# A Comparative Neural Radiance Field (NeRF) 3D Analysis of Camera Poses from
HoloLens Trajectories and Structure from Motion

Miriam Jäger Corresponding author Patrick Hübner  Dennis Haitz  Boris Jutzi
Institute of Photogrammetry and Remote Sensing (IPF)
Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
(miriam.jaeger, patrick.huebner, dennis.haitz<EMAIL_ADDRESS>

###### Abstract

Neural Radiance Fields (NeRFs) take a set of camera poses with associated
images as input and train a Neural Network which estimates a density value and
color values for each position. The position-dependent learning of density is
of particular interest for photogrammetry. The coordinate system of the
trained NeRF can be retrieved and filtered according to its density on
objects, which enables a 3D reconstruction. Most commonly, traditional methods
like Structure from Motion (SfM) are used to calculate the camera poses in
pre-processing needed for training NeRFs. From this perspective, the HoloLens
provides an interesting interface, that enables the extraction of the required
input data for NeRFs, the camera poses and associated sensor RGB images. In
this work we presented a workflow for the extraction of high resolution 3D
reconstructions almost directly from HoloLens data under the application of
NeRFs. On the one hand, internal camera poses retrieved from the HoloLens
trajectory via a server application, On the other hand, camera poses via
Structure from Motion were considered. For both camera poses, an enhanced
variant was additionally applied by a pose refinement while training. The
results show that the internal camera poses lead to convergence of the NeRF
after a simple rotation around the x-axis and enable a 3D reconstruction.
Improvements in training and reconstruction can be achieved by pose refinement
while training. This enables a comparable quality in the training process and
resulting point cloud as achieved by external camera poses in pre-processing
caculated using Structure from Motion. In general all NeRF reconstructions
outperform the conventional photogrammetric method using Multi-View Stereo.

###### Abstract

Neural Radiance Fields (NeRFs) are trained using a set of camera poses and
associated images as input to estimate density and color values for each
position. The position-dependent density learning is of particular interest
for photogrammetry, enabling 3D reconstruction by querying and filtering the
NeRF coordinate system based on the object density. While traditional methods
like Structure from Motion are commonly used for camera pose calculation in
pre-processing for NeRFs, the HoloLens offers an interesting interface for
extracting the required input data directly. We present a workflow for high-
resolution 3D reconstructions almost directly from HoloLens data using NeRFs.
Thereby, different investigations are considered: Internal camera poses from
the HoloLens trajectory via a server application, and external camera poses
from Structure from Motion, both with an enhanced variant applied through pose
refinement. Results show that the internal camera poses lead to NeRF
convergence with a PSNR of 25 dB with a simple rotation around the x-axis and
enable a 3D reconstruction. Pose refinement enables comparable quality
compared to external camera poses, resulting in improved training process with
a PSNR of 27 dB and a better 3D reconstruction. Overall, NeRF reconstructions
outperform the conventional photogrammetric dense reconstruction using Multi-
View Stereo in terms of completeness and level of detail.

###### keywords:

Neural Radiance Fields, Microsoft HoloLens, Structure from Motion, Trajectory,
3D Reconstruction, Point Cloud

##  1 INTRODUCTION

With the pioneering research on Neural Radiance Fields (NeRFs) (Mildenhall et
al.,, 2020), that enable the rendering of new views with the so-called view
synthesis based on image data and associated camera poses in space, a new
epoch in computer graphics started. Novel inventions like Instant NGP (Müller
et al.,, 2022) advanced NeRFs once again, as they reduce training and
rendering time to minutes or even seconds.

However, these methods also arouse interest beyond the field of computer
graphics for research and development in photogrammetry. NeRFs take a sparse
set of camera poses with associated images as input and train a Neural Network
which estimates a density value $\delta$ and color values c = (R,G,B) for each
position X = (x,y,z). The position-dependent learning of density, and color
values, is of particular interest for photogrammetry with regard to mobile 3D
mapping. We consider density as a kind of pseudo-probability for the
occurrence of an object in 3D space. Thus, the positions of the trained NeRF
can be accessed and filtered by their density on objects, which allows the
extraction of 3D point clouds of the scene.

Most commonly, traditional methods like Structure from Motion (SfM) are used
to calculate the camera poses in pre-processing needed for training the NeRFs.
From this perspective, the HoloLens provides an interesting interface, that
enables the extraction of the required input data, the camera poses and
associated sensor RGB images. This allows to create a 3D reconstruction,
including color information, nearly directly from the sensor data.

In this study, we investigate whether the trajectory from the HoloLens is
sufficient to achieve convergence of the NeRF and the potential to achieve a
3D reconstruction of the scene based on the density values of the NeRF trained
with HoloLens data. Four different types of camera poses using the same
HoloLens images are compared regarding their training process and the
resulting 3D reconstructions. On the one hand, the internal HoloLens camera
poses, as well as a including pose refinement during training are considered.
On the other hand, external generated camera poses via SfM, as well as
including the pose refinement are investigated. In order to compare the point
clouds based on NeRFs with traditional methods, a dense Multi-View Stereo
(MVS) point cloud from the camera poses is reconstructed.

Firstly, it was demonstrated that the internal HoloLens camera poses and
images are suitable for the convergence of the NeRF, as shown by the
quantitative results of the training process in Figure 4. After a simple
rotation around the x-axis, the convergence occurs from approximately 20,000
training epochs with a Peak-Signal-to-Noise-Ratio (PSNR) of 25 dB. Secondly,
the trained NeRF is suitable for a 3D reconstruction of the scene, as shown by
the qualitative results in Figure 5. After additional training of extrinsics
alias pose refinement, HoloLens camera poses lead to comparable PSNR values of
27 dB and a comparable 3D reconstruction with respect to the separate pose
determination via SfM in pre-processing. Furthermore, the reconstruction from
the NeRF exhibited advantages over a conventional dense 3D reconstruction
through MVS. The reconstructions from the NeRF using SfM and internal HoloLens
camera poses lead to a higher point density, less artefacts, and a better
mapping of untextured surfaces in terms of completeness than MVS.

##  2 RELATED WORK

In this section, we briefly summarize related work to our study. Thereby we
give an overview on basic and recent research and development on NeRFs.

The foundation for the Neural Radiance Fields (NeRFs) was established by the
Scene Representation Networks (SNR) (Sitzmann et al.,, 2019). Their underlying
principle is modeling the scene as a function of 3D coordinates within it. It
was followed by the groundbreaking research work of Neural Radiance Fields
(Mildenhall et al.,, 2020). These enable estimation of color values and
densities for each 3D coordinate through 6D camera poses and associated 2D
images by learning a deep neural network with multi-layer perceptrons (MLPs).
The initial NeRF was followed by thousands of publications driving research
and development in various domains.

To address scalability, the augmentation to large scale scenes is achieved by
Mega-NeRF (Turki et al.,, 2022) using a data partitioning based on visibility
analysis or Block-NeRF (Tancik et al.,, 2022) with distance-dependent
partitioning based on street segments. Other approaches, such as Bundle
Adjusting Radiance Fields (BaRF) (Lin et al.,, 2021) and Gaussian Activated
Radiance Fields (GaRF) (Chng et al.,, 2022), address the purpose of a camera
pose estimation. In addition to neural methods, non-neural research like
Plenoxels (Fridovich-Keil et al.,, 2022) have been proposed. Dynamic
contributions, such as (Pumarola et al.,, 2021), utilize time as an additional
input dimension for time-dependent rendering of novel images, while (Gao et
al.,, 2021) employ time components for preventing the occurrence of artifacts
caused by dynamic pixels. Furthermore, 3D reconstruction from NeRFs are
considered (Rosinol et al.,, 2022; Oechsle et al.,, 2021). Methods such as
AdaNeRF (Kurz et al.,, 2022), FastNeRF (Garbin et al.,, 2021) and Instant NGP
(Müller et al.,, 2022) aim to improve rendering or training efficiency.
Thereby Instant NGP, which we utilize in our study, uses a combination of
small MLPs and spatial hash table encoding for real-time training and
rendering.

##  3 METHODOLOGY

In Section 3.1 the principles of the methods used to generate the input data
according to Figure 1 for the NeRFs are presented. The essential
transformations of the internal HoloLens camera poses in order to use them for
the NeRFs are explained. After that, the standard method used to determine the
camera poses in pre-processing is introduced. Subsequently, in Section 3.2 the
methodology for the extraction of a 3D point cloud from NeRFs as well as the
used conventional photogrammetric reconstruction are described.

Figure 1: Flowchart of the applied investigations. Input data are two
different types of camera poses: internal poses of the HoloLens and externally
calculated poses via SfM. Subsequently, in each case a pose refinement variant
is performed during the training. The four resulting point clouds are
extracted from the NeRF using a global density threshold.

###  3.1 Camera Poses of the Trajectories

#### Transformation of the HoloLens camera poses

As the first and central step, the HoloLens camera poses have to be
transformed into a compatible input format for the NeRF. Such a format is
given by the representation as a so-called view matrix
$\text{T}_{\text{view}}$, which is a 4$\times$4 transformation matrix in the
form of homogeneous coordinates. It consists of translation, rotation and
scaling. The input to the used NeRF follows the
OpenGL111https://learnopengl.com/Getting-started/OpenGL
(last access 01/03/2023) convention, with the camera placed in a right-handed
coordinate system, the positive z-axis pointing away from the camera, and the
positive x-axis pointing to the right when looking through the camera lens.
The y-axes must face the global z-axis, since they correspond to the so-called
up-vector. As the z-axis of the HoloLens camera poses
$\text{T}_{\text{HoloLens}}$ is orientated in the direction of the z-axis in
the global coordinate system, see Figure 2, a transformation by 90 degrees
around the global x-axis is required as shown in Figure 2 by:

$\displaystyle\text{T}_{\text{view}}$
$\displaystyle=\text{T}_{\text{x,$\alpha$}}\text{T}_{\text{HoloLens}},$ (1)

with

$\displaystyle\text{T}_{\text{x,$\alpha=90^{\circ}$}}$
$\displaystyle=\begin{bmatrix}1&0&0&0\\\
0&\text{cos($\alpha$)}&\text{-sin($\alpha$)}&0\\\
0&\text{sin($\alpha$)}&\text{cos($\alpha$)}&0\\\ 0&0&0&1\end{bmatrix}$ (2)
$\displaystyle=\begin{bmatrix}1&0&0&0\\\ 0&0&$-$1&0\\\ 0&1&0&0\\\
0&0&0&1\end{bmatrix}.$

Figure 2: Camera poses for the trajectory from SfM (trajectory top) via the
transformations of Instant NGP implementation versus the internal camera poses
from HoloLens (trajectory bottom): 2 shows the external versus the original
internal camera poses, 2 shows the transformation of the HoloLens camera poses
by rotating them 90 degrees around the global x-axis, 2 shows the translation
to the center of attention and 2 shows scaling to the NeRF coordinate system.

Two additional transformations are performed for translation and scaling,
based on those in the Instant NGP
implementation222https://github.com/NVlabs/instant-ngp
(last access 09/12/2022). Firstly, for translation, the transformation
matrices are transformed to a common focal point (center of attention), as
shown in Figure 2. For each camera pair, the intersection point between the
optical axes is computed, resulting in the focal point. Afterwards the point
is subtracted from the current position of the camera, aligning the cameras
towards the focal point. This allows the camera poses to be used for the
visualization of the object in focus.

Subsequently, scaling is performed on the camera poses to the size of the NeRF
coordinate system, see Figure 2. First, the average distance of all cameras
from the origin is calculated by computing the Euclidean norm of the
displacement vectors of the camera transformations and summing them. This
value is divided by the number of camera transformations to obtain the average
distance. The scaling factor is then computed by dividing the camera distances
by the average distance and multiplying it with a factor, which is set to 4 by
Instant NGP. Final scaling is performed by multiplying each transformation
matrix with the final scaling factor.

#### Structure from Motion

Structure from Motion (SfM) generally describes the procedure of
reconstructing a 3D scene from a set of images taken from different directions
and positions. It relies on the calculation and matching of point
correspondences within an image sequence from overlapping images by using
methods such as SIFT (Lowe,, 2004). In this study, the (incremental) Structure
from Motion technique by (Schönberger and Frahm,, 2016) is used for the
external calculation of the camera poses.

###  3.2 3D Reconstruction

#### NeRF

NeRFs enable novel view synthesis of scenes. However, from the perspective of
photogrammetry, instead of rendering new 2D views (Mueller et al.,, 2019), we
are interested in the 3D geometry and corresponding color values of the scene.
We consider the density as a kind of pseudo-probability for the occurrence of
a surface in 3D space. Considering, positions with high densities indicate a
high possibility to be an object point. In the first step, uniform sampling of
the density field is achieved by sampling density and color values in the
coordinate system of the trained NeRF at equidistant sampling points in a
bounding box. In the second step, we filter the positions X = (x,y,z) with
high density values using a global threshold $\delta_{\text{t}}$. Thereby, we
assume that object points maintain a higher density $\delta>\delta_{\text{t}}$
compared to non-object points.

For the 3D mapping we investigate 3D reconstructions of our scene based on
four different types of input data, as shown in Figure 1. We use the internal
HoloLens camera poses and external camera poses calculated by a SfM workflow
as described in Section 3.1. In addition to the internal and external camera
poses and corresponding images, the pose refinement of the Instant NGP
implementation is used. The implementation requires initial poses in order to
refine them and is unable to compute poses completely from the scratch. By
using the camera pose as an additional variable in the training process, it
propagates gradients back onto the camera parameters in order to minimize the
loss.

#### Multi-View Stereo

In order to compare the reconstructions from NeRFs with a reconstruction from
a conventional method, we use a classical Multi-View Stereo (MVS) pipeline
(Schönberger and Frahm,, 2016) on the basis of the output of the Structure
from Motion in Section 3.1. Accordingly, the same HoloLens camera poses as for
the NeRFs serve as input here, which makes the reconstructions comparable in
the same coordinate system. On the one hand, we use the output of SfM for a
sparse reconstruction. On the other, hand we generate a dense reconstruction
with MVS. MVS (Schönberger et al.,, 2016) takes the information from the
sparse model from SfM to for pixelwise computation of depth information in an
image.

##  4 Dataset

Our expertiments are based on a dataset captured by the Microsoft HoloLens,
which includes an indoor scene of a plant (Ficus) on a plane surface, see
Figure 3. The HoloLens provides an interesting interface for the NeRF, as it
generates the required input data, camera poses and associated sensor images.
In general, HoloLens, developed by Microsoft and firstly released in 2018,
embodies the world’s first fully autonomous holographic computer and has
become an important device for all kinds of applications, such as 3D mapping
and modelling of indoor scenes (Weinmann et al.,, 2020; Weinmann et al.,,
2021).

Figure 3: Visualization of an image of the captured Ficus plant as our
measurement object using the Microsoft HoloLens RGB camera. Figure 4:
Comparison of the Peak-Signal-to-Noise-Ratio (PSNR) in $\mathrm{dB}$
$\uparrow$ and loss $\downarrow$ during the training processes. The red curves
show the PSNR, the blue curves the loss. The HoloLens images with internal
HoloLens camera poses, internal HoloLens camera poses with pose refinement,
external (SfM) camera poses, and external (SfM) camera poses with pose
refinement are considered.

Figure 5: Comparison of the 3D reconstructions from NeRFs using a global
density threshold $\delta_{\text{t=15}}$. For HoloLens images and 5 internal
camera poses, 5 with pose refinement and 5 external camera poses, 5 with pose
refinement. Compared to the 5 sparse and 5 dense point cloud from external
camera poses with MVS.

HoloLens333https://www.microsoft.com/en-us/hololens/hardware
(last access 02/20/2023) generation 2 was released in 2019 and features
improved camera technology compared to the first generation such as higher
resolution and better color depth, resulting in sharper and more detailed
images.

The HoloLens 2 server application (Dibene and Dunn,, 2022) is used for
requesting the data in the HoloLens. The system provides access to all the
HoloLens 2 sensors, including the images from the 1920 $\times$ 1080
photo/video RGB camera and corresponding camera pose of the device in 3D
space. In addition, device calibration data can be retrieved by the internal
orientation (camera intrinsics). The HoloLens images and corresponding camera
poses were captured with a hemispherical camera framing. Thereby step sizes of
32 scanning points at a height of about 120 cm, with two different viewing
angles have been employed, which results in a total of 64 images.

##  5 EXPERIMENTS AND RESULTS

In this section, we present our experiments and results by a quantitative
evaluation on analyzing the training process in Section 5.1. This is followed
by a qualitative analysis in Section 5.2 of the resulting 3D reconstructions.
We investigate the impact of the camera poses in general, the comparison of
point clouds from NeRFs trained with different input sets based on HoloLens
data in Section 4 as well as photogrammetric reconstructions.

###  5.1 Training

The training process of the NeRFs proceeds differently based on the chosen
configurations, as shown in Figure 4. In particular, we use the Peak-Signal-
to-Noise-Ratio (PSNR) in $\mathrm{dB}$ between the input RGB images of the
training data and the rendered images for the accuracy measurement while
training. Comparing the training, both the internal HoloLens and the external
SfM camera poses lead to a convergence of the NeRF. This occurs at
approximately 20,000 training epochs, which corresponds to a duration of
between 2 and 5 min training. The internal HoloLens camera poses achieve best
results of about 25 dB. In contrast, more than 27 dB can be achieved with the
external camera poses from SfM in pre-processing. Remarkably, pose refinement
by training the extrinsics can increase the performance for the internal
camera poses to a comparable level of about 27 dB. However, no further
increase in PSNR is achieved for the external camera poses by pose refinement.
For all configurations, the loss behaves inversely proportional. Based on
these training results, conclusions can be drawn about the relative precisions
of the different type of camera poses.

###  5.2 3D Reconstruction

Finally, Figure 5 compares the 3D reconstructions from the NeRFs trained on
different input data by using a global threshold $\delta_{\text{t=15}}$, and
the sparse and dense point clouds. In general, 3D reconstructions from NeRF on
HoloLens data can be generated directly with the internal camera poses as well
as from the external camera poses calculated via SfM.

The visual quality of the reconstructions corresponds to the achieved PSNR
values in Figure 4. This is particularly evident from the artifacts in the
reconstruction from HoloLens internal camera poses with no pose refinement in
Figure 5. The training course also shows a lower maximum PSNR of 25 dB
compared to those of the other three training processes with PSNR values of 27
dB. In this case, artifacts are located in empty space. This effect rarely
occurs during 3D reconstruction based on the internal camera poses with pose
refinement, as Figure 5 shows. The external camera poses provide adequate
qualitative reconstruction results without in Figure 5 and with pose
refinement in Figure 5. Only small artifacts disappear with pose refinement.
Overall, all input data provide sufficient 3D reconstructions from NeRFs with
minor color differences. In particular, the surface of the pot of the plant
can be reconstructed well using NeRFs.

In contrast, the MVS approach does not provide a complete reconstruction of
the object, which is especially noticeable on the pot. This occurs for both
the sparse reconstruction in Figure 5 and dense reconstruction in Figure 5.
Additionally, the dense point cloud contains gray artifacts at the fine
structures of the branches.

##  6 DISCUSSION

This research investigates the application of camera poses from Microsoft
HoloLens trajectories for 3D reconstruction. On the one hand, internal camera
poses directly retrieved from the HoloLens trajectory via a server application
have been investigated. On the other hand, external camera poses that were
calculated in the conventional manner via Structure from Motion were
considered. For both scenarios, an enhanced pose refinement was additionally
applied by training the camera extrinsics.

It could be demonstrated that, after a simple rotation around the x-axis, the
internal HoloLens camera poses are sufficient for NeRF convergence in
approximately 20,000 training epochs. This enables a 3D reconstruction using
NeRF coordinates by sampling. Four investigations are considered as input for
the corresponding images: The internal HoloLens camera poses, external camera
poses from SfM, both with and without pose refinement. Overall, the results
show varying quantitative and qualitative performance in training and 3D
reconstruction based on the utilized camera poses. Considering the training
process the unrefined internal HoloLens camera poses provide PSNR of about 25
dB. With pose refinement of the internal camera poses, the training process
improves to about 27 dB. This is comparable with the external camera poses
from SfM, which achieve higher PSNR values of around 27 dB, both unrefined and
refined. We assume that improved poses lead to a superior training process in
terms of the PSNR values and consequently better 3D reconstructions, which is
confirmed by the qualitative results. Thereby the unrefined internal HoloLens
camera poses contains more huge artifacts in the 3D reconstruction. However,
by pose refinement of the internal camera poses, the artifacts are reduced,
and the reconstruction is comparable to the reconstruction from external
calculated camera poses. Each external camera poses without and with pose
refinement, contain only a few small artifacts. We suggest that the externally
calculated poses are already quite accurate and therefore do not improve
further with pose refinement.

Nevertheless, the results from 3D mapping using NeRF are notably superior to
the classical photogrammetric method of dense Multi-View stereo (MVS)
reconstruction from camera poses via SfM for our dataset. The NeRF
reconstructions yield better results on untextured, homogeneous surfaces. This
is especially evident for the pot of the plant, which apparently fails to be
reconstructed with the conventional MVS. In addition, fine structures in MVS
reconstruction contain gray artifacts, as can be seen in the branches of the
plant and an inferior level of detail. Some color differences within the NeRF
reconstructions are caused by the directionality of color in the NeRFs, as
opposed to density. However, the color differences are minor and do not harm
the overall impression of the reconstruction.

##  7 CONCLUSION

In this paper, we presented a workflow for the extraction of high resolution
3D reconstructions almost directly from Microsoft HoloLens data under the
application of Neural Radiance Fields (NeRFs). Thereby, the impact of the
camera poses has been investigated using a quantitative analysis by
considering the training process, as well as a qualitative analysis by
regarding the final 3D reconstructions.

We demonstrated that the internal HoloLens camera poses und corresponding
images as input data are able to provide convergence of the NeRF during
training. This enables the generation of a 3D reconstruction from positions
with high density values in the NeRF coordinate system. Improvements in the
training process and resulting 3D reconstruction can be achieved by pose
refinement while training the NeRF. This enables a comparable quality in the
training process and resulting point cloud as achieved by external camera
poses calculated in pre-processing using approaches such as Structure from
Motion. It demonstrates the impact of the camera poses on the quality of the
3D reconstruction. In addition, among all pose investigations, the NeRF
reconstructions outperform the conventional photogrammetric method using
Multi-View Stereo.

In summary, the combination of internal HoloLens camera poses and associated
images with NeRFs offers an immense potential for enabling highly detailed,
colored, mobile 3D mappings of a scene in a straightforward workflow. In
future work, we suggest using a 3D region growing algorithm instead of a
global density threshold in terms of artifacts removal, assuming that all
object points in the scene are spatially connected.

## References

  * Chng et al., (2022) Chng, S.-F., Ramasinghe, S., Sherrah, J., Lucey, S., 2022. Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation. _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII_ , Springer, 264–280.
  * Dibene and Dunn, (2022) Dibene, J. C., Dunn, E., 2022. HoloLens 2 Sensor Streaming. arXiv. https://arxiv.org/abs/2211.02648.
  * Fridovich-Keil et al., (2022) Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., Kanazawa, A., 2022\. Plenoxels: Radiance fields without neural networks. _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5501–5510.
  * Gao et al., (2021) Gao, C., Saraf, A., Kopf, J., Huang, J.-B., 2021. Dynamic view synthesis from dynamic monocular video. _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5712–5721.
  * Garbin et al., (2021) Garbin, S. J., Kowalski, M., Johnson, M., Shotton, J., Valentin, J., 2021. Fastnerf: High-fidelity neural rendering at 200fps. _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 14346–14355.
  * Kurz et al., (2022) Kurz, A., Neff, T., Lv, Z., Zollhöfer, M., Steinberger, M., 2022. Adanerf: Adaptive sampling for real-time rendering of neural radiance fields. _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVII_ , Springer, 254–270.
  * Lin et al., (2021) Lin, C.-H., Ma, W.-C., Torralba, A., Lucey, S., 2021. Barf: Bundle-adjusting neural radiance fields. _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5741–5751.
  * Lowe, (2004) Lowe, D. G., 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60, 91–110.
  * Mildenhall et al., (2020) Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., Ng, R., 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. _European Conference on Computer Vision (ECCV)_ , 405–421.
  * Mueller et al., (2019) Mueller, M. S., Sattler, T., Pollefeys, M., Jutzi, B., 2019. Image-to-image translation for enhanced feature matching, image retrieval and visual localization. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 4, 111–119.
  * Müller et al., (2022) Müller, T., Evans, A., Schied, C., Keller, A., 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics, 41(4), 102:1-15.
  * Oechsle et al., (2021) Oechsle, M., Peng, S., Geiger, A., 2021. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5589–5599.
  * Pumarola et al., (2021) Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F., 2021. D-nerf: Neural radiance fields for dynamic scenes. _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10318–10327.
  * Rosinol et al., (2022) Rosinol, A., Leonard, J. J., Carlone, L., 2022. Nerf-slam: Real-time dense monocular slam with neural radiance fields.
  * Schönberger and Frahm, (2016) Schönberger, J. L., Frahm, J.-M., 2016. Structure-from-motion revisited. _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Schönberger et al., (2016) Schönberger, J. L., Zheng, E., Frahm, J.-M., Pollefeys, M., 2016. Pixelwise view selection for unstructured multi-view stereo. _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14_ , Springer, 501–518.
  * Sitzmann et al., (2019) Sitzmann, V., Zollhoefer, M., Wetzstein, G., 2019. Scene representation networks: Continuous 3d-structure-aware neural scene representations. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, R. Garnett (eds), _Advances in Neural Information Processing Systems_ , 32, Curran Associates, Inc.
  * Tancik et al., (2022) Tancik, M., Casser, V., Yan, X., Pradhan, S., Mildenhall, B., Srinivasan, P. P., Barron, J. T., Kretzschmar, H., 2022. Block-nerf: Scalable large scene neural view synthesis. _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 8248–8258.
  * Turki et al., (2022) Turki, H., Ramanan, D., Satyanarayanan, M., 2022. Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12922–12931.
  * Weinmann et al., (2020) Weinmann, M., Jäger, M. A., Wursthorn, S., Jutzi, B., Hübner, P., 2020. 3D indoor mapping with the Microsoft HoloLens: qualitative and quantitative evaluation by means of geometric features. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 1, 165–172.
  * Weinmann et al., (2021) Weinmann, M., Wursthorn, S., Weinmann, M., Hübner, P., 2021. Efficient 3D Mapping and Modelling of Indoor Scenes with the Microsoft HoloLens: A Survey. Journal of photogrammetry, remote sensing and geoinformation science, 89, 319-333.


# HyperMAML: Few-Shot Adaptation of Deep Models with Hypernetworks

M. Przewięźlikowski, P. Przybysz, J. Tabor, M. Zięba, P. Spurek
Faculty of Mathematics and Computer Science,
Jagiellonian University, Kraków, Poland
<EMAIL_ADDRESS>

###### Abstract

The aim of Few-Shot learning methods is to train models which can easily adapt
to previously unseen tasks, based on small amounts of data. One of the most
popular and elegant Few-Shot learning approaches is Model-Agnostic Meta-
Learning (MAML). The main idea behind this method is to learn the general
weights of the meta-model, which are further adapted to specific problems in a
small number of gradient steps. However, the model’s main limitation lies in
the fact that the update procedure is realized by gradient-based optimisation.
In consequence, MAML cannot always modify weights to the essential level in
one or even a few gradient iterations. On the other hand, using many gradient
steps results in a complex and time-consuming optimization procedure, which is
hard to train in practice, and may lead to overfitting. In this paper, we
propose HyperMAML, a novel generalization of MAML, where the training of the
update procedure is also part of the model. Namely, in HyperMAML, instead of
updating the weights with gradient descent, we use for this purpose a
trainable Hypernetwork. Consequently, in this framework, the model can
generate significant updates whose range is not limited to a fixed number of
gradient steps. Experiments show that HyperMAML consistently outperforms MAML
and performs comparably to other state-of-the-art techniques in a number of
standard Few-Shot learning benchmarks.

##  1 Introduction

In the typical Few-Shot learning setting, the aim is to adapt to new tasks
under the assumption that only a few examples are given. As we know, people
typically learn new tasks easily by using only a few training examples. On the
contrary, a standard deep neural network must be trained on an extensive
amount of data to obtain a similar accuracy. Thus, the aim of Few-Shot
learning models is to bring neural networks closer to the human brain’s
capabilities. The most famous and, in our opinion, the most elegant approach
to Few-Shot learning is Model-Agnostic Meta-Learning (MAML) (Finn et al.,
2017), where the model is trained to adapt universal weights to new Few-Shot
learning tasks quickly. It seems that the brain’s neural networks can adapt to
new tasks too, by applying the fact that during the process of evolution, some
of its parts have developed universal weights which are easily adaptable to
typical tasks we encounter in real life. Thus the idea behind MAML gives us a
possible insight into the working of the brain.

The fascinating factor of human intelligence is that the human learning
process, although still not understood, is clearly not based on the gradient
descent algorithm, as we cannot in general backpropagate the information
(Lillicrap et al., 2020; Song et al., 2020; Whittington and Bogacz, 2019).
Thus, from the biological point of view, the main limitation of MAML is the
fact that it uses the gradient descent method for weight updates. The main
research problem that we set ourselves is whether one can modify MAML to be
more biologically feasible, i.e. keep its ability to find universal weight but
remove the necessity of using gradient-based update methods.

Figure 1:  Let us consider a two-dimensional dataset consisting of four
Gaussian data clusters (the first column). In the Meta-Learning scenario, we
produce a task that consists of samples from two horizontal or vertical
ellipses with permuted labels (columns 2-5). The MAML model with one gradient
step in the inner loop cannot update all four tasks (see the first row). In
practice, in the case of high-dimensional data, using many gradient updates in
MAML introduces significant computational overhead (see Section 4.3). Our
HyperMAML (pictured in the second row) can solve the task by using the
Hypernetwork paradigm. We use only a single update in our method, which can
dramatically change weights. We include the details of this experiment in the
supplementary material.

We solve this problem by constructing HyperMAML111We make our implementation
available at https://github.com/gmum/few-shot-hypernets-public., a model which
replaces the gradient optimization in the update of weights by trainable
update procedure with the use of the Hypernetwork paradigm. Hypernetworks,
introduced in (Ha et al., 2016) are defined as neural models that generate
weights for a separate target network solving a specific task. In our model,
HyperMAML, the Hypernetwork aggregates the information from the support set
and produces an update to the main model. Thanks to such an approach, we can
create various types of updates that are not limited to a few gradient steps.
In practice, MAML works when there exist universal weights that are close
enough to the optimal solution for each task. To visualize such a situation,
we present a simple 2D example, see Fig. 1 where a single gradient update
fails to sufficiently adapt the model to a given task. We cannot effectively
switch weight in one gradient step. On the other hand, when we use many
gradient steps, we obtain a complex optimization procedure that uses an inner
and outer loop. Such a procedure can be seen as second-order optimization,
which is complex to train (Finn et al., 2017). Contrary to MAML we do not need
an inner loop in the optimization procedure, and consequently, we do not have
second-order optimization. We also reduce the number of hyperparameters, which
are used in the inner loop of MAML approach, which would need to be tuned in a
grid search. As a result, our algorithm obtains better results than the
classical MAML algorithm and produces results comparable to other state-of-
the-art algorithms.

The contributions of our work can be summarized as follows:

  * •

We introduce HyperMAML, a novel approach to the Few-Shot learning problem by
aggregating information from the support set and directly producing weights
updates.

  * •

In HyperMAML, we do not use loss calculation or gradient backpropagation for
the update to the new task, thus making the model more biologically feasible
and computationally efficient.

  * •

We significantly increase the update ability compared to the classical MAML
algorithm, as evidenced by the increased accuracy in numerous benchmarks we
perform.

##  2 Related Work

The problem of Meta-Learning and Few-Shot learning (Hospedales et al., 2020;
Schmidhuber, 1992; Bengio et al., 1992) has received a growing amount of
attention from the scientific community over the recent years, with the
abundance of methods emerging as a result. They can be roughly categorized
into several groups:

#### Model-based methods

aim to adapt to novel tasks quickly by utilizing mechanisms such as memory
(Ravi and Larochelle, 2017; Santoro et al., 2016; Mishra et al., 2018; Zhen et
al., 2020), Gaussian Processes (Rasmussen, 2003; Patacchiola et al., 2020;
Wang et al., 2021), or generating fast weights based on the support set with
set-to-set architectures (Qiao et al., 2017; Bauer et al., 2017; Ye et al.,
2018). The fast weights approaches can be interpreted as using Hypernetworks
(Ha et al., 2016) – models which learn to generate the parameters of neural
networks performing the designated tasks.

Similarly, HyperMAML utilizes a Hypernetwork to generate weights updates for
performing specific tasks. The key difference is that in HyperMAML, the
Hypernetwork is not the sole source of model weights, but is used to generate
the updates to the weights of the base model.

#### Metric-based methods

learn a transformation to a feature space where the distance between examples
from the same class is small. The earliest examples of such methods are
Matching Networks (Vinyals et al., 2016) and Prototypical Networks (Snell et
al., 2017). Subsequent works show that metric-based approaches can be improved
by techniques such as learnable metric functions (Sung et al., 2018),
conditioning the model on tasks (Oreshkin et al., 2018) or predicting the
parameters of the kernel function to be calculated between support and query
data with Hypernetworks (Sendera et al., 2022). HyperMAML does not consider
relations between the support and query examples during inference and
therefore does not fall into the category of metric-based models.

#### Optimization-based methods

such as MetaOptNet (Lee et al., 2019) are based on the idea of an optimization
process over the support set within the Meta-Learning framework. Arguably, the
most popular of this family of methods is Model-Agnostic Meta-Learning (MAML)
(Finn et al., 2017), which inspired a multitude of research and numerous
extensions to the original algorithm. This includes various techniques for
stabilizing its training and improving performance, such as Multi-Step Loss
Optimization, and scheduling the learning rate of the meta-optimizer (Antoniou
et al., 2018) , or using the Bayesian variant of MAML (Yoon et al., 2018).

Due to a need for calculating second-order derivatives when computing the
gradient of the meta-training loss, training the classical MAML introduces a
significant computational overhead. The authors show that in practice the
second-order derivatives can be omitted at the cost of small gradient
estimation error and minimally reduced accuracy of the model (Finn et al.,
2017; Nichol et al., 2018). Methods such as iMAML and Sign-MAML propose to
solve this issue with implicit gradients or Sign-SGD optimization (Rajeswaran
et al., 2019; Fan et al., 2021). The optimization process can also be improved
by training not only the base initialization of the model but also the
optimizer itself – namely, training a neural network that transforms gradients
calculated w.r.t. loss of the support set predictions into weight updates
(Munkhdalai and Yu, 2017; Munkhdalai et al., 2018; Li et al., 2017;
Rajasegaran et al., 2020).

HyperMAML shares a key characteristic with the optimization-based methods –
namely, it also utilizes a base set of weights, which are updated to obtain a
model fit for a given task. The key difference between HyperMAML and MAML is
that while MAML adapts to novel tasks through multiple steps of gradient-based
optimization, HyperMAML generates the updates in a single step using a
Hypernetwork. This makes HyperMAML more similar to methods like (Li et al.,
2017; Munkhdalai and Yu, 2017), which generate weight updates through trained
meta-optimizers. However, contrary to those approaches, in HyperMAML the
Hypernetwork predicts the weight updates based on (i) latent representation of
the support set, (ii) predictions of the base model for the support set, (iii)
ground-truth labels of the support examples (see Fig. 2). Thus HyperMAML does
not require calculating either the loss function or its gradients during
generating of the task-specific weight updates, making it more computationally
efficient.

##  3 HyperMAML: Hypernetwork for Few-Shot learning

In this section, we present our HyperMAML model for Few-Shot learning. First,
we start by presenting background and notations for Few-Shot learning. Then we
describe how the MAML algorithm works. Finally, we present HyperMAML, which
can be understood as an extension of the classical MAML.

###  3.1 Background

#### The terminology

describing the Few-Shot learning setup is dispersive due to the colliding
definitions used in the literature. Here, we use the nomenclature derived from
the Meta-Learning literature, which is the most prevalent at the time of
writing. Let $\mathcal{S}=\\{(\mathbf{x}_{l},\mathbf{y}_{l})\\}_{l=1}^{L}$ be
a support-set containing input-output pairs, with $L$ examples with the equal
class distribution. In the one-shot scenario, each class is represented by a
single example, and $L=K$, where $K$ is the number of the considered classes
in the given task. Whereas, for Few-Shot scenarios, each class usually has
from $2$ to $5$ representatives in the support set $\mathcal{S}$.

Let $\mathcal{Q}=\\{(\mathbf{x}_{m},\mathbf{y}_{m})\\}_{m=1}^{M}$ be a query-
set (sometimes referred to in the literature as a target-set), with $M$
examples, where $M$ is typically one order of magnitude greater than $K$. For
clarity of notation, the support and query sets are grouped in a task
$\mathcal{T}=\\{\mathcal{S},\mathcal{Q}\\}$. During the training stage, the
models for Few-Shot applications are fed by randomly selected examples from
training set $\mathcal{D}=\\{\mathcal{T}_{n}\\}^{N}_{n=1}$, defined as a
collection of such tasks.

During the inference stage, we consider task
$\mathcal{T}_{*}=\\{\mathcal{S}_{*},\mathcal{X}_{*}\\}$, where
$\mathcal{S}_{*}$ is a support set with the known class values for a given
task, and $\mathcal{X}_{*}$ is a set of query (unlabeled) inputs. The goal is
to predict the class labels for query inputs $\mathbf{x}\in\mathcal{X}_{*}$,
assuming support set $\mathcal{S}_{*}$ and using the model trained on
$\mathcal{D}$.

Algorithm 1 MAML - Model-Agnostic Meta-Learning (Finn et al., 2017)

Require: $\mathcal{D}=\\{\mathcal{T}_{n}\\}_{n=1}^{N}$: set of training tasks
Require: $\alpha$, $\beta$: step size hyper parameters

1:randomly initialize $\theta$

2:while not done do

3: Sample batch of tasks $\mathcal{B}$ from $\mathcal{D}$

4: for each task $\mathcal{T}_{i}=\\{\mathcal{S}_{i},\mathcal{Q}_{i}\\}$ from
batch $\mathcal{B}$ do

5: Evaluate $\nabla_{\theta}\mathcal{L}_{S_{i}}(f_{\theta})$ with respect to
examples from $\mathcal{S}_{i}$ given loss by (2)

6: Compute adapted parameters $\theta^{\prime}_{i}$ with gradient descent
using formula given by eq. (1)

7: end for

8: Update the global parameters of the model $\theta$ with formula given by
eq. (4).

9:end while

#### Model-Agnostic Meta-Learning (MAML)

is one of the current standard algorithms for Few-Shot learning, which learns
the parameters of a model so that it can adapt to a new task in a few gradient
steps.

We consider a model represented by a function $f_{\theta}$ with parameters
$\theta$. In the Few-Shot problems $f_{\theta}$ models discriminative
probabilities for the classes,
$f_{\theta}(\mathbf{x})=p(\mathbf{y}|\mathbf{x},\theta)$. The standard MAML
model is trained with the procedure given by Algorithm 1. In each of the
training iterations the batch of tasks $\mathcal{B}$ is sampled from
$\mathcal{D}$. Further, for each task
$\mathcal{T}_{i}=\\{\mathcal{S}_{i},\mathcal{Q}_{i}\\}$ from $\mathcal{B}$
MAML adapts the model’s parameters $\theta^{\prime}_{i}$ that are specific for
a given task. The actual values of $\theta^{\prime}_{i}$ are calculated using
one or more gradient descent updates. In the simplest case of one gradient
iteration, the parameters are updated as follows:

$\theta^{\prime}_{i}=\theta-\alpha\nabla_{\theta}\mathcal{L}_{\mathcal{S}_{i}}(f_{\theta}),$
(1)

where $\alpha$ is the step size that may be fixed as a hyperparameter or meta-
learned, and the loss function for a set of observations $\mathcal{Z}$ is
defined as $\mathcal{L}_{\mathcal{Z}}$ for the few shot scenario is
represented as a simple cross-entropy:

$\mathcal{L}_{\mathcal{Z}}(f_{\theta})=\sum_{(\mathbf{x}_{i,l},\mathbf{y}_{i,l})\in\mathcal{Z}}\sum_{k=1}^{K}-y_{i,l}^{k}\log{f_{\theta,k}(\mathbf{x}_{i,j})},$
(2)

where $f_{\theta,k}(\mathbf{x}_{i,j})$ denotes $k$-th output of the model
$f_{\theta}$, for a given input $\mathbf{x}_{i,l}$, and $\mathbf{y}_{i,l}$ is
corresponding class in one-hot coding. For simplicity of notation, we will
consider one gradient update for the rest of this section, but using multiple
gradient updates is a straightforward extension. After calculating the tasks-
specific updates $\theta^{\prime}_{i}$ the general model parameters are
trained by optimizing for the performance of $f_{\theta^{\prime}_{i}}$ with
respect to $\theta$ across tasks from batch $\mathcal{B}$. More concretely,
the meta-objective used to train the general parameters of the models is as
follows:

$\mathcal{L}_{MAML}(f_{\theta})=\sum_{\mathcal{T}_{i}\in\mathcal{B}}\mathcal{L}_{\mathcal{Q}_{i}}(f_{\theta^{\prime}})=\sum_{\mathcal{T}_{i}\in\mathcal{B}}\mathcal{L}_{\mathcal{Q}_{i}}(f_{\theta-\alpha\nabla_{\theta}\mathcal{L}_{\mathcal{S}_{i}}(f_{\theta})}),$
(3)

Note that the meta-optimization is performed over the model parameters
$\theta$, whereas the objective is computed using the updated model parameters
$\theta^{\prime}$. In effect, our proposed method aims to optimize the model
parameters such that one or a small number of gradient steps on a new task
will produce maximally effective behavior on that task.

The meta-optimization across tasks is performed via stochastic gradient
descent (SGD) such that the model parameters $\theta$ are updated as follows:

$\theta\leftarrow\theta-\beta\nabla_{\theta}\mathcal{L}_{MAML}(f_{\theta}),$
(4)

where $\beta$ is the meta step size.

During the inference stage, in order to perform predictions for newly observed
task $\mathcal{T}_{*}=\\{\mathcal{S}_{*},\mathcal{X}_{*}\\}$ the loss function
$\mathcal{L}_{\mathcal{S}_{*}}(f_{\theta})$ is calculated first using eq. (2).
Next, the parameters $\theta^{\prime}_{*}$ for task $\mathcal{T}_{*}$ are
calculated from eq. (1). The final predictions for query examples
$\mathcal{X}_{*}$ are performed by the model $f_{\theta^{\prime}_{*}}$, where
for selected query example $\mathbf{x}_{q}\in\mathcal{X}_{*}$ we have
$p(\mathbf{y}|\mathbf{x}_{q},\theta^{\prime}_{*})=f_{\theta^{\prime}_{*}}(\mathbf{x}_{q},\theta^{\prime}_{*})$.

The main limitation of the approach is that it produces the general weights
for all possible tasks, and the adjustment is performed via a gradient-based
approach performed on the support set. For some non-trivial challenging tasks,
the dedicated parameters $\theta^{\prime}_{*}$ may be located far from the
base weights, $\theta$. Consequently, the adaptation procedure would require
significantly more gradient steps, the training may be unstable, and the model
will tend to overfit to support set. To overcome this limitation, we propose
to replace the gradient-based adaption with the Hypernetwork approach, where
the update step is returned by an additional deep model that extracts the
information from the support set (see toy example in Fig. 1).

###  3.2 HyperMAML - overview

We introduce our HyperMAML – a model that utilizes Hypernetworks for modeling
weights updates in the classical MAML algorithm. The main idea of the proposed
updating scenario is to use information extracted from support examples and
predictions given by universal weights to find optimal updates dedicated to a
given task. Thanks to this approach, we can switch the classifier’s parameters
between completely different tasks based on the support set and existing
prediction of the universal weights.

The architecture of the HyperMAML is provided in Fig. 2. In this section we
present the model for one-shot scenario, we further discuss, how to extend it
to Few-Shot problems. We aim at predicting the class distribution
$p(\mathbf{y}|\textbf{x}_{q},\mathcal{S})$, assuming given single query
example $\mathbf{x}_{q}$, and the set of support examples $\mathcal{S}$.
Following the idea from MAML we consider the parameterized function
$f_{\theta}$, that models the discriminative distribution for the classes. In
addition, in our architecture we distinguish the trainable encoding network
$E(\cdot)$, that transforms data to low-dimensional representation. We
postulate to calculate
$p(\mathbf{y}|\textbf{x}_{q},\theta^{\prime})=f_{\theta^{\prime}}(\mathbf{e}_{q})$,
where $\mathbf{e}_{q}$ is the query example $\mathbf{x}_{q}$ transformed using
encoder $E(\cdot)$, and $\theta^{\prime}$ represents the updated parameters
for a considered task, $\theta^{\prime}=\theta+\Delta_{\theta}$. Compared to
gradient-based adaptation step described by eq. (1) used to calculate
$\Delta_{\theta}$ we propose to predict the values using hypernetwork,
directly from support set.

Figure 2: The overview of HyperMAML architecture. The input support examples
are processed by encoding network $E(\cdot)$ and delivered to the hypernetwork
$H(\cdot)$ together with the true support labels and predictions from general
model $f_{\theta}(\cdot)$. The hypernetwork transforms them, and returns the
update of weigths $\Delta\theta$ for target classifier $f_{\theta^{\prime}}$.
The query example is transformed by Encoder $E(\cdot)$, and the final class
distribution is returned by the target model $f_{\theta^{\prime}}$ dedicated
to the considered task.

Each of the inputs from support set $\mathcal{X}_{S}$ is transformed by
Encoder $E(\cdot)$ in order to obtain low-dimensional matrix of embeddings
$\mathbf{E}_{\mathcal{S}}=[\mathbf{e}_{\mathcal{S},1},\dots,\mathbf{e}_{\mathcal{S},K}]^{\mathrm{T}}$.
Next, the corresponding class labels for support examples,
$\mathbf{Y}_{\mathcal{S}}=[\mathbf{y}_{\mathcal{S},1},\dots,\mathbf{y}_{\mathcal{S},K}]^{\mathrm{T}}$
are concatenated to the corresponding embeddings stored in the rows of matrix
$\mathbf{E}_{\mathcal{S}}$. In addition, we also calculate the predicted
values for the examples from the support set using the general model
$f_{\theta}(\mathbf{E}_{\mathcal{S}})=\mathbf{\hat{Y}_{\mathcal{S}}}$, and
also concatenate them to $\mathbf{E}_{\mathcal{S}}$. The matrix transformed
support inputs $\mathbf{E}_{\mathcal{S}}$, together with true support labels
$\mathbf{Y}_{\mathcal{S}}$, and corresponding predictions
$\mathbf{\hat{Y}_{\mathcal{S}}}$ returned by genreal model are delivered as an
input to the hypernetwork $H(\cdot)$ that returns the update
$\Delta_{\theta}$. The parameters for final target model are calculated with
the following formula:

$\theta^{\prime}=\theta+\Delta_{\theta}=\theta+H(\mathbf{E}_{\mathcal{S}},\mathbf{\hat{Y}}_{\mathcal{S}},\mathbf{Y}_{\mathcal{S}}).$
(5)

Practically, the Hypernetwork observes the support examples with the
corresponding true values and decides how the global parameters $\theta$
should be adjusted to the considered task. In addition, the predictions from
global model $f_{\theta}$ are also delivered to the model in order to identify
the misclassifications and try to correct them during the update state.

###  3.3 HyperMAML - training

For training the model we assume that encoder $E(\cdot)$ is parametrized by
$\gamma$, $E:=E_{\gamma}$, and the hypernetwork $H(\cdot)$ by $\eta$,
$H:=H_{\eta}$. The training procedure is described in Algorithm 2. First, we
sample the batch of tasks $\mathcal{B}$ from the given dataset $\mathcal{D}$.
Next, for each task $\mathcal{T}_{i}$ in batch $\mathcal{B}$ we calculate the
update $\Delta_{\theta_{i}}$ using the support set $\mathcal{S}_{i}$, and
provide the updated parameters $\theta^{\prime}$ according to the rule given
by eq. (5). Finally, the objective to train the parameters of the system is
calculated using the query sets from the batch tasks $\mathcal{T}_{i}$:

$\mathcal{L}_{HyperMAML}(f_{\theta})=\sum_{\mathcal{T}_{i}\in\mathcal{B}}\mathcal{L}_{\mathcal{Q}_{i}}(f_{\theta^{\prime}})=\sum_{\mathcal{T}_{i}\in\mathcal{B}}\mathcal{L}_{\mathcal{Q}_{i}}(f_{\theta+\Delta_{\theta}}),$
(6)

where $\mathcal{L}_{\mathcal{Q}_{i}}(f_{\theta^{\prime}})$ is given by eq.
(2). The parameters of the encoder, hypernetwork, and global parameters
$\theta$ represent the meta parameters of the system, and they are updated
with stochastic gradient descent (SGD) by optimizing
$\mathcal{L}_{HyperMAML}(f_{\theta})$.

Algorithm 2 HyperMAML

Require: $\mathcal{D}=\\{\mathcal{T}_{n}\\}_{n=1}^{N}$: set of training tasks
Require: $\beta$: step size hyper parameter

1:randomly initialize $\theta$, $\gamma$, $\eta$

2:while not done do

3: Sample batch of tasks $\mathcal{B}$ from $\mathcal{D}$

4: for each task $\mathcal{T}_{i}=\\{\mathcal{S}_{i},\mathcal{Q}_{i}\\}$ from
batch $\mathcal{B}$ do

5: Compute adapted parameters $\theta^{\prime}_{i}$ from $\mathcal{S}_{i}$
using formula given by eq. (5)

6: end for

7: Calculate the loss $\mathcal{L}_{HyperMAML}(f_{\theta})$ given by eq. (6).

8:
$\theta\leftarrow\theta-\beta\nabla_{\theta}\mathcal{L}_{HyperMAML}(f_{\theta})$
$\triangleright$ Update the global target parameters $\theta$

9: $\eta\leftarrow\eta-\beta\nabla_{\eta}\mathcal{L}_{HyperMAML}(f_{\theta})$
$\triangleright$ Update parameters of the hypernetwork $H_{\eta}$

10:
$\gamma\leftarrow\gamma-\beta\nabla_{\gamma}\mathcal{L}_{HyperMAML}(f_{\theta})$
$\triangleright$ Update the parameters of the Encoder $E_{\gamma}$

11:end while

#### Adaptation to the Few-Shot scenario.

The proposed method can be easily extended for Few-Shot scenarios following
the aggregation technique from (Sendera et al., 2022). For our approach, we
aggregate the embedding values of the support examples from the same class
using _mean_ operation. In addition, the corresponding predictions within the
class are also averaged and concatenated to the averaged per class embedding
together with the true class label, and further processed via hypernetwork.

#### Warming-up universal weights

In practice, it is not trivial to initialize the universal weights of
HyperMAML. Classical initialization does not allow to update the universal
weights and only the Hypernetwork’s parameters are changing. To solve this
problem we use a smooth transition from gradient to Hypernetwork update:

$\theta^{\prime}=\theta+\lambda\cdot
H(\mathbf{E}_{\mathcal{S}},\mathbf{\hat{Y}}_{\mathcal{S}},\mathbf{Y}_{\mathcal{S}})-(1-\lambda)\cdot\alpha\nabla_{\theta}\mathcal{L}_{\mathcal{S}_{i}}(f_{\theta})$
(7)

where $\lambda$ is changing from zero to one in a few initial training epochs.

##  4 Experiments

In the typical Few-Shot learning setting, making a valuable and fair
comparison between proposed models is often complicated because of the
existence of the significant differences in architectures and implementations
of known methods. In order to limit the influence of the deeper backbone
(feature extractor) architectures, we follow the unified procedure proposed by
(Chen et al., 2019)222We shall release the code with our experiments after the
end of the review period..

In all of the reported experiments, the tasks consist of 5 classes (5-way) and
1 or 5 support examples (1 or 5-shot). Unless indicated otherwise, all
compared models use a known and widely utilized backbone consisting of four
convolutional layers (each consisting of a 2D convolution, a batch-norm layer,
and a ReLU non-linearity; each layer consists of 64 channels) and have been
trained from scratch (Chen et al., 2019). In all experiments, the query set of
each task consists of 16 samples for each class (80 in total). We split the
datasets into the standard train, validation, and test class subsets, used
commonly in the literature (Ravi and Larochelle, 2017; Chen et al., 2019;
Patacchiola et al., 2020). For each setting, we report the performance of
HyperMAML averaged over three training runs. We provide the additional
training details in the Appendix.

###  4.1 Classification

First, we consider a classical Few-Shot learning scenario. We benchmark the
performance of the HyperMAML and other methods on two challenging and widely
considered datasets: Caltech-USCD Birds (CUB) (Wah et al., 2011) and mini-
ImageNet (Ravi and Larochelle, 2017). The following experiments are in the
most popular setting, 5-way, consisting of 5 random classes. We compare
HyperMAML to a vast pool of state-of-the-art algorithms in the tasks of 1-shot
and 5-shot classification.

In the $1$-shot scenario, HyperMAML yields top performing results ($66.11\%$)
on the CUB dataset, inferior only to FEAT (Ye et al., 2018) ($68.87\%$) and on
par with HyperShot with finetuning (Sendera et al., 2022) ($66.13\%$). In the
mini-ImageNet dataset, our approach is not among the best models but returns
decent outcome ($51.84\%$).

In the $5$-shot setting, HyperMAML is among the top-3 best performing models
achieving $78.89\%$ in the CUB dataset. On the contrary, on the mini-ImageNet
dataset, HyperMAML performs slightly worse than multiple other models (Ye et
al., 2018; Rajasegaran et al., 2020; Sendera et al., 2022; Antoniou et al.,
2018), achieving the accuracy of $66.29\%$.

The obtained results show that HyperMAML achieves performance better or
comparable to a vast pool of standard Few-Shot learning methods, in particular
MAML (Finn et al., 2017).

Table 1: The classification accuracy results for the inference tasks on CUB and mini-ImageNet datasets in the $1$-shot and $5$-shot settings. The highest results are in bold and second-highest in italic (the larger, the better).  | CUB | mini-ImageNet
---|---|---
Method | 1-shot | 5-shot | 1-shot | 5-shot
Feature Transfer (Zhuang et al., 2020) | $46.19\pm 0.64$ | $68.40\pm 0.79$ | $39.51\pm 0.23$ | $60.51\pm 0.55$
Baseline++ (Chen et al., 2019) | $61.75\pm 0.95$ | $78.51\pm 0.59$ | $47.15\pm 0.49$ | $66.18\pm 0.18$
MatchingNet (Vinyals et al., 2016) | $60.19\pm 1.02$ | $75.11\pm 0.35$ | $48.25\pm 0.65$ | $62.71\pm 0.44$
ProtoNet (Snell et al., 2017) | $52.52\pm 1.90$ | $75.93\pm 0.46$ | $44.19\pm 1.30$ | $64.07\pm 0.65$
RelationNet (Sung et al., 2018) | $62.52\pm 0.34$ | $78.22\pm 0.07$ | $48.76\pm 0.17$ | $64.20\pm 0.28$
DKT + BNCosSim (Patacchiola et al., 2020) | $62.96\pm 0.62$ | $77.76\pm 0.62$ | $49.73\pm 0.07$ | $64.00\pm 0.09$
PPA (Qiao et al., 2017) | – | – | $54.53\pm 0.40$ | –
FEAT (Ye et al., 2018) | $\mathbf{68.87\pm 0.22}$ | $\mathbf{82.90\pm 0.15}$ | $\mathbf{55.15\pm 0.20}$ | $\mathbf{71.61\pm 0.16}$
HyperShot (Sendera et al., 2022) | $65.27\pm 0.24$ | $79.80\pm 0.16$ | $52.42\pm 0.46$ | $68.78\pm 0.29$
HyperShot+ finetuning (Sendera et al., 2022) | $\mathit{66.13\pm 0.26}$ | $\mathit{80.07\pm 0.22}$ | $53.18\pm 0.45$ | $69.62\pm 0.2$
MAML (Finn et al., 2017) | $56.11\pm 0.69$ | $74.84\pm 0.62$ | $45.39\pm 0.49$ | $61.58\pm 0.53$
MAML++ (Antoniou et al., 2018) | – | – | $52.15\pm 0.26$ | $68.32\pm 0.44$
iMAML-HF (Rajeswaran et al., 2019) | – | – | $49.30\pm 1.88$ | –
SignMAML (Fan et al., 2021) | – | – | $42.90\pm 1.50$ | $60.70\pm 0.70$
Bayesian MAML (Yoon et al., 2018) | $55.93\pm 0.71$ |  | $\mathit{53.80\pm 1.46}$ | $64.23\pm 0.69$
Unicorn-MAML (Ye and Chao, 2021) | – | – | $54.89$ | –
Meta-SGD (Li et al., 2017) | – | – | $50.47\pm 1.87$ | $64.03\pm 0.94$
MetaNet (Munkhdalai and Yu, 2017) | – | – | $49.21\pm 0.96$ | –
PAMELA (Rajasegaran et al., 2020) | – | – | $53.50\pm 0.89$ | $\mathit{70.51\pm 0.67}$
HyperMAML | $\mathit{66.11\pm 0.28}$ | $78.89\pm 0.19$ | $51.84\pm 0.57$ | $66.29\pm 0.43$
Table 2: The classification accuracy results for the inference tasks on cross-domain tasks (Omniglot$\rightarrow$EMNIST and mini-ImageNet$\rightarrow$CUB) datasets in the $1$-shot and $5$-shot setting. The highest results are bold and second-highest in italic (the larger, the better).  | Omniglot$\rightarrow$EMNIST | mini-ImageNet$\rightarrow$CUB
---|---|---
Method | 1-shot | 5-shot | 1-shot | 5-shot
Feature Transfer (Zhuang et al., 2020) |  64.22 $\pm$ 1.24 |  86.10 $\pm$ 0.84 |  32.77 $\pm$ 0.35 |  50.34 $\pm$ 0.27
Baseline$++$ (Chen et al., 2019) |  56.84 $\pm$ 0.91 |  80.01 $\pm$ 0.92 |  39.19 $\pm$ 0.12 | ${57.31\pm 0.11}$
MatchingNet (Vinyals et al., 2016) |  75.01 $\pm$ 2.09 |  87.41 $\pm$ 1.79 |  36.98 $\pm$ 0.06 |  50.72 $\pm$ 0.36
ProtoNet (Snell et al., 2017) |  72.04 $\pm$ 0.82 |  87.22 $\pm$ 1.01 |  33.27 $\pm$ 1.09 |  52.16 $\pm$ 0.17
RelationNet (Sung et al., 2018) |  75.62 $\pm$ 1.00 |  87.84 $\pm$ 0.27 |  37.13 $\pm$ 0.20 |  51.76 $\pm$ 1.48
DKT (Patacchiola et al., 2020) |  75.40 $\pm$ 1.10 | $\mathit{90.30\pm 0.49}$ | $\mathbf{40.14\pm{0.18}}$ |  56.40 $\pm$ 1.34
OVE PG GP + Cosine (ML) (Snell and Zemel, 2020) | $68.43\pm 0.67$ | $86.22\pm 0.20$ | $39.66\pm 0.18$ | $55.71\pm 0.31$
OVE PG GP + Cosine (PL) (Snell and Zemel, 2020) | $77.00\pm 0.50$ | $87.52\pm 0.19$ | $37.49\pm 0.11$ | $57.23\pm 0.31$
HyperShot  (Sendera et al., 2022) | ${78.06\pm 0.24}$ | $89.04\pm 0.18$ | $39.09\pm 0.28$ | $\mathit{57.77\pm 0.33}$
HyperShot + finetuning | $\mathbf{80.65\pm 0.30}$ | $\mathbf{90.81\pm 0.16}$ | $\mathit{40.03\pm 0.41}$ | $\mathbf{58.86\pm 0.38}$
MAML (Finn et al., 2017) |  74.81 $\pm$ 0.25 |  83.54 $\pm$ 1.79 |  34.01 $\pm$ 1.25 |  48.83 $\pm$ 0.62
Bayesian MAML (Yoon et al., 2018) | $63.94\pm 0.47$ | $65.26\pm 0.30$ | $33.52\pm 0.36$ | $51.35\pm 0.16$
HyperMAML | $\mathit{79.07\pm 1.09}$ | ${89.22\pm 0.78}$ | $36.32\pm 0.61$ | $49.43\pm 0.14$

###  4.2 Cross-domain adaptation

In the cross-domain adaptation setting, the model is evaluated on tasks coming
from a different distribution than the one it had been trained on. Therefore,
such a task is more challenging than standard classification and is a
plausible indicator of a model’s ability to generalize. In order to benchmark
the performance of HyperMAML in cross-domain adaptation, we combine two
datasets so that the training fold is drawn from the first dataset and
validation and the testing fold – from another one. We report the results in
Table 2. In the task of 1-shot Omniglot$\rightarrow$EMNIST classification,
HyperMAML achieves the second-best result ($79.84\%$), with
HyperShot+finetuning ($80.65\%$) being the top one. Compared to the other
methods, we observe relatively smaller performance growth as more data becomes
available. In the $5$-shot Omniglot$\rightarrow$EMNIST classification task
HyperMAML yields comparable results ($89.22\%$) to HyperShot (Sendera et al.,
2022) ($90.81\%$) and DKT (Patacchiola et al., 2020) ($90.30\%$), which are
the state-of-the-art in this setting. In the most challenging task of mini-
ImageNet$\rightarrow$CUB classification, our method performs comparably to
baseline methods such as MAML, ProtoNet and Matching Net (Finn et al., 2017;
Vinyals et al., 2016; Snell et al., 2017), particularly in the $1$-shot
setting.

###  4.3 Computational efficiency

Finally, we verify the hypothesis that HyperMAML offers an increased
computational efficiency compared to MAML. To this end, we measure the times
of processing the entire Omniglot $\rightarrow$ EMNIST test dataset (600 tasks
in total) by MAML with different numbers of gradient steps and HyperMAML and
report the results in Table 3. We provide additional results for other
datasets in the supplementary material.

We find that processing the test data with HyperMAML takes approximately the
same time as using MAML with just 2 gradient updates. We also note that even
given the budget of 100 gradient updates, MAML never matches the accuracy
achieved by a single update generated by HyperMAML.

Model | Steps | Time | Accuracy
---|---|---|---
MAML | 0 | $2.17\pm 0.28$ | $19.978\pm 0.348$
1 | $5.16\pm 0.13$ | $74.67\pm 0.25$
2 | $7.30\pm 0.11$ | $74.69\pm 0.24$
3 | $8.87\pm 0.08$ | $74.71\pm 0.25$
4 | $10.79\pm 0.19$ | $74.71\pm 0.24$
5 | $12.66\pm 0.14$ | $74.71\pm 0.23$
10 | $22.60\pm 0.28$ | $74.74\pm 0.25$
25 | $51.75\pm 0.24$ | $74.78\pm 0.25$
50 | $103.88\pm 2.24$ | $74.82\pm 0.25$
| 100 | $212.62\pm 2.24$ | $74.86\pm 0.27$
HyperMAML | – | $8.21\pm 0.41$ | $80.778\pm 0.598$
Table 3: Time (in seconds) spent on processing the entire Omniglot
$\rightarrow$ EMNIST test dataset (600 tasks) by MAML with different numbers
of gradient steps and HyperMAML.

##  5 Conclusions

In this work, we introduced HyperMAML – a novel Meta-Learning algorithm
strongly motivated by MAML (Finn et al., 2017). The crucial difference between
the two methods lies in the fact that in HyperMAML the update of the model is
given not by the gradient optimization, but by the trainable Hypernetwork.
Consequently, HyperMAML is more computationally efficient than MAML, as during
adaptation it performs only a single parameter update. Moreover, HyperMAML is
not only more biologically feasible as it does not use backpropagation, but it
can adapt easier to different settings and has a smaller number of
hyperparameters. Our experiments show that HyperMAML outperforms the classical
MAML in a number of standard Few-Shot learning benchmarks and achieves results
similar to various other state-of-the-art methods in most cases.

#### Limitations

The main limitation of the method is the need for initialization of the
universal weights by a continuous transition from gradient update to
Hypernetwork. We plan to investigate this issue in future work.

## References

  * Antoniou et al. [2018] A. Antoniou, H. Edwards, and A. Storkey.  How to train your maml, 2018.  URL https://arxiv.org/abs/1810.09502.
  * Bauer et al. [2017] M. Bauer, M. Rojas-Carulla, J. B. Światkowski, B. Schölkopf, and R. E. Turner.  Discriminative k-shot learning using probabilistic models, 2017.
  * Bengio et al. [1992] S. Bengio, Y. Bengio, J. Cloutier, and J. Gecsei.  On the optimization of a synaptic learning rule.  1992\.
  * Bertinetto et al. [2018] L. Bertinetto, J. F. Henriques, P. Torr, and A. Vedaldi.  Meta-learning with differentiable closed-form solvers.  In _International Conference on Learning Representations_ , 2018.
  * Chen et al. [2019] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, and J.-B. Huang.  A closer look at few-shot classification.  _arXiv preprint arXiv:1904.04232_ , 2019.
  * Fan et al. [2021] C. Fan, P. Ram, and S. Liu.  Sign-maml: Efficient model-agnostic meta-learning by signsgd.  _CoRR_ , abs/2109.07497, 2021.  URL https://arxiv.org/abs/2109.07497.
  * Finn et al. [2017] C. Finn, P. Abbeel, and S. Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International Conference on Machine Learning_ , pages 1126–1135. PMLR, 2017.
  * Finn et al. [2018] C. Finn, K. Xu, and S. Levine.  Probabilistic model-agnostic meta-learning.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pages 9537–9548, 2018.
  * Gidaris and Komodakis [2018] S. Gidaris and N. Komodakis.  Dynamic few-shot visual learning without forgetting.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4367–4375, 2018.
  * Gordon et al. [2018] J. Gordon, J. Bronskill, M. Bauer, S. Nowozin, and R. Turner.  Meta-learning probabilistic inference for prediction.  In _International Conference on Learning Representations_ , 2018.
  * Grant et al. [2018] E. Grant, C. Finn, S. Levine, T. Darrell, and T. Griffiths.  Recasting gradient-based meta-learning as hierarchical bayes.  In _International Conference on Learning Representations_ , 2018.
  * Ha et al. [2016] D. Ha, A. Dai, and Q. V. Le.  Hypernetworks.  _arXiv preprint arXiv:1609.09106_ , 2016.
  * Hospedales et al. [2020] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey.  Meta-learning in neural networks: A survey, 2020.
  * Jerfel et al. [2019] G. Jerfel, E. Grant, T. L. Griffiths, and K. Heller.  Reconciling meta-learning and continual learning with online mixtures of tasks.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , pages 9122–9133, 2019.
  * Kim and Hospedales [2021] M. Kim and T. Hospedales.  Gaussian process meta few-shot classifier learning via linear discriminant laplace approximation.  _arXiv preprint arXiv:2111.05392_ , 2021.
  * Lee et al. [2019] K. Lee, S. Maji, A. Ravichandran, and S. Soatto.  Meta-learning with differentiable convex optimization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10657–10665, 2019.
  * Li et al. [2017] Z. Li, F. Zhou, F. Chen, and H. Li.  Meta-sgd: Learning to learn quickly for few-shot learning, 2017.  URL https://arxiv.org/abs/1707.09835.
  * Lillicrap et al. [2020] T. P. Lillicrap, A. Santoro, L. Marris, C. J. Akerman, and G. Hinton.  Backpropagation and the brain.  _Nature Reviews Neuroscience_ , 21(6):335–346, 2020.
  * Mishra et al. [2018] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel.  A simple neural attentive meta-learner.  In _International Conference on Learning Representations_ , 2018.
  * Munkhdalai and Yu [2017] T. Munkhdalai and H. Yu.  Meta networks.  In _International Conference on Machine Learning_ , pages 2554–2563. PMLR, 2017.
  * Munkhdalai et al. [2018] T. Munkhdalai, X. Yuan, S. Mehri, and A. Trischler.  Rapid adaptation with conditionally shifted neurons.  In _International Conference on Machine Learning_ , pages 3664–3673. PMLR, 2018.
  * Nguyen et al. [2020] C. Nguyen, T.-T. Do, and G. Carneiro.  Uncertainty in model-agnostic meta-learning using variational inference.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 3090–3100, 2020.
  * Nichol et al. [2018] A. Nichol, J. Achiam, and J. Schulman.  On first-order meta-learning algorithms.  _arXiv preprint arXiv:1803.02999_ , 2018.
  * Oreshkin et al. [2018] B. N. Oreshkin, P. Rodriguez, and A. Lacoste.  Tadam: Task dependent adaptive metric for improved few-shot learning.  _arXiv preprint arXiv:1805.10123_ , 2018.
  * Patacchiola et al. [2020] M. Patacchiola, J. Turner, E. J. Crowley, M. O’Boyle, and A. J. Storkey.  Bayesian meta-learning for the few-shot setting via deep kernels.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Qiao et al. [2017] S. Qiao, C. Liu, W. Shen, and A. Yuille.  Few-shot image recognition by predicting parameters from activations, 2017\.
  * Rajasegaran et al. [2020] J. Rajasegaran, S. H. Khan, M. Hayat, F. S. Khan, and M. Shah.  Meta-learning the learning trends shared across tasks.  _CoRR_ , abs/2010.09291, 2020.  URL https://arxiv.org/abs/2010.09291.
  * Rajeswaran et al. [2019] A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine.  Meta-learning with implicit gradients.  _Advances in Neural Information Processing Systems_ , 32:113–124, 2019.
  * Rasmussen [2003] C. E. Rasmussen.  Gaussian processes in machine learning.  In _Summer school on machine learning_ , pages 63–71. Springer, 2003\.
  * Ravi and Beatson [2018] S. Ravi and A. Beatson.  Amortized bayesian meta-learning.  In _International Conference on Learning Representations_ , 2018.
  * Ravi and Larochelle [2017] S. Ravi and H. Larochelle.  Optimization as a model for few-shot learning.  In _ICLR_ , 2017.
  * Santoro et al. [2016] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap.  Meta-learning with memory-augmented neural networks.  In _International conference on machine learning_ , pages 1842–1850. PMLR, 2016.
  * Schmidhuber [1992] J. Schmidhuber.  Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks.  _Neural Computation_ , 4(1):131–139, 01 1992\.
  * Sendera et al. [2022] M. Sendera, M. Przewięźlikowski, K. Karanowski, M. Zięba, J. Tabor, and P. Spurek.  Hypershot: Few-shot learning by kernel hypernetworks.  _arXiv preprint arXiv:2203.11378_ , 2022.
  * Snell and Zemel [2020] J. Snell and R. Zemel.  Bayesian few-shot classification with one-vs-each pólya-gamma augmented gaussian processes.  In _International Conference on Learning Representations_ , 2020.
  * Snell et al. [2017] J. Snell, K. Swersky, and R. S. Zemel.  Prototypical networks for few-shot learning.  _arXiv preprint arXiv:1703.05175_ , 2017.
  * Song et al. [2020] Y. Song, T. Lukasiewicz, Z. Xu, and R. Bogacz.  Can the brain do backpropagation?—exact implementation of backpropagation in predictive coding networks.  _Advances in neural information processing systems_ , 33:22566–22579, 2020.
  * Sun et al. [2021] Z. Sun, J. Wu, X. Li, W. Yang, and J.-H. Xue.  Amortized bayesian prototype meta-learning: A new probabilistic meta-learning approach to few-shot image classification.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1414–1422. PMLR, 2021.
  * Sung et al. [2018] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales.  Learning to compare: Relation network for few-shot learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1199–1208, 2018.
  * Vinyals et al. [2016] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al.  Matching networks for one shot learning.  _Advances in neural information processing systems_ , 29:3630–3638, 2016.
  * Wah et al. [2011] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The caltech-ucsd birds-200-2011 dataset.  Technical Report CNS-TR-2011-001, 2011.
  * Wang et al. [2019] Y. Wang, W.-L. Chao, K. Q. Weinberger, and L. van der Maaten.  Simpleshot: Revisiting nearest-neighbor classification for few-shot learning.  _arXiv preprint arXiv:1911.04623_ , 2019.
  * Wang et al. [2021] Z. Wang, Z. Miao, X. Zhen, and Q. Qiu.  Learning to learn dense gaussian processes for few-shot learning.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Whittington and Bogacz [2019] J. C. Whittington and R. Bogacz.  Theories of error back-propagation in the brain.  _Trends in cognitive sciences_ , 23(3):235–250, 2019.
  * Ye et al. [2018] H. Ye, H. Hu, D. Zhan, and F. Sha.  Learning embedding adaptation for few-shot learning.  _CoRR_ , abs/1812.03664, 2018.  URL http://arxiv.org/abs/1812.03664.
  * Ye and Chao [2021] H.-J. Ye and W.-L. Chao.  How to train your maml to excel in few-shot classification, 2021.
  * Yoon et al. [2018] J. Yoon, T. Kim, O. Dia, S. Kim, Y. Bengio, and S. Ahn.  Bayesian model-agnostic meta-learning.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pages 7343–7353, 2018.
  * Zhen et al. [2020] X. Zhen, Y.-J. Du, H. Xiong, Q. Qiu, C. Snoek, and L. Shao.  Learning to learn variational semantic memory.  In _NeurIPS_ , 2020.
  * Zhuang et al. [2020] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He.  A comprehensive survey on transfer learning, 2020.

##  6 Appendix: 2D example

To visualize the fact that the MAML algorithm is significantly limited by the
weight update procedure based on the gradient method, we will analyze a simple
2D example. Let us consider a data set consisting of 4 Gaussian data; see the
first column in Fig. 3. In the meta-learning scenario, we produce a task that
consists of samples from two horizontal or vertical ellipses with permuted
labels. In practice, we would like to force a method to adapt to four tasks
presented in the second to the fifth column in Fig. 3. In the training
procedure, we use a neural network with one layer – in practice, a linear
model. We can visualize weights and decision boundaries thanks to using such
simple architecture. We marked the universal weights with the blue color and
the updated parameters – with red.

The MAML model with one gradient step in the inner loop (see the first row in
Fig. 3) cannot update all four tasks. On the other hand, when we use five
gradient updates, the model gives a reasonable solution (see the second row in
Fig. 3). In practice, for MAML, we cannot effectively use many steps in inner
lop in the case of really high dimensional data.

Our HyperMAML can solve the task by using the hypernetwork paradigm. We use
only one update in our method, which can dramatically change weights. Notice
that even for five steps, MAML cannot significantly update weight. In
practice, it uses only rotations.

Figure 3:  Let us consider a data set consisting of four Gaussian data (the
first column). In the Meta-Learning scenario, we produce a task that consists
of samples from two horizontal or vertical ellipses with permuted labels
(second to the fifth column). The MAML model with one gradient step in the
inner loop (see the first row) cannot update all four tasks. On the other
hand, when we use five gradient updates, the model gives a reasonable solution
(see the second row). In practice, for MAML, we cannot effectively use many
steps in inner lop in the case of really high dimensional data. Our HyperMAML
can solve the task by using the hypernetwork paradigm. We use only one update
in our method, which can dramatically change weights. Details of the
experiment we add to supplementary material.

##  7 Appendix: Ablation study of mechanisms used in HyperMAML

In this section, we present two mechanisms we utilize when training HyperMAML.

#### Switching

mechanism is a smooth transition between training the convolutional encoder
through MAML and HyperMAML objective. We consider the MAML warm-up as a
starting point of the training loop and then smoothly move towards HyperMAML
training. During training, we define two "milestone" epochs (see Section 10.3,
between which the transition occurs. During the transition, we continuously
decrease the participation of the MAML objective in the training process. It
is done by multiplying MAML loss by $p$, ranging from $1.0$ to $0.0$ for a
given number of epochs, and multiplying the HyperMAML loss by $1-p$. Our
motivation for this mechanism is to train a better universal optimizing the
MAML objective during the warm-up part of the training and then Switch to the
HyperMAML objective gradually.

#### Enhancement

of the embeddings is another mechanism in our framework. When preparing the
input to the Hypernetwork from the support set, we first obtain support
embeddings from the encoder. Then, we forward those embeddings through the
universal classifier and obtain its predictions for support examples. We
concatenate those predictions to the support embeddings, as well as their
respective ground-truth labels. The whole process is visualized in Figure 4.
Our motivation to perform such an operation is to give the Hypernetwork the
information about the current decision of the classifier, to better estimate
the updates to its weights. We note that even though the Hypernetwork
generates the weights of the classifier based on the enhanced support
embeddings, the generated downstream classifier does not use enhancements when
processing the query set. Instead, it only processes the raw embeddings
obtained from the encoder (see Figure 2 from the main paper).

Figure 4: Illustration of the embedding enhancement mechanism. The support
embeddings (which serve as the input to the HyperMAML Hypernetwork) are
enhanced with the predictions of the base classifier and their respective
ground-truth labels.

We perform an ablation study of both mechanisms on the task 5-shot 1-way
Omniglot$\rightarrow$EMNIST classification. The results, reported in Table 4,
indicate that both mechanisms utilized individually improve the performance of
HyperMAML, and the combination of the two yields the best results.

Model | Switch | Enhancement | Accuracy | Learning rate | Milestones
---|---|---|---|---|---
HyperMAML | no | no | $74.39\pm 0.88$ | $0.0001$ | -
no | yes | $75.28\pm 1.29$ | $0.0001$ | -
yes | no | $76.89\pm 0.71$ | $0.01$ | $51,550$
yes | yes | $79.10\pm 1.43$ | $0.01$ | $51,550$
Table 4: The classification accuracy of HyperMAML on
Omniglot$\rightarrow$EMNIST task in the 5-way 1-shot scenario. We consider
different strategies by adding switch or embedding enhancement strategy to our
model.

##  8 Appendix: Full classification results

We provide an expanded version of Table 1 from the main paper, with numerous
additional baseline methods – see Table 1.

Table 5: The classification accuracy results for the inference tasks on CUB and mini-ImageNet datasets in the $5$-shot setting. The highest results are bold and second-highest in italic (the larger, the better).  | CUB | mini-ImageNet
---|---|---
Method | 1-shot | 5-shot | 1-shot | 5-shot
ML-LSTM [31] | – | – | $43.44\pm 0.77$ | $60.60\pm 0.71$
SNAIL [19] | – | – | $45.10$ | $55.20$
LLAMA [11] | – | – | $49.40\pm 1.83$ |
VERSA [10] | – | – | $48.53\pm 1.84$ | $67.37\pm 0.86$
Amortized VI [10] | – | – | $44.13\pm 1.78$ | $55.68\pm 0.91$
Meta-Mixture [14] | – | – | $49.60\pm 1.50$ | $64.60\pm 0.92$
SimpleShot [42] | – | – | $49.69\pm 0.19$ | $66.92\pm 0.17$
Feature Transfer [49] | $46.19\pm 0.64$ | $68.40\pm 0.79$ | $39.51\pm 0.23$ | $60.51\pm 0.55$
Baseline++ [5] | $61.75\pm 0.95$ | $78.51\pm 0.59$ | $47.15\pm 0.49$ | $66.18\pm 0.18$
MatchingNet [40] | $60.19\pm 1.02$ | $75.11\pm 0.35$ | $48.25\pm 0.65$ | $62.71\pm 0.44$
ProtoNet [36] | $52.52\pm 1.90$ | $75.93\pm 0.46$ | $44.19\pm 1.30$ | $64.07\pm 0.65$
RelationNet [39] | $62.52\pm 0.34$ | $78.22\pm 0.07$ | $48.76\pm 0.17$ | $64.20\pm 0.28$
DKT + CosSim [25] | $63.37\pm 0.19$ | $77.73\pm 0.26$ | $48.64\pm 0.45$ | $62.85\pm 0.37$
DKT + BNCosSim [25] | $62.96\pm 0.62$ | $77.76\pm 0.62$ | $49.73\pm 0.07$ | $64.00\pm 0.09$
GPLDLA [15] | $63.40\pm 0.14$ | $78.86\pm 0.35$ | $52.58\pm 0.19$ | –
amortized Bayesian prototype meta-learning [38] | $63.46\pm 0.98$ | $\mathit{80.94\pm 0.62}$ | $53.28\pm 0.91$ | ${70.44\pm 0.72}$
VAMPIRE [22] | – | – | $51.54\pm 0.74$ | $64.31\pm 0.74$
PLATIPUS [8] | – | – | $50.13\pm 1.86$ | –
ABML [30] | $49.57\pm 0.42$ | $68.94\pm 0.16$ | $45.00\pm 0.60$ | –
OVE PG GP + Cosine (ML) [35] | $63.98\pm 0.43$ | $77.44\pm 0.18$ | $50.02\pm 0.35$ | $64.58\pm 0.31$
OVE PG GP + Cosine (PL) [35] | $60.11\pm 0.26$ | $79.07\pm 0.05$ | $48.00\pm 0.24$ | $67.14\pm 0.23$
Reptile [23] | – | – | $49.97\pm 0.32$ | $65.99\pm 0.58$
R2-D2 [4] | – | – | $48.70\pm 0.60$ | $65.50\pm 0.60$
VSM [48] | – | – | ${54.73\pm 1.60}$ | $68.01\pm 0.90$
PPA [26] | – | – | $54.53\pm 0.40$ | –
DFSVLwF [9] | – | – | $\mathbf{56.20\pm 0.86}$ | –
FEAT [45] | $\mathbf{68.87\pm 0.22}$ | $\mathbf{82.90\pm 0.15}$ | $\mathit{55.15\pm 0.20}$ | $\mathbf{71.61\pm 0.16}$
HyperShot [34] | ${65.27\pm 0.24}$ | $79.80\pm 0.16$ | $52.42\pm 0.46$ | $68.78\pm 0.29$
HyperShot+ finetuning [34] | $\mathit{66.13\pm 0.26}$ | ${80.07\pm 0.22}$ | $53.18\pm 0.45$ | $69.62\pm 0.28$
MAML [7] | $56.11\pm 0.69$ | $74.84\pm 0.62$ | $45.39\pm 0.49$ | $61.58\pm 0.53$
MAML++ [1] | – | – | $52.15\pm 0.26$ | $68.32\pm 0.44$
iMAML-HF [28] | – | – | $49.30\pm 1.88$ | –
SignMAML [6] | – | – | $42.90\pm 1.50$ | $60.70\pm 0.70$
Bayesian MAML [47] | $55.93\pm 0.71$ |  | $53.80\pm 1.46$ | $64.23\pm 0.69$
Unicorn-MAML [46] | – | – | $54.89$ | –
Meta-SGD [17] | – | – | $50.47\pm 1.87$ | $64.03\pm 0.94$
MetaNet [20] | – | – | ${49.21\pm 0.96}$ | –
PAMELA [27] | – | – | $53.50\pm 0.89$ | $\mathit{70.51\pm 0.67}$
HyperMAML | $\mathit{66.11\pm 0.28}$ | $78.89\pm 0.19$ | $51.84\pm 0.57$ | $66.29\pm 0.43$

##  9 Appendix: Computational efficiency – results on natural image datasets

In this section, we perform similar experiments to those described in Section
4.3 of the main paper and measure the inference time of HyperMAML and MAML
with different numbers of gradient steps on CUB and mini-ImageNet datasets. We
summarize the results in Table 6, 7, 8. Similarly to the benchmark performed
on smaller images from the Omniglot $\rightarrow$ EMNIST dataset, the
inference time of HyperMAML is comparable to MAML with two gradient steps.
Likewise, MAML never achieves accuracy higher than HyperMAML.

As opposed to Section 4.3 of the main paper, we report the accuracies of MAML
only up to seven gradient steps. This is due to an insufficient amount of GPU
memory available for making more steps in the MAML implementation we used [5].
We also note that the accuracies of MAML reported here are significantly
higher than the ones reported in Table 1 of the main paper. The MAML
accuracies from that table were previously reported in the literature [25],
whereas the results in Table 6, 7, 8 have been obtained by the MAML
implementation in our codebase [5].

|  | CUB | mini-ImageNet
---|---|---|---
Model | Steps | Time | Accuracy | Time | Accuracy
MAML | 0 | $4.15\pm 0.03$ | $20.01\pm 0.02$ | $4.24\pm 0.14$ | $20.10\pm 0.31$
1 | $6.89\pm 0.10$ | $47.99\pm 0.22$ | $8.48\pm 0.14$ | $43.46\pm 0.42$
2 | $9.44\pm 0.15$ | $58.53\pm 0.55$ | $10.50\pm 0.80$ | $48.71\pm 0.19$
3 | $11.60\pm 0.02$ | $61.54\pm 0.41$ | $13.17\pm 0.26$ | $48.32\pm 0.38$
4 | $12.97\pm 0.09$ | $62.18\pm 0.38$ | $15.17\pm 0.18$ | $48.82\pm 0.25$
5 | $15.00\pm 0.17$ | $62.66\pm 0.40$ | $17.29\pm 0.32$ | $48.89\pm 0.31$
6 | $16.90\pm 0.12$ | $62.77\pm 0.36$ | $19.01\pm 0.26$ | $48.80\pm 0.24$
7 | $19.07\pm 0.17$ | $62.83\pm 0.34$ | $20.55\pm 0.30$ | $48.87\pm 0.29$
HyperMAML | – | $8.46\pm 0.37$ | $65.79\pm 0.88$ | $11.11\pm 0.25$ | $51.38\pm 0.37$
Table 6: Time (in seconds) spent on processing the entire CUB and mini-
ImageNet test datasets (600 tasks each) by MAML with different numbers of
gradient steps and HyperMAML.

##  10 Appendix: Training details

In this section, we present details of the training and architecture overview.

###  10.1 Architecture overview

The architecture of HyperMAML consists of the following parts (as outlined in
the Fig. 5 in the main body of the work):

#### Encoder

For each experiment described in the main body of this work, we utilize a
shallow convolutional encoder (feature extractor), commonly used in the
literature [7, 5, 25]. This encoder consists of four convolutional layers,
each consisting of a convolution, batch normalization, and ReLU nonlinearity.
Each of the convolutional layers has an input and output size of 64, except
for the first layer, where the input size is equal to the number of image
channels. We also apply max-pooling between each convolution, by which the
resolution of the processed feature maps is decreased by half. The output of
the encoder is flattened to process it in the next layers.

In the case of the Omniglot and EMNIST images, such encoder compresses the
images into 64-element embedding vectors, which serve as input to the
Hypernetwork (with the above-described enhancements) and the classifier.
However, in the case of substantially larger mini-ImageNet and CUB images, the
backbone outputs a feature map of shape $[64\times 5\times 5]$, which would
translate to $1600$-element embeddings and lead to an over parametrization of
the Hypernetwork and classifier which processes them and increase the
computational load. Therefore, we apply an average pooling operation to the
obtained feature maps and ultimately also obtain embeddings of shape $64$.
Thus, we can use significantly smaller Hypernetworks.

#### Hypernetwork

The Hypernetwork transforms the enhanced embeddings of the support examples of
each class in a task into the updates for the portion of classifier weights
predicting that class. It consists of three fully-connected layers with ReLU
activation function between each consecutive pair of layers. In the
hypernetwork, we use a hidden size of $256$ or $512$.

#### Classifier

The universal classifier is a single fully-connected layer with the input size
equal to the encoder embedding size (in our case 64) and the output size equal
to the number of classes. When using the strategy with embeddings enhancement,
we freeze the classifier to get only the information about the behavior of the
classifier, this means we do not calculate the gradient for the classifier in
this step of the forward pass. Instead, gradient calculation for the
classifier takes place during the classification of the query data.

###  10.2 Training details

In all of the experiments described in the main body of this work, we utilize
the switch and the embedding enhancement mechanisms. During training, we use
the Adam optimizer and the MultiStepLR learning rate scheduler with the decay
of $0.3$ and learning rate starting from $0.01$ or $0.001$. We train HyperMAML
for 4000 epochs on all the datasets, save for the simpler Omniglot
$\rightarrow$ EMNIST classification task, where we train for 2048 epochs
instead.

###  10.3 Hyperparameters

Below, we outline the hyperparameters of architecture and training procedures
used in each experiment.

hyperparameter | CUB | mini-ImageNet | mini-ImageNet $\rightarrow$ CUB | Omniglot $\rightarrow$ EMNIST
---|---|---|---|---
learning rate | $0.01$ | $0.001$ | $0.001$ | $0.01$
Hyper Network depth | $3$ | $3$ | $3$ | $3$
Hyper Network width | $512$ | $256$ | $256$ | $512$
epochs no. | $4000$ | $4000$ | $4000$ | $2048$
warmup milestones | $51,550$ | $101,1100$ | $101,1100$ | $51,550$
Table 7: Hyperparameters for each of conducted 1-shot experiments. hyperparameter | CUB | mini-ImageNet | mini-ImageNet $\rightarrow$ CUB | Omniglot $\rightarrow$ EMNIST
---|---|---|---|---
learning rate | $0.001$ | $0.001$ | $0.001$ | $0.01$
Hyper Network depth | $3$ | $3$ | $3$ | $3$
Hyper Network width | $256$ | $256$ | $256$ | $512$
epochs no. | $4000$ | $4000$ | $4000$ | $2048$
milestones | $101,1100$ | $101,1000$ | $101,1000$ | $51,550$
Table 8: Hyperparameters for each of the conducted 5-shot experiments.

##  11 Appendix: Implementation details

We implement HyperMAML using the PyTorch framework. We attach the zipped
folder containing the source code with our experiments as a part of the
supplementary material. We shall release the code publicly after the end of
the review period.

Each experiment described in this work was run on a single NVIDIA RTX 2080
GPU. Typical HyperMAML model trained on the Omniglot $\rightarrow$ EMNIST and
mini-ImageNet datasets takes about 2 and 100 hours, respectively.