# Zoo-Tuning: Adaptive Transfer from a Zoo of Models

Yang Shu  Zhi Kou  Zhangjie Cao  Jianmin Wang  Mingsheng Long

###### Abstract

With the development of deep networks on various large-scale datasets, a large
zoo of pretrained models are available. When transferring from a model zoo,
applying classic single-model based transfer learning methods to each source
model suffers from high computational burden and cannot fully utilize the rich
knowledge in the zoo. We propose _Zoo-Tuning_ to address these challenges,
which learns to adaptively transfer the parameters of pretrained models to the
target task. With the learnable channel alignment layer and adaptive
aggregation layer, Zoo-Tuning _adaptively aggregates channel aligned
pretrained parameters_ to derive the target model, which promotes knowledge
transfer by simultaneously adapting multiple source models to downstream
tasks. The adaptive aggregation substantially reduces the computation cost at
both training and inference. We further propose lite Zoo-Tuning with the
temporal ensemble of batch average gating values to reduce the storage cost at
the inference time. We evaluate our approach on a variety of tasks, including
reinforcement learning, image classification, and facial landmark detection.
Experiment results demonstrate that the proposed adaptive transfer learning
approach can transfer knowledge from a zoo of models more effectively and
efficiently.

Transfer Learning, Fine-Tuning, Model Zoo

##  1 Introduction

Transfer learning leverages knowledge from existing datasets to improve the
learning on the target task (Pan & Yang, 2009). With deep learning achieving
state-of-the-art performance on various machine learning tasks (LeCun et al.,
2015; Goodfellow et al., 2016), deep transfer learning has attracted more
attention in recent years (Tan et al., 2018). A standard deep transfer
learning paradigm is to leverage models pretrained on large-scale datasets
(Russakovsky et al., 2015) and _fine-tune_ the model on the target task
(Yosinski et al., 2014), which is demonstrated to be a simple and effective
solution in real-world applications (Kornblith et al., 2019).

However, most of the existing fine-tuning approaches only transfer from a
single pretrained deep network to the target task (Xuhong et al., 2018; Li et
al., 2019; Chen et al., 2019). With the increasing number of large-scale
datasets in various fields, we usually have access to a zoo of deep models
pretrained on various datasets with different methods such as supervised
learning (He et al., 2016), self-supervised learning (Dosovitskiy et al.,
2014) and unsupervised learning (Chen et al., 2020). Therefore, the large
diverse model zoo defines a new problem setting, Transfer Learning from a Zoo
of Models, which aims to transfer knowledge from multiple models to promote
the learning of the target task.

The new problem setting introduces new challenges: (1) How to decide the
extent of knowledge to transfer from different pretrained models? The
different pretrained models contain a diverse range of knowledge. Some
pretrained models are more related to the target tasks, while some models are
unrelated to or even negatively influence the target learning. For example,
although it is common to initialize models with parameters trained on
ImageNet, empirical evidence has shown that this practice offers little
benefit to applications in medical imaging (Raghu et al., 2019). Therefore, it
is important to decide the correct models to transfer from. (2) How to
aggregate the knowledge from different pretrained models? The diverse
pretrained models can be complementary to each other, which serve as a more
complete knowledge base for the target task. Thus, a model aggregating
mechanism is required to integrate knowledge from multiple pretrained models,
which further improves the target task. Directly transferring each model to
the target and assembling the fine-tuned models may aggregate knowledge but
suffers from the large training and inference cost linearly increasing with
the size of the model zoo.

In this paper, we propose Zoo-Tuning, an effective and efficient solution that
enables adaptive transfer from a zoo of models to downstream tasks. We decide
to transfer model parameters to avoid the high computation burden of
forwarding data through all the pretrained models. We first align channels of
different pretrained model parameters by a channel alignment layer since
pretrained models are trained on diverse datasets. We then aggregate the
pretrained model parameters with the weights controlled by a gating network.
The channel alignment layer and the aggregation module are learned, and
simultaneously, the source pretrained models are tuned by the target loss
signals to adaptively fit the target task. We further develop a lite version
of Zoo-Tuning with a unified gating value for all data in the target task
computed by the temporal ensemble of average gating values of each data batch.
Lite Zoo-Tuning further reduces the computation and storage cost at inference
time, which serves as an option to trade-off between efficiency and
performance.

The contributions of the paper can be summarized as:

  * •

We propose Zoo-Tuning, an adaptive transfer method to enable transfer learning
from a zoo of models. Zoo-Tuning aligns channels of source pretrained models
and learns data-dependent gating networks to aggregate source model
parameters, which transfers knowledge from a zoo of source pretrained models
with a low computation budget. All the modules are learned, and
simultaneously, the pretrained models are tuned by the target loss signals to
fit into the target task adaptively.

  * •

We develop a lite version of Zoo-Tuning by the temporal ensemble of average
gating values of each data batch, which further saves the inference and
storage cost and serves as an optional method to trade-off efficiency and
accuracy.

  * •

We conduct experiments on a variety of tasks, including reinforcement
learning, image classification, and facial landmark detection. Experimental
results show that the proposed method outperforms single-model transfer
methods (and their extensions) and multiple-model transfer methods while
remaining efficient.

##  2 Related Work

Transfer Learning. Transfer learning is a machine learning paradigm to
transfer knowledge from source domains to improve the learning of the target
task (Pan & Yang, 2009). A promising way to leverage knowledge in pretrained
models is to use features extracted by the networks (Oquab et al., 2014;
Donahue et al., 2014) or fine-tune from pretrained networks (Agrawal et al.,
2014; Girshick et al., 2014). To study the transferability of a pretrained
model, Yosinski et al. (2014) experimentally quantified the generality versus
specificity of neurons in each layer of a deep convolutional neural network.
Long et al. (2015, 2018) further proposed to learn transferable
representations. Recently, many empirical studies consider the effect of
transferring from a pretrained model on various downstream tasks and
scenarios, such as classification (Kornblith et al., 2019; Zhai et al., 2019),
few-shot learning (Ramalho et al., 2019), medical imaging (Raghu et al.,
2019), object detection and instance segmentation (He et al., 2019). Some
works utilize knowledge of a pretrained model by regularizing the parameters
(Xuhong et al., 2018) or features (Li et al., 2019; Chen et al., 2019). Others
propose auxiliary tasks to explore category relationships (You et al., 2020)
or intrinsic structures (Zhong et al., 2020) of data. However, it remains
unclear how to extend single model transfer learning techniques to a zoo of
models, where simply assembling every single model is fairly inefficient.

Transfer from Multiple Models. In the face of a zoo of models, recent works
seek to predict the transferability of pretrained models to select the best
one (Tran et al., 2019; Bao et al., 2019; Nguyen et al., 2020; You et al.,
2021). These methods suffer from two shortcomings: The selection may be
inaccurate, which hurts the performance; Besides, only a single best model is
utilized, which wastes other rich knowledge in the whole model zoo. Some works
leverage prior knowledge by inserting features from pretrained models (Rusu et
al., 2016; Liu et al., 2019). These methods need to pass the input data
through all models during training or even inference, which may cause high
computation and memory costs. Furthermore, these methods use pretrained models
to guide the learning of a student model without tuning and adapting the whole
zoo to the target. Guidance without adaptation may fail when the downstream
tasks are more complex or less similar to the pretrained tasks.

Conditional Computation. Our method is also related to conditional computation
(Davis & Arel, 2013; Cho & Bengio, 2014), where parts of the network are
active on a per-example basis. Eigen et al. (2013) introduced the idea of
using multiple mixture-of-experts (Masoudnia & Ebrahimpour, 2014) with their
own gating networks as parts of a deep model. Shazeer et al. (2017) introduced
sparsely-gated mixture-of-experts layers to form outrageously large neural
networks. Different from these methods which increase model capacity to absorb
sufﬁciently large data by mixing outputs of sub-networks, we aim to transfer
knowledge in source models by adaptively aggregating model parameters.

Figure 1: The framework of our proposed method. We derive the target model
$\widehat{\mathbf{W}}$ by aggregating the parameters of the source models
$\mathbf{W}_{i}$ in each layer, controlled by the learnable adaptive
aggregation modules based on the input data. During training, the adaptive
modules are trained, and the source parameters are tuned to transfer to the
target task. After training, the tuned source models are aggregated depending
on each query data for inference.

##  3 Approach

In this section, we introduce the problem of transfer learning from a zoo of
models. Then we introduce our Zoo-Tuning approach, with channel alignment and
adaptive aggregation. We further propose a more efficient variant, lite Zoo-
Tuning.

###  3.1 Transfer Learning from a Zoo of Models

The most common transfer learning scenario considers only one single
pretrained model $M$ at hand to serve the target task
$\mathcal{D}=\\{(\mathbf{x},\mathbf{y})\\}$. In transfer learning from a zoo
of models, we consider a more complicated situation where we have a zoo of
pretrained models $\mathcal{M}=\\{M_{1},M_{2},\cdots,M_{m}\\}$. This problem
is challenging in two ways: (1) The diverse pretrained models hold different
relationships to the target tasks, which need transferring knowledge from
different pretrained models to different extents; (2) Different models are
pretrained on various data and thus store different knowledge, which may be
complementary to each other to solve the downstream tasks. How to aggregate
knowledge from various pretrained models is an essential but difficult problem
for model zoo transfer learning.

In this paper, we consider the situation that different models in the zoo have
the same architecture but are trained with different data, tasks, or
pretraining algorithms. This assumption of the same architecture is reasonable
and has its value in practice since architectures such as ResNet (He et al.,
2016) can be widely used in various datasets and tasks, and diverse pretrained
models of these architectures with rich source knowledge are provided in the
open-source community. It is easier and more reliable to apply these models
with the same simple and familiar architectures, especially on new problems.
Besides, The same architecture enables more effective layer-wise knowledge
transfer, which is hard to realize on different architectures. A more
generalized situation where models have arbitrary architectures would be
interesting and challenging to explore for future work.

###  3.2 Zoo-Tuning

We address the problem of transfer learning from a zoo of models by Zoo-
Tuning. The framework is shown in Figure 1. Zoo-Tuning enables knowledge
transfer from multiple models by adaptively aggregating source model
parameters in each layer, based on the input data, to form the target model.
The adaptive aggregation consists of channel alignment and gating networks to
control the extent of each source model in transfer learning. As the adaptive
aggregation mechanism is lightweight and the target data pass through the
derived target model instead of all source models, Zoo-Tuning only introduces
similar inference time to a single model, which is computationally efficient.
We further propose a lite version of Zoo-Tuning to reduce the storage cost.

(a) Adaptive Aggregation (AdaAgg) Layer

(b) Integrating AdaAgg Layers in a Residual Block

Figure 2: (a) Illustration of the Adaptive Aggregation Layer. The target input
$\hat{\mathbf{h}}^{l-1}$ goes through the gating networks $\mathbf{A}_{i}^{l}$
to compute gating values. The source parameters $\mathbf{W}_{i}^{l}$ are first
aligned by $\mathbf{T}_{i}^{l}$ and then aggregated with these gating values
to form the target parameters $\widehat{\mathbf{W}}^{l}$. The input
$\hat{\mathbf{h}}^{l-1}$ is finally forwarded through the layer parameterized
by $\widehat{\mathbf{W}}^{l}$. (b) We can change the layers of the network
backbone into AdaAgg layers to aggregate models in the zoo. Here is an example
where the backbone is composed of residual blocks.

Channel Alignment. Different models are separately trained on diverse datasets
or tasks, so even parameters at the same channel of the same layer in
different pretrained models may indicate different semantic meanings. The
misaligned channels cause difficulty in aggregating parameters of different
pretrained models. To address the problem, we adopt a channel alignment module
that transforms and aligns channels of different pretrained models. We
consider parameters $\mathbf{W}_{i}^{l}$ of the $l$-th convolutional layer in
any source model $M_{i}$ with the size $C_{\text{out}}\times
C_{\text{in}}\times K\times K$, where $C_{\text{out}}$ is the number of output
channels, $C_{\text{in}}$ indicates input channels, and $K$ is the kernel size
of the convolutional layer. We adopt a lightweight convolutional layer
$\mathbf{T}_{i}^{l}$ with $1\times 1$ kernel of size $C_{\text{out}}\times
C_{\text{out}}\times 1\times 1$ as the channel alignment module. Specifically,
the channels in the source convolutional parameters $\mathbf{W}_{i}^{l}$ are
reorganized by the channel alignment layer to derive the transformed
parameters $\widetilde{\mathbf{W}}_{i}^{l}$ as follows:

$\widetilde{\mathbf{W}}_{i}^{l}=\mathbf{T}_{i}^{l}*\mathbf{W}_{i}^{l},$ (1)

where we also use $\mathbf{T}_{i}^{l}$ to denote the parameters of the
alignment module. We show an implementation of the channel alignment module
for source parameters of convolutional layers here. The idea is easy to extend
to other kinds of layers, such as employing a linear alignment layer for fully
connected layers. We initialize the channel alignment layer as an identical
mapping, which gives the target model a smooth warm-up from the pretrained
weights.

Adaptive Aggregation. With channel-aligned source parameters, we develop an
adaptive aggregation (AdaAgg) layer to dynamically aggregate source model
parameters. We have two key insights in the design of the AdaAgg layer: (1)
Each data point of each downstream task should have a different aggregation
since each data point holds specific relationships with source tasks; (2) The
aggregation should be computationally efficient for a large number of source
models. We integrate these two key insights into the design of the AdaAgg
layer. As shown in Figure 2(a), considering the $l$-th layer of the network,
the AdaAgg layer is equipped with a gating network $\mathbf{A}_{i}^{l}$ for
each source model $M_{i}$, which controls the mixing of its corresponding
parameters $\mathbf{W}_{i}^{l}$. The gating network $\mathbf{A}_{i}^{l}$ takes
the feature of the previous layer in the target model $\hat{\mathbf{h}}^{l-1}$
as the input and outputs the gating value $a_{i}^{l}$. The aligned source
parameters $\widetilde{\mathbf{W}}_{i}^{l}$ are aggregated with the gating
values to derive the parameters of the target model in this layer
$\widehat{\mathbf{W}}^{l}$ as follows:

$\widehat{\mathbf{W}}^{l}=\sum_{i=1}^{m}a_{i}^{l}\widetilde{\mathbf{W}}_{i}^{l}=\sum_{i=1}^{m}\mathbf{A}_{i}^{l}(\hat{\mathbf{h}}^{l-1})\left(\mathbf{T}_{i}^{l}*\mathbf{W}_{i}^{l}\right),$
(2)

We consider lightweight gating networks to reduce the computation and storage
cost of the gating network. For example, for a convolutional layer, the gating
network consists of a global average pooling layer, $2$ convolutional layers
with $1\times 1$ kernel, and a sigmoid activation function. Such design brings
little additional computational cost of the gating network compared to
processing data with the original convolution operation, even with a large-
scale model zoo.

We can easily change the backbone layers of source models into AdaAgg layers
to aggregate source models’ parameters in each layer. In Figure 2(b), we give
an example of the residual block. With the target model parameters, the target
data are passed through the target model for training and inference. We can
solve the optimization problem of adapting the model zoo to the target task as
follows:

$\displaystyle\min_{\mathbf{\Theta}}\mathbb{E}_{(\mathbf{x},\mathbf{y})\sim\mathcal{D}}\mathcal{J}\left(f^{L}(\cdot;\widehat{\mathbf{W}}^{L})\circ\cdots\circ
f^{1}(\mathbf{x};\widehat{\mathbf{W}}^{1}),\mathbf{y}\right),$ (3)

where $\mathcal{D}=\\{(\mathbf{x},\mathbf{y})\\}$ denotes the target dataset,
$L$ is the total number of layers, $f^{l}$ is the operation of the $l$-th
layer parameterized by $\widehat{\mathbf{W}}^{l}$ and $\mathcal{J}$ is the
loss for the target task.
$\mathbf{\Theta}=\left(\mathbf{W}_{i},\mathbf{A}_{i},\mathbf{T}_{i}\right)$
includes source model parameters $\mathbf{W}_{i}^{l}$, channel alignment
parameters $\mathbf{T}_{i}^{l}$, and gating network parameters
$\mathbf{A}_{i}^{l}$ in all AdaAgg layers. All of these parameters are
adaptively trained or tuned to fit for the target task.

Figure 3: Lite Zoo-Tuning learns a unified inference model for all data in the
target task based on the temporal ensemble of adaptive aggregation values over
batches. Thus, we just need to store the aggregated target model
$\widehat{\mathbf{W}}$ for all target data during inference.

###  3.3 Lite Zoo-Tuning

The adaptive aggregation introduced above is computationally efficient but
still requires all the source parameters at the inference stage, as the gating
values can be computed only when the target data is presented at inference
time. As shown in Figure 3, to further save the storage for applying Zoo-
Tuning to devices with limited resource, we relax the dependency of the gating
values on each individual target sample to the dependency on the entire
dataset, by temporal ensemble of the adaptive aggregation over batches. This
results in a unified gating value and model for all data during inference.
During training, for a layer $l$ of a source model $i$, we firstly compute
gating values for each sample in the batch, denoted as $a_{i,j}^{l}$, where
$j$ is the index of the sample. Then we compute the batch average gating
values and their temporal ensemble (TE) over the training batches as follows:

$\displaystyle\bar{a}_{i}^{l}=\lambda\cdot\bar{a}_{i}^{l}+(1-\lambda)\left(\frac{1}{b}\cdot\sum_{j=1}^{b}a_{i,j}^{l}\right).$
(4)

We use the batch average values for all training data in the batch and update
the temporal ensemble values with $\lambda=0.9$, which is commonly used in
temporal ensemble techniques. The temporal ensemble values reflect how the
target task relies on each source pretrained model and thus serve as the
unified gating values for all target data in inference. Now all target data
share the same gating values, so we can pre-compute the aggregation of source
parameters to form the target model before inference as follows:

$\widehat{\mathbf{W}}^{l}=\sum_{i=1}^{m}\bar{a}_{i}^{l}\widetilde{\mathbf{W}}_{i}^{l}.$
(5)

The key difference between Eqn. (2) and Eqn. (5) is that $\bar{a}_{i}^{l}$ in
Eqn. (5) is shared by all test data and does not change with each sample. So
at inference, the target data can be directly forwarded through the pre-
aggregated target model. Thus, the cost of lite Zoo-Tuning model in
computation and storage is close to a single model. We coin Eqn. (5) Temporal
Ensemble Adaptive Aggregation (TE-AdaAgg).

###  3.4 Complexity Analysis

As we propose layer-wise adaptive transfer of source parameters, for
simplicity, we only analyze the complexity of one layer, which can be extended
to the whole model. We consider aggregating convolutional layers of $m$
pretrained models. Suppose the dimension of the layer is $C_{\text{out}}\times
C_{\text{in}}\times K\times K$ where $C_{\text{out}}$ and $C_{\text{in}}$ are
the numbers of output and input channels, and $K$ is the kernel size. The
input feature map has the dimension $C_{\text{in}}\times H\times W$, where $H$
and $W$ are spatial dimensions. $W$ means the width of the feature map only in
this Section 3.4 to avoid notation abuse. The original convolution operation
has the complexity $O\left(HWK^{2}C_{\text{out}}C_{\text{in}}\right)$. Zoo-
Tuning introduces additional computation for channel alignment, gating
networks, and adaptive aggregation with the cost of
$O\left(mK^{2}C_{\text{out}}^{2}C_{\text{in}}\right)$,
$O\left(HWC_{\text{in}}+mC^{2}_{\text{in}}\right)$ and
$O\left(mK^{2}C_{\text{out}}C_{\text{in}}\right)$ respectively, which is
$O\left(mK^{2}C_{\text{out}}^{2}C_{\text{in}}+HWC_{\text{in}}+mC^{2}_{\text{in}}\right)$
in total. At inference, since the channel alignment is data-independent and
can be pre-computed, the computational cost becomes
$O\left(mK^{2}C_{\text{out}}C_{\text{in}}+HWC_{\text{in}}+mC^{2}_{\text{in}}\right)$.
Compared with the original cost of convolution operation, the additional cost
for Zoo-Tuning is small. We will also empirically compare the computational
complexity of other transfer learning methods and different variants of Zoo-
Tuning.

##  4 Experiments

We conduct experiments within three experimental settings. In the first
setting, we use a zoo of reinforcement learning models pretrained on various
Atari games and transfer to a different set of games. In the other two
settings, we use a zoo of diverse computer vision models pretrained on various
large-scale datasets and transfer to multiple downstream tasks for
classification and facial landmark detection. All experiments are implemented
in the PyTorch framework.

(a) Alien

(b) Gopher

(c) JamesBond

Figure 4: Results of transferring pretrained models to downstream tasks in the
reinforcement learning of Atari games.

###  4.1 Transfer Learning in Reinforcement Learning

To demonstrate the generalizability of the proposed Zoo-Tuning method to
various domains, we first conduct experiments on reinforcement learning
models.

Benchmarks. Although many reinforcement learning algorithms can achieve much
better performance than humans on Atari games, they are still far less
efficient than a human learner. So we measure our method at 100k interaction
steps (400k environment steps with action repeat of 4) on Atari, which
corresponds to the time for a human learner. We use the Seaquest and Riverraid
tasks as source tasks and learn an optimal policy by reinforcement learning
for each task. We transfer the pretrained reinforcement learning models to $3$
downstream tasks: Alien, Gopher, and JamesBond.

Implementation Details. For the learning algorithm, we follow the
implementation of Data-Efficient Rainbow (van Hasselt et al., 2019), which
modifies hyper-parameters of Rainbow (Hessel et al., 2018) for data
efficiency. The model of Data-Efficient Rainbow consists of $2$ convolutional
layers: $32$ filters of size $5\times 5$ with the stride of $5$ and $64$
filters of size $5\times 5$ with the stride of $5$, followed by a flatten
layer and $2$ noisy linear layers (Fortunato et al., 2018) with the hidden
size of $256$. We use Adam optimizer (Kingma & Ba, 2015) with a learning rate
of $1\times 10^{-4}$. Other hyper-parameters are kept the same as those in van
Hasselt et al. (2019). We repeat each experiment 5 times with different seeds
and report the mean and variance of the results.

Results. As shown in Figure 4, in all the downstream tasks, Zoo-Tuning
outperforms transferring from a single pretrained model and Knowledge Flow
(Liu et al., 2019), which also transfers from multiple pretrained models. The
results indicate that Zoo-Tuning enables utilizing knowledge from pretrained
policies to promote target tasks by adaptive transfer. Simply fine-tuning from
each pretrained model performs similarly with training from scratch, and even
slightly worse, demonstrating that when the source and target tasks have a
large distribution shift, brutely transferring knowledge may cause negative
transfer.

###  4.2 Transfer Learning in Image Classification

For the zoo of models in the classification setting, we use $5$ ResNet-50
models pretrained on representative computer vision datasets: (1) Supervised
pretrained model and (2) Unsupervised pretrained model with MOCO (He et al.,
2020) on ImageNet (Russakovsky et al., 2015), (3) Mask R-CNN (He et al., 2017)
model for detection and instance segmentation, (4) DeepLabV3 (Chen et al.,
2018) model for semantic segmentation, and (5) Keypoint R-CNN model for
keypoint detection, pretrained on COCO-2017 challenge datasets of each task.
In total, the zoo of models are trained on millions of images across a wide
range of computer vision tasks, which contains _abundant knowledge in the
computer vision domain._ All pretrained models are found in torchvision
(Paszke et al., 2017) or original implementation.

Table 1: Comparison of top-1 accuracy($\%$) and complexity on the
classification benchmarks including the general benchmark, fine-grained
benchmark, and specialized benchmark.

Model | General | Fine-grained | Specialized |  | Train | Inference
---|---|---|---|---|---|---
Cifar-100 |  COCO-70 |  Aircraft |  Cars |  Indoors |  DMLab |  EuroSAT |  Avg. Acc. |  GFLOPs |  Params |  GFLOPs |  Params
ImageNet sup. | 81.18 | 81.97 | 84.63 | 89.38 | 73.69 | 74.57 | 98.43 | 83.41 | 4.12 | 23.71M | 4.12 | 23.71M
MOCO pt. | 75.31 | 75.66 | 83.44 | 85.38 | 70.98 | 75.06 | 98.82 | 80.66 | 4.12 | 23.71M | 4.12 | 23.71M
MaskRCNN pt. | 79.12 | 81.64 | 84.76 | 87.12 | 73.01 | 74.73 | 98.65 | 82.72 | 4.12 | 23.71M | 4.12 | 23.71M
DeepLab pt. | 78.76 | 80.70 | 84.97 | 88.03 | 73.09 | 74.34 | 98.54 | 82.63 | 4.12 | 23.71M | 4.12 | 23.71M
Keypoint pt. | 76.38 | 76.53 | 84.43 | 86.52 | 71.35 | 74.58 | 98.34 | 81.16 | 4.12 | 23.71M | 4.12 | 23.71M
ensemble | 82.26 | 82.81 | 87.02 | 91.06 | 73.46 | 76.01 | 98.88 | 84.50 | 20.60 | 118.55M | 20.60 | 118.55M
distill | 82.32 | 82.44 | 85.00 | 89.47 | 73.97 | 74.57 | 98.95 | 83.82 | 24.72 | 142.28M | 4.12 | 23.71M
Knowledge Flow | 81.56 | 81.91 | 85.27 | 89.22 | 73.37 | 75.55 | 97.99 | 83.55 | 28.83 | 169.11M | 4.12 | 23.71M
Lite Zoo-Tuning | 83.39 | 83.50 | 85.51 | 89.73 | 75.12 | 75.22 | 99.12 | 84.51 | 4.53 | 130.43M | 4.12 | 23.71M
Zoo-Tuning | 83.77 | 84.91 | 86.54 | 90.76 | 75.39 | 75.64 | 99.12 | 85.16 | 4.53 | 130.43M | 4.18 | 122.54M

Benchmarks. We divide the $7$ downstream tasks into three types of benchmarks:
general benchmarks, fine-grained benchmarks, and specialized benchmarks, to
verify the efficacy of the proposed Zoo-Tuning on different kinds of images.
(1) General benchmarks with CIFAR-100 (Krizhevsky et al., 2009) and COCO-70:
CIFAR-100 contains $100$ classes with $600$ images per class, which are split
into $500$ training images and $100$ testing images. COCO-70 is constructed by
cropping objects for each image in COCO dataset (Lin et al., 2014) and
removing minimal items. It contains $70$ classes with more than $1,000$ images
per category. (2) Fine-grained benchmarks with FGVC Aircraft (Maji et al.,
2013), Stanford Cars (Krause et al., 2013) and MIT-Indoors (Quattoni &
Torralba, 2009): FGVC Aircraft is a benchmark for the fine-grained aircraft
categorization. It has $100$ categories containing $100$ images each. Stanford
Cars contains $16,185$ images for $196$ classes of cars. The data are split
into $8,144$ training images and $8,041$ testing images. MIT-Indoors contains
$67$ Indoor categories, and a total of $15,620$ images. We use a subset of the
dataset that has $80$ images for training and $20$ images for testing per
class. (3) Specialized benchmarks with DMLab (Beattie et al., 2016) and
EuroSAT (Helber et al., 2019): DMLab contains frames observed by the agent
acting in the DeepMind Lab environment, which are annotated by the distance
between the agent and various objects present in the environment. The data are
split into $65,550$ training images, $22,628$ validation images and $22,735$
test images. EuroSAT dataset is based on Sentinel-2 satellite images covering
$13$ spectral bands and consisting of $10$ classes with $27,000$ labeled and
geo-referenced samples.

Implementation Details. We follow the common fine-tuning principle described
in (Yosinski et al., 2014) and replace the last task-specific classification
layer with a randomly initialized fully connected layer. We adopt SGD with a
learning rate of $0.01$ and momentum of $0.9$ with the same training strategy
(total $15$k iterations for fine-tuning with learning rate decay per $6$k
iterations) for all pretrained models, compared methods and the proposed Zoo-
Tuning. This ensures a fair comparison between different methods and avoids
over-tuning on specific tasks. We adopt a batch size of $48$, and all images
are randomly resized and cropped to $224\times 224$ as the input of the
network. More details can be found in supplementary materials.

Results. We report the top-1 accuracy on the test data of each task and the
complexity of each method. For our method, we report Zoo-Tuning and lite Zoo-
Tuning. For the single-model transfer method, we compare with fine-tuning from
every single pretrained model. For methods using all pretrained models, we
compare with three methods: using the ensemble of fine-tuned source models for
prediction, distilling from the ensemble, and Knowledge Flow (Liu et al.,
2019), which is designed to transfer from multiple models. From Table 1. We
have the following observations:

On all the three benchmarks, Zoo-Tuning consistently outperforms fine-tuning
from each single pretrained model, which indicates that Zoo-Tuning
successfully aggregates and utilizes the rich knowledge in the whole zoo of
models.

Compared with the methods using all pretrained models, Zoo-Tuning shows higher
or comparable performance on most of the tasks. Compared with the parameters
in the model zoo, the additional parameters in Zoo-Tuning is about $10\%$,
which shows that the adaptive modules are lightweight. With the adaptive
parameter aggregation mechanism, Zoo-Tuning is more computationally efficient.
Note that the ensemble predictions require ﬁne-tuning all the candidate
pretrained models on the target task firstly. Even at inference time, each
query sample should go through all the fine-tuned models to get the ﬁnal
prediction, causing high inference cost. Distilling and knowledge ﬂow show
similar inference costs as Zoo-Tuning, but Zoo-Tuning achieves higher
performance on almost all the tasks. The results demonstrate that Zoo-Tuning
is a both effective and efficient solution to transfer learning from a zoo of
models.

Lite Zoo-Tuning also outperforms compared methods in average accuracy. We
specially compare it with distilling from the ensemble (Distill) since they
are both efficient in inference. Although the performance gain is not large,
lite Zoo-Tuning still outperforms Distill consistently on all tasks and
achieves greater advantages in the training cost. This is because Distill
still needs to forward data through all source models, while lite Zoo-Tuning
only needs to pass the data through the aggregated model. Furthermore, Distill
needs to fine-tune all the pretrained models on the target data first and then
distill a target model from the ensemble outputs of fine-tuned models, which
requires a high training cost linearly increasing with the number of source
models. The results match the goal of lite Zoo-Tuning to substantially reduce
the storage cost in inference while keeping relatively high performance, which
is more scalable when training with a large number of source models.

Zoo-Tuning achieves higher accuracy than lite Zoo-Tuning, which demonstrates
that capturing fine-grained data-dependent gating values would help adapt the
pretrained models to the target task but with more cost of storage and
computation in inference. Lite Zoo-Tuning costs the same GFLOPs and parameters
as the single model in inference, with slight performance drop than Zoo-
Tuning, which serves as a trade-off between accuracy and efficiency.

###  4.3 Transfer Learning in Facial Landmark Detection

Benchmarks. To explore the usage of Zoo-Tuning on more diverse and complex
downstream vision tasks, we use the same model zoo as the image classification
tasks in Section 4.2 and consider transferring to three facial landmark
detection tasks, 300W (Sagonas et al., 2013), WFLW (Wu et al., 2018), and COFW
(Burgos-Artizzu et al., 2013). The 300W is a combination of HELEN (Le et al.,
2012), LFPW (Belhumeur et al., 2013), AFW (Zhu & Ramanan, 2012), XM2VTS and
IBUG datasets, where each face has $68$ landmarks. We follow (Ren et al.,
2016) and use the $3148$ training images. We evaluate the performance using
the full set containing $689$ images. The WFLW dataset is a built on WIDER
Face (Yang et al., 2016). There are $7500$ training images and 2500 testing
images with 98 manual annotated landmarks. The COFW dataset consists of $1345$
training faces and $507$ testing faces with $29$ facial landmarks.

Table 2: Comparison of NME results on facial landmark detection tasks: 300W, WFLW, and COFW. Model | 300W | WFLW | COFW
---|---|---|---
scratch | $3.66$ | $5.33$ | $4.20$
imagenet sup. | $3.52$ | $4.90$ | $3.66$
moco pt. | $3.45$ | $4.75$ | $3.63$
maskrcnn pt. | $3.53$ | $4.87$ | $3.67$
deeplab pt. | $3.53$ | $4.89$ | $3.73$
keypoint pt. | $3.50$ | $4.90$ | $3.66$
ensemble | $\mathbf{3.33}$ | $4.64$ | $\mathbf{3.46}$
distill | $3.45$ | $4.74$ | $3.53$
Knowledge Flow | $3.71$ | $5.28$ | $4.58$
Zoo-Tuning | $3.41$ | $\mathbf{4.58}$ | $3.51$

Implementation Details. We generally follow the protocol in Sun et al. (2019).
We follow the standard training scheme in (Wu et al., 2018). All the faces are
cropped by the provided boxes according to the center location and resized to
$256\times 256$. We augment the data by $\pm 30$ degrees in-plane rotation,
$0.75\sim 1.25$ scaling, and random ﬂipping. The models are trained for $60$
epochs with a batch size of $16$. We use Adam optimizer (Kingma & Ba, 2015).
The base learning rate is $1\times 10^{-4}$ and is decayed by a rate of $0.1$
at the $30$-th and $50$-th epochs. In testing, each keypoint location is
predicted by transforming the highest heat value location to the original
image space and adjusting it with a quarter offset in the direction from the
highest response to the second highest response (Chen et al., 2017).

Results. We use the inter-ocular distance as normalization and report the
normalized mean error (NME) for evaluation in Table 2. Comparing fine-tuning
from each single pretrained model, we can observe that the MOCO (He et al.,
2020) pretrained model generally outperforms other pretrained models when
transferring to the facial landmark detection tasks. The results confirm that
even commonly-used models such as the supervised ImageNet pretrained model
cannot dominate all downstream tasks, and it is important to select the more
suitable pretrained models for the target task. Zoo-Tuning addresses the
challenge by gating networks trained on the target task and consistently
outperforms transferring from each single model. Knowledge Flow achieves
little improvement than training from scratch and even performs worse on 300W
and COFW. This method uses pretrained models as teachers to guide the learning
of the student network. But this kind of guidance cannot fully utilize and
adapt the knowledge in the zoo, especially when the target tasks are not close
to pretrained tasks. Zoo-Tuning adapts the whole model zoo to the target,
which is a more effective way of knowledge transfer. Zoo-Tuning achieves
comparable performance with the ensemble, but with much less computational
cost during training and inference.

Figure 5: Visualization of gating values in each layer of source models using
the five vision pretrained models as source models and using CIFAR-100 as the
target task. From the left to right is from the bottom layer to the top layer.
The darker color means a higher gating value.

###  4.4 Analysis

Variants of Zoo-Tuning. We compare Zoo-Tuning with its variants to demonstrate
the efficacy of different modules in Zoo-Tuning. We use all the five
pretrained computer vision models described above and transfer them to the
COCO-70 dataset and the WFLW dataset. From Table 3, we have the following
observations: (1) Zoo-Tuning outperforms Zoo-Tuning w/o $\mathbf{T}$, which
demonstrates that channel alignment of source parameters improves the
performance for adaptive aggregation. (2) Zoo-Tuning w/o $\mathbf{T}$ and Zoo-
Tuning outperform average aggregation with a large margin. Average aggregation
aggregates all the source parameters with the same weight, which treats all
source parameters equally. The results demonstrate that it is essential to
learn the gating values on the target task and adaptively fit the aggregation
to better serve the target task.

Table 3: Ablation study on variants of Zoo-Tuning. Method | COCO | WFLW
---|---|---
Average Aggregation | 80.53 | 4.75
Zoo-Tuning w/o $\mathbf{T}$ | 83.92 | 4.64
Zoo-Tuning | 84.91 | 4.58

Visualization of Gating Values. We visualize the gating values in each layer
of each source model learned by Zoo-Tuning on the CIFAR-100 dataset. As shown
in Figure 5, different source models have diverse gating values, which
indicates that different source pretrained models have different relationships
to the target task, and Zoo-Tuning learns to aggregate the source models for
the target task adaptively. Different layers show different gating values,
which matches the study on deep networks that different layers contain
different knowledge and exhibit different transferability. The gating values
show some insights on the transferability of the pretrained models. Though
both trained on ImageNet, the supervised model has higher values than the
unsupervised one, especially on top layers. Overall, ImageNet supervised model
has the highest values, indicating that more related source tasks have more
transferable knowledge for the target. Besides, the advantage of the ImageNet
supervised model is mainly on top layers, and other networks also have high
values in the bottom and intermediate layers. This matches the previous
observation that knowledge in deep networks goes from general to task-specific
as the layer goes deeper.

Number of Parameters. Zoo-Tuning from a zoo of models outperforms the best
single model in the zoo with large margins. However, there is a concern that
this improvement may be due to the increase in model parameters. To evaluate
how the number of parameters influences the final result, we conduct an
ablation study where we change the $5$ diverse pretrained models to $5$
ImageNet supervised pretrained models and perform Zoo-Tuning from the $5$ same
models (denoted as $5\times$ ImageNet). We report the results on the image
classification datasets COCO-70 and Aircraft.

Table 4: Ablation study on parameters of the network. Method | Aircraft | COCO | Param
---|---|---|---
ImageNet | 84.63 | 81.97 | $1\times$
$5\times$ ImageNet | 84.92 | 82.13 | $5\times$
Lite Zoo-Tuning | 85.51 | 83.50 | $1\times$
Zoo-Tuning | 86.54 | 84.91 | $5\times$

As shown in Table 4, although the model of transferring from $5\times$
ImageNet has the same parameters as Zoo-Tuning from $5$ diverse pretrained
models, it achieves minor improvements compared with fine-tuning from one
single model. Note that the proposed lite Zoo-Tuning also outperforms fine-
tuning from a single ImageNet model, while lite Zoo-Tuning has the same number
of parameters with a single model. The results indicate that the key to the
superior performance of Zoo-Tuning is not simply increasing model parameters
but adaptively aggregating rich knowledge from the source models. The results
not only show the effectiveness of the proposed Zoo-Tuning, but also show the
importance of knowledge aggregation in the problem of transfer learning from a
zoo of diverse models.

Number of Pretrained Models. We study how the number of pretrained models in
the zoo affects the proposed method. We experiment on the CIFAR-100 dataset by
sequentially adding the MoCo, Keypoint R-CNN, DeepLabV3, Mask R-CNN, and
ImageNet supervised models into the model zoo. We report the results of Zoo-
Tuning with different numbers of pretrained models as well as the results of
the single best models in the corresponding zoo in Table 5.

Table 5: Ablation study on the number of models. # of models | 2 | 3 | 4 | 5
---|---|---|---|---
Single Best | 76.38 | 78.76 | 79.12 | 81.18
Zoo-Tuning | 78.62 | 79.78 | 81.50 | 83.77

From the results, with different model zoo sizes of $2$, $3$, $4$, and $5$
pretrained models, Zoo-Tuning consistently outperforms fine-tuning from the
best single model in the zoo, respectively. Note that it is even more
difficult to quickly select out the best pretrained model in practice. This
indicates that Zoo-Tuning can effectively and efficiently utilize the
knowledge in the whole model zoo to promote transfer learning performance,
which makes it a better choice in real-world applications. Also, we can find
that the performance of Zoo-Tuning increases with the increasing number of
pretrained models, which demonstrates that Zoo-Tuning can hold a varying
number of pretrained models and is extendable to more source models.

##  5 Conclusion

In this paper, we propose Zoo-Tuning to enable transfer learning from a zoo of
models. We align the channels of source parameters with a channel alignment
layer and adopt a gating network depending on the input data for each source
model to aggregate their parameters, which derives the target model. The
channel alignment layer and the gating network are trained, and the source
pretrained parameters are tuned by the target task to adaptively transfer
knowledge from the zoo of source models to the target task. We further propose
lite Zoo-Tuning with the temporal ensemble of batch average gating values,
which further reduces the storage cost at inference time. Experiment results
in reinforcement learning, image classification, and facial landmark detection
demonstrate that Zoo-Tuning achieves state-of-the-art performance with small
computational and storage costs.

## Acknowledgments

This work was supported by the National Key R&D Program of China
(2020AAA0109201), NSFC grants (62022050, 62021002, 61772299), Beijing Nova
Program (Z201100006820041), and MOE Innovation Plan of China.

## References

  * Agrawal et al. (2014) Agrawal, P., Girshick, R., and Malik, J.  Analyzing the performance of multilayer neural networks for object recognition.  In _ECCV_ , 2014.
  * Bao et al. (2019) Bao, Y., Li, Y., Huang, S.-L., Zhang, L., Zheng, L., Zamir, A., and Guibas, L.  An information-theoretic approach to transferability in task transfer learning.  In _ICIP_ , 2019.
  * Beattie et al. (2016) Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., et al.  Deepmind lab.  _arXiv preprint arXiv:1612.03801_ , 2016.
  * Belhumeur et al. (2013) Belhumeur, P. N., Jacobs, D. W., Kriegman, D. J., and Kumar, N.  Localizing parts of faces using a consensus of exemplars.  _TPAMI_ , 35(12):2930–2940, 2013.
  * Burgos-Artizzu et al. (2013) Burgos-Artizzu, X. P., Perona, P., and Dollár, P.  Robust face landmark estimation under occlusion.  In _ICCV_ , 2013.
  * Chen et al. (2018) Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In _ECCV_ , 2018.
  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.  A simple framework for contrastive learning of visual representations.  In _ICML_ , 2020.
  * Chen et al. (2019) Chen, X., Wang, S., Fu, B., Long, M., and Wang, J.  Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning.  In _NeurIPS_ , 2019.
  * Chen et al. (2017) Chen, Y., Shen, C., Wei, X.-S., Liu, L., and Yang, J.  Adversarial posenet: A structure-aware convolutional network for human pose estimation.  In _ICCV_ , 2017.
  * Cho & Bengio (2014) Cho, K. and Bengio, Y.  Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning.  _arXiv preprint arXiv:1406.7362_ , 2014.
  * Davis & Arel (2013) Davis, A. and Arel, I.  Low-rank approximations for conditional feedforward computation in deep neural networks.  _arXiv preprint arXiv:1312.4461_ , 2013.
  * Donahue et al. (2014) Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T.  Decaf: A deep convolutional activation feature for generic visual recognition.  In _ICML_ , 2014.
  * Dosovitskiy et al. (2014) Dosovitskiy, A., Springenberg, J. T., Riedmiller, M., and Brox, T.  Discriminative unsupervised feature learning with convolutional neural networks.  In _NeurIPS_ , 2014.
  * Eigen et al. (2013) Eigen, D., Ranzato, M., and Sutskever, I.  Learning factored representations in a deep mixture of experts.  _arXiv preprint arXiv:1312.4314_ , 2013.
  * Fortunato et al. (2018) Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., et al.  Noisy networks for exploration.  In _ICLR_ , 2018.
  * Girshick et al. (2014) Girshick, R., Donahue, J., Darrell, T., and Malik, J.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In _CVPR_ , 2014.
  * Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y.  _Deep learning_ , volume 1.  MIT press Cambridge, 2016.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * He et al. (2017) He, K., Gkioxari, G., Dollár, P., and Girshick, R.  Mask r-cnn.  In _ICCV_ , 2017.
  * He et al. (2019) He, K., Girshick, R., and Dollár, P.  Rethinking imagenet pre-training.  In _ICCV_ , 2019.
  * He et al. (2020) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R.  Momentum contrast for unsupervised visual representation learning.  In _CVPR_ , 2020.
  * Helber et al. (2019) Helber, P., Bischke, B., Dengel, A., and Borth, D.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _JSTARS_ , 12(7):2217–2226, 2019.
  * Hessel et al. (2018) Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D.  Rainbow: Combining improvements in deep reinforcement learning.  In _AAAI_ , 2018.
  * Kingma & Ba (2015) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  In _ICLR_ , 2015.
  * Kornblith et al. (2019) Kornblith, S., Shlens, J., and Le, Q. V.  Do better imagenet models transfer better?  In _CVPR_ , 2019.
  * Krause et al. (2013) Krause, J., Stark, M., Deng, J., and Fei-Fei, L.  3d Object Representations for Fine-Grained Categorization.  In _ICCV Workshop_ , 2013.
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al.  Learning multiple layers of features from tiny images.  2009\.
  * Le et al. (2012) Le, V., Brandt, J., Lin, Z., Bourdev, L., and Huang, T. S.  Interactive facial feature localization.  In _ECCV_ , 2012.
  * LeCun et al. (2015) LeCun, Y., Bengio, Y., and Hinton, G.  Deep learning.  _Nature_ , 521(7553):436–444, 2015.
  * Li et al. (2019) Li, X., Xiong, H., Wang, H., Rao, Y., Liu, L., Chen, Z., and Huan, J.  Delta: Deep learning transfer using feature map with attention for convolutional networks.  In _ICLR_ , 2019.
  * Lin et al. (2014) Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.  Microsoft coco: Common objects in context.  In _ECCV_ , 2014.
  * Liu et al. (2019) Liu, I.-J., Peng, J., and Schwing, A. G.  Knowledge flow: Improve upon your teachers.  In _ICLR_ , 2019.
  * Long et al. (2015) Long, M., Cao, Y., Wang, J., and Jordan, M.  Learning transferable features with deep adaptation networks.  In _ICML_ , 2015.
  * Long et al. (2018) Long, M., Cao, Y., Cao, Z., Wang, J., and Jordan, M. I.  Transferable representation learning with deep adaptation networks.  _TPAMI_ , 41(12):3071–3085, 2018.
  * Maji et al. (2013) Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A.  Fine-grained visual classification of aircraft.  _Technical report_ , 2013.
  * Masoudnia & Ebrahimpour (2014) Masoudnia, S. and Ebrahimpour, R.  Mixture of experts: a literature survey.  _Artificial Intelligence Review_ , 42(2):275–293, 2014.
  * Nguyen et al. (2020) Nguyen, C., Hassner, T., Seeger, M., and Archambeau, C.  Leep: A new measure to evaluate transferability of learned representations.  In _ICML_ , 2020.
  * Oquab et al. (2014) Oquab, M., Bottou, L., Laptev, I., and Sivic, J.  Learning and transferring mid-level image representations using convolutional neural networks.  In _CVPR_ , 2014.
  * Pan & Yang (2009) Pan, S. J. and Yang, Q.  A survey on transfer learning.  _TKDE_ , 22(10):1345–1359, 2009.
  * Paszke et al. (2017) Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A.  Automatic differentiation in pytorch.  2017\.
  * Quattoni & Torralba (2009) Quattoni, A. and Torralba, A.  Recognizing indoor scenes.  In _CVPR_ , 2009.
  * Raghu et al. (2019) Raghu, M., Zhang, C., Kleinberg, J., and Bengio, S.  Transfusion: Understanding transfer learning for medical imaging.  In _NeurIPS_ , 2019.
  * Ramalho et al. (2019) Ramalho, T., Sousbie, T., and Peluchetti, S.  An empirical study of pretrained representations for few-shot classification.  _arXiv preprint arXiv:1910.01319_ , 2019.
  * Ren et al. (2016) Ren, S., Cao, X., Wei, Y., and Sun, J.  Face alignment via regressing local binary features.  _TIP_ , 25(3):1233–1245, 2016.
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.  Imagenet large scale visual recognition challenge.  _IJCV_ , 115(3):211–252, 2015.
  * Rusu et al. (2016) Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_ , 2016.
  * Sagonas et al. (2013) Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., and Pantic, M.  300 faces in-the-wild challenge: The first facial landmark localization challenge.  In _ICCV Workshops_ , 2013.
  * Shazeer et al. (2017) Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  _arXiv preprint arXiv:1701.06538_ , 2017.
  * Sun et al. (2019) Sun, K., Zhao, Y., Jiang, B., Cheng, T., Xiao, B., Liu, D., Mu, Y., Wang, X., Liu, W., and Wang, J.  High-resolution representations for labeling pixels and regions.  _arXiv preprint arXiv:1904.04514_ , 2019.
  * Tan et al. (2018) Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., and Liu, C.  A survey on deep transfer learning.  In _ICANN_ , 2018.
  * Tran et al. (2019) Tran, A. T., Nguyen, C. V., and Hassner, T.  Transferability and hardness of supervised classification tasks.  In _ICCV_ , 2019.
  * van Hasselt et al. (2019) van Hasselt, H., Hessel, M., and Aslanides, J.  When to use parametric models in reinforcement learning?  In _NeurIPS_ , 2019.
  * Wu et al. (2018) Wu, W., Qian, C., Yang, S., Wang, Q., Cai, Y., and Zhou, Q.  Look at boundary: A boundary-aware face alignment algorithm.  In _CVPR_ , 2018.
  * Xuhong et al. (2018) Xuhong, L., Grandvalet, Y., and Davoine, F.  Explicit inductive bias for transfer learning with convolutional networks.  In _ICML_ , 2018.
  * Yang et al. (2016) Yang, S., Luo, P., Loy, C.-C., and Tang, X.  Wider face: A face detection benchmark.  In _CVPR_ , 2016.
  * Yosinski et al. (2014) Yosinski, J., Clune, J., Bengio, Y., and Lipson, H.  How transferable are features in deep neural networks?  In _NeurIPS_ , 2014.
  * You et al. (2020) You, K., Kou, Z., Long, M., and Wang, J.  Co-tuning for transfer learning.  In _NeurIPS_ , 2020.
  * You et al. (2021) You, K., Liu, Y., Wang, J., and Long, M.  Logme: Practical assessment of pre-trained models for transfer learning, 2021.
  * Zhai et al. (2019) Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., et al.  A large-scale study of representation learning with the visual task adaptation benchmark.  _arXiv preprint arXiv:1910.04867_ , 2019.
  * Zhong et al. (2020) Zhong, J., Wang, X., Kou, Z., Wang, J., and Long, M.  Bi-tuning of pre-trained representations.  _arXiv preprint arXiv:2011.06182_ , 2020.
  * Zhu & Ramanan (2012) Zhu, X. and Ramanan, D.  Face detection, pose estimation, and landmark localization in the wild.  In _CVPR_ , 2012.

##  Appendix A Experiment Details

In this section, we supplement more experiment details for reproducing the
experiment results.

###  A.1 Implementation Details in Reinforcement Learning

In the experiments of transfer learning in reinforcement learning, we use the
network architecture in Data-Efficient Rainbow, which consists of $2$
convolutional layers: $32$ filters of size $5\times 5$ with the stride of $5$
and $64$ filters of size $5\times 5$ with the stride of $5$, followed by a
flatten layer and $2$ noisy linear layer with the hidden size of $256$. We
train two models with this architecture on the Seaquest and Riverraid games,
respectively, as our source models. We train for $1\times 10^{5}$ steps with
the same hyper-parameters as those in Data-Efficient Rainbow. We then perform
Zoo-Tuning on the first $2$ layers for each target task. To be specific, we
replace the first $2$ layers in the original architecture with adaptive
aggregation (AdaAgg) layers, which learn to adaptively aggregate the
parameters of these layers in source models to form the corresponding layers
of the target model. The last $2$ layers of the architecture are not modified,
which are randomly initialized and trained together with the previous AdaAgg
layers. During the test stage, we calculate an average of $10$ episodes as the
final result, and report the mean and variance of results with 5 different
random seeds.

###  A.2 Implementation Details in Image Classification and Facial Landmark
Detection

In the experiments of classification and facial landmark detection, we use $5$
pretrained models trained from various datasets and tasks as stated in the
main text. Each pretrained model consists of a ResNet-50 backbone and several
task-specific head layers. For each target task, we perform Zoo-Tuning on the
ResNet-50 backbone. We modify each convolutional layer in the backbone with an
AdaAgg layer to aggregate knowledge from all $5$ pretrained models. We do not
perform channel alignment on top $1\times 1$ convolutional layers in the
backbone, which preserves the accuracy and further reduces the parameters and
computational complexity of our method. We keep other layers such as pooling
layers, batch normalization layers, and activation layers unchanged. For each
batch normalization layer in the backbone, we initialize its learnable linear
transformation parameters with the average of the corresponding parameters in
$5$ pretrained models, which gives the layer a smooth warm-up.

Based on the modified backbone with AdaAgg layers, we add a task-specific head
for each task. The task-specific head is randomly initialized and trained
together with the AdaAgg backbone. For classification tasks, the head is
composed of a linear layer. For facial landmark detection, the head is
composed of three $4\times 4$ transpose convolution layers with the stride of
$2$, and a $1\times 1$ convolution layer. For each target task, we run
experiments for $3$ times and report the average results of the $3$ runs.

###  A.3 Computing Infrastructure

For reinforcement learning experiments, we implement all the methods based on
PyTorch 1.5.0, torchvison 0.6.0, and CUDA 10.2 libraries. For the other two
computer vision experiments, we use PyTorch 1.1.0, torchvison 0.3.0, and CUDA
10.0 libraries for the classification tasks. We use PyTorch 1.0.0, torchvison
0.2.1, and CUDA 9.0 libraries for facial landmark detection tasks. We use a
machine with 32 CPUs, 256 GB memory, and one NVIDIA TITAN X GPU.

Figure 6: Visualization of gating values in each layer of source models. We
use the $5$ vision pretrained models as source models and use the facial
landmark detection dataset COFW as the target task. From the left to right is
from the bottom layer to the top layer. The darker color means a higher gating
value.

##  Appendix B Experiment Results

In this section, we supplement more experiment results.

###  B.1 More Transfer Learning Baselines

In this section, we design two more transfer learning methods for transferring
from a zoo of models. The first method, called Concat-Ensemble-Distill (CED),
performs a knowledge distillation from model ensembles but uses the
concatenation of their features instead of direct averaging their predictions.
Specifically, it adopts a network architecture consisting of a shared backbone
with multi-heads to distill the high-level features from each source model in
the zoo. Then it uses the concatenation of all these features from multi-heads
for the target task. The second method, called Feature Extraction (FE),
extracts the features (specifically, features of the penultimate layers) from
all the source models in the model zoo and concatenates them to serve as the
new features for the target data. It then trains a target classifier with the
concatenated features. The two methods are extensions of feature-level
transfer learning methods to the model zoo transfer setting. We perform two
variants for each method based on whether the source models are firstly fine-
tuned with target data (+Finetuned). Note that fine-tuning all source models
causes high computation costs.

Table 6: Comparison with the proposed baselines. Model | Cifar | COCO | Indoors
---|---|---|---
CED | $81.84$ | $82.19$ | $73.15$
CED+Finetuned | $81.91$ | $82.01$ | $74.57$
FE | $80.01$ | $56.52$ | $66.15$
FE+Finetuned | $83.00$ | $83.34$ | $74.66$
Zoo-Tuning | $\mathbf{83.77}$ | $\mathbf{84.91}$ | $\mathbf{75.39}$

As shown in Table 6, Zoo-Tuning still outperforms these baselines. Compared
with them, Zoo-Tuning has two main advantages. (1) It adaptively leverages the
knowledge in pretrained models to facilitate target learning tasks instead of
using the fixed features. (2) These baselines only consider high-level
features while Zoo-Tuning enables aggregation in all layers, leading to a
deeper transfer of knowledge.

###  B.2 Visualization of Gating Values

We also visualize the gating values of each layer in each pretrained model
learned by Zoo-Tuning on the facial landmark detection benchmark COFW, as
shown in Figure 6.

Different layers in different source models have diverse gating values, which
indicates that different layers and source pretrained models have different
underlying influences on the target task. Note that this task is more
complicated than classification, as the relationship between source and target
tasks is unclear, and there is little prior knowledge. We also find that the
single model performance cannot necessarily determine its importance when
transferring from multiple models under this situation. For example, though
fine-tuning from the MoCo pretrained model performs better than other models,
it does not have dominant gating values in Zoo-Tuning. This makes transferring
from a zoo of models difficult, especially on more complex downstream tasks,
and shows the advantage of using Zoo-Tuning, which learns to adaptively
aggregate the source models without handcrafted designs.

Figure 7: The normalized mean error of the fine-tuning baseline and Zoo-Tuning
on WFLW dataset.

###  B.3 Training Curves

We plot the curves of the normalized mean error (NME) on target test data
while training with the fine-tuning baseline and Zoo-Tuning methods on the
WFLW dataset in Figure 7. We can observe that Zoo-Tuning shows a similar
convergence speed compared with fine-tuning from one model but consistently
achieves better generalization results.


SI-HEP-2022-25, P3H-22-093

#  $H\to\gamma\gamma$ to all orders in $\alpha_{s}$ in the large-$\beta_{0}$
limit of QCD

D. Boito  Instituto de Física de São Carlos, Universidade de São Paulo, CP
369, 13560-970, São Carlos, SP, Brazil G. das Neves  Instituto de Física de
São Carlos, Universidade de São Paulo, CP 369, 13560-970, São Carlos, SP,
Brazil J. Piclum  Theoretische Physik 1,Center for Particle Physics Siegen
(CPPS),
Universität Siegen, Walter-Flex-Str. 3, 57068 Siegen, Germany

###### Abstract

In this note, we present the result for the QCD corrections to the decay of
the Higgs boson into two photons in the large-$\beta_{0}$ limit of QCD, in the
heavy-top limit. From our results, one can easily read off the exact
leading-$n_{f}$ QCD contributions in analytic form to all orders in the strong
coupling, $\alpha_{s}$, where $n_{f}$ is the number of massless quarks, and
identify the leading renormalon singularities. We give explicit results for
the leading-$n_{f}$ coefficients at 5 and 6 loops and use the
large-$\beta_{0}$ result to speculate about the size of yet unknown (but
small) higher-order contributions to the QCD series.

##  1 Introduction

The Higgs decay into two photons played a central role, ten years ago, in the
discovery of the Higgs boson at the Large Hadron Collider (LHC). This channel
remains crucial for precision studies of Higgs properties at the LHC as well
as at future colliders. Given the expected increase in precision on the
experimental side, the theoretical calculation of Higgs decays within the
Standard Model (SM) must be performed at higher orders in perturbation theory.

Because the photon is massless, this decay is a loop-induced process at
leading order (LO) within the SM. Its amplitude can be decomposed into a
bosonic contribution, stemming from the $W$ boson, and fermionic
contributions. Here we are concerned with the fermionic contribution arising
from the top quark, which dominates over the subleading $b$-quark and
$\tau$-lepton fermion loops (contributions from even lighter fermions are
irrelevant).

The decay amplitude has been known at LO for many years [1, 2, 3, 4]. The
next-to-leading order (NLO) result was first obtained as an expansion in
$\tau_{t}=\frac{m_{H}^{2}}{4m_{t}^{2}}\approx 0.13$, with $m_{H}$ and $m_{t}$
being the Higgs and the top-quark masses, respectively, but later the full
$m_{t}$ dependence of the NLO result was obtained[5, 6, 7, 8, 9, 10, 11, 12].
The NNLO QCD contribution, which is a three-loop calculation, was obtained
about ten years ago, again as an asymptotic expansion in the top mass;
numerical results with full top-quark mass dependence were obtained only much
more recently[13, 14]. The NNNLO (four-loop), $\mathcal{O}(\alpha_{s}^{3})$,
corrections were first partially computed in Ref. [15] and later completed in
Ref. [16]. Partial results for the N4LO contributions where both photons
couple to a massive top-quark loop can also be found in Ref. [15].

In this work we present, for the first time, the result for the QCD
corrections in the large-$\beta_{0}$ limit of QCD within the heavy-top limit.
From our results, the exact leading-$n_{f}$ QCD contributions, with $n_{f}$
being the number of active (massless) quark flavors, can be directly read off
in analytic form to all orders in the strong coupling, $\alpha_{s}$. In our
calculation, we are concerned with non-singlet diagrams only, i.e. diagrams in
which the Higgs boson and the two photons in the final state couple to the
same top-quark loop. Contributions from the singlet diagrams are subleading in
the large-$\beta_{0}$ limit.111These contributions are not small in full QCD
and we comment on singlet-diagram contributions in Sec 3.

In the large-$\beta_{0}$ limit of QCD [17], one considers first the limit of
large $n_{f}$, keeping $\alpha_{s}n_{f}\sim\mathcal{O}(1)$. The fermionic
corrections to the gluon propagator acquire a special character since they
count as $\mathcal{O}(1)$, and a dressed gluon propagator, known to all orders
in $\alpha_{s}$, can be obtained. When no external gluons are present, such as
in the case of this work, the leading-$n_{f}$ contributions to a given process
can be obtained, to all orders, by replacing the gluon propagator in the
calculation of the $\mathcal{O}(\alpha_{s})$ QCD correction by the Borel
transform of the dressed gluon propagator. The large-$\beta_{0}$ limit
consists then in the replacement of the fermionic contribution to the QCD
$\beta$ function, $\beta_{0f}$, proportional to $n_{f}$, by the complete one-
loop QCD beta-function coefficient, $\beta_{0}$. This procedure, known as
naive non-abelianization [18, 19], effectively resums a set of non-Abelian
graphs related to the running of the coupling, thereby restoring the non-
Abelian character of the results.

Our calculation is performed with a modified version of the publicly available
package MATAD [20], which is written in FORM [21]. Working with the Borel-
transformed dressed gluon propagator in the large-$\beta_{0}$ limit amounts,
essentially, to a perturbative calculation with an analytically regularized
gluon propagator. This requires that the exponent of the squared momentum in
the gluon propagator be generalized to a real number [17]. This was
implemented through modifications of the original source code of MATAD, since
this generalization is not supported by the standard version.

The importance of calculations in the large-$\beta_{0}$ limit should not be
overstated. Albeit realistic, it is a toy model for higher orders, but one
that can lead to insights about the physics of renormalons, non-perturbative
corrections, and even, in some cases, allow for estimates of unknown higher-
order contributions. The fact that the leading-$n_{f}$ terms are exactly
obtained can also be useful as an independent cross-check of future
calculations.

We present, in analytical form, the Borel transform of the decay amplitude in
the large-$\beta_{0}$ limit, for an infinitely heavy top quark. From this
result, it is simple to extract in analytical form the exact leading-$n_{f}$
contributions to the QCD series to all orders in $\alpha_{s}$, with $n_{f}$
being the number of massless quarks. We reproduce the known leading-$n_{f}$
results up to 4 loops, and give for the first time the explicit expression at
5 and 6 loops. The large-$\beta_{0}$ result allows for a discussion of the
different renormalon singularities, and we show that the leading UV renormalon
is very prominent, leading to a sign alternating perturbative series (for not-
so-large renormalization scale choices). The large-$\beta_{0}$ series
reproduces the approximate magnitude of the known QCD perturbative
coefficients, although with incorrect signs. This allows for a speculation
about the size of yet unknown contributions to the QCD series.

This work is organized as follows. In Sec. 2 we set up the theoretical
framework and the notation. Our main results are given in Sec. 3 together with
the discussion of some of their potential implications. In Sec. 4 we present
our conclusions. We relegate to App. A a brief description of the
modifications to MATAD that are required to work in the large-$\beta_{0}$
limit.

##  2 Theoretical framework

The decay width of the process $H\rightarrow\gamma\gamma$ starts at one-loop
order in the SM and can be written in terms of bosonic and fermionic
contributions as

$\Gamma(H\rightarrow\gamma\gamma)=\frac{M_{H}^{3}}{64\pi}\mid
A_{W}(\tau_{W})+\sum_{f}A_{f}(\tau_{f})\mid^{2},$ (1)

where the first amplitude, $A_{W}$, is due to $W$ boson diagrams, while
$A_{f}$ stems from the decay mediated by charged fermions with mass $m_{f}$
[22]. In the above expression, $\tau_{f}=\frac{M_{H}^{2}}{4m_{f}^{2}}$ and
$\tau_{W}=\frac{M_{H}^{2}}{4M_{W}^{2}}$, where $m_{W}$ is the $W$-boson mass.
The fermionic contributions are strongly dominated by the top-quark loop, with
small subleading components due to the bottom quark and the tau lepton. Here,
we focus exclusively on the top-quark contribution, $A_{t}$. We work in an
expansion in $\tau_{t}$ in the limit $m_{t}\to\infty$ (the heavy top limit),
which corresponds to the leading term in this expansion. To estimate effects
due to higher orders in perturbation theory, this limit should be sufficient
since the first $\tau_{t}$ correction is of about 10%.

We write the amplitude $A_{t}$ as an expansion in powers of the strong
coupling, $\alpha_{s}$, as

$A_{t}=\hat{A}_{t}\sum_{n=0}^{\infty}A_{t}^{(n)}\bigg{(}\frac{\alpha_{s}}{\pi}\bigg{)}^{n},$
(2)

where

$\hat{A}_{t}=N_{c}\frac{2Q_{t}^{2}\alpha}{3\pi v},$ (3)

with $\alpha$ being the fine-structure constant, $Q_{t}$ is the
electromagnetic top charge, $N_{c}=3$ is the number of colors, and
$v=1/\sqrt{\sqrt{2}G_{F}}$ is the vacuum expectation value of the Higgs field
with the Fermi constant as input. With this convention (which is the same as
Ref. [15], for example), the leading order contribution to the amplitude
$A_{t}$, Eq. (2), reads, after the expansion in $\tau_{t}$,

$A_{t}^{(0)}=1+\frac{7}{30}\tau_{t}+\frac{2}{21}\tau_{t}^{2}+\frac{26}{525}\tau_{t}^{3}+\frac{512}{17325}\tau_{t}^{4}+\frac{1216}{63063}\tau_{t}^{5}+\mathcal{O}(\tau_{t}^{6}).$
(4)

Using expansion by regions (see for example Ref. [23]), the leading term in
this expansion, corresponding to $m_{t}\to\infty$, can be obtained from the
computation of the loop integrals in the hard region, which leads to so-called
massive tadpole integrals.

The NLO term due to the top loop is obtained through the exchange of a virtual
gluon between the internal top propagators [22]. The topology of the main
diagrams for the NLO process is shown in Fig. 1. There are a total of 12
diagrams at the two-loop level, and the final result for the amplitude in the
infinitely heavy top-quark limit reads

$A_{t}^{(1)}=-\frac{3}{4}C_{F}+\mathcal{O}(\tau_{t}),$ (5)

with $C_{F}=(N_{c}^{2}-1)/(2N_{c})=4/3$. Here, we have reproduced the NLO
calculation using qgraf[24] and MATAD [20], since this forms the basis for the
computation in the large-$\beta_{0}$ limit, as we discuss further below.
Analytical expressions for higher order corrections, $A_{t}^{(2)}$ and
$A_{t}^{(3)}$, up to four loops, can be found in Refs. [15, 16].

We turn now to the setup of the large-$\beta_{0}$ limit calculation. In the
large-$\beta_{0}$ limit we deal with the perturbative series to all orders in
$\alpha_{s}$. As is well known, in QCD, series of this type have coefficients
that diverge factorially and the perturbative series is, at best, asymptotic.
In this context it is convenient to work with the Borel transform of the
series, which suppresses the factorial divergence of the coefficients and can
have a finite radius of convergence. For the perturbative expansion of a
quantity $R$ starting at $\mathcal{O}(\alpha_{s})$ (without any loss of
generality),

$R\sim\sum_{n=0}^{\infty}r_{n}\alpha_{s}^{n+1},$ (6)

we define its Borel transform in the following way [17]

$B[R](t)=\sum_{n=0}^{\infty}r_{n}\frac{t^{n}}{n!}.$ (7)

The Borel transform is the inverse Laplace transform of the original series.
The procedure can be inverted and a ‘true value’ of the asymptotic series can
be assigned (in the Borel sense) through the Laplace transform as

$R(\alpha_{s})=\int_{0}^{\infty}{\rm d}t\,e^{-t/\alpha_{s}}B[R](t),$ (8)

provided the integral exists. In the Borel $t$-plane, singularities known as
renormalons appear. They arise from infra-red (IR) and ultra-violet (UV)
regions in the loop sub-graphs. The IR renormalons are particularly important
here, since they appear on the right-hand side and obstruct the integration in
Eq. (8). Circumventing these singularities generates an imaginary ambiguity in
the Borel integral. This ambiguity is related to non-perturbative terms from
condensate matrix elements in the operator product expansion (OPE). Here,
since the typical scale of the problem is quite high compared with
$\Lambda_{\rm QCD}$, these OPE corrections — and accordingly the imaginary
ambiguities of IR renormalons — are tiny and can, for all practical purposes,
be neglected.

Using the definition in Eq. (7), the dressed gluon propagator with a four-
momentum $k$ obtained after resumming the massless-quark bubble-loop
corrections in Borel space reads [17]

$B[\alpha_{s}G_{\mu\nu}](u)=\frac{(-i)}{k^{2}}\bigg{(}g_{\mu\nu}-\frac{k_{\mu}k_{\nu}}{k^{2}}\bigg{)}\bigg{(}-\frac{\mu^{2}}{k^{2}}e^{-C}\bigg{)}^{u}+(-i)\xi\frac{k_{\mu}k_{\nu}}{k^{4}},$
(9)

where we introduced the variable $u$, defined as

$u=-\beta_{0,f}t.$ (10)

(We recall that $\beta_{0f}$ are the fermionic contributions to $\beta_{0}$.)
In Eq. (9), $\mu^{2}$ is the renormalization scale and $\xi$ is the gauge
parameter. The constant $C$ sets the renormalization scheme: in the
$\overline{\text{MS}}$ scheme, $C=-5/3$; whereas in the MS scheme, for
instance, $C=-5/3+\gamma_{E}-\ln 4\pi$. Notice that, in the definition of the
transformation, Eq. (9), we multiplied the gluon propagator by $\alpha_{s}$;
thus, the lowest order term in the $u$ expansion corresponds already to the
NLO QCD correction. In Eq. (9), the factor $(-\mu^{2}e^{-C})^{u}$ ensures
scheme and scale invariance of the Borel integral result, Eq. (8). We use the
following definition for the QCD $\beta$ function and its coefficients:

$\beta(\alpha_{s})=\beta_{0}\alpha_{s}^{2}+\beta_{1}\alpha_{s}^{3}+\cdots=\mu^{2}\frac{{{\rm
d}\,\alpha_{s}}}{{\rm d}\mu^{2}},$ (11)

with

$\beta_{0}=-\frac{11\,C_{A}}{12\pi}+\frac{n_{f}T_{F}}{3\pi},$ (12)

where $C_{A}=N_{c}=3$, $T_{F}=1/2$, and $n_{f}$ is the number of massless
quark flavors. The second contribution on the right-hand side of Eq. (12) is
the fermionic contribution, $\beta_{0,f}=n_{f}/(6\pi)$.

Working in Landau gauge, with $\xi=0$, the Borel transform of the dressed
propagator is essentially the original propagator with a modification in the
exponent of the denominator momentum, $k^{2}\rightarrow(k^{2})^{1+u}$ — which
amounts to working with an analytically regularized gluon propagator. In our
case, using Eq. (9) as the gluon propagator in the NLO QCD calculation is
sufficient to generate the exact leading-$n_{f}$ terms to all orders in
$\alpha_{s}$. This is depicted in Fig. 1, where the main topologies involved
in the diagrammatic calculation are shown and the dashed line represents the
resummed, dressed, gluon propagator. The calculation of the diagrams of Fig. 1
yields the main result of this paper. We implemented this calculation in MATAD
with suitable modifications to the source code to account, essentially, for
the new gluon propagator. Some technical details of this implementation can be
found in Appendix A. The source is also publicly available on GitHub.222The
source files can be found in the following link
https://github.com/g-neves/hgg-large-
beta0/tree/master/c_hgagalargeb0/hgagalargeb0 In the next section we discuss
the final results.

Figure 1: Sample of diagrams for the calculation of the leading-$n_{f}$ terms
for the decay $H\rightarrow\gamma\gamma$. The internal dashed lines represent
the resummed gluon propagator, Eq (9).

##  3 Results

The Borel transform of the amplitude $A_{t}$ in the large-$\beta_{0}$ limit,
for $m_{t}\to\infty$, denoted $B[A_{t,\text{large-}\beta_{0}}]$, in closed
form, reads

$B[A_{t,\text{large-}\beta_{0}}]=\bigg{(}\frac{Q_{t}^{2}\alpha}{v}\bigg{)}\bigg{(}\frac{2N_{c}\,C_{F}}{4\pi^{2}}\bigg{)}\left(\frac{\mu^{2}}{\overline{m}_{t}^{2}}\right)^{u}\frac{e^{5u/3}(u^{2}-1)\Gamma(1-u)\Gamma(1+u)^{3}}{(1+2u)\Gamma(1+2u)},$
(13)

where $\overline{m}_{t}\equiv\overline{m}_{t}(\mu)$ is the $\overline{\rm MS}$
top-quark mass. This expression exhibits the renormalon singularities arising
from UV and IR regions of loop sub-graphs, which are encoded in the $\Gamma$
functions. The UV renormalons, which are located at negative integer values of
the variable $u$ are all double poles, with the sole exception of the leading
UV pole at $u=-1$, which is simple. This happens due to a partial cancellation
with the $(u^{2}-1)/\Gamma(1+2u)$ term. Note also that there is no singularity
at $u=-1/2$ and the function is regular at this point, as expected. The IR
renormalons, on the other hand, are all simple poles with the leading
singularity at $u=2$ being a quartic IR sensitivity, connected with dimension
four corrections in the Operator Product Expansion.

Regarding the dominant renormalon singularities another observation is that
the residue of the leading UV pole and that of the leading IR pole are of the
same order. Their ratio being of order 1 implies that the UV renormalon, being
the closest to the origin, should take over rather earlier and lead to a sign
alternating series. This, however, depends on the renormalization scale that
is chosen: larger values of $\mu$ enhance the IR poles and suppress the UV
contributions, which postpones the dominance of the UV renormalon, reducing
its residue, and consequently delays the sign alternation.

From the Taylor expansion of Eq. (13) one recovers the perturbative expansion
in $\alpha_{s}$ in the large-$\beta_{0}$ limit. Of course, the leading term
must correspond to the exact NLO QCD result, and it does lead to $A_{t}^{(1)}$
of Eq. (5), as expected. Then, further expanding in $u$ and performing the
substitution $u=-\beta_{0,f}t$, one can re-obtain the leading-$n_{f}$
contributions of the exactly known QCD corrections in perturbation theory. The
coefficient of order $\alpha_{s}^{n}$ has a leading $n_{f}$ term of order
$n_{f}^{n-1}$, which we denote by $A_{t,{n_{f}^{n-1}}}^{(n)}$. At 3-, 4-, and
5-loops, we get

$\displaystyle
A_{t,n_{f}}^{(2)}=-C_{F}T_{F}n_{f}\left(\frac{1}{12}-\frac{\ell_{\mu}}{4}\right),$
(14) $\displaystyle
A_{t,n_{f}^{2}}^{(3)}=-C_{F}T_{F}^{2}n_{f}^{2}\left(\frac{19}{108}-\frac{\ell_{\mu}}{18}+\frac{\ell_{\mu}^{2}}{12}\right),$
(15) $\displaystyle
A_{t,n_{f}^{3}}^{(4)}=-C_{F}T_{F}^{3}n_{f}^{3}\left(\frac{487}{972}-\frac{\zeta_{3}}{3}-\frac{19}{108}\ell_{\mu}+\frac{\ell_{\mu}^{2}}{36}-\frac{\ell_{\mu}^{3}}{36}\right),$
(16)

where $\ell_{\mu}\equiv\ln\left(\frac{\mu^{2}}{\overline{m}_{t}^{2}}\right)$
and $\zeta_{i}$ is the Riemann Zeta Function evaluated at $i$. These results
are in agreement with Refs. [15, 14].

A new result of this work is the leading $n_{f}$-terms to all orders in
perturbation theory. As an example, the N5LO leading-$n_{f}$ coefficient,
$A_{t,n_{f}^{4}}^{(5)}$, reads

$\displaystyle
A_{t,n_{f}^{4}}^{(5)}=-C_{F}T_{F}^{4}n_{f}^{4}\bigg{[}\frac{9613}{8748}-\frac{\pi^{4}}{135}-\frac{4\zeta_{3}}{27}+\ell_{\mu}\left(\frac{4\zeta_{3}}{9}-\frac{487}{729}\right)+\frac{19}{162}\ell_{\mu}^{2}-\frac{\ell_{\mu}^{3}}{81}+\frac{\ell_{\mu}^{4}}{108}\bigg{]},$
(17)

while the N6LO leading-$n_{f}$ coefficient, $A_{t,n_{f}^{5}}^{(6)}$ is

$\displaystyle A_{t,n_{f}^{5}}^{(6)}=-C_{F}T_{F}^{5}n_{f}^{5}\bigg{[}$
$\displaystyle\frac{307765}{78732}-\frac{\pi^{4}}{243}-\frac{190}{243}\zeta_{3}-\frac{20}{9}\zeta_{5}+\ell_{\mu}\bigg{(}-\frac{48065}{26244}+\frac{\pi^{4}}{81}+\frac{20}{81}\zeta_{3}\bigg{)}$
$\displaystyle+\ell_{\mu}^{2}\bigg{(}\frac{2435}{4374}-\frac{10}{27}\zeta_{3}\bigg{)}-\frac{95}{1458}\ell_{\mu}^{3}+\frac{5}{972}\ell_{\mu}^{4}-\frac{1}{324}\ell_{\mu}^{5}\bigg{]}.$
(18)

The higher-order coefficients can be easily generated from Eq. (13) following
the procedure outlined above.

Let us now discuss the series in the large-$\beta_{0}$ limit, after naive non-
abelianization, with the replacement $\beta_{0f}\to\beta_{0}$. This
transformation preserves the leading $n_{f}$ coefficients
$A_{t,{n_{f}^{n-1}}}^{(n)}$ and generates approximate results for all sub-
leading powers of $n_{f}$. The perturbative series thus obtained is, at best,
a good qualitative approximation to the full QCD results. With this caveat in
mind, we show in Fig. 2a the perturbative series for four different choices of
the renormalization scale $\mu$, order by order in perturbation theory. The
horizontal line is the Borel sum of the series, given by the integral of Eq.
(8).333The series, strictly speaking, is not Borel summable due to the IR
renormalons, which obstruct the integration. A prescription to circumvent them
must be chosen, which generates an imaginary ambiguity which scales as
$e^{-p/\alpha_{s}}$ (where $p>0$), related to the size of non-perturbative
corrections. However, these contributions, at the Higgs mass scale, are very
small, which leads to only a tiny ambiguity which is not visible in our plots
and can be ignored here. In Fig. 2b we display the scale dependence of the
result, order by order, varying the renormalization scale $\mu$ in the
logarithms $\ell_{\mu}$ (see Eqs. (14)-(3)) and in
$\alpha_{s}^{(n_{f}=5)}(\mu)$. The running of $\alpha_{s}^{(n_{f}=5)}(\mu)$ is
performed at one loop, for consistency with the large-$\beta_{0}$ limit, using
as input $\alpha_{s}^{(n_{f}=5)}(m_{Z})=0.1179$ while for the top mass the
value of $\overline{m}_{t}(\overline{m}_{\rm t})=163.643$ GeV [25] is kept
fixed, since the $\overline{\rm MS}$ quark-mass running is of order
$1/\beta_{0}$ itself, which would then generate $1/\beta_{0}^{2}$ terms not
included in the perturbative coefficients [26, 27]. The stabilization with
respect to scale variations is clearly observed when the order is increased,
starting from N4LO, or $\mathcal{O}(\alpha_{s}^{4})$, with the N5LO result
being extremely stable and in excellent agreement with the Borel sum.

(a)

(b)

Figure 2: (a) Perturbative series, Eq. (2), in the large-$\beta_{0}$ limit for
four different values of the renormalization scale $\mu$, order by order. The
Borel sum of the series (see text) is shown as the horizontal dashed line. (b)
Perturbative series as a function of the renormalization scale up to N5LO (or
$\mathcal{O}(\alpha_{s}^{5})$). The scale variation range $50~{}{\rm
GeV}\leq\mu\leq 350~{}{\rm GeV}$ contains the interval $m_{H}/2\leq\mu\leq
2m_{t}$.

Before comparing with the QCD calculation, an observation regarding the
singlet diagrams is in order. In Ref. [15], it is shown that with the scale
choice $\mu=\overline{m}_{t}(\overline{m}_{t})$ (which resums the logarithms
$\ell_{\mu}$) at the 3-loop level the singlet diagrams are approximately of
the same magnitude as the non-singlet diagrams, while at a lower scale,
$\mu=m_{H}$ the singlet contribution is significantly enhanced and is
approximately three times larger than the non-singlet contribution. It is
argued that one might expect the same to happen at 4- and 5-loop orders. Thus,
if the large-$\beta_{0}$ limit represents well quantitatively the non-singlet
diagrams in full QCD, at higher values of $\mu$, the difference between the
result in QCD and in large-$\beta_{0}$ should decrease. Below, the results we
discuss are for $\mu=\overline{m}_{t}(\overline{m}_{t})$, which reduces the
singlet contributions.

After naive nonabelianization, our result for $A_{t,\text{large-}\beta_{0}}$
at the scale $\mu=\overline{m}_{t}(\overline{m}_{t})$ reads
($a_{s}\equiv\alpha_{s}(\overline{m}_{t}(\overline{m}_{t}))/\pi$)

$\frac{A_{t,\text{large-}\beta_{0}}}{\hat{A}_{t}}=1-a_{s}+0.6389\,a_{s}^{2}-7.755\,a_{s}^{3}+25.43\,a_{s}^{4}-290.4\,a_{s}^{5}+2212\,a_{s}^{6}+\cdots$
(19)

This result shows that in the large-$\beta_{0}$ limit the pattern of sign
alternation is already present, systematically, starting at
$\mathcal{O}(\alpha_{s}^{2})$. This is a consequence of the dominant behavior
of the leading UV singularity. (This pattern disappears if even larger values
of $\mu$ are used. For example, for $\mu=280$ GeV the coefficients have fixed
signs, as can be seen in Fig. 2a.) In the same conventions, and again for
$\mu=\overline{m}_{t}(\overline{m}_{t})$ and
$a_{s}=\alpha_{s}^{(n_{f}=5)}(\overline{m}_{t}(\overline{m}_{t}))/\pi$, using
$m_{H}=125.25$ GeV, from the results of Refs. [15, 16] we find for the
equivalent result in QCD

$\displaystyle\frac{A_{t}}{\hat{A}_{t}}=1-a_{s}$
$\displaystyle-(1.292+(0.8955-1.440i)_{\rm si})\,a_{s}^{2}$
$\displaystyle+(7.440+16.49\,i)\,a_{s}^{3}-(22.31+c_{4})\,a_{s}^{4}+\cdots$
(20)

At $\mathcal{O}(a_{s}^{4})$ only the contributions where the photons couple to
a massive top-quark loop are known from Ref. [15] (we have re-expressed the
result of Ref. [15] in terms of $a_{s}=\alpha_{s}^{(n_{f}=5)}/\pi$) and the
constant $c_{4}$ has yet to be calculated. But these contributions do dominate
the real part of the $\mathcal{O}(\alpha_{s}^{3})$ coefficient, as can be seen
comparing the results of Refs. [15, 16]. The singlet contributions to
$\mathcal{O}(a_{s}^{2})$ are indicated by ‘si’. The imaginary part in the
singlet contributions is required by unitarity. In the decay width, however,
the first contribution arising from an imaginary part, namely from ${\rm
Im}(A_{t}^{(2)})$, starts at $\mathcal{O}(\alpha_{s}^{4})$ and can be expected
to be suppressed, since it competes with the real parts from higher-order
coefficients, which tend to be significantly larger, as seen in the QCD
result. We therefore compare the real parts of the coefficients of Eqs. (19)
and (3). From these results we see that the magnitude of the coefficients
obtained in the large-$\beta_{0}$ limit are not too far from the magnitude of
the known QCD coefficients, but the sign is wrong in all cases. In QCD, the
systematic sign alternation is not yet established at order
$\mathcal{O}(\alpha_{s}^{2})$ although the signs do alternate starting from
$\mathcal{O}(\alpha_{s}^{3})$ for the known terms. This is an indication that
the UV renormalon in QCD should be less prominent than in the
large-$\beta_{0}$ limit, in agreement with other processes where the
large-$\beta_{0}$ result is known [28, 27, 29].

Based on the observation that the large-$\beta_{0}$ result roughly reproduces
the size (but not the sign) of the QCD contributions, we can speculate that
the $\alpha_{s}^{5}$ non-singlet coefficient would have the same magnitude as
the one shown in Eq. (19). Given the important contribution of the singlet
diagrams, it is safer to attach a 100% uncertainty to this result, which leads
to the following estimate for the $\mathcal{O}(a_{s}^{5})$ contribution to
$A_{t}$ for $\mu=\overline{m}_{t}(\overline{m}_{t})$:

$A_{t}^{(5)}\approx 300\pm 300.$ (21)

In the decay width, in turn, this implies a correction of, at most,

$\Gamma(H\to\gamma\gamma)\Big{|}_{\alpha_{s}^{5}}-\Gamma(H\to\gamma\gamma)\Big{|}_{\alpha_{s}^{4}}\approx+0.00015\,{\rm
keV},$ (22)

which is about three times smaller than the contribution from the partial
$\alpha_{s}^{4}$ correction calculated in Ref. [15], which amounted to mere
0.04‰ of the total decay width. (For this estimate we have added the LO $W$
contribution and top-mass corrections in the 1- and 2-loop QCD results — see,
e.g., the expressions given in Ref. [14].) This clearly indicates that the QCD
contributions are under very good control here, with an uncertainty from the
truncation which is about 10 times smaller than the uncertainty arising
parametrically from the value of $\alpha_{s}$ itself, for example, and much
smaller than the uncertainty from $m_{H}$ which amounts to about $0.04$ keV.

##  4 Conclusions

In this work we presented the result for the QCD corrections to
$H\to\gamma\gamma$ in the large-$\beta_{0}$ limit for an infinitely heavy top
quark. The analytical result was obtained with a modified version of MATAD
[20]. From the Borel transformed amplitude, upon re-expanding in the Borel
variable, we can reconstruct the perturbative series to all orders in
$\alpha_{s}$ in the large-$\beta_{0}$ limit. In particular, the exact
analytical results for the leading-$n_{f}$ contribution at each order can be
extracted and, apart from reproducing the known results, we have given
explicit expressions for the previously unknown contributions at 5 and 6
loops, in Eqs. (17) and (3). The higher-order leading-$n_{f}$ coefficients can
easily be extracted from the analytical result for the Borel transformed
amplitude, Eq. (13). These results can serve as a partial cross-check for
future calculations, should the full QCD corrections be independently
calculated by other groups.

Furthermore, the exact knowledge of the Borel transform in the
large-$\beta_{0}$ limit allows for a study of the renormalon singularities. We
find the usual and expected towers of UV and IR renormalons: the leading UV
renormalon, located at $u=-1$ in the Borel plane, being the closest to the
origin and dominating the series at high orders, while the leading IR
renormalon is related to a quartic IR sensitivity, and appears at $u=2$. In
the large-$\beta_{0}$ limit, the UV renormalon has a somewhat large residue
and it dominates the series for values of the renormalization scale below 200
GeV or so. The perturbative series has a systematic sign alternation which is
not observed in QCD. In QCD, the coefficients can have different signs but no
systematic alternation is observed, which indicates a weaker UV renormalon, as
observed in other processes [28, 27, 29], and a more complicated interplay
between UV and IR contributions.

The perturbative series in the large-$\beta_{0}$ limit has coefficients of
roughly the same order of magnitude as in full QCD up to
$\mathcal{O}(\alpha_{s}^{3})$, or even $\mathcal{O}(\alpha_{s}^{4})$ if we
compare with the partial results of Ref. [15], although the signs of the
coefficients are opposite. Assuming this observation survives at even higher
orders, we estimate the six-loop coefficient to be $A_{t}^{(5)}\approx 300\pm
300$, for $\mu=\overline{m}_{t}(\overline{m}_{t})$. This, in turn, leads to a
contribution to the decay width of at most 0.00015 keV, which is significantly
below the parametric uncertainty induced by $m_{H}$ and $\alpha_{s}$ and much
smaller than the available estimates of the $\mathcal{O}(\alpha_{s}^{4})$
contribution [15].

##  Appendix A Modifications to MATAD’s source code

In this appendix we briefly describe the technical implementation in MATAD of
the calculation of the diagrams of Fig. 1. The implementation of the
large-$\beta_{0}$ limit calculation in MATAD requires some modifications to
the source code. The first modification deals with the Borel transformed
dressed gluon propagator, where the variable $u$ of Eq. (9), which is not
supported in the original package, must be included. This basically
generalizes the exponent of the $k^{2}$ term in the denominator of the gluon
propagator to the real domain. The second modification takes into account the
inclusion of the variable $u$ into the 2-loop tadpole integrals, where the
modified gluon propagator enters the calculation.

The first modification is to the gluon propagator to account for Eq. (9). We
arranged the diagrams such that the gluon propagators always carried momentum
$p_{3}$, without any external momenta. With this, the modification is rather
simple, and we only need to multiply the original propagator by a new function
that we call Denu, which accounts for the factors of $u$ in the modified
propagator. In standard FORM notation, this amounts to

⬇

id Dg(?x) = Dg(?x)*Denu(u);

In the next step, one identifies the powers in the quark and gluon propagators
and unifies the scalar integrals into a single function with the relevant
coefficients in order to make contact with the analytic calculation of the
relevant two-loop integrals. As discussed previously, $p_{3}$ was reserved for
the gluon propagator; the momenta $p_{1}$ and $p_{2}$ were assigned to the top
propagators. In the following expression, s1m and s2m represent the quark
propagators with momentum $p_{1}$ and $p_{2}$, respectively. In FORM code, the
unification of the scalar integrals into a function with the relevant
coefficients reads

⬇

id s1m^a1?*s2m^a2?/p3.p3^a3? = f(a1,a2,a3);

After this, we simply exclude the scaleless integrals which integrate to zero
in dimensional regularization.

Since we are interested in the leading term in the expansion in $\tau_{t}$,
corresponding to the limit $m_{t}\to\infty$, we need to consider only the hard
region of the loop integrals. In this region the integrals become massive
tadpoles. We can then perform the corresponding scalar loop integrals using
the well-known formula

$\displaystyle\int\frac{\mathrm{d}^{D}k\,\mathrm{d}^{D}\ell}{(-k^{2}+m^{2})^{a_{1}}(-\ell^{2}+m^{2})^{a_{2}}[-(k+\ell)^{2}]^{a_{3}}}$
$\displaystyle=(i\pi^{D/2})^{2}\frac{\Gamma(a_{1}+a_{3}+\varepsilon-2)\Gamma(a_{2}+a_{3}+\varepsilon-2)\Gamma(2-\varepsilon-
a_{3})}{\Gamma(a_{1})\Gamma(a_{2})\Gamma(2-\varepsilon)}$
$\displaystyle\times\frac{\Gamma(a_{1}+a_{2}+a_{3}+2\varepsilon-4)}{\Gamma(a_{1}+a_{2}+2a_{3}+2\varepsilon-4)}(m^{-2})^{a_{1}+a_{2}+a_{3}+2\varepsilon-4}.$
(23)

The $u$ variable is attached to the massless propagator, i.e., in our case it
appears together with the $a_{3}$ variable. We implement this with the
following id statement:

⬇

id f(a1?,a2?,a3?)*Denu(u?) = eMu(u) * M^(2*(4 - a1-a2-a3))

* Gam(a1+a3-2,1,u)*Gam(a2+a3-2,1,u)*Gam(2-a3,-1,-u)

* iGam(a1,0,0)*iGam(a2,0,0)*iGam(2,-1,0)

* Gam(a1+a2+a3-4,2,u)*iGam(a1+a2+2*a3-4,2,2*u);

.sort

On the r.h.s. of the equality sign, eMu is the function containing the
additional factors of the Borel transformed gluon propagator in Eq. (9), that
is

$\text{{eMu}(u)}\equiv\bigg{(}-\frac{\mu^{2}}{m_{t}^{2}}e^{-C}\bigg{)}^{u}.$

The functions Gam and iGam (iGam $\equiv$ 1/Gam) are the $\Gamma$-function and
its inverse using the following notation

$\Gamma(a+b\epsilon+cu)\equiv\texttt{Gam(a,b,c)},$ (24)

and analogously for the inverse of the $\Gamma$ function.

## Acknowledgments

GdN thanks the Particle Physics Group of the Universität Siegen, where part of
this work was carried out, for hospitality. DB’s work was supported in part by
the São Paulo Research Foundation (FAPESP) Grants No. 2015/20689-9 and No.
2021/06756-6, by CNPq Grant No. 308979/2021-4. The work of GdN was suppported
by FAPESP Grants No. 2018/12305-4 and No. 2019/17046-0. The work of JP was
supported by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under grant 396021762 – TRR 257 “Particle Physics Phenomenology
after the Higgs Discovery.”

## References

  * [1] J. R. Ellis, M. K. Gaillard and D. V. Nanopoulos, _A Phenomenological Profile of the Higgs Boson_ , _Nucl. Phys. B_ 106 (1976) 292.
  * [2] M. A. Shifman, A. I. Vainshtein, M. B. Voloshin and V. I. Zakharov, _Low-Energy Theorems for Higgs Boson Couplings to Photons_ , _Sov. J. Nucl. Phys._ 30 (1979) 711.
  * [3] L. B. Okun, _Leptons and Quarks: Special Edition Commemorating the Discovery of the Higgs Boson_. North-Holland, Amsterdam, Netherlands, 1982, 10.1142/9162.
  * [4] M. B. Gavela, G. Girardi, C. Malleville and P. Sorba, _A Nonlinear R(xi) Gauge Condition for the Electroweak SU(2) X U(1) Model_ , _Nucl. Phys. B_ 193 (1981) 257.
  * [5] A. Djouadi, M. Spira, J. J. van der Bij and P. M. Zerwas, _QCD corrections to gamma gamma decays of Higgs particles in the intermediate mass range_ , _Phys. Lett. B_ 257 (1991) 187.
  * [6] K. Melnikov and O. I. Yakovlev, _Higgs $\to$ two photon decay: QCD radiative correction_, _Phys. Lett. B_ 312 (1993) 179 [hep-ph/9302281].
  * [7] A. Djouadi, M. Spira and P. M. Zerwas, _Two photon decay widths of Higgs particles_ , _Phys. Lett. B_ 311 (1993) 255 [hep-ph/9305335].
  * [8] M. Inoue, R. Najima, T. Oka and J. Saito, _QCD corrections to two photon decay of the Higgs boson and its reverse process_ , _Mod. Phys. Lett. A_ 9 (1994) 1189.
  * [9] M. Spira, A. Djouadi, D. Graudenz and P. M. Zerwas, _Higgs boson production at the LHC_ , _Nucl. Phys. B_ 453 (1995) 17 [hep-ph/9504378].
  * [10] J. Fleischer, O. V. Tarasov and V. O. Tarasov, _Analytical result for the two loop QCD correction to the decay H $\to$ 2 gamma_, _Phys. Lett. B_ 584 (2004) 294 [hep-ph/0401090].
  * [11] R. Harlander and P. Kant, _Higgs production and decay: Analytic results at next-to-leading order QCD_ , _JHEP_ 12 (2005) 015 [hep-ph/0509189].
  * [12] U. Aglietti, R. Bonciani, G. Degrassi and A. Vicini, _Analytic Results for Virtual QCD Corrections to Higgs Production and Decay_ , _JHEP_ 01 (2007) 021 [hep-ph/0611266].
  * [13] M. Steinhauser, _Corrections of O $(\alpha_{s}^{2})$ to the decay of an intermediate mass Higgs boson into two photons_, in _Ringberg Workshop: The Higgs Puzzle - What can We Learn from LEP2, LHC, NLC, and FMC?_ , pp. 177–185, 12, 1996, hep-ph/9612395.
  * [14] P. Maierhöfer and P. Marquard, _Complete three-loop QCD corrections to the decay $H\to\gamma\gamma$_, _Phys. Lett. B_ 721 (2013) 131 [1212.6233].
  * [15] C. Sturm, _Higher order QCD results for the fermionic contributions of the Higgs-boson decay into two photons and the decoupling function for the $\overline{\text{MS}}$ renormalized fine-structure constant_, _Eur. Phys. J. C_ 74 (2014) 2978 [1404.3433].
  * [16] J. Davies and F. Herren, _Higgs boson decay into photons at four loops_ , _Phys. Rev. D_ 104 (2021) 053010 [2104.12780].
  * [17] M. Beneke, _Renormalons_ , _Phys. Rept._ 317 (1999) 1 [hep-ph/9807443].
  * [18] D. J. Broadhurst and A. G. Grozin, _Matching QCD and HQET heavy - light currents at two loops and beyond_ , _Phys. Rev. D_ 52 (1995) 4082 [hep-ph/9410240].
  * [19] M. Beneke and V. M. Braun, _Naive nonAbelianization and resummation of fermion bubble chains_ , _Phys. Lett. B_ 348 (1995) 513 [hep-ph/9411229].
  * [20] M. Steinhauser, _MATAD: A Program package for the computation of MAssive TADpoles_ , _Comput. Phys. Commun._ 134 (2001) 335 [hep-ph/0009029].
  * [21] B. Ruijl, T. Ueda and J. Vermaseren, _FORM version 4.2_ , 1707.06453.
  * [22] A. Djouadi, _The Anatomy of electro-weak symmetry breaking. I: The Higgs boson in the standard model_ , _Phys. Rept._ 457 (2008) 1 [hep-ph/0503172].
  * [23] V. A. Smirnov, _Applied asymptotic expansions in momenta and masses_ , _Springer Tracts Mod. Phys._ 177 (2002) 1.
  * [24] P. Nogueira, _Automatic Feynman graph generation_ , _J. Comput. Phys._ 105 (1993) 279.
  * [25] M. Beneke, _Pole mass renormalon and its ramifications_ , _Eur. Phys. J. ST_ 230 (2021) 2565 [2108.04861].
  * [26] A. G. Grozin, _Renormalons: Technical introduction_ , hep-ph/0311050.
  * [27] D. Boito, V. Mateu and M. V. Rodrigues, _Small-momentum expansion of heavy-quark correlators in the large- $\beta$0 limit and $\alpha$s extractions_, _JHEP_ 08 (2021) 027 [2106.05660].
  * [28] D. Boito, P. Masjuan and F. Oliani, _Higher-order QCD corrections to hadronic $\tau$ decays from Padé approximants_, _JHEP_ 08 (2018) 075 [1807.01567].
  * [29] M. A. Benitez-Rathgeb, D. Boito, A. H. Hoang and M. Jamin, _Reconciling the contour-improved and fixed-order approaches for $\tau$ hadronic spectral moments. Part I. Renormalon-free gluon condensate scheme_, _JHEP_ 07 (2022) 016 [2202.10957].