#  PIA: Your Personalized Image Animator
via Plug-and-Play Modules in Text-to-Image Models

Yiming Zhang1222Equal contribution. Zhening Xing1222Equal contribution.
Yanhong Zeng1 Youqing Fang1 Kai Chen1111Corresponding author.

1Shanghai Artificial Intelligence Laboratory

https://PI-Animator.github.io

###### Abstract

Recent advancements in personalized text-to-image (T2I) models have
revolutionized content creation, empowering non-experts to generate stunning
images with unique styles. While promising, adding realistic motions into
these personalized images by text poses significant challenges in preserving
distinct styles, high-fidelity details, and achieving motion controllability
by text. In this paper, we present PIA, a Personalized Image Animator that
excels in aligning with condition images, achieving motion controllability by
text, and the compatibility with various personalized T2I models without
specific tuning. To achieve these goals, PIA builds upon a base T2I model with
well-trained temporal alignment layers, allowing for the seamless
transformation of any personalized T2I model into an image animation model. A
key component of PIA is the introduction of the condition module, which
utilizes the condition frame and inter-frame affinity as input to transfer
appearance information guided by the affinity hint for individual frame
synthesis in the latent space. This design mitigates the challenges of
appearance-related image alignment within PIA and allows for a stronger focus
on aligning with motion-related guidance. To address the lack of a benchmark
for this field, we introduce AnimateBench, a comprehensive benchmark
comprising diverse personalized T2I models, curated images, and motion-related
prompts. We show extensive experiments on AnimateBench to verify the
superiority of PIA.

[autoplay,loop,width=trim=5 0 5 0, clip]8figs/ours/gif_teaser2/015

Figure 1:  Given an elaborated image generated by a personalized text-to-image
model, the proposed Personalized Image Animator (PIA) animates it with
realistic motions according to different text prompts while preserving the
original distinct styles and high-fidelity details. We recommend using Abode
Arobat and clicking the images to play the animation clips. [Best viewed in
color with zoom-in]

##  1 Introduction

Recent democratization of text-to-image (T2I) generation has seen incredible
progress by the growing Stable Diffusion community [31, 1] and the explosion
of personalized models [32, 19]. Numerous AI artists and enthusiasts create,
share, and use personalized models on model-sharing platforms such as Civitai
[6] and Hugging Face [20], turning imagination into images at an affordable
cost (e.g., a laptop with an RTX3080). However, there is still limited
research exploring the realm of infusing these unique personalized images with
dynamic motions [39, 11, 4].

We are intrigued by the potential of animating personalized images, given
their distinctive styles and impressive visual quality. Moreover, the
incorporation of text prompts to guide the animation process can significantly
streamline various creative endeavors within the vibrant personalized
community, minimizing the required effort. While promising, this task presents
two distinct challenges: image alignment with the detailed personalized image
and achieving motion controllability through text prompt. While recent
advancements in training large-scale text-to-video (T2V) models have enabled
users to generate videos based on text or images [39], these approaches
struggle to preserve the unique styles and high-fidelity details of
conditional images due to the absence of personalized domain knowledge. To
address this limitation, researchers have developed temporal alignment layers
[4, 11]. These plug-and-play modules effectively transform personalized T2I
models into T2V models while maintaining their distinct styles [4, 11].
However, despite numerous attempts to adapt these personalized T2V models for
image animation, the primary focus of the temporal alignment layers remains on
aligning the appearance of individually generated frames in latent space to
achieve smooth video results. As a result, their ability to effectively
respond to motion-related text guidance is limited.

To address the aforementioned limitations, we introduce PIA, a Personalized
Image Animator that excels in both image alignment and motion controllability.
Initially, we choose to build our framework upon a base T2I model (_i.e_.,
Stable Diffusion [31]) and incorporate well-established temporal alignment
layers following previous works [11]. This approach allows PIA to effectively
leverage personalized domain knowledge by seamlessly replacing the base T2I
model with any personalized T2I model during inference. Next, to empower our
framework with image animation ability, we introduce a trainable condition
module into the input layer of the T2I model. This module plays a crucial role
in generating each frame of the animation by taking the conditional image and
the inter-frame affinity between the current frame and the conditional frame
as inputs. Through this design, the condition module effectively borrows
appearance features from the conditional frame, guided by the affinity hint,
resulting in improved alignment with the condition image. Finally, we fine-
tune the condition module and the temporal alignment layers while keeping the
T2I model fixed to preserve its open-domain knowledge. Leveraging the enhanced
appearance-related alignment facilitated by the condition module within the
input layer, the temporal alignment layers can focus more on motion-related
alignment, leading to improved motion controllability. Importantly, we
demonstrate the effortless generalizability of PIA’s animation capability by
replacing the T2I model with any other personalized model. This flexibility
empowers users to animate their elaborated personal images using text prompts
while preserving distinct features and high-fidelity details.

In summary, PIA presents a powerful solution for personalized image animation,
offering superior image alignment, motion controllability by text, and the
flexibility to integrate various personalized models. This comprehensive
approach ensures an engaging and customizable animation experience for users.
To address the lack of a benchmark in personalized image animation, we
introduce a new benchmark called AnimateBench, which comprises various
personalized T2Is, curated images, and tailored motion-related prompts. To
validate the effectiveness of PIA, we conducted extensive quantitative and
qualitative evaluations using AnimateBench. These evaluations provide a robust
assessment of the capabilities and performance of our approach.

Figure 2:  Personalized Image Animator (PIA). As shown in (a), PIA consists of
a text-to-image (T2I) model, well-trained temporal alignment layers (TA), and
a new condition module $\mathcal{W}_{cond}$ responsible for encoding the
condition image $z^{I}$ and inter-frame affinity $s^{1:F}$. In particular, the
T2I model consists of U-Net blocks, including a ResBlock (Res) [13], a self-
attention layer (SA), and a cross-attention layer (CA), as depicted in (b).
During training, the condition module learns to leverage the affinity hints
and incorporate appearance information from the condition images, facilitating
image alignment and enabling a stronger emphasis on motion-related alignment.

##  2 Related Work

#### Single Image Animation.

Enhancing still pictures with motions has attracted decades of attention and
efforts in the research field [5, 8, 3, 41, 18, 35, 36]. Chuang _et al_.
present a pioneer work for animating a variety of photographs and paintings
[5]. Specifically, the still scene is segmented into a series of layers by a
human user and each layer is rendered with a stochastic motion texture to form
the final animated video. With the success of deep learning, more recent works
get rid of the manual segmentation and can synthesize more natural motion
textures [8, 3, 41, 18, 35, 36]. Endo _et al_. propose learning and predicting
the motion and appearance separately using two types of DNN models for
animating landscape [8]. Holynski _et al_. learns an image-to-image
translation network to encode motion priors of natural scenes with continuous
fluid [18]. Bertiche _et al_. focus on learning garment motion dynamics for
blowing the dressed humans under the wind [3]. Xiao _et al_. propose
automatically segmenting hair wisps and animating hair wisps with a wisp-aware
animation module with pleasing motions [41]. Some works focus on synthesizing
natural lip motion for talking heads based on audio [36, 35]. Recent works
also have been significantly prompted by the powerful diffusion models [21,
23, 31]. Mahapatra _et al_. transfer the estimated optical flow from real
images to artistic paintings by the concept of twin image synthesis with a
pre-trained text-to-image diffusion model [23]. Li _et al_. leverages a latent
diffusion model for modeling natural oscillating motion such as flowers and
candles blowing in the wind [21]. Despite the significant progress, these
animation techniques can only synthesize specific types of content and motion
such as time-lapse videos, and facial and body animation. In contrast, our
model, PIA, can animate the image without segmenting the region of interest
and drive the image with any motion through a text prompt.

#### Text-to-video Synthesis.

Recent large text-to-image models have shown remarkable progress and attracted
significant attention, by enabling diverse and high-fidelity image synthesis
based on a text prompt written in natural language [31, 34, 29, 30, 26]. The
video latent diffusion model (VLDM) has demonstrated a strong ability to
capture motion priors [40, 4, 11, 9, 17, 12, 22, 42]. Some text-to-video
generation models[4, 11] are trained to synthesize natural videos. However,
most of them can not take an image as input. Current approaches[33] have tried
to improve image animation based on VLDM, but they suffer from issues such as
lack of fidelity to the conditioning frames and poor control over generated
content. Some works[39, 45] finetune models to take images as condition,
however, most image details are lost. Besides, most of the prompts are used to
describe the image content, users can not animate the image according to their
intent. AnimateDiff [11] is a pipeline that aims at training a plug-and-play
motion module from large-scale video datasets and transferring the learned
reasonable motion priors to new domains by inserting the motion module into
customized T2I models. Although it has shown promising results, AnimateDiff
fails to animate a given image with a text-to-video model.

##  3 PIA: Your Personalized Image Animator

For an image $I$ generated from a personalized text-to-image model, our target
is to generate a $F$ frame video clip $V=\\{v^{1},\dots,v^{F}\\}$, where the
first frame $v^{1}$ should resemble condition image $I$, and the motion should
follow the input prompt $c$. As mentioned in the previous section, since the
appearance information between frames is independent of each other, the
current temporal alignment layers need to align appearance and motion
information across frames simultaneously. This prevents temporal alignment
layers from learning motion prior effectively. To address this issue, we
design PIA, as shown in Fig. 2. We propose two frame-wise inputs: the
conditional image $I$ and the inter-frame affinity $s^{1:F}$. These inputs are
encoded through a condition module, which provides the necessary appearance
information for each frame.

In this section, Sec. 3.1 introduces preliminary information about Stable
Diffusion model and temporal alignment techniques in text-to-video methods.
Then, we describe the design of our proposed PIA in Sec. 3.2. Finally, in Sec.
3.3, we present the training and inference details.

###  3.1 Preliminaries

#### Stable Diffusion.

Stable Diffusion (SD) [31] is one of the most popular large-scale open-source
text-to-image models and has a well-developed community. SD has 4 main
components: a encoder $\mathcal{E}$, a decoder $\mathcal{D}$, a U-Net denoiser
$\epsilon_{\theta}$, and a pretrained CLIP-based [27] text encoder
$\mathcal{T}$. In the training stage, the input image $x$ is first encoded to
the latent code $z_{0}=\mathcal{E}(x)$. Then $z_{0}$ is perturbed via pre-
defined diffusion process [16, 24, 7]:

$z_{t}=\sqrt{\bar{\alpha_{t}}}z_{0}+\sqrt{1-\bar{\alpha_{t}}}\epsilon_{t},\epsilon_{t}\sim\mathcal{N}(0,I)$
(1)

where timestep $t\sim\mathcal{U}[1,T]$, and $\bar{\alpha_{t}}$ is the noise
strength at $t$. Then the denoising U-Net $\epsilon$ is trained to predict the
added noise with the following loss:

$\mathcal{L}=\mathbb{E}_{x,\epsilon\sim
N(0,1),t,c}\left[\|\epsilon-\epsilon_{\theta}(z_{t},t,\mathcal{T}(c))\|\right]$
(2)

where $c$ is the corresponding prompt for $x_{0}$.

#### Temporal Alignment Layer.

The recent T2V methods [40, 11, 4] have discovered that incorporating temporal
transformers[38] into SD enables the alignment of appearance and motion
information across frames, thereby transforming the T2I model into a T2V
model. Taking feature with a shape of $f\in\mathbb{R}^{F,C,H,W}$ as an
example, where $F$ represents the frame length, $C$ denotes the number of
channels, and $H$ and $W$ represent the spatial dimensions. We begin by
permuting its shape to $((H,W),F,C)$, and then the transformers layers for
temporal alignment can be defined as follows:

$f_{out}=\texttt{Softmax}(\frac{QK^{\intercal}}{\sqrt{c}})\cdot V,$ (3)

where $Q=\mathcal{W}^{Q}z$, $K=\mathcal{W}^{K}z$, $V=\mathcal{W}^{V}z$ are
projection operation. Due to the temporal alignment layer needing to consider
both appearance and motion information simultaneously, a trade-off arises
between appearance consistency and motion controllability in the T2V task. In
our work, we introduce explicitly appearance information and corresponding
encoding module to make the temporal alignment layer focus more on motion-
related information.

###  3.2 Plug-and-Play Modules for Animation

Figure 3:  Illustration of the condition module (CM). A vanilla personalized
T2V model (shown in (a)) needs to align both the appearance and motion of
individual frames simultaneously. PIA with CM (shown in (b)) can borrow
appearance information from condition image $z^{I}$ with affinity hints $s$,
easing the challenge of both appearance and motion alignment. We use the color
and strip to denote appearance and motion, respectively.

We introduce the additional conditions and how conditional module can borrow
appearance information from them to improve the motion controllability. In the
T2V model, the initial latent for each frame is independently sampled from
Gaussian Distribution, this results in inconsistent appearance identities for
each frame, as illustrated in the upper row of Fig. 3. As the only frame-aware
modules, temporal alignment layers need to devote effort to simultaneously
learning motion prior and aligning appearance consistency. To address this
problem, we encode the conditional image $I$ to the latent space, _i.e_.
$z^{I}=\mathcal{E}(I)$, and utilize it as the explicit appearance information
for each frame. Furthermore, we propose inter-frame affinity $s$ to quantify
the degree of motion for each frame in relation to the conditional image. In
the training stage, we calculate the affinity score based on the training
data. Given a video clip $v^{1:F}$, we set the first frame $v^{1}$ as the
conditional image. For frame $v^{i}$, we first calculate the L1 distance
between it and the conditional frame $v^{1}$ in HSV space, and denote this
distance as $d^{i}$. Subsequently, we apply this operation to all frames of
video clips in the dataset, resulting in the maximum distance value
$d_{\mathrm{max}}$. Then we normalize the distance $d^{i}$ to $[0,1]$ via
$d_{\mathrm{max}}$. The affinity score can be calculated as
$s^{i}=1-d^{i}/d_{\mathrm{max}}\times(s_{\mathrm{max}}-s_{\mathrm{min}})$.
Here, $s_{\mathrm{max}}$ and $s_{\mathrm{min}}$ are hyperparameters to scale
affinity scores to specific ranges. In order to align $s^{i}$ with $z^{I}$, we
expand it to a single channel feature map with the same size as latent code
(_i.e_. $[1\times h\times w]$). During the inference stage, we allow the user
to provide a user-defined input for $s^{i}$. This enables the generation of
animation results with varying motion scales, providing customization in the
output, as shown in Fig. 7.

To expand the capabilities of the U-Net without compromising its original
functionality, we introduce a lightweight single-layer convolution as the
conditional module. For condition $z^{I}\oplus s^{i}$, where $\oplus$ denotes
concatenation operation, we encode it with $\mathcal{W}_{cond}$ and add it to
the output of the first convolution layer. As shown in the lower row of Fig.
3, the inclusion of the conditional module results in enhanced consistency of
the appearance information in the output of the first convolutional layer
compared to the original model. This makes subsequent temporal alignment
layers focus on aligning motion guidance.

This operation is equivalent to concatenating $\mathcal{W}_{\mathrm{cond}}$
and $\mathcal{W}$, the weight of the original convolution block, at the first
dimension. To ensure the newly added weight does not hurt the performance of
the pretrained model, we apply zero-initialization [44] to
$\mathcal{W}_{\mathrm{cond}}$.

###  3.3 Training and inference

Prior to the training of PIA, following AnimateDiff [11], we initiate the
process by pretraining a domain adapter and the temporal alignment layers on a
single frame dataset and a video dataset, respectively. Next, we expand the
number of channels in the first convolution block by initializing the
additional channels with zero weights, as shown in Fig. 3. In the training
stage, for sampled video clip $v^{1:F}$, we designate the first frame as the
conditional image, _i.e_. $I:=v^{0}$. We only update the
$\mathcal{W}_{\mathrm{cond}}$ and the temporal alignment layers the following
loss function, while keeping the other parameters unchanged.

$\mathcal{L}=\mathbb{E}_{v^{1:F},\epsilon\sim\mathcal{N}(0,I),t,c}[\|\epsilon-\epsilon_{\theta}(z_{t}^{1:F},z_{0}^{0},s^{1:F},t,\mathcal{T}(c))||_{2}^{2}],$
(4)

where $c$ is the corresponding prompt, $z_{0}^{0}$ is the clean latent code
for the first frame and $z_{t}^{1:F}$ is perturbed latent code follow Eq. (1).
To retain the model’s text-to-video capability, we randomly drop inputs with a
20% probability during training, effectively transforming the training task
into a T2V task. Specifically, we replace both $s^{1:F}$ and $z^{I}$ with
zeros. When calculating the affinity score, we set $s_{\mathrm{max}}$ and
$s_{\mathrm{min}}$ to be 1 and 0.2, respectively. Both single frame and video
data training are conducted on WebVid dataset [2]. The number of frames $F$ is
fixed as 16 during training. In the inference stage, for image $I$ from
personalized models, we replace the domain adapter with personalized models
and keep the temporal alignment layers and extended convolution block
unchanged. Affinity score $s$ serves as user input and plays a role in
adjusting the magnitude of motion.

##  4 Experiments

###  4.1 AnimateBench

#### AnimateBench.

Existing benchmarks are restricted to specific domains like human faces, fluid
elements, etc. To this end, we introduce AnimateBench for comparisons in the
field of personalized image animation. AnimateBench contains images in
different domains and multiple prompts to test the text-based image animation
ability. Specifically, we evaluated 105 personalized cases, which contain
different contents, styles, and concepts. These personalized cases are
generated from 7 different personalized text-to-image models, which are used
to evaluate the domain generalization ability. We use five images generated by
each model for a comprehensive comparison. In addition, we carefully elaborate
three motion-related prompts for each image to evaluate the motion
controllability by the text of different approaches. Specifically, the motion-
related text prompts typically describe the following motions that the image
probably happens within a single shot shot.

#### Evaluation Metrics.

Our quantitative comparison evaluates animation quality including image
alignment and text alignment, following previous works [40, 11]. Specifically,
the CLIP score is calculated by computing the cosine similarity between
different embeddings, _e.g_., text and image. Therefore, we can use the CLIP
score to evaluate both text and input image similarity with videos.

  * •

Image Alignment We compute the cosine similarity between input image
embeddings and each video frame embeddings to evaluate the image-video
alignment.

  * •

Text Alignment We also compute the cosine similarity between text embeddings
and frame embeddings to evaluate the text-video alignment.

Figure 4: An example of AnimateBench. The images in AnimateBench are carefully
crafted using a set of collected personalized text-to-image models. Each image
has three carefully designed prompts, describing the following motions that
the image probably happens within a single short shot. Figure 5: Qualitative
comparison with state-of-the art approaches. Compared with other methods, PIA
shows excellent motion controllability and strong image alignment.
Specifically, in the first case, PIA generates a “walking” motion for the toy
bear (in its feet), while other methods can only remain static frames, showing
a lack of motion controllability. In the second case, PIA adds a new element,
_i.e_., fire, with realistic motion. We show more video cases in supplementary
materials due to the file size limit of the main paper.

#### Baselines.

We carefully choose the most recent and competitive approaches for
personalized image animation with their brief introduction as follows,

  * •

VideoComposer [39] can generate controllable videos using different conditions
such as text, image, motion vector, or a combination of them.

  * •

AnimateDiff [11] is a milestone work that largely facilitates the development
of personalized video generation by learning motion prior. We extend it for
the application of image animation following previous best practices by using
ControlNet [44] and image prompt adapter [43].

  * •

Pika Labs [25] and Gen2 [10] are two of the most popular commercial tools for
video generation, which are widely recognized as state-of-the-art in image
animation.

###  4.2 Comparisons with State-of-the-Art

We compare our method with state-of-the-art for personalized image animation
on AnimateBench. The performance of these models was evaluated in various
terms of text alignment, and image alignment.

#### Qualitative Comparison.

We qualitatively compared PIA with the most recent animation methods,
including AnimateDiff-ControlNet, Gen-2, Pika Labs, and VideoComposer. Visual
results can be found in Fig. 5. It can be observed that the videos generated
by PIA exhibit better responses to motion-related keywords in the prompts and
better preservation of details from the input image content. Videocomposer
produces video clips with poor text alignment and incoherent motion.
Animatediff-ControlNet, on the other hand, generates videos with smooth motion
but loses image details. Gen2 and Pika Labs are prone to generate videos with
high identity but respond less to the prompt.

#### Quantitative Evaluation.

With the personalized cases, we computed the CLIP Score to compare text-video
alignment and image-video alignment. Detailed results are shown in Table 1.
The results demonstrate that PIA achieves the highest CLIP Scores, indicating
strong text-video alignment and high image fidelity.

We generated videos with different methods on AnimateBench and solicited
ratings from participants for the ranking in text alignment and image
alignment. Specifically, we set up twenty questions and asked which of the
videos best matched the text or image. Note that all the cases are selected
randomly. We then compute the preference rate to evaluate the performance of
all the methods. Results can be found in Table 1.

Methods | CLIP Score | User Study
---|---|---
image | text | image | text
VideoComposer[39] | 225.3 | 62.85 | 0.180 | 0.110
AnimateDiff[11] | 218.0 | 63.31 | 0.295 | 0.220
PIA(ours) | 225.9 | 63.68 | 0.525 | 0.670
Table 1: Quantitative comparison on AnimateBench.

###  4.3 Analysis

In this section, we present some interesting applications and conduct ablative
study. Specifically, we first introduce three abilities of PIA, which are
motion control by text prompt, motion magnitude controllability and style
transfer. We show visual results for each application. Then we compare two
different frame affinities to clarify the effectiveness. Details of different
constructions of frame affinity will be explained in the following.

#### Motion Control by Text Prompt.

The condition module enables PIA to borrow appearance features from the
conditional frame so that temporal alignment layers can focus more on motion
generation. These designs enable PIA to perform better image and text
alignment in generated videos. The strong prompt control ability of PIA can be
seen in Fig. 6. During the test, we observed that a simple prompt design
produces great visual effects, as shown in the figure. PIA is able to generate
fancy results by adding new elements to the generated videos with smooth
motion quality based on text prompts. Unlike Pika Labs or Gen-2 prone to
generate small motion within 3 seconds, PIA generates video with more text
alignment motion.

Figure 6: Motion Control by Text Prompt. PIA can effectively capture the
motion-related guidance in text prompt and add realistic related motion in the
results. Figure 7: Motion Magnitude Controllability. PIA enables users to
adjust the motion magnitude as light, middle, or large by setting the input
affinity information as different values.

#### Motion Magnitude Controllability.

PIA can control the magnitude of motion by adjusting the frame affinity. In
the inference stage, we constructed three types of frame affinity patterns
based on real video data, and the test results are shown in Fig. 7.
Specifically, we use the statistics in the training stage, to construct the
similarity affinity pattern. We broadly divide the range of motion into three
categories and provide the corresponding affinity in inference time. In Fig.
7, the frame affinity gradually decreases from top to bottom, indicating an
increasing magnitude of motion. From the results, it can be observed that the
generated motion aligns with our expectations, demonstrating the effectiveness
of the similarity channel in controlling motion magnitude. Meanwhile, PIA can
also control the position of the input image in the video by adjusting the
Similarity channel. In short, by using different frame affinity during the
inference stage, users can flexibly control the motion magnitude and
conditional frame position in the generated video, leading to a more
controllable animation.

#### Style Transfer.

So far, we have discussed using models and input images in the same domain to
generate videos. However, we have found interesting facts that if the two have
different domains, such as providing a real-style image and using an anime-
style model to generate motion, PIA can achieve style transfer effects. As
shown in Fig. 8. Frame affinity helps generate videos with smooth style
transfer between image style and model style. This feature of PIA allows users
to have a better-personalized experience.

Figure 8: Style transfer. PIA is able to achieve style transfer in generated
videos by applying a personalized text-to-image model with a different domain
style to the given image.

#### Effectiveness of Frame Affinity.

In the training stage, we calculate the similarity score for affinity hints as
mentioned before. We perform an ablation study with one-hot frame affinity to
prove the effectiveness of similarity frame affinity. One-hot frame affinity
is constructed by 1 in the condition frame and 0 in the other frames which is
able to indicate the position of the condition frame. We evaluate two methods
on the AnimateBench. One-hot frame affinity obtains the image CLIP score of
210.9, while the similarity frame affinity reaches 225.9. Moreover, we also
show the visual result of video frames generated by the two constructions in
Fig. 9. We can find that frame affinity constructed by similarity allows the
model to learn identity better. Note that in different constructions, we use
the same pattern of affinity in training and inference.

Figure 9: Effectiveness of frame affinity. With frame affinity, PIA is able to
generate videos with stronger image alignment.

#### Limitation.

PIA is capable of animating personalized images with high text and image
alignment with text conditions. However, we have observed that videos
generated by PIA exhibit color discrepancy when applied to images with
significantly different styles from the training data, as shown in Fig. 10. We
hypothesize the color discrepancy is caused by the WebVid [2] domain. PIA
learns the color distribution of WebVid, and when there is a notable
difference between the personalized image during inference, the color
discrepancy may arise. Additionally, when generating images, specific trigger
words such as a certain style may be required. However, when generating
videos, if the prompts do not include these words, it can also result in color
discrepancy. We further speculate that training on a more diverse range of
video data with various styles and content and providing complete trigger
words during inference could mitigate this phenomenon.

Figure 10: Limitation. When the input image is far from the training dataset
domain, PIA is prone to generate videos with significant shifts of color.

##  5 Conclusions

In this paper, we introduce PIA a powerful solution for personalized image
animation. Our method shows excellent image alignment and motion
controllability, ensuring text-based customized animation for users. To
achieve high alignment, we introduce condition module and inter-frame
affinity. They endow PIA borrow appearance features from the conditional
frame, leading temporal layers to focus more on motion generation.
Furthermore, we construct an animation benchmark named AnimateBench to
evaluate the animation performance of PIA and other methods. Experimental
results demonstrate that PIA performs excellent on image animation tasks and
shows various interesting extended applications.

In future work, we first plan to extend our method to a more powerful base
text-to-image model, _e.g_., SDXL [26], which is becoming the new trend in the
Stable Diffusion community. Second, we plan to study the limitation of color
shift caused by the low-quality training dataset.

## References

  * AUTOMATIC [1111] AUTOMATIC1111.  https://github.com/AUTOMATIC1111/stable-diffusion-webui.
  * Bain et al. [2021] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1728–1738, 2021.
  * Bertiche et al. [2023] Hugo Bertiche, Niloy J Mitra, Kuldeep Kulkarni, Chun-Hao P Huang, Tuanfeng Y Wang, Meysam Madadi, Sergio Escalera, and Duygu Ceylan.  Blowing in the wind: Cyclenet for human cinemagraphs from still images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 459–468, 2023.
  * Blattmann et al. [2023] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22563–22575, 2023.
  * Chuang et al. [2005] Yung-Yu Chuang, Dan B Goldman, Ke Colin Zheng, Brian Curless, David H Salesin, and Richard Szeliski.  Animating pictures with stochastic motion textures.  In _ACM SIGGRAPH 2005 Papers_ , pages 853–860. 2005.
  * [6] CIVITAI, 2022.  https://civitai.com/.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Endo et al. [2019] Yuki Endo, Yoshihiro Kanamori, and Shigeru Kuriyama.  Animating landscape: self-supervised learning of decoupled motion and appearance for single-image video synthesis.  _ACM Transactions on Graphics (TOG)_ , 38(6):1–19, 2019.
  * Esser et al. [2023] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7346–7356, 2023.
  * Gen [2] Gen2, 2023.  https://research.runwayml.com/gen2.
  * Guo et al. [2023] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  _arXiv preprint arXiv:2307.04725_ , 2023.
  * Harvey et al. [2022] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood.  Flexible diffusion modeling of long videos.  _Advances in Neural Information Processing Systems_ , 35:27953–27965, 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  CLIPScore: a reference-free evaluation metric for image captioning.  In _EMNLP_ , 2021.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ho et al. [2022] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet.  Video diffusion models, 2022.
  * Holynski et al. [2021] Aleksander Holynski, Brian L Curless, Steven M Seitz, and Richard Szeliski.  Animating pictures with eulerian motion fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5810–5819, 2021.
  * Hu et al. [2021] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2021.
  * [20] Huggingface, 2022.  https://huggingface.co/.
  * Li et al. [2023] Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski.  Generative image dynamics.  _arXiv preprint arXiv:2309.07906_ , 2023.
  * Luo et al. [2023] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan.  Videofusion: Decomposed diffusion models for high-quality video generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10209–10218, 2023.
  * Mahapatra et al. [2023] Aniruddha Mahapatra, Aliaksandr Siarohin, Hsin-Ying Lee, Sergey Tulyakov, and Jun-Yan Zhu.  Text-guided synthesis of eulerian cinemagraphs.  _SIGGRAPH ASIA_ , 2023.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * [25] Pika Labs, 2023.  https://www.pika.art/.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Radford et al. [2021a] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021a.
  * Radford et al. [2021b] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021b.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * s9roll [7] s9roll7.  https://github.com/s9roll7/animatediff-cli-prompt-travel.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Shen et al. [2022] Shuai Shen, Wanhua Li, Zheng Zhu, Yueqi Duan, Jie Zhou, and Jiwen Lu.  Learning dynamic facial radiance fields for few-shot talking head synthesis.  In _European Conference on Computer Vision_ , pages 666–682. Springer, 2022.
  * Shen et al. [2023] Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu.  Difftalk: Crafting diffusion models for generalized audio-driven portraits animation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1982–1991, 2023.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2023] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou.  Videocomposer: Compositional video synthesis with motion controllability.  _arXiv preprint arXiv:2306.02018_ , 2023.
  * Wu et al. [2023] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7623–7633, 2023.
  * Xiao et al. [2023] Wenpeng Xiao, Wentao Liu, Yitong Wang, Bernard Ghanem, and Bing Li.  Automatic animation of hair blowing in still portrait photos.  In _ICCV_ , 2023.
  * Yang et al. [2023] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt.  Diffusion probabilistic modeling for video generation.  _Entropy_ , 25(10):1469, 2023.
  * Ye et al. [2023] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  2023\.
  * Zhang et al. [2023a] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023a.
  * Zhang et al. [2023b] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou.  I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models.  _arXiv preprint arXiv:2311.04145_ , 2023b.

##  6 Implementation Details

###  6.1 Training

We train PIA on WebVid10M [2] with only condition module and temporal
alignment layers trainable. Specifically, we use the motion module in
AnimateDiff [11] as a pre-trained model for temporal alignment layers. We
compute the L1 distance between the condition frame and other frames in HSV
space. Subsequently, we utilize this distance to calculate the affinity score.
Specifically, we consider the top 2.5th percentile of the samples as the
minimum and the 97.5th as the maximum value to linearly scale the affinity
score to $[0.2,1]$. In addition, we set a probability of $20\%$ to zero out
the entire input of the condition module. This ensures that PIA retains text-
to-video capabilities and promotes the training of the condition module. We
train condition module and temporal alignment layers on 16 NVIDIA A100s for
4.5k steps and use a learning rate of $1\times 10^{-5}$.

###  6.2 Inference

During the inference stage, users can replace the base model with the
personalized T2I model to realize the image animation. Besides, we construct
the inter-frame affinity according to the affinity score obtained from the
training stage. We design three affinity ranges for three different amplitude
motions. The maximum value of all three affinity ranges is 1, achieved at the
conditional frame. The minimum values are 0.2, 0.4, and 0.8, respectively,
with corresponding decreases in motion magnitude. We use classifier-free
guidance during the DDIM process [37] and set the classifier-free guidance
[15] scale as 7.5.

##  7 AnimateBench

AnimateBench is a comprehensive benchmark, which consists of 105 image and
prompt pairs. To cover a wide variety of contents, styles, and concepts, we
choose seven base models [32] and LoRA [19]. An example case of AnimateBench
is depicted in Fig. 11.

Figure 11: AnimateBench case. Each curated personalized image corresponds to a
personalized text-to-image model and three tailored motion-related text
prompts.

###  7.1 Images in AnimateBench

We carefully choose seven of the most popular base models [32] and LoRAs [19]
in Cvitai [6]. Each personalized model has very distinct styles and we use
them to curate images with impressive high quality by tailored text prompts
for image generation. Specifically, these images differ in styles, contents,
and concepts and ensure that AnimateBench covers three categories: people,
animals, and landscapes.

###  7.2 Prompts in AnimateBench

For each generated image, we design three prompts describing different motions
to test the text alignment ability of models. Prompts are mainly composed of
three parts: the subject, the motion descriptor, and the trigger words.
Subject and motion descriptors specify the content of motion in the generated
videos. The trigger word is a well-known technique that is able to activate
the DreamBooth or LoRA to generate personalized effects [6]. Only when these
prompts are included during inference, DreamBooth or LoRA can achieve optimal
performance. Then we can get the complete prompt in AnimateBench. For example,
we use ‘1girl is smiling, white hair by atey ghailan, by greg rutkowski, by
greg tocchini.’ to generate a personalized image, and then we can get the
complete prompt as ‘1girl is smiling, white hair by atey ghailan, by greg
rutkowski, by greg tocchini’. In this case, ‘1gril’ represents the subject,
smiling represents the motion descriptor, and ‘white hair by atey ghailan, by
greg rutkowski, by greg tocchini’ represents the trigger word. We also
distinguish motion between different types of subjects. For example, the
prompt of people or animals contains more descriptors such as smiling, crying,
etc, while the prompt of landscapes or scenes contains more like raining,
lightning, etc.

##  8 Evaluation Details

###  8.1 CLIP Score

Following previous work [11, 39, 4], we compute CLIP score to quantitatively
evaluate the alignment in generated videos. In addition to calculating text
alignment, we measure image alignment by computing the similarity between the
embeddings of the generated video frames and the input images. The two average
CLIP scores are calculated on AnimateBench which contains 1680 frames. We
leverage the code provided by [14] and use ViT-B/32 [28] model to extract the
embedding of images and prompts.

###  8.2 User Study

For user study, we randomly select input image and prompt pairs in
AnimateBench and then generate videos by using PIA, VideoComposer [39] and
AnimateDiff [11, 33] with ControlNet [44] and IP-Adapter [43]. We ask the
participants to choose from the three generated videos with the best image
alignment or text alignment in each question. We show an illustration of the
question cases in Fig. 12. There are 20 questions in total, and the order of
options is shuffled. A total of 20 participants were involved in the survey.
Following the previous work [4], we calculated the preference rate, and the
results are shown in the main paper.

Figure 12: User Study. Example of user study questionnaires.

##  9 Ablation

In this section, we introduce more ablation studies to verify the
effectiveness of the inter-frame affinity and the fine-tuning of temporal
alignment layers.

###  9.1 Inter-frame Affinity

To further verify the effectiveness of inter-frame affinity, we train a model
without affinity hints for ablation study. We remove the affinity channel from
the input of the condition module and the result is shown in Fig. 13. Compared
to our method, videos generated by the model without inter-frame affinity are
more incoherent and change suddenly.

Figure 13: Ablation study for Inter-frame Affinity. Without an affinity hint,
the generated videos become incoherent and may change significantly after the
first given frame. With the inter-frame affinity as inputs, PIA is able to
animate images that are faithful to the condition frame.

###  9.2 Fine-tuning Temporal Alignment Layers

In the training stage, we train both the condition module and temporal
alignment layers. We now show the result of only training the condition module
with temporal alignment layers frozen in Fig. 14. The result proves that the
frozen temporary alignment layers failed to align the condition frame with
other frames.

Figure 14: Ablation study for fine-tuning the Temporal Alignment Layers. Pre-
trained temporal alignment layers fail to align the condition frame in
generated videos. PIA fine-tunes both the condition module and the temporal
alignment layers, leading to better preservation of the information in the
condition frames.

##  10 Visualization Results

###  10.1 Visualization of Attention Map

To demonstrate that the motion alignment of PIA is better than other methods,
we visualize the average cross attention map of motion descriptor token. We
use prompt ‘the rabbit on is on fire’ as an example and visualize the cross
attention map corresponding to token ‘fire’, as shown in Fig. 15. We can
observe that in our method, the region attended by the ‘fire’ matches the
region of flames. In contrast, the motion descriptor token in the baseline
method randomly attends to the entire context and cannot form a meaningful
structure. This phenomenon demonstrates that our method exhibits better motion
alignment performance.

Figure 15:  Visualization of Cross-attention map. We generate the video using
prompt ‘a rabbit is on fire’ and visualize the cross-attention map
corresponding to the token ‘fire’ for both AnimateDiff [11] and our own
method. In PIA, token ‘fire’ shows more accurate attention to the shape of
flames, while in AnimateDiff, the token randomly attends to the entire
context. This demonstrates the superior motion alignment performance of our
method.  Figure 16: Using PIA to animate open-domain images.Without providing
personalized T2I models, PIA is able to animate the open-domain images with
realistic motions by text while preserving the details and identity in
condition frame with IP-Adapter[43]

###  10.2 PIA for Open-Domain Images

In this section, we further explore animating open-domain images with PIA
without using personalized T2I models. To further enhance the preservation of
the information and details of the condition frame, we combine the Image
Prompt Adapter (IP-Adapter) [43] with PIA. Specifically, we use a CLIP image
encoder to extract features from the input images. Then, these image features
are incorporated into each frame through the cross-attention mechanism in the
UNet. As shown in Fig. 16, without using personalized models, our model
successfully animates an open-domain image with realistic motion by text while
preserving the identity of the given image.


[a]Ben Page

# Novel Bottomonium Results

Chris Allton  Seyong Kim

###### Abstract

We present the latest results from the use of the Backus-Gilbert method for
reconstructing the spectra of NRQCD bottomonium mesons using anisotropic
FASTSUM ensembles at non-zero temperature. We focus in particular on results
from the $\eta_{b}$, $\Upsilon$, $\chi_{b1}$ and $h_{b}$ generated from
Tikhonov-regularized Backus-Gilbert coefficient sets. We extend previous work
on the Laplace shifting theorem as a means of resolution improvement and
present new results from its use. We conclude with a discussion of the
limitations of the improvement routine and elucidate a connection with Parisi-
Lepage statistical scaling.

##  1 Introduction

Bottomonium has previously been studied as a means of estimating the
properties of the quark-gluon plasma as produced in relativistic heavy-ion
collisions. [1, 2, 3]. The problem of reconstructing the spectrum of
bottomonium states at non-zero temperature is an ill-posed one due to the
presence of finite uncertainties in data collected from lattice QCD
simulations. This work represents one of the latest in a collection of studies
performed by the Fastsum Collaboration focussing on bottomonium states
simulated using anisotropic lattices at nonzero temperature. In the following,
we build upon the results of previous work [4] and present a discussion on the
use of the Backus-Gilbert method for reconstructing bottomonium spectra,
showing how the Laplacian nature of the reconstruction formula my be exploited
to give improved resolution.

##  2 Lattice details

Non-relativistic QCD (NRQCD) is an effective field theory for heavy quarkonia,
which approximates fully relativistic QCD by expanding the Lagrangian in
powers of the heavy quark velocity [5]. One of the principal benefits of the
NRQCD formulation is that the calculation of the time evolution of the heavy
quarks reduces to an initial-value problem, as the heavy quarks and antiquarks
decouple in the non-relativistic regime. This decoupling effect turns the
spectral representation of the Euclidean correlator $G(\tau)$ into a Laplace
transformation of the spectral density function $\rho(\omega)$:

$G(\tau;T)=\int_{\omega_{\text{min}}}^{\omega_{\text{max}}}\frac{d\omega}{2\pi}K(\tau,\omega)\rho(\omega;T)$
(1)

where $K(\tau,\omega)=e^{-\omega\tau}$ is the temperature independent kernel
function and $T=(a_{\tau}N_{\tau})^{-1}$ is the lattice temperature as a
function of the temporal extent $N_{\tau}$. In order to relate NRQCD energies
to physical energies, the reconstruction window
$\omega\in[\omega_{\text{min}},\omega_{\text{max}}]$ must be additively
renormalised by the NRQCD energy shift, $\Delta E=7.46$GeV.

We make use of Fastsum’s Generation 2L ensembles, generated using anisotropic
lattices ($\xi=a_{s}/a_{\tau}\sim 3.5$) with 2+1 flavour, clover-improved
Wilson fermions using a physical $s$ quark and lighter, degenerate $u$ and $d$
quarks[6]. The spatial extent of the lattice $N_{s}=32$ and the temporal
extent along with the corresponding lattice temperatures are detailed in Table
1.

$N_{\tau}$ | 128 | 64 | 56 | 48 | 40 | 36 | 32 | 28 | 24 | 20
---|---|---|---|---|---|---|---|---|---|---
$T=1/(a_{\tau}N_{\tau})$ [MeV] | 47 | 95 | 109 | 127 | 152 | 169 | 190 | 217 | 253 | 304
$N_{\rm cfg}$ | 1024 | 1041 | 1042 | 1123 | 1102 | 1119 | 1090 | 1031 | 1016 | 1030
Table 1: Temporal extent, corresponding lattice temperature in MeV and number
of configurations for the Fastsum Generation 2L ensembles. The double vertical
line mid-table represents our value of $T_{\rm pc}$[6].

##  3 The Backus-Gilbert Method

The Backus-Gilbert method [7] is a reconstruction technique which extracts
regularised solutions from the ill-posed inverse problem by imposing
constraints on the stability of its predictions under a change of input. Since
$G(\tau)$ is only known to at most $\mathcal{O}(100)$ but $\rho(\omega)$ is
continuous ($\mathcal{O}(1000+)$ points), there are theoretically an infinite
number of possible spectra $\rho$ which produce the correct $G(\tau)$ within
numerical errors. Backus-Gilbert attempts to estimate a solution of Eq. 1,
denoted $\hat{\rho}$, on a point-by-point basis by constructing averaging
functions $A(\omega,\omega_{0})$ centred about some point $\omega_{0}$
generated using the data kernel $K(\tau,\omega)$:

$\hat{\rho}({\omega_{0}})=\int^{\omega_{\text{max}}}_{\omega_{\text{min}}}A(\omega,\omega_{0})\rho(\omega)~{}d\omega$
(2)

with $A(\omega,\omega_{0})=\sum_{\tau}c_{\tau}(\omega_{0})K(\tau,\omega)$. In
the limit $A(\omega,\omega_{0})\longrightarrow\delta(\omega-\omega_{0})$, we
obtain a perfect reconstrucion of the target spectrum $\rho$. It is easily
seen that plugging Eq. 1 into

$\hat{\rho}(\omega_{0})=\sum_{\tau}c_{\tau}(\omega_{0})G(\tau).$ (3)

gives Eq. 2. The coefficients $c_{\tau}(\omega_{0})$ control the shape of
$A(\omega,\omega_{0})$ and are found by minimising the cost function
$J(\omega_{0})$ representing the least-squares distance between the averaging
function and the delta function at $\omega_{0}$[8]:

$J(\omega_{0})=\int_{\omega_{\text{min}}}^{\omega_{\text{max}}}\left[A(\omega,\omega_{0})-\delta(\omega-\omega_{0})\right]^{2}~{}d\omega.$
(4)

Setting $\partial_{c_{\tau}}J(\omega_{0})=0$ reduces the problem to an
inversion of a matrix-vector product:

$\mathcal{K}_{\tau\tau^{\prime}}\cdot
c_{\tau^{\prime}}(\omega_{0})=K(\omega_{0},\tau),\qquad\text{where}\quad\mathcal{K}_{\tau\tau^{\prime}}=\int_{\omega_{\text{min}}}^{\omega_{\text{max}}}K(\tau,\omega)K(\tau^{\prime},\omega)d\omega.$
(5)

The kernel width matrix $\mathcal{K}$ is near-singular (worsened by the
exponential nature of the kernel) and so must be treated by a conditioning
routine before inversion. One such conditioning routine is the addition of a
small constant term to the diagonal entries in a Tikhonov-like fashion[9]:

$\mathcal{K}(\alpha)=\mathcal{K}+\alpha I$ (6)

where $\alpha$ is a parameter which controls the strength of the
regularisation. The benefit of such scalar conditioning over other methods is
that the coefficients $c_{\tau}$ are constructed without prior knowledge of
$G(\tau)$, enabling their use in the reconstruction of any spectra obeying Eq.
1, regardless of choices of quantum numbers, see [4].

##  4 Improvement via the Laplace Shift Transform

It has been previously shown [10] that, for the case of the maximum entropy
method (MEM), there exists a relationship between the choice of
$\omega_{\text{min}}$ and the resolving power of the method. This effect also
occurs in the Backus-Gilbert method which shares a similar basis-function
mechanism of reconstruction as the MEM. The application of a Laplace shift
transform:

$G^{\prime}(\tau)=e^{\Delta\cdot\tau}G(\tau)\overset{\mathcal{L}}{\Longrightarrow}\rho^{\prime}(\omega)=\rho(\omega+\Delta),$
(7)

where $\Delta>0$ shifts the spectral features closer to $\omega_{\text{min}}$
where the resolving power of the method is improved [4] offering improved
predictions for mass and width estimates. Building upon this feature, we have
opted to combine $\omega_{\text{min}}$ and the Laplace shift transform
parameter $\Delta$ into a single parameter
$\widetilde{\Delta}=\omega_{\text{min}}+\Delta$.

##  5 Systematic analysis: Removing $\widetilde{\Delta}$ and $\alpha$
dependence

There is a systematic dependence of the ground state mass $M$ and width
$\Gamma$ on the value of $\widetilde{\Delta}$ and $\alpha$ which must be
removed during analysis. This is illustrated in Fig. 1 where the mass (Left)
and width (Right) are shown as a function of $\widetilde{\Delta}$ for various
$\alpha$ values.

As can be seen, for fixed $\alpha$, the mass increases and width decreases
monotonically with $\widetilde{\Delta}$ due to the resolution improvement. We
also note that in the mass case, this slope approaches zero as
$\alpha\rightarrow 0$, indicating that the $\widetilde{\Delta}$ dependence
falls away. Finally, we note that the maximum theoretical shift occurs when
$\widetilde{\Delta}$ equals the mass. This is indicated by the boundary of the
hatched region in Fig. 1 (Left)). We therefore obtain our mass estimate by
firstly performing a linear extrapolation of the mass, at fixed $\alpha$, to
the boundary of the hatched region, obtaining $M(\alpha)$. The $\alpha$
dependence is then removed by noting empirically that $M(\alpha)$ becomes
independent of $\alpha$, for small $\alpha$, and so can be fit with a
constant. A bootstrap analysis produces the error estimate.

The ground state width is determined by a similar procedure. First a linear
fit in $\widetilde{\Delta}$ is performed, at fixed $\alpha$, and extrapolated
to the boundary region. The $\alpha$ dependence is then removed in the same
manner as in the mass case. However we note that, due to the finite resolving
power of the Backus Gilbert method, the widths obtained should be considered
upper bounds on the physical width of the state.

Figure 1: _Left:_ Plot of the ground state mass versus the shift parameter
$\widetilde{\Delta}$ for the (smeared) $\Upsilon$ meson over a range of
$\alpha$ values. The hatched region represents the maximum possible shift,
beyond which the ground state feature falls outside of the sampling window.
The red dashed line is the PDG estimate [11] and the black dashed line
represents our best estimate after extrapolating the $\widetilde{\Delta}$ and
$\alpha$ hyper-parameters. _Right_ : Plot of our estimate of the upper bound
for the ground state FWHM width versus $\widetilde{\Delta}$ for the $\Upsilon$
meson over a range of $\alpha$ values. The black dashed line is our best
estimate of the upper bound and the red band denotes its uncertainty. The
temperature used in these plots is $T=47$ MeV.

The value of $\omega_{\text{min}}$ is held fixed in this analysis (at a value
of $\omega_{\text{min}}=-0.1a_{\tau}^{-1}\approx 6.8$GeV). We have chosen to
include all possible Euclidean times in our analysis, i.e.
$0\leq\tau<N_{\tau}$, because the resolving power of Backus Gilbert method is
greatest for the largest time windows.

##  6 Results for the bottomonium sector: $\eta_{b}$, $\Upsilon$, $\chi_{b1}$
and $h_{b}$

Figure 2: Comparison between new results from this work and those from [4],
labelled (2021), for the mass (_left_) and upper bound on the width (_right_).
The magenta line on the left pane is the PDG estimate for the $\Upsilon$ mass
[11]. The red shaded band on the right pane indicates our estimate of the
maximum resolving power of the new method.

We have improved our previous results [4]) by including the Laplace shifting
(see §4) and an improved fitting analysis (see §5). In Fig. 2, we display our
results for the $\Upsilon$ mass and width as a function of temperature,
including the results from our earlier analysis. We have also extended our
previous work by including the $\eta_{b}$, $\chi_{b1}$ and $h_{b}$ states,
with results shown in Figs. 3 and 4. For this analysis, we restricted
ourselves to data generated using smeared quark sources which have improved
overlap with the ground state over local sources. We also conducted a more
rigorous study of the dependence of the mass and width on the parameters
$\alpha$ and $\widetilde{\Delta}$ in an attempt to measure their contribution
to the systematic error.

In particular, in the case of the $\Upsilon$ mass, we wish to highlight in
Fig.2 the comparison between the results of this work and the estimates
presented in [4] which appear to be contaminated by systematics of the method,
as the dependence of the the mass and width on $\alpha$ was not fully
accounted for. We also note that below the pseudocritical temperature ($T_{\rm
pc}=162~{}$MeV for our Gen-2L ensembles[6, 12]), the experimental widths for
the $\Upsilon$ and $\eta_{b}$ are $54.02\pm 1.25$ keV and
$10$$\begin{subarray}{c}+4\\\ -5\end{subarray}$ MeV respectively[11], over an
order of magnitude smaller than even the minimum resolvable width for our
method (see Fig. 4). We once again reiterate that our presented values for the
ground state width represent an upper bound on the true value, and we leave an
investigation into the effect of changing the Euclidean time extent and the
lower bound $\omega_{\text{min}}$ to a future study.

Figure 3: Plots showing the estimate of the mass versus lattice temperature
for select bottomonium states. The horizontal dashed line represents the PDG
estimate for the given state [11]. Figure 4: Plots showing the estimate of the
upper bound on the width versus lattice temperature for select bottomonium
states. The shaded region represents our best estimate of the resolving power
of the method. Figure 5: Value of the energy shift $\Delta_{\text{sing}}$
(i.e. the predicted mass) which gives the most singular shifted covariance
matrix (see Eq. 9) for a variety of bottomonium channels as a function of
$1/\tau_{2}$. The covariance matrices are defined over the time interval
$0\leq\tau<\tau_{2}$, and therefore the best results are obtained as
$\tau_{2}\rightarrow\infty$. The lattice temperature is 47 MeV. The predicted
masses each meson tends toward the experimental estimate for the pseudoscalar
mass. Experimental values for the meson masses are shown as horizontal lines
[11].

##  7 Connection with Parisi-Lepage Statistical Scaling

To estimate the error in the resulting reconstruction, the uncertainty in the
Euclidean correlator $\Delta G(\tau)$ must be combined with the Backus Gilbert
coefficients $c_{\tau}$. The uncertainty corresponding to Eq. 3 is simply

$\Delta\hat{\rho}^{2}=\sum_{\tau,\tau^{\prime}}c_{\tau}\text{Cov}[G]_{\tau,\tau^{\prime}}c_{\tau^{\prime}}$
(8)

where $\text{Cov}[G]$ is the covariance in $G(\tau)$. Under the Laplace
transformation outlined in Eq. 7, the covariance matrix transforms as

$\text{Cov}[G;\Delta]_{\tau,\tau^{\prime}}=e^{\Delta\cdot\tau}\text{Cov}[G]_{\tau\tau^{\prime}}e^{\Delta\cdot\tau^{\prime}}$
(9)

which in turn influences the spectral error $\Delta\hat{\rho}$. This effect
can be probed by measuring the condition number of the resulting matrix,
defined by

$\kappa\left(\text{Cov}[G;\Delta]\right)=\dfrac{\sigma_{\text{max}}}{\sigma_{\text{min}}}$
(10)

where $\sigma$ are the singular values of the matrix.

It is interesting to study $\kappa$ as a function of $\Delta$ and determine
the value, $\Delta_{\text{sing}}$ when $\text{Cov}[G;\Delta]$ becomes
singular. One may imagine that $\Delta_{\text{sing}}$ is the ground state mass
of the $G(\tau)$ channel. However, as pointed out by Parisi [13] and
elucidated further by Lepage [14], the covariance matrix has a special
physical significance. It can be expressed as a correlation function of the
square of the interpolating operators of the original correlation function,
$G(\tau)$. Analysing this further, one finds that the lightest state which
contributes to the covariance matrix is the pseudoscalar, no matter what state
was being probed by $G(\tau)$. This therefore implies that
$\text{Cov}[G;\Delta]$ becomes singular when $\Delta=\Delta_{\text{sing}}$ is
the pseudoscalar mass (i.e. the $\eta_{b}$ mass in our case) independent of
the channel.

We illustrate this in Fig. 5 where $\Delta_{\text{sing}}$ is plotted for a
variety of channels. The covariance matrix was defined over the time interval
$0\leq\tau<\tau_{2}$ meaning that the large time limit (where the ground state
dominates) is obtained as $\tau_{2}\rightarrow\infty$. As can be seen, in this
limit we recover the $\eta_{b}$ (i.e. pseudoscalar) mass, thereby confirming
the prediction of [13, 14].

##  8 Summary

We have presented results for the ground state mass and an estimate for the
upper bound on the width for several bottomonium states using smeared quark
sources, showing improved resolution compared to our previous results. We have
demonstrated the ability of the Laplace shift to naively increase the
resolving power of the method, but show that is still insufficient to resolve
the ground state widths of the system. The effect of the Laplace shift
transform on the covariance matrix of the Euclidean correlation function was
also studied, where the condition number of the matrix was found to confirm
Parisi-Lepage statistical scaling in the long-time limit.

## Acknowledgments

This work is supported by STFC grant ST/T000813/1. SK is supported by the
National Research Foundation of Korea under grant NRF-2021R1A2C1092701 and
grant NRF-2021K1A3A1A16096820, funded by the Korean government (MEST). BP has
been supported by a Swansea University Research Excellence Scholarship
(SURES). This work used the DiRAC Extreme Scaling service at the University of
Edinburgh, operated by the Edinburgh Parallel Computing Centre and the DiRAC
Data Intensive service operated by the University of Leicester IT Services on
behalf of the STFC DiRAC HPC Facility (www.dirac.ac.uk). This equipment was
funded by BEIS capital funding via STFC capital grants ST/R00238X/1,
ST/K000373/1 and ST/R002363/1 and STFC DiRAC Operations grants ST/R001006/1
and ST/R001014/1. DiRAC is part of the UK National e-Infrastructure. This work
was performed using PRACE resources at Cineca (Italy), CEA (France) and
Stuttgart (Germany) via grants 2015133079, 2018194714, 2019214714 and
2020214714. We acknowledge the support of the Swansea Academy for Advanced
Computing, the Supercomputing Wales project, which is part-funded by the
European Regional Development Fund (ERDF) via Welsh Government, and the
University of Southern Denmark and ICHEC, Ireland for use of computing
facilities. We are grateful to the Hadron Spectrum Collaboration for the use
of their zero temperature ensemble.

## References

  * [1] G. Aarts, S. Kim, M. P. Lombardo, M. B. Oktay, S. M. Ryan, D. K. Sinclair et al.__,arXiv:1010.3725 [hep-lat, physics:hep-ph, physics:nucl-th].
  * [2] G. Aarts, C. Allton, S. Kim, M. P. Lombardo, M. B. Oktay, S. M. Ryan et al.__,arXiv:1109.4496 [hep-lat, physics:hep-ph, physics:nucl-th].
  * [3] G. Aarts, C. Allton, T. Harris, S. Kim, M. P. Lombardo, S. M. Ryan et al.__,arXiv:1402.6210 [hep-lat, physics:hep-ph].
  * [4] B. Page, G. Aarts, C. Allton, B. Jäger, S. Kim, M. P. Lombardo et al.__,arXiv:2112.02075 [hep-lat].
  * [5] G. Lepage __, in _Nuclear Physics B - Proceedings Supplements_ , vol. 26, pp. 45–56.  DOI.
  * [6] G. Aarts, C. Allton, J. Glesaaen, S. Hands, B. Jäger, S. Kim et al.__,arXiv:2007.04188 [hep-lat, physics:hep-ph, physics:nucl-th].
  * [7] G. Backus and F. Gilbert, _The Resolving Power of Gross Earth Data_ , vol. 16.  10.1111/j.1365-246X.1968.tb00216.x.
  * [8] D. W. Oldenburg, _An introduction to linear inverse theory_ , vol. GE-22.  10.1109/TGRS.1984.6499187.
  * [9] A. N. Tikhonov __, in _On the Stability of Inverse Problems_ , vol. 39, pp. 195–198.
  * [10] A. Rothkopf __,arXiv:1208.5162 [hep-lat, physics:nucl-th, physics:physics].
  * [11] Particle Data Group, R. L. Workman, V. D. Burkert, V. Crede, E. Klempt, U. Thoma et al.__, .
  * [12] G. Aarts, C. Allton, R. Bignell, T. J. Burns, S. C. García-Mascaraque, S. Hands et al.__,arXiv:2209.14681 [hep-lat, physics:nucl-th].
  * [13] G. Parisi, _The strategy for computing the hadronic mass spectrum_ , vol. 103.  10.1016/0370-1573(84)90081-4.
  * [14] G. P. Lepage __, in _The Analysis of Algorithms for Lattice Field Theory_ , pp. 97–120.