# Optimizing Intensive Database Tasks Through Caching Proxy Mechanisms

Ionuț-Alex Moise1 and Alexandra Băicoianu2
1Faculty of Mathematics and Computer Science, Transilvania University of
Brașov, Brașov, Romania
2Faculty of Mathematics and Computer Science, Department of Mathematics and
Computer Science, Transilvania University of Brașov, Brașov, Romania
<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://orcid.org/0000-0002-1264-3404

###### Abstract

Web caching is essential for the World Wide Web, saving processing power,
bandwidth, and reducing latency. Many proxy caching solutions focus on
buffering data from the main server, neglecting cacheable information meant
for server writes. Existing systems addressing this issue are often intrusive,
requiring modifications to the main application for integration. We identify
opportunities for enhancement in conventional caching proxies. This paper
explores, designs, and implements a potential prototype for such an
application. Our focus is on harnessing a faster bulk-data-write approach
compared to single-data-write within the context of relational databases. If a
(upload) request matches a specified cacheable URL, then the data will be
extracted and buffered on the local disk for later bulk-write. In contrast
with already existing caching proxies, Squid for example, in a similar
uploading scenario, the request would simply get redirected, leaving out
potentially gains such us minimized processing power, lower server load and
bandwidth. After prototyping and testing the suggested application against
Squid, concerning data uploads with $1,100,1.000,\ldots,100.000$ requests, we
consistently observed query execution improvements ranging from 5 to 9 times.
This enhancement was achieved through buffering and bulk-writing the data, the
extent of which depended on the specific test conditions.

##  1 Introduction

The wide use of the internet by people around the world has posed scalability
challenges for many businesses and service providers [Datta et al., 2003].
Long response times or even inaccessibility is a factor that affects the
revenues of web-centric companies, leading to lower earnings [Wessels, 2001],
[Datta et al., 2003]. Web caches have been shown to solve some of the
scalability problems. They helped bring down latencies, bandwidth usage and
save processing power [Barish and Obraczka, 2000], [Wessels, 2001], [Datta et
al., 2003].

There are generally a few widely used approaches to caching the data: browser
cache, proxy cache and server cache [Barish and Obraczka, 2000], [Wessels,
2001], [Ali et al., 2011], [Zulfa et al., 2020]. The browser cache is the
closest one to the user. It can save, in the memory of the local computer,
static data like images, videos, CSS and JS code, etc. [Datta et al., 2003]. A
proxy cache is a dedicated server that sits between one or more clients and
one or more servers. Compared to the browser cache, which is tied to a single
machine, a proxy cache can be placed anywhere on the web, at different levels:
ISP (local, regional, national) or right in front of the primary server [Datta
et al., 2003]. Lastly, there is the option to cache your data on the computer
that is running the web server/database, either by using your own/a third-
party solution or indirectly through the caching system of your operating
system or database.

Research in the field primarily targets cache replacement algorithms and
prefetching. Crucially, caching solutions must decide what objects to retain
and which to evict due to limited memory space. Managing the resources
incorrectly and keeping unused objects cached for long enough, results in
what’s known as cache pollution [Ali et al., 2011], [Mertz and Nunes, 2017].
Some of the most popular caching policies include: LFU (least frequently
used), LRU (least recently used), GDS (greedy dual size), GDS-Frequency and
many more [Ali et al., 2011], [Zulfa et al., 2020], [Ioannou and Weber, 2016],
some of them also using machine learning to enhance the already used ones [Ali
et al., 2012].

The effectiveness of caching is typically measured in hit rate or byte hit
rate. Hit rates are determined as the percent of requests that could be
satisfied directly by the cache, while byte hit rate represents the percent of
the data (numbered in bytes) that were already cached before answering the
request [Nanda et al., 2015], [Dilley et al., 1999].

Cache prefetching is a technique that aims to request and cache objects before
they are needed. Although useful, the solution must be carefully implemented
to avoid polluting the cache in an excessive way [Ali et al., 2011], [Mertz
and Nunes, 2017], [Seshadri et al., 2015]. In their studies, [Ali et al.,
2011] showed that based on surveys, web caching along prefetching can reduce
the latency of responses by up to 60%, compared to web caching along, yielding
an improvement of only 26% in latency. Likewise, [Barish and Obraczka, 2000]
published a survey where we can find similar results, incorporating
prefetching resulting in a 41% or even 57% latency improvement.

The majority of the developed solutions for web caching are dedicated towards
storing data generated by a server. Few of them offer a solution for buffering
the incoming information that is meant to be stored by the SQL/NoSQL database.
Indeed, Redis, Memcached, Apache Kafka, RabbitMQ and others offer the
possibility to achieve this goal, but it is done intrusively, meaning that the
underlying main server needs to suffer modifications to accommodate these
solutions. In the following sections, we will introduce a system and
architecture specifically engineered to cache incoming (uploaded) and fetched
data, requiring minimal to no modifications to the core application.

##  2 Materials and Methods

Prior to delving into the construction and testing of the proposed solution,
it is imperative to provide an overview of its counterpart. This would be
Squid [Squid-Cache, 2024], a proxy server solution that is most often used as
a caching proxy. It is a popular application used for caching and managing
both static and dynamic content generated as response by a web server for a
user’s request, supporting protocols such as HTTP, HTTPS, FTP and more.

In order to save bandwidth, speed up load times, and conserve computing power,
hundreds of Internet providers employ Squid in addition to thousands of
standalone websites, as stated in [Squid-Cache, 2024].

Squid is a battle-hardened application. It provides a powerful configuration
file with the ability to create very complex distributed caching
infrastructures. It can deploy on multiple computers and create a caching
hierarchy consisting of Parents, Kids and Coordinators, all communicating with
each other for better buffering and cache management to save bandwidth,
processing power and lower latencies. Besides that, Squid also offers
administrators the possibility to configure both Memory and Disk caches
independently, each with its own replacement policy like: LRU, heap GDSF, heap
LFUDA, heap LRU.

Trying to match the power that Squid provides would be a very tedious and long
process. Thus, we will try to optimize only a small chunk of it. Our focus
falls on how Squid handles requests that upload data. As of now, Squid will
simply redirect those to the main server. This approach may be improved.
Instead of redirecting the request, we could extract the data (if its URL is
marked as cacheable in the configuration file) and store it locally into a
buffer till the caching time expires, sending it all at once afterward. This
way, if the data is meant to be written in a relational database, the time it
takes to execute for a single, multiple rows insertion query is far lower than
overall multiple single rows insertions. The following sections will explore
the concept further and present a viable solution.

Figure 1: RcSys flow.

##  3 The Proposed System Architecture

The proposed application, referred to as RcSys (Resource Caching System),
functions as a caching proxy for handling both user upload and download
requests. Prior to a detailed technical examination of the system’s
architecture, a general schematic overview of its operational flow will be
presented.

The diagram 1 illustrates three primary actors: clients, the RcSys server, and
the main server. All communication between these components occurs through
HTTP(S).

The interaction commences as a client initiates a request over the internet,
either for uploading (e.g., sending emails, creating posts) or downloading
(e.g., reading messages, shopping). Upon reaching RcSys, a rapid evaluation
occurs to determine if the requested resource should be cached. This decision
relies on details outlined in a configuration file created by the
administrator, specifying the paths designated for caching.

If the requested URL is not known, then it gets redirected to the main server.
In case it is known and is of type upload, it will be stored in a buffer and
later, when the specified amount of caching time expires, it will be written
to the main server (along with the other data that share the same URL). If the
accessed resource is of type download, the system first checks whether it
exists or is expired (if so, it makes a call for the updated version to the
main server) and then replies to the client.

###  3.1 Multi-Thread Request Processing

Regarding the architecture of RcSys, a notable technical aspect involves
request processing. It employs a dedicated Server thread for managing the web
server and accepting connections, along with a pool of Worker threads
controlled by a master thread. Upon a new connection, it becomes a task in a
shared queue between Server and Worker. The master thread retrieves and
assigns tasks to workers. Simultaneously, new connections can be established
and added to the queue.

The Figure 2 offers a more detailed view of the system’s functionality. Upon
the application’s initiation, a configuration file undergoes processing, and
its supplied information is stored within an IConfiguration object for
convenient access. This file encompasses various details, including the
desired thread pool size. Concurrently, the creation of the Worker results in
the instantiation of X threads, as dictated by the configuration.

This architectural decision is predicated on a challenge encountered during
the design phase of the proxy caching system. To illustrate, consider a
scenario where a singular Worker thread manages user requests. In a context
where numerous resources are cached, each with its specific expiration time
ranging from seconds to hours, and a substantial number of these resources
total in the hundreds. For instance, if there are 1000 requests awaiting
processing, some requiring the update of cached data, the consequence is that
over 900 requests must wait for the arrival of new information, irrespective
of whether it pertains to their specific data or not. In contrast, utilizing a
thread pool, the proposed architecture enables concurrent processing of new
requests by the CPU while the operating system monitors the arrival of
necessary resources from the main server to fulfill other pending tasks.

The requests are distributed for processing among the worker threads in a
Round-robin fashion [Balharith and Alhaidari, 2019]. An index of the thread
that is next for receiving a new task is kept in the Master Worker state. If a
new request arrives, the thread that is pointed at by the index will be
assigned to serve it and the index is incremented, therefor the next one will
be handled by a different worker.

A second solution for solving the earlier mentioned problem would be to build
an event-loop based architecture. However, such a resolve imposes new
challenges and a harder to comprehend implementation, even if it is used a
library like libev[libev, 2024]. The more natural way is to use threads,
performing as good as event-loop programming and easier to execute [von Behren
et al., 2003].

Figure 2: RcSys thread pool.

###  3.2 One-Tiered Cache Replacement Policy

The main mean by which RcSys stores the cached data is disk, be it HDD or SSD.
Although it may not be an option as faster to access and retrieve information
as RAM, it has its own advantages and the fact that it is “disk-only” is not
entirely true.

Caching data into RAM comes with a lot of careful managing and designing. If
your process ends up leaking memory or not allocating/deallocating it
efficiently, it can use all computer’s memory and slow the entire system
performance or even crash. RAM is also a much smaller sized resource on
commodity computers that can be used as servers, being significantly more
expensive than the disk. A $16-32-64-128\ldots GB$ of RAM machine could also
store way less cached data compared to a 2-4TB non-volatile memory option.

Despite its inherent drawbacks, the decision to utilize the disk as a storage
medium is justified by the support provided by the operating system in
managing file access. Therefore, the characterization of it being a ”disk-only
solution” is not entirely accurate. In order to facilitate access to memory
for software applications, the OS allocates pages of memory. Those pages
represent virtual memory addresses that are later translated to physical
addresses when a request to the memory controller is made to get the stored
data. The OS also uses the main memory of a computer to load accessed disk
files into and minimize the IO operations that would be performed. It maps the
file opened for reading and writing from the disk to the RAM and takes care of
evicting them if the memory is full. This way, the user can manipulate the
file’s data and the system does not have to issue a write to disk every time a
new letter is typed. Instead, it marks the memory pages as “dirty” and updates
the file later. Managing resources this way increases the machines’ general
performance [The kernel development community. Sphinx 5.0.1 & Alabaster 0.7.12
,].

RcSys’s architecture takes advantage of the OS file caching behavior to
simplify the proposed solution and to also benefit from the performance gains
that come along writing and reading from RAM. A similar approach is
implemented by Nginx for caching, as detailed by one of their Senior Director
here [Garrett, 2016].

Disk’s higher memory availability does not mean that it is infinite. It might
as well get full if enough data requires caching. To deal with this problem, a
simple cache eviction policy was implemented, that would delete files from
disk according to LRU. The least recently accessed file for either reading or
writing will be erased to make space for a new one. The configuration file
also specifies the maximum size of disk memory that can be used for caching.
The architecture, however, does not guarantee that the quantum won’t be
exceeded. Instead, when a new request comes in, before processing it, we
evaluate whether the maximum cache size was exceeded. If so, cached resources
from the disk will get deleted in a LRU manner until the used space is below
the maximum one. This means that if the incoming request has to write new
files on disk, the memory limit could be again surpassed, till another request
arrives and the cycle repeats. The OS also implements a variation of the LRU
for deleting unused pages [Linux Kernel Organization,].

###  3.3 Use Cases

As a caching proxy, RcSys aligns with established use cases observed in
existing solutions. Its deployment serves purposes such as conserving
processing power and bandwidth for a web server. By caching non-real-time
critical content, RcSys alleviates the proxied server’s burden, including
items like blog posts, images, messages, entire web pages, etc. Moreover, it
enhances the user experience by positioning itself in proximity to the user,
handling requests that do not necessitate database queries or data
regeneration, thereby minimizing overhead.

An additional advantage that sets it apart from alternative solutions is its
capability to temporarily store data intended for server writes and
subsequently perform bulk writes. This feature substantially reduces the
processing load on the main server and database, further enhancing efficiency.

###  3.4 A Final Overview of System’s Flow

For providing a comprehensive overview and connecting all the components
comprising the architecture, a sequence diagram has been crafted, see Figure
3. In this diagram, we explain some of the most important objects’ states and
their role in RcSys, as well as showing the path that a user’s request takes
once in the system.

If we delve into the details, particularly in the context of caching incoming
data, the process unfolds as follows in 4.

Figure 3: RcSys flow chart. Figure 4: RcSys flow chart zoomed in on POST
cache. Figure 5: Overall performance gains RcSys vs Squid.

##  4 Strengths and Shortcomings of the Solution

In order to compare the overall performance of the proposed application, we
chose to test it against Squid-Cache [Squid-Cache, 2024], both being a caching
proxy. Before going forward, we must make a big disclaimer. RcSys is nowhere
near as powerful and well-rounded as squid, nor is it production ready. It is
just an experimental application built to explore new architectures and
capabilities that a caching proxy can bring.

The proxied application utilized in our research serves as a conceptual,
abstract representation—a pattern mirroring the structure typical of a
conventional web server. Developed in Golang, the choice of this language
stems from our desire for a swift prototype creation process, minimizing
unnecessary code verbosity.

Golang’s net/http [Golang, v122] handles the requests along Gorm [Gorm, 1257]
for interacting with a PostgreSQL database. The database’s schema is modest,
consisting of four entities (4c0fk, 4c2fk, 10c0fk, 10c2fk), modeled in such a
way as to cover some real-world operations. The names are self-explanatory,
each entity consisting of several string fields equal to the digits previous
to the ”c” character and some foreign keys equal to the digit previous to
“fk”. For example, entity 4c0fk has 4 string columns and 0 foreign keys, while
entity 10c2fk has 10 string columns and 2 foreign keys (mapped to entity 4c0fk
and entity 10c0fk, same as for entity 4c2fk).

We aimed to explore fundamental database use cases by conducting tests for
different results on tables of various sizes, ranging from small to large, and
considering scenarios with and without foreign keys. Foreign keys play an
important role in a relational database, impacting a query’s performance. For
each foreign key that the database must insert, it has to perform a look up in
the referenced table to make sure that the new created row is valid and does
not point to a non-existing record.

The application that simulates clients sending requests to the web server was
also built with Golang for the same fast-prototyping reason. It defines basic
functions for getting as well as posting data for several times in order to
create metrics and compare the proxies.

For the first set of tests, we simulated basic POST operations, with the
requests being proxied by both RcSys and Squid. As already mentioned, Squid
does not possess the ability to cache the uploaded data in order to send it
later to the proxied server all at once. Therefore, every such operation was
redirected to the main server. Both applications went through the same tests,
with the same data (different strings with the same size). For each entity
(4c0fk, 4c2fk, 10c0fk, 10c2fk), a client sent
$1,100,1000,5000,10000,25000,50000$ and 100000 POST HTTP requests to create a
new record in the database. The data was collected and analyzed for each batch
of tests $(1,100,\ldots)$. In order to obtain accurate results of the
execution time, PostgreSQL provides us with a query prefix EXPLAIN ANALYZE
[PostgreSQL Global Dev Group, v16b], [PostgreSQL Global Dev Group, v16a],
[PostgreSQL Global Dev Group, v16c]. Running a simple query like ”INSERT INTO
… VALUES …” and prefixing it with ”EXPLAIN ANALYZE =>EXPLAIN ANALYZE INSERT
INTO … VALUES … ” results in both executing the query and outputting different
real information about the execution process. From among the returned data, we
extracted two performance metrics, namely the ”Execution time” (the actual
time that it took to execute the query) and ”Planning time” (the actual time
that it took to plan the query execution), expressed in milliseconds. The
information that EXPLAIN ANALYZE provides differs for each type of query. For
example, if the inserted data links through foreign keys to other tables, a
trigger will be executed to check if the referenced row exists. This ”Trigger
time” is summed up in the ”Execution time” parameter.

Figures 5 compare the execution and planning time data for all mentioned
entities, the $X$ axis being the number of requests $1,100,1.000,5.000,\ldots
100.000$ and the $Y$ axis being the execution time corresponding to the $X$
number of requests. Observing these results, it becomes evident that RcSys
outperforms Squid significantly in handling upload requests. Saving multiple
rows of data is faster than saving only one at a time (one per query, as Squid
did) and significantly mitigates the overall inserting cost. Although RcSys
could buffer all the incoming upload requests in a span as long as the caching
expiration time, we chose to limit the amount of data to as much as 10.000
requests, after which the buffer would be emptied. MySQL’s documentation
[Oracle, v8] breaks down the cost of an INSERT statement in proportions as
follows:

  * •

Connecting (3)

  * •

Sending query to server (2)

  * •

Parsing query (2)

  * •

Inserting row (1 x size row)

  * •

Inserting indexes (1 x number of indexes)

  * •

Closing (1)

. Inserting multiple rows at once allowed avoiding the costs associated with
multiple connections, sending queries to the server, and closures. Examination
of the underlying data used in generating the four charts reveals the
corresponding tables, see Tables 1 and 2.

Table 1: Raw execution tests data RcSys vs Squid in ms. Entity | 1 | 100 | 1.000 | 10.000 | 50.000 | 100.000
---|---|---|---|---|---|---
4c0fk | RcSys | 0.055 | 1.831 | 9.64 | 48.106 | 254.99 | 514.69
Squid | 0.055 | 8.249 | 65.345 | 633.505 | 3174.3 | 6413.4
4c2fk | RcSys | 0.313 | 2.348 | 20.379 | 198.445 | 984.092 | 1956.51
Squid | 0.313 | 11.431 | 131.999 | 1314.13 | 6791.7 | 13395.5
10c0fk | RcSys | 0.299 | 0.758 | 6.368 | 59.859 | 296.108 | 578.355
Squid | 0.299 | 6.11 | 57.113 | 730.357 | 3446.2 | 6458.6
10c2fk | RcSys | 0.281 | 2.781 | 45.473 | 211.247 | 1058.82 | 2121.50
Squid | 0.281 | 13.23 | 142.361 | 1396.49 | 6979.6 | 13692.9
Table 2: Raw planning tests data RcSys vs Squid in ms. Entity | 1 | 100 | 1.000 | 10.000 | 50.000 | 100.000
---|---|---|---|---|---|---
4c0fk | RcSys | 0.019 | 0.101 | 0.318 | 4.621 | 17.75 | 41.99
Squid | 0.019 | 2.988 | 25.141 | 225.035 | 1112.1 | 2265.0
4c2fk | RcSys | 0.024 | 0.124 | 1.095 | 17.235 | 70.487 | 150.85
Squid | 0.024 | 2.639 | 30.298 | 311.137 | 1611.1 | 3182.5
10c0fk | RcSys | 0.026 | 0.052 | 0.32 | 7.581 | 26.919 | 52.237
Squid | 0.026 | 2.365 | 21.413 | 272.29 | 1280.3 | 2427.8
10c2fk | RcSys | 0.022 | 0.464 | 4.161 | 20.39 | 79.134 | 165.84
Squid | 0.022 | 3.134 | 33.755 | 335.182 | 1679.7 | 3310.1

Extracting additional information about the gains is possible from the raw
data, such as the average time improvement for each test, as indicated in
Table 3. Besides its advantage in caching upload requests, RcSys lacks many
features and critical functionalities that Squid has, see Table 4.

Table 3: RcSys vs Squid performance gains. Average query speedups | Test 4c0fk | Test 10c0fk | Test 4c2fk | Test 10c2fk
---|---|---|---|---
Execution Speedup | x9.12 | x9.48 | x5.93 | x5.20
Planning Speedup | x52.11 | x40.18 | x19.13 | x14.17
Table 4: RcSys vs Squid features comparison. Feature | RcSys | Squid
---|---|---
Security | No HTTPS (SSL/TLS) or Authentication | HTTPS and Authentication
Protocols supported | Only HTTP | HTTP, HTTPS, FTP and more
Distributed cache | No |  Yes, with complex hierarchy of parents, kids and coordinators
Cache replacement policies | LRU |  LRU, heap GDSF, heap LFUDA, heap LRU
Caching options | Only disk | Disk and Memory
Configuration | Minimal | Very robust

##  5 Conclusions

The paper has proposed an innovative enhancement designed to bring added value
to conventional caching proxy solutions. Through the implementation and
testing of the proposed architecture, it was evident that there is a
significant improvement opportunity in reducing bandwidth consumption and
optimizing data upload speeds, especially in contexts utilizing relational
databases. The strategy of locally extracting and storing data from upload-
type requests on the caching server’s disk or memory, followed by later bulk-
write, showcased marked enhancements in overall database write performance.
While typical web caching proxies would redirect those types of requests,
RcSys obtained by buffering them between 5.20 and 9.12 times SQL query
execution speedups and between 14.17 and 52.11 times SQL query planning
speedups.

## REFERENCES

  * Ali et al., 2011 Ali, W., Shamsuddin, S. M., and Ismail, A. S. (2011).  A survey of web caching and prefetching a survey of web caching and prefetching.  International Journal of Advances in Soft Computing and its Applications, 3.
  * Ali et al., 2012 Ali, W., Shamsuddin, S. M., and Ismail, A. S. (2012).  Intelligent web proxy caching approaches based on machine learning techniques.  Decision Support Systems, 53(3):565–579.
  * Balharith and Alhaidari, 2019 Balharith, T. and Alhaidari, F. (2019).  Round robin scheduling algorithm in cpu and cloud computing: A review.  In 2019 2nd International Conference on Computer Applications & Information Security (ICCAIS), pages 1–7.
  * Barish and Obraczka, 2000 Barish, G. and Obraczka, K. (2000).  World wide web caching: trends and techniques.  IEEE Communications Magazine, 38(5):178–184.
  * Datta et al., 2003 Datta, A., Dutta, K., Thomas, H., and VanderMeer, D. (2003).  World wide wait: A study of internet scalability and cache-based approaches to alleviate it.  Management Science, 49(10):1425–1444.
  * Dilley et al., 1999 Dilley, J., Arlitt, M., and Perret, S. (1999).  Abstract enhancement and validation of squid’s cache replacement policy.
  * Garrett, 2016 Garrett, O. (2016).  Cache Placement Strategies for NGINX and NGINX Plus.
  * Golang, v122 Golang (v1.22).  Documentation net/http.
  * Gorm, 1257 Gorm (v1.25.7).  The fantastic ORM library for Golang.
  * Ioannou and Weber, 2016 Ioannou, A. and Weber, S. (2016).  A survey of caching policies and forwarding mechanisms in information-centric networking.  IEEE Communications Surveys & Tutorials, 18(4):2847–2886.
  * libev, 2024 libev (2024).  Documentation libev.
  * Linux Kernel Organization,  Linux Kernel Organization.  Page Frame Reclamation.
  * Mertz and Nunes, 2017 Mertz, J. and Nunes, I. (2017).  Understanding application-level caching in web applications: A comprehensive introduction and survey of state-of-the-art approaches.  ACM Comput. Surv., 50(6).
  * Nanda et al., 2015 Nanda, P., Singh, S., and Saini, G. (2015).  A review of web caching techniques and caching algorithms for effective and improved caching.  International Journal of Computer Applications, 128:41–45.
  * Oracle, v8 Oracle (v8).  Optimizing INSERT Statements.
  * PostgreSQL Global Dev Group, v16a PostgreSQL Global Dev Group (v16a).  SQL ANALYZE.
  * PostgreSQL Global Dev Group, v16b PostgreSQL Global Dev Group (v16b).  SQL EXPLAIN.
  * PostgreSQL Global Dev Group, v16c PostgreSQL Global Dev Group (v16c).  Using EXPLAIN.
  * Seshadri et al., 2015 Seshadri, V., Yedkar, S., Xin, H., Mutlu, O., Gibbons, P. B., Kozuch, M. A., and Mowry, T. C. (2015).  Mitigating prefetcher-caused pollution using informed caching policies for prefetched blocks.  ACM Trans. Archit. Code Optim., 11(4).
  * Squid-Cache, 2024 Squid-Cache (2024).  Squid: Optimising Web Delivery.
  * The kernel development community. Sphinx 5.0.1 & Alabaster 0.7.12 ,  The kernel development community. Sphinx 5.0.1 & Alabaster 0.7.12 .  Page cache.
  * von Behren et al., 2003 von Behren, R., Condit, J., and Brewer, E. (2003).  Why events are a bad idea (for high-concurrency servers).  In Proceedings of the 9th Conference on Hot Topics in Operating Systems - Volume 9, HOTOS’03, page 4, USA. USENIX Association.
  * Wessels, 2001 Wessels, D. (2001).  Web Caching.  O’Reilly Series. O’Reilly & Associates.
  * Zulfa et al., 2020 Zulfa, M., Hartanto, R., and Permanasari, A. (2020).  Caching strategy for web application – a systematic literature review.  International Journal of Web Information Systems, 16:545–569.


# Disentangling Foreground and Background Motion for Enhanced Realism in Human
Video Generation

Jinlin Liu, Kai Yu, Mengyang Feng, Xiefan Guo, Miaomiao Cui
<EMAIL_ADDRESS>
https://liujl09.github.io/humanvideo_movingbackground/
corresponding author

###### Abstract

Recent advancements in human video synthesis have enabled the generation of
high-quality videos through the application of stable diffusion models.
However, existing methods predominantly concentrate on animating solely the
human element (the foreground) guided by pose information, while leaving the
background entirely static. Contrary to this, in authentic, high-quality
videos, backgrounds often dynamically adjust in harmony with foreground
movements, eschewing stagnancy. We introduce a technique that concurrently
learns both foreground and background dynamics by segregating their movements
using distinct motion representations. Human figures are animated leveraging
pose-based motion, capturing intricate actions. Conversely, for backgrounds,
we employ sparse tracking points to model motion, thereby reflecting the
natural interaction between foreground activity and environmental changes.
Training on real-world videos enhanced with this innovative motion depiction
approach, our model generates videos exhibiting coherent movement in both
foreground subjects and their surrounding contexts. To further extend video
generation to longer sequences without accumulating errors, we adopt a clip-
by-clip generation strategy, introducing global features at each step. To
ensure seamless continuity across these segments, we ingeniously link the
final frame of a produced clip with input noise to spawn the succeeding one,
maintaining narrative flow. Throughout the sequential generation process, we
infuse the feature representation of the initial reference image into the
network, effectively curtailing any cumulative color inconsistencies that may
otherwise arise. Empirical evaluations attest to the superiority of our method
in producing videos that exhibit harmonious interplay between foreground
actions and responsive background dynamics, surpassing prior methodologies in
this regard.

##  1 Introduction and Related Work

The objective of image animation is to transform static images into dynamic
videos, harnessing the original image’s content. A prominent application lies
in imbuing characters within input images with targeted actions, guided by
reference footage [19; 18; 20; 28; 30; 31; 23; 26; 9; 6]. Initial endeavors in
this domain relied upon meticulously crafted loss functions alongside pure
convolutional or transformer-based architectures to transpose the input image
into varying poses [19; 18; 20; 28; 30; 31].

Recently, however, diffusion models have emerged as a generative strategy in
works like Disco [23], Magicanimate [26], Dreampose [9], and Animateanyone
[6], capitalizing on pre-trained, high-performance models to yield outputs
with heightened resolution and finer detail. Disco introduces ControlNet for
background inpainting and dynamic foreground figure generation, while
Dreampose integrates detailed cross-attention features from reference images
and Densepose [3] for refined guidance. Despite their advancements, these
image-centric models encounter limitations in providing seamless frame-to-
frame transitions for video generation. Magicanimate and Animateanyone tackle
this issue by incorporating a temporal motion module [4], utilizing additional
U-Nets to capture intricate features for videos with meticulous detail.

Despite significant strides achieved in the past two years, we have identified
an inherent shortcoming in image animation research that has hitherto received
inadequate attention. The overwhelming majority of preceding efforts
concentrate exclusively on animating the human subject within input images,
transitioning it to various poses, while inadvertently overlooking the
background. This oversight results in generated videos with consistently
static background, a stark contrast to the reality observed in most online
video content, where backgrounds are typically in motion or exhibit dynamism.
Indeed, it is only in a minority of instances, typically those filmed with a
stationary camera, that a truly static background is encountered. To enhance
the authenticity of image-animated videos and bridge this gap, it becomes
imperative to transcend the confines of static background generation.
Incorporating dynamic backgrounds into synthesized videos is a pivotal step
towards creating animations that more faithfully mimic the richness and
variability of real-world footage, thereby elevating the overall realism and
immersive quality of the output.

This research presents a groundbreaking method that isolates the modeling of
foreground and background movement in video generation. We’ve developed an
ingenious motion depiction system that skillfully captures intricate human
actions along with environmental changes. Human activity is modeled using pose
data, while background motion is traced via sparse tracking points. This
separation allows our model to learn and create harmonious foreground-
background interactions after being trained on real-world videos. To
synthesize longer sequences without typical long-term errors, we employ a
segmented generation technique. Videos are produced in sections, with each new
clip informed by the final frame of the preceding one and fresh noise. To
maintain continuity, the base reference image’s features are continuously
infused into the network throughout clip generation, preventing color
inconsistencies from accruing as the video grows. In summary, our research
advances the field in two pivotal areas:

\- Our work pioneers a breakthrough in addressing and transcending the long-
standing limitation of static backgrounds prevalent in image animation. By
introducing a groundbreaking decoupling of foreground and background motion
representation, we present the inaugural method capable of synthesizing human
videos that not only feature naturalistic foreground actions but also
incorporate authentically dynamic backgrounds, thereby setting a new benchmark
for realism in generated video content.

\- We introduce an efficient pipeline designed to produce extended videos
devoid of error accumulation issues typically encountered in prolonged
sequences. This is achieved through a strategic combination of conditional
concatenation and the utilization of global feature extraction, enabling the
seamless generation of prolonged video clips with maintained content
consistency, thereby ensuring a cohesive and high-quality viewing experience
throughout.

##  2 Methodology

Figure 1:  Our proposed method features a pipeline that meticulously models
foreground motions with pose guidance and captures background dynamics via a
sparse tracking system. To tackle the challenge of video synthesis beyond
training sample lengths, we use a sequential strategy, generating video clips
that build on the last frame of the previous clip to maintain continuity. A
global feature vector derived from the initial image ensures visual coherence
across the iterative process, preventing color or context inconsistencies.
Additionally, a Temporal Motion Block ensures smooth transitions and temporal
consistency between frames, enhancing realism and fluidity in the generated
sequences.

The methodology’s workflow is illustrated in Figure 1, outlining our approach.
We adopt a dual strategy: foreground movement is modeled leveraging pose
direction, while background motion is handled through sparse tracking markers.
To synthesize videos exceeding the length of our training frame number, we
adopt a clip-by-clip generation technique. Here, the final frame of each
produced clip serves as the springboard for generating the subsequent one. To
uphold coherence throughout this sequential generation, the features extracted
from the initial reference image are utilized as a global anchor and are fed
into the core network structure. Furthermore, we integrate a Temporal Motion
Block, as introduced in [4], to guarantee seamless frame-to-frame transitions
and overall video smoothness.

Our network’s fundamental architecture is rooted in Latent Diffusion Models
(LDMs) [15], integrating a Variational Autoencoder (VAE) [22] for encoding and
decoding, along with a U-Net [16] architecture. The VAE component is
instrumental in mapping input images into a compact latent space, thereby
streamlining computations. The U-Net, on the other hand, accepts noise and
conditional inputs to forecast the output noise, facilitated by employing a
Clip image encoder [14] that transforms the reference image into high-
dimensional features. These features are then utilized within the U-Net for
cross-attention calculations. During training, we intentionally inject noise
into the image frames, tasking the network with predicting this added noise.
The optimization goal is formally established as follows:

$\mathcal{L}=\mathbb{E}_{\mathbf{z}_{t},c,\epsilon,t}(\left\|\epsilon-\epsilon_{\theta}(\mathbf{z}_{t},c,t)\right\|),$
(1)

where $\mathbf{z}_{t}$ is the input noise, $c$ represents the condition, $t$
is the timestep.

Aside from adopting the conventional configurations prevalent in U-Net-based
diffusion models, we have devised several bespoke modules to specifically
cater to learning both foreground and background dynamics. These modules
encompass a Movement Representation system for capturing motion
characteristics and a Progressive Long Video Generation mechanism, enabling
the creation of extended video sequences.

Figure 2: We separately extract body poses and tracking points from the
reference video, with body poses encapsulating the movements of the foreground
subjects and tracking points signifying the dynamics of the background. In
order to address any potential overlap between these two distinct
representations, we employ the inverse of the foreground mask, which is
multiplied with the extracted tracking points. This operation effectively
removes any overlap, ensuring a clean separation of foreground and background
motion elements.

###  2.1 Movement Representation

Previous research predominantly concentrates on rendering foreground activity
through pose-guided frameworks, often disregarding the concurrent background
motion. In genuine video content, backgrounds typically shift in harmony with
the foreground actions. Foreground human movements can be arbitrary,
intertwining bodily gestures and alterations in camera perspective. Meanwhile,
the motion of the background can stem from initial camera maneuvers or
subsequent video post-production edits, rendering the overall movement
intricate. Acknowledging the divergent characteristics between foreground and
background motions, we suggest isolating their representations. Estimating
camera movement directly from videos where the foreground is dynamically in
motion proves imprecise; hence, we introduce a method utilizing tracking
points to articulate background motion. These points meticulously record the
positions of identical elements across sequential frames, furnishing a precise
illustration of the true background dynamics. It is noteworthy that the
intricacy of human bodily movements poses considerable difficulties for
tracking. In alignment with established literature, we adopt the use of
keypoint-based poses to analyze human motion. Specifically, we harness DWPose
[27], a tool proven effective in extracting pose details from both training
and reference footage, thereby efficiently encapsulating the complexities
inherent in human kinetics.

Cotracker [8] is employed to extract tracking points from video content. To
bolster the robustness of our proposed method against variations in the number
of tracking points, we randomly select a subset ranging from 5 to 50 points
for each training video. Considering that the extraction of foreground poses
and background tracking points is carried out through separate methods,
instances may arise where these pose keypoints and tracking points intersect.
To mitigate such overlaps, we undertake a meticulous segmentation of the
foreground, applying a mask to the background representation to eliminate
these superimpositions. We adopt a sophisticated human matting algorithm [10]
to yield an accurate alpha mask that delineates the contours of the foreground
figure with precision. This way, the resultant depiction of background motion
is attained by combining the inverted foreground mask with the extracted
tracking points, thereby assuring a cohesive and non-interfering portrayal of
movement for both foreground and background components.

$M=(1-\alpha)\times T,$ (2)

where $T$ denotes the tracked key points, and $\alpha$ signifies the computed
foreground mask, we illustrate this procedural workflow in Figure 2.

We utilize dual encoder networks, specifically a foreground encoder and a
background encoder, as illustrated in Figure 1, to separately derive features
from the representations of foreground and background motions. These extracted
motion features—pertaining to both the foreground and background—are then
added to features obtained from an input noise vector and a conditional image.
The consolidated set of features is subsequently fed into the primary U-Net
architecture, facilitating comprehensive motion synthesis that accounts for
both contextual and stochastic elements.

###  2.2 Progressive Long Video Generation

Constrained by the finite capacity of GPU memory, our model’s training is
confined to a restricted number of frames. During singular inference
instances, our methodology facilitates the generation of up to 32 consecutive
frames. To extend this capability and produce lengthier videos, approaches
like those outlined in [26] and [6] have adopted a sliding window technique,
allowing clips to overlap. This strategy proves efficacious when dealing with
static background; however, it encounters substantial limitations when applied
to scenarios involving dynamic background motion. In the presence of
background movement, the disparity between successive clips becomes more
pronounced compared to scenes with a stationary background. Consequently,
averaging operations conducted in the latent space under these conditions can
induce blurriness and inconsistency in the generated video sequences,
undermining visual coherence and quality.

Addressing this challenge, we introduce a novel strategy aimed at producing
coherent outputs directly between successive video segments. Initiation of the
video generation begins with the initial reference image serving as the
conditional input for the first clip. Progressing onwards, we adopt a
mechanism where the terminal frame of each preceding clip acts as the
conditioning input for the succeeding one, as illustrated in Figure 1. To
ascertain the alignment of the generated sequence, a condition concatenation
technique is employed, thereby positioning the conditional input image as the
opening frame of the output sequence. Consequently, the initial frame of any
subsequent clip seamlessly aligns with the finale of its antecedent, fostering
continuity. Anticipating potential accumulative errors throughout this
iterative procedure, we devise a solution entailing the extraction of global
features from the initial reference image. These features are then
consistently infused throughout the entire generative process. This measure
bolsters the preservation of visual consistency, empowering our model to
synthesize videos extending beyond 10 seconds without compromising on
coherence or quality.

#### Condition concatenation.

In prior studies [6; 9; 26; 23], the produced frames are primarily governed by
the input pose directives and might not closely resemble the conditioning
image in detail. We introduce an approach where the image latent
representation of the conditional image is directly concatenated with the
noise latent, forming the input to the U-Net architecture. By implementing
this methodology, the network is encouraged to initiate frame generation with
an explicit directive, ensuring that the output sequence begins precisely with
the conditioned image, thereby enhancing visual coherence and control over the
synthesis process. For the situation where the initial pose guide frame
doesn’t match the reference image, we substitute a blank image for generating
the first video clip. When generating consecutive clips, we employ the last
frame of previous clip for condition concatenation.

#### Global feature extraction and injection.

Within the progressive generation sequence, we adopt the strategy of using the
most recently generated frame as the basis for subsequent iterations. However,
as observed by [12], this practice can lead to error accumulation, causing
marked deviations in image content over time. To tackle this issue, we propose
incorporating features derived from the initial reference image as global
features, infusing them throughout the entire generation workflow. These
persistent global features act as a unifying force, maintaining content and
stylistic consistency across different stages of generation and effectively
mitigating error build-up. To ensure the capture of rich detail from the input
image, a potent feature extractor is imperative. While a clip image encoder is
capable of extracting high-level features, these may lack the finer nuances
necessary for precision. Although Anydoor [2] employs DINOV2 [13] to retrieve
more detailed features, these still fall short in high-frequency detail
representation. Notably, U-Net architectures have demonstrated exceptional
capability in extracting fine, detailed features, as evidenced by various
studies including [5; 11; 6; 25; 32; 24; 21; 1]. To harness this strength, we
employ an identical U-Net for the extraction of global features, subsequently
integrating these features into the primary network infrastructure using
concatenation of self-attention features. This strategic utilization of U-Net
not only ensures a rich feature extraction process but also reinforces the
consistency and fidelity of the generated content over prolonged sequences.

##  3 Experiments

###  3.1 Implementation Details

#### Network training.

The U-Net’s weights, along with the global feature extraction component, are
initialized leveraging a pretrained stable diffusion model [runwayml2023sd].
All network modules except for VAE encode and decoder, undergo joint training,
adopting a learning rate of $1e^{-5}$. To facilitate this training process, we
employ a setup consisting of 4 NVIDIA A100 GPUs, with a batch size set to 4.
The chosen resolution for training is $640\times 448$ pixels, and to economize
on memory usage, we implement gradient checkpointing techniques.

#### Dataset.

Current publicly available human motion datasets include the TikTok dataset
[7] and the UBC Fashion dataset [29], both of which consist of videos
meticulously handpicked for their content. Notably, every video within these
datasets features a static background across both training and testing
subsets. Conversely, videos captured in real-world scenarios using handheld
devices like smartphones characteristically incorporate dynamic backgrounds.
In recognition of this discrepancy, we curated the Human-5000 dataset by
gathering unaudited, genuine videos from the web, thereby encompassing a
diverse array of background movements. This dataset, focusing on human-centric
videos, was sourced ethically from copyright-free platforms. Unlike the
aforementioned datasets, the Human-5000 does not impose manual curation on the
foreground or background motion, allowing for a broad spectrum of naturalistic
movements in both elements. The dataset is systematically divided into two
parts: a training set comprising 4000 videos and a test set consisting of 1000
videos, with this allocation performed randomly to ensure unbiased evaluation
of our model’s performance across varied motion scenarios.

#### Baseline.

Our primary comparative analysis focuses on similar methodologies, notably
Disco [23], Magicanimate [1], and Animateanyone [6]. We undertake a
comprehensive evaluation of these approaches, encompassing both qualitative
assessments that gauge visual quality and quantitative measures for objective
comparison. Considering that all the mentioned methods have referenced the
TikTok dataset [7], we initially pit our method against theirs on this common
ground, providing a consistent benchmark for comparison. Furthermore, given
the unique challenges posed by dynamic backgrounds, we extend our evaluation
by contrasting our approach with the state-of-the-art Animateanyone [6] on our
newly compiled Human-5000 dataset, which specifically incorporates a wide
range of background movements. This additional comparison highlights the
efficacy of our method in handling complex, real-world scenarios beyond static
backgrounds.

Figure 3: Inputting previously unseen data, our method’s qualitative
performance highlights its success in generating videos that seamlessly
integrate foreground and background motion.

###  3.2 Results and Comparisons

In Figure 3, We demonstrate qualitative examples of outputs generated by our
proposed method using unseen data not part of either the TikTok or Human-5000
datasets. Our technique effectively produces lifelike human videos, marked by
natural foreground motion and believable background actions.

We employ the TikTok dataset [7], comprising 440 individual dance videos each
lasting 10 to 15 seconds. Out of these, 50 videos constitute our test set, and
the rest form the training set. All comparative algorithms are trained on the
identical training subset. DisCo [23] and MagicAnimate [1] leverage the pre-
trained models provided by their respective authors for initialization,
whereas AnimateAnyone [6] and our model utilize a pre-trained stable diffusion
model for this purpose. A quantitative assessment summarized in Table 1 was
conducted, and qualitative comparisons with MagicAnimate [1] and AnimateAnyone
[6] are illustrated in Figure 4. Both the qualitative and quantitative
evaluations confirm the efficacy of our approach, with our method
outperforming prior works in frame quality and level of detail.

Figure 4: On the TikTok dataset, qualitative analysis reveals that MagicAnimate [26] outputs suffer from color distortion, while AnimateAnyone’s [6] also show detail inconsistencies like face and text. Our method notably enhances these aspects. Table 1: Quantitative comparison on TikTok dataset Methods | SSIM$\uparrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ | FVD$\downarrow$
---|---|---|---|---
DisCo [23] | 0.665 | 28.60 | 0.299 | 302.2
MagicAnimate [26] | 0.714 | 29.16 | 0.253 | 182.1
AnimateAnyone [6] | 0.718 | 29.56 | 0.278 | 171.9
Ours | 0.738 | 29.43 | 0.227 | 168.2

In Figure 5 and Table 2, we specifically pit our method against Animateanyone
[6], utilizing our custom Human-5000 dataset as the testing arena. This unique
dataset is distinguished by its inclusion of videos characterized by
concurrent foreground and background motion. We observe that Animateanyone
struggles to produce convincing videos when confronted with dynamic
backgrounds. Conversely, our methodology excels in generating videos that
seamlessly integrate natural and fluid background movements, highlighting its
superiority in managing complex, real-world motion scenarios.

Figure 5: When compared with AnimateAnyone ([6]) in Human-5000 datasets featuring background motion, our method excels with superior background movement generation capabilities. Table 2: Quantitative comparison on Human-5000 dataset Methods | SSIM$\uparrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ | FVD$\downarrow$
---|---|---|---|---
AnimateAnyone[6] | 0.839 | 29.31 | 0.159 | 201.2
W/O foreground representation | 0.828 | 25.81 | 0.216 | 464.9
W/O background representation | 0.842 | 29.11 | 0.160 | 197.8
W/O gloabl feature | 0.847 | 27.87 | 0.184 | 313.0
W/O condition concatenation | 0.829 | 27.96 | 0.168 | 204.1
Ours | 0.866 | 29.69 | 0.139 | 182.6

Figure 6: Ablation study with different settings.

###  3.3 Ablation Study

#### Without foreground movement representation.

Pose information is incorporated to better illustrate the movement of
foreground humans since tracking points alone prove insufficient in accurately
capturing the intricacies of bodily motion. As a point of comparison, we
conduct experiments omitting foreground representation and relying solely on
tracking points to convey motion. The resultant data, depicted in Figure 6(a)
and quantified in Table 2, reveal a deficiency in effectively generating
realistic human body movements.

#### Without background movement representation.

We introduce the use of sparse tracking points as a means to encode background
motion during both training and inference phases. To validate the efficacy of
this component, we conduct an ablation study by re-training our network sans
the background representation module. As evidenced in Figure 6(b) and Table 2,
the resultant videos produced by this modified network lack any discernible
background movement, leading to outputs that appear less authentic and
lifelike. This comparison underscores the pivotal role of background
representation in enhancing the realism of our generated videos.

#### Without condition concatenation.

Condition concatenation serves as a bridge, seamlessly linking the terminal
frame of one video clip to the initial frame of the succeeding one. To
illustrate its necessity, Figure 6(c) presents examples generated without
employing condition concatenation. For the generation of extended sequences,
we adopt the sliding window technique, as utilized in [26] and [6]. Evidently,
the absence of condition concatenation impedes the attainment of smooth frame
transitions between clips, underscoring its importance in ensuring contiguous
and fluid video sequences.

#### Without global feature extraction.

The integration of a global feature extraction component is pivotal in
maintaining content and style uniformity throughout the progression of lengthy
video generation. To assess its impact, we undertake an experiment where we
omit this component and retrain the network using identical training
configurations. The outcomes depicted in Figure 6(d) and Table 2 illustrate
that, in the absence of global feature extraction, the video content exhibits
marked inconsistencies and notable alterations across consecutive clips.
Conversely, when the global feature extraction is included, the resultant
video demonstrates remarkable consistency and retains a high level of visual
quality, emphasizing its critical contribution to coherent, high-fidelity
video synthesis.

##  4 Limitations and Discussions

In our movement representation approach, we employ DWPose [27] for extracting
human poses and Cotracker [8] to identify background motion. Nonetheless, our
method’s performance is contingent upon the accuracy of these extractions;
inaccuracies or omissions in the extracted movements by either technique could
undermine the final output.

Furthermore, given the sparsity of tracking points allocated for background
motion, there exists the possibility that the full breadth of background
dynamics may not be comprehensively captured by these points alone.
Consequently, the synthesized video might only reflect a portion of the
background movement present in the reference video. This shortcoming can be
somewhat alleviated by increasing the number of tracking points dedicated to
background motion representation, thereby enhancing the system’s capacity to
replicate intricate background activities more faithfully.

Our method excels in producing lifelike videos. However, it’s noteworthy that
advancements in generative model capabilities could pave the way for deepfake
production, posing risks of misuse for disseminating false information. On the
flip side, the absence of audio in our generated videos serves as a
distinguishing feature, facilitating their differentiation from genuine
footage.

##  5 Conclusion

This paper introduces a novel approach to video generation that segregates the
representation of foreground and background motion. By adopting pose
estimation for foreground dynamics and sparse tracking points for background
movement, our method achieves the creation of videos that exhibit both natural
human action and authentic environmental motion, surpassing the constraints of
prior techniques confined to static backgrounds. Moreover, we present an
innovative pipeline designed to facilitate the synthesis of extended video
sequences without encountering cumulative errors over time. Central to this
advancement are techniques of condition concatenation and global feature
extraction, which collectively empower the generation of prolonged, clip-by-
clip video content while preserving visual coherence and consistency
throughout.

## References

  * [1] W. Chen, T. Gu, Y. Xu, and C. Chen.  Magic clothing: Controllable garment-driven image synthesis.  arXiv preprint arXiv:2404.09512, 2024.
  * [2] X. Chen, L. Huang, Y. Liu, Y. Shen, D. Zhao, and H. Zhao.  Anydoor: Zero-shot object-level image customization.  arXiv preprint arXiv:2307.09481, 2023.
  * [3] R. A. Güler, N. Neverova, and I. Kokkinos.  Densepose: Dense human pose estimation in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7297–7306, 2018.
  * [4] Y. Guo, C. Yang, A. Rao, Y. Wang, Y. Qiao, D. Lin, and B. Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  arXiv preprint arXiv:2307.04725, 2023.
  * [5] E. Hedlin, G. Sharma, S. Mahajan, H. Isack, A. Kar, A. Tagliasacchi, and K. M. Yi.  Unsupervised semantic correspondence using stable diffusion.  Advances in Neural Information Processing Systems, 36, 2024.
  * [6] L. Hu, X. Gao, P. Zhang, K. Sun, B. Zhang, and L. Bo.  Animate anyone: Consistent and controllable image-to-video synthesis for character animation.  arXiv preprint arXiv:2311.17117, 2023.
  * [7] Y. Jafarian and H. S. Park.  Learning high fidelity depths of dressed humans by watching social media dance videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12753–12762, 2021.
  * [8] N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht.  Cotracker: It is better to track together.  arXiv preprint arXiv:2307.07635, 2023.
  * [9] J. Karras, A. Holynski, T.-C. Wang, and I. Kemelmacher-Shlizerman.  Dreampose: Fashion image-to-video synthesis via stable diffusion.  In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 22623–22633. IEEE, 2023.
  * [10] J. Liu, Y. Yao, W. Hou, M. Cui, X. Xie, C. Zhang, and X.-s. Hua.  Boosting semantic human matting with coarse annotations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8563–8572, 2020.
  * [11] G. Luo, L. Dunlap, D. H. Park, A. Holynski, and T. Darrell.  Diffusion hyperfeatures: Searching through time and space for semantic correspondence.  Advances in Neural Information Processing Systems, 36, 2024.
  * [12] G. Oh, J. Jeong, S. Kim, W. Byeon, J. Kim, S. Kim, H. Kwon, and S. Kim.  Mtvg: Multi-text video generation with text-to-video models.  arXiv preprint arXiv:2312.04086, 2023.
  * [13] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  arXiv preprint arXiv:2304.07193, 2023.
  * [14] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [15] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [16] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234–241. Springer, 2015.
  * [17] runwayml.  Stable diffusion v1-5 model, 2023.
  * [18] A. Siarohin, S. Lathuilière, S. Tulyakov, E. Ricci, and N. Sebe.  Animating arbitrary objects via deep motion transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2377–2386, 2019.
  * [19] A. Siarohin, S. Lathuilière, S. Tulyakov, E. Ricci, and N. Sebe.  First order motion model for image animation.  Advances in neural information processing systems, 32, 2019.
  * [20] A. Siarohin, O. J. Woodford, J. Ren, M. Chai, and S. Tulyakov.  Motion representations for articulated animation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13653–13662, 2021.
  * [21] L. Tian, Q. Wang, B. Zhang, and L. Bo.  Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions.  arXiv preprint arXiv:2402.17485, 2024.
  * [22] A. Van Den Oord, O. Vinyals, et al.  Neural discrete representation learning.  Advances in neural information processing systems, 30, 2017.
  * [23] T. Wang, L. Li, K. Lin, Y. Zhai, C.-C. Lin, Z. Yang, H. Zhang, Z. Liu, and L. Wang.  Disco: Disentangled control for realistic human dance generation.  arXiv preprint arXiv:2307.00040, 2023.
  * [24] H. Wei, Z. Yang, and Z. Wang.  Aniportrait: Audio-driven synthesis of photorealistic portrait animation.  arXiv preprint arXiv:2403.17694, 2024.
  * [25] Y. Xu, T. Gu, W. Chen, and C. Chen.  Ootdiffusion: Outfitting fusion based latent diffusion for controllable virtual try-on.  arXiv preprint arXiv:2403.01779, 2024.
  * [26] Z. Xu, J. Zhang, J. H. Liew, H. Yan, J.-W. Liu, C. Zhang, J. Feng, and M. Z. Shou.  Magicanimate: Temporally consistent human image animation using diffusion model.  arXiv preprint arXiv:2311.16498, 2023.
  * [27] Z. Yang, A. Zeng, C. Yuan, and Y. Li.  Effective whole-body pose estimation with two-stages distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4210–4220, 2023.
  * [28] W.-Y. Yu, L.-M. Po, R. C. Cheung, Y. Zhao, Y. Xue, and K. Li.  Bidirectionally deformable motion modulation for video-based human pose transfer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7502–7512, 2023.
  * [29] P. Zablotskaia, A. Siarohin, B. Zhao, and L. Sigal.  Dwnet: Dense warp-based network for pose-guided human video generation.  arXiv preprint arXiv:1910.09139, 2019.
  * [30] P. Zhang, L. Yang, J.-H. Lai, and X. Xie.  Exploring dual-task correlation for pose guided person image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7713–7722, 2022.
  * [31] J. Zhao and H. Zhang.  Thin-plate spline motion model for image animation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3657–3666, 2022.
  * [32] S. Zhu, J. L. Chen, Z. Dai, Y. Xu, X. Cao, Y. Yao, H. Zhu, and S. Zhu.  Champ: Controllable and consistent human image animation with 3d parametric guidance.  arXiv preprint arXiv:2403.14781, 2024.