# Lightweight Frequency-Based Tiering
for CXL Memory Systems

Kevin Song
University of Toronto
Canada
Jiacheng Yang
University of Toronto
Canada
Sihang Liu
University of Waterloo
Canada
Gennady Pekhimenko
University of Toronto
Canada

###### Abstract

Modern workloads are demanding increasingly larger memory capacity. Compute
Express Link (CXL)-based memory tiering has emerged as a promising solution
for addressing this trend by utilizing traditional DRAM alongside slow-tier
CXL-memory devices in the same system. Unfortunately, most prior tiering
systems are recency-based, which cannot accurately identify hot and cold
pages, since a recently accessed page is not necessarily a hot page. On the
other hand, more accurate frequency-based systems suffer from high memory and
runtime overhead as a result of tracking large memories.

In this paper, we propose FreqTier, a fast and accurate frequency-based
tiering system for CXL memory. We observe that memory tiering systems can
tolerate a small amount of tracking inaccuracy without compromising the
overall application performance. Based on this observation, FreqTier
probablistically tracks the access frequency of each page, enabling accurate
identification of hot and cold pages while maintaining minimal memory
overhead. Finally, FreqTier intelligently adjusts the intensity of tiering
operations based on the application’s memory access behavior, thereby
significantly reducing the amount of migration traffic and application
interference.

We evaluate FreqTier on two emulated CXL memory devices with different
bandwidths. On the high bandwidth CXL device, FreqTier can outperform state-
of-the-art tiering systems while using 4$\times$ less local DRAM memory for
in-memory caching workloads. On GAP graph analytics and XGBoost workloads with
1:32 local DRAM to CXL-memory ratio, FreqTier outperforms prior works by
1.04$-$2.04$\times$ (1.39$\times$ on average). Even on the low bandwidth CXL
device, FreqTier outperforms AutoNUMA by 1.14$\times$ on average.

##  I Introduction

Modern applications [7, 70, 41] are demanding memory with increasingly larger
capacity and bandwidth. To keep up with this fast growth, one potential
solution is to install more and higher-capacity DRAM modules. However, since
main memory is already one of the most expensive components in data center
servers, purchasing more DRAM modules will only exacerbate this cost. Meta
reports that 37$\%$ of the total cost of ownership per rack is spent on memory
alone [40]. Moreover, the growth in DRAM density has been slowing down since
the last decade [50, 45], further limiting the capacity scalability of main
memory.

Compute Express Link (CXL) based memory tiering is one of the most promising
solutions to this challenge. CXL is an industry-standard interconnect
protocol. Memory (e.g. DDR4 and DDR5 DRAM) attached through the CXL interface
is byte-addressable, directly accessible by the host CPU, and supports
standard memory allocation interfaces. Compared to local DRAM, CXL-attached
memory has a larger capacity and lower cost-per-GB, but higher latency and
lower bandwidth. Different CXL memory devices can have different latency and
bandwidth, depending on the CXL controllers and memory technologies [60]. As
shown in Figure 1, compared to local DRAM, current CXL memory devices
introduce 50$-$100 ns of additional access latency [40], while achieving
20$-$70$\%$ of its bandwidth [60]. Therefore, placing only the hot data in
local DRAM and the rest in CXL-memory can improve the system memory capacity
at a lower cost.

Figure 1: Illustration for CXL-attached memory expansion. Performance numbers
obtained from [40, 60].

To fully exploit the benefits of CXL-memory, the prerequisite is to understand
the workloads. Modern large memory workloads tend to exhibit skewed memory
accesses, where the majority of memory accesses are focused on a small portion
of data. Twitter and Meta both report that the popularity distribution of in-
memory caching workloads largely follows Zipfian distribution with a high
degree of skewness [7, 70], where a small fraction of memory consumes the
majority of accesses. For instance, the Meta content delivery network (CDN)
cache follows a Zipfian distribution where the top 20% hottest items consume
60% of all accesses [7]. This skewness in access distribution synergies well
with memory tiering.

With the rising popularity of CXL hardware and growing application demands,
designing an efficient tiering system for CXL memory is becoming an
increasingly important topic. However, the design of a practical CXL tiering
system faces two key challenges. First, _inaccurate data placement between
local DRAM and CXL-memory can significantly degrade application performance_ ,
due to CXL-memory’s longer access latency and lower bandwidth. Second, _the
tiering system can induce significant memory and runtime overhead_. Keeping
track of the access information for TBs of memory can result in significant
memory overhead. In addition, operations, such as page migrations and memory
access tracking, consume memory and compute resources.

Several prior tiering systems attempt to address these challenges. One class
of memory tiering system is _recency-based tiering_ , which approximates data
hotness using access recency. In general, such systems make data migration
decisions based on access information from a relatively short time window,
typically in the order of seconds or less [24, 40]. AutoNUMA [24] and TPP [40]
are two state-of-the-art recency-based tiering systems. While tracking access
recency is relatively low-overhead, a recently accessed page is not
necessarily a hot page. For instance, a page that was recently accessed once
but never again should not be promoted to local DRAM. Therefore, recency-base
tiering systems are often not accurate in identifying hot pages, leading to
lower tiering quality. This issue is especially prominent in systems with
lower local DRAM capacities. For example, recency-based systems can suffer up
to 20$\%$ local DRAM hit ratio drop when the local DRAM capacity is reduced
from 64GB to 16GB.

To better track memory hotness, another class of tiering systems identifies
hot data using _frequency_ instead of _recency_. Compared to recency-based
tiering systems, frequency-based tiering has the potential to more accurately
identify hot pages since migration decisions are based on access information
from a longer window of time in the order of minutes to hours. However,
tracking the access frequency of up to TBs of memory pages over a long period
of time can incur significant memory overhead. HeMem [54], a state-of-the-art
frequency-based tiering system, can take 4% memory overhead when tiering at
4KB page granularity. As tiering metadata is accessed frequently, they place
additional pressure on the limited local DRAM, greatly reducing the cost
savings of tiered memory systems.

We summarize two major challenges in CXL tiering: (1) data placement accuracy
and (2) memory and runtime overhead. Addressing both challenges at the same
time is difficult and prior approaches tradeoff one for the other. As
different CXL memories that feature various performance characteristics are
around the corner, there is an urgent need for a tiering solution that
fundamentally addresses these challenges.

This work aims to overcome both challenges. We propose FreqTier, a frequency-
based application-transparent tiering system that achieves both high tiering
accuracy and low memory and runtime overhead. To accurately identify hot pages
and maximize tiering performance under low local DRAM capacities (Challenge
1), FreqTier adopts frequency-based tiering by tracking memory accesses at the
page granularity using high-performance hardware counters. To reduce memory
overhead (Challenge 2), we observe that access frequency tracking can tolerate
a small amount of inaccuracy without degrading the overall application
performance. Based on this insight, FreqTier tracks page access frequencies
using counting bloom filter (CBF) [18], a probabilistic data structure. The
key advantage of the CBF is that it purposely introduces a controllable level
of tracking inaccuracy to achieve better memory efficiency. We demonstrate in
Section VII-E that a moderately sized CBF is sufficient to achieve high
tracking accuracy. Lastly, to reduce the runtime and memory overhead due to
page migrations and access sampling (Challenge 2), FreqTier adopts a dynamic
approach to adjust the intensity of tiering operations based on the
application memory access behavior. Our evaluation shows this technique
reduces page migration traffic by on average 4.2$\times$ compared to prior
works to effectively decrease application interference (Section III).

To evaluate the effectiveness of FreqTier, we evaluate the performance of
FreqTier with CacheLib (in-memory caching) [17], GAP (graph analytics) [5],
and XGBoost (gradient boosting) [68], and compare against three state-of-the-
art tiering systems, AutoNUMA [24], TPP [40], and HeMem [54]. In terms of
hardware, we emulate two CXL memory devices, one configuration with a higher
CXL bandwidth and the other with a lower bandwidth, following the methodology
from prior works [33, 40]. The emulated performance akins to performance
numbers reported in a recent CXL characterization work [60].

In summary, this work makes the following contributions:

  * •

We propose a new probabilistic access frequency tracking mechanism based on
the insight that memory tiering systems can tolerate a certain amount of
tracking inaccuracy.

  * •

Based on the above insight, we introduce a novel application-transparent
memory tiering system, FreqTier. This design can more accurately identify hot
and cold pages with low memory and runtime overhead.

  * •

We evaluate FreqTier on large memory workloads using two emulated CXL memory
devices. On the high bandwidth CXL device, FreqTier can outperform prior works
while using 4$\times$ less local DRAM memory on CacheLib. On GAP and XGBoost,
FreqTier outperforms prior works by 1.39$\times$ on average with 16GB of local
DRAM. Even with low bandwidth CXL, FreqTier consistently outperforms AutoNUMA
by an average of 1.14$\times$.

##  II Background and Motivation

###  II-A CXL-Enabled Memory Tiering Systems

CXL is an open industry standard interconnect running on top of the PCIe
physical layer [59, 43, 58, 48]. The key goal of CXL is to better support
heterogeneous computing and memory capabilities in future data center
architectures. Any device attached through the CXL interface will share a
unified and cache-coherent memory space with the host processor and other CXL
devices. Since its introduction in 2019, the CXL ecosystem has been under
rapid development. With support from major vendors across the data center
stack, CXL is widely believed to make a significant impact in future data
center architectures.

A CXL-enabled memory tiering system utilizes both regular local DRAM and CXL-
attached memory technologies in the same system. Local DRAM offers better
latency and bandwidth but has greater cost-per-GB, while CXL memory provides
higher capacity and lower cost-per-GB but suffers from lower performance. The
key goal of a tiered memory system is to combine the benefits of both local
DRAM and CXL-memory while minimizing the amount of performance degradation due
to occasional accesses to CXL-memory. To achieve this, memory tiering systems
must accurately identify hot and cold data and place them into the local DRAM
and CXL-memory, respectively.

###  II-B Skewness in Large Memory Workloads

Large memory workloads often exhibit skewed memory access distribution, where
the small set of addresses accounts for the majority of memory accesses. For
instance, Twitter and Meta both report that the popularity of in-memory
caching workloads largely follows Zipfian distribution with a high degree of
skewness [70, 7] – a maximum alpha value of 2.5. For reference, a Zipfian
distribution with alpha = 0.9 indicates the top 10% most popular items make up
approximately 80% of all accesses. TPP [40] reports that certain Meta web
serving and advertisement workloads only use 20-40% of the total allocated
memory within a two-minute interval, while the rest of the memory remains cold
[40]. For graph analytics workloads, Moura et. al [44] observe that for the
betweenness centrality kernel in the GAP benchmark suite, the top two hottest
objects consume nearly 40% of all accesses.

Skewed access distributions make memory tiering especially promising. Since
the hottest data typically only constitute a small portion of the total
application footprint, an effective tiering system should achieve near-optimal
performance when its fast-tier memory capacity is large enough to fit the
application’s hot pages. However, as the fast-tier memory capacity reduces,
the margin for tiering inaccuracies is also reduced. For example, when the
local DRAM to CXL-memory capacity ratio drops from 1:8 to 1:32, TPP
experiences up to 9.6$\%$ drop in throughput on a CacheLib [7] workload. Thus,
_the ability to accurately identify hot and cold pages_ is crucial for high-
performance tiering systems.

###  II-C State-of-the-art and Limitations

While skewness makes tiering promising in theory, designing a practical and
performant tiering system is challenging.

####  II-C1 Prior Recency-based Tiering Systems

Recency-based tiering uses data access recency to approximate data hotness. In
general, such systems make data migration decisions based on data access
information from a relatively short window of time, usually in the order of
seconds. AutoNUMA [24] and TPP [40] are two state-of-the-art recency-based
tiering systems

AutoNUMA is the tiering system used by the Linux kernel. To sample memory
accesses, AutoNUMA periodically scans the application address space and unmaps
256MB of pages (the scan window). The next access to an unmapped page will
trigger a minor page fault, at which time AutoNUMA calculates the time elapsed
since the page was unmapped (hint fault latency). If the hint fault latency is
less than a hot threshold, this page is considered hot and promoted [23]. TPP
[40] uses the same hint fault mechanism to sample memory accesses. Different
from AutoNUMA, TPP promotes a faulted page only if it is in the active LRU
list.

(a) 16GB local DRAM.

(b) 32GB local DRAM.

Figure 2: Memory traffic volume breakdown. Local DRAM access and CXL memory
access are accesses that missed L3 but serviced from local DRAM/CXL memory
respectively. Evaluated on two CacheLib workloads.

The advantage of recency-based systems is that they can be lightweight.
However, their classification accuracy suffers. AutoNUMA can suffer from
inaccurate tiering for two reasons. _(1) Missed frequency information:_ As
illustrated in the first timeline of Figure 3, when a page is accessed
multiple times within the hot window, only the first access is considered,
resulting in the loss of frequency information. _(2) Page misclassification:_
In the second and third timelines, a cold page can be mistakenly classified as
a hot page and promoted if just one access falls within the hot window, while
a hot page may not get promoted because none of its recent accesses happen to
fall within the hot window [20, 32]. Similarly, TPP’s LRU-based promotion
mechanism treats all pages within the active LRU list equally, even though
they can have varying degrees of hotness.

To demonstrate, we evaluate AutoNUMA and TPP’s accuracy by measuring the local
DRAM hit ratio (i.e., the faction of memory accesses to local DRAM out of the
total memory accesses) on a CacheLib [7] workload with a memory footprint of
267GB. When given 16GB of local DRAM and 512GB of CXL-memory, AutoNUMA, and
TPP suffer from low hit ratios of 71% and 70% respectively. In Section VII, we
demonstrate that it is possible for a tiering system to achieve 90% hit ratio.
This lower hit ratio not only limits the effective use of local memory but
also leads to interference with the application. Figure 2a shows the migration
traffic due to tiering, local DRAM accesses, and CXL accesses in CacheLib,
with 16GB of local DRAM. AutoNUMA and TPP’s page migrations generate on
average 10.4% and 43.5% of the total memory traffic, interfering with the
application. While the percentages of local DRAM accesses improve with 32GB of
local DRAM (Figure 2b), the amount of page migration traffic remains
significant.

Figure 3: Drawbacks of AutoNUMA promotion mechanism.

####  II-C2 Prior Frequency-based Tiering Systems

In comparison to recency-based tiering, frequency-based tiering can achieve a
higher local DRAM hit ratio when targeting skewed workloads. Frequency-based
tiering measures data hotness based on access information from a relatively
longer window of time in the order of minutes to hours. State-of-the-art
frequency-based tiering systems such as HeMem [54] adopt data structures such
as a hash table to track page access frequencies.

While prior frequency-based systems do not suffer from low classification
accuracy, their memory overhead is high. In Section VII-C, we demonstrate that
HeMem metadata consumes 4$\%$ of the total memory capacity. This high memory
overhead significantly reduces the cost savings of tiering systems.

##  III FreqTier Key Ideas

In this section, we first summarize the challenges of designing an effective
CXL tiering system and then describe how we tackle them.

Challenge 1. Inaccurate data placement between DRAM and CXL memory can
significantly degrade application performance. This is because CXL-memory has
considerably longer access latency and lower bandwidth compared to local DRAM,
as studied by prior works [40, 33, 60]. Thus, the first challenge is how a
memory tiering system can accurately identify hot pages and place them in
local memory to minimize the number of accesses to CXL-memory.

Solution. For better classification accuracy, we adopt frequency-based tiering
at page granularity. FreqTier employs high-performance Processor Event-Based
Sampling (PEBS) counters to efficiently collect local DRAM and CXL-memory
access samples. FreqTier records separate frequency counts for each 4KB page,
the smallest granularity supported by Linux memory migration interfaces. Based
on the access frequency of each page, FreqTier determines the appropriate hot
threshold based on the current access hotness distribution and performs
promotion and demotion accordingly. Our approach effectively reduces the
amount of page migration traffic by 4.2$\times$ on average over prior works,
as shown in Figure 2.

Challenge 2. The tiering system can induce significant memory and runtime
overhead, especially for frequency-based tiering. In modern large memory
systems with TBs of memory, storing the hotness of all pages for a long period
of time can require a significant amount of memory. Thus, tiering systems must
minimize their memory overhead, otherwise, the cost saving of memory tiering
systems is diminished. To reduce the memory overhead, prior works rely on
techniques such as tracking at the huge page granularity. However, such
approaches sacrifice classification accuracy. In addition, page migrations
between local DRAM and CXL-memory consume memory and compute resources due to
page copying and updating the OS page table entries [60]. If not managed
carefully, this overhead can interfere with applications and diminish their
performance.

Solution. FreqTier exploits an observation that tiering systems can tolerate
tracking inaccuracy. Ultimately, the goal of the tiering system is to
differentiate between hot and cold pages. Whether a page is extremely hot or
moderately hot does not change the fact that this page will be promoted to
local DRAM. Thus, if the overall access frequency distribution of the
application can be accurately captured, the access frequency of a page can
often be reliably approximated. Using this insight, FreqTier _tracks page
access frequencies probabilistically_ to significantly reduce the memory
overhead. To mitigate interference with the application, FreqTier adopts an
adaptive method that adjusts the tiering intensity based on the application’s
memory behavior.

##  IV FreqTier Design

In this section, we describe the design of FreqTier based on key ideas in
Section III.

###  IV-A Workflow Overview

Figure 4 illustrates the high-level design of FreqTier. FreqTier is
implemented as a userspace runtime thread that performs tiering for a workload
process. \bf\raisebox{-0.8pt} {\small1}⃝ FreqTier dynamically links the
FreqTier shared library into the target application binary using LD_PRELOAD
dynamic link mechanism. This process is _transparent_ to the application and
does not require recompiling the target workload. \bf\raisebox{-0.8pt}
{\small2}⃝ The shared library spawns the tiering thread just before the main
function of the target workload is invoked. \bf\raisebox{-0.8pt} {\small3}⃝
FreqTier uses hardware performance counters to sample local DRAM and CXL
memory accesses performed by the target workload. By utilizing hardware
counters, FreqTier can record access samples from the entire process address
space, compared to AutoNUMA whose access samples are limited to one scan
window at a time. \bf\raisebox{-0.8pt} {\small4}⃝ Sampled memory addresses
collected are recorded in the counting bloom filter and the tiering policy
makes promotion and demotion decisions based on the hotness of pages.
\bf\raisebox{-0.8pt} {\small5}⃝ Finally, FreqTier utilizes Linux kernel system
call to promote hot pages to local DRAM and demote cold pages to CXL-memory.

Figure 4: An overview of FreqTier.

###  IV-B Probabilistic Frequency Tracking

Data structures such as a hash table are precise, ensuring that page access
frequencies are stored exactly as initially recorded. However, absolute
accuracy is not always required in the context of memory tiering. For
instance, even if a page is inaccurately recorded as having an access count of
one when in fact it has never been accessed, this page will likely still be
classified as a cold page and no migration decisions will be affected by this
particular inaccuracy. We differentiate between this tracking inaccuracy and
the inaccuracies inherent in recency-based tiering systems. For frequency
tracking, a small amount of inaccuracy does not substantially alter the
overall frequency distribution among all memory pages. As a result, it has a
negligible impact on tiering decisions, as we demonstrate in Section VII-E1.
On the other hand, recency-based tiering is fundamentally incapable of
capturing the access frequency distribution.

Based on the above observation, we propose to use the counting bloom filter
(CBF) [18], a probabilistic data structure, to track access frequencies of
each page. The CBF builds upon the bloom filter, where each element in the
array is a counter value instead of a single bit. Intuitively, CBF introduces
a controllable level of tracking inaccuracy to achieve better memory
efficiency. To store the access frequencies of TBs of pages, one obvious data
structure is the hash table, where the hash key is the page address and the
value is the access count. Unlike hash tables, CBF does not store the hash
keys to significantly reduce its memory consumption. The CBF purposely allows
hash collisions to occur and controls the likelihood of collisions with the
array size, where a larger CBF provides higher accuracy. We perform a detailed
study on how different CBF sizes affect performance in Section VII-E and
demonstrate that a moderately sized CBF can achieve high tracking accuracy. A
detailed description of CBF is presented in Section V-A.

###  IV-C Migration Policy

Promotion. FreqTier adopts a threshold-based promotion mechanism illustrated
in Algorithm 1. For each access sample collected, FreqTier first records this
page access in the CBF. If the incremented frequency is greater or equal to a
global hot threshold, FreqTier promotes this page. This mechanism is effective
because hot pages are sampled more frequently, providing plenty of promotion
opportunities. To reduce the migration system call overhead, FreqTier batches
100K samples at a time and promotes all hot pages in a batch with a single
system call. The exact batch size is an adjustable parameter. We present a
sensitivity study in Section VII-E.

1 Input: MemoryAccessSamples

2 foreach  _PageAddress $\in$ MemoryAccessSamples_ do

3 CBF.Increment(PageAddress)

4 if  _CBF.Get(PageAddress) $\geq$ HotThreshold_ then

5 PromotePage(PageAddress)

6

Algorithm 1 Hot page promotion

Demotion. Unlike promotions, FreqTier cannot rely on access samples to trigger
demotions because cold pages are rarely sampled. FreqTier identifies cold
pages in local DRAM memory by linearly scanning the application virtual
address space, as illustrated in Algorithm 2. FreqTier utilizes /proc/PID/maps
to find the next demotion candidate and /proc/PID/pagemaps to check if the
candidate page is mapped to local DRAM. For each page in local DRAM, if its
frequency is lower than the hot threshold, this page is prepared for demotion.
Naively implementing this demotion algorithm will incur large overhead due to
excessive reads to the Linux pseudo-filesystem. We discuss optimizations that
make it practical in Section V-B.

1 Input: LastScannedPage, PagesToDemote

2 Output: LastScannedPageOut

3 CurPageAddr = LastScannedPage

4 PagesDemoted = 0

5 while  _PagesDemoted $<$ PagesToDemote_ do

6 if  _CurPageAddr is in local DRAM_ then

7 if  _CBF.Get(CurPageAddr) $<$ HotThreshold_ then

8 DemotePage(CurPageAddr)

9

10 CurPageAddr = next address in virtual address space

11 PagesDemoted = PagesDemoted + 1

12

13LastScannedPageOut = CurPageAddr

14

Algorithm 2 Cold page demotion

Dynamic tiering intensity.  Constantly collecting memory access samples with
high sampling frequency enables FreqTier to make more accurate tiering
decisions, but also incurs high runtime overhead. We observe that once the
majority of hot pages are identified, further page migrations cease to improve
application performance. FreqTier monitors the local DRAM hit ratio and
predicts whether further tiering can continue to provide performance benefits.
If the local DRAM hit ratio improvement becomes marginal, FreqTier
automatically reduces the sampling frequency to reduce migration traffic and
sampling overhead.

Real-world workloads exhibit dynamically changing memory access distributions
over time[7, 70] and tiering systems must adapt to such changes in a timely
manner. Upon detecting a sudden change in local DRAM hit ratio, FreqTier
automatically increases the sampling frequency to swiftly adjust to the new
distribution. Additional details on the algorithm are provided in Section
V-B2.

##  V Implementation Details

###  V-A Counting Bloom Filter

The CBF consists of $k$ hash functions and an array of size $N$ storing
integer access frequency counts. A CBF supports two operations: GET and
INCREMENT. GET calculates $k$ array indices from $k$ hash functions and
returns the minimum counters within the $k$ counters. INCREMENT calculates $k$
indices from $k$ hash functions and increment the minimum counters. An example
is illustrated in Figure 5 with $k$=3. GET on page A will return 3 and
INCREMENT on page A will increase the counters at indices 0 and 4 to the value
4. Similarly, INCREMENT on page B will increase the counters at indices 2, 5,
and 7 to the value 10, and GET on page B will return 10. To ensure the
freshness of counters, FreqTier performs aging periodically by dividing all
counters by 2, similar to [54, 16].

Figure 5: Counting bloom filter illustration.

The number of bits used for each frequency counter in the CBF array should be
large enough to accurately capture the frequency distribution. On the other
hand, allocating more bits to each counter increases the memory overhead. By
default, FreqTier empirically allocates 4 bits for each counter, allowing a
maximum frequency count of $2^{4}-1=15$. All pages with access counts $\geq$
15 are treated equally. We present results in Section VII-E to demonstrate
this bit count is sufficient.

In terms of the number of counters, FreqTier by default allocates a CBF large
enough to store all pages in local DRAM while achieving a false positive rate
of $10^{-3}$ (based on methods in a prior study [8]). We evaluate the effect
of CBF size on application performance in Section VII-E.

###  V-B Migration Policy

Figure 6 shows FreqTier state transitions. At initialization, FreqTier starts
at the promoting state. We adopt the default Linux policy where memory
allocations are served from local DRAM if space is available. Similar to
AutoNUMA and TPP, FreqTier uses two watermarks, DEMOTE_WMARK and PROMO_WMARK,
to trigger promotion and demotions. Both watermarks measure against the amount
of free capacity on local DRAM, where DEMOTE_WMARK is greater than
PROMO_WMARK. When the amount of free memory is below PROMO_WMARK, FreqTier
demotes until the amount of free memory is greater than DEMOTE_WMARK. These
watermarks follow the default setting in the Linux kernel [71] (same in
AutoNUMA).

Figure 6: FreqTier migration policy.

####  V-B1 Demotion Algorithm

Once enough pages are demoted to reach DEMOTE_WMARK, FreqTier checkpoints the
last scanned address and resumes from this address the next time demotion is
triggered. This allows FreqTier to save its demotion scanning progress, as
illustrated in Figure 7. Once the demotion scan reaches the last address in
the process address space, it returns to the first address and continues
scanning. To further reduce the demotion overhead, FreqTier queries
/proc/PID/pagemaps in a batch of contiguous pages. Once enough cold pages are
identified and collected, one numa_move_pages() call demotes all cold pages.
With the above optimizations, on CacheLib workloads with 16GB of local DRAM,
FreqTier demotes roughly 160k pages every time demotion is triggered, a
process that takes less than 3 seconds.

####  V-B2 Dynamic tiering intensity

By default, FreqTier offers three levels of sampling frequencies by default:
100kHz, 10kHz, and 1kHz. At initialization, FreqTier samples accesses at
100kHz. While pages are being migrated, FreqTier records the instantaneous
local DRAM hit ratio every minute. If the hot ratios stay largely the same
(i.e., vary within 0.5% in our implementation), the hit ratio is deemed
stable.

If the local DRAM hit ratio is deemed stable, FreqTier reduces the sampling
frequency by one level. If the hit ratio becomes unstable, FreqTier increases
the sampling frequency by one level. This allows FreqTier to continue sampling
at a high rate if the hit ratio is only stable temporarily.

If the hit ratio is stable and the current sampling frequency is already at
the lowest level, FreqTier enters low overhead monitoring mode. In monitoring
mode, FreqTier utilizes perf-stat [51] to continue tracking the local DRAM hit
ratio. Compared to access sampling, perf-stat only counts the number of local
DRAM and CXL-memory accesses, thus incurring negligible overhead. Within
monitoring mode, if the local DRAM hit ratio becomes unstable, FreqTier
considers the access distribution to have changed, returns to the promotion
state, and restarts sampling with the highest sampling frequency.

In addition to a stable hit ratio, FreqTier enters monitoring mode if any of
the two following conditions are true: _(1) Promotion plateau:_ In addition to
local DRAM hit ratio, since applications such as GAP workloads have naturally
unstable local DRAM hit ratios, FreqTier also monitors the number of pages
promoted for each one-minute window. If no pages are promoted in the past
window, we refer to this as a promotion plateau. When promotion occurs at such
slow rates, the small amount of tiering improvement does not justify the
tiering overhead and FreqTier enters monitoring mode. _(2) Empty demotion
scan:_ As more cold pages are demoted, eventually no more cold pages remain in
local DRAM. At this point, further page migrations are likely not beneficial,
as local DRAM is already full of hot pages. If FreqTier reaches the end of the
address space, it restarts the demotion scan from the top of the process
address space to demote any newly allocated cold pages, as illustrated in
Figure 7. If no more cold pages are found, FreqTier enters monitoring mode.

Figure 7: FreqTier demotion mechanism.

###  V-C FreqTier Optimizations

In this section, we describe three optimizations to FreqTier: (a) dynamic hot
threshold, (b) blocked CBF, and (c) CBF increment coalescing.

##### Dynamic Hot Threshold.

The hot threshold determines the minimum number of accesses a page should
possess to be considered hot. FreqTier dynamically adjusts the hot threshold
to maximize the tiering quality. To utilize local DRAM most efficiently, the
hot threshold should be set such that the total size of hot pages is roughly
equal to the size of local DRAM. If the number of hot pages (based on the
current hot threshold) is larger than the local DRAM capacity, FreqTier
increases the hot threshold to only keep the hottest data. Similarly, if the
hot pages cannot fill the entire local DRAM, FreqTier decreases the hot
threshold. By default, the initial hot threshold is 5, which we observe to
work well experimentally.

##### Blocked Counting Bloom Filter.

In the traditional CBF described in Section V-A, since the counter array can
be larger than the L3 cache, each access to the CBF can require up to three
cache line accesses and worst case three DRAM accesses. Block CBF is a
variation that bounds the max number of cache lines and/or DRAM accesses to
one. With Blocked CBF, all counters of the same page belong to the same
64-byte block, the size of one cache line. We follow the implementation of
Caffeine [35] to use portions of one hash function to first select a 64-byte
block. The rest of the hash functions select which three counters within this
block to use. We observe negligible counting accuracy loss when switching to
blocked CBF.

##### Counting Bloom Filter Increment Coalescing.

Since the memory access distribution is often skewed, accesses to the bloom
filter are often focused on a few hot pages. Thus, repeating
increment_frequency() calls to the same page can be coalesced to a single
increase_frequency(page_address, amount) call, which increases the frequency
of page_address by amount. While collecting access samples, FreqTier tracks
the number of samples collected for each page address using a hash table.
After a batch of samples is collected, we invoke increase_frequency instead of
increment_frequency for each unique page address in the hash table. We observe
4$\times$ fewer CBF accesses with this optimization.

##  VI Methodology

###  VI-A CXL Emulation

While recent studies have revealed performance characteristics of real CXL-
memory devices (CXL 1.1 specification) [60], these devices are not
commercially available on the market. We use a remote NUMA node on a two-
socket system to emulate the performance of CXL because it is the most
accurate performance emulation methodology we are aware of [40, 33]. Each
socket has a 16-core Intel Xeon 4314 processor and 512GB of DDR4 memory.
Memory attached to the remote NUMA node emulates CXL-memory. The application
runs on the CPUs of the local NUMA node. To adjust the local:CXL memory ratio,
we use the memmap kernel parameter to disable portions of DRAM on the local
NUMA node.

We emulate two CXL configurations. These configurations follow the CXL latency
and bandwidth numbers reported in a recent work [60]. CXL-1 emulates a fast,
high-bandwidth CXL memory system, and CXL-2 emulates a slower, low-bandwidth
CXL system. The faster emulated CXL in CXL-1 is based on enabling full memory
channels (8) on the NUMA node. And, the reduction in bandwidth in CXL-2 is
achieved by reducing the number of memory channels to 1 in the remote NUMA
node. Figure 8 shows the latency and bandwidth of the emulation measured using
Intel Memory Latency Checker [63]. These numbers are similar to the fast and
slow CXL devices reported in a recent work [60].

The evaluation (Section VII) will focus on CXL-1, as future CXL systems are
projected to have high bandwidth. We evaluate CXL-2 in Section VII-B to
examine FreqTier’s capability of accurate and low-overhead tiering at low
bandwidth.

(a) Latency. (b) Bandwidth.

Figure 8: Access latency and bandwidth for local DRAM and emulated CXL-memory
on two CXL configurations.

###  VI-B Baselines and FreqTier Configurations

We compare FreqTier against AutoNUMA [24], TPP, HeMem, and all-local
configuration. Both AutoNUMA and TPP represent state-of-the-art recency-based
tiering systems. As an industry-standard tiering system, AutoNUMA has
incorporated state-of-the-art features from both academia and industry [71,
11, 38]. We have confirmed with the TPP author that the majority of TPP
features have been merged into AutoNUMA since v6.0 [21, 22]. The key
difference remaining is that TPP uses LRU-based promotion whereas AutoNUMA
adopts hint fault latency-based promotion. HeMem [54] represents the state-of-
the-art frequency-based tiering system. Since the original HeMem is designed
specifically for persistent memory systems (e.g., Intel’s Optane [25]), we
reimplement and adapt HeMem for CXL memory.

The all-local baseline places all data in local DRAM and represents the
performance upper bound. For AutoNUMA, FreqTier, HeMem, and all-local, we use
Linux kernel v6.2. While the original TPP was integrated into Linux kernel
v5.18, we evaluate TPP on the newest kernel (v6.0) that it is compatible with
to minimize performance differences caused by other parts of the kernel. We
set up and configure TPP based on guidelines provided by the TPP author. For
AutoNUMA, we enable multi-generational LRU (MGLRU) based demotion due to its
better performance. Since kernel v6.0 does not support MGLRU, we use regular
LRU-based demotion for TPP. We acknowledge this as a disadvantage for TPP when
compared to AutoNUMA, which may explain the inferior performance of TPP in
general.

TABLE I: Workloads for evaluation. Application | Input | Footprint
---|---|---
Content-delivery network | CacheLib generator [17] | 267GB
Social-graph
Betweenness centrality (BC) | Kronecker graph [5] | 335GB
Breadth first search (BFS)
Connected components (CC)
XGBoost [68] |  Criteo Click Logs [12] | 248GB

###  VI-C Workloads

We evaluate FreqTier on workloads shown in Table I. All workloads use 16
threads mapped to 16 physical cores. The runtime of all tiering systems shares
the same 16 physical cores with the application.

CacheLib is an in-memory caching engine used by Meta in various use cases
including content delivery networks (CDN) and social graphs [7]. We use the
benchmarking framework from Meta, cachebench [9], to evaluate two workloads:
CDN and social graph. Each workload is defined by a unique popularity
distribution, size distribution, and operation composition, all of which are
provided as a part of the CacheLib project. We run each experiment for 1 hour
and report the average GET latency and throughput every 3 seconds. We allocate
256GB of memory for CacheLib to store items. The item popularity distribution
and total number of items are scaled up to achieve realistic CacheLib hit
ratios.

GAP benchmark suite is a collection of standard graph processing kernel
implementations [6]. We generate a Kronecker power-law graph, which has been
used frequently in prior research [6], with 2 billion nodes and 8 billion
edges (average degree of 4). We perform enough trials such that the total
runtime per kernel is roughly one hour. The average runtimes reported exclude
graph building time and the first 1/4 of trials, which are considered warmup
trials.

XGBoost is a widely used gradient-boosting library implemented using C++
commonly executed on CPU systems [2, 26]. We focus on XGBoost training, which
is known for consuming a large amount of memory [67]. To evaluate FreqTier, we
use portions of the Criteo Click Logs dataset [12] and train for 400 rounds
using 16 threads, amounting to roughly one hour of total execution time.

##  VII Evaluation

TABLE II: CacheLib CDN workload performance. P50 GET latency (µs) / % all-
local
---
Config | %local | FreqTier | AutoNUMA | TPP | HeMem
1:32 | 6% | 4.1 / 89.1% | 4.4 / 83.4% | 4.7 / 76.9% | 4.8 / 75.7%
1:16 | 12% | 3.9 / 93.2% | 4.2 / 87.2% | 4.6 / 79.7% | 4.7 / 77.0%
1:8 | 24% | 3.9 / 94.0% | 4.0 / 91.0% | 4.3 / 84.3% | 4.9 / 74.8%
Throughput (Mop/s) / % all-local
1:32 | 6% | 0.87 / 85.9% | 0.84 / 82.9% | 0.72 / 71.0% | 0.82 / 80.6%
1:16 | 12% | 0.88 / 86.9% | 0.86 / 85.0% | 0.73 / 72.3% | 0.83 / 81.4%
1:8 | 24% | 0.90 / 88.8% | 0.90 / 88.4% | 0.76 / 74.8% | 0.80 / 79.1%
TABLE III: CacheLib social-graph workload performance. P50 GET latency (µs) /
% all-local
---
Config | %local | FreqTier | AutoNUMA | TPP | HeMem
1:32 | 6% | 0.82 / 90.6% | 0.89 / 83.6% | 0.94 / 79.2% | 0.93 / 79.9%
1:16 | 12% | 0.79 / 94.8% | 0.81 / 91.4% | 0.88 / 85.1% | 0.90 / 82.6%
1:8 | 24% | 0.78 / 96.0% | 0.80 / 92.9% | 0.82 / 90.4% | 0.91 / 82.2%
Throughput (Mop/s)
1:32 | 6% | 5.6 / 95.6% | 5.2 / 87.7% | 4.6 / 77.8% | 5.0 / 84.7%
1:16 | 12% | 5.8 / 97.4% | 5.5 / 93.1% | 4.8 / 82.0% | 5.1 / 86.2%
1:8 | 24% | 5.8 / 98.4% | 5.6 / 95.3% | 5.0 / 85.3% | 5.0 / 83.8%

###  VII-A Performance Comparisons

We first present results for configuration CXL-1. For all workloads, we list
the results in Table II–V, where %all-local indicates relative performance
over all-local configurations, and %local column indicates the percentage of
local DRAM capacity over the total application memory footprint. The %Config
column indicates the ratio between local DRAM and CXL-memory capacity.

CacheLib. Tables II and III show the performance of FreqTier compared to prior
works at three different local:CXL memory capacity ratios (Config column).
Both latency and throughput results are steady-state averages (the maximum
relative standard error is 0.29%). We make three observations from this
result.

1) Under the same local:CXL memory ratios, FreqTier outperforms AutoNUMA, TPP,
and HeMem in all experiments. On average, FreqTier outperforms AutoNUMA, TPP,
and HeMem by 5.4%, 12.7%, and 18.2% respectively in terms of P50 latency, and
3.9%, 19.5%, and 11.5% in terms of throughput. This improvement is largely due
to FreqTier’s ability to more accurately identify hot and cold pages. Figure 9
compares the hit ratio of FreqTier against baselines at different local DRAM
sizes. With 16GB of local DRAM, FreqTier produces on average 20% and 21%
higher local DRAM hit ratio than AutoNUMA and TPP respectively. HeMem achieves
higher hit ratios than AutoNUMA and TPP as its frequency-based tiering policy
can more accurately identify hot pages. However, HeMem’s high runtime overhead
interferes with the application execution, thus resulting in lower
performance. We evaluate HeMem’s memory overhead in Section VII-C.

2) FreqTier often requires 2$\times$ less local DRAM than AutoNUMA to achieve
the same level of performance. For example, on the CDN workload, FreqTier with
1:32 configuration (16GB local DRAM) outperforms AutoNUMA with $2\times$ local
DRAM. Similarly, on the social graph workload, FreqTier with 1:16
configuration outperforms AutoNUMA with 1:8, resulting in a 2$\times$
reduction in local DRAM cost.

3) FreqTier can achieve close-to-optimal performance with low amounts of local
DRAM. For social graph, FreqTier only requires 16GB (1:32 ratio) of local DRAM
to achieve 90% of all-local performance and 64GB to achieve 95% in terms of
both P50 latency and throughput. This confirms the high level of access
skewness within the memory access pattern.

Figure 9: Local DRAM hit ratio on CacheLib workloads.

GAP graph processing. Table IV shows the average runtime of FreqTier compared
to AutoNUMA and TPP. The maximum relative standard error for CC, BC, and BFS
are 0.6%, 1.8%, and 3.7%, respectively. For all three kernels, FreqTier with
1:16 configuration outperforms AutoNUMA with 1:8, reducing the amount of local
DRAM by 2$\times$. In the 1:32 configuration, FreqTier outperforms AutoNUMA by
3.8%, 17.3%, and 18.2% for BC, BFS, and CC respectively. At 1:16, the
performance improvements drop to 6.0%, 3.9%, 2.4%. At 1:8, FreqTier and
AutoNUMA have similar performances. We again observe that the advantage of
FreqTier increases with a lower amount of local DRAM, which is consistent with
the observation from CacheLib.

TABLE IV: GAP workload performance. Config | %local | FreqTier | AutoNUMA | TPP | HeMem
---|---|---|---|---|---
BC execution time (s) / % all-local
1:32 | 5% | 483 / 86.6% | 501 / 83.4% | 624 / 66.9% | 649 / 64.3%
1:16 | 10% | 419 / 99.7% | 444 / 94.0% | 583 / 72.6% | 641 / 65.2%
1:8 | 19% | 418 / 99.9% | 425 / 98.3% | 510 / 81.8% | 648 / 64.5%
BFS execution time (s) / % all-local
1:32 | 5% | 8.1 / 80.7% | 9.6 / 68.8% | 15.5 / 42.3% | 11.9 / 55.4%
1:16 | 10% | 7.1 / 92.2% | 7.4 / 88.8% | 14.5 / 45.5% | 11.8 / 55.7%
1:8 | 19% | 6.6 / 99.9% | 7.3 / 90.1% | 9.9 / 66.2% | 11.7 / 56.2%
CC execution time (s) / % all-local
1:32 | 5% | 11.8 / 92.3% | 13.9 / 78.1% | 12.9 / 84.0% | 19.3 / 56.2%
1:16 | 10% | 11.2 / 96.7% | 11.5 / 94.4% | 13.5 / 80.4% | 19.4 / 56.0%
1:8 | 19% | 11.7 / 93.1% | 11.4 / 94.9% | 13.3 / 81.6% | 19.6 / 55.6%
TABLE V: XGBoost performance. Average runtime per boosting round (s) / % all-
local
---
Config | %local | FreqTier | AutoNUMA | TPP | HeMem
1:32 | 6.5% | 6.0 / 95.9% | 6.6 / 88.3% | 12.3 / 47.1% | 8.4 / 68.9%
1:16 | 13% | 5.9 / 97.5% | 6.2 / 93.6% | 10.7 / 54.1% | 7.9 / 73.0%
1:8 | 26% | 5.9 / 98.3% | 6.0 / 97.3% | 7.4 / 78.8% | 8.4 / 69.1%

XGBoost. Table V shows the average runtime per training round. The maximum
relative standard errors are 1.4%, 1.3%, 3.9% for FreqTier, AutoNUMA, and TPP.
Under the 1:32 configuration, FreqTier achieves 95.9% of all-local
performance, 7.6%, 49%, and 27% higher than AutoNUMA, TPP, and HeMem
respectively. To achieve 95% performance of all-local, FreqTier only requires
16GB of local DRAM while AutoNUMA requires 64GB.

We conclude that FreqTier is an effective tiering system that can improve
large memory workload performance and significantly reduce the amount of local
DRAM required.

###  VII-B Different CXL Configurations

In this section, we investigate whether FreqTier’s performance improvements
can be generalized to CXL memory devices with different performance
characteristics. Using configuration CXL-2, we re-evaluate all workloads. To
accommodate the reduced CXL memory capacity (64GB), we scale down the memory
footprint of each application. All experiments use 8GB of local DRAM. We
compare the performance of FreqTier and AutoNUMA, the best-performing
baseline, in Figure 10. We observe that FreqTier outperforms AutoNUMA on all
workloads (on average 1.14$\times$). This is largely because FreqTier achieves
a higher local DRAM hit ratio, which is independent of the CXL memory
bandwidth. We conclude that the performance benefits of FreqTier can be
generalized to CXL memory devices with different performances.

Figure 10: Performance in low-bandwidth CXL configuration.

###  VII-C Memory Overhead

In this section, we compare the memory overhead of FreqTier and HeMem, another
frequency-based tiering system.

The memory consumption of FreqTier consists of two main parts, the CBF and
perf ring buffers. The size of CBF is discussed in Section VII-E. The perf
ring buffers are used by Linux perf to temporarily store collected memory
access samples. FreqTier allocates 512KB111The perf ring buffers should be
sized based on the sampling frequency. In our experiments, 512KB ring buffer
is sufficiently large to minimize the number of lost samples at 100kHz
sampling frequency. per hardware counter per core. With 16 physical cores and
2 counters per core, the perf ring buffer consumes 16MB in total. In the case
of in-memory caching workloads with a 267GB memory footprint, FreqTier
consumes less than 100MB of memory, amounting to less than 0.04% of memory
overhead.

In comparison, HeMem maintains 168 bytes of metadata for each page. In total,
HeMem requires more than 11GB of memory to manage pages in a 267GB workload,
amounting to 110$\times$ more memory used than FreqTier.

###  VII-D Adapting to Changing Access Distribution

Access distribution may change over time. For example, Meta reports that its
CacheLib workloads experience a high degree of churn [7], i.e., the change in
key popularity. The tiering system must keep the recorded frequencies fresh in
order to effectively identify and promote new hot data.

To evaluate how effectively FreqTier can detect and adapt to changes in memory
access distribution, we modify the CacheLib benchmark framework to change the
item access distribution. In this experiment, we use the popularity and size
distribution from the original CDN workload provided by Meta. First, all items
are inserted into the cache. During the first phase, we redirect all generated
accesses to the first half of items. From 430 seconds onward, we redirect all
accesses to the second half of items. This represents a worst-case
distribution change, where all data accessed during the first phase become
cold in the second phase.

Figure 11 shows how well FreqTier and the fastest baseline AutoNUMA adjust to
the changing distribution under the 1:32 configuration. After the distribution
changes, FreqTier’s low overhead monitoring mechanism detects this change in
distribution within 30 seconds and increases the hardware counter sample
frequency. From 500 to 1100 seconds, FreqTier identifies the new set of hot
pages while demoting cold pages out of the local DRAM. FreqTier experiences
slightly higher P50 latency than AutoNUMA from 500 to 900 seconds largely due
to the overhead associated with counter sampling. That being said, after the
temporary slowdown, FreqTier continues to outperform AutoNUMA. We conclude
that FreqTier can effectively detect and adjust to changing access
distributions.

Figure 11: Changing access distribution. Figure 12: FreqTier sensitivity to
counting bloom filter size. All performance numbers are relative to the 128MB
configuration. Higher is better for both P50 and throughput series.

###  VII-E Sensitivity Studies

####  VII-E1 Counting Bloom Filter size

To understand the effect of different CBF sizes on FreqTier performance, we
conduct a sensitivity study. We evaluate the performance of two in-memory
caching workloads, CDN and social-graph, under various CBF sizes. Experiment
setups are identical to those described in Section VII-A. Figure 12 shows the
results. As expected, reducing the CBF size generally leads to lower overall
application performance due to lower tracking accuracy caused by hash
collisions. On the other hand, increasing the CBF size beyond 128MB does not
improve performance. For both workloads, each with 256GB memory footprint,
using a 32MB CBF is sufficient to achieve accurate frequency tracking.
FreqTier is a flexible framework. To accommodate workloads with larger memory
footprints, the CBF size can be easily scaled.

####  VII-E2 Sample batch size

FreqTier processes memory access samples in batches. Figure 13 shows the P50
latency and throughput of FreqTier under various batch sizes. In general,
higher batch sizes achieve lower P50 latencies due to amortized system call
overhead. However, a larger batch size also incurs higher memory overhead,
since an entire batch of samples must be stored at once. The memory overhead
is equal to 16 bytes times the sample batch size. FreqTier uses the default
batch size of 100k, which achieves a balance between performance and memory
overhead.

Figure 13: FreqTier sensitivity to sample batch size, normalized to batch size
1. Evaluated on CacheLib CDN with 1:32 local DRAM to CXL-memory ratio.  Figure
14: Access frequency distribution captured by FreqTier on CacheLib and GAP
workloads using 1:32 local DRAM to CXL-memory ratio.

####  VII-E3 Number of bits per frequency counter:

In the CBF, the number of bits allocated for each counter determines the
maximum frequency count any page can have. For example, if 4 bits are
allocated for each counter, then all pages with a frequency count $\geq$ 15
are treated equally. To determine the appropriate max frequency count, we
characterize the access frequency distribution of the target workloads. Figure
14 shows the cumulative distributions of pages in the CBF. For each workload,
one frequency distribution is recorded for every 100k samples. The
distribution with the highest number of pages having frequency $=$ 15 is
selected.

We observe that the amount of pages with frequency $\geq$ 15 is less than
2$\%$ among our workloads. Since the ratio between local DRAM and CXL-memory
is usually much larger than 2%, additional frequency counter bits is
unnecessary – pages with access count $\geq$ 15 can be safely be classified as
hot and placed in the local DRAM. Thus, FreqTier allocates 4 bits for each
frequency counter by default. Similar to other parameters, the number of bits
can also be scaled if needed.

##  VIII Discussions

##### Non-skewed Applications

FreqTier and frequency-based tiering systems in general are intended for
applications with skewed memory access distributions. Extensive studies in the
literature have shown that this is true for a significant portion of modern
large memory workloads [7, 6, 70]. Applications that do not exhibit this
property will see little performance benefits from FreqTier.

##### CXL Memory Pooling

This work emulates CXL 1.1 due to the lack of commercial CXL devices on the
market. Newer generations of CXL standards, namely CXL 2.0 and 3.0, extend
CXL’s tiering capability to memory pooling across multi-host systems. This
work focuses on single-machine memory expansion, which we view as orthogonal
to multi-host tiering. Fundamentally, FreqTier aims to address the problem of
identifying hot/cold data, which is also applicable to multi-host tiering. We
leave such extensions to future works.

##### Userspace vs. Kernel Tiering

We implement FreqTier in the userspace due to its flexibility. This advantage
is recognized by both prior memory tiering systems [54, 57] and software
systems such as userspace networking systems [36, 13] and file systems [4].
Kernel-based memory tiering has its advantages, such as avoiding context-
switch overheads and allowing for the management of multiple applications.
However, the ideas behind FreqTier are not limited to userspace and can be
implemented in the kernel.

##  IX Related Works

##### Memory Tiering Systems.

Tiered memory systems can utilize various types of memory technologies as the
slower tier. Examples include solid state drives [10, 19, 28] or persistent
memory [54, 37, 1, 29, 15, 30]. The slow-tier memory can be placed on the same
server as the fast-tier memory (near memory) [54, 40, 24], or on remote
servers (far memory) [69, 57, 72], where memory accesses are served from the
network. In terms of programmability, tiering systems range from application-
transparent [56, 40, 54, 31, 65] to requiring manual modifications to the
application [57, 52].

AutoNUMA [24] and TPP [40] are recency-based CXL tiering systems. HeMem [54]
is a persistent memory tiering system that performs frequency-based tiering at
the page granularity using a hash table. MULTI-CLOCK [37] differentiates
between pages accessed exactly once and those accessed more than once.
However, MULTI-CLOCK treats all pages accessed more than once equally,
resulting in low classification accuracy. DAOS (Data Access-aware Operating
System) [49, 62, 61] performs access monitoring and migrations in units of
variable-sized memory regions, where all pages in the same region share the
same access frequency. These prior works either cannot accurately identify
hot/cold pages or require large memory overhead. FreqTier addresses both of
these challenges.

Application-guided tiering systems such as memkind [52], SMDK [47], Unimem
[66], Xmem [14], and 2PP [64] offer the potential for the tiering system to
understand application semantics through profiling or application
modifications. [27, 42, 53] accelerates tiering process using specialized
hardware modifications. However, both approaches have limited scalability in
data center environments, as they require modifying the application or
additional hardware support. In contrast, FreqTier does not require any
hardware or application changes.

##### Counting Bloom Filter.

Prior works on CBF are either theoretical [46, 55] or apply the CBF in domains
such as database, in-memory caching, and network communication [3, 16, 35,
34]. While we drew inspiration from prior works on CBF, FreqTier has several
key differences. FreqTier is built based on the key observation that memory
tiering can tolerate a small amount of tracking inaccuracy. To our knowledge,
FreqTier is the first work to practically apply the unique advantages of CBF
to memory tiering. While in-memory cache replacement policies such as TinyLFU
[16] focus on utilizing CBF to find the optimal cache entries to evict, the
CBF itself is not capable of performing memory tiering. FreqTier introduces
efficient promotion/demotion mechanisms based on CBF’s unique properties,
alongside flexible migration policies to make fine-grained frequency-based
tiering practical.

##### Disaggregated Memory Systems.

Memory disaggregation [72, 57, 69, 39] expands the main memory capacity by
placing additional memory modules in remote servers. Similar to memory tiering
on a single machine, disaggregated memory systems also aim to place the
hottest data on the local fast-tier memory. Our work is orthogonal to works in
this domain. In addition to locally-attached CXL memory managed by FreqTier,
the target server can further expand its memory capacity through disaggregated
tiering systems.

##  X Conclusion

We propose FreqTier, an application-transparent fine-grained frequency-based
tiering system. FreqTier achieves both high classification accuracy and low
overhead by adopting novel probabilistic access frequency tracking and
flexible migration policy. On in-memory caching workloads, FreqTier
outperforms prior works while using 4× less local DRAM memory. while requiring
less than 100MB of memory overhead. On graph analytic and gradient boosting
workloads, FreqTier outperforms prior works by 1.39$\times$ on average under
1:32 local DRAM to CXL-memory ratio. FreqTier’s performance gains generalize
to a CXL device with lower bandwidth, outperforming AutoNUMA by an average of
1.14$\times$.

## References

  * [1] N. Agarwal and T. F. Wenisch, “Thermostat: Application-transparent page management for two-tiered main memory,” in _Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)_. New York, NY, USA: Association for Computing Machinery, 2017, p. 631–644.
  * [2] Amazon, “Xgboost algorithm,” https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html#Instance-XGBoost-training-cpu.
  * [3] Apache Software Foundation, “Class countingbloomfilter,” https://hadoop.apache.org/docs/r2.7.5/api/org/apache/hadoop/util/bloom/CountingBloomFilter.html.
  * [4] T. L. K. Archives, “Fuse,” https://www.kernel.org/doc/html/next/filesystems/fuse.html.
  * [5] S. Beamer, K. Asanović, and D. Patterson, “GAP benchmark suite,” https://github.com/sbeamer/gapbs.
  * [6] S. Beamer, K. Asanović, and D. Patterson, “The gap benchmark suite,” 2015. [Online]. Available: https://arxiv.org/abs/1508.03619
  * [7] B. Berg, D. S. Berger, S. McAllister, I. Grosof, S. Gunasekar, J. Lu, M. Uhlar, J. Carrig, N. Beckmann, M. Harchol-Balter, and G. R. Ganger, “The CacheLib caching engine: Design and experiences at scale,” in _14th USENIX Symposium on Operating Systems Design and Implementation (OSDI)_. USENIX Association, Nov. 2020, pp. 753–768.
  * [8] A. Broder and M. Mitzenmacher, “Survey: Network applications of bloom filters: A survey.” _Internet Mathematics_ , vol. 1, 11 2003.
  * [9] CacheLib, “Cachebench overview,” https://cachelib.org/docs/Cache_Library_User_Guides/Cachebench_Overview/.
  * [10] CacheLib, “Hybrid cache,” https://cachelib.org/docs/Cache_Library_Architecture_Guide/hybrid_cache.
  * [11] J. Corbet, “Two memory-tiering patch sets,” https://lwn.net/Articles/898766/.
  * [12] Criteo, “Criteo 1TB click logs dataset,” https://ailab.criteo.com/criteo-1tb-click-logs-dataset/.
  * [13] DPDK, “Data plane development kit,” https://github.com/DPDK/dpdk.
  * [14] S. R. Dulloor, A. Roy, Z. Zhao, N. Sundaram, N. Satish, R. Sankaran, J. Jackson, and K. Schwan, “Data tiering in heterogeneous memory systems,” in _Proceedings of the Eleventh European Conference on Computer Systems (EuroSys)_. New York, NY, USA: Association for Computing Machinery, 2016.
  * [15] P. Duraisamy, W. Xu, S. Hare, R. Rajwar, D. Culler, Z. Xu, J. Fan, C. Kennelly, B. McCloskey, D. Mijailovic, B. Morris, C. Mukherjee, J. Ren, G. Thelen, P. Turner, C. Villavieja, P. Ranganathan, and A. Vahdat, “Towards an adaptable systems architecture for memory tiering at warehouse-scale,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)_. New York, NY, USA: Association for Computing Machinery, 2023, p. 727–741.
  * [16] G. Einziger, R. Friedman, and B. Manes, “TinyLFU: A highly efficient cache admission policy,” _ACM Trans. Storage_ , vol. 13, no. 4, nov 2017. [Online]. Available: https://doi.org/10.1145/3149371
  * [17] Facebook, “Cachelib,” https://github.com/facebook/CacheLib.
  * [18] L. Fan, P. Cao, J. Almeida, and A. Z. Broder, “Summary cache: A scalable wide-area web cache sharing protocol,” _SIGCOMM Comput. Commun. Rev._ , vol. 28, no. 4, p. 254–265, oct 1998.
  * [19] H. Herodotou and E. Kakoulli, “Automating distributed tiered storage management in cluster computing,” _Proc. VLDB Endow._ , vol. 13, no. 1, p. 43–56, sep 2019. [Online]. Available: https://doi.org/10.14778/3357377.3357381
  * [20] Y. Huang, “memory tiering: hot page selection with hint page fault latency,”<EMAIL_ADDRESS>
  * [21] Y. Huang, “memory tiering: skip to scan fast memory,” https://github.com/torvalds/linux/commit/a1a3a2fc304df326ff67a1814364f640f2d5121c.
  * [22] Y. Huang, “Numa balancing: optimize page placement for memory tiering system,” https://github.com/torvalds/linux/commit/c574bbe917036c8968b984c82c7b13194fe5ce98.
  * [23] Y. Huang, “Optimize page placement in tiered memory system,” https://lpc.events/event/11/contributions/967/attachments/811/1654/Optimize%20Page%20Placement%20in%20Tiered%20Memory%20System.pdf.
  * [24] Y. Huang, “[patch -v4 0/3] memory tiering: hot page selection,”<EMAIL_ADDRESS>
  * [25] Intel, “Intel® optane™ persistent memory,” https://www.intel.com/content/www/us/en/products/details/memory-storage/optane-dc-persistent-memory.htmlk.
  * [26] Intel, “Maximize your CPU resources for XGBoost training and inference,” https://www.intel.com/content/www/us/en/developer/videos/maximize-cpu-resources-xgboost-training-inference.html#gs.47qye6.
  * [27] Intel, “Why is the intel optane persistent memory in memory mode not persistent?” https://www.intel.com/content/www/us/en/support/articles/000055895/memory-and-storage/intel-optane-persistent-memory.html.
  * [28] E. Kakoulli and H. Herodotou, “OctopusFS: A distributed file system with tiered storage management,” in _Proceedings of the 2017 ACM International Conference on Management of Data (SIGMOD)_. New York, NY, USA: Association for Computing Machinery, 2017, p. 65–78. [Online]. Available: https://doi.org/10.1145/3035918.3064023
  * [29] S. Kannan, A. Gavrilovska, V. Gupta, and K. Schwan, “HeteroOS - OS design for heterogeneous memory management in datacenter,” in _ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA)_ , 2017, pp. 521–534.
  * [30] H. T. Kassa, J. Akers, M. Ghosh, Z. Cao, V. Gogte, and R. Dreslinski, “Improving performance of flash based Key-Value stores using storage class memory as a volatile memory extension,” in _USENIX Annual Technical Conference (ATC)_. USENIX Association, Jul. 2021, pp. 821–837. [Online]. Available: https://www.usenix.org/conference/atc21/presentation/kassa
  * [31] A. Lagar-Cavilla, J. Ahn, S. Souhlal, N. Agarwal, R. Burny, S. Butt, J. Chang, A. Chaugule, N. Deng, J. Shahid, G. Thelen, K. A. Yurtsever, Y. Zhao, and P. Ranganathan, “Software-defined far memory in warehouse-scale computers,” in _Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)_. New York, NY, USA: Association for Computing Machinery, 2019, p. 317–330. [Online]. Available: https://doi.org/10.1145/3297858.3304053
  * [32] T. Lee and Y. I. Eom, “Optimizing the page hotness measurement with re-fault latency for tiered memory systems,” in _IEEE International Conference on Big Data and Smart Computing (BigComp)_ , 2022, pp. 275–279.
  * [33] H. Li, D. S. Berger, L. Hsu, D. Ernst, P. Zardoshti, S. Novakovic, M. Shah, S. Rajadnya, S. Lee, I. Agarwal, M. D. Hill, M. Fontoura, and R. Bianchini, “Pond: CXL-based memory pooling systems for cloud platforms,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)_. New York, NY, USA: Association for Computing Machinery, 2023, p. 574–587. [Online]. Available: https://doi.org/10.1145/3575693.3578835
  * [34] W. Liu, Z. Xu, J. Tian, and Y. Zhang, “Towards in-network compact representation: Mergeable counting bloom filter vis cuckoo scheduling,” _IEEE Access_ , vol. PP, pp. 1–1, 04 2021.
  * [35] B. Manes, “Caffeine,” https://github.com/ben-manes/caffeine.
  * [36] M. Marty, M. de Kruijf, J. Adriaens, C. Alfeld, S. Bauer, C. Contavalli, M. Dalton, N. Dukkipati, W. C. Evans, S. Gribble, N. Kidd, R. Kononov, G. Kumar, C. Mauer, E. Musick, L. Olson, M. Ryan, E. Rubow, K. Springborn, P. Turner, V. Valancius, X. Wang, and A. Vahdat, “Snap: a microkernel approach to host networking,” in _In ACM SIGOPS 27th Symposium on Operating Systems Principles (SOSP)_ , 2019.
  * [37] A. Maruf, A. Ghosh, J. Bhimani, D. Campello, A. Rudoff, and R. Rangaswami, “MULTI-CLOCK: Dynamic tiering for hybrid memory systems,” in _IEEE International Symposium on High-Performance Computer Architecture (HPCA)_ , 2022, pp. 925–937.
  * [38] H. A. Maruf, “Transparent page placement for tiered-memory,”<EMAIL_ADDRESS>
  * [39] H. A. Maruf and M. Chowdhury, “Effectively prefetching remote memory with leap,” in _USENIX Annual Technical Conference (ATC)_. USENIX Association, Jul. 2020, pp. 843–857. [Online]. Available: https://www.usenix.org/conference/atc20/presentation/al-maruf
  * [40] H. A. Maruf, H. Wang, A. Dhanotia, J. Weiner, N. Agarwal, P. Bhattacharya, C. Petersen, M. Chowdhury, S. Kanaujia, and P. Chauhan, “TPP: Transparent page placement for CXL-enabled tiered-memory,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)_. New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3582016.3582063
  * [41] G. Marçais and C. Kingsford, “A fast, lock-free approach for efficient parallel counting of occurrences of k-mers,” _Bioinformatics_ , vol. 27, no. 6, pp. 764–770, 01 2011. [Online]. Available: https://doi.org/10.1093/bioinformatics/btr011
  * [42] M. R. Meswani, S. Blagodurov, D. Roberts, J. Slice, M. Ignatowski, and G. H. Loh, “Heterogeneous memory architectures: A hw/sw approach for mixing die-stacked and off-package memories,” in _IEEE 21st International Symposium on High Performance Computer Architecture (HPCA)_ , 2015, pp. 126–136.
  * [43] Micron, “CZ120 memory expansion module,” https://www.micron.com/solutions/server/cxl#:~:text=CXL%20memory%20expansion%20serves%20as,workloads%20for%20CXL%2Denabled%20servers.
  * [44] D. Moura, D. Mosse, and V. Petrucci, “Performance characterization of AutoNUMA memory tiering on graph analytics,” in _IEEE International Symposium on Workload Characterization (IISWC)_. IEEE, nov 2022. [Online]. Available: https://doi.org/10.1109%2Fiiswc55918.2022.00024
  * [45] O. Mutlu, “Memory scaling: A systems architecture perspective,” https://users.ece.cmu.edu/~omutlu/pub/mutlu_memory-scaling_memcon13_talk.pdf.
  * [46] S. Nayak and R. Patgiri, “countbf: A general-purpose high accuracy and space efficient counting bloom filter,” in _17th International Conference on Network and Service Management (CNSM)_ , 2021.
  * [47] OpenMPDK, “Scalable memory development kit,” https://github.com/OpenMPDK/SMDK.
  * [48] Panmnesia, “Panmnesia technologies,” https://panmnesia.com/#technology.
  * [49] S. Park, M. Bhowmik, and A. Uta, “Daos: Data access-aware operating system,” in _Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing (HPDC)_. New York, NY, USA: Association for Computing Machinery, 2022, p. 4–15. [Online]. Available: https://doi.org/10.1145/3502181.3531466
  * [50] J. T. Pawlowski, “Prospects for memory,” https://passlab.github.io/mchpc/mchpc2019/presentations/MCHPC_Pawlowski_keynote.pdf.
  * [51] perf stat, “perf-stat(1) — linux manual page,” https://man7.org/linux/man-pages/man1/perf-stat.1.html.
  * [52] pmem.io, “memkind,” https://pmem.io/memkind/.
  * [53] L. E. Ramos, E. Gorbatov, and R. Bianchini, “Page placement in hybrid memory systems,” in _Proceedings of the International Conference on Supercomputing (ICS)_. New York, NY, USA: Association for Computing Machinery, 2011, p. 85–95. [Online]. Available: https://doi.org/10.1145/1995896.1995911
  * [54] A. Raybuck, T. Stamler, W. Zhang, M. Erez, and S. Peter, “HeMem: Scalable tiered memory management for big data applications and real NVM,” in _Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles (SOSP)_. New York, NY, USA: Association for Computing Machinery, 2021, p. 392–407. [Online]. Available: https://doi.org/10.1145/3477132.3483550
  * [55] P. Reviriego and O. Rottenstreich, “The tandem counting bloom filter - it takes two counters to tango,” _IEEE/ACM Transactions on Networking_ , vol. 27, no. 6, pp. 2252–2265, 2019.
  * [56] V. C. Rik van Riel, “Automatic numa balancing,” https://www.linux-kvm.org/images/7/75/01x07b-NumaAutobalancing.pdf, 2014.
  * [57] Z. Ruan, M. Schwarzkopf, M. K. Aguilera, and A. Belay, “AIFM: High-Performance, Application-Integrated far memory,” in _14th USENIX Symposium on Operating Systems Design and Implementation (OSDI)_. USENIX Association, Nov. 2020, pp. 315–332. [Online]. Available: https://www.usenix.org/conference/osdi20/presentation/ruan
  * [58] Samsung, “Samsung electronics introduces industry’s first 512gb CXL memory module,” https://news.samsung.com/us/samsung-electronics-introduces-industrys-first-512gb-cxl-memory-module/.
  * [59] D. D. Sharma, “Introduction to compute express link,” https://docs.wixstatic.com/ugd/0c1418_d9878707bbb7427786b70c3c91d5fbd1.pdf.
  * [60] Y. Sun, Y. Yuan, Z. Yu, R. Kuper, C. Song, J. Huang, H. Ji, S. Agarwal, J. Lou, I. Jeong, R. Wang, J. H. Ahn, T. Xu, and K. N. S., “Demystifying cxl memory with genuine cxl-ready systems and devices,” in _2023 56th IEEE/ACM International Symposium on Microarchitecture (MICRO)_ , 2023.
  * [61] The kernel development community, “Damon-based reclamation,” https://www.kernel.org/doc/html/next/admin-guide/mm/damon/reclaim.html?highlight=damon.
  * [62] The kernel development community, “DAMON: Data access monitor,” https://www.kernel.org/doc/html/next/mm/damon/index.html?highlight=damon.
  * [63] K. Viswanathan, “Intel memory latency checker v3.11,” https://www.intel.com/content/www/us/en/developer/articles/tool/intelr-memory-latency-checker.html, 2021.
  * [64] W. Wei, D. Jiang, S. A. McKee, J. Xiong, and M. Chen, “Exploiting program semantics to place data in hybrid memory,” in _International Conference on Parallel Architecture and Compilation (PACT)_ , 2015, pp. 163–173.
  * [65] J. Weiner, N. Agarwal, D. Schatzberg, L. Yang, H. Wang, B. Sanouillet, B. Sharma, T. Heo, M. Jain, C. Tang, and D. Skarlatos, “TMO: Transparent memory offloading in datacenters,” in _Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)_. New York, NY, USA: Association for Computing Machinery, 2022, p. 609–621. [Online]. Available: https://doi.org/10.1145/3503222.3507731
  * [66] K. Wu, Y. Huang, and D. Li, “Unimem: Runtime data management on non-volatile memory-based heterogeneous main memory,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC)_. New York, NY, USA: Association for Computing Machinery, 2017. [Online]. Available: https://doi.org/10.1145/3126908.3126923
  * [67] XGBoost, “Using XGBoost external memory version,” https://xgboost.readthedocs.io/en/stable/tutorials/external_memory.html.
  * [68] XGBoost, “XGBoost: eXtreme gradient boosting,” https://github.com/dmlc/xgboost.
  * [69] Z. Yan, D. Lustig, D. Nellans, and A. Bhattacharjee, “Nimble page management for tiered memory systems,” in _Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)_. New York, NY, USA: Association for Computing Machinery, 2019, p. 331–345.
  * [70] J. Yang, Y. Yue, and K. V. Rashmi, “A large scale analysis of hundreds of in-memory cache clusters at Twitter,” in _14th USENIX Symposium on Operating Systems Design and Implementation (OSDI)_. USENIX Association, Nov. 2020, pp. 191–208.
  * [71] H. Ying, “autonuma: Optimize memory placement for memory tiering system,” https://lwn.net/Articles/845102/.
  * [72] Y. Zhou, H. M. G. Wassel, S. Liu, J. Gao, J. Mickens, M. Yu, C. Kennelly, P. Turner, D. E. Culler, H. M. Levy, and A. Vahdat, “Carbink: Fault-tolerant far memory,” in _16th USENIX Symposium on Operating Systems Design and Implementation (OSDI)_ , Carlsbad, CA, Jul. 2022, pp. 55–71.


# Turning Text and Imagery into Captivating Visual Video

Mingming Wang, Elijah Miller<EMAIL_ADDRESS>
(2024)

###### Abstract.

The ability to visualize a structure from multiple perspectives is crucial for
comprehensive planning and presentation. This paper introduces an advanced
application of generative models, akin to Stable Video Diffusion, tailored for
architectural visualization. We explore the potential of these models to
create consistent multi-perspective videos of buildings from single images and
to generate design videos directly from textual descriptions. The proposed
method enhances the design process by offering rapid prototyping, cost and
time efficiency, and an enriched creative space for architects and designers.
By harnessing the power of AI, our approach not only accelerates the
visualization of architectural concepts but also enables a more interactive
and immersive experience for clients and stakeholders. This advancement in
architectural visualization represents a significant leap forward, allowing
for a deeper exploration of design possibilities and a more effective
communication of complex architectural ideas.

video generation, Stable diffusion

††copyright: acmcopyright††journalyear: 2024††doi:
XXXXXXX.XXXXXXX††conference: ACM conference; 2024; ††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Modeling and simulation

##  1\. Introduction

The realm of architectural design (groat2013architectural, ;
aliakseyeu2006computer, ; akin1996frames, ; lomas2007architectural, ;
li2023archi, ) is inherently visual, with the ability to accurately and
compellingly represent a structure from various perspectives being paramount
to the success of any project. Traditional methods of architectural
visualization, while effective, can be time-consuming and resource-intensive,
often requiring specialized skills and extensive post-production work. With
the advent of artificial intelligence and the development of advanced
generative models, a new horizon in architectural visualization has emerged.

Diffusion-based methods (rombach2022high, ; song2020improved, ; imagen, ;
improvedddpm, ; ho2020denoising, ; song2020score, ) are favored for their
controllability, realism, and diversity, with applications in computer vision
tasks such as image editing (brooks2023instructpix2pix, ;
hertz2022prompt2prompt, ; li2023layerdiffusion, ; ruiz2023dreambooth, ;
tumanyan2023plug, ) and dense prediction (diffusiondet, ; li2023efficient, ;
diffmatch, ). They are also used in video synthesis (singer2022make, ;
miller2024enhanced; ho2022imagenvideo, ; videocontrolnet, ; videoLDM, ;
wang2018video2video, ; magicvideo, ; vdm, ) and 3D generation (magic3d, ;
dreamfusion, ; raj2023dreambooth3d, ; text-to-3d, ; tang2023dreamgaussian, ;
li2024generating, ; diffusion3d, ). In this paper, we present an innovative
approach to architectural visualization that leverages the capabilities of
models akin to Stable Video Diffusion (blattmann2023stable, ) to generate
multi-perspective videos of buildings from single images and to create design
videos directly from textual descriptions. We focus on the application of
these models in the context of architectural design, exploring how they can be
adapted and utilized to enhance the visualization process for architects,
designers, and stakeholders.

Our approach to image-based multi-perspective video generation involves a
meticulous preprocessing stage where architectural images are analyzed to
identify key structural elements and potential viewpoints. This is followed by
a sophisticated diffusion process that expands on the principles of SVD,
generating a sequence of videos that showcase the building from various angles
while maintaining visual coherence and continuity across frames.

For text-to-video generation in architectural design, we employ a model
trained to interpret and encode textual descriptions of architectural concepts
into a latent space. This encoded information serves as a foundation for the
model to synthesize videos that bring the described architectural designs to
life, offering a powerful tool for conceptual exploration and communication.

We also introduce innovations in camera motion control within the generated
videos, allowing for the manipulation of viewing angles and paths to simulate
realistic walkthroughs of the architectural designs. This feature is
particularly beneficial for virtual presentations and client demonstrations,
where the ability to navigate through a design is crucial.

Throughout the paper, we highlight the applications and benefits of our
approach, demonstrating its potential to revolutionize the field of
architectural visualization. From rapid prototyping to virtual presentations,
our method streamlines the design process, making high-quality architectural
visualization accessible and efficient.

Furthermore, we discuss the challenges and future work in this domain,
including the need for broader generalization across diverse architectural
styles and the integration of these models into existing design workflows. We
also consider the ethical implications and the importance of maintaining the
creative integrity of human designers in the age of AI.

Figure 1. We show the capabilities of the generative model in creating multi-
perspective architectural visualizations and transforming textual descriptions
into dynamic video content.

##  2\. Application Details

As shown in Fig. 1, our methodology is crafted on a foundation akin to Stable
Video Diffusion (SVD), tailored and enhanced to address the unique demands of
architectural visualization. The approach encompasses several pivotal
components:

Image-Based Multi-Perspective Video Generation:

Preprocessing and Viewpoint Analysis: We initiate the process by meticulously
preprocessing architectural images to extract essential structural elements
and pinpoint optimal viewpoints. Multi-Step Diffusion Process: Implementing a
sophisticated multi-step diffusion process, our model crafts a sequence of
videos that display the building from diverse vantage points, maintaining
visual uniformity across frames. Text-to-Video Generation for Architectural
Design:

Text Encoding and Semantic Understanding: The model is adeptly trained to
comprehend and encode textual depictions of architectural designs into a rich
latent space. Video Synthesis from Text: Capitalizing on this encoded data,
the model synthesizes videos that vividly materialize the architectural
designs as articulated in the text. Model Architecture and Training:

UNet with Temporal Extensions: We harness a UNet architecture, thoughtfully
expanded with temporal layers, to encapsulate the spatial and temporal
intricacies of architectural structures. Fine-Tuning for Architectural Data:
The model is subjected to fine-tuning using a meticulously curated dataset of
architectural images and videos, thereby enhancing its proficiency in
generating nuanced building perspectives. Innovations in Architectural
Visualization:

Controlled Camera Motion: We introduce advanced modules that facilitate
precise manipulation of camera motion within generated videos, offering
granular control over viewing angles and trajectories. Multi-View Consistency:
The model steadfastly ensures consistency across various views, effectively
simulating a seamless and realistic walkthrough of the architectural design.
Applications and Case Studies:

Rapid Prototyping: Designers can swiftly prototype architectural concepts
through our video generation framework, streamlining the iteration process and
fostering prompt feedback. Virtual Presentations: The videos generated serve
as an engaging medium for virtual presentations, allowing stakeholders to
immerse themselves in the design from a multitude of perspectives. Challenges
and Future Work:

Generalization Across Architectural Styles: We confront the challenge of
enhancing model generalization to encompass a wide array of architectural
styles and their inherent complexities. Integration with Design Workflows: We
are committed to investigating more profound integrations with current
architectural design and visualization workflows. By fusing the principles of
Stable Video Diffusion with the architectural design domain, our research
introduces an innovative multi-perspective visualization approach, set to
transform the conceptualization, development, and presentation paradigms for
architects and designers.

##  3\. Conclusion

In this paper, we introduce a application of diffusion-based models for
architectural visualization, demonstrating the potential to revolutionize how
architects and designers conceptualize and present their work. By harnessing
the capabilities of Stable Video Diffusion-like models, we’ve presented a
method that generates multi-perspective videos from single images and creates
design videos directly from textual descriptions. This approach not only
accelerates the design visualization process but also enhances the interactive
and immersive experience for clients and stakeholders. As we look to the
future, the integration of these models into standard design workflows and
their ethical implications will be critical areas of focus, ensuring that the
creative integrity of human designers remains at the forefront of the
industry’s evolution.

## References

  * (1) L. N. Groat and D. Wang, _Architectural research methods_. John Wiley & Sons, 2013.
  * (2) D. Aliakseyeu, J.-B. Martens, and M. Rauterberg, “A computer support tool for the early stages of architectural design,” _Interacting with Computers_ , vol. 18, no. 4, pp. 528–555, 2006.
  * (3) Ö. Akin and C. Akin, “Frames of reference in architectural design: analysing the hyperacclamation (aha-!),” _Design studies_ , vol. 17, no. 4, pp. 341–361, 1996.
  * (4) K. J. Lomas, “Architectural design of an advanced naturally ventilated building form,” _Energy and Buildings_ , vol. 39, no. 2, pp. 166–181, 2007.
  * (5) P. Li, B. Li, and Z. Li, “Sketch-to-Architecture: Generative AI-aided Architectural Design,” in _Pacific Graphics Short Papers and Posters_. The Eurographics Association, 2023.
  * (6) R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _CVPR_ , 2022.
  * (7) Y. Song and S. Ermon, “Improved techniques for training score-based generative models,” in _NeurIPS_ , 2020.
  * (8) C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans _et al._ , “Photorealistic text-to-image diffusion models with deep language understanding,” in _NeurIPS_ , 2022.
  * (9) A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in _ICML_ , 2021.
  * (10) J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in _NeurIPS_ , 2020.
  * (11) Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” in _ICLR_ , 2021.
  * (12) T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” in _CVPR_ , 2023.
  * (13) A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, “Prompt-to-prompt image editing with cross attention control,” in _ICLR_ , 2023.
  * (14) P. Li, Q. Huang, Y. Ding, and Z. Li, “Layerdiffusion: Layered controlled image editing with diffusion models,” in _SIGGRAPH Asia 2023 Technical Communications_ , 2023, pp. 1–4.
  * (15) N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,” in _CVPR_ , 2023.
  * (16) N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel, “Plug-and-play diffusion features for text-driven image-to-image translation,” in _CVPR_ , 2023.
  * (17) S. Chen, P. Sun, Y. Song, and P. Luo, “Diffusiondet: Diffusion model for object detection,” in _ICCV_ , 2022.
  * (18) P. Li and Z. Li, “Efficient temporal denoising for improved depth map applications,” in _Proc. Int. Conf. Learn. Representations, Tiny papers_ , 2023.
  * (19) J. Nam, G. Lee, S. Kim, H. Kim, H. Cho, S. Kim, and S. Kim, “Diffmatch: Diffusion model for dense matching,” _arXiv:2305.19094_ , 2023.
  * (20) U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni _et al._ , “Make-a-video: Text-to-video generation without text-video data,” in _ICLR_ , 2023.
  * (21) J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet _et al._ , “Imagen video: High definition video generation with diffusion models,” _arXiv:2210.02303_ , 2022.
  * (22) Z. Hu and D. Xu, “Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet,” _arXiv:2307.14073_ , 2023.
  * (23) A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis, “Align your latents: High-resolution video synthesis with latent diffusion models,” in _CVPR_ , 2023.
  * (24) T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, G. Liu, A. Tao, J. Kautz, and B. Catanzaro, “Video-to-video synthesis,” in _NeurIPS_ , 2018.
  * (25) D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng, “Magicvideo: Efficient video generation with latent diffusion models,” _arXiv:2211.11018_ , 2022.
  * (26) J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” in _NeurIPS_ , 2022.
  * (27) C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin, “Magic3d: High-resolution text-to-3d content creation,” in _CVPR_ , 2023.
  * (28) B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “Dreamfusion: Text-to-3d using 2d diffusion,” _arXiv preprint arXiv:2209.14988_ , 2022.
  * (29) A. Raj, S. Kaza, B. Poole, M. Niemeyer, N. Ruiz, B. Mildenhall, S. Zada, K. Aberman, M. Rubinstein, J. Barron _et al._ , “Dreambooth3d: Subject-driven text-to-3d generation,” _arXiv preprint arXiv:2303.13508_ , 2023.
  * (30) C. Li, C. Zhang, A. Waghwase, L.-H. Lee, F. Rameau, Y. Yang, S.-H. Bae, and C. S. Hong, “Generative ai meets 3d: A survey on text-to-3d in aigc era,” _arXiv:2305.06131_ , 2023.
  * (31) J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng, “Dreamgaussian: Generative gaussian splatting for efficient 3d content creation,” _arXiv preprint arXiv:2309.16653_ , 2023.
  * (32) P. Li and B. Li, “Generating daylight-driven architectural design via diffusion models,” _arXiv preprint arXiv:2404.13353_ , 2024.
  * (33) S. Luo and W. Hu, “Diffusion probabilistic models for 3d point cloud generation,” in _CVPR_ , 2021.
  * (34) A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts _et al._ , “Stable video diffusion: Scaling latent video diffusion models to large datasets,” _arXiv preprint arXiv:2311.15127_ , 2023.