# Lumina-Next : Making Lumina-T2X
Stronger and Faster with Next-DiT

Le Zhuo1 Ruoyi Du1,5∗ Han Xiao1,2∗ Yangguang Li1∗ Dongyang Liu1,2∗
Rongjie Huang1∗ Wenze Liu1∗ Lirui Zhao1 Fu-Yun Wang1 Zhanyu Ma5
Xu Luo1 Zehan Wang1 Kaipeng Zhang1 Xiangyang Zhu1 Si Liu4 Xiangyu Yue1,2
Dingning Liu1 Wanli Ouyang1 Ziwei Liu3 Yu Qiao122footnotemark: 2 Hongsheng
Li1,222footnotemark: 2 Peng Gao1 ∗
1Shanghai AI Laboratory 2The Chinese University of Hong Kong
3Nanyang Technological University 4Beihang University
5Beijing University of Posts and Telecommunications  Equal
ContributionCorresponding AuthorsProject Lead

###### Abstract

Lumina-T2X is a nascent family of Flow-based Large Diffusion Transformers
(Flag-DiT) that establishes a unified framework for transforming noise into
various modalities, such as images and videos, conditioned on text
instructions. Despite its promising capabilities, Lumina-T2X still encounters
challenges including training instability, slow inference, and extrapolation
artifacts. In this paper, we present Lumina-Next, an improved version of
Lumina-T2X, showcasing stronger generation performance with increased training
and inference efficiency. We begin with a comprehensive analysis of the Flag-
DiT architecture and identify several suboptimal components, which we address
by introducing the Next-DiT architecture with 3D RoPE and sandwich
normalizations. To enable better resolution extrapolation, we thoroughly
compare different context extrapolation methods applied to text-to-image
generation with 3D RoPE, and propose Frequency- and Time-Aware Scaled RoPE
tailored for diffusion transformers. Additionally, we introduce a sigmoid time
discretization schedule to reduce sampling steps in solving the Flow ODE and
the Context Drop method to merge redundant visual tokens for faster network
evaluation, effectively boosting the overall sampling speed. Thanks to these
improvements, Lumina-Next not only improves the quality and efficiency of
basic text-to-image generation but also demonstrates superior resolution
extrapolation capabilities and multilingual generation using decoder-based
LLMs as the text encoder, all in a zero-shot manner. To further validate
Lumina-Next as a versatile generative framework, we instantiate it on diverse
tasks including visual recognition, multi-view, audio, music, and point cloud
generation, showcasing strong performance across these domains. By releasing
all codes and model weights at https://github.com/Alpha-VLLM/Lumina-T2X, we
aim to advance the development of next-generation generative AI capable of
universal modeling.

##  1 Introduction

Scaling diffusion transformers has unveiled significant improvements in text-
conditional image and video generation [14, 13, 29, 32, 57]. Notably,
Lumina-T2X [32] introduce flow-based large diffusion transformer (Flag-DiT),
which has proven to be a stable, scalable, flexible, and training-efficient
architecture for generative modeling across various data domains. For
instance, the use of advanced normalization techniques like KQ-Norm and
RMSNorm enable stable mixed-precision training when scaling Lumina-T2X to a 5B
Flag-DiT with a 7B LLaMA [74]. Besides, the flow matching framework and Rotary
Position Embedding (RoPE) unlock the potential of Lumina-T2X to generate data
with arbitrary resolutions, aspect ratios, and durations during sampling.
While Lumina-T2X achieves superior visual aesthetics quality with remarkably
low training resources, it suffers from weak image-text alignment, slow
inference, and extrapolation artifacts due to inadequate training,
insufficient training data, and inappropriate context extension strategy.

Figure 1: As a foundational generative framework, we demonstrate Lumina-Next’s
capabilities to generate high-resolution images, multi-view images, general
audio and music, and 16K point clouds.

To fully unleash the potential of scaling diffusion transformers for
generative modeling, we present the next generation of Lumina-T2X, Lumina-Next
with improved architecture, better resolution extrapolation strategy, and
optimized sampling techniques, all of which make Lumina-Next stronger and
faster. Specifically, the key improvements of Lumina-Next are listed as
follows:

Architecture of Next-DiT We revisit the architecture design of Flag-DiT and
find suboptimal components for scalable generative modeling in the visual
domain. First, we replace the 1D RoPE with 3D RoPE to eliminate the
inappropriate positional priors to model images and videos using the attention
mechanism. We further remove all learnable identifiers in Flag-DiT, such as
`[nextline]` and `[nextframe]`, since our 3D RoPE already provides sufficient
3D positional information. Next, we task an in-depth analysis of the
instability during both training and sampling and find it stems from the
uncontrollable growth of network activations across layers. Therefore, we
introduce the sandwich normalization block [23] in attention modules, which is
proven to effectively control the activation magnitudes. Additionally, we
employ the Grouped-Qurey Attention [4] to reduce computational demand
especially when generating high-resolution images. The improved architecture
of Next-DiT is validated on the ImageNet-256 benchmark, demonstrating a faster
convergence rate compared to both Flag-DiT and SiT [54].

Frequency- and Time-Aware Scaled RoPE Inspired by the recent progress of
context extrapolation in LLMs [16, 1, 59], Lumina-T2X directly applies NTK-
Aware Scaled RoPE to generate higher resolution images thanks to the design of
1D RoPE. However, a comprehensive analysis of using 3D RoPE for length
extrapolation in the vision domain is still lacking, especially tailored for
diffusion and flow models. To bridge this gap, we first conduct a holistic
comparison between existing extrapolation methods applied to text-to-image
diffusion transformers, including Position Extrapolation, Position
Interpolation [16], and NTK-Aware Scaled RoPE [1]. We rethink the relationship
between the encoding frequency in RoPE and the generated content and propose a
Frequency-Aware Scaled RoPE, significantly reducing content repetition during
extrapolation. Furthermore, considering the difference between the auto-
regressive generation of LLMs and the time-aware generation process with fixed
sequence length in diffusion models, we propose a novel Time-Aware Scaled RoPE
for diffusion transformers to generate high-resolution images with global
consistency and local details.

Optimized Time Schedule with Higher-Order Solvers In contrast to various time
schedules [56, 68, 39] and advanced SDE/ODE solvers [48, 49, 24, 39] in
diffusion models, most flow models [41, 45, 29] still adopts Euler’s method
with uniform time discretization during sampling. SiT [54] conducts a
preliminary exploration of using different diffusion schedules and higher-
order ODE solvers on the ImageNet benchmark. In this work, we demonstrate that
existing diffusion schedules are not suitable for flow models and propose
novel time schedules tailored for flow models to minimize discretization
errors. We further combine the optimized schedules with higher-order ODE
solvers, achieving high-quality text-to-image generation samples within only 5
to 10 steps.

Time-Aware Context Drop To reduce the network evaluation time of each single
step, we propose dynamically merging latent tokens to reduce redundancy in
attention blocks. Different from the complex merge-and-unmerge algorithm in
Token Merging [8], we employ a simple average pooling for keys and values to
aggregate similar context tokens in spatial space. Additionally, we augment
this Context Drop with time awareness to align with the dynamic sampling
process of diffusion models. The resulting Time-Aware Context Drop is
validated to effectively enhance inference speed while maintaining visual
quality, achieving a $2\times$ inference-time speed up in 1K resolution image
generation. When integrated with advanced attention inference techniques like
Flash Attention [20], our method further improves inference speed of
generating ultra-high resolution images.

By composing all improvements over Lumina-T2X, we reach Lumina-Next. Our
Lumina-Next, featuring a 2B Next-DiT and Gemma-2B as text encoder, achieves
better text-to-image generation compared to Lumina-T2X with a 5B Flag-DiT and
LLaMA-7B [74] as text encoder, while significantly reducing training and
inference costs. Unlike previous models [62, 60, 29, 14, 13] that rely on CLIP
[61] or T5 [18], Lumina-Next demonstrates strong zero-shot multilingual
ability with LLMs as text encoder. Moreover, equipped with improved
architecture and the proposed inference techniques, Lumina-Next can achieve
tuning-free 2K and few-step generation. We also show that Lumina-Next’s
framework can be easily extended to other modalities with excellent
performance, demonstrating that Lumina-Next is a unified, versatile, powerful,
and efficient framework for generative modeling. All codes and checkpoints of
Lumina-Next are released. We hope the reproducibility of Lumina-Next can
foster transparency and innovations in the generative AI community.

##  2 Improving Lumina-T2X

###  2.1 Architecture of Next-DiT

Figure 2: Architecture details of Flag-DiT and Next-DiT. The main improvements
of Next-DiT include 3D RoPE, sandwich normalization, group-query attention,
_etc_.

In Lumina-T2X [32], Flag-DiT serve as the core architecture with flow matching
formulation, ensuring training stability and scalability for larger models. In
this section, we propose Next-DiT, an improved version of Flag-DiT with the
following key modifications. The comparison of detailed architectures of Next-
DiT and Flag-DiT is illustrated in Figure 2.

#### Replacing 1D RoPE and Identifiers with 3D RoPE

Flag-DiT replaces absolute positional embedding (APE) with RoPE to enable
flexibility of extrapolating to unseen resolution during training.

Figure 3: Visualization of attention score using (a) 1D RoPE and (b) 2D RoPE
on images. We set the central point in the image as the anchor query.

It further introduces learnable tokens including `[nextline]` and
`[nextframe]` as identifiers to model images and videos with different aspect
ratios and durations. We discovered that 1D RoPE is a lossy representation to
encode accurate spatial-temporal positions and the learnable identifiers
introduce additional design complexity, all of which can be simplified to 3D
RoPE without losing any information. Specifally, Lumina-T2X encode input from
different modalities into latent frames of shape $[H,W,T,C]$, where
$T=\texttt{1}$ for images, $T=\texttt{numframes}$ for videos, and
$T=\texttt{numviews}$ for multi-view images. Then, 1D RoPE is applied to the
flattened 1D token sequences, which is formulated as follows given the $m$-th
query and $n$-th key $q_{m},k_{n}\in\mathbb{R}^{d_{\text{head}}}$:

$\tilde{q}_{m}=f(q_{m},m)=q_{m}e^{im\Theta},\quad\tilde{k}_{n}=f(k_{n},n)=k_{n}e^{in\Theta},$
(1)

where $\Theta=\text{Diag}({\theta_{1},...,\theta_{d_{\text{head}/2}}})$ is the
frequency matrix. However, 1D RoPE directly treats flattened frame tokens as
1D language sequences, discarding the spatial and temporal relations between
different positions. As shown in Figure 3, when applying 1D RoPE to the 2D
images, the resulting long-term decaying property is obviously incorrect for
the 2D scenario. Inspired by recent works that introduce 2D RoPE in the image
domain [50, 30], it is natural to extend the original 1D RoPE to 3D RoPE for
spatial-temporal modeling. We divide the embedding dimensions into three
independent parts and calculate positional embedding for the $x$-axis,
$y$-axis, and $z$-axis separately. Given then 3D coordinates of the $m$-th
query and $n$-th key, the 3D RoPE is defined as:

$\displaystyle\tilde{q}_{m}$
$\displaystyle=f(q_{m},t_{m},h_{m},w_{m})=q_{m}[e^{it_{m}\Theta_{t}}\parallel
e^{ih_{m}\Theta_{h}}\parallel e^{iw_{m}\Theta_{w}}],$ (2)
$\displaystyle\tilde{k}_{n}$
$\displaystyle=f(k_{n},t_{n},h_{n},w_{n})=k_{n}[e^{it_{n}\Theta_{t}}\parallel
e^{ih_{n}\Theta_{h}}\parallel e^{iw_{n}\Theta_{w}}],$

where
$\Theta_{t}=\Theta_{h}=\Theta_{w}=\text{Diag}({\theta_{1},...,\theta_{d_{\text{head}}/6}})$
is the divided frequency matrix for each axis and $\parallel$ denotes
concatenating complex vectors of different axes at the last dimension. The
attention scores with 3D RoPE are calculated by taking the real part of the
standard Hermitian inner product:

$\displaystyle\text{Re}[f(q_{m},t_{m},h_{m},w_{m})f^{*}(k_{n},t_{n},h_{n},w_{n})]$
(3)
$\displaystyle=\text{Re}[q_{m}k_{n}^{*}e^{i\Theta_{t}(t_{m}-t_{n})}\parallel
q_{m}k_{n}^{*}e^{i\Theta_{h}(h_{m}-h_{n})}\parallel
q_{m}k_{n}^{*}e^{i\Theta_{w}(w_{m}-w_{n})}],$

where $\text{Re}[\cdot]$ denotes the real part of complex numbers and ∗
indicates complex conjugates.

With the introduction of 3D RoPE, we provide a unified and accurate spatial-
temporal representation of positional encoding for different modalities,
without the need to add extra learnable tokens such as `[nextline]` and
`[nextframe]` to indicate the row, column, and frame index. Our 3D RoPE also
incorporates the 2D RoPE formulation since it naturally becomes 2D RoPE when
applied to images with only 1 frame. Figure 2 shows the 2D RoPE assigns high
attention scores to positions within the same row and column, which is a
natural prior for images. Furthermore, our decoupled implementation of 3D RoPE
on spatial and temporal axes better unlocks the potential for length
extrapolation when combined with inference-time techniques mentioned in
Section 2.2. We demonstrate the superior zero-shot extrapolation results with
arbitrary resolutions in Section 3.1.

#### Controling Magnititude of Network Activations

Figure 4: Sandwich normalization effectively controls activation magnitudes
over layers.

Figure 5: Discretization errors and local curvatures grow at the start and end
of sampling.

Figure 6: Next-DiT converges faster than SiT and Flag-DiT in terms of FID and
IS.

To stabilize the training process when scaling up model size and token length,
Lumin-T2X replaces all LayerNorm with RMSNorm and add QK-Norm before key-query
dot product attention computation. Though effective to some extent, this
recipe is insufficient for stabilizing the training of extremely large
diffusion transformers on long sequences. A closer examination of the Flag-DiT
architecture reveals the presence of long signal paths without any
normalization due to the residue structure of attention and MLP layers. We
argue that these unnormalized paths appeared in many diffusion transformer
architectures [58, 14, 54] can accumulate signal values across layers, causing
uncontrollable growth of network activations, especially in deeper layers. We
validate this hypothesis by visualizing the evolution of activation magnitudes
over different depths of the text-to-image model using 500 random samples at
different timesteps, as shown in Figure 6. What’s worse, the uncontrollable
network activations not only lead to an unstable training process but also
deteriorate the sampling process. Particularly for resolution extrapolation,
small errors in the early layers or sampling steps are amplified in subsequent
layers or steps. These compounding errors gradually cause extremely out-of-
distribution inputs for the network, leading to failure in generated samples.

Therefore, we propose a simple yet effective approach that adds RMSNorm both
before and after each attention and MLP layer. The modified architecture is
equivalent to using the _sandwich normalization_ [23] in transformer blocks.
Importantly, both pre-norm and post-norm are placed before the scale operation
in AdaLN-Zero to prevent normalizing an all-zero tensor at the start of
training, which is caused by the zero-initialized scale parameter in AdaLN-
Zero. However, after the scale operation, the activation magnitudes still
cannot be preserved. We further add a tanh gating to the second scale
prediction after post-norm to prevent extremely large modulation values from
contributing to the residual branch. With all these efforts including the
original QK-Norm in Flag-DiT, we effectively control both then input and
output activation magnitudes of each attention and MLP layer, as validated in
Figure 6.

#### Grouped-Query Attention

We leverage grouped-query attention [4] in transformer blocks, which is an
interpolation between multi-head and multi-query attention to improve
inference speed as well as reduce memory consumption. For instance, our 2B
model divides 32 query heads into 8 groups, resulting in 8 shared key-value
heads. This modification reduces the parameter size while achieving similar
performance compared to the original multi-head attention.

#### Experiments on ImageNet

To quantitatively assess the effects of Next-DiT with the above improvements,
we conduct experiments on the label-conditional ImageNet-256 benchmark. We
follow the training setups and evaluation protocols of SiT [54] and Flag-DiT
[32]. As depicted in Figure 6, Next-DiT converges significantly faster than
both Flag-DiT and SiT evaluated by FID and Inception Score (IS). This
observation confirms that the improved design of Next-DiT can enhance its
fundamental generative capabilities. Additionally, we ablate the effectiveness
of long-skip connections[6] designed to provide shortcuts for low-level
features. Unfortunately, long-skip connections lead to training instability
and significantly worsen the performance of Next-DiT. Therefore, Next-DiT
chooses to not incorporate long-skip connection.

###  2.2 Improving NTK-Aware Scaled RoPE with Frequency- and Time-Awareness

Figure 7: (a) Toy illustration of RoPE embeddings’ wavelength with different
extrapolation strategies. (b)-(g) Results of different resolution
extrapolation strategies of 2K generation.

In the large language models (LLMs) community, tuning-free or few-shot tuning
length extrapolation is a popular focus currently and has been thoroughly
analyzed, _e.g._ , position interpolation [16], NTK-Aware Scaled RoPE [1],
YaRN [59], _etc_. In Lumina-T2X, due to the adoption of 1D RoPE same as LLMs,
NTK-Aware Scaled RoPE has been directly applied to achieve a certain degree of
resolution extrapolation. In Lumina-Next, we further investigate the effects
of different length extrapolation methods on 3D RoPE for the first time. Based
on our findings, we introduce frequency- and time-awareness grounded in NTK-
Aware Scaled RoPE, resulting in novel Frequency-Aware Scaled RoPE and Time-
Aware Scaled RoPE targeted to the visual generation task. In Figure 7 (a), we
demonstrate the wavelength of each dimension of the RoPE embedding under
different extrapolation techniques with a toy setting of $b=5$,
$h_{\text{head}}=24$, and $T=H=W=16$.

#### Revisiting NTK-Aware Scaled RoPE

3D RoPE encodes position information of each axis using a frequency matrix
$\Theta=\text{Diag}(\theta_{1},\cdots,\theta_{d},\cdots,\theta_{d_{\text{head}}/6})$
with $\theta_{d}=b^{-6d/d_{\text{head}}}$, where $b$ is the rotary base. When
we want to perform a $s$ times resolution extrapolation, the most
straightforward way is encoding unseen positions with no change of RoPE, known
as Position Extrapolation. However, these unseen positions will directly
confuse the model’s spatial understanding, leading to the generation of
unreasonable or repetitive content (refer to Figure 7 (c)).

In contrast, another straightforward approach, Position Interpolation [16],
involves linearly rescaling the frequency matrix as
$\theta_{d}^{\text{inter}}=\theta_{d}\cdot s$, thereby bringing all position
embeddings back into the training value domain. However, as noted by [1], this
linear interpolation makes it difficult for RoPE to distinguish between
positions of tokens that are very close to each other. The same conclusion
exists in the diffusion transformer – as shown in Figure 7 (d), the global
structure of the image is maintained at the cost of losing all local details.

To address these issues, NTK-Aware Scaled RoPE [1] is proposed to achieve
training-free length extrapolation for LLMs. Particularly, it scales the
rotary base as $b^{\prime}=b\cdot s$ and then calculates
$\theta_{d}^{\text{ntk}}=b^{\prime-6d/d_{\text{head}}}$ such that the lowest
frequency term of RoPE is equivalent to performing position interpolation,
allowing a gradual transition from position extrapolation of high-frequency
terms to position interpolation of low-frequency ones. In Lumina-T2X [32],
NTK-Aware Scaled RoPE has been shown to seamlessly integrate with diffusion
transformers to achieve resolution extrapolation to a certain extent. However,
failure cases still occur, such as the repetition issue shown in Figure 7 (e),
indicating that the model still struggles with spatial awareness during
resolution extrapolation.

#### Frequency-Aware Scaled RoPE

Rethinking the limitations of NTK-Aware Scaled RoPE, we believe that
performing interpolation on the lowest frequency component is a suboptimal
design. Considering that RoPE uses periodic functions with frequencies
$\Theta$ to encode relative positions, many dimensions encounter unseen
repeated cycles during resolution extrapolation, which in turn causes the
model to output repetitive content. This echoes the statement of YaRN [59]
that position embedding should be divided into parts using wavelength as the
boundary and treated differently. To solve this problem, we first identify the
dimension with the wavelength equivalent to the training sequence length as
$d_{\text{target}}=d_{\text{head}}\cdot\text{log}_{b}(\frac{L}{2\pi})$, where
$L$ is the maximum training sequence length. After that, a scaled base with
frequency-awareness can be formulated by $b^{\prime}=b\cdot
s^{\frac{d_{\text{head}}}{d_{\text{target}}}}$, which enable the component at
$d_{\text{target}}$ equivalent to position interpolation. Note that
$d_{\text{target}}$ may not be an integer, but we do not need to use it as an
index so that we can use its exact value. Additionally, to prevent the part
where $d>d_{\text{target}}$ from being excessively interpolated, we take the
larger value comparing with the position interpolation result as
$\theta_{d}^{\text{freq}}=\text{max}(b^{\prime-6d/d_{\text{head}}},b^{-6d/d_{\text{head}}}\cdot
s)$. This Frequency-Aware Scaled RoPE further addresses the issue of content
repetition, as shown in Figure 7 (f).

Figure 8: Comparison of few-step generation using different time schedules
with Euler’s method.

#### Time-Aware Scaled RoPE

Observing Figure 7, we can conclude that: (1) while position interpolation
confuses local details, it effectively ensures a reasonable global structure,
and (2) while NTK-Aware Scaled RoPE has content repetition, it retains good
local details. By combining these observations with the time-conditioned
feature of diffusion models, we can achieve a more targeted resolution
extrapolation approach. A common understanding of diffusion models is that
they first restore global concepts before local details [17]. This motivates
us to design a time-aware strategy to benefit from different approaches at
different denoising steps – using position interpolation in the early stages
of denoising to ensure the overall structure and gradually shifting to NTK-
Aware Scaled RoPE to preserve local details. This echoes DemoFusion’s [27]
strategy of gradually reducing global guidance over the denoising process,
allowing the model to focus on adding local details. Specifically, the
previously proposed Frequency-Aware Scaled RoPE is naturally adaptable to this
progressive shift. When $d_{\text{target}}=1$, it is equivalent to position
interpolation; when $d_{\text{target}}=d_{\text{head}}$, it is equivalent to
NTK-Aware Scaled RoPE. Therefore, we design a coefficient $d_{t}$ that varies
with time $t$ in the most direct way, making the frequency base time-
conditioned, _i.e._ , ${b^{\prime}}_{t}=b\cdot
s^{\frac{d_{\text{head}}}{d_{t}}}$, where $d_{t}=(d_{\text{head}}-1)\cdot
t+1$. Then the time-aware frequency base can be expressed as
$\theta_{d,t}^{\text{time}}=\max({b^{\prime}}_{t}^{-6d/d_{\text{head}}},b^{-6d/d_{\text{head}}}\cdot
s)$. This Time-Aware Scaled RoPE makes globally consistent high-detail
resolution extrapolation possible, as shown in Figure 7 (g). In the following
experiments on resolution extrapolation, we use Time-Aware Scaled RoPE for the
best tuning-free performance.

###  2.3 Improving Sampling Efficiency with Fewer and Faster Steps

Different from one-step generative models, the sampling of diffusion and flow
models involves an iterative denoising process. The overall time complexity of
sampling can be written as $N\times T$, which is controlled by the number of
function evaluations $N$ and the inference time of a single function
evaluation $T$. Therefore, to improve the efficiency during inference, we
propose corresponding strategies for each factor to reduce $N$ and $T$
respectively.

#### Fewer Sampling Steps with Optimized Time Schedule and Higher-Order
Solvers

Both diffusion models and flow-based models numerically solve an SDE or ODE
within the interval $[t_{min},t_{max}]$ during sampling. A common approach is
to discretize the continuous time interval into multiple sub-intervals and use
Euler’s method with $\mathcal{O}(h^{2})$ discretization error with respect to
step size $h$ or other advanced solvers. Given a constant budget of $N$
sampling steps, the time discretization strategy
$\\{t_{0},t_{1},...,t_{N}\\}$, or time schedule, determines the grid of
timesteps on which the network is evaluated. The resulting accumulated
truncation errors can greatly impact the quality of generated samples,
especially when $N$ is small. Various time schedules have been explored for
diffusion models, such as Cosine [56], Linear [68], and EDM [39]. However,
most flow-based models still adopt the simplest uniform time intervals and
first-order Euler’s method for sampling, resulting in poor-quality images in
low-resource scenarios, as shown in Figure 8.

Algorithm 1 Few-step sampling using midpoint method and optimized schedule

1:procedure
MidpointSampler(${\mathbf{v}}_{\theta}({\mathbf{x}},t),t_{i\in\\{0,\cdots,N\\}}$)

2: sample ${\mathbf{x}}_{0}\sim\mathcal{N}(0,\mathbf{I})$

3: for $i\in\\{0,\cdots,N-1\\}$ do

4: $\Delta t\leftarrow t_{i+1}-t_{i}$$\triangleright$ Adaptive step size

5: $\mathbf{d}_{i}\leftarrow{\mathbf{v}}_{\theta}({\mathbf{x}}_{i},t_{i})$
$\triangleright$ Evaluate ${\mathrm{d}}{\mathbf{x}}/{\mathrm{d}}t$ at
$({\mathbf{x}},t_{i})$

6:
$({\mathbf{x}}^{\prime}_{i},t^{\prime})\leftarrow({\mathbf{x}}_{i}+\frac{\Delta
t}{2}\mathbf{d}_{i},t+\frac{\Delta t}{2})$ $\triangleright$ Additional
evaluation point

7:
$\mathbf{d}_{i+1}\leftarrow{\mathbf{v}}_{\theta}({\mathbf{x}}^{\prime}_{i},t^{\prime})$
$\triangleright$ Evaluate ${\mathrm{d}}{\mathbf{x}}/{\mathrm{d}}t$ at
$(\tilde{{\mathbf{x}}}_{i+1},t_{i+1})$

8: ${\mathbf{x}}_{i+1}\leftarrow{\mathbf{x}}_{i}+\Delta t\mathbf{d}_{i+1}$
$\triangleright$ Euler step from $t_{i}$ to $t_{i+1}$

9: end for

10: return ${\mathbf{x}}_{N}$

11:end procedure

Therefore, we provide a detailed analysis of truncation errors at different
timesteps using Euler’s method to solve the Flow ODE for text-to-image
generation. We begin by setting the anchor sampling steps, $N=50$, with
uniform intervals for evaluation. For each timestep $t_{i}$, the sample
produced by one-step Euler’s method is defined as
$\hat{{\mathbf{x}}}_{t_{i+1}}={\mathbf{x}}_{t_{i}}+(t_{i+1}-t_{i}){\mathbf{v}}_{\theta}({\mathbf{x}}_{t_{i}},t_{i})$.
To approximate the local truncation error $\tau_{t_{i}}$ at time $t_{i}$, we
estimate the ground truth sample ${\mathbf{x}}_{t_{i+1}}$ by running a large
number of Euler steps from $t_{i}$ to $t_{i+1}$ with uniform intervals and
compute the discretization error as
$\tau_{t_{i}}={\mathbf{x}}_{t_{i+1}}-\hat{{\mathbf{x}}}_{t_{i+1}}$. In
addition, since the flow model builds linear interpolation between noise and
data and predicts the constant velocity field, it is natural to use the
curvature to describe local errors on the trajectory,  _i.e._ , the straighter
the trajectory, the smaller the discretization error. Following [55], we
compute the second-order derivative to estimate the local curvature of each
step as
$\kappa_{t_{i}}={\mathbf{x}}_{t_{i}}-({\mathbf{x}}_{t_{i+1}}+{\mathbf{x}}_{t_{i-1}})\times
0.5$. The average L2-norm of discretization error $\tau_{t_{i}}$ and curvature
$\kappa_{t_{i}}$ are visualized in Figure 6. Our analysis reveals that
discretization errors are much larger when $t$ is near pure noise ($t=0$) and
are relatively larger near clean data ($t=1$), compared to intermediate
timesteps. This observation contradicts the findings in diffusion models,
where EDM [39] discover the discretization errors monotonically increase as
samples get closer to clean data. Hence, diffusion schedules like EDM are not
ideal for flow-based models.

To optimize the time schedules for sampling, we propose two parametric
functions to map $t\in[0,1]$:

$\textsc{Rational}\quad t^{\prime}=\frac{t}{\sigma-t+\sigma
t},\quad\quad\textsc{Sigmoid}\quad
t^{\prime}=\begin{cases}\frac{1}{1+\exp(-\alpha(t-\mu))}&\text{if }t<\mu\\\
1-\frac{1}{1+\exp(\beta(t-\mu))}&\text{if }t\geq\mu\end{cases}.\\\ $ (4)

These time schedules are tailored for flow-based models. The first, a rational
function parametrized by $\sigma$, maps all timesteps to smaller values,
minimizing the discretization errors near pure noise. This schedule reduces to
the uniform time schedule when $\sigma=1$ and is also used in prior works [32,
29, 35] for high-resolution image generation. The second schedule, a piecewise
sigmoid-like function, ensures that the step sizes at the start and end of
sampling are larger than those at intermediate steps. The central point is
controlled by $\mu$, while the weights on lower and higher timesteps are
adjusted by $\alpha$ and $\beta$, respectively. As shown in Figure 8, both
rational and sigmoid schedules improve few-step sampling performance compared
to all-black or all-white images generated by the uniform schedule. This
highlights the importance of the early stage of sampling in creating the main
subject in the image. The sigmoid schedule generates images with better
details and is adopted for the remainder of this paper with $\mu=0.6$,
$\alpha=6$, and $\beta=20$.

Notably, this optimized schedule requires no extra computation but a free
launch for fast sampling using our flow model. It is also compatible with
advanced solvers applicable to flow models, which can unleash its full
potential. However, limited work has explored the use of higher-order solvers
for solving the Flow ODE. Unlike the Probability Flow ODE [69] in diffusion
models, which has a semi-linear structure, the Flow ODE derived from the
continuity equation is in a simpler form of
$\dot{\mathbf{x}}={\mathbf{v}}_{\theta}({\mathbf{x}}_{t},t)$. This simplicity
allows for the direct use of existing higher-order ODE solvers, such as the
traditional explicit Runge-Kutta (RK) methods in the form of
${\mathbf{x}}_{t}={\mathbf{x}}_{s}+\int^{t}_{s}{\mathbf{v}}_{\theta}({\mathbf{x}}_{\tau},\tau){\mathrm{d}}\tau$,
where the $n$-th order solver ensembles $n$ intermediate steps between $[s,t]$
to reduce the approximation errors. During sampling, explicit RK family
solvers also need to specify the timesteps $\\{t_{0},t_{1},...,t_{N}\\}$ in
advance and adopt the uniform intervals as default. Thus, our improved
schedules, such as the rational or sigmoid schedule, can be directly applied
to these higher-order solvers. Algorithm 1 illustrates the pseudocode for
combining the midpoint method and sigmoid schedule for sampling.

Figure 9: (a) Qualitative results of 2K images generated by Lumina-Next with
and without context drop. (b) Comparison of inference speed between different
settings using 50 Euler steps.

#### Faster Network Evaluation with Time-Aware Context Drop

After reducing the denoising steps during sampling, our next goal is to
optimize the network evaluation time $T$ for each single step, which is
constrained by the quadratic complexity of attention blocks. Among various
inference-time acceleration methods, Token Merging (ToMe) [8] is a simple yet
effective approach that merges similarly visual tokens to reduce redundancy in
attention blocks, thereby speeding up inference for image classification.
Although ToMe has been successfully applied to diffusion U-Net [9, 67], we
find that a straightforward application of ToMe to Next-DiT, a purely
transformer-based diffusion model, completely fails to produce satisfactory
images. We hypothesize that performing the original token merging in all
transformer layers across all diffusion steps is clearly unsuitable for the
task of generative modeling with diffusion transformers.

To bridge this gap, we propose Time-Aware Context Drop, an optimized method to
drop redundant visual tokens during inference for Next-DiT. First, the
original bipartite soft matching in ToMe not only introduces additional
complexity but also contradicts the design of 3D RoPE in Next-DiT. Therefore,
we replace this complex token partitioning strategy in ToMe with a simple
average pooling for downsampling, aligning with the spatial prior introduced
in 3D RoPE. Similar to [67], we find that applying the merge-and-unmerge
operation to all queries, keys, and values greatly undermines the quality of
generated images as this strategy is designed for visual recognition. To
preserve the complete visual content, we only perform average pooling on keys
and values for downsampling. Finally, considering the time-awareness in
diffusion models, we perform token dropping for all layers at $t=0$ to enhance
efficiency and gradually transform to no token dropping at $t=1$ to maintain
visual quality. The Context Drop without time-awareness is similar to the key-
value (KV) token compression in PixArt-$\Sigma$ [13], which merges KV tokens
in spatial space using $2\times 2$ convolution to reduce complexity. However,
unlike their fine-tuning methods, Context Drop is a training-free technique
with an adaptive compression rate that aligns with the diffusion time
schedule.

As shown in Figure 9 (a), Next-DiT using the proposed Time-Aware Context Drop
with a 75% downsampling ratio generates comparable and even better 2K images
compared to the baseline. The inference time comparison in Figure 9 (b)
highlights that our Context Drop can further enhance inference speed when
combined with advanced inference techniques for transformers, such as Flash
Attention [20], which is more significant for higher-resolution image
generation. Notably, our Context Drop is similar to masked attention or
dilated convolution, which has been proven to be the key of resolution
extrapolation in diffusion U-Net [34, 37]. Hence, this high-level resemblance
in our diffusion transformer architecture may explain why Context Drop can
reduce repetitive and unreasonable artifacts in high-resolution image
generation, presenting an interesting avenue for future research.

##  3 Lumina-Next by Composing Everything

By composing all improvements, we finally reached Lumina-Next, a powerful
generative framework for various modalities with improved efficiency. We first
instantiate Lumina-Next for text-to-image generation, which is stronger and
faster than Lumina-T2X. Besides, compared with SDXL and PixArt-$\alpha$, it
demonstrates preliminary supports of multilingual prompts (refer to Section
3.3), better visual quality with smaller sampling steps, and more flexibility
at synthesizing images with any resolution. After this, we further verified
that the Lumina-Next framework also exhibits excellent performance in any-
resolution recognition (refer to Section 4) and multi-view images, music,
audio, point clouds generation (refer to Section 5).

Figure 10: Comparison of $4\times$ resolution extrapolation.

###  3.1 Flexible Resolution Extrapolation

#### Failure of SDXL and PixArt-$\alpha$

SDXL [60] and PixArt-$\alpha$ [14] are representative methods of U-Net and
transformer-based diffusion models, respectively. Despite their ability to
generate impressive images, they fail to produce images beyond the training
resolution, resulting in repetitive and unreasonable content at higher
resolutions, as shown in Figure 10. This is because the receptive field of
U-Net and the absolute position embedding of transformers are significantly
biased towards the training resolution or sequence length.

#### Comparison with Tuning-free Resolution Extrapolation Methods

Due to the enormous cost of directly training high-resolution (greater than
2K) generative models, researchers have sought to enable pre-trained diffusion
models to infer beyond the training resolution in a tuning-free manner.
MultiDiffusion [7] is the first attempt to generate high-resolution panoramas
via sampling overlapped sliding windows at the original resolution. DemoFusion
[27] further utilizes the generated results at training resolution as global
guidance, achieving object-oriented high-resolution image generation.
Meanwhile, ScaleCrafter [34] adopts dilated convolution to achieve resolution
extrapolation tiled for U-Net based diffusion models (_e.g._ , SDXL [60]).

In Figure 10, we compare Lumina-Next with the methods above on the 2K
generation task. We can conclude that MultiDiffusion, proposed for panorama
generation, lacks global awareness, leading to the issue of repetitive content
still being present. DemoFusion and ScaleCrafter work well, but DemoFusion
requires multiple runs, resulting in a non-negligible inference budget, and
ScaleCrafter is built upon dilated convolution, which is unsuitable for modern
transformer-based diffusion models. In contrast, Lumina-Next inherently
supports a direct resolution extrapolation with the carefully designed Next-
DiT architecture and frequency- and time-aware scaled RoPE at inference time,
offering a more fundamental solution in the era of diffusion transformer.

#### On the Importance of Next-DiT for Resolution Extrapolation

Regarding Figure 10, we particularly emphasize that Lumina-Next’s 2K
extrapolation performance significantly surpasses that of Lumina-T2X. In
addition to the newly proposed Time-Aware Scaled RoPE, we attribute this to
the advanced design of Next-DiT: (1) 3D RoPE enables independent modelling of
the $t$, $h$, $w$ dimensions, decoupling extrapolation in three dimensions so
that they do not affect each other, and (2) sandwich norm stabilizes the
output magnitude of each transformer block, reducing the model’s sensitivity
to sequence length. Composing them together, Lumina-Next possesses excellent
resolution extrapolation capabilities without the need for any special
inference algorithms like DemoFusion [27].

#### Resolution Extrapolation with Any Aspect-Ratio

Figure 11: Results of $4\times$ resolution extrapolation with extreme aspect
ratio.

In Figure 11, we further compare the panorama ($1024\times 4096$ images)
generation performance of the proposed Lumina-Next with MultiDiffusion [7] and
DemoFusion [27] to highlight its any aspect ratio extrapolation ability. We
found that although panorama generation is one of the primary goals of
MultiDiffusion, its “copy-paste” inference scheme easily leads to unreasonable
content. DemoFusion adopts a progressive upscaling strategy, limiting its
ability to generate arbitrary aspect ratios by the base model – in extreme
aspect ratio scenarios, it also fails to produce satisfactory content. In
contrast, Lumina-Next perfectly maintains the ability to generate at any
aspect ratio under the training resolution, even when performing $4\times$
resolution extrapolation.

###  3.2 Few-Step Text-to-Image Generation

Figure 12: Comparison of few-step generation using second-order ODE solvers.

We demonstrate how our improved sampling schedules, together with higher-order
ODE solvers, can significantly enhance the sampling efficiency for flow
models. Specifically, we employ the midpoint solver, a second-order Runge-
Kutta method, combined with our sigmoid schedule following Algorithm 1. For
comparison, we choose open-source text-to-image models, including SDXL [60]
and PixArt-$\alpha$ [14] equipped with DPM-Solver [48, 49]. Figure 12 shows
Lumina-Next using the midpoint solver significantly improves the sample
quality in 10-20 number of function evaluations (NFEs), consistently achieving
better conditional sampling performance compared to PixArt-$\alpha$and SDXL
using DPM-Solver. Switching to higher-order solvers can further enhance few-
step sampling performance with the cost of linearly increasing NFEs,
presenting a speed-quality trade-off.

###  3.3 Zero-Shot Multilingual Generation

Figure 13: Results of multilingual text-to-image generation by Lumina-Next,
SDXL, and PixArt-$\alpha$. Figure 14: Results of text-to-image generation with
emojis by Lumina-Next, SDXL, and PixArt-$\alpha$. Figure 15: Results of
multilingual text-to-image generation using different LLMs as text encoders.
Figure 16: Generated images of Lumina-Next with long, detailed prompts.

Previous T2I models, such as Imagen [62], PixArt-$\alpha$ [14] and Stable
Diffusion [29, 60], utilized CLIP and T5 as text encoder for generating high
aesthetics images. Such choices took advantage of pretrained text encoders or
multimodal aligned text encoders as good representations for text-conditional
image synthesis. However, employing T5 and CLIP as text embedding shows
limited effectiveness in handling multilingual prompts. Different from those
approaches, the Lumina series adopts decoder-only LLMs as text encoders. For
instance, Lumina-T2X uses a 7 billion LLaMA as text encoder, while Lumina-Next
employs the smaller Gemma-2B to decrease GPU memory costs and increase
training throughput. Remarkably, both LLaMA-7B and Gemma-2B endow Lumina T2I
models with the ability to understand multilingual prompts in a zero-shot
manner, despite being primarily pretrained on English-only corpora.

As shown in Figure 13, our T2I models not only accurately interpret
multilingual prompts but also generate images with cultural nuances, even
though it was mainly trained with English-only image-text pairs. Figure 14
demonstrates Lumina-Next’s preliminary capability to understand emojis.
Compared to SDXL and PixArt-$\alpha$, which use CLIP and T5, there is a
significant improvement in understanding multilingual prompts. Furthermore,
Lumina-Next excels in handling image generation with extremely long prompts.
As shown in Figure 16, Lumina-Next can produce high-quality images that
strictly follow the detailed instructions in the long prompts, even in other
languages such as Chinese. We argue that decoder-only LLMs with strong
language understanding abilities better align multilingual semantics than the
encoder-decoder architecture of T5 or the multimodal alignment of CLIP text
embeddings.

To further explore the relationships between multilingual generation abilities
and text encoders, we train Lumina-Next with different open-source LLMs,
including Gemma-2B, Qwen-1.8B [5], and InternLM-7B [11]. As demonstrated in
Figure 15, Lumina-Next with Qwen-1.8B and InternLM-7B showcase significantly
better text-image alignment compared to Gemma-2B when using diverse Chinese
prompts as inputs. Remarkably, Lumina-Next with Qwen-1.8B and InternLM-7B
accurately captured the meanings and emotions of classic Chinese poems,
reflecting these nuances in the generated images. This observation corresponds
with the stronger multilingual abilities of Qwen-1.8B and InternLM-7B.
Therefore, we argue that using more advanced LLMs as text encoders facilitates
expressive language understanding and enhances overall text-to-image
generation performance.

##  4 Any Resolution Recognition with Lumina-Next

Lumina-Next is not only a generative modeling framework for generating images
with various resolutions but also a framework for recognizing images at any
resolution. As illustrated in Figure 17, our Next-DiT can be seamlessly
adapted into a robust backbone for image representation learning. With this
novel architecture, Next-DiT achieves superior performance and faster
convergence while maintaining computational efficiency compared to the
original vision transformer in image recognition tasks. Furthermore, Next-DiT
overcomes the limitations of previous vision transformers in processing images
of varying resolutions, demonstrating remarkable generalization capabilities
to handle input images with arbitrary resolutions and aspect ratios.

###  4.1 Pipelines

The adaptation of Lumina-Next to image understanding at any resolution is
depicted in Figure 17. Specifically, we introduce a learnable gate factor in
the attention module, thereby avoiding the incorporation of time embeddings
and label embeddings in image recognition tasks. We apply global average
pooling to all image tokens to obtain the class token, which is then fed into
the classification head implemented by an MLP network. By integrating 2D RoPE
during attention computation and incorporating RMSNorm before and after each
attention and MLP layer, our Next-DiT architecture exhibits enhanced
resolution extrapolation capabilities. We employ a progressive training scheme
with a fixed-resolution pre-training stage followed by an arbitrary-resolution
fine-tuning stage, enabling Next-DiT to achieve robust image understanding
across various resolutions.

Figure 17: Comparison between the original architecture of Next-DiT and
modified version for visual recognition.

#### Fixed Resolution Pre-training

We first pre-train Next-DiT at a fixed resolution by cropping and resizing
images to $224\times 224$, following the approach of the original ViT. This
resolution balances the trade-off between training speed and input resolution,
as higher resolutions require more FLOPs and reduced throughput. After pre-
training, we directly evaluate our model on unseen resolutions to guide a
subsequent fine-tuning stage with arbitrary resolutions.

#### Any Resolution Fine-tuning

To enable our model to handle images with any resolution and aspect ratio, we
employ an any-resolution fine-tuning stage. While previous works [21, 72] also
utilize a multi-resolution training approach, they rely on a set of predefined
input resolutions and randomly sample different resolutions for each input
image from this set. These methods often alter the original resolution of
aspect ratio of the input images a lot, potentially causing a performance
drop, and they incur additional computational costs by resizing low-resolution
images to higher resolutions unnecessarily. In contrast, we propose a dynamic
padding scheme that allows efficient model training while preserving the
original resolution and aspect ratio of the input images. This approach
enables us to fully exploit the multi-resolution understanding capability of
Next-DiT in a computationally efficient manner.

Figure 18: Illustration of our dynamic partitioning and padding scheme with
masked attention for handling input images of arbitrary resolutions.

#### Dynamic Partitioning and Padding with Masked Attention

Our aim is to retain the original resolution and aspect ratio of each input
image to the fullest extent during the subsequent fine-tuning stage.
Additionally, we seek to ensure the width and height of the input image are
divisible by the patch size of the patch embed layer, enabling us to fully
utilize the image information during the embedding process. To achieve this,
we propose a dynamic partitioning and padding scheme to optimize the handling
of image tokens instead of relying on a fixed resolution, as illustrated in
Figure 18. Specifically, given constraints on the maximum number of patches
$N$ and the maximum aspect ratio $R_{\max}$, we define a set of candidate
patch partitions $\mathcal{C}=\left\\{(H_{p},W_{p})\mid H_{p}\cdot W_{p}\leq
N,\frac{\max(H_{p},W_{p})}{\min(H_{p},W_{p})}\leq R_{\max}\right\\}$. For an
input image of size $(H_{I},W_{I})$, we determine its optimal patch partition
by calculating the matching ratio between the original input size and the
target size, given by
$(H^{*}_{p},W^{*}_{p})=\text{argmax}_{(H_{p},W_{p})\in\mathcal{C})}\frac{\min\left(\frac{H_{p}}{H_{I}},\frac{W_{p}}{W_{I}}\right)}{\max\left(\frac{H_{p}}{H_{I}},\frac{W_{p}}{W_{I}}\right)}$.
Subsequently, we resize the input image to $(H^{*}_{p}\cdot P,W^{*}_{p}\cdot
P)$, where $P$ denotes the patch size. This dynamic partitioning process leads
to varying sequence lengths of patch tokens for each input image. To handle
batches containing inputs with different token lengths, we pad all image token
sequences to match the length of the longest sequence using a pad token.
Additionally, we introduce attention masks within each attention module to
prevent unwanted interactions between pad tokens and regular image tokens.

#### Flexible Resolution Inference

By incorporating NTK-Aware Scaled RoPE and sandwich normalization, Next-DiT
demonstrates exceptional resolution extrapolation during inference, even
without fine-tuning on varied resolutions. Furthermore, the proposed dynamic
partitioning and padding scheme allows for efficient fine-tuning while
preserving image resolutions and aspect ratios. Consequently, our model can
perform flexible inference on images with arbitrary resolutions and aspect
ratios.

###  4.2 Setups

We conduct experiments on the ImageNet-1K dataset to validate the
effectiveness of Next-DiT for image recognition. We build our Next-DiT model
following the architecture hyperparameters of ViT-base [26], stacking 12
transformer layers with a hidden size of 768 and 12 attention heads. This
configuration ensures that our architecture has a comparable number of
parameters to the original ViT. During the fixed-resolution pre-training
stage, we train the models from scratch for 300 epochs with an input size of
$224\times 224$. We use the AdamW optimizer with a cosine decay learning rate
scheduler, setting the initial learning rate, weight decay, and batch size to
1e-3, 0.05, and 1024, respectively. We follow the same training recipe as the
DeiT model [73], including its data augmentation and regularization
strategies. In the subsequent any-resolution fine-tuning stage, we further
train the pre-trained model using the proposed dynamic partitioning and
padding scheme. This fine-tuning is performed for an additional 30 epochs with
a constant learning rate of 1e-5 and a weight decay of 1e-8.

Table 1: Comparison of Next-DiT with DeiT [73] on ImageNet classification.

Model | Params | Setting | Resolution | Top-1 Acc(%)
---|---|---|---|---
DeiT-base [73] | 86M | 300E | $224\times 224$ | 81.8
Next-DiT | 86M | 100E | $224\times 224$ | 81.6
Next-DiT | 86M | 300E | $224\times 224$ | 82.3
DeiT-base [73] | 86M | 300E | Flexible | 67.2
Next-DiT | 86M | 300E+30E | Flexible | 84.2

Figure 19: Performance of Next-DiT across different resolutions.

###  4.3 Experiments

We report the performance of our Next-DiT in Table 1. We first test our model
at a fixed resolution of $224\times 224$. Compared with DeiT-base, which has
comparable parameters, Next-DiT exhibits superior performance and faster
convergence. Notably, training the model for only 100 epochs achieves a Top-1
accuracy of 81.6%, nearly matching DeiT-base’s result achieved after 300
epochs. Under the same training setting with 300 epochs of pre-training, Next-
DiT surpasses DeiT-base with an accuracy of 82.3% versus 81.6%, respectively,
demonstrating its effectiveness as a robust image backbone. We also adopt the
flexible resolution inference strategy, preserving the original resolution and
aspect ratio of the input images to validate the model’s generalization
capability across varied resolutions. After the additional fine-tuning stage,
Next-DiT achieves 84.2% accuracy when handling arbitrary resolutions. We
report the performance of DeiT-base pre-trained at $224\times 224$, using the
same inference configuration. To align with the input resolution, we
interpolate its positional embeddings accordingly. Compared to our fine-tuned
Next-DiT, the performance of DeiT-base drops significantly, suggesting its
limitations in processing images with varied resolutions.

We also evaluate the performance across different image resolutions, as
illustrated in Figure 1. Next-DiT demonstrates better generalization to larger
image sizes compared to DeiT-base, even without fine-tuning. With additional
fine-tuning using the proposed dynamic partitioning scheme, our model
significantly enhances its ability to handle varied resolutions, particularly
high resolutions such as $1024\times 1024$, outperforming DeiT-base by a
significant margin. These experimental results highlight the effectiveness of
Next-DiT as a powerful image backbone for recognizing images at any
resolution, in addition to its capability to generate images at various
resolutions.

##  5 Generating Multi-Views, Music and Audio with Lumina-Next

Figure 20: An illustration of multi-view images generation. Left: All view
training paradigm of MV-Next-DiT. Right: Any view inference paradigm of MV-
Next-DiT.

Lumina-Next is a versatile and expandable framework for generative modeling
that goes beyond text-conditional image generation. In this section, we first
demonstrate the application of Lumina-Next to multi-view generation using both
image and text conditioning. Additionally, we extend Lumina-Next to encompass
a wider range of modalities, including text-conditional audio, music, and
point cloud generation.

###  5.1 Image- and Text-conditional Multi-View Generation

Figure 21: Results generated by MV-Next-DiT. The upper right is the result of
text-to-multiview, and the rest is the result of text-&image-to-multiview,
where the first column is the corresponding input image with the background
removed. In addition, every 3 rows from top to bottom are the generated
results of 4 views, 6 views, and 8 views respectively.

#### Pipelines

Having witnessed the success of Next-DiT on text-to-image generation, we use
Next-DiT’s text-to-image model as a pre-trained model and extend it to image-
and text-conditional multi-view generation, called MV-Next-DiT, as shown in
Figure 20. Firstly, we introduce relative pose control to Next-DiT to
distinguish between views. Specifically, for $N$ multi-view images with the
same elevation and random azimuths rendering from 3D objects, taking the
azimuth of the first image as the reference view, the relative azimuths of the
remaining $N-1$ images to the first image are calculated sequentially. The
obtained relative azimuths are encoded into the same dimension as time
embedding using a 2-layer MLP network after $sin$ and $cos$ transformation,
and are added to the time embedding to inject camera control into the model.
Secondly, we expanded the original text condition of Next-DiT into text and
image conditions. Specifically, the same encoding and injection method as
Next-DiT is used for text conditioning. For the image condition, we use VAE to
encode the input image, and then concatenate it to the multi-view noises
($N+1$). This image is not used for supervision during training. In addition,
when building the training data, this condition image has the same azimuth as
the first supervised view (both are 0), but it is a random elevation. It is
worth noting that the image condition is dropped based on probability during
training, and the input of the image condition is optional during inference.
Finally, a progressive training strategy of views and resolution is adopted
during training, and a flexible-views inference strategy is used during
inference. Specifically, during training, we first performed training at
$256\times 256$ resolution with $N=4$. Then it was expanded to the training of
$512\times 512$ resolution with $N=4$, and finally, it was expanded to the
training of $512\times 512$ resolution with $N=8$. In inference, since each
object is trained using all random views, any number of views can be
inferenced at one time. These improvements allow MV-Next-DiT to output multi-
view images with high-quality and flexible azimuth.

Table 2: The detailed training setting of MV-Next-DiT. Training config | Pre-train model | Total batch (Images) | Learning rate | Iteration | A100 cost
---|---|---|---|---|---
$256\times 256$, $N=4$ | Next-DiT, Text-to-Image | $256\times(N+1)$ | 1e-4 | 100k | $16GPUs\times 45h$
$512\times 512$, $N=4$ |  MV-Next-DiT, $256\times 256$, $N=4$ | $128\times(N+1)$ | 1e-4 | 100k | $16GPUs\times 57h$
$512\times 512$, $N=8$ |  MV-Next-DiT, $512\times 512$, $N=4$ | $32\times(N+1)$ | 1e-4 | 100k | $16GPUs\times 72h$

#### Setups

Data preparation: we use the Objaverse [22] list and caption provided by Cap3D
[53] as training data. For data rendering, we first use the same elevation and
8 random azimuths for each object, render with the same camera setting, and
obtain supervision data. Then the azimuths corresponding to 4 images are
randomly selected from the 8 rendered views and cooperate with random
elevations to render input images. Each object then has 4 groups of training
samples. Training details: We use Next-DiT with 600M parameters for training
of MV-Next-DiT. The specific training settings are shown in Table 2.

Table 3: Comparison of capabilities between MV-Next-DiT and other mutli-view methods. Methods |  Base model |  Resolution |  Condition type |  Inference views
---|---|---|---|---
Zero-123 [44] |  SD Image Variations v2 |  $256\times 256$ |  Image |  1
MVDream [65] |  SD v2.1 |  $256\times 256$ |  Text |  4
UniDream [46] |  SD v2.1 |  $256\times 256$ |  Text |  4
ImageDream [75] |  MVDream (SD v2.1) |  $256\times 256$ |  Image |  4
Wonder3D [47] |  SD Image Variations v2 |  $256\times 256$ |  Image |  6
Zero-123++ [64] |  SD v2 |  $256\times 256$ |  Image |  6
CRM [76] |  ImageDream (SD v2.1) |  $256\times 256$ |  Image |  6
MV-Next-DiT |  Next-DiT |  $512\times 512$ |  Text & Image |  1$\sim$8

#### Experiments

We collected images provided by previous work, such as CRM [76], to test the
performance of our MV-Next-DiT. In particular, we provide the generated
results of only text condition, text and image conditions, and the results of
sampling 4 views, 6 views, and 8 views respectively from the setting of N=8,
as shown in Figure 21. In addition, we compared the capabilities with existing
multi-view methods as shown in Table 3. The overall performance demonstrates
the advantages of our MV-Next-DiT. The rich higher-resolution flexible-view
results demonstrate MV-Next-DiT’s excellent multi-view generation capabilities
and the superiority of the Next-DiT architecture.

Figure 22: An illustration of text-guided music/audio generation. It consists
of the following main components: 1) VAE to encode spectrogram into a latent
and convert it back to spectrogram; 2) text encoder to derive high-level
textual representation; 3) flow transformer to inject condition; and 4)
separately-trained neural vocoder to convert mel-spectrograms to raw
waveforms. In the following sections, we describe these components in detail.

###  5.2 Text-Conditional Audio and Music Generation

#### Pipelines

The overall pipeline of text-to-audio and text-to-music generation using Next-
DiT is illustrated in Figure 22. Specifially, we use a modified audio VAE with
a 1D-convolution-based model to derive audio latent. The audio signal is a
sequence of mel-spectrogram sample ${\mathbf{x}}\in[0,1]^{C_{\mathrm{a}}\times
T}$, where $C_{\mathrm{a}},T$ respectively denote the mel channels and the
number of frames. Our spectrogram autoencoder is composed of 1) an encoder
network $E$ which takes samples ${\mathbf{x}}$ as input and outputs latent
representations $z$; 2) a decoder network $G$ reconstructs the mel-spectrogram
signals ${\mathbf{x}}^{\prime}$ from the compressed representation
${\mathbf{z}}$; and 3) a multi-window discriminator learns to distinguish the
generated samples $G({\mathbf{z}})$ from real ones in different multi-
receptive fields of mel-spectrograms.

In text2audio generation, we include the dual text encoder architecture
consisting of a main text encoder CLAP [28] that takes the original natural
language caption ${\mathbf{y}}$ as input and a temporal encoder FLAN-T5 [18]
which takes the structured caption ${\mathbf{y}}_{s}$ passed by LLM as input.
The final conditional representation is expressed as:

$\mathbf{c}=\text{Linear}(\text{Concat}(f_{\text{text}}({\mathbf{y}}),f_{\text{temp}}({\mathbf{y}}_{s}))),$
(5)

where $f_{\text{text}}$ is the main text encoder and $f_{\text{temp}}$ is the
temporal encoder. For text2music generation, we use the FLAN-T5 [18] as the
text encoder to encode the original natural language caption ${\mathbf{y}}$.

#### Setups of Text-to-Audio Generation

Following the benchmark studies [40, 38], we train on AudioSet-SL [42] and
finetune the model in the Audiocaps dataset. Overall, we have $\sim$3k hours
with 1M audio-text pairs for training data. For evaluating text-to-audio
models, the AudioCaption validation set is adopted as the standard benchmark,
which contains 494 samples with five human-annotated captions in each audio
clip.

We conduct preprocessing on the text and audio data: 1) convert the sampling
rate of audios to 16kHz and pad short clips to 10-second long; 2) extract the
spectrogram with the FFT size of 1024, hop size of 256 and crop it to a mel-
spectrogram of size $80\times 624$; 3) non-standard words (e.g.,
abbreviations, numbers, and currency expressions) and semiotic classes [71]
(text tokens that represent particular entities that are semantically
constrained, such as measure phrases, addresses, and dates) are normalized.

Our models conduct a comprehensive evaluation using both objective and
subjective metrics to measure audio quality, text-audio alignment fidelity,
and inference speed. Objective assessment includes Kullback-Leibler (KL)
divergence, Frechet audio distance (FAD), and CLAP score to quantify audio
quality. In terms of subjective evaluation, we conduct crowd-sourced human
assessments employing the Mean Opinion Score (MOS) to evaluate both audio
quality (MOS-Q) and text-audio alignment faithfulness (MOS-F).

Table 4: The audio quality comparisons with baselines. Model | Objective Metrics | Subjective Metrics
---|---|---
FAD ($\downarrow$) | KL ($\downarrow$) | CLAP ($\uparrow$) | MOS-Q($\uparrow$) | MOS-F($\uparrow$)
GT | / | / | 0.670 | 86.65 | 84.23
AudioGen-Large | 1.74 | 1.43 | 0.601 | / | /
Make-An-Audio | 2.45 | 1.59 | 0.616 | 70.32 | 66.24
AudioLDM | 4.40 | 2.01 | 0.610 | 64.21 | 60.96
Tango | 1.87 | 1.37 | 0.650 | 74.35 | 72.86
AudioLDM 2 | 1.90 | 1.48 | 0.622 | / | /
AudioLDM 2-Large | 1.75 | 1.33 | 0.652 | 75.86 | 73.75
Make-An-Audio 2 | 1.80 | 1.32 | 0.645 | 75.31 | 73.44
Ours | 1.03 | 1.45 | 0.630 | 77.53 | 76.52
Ours (w/o Dual-encoder) | 1.48 | 1.53 | 0.601 | 75.13 | 72.01
Ours (w/ Audioset) | 2.09 | 2.06 | 0.562 | / | /
Table 5: Ablation studies of text-to-audio generation. Model | FAD ($\downarrow$) | KL ($\downarrow$) | CLAP ($\uparrow$)
---|---|---|---
Next-DiT | 1.03 | 1.45 | 0.630
\- w/o Dual-encoder | 1.48 | 1.53 | 0.601
\- w/o Audiocaps | 2.09 | 2.06 | 0.562
Flag-DiT | 1.56 | 1.91 | 0.573

#### Experiments of Text-to-Audio Generation

We conduct a comparative analysis of the quality of generated audio samples
and inference latency across various systems, including GT (i.e., ground-truth
audio), AudioGen [40], Make-An-Audio [38], AudioLDM-L [43], TANGO [33], Make-
An-Audio 2 [36], and AudioLDM 2 [43], utilizing the published models as per
the respective paper and the same inference steps of 100 for a fair
comparison. The evaluations are conducted using the AudioCaps test set and
then calculate the objective and subjective metrics. The results are compiled
and presented in Table 4. From these findings, we draw the following
conclusion:

In terms of audio quality, our proposed system demonstrates outstanding
performance with an FAD of 1.03, demonstrating minimal spectral and
distributional discrepancies between the generated audio and ground truth.
Human evaluation results further confirm the superiority, with MOS-Q and MOS-F
scores of 77.53 and 76.52, respectively. These findings suggest a preference
among evaluators for the naturalness and faithfulness of audio synthesized by
our model over baseline approaches.

For ablation depicted in Table 5, we 1) explore the effectiveness of dual text
encoders for a deep understanding of arbitrary natural language input and
observe that the drop in CLAP encoders has witnessed the distinct degradation
of text-audio alignment faithfulness; 2) study the usage of Audioset data, and
find that without fine-tuning on Audiocaps following previous works [43], a
significant drop on objective evaluation happens; 3) validate the architecture
improvements of Next-DiT compared to original Flag-DiT [32], which showcases
significant improvements in all metrics.

Table 6: The comparison with baseline models on the MusicCaps Evaluation set. We borrow the results of Mousai, Melody, and MusicLM from MusicGen [19]. Model | Objective Metrics | Subjective Metrics
---|---|---
FAD ($\downarrow$) | KL ($\downarrow$) | MOS-Q($\uparrow$) | MOS-F($\uparrow$)
GroundTruth | / | / | 88.42 | 90.34
Riffusion | 13.31 | 2.10 | 76.11 | 77.35
Mousai | 7.50 | / | / | /
Melody | 5.41 | / | / | /
MusicLM | 4.00 | / | / | /
MusicGen | 4.50 | 1.41 | 80.74 | 83.70
MusicLDM | 5.20 | 1.47 | 80.51 | 82.35
AudioLDM 2 | 3.81 | 1.22 | 82.24 | 84.35
Ours | 3.75 | 1.24 | 83.56 | 85.69

#### Setups of Text-to-Music Generation

We employ a diverse combination of datasets to facilitate the training process
of our model. In the context of text-to-music synthesis, we exclusively employ
the LP-MusicCaps [25] dataset. The culmination of these efforts yields a
dataset comprising 0.92 million audio-text pairs, boasting a cumulative
duration of approximately 3.7K hours.

We conduct preprocessing on both text and audio data as follows: 1) We convert
the sampling rate of audio to 16kHz. Prior works [77, 38, 42] pad or truncate
the audio to a fixed length (10s), while we group audio files with similar
durations together to form batches to avoid excessive padding which could
potentially impair model performance and slow down the training speed. This
approach also allows for improved variable-length generation performance. We
truncate any audio file that exceeds 20 seconds, to speed up the training
process. 2) For audios without natural language annotation, we apply the
pseudo prompt enhancement method from Make-An-Audio [38] to construct captions
aligned with the audio. We traverse the AudioCaps training set and the LLM
augmented data with a probability of 50%, while randomly selecting data from
all other sources with a probability of 50%. For the latter dataset, we use
"<text & all>" as their structured caption.

#### Experiments of Text-to-Music Generation

In this part, we compare the generated audio samples with other systems,
including 1) GT, the ground-truth audio; 2) MusicGen [19]; 3) MusicLM [3]; 4)
Mousai [63]; 5) Riffusion [31]; 6) MusicLDM [15]; 7) AudioLDM 2 [43]. The
results are presented in Table 6, and we have the following observations:
Regarding audio quality, our model consistently surpasses diffusion-based
methods and language models across FAD and KL metrics. In terms of subjective
evaluation, our model demonstrates the strong text-music alignment
faithfulness.

###  5.3 Label- and Text-Conditional Point Cloud Generation

#### Pipelines

We propose a density-invariant point cloud generator to validate the
effectiveness of Lumina-Next in the 3D domain. The density-invariant property
allows an efficient training on a fixed point number (_e.g_., 256 points) and
an inference of any point number (_e.g_., 1024, 2048, or more points).
Different from 2D images, increasing the density of a point cloud does not
affect the overall shape but rather enhances the details. To achieve any point
inference, we combine the Time-Aware Scaled RoPE with Fourier Features [70]
and design a Time-Aware Scaled Fourier Features.

$[\cos(2\pi\bm{\theta}_{1}^{\top}\mathbf{p}),\sin(2\pi\bm{\theta}_{1}^{\top}\mathbf{p}),...,\cos(2\pi\bm{\theta}_{d}^{\top}\mathbf{p}),\sin(2\pi\bm{\theta}_{d}^{\top}\mathbf{p}),...]$.
To inject the time condition of diffusion steps into the Fourier features, we
scale the frequency via ${\bm{\theta}^{\prime}}_{d}=\bm{\theta}_{d}\cdot
s^{\frac{d_{\text{head}}}{d_{t}}}$, where $d_{t}=(d_{\text{head}}-1)\cdot
t+1$. This leads the generator to focus on the global information in the early
steps of denoising, and include more shape details in later steps. This Time-
Aware Scaled Fourier Features can effectively distinguish continual positions
in 3D space, thus our model guarantees any point inference even though the
model is only trained on a fixed low density.

Table 7: Quantitative results of point cloud generation. We multiplied the value of CD by $10^{3}$. Shape | Model | MMD ($\downarrow$) | COV (%, $\uparrow$) | Shape | Model | MMD ($\downarrow$) | COV (%, $\uparrow$)
---|---|---|---|---|---|---|---
Airplane | PC-GAN | 3.819 | 42.17 | Chair | PC-GAN | 13.436 | 46.23
TreeGAN | 4.323 | 39.37 | TreeGAN | 14.936 | 38.02
PointFlow | 3.688 | 44.98 | PointFlow | 13.631 | 41.86
ShapeGF | 3.306 | 50.41 | ShapeGF | 13.175 | 48.53
PDiffusion | 3.276 | 48.71 | PDiffusion | 12.276 | 48.94
Ours | 3.371 | 49.21 | Ours | 12.975 | 48.33
Figure 23: Examples of generated point clouds with different densities,
sampled from the generator trained on 256 points.

#### Setups

We train a label- and text-conditioned 3D generator respectively. For the
label-conditioned model, we employ the ShapeNet dataset [12] and follow [51]
to preprocess it. We randomly sample 256 points from each point cloud to train
the model. After training, we synthesize point clouds of densities in
$\\{256,512,1024,2048,4096,8192\\}$. To evaluate the generation, we adopt the
Minimum Matching Distance (MMD) and the Coverage score (COV) as quality
metrics and use Chamfer Distance (CD) as the distance measure to calculate
them. For the text-guided generator, we adopt the Cap3D dataset [52] for
training.

#### Experiments

We compare the generation performance of two categories: airplane and chair.
We fine-tune our pre-trained generator on 2048 points to further improve the
generation quality. We mainly compare our performance with PC-GAN [2], TreeGAN
[66], PointFlow [78], ShapeGF [10], and PDiffusion [51] in Table 7. We observe
that our generator achieves a satisfactory performance compared to existing
models. We also visualize some generated point clouds in Figure 23, where we
directly apply the generator pre-trained on 256 points to sample higher-
density point clouds. We observe the density of generated point clouds
decreases when we increase the density (_e.g_., airplane and chair), which
could be due to the fact that more points would cause a smoother attention
matrix in self-attention layers.

##  6 Conclusion

In this paper, we introduce Lumina-Next, which successfully addressed the
limitations of its predecessor, Lumina-T2X. With the improved Next-DiT
architecture, context extrapolation together, and fast sampling techniques
tailored for our flow-based diffusion transformers, Lumina-Next showcases
strong generation capabilities, such as generating high-quality images
significantly larger than its training resolution and multilingual text-to-
image generation. Remarkably, these results are all produced in a training-
free manner and outperform previous methods including SDXL and PixArt-$\alpha$
by a considerable margin. Furthermore, we extend Lumina-Next’s versatility to
other tasks and modalities, such as visual recognition, multiviews, audio,
music, and point clouds generation, with minimal modifications, achieving
superior results across these diverse applications.

## References

  * [1] Ntk-aware Scaled Rope Allows Llama Models to Have Extended (8k+) Context Size Without Any Fine-tuning and Minimal Perplexity Degradation.  https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/, 2024.  Accessed: 2024-4-10.
  * [2] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas.  Learning representations and generative models for 3d point clouds.  In International Conference on Machine Learning, pages 40–49. PMLR, 2018.
  * [3] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al.  Musiclm: Generating music from text.  arXiv preprint arXiv:2301.11325, 2023.
  * [4] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai.  Gqa: Training generalized multi-query transformer models from multi-head checkpoints.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, 2023.
  * [5] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  arXiv preprint arXiv:2309.16609, 2023.
  * [6] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu.  All are worth words: A vit backbone for diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22669–22679, 2023.
  * [7] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel.  Multidiffusion: fusing diffusion paths for controlled image generation.  In Proceedings of the 40th International Conference on Machine Learning, pages 1737–1752, 2023.
  * [8] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman.  Token merging: Your vit but faster.  In The Eleventh International Conference on Learning Representations, 2022.
  * [9] Daniel Bolya and Judy Hoffman.  Token merging for fast stable diffusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4598–4602, 2023.
  * [10] Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan.  Learning gradient fields for shape generation.  arXiv preprint arXiv:2008.06520, 2020.
  * [11] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al.  Internlm2 technical report.  arXiv preprint arXiv:2403.17297, 2024.
  * [12] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], 2015.
  * [13] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li.  Pixart-$\sigma$: Weak-to-strong training of diffusion transformer for 4k text-to-image generation.  arXiv preprint arXiv:2403.04692, 2024.
  * [14] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al.  Pixart-$\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis.  arXiv preprint arXiv:2310.00426, 2023.
  * [15] Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg-Kirkpatrick, and Shlomo Dubnov.  Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies.  In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1206–1210. IEEE, 2024.
  * [16] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian.  Extending context window of large language models via positional interpolation.  arXiv preprint arXiv:2306.15595, 2023.
  * [17] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon.  Perception prioritized training of diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472–11481, 2022.
  * [18] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and others.  Scaling instruction-finetuned language models.  arXiv preprint arXiv:2210.11416, 2022.
  * [19] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez.  Simple and controllable music generation, 2023.
  * [20] Tri Dao.  Flashattention-2: Faster attention with better parallelism and work partitioning.  arXiv preprint arXiv:2307.08691, 2023.
  * [21] Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al.  Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution.  Advances in Neural Information Processing Systems, 36, 2024.
  * [22] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142–13153, 2023.
  * [23] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  Advances in Neural Information Processing Systems, 34:19822–19835, 2021.
  * [24] Tim Dockhorn, Arash Vahdat, and Karsten Kreis.  Genie: Higher-order denoising diffusion solvers.  Advances in Neural Information Processing Systems, 35:30150–30166, 2022.
  * [25] SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam.  Lp-musiccaps: Llm-based pseudo music captioning.  arXiv preprint arXiv:2307.16372, 2023.
  * [26] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [27] Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma.  Demofusion: Democratising high-resolution image generation with no $$$.  In CVPR, 2024.
  * [28] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang.  Clap: Learning audio concepts from natural language supervision.  2022\.
  * [29] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al.  Scaling rectified flow transformers for high-resolution image synthesis.  arXiv preprint arXiv:2403.03206, 2024.
  * [30] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva-02: A visual representation for neon genesis.  arXiv preprint arXiv:2303.11331, 2023.
  * [31] Seth Forsgren and Hayk Martiros.  Riffusion-stable diffusion for real-time music generation.  URL https://riffusion. com, 2022.
  * [32] Peng Gao, Le Zhuo, Ziyi Lin, Dongyang Liu, Ruoyi Du, Xu Luo, Longtian Qiu, Yuhang Zhang, et al.  Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers.  arXiv preprint arXiv:2405.05945, 2024.
  * [33] Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria.  Text-to-audio generation using instruction-tuned llm and latent diffusion model.  arXiv preprint arXiv:2304.13731, 2023.
  * [34] Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan.  Scalecrafter: Tuning-free higher-resolution visual generation with diffusion models.  In The Twelfth International Conference on Learning Representations, 2023.
  * [35] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans.  simple diffusion: End-to-end diffusion for high resolution images.  In International Conference on Machine Learning, pages 13213–13232. PMLR, 2023.
  * [36] Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao.  Make-an-audio 2: Temporal-enhanced text-to-audio generation.  arXiv preprint arXiv:2305.18474, 2023.
  * [37] Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, and Hongsheng Li.  Fouriscale: A frequency perspective on training-free high-resolution image synthesis.  arXiv preprint arXiv:2403.12963, 2024.
  * [38] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao.  Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models, 2023.
  * [39] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  Advances in Neural Information Processing Systems, 35:26565–26577, 2022.
  * [40] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi.  AudioGen: Textually Guided Audio Generation.
  * [41] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le.  Flow matching for generative modeling.  arXiv preprint arXiv:2210.02747, 2022.
  * [42] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley.  AudioLDM: Text-to-Audio Generation with Latent Diffusion Models.
  * [43] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley.  Audioldm 2: Learning holistic audio generation with self-supervised pretraining.  arXiv preprint arXiv:2308.05734, 2023.
  * [44] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298–9309, 2023.
  * [45] Xingchao Liu, Chengyue Gong, and Qiang Liu.  Flow straight and fast: Learning to generate and transfer data with rectified flow.  arXiv preprint arXiv:2209.03003, 2022.
  * [46] Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu, Sida Peng, Yan-Pei Cao, Xiaojuan Qi, Xiaoshui Huang, Ding Liang, and Wanli Ouyang.  Unidream: Unifying diffusion priors for relightable text-to-3d generation.  arXiv preprint arXiv:2312.08754, 2023.
  * [47] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al.  Wonder3d: Single image to 3d using cross-domain diffusion.  arXiv preprint arXiv:2310.15008, 2023.
  * [48] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  Advances in Neural Information Processing Systems, 35:5775–5787, 2022.
  * [49] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  arXiv preprint arXiv:2211.01095, 2022.
  * [50] Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai.  Fit: Flexible vision transformer for diffusion model.  arXiv preprint arXiv:2402.12376, 2024.
  * [51] Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 2837–2845, 2021.
  * [52] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson.  Scalable 3d captioning with pretrained models.  arXiv preprint arXiv:2306.07279, 2023.
  * [53] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson.  Scalable 3d captioning with pretrained models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [54] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie.  Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers.  arXiv preprint arXiv:2401.08740, 2024.
  * [55] Bao Nguyen, Binh Nguyen, and Viet Anh Nguyen.  Bellman optimal step-size straightening of flow-matching models.  arXiv preprint arXiv:2312.16414, 2023.
  * [56] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In International conference on machine learning, pages 8162–8171. PMLR, 2021.
  * [57] OpenAI.  https://openai.com/sora.  In OpenAI blog, 2024.
  * [58] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.
  * [59] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.  Yarn: Efficient context window extension of large language models.  arXiv preprint arXiv:2309.00071, 2023.
  * [60] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  arXiv preprint arXiv:2307.01952, 2023.
  * [61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [62] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in neural information processing systems, 35:36479–36494, 2022.
  * [63] Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf.  Moûsai: Text-to-music generation with long-context latent diffusion, 2023.
  * [64] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su.  Zero123++: a single image to consistent multi-view diffusion base model.  arXiv preprint arXiv:2310.15110, 2023.
  * [65] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang.  Mvdream: Multi-view diffusion for 3d generation.  arXiv preprint arXiv:2308.16512, 2023.
  * [66] Dong Wook Shu, Sung Woo Park, and Junseok Kwon.  3d point cloud generative adversarial network based on tree structured graph convolutions.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3859–3868, 2019.
  * [67] Ethan Smith, Nayan Saxena, and Aninda Saha.  Todo: Token downsampling for efficient generation of high-resolution images.  arXiv preprint arXiv:2402.13573, 2024.
  * [68] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [69] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * [70] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
  * [71] Paul Taylor.  Text-to-speech synthesis.  Cambridge university press, 2009.
  * [72] Rui Tian, Zuxuan Wu, Qi Dai, Han Hu, Yu Qiao, and Yu-Gang Jiang.  Resformer: Scaling vits with multi-resolution training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22721–22731, 2023.
  * [73] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International conference on machine learning, pages 10347–10357. PMLR, 2021.
  * [74] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [75] Peng Wang and Yichun Shi.  Imagedream: Image-prompt multi-view diffusion for 3d generation.  arXiv preprint arXiv:2312.02201, 2023.
  * [76] Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu.  Crm: Single image to 3d textured mesh with convolutional reconstruction model.  arXiv preprint arXiv:2403.05034, 2024.
  * [77] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu.  Diffsound: Discrete Diffusion Model for Text-to-sound Generation.
  * [78] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan.  Pointflow: 3d point cloud generation with continuous normalizing flows.  In IEEE International Conference on Computer Vision, pages 4541–4550, 2019.