# Combinatorial Music generation model with song structure graph analysis

###### Abstract

In this work, we propose a symbolic music generation model with the song
structure graph analysis network. We construct a graph that uses information
such as note sequence and instrument as node features, while the correlation
between note sequences acts as the edge feature. We trained a Graph Neural
Network to obtain node representation in the graph, then we use node
representation as input of Unet to generate CONLON pianoroll image latent. The
outcomes of our experimental results show that the proposed model can generate
a comprehensive form of music. Our approach represents a promising and
innovative method for symbolic music generation and holds potential
applications in various fields in Music Information Retreival, including music
composition, music classification, and music inpainting systems.

Index Terms—  combinatorial music generation, graph neural network, song
structure analysis

##  1 Introduction

Recent research in the field of symbolic music generation has been focused on
utilizing deep learning-based generation models like GAN[1] and Transformer[2]
to produce high-quality music. However, several challenges remain for music-
generating models. For example, the ability to consider the structure of music
during the generation process, and the creation of commercially viable music
still poses a significant challenge.

Theme transformer[3] model has been used to maintain a regular song structure
by directly conditioning the melody used as the main theme. Another model
called MELONS[4] has divided songs into nodes to form a graph and conducted a
study on generating music by analyzing the relationship between these nodes.
These models were suggested to consider the song structure, but they do not
consider the song structure in multi-track case.

Lee et al.[5] proposed the concept of combinatorial music generation to
increase the commercial applicability of music generation models. In stage 1,
we should generate note sequences containing a specific role, like instrument,
track-role, or genre. In stage 2, the note sequences are combined to create
complete music. This form of music generation resembles the form of music
produced by real people, and it is easy to commercialize the generated
results.

To address these challenges, we introduce a new framework that analyzes song
structure using graphs for combinatorial music generation. We analyzed the
given music’s song structure based on the simplicity matrix to construct a
song structure graph. Example of song structure graph is illustrated in Figure
1. Each node in the graph represents pieces of information containing metadata
of bars, such as note sequences, keys, and instruments. We denote these pieces
of information as ”Musical Pattern” in this paper. Each edge in the graph
represents the relationship between musical patterns. We then use Relational
Graph Neural Network to obtain the representation of each musical pattern
node. This node representation is then used to generate a piano-roll image.
Our proposed framework enables the understanding of the overall structure of
multi-track music and facilitates the generation of each musical pattern for
music generation. We also show that it can be used for symbolic music
generation or inpainting by proposing a framework that can directly obtain
structure structures by using graphs at the bar level.

##  2 Background

Fig. 1: Song Structure Graph example. There are edges related to a musical
pattern in the red box, except for some ”Same Instrument Flow” edges. Note
that this figure plots general pianoroll, not CONLON pianoroll[6] which used
in our study.

###  2.1 Song Structure Analysis

Song structure analysis based on the self-similarity matrix is a well-
established method, especially in the audio domain[7]. Various studies have
also been conducted on song structure analysis in symbolic music[8][9].
However, in the symbolic music domain, this method is not robust to variations
like pitch shifts. With this issue, we adopted the idea of content-based image
retrieval[10]. We employed a Wasserstein autoencoder[11] to transform each
piano-roll into the embedding of the latent space, and then we computed cosine
similarity to conduct song structure analysis.

###  2.2 Graph Neural Network

Graph Neural Network (GNN) was first proposed by [12], which makes it easy to
obtain a representation for each node for graph data. Recently, numerous
studies have applied GNNs to Music Information Retrieval (MIR)
tasks[13][14][15]. In most of these studies[13][15], a single note was treated
as a node, and the relationship between notes was represented as an edge. This
approach might provide only a limited view of the entire song structure. To
address this issue, we expand the scope to include musical patterns as nodes,
and each edge represents the relationship between musical patterns. In this
manner, we treat each complete music as a single graph.

We used the Relational Graph Convolutional Network (RGCN) to analyze the
features of various edges and nodes. For directed and labeled graphs as
$G=(\mathcal{V},\mathcal{E},\mathcal{R})$ with nodes $v_{i}\in\mathcal{V}$ and
edges $(v_{i},r,v_{j})\in\mathcal{E}$ and relation type $r\in\mathcal{R}$, we
can compute hidden representation in $(l+1)^{th}$ layer in RGCN as the
following equation:

$h_{i}^{l+1}=\sigma\left(W_{0}^{(l)}h_{i}^{(l)}+\sum_{r\in\mathcal{R}}\sum_{j\in
N_{i}^{r}}\frac{1}{c_{i,r}}W_{r}^{(l)}h_{j}^{(l)}\right)$ (1)

where $N_{i}^{r}$ denotes the set of neighbor indices of node $i$ under
relation $r\in\mathcal{R}$, $c_{i,r}$ is a problem-specific normalization
constant parameter. we use $c_{i,r}=|N_{i}^{r}|$ for experiment.

##  3 Proposed method

###  3.1 Song Structure Graph

Fig. 2: Kernel function for compute novelty function $n_{f}$. Algorithm 1 Song
Structure Graph Constructing Algorithm

1:Music’s MIDI data $M$

2:Music’s metadata $meta$

3:function getGraph($M,meta$)

4: $B_{N,I}\leftarrow$ split($M$)

5: $SM_{i,j}\leftarrow$ SimilarityMetric($B_{i},B_{j}$) $\forall i,j\in N$

6: for $k$ in range(N) do

7: $NC_{k}\leftarrow$ kernel function($SM,k$)

8: end for

9: $SP\leftarrow\\{\\}$ $\triangleright$ Starting Points for Musical Patterns

10: $MP\leftarrow\\{\\}$ $\triangleright$ Musical Patterns

11: for $l$ in range(N) do

12: if $NC_{l}>$ threshold$\And$it is local maxima then

13: $SP$.insert($l$)

14: end if

15: end for

16: for $t$ in $SP$ do

17: if $t$ is not in $Used$ then

18: if $B_{[t,t+pl],inst}$ is not empty then

19: $MP.$insert$({[B,meta]_{[t,t+pl],inst}})$

20: $Used\leftarrow{[t,t+pl]}$

21: end if

22: end if

23: end for

24: $V\leftarrow MP$

25: $E\leftarrow\\{\\}$

26: for $u\in V$ do

27: for $v\in V$ do

28: if Rel($u,v$) then

29: $E$.insert$(u,v,\textsc{Rel}(u,v))$

30: end if

31: end for

32: end for

33: $g\leftarrow G(V,E)$ return $g$

34:end function

Fig. 3: Red-circled node’s hidden state update in the RGCN layer. $\epsilon$
means encoder and $\delta$ means decoder.

The construction of the song structure graph involved the application of the
self-similarity matrix to the analysis of song structures. The given MIDI data
was divided into bar sequence $BS$ with length $N$. The image was compressed
into a Wasserstein Autoencoder (WAE) to compute the cosine similarity in
latent space. The self-similarity matrix was computed by the formula:

$SM_{i,j}=\frac{1}{2}+\frac{w(BS_{i})\cdot w(BS_{j})}{2\lVert
w(BS_{i})\rVert\lVert w(BS_{j})\rVert},i,j\in N$ (2)

where $w$ is the encoder of the WAE. We modified some terms to make
$SSM_{i,j}\in[0,1]$ for all $i,j\in N$. We can compute novelty function
$n_{f}$ with kernel matrix $K_{N}$[16][17],

$K_{N}(i,j)=sign(a)\cdot sign(b),a,b\in[-L,L]$ (3)

where $L$ is the range for the kernel function, and $sign$ is the sign
function (-1, 0, or 1). We use the radially Gaussian function to smooth the
kernel,

$\phi(a,b)=exp\left(-\frac{a^{2}+b^{2}}{2L\sigma^{2}}\right)$ (4)

where $\sigma$ is the standard deviation for $a$ and $b$. The novelty function
$n_{f}$ can be calculated by correlating the kernel with the diagonal of the
self-similarity matrix. These processes are described in Figure 2.

$n_{f}(m)=\sum_{a,b=-L}^{L}K_{N}(a,b)\phi(a,b)SSM_{m+a,m+b}$ (5)

This study employs a homogeneity and repetition-based approach to gauge the
similarity of given musical patterns and to identify any recurring repetitions
of these patterns at subsequent times. Using the Similarity Matrix, we can
detect repetitions in music by identifying high values in the same row. We
used 7 Hu moment based image similarity[18] to compute Homogeneity between two
pianoroll. The resulting information is then employed to define the various
edges of the song structure graph. Our whole algorithm for constructing the
song structure graph is described in Algorithm 1. Edges used in our
experiments are explained below.

Same Time Relation

Connect if two musical patterns playing at the same time.

Same Instrument Flow

Connect if the same instrument is played continuously. Note that these are
direct edges.

Same Song Structure

Connect musical patterns in time $i$ and $j$, if $SSM_{i,j}>\mathbf{t}$, where
$\mathbf{t}$ is threshold.

Similar Homogeneity

Connect if two musical Pattern has a similar shape based on 7 Hu invariants.

###  3.2 Model Architecture

In our training framework, we employed a Deep Convolutional Autoencoder with a
residual block to compress the piano roll image data into the latent space. We
used the Mean Squared Error (MSE) loss function to reduce reconstruction
errors. We also obtained the feature vector through the embedding layer to use
the key, instrument, and time information given in the musical pattern.

In the inference process of generating music, there are cases where the
pianoroll feature is not given to the musical Pattern node. Based on [19], We
can get proper embedding by simply adding the ”Observed” feature’s and a
partially observed feature in the same embedding space. Therefore, we used the
sum of features that embeds key/instrument information and piano roll latent
embedding. We use zero vector for masked piano-roll’s latent embedding. The
time information embedding was concatenated into a combined feature vector to
serve as a positional embedding. Thus, the initial state $h_{0}$ can be
computed as:

$h_{0}=(X+\hat{Y})\oplus
T,X\in\mathbb{R}^{d_{h}},\hat{Y}\in\mathbb{R}^{d_{h}},T\in\mathbb{R}^{d_{t}}$
(6)

where $d_{h},d_{t}$ is hidden dimension of features and time embedding, $X$ is
the embedding of observed information(key/instrument), $\hat{Y}$ is embedding
of partially observed information(pianoroll), and $T$ is the positional
embedding for each musical patterns. Shi et al. [19] have shown that this
model can be approximately decomposed into observed feature propagation and
partially observed information. Augmentation methods such as DropEdge[20] and
DropNode[21] were used to obtain more generalized performance. And we used two
additional loss functions. MSE Loss function was used to obtain the latent of
the masked image, and the other was to use cross-entropy loss to proceed with
the task of predicting genre by averaging all node representations. With this,
our objective function becomes:

$\begin{array}[]{l}L=\min(L_{r}+\lambda L_{C}),\text{where}\\\
L_{r}=\sum_{v}^{\hat{V}}MSE(o_{v},y_{v})\\\
L_{c}=-\sum_{g}^{G}L(\sigma(O_{g}),Y_{g})\end{array}$ (7)

where $\lambda$ is a factor for balancing loss, $\hat{V}$ is the set of
musical patterns in which pianoroll latent is masked, $o_{v}$ is the
representation output of a model, $y_{v}$ is target pianoroll latent, $G$ is
the set of the song structure graph, and $O_{g}$ is the classification output
of the model, and $Y_{g}$ is the genre label of the graph.

This allows node representation to be obtained not only to predict musical
pattern’s pianoroll, but also to predict genre features given in music. These
node representations are used for another model, by conditioning the
generation process. We constructed an Unet[22] model with node representation
as input and piano roll’s latent as a target so that the pianoroll of each
musical pattern’s pianoroll latent can be generated by node representation
condition. The whole model architecture is described in Figure 3.

Task Metric | ND | UP | KS | VA | DA
---|---|---|---|---|---
ground-truth | 20.85 | 6.23 | 0.74 | 94.81 | 0.81
Generation | 23.04 | 6.81 | 0.70 | 44.66 | 0.29
Inpainting | 17.60 | 6.26 | 0.75 | 79.10 | 0.54
Cond. Generation | 22.14 | 6.46 | 0.72 | 41.55 | 0.17
Table 1: This is the result of the experiment.

##  4 Experiment

###  4.1 Implementation Details

####  4.1.1 Dataset

We conducted an experiment using cleansed LPD (Lakh Pianoroll Dataset)-17
which is a subset of the Lakh MIDI dataset[1][23]. It consists multitrack of
17 instruments each, and all track has 4/4 time signature. In addition, we
collected 4240 pieces of music, which key information is given in MIDI’s
internal metadata, and the genre is one of ’country’, ’piano’, ’rock’, ’pop’,
’folk’, ’electronic’, ’rap’, ’chill’, ’dance’, ’jazz’, ’rnb’, ’reggae’,
’house’, ’techno’, ’trance’, ’metal’, ’pop_rock’, ’latin’, and ’catchy’. We
quantized pianoroll to handle minimum length to 48th note.

####  4.1.2 Experiment Details

During training, we set a loss balancing factor $\lambda$ as 1, and a masking
ratio of 30% for each node in the pianoroll. We used a novel format of
pianoroll, CONLON. The CONLON pianoroll incorporates the Velocity channel and
Duration channel, so it can contain velocity information that cannot be
contained by a classic pianoroll. Also, it contains note data in only onset,
so focusing on more crucial features compared to classic pianorolls.

###  4.2 Evaluation

We conducted three experiments to assess the efficacy of our proposed
framework. Firstly, we explored inpainting by masking 30% of a musical pattern
to test the model’s ability to accurately reconstruct the masked section.
Secondly, we embarked on music generation by preserving only musical patterns
with a playing time of up to 8 bars, to see how the model generates
corresponding patterns. Lastly, in the melody-conditioned music generation
task, we hypothesized that the instrument with the highest number of
connections via the ”Same Homogeneity Edge” serves as the primary instrument.
Using a pre-composed melody, we analyzed how the model composes accompaniments
for other instruments. To evaluate the performance across these tasks, we
employed several metrics that capture musical characteristics. It is crucial
to note that these metrics were computed across all masked musical pattern
nodes, excluding the drum instrument. Detailed methodologies are further
explained in the below.

Key Score (KS)

According to 24 basic key scales(C maj, C#maj, …, B min), check if notes’
pitches are on the scale in the musical pattern. $KS=\frac{\text{\\# of notes
in key}}{\text{\\# of notes}}$

Unique Pitch (UP)

Unique pitches that occur at least once in a musical pattern.

Note Density (ND)

Count the number of notes in musical pattern.

Velocity Average (VA)

Average value of velocities.

Duration Average (DA)

Average value of durations.

##  5 Result

The results of the experiment above are shown in Table 1. For the ground-
truth, We averaged metric for randomly selected 1000 pieces of music in
dataset. to evaluate the models, we used 43 pieces of music which were not
used for training.

In the case of Inpainting, it was confirmed that song metrics did not differ
significantly from the ground-truth, but in some instances, the metrics were
slightly off. For the generation or melody-conditional generation tasks, the
Velocity Average or Duration Average indicators were quite far from ground-
truth. As the proportion of provided musical pattern information increased,
the metrics approached the ground truth more closely, leading to better
performance in the inpainting task. Still, these metrics show that our model
can construct a general form of music.

While our metrics provide objective measurements, our subjective assessment of
the generated music sequences is less favorable. While each instrument’s
individual quality may be satisfactory, the combined multi-track output often
displays inconsistent, which undermines the overall performance. However, our
results confirm that each musical pattern is generated with consideration of
key and instrument information, and graph structure. Specifically, in the case
of drum patterns, it is inferred that direct information input could yield
high performance, especially when considering the proximity of the generated
output to actual usability. Generally, for drum patterns represented in
pianoroll, we think it is a phenomenon that occurs because it has distinct
characteristics compared to other instruments.

Even if our model did not succeed in generating music with complete quality,
it was shown that notes could be configured to have a musical distribution
similar to the original data using the information provided in generating
musical patterns, to construct the entire song structure graph.

##  6 Conclusion

We have proposed a framework for performing a combinatorial music generation
task, which involves expressing multitrack symbolic music as a single song
structure graph. The framework has shown its ability to generate new music
that is similar in form to existing music, as well as perform tasks such as
generation and inpainting tasks.

However, the research has several limitations at this point. It is hard to
ensure stable generation during the process of connecting various models, like
GNN, Autoencoder, and Unet. Furthermore, we believe our proposed framework can
be enhanced by utilizing various models such as transformer and diffusion
models. Currently, there isn’t a process for generating a song structure graph
from scratch, so it is difficult to see it as a model that can completely
produce entire music.

Although our song structure analysis considered the novelty, reposition, and
homogeneity of each pattern, the study did not include any track-role (e.g.,
verse or chorus) or explicit expression (e.g., A-B-A’-B). It is hoped that
future research will address this limitation based on more detailed analysis
and datasets.

Our framework well embodies the actual producer’s composition method, and has
demonstrated certain quality in composition. Beyond this, our framework is
expected to have wide applicability not only for music generation but also for
other tasks that account for the overall song structure in symbolic music,
such as music modeling and harmonization.

## References

  * [1] Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang,  “Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2018, vol. 32.
  * [2] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck,  “Music transformer,”  arXiv preprint arXiv:1809.04281, 2018.
  * [3] Yi-Jen Shih, Shih-Lun Wu, Frank Zalkow, Meinard Muller, and Yi-Hsuan Yang,  “Theme transformer: Symbolic music generation with theme-conditioned transformer,”  IEEE Transactions on Multimedia, 2022.
  * [4] Yi Zou, Pei Zou, Yi Zhao, Kaixiang Zhang, Ran Zhang, and Xiaorui Wang,  “Melons: generating melody with long-term structure using transformers and structure graph,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 191–195.
  * [5] Hyun Lee, Taehyun Kim, Hyolim Kang, Minjoo Ki, Hyeonchan Hwang, Sharang Han, Seon Joo Kim, et al.,  “Commu: Dataset for combinatorial music generation,”  Advances in Neural Information Processing Systems, vol. 35, pp. 39103–39114, 2022.
  * [6] Luca Angioloni, Valentijn Borghuis, Lorenzo Brusci, and Paolo Frasconi,  “Conlon: A pseudo-song generator based on a new pianoroll, wasserstein autoencoders, and optimal interpolations.,”  in ISMIR, 2020, pp. 876–883.
  * [7] Jouni Paulus, Meinard Müller, and Anssi Klapuri,  “Audio-based music structure analysis,”  Proceedings of the 11th International Society for Music Information Retrieval Conference, ISMIR 2010, pp. 625–636, 01 2010.
  * [8] Nicolas Lazzari, Andrea Poltronieri, and Valentina Presutti,  “Pitchclass2vec: Symbolic music structure segmentation with chord embeddings,”  arXiv preprint arXiv:2303.15306, 2023.
  * [9] Sangjun Han, Hyeongrae Ihm, Moontae Lee, and Woohyung Lim,  “Symbolic music loop generation with neural discrete representations,”  arXiv preprint arXiv:2208.05605, 2022.
  * [10] Sahil Jain, Kiranmai Pulaparthi, and Chetan Fulara,  “Content based image retrieval,”  Int. J. Adv. Eng. Glob. Technol, vol. 3, pp. 1251–1258, 2015.
  * [11] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf,  “Wasserstein auto-encoders,”  arXiv preprint arXiv:1711.01558, 2017.
  * [12] Marco Gori, Gabriele Monfardini, and Franco Scarselli,  “A new model for learning in graph domains,”  in Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. IEEE, 2005, vol. 2, pp. 729–734.
  * [13] Emmanouil Karystinaios and Gerhard Widmer,  “Cadence detection in symbolic classical music using graph neural networks,”  arXiv preprint arXiv:2208.14819, 2022.
  * [14] Angelo Cesar Mendes da Silva, Diego Furtado Silva, and Ricardo Marcondes Marcacini,  “Heterogeneous graph neural network for music emotion recognition,”  2022\.
  * [15] Dasaem Jeong, Taegyun Kwon, Yoojin Kim, and Juhan Nam,  “Graph neural network for music score data and modeling expressive piano performance,”  in International Conference on Machine Learning. PMLR, 2019, pp. 3060–3070.
  * [16] Meinard Müller,  Fundamentals of music processing: Audio, analysis, algorithms, applications, vol. 5,  Springer, 2015.
  * [17] João Rodrigues, Hui Liu, Duarte Folgado, David Belo, Tanja Schultz, and Hugo Gamboa,  “Feature-based information retrieval of multimodal biosignals with a self-similarity matrix: Focus on automatic segmentation,”  Biosensors, vol. 12, no. 12, pp. 1182, 2022.
  * [18] Ming-Kuei Hu,  “Visual pattern recognition by moment invariants,”  IRE transactions on information theory, vol. 8, no. 2, pp. 179–187, 1962.
  * [19] Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun,  “Masked label prediction: Unified message passing model for semi-supervised classification,”  arXiv preprint arXiv:2009.03509, 2020.
  * [20] Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang,  “Dropedge: Towards deep graph convolutional networks on node classification,”  arXiv preprint arXiv:1907.10903, 2019.
  * [21] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen,  “Graph contrastive learning with augmentations,”  Advances in neural information processing systems, vol. 33, pp. 5812–5823, 2020.
  * [22] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 2015, pp. 234–241.
  * [23] C Raffel,  Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching. 331 Ph. D,  Ph.D. thesis, thesis, Columbia University, 2016.


(a)

(b)

Figure 1. Content-adaptive image representation. Content-adaptive image
representation. Leveraging a tailored differentiable renderer, Image-GS
adaptively distributes and progressively optimizes a set of anisotropic 2D
Gaussians to fit a target image. Image-GS is a flexible image representation
that has high memory & computation efficiency, supports fast random pixel
access, and offers a natural level of detail. (1(a)) shows the learned
Gaussian position distribution (green dots); 20% of Gaussians are plotted for
better visibility. (1(b)) Image-GS’s content-adaptive nature enables it to
wisely allocate resources based on the local signal complexity and preserve
fine image details with higher fidelity than alternative methods. The insets
visualize the corresponding error images, with brighter colors indicating
higher errors.

# Image-GS: Content-Adaptive Image Representation via 2D Gaussians

Yunxiang Zhang  0000-0003-0189-1776<EMAIL_ADDRESS>New York
UniversityUSA ,  Alexandr Kuznetsov  0009-0001-7084-3391
<EMAIL_ADDRESS>,  Akshay Jindal  0000-0003-0557-0726
<EMAIL_ADDRESS>Intel CorporationUSA ,  Kenneth Chen
0000-0002-8095-4407<EMAIL_ADDRESS>New York UniversityUSA ,  Anton Sochenov
0009-0002-7496-8586<EMAIL_ADDRESS>,  Anton Kaplanyan
0000-0002-8376-6719<EMAIL_ADDRESS>Intel CorporationUSA and  Qi Sun
0000-0002-3094-5844<EMAIL_ADDRESS>New York UniversityUSA

###### Abstract.

Neural image representations have recently emerged as a promising technique
for storing, streaming, and rendering visual data. Coupled with learning-based
workflows, these novel representations have demonstrated remarkable visual
fidelity and memory efficiency. However, existing neural image representations
often rely on explicit uniform data structures without content adaptivity or
computation-intensive implicit models, limiting their adoption in real-time
graphics applications.

Inspired by recent advances in radiance field rendering, we propose Image-GS,
a content-adaptive image representation. Using anisotropic 2D Gaussians as the
basis, Image-GS shows high memory efficiency, supports fast random access, and
offers a natural level of detail stack. Leveraging a tailored differentiable
renderer, Image-GS fits a target image by adaptively allocating and
progressively optimizing a set of 2D Gaussians. The generalizable efficiency
and fidelity of Image-GS are validated against several recent neural image
representations and industry-standard texture compressors on a diverse set of
images. Notably, its memory and computation requirements solely depend on and
linearly scale with the number of 2D Gaussians, providing flexible controls
over the trade-off between visual fidelity and run-time efficiency. We hope
this research offers insights for developing new applications that require
adaptive quality and resource control, such as machine perception, asset
streaming, and content generation.

Image Representation

††ccs: Computing methodologies Computer graphics

##  1\. Introduction

For a long time, images have been digitized as uniform pixel grids in both
hardware and software. Such discrete representations do not align with the
physical world, where visual content is continuous and non-uniform. As a
result, these representations suffer from limited efficiency, especially for
domain-specific tasks such as machine vision and content streaming [Li et al.,
2020; Chen et al., 2021].

Neural representations have recently emerged to encode, process, and render
images using neural networks [Chen et al., 2021; Karnewar et al., 2022;
Dosovitskiy et al., 2020; Martel et al., 2021]. Incorporated into learning-
based frameworks, these representations have demonstrated significant
advantages over traditional image formats in terms of visual fidelity, memory
efficiency, and machine vision task performance. However, neural image
representations commonly rely on explicit, uniform data structures that do not
adapt to the image content or computationally heavy implicit networks for
image decoding. Such characteristics become a major barrier in real-time
graphics applications that require fast memory access or resource-dependent
quality adaptation.

To this end, we propose Image-GS, a flexible, compact, and content-adaptive
image representation based on anisotropic, colored 2D Gaussians. Specifically,
each 2D Gaussian is characterized by its mean, covariance, and color. Given a
target image, a group of 2D Gaussians is adaptively spawned based on the
magnitude of local image gradients, with more Gaussians allocated to regions
with high-frequency details. The Gaussian parameters are then optimized via a
tailored differentiable renderer to reconstruct the target image, and
additional Gaussians are progressively added to image regions exhibiting high
reconstruction errors. To accelerate Image-GS’s inference speed, we introduce
a hierarchical grid structure based on binary space partitioning. The content-
adaptive nature of Image-GS enables it to wisely allocate resources based on
the local signal complexity and preserve fine image details with high
fidelity.

Through a series of comparative experiments with several recent neural image
representations and industry-standard texture compression algorithms, we
validate Image-GS’s generalizable performance in terms of visual quality and
memory & computation efficiency. Additionally, Image-GS supports hardware-
friendly fast random access, continuous level-of-detail adaptation, and real-
time inference performance. We hope this research provides insights for
developing novel image representations that have the advantages of both
explicit (direct memory access & adaptive level-of-details) and implicit
(efficient storage & learning-compatible) encoding, and therefore, supporting
specific hardware and application needs.

In summary, this research contributes:

  * •

a flexible, compact, and content-adaptive image representation based on
anisotropic colored 2D Gaussians;

  * •

a tailored differentiable renderer that efficiently aggregates
anisotropic colored 2D Gaussians into images;

  * •

a hierarchical spatial partitioning for accelerated inference.

Figure 2. Optimization pipeline of our Gaussian-based adaptive image
representation. Optimization pipeline of our Gaussian-based adaptive image
representation. At initialization, a group of 2D Gaussian primitives are
adaptively spawned based on the magnitude of local image gradients, with more
Gaussians allocated to regions with fine details (Section 3.3). During
training, the parameters associated with the Gaussian primitives (Section 3.1)
are optimized via a tailored differentiable renderer (Section 3.2) to
reconstruct the target image, and additional Gaussians are progressively added
to image regions exhibiting high reconstruction errors (Section 3.3). Note
that we plot Gaussians as colored elliptical discs with red frames based on
their mean and covariance, and overlay them with rendered images for better
visibility of the training progress.

##  2\. Related Work

###  2.1. Image and Texture Compression

Traditional image compression methods have prioritized efficient storage and
transmission over real-time graphics. Lossless methods optimize pixel
permutation and use entropy encoding [Welch, 1985], while lossy methods
transform image blocks into frequency domains using wavelet [Antonini et al.,
1992] or cosine transforms [Wallace, 1992] followed by quantization and
entropy encoding. Advanced lossy methods also consider human color
sensitivity, higher bit depths, wide color gamuts, user statistics [Alakuijala
et al., 2019], and employ content-adaptive block sizes and looped filtering to
reduce artifacts [Chen et al., 2018]. Despite high compression ratios, these
methods are complex to decode, slow, and unsuitable for non-color data like
normal maps in real-time graphics. In contrast, texture compression methods
aim to reduce GPU bandwidth and support non-color data, random access, and
fast decompression. They operate on small 4x4 pixel blocks, maintaining local
statistics (such as mean and variance) while reducing bits-per-pixel (bpp)
within each block [Delp and Mitchell, 1979]. Each block is compressed
independently, storing per-pixel color values [Campbell et al., 1986], base
color with adjustments [Ström and Akenine-Möller, 2005; Ström and Pettersson,
2007], or color endpoints with interpolation indices [BC, 2024]. Advanced
methods allow dynamic block sizes, HDR content, and content-adaptive
compression strategy per block [Nystad et al., 2012]. These methods offer fast
random access but are limited to an 8:1 compression ratio. Recently,
Vaidyanathan et al. [2023] proposed compressing multiple material textures and
mipmap chains with a small multilayer perceptron (MLP), achieving high
compression and real-time random access. Our Image-GS representation also
achieves a high compression ratio, good visual fidelity, and real-time
inference, with additional content-adaptive optimization, at-will level-of-
detail queries, and variable bit rate.

###  2.2. Neural Image Representation

Neural image representation is an emerging field that diverges from
traditional pixel-based methods, using deep features or implicit neural
functions to encode images. Ballé et al. [2018] use variational autoencoders
(VAEs) with deep hyperprior to create compact latent space image
representation. Chen et al. [2021] transform images into continuous signals
via a 2D feature map and a shared decoder. These methods, trained on fixed
image sets, may struggle to generalize to new images. In contrast, per-image
encoder/decoder approaches employ MLPs to approximate 2D image signals,
enhanced with activation functions like sinusoids [Sitzmann et al., 2020] and
Gabor wavelets [Saragadam et al., 2023], or positional encoding [Tancik et
al., 2020] to capture fine details. These methods can also be extended to
discontinuous signals through hybrid neural-mesh representations [Belhe et
al., 2023]. Karnewar et al. [2022] showed that even traditional grid-based
representations can be improved by adding a fixed non-linearity to
interpolated values. Martel et al. [2021] proposed a hybrid method that uses
multiscale block-coordinate decomposition to adaptively allocate resources
based on local signal complexity. These methods, though capable of
representing complex images with high quality and low memory, often have long
training and inference times and struggle with high-frequency variations due
to single-scale representation. Müller et al. [2022] addressed these issues
with fully fused MLPs and multi-resolution hash grids, enabling quick, memory-
efficient learning and inference of gigapixel images. In Section 4, we
demonstrate that our Image-GS representation achieves superior visual quality
at low bit rate compared to these neural methods while supporting real-time
inference.

#### Gaussian mixtures in graphics

In parallel to neural representation, Gaussian mixture representations are
also seeing an emerging trend in computer graphics. Our work is inspired by
the recent 3D Gaussian Splatting method [Kerbl et al., 2023] that uses
explicit 3D Gaussian functions for real-time high-quality novel-view
synthesis. Many follow-up works have extended this method to dynamic scenes
[Luiten et al., 2023], on-the-fly training and streaming [Sun et al., 2024],
2D Gaussians for surface modeling [Huang et al., 2024], and a wide variety of
other applications [Chen and Wang, 2024]. Although Gaussian mixture models
have been used for image generation [Gepperth and Pfülb, 2021], compression
[Sun et al., 2019, 2021], and stylization [Cheng, 2024], their application as
2D image representation remains largely unexplored in real-time graphics.

##  3\. Method

We first introduce a flexible and compact image representation based on
anisotropic, colored 2D Gaussians (Section 3.1), then present a tailored
differentiable renderer to aggregate Gaussians into pixel values (Section
3.2). The Gaussian parameters are optimized to reconstruct the image content,
while additional 2D Gaussians are progressively added to error-prone regions
(Section 3.3). The resulting representation, named Image-GS, adaptively
allocates 2D Gaussians based on the complexity of local image regions,
achieving a favorable trade-off between visual fidelity and memory/computation
efficiency. To enable low-cost random access and fast rendering speed, we
build an image-specific hierarchical grid to spatially partition optimized
Gaussians and reduce run-time computations (Section 3.4).

###  3.1. Images as Anisotropic 2D Gaussians

Due to the adoption of memory-consuming feature grids [Lombardi et al., 2019;
Sitzmann et al., 2019a] and computation-intensive implicit models [Sitzmann et
al., 2019b; Niemeyer et al., 2020; Martel et al., 2021], existing neural
representations exhibit limited scalability for complex visual data in real-
time graphics applications. To address this issue, we draw inspiration from
the recent success of 3D Gaussian splatting [Kerbl et al., 2023], an explicit
scene representation supporting high-quality and real-time rendering, and
propose to use anisotropic 2D Gaussians as the representation basis for
images.

Similar to the Gaussian primitives in 3D, the geometry and orientation of an
anisotropic 2D Gaussian is characterized by a covariance matrix
$\bm{\Sigma}\in\mathbb{R}^{2\times 2}$ centered at image coordinates
$\bm{\mu}=(u,v)$. Its density value evaluated at an arbitrary pixel location
$\textbf{x}\in\mathbb{R}^{2}$ gives:

(1) $\displaystyle
G(\textbf{x})=\exp\left({-\frac{1}{2}(\textbf{x}-\bm{\mu})^{T}\bm{\Sigma}^{-1}(\textbf{x}-\bm{\mu})}\right),$

To ensure that the covariance matrix $\bm{\Sigma}$ remains physically
feasible, i.e., positive semi-definite, during the numerical optimization
process detailed in Section 3.3, we factorize it into a rotation matrix
$\textbf{R}\in\mathbb{R}^{2\times 2}$ and a diagonal scaling matrix
$\textbf{S}\in\mathbb{R}^{2\times 2}$:

(2)
$\displaystyle\bm{\Sigma}=\textbf{R}\,\textbf{S}\,\textbf{S}^{T}\,\textbf{R}^{T},$

Specifically, we create and maintain a rotation angle $\theta\in[0,\pi]$ and a
scaling vector $\textbf{s}\in\mathbb{R}_{+}^{2}$ for each 2D Gaussian. These
parameters are optimized via stochastic gradient descent and clipped to their
allowed value range during the training process. The rotation matrix R and the
scaling matrix S are constructed on the fly using $\theta$ and s at both the
training and inference stages.

Unlike Gaussian splatting in 3D that adopts spherical harmonics to model view-
dependent color effects [Kerbl et al., 2023], we only utilize a 3-dimensional
vector $\textbf{c}\in\mathbb{R}_{+}^{3}$ for each 2D Gaussian to store its RGB
values, as an image essentially captures a single view of a scene. In
addition, 3D Gaussian splatting relies on a per-Gaussian trainable opacity
parameter for depth-based occlusion computation and $\alpha$-blending during
the rendering process. By contrast, depth information is not necessary for
accurate rendering in the 2D space, and 2D Gaussian primitives can be
effectively aggregated regardless of their relative order, as we explain in
Section 3.2. Therefore, our 2D Gaussian primitives do not have an opacity
property.

Based on the discussion above, an arbitrary 2D Gaussian primitive in our image
representation, $G_{i}$ with $1\leq i\leq N_{\text{g}}$, is fully
characterized by a vector containing 8 trainable parameters
$\textbf{p}_{i}\in\mathbb{R}^{8}$:

(3)
$\displaystyle\textbf{p}_{i}\coloneqq\textbf{p}_{i}(\bm{\mu}_{i},\theta_{i},\textbf{s}_{i},\textbf{c}_{i}),$

###  3.2. Aggregating 2D Gaussians into Images

While Gaussian splatting in 3D necessitates per-view depth sorting and per-
Gaussian opacity to handle occlusions and enforce multi-view consistency,
especially when there exist objects that are visible in some views but not in
others, we argue that depth sorting and occlusion modeling can be safely
omitted in the 2D case without negatively impacting the rendering quality.
Since an image only captures a single view of an underlying 3D scene, there is
no need to account for any potential multi-view consistency issues in other
viewing directions due to inconsistent depth information. As a result, an
image represented by a set of 2D Gaussians can be rendered by applying the
Gaussians in arbitrary order, as long as the final rendering result is pixel-
wise accurate for that particular image.

With this insight in mind, we simplify the standard point-based
$\alpha$-blending approach from the literature [Yifan et al., 2019; Kopanas et
al., 2021, 2022] by treating our Gaussian primitives as an unordered set of
semi-transparent anisotropic points, and aggregate their color contributions
to form an image pixel $\textbf{x}\in\mathbb{R}^{2}$:

(4)
$\displaystyle\textbf{c}_{\text{r}}(\textbf{x})=\sum_{i=1}^{N_{\text{g}}}G_{i}(\textbf{x})\cdot\textbf{c}_{i},$

While this naive formulation effectively renders 2D Gaussians in an order-
agnostic manner, it involves all Gaussians to compute the color of each pixel.
Such global pixel-Gaussian correlation breaks the data locality that is
necessary for efficient random pixel access on GPUs and largely limits the
overall rendering speed [Vaidyanathan et al., 2023]. Moreover, the fact that
each Gaussian receives gradients through pixel errors across the whole image
domain makes the optimization less consistent across iterations and slower to
converge, as training updates in irrelevant, faraway image regions can
influence a 2D Gaussian in unpredictable and undesirable ways.

For this reason, we limit the number of Gaussians that contribute to a given
pixel based on their density. Specifically, we first evaluate and rank the
density values of all Gaussians at $\textbf{x}\in\mathbb{R}^{2}$, then only
keep the top-K and use their density values as weights for rendering. Besides,
we also normalize these weights before the aggregation to avoid situations
where certain pixels are not adjacent enough to their top-K Gaussians to
receive sufficient color contributions. Overall, our rendering algorithm for
2D Gaussians is formulated as:

(5)
$\displaystyle\textbf{c}_{\text{r}}(\textbf{x})=\frac{1}{\sum_{j\in\mathcal{N}_{\text{K}}(\textbf{x})}G_{j}(\textbf{x})}\sum_{j\in\mathcal{N}_{\text{K}}(\textbf{x})}G_{j}(\textbf{x})\cdot\textbf{c}_{j},$

where $\mathcal{N}_{\text{K}}(\textbf{x})$ represents the set of top-K
Gaussians for x, defined as
$\mathcal{N}_{\text{K}}(\textbf{x})=\text{top-K}(\\{G_{i}(\textbf{x})\\}_{i=1}^{N_{\text{g}}})$.
Throughout our experiments, K is set to $10$ if not explicitly specified
otherwise.

Figure 3. Accelerated inference via hierarchical spatial partitioning.
Accelerated inference via hierarchical spatial partitioning. After training,
the target image is adaptively subdivided into $N_{\text{b}}$ disjoint blocks
(solid orange) such that each block covers a subset of optimized Gaussians
(red dots show their locations) with similar cardinality ($\approx
N_{\text{g}}/N_{\text{b}}$). Instead of querying all $N_{\text{g}}$ Gaussians,
rendering a pixel in block $B_{k}$ only involves the Gaussians within its
corresponding outer shell $S_{k}$ (dotted yellow). The size difference between
each block-shell pair is designed to mitigate boundary artifacts. Only 10% of
Gaussians are plotted for better visibility.

###  3.3. Content-Adaptive Initialization and Optimization

Unlike scene geometries which are commonly sparse in the 3D space, images
(except the ones with an alpha channel) typically contain dense color
information everywhere in the 2D image domain. A good initialization of our 2D
Gaussian primitives, therefore, should output a spatial distribution that
covers the entire image domain while emphasizing regions with high-frequency,
fine details.

To this end, we propose a content-adaptive sampling strategy that combines
local image gradient guidance with uniform sampling. Specifically, we only
sample pixel locations to initialize the position of Gaussians. During the
position sampling for each Gaussian, the probability of a given pixel x being
sampled is a weighted sum of the relative magnitude of its local image
gradient and a constant shared across all pixel locations, as formulated
below. In particular, the left term advocates image-content adaptivity, while
the right term ensures appropriate image-domain coverage.

(6)
$\displaystyle\mathds{P}_{\text{init}}(\textbf{x})=\frac{(1-\lambda_{\text{init}})\cdot\lVert\nabla
I(\textbf{x})\rVert_{2}}{\sum_{h=1}^{H}\sum_{w=1}^{W}\lVert\nabla
I(\textbf{x}_{h,w})\rVert_{2}}+\frac{\lambda_{\text{init}}}{H\cdot W},$

where $H/W$ give the height/width of the image, and $\nabla I(\cdot)$ denotes
the image gradient operator. $\lambda_{\text{init}}\in[0,1]$ balances local
content adaptivity and uniform coverage, and is set to $0.3$ in our
experiments. In addition to position initialization, all Gaussians are
assigned the target pixel color at their initialized location
$\textbf{c}_{\text{t}}(\textbf{x})$.

To optimize the Gaussian parameters toward reconstructing the target image,
during each training iteration, we sample a large set of pixel locations and
compute the $L_{1}$ loss between rendered (using the differentiable renderer
in Section 3.2) and ground-truth pixel values at sampled locations. Similar to
Equation 6, the sampling distribution again has an image-gradient term to
emphasize complex local image content and a constant term to enforce broad
coverage.

(7)
$\displaystyle\mathds{P}_{\text{opt}}(\textbf{x})=\frac{(1-\lambda_{\text{opt}})\cdot\lVert\nabla
I(\textbf{x})\rVert_{2}}{\sum_{h=1}^{H}\sum_{w=1}^{W}\lVert\nabla
I(\textbf{x}_{h,w})\rVert_{2}}+\frac{\lambda_{\text{opt}}}{H\cdot W},$

The trade-off parameter $\lambda_{\text{opt}}\in[0,1]$ is set to $0.8$ in our
experiments.

Besides the initially created Gaussians, we also periodically add new Gaussian
primitives to image regions having high reconstruction errors as the training
progresses. This is achieved by sampling pixel locations based on their
relative error magnitude.

(8)
$\displaystyle\mathds{P}_{\text{add}}(\textbf{x})=\frac{\left|\textbf{c}_{\text{r}}(\textbf{x})-\textbf{c}_{\text{t}}(\textbf{x})\right|}{\sum_{h=1}^{H}\sum_{w=1}^{W}\left|\textbf{c}_{\text{r}}(\textbf{x}_{h,w})-\textbf{c}_{\text{t}}(\textbf{x}_{h,w})\right|},$

Figure 2 illustrates the optimization pipeline of Image-GS.

###  3.4. Hierarchical Spatial Partitioning for Efficient Inference

For real-time graphics applications with demanding performance requirements,
the fact that rendering a single pixel requires evaluating and ranking the
density values of all Gaussian primitives is computationally infeasible.
Fortunately, the constraint on the number of Gaussians making contributions to
each pixel (Equation 5) enforces pixel-Gaussian locality during the
optimization and ensures that the top-K Gaussians of each pixel are located
within a small local neighborhood after the training.

Following this intuition, we tailor a hierarchical grid structure to each
Image-GS-represented image to spatially partition the set of all
$N_{\text{g}}$ optimized 2D Gaussians into $N_{\text{b}}$ smaller subsets of
similar cardinality ($\approx N_{\text{g}}/N_{\text{b}}$). This is achieved
via a variant of binary space partitioning (BSP), where only horizontal and
vertical splitting lines are employed. Given the maximum number of Gaussians
allowed in each subset $N_{\text{max}}$, a BSP tree is constructed to
iteratively and adaptively subdivide the image domain into smaller blocks
until the termination criterion is met. Specifically, each node in the BSP
tree corresponds to a block in the image space. The ensemble of leaf nodes
forms a set of disjoint blocks that together cover the entire image domain.
Each leaf block undergoes a sequence of alternating horizontal and vertical
splitting. Each splitting line is computed such that it separates a leaf block
into two smaller blocks with an equal number of Gaussians. This process
continues until no leaf block contains more than $N_{\text{max}}$ Gaussians.
The resulting leaf blocks give the desired subsets of Gaussians.

However, rendering pixels in a leaf block $B_{k}$ using only the Gaussians in
$B_{k}$ can result in severe boundary artifacts, where neighboring pixels in
two adjacent leaf blocks exhibit unnatural color changes. To address this
issue, we extend the boundaries of each block $B_{k}$ by $1/4$ to obtain its
corresponding encompassing shell $S_{k}$ and make all Gaussians within $S_{k}$
available for rendering pixels in $B_{k}$. Figure 3 illustrates the resulting
hierarchical spatial partitioning. Instead of querying all $N_{\text{g}}$
Gaussians, rendering a pixel only involves $\approx
9N_{\text{g}}/4N_{\text{b}}$ Gaussians now. Figure 4 shows the inference
acceleration performance for a varying number of blocks.

##  4\. Evaluation

###  4.1. Experimental Setup

#### Evaluation dataset

To comprehensively understand and validate Image-GS’s image representation
performance, we prepared an evaluation dataset of 30 RGB images (2K$\times$2K
resolution) with diverse characteristics, including 10 photographs, 4 vector-
style images, 8 texture maps, 4 anime posters, and 4 watercolor/oil paintings.

#### Evaluation metrics

We adopt 4 image quality metrics, PSNR, SSIM [Wang et al., 2004], LPIPS [Zhang
et al., 2018], and FLIP [Andersson et al., 2020], to evaluate the visual
fidelity of Image-GS and establish quantitative comparisons with baseline
methods. We also report the parameter size of each representation and the
corresponding bit rate in bpp (bits per pixel) to evaluate their memory
efficiency.

#### Implementation

The only trainable parameters in Image-GS are the Gaussian parameters in
Equation 3. By mapping image domains to $[0,1]^{2}$ grids, Image-GS works for
target images of any aspect ratio and resolution. At initialization, the
Gaussian positions $\bm{\mu}$ and colors c are populated by sampling pixel
coordinates based on Equation 6. Their scaling vectors s and rotation angles
$\theta$ are set to $2/\max(H,W)$ and $0$, respectively. We adopt the Adam
optimizer [Kingma and Ba, 2015] to iteratively update these parameters for 50K
iterations. The learning rates for $(\bm{\mu},\textbf{c},\textbf{s},\theta)$
start at $($2$e$-4, 2$e$-3, 1$e$-3, 1$e$-3$)$ and decay by 10 (only once) if
no improvement (PSNR and SSIM are computed every 1K iterations) has been made
for 3 consecutive measurements. During each iteration, 10K pixel locations
$\mathcal{P}$ are sampled based on Equation 7 to evaluate the loss function:

(9) $\displaystyle
L=\frac{1}{\left|\mathcal{P}\right|}\sum_{\textbf{x}\in\mathcal{P}}\left|\textbf{c}_{\text{r}}(\textbf{x})-\textbf{c}_{\text{t}}(\textbf{x})\right|,$

During training, we progressively allocate additional Gaussians to image
regions with high fitting errors based on Equation 8. For a budget of
$N_{\text{g}}$ Gaussians, we initialize training with $N_{\text{g}}/2$
Gaussians and optimize for 10K iterations. $N_{\text{g}}/8$ additional
Gaussians are added every 5K iterations until the budget runs out. We
implement Image-GS in PyTorch [Paszke et al., 2019] and use half-precision
floating-point numbers (float16) for all parameters.

Figure 4. Performance analysis with hierarchical grid partitioning.
Performance analysis with hierarchical grid partitioning. We measure the time
of rendering 10K pixels with Image-GS (8K Gaussians) for a varying number of
BSP blocks (Section 3.4). The measurement is performed on an NVIDIA GeForce
RTX 3080 GPU. Each data point is averaged over 100 trials. Note that our
proof-of-concept implementation uses pure PyTorch code only. Further
acceleration can be achieved with customized CUDA kernels.

###  4.2. Acceleration via Hierarchical Spatial Partitioning

Using the acceleration technique introduced in Section 3.4, we analyze Image-
GS’s system performance under a varying number of blocks. With the BSP tree
constructed, the image domain $[0,1]^{2}$ is hierarchically partitioned into
$N_{\text{b}}$ disjoint blocks. Each block $B_{k}$ is defined by the
coordinates of its top-left and bottom-right corners
$(\textbf{x}_{1},\textbf{x}_{2})$. At run-time, the boundary locations of its
corresponding shell $S_{k}$ are derived from $(\textbf{x}_{1},\textbf{x}_{2})$
on the fly and used to filter the Gaussians within $S_{k}$ for rendering the
pixels within $B_{k}$.

#### Results and discussion

Figure 4 illustrates the run-time efficiency of Image-GS (8K Gaussians) for a
varying number of spatial partitioning blocks. The measurement is performed on
an NVIDIA GeForce RTX 3080 GPU. The inference time required for rendering 10K
pixels decreases from 4.59 ms for 16 blocks to 0.70 ms for 192 blocks, showing
a 6.56$\times$ acceleration. Further engineering efforts, such as customized
CUDA kernels, could enable substantial acceleration over our proof-of-concept
implementation in pure PyTorch code but are beyond the scope of this work.
Notably, storing these block coordinates for accelerated rendering only
requires $4N_{\text{b}}$ float16 parameters, and incurs $8N_{\text{b}}$ bytes
additional size to our representation, which is practically negligible
compared to memory consumed by the Gaussian parameters. For instance, 150
blocks (1.20 KB) only account for $0.93\%$ memory consumption of an Image-GS-
represented image with 8K Gaussians. These results demonstrate the scalability
of our acceleration approach via hierarchical spatial partitioning.

###  4.3. Visual Fidelity vs Memory Efficiency

Table 1. Trade-off between visual fidelity and memory efficiency. Trade-off between visual fidelity and memory efficiency. We adjust the size of Image-GS by varying the number of 2D Gaussians therein (“nK”). $\uparrow$/$\downarrow$ symbols indicate that higher/lower values are better. Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FLIP$\downarrow$ | Size$\downarrow$
---|---|---|---|---|---
Ours (1K) | 24.48 | 0.7946 | 0.2734 | 0.1748 | 16 KB
Ours (2K) | 26.80 | 0.8316 | 0.2076 | 0.1392 | 32 KB
Ours (4K) | 29.43 | 0.8706 | 0.1573 | 0.1074 | 64 KB
Ours (6K) | 30.85 | 0.8828 | 0.1310 | 0.0967 | 96 KB
Ours (8K) | 32.19 | 0.8912 | 0.1176 | 0.0866 | 128 KB

We optimize Image-GS to the 22 non-texture images in our evaluation dataset
under varying memory consumption. By adjusting the number of 2D Gaussians, we
obtain 5 different bit-rate levels (in bpp) for Image-GS: 0.244, 0.183, 0.122,
0.061, 0.031.

#### Results and discussion

For all 5 bit-rate levels, we render Image-GS-represented images at
2K$\times$2K resolution and evaluate their visual fidelity against the
reference images. As summarized in Table 1, Image-GS achieves an average
performance of 32.19 (PSNR), 0.89 (SSIM), 0.12 (LPIPS), and 0.09 (FLIP) with a
memory size of 128 KB. Even at an ultra-low bit rate of 0.031 bpp (16 KB),
Image-GS is able to fit the target images at 24.48 (PSNR), 0.79 (SSIM), 0.27
(LPIPS), and 0.17 (FLIP) on average. Notably, our progressive optimization
strategy (Section 3.3) automatically generates a sequence of Image-GS-
represented images at varying bit rates during training. This forms a natural
level of detail (LoD) stack for the target image. Figure 6 shows several
samples rendered at 2K$\times$2K resolution.

###  4.4. Comparison with Neural Image Representations

Table 2. Quantitative comparison with previous neural image representations. Quantitative comparison with previous neural image representations. We measure the representation efficiency of Image-GS in terms of visual quality and memory consumption against four baseline methods. All metrics are computed on images rendered at 2K$\times$2K resolution. Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FLIP$\downarrow$ | Size$\downarrow$
---|---|---|---|---|---
ReLU | 23.25 | 0.7148 | 0.4288 | 0.2012 | 132 KB
SIREN | 27.48 | 0.7760 | 0.3662 | 0.1613 | 135 KB
WIRE | 26.53 | 0.6996 | 0.4062 | 0.2117 | 134 KB
I-NGP | 27.12 | 0.7815 | 0.2641 | 0.1786 | 141 KB
Ours | 32.19 | 0.8912 | 0.1176 | 0.0866 | 128 KB

#### Baseline methods

We establish comparisons with 4 recent neural image representations: ReLU
fields [Karnewar et al., 2022], SIREN [Sitzmann et al., 2020], WIRE [Saragadam
et al., 2023], and Instant NGP [Müller et al., 2022]. For fair comparisons
under similar bit rates, we modify these baseline models to match our bit
rates by decreasing the resolution of feature grids (ReLU, Instant NGP) and/or
reducing the number of hidden layers/features (Instant NGP, WIRE, SIREN). The
22 non-texture images in our evaluation dataset are employed.

#### Results and discussion

As summarized in Table 2, our Gaussian-based representation Image-GS achieves
an average performance of 32.19 (PSNR), 0.89 (SSIM), 0.12 (LPIPS), and 0.09
(FLIP), significantly outperforming all baseline methods despite a smaller
memory footprint. Figure 7 shows several visual examples. At an ultra-low bit
rate of 0.244 bpp, all baselines show different levels of image distortions
and artifacts. For instance, decreasing the resolution of grid-based methods
(ReLU, Instant NGP) leads to block artifacts due to feature vectors being
interpolated at sparser locations. Implicit functions (SIREN, WIRE) exhibit
artifacts such as ringing or blurring, which are more pronounced after
reducing the base network parameters. By contrast, Image-GS exhibits much less
visible artifacts.

(a)

(b)

(c)

(d)

(a)

(b)

(c)

(d)

Figure 5. Qualitative comparison with GPU texture compressors. Qualitative
comparison with GPU texture compressors. For comparisons under similar
compression rates, we use the mipmap level 2 for both BC1 and BC7 to match
their bit rates to ours. The bottom-right insets visualize the corresponding
error images, with brighter colors indicating higher errors.

###  4.5. Comparison with Texture Compression Methods

#### Baseline methods

We compare to two BCx variants designed for RGB texture compression, BC1
(highest settings) and BC7 (quality 0.25), using their AMD Compressonator
[AMD, 2024] implementations. Since these settings only offer a maximum
compression of up to 4 bpp, we use the compressed images in the mipmap level 2
for both algorithms and upsample them back to the original resolution using
bilinear interpolation to match our 0.244 bpp for fair comparisons. The 8
texture maps in our evaluation dataset are employed.

#### Results and discussion

As summarized in Table 3, our Gaussian-based representation Image-GS achieves
an average performance of 33.03 (PSNR), 0.89 (SSIM), 0.16 (LPIPS), and 0.08
(FLIP), consistently outperforming the two baseline methods despite a smaller
memory footprint. Figure 5 presents visual comparisons of Image-GS versus BC1
and BC7. The error maps indicate that Image-GS shows visibly less distortions
from the uncompressed reference images. More samples can be found in
LABEL:fig:supplementary-texture-1 and LABEL:fig:supplementary-texture-2 in the
appendix.

Table 3. Quantitative comparison with GPU texture compressors. Quantitative comparison with GPU texture compressors. All metrics are computed on textures rendered at 2K$\times$2K resolution. We use the mipmap level 2 for BC1/BC7 to match their bit rates to ours. Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FLIP$\downarrow$ | Bit Rate$\downarrow$
---|---|---|---|---|---
BC1 | 28.86 | 0.8714 | 0.2369 | 0.1012 | 0.250 bpp
BC7 | 27.31 | 0.8425 | 0.3006 | 0.1054 | 0.253 bpp
Ours | 33.03 | 0.8922 | 0.1591 | 0.0757 | 0.244 bpp

##  5\. Potential Applications

###  5.1. Adaptive Image Representation for Machine Vision

Deep-learning techniques have found tremendous success across a wide range of
machine vision tasks, such as object tracking, facial recognition, and depth
estimation. These achievements typically require high-resolution images as
inputs for large neural network models to extract relevant information. While
images represent visual information using a uniform grid structure, the task-
related information often only resides in a few scattered local image regions
exhibiting distinct features. Such spatially uniform representation for visual
inputs results in an efficient usage of processing time and energy
consumption. By contrast, Image-GS adaptively distributes representation
resources across the image domain, with more bits allocated to image regions
showing fine details. Compared to standard images, our Gaussian-based
representation offers a more flexible and efficient encoding of visual inputs
for machine vision models to operate on. Directly running machine vision
models on Image-GS-represented images may improve their computation
efficiency.

###  5.2. Resource-Adaptive Image Representation

The only trainable parameters in Image-GS are the ones associated with the 2D
Gaussians. Consequently, its memory and computation requirements solely depend
on and linearly scale with the number of 2D Gaussians. This provides a
flexible way to trade off between visual fidelity and run-time efficiency.
Moreover, our progressive optimization strategy with error-guided Gaussian
addition produces a sequence of Image-GS-represented images at varying bit
rates during training, which forms a natural level of detail stack for the
target image. These properties of Image-GS are useful in situations where
computational resources or network bandwidths are limited, such as mobile
computing or web album streaming. While conventional compression algorithms
may uniformly degrade the visual quality across the entire image domain,
Image-GS is capable of maximizing the usage of available resources for
optimized visual quality.

###  5.3. Image Restoration and Enhancement

The inherent low-frequency nature of Gaussian functions makes our Image-GS
representation robust to various high-frequency image distortions and
artifacts caused by JPEG compression (blockiness, ringing, and contouring),
quantization errors (color banding and aliasing), transmission over noisy
channels and low-light imagery (salt and pepper noise). We empirically observe
in our experiments that, when using Image-GS to represent images containing
such high-frequency artifacts, Image-GS effectively eliminates them during
optimization and outputs artifact-free rendering. These observations suggest
that Image-GS has the potential to accomplish certain image restoration and
enhancement tasks.

##  6\. Limitations and Future Work

#### Hierarchical spatial guidance during optimization

While Image-GS is designed to be content-adaptive, numerical optimization
algorithms, such as stochastic gradient descent, sometimes have trouble
shifting the spatial distribution of Gaussians toward the global optimum.
Following the acceleration technique in Section 3.4, we plan to introduce a
dynamic BSP tree that guides the spatial allocation of Gaussians. During
training, the tree structure and Gaussian parameters can be jointly optimized
by formulating and solving an integer programming problem similar to [Martel
et al., 2021].

#### Dynamic visual content

In this work, we demonstrate that images can be adaptively and efficiently
represented with an explicit basis, anisotropic 2D Gaussian. Motivated by
recent research that incorporates dynamics into Gaussian-based 3D scene
representations [Luiten et al., 2023; Stavros et al., 2024], we plan to apply
our method to represent videos by additionally modeling the movements of 2D
Gaussians within the image plane. We envision this extension to benefit
graphics applications such as video streaming.

##  7\. Conclusion

In this paper, we proposed Image-GS, a flexible, compact, and content-adaptive
image representation based on anisotropic, colored 2D Gaussians. Image-GS has
high memory & computation efficiency, supports fast random access, and offers
a natural level of detail through progressive optimization. The content-
adaptive nature of Image-GS enables it to wisely allocate resources based on
the signal complexity of local image regions and preserve fine image details
with higher fidelity than alternative methods. Through a series of
quantitative comparisons with recent neural image representations and
industry-standard texture compression algorithms, we validated the visual
fidelity and memory efficiency of Image-GS. We hope this research will
establish new grounds for developing new representations for visual data.

## References

  * [1]
  * AMD [2024] 2024\.  AMD Compressonator.  https://gpuopen.com/compressonator/.
  * BC [2024] 2024\.  Texture Block Compression in Direct3D 11.  https://learn.microsoft.com/en-us/windows/win32/direct3d11/texture-block-compression-in-direct3d-11.
  * Alakuijala et al. [2019] Jyrki Alakuijala, Ruud Van Asseldonk, Sami Boukortt, Martin Bruse, Iulia-Maria Comșa, Moritz Firsching, Thomas Fischbacher, Evgenii Kliuchnikov, Sebastian Gomez, Robert Obryk, et al. 2019\.  JPEG XL next-generation image compression architecture and coding tools. In _Applications of digital image processing XLII_ , Vol. 11137. SPIE, 112–124.
  * Andersson et al. [2020] Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D Fairchild. 2020.  FLIP: A Difference Evaluator for Alternating Images.  _Proc. ACM Comput. Graph. Interact. Tech._ 3, 2 (2020), 15–1.
  * Antonini et al. [1992] Marc Antonini, Michel Barlaud, Pierre Mathieu, and Ingrid Daubechies. 1992.  Image coding using wavelet transform.  _IEEE Trans. Image Processing_ 1 (1992), 20–5.
  * Ballé et al. [2018] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. 2018.  Variational image compression with a scale hyperprior.  _arXiv preprint arXiv:1802.01436_ (2018).
  * Belhe et al. [2023] Yash Belhe, Michaël Gharbi, Matthew Fisher, Iliyan Georgiev, Ravi Ramamoorthi, and Tzu-Mao Li. 2023.  Discontinuity-Aware 2D Neural Fields.  _ACM Transactions on Graphics (TOG)_ 42, 6 (2023), 1–11.
  * Campbell et al. [1986] Graham Campbell, Thomas A DeFanti, Jeff Frederiksen, Stephen A Joyce, Lawrence A Leske, John A Lindberg, and Daniel J Sandin. 1986.  Two bit/pixel full color encoding.  _ACM SIGGRAPH Computer Graphics_ 20, 4 (1986), 215–223.
  * Chen and Wang [2024] Guikun Chen and Wenguan Wang. 2024.  A survey on 3d gaussian splatting.  _arXiv preprint arXiv:2401.03890_ (2024).
  * Chen et al. [2021] Yinbo Chen, Sifei Liu, and Xiaolong Wang. 2021.  Learning continuous image representation with local implicit image function. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 8628–8638.
  * Chen et al. [2018] Yue Chen, Debargha Murherjee, Jingning Han, Adrian Grange, Yaowu Xu, Zoe Liu, Sarah Parker, Cheng Chen, Hui Su, Urvang Joshi, et al. 2018\.  An overview of core coding tools in the AV1 video codec. In _2018 picture coding symposium (PCS)_. IEEE, 41–45.
  * Cheng [2024] Chang-Chieh Cheng. 2024.  Image representation and reconstruction by compositing Gaussian ellipses.  _IET Image Processing_ 18, 2 (2024), 493–506.
  * Delp and Mitchell [1979] Edward Delp and O Mitchell. 1979.  Image compression using block truncation coding.  _IEEE transactions on Communications_ 27, 9 (1979), 1335–1342.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In _International Conference on Learning Representations_.
  * Gepperth and Pfülb [2021] Alexander Gepperth and Benedikt Pfülb. 2021.  Image modeling with deep convolutional gaussian mixture models. In _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1–9.
  * Huang et al. [2024] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2024.  2D Gaussian Splatting for Geometrically Accurate Radiance Fields.  _arXiv preprint arXiv:2403.17888_ (2024).
  * Karnewar et al. [2022] Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra. 2022.  Relu fields: The little non-linearity that could. In _ACM SIGGRAPH 2022 Conference Proceedings_. 1–9.
  * Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. 2023.  3D Gaussian Splatting for Real-Time Radiance Field Rendering.  _ACM Transactions on Graphics (TOG)_ 42, 4 (2023), 1–14.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kopanas et al. [2022] Georgios Kopanas, Thomas Leimkühler, Gilles Rainer, Clément Jambon, and George Drettakis. 2022.  Neural point catacaustics for novel-view synthesis of reflections.  _ACM Transactions on Graphics (TOG)_ 41, 6 (2022), 1–15.
  * Kopanas et al. [2021] Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis. 2021.  Point-Based Neural Rendering with Per-View Optimization. In _Computer Graphics Forum_ , Vol. 40. Wiley Online Library, 29–43.
  * Li et al. [2020] Mengtian Li, Yu-Xiong Wang, and Deva Ramanan. 2020.  Towards streaming perception. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_. Springer, 473–488.
  * Lombardi et al. [2019] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019.  Neural volumes: learning dynamic renderable volumes from images.  _ACM Transactions on Graphics (TOG)_ 38, 4 (2019), 1–14.
  * Luiten et al. [2023] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. 2023.  Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis.  _arXiv preprint arXiv:2308.09713_ (2023).
  * Martel et al. [2021] Julien NP Martel, David B Lindell, Connor Z Lin, Eric R Chan, Marco Monteiro, and Gordon Wetzstein. 2021.  Acorn: adaptive coordinate networks for neural scene representation.  _ACM Transactions on Graphics (TOG)_ 40, 4 (2021), 1–13.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM transactions on graphics (TOG)_ 41, 4 (2022), 1–15.
  * Niemeyer et al. [2020] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. 2020.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 3504–3515.
  * Nystad et al. [2012] Jörn Nystad, Anders Lassen, Andy Pomianowski, Sean Ellis, and Tom Olson. 2012.  Adaptive scalable texture compression. In _Proceedings of the Fourth ACM SIGGRAPH/Eurographics Conference on High-Performance Graphics_. 105–114.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ 32 (2019), 8026–8037.
  * Saragadam et al. [2023] Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. 2023.  Wire: Wavelet implicit neural representations. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18507–18516.
  * Sitzmann et al. [2020] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. 2020.  Implicit neural representations with periodic activation functions.  _Advances in neural information processing systems_ 33 (2020), 7462–7473.
  * Sitzmann et al. [2019a] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. 2019a.  Deepvoxels: Learning persistent 3d feature embeddings. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2437–2446.
  * Sitzmann et al. [2019b] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. 2019b.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Stavros et al. [2024] Stavros Stavros, Tobias Zirr1, Alexandr Kuznetsov, Georgios Kopanas, and Anton Kaplanyan. 2024.  N-Dimensional Gaussians for Fitting of High Dimensional Functions. In _ACM SIGGRAPH 2024 Conference Proceedings_. 1–9.
  * Ström and Akenine-Möller [2005] Jacob Ström and Tomas Akenine-Möller. 2005.  i PACKMAN: High-quality, low-complexity texture compression for mobile phones. In _Proceedings of the ACM SIGGRAPH/EUROGRAPHICS conference on Graphics hardware_. 63–70.
  * Ström and Pettersson [2007] Jacob Ström and Martin Pettersson. 2007.  ETC 2: texture compression using invalid combinations. In _Graphics Hardware_ , Vol. 7. 49–54.
  * Sun et al. [2024] Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 2024.  3dgstream: On-the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint videos.  _arXiv preprint arXiv:2403.01444_ (2024).
  * Sun et al. [2019] Jianjun Sun, Yan Zhao, and Shigang Wang. 2019.  Image compression using GMM model optimization. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 1797–1801.
  * Sun et al. [2021] Jianjun Sun, Yan Zhao, Shigang Wang, and Jian Wei. 2021.  Image compression based on Gaussian mixture model constrained using Markov random field.  _Signal Processing_ 183 (2021), 107990.
  * Tancik et al. [2020] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. 2020.  Fourier features let networks learn high frequency functions in low dimensional domains.  _Advances in neural information processing systems_ 33 (2020), 7537–7547.
  * Vaidyanathan et al. [2023] Karthik Vaidyanathan, Marco Salvi, Bartlomiej Wronski, Tomas Akenine-Moller, Pontus Ebelin, and Aaron Lefohn. 2023.  Random-Access Neural Compression of Material Textures.  _ACM Transactions on Graphics (TOG)_ 42, 4 (2023), 1–25.
  * Wallace [1992] Gregory K Wallace. 1992.  The JPEG still picture compression standard.  _IEEE transactions on consumer electronics_ 38, 1 (1992), xviii–xxxiv.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ 13, 4 (2004), 600–612.
  * Welch [1985] Terry A Welch. 1985.  High speed data compression and decompression apparatus and method.  US Patent 4,558,302.
  * Yifan et al. [2019] Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung. 2019.  Differentiable surface splatting for point-based geometry processing.  _ACM Transactions on Graphics (TOG)_ 38, 6 (2019), 1–14.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 586–595.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(a)

(b)

(c)

(d)

(e)

(f)

Figure 6. Illustration of the trade-off between visual quality and
memory/computation efficiency with Image-GS. Illustration of the trade-off
between visual quality and memory/computation efficiency with Image-GS. The
bit rate of Image-GS is adjusted by controlling the maximum number of 2D
Gaussians that are allowed. Notably, our progressive optimization with error-
guided Gaussian addition generates a sequence of Image-GS-represented images
at varying bit rates along the way, which forms a natural level of detail for
the target image.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(a)

(b)

(c)

(d)

(e)

(f)

Figure 7. Qualitative comparison with previous neural image representations.
Qualitative comparison with previous neural image representations. Our
evaluation dataset covers various image types, including photographs,
watercolor/oil paintings, anime posters, and vector-style images. Notably, the
content-adaptive nature of Image-GS enables it to wisely allocate resources
based on the complexity of local image regions and better preserve fine image
details than the baseline methods under similar memory consumption. The model
size (in KB) for ReLU fields, SIREN, WIRE, Instant NGP, and our Image-GS are
132, 135, 134, 141, and 128, respectively.