Colin Cleveland
Department of Informatics, King’s College London, London, UK
E-mail:
Chin-Yen Lee
Department of Mathematics, National Central University, Taoyuan, 32056, Taiwan
E-mail:
Shen-Fu Tsai
Department of Mathematics, National Central University, Taoyuan, 32056, Taiwan
E-mail:
Wei-Hsuan Yu
Department of Mathematics, National Central University, Taoyuan, 32056, Taiwan
E-mail:
Hsuan-Wei Lee†
Institute of Sociology, Academia Sinica, Taipei City, Taiwan
E-mail: Many applications, ranging from natural to social sciences, rely on
graphlet analysis for the intuitive and meaningful characterization of
networks employing micro-level structures as building blocks. However, it has
not been thoroughly explored in heterogeneous graphs, which comprise various
types of nodes and edges. Finding graphlets and orbits for heterogeneous
graphs is difficult because of the heterogeneity and abundance of semantic
information. We consider heterogeneous graphs, which can be treated as colored
graphs. By applying the canonical label technique, we determine the graph
isomorphism problem with multiple states on nodes and edges. With minimal
parameters, we build all non-isomorphic graphs and associated orbits. We
provide a Python package that can be used to generate orbits for colored
directed graphs and determine the frequency of orbit occurrence. Finally, we
provide four examples to illustrate the use of the Python package. graph
isomorphism, orbit isomorphism, graphlet

**footnotetext: These authors contributed equally to this work††footnotetext:
Corresponding author

# Graphlet and Orbit Computation on Heterogeneous Graphs

Colin Cleveland∗ Chin-Yen Lee∗ Shen-Fu Tsai  Wei-Hsuan Yu  Hsuan-Wei Lee†
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
##  1 Introduction

We live in a data-rich world and it’s easy to obtain immense, directed,
dynamic, and networked data. Graphlet analysis is a useful analytical method
for studying graph-based data by decomposing graphs into many basic and often
meaningful elements. Researchers can explain the mechanism of these small unit
graphs by focusing on the numbers of these elements and conducting statistical
tests and other analyses. This approach has a wide range of applications,
including biology Milenković and Pržulj (2008), chemistry Wang _et al._
(2021), business Zhou _et al._ (2021), and social sciences Charbey and Prieur
(2019); Ashford _et al._ (2022).

Graphlets are described as small induced subgraphs of a potentially huge
network that arise at any frequency. An induced subgraph indicates that after
selecting the nodes in the major network, all the edges connecting them must
be selected to form the subgraph. Within graphlets, symmetry groups of nodes,
known as automorphism orbits (or orbits for brevity), are utilized to
characterize the various topological locations that a node might occupy.
Orbits have also been utilized to generalize the concept of node degree Pržulj
(2007); Hočevar and Demšar (2014). If two nodes map to one another in an
isomorphic projection of the graphlet onto itself, they belong to the same
orbit. To completely decompose a graph of a given size into a certain graphlet
and orbit requires good combinatorial mathematics so that nothing is missed.
Existing approaches for counting graphlets and orbits rely on direct
enumeration; to count them, all network embeddings must be located. Good
algorithms are also required to help researchers decompose their networks,
number images and nodes, and record the number Hočevar and Demšar (2014);
Ahmed _et al._ (2017); Ribeiro _et al._ (2021).

Complex networks are composed of numerous nodes and edges, each containing a
large amount of information. Heterogeneity is an inherent trait of
heterogeneous graphs, that is, the nodes and edges of diverse types. For
instance, different types of nodes have distinct characteristics, and their
features may exist in distinct feature spaces. Nodes can have multiple states,
such as atoms, gender, and links, as well as multiple states, such as type,
direction, and sign. Such heterogeneous networks are pervasive in the natural
world, where many types of nodes and edges are detected. This variety of real-
world systems is frequently attributed to the fact that the data in
applications typically contain semantic information. Researchers have recently
used graphlets and orbits with multiple types of nodes or edges in their
analyses. For instance, by generalizing the graphlet-based node role concept
to directed graphlets, the authors of Sarajlić _et al._ (2016) conducted
sophisticated descriptive and predictive analyses of directed networked data
that were not possible with undirected graphlets or other directed network
statistics. In addition, by using edge information, Jia _et al._ (2022)
demonstrated that edge-type encoded graphlet techniques can outperform the
classic graphlet degree vector approach by a large margin in a node
classification challenge. Guan _et al._ (2017) introduced evolutionary
characteristics of weighted graphlets to model investors of various categories
as a single entity, taking their co-holding strength into account.

To the best of our knowledge, the current graphlet-related algorithms and
packages cannot easily handle networks with multiple state nodes and links
simultaneously in a systematic manner. Most graphlet packages and algorithms
deal with homogeneous graphs. More importantly, the number of possible
enumerations grows exponentially if the underlying complex networks are
heterogeneous. We hope to develop a suite that can help researchers determine
the number of graphlets and orbits they are looking for within a reasonable
amount of computation, given the numbers of points, node states, and link
states. Additionally, we will help researchers to decompose and number
graphlets and orbits within the network. There have been related studies on
graphlets on heterogeneous graphs. In Rossi _et al._ (2020); Carranza _et al._
(2020), the authors proposed a typed graphlet that generalizes graphlets to
heterogeneous graphs, and they outlined a system for counting the occurrences
of such typed graphlets and higher-order spectral clustering. However,
compared to their work, we focus on enumerating all graphs and orbits and
return a vector consisting of the number of occurrences of graphs and orbits
of a given graph.

In Section 2, the definitions of colored graphs and orbits are presented.
Colors are used to denote various states or types of nodes and edges. We will
examine how a colored graph can be used to depict its orbits. Given a
heterogeneous graph with $n$ vertices, $v_{c}$ types of vertices, and $e_{c}$
types of edges, we calculate the number of graphs and orbits it contains. See
Table 1 for the numbers of graphs and orbits for a given $(n,v_{c},e_{c})$.
For example, there are 900 graphs and 2,925 orbits of type at most
$(n,v_{c},e_{c})=(4,1,5)$, which means the graphs with 4 vertices, one type of
vertex, and at most 5 types of edges. In short, we assign the same graph a
fixed label, add new vertices and edges to the small graph, and finally
complete the classification. However, because the possible enumeration of
graphlets and orbits would soon become huge and our goal is to completely
classify the orbits of a colored directed graph, it is not realistic to
exhaust all possible enumerations. We focus only on heterogeneous graphs with
a reasonably small parameter set $(n,v_{c},e_{c})$. However, by converting the
problem of counting orbits into solving a linear system of equations, Hočevar
and Demšar (2016) showed that one only needs to count the frequency of some
special graphs but not all subgraphs. Hence, determining whether two orbits
are isomorphic is a graph isomorphic problem, for which there are several
effective algorithms based on locating the isomorphism or canonical form.
Because the orbit isomorphism problem is an exact graph isomorphism problem,
canonical labeling methods are not required. Therefore, the frequency of a
subgraph can be determined by computing and comparing the canonical form of
each subgraph. One can refer to Melckenbeeck _et al._ (2016, 2018); Ribeiro
_et al._ (2021); Sarajlić _et al._ (2016) for methods for solving linear
equations. Moreover, Kreher and Stinson (1999) provided a robust method for
solving canonical forms. We utilize these techniques to obtain the graphlets
and orbit information. In Section 3, we present four examples to demonstrate
the proposed package. We introduce three main functions that can be used to
distinguish graphlet orbits, generate a list of non-isomorphic heterogeneous
graphs and orbits, and return a vector consisting of the numbers of
occurrences of graphs and orbits of a given graph. Finally, in Section 4, we
present our concluding remarks.

##  2 Algorithm

In this section, we first define the orbits of heterogeneous graphs. Then we
describe our underlying orbit generation algorithm when (Di)Generator is
called to generate all orbits of order up to n, with number of vertex colors
up to sizev and number of edge colors up to sizee.

###  2.1 Orbit definition

A colored (di-)graph $M$ of order $n$ can be represented by an $n\times n$
matrix $A_{M}$ with entries from $\mathbb{N}\cup\\{0\\}$. The diagonal and
off-diagonal entries represent the colors of the vertices and edges,
respectively. Here, colors represent the types of nodes and edges in the
heterogeneous graphs. Thus, a colored graph is a complete graph $K_{n}$ with
colors for each vertex and edge. We say $M$ is of _type_ $(n,v_{c},e_{c})$ if
its order is $n$ and vertices and edges have $v_{c}$ and $e_{c}$ colors,
respectively.

###  2.2 Overview

We first generate all non-isomorphic graphs of order up to n with up to sizev
vertex colors and sizee edge colors, as described in Section 2.3. Then for
each of the generated graph $M$ we collect all its associated orbits as
described in Section 2.4. Finally we remove duplicate, i.e., isomorphic
orbits. We summarize the overall algorithm in the pseudocode below.

    GenerateOrbits(n, sizev, sizee) {
      Let Q be an empty set

      Generate Gs, the set of all non-isomorphic graphs

      For each G in Gs
        Insert into Q all orbits associated with G

      Remove isomorphic orbits from Q

      return Q
    }

###  2.3 Graph enumeration

To enumerate all graphs of order up to n with up to sizev vertex colors and
sizee edge colors, we first generate all graphs of order up to 3 with single
vertex color and up to 2 edge colors. Then based on that in each of the
following (sizee-2) iterations we enumerate all possible outcomes of adding
one edge color. Then based on that in each of the following n-3 iterations we
enumerate all possblie outcomes of adding one vertex. Finally we enumerate all
possible vertex colorings using up to sizev colors. We summarize in the
pseudocode below. In our implementation, we save as much partial results as
possible to files so that successive calls do not cause redundant computation.

    GenerateGraphs(n, sizev, sizee) {
      Let Gs = GenerateGraphs(3, 1, 2)

      For e = 3, 4, ..., sizee
        Let Gt be an empty set
        For each G in Gs
          Insert to Gt all possible result of adding one edge color to G

        Append Gt to Gs
        Deduplicate Gs

      For m = 4, 5, ..., n
        Let Gt be an empty set
        For each G in Gs
          Insert to Gt all possible results of adding one vertex to G

        Append Gt to Gs
        Deduplicate Gs

      Let Gt be an empty set
      For each G in Gs
        Insert to Gt all possible results of coloring vertices of G using up to sizev colors

        Deduplicate Gt

      return Gt
    }

###  2.4 Orbits enumeration from a graph

For $v\in V$, we denote by $v^{g}$ the action of $g$ in symmetric group
$S_{n}$ on $V$ (applying the function $g$ to $v$ and obtaining the value
$g(v)$). This action induces a new action on $\mathcal{G}_{n}$. In other
words, $M^{g}$ is the graph obtained by executing action $g$ on $V$. Two
graphs $M_{1}$ and $M_{2}$ are said to be isomorphic, denoted by $M_{1}\simeq
M_{2}$, if there exists a $g\in S_{n}$ such that $M_{1}^{g}=M_{2}$. The
automorphism group of graph $M$ is the set of actions $g$ in $S_{n}$ that
produce an identical graph when acting on $M$:

$Aut(M)=\left\\{g\in S_{n}:M^{g}=M\right\\}.$ (1)

The graphlet orbit of interest is the orbit of a vertex $v$ of
$M\in\mathcal{G}$ with respect to the automorphism group $Aut(M)$:

$\mathcal{O}_{v,M}:=\\{v^{g}:g\in Aut(M)\\}.$ (2)

Note that $\mathcal{O}_{w,M}=\mathcal{O}_{v,M}$ for all
$w\in\mathcal{O}_{v,M}$. We assign a labeled graph $O_{v}$ to
$\mathcal{O}_{v,M}$ such that the first vertex corresponds to reference point
$v$, i.e., $O_{v}=O_{w}$.

We list all connected orbits of type $(n,v_{c},e_{c})=(3,2,1)$ in Figure 1.

Figure 1: Connected orbits of type $(n,v_{c},e_{c})=(3,2,1)$; there are four host graphs and 10 orbits. $(v_{c},e_{c})$\$n$ | 3 | 4 | 5
---|---|---|---
(1,1) | (4,6) | (11,20) | (34,90)
(1,2) | (10,18) | (66,165) | (792,3132)
(1,3) | (20,40) | (276,816) | (10688,48400)
(1,4) | (35,75) | (900,2925) | (90005,428625)
(1,5) | (56,126) | (2451,8436) |
(2,1) | (20,40) | (90,240) | (544,1992)
(2,2) | (56,126) | (705,2280) | (19548,88452)
(2,3) | (120,288) | (3400,11968) | (306016,1468480)
(2,4) | (220,550) | (12025,44200) | (2725010,13350750)
(2,5) | (364,936) | (34410,129648) |
(3,1) | (56,126) | (357,1092) | (3258,13338)
(3,2) | (165,405) | (3132,10962) | (137268,645408)
(3,3) | (364,936) | (15900,58800) | (2249184,10964880)
(3,4) | (680,1800) | (57750,219450) |
(3,5) | (1140,3078) | (167805,647460) |
Table 1: Numbers of non-isomorphic graphs/orbits of type at most
$(n,v_{c},e_{c})$.

Isomorphism naturally results in canonical labels, an invariant among
isomorphic graphs. Our orbit generation problem can be generalized to a graph
plus an ordered partition. Let $\Pi$ denote the set of ordered partitions on
$V$. Then the action $g$ on $V$ induces a new action on $\Pi$ by
$\pi^{g}=(W_{1},\dots,W_{m})^{g}=(W^{g}_{1},\dots,W^{g}_{m})=(\\{v^{g}:v\in
W\\})_{W\in\pi}$ for $\pi=(W_{1},\dots,W_{m})\in\Pi$. Two pairs
$(M,\pi),(M^{\prime},\pi^{\prime})$ are _isomorphic_ if there is a $g\in
S_{n}$ such that $(M,\pi)^{g}=(M^{g},\pi^{g})$.
A _canonical label_ is a function

$C:\mathcal{G}\times\Pi\to\mathcal{G}\times\Pi$ (3)

such that, for all $M\in\mathcal{G},\pi\in\Pi$ and $g\in S_{n}$,

  1. (C1)

$C(M,\pi)\simeq(M,\pi)$

  2. (C2)

$C(M^{g},\pi^{g})=C(M,\pi)$.

Note that $(M,\pi)\simeq(M^{\prime},\pi^{\prime})\iff C(M,\pi)\simeq
C(M^{\prime},\pi^{\prime})$.

The equitable partitioning technique is crucial for McKay’s algorithm McKay
(1981). It can be extended to canonical labels of colored graphs. Let $v\in V$
and $\pi=(W_{1},\dots,W_{m})\in\Pi$. The _degree of $v$ in $W_{i}$_, denoted
by $\deg_{W_{i}}(v)$, is the sum of the entries at the intersection of the row
of adjacency matrix $A_{M}$ corresponding to $v$ and columns corresponding to
$W_{i}$. The ordered partition $\pi$ is _equitable_ if

$\deg_{W_{i}}(v)=\deg_{W_{i}}(w)\,\,\,\forall v,w\in W_{j}$ (4)

holds for all $i,j\in\\{1,\dots,m\\}$.

Based on equitable partition, McKay (1981) provided an algorithm to compute
the permutation required to arrange $V$ to obtain the canonical labels for
simple graphs. That is, $\pi^{M}_{lab}$ is a discrete order partition, i.e., a
permutation such that $(M,\pi)^{\pi^{M}_{lab}}$ is the canonical label of
$(M,\pi)$. Although their proof of the algorithm’s correctness is for simple
graphs, it also works for heterogeneous graphs.

Using this tool, the vertex set $\mathcal{O}_{v,M}$ can be represented by a
unique graph, with vertex $1$ separated from all other vertices in the ordered
partition $\pi$ of $V$

$O_{v}^{\pi^{M_{v}}_{lab}},$

where $O_{v}$ is the graph obtained from $M$ by swapping vertices $v$ and $1$,
and $\pi=(\\{1\\},V-1)$. More formally,

$\mathcal{O}_{v,M}=\\{u:O_{u}^{\pi^{O_{u}}_{lab}}=O_{v}^{\pi_{lab}^{O_{v}}}\\}.$

In other words, $O_{v}$ is the canonical form when the first vertex is
reference point $v$. Figure 2 shows an example for type
$(n,v_{c},e_{c})=(5,1,2)$, where there are four orbits relative to this graph.

Figure 2: Graph and orbits of a heterogeneous graph with type
$(n,v_{c},e_{c})=(5,1,2)$.

Therefore, we obtain four graphs $O_{a},O_{b},O_{c},O_{d}$ corresponding to
the four orbitals $O_{a,M},O_{b,M},O_{c,M},O_{d,M}$, respectively. Vertex $1$
in $O_{a}$ is vertex $a$ in $M$. The remaining vertices, say $e$, are in orbit
$b$. Therefore, the graph it produces is $O_{b}$; that is, $O_{e}=O_{b}$.

Using other vertices to assign the graphs gives the same result. Thus, the
first vertex is selected for convenience.

##  3 Proposed package

This package was written in Python 3 with the NumPy library. The three entry
functions are (Di)Canonical, (Di)Generator, and (Di)Count, which can be used
to distinguish graphlet orbits, generate a list of non-isomorphic
heterogeneous graphs and orbits, and return a vector consisting of the numbers
of occurrences of graphs and orbits of a given graph, respectively.

(Di)Canonical(G, ref = -1): This function computes the invariant of isomorphic
graphs or graphlet orbits, which can be used to distinguish non-isomorphic
graphs or graphlet orbits.

If ref $<1$, the function computes an invariant of graphs isomorphic to the
input graph G, e.g., $G_{1}$ in Figure 1. Otherwise, it computes an invariant
of the graphlet orbit associated with $\mathcal{O}_{ref,G}$ defined in Section
2, e.g., graphlet orbit $3$ in Figure 1.

Input: An adjacency matrix G representing a graph and, optionally, a vertex
ref $\geq 0$.

###  3.1 Output

The serialized adjacency matrix of the canonical label $C(G)$ of the input
graph G, or if vertex ref $\geq 0$, the serialized adjacency matrix of

$M_{ref}^{\pi_{lab}^{M_{ref}}}$

defined in Section 2. Implementation: If ref $<1$, then it returns the
lexicographically smallest serialized adjacency matrix of G for all $|V(G)|!$
permutations of its vertices. Otherwise, the algorithm swaps vertex 0 with
vertex ref and returns the lexicographically smallest serialized adjacency
matrix for all $|V(G)-1|!$ permutations of vertices $1,2,3,\ldots,|V(G)|-1$.
When comparing the serialized adjacency matrix for undirected graphs, we only
need to consider the entries on or above the diagonal because the adjacency
matrix is symmetric. With directed graphs, the entire adjacency matrix is
required.

(Di)Generator(n, sizev, sizee, orbg = False, connect = False, Time = True):

Built upon (Di)Canonical, this function generates a list of non-isomorphic
graphs of order n or graphlet orbits from graphs of order n, where each vertex
has a color in [sizev] and each edge has a color in [sizee].

Input:

  * n

: The order of graphs to be considered.

  * sizev

: Maximum number of vertex colors.

  * sizee

: Maximum number of edge colors.

  * orbg

: Whether to generate graphs (False) or graphlet orbits (True). Default is
False.

  * connect

: Whether to restrict to connected graphs (True) or not (False). Default is
False.

  * Time

: Whether to log the running time (True) or not (False). Default is True.

Output: The list of adjacency matrices representing the non-isomorphic graphs
or graphlet orbits.

Implementation: The function uses Dynamic Programming. It recursively calls
itself with a smaller number of vertices, vertex colors, and edge colors to
gradually obtain the intended list from the base cases, where graph order and
numbers of edge and vertex colors are small. See Section 2.3 and Section 2.4
for more details. It uses (Di)Canonical to determine whether two graphs or
graphlet orbits are isomorphic.

(Di)Count(G, ref, k, sizev, sizee, connect = False): Given a host graph G and
its vertex ref, this function returns a vector consisting of the number of
occurrences, at ref in G, of graphlet orbits belonging to graphs of order k,
where each vertex has a color in [sizev] and each edge has a color in [sizee].

Input:

  * G

: The host graph in which the occurrences of graphlet orbits are to be
counted.

  * ref

: A vertex in G at which the function counts the number of occurrences of the
specified graphlet orbits.

  * k

: The function counts in G the occurrences of all graphlet orbits belonging to
graphs of order k.

  * sizev

: Maximum number of vertex colors in graphs of order k in which the graphlet
orbits are considered.

  * sizee

: Maximum number of edge colors in graphs of order k in which the graphlet
orbits are considered.

  * connect

: Whether to restrict to connected graphs (True) or not (False). Default is
False.

Output: A vector containing the number of occurrences of all graphlet orbits
considered.

Implementation: The function first finds all graphlet orbits of interest via
(Di)Generator. The number of occurrences of these graphlet orbits is then
calculated.

###  3.2 Examples

To illustrate our package, we provide four examples containing both synthetic
and real-world data.

######  Example 3.1.

Let

$M=\left(\begin{array}[]{cccccc}0&1&1&0&0&0\\\ 1&0&1&0&0&0\\\ 1&1&0&1&1&0\\\
0&0&1&0&0&0\\\ 0&0&1&0&0&1\\\ 0&0&0&0&1&0\end{array}\right).$ (5)

Then, Canonical(M) will return the list [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
0, 1, 0, 1, 0, 1, 1, 1, 0]. The matrix can be recovered from the list using
lis2mat(L,6). Finally, we obtain a graph under canonical labeling. We also
treat $M$ as a digraph. Then, DiCanonical(M) will return the list [0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1,
0, 1, 1, 1, 0, 1].

Again, lis2mat(L,6) returns a graph under canonical labeling. Note that we get
different canonical labeling for graphs and digraphs; refer to Figure 3. To
compute the canonical form of orbit $O_{5,M}$, we enter Canonical(M,4).

Figure 3: Canonical labeling for graphs and digraphs

######  Example 3.2.

If we enter Generator(4,1,1), it returns a list of all graphs of type at most
$(n,v_{c},e_{c})=(4,1,1)$. Thus, we get [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0,
0, 0, 0, 0, 0, 0, 0, 1, 0], [0, 0, 0, 0, 0, 0, 0, 1, 1, 0], [0, 0, 0, 0, 0, 0,
1, 1, 1, 0]
,…]. Two parameters can be changed: orbg and connect. Thus, the command
Generator(4,
1,1,orbg=True,connect=True) will return the list of all connected orbits of
type at most $(n,v_{c},e_{c})=(4,1,1)$. Therefore, we obtain [0, 0, 0, 0, 0,
0, 1, 1, 1, 0], [0, 0, 0, 0, 1, 0, 1, 1, 1, 0], [0, 0, 0, 1, 0, 0, 0, 1, 1,
0], [0, 0, 0, 1, 0, 0, 1, 1, 1, 0],…].

######  Example 3.3 (Dolphin network).

The dolphin network was used to assess the frequency at which two dolphins
were observed together. The sexes of all but four dolphins were known.
Whenever a dolphin pod was spotted in a particular fjord between 1995 and
2001, each adult member of the pod was photographed and identified based on
the natural markings on its dorsal fin Lusseau (2003). The whole graph is
presented in 4. The network is a heterogeneous graph of type
$(n,v_{c},e_{c})=(62,3,1)$. The orbits that second dolphin appears in are
listed in Table 2.

Figure 4: Dolphin network: The structure of the whole dolphin network. Red
(blue, black) vertices indicate that the dolphins are female (male, unknown).

Let $M$ be the adjacency matrix of the dolphin network. If we enter Count(M,
1, 4, 3, 1, True), it will return the statistical data for the second dolphin
(Python starts from 0). This is a list of length 627. Figure 5 shows Orbits 0,
4, 6, 13, and 24.

Figure 5: Some orbits of the dolphin network. The second dolphin is in the upper left. frequency | Orbit
---|---
1 | 2, 7, 8, 18, 21, 23, 27, 29, 35, 41, 43, 49, 50, 52, 59, 66,
| 74, 75, 95, 99, 105, 118, 140, 142, 153, 154, 181, 186
2 | 0, 4, 6, 13, 24
3 | 3, 124
4 | 90, 93, 126, 136, 138
5 | 63, 107, 123, 135
6 | 38
7 | 121
9 | 33
10 | 131
12 | 19
14 | 37
15 | 93
18 | 122
20 | 116
Table 2: Numbers of occurrences

######  Example 3.4.

For illustrative purposes, we constructed an artificial heterogeneous graph.
The directed network in Figure 8 has 10 vertices, 3 types of nodes, and 2
types of directed edges. We now count the frequency of vertex 0 as an orbit of
type $(n,v_{c},e_{c})=(4,3,2)$. Therefore, we enter DiCount(M, 0, 4, 3, 2,
True). There are 7161912 connected orbits, and only 84 orbits occur all with
frequency 1. Figure 9 shows the first five orbits: 5145192, 5145203, 5145456,
5145776, and 5146300.

Figure 6: A heterogeneous network with directed edges. Figure 7: First five
orbits of directed graph.

######  Example 3.5 (New example).

$M=\left(\begin{array}[]{ccccccc}0&1&0&0&0&0&0\\\ 0&0&1&0&1&1&0\\\
0&0&0&1&0&0&0\\\ 1&0&0&0&0&0&1\\\ 0&0&0&0&0&0&0\\\ 0&0&0&0&0&0&0\\\
0&0&1&0&0&0&0\end{array}\right)$ (6)

The directed network in Figure 8 has 7 vertices, 1 types of nodes, and 1 types
of directed edges. The DiCanonical(M) gives [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1,
0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0]

We now count the frequency of vertex $2$ as an orbit of type
$(n,v_{c},e_{c})=(4,1,1)$. Therefore, we enter DiCount(M, 0, 4, 1, 1, True).

Figure 8: A heterogeneous network with directed edges. Figure 9: First five
orbits of directed graph.

##  4 Conclusion

Graphlets provide an efficient topological representation of how each node and
edge is integrated inside the mesoscale structure of a network and are widely
utilized in data science. In this study, we built a Python package to generate
a canonical form of graphlets and to count the occurrences of graphlet orbits
in heterogeneous graphs. By generalizing the techniques for distinguishing
graphlet orbits and generating a list of non-isomorphic graphs and orbits for
heterogeneous graphs, we made possible sophisticated descriptions of directed
and directed complex networked data that were not possible with single-type
graphlets. Once the order and type are calculated, the information can be
saved, and the program will directly use the file the next time. In the
package, we count the frequency of some special subgraphs, but not all
subgraphs, by counting orbits to solve a linear system of equations.
Therefore, our package can save considerable time for researchers analyzing
graphlets and orbits of heterogeneous graphs. However, the number of non-
isomorphic graphs/orbits increases exponentially with type and order. The size
of the file that stores all the canonical forms also increases significantly
as the number of parameters increases. For example, the file size for directed
graph type $(n,v_{c},e_{c})=(4,3,2)$ is approximately 0.8 GB with 1426032
orbits. Because of the essentially exponential-ordered complexity of the
enumerations of graphlets and orbits, our package is more suitable for
heterogeneous graphs with small order, less node type, and less edge type
parameters, which can be reused many times. A novel contribution of this study
is the use of a generating method to count the orbits of edges. Consequently,
the analytical methods and computing techniques used in this study can
potentially yield new discoveries in various heterogeneous graphs in various
fields. Our package is available at Colin Cleveland, Chin-Yen Lee, Shen-Fu
Tsai, Wei-Hsuan Yu and Hsuan-Wei Lee (2023) and users are encouraged to
contribute to it.

##  5 Acknowledgements

Shen-Fu Tsai is supported by the Ministry of Science and Technology of Taiwan
under grant MOST 111-2115-M-008-010-MY2.

## References

  * Ahmed _et al._ (2017) Ahmed NK, Neville J, Rossi RA, Duffield NG, Willke TL (2017).  “Graphlet decomposition: Framework, algorithms, and applications.”  _Knowledge and Information Systems_ , 50, 689–722.
  * Ashford _et al._ (2022) Ashford JR, Turner LD, Whitaker RM, Preece A, Felmlee D (2022).  “Understanding the characteristics of COVID-19 misinformation communities through graphlet analysis.”  _Online Social Networks and Media_ , 27, 100178.
  * Carranza _et al._ (2020) Carranza AG, Rossi RA, Rao A, Koh E (2020).  “Higher-order clustering in complex heterogeneous networks.”  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 25–35.
  * Charbey and Prieur (2019) Charbey R, Prieur C (2019).  “Stars, holes, or paths across your facebook friends: A graphlet-based characterization of many networks.”  _Network Science_ , 7(4), 476–497.
  * Colin Cleveland, Chin-Yen Lee, Shen-Fu Tsai, Wei-Hsuan Yu and Hsuan-Wei Lee (2023) Colin Cleveland, Chin-Yen Lee, Shen-Fu Tsai, Wei-Hsuan Yu and Hsuan-Wei Lee (2023).  _GEOMINE package_.  URL https://pypi.org/project/GEOMINE/.
  * Guan _et al._ (2017) Guan Q, An H, Liu N, An F, Jiang M (2017).  “Information connections among multiple investors: evolutionary local patterns revealed by motifs.”  _Scientific Reports_ , 7(1), 14034.
  * Hočevar and Demšar (2014) Hočevar T, Demšar J (2014).  “A combinatorial approach to graphlet counting.”  _Bioinformatics_ , 30(4), 559–565.
  * Hočevar and Demšar (2016) Hočevar T, Demšar J (2016).  “Computation of graphlet orbits for nodes and edges in sparse graphs.”  _Journal of Statistical Software_ , 71, 1–24.
  * Jia _et al._ (2022) Jia M, Van Alboom M, Goubert L, Bracke P, Gabrys B, Musial K (2022).  “Encoding edge type information in graphlets.”  _Plos one_ , 17(8), e0273609.
  * Kreher and Stinson (1999) Kreher DL, Stinson DR (1999).  “Combinatorial algorithms: generation, enumeration, and search.”  _ACM SIGACT News_ , 30(1), 33–35.
  * Lusseau (2003) Lusseau D (2003).  “The emergent properties of a dolphin social network.”  _Proceedings of the Royal Society of London. Series B: Biological Sciences_ , 270(suppl_2), S186–S188.
  * McKay (1981) McKay BD (1981).  “Practical graph isomorphism.”
  * Melckenbeeck _et al._ (2018) Melckenbeeck I, Audenaert P, Colle D, Pickavet M (2018).  “Efficiently counting all orbits of graphlets of any order in a graph using autogenerated equations.”  _Bioinformatics_ , 34(8), 1372–1380.
  * Melckenbeeck _et al._ (2016) Melckenbeeck I, Audenaert P, Michoel T, Colle D, Pickavet M (2016).  “An algorithm to automatically generate the combinatorial orbit counting equations.”  _PloS one_ , 11(1), e0147078.
  * Milenković and Pržulj (2008) Milenković T, Pržulj N (2008).  “Uncovering biological network function via graphlet degree signatures.”  _Cancer informatics_ , 6, CIN–S680.
  * Pržulj (2007) Pržulj N (2007).  “Biological network comparison using graphlet degree distribution.”  _Bioinformatics_ , 23(2), e177–e183.
  * Ribeiro _et al._ (2021) Ribeiro P, Paredes P, Silva ME, Aparicio D, Silva F (2021).  “A survey on subgraph counting: concepts, algorithms, and applications to network motifs and graphlets.”  _ACM Computing Surveys (CSUR)_ , 54(2), 1–36.
  * Rossi _et al._ (2020) Rossi RA, Ahmed NK, Carranza A, Arbour D, Rao A, Kim S, Koh E (2020).  “Heterogeneous graphlets.”  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ , 15(1), 1–43.
  * Sarajlić _et al._ (2016) Sarajlić A, Malod-Dognin N, Yaveroğlu ÖN, Pržulj N (2016).  “Graphlet-based characterization of directed networks.”  _Scientific reports_ , 6(1), 1–14.
  * Wang _et al._ (2021) Wang Z, Han Y, Li C, Geng Z, Fan J (2021).  “Input-output networks considering graphlet-based analysis for production optimization: Application in ethylene plants.”  _Journal of Cleaner Production_ , 278, 123955.
  * Zhou _et al._ (2021) Zhou J, Huang T, Li S, Hu R, Liu Y, Fu Y, Xiong H (2021).  “Competitive relationship prediction for points of interest: A neural graphlet based approach.”  _IEEE Transactions on Knowledge and Data Engineering_ , 34(12), 5681–5692.


# Concept-centric Personalization with Large-scale Diffusion Priors

Pu Cao1,2 Lu Yang1 Feng Zhou1 Tianrui Huang1 Qing Song1
1Beijing University of Posts and Telecommunications 2Metavatar
{caopu, soeaver, zhoufeng, htr<EMAIL_ADDRESS>corresponding author.

###### Abstract

Despite large-scale diffusion models being highly capable of generating
diverse open-world content, they still struggle to match the photorealism and
fidelity of concept-specific generators. In this work, we present the task of
customizing large-scale diffusion priors for specific concepts as concept-
centric personalization. Our goal is to generate high-quality concept-centric
images while maintaining the versatile controllability inherent to open-world
models, enabling applications in diverse tasks such as concept-centric
stylization and image translation. Distinct from existing personalization
tasks that focus on specific entities, the proposed task requires large-scale
training to achieve reasonable results. It brings forth unique challenges:
difficult fidelity and controllability balancing and severe unconditional
guidance drift in classifier-free guidance.

To tackle these challenges, we identify catastrophic forgetting of guidance
prediction from diffusion priors as the fundamental issue. Consequently, we
develop a guidance-decoupled personalization framework specifically designed
to address this task. We propose Generalized Classifier-free Guidance (GCFG)
as the foundational theory for our framework. This approach extends
Classifier-free Guidance (CFG) to accommodate an arbitrary number of
guidances, sourced from a variety of conditions and models. Employing GCFG
enables us to separate conditional guidance into two distinct components:
concept guidance for fidelity and control guidance for controllability. This
division makes it feasible to train a specialized model for concept guidance,
while ensuring both control and unconditional guidance remain intact. We then
present a null-text Concept-centric Diffusion Model as a concept-specific
generator to learn concept guidance without the need for text annotations.
Furthermore, we demonstrate the seamless integration of our framework with
existing techniques and underscore the vast potential of GCFG. Our extensive
experiments confirm the superior efficacy of our proposed method in achieving
concept-centric personalization. Code will be available at
https://github.com/PRIV-Creation/Concept-centric-Personalization.

Figure 1: Concept-centric personalization. In this work, we introduce a novel
diffusion-based task, termed concept-centric personalization, where diffusion
priors are customized to achieve faithful, diverse, and controllable
generation of a given concept.

##  1 Introduction

Thanks to billion-scale image-text datasets [2, 39, 33, 40] and the
development of large-scale natural language models [8, 32, 31], the research
on diffusion models in recent years has expanded the boundaries of artificial
intelligence generated content, significantly enhancing the diversity and
controllability of the generated content [35]. However, a critical problem
remains: these models fall short in achieving high levels of photorealism and
fidelity in specific concepts, lagging notably behind concept-specific
generators, as shown in gray block in Fig. 1. For instance, a face generator
(_e.g._ , StyleGAN3 [19]) trained on FFHQ [18] can produce lifelike facial
images, whereas large-scale diffusion models, such as Stable Diffusion [35],
struggle to achieve this.

In this work, we present the Concept-centric Personalization task, aiming to
develop a concept generator that not only rivals dedicated generators in
fidelity but also leverages the versatile open-world control capabilities
inherent in large-scale diffusion priors. This task is designed for a wide
range of applications, as shown in Fig. 1. Although recent research has
extensively explored personalization, the existing methods [12, 37, 30]
primarily concentrate on learning one or a few specific subjects instead of a
concept, aiming to produce varied subject-centric results (_e.g._ , generate
images of Geoffrey Hinton wearing glasses by several exampled images [25]). We
categorize these as subject-centric personalization111Since previous papers
did not differentiate between _subject_ and _concept_ , many papers utilize
_concept_ to denote a specific object. In this paper, we categorize all
existing methods as subject-centric, even if their papers use _concept_.. We
show the distinctions between these two tasks below.

Task | Subject-centric | Concept-centric
---|---|---
Objective | Consistency & Efficiency | Fidelity & Diversity
Controllability Preservation
Scale | Small, $<10^{2}$ images, | Large, $>10^{3}$ images,
minutes/hours of training | days of training

Our proposed task emphasizes fidelity and diversity in the generative results,
requiring a larger dataset and training process.

Hence, our task inevitably faces two significant yet unresolved challenges.
Firstly, achieving a balance between fidelity and controllability is
increasingly complex due to the broader scope and scale of our task. Secondly,
personalizing diffusion priors can disrupt their ability to model
unconditional guidance for the open world. Such disruption can lead to
inaccurate estimations in the sampling distribution under Classifier-free
Guidance (CFG) [15], resulting in a marked degradation of generation results.
Although some existing subject-centric personalization methods [37, 30] have
considered the first challenge, their solutions are not sufficient for our
task, as evidenced by the performance degradation marked in orange area of
Fig. 1. This highlights the urgent need for a novel perspective in our
proposed task.

Confronted with the aforementioned challenges, we propose a guidance-decoupled
personalization framework. We pinpoint the root cause of these challenges as
catastrophic forgetting of guidance in diffusion priors. Our key insight is
that it is effective by directly predicting guidance we wish to maintain
unchanged through diffusion priors, while independently learning concept
information via an additional model. Hence, we decouple conditional guidance
into two parts: concept guidance , which is optimized to enhance fidelity, and
control guidance, which is preserved for controllability. Our proposed
framework consists of Generalized Classifier-free Guidance (GCFG) as the
theoretical foundation and Concept-centric Diffusion Model to learn concept
guidance. GCFG is extended from classifier-free guidance to a more generalized
context, allowing for the integration of multiple guidances from different
models and conditions to guide the denoising process. In parallel, the
Concept-centric Diffusion Model, built upon diffusion priors, is adopted in a
null-text mode to avoid the need for text annotation. For concept-centric
generation, we utilize GCFG to fuse concept guidance, control guidance, and
unconditional guidance. Notably, control guidance can also be predicted with
other techniques like ControlNet. We also explore GCFG in diffusion model,
explaining the negative prompt and proposing a condition decoupling to
generate diverse images.

We conduct extensive experiments to validate our proposed approach and
demonstrate that our method can be integrated with existing methods and open-
sourced customized diffusion models. To summarize, our contributions are as
follows:

  * •

We introduce the concept-centric personalization task and identify its
challenges of fidelity-controllability tradeoff and unconditional guidance
drift.

  * •

We present a novel insight that utilizes an independent diffusion model to
learn given concept and introduce a guidance-decoupled personalization
framework. It consists of Generalized Classifier-free Guidance as a
theoretical foundation and null-text Concept-centric Diffusion Model for
concise.

  * •

We further explore GCFG in diffusion model to illustrate its great potential.
We employ it to explain and extend the negative prompt and introduce a new
technique, condition decoupling, which decouple the relation between contents
in prompt to generate diverse results.

Figure 2: Analysis of concept-centric personalization There are two
challenges: (a) tradeoff between fidelity and controllability, caused by
inconsistency between data-centric condition $Y$ in training and general
condition $C$ in inference; (b) unconditional guidance drift, which makes
inaccurate estimation of $p(C|z)$ and degrades generation ability.

##  2 Related Works

###  2.1 Text-to-image Generation

As the scale of image-text data grows [2, 39, 33, 40], large-scale diffusion
models [9, 29, 16, 35] have demonstrated astonishing generative capabilities,
able to produce images based on open-world prompts. Some models, such as
Imagen [38] and Stable Diffusion [35], have been explored as diffusion priors,
applied in downstream tasks like controlled generation [44, 28, 56, 22] and
image editing [14, 20, 7]. For instance, ControlNet [56] proposes using an
additional network to receive more spatial controls to guide generation.
Furthermore, diffusion models have demonstrated powerful capabilities in many
tasks, such as face parsing [49, 50, 51, 53, 52, 54], style transfer [23, 55,
46], image manipulation [10, 48] and super-resolution [47, 45].

###  2.2 Subject-centric Personalization

The task of subject-centric personalization aims to inject subjects into
large-scale diffusion priors and generate variant images guided by different
prompts [12, 37, 30, 5, 25, 26, 1, 42, 41]. Some methods fine-tune the part of
diffusion models with one or a few images of the specific subject [12, 37, 30,
5]. Textual inversion [12] tunes the text embedding of the subject
placeholder, while Dreambooth [37] further updates the weight of UNet [36].
Some methods further study training-free [1, 41], continual customization
[42], or multi-subjects personalization [25, 26] and layout-guided generation
[13]. However, these methods concentrate on consistency of given subjects and
customization efficiency, tailored for the situation with small-scale image
dataset and limited training process.

###  2.3 Fidelity and Controllability tradeoff

In the field of generation, the tradeoff between fidelity and controllability
has received widespread attention. In GANs, many inversion methods address
this problem through regularization [34, 3], latent code discriminator [43],
and invertibility prediction [parmar2022sam, 4]. In subject-centric
personalization, thanks to the simple customization process and excellent
generalization of diffusion priors, this problem is easily solved by
regularization [37], parameter efficient tuning [17], orthogonal fine-tuning
[30], and so on. However, a longer training process is required for concept-
centric personalization and the existing techniques struggle to solve this
problem, highlighting an urgent need for a new approach.

###  2.4 Guidance of Diffusion Models

In diffusion models, the control capability primarily stems from applying
guidance during the denoising process to increase the likelihood of the
condition. Classifier Guidance [9] introduced an auxiliary classifier, which
guides the sampling with $\nabla\log p_{\phi}(c|x_{t})$ to increase the
likelihood of condition. To attain the tradeoff between sample quality and
diversity without additional training, Classifier-free Guidance (CFG) [15]
combines the conditional and unconditional guidance from the diffusion model.
Some previous works [11, 24] interpret diffusion models as energy-based models
for concept composition. In this work, we consider a generalized situation
with multiple diffusion models, conditions, and intensities for better
concept-centric generation and further discuss the great potential in
diffusion models.

##  3 Concept-centric Personalization Task

###  3.1 Preliminaries

Text-to-Image Generation. Stable Diffusion forms Latent Diffusion Model [35]
(LDM), which introduced latent space for high-resolution image synthesis. For
given input image $x\in\mathbb{R}^{H\times W\times 3}$, an encoder of
variational auto-encoder (VAE) $\varepsilon$ converts it into a latent
representation $z_{0}\in\mathbb{R}^{h\times w\times c}$, where image is
downsampled by a factor $f$ and channels are increased to $c$. Then a text-
conditioned UNet $\epsilon$ in latent space is used to predict the noise of
each timestamp $t$ from $z_{t}$ and prompt $p$ to recover $z_{0}$. After the
denoising process, VAE’s decoder transforms $z_{0}$ to image space to generate
the image.

Classifier-free Guidance. To achieve a better tradeoff between quality and
diversity, CFG [15] is utilized to combine the conditional and unconditional
score estimates with a hyperparameter $w$:

$\displaystyle\widetilde{\epsilon}_{\theta}(z_{t},c)=(1+w)\epsilon_{\theta}(z_{t},c)-w\epsilon_{\theta}(z_{t})$
(1)

It is noteworthy that CFG treats condition $c$ as a whole and uses the same
model (_i.e._ , $\epsilon_{\theta}$) to handle both guidances, which has
become a paradigm in the current community.

Personalization. The personalization task aims to learn subject/concept from
given data. Denote the training data as conditional latent distribution
$p(z|Y)$ and $Y$ as a trigger, the personalization process can be formulated
as:

$\displaystyle\theta^{*}=\underset{\theta}{\mathrm{arg\,min}}\mathcal{L}(p_{\theta}(z|Y),p(z|Y))$
(2)

where $\theta$ is the parameters of trainable models (_e.g._ , UNet and text
embedding) and $\theta^{*}$ is the parameters of customized model. We then
utilize customized models to generate varied images of given subject/concept.

###  3.2 Task Analysis

In concept-centric personalization, a large-scale concept image dataset and
training process are required to model concept information. Two significant
challenges are concluded as follows:

Fidelity-Controllability Tradeoff. The inherent inconsistency between the
training condition $Y$ and the inference condition $C$ in personalization
often leads to the generator overfitting on given data, demonstrating
controllability degradation. In subject-centric personalization, a brief
training period (_e.g._ , less than one hour on a single GPU card) would not
affect controllability significantly, which is marked in purple in Fig. 2.
However, to attain a comparable model to a dedicated generator, an extensive
training process is necessary, _i.e._ , millions of images throughout, which
almost diminish controllability and is marked as Training-only in Fig. 2.
Hence, an important question is how to preserve controllability while
improving fidelity.

Unconditional Guidance Drift. The updating of parameters during the
personalization process inadvertently affects the unconditional guidance,
leading to an inaccurate estimation of the posterior probability $p(C|z)$ for
conditional generation. This issue is demonstrated in Fig. 2, where we display
unconditional generation results using the same random seed across several
open-sourced models. Despite all models being initialized by Stable Diffusion
v1.5, the outcomes exhibit significant variations and reflect the feature of
customization data.

##  4 Method

Figure 3: Motivation of our proposed guidance-decoupled personalization
framework. The blue and gray blocks symbolize various guidances, such as
$C^{prior}:=\epsilon_{\theta_{prior}}(z,C)$, and $C_{1},C_{2}$. The elements
$C_{1}$ and $C_{2}$ represent the concept guidance and control guidance,
respectively, which have been decoupled from the conditional guidance.
Previous methods predict all guidances by customized diffusion models, and our
framework enables employing an additional model, _i.e._ , Concept Diffusion,
to learn concept guidance, keeping the others unaffected. Concept D.: Concept
Diffusion, used to learn concept guidance in personalization.

###  4.1 Motivation

Our work is primarily motivated by addressing the foundational challenges in
concept-centric personalization: the catastrophic forgetting of unconditional
guidance and conditional guidance. In personalization, only the guidance
specific to concept generation, which we refer to as concept guidance, should
be updated. We define the remaining portion of conditional guidance as control
guidance, which is used for controlling in generation and should be preserved
during customization. However, previous methods, which employ customized
diffusion models to predict all types of guidance and are shown in Fig. 3(a),
often struggle with these issues, a gap our research seeks to bridge.

In this work, we present a novel insight: decoupling conditional guidance into
concept guidance and control guidance and utilizing an independent model to
predict the former with other guidance unchanged, as illustrated in Fig. 3(b).
Our approach begins with the introduction of Generalized Classifier-free
Guidance in Sec. 4.2, allowing integration of multiple guidances of diffusion
models. Subsequently, we present the Concept-centric Diffusion Model in Sec.
4.3, designed to streamline the training and inference processes without
requiring concept-centric data prompts. Finally, in Sec. 4.4, we demonstrate
the extensive potential of our method within the diffusion model context.

###  4.2 Generalized Classifier-free Guidance

To combine guidance from different diffusion models, we proposed the
Generalized Classifier-free Guidance (GCFG) by extending Classifier-free
Guidance [15]. We consider a general circumstance, using multiple models,
conditions, and intensities, for conditional image generation. Denoting a sets
of independent conditions as $C=\\{c_{1},\cdots,c_{k}\\}$ with manually
defined intensities $w=\\{w_{1},\cdots,w_{k}\\}$ and noised latent $z_{i}$, we
define posterior probability as $p(C|z_{t+1})=\prod p(c_{i}|z_{t+1})^{w_{i}}$.
Although conditions are not strictly independent, we can adjust their
intensities to approximately maximize the joint posterior probability in
practice.

Following [15], the reverse process aims to denoise latent according to the
gradient of the log-likelihood of probability $p(z_{t})p(C|z_{t})$ as follows:

$\displaystyle\nabla_{z_{t}}\log p(z_{t})+\nabla_{z_{t}}\log p(C|z_{t})$
$\displaystyle=$ $\displaystyle\nabla_{z_{t}}\log
p(z_{t})+\sum_{i=1}^{K}w_{i}\nabla_{z_{t}}\log p(c_{i}|z_{t})$
$\displaystyle=$ $\displaystyle\nabla_{z_{t}}\log
p(z_{t})+\sum_{i=1}^{K}w_{i}\nabla_{z_{t}}\log\frac{p(z_{t}|c_{i})}{p(z_{t})}$
$\displaystyle=$ $\displaystyle\nabla_{z_{t}}\log
p(z_{t})+\sum_{i=1}^{K}w_{i}(\nabla_{z_{t}}\log
p(z_{t}|c_{i})-\nabla_{z_{t}}\log{p(z_{t})})$

In Stable Diffusion, we use a UNet $\epsilon$ to predict each term:
$\epsilon(z_{t})=\nabla_{z}\log p(z_{t})$ and
$\epsilon(z_{t},c_{i})=\nabla_{z}\log p(z_{t}|c_{i})$. In GCFG, we can employ
different diffusion models with parameters
$\\{\theta_{0},\theta_{1},\cdots,\theta_{K}\\}$ for each condition. Hence, we
have:

$\displaystyle\hat{\epsilon}(z_{t},C)=\epsilon_{\theta_{0}}(z_{t})+\sum_{i=1}^{K}w_{i}(\epsilon_{\theta_{i}}(z_{t},c_{i})-\epsilon_{\theta_{0}}(z_{t}))$
(3)

where we use $\epsilon_{\theta_{0}}$ to predict the unconditional guidance and
use $\epsilon_{\theta_{i}}$ to predict the $i$th conditional guidance. We
illustrate the sampling process in Fig. 4(a) and Algorithm 1.

(a)

(b)

Figure 4: Details of Generalized Classifier-free Guidance. We present: (a) a
schematic diagram of GCFG, and (b) the inference process in concept-centric
generation.

Concept-centric Generation by GCFG. We can train a concept generator
$\epsilon_{concept}$ to provide concept guidance and employ the Stable
Diffusion $\epsilon_{sd}$ to provide unconditional guidance. For control
guidance, we can flexibly adapt existing techniques, including Stable
Diffusion, ControlNet, or any open-sourced customized models, as shown in Fig.
4(b). The reverse process suffices:

$\displaystyle\underbrace{(1-w_{1}-w_{2})\epsilon_{sd}(z_{t})}_{\hbox{\pagecolor[RGB]{191,191,191}\small
unconditional\
guidance}}+\underbrace{w_{1}\epsilon_{concept}(z_{t})}_{\hbox{\pagecolor[RGB]{254,242,204}\small
concept\
guidance}}+\underbrace{w_{2}\epsilon_{control}(z_{t},p)}_{\hbox{\pagecolor[RGB]{219,232,244}\small
control\ guidance}}$

where $p$ is control signal and $\epsilon_{control}$ represents any
controlling model or technique.

Algorithm 1 Generalized Classifier-free Guidance

1:$\\{\epsilon_{\theta_{i}}\\}_{i=0}^{K}$: UNets

2:$\\{c_{i},w_{i}\\}_{i=1}^{K}$: conditions and weights

3:$p_{\text{uncond}}\leftarrow\epsilon_{0}(z_{t})$

4:$p\leftarrow p_{\text{uncond}}$ $\triangleright$ unconditional guidance

5:for $i=1$ to $K$ do $\triangleright$ conditional guidance

6: $p_{i}\leftarrow\epsilon_{i}(z_{t},c_{i})$

7: $p\leftarrow p+w_{i}\times(p_{i}-p_{\text{uncond}})$

8:end for

9:return $p$

###  4.3 Concept-centric Diffusion Model

The concept-centric diffusion model aims to provide the concept guidance in a
concise way. Any generator that can provide concept guidance in diffusion
priors’ latent space can serve as a concept generator. To avoid costly text
annotations for image datasets or engaging in prompt engineering, we use the
UNet from Stable Diffusion v1.5 and convert it into a null-text mode.

Null-text Unconditional Generator. Unlike existing personalization methods,
our concept-centric diffusion model only predicts the concept guidance in the
latent space of diffusion prior without the need for controllability
preservation. Hence, we can convert the original UNet into the null-text mode.
In Stable Diffusion, cross-attention is used to fuse text feature and image
feature with residual connection, which follows:

$\displaystyle\mathcal{F}_{img}^{\prime}=\underbrace{W(Softmax(\frac{\mathcal{Q}_{img}\mathcal{K}^{T}_{text}}{d})\times\mathcal{V}_{text})}_{text\
feature\ injection}+\mathcal{F}_{img}$ (4)

where $\mathcal{Q}_{img}$ is transformed from input image feature
$\mathcal{F}_{img}$, $\mathcal{K}^{T}_{text},\mathcal{V}_{text}$ is
transformed from text feature, $d$ is the num of channels, and $W$ is a linear
projection.

We aim to discard text input and maintain the input and output distribution of
cross-attention for training stability and efficiency. Note that this operator
can be viewed as having two components: attention-based text feature injection
and original image feature. We change the former term to $\mathcal{F}_{null-
text}$:

$\displaystyle\mathcal{F}_{null-text}$
$\displaystyle=W(Norm(\mathcal{V}_{null-text}))$ (5)
$\displaystyle\mathcal{F}_{img}^{\prime}$ $\displaystyle=\mathcal{F}_{null-
text}+\mathcal{F}_{img}$ (6)

where $\mathcal{V}_{null-text}$ is predicted by text encoder and value
projection from null-text $\phi$. $Norm$ layer is used to normalize
$\mathcal{V}_{null-text}$ in token dimension, since we assume that attention
between image feature and null-text is unimportant and use a normalization
operator to maintain the feature distribution. $\mathcal{F}_{null-text}$ is
only related to $\mathcal{V}_{null-text}$ and can be fixed in training and
inference.

###  4.4 Exploration: GCFG in Diffusion Models

Figure 5: Example of condition decoupling. Decouple conditions by GCFG can
make contents independent. It helps mitigate the bias in diffusion priors,
leading to more diverse results.

Although we introduced GCFG to address concept-centric personalization, our
use of it is merely a special case. In this part, we further explore its
capabilities in diffusion models and hope that the features of GCFG can
further benefit diffusion generation. We first use GCFG to explain and expand
upon the negative prompt mechanism. Secondly, we propose a new technique,
condition decoupling, which employs GCFG to decompose related content within
prompts.

Negative Prompt. Negative prompt [11, 24] is an important application in text-
to-image synthesis, allowing the user to provide a prompt to be eliminated in
results. In practice, negative prompts replace unconditional guidance in CFG,
as follows:

$\displaystyle\hat{\epsilon}(z_{t},c_{pos},c_{neg})=(1-w)\epsilon(z_{t},c_{neg})+w\epsilon(z_{t},c_{pos})$
(7)

where $c_{pos}$ and $c_{neg}$ are positive and negative prompts. Following
Sec. 4.2, the denoising process is actually sampled to maximize the likelihood
of $\frac{p(c_{pos}|z_{t})}{p(c_{neg}|z_{t})}$. In GCFG, conditions with
positive scales can be regarded as positive prompts, while those with negative
scales are negative prompts. Eq. 7 is a special case when we use two
conditions $c_{pos}$ and $c_{neg}$ with guidance scales $w$ and $w+1$
respectively. Therefore, GCFG provides a theoretical guideline for more
diversified control over negative prompts. Based on it, negative prompts can
be more flexible and we illustrate the performance in Appendix C.

Condition Decoupling. GCFG also has the unique ability to decouple
relationships between contents that are typically related in the real world,
thereby enabling the generation of more diverse images. For instance, if the
goal is to generate images of a sunflower in the style of Van Gogh, as opposed
to replicating Van Gogh’s specific sunflower paintings, we can apply two
conditions: sunflower and in Van Gogh style. This distinction is showcased in
Fig. 5. Using the combined prompt sunflower in Van Gogh Style tends to produce
images closely resembling Van Gogh’s paintings. In contrast, employing
decoupled conditions results in a much broader diversity of images. This
method proves equally effective for generating concepts similar to those in
the training dataset, such as rose.

##  5 Experiments

Figure 6: Comparisons to previous methods. We evaluate concept-centric
generation in unconditional, text-guided, and spatial-guided cases. We fix the
random seed to illustrate the controllability for the former two.

(a)

(b)

Figure 7: Qualitative results of our proposed guidance-decoupled
personalization framework. We present the qualitative results on FFHQ and
AFHQv2. We also integrate our method with an open-sourced model to control
style. Images are $512\times 512$ resolution. Guidance is predicted in Stable
Diffusion v1.5,Concept-centric diffusion model, and Other open-sourced
diffusion model.

###  5.1 Settings

In this study, we evaluate methods on two concepts: face (_i.e._ , FFHQ [18])
and animal (_i.e._ , AFHQv2 [6]), chosen for their diverse image collections
and high-resolution (larger than $512\times 512$). All experiments are
conducted using images with a resolution of $512\times 512$. Concept-centric
Diffusion Models are initialed by Stable Diffusion v1.5 and all parameters of
UNet are trainable. To establish a comprehensive baseline, we employ several
state-of-the-art models: Textual Inversion [12], Dreambooth [37], Custom
Diffusion [21], and OFT [30], along with our method. Textual Inversion and
Custom Diffusion serve as parameter-efficient comparisons, streamlining the
experimental setup, while OFT is included for its regularization properties.
All models are trained diligently until loss convergence. To quantitatively
measure the fidelity and controllability of the generated images, we
incorporate the FID in unconditional generation, and alignment in conditional
generation. Moreover, we also undertake a user study to gather human
preferences, providing a qualitative dimension to our evaluation criteria and
ensuring a holistic view of the performance of the methods under scrutiny.

###  5.2 Quantitative Results

Method | #param. | Face | Animal
---|---|---|---
FID$\downarrow$ | Pref.$\uparrow$ | FID$\downarrow$ | Pref.$\uparrow$
Textual Inversion [12] |  $<1$K | 70.62 | 0.4% | 44.75 | 2.6%
DreamBooth [37] | 859M | 12.37 | 7.6% | 23.92 | 12.8%
Custom Diffusion [21] | 44M | 40.98 | 1.6% | 49.98 | 1.9%
OFT [30] | 23M | 27.50 | 3.0% | 38.74 | 1.5%
Ours | 859M | 6.57 | 89.4% | 18.82 | 81.2%
StyleGAN3-T | - | 3.16† | - | 4.04 | -

Table 1: Quantitative results of unconditional generation We evaluate FID and
human preference on two datasets. Our method achieves excellent generation
results among baselines.†: reproduced results in $512\times 512$ resolution.
Bold: the best result. Underline: the second best result.

Method | Text-Guided | Spatial-Guided
---|---|---
Align.$\uparrow$ | Pref.$\uparrow$ | Align.$\uparrow$ | Pref. $\uparrow$
Textual Inversion [12] | 0.97 | 0.2% | 0.22 | 1.5%
DreamBooth [37] | 0.24 | 4.5% | 0.22 | 9.5%
Custom Diffusion [21] | 0.74 | 1.4% | 0.20 | 1.7%
OFT [30] | 0.77 | 9.4% | 0.21 | 9.8%
Ours | 0.89 | 84.6% | 0.20 | 77.5%

Table 2: Controllability evaluation results. We evaluate alignment and human
preference of text-guided and spatial-guided (ControlNet) generation. Bold:
the best result. Underline: the second best result.

We conduct the evaluation of concept-centric unconditional generation and
report the results in Tab. 1. Our approach demonstrates state-of-the-art
generation quality, evidenced by a significantly lower FID compared to
baseline methods and the highest human preference rating. Additionally, the
complexity of learning to generate a concept, as opposed to a subject, is
underscored by the need for extensive parameterization to capture sufficient
information. This finding aligns with the analysis presented in Sec. 3.2,
highlighting challenges that are not readily addressed by subject-centric
approaches. It is observed that while our method narrows the gap, there
remains a discernible difference between our results and those of concept-
specific generators like StyleGan3. This is partly because our method also
prioritizes powerful controllability and does not involve fine-tuning the VAE,
in order to maintain generalization capabilities. On datasets such as AFHQv2,
this gap is more pronounced, particularly due to the VAE’s limitations in
robustly generating textures like fur, which is further illustrated in the
magnified Fig. 7(a).

In the domain of text-guided generation, Textual Inversion shows a higher
alignment with the text prompts than our method. However, this is achieved at
the cost of image quality, owing to its reliance on a very limited set of
learnable parameters. This compromise in quality is clearly evident in the
comparisons presented in Fig. 6. In contrast, in spatial-guided generation,
where semantic features are integrated into the UNet, the alignment degree
among different methods is similar. Notably, our results not only align well
with the control signals but also achieve the highest ratings in human
preference assessments.

###  5.3 Qualitative Results

We conducted a qualitative comparison of our method against previous subject-
centric personalization methods, as depicted in Fig. 6. Methods like textual
inversion and custom diffusion, which adhere to a parameter-efficient format,
exhibit noticeably lower image quality compared to others. Dreambooth, in
particular, demonstrates a decline in controllability during extended training
periods. This results in a diminished capacity for text-guided generation and
the emergence of unnatural artifacts, especially when used in conjunction with
ControlNet. In contrast, our method excels in both fidelity and control
capabilities.

Illustrated in Fig. 7 is a gallery of images generated by our method,
encompassing both unconditional and conditional scenarios. In the realm of
unconditional generation, our approach yields exceptional results, producing
high-resolution concept images that are virtually indistinguishable from
actual photographs. When it comes to conditional generation, our method
performs admirably in both text-guided and spatial-guided contexts. It also
demonstrates compatibility with open-sourced models, enabling precise control
over the generation outcomes.

###  5.4 Discussion and Ablation Studies

To further elucidate the challenges and the efficacy of our proposed solution,
we conduct in-depth ablation studies.

Effects of decoupled guidance on fidelity-controllability tradeoff.

(a)

(b)

Figure 8: Ablation studies. (a) the effects of decoupled guidance are
displayed by adjusting the scales of concept (vertical axis) and control
(horizontal axis) guidances. (b) The benefits of guidance rectification are
evident when comparing the generation results without GCFG (left column) to
those with GCFG (right column).

We demonstrate the distinct effects of concept guidance and control guidance
by using the control prompt a white and black dog with varying guidance scales
in Fig. 8(a). A higher scale of control guidance amplifies the prompt’s
influence, while a higher scale of concept guidance increases the fidelity.
This visual representation clearly shows how different guidance contributions
can be balanced, and how GCFG effectively combines them to achieve an optimal
tradeoff between fidelity and controllability. While certain scale
combinations may yield suboptimal results (e.g., $w_{1}=1.5$ and $w_{2}=5.0$),
in practice, it is feasible to find a suitable and stable scale range, like
$7.5$ used in Stable Diffusion, to guide the generation process effectively.

Effects of GCFG on unconditional guidance drift. In previous methods,
unconditional guidance is predicted by customized model, leading to inaccurate
estimation of probability in sampling. With GCFG, the unconditional guidance
is rectified by straightforwardly predicting by diffusion priors. We compare
generation results with and without GCFG in Fig. 8(b). Our method
significantly improves image quality, generating less artifacts and attains
photorealistic results.

##  6 Conclusion

In this research, we introduce the task of concept-centric personalization,
which strives to generate images that faithfully represent given concepts with
high fidelity and controllability. To address this task, we design a guidance-
decoupled personalization framework that separates conditional guidance into
concept guidance for fidelity and control guidance for maintaining
controllability. Our innovation begins with proposing Generalized Classifier-
free Guidance (GCFG) as the theoretical foundation for synthesizing images
utilizing multiple guidances and models. Then we introduce the null-text
Concept-centric Diffusion Model, learning concept guidance without text
requirement. This framework is designed to inherit the intrinsic
controllability and unconditional generation capabilities of diffusion priors
with fidelity increasing. Additionally, we explore GCFG in the context of
diffusion model, explaining negative prompt mechanism and proposing a new
technique, condition decoupling, to decouple the relation between contents in
text prompts.

## References

  * Arar et al. [2023] Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H Bermano.  Domain-agnostic tuning-encoder for fast personalization of text-to-image models.  _arXiv preprint arXiv:2307.06925_ , 2023.
  * Caesar et al. [2018] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1209–1218, 2018.
  * Cao et al. [2022] Pu Cao, Lu Yang, Dongxu Liu, Zhiwei Liu, Shan Li, and Qing Song.  Lsap: Rethinking inversion fidelity, perception and editability in gan latent space.  _arXiv preprint arXiv:2209.12746_ , 2022.
  * Cao et al. [2023] Pu Cao, Lu Yang, Dongxu Liu, Zhiwei Liu, Shan Li, and Qing Song.  What decreases editing capability? domain-specific hybrid refinement for improved gan inversion.  _arXiv preprint arXiv:2301.12141_ , 2023.
  * Chen et al. [2023] Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu.  Disenbooth: Disentangled parameter-efficient tuning for subject-driven text-to-image generation.  _arXiv preprint arXiv:2305.03374_ , 2023.
  * Choi et al. [2020] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Couairon et al. [2022] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance.  _arXiv preprint arXiv:2210.11427_ , 2022.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Dong et al. [2023] Wenkai Dong, Song Xue, Xiaoyue Duan, and Shumin Han.  Prompt tuning inversion for text-driven image editing using diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7430–7440, 2023.
  * Du et al. [2020] Yilun Du, Shuang Li, and Igor Mordatch.  Compositional visual generation with energy based models.  _Advances in Neural Information Processing Systems_ , 33:6637–6647, 2020.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Gu et al. [2023] Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al.  Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models.  _arXiv preprint arXiv:2305.18292_ , 2023.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4401–4410, 2019.
  * Karras et al. [2021] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  _Advances in Neural Information Processing Systems_ , 34:852–863, 2021.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6007–6017, 2023.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1931–1941, 2023.
  * Li et al. [2023] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22511–22521, 2023.
  * Li et al. [2020] Zuoxin Li, Fuqiang Zhou, Lu Yang, Xiaojie Li, and Juan Li.  Accelerate neural style transfer with super-resolution.  _Multimedia Tools and Applications_ , 79(7):4347–4364, 2020.
  * Liu et al. [2022] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum.  Compositional visual generation with composable diffusion models.  In _European Conference on Computer Vision_ , pages 423–439. Springer, 2022.
  * Liu et al. [2023a] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao.  Cones: Concept neurons in diffusion models for customized generation.  _arXiv preprint arXiv:2303.05125_ , 2023a.
  * Liu et al. [2023b] Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao.  Cones 2: Customizable image synthesis with multiple subjects.  _arXiv preprint arXiv:2305.19327_ , 2023b.
  * Lu et al. [2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _arXiv preprint arXiv:2211.01095_ , 2022.
  * Mou et al. [2023] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  _arXiv preprint arXiv:2302.08453_ , 2023.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Qiu et al. [2023] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf.  Controlling text-to-image diffusion by orthogonal finetuning.  _arXiv preprint arXiv:2306.07280_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Roich et al. [2022] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  _ACM Transactions on graphics (TOG)_ , 42(1):1–13, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Shi et al. [2023] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung.  Instantbooth: Personalized text-to-image generation without test-time finetuning.  _arXiv preprint arXiv:2304.03411_ , 2023.
  * Smith et al. [2023] James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin.  Continual diffusion: Continual customization of text-to-image diffusion with c-lora.  _arXiv preprint arXiv:2304.06027_ , 2023.
  * Tov et al. [2021] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  _ACM Transactions on Graphics (TOG)_ , 40(4):1–14, 2021.
  * Voynov et al. [2023] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or.  Sketch-guided text-to-image diffusion models.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–11, 2023.
  * Wang et al. [2023a] Yujin Wang, Lingen Li, Tianfan Xue, and Jinwei Gu.  Reconstruct-and-generate diffusion model for detail-preserving image denoising.  _arXiv preprint arXiv:2309.10714_ , 2023a.
  * Wang et al. [2023b] Zhizhong Wang, Lei Zhao, and Wei Xing.  Stylediffusion: Controllable disentangled style transfer via diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7677–7689, 2023b.
  * Wu et al. [2023] Chanyue Wu, Dong Wang, Yunpeng Bai, Hanyu Mao, Ying Li, and Qiang Shen.  Hsr-diff: hyperspectral image super-resolution via conditional diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7083–7093, 2023.
  * Wu and De la Torre [2023] Chen Henry Wu and Fernando De la Torre.  A latent space of stochastic diffusion models for zero-shot image editing and guidance.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7378–7387, 2023.
  * Yang et al. [2019] Lu Yang, Qing Song, Zhihui Wang, and Ming Jiang.  Parsing r-cnn for instance-level human analysis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 364–373, 2019.
  * Yang et al. [2020] Lu Yang, Qing Song, Zhihui Wang, Mengjie Hu, Chun Liu, Xueshi Xin, Wenhe Jia, and Songcen Xu.  Renovating parsing r-cnn for accurate multiple human parsing.  In _European Conference on Computer Vision_ , pages 421–437. Springer, 2020.
  * Yang et al. [2021] Lu Yang, Qing Song, Xueshi Xin, and Zhiwei Liu.  Quality-aware network for face parsing.  _arXiv preprint arXiv:2106.07368_ , 2021.
  * Yang et al. [2022a] Lu Yang, Zhiwei Liu, Tianfei Zhou, and Qing Song.  Part decomposition and refinement network for human parsing.  _IEEE/CAA Journal of Automatica Sinica_ , 9(6):1111–1114, 2022a.
  * Yang et al. [2022b] Lu Yang, Qing Song, Zhihui Wang, Zhiwei Liu, Songcen Xu, and Zhihao Li.  Quality-aware network for human parsing.  _IEEE Transactions on Multimedia_ , 2022b.
  * Yang et al. [2023a] Lu Yang, Wenhe Jia, Shan Li, and Qing Song.  Deep learning technique for human parsing: A survey and outlook.  _arXiv preprint arXiv:2301.00394_ , 2023a.
  * Yang et al. [2023b] Serin Yang, Hyunmin Hwang, and Jong Chul Ye.  Zero-shot contrastive loss for text-guided diffusion image style transfer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 22873–22882, 2023b.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models, 2023.
  * Zhou et al. [2022] Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, and Chen Change Loy.  Towards robust blind face restoration with codebook lookup transformer.  In _NeurIPS_ , 2022.

Supplementary Material

##  Appendix A Why We Need New Solution for Concept-centric Personalization
Task

As we have mentioned above, subject-centric methods are not suitable in our
proposed task, since they cannot address the challenges in Sec. 3.2. In this
section, we further analyze their performance and discuss why we need a new
solution for concept-centric personalization task.

In our study, we generated a scatter plot to compare the performance of
various methods in terms of controllability and fidelity, also considering
their trainable parameters. We utilized text-alignment as a metric for
controllability, plotted on the y-axis, and employed $-\log({\text{FID}/100})$
for fidelity, represented on the x-axis. Consistent with our task analysis
findings Sec. 3.2, we observed that existing methods typically exhibit a
tradeoff, where an increase in fidelity corresponds with a decrease in control
capabilities, making it challenging to achieve an optimal balance.

Furthermore, our analysis revealed a notable correlation between the number of
trainable parameters and the level of fidelity achieved. Specifically, methods
with fewer trainable parameters tend to maintain control capabilities to some
extent but fall short in achieving the desired level of fidelity. This
observation underlines our decision to exclude the LoRA [17] from our
experiments. In contrast to subject-centric personalization, our concept-
centric task necessitates extensive training involving substantial data,
numerous iterations, and a high count of trainable parameters.

Interestingly, while Dreambooth and our framework have comparable numbers of
trainable parameters, they exhibit distinct outcomes in terms of
controllability and fidelity. This difference can be attributed to our
adoption of a guidance-decoupled approach, which effectively preserves control
guidance and unconditional guidance from being overshadowed. As a result,
control guidance significantly bolsters our model’s control capabilities, and
the preservation of unconditional guidance contributes to the enhancement of
the generative results.

Figure 9: Results of controllability and fidelity on FFHQ. The diameter of
each circle in this plot represents the number of trainable parameters
associated with each method. Our proposed framework demonstrates a significant
advantage over existing subject-centric personalization approaches in terms of
fidelity. Additionally, it exhibits better controllability compared to nearly
all baseline methods.

##  Appendix B Unconditional Guidance Drift in Existing Customized Diffusion
Models

As shown in Fig. 2, the open-sourced customized diffusion models suffer from
severe unconditional guidance drift, where the features of the training data
unduly influence unconditional generation results. We previously analyzed the
nature of this discrepancy in Sec. 3.2. In this section, we extend our
examination to include the impact of this challenge on general generation
quality and demonstrate how GCFG can mitigate these effects.

We evaluate four widely-used open-sourced models: Counterfeit
222https://civitai.com/models/4468, BeautifulRealisticAsians
333https://civitai.com/models/25494, Anything
444https://civitai.com/models/66, and Dreamshaper
555https://civitai.com/models/4384. The comparative results are presented in
Fig. 10. Without GCFG, the generation quality shows slight degradation, as
evidenced by less detailed eyes and less realistic hair textures. The
implementation of GCFG substantially enhances generative performance through
the rectification of unconditional guidance.

In our experiments, we have also found that GCFG notably excels with not
intricately designed prompts and leads to fewer failure cases. The efficacy of
our method becomes particularly apparent when negative prompts are not
utilized. As we discuss in Sec. 4.4, the guidance provided by negative prompts
often entirely supplants the unconditional guidance. This observation is in
line with the prevalent community practice of employing complex and well-
crafted prompts, including negative prompts, to elevate the quality of
generated images.

Our findings indicate that GCFG allows simple prompt designing, significantly
improving the model’s capability to produce high-quality images, even without
the use of negative prompts. This flexibility greatly simplifies user
interaction, rendering the generation process more intuitive and less reliant
on intricate prompt engineering. As a result, GCFG not only enhances the
generative quality but also broadens accessibility, allowing users with
varying levels of expertise to engage with the model effectively.

Figure 10: Effects of GCFG for unconditional guidance rectification. The first
row is generated by the original inference process, which utilizes a
customized model to predict unconditional guidance. The sampling process of
second row is modified to use Stable Diffusion v1.5 to predict unconditional
guidance.

##  Appendix C Flexible Guidance Controlling with GCFG

In Sec. 4.4, we highlighted the potential for increased flexibility of the
negative prompt mechanism. We now present the capacity for flexible guidance
control offered by Generalized Classifier-free Guidance (GCFG). In current
implementations utilizing Classifier-free Guidance (CFG), the guidance derived
from negative prompts is generally configured to supplant unconditional
guidance. This configuration inherently constrains the scale of the negative
prompt to a fixed value of $1-w$. However, GCFG introduces a nuanced
understanding: the primary distinction between positive and negative prompts
lies in the scale of the guidance.

With this insight, GCFG enables us to establish a flexible mechanism for
controlling guidance in conditional generation, moving beyond the conventional
notion of negative prompts. Coupled with the concept of condition decoupling
discussed in Sec. 4.4, we propose a straightforward guideline for utilizing
this mechanism:

  * •

Employ any number of prompts as conditions, with each condition independently
influencing the generation process.

  * •

Assign a positive and large scale to those conditions that should be
emphasized in the generative results, while conditions intended to be
minimized or excluded are given a negative scale.

We demonstrate this approach with examples in Fig. 11. This mechanism is
applicable in both concept-centric and generic generation contexts. Compared
to traditional negative prompt techniques, GCFG’s continuous guidance scale
offers a more versatile and nuanced means of control.

Figure 11: Flexible Guidance Control with GCFG: This figure illustrates the
control techniques enabled by GCFG in both concept-centric and generic image
generation contexts. We maintain a constant scale for guidance1 while varying
the scale of guidance2. Notably, the middle images are generated with the
scale of guidance2 set to 0. This setup results in the left segment of the
images being guided by negative scales (negative prompts), and the right
segment by positive scales (positive prompts). By judiciously setting these
conditions and their respective scales, GCFG facilitates the creation of a
more generalized and adaptable control mechanism, enhancing the versatility of
the image generation process.

##  Appendix D Implementation Details

###  D.1 Datasets

We use two high-resolution (larger than $512\times 512$ resolution) datasets
in different domains to evaluate our method.

FFHQ. FFHQ is a high-quality human face dataset, containing 70,000 images.
Images come from Flickr and are aligned into $1024\times 1024$. We further
resize images into $512\times 512$ without crop.

AFHQv2. AFHQv2 is an animal face dataset with a resolution of $512\times 512$,
containing 15803 images. We use all images in the training and evaluation
process.

###  D.2 Training Details

Concept-centric Diffusion Models. Owing to the design of our concept-centric
diffusion model, which eliminates the need for text input, we utilize image-
only datasets for training, thereby obviating the requirement for text
annotations. Our training approach adheres to a standard UNet fine-tuning
protocol. This involves encoding images into a latent space using a VAE
encoder and subsequently predicting the noise pattern from these diffused
latent representations.

Common Settings. The experimental distinction between baselines and our
concept-centric diffusion models lies in the use of text prompts. We adopt
prompt templates inspired by Dreambooth [37], utilizing formats such as a
photo of <ffhq> face and a photo of <afhq> animal face. It’s important to note
that the additional tokens <ffhq> and <afhq> are omitted in cases where text
embedding is not designed to be trainable.

For all experiments, we maintain a constant learning rate of 1e-5 with the
optimizer of AdamW and a batch size of 32. These experiments are conducted on
8 NVIDIA A100 GPUs, with each running until convergence.

Inference. We employ the DPM-Solver++ scheduler [27] with 100 steps across all
models while using 25 steps for stylized results to achieve better results.
Regarding the guidance scale, we perform grid searches on all models and
conditions, with an interval of $0.25$. In text-guided generation, we combine
the prompts for baselines, like a photo of <ffhq> face, wearing glasses, and
only use control prompt for our method, _i.e._ , wearing glasses.

###  D.3 Evaluation

Unconditional Generation. Following previous methods, we evaluate FID on
50,000 images. For personalization-based methods, we use
torchmetrics666https://github.com/Lightning-AI/torchmetrics to calculate FID
with diffusers777https://github.com/huggingface/diffusers. For StyleGAN3
[19]888https://github.com/NVlabs/stylegan3, we use the official repository to
reproduce the result in the resolution of $512\times 512$.

Text-guided Generation. Although the measure of text-image similarity, often
quantified by CLIP scores, is a common metric for evaluating the alignment in
text-guided image generation, we observed its limitations in accurately
reflecting facial attributes. For instance, in the case of the prompt ‘wearing
glasses,’ the difference in CLIP scores between images with and without
glasses is relatively marginal, often around a value of 1 (e.g., 18 to 19).
Due to this lack of pronounced differentiation, we adopt an alternative
approach for evaluation.

We utilize a set of attribute-specific prompts, detailed in Tab. 3, to
generate images. Subsequently, we employ facial attribute predictors to
ascertain whether the specified attribute (e.g., glasses) is accurately
represented in the generated images. The effectiveness of our method is then
quantified by the ratio of successful samples where the controlled attribute
is correctly manifested in the results.

wearing glasses | wearing sunglasses
---|---
wearing hat | smilling
male | female
white people | black people
asian people | square face

Table 3: Prompts used for text-guided generation evaluation. We generate 100
images of each prompt for evaluation.

Spatial-guided Generation. To assess the effectiveness of spatial-guided
generation, we select a sample of 200 images from the CelebA-HQ dataset. For
each image, we generate corresponding canny images to use as conditions. We
then produce 5 generated results per condition and compute the discrepancy
between the canny conditions and the corresponding generative results,
providing a quantitative measure of the alignment accuracy.

User Study. To gauge human preferences in both unconditional and conditional
generation contexts, we organize a user study. We gain generative results from
all methods using the same random seed across all methods and present these
images to participants in a random sequence. Participants are instructed to
choose the best image from the given set. We record the frequency with which
each method is selected as the best, and this data is used to calculate the
win rate for each method. The win rates serve as an indicator of human
preference (denoted as ’Pref.’) and are presented in Tab. 1 and Tab. 2.

##  Appendix E More Qualitative Results

###  E.1 Unconditional Concept-centric Generation

  * •

Fig. 12 illustrates the comparison of unconditional generative results on
FFHQ.

  * •

Fig. 13 illustrates the comparison of unconditional generative results on
AFHQv2.

Figure 12: Unconditional generation results on FFHQ. We illustrate the
unconditional results of all models trained from our method and baselines.
Figure 13: Unconditional generation results on AFHQv2. We illustrate the
unconditional results of all models trained from our method and baselines.

###  E.2 Conditional Generation

  * •

Fig. 14 illustrates the comparison of text-guided generative results on FFHQ.

  * •

Fig. 15 illustrates the generation results under different ethnic conditions.

  * •

Fig. 16 illustrates the generation results under multiple conditions.

Figure 14: Text-guided generation comparison across baselines and our method.
Since Dreambooth with large-scale training process almost loses
controllability (as shown in Fig. 6), we evaluate other methods in this part.
Figure 15: Facial generation under different ethnic conditions. We use the
proposed method to generate facial images from different ethnic conditions. We
fix the random seeds for each column. Figure 16: Generative results under
multiple conditions.

##  Appendix F Limitations

Figure 17: Enhancing proposed frameworks with face swapping and enhancement.
This process effectively demonstrates the potential of combining our
generation framework with advanced image processing techniques to overcome
limitations in generalizability, particularly in out-of-distribution scenarios
such as full-body image generation.

Although the guidance-decoupled framework can learn a new concept with high
fidelity and controllability, there are still some limitations. Firstly,
similar to dedicated generators, our method requires a longer training time
(_i.e._ , several days) on the concept dataset. Our future work plans to
explore how to leverage diffusion priors to reduce training duration.
Secondly, due to the concentrated distribution of concept-centric data, even
when using diffusion priors for control, the generalizability to out-of-
distribution scenes remains limited. For example, in the domain of human
faces, our customized generation achieves excellent results in facial image
generation, but struggles in generating full-body images. However, we can
leverage other technologies (such as face swapping and super-resolution, etc.)
to extend our customized generation framework to a broader application space.
We demonstrate this using roop999https://github.com/s0md3v/roop and codeformer
[57], which is shown in Fig. 17. Body images are generated by open-sourced
diffusion models, which are borrowed from https://civitai.com/.