# Venue Analytics: A Simple Alternative to Citation-Based Metrics

Leonid Keselman  Carnegie Mellon University5000 Forbes AvePittsburghPA15213
<EMAIL_ADDRESS>
###### Abstract.

We present a method for automatically organizing and evaluating the quality of
different publishing venues in Computer Science. Since this method only
requires paper publication data as its input, we can demonstrate our method on
a large portion of the DBLP dataset, spanning 50 years, with millions of
authors and thousands of publishing venues. By formulating venue authorship as
a regression problem and targeting metrics of interest, we obtain venue scores
for every conference and journal in our dataset. The obtained scores can also
provide a per-year model of conference quality, showing how fields develop and
change over time. Additionally, these venue scores can be used to evaluate
individual academic authors and academic institutions. We show that using
venue scores to evaluate both authors and institutions produces quantitative
measures that are comparable to approaches using citations or peer assessment.
In contrast to many other existing evaluation metrics, our use of large-scale,
openly available data enables this approach to be repeatable and transparent.

To help others build upon this work, all of our code and data is available at
https://github.com/leonidk/venue_scores.

††copyright: none

Figure 1. Results from our method, demonstrating differences and changes in
conference quality over time in various subfields. These graphs includes size
and year normalization and are the result of combining multiple different
metrics of interest.

Correlation Results

##  1\. Introduction

There exist many tools to evaluate professional academic scholarship. For
example Elseiver’s Scorpus provides many author-level and journal-level
metrics to measure the impact of scholars and their work (Colledge et al.,
2010; da Silva and Memon, 2017). Other publishers, such as the Public Library
of Science, provide article-level metrics for their published work (Fenner,
2013). Large technology companies, such as Google and Microsoft provide their
own publicly available metrics for scholarship (Butler, 2011). Even
independent research institutes, such as the Allen Institute’s Semantic
Scholar (Ammar et al., 2018), manage their own corpus and metrics for
scholarly productivity. However, these author-based metrics (often derived
from citation measurements) can be inconsistent, even across these large,
established providers (da Silva and Dobránszki, 2018).

In this work, we propose a method for evaluating a comprehensive collection of
published academic work by using an external evaluation metric. By taking a
large collection of papers and using only information about their publication
venue, who wrote them, and when, we provide a largely automated way of
discovering not only venue’s value. Further, we also develop a system for
automatic organization of venues. This is motivated by the desire for an open,
reproducible, and objective metric that is not subject to some of the
challenges inherent to citation-based methods (da Silva and Dobránszki, 2018;
Bornmann and Daniel, 2008; Galiani and Gálvez, 2017).

We accomplish this by setting up a linear regression from a publication record
to some metric of interest. We demonstrate three valid regression targets:
status as a faculty member (a classification task), awarded grant amounts, and
salaries. By using DBLP (Ley, 2002) as our source of publication data, NSF
grants as our source of awards, University of California data for salaries,
and CSRankings (Berger, 2018) for faculty affiliation status, we’re able to
formulate these as large data regression tasks, with design matrix dimensions
on the order of a million in each dimension. However, since these matrices are
sparse, regression weights can be obtained efficiently on a single laptop
computer. Details of our method are explained in section 4.

We call our results venue scores and validate their performance in the tasks
of evaluating conferences, evaluating professors, and ranking universities. We
show that our venue scores correlate highly with other influence metrics, such
as h-index (Hirsch, 2005), citations or highly-influential citations
(Valenzuela et al., 2015). Additionally, we show that university rankings,
derived from publication records correlate highly with both established
rankings (News and Report, 2018; Education, 2018; Rankings, 2018) and with
recently published quantitative metrics (Berger, 2018; Blackburn et al., 2018;
Vucetic et al., 2018; Clauset et al., 2015).

##  2\. Related Work

Quantitative measures of academic productivity tend to focus on methods
derived from citation counts. By using citation count as the primary method of
scoring a paper, one can decouple an individual article from the authors who
wrote it and the venue it was published in. Then, robust citation count
statistics, such as h-index (Hirsch, 2005), can be used as a method of scoring
either individual authors, or a specific venue. Specific critiques of h-index
scores arose almost as soon as the h-index was published, ranging from a
claimed lack-of-utility (Lehmann et al., 2006), to a loss of discriminatory
power (Tol, 2008).

Citations can also be automatically analyzed for whether or not they’re highly
influential to the citing paper, producing a ”influential citations” metric
used by Semantic Scholar (Valenzuela et al., 2015). Even further, techniques
from graph and network analysis can be used to understand systematic
relationships in the citation graph (Bergstrom, 2007). Citation-based metrics
can even be used to provide a ranking of different universities (Blackburn et
al., 2018).

Citations-based metrics, despite their wide deployment in the scientometrics
community, have several problems. For one, citation behavior varies widely by
fields (Bornmann and Daniel, 2008). Additionally, citations often exhibits
non-trivial temporal behavior (Galiani and Gálvez, 2017), which also varies
greatly by sub-field. These issues highly affect one’s ability to compare
across disciplines and produce different scores at different times. Recent
work suggests that citation-based metrics struggle to effectively capture a
venue’s quality with a single number (Walters, 2017). Comparing citation
counts with statistical significance requires an order-of-magnitude difference
in the citation counts (Kurtz and Henneken, 2017), which limits their utility
in making fine-grained choices. Despite these quality issues, recent work
(Vucetic et al., 2018) has demonstrated that citation-based metrics can be
used to build a university ranking that correlates highly with peer
assessment; we show that our method provides a similar quality of correlation.

Our use of straightforward publication data (Section 3) enables a much simpler
model. This simplicity is key, as the challenges in maintaining good citation
data have resulted in the major sources of h-index scores being inconsistent
with one another (da Silva and Dobránszki, 2018).

While there exist many forms of ranking journals such as Eigenfactor
(Bergstrom, 2007) or SJR (Falagas et al., 2008), these tend to focus on
journal-level metrics while our work focuses on all venues, including
conferences.

###  2.1. Venue Metrics

We are not the first to propose that scholars and institutions can be ranked
by assigning scores to published papers(Ren and Taylor, 2007). However, in
prior work the list of venues is often manually curated and assigned equal
credit, a trend that is true for studies in 1990s (Geist et al., 1996) and
their modern online versions (Berger, 2018). Instead, we propose a method for
obtaining automatic scores for each venue, and in doing so, requires no manual
curation of valid venues.

Previous work (Yan and Lee, 2007) has developed methods for ranking venues
automatically, generating unique scores based only on author data by labeling
and propagating notions of ”good” papers and authoritative authors. However,
this work required a manually curated seed of what good work is. It was only
demonstrated to work on a small sub-field of conferences, as new publication
cliques would require new labeling of ”good” papers. Recent developments in
network-based techniques for ranking venues have included citation information
(Zhang and Wu, 2018), and are able to produce temporal models of quality. In
contrast, our proposed model doesn’t require citation data to produce sensible
venue scores.

Our work, in some ways, is most similar to that of CSRankings (Berger, 2018).
CSRankings maintains a highly curated set of top-tier venues; venues selected
for inclusion are given 1 point per paper, while excluded venues are given 0
points. Additionally, university rankings produced by CSRankings include a
manually curated set of categories, and rankings are produced via a geometric
mean over these categories. In comparison, in this work, we produce unique
scores for every venue and simply sum together scores for evaluating authors
and institutions.

In one of the formulations of our method, we use authors status as faculty (or
not faculty) to generate our venue scores. We are not alone in this line of
analysis, as recent work has demonstrated that faculty hiring information can
be used to generate university prestige rankings (Clauset et al., 2015).

Many existing approaches either focus only on journals (Falagas et al., 2008),
or do not have their rankings available online. For our dataset, we do not
have citation-level data available, so we are unable to compare against
certain existing methods on our dataset. However, as these methods often
deploy a variant of PageRank (Page et al., 1999), we describe a PageRank
baseline in Section 6.1 and report its results.

##  3\. Data

Our primary data is the dblp computer science bibliography (Ley, 2002). DBLP
contains millions of articles, with millions of authors across thousands of
venues in Computer Science. We produced the results in this paper by using the
dblp-2019-01-01 snapshot. We restricted ourselves to only consider conference
and journal publications, skipping books preprints, articles below 6 pages and
over 100 pages. We also merged dblp entries corresponding to the same
conference. This lead to a dataset featuring 2,965,464 papers, written by
1,766,675 authors, across 11,255 uniquely named venues and 50 years of
publications (1970 through 2019).

Our first metric of interest is an individual’s status as a faculty member at
a university. For this, we used the faculty affiliation data from CSRankings
(Berger, 2018), which are manually curated and contain hundreds of
universities across the world and about 15,000 professors. For evaluation
against other university rankings, we used the ScholarRank (Vucetic et al.,
2018) data to obtain faculty affiliation, which contains a more complete
survey of American universities (including more than 50 not currently included
in CS Rankings). While CSRankings data is curated to have correct DBLP names
for faculty, the ScholarRank data does not. To obtain a valid affiliation, the
names were automatically aligned with fuzzy string matching, resulting in
about 4,000 faculty with good seemingly unique DBLP names and correct
university affiliation. Although those two methods are manually curated,
automatic surveys of faculty affiliations have recently been demonstrated
(Morgan et al., 2018b).

Our second metric of interest was National Science Foundation grants, where we
used awards from 1970 until 2018. This data is available directly from the NSF
(Foundation, 2018). We adjusted award amounts using annual CPI inflation data.
We restricted ourselves to awards that had finite amount, where we could match
at least half the Principal Investigators on the grant to DBLP names and the
grant was above $20,000. Award amounts over 10 million dollars were clipped in
a smooth way to avoid matching to a few extreme outliers. This resulted in
407,012 NSF grants used in building our model.

Our third and final metric of interest was University of California salary
data (Institute, 2018). This was inspired by a paper that predicted ACM/IEEE
Fellowships for 87 professors and used salary data (Nocka et al., 2014). We
looked at professors across the entire University of California system,
matching their names to DBLP entries in an automated way. We used the maximum
salary amount for a given individual across the 2015, 2016 and 2017 datasets,
skipping individuals making less than $120,000$ or over $800,000$ dollars.
This resulted in 2,436 individuals, down from 3,102 names that we matched and
an initial set of about 20,000 initial professors. As DBLP contains some
Chemistry, Biology, and Economics venues, we expect that some of these are
likely not Computer Science professors.

Table 1. Spearman’s $\rho$ correlation between rankings produced by targeting different metrics of interest. | Faculty | NSF | Salary
---|---|---|---
Faculty | 1.00 | 0.91 | 0.84
NSF | 0.91 | 1.00 | 0.86
Salary | 0.84 | 0.86 | 1.00

##  4\. Method

Our basic model is that a paper in a given publication venue (either a
conference or a journal), having passed peer review, is worth a certain amount
of value. Certain venues are more prestigious, impactful or selective, and
thus should have higher scores. Other venues have less strict standards, or
perhaps provide less opportunity to disseminate their ideas (Morgan et al.,
2018a) and should be worth less. While this model explicitly ignores the
differences in paper quality at a publication venue, discarding this
information enables the use of a large quantity of data to develop a
statistical scoring system.

This methodology is not valid for all fields of science, nor all models of how
impactful ideas are developed and disseminated. Our method requires individual
authors have multiple publications across many different venues, which is more
true in Computer Science than the natural sciences or humanities, where
publishing rates are lower (Lubienski et al., 2018). If instead we assume that
all research ideas produce only a single paper, or that passing peer-review is
a noisy measurement of quality (Shah et al., 2018), then our proposed method
would not work very well. Instead, the underlying process which supports our
methodology is that good research ideas produce multiple research publications
in selective venues; better ideas would produce more individual publications
in higher quality venues. The concept of All models are wrong, but some are
useful is our guide here. This assumption allows us to obtain venue scores in
an automatic way, and then use these scores to evaluate both authors and
institutions.

Our approach to ranking venues is to construct a regression task, apply an
optimization method to solve it, and use the resulting weights. Optimization’s
ability to generate powerful representations is well-studied in machine
learning (Rumelhart et al., 1986) and statistics (Huber, 1985). The resulting
weights are often useful (Coates and Ng, 2011), and this methodology is widely
used in natural language processing (Mikolov et al., 2013; Pennington et al.,
2014).

###  4.1. Formal Setup

In general, we will obtain a score for each venue by setting up a linear
regression task in the form of equation 1.

(1)
$\bordermatrix{&\text{conf}_{1}&\text{conf}_{2}&\ldots&\text{conf}_{n}&\cr\text{auth}_{1}&1&3&\ldots&0&1\cr\text{auth}_{2}&1&0&\ldots&1&1\cr\vdots&\vdots&\vdots&\ddots&\vdots&\vdots\cr\text{auth}_{m}&0&2&\ldots&4&1}\begin{pmatrix}x_{0}\\\
x_{1}\\\ \vdots\\\ x_{n}\end{pmatrix}=\begin{pmatrix}\text{isProf}_{1}\\\
\text{isProf}_{2}\\\ \vdots\\\ \text{isProf}_{m}\end{pmatrix}$

Authors are listed along the rows, and venues are listed along the columns.
The number of publications that an author has in a conference is noted in the
design matrix. There is an additional column of 1s to learn a bias offset in
the regression.

Different forms of counting author credit are discussed in section 4.6, while
different regression targets are discussed in section 3. In equation 1, the
regression target is shown as a binary variable indicating whether or not that
author is currently a professor. If this linear system, $Ax=b$ is solved, then
the vector $x$ will contain real-valued scores for every single publishing
venue. Since our system is over-determined, there is generally no exact
solution.

Instead of solving this sparse linear system directly, we instead solve a
regularized regression, using a robust loss and $L_{2}$ regularization. That
is, we iteratively minimize the following expression via stochastic gradient
descent (Robbins and Monro, 1951)

(2) $L(Ax,b)+\lambda||x||^{2}$

The $L_{2}$ regularization enforces a Gaussian prior on the learned conference
scores. We can perform this minimization in Python using common machine
learning software packages (Pedregosa et al., 2011). We tend to use a robust
loss function, such as the Huber loss in the case of regression (Huber, 1964),
which is quadratic for errors of less than $\delta$ and linear for errors of
larger than $\delta$. It can be written as

(3) $L(\hat{y},y)=\begin{cases}\frac{1}{2}(y-\hat{y})^{2},&\text{if
}|y-\hat{y}|\leq\delta\\\
\delta|y-\hat{y}|-\frac{1}{2}\delta^{2},&\text{otherwise}\end{cases}$

In the case of classification, we have labels $y\in\\{-1,1\\}$ and use the
modified Huber loss (Zhang, 2004),

(4) $L(\hat{y},y)=\begin{cases}\max(0,1-y\hat{y})^{2},&\text{if
}y\hat{y}\geq-1\\\ -4y\hat{y},&\text{otherwise}\end{cases}$

We experimented with other loss functions, such as the logistic loss, and
while they tended to produce similar rankings and results, we found that the
modified Huber loss provided better empirical performance in our test metrics,
even though the resulting curves looked very similar upon qualitative
inspection.

###  4.2. Metrics of Interest

As detailed in section 3, we targeted three metrics of interest: status as a
faculty member, NSF award sizes, and professor salaries. Each of these metrics
came from a completely independent data source, and we found that they each
had their own biases and strengths (more in section 5).

For faculty status classification, we used the modified huber loss and
CSRankings (Berger, 2018) faculty affiliations. To build venue scores that
reward top-tier conferences more highly, we only gave professors in the
top-$k$ ranked universities positive labels. We tried $k=5,16,40,80$, and
found that we got qualitatively different results with quantitative similar
performance. Unless otherwise stated, we used $k=40$. The university ranking
used to select top-$k$ was CSRankings itself, and included international
universities, covering the Americas, Europe and Asia. This classification is
performed across all authors, leading to 1.7 million rows in our design
matrix.

For the NSF awards, every Principal Investigator on the award had their papers
up to the award year as the input features. We used a Huber loss,
$\lambda=0.03$, and experimented with normalizing our award data to have zero
mean and unit variance. Additionally, we built models for both raw award sizes
and log of award sizes; the raw award sizes seem to follow a power-law while
the log award sizes seem distributed as approximately a Gaussian. Another
model choice is whether to regress each NSF grant as an independent
measurement or instead a marginal measurement which tracks the cumulative
total of NSF grants received by the authors. If not all authors on the grant
were matched to DBLP names, we only used the fraction of the award
corresponding to the fraction of identified authors. This regression had
$\sim\frac{1}{2}$ million rows in its design matrix .

For the salary data, we found that normalizing the salary data to have zero
mean and unit variance led to a very poor regression result, while having no
normalization produced a good result. This regression only had $\sim 2,400$
datapoints, and thus provided information about fewer venues than the other
metrics of interest.

###  4.3. Modeling Change Over Time

In modeling conference values, we wanted to build a model that could adjust
for different values in different years. For example, a venue may be
considered excellent in the 1980s, but may have declined in influence and
prestige since then. To account for this behavior, we break our dataset into
chunks of $n$ years and create a different regression variable for each
conference for each chunk. The non-temporal model is obtained simply by
setting $n\geq 50$.

We also examine a model that creates an independent variable for each year,
for each conference. After setting $n=1$ in the block model, we splat each
publication as a Gaussian at a given year. By modifying $\sigma$, we can
control the smoothness of the obtained weights. The Gaussian is applied via a
sparse matrix multiply of the design matrix $A$ with the appropriate band
diagonal sparse matrix $G$. The use of a truncated Gaussian (where $p<0.05$ is
clipped and the Gaussian is re-normalized) enables our matrix to maintain
sparsity. We used a $\sigma=4.5$, which produced an effective window size of
about 10 years. This can be seen visually in Figure 2.

Our different temporal models are compared in Table 5 by correlating against
existing author-level, journal-level and university-level metrics. For
evaluation details see Section 6.

###  4.4. Normalizing Differences Across Years

The temporal models described in the previous section have an inherent bias.
Due to the temporal window of the DBLP publication history, there is variation
in distributed value due to changes annual NSF funding, the survivorship of
current academic faculty, etc. To adjust for this bias, we scale each
conference-year value by the standard deviation of conference values in that
year. This scaling can help or hurt performance, depending on which metric of
interest the model is built against. It generally produces flatter value
scores over time but leads to some artifacts. The effects of this
normalization are shown in Figure 3 and Table 2. In our final version, we
instead normalize by the average of the top 10 conferences in each year, which
produced flatter results over time,

Figure 2. Truncated Gaussian ($\sigma=4.5$) used to splat a publication’s
value across multiple years. Examples centered at the year 2000 and the year
2018

Truncated Gaussian

Figure 3. Results showing the effect of performing a normalization for venue
year and size. See Sections 4.4 and 4.5.

###  4.5. Normalizing Differences In Venue Size

Our model uses $L_{2}$ regularization on the venue scores, which tends to
squash the value of variables with less explanatory power.This process often
resulted in the under-valuing of smaller, more selective venues. To correct
for this size bias, instead of giving each paper $1$ point of value in the
design matrix, we give each paper $\frac{1}{M}$ credit, where $M$ is the
number of papers at that venue in that year; this step is performed before
Gaussian splatting. This produced rankings which emphasized small venues and
provided a good notion of efficient venues. In practice, we instead used a
blended result with a multiplier of $\frac{1}{\sqrt[\alpha]{M}}$ with
$\alpha=1.5849$, the Hausdorff dimension of the Sierpinski triangle, an
arbitrary constant.

Table 2. Spearman correlation between our model and existing metrics, showing the effect of different normalization schemes. See Sections 4.4 and 4.5 for model details. See Section 6 for evaluation details.  Normalization |  influential citations (author) |  h-index (author) |  h-index (university) |  h-index (venue)
---|---|---|---|---
None |  0.72 |  0.61 |  0.60 |  0.26
Year |  0.72 |  0.66 |  0.58 |  0.18
Size |  0.74 |  0.63 |  0.61 |  0.25
Year + Size |  0.72 |  0.61 |  0.60 |  0.26

###  4.6. Modeling Author Position

Another question to consider is how credit for a paper is divided up amongst
the authors. We consider four models of authorship credit assignment:

  1. (1)

Authors get $\frac{1}{n}$ credit for each paper, where $n$ is the number of
authors on the paper. Used by (Berger, 2018).

  2. (2)

All authors get full credit (1 point) for each paper

  3. (3)

Authors receive less credit for later positions
($\frac{1}{1},\frac{1}{2},\frac{1}{3}\cdots\frac{1}{n}$), normalized so total
credit sums to 1). This model assigns more credit to earlier authors and is
used by certain practitioners (Sekercioglu, 2008; Huang, 2018).

  4. (4)

The same as (3), except the last author is explicitly assigned equal credit
with the first author before normalization.

Using Spearman correlation with Semantic Scholar’s ”highly influential
citations”, an evaluation metric described in depth in Sec. 6.4, we can
evaluate each of these models. Specifically, there are two places where venues
scores require a selection of authorship model. The first is how much credit
is assigned to each paper when performing regression (in the case of our
faculty metric of interest). The second is when evaluating authors with the
obtained regression vector. See table 3 for a summary of experimental results.

Table 3. Correlation (Spearman’s $\rho$) between our model and Semantic Scholar (Valenzuela et al., 2015), showing the properties of different authorship models. For details see section 4.6. |  | Evaluation Author Model
---|---|---
|  | 1 | 2 | 3 | 4
Regression Author Model | 1 | 0.70 | 0.72 | 0.65 | 0.70
2 | 0.68 | 0.71 | 0.61 | 0.67
3 | 0.71 | 0.73 | 0.66 | 0.71
4 | 0.70 | 0.72 | 0.65 | 0.71

For the purposes of evaluation, assigning full credit to authors (model 2)
produced the best results, while model 3 consistently produced the lowest
quality correlations. On the other hand, for the purposes of performing the
classification task, the roles are flipped. Assigning full credit (model 2)
consistently produces the worst quality correlations while using model 3
produces the highest quality correlations.

###  4.7. Combining Models

Since the proposed metrics of interest (faculty status, NSF awards, salaries)
were generated from different independent regression targets, with different
sized design matrices, there may be value in combining them to produce a joint
model. The value of ensemble models is well documented in both theory (Freund
and Schapire, 1997) and practice (Bell et al., 2009). In the absence of a
preferred metric with which to cross-validate our model, we simply perform an
unweighted average of our models to obtain a gold model. To ensure that the
weights are of similar scale, the conference scores are normalized to have
zero mean and unit variance before combining them. Venue scores that are too
large or too small are clipped at 12 standard deviations. For the temporal
models, this normalization is performed on a per-year basis. While table 4
shows results for a simple combination, one could average together many models
with different choices of hyperparameters, regression functions, datasets,
filters to scrub the data, etc.

Table 4. Correlation between our model and traditional measures of scholarly output on the dataset of CMU faculty. For model details see Section 4.7. For evaluation details see Section 6.4. Model | citations | h-index (Hirsch, 2005) | influential citations (Valenzuela et al., 2015)
---|---|---|---
Faculty | 0.59 | 0.68 | 0.71
NSF | 0.63 | 0.66 | 0.67
Salary | 0.36 | 0.36 | 0.41
Combined | 0.69 | 0.77 | 0.75

##  5\. Results

A visual example of some of venue scores is shown in Figure 1. We kept the
y-axis fixed across all the different Computer Science sub-disciplines to show
how venue scores can be used to compare different fields in a unified metric.
There are additional results in our qualitative demonstration of normalization
methods, Figure 3.

Due to the variation in rankings produced by one’s choice of hyperparameters,
and the large set of venues being evaluated, we do not have a canonical set of
rankings that can be presented succinctly here. Instead, we will focus on
quantitative evaluations of our results in the following section.

Table 5. Correlation between rankings produced by our model against rankings produced by traditional scholarly metrics. Different rows correspond to different hyperparameters choices for our model. Each column is a corresponds to a traditional metric. AI = Author Highly Influential Citations, AH = Author H-index, USN = US News 2018, VH = Venue H-index, VC = Venue Citations.  Years | Metric |  AI |  AH |  USN |  VH |  VC
---|---|---|---|---|---|---
$\sigma=4.5$ | Faculty |  0.73 |  0.69 |  0.74 |  0.63 |  0.42
10 |  |  0.67 |  0.57 |  0.76 |  0.57 |  0.35
50 |  |  0.75 |  0.68 |  0.76 |  0.38 |  0.21
$\sigma=4.5$ | NSF |  0.64 |  0.62 |  0.62 |  0.61 |  0.59
10 |  |  0.68 |  0.68 |  0.60 |  0.59 |  0.60
50 |  |  0.67 |  0.65 |  0.63 |  0.64 |  0.67
$\sigma=4.5$ | Salary |  0.62 |  0.58 |  0.59 |  0.48 |  0.55
10 |  |  0.65 |  0.62 |  0.57 |  0.45 |  0.55
50 |  |  0.66 |  0.63 |  0.56 |  0.43 |  0.63

##  6\. Evaluation

To validate the venue scores obtained by our regression methods, our
evaluation consists of correlating our results against existing rankings and
metrics. We consider three classes of existing scholarly measurements to
correlate against: those evaluating universities, authors, and venues. Each of
these classes has different standard techniques, and a different evaluation
dataset, so they will be described separately in Sections 6.2, 6.4, and 6.3.

In the case of our proposed method, venue scores, we have a simple way to turn
them from a journal-based to an author-based or institution-based metric.
Venues are evaluated directly with the scores. Authors are evaluated as the
dot product of venue scores and the publication vector of an author.
Universities are evaluated as the dot product of venue scores and the total
publication vector of all faculty affiliated with that university.

###  6.1. PageRank Baseline

Many existing approaches build on the idea of eigenvalue centrality
(Bergstrom, 2007; Yan and Lee, 2007; Zhang and Wu, 2018). We implemented
PageRank (Page et al., 1999) using the power iteration method to compute a
centrality measure to use for both author-level and venue-level metrics.
Unlike most versions of PageRank, which use citation counts, we implement two
variants based solely on co-authorship information.

Author-level PageRank (PageRankA) is computed on the 1.7M x 1.7M sized co-
authorship graph, where an edge is added for every time two authors co-author
a paper. We found that the authors with highest centrality measures are often
common names with insufficient disambiguation information in DBLP.

Journal-level PageRank (PageRankC) is computed on the 11,000 x 11,000 co-
authorship graph, where an edge is added for every author who publishes in
both venues. When ran on the unfiltered DBLP data, the highest scoring venue
was arXiv, an expected result.

###  6.2. University Ranks

For this work, we produce university rankings simply as an evaluation method
to demonstrate the quality and utility of our venue scoring system. The reader
is cautioned that university ranking systems can tend to produce undesirable
gaming behavior (Johnes, 2018), and are prone to manipulation.

We obtained and aligned many existing university rankings for Computer Science
departments. These include rankings curated by journalistic sources, such as
the US News Rankings (News and Report, 2018), the QS Rankings (Rankings,
2018), Shanghai Ranking (Rankings, 2015), Times Higher Education Rankings
(Education, 2018) and the National Research Council report (Clauset et al.,
2015). In addition, we consider purely quantitative evaluation systems such as
ScholarRank (Vucetic et al., 2018), CSRankings (Berger, 2018), CSMetrics
(Blackburn et al., 2018), and Prestige Rankings (Clauset et al., 2015). We
additionally include ScholarRank’s t10sum across the matched faculty that our
venue scores result uses.

We follow a recent paper (Vucetic et al., 2018), which demonstrated the
efficacy of a citation-based metric in producing rankings with large
correlation against US News rankings. We extend these experiments to include
more baselines. In contrast with (Vucetic et al., 2018), we use a rank
correlation metric (namely Kendall’s $\tau$), which naturally handles ordinal
ranking systems. While ScholarRank (Vucetic et al., 2018) claimed a
correlation of $>0.9$ with US News, this was under Pearson’s correlation
coefficient, and the result under Kendall’s $\tau$ is 0.768 in the published
version and 0.757 using full precision ScholarRank.

Our faculty-based regression is able to generate a result with the highest
correlation against the US News rankings. We perform even better than
ScholarRank, which was designed to optimize this metric (although under a non-
rank correlation metric).

Table 6. Section 6.2. Kendall’s $\tau$ correlation across different University Rankings. Ranking |  Correlation with US News 2018
---|---
USN2018 (News and Report, 2018) |  1.000
USN2010 (Clauset et al., 2015) |  0.928
Venue Scores |  0.780
ScholarRank (Vucetic et al., 2018) |  0.768
ScholarRankFull |  0.757
CSMetrics (Blackburn et al., 2018) |  0.746
CSRankings (Berger, 2018) |  0.724
Times (Education, 2018) |  0.721
NRC95 (Clauset et al., 2015) |  0.713
t10Sum (Vucetic et al., 2018) |  0.713
Prestige (Clauset et al., 2015) |  0.666
Citations (Vucetic et al., 2018) |  0.665
Shanghai (Rankings, 2015) |  0.586
# of papers |  0.585
BestPaper (Huang, 2018) |  0.559
PageRankA |  0.535
PageRankC |  0.532
QS (Rankings, 2018) |  0.518
Table 7.  Spearman’s $\rho$ correlation between different journal-level metrics (N=1008). For details see Section 6.3. | papers | citations | h-index |  PageRank C |  venue scores
---|---|---|---|---|---
papers |  | 0.75 | 0.41 |  0.91 |  0.25
citations | 0.75 |  | 0.60 |  0.69 |  0.27
h-index | 0.41 | 0.60 |  |  0.37 |  0.65
PageRankC | 0.91 | 0.69 | 0.37 |  |  0.27
venue scores | 0.25 | 0.27 | 0.65 |  0.27 |

###  6.3. Journal-level metrics

To evaluate the fidelity of our venue scores for journals and conferences, we
obtain the h-index (Hirsch, 2005) and citation count for 1,308 conferences
from Microsoft Academic Graph (Schauerte, 2014). We continue to use Spearman’s
$\rho$ as our correlation metric, even though rank-correlation metrics can be
highly impacted by noisy data (Abdullah, 1990).

Under this metric, venue scores correlated highly with h-index. Notably,
h-index and venue scores are highly correlated with each other, even though
venue scores are much less correlated with raw conference size. See Figure 7
for detailed results.

###  6.4. Author-level Metrics

To evaluate our venue scores in the application of generating author-level
metrics, we will use rank correlation (also known as Spearman’s $\rho$)
(Spearman, 1904) between our venue scores and traditional author-level metrics
such as h-index. Google Scholar was used to obtain citations, h-index (Hirsch,
2005), i10-index, and Semantic Scholar used to obtain highly influential
citations (Valenzuela et al., 2015). Prior work has critiqued the h-index
measure (Yong, 2014) and proposed an alternative metric, derived from a
citation count. However, our use of a rank correlation means that
monotonically transformed approximations of citation counts would lead to
identical scores.

For evaluation, we collected a dataset for the largest Computer Science
department in CSRankings ($n=148$). The results are shown in Table 8. We can
see that venue scores highly correlate with h-index, influential citations,
i10-scores and CSRankings scores. The results from the author-based PageRank
are surprisingly similar to our venue scores. However, the conference-based
PageRank performed worse than venue scores on every correlation metric.

Table 8. Correlation between different author-level metrics for a dataset of professors (N=148). Details are in Section 6.4.  | papers | citations | h-index | i10 | CSR (Berger, 2018) | venue score | PageRankA | PageRankC | influence (Valenzuela et al., 2015)
---|---|---|---|---|---|---|---|---|---
papers |  | 0.66 | 0.79 | 0.81 | 0.71 | 0.94 | 0.94 | 0.89 | 0.76
citations | 0.66 |  | 0.93 | 0.88 | 0.49 | 0.66 | 0.68 | 0.60 | 0.81
h-index | 0.79 | 0.93 |  | 0.97 | 0.56 | 0.75 | 0.81 | 0.68 | 0.80
i10 | 0.81 | 0.88 | 0.97 |  | 0.53 | 0.75 | 0.82 | 0.69 | 0.73
CSR (Berger, 2018) | 0.71 | 0.49 | 0.56 | 0.53 |  | 0.84 | 0.64 | 0.80 | 0.64
venue score | 0.94 | 0.66 | 0.75 | 0.75 | 0.84 |  | 0.86 | 0.92 | 0.78
PageRankA | 0.94 | 0.68 | 0.81 | 0.82 | 0.64 | 0.86 |  | 0.83 | 0.72
PageRankC | 0.89 | 0.60 | 0.68 | 0.69 | 0.80 | 0.92 | 0.83 |  | 0.67
influence (Valenzuela et al., 2015) | 0.76 | 0.81 | 0.80 | 0.73 | 0.64 | 0.78 | 0.72 | 0.67 |
Figure 4. The career arcs of several accomplished Computer Scientists. The
first row uses a simple model where all papers are all given equal weight;
first using raw counts and then normalizing by the number of papers published
each year. The second row shows our model.

##  7\. Discussion

Our results show a medium to strong correlation of venue scores against
existing scholarly metrics, such as citation count and h-index. For author
metrics, venue scores correlate with influential citations (Valenzuela et al.,
2015) or h-index about as well as such measures correlate against each other
or raw citation counts (see Table 8). For venue metrics, venue scores
correlate with h-index (0.64) and citations (0.61) nearly as well as citations
correlate with h-index (0.66). For university metrics, venue scores correlate
as well with measures of peer assessment as citation-based metrics do (Vucetic
et al., 2018).

As h-index and citation counts have their flaws, obtaining perfect correlation
is not necessarily a desirable goal. Instead, these strong correlations serve
as evidence for the viability of venue scores.

Venue scores have been shown to be robust against hyperparameter choices
(Tables 2, 3, 4, 5). Even venue scores produced from completely different data
sources tend to look very similar (Table 1). Additionally, venue scores can
naturally capture the variation of conference quality over time (Figures 1,
3).

As with any inductive method, venue scores are data-driven and will be subject
to past biases. For example, venue scores can clearly be biased by hiring
practices, pay inequality and NSF funding priorities. As these are the
supervising metrics, bias in those datasets will be encoded in our results.
For example, we found that the faculty hiring metric prioritized Theoretical
Computer Science, while using NSF awards prioritized Robotics. The faculty
classification task may devalue publishing areas where candidates pursue
industry jobs, while the NSF grant regression task may devalue areas with
smaller capital requirements. By using large datasets and combining multiple
metrics in a single model (Section 4.7), the final model could reduce the
biases in any individual dataset.

Each of our metrics of interest has an inherent bias in timescale, which our
temporal normalization tries to correct for, but likely does an incomplete job
of. Salaries are often higher for senior faculty. NSF Awards can have a long
response time and a preferences towards established researchers. Faculty
classification prioritizes the the productive years of existing faculty.
Additionally, faculty hiring as a metric will have a bias towards work from
prestigious universities (Clauset et al., 2015) and their venue preferences.
Some of these issues also exist in citation metrics, and may be why our
uncorrected models correlated better with them (Table 2).

Figure 5. Automatic clustering for venues in Computer Science, the largest
venues in each cluster are labeled.

Automatic clustering for venues in Computer Science

Figure 6. Heatmap showing the differences in research focus across different
universities. Figure 5 can be used as a guide.

Different university’s fingerprints

Figure 7. An embedding of Carnegie Mellon University’s School of Computer
Science with colors indicating sub-departments. For example, the Robotics
Institute (RI) has clear clusters for Robotics, Graphics, CV and HRI.

##  8\. Similarity Metrics

While the previous sections of this paper have focused on evaluation, the same
dataset can be used to organize venues into groups. For organization, we use a
much smaller dataset, using data since 2005 and only evaluating the 1,155
venues that have at least 20 R1 universities with faculty publishing in them.
We then build the venue $\times$ author matrix, counting the number of papers
each that author published in each venue. Performing a $d$-dimensional Latent
Dirichlet Allocation (Blei et al., 2003), we obtain a 50-dimensional vector
representing each conference in a meaningful way.

These vectors can then be clustered (Arthur and Vassilvitskii, 2007) to
produce automatic categories for each conference. These high dimensional
vectors can also be embedded (van der Maaten and Hinton, 2008) into two
dimensions to produce a visual map of Computer Science. See Figure 5 for our
result. These clusters represent natural categories in Computer Science. For
example, it is easy to see groups that could be called Theory, Artificial
Intelligence, Machine Learning, Graphics, Vision, Parallel Computing, Software
Engineering, Human-Computer Interaction, among many others.

While some clusters are distinct and repeatable, others are not. When datasets
contain challenging cases, ideal clustering can be hard to estimate (Ben-
David, 2015). Using silhouette scores (Rousseeuw, 1987), we can estimate how
many natural clusters of publishing exist in Computer Science. In our
experiments, silhouette scores were maximized with 20 to 28 clusters. As the
clustering process is stochastic, we were unable to determine the optimal
cluster number with statistical significance.

By embedding each author with the weighted average of their publication’s
vectors, we can also obtain a fingerprint that shows which areas of Computer
Science each university focuses on. See Figure 6 for an example of such
fingerprints for many top departments. The same clustering method can be used
to analyze the focus areas of a single department, for an example see Figure
7.

##  9\. Conclusion

We have presented a method for ranking and organizing a scholarly field based
only on simple publication information- namely a list of papers, each labeled
with only their published venue, authors, and year. By regressing venue scores
from metrics of interest, one obtains a plausible set of venue scores. These
scores can be compared across sub-fields and aggregated into author-level and
institution-level metrics. The scores provided by this system, and their
resulting rankings, correlate highly with other established metrics. As this
system is based on easily obtainable, publicly available data, it is
transparent and reproducible. Our method builds on simple techniques and
demonstrates that their application to large-scale data can produce
surprisingly robust and useful tools for scientometric analysis.

###### Acknowledgements.

Martial Hebert suggested the use of correlation metrics as a technique for
quantitative evaluation, thereby providing the framework for every table of
results in this paper. Emery Berger developed CSRankings (Berger, 2018), which
was highly influential in the design and implementation of this project.
Joseph Sill (Sill, 2010), Wayne Winston, Jeff Sagarin and Dan Rosenbaum
developed Adjusted Plus-Minus, a sports analytics technique that partially
inspired this work. Kevinjeet Gill was unrelenting in advocating for a year-
by-year regression model to avoid sampling and quantization artifacts.

## References

  * (1)
  * Abdullah (1990) Mokhtar Bin Abdullah. 1990\.  On a Robust Correlation Coefficient.  _Journal of the Royal Statistical Society. Series D (The Statistician)_ 39, 4 (1990), 455–460.  http://www.jstor.org/stable/2349088
  * Ammar et al. (2018) Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Wang, Chris Willhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni. 2018.  Construction of the Literature Graph in Semantic Scholar. In _NAACL HLT_. Association for Computational Linguistics, 84–91.  https://doi.org/10.18653/v1/N18-3011
  * Arthur and Vassilvitskii (2007) David Arthur and Sergei Vassilvitskii. 2007.  K-means++: The Advantages of Careful Seeding. In _SODA_. SIAM, Philadelphia, PA, USA, 1027–1035.
  * Bell et al. (2009) Robert M. Bell, Yehuda Koren, and Chris Volinsky. 2009\.  The BellKor solution to the Netflix Prize.
  * Ben-David (2015) Shai Ben-David. 2015\.  Clustering is Easy When ….What?  _arXiv e-prints_ , Article arXiv:1510.05336 (Oct 2015), arXiv:1510.05336 pages.  arXiv:stat.ML/1510.05336
  * Berger (2018) Emery Berger. 2018\.  CSRankings: Computer Science Rankings.  http://csrankings.org/.
  * Bergstrom (2007) Carl Bergstrom. 2007\.  Eigenfactor: Measuring the value and prestige of scholarly journals.  _College & Research Libraries News_ 68, 5 (2007), 314–316.
  * Blackburn et al. (2018) Steve Blackburn, Carla Brodley, H. V. Jagadish, Kathryn S McKinley, Mario Nascimento, Minjeong Shin, Sean Stockwel, Lexing Xie, and Qiongkai Xu. 2018.  csmetrics.org: Institutional Publication Metrics for Computer Science.  https://github.com/csmetrics/csmetrics.org/blob/master/docs/Overview.md.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003\.  Latent dirichlet allocation.  _Journal of machine Learning research_ 3, Jan (2003), 993–1022.
  * Bornmann and Daniel (2008) Lutz Bornmann and Hans-Dieter Daniel. 2008.  What do citation counts measure? A review of studies on citing behavior.  _J. Doc._ 64, 1 (2008), 45–80.
  * Butler (2011) D. Butler. 2011\.  Computing giants launch free science metrics.  _Nature_ 476, 7358 (Aug 2011), 18\.
  * Clauset et al. (2015) Aaron Clauset, Samuel Arbesman, and Daniel B Larremore. 2015\.  Systematic inequality and hierarchy in faculty hiring networks.  _Science Advances_ (2015).
  * Coates and Ng (2011) Adam Coates and Andrew Y Ng. 2011.  The importance of encoding versus training with sparse coding and vector quantization. In _ICML_. 921–928.
  * Colledge et al. (2010) Lisa Colledge, Félix de Moya-Anegón, Vicente P Guerrero-Bote, Carmen López-Illescas, Henk F Moed, et al. 2010\.  SJR and SNIP: two new journal metrics in Elsevier’s Scopus.  _Insights_ 23, 3 (2010), 215.
  * da Silva and Dobránszki (2018) Jaime A Teixeira da Silva and Judit Dobránszki. 2018.  Multiple versions of the h-index: Cautionary use for formal academic purposes.  _Scientometrics_ 115, 2 (2018), 1107–1113.
  * da Silva and Memon (2017) Jaime A Teixeira da Silva and Aamir Raoof Memon. 2017.  CiteScore: A cite for sore eyes, or a valuable, transparent metric?  _Scientometrics_ 111, 1 (2017), 553–556.
  * Education (2018) Times Higher Education. 2018\.  World University Rankings, Computer Science.  https://www.timeshighereducation.com/world-university-rankings/2018/subject-ranking/computer-science.
  * Falagas et al. (2008) Matthew E. Falagas, Vasilios D. Kouranos, Ricardo Arencibia-Jorge, and Drosos E. Karageorgopoulos. 2008\.  Comparison of SCImago journal rank indicator with journal impact factor.  _The FASEB Journal_ 22, 8 (2008), 2623–2628.
  * Fenner (2013) Martin Fenner. 2013\.  What can article-level metrics do for you?  _PLoS biology_ 11, 10 (2013), e1001687.
  * Foundation (2018) National Science Foundation. 2018\.  Download Awards by Year.  https://www.nsf.gov/awardsearch/download.jsp.
  * Freund and Schapire (1997) Yoav Freund and Robert E Schapire. 1997.  A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting.  _J. Comput. System Sci._ 55, 1 (1997), 119 – 139.  https://doi.org/10.1006/jcss.1997.1504
  * Galiani and Gálvez (2017) Sebastian Galiani and Ramiro H Gálvez. 2017.  _The life cycle of scholarly articles across fields of research_.  Technical Report. National Bureau of Economic Research.
  * Geist et al. (1996) Robert Geist, Madhu Chetuparambil, Stephen Hedetniemi, and A. Joe Turner. 1996.  Computing Research Programs in the U.S.  _Commun. ACM_ 39, 12 (Dec. 1996), 96–99.  https://doi.org/10.1145/240483.240505
  * Hirsch (2005) J E Hirsch. 2005\.  An index to quantify an individual’s scientific research output.  _Proc. Natl. Acad. Sci. U. S. A._ 102, 46 (nov 2005), 16569–16572.  https://doi.org/10.1073/pnas.0507655102
  * Huang (2018) Jeff Huang. 2018\.  Best Paper Awards in Computer Science (since 1996).  https://jeffhuang.com/best_paper_awards.html.
  * Huber (1964) Peter J. Huber. 1964\.  Robust Estimation of a Location Parameter.  _Ann. Math. Statist._ 35, 1 (03 1964), 73–101.  https://doi.org/10.1214/aoms/1177703732
  * Huber (1985) Peter J. Huber. 1985\.  Projection Pursuit.  _The Annals of Statistics_ 13, 2 (1985), 435–475.  http://www.jstor.org/stable/2241175
  * Institute (2018) Nevada Policy Research Institute. 2018\.  Transparent California.  https://transparentcalifornia.com/agencies/salaries/#university-system.
  * Johnes (2018) Jill Johnes. 2018\.  University rankings: What do they really show?  _Scientometrics_ 115, 1 (01 Apr 2018), 585–606.  https://doi.org/10.1007/s11192-018-2666-1
  * Kurtz and Henneken (2017) Michael J. Kurtz and Edwin A. Henneken. 2017.  Measuring metrics - a 40-year longitudinal cross-validation of citations, downloads, and peer review in astrophysics.  _JAIST_ 68, 3 (2017), 695–708.  https://doi.org/10.1002/asi.23689
  * Lehmann et al. (2006) Sune Lehmann, Andrew D Jackson, and Benny E Lautrup. 2006\.  Measures for measures.  _Nature_ 444, 7122 (2006), 1003.
  * Ley (2002) Michael Ley. 2002\.  The DBLP Computer Science Bibliography: Evolution, Research Issues, Perspectives. In _String Processing and Information Retrieval_ , Alberto H. F. Laender and Arlindo L. Oliveira (Eds.). Springer Berlin Heidelberg, 1–10.
  * Lubienski et al. (2018) Sarah Theule Lubienski, Emily K. Miller, and Evthokia Stephanie Saclarides. 2018.  Sex Differences in Doctoral Student Publication Rates.  _Educational Researcher_ 47, 1 (2018), 76–81.  https://doi.org/10.3102/0013189X17738746
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013\.  Efficient estimation of word representations in vector space.  _ICLR_ (2013).
  * Morgan et al. (2018a) Allison C. Morgan, Dimitrios J. Economou, Samuel F. Way, and Aaron Clauset. 2018a.  Prestige drives epistemic inequality in the diffusion of scientific ideas.  _EPJ Data Science_ 7, 1 (19 Oct 2018), 40\.
  * Morgan et al. (2018b) Allison C. Morgan, Samuel F. Way, and Aaron Clauset. 2018b.  Automatically assembling a full census of an academic field.  _PLOS ONE_ 13, 8 (08 2018), 1–18.  https://doi.org/10.1371/journal.pone.0202223
  * News and Report (2018) US News and World Report. 2018.  Best Computer Science Schools.  https://www.usnews.com/best-graduate-schools/top-science-schools/computer-science-rankings.
  * Nocka et al. (2014) Andrew Nocka, Danning Zheng, Tianran Hu, and Jiebo Luo. 2014\.  Moneyball for Academia: Toward Measuring and Maximizing Faculty Performance and Impact. In _2014 IEEE International Conference on Data Mining Workshop_. 193–197.
  * Page et al. (1999) Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999.  _The PageRank citation ranking: Bringing order to the web._ Technical Report. Stanford.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ 12 (2011), 2825–2830.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation. In _EMNLP_. 1532–1543.
  * Rankings (2018) QS World University Rankings. 2018\.  Computer Science & Information Systems.  https://www.topuniversities.com/university-rankings/university-subject-rankings/2018/computer-science-information-systems.
  * Rankings (2015) Shanghai Rankings. 2015\.  Academic Ranking of World Universities in Computer Science.  http://www.shanghairanking.com/SubjectCS2015.html.
  * Ren and Taylor (2007) Jie Ren and Richard N Taylor. 2007.  Automatic and versatile publications ranking for research institutions and scholars.  _Commun. ACM_ 50, 6 (2007), 81–85.
  * Robbins and Monro (1951) Herbert Robbins and Sutton Monro. 1951.  A Stochastic Approximation Method.  _The Annals of Mathematical Statistics_ 22, 3 (1951), 400–407.
  * Rousseeuw (1987) Peter J. Rousseeuw. 1987\.  Silhouettes: A graphical aid to the interpretation and validation of cluster analysis.  _J. Comput. Appl. Math._ 20 (1987), 53 – 65.  https://doi.org/10.1016/0377-0427(87)90125-7
  * Rumelhart et al. (1986) David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1986.  Learning representations by back-propagating errors.  _Nature_ 323, 6088 (1986), 533–536.
  * Schauerte (2014) Boris Schauerte. 2014\.  Microsoft Academic: conference field ratings.  http://www.conferenceranks.com/visualization/msar2014.html.
  * Sekercioglu (2008) Cagan H. Sekercioglu. 2008\.  Quantifying Coauthor Contributions.  _Science_ 322, 5900 (2008), 371–371.  https://doi.org/10.1126/science.322.5900.371a
  * Shah et al. (2018) Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike von Luxburg. 2018\.  Design and Analysis of the NIPS 2016 Review Process.  _JMLR_ 19, 49 (2018), 1–34.  http://jmlr.org/papers/v19/17-511.html
  * Sill (2010) Joseph Sill. 2010\.  Improved NBA adjusted+/-using regularization and out-of-sample testing. In _MIT Sloan Sports Analytics Conference_.
  * Spearman (1904) Charles Spearman. 1904\.  The proof and measurement of association between two things.  _The American journal of psychology_ 15, 1 (1904), 72–101.
  * Tol (2008) Richard S. J. Tol. 2008\.  A rational, successive g-index applied to economics departments in Ireland.  _J. Informetrics_ 2 (2008), 149–155.
  * Valenzuela et al. (2015) Marco Valenzuela, Vu Ha, and Oren Etzioni. 2015.  Identifying meaningful citations. In _AAAI Workshop: Scholarly Big Data_.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing Data using t-SNE.  _Journal of Machine Learning Research_ 9 (2008), 2579–2605.
  * Vucetic et al. (2018) Slobodan Vucetic, Ashis Kumar Chanda, Shanshan Zhang, Tian Bai, and Aniruddha Maiti. 2018\.  Peer Assessment of CS Doctoral Programs Shows Strong Correlation with Faculty Citations.  _Commun. ACM_ 61, 9 (Aug. 2018), 70–76.
  * Walters (2017) W. H. Walters. 2017\.  Citation-Based Journal Rankings: Key Questions, Metrics, and Data Sources.  _IEEE Access_ 5 (2017), 22036–22053.  https://doi.org/10.1109/ACCESS.2017.2761400
  * Yan and Lee (2007) Su Yan and Dongwon Lee. 2007.  Toward Alternative Measures for Ranking Venues: A Case of Database Research Community. In _Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries_ _(JCDL ’07)_. ACM, New York, NY, USA, 235–244.
  * Yong (2014) Alexander Yong. 2014\.  Critique of Hirsch’s citation index: A combinatorial Fermi problem.  _Notices of the AMS_ 61, 9 (2014), 1040–1050.
  * Zhang and Wu (2018) Fang Zhang and Shengli Wu. 2018.  Ranking Scientific Papers and Venues in Heterogeneous Academic Networks by Mutual Reinforcement. In _JCDL_. ACM, New York, NY, USA, 127–130.
  * Zhang (2004) Tong Zhang. 2004\.  Solving Large Scale Linear Prediction Problems Using Stochastic Gradient Descent Algorithms. In _ICML_. ACM, New York, NY, USA, 116–.

##  Appendix A Credit Assignment

In order to address issues of collinearity raised by having authors who
publish papers together, we wanted to solve a credit assignment problem. We
adapted a well-known method for addressing this by adapting regularized plus
minus (Sill, 2010) from the sports analytics literature. In our case, we
simply regress each publications values from its authors, as shown below.

$\bordermatrix{&\text{author}_{1}&\text{author}_{2}&\ldots&\text{author}_{n}\cr\text{paper}_{1}&1&1&\ldots&0\cr\text{paper}_{2}&1&0&\ldots&0\cr\vdots&\vdots&\vdots&\ddots&\vdots\cr\text{paper}_{m}&0&1&\ldots&1}\begin{pmatrix}x_{1}\\\
x_{2}\\\ \vdots\\\ x_{n}\end{pmatrix}=\begin{pmatrix}\text{score}_{1}\\\
\text{score}_{2}\\\ \vdots\\\ \text{score}_{n}\end{pmatrix}$

This technique produced scores that correlated highly with total value scores.
While this may be valuable for understanding an individual’s contribution, but
we were unable to find an evaluation for this method. The form of this model
that considers n-wise or pairwise relationships was also considered, but these
matrices were too large for us to run trials.

Figure 8. Curve showing the natural number of clusters in the dataset using R1
authors

##  Appendix B Aging Curve

To evaluate if our model makes a sensible prediction over the timescale of a
scholar’s career, we built a model to see what an average academic career
looks like, given that the author is still publishing in those years. See
Figure 9. Our model suggests a rise in productivity for the first 20 years of
one’s publishing history, and then a steady decline.

Figure 9. The average productivity of all DBLP authors for that year of their publishing career. Figure 10. An automatic clustering and rating of the entirity of CS. Table 9. Top 55 Venues Name | Score | Size
---|---|---
STOC | 11.71 | 128
FOCS | 11.04 | 120
NIPS | 10.94 | 484
Conference on Computational Complexity | 10.41 | 53
SODA | 10.17 | 224
SIAM J. Comput. | 9.98 | 138
HotNets | 9.53 | 46
ICML | 9.46 | 303
TCC | 9.06 | 76
NSDI | 8.76 | 69
CCS | 8.76 | 114
INFOCOM | 8.59 | 434
ITCS | 8.56 | 103
Allerton | 8.53 | 331
ASPLOS | 8.43 | 48
SIGCOMM | 8.31 | 55
CVPR | 8.22 | 606
PLDI | 8.05 | 68
J. ACM | 8.03 | 82
Theory of Computing | 7.94 | 29
USENIX Security Symposium | 7.91 | 62
NDSS | 7.86 | 63
APPROX-RANDOM | 7.86 | 74
COLT | 7.78 | 82
VLDB | 7.75 | 185
IEEE/ACM Trans. Netw. | 7.74 | 226
AAAI | 7.73 | 403
SIGMOD Conference | 7.62 | 98
SIGGRAPH | 7.59 | 125
AISTATS | 7.52 | 138
OOPSLA | 7.41 | 75
SIGGRAPH Asia | 7.38 | 140
CRYPTO | 7.24 | 80
EUROCRYPT | 7.24 | 78
POPL | 7.22 | 68
CHI | 7.15 | 431
IEEE SSP | 7.11 | 54
WWW | 6.82 | 232
HotOS | 6.75 | 27
EMNLP | 6.61 | 263
EC | 6.60 | 63
Commun. ACM | 6.59 | 222
GetMobile | 6.50 | 41
IMC | 6.49 | 70
OSDI | 6.46 | 29
ICDE | 6.44 | 183
SIGSOFT FSE | 6.42 | 93
ECCV | 6.42 | 252
ICALP | 6.40 | 148
ISIT | 6.28 | 1032
KDD | 6.23 | 240
ICCV | 6.22 | 292
Robotics: Science and Systems | 6.20 | 88
MICRO | 6.19 | 62
ISCA | 6.18 | 73
Table 10. Top 55 Universities school | authors | papers | venue score | size normed
---|---|---|---|---
Carnegie Mellon University | 174 | 17275 | 73126 | 14159
University of California - Berkeley | 108 | 11011 | 51062 | 10884
Massachusetts Institute of Technology | 108 | 9613 | 47037 | 10026
Univ. of Illinois at Urbana-Champaign | 103 | 11248 | 44714 | 9627
Technion | 96 | 8795 | 39601 | 8656
Stanford University | 69 | 7028 | 36169 | 8513
Georgia Institute of Technology | 108 | 9337 | 38083 | 8118
Tsinghua University | 150 | 14643 | 40662 | 8104
University of California - Los Angeles | 46 | 6589 | 30368 | 7888
University of Michigan | 83 | 7897 | 33666 | 7598
Tel Aviv University | 48 | 5468 | 28816 | 7404
University of California - San Diego | 74 | 6646 | 30836 | 7142
University of Maryland - College Park | 76 | 7751 | 30795 | 7089
ETH Zurich | 39 | 6673 | 26120 | 7081
Cornell University | 81 | 5918 | 30278 | 6871
University of Washington | 69 | 5727 | 29087 | 6846
University of Southern California | 58 | 7206 | 26044 | 6387
Columbia University | 53 | 5064 | 25251 | 6330
EPFL | 55 | 6927 | 25320 | 6290
Princeton University | 47 | 4991 | 24203 | 6252
HKUST | 57 | 6640 | 25134 | 6190
National University of Singapore | 75 | 7210 | 25123 | 5801
University of Pennsylvania | 53 | 5340 | 22696 | 5690
University of California - Irvine | 71 | 6624 | 24070 | 5628
Pennsylvania State University | 49 | 5668 | 21325 | 5451
Peking University | 147 | 11858 | 27050 | 5413
University of Toronto | 99 | 5955 | 24759 | 5376
University of Texas at Austin | 52 | 4485 | 21225 | 5346
University of California - Santa Barbara | 38 | 4698 | 19137 | 5224
University of Waterloo | 105 | 7316 | 23781 | 5099
Max Planck Institute | 32 | 4307 | 17808 | 5093
Purdue University | 71 | 5659 | 21734 | 5082
New York University | 67 | 4915 | 20752 | 4918
Rutgers University | 63 | 4935 | 20311 | 4884
Nanyang Technological University | 71 | 9143 | 20827 | 4870
University of Wisconsin - Madison | 54 | 3988 | 18986 | 4738
University of Massachusetts Amherst | 57 | 4129 | 19155 | 4717
Hebrew University of Jerusalem | 42 | 3316 | 17478 | 4647
University of California - Riverside | 43 | 4230 | 17114 | 4522
Shanghai Jiao Tong University | 77 | 8220 | 19652 | 4511
Zhejiang University | 89 | 7531 | 20232 | 4496
Ohio State University | 42 | 4121 | 16740 | 4451
Duke University | 23 | 2879 | 13936 | 4385
Chinese Academy of Sciences | 46 | 5795 | 16497 | 4285
University of Minnesota | 43 | 4346 | 15855 | 4190
Northwestern University | 49 | 3993 | 16183 | 4137
Stony Brook University | 47 | 4565 | 15819 | 4086
University of Edinburgh | 115 | 6461 | 19292 | 4058
University of Tokyo | 80 | 7524 | 17762 | 4042
University of Oxford | 69 | 5549 | 17160 | 4039
Chinese University of Hong Kong | 32 | 3927 | 13843 | 3959
TU Munich | 53 | 6432 | 15521 | 3891
KAIST | 81 | 5591 | 17114 | 3884
Harvard University | 34 | 2471 | 13640 | 3837
Imperial College London | 65 | 6622 | 15834 | 3779
Table 11. Top 50 Authors Name | Score
---|---
Philip S. Yu | 4354
H. Vincent Poor | 3873
Kang G. Shin | 3540
Jiawei Han 0001 | 3143
Micha Sharir | 3086
Thomas S. Huang | 2992
Don Towsley | 2913
Xuemin Shen | 2755
Luc J. Van Gool | 2700
Noga Alon | 2596
Christos H. Papadimitriou | 2531
Leonidas J. Guibas | 2525
Mahmut T. Kandemir | 2467
Jie Wu 0001 | 2452
Wen Gao 0001 | 2416
Shuicheng Yan | 2373
Rama Chellappa | 2286
Alberto L. Sangiovanni-Vincentelli | 2266
Georgios B. Giannakis | 2266
Yunhao Liu | 2257
Mohamed-Slim Alouini | 2238
Michael I. Jordan | 2213
Christos Faloutsos | 2173
Massoud Pedram | 2153
Dacheng Tao | 2149
Hans-Peter Seidel | 2145
Luca Benini | 2139
Moshe Y. Vardi | 2092
Lajos Hanzo | 2081
Yishay Mansour | 2073
Sudhakar M. Reddy | 2070
Sajal K. Das | 2045
Pankaj K. Agarwal | 2042
Avi Wigderson | 2024
Robert E. Tarjan | 2019
Irith Pomeranz | 2003
Elisa Bertino | 2003
Zhu Han | 1992
Shlomo Shamai | 1976
David Peleg | 1974
Victor C. M. Leung | 1966
Kaushik Roy 0001 | 1949
Xiaoou Tang | 1938
Hector Garcia-Molina | 1923
Larry S. Davis | 1918
Rafail Ostrovsky | 1895
Eric P. Xing | 1881
Ness B. Shroff | 1879
Krishnendu Chakrabarty | 1868
Hai Jin 0001 | 1867


# Personalizing Text-to-Image Generation via
Aesthetic Gradients

Victor Gallego
Komorebi AI Technologies
<EMAIL_ADDRESS>
###### Abstract

This work proposes aesthetic gradients, a method to personalize a CLIP-
conditioned diffusion model by guiding the generative process towards custom
aesthetics defined by the user from a set of images. The approach is validated
with qualitative and quantitative experiments, using the recent stable
diffusion model and several aesthetically-filtered datasets. Code is released
at https://github.com/vicgalle/stable-diffusion-aesthetic-gradients

|  Original SD |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Personalized with $\mbox{SAC}_{8+}$ |  |  |  |  |  |  |  |
|  | A fountain, sculpture
---
| A giant octopus, bioluminescence
---
| A nighttime cityscape, concept art
---
| A gateway between dreams
---
Original SD |  |  |  |  |  |  |  |
Personalized with $\mbox{SAC}_{8+}$ |  |  |  |  |  |  |  |
|  | A clearing filled with colorful
---
plants in a thick wood where time
has stopped, trending on Artstation
| A beautiful mannequin made
---
of marble printed in 3d
geometric neon + kintsugi…
| Photorealistic white marble greek
---
goddess face profile sculpture
entwined by crimson vines…
| Marble Polished Tile.
---
Dark Volcano is an impressive
dark red quartzite…
Original SD |  |  |  |  |  |  |  |
Personalized with $\mbox{LAION}_{7+}$ |  |  |  |  |  |  |  |
|  | A whole train covered in wildstyle
---
graffiti in the style of Seen, HQ
masterpiece, vibrant
| Art Deco in Santorini island
---
palm trees crystal clear neon water,
trending Artstation, octane render…
| A gothic cathedral in the
---
middle of the forest
| A painting of a path going up
---
a hill, oil on canvas
Figure 1: Stable diffusion generations for the original model and personalized
variants using $\mbox{SAC}_{8+}$ and $\mbox{LAION}_{7+}$ aesthetic embeddings.

### Introduction

Recently developed text-to-image models have demonstrated an unprecedented
capability to reason over natural language descriptions [8]. Their use,
however, is constrained by the user’s ability to describe the desired target
through text. But can generative models really grasp the aesthetic vibe
preferred by its users? Recent works such as _textual inversion_ [2] and
_dreambooth_ [9] aim to provide user personalization to diffusion models, but
rather focus on learning custom objects from few images. Instead, in this work
we present an alternative approach for _personalization_ of text-to-image
diffusion models. Our goal is to guide the generative process towards custom
aesthetics defined by the user, without restricting us to single objects, but
rather aesthetic patterns specified by arbitrarily large sets of images.

### Method

At first, the user chooses a textual prompt $y$ to guide the generation. This
prompt is passed through the CLIP text encoder [7] to obtain a textual
embedding $c=\mbox{CLIP}_{\theta,txt}(y)$. Conditioned on this representation
$c$, the diffusion process generates a image that matches the prompt. We
propose to modify the previous representation $c$ by taking into account
another embedding, representing the aesthetic preferences of the user. Let
$\\{x_{i}\\}_{i=1}^{K}$ be a set of $K$ images representing the aesthetic
preference of an user. We define its corresponding _aesthetic embedding_ $e$
as the average of the visual embeddings of the previous images, that is,
$e=\frac{1}{K}\sum_{i=1}^{K}\mbox{CLIP}_{\theta,vis}(x_{i})$. Then we
normalize the resulting vector to be of unitary norm. The similarity between
the two embeddings, computed as the dot product $ce^{\intercal}$, can be used
to measure the agreement between CLIP representation of the textual prompt and
the preferences of the user. Thus, the previous expression can be used as a
loss and we can perform gradient descent with respect to CLIP text encoder
weights to drive the prompt representation towards the aesthetics of the user:
$\theta^{\prime}=\theta+\epsilon\,\nabla_{\theta}\mbox{CLIP}_{\theta,txt}(y)e^{\intercal}$,
with $\epsilon$ being a user-defined step size. After a few iterations, we
compute the new, personalized prompt representation,
$c^{\prime}=\mbox{CLIP}_{\theta^{\prime},txt}(y)$, and the generation
continues using the underlying diffusion process. The resulting representation
$c^{\prime}$ is more aligned to the user preference, while preserving the
original semantics, as we will see in the experiments from the next section.
Note that only the weights of the CLIP text encoder are modified, nor the
visual encoder nor any other component of the diffusion model.

The benefits of our approach to personalization are several: (i) it works
agnostically of the diffusion model, that is, it only requires a diffusion
model which conditions on a textual prompt processed by CLIP. In the
experiments we use the recent _stable diffusion_ (SD) model. (ii) it is
computationally cheap, as it only requires a few gradient steps (less than 20
in the experiments) of the CLIP text encoder. It is not necessary to fine-tune
the diffusion model, thus making it GPU-friendly. (iii) the user only needs to
store one aesthetic embedding per set of images, thus saving storage space and
being amenable to sharing. In the case of CLIP-L/14, the variant used by SD,
$e$ is a vector of 768 dimensions. A potential drawback of our approach is
that we introduce two new hyperparameters: the step size $\epsilon$ and the
number of iterations. For our experiments, we set $\epsilon=1e-4$ and vary the
number of iterations from 5 to 20; yet these are two levers that the user can
control.

### Experiments and results

First, we qualitatively show the effectiveness of our aesthetic gradients
approach using several aesthetic embeddings and a collection of prompts of
varying length and complexity. As the aesthetic embeddings, for this
experiment we use two sets of images: $\mbox{SAC}_{8+}$, a subset of Simulacra
Aesthetic Captions [6] with images filtered to have a rating greater than 8;
and $\mbox{LAION}_{7+}$: a subset of LAION Aesthetics v1 [4] with images
filtered to have a rating greater than 7. Figure 1 shows several generations,
comparing the original SD model with the personalizations. Note that the
personalized generations better reflect the aesthetics of each embedding, such
as more fantasy-like imagery in the case of $\mbox{SAC}_{8+}$, and more floral
patterns in $\mbox{LAION}_{7+}$.

Figure 2: Aesthetic scores distribution

We also perform an experiment to quantitatively assess the performance of the
aesthetic gradient approach. To do so, we generate a list of 25 prompts, of
varying length and complexity (Table 1). For each, we generate six images,
both using the original SD model and personalizing with $\mbox{SAC}_{8+}$.
Finally, for each image we compute its aesthetic score using an open source
model [1]. The distribution of the scores for each of the two groups of
generations is shown in Figure 2. Note the personalized models has improved
aesthetic score, even though the diffusion model has not been optimized with
respect to it. Appendix Additional experiments and results includes another
batch of experiments, using different aesthetics.

### Conclusion

We have proposed a flexible and efficient approach for personalizing text-to-
image models, in particular by guiding the generation towards the aesthetic
preferences of the user. As further work, it is straightforward to adapt our
method to CLIP-guided diffusion [5], in which the customized CLIP model guides
the generation at every timestep of the diffusion process. Also, instead of
gradient descent optimizers, we could use SG-MCMC samplers [10, 3] to explore
a greater region of the latent space and improve the diversity of the results.

## Ethical Implications

This work aims to provide users with an effective framework for adapting the
output of diffusion models to the preferred aesthetics of the user. While
general text-to-image models might be biased towards specific attributes when
synthesizing images from text, our approach enables the user to better reflect
the desirable effects. On the other hand, malicious agents might try to use
generated images to mislead viewers. This is a common issue, existing in other
generative modeling approaches. Future research in generative modeling and
personalization must continue investigating and revaluing these issues.

## References

  * [1] Katherine Crowson.  Simulacra aesthetic models.  Technical Report Version 1.0, 2022.  url https://github.com/crowsonkb/simulacra-aesthetic-models .
  * [2] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.
  * [3] Victor Gallego and David Rios Insua.  Stochastic gradient mcmc with repulsive forces.  arXiv preprint arXiv:1812.00071, 2018.
  * [4] LAION.  Laion aesthetics v1.  Technical Report Version 1.0, LAION AI, 2022.  url https://github.com/LAION-AI/aesthetic-predictor .
  * [5] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [6] John David Pressman, Katherine Crowson, and Simulacra Captions Contributors.  Simulacra aesthetic captions.  Technical Report Version 1.0, Stability AI, 2022.  url https://github.com/JD-P/simulacra-aesthetic-captions .
  * [7] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [8] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [9] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  2022\.
  * [10] Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson.  Cyclical stochastic gradient mcmc for bayesian deep learning.  arXiv preprint arXiv:1902.03932, 2019.

## Additional experiments and results

Table 1: Prompts used for the quantitative experiment

A fountain, sculpture
---
A pyramid over a snowy scenery
A giant octopus, bioluminescence
A still life of flowers, volumetric lighting
A still life of flowers, stained glass
A nighttime cityscape, concept art
The sacred library by Simon Stålenhag and Thomas Kinkade, oil on canvas
A gateway between dreams
Space jellyfish, watercolor
Giant skull without a lower jaw, floating above a pile of gems while
it leaks gems and bone. An orange, cloudy sky fills the background
An orange overstuffed chair, custom design
Ethereal
A clearing filled with colorful plants in a thick woods
where time has stopped, trending on Artstation
An archer lounging against a tree with petals falling,
painting by Horace Vernet
Textless, 8k, hyperdetail Papier-mache, Ambient occlusion High key light,
Contour rivalry, octane render redshift render, Porcelain painted
ceramics by Krystle Mitchell, The efficient panda surrounds bangle,
ascot plain peel postfix circadian sunroom
Dimming dares to swifting ruins lights, charges changes on the skies
from above, blinks true to throughout, to a closing of hands on spacing
world to binding breaks of recreating strings, then to dusting fantasy
of hands that try to wave a reach of each, and a
spine of splitting reeks of falling sense of decaying skying
Centuries of citadels, and been tuning in tones that been crystalize in a
field that felt a widing in its own, still a lighten
abyss vision for depths, it still crystalline in souls
that truly enjoyed, of a meaning
White marble, white marble bas relief profile sculpture of a
beautiful black haired woman with pale skin and a crown on her head
sitted on an intricate metal throne, medusa, white and gold kintsugi,
feminine shapes, crabs, spiders, scorpions, tarantulas, stunning,
art by hr geiger and ridley scott and alphonse mucha
and josephine wall, highly detailed, intricately detailed
Photorealistic white marble greek goddess face profile sculpture
entwined by golden and crimson vines and roots, flesh shows at
some parts under the broken marble, swirling liquified meat and
red kintsugi, symbolist, visionary, etheric, entwined with iridiscent
fractal lace, alien botanicals, cinematic composition, cinematic lighting
A beautiful mannequin made of marble printed in 3 d geometric neon
\+ kintsugi, facing a giant doorway opening with a neon pink light,
flowering iridescent pineapples + orchids, transcendent, vibrant color,
clean linework, finely detailed, 4k, trending on artstation,
photorealistic, volumetric lighting, octane render
A pirate ship, sepia coloring, hyper-detailed, dusk, 4k octane render
Vaporwave soviet skyline at sunrise, trending on Artstation.
Many intrincate details
Marble Polished Tile. Sky Blue is an impressive pale blue quartzite.
Its appearance is reminiscent of a splendid blue sky interspersed with
fluffy white clouds, as its name suggests. This natural stone’s
base shuffles different soft blues such as blue lavender,
pale blue, and pastel indigo. The veins look like clouds.
Decorative marble tile
A photograph of an astronaut riding a horse
A painting of a tree, oil on canvas

Figure 3 depicts further qualitative experiments, with different aesthetics of
those from the main text. In particular, we target the following aesthetic
preferences: Aivazovsky (five paintings from the artist), cloudcore,
gloomcore, and glowwave, with the last three consisting in 100 random images
scraped from Pinterest using those terms as the keyword. In the Figure, the
first column shows the generations for the original SD model, the second
column shows the generations also for the original model but appending the
aesthetic keyword at the end of the prompt; and the third column show the
generations personalized with the proposed aesthetic gradients approach. Note
that whereas the second column shows that the effect of appending the
aesthetic keyword in the prompt has a negligible or very dim effect due to the
limitations of the CLIP text encoder, the aesthetic gradients approach can
better reflect the aesthetic effect that is to be expected, dramatically
improving the corresponding aesthetic vibe of the generations.

|  | Original SD | Original SD | Personalized
---|---|---|---
aivazovsky |  |  |
|  | A painting of a tree, oil on canvas
---
| A painting of a tree, oil on canvas
---
by Ivan Aivazovsky
| A painting of a tree, oil on canvas
---
cloudcore |  |  |
|  | A pirate ship, sepia coloring, hyper-
---
detailed, dusk, 4k octane render
| A pirate ship, sepia coloring, hyper-
---
…, cloudcore
| A pirate ship, sepia coloring, hyper-
---
detailed, dusk, 4k octane render
gloomcore |  |  |
|  | A clearing filled with colorful
---
plants in a thick wood where time
has stopped, trending on Artstation
| A clearing filled with colorful
---
plants in a thick wood where time
…, gloomcore
| A clearing filled with colorful
---
plants in a thick wood where time
has stopped, trending on Artstation
gloomcore |  |  |
|  | Award-wining photograph of a
---
brutalist concrete building with
exuberant vegetation, Provia, Velvia
| Award-wining photograph of a
---
brutalist concrete building with
…, gloomcore
| Award-wining photograph of a
---
brutalist concrete building with
exuberant vegetation, Provia, Velvia
glowwave |  |  |
|  | Award-wining photograph of a dark
---
corridor, Provia, Velvia
| Award-wining photograph of a dark
---
corridor, Provia, Velvia, glowwave
| Award-wining photograph of a dark
---
corridor, Provia, Velvia
glowwave |  |  |
|  | Award-wining photograph of a dark
---
corridor, Provia, Velvia
| Award-wining photograph of a dark
---
corridor, Provia, Velvia, glowwave
| Award-wining photograph of a dark
---
corridor, Provia, Velvia
glowwave |  |  |
|  | Award-wining photograph of japanese
---
town street at night, Provia, Velvia
| Award-wining photograph of japanese
---
…, glowwave
| Award-wining photograph of japanese
---
town street at night, Provia, Velvia
Figure 3: Further qualitative results using different aesthetic embeddings.