# The Right Variety: Improving Expressive Range Analysis with Metric Selection
Methods

Oliver Withington<EMAIL_ADDRESS>Queen Mary University of
LondonLondonUK and  Laurissa Tokarchuk<EMAIL_ADDRESS>Queen
Mary University of LondonLondonUK

††conference: Foundations of Digital Games; April 11-14, 2022; Lisbon,
Portugal

##  1\. Abstract

Expressive Range Analysis (ERA), an approach for visualising the output of
Procedural Content Generation (PCG) systems, is widely used within PCG
research to evaluate and compare generators, often to make comparative
statements about their relative performance in terms of output diversity and
search space exploration. Producing a standard ERA visualisation requires the
selection of two metrics which can be calculated for all generated artefacts
to be visualised. However, to our knowledge there are no methodologies or
heuristics for justifying the selection of a specific metric pair over
alternatives. Prior work has typically either made a selection based on
established but unjustified norms, designer intuition, or has produced
multiple visualisations across all possible pairs, which limits the clarity
and usability of the approach. This work aims to contribute to this area by
identifying valuable characteristics of metric pairings. Pairings that have
these characteristics have an increased probability of producing an
informative ERA projection of the underlying generator. We introduce and
investigate three quantifiable selection criteria for assessing metric pairs,
and demonstrate how these criteria can be operationalized to rank those
available. Though this is an early exploration of the concept of quantifying
the utility of ERA metric pairs, we argue that the contribution of this paper
can make ERA more useful and usable for both researchers and game designers.

##  2\. Introduction

Procedural Content Generation (PCG) systems for games can often produce huge
amounts of diverse artefacts and experiences. Whether it is the billions of
possible maps of Minecraft (persson2011) or the quintillions of planets in No
Man Sky (hellogamesfirm2016), the total volume of a PCG system’s possible
output, commonly referred to as its ‘generative space’ can be huge and
complex. The challenge of understanding and usefully comparing generative
spaces is a vital one to both games researchers and game designers.
Researchers want to be able to credibly claim that their novel systems are an
improvement over prior alternatives in terms of the diversity or quality of
output. Similarly, commercial game designers want to be confident that they
are selecting the right generative approach, and tuning their chosen
approaches correctly to produce the optimal generative space.

A popular approach for gaining an understanding of a generative space, is to
produce visualisations of it which condense the highly convoluted
n-dimensional spaces of possible artefacts, into something 2-dimensional and
human understandable. A common approach for visualising generative spaces in
prior PCG research is Expressive Range Analysis (ERA), a technique pioneered
by Smith et al (smith2010). ERA allows us to visualise sets of game levels
using two or more quantifiable metrics of interest such as heuristics for
difficulty or aesthetic qualities. These metrics are used to localise each
artefact in a 2-dimensional plot, which can then be used to understand how the
generated content is distributed in this metric defined space. This helps
designers to understand how the generative space of their generator is
positioned within the larger possibility space of levels that could be
produced within that domain, and to compare that generative space to
alternatives.

Arguably the most important choice when applying ERA is the pair of metrics
that are used to produce the plot. However, when ERA has been deployed in
prior research this choice is typically not justified, and the impact of it is
normally not discussed. There are to our knowledge no prior works that aim to
justify the value of specific metric pairs over the alternatives available, or
methods to assist a researcher or designer in deciding on the best pair to
use. Most commonly, research using ERA either repurposes metric pairings from
prior work, often Smith et als original paper, or a large number of ERA plots
is produced for all metric pairs which limits the impact of making a poor
choice but also reduces the usefulness and readability using ERA.

It is to this area of selecting metrics for applying ERA that this paper makes
a contribution. We argue that there are actually several characteristics of
metric pairs which could make them more desirable than alternatives when
applying ERA, and that the calculation of these characteristics across
candidate pairs of metrics can form the basis of helpful selection criteria.
Specifically the selection criteria we look at are:

  * •

Fitness Independence, meaning that fit content is evenly distributed in the
metric pair defined space

  * •

Mutual Independence, meaning there is minimal correlation between the pair of
metrics selected

  * •

Minimal Alternative Metric Independence, meaning there is minimal metric
diversity which is unaccounted for by a selected metric pair

This work makes the case that the more a metric pair satisfies these criteria,
the more useful an ERA visualisation using them will be. Combined, they help
to ensure that as much of the ERA plot as possible is accessible for the
generator, that as much of the plot is useful for the generator to explore,
and that what appears to be diversity for generative spaces using it actually
represents diversity across many possible metrics.

In this paper we introduce these selection criteria for ERA metric pairs and
discuss their value to both game designers and researchers looking to
understand or compare their systems’ generative spaces. We show how these
characteristics can be calculated to differentiate between and rank
alternative metric pairings, allowing a more informed choice to be made about
which pair to visualise in cases where it is not obvious. We also conduct a
series of pilot experiments in the popular research domain of generated levels
for Super Mario Bros, using the approach to evaluate and rank over 150
possible metric pairings for visualising generated output using ERA. The
results of these experiments suggest that in spite of being an early
exploration of these ideas, applying these selection criteria to metric
selection can be a valuable source of insight into how best to conduct ERA
within specific game domains, as well as potentially a source of more general
insight into the design of good metrics. We conclude that this approach could
substantially improve on the usability and utility ERA as it is currently
applied while maintaining its benefits, allowing it to be used as a more
robust approach for understanding and comparing the generative spaces of PCG
systems.

##  3\. Related Work

In this section we introduce and discuss the work that most directly informed
and inspired this project, both to highlight the core concepts that underpin
it and to show where this project aims to improve on the current state-of-the-
art.

###  3.1. Expressive Range Analysis

ERA was first introduced by Gillian Smith and Jim Whitehead in 2010
(smith2010), and was proposed as a novel method for visualising and
understanding generative spaces. Its mode of operation is appealingly simple,
which has no doubt contributed to its widespread adoption. To conduct basic
ERA a designer first generates a large set of levels from a generator, and
then calculates two or more metrics for each level. The set of levels can then
be visualised on a 2D plot, most commonly a heatmap. These 2D plots can then
be used to understand the character and diversity present in individual
generative spaces, but also those from alternative generators compare to each
other. ERA has most commonly been applied to work within its original domain
of PCG for game levels, but it has also found a place in alternative areas of
PCG research such as visualising the output of poetry
generators(kreminski2022), tabletop game outcomes (guzdial2020) and emergent
narrative systems(kybartas2018).

Over the intervening years ERA has been used for several alternative purposes
within PCG research. In Smith et al’s original paper (smith2010) it was
primarily used to qualitatively understand their systems output tendency, with
a focus on characteristics such as what kinds of levels are likely to be
produced and which levels are hard or impossible to produce. However they also
highlighted that ERA could form the basis of quantitative comparisons between
generators, a use case that has proved very popular in research since. It has
often been used to heuristically assess the relative amounts of output
diversity which can be produced from alternative systems. Output diversity is
often regarded as an inherently desirable quality of PCG systems, and ERA
provides a way of comparing the output diversity of alternative systems by
comparing which generative space is more spread out in the ERA plot. As a
result it commonly appears in novel PCG research to support arguments about
how PCG systems compare in terms of the diversity present in their generative
spaces (horn2014; jadhav2021; zakaria2021; alvarez2022).

Though ERA is still used in contemporary research in its original form, there
have been several evolutions on the underlying concept to improve its
applicability and functionality. Adam Summerville introduced the concept of
using corner plots to visualise multiple metric pairs simultaneously,
overcoming the prior limitation of only viewing two metrics at a time
(summerville2018). They also introduced the use of the e-distance metric for
comparing the distributions of generative spaces in n-dimensional space. Cook
et al developed the Danesh system for conducting automatic ERA within Unity,
as well as for identifying the trends in ERA across alternative
parameterisations of the same generator using what they termed Randomised ERA,
or RERA (cook2021). Kremenski et al introduced ‘expressive range coverage
analysis’ to visualise trends in how designers interacted with a generative
system over time(kreminski2022). Withington et al explored the application of
dimensionality reduction algorithms to game level representations to produce
ERA style plots without the need of calculating the underlying
metrics(withington2022).

###  3.2. Generative Space, Possibility Spaces and ERA

An important concept for this work, but also ERA more broadly, is that between
generative space and possibility space. These terms were popularised by Mike
Cook in their work (cook2022a) and they help to highlight the strengths and
weaknesses of ERA, as well as the potential benefits of this work’s approach.
Unfortunately the definition of these terms is not universally accepted, and
there are high profile works that use generative space and possibility space
interchangeably(guzdial2020). The distinction remains very conceptually useful
though, and we hope the terminology will become more ubiquitous in future.

As we have previously discussed, the generative space of a PCG system refers
to the volume that contains every artefact that a specific system could
generate. In contrast, possibility space refers to the volume that contains
every artefact that could be represented by any generator for that domain.
Typically, a generative space is a smaller volume contained within the larger
possibility space. When a game designer is choosing between alternative
generative approaches for a certain task, or tuning the parameters of a chosen
generative approach, this can be conceptualised as the choosing of an optimal
generative space for a given possibility space. Similarly, when a designer is
comparing two generators in terms of their output, they are comparing two
different localisations of generative spaces within the same possibility
space.

This terminology gives us a lens with which to understand ERA. ERA plots
simultaneously produce a 2D dimensional projection of both a PCG systems
generative space, and also the domains underlying possibility space. Just as
the n dimensional volume of the generative space is compressed into a 2D
metric determined plot, so is the potentially vastly more complex underlying
possibility space. The utility of an ERA visualisation of a generative space
is heavily influenced by the impact of the selected metric pair on the
projection of the possibility space. When conducting ERA it is very possible
to decide on metric pairs which mean that the majority of the plot does not
contain good content, or is not even within the possibility space of content
due to the majority of metric values being mutually incompatible. The
intention with this work’s metric pair selection criteria is that they can
help designers use ERA to be a more useful projection of the underlying
possibility space, and therefore a more useful projection of a specific
generative space.

###  3.3. Metric Selection for ERA

The main decision a designer has to make when applying conventional ERA to a
generative space is the choice of the pair of metrics to visualise the space.
This choice dictates the types of diversity that will be captured in the
visualisation, as well as potentially what types of diversity are ignored. The
distribution of any metric that is of interest but is not included in the plot
is unknown. Given three metrics of interest, a plot could show a high level of
diversity in terms of metrics 1 and 2, while concealing homogeneity in metric
3 for example. If ERA is being deployed to allow a designer to qualitatively
explore the distribution of a generative space in terms of two metrics that
they know are of interest, then no choice needs to be made. However, if ERA is
being deployed to argue that a generator produces more generally diverse
content than an alternative, or if there are many metrics of interest for a
designer, then this choice is less obvious.

In general there are two approaches that are taken for selecting metrics.
Firstly, researchers can reuse metrics that have been used in prior
literature, most commonly the two metrics from Smith et als original paper. In
that paper Smith et al introduced a ‘linearity’ metric, which is a heuristic
for how flat a level is, and ‘leniency’, a heuristic for level difficulty.
Though these two metrics were designed to be insightful within the context of
their specific level generation system, they have since been widely reused
within PCG research for conducting ERA (horn2014; alvarez2022; jadhav2021),
and also as metrics to intentionally aim to find diversity in as part of
Quality-Diversity search based PCG (alvarez2019; sarkar2021). While there is
nothing inherently problematic about reusing metrics in this way, the lack of
justification given for their reuse in novel domains means researchers risk
using suboptimal metrics for their purposes.

We also note that while the names linearity and leniency appear frequently to
describe metric choices, the underlying calculation to arrive at them is often
altered to fit with the specific game domain of interest. Leniency was first
calculated as a score based on the number of jumps, moving platforms and
enemies in the level (smith2010), but has since been calculated as a
normalised rating on the average difficulty of every point of the level
(horn2014), or more simply as count of enemies subtracted from count of
rewards (jadhav2021). Linearity similarly was originally calculated as a
linear regression on the gaps between the highest points of the level for each
horizontal position and the centre line (smith2010; shaker2012), but has since
been calculated in very different ways in different contexts, such as using a
normalised count of the number of paths between doors in the case of Alvarez
et al’s work on the Evolutionary Dungeon Designer (EDD) (alvarez2022). These
alternative calculations usually make pragmatic sense. The EDD for example
creates levels that are viewed top down and contain no deadly gaps in the
scenery which makes the original formula inappropriate. However, the degree of
variation in even these ostensibly similar metric pairs for ERA highlights, in
our view, the potential value of heuristics for making a more informed choice.

The alternative approach for metric selection is to not select at all, and
instead visualise the plots for all metric pairs that might be of interest.
This longer list of metrics can again be sourced inclusively from the prior
literature on similar game domains (alvarez2022; hervé2021), or a list can be
designed by a game domain expert (kreminski2022). This list of n metrics can
then be visualised using ERA of all possible pairs, commonly using
Summerville’s corner plot concept(summerville2018). However, the use of
multiple metrics within ERA has the weaknesses of both sacrificing the
simplicity and readability of a singular ERA plot, as well as relying on the
presumption that all of the metric pair plots are equally informative.

There have been efforts to add more robust justification for choosing
individual metrics within ERA, most commonly in the form of validating whether
or not commonly used metrics correlate with human perceptions of quality and
diversity. Summerville et al explored the power of common metrics when
predicting player experience and found strong correlations (summerville2017).
Marino et al conducted a similar study and found much more limited correlation
(marino2015). Herve & Salge explored the relationship between common metrics
and expert evaluations of Minecraft maps (hervé2021). These works have been
invaluable for confirming or refuting that specific metrics are capturing
forms of quality or diversity which are of real interest to designers.
However, as their focus has been on the informative power of individual
metrics their utility in aiding the selection of metric pairings is limited.
As we shall see in Section 4, applying ERA with two metrics which are strongly
correlated with human perceptions does not ensure the plot will be useful. It
is still possible for two such metrics to produce an unhelpful projection of
the possibility space, one in which for example, the majority of the space is
undesirable or inaccessible.

###  3.4. Mario AI Benchmark

This work’s pilot experiments make use of the Mario AI Benchmark, a widely
used AI research platform used for the development and evaluation of AI agents
for level playing and level generation. While we share the sentiment of Mike
Cook (cook2022) that Super Mario has received an unreasonable amount of focus
within PCG literature as a test bed, using it within this work is justified by
the amount of prior work that has applied ERA specifically to Mario level
generators. This gives us both a wide variety of previously used metrics to
draw upon, as well as making any insights we gain into optimal metric
selection more relevant to the research community.

We use the level corpuses generated as part of Horn et als 2014 work
(horn2014), generated from a diverse set of nine Mario level generators which
are included in the latest version of the benchmark, which is bundled with 14
levels from the original Mario game giving a total of 9014 levels.. These
level sets are ideal for this research for several reasons. They all use the
same encoding system and are represented by 2D matrices of characters, in
which each character represents a tile type. This means we can use the same
metric calculation approaches across all levels in the corpus. Additionally,
prior research has confirmed that the sets from the alternative generators are
meaningfully diverse in a way that has been detectable using alternative
visualisation approaches(horn2014; withington2022).

To automatically evaluate the levels we make use of Robin Baumgarten’s
A*-based Mario level playing agent, the winning agent in the first Mario AI
competition(togelius2010), which is included in the benchmark and is often
used in PCG research to confirm that a level is completable (fontaine2021;
cernygreen2020).

##  4\. Metric Pair Selection Criteria

As we have seen there is little consistency in how ERA has been applied in
terms of the metrics selected, and that there is typically limited
justification given for the selection of certain metrics and pairings over
alternatives. However, we argue that there are characteristics of useful ERA
plots that we can infer from its design, characteristics that can be used as
the basis for heuristics for guiding metric pair selection without the need
for human validation. In this section we introduce and discuss the relevance
of these characteristics, as well as how they can be operationalised to
produce metric pair selection criteria. All of the selection criteria we
introduce are intended to be calculated for a given pair of metrics, selected
from a larger pool of candidate metrics. For a set of n metrics that may be of
interest, there are n(n-1)/2 potential pairs of metrics which could be
selected to produce ERA plots. The goal is for these selection criteria to
give a quantifiable justification for selecting one of these potential pairs
over the alternatives.

###  4.1. Criteria 1: Fitness Independence

The first selection criteria for metric pairs that we argue to be valuable is
that of fitness independence. What we mean by fitness independence is that fit
or desirable levels are as evenly distributed within the 2D ERA projection of
the underlying possibility space as possible (See Figure 1 for an
illustration). Fitness is a term popularised by the field of search-based
algorithms to refer to quantifiable measures of how good or strong an artefact
is. It is important to note that this criteria is naturally only applicable to
domains where there is a concept of fitness, though in the domain of
generating game levels such a concept is extremely common due to how easy it
is for PCG systems to generate game levels that are uncompletable, or only
completable to a point. Fitness in PCG for game levels is often contextualised
as a more binary concept (khalifa2018), and we consider that this fitness
independence selection criteria would be useful both for domains with either
continuous or binary fitness concepts. Either way, for exploration within ERA
to be desirable, there needs to be a good distribution of acceptable content.

We argue that this criteria is implicitly desirable for an ERA visualisation
to be as informative as possible. With it, alternative generative spaces can
be directly compared in terms of their relative explorations of the space, or
an individual generator can be tuned to locate its generative space wherever
the designer wishes within the space, while being confident that each location
in the space is equally likely to be valuable. Contrastingly, in
visualisations with low fitness independence exploration is likely to be
undesirable, and a designer would not feel as free to optimise towards any
chosen location of the space, as all of the fit levels would be tightly
clustered in the space. An ERA plot with low fitness independence could still
be useful to a designer confident that the two metrics represent the diversity
they are most interested in, but they would still need to be more wary of the
location of fit content within the space. High fitness independence reduces
the need to be concerned about the location of fit content, allowing
researchers and designers to focus more just on the visualisation.

The method for calculating the fitness independence that we explore in this
paper is to discretise the ERA plot into an evenly sized grid, in which each
cell in the grid is defined by a range of metric values. Each level to be
visualised is then sorted into this grid. We can then calculate the average
fitness of each grid cell in terms of the levels placed there, and then
further calculate the average cell fitness for the entire grid. Low values for
a grid defined by a certain metric pair would therefore indicate that fit
levels were highly concentrated in specific cells of the grid with more of the
grid dominated by either empty cells or cells with exclusively low fitness
scores, whereas higher values would indicate a more even distribution.

Figure 1. Illustration of the possible effect of high and low fitness
independence on the sparseness of acceptable content in ERA space

###  4.2. Criteria 2: Mutual Correlation

The second selection criteria for metric pairs is that of mutual correlation.
What this means practically is that we want there to be a minimal or non-
existent predictive relationship between the values of one metric and the
other in its pair (See Figure 2 for an illustration of this criteria). This
criteria is important for two reasons. Firstly, to ensure that as much of the
plot is actually accessible, in other words that as much of the plot is within
the underlying possibility space of the domain. Secondly, it is important to
minimise the amount of redundant information within the ERA plot. In the case
that the value of one metric is highly predictive of the other, then one could
be discarded while maintaining the same amount of information about the
distribution of the generator’s levels. It would also likely make the majority
of the overall ERA plot inaccessible, as any paired values outside of a
tightly confined range would not be within the underlying possibility space or
artefacts. In the alternative desirable case in which there is minimal
correlation between the metric values in a selected pair we are instead
maximising the information gained within the plot, and also ensuring that more
of the plot is at least hypothetically possible.

Meeting this criteria makes the ERA plot using the pair more meaningful. With
the criteria met we can be confident that what appears to be lack of diversity
in a generative space means that a generator is exploring a small amount of a
possibility space, not that the projection of the possibility space itself is
heavily restricted. It also makes it more likely that a designer will be able
to distinguish between the properties of alternative generative spaces. The
stronger the mutual correlation between metrics, the more that any projected
generative spaces would necessarily overlap in the amount of the plot that
represented the restricted possibility space. As we shall discuss in the next
section on the third and final criteria, low mutual correlation between a
metric pair also increases the likelihood that one of them is correlated with
a given alternative metric, a feature we shall introduce as desirable shortly.

To calculate the mutual independence of metrics in our pilot studies we
calculate the linear correlation between the values found for each metric
using Spearman’s rank correlation coefficient, commonly referred to as
Spearman’s $\rho$. The stronger the correlation between the two metrics the
more the value of one determines the other, and the less explorable the full
ERA plot will be. We store the absolute value of Spearman’s $\rho$, as whether
the correlation is strongly positive or strongly negative we consider it to be
equally undesirable. The major disadvantage of this approach is that it does
not help us discover non-linear correlations between metrics which if they
were present would still be harmful to how explorable the plot is, something
we discuss further in Section 7. However it has the advantage of being far
simpler and less computationally intensive to calculate than alternative
methods for investigating non-linear correlation.

Figure 2. Illustration of the effect of low and high mutual correlation on the
location of a possibility space within an ERA plot, as well as the effect on
the location of generative spaces

###  4.3. Criteria 3: Alternative Metric Correlation

The third selection criteria we investigate in this work is the hardest to
conceptualise and calculate, but also arguably the most important. It is that
of the level of correlation with alternative metrics. What this means in
practice is that we want to minimise the existence of alternative but valuable
metrics to the pair used to create the plot, which are uncorrelated with the
plotted metrics (See Figure 3 for an illustration of high and low alternative
metric correlation with a single metric). The underlying goal is that the ERA
plot projection of the underlying possibility space is as information rich as
possible, and that what appears to be diversity or a lack of it represents
diversity or a lack of it in a more holistic way than one determined by just
the two metrics.

To understand why this criteria is valuable we shall use a short example case.
If we imagine we were interested in only three metrics for a set of generated
levels, called A, B and C. If we used A and B as the metric pair to conduct
ERA, there is a chance that the generative space could appear widely
distributed in the plot even if all content had highly homogenous values for
metric C. However, if high linear correlation was found between either metrics
A or B and metric C, then we could be confident that there was no risk of
this, and that what appeared as diversity in A and B also represented
diversity in C. This is what we aim to assess with this selection criteria,
except that we aim to find minimal independence between a selected pair and
all other metrics of interest, rather than just a specific alternative metric.
The higher the values for this criteria, the more we can be confident that
diversity in an ERA plot represents diversity for multiple metrics, and
therefore represents diversity in a more general sense than when the criteria
values are low.

To operationalise this criteria we use a similar approach to the mutual
independence criteria, except in this case high correlation values for
Spearman’s $\rho$ are desirable. For a candidate metric pair M1 and M2 we
calculate $\rho$ for the correlation between every alternative metric and M1
and M2 separately, giving two values for each alternative metric. We only
store the highest absolute value found for either M1 or M2 as it does not
matter which metric in the pair has the correlation, nor whether the
correlation is positive or negative. This is why the mutual correlation
criteria supports this one, as having two independent metrics increases the
likelihood of finding one that performs well for alternative metrics here.
This set of $\rho$ values for each other metric can then be divided by the
number of alternative metrics to give a mean average correlation score with
alternative metrics. The higher this score, the more inclusive an ERA plot is
of alternative forms of diversity.

Figure 3. Illustration of the effect of high and low alternative metric
correlation in terms of a single alternative metric, and the effect this can
have on the distribution of alt metric values in and ERA plot

##  5\. Experiment Design

In this section we introduce the design of our pilot experiments using this
approach for selecting ERA metric pairs, as well as explaining the reasoning
behind the key design decisions. The intention for these experiments is that
they demonstrate the potential utility of this work’s novel selection
criteria, as well as highlighting the strengths and weaknesses of our current
approach for calculating them.

###  5.1. Game Domains Assessed

The PCG generative spaces analysed all come from the Horn et al corpus of
generated levels (horn2014) that we discussed in more detail in Section 3.
This set of 1000 generated levels from 9 different generators, as well as 14
of the levels from the original Super Mario Bros is included in the current
version of the Mario AI Framework.

When calculating our selection criteria and evaluating which metric pair is
most appropriate for performing ERA, we focus on the complete assembled set of
9014 levels as we argue this will give a more representative view of the
underlying possibility space of Mario levels than could be obtained from any
individual generator in the set. This is a decision we discuss and critique in
more detail in Section 7, as while it gives a more nuanced view of the
possibility space it is also a less realistic implementation. In practice game
designers and researchers will rarely have access to multiple different
generators for a game domain to use to gain this kind of view.

###  5.2. Metrics Assessed

For each of the 9014 Mario levels in our dataset we calculated 18 metrics. For
the full list of metrics and how they are calculated see Table 1. The list of
18 metrics and how they were calculated were largely inspired by prior works
which used them or similar variants in ERA or QD-search. Metrics related to
block counts, or repeated patterns such as locations in which the agent can
stand are commonly used, as are counts or ratios of agent actions when
simulated play is involved in artefact evaluation (summerville2017;
hervé2021). Our MaxJumpAirTime and AverageY metrics are more novel.
MaxJumpAirTime captures the largest jump performed by the agent, as performing
a large jump can often be a memorable moment or difficulty spike within the
level. AverageY relates more to the level’s structure, capturing how the
average height of the agent during its run to get an idea of the average
height of standable terrain. As this work is focused on metric pair
evaluation, the design of the metrics themselves was not a focus. However we
did want to have an even amount of ‘structural’ metrics calculated from the
level’s structure itself, and ‘agent evaluated’ metrics calculated from the
play trace of an AI agent, to draw comparison between the presence of both in
high performing metric pairs.

Also extracted from the agent simulations was the amount of horizontal
progress the agent was able to make as a proportion of the whole level width.
This feature is often referred to as ‘playability’ or ‘fitness’ in the prior
work using the Mario AI benchmark(cernygreen2020; fontaine2021), though we
refer to it as playability as it is the more specific term. Playability is
calculated as a score from 0 to 1, where 1 indicates the agent was able to
reach the end of the level. This metric was not included as a candidate for
inclusion in metric pairs, as it was held back for use in the FI selection
criteria calculation.

Table 1. Metrics investigated and how they are calculated Type | Name | How Calculated
---|---|---
| Contiguity | +1 per solid tile adjacent to a solid tile
| Linearity | +1 per solid tile horizontally adjacent to a solid tile
| BlockCount | +1 per solid tile
| EnemyCount | +1 per enemy
Structural | RewardCount | +1 per tile containing a reward
| EmptyCount | +1 per empty tile
| PipeCount | +1 per tile containing a pipe block
| Density | Total places Mario can stand divided by level length
| ClearColumns | +1 per column containing only empty tiles
| Playability | Proportion of level the agent was able to complete
| JumpCount | +1 per jump action used by agent
| JumpEntropy | JumpCount divided by total actions
| Speed | Proportion of level navigated per second on average
Agent | TimeTaken | Time used by the agent for the run in seconds
Extracted | TotalEnemyDeaths | +1 per enemy who died during the run
| KillsByStomp | +1 per enemy killed by agent by jumping on them
| MaxJumpAirTime | Longest jump by agent in terms of number of frames in
| OnGroundRatio | Proportion of time agent spent on the ground
| AverageY | Average Y axis position of Mario during the run

###  5.3. Metric Selection Criteria

The main results calculated the three selection criteria explained in Section
4, namely Fitness Independence (FI), Mutual Correlation (MC) and Alternative
Metric Correlation (AMC). Each one gives a score between 0 and 1, with higher
scores considered more desirable for FI and AMC, and lower scores more
desirable for MC. These three criteria are calculated for each of the 153
unique metric pairs within the set.

Additionally we calculate the rank for each pair in terms of the three
criteria, which we can then use to identify the best N metric pairs for each
criteria. We also calculate the average rank for each pair across the three
criteria. This gives a heuristic for the overall performance of the pair, with
lower values indicating better general performance across the three metrics.
This average ranking heuristic has the weakness of equally weighting
performance across all criteria when their relative importance has not yet
been established, something we discuss in more detail in Section 7.

###  5.4. Agent Simulation

The agent we use to calculate the agent-evaluated metrics, as well as the
playability of the levels is Robin Baumgarten’s A* Mario agent which we
introduced in Section 3. This agent has been widely used to assess the quality
and characteristics of Mario levels in prior research up to the present day,
though it does have significant limitations such as being unable to backtrack.

As a result of the variance possible in the agent’s decision making we run the
agent on each level five times and then calculate the average results across
these five runs for all of the agent extracted features.

###  5.5. Heatmap Generation and Distribution Analysis

In order to calculate the fitness independence selection criteria, as well as
some of the results visualisations, the expressive range plot needs to be
divided into a grid so that the average fitness of levels in each grid
location can be calculated. Our approach for this is to divide each metric
pair ERA plot into a 20 by 20 grid, with the range of values for each cell
determined by subtracting the minimum value found for a metric from the
maximum value and then dividing by 20. This approach ensures that all levels
can be placed within the plot, and that no metric specific ranges have to be
chosen. However, it also means that the distribution within the plot is
heavily influenced by the presence of outlying metric values, something we
discuss further in Section 7.

###  5.6. Data and Visualisations Presented

In the following section we present several results tables and visualisations
to illustrate the important and interesting findings from these experiments.
First, we present the ERA visualisations of the highest and lowest ranked
metric pairings in terms of their average rank across the three selection
criteria. These are presented both with the playability heatmaps to show how
playable levels are distributed in the space, along with ERA visualisations of
the full level corpus with different generators highlighted in different
colours. These are presented to highlight some of the features of strong and
weak metric pairings when using our selection criteria.

Second, we present top 5 ranked metric pairings for each selection criteria in
terms of their values, as well as the top 5 ranked pairings in terms of their
average rank across the three other ranks. These are primarily presented to
demonstrate what the final data obtained from applying the selection criteria
looks like, and to highlight patterns in the ranks obtained by high performing
metric pairs across the three criteria. This set of 20 pairings is also used
as our set of high performing metric pairs, to be used in the final two
figures which illustrate patterns found in metric pairings which obtained high
rankings in at least one criteria.

Finally, we present both a histogram of the appearance rate of each individual
metric in the list of top high performing pairings, as well as a table
enumerating the count of pairings in this list which were either both agent
selected or structural metrics, or were composed of one of each. These are
presented to highlight valuable trends in the makeup of high performing pairs,
both to present avenues for future work but also to highlight weaknesses in
the presented implementation.

###  5.7. Hardware & Software Used

In order to extract and save the metrics for each of the Mario levels included
in the experiments, custom Java code was used to interface with the Mario AI
Benchmark. Code available at [Anonymised for submission]. The calculation of
the selection criteria themselves as well as the generation of visualisations
was done in Python, with code available at [Anonymised for submission].

All software was run on a Dell laptop with an i5-10310U CPU with 16.0 GB of
RAM. The process of metric extraction from the Mario AI Benchmark took
approximately 12 hours, and the calculation of the selection criteria took
approximately 1 hour and 30 minutes.

##  6\. Results and Discussion

Table 2. Top 5 metric pairs for each selection criteria, as well as for the average rank across all three criteria. All values rounded to 3sf |  | FI | MC | AMC |
---|---|---|---|---|---
| Metric Pair | Score | Rank | Score | Rank | Score | Rank | Average Rank
| Density-JumpCount | -0.115 | 1 | -0.115 | 30 | 0.518 | 77 | 36
| EnemyCount-JumpCount | 0.506 | 2 | 0.506 | 101 | 0.521 | 74 | 59
| Density-AverageY | -0.0154 | 3 | -0.0154 | 3 | 0.551 | 39 | 15
| JumpCount-TotalEnemyDeaths | 0.655 | 4 | 0.655 | 130 | 0.541 | 47 | 60.3
Top 5 FI Rank | PipeCount-TotalEnemyDeaths | 0.594 | 5 | 0.594 | 122 | 0.5831 | 15 | 47.3
|  |  |  |  |  |  |  |
| EmptySpaceCount-MaxJumpAirTime | 0.0564 | 153 | -0.00253 | 1 | 0.583 | 17 | 57
| Speed-MaxJumpAirTime | 0.124 | 132 | 0.0149 | 2 | 0.4521 | 128 | 87.3
| Density-AverageY | 0.496 | 3 | -0.0154 | 3 | 0.551 | 39 | 15
| Density-ClearColumns | 0.296 | 67 | 0.0166 | 4 | 0.286 | 148 | 73
Top 5 MC Rank | KillsByStomp-AverageY | 0.321 | 53 | 0.0168 | 5 | 0.536 | 54 | 37.3
|  |  |  |  |  |  |  |
| EmptySpaceCount-JumpCount | 0.122 | 133 | -0.535 | 106 | 0.610 | 1 | 80
| BlockCount-TotalEnemyDeaths | 0.336 | 44 | 0.379 | 83 | 0.609 | 2 | 43
| BlockCount-AverageY | 0.321 | 52 | -0.629 | 125 | 0.606 | 3 | 60
| Contiguity-TotalEnemyDeaths | 0.334 | 45 | 0.263 | 54 | 0.606 | 4 | 34.3
Top 5 AMC Rank | BlockCount-JumpCount | 0.359 | 30 | 0.478 | 99 | 0.604 | 5 | 44.7
|  |  |  |  |  |  |  |
| Density-AverageY | 0.496 | 3 | -0.0154 | 3 | 0.551 | 39 | 15
| PipeCount-Density | 0.365 | 26 | 0.0531 | 20 | 0.574 | 22 | 22.7
| Density-JumpEntropy | 0.456 | 6 | -0.104 | 29 | 0.530 | 60 | 31.7
| ClearColumns-AverageY | 0.396 | 17 | -0.0336 | 12 | 0.522 | 71 | 33.3
Top 5 Average Rank | Contiguity-TotalEnemyDeaths | 0.334 | 45 | 0.263 | 54 | 0.606 | 4 | 34.3
Figure 4. Heatmap of highest ranked metric pair, Density & AverageY, in terms of average rank. Presented with both the fitness heatmap, and the derived ERA plot when using the metrics to project the level corpus, with each generator highlighted in a different colour Figure 5. Heatmap of lowest ranked metric pair, Contiguity & BlockCount, in terms of average rank. Presented with both the fitness heatmap, and the derived ERA plot when using the metrics to project the level corpus, with each generator highlighted in a different colour Table 3. Frequency of alternative combinations of structural and agent extracted features across the 5 top ranked pairs for each criteria and for average rank | Count
---|---
Pair Category | Avg Rank | FI | MC | AMC | Total
Structural-Structural | 1 | 0 | 1 | 0 | 2
Structural-Agent | 4 | 4 | 2 | 5 | 15
Agent-Agent | 0 | 1 | 2 | 0 | 3
Figure 6. Relative appearance frequencies of each metric in the top 5 pairs
for each selection criteria and average rank. Structural metrics colored in
green, agent-extracted in orange

In this section we discuss the approach we have presented, as well as the
insights into it that were gained from our pilot experiments. Overall we find
many reasons to be enthusiastic about the approach we have presented. The
process for calculating the section criteria for metric pairs is
straightforward and relatively fast, and the insights that can be gained from
using it can be both useful and also unobvious. We are confident that using
them would likely lead to more informative and fair ERA plots than making an
uninformed choice. To get an idea of this we can look at the best and worst
performing metric pairs in terms of their average ranks in Figures 4 and 5.
Not only is it easier to distinguish between the generative spaces of the
different generators in the best performing metric pair in Figure 4, but as a
result of the FI and AMC selection criteria one can be more confident that
what appears to be diversity is both useful in terms of its fitness, and also
represents diversity across multiple alternative metrics.

One of the interesting patterns in our pilot results is the relationship, or
lack of one, between the performance of metric pairs across the three
criteria. As we can see in Table 2, metric pairs that ranked highly in one
selection criteria often ranked very poorly for other selection criteria. We
expected the FI and MC criteria to be correlated to an extent as they both
prioritise the wide distribution of levels in the space. We also expected MC
and AMC criteria to be correlated as the more uncorrelated the two metrics are
the more likely it is that one of them will have a strong correlation with a
given alternative metric. These correlation effects do not appear strong
enough in this domain to ensure the existence of metric pairs which are high
performing in all criteria. For an example of this, the 2nd ranked metric pair
in terms of average rank, PipeCount-Density, does not appear in the top 20 of
ranked pairs for any individual criteria. It is possible however, that
alternative untested metrics or improved designs of tested ones could attain
good performance on all three criteria.

As we can see in Figure 6, the individual metrics we investigated were widely
distributed in terms of how often they appeared in high ranked metric pairs.
We looked at the Top 5 ranked pairs for each criteria and the Top 5 averaged
ranked, giving a total of 20 high performing pairs and 40 constituent metrics.
At the high end was the density metric which appeared in 7 of 20 possible
pairs. At the low end, four metrics did not appear at all. While this sample
is too small and domain specific to make any general statements, the level of
skew is interesting and implies that we as researchers should be more cautious
about making an unconsidered choice about which metrics to use.

A thorough investigation of what the underlying causes of individual metrics
underperforming is outside the scope of this work. However some causes were
immediately apparent, such as the damaging effect of outliers on how we
calculated the FI selection criteria. Extreme outliers cause the non outlying
levels to be concentrated in the heatmap, as the heat maps dimensions are
determined by the total range of the metric values, which harms their level of
distribution and their subsequent score. This was most obvious in the
MaxJumpAirTime metric, in which one level Mario was able to achieve a value of
148 frames, well outside of the rest of the values for the remaining hundreds
of levels whose values were found in the range of 0 to 62. This high value was
achieved due to the unique design of the level which was so highly populated
by enemies that the most efficient route for the A* agent was to bounce on
them for the majority of the level, leading to long periods of time not spent
on the ground. This directly caused the low ranking of any metric pair using
it in terms of the FI score, as we can see in Table2 with the otherwise
strongly performing ‘EmptySpaceCount-MaxJumpAirTime’ metric pair which is top
ranked for the MC selection criteria. While this is interesting, it should
also caution us against using this method of calculating Fitness Independence
again, without further considering how to handle outliers.

An interesting finding was the prevalence of metric pairs involving a
combination of structural and agent evaluated features in high ranking pairs.
As we see in Table 3 15 out of the 20 high scoring pairs conformed to this
format. Statistically there is a bias towards structural-agent pairs, as due
to us not testing pairs of the same metric a slight majority of 81 of the 153
total pairs were structural-agent pairs. However the skew was more extreme
than can be explained by this bias, and it also makes intuitive sense, as the
way a game playing agent interacts with the level as a series of required
actions is very different to structurally analysing it as a matrix of tiles.
We expect this to mean they are less likely to be mutually correlated, and
therefore more likely to score well on the AMC criteria. This result suggests
that ERA implementations could benefit from calculating each metric in their
pairs using differing approaches to increase the likelihood that the metrics
are performant in terms of our selection criteria. Though we should be careful
to not extrapolate too far from this single data point, as it could easily be
a quirk of the specific metrics we selected to evaluate.

##  7\. Limitations and Future Work

While we are excited about the concept and high level approach we have
presented, it has several weaknesses and limitations that could be usefully
improved upon in future. Additionally there is much further work that could be
done to both expand on the ideas we have presented, and to generally advance
the research domain of understanding and visualising PCG system generative
spaces.

An important avenue of future work is to explore alternative ways of
calculating the selection criteria that were introduced in this paper. Rank
correlation calculated using Spearman’s Rho for the MC and AMC criteria is
fast and efficient, but it has the severe limitation of only detecting
monotonic relationships between metrics, and could not be able to detect more
complex non-linear relationships. Future work could usefully explore more
complex but computationally intensive alternatives such as the use of
smoothing splines. There are also alternatives for calculating the FI metric.
A simple alternative could be to calculate the maximum level fitness found in
a grid cell rather than the average which could be fairer and more valid, as
what we are interested in with this criteria is that it is possible to find
levels widely within the ERA plot, not whether a high proportion of them have
already been found.

This area could also benefit highly from further exploration and introspection
on what additional selection criteria would be valuable for metric pairs. One
which could be valuable but was outside of scope for this paper is that of
emergence, meaning that values for metrics are not determined by how the
generator is parameterised. This was argued as important by Smith et al in the
paper that first introduced ERA and we think it would still be valuable for
making ERA more informative, as well as making comparisons between generative
spaces fairer. Opperationalising this criteria would be complex as one would
need sets of levels generated from multiple parameterisations of the same
generator, but future work could explore an approach similar to RERA,
developed by Cook et al for the Danesh platform (cook2021).

A conceptual weakness of both our approach and ERA more generally, is the
unknowability of the underlying possibility space. As we discussed in Section
3.2 the process of tuning a PCG system can be thought of as trying to optimise
the size and location of the system’s generative space within the wider
possibility space of possible content. When ERA is being used to quantify the
search space exploration of a generator, what researchers want to know is how
much of the underlying possibility space is being explored. However, there has
been little prior work which aims to understand or quantify the possibility
spaces which encompass generative spaces. Future work to explore methods for
getting a PCG system agnostic view of underlying possibility spaces could be
highly beneficial for both robust ERA, but also PCG research in general.
Quality-Diversity based PCG approaches (gravina2019) provide a possible avenue
for this as they are well optimised for maximising exploration. As they also
create projections of the search space using metrics in an identical way to
ERA there would need to be careful consideration of what metrics to choose.
This work’s selection criteria could potentially be deployed for selecting
metrics to increase the possibility space exploration of QD search, something
we intend to investigate in future.

A practical weakness of the approach we have presented is simply the time it
takes to identify the best metric pairs. While the computation time was
relatively short, it obviously takes far longer than conducting the ERA
itself. This makes it substantially less appealing for practical ERA
applications, and would only get less appealing as the number of candidate
metrics increased, though increasing the number of candidates would also
arguably increase the value of our selection criteria by helping a designer to
filter them. In future work we would like to explore whether metrics can be
found which are highly performant across multiple game domains of similar
genre. This could synergise well with papers discussed previously which
explored the relationship between metrics and player perceptions (marino2015;
summerville2017; hervé2021). Metric pairs that scored highly in multiple
domains using our criteria, while also correlating with human perceptions of
quality or diversity would have the potential to be robustly useful in novel
PCG domains without the need for evaluation.

##  8\. Conclusion

In this paper we have presented our approach for enhancing implementations of
ERA through the use of metric selection criteria to enable designers to make a
more informed choice on which pair of metrics is most appropriate for their
domain. We also presented our implementation of these criteria and the results
and insights we obtained from applying them to a pre-existing corpus of
generated levels. We find that using these criteria can give valuable insights
on how best to produce ERA visualisations of a domain, as well as how best to
design metrics for conducting ERA. We hope that this work might form the basis
of future work exploring novel methods for selecting the best pairs of metrics
for expressive visualisation within PCG.


[a,e]Ryan Bignell

# Reconstructed (charm) baryon methods at finite temperature on anisotropic
lattices

Gert Aarts  Chris Allton  M. Naeem Anwar  Timothy J. Burns  Benjamin Jäger
Jon-Ivar Skullerud

###### Abstract

Reconstructed-correlator methods have been used to investigate thermal effects
in mesonic correlation functions in a fit-independent manner. This technique
has recently been extended to the baryonic sector. In this work different ways
of implementing this approach for baryon correlators are examined. Using both
real and synthetic data it is found that for heavy baryons, such as the
$\Xi_{cc}(ccu)$ baryon, different choices are equivalent and that for the
lighter nucleon the effect of different implementations is minimal. Further
comparison to the so-called double ratio using the FASTSUM Generation 2L
thermal ensembles shows that reconstructed-correlator ratios and double ratios
contain nearly identical quantitative information.

##  1 Introduction

In this contribution we examine the reconstructed-correlator technique [1, 2,
3, 4] applied to baryonic correlation functions. This technique is motivated
by the spectral relation for fermionic correlators,

$\displaystyle
G\\!\left(\tau;T\right)=\int_{-\infty}^{\infty}\frac{d\omega}{2\pi}\,K_{F}\\!\left(\tau,\omega;T\right)\rho\\!\left(\omega;T\right),$
(1)

where the kernel $K_{F}\\!\left(\tau,\omega;T\right)$ has a known analytical
form and temperature dependence, while the spectral function
$\rho\\!\left(\omega;T\right)$ is not known and is the item of interest, in
particular its temperature dependence. By using the reconstructed correlator,
one can examine changes in the spectral function
$\rho\\!\left(\omega;T\right)$ separate from the changes in the kernel
$K_{F}\\!\left(\tau,\omega;T\right)$ as the temperature is changed.

In particular we consider methods which enable the use of this technique for
two (fixed-scale) ensembles with different temporal extents $N_{0,1}$ — and
hence different temperatures $T_{0,1}=1/(a_{\tau}N_{0,1})$, with $a_{\tau}$
the temporal lattice spacing — which do not align with the odd factor of
$m=N_{0}/N_{1}=T_{1}/T_{0}$ required [4]. Subsequently we compare this
technique to the “double correlator ratio” method used in Refs. [3, 4] using
real lattice QCD data from the FASTSUM thermal ensembles [5, 3]. The double
ratio is defined as

$\displaystyle R\\!\left(\tau;T,T_{0}\right)$
$\displaystyle=\left.\frac{G\\!\left(\tau;T\right)}{G_{\rm
model}\\!\left(\tau;T,T_{0}\right)}\right/\frac{G\\!\left(\tau;T_{0}\right)}{G_{\rm
model}\\!\left(\tau;T_{0},T_{0}\right)},$ (2)

where $G\\!\left(\tau;T\right)$ is the lattice correlator at temperature $T$
and $G_{\rm model}\\!\left(\tau;T,T_{0}\right)$ a model correlator at
temperature $T$ informed by the physics at a lower temperature $T_{0}$, which
in practice is the mass of the ground state at the lowest temperature
available.

##  2 Ensemble and correlator details

We use the anisotropic “Generation 2L” thermal ensembles of the FASTSUM
collaboration [5, 3], consisting of $2+1$ flavours of Wilson fermions and a
Symanzik-improved anisotropic gauge action, following the Hadron Spectrum
Collaboration [6]. Full details of the action and parameter values can be
found in Refs. [5, 3]. Ensembles are generated using a fixed-scale approach,
such that the temperature is varied by changing $N_{\tau}$, as
$T=1/\\!\left(a_{\tau}N_{\tau}\right)$, see Table 1.

Table 1: FASTSUM Generation 2L ensembles. The lattice size is $32^{3}\times N_{\tau}$, with temperature $T=1/\\!\left(a_{\tau}N_{\tau}\right)$. The spatial lattice spacing is $a_{s}=0.11208\\!\left(31\right)$ fm, the renormalised anisotropy $\xi=a_{s}/a_{\tau}=3.453(6)$ [7, 5] and the pion mass $m_{\pi}=239(1)$ MeV [8]. We use $\sim 1000$ configurations and eight (random) sources for a total of $\sim 8000$ measurements at each temperature. The estimate for $T_{\rm pc}$ comes from an analysis of the renormalised chiral condensate and equals $T_{\rm pc}=167(2)(1)$ MeV [5, 3].  $N_{\tau}$ | 128 | 64 | 56 | 48 | 40 | 36 | 32 | 28 | 24 | 20 | 16
---|---|---|---|---|---|---|---|---|---|---|---
$T\,\\!\left(\text{MeV}\right)$ | 47 | 95 | 109 | 127 | 152 | 169 | 190 | 217 | 253 | 304 | 380

Baryon correlators are of the form

$\displaystyle
G^{\alpha\alpha^{\prime}}(x)=\Braket{\mathcal{O}^{\alpha}(x)\overline{\mathcal{O}}^{\alpha^{\prime}}(0)},$
(3)

where $\overline{\mathcal{O}}=\mathcal{O}^{\dagger}\gamma_{4}$ and $\alpha$,
$\alpha^{\prime}$ are Dirac indices. The parity-projected correlation
functions at vanishing spatial momentum are [9, 10, 11]
$G_{\pm}(\tau)=\mbox{tr}\,P_{\pm}G(\tau),$ where $P_{\pm}=(1\pm\gamma_{4})/2$.
These are related as [10] $G_{\pm}(\tau)=-G_{\mp}(1/T-\tau),$ implying that
the forward- (backward-) propagating states of $G_{+}(\tau)$ are states with
positive (negative) parity. The three-quark operators used follow Refs. [12,
13] and are described explicitly in Ref. [4]. In the following we focus upon
the $\Xi_{cc}\\!\left(ccu\right)$ as an example of a heavy baryon and the
nucleon $N\\!\left(uud\right)$ as the lightest baryon.

##  3 Reconstructed Correlator

Here we give a brief recap of the method [4]. The spectral relation for
baryonic correlators is given in Eq. (1) and the fermionic kernel reads [10]

$\displaystyle
K_{F}\\!\left(\tau,\omega;T\right)=\frac{e^{-\omega\tau}}{1+e^{-\omega/T}}.$
(4)

We wish to arrive at an expression relating the correlator at a higher
temperature $T$ to one at a lower temperature $T_{0}$. To do so, we
temporarily switch to lattice units, such that $T=1/N_{\tau}$, $T_{0}=1/N_{0}$
and $N_{0}/N_{\tau}=m>1$, and introduce the identity, relevant for the
fermionic case,

$\displaystyle 1+\text{e}^{-\omega mN_{\tau}}=\\!\left(1+\text{e}^{-\omega
N_{\tau}}\right)\,\\!\left(\sum_{n=0}^{m-1}\\!\left(-1\right)^{n}\,\text{e}^{-n\omega
N_{\tau}}\right).$ (5)

This identity requires that $m$ is integer and odd. The fermionic kernel can
hence be expressed as

$\displaystyle K_{F}(\tau,\omega;$
$\displaystyle\,1/N_{\tau})=\frac{\text{e}^{-\omega\tau}}{1+\text{e}^{-\omega
N_{\tau}}}=\sum_{n=0}^{m-1}\frac{\\!\left(-1\right)^{n}\,\text{e}^{-\omega\\!\left(\tau+n\,N_{\tau}\right)}}{1+\text{e}^{-\omega
mN_{\tau}}}\quad$
$\displaystyle=\sum_{n=0}^{m-1}\\!\left(-1\right)^{n}\,K_{F}\\!\left(\tau+n\,N_{\tau},\omega;1/(mN_{\tau})\right),$
(6)

The kernel has therefore been expressed as a summation over a kernel with a
longer time extent $mN_{\tau}>N_{\tau}$.

Inserting this resummation into the spectral relation (1) relates a correlator
at $T=1/N_{\tau}$ to one at a lower temperature
$T_{0}=1/N_{0}=1/\\!\left(m\,N_{\tau}\right)$, assuming that the spectral
content, i.e. $\rho\\!\left(\omega\right)$, is unchanged. This yields the
reconstructed correlator for fermions

$\displaystyle
G_{\text{rec}}\\!\left(\tau;1/N_{\tau},1/N_{0}\right)=\sum_{n=0}^{m-1}\\!\left(-1\right)^{n}\,G\\!\left(\tau+n\,N_{\tau};1/N_{0}\right).$
(7)

If we now switch back to denoting the temperatures with $T$ and $T_{0}$, the
relationship becomes explicitly

$\displaystyle
G_{\text{rec}}\\!\left(\tau;T,T_{0}\right)=\sum_{n=0}^{m-1}\\!\left(-1\right)^{n}\,G\\!\left(\tau+n/T;T_{0}\right).$
(8)

As $m=N_{0}/N_{\tau}=T/T_{0}$ must be an odd integer, the lattice sizes where
this technique can be used are limited in principle. In fact, none of the
ensembles in Table 1 have this odd integer relation with respect to the
ensemble at the lowest temperature ($N_{0}=128$). To get around this
limitation, we may consider adding or removing points from the zero-
temperature correlator to ensure such an odd-integer ratio. This should be
done in such a way to not affect the physics encoded in the correlator. One
may consider “padding” the correlator at the minimum of the correlator, with
either zeroes or with the minimum value of the correlator. This is done in
analogy to the mesonic case [3]. An alternative is removing points from the
correlator, again done symmetrically at the minimum of the correlator. In
contrast to the padding method, removing points can only be done when
$N_{0}>3N_{\tau}$, which corresponds to $N_{\tau}\leq 40$ for our ensembles.

To test these methods, we first apply them to a model correlator

$\displaystyle G_{\rm
model}\\!\left(\tau;T,T_{0}\right)=A_{+}K_{F}\\!\left(\tau,M^{+}_{0}\right)+A_{-}K_{F}\\!\left(\tau,-M^{-}_{0}\right)=\frac{A_{+}\text{e}^{-M^{+}_{0}\tau}}{1+\text{e}^{-M^{+}_{0}/T}}+\frac{A_{-}\text{e}^{M^{-}_{0}\tau}}{1+\text{e}^{M^{-}_{0}/T}},$
(9)

where $A_{\pm}$ is some normalisation chosen to be $1$ and $M^{\pm}_{0}$ are
the zero-temperature positive and negative parity masses of the
$\Xi_{cc}\\!\left(ccu\right)$ and the $N\\!\left(uud\right)$ ground states as
determined by us in Ref. [4] at temperature $T_{0}=47$ MeV ($N_{0}=128$). In
units of $a_{\tau}$, these are

$\displaystyle a_{\tau}M^{+}_{0}\\!\left(ccu\right)$
$\displaystyle=0.59642(85),\quad$ $\displaystyle
a_{\tau}M^{+}_{0}\\!\left(uud\right)=0.1740(17),$ $\displaystyle
a_{\tau}M^{-}_{0}\\!\left(ccu\right)$ $\displaystyle=0.6575(24),\quad$
$\displaystyle a_{\tau}M^{-}_{0}\\!\left(uud\right)=0.2501(65).$ (10)

Note that we neglect the uncertainties when constructing the model in Eq.
$\left(\ref{eqn:modelG}\right)$. The key assumption in this model is that the
width of the state is negligible.

Next, we consider the correlators at $N_{\tau}=40$ and $N_{0}=128$. These
temporal extents are chosen as they allow us to consider both adding and
removing points. Note that for the doubly-charmed baryon the correlator is
exponentially suppressed around the centre of the lattice,
$G(\tau=20a_{\tau})/G(\tau=0)\sim 10^{-5}$, whereas this is less so for the
nucleon, with $G(\tau=20a_{\tau})/G(\tau=0)\sim 10^{-2}$. Padding with zeroes
or with the minimum value are therefore expected to produce quite similar
results for the heavier state while somewhat more distinct results can be
expected for the lighter nucleon.

To compare the results of the reconstructed correlator, Eq. (8), at
$N_{\tau}=40$ with the actual (model) correlator at the same temperature, we
consider the ratio

$\displaystyle r\\!\left(\tau;T,T_{0}\right)=G_{\rm
rec}\\!\left(\tau;T,T_{0}\right)/G_{\rm model}\\!\left(\tau;T,T_{0}\right).$
(11)

Note that both correlators in the ratio are evaluated at $N_{\tau}=40$, but
one using the reconstruction method starting from the $N_{0}=128$ correlator
and one using the model parameters $M_{0}^{\pm}$ determined at $N_{0}=128$.

Figure 1: Ratio of reconstructed correlator to model correlator, $(1-G_{\rm
rec}/G_{\rm model})\times 10^{\alpha}$, with $\alpha=5$ for the nucleon (left)
and $\alpha=12$ for the $\Xi_{cc}$ (right). The three methods (padding with
the ‘Min’imum value, ‘Sub’traction and padding with ‘Zero’es) are explained in
the main text.

In Fig. 1 the ratio is shown using three methods to determine the
reconstructed correlator: by padding the $N_{0}=128$ correlator with the
minimal value (Min_Model), with zeroes (Zero_Model), and with points removed
(Sub_Model). Note that what is shown is ($1-G_{\rm rec}/G_{\rm model})\times
10^{\alpha}$, with $\alpha=5$ for the nucleon and $\alpha=12$ for the charmed
baryon, to highlight the very small difference between the reconstructed and
model correlators, on the order of $10^{-\alpha}$. The “padding with zeroes”
method behaves in an opposite manner to the other two approaches, due to the
addition of points with value ‘0’ rather than simply exponentially small.

Figure 2:  Left: Ratio of reconstructed correlator at $N_{\tau}=40$ to lattice
correlator at $N_{\tau}=40$ as well as the double ratio (2). A deviation from
1 shows changes in the spectral content. Right: Ratios of the reconstructed
correlator ratio with the double ratio. A deviation from 1 shows differences
between the two methods. All correlators are normalised to be equal to one at
the source.

##  4 Lattice Data

Following Ref. [4], we now apply these methods to the correlator data from the
FASTSUM ensembles, considering in particular the $\Xi_{cc}(ccu)$ and $N(uud)$
correlators. We compare the reconstructed correlator methods discussed above
with the double ratio introduced in Refs. [3, 4] and written in Eq. (2). As
stated, both the ratio with the reconstructed correlator and the double ratio
aim to examine whether changes due to the increase in temperature are present.
When constructing the double ratio, the statistical uncertainty comes from the
statistical uncertainty in the mass fit parameters in Eq.
$\left(\ref{eqn:masses}\right)$ as well as from the statistical uncertainty in
the correlators.

We explicitly compare these two methods in Fig. 2 at $N_{\tau}=40$, i.e. just
below the pseudocritical temperature. The left hand plot shows the
reconstructed and double ratios. For the heavier $\Xi_{cc}$, the results agree
within uncertainty. For the lighter $N$ there is a visible difference at later
times between the double ratio and each of the reconstructed correlator
ratios. The qualitative behaviour of interest is, however, still the same. To
further highlight any differences we take ratios of the ratios of correlators
shown in the left hand plot. The results are shown in the right hand plot.
Here we have removed any uncertainties normally used in the mass parameter
which would otherwise obscure any interesting behaviour. This shows that the
differences between the three reconstructed correlator methods are indeed very
small. Note that this test is different from the one in Fig. 1, as the
reconstructed correlator now uses real lattice data as input rather than a
model. Finally, the reconstructed and double ratios are in very good agreement
as well.

##  5 Summary

In this work, the reconstructed correlator method for baryons was considered.
In the “fixed-scale” approach to thermal lattice QCD, this allows a baryon
correlator at a higher temperature to be reconstructed from one at a lower
temperature, assuming that the spectral content is unchanged. Hence
reconstructed correlators can be used to examine the presence of thermal
effects in baryon spectral functions [4].

To use the technique for two ensembles with temporal extents $N_{0}$ and
$N_{\tau}$, the ratio $m=N_{0}/N_{\tau}=T/T_{0}$ should be an odd integer. As
this is usually not the case, some form of alteration of the correlator at the
lowest temperature should be performed. Here we considered three approaches:

  * •

Removing data points symmetrically from the minimum of the correlator;

  * •

Adding data points symmetrically at the minimum of the correlator, by

    * –

Adding zeroes;

    * –

Adding the minimum value of the correlator.

No significant difference is observed between these methods for both real and
synthetic test data. Since the correlator is exponentially suppressed in the
region where data points are added or removed, this is not unexpected. This is
in contrast to the case of light mesons, where artefacts due to padding with
zeroes can be observed [14], and more sophisticated methods must be applied
[15]. Padding with data points can be done at all temperatures in contrast to
removing points where $N_{0}>3N_{\tau}$ is required.

Next we compared the ratio of the actual and the reconstructed correlator with
the double ratio introduced in Refs. [3, 4]. For the latter, no padding or
subtraction is required. We found that both methods give comparable insight
into the possible temperature of the spectral content in the correlator.

## Acknowledgments

GA, CA, RB and TJB are grateful for support via STFC grant ST/T000813/1. MNA
acknowledges support from The Royal Society Newton International Fellowship.
RB acknowledges support from a Science Foundation Ireland Frontiers for the
Future Project award with grant number SFI-21/FFP-P/10186. This work used the
DiRAC Extreme Scaling service at the University of Edinburgh, operated by the
Edinburgh Parallel Computing Centre and the DiRAC Data Intensive service
operated by the University of Leicester IT Services on behalf of the STFC
DiRAC HPC Facility (www.dirac.ac.uk). This equipment was funded by BEIS
capital funding via STFC capital grants ST/R00238X/1, ST/K000373/1 and
ST/R002363/1 and STFC DiRAC Operations grants ST/R001006/1 and ST/R001014/1.
DiRAC is part of the UK National e-Infrastructure. We acknowledge the support
of the Swansea Academy for Advanced Computing, the Supercomputing Wales
project, which is part-funded by the European Regional Development Fund (ERDF)
via Welsh Government, and the University of Southern Denmark and ICHEC,
Ireland for use of computing facilities. This work was performed using PRACE
resources at Cineca (Italy), CEA (France) and Stuttgart (Germany) via grants
2015133079, 2018194714, 2019214714 and 2020214714.

## Software and Data

Correlation functions were generated using openQCD-hadspec [16], an extension
to openQCD-FASTSUM [17] for correlation functions. openQCD-FASTSUM was used
for ensemble generation [5, 3]. The analysis in this work, available at Ref.
[18], makes extensive use of the python package gvar [19]. Additional data
analysis tools included matplotlib [20, 21], numpy [22] and lsqfit [23]. The
zero temperature mass results of Ref. [4] were used in this study. Nucleon
mass fits used the scripts available at Ref. [24] with an input file available
at Ref. [18].

## Authors’ Contributions

Bignell: Data analysis, plot generation and primary manuscript production.
Jäger: Data production. Aarts, Allton, Anwar, Burns, Skullerud: Contributions
to analysis and physics interpretations of results, and draft of the
manuscript.

### Open Access Statement

For the purposes of open access, the authors have applied a Creative Commons
Attribution (CC BY) to any Author Accepted Manuscript version arising.

## References

  * [1] H.T. Ding, A. Francis, O. Kaczmarek, F. Karsch, H. Satz and W. Soeldner, _Charmonium properties in hot quenched lattice QCD_ , _Phys. Rev. D_ 86 (2012) 014509 [1204.4945].
  * [2] A. Kelly, A. Rothkopf and J.-I. Skullerud, _Bayesian study of relativistic open and hidden charm in anisotropic lattice QCD_ , _Phys. Rev. D_ 97 (2018) 114509 [1802.00667].
  * [3] G. Aarts, C. Allton, R. Bignell, T.J. Burns, S.C. García-Mascaraque, S. Hands et al., _Open charm mesons at nonzero temperature: results in the hadronic phase from lattice QCD_ , 2209.14681.
  * [4] G. Aarts, C. Allton, M.N. Anwar, R. Bignell, T.J. Burns, B. Jäger et al., _Non-zero temperature study of spin 1/2 charmed baryons using lattice gauge theory_ , 2308.12207.
  * [5] G. Aarts et al., _Properties of the QCD thermal transition with Nf=2+1 flavors of Wilson quark_ , _Phys. Rev. D_ 105 (2022) 034504 [2007.04188].
  * [6] R.G. Edwards, B. Joo and H.-W. Lin, _Tuning for Three-flavors of Anisotropic Clover Fermions with Stout-link Smearing_ , _Phys. Rev. D_ 78 (2008) 054501 [0803.3960].
  * [7] J.J. Dudek, R.G. Edwards and C.E. Thomas, _S and D-wave phase shifts in isospin-2 pi pi scattering from lattice QCD_ , _Phys. Rev. D_ 86 (2012) 034031 [1203.6041].
  * [8] D.J. Wilson, R.A. Briceno, J.J. Dudek, R.G. Edwards and C.E. Thomas, _The quark-mass dependence of elastic $\pi K$ scattering from QCD_, _Phys. Rev. Lett._ 123 (2019) 042002 [1904.03188].
  * [9] G. Aarts, C. Allton, S. Hands, B. Jäger, C. Praki and J.-I. Skullerud, _Nucleons and parity doubling across the deconfinement transition_ , _Phys. Rev. D_ 92 (2015) 014503 [1502.03603].
  * [10] G. Aarts, C. Allton, D. De Boni, S. Hands, B. Jäger, C. Praki et al., _Light baryons below and above the deconfinement transition: medium effects and parity doubling_ , _JHEP_ 06 (2017) 034 [1703.09246].
  * [11] G. Aarts, C. Allton, D. De Boni and B. Jäger, _Hyperons in thermal QCD: A lattice view_ , _Phys. Rev. D_ 99 (2019) 074503 [1812.07393].
  * [12] D.B. Leinweber, W. Melnitchouk, D.G. Richards, A.G. Williams and J.M. Zanotti, _Baryon spectroscopy in lattice QCD_ , _Lect. Notes Phys._ 663 (2005) 71 [nucl-th/0406032].
  * [13] SciDAC, LHPC, UKQCD collaboration, _The Chroma software system for lattice QCD_ , _Nucl. Phys. B Proc. Suppl._ 140 (2005) 832 [hep-lat/0409003].
  * [14] R. Quinn, J. Glesaaen, A. Rothkopf and J.-I. Skullerud, _Spectral properties of light and charm mesons from $N_{f}=2+1$ anisotropic lattice QCD_, _PoS_ Confinement2018 (2019) 272 [1903.11006].
  * [15] R. Quinn, _Improved methods for creating reconstructed correlators_ , May, 2019\.
  * [16] J. Glesaaen, _openqcd-hadspec_ , Sept., 2018.  10.5281/zenodo.2217027.
  * [17] J.R. Glesaaen and B. Jäger, _openqcd-fastsum_ , Apr., 2018.  10.5281/zenodo.2216355.
  * [18] R. Bignell, C. Allton, G. Aarts, J.-I. Skullerud, M.N. Anwar and B. Jaeger, _Dataset and Conference Proceeding "Reconstructed (charm) baryon methods at finite temperature on anisotropic lattices"_ , Dec., 2023.  10.5281/zenodo.10033629.
  * [19] P. Lepage, C. Gohlke and D. Hackett, _gplepage/gvar: gvar version 11.11.2_ , Mar., 2023.  10.5281/zenodo.7697309.
  * [20] J.D. Hunter, _Matplotlib: A 2d graphics environment_ , _Computing in Science & Engineering_ 9 (2007) 90.
  * [21] T.A. Caswell, A. Lee, E.S. de Andrade, M. Droettboom, T. Hoffmann, J. Klymak et al., _matplotlib/matplotlib: Rel: v3.7.1_ , Mar., 2023.  10.5281/zenodo.7697899.
  * [22] C.R. Harris, K.J. Millman, S.J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau et al., _Array programming with NumPy_ , _Nature_ 585 (2020) 357.
  * [23] P. Lepage and C. Gohlke, _gplepage/lsqfit: lsqfit version 13.0.1_ , May, 2023\.  10.5281/zenodo.7931361.
  * [24] G. Aarts, C. Allton, M.N. Anwar, R. Bignell, T.J. Burns, B. Jaeger et al., _Dataset and scripts for "Non-zero temperature study of spin 1/2 charmed baryons using lattice gauge theory"_ , Aug., 2023.  10.5281/zenodo.8273590.