# Communicating on Security within Software Development Issue Tracking

Léon McGregor
Heriot-Watt University
<EMAIL_ADDRESS>
Manuel Maarek
Heriot-Watt University
<EMAIL_ADDRESS>
Hans-Wolfgang Loidl
Heriot-Watt University
<EMAIL_ADDRESS>

###### Abstract

During software development, balancing security and non security issues is
challenging. We focus on security awareness and approaches taken by non-
security experts using software development issue trackers when considering
security. We first analyse interfaces from prominent issue trackers to see how
they support security communication and how they integrate security scoring.
Then, we investigate through a small scale user study what criteria developers
take when prioritising issues, in particular observing their attitudes to
security.

We find projects make reference to CVSS summaries (Common Vulnerability
Scoring System), often alongside CVE reports (Common Vulnerabilities and
Exposures), but issue trackers do not often have interfaces designed for this.
Users in our study were not comfortable with CVSS analysis, though were able
to reason in a manner compatible with CVSS. Detailed explanations and advice
were seen as helpful in making security decisions. This suggests that adding
improvements to communication through CVSS-like questioning in issue tracking
software can elicit better security interactions.

Copyright is held by the author/owner. Permission to make digital or hard
copies of all or part of this work for personal or classroom use is granted
without fee.

_USENIX Symposium on Usable Privacy and Security (SOUPS)_ 2023\.
August 6–8, 2023, Anaheim, CA, USA

##  1 Introduction

Discrepancies exist between what security experts desire and the guidance
developers end up following [10], so building better security practices into
software development is key for exposing security to non-experts. Modern
Development Operations (DevOps) processes, streamline tracking and
implementing features, bug fixes, and management of issues arising in software
development [17]. A key concern for security is whether, amongst the many
DevOps security practices [16], processes include prioritising reported
security issues. Prioritisation of issues during development is an important
step, as it dictates the approach that a whole project will take towards
developing their product. Priority tensions in a project mean that many
different aspects will compete for priority, and have an impact on developer’s
approaches to security and analytical thinking. “Consideration of revenue is
rational” for developers to prioritise [4], so care is needed to balance
security against functional requirements. In order for an application to be
secure, security issues ought to get a high priority. Making this decision
well requires project members to be informed on security risks and to be
involved in the decision making process, beyond just the security experts.

Investigating whether tools used during DevOps do enable security is important
as there are many factors that impact adoption [18]. DevOps must also enable
security professionals to use “communication and methods by which [non-
security developers] already share knowledge as part of their workflow” [2].
When considering security, communication and management during development are
key aspects to improving security motivation [3]. There is a need to
investigate if DevOps processes and tools encourage good security among non-
security experts.

Our investigation is twofold. First, we survey the design and security
approaches within large projects which track issues openly (Section 2). Then,
we conduct a developer study observing non-security experts approaches to
security prioritisation of issues and perception of the role of CVSS (Common
Vulnerability Scoring System) [5] as a security analysis system (Section 3).
We frame our work around the following research questions.

RQ1 How do software development issue tracking systems integrate security
considerations?
RQ2 Does prompting for CVSS during issue management have the potential to be a
useful interaction?
RQ3 How can non-security experts better engage with security during project
issue management?

#### Contributions

Our survey covers four large open software development issue trackers and
their usage in openly tracked projects (Section 2 details our selection
criteria). Then we run a developer study investigating how non-security-
experts make security decisions. Four participants engaged, making it a scale
small investigation, but we nevertheless draw the following preliminary
findings which form the main contributions of this paper. 1) Existing software
development issue tracking tools lack the design to fully convey security
concerns. 2) Non-experts seem not comfortable with using CVSS analysis. 3)
However, CVSS seems to be considered helpful by a non-security
expert/experienced project manager to prioritise security-impacting issues. 4)
Security inquiry through questioning and sharing advice could make security
more accessible to non-experts. 5) Security related metadata could be
integrated into issue trackers, elicited by text answers to security
questions, or optionally CVSS scores.

#### Security classifications and scoring

Numerous methods have been adopted across the security industry to help
classify the impact of security flaws. CWE identifiers (Common Weakness
Enumeration) [12] represent common classes of bug with similar behaviours.
They can be assigned to a bug to better describe what the problem is in
relation to others. CVE identifiers (Common Vulnerabilities and Exposures)
[11] are assigned to specific cases of flaws and uniquely identify a single
case of a security vulnerability or exposure of a sensitive system. CVSS
scores are values given to a reported vulnerability, calculated by measuring
the impact of a security flaw across several dimensions. CVE records contain
CVSS analysis, and many CVE records reference CWE classifications. CVSS
analysis can either be shared as a score between 0 and 10 (most severe) or as
a string which encodes all of the individual components of the scoring
metrics, allowing individuals to see precisely what the risks are. CVSS is a
method of measuring the severity of a security issue, and is often attached to
a specific CVE, though there is nothing that precludes it being used outside
of CVE only. Some research suggests there is significant disagreement amongst
security experts over the individual scores that might be generated through
CVSS [7], however there are findings suggesting that in general the CVSS
scores produced by databases are trustworthy [9]. This tension suggests that
these scoring mechanisms are worth investigating further, particularly with
respect to non-security experts. There exist other classification and scoring
systems, such as CWSS scoring system [13], similar to CVSS, but not focused on
specific security incidents. We focus on CVSS as it was specifically designed
to emphasise and rank which bugs should be prioritised for patching [14].

##  2 Issue Tracker Survey

To answer our research questions we investigate the design characteristics and
security approaches of platforms, before we conduct a developer study
investigating developer approaches within issue trackers. In this section, we
investigate issue tracking tools in order to see how well they integrate and
promote security. We have decided to analyse the public facing instances of
issue trackers for projects run by the same groups that created the bug
trackers, assuming that these developers would be the most likely to fully
utilise the capabilities of their issue tracking tools. Note that we are not
focused on performing in-depth ethnographic analysis of these projects, but
simply interested to see how they publicly use their own issue tracking
platforms in relation to security. Platforms referenced here were accessed in
July 2022.

#### Investigation steps

Our investigation followed steps for each issue tracker: 1) Explore and
identify all the interface elements, 2) Find security issues reported within
the project and observe interesting interactions, 3) Find mentions of CVE,
CVSS, CWE, or discussion relating to security choices.

Interface exploration involved identifying common and differing interface
elements on the individual bug/issue pages and seeing whether the projects
made use of these available elements. Security issues needed to be located by
a broad search for “security” in body text or labelling and identifying how
particular project demarcates security issues, then doing a deeper
investigation on terms such as “CVE” or “CVSS”.

#### Selection criteria

There are many DevOps, issue tracking, and project management tools, so we
narrowed down to a specific subset with publicly visible trackers to allow for
the focused analysis detailed above. We selected projects and tracking systems
when the system would 1) Track issues in a software development project, 2)
Not require a special plugin or extension, 3) Have a public tracker used by
the manufacturers, 4) Serve large projects.

We settled on the following 4 issue trackers: Bugzilla (Firefox Web Browser),
Monorail (Chrome Web Browser), Jira (Atlassian DevOps Tools), GitLab. Among
other major open systems, we discarded GitHub as it does have one central bug
tracking system, and Linux Kernel development as it relies on mailing lists
which are out of scope of this review. We note other platforms (Trac, Redmine)
met the criteria but these were not selected for full investigation.

###  2.1 Investigation

#### Bugzilla

Bugzilla is a bug tracker developed by Mozilla for use with developing the
Firefox browser. Within the Firefox project, the project uses all of the
fields that are available in the interface. In addition, Bugzilla allows site
administrators to add custom fields and properties if the project managers
feel they would benefit from it .

Bugzilla does not include any built-in support for tracking or measuring CVSS
scores, but there are cases where bug report participants have included
comments that reference CVSS scores. From searching the Firefox Bugzilla
instance for references to CVSS, we find that most of the bugs that include a
CVSS score are CVEs which have been copied to the Mozilla-run bug tracker from
other sources. Bug 1246014 is a good example of this. Coming via email from an
external security investigator, the CVSS and related analysis needs to be
included in a comment to the bug, as there is no standard field in the UI to
place it.

We see that when a CVE is reported, it will include a CVSS score, describing
the severity of the bug. In this case the bug was given a keyword, “sec-
critical”, as a way of tracking the severity. What is interesting in this case
is that the dedicated fields for bug priority and severity were not changed.
This behaviour is seen in other CVEs reported to the Mozilla tracker in this
fashion, including Bug 814001 and Bug 805121. Some CVEs reported as bugs do
however get the relevant fields set in addition to a priority keyword,
including Bug 1274637 and Bug 1631618. Outside of CVE and CVSS related bugs,
keywords, severity and priorities are used to keep track of bugs, as can be
seen in Bug 1538007. It is unclear just from these observations if there is a
consistent or formal procedure that Mozilla has for assigning security
priorities in their bug tracker, but it does indicate that a standardised
interface that can keep track of CVSS scores in addition to or in place of
many separate labels might be useful.

By performing a broad search across the entire bug tracker for mentions of
CVSS excepting when used to describe CVEs, we find that no bugs use CVSS
scores despite the intended use of CVSS as a means of analysing security bugs.

#### Monorail

Monorail is a bug tracker developed by Google, for use in developing Chrome,
the world’s largest web browser. One differentiating aspect to Bugzilla issue
tracking is the use of labels to track different aspects, rather than using
individual fields . Labels can be related using dashes - as an alternative to
defining specific fields, such as Target-102, which indicates that the team
aims to resolve an issue by the time version 102 is released. This scoped
labelling approach might offer more autonomy for individual project members to
categorise and prioritise issues than having to rely on a project
administrator to add specific fields. This capability is used frequently
within the Chrome project.

Searching for CVSS scores  given in bugs reported in the Chromium project
tracker reveals a similar pattern to that found in the Mozilla tracker.
Comments and descriptions of bugs only tend to mention CVSS scores when they
were originally formed as CVEs, cross-posted to the public tracker. An
illustrative example is found in Bug 1313172. The interface includes a
priority field, but in contrast no severity field, and instead severity labels
are used. Monorail does not have a built-in CVSS field or method to track this
so when required the comments section has to be used to display CVSS scores.

The Chrome project has over 1000 bugs referencing CVSS, but only 2 confirmed
bugs that reference CVSS scores without also mentioning CVE . These are Bug
571480 and Bug 695474. Both have a priority, but only the first has a security
severity label. This may indicate that there may either be limited utility or
a lack of recognition of these metrics outside of the context of CVE
reporting.

#### Jira

Jira is a bug tracker created by Atlassian. It is used by many projects
including MongoDB and Qt. Atlassian use Jira as their public facing issue
tracker for their own products, which we analyse here. In a Jira issue many
fields, properties, and links are present similar to other issue trackers.
Jira also allows Jira administrators to change the fields available . In the
open source project Qt, we see that their issue tracker only has the default
fields, and so is missing the “severity” fields present in Atlassian’s Jira
instance (seen in Bug QTBUG-105931). This approach means that different
projects may approach security in different ways, depending on how they have
configured their setup.

Within Atlassian’s public issue tracker there are references to CVSS scores
within the comments of bugs . It is interesting that many of the CVSS scores
are all presented in the same way, through a table in comments, yet Jira’s
custom fields are not used to standardise this. We see this presentation added
by a ‘bot’ in Bug JRASERVER-71198, indicating desire for some kind of
automation for collecting this data. All the comments in this fashion include
a link to an Atlassian CVSS Calculator  to explain the severity rating. The
CVSS scores seem to come from CVE notifications which state it _“is an
independent assessment and you should evaluate its applicability to your own
IT environment.”_ Whether this phrase is included as a way to instruct the
community reading these bugs to be alert, or simply as a form of legal
disclaimer is unclear. CVSS scores are not limited to just CVE reports, but
also appear in 122 issues unrelated to CVE, indicating that CVSS
identifications bear some importance to the Jira project. Tracking security is
an important part of Atlassian bugs, across their projects 6646 bugs are
related in some way to security, even excluding CVE .

While researching which open source projects use Jira, we found MongoDB
recommends voting as a way to gauge community priorities for fixing some
issue. This is an insight that the addition of a “voting” property may be
incentivising projects to use it as a way to decide priorities collectively.

#### GitLab

GitLab targets the whole DevOps process, including issue tracking. GitLab is
open source and the development team use their own project to track issues.

GitLab’s tracker shows many bugs mentioning CVSS without an associated CVE
report. There is a GitLab Bug 218601 which is a proposal to standardise the
way GitLab tracks CVSS scores, notably suggesting _“scoring can then be
exposed to the user in relevant parts of the UI”._ GitLab’s handbook describes
that priorities and severity can be assigned based on the CVSS scores that are
generated . The handbook also suggests that issues with high CVSS scores ought
to be labelled as high priority and be mitigated within 24 hours.

For individual projects, GitLab’s issue tracking interface does have a weight
field. However, the concrete meaning of this field is not defined and could
change project to project. From GitLab’s documentation , it could refer to any
of _“how much time, value, or complexity a given issue has or costs.”_ This is
still an important metric when it comes to prioritisation, however it is not
clear if it alone can be leveraged or relied on to handle the priorities of
security issues.

GitLab already has a CVSS calculator , but it is not used for generating
scores to prioritise issues. Instead it is for defining the “bounty” to be
rewarded for certain bugs. When issues are reported through the bounty
program, the CVSS scores generated as part of the reporting are converted to
priority and severity indicated through labelling. In some cases the CVSS
scores, though used as part of reporting, are not linked on the issue page
itself. Instead developers need to navigate to a separate page to view it. An
example of this can be see in Bug 336535: _Severity is set as per CVSS
calculated on hackerone report._ Not presenting this inline could add to
developer workload. Designing an interface that can present this information
could improve issue prioritisation to draw developer attention to more severe
issues. Other bugs, such as Bug 360986, include basic information of the final
CVSS score, and link to more detailed reports again held elsewhere.

GitLab’s development team currently tracks CWEs, and other weakness and
classifications through labels. GitLab’s team use scoped labels, which allows
all CWEs of a specific type to be grouped together . An open GitLab Bug 300978
is an issue discussing a proposal of whether to adopt CWEs as a means of
tracking and codifying security issues in GitLab’s Secure group. The evidence
of interest in CWE and CVSS indicates intent from GitLab to include more
support for these security vulnerability classification systems.

###  2.2 Comparison and summary

Table 1: Comparison of metadata fields across platforms

Field |  Bugzilla |  Monorail |  Jira |  GitLab
---|---|---|---|---
Priority | $\bullet{}$ | $\bullet{}$ | $\bullet{}$ | $\circ$
Severity | $\bullet{}$ | $\circ$ | $\bullet{}$ | $\circ$
Weight |  |  |  | $\bullet{}$
Votes |  |  | $\bullet{}$ | $\bullet{}$
Scoped Labels |  | $\bullet{}$ |  | $\bullet{}$
Milestones | $\bullet{}$ | $\circ$ | $\bullet{}$ | $\bullet{}$
Estimated Completion Times |  | $\bullet{}$ |  | $\bullet{}$
Epics |  | $\circ$ | $\bullet{}$ | $\bullet{}$
Colours | $\bullet{}$ |  | $\bullet{}$ | $\bullet{}$
Custom Fields | $\bullet{}$ | $\circ$ | $\bullet{}$ |
CWE |  |  |  | $\circ$

Native to all: Title and Description, Timestamps, Component Hierarchy, Issue
types, Labels, Attachments, Related Issues, Confirmation and Resolution,
Project Member Links Metadata support: native fields ($\bullet{}$), possible
with labels ($\circ$)

Table 1 compares the available features focusing on non-security specific
features as none of the investigated issue trackers include dedicated space to
discuss security specific concerns. We see that many features are shared
amongst the issue trackers. There are some outliers. GitLab seems to offer as
many capabilities as is possible. Where certain features are not present, many
projects using these systems utilise labels or keywords as a means of tracking
certain properties of issues. Only GitLab has a ‘weight’ field, however its
use might be duplicated by severity or estimated completion times in other
tracking systems. Most of the tracking systems make use of colour to draw
attention to certain features, with Bugzilla highlighting important issues in
red when in list view, to GitLab which uses very colourful labels, chosen by
project organisers. Some bug tracking systems have built-in priority tracking
fields for their bugs, and every bug tracking system has some form of keyword
or labelling fields. Developers often seem to use labelling as a means of
tracking severity, suggesting labels may be a more usable interface element.
The use of labelling varies between development projects.

The contrast between Bugzilla and Monorail is interesting when considering
severity labelling. Both Bugzilla and Jira include severity fields, others use
labels. But Firefox development includes a specific keyword despite Bugzilla
having a severity field, and Chrome development uses a severity label as an
alternative to a severity field. This indicates different approaches in how
strictly severity is assigned, and that one single approach is perhaps not
suitable for all projects.

The GitLab project tracks certain weaknesses classified by CWE through scoped
labelling. CVSS scores are mentioned within bug tracking for open source
projects, often in relation to externally reported issues such as through CVEs
or bug bounties. The CVSS scores appear within the text of the discussion of
the bug, instead of alongside the labels or fields that describe severity and
priority. This could indicate that though there is a desire for CVSS when
discussing critical security bugs like CVEs, the bug tracker interfaces do not
leverage CVSS at all for other kinds of reported security issues, and offer
little support for CVSS when they are used. Note that CVSS targets reported
vulnerabilities and might not suit all security planning discussions.

In summary, the selected DevOps issue trackers have many commonalities, but
also a few differences. We see wide variation in the approaches taken to
labelling, handling severity, and handling references to security issues and
reports. External metrics and classifications are used, but interface support
for referencing these is missing, and interfaces offer no automated support
for making decisions based on such metrics.

##  3 Developer Study

After investigating how trackers support security, we focus on how users
themselves make security decisions through our developer study. The purpose of
this study is to investigate approaches that non-security-experts take when
analysing and prioritising security issues. We examine whether techniques like
CVSS analysis are useful, and what parts of an issue presentation impact most.
This study is conducted online with non-security-expert developers and project
managers.

###  3.1 Protocol

The participants will be placed in a fictional setting, a software development
company developing financial applications. When presented with issues, the
participant are asked to consider each one, then assign a priority to it,
relative to the others, and answer questions about CVSS scoring. All
participants see the same issues, but in a random order. The prioritisation
will be a scale where the top issue needs to be dealt with first, and both
security and business-critical issues are rated together, to simulate the
prioritisation stresses faced in a real environment. To incur some amount of
time pressure to simulate a working environment, but allow for enough time to
have a reasonable attempt at seeing and briefly investigating all of the
issues, we allocate participants 30 minutes followed by however long needed to
answer a final questionnaire. Participants were awarded online vouchers for
their time regardless if they completed the experiment. Our protocol was
reviewed and approved by our university’s ethical committee.

Issues We add 14 issues to a tracker. 7 issues each focused on security and
functionality. The security issues were created by looking at top Mitre issues
, with a corresponding entry in the find-sec-bugs library . The functionality
issues were created by considering what a financial app would need to offer,
and potential requirements that may be faulty. For example, one issue
concerned “Improper input validation”, with consideration to security and
CWE-20. During the experiment issues are not explicitly named as being
security issues or not. For each of these issues, the following is given: a
title, a description, and a code snippet. The issues were designed to include
all the information relevant to the business, and which would allow for full
CVSS analysis. When prioritising these issues, there will be security and
functionality considerations for all, however some issues will be more or less
_security critical_ or _business critical_.

Experiment platform For this experiment, we worked with a customised GitLab
server. We chose GitLab as an easily self-hostable DevOps platform we could
customise for our experiment. We added custom questionnaires to this server:
an easy-to-use drag and drop interface for choosing relative priorities, and a
questionnaire shown next to issues with CVSS and other questions (shown in
Figure 1). These are shown as an overlay so that the primary activity of issue
analysis is always present. The drag and drop interface ensures that all of
the prioritisation was conducted in a relative fashion forcing a choice and
preventing giving the same priority to issues. Although we build on the GitLab
interface for our study, we are exploring the approaches developers take
generally rather than specifically in a GitLab context.

Figure 1: Screenshot of the issue questionnaire interface

###  3.2 Developer Study Outcomes

Here we present the outcomes from our developer study. We recruited 10
participants from computer science alumni, all of whom consented to take part.
4 started and completed the experiment, and all of these participants
identified as male. We recruited a balance of project managers and software
developers, 2 participants identifying expertise in each. None of the
participants felt their expertise in security was strong. We name the less
experienced participants SD1, SD2, PM1, and one project manager with more
experience PME.

Approaching prioritisation We viewed participants’ approaches to choosing
priorities through logged behaviour. 3 participants looked at most of the
issues when making their prioritisations, while PM1 only looked at the details
of 3 issues.

SD1 and SD2 changed their prioritisation as they read through the issues, and
PME preferred to make multiple prioritisations at once. PM1, who did not look
at many issues, did not change the priorities from the default random order,
so we cannot draw concrete conclusions for that participant’s priorities. SD1,
SD2, and PM1 on average changed 13 issue priorities from the initial random
assignment, which suggests they did engage well in this activity.

Looking at the final priorities chosen, we can see some trends. There was
agreement that “CSRF or Referrer Missing”, “SQL Injection”, “Input Validation”
were the most important as they appeared in the top 5 highest priorities;
“Adding and remembering payees” and “Chequing” and “Currency converter” were
all placed in the lowest 4 priorities; “Downloading PDF Summaries” was the
lowest priority issue. This is interesting to see as it suggests, from our
population sample, that they want to rate security issues higher than non-
security ones. Participants never contacted each other yet chose similar
priorities.

Usefulness of CVSS PME, SD2, and PM1 attempted to generate CVSS scores for
some issues. We asked how comfortable participants felt using GitLab, choosing
priorities, and completing the tasks. Responses showed mixed responses with no
clear trends across participant demographics. Only PME felt comfortable with
CVSS. The participants showed evidence of critical security thinking,
identifying where there were some security issues, but suggested due to them
not being readily exploitable, they could be de-prioritised. When asked about
the SSL issue, PME stated “[it] is serious but is not obvious or easy to
exploit so it is not as important,” and on an SQL Injection issue, SD1
comments “[issues] that have the potential for data leaks, is given highest
priority”. Analyses like the ease of exploitation and impact on
confidentiality are captured in CVSS, so standardised questioning like used in
CVSS could be a useful way to discuss issues amongst developers. PME’s comment
about risk mirrors the view that CVSS is concerned with the severity over risk
[15], and, in that sense, a framing of CVSS that better indicates context may
be helpful.

Engaging with security Participants feel there should be collective
responsibility for security. SD1 suggests “A sorted list of developer/security
analysts etc… so i would know who to ping for” is useful for issue tracking,
to expedite seeking advice or guidance. PM1 suggests employing “a dedicated
cyber team to review and ensure best practice”.

During the per-issue questions, we asked participants what aspect of an issue
was most influential to their choice of priority. PME and SD2 gave an
explanation for every issue, and SD1 gave explanations for the 6 issues they
prioritised highest. SD2 and PME, a software developer and a project manager,
favoured ‘advice given’ and ‘legal impact’, respectively. The most common
aspect impacting a decision was when a fellow staffer in the scenario gave
advice explaining how an attack worked. This is also backed up by one comment
that mentioned that “links” to standards are an important part of prioritising
an issue. This suggests the importance of communication between a team and
utilising relevant knowledge. The next biggest impact is when there may be a
legal impact to the business. GDPR and “privacy laws” were specifically
mentioned within this aspect. Potentially we may see more priority given to
security if laws surrounding secure programming and secure delivery of
services are known. The third biggest impact is when there is either an impact
to customers, or a need to weigh between commercial and security critical
interests. PME described this tension in a comment on an issue saying “this is
a commercial issue but not a critical issue,” and gave it a low priority,
showing that even in the face of business, they felt security was more
important. Encouraging more dialogue in the form of advice from relevant
parties during issue analysis, taking into account tradeoffs required between
commercial and security interests, could help less experienced developers
better engage with security decisions.

##  4 Discussion and Conclusion

We discuss our findings according to our 3 research questions.

RQ1: How do software development issue tracking systems integrate security
considerations? We see many large software projects reference security
analyses in issues reported to their trackers. Vulnerabilities like CVEs are
often included in the body or comment of a report, CVSS scores are likewise
pasted into comments, and labels are used to track the classification
according to external sources like CWE classes. Despite the apparent desire to
reference such security measurements and analyses, basic issue tracking
interfaces do not offer any built-in fields to support security metadata, with
projects like Atlassian opting for a bot to add security information in Jira.

RQ2: Does prompting for CVSS have the potential to be a useful interaction? In
the parts of the experiment where we analyse CVSS we find mixed evidence that
it could be a useful addition. Participants were able to identify the tensions
between prioritising security and functional interests. They gave comments
that mirror the approach taken by CVSS for conducting analysis. When directly
asked if they felt comfortable with CVSS, only 1 out of 4 participants felt
comfortable, so this could suggest that CVSS alone may not be suitable or
would require training to increase confidence before use. Though our sample
size is low, the fact that our more experienced project manager participant
felt most comfortable with CVSS could suggest that more experienced roles are
more suited to using CVSS for prioritisation, or that CVSS is more relevant to
such roles by potentially helping them liaise with software developers who
actually handle the issues. There is evidence from prior study that giving
additional advice during CVSS analysis helps to provide more accurate scoring
[1], so any future CVSS integration should come with guidance. Alternatives
such as the Exploit Prediction Scoring System (EPSS)  [8] are explored to
better suit proactive security analysis (EPSS was in early release when our
project started).

RQ3: How can non-security experts better engage with security during project
issue management? Experts can find the current design of DevOps tools more
useful than less experienced users. Explanations and colour are the most
useful design considerations when displaying important information for
security choices. Related studies into API design [6] find benefits in
involving developers when choosing such design considerations and this may
benefit issue management tools also. Developers should be supported with
external knowledge where relevant. To best engage with security discussions,
all experts should offer their advice and context, and if possible DevOps
processes should guide those making priorities towards people best able to
give this advice. Combined with security analysis from CVSS or otherwise, this
could improve security dialogue between team members.

## References

  * [1] Luca Allodi, Sebastian Banescu, Henning Femmer, and Kristian Beckers.  Identifying Relevant Information Cues for Vulnerability Assessment Using CVSS.  In Proceedings of the Eighth ACM Conference on Data and Application Security and Privacy, CODASPY ’18, pages 119–126, New York, NY, USA, March 2018. Association for Computing Machinery.
  * [2] Debi Ashenden and Gail Ollis.  Putting the Sec in DevSecOps: Using Social Practice Theory to Improve Secure Software Development.  In New Security Paradigms Workshop 2020, NSPW ’20, pages 34–44, New York, NY, USA, October 2020. Association for Computing Machinery.
  * [3] Sarah Beecham, Nathan Baddoo, Tracy Hall, Hugh Robinson, and Helen Sharp.  Motivation in Software Engineering: A systematic literature review.  Information and Software Technology, 50(9):860–878, August 2008\.
  * [4] Partha Das Chowdhury, Joseph Hallett, Nikhil Patnaik, Mohammad Tahaei, and Awais Rashid.  Developers Are Neither Enemies Nor Users: They Are Collaborators.  In 2021 IEEE Secure Development Conference (SecDev), pages 47–55, October 2021.
  * [5] FIRST.  CVSS v3.1 Specification Document.  https://www.first.org/cvss/v3.1/specification-document, 2019.
  * [6] Peter Leo Gorski, Yasemin Acar, Luigi Lo Iacono, and Sascha Fahl.  Listen to Developers! A Participatory Design Study on Security Warnings for Cryptographic APIs.  In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, pages 1–13, Honolulu, HI, USA, April 2020. Association for Computing Machinery.
  * [7] Hannes Holm and Khalid Khan Afridi.  An expert-based investigation of the Common Vulnerability Scoring System.  Computers & Security, 53:18–30, September 2015.
  * [8] Jay Jacobs, Sasha Romanosky, Octavian Suciu, Benjamin Edwards, and Armin Sarabi.  Enhancing vulnerability prioritization: Data-driven exploit predictions with community-driven insights, 2023.
  * [9] Pontus Johnson, Robert Lagerström, Mathias Ekstedt, and Ulrik Franke.  Can the Common Vulnerability Scoring System be Trusted? A Bayesian Analysis.  IEEE Transactions on Dependable and Secure Computing, 15(6):1002–1015, November 2018.
  * [10] Michelle Mazurek.  We Are the Experts, and We Are the Problem: The Security Advice Fiasco.  In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS ’22, page 7, New York, NY, USA, November 2022. Association for Computing Machinery.
  * [11] MITRE.  Overview | CVE.  https://www.cve.org/About/Overview, 1999.
  * [12] MITRE.  CWE - Common Weakness Enumeration.  https://cwe.mitre.org/about/index.html, 2006.
  * [13] MITRE.  CWSS - Common Weakness Scoring System.  https://cwe.mitre.org/cwss/cwss_v1.0.1.html, 2014.
  * [14] Marcus Pendleton, Richard Garcia-Lebron, Jin-Hee Cho, and Shouhuai Xu.  A Survey on Systems Security Metrics.  ACM Computing Surveys, 49(4):62:1–62:35, December 2016.
  * [15] Jonathan Spring, Eric Hatleback, Allen Householder, Art Manion, and Deana Shick.  Time to Change the CVSS?  IEEE Security & Privacy, 19(2):74–78, March 2021.
  * [16] Akond Ashfaque Ur Rahman and Laurie Williams.  Software security in DevOps: Synthesizing practitioners’ perceptions and practices.  In Proceedings of the International Workshop on Continuous Software Evolution and Delivery, CSED ’16, pages 70–76, New York, NY, USA, May 2016. Association for Computing Machinery.
  * [17] Anna Wiedemann, Nicole Forsgren, Manuel Wiesche, Heiko Gewald, and Helmut Krcmar.  The DevOps Phenomenon: An executive crash course.  Queue, 17(2):Pages 40:93–Pages 40:112, April 2019.
  * [18] Shundan Xiao, Jim Witschey, and Emerson Murphy-Hill.  Social influences on secure development tool adoption: Why security tools spread.  In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW ’14, pages 1095–1106, Baltimore, Maryland, USA, 2014. ACM Press.


# The droplet-scaling versus replica symmetry breaking debate in spin glasses
revisited

M. A. Moore  Department of Physics and Astronomy, University of Manchester,
Manchester M13 9PL, United Kingdom

###### Abstract

Simulational studies of spin glasses in the last decade have focussed on the
so-called replicon exponent $\alpha$ as a means of determining whether the
low-temperature phase of spin glasses is described by the replica symmetry
breaking picture of Parisi or by the droplet-scaling picture. On the latter
picture, it should be zero, but we shall argue that it will only be zero for
systems of linear dimension $L>L^{*}$. The crossover length $L^{*}$ may be of
the order of hundreds of lattice spacings in three dimensions and approach
infinity in 6 dimensions. We use the droplet-scaling picture to show that the
apparent non-zero value of $\alpha$ when $L<L^{*}$ should be $2\theta$, where
$\theta$ is the domain wall energy scaling exponent, This formula is in
reasonable agreement with the reported values of $\alpha$.

##  I Introduction

The nature of the low temperatures phase of Ising spin glasses in finite
dimensional spin glasses has been controversial for decades. A nice review of
the situation was given by Newman and Stein in 2003 Newman and Stein (2003a).
The two descriptions which are the most developed are the replica symmetry
breaking picture (RSB) which derives from Parisi’s exact solution Parisi
(1979, 1983); Rammal  _et al._ (1986); Mézard  _et al._ (1987); Parisi (2008)
of the Sherrington-Kirkpatrick model Sherrington and Kirkpatrick (1975) and
the droplet-scaling picture McMillan (1984); Bray and Moore (1986); Fisher and
Huse (1988). There are two other pictures, the TNT picture of Krzakala and
Martin Krzakala and Martin (2000) and of Palassini and Young Palassini and
Young (2000) and the chaotic pairs picture of Newman and Stein Newman and
Stein (1998). These four different pictures can be most readily distinguished
by the nature of excitations or droplets produced from their ground state and
the nature of the interfaces of the droplets or domain walls. Thus in $d$
dimensions consider the interface generated through changing the boundary
conditions from periodic to anti-periodic in one direction in a cube of length
$L$. The number of bonds in the interface will scale as $L^{d_{s}}$. If
$d_{s}=d$ the interface is said to be space filling. In the RSB and chaotic
pairs picture, interfaces are space filling. In the droplet-scaling and TNT
picture the fractal dimension $d_{s}<d$. The other distinguishing feature of
the four pictures is the (free) energy of the interfaces or droplets. In an
Ising ferromagnet the energy of a domain wall separating ‘up’ spins from
‘down’ spins scales as $L^{d-1}$. In the droplet picture (and also the chaotic
pairs picture) the energy of a spin glass interface or droplet is similar,
increasing as $L^{\theta}$ and $\theta>0$ when there is a finite temperature
spin glass phase. However, it is different in the RSB and TNT picture. There
an excitation or droplet can have an energy $O(1)$ even when it contains
$O(L^{d})$ spins.

It is my belief that what is the correct picture may change with the
dimensionality $d$ of the system. The strong coupling renormalization group
has been used Monthus (2015); Wang  _et al._ (2017, 2018) to study the value
of $d_{s}$ as a function of dimensionality $d$. It was found that $d_{s}$
became equal to $d$ in six dimensions. This suggests that for dimensions $d>6$
either the RSB picture or chaotic pairs could apply, while for $d<6$ the
droplet-scaling or TNT picture could apply.

Back in 2000 the TNT picture seemed to provide the description of the spin
glass state which was best supported by simulational work in $d=3$.
Simulations have mostly been done on the Edwards-Anderson Ising spin
Hamiltonian Edwards and Anderson (1975) where the bonds $J_{ij}$ are between
nearest-neighbors:

$\mathcal{H}=-\sum_{<ij>}J_{ij}S_{i}S_{j}-h\sum_{i}S_{i}.$ (1)

A focus of many studies has been the Parisi overlap function Parisi (1979,
1983, 2008) between spins in two copies, $a$ and $b$ of the system, defined by

$P(q)=\overline{\langle\delta(q-\frac{1}{N}\sum_{i}S_{i}^{a}S_{i}^{b})\rangle},$
(2)

where the overline denotes the bond-average over the couplings $J_{ij}$. When
the field $h=0$, $P(q)$ takes the trivial form of two delta functions in the
droplet-scaling and chaotic pairs pictures, (at least in the thermodynamic
limit when the number of sites $N\to\infty$);

$P(q)=(1/2)\delta(q-q_{EA})+(1/2)\delta(q+q_{EA}),$ (3)

where $q_{EA}=(1/N)\sum_{i}\langle S_{i}\rangle^{2}$, calculated in the limit
$h\to 0$. In the RSB and TNT pictures $P(q)$ is non-zero in the interval
$-q_{EA}<q<q_{EA}$. Studies of $P(q)$ at, say, $q=0$ showed that it remains
finite as $L$, the linear dimension of the simulational box, is increased.
However, in the droplet scaling picture it is predicted that $P(0)$ should
decrease with $L$ at finite temperatures $T$, as $T/L^{\theta}$. No
simulational study has ever seen any significant decrease of $P(0)$ with $L$
(Baños  _et al._ , 2010; Yucesoy  _et al._ , 2012; Billoire  _et al._ , 2013;
Yucesoy  _et al._ , 2013). On the other hand the study of interfaces seems to
strongly support the idea that they are not space filling as $d_{s}<d$
(although naturally this was disputed Marinari and Parisi (2000)). The
initials TNT refer to the fact that the behavior of the interface is trivial,
that is, as predicted by droplet scaling, but that the overlap function $P(q)$
is non-trivial as in the RSB picture of Parisi and not as given by the trivial
droplet-scaling prediction of Eq. (3).

Supporters of the droplet-scaling picture like the author of this paper would
explain away this failure to predict the observed form of $P(q)$ in
simulations as a finite size effect. Studies of $P(q)$ have been restricted by
computational limitations to systems whose linear dimension $L$ are usually
less than 30. It is postulated that there is a length scale, $L^{*}$, which
has to be surpassed before the true asymptotic behavior as $P(0)\sim
T/L^{\theta}$ reveals itself. Evidence that this might be a possibility has
come from studies on the $d=2$ spin glass problem, which does not have a
finite temperature spin glass phase ($\theta<0$) but it has features which
seem to have their analogue in $d=3$. In $d=3$ Krzakala and Martin Krzakala
and Martin (2000) noted that there were excitations on the scale of their
system size $L$ involving $O(L^{d})$ spins whose energies were not as large as
$~{}L^{\theta}$ but were instead of $O(1)$. Domain walls in $d=3$ do have
energies of $O(L^{\theta})$, but droplets seemed to exist which were as large
as the system (in fact they often touched the boundaries of the system) but
were of lower energy. In the droplet-scaling picture one considers a compact,
connected cluster of $N$ spins, of linear dimension $L$, such that
$L^{d}<N<(2L)^{d}$, containing the chosen spin. It is assumed Fisher and Huse
(1988); Bray and Moore (1986) that the distribution $\rho_{L}(E_{L})$ of
minimal energy clusters i.e. excitations has the scaling form

$\rho_{L}(E_{L})\approx\frac{1}{\Upsilon
L^{\theta}}\tilde{\rho}\bigg{[}\frac{E_{L}}{\Upsilon L^{\theta}}\bigg{]},$ (4)

where $\Upsilon$ is a constant of the order of the standard deviation of the
bonds $J_{ij}$ and $\tilde{\rho}(0)>0$ for $d\geq 2$. (In Appendix A we shall
calculate this distribution function analytically for the case of $d=1$). Eq.
(4) implies that the typical minimal droplet should have energy of
$O(L^{\theta})$ and that the probability that the minimal energy droplet has
energy of $O(1)$ should fall off as $1/\Upsilon L^{\theta}$. It is this which
lies behind the droplet-scaling prediction that $P(0)$ should decrease at
temperature $T$ as $\sim T/\Upsilon L^{\theta}$. In fact in the study of Ref.
Krzakala and Martin (2000) there seemed to be more low energy droplets than
expected from this formula and it is this which is the basis of the TNT
picture. A supporter of the droplet-scaling picture has to assert that the
systems studied always have a size $L<L^{*}$.

This seems plausible if one looks at the behavior of droplets in $d=2$
dimensions. The great advantage of studying two dimensions is that there exist
polynomial time algorithms which enable one to obtain the ground state of very
large systems. Thus by studying systems of size up to $10000\times 10000$ it
has been found that Khoshbakht and Weigel (2018) the energy associated with
the change from periodic to anti-periodic boundary conditions has
$\theta=-0.2793(3)$. The associated domain wall has a fractal dimension
$d_{s}=1.27319(9)$. However, the situation with droplets is more complicated
and produced a situation not unlike the debate between the advocates of the
droplet picture and the RSB picture in three dimensions. The droplet scaling
picture predicts that the correlation length, (as determined from the spin
glass susceptibility), should grows as the temperature is reduced as
$\xi(T)\sim 1/T^{1/|\theta|}$ Bray and Moore (1986) but the simulations at
finite temperature found that they appeared to grow with an effective exponent
$\tilde{\theta}\sim-0.48$ Kawashima  _et al._ (1992) down to the lowest
temperatures they could simulate. The origin of this discrepancy produced much
controversy Kawashima (1999); Berthier and Young (2003); Hartmann and Young
(2002); Picco  _et al._ (2003), before the correct explanation of the puzzle
emerged Hartmann and Moore (2003, 2004). The key to its understanding lies in
the fact that the droplet-scaling picture is indeed a scaling picture and that
there are always corrections to the leading terms. These can be much larger in
some quantities than in others. For example consider the domain wall produced
by changing the boundary conditions in one direction from periodic to
antiperiodic. The standard deviation of the energy difference takes the form

$\Delta E=AL^{\theta}+BL^{-\omega},$ (5)

where the second term is the “correction to scaling”. Note that $\theta$ in
two dimensions is negative (but is positive in three dimensions). According to
Ref. Khoshbakht and Weigel (2018) the correction to scaling term is very small
for domain walls. The situation for droplets is very different and depends on
how they are generated Hartmann and Moore (2003, 2004). Those which involve
flipping the central spin but for which the resulting droplet did not touch
the boundaries (the spins at the boundaries were fixed in their ground state
orientations) e.g the cross droplets (see Ref. Hartmann and Moore (2003, 2004)
for details) are such that at small values of $L$, their energy seemed to
decrease with an effective exponent $\tilde{\theta}=-0.47$ but for values of
$L>L^{*}\approx 60$ a value close to the expected value of $-0.279$ was seen.
Thus the simple droplet scaling behavior did emerge when droplets of large
enough size could be studied. It would also be expected that the correlation
length $\xi(T)$ would also grow according to droplet-scaling expectations if
studied at low enough temperatures and this has now been confirmed by recent
simulations Fernandez  _et al._ (2019).

We turn now to three dimensional spin glasses. It is the contention of this
paper that the finite energy droplets generated by the procedures used in
Refs. Krzakala and Martin (2000); Palassini and Young (2000) would not be of
size $O(L^{d})$ with a fractal dimension $d_{s}<d$ when $L>>L^{*}$. Here
$L^{*}$ denotes the crossover length in three dimensions, which I suspect
might be even longer than its two-dimensional counterpart. Newman and Stein
Newman and Stein (2001) have proved that in the large $L$ limit that
excitations or droplets of size $O(L^{d})$ with a fractal dimension $d_{s}<d$
cannot exist. (They proved that the interfaces of these excitations must
eventually pass outside a fixed finite window, no matter how large, as the
linear size L of the volume under consideration goes to infinity. So if TNT
applied, then inside of any fixed window one eventually sees the same, single
ground state pair (with, say, free or periodic boundary conditions) in the
large L limit, just as in the droplet-scaling picture). The apparent evidence
to the contrary in the numerical work of Refs. Krzakala and Martin (2000);
Palassini and Young (2000) is because they were unable to simulate large
enough systems and were working for system sizes $L<L^{*}$. For systems larger
than $L^{*}$ droplet-scaling features should emerge: If the finite energy
droplets do not involve $O(L^{d})$ spins when $L$ is large, they will not make
the Parisi overlap function non-trivial and for $L>L^{*}$ the behavior in Eq.
(3) will emerge. Given that in two dimensions, the crossover length
$L^{*}\approx 60$, then it seems likely in three dimensions that the length
scale $L^{*}$ will be even larger, perhaps of the order of hundreds of lattice
spacings. I would anticipate that it will approach infinity as $d\to 6$ when
these droplet states of $O(1)$ energy will have $d_{s}=d$ Wang  _et al._
(2017, 2018) and produce the RSB states expected for $d>6$.

On this viewpoint the TNT picture is just the droplet-scaling picture, with
the recognition that there are droplets whose energies are of $O(1)$ in
systems whose linear dimensions are less than $L^{*}$ and only there do they
have size $O(L^{d})$. This is all due to scaling corrections. In Ref. Hartmann
and Moore (2003, 2004) it was suggested that the possible origin of these
droplets whose energies are of $O(1)$ might arise from the fact that in Eq.
(5) that $\Delta E$, now being used to describe the energy of droplets, has a
minimum at some value of $L$ when $B>0$ and $\theta>0$. Around this minimum
the $L$ dependence of $\Delta E$ will be small, giving rise to an effective
value of $\theta$, $\tilde{\theta}$, which would be close to zero at the $L$
values near the minimum. It will be only at large $L>L^{*}$ that $\Delta E$
will clearly increase as $L^{\theta}$, just as occurred for the cross-droplets
in two dimensions when they were larger than $L^{*}$ Hartmann and Moore (2003,
2004). The size of corrections to scaling depend on the quantity being
studied: Domain wall energies (for $\theta$) and the interface size (for the
exponent $d_{s}$) may only have small corrections and the existing studies for
$L<L^{*}$ may still be giving accurate answers for these exponents.

It would clearly be very desirable to have estimates of the value of $L^{*}$.
This has been done by Middleton Middleton (2013) for a particular form of the
bond distribution, the $\pm J$ model. This bond distribution produces a
macroscopic degeneracy for the ground state of the system and has zero-energy
droplets. The finite temperature excitations with free energies of $O(1)$ are
discussed in Thomas  _et al._ (2011). At any non-zero temperature the
properties of the $\pm J$ model should be similar to models with a continuous
bond distribution Jörg  _et al._ (2006) (but the value of $L^{*}$ will not be
universal). Middleton Middleton (2013) estimated a value of $L^{*}$ in two
dimensions of $\approx 64$. This is rather similar to the value $\approx 60$
obtained from the behavior of the cross-droplets which were studied for a
Gaussian bond distribution in Hartmann and Moore (2003, 2004). One of his
methods for getting a value of $L^{*}$ was similar to that used in Ref. Hatano
and Gubernatis (2002) and using it Middleton obtained $L^{*}\approx 500$ in
three dimensions.

Over the last decade the Janus collaboration and others F. Belletti et al.
(2009); Baños  _et al._ (2012); Manssen  _et al._ (2015); Billoire  _et al._
(2017) have presented results which seem at first sight to be at variance with
droplet-scaling expectations. They have found evidence which suggests that
spin glasses in three dimensions have the behavior only expected of a system
with RSB or chaotic pairs ordering. In Refs. F. Belletti et al. (2009);
Manssen  _et al._ (2015) they carried out the following simulation. Starting
from a randomly chosen set of spin configurations, they quenched to a
temperature $T<T_{c}$, where $T_{c}$ is the spin glass transition temperature.
They then let the spins evolve according to heat bath dynamics for a time
$t_{W}$. This results in domains of spin glass order whose size is measured by
a coherence length $\xi(t_{W})$ which grows as $t_{W}$ increases. They found
that the correlation function

$C_{4}(R_{ij},t_{W})\equiv\overline{\langle
S_{i}S_{j}\rangle^{2}}\sim\frac{1}{R_{ij}^{\alpha}}f\bigg{(}\frac{R_{ij}}{\xi(t_{W})}\bigg{)}.$
(6)

The overline is the usual bond average. In practice this was done by
simulating two copies of the system with the same interaction but quenched
into different initial random configurations, which allows an unbiased
estimate of the thermal averages. The function $f(x)$ falls off with
increasing $x$ faster than exponentially and $f(x\to 0)={\rm const}$. The
coherence length $\xi(t_{W})$ is itself determined via the ratio of the second
and zeroth moments of $C_{4}(R_{ij},t_{W})$. The $k^{{\rm th}}$ moment is
defined by

$I_{k}(t_{W})=\int d^{d}r\,r^{k}C_{4}(r,t_{W}),$ (7)

and then

$\xi(t_{W})=\sqrt{\frac{I_{2}(t_{W})}{I_{0}(t_{W})}}.$ (8)

The coherence length is found to grow slowly (coarsen) with $t_{W}$ as
$\xi(t_{W})\sim t_{W}^{{\rm const}/T}$. In simulations it can grow as large as
$20$ to $30$ lattice spacings. Much interest attaches to the exponent $\alpha$
which is called the “replicon exponent” by the Janus collaboration. An early
estimate of its value in $d=3$ was $0.38(2)$ F. Belletti et al. (2009), while
in a more recent paper this was revised downwards to $0.35>\alpha>0.25$ Baity-
Jesi  _et al._ (2018).

It seems likely that the coherence length $\xi(t_{W})$ was always less than
the crossover length $L^{*}$ in these simulations so one should expect TNT
effects. Then the system will behave as if it has some RSB features (in
particular, it will have droplets of size $O(L^{d})$ with energy cost $O(1)$
which change as $\xi(t_{W})$ grows). This will make the value of $\alpha$
appear to be non-zero. The droplet-scaling prediction is Manssen  _et al._
(2015); Billoire  _et al._ (2017), $C_{4}(R_{ij},t_{W})\to q_{EA}^{2}$ but
this result will only be seen when $\xi(t_{W})>>L^{*}$. This approach to a
constant (which corresponds to $\alpha=0$) will emerge only in the limit
$L^{*}<<R_{ij}<<\xi(t_{W})$. The present simulations are a very long way off
this limit. It means that the values for $\alpha$ currently being reported for
$d=3$ are just effective values of this exponent, as they are only valid over
a limited range of $R_{ij}$ and $\xi(t_{W})$. However, because $L^{*}$ may be
quite large, the value for the replicon exponent $\alpha$ could be well-
defined: The crossover region where it gradually goes to its true value of
zero has not been reached. In Sec. II the values for $\alpha$ currently being
reported are predicted by a simple argument which rests on the assumption that
the correct picture of the three dimensional spin glass ordered phase is that
of droplet-scaling. We show that in dimensions $d<6$ that this effective value
for $\alpha$ is $2\theta$, where $\theta$ is the usual exponent describing the
energy cost of a domain wall. This result is consistent with the numerical
data on $\theta$ and $\alpha$, neither of which alas are very accurately
determined at the present time.

The exponent $\alpha$ appears in another form in studies of the metastates of
spin glasses. Most metastates discussions in spin glasses concern equilibrium
properties Newman and Stein (1996, 1997, 1992, 2003b); Arguin  _et al._
(2015). (One exception is Ref. White and Fisher (2006)). An exponent $\zeta$
has been introduced and discussed at length by Read Read (2014). It is defined
via the logarithm of the number of metastates which can be distinguished in a
window of size $W$ which scales as $W^{d-\zeta}$. Read Read (2014) showed that
the RSB picture predicted that $\zeta=4$ when $d>6$. There is an assumed
equivalence between the equilibrium metastates and those generated using a
dynamical coarsening procedure to define an Aizenman-Wehr metastate Aizenman
and Wehr (1990). If they are equivalent $d-\zeta\equiv\alpha$, where $\alpha$
is defined from Eq. (6). In three dimensions Billoire et al. Billoire  _et
al._ (2017) found using the equilibrium metastate approach $\alpha=0.7\pm
0.3$, while a coarsening dynamical metastate procedure was used by Manssen et
al. who analysed their data with $\alpha=0.438$ Manssen  _et al._ (2015). In
Ref. Wittmann and Young (2016) a simulation on a one-dimensional system with
long-range interactions thought to be equivalent in its behavior to that of
the EA model in $d=8$ was used to construct the dynamical metastate and it
gave a value for $\alpha$ consistent with Read’s predictions and the assumed
equivalence of static and dynamical metastate constructions. Our expression
for the effective exponent $\alpha$ agrees with Read’s prediction at $d=6$.

In Sec. III we make suggestions for further (mostly simulational) work which
could help in checking the validity of the scenario advocated in this paper.

##  II The replicon exponent

The Janus collaboration Baity-Jesi  _et al._ (2017); Zhai  _et al._ (2020);
Paga  _et al._ also studied the effect of turning on a small field $h$ at time
$t_{W}$ and determining the magnetization
$m(t+t_{W})=(1/N)\sum_{i}S_{i}(t+t_{W})$ at time $t+t_{W}$. They make the
“bold” claim Baity-Jesi  _et al._ (2017) that $m(t+t_{W})$ has the form which
one might have written down using equilibrium arguments, which they take to be

$m(t+t_{w},t_{w};h)=\xi(t+t_{W})^{y_{h}-d}\mathcal{F}(h[\xi(t+t_{W})]^{y_{h}};\mathcal{R}_{t,t_{W}}),$
(9)

where $\mathcal{R}_{t,t_{W}}\equiv\xi(t+t_{W})/\xi(t_{W})$. The scaling
function $\mathcal{F}(x,\mathcal{R})$ is odd in the $x$ argument for symmetry
reasons. $t$ is of order $t_{W}$ so $\mathcal{R}_{t,t_{W}}\approx 1$ and
dependence on this variable is ignored. The relevant length scale in this
study is $\xi(t_{W})$. The exponent $y_{h}$ was claimed to be related to
$\alpha$ by

$y_{h}=\frac{d}{2}-\frac{\alpha}{4}$ (10)

We caution the reader at this point that the Janus collaboration have taken to
writing what we call $\alpha$ as $\theta$. In this paper $\theta$ has its
conventional spin glass meaning as the exponent associated with the domain or
droplet energy (Janus call that $\zeta$).

The magnetization can be written as a series expansion in odd powers of $h$:

$m(h)=\chi_{1}h+\frac{\chi_{3}}{3!}h^{3}+\frac{\chi_{5}}{5!}h^{5}+O(h^{7}),$
(11)

where the dependency of the susceptibilities $\chi_{1}$, $\chi_{3}$,
$\chi_{5}$ and $m$ on $t$ and $t_{W}$ has been omitted to simplify the
notation. Combining Eq. (9) and Eq. (11), one deduces for example, that

$\chi_{3}\propto-\xi(t_{W})^{4y_{h}-d}\equiv-\xi(t_{W})^{d-\alpha}.$ (12)

In Ref. Baity-Jesi  _et al._ (2017) they wrote that “At least in equilibrium
$\chi_{3}$ is related to the space integral of the microscopic correlation
function $C_{4}(R_{ij},t_{W})$”. This if true would explain why $\alpha$
appears in $y_{h}$ as in Eq. (10). But it is only true when $T>T_{c}$. For
$T<T_{c}$ the equilibrium expression for $\chi_{3}$ in terms of correlation
functions was given long ago Chalupa (1977) and for a symmetric bond-
distribution is the bond-average of

$\displaystyle\chi_{3}$ $\displaystyle=$
$\displaystyle-\frac{6\beta^{3}}{N}\sum_{ij}G_{B}(R_{ij})$ (13)
$\displaystyle+$ $\displaystyle\frac{4\beta^{3}}{N}\sum_{i}(1-4\langle
S_{i}\rangle^{2}+3\langle S_{i}\rangle^{4}).$

where the correlation function $G_{B}(R_{ij})$ is the breather or longitudinal
correlation function studied in Bray and Moore (1979) and given by the bond
average of

$G_{B}(R_{ij})=\langle S_{i}S_{j}\rangle^{2}-4\langle S_{i}\rangle\langle
S_{j}\rangle\langle S_{i}S_{j}\rangle+3\langle S_{i}\rangle^{2}\langle
S_{j}\rangle^{2}.$ (14)

The second line in Eq. (13) only contributes a finite term to $\chi_{3}$,
whereas the term involving $G_{B}$ gives a contribution which diverges with
the size of the system. $G_{B}$ has been studied within the droplet-scaling
picture Bray and Moore (1986); Fisher and Huse (1988). There it was shown that
it was given in terms of an integral involving the scaling function
$\tilde{\rho}$ of Eq. (4):

$G_{B}(R_{ij})\sim\frac{Tq_{EA}^{2}}{\Upsilon
R_{ij}^{\theta}}\int_{0}^{\infty}dx\,\tilde{\rho}\bigg{(}\frac{Tx}{\Upsilon
R_{ij}^{\theta}}\bigg{)}{\rm sech}^{2}x(1-3{\rm tanh}^{2}x).$ (15)

For small values of the ratio $T/(\Upsilon R_{ij}^{\theta})$ the integral can
be approximated by setting the term in $\tilde{\rho}$ to its value at
$\tilde{\rho}(0)$. However, the integral then is zero, so one has to expand
$\tilde{\rho}(x)$ to next order in its Taylor series expansion;
$\tilde{\rho}(x)=\tilde{\rho}(0)-Ax+\cdots$. Then $G_{B}(R_{ij})$ becomes

$G_{B}(R_{ij})\sim Aq_{EA}^{2}\bigg{(}\frac{T}{\Upsilon
R_{ij}^{\theta}}\bigg{)}^{2}.$ (16)

Previously Bray and Moore (1986); Fisher and Huse (1988), out of an abundance
of caution, the possibility that $\tilde{\rho}(x)=\tilde{\rho}(0)-Ax^{\phi}$
was considered. This changes the exponent of the term in $T/\Upsilon
R_{ij}^{\theta}$ in Eq. (16) from $2$ to $(1+\phi)$. In Appendix A it is shown
for the case of $d=1$ that the Taylor series form for $\tilde{\rho}(x)$ is
appropriate. The numerical data in $d=2$ Hartmann and Moore (2004) is at least
consistent with the Taylor series expansion form $\phi=1$. We shall in this
paper from now on just take $\phi=1$.

Using Eq. (16) for $G_{B}(R_{ij})$ one deduces that
$\chi_{3}\sim-L^{d-2\theta}$. Compare this with Eq. (12);
$\chi_{3}\sim-\xi(t_{W})^{d-\alpha}$ in the coarsening investigation. In
coarsening the relaxation modes of the system with wavevectors greater than
$1/\xi(t_{W})$ are equilibrated. At wavevectors $k<1/\xi(t_{W})$ the system
will still have the imprint of the infinite temperature system it was before
the quench. This suggests that this region of $k$-space will only make a
finite contribution to $\chi_{3}$. Hence we should be able to equate the
exponents of $L$ and $\xi(t_{W})$ in the two expressions for $\chi_{3}$, so
making

$\alpha=2\theta.$ (17)

The equilibration of the system within the length scale $\xi(t_{W})$ is not
such as happens in a finite system on ergodic time scales which drives
$\langle S_{i}\rangle$ to zero. Instead it is more like the equilibration of
an infinite system in which boundary conditions are applied to break the up-
down symmetry and leave $\langle S_{i}\rangle$ non-zero. It is to this
situation that Eqs. (13) and (14) are applicable.

We next compare Eq. (17) with the current numerical estimates of $\alpha$ and
$\theta$. Alas neither are known with great precision. In $d=3$, $\alpha$ was
found to be $0.38(2)$ in F. Belletti et al. (2009). In later work Baity-Jesi
_et al._ (2018), they noticed there was an apparent temperature dependence in
the value of $\alpha$. If the quench were not to a temperature less than
$T_{c}$ but instead to $T_{c}$ itself $\alpha=d-2+\eta\approx 0.610(4)$ (see
the definition of $C_{4}(R_{ij},t_{W})$ in Eq. (6)). They argued that their
apparent temperature dependence was due to proximity to $T_{c}$ in their work.
This effect would go away if $\xi(t_{W})\to\infty$. However, by using an
extrapolation to this limit they estimated that $0.35>\alpha>0.25$. The value
for $\theta$ was given by Hartmann Hartmann (1999a) to be $0.19(2)$ and by
Boettcher Boettcher (2005) as $0.24(1)$. For $d=4$ both exponents are even
less precisely determined: $\alpha=1.03(2)$ according to Ref. Nicolao  _et
al._ (2014), while $\theta$ is $0.64(5)$ according to Ref. Hartmann (1999b)
and $0.61(1)$ according to Ref. Boettcher (2005). In $d=6$, the RSB formula of
Read Read (2014) gives $\alpha=2$, while the RSB based calculations of Ref.
Aspelmeier  _et al._ (2016a) give $\theta=1$. The numerical estimate in $d=6$
by Boettcher in Boettcher (2005) was $\theta=1.1(1)$. I would judge that the
agreement of Eq. (17) with the data for $d\leq 6$ to be satisfactory, given
the large uncertainties in the numerical values of $\alpha$ and $\theta$.

##  III Discussion

In this paper we have argued that the old TNT picture of spin glasses can
explain not only the old puzzle of the behavior of the Parisi overlap function
$P(q)$ but also the more recent results of the Janus group on the correlation
function $C_{4}(R_{ij},t_{W})$. It has also been argued that the TNT picture
is only relevant when phenomena on length scales $<L^{*}$ are studied, and
that the RSB-like behavior seen on length scales less than $L^{*}$ will change
to droplet-scaling behavior on the longest length scales. It has been possible
to show the replicon exponent $\alpha$ of the Janus collaboration is equal to
$2\theta$ for $d<6$. This result shows that even in the RSB-like region
$L<L^{*}$, droplet-scaling calculations have utility.

We suspect that the length scale $L^{*}$ in $d=3$ might be so large that any
crossover in behavior might be impossible to see in simulations at present,
where $\xi(t_{W})$ or the linear size $L$ of an equilibrated system are
usually less than $30$. However, in experiments it has been claimed Zhai  _et
al._ (2019) that values of $\xi(t_{W})$ of order $250$ are being seen. In
$d=2$ a crossover was seen at values of $L^{*}=60$ Hartmann and Moore (2004)
which suggests that $L^{*}$ could be hundreds of lattice spacings for $d=3$.
Indeed $L^{*}$ probably grows to infinity as $d\to 6$ when the droplets of
energy $O(1)$ of the TNT picture become the pure states of RSB. Our value for
$\alpha$ coincides with that derived from the RSB picture by Read Read (2014)
in six dimensions. However, the large value of $L^{*}$ allow the possibility
that for sizes $L<L^{*}$ the apparent values quoted for $\alpha$ could be
well-defined and not influenced by the expected creep towards $0$ when $L$
grows past $L^{*}$.

I shall now make a few suggestions for a number of investigations which might
help to clarify what is going on.

  1. 1.

If indeed $L^{*}$ is of order $500$ in three dimensions many properties of
spin glasses will appear both in simulations and experiments to be just as
expected from the RSB picture. However, a key difference between RSB (and
chaotic pairs) and droplet-scaling is that on the droplet-scaling picture
$d_{s}<d$, whereas on the other two pictures, the domain wall produced by (for
example) a change of boundary conditions from periodic to anti-periodic in one
direction, is space filling with $d_{s}=d$. Domain wall energies seem to have
much smaller corrections to scaling than those of droplets and so studying
domain walls would seem likely to be a good way of also getting at $d_{s}$.
Much numerical work, although it was done for $L<L^{*}$, does favor $d_{s}<d$,
thus supporting the droplet-scaling picture. Exponents like $\theta$ and
$d_{s}$ are exponents associated with the zero-temperature fixed point and it
would be natural to expect that the best results for their value would be
obtained from $T=0$ studies. Alas, that requires finding ground states of the
Hamiltonian which is NP hard for $d>2$. The methods which have been used at
finite temperature usually involve extensive data manipulation F. Belletti et
al. (2009). An old simulation by Huse Huse (1991) gave a way of determining
$d_{s}$ at finite temperatures using a coarsening procedure which avoided
extensive data manipulation (and yielded $d_{s}<d$). That approach could
nowadays be pushed to larger values of $\xi(t_{W})$.

  2. 2.

The susceptibility $\chi_{3}$ is an integral over all space of the correlation
function $G_{B}(R_{ij})$. This correlation function was studied using
simulations in $d=3$ in Ref. De Dominicis  _et al._ (2005) but only for rather
modest system sizes ($L=12$). Both RSB and droplet scaling predict a power law
decrease of $G_{B}$ with $R_{ij}$. In Ref. De Dominicis  _et al._ (2005) a
more rapid decay, possibly exponential, with $R_{ij}$, was seen. That might be
due to finite size effects, but it would be useful if this topic could now be
re-visited.

  3. 3.

The predictions of the Janus collaboration of the behavior of $\chi_{3}$ as
$L^{d-\alpha}$ should presumably extend to the RSB region $d>6$. Using Read’s
result $\alpha=d-4$ for $d>6$, the divergence of $\chi_{3}$ is then
$\chi_{3}\sim L^{4}$. The analytical work in Ref. De Dominicis  _et al._
(2005) predicts a divergence of $\chi_{3}$ from $G_{B}$ as $L$ in dimensions
$d>6$. It would be interesting to study this discrepancy using the one-
dimensional proxy model for high dimensions used in Ref. Wittmann and Young
(2016). However, ageing a system with RSB towards equilibrium needs to be re-
examined in the light of the recent findings of Bernaschi  _et al._ (2020) for
the Viana-Bray model Viana and Bray (1985) who found that the system stayed
trapped in a confined region of the configuration space.

Within the spin glass phase itself the large value of the crossover length
$L^{*}$ will make it difficult to provide good numerical or experimental
evidence as to which picture of spin glasses, droplet-scaling or RSB, is
correct. $L^{*}$ is, however, a feature of the zero-temperature fixed point.
Fortunately there is another way to resolve the debate which avoids features
produced by the zero-temperature fixed point and that is to determine whether
there is an de Almeida-Thouless transition de Almeida and Thouless (1978) when
a magnetic field is applied. This line marks the onset to a state with RSB, on
cooling in a field and is absent according to the droplet-scaling picture Bray
and Moore (1986); Fisher and Huse (1988). There have been doubts as to its
existence below six dimensions ever since Bray and Roberts Bray and Roberts
(1980) were unable to find a stable perturbative critical fixed point for it
in dimensions $d$ just below $6$. Further arguments to this effect have been
given Moore and Bray (2011); Moore (2012). In three dimensions there is
experimental evidence Mattsson  _et al._ (1995) supporting the absence of the
de Almeida-Thouless line. Simulations on this issue Larson  _et al._ (2013)
provide in the view of this author excellent evidence that there is no de
Almeida-Thouless transition, (but some still remain of the view that there is
a transition Paga  _et al._ ; finite size complications Yeo and Moore (2015);
Aspelmeier  _et al._ (2016b) are not insignificant).

###### Acknowledgements.

I should like to acknowledge very useful exchanges with Alex Hartmann, Chuck
Newman, Daniel Stein, Victor Martin-Mayor, Nick Read, and Peter Young.

*

##  Appendix A The droplet distribution function in one dimension

In Sec. II we made the assumption that the scaling function of Eq. (4),
$\tilde{\rho}(x)$, had a Taylor series expansion as
$\tilde{\rho}(x)=\tilde{\rho}(0)-Ax+\cdots$ rather than as
$\tilde{\rho}(x)=\tilde{\rho}(0)-Ax^{\phi}$. In this Appendix the scaling
function is obtained analytically for $d=1$ and it is shown that in this case
$\phi=1$ and that the Taylor series expansion of $\tilde{\rho}(x)$ is valid.

Figure 1: The scaling function $f(x)=\tilde{\rho}(x)$ for the droplet energy
distribution scaling function in one dimension.

The ground state of a one-dimensional spin system with open boundary
conditions is found by making (say) the spin at one end, $S_{1}=1$ and fixing
the orientation of the remaining spins using $S_{i}S_{i+1}=sign(J_{i,i+1})$.
To find the domain wall energy one flips the spin $S_{N}$. This causes all the
spins up to the bond of smallest magnitude to flip. In a system of $L$ bonds,
the bond $|J|$ of smallest magnitude has the distribution for large $L$ Bray
and Moore (1986)

$P_{L}(|J|)=\frac{1}{J(L)}\exp\bigg{(}-\frac{|J|}{J(L)}\bigg{)},$ (18)

and

$J(L)=\frac{1}{P_{J}(0)L}.$ (19)

Here $P_{J}(0)$ denotes the value of the bond distribution function at $J=0$,
and provided it is non-zero, Eq. (19) implies that $\theta=-1$ for $d=1$. Eq.
(18) is the distribution function for “domain wall energies” $E$, which is
equal to that of $|J|$.

The minimal droplet energy $E$ around site $i$ is the sum of the energy of the
bond, $E_{1}$, to the right of site $i$ where all the bonds between $i$ and
the bond to the right of the site at $i+L_{1}$ have magnitudes greater than
$E_{1}=|J_{i+L_{1},i+L_{1}+1}|$ plus the energy of the bond, $E_{2}$, to the
left of site $i$ where all the bonds between $i$ and the bond to the left of
the site at $i-L_{2}$ have magnitudes larger than
$E_{2}=|J_{i-L_{2},i-L_{2}-1}|$. Then the $L=L_{1}+L_{2}$ spins lying between
$i-L_{2}$ and $i+L1$ can all be flipped together at a total energy cost of
$E=E_{1}+E_{2}$. For large $L_{1}$ and $L_{2}$ the distribution of $E_{1}$ and
$E_{2}$ will be as given in Eq. (18), so the probability distribution of
droplets of energy $E$ and size $L$ will be

$\rho_{L}(E)=\frac{1}{L}\int dL_{1}\int dL_{2}\int dE_{1}\int dE_{2}\\\
\delta(E-E1-E_{2})\delta(L-L_{1}-L_{2})P_{L_{1}}(E_{1})P_{L_{2}}(E_{2}).$ (20)

The coefficient $1/L$ arises as the spin i can be in any of the $L$ sites
between the weak bonds which are broken when the droplet is flipped. Writing
$\rho_{L}(E)$ in terms of its scaling form
$\rho_{L}(E)=(1/J(L))\tilde{\rho}(E/J(L))$ one finds

$\displaystyle\tilde{\rho}(x)$ $\displaystyle=$
$\displaystyle\int_{0}^{1}\,dy\frac{y(1-y)[\exp(-xy)-\exp(-x(1-y))]}{1-2y}$
$\displaystyle=$
$\displaystyle\frac{1}{2}e^{-x}\bigg{(}-\frac{2+e^{x}(-2+x)+x}{x^{2}}+e^{x/2}{\rm
Shi(x/2)}\bigg{)},$

where ${\rm Shi}$ is the $\sinh$ integral and $x=E/J(L)$. The function
$\tilde{\rho}(x)$ in shown in Fig. 1.

In two dimensions $\tilde{\rho}(0)$ is finite rather than as here zero. The
function $\tilde{\rho}(x)$ is an increasing function of $x$ at small values of
$x$ just as in two dimensions Hartmann and Moore (2004). In $d=3$ I suspect
that $\tilde{\rho}(x)$ is actually a decreasing function of $x$. Fig. 1 shows
that the scaling function has a long tail at large $x$; in fact it is so long-
tailed that the mean droplet size is not well-defined. $\tilde{\rho}(x)$ has
the Taylor series expansion at small $x$

$\tilde{\rho}(x)=x/6-x^{2}/12+x^{3}/45-O(x)^{4},$ (21)

which implies that $\phi=1$ in one dimension.

## References

  * Newman and Stein (2003a) C. M. Newman and D. L. Stein, “Finite-Dimensional Spin Glasses: States, Excitations, and Interfaces,” Ann. Henri Poincaré 4, 497 (2003a).
  * Parisi (1979) G. Parisi, “Infinite number of order parameters for spin-glasses,” Phys. Rev. Lett. 43, 1754 (1979).
  * Parisi (1983) G. Parisi, “Order parameter for spin-glasses,” Phys. Rev. Lett. 50, 1946 (1983).
  * Rammal  _et al._ (1986) R. Rammal, G. Toulouse, and M. A. Virasoro, “Ultrametricity for physicists,” Rev. Mod. Phys. 58, 765 (1986).
  * Mézard  _et al._ (1987) M. Mézard, G. Parisi, and M. A. Virasoro,  _Spin Glass Theory and Beyond_ (World Scientific, Singapore, 1987).
  * Parisi (2008) G. Parisi, “Some considerations of finite dimensional spin glasses,” J. Phys. A 41, 324002 (2008).
  * Sherrington and Kirkpatrick (1975) David Sherrington and Scott Kirkpatrick, “Solvable Model of a Spin-Glass,” Phys. Rev. Lett. 35, 1792 (1975).
  * McMillan (1984) W. L. McMillan, “Scaling theory of Ising spin glasses,” J. Phys. C 17, 3179 (1984).
  * Bray and Moore (1986) A. J. Bray and M. A. Moore, “Scaling Theory of the ordered phase of spin glasses,” in  _Heidelberg Colloquium on Glassy Dynamics and Optimization_ , edited by L. Van Hemmen and I. Morgenstern (Springer, New York, 1986) p. 121.
  * Fisher and Huse (1988) D. S. Fisher and D. A. Huse, “Equilibrium behavior of the spin-glass ordered phase,” Phys. Rev. B 38, 386 (1988).
  * Krzakala and Martin (2000) F. Krzakala and O. C. Martin, “Spin and Link Overlaps in Three-Dimensional Spin Glasses,” Phys. Rev. Lett. 85, 3013–3016 (2000).
  * Palassini and Young (2000) Matteo Palassini and A. P. Young, “Nature of the Spin Glass State,” Phys. Rev. Lett. 85, 3017–3020 (2000).
  * Newman and Stein (1998) C. M. Newman and D. L. Stein, “Simplicity of state and overlap structure in finite-volume realistic spin glasses,” Phys. Rev. E 57, 1356–1366 (1998).
  * Monthus (2015) C. Monthus, “Fractal dimension of spin-glasses interfaces in dimension $d=2$ and $d=3$ via strong disorder renormalization at zero temperature,” Fractals 23, 1550042 (2015).
  * Wang  _et al._ (2017) Wenlong Wang, M. A. Moore, and Helmut G. Katzgraber, “Fractal Dimension of Interfaces in Edwards-Anderson and Long-range Ising Spin Glasses: Determining the Applicability of Different Theoretical Descriptions,” Phys. Rev. Lett. 119, 100602 (2017).
  * Wang  _et al._ (2018) Wenlong Wang, M. A. Moore, and Helmut G. Katzgraber, “Fractal dimension of interfaces in Edwards-Anderson spin glasses for up to six space dimensions,” Phys. Rev. E 97, 032104 (2018).
  * Edwards and Anderson (1975) S. F. Edwards and P. W. Anderson, “Theory of spin glasses,” J. Phys. F: Met. Phys. 5, 965 (1975).
  * Baños  _et al._ (2010) R Alvarez Baños, A Cruz, L A Fernandez, J M Gil-Narvion, A Gordillo-Guerrero, M Guidetti, A Maiorano, F Mantovani, E Marinari, V Martin-Mayor, J Monforte-Garcia, A Muñoz Sudupe, D Navarro, G Parisi, S Perez-Gaviro, J J Ruiz-Lorenzo, S F Schifano, B Seoane, A Tarancon, R Tripiccione, and D Yllanes, “Nature of the spin-glass phase at experimental length scales,” Journal of Statistical Mechanics: Theory and Experiment 2010, P06026 (2010).
  * Yucesoy  _et al._ (2012) B. Yucesoy, Helmut G. Katzgraber, and J. Machta, “Evidence of Non-Mean-Field-Like Low-Temperature Behavior in the Edwards-Anderson Spin-Glass Model,” Phys. Rev. Lett. 109, 177204 (2012).
  * Billoire  _et al._ (2013) A. Billoire, L. A. Fernandez, A. Maiorano, E. Marinari, V. Martin-Mayor, G. Parisi, F. Ricci-Tersenghi, J. J. Ruiz-Lorenzo, and D. Yllanes, “Comment on “Evidence of Non-Mean-Field-Like Low-Temperature Behavior in the Edwards-Anderson Spin-Glass Model”,” Phys. Rev. Lett. 110, 219701 (2013).
  * Yucesoy  _et al._ (2013) B. Yucesoy, H. G. Katzgraber, and J. Machta, “Yucesoy, Katzgraber, and Machta Reply:,” Phys. Rev. Lett. 110, 219702 (2013).
  * Marinari and Parisi (2000) Enzo Marinari and Giorgio Parisi, “Effects of changing the boundary conditions on the ground state of Ising spin glasses,” Phys. Rev. B 62, 11677–11685 (2000).
  * Khoshbakht and Weigel (2018) Hamid Khoshbakht and Martin Weigel, “Domain-wall excitations in the two-dimensional Ising spin glass,” Phys. Rev. B 97, 064410 (2018).
  * Kawashima  _et al._ (1992) N Kawashima, N Hatano, and M Suzuki, “Critical behaviour of the two-dimensional EA model with a Gaussian bond distribution,” Journal of Physics A: Mathematical and General 25, 4985–5003 (1992).
  * Kawashima (1999) N Kawashima, “Fractal droplets in two-dimensional spin glass,” J. Phys. Soc. Jpn. 69, 987 (1999).
  * Berthier and Young (2003) Ludovic Berthier and A P Young, “Energetics of clusters in the two-dimensional Gaussian Ising spin glass,” Journal of Physics A: Mathematical and General 36, 10835–10846 (2003).
  * Hartmann and Young (2002) A. K. Hartmann and A. P. Young, “Large-scale low-energy excitations in the two-dimensional Ising spin glass,” Phys. Rev. B 66, 094419 (2002).
  * Picco  _et al._ (2003) M. Picco, F. Ritort, and M. Sales, “Statistics of lowest droplets in two-dimensional Gaussian Ising spin glasses,” Phys. Rev. B 67, 184421 (2003).
  * Hartmann and Moore (2003) A. K. Hartmann and M. A. Moore, “Corrections to Scaling are Large for Droplets in Two-Dimensional Spin Glasses,” Phys. Rev. Lett. 90, 127201 (2003).
  * Hartmann and Moore (2004) A. K. Hartmann and M. A. Moore, “Generating droplets in two-dimensional Ising spin glasses using matching algorithms,” Phys. Rev. B 69, 104409 (2004).
  * Fernandez  _et al._ (2019) L A Fernandez, E Marinari, V Martin-Mayor, G Parisi, and J J Ruiz-Lorenzo, “An experiment-oriented analysis of 2D spin-glass dynamics: a twelve time-decades scaling study,” Journal of Physics A: Mathematical and Theoretical 52, 224002 (2019).
  * Newman and Stein (2001) C. M. Newman and D. L. Stein, “Interfaces and the Question of Regional Congruence in Spin Glasses,” Phys. Rev. Lett. 87, 077201 (2001).
  * Middleton (2013) A. Alan Middleton, “Extracting thermodynamic behavior of spin glasses from the overlap function,” Phys. Rev. B 87, 220201 (2013).
  * Thomas  _et al._ (2011) Creighton K. Thomas, David A. Huse, and A. Alan Middleton, “Zero- and Low-Temperature Behavior of the Two-Dimensional $\pm{}J$ Ising Spin Glass,” Phys. Rev. Lett. 107, 047203 (2011).
  * Jörg  _et al._ (2006) T. Jörg, J. Lukic, E. Marinari, and O. C. Martin, “Strong Universality and Algebraic Scaling in Two-Dimensional Ising Spin Glasses,” Phys. Rev. Lett. 96, 237205 (2006).
  * Hatano and Gubernatis (2002) Naomichi Hatano and J. E. Gubernatis, “Evidence for the double degeneracy of the ground state in the three-dimensional $\pm{}J$ spin glass,” Phys. Rev. B 66, 054437 (2002).
  * F. Belletti et al. (2009) (Janus Collaboration) F. Belletti et al., “An in-depth view of the microscopic dynamics of Ising spin glasses at fixed temperature,” J. Stat. Phys. 136, 1121 (2009).
  * Baños  _et al._ (2012) Raquel Alvarez Baños, Andres Cruz, Luis Antonio Fernandez, Jose Miguel Gil-Narvion, Antonio Gordillo-Guerrero, Marco Guidetti, David Iñiguez, Andrea Maiorano, Enzo Marinari, Victor Martin-Mayor, Jorge Monforte-Garcia, Antonio Muñoz Sudupe, Denis Navarro, Giorgio Parisi, Sergio Perez-Gaviro, Juan Jesus Ruiz-Lorenzo, Sebastiano Fabio Schifano, Beatriz Seoane, Alfonso Tarancon, Pedro Tellez, Raffaele Tripiccione, and David Yllanes, “Thermodynamic glass transition in a spin glass without time-reversal symmetry,” PNAS 109, 6452 (2012).
  * Manssen  _et al._ (2015) Markus Manssen, Alexander K. Hartmann, and A. P. Young, “Nonequilibrium evolution of window overlaps in spin glasses,” Phys. Rev. B 91, 104430 (2015).
  * Billoire  _et al._ (2017) A. Billoire, L. A. Fernandez, A. Maiorano, E. Marinari, V. Martin-Mayor, J. Moreno-Gordo, G. Parisi, F. Ricci-Tersenghi, and J. J. Ruiz-Lorenzo, “Numerical Construction of the Aizenman-Wehr Metastate,” Phys. Rev. Lett. 119, 037203 (2017).
  * Baity-Jesi  _et al._ (2018) M. Baity-Jesi, E. Calore, A. Cruz, L. A. Fernandez, J. M. Gil-Narvion, A. Gordillo-Guerrero, D. Iñiguez, A. Maiorano, E. Marinari, V. Martin-Mayor, J. Moreno-Gordo, A. Muñoz Sudupe, D. Navarro, G. Parisi, S. Perez-Gaviro, F. Ricci-Tersenghi, J. J. Ruiz-Lorenzo, S. F. Schifano, B. Seoane, A. Tarancon, R. Tripiccione, and D. Yllanes (Janus Collaboration), “Aging Rate of Spin Glasses from Simulations Matches Experiments,” Phys. Rev. Lett. 120, 267203 (2018).
  * Newman and Stein (1996) C. M. Newman and D. L. Stein, “Spatial Inhomogeneity and Thermodynamic Chaos,” Phys. Rev. Lett. 76, 4821–4824 (1996).
  * Newman and Stein (1997) C. M. Newman and D. L. Stein, “Metastate approach to thermodynamic chaos,” Phys. Rev. E 55, 5194–5211 (1997).
  * Newman and Stein (1992) C. M. Newman and D. L. Stein, “Multiple states and thermodynamic limits in short-ranged Ising spin-glass models,” Phys. Rev. B 46, 973–982 (1992).
  * Newman and Stein (2003b) C M Newman and D L Stein, “Ordering and broken symmetry in short-ranged spin glasses,” Journal of Physics: Condensed Matter 15, R1319–R1364 (2003b).
  * Arguin  _et al._ (2015) L.-P. Arguin, C. M. Newman, and D. L. Stein, “Thermodynamic identities and symmetry breaking in short-range spin glasses,” Phys. Rev. Lett. 115, 187202 (2015).
  * White and Fisher (2006) Olivia L. White and Daniel S. Fisher, “Scenario for Spin-Glass Phase with Infinitely Many States,” Phys. Rev. Lett. 96, 137204 (2006).
  * Read (2014) N. Read, “Short-range Ising spin glasses: The metastate interpretation of replica symmetry breaking,” Phys. Rev. E 90, 032142 (2014).
  * Aizenman and Wehr (1990) M. Aizenman and J. Wehr, “Rounding Effects of Quenched Randomnesson First Order Phase Transitions,” Commun. Math. Phys. 130, 489 (1990).
  * Wittmann and Young (2016) Matthew Wittmann and A P Young, “The connection between statics and dynamics of spin glasses,” Journal of Statistical Mechanics: Theory and Experiment 2016, 013301 (2016).
  * Baity-Jesi  _et al._ (2017) M. Baity-Jesi, E. Calore, A. Cruz, L. A. Fernandez, J. M. Gil-Narvion, A. Gordillo-Guerrero, D. Iñiguez, A. Maiorano, E. Marinari, V. Martin-Mayor, J. Monforte-Garcia, A. Muñoz Sudupe, D. Navarro, G. Parisi, S. Perez-Gaviro, F. Ricci-Tersenghi, J. J. Ruiz-Lorenzo, S. F. Schifano, B. Seoane, A. Tarancon, R. Tripiccione, and D. Yllanes (Janus Collaboration), “Matching Microscopic and Macroscopic Responses in Glasses,” Phys. Rev. Lett. 118, 157202 (2017).
  * Zhai  _et al._ (2020) Q. Zhai, I. Paga, M. Baity-Jesi, E. Calore, A. Cruz, L. A. Fernandez, J. M. Gil-Narvion, I. Gonzalez-Adalid Pemartin, A. Gordillo-Guerrero, D. Iñiguez, A. Maiorano, E. Marinari, V. Martin-Mayor, J. Moreno-Gordo, A. Muñoz Sudupe, D. Navarro, R. L. Orbach, G. Parisi, S. Perez-Gaviro, F. Ricci-Tersenghi, J. J. Ruiz-Lorenzo, S. F. Schifano, D. L. Schlagel, B. Seoane, A. Tarancon, R. Tripiccione, and D. Yllanes, “Scaling Law Describes the Spin-Glass Response in Theory, Experiments, and Simulations,” Phys. Rev. Lett. 125, 237202 (2020).
  * (53) I Paga, Q Zhai, M Baity-Jesi, E Calore, A Cruz, L A Fernandez, J M Gil-Narvion, I Gonzalez-Adalid Pemartin, A Gordillo-Guerrero, D Iñiguez, A Maiorano, E Marinari, V Martin-Mayor, J Moreno-Gordo, A Muñoz-Sudupe, D Navarro, R L Orbach, G Parisi, S Perez-Gaviro, F Ricci-Tersenghi, J J Ruiz-Lorenzo, S F Schifano, D L Schlagel, B Seoane, A Tarancon, R Tripiccione, and D Yllanes, “Spin-glass dynamics in the presence of a magnetic field: exploration of microscopic properties,” Journal of Statistical Mechanics: Theory and Experiment 2021, 033301.
  * Chalupa (1977) J. Chalupa, “The Susceptibilities of Spin Glasses,” Solid State Communications 22, 315 (1977).
  * Bray and Moore (1979) A. J. Bray and M. A. Moore, “Replica symmetry and massless modes in the Ising spin glass,” Journal of Physics C: Solid State Physics 12, 79 (1979).
  * Hartmann (1999a) Alexander K. Hartmann, “Scaling of stiffness energy for three-dimensional $\pm{}J$ Ising spin glasses,” Phys. Rev. E 59, 84–87 (1999a).
  * Boettcher (2005) Stefan Boettcher, “Stiffness of the Edwards-Anderson Model in all Dimensions,” Phys. Rev. Lett. 95, 197205 (2005).
  * Nicolao  _et al._ (2014) Lucas Nicolao, Giorgio Parisi, and Federico Ricci-Tersenghi, “Spatial correlation functions and dynamical exponents in very large samples of four-dimensional spin glasses,” Phys. Rev. E 89, 032127 (2014).
  * Hartmann (1999b) Alexander K. Hartmann, “Calculation of ground states of four-dimensional $\pm{}J$ Ising spin glasses,” Phys. Rev. E 60, 5135–5138 (1999b).
  * Aspelmeier  _et al._ (2016a) T. Aspelmeier, Wenlong Wang, M. A. Moore, and Helmut G. Katzgraber, “Interface free-energy exponent in the one-dimensional Ising spin glass with long-range interactions in both the droplet and broken replica symmetry regions,” Phys. Rev. E 94, 022116 (2016a).
  * Zhai  _et al._ (2019) Qiang Zhai, V. Martin-Mayor, Deborah L. Schlagel, Gregory G. Kenning, and Raymond L. Orbach, “Slowing down of spin glass correlation length growth: Simulations meet experiments,” Phys. Rev. B 100, 094202 (2019).
  * Huse (1991) David A. Huse, “Monte Carlo simulation study of domain growth in an Ising spin glass,” Phys. Rev. B 43, 8673–8675 (1991).
  * De Dominicis  _et al._ (2005) Cirano De Dominicis, Irene Giardina, Enzo Marinari, Olivier C. Martin, and Francesco Zuliani, “Spatial correlation functions in three-dimensional Ising spin glasses,” Phys. Rev. B 72, 014443 (2005).
  * Bernaschi  _et al._ (2020) Massimo Bernaschi, Alain Billoire, Andrea Maiorano, Giorgio Parisi, and Federico Ricci-Tersenghi, “Strong ergodicity breaking in aging of mean-field spin glasses,” Proceedings of the National Academy of Sciences 117, 17522–17527 (2020).
  * Viana and Bray (1985) L Viana and A J Bray, “Phase diagrams for dilute spin glasses,” Journal of Physics C: Solid State Physics 18, 3037–3051 (1985).
  * de Almeida and Thouless (1978) J. R. L. de Almeida and D. J. Thouless, “Stability of the Sherrington-Kirkpatrick solution of a spin glass model,” J. Phys. A 11, 983 (1978).
  * Bray and Roberts (1980) A. J. Bray and S. A. Roberts, “Renormalisation-group approach to the spin glass transition in finite magnetic fields,” Journal of Physics C: Solid State Physics 13, 5405 (1980).
  * Moore and Bray (2011) M. A. Moore and A. J. Bray, “Disappearance of the de Almeida-Thouless line in six dimensions,” Phys. Rev. B 83, 224408 (2011).
  * Moore (2012) M. A. Moore, “$1/m$ expansion in spin glasses and the de Almeida-Thouless line,” Phys. Rev. E 86, 031114 (2012).
  * Mattsson  _et al._ (1995) J. Mattsson, T. Jonsson, P. Nordblad, H. Aruga Katori, and A. Ito, “No Phase Transition in a Magnetic Field in the Ising Spin Glass F${\mathrm{e}}_{0.5}$M${\mathrm{n}}_{0.5}$Ti${\mathrm{O}}_{3}$,” Phys. Rev. Lett. 74, 4305–4308 (1995).
  * Larson  _et al._ (2013) Derek Larson, Helmut G. Katzgraber, M. A. Moore, and A. P. Young, “Spin glasses in a field: Three and four dimensions as seen from one space dimension,” Phys. Rev. B 87, 024414 (2013).
  * Yeo and Moore (2015) Joonhyun Yeo and M. A. Moore, “Critical point scaling of Ising spin glasses in a magnetic field,” Phys. Rev. B 91, 104432 (2015).
  * Aspelmeier  _et al._ (2016b) T. Aspelmeier, Helmut G. Katzgraber, Derek Larson, M. A. Moore, Matthew Wittmann, and Joonhyun Yeo, “Finite-size critical scaling in Ising spin glasses in the mean-field regime,” Phys. Rev. E 93, 032123 (2016b).