# Practitioner–generated blog posts as evidence for software engineering
research: attitudinal survey and preliminary checklist

Austen Rainer
Queens University Belfast
Belfast, UK
<EMAIL_ADDRESS>Ashley Williams111Corresponding Author
Manchester Metropolitan University
Manchester, UK
<EMAIL_ADDRESS>
(June 2020)

###### Abstract

Background: Blog posts are frequently used by software practitioners to share
information about their practice. Blog posts therefore provide a potential
source of evidence for software engineering (SE) research. The use of blog
posts as evidence for research appears contentious amongst some SE
researchers. There is also the significant challenge of assuring the quality
of blog posts for such research.

Objective: To better understand the actual and perceived value of blog posts
as evidence for SE research, and to develop guidance for SE researchers on the
use of blog posts as evidence.

Method: We further analyse responses from a previously conducted attitudinal
survey of 44 software engineering researchers. We conduct a heatmap analysis,
simple statistical analysis, and a thematic analysis.

Results: We find no clear consensus from respondents on researchers’ attitudes
to the credibility of blog posts, or on a standard set of criteria to evaluate
blog–post credibility. We show that some of the responses to the survey
exhibit characteristics similar to the content of blog posts, e.g., asserting
prior beliefs as claims, with no citations and little supporting rationale. We
illustrate our insights with ~60 qualitative examples from the survey (~40% of
the total responses). We complement our quantitative and qualitative analyses
with preliminary checklists to guide SE researchers in their use of blog posts
as evidence.

Conclusion: Blog posts are relevant to research because they are written by
software practitioners describing their practice and experience. But
evaluating the credibility of blog posts, so as to select the higher–quality
content, remains an ongoing challenge. The quantitative and qualitative
results, together with the proposed checklists, are intended to stimulate
reflection and action in the research community on the role of blog posts as
evidence in software engineering research. Finally, our findings on
researchers’ attitudes to blog posts also provide more general insights into
researchers’ values for SE research.

Keywords: survey, blog posts, grey literature, software practice, data quality

###### Contents

  1. 1 Introduction
    1. 1.1 Context and motivation
    2. 1.2 Research Questions
    3. 1.3 Contributions
    4. 1.4 Structure of this paper
  2. 2 Related work
    1. 2.1 The use of blog posts in SE research
    2. 2.2 Blog posts and classifications of evidence
    3. 2.3 Standards of evidence
    4. 2.4 Quality–assurance of publications
    5. 2.5 Summary
  3. 3 Survey Design
    1. 3.1 Our previous study
    2. 3.2 Candidate criteria
    3. 3.3 Survey development and refinement
    4. 3.4 Participants
    5. 3.5 Post–survey follow–up
    6. 3.6 Excluded–case analyses
    7. 3.7 Quantitative analysis
    8. 3.8 Thematic analyses
    9. 3.9 Public access to data
  4. 4 Summary of previous results
    1. 4.1 The general credibility of blog posts
    2. 4.2 Summary of results from initial analysis
  5. 5 Exploratory analyses of the quantitative data
    1. 5.1 Summary of responses
    2. 5.2 Heatmap analysis
    3. 5.3 Actual response proportions relative to theoretical range
    4. 5.4 Confidence intervals
    5. 5.5 Correlation analysis
    6. 5.6 Summary
  6. 6 Thematic analyses
    1. 6.1 Introduction
    2. 6.2 General comments on respondents and responses
    3. 6.3 Author bias, subjectivity and sincerity
    4. 6.4 Topic and content
    5. 6.5 Professional experience
    6. 6.6 The credibility of the author
    7. 6.7 Anecdotal, (a)contextual information and evidence
  7. 7 Preliminary checklists for evaluating blog posts
    1. 7.1 Overview
    2. 7.2 Applying the checklist to the survey itself
  8. 8 Discussion
    1. 8.1 Reviewing the research questions
    2. 8.2 Implications of the findings
    3. 8.3 Threats to validity
    4. 8.4 Further research
    5. 8.5 Conclusions

###### List of Figures

  1. 1 The general credibility of practitioner blog posts (n=43, Mod=3, Med=3, Mn=2.7)
  2. 2 Stacked barchart of responses to the nine criteria
  3. 3 Heatmap of responses
  4. 4 Bubble charts with Loess smoother and confidence interval
  5. 5 Bubble charts with Loess smoother and confidence interval
  6. 6 Dotplots with Loess smoother and confidence interval
  7. 7 Dotplots with Loess smoother and confidence interval

###### List of Tables

  1. 1 Research into software practice using blogs
  2. 2 Quality checklists.
  3. 3 Summary of the nine criteria from the literature review
  4. 4 Confidence intervals for proportions of responses for the nine criteria
  5. 5 Spearman Rho rank correlations for the nine credibility criteria
  6. 6 General observations on respondents and responses
  7. 7 Observations on author subjectivity, blog content, and professional experience
  8. 8 The credibility of the author (grouped by GC score)
  9. 9 Context and anecdote (grouped by GC score)
  10. 10 Summary of the checklists
  11. 11 Checklist
  12. 12 Checklist (continued)
  13. 13 Checklist (continued)
  14. 14 Checklist (continued)
  15. 15 Characteristics of the responses from survey respondents
  16. 16 Statistics on researchers’ responses
  17. 17 Statistics on Joel Spolsky’s blog posts (n=1024)
  18. 18 Blog posts compared with interviews
  19. 19 Summary of survey questions

##  1 Introduction

###  1.1 Context and motivation

The emergence of social media and of the social programmer (Storey et al.,
2010, 2014) has dramatically changed the way that software practitioners share
information about their practice. Blog posts are one type of social media and
therefore provide a potential source of evidence. Blog posts have, however,
been used only sporadically as evidence in software engineering (SE) research
e.g., (Aniche et al., 2018; Parnin et al., 2013; Parnin and Treude, 2011;
Pagano and Maalej, 2011).

There are significant challenges to using blog posts in research (Rainer and
Williams, 2018, 2019). We consider two, related challenges in this paper.
First, there is the challenge of quality–assuring blog posts to help ensure
that research uses higher–quality evidence. For example, Garousi et al. (2018)
and Soldani et al. (2018) both developed checklists for the quality–assurance
of grey literature, however neither checklist was developed specifically for
blog posts. Second, informal communication — for example, at conferences and
via reviewers’ comments on our submitted manuscripts — suggests that the use
of blog posts in research (and, more generally, the use of grey literature) is
contentious for at least some researchers within the community. A research
community comprises a diversity of perspectives, values and goals. There is
therefore the subtler challenge of understanding the research community’s
diversity of attitudes to blog posts as evidence for research.

In a previous paper (Williams and Rainer, 2019a), a conference paper, we
reported initial results of a survey we conducted to study SE researchers’
attitudes to the credibility of blog posts, and to criteria for evaluating
such credibility. We observed in that paper that respondents adopt a
conditional evaluation of the credibility of blog posts: essentially, it
depends. The survey responses — there were approximately 150 qualitative
comments from 37 of the 44 respondents — are clearly complex and rich in
detail. The responses, as well as the two challenges we identify above, all
warrant more attention than we were able to direct in the conference paper. We
therefore further analyse the survey responses and report that analyses in
this paper. We complement the analyses with checklists synthesised from the
analyses of the survey and from prior literature.

###  1.2 Research Questions

For the current paper, we investigate the following research questions:

  1. 1.

To what degree do researchers consider blog posts to be credible? We explored
this question in our conference paper (Rainer and Williams, 2018) and
re–consider it here in more detail.

  2. 2.

What criteria do researchers claim to use when evaluating the credibility of a
blog post? We consider both the pre–defined criteria identified a priori for
the survey, and the additional criteria that are either implied or explicitly
proposed by survey respondents.

  3. 3.

What guidance on the quality–assurance of blog posts can be synthesised from
prior research and the survey responses? Such guidance is helpful for
establishing a standard with which researchers can evaluate blog posts.

  4. 4.

How do the criteria that the respondents say they use compare with the
responding behaviour of those respondents in the survey itself? For example, a
respondent may identify the lack of citations in blog posts as an indicator
that blog posts lack credibility, but then the respondent may themselves not
cite research in their survey responses.

###  1.3 Contributions

The current paper substantially extends our previous analyses to make five
contributions. First, we present the heatmap analysis of the quantitative
responses from the survey, together with statistics complementing that
heatmap. The heatmap and complementary statistics provide a much richer
representation of the diversity of researchers’ attitudes than the simple, and
simplifying, descriptive statistics we report in the first paper (Williams and
Rainer, 2019a). Second, we present a thematic analysis of all the qualitative
responses. Our first paper simply reported a small selection of illustrative
qualitative responses. Third, we present a rich sample of responses
(approximately 40% of the dataset). One reason we present that rich sample is
transparency to our analysis. But the main reason we present the sample is to
encourage the reader to reflect on their own attitudes to blog posts as
evidence in SE research. Fourth, and as a complement to our third
contribution, we show that respondents’ ‘behaviour’ in the survey (i.e., the
content of some of the survey responses) has similarities with software
practitioners’ behaviour in their blog posts, e.g., both respondents and
blog–post writers make claims, present limited reasoning, few examples, and no
citations. In other words, we show that at least some threats to the
credibility of blog posts are also threats to more traditional methods of data
collection, such as surveys and interviews. As with our rich sample, we use
this comparison of behaviours to encourage SE readers to reflect on their
attitudes. Fifth, and finally, we synthesise the quantitative and qualitative
analyses, together with insights from prior work, into a set of preliminary
checklists that researchers can use to help them evaluate blog posts.

Overall, our analyses — first reported in (Williams and Rainer, 2019a) and now
substantially extended in the current paper — provides the first empirical
investigation of software engineering researchers’ attitudes to the value of
blog posts for software engineering research, and provides the first specific
guidance within the SE research community for evaluating the credibility of
blog posts.

###  1.4 Structure of this paper

The remainder of the paper is structured as follows. Section 2 considers
relevant, previous work, including our previous paper (Williams and Rainer,
2019a). Section 3 presents and discusses the design of the survey. Section 4
briefly re–reports and summarises the quantitative results from our previous
paper so as to provide context for the additional analyses reported in the
current paper. Section 5 presents the heatmap analysis of the quantitative
responses, together with statistics complementing that heatmap. Section 6
presents the thematic analysis of the qualitative responses. Section 7
proposes checklists for evaluating facets of blog posts. The section then uses
the checklists to evaluate the survey responses, comparing those with blog
posts. Finally, section 8 discusses our findings, considers threats to
validity, identifies several directions for further research, and briefly
concludes.

##  2 Related work

###  2.1 The use of blog posts in SE research

Table 1 presents examples of empirical research into software practice that
make use of blog posts. As the table indicates, blog posts have been used in
software engineering research for about a decade, albeit sporadically, with
the quantity of blog posts used in the research ranging from one blog post
(Rainer, 2017) to 50,000 blog posts (Pagano and Maalej, 2011).

The examples presented in the table are organised into three broad categories:
1) primary studies that use blog posts to gain insights into software
practice; 2) secondary studies that use grey literature, such as blog posts,
to gain insights into software practice; and 3) guidelines papers that present
advice on the use of grey literature (including blog posts) in research, these
guidelines including quality checklists.

Our focus in this paper is on the use of blog posts as evidence for software
engineering research. We recognise that there are other fields of research
that use blog posts. These fields include opinion mining, sentiment analyses,
argumentation mining and experience mining, e.g., (Swanson et al., 2014;
Gordon and Swanson, 2009; Burton et al., 2009; Park et al., 2010; Kurashima et
al., 2009; Inui et al., 2008; Kurashima et al., 2006). Other research, e.g.,
(Lakshmanan and Oberhofer, 2010; Chau et al., 2009; Khan et al., 2017), is
also interested in blog posts but not necessarily empirically, or with the
intent to study software practice.

The three categories identified in Table 1 approach the credibility of blog
posts in contrasting ways. Secondary studies and guidelines develop and apply
quality checklists for assessing the quality of grey literature, with Garousi
et al. (2018) proposing a more generic checklist as part of their guidelines
for Multivocal Literature Reviews (MLRs)222There are important differences
between the systematic review of previously published primary studies and the
grey literature review of blog posts.. Primary studies of software practice
tend to accept the quality of selected blog posts without evaluation (e.g.,
there is no post–selection quality evaluation) however in their respective
primary studies Williams (2018b) used reasoning indicators as a proxy for
quality, and Rainer (2017) selected a blog post from the blog of one highly
regarded practitioner.

Table 1: Research into software practice using blogs First author | Year |  # blog posts
---|---|---
Guidelines on secondary studies of GL in research
(Garousi et al., 2018) | 2018 |  N/A
Secondary studies that use blog posts in research
(Soldani et al., 2018) | 2018 |  20/51 blog posts (40% of dataset)
(Raulamo-Jurvanen et al., 2017) | 2017 |  60 GL sources
(Garousi and Mäntylä, 2016) | 2016 |  46 internet articles & white papers
Primary studies that use blog posts in research
(Williams, 2018a) | 2018 |  2852 blogs posts
(Rainer, 2017) | 2017 |  One blog post
(Parnin et al., 2013) | 2013 |  300 blog posts
(Parnin and Treude, 2011) | 2011 |  376 blog posts
(Pagano and Maalej, 2011) | 2011 |  50,000 blog posts

###  2.2 Blog posts and classifications of evidence

The research community already classifies blog posts as a candidate source of
evidence. We consider two examples of that recognition here: Prechelt and
Petre’s (2010) discussion of credibility, and Wohlin’s (2013) evidence profile
for software engineering research and practice.

Prechelt and Petre (2010) identify five sources of evidence: (direct)
experience, other people, reflection, reading, and scientific (or
quasi–scientific) exploration. For the category of reading, they state,
“Written materials, both informal (such as high–quality blog posts) or formal
(such as scientific articles) transport insights from other parties.”
(emphasis added here). There are relationships between these sources of
evidence. Two relationships between the sources are particularly relevant to
the current paper. First, a practitioner gains experience (we italicise the
categories proposed by Prechelt and Petre), reflects on that experience, and
then shares that reflective experience as a blog post for others to read.
Second, a practitioner gains experience, reflects on that experience, and then
shares that reflective experience with researchers as part of a scientific
exploration that is then published for others to read. Prechelt and Petre
(2010) highlight an important qualifier, of course: the reading of
high–quality blog posts. This is precisely the category of blog post that our
work (in the current paper and elsewhere) aims to identify.

Wohlin (2013) developed an evidence profile to be used to organise different
types of evidence for a particular case. Wohlin identifies seven sources of
information, from the strongest source to the weakest, and discounts the three
weakest sources as not evidence. Of the remaining four sources, two are
relevant here: statements from trustworthy witnesses, and statements from
expert witnesses. Blog posts may contain information from trustworthy
witnesses and expert witnesses. As with Prechelt and Petre (2010), there is an
important qualifier: identifying those blog posts that are written by
trustworthy witnesses and expert witnesses.

###  2.3 Standards of evidence

Prechelt and Petre (2010) also recognise that “Different purposes require
different standards of evidence.” ((Prechelt and Petre, 2010); emphasis in
original). (Devanbu et al., 2016) and (Rainer et al., 2003) both show that
practitioners tend to hold a different standard of evidence to researchers:
practitioners prefer the opinion of other practitioners as their primary
source of evidence. Our interest in the current paper is of course on
researchers’ standards.

A further complication is that researchers hold different worldviews. Petersen
and Gencel (2013) consider the relationship between worldviews, research
methods and validity. They assert that the dominant worldview in empirical
software engineering is the pragmatist worldview, in which multiple research
methods from the other worldviews (i.e., positivist, constructivist, and
participative) are used. For Petersen and Gencel, the multi–method approach
creates difficulties in reporting threats to validity, and creates confusion
when evaluating research. Petersen and Gencel’s work highlights the challenges
of defining a standard or standards for evidence. Our analysis later in this
paper provides insights into the diversity of attitudes of researchers. Also,
in developing a suite of checklists, we contribute toward the establishment of
a standard (or standards) for evaluating blog posts as evidence.

Aside from the challenges that arise from different worldviews, there is a
broad distinction between the relevance of evidence, the validity of that
evidence, and the rigour of the process by which that evidence has been
generated. We draw on two prior publications to consider relevance, validity
and rigour: Prechelt and Petre (2010)’s book chapter on credibility, and
Ivarsson and Gorschek’s (2011) evaluation of the industrial relevance and
rigour of technology evaluations in software engineering.

Prechelt and Petre (2010 broadly distinguish between relevance and credibility
(for Prechelt and Petre, credibility comprises high validity and good
reporting). An implication from Prechelt and Petre (2010)’s position on
relevance and credibility is that an evaluator would typically consider
relevance first, and then go on to consider the rigour of relevant claims and
evidence. It follows that blog posts might be relevant to a topic of interest
to a researcher, for which these blog posts are then evaluated for their
validity of claims, and possibly for the rigour by which the claims were
formed.

Ivarsson and Gorschek (2011) develop a model for evaluating the industrial
relevance and rigour of technology evaluations in software engineering. They
define methodological rigour in terms of conformance to standards for that
methodology i.e., a study is methodologically rigorous if that study conforms
to guidelines, protocols and best practices for that methodology. One way in
which researchers quality–assure evidence published in research papers is by
checking the rigour of the process by which the evidence was generated. One
significant difficulty with blog posts is that there is no standard, de facto
or otherwise, for the writing of a blog post, or for the enquiry process that
informs the claims then stated in the blog post, or indeed for the reporting
of that enquiry process.

###  2.4 Quality–assurance of publications

Over many years, the empirical software engineering community has reflected on
how to quality–assure publications. For example, twenty–five years ago, Fenton
et al. (1994) proposed several questions to ask about any claim arising from
software engineering research. And in a founding paper on Evidence Based
Software Engineering (EBSE) Dybå et al. (2005) proposed five questions to help
practitioners assess the credibility of evidence: 1) Is there any vested
interest? 2) Is the evidence valid?, 3) Is the evidence important? 4) Can the
evidence be used in practice? 5) Is the evidence consistent with evidence from
other studies?

More recently, questions asking about claims and evidence have been structured
into checklists. Two checklists are particularly relevant for the current
paper because both relate to grey literature: Garousi et al.’s (2018) generic
checklist for MLRs, and Soldani et al.’s (2018) specific checklist for a GLR
of microservices. We summarise these checklists here in Table 2. Later in this
paper we integrate these checklists with the quantitative and qualitative
results from our survey.

Table 2: Quality checklists. |  Question
---|---
Garousi et al.’s (2018) quality checklist.
Authority of the producer
|  Is the publishing organization reputable?
|  Is an individual author associated with a reputable organization?
|  Has the author published other work in the field?
|  Does the author have expertise in the area?
Methodology
|  Does the source have a clearly stated aim?
|  Does the source have a stated methodology?
|  Is the source supported by authoritative, contemporary references?
|  Are any limits clearly stated?
|  Does the work cover a specific question?
|  Does the work refer to a particular population or case?
Objectivity
|  Does the work seem to be balanced in presentation?
|  Is the statement in the sources as objective as possible? Or, is the
statement a subjective opinion?
|  Is there vested interest?
|  Are the conclusions supported by the data?
Date
|  Does the item have a clearly stated date?
Position regarding related resources
|  Have key related grey literature or formal sources been linked to /
discussed?
Novelty
|  Does it enrich or add something unique to the research?
|  Does it strengthen or refute a current position?
Impact: A normalisation of several impact metrics
|  Number of citations, backlinks, media shares, comments, views
Outlet type (see (Garousi et al., 2018))
|  1st Tier, 2nd Tier, 3rd Tier
Soldani et al.’s (2018) quality checklist.
# |  Explanation
Inclusion criteria
I1 |  The study discusses the industrial application of microservices.
I2 |  The study discusses the benefits or shortcomings of microservice design, development or operation.
I3 |  The study reports on direct experiences, opinions or practices on microservices by educated practitioners.
I4 |  The study refers to a practical case–study of design, development or operation of microservices.
Exclusion criteria
E1 |  The study does not offer details on design or implementation of microservices.
E2 |  The study is not referred to industrial cases or other factual evidence.
E3 |  The benefits or pitfalls of microservices are not justified/quantified by the study.
E4 |  The study does not provide scope and limitations of proposed solutions/patterns.
E5 |  The study does not offer evidence of a practitioner perspective.
Additional control factors
C1 |  Practical experience: A study is to be selected only if it is written by practitioners with 5+ experience in service-oriented design, development and operation, or if it refers to established microservices solutions with 2+ years of operation.
C2 |  Industrial case-study: A study is to be selected only if it refers to at least 1 industrial case-study where a quantifiable number of microservices are operated.
C3 |  Heterogeneity: The selected studies reflect at least 5 top industrial domains and markets where microservices were successfully applied.
C4 |  Implementation quantity: The selected studies refer to/show implementation details for the benefits and pitfall they discuss, so that other researchers and practitioners can use them in action.

As well as checklists, there are a range of guidelines available, including:
Kitchenham and Charters’ (2007) guidelines for conducting systematic reviews;
Petersen et al.’s (2015) guidelines for systematic mapping studies; Garousi et
al.’s (2018) guidelines for MLRs; Runeson and Höst (2009) guidelines and
checklists for case study research (see also (Runeson et al., 2012));
Kitchenham and Pfleeger’s (2002) guidelines for survey research; and
Kitchenham et al.’s (2002) general guidelines on empirical research. Whilst
all of these questions, checklists and guidelines assert, or prescribe, how to
evaluate information, none of these guidelines are particular to blog posts.

###  2.5 Summary

Blog posts are already being used in SE research and blog posts are already
recognised within existing classifications of evidence. There are concerns
about the credibility of blog posts and these concerns broadly relate to the
validity of the statements made in blog posts as well as the rigour of the
processes of enquiry (if any) that inform those statements. Prior to a
consideration of validity or of rigour, the evaluator might (typically) first
consider the relevance of the blog post, i.e., whether the content of the blog
post is relevant to the research topic of interest to the researcher. In
principle, many blog posts would be relevant to research because those blog
posts refer to aspects of software practice. There are, legitimately,
different standards to evidence. We are of course interested here in the
standards of evidence as they concern researchers, and therefore the standards
that researcher do or ought to place on blog posts as evidence for research.
Finally, different worldviews influence what the respective researchers
consider acceptable standards of evidence and therefore influence what any
given researcher would accept as a relevant and valid blog post for research.

##  3 Survey Design

###  3.1 Our previous study

Given the current status of software engineering research that uses blog
posts, we decided to conduct an investigation into researchers’ attitudes to
the credibility of blog posts. We discuss the design of our survey later in
this section. For the current subsection, we briefly present a critical review
of the first paper (Williams and Rainer, 2019a) we published that reported the
preliminary results of the survey.

Our initial analysis (Williams and Rainer, 2019a) suggested a complex
relationship between researchers’ general attitude to the credibility of blog
posts and the researchers’ credibility criteria for evaluating blog posts. As
one example, and as we show later in this paper, it appears that some
researchers’ place a greater value on criteria such as reasoning and
professional experience, with relatively less value placed on reporting
empirical data and methods of data collection. By contrast, other researchers
placed greater value on reporting empirical data and methods of data
collection. These contrasting positions appear consistent with Prechelt and
Petre’s (2010) assertion about different standards of evidence, and Petersen
and Gencel’s (2013) arguments concerning the effect of worldview on the
assessment of validity.

It became clear, as we continued to analyse the data, that there were several
limitations to our survey design. For example, the dataset is relatively small
(as is the relative size of the software engineering research community),
there were a limited number of questions in our survey (though this was
deliberate to encourage responses to the survey) and the nature and range of
permissible responses (because of the Likert scaling) limited the ability to
explore researchers’ attitudes. We did however allow for qualitative
responses.

For the current paper, we report a more detailed analyses of the data. We
report quantitative analyses, comprising a heatmap and statistical analysis
(whilst recognising that meaningful statistical analyses of the data may be
both difficult and potentially misleading). We complement the quantitative
analyses with a thematic analysis of the qualitative responses.

Initial results for the survey, together with a full description of the survey
design, were first reported in (Williams and Rainer, 2019a). For completeness,
we restate in this section the description of the survey design. We complement
that description with a brief explanation of our heatmap analysis of the
quantitative data and our thematic analysis of the qualitative data. In
section 4 we briefly re–report the results from the first paper.

###  3.2 Candidate criteria

To inform the design of our survey, we conducted a broad review of previous
empirical studies of credibility. Many of these studies were published outside
of software engineering research. We identified 833 candidate articles for
review, reducing that candidate set to a final set of 13 papers. The full
details of the literature review are reported in a technical report (Williams
and Rainer, 2018).

From the set of 13 papers, we identified 88 candidate criteria, which we
subsequently distilled to the nine criteria we used in the survey. These nine
criteria are summarised in Table 3. Respondents to the survey identified
additional criteria which we discuss later in this paper.

Table 3: Summary of the nine criteria from the literature review Criterion |  Acronym
---|---
Clarity of writing |  CoW
Reporting empirical data |  RED
Reporting the method of data collection |  RM
Reporting professional experience |  PExp
Reasoning |  Rsn
Citing practitioner sources |  URL–P
Citing research sources |  URL–R
Prior beliefs of the reader |  PB
Prior beliefs of others who influence the reader |  IoO

###  3.3 Survey development and refinement

We developed a draft survey that was reviewed by four colleagues who were
familiar with the research, and revised the survey in response to their
feedback. We then conducted a pilot study, inviting responses from a network
of software engineering researchers within New Zealand
(SI^NZ333http://softwareinnovation.nz/). Based on the feedback from the pilot
study we clarified some survey questions. The final survey comprised 20
questions, comprising 12 main questions of which one question had nine
sub–questions. The questions and permissible answers are summarised in the
Appendix.

We used the Qualtrics444https://qualtrics.com online survey instrument to
administer the survey. An invitation email was sent to each participant, with
instructions and an anonymous link to the survey. The survey was approved by
the appropriate University of Canterbury (New Zealand) Ethics Committee (HEC
2017/68/LR-PS).

The main questions in the survey asked about: 1) the researcher’s opinion on
the general credibility of blog articles, 2) the criteria they (say they) use
to assess the credibility of blog articles, 3) whether there are any criteria
that we have not identified; 4) whether they think the criteria generalise to
other grey literature, 5) whether they think the criteria generalise to
research, and 6) whether they had any other comments to make on the survey.
Each of these questions included the opportunity for the respondent to provide
open–ended, additional comments. It is these comments that form the bases for
the thematic analysis.

As optional questions, we also asked respondents for their contact details,
whether they would be willing to participate in a follow–up interview, and
whether they would like to receive an anonymised copy of the data. 21
respondents (47%) confirmed they would like a copy of the data, and the
anonymised raw data has been sent to those respondents. The full list of
questions is available online at
https://www.researchgate.net/publication/324784268_Design_of_a_survey_on_credibility.

###  3.4 Participants

The survey was conducted between 13th February 2018 and 26th March 2018.
Invitations were sent out to the Programme Committees of the Evaluation and
Assessment in Software Engineering (EASE) conference and the Empirical
Software Engineering and Measurement (ESEM) conference for that year.
‘Overlapping’ members for each Programme Committee were emailed only once, and
respondents involved in the development and refinement of the survey were not
invited to participate in the full survey.

Overall, 138 researchers were invited to participate. Four of these invitees
asked us whether they could forward the survey to their colleagues. We
approved these requests but were not able to track increase in numbers of
invited participants. Consequently, we are unable to precisely report the
number of people who actually received the invitation. 57 invitees started the
survey and 44 completed it, giving a response rate of 32% (assuming 138
invitees). The participants’ experience in research ranged from two years to
35 years, with a mean average of 16.2 years.

We intentionally designed a short survey to encourage participation. We
estimated, from our draft and pilot study, that the survey would take about 10
minutes to complete. The total time taken to complete the survey ranged from
2.4 minutes to 22 hours with an overall average of 75.7 minutes. Ignoring the
completion times of the five responses that took longer than one hour to
complete gives a range from 2.4 minutes to 47.1 minutes, with an average time
of 11.7 minutes.

###  3.5 Post–survey follow–up

We performed two stages of post–survey follow–up. For the first stage, at the
close of the survey, we contacted invitees to ask them for reasons for why she
or he did not start the survey, or started the survey but did not complete it.
As the survey was anonymous, we emailed all invitees as we could not email
only those who had not completed the study. A similar follow–up was also
conducted with the SI^NZ trial. The reasons given with both survey–trial
follow–up and the full survey follow–up was that respondents were too busy to
start, or complete, the survey.

For the second stage, upon completion of the initial analysis and acceptance
of the first paper (Williams and Rainer, 2019a), we sent a copy of the results
to those respondents who had expressed an interest in receiving the results.
As noted earlier, 21 respondents (47%) were sent a copy of the data.

###  3.6 Excluded–case analyses

During the quantitative analysis for our first paper (Williams and Rainer,
2019a), we identified one respondent — respondent #22 in Table 3 — who scored
0 (zero; no blog post is credible) to the question asking about the general
credibility of blog posts. The same respondent also provided maximum scores of
1 for two of the criteria, Reporting empirical data and Reporting research
methods, with scores of 0 for all other criteria. This is a surprising (even
confusing) set of quantitative responses because respondent #22 appears to be
stating that criteria such as Reasoning and citations have no value at all (a
score of 0) and that, with scores of 1, Reporting empirical data and Reporting
research methods have minimal importance.

In her or his qualitative responses, respondent #22 states, “It is simply
impossible to evaluate the value [of blogs] since no real evidence is
provided.”, and “If we start trusting blogs, we might as well stop doing
scientific research in software engineering.” The survey was not asking
whether blog posts reported empirical evidence or reported research methods
but rather was asking what the respondent considered to be important criteria
for evaluating the trustworthiness of blog posts. Given respondent #22’s
qualitative comments, we might expect their scores on the credibility criteria
to be high, i.e., at least high scores for Reporting empirical data and
Reporting research methods, perhaps with at least relatively high scores for
Reasoning and citations to research. Yet, as noted above, these are not the
scores that were reported. It seems that respondent 22 has misinterpreted the
questions that were asked in the survey. Given the unusual responses from
respondent #22, and also to remain consistent with our previous paper
(Williams and Rainer, 2019a), we remove case #22 from our quantitative
analyses. We do however retain the respondent’s comments for our thematic
analysis.

A second respondent — #39 in Table 3 — also demonstrates unusual responding
behaviour. This respondent completed the survey in 2.4m, the quickest time to
complete the survey, scored 3 for the General Credibility question, and then
scored 5 consistently for all nine credibility criteria, returning one of the
highest total scores across all 44 respondents. Again, this response is
surprising. The respondent completed the survey about 5 times quicker than the
adjusted average for all respondents and reports a uniform set of responses
across all nine criteria. The respondent makes few qualitative comments, the
only comment being “It [the general quality of blog posts] depends. Quality
and credibility of blog articles vary wildly.”. For our quantitative and
thematic analysis we retain case #39.

In section 6 we look again at possible excluded cases when we consider the
qualitative responses.

###  3.7 Quantitative analysis

Although our dataset is relatively small (n=44) with a limited number of
questions and a limited range of values (typically 0 – 6), the attitudinal
data is complex and we don’t want to over–simplify the data by only using
descriptive statistics. We also don’t want to mislead: whilst a statistical
test might be more sophisticated it does not necessarily produce a more
reliable insight. We therefore report a heatmap of all 44 responses and
complement that heatmap with statistical analysis. We use RStudio (v1.2.5033)
with the tidyverse (v1.3.0) collection of R packages.

###  3.8 Thematic analyses

To complement each quantitative question in the survey we invited the
respondent to provide open–ended comments. A total of 143 comments were
received from the 44 respondents.

We first analysed separately the qualitative comments associated with each
quantitative question. We read several times each set of comments, and
iteratively developed nodes to denote respondents’ concepts, discussed the
nodes, then developed themes and discussed the themes.

For the current paper, we report thematic analyses of all comments as one
dataset. We present verbatim quotes from a relatively large proportion
(approximately 40%) of the comments. We do this to convey the richness of the
opinions held by the 44 respondents. We used NVivo for Mac (v11.4.2) to
qualitatively analysis the data.

We choose not to report frequencies of coded qualitative data for several
reasons. First, our motivation for reporting qualitative comments is to
provide a rich complement to the quantitative analyses. Quantifying the
qualitative data dilutes that richness and ‘just’ provides another set of
quantitative results. Second, a respondent can provide qualitative comments
for each and all of the criteria. Given that we analyse the dataset as a
whole, there could be a ‘double counting’ of frequencies of qualitative codes
(where a code repeats across comments because the comment is repeated by a
respondent). As one illustrative example, we found some respondents proposing
additional criteria that were already present in the survey. As another
illustrative example, we found cases were respondents were making comments
such as, ‘Please refer to my previous comments’ which raises complications in
objectively counting nodes. Third, the quantity of comments declined as the
respondents progressed through the survey, with reduced variation and
frequency of codes identified for later criteria. This may lead to an
imbalanced dataset, were one to compare comments across different questions.

###  3.9 Public access to data

For transparency, the raw responses from all respondents are available
online555https://www.researchgate.net/publication/331704210.

##  4 Summary of previous results

As noted, we first reported initial results from the survey in a conference
paper (Williams and Rainer, 2019a). We seek a balance of providing sufficient
foundation and background from the previous conference paper, so as to support
the new quantitative and thematic analyses reported in the current paper,
whilst not just re–reporting here the results of that previous paper. We
re–report from (Williams and Rainer, 2019a) our analyses of the respondents’
overall attitude to the credibility of blog posts (see section 4.1 and 1) to
provide necessary context for the current paper, and then summarise the
findings from the previous paper.

###  4.1 The general credibility of blog posts

The survey asked respondents whether, in general, they consider blog posts to
be credible. Figure 1 presents the results. We are conscious that respondents
are being asked to evaluate a complex situation (i.e., a very large volume of
blog posts that vary in content and quality) with a one–score response. Given
the complexity of the situation, a score of 3 might constitute the ‘safest’
response, or the most conservative response, for many respondents.

Figure 1: The general credibility of practitioner blog posts (n=43, Mod=3,
Med=3, Mn=2.7)

###  4.2 Summary of results from initial analysis

We conducted the following analyses, with observations and conclusions in our
conference paper (Williams and Rainer, 2019a):

  1. 1.

We ranked the nine credibility criteria according to the 7–point Likert scale.
We found

    1. (a)

A surprisingly low level of rankings, e.g., 32% of respondents consider the
Reporting of empirical data to be an extremely important criteria. We
speculated that these low percentages may indicate that respondents do not
consider criteria to be essential, or perhaps the respondents were making some
kind of adjustment in their valuation for blog posts.

    2. (b)

The Reasoning criteria was most frequently ranked as extremely important (
40%).

    3. (c)

Devanbu et al. 2016 and Rainer et al. 2003 found that software engineering
practitioners valued their own personal experience, and that of their
colleagues, over independent, third–party empirical evidence. By contrast,
researchers assign a low importance to Prior beliefs and Influence of others.

  2. 2.

We separated the data into three subsamples based on the degree to which
respondents considered blog posts credible: Low credibility (scores of 0 and
1), Medium credibility (scores of 2 and 3) and High credibility (scores of 4
and 5. We found:

    1. (a)

The Low subsample ranked four criteria — Clarity of writing (CoW), Reporting
empirical data (RED), Reporting methods of data collection (RM) and Reasoning
(Reason) — relatively highly. These four criteria most clearly align with the
overt values of the empirical software engineering research community.

    2. (b)

The Low subsample had a minimum value of 5 for Reporting empirical data (RED).
This could suggest that respondents in the Low subsample place a particularly
high value on reporting empirical data. This might be a spurious result in our
data, but as was noted it is a result consistent with the overt values of an
empirical research community.

    3. (c)

71% of respondents in the High subsample rated the Reason criterion as
extremely important. This contrasted with 43% and 31% in the Low and Medium
subsamples retrospectively, and suggests that respondents in the High
subsample place a particularly high value on reasoning. This observation might
explain why this subsample considers blog posts to generally be credible: the
respondents in this subsample primarily evaluate credibility in terms of the
presence of reasoning, and blog posts provide a flexible medium for
practitioners to express their reasoning. A related observation is that none
of the respondents in the High subsample consider the Reporting methods of
data collection (RM), citations to practitioner sources (URL-P), or citations
to research sources (URL-R) to be extremely important.

    4. (d)

The contrasting views between the three subsamples together with the
contrasting views within each subsample may help to explain why blog posts are
contentious. As one example, the High–credibility respondents place a high
value on Reasoning and a low value on Reporting empirical data for blog posts,
in contrast to the Low–credibility respondents who place a relatively high
value on Reporting empirical data.

  3. 3.

We also asked respondents whether they thought the identified nine criteria
generalised to other practitioner–generated content and to research content.
We found that:

    1. (a)

Over 60% of respondents thought that the criteria generalise to other
practitioner–generated content.

    2. (b)

A very similar percentage, over 58%, thought the criteria also apply to
researcher–generated content.

##  5 Exploratory analyses of the quantitative data

###  5.1 Summary of responses

Figure 2 presents a stacked barchart of respondents’ ratings of the nine
credibility criteria. We report this stacked barchart to provide a context for
the subsequent analyses.

Figure 2: Stacked barchart of responses to the nine criteria

###  5.2 Heatmap analysis

Figure 3 presents a heatmap of the respondents’ scores for the nine
credibility criteria and the general credibility of blog posts, together with
descriptive statistics for those scores.

Figure 3: Heatmap of quantitative responses (n=44)

The heatmap is ordered using the following ‘logic’. The heatmap is first
ordered from left to right based on the sum of all respondents’ scores for
each criteria (see the penultimate row at the foot of the heatmap, labelled
Sum). The heatmap is then ranked from top to bottom, first based on the
respondent’s score for general credibility (see the column labelled GC) and
then based on seven of the credibility criteria (see the column labelled
R(7)). The seven criteria used in the ranking are the first seven ordered in
the figure i.e. RED, Rsn, CoW, RM, PExp, URL–R and URL–P. Our previous paper
(Rainer and Williams, 2018) found that the last two criteria, IoO and PB, were
considered to be qualitatively different to the first seven criteria. We
discuss this difference below.

Recall that scores for each of the credibility criteria are in the range
0666Does a score of 0 mean “I don’t know”? Check. to 6, whilst scores for the
general credibility of blog posts are in the range 0 to 5 (though no
respondent provided a score of 5). Cells shaded entirely black have the score
of 6. All other cells contain the respective respondent’s score. Cells not
containing a value represent responses of “I don’t know” or missing values.
Respondents that are excluded from the quantitative analysis are indicated
with an x suffix against the ID, i.e., 22x. Respondents that are candidates
for exclusion, as identified during the subsequent qualitative analysis, are
indicated with an q suffix against the ID, i.e., 13q, 37q. Subsection 3.6
discusses excluding cases on the basis of the quantitative analysis and
subsection 6.2 discusses excluding cases on the basis of the qualitative
analysis.

The logic by which the heatmap has been ordered helps to highlight a general
pattern of attitudes amongst the respondents. Respondents tend to show a
preference for the criteria toward the left side of the heatmap, and they show
this preference regardless of how they rate the credibility of blog posts.

###  5.3 Actual response proportions relative to theoretical range

The theoretical range for an individual respondent’s sum of scores for all
nine credibility criteria is 0 to 54 (nine credibility criteria each with a
range of scores from 0 to 6), with a theoretical median average score for a
respondent of 27.5 (as there are 55 values [0 - 54] available in the sum of
scores). Similarly, the theoretical range for the sum of scores for an
individual criterion, across all 44 respondents, is 0 to 264 (44 respondents
with a maximum score of 6; this range obviously reduces as one excludes
cases).

###  5.4 Confidence intervals

Table 4 reports 95% confidence intervals777Calculated using the following
guidance: http://www.stat.yale.edu/Courses/1997-98/101/catinf.htm. for the
percentage responses to the nine credibility criteria888We have not done a
pairwise exclusion of cases to compensate for missing responses later in the
survey.. In other words, given the proportions reported in the sample, the
confidence intervals provide a range estimate for the population’s proportion
for each criteria. Confidence intervals are calculated using the z–score of a
Normal population, of course, so the estimates reported in Table 4 obviously
assume that the population of survey responses for each criterion would
conform to a Normal distribution. This is unlikely to be a reasonable
assumption, however reporting confidence intervals helps us to explore the
data.

(To digress briefly: one candidate explanation for a non–Normal distribution
is that there may be distinct sub–populations, e.g., a population of software
engineering researchers who have not and do not read blog posts, in contrast
to a population of software engineering researchers who have and do read blog
posts. This explanation is based on the qualitative comments relating to
self–excluding cases, as discussed in subsection 6.2. Each of these
sub–populations will have different experiences of the value and credibility
of blog posts and therefore their attitudes will likely contrast.)

Accepting the difficulties with the confidence intervals, the table suggests
that the upper limits to the confidence intervals for the Others and Beliefs
criteria are very close to the lower limits for the top four criteria. This
suggests that variation in the sample means – perhaps arising because of the
convenience sampling – would be unlikely to affect a number of the results.
Acknowledging the difficulties with the confidence intervals, the best we can
conclude at this stages is that the broad ‘profile’ of proportions in the
sample may be present in the population.

Table 4: Confidence intervals for proportions of responses for the nine criteria Criteria |  Count |  Proportion |  CI Upper |  CI Lower
---|---|---|---|---
RED |  210 |  0.795 |  0.915 |  0.676
Rsn |  208 |  0.788 |  0.909 |  0.667
CoW |  199 |  0.754 |  0.881 |  0.626
RM |  196 |  0.742 |  0.872 |  0.613
Pexp |  193 |  0.731 |  0.862 |  0.600
URLR |  184 |  0.697 |  0.833 |  0.561
URLP |  167 |  0.633 |  0.775 |  0.490
Others |  127 |  0.481 |  0.629 |  0.333
Beliefs |  117 |  0.443 |  0.590 |  0.296

###  5.5 Correlation analysis

Table 5 presents the Spearman Rho correlations for each pair of the nine
criteria. The correlations may be grouped as follows:

  1. 1.

positive correlations that have a high Rho value and are statistically
significant (p $<$ 0.05);

  2. 2.

positive correlations that have a relatively high Rho value but are not
statistically significant (p $>=$ 0.05);

  3. 3.

positive correlations that have a relatively low Rho value;

  4. 4.

correlations that have a negative Rho value (negative correlations);

  5. 5.

those correlations that have a very low, or even no, correlation

Table 5: Spearman Rho rank correlations for the nine credibility criteria Criterion | RED | Rsn | CoW | RM | Pexp | URLR | URLP | Others
---|---|---|---|---|---|---|---|---
Rsn | -0.07 |  |  |  |  |  |  |
CoW | -0.01 | 0.23 |  |  |  |  |  |
RM | 0.72 | -0.04 | -0.16 |  |  |  |  |
Pexp | -0.15 | 0.05 | 0.00 | -0.03 |  |  |  |
URLR | 0.21 | 0.19 | 0.06 | 0.24 | -0.17 |  |  |
URLP | -0.06 | 0.12 | 0.06 | 0.09 | 0.19 | 0.56 |  |
Others | -0.29 | 0.05 | 0.06 | -0.32 | 0.10 | -0.08 | 0.25 |
Beliefs | -0.33 | -0.06 | 0.09 | -0.33 | 0.02 | -0.24 | 0.11 | 0.85

As indicated by the groupings, most correlations are not statistically
significant. This may partly be explained by the low sample size (and hence
low statistically power). We focus here on the two correlations that are
significant, and on the negative correlations.

  1. 1.

There is a very strong statistical correlation between the Influence of others
(Others) criterion and the Prior beliefs (Beliefs) criterion (Rho = 0.85). The
correlation is visualised in Figure 4a. These two criteria are likely to be
qualitatively different to the other seven criteria because these two criteria
are not directly ‘measurable’ within the blog post itself. Our heatmap also
suggests these two criteria are different. We think that these two criteria
and their correlation would be ‘caused’ by an underlying effect that is
different to the other seven criteria. We return to this pair of criteria in
due course.

  2. 2.

There is a strong correlation between the Reporting empirical data (RED)
criterion and the Reporting research method (RM) criterion (Rho = 0.72). The
correlation is visualised in Figure 4b. That there is a correlation, and that
this correlation is one of the stronger correlations, is not surprising. We
expect researchers to independently rank both criteria highly (or relatively
highly) because the research community places high importance on evidence and
on the rigour of evidence–collection. Given the manifest importance of these
criteria to the community, and given the strength of their correlation, these
two criteria and their collection could act as a kind of benchmark against
which to consider the other criteria and correlations. For example, earlier
(see subsection 5.2) we speculated on the possibility of different
sub–populations. The strength of correlation between the Reporting empirical
data (RED) criterion and the Reporting research method (RM) criterion suggests
two criteria that are relatively constant across sub–populations.

We also observe negative correlations between the pair of criteria, Reporting
empirical data and Reporting research methods, and the pair of criteria, Prior
beliefs and the Influence of others. Two examples of the negative correlations
are shown in Figures 4c and 4d for illustration. These correlations are not
statistically significant, however they are consistent with the values of the
software engineering research community, i.e., to value evidence over one’s
prior beliefs and over the influence of others.

(a) Beliefs & influence

(b) Data & method

(c) Influence & Methods

(d) Data & Beliefs

Figure 4: Bubble charts with Loess smoother and confidence interval

Bubble plots for each of the criteria can be found in the Appendix. The plots
present a kind of transposition of the data presented in the heatmap.

###  5.6 Summary

We have examined the quantitative data from various perspectives in the search
for patterns within the data that would provide insights into respondents’
attitudes to the use of blog posts in research. At best, our overall analyses
provide the following limited insights:

  1. 1.

The sample is relatively small and imbalanced, and this effects the ability to
quantitatively analyse the data, and the confidence we can draw from that
analysis.

  2. 2.

There appears to be a general preference for certain credibility criteria,
i.e. Reporting empirical data, Reasoning, Clarity of writing, Reporting
research methods and Professional experience. These preferences appear to be
regardless of the rating for General Credibility.

  3. 3.

Although there is a quantitative ranking to these preferred criteria, the
closeness of the rankings and the confidence intervals suggest the ordering of
the sample may not reflect the ordering in the population (but the confidence
intervals are likely untrustworthy because of the nature of the data).

  4. 4.

There are two pairs of criteria that have within–pair correlations. These are:

    1. (a)

Reporting empirical data correlating with Reporting research methods; and

    2. (b)

Influence of others correlating with Prior beliefs.

  5. 5.

The correlation of Reporting empirical data with Reporting research methods is
not unsurprising, given the nature of the research discipline, i.e., with
importance placed on evidence and evidence–collection.

  6. 6.

The correlation of Influence of others correlating with Prior beliefs may be
due to a misunderstanding of these criteria, or because these two criteria are
qualitatively different to the seven other criteria.

  7. 7.

There may be a negative correlation between to the two pairs of criteria,
i.e., Reporting empirical data and Reporting research methods negatively
correlating with Influence of others and Prior beliefs. Again this is not
unsurprising, given the nature of the research discipline, i.e., with
importance placed on evidence and evidence–collection.

  8. 8.

There may be sub–populations in the data.

##  6 Thematic analyses

###  6.1 Introduction

Our qualitative analyses identified a number of themes. In the following
subsections, we present tables of verbatim quotes from the respondents and
discuss those quotes. To aid cross–referencing, each quote has a unique
identifier of the format Q#. Inevitably, some quotes refer to more than one
theme. To save space, in most cases we present the quote in one table but
embolden appropriate phrases to emphasise the themes as well as the richness
of the data. For each quote, we include the respondent’s rating of the general
credibility of blog posts, of the format GC=n, where n is the rating in the
range 0 (lowest) to 5 (highest). For each table, we order the quotes according
to the respondents’ scores for General Credibility.

###  6.2 General comments on respondents and responses

Table 6 presents examples of three issues relating to respondents and their
responses: examples where respondents and their responses might be excluded;
examples of where respondents are aware of variability in the quality of blog
posts; and examples of respondents’ expectations of blog posts.

Table 6a) identifies examples of respondents who could be excluded/included
from the analyses, depending on how strict one sets the exclusion/inclusion
criteria. Examples for exclusion could be treated as either outliers or
inaccurate (noisy) cases. Respondent Q6 is the respondent already identified
as an outlier from our earlier analyses (Williams and Rainer, 2019a).
Respondent Q6 bases her or his assessment of the general credibility of blog
posts (GC=3) on her or his “impression”. Similarly, Q6 is unusual as it is one
of the few respondents in the survey who confirm they read a lot of blog
posts.

Table 6b) indicates that survey respondents are aware of variability in the
quality of blog posts and therefore, by implication, of the difficulty of
responding.

Table 6c) provides comments on the respondents’ expectations of blog posts.
Expectations may also be understood as a kind of prior belief, and may give
rise to biases, such as anchoring bias or confirmation bias. For example, if I
expect blog posts to lack credibility I may not be alert to those blog posts
that provide credible information.

We did not explicitly ask respondents to state their expectations. One
interpretation of our results is that, in fact, the results constitute a
baseline of researchers’ expectations of blog posts.

Table 6: General observations on respondents and responses Q# |  Comment
---|---
6a) Example cases for exclusion and inclusion
Candidate cases for exclusion
1 |  This is just a feeling. I do not read many blog articles. [GC=3]
2 |  I do not follow blogs. My comments relate to my impression of the work of practitioners such as Don Reifer, Capers Jones, Larry Putnam, Dan Galorath; organizations such as NESMA and IFPUG; and data providers such as ISBSG. [GC=3]
3 |  It is simply impossible to evaluate the value since no real evidence is provided. [GC=0]
Candidate cases for inclusion
4 |  I like to read some blog articles to get a closer look at what practitioners are doing. [GC=2]
5 |  I do read a lot of blogs. The quality varies. [GC=3]
Other cases
6 |  I have never used blog info in my research, rather use empirical data collected directly from companies. Pls, consider this when analysing my answers! [GC=1]
6b) Recognition of the variability of blog posts
7 |  I think we should assess the type of information according to its [the author’s] intention. It is unfair to assess a blog post as we would do to a scientific paper, and vice–versa. They have different target audience. I understand that blog posts that present some empirical data or method [of data collection] have additional positive characteristics, while scientific works that present neither of them have additional negative characteristics. Both can be assessed regarding a their reasoning and the use of practical experiences, though. [GC=3]
8 |  You can go from a really good, well-backed article to a completely non-sense, personal feeling-based article [bias]. Since anyone can write anything, there should be a way to filter it, recommend, or something like this. [GC=3]
9 |  It really depends. Some are sh*t and some make a lot of sense. [GC=3]
10 |  I do read a lot of blogs. The quality varies. [GC=3]
11 |  It depends. Quality and credibility of blog articles vary wildly. [GC=3]
6c) Expectations of blog posts (grouped by GC score)
12 |  I dont (sic) usually read a blog post from a practitioner expecting it to contain any of this information. If they present this type of information is a plus to me. [GC=3]
13 |  What I expect to find in a blog post to assess its quality is a good reasoning for the writers’ argumentation and, when possible, some experience report. The level of detail in both the reasoning the writer uses and the experience report the writer presents is what I consider the most important to assess the post’s quality. [GC=3]
14 |  It really depends. Some are sh*t and some make a lot of sense. [GC=3]
15 |  It depends. Quality and credibility of blog articles vary wildly. [GC=3]
16 |  I expect blogs to be principally anecdotal \- that is the value for me in them - do the anecdotes/war stories align with my experience (as practitioner) and do they align with empirical results or offer counter-evidence. I would be pleasantly surprised if they offered more than simple empirical observations that could be calibrated. [GC=2]

###  6.3 Author bias, subjectivity and sincerity

Table 7a) presents contrasting comments on (i) author bias and subjectivity,
and (ii) author sincerity. A sincere practitioner may unintentionally be bias,
a contrast that is also found with researchers e.g., as demonstrated with the
need for calculating inter–rater agreement on assessments. Bias, subjectivity
and sincerity are not properties that only apply to the writing of blog posts.
Information shared by any practitioner, or researcher, for example through
interview, survey, focus group or blog post has a degree of bias, subjectivity
and sincerity.

###  6.4 Topic and content

Table 7b) presents comments on the content of blog posts. Quote Q7 suggests
that researchers’ evaluations of the credibility of a blog post is partly
dependent on the researchers’ interest in specific content of the blog post.
For example, Treude, Parnin, Storey and Aniche (e.g. (Parnin and Treude, 2011;
Parnin et al., 2013; Aniche et al., 2018)) have a particular interest in the
degree to which practitioners discuss API methods in blog posts. They
therefore also have an interest in the presence of code fragments that
demonstrate or illustrate the API method/s. Pagano and Maalej (2011) observed
that only 1.8% (934) of the 50,701 blog posts they examined (from four active
open source projects) contained source code paragraphs. This contrasts with
Parnin and Treude (2011) who observed that 90% of the posts they examined (336
posts from 373 posts) had code snippets in the post, a median of 3 code
snippets per post. In both studies, the researchers were interested in the
presence of code, but found a very different prevalence of the presence of
code. This difference could affect how one interprets the credibility of a
blog post or the dataset of blog posts, or indeed blog posts in general.
Evaluating the credibility of a blog post in relation to the researchers’
interest in specific content connects back to Petersen and Gencel’s (2013)
work on worldviews.

Another example is Stack Overflow. The nature of the question–and–answer site
is that, to the degree possible, the questioner is encouraged to ask a
Minimal, Complete and Verifiable Example
(MCVE)999https://stackoverflow.com/help/mcve; an alternative acronym is
Minimal Worked Example (MWE): a collection of source code and other resources
that allow the bug or problem to be demonstrated and reproduced.: use the
minimal amount of code needed to (re)produce the problem; complete all parts
(of the code) to (re)produce the problem; and test the code first to verify
that you can (re)produce the problem. A blog post of this kind would be
different to, for example, a discussion of the relative benefits and drawbacks
of different testing techniques,

The qualitative data therefore suggests the following types of content (this
list is not exhaustive): empirical data; opinions (beliefs); experience;
Reasoning; and source code.

Table 7: Observations on author subjectivity, blog content, and professional experience Q# |  Comment
---|---
7a) Author bias, subjectivity and sincerity (grouped by GC score)
(i) Comments relating to author bias and subjectivity
17 |  They are based on personal opinions or gut feeling, very often subjective or inspired by someone else. [GC=3]
18 |  The blogs are credible in that they represent the view of the bloggers. I don’t believe they fake up their opinions. Of course, the views may be highly biased. [GC=3]
19 |  In general, blog discuss ideas or advocate for technique/tools. Assesments (sic) could be biased. [GC=3]
20 |  You can go from a really good, well-backed article to a completely non-sense, personal feeling-based article [bias]. Since anyone can write anything, there should be a way to filter it, recommend, or something like this. [GC=3]
21 |  While I do not think that blog articles are deliberately wrong or misleading, I believe that they are very subjective opinions and just should be treated accordingly (that is: cautiously). [GC=2]
22 |  It is hard to read such materials. I am always concerned with their validity and fairness. Usually they represent opinions (sometimes biased) on some topics. [GC=1]
(ii) Comments relating to author sincerity
23 |  Blogs provide opinions, so from the point of view of those writing the blogs, they believe that what they are stating is true and reliable. However, such assertions may fail a more detailed scrutiny. [GC=3]
24 |  Blog articles represent the experience of the writer and so are “credible” from that point of view. I would be surprised to find any blog writer deliberately misrepresenting his or her observations. [GC=3]
25 |  It’s often there own Opinon; mostly serious, cause they don’t want to blame themselves with stupid staff; interesting views Often experiences that can be very useful and that are from practice. [GC=3]
26 |  In my experience blog articles are typically written in good faith by practitioners who have a degree of familiarity with their chosen topic. Whether such individuals necessarily have the wider context of their chosen topic or perspective on the topic can be more open to question. [GC=2]
7b) Content of the blog posts
27 |  I mostly think it depends on the subject matter of the post. Generally blogs and Q&A sites on programming are credible to me and quite useful as well. On the other hand, when a blog post regards other fields of software engineering such as management, requirements, measurement and even testing, it is quite hard to identify a post that has zero or little bias. When they try to make some comparison with alternatives to their recommendation, it is usually based on outdated information or unfair comparison. [GC=3]
28 |  It depends on the topic as well as on the amount of provided content and its level of detail. [GC=3]
29 |  It depends on what they are reporting. [GC=3]
7c) Professional experience (grouped by GC score)
30 |  (In the field of agile development) practitioners experiences are very important. The credibility of course depends on the quality of the content. It also depends on the format, would probably also trust a blog article more than a single short answer in a Q&A section. [GC=4]
31 |  The main goal of blogs articles is not to provide empirical evidence, each blogger has his/her own goal from sharing experiences, self-marketing, getting public attention, etc. [GC=2]
32 |  Experience that are grounded in prior research, supports or refutes prior finding, explains the context and data collection method clearly seems credible to me. [GC=2]
33 |  Opi[n]on-driven and mostly grounded in personal experience – not viewed from a more general perspective. Quite often focused on particular business goals…[GC=1]

###  6.5 Professional experience

Table 7c) presents comments on professional experience. As noted in the
previous subsection, experience is one type of content in blog posts. Table
7c) suggests that experience seems to be the type of content typically of most
interest to researchers.

###  6.6 The credibility of the author

Table 8 presents comments on the credibility of the author. Respondents appear
to place significant value on the credibility of the author of the blog post.
The table provides indications of a complex set of attributes that determine
credibility i.e., the more experienced and skilled practitioners, who reflect
carefully on their work, who are appropriately motivated, who have a high/er
reputation, who write regularly on a topic, who are affiliated with a
well–known organisation, and who have some history of academic participation.

Table 8: The credibility of the author (grouped by GC score) Q# |  Comment
---|---
34 |  I tend to judge articles on the credibility of the authors. In this case, the best practitioners are absolutely credible. However, no one is right all the time. [GC=4]
35 |  I also appreciate if the prior beliefs/assumptions of the author(s) are made clear in the article. [GC=4]
36 |  Implicitly, the popularity of the authors, the popularity of the blog (is it well known, can everyone post there or was the author accepted to present his content there). [GC=4]
37 |  The writer experience (previous worked companies, for instance) The blog credibility (is it part of a larger portal?) [GC=3]
38 |  I think we should assess the type of information according to its [the author’s] intention. It is unfair to assess a blog post as we would do to a scientific paper, and vice–versa. They have different target audience. I understand that blog posts that present some empirical data or method [of data collection] have additional positive characteristics, while scientific works that present neither of them have additional negative characteristics. Both can be assessed regarding a their reasoning and the use of practical experiences, though. [GC=3]
39 |  It depends on who wrote the blog post. I’m highly biased by blog posts by Martin Folwer (sic) and others. But I read with care posts by other that I don’t know. [GC=3]
40 |  It really depends on who is the practitioner, and the blog itself. If it is part of a larger portal, or independent. [GC=3]
41 |  The prestige of the blogger, e.g. Kent Beck [GC=3]
42 |  Unlike academic articles, which can be judged on the basis of the method and data collected, the quality of a blog articles also depends on the reputation and experience of the author: everyone can have an opinion (which is what most blog articles express I think), but opinions of reputable authors carry more weight. [GC=3]
43 |  Author affiliation and expertise. Martin Fowler vs a less experienced [person] [GC=3]
44 |  Frank disclosure of the writer’s background and experience that has shaped its understanding of the world and software development processes. [GC=3]
45 |  If a practitioner is writing about something regularly, then any post they write within that area is much more credible than a post in an unrelated or even just different area. [GC=3]
46 |  Motivation of the author, for example: promoting company technology/practices, consultant looking for publicity. [GC=3]
47 |  Reputation of the author. [GC=3]
48 |  I tend to trust more blogs by practitioners of well known companies. [GC=2]
49 |  Reputation of the blogger. [GC=2]
50 |  The prior beliefs of the blog writer \- does the writer offer evidence of challenging/reflecting upon their own personal beliefs in response to any learning/understanding they have acquired on the topic they have written the blog on. [GC=2]
51 |  Quite hard. Only well known sources [authors] and depending on the topic has some minimal credibility. [GC=1]
52 |  practitioner previous academic participation [and] practitioner affiliation. [GC=1]

###  6.7 Anecdotal, (a)contextual information and evidence

Table 9 presents comments anecdote and context. The concepts of context and
anecdote are related to each other in that anecdotal evidence appears to lack
(amongst other things) important details, such as contextual information and
concrete detail. Some respondents express concerns that practitioners in blog
posts over–generalise beyond their experiences.

Anecdote and context have connections with case studies (e.g. (Runeson et al.,
2012; Runeson and Höst, 2009)) and their generalisation. One recurring threat
to the validity of case studies, and also of survey studies, is the limited
number of cases reported. Another threat is that often researchers simply do
not know the characteristics of the population from which the cases are drawn.
And a further threat is that the sampling process itself is uncertain e.g.,
self–selection on the part of respondents. These threats, and the recurring
problem of context in software engineering research (e.g. (Petersen and
Wohlin, 2009; Dybå et al., 2012; Dyba, 2013; Clarke and O’Connor, 2012)) raise
concerns about generalisation.

One difference between the anecdotes of practitioners and the case study by
researchers is that researchers are much more likely to recognise and report
these challenges, in contrast to practitioners writing blog posts. Simply
reporting the presence of a threat does not in itself diminish the threat,
however. Even when researchers report contextual information, the discipline
lacks an accepted framework for sharing contextual information.

Table 9: Context and anecdote (grouped by GC score) Q# |  Comment
---|---
53 |  Anecdotal, but surely relevant. [GC=4]
54 |  Many practitioners lack the larger picture to give objective reports. As such, they report their perspective which is often skewed by their limited knowledge of tools and techniques…As such, vast majority of practitioners are behind the state of art and practice and report anecdotal or otherwise out–of–context findings that should not be considered by anyone thriving towards excellence in any field. On the other hand, there are a select few practitioners who have sufficient background knowledge and larger perspective whose blogs are excellent and far beyond any researcher’s blogs or in some cases even surpassing scientific journals in their value. These practitioners always: 1) avoid generalisation, and 2) report in detail sufficient for reproduction, verification and scoping (as they do list the biases in their analysis, which rarely happens in journal articles). [GC=3]
55 |  Blog entries (e.g. position statements / manifestos) are to be seen as anecdotal evidence. This is not a problem per–se, but such evidence suffers especially from the lack of context information that makes the claims empirically usable rather than suffering from the lack of evidence-based accuracy [GC=3]
56 |  Context description (good description) of the blog, e.g. Agile text can be many versions and variants of Agile. [GC=3]
57 |  The context information (e.g. project setting, practitioners’ characteristics, etc.) is rather important. Setting the results [practitioner’s observations and beliefs] in relation to existing evidence can be done by researchers provided [there is] a certain clarity of the [practitioner’s] claims. Without clarity, claims tend to result in universal propositions and solutions. [GC=3]
58 |  Blogs contain personal opinions that stem from concrete cases. They cannot be generalized. Furthermore, opinions evolve over time as technology matures, the writer acquires more experience, etc. In other words: blogs are snapshots (on time / contex[t]) regarding some issue / technology. [GC=2]
59 |  Some blogs are good (e.g., Jeff Atwood or Martin Fowler). However, most of those are not scientifically validated. Does not contain enough information to understand the context or replicate. [GC=2]
60 |  Very few are credible. Main problem is they are most often anecdotal and based on often a limited set of cases where the blog authors has worked/experience from. Since this is their experience there is rarely a realization that it is limited. I know practitioners who spent their whole adult work life in one and the same company and this gives them the right to know much more and “correct” researcher even if the latter are working closely with maybe 10–15 different companies. I think we often think way too binary on these things (“all practitioners know a lot about industrial practice, while no researchers do”). [GC=1l; parenthesis are in the original text]
61 |  Reporting the context for and how empirical evidence has been gathered is the key thing. Anecdotal “stories” based on long experience can be helpful to create ideas of what to study more but can and often is misused by authors to spread their opinions. [GC=1]
62 |  some blogs are written by practitioners who have access to data from the company they work for (see StackOverflow blog). That is very valuable knowledge. Many others (especially the ones not explicitly affiliated) are less reliable since based on anecdotes. [GC=1]
63 |  Describing the context for ones statements/blog/info. What is your experience, how broad is it, from which industries/domains, how many years etc. See Peterseon and Wohlin on describing context for empirical SE research, for example. [GC=1]

Table 9 presents verbatim quotes on the relationship between anecdote and
empirical evidence. Essentially, the respondent is suggesting we seek
alignment – a triangulation – between the beliefs reported by the
practitioner, one’s professional experience (as a practitioner) and
independent empirical evidence. This is the intended aim of Multivocal
Literature Reviews (MLRs) and also of Evidence Based Software Engineering
(EBSE).

##  7 Preliminary checklists for evaluating blog posts

###  7.1 Overview

Building on the analyses presented earlier in this paper, we propose a
preliminary checklist that researchers can use to evaluate blog posts. We also
briefly demonstrate the checklist in section 7.2.

The checklist is synthesised from four sources: the general review of
literature we present in section 2, the pre–defined criteria used in the
survey (see section 3.2 and also (Williams and Rainer, 2018)), the thematic
analyses of the reviewers’ qualitative responses (see section 6), and two
quality checklists, Garousi et al.’s (2018) generic checklist for multivocal
literature reviews (MLRs), and Soldani et al.’s (2018) checklist used in their
grey literature review (GLR) of microservices. The checklist is organised into
sections as summarised in Table 10. The checklist itself is presented in
Tables 11 through 14.

Table 10: Summary of the checklists Checklists |  Brief explanation
---|---
Foundational |  Questions intended to raise the self–awareness of users of the other checklists.
Process |  Questions for evaluating the process of enquiry – if any – that informs the writing of the blog posts, e.g., any formal or informal investigation conducted by the author of the blog post
Author |  Questions for evaluating the credibility of the author of the blog post.
Content |  Questions for evaluating the content of the blog post itself.
Reader |  Questions for evaluating reader feedback in the blog post.
Influencer |  Questions for evaluating influences of others on the author of the blog post.
Media |  Questions for evaluating the media through which the blog post is published, e.g., the platform on which the blog post is posted.

All of the sections of the checklist are concerned with credibility (broadly
defined) and relevance, though some sections are more explicit in their
separation of these two constructs, and some sections focus on a particular
aspect of credibility. Some sections are also more developed the others, and
we briefly discuss three sections here: the Foundational, Process and Author
sections.

For the Foundational section, the worldview provides a foundation for framing
the topic, objectives, and questions for the enquiry, the features of the
empirical phenomena of interest, and the foundations of how credibility will
be evaluated. Individuals have worldviews. A worldview may also be implicitly
or explicitly stated in guidelines, checklists or methodology.

The Process section is a special case because it focuses only on the rigour on
an enquiry process and not on the relevance.

For the Author section, credibility is demonstrated and assessed through
competence and relevance e.g., that the author has sufficient expertise in a
relevant domain. Researchers select practitioners for studies, or allow
practitioners to self–select, on the basis of competence and relevance.
Researchers have concerns around the subjectivity and hence bias of the
enquirer, and this raises a paradox: researchers appear to value (some)
practitioners on the basis of their professional experiences and the reporting
of that experience (e.g., Martin Fowler) but also have concerns around the
subjectivity that arises during the very formation of such experiences and
their reporting.

We deliberately do not provide a scoring mechanism or thresholds, or similar,
for the checklist. Neither do we advise on how to aggregate ‘scores’ within
the checklist. We do this in recognition of the insights arising from our
qualitative and quantitative analyses, and in recognition of Petersen and
Gencel’s (2013 discussion of the relationship of worldviews and validity,
e.g., to set a scoring mechanism would be to prematurely set a standard for
assessing a complex type of evidence.

We present a preliminary checklist because the checklist would benefit from
further development, validation and evaluation. The checklist therefore
constitutes a foundation for further research, e.g., through its application,
evaluation and revision.

Table 11: Checklist # |  Question
---|---
Foundational
Familiarity and expectations of blog posts
1 |  How frequently do you read blog posts?
2 |  Have you used blog posts in your research?
3 |  What are your prior beliefs about blog posts?
Worldview
4 |  What is your worldview of research, e.g., pragmatic, constructivist, positivist?
5 |  Have you reflected on your worldview?
6 |  Have you reflected on how your worldview influences, or might influence, your approach to the evaluation of blog posts?
Research design
7 |  These checklists complement other checklists.
8 |  Have you clearly defined the empirical phenomenon, or phenomena, that interests you and to which the blog posts are expected to relate?
9 |  Have you clearly defined the context for the empirical phenomenon, or phenomena, that interests you and to which the blog posts are expected to relate??
10 |  Have you a clear conceptual model of the theorised conditions that give rise to the empirical phenomena that interest you?
11 |  Have you developed a clear set of research questions, or similar, to direct your evaluation?
Table 12: Checklist (continued) # |  Question
---|---
Process
12 |  Is there a process of enquiry reported in the blog post at least in some general way?
13 |  Are specific steps of the enquiry reported, in particular is there some explanation for:
13.1 |  how observations were made;
13.2 |  how data was collected;
13.3 |  how data was analysed;
13.4 |  how results were interpreted; and
13.5 |  how interpretations were reported?
14 |  Are the steps of the process reported accurately?
15 |  Are the steps of the process reported completely?
16 |  Does the process conform to available standards?
17 |  Is the process fit–for–purpose?
18 |  Is the process reported completely?
19 |  Is the process conducted in a timely way?
20 |  Is the process reported in a timely way?
21 |  Does the enquiry process support realism?
22 |  Does the enquiry process use realistic subjects?
23 |  Does the enquiry process occur in a realistic context?
24 |  Does the enquiry process investigate a phenomena at an appropriate scale?
Table 13: Checklist (continued) # |  Question
---|---
Author
Reputation:
25 |  What is the author’s professional reputation?
26 |  What is the author’s affiliation?
27 |  What is the affiliation’s reputation?
28 |  How regularly does the author publish posts?
Bias, subjectivity, intentions & sincerity:
29 |  What are the intentions of the author?
30 |  Does the author have any bias?
31 |  Does the author have vested interests?
32 |  Is the author sincere?
33 |  Is there any relevant subjectivity?
Expertise and experience
34 |  What is the author’s relevant experience?
35 |  What is the author’s relevant expertise?
Table 14: Checklist (continued) # |  Question
---|---
Content
Quality of writing
36 |  How clearly written is the post?
37 |  How impartially written is the post?
Anecdotal content
38 |  Does the post present anecdote?
39 |  Is the anecdote supported by professional experience?
40 |  Does the post report concrete details of the anecdote?
41 |  Are concrete details reported about discrete situations?
42 |  Does the post report context information?
Empirical content
43 |  Does the post report empirical data?
Practical content
44 |  Does the post report a Minimum, Complete and Verifiable Example (MCVE)?
45 |  Does the post report source code example(s)?
Conceptual content
46 |  Does the post report a conceptual model or similar?
Rhetorical content
47 |  Does the post make claims about relevant professional situations?
48 |  Are claims supported by reasons, explanations etc?
49 |  Are claims supported by anecdotal content?
50 |  Are claims supported by empirical content?
51 |  Are claims supported by practical content?
52 |  Are claims supported by conceptual content?
53 |  Does the post show a clear connection between claims, rhetoric, conceptual, and/or anecdotal or empirical content
Reader
54 |  How many times has the post been cited by others (backlink)?
55 |  How many times has the post been media–shared
56 |  How many comments does the post have?
57 |  How many times has the post been viewed?
Influencer
Influence of practitioners’ written work
58 |  How many citations (e.g., backlinks) are there to practitioner’s published work
Influence of researchers’ written work
59 |  How many citations (e.g., references) are there to researcher’s published work
Media
60 |  Medium
61 |  Platform

###  7.2 Applying the checklist to the survey itself

Although the checklists were developed to evaluate blog posts, we demonstrate
the checklists by applying them to the survey responses. We apply the
checklists to the survey for several reasons. First, it is more efficient to
demonstrate, in this paper, the checklists against the dataset we have already
been exploring, rather than introduce and discuss a further dataset. Second,
applying the checklists to the survey is another way of evaluating the survey
data. Third, by applying the checklists to the survey data, we begin to show
that the checklists have the potential for wider applicability. Fourth, and
finally, by applying the checklists to the survey data we can contrast the
credibility of survey data with the credibility of blog posts as data.

There is the risk in this demonstration of a circular argument: we use the
survey data to generate checklists and then use the checklists to evaluate the
same survey data. To clarify: we have used the content of the survey responses
to generate the checklists, and now use the checklists to examine potential
contradictions in the survey data and to compare that data with blog posts.

Table 15: Characteristics of the responses from survey respondents Criteria |  Examples
---|---
Items from the Foundational category
Worldview |  Table 18 suggests that some researchers conceive of blog posts as ‘types’ of interview.
Values |  Some responses suggest that respondents are trading–off the value of rigour and relevance e.g., Q9.
Items from the Process category
Report |  We show in Table 16 that the ‘reports’ from respondents are short, even very short. Table 17 presents contrasting statistics for the set of blog posts on the Joel Spolsky blog.
Steps of the enquiry |  Responses such as those presented in Table 6 imply that some respondents have not conducted any kind of enquiry into blog posts, however informal.
Items from the Author category
Experience |  Table 6 provides examples of where respondents declare they lack experience. And yet, as far as we are aware,no respondent explicitly considered themselves unqualified to complete the survey. In the follow–up with non–respondents to better understand why they did not participate in (did not start, or did not complete) the survey, the only declared reason for not participating in the survey was the lack of time.
Vested interests |  By definition, the respondents have vested interests as they are being asked to evaluate the credibility of blog posts for use in research in their discipline.
Items from the Content category
Concreteness |  Respondents referred to particular practitioners e.g., in Q6 the respondent refers to, “… the work of practitioners such as Don Reifer, Capers Jones, Larry Putnam, Dan Galorath; organizations such as NESMA and IFPUG; and data providers such as ISBSG.”
Reasoning |  There are short responses (e.g. Q6, Q8, Q8) that contain assertions that are not supported by explanations, arguments, citations etc.
Experience |  Some respondents refer to their personal experience, but don’t describe that experience e.g., Q6, Q7.
Beliefs |  Some respondents refer to their beliefs e.g., Q7, Q7, Q7.
Impartiality of writing |  In general the responses are (relatively) impartial, though there are some exceptions e.g., Q6.
Influence of others |  Considering that all 44 respondents were researchers, only one response refers to research (and even then with no explicit citation): “… See Peterseon (sic) and Wohlin on describing context for empirical SE research…”. (We presume the respondent was referring to (Petersen and Wohlin, 2009).)
|  A greater number of responses explicitly recognised specific practitioners,
the most frequent being Martin Fowler. Again, there were no explicit citations
e.g. no URLs were provided.
Table 16: Statistics on researchers’ responses Statistic | Value
---|---
Total number of words (across all questions) | 3,972
Total actual number of qualitative responses received (across all questions) | 143
Maximum number of potential qualitative responses (across all questions) | 264
Percentage of actual number of qualitative responses: | 54%
Mean number of words per response | 15
Median number of words per response | 5
Total number of reasoning markers | 33
Mean number of reasoning markers (264 responses) | 0.125
Median number of reasoning markers | 0
Table 17: Statistics on Joel Spolsky’s blog posts (n=1024) Statistic | Excluding outlier | Including outlier
---|---|---
Minimum number of words | 0 | 0
Maximum number of words | 6222 | 16432
Mean number of words | 515.3 | 530.7
Median number of words | 181 | 181.5
Note: The minimum count of zero words is due to the post containing a photo.
Table 18: Blog posts compared with interviews Q# |  Researchers’ expectations
---|---
64 |  Blog articles are credible as a witness of a practitioner, and has to be treated accordingly. I would never publish a practitioner interview in extenso, neither is a practitioner blog post to be trusted as rigorous research outputs. [GC=2]
65 |  I consider them mostly equivalent to interviews. While practitioners are very free to edit their words, they still represent their opinions/perceptions/emotions. [GC=4]

Table 15 summarises criteria from the models with illustrative examples from
the survey responses. Three further tables support Table 15: Table 16 presents
statistics on the responses from survey respondents; Table 17 presents
statistics on blog posts from one blog maintained by Joel Spolsky; and Table
18 provides quotes from the survey respondents on the nature of blog posts.
For Table 16, the reasoning markers are discussed and validated in (Williams,
2018b). For Table 17, we have used the Joel Spolsky dataset in previous work
e.g., (Williams and Rainer, 2019b; Williams, 2018a).

The comparisons presented in Table 15 through Table 18 suggest that
respondents to the survey are behaving a bit like practitioners who write blog
posts. Respondents are reporting their beliefs, but these beliefs are often
supported by limited reporting of experience, reasoning, anecdote etc.; or are
even unsupported. We have found instances where respondents self–report a lack
of experience of blog posts, and yet continue to provide opinions on the
credibility of blog posts: these respondents should probably be excluded from
the analyses. As another example, many survey respondents value reasoning, and
yet there are many statements made in the survey that are not supported by
reasoning but, instead, are simply assertions. Despite these constraints and
limitations in the data, we have still been able to develop a meta–model and
several simple models. A similar approach applies to blog posts: though
individual blog posts have constraints and limitations on quality, it remains
possible to aggregate insights from an appropriate set of blog posts, as
Soldani et al. (2018) and Raulamo–Jurvanen et al. (2017) have done.

##  8 Discussion

###  8.1 Reviewing the research questions

For question RQ1, viz. To what degree do researchers consider blog posts to be
credible?, we report a heatmap analysis of the survey responses. Compared to
our preliminary analysis (Williams and Rainer, 2019a), this heatmap more
concisely and effectively represents the diversity of attitudes within the
research community. The heatmap suggests the presence of different worldviews
(Petersen and Gencel, 2013), as discussed in section 2, and begins to explain
the contention present within the research community e.g., between a
researcher who values reasoning and relevance in contrast to a researcher who
values empirical data and reporting of data collection processes.

For question RQ2, viz. What criteria do researchers claim to use when
evaluating the credibility of a blog post?, we report the thematic analyses of
over 60 qualitative comments contained in the survey (see section 6). The
comments reveal a complex set of criteria that researchers claim to use when
evaluating blog posts. Of course, any individual researcher is unlikely to be
using all of these criteria.

For question RQ3, viz. What guidance on the quality–assurance of blog posts
can be synthesised from prior research and the survey responses?, we have
drawn on four sources to synthesis a preliminary suite of checklists: our nine
original credibility criteria, the literature review, two quality checklists
for grey literature (Garousi et al., 2018; Soldani et al., 2018), and the
quantitative and qualitative re–analyses of the survey responses. Each
checklist is intended to evaluate a different facet of blog posts.

For question RQ4, viz. How do the criteria that the respondents say they use
compare with the responding behaviour of those respondents in the survey
itself?, we perform two comparisons. First, we compare characteristics of the
survey responses against a dataset of blog posts. Second, we use the
synthesised checklists to analyse selected survey responses. We observe that
survey respondents’ behaviour appears similar in some aspects to the behaviour
of authors of blog posts, e.g., survey respondents made assertions and these
assertions were rarely supported by argument or reasoning, and researchers did
not formally cite previous research to support their assertions. The number of
criteria within the checklists indicate the complexity of evaluating the
credibility of blog posts, and also partially explains the contrasts in
attitudes of the researchers who responded to our survey. Again, the variety
of responses from the researchers suggests differences in worldviews, and
therefore differences in researchers’ research questions and topics of
interest, and in the ‘logic’ that researchers use to define and evaluate the
credibility of data.

###  8.2 Implications of the findings

The current paper, and the associated survey, were motivated by two issues:
the recognition that the use of blog posts as evidence in software engineering
is contentious for some, and the recognition of the significant challenge of
quality–assuring blog posts. These two issues are clearly connected.

With regards to the first motivation, concerning contention, the findings
reported in this paper are not intended to resolve the contention(s) present
in the research community; rather, the paper seeks to increase the community’s
awareness of these issues, as well as the complexity of these issues, and to
encourage the community to engage with these issues.

With regards to the second motivation, concerning quality–assurance, the paper
develops a preliminary suite of checklists specifically for blog posts to help
the community quality–assure blog posts.

###  8.3 Threats to validity

There are a number of threats to the validity of this study, these threats
also indicating directions for future research. We enumerate the responses:

  1. 1.

Whilst the response rate is reasonable (32%) the absolute number of responses
(44) is relatively low. We have managed this threat through retaining all
responses and conducting and reporting a rich sample of qualitative responses
(over 60 quotes) to complement the quantitative analyses.

  2. 2.

Kitchenham and Pfleeger (2002) advise that survey designers should specify
their population of interest. We have implicitly defined a population by
targeting members of programme committees for two very well–established
conferences in the field. The implied population was therefore empirical
software engineering researchers. We could have been both more explicit on
that population but also more specific, e.g., the population of empirical
software engineering researchers who have some familiarity with blogs and blog
posts.

  3. 3.

We didn’t explicitly ask respondents about their experience with blog posts
e.g., the degree to which they read blog posts and follow different blogs. We
also didn’t explicitly ask respondents about their expectations of blog posts.
These limitation prevent us from distinguishing the more informed and
experienced respondents from those who are less informed and less experienced.
We have considered the exclusion of cases as part of our survey design in
section 3 and then in our thematic analyses in section 6. With this survey
being the first survey of its kind, and given the limited sample size, we
decided to retain all responses for the current paper, to maximise the
insights on this topic. (In our original paper, we excluded one respondents.)

  4. 4.

A related concern is that we didn’t provide guidelines to potential
respondents on whether they should exclude themselves, or whether and how they
should declare their degree of confidence in their responses. The respondent
(see quote 6 in Table 6) who stated that she or he does not follow blogs is an
example of a respondent who likely should not participate in a future study of
this kind. We have managed this threat in several ways: 1) retaining the
entire dataset for completeness, 2) reporting a rich sample of quotes, 3)
reporting the respondent’s overall rating for the credibility of blog posts
with each quote, and 4) synthesising a suite of checklists from the quotes.

  5. 5.

We asked respondents for their general views on blogs and blog posts, whereas
we could have provided specific examples of blog posts and asked respondents
about those examples. A different survey design might have revealed different
criteria, however the current analyses and the synthesised checklists indicate
that a rich set of criteria have been developed.

  6. 6.

We could have conducted more detailed follow–ups with both non–respondents and
respondents. We intend to do this with those respondents who have expressed a
willingness for a follow–up.

  7. 7.

A more sophisticated survey could ask multiple questions (phrased differently)
about each credibility criteria so as to establish with confidence consistent
responses. One explanation for the above limitations is that we wanted to
conduct a survey that could be completed relatively quickly so as to reduce
drop–out rates and increase the number of completed responses.

  8. 8.

There is uncertainty as to what cases we should exclude and also uncertainty
over what cases we should retain for the analyses. We have three different
types of indicator:

     * •

At least one respondent who completed the survey very quickly and who also
responded with the same values (i.e., scored five for all nine criteria).

     * •

One respondent who scored all Likert criteria consistently low.

     * •

A small number of respondents who state in their qualitative comments that
they do read blog posts, or do not read them often.

###  8.4 Further research

There are a number of directions for further research in this area, and again
we enumerate them.

  1. 1.

An obvious direction for further research is to conduct more studies that
address the threats and limitations identified in subsection 8.3.

  2. 2.

With the raw responses from this survey available online, there is the
opportunity for others to independently analyse the published data from the
current study.

  3. 3.

Similarly, there is the opportunity for others to independently replicate the
existing study.

  4. 4.

The preliminary checklists warrant application, evaluation and revision,
including extension.

  5. 5.

It would also be valuable to widen the scope of this study to consider other
kinds of grey literature and to compare the design, collection and analyses of
grey literature in comparison to white literature. Such comparisons are timely
as the research community increasingly uses social media in its research.

###  8.5 Conclusions

Blog posts are a potential source of evidence for software engineering
research however there are concerns relating to the quality of blog posts. We
conducted a survey to better understand researchers’ attitudes to blog posts.
We explicitly asked respondents about nine pre–defined credibility criteria,
as well as inviting respondents to identify additional criteria. The nine
criteria were identified from an extensive review of prior literature on
credibility. We found no clear majority consensus on which of the nine
criteria to use to evaluate the credibility of blog posts. Instead, there
appeared to be subgroups of respondents who broadly agreed on particular
criteria. We conducted thematic analyses of the qualitative comments from
respondents to identify additional criteria. We then developed a preliminary
suite of checklists for evaluating credibility, comprising several categories
of criteria: foundational, process, author, content, media, influencer and
reader. The checklists were synthesised from four sources of information:
respondents’ attitudes to the nine criteria, additional criteria proposed by
respondents, criteria identified from the literature reviewed, and two quality
checklists, one generic checklist for MLRs and one checklist for a specific
GLR. We partially demonstrated the checklists by applying them back to the
survey itself, and in so doing we exposed similarities between survey
responses and blog posts. Finally, we reflected on implications for research
and practice, threats to the validity of our work, and opportunities for
further research.

## Acknowledgements

We thank the 44 survey respondents for their engagement in the survey. We
thank members of Software Innovation NZ for their support in the development
of the survey. We thank the reviewers in anticipation of their reviews. The
survey was approved by the appropriate University of Canterbury (New Zealand)
Ethics Committee (HEC 2017/68/LR-PS).

## References

  * Aniche et al. [2018] Maurício Aniche, Christoph Treude, Igor Steinmacher, Igor Wiese, Gustavo Pinto, Margaret-Anne Storey, and Marco Aurélio Gerosa.  How modern news aggregators help development communities shape and share knowledge.  In _Int’l. Conf. on Software Engineering_ , 2018.
  * Burton et al. [2009] Kevin Burton, Akshay Java, Ian Soboroff, et al.  The icwsm 2009 spinn3r dataset.  In _Third Annual Conference on Weblogs and Social Media (ICWSM 2009)_ , 2009.
  * Chau et al. [2009] Michael Chau, Jennifer Xu, Jinwei Cao, Porsche Lam, and Boby Shiu.  A blog mining framework.  _It Professional_ , 11(1):36–41, 2009.
  * Clarke and O’Connor [2012] Paul Clarke and Rory V O’Connor.  The situational factors that affect the software development process: Towards a comprehensive reference framework.  _Information and Software Technology_ , 54(5):433–447, 2012.
  * Devanbu et al. [2016] Prem Devanbu, Thomas Zimmermann, and Christian Bird.  Belief & evidence in empirical software engineering.  In _Proceedings of the 38th international conference on software engineering_ , pages 108–119. ACM, 2016.
  * Dyba [2013] Tore Dyba.  Contextualizing empirical evidence.  _IEEE software_ , 30(1):81–83, 2013.
  * Dybå et al. [2005] Tore Dybå, Barbara A Kitchenham, and Magne Jorgensen.  Evidence–based software engineering for practitioners.  _IEEE software_ , 22(1):58–65, 2005.
  * Dybå et al. [2012] Tore Dybå, Dag IK Sjøberg, and Daniela S Cruzes.  What works for whom, where, when, and why?: on the role of context in empirical software engineering.  In _Proceedings of the ACM–IEEE international symposium on Empirical software engineering and measurement_ , pages 19–28. ACM, 2012.
  * Fenton et al. [1994] Norman Fenton, Shari Lawrence Pfleeger, and Robert L. Glass.  Science and substance: A challenge to software engineers.  _IEEE software_ , 11(4):86–95, 1994.
  * Garousi and Mäntylä [2016] Vahid Garousi and Mika V Mäntylä.  When and what to automate in software testing? a multi–vocal literature review.  _Information and Software Technology_ , 76:92–117, 2016\.
  * Garousi et al. [2018] Vahid Garousi, Michael Felderer, and Mika V Mäntylä.  Guidelines for including grey literature and conducting multivocal literature reviews in software engineering.  _Information and Software Technology_ , 2018.
  * Gordon and Swanson [2009] Andrew Gordon and Reid Swanson.  Identifying personal stories in millions of weblog entries.  In _Third International Conference on Weblogs and Social Media, Data Challenge Workshop, San Jose, CA_ , volume 46, 2009.
  * Inui et al. [2008] Kentaro Inui, Shuya Abe, Kazuo Hara, Hiraku Morita, Chitose Sao, Megumi Eguchi, Asuka Sumida, Koji Murakami, and Suguru Matsuyoshi.  Experience mining: Building a large–scale database of personal experiences and opinions from web documents.  In _Proceedings of the 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology–Volume 01_ , pages 314–321. IEEE Computer Society, 2008.
  * Ivarsson and Gorschek [2011] Martin Ivarsson and Tony Gorschek.  A method for evaluating rigor and industrial relevance of technology evaluations.  _Empirical Software Engineering_ , 16(3):365–395, Jun 2011.  ISSN 1573–7616.  10.1007/s10664–010–9146–4.  URL https://doi.org/10.1007/s10664-010-9146-4.
  * Khan et al. [2017] Hikmat Ullah Khan, Ali Daud, Umer Ishfaq, Tehmina Amjad, Naif Aljohani, Rabeeh Ayyaz Abbasi, and Jalal S Alowibdi.  Modelling to identify influential bloggers in the blogosphere: A survey.  _Computers in Human Behavior_ , 68:64–82, 2017.
  * Kitchenham and Charters [2007] B. Kitchenham and S Charters.  Guidelines for performing systematic literature reviews in software engineering, 2007.
  * Kitchenham and Pfleeger [2002] Barbara Kitchenham and Shari Lawrence Pfleeger.  Principles of survey research: part 5: populations and samples.  _ACM SIGSOFT Software Engineering Notes_ , 27(5):17–20, 2002.
  * Kitchenham et al. [2002] Barbara A Kitchenham, Shari Lawrence Pfleeger, Lesley M Pickard, Peter W Jones, David C. Hoaglin, Khaled El Emam, and Jarrett Rosenberg.  Preliminary guidelines for empirical research in software engineering.  _IEEE Transactions on software engineering_ , 28(8):721–734, 2002.
  * Kurashima et al. [2006] Takeshi Kurashima, Taro Tezuka, and Katsumi Tanaka.  Mining and visualizing local experiences from blog entries.  In _DEXA_ , pages 213–222. Springer, 2006.
  * Kurashima et al. [2009] Takeshi Kurashima, Ko Fujimura, and Hidenori Okuda.  Discovering association rules on experiences from large–scale blog entries.  In _European Conference on Information Retrieval_ , pages 546–553. Springer, 2009.
  * Lakshmanan and Oberhofer [2010] Geetika Lakshmanan and Martin Oberhofer.  Knowledge discovery in the blogosphere: Approaches and challenges.  _IEEE internet computing_ , 14(2):24–32, 2010\.
  * Pagano and Maalej [2011] Dennis Pagano and Walid Maalej.  How do developers blog?: an exploratory study.  In _Proceedings of the 8th working conference on Mining software repositories_ , pages 123–132. ACM, 2011.
  * Park et al. [2010] Keun Chan Park, Yoonjae Jeong, and Sung Hyon Myaeng.  Detecting experiences from weblogs.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_ , pages 1464–1472. Association for Computational Linguistics, 2010.
  * Parnin and Treude [2011] Chris Parnin and Christoph Treude.  Measuring api documentation on the web.  In _Proceedings of the 2nd international workshop on Web 2.0 for software engineering_ , pages 25–30. ACM, 2011.
  * Parnin et al. [2013] Chris Parnin, Christoph Treude, and Margaret-Anne Storey.  Blogging developer knowledge: Motivations, challenges, and future directions.  In _Program Comprehension (ICPC), 2013 IEEE 21st International Conference on_ , pages 211–214. IEEE, 2013.
  * Petersen and Gencel [2013] Kai Petersen and Cigdem Gencel.  Worldviews, research methods, and their relationship to validity in empirical software engineering research.  In _2013 Joint Conference of the 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement_ , pages 81–89. IEEE, 2013.
  * Petersen and Wohlin [2009] Kai Petersen and Claes Wohlin.  Context in industrial software engineering research.  In _Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement_ , pages 401–404. IEEE Computer Society, 2009.
  * Petersen et al. [2015] Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz.  Guidelines for conducting systematic mapping studies in software engineering: An update.  _Information and Software Technology_ , 64:1–18, 2015.
  * Prechelt and Petre [2010] Lutz Prechelt and Marian Petre.  Credibility, or why should i insist on being convinced?  _Making Software: What Really Works, and Why We Believe It_ , page 17, 2010.
  * Rainer [2017] Austen Rainer.  Using argumentation theory to analyse software practitioners’ defeasible evidence, inference and belief.  _Information and Software Technology_ , 2017.
  * Rainer and Williams [2018] Austen Rainer and Ashley Williams.  Using blog articles in software engineering research: benefits, challenges and case–survey method.  In _Proceedings of the 25th Australasian Software Engineering Conference (ASWEC 2018)_ , 2018.
  * Rainer and Williams [2019] Austen Rainer and Ashley Williams.  Using blog-like documents to investigate software practice: Benefits, challenges, and research directions.  _Journal of Software: Evolution and Process_ , 31(11):e2197, 2019.
  * Rainer et al. [2003] Austen Rainer, Tracy Hall, and Nathan Baddoo.  Persuading developers to” buy into” software process improvement: a local opinion and empirical evidence.  In _Empirical Software Engineering, 2003. ISESE 2003. Proceedings. 2003 International Symposium on_ , pages 326–335. IEEE, 2003.
  * Raulamo-Jurvanen et al. [2017] Päivi Raulamo-Jurvanen, Mika Mäntylä, and Vahid Garousi.  Choosing the right test automation tool: A grey literature review of practitioner sources.  In _Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering_ , EASE’17, pages 21–30, New York, NY, USA, 2017. ACM.  ISBN 978–1–4503–4804–1.  10.1145/3084226.3084252.  URL http://doi.acm.org/10.1145/3084226.3084252.
  * Runeson and Höst [2009] Per Runeson and Martin Höst.  Guidelines for conducting and reporting case study research in software engineering.  _Empirical software engineering_ , 14(2):131, 2009\.
  * Runeson et al. [2012] Per Runeson, Martin Höst, Austen Rainer, and Bjorn Regnell.  _Case study research in software engineering: Guidelines and examples_.  John Wiley & Sons, 2012.
  * Soldani et al. [2018] Jacopo Soldani, Damian Andrew Tamburri, and Willem-Jan Van Den Heuvel.  The pains and gains of microservices: A systematic grey literature review.  _Journal of Systems and Software_ , 2018.
  * Storey et al. [2010] Margaret-Anne Storey, Christoph Treude, Arie van Deursen, and Li-Te Cheng.  The impact of social media on software engineering practices and tools.  In _Proceedings of the FSE/SDP workshop on Future of software engineering research_ , pages 359–364. ACM, 2010.
  * Storey et al. [2014] Margaret-Anne Storey, Leif Singer, Brendan Cleary, Fernando Figueira Filho, and Alexey Zagalsky.  The (r) evolution of social media in software engineering.  In _Proceedings of the on Future of Software Engineering_ , pages 100–116. ACM, 2014.
  * Swanson et al. [2014] Reid Swanson, Elahe Rahimtoroghi, Thomas Corcoran, and Marilyn Walker.  Identifying narrative clause types in personal stories.  In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_ , pages 171–180, 2014.
  * Williams [2018a] Ashley Williams.  Do software engineering practitioners cite research on software testing in their online articles?: A preliminary survey.  In _Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018_ , pages 151–156. ACM, 2018a.
  * Williams [2018b] Ashley Williams.  Using reasoning markers to select the more rigorous software practitioners’ online content when searching for grey literature.  In _Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018_ , pages 46–56. ACM, 2018b.
  * Williams and Rainer [2018] Ashley Williams and Austen Rainer.  _The analysis and synthesis of previous work on credibility assessment in online media: technical report_.  University of Canterbury, New Zealand, 04 2018.  URL https://www.researchgate.net/publication/324765770_The_analysis_and_synthesis_of_previous_work_on_credibility_assessment_in_online_media_technical_report.
  * Williams and Rainer [2019a] Ashley Williams and Austen Rainer.  How do software engineering researchers assess the credibility of practitioner–generated blog posts?  In _23rd International Conference on Evaluation and Assessment in Software Engineering (EASE’19), 14 – 17 April 2019, Copenhagen, Denmark_ , 2019a.
  * Williams and Rainer [2019b] Ashley Williams and Austen Rainer.  Do software engineering practitioners cite research in their online articles? a larger scale replication.  In _23rd International Conference on Evaluation and Assessment in Software Engineering (EASE’19), 14 – 17 April 2019, Copenhagen, Denmark_ , 2019b.
  * Wohlin [2013] Claes Wohlin.  An evidence profile for software engineering research and practice.  In _Perspectives on the Future of Software Engineering_ , pages 145–157. Springer, 2013.

## Appendix

### Appendix A - Survey questions

Table 19 presents the questions used in the survey together with permissible
values. For Q2, we used a text field to allow respondents flexibility in
response. For Q3, the range of Likert values was O: No blog is credible to 5:
All blogs are credible. For the Q5 sub–questions, the range of Likert values
was 0: Not at all important to 6: Extremely important.

Table 19: Summary of survey questions # |  Question |  Values
---|---|---
Q1 |  Please summarise your specific areas of research? Please be more specific than ’software engineering’ :). Separate each area with a semi-colon e.g. software testing; requirements engineering. |  Text
Q2 |  How many years experience do you have conducting software engineering research? (Enter number to one decimal place.) |  Text
Q3 |  In general, how credible do you consider blog articles that are written by practitioners? |  Likert
Q4 |  Please provide general comments to complement your rating. (Optional.) |  Text
Q5.1 |  For each of the following criteria, please rate how important you consider the criterion to be when assessing quality: - The clarity of writing within the blog article |  Likert
Q5.2 |  For each of the following criteria, please rate how important you consider the criterion to be when assessing quality: - The reporting of empirical data within the blog article that relates to the claims made in the article |  Likert
Q5.3 |  For each of the following criteria, please rate how important you consider the criterion to be when assessing quality: - The reporting of the method by which the empirical data was collected and analysed |  Likert
Q5.4 |  For each of the following criteria, please rate how important you consider the criterion to be when assessing quality: - The reporting of professional experience within the blog article that relates to the claims made in the blog article |  Likert
Q5.5 |  For each of the following criteria, please rate how important you consider the criterion to be when assessing quality: - The presence of web links to other practitioner sources that relates to the claims made in the article |  Likert
Q5.6 |  For each of the following criteria, please rate how important you consider the criterion to be when assessing quality: - The presence of web links to peer-reviewed research that relates to the claims made in the article |  Likert
Q5.7 |  For each of the following criteria, please rate how important you consider the criterion to be when assessing quality: - The presence of reasoning within the blog that relates to the claims made in the article |  Likert
Q5.8 |  For each of the following criteria, please rate how important you consider the criterion to be when assessing quality: - The prior beliefs of the reader when reading the article |  Likert
Q5.9 |  For each of the following criteria, please rate how important you consider the criterion to be when assessing quality: - The influence of other people’s opinions on the reader’s beliefs e.g. a recommendation to read the blog article |  Likert
Q6 |  Please provide more information to explain your answers. |  Text
Q7 |  Are there any other criteria that you think are missing or have not been considered by this survey? Please summarise those other criteria. |  Text
Q8 |  Do you think that the criteria identified in Q5 and Q7 generalise to assessing the quality of content written by practitioners, other than blogs (e.g. emails, Q&A sites such as Stack Exchange, comments that have been provided in response to blog articles)? |  Text
Q9 |  Please provide more information to explain your answer |  Text
Q10 |  Do you think the criteria identified in Q5 and Q7 generalise to assessing the quality of content written by researchers e.g. journal articles, conference papers? |  Text
Q11 |  Please provide more information to explain your answer. |  Text
Q12 |  Please provide any other comments you would like to make on this survey. |  Text

### Appendix B - Bubble plots for the criteria within the survey

Figures 5, 6, and 7 present bubble plots for each of the nine criteria and, in
so doing, the plots provide a kind of transposition of the data presented in
the heatmap.

For each bubble plot the x–axis presents the scores for the General
Credibility (see GC in Table 3), the y–axis presents the scores for the
respective criterion (in the range 0 – 6), and the size of the dot indicates
the number of responses. A LOESS (locally estimated scatterplot smoothing)
line is plotted, together with (again) a confidence interval. Once again, we
use these plots for the purpose of exploratory data analyses. And once again,
we proceed cautiously in the interpretation of the data, e.g., the dataset is
clearly imbalanced with about half of the responses plotted at $x=3$.

Figure 5a and Figure 5c both suggest, unsurprisingly, an inverse relationship
between the credibility of blogs posts and the importance of reporting
empirical data and reporting research methods: respondents who place a higher
value on reporting empirical data and on reporting research methods also place
a lower value on the credibility of blog posts, and vice versa. As noted
earlier, this is unsurprising given the value that researchers place on
evidence and on evidence–collection. In our excluded–case analysis, respondent
#22 appears to take a ‘purest’ view of this relationship, i.e., rejecting
entirely the other seven criteria.

For the three sub–figures in Figure 7, there is curious behaviour where $x=1$
and where $x=2$, i.e., when the score for General Credibility is 1 or 2. In
all three figures, it is not clear whether the ratings for the criterion are
unusually high for $x=1$ or unusually low for $x=2$; or whether instead the
data is dominated by responses at $x=3$. And these patterns suggest the
presence of sub–populations.

(a) Empirical data

(b) Reasoning

(c) Research method

(d) Clarity of writing

Figure 5: Bubble charts with Loess smoother and confidence interval

(a) Citing research

(b) Citing practice

Figure 6: Dotplots with Loess smoother and confidence interval

(a) Prof. experience

(b) Others’ influence

(c) Prior beliefs

Figure 7: Dotplots with Loess smoother and confidence interval