# Merits and Limits: Applying open data to monitor open access publications in
bibliometric databases

Aliakbar Akbaritabar111German Centre for Higher Education Research and Science
Studies (DZHW), Schützenstr. 6a, Berlin, 10117 (Germany<EMAIL_ADDRESS>(corresponding author)
Stephan Stahlschmidt222German Centre for Higher Education Research and Science
Studies (DZHW<EMAIL_ADDRESS>
###### Abstract

Identifying and monitoring Open Access (OA) publications might seem a trivial
task while practical efforts prove otherwise. Contradictory information arise
often depending on metadata employed. We strive to assign OA status to
publications in Web of Science (WOS) and Scopus while complementing it with
different sources of OA information to resolve contradicting cases. We linked
publications from WOS and Scopus via DOIs and ISSNs to Unpaywall, Crossref,
DOAJ and ROAD. Only about 50% of articles and reviews from WOS and Scopus
could be matched via a DOI to Unpaywall. Matching with Crossref brought 56
distinct licences, which define in many cases the legally binding access
status of publications. But only 44% of publications hold only a single
licence on Crossref, while more than 50% have no licence information submitted
to Crossref. Contrasting OA information from Crossref licences with Unpaywall
we found contradictory cases overall amounting to more than 25%, which might
be partially explained by (ex-)including green OA. A further manual check
found about 17% of OA publications that are _not accessible_ and 15% non-OA
publications that are _accessible_ through publishers’ websites. These
preliminary results suggest that identification of OA state of publications
denotes a difficult and currently unfulfilled task.

Keywords: Open Access, Unpaywall, Crossref, Web Of Science, Scopus

## Introduction

Open access (henceforth OA) in scholarly communication describes unrestricted
access to published peer-reviewed documents written by and addressed to
researchers. These documents have traditionally been disseminated via
publications in scientific journals, which charge for access to the respective
content. Stimulated by a call for greater openness and transparency in general
(“open science”), the OA movement has nowadays been accepted as one, though
not the only, alternative for the dissemination of scholarly documents. Even
publishers seem to embrace this new model as providing a suitable
infrastructure while at the same time securing their own economic interests.

This inter-mixture of interests has resulted not only in one, but several
forms of OA publications such as _Gold_ , _Hidden Gold_ , _Hybrid_ , _Green_ ,
_Delayed_ , _Bronze_ and _Black_ which are mainly based on right to access and
pay to publish models depending on venues where the OA publication is
accessible.

Due to the individual ascription of single publications to one or several of
these categories and the decentralized structure of the scientific publishing
market with a variety of diverse publishers, the identification of OA is less
trivial than it might seem. Even large bibliometric data provider rely on
external information to provide information on
OA333https://clarivate.com/blog/easing-access-to-open-access-clarivate-
analytics-partners-with-impactstory/ and most large scale undertakings by the
scientometric community to obtain reliable information on OA prevalence rely
on the use of web crawlers (Archambault et al., 2013; Piwowar et al., 2018)

Inspired by the Hybrid OA Dashboard (Jahn, 2017) we applied licensing
information detailing the legally binding access state supplied by publishers
to the publisher association Crossref to identify OA publications. We
determined the OA status of all publications retrieved from Web of Science
(henceforth WOS) and Scopus in-house databases of 2017 by confronting them to
two sources of OA information, i.e., Unpaywall and Crossref. In Section 2, we
present our data and methods. In Section 3 we present our findings, while we
discuss our main results in Section 4.

## Data & Method

We queried all publications from Scopus and WOS in in-house databases of
2017\. Data included article’s unique ID from database and DOI. We matched
those DOIs with Unpaywall database from April 18th 2018 to determine the OA
status for each single publication. In parallel, we matched those DOIs with
Crossref data (using snapshot of the data from April 2018 based on plus
service described in Crossref (2018)) and retrieved the available information
on the licences of publications444It is neccessary to note that our effort to
send large number of requests to Crossref API (even while using plus service
and through both rcrossref package in R and more fine-grained httr requests
directly to Crossref API) faced timeout and response time errors and
alternatively we chose to use the in-house snapshot of the Crossref data to
circumvent the above error. This meant parsing large corpus of JSON files
which can be time consuming depending on the goals of the analysis. Any effort
on automating the proposed OA identification procedure needs to overcome the
technical issues like this..

Additionally, we used the journals’ ISSNs provided by Wohlgemuth, Rimmert, &
Winterhager (2016) (and the updated version in Rimmert, Bruns, Lenke, &
Taubert (2017)) to identify Gold OA publications. They use different known OA
indexes (e.g., DOAJ555https://doaj.org/ (Directory of Open Access Journals)
and ROAD666https://road.issn.org/ (Directory of Open Access scholarly
Resources) and determine if the respective ISSN is listed in those databases.
They differentiate between _ISSN_ and _ISSNL_ which is more fine-grained by
adding a specific ISSN to some special issues. We tried both ISSN and ISSNL,
sicne the latter had higher matching records, therefore in our analysis
presented in the Results section we use the _ISSNL_.

It is necessary to note that some publications had multiple licence URLs in
Crossref database, we followed a procedure with four steps to ensure using
only one licence per publication (see Table 2 for the frequencies of these
publications):

  1. 1.

If a publication had only one record in Crossref database, whether it had an
_OA_ , _non-OA_ , _unclear_ licence or _no licence information (i.e. NA)_ , we
used this status and categorized the publication as a unique one.

  2. 2.

If a publication had multiple _OA_ licence URLs, we removed the duplicates and
categorized it as _OA_.

  3. 3.

If a publication had a mixture of _OA_ and _non-OA_ licence URLs, we removed
the duplicates and categorized it as _OA_.

  4. 4.

If a publication had multiple _non-OA_ licence URLs, we removed the duplicates
and categorized it as _non-OA_.

A research assistant controlled the unique licences (a total of 56) we
extracted from Crossref with available information online to categorize them
as _OA_ and _non-OA_. We used this categorization in parallel to established
OA identification procedures (e.g., searching for journal’s ISSN in DOAJ and
ROAD in Gold OA identification) to ensure a higher level of robustness in our
results.

In OA Identification process and in order to determine if a publication was OA
or not, we applied a multi-category view separating Gold, Hidden Gold, Hybrid
and Delayed OA, while doing so, we reached a new category of _Probable Hybrid
OA_. Our investigation strategy for each category was as follows:

  * •

Gold OA: As described earlier, we used the ISSNs provided by Rimmert et al.
(2017) to determine Gold OA. We matched the respective ISSN (from both WOS and
Scopus) with DOAJ and ROAD. If the respective ISSN was listed in one of those
directories, the publication is categorized as _Gold OA_. We confronted Gold
OA from DOAJ and ROAD with our research assistant’s categorization of Crossref
licences after the manual check of unique licence URLs.

  * •

Hidden Gold OA: we used metadata from WOS and Scopus to determine the journal
issue and looking at the licences of all publications in a single issue, if
all publications had _OA_ licences, but the ISSN was not indexed in DOAJ or
ROAD we categorized it as _Hidden Gold OA_.

  * •

Hybrid OA: If an issue had at least one _non-OA_ publication while having one
or more _OA_ publications, we categorized the OA publications as _Hybrid OA_.

  * •

Probable Hybrid OA: If an issue did not have a _non-OA_ publication while
having one or more _OA_ publications and some publications in the issue didn’t
have licence information, we categorized them as _Probable Hybrid OA_.

  * •

Delayed OA: In all of the above cases, we looked into delays based on Crossref
metadata (a difference in terms of days from day of publication and the date
licence was assigned to the publication as described in CrossRef-API (2019),
this is the time period known as _embargo time_) to determine if they were
_Delayed_ , therefore each of the above categories were split to two groups,
_delayed_ and _not-delayed_. If a publication had multiple licence URLs on
Crossref, we controlled their respective delay times, if any of those were
_not-delayed_ we categorized the publication as such, while if any of the
licences were _delayed_ , the publication is identified as a _delayed_ one.

  * •

Closed Access: Strictly speaking, if the number of publications in an issue
was equal to the number of _non-OA_ publications and the ISSN was not indexed
in DOAJ or ROAD, we categorized them as _Closed Access_.

  * •

NA (Not available): A publication that was not fitting in any of the above
categories or did not have a licence URL to determine its condition was
categorised as _NA_. Number of NAs are higher than _Closed Access_
publications, since we aimed to keep the definitions as strict as possible.

## Results

We present the results in two main sections, one regarding _Unpaywall_ and the
other on licences extracted from _Crossref_. We then present the _comparison
between Unpaywall and Crossref_ and the results of our _manual checks on
random samples_ for robustness of the results.

Table 1 shows the number of _articles_ and _review papers_ from WOS and Scopus
with an equivalent record in Unpaywall database. It presents also the total
number of _articles_ and _review papers_ in WOS/Scopus to provide a baseline
for comparison. Unpaywall has higher than 50% coverage in both cases while
coverage of WOS is slightly higher (can be due to different indexing
philosophy or DOIs completeness). In the following tables (in _Unpaywall_
results), publications are limited to only _articles_ and _review papers_
published in 2000-2017.

Table 1: All publications from WOS (2000-2017) and Scopus (2000-2017) that have an equivalent record in Unpaywall database (joined by DOIs) Data Source | Frequency | Percent
---|---|---
WOS (matched Unpaywall Only articles & reviews) | 11,661,206 | 57.5%
WOS (Only articles & reviews 2000-2017) | 20,280,606 | -
Scopus (matched Unpaywall Only articles & reviews) | 14,188,983 | 53.48%
Scopus (Only articles & reviews 2000-2017) | 26,532,295 | -

Figure 1 presents the distribution of journals and publications indexed in WOS (top) and Scopus (bottom) matched with Unpaywall database and crosschecked the ISSNs with DOAJ. _Missing on DOAJ_ in these Figures refer to those journals whose ISSN was missing from Rimmert et al. (2017) data, therefore we could not check if the ISSN is listed in DOAJ or not while _Others_ means the ISSN was existing in Rimmert et al. (2017) but it was not listed as _OA_ in DOAJ. Share of pubilcations which don’t have a matching ISSN in DOAJ (meaning they are not Gold OA) and are identified as OA in Unpaywall is interesting on both Figures (designated with “Missing on DOAJ | Unpaywall OA” as label). They could be other OA types (green, hybrid, hidden gold).

Figure 1: Publications indexed in WOS (top) and Scopus (bottom) matched with
Unpaywall database and crosschecked the ISSNs with DOAJ (Gold OA) (X-axis
denotes the years, Y-axis denotes the number of publications in each year)

We matched publications to Crossref data from April 2018 and found 56 distinct
licence types for all of the publications. Table 2 presents a descriptive view
on whether publications have licence information recorded in Crossref. It
shows that about 50% of publications from WOS or Scopus with a matching DOI
indexed in Crossref do not have a licence URL. Some of the publications had
more than one licence information in Crossref (as an example, the number of
DOIs that each have 6 licence records on Crossref are 7). In case of multiple
licences, if a publication had at least one OA licence, we categorized it as
_OA_.

Table 2: Number of licences per DOI found in Crossref Number of licences per DOI | Frequency of DOIs | Percent
---|---|---
0 | 9,892,208 | 51.41
1 | 8,520,158 | 44.28
2 | 824,975 | 4.29
3 | 5,770 | 0.03
5 | 25 | 0.00
6 | 7 | 0.00

Figure 2 present the _Gold_ , _Hidden Gold_ , _Hybrid_ and _Delayed OA_ status
of the publications from WOS (top) and Scopus (bottom), which is presented as
trends over the years. We limited the years to 2000-2017 to show the most
recent trends. To make these Figures more readable, we removed _NA_ (those
without a matching DOI or without a licence information on Crossref).

Figure 2: Comparison of OA publications 2000-2017 (WOS (top) and Scopus
(bottom) data matched with Crossref) (X-axis denotes the years, Y-axis denotes
the number of publications in each year)

Tables 3 and 4 present the _OA status_ comparison between Unpaywall and
Crossref in WOS and Scopus publications, respectively. Note, Crossref OA
status in the Tables is the categorization we developed using respective
licence URLs. We double checked the contradictory cases and improved our
while-list of OA licences, while some of the contradictions still remain
(e.g., Unpaywall declares those publications as OA while they are closed
access or vice versa, in case of licences on Crossref that are open access
while the publication is declared as non-OA on Unpaywall). Overall
contradictory cases amount to 27.95% in WOS and 27.57% in Scopus which might
partly be explained by the wider scope of Unpaywall including also green OA
publications that might not be identified via license information only.

Table 3: OA status comparison between Unpaywall and Crossref in WOS publications Crossref OA Status | Unpaywall OA Status | Frequency | Percent
---|---|---|---
Closed Access | Closed Access | 4,452,185 | 38.18
NA | Closed Access | 3,512,794 | 30.12
NA | Open Access | 1,770,612 | 15.18
Closed Access | Open Access | 1,363,525 | 11.69
Open Access | Open Access | 435,516 | 3.73
Open Access | Closed Access | 126,354 | 1.08
Closed Access | NA | 26 | 0.00
NA | NA | 19 | 0.00
Table 4: OA status comparison between Unpaywall and Crossref in Scopus publications Crossref OA Status | Unpaywall OA Status | Frequency | Percent
---|---|---|---
Closed Access | Closed Access | 5,138,444 | 36.21
NA | Closed Access | 4,635,801 | 32.67
NA | Open Access | 2,201,936 | 15.52
Closed Access | Open Access | 1,549,902 | 10.92
Open Access | Open Access | 502,510 | 3.54
Open Access | Closed Access | 160,132 | 1.13
NA | NA | 15 | 0.00
Open Access | NA | 4 | 0.00
Closed Access | NA | 1 | 0.00

Tables 5 and 6 present the result of our research assistant’s manual check for
accessibility to article’s PDF file from publishers websites compared to the
respecitve licence in Crossref and the OA status we manually assignded to
those URLs in contrast to OA status from Unpaywall. It is interesting to see
there are publications defined as _Non-OA_ while their PDF is accessible from
the publisher (14.42% in WOS and 14.98% in Scopus) or vice versa, OA
publications (based on either Unpaywall, Crossref or both) that are not
accessible online (17.57% in WOS and 16.74% in Scopus). Note also the
contradictory cases between Crossref and Unpaywall, where metadata from one
shows _OA_ and the other _Closed_ , which requires further probes (22.98% in
WOS and 22.91% in Scopus, these percentages are quite close to contradictions
observed in the overall sample presented in Tables 3 and 4). Our effort to
complement these databases proves that none of them could be used in
isolation. We aim to follow-up and use PDF URLs provided by Unpaywall in large
scale to control the ratio of publications which can be accessed.

Table 5: Random sample OA status check on publications from WOS PDF Manually accessible? | Licence status | Pub OA? | Frequency | Percent
---|---|---|---|---
PDF Accessible | Open Access | Unpaywall OA | 104 | 46.85
No Access to PDF | Closed Access | Unpaywall non-OA | 44 | 19.82
No Access to PDF | Open Access | Unpaywall non-OA | 18 | 8.11
No Access to PDF | Closed Access | Unpaywall OA | 16 | 7.21
PDF Accessible | Closed Access | Unpaywall OA | 16 | 7.21
PDF Accessible | Closed Access | Unpaywall non-OA | 14 | 6.31
No Access to PDF | Open Access | Unpaywall OA | 5 | 2.25
NA | Closed Access | Unpaywall non-OA | 1 | 0.45
No Access to PDF | Closed Access | Missing on Unpaywall | 1 | 0.45
PDF Accessible | NA | Unpaywall non-OA | 1 | 0.45
PDF Accessible | Open Access | Unpaywall non-OA | 1 | 0.45
PDF Accessible | NA | Unpaywall OA | 1 | 0.45
Table 6: Random sample OA status check on publications from Scopus PDF Manually accessible? | Licence status | Pub OA? | Frequency | Percent
---|---|---|---|---
PDF Accessible | Open Access | Unpaywall OA | 105 | 46.26
No Access to PDF | Closed Access | Unpaywall non-OA | 48 | 21.15
No Access to PDF | Closed Access | Unpaywall OA | 17 | 7.49
PDF Accessible | Closed Access | Unpaywall OA | 17 | 7.49
No Access to PDF | Open Access | Unpaywall non-OA | 17 | 7.49
PDF Accessible | Closed Access | Unpaywall non-OA | 14 | 6.17
No Access to PDF | Open Access | Unpaywall OA | 4 | 1.76
NA | Closed Access | Unpaywall non-OA | 1 | 0.44
PDF Accessible | NA | Unpaywall non-OA | 1 | 0.44
PDF Accessible | Open Access | Unpaywall non-OA | 1 | 0.44
PDF Accessible | NA | Missing on Unpaywall | 1 | 0.44
PDF Accessible | NA | Unpaywall OA | 1 | 0.44

## Conclusions

It is clear that publishing as OA is on the rise in recent years. This trend
is observed similarly in WOS and Scopus (while Scopus has higher raw
publication counts but trends are identical) and based on OA identification
stemming from both Unpaywall and Crossref. But still the majority of
publications are closed access. We observed that despite the high coverage of
Unpaywall (higher than 50% of _articles_ and _reviews_ in both WOS and
Scopus), it doesn’t provide enough metadata (as of April 2018) for OA
categorization thus could be limiting for large scale OA monitoring in the
leading bibliometric databases. Licence information from Crossref is more
detailed and it gives a good possibility to complement Unpaywall metadata.
Although we overcame the downsides by complementing these databases, we still
found further contradictions between them with manual random checks. Some
publications were OA (based on their licences or Unpaywall status) while their
PDF files were _not accessible_ through publishers’ websites. Some
publications were closed access, while their PDF files were _accessible_. We
found that the issue of multiple records for some publications or multiple
licence information is something that needs to be seriously considered in OA
monitoring. While we tried to test different scenarios in OA identification,
still there are publications that won’t fit into any of the scenarios and we
had to categorize them as _NA_ (since we wanted to keep the _Closed Access_
definition as strict as possible), these are the publications that need to be
further studied and usually the metadata of the OA databases are lacking for
them. We propose OA monitoring activities to try to benefit from our approach
in compelemting the metadata from OA databases, i.e. Unpaywall and Crossref,
while noting that there are contradictions between these sources. Our effort
to complement these databases proves that none of them could be used in
isolation.

## References

Archambault, E., Amyot, D., Deschamps, P., Nicol, A., Rebout, L., & Roberge,
G. (2013). Peer-reviewed papers at the european and world levels—2004-2011.
_Info@ Science_ , _1_ , 495–6505.

Crossref. (2018, October). Crossref. Retrieved from https://www.crossref.org/

CrossRef-API. (2019, October). CrossRef api. Retrieved from
https://github.com/CrossRef/rest-api-doc#filter-names

Jahn. (2017, January). About the hybrid oa dashboard. Retrieved from
https://subugoe.github.io/hybrid_oa_dashboard/about.html

Piwowar, H., Priem, J., Larivière, V., Alperin, J. P., Matthias, L.,
Norlander, B., … Haustein, S. (2018). The state of oa: A large-scale analysis
of the prevalence and impact of open access articles. _PeerJ_ , _6_ , e4375.

Rimmert, C., Bruns, A., Lenke, C., & Taubert, N. C. (2017). ISSN-matching of
gold oa journals (issn-gold-oa) 2.0.

Wohlgemuth, M., Rimmert, C., & Winterhager, M. (2016). ISSN-matching of gold
oa journals (issn-gold-oa).