# Land Use Detection & Identification using Geo-tagged Tweets

Saeed Khan  School of ITEE
University of Queensland
Brisbane, Australia
<EMAIL_ADDRESS>Md Shahzamal  Department of Computing
Macquarie University
Sydney, Australia
<EMAIL_ADDRESS>
###### Abstract

Geo-tagged tweets can potentially help with ‘sensing’ the interaction of
people with their surrounding environment. Based on this hypothesis, this
paper makes use of geo-tagged tweets in order to ascertain various land uses
with a broader goal to help with urban/city planning. The proposed method
utilises supervised learning to reveal spatial land use within cities with the
help of Twitter activity signatures. Specifically, the technique involves
using tweets from three cities of Australia namely Brisbane, Melbourne and
Sydney. Analytical results are checked against the zoning data provided by
respective city councils and a good match is observed between the predicted
land use and existing land zoning by the city councils. We show that geo-
tagged tweets contain features that can be useful for land use identification.

###### Index Terms:

land use, twitter, spatio-temporal, urban zoning

##  I Introduction

The goal of urban planning is to develop cities with the best possible
productive use of land, the ultimate purpose being improving peoples’ day to
day life and better management of resources. Generally, the urban planning
characterises residential, commercial or recreational use of the land. The
land classification or zoning determines how each segment of the land is to be
used. Zoning can be defined as the process of dividing land in a city council
(municipality) into various segments based upon its intended use. Thus, it may
enforce a variety of specific or conditional uses of land. It may also
indicate the size and dimensions of land area as well as the form and scale of
constructions. One of the issues associated with zoning is that it requires
actual assessment with regards to the use of land as originally planned. This
information needs to be collected on site, and is usually obtained through
survey or questionnaire to record how people interact with their land. This
approach has restrictions in the sense that it is very much dependent upon the
willingness of people to participate in such activity, involves costs, and,
essentially is just a snapshot in time. Technologies such as GIS [1] may
reveal some of the uses of land through image processing. However, since
images are not acquired very frequently, such techniques may not be very
productive.

As a result of using a large number of mobile devices, people leave behind
digital footprints of their interaction with the environment. Availability of
a vast number of mobile applications has turned the mobile devices into
sensors of human activity in space and time. This has led to the advent of new
research areas such as smart cities and urban computing with a goal to
understand the city dynamics through ubiquitous data, and thus improving the
quality of life. Data sources such as GPS, Bluetooth, Wifi and cell phone
records are becoming more and more useful for applications such as transport
planning [2], traffic estimation [3] and social studies [4]. This paper
suggests a method to use Twitter data to determine the type of land people
tweet from by analysing the temporal patterns of their tweeting behaviour. The
method is scalable and efficient compared to some other techniques in the
literature and can utilise any number of data points in order to capture the
best possible spatial features.

##  II Related Work

For urban planning, a number of different data sources have been used by
various researchers in their studies to model and characterise land use. The
data for these studies comes from either a GPS device, cell phone, or a
Location Based Social Network service. A quick review of work related to each
of these technologies is provided here.

#### GPS:

This data source has limited availability, is recorded every few seconds and
is usually based on a small number of subjects. It is not very widely used for
land classification purposes. The data set is usually acquired through buses
and taxis e.g. in [5], the authors conduct analysis of GPS tracks obtained
from taxis to infer the mobility of individuals and detect areas of various
activities using a pre-defined map of the places of interest.

#### Cell Phone Data:

Usually known as Call Detail Records (CDRs) contain information for a large
number of people, but is not easy to acquire because of privacy issues and
with location precision at the level of mobile tower only when some activity
is initiated. In [6], the authors use cell phone data to analyse the
relationship between cellular activity and commercial land use to identify the
prevailing pattern in each area. They qualitatively presented their results
and did not use any land use information. In another similar study [7], the
authors analysed the cell phone activity to characterise and cluster similar
locations.

#### Location Based Social Networks:

These offer precise location information in the form of latitude and longitude
when users enable the geo-location feature. This data set can be most up to
date and is easy to acquire as a result of ubiquitous use of various mobile
and social network applications by people. [8] model activity patterns in New
York City and London by using the geo-located information provided by a social
networking platform. They characterise the activity patterns by using a set of
pre-defined categories which indicate the type of check-in locations. Due to
this approach, their results provide a coarse idea of the land use. In another
work, [9] present a method to explore land use and user activities at a large
scale with the help of Foursquare platform. They validate the results with
personal interviews to confirm the identified land uses. In [10], the authors
group twitter users by city zone and various times of the day to study the
daily dynamics of the city. For a zone, they compare daytime activity with
night time to determine zones showing increased activity. Using multiple
regression, they determine how various land uses influence different time of
the day via changing coefficients. They observed that user activity seemed to
decrease throughout the day for land uses such as education, health and
offices, remained constant for parks, and increased for retail and residential
zones. This study aims to establish the link between twitter activity and the
known land use types and does not intend to determine the land uses from
twitter activity itself. In [11] and [12], the authors propose a method to
characterise geographic regions by focusing on day-to-day living patterns of
crowds using Twitter data. To grasp images of a city, the regularity of a
region is defined by using activity patterns such as the number of tweets,
their contents, the number of users and the movement of crowds. Moreover, the
changes in regularity patterns with respect to time are analysed and the
grouped urban types are categorised by tracking similar patterns across the
regions.

In this study, we also make use of publicly available geo-tagged tweets for
the identification of land use in a city. However, we purely use the temporal
patterns of raw twitter activity without using any textual information or user
profiles. Our method is simplistic, requires less processing and thus can be
applied on any volume of data. Specifically, we use temporal signatures of
Brisbane tweets as a template (training data) against which land use in other
cities is subsequently detected and predicted. We validate the results against
land use data provided by councils of test cities and observe a good match
between the predicted land use and actual land classification.

##  III Data Set & Experimental Setup

The data set has been collected using Twitter Streaming API [13] covering
Australia and spanning a period from January 2015 till July 2016. It consists
of 9462345 tweets from 245796 unique users. For Sydney we have 947964 geo-
tagged tweets and 40281 unique users, for Melbourne we have 854821 geo-tagged
tweets and 35556 unique users and for Brisbane we have 276394 geo-tagged
tweets and 14555 unique users in the data set. The proposed method involves
using the spatial location of tweets for above three cities of Australia.
Brisbane is chosen as the baseline city, and Melbourne and Sydney are used as
test cities on which the method is applied. For Brisbane, Central Business
District (CBD) is chosen as the reference around which various land use zones
are identified based on its city council’s scheme.111Schemes can be found
here. Although the council divides land into numerous types, key zones consist
of business, residential, education and recreation since these zones account
for the highest twitter activity compared to other zones.

Starting from the Brisbane CBD, corresponding temporal signatures are
extracted for each key zone. These signatures are later used as a template.
The hypothesis is that areas in different cities can be classified by their
similarity to these temporal signatures. To detect land use in our test
cities, CBDs are selected again and various activity clusters are generated
from the city-blocks until some meaningful temporal patterns are obtained.
Then, the closest matching temporal pattern from Brisbane is found for each
unknown cluster and the land use is thus predicted. This predicted cluster is
then compared to actual land use based on its city zoning scheme and the
overlap percentage is calculated for evaluation.

##  IV Temporal Signature Generation

Using Brisbane as baseline city, each of the key identified zone is analysed
for its twitter activity. Figure 1 shows the city council’s land zoning scheme
based on areas surrounding the CBD.

Figure 1: Brisbane City Zoning Map: land use is identified by a combination of
colour and number e.g. business (Blue 1), residential (Light pink 1, 2, 3 &
Maroon 1, 2), education (Yellow 5) and recreation (Yellow 2 & Grey 2).
Source: cityplan.brisbane.qld.gov.au/eplan/#/Property/0

Since each geo-tagged tweet contains its location information and time-stamp,
the temporal pattern for every hour of the day is extracted and plotted. This
temporal signature reflects the twitter activity for a given land zone and
spans the entire period of the data set. As a result, four temporal signatures
are generated for business (Blue 1), residential (Light pink 1, 2, 3 & Maroon
1, 2), education (Yellow 5) and recreation (Yellow 2 & Grey 2) land use,
respectively.

Figure 2 shows the temporal behaviour of each key land zone as the normalised
hourly tweets for the entire data set. Normalisation is the process of
adjusting values from different scales to some common scale in order to
provide a better comparison of the values. For example, the raw data for
Business cluster consists of very high values for each hour compared to other
clusters, and gives an impression that it is probably the most ‘active’
cluster. However, a better analysis would be to figure out the average
percentage of tweets posted in each hour of a day within each of the clusters.
Consequently, each hourly record value is divided by the overall average of
all hours to get the corresponding normalised value. In this way, the average
for all normalised values is 1 for each cluster.

$T_{h}^{c}=\frac{\\#T_{h}^{c}}{\sum_{h=0}^{23}\\#T_{h}^{c}}$

where $T_{h}^{c}$ is the percentage of tweets in hour $h\in\\{0,\dots,23\\}$
within cluster $c$, and $\\#T_{h}^{c}$ is the raw count of such tweets.

Figure 2: Brisbane Temporal Signatures

As shown in Figure 2, the business zone is characterised by a large tweeting
activity during day time. The first peak in the tweeting activity is observed
around 1 p.m. and the second peak is seen at 5 p.m. This might be related to
the times corresponding to activities such as going for a lunch or going home.
Spatially, this zone covers areas such as hotels, restaurants, shopping
centres and offices. Some bus and train stations are also included in it.
Hence, overall this zone represents ‘business’ activity in the city. The
residential zone shows increasing twitter activity in the afternoon with
maximum shown at 10 p.m. It covers areas comprising of private residences,
apartments and hostels. The education zone has its maximum peak at 10 a.m.
followed by another one at 12 p.m. After this, the trend in the twitter
activity gradually declines. This zone covers area including a school and a
TAFE in South Brisbane. The recreation zone has its distinctive peak at 7 p.m.
after which there is a sharp decline in twitter activity. Spatially it covers
areas including Suncorp Stadium and Brisbane Convention & Exhibition Centre.
We hypothesise that Twitter activities in other cities such as Melbourne and
Sydney have the same pattern for each of the four land uses. We test this
hypothesis through our proposed land use detection model.

##  V Land Use Detection

In this section, land use is detected for each test city: Melbourne and Sydney
by using the spatial location of tweets. The cluster building process is
manual and incremental as per following steps:

  1. 1.

Start from CBD, randomly pickup a city-block, get its latitude-longitude
bounding box, retrieve the data points contained inside and plot its temporal
curve. If the curve contains data points for every hour of the day, keep the
city-block as one of the clusters. Otherwise:

  2. 2.

Manually expand the cluster by adding adjoining areas by enlarging the
latitude-longitude bounding box. Check the temporal pattern again and expand
further if necessary until a full temporal curve showing data for every hour
of the day is obtained.

  3. 3.

Choose another city-block and generate clusters in above manner.

  4. 4.

Discard any cluster with incomplete temporal curve i.e missing data points for
certain hour(s) of the day.

This cluster generation process emphasises that there should be no zero
reading for any hour because this might indicate that such a cluster is either
just random or irregular in nature with not much activity taking place in it.
In comparison, a cluster which has activity for every hour of the day shows
that it is consistently and regularly used by people and this is important for
determining an activity or land use eventually. It should also be noted that
some of the clusters may eventually form a little far from the CBD as a result
of this process.

###  V-A Melbourne Clusters

For Melbourne, the cluster building process explained above is followed and
eventually four clusters are generated which are then subjected to further
processing and analysis. These clusters are shown in figure 3.

(a) Cluster 1

(b) Cluster 2

(c) Cluster 3

(d) Cluster 4

Figure 3: Melbourne Clusters. Cluster 1 ranges from -37.820565 to -37.806470
(latitude) and 144.955165 to 144.974617 (longitude). Cluster 2: -37.799383 to
-37.787955 (latitude) and 144.916789 to 144.935543 (longitude). Cluster 3:
-37.800277 to -37.792296 (latitude) and 144.957810 to 144.964049 (longitude).
Cluster 4: -37.824938 to -37.817277 (latitude) and 144.971911 to 144.988133
(longitude).

The temporal patterns for each of these unknown clusters are shown in figure 4
and these plots represent normalised data.

Figure 4: Melbourne Temporal Signatures

Each of these clusters is yet unknown in terms of the land use that it
actually represents. As a next step, these clusters are compared with
Brisbane’s known zones to judge their similarity. This similarity is evaluated
quantitatively by using the measure of mean-squared error. The mean-squared
error (MSE) function calculates the average of squares of errors, or in other
words, the average squared difference between the observed values (here a
temporal activity signature from Brisbane) and predicted values (here a
temporal activity signature from Melbourne). It is always non-negative and
values closer to zero are better as they indicate less error, with zero
signifying no error at all.

$\,\textrm{MSE}=\ \frac{1}{n}\sum_{i=1}^{n}(X_{i}-Y_{i})^{2}$ (1)

where n is the number of data points,$\ \textit{X}_{i}$ represents observed
values and$\ \textit{Y}_{i}$ represents predicted values. The similarity
between values can also be measured with other techniques such as Principal
Component Analysis (PCA) [14]. PCA is based on dimensionality reduction
concept and involves building vectors based on observed and predicted values.
This work, however, uses MSE technique because of its simplicity in
calculating errors. Each extracted cluster of Melbourne is checked against
every zone of Brisbane and the corresponding mean-squared error is calculated.
This error is calculated on the basis of each hourly normalised value for a
cluster and then the overall error is summed. Table I lists errors for each
pair of Melbourne and Brisbane cluster below:

| Brisbane Zones
---|---
| Business | Residential | Education | Recreation
Melb. Cluster 1 | 0.045 | 0.156 | 0.419 | 0.387
Melb. Cluster 2 | 0.206 | 0.204 | 0.611 | 0.607
Melb. Cluster 3 | 0.098 | 0.613 | 0.184 | 0.90
Melb. Cluster 4 | 0.239 | 0.190 | 0.834 | 0.117
TABLE I: MSE between Melbourne & Brisbane reference clusters

The minimum error for each pair of clusters is highlighted in bold and based
on this, the cluster for Melbourne is assigned. e.g. Cluster 1 has the least
error when compared against the Business zone of Brisbane. So the model has
predicted that Cluster 1 of Melbourne probably belongs to Business area of the
city. Similarly, Cluster 2 of Melbourne has least error against Residential
zone of Brisbane and so this cluster is predicted as Residential. In this way,
a land use prediction is made for all Melbourne clusters to be later validated
against actual land use based on its city council’s classification scheme. If
the predictions are mostly correct, it is indicative that we can use a city’s
twitter activity pattern as a reference to identify the land use in other
similar cities.

###  V-B Sydney Clusters

For Sydney, the clusters are built in a similar fashion and four clusters are
generated for further processing and analysis. These clusters are shown in
figure 5.

(a) Cluster 1

(b) Cluster 2

(c) Cluster 3

(d) Cluster 4

Figure 5: Sydney Clusters. Cluster 1 ranges from -33.868316 to -33.863256
(latitude) and 151.202142 to 151.213085 (longitude). Cluster 2: -33.890915 to
-33.888452 (latitude) and 151.196287 to 151.200373 (longitude). Cluster 3:
-33.891788 to -33.886575 (latitude) and 151.182370 to 151.190743 (longitude).
Cluster 4: -33.895203 to -33.888043 (latitude) and 151.220731 to 151.227083
(longitude).

The temporal patterns of each of these unknown clusters are shown in figure 6.
The plots represent normalised hourly twitter activity for the corresponding
cluster.

Figure 6: Sydney Temporal Signatures

These clusters are yet unknown in terms of the land use that they actually
represent. Like Melbourne, these clusters are compared with Brisbane’s known
zones to judge their mutual similarity. Each Sydney cluster is matched against
every zone of Brisbane and their corresponding mean-squared error is
calculated. Table II lists errors for each pair of Sydney and Brisbane
cluster. The minimum MSE error for each pair of clusters is highlighted and
based on this, the cluster for Sydney is assigned. Cluster 1 is predicted as
Business, Cluster 2 as Residential, Cluster 3 as Education, and Cluster 4 as
Recreation.

| Brisbane Zones
---|---
| Business | Residential | Education | Recreation
Sydney Cluster 1 | 0.030 | 0.238 | 0.267 | 0.536
Sydney Cluster 2 | 0.266 | 0.114 | 0.691 | 0.355
Sydney Cluster 3 | 0.398 | 1.204 | 0.198 | 1.613
Sydney Cluster 4 | 0.397 | 0.321 | 1.190 | 0.149
TABLE II: MSE between Sydney & Brisbane reference clusters

##  VI Land Use Validation

In this section, we validate the extracted clusters of Melbourne and Sydney
against actual land use and evaluate the overall methodology. To validate the
empirical results obtained so far, the actual land zoning information provided
by the city councils of Melbourne and Sydney is utilised. Comparison is made
against the type of land zone corresponding to the predicted cluster type. The
Melbourne city council divides land into commercial, recreation, residential
and mixed-use zones etc. as shown in figure 7. Similarly, the Sydney city
council divides land into commercial, mixed-use, residential and industrial
zones etc. as shown in figure 8.

Figure 7: Melbourne Zoning Map Figure 8: Sydney Zoning Map

In order to determine how closely the identified clusters signify the official
land uses, we calculate the degree of overlap between polygon shape of each
extracted cluster and the polygon of relevant underlying space in the official
zoning map of a city. Table III shows the percentages of overlap for various
types of clusters for both cities. We find that in general there is a high
similarity between the official land use types and the predicted land use
types from twitter.

Predicted Land Use Official Land Use Business Residential Education Recreation
Melbourne Business 52.4% - - - Residential - 56% - - Education - - 0% -
Recreation - - - 67.2% Sydney Business 62.5% - - - Residential - 50% - -
Education - - 53.5% - Recreation - - - 52.6%

TABLE III: Overlap between the land use predicted by Twitter activity and the
Official land use.

According to Table III, the business areas in the official zoning maps are
identified appropriately by our predicted business clusters with accuracy
ranging between 52% and 62%. Similarly, the percentage of similarity between
the residential land zone and our predicted residential cluster is between 50%
and 56%. The education cluster of Melbourne is predicted incorrectly and hence
its accuracy against actual education zone is 0%, whereas for Sydney it is
53.5%. The highest accuracy is observed for recreation zone ranging from 52%
to 67%.

Not all types of official land zones are necessarily predicted in all cities,
as is the case with Melbourne’s education cluster. This is because its
corresponding extracted cluster (Cluster 3) generated slightly higher error
with respect to it. In terms of calculating the overlap between extracted
cluster and its underlying actual land zone, the area of underlying zone plays
a key part. The cluster itself may have been predicted correctly for its
actual use, but its overlap percentage is influenced by the size of the
council’s land zone area. For instance, the predicted cluster may have a small
area owing to concentration of tweets in a limited space but its corresponding
actual land zone area may be much larger. This is possible for places like
parks where people might gather in a specific part of it and their resulting
cluster will be much smaller than the rest of the park, hence yielding smaller
overlap percentage. Another issue is the best number of clusters. Current
methodology relies on a limited number of clusters and determining the best
number of clusters may also be a fruitful area for further research.

Overall, it can be argued that the predicted clusters in both cities
predominantly match their respective actual land zones. Thus, the proposed
methodology in this study demonstrates that geo-tagged tweets show great
promise for land use identification, and this suggests profitable areas for
future research in how best to tune this method for maximum accuracy.
Although, it has been shown that key official land use types are reasonably
represented by our predicted clusters, this whole analysis is still
preliminary and needs further investigation. Moreover, it would also be worth
validating the empirical results against some survey data when available.

##  VII Conclusions

The availability of massive geo-data generated as a result of ubiquitous use
of various online and mobile applications has highlighted its prospective use
in various fields, one of which is urban development and planning. In this
study, a method has been proposed and demonstrated for identifying and
predicting land use using Twitter data from Brisbane and subsequently testing
on Melbourne and Sydney. The results indicate that geo-tagged tweets can
represent the land use in line with official land zoning to a good extent. The
percentage of overlap is found to be as high as 67.2% for Melbourne’s
recreation zone followed by 62.5% for Sydney’s business zone. The results of
this work are comparable to those of [9] and [15], but the cluster generation
method used here is simpler yet robust compared to those studies which use a
much slower clustering method requiring filtering out of many raw data points.
On the other hand, our method can work with virtually any volume of data and
without removing any records or losing underlying spatial features.

It should be noted that the precise nature of activities being undertaken by
twitter users is not determined as a result of our technique at this stage.

## References

  * [1] J. Yin, Z. Yin, H. Zhong, S. Xu, X. Hu, J. Wang, and J. Wu, “Monitoring urban expansion and land use/land cover changes of shanghai metropolitan area during the transitional economy (1979–2009) in china,” _Environmental monitoring and assessment_ , vol. 177, no. 1-4, pp. 609–621, 2011.
  * [2] V. Frias-Martinez, C. Soguero, and E. Frias-Martinez, “Estimation of urban commuting patterns using cellphone network data,” in _ACM SIGKDD International Workshop on Urban Computing_ , ser. UrbComp ’12. New York, NY, USA: ACM, 2012, pp. 9–16.
  * [3] N. Caceres, L. M. Romero, F. G. Benitez, and J. M. del Castillo, “Traffic flow estimation models using cellular phone data,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 13, no. 3, pp. 1430–1441, Sept 2012\.
  * [4] R. O. Oloritun, A. Madan, A. Pentland, and I. Khayal, “Identifying close friendships in a sensed social network,” _Procedia-Social and Behavioral Sciences_ , vol. 79, pp. 18–26, 2013.
  * [5] J. Yuan, Y. Zheng, and X. Xie, “Discovering regions of different functions in a city using human mobility and pois,” in _18th ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, 2012, pp. 186–194.
  * [6] F. Calabrese, M. Colonna, P. Lovisolo, D. Parata, and C. Ratti, “Real-time urban monitoring using cell phones: A case study in rome,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 12, no. 1, pp. 141–151, March 2011.
  * [7] V. Soto and E. Frias-Martinez, “Robust land use characterization of urban landscapes using cell phone data,” in _1st Workshop on Pervasive Urban Applications, in conjunction with 9th Int. Conf. Pervasive Computing_ , 2011.
  * [8] A. Noulas, S. Scellato, C. Mascolo, and M. Pontil, “Exploiting semantic annotations for clustering geographic areas and users in location-based social networks,” _The social mobile web_ , vol. 11, no. 2, 2011.
  * [9] J. Cranshaw, R. Schwartz, J. Hong, and N. Sadeh, “The livehoods project: Utilizing social media to understand the dynamics of a city,” in _Sixth International AAAI Conference on Weblogs and Social Media_ , 2012.
  * [10] J. C. García-Palomares, M. H. Salas-Olmedo, B. Moya-Gómez, A. Condeço-Melhorado, and J. Gutiérrez, “City dynamics through twitter: Relationships between land use and spatiotemporal demographics,” _Cities_ , vol. 72, pp. 310–319, 2018.
  * [11] R. Lee, S. Wakamiya, and K. Sumiya, “Urban area characterization based on crowd behavioral lifelogs over twitter,” _Personal and ubiquitous computing_ , vol. 17, no. 4, pp. 605–620, 2012.
  * [12] S. Wakamiya, R. Lee, and K. Sumiya, “Crowd-sourced urban life monitoring: urban area characterization based crowd behavioral patterns from twitter,” ser. ICUIMC ’12. ACM, 2012, pp. 1–9.
  * [13] Twitter. (2013, Jun.) Streaming apis. https://dev.twitter.com/streaming/overview.
  * [14] K. Pearson, “Liii. on lines and planes of closest fit to systems of points in space,” _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ , vol. 2, no. 11, pp. 559–572, 1901.
  * [15] V. Frias-Martinez and E. Frias-Martinez, “Spectral clustering for sensing urban land use using twitter activity,” _Engineering Applications of Artificial Intelligence_ , vol. 35, pp. 237–245, 2014.


# OpenChemIE: An Information Extraction Toolkit For Chemistry Literature

Vincent Fan  Computer Science and Artificial Intelligence Laboratory,
Massachusetts Institute of Technology, Cambridge, MA, 02139<EMAIL_ADDRESS>Yujie Qian  Computer Science and Artificial Intelligence Laboratory,
Massachusetts Institute of Technology, Cambridge, MA, 02139
<EMAIL_ADDRESS>Alex Wang  Computer Science and Artificial Intelligence
Laboratory, Massachusetts Institute of Technology, Cambridge, MA, 02139  Amber
Wang  Computer Science and Artificial Intelligence Laboratory, Massachusetts
Institute of Technology, Cambridge, MA, 02139  Connor W. Coley  Department of
Chemical Engineering, Massachusetts Institute of Technology, Cambridge, MA,
02139  Regina Barzilay<EMAIL_ADDRESS>Computer Science and Artificial
Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA,
02139

###### Abstract

Information extraction from chemistry literature is vital for constructing up-
to-date reaction databases for data-driven chemistry. Complete extraction
requires combining information across text, tables, and figures, whereas prior
work has mainly investigated extracting reactions from single modalities. In
this paper, we present OpenChemIE to address this complex challenge and enable
the extraction of reaction data at the document level. OpenChemIE approaches
the problem in two steps: extracting relevant information from individual
modalities and then integrating the results to obtain a final list of
reactions. For the first step, we employ specialized neural models that each
address a specific task for chemistry information extraction, such as parsing
molecules or reactions from text or figures. We then integrate the information
from these modules using chemistry-informed algorithms, allowing for the
extraction of fine-grained reaction data from reaction condition and substrate
scope investigations. Our machine learning models attain state-of-the-art
performance when evaluated individually, and we meticulously annotate a
challenging dataset of reaction schemes with R-groups to evaluate our pipeline
as a whole, achieving an F1 score of 69.5%. Additionally, the reaction
extraction results of OpenChemIE attain an accuracy score of 64.3% when
directly compared against the Reaxys chemical database. We provide OpenChemIE
freely to the public as an open-source package, as well as through a web
interface.

Equal Contribution Equal Contribution Equal Contribution Department of
Electrical Engineering and Computer Science, Massachusetts Institute of
Technology, Cambridge, MA, 02139 Department of Electrical Engineering and
Computer Science, Massachusetts Institute of Technology, Cambridge, MA, 02139

##  1 Introduction

Reaction data curated from scientific literature is commonly used to train
models for cheminformatics. Today, this data is collected and maintained by
experts in databases such as Reaxys 1. However, this manual extraction comes
with prohibitive cost and delayed updates. Moreover, increasingly nuanced
machine learning models for reaction development require more fine-grained and
comprehensive data, pertaining to reaction conditions, substrate scope, and
other screening processes in synthetic chemistry2, 3, 4, 5. Existing automated
techniques can only partially address this task, focusing on specific
subproblems, such as reaction parsing from individual diagrams or text
passages6, 7, 8, 9. In this paper, we present OpenChemIE, a system that
extracts reaction data from chemical literature at the document level.

Figure 1:  Example of a multimodal reaction description drawn from Zhao et
al.10. The reaction template is displayed in a figure, but information
regarding R-groups is only contained in the highlighted sections of the table.
Moreover, detailed reaction conditions are described in the table and
accompanying footnotes.

This extraction task is difficult because large swathes of reaction data are
realized in multiple modalities, often requiring chemical reasoning to fully
determine relevant molecular structures. Figure 1 illustrates two challenges.
First, the molecular structures are not entirely depicted in the figure, as
they contain R-groups. The abbreviated structures can be identified by
directly parsing phenol for R2 in entry 1 of the accompanying table, or by
comparing the differences in the molecular graphs of 1 and 1u in entry 21.
Second, the system must align additional reaction metadata with the correct
structures. In Figure 1 the highlighted molecule is only referred to by the
label 1 in the footnote text, which contains detailed reaction conditions. In
other cases, conditions may also be defined in a figure or table, or the
reaction itself may be described in text.

To address these challenges, OpenChemIE provides a streamlined computational
pipeline, which analyzes individual modalities and combines the extracted
information together to recover implicitly defined reactions. Building on our
prior research in reaction extraction7, molecular optical recognition11, and
reaction diagram parsing6, we design additional modules that enable OpenChemIE
to fuse information at three different levels. First, we train a machine
learning model to associate molecules depicted in diagrams with their text
labels, performing a multimodal coreference resolution. Second, OpenChemIE
aligns reactions with reaction conditions and other data presented in tables,
annotated in figures, or discussed in texts by utilizing the coreference
information. Lastly, OpenChemIE recognizes R-groups in a diagram by comparing
molecules with the same label, and substitutes them with additional
substructures listed in substrate scope tables and figures, yielding complete
substrate data.

We evaluate the performance of each individual machine learning module as well
as the system as a whole. To evaluate the overall performance of OpenChemIE,
we manually curated a dataset of 1007 reactions described in 78 substrate
scope figures involving R-groups across five different organic chemistry
journals. The extraction task requires all reaction components to be correctly
predicted and R-groups to be resolved. OpenChemIE achieves an F1 score of
69.5% on this dataset, and we performed a meticulous evaluation to analyze the
error contributions of the different modules of the pipeline, identifying
areas where the system could be further enhanced in future work. Furthermore,
in an end-to-end evaluation of OpenChemIE on extracting reaction data from
journal articles, we attain an accuracy of 64.3% when comparing against
existing extractions in Reaxys. Notably, our models for reaction diagram
parsing, molecule detection, and coreference resolution all perform robustly
under evaluation on independent benchmarks, and our R-group resolution
algorithm only contributes to a small amount of mistakes. The majority of
errors were due to mistakes in molecule recognition or optical character
recognition.

OpenChemIE is available on a public web portal (https://mit.openchemie.info)
as an easily accessible demonstration of key forms of analysis that we
incorporate. The full pipeline and its individual methods for analysis are
provided in a Python package (https://github.com/CrystalEye42/OpenChemIE) that
is suitable for larger-scale information extraction. Our Python package allows
for comprehensive extractions of molecules and reactions from PDF files, as
well as from only text and images. The toolkit is fully open-source to
facilitate future development in this area.

##  2 Related Work

### Extracting From Figures

This task includes molecule recognition and reaction extraction. Molecule
recognition involves translating molecular images into SMILES strings. Initial
approaches employed rule-based methods for determining the structures of
molecules, utilizing a suite of algorithms and heuristics to detect individual
components such as bonds and atoms 12, 13. Later works instead leveraged CNN-
based encoder-decoder architectures from deep learning to perform this
segmentation, allowing for robust recognition across diverse styles 14, 15,
16, 17, 18, 11. Recent research has also enabled the extraction of reaction
schemes from figures, in which the reactants, products, conditions, and yield
for each reaction are identified. These works approached the segmentation
either by using a series of heuristics and image filters19 or by applying
data-driven models for object detection and sequence generation6, 20.

Several works have additionally developed systems for automatic extraction
from figures of real-world documents. These include ReactionDataExtractor 19
and its 2.0 version 20, which involve parsing reaction schemes, and also
include MolMiner 21 and DECIMER.ai 22, which focus on molecule recognition.
ChemSchematicResolver8 is another molecule recognition system that
additionally resolves R-groups defined in text labels. However, achieving a
robust figure-based extraction system remains challenging due to the wide
variation of styles and possible complexities for molecules and reaction
schemes.

### Extracting From Text

This task consists of identifying chemical entities and their roles, as well
as parsing described reactions. Several studies for the former have centered
on dataset curation23, 24. For both chemical entity identification and
reaction extraction, proposed solutions include parsers that employ a series
of regular expressions and classifiers to detect key terms25, 26. A deep-
learning solution for extracting reactions instead formulates the problem as a
sequence labeling task and utilizes a fine-tuned transformer encoder
architecture.

A few works have created systems that extract from PDF files of documents
instead of from plain text, presenting additional engineering challenges.
These include ChemDataExtractor9 and PDFDataExtractor27, which identify
chemical entities and their associated properties. While these can extract
important information available in text, they do not process relevant
information from figures that would augment the data. Text-based descriptions
of reactions and molecules are often underspecified, generally referring to
these entities using families of compounds or by labels defined in figures.
Resolving these mentions to obtain specific molecular structures is vital. In
contrast to both these and the figure-based extraction systems, OpenChemIE
aims for a more versatile, unified system. The advantage of OpenChemIE is our
usage of specialized chemistry-informed algorithms to integrate extractions
from multiple modalities, namely text, tables, and figures, thus overcoming
the single modality barrier and enabling more comprehensive extractions.

##  3 Problem Formulation

Figure 2:  OpenChemIE addresses the problem of extracting a list of reactions,
containing chemical structures for reactants and products, as well as reaction
conditions from a PDF document segmented into figures, text, and tables.

As seen in Figure 2, OpenChemIE addresses the task of extracting detailed
chemical reactions at the document level. We consider the input journal
article to be a triple of (Figures, Texts, Tables), which can be automatically
segmented with existing PDF parsing tools. We seek to extract the reactions
described in the paper by identifying machine-readable structures for their
reactants and products, as well as other metadata. The expected output is a
list of reactions $\\{\mathbf{R_{1}},\mathbf{R_{2}},\dots,\mathbf{R_{n}}\\}$,
where each reaction $\mathbf{R_{i}}$ is a triple $(R_{i},P_{i},C_{i})$.
$R_{i}$ is the set of reactants and $P_{i}$ is the set of products, each
consisting of one or more molecules expressed as SMILES strings. $C_{i}$ is
the set of metadata associated with the reaction, including detailed
conditions and yield information, and may be empty if no such information is
parsed from the paper. We do not capture information contained in other plots,
such as reaction coordinate diagrams or spectral data.

The information extraction task is thus expressed as a function

$f:(\text{Figures, Texts,
Tables})\rightarrow\\{\mathbf{R}_{1},\mathbf{R}_{2},\dots,\mathbf{R}_{n}\\}$
(1)

Crucially, OpenChemIE establishes relationships between its three inputs to
inform its output, such that $f(\text{Figures, Text, Tables})$ contains more
data than $f(\text{Figures})\cup f(\text{Texts})\cup f(\text{Tables})$, the
result of individually extracting from each modality.

##  4 OpenChemIE Overview

Figure 3: Overview of OpenChemIE, which receives segmented figures, texts,and
tables for processing. The results from individual neural models for each
modality are combined through reaction condition alignment and R-group
resolution in OpenChemIE to yield a final list of reactions.

In the following section, we present an overview of OpenChemIE, a dedicated
toolkit designed to extract full reaction data from chemistry papers. A
summary of our system can be found in Figure 3. Initially, OpenChemIE receives
the document which has been segmented into figures, text, and tables for use
in the downstream steps of our pipeline (implementation details are provided
in the Supporting Information). For each modality, we have developed
specialized machine learning models capable of effectively parsing molecules
and reactions, as well as inferring relationships between text and diagrams.
To further expand the scope of information captured by OpenChemIE, we
implement two general procedures that fuse the outputs of individual models to
produce a more complete reaction list. A description of the individual
components of OpenChemIE follows.

  * •

Figure Analysis. Analysis of chemistry figures requires strong visual
understanding, ranging from high-level comprehension of reaction schemes and
relations between entities to low-level recognition of molecules. To address
this multifaceted challenge, we provide four models for figure/scheme
analysis. The first of these models is designed for detecting sub-images of
molecules within figures (molecule detection, MolDetect) and providing the
relevant bounding box. Another model is for resolving the coreference between
detected molecules in the figure and labels in the text (text-figure
coreference, MolCoref). Additionally, OpenChemIE utilizes our previous
research for parsing reaction schemes and relevant condition information
(reaction diagram parsing, RxnScribe) 6 and translating molecular images into
their chemical structures (molecule recognition, MolScribe) 11.

  * •

Text Analysis. Extracting from chemistry texts involves identifying mentions
of molecules and chemical reactions. To this end, we provide two models to
address both of these subtasks. The first one is a model for extracting
chemical entities from texts (named entity recognition, ChemNER). The second
model, from our previous research, identifies chemical reactions and their
reaction conditions (reaction extraction, ChemRxnExtractor)7.

  * •

Multimodal Integration. We implement two additional procedures that integrate
information across our single modality models. R-group resolution is the
process of identifying and substituting R-group structures into reaction
templates, which allows for a more complete extraction of reactions. This
process utilizes parsed data from tables as well as molecule information from
figure analysis models. Reaction condition alignment enhances text-based
reaction descriptions with molecular structures identified in relevant
diagrams. These multimodal integration components enable OpenChemIE to serve
as an end-to-end pipeline for reaction extraction.

In the subsequent sections, we will elaborate on the technical details of each
module comprising OpenChemIE.

##  5 Figure Analysis

Figures in chemistry literature contain essential molecular structures and
reactions, as well as relational information to text in or surrounding the
diagram. In particular, the new methods we develop focus on this latter
aspect, which enable the success of our downstream multimodal integration
modules. As in Figure 4, OpenChemIE addresses four facets of figure analysis,
which we detail in this section.

Figure 4: OpenChemIE provides four models for analyzing figures in chemistry
literature, including molecule detection, text-figure coreference, reaction
diagram parsing, and molecule recognition.

### Molecule Detection

In chemistry literature, a single figure often contains multiple molecules.
The task of molecule detection is to segment the figure into sub-images of
molecules such that we can later recognize the structure of each individual
molecule. Molecule detection shares similarities with the extensively studied
object detection task in computer vision 28, 29, which focuses on identifying
sub-images of objects within natural photographs.

In OpenChemIE, we provide MolDetect, a molecule detection model formulated
with sequence generation. Inspired by the Pix2Seq 29 model designed for object
detection, MolDetect identifies molecular sub-images by predicting their
bounding boxes as a sequence. Given a figure, a molecule entity whose bounding
box has top-left coordinates $(\mathrm{x_{1}},\mathrm{y_{1}})$ and bottom-
right coordinates $(\mathrm{x_{2}},\mathrm{y_{2}})$ is represented as five
discrete tokens,

$\text{Molecule}:=\ \mathrm{x_{1}}\ \mathrm{y_{1}}\ \mathrm{x_{2}}\
\mathrm{y_{2}}\ \texttt{[Mol]}$ (2)

where [Mol] is a special token indicating the detection of a molecule.
MolDetect sequentially generates all the molecule entities within the figure,

$\text{MolDetectOutput}:=(\text{Molecule})^{*}$ (3)

where $(\cdot)^{*}$ means zero or more occurrences.

MolDetect is implemented as an encoder-decoder architecture. The figure is
encoded using a convolutional neural network to obtain hidden representations.
Then, the decoder is a Transformer which generates the output sequence as
defined in Equations 2 and 3.

### Text-Figure Coreference

Figure 5: Illustration of extracting molecules and identifier coreferences
from a figure. First, MolCoref determines entity bounding boxes and their
correspondence. Then, the molecules are recognized by MolScribe.

It is common practice in chemistry literature to assign unique identifiers to
the molecules depicted in a figure and subsequently refer to them by their
respective identifiers in the accompanying text. To establish a clear link
between the information in the text and the figure, we have developed
MolCoref, a model that pairs molecules with their respective identifiers in
the figure. An example prediction from MolCoref can be found in Figure 5,
which depicts the molecules and identifiers being successfully detected and
the corresponding links established.

MolCoref also employs a sequence generation approach to resolve the
coreference between identifiers and molecular structures. Specifically, its
output format is defined as

$\begin{split}&\text{MolCorefOutput}:=\ (\text{Molecule}\
[\text{Identifier}]?)^{*}\\\ &\text{Molecule}:=\ \mathrm{x_{1}}\
\mathrm{y_{1}}\ \mathrm{x_{2}}\ \mathrm{y_{2}}\ \texttt{[Mol]}\\\
&\text{Identifier}:=\ \mathrm{x_{1}}\ \mathrm{y_{1}}\ \mathrm{x_{2}}\
\mathrm{y_{2}}\ \texttt{[Idt]}\end{split}$ (4)

where $[\cdot]?$ means optional. Both the Molecule and Identifier are
represented using five tokens, consisting of four coordinates and a final
token to differentiate them. When a Molecule is paired with an Identifier in
the figure, the model generates the Molecule first, followed by the
corresponding Identifier. Otherwise, the model generates only the Molecule
without the Identifier. Based on MolCoref’s output, we use a molecular
recognition model to parse the chemical structures and an optical character
recognition (OCR) model to parse the text strings.

Compared to the previous approach 8 that relies on heuristic rules for
aligning molecules with their identifiers, MolCoref integrates the detection
of molecule bounding boxes and the resolution of text-figure coreference into
a single model. This simplifies the process and mitigates the risk of error
propagation. Furthermore, as our experiments will demonstrate, our data-driven
model yields more accurate and reliable predictions.

### Reaction Diagram Parsing

Figure 6: Illustration of extracting reactions from a figure. First, RxnScribe
parses the two reactions in the figure. Then, the molecules and text are
recognized by MolScribe and OCR models respectively.

Reaction schemes are often defined graphically within figures and in a wide
range of styles, requiring a sophisticated level of visual understanding to
correctly extract. To this end, OpenChemIE incorporates RxnScribe 6, a model
previously designed for the extraction of reaction schemes from figures.
Figure 6 demonstrates the extraction process on a figure with two reactions.
In it, RxnScribe predicts the structure of both reactions correctly, with
reactants, conditions, and products highlighted in red, green, and blue boxes,
respectively.

### Molecule Recognition

Molecule recognition is the task of translating an image of a molecule into
its corresponding chemical structure, typically represented as a SMILES string
in a computer-readable format. OpenChemIE includes MolScribe 11, a model we
developed earlier for molecule recognition. Our previous modules for text-
figure coreference and reaction diagram parsing only extract the high-level
structure of diagrams. To fully extract the reaction or molecular information,
we further crop the bounding boxes and pass the individual subdiagrams to
downstream modules. As shown in Figure 5, we use MolScribe and EasyOCR30, an
off-the-shelf optical character recognition tool, to translate the content in
each bounding box to paired SMILES strings and text labels. Similarly, in
Figure 6, we use the same tools to extract the molecular structures of
products and reactants, as well as text descriptions of accompanying reaction
conditions.

##  6 Text Analysis

Figure 7: Illustration of extracting chemical entities and reactions from
text. The passages are drawn from Liu et al..31

As seen in Figure 7, OpenChemIE contains two powerful natural language
processing models which excel at extracting chemical entities and reactions
from the text in chemistry literature.able to extract chemical entities and
reactions from text. The machine learning models are named ChemNER and
ChenRxnExtractor respectively.

### Named Entity Recognition

Our first model is dedicated to the task of extracting chemical entities from
a given text excerpt. In scientific literature, chemical entities can take on
diverse forms, including molecular formulas (e.g., NaOH), IUPAC systematic
names (e.g., 1,3,4-oxadiazole), abbreviations (e.g., GABA), or database
identifiers (e.g., CID16020046). For successful extraction, it is essential to
locate these mentions in the text and accurately determine their specific
forms.

In OpenChemIE, we provide a model named ChemNER, which is trained on the
publicly available CHEMDNER corpus 23. This corpus comprises a collection of
PubMed abstracts with expert-annotated chemical entity mentions. Our model
adopts a sequence tagging approach using the BIO format. We fine-tune a
language model that has been pre-trained on biomedical literature,32 further
enhancing the model’s performance and domain-specific understanding. In Figure
7, ChemNER detects all three chemical entities and correctly makes the
distinction that “cyanamide” refers to a specific compound whereas the other
chemical mentions refer to general families of compounds.

### Reaction Extraction

Reaction extraction is a structured prediction task that involves identifying
the reactions presented in the text. OpenChemIE includes ChemRxnExtractor 7, a
model previously developed for text-based reaction extraction. Figure 7
displays a processed reaction description where the product is represented by
the identifier “4a”, with additional information about reaction conditions and
yield also highlighted. However, the chemical structures of the extracted
reactant “1a” and product “4a” are omitted in the text, highlighting the
importance of our models for coreference resolution and molecule recognition
in diagrams.

##  7 Multimodal Integration

Complete reaction schemes, which require full structural information of
reactants and products as well as complex descriptions of reaction conditions,
are often specified across multiple paragraphs, tables, and figures.
Understanding the connections between these modalities is challenging, as
demonstrated in Figure 1, and has not been significantly explored by previous
works. OpenChemIE begins to address the general task of multimodal integration
by dividing the problem into two main challenges. For one, detailed reaction
condition and yield data must be properly aligned with machine readable
molecular structures of the reactions they refer to. Furthermore, many
diagrams are underspecified, and the R-groups they contain must be inferred
from a separate molecule in the diagram or a completely different table
altogether. In the following sections, we describe how we integrate our
individual model results together for multimodal understanding.

### Reaction Condition Alignment

Figure 8: Reaction Condition Alignment. We augment incomplete reaction
descriptions in text with resolved molecule identifier pairs and parse
additional reaction condition tables. Example adapted from Liu et al..31

In OpenChemIE, we provide methods to align the reaction data contained in
figures with the information from text and tables, obtaining more complete
reaction descriptions.

One type of reaction condition alignment we address is the task of integrating
information from condition screening tables with their corresponding reactions
displayed in figures, such as in Figure 8. For this, we created a parser to
extract the table headers and columns. We use a dictionary-based classifier to
categorize each column based on its header, such as being for temperature,
solvent, yield, or other common types of metadata. Each row in the table
corresponds to a complete configuration of reaction conditions, which we add
to the set of reaction conditions for the relevant reaction.

Reactions and their details are often described within the accompanying text
as well. However, the reactants and products in this modality are often
distinguished by their unique identifier, with the structural information of
the molecules defined separately in figures. With only the identifier, these
text-based reactions would be incomplete due to the missing of molecular
structures. To address this issue, we align the molecular structure
information from figures with their identifiers in text. From our figure
analysis module, we first obtain a mapping between the identifiers and their
structures using MolCoref. Whenever an identifier is encountered during the
text-based reaction extraction stage, we substitute the identifiers with their
SMILES representation. This integration along with our table-figure
integration allows for the unification of information across three modalities
to extract significantly more complete reaction data.

### R-Group Resolution

Figure 9: R-Group Resolution. R-groups are first identified from diagrams and
tables and then substituted into the appropriate reactant molecule templates.
Example adapted from Zhang et al.33

Previous work from ChemSchematicResolver 8 has been done to parse simple
definitions of R-groups that are explicitly expressed as text chemical
formulae within figures (e.g., “R=Me”) and perform the corresponding
substitutions. In addition to this case, OpenChemIE seeks to comprehend other
forms of substrate scope, namely the cases where products are depicted as
different molecular structures or where a table separate from the reaction
scheme defines the R-groups, which require further reasoning to determine the
resolved molecular structures.

We address the two most common modes of presentation for substrate scope,
which are shown in Figure 9. In the first case, the reaction template is
displayed graphically and the R-groups are defined as text in an associated
table or label. For this, we parse the R-group information from the text
formula and use MolScribe11 to predict the graph structures of the template
molecules. We then directly substitute the chemical formulas of the R-groups
into their placeholders in the graphs. These structures are then expanded and
converted to SMILES strings by MolScribe’s postprocessing methods.

In the second case, in addition to a reaction template, there is a set of
possible products defined in the diagram with which one must infer the
structures of the R-groups and the reactants. To approach this, we first
leverage MolCoref to identify the labels of all molecules in the figure and
match the label prefixes to associate the specific products with their
template molecule. Given the reaction template and the specific product, we
use a subgraph isomorphism algorithm implemented in RDKit 34 to identify the
atom mapping between the two molecular structures. The unmapped atoms in the
specific product are molecular fragments that correspond to the substructures
of the R-groups. We substitute the identified R-group fragments into the
reactant templates in order to obtain the full molecular structures of the
entire reaction.

With the two integration steps presented above, we obtain a reaction list with
complete molecule structures and conditions. In the experiments, we evaluate
the performance of the pipeline.

##  8 Experiments

One of the central challenges in developing a reaction extractor is a lack of
high-quality benchmark datasets with corresponding evaluation metrics. To this
end, we created our own dataset consisting of reactions and diagrams from
chemistry literature, with manually produced annotations. In addition, we
compared the system output against reactions in the Reaxys database. While our
annotation scheme and extraction scope are not fully aligned with that of
Reaxys (e.g., Reaxys does not include reactions with low or no yield), this
challenging evaluation provides another measure of the system performance. We
conduct a meticulous error analysis for both evaluation settings and further
discuss the performance of individual modules in OpenChemIE.

### Evaluation With Annotated Data

We evaluate OpenChemIE on a newly annotated reaction extraction dataset. This
dataset contains 1007 reactions collected from 78 figures from recent issues
of five chemistry journals: Journal of Organic Chemistry, Organic Letters,
Angewandte Chemie International Edition, European Journal of Organic
Chemistry, and Asian Journal of Organic Chemistry. The figures in this dataset
are substrate scope diagrams. Using ChemDraw, we annotated SMILES strings for
every reaction by inferring the structure of reactants from the structures of
the template product and table of full products. A set of example annotations
for this dataset is displayed in Figure 10. For each substrate scope diagram,
we first annotate the reaction template $(R,P)$, where $R$ and $P$ are the
sets of SMILES strings for the reactants and products respectively, which may
contain R-groups. Then, we annotate the substrate scope $\\{(R_{i},P_{i})\\}$,
where $R_{i}$ is a set of reactants whose R-groups have been substituted based
on $P_{i}$.

Table 1: Performance of OpenChemIE for extracting reactions from substrate
scope diagrams, as well as the individual performance of each module in
OpenChemIE.

Module | Evaluation Score*
---|---
OpenChemIE | 79.1 / 62.0 / 69.5
Evaluation of individual models |
\- Molecule Detection (MolDetect) | 86.0
\- Coreference Resolution (MolCoref) | 91.4 / 88.9 / 90.1
\- Reaction Diagram Parsing (RxnScribe) | 91.9 / 90.1 / 91.0
\- Molecule Recognition (MolScribe) | 71.9
\- Named Entity Recognition (ChemNER) | 87.1 / 88.1 / 87.6
\- Reaction Extraction (ChemRxnExtractor) | 79.3 / 78.1 / 78.7

  * *

Precision/Recall/F1 by default. For molecule detection, we use Average
Precision 35. For molecular recognition, we use accuracy.

Figure 10: Illustration of annotation process, where we parse the SMILES
strings of the template reaction $(R,P)$ and provide each detailed reaction
$(R_{i},P_{i})$.

For this dataset, we evaluate the model’s predictions using exact match, i.e.,
a predicted reaction $(\hat{R},\hat{P})$ is considered correct only if all the
molecular structures of its reactants and products match those in a ground
truth reaction. We compute the precision, recall, and F1 to assess the model’s
performance. Here, the precision measures what fraction of the model’s
predictions is correct, the recall measures what fraction of the ground truth
reactions is correctly predicted, and F1 score is the harmonic mean of
precision and recall. As seen in Table 1, OpenChemIE achieves a precision of
79.1%, recall of 62.0% and F1 score of 69.5 % on this task.

Each individual module in OpenChemIE is also evaluated on an independent
benchmark to measure its performance in isolation of the entire system. Table
1 shows state-of-the-art performances of all six individual machine learning
models on their respective benchmarks. In particular, MolCoref achieves an
90.1% F1 score on a dataset of 1696 diagrams meticulously annotated with
molecule-identifier information. As detailed in our past research, RxnScribe
achieves a strong 91.0% F1 score on identifying single line reaction diagrams,
and MolScribe has an accuracy of 71.9% on realistic molecular structures drawn
from past ACS publications. We discuss additional evaluation details and error
contribution rates to the next section.

### Evaluation With Reaxys

| Correct | Total Predictions | Accuracy
---|---|---|---
OpenChemIE | 257 | 400 | 64.3%
ReactionDataExtractor 2.0 | 9 | 102 | 8.8%
Table 2: Reaction extraction results on journal articles compared against
Reaxys.

We evaluate the performance of OpenChemIE by comparing the extractions against
those in Reaxys. Reaxys is a large commercial database of reactions that is
periodically updated by chemical experts who manually extract the data from
journal articles.

We construct the dataset for this task by collecting 19 journal articles
containing 155 figures from recent issues of The Journal of Organic Chemistry
and Organic Letters that contained reaction condition and substrate scope
screening tables. These journal articles were each converted into a triple of
figures, texts, and tables for input to OpenChemIE with a set of off-the-shelf
PDF-parsing tools corresponding to each modality36, 27, 37. Due to errors in
diagram parsing frequently yielding inaccurate borders, we manually adjusted
diagram segmentations for this dataset. Existing reaction extractions in
Reaxys provided the groundtruth annotations. For each article, the groundtruth
thus contains a list $\\{(R_{i},P_{i})\\}$ where $R_{i}$ and $P_{i}$ are sets
of SMILES strings for the reactants and products respectively.

We use a soft match to evaluate the accuracy of our pipeline’s predictions.
First, molecular structures are considered to be equivalent if they are
tautomers to each other since some compounds rearrange to specific isomers in
solution. Second, a predicted reaction $(\hat{R},\hat{P})$ is considered
correct if Reaxys contains an entry $(R,P)$ such that $\hat{R}$ is a subset of
$R$ and $\hat{P}$ is a subset of $P$. We choose to use this evaluation metric
because of ambiguities that arise during the annotation process, for example,
whether certain compounds are considered reactants or reagents specified in
the set of conditions instead.

As seen in table 2 OpenChemIE extracts 400 reactions from this dataset, of
which 257 have a soft match in the Reaxys database, for an accuracy of 64.3%.
Since ReactionDataExtractor 2.020 does not extract from texts or tables, we
only provide the segmented diagrams for each journal article to
ReactionDataExtractor 2.0. It achieves an accuracy of 8.8% with 102 total
predictions in this evaluation setting. Besides reactions described in texts
or tables, ReactionDataExtractor was also unable to resolve reactions whose
depictions involved R-groups, which comprised the majority of reactions
extracted by OpenChemIE. Moreover, we applied the same evaluation to the fully
automatic version of OpenChemIE provided through our code package and web
portal. In this setting, OpenChemIE achieved an accuracy of 46.0% on 359 total
predictions. The decrease in accuracy can mainly be attributed to inaccurate
diagram segmentations during the PDF parsing process, since the automatic tool
LayoutParser was not trained specifically on chemistry literature. Additional
implementation details can be found in the Supporting Information.

### Analysis and Discussion

Figure 11: Error contribution of each relevant module to the R-group
Resolution process Figure 12: Examples of predictions and common errors of
OpenChemIE on substrate scope diagrams.

We analyze the error contribution of individual components in OpenChemIE.
Since the majority of reactions extracted in the previous evaluations are
described in substrate scope diagrams, we focus our analysis towards the
performance on the task of extracting from this setting. Figure 11 displays
the error contribution of each module in the R-group resolution process. The
full evaluation scores of each model are displayed in Table 1.

Figure 12 illustrates examples of successful predictions and common errors by
OpenChemIE. In Example (1), OpenChemIE is able to correctly identify the
reaction template, determine the molecular bounding boxes, and resolve all of
the coreferences correctly. The overall pipeline is able to extract all four
reactions depicted in the substrate scope diagram correctly, which can be
attributed to the strong performance of MolCoref and RxnScribe in the initial
stage. We provide evaluation results of MolDetect and MolCoref in Table 3 and
also compare the models against ChemSchematicResolver 8. MolDetect and
MolCoref leverage the simple sequential learning framework to achieve strong
performance in both tasks, whereas errors propagate throughout
ChemSchematicResolver’s rule-based pipeline. Per Figure 11, MolCoref
ultimately caused 15% of incorrect predictions. This outsized error
contribution was primarily due to diagrams in which one molecule had multiple
labels, a presentation style not seen in the training dataset. On the other
hand, RxnScribe achieves an F1 score of 91% for parsing single line reaction
diagrams, which make up the majority of reaction templates in substrate scope
diagrams, and contributed to 0% of overall errors. Our prior work6 provides a
more detailed quantitative evaluation of RxnScribe on extracting reactions
from diagrams of various styles.

| Detection | Coreference
---|---|---
| Average Precision | Precision | Recall | F1
ChemSchematicResolver | 28.8 | 83.8 | 31.7 | 46.0
MolDetect | $\mathbf{86.0}$ | – | – | –
MolCoref | $82.9$ | $\mathbf{91.4}$ | $\mathbf{88.9}$ | $\mathbf{90.1}$
Table 3: Evaluation of chemical diagram entity detection and coreference
performance (scores are in %).

Our R-group resolution algorithm also performs robustly and is generally able
to correctly identify the R-groups from each product and perform the
corresponding substitutions in the reactant template when the input is free of
errors. However, there were a small number of cases where the algorithm
returned an incorrect prediction. For example, some product templates are
completely symmetric but contain two different R-groups. Since the R-group
resolution algorithm does not take into account information about the layout
or color of the original diagram, it is unable to differentiate between the
two correctly extracted R-group fragments. In other diagrams where errors
occurred, specific presentation choices violated assumptions made in the
design of the algorithm. Some authors switched the chirality of certain atoms
between the template and product, and others included products where not every
R-group in the original template had a substituent. A more detailed discussion
of specific errors can be found in the Supporting Information.

In contrast, over half of the errors in the OpenChemIE pipeline occurred
during molecule recognition. In Figure 12, example (2) displays a MolScribe
error occurring on a MolCoref prediction, where a molecule with label 3u is
parsed incorrectly. Example (3) displays an instance where there is a
MolScribe error in the original reaction template. From this, we observe that
there are two reasons for MolScribe’s outsized error contribution. First, if
there is a single MolScribe error in the original reaction template, the
extraction results for the entire diagram will be incorrect. This scenario
contributed 41.6% of all errors. Second, MolScribe only achieves a 71.9%
accuracy on molecules from ACS publications, which are often drawn in diverse
styles 11. Furthermore, the tool we employed for optical character recognition
of molecule labels, EasyOCR, was another large source of error. Many labels
were parsed incorrectly. In example (2), the label “3g” was mistakenly parsed
as “39”, which meant that the product was not processed by the downstream
algorithm, as it was not associated with the product with label “3”.

| Percentage | Precision | Recall | F1
---|---|---|---|---
ABBREVIATION | 16.5% | 85.9 | 84.5 | 85.2
FAMILY | 13.0% | 76.5 | 85.4 | 80.7
FORMULA | 14.0% | 86.3 | 84.8 | 85.6
IDENTIFIER | 2.4% | 92.2 | 78.1 | 84.5
MULTIPLE | 0.7% | 68.7 | 74.4 | 71.5
SYSTEMATIC | 22.9% | 89.8 | 88.6 | 89.2
TRIVIAL | 30.5% | 91.5 | 93.5 | 92.5
Overall | 100% | 87.1 | 88.1 | 87.6
Table 4: Evaluation of chemical entity named entity recognition by entity type
(scores are in %).

We further analyze the two text-based extraction models in OpenChemIE, namely
ChemNER and ChemRxnExtractor. ChemNER is trained based on the BioBERT-large 32
checkpoint using the CHEMDNER dataset.23 The dataset annotates mentions of
chemical entities in seven types. Whereas past work does not make distinctions
based on entity type during the evaluation, we evaluate the entity-level
precision, recall and F1 scores on each type and also report the micro-
averaged overall performance in Table 4. The model achieves an F1 score of
87.57 % on the entire test set, with stronger performance on the two most
common classes, SYSTEMATIC and TRIVIAL. Complete evaluation results for
ChemRxnExtractor can be found in previous work 7.

##  9 OpenChemIE Interfaces

OpenChemIE is accessible through two interfaces: (1) a comprehensive Python
package that integrates all our models and utility functions, and (2) a user-
friendly Web portal that simplifies the toolkit’s usage, making it accessible
to a wider audience, even those without programming knowledge.

### Python Package

We provide an open-source Python package
(https://github.com/CrystalEye42/OpenChemIE) that integrates all our models
and utility functions, including the PDF parser and the models for text,
figure, and table analysis. We further implement methods that take a PDF
document as input and effortlessly execute the information extraction
pipeline, returning the extracted molecule and reaction data in a structured
format. To ensure smooth usage, we provide detailed installation instructions
and example use cases, enabling chemists with basic programming skills to
efficiently process literature data using the toolkit.

### Web Portal

We have developed a user-friendly web portal (https://mit.openchemie.info)
that streamlines the PDF upload process, automatically executes extraction
models, and conveniently displays the results. Users can upload a PDF document
of a chemistry paper, which will be processed on our backend server using the
OpenChemIE toolkit. The extraction results will be visualized on the portal.
As in Figure 13, predicted molecule structures are displayed in a web-based
Ketcher editor, enabling the user to edit the model’s predictions if desired.
Due to computational constraints, our public portal can process a maximum of
five pages from each paper. However, users can freely download and deploy the
web portal on their own machines, granting access to all functionalities.

Figure 13: Illustration of the web interface for extracting molecular
structures from PDF files. The uploaded PDF document is from Wu et al..38

##  10 Conclusion

In this paper we present OpenChemIE, a comprehensive system for information
extraction from chemistry literature at the document level. OpenChemIE
addresses the need for the integration of information across multiple
modalities in order to provide complete extractions of molecules and
reactions. We approach the general challenge of chemistry information
extraction by incorporating chemistry-informed algorithms to integrate the
results from individual modalities to obtain the final outputs. This approach
allows for the extraction of previously unresolvable information, such as
substrate scope investigations, and is a notable step toward achieving
multimodal analysis of chemistry literature.

OpenChemIE has made remarkable progress toward its objective of extracting
reaction data comprehensively from chemical literature, although some
challenges remain to be addressed. For instance, there is room for enhancing
the performance of machine learning models on diverse literature data:
MolScribe might be further developed to more precisely capture less common
representations of molecular structures, including Markush structures; the PDF
parsing tool may benefit from adjustments to better cater to chemical
documents. Additionally, while our system is adept at parsing multiple
multimodal relationships, enhancing its ability to understand the complex
interdependencies between different modalities in chemical documents
represents an exciting area for future development. The emergent abilities of
large language models hold promise for providing a more integrated end-to-end
solution for chemical information extraction, suggesting an optimistic pathway
forward.

##  11 Data and Software Availability

The OpenChemIE toolkit is publicly available:

  * •

Source code: https://github.com/CrystalEye42/OpenChemIE

  * •

Web interface: https://mit.openchemie.info

. Individual machine learning models in OpenChemIE can be found at the
following links:

  * •

MolScribe: https://github.com/thomas0809/MolScribe

  * •

RxnScribe: https://github.com/thomas0809/RxnScribe

  * •

MolDetect/MolCoref: https://github.com/Ozymandias314/MolDetect

  * •

ChemNER: https://github.com/Ozymandias314/ChemIENER

  * •

ChemRxnExtractor: https://github.com/jiangfeng1124/ChemRxnExtractor

The datasets for our molecule detection, molecule coreference resolution, and
R-group resolution processes are constructed from journal articles shared
between the American Chemical Society (ACS) and MIT under a private access
agreement.

  * •

The annotated images for the molecule coreference and detection task, as well
as their train/validation/test splits can be downloaded at
https://huggingface.co/datasets/Ozymandias314/MolCorefData.

  * •

The diagrams and annotations for the R-group resolution dataset, as well as
data for the comparison against Reaxys are located at
https://huggingface.co/datasets/Ozymandias314/OpenChemIEData for download.

Detailed evaluation results for our new models (ChemNER, MolDetect, and
MolCoref) and the overall OpenChemIE pipeline, a description of the data
annotation process, and implementation details for our PDF Parser are
available in the supporting information.

The authors thank Guy Zylberberg for his contribution to the web interface
development, and Zhengkai Tu for his chemical expertise. The authors
additionally thank the members of Regina Barzilay’s group and Connor Coley’s
group for helpful discussion and feedback. This work was supported by the
DARPA Accelerated Molecular Discovery (AMD) program under contract
HR00111920025 and the Machine Learning for Pharmaceutical Discovery and
Synthesis Consortium (MLPDS).

## References

  * 1 Reaxys. https://www.reaxys.com, (accessed on 07/01/2023)
  * Tu et al. 2023 Tu, Z.; Stuyver, T.; Coley, C. W. Predictive chemistry: machine learning for reaction deployment, reaction development, and reaction discovery. _Chem. Sci._ 2023, _14_ , 226–244
  * Maser et al. 2021 Maser, M. R.; Cui, A. Y.; Ryou, S.; DeLano, T. J.; Yue, Y.; Reisman, S. E. Multilabel Classification Models for the Prediction of Cross-Coupling Reaction Conditions. _Journal of Chemical Information and Modeling_ 2021, _61_ , 156–166, PMID: 33417449
  * Gao et al. 2018 Gao, H.; Struble, T. J.; Coley, C. W.; Wang, Y.; Green, W. H.; Jensen, K. F. Using Machine Learning To Predict Suitable Conditions for Organic Reactions. _ACS Central Science_ 2018, _4_ , 1465–1476, PMID: 30555898
  * Qian et al. 2023 Qian, Y.; Li, Z.; Tu, Z.; Coley, C.; Barzilay, R. Predictive Chemistry Augmented with Text Retrieval. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore, 2023; pp 12731–12745
  * Qian et al. 2023 Qian, Y.; Guo, J.; Tu, Z.; Coley, C. W.; Barzilay, R. RxnScribe: A Sequence Generation Model for Reaction Diagram Parsing. _Journal of Chemical Information and Modeling_ 2023, _63_ , 4030–4041
  * Guo et al. 2022 Guo, J.; Ibanez-Lopez, A. S.; Gao, H.; Quach, V.; Coley, C. W.; Jensen, K. F.; Barzilay, R. Automated Chemical Reaction Extraction from Scientific Literature. _J. Chem. Inf. Model._ 2022, _62_ , 2035–2045
  * Beard and Cole 2020 Beard, E. J.; Cole, J. M. ChemSchematicResolver: A Toolkit to Decode 2D Chemical Diagrams with Labels and R-Groups into Annotated Chemical Named Entities. _Journal of Chemical Information and Modeling_ 2020, _60_ , 2059–2072, PMID: 32212690
  * Swain and Cole 2016 Swain, M. C.; Cole, J. M. ChemDataExtractor: A Toolkit for Automated Extraction of Chemical Information from the Scientific Literature. _J. Chem. Inf. Model._ 2016, _56_ , 1894–1904
  * Zhao et al. 2012 Zhao, S.-C.; Ji, K.-G.; Lu, L.; He, T.; Zhou, A.-X.; Yan, R.-L.; Ali, S.; Liu, X.-Y.; Liang, Y.-M. Palladium-Catalyzed Divergent Reactions of 1,6-Enyne Carbonates: Synthesis of Vinylidenepyridines and Vinylidenepyrrolidines. _The Journal of Organic Chemistry_ 2012, _77_ , 2763–2772, PMID: 22364228
  * Qian et al. 2023 Qian, Y.; Guo, J.; Tu, Z.; Li, Z.; Coley, C. W.; Barzilay, R. MolScribe: Robust Molecular Structure Recognition with Image-to-Graph Generation. _Journal of Chemical Information and Modeling_ 2023, _63_ , 1925–1934
  * Filippov and Nicklaus 2009 Filippov, I. V.; Nicklaus, M. C. Optical structure recognition software to recover chemical information: OSRA, an open source solution. 2009
  * Peryea et al. 2019 Peryea, T.; Katzel, D.; Zhao, T.; Southall, N.; Nguyen, D.-T. MOLVEC: Open source library for chemical structure recognition. Abstracts of papers of the American Chemical Society. 2019
  * Staker et al. 2019 Staker, J.; Marshall, K.; Abel, R.; McQuaw, C. M. Molecular Structure Extraction from Documents Using Deep Learning. _J. Chem. Inf. Model._ 2019, _59_ , 1017–1029
  * Rajan et al. 2020 Rajan, K.; Brinkhaus, H. O.; Zielesny, A.; Steinbeck, C. A review of optical chemical structure recognition tools. _J. Cheminf._ 2020, _12_ , 1–13
  * Oldenhof et al. 2020 Oldenhof, M.; Arany, A.; Moreau, Y.; Simm, J. ChemGrapher: optical graph recognition of chemical compounds by deep learning. _J. Chem. Inf. Model._ 2020, _60_ , 4506–4517
  * Clevert et al. 2021 Clevert, D.-A.; Le, T.; Winter, R.; Montanari, F. Img2Mol – accurate SMILES recognition from molecular graphical depictions. _Chem. Sci._ 2021, _12_ , 14174–14181
  * Rajan et al. 2021 Rajan, K.; Zielesny, A.; Steinbeck, C. DECIMER 1.0: deep learning for chemical image recognition using transformers. _J. Cheminf._ 2021, _13_ , 1–16
  * Wilary and Cole 2021 Wilary, D. M.; Cole, J. M. ReactionDataExtractor: A Tool for Automated Extraction of Information from Chemical Reaction Schemes. _J. Chem. Inf. Model._ 2021, _61_ , 4962–4974
  * Wilary and Cole 2023 Wilary, D. M.; Cole, J. M. ReactionDataExtractor 2.0: A Deep Learning Approach for Data Extraction from Chemical Reaction Schemes. _Journal of Chemical Information and Modeling_ 2023, _63_ , 6053–6067, PMID: 37729111
  * Xu et al. 2022 Xu, Y. et al. MolMiner: You Only Look Once for Chemical Structure Recognition. _J. Chem. Inf. Model._ 2022, _62_ , 5321–5328
  * Rajan et al. 2023 Rajan, K.; Brinkhaus, H. O.; Agea, M. I.; Zielesny, A.; Steinbeck, C. DECIMER.ai - An open platform for automated optical chemical structure identification, segmentation and recognition in scientific publications. 2023,
  * Krallinger et al. 2015 Krallinger, M.; Leitner, F.; Rabal, O.; Vazquez, M.; Oyarzabal, J.; Valencia, A. CHEMDNER: The drugs and chemical names extraction challenge. _J. Cheminformatics_ 2015, _7_ , S1
  * Nguyen et al. 2020 Nguyen, D. Q.; Zhai, Z.; Yoshikawa, H.; Fang, B.; Druckenbrodt, C.; Thorne, C.; Hoessel, R.; Akhondi, S. A.; Cohn, T.; Baldwin, T.; Verspoor, K. ChEMU: Named Entity Recognition and Event Extraction of Chemical Reactions from Patents. Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part II. 2020; pp 572–579
  * Jessop et al. 2011 Jessop, D. M.; Adams, S. E.; Willighagen, E. L.; Hawizy, L.; Murray-Rust, P. OSCAR4: a flexible architecture for chemical text-mining. _J. Cheminformatics_ 2011, _3_ , 41
  * Hawizy et al. 2011 Hawizy, L.; Jessop, D. M.; Adams, N.; Murray-Rust, P. ChemicalTagger: A tool for semantic text-mining in chemistry. _J. Cheminformatics_ 2011, _3_ , 17
  * Zhu and Cole 2022 Zhu, M.; Cole, J. M. PDFDataExtractor: A Tool for Reading Scientific Text and Interpreting Metadata from the Typeset Literature in the Portable Document Format. _J. Chem. Inf. Model._ 2022, _62_ , 1633–1643
  * Carion et al. 2020 Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; Zagoruyko, S. End-to-End Object Detection with Transformers. Computer Vision - ECCV 2020 \- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I. 2020; pp 213–229
  * Chen et al. 2022 Chen, T.; Saxena, S.; Li, L.; Fleet, D. J.; Hinton, G. E. Pix2seq: A Language Modeling Framework for Object Detection. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. 2022
  * 30 EasyOCR. https://github.com/JaidedAI/EasyOCR, (accessed on 07/01/2023)
  * Liu et al. 2022 Liu, X.; Liu, H.; Bian, C.; Wang, K.-H.; Wang, J.; Huang, D.; Su, Y.; Lv, X.; Hu, Y. Synthesis of 3-Trifluoromethyl-1,2,4-triazolines and 1,2,4-Triazoles via Tandem Addition/Cyclization of Trifluoromethyl N-Acylhydrazones with Cyanamide. _The Journal of Organic Chemistry_ 2022, _87_ , 5882–5892, PMID: 35412831
  * Lee et al. 2019 Lee, J.; Yoon, W.; Kim, S.; Kim, D.; Kim, S.; So, C. H.; Kang, J. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. _Bioinformatics_ 2019, _36_ , 1234–1240
  * Zhang et al. 2023 Zhang, Y.; Zhang, Z.; Xia, Y.; Wang, J.; Peng, Y.; Song, G. TfOH-Catalyzed Cascade Reaction: Metal-Free Access to 3,3-Disubstituted Phthalides from o-Alkynylbenzoic Acids. _The Journal of Organic Chemistry_ 2023, _88_ , 12924–12934, PMID: 37643422
  * Landrum 2010 Landrum, G. RDKit. https://www.rdkit.org/, 2010
  * Lin et al. 2015 Lin, T.-Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C. L.; Dollár, P. Microsoft COCO: Common Objects in Context. 2015
  * Shen et al. 2021 Shen, Z.; Zhang, R.; Dell, M.; Lee, B. C. G.; Carlson, J.; Li, W. LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis. 16th International Conference on Document Analysis and Recognition, ICDAR 2021, Lausanne, Switzerland, September 5-10, 2021, Proceedings, Part I. 2021; pp 131–146
  * 37 pdfminer.six. https://pdfminersix.readthedocs.io/, (accessed on 07/01/2023)
  * Wu et al. 2022 Wu, S.; Song, W.; Zhu, R.; Hu, J.; Zhao, L.; Li, Z.; Yu, X.; Xia, C.; Zhao, J. Catalyst-Free α-Alkylation-α-Hydroxylation of Oxindole with Alcohols. _The Journal of Organic Chemistry_ 2022, _87_ , 5464–5471, PMID: 35389661