# Fire and Smoke Digital Twin – A computational framework for modeling fire
incident outcomes

Junfeng Jiao<EMAIL_ADDRESS>1234-5678-9012 ,  Ryan Hardesty Lewis
<EMAIL_ADDRESS>University of Texas at AustinP.O. Box
1212AustinTexasUSA43017-6221 ,  Kijin Seong  University of Texas at Austin1
Thørväld CircleAustinTexasUSA<EMAIL_ADDRESS>,  Arya Farahi
University of Texas at AustinAustinTexasUSA<EMAIL_ADDRESS>,
Paul Navratil  Texas Advanced Computing CenterRono-HillsAustinTexasUSA ,  Nate
Casebeer  Austin Fire Department30 Shuangqing RdAustinTexasUSA and  Dev Niyogi
University of Texas at Austin8600 Datapoint DriveAustinTexasUSA78229
<EMAIL_ADDRESS>
(2023)

###### Abstract.

Fires and burning are the chief causes of particulate matter (PM2.5), a key
measurement of air quality in communities and cities worldwide. This work
develops a live fire tracking platform to show active reported fires from over
twenty cities in the U.S., as well as predict their smoke paths and impacts on
the air quality of regions within their range. Specifically, our close to
real-time tracking and predictions culminates in a digital twin to protect
public health and inform the public of fire and air quality risk. This tool
tracks fire incidents in real-time, utilizes the 3D building footprints of
Austin to simulate smoke outputs, and predicts fire incident smoke falloffs
within the complex city environment. Results from this study include a
complete fire and smoke digital twin model for Austin. We work in cooperation
with the City of Austin Fire Department to ensure the accuracy of our forecast
and also show that air quality sensor density within our cities cannot
validate urban fire presence. We additionally release code and methodology to
replicate these results for any city in the world. This work paves the path
for similar digital twin models to be developed and deployed to better protect
the health and safety of citizens.

smoke prediction, physical simulation, digital twin, urban fire

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation email; Unknown Conference, USA††booktitle:
;††ccs: Computer systems organization Embedded systems††ccs: Computer systems
organization Real-time systems††ccs: Computing methodologies Modeling and
simulation††ccs: Applied computing Physical sciences and engineering

##  1\. Introduction

A digital twin is often described as “a digital representation of a physical
item or assembly using integrated simulations and service data” (Vrabi et al.,
2018). In the pursuit of an eventual all-encompassing “Smart City,” or a
digital twin of an entire city’s infrastructure, there have been numerous
research ventures into digitizing city information, such as flood modeling,
traffic prediction, emergency evacuation routing, among other applications
(Gil et al., 2021; Ong et al., 2022). The United Nations predicts that most of
the world’s population currently lives within cities. Coupled with the fact
that the probability of natural disasters will only increase as climate change
progresses (inbook, USA et al.(2014), emergency management and public services
must be scaled to meet this future. In particular, active monitoring systems
be developed to track, warn, and even predict disastrous events (Allen and
Melgar, 2019; Antwi-Agyakwa et al., 2023). A large number of disastrous events
are dangerous because resulting air pollutants, like asbestos, pesticides, and
cyanide, find themselves airborne and transported downwind via smoke to affect
nearby populations (Carlin et al., 2015; Nanjappan et al., 2021; Bush et al.,
2000). As wildfires and other fire events grow in magnitude, it’s hard not to
realize that urban fires are not normal, or at least they used to be much less
frequent.

Smoke impacts from fire events have dangerous consequences (Stefanidou et al.,
2008). Air pollution has been found to create an increased rate of morbidity
and mortality, both causing and accelerating the progress of many diseases and
conditions (Goswami et al., 2020). Moreover, 80%-90% of fire-related deaths
are attributed to smoke inhalation (Toon et al., 2010), and even a sustained
small change in air quality can increase the mortality rate of some diseases
by twentyfold (Wu et al., 2020). To meet this challenge of often-occurring yet
widely ignored health hazards from local fires, we develop FireCOM, a “digital
twin” model that focuses on urban fires with real-time tracking and prediction
of their smoke falloff, for community and neighborhood level awareness. This
tool is designed to help the city officials in planning stage and mitigating
the threat of urban fires and their aftermaths.

Fires detection and tracking is usually performed using satellite data. The
fire detection models often relies on abnormal heat signatures and bright
spots on the Earth’s surface (Land Atmosphere Near Real-Time Capability For
EOS Fire Information For Resource Management System, 2021; Matson and Dozier,
1981). This type of tracking is helpful for wildfires, which are so large that
they appear on satellite data. Their distinct and high signal-to-noise heat
signal makes it easy to rely on satellite data. These signals for the fire
incidents in an urban environment, however, are low signal-to-noise and often
indistinguishable from the white noise, which makes detection difficult and
leads to many false positives due to other source of noise, such as sun-glint
off metallic structures, or false negatives (Roy et al., 2006). False
negatives are more common, though. Often there are no detections, as not all
urban fires are large enough for spatial satellite resolution and often remain
covered by trees, buildings, and other obstructions. An alternative route
would be to use 911 calls, but not much is done with this data. Many U.S.
cities provide live pages to the public of emergency incidents as they are
reported. These reports can be accurately pinpointed to a street address and
the time lags are insignificant. FireCOM utilizes this information to track
urban fires.

Tracking fires in real-time is not the only challenge. It is more important to
model and predict the impact of urban fire incidents, specifically particulate
matter exposure, such that the decision-makers can react rapidly as the
situation unfolds. The existing fire and smoke studies mainly focus on
wildfires (Chow et al., 2022; Mandel et al., 2014; Prichard et al., 2019).
Wildfire impact is often larger – spans hundreds and thousands of acres –
making these models less practical for vulnerable urban residents avoiding
exposure to smoke. While urban fire studies are not non-existing (Himoto and
Tanaka, 2003; Ferreira et al., 2014), not much work is done in this area. This
is despite the fact that fire incidents are more widespread and common in
urban communities. FireCOM fills this gap.

FireCOM gathers real-time fire and building footprint data, and combines this
information with weather data to accurately predict smoke dispersion within
one, two, and three hours following a fire incident. This model not only helps
to warn individuals about deteriorating air quality in their vicinity, but
also assists fire departments and decision-makers in coordinating their
efforts to mitigate the fire and its smoke-related aftermath.

The primary users of FireCOM include urban residents, emergency responders,
and decision-makers involved in fire management and public health. By serving
as an early warning system, FireCOM aims to improve health conditions and
facilitate a more effective response to urban fires. The tool’s accuracy and
effectiveness are validated through data from thousands of air quality sensors
and through collaborative case studies with the Austin Fire Department,
ensuring its reliability and practical applicability in predicting urban fire
impacts on air quality.

##  2\. Related Work

The toxicity of wildfire smoke and urban smoke is a significant health hazard
(Alarie, 2002; Bush et al., 2000; Carlin et al., 2015). Carbon monoxide is
often the leading cause of deaths in fires, but burning synthetic materials,
such as those in urban environments, produces cyanide in fires that causes up
to 50% of fire deaths (Alarie, 2002). Building structures are often made of
inorganic materials, implying that fumes from burning structures could be more
harmful than forest fires. Fire incidents in urban environments, where the
majority of fires are structure fires (Singh et al., 2021) compounded with
high population density compared to rural areas, present an acute and
immediate danger to vulnerable communities. Hence, real-time tracking of smoke
falloff and mitigating its harms can save lives. Actively tracking,
predicting, and managing smoke is usually focused on wildfires, with examples
including NOAA’s AirNow forecast (Stephens et al., 2001) and the US Forest
Service’s BlueSky (Larkin et al., 2009). However, little work has been done in
predicted smoke from urban fires, with existing models focusing on evaluating
risk and fire spread (Li and Davidson, 2013; Himoto and Tanaka, 2003; Ferreira
et al., 2014). To the best of our knowledge, our model provides the first
city-wide tracking and prediction of urban smoke.

Prediction of the fire smoke diffusion process requires running
computationally expensive fluid dynamics numerical simulations, but results in
near-realistic outcomes. For instance, Zhao et al. (2022) used their smoke
simulation to recommend changes to typical subway carriages and proposed
guidelines for evacuation protocols. Their results achieved the same
performance as an empirical miniature tunnel study done by Li et al. (2012)
almost a decade earlier. Fluid dynamic simulations have become a tool of
engineering, design, and prediction with verifiable accuracy (Wang et al.,
2010). However, for these simulations to be useful outside of a scientific
audience, visualization is key. We combine domain-specific visualizations of
both fluid-simulated smoke and a 3D city superimposed in order to make our
forecasts publicly available, through a web dashboard, to urban residents.
Alongside the help of real-time air sensors, the combination of these various
tools allows us to model smoke dynamics on a city-wide scale. Figure 1 shows
the early ability of our approximate VSmoke forecast in determining the land
uses an urban fire affects and the degree to which they will be affected.

Figure 1. A VSmoke smoke simulation in-browser, with residences affected also
highlighted according to smoke impact.

A VSmoke smoke simulation in-browser, with residences affected also
highlighted according to smoke impact. (Arya: Please never include a figure
not referenced in the paper. Figures are there to guide and provide support
for the arguments and storytelling. The reader cannot understand what we have
to say about this figure, how it is related to this work, and where in the
text they should look at this figure, and how it should be interpreted. )

Figure 2. A diagram of the urban smoke prediction workflow.

A diagram showcasing the urban fire workflow.

##  3\. FireCOM a digital twin architecture for active fires

FireCOM is a digital twin architecture designed for real-time monitoring and
prediction of active fires and their impact on urban environments. Our data
comprises three main components: (1) fire incidents, (2) weather data, and (3)
air quality data. We obtain real-time fire incident locations from the Austin
Fire Department, weather and wind conditions surrounding each fire from the
National Weather Service, and air quality data from nearby outdoor air quality
sensors provided by PurpleAir’s sensors111https://api.purpleair.com/.

Blender, a powerful open-source 3D creation suite, is chosen as the research
platform due to its easy integration with Python, extensibility, and
compatibility with GIS tools and accurate fluid simulations. Although Blender
may not be primarily designed for real-time work, it offers precise models and
simulations, essential for our research. We integrate OpenTopography’s
topographic map of Austin and OpenStreetMap’s 3D city structures to create the
digital twin’s base map.

To improve the accuracy of our simulations, we incorporate real-world
infrastructure, including building and street classifications, such as
schools, hospitals, and speed limits. This enables our model to identify areas
at higher risk from reduced air quality.

Our fire model leverages live information from local fire departments, weather
services, and air quality sensors to predict smoke paths and monitor fire
incidents. We map reported fire incidents using geographical coordinates, and
then query the National Weather Service for recent weather and wind
information at each location. Subsequently, we generate predicted smoke paths
for each fire and validate our predictions using data from air quality sensors
to ensure accuracy in the upcoming hours. A general workflow of FireCOM can be
seen in Figure 2.

By integrating real-time data, GIS tools, and fluid simulations within the
Blender platform, FireCOM serves as a reliable digital twin architecture for
active fire monitoring and prediction, ultimately enhancing urban safety and
public health.

##  4\. Data

Real-time fire incident data are procured from the 911 calls’ active incident
page as updated for over twenty cities. We note that these cities have vastly
different schemes for all emergency information, mixing different date
formats, missing certain information like coordinates, and calling the same
category different things. To deal with this, we created a unified schema with
only the needed information. This includes incident name, date, longitude and
latitude coordinates, street address, and reporting department. Although
latitude and longitude and street address are translatable to one another, we
included both to reduce any geocoding or confusion on the part of the end-
user, whose interest is in the street address. We, however, only need the
exact coordinates to display on a map. From all of our cities, we manually
annotated what each live incident feed returned and created scripts to convert
all the different formats we received back, including HTML tables, RSS feeds,
plain text files, and JSON documents, into our format. An example of such a
script is provided in Algorithm 1.

Algorithm 1 Urban Fire Retrieval

1:for $city=1,2,\ldots$ do

2: Fetch city data from permanent live URL

3: Based on city, convert to custom format (JSON)

4: Clean data with timestamps and geolocations

5: Store data in date-based structure

6:end for

After variations of this script run for each and every annotated city’s live
data, we compare the latest data with the same information retrieved on the
previous request, actively updating which fires are still burning and which
ones have been reportedly fixed. With this information, we track active fires
in all reported cities. We additionally determine where most fires in each
city are likely to occur by looking at the average amount of fires per region,
creating a ”fire risk” map of Austin.

In 2022, over 20,000 fires occurred in Austin alone. Analyzing this data
revealed several trends. By intersecting each fire event with a tract-level
map of Austin, we created a ”fire risk” analysis map where orange represents
normal fire risk, while yellow and red represent below-average and above-
average risk, respectively. This map exhibits a correlation with the City of
Austin’s FireCAT Wildfire Risk Assessment (Pohl, 2022), as seen in Figure 3,
indicating that wildfires remain a significant source of fire incidents.
However, it also highlights the elevated probability of fires in downtown
Austin.

Our results suggest that most fire incidents occur in the afternoon, typically
between 5 p.m. and 10 p.m. (see Figure 4). However, the transition between
different time periods is not as smooth as one might expect, warranting
further investigation into the underlying factors influencing these trends.

Figure 3. A comparison of FireCAT Wildfire Risk Assessment (left) and our Fire
Averages per tract (right).

A comparison of FireCAT Wildfire Risk Assessment and our Fire Averages per
tract.

Figure 4. A graph of the number of reported fires by the hour.

A graph of the number of reported fires by the hour.

Weather service data is incredibly important while surveying smoke impacts, as
wind direction and strength, among other weather conditions, ultimately
determine the region of impact. As demonstrated by the HRRR-Smoke model for
wildfires, weather is perhaps the single most important factor that determines
the spatial progress of smoke (Chow et al., 2022). City weather is very well-
researched and provided mainly by Weather.gov, a government service that
provides hourly updates of weather conditions across the entire continental
United States. We query Weather.gov and selectively retrieve the important
information, namely, wind speed, direction, and conditions.

Finally, we create a large data set of time-independent data. This data set
includes elevation maps, building footprint data sets, and locations of
essential points of interest like hospitals and fire departments. The
elevation data is similarly available to the public from OpenTopography, and
the rest are publicly available from Open Data Portals for each respective
city.

##  5\. Smoke Prediction and Validation

###  5.1. Methods

To predict smoke output from various tracked fires, we initially employed a
Gaussian smoke prediction model based on the VSmoke particulate matter
concentration program developed by the Georgia Forestry Commission (GFC)
(Lavdas, 1996a, b). This model calculates smoke dispersion using a normal
probability distribution, which is common for ground-level fires (Bosanquet
and Pearson, 1936).

We gathered real-time weather information, such as wind speed, wind direction,
and humidity, from Weather.gov and NOAA’s HRRR model to input into the VSmoke
model. From the live fire information, we can make average assumptions based
on the size of each fire, such as an “appliance fire” versus a “brush fire,”
what type of fuel the fire uses, as well as tweak other qualities of the
active fire.

However, it’s important to note that VSmoke doesn’t account for the
surrounding buildings’ geometry. To address this, we moved to a 3D city model
using Blender and OpenStreetMaps 3D (Over et al., 2010), which allowed us to
implement a more accurate smoke prediction model using MantaFlow (Chu and
Thuerey, 2017; Xie et al., 2018).

To do this, Blender was used, alongside OpenStreetMaps 3D, which is a
generated mesh of an entire city using building footprints and available
height information to create approximate models of each structure (Over et
al., 2010). We also overlay it on an accurate topographic height map of Austin
using OpenTopography, down to 30m precision. With the surrounding environment
in-place, we created a new BlenderPy script to spawn a smoke fluid simulation
at the exact longitude and latitude within the “3D City”, with parameters for
weather and fire matching those input into VSmoke. In this way, we run a smoke
simulation in a near-realistic digital twin of any city. An example of this
visualization, converted from Blender to a browser-view, can be seen in Figure
5.

Figure 5. A supervised smoke fluid simulation in-browser.

A supervised smoke fluid simulation in-browser.

###  5.2. Results

Using the VSmoke model as a reference, we tuned the MantaFlow fluid simulation
parameters to match the VSmoke prediction area. Figure 6 shows a comparison
between the VSmoke and MantaFlow predicted smoke outputs.

Figure 6. A comparison of the VSmoke and MantaFlow predicted smoke outputs.

A comparison of the VSmoke and MantaFlow predicted smoke outputs.

Typically, smoke systems are validated from air sensors, satellite imagery,
and meteorology stations (Chow et al., 2022). Since satellite imagery does not
work well for fires in an urban context, as well as that meteorology stations
were often sparse and located far from cities, we assessed the accuracy of our
smoke predictions by comparing them with air sensor data. We collaborated with
researchers from the Texas Advanced Computing Center (TACC) to evaluate the
overlap between our generated smoke trails and PurpleAir sensor data and found
that sensors were too sparse to detect urban smoke impacts. Instead, we used a
validation process by case studies with the City of Austin, where we were able
to use prescribed burns and set up local air sensors ahead of time to test the
accuracy of our model.

In terms of pipeline performance, our system proved to be efficient in
generating smoke predictions and displaying them in the browser. Upon
receiving a notification from the active fire query API, the system takes less
than a second to generate an approximate VSmoke prediction for up to three
hours in advance (not considering city geometry) and communicate that result
in-browser. For the more accurate MantaFlow 3D smoke predictions, which take
into account the city geometry, the pipeline takes approximately thirty
seconds to generate predictions for up to three hours in advance. This timely
generation and display of smoke predictions can be critical in facilitating
rapid decision-making and response efforts in urban fire scenarios.

###  5.3. Discussion

Our study resulted in two models: the VSmoke-based model, which is
computationally efficient but less accurate, and the MantaFlow-based model,
which is computationally intensive but provides more accurate results by
considering complex city geometry.

As for validation, our partners at TACC demonstrated that air sensors within
our cities were too sparse, never being even within a mile of a documented
fire. On the few collisions, they suggested that our findings were accurate,
as air sensors reported lower than average values within the predicted smoke
area. However, it’s important to note that air quality sensors in Austin and
major cities are too sparse to detect acute events like fires, even a decade
after a similar issue was reported by NASA (Pasch et al., 2011). NASA’s
solutions to this problem of ”sparse data” are generally smoothing of data
points over large regions, which is still not suitable for the smoke from
urban fires, which would be a spike in a small area.

Instead, we used experimental data provided from a series of case studies
conducted with the City of Austin. The Austin Fire Department, like many fire
departments, prescribes fires weeks in advance that need to occur for
firefighter training, brush removal, or forest clearing. Knowing the location
of a future fire, we could set up our air sensors downwind of the location and
detect if our model was accurate. We had predicted that air sensors that did
not change under downwind conditions were often behind buildings and other
obstructive geometry, which could be accounted for with our 3D smoke model.
The MantaFlow-based model demonstrated promising accuracy in our case study
with the City of Austin, finding that the air quality of downwind regions
correlates highly with our predicted smoke impact.

Despite the positive results, our study faced limitations due to the sparsity
of air quality sensors in Austin and other major cities. We recommend future
research in this area to consider deploying low-cost air sensors across cities
for improved detection and reporting of immediate threats to air quality.

However, we also recognize that some of our tracked cities, like Los Angeles,
had thousands of air sensors, yet exhibited the same behaviour. For this
reason, we also recommend the development of novel air quality tracking
techniques, like geostationary satellite detection, currently being worked on
by NASA ARSET (NASA Applied Remote Sensing Training Program (2022), ARSET).
With tools like these, validation of fire and smoke digital twins like FireCOM
will become much easier.

##  6\. Key challenges, solutions, and innovations

A key challenge in conducting this research was the relative scarcity of smoke
prediction models. While models do, to an extent, exist, they are often very
specific or meant for non-scientific usage. For example, the EPA’s smoke model
predicts long continuous regions of smoke across North America, without really
offering much useful information in terms of actual air quality impacts on
communities. Another example is the fluid simulations of game engines, such
Unreal Engine, which are meant to be optimized for visualization rather than
any consideration to accurately portray how smoke and fire act in life. For
these reasons, we chose to use smoke models from VSmoke from the Georgia
Forestry Commission and MantaFlow provided by the Technical University of
Munich.

Another challenge to overcome was the inability to present these models in any
direct way to viewers. Both fluid simulations and real-time active maps are
rather costly to compute and maintain on a running basis, and doing these
smoke predictions on-demand to display immediately within a live map required
some technical innovations. Displaying fluid simulations interactively to
users is a pipe dream, as well, as they take a powerful computer to compute,
yet alone visualize. We needed to make the technology accessible at the whim
of a smartphone’s web browser. To this end, we created APIs for both our 2D
and 3D smoke pipelines, as well as used some tricks to display the output of
each respective simulation.

Firstly, we rewrote a lot of GFC’s codebase to optimize it for newer
computers, as well as run parallel smoke simulations at the request of an API.
The program outputs Keyhole Markup (.KML) files to visualize each 2D smoke
simulation, which we spaced out at one hour, two hour, and three hour
predicted simulations. Secondly, we created an API to run fluid simulations
for fires on-demand in Blender. While the model expanded for a full three
hours in Blender, we timestamped and exported the one, two, and three hour
marks to Filmbox (.FBX) object files, which can be rendered in-browser.

With the smoke generated in six different iterations, we just needed to
display our results on each browser. To reduce confusion between the two
separate models, we displayed VSmoke’s simulations on a standard map labelled
“2D”, while presenting the MantaFlow simulations on a vector-tiled 3D map,
courtesy of MapBox GL, which quickly renders the same OpenStreetMap 3D tiles
from Blender in-browser. From this, we just used some extension libraries,
like Threebox.js, to display our smoke simulations within the 3D space from
the fire’s origin, all without requiring anything other than a machine capable
of viewing a model, which each computer and smartphone is capable of. Our
computationally efficient VSmoke-enabled 2D map can be seen in Figure 7, while
our fluid-simulated MantaFlow 3D map can be seen in Figure 8. We also overlay
a dynamically updating ”fire risk” layer, which changes as the average amount
of fires per tract in Austin updates day by day.

The novelty of our work lends itself to three areas: (1) data integration, (2)
shifting from 2D to 3D, and (3) use of public and generalized data. We
integrated information from a multitude of resources in creating this real-
time model, including building footprints, air sensors, wind and weather
updates, as well as census data and points of interest around the city. With
all of this data, we can easily begin to generate smoke models across the
city, but with only an approximate accuracy, given the complexity of an urban
environment. Shifting to a 3D perspective of a city, complete with topography
and building geometry, allows for a more realistic and better detailed
understanding of how each fire falloff will develop, and where exactly it will
affect, allowing both citizens and firefighters to make informed decisions
when dealing with fire in their communities. Finally, our work was done
completely with public data, and alongside our released open-source code,
makes this model completely generalizable and replicable to any city.

Figure 7. Our 2D real-time fire and smoke map in-browser.

Our 2D real-time fire and smoke map in-browser.

Figure 8. Our 3D real-time fire and smoke map in-browser.

Our 3D real-time fire and smoke map in-browser.

##  7\. Conclusions

On-demand smoke simulations for urban fires were never attempted because of
the complex geometry and insights needed to perform such a task, but with our
model, one can easily predict both approximate smoke outcomes, as well as
hyper-realistic smoke fluid simulations for any fire. With our APIs,
researchers can begin to predict future fires, as well as analyze past fires
for any city. With our map, citizens can become well-informed of the air
quality risks posed by urban fires around their city, as well as better
understand how smoke might impact their nearby communities. We could
potentially serve as an early warning system for lowered air quality and smoke
in areas of high risk, where exposure to reduced PM2.5 could endanger lives
and hinder child development.

In the near future, we expect to see our model used to conduct air quality
research into respiratory diseases. We can detect if there is a correlation
between areas of high urban fire risk alongside areas of high COPD, Asthma,
and other respiratory issues. On top of this, we wish to see our model used in
growing digital twins of Smart City environments, whereas every aspect of a
city can be visualized, predicted, and optimized to save lives and serve
communities better.

Within a full digital twin model, our current data would be a marginal list of
the information necessary. A full twin might take advantage of the traffic
conditions on a right or left lane to indicate which side of the road a
burning car might be on, or even make estimations based on the total
aggregation of phone signals in an area about how many people might be
potentially affected by a smoke falloff. For future research in this area, we
would encourage the use of Big Data in retrieving all relevant local
conditions and creating a more full-scale model.

## Funding

This research is supported by the Bridging Barriers Initiative Good Systems
Grand Challenge at The University of Texas at Austin, the City of Austin
(UTA19-000382), National Science Foundation (2043060, 2133302, 1952193,
2125858, 2236305), NSF-GOLD (2228205), and CSE-OCE (1835739).

###### Acknowledgements.

The authors extend their sincere gratitude to Dan Chen (Georgia Forestry
Association), Marc Coudert (Office of Resilience, City of Austin), Braniff
Davis (Austin Fire Department), and Chief Joel Baker (Austin Fire Department)
for supporting our project. The authors also acknowledge an interagency
agreement between UT and the City of Austin through the Bridging Barriers
Initiative.

## References

  * (1)
  * Alarie (2002) Yves Alarie. 2002\.  Toxicity of fire smoke.  _Critical Reviews in Toxicology_ 32, 4 (2002), 259–289.  https://doi.org/10.1080/20024091064246
  * Allen and Melgar (2019) Richard Allen and Diego Melgar. 2019.  Earthquake Early Warning: Advances, Scientific Challenges, and Societal Needs.  _Annual Review of Earth and Planetary Sciences_ 47 (05 2019).  https://doi.org/10.1146/annurev-earth-053018-060457
  * Antwi-Agyakwa et al. (2023) Kwasi Antwi-Agyakwa, Mawuli Afenyo, and Donatus Angnuureng. 2023\.  Know to Predict, Forecast to Warn: A Review of Flood Risk Prediction Tools.  _Water_ 15 (01 2023).  https://doi.org/10.3390/w15030427
  * Bosanquet and Pearson (1936) C. H. Bosanquet and James L. Pearson. 1936.  The spread of smoke and gases from chimneys.  _Transactions of The Faraday Society_ 32 (1936), 1249–1263.
  * Bush et al. (2000) Parshall B. Bush, Daniel G. Neary, and Charles K. McMahon. 2000.  Fire and Pesticides: A Review of Air Quality Considerations.
  * Carlin et al. (2015) Danielle Carlin, Theodore Larson, Jean Pfau, Stephen Gavett, Arti Shukla, Aubrey Miller, and Ronald Hines. 2015.  Current Research and Opportunities to Address Environmental Asbestos Exposures.  _Environmental health perspectives_ 123 (08 2015), A194–A197.  https://doi.org/10.1289/ehp.1409662
  * Chow et al. (2022) Fotini Katopodes Chow, Katelyn A. Yu, Alexander Young, Eric James, Georg A. Grell, Ivan Csiszar, Marina Tsidulko, Saulo Freitas, Gabriel Pereira, Louis Giglio, Mariel D. Friberg, and Ravan Ahmadov. 2022.  High-Resolution Smoke Forecasting for the 2018 Camp Fire in California.  _Bulletin of the American Meteorological Society_ 103, 6 (2022), E1531 – E1552.  https://doi.org/10.1175/BAMS-D-20-0329.1
  * Chu and Thuerey (2017) Mengyu Chu and Nils Thuerey. 2017.  Data-Driven Synthesis of Smoke Flows with CNN-Based Feature Descriptors.  _ACM Trans. Graph._ 36, 4, Article 69 (jul 2017), 14 pages.  https://doi.org/10.1145/3072959.3073643
  * Ferreira et al. (2014) Tiago Ferreira, Romeu Vicente, Raimundo Mendes Silva, H. Varum, A. Costa, and Rui Maio. 2014\.  Urban Fire Risk: evaluation and emergency planning.  https://doi.org/10.13140/2.1.5053.6648
  * Gil et al. (2021) Yolanda Gil, Daniel Garijo, Deborah Khider, Craig A. Knoblock, Varun Ratnakar, Maximiliano Osorio, Hernán Vargas, Minh Pham, Jay Pujara, Basel Shbita, Binh Vu, Yao-Yi Chiang, Dan Feldman, Yijun Lin, Hayley Song, Vipin Kumar, Ankush Khandelwal, Michael Steinbach, Kshitij Tayal, Shaoming Xu, Suzanne A. Pierce, Lissa Pearson, Daniel Hardesty-Lewis, Ewa Deelman, Rafael Ferreira Da Silva, Rajiv Mayani, Armen R. Kemanian, Yuning Shi, Lorne Leonard, Scott Peckham, Maria Stoica, Kelly Cobourn, Zeya Zhang, Christopher Duffy, and Lele Shu. 2021.  Artificial Intelligence for Modeling Complex Systems: Taming the Complexity of Expert Models to Improve Decision Making.  _ACM Trans. Interact. Intell. Syst._ 11, 2, Article 11 (jul 2021), 49 pages.  https://doi.org/10.1145/3453172
  * Goswami et al. (2020) Meera Goswami, Dalip Mansotra, Shivalika Sharma, Gaurav Pant, and Prakash Joshi. 2020.  _EFFECTS OF AIR POLLUTION ON HUMAN HEALTH_.  135–144.
  * Himoto and Tanaka (2003) Keisuke Himoto and Takeyoshi Tanaka. 2003.  A Physically-Based Model for Urban Fire Spread.  _Fire Safety Science_ 7 (01 2003), 129–140.  https://doi.org/10.3801/IAFSS.FSS.7-129
  * Land Atmosphere Near Real-Time Capability For EOS Fire Information For Resource Management System (2021) Land Atmosphere Near Real-Time Capability For EOS Fire Information For Resource Management System. 2021\.  MODIS/Aqua+Terra Thermal Anomalies/Fire locations 1km FIRMS V006 NRT (Vector data).  https://doi.org/10.5067/FIRMS/MODIS/MCD14DL.NRT.0061
  * Larkin et al. (2009) Narasimhan Larkin, Susan O’Neill, Robert Solomon, Sean Raffuse, Tara Strand, Dana Coe, Candace Krull, Miriam Rorig, Janice Peterson, and Sue Ferguson. 2009\.  The BlueSky smoke modeling framework.  _Int. J. Wildland Fire_ 18 (01 2009).  https://doi.org/10.1071/WF07086
  * Lavdas (1996a) L. G. Lavdas. 1996a.  Improving Control of Smoke from Prescribed Fire Using Low Visibility Occurrence Risk Index.  _Southern Journal of Applied Forestry_ 20, 1 (02 1996), 10–14.  https://doi.org/10.1093/sjaf/20.1.10 arXiv:https://academic.oup.com/sjaf/article-pdf/20/1/10/23457356/sjaf0010.pdf
  * Lavdas (1996b) Leonidas G. Lavdas. 1996b.  _Program VSMOKE–Users Manual_.  Technical Report.  https://doi.org/10.2737/srs-gtr-6
  * Li et al. (2012) Junmei Li, Shanshan Liu, Yanfeng Li, Chao Chen, Xuan Liu, and Chenchen Yin. 2012\.  Experimental Study of Smoke Spread in Titled Urban Traffic Tunnels Fires.  _Procedia Engineering_ 45 (2012), 690–694.  https://doi.org/10.1016/j.proeng.2012.08.224 2012 International Symposium on Safety Science and Technology.
  * Li and Davidson (2013) Sizheng Li and Rachel A. Davidson. 2013.  Parametric study of urban fire spread using an urban fire simulation model with fire department suppression.  _Fire Safety Journal_ 61 (Oct. 2013), 217–225.  https://doi.org/10.1016/j.firesaf.2013.09.017
  * Mandel et al. (2014) J. Mandel, Shai Amram, J. Beezley, Guy Kelman, Adam Kochanski, Volodymyr Kondratenko, B. Lynn, B. Regev, and Martin Vejmelka. 2014.  Recent advances and applications of WRF–SFIRE.  _Natural Hazards and Earth System Science_ 14 (10 2014), 2829–2845.  https://doi.org/10.5194/nhess-14-2829-2014
  * Matson and Dozier (1981) M. Matson and Jeff Dozier. 1981.  Identification of subresolution high temperature sources using thermal IR sensor.  _Photogrammetric Engineering and Remote Sensing_ 47 (01 1981), 1311–1318.
  * Nanjappan et al. (2021) Eshwari Nanjappan, Emily Sullo, Srijain Shrestha, Shara Thomas, and Elysée Nouvet. 2021.  Californian Wildfires and Associated Human Health Outcomes: An Epidemiological Scoping Review of the Creative Commons Attribution License (CC BY 4.0).  5 (07 2021), 944–953.
  * NASA Applied Remote Sensing Training Program (2022) (ARSET) NASA Applied Remote Sensing Training Program (ARSET). 2022.  ARSET - Accessing and Analyzing Air Quality Data from Geostationary Satellites.  http://appliedsciences.nasa.gov/join-mission/training/english/arset-accessing-and-analyzing-air-quality-data-geostationary
  * Ong et al. (2022) Joshua Ong, David Kulpanowski, Yangxinyu Xie, Evdokia Nikolova, and Ngoc Mai Tran. 2022\.  OpenEMS: an open-source Package for Two-Stage Stochastic and Robust Optimization for Ambulance Location and Routing with Applications to Austin-Travis County EMS Data.  _arXiv preprint arXiv:2201.11208_ (2022).
  * Over et al. (2010) Martin Over, Arne Schilling, S. Neubauer, and Alexander Zipf. 2010.  Generating web-based 3D City Models from OpenStreetMap: The current situation in Germany.  _Comput. Environ. Urban Syst._ 34 (2010), 496–507.
  * Pasch et al. (2011) A. Pasch, Patrick Zahn, J. Dewinter, M. Haderman, J. White, P. Dickerson, Timothy Dye, and Randall Martin. 2011\.  Improve EPA’s AIRNow Air Quality Index Maps with NASA/NOAA Satellite Data.  _AGU Fall Meeting Abstracts_ (12 2011), 0376–.
  * Pohl (2022) Kelly Pohl. 2022\.  Austin wildfire and vulnerable populations tool.  https://headwaterseconomics.org/wildfire/homes-risk/austin-wildfire-population-risk/
  * Prichard et al. (2019) Susan Prichard, Sim N. Larkin, Roger Ottmar, Nancy H.F. French, Kirk Baker, Tim Brown, Craig Clements, Matt Dickinson, Andrew Hudak, Adam Kochanski, Rod Linn, Yongqiang Liu, Brian Potter, William Mell, Danielle Tanzer, Shawn Urbanski, and Adam Watts. 2019.  The fire and smoke model evaluation experiment—a plan for integrated, large fire–atmosphere field campaigns.  _Atmosphere. 10(2): 66-._ 10, 2 (2019).  https://doi.org/10.3390/atmos10020066
  * Roy et al. (2006) David P. Roy, S. N. Trigg, Roy Bhima, Bruce H. Brockett, Opha Pauline Dube, Philip E. Frost, Navashni Govender, Tobias Landmann, Johan Le Roux, Tsepo Lepono, Joaquim Macuacua, Cheikh Mbow, K. L. Mhwandangara, Belda Mosepele, Onisimo Mutanga, Gosiame Neo-Mahupeleng, Mario Norman, and S. Virgilo. 2006\.  The utility of satellite fire product accuracy Information-perspectives and recommendations from the Southern Africa fire network.  _IEEE Transactions on Geoscience and Remote Sensing_ 44 (2006), 1928–1930.
  * Singh et al. (2021) Priya Singh, Chandra Sabnani, and Vijay Kapse. 2021\.  Hotspot Analysis of Structure Fires in Urban Agglomeration: A Case of Nagpur City, India.  _Fire_ 4 (07 2021), 38.  https://doi.org/10.3390/fire4030038
  * Stefanidou et al. (2008) Maria Stefanidou, Sotiris Athanaselis, and Chara Spiliopoulou. 2008\.  Health Impacts of Fire Smoke Inhalation.  _Inhalation toxicology_ 20 (07 2008), 761–6.  https://doi.org/10.1080/08958370801975311
  * Stephens et al. (2001) G. Stephens, D. McNamara, R. Fennimore, B. Ramsay, and Mark Ruminski. 2001.  NOAA’s Improved Fire and Smoke Analysis, A Global Disaster Information Network Initiative.  _AGU Spring Meeting Abstracts_ -1 (04 2001).
  * Toon et al. (2010) Michael Toon, Marc Maybauer, John Greenwood, Dirk Maybauer, and John Fraser. 2010.  Management of acute smoke inhalation injury.  _Critical care and resuscitation : journal of the Australasian Academy of Critical Care Medicine_ 12 (03 2010), 53–61.
  * (34) Christopher (USA, Vicente Barros, Michael Mastrandrea, Katharine (USA, Mohamed Abdrabo, W. Adger, Yury Federation, O. Anisimov, Doug Arent, Jonathon (Australia, Virginia (USA, Rongshuo (China, Monalisa (USA/India, Stewart Cohen, Wolfgang (Germany/France, Purnamita Dasgupta, Debra Davidson, Fatima Denton, Petra Doell, and Gary Yohe. 2014\.  _Climate change 2014: impacts, adaptation, and vulnerability – IPCC WGII AR5 summary for policymakers_.  1–32.
  * Vrabi et al. (2018) Rok Vrabi, John Ahmet Erkoyuncu, Peter Butala, and Rajkumar Roy. 2018.  Digital twins: Understanding the added value of integrated models for through-life engineering services.  _Procedia Manufacturing_ 16 (2018), 139–146.
  * Wang et al. (2010) S.-J Wang, C. Xu, P. Yuan, and Y.-Y Wang. 2010\.  Research progress of realistic fluid simulation at home and abroad.  22 (02 2010), 280–286.
  * Wu et al. (2020) X. Wu, R. C. Nethery, M. B. Sabath, D. Braun, and F. Dominici. 2020.  Air pollution and covid-19 mortality in the United States: Strengths and limitations of an ecological regression analysis.  _Science Advances_ 6, 45 (2020).  https://doi.org/10.1126/sciadv.abd4049
  * Xie et al. (2018) You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. 2018\.  TempoGAN: A Temporally Coherent, Volumetric GAN for Super-Resolution Fluid Flow.  _ACM Trans. Graph._ 37, 4, Article 95 (jul 2018), 15 pages.  https://doi.org/10.1145/3197517.3201304
  * Zhao et al. (2022) Jiaming Zhao, Zhisheng Xu, Houlin Ying, Xueqi Guan, Kunkun Chu, Sylvain Marcial Sakepa Tagne, and Haowen Tao. 2022\.  Study on smoke spread characteristic in urban interval tunnel fire.  _Case Studies in Thermal Engineering_ 30 (2022), 101755.  https://doi.org/10.1016/j.csite.2022.101755

##  Appendix A Online Resources

To help others use our model, we have made all source code developed publicly
available on GitHub, along with documentation. We have divided the work into
the following three repositories.

FireIncidentFrontend

https://github.com/UrbanInfoLab/FireIncidentFrontend

This repository contains strictly the front-end files required to run our
model. The 2D and 3D front-ends are included, both being statically served via
JavaScript libraries through MapBox and Leaflet. One can also find the code to
display both the 2D (.KML) and 3D (.FBX) predicted smoke outputs in their
browser with accurate positioning on a geographic map.

FireIncidentBackend

https://github.com/UrbanInfoLab/FireIncidentBackend

This repository contains strictly the back-end files required to run our
model. The base back-end finds itself in a curated list of executable events
to fetch, clean, and synthesize data on an hourly basis from chosen fire
departments. Our custom implementation of the VSmoke algorithm is also
included, alongside our MantaFlow algorithm, with proper usage examples.

FireIncidentData

https://github.com/UrbanInfoLab/FireIncidentData

This repository contains all accumulated data from running our model for the
last year. Multiple cities, like Los Angeles, Houston, Seattle, and others are
included in addition to Austin, and all data is collected and updated on a
nightly basis to the repository. We also provide access to an API integrating
with our live server in case the data is needed on a real-time basis.


# A Relation-Oriented Clustering Method for Open Relation Extraction

Jun Zhao1, Tao Gui2111Corresponding authors., Qi Zhang1111Corresponding
authors., Yaqian Zhou1
1School of Computer Science, Shanghai Key Laboratory of Intelligent
Information Processing,
Fudan University, Shanghai, China
2Institute of Modern Languages and Linguistics, Fudan University
<EMAIL_ADDRESS>
###### Abstract

The clustering-based unsupervised relation discovery method has gradually
become one of the important methods of open relation extraction (OpenRE).
However, high-dimensional vectors can encode complex linguistic information
which leads to the problem that the derived clusters cannot explicitly align
with the relational semantic classes. In this work, we propose a relation-
oriented clustering model and use it to identify the novel relations in the
unlabeled data. Specifically, to enable the model to learn to cluster
relational data, our method leverages the readily available labeled data of
pre-defined relations to learn a relation-oriented representation. We minimize
distance between the instance with same relation by gathering the instances
towards their corresponding relation centroids to form a cluster structure, so
that the learned representation is cluster-friendly. To reduce the clustering
bias on predefined classes, we optimize the model by minimizing a joint
objective on both labeled and unlabeled data. Experimental results show that
our method reduces the error rate by 29.2% and 15.7%, on two datasets
respectively, compared with current SOTA methods.

##  1 Introduction

Relation extraction (RE), a crucial basic task in the field of information
extraction, is of the utmost practical interest to various fields including
web search (Xiong et al., 2017), knowledge base completion (Bordes et al.,
2013), and question answering (Yu et al., 2017). However, conventional RE
paradigms such as supervision and distant supervision are generally designed
for pre-defined relations, which cannot deal with new emerging relations in
the real world.

Figure 1: Although both instances $\mathcal{S}_{2}$ and $\mathcal{S}_{3}$
express founded relation while $\mathcal{S}_{1}$ expresses CEO relation, the
distance between $\mathcal{S}_{1}$ and $\mathcal{S}_{2}$ is still smaller than
that between $\mathcal{S}_{2}$ and $\mathcal{S}_{3}$. This is because there
may be more similar surface information (e.g. word overlapping) or syntactic
structure between $\mathcal{S}_{1}$ and $\mathcal{S}_{2}$, thus the derived
clusters cannot explicitly align with relations.

Under this background, open relation extraction (OpenRE) has been widely
studied for its use in extracting new emerging relational types from open-
domain corpora. The approaches used to handle open relations roughly fall into
one of two groups. The first group is open information extraction (OpenIE)
(Etzioni et al., 2008; Yates et al., 2007; Fader et al., 2011), which directly
extracts related phrases as representations of different relational types.
However, if not properly canonicalized, the extracted relational facts can be
redundant and ambiguous. The second group is unsupervised relation discovery
(Yao et al., 2011; Shinyama and Sekine, 2006; Simon et al., 2019). In this
type of research, much attention has been focused on unsupervised clustering-
based RE methods, which cluster and recognize relations from high-dimensional
representations (Elsahar et al., 2017). Recently, the self-supervised signals
in pretrained language model are further exploited for clustering optimization
(Hu et al., 2020).

However, many studies show that high-dimensional embeddings can encode complex
linguistic information such as morphological (Peters et al., 2018), local
syntactic (Hewitt and Manning, 2019), and longer range semantic information
(Jawahar et al., 2019). Consequently, the distance of representation is not
completely consistent with relational semantic similarity. Although Hu et al.
(2020) use self-supervised signals to optimize clustering, there is still no
guarantee that the learned clusters will explicitly align with the desired
relational semantic classes (Xing et al., 2002). As shown in Figure 1, we use
the method proposed by Hu et al. (2020) to get the instance representations.
Although both instances $\mathcal{S}_{2}$ and $\mathcal{S}_{3}$ express the
founded relation, the euclidean distance between them is larger than that
between $\mathcal{S}_{1}$ and $\mathcal{S}_{2}$, which express different
relation. Obviously, the clustering algorithm tends to group instances
$\mathcal{S}_{1}$ and $\mathcal{S}_{2}$ together, rather than
$\mathcal{S}_{2}$ and $\mathcal{S}_{3}$ which express the same relation.

In this work, we propose a relation-oriented clustering method. To enable the
model to learn to cluster relational data, pre-defined relations and their
existing labeled instances are leveraged to optimize a non-linear mapping,
which transforms high-dimensional entity pair representations into relation-
oriented representations. Specifically, we minimize distance between the
instances with same relation by gathering the instances representation towards
their corresponding relation centroids to form the cluster structure, so that
the learned representation is cluster-friendly. In order to reduce the
clustering bias on the predefined classes, we iteratively train the entity
pair representations by optimizing a joint objective function on the labeled
and unlabeled subsets of the data, improving both the supervised
classification of the labeled data, and the clustering of the unlabeled data.
In addition, the proposed method can be easily extended to incremental
learning by classifying the pre-defined and novel relations with a unified
classifier, which is often desirable in real-world applications. Our
experimental results show that our method outperforms current state-of-the-art
methods for OpenRE. Our codes are publicly available at
Github***https://github.com/Ac-Zyx/RoCORE..

To summarize, the main contributions of our work are as follows: (1) we
propose a novel relation-oriented clustering method RoCORE to enable model to
learn to cluster relational data; (2) the proposed method achieves the
incremental learning of unlabeled novel relations, which is often desirable in
real-world applications; (3) experimental results show that our method reduces
the error rate by 29.2% and 15.7%, on two real-world datasets respectively,
compared with current state-of-the-art OpenRE methods.

Figure 2: Overview of our RoCORE method. At the first step, we encode both the
labeled and unlabeled instances in to entity pair representations. Then the
entity pair representations are transformed to relation-oriented
representations by gathering towards their relational centroids in the second
step. Finally, based on the pseudo labels generated by clustering on unlabeled
data, we optimize the entity pair representations and classifier by minimizing
a joint objective function to reduce the clustering bias on predefined
classes. The above three steps are performed iteratively to gradually improve
model performance on novel relations.

##  2 Related Work

Open Relation Extraction. To meet the needs of extracting new emerging
relation types, many efforts have been undertaken to exploring methods for
open relation extraction (OpenRE). The first line of research is Open
Information Extraction (Etzioni et al., 2008; Yates et al., 2007; Fader et
al., 2011), in which relation phrases are extracted directly to represent
different relation types. However, using surface forms to represent relations
results in an associated lack of generality since many surface forms can
express the same relation. Recently, unsupervised clustering-based RE methods
is attracting lots of attentions. Elsahar et al. (2017) proposed to extract
and cluster open relations by re-weighting word embeddings and using the types
of named entities as additional features. Hu et al. (2020) proposed to exploit
weak, self-supervised signals in pretrained language model for adaptive
clustering on contextualized relational features. However, the self-supervised
signals are sensitive to the initial representation (Gansbeke et al., 2020)
and there is still no guarantee that the learned clusters will align with the
relational semantic classes (Xing et al., 2002). Wu et al. (2019) proposed the
relation similarity metrics from labeled data, and then transfers the
relational knowledge to identify novel relations in unlabeled data. Different
from them, we propose a relation-oriented method explicitly clustering data
based on relational information.

Knowledge in High-Dimensional Vector. Pre-trained static and contextual word
representations can provide valuable prior knowledge for constructing
relational representations (Soares et al., 2019; Elsahar et al., 2017). Peters
et al. (2018) showed that different neural architectures (e.g., LSTM, CNN, and
Transformers) can hierarchically structure linguistic information that varies
with network depth. Recently, many studies (Jawahar et al., 2019; Clark et
al., 2019; Goldberg, 2019) have shown that such hierarchy also exists in
pretraining models like BERT. These results suggest that high-dimensional
embeddings, independent of model architecture, learn much about the structure
of language. Directly clustering on these high-dimensional embeddings should
hardly produce ideal clusters in our desired way, which motivates us to extend
current unsupervised clustering-based RE methods to learn the representations
tailored for clustering relational data.

##  3 Approach

In this work, we propose a relation-oriented clustering method, which takes
advantage of the relational information in the existing labeled data to enable
model to learn to cluster relational data. In order to reduce the clustering
bias on the predefined classes, we iteratively train the entity pair
representations by optimizing a joint objective function on the labeled and
unlabeled subsets of the data, improving both the supervised classification of
the labeled data, and the clustering of the unlabeled data. The proposed
method is shown in Figure 2.

Specifically, given an unlabeled dataset
$\mathcal{D}^{u}=\\{s_{i}^{u}\\}_{i=1,...,M}$ of relational instances
$s_{i}^{u}$, our goal is to automatically cluster the relational instances
into a number of classes $C^{u}$, which we assume to be known a priori. To
enable the model to learn to cluster data, we incorporate a second labeled
dataset of pre-defined relations
$\mathcal{D}^{l}=\\{(s_{i}^{\ell},y_{i}^{\ell})\\}_{i=1,...,N}$ where
$y_{i}^{\ell}\in\\{1,...,C^{\ell}\\}$ is the relational label for instance
$s_{i}^{\ell}$.

###  3.1 Method Overview

We approach the problem by learning a relation-oriented representation, from
which the derived clusters can be explicitly aligned with the desired
relational semantic classes. As illustrated in Figure 2, we learn the
representation and optimize the model by performing three iterative steps:

(1) First, we encode relation instances in $\mathcal{D}^{\ell}$ and
$\mathcal{D}^{u}$ using the entity pair encoder implemented as the pretrained
BERT (Devlin et al., 2018), which takes relation instances
$\\{s_{i}^{\ell}\\}_{i=1,...,N}$, and $\\{s_{j}^{u}\\}_{j=1,...,M}$, as input,
and output relation representation $\bm{h_{i}^{\ell}}$, $\bm{h_{j}^{u}}$.
However, high-dimensional $\bm{h}$ can encode a mixture of various aspects of
linguistic features and the derived clusters from $\bm{h}$ cannot explicit
align with desired relational classes.

(2) To make the distance between the representations accurately reflect the
relational semantic similarity, the obtained $\bm{h}_{i}^{\ell}$ are
transformed to low-dimensional relation-oriented representations
$\bm{h}_{i}^{\ell\prime}$ by a non-linear mapping $\bm{g}$. Under the
supervision of labels $y_{i}^{\ell}$ in $\mathcal{D}^{\ell}$, $\bm{g}$ is
optimized by the gathering of $\bm{h}_{i}^{\ell\prime}$ towards their
relational centroids to form a cluster structure, thereby we obtain
$\bm{h}_{j}^{u\prime}$ from unlabeled data using the optimized $\bm{g}$ and
generate the pseudo labels $\hat{y}^{u}$ according to clustering on
$\bm{h}_{i}^{u\prime}$.

(3) Because using labeled data to guide the $\bm{h^{\prime}}$ towards their
relational centroids will produce clustering bias on pre-defined relations, it
is difficult to directly generate high-quality pseudo labels. To reduce the
negative effect of errors in pseudo labels, we optimize classifier and entity
pair representations by minimizing a joint objective function, containing
terms for both pre-defined and novel relations, using respectively the given
labels $y^{\ell}$ and generated pseudo label $\hat{y}^{u}$. Based on the
refined entity pair representation $\bm{h}$ which encode more contextual
relational information, the above three steps are performed iteratively to
gradually improve the quality of pseudo labels $\hat{y}^{u}$ and model
performance.

###  3.2 Entity Pair Encoder

Given a relation instance $s_{i}=(\bm{x}_{i},h_{i},t_{i})$, which consists of
a sentence $\bm{x}_{i}=\\{x_{1},x_{2},...,x_{n}\\}$ and two entity spans
$h_{i}=(s_{h},e_{h})$, $t_{i}=(s_{t},e_{t})$ marking the position of the
entity pair, the entity pair encoder $\bm{f}$ aims to map relation instance
$s_{i}$ to a fixed-length embedding
$\bm{h}_{i}=\bm{f}(s_{i})\in\mathbb{R}^{d}$ that encode contextual information
in $s_{i}$. We adopt BERT (Devlin et al., 2018) as the implemention of our
encoder $\bm{f}$ due to its strong performance on extracting contextual
information. Formally:

$\displaystyle\bm{h}_{1}^{r},...,\bm{h}_{n}^{r}={\rm
BERT}^{r}(x_{1},...,x_{n})$ (1) $\displaystyle\bm{h}_{ent}={\rm
MAXPOOL}(\bm{h}_{s}^{r},...,\bm{h}_{e}^{r})$ (2)
$\displaystyle\bm{h}_{i}=\bm{h}_{head}\oplus\bm{h}_{tail},$ (3)

where $r$ is a hyperparameter that denotes the output layer of BERT. $s$ and
$e$ represent start and end position of the corresponding entity respectively.
$\oplus$ denotes the concatenation operator. This structure of entity pair
representation encoder has been widely used in previous RE methods (Wang et
al., 2021; Hu et al., 2020).

###  3.3 Relation-Oriented Clustering Module

In order to make the distance between representation accurately reflect the
relational semantic similarity, the obtained
$\\{\bm{h}^{\ell}_{i}\\}_{i=1,...,N}$ are transformed to low-dimensional
relation-oriented representations $\bm{h}_{i}^{\ell\prime}$ by a non-linear
mapping $\bm{g}(\cdot):\mathbb{R}^{d}\rightarrow\mathbb{R}^{m}$. Under the
supervision of labels $y_{i}^{\ell}$ in $\mathcal{D}^{\ell}$, $\bm{g}$ is
optimized by the gathering of $\bm{h}_{i}^{\ell\prime}$ towards their
relational centroids as follows:

$\displaystyle\mathcal{L}_{center}=\frac{1}{2N}\sum_{i=1}^{N}\left\|\bm{h^{\ell\prime}}_{i}-\bm{c}_{y_{i}}\right\|_{2}^{2}$
(4)
$\displaystyle\bm{c}_{r}=\frac{1}{|\mathcal{D}_{r}|}\sum_{i\in\mathcal{D}_{r}}\bm{h}_{i}^{\ell\prime},$
(5)

where $\bm{c}_{r}$ denotes the centroids of relation $r$. The center loss
$\mathcal{L}_{center}$ seems reasonable, but problematic. A global optimal
solution to minimize $\mathcal{L}_{center}$ is $g(\bm{h}_{i})=\bm{0}$, which
is far from being desired. This motivates us to incorporate a reconstruction
term to prevent the semantic space from collapsing. Specifically,, a decoding
network $\bm{d}(\cdot)$ is used to map the representation
$\bm{h}_{i}^{\prime}$ back to the original representation $\bm{h}_{i}$.Thus,
we can derive the following loss function:

$\mathcal{L}_{C}=\frac{1}{2N}\sum_{i=1}^{N}\ell(\bm{d}(\bm{h}_{i}^{\prime}),\bm{h}_{i})+\lambda\mathcal{L}_{center},$
(6)

where both the encoder $\bm{g}(\bm{h}_{i})$ and decoder
$\bm{d}(\bm{h}_{i}^{\prime})$ are implemented as DNN. The function
$\ell(\cdot,\cdot):\mathcal{R}^{d}\rightarrow\mathcal{R}$ is the least-squares
loss $\ell(\bm{x},\bm{y})=\left\|\bm{x}-\bm{y}\right\|_{2}^{2}$ that measures
the reconstruction error and other choices such as $\ell_{1}$-norm also can be
considered. $\lambda$ is a hyper-parameter that balances the reconstruction
error versus center loss.

Finally, we obtain $\\{\bm{h}_{j}^{u\prime}\\}_{j=1,...,M}$ using the
optimized $\bm{g}$ and generate pseudo labels $\hat{y}^{u}$ using k-means
algorithm as follows:

$\hat{y}^{u}=\text{ k-means}(\bm{h}^{u\prime}),$ (7)

###  3.4 Relation Classification Module

Based on the pseudo labels $\hat{y}^{u}$ generated by clustering, we can train
the classifier and refine entity pair representation $\bm{h}$ to encode more
contextual relation information. Since it’s difficult to keep the order of
clusters consistent in multiple clustering, instead of using standard cross
entropy loss, we propose to use the pairwise similarities for novel relation
learning.

$q_{ij}=\mathbbm{1}\\{\hat{y}_{i}^{u}=\hat{y}_{j}^{u}\\},$ (8)

where the symbol $q_{ij}$ denotes whether $s_{i}^{u}$ and $s_{j}^{u}$ belong
to the same cluster. If a pair is from the same cluster, the classifier
$\bm{\eta}^{u}:\mathcal{R}^{d}\rightarrow\mathcal{R}^{C^{u}}$ outputs similar
distributions, and vice-versa. Specifically, we use the pair-wise KL-
divergence to evaluate the distance of two relation instances. Given a pair of
instance $s_{i}^{u}$, $s_{j}^{u}$, their corresponding output distributions
are defined as $\mathcal{P}=\bm{\eta}^{u}(\bm{f}(s_{i}^{u}))$ and
$\mathcal{Q}=\bm{\eta}^{u}(\bm{f}(s_{j}^{u}))$. For the pair from the same
cluster, the cost is described as:

$\displaystyle\mathcal{L}^{+}(s_{i}^{u},s_{j}^{u})=\mathcal{D}_{KL}(\mathcal{P^{*}}||\mathcal{Q})+\mathcal{D}_{KL}(\mathcal{Q^{*}}||\mathcal{P})$
(9)
$\displaystyle\mathcal{D}_{KL}(\mathcal{P^{*}}||\mathcal{Q})=\sum_{c=1}^{C^{u}}p_{c}log\frac{p_{c}}{q_{c}},$
(10)

where $\mathcal{P}^{*}$ denotes that $\mathcal{P}$ is assumed to be a constant
and each KL-divergence factor $\mathcal{D}_{KL}(\mathcal{P}||\mathcal{Q})$ is
a unary function whose gradient is simply
$\partial\mathcal{D}_{KL}(\mathcal{P}^{*}||\mathcal{Q})/\partial\mathcal{Q}$.

If $s_{i}^{u}$, $s_{j}^{u}$ comes from different clusters, their output
distributions are expected to be different, which can be defined as a hinge-
loss function:

$\displaystyle\begin{split}\mathcal{L}^{-}(s_{i}^{u},s_{j}^{u})=L_{h}(\mathcal{D}_{KL}(\mathcal{P^{*}}||\mathcal{Q}),\sigma)+\\\
L_{h}(\mathcal{D}_{KL}(\mathcal{Q^{*}}||\mathcal{P}),\sigma)\end{split}$ (11)
$\displaystyle L_{h}(e,\sigma)=\max(0,\sigma-e),$ (12)

and the total loss can be defined as a contrastive loss:

$\begin{split}\mathcal{L}_{BCE}(s_{i}^{u},s_{j}^{u})=q_{i,j}\mathcal{L}^{+}(s_{i}^{u},s_{j}^{u})+\\\
(1-q_{ij})\mathcal{L}^{-}(s_{i}^{u},s_{j}^{u}).\end{split}$ (13)

Note that $\mathcal{L}_{BCE}$ is a symmetric loss w.r.t.
$s_{i}^{u}$,$s_{j}^{u}$ since $\mathcal{P}$ and $\mathcal{Q}$ are
alternatively assumed to be constant in $\mathcal{L}^{+}$ and
$\mathcal{L}^{-}$. Finally, we get the prediction for a relation instance
$s_{i}^{u}$ as follows:

$\hat{y}_{i}^{u}=\arg\max_{y}[\bm{\eta}^{u}(\bm{f}(s_{i}^{u}))]_{y}$ (14)

###  3.5 Training Methods

####  3.5.1 Iterative Joint Training

Because using labeled data to guide the $\bm{h}$ towards their relational
centroids will produce clustering bias on pre-defined relations, it is
difficult to directly generate high-quality pseudo labels $\hat{y}^{u}$ for
novel relations. To reduce the negative effect of errors in pseudo labels, we
incorporate a classifier
$\bm{\eta}^{\ell}:\mathcal{R}^{d}\rightarrow\mathcal{R}^{C^{l}}$ for pre-
defined relations and refine $\bm{h}$ by minimizing a joint objective
function, containing terms for both pre-defined and novel relations, using
respectively the given labels $y^{\ell}$ and generated pseudo label
$\hat{y}^{u}$ as follows:

$\displaystyle\mathcal{L}_{CE}=-\frac{1}{N}\sum_{i=1}^{N}log\bm{\eta}^{\ell}_{y_{i}}(\bm{h_{i}})$
(15) $\displaystyle\mathcal{L}_{CLS}=\mathcal{L}_{CE}+\mathcal{L}_{BCE}.$ (16)

The refined entity pair representation $\bm{h}$ encode more contextual
relation information, which in turn promote clustering optimization and
generate pseudo labels $\hat{y}^{u}$ with higher accuracy. We refine
representation $\bm{h}$ and optimize clustering in a iterative manner to
gradually improve the quality of the pseudo labels and model performance. This
iterative procedure is detailed in Algorithm 1.

Input: novel relation dataset $\mathcal{D}^{u}=\\{s_{j}^{u}\\}$,

predefined relation dataset
$\mathcal{D}^{\ell}=\\{(s_{i}^{\ell},y_{i}^{\ell})\\}$,

model parameters $\Theta$, $\Phi$, $\Psi$ for Entity pair encoder, Relation-
oriented clustering module, Relation classifiers, respectively,

and learning rate $\eta$.

1 for  _$epoch\leftarrow 1$ to $L$_ do

2 Pre-train clustering network by minimize reconstruction loss
$\Phi=\Phi-\eta\nabla_{\Phi}\ell(\bm{g}(\bm{h}_{i}^{\prime}),\bm{h}_{i})$;

3

4 end for

5repeat

6 generate pseudo labels $\hat{y}$ by equation 7;

7 refine entity pair representation

8 $\Theta=\Theta-\eta\nabla_{\Theta}\mathcal{L}_{CLS}$;

9 $\Psi=\Psi-\eta\nabla_{\Psi}\mathcal{L}_{CLS}$;

10 optimize clustering $\Phi=\Phi-\eta\nabla_{\Phi}\mathcal{L}_{C}$;

11

12until  _convergence_ ;

Algorithm 1 The RoCORE Method

####  3.5.2 Incremental Learning Scheme

In real-world settings, when facing a new sentence, we often don’t know
whether it belongs to pre-defined relations or novel relations. In this work,
we explore the incremental learning of novel relations to enable
$\bm{\eta}^{l}$ to discriminate both pre-defined and novel relations. Under
incremental learning settings, we extend the classifier $\bm{\eta}^{l}$ to
$C^{u}$ novel relation types, so that
$\bm{\eta}^{l}:\mathcal{R}^{d}\rightarrow\mathcal{R}^{C^{l}+C^{u}}$. Then, the
model is trained using cross-entropy loss instead of equation 15 as follows:

$\displaystyle\mathcal{L}_{CE}=$
$\displaystyle-\frac{1}{N}\sum_{i=1}^{N}log\bm{\eta}^{\ell}_{y_{i}}(\bm{h}_{i})$
(17)
$\displaystyle-\frac{\mu(t)}{M}\sum_{j=1}^{M}log\bm{\eta}^{\ell}_{\hat{y}_{j}}(\bm{h}_{j}),$

where we obtain $\hat{y}_{j}$ using equation 14 and the coefficient $\mu(t)$
balances the cross entropy loss of pre-defined and novel relations. We
implemented it as a ramp-up function $\mu(t)=\mu_{0}e^{-5(1-\frac{t}{T})^{2}}$
where $t$ is current epoch and $T$ is the ramp-up length and coefficient
$\mu_{0}\in\mathbb{R}^{+}$.

##  4 Experimental Setup

In this section, we describe the datasets for training and evaluating the
proposed method. We also detail the baseline models for comparison. Finally,
we clarify the implementation details and hyperparameter configuration of our
method.

###  4.1 Datasets

We conduct experiments on two relation extraction datasets.

FewRel. Few-Shot Relation Classification Dataset (Han et al., 2018). FewRel is
a human-annotated dataset containing 80 types of relations, each with 700
instances. We follow the setting in (Wu et al., 2019) to use the original
train set of FewRel, which contains 64 relations, as labeled set with
predefined relations, and the original validation set of FewRel, which
contains 16 new relations, as the unlabeled set with novel relations to
extract. 1,600 instances were randomly selected from the unlabeled set as the
test set. The rest of labeled and unlabeled instances are considered as the
train set.

TACRED. The TAC Relation Extraction Dataset (Zhang et al., 2017). TACRED is a
human-annotated large-scale relation extraction dataset that covers 41
relation types. We remove the instances labeled as no_relation and use the
remaining 21,773 instances for training and evaluation. Similar to the setting
of FewRel, we select the 0-30 relation types as labeled set with pre-defined
relations and the 31-40 relation types as unlabeled set with novel relations.
We randomly selected 15% of the instances from the unlabeled set as the test
set. The rest of the labeled and unlabeled instances are considered as the
train set.

###  4.2 Compared Methods

To evaluate the effectiveness of our method, we select the following SOTA
OpenRE models for comparison. Note that the first four methods are
unsupervised and RSN as well as RSN-BERT leverages labeled data of predefined
relations.

HAC with Re-weighted Word Embeddings (RW-HAC) (Elsahar et al., 2017). RW-HAC
is a feature clustering method for OpenRE. The model contructs relational
feature based on the weighted word embeddings as well as entity types.

Discrete-state Variational Autoencoder (VAE) (Marcheggiani and Titov, 2016).
VAE is a reconstruction-based method for OpenRE. The model is optimized by
reconstructing entities from pairing entities and predicted relations.

Entity Based URE (Etype+) (Tran et al., 2020). Etype+ is a simple and
effective method relying only on entity types. The same link predictor as in
(Marcheggiani and Titov, 2016) is employed and two additional regularisers are
used.

Self-supervised Feature Learning for OpenRE (SelfORE) (Hu et al., 2020).
SelfORE exploits weak, self-supervised signals by leveraging large pretrained
language model for adaptive clustering on contextualized relational features.

Relational Siamese Network (RSN) (Wu et al., 2019). This method learns
similarity metrics of relations from labeled data of pre-defined relations,
and then transfer the relational knowledge to identify novel relations in
unlabeled data.

RSN with BERT Embedding (RSN-BERT). A variant of RSN, the static word vector
is replaced by the BERT embedding for fair comparison.

###  4.3 Implementation Details

Our entity pair encoder is implemented as the  bert-base-uncased which
consists of 12 layers and we use layer 8 as the output layer for best
performance. Note that we only fine-tune the parameters of the output layer in
the iterative training process to avoid overfitting. Non-linear mapping
$\bm{g}(\cdot)$ and $\bm{d}(\cdot)$ are both implemented as a DNN with relu
activation, specifically $\mathbb{R}^{d}$-512-512-256 for $\bm{g}(\cdot)$ and
256-512-512-$\mathbb{R}^{d}$ for $\bm{d}(\cdot)$. All experiments are
conducted using a GeForce GTX 1080Ti with 11GB memory and table 1 shows our
best hyper-parameter settings.

Hyper-parameters | value
---|---
optimizer | Adam
learning rate | 1e-4
batch size | 100
pre-training epochs $L$ | 10
BCE loss coefficient $\sigma$ | 2
center loss coefficient $\lambda$ for FewRel | 0.005
center loss coefficient $\lambda$ for TACRED | 0.001
ramp-up coefficient $\mu_{0}$ | 1.0
ramp-up length $T$ | 10
Table 1: Hyper-parameter settings.

Dataset | Method | $B^{3}$ | V-measure | ARI
---|---|---|---|---
Prec. | Rec. | $F_{1}$ | Hom. | Comp. | $F_{1}$
FewRel | VAE(Marcheggiani and Titov, 2016) | 0.309 | 0.446 | 0.365 | 0.448 | 0.500 | 0.473 | 0.291
RW-HAC(Elsahar et al., 2017) | 0.256 | 0.492 | 0.337 | 0.391 | 0.485 | 0.433 | 0.250
EType+(Tran et al., 2020) | 0.238 | 0.485 | 0.319 | 0.364 | 0.463 | 0.408 | 0.249
SelfORE(Hu et al., 2020) | 0.672 | 0.685 | 0.678 | 0.779 | 0.788 | 0.783 | 0.647
RSN(Wu et al., 2019) | 0.486 | 0.742 | 0.589 | 0.644 | 0.787 | 0.708 | 0.453
RSN-BERT | 0.585 | 0.899 | 0.709 | 0.696 | 0.889 | 0.781 | 0.532
RoCORE | $0.752_{17}$ | $0.846_{09}$ | $\textbf{0.796}_{11}$ | $0.838_{10}$ | $0.883_{06}$ | $\textbf{0.860}_{07}$ | $\textbf{0.709}_{23}$
TACRED | VAE(Marcheggiani and Titov, 2016) | 0.247 | 0.564 | 0.343 | 0.208 | 0.362 | 0.264 | 0.159
RW-HAC(Elsahar et al., 2017) | 0.426 | 0.633 | 0.509 | 0.469 | 0.597 | 0.526 | 0.281
EType+(Tran et al., 2020) | 0.302 | 0.803 | 0.439 | 0.260 | 0.607 | 0.364 | 0.143
SelfORE(Hu et al., 2020) | 0.576 | 0.510 | 0.541 | 0.630 | 0.608 | 0.619 | 0.447
RSN(Wu et al., 2019) | 0.628 | 0.634 | 0.631 | 0.624 | 0.663 | 0.643 | 0.459
RSN-BERT | 0.795 | 0.878 | 0.834 | 0.849 | 0.870 | 0.859 | 0.756
RoCORE | $0.871_{42}$ | $0.849_{37}$ | $\textbf{0.860}_{35}$ | $0.895_{32}$ | $0.881_{20}$ | $\textbf{0.888}_{24}$ | $\textbf{0.812}_{64}$

Table 2: Main results on two relation extraction datasets. The subscript
represents the corresponding standard deviation (e.g., $0.796_{11}$ indicates
$0.796\pm 0.011$). Experimental results show that our method reduces the error
rate by 29.2%(0.709$\rightarrow$0.796) and 15.7%(0.834$\rightarrow$0.860), on
two datasets respectively.

##  5 Results and Analysis

In this section, we present the experimental results of our model on two real-
world datasets to demonstrate the effectiveness of our method. We also provide
additional experimental results on hyper-parameter analysis and relation
representation visualization in appendix A and B.

###  5.1 Main Results

Table 2 reports model performances on FewRel, TACRED dataset, which shows that
the proposed method achieves state-of-the-art results on OpenRE task.
Benefitting from the valuable information in the labeled instances of pre-
defined relations, RoCORE effectively learns the relation-oriented
representation from which the derived clusters explicitly align with
relational semantic classes, thereby outperforming previous clustering-based
baseline such as SelfORE by a large margin. In addition, despite the fact that
RSN and its variant RSN-BERT also leverage relational information in labeled
data, the learning of similarity metrics and clustering are mutually
independent. In our method, relation representation learning and cluster
optimization are mutually dependent. Thus, the learned representations are
tailored for clustering. As a result, our method outperforms RSN and RSN-BERT
on the two datasets.

###  5.2 Ablation Study

Dataset | Method | Prec. | Rec. | $F_{1}$
---|---|---|---|---
FewRel | w/o center loss | $0.726_{32}$ | $0.774_{31}$ | $0.749_{31}$
w/o reconstruction | $0.512_{38}$ | $0.573_{17}$ | $0.540_{25}$
w/o CE | $0.662_{67}$ | $0.787_{54}$ | $0.719_{47}$
RoCORE | $0.752_{17}$ | $0.846_{09}$ | $\textbf{0.796}_{11}$
TACRED | w/o center loss | $0.818_{57}$ | $0.842_{37}$ | $0.830_{41}$
w/o reconstruction | $0.549_{35}$ | $0.483_{30}$ | $0.514_{31}$
w/o CE | $0.706_{45}$ | $0.776_{51}$ | $0.739_{47}$
RoCORE | $0.871_{42}$ | $0.849_{37}$ | $\textbf{0.860}_{35}$

Table 3: Abalation study of our method. This table only lists the results of
metric $B^{3}$. For results of other metrics, please refer to the Table 5 in
Appendix C.

To study the contribution of each component in the proposed method, we conduct
ablation experiments on the two datasets and display the results in Table 3.
The results show that the model performance is degraded if
$\mathcal{L}_{center}$ is removed, indicating that the guidance of supervision
signals from pre-defined relations provide valuable information for learning
the relation-oriented representations. It is worth noting that the
reconstruction term has an important role in the clustering module. Without
the reconstruction term, the semantic space will collapse and the performance
will be seriously hurt. In addition, joint optimizing on both the labeled and
unlabeled data is also very important. The initial pseudo labels for novel
relations are not accurate due to the unwanted clustering bias on pre-defined
relations. Without $\mathcal{L}_{CE}$, the error in pseudo labels will lead
the refinement of the entity pair representation to a wrong direction, which
affects the model performance.

###  5.3 The Influence of Pre-defined Relation Number on Performance

Figure 3: Clustering results with different numbers of pre-defined training
relations.

In this subsection, we conduct experiments on two different datasets to
explore the influence of pre-defined relation number on performance of our
method. For FewRel dataset, following the setting in (Wu et al., 2019), we
change the number of pre-defined relations from 40 to 64 while fixing the
total number of labeled instances to 25,000. Similarly, the settings for
TACRED dataset is 18, 31 and 12, 000, respectively.

From figure 3 we can see the following: (1) The increase of pre-defined
relation number do improve the generalization of our method on novel
relations. The models trained on 64/31 relations slightly perform better than
the models trained on 40/18 relations on FewRel/TACRED dataset (2) Our method
constantly performs better than RSN and RSN-BERT with the number of predefined
relations vary. This indicates the effectiveness of our method.

###  5.4 Cross Domain Analysis

Task | Method | Prec. | Rec. | $F_{1}$
---|---|---|---|---
$\textbf{F}\rightarrow\textbf{T}$ | RSN | 0.349 | 0.590 | 0.439
RSN-BERT | 0.337 | 0.866 | 0.486
RoCORE | $0.621_{28}$ | $0.602_{51}$ | $\textbf{0.611}_{34}$
$\textbf{T}\rightarrow\textbf{F}$ | RSN | 0.225 | 0.529 | 0.316
RSN-BERT | 0.261 | 0.861 | 0.400
RoCORE | $0.687_{36}$ | $0.766_{46}$ | $\textbf{0.724}_{26}$

Table 4:  Results on two constructed cross-domain tasks. F means FewRel, which
is from encyclopedia domain. T means TACRED, which is from news and web
domain. This table only lists the results of metric $B^{3}$. For results of
other metrics, please refer to the Table 6 in Appendix C.

In real-world settings, pre-defined relations and novel relations of interest
usually come from different domains. To study the model performance in cross-
domain settings, we conducted experiments on two cross-domain tasks, i.e,:
FewRel to TACRED and TACRED to FewRel. Pre-defined relations and their labeled
instances come from the source domain training dataset, and we evaluate
performance on the target domain testing dataset.

Table 4 shows the experimental results, from which we can observe that: (1)
the change of domain increases the semantic gap between the pre-defined and
novel relations. As the result of that, the performance of the model using
labeled data of predefined relations is degraded. (2) compared with RSN and
RSN-BERT, our method shows better generalization performance on novel
relations, which shows that our proposed iterative joint training method
effectively reduces the unwanted bias on source domain labeled data. (3) In
addition, when a model has the tendency to cluster multiple relation into one,
an unbalanced PR value (i.e., high rec. and low prec. in RSN-BERT) will be
produced, which is undesired in real-world applications.

###  5.5 Incremental Learning of Novel Relations

Figure 4: Model performance with different amounts of labeled data.

In this subsection, we evaluate the effectiveness of our incremental learning
scheme and explore the influence of the amount of labeled data on model
performance. We use BERT with a linear softmax classifer as the baseline for
comparison. We train the baseline model using the labeled data of both pre-
defined and novel relations, following the supervised learning paradigm. For
our method, we still use only the labels of pre-defined relations.

From figure 4 we can observe the following: (1) The performance of the models
improve gradually as labeled data increase. Our method can still maintain good
performance when there is a lack of labeled data. This indicates that the
proposed method is robust to the reduction of labeled data. (2) Our method
achieves similar performance compared with the supervised baseline on two
experiments, which use 40% labels of novel relations on FewRel dataset and 82%
on TACRED respectively. It indicates that we successfully achieve the
incremental learning of novel relations.

##  6 Conclusions

In this work, we introduce a relation-oriented clustering method that extends
the current unsupervised clustering-based OpenRE method. The proposed method
leverages the labeled data of pre-defined relations to learn a relation-
oriented representation from which the derived clusters explicitly align with
relational classes. Iterative joint training method effectively reduces the
unwanted bias on labeled data. In addition, the proposed method can be easily
extended to incremental learning of novel relations. Experimental results show
that our method outperforms SOTA methods for OpenRE.

## Acknowledgements

The authors wish to thank the anonymous reviewers for their helpful comments.
This work was partially funded by China National Key R&D Program (No.
2018YFB1005104), National Natural Science Foundation of China (No. 62076069,
61976056), Shanghai Municipal Science and Technology Major Project
(No.2021SHZDZX0103).

## References

  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston, and Oksana Yakhnenko. 2013.  Translating embeddings for modeling multi-relational data.  In _Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’13, page 2787–2795, Red Hook, NY, USA. Curran Associates Inc.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? an analysis of bert’s attention.  _CoRR_ , abs/1906.04341.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Elsahar et al. (2017) Hady Elsahar, Elena Demidova, Simon Gottschalk, Christophe Gravier, and Frederique Laforest. 2017.  Unsupervised open relation extraction.  In _The Semantic Web: ESWC 2017 Satellite Events_ , pages 12–16, Cham. Springer International Publishing.
  * Etzioni et al. (2008) Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S. Weld. 2008.  Open information extraction from the web.  _Commun. ACM_ , 51(12):68–74.
  * Fader et al. (2011) Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011.  Identifying relations for open information extraction.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 1535–1545, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Gansbeke et al. (2020) Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. 2020.  Scan: Learning to classify images without labels.
  * Goldberg (2019) Yoav Goldberg. 2019.  Assessing bert’s syntactic abilities.  _CoRR_ , abs/1901.05287.
  * Han et al. (2018) Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018.  FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4803–4809, Brussels, Belgium. Association for Computational Linguistics.
  * Hewitt and Manning (2019) John Hewitt and Christopher D. Manning. 2019.  A structural probe for finding syntax in word representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hu et al. (2020) Xuming Hu, Chenwei Zhang, Yusong Xu, Lijie Wen, and Philip S. Yu. 2020.  Selfore: Self-supervised relational feature learning for open relation extraction.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Marcheggiani and Titov (2016) Diego Marcheggiani and Ivan Titov. 2016.  Discrete-state variational autoencoders for joint discovery and factorization of relations.  _Transactions of the Association for Computational Linguistics_ , 4:231–244.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018.  Dissecting contextual word embeddings: Architecture and representation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1499–1509, Brussels, Belgium. Association for Computational Linguistics.
  * Shinyama and Sekine (2006) Yusuke Shinyama and Satoshi Sekine. 2006.  Preemptive information extraction using unrestricted relation discovery.  In _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_ , pages 304–311, New York City, USA. Association for Computational Linguistics.
  * Simon et al. (2019) Étienne Simon, Vincent Guigue, and Benjamin Piwowarski. 2019.  Unsupervised information extraction: Regularizing discriminative approaches with relation distribution losses.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1378–1387, Florence, Italy. Association for Computational Linguistics.
  * Soares et al. (2019) Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019\.  Matching the blanks: Distributional similarity for relation learning.  _CoRR_ , abs/1906.03158.
  * Tran et al. (2020) Thy Thy Tran, Phong Le, and Sophia Ananiadou. 2020.  Revisiting unsupervised relation extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7498–7505, Online. Association for Computational Linguistics.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-sne.  _Journal of Machine Learning Research_ , 9(86):2579–2605.
  * Wang et al. (2021) Yijun Wang, Changzhi Sun, Yuanbin Wu, Hao Zhou, Lei Li, and Junchi Yan. 2021.  ENPAR:enhancing entity and entity pair representations for joint entity relation extraction.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2877–2887, Online. Association for Computational Linguistics.
  * Wu et al. (2019) Ruidong Wu, Yuan Yao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. 2019.  Open relation extraction: Relational knowledge transfer from supervised data to unsupervised data.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 219–228, Hong Kong, China. Association for Computational Linguistics.
  * Xing et al. (2002) Eric P. Xing, Andrew Y. Ng, Michael I. Jordan, and Stuart Russell. 2002.  Distance metric learning, with application to clustering with side-information.  In _Proceedings of the 15th International Conference on Neural Information Processing Systems_ , NIPS’02, page 521–528, Cambridge, MA, USA. MIT Press.
  * Xiong et al. (2017) Chenyan Xiong, Russell Power, and Jamie Callan. 2017.  Explicit semantic ranking for academic search via knowledge graph embedding.  In _Proceedings of the 26th International Conference on World Wide Web_ , WWW ’17, page 1271–1279, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Yao et al. (2011) Limin Yao, Aria Haghighi, Sebastian Riedel, and Andrew McCallum. 2011.  Structured relation discovery using generative models.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 1456–1466, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Yates et al. (2007) Alexander Yates, Michele Banko, Matthew Broadhead, Michael Cafarella, Oren Etzioni, and Stephen Soderland. 2007.  TextRunner: Open information extraction on the web.  In _Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT)_ , pages 25–26, Rochester, New York, USA. Association for Computational Linguistics.
  * Yu et al. (2017) Mo Yu, Wenpeng Yin, Kazi Saidul Hasan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2017.  Improved neural relation detection for knowledge base question answering.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 571–581, Vancouver, Canada. Association for Computational Linguistics.
  * Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017.  Position-aware attention and supervised data improve slot filling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 35–45, Copenhagen, Denmark. Association for Computational Linguistics.

##  Appendix A Hyperparameter Analysis

Figure 5: Model performance with different $\lambda$.

From the experimental results of ablation study, it can be seen that
reconstruction loss and center loss have a great impact on the performance of
the model. $\lambda$ is a key hyperparameter that balances the reconstruction
loss versus center loss. In this section, we conduct experiments to study the
influence of the value of $\lambda$ on the performance of the model. From
Figure 5 we can see that: (1) When $\lambda$ gradually increases from 0, the
center loss begins to affect the optimization. The model learns that instances
with the same relation should be mapped to relatively close positions in the
representation space, and the performance of the model gradually improves. (2)
When the lambda exceeds a certain threshold, further increasing the $\lambda$
will leads to unwanted bias to the predefined relations, which will degrade
the performance of the model.

##  Appendix B Relation Representation Visualization

Figure 6: Visualization of the relation representation after t-SNE dimension
reduction. The representations are colored with their ground-truth relation
labels. These three sequentially illustrate the feature representation of
initial sate, after reconstruction pre-training, and after training. All
figures visualize the clustering result for 600 instances of randomly selected
6 novel relations on FewRel test dataset.

To intuitively show how the RoCORE method learns the constantly optimized
relation-oriented representation, we visualize the relational representation
with t-SNE (van der Maaten and Hinton, 2008). The visualization results are
shown in Figure 6. It is apparent that, before training (left), the relational
representations are distributed randomly at different locations in the
semantic space. After pre-training (middle), the relational representations
still are not tailored for the relations. For example, the instances with blue
and light green colors may have similar syntactic or surface features and
clustering them directly will lead to a poor result. After training (right),
the relational representations are well separated and the distribution is
based on relation types.

##  Appendix C Detailed Results of Other Experiments

In this section, the detailed results of ablation experiments and cross domain
analysis are listed in Table 5 and Table 6 respectively.

Dataset | Method | $B^{3}$ | V-measure | ARI
---|---|---|---|---
Prec. | Rec. | $F_{1}$ | Hom. | Comp. | $F_{1}$
FewRel | w/o center loss | $0.726_{32}$ | $0.774_{31}$ | $0.749_{31}$ | $0.818_{19}$ | $0.842_{18}$ | $0.830_{19}$ | $0.702_{38}$
w/o reconstruction | $0.512_{38}$ | $0.573_{17}$ | $0.540_{25}$ | $0.665_{24}$ | $0.689_{12}$ | $0.676_{16}$ | $0.495_{40}$
w/o CE | $0.662_{67}$ | $0.787_{54}$ | $0.719_{47}$ | $0.772_{41}$ | $0.844_{27}$ | $0.806_{28}$ | $0.617_{68}$
RoCORE | $0.752_{17}$ | $0.846_{09}$ | $\textbf{0.796}_{11}$ | $0.838_{10}$ | $0.883_{06}$ | $\textbf{0.860}_{07}$ | $\textbf{0.709}_{23}$
TACRED | w/o center loss | $0.818_{57}$ | $0.842_{37}$ | $0.830_{41}$ | $0.855_{41}$ | $0.867_{32}$ | $0.864_{32}$ | $0.783_{69}$
w/o reconstruction | $0.549_{35}$ | $0.483_{30}$ | $0.514_{31}$ | $0.589_{37}$ | $0.570_{28}$ | $0.579_{32}$ | $0.393_{54}$
w/o CE | $0.706_{45}$ | $0.776_{51}$ | $0.739_{47}$ | $0.753_{30}$ | $0.803_{37}$ | $0.777_{32}$ | $0.656_{85}$
RoCORE | $0.871_{42}$ | $0.849_{37}$ | $\textbf{0.860}_{35}$ | $0.895_{32}$ | $0.881_{20}$ | $\textbf{0.888}_{24}$ | $\textbf{0.812}_{64}$

Table 5: The detailed results of abalation study. The subscript represents the corresponding standard deviation (e.g., $0.749_{12}$ indicates $0.749\pm 0.012$) Task | Method | $B^{3}$ | V-measure | ARI
---|---|---|---|---
Prec. | Rec. | $F_{1}$ | Hom. | Comp. | $F_{1}$
$\textbf{F}\rightarrow\textbf{T}$ | RSN | 0.349 | 0.590 | 0.439 | 0.387 | 0.533 | 0.448 | 0.279
RSN-BERT | 0.337 | 0.866 | 0.486 | 0.400 | 0.777 | 0.528 | 0.352
RoCORE | $0.621_{28}$ | $0.602_{51}$ | $0.611_{34}$ | $0.642_{37}$ | $0.666_{32}$ | $0.654_{31}$ | $0.451_{65}$
$\textbf{F}\rightarrow\textbf{T}$ | RSN | 0.225 | 0.529 | 0.316 | 0.359 | 0.507 | 0.420 | 0.243
RSN-BERT | 0.261 | 0.861 | 0.400 | 0.438 | 0.822 | 0.571 | 0.263
RoCORE | $0.687_{36}$ | $0.766_{46}$ | $\textbf{0.724}_{26}$ | $0.796_{22}$ | $0.836_{22}$ | $\textbf{0.815}_{16}$ | $\textbf{0.658}_{43}$
Table 6:  The detailed results of cross domain analysis. The subscript
represents the corresponding standard deviation (e.g., $0.724_{26}$ indicates
$0.724\pm 0.026$)