# A spatio-temporal LSTM model to forecast across multiple temporal and
spatial scales

Yihao Hu,1 Fearghal O’Donncha,2 Paulito Palmes,2
Meredith Burke,3 Ramon Filgueira,3 Jon Grant.3

###### Abstract

This paper presents a novel spatio-temporal LSTM (SPATIAL) architecture for
time series forecasting applied to environmental datasets. The framework was
evaluated across multiple sensors and for three different oceanic variables:
current speed, temperature, and dissolved oxygen. Network implementation
proceeded in two directions that are nominally separated but connected as part
of a natural environmental system – across the spatial (between individual
sensors) and temporal components of the sensor data. Data from four sensors
sampling current speed, and eight measuring both temperature and dissolved
oxygen evaluated the framework. Results were compared against RF and XGB
baseline models that learned on the temporal signal of each sensor
independently by extracting the date-time features together with the past
history of data using sliding window matrix. Results demonstrated ability to
accurately replicate complex signals and provide comparable performance to
state-of-the-art benchmarks. Notably, the novel framework provided a simpler
pre-processing and training pipeline that handles missing values via a simple
masking layer. Enabling learning across the spatial and temporal directions,
this paper addresses two fundamental challenges of ML applications to
environmental science: 1) data sparsity and the challenges and costs of
collecting measurements of environmental conditions such as ocean dynamics,
and 2) environmental datasets are inherently connected in the spatial and
temporal directions while classical ML approaches only consider one of these
directions. Furthermore, sharing of parameters across all input steps makes
SPATIAL a fast, scalable, and easily-parameterized forecasting framework.

## Introduction

The coastal ocean represents perhaps the most complex modeling challenge,
intimately connected as it is to the deep ocean and the atmosphere (Song and
Haidvogel 1994). Irregular coastlines, and steep highly variable bottom
topography can generate highly complex patterns of flow; wind forcings produce
both surface and internal waves and contributes to surface flows directly
through wind drift and Ekman transport; tidal forcing in addition to
barotropic flow processes induce internal tides; while freshwater discharges
add buoyancy fluxes, which further complicate local water motion (O’Donncha et
al. 2015). Consequently, the accurate quantification of flow processes in the
ocean is a complex, nonlinear task requiring a holistic consideration of
dynamics and driving forces in both space and time. Further, understanding,
quantifying, and forecasting ocean dynamics is of major importance for
societal and economic factors: we look to the oceans for food, water, and
energy, while ocean-atmosphere dynamics are critical to understanding weather
and climate.

Traditionally, forecasting of ocean processes rely on physics-based approaches
that resolve a set of governing equations such as Navier-Stokes (Constantin
and Foias 1988) or convective-diffusion equations (Stockie 2011). An extensive
body of literature exists on this subject with early investigations on the
topic described in Bryan (1969) and a useful overview provided in Ji (2017).
The primary challenge of physics-based approaches are the immense
computational expense to deploy at high resolution across broad spatial and
temporal scales (generally requiring high performance computing facilities),
and the complexity of configuring and parameterizing the model that typically
requires an expert user (DeYoung et al. 2004).

Machine-learning (ML) approaches to forecast ocean processes are in a nascent
stage and have traditionally been limited by the challenge of data sparsity.
To circumvent this limitation, recent years have seen interest in using ML to
develop low cost approximates or surrogates of physic-based models (e.g. Lary,
Müller, and Mussa (2004); Ashkezari et al. (2016). In this paradigm, there is
the luxury of being able to run the model as many times as necessary to
develop a sufficient data set to train the ML model. A key issue with this
approach is that one is limited by the performance of the physics-based model;
as an example James, Zhang, and O’Donncha (2018) have recently observed a
factor of 12,000 improvement in the speed of obtaining results comparable to
that of the physics model, but because the model itself is utilized to train
the deep-learning network, the accuracy and spatial extents are limited to the
levels dictated by the traditional approaches.

Progress in satellite technology generates large volumes of remotely sensed
observations providing long-term global measurements at varying spatial and
temporal resolution. These datasets provide a natural fit for ML-based
approaches and there is a wide body of literature related to mining and
forecasting ocean processes from satellite observations. Deep Learning (DL)
models have been adopted to increase the resolution of satellite imagery
through down-scaling techniques (Ducournau and Fablet 2016), while a number of
studies have presented data-mining approaches that extract pertinent events
from measured data such as detection of harmful algal blooms (Gokaraju et al.
2011). Some caution is necessary here since satellite estimates of ocean
processes are invariably contaminated by external influence on the optical
properties (Holt et al. 2009; Smyth et al. 2006), while one is naturally
limited to sampling surface processes and cannot penetrate the ocean depth.

This paper investigates a framework that extends model accuracy by explicitly
learning across the spatial and temporal components of a disparate – but
related – time-series signal. We refer to our framework as Spatial-LSTM or
SPATIAL.

SPATIAL is applied to three real-world datasets, namely, ocean current speeds,
temperatures, and dissolved oxygen. Incorporating both physical and
biogeochemical datasets, the performance of the model with different temporal
scales and process memory are evaluated: ocean current patterns are influenced
by external physical drivers such as tidal effects and surface winds stress
together with density driven flows generated by changes in temperature and
salinity; variations in ocean temperature are explained by multiple exogenous
processes such as solar radiation, air temperature, sea bed heat transfer, and
external heat flux from rivers and the open ocean (Pidgeon and Winant 2005);
oceanic oxygen content, on the other hand is a biogeochemical process
influenced by hydrodynamics (horizontal and vertical mixing, residence times,
etc.), weather (temperature reducing the solubility of oxygen), and nutrient
loads (both anthropogenic and organic enrichment) (Caballero-Alfonso,
Carstensen, and Conley 2015). The combination of nonlinear response and
sensitivity to multiple, opaque variables makes the capabilities of DL an
appealing solution for these prediction-requirements.

## Related Work

There is a large number of application papers on using ML for time-series
forecasting. For classical (linear) time series when all relevant information
about the next time step is included in the recent past, multilayer perceptron
(MLP) is typically sufficient to model the short-term temporal dependence
(Gers and Schmidhuber 2001). Compared to MLPs, convolutional neural networks
(CNN) automatically identify and extract features from the input time series.
A convolution can be seen as a sliding window over the data, similar to the
moving average (MA) model but with nonlinear transformation. Different from
MLPs and CNNs, recurrent neural networks (RNN) explicitly allow long-term
dependence in the data to persist over time, which clearly is a desirable
feature for modeling time series. A review of many of these approaches for
time series-based forecasting is provided in Esling and Agon (2012); Gamboa
(2017); Fawaz et al. (2019).

Forecasting of environmental variables relies heavily on physics-based
approaches and a well-established literature exists related to hydro-
environmental modeling and applications to numerous case studies. An excellent
operational forecasting product are the Copernicus Marine Service ocean model
outputs which provide daily forecasts of variables such as current speed,
temperature, and dissolved oxygen (DO) at a 10-day forecasting window (EU
Copernicus 2020). Data are provided at varying horizontal resolution from
approximately $10\text{\,}\mathrm{k}\mathrm{m}$ globally, to regional models
at resolution of up to $3\text{\,}\mathrm{k}\mathrm{m}$. These datasets serve
as a valuable forecasting tool but are generally too coarse for coastal
process studies and for most practical applications (e.g. predicting water
quality properties within a bay) are rescaled to higher resolution using local
regional models (e.g. Gan and Allen (2005).

Due to the heavy computational overhead of physical models, there is an
increasing trend to apply data-driven DL or ML methods to model physical
phenomena (de Bezenac, Pajot, and Gallinari 2017; Wiewel, Becher, and Thuerey
2018). A number of studies have investigated data-driven approaches to provide
computationally cheaper surrogate models, applied to such things as wave
forecasting (James, Zhang, and O’Donncha 2018), viscoelastic earthquake
simulation (DeVries, Thompson, and Meade 2017), and water-quality
investigation (Arandia et al. 2018). In all examples, the ML has been used to
perform non-linear regression between the inputs and outputs.

An alternative approach aims to embed information from physics or heuristic
knowledge within the network. Physics-informed DL is a novel approach for
resolving information from physics. The philosophy behind it is to approximate
the quantity of interest (e.g., governing equation variables) by a deep neural
network (DNN) and embed the physical law to regularize the network. To this
end, training the network is equivalent to minimization of a well-designed
loss function that contains the PDE residuals and initial/boundary conditions
(Rao, Sun, and Liu 2020).

A further stream of related work has been started by Chen et al. (2018), who
presented a novel approach to approximate the discrete series of layers
between the input and output state by acting on the derivative of the hidden
units. At each stage, the output of the network is computed using a black-box
differential equation solver which evaluates the hidden unit dynamics to
determine the solution with the desired accuracy. In effect, the parameters of
the hidden unit dynamics are defined as a continuous function, which may
provide greater memory efficiency and balancing of model cost against problem
complexity. The approach aims to achieve comparable performance to existing
state-of-the-art with far fewer parameters, and suggests potential advantages
for time series modeling. For follow-up work in this stream, see Grathwohl et
al. (2019); Gulgec et al. (2019).

Figure 1: SPATIAL allows multidimensional input, which enables the deep neural
network to extract features from different sensors and exploit the information
learned to predict the time series for every sensor. That’s being said for
each sensor, the prediction is not only based on its previous time series, but
also the information from other sensors.

(a) Classical LSTM Model for single sensor $i$

(b) Unfolded Orthogonal Bidirectional LSTM model

Figure 2: Left figure (a) presents a schematic of a classical LSTM
architecture to predict a single time series signal while (b) presents our
SPATIAL implementation in which the spatial (sensor-to-sensor) and temporal
patterns are explicitly learned by the network. In this representation, m is
the number of sensors, n is the training time period, and k is the time period
we wish to predict.

## Methodology

SPATIAL is a first attempt at using a bidirectional LSTM model on both the
spatial and temporal directions of a time-series signal. More specifically,
based on geographical proximity or domain expertise, it is known that the
signals that we wish to predict (ocean currents, temperature, and dissolved
oxygen) have some dependency on neighbouring signals. We implement a
bidirectional LSTM model across the spatial direction of pertinent sensors and
train the model to learn the spatial and temporal structure. An overview of
SPATIAL is provided in Figure 1, and described in detail in the following
section.

### Deep Neural Networks

RNN and its variants are widely studied for time series forecasting
(Hochreiter and Schmidhuber 1997). A fundamental extension of RNNs compared to
ANN approaches is parameter sharing across different parts of the model. This
has natural applicability to the forecasting of time-series variables with
historical dependency. A schematic representation is provided in Figure 2(a)
illustrating the neural network mapping framework from an input time series,
$(x_{1},x_{2},\cdots,x_{t})$, to output label vector,
$(y_{t+1},y_{t+2},\cdots,y_{T})$. In effect, the RNN has two inputs, the
present state and the past.

Standard RNN approaches have been shown to fail when lags between response and
explanatory variables exceed 5–10 discrete timesteps (Gers 1999). Repeated
applications of the same parameters can give rise to vanishing, or exploding
gradients leading to model stagnation or instability. A number of approaches
have been proposed in the literature to address this, with the most popular
being LSTM (Goodfellow, Bengio, and Courville 2016).

An extension of the parameter sharing enabled by recurrent networks is
bidirectional LSTM which processes sequence data in backward and forward
directions (Schuster and Paliwal 1997). This has natural application to
sequential or time series data, and has demonstrated improved performance
against unidirectional LSTM in areas such as language processing (Wang et al.
2016), and speech recognition (Graves, Jaitly, and Mohamed 2013). While
bidirectional LSTM has previously been applied to time series forecasting,
this paper is the first that applies across the spatial as well as the
temporal components. Figure 2(b) provides a schematic of our SPATIAL
implementation. A unidirectional LSTM is applied across the time direction of
each sensor, and between individual sensors a series of stacked bidirectional
layers support learning between distinct time series. The number of stacked
layers is a hyperparameter selected during training. The features to the
network consist of an m $\times$ n array (where m is number of sensors and n
is number of time points) and labels consist a corresponding m $\times$ k
array (where k is the prediction window).

### The data

In this study, SPATIAL is applied to the following datasets: ocean current
speed or velocity collected along the Norwegian coast, ocean temperature, and
dissolved oxygen sampled from a high-density network of sensors at an
aquaculture site in Atlantic Canada.

Acoustic Doppler Current Profiler (ADCP) is used to measure water current
velocities over a depth range, using the Doppler effect of sound waves
scattered back from particles within the water column. Four ADCPs were
deployed at individual locations along the Norwegian coast. The sensors
collect 1-minute interval measurements at vertical resolution or bins of
between 5 and $15\text{\,}\mathrm{m}$ over a 1-year period (January 1st –
December 31st 2018). Observations were averaged over the depth to generate a
single time series for each ADCP.

Table 1 provides summary metrics of the ADCP deployments including longitude,
latitude, and distance between ADCPs (relative to arbitrarily selected, ADCP
1). The data were downloaded from the OPeNDAP server provisioned by the
Norwegian Meteorological Institute (Institute 2020). While these sensors are
geographically distant (up to $60\text{\,}\mathrm{k}\mathrm{m}$ apart), theory
imparted from transfer learning (Pan and Yang 2009) suggests that knowledge of
the dynamics at one location could be used to guide training at others.

# | ADCP name | Long | Lat | Dist. (km)
---|---|---|---|---
1 | E39_A_Sulafjorden | 6.18 | 62.56 | 00.0
2 | E39_B_Sulafjorden | 6.35 | 62.67 | 22.4
3 | E39_D_Breisundet | 6.09 | 62.60 | 11.1
4 | E39_F_Vartalsfjorden | 5.74 | 62.16 | 65.2
Table 1: The geographical longitude and latitude coordinates, and reference
name (complying with the naming convention used on the OpeNDAP portal
(Institute 2020) of the four ADCP datasets used in this study, together with
the distance between each sensor and the first one listed

Temperature and dissolved oxygen observations were collected within an
aquaculture farm of sea cages cultivating Atlantic salmon. Each cage was
$100\text{\,}\mathrm{m}$ in circumference, $12\text{\,}\mathrm{m}$ deep, with
around 25,000 fish. The cages were equipped with an array of sensors sampling
both temperature and dissolved oxygen. We gathered information from eight
sensors distributed across a cage: 4 at $2\text{\,}\mathrm{m}$, and 4 at
$8\text{\,}\mathrm{m}$ (North, South, East, West configuration).

These sensors sample a complex system described by natural conditions (natural
ocean variation due to dynamics) and the effects of fish farming (modified
temperature stratification patterns, consumption of oxygen by fish, etc.).
Furthermore, the geographical proximity of cages exerted an influence:
modification of upstream dynamics will be amplified at downstream sensors as
effects accumulate (e.g. oxygen depletion) (O’Donncha, Hartnett, and Nash
2013). Tidally driven flows have the most significant effect on dissolved
oxygen levels, with its influence a function of cage location within the farm.
As waters flow from one end of the farm to the other, fish behaviour and
physiology, as well as flow restriction from cage infrastructure reduce oxygen
levels. This results in significant oxygen variations throughout the farm with
high oxygen levels at one end of the farm and low oxygen levels at the other.
Since farmers use low oxygen as an indicator as to whether to feed, it is
important to implement robust predictive models that reflect spatial
variations at high resolutions to enable informed decisions that impact fish
growth and welfare (Burke et al. 2020).

The DO and temperature data was collected over a four-month period from August
16th – December 21st 2018 over relatively small geographical extents,
characterised by significant dynamics and aquaculture-industry activities.
Additional details on the data collection and application to directing
aquaculture operations are provided in Burke et al. (2020).

### Data Preprocessing

Anomalous data and gaps in the sensor data are common in ocean monitoring.
Table 2 presents summary statistics including information on the frequency of
occurrence of missing data. The BMean column quantifies the mean length of
contiguous blocks of missing data for each sensor dataset (averaged across all
sensors). The ADCP data in particular reports large data gaps which
complicates learning and requires bespoke data cleaning routines to adjust to
the characteristics of the models.

Each individual sensor data consists of a one-dimensional time series at
1-minute interval. We resampled each dataset to 30-minute intervals to avoid
spurious fluctuations while maintaining the fidelity of the data. Data
processing proceeded according to the requirements of the particular ML
pipeline.

SPATIAL has the simplest data preparation requirement with each time series
loaded into a single array of $[N_{sensors},N_{timepoints}]$ for each of the
three variables.

Dataset | Mean | $\sigma$ | Skewness | Kurtosis | BMean
---|---|---|---|---|---
ADCP | 20.22 | 7.54 | 2.51 | 9.11 | 216.61
Temp | 12.21 | 0.85 | -0.30 | -1.34 | 4.36
DO | 08.50 | 0.55 | -0.13 | -0.66 | 4.44
Table 2: Summary statistics of the three datasets.

The AutoAI and AMLP pipelines follow the tabular ML workflow of data
preparation. We used the TSML (Palmes, Ploennigs, and Brady 2020) which is an
open-source package to automatically convert the time series data into matrix
form by _sliding window_. Each row represents the autoregressive features
together with the date-time features (year, month, day hour, day of week,
etc), time-aligned with the corresponding label at the desired prediction
window. We setup the problem as a 24-hour ahead prediction using half hour
stride with 6-hour autoregressive window size predetermined based on the
initial experiments.

For the AutoAI and AMLP approaches, missing data were flagged and any rows
that contained missing data were dropped in the sliding window matrix
generated. SPATIAL on the other hand simply applied a masking layer that acted
on the data flagged as missing and automatically ignored them during learning.

The complete source code of SPATIAL has been released publicly under Apache
license at https://github.com/IBM/spatial-lstm, whereby further details of the
implementation can be studied in detail. The ADCP data we used is freely
available and the open source code contains scripts to download and preprocess
it. We currently do not have permission to release the oxygen and temperature
data as it pertains to commercial aquaculture operations.

Figure 3: Time series plot of the three datasets. Top figure represents
measurements of ocean current speed from four locations (summarised in Table
XXX), while middle and bottom figure denotes dissolved oxygen and temperature
observations respectively at eight locations.

## Results and Discussion

To evaluate the relative performance of SPATIAL with respect to existing
approaches, we included two optimal baseline models based on the AutoML (Drori
et al. 2018) technology:

  * •

IBM AutoAI (IBM 2020): a technology that is directed at automating the end-to-
end AI Lifecycle, from data cleaning, to algorithm selection, and to model
deployment and monitoring in the ML workflow (Wang et al. 2020).

  * •

AutoMLPipeline (AMLP) (Palmes 2020): a toolbox that provides a semi-automated
ML model generation and forecasting.

Models generated by the above AutoML technology were used to benchmark the
predictive skill of the SPATIAL framework, provide additional insight into
data characteristics and pattern extraction of environmental datasets, and
assess the ease of deployment of the different model frameworks.

Figure 4: Comparison of SPATIAL performance against two benchmark models: An
XGBoost model configured and deployed by AutoAI (red line), and a Random
Forest model generated by using the AMLP pipeline (blue line). Black line
indicates observations. The top row compares each model against observations
while the bottom row presents the residual plots between observation and
prediction. Each model predicts a 24-hour ahead forecast over 24 hours with a
stride of 30 minutes (new forecast generated every 30 minutes). Data is
presented for ADCP (left column), Dissolved Oxygen (middle column), and
Temperature (right column).

Figure 3 presents all sensor data illustrating a number of characteristics:

  * •

The current speed is highly volatile and reports rapid variations over a
relatively large range between 0 –
$75\text{\,}\mathrm{c}\mathrm{m}\mathrm{/}\mathrm{s}$.

  * •

Temperature data exhibits a pronounced seasonal pattern, reducing from a peak
of about 18 ∘C to 3 ∘C in latter half of December.

  * •

DO is exposed to relatively large fluctuations that are influenced by
exogenous, non-seasonal factors (particularly aquaculture-driven impacts such
as supplemental oxygenation and fish metabolism). Values drop rapidly from a
peak of about $10\text{\,}\mathrm{m}\mathrm{g}\mathrm{/}\mathrm{L}$ in late
August to a minimum of about
$7\text{\,}\mathrm{m}\mathrm{g}\mathrm{/}\mathrm{L}$ in mid-October before
increasing back to approximately
$10\text{\,}\mathrm{m}\mathrm{g}\mathrm{/}\mathrm{L}$.

These pronounced patterns encapsulated a wide-ranging array of time-series
behaviours including volatility, seasonality, and exogenous drivers. Of
particular interest are seasonal patterns in temperature data and the
capabilities to learn seasonally-varying patterns from short, sub-seasonal
datasets (both temperature and DO contain 17 weeks of observation data). These
constitute a challenging time-series forecasting problem and early analysis of
standard approaches such as GAM and ARIMA modeling reported substandard
predictive skill.

| SPATIAL | AMLP-RF | AutoAI-XGB
---|---|---|---
Dataset | MAE | $\sigma$ | MAE | $\sigma$ | MAE | $\sigma$
Temp | 0.86 | 0.54 | 0.25 | 0.04 | 0.21 | 0.03
DO | 0.32 | 0.25 | 0.27 | 0.04 | 0.28 | 0.04
ADCP | 4.67 | 5.40 | 3.80 | 1.68 | 4.51 | 1.94
Table 3: The MAE and standard deviation of reported MAE across all three
datasets for a five-fold cross-validation interrogation.

We compared our SPATIAL model against the two baseline implementations
described previously. Tuning of model performance considered effects of
appropriate number of time lags and hyperparameter selection, and adopted a
5-fold cross-validation for all models. The baseline models relied on a robust
automated hyperparameter optimisation routine, while hyperparameter tuning of
the SPATIAL adopted a greedy grid-search approach.

Table 3 summarises model performance in terms of MAE and the standard
deviation of the error across series of cross-validation tests. The baseline
models that demonstrated best performance across all datasets were XGBoost and
Random Forest, for AutoAI (AutoAI-XGB) and AMLP (AMLP-RF), respectively. We
see strong performance of the different models across all datasets, despite
the high complexity of the data. ADCP predictions reported significantly
higher MAE for all models, which was largely a result of larger volatility and
range in the data. Table 2 presents summary statistics of the three datasets
indicating that the standard deviation varies across the datasets. Considering
the reported MAE relative to the standard deviation of the data, we see
comparable predictive skill across the three datasets.

Figure 4 provides a time series plot comparing the three models against
observations (we selected one sensor from each of the dataset to illustrate).
All models closely capture both the short-scale fluctuations and seasonal
patterns of the data. This is particularly true for data such as temperature
which has a pronounced seasonal pattern and can be challenging for ML models
to learn with less than one-year of data (in our case, our training data was
17 weeks).

Performance comparisons demonstrated that existing algorithms give strong
predictive skill on these datasets. This performance is broadly replicated by
our SPATIAL model. Namely, adopting a simpler data preprocessing, and model
generation and implementation pipeline (fewer models to support by grouping
sensor data), we get performance comparable to existing state-of-the-art. Our
SPATIAL model has a number of practical advantages against existing
algorithms, particularly:

  1. 1.

The data preprocessing pipeline is simplified as the SPATIAL pipeline loads
all data into a single array that is fed to the networks (the model does not
rely on exogenous variables or feature transformations). Missing data is
handled readily with masking layer. This compares with the the benchmark
models that requires a bespoke model for each sensor signal and transforming
into appropriate matrix structure dependent on the desired prediction window,
historic features and stride forward step length (Palmes, Ploennigs, and Brady
2020).

  2. 2.

By feeding data from multiple sensors that are geographically in proximity or
share certain characteristics, the network can capture certain physical
relationships that exist in nature and potentially serves to regularize the
model.

  3. 3.

Treating all sensors simultaneously provides a large increase in computational
efficiency – only one model is trained rather than a model for each signal.

Points 1 and 2 above are closely connected. In forecasting, we wish to use the
simplest model that most closely mirrors what we know about the system. The
proposed SPATIAL framework simplifies the model training and deployment
process for environmental modeling by:

  * •

requiring only one model be trained and maintained;

  * •

eliminating need to implement transformations or matrix manipulations of the
input data;

  * •

naturally generating a time series sequence for the desired forecasting period
rather than a single value for each direct model (although classical
approaches can adopt an iterative method to generate a time-sequence
prediction via multi-step forecasting, predictive skill is generally limited
(Hamzaçebi, Akay, and Kutay 2009)).

An important point is that the SPATIAL model processes information from
multiple sensors that are known to be connected (based on the spatiotemporal
nature of the dataset). There is a large volume of literature related to
applying physics-informed constraints to ML which is discussed in the Related
Work section. The objective in many of those studies is to augment the models
with data external to the training data via aspects such as modified loss
functions, data augmentation, or specifying consensus filters to guide
disparate models or data towards convergence (Haehnel et al. 2020). The
proposed methodology presents a natural framework to ingest information
external to the time series signal to the predictive model positing the
opportunity to enhance learning. Within a time series forecasting paradigm,
the nominal architectures that deliver performance are well-established, and
often the challenge reduces to appropriate data conditioning and feature
engineering. Missing data, anomalous data, and sensor drift are major
obstacles, particularly for ocean-related deployments. SPATIAL presents a
natural framework to reduce the impacts of individual sensor error or bias by
enabling parameter sharing and pattern learning across multiple sensors that
can serve to regularize individual deviations.

Further, we investigated how interdependence of different variables can inform
learning. Dissolved oxygen has a nonlinear dependency on temperature and both
data were sampled at the same location. In a classical sense, one could adopt
temperature as a feature to a ML framework. We explored a combined oxygen and
temperature SPATIAL that fed data from both simultaneously to the network.
Results demonstrated reduced MAE compared to treating both independently
indicating the ability to learn from multiple scales. The combined SPATIAL
reported MAE of $0.3\text{\,}\mathrm{m}\mathrm{g}\mathrm{/}\mathrm{L}$, and
$0.65\text{\,}{}^{\circ}\mathrm{C}$ for oxygen and temperature, respectively,
when SPATIAL was applied across both datasets simultaneously, while the
corresponding results for the individual models were 0.32 and 0.86 for oxygen
and temperature.

Ocean variables such as dissolved oxygen and temperature are intrinsically
connected and it is intuitive to apply the capabilities of DL to extract and
exploit these relationships to achieve uplift in predictive skill.
Traditionally, this is a part of the feature engineering phase. SPATIAL
however, provides a natural framework to readily explore these relationships
via a simple DL framework that extracts these spatiotemporal patterns.

Finally, this paper compares SPATIAL against two baseline frameworks.
Importantly, the baseline models required training on each dataset
individually, representing a total of 20 model training iterations compared to
three for the SPATIAL. The computational saving is approximately similar (six-
fold reduction), although in reality the savings are larger, as the SPATIAL
architecture is relatively simple and required minimal hyperparameter tuning
compared to a full algorithm search implemented by AutoML approaches. These
computational savings are particularly attractive for edge-computing
applications and is a major area of interest for marine industry applications
(O’Donncha and Grant 2020). Many ocean sensor deployments consist of multiple
sensors connected to a low-power compute device to manage storage, data
processing, and connectivity. Extending these with a lightweight predictive
framework that can forecast based only on the data collected from the sensors
can allow for more intelligent data processing and enable adaptive sampling
for sensor networks (Jain and Chang 2004).

## Conclusions

This paper presents a DL framework based on bidirectional LSTM to extend
learning of time series signals. The resulting model provides a scalable
forecasting system that adjusts naturally to spatiotemporal patterns. The cost
of model training was reduced by an order of magnitude as we trained a single
model for each dataset rather than each signal. SPATIAL adjusts naturally to
missing data through a simple masking layer and the random distribution of
missing data between sensors can be leveraged to lessen the effect of data
gaps on learning (i.e. since typically sensors exhibit data gaps at different
times, combining information from multiple sensors in a single framework can
improve learning on noisy and error-prone data). Finally extending the
recurrent structure of LSTM-type approaches for time series applications to
the spatiotemporal direction has potential to translate to many different
industry applications such as weather, transport, and epidemiology.

The ML approach is extremely useful in forecasting oceanographic conditions of
water quality (e.g. oxygen), especially as it assimilates realtime forcing
variables such as temperature from sources including sensor networks or remote
sensing. As additional big data layers become available from instruments such
as underwater autonomous vehicles, this approach can be extended to 3D and 4D
geospatial data to map nearshore dynamics of oxygen or other variables. These
forecasts are essential in management decisions regarding coastal resources
including aquaculture, fisheries, and pollution discharge.

## References

  * Arandia et al. (2018) Arandia, E.; O’Donncha, F.; McKenna, S.; Tirupathi, S.; and Ragnoli, E. 2018.  Surrogate modeling and risk-based analysis for solute transport simulations.  _Stochastic Environmental Research and Risk Assessment_ 1–15.  ISSN 1436-3240.  doi:10.1007/s00477-018-1549-6.  URL http://link.springer.com/10.1007/s00477-018-1549-6.
  * Ashkezari et al. (2016) Ashkezari, M. D.; Hill, C. N.; Follett, C. N.; Forget, G.; and Follows, M. J. 2016\.  Oceanic eddy detection and lifetime forecast using machine learning methods.  _Geophysical Research Letters_ 43(23): 12–234.
  * Bryan (1969) Bryan, K. 1969.  Climate and the ocean circulation.  _Mon. Weather Rev_ 97(11): 806–827.
  * Burke et al. (2020) Burke, M.; Grant, J.; Filgueira, R.; and Stone, T. 2020.  Oceanographic processes control dissolved oxygen variability at an Atlantic salmon farm: Application of a real-time sensor network.  _Aquaculture_ .
  * Caballero-Alfonso, Carstensen, and Conley (2015) Caballero-Alfonso, A. M.; Carstensen, J.; and Conley, D. J. 2015.  Biogeochemical and environmental drivers of coastal hypoxia.  _Journal of Marine Systems_ 141: 190–199.
  * Chen et al. (2018) Chen, T. Q.; Rubanova, Y.; Bettencourt, J.; and Duvenaud, D. K. 2018.  Neural Ordinary Differential Equations.  In Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; and Garnett, R., eds., _Advances in Neural Information Processing Systems 31_ , 6572–6583. Curran Associates, Inc.  URL http://papers.nips.cc/paper/7892-neural-ordinary-differential-equations.pdf.
  * Constantin and Foias (1988) Constantin, P.; and Foias, C. 1988.  _Navier-stokes equations_.  University of Chicago Press.
  * de Bezenac, Pajot, and Gallinari (2017) de Bezenac, E.; Pajot, A.; and Gallinari, P. 2017.  Deep Learning for Physical Processes: Incorporating Prior Scientific Knowledge.  _arxiv preprint arXiv:1711.07970_ URL http://arxiv.org/abs/1711.07970.
  * DeVries, Thompson, and Meade (2017) DeVries, P. M. R.; Thompson, T. B.; and Meade, B. J. 2017.  Enabling large-scale viscoelastic calculations via neural network acceleration.  _Geophysical Research Letters_ 44(6): 2662–2669.  ISSN 00948276.  doi:10.1002/2017GL072716.  URL http://doi.wiley.com/10.1002/2017GL072716.
  * DeYoung et al. (2004) DeYoung, B.; Heath, M.; Werner, F.; Chai, F.; Megrey, B.; and Monfray, P. 2004.  Challenges of modeling ocean basin ecosystems.  _Science_ 304(5676): 1463–1466.
  * Drori et al. (2018) Drori, I.; Krishnamurthy, Y.; Rampin, R.; de Paula Lourenco, R.; Ono, J.; Cho, K.; Silva, C.; and Freire, J. 2018.  Alphad3m: Machine learning pipeline synthesis.  In _Workshop on AutoML (ICML)_.
  * Ducournau and Fablet (2016) Ducournau, A.; and Fablet, R. 2016.  Deep learning for ocean remote sensing: an application of convolutional neural networks for super-resolution on satellite-derived SST data.  In _2016 9th IAPR Workshop on Pattern Recogniton in Remote Sensing (PRRS)_ , 1–6. IEEE.  ISBN 978-1-5090-5041-3.  doi:10.1109/PRRS.2016.7867019.  URL http://ieeexplore.ieee.org/document/7867019/.
  * Esling and Agon (2012) Esling, P.; and Agon, C. 2012.  Time-series data mining.  _ACM Computing Surveys (CSUR)_ 45(1): 1–34.
  * EU Copernicus (2020) EU Copernicus. 2020.  Copernicus - Marine environment monitoring service.  URL https://marine.copernicus.eu/.
  * Fawaz et al. (2019) Fawaz, H. I.; Forestier, G.; Weber, J.; Idoumghar, L.; and Muller, P.-A. 2019.  Deep learning for time series classification: a review.  _Data Mining and Knowledge Discovery_ 33(4): 917–963.
  * Gamboa (2017) Gamboa, J. C. B. 2017.  Deep learning for time-series analysis.  _arXiv preprint arXiv:1701.01887_ .
  * Gan and Allen (2005) Gan, J.; and Allen, J. S. 2005.  On open boundary conditions for a limited-area coastal model off Oregon. Part 1: Response to idealized wind forcing.  _Ocean Modelling_ 8(1-2): 115–133.
  * Gers (1999) Gers, F. 1999.  Learning to forget: continual prediction with LSTM.  In _9th International Conference on Artificial Neural Networks: ICANN ’99_ , volume 1999, 850–855. IEE.  ISBN 0 85296 721 7.  doi:10.1049/cp:19991218.  URL http://digital-library.theiet.org/content/conferences/10.1049/cp–“˙˝19991218.
  * Gers and Schmidhuber (2001) Gers, F. A.; and Schmidhuber, E. 2001.  LSTM recurrent networks learn simple context-free and context-sensitive languages.  _IEEE Transactions on Neural Networks_ 12(6): 1333–1340.
  * Gokaraju et al. (2011) Gokaraju, B.; Durbha, S. S.; King, R. L.; and Younan, N. H. 2011.  A Machine Learning Based Spatio-Temporal Data Mining Approach for Detection of Harmful Algal Blooms in the Gulf of Mexico.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ 4(3): 710–720.  ISSN 1939-1404.  doi:10.1109/JSTARS.2010.2103927.  URL http://ieeexplore.ieee.org/document/5701668/.
  * Goodfellow, Bengio, and Courville (2016) Goodfellow, I. J.; Bengio, Y.; and Courville, A. 2016.  _Deep Learning_.  Cambridge, MA, USA: MIT Press.  http://www.deeplearningbook.org.
  * Grathwohl et al. (2019) Grathwohl, W.; Chen, R. T. Q.; Bettencourt, J.; and Duvenaud, D. 2019.  Scalable Reversible Generative Models with Free-form Continuous Dynamics.  In _International Conference on Learning Representations_.
  * Graves, Jaitly, and Mohamed (2013) Graves, A.; Jaitly, N.; and Mohamed, A.-r. 2013.  Hybrid speech recognition with deep bidirectional LSTM.  In _2013 IEEE workshop on automatic speech recognition and understanding_ , 273–278. IEEE.
  * Gulgec et al. (2019) Gulgec, N. S.; Shi, Z.; Deshmukh, N.; Pakzad, S.; and Takáč, M. 2019\.  FD-Net with Auxiliary Time Steps: Fast Prediction of PDEs using Hessian-Free Trust-Region Methods.  In _NeurIPS 2019 Workshop (Beyond First Order Methods in ML)_.
  * Haehnel et al. (2020) Haehnel, P.; Mareček, J.; Monteil, J.; and O’Donncha, F. 2020.  Using deep learning to extend the range of air pollution monitoring and forecasting.  _Journal of Computational Physics_ 408: 109278.
  * Hamzaçebi, Akay, and Kutay (2009) Hamzaçebi, C.; Akay, D.; and Kutay, F. 2009.  Comparison of direct and iterative artificial neural network forecast approaches in multi-periodic time series forecasting.  _Expert Systems with Applications_ 36(2): 3839–3844.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long short-term memory.  _Neural computation_ 9(8): 1735–1780.
  * Holt et al. (2009) Holt, J.; Harle, J.; Proctor, R.; Michel, S.; Ashworth, M.; Batstone, C.; Allen, I.; Holmes, R.; Smyth, T.; Haines, K.; et al. 2009.  Modelling the global coastal ocean.  _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ 367(1890): 939–951.
  * IBM (2020) IBM. 2020.  IBM Watson Studio - AutoAI.  URL https://www.ibm.com/ie-en/cloud/watson-studio/autoai.
  * Institute (2020) Institute, N. M. 2020.  Norkyst Model.  URL https://thredds.met.no/thredds/catalog.html.
  * Jain and Chang (2004) Jain, A.; and Chang, E. Y. 2004.  Adaptive sampling for sensor networks.  In _Proceeedings of the 1st international workshop on Data management for sensor networks: in conjunction with VLDB 2004_ , 10–16.
  * James, Zhang, and O’Donncha (2018) James, S.; Zhang, Y.; and O’Donncha, F. 2018.  A machine learning framework to forecast wave conditions.  _Coastal Engineering_ 137\.  ISSN 03783839.  doi:10.1016/j.coastaleng.2018.03.004.
  * Ji (2017) Ji, Z.-G. 2017.  _Hydrodynamics and water quality: modeling rivers, lakes, and estuaries_.  John Wiley & Sons.
  * Lary, Müller, and Mussa (2004) Lary, D. J.; Müller, M. D.; and Mussa, H. Y. 2004.  Using neural networks to describe tracer correlations.  _Atmospheric Chemistry and Physics_ 4(1): 143–146.
  * O’Donncha and Grant (2020) O’Donncha, F.; and Grant, J. 2020.  Precision Aquaculture.  _IEEE Internet of Things Magazine_ 2(4): 26–30.  ISSN 2576-3180.  doi:10.1109/iotm.0001.1900033.
  * O’Donncha, Hartnett, and Nash (2013) O’Donncha, F.; Hartnett, M.; and Nash, S. 2013.  Physical and numerical investigation of the hydrodynamic implications of aquaculture farms.  _Aquacultural Engineering_ 52: 14–26.
  * O’Donncha et al. (2015) O’Donncha, F.; Hartnett, M.; Nash, S.; Ren, L.; and Ragnoli, E. 2015.  Characterizing observed circulation patterns within a bay using HF radar and numerical model simulations.  _Journal of Marine Systems_ 142: 96–110.
  * Palmes (2020) Palmes, P. 2020.  AutoMLPipeline: A Toolbox for Building ML Pipelines.  doi:10.5281/zenodo.3980593.  https://doi.org/10.5281/zenodo.3980593.
  * Palmes, Ploennigs, and Brady (2020) Palmes, P.; Ploennigs, J.; and Brady, N. 2020.  TSML (Time Series Machine Learning).  _Proceedings of the JuliaCon Conferences_ 1(1): 51.  doi:10.21105/jcon.00051.  https://doi.org/10.21105/jcon.00051.
  * Pan and Yang (2009) Pan, S. J.; and Yang, Q. 2009.  A survey on transfer learning.  _IEEE Transactions on knowledge and data engineering_ 22(10): 1345–1359.
  * Pidgeon and Winant (2005) Pidgeon, E. J.; and Winant, C. D. 2005.  Diurnal variability in currents and temperature on the continental shelf between central and southern California.  _Journal of Geophysical Research: Oceans_ 110(C3).
  * Rao, Sun, and Liu (2020) Rao, C.; Sun, H.; and Liu, Y. 2020.  Physics informed deep learning for computational elastodynamics without labeled data.  _arXiv preprint arXiv:2006.08472_ .
  * Schuster and Paliwal (1997) Schuster, M.; and Paliwal, K. K. 1997.  Bidirectional recurrent neural networks.  _IEEE transactions on Signal Processing_ 45(11): 2673–2681.
  * Smyth et al. (2006) Smyth, T. J.; Moore, G. F.; Hirata, T.; and Aiken, J. 2006.  Semianalytical model for the derivation of ocean color inherent optical properties: description, implementation, and performance assessment.  _Applied Optics_ 45(31): 8116–8131.
  * Song and Haidvogel (1994) Song, Y.; and Haidvogel, D. 1994.  A semi-implicit ocean circulation model using a generalized topography-following coordinate system.  _Journal of Computational Physics_ 115(1): 228–244.
  * Stockie (2011) Stockie, J. M. 2011.  The mathematics of atmospheric dispersion modeling.  _SIAM Review_ 53(2): 349–372.
  * Wang et al. (2016) Wang, C.; Yang, H.; Bartz, C.; and Meinel, C. 2016.  Image captioning with deep bidirectional LSTMs.  In _Proceedings of the 24th ACM international conference on Multimedia_ , 988–997.
  * Wang et al. (2020) Wang, D.; Ram, P.; Weidele, D. K. I.; Liu, S.; Muller, M.; Weisz, J. D.; Valente, A.; Chaudhary, A.; Torres, D.; Samulowitz, H.; et al. 2020.  AutoAI: Automating the End-to-End AI Lifecycle with Humans-in-the-Loop.  In _Proceedings of the 25th International Conference on Intelligent User Interfaces Companion_ , 77–78.
  * Wiewel, Becher, and Thuerey (2018) Wiewel, S.; Becher, M.; and Thuerey, N. 2018.  Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flow.  _arxiv preprint arXiv:1802.10123_ URL http://arxiv.org/abs/1802.10123.


# UWB at SemEval-2020 Task 1: Lexical Semantic Change Detection

Ondřej Pražák Department of Computer Science and Engineering,  Pavel Přibáň
Department of Computer Science and Engineering,  Stephen Taylor Department of
Computer Science and Engineering,  Jakub Sido NTIS – New Technologies for the
Information Society,  Department of Computer Science and Engineering,

###### Abstract

In this paper, we describe our method for detection of lexical semantic
change, i.e., word sense changes over time. We examine semantic differences
between specific words in two corpora, chosen from different time periods, for
English, German, Latin, and Swedish. Our method was created for the SemEval
2020 Task 1: Unsupervised Lexical Semantic Change Detection. We ranked
$1^{st}$ in Sub-task 1: binary change detection, and $4^{th}$ in Sub-task 2:
ranked change detection. Our method is fully unsupervised and language
independent. It consists of preparing a semantic vector space for each corpus,
earlier and later; computing a linear transformation between earlier and later
spaces, using Canonical Correlation Analysis and Orthogonal Transformation;
and measuring the cosines between the transformed vector for the target word
from the earlier corpus and the vector for the target word in the later
corpus.

##  1 Introduction

00footnotetext:  ${}^{\text{*}}$Equal contribution. 00footnotetext:  This work
is licensed under a Creative Commons Attribution 4.0 International Licence.
Licence details: http://creativecommons.org/licenses/by/4.0/.

Language evolves with time. New words appear, old words fall out of use, the
meanings of some words shift. The culture changes as well as the expected
audience of the printed word. There are changes in topics, in syntax, in
presentation structure. Reading the natural philosophy musings of aristocratic
amateurs from the eighteenth century, and comparing with a monograph from the
nineteenth century, or a medical study from the twentieth century, we can
observe differences in many dimensions, some of which seem hard to study.
Changes in word senses are both a visible and a tractable part of language
evolution. Computational methods for researching words’ stories have the
potential of helping us understand this small corner of linguistic evolution.
The tools for measuring these diachronic semantic shifts might also be useful
for measuring whether the same word is used in different ways in synchronic
documents. The task of finding word sense changes over time is called
diachronic Lexical Semantic Change (LSC) detection. The task is getting more
attention in recent years [Hamilton et al., 2016b, Frermann and Lapata, 2016,
Schlechtweg et al., 2017]. There is also the synchronic LSC task, which aims
to identify domain-specific changes of word senses compared to general-
language usage [Schlechtweg et al., 2019].

?) provides a comprehensive survey of techniques for the LSC task, as does ?).
?) evaluated available approaches for LSC detection using the DURel dataset
[Schlechtweg et al., 2018]. Some of the methodologies for finding time-
sensitive meanings were borrowed from information retrieval techniques in the
first place. According to ?), there are mainly three types of approaches. (1)
Semantic vector spaces approaches [Gulordava and Baroni, 2011, Kim et al.,
2014, Xu and Kemp, 2015, Eger and Mehler, 2016, Hamilton et al., 2016a,
Hamilton et al., 2016b, Rosenfeld and Erk, 2018] represent each word with two
vectors for two different time periods. The change of meaning is then measured
by the cosine distance between the two vectors. (2) Topic modeling approaches
[Wang and McCallum, 2006, Bamman and Crane, 2011, Wijaya and Yeniterzi, 2011,
Mihalcea and Nastase, 2012, Cook et al., 2014, Frermann and Lapata, 2016,
Schlechtweg and Walde, 2020] estimate a probability distribution of words over
their different senses, i.e., topics. (3) Clustering models [Mitra et al.,
2015, Tahmasebi and Risse, 2017] are used to cluster words into clusters
representing different senses.

We participated in the SemEval 2020 Task 1: Unsupervised Lexical Semantic
Change Detection [Schlechtweg et al., 2020] competition. In this paper, we
describe our solution and submitted systems for this competition. The task
consists of two sub-tasks, a binary classification task (Sub-task 1) and a
ranking task (Sub-task 2), which involve comparing usage of target words
between two lemmatized corpora, each drawn from documents from a different
time period for four languages: English, German, Latin, and Swedish. For both
sub-tasks, only the target words and two corpora for each language were
provided by organizers, no annotated data. The task is intended to be solved
in a completely unsupervised way.

In the binary classification task, the goal is for two given corpora $C_{1}$
and $C_{2}$ (for time $t_{1}$ and $t_{2}$) and for a set of target words,
decide which of these words changed or did not change their sense (semantic)
between $t_{1}$ and $t_{2}$. Change of sense is whether the word lost or
gained any sense between the two periods (corpora). The objective of the Sub-
task 2 is for two given corpora $C_{1}$ and $C_{2}$, rank a set of target
words according to their degree of lexical semantic change between $t_{1}$ and
$t_{2}$. A higher rank means a stronger change. Target words are the same for
both sub-tasks.

Because language is evolving, expressions, words, and sentence constructions
in two corpora from different time periods about the same topic will be
written in languages that are quite similar but slightly different. They will
share the majority of their words, grammar, and syntax.

The main idea behind our solution is that we treat each pair of corpora
$C_{1}$ and $C_{2}$ as different languages $L_{1}$ and $L_{2}$ even though
that text from both corpora is written in the same language. We believe that
these two languages $L_{1}$ and $L_{2}$ will be extremely similar in all
aspects, including semantic. We train separate semantic space for each corpus
and subsequently, we map these two spaces into one common cross-lingual space.
We use methods for cross-lingual mapping [Brychcín et al., 2019, Artetxe et
al., 2016, Artetxe et al., 2017, Artetxe et al., 2018a, Artetxe et al., 2018b]
and thanks to the large similarity between $L_{1}$ and $L_{2}$ the quality of
transformation should be high. We compute cosine similarity to classify and
rank the target words, see Section 3 for details.

Our systems111Our code is available at
https://github.com/pauli31/SemEval2020-task1 ranked $1^{st}$ out of $33$ teams
in Sub-task 1 with an average accuracy of $0.687$, and $4^{th}$ out of $32$
teams in Sub-task 2 with an average Spearman’s rank correlation of $0.481$.

##  2 Data

The corpora are drawn from several sources, described in [Schlechtweg et al.,
2020]. Table 1 shows periods and sizes. For each language, items in the
earlier corpus are separated from items in the later one by at least 46 years
(German) and as much as 2200 years (Latin). All corpora are lemmatized,
punctuation is removed, and sentences are randomly reordered. For English,
target words have been marked with their part-of-speech. Two example
sentences:(1) there be no pleasing any of you do as we may, and (2) rise upon
the instant in his stirrup the bold cavalier hurl with a sure and steady hand
the discharge weapon in the face_nn of his next opponent. (1) illustrates a
failure of lemmatization -- the word ’pleasing’, which is a form of the verb
’please’; (2) shows a target word ’face’, marked as a noun. Less than 10% of
the English sentences contain a target word.

| Corpus 1 | Corpus 2 |
---|---|---|---
Language | Period | # Tokens | Period | # Tokens | # Targets
English | 1810-1860 | 6 559 657 | 1960-2010 | 6 769 814 | 37
German | 1800-1900 | 70 244 484 | 1946-1990 | 72 397 546 | 48
Latin | 200BC-1BC | 1 751 405 | 100AD-present | 9 417 033 | 40
Swedish | 1790-1830 | 71 091 465 | 1895-1903 | 110 792 658 | 31

Table 1: Corpus statistics. The last column # Targets denotes the number of
target words.

Lemmatization reduces the vocabulary so that there are more examples of each
word. It also introduces ambiguity; the decisions to add a POS tag to English
target words and retain German noun capitalization shows that the organizers
were aware of this problem.

##  3 System Description

First, we train two semantic spaces from corpus $C_{1}$ and $C_{2}$. We
represent the semantic spaces by a matrix $\mathbf{X}^{s}$ (i.e., a source
space $s$) and a matrix $\mathbf{X}^{t}$ (i.e, a target space $t$)222The
source space $\mathbf{X}^{s}$ is created from the corpus $C_{1}$ and the
target space $\mathbf{X}^{t}$ is created from the corpus $C_{2}$. using
word2vec Skip-gram with negative sampling [Mikolov et al., 2013]. We perform a
cross-lingual mapping of the two vector spaces, getting two matrices
$\mathbf{\hat{X}}^{s}$ and $\mathbf{\hat{X}}^{t}$ projected into a shared
space. We select two methods for the cross-lingual mapping Canonical
Correlation Analysis (CCA) using the implementation from [Brychcín et al.,
2019] and a modification of the Orthogonal Transformation from VecMap [Artetxe
et al., 2018b]. Both of these methods are linear transformations. In our case,
the transformation can be written as follows:

$\mathbf{\hat{X}}^{s}=\mathbf{W}^{s\rightarrow t}\mathbf{X}^{s}$ (1)

where $\mathbf{W}^{s\rightarrow t}$ is a matrix that performs linear
transformation from the source space $s$ (matrix $\mathbf{X}^{s}$) into a
target space $t$ and $\mathbf{\hat{X}}^{s}$ is the source space transformed
into the target space $t$ (the matrix $\mathbf{X}^{t}$ does not have to be
transformed because $\mathbf{X}^{t}$ is already in the target space $t$ and
$\mathbf{X}^{t}=\mathbf{\hat{X}}^{t}$).

Generally, the CCA transformation transforms both spaces $\mathbf{X}^{s}$ and
$\mathbf{X}^{t}$ into a third shared space $o$ (where
$\mathbf{X}^{s}\neq\mathbf{\hat{X}}^{s}$ and
$\mathbf{X}^{t}\neq\mathbf{\hat{X}}^{t}$). Thus, CCA computes two
transformation matrices $\mathbf{W}^{s\rightarrow o}$ for the source space and
$\mathbf{W}^{t\rightarrow o}$ for the target space. The transformation
matrices are computed by minimizing the negative correlation between the
vectors $\mathbf{x}_{i}^{s}\in\mathbf{X}^{s}$ and
$\mathbf{x}_{i}^{t}\in\mathbf{X}^{t}$ that are projected into the shared space
$o$. The negative correlation is defined as follows:

$\operatorname*{argmin~{}}_{\mathbf{W}^{s\rightarrow
o},\mathbf{W}^{t\rightarrow o}}-\sum_{i=1}^{n}\rho(\mathbf{W}^{s\rightarrow
o}\mathbf{x}_{i}^{s},\mathbf{W}^{t\rightarrow
o}\mathbf{x}_{i}^{t})=-\sum_{i=1}^{n}\frac{cov(\mathbf{W}^{s\rightarrow
o}\mathbf{x}_{i}^{s},\mathbf{W}^{t\rightarrow
o}\mathbf{x}_{i}^{t})}{\sqrt{var(\mathbf{W}^{s\rightarrow
o}\mathbf{x}_{i}^{s})\times var(\mathbf{W}^{t\rightarrow
o}\mathbf{x}_{i}^{t})}}$ (2)

where $cov$ the covariance, $var$ is the variance and $n$ is a number of
vectors. In our implementation of CCA, the matrix $\mathbf{\hat{X}}^{t}$ is
equal to the matrix $\mathbf{X}^{t}$ because it transforms only the source
space $s$ (matrix $\mathbf{X}^{s}$) into the target space $t$ from the common
shared space with a pseudo-inversion, and the target space does not change.
The matrix $\mathbf{W}^{s\rightarrow t}$ for this transformation is then given
by:

$\mathbf{W}^{s\rightarrow t}=\mathbf{W}^{s\rightarrow
o}(\mathbf{W}^{t\rightarrow o})^{-1}$ (3)

The submissions that use CCA are referred to as cca-nn, cca-bin, cca-nn-r and
cca-bin-r where the -r part means that the source and target spaces are
reversed, see Section 4. The -nn and -bin parts refer to a type of threshold
used only in the Sub-task 1, see Section 3.1. Thus, in Sub-task 2, there is no
difference for the following pairs of submissions: cca-nn \-- cca-bin and cca-
nn-r \-- cca-bin-r.

In the case of the Orthogonal Transformation, the submissions are referred to
as ort & uns. We use Orthogonal Transformation with a supervised seed
dictionary consisting of all words common to both semantic spaces. (ort). The
transformation matrix $\mathbf{W}^{s\rightarrow t}$ is given by:

$\operatorname*{argmin~{}}_{\mathbf{W}^{s\rightarrow
t}}{\sum_{i}^{|V|}{(\mathbf{W}^{s\rightarrow
t}\mathbf{x}^{s}_{i}-\mathbf{x}^{t}_{i})^{2}}}$ (4)

under the hard condition that $\mathbf{W}^{s\rightarrow t}$ needs to be
orthogonal, where V is the vocabulary of correct word translations from source
to target space. The reason for the orthogonality constraint is that linear
transformation with the orthogonal matrix does not squeeze or re-scale the
transformed space. It only rotates the space, thus it preserves most of the
relationships of its elements (in our case it is important that orthogonal
transformation preserves angles between the words, so it preserves the cosine
similarity).

?) also proposed a method for automatic dictionary induction. This is a fully
unsupervised method for finding orthogonal cross-lingual transformations. We
used this approach for our uns submissions.

Finally in all transformation methods, for each word $w_{i}$ from the set of
target words $T$, we select its corresponding vectors $\mathbf{v}_{w_{i}}^{s}$
and $\mathbf{v}_{w_{i}}^{t}$ from matrices $\mathbf{\hat{X}}^{s}$ and
$\mathbf{\hat{X}}^{t}$, respectively
($\mathbf{v}_{w_{i}}^{s}\in\mathbf{\hat{X}}^{s}$ and
$\mathbf{v}_{w_{i}}^{t}\in\mathbf{\hat{X}}^{t}$), and we compute cosine
similarity between these two vectors. The cosine similarity is then used to
generate an output for each sub-task. For Sub-task 2, we compute the degree of
change for word $w_{i}$ as
$1-cos(\mathbf{v}_{w_{i}}^{s},\mathbf{v}_{w_{i}}^{t})$. The settings of hyper-
parameters for both methods give us several combinations, see sections 4 and
5.

###  3.1 Binary System

The organizers provided a definition for binary change in terms of specific
numbers of usages of senses of a target word. We decided against attempting to
model and group individual word usages. Instead, we decided to use the
continuous scores from Sub-task 2 for the binary task. We assume that there is
a threshold $t$ for which the target words with a continuous score greater
than $t$ changed meaning and words with the score lower than $t$ did not. We
know that this assumption is generally wrong (because using the threshold we
introduce some error into the classification), but we still believe it holds
for most cases and it is the best choice. In order to examine this assumption
after the evaluation period, we computed the accuracy of the gold ranking
scores with the optimal threshold (selected to maximize test accuracy). As you
can see in Table 2, even if an optimal threshold is chosen, the best accuracy
that can be achieved is, on average, $87.6\%$.

| Avg | Eng | Ger | Lat | Swe
---|---|---|---|---|---
acc | .876 | .838 | .854 | .875 | .936
$t$ | - | .207 | .190 | .235 | .244

Table 2: Sub-task 1 accuracy with gold ranking and optimal thresholds.

In order to find the threshold $t$, we tried several approaches. We call the
first approach binary-threshold (-bin in Table 3). For each target word
$w_{i}$ we compute cosine similarity of its vectors $\mathbf{v}_{w_{i}}^{s}$
and $\mathbf{v}_{w_{i}}^{t}$, then we average these similarities for all
words. The resulting averaged value is used as the threshold. Another approach
called global-threshold (-gl) is done similarly, but the average similarity is
computed across all four languages. The last approach, called nearest-
neighbors (-nn), compares sets of nearest neighbours. For target word $w_{i}$
we find $100$ nearest (most similar) words from both transformed spaces
(matrices $\mathbf{\hat{X}}^{s}$ and $\mathbf{\hat{X}}^{t}$), getting two sets
of nearest neighbours $N^{s}$ and $N^{t}$ for each target word. Then, we
compute the size of the intersection of these two sets for each target word.
From the array of intersection sizes, we select the second highest value333We
observed that the largest size of intersection is usually significantly
greater than the other sizes so we select the second highest number. We are
aware that this is a very simple and data-dependent approach and could be
further improved., and we divide it by two. The resulting number is a
threshold for all target words. If the size of a target word’s intersection is
greater or equal to the threshold, we classify the word unchanged; otherwise
changed. This threshold is set for each language independently.

##  4 Experimental Setup

To obtain the semantic spaces, we employ Skip-gram with negative sampling
[Mikolov et al., 2013]. We use the Gensim framework [Řehůřek and Sojka, 2010]
for training the semantic spaces. For the final submission, we trained the
semantic spaces with $100$ dimensions for five iterations with five negative
samples and window size set to five. Each word has to appear at least five
times in the corpus to be used in the training. For all cca- submissions, we
build the translation dictionary for the cross-lingual transformation of the
two spaces by removing the target words from the intersection of their
vocabularies.

For cca-nn-r and cca-bin-r we change the direction of the cross-lingual
transformation. The initial setup for the transformation is that the source
space is the space of the older corpus $C_{1}$ (represented by the matrix
$\mathbf{X}^{s}$), and the target space is the semantic space of the later
corpus $C_{2}$ (represented by the matrix $\mathbf{X}^{t}$). We reversed the
order, and we use the matrix $\mathbf{X}^{t}$ as the source space, which is
transformed into semantic space (matrix $\mathbf{X}^{s}$) of the older corpus,
i.e. into the original source space.

The two other methods are Orthogonal Transformations with identical words as
the seed dictionary (ort) and unsupervised transformation (uns) from the
VecMap tool. In these methods, we use median similarity as the threshold for
the binary task. We experimented with a separate threshold for each language
(bin suffix) and with a single global threshold for all the languages (gl
suffix).

The sample data provided with the task did not contain any labeled development
target words, so we used the DURel [Schlechtweg et al., 2018] corpus and WOCC
[Schlechtweg et al., 2019] corpus444Both available at www.ims.uni-
stuttgart.de/en/research/resources/corpora/wocc to validate our solutions’
performance. The WOCC corpus contains lemmatized German sentences from
Deutsches Textarchiv [Deutsches Textarchiv, 2017] for two time periods
1750-1799 and 1850-1899 , which we used to train the semantic spaces. We
evaluated our system on the DURel corpus, and we achieved $0.78$ of Spearman’s
correlation rank with settings that correspond to the column named cca-nn-r in
Table 3.

##  5 Results

We submitted eight different submissions for the Sub-task 1 and four for the
Sub-task 2 obtained by CCA and the VecMap tool. We also submitted predictions
based on Latent Dirichlet Allocation [Blei et al., 2003] (LDA), but because of
its poor results and limited paper size, we do not include the description
here. The results, along with the ranking555The ranking in the table is the
ranking among 186 submissions and does not correspond to ranking among 33
teams because team ranks are based on their best submission. are shown in
Table 3. The bold results denote the best result we achieved for each
language, and the underlined results were used in our team’s final ranking.

Task |  | cca-nn | cca-nn-r | cca-bin | cca-bin-r | ort-bin | ort-gl | uns-bin | uns-gl
---|---|---|---|---|---|---|---|---|---
1 | Avg | .572 (65) | .607 (28) | .663 (6) |  0.687 (1) | .634 (21) | .639 (16) | .659 (11) | .655 (13)
Eng | .595 (6) | .730 (1) | .595 (6) | .622 (5) | .568 (7) | .622 (5) | .568 (7) | .622 (5)
Ger | .521 (14) | .542 (13) | .812 (1) | .750 (3) | .750 (3) | .688 (6) | .750 (3) | .646 (8)
Lat | .625 (5) | .575 (7) | .600 (6) | .700 (2) | .575 (7) | .600 (6) | .675 (3) | .675 (3)
Swe | .548 (8) | .581 (7) | .645 (5) | .677 (4) | .645 (5) | .645 (5) | .645 (5) | .677 (4)
2 | Avg | .411 (14) | .469 (7) | - | - |  .481 (6) | - | .455 (8) | -
Eng | .279 (33) | .421 (5) | - | - | .367 (11) | - | .365 (12) | -
Ger | .707 (4) | .696 (7) | - | - | .697 (6) | - | .687 (9) | -
Lat | .181 (66) | .251 (58) | - | - | .254 (55) | - | .181 (65) | -
Swe | .476 (10) | .509 (8) | - | - | .604 (1) | - | .587 (3) | -
Table 3: Results for our final submissions.

The cca-bin-r system settings achieved the first-place rank out of 189
submissions by 33 teams on Sub-task 1, with an absolute accuracy of 0.687. The
majority class was unchanged, and always choosing it would have given a score
of 0.571. The submissions with the top 15 best accuracies (four teams) had
scores ranging from 0.659 to 0.687 and assorted scores on the different
languages. The four of our systems using the -bin approach, which included the
top-scoring system, have a mean percentile of 94.6; the two -gl strategy
systems had a mean percentile of 92.7, and the two systems with the -nn
strategy had a mean percentile of 75.5. On Sub-task 2, our best system
achieved the fourth-place rank out of 33 teams. It had an average correlation
against the gold ranking for all four languages of 0.481.

Since the threshold strategy was used only for Sub-task 1, there is no
difference in results for Sub-task 2 in Table 3 in the following pairs of
columns cca-nn \-- cca-bin, cca-nn-r \-- cca-bin-r, ort-bin \-- ort-gl and
uns-bin \-- uns-gl. Thus, for Sub-task 2 we provide numbers only in the cca-
nn, cca-nn-r, ort-bin and uns-bin columns.

Examining Table 3, the ranking scores for Latin are not only worse than the
other languages in absolute terms, but their position relative to other
submissions is also much worse. The Latin corpora have several anomalies, but
only the small size of the earlier corpus (a third the size of the next
smallest corpus) seems likely to be a problem. For example, although both
Latin corpora have a much larger proportion of short lines than the others, a
measurement of the mean context size for target words shows that for all of
the corpora, the mean context size for target words is between 8.38 (German 2)
and 8.95 (German 1).

A different problem is syntax. The lemmatization of the corpora is no obstacle
to the English reader, who can usually guess the correct form, because word-
order in English (and also in Swedish and German) is somewhat rigid, and word
inflexions are largely replaced by separate words, which have their own
lemmas. In Latin, word-order is quite flexible, and different authors have
different preferred schemes for emphasis and euphony. Two adjacent words are
not required to be part of each other’s context, and the most semantically
related words in a sentence may be separated precisely to emphasize them.

We performed post-evaluation experiments with word embedding vector size in
order to discover its effect on system performance, see Figure 1 containing
visualization of results for Sub-task 2. It shows that the optimal size for
this task is in most cases between 100 and 175 for Sub-task 2, and between 75
and 125 for Sub-task 1 (not shown here). We also tried using fastText
[Bojanowski et al., 2017] instead of word2vec Skip-gram to get the semantic
space, with settings that correspond to the ones we used for the final
submissions. The performance using fastText was not improved in general, but
for some settings we obtained slightly better results. Other experiments
suggest that Latin results can benefit when the context size is increased
during the training of the semantic spaces. According to submitted results, it
seems that CCA method with reversed order (columns cca-nn-r, cca-bin-r in
Table 3) works better than without reversed but from the Figure 1 is evident
that it is valid only for English and Latin.

[width=0.33]graphs/SemEval-Method-a-rank-5-iter

(a) Nearest neighbors mapping -- cca-nn

[width=0.33]graphs/SemEval-Method-b-rank-5-iter

(b) Reversed nn. mapping -- cca-nn-r

[width=0.33]graphs/SemEval-Avg-rank-5-iter

(c) Average of all languages

Figure 1: Performance using the CCA method with different word embeddings size
for Sub-task 2.

##  6 Conclusion

Applying a threshold to semantic distance is a sensible architecture for
detecting the binary semantic change in target words between two corpora. Our
binary-threshold strategy succeeded quite well. We did have a small advantage
in that the list of target words turned out to be nearly equally divided
between changed (0.57) and unchanged (0.43) words. Thus, choosing thresholds
assuming that the division was 50/50 was not a severe problem. Our experiments
also reveal the limits of a threshold strategy, as shown in Table 2. Second,
although our systems did not win Sub-task 2 change-ranking competition, they
show that our architecture is a strong contender for this task when there is
sufficient data to build good vector semantic spaces. The results for Latin
illustrate that there is still room for other, less statistical approaches;
one might have predicted, falsely, that 1.7M tokens was plenty of data. The
variety of experiments which were possible in the post-evaluation period
suggest that the corpora developed for the task will encourage work in the
area. In future work, we expect to focus on other cross-lingual techniques and
other methods of measuring similarity between words besides cosine similarity.

## Acknowledgments

This work has been partly supported by ERDF ”Research and Development of
Intelligent Components of Advanced Technologies for the Pilsen Metropolitan
Area (InteCom)” (no.: CZ.02.1.01/0.0/0.0/17 048/0007267), by the project
LO1506 of the Czech Ministry of Education, Youth and Sports and by Grant No.
SGS-2019-018 Processing of heterogeneous data and its specialized
applications. Access to computing and storage facilities owned by parties and
projects contributing to the National Grid Infrastructure MetaCentrum provided
under the programme "Projects of Large Research, Development, and Innovations
Infrastructures" (CESNET LM2015042), is greatly appreciated.

## References

  * [Ammar et al., 2016]  Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A. Smith.  2016\.  Massively multilingual word embeddings.  Computing Research Repository, arXiv:1602.01925.
  * [Artetxe et al., 2016]  Mikel Artetxe, Gorka Labaka, and Eneko Agirre.  2016\.  Learning principled bilingual mappings of word embeddings while preserving monolingual invariance.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289--2294, Austin, Texas, November. Association for Computational Linguistics.
  * [Artetxe et al., 2017]  Mikel Artetxe, Gorka Labaka, and Eneko Agirre.  2017\.  Learning bilingual word embeddings with (almost) no bilingual data.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451--462, Vancouver, Canada, July. Association for Computational Linguistics.
  * [Artetxe et al., 2018a]  Mikel Artetxe, Gorka Labaka, , and Eneko Agirre.  2018a.  Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations.  In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), pages 5012--5019.
  * [Artetxe et al., 2018b]  Mikel Artetxe, Gorka Labaka, and Eneko Agirre.  2018b.  A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789--798, Melbourne, Australia, July. Association for Computational Linguistics.
  * [Bamman and Crane, 2011]  David Bamman and Gregory Crane.  2011\.  Measuring historical word sense variation.  In Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries, JCDL ’11, page 1–10, New York, NY, USA. Association for Computing Machinery.
  * [Blei et al., 2003]  David M Blei, Andrew Y Ng, and Michael I Jordan.  2003\.  Latent dirichlet allocation.  Journal of Machine Learning Research, 3:2003.
  * [Bojanowski et al., 2017]  Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.  2017\.  Enriching word vectors with subword information.  Transactions of the Association for Computational Linguistics, 5:135--146.
  * [Brychcín et al., 2019]  Tomáš Brychcín, Stephen Taylor, and Lukáš Svoboda.  2019\.  Cross-lingual word analogies using linear transformations between semantic spaces.  Expert Systems with Applications, 135:287--295.
  * [Cook et al., 2014]  Paul Cook, Jey Han Lau, Diana McCarthy, and Timothy Baldwin.  2014\.  Novel word-sense identification.  In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1624--1635, Dublin, Ireland, August. Dublin City University and Association for Computational Linguistics.
  * [Deutsches Textarchiv, 2017]  Deutsches Textarchiv.  2017\.  Grundlage für ein referenzkorpus der neuhochdeutschen sprache.  Herausgegaben von der Berlin-Brandenburgischen Akademie der Wissenschaften. http://deutschestextarchiv.de.
  * [Eger and Mehler, 2016]  Steffen Eger and Alexander Mehler.  2016\.  On the linearity of semantic change: Investigating meaning variation via dynamic graph models.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 52--58, Berlin, Germany, August. Association for Computational Linguistics.
  * [Faruqui and Dyer, 2014]  Manaal Faruqui and Chris Dyer.  2014\.  Improving vector space word representations using multilingual correlation.  In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 462--471.
  * [Frermann and Lapata, 2016]  Lea Frermann and Mirella Lapata.  2016\.  A Bayesian model of diachronic meaning change.  Transactions of the Association for Computational Linguistics, 4:31--45.
  * [Gulordava and Baroni, 2011]  Kristina Gulordava and Marco Baroni.  2011\.  A distributional similarity approach to the detection of semantic change in the Google books ngram corpus.  In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pages 67--71, Edinburgh, UK, July. Association for Computational Linguistics.
  * [Hamilton et al., 2016a]  William L. Hamilton, Jure Leskovec, and Dan Jurafsky.  2016a.  Cultural shift or linguistic drift? comparing two computational measures of semantic change.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2116--2121, Austin, Texas, November. Association for Computational Linguistics.
  * [Hamilton et al., 2016b]  William L. Hamilton, Jure Leskovec, and Dan Jurafsky.  2016b.  Diachronic word embeddings reveal statistical laws of semantic change.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489--1501, Berlin, Germany, August. Association for Computational Linguistics.
  * [Hardoon et al., 2004]  David R. Hardoon, Sandor R. Szedmak, and John R. Shawe-Taylor.  2004\.  Canonical correlation analysis: An overview with application to learning methods.  Neural Computation, 16(12):2639--2664, December.
  * [Kim et al., 2014]  Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov.  2014\.  Temporal analysis of language through neural language models.  In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 61--65, Baltimore, MD, USA, June. Association for Computational Linguistics.
  * [Kutuzov et al., 2018]  Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal.  2018\.  Diachronic word embeddings and semantic shifts: a survey.  In Proceedings of the 27th International Conference on Computational Linguistics, pages 1384--1397, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics.
  * [Mihalcea and Nastase, 2012]  Rada Mihalcea and Vivi Nastase.  2012\.  Word epoch disambiguation: Finding how words change over time.  In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 259--263, Jeju Island, Korea, July. Association for Computational Linguistics.
  * [Mikolov et al., 2013]  Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  2013\.  Efficient estimation of word representations in vector space.  In Proceedings of workshop at ICLR. arXiv.
  * [Mitra et al., 2015]  Sunny Mitra, Ritwik Mitra, Suman Kalyan Maity, Martin Riedl, Chris Biemann, Pawan Goyal, and Animesh Mukherjee.  2015\.  An automatic approach to identify word sense changes in text media across timescales.  Natural Language Engineering, 21(5):773--798.
  * [Řehůřek and Sojka, 2010]  Radim Řehůřek and Petr Sojka.  2010\.  Software Framework for Topic Modelling with Large Corpora.  In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45--50, Valletta, Malta, May. ELRA.  http://is.muni.cz/publication/884893/en.
  * [Rosenfeld and Erk, 2018]  Alex Rosenfeld and Katrin Erk.  2018\.  Deep neural models of semantic shift.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 474--484, New Orleans, Louisiana, June. Association for Computational Linguistics.
  * [Schlechtweg and Walde, 2020]  Dominik Schlechtweg and Sabine Schulte im Walde.  2020\.  Simulating lexical semantic change from sense-annotated data.  In A. Ravignani, C. Barbieri, M. Martins, M. Flaherty, Y. Jadoul, E. Lattenkamp, H. Little, K. Mudd, and T. Verhoef, editors, The Evolution of Language: Proceedings of the 13th International Conference (EvoLang13).
  * [Schlechtweg et al., 2017]  Dominik Schlechtweg, Stefanie Eckmann, Enrico Santus, Sabine Schulte im Walde, and Daniel Hole.  2017\.  German in flux: Detecting metaphoric change via word entropy.  In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 354--367, Vancouver, Canada, August. Association for Computational Linguistics.
  * [Schlechtweg et al., 2018]  Dominink Schlechtweg, Sabine Schulte im Wlade, and Stefanie Eckmann.  2018\.  Diachronic usage relatedness (durel): A framework for the annotation of lexical semantic change.  In Proceedings of NAACL-HLT 2018, pages 169--174.
  * [Schlechtweg et al., 2019]  Dominik Schlechtweg, Anna Hätty, Marco Del Tredici, and Sabine Schulte im Walde.  2019\.  A wind of change: Detecting and evaluating lexical semantic change across times and domains.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 732--746, Florence, Italy, July. Association for Computational Linguistics.
  * [Schlechtweg et al., 2020]  Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi.  2020\.  SemEval 2020 Task 1: Unsupervised Lexical Semantic Change Detection.  In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics.
  * [Tahmasebi and Risse, 2017]  Nina Tahmasebi and Thomas Risse.  2017\.  Finding individual word sense changes and their delay in appearance.  In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 741--749, Varna, Bulgaria, September. INCOMA Ltd.
  * [Tahmasebi et al., 2018]  Nina Tahmasebi, Lars Borin, and Adam Jatowt.  2018\.  Survey of computational approaches to lexical semantic change.  arXiv preprint arXiv:1811.06278.
  * [Wang and McCallum, 2006]  Xuerui Wang and Andrew McCallum.  2006\.  Topics over time: A non-markov continuous-time model of topical trends.  In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, page 424–433, New York, NY, USA. Association for Computing Machinery.
  * [Wijaya and Yeniterzi, 2011]  Derry Tanti Wijaya and Reyyan Yeniterzi.  2011\.  Understanding semantic change of words over centuries.  In Proceedings of the 2011 International Workshop on DETecting and Exploiting Cultural DiversiTy on the Social Web, DETECT ’11, page 35–40, New York, NY, USA. Association for Computing Machinery.
  * [Xu and Kemp, 2015]  Yang Xu and Charles Kemp.  2015\.  A computational evaluation of two laws of semantic change.  In CogSci.