# Apparent molar volume anomaly in water-dimethyl sulfoxide liquid mixtures.
Molecular dynamics computer simulations

M. Aguilar  $|$ ${}^{\textsf{\footnotesize{\mbox{?}}}}$, H.Dominguez  $|$
${}^{\textsf{\footnotesize{\mbox{?}}}}$, O. Pizio  $|$
${}^{\textsf{\footnotesize{\mbox{?}}}}$ Corresponding author:
<EMAIL_ADDRESS>
(Received August 9, 2022, in final form October 19, 2022)

###### Abstract

We have studied the composition dependence of density of liquid water-DMSO
mixtures at different temperatures by using the isobaric-isothermal (NPT)
molecular dynamics computer simulations. The non-polarizable semi-flexible, P1
and P2 models for the DMSO molecule combined with the TIP4P-2005 water model
are considered. The excess mixing volume and the apparent molar volumes of the
species are reported. We have established that the P1-TIP4P-2005 model for the
mixture provides a very good description of the location of the minimum of
apparent molar volume for DMSO species indicating the anomaly. Most important
is that the temperature interval where the hydrophobic effect exists, is
correctly captured with this modelling, in contrast to the P2-TIP4P-2005
model.

Key words: molecular dynamics, water-dimethylsolfoxide mixtures, apparent
molar volume

###### Abstract

Äîñëäæåíî êîíöåíòðàöéíó çàëåæíñòü ãóñòèíè ðäêèõ ñóìøåé âîäà-äèìåòèëñóëüôîêñèä
(ÄÌÑÎ) ïðè ðçíèõ òåìïåðàòóðàõ ç âèêîðèñòàííÿì êîìïþòåðíîãî ìîäåëþâàííÿ ìåòîäîì
içîáàðèíî-içîòåðìiíî¿ ìîëåêóëÿðíî¿ äèíàìiêè. Ðîçãëÿäàþòüñÿ íåïîëÿðèçîâí
íàïâãíóê ìîäåë P1 P2 äëÿ ìîëåêóëè ÄÌÑÎ ó ïîäíàíí ç ìîäåëëþ TIP4P-2005 äëÿ
âîäè. Íàâåäåíî ðåçóëüòàòè îáèñëåíü äëÿ íàäëèøêîâîãî îáìó çìiøóâàííÿ òà âèäèìèõ
ìîëÿðíèõ îáìiâ êîìïîíåíò ñóìø. Âñòàíîâëåíî, ùî ìîäåëü P1-TIP4P-2005 äëÿ ö¿
ñóìø çàáåçïåó äóæå äîáðèé îïèñ ïîëîæåííÿ ìíìóìó âèäèìîãî ìîëÿðíîãî îáìó
êîìïîíåíòè ÄÌÑÎ, ÿêèé âêàçó íà àíîìàëþ. Íàéáëüø âàæëèâèì òå, ùî âèáðàíà
ìîäåëü, íà âäìíó âä ¿¿ âäïîâäíèêà P2-TIP4P-2005, ïðàâèëüíî ïåðåäáàà íàÿâíñòü
òåìïåðàòóðíîãî íòåðâàëó, â ÿêîìó ñíó ãäðîôîáíèé åôåêò.

Ключов слова: ìîëåêóëÿðíà äèíàìêà, ñóìiøi âîäà-äèìåòèëñóëüôîêñèä (ÄÌÑÎ),
âèäèìèé ìîëÿðíèé îáì

Liquid mixtures of dimethyl sulfoxide (DMSO) and water are widely used in
organic chemistry, chemical engineering, and biomedical applications. In this
rapid communication we refer to the static light scattering experiment
performed on aqueous solutions of dimethyl sulfoxide (DMSO) at various
compositions and at two temperatures, 20∘C and 50∘C [1]. In the concentration
range around 10 mol percent of DMSO, an abnormal maximum of scattered light
was detected, the intensity of which decreases with an increase of
temperature. This maximum has been attributed to the hydrophobic effect and
has been interpreted in terms of the minimum of apparent molar volume of DMSO
upon concentration. Inspired by this report, we have undertaken the study of
the composition dependence of density of water-DMSO liquid mixtures at 298.15
K, 318.15 K and 338.15 K by using molecular dynamics computer simulations
complementing our recent study performed at 298.15 K [2]. Our focus is on the
capability of simple non-polarizable models for DMSO and water to reproduce
the experimental observations by capturing peculiar composition interval in
the restricted temperature range.

Previously, binary mixtures of water with DMSO, in particular at a low DMSO
concentration, were the subject of study in different laboratories [3, 4, 5,
6, 7, 2]. Specifically, in [3, 4] the focus is on the composition trends of
certain properties at a single temperature 300 K. The NPT setup was used. On
the other hand, the NVT simulations at 298.15 K, 318.15 K and 338.15 K and at
a fixed DMSO molar fraction ($X_{\text{dmso}}=0.055$) were performed. The
experimental density values were assumed. In this study, we have chosen quite
popular, nonpolarizable, P1 and P2 models for DMSO species [8], from a huge
set of force fields for DMSO [9, 10], and well tested TIP4P-2005 water model
[11]. Technical details of our isobaric-isothermal molecular dynamics computer
simulations were described very recently in the previous publication on the
topic [2].

Figure 1: (Colour online) Panel a: Composition dependence of the density of
water-DMSO mixtures from the NPT MD simulations of P2-TIP4P-2005 model (solid
red lines), P1-TIP4P-2005 model (solid blue lines) in comparison with the
experimental data (dashed lines with triangles) at 298.15 K [12] and at 318.15
K (dashed line with squares) [13]. Panel b: Similar to the panel a, but at
338.15 K. Experimental data (dashed line with triangles) are from [14].

As common, the strategy of exploration is to describe a desired property on
composition and to capture its deviation from ideality as well. We do that for
the mixture density at each fixed temperature. A set of results is shown in
figure 1. The panel a of this figure refers to $T=298.15$ K and $T=318.15$ K,
whereas the panel b illustrates the results at $T=338.15$ K. This temperature
was also chosen in [5] due to the availability of experimental data. From the
inspection of the results, we observe that the P1-TIP4P-2005 and P2-TIP4P-2005
are quite accurate in the interval of composition from pure water,
$X_{\text{dmso}}=0$, up to $X_{\text{dmso}}\approx 0.2$. At a higher DMSO
content in the mixture, the simulation data begin to deviate from the
experimental results. The maximum density at a certain composition,
$X_{\text{dmso}}\approx 0.58$, is captured by both models in question.
Nevertheless, it can be seen that the P1-TIP4P-2005 model performs much better
than its P2-TIP4P-2005 counterpart. This latter model overestimates the
density in the interval above $X_{\text{dmso}}>$ 0.2. Moreover, the deviation
from the experimental points does not decrease while the temperature grows
from 298.15 K up to 338.15 K. The P1-TIP4P-2005 model exhibits a small
inaccuracy for density at “intermediate” compositions, but performs much
better than P2-TIP4P-2005. It is difficult to judge the precision of computer
simulations data at low $X_{\text{dmso}}$ values at the scale in the figure 1.
Better insights follow from the excess properties.

We describe geometric aspects of mixing of the species in terms of the excess
mixing volume defined as common, $\Delta
V_{\text{mix}}=V_{\text{mix}}-X_{d}V_{d}-(1-X_{d})V_{w}$, where
$V_{\text{mix}}$, $V_{d}$ and $V_{w}$ refer to the molar volume of the mixture
and of the individual components, DMSO and water, respectively. Apparently,
this property is not strongly dependent on temperature in the interval we deal
with. This follows from the experimental results shown in figure 2. The mixing
volume slightly decreases in magnitude upon increasing temperature, as
expected. In general terms, computer simulation results show that the excess
mixing volume is overestimated in the framework of models assumed. Thus, the
mixture of water and DMSO species from simulations is predicted to be more
non-ideal than its laboratory counterpart in almost entire composition range
at $T=$ 298.15 K. The $\Delta V_{\text{mix}}$ values from simulations decrease
in magnitude in agreement with experimental trends. However, the location of
the minimum along $X_{\text{dmso}}$ axis from simulations and experiments is
slightly different. Finally, one can observe that the performance of the
P1-TIP4P-2005 model is slightly better compared to the P2-TIP4P-2005 model.

Figure 2: (Colour online) A comparison of the composition dependence of the
excess mixing volume of water-DMSO mixtures for two DMSO models, P1 and P2,
combined with TIP4P-2005 water model, with the experimental data at 298.15 K
([12]—black triangles) and at 338.15K ([14]—black squares).

In order to discern the contribution of each species of the mixture into the
mixing volume, it is common to resort to the excess partial molar volumes as
we have done recently [2]. However, similar insights into the geometric
aspects of mixing on composition, both from experiments and simulations, can
be obtained by resorting to the notion of the apparent molar volume of
species, rather than to the partial molar volumes.

Figure 3: (Colour online) A comparison of the composition dependence of the
apparent molar volume of DMSO from simulations with the experimental data
marked as triangles from [12] at 298.15 K, from [13] at 318.15 K and from [14]
at 338.15 K.

The apparent molar volume for each species, according to the definition [15],
is: $V_{\phi}^{(w)}=V_{w}+\Delta V_{\text{mix}}/(1-X_{d})$ and
$V_{\phi}^{(d)}=V_{d}+\Delta V_{\text{mix}}/X_{d}$. We elaborated the
experimental density data from [12, 13] and the results from our simulations
to construct the plots shown in figures 3 and 4. The most important and
illuminating results for the apparent molar volume for DMSO species,
$V_{\phi}^{(d)}$, are shown in figure 3. Namely, we observe that both models
in question predict the existence of miminum at certain composition
$X_{\text{dmso}}$. The points describing the location of the minimum at
different temperatures from simulations and experiments are joined by
respective lines. Evolution of the minimum with temperature is appropriately
captured by the P1-TIP4P-2005 model in contrast to the P2-TIP4P-2005 model.
Moreover, the P1-TIP4P-2005 model correctly predicts that the hydrophobic
trends diminish with increasing temperature. Performance of the P2-TIP4P-2005
model in this aspect contradicts the experimental trends and is not
satisfactory. We have obtained the minimum for $V_{\phi}^{(d)}$ for this
model, even at a higher temperature equal to 358.15 K (the respective plot is
not shown for economy of space).

Concerning the behaviour of the apparent molar volume of water species,
$V_{\phi}^{(w)}$, upon composition (figure 4), it is worth to mention a rather
satisfatory performance of the P1-TIP4P-2005 model in the entire composition
range, especially for higher temperatures. The respective figure at 298.15 K
is given in [2] (figure 3d). It is necessary to mention, however, that the
experimental results for DMSO-rich (more specifically, extremely rich)
mixtures are difficult to obtain precisely, because high-purity DMSO is an
exceedingly hygroscopic solvent, the discussion of this issue is given in
[12].

To conclude, we have established that the P1 and P2 models for DMSO combined
with the TIP4P-2005 water model yield a minimum of the apparent molar volume
for DMSO species at low values for $X_{\text{dmso}}$. This behavior is the
manifestation of hydrophobic effects in these mixtures at a specific
composition interval. However, it appears that the predictions from
P1-TIP4P-2005 model agree better with the experimental trends than the ones
from P2-TIP4P-2005 model. Moreover, the P1-TIP4P-2005 model predicts the
existence of a minimum of $V_{\phi}^{(d)}$ for DMSO within the correct
temperature interval from 298.15 K up to 338.15 K. At this highest
temperature, the minimum almost disappears, as deduced from experimental data.
By contrast, P2-TIP4P-2005 model does not provide an accurate estimate for a
peculiar composition interval and does not yield a decay for the
hydrophobicity trends on temperature. In various publications this model is
claimed to be the best, see for example [18]. Here, we show that this
conclusion is valid in certain aspects at a room temperature only. In
addition, it is important to mention that the combination of the OPLS model
for DMSO with TIP4P-2005 model for water does not provide a correct
composition dependence of density at room temperature, cf. figure 2 of [2] and
at temperatures of this study. In consequence, the anomaly of the apparent
molar volume is not captured by this type of model at all. These results are
available upon request.

Figure 4: (Colour online) Panels a and b: A comparison of the composition
dependence of the apparent molar volume of water from simulations with the
experimental data (triangles [13]) at 318.15 K and [14] at 338.15 K. Similar
plot at 298.15 K is shown as figure 3d of [2].

Previously, we explored the minimum of the methanol apparent molar volume on
composition for water-methanol mixtures [16] at 298.15 K, see also [17] for
the interpretation of the observed phenomena. This kind of system is the
mixture of two hydrogen bonded liquids. By contrast, in the systems of the
present study there is no bonding between the DMSO solutes in water.
Apparently, the temperature trends should be slightly different for these two
classes of systems.

In addition, it seems intriguing to investigate the evolution of trends of
hydrophobicity in water-DMSO mixtures upon changing the external pressure as
well. One can be guided by important experimental observations, see e.g., [19,
20]. More generally, from the simulations perspective it is challenging to
construct a more complete map of anomalies, in pressure, temperature and
composition variables, that would involve, for example, the temperature of
maximum density and the minimum of isothermal compressibility, even by using
the non-polarizable models. These data would serve as a useful benchmark for
the following development of polarizable models. In the case of water-DMSO
mixtures, certain progress has been reached at this level of modelling in [21,
22]. However, the BK3 polarizable water model seems to be a more convenient
starting point, because the principal anomalies of pure water are quite well
captured in this framework [23, 24]. Some of these issues are under study in
our laboratory.

## References

  * [1] Rodnikova M. N., Zakharova Y. A., Solonina I. A., Sirotkin D. A., Russ. J. Phys. Chem. A, 2012, 86, 892, doi:10.1134/S0036024412050317.
  * [2] Aguilar M., Dominguez H., Pizio O., Condens. Matter Phys., 2022, 25, 33202, doi:10.5488/CMP.25.33202.
  * [3] Banerjee S., Roy S., Bagchi B., J. Phys. Chem. B, 2010, 114, 12875, doi:10.1021/jp1045645.
  * [4] Roy S., Banerjee S., Biyani N., Jana B., Bagchi B., J. Phys. Chem. B, 2011, 115, 685, doi:10.1021/jp109622h.
  * [5] Mancera R. L., Chalaris M., Refson K., Samios J., Phys. Chem. Chem. Phys., 2004, 6, 94,
doi:10.1039/B308989D.

  * [6] Mandumpal J. B., Kreck C. A., Mancera R. L., Phys. Chem. Chem. Phys., 2011, 13, 3839,
doi:10.1039/c0cp02326d.

  * [7] Gujt J., Cazares Vargas E., Pusztai L., Pizio O., J. Mol. Liq., 2017, 228, 71, doi:10.1016/j.molliq.2016.09.024.
  * [8] Luzar A., Chandler D., J. Chem. Phys., 1993, 98, 8160, doi:10.1063/1.464521.
  * [9] Chalaris M., Marinakis S., Dellis D., Fluid Phase Equilib., 2008, 267, 47, doi:10.1016/j.fluid.2008.02.019.
  * [10] Idrissi A., Marekha B., Barj M., Jedlovszky P., J. Phys. Chem. B, 2014, 118, 8724, doi:10.1021/jp503352f.
  * [11] Abascal J. L. F., Vega C., J. Chem. Phys., 2005, 123, 234505, doi:10.1063/1.2121687.
  * [12] Egorov G. I., Makarov D. M., Russ. J. Phys. Chem. A, 2009, 83, 693, doi:10.1134/S003602440905001X.
  * [13] Grande M., Alvarez Juliá J., García M., Marschoff C. M., J. Chem. Thermodyn., 2007, 39, 1049,
doi:10.1016/j.jct.2006.12.012.

  * [14] Cowie J. M., Toporowski P. M., Can. J. Chem., 1961, 39, 2240, doi:10.1139/v61-296.
  * [15] Torres R. B., Marchiore A., Volpe P., J. Chem. Thermodyn., 2006, 38, 526, doi:10.1016/j.jct.2005.07.012.
  * [16] Cruz Sanchez M., Aguilar M., Pizio O., Condens. Matter Phys., 2020, 23, 34601, doi:10.5488/CMP.23.34601.
  * [17] Chechko V. E., Gotsulsky V. Ya., Malomuzh M. P., Condens. Matter Phys., 2013, 16, 23006,
doi:10.5488/CMP.16.23006.

  * [18] Roy D., Kovalenko A., J. Comput.-Aided Mol. Des., 2019, 33, 905, doi:10.1007/s10822-019-00238-4.
  * [19] Kanno H., Kajiwara K., Miyata K., J. Chem. Phys., 2010, 132, 194503, doi:10.1063/1.3429973.
  * [20] Taschin A., Bartolini P., Fanetti S., Lapini A., Citroni M., Righini R., Bini R., Torre R., J. Phys. Chem. Lett., 2020, 11, 3063, doi:10.1021/acs.jpclett.0c00363.
  * [21] Bachmann S. J., van Gunsteren W. F., J. Phys. Chem. B, 2014, 118, 10175, doi:10.1021/jp50356951.
  * [22] Bachmann S. J., van Gunsteren W. F., J. Chem. Phys., 2014, 141, 22D515, doi:10.1063/1.4897976.
  * [23] Kiss P. T., Baranyai A., J. Chem. Phys., 2012, 137, 084506, doi:10.1063/1.474641.
  * [24] Kiss P. T., Baranyai A., J. Chem. Phys., 2014, 140, 154505, doi:10.1063/1.4871390.

Ukrainian 0

Àíîìàëÿ âèäèìîãî ìîëÿðíîãî îáìó ðäêèõ ñóìiøåé âîäà-äèìåòèëñóëüôîêñèä.
Êîìï’þòåðíå ìîäåëþâàííÿ ìåòîäîì ìîëåêóëÿðíî¿ äèíàìiêè M. Àãëàð  $|$
${}^{\textsf{\footnotesize{\mbox{?}}}}$, Å. Äîìíãåñ  $|$
${}^{\textsf{\footnotesize{\mbox{?}}}}$, O. Ïçî  $|$
${}^{\textsf{\footnotesize{\mbox{?}}}}$

${}^{\textsf{\footnotesize 1}}$ íñòèòóò õì¿, Íàöîíàëüíèé àâòîíîìíèé óíâåðñèòåò
Ìåõêî, Circuito Exterior, 04510, Ìåõêî, Ìåêñèêà ${}^{\textsf{\footnotesize
2}}$ íñòèòóò ìàòåðàëîçíàâñòâà, Íàöîíàëüíèé àâòîíîìíèé óíâåðñèòåò Ìåõêî,
Circuito Exterior, 04510, Ìåõêî, Meêñèêà


11institutetext: Mastercard AI, Gurgaon, India
11email: $\\{$avi.chawla, nidhi.mulay, vikas.bishnoi,
<EMAIL_ADDRESS>22institutetext: Indian Institute of
Technology (BHU), Varanasi, India
22email<EMAIL_ADDRESS>
# A Comparative Study of Transformers on
Word Sense Disambiguation

Avi Chawla 11 Nidhi Mulay 11 Vikas Bishnoi 11 Gaurav Dhama 11
Anil Kumar Singh 22

###### Abstract

Recent years of research in Natural Language Processing (NLP) have witnessed
dramatic growth in training large models for generating context-aware language
representations. In this regard, numerous NLP systems have leveraged the power
of neural network-based architectures to incorporate sense information in
embeddings, resulting in Contextualized Word Embeddings (CWEs). Despite this
progress, the NLP community has not witnessed any significant work performing
a comparative study on the contextualization power of such architectures. This
paper presents a comparative study and an extensive analysis of nine widely
adopted Transformer models. These models are BERT, CTRL, DistilBERT, OpenAI-
GPT, OpenAI-GPT2, Transformer-XL, XLNet, ELECTRA, and ALBERT. We evaluate
their contextualization power using two lexical sample Word Sense
Disambiguation (WSD) tasks, SensEval-2 and SensEval-3. We adopt a simple yet
effective approach to WSD that uses a _k_ -Nearest Neighbor (kNN)
classification on CWEs. Experimental results show that the proposed techniques
also achieve superior results over the current state-of-the-art on both the
WSD tasks.

###### Keywords:

Word Sense Disambiguation Transformers

##  1 Introduction

Developing powerful language representations technique has been a key area of
research in Natural Language Processing (NLP). The employment of effective
representational models has also been an essential contributor in improving
the performance of many NLP systems. Word vectors or embeddings are fixed-
length vectors that are proficient in capturing the semantic properties of
words. Emerging from a simple Neural Network-based Word2Vec model and recently
transitioning to Contextualised Word Embeddings (CWEs), the advancements have
consistently brought a revolution to every NLP sub-domain. The introduction of
naive Word2Vec model not only brought an unprecedented increase in the
performance of a wide variety of downstream tasks such as Machine Translation,
Sentiment Analysis, and Question Answering, but it also laid the foundation
for a majority of Natural Language Understanding (NLU) architectures that we
use today.

Recent attempts in NLU research have fundamentally focused on generating
context-aware word representations, i.e., embeddings that take into account
the polysemous nature of words. Polysemy refers to the changes in the meaning
of a word when the context around it changes. One related task in NLP is Word
Sense Disambiguation (WSD) which deals with the automatic recognition of the
correct sense of a word appearing in a specific context. WSD is an essential
component of any NLP system as it helps in generating better semantic
representations of words.

Contribution: The Transformer architectures implemented in the HuggingFace
framework [19] implicitly provide a model for WSD. We test the performance of
nine such pre-trained models on WSD and extensively analyse each one of them.
These models are BERT [7], OpenAI-GPT [12], OpenAI-GPT2 [13], CTRL [8],
DistilBERT [15], Transformer-XL [6], XLNet [20], ELECTRA [4] and ALBERT [9].
This comprehensive study helps us in comparing the ability of different
transformer models in incorporating polysemy in embeddings, i.e., their power
of segregating various senses of a word in the word-vector space. Through our
experiments, we also report a new state-of-the-art on both the lexical sample
WSD datasets we experimented on, i.e., SensEval-2 and SensEval-3.

Note: Although the prime use of the CTRL, OpenAI-GPT, and OpenAI-GPT2 model is
Natural Language Generation (NLG), we still include them in our comparative
study. We do this to determine the extent to which these models consider
polysemy while carrying out NLG as their primary objective.

##  2 Related Work

Word Sense Disambiguation (WSD) is an old and common problem in NLP. In the
early days of Artificial Intelligence, WSD was conceived as a fundamental task
of Machine Translation [17]. Since then, advancements in NLP have led to the
development of a variety of WSD systems. Recent attempts in this respect have
tried to tackle the problem by introducing the concept of sense embeddings.
For instance, [1] induced sense embeddings using a pre-training based
approach. [10] proposed methods that focus on generating sense embeddings
using pre-trained word embeddings such as Glove vectors [11]. [16] proposed
‘Sense2Vec’, which utilized the part-of-speech and named entity tag
information to distinguish between different meanings of a word. An extensive
survey on further ideas and research on sense representations of words is
given by [3].

Most of the recent approaches have leveraged the power of Deep Learning to
build WSD systems. [2] proposed an auto-encoder-based approach that goes from
the target word embedding back to the word definition. The method proposed by
[21] revolved around the computation of sentence context vector for ambiguous
words. They adopted a _k_ -Nearest Neighbor (kNN) [5] based approach for
classification of ambiguous words. In contrast to all the approaches described
above, [18] proposed a simple yet effective approach for the classification of
ambiguous words. Instead of using any pre-trained embeddings like the glove
embeddings, they used the BERT [7] to obtain Contextualised Word Embeddings
(CWEs). For prediction, they used a kNN based approach. The use of BERT also
achieved new state-of-the-art results over previously proposed approaches.

##  3 Datasets

In our experiments, we use two widely-adopted lexical sample corpora available
for WSD, SensEval-2 and SensEval-3. Both come with a train and test set to
train and evaluate a WSD model. The words in these datasets are annotated with
the sense identifiers defined in WordNet 3.0. A brief overview of both
datasets is shown in Table 1. To evaluate the performance of a WSD model, we
refer to the testing scripts from the comprehensive framework of
[14]111https://github.com/getalp/UFSAC..

Table 1: An overview of the datasets used for the study of nine Transformer
Models. Average Sentence Length has been rounded off to the nearest integer.

Dataset |  No. of Sentences |  Avg. Sentence Length |  No. of Distinct Sense Identifiers |  No. of Sense Embeddings |  Distinct Words | Nouns | Adjectives | Verbs
---|---|---|---|---|---|---|---|---
SensEval-3 Train | 7860 | 30 | 285 | 9280 | 172 | 3632 | 308 | 3879
SensEval-3 Test | 3944 | 30 | 260 | 4520 | 168 | 1777 | 153 | 1999
SensEval-2 Train | 8611 | 29 | 783 | 8742 | 187 | 3492 | 1400 | 2559
SensEval-2 Test | 4328 | 29 | 620 | 4385 | 184 | 1737 | 702 | 1800

##  4 Experiments

For our experimentation, we take inspiration from a simple yet effective kNN
based approach on CWEs to WSD proposed by [18]. This approach uses a cosine
similarity-based distance metric for the classification of ambiguous words in
the test data. In a nutshell, we obtain the CWEs of all the ambiguous words in
the training data by providing their respective contexts to one of the nine
contextualization approaches. While classifying an ambiguous word in a test
sentence, a kNN classification approach is used, with cosine similarity
between the CWE of the ambiguous word and all its instances observed during
training as the similarity metric. Such an experiment is carried out for six
different values of the hyper-parameter $k\in\\{1,3,5,7,10,11\\}$ in the kNN
classifier. An ambiguous word is classified to the sense with the maximum
number of nearest neighbors in the “ _k_ ” nearest neighbors.

We propose few additions to existing approach to improve the overall
performance. The first improvement lies in the way data is collected. [18]
used the lemma of every word in a sentence to obtain sentences from the
dataset. This, in some cases, generated inappropriate sentences such as: _“Nor
be this feeling only provoked by the sight or the thought of art, he write.”_
instead of _“Nor is this feeling only provoked by the sight or the thought of
art, he wrote.”_. Another sentence collected by their method and our method is
_“The art_critic critic be thus bind to consider with care what standard of
comparison should be use.”_ and _“The art critic is thus bound to consider
with care what standards of comparison should be used.”_ respectively. The
sentences collected by their method lack a proper grammatical sense and
structure. We improve this by collecting the lemma only for ambiguous words
and surface form for every other word in the sentence.

Our second improvement is an empirical finding. While obtaining the CWEs from
BERT, they treated the concatenation of the output of the last four layers of
BERT as the word embeddings. Instead, we used only the final layer of BERT to
obtain the embedding of a word.

##  5 Experimental Results

To study and analyze each of the transformer models in detail, we conduct
three rounds of experiments. In the first round, we carry out the task of WSD
on nine pre-trained Transformer architectures using the kNN approach described
above and compare their performance on two WSD Lexical Sample tasks. Further,
to visualize each model’s power to separate different senses of a word in
their embedding space, we draw the t-SNE plots for the CWEs generated by the
transformer models. Lastly, we provide a qualitative analysis by examining the
correct predictions and the wrong predictions made by each of our WSD models.

###  5.1 Contextualized Embeddings

To compare the models based on their contextualization power, we perform the
task of WSD using the language representation provided by them. Table 2 lists
the results obtained by each of these models for $k\in\\{1,3,5,7,10,11\\}$.
The BERT model achieved a new state-of-the-art on the SensEval-2 and
SensEval-3 tasks [18]. The modification also facilitates the DistilBERT model
in beating the current state-of-the-art on SensEval-3 dataset. Also, it
becomes evident from the results obtained by the ALBERT and the DistilBERT
model that they highly resemble BERT’s architecture. Through our observations,
we also state that the employment of DistilBERT and ALBERT in place of BERT
could take off a major overhead of the training time without incurring a
significant loss in the performance.

An unexpected drop in performance is observed for the XLNet and Transformer-XL
model compared to the other well-performing models. Though both the models are
effective on various NLP tasks using their powerful recurrence-based
Transformer architectures, we notice that the model still underperforms.

Coming towards the end, the three NLG models — OpenAI-GPT, OpenAI-GPT2, and
CTRL also performed poorly. CTRL and OpenAI-GPT2 performed slightly better
than the Most Frequent Sense (MFS) baseline on SensEval-3 dataset. In addition
to this, they even failed to beat the MFS baseline on the SensEval-2 dataset,
demonstrating that they are ineffective in capturing polysemy.

Table 2: Results(F1%) of all the Transformer models for different values of
_k_ on the _k_ -Nearest Neighbor classification approach vs. the Most Frequent
Sense (MFS) baseline and current state-of-the-art results. The Best results
for each model are underlined, and the best result on a particular dataset is
bold. The previous state-of-the-art is in italics.

Model | SensEval-2 | SensEval-3
---|---|---
| k=1 | k=3 | k=5 | k=7 | k=10 | k=11 | k=1 | k=3 | k=5 | k=7 | k=10 | k=11
BERT | 76.02 | 76.78 | 76.62 | 76.62 | 76.76 | 76.81 | 79.40 | 80.31 | 80.49 | 80.96 | 80.75 | 80.72
DistilBERT | 74.81 | 75.64 | 75.36 | 75.43 | 75.41 | 75.43 | 78.62 | 79.71 | 80.05 | 80.15 | 80.23 | 80.07
ALBERT | 74.84 | 75.33 | 75.43 | 74.98 | 75.07 | 75.07 | 77.94 | 78.93 | 79.44 | 79.60 | 79.71 | 79.57
XLNet | 64.74 | 66.24 | 66.48 | 66.45 | 66.38 | 66.45 | 69.97 | 70.64 | 71.50 | 71.78 | 71.42 | 71.42
ELECTRA | 65.98 | 65.88 | 65.98 | 66.10 | 66.07 | 65.95 | 69.45 | 70.10 | 70.82 | 71.14 | 71.11 | 71.01
GPT | 59.80 | 60.84 | 61.29 | 61.24 | 61.15 | 61.54 | 65.63 | 67.65 | 68.51 | 69.29 | 69.58 | 69.60
Trans-XL | 53.36 | 54.35 | 55.01 | 55.18 | 55.01 | 54.45 | 62.07 | 62.82 | 63.32 | 63.99 | 63.50 | 63.50
CTRL | 52.39 | 53.64 | 54.28 | 54.45 | 54.49 | 54.82 | 58.09 | 60.38 | 60.92 | 61.50 | 61.78 | 61.63
GPT2 | 50.96 | 53.57 | 53.88 | 53.88 | 53.86 | 53.80 | 57.03 | 59.83 | 60.92 | 61.21 | 61.29 | 61.19
MFS | 54.79 | 58.95
kNN [18] | _76.52_ | _80.12_

###  5.2 Sense-space analysis using t-SNE plots

To understand and interpret a model’s power to segregate different senses of a
word in the embedding space, we draw the t-SNE plots of the embeddings
obtained for the word ‘bank’ from the training data of SensEval-3 for each of
the nine models. Figure 2 represents the t-SNE plots thus obtained. Sub-figure
2.(j) represents the interpretable meanings of the senses represented in the
t-SNE plots along with their respective frequencies in the SensEval-3 training
corpus. We exclude any sense with a frequency of less than three from the
t-SNE plot for clarity. It is evident from the t-SNE plot of OpenAI-GPT2 that
it hardly distinguishes between different senses, and we see this as a
possible reason why its accuracy is very close to the MFS baseline. As all the
sense embeddings are in the vicinity of each other, the model hardly learns
any decision boundary for sense classification. Therefore, the approach
performs slightly better than the MFS baseline. We can draw a similar
conclusion by observing the t-SNE plots of CTRL and Transformer-XL, implying
that the NLG objective of OpenAI-GPT2 and CTRL hardly takes polysemy into
account.

|  |  |  |
---|---|---|---|---
(a) BERT | (b) DistilBERT | (c) ALBERT | (d) XLNet | (e) ELECTRA

|  |  |  |
---|---|---|---|---
(f) GPT | (g) Trans-XL | (h) CTRL | (i) GPT2 | (j) Sense Labels

Figure 1: t-SNE plots of different senses of ‘bank’ and their contextualized
embeddings. The legend(shown separately in Sub-figure (j)) shows a short
description of the respective WordNet sense and the frequency of occurrence in
the training data. We used the SensEval-3 training dataset for obtaining these
plots.

On the other hand, plots obtained for the OpenAI-GPT, ELECTRA, and XLNet
models depict that these models capture polysemy relatively better than the
NLG models. They do stress a little on making a distinction between different
senses of a word. Lastly, models that performed the best among the nine models
we experimented on are BERT, DistilBERT, and ALBERT. These models possess
exceptional proficiency in identifying polysemy, which is evident from their
t-SNE plots as well as their accuracy on both the datasets.

###  5.3 Additional Experiments

Part-of-Speech information of a word has been regarded as a crucial influencer
in determining its possible sense. [18] proposed a POS-sensitive approach to
WSD for the determination of the sense of an ambiguous word. Their experiments
resulted in an accuracy lift of approximately 2-3 F1 on the SemEval datasets.
Still, this approach did not prove to be beneficial for models trained on
SensEval-2 and SensEval-3 datasets. This was because each word in these
datasets is annotated with only one POS. Aligning our analysis on similar
lines, as a final set of error analyses in our comparative study, we attempt
to understand each model’s behavior to different POS tags. We estimate the
percentage of correct classifications made by each model for Nouns, Verbs, and
Adjectives in the two datasets. This is presented in Table 4 for $\emph{k}=1$.

For SensEval-2 dataset, we observe that each model was able to classify both
Nouns and Adjectives correctly to a considerable extent. But, for Verbs, a
difference of approximately 15-20% was observed from that of Nouns and
Adjectives. A similar drop in classification accuracy was observed in
SensEval-3 for Adjectives. Each model classified Nouns and Verbs in this
dataset to a reasonable extent but underperformed during the classification of
Adjectives.

Table 3: The percentage of correct classifications made by the models for Nouns, Verbs and Adjectives on SensEval-2 Test and SensEval-3 Test data. The values are for _k_ =1. The best results on a particular POS tag have been marked in bold. Model | SensEval-2 | SensEval-3
---|---|---
|  Nouns |  Verbs |  Adj |  Nouns |  Verbs |  Adj
BERT |  81.64 |  67.22 |  81.62 |  78.17 |  82.33 |  56.86
DistilBERT |  81.00 |  65.61 |  80.06 |  76.14 |  82.86 |  54.25
ALBERT |  82.38 |  64.33 |  80.06 |  75.63 |  81.87 |  55.56
XLNet |  71.50 |  56.39 |  66.81 |  67.92 |  73.58 |  48.37
ELECTRA |  75.30 |  54.83 |  68.80 |  67.02 |  73.16 |  50.98
OpenAI-GPT |  70.87 |  47.44 |  64.10 |  64.15 |  68.06 |  52.29
Tranformer-XL |  61.95 |  42.67 |  59.54 |  60.89 |  64.36 |  47.06
CTRL |  62.29 |  41.11 |  56.84 |  57.01 |  60.55 |  39.87
OpenAI-GPT2 |  55.79 |  42.72 |  60.11 |  51.55 |  63.42 |  40.52

##  6 Conclusion

In this paper, we evaluated the contextualisation power of nine pre-trained
Transformer Models on a WSD task. We presented a comparative study on each
model’s power to capture polysemy in the embeddings they generate. To
accomplish this, we used a kNN based approach to WSD proposed by [18] and
proposed two improvements in their method that also accompanied us in
establishing a new state-of-the-art on WSD Lexical Sample Task of SensEval-2
and SensEval-3. We concluded our study by stating that BERT, DistilBERT, and
ALBERT models prove out to be most effective on the WSD task solely based on
text encodings they provide. We found these models to possess an extraordinary
potential to identify a word’s different senses compared to all the other
models.

As future work, we plan to make use of POS information as well to classify an
ambiguous word. We firmly believe that incorporating POS information in WSD
could be very useful and further increase the performance of these models. In
addition to this, we also believe that fine-tuning these models could be a
potential area of focus. In our experiments, we leveraged the pre-trained
models as provided by the authors, and a bit of fine-tuning could be
beneficial.

## References

  * [1] Bartunov, S., Kondrashkin, D., Osokin, A., Vetrov, D.: Breaking sticks and ambiguities with adaptive skip-gram. In: Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS). pp. 130–138. Cadiz, Spain (2016)
  * [2] Bosc, T., Vincent, P.: Auto-encoding dictionary definitions into consistent word embeddings. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 1522–1532. Association for Computational Linguistics, Brussels, Belgium (Oct-Nov 2018). https://doi.org/10.18653/v1/D18-1181, https://www.aclweb.org/anthology/D18-1181
  * [3] Camacho-Collados, J., Pilehvar, M.T.: From word to sense embeddings: A survey on vector representations of meaning. CoRR abs/1805.04032 (2018), http://arxiv.org/abs/1805.04032
  * [4] Clark, K., Luong, M.T., Le, Q.V., Manning, C.D.: ELECTRA: Pre-training text encoders as discriminators rather than generators. In: ICLR (2020), https://openreview.net/pdf?id=r1xMH1BtvB
  * [5] Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Transactions on Information Theory 13(1), 21–27 (1967)
  * [6] Dai, Z., Yang, Z., Yang, Y., Carbonell, J.G., Le, Q.V., Salakhutdinov, R.: Transformer-xl: Attentive language models beyond a fixed-length context. CoRR abs/1901.02860 (2019), http://arxiv.org/abs/1901.02860
  * [7] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [8] Keskar, N.S., McCann, B., Varshney, L., Xiong, C., Socher, R.: CTRL - A Conditional Transformer Language Model for Controllable Generation. arXiv preprint arXiv:1909.05858 (2019)
  * [9] Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: A lite bert for self-supervised learning of language representations (2019)
  * [10] Pelevina, M., Arefiev, N., Biemann, C., Panchenko, A.: Making sense of word embeddings. In: Proceedings of the 1st Workshop on Representation Learning for NLP. pp. 174–183. Berlin, Germany (2016)
  * [11] Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP). pp. 1532–1543 (2014), http://www.aclweb.org/anthology/D14-1162
  * [12] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training. (2018)
  * [13] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019)
  * [14] Raganato, A., Camacho-Collados, J., Navigli, R.: Word sense disambiguation: A unified evaluation framework and empirical comparison. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. pp. 99–110. Valencia, Spain (2017)
  * [15] Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter (2019)
  * [16] Trask, A., Michalak, P., Liu, J.: sense2vec - A fast and accurate method for word sense disambiguation in neural word embeddings. CoRR abs/1511.06388 (2015), http://arxiv.org/abs/1511.06388
  * [17] Weaver, W.: Machine Translation of Languages: Fourteen Essays. Technology Press of M.I.T., Cambridge, MA, USA (1955)
  * [18] Wiedemann, G., Remus, S., Chawla, A., Biemann, C.: Does bert make any sense? interpretable word sense disambiguation with contextualized embeddings. ArXiv abs/1909.10430 (2019)
  * [19] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Brew, J.: Huggingface’s transformers: State-of-the-art natural language processing. ArXiv abs/1910.03771 (2019)
  * [20] Yang, Z., Dai, Z., Yang, Y., Carbonell, J.G., Salakhutdinov, R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. CoRR abs/1906.08237 (2019), http://arxiv.org/abs/1906.08237
  * [21] Yuan, D., Richardson, J., Doherty, R., Evans, C., Altendorf, E.: Semi-supervised word sense disambiguation with neural models. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. pp. 1374–1385. Osaka, Japan (2016)