# Do Language Models Know the Way to Rome?

Bastien Liétard
University of Lille
bastien.lietard.etu
@univ-lille.fr
&Mostafa Abdou and Anders Søgaard
University of Copenhagen
<EMAIL_ADDRESS>
###### Abstract

The global geometry of language models is important for a range of
applications, but language model probes tend to evaluate rather local
relations, for which ground truths are easily obtained. In this paper we
exploit the fact that in geography, ground truths are available beyond local
relations. In a series of experiments, we evaluate the extent to which
language model representations of city and country names are isomorphic to
real-world geography, e.g., if you tell a language model where Paris and
Berlin are, does it know the way to Rome? We find that language models
generally encode limited geographic information, but with larger models
performing the best, suggesting that geographic knowledge can be induced from
higher-order co-occurrence statistics.

##  1 Introduction

Language models (LMs) are fundamental building blocks in state-of-the-art
natural language processing. These models are trained to predict words in
context, and as a side product, they learn to compress higher-order co-
occurrence statistics to represent the distributional properties of words and
phrases.

It is well-documented that the representations of modern language models
encode some syntactic Tenney et al. (2019) and semantic knowledge (Reif et al.
(2019)), as well as some real-world knowledge Davison et al. (2019); Petroni
et al. (2019); Jiang et al. (2020); Roberts et al. (2020), i.e., some
knowledge base relations can be extracted directly from language models. Much
of this information was available in older language models such as Word2Vec,
too Mikolov et al. (2013).

The probes that have been designed for the above studies, however, only probe
for one-dimensional information (e.g., part of speech or semantic class) or
very local relations (e.g., is Daniel Kehlmann a writer?). It is an open
question whether higher-order co-occurrence statistics found in textual
corpora can induce representations show isomorphism to human mental
representations of entities and relations. To test this we operationalize a
dataset of geographic information. Geographic knowledge is a domain in which
we can go beyond local relations, since we can access global representations
of geographic knowledge (to the extent it is isomorphic to the physical
world).

Cross-lingual language model alignment provides additional motivation for
probing for geographic knowledge: We know that language models of different
languages are sometimes near-isomorphic Søgaard et al. (2018); Vulić et al.
(2020), and it is tempting to think that this is because language models
reflect real-world structures, and that the (occasional) near-isomorphism of
language models is simply the result of the fact that we use language(s) to
talk about the same world. Is the isomorphism of language models a result of
them being in part isomorphic to the physical world?

Why would language models potentially encode physical geometry? Higher-order
co-occurrence statistics can be surprisingly informative. Copenhagen and Malmö
are in different countries, but connected by a bridge, often referred to as
the bridge between Copenhagen and Malmö. Other cities belong to the same
municipalities and their names therefore co-occur with the name of the
municipality. Other cities were, for instance, potentially impacted by the
same natural disaster or part of the same development project. By distilling
thousands, or maybe hundreds of thousands, of such co-occurrences, we
conjecture that language models might be able to induce somewhat fine-grained
maps of physical geometry.

In this work, we study the extent to which geographical information is encoded
by language models. Based on geographic information from multiple sources, we
train probing models on language model representations and evaluate their
ability to predict the locations of cities and countries, as well as their
ability to predict (one-dimensional) population numbers and (local) neighbor
relations, relative to control probe performance levels.

#### Contributions

We collect geographic information from multiple sources and design
experimental protocols to probe language models for such information. We
evaluate two BERT models Devlin et al. (2019) of different sizes, two RoBERTa
models Liu et al. (2019) of different sizes, as well as GPT-2 Radford et al.
(2019) and Word2Vec Mikolov et al. (2013) across three different tasks and
various protocols. Our results show that modern language models do not encode
much more geographic information than Word2Vec, but that larger (non-
autoregressive) language models encode more information than their smaller
counterparts, suggesting that such information is nevertheless available
through higher-order co-occurrence statistics.

##  2 Related Work

Word representations based on distributional statistics have been theorised to
capture a wide range of information Schütze (1992). To evaluate this, a
considerable body of literature has made use of semantic similarity,
relatedness, and analogy datasets Agirre et al. (2009); Bruni et al. (2012);
Baroni et al. (2014); Faruqui et al. (2014); Hill et al. (2015); Drozd et al.
(2016); Abdou et al. (2018). Asking a broader question, Rubinstein et al.
(2015) investigated the types of semantic information which are encoded by
different classes of word embedding models, finding that taxonomic properties
(such as animacy) are well-modelled. In a similar direction, Collell Talleda
and Moens (2016) and Lucy and Gauthier (2017) draw on semantic norm datasets
to test how well these models can encode a range of perceptual and conceptual
features.

In the context of the neural language models, several recent works such as
Davison et al. (2019) and Petroni et al. (2019) have attempted to extract
factual and commonsense knowledge from them by posing knowledge base triplets
as close statements which are used to query the models. Most related to this
work, Forbes et al. (2019) investigate whether LMs can learn physical
commonsense through language. They find that LM representations do indeed
encode information regarding object properties (e.g., bananas are yellow), and
affordances (e.g. bananas can be eaten) but they do not to capture the more
subtle interplay between the two.

Studies investigating the geometry of word representations have focused on
intrinsic dimensionality and subspaces Yaghoobzadeh and Schütze (2016); Coenen
et al. (2019), embedding concentration in narrow cones and anisotropy Mimno
and Thompson (2017); Ethayarajh (2019), and comparisons to the geometry of
cognitive measurements or perceptual spaces Abnar et al. (2019); Abdou et al.
(2021). In this work we investigate the degree of isomorphism between language
model representations of geographic location names and their real-world
counterparts. See Vulić et al. (2020) for a general discussion of isomorphism
across language models, what explains this, and what it depends on.

##  3 Methodology

We probe language model representations of city and country names. Below, we
propose three classification/regression probing tasks, as an well as an
analysis based on (relational) similarity.

###  3.1 Probing tasks

#### Task 1: Predicting geo-coordinates

We probe LMs’ representation about whether they convey information about the
actual position of the location. To this extent, we train a model to predict
GPS coordinates (latitude and longitude) given the location’s representation
through language models. We evaluate prediction error by computing the
distance in kilometers between the predicted GPS point and the expected point,
and compute the average of error distances. Since we use linear models
(Lasso111We also include an MLP regressor for comparison.) to predict geo-
coordinates, we effectively evaluate the isomorphism of language model
representations to the physical world. Our target variable is two-dimensional,
and we evaluate non-local relations.

#### Task 2: Predicting population sizes

In the physical world, cities are characterized not only by their location,
but also by their size and population. There is indeed a major difference
between Shanghai (27M people) and Worcester (101K people). Given the
representation of a country name or a city name, we train a probe to predict
the population living in the location. We evaluate the performance by
computing the mean squared error. This is a one-dimensional attribute included
here for comparison.

#### Task 3: Predicting neighboring countries

Borders between countries are also an important geographic relation that is
closely related to a country’s international policy. It also provides
information about the continental situation ; indeed, a country with no land
borders must be an island. To probe language models for neighboring relations,
we train an classifier to predict whether two countries share a border or not,
given the pair of representations. We report the probe accuracy. Neighboring
relations are local and included here for comparison.

###  3.2 Control tasks and scores

Hewitt and Liang (2019) have shown the importance of a control task to balance
the probe accuracy. We construct our control tasks by randomly permuting the
target variables and train the model on this randomly-permuted dataset. We
repeat this control task 10 times and take the mean error/accuracy.

#### Probe classifier selectivity

For the country borders classification task, we define the probe selectivity
as the difference between the probe accuracy on the original task and the
accuracy on the control task Hewitt and Liang (2019).

#### Probe error reduction

Similarly, we define the probe error reduction, which measures how well the
probe performs on the original task compared to the control task, by comparing
positive error measures of both tasks through error reduction, i.e., the
proportion of control task error that is reduced in our probe.

$PER=1-\frac{\textit{Probing task error}}{\textit{Control task error}}$

Less formally, if the PER is between 0 and 1, its upper bound, it indicates to
what extent the probe is doing better on the original task than on the control
task ; a negative PER indicates the probe is doing worse on the original task
than on the control task. We use PER for regression tasks rather than absolute
error or MSE, because figures are more easily interpretable when properly
baselined. Note that if absolute error is 0, PER is 1; if absolute error is as
high as the random baseline, PER is 0. Under the assumption performance is
better than random, PER thus ranges between 0 and 1.

###  3.3 Similarity Analysis

For each language models, we compute the cosine similarity between each pair
of cities. Our hypothesis is that cities belonging to the same country will be
more similar than two cities from different countries. We report both average
similarities (cities in the same country and cities in different countries)
and also compute the histogram of the distribution of these two sets of
similarities.

##  4 Experimental Settings and Data

#### Language models

We compare three different state-of-the-art contextualized LMs, namely BERT
(Devlin et al., 2019), RoBERTa (Liu et al., 2019) and GPT-2 (Radford et al.,
2019). A major difference between GPT-2 and the two other models is that it is
trained to be unidirectional (_i.e._ using next-token language modelling),
whereas BERT and RoBERTa are trained to be bidirectionnal (_i.e._ using masked
language modelling). For BERT and RoBERTa, we used both base ($L=12$ layers
and $D=768$ output dimensions) and large ($L=24$ layers and $D=1024$ output
dimensions) version, so we can study the influence of the model’s size. We
probe widely-used pre-trained versions of these models, as they have been
trained in their respective original papers. Both versions of BERT and RoBERTa
have been pre-trained on the BookCorpus (800M words) and English Wikipedia
(2.5B words). GPT-2 is also trained on the Book Corpus.

#### Geographic data

We get a list of names of cities around the world having a population of 100K
or more people, with their GPS coordinates (Latitude and Longitude), their
country and their population, from a MaxMind
database222https://www.kaggle.com/max-mind/world-cities-database. Country geo-
coordinates correspond to the centroid of each country. We obtain population
size data from the United Nations’ Human Development Data
Center333http://hdr.undp.org/en/data, given in millions of inhabitants. Our
datasets are then composed of 3527 cities and 249 countries and territories.

#### Representations extraction

Since the LMs provide contextualised token representations, we provide three
linguistically-basic contexts : He lives in $X$, She moved to $X$ and I come
from $X$, replacing $X$ with either a city name or a country name. For each
context, we extract the representation corresponding to the name by averaging
the hidden states of the last 4 layers of the LMs. Following Bommasani et al.
(2020), we use a mean pooling of subwords’ representations when necessary.

#### Word2Vec

To compare static and contextualized models, we also use pre-trained Word2Vec
static representations learned over the Google News dataset (150B words),
provided by the gensim444https://radimrehurek.com/gensim/ library. The
representational vectors are in 300 dimensions.

Task | Probe | Dataset | Word2Vec | BERT | BERT-L | GPT-2 | RoBERTa | RoBERTa-L
---|---|---|---|---|---|---|---|---
GPS | MLP | city | 0.666 | 0.479 | 0.585 | 0.424 | 0.466 | 0.517
country | 0.57 | 0.424 | 0.58 | 0.422 | 0.405 | 0.468
Lasso | city | 0.498 | 0.454 | 0.555 | 0.364 | 0.398 | 0.466
country | 0.395 | 0.394 | 0.465 | 0.349 | 0.378 | 0.454
Population | MLP | city | 0.507 | 0.694 | 0.704 | 0.624 | 0.696 | 0.707
country | 0.618 | 0.59 | 0.63 | 0.568 | 0.528 | 0.505
Lasso | city | 0.628 | 0.539 | 0.55 | 0.447 | 0.403 | 0.531
country | -0.242 | 0.116 | 0.276 | 0.372 | 0.078 | 0.116
Table 1: Probe error reduction. “BERT-L” and “RoBERTa-L” denote large versions
of BERT and RoBERTa. Column name “Probe” designates the different regression
architectures used for probing. For GPS coordinates, error reduction is
computed with average error in kilometers, whereas it is computed using MSE
for Population. Control tasks are performed on $n=10$ trials, and the control
error is obtained by averaging all trials errors.

#### Probe models

For the classification problem (task 3), we use a 100-units single-layered MLP
as a classifier. As a probe for regression task (1 and 2), we train both a
Lasso regressor with a L2-penalty of $\alpha$ and a single hidden layer MLP
with 100 units. We arbitrarily choose $\alpha=1$, except for task 1 for which
we remarked that $\alpha=0.5$ significantly decreased the training time
without affecting to much the error ratio. No other hyperparameters-tuning was
done, since we are not interested in designing the best task-specific models.
Probes are trained on a random split of 80% of the dataset, and evaluate on
the 20% remaining.

| Accuracy | Probe
---|---|---
Model | Prb. | Ctrl. | Selectivity
Word2Vec | 0.849 | 0.49 | 0.36
BERT | 0.856 | 0.51 | 0.34
BERT-L | 0.873 | 0.51 | 0.37
GPT-2 | 0.808 | 0.51 | 0.3
RoBERTa | 0.817 | 0.51 | 0.31
RoBERTa-L | 0.843 | 0.52 | 0.32
Table 2: Probe accuracy (Prb.), control accuracy (Ctrl.) and probe selectivity
for country borders prediction.

##  5 Results

In Table 1, probe error reduction scores are displayed for both regression
tasks, namely Task 1 and 2. The full list of performances (error distances in
kilometers) for Task 1 is provided in the appendix A.1. In Appendix A.2, error
values for population prediction with countries are reported. Since the number
of countries is relatively small, we perform a 5-fold cross-validation with
this dataset.

When predicting a location’s GPS coordinates, all models have an error
reduction score significantly larger than 0, across both families of
regressors and across cities and countries. The Word2Vec model especially has
often the lowest error and a high PER, showing no real weakness compared to
contextualised models. GPT-2 has the lowest PER and the worst error value,
across almost all tasks and probes. The two regressors globally lead to the
same error values, even though MLP allows a slightly bigger error reduction
than Lasso. We can also note that, for both BERT and RoBERTa, increasing model
size (moving from base models to large), one increases the gap between the
performance on the task and on the random control, reducing overall error.
Finally, we can also see that the models are generally not very accurate for
predicting GPS coordinates, leading to an average error of 2’500-5’000
kilometers. It is relatively accurate continent-wise, but not precise.

In sum, we can conclude that it is non-trivial to learn linear maps from
language models to physical world geo-coordinates, presumably because
representations are non-isomorphic. On the other hand, the improvements from
larger models seem to suggest that geographical knowledge is available from
higher-order co-occurrence statistics. Static model also provide a strong
baseline compared to contextualised models.

Figure 1: Representational similarity histograms for BERT, BERT large, RoBERTa and Word2Vec. The similarity is computed through cosine similarity. Blue is the curve of intra-country (same country) similarity ; orange is the one for inter-countries (different countries) similarity. Model | intra | inter | gap
---|---|---|---
Word2Vec | 0.51 | 0.24 | 0.27
BERT | 0.83 | 0.77 | 0.06
BERT-L | 0.77 | 0.65 | 0.12
GPT-2 | 0.90 | 0.89 | 0.01
RoBERTa | 0.92 | 0.89 | 0.03
RoBERTa-L | 0.94 | 0.93 | 0.01
Table 3: Average representational similarities between cities in the same
country (intra), in different countries (inter) and the gap between them.

In Task 2, the gap between the two regressors is more noticeable compared to
Task 1: MLP exhibits much higher error reductions between the original task
and the random control, and generally allow for lower error numbers. We remark
that our Lasso model is particularly ineffective for country population
prediction, leading only to low PER and high errors. On this task and dataset,
GPT-2 random control error is high, leading to an outlying PER for this LM,
even if its probing-task error is still the biggest. The scheme observed in
Task 1 between base and large version of BERT and RoBERTa seems to hold, with
only an exception for countries with MLP. This overall indicates that a one-
dimensional attribute is easier to learn than a two-dimensional space, giving
more expressive models an upper hand.

Table 2 shows the probe’s accuracy and selectivity on Task 3, predicting
country borders. This is a local relation, included for comparison. Probe
accuracy scores exhibit little variance, and control accuracies are almost
identical, close to 0.5, which is expected for random binary classification.
All language models allow exhibit probe accuracies above $80\%$. This
indicates the insufficiency of probing only for local relations.

The results of the similarity analysis on city names are displayed in Table 3
and in Figure 1. It is to be observed that representations are highly similar
in contextualized LMs, especially compared to those from Word2Vec, which is in
accordance with Ethayarajh (2019) results about anisotropy in higher layers of
LMs. On the other hand, we observe that representations of cities in the same
country are always more similar than cities in different countries, even if
the gap is small for some models. Finally, we can note that no conclusion can
be made about the effect of increasing the LM’s size in this analysis: the gap
is bigger in large version of BERT than in the base version, but this relation
does not hold for RoBERTa.

##  6 Discussion

Tasks 1 to 3 suggest that the amount of geographic knowledge learned by the
language models is limited. On the one hand, trained predictors for both GPS
coordinates and population are of poor quality, making average errors of
thousands of kilometers or hundreds of millions of people. On the other hand,
the probes are doing significantly better on these tasks than during random
control, and information about country borders have been shown by results of
Task 3 to be embedded in pre-trained language models’ representations. This
suggests that approximate geographic positions and their neighborhood are
learned during training by these models. Similarity analysis also indicates
that pre-trained models have learned to associate representations of cities in
the same country.

In the end, language models, whether they are static or contextualized, seem
to be able to extract limited geographical knowledge from higher-order co-
occurrence statistics. The size of the language models, as well as the amount
of data they were trained on, seem to be important for how well they encode
this information. Increasing the model’s size leads to best performances and
error reduction scores, indicating a better learning of geographic knowledge.
Overall, context-sensitive language models do not show significantly better
performances than Word2Vec. Finally, our experiments demonstrate the
importance of probing for global, multi-dimensional relations that require
near-isomorphism and are not easily saturated.

In the future, it would be of interest to test models finetuned on corpora
which contain a wealth of geographic information (e.g. a corpus of atlases) in
order to evaluate a) whether representational alignment to real-world
geography can be affected by seeing factual expressions such as “Italy,
France, and Switzerland share borders." and b) how exactly representations of
entities (e.g. France) alter when the model sees factual statements as the one
above.

## Acknowledgements

We thank the anonymous reviewers. Mostafa Abdou was funded by a Google Focused
Research Award. We used data created by MaxMind, available from
http://www.maxmind.com/.

## References

  * Abdou et al. (2021) Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. 2021.  Can language models encode perceptual structure without grounding? a case study in color.
  * Abdou et al. (2018) Mostafa Abdou, Artur Kulmizev, and Vinit Ravishankar. 2018.  Mgad: Multilingual generation of analogy datasets.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Abnar et al. (2019) Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Willem Zuidema. 2019.  Blackbox meets blackbox: Representational similarity and stability analysis of neural language models and brains.  _arXiv preprint arXiv:1906.01539_.
  * Agirre et al. (2009) Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009.  A study on similarity and relatedness using distributional and wordnet-based approaches.
  * Baroni et al. (2014) Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014.  Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 238–247.
  * Bommasani et al. (2020) Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020.  Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4758–4781, Online. Association for Computational Linguistics.
  * Bruni et al. (2012) Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012.  Distributional semantics in technicolor.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 136–145.
  * Coenen et al. (2019) Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. 2019.  Visualizing and measuring the geometry of bert.  _arXiv preprint arXiv:1906.02715_.
  * Collell Talleda and Moens (2016) Guillem Collell Talleda and Marie-Francine Moens. 2016.  Is an image worth more than a thousand words? on the fine-grain semantic differences between visual and linguistic representations.  In _Proceedings of the 26th International Conference on Computational Linguistics_ , pages 2807–2817. ACL.
  * Davison et al. (2019) Joe Davison, Joshua Feldman, and Alexander Rush. 2019.  Commonsense knowledge mining from pretrained models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1173–1178, Hong Kong, China. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Drozd et al. (2016) Aleksandr Drozd, Anna Gladkova, and Satoshi Matsuoka. 2016.  Word embeddings, analogies, and machine learning: Beyond king-man+ woman= queen.  In _Proceedings of coling 2016, the 26th international conference on computational linguistics: Technical papers_ , pages 3519–3530.
  * Ethayarajh (2019) Kawin Ethayarajh. 2019.  How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 55–65. Association for Computational Linguistics.
  * Faruqui et al. (2014) Manaal Faruqui, Jesse Dodge, Sujay K Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. 2014.  Retrofitting word vectors to semantic lexicons.  _arXiv preprint arXiv:1411.4166_.
  * Forbes et al. (2019) Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019.  Do neural language representations learn physical commonsense?  _arXiv preprint arXiv:1908.02899_.
  * Hewitt and Liang (2019) John Hewitt and Percy Liang. 2019.  Designing and interpreting probes with control tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2733–2743, Hong Kong, China. Association for Computational Linguistics.
  * Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015.  Simlex-999: Evaluating semantic models with (genuine) similarity estimation.  _Computational Linguistics_ , 41(4):665–695.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know?  _Transactions of the Association for Computational Linguistics_ , 8:423–438.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Lucy and Gauthier (2017) Li Lucy and Jon Gauthier. 2017.  Are distributional representations ready for the real world? evaluating word vectors for grounded perceptual meaning.  _arXiv preprint arXiv:1705.11168_.
  * Mikolov et al. (2013) Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  In _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_.
  * Mimno and Thompson (2017) David Mimno and Laure Thompson. 2017.  The strange geometry of skip-gram with negative sampling.  In _Empirical Methods in Natural Language Processing_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language models are unsupervised multitask learners.
  * Reif et al. (2019) Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019.  Visualizing and measuring the geometry of bert.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model?  _arXiv preprint arXiv:2002.08910_.
  * Rubinstein et al. (2015) Dana Rubinstein, Effi Levi, Roy Schwartz, and Ari Rappoport. 2015.  How well do distributional models capture different types of semantic knowledge?  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 726–730.
  * Schütze (1992) Hinrich Schütze. 1992.  Dimensions of meaning.  In _SC_ , pages 787–796.
  * Søgaard et al. (2018) Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018.  On the limitations of unsupervised bilingual dictionary induction.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 778–788, Melbourne, Australia. Association for Computational Linguistics.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.  BERT rediscovers the classical NLP pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601, Florence, Italy. Association for Computational Linguistics.
  * Vulić et al. (2020) Ivan Vulić, Sebastian Ruder, and Anders Søgaard. 2020.  Are all good word vector spaces isomorphic? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3178–3192, Online. Association for Computational Linguistics.
  * Yaghoobzadeh and Schütze (2016) Yadollah Yaghoobzadeh and Hinrich Schütze. 2016.  Intrinsic subspace evaluation of word embedding representations.  _arXiv preprint arXiv:1606.07902_.

##  Appendix A Appendices

###  A.1 Task 1 performances

On the following tables, Prb. stands for “probing task” and Ctl. for “control
task”.

Model | MLP | Lasso
---|---|---
Prb. | Ctl. | Prb. | Ctl.
Word2Vec | 2612 | 7825 | 3447 | 6870
BERT | 4195 | 8057 | 3780 | 6920
BERT-L | 3315 | 7997 | 3077 | 6911
GPT-2 | 4613 | 8011 | 4498 | 7070
RoBERTa | 4278 | 8007 | 4148 | 6894
RoBERTa-L | 3876 | 8029 | 3686 | 6903
Mean error distances in kilometers for GPS prediction of cities Model | MLP | Lasso
---|---|---
Prb. | Ctrl. | Prb. | Ctrl.
Word2Vec | 3738 | 8695 | 4379 | 7234
BERT | 4950 | 8598 | 4944 | 8152
BERT-L | 3603 | 8578 | 4488 | 8394
GPT-2 | 5111 | 8840 | 5658 | 8684
RoBERTa | 5522 | 9275 | 5036 | 6091
RoBERTa-L | 4764 | 8960 | 4433 | 8125
Mean error distances in kilometers for GPS prediction of countries

###  A.2 Task 2 performances on countries

Reported results are mean squared errors. Therefore, mean absolute errors are
at most of the magnitude of few hundreds (of millions) of people.

Model | MLP | Lasso
---|---|---
Prb. | Ctrl. | Prb. | Ctrl.
Word2Vec | 12142 | 31815 | 22112 | 17810
BERT | 16382 | 39952 | 26583 | 30063
BERT-L | 17166 | 46365 | 22559 | 31174
GPT-2 | 15264 | 3566 | 32130 | 51171
RoBERTa | 15266 | 32338 | 26375 | 28592
RoBERTa-L | 16390 | 33112 | 22927 | 25923
Mean squared error distances (in millions of people) for prediction of
country’s population.


11institutetext: School of Information Science and Engineering, Southeast
University, Nanjing 210096, China 22institutetext: Key Laboratory of Child
Development and Learning Science of Ministry of Education, Southeast
University, Nanjing 210096, China 33institutetext: School of Biological
Science and Medical Engineering, Southeast University, Nanjing 210096, China
33email: {cheng.lu<EMAIL_ADDRESS>
∗ Equal Contributions. ✉ Corresponding Authors.

# Time-Frequency Transformer: A Novel Time Frequency Joint Learning Method for
Speech Emotion Recognition

Yong Wang∗, 11 Cheng Lu∗,✉, 2233 Yuan Zong✉, 2233 Hailun Lian 1133 Yan Zhao
1133 Sunan Li 1133

###### Abstract

In this paper, we propose a novel time-frequency joint learning method for
speech emotion recognition, called Time-Frequency Transformer. Its advantage
is that the Time-Frequency Transformer can excavate global emotion patterns in
the time-frequency domain of speech signal while modeling the local emotional
correlations in the time domain and frequency domain respectively. For the
purpose, we first design a Time Transformer and Frequency Transformer to
capture the local emotion patterns between frames and inside frequency bands
respectively, so as to ensure the integrity of the emotion information
modeling in both time and frequency domains. Then, a Time-Frequency
Transformer is proposed to mine the time-frequency emotional correlations
through the local time-domain and frequency-domain emotion features for
learning more discriminative global speech emotion representation. The whole
process is a time-frequency joint learning process implemented by a series of
Transformer models. Experiments on IEMOCAP and CASIA databases indicate that
our proposed method outdoes the state-of-the-art methods.

###### Keywords:

Speech emotion recognition Time-frequency domain Transformer.

##  1 Introduction

Speech Emotion Recognition (SER) aims to use computers to automatically
analyze and recognize the emotional state in human speech[1], which has become
a hotspot in many fields, e. g., affective computing and Human-Computer
Interaction (HCI) [2]. For SER task, how to obtain a speech emotion
representation with high discrimination and strong generalization is a key
step to realize the superior performance of speech emotion classification [3],
[4].

In order to recognition speech emotions well, early SER works mainly combined
some low-level descriptor (LLD) features or their combinations [5], e. g.,
Mel-Frequency Cepstral Coefficients (MFCC), Zero-Crossing Rate, and Pitch,
with classifiers [6], e. g., k-Nearest Neighbor (KNN) and Support Vector
Machine (SVM), for emotion prediction. Furthermore, with the rapid development
of deep learning, high-dimensional speech emotion features generated by deep
neural networks (DNN), e. g., Convolutional Neural Network (CNN) [7] and
Recurrent Neural Network (RNN) [8], have emerged on SER and achieved superior
performance. Currently, the input features of DNNs are mainly based on
spectrogram, e. g., magnitude spectrogram [9] and Mel-spectrogram [10], which
are the time-frequency representations of speech signals.

The time and frequency domains of the spectrogram contain rich emotional
information. To excavate them, a practical approach is joint time-frequency
modeling strategy with the input features of spectrograms [10], [3]. Among
these methods, combining CNN and RNN structure, i. e., CNN$+$LSTM, is a
classic method, which utilizes CNN and RNN to encode the information in
frequency and time domains, respectively. For instance, Satt et al. [11]
combined CNN with a special RNN (i. e., LSTM) to model the time-frequency
domain of emotional speech. Wu et al. [12] proposed a recurrent capsules
network to extract time-frequency emotional information from the spectrogram.

Although current time-frequency joint learning methods have achieved certain
success on SER, they still suffer from two issues. The first one is that they
usually shared the modeling both in time and frequency domains, ignoring the
specificality of the respective domains. For instance, the time-frequency
domain shares a uniform size convolution kernel in CNN [9] and a uniform-scale
feature map is performed on the time-frequency domain in RNN [11]. Therefore,
separate modeling of time-domain and frequency-domain information should be
considered to ensure the specificity and integrity of the encoding of time-
frequency domain information. The other issue is that only some low-level
feature fusion operations (e. g., splicing and weighting) are adopted in time
frequency joint learning process, leading to poor discriminativeness of fusion
features [10]. This indicates that the effective fusion of emotional
information in the time-frequency domain is also the key for time-frequency
joint learning.

To cope with the above issues, we propose a novel Transformer-based time
frequency domain joint learning method for SER, called Time-Frequency
Transformer, which consisting of three modules, i. e., Time Transformer
module, Frequency Transformer module, and Time-Frequency Transformer module,
as shown in Figure. 1. Firstly, Time Transformer module and Frequency
Transformer module are designed to model the local emotion correlations
between frames and inside frequency bands respectively, which can ensure the
integrity of the emotion information modeling in both time and frequency
domains. Then, we also propose a Time-Frequency Transformer module to excavate
the time-frequency emotional correlations through the local time-domain and
frequency-domain emotion features for learning more discriminative global
speech emotion representation. The whole process is a time-frequency joint
learning process. Overall, our contributions can be summarized as the
following three points:

  * •

We propose a novel time frequency joint learning method based on Transformers
(i. e., Time-Frequency Transformer), which can effectively excavate the local
emotion information both in time frames and frequency bands to aggregate
global speech emotion representations.

  * •

We propose a Time Transformer and Frequency Transformer to ensure the
integrity of modeling time-frequency local emotion representations.

  * •

Our proposed Time-Frequency Transformer outperform on the state-of-the art
methods on IEMOCAP database and CASIA database.

Figure 1: Overview of Time-Frequency Transformer for speech emotion
recognition. It mainly consists of three modules, i. e., Time Transformer
module (light orange background), Frequency Transformer module (light blue
background), and Time-Frequency Transformer module (light green background).

##  2 Proposed Method

In this section, we will introduce our proposed Time-Frequency Transformer
shown in Figure. 1, including three modules, i.e., Time Transformer module,
Frequency Transformer module and Time-Frequency Transformer module.

###  2.1 Time Transformer Module

The role of Time Transformer is to capture the local emotion correlations
across time frames for time domain encoding of emotional speech. This module
utilizes the time encoder to reduce the dimensionality of the input feature
$\boldsymbol{x}$ of the model and extract part of the emotional information in
time-domain of $\boldsymbol{x}$. Then, MSA mechanism of the Transformer
encoder is used to calculate the emotional correlation between frames, making
the model focus on emotion related frames and reducing the impact of emotion
unrelated frames on speech emotion recognition.

Basically, the time encoder consists of a 2D convolutional layer, a 2D batch
normalization layer, an activation function layer, a 2D convolutional layer, a
2D batch normalization layer and an activation function layer in sequence. We
convolute the input spectrogram feature $\boldsymbol{x}\in\mathbb{R}^{b\times
c\times f\times d}$ frame by frame and extract temporal emotional information.
Each convolution layer is followed by a batch normalization layer and an
activation function layer to speed up model training and improve the model’s
nonlinear representation capabilities. The output of the second activation
function layer is $\boldsymbol{\hat{x}}^{\prime}\in\mathbb{R}^{b\times
c1\times(f/4)\times d}$, where _b_ and _f_ represent the number of samples
selected for each training and the number of Mel-filter banks, _c_ and _c1_
represent the number of input channels of the spectrogram feature and the
number of output channels of the last convolutional layer and d is the number
of frames of the spectrogram feature.The process can be represented as

$\boldsymbol{\hat{x}}^{\prime}=Act(BN(C_{t}(Act(BN(C_{t}(\boldsymbol{x}))))))$
(1)

where $C_{t}(\cdot)$ is the convolutional operation in each frame by a 2D
convolutional layer, $BN(\cdot)$ and $Act(\cdot)$ is batch normalization and
activation function operations respectively.

We utilizes a Transformer encoder to perform temporal attention focusing on
$\boldsymbol{\hat{x}}^{\prime}$. The Transformer encoder consists of a MSA
sub-layer and a feed-forward neural network using residual connections.The
Transformer encoder applies multiple attention heads to achieve the model
parallel training. By dividing the input into multiple feature subspaces and
applying a self-attention mechanism in the subspaces, the model can be trained
in parallel while capturing emotional information.

We take the mean value of the convolution output channel dimension of
$\boldsymbol{\hat{x}}^{\prime}$ to get
$\boldsymbol{s}^{\prime}\in\mathbb{R}^{b\times(f/4)\times d}$ and then
transpose $\boldsymbol{s}^{\prime}$ to get
$\boldsymbol{\hat{s}}^{\prime}\in\mathbb{R}^{b\times d\times(f/4)}$. The
feature $\boldsymbol{\hat{s}}^{\prime}\in\mathbb{R}^{b\times d\times(f/4)}$ is
used as the input of the time-domain Transformer encoder. We can represent the
process as

$\boldsymbol{m}=LN(MSA(\boldsymbol{\hat{s}}^{\prime})+\boldsymbol{\hat{s}}^{\prime})$
(2) $\boldsymbol{\hat{q}}=LN(MLP(\boldsymbol{m})+\boldsymbol{m})$ (3)

where $MSA(\cdot)$, $LN(\cdot)$ and $MLP(\cdot)$ is MSA, layer normalization
and feed-forward neural network respectively in a Transformer encoder.
Besides, $\boldsymbol{m}$ is the output of $\boldsymbol{\hat{s}}^{\prime}$
after the MSA and layer normalization of a Transformer encoder and
$\boldsymbol{\hat{q}}\in\mathbb{R}^{b\times d\times(f/4)}$ is the output of a
Transformer encoder. The output feature $\boldsymbol{\hat{q}}$ will be used as
one of the inputs of time-frequency Transformer module after linear mapping.

###  2.2 Frequency Transformer Module

This module has a similar structure to the time Transformer module, both
containing a Transformer encoder. The difference between the two modules is
that this module performs 2D convolutional operation on the input features
$\boldsymbol{x}$ frequency band by frequency band, which is used for reducing
dimension of feature and extracting frequency-domain emotional information. We
use 2D convolutional operation $C_{f}(\cdot)$ on each frequency band of
feature $\boldsymbol{x}$. Then, we normalize the feature and activate the
feature using the activation function. Finally get the output
$\boldsymbol{\hat{x}}^{\prime\prime}\in\mathbb{R}^{b\times c1\times
f\times(d/4)}$ of the last activation function layer. The operations can be
represented as

$\boldsymbol{\hat{x}}^{\prime\prime}=Act(BN(C_{f}(Act(BN(C_{f}(\boldsymbol{x}))))))$
(4)

We take the mean value of the convolution output channel dimension of
$\boldsymbol{\hat{x}}^{\prime\prime}$ to get
$\boldsymbol{\hat{s}}^{\prime\prime}\in\mathbb{R}^{b\times f\times(d/4)}$.
Then, we use a Transformer encoder to calculate the emotional correlation
between frequency bands for $\boldsymbol{\hat{s}}^{\prime\prime}$, so that the
model focuses on the emotional related parts in frequency domain, reducing the
impact of emotional unrelated frequency bands on speech emotion recognition.
The operations can be represented as

$\boldsymbol{n}=LN(MSA(\boldsymbol{\hat{s}}^{\prime\prime})+\boldsymbol{\hat{s}}^{\prime\prime})$
(5) $\boldsymbol{\hat{k}}=LN(MLP(\boldsymbol{n})+\boldsymbol{n})$ (6)

where $\boldsymbol{n}$ is the output of $\boldsymbol{\hat{s}}^{\prime\prime}$
after the MSA and layer normalization of the Transformer encoder and
$\boldsymbol{\hat{k}}\in\mathbb{R}^{b\times f\times(d/4)}$ is the output of
the Transformer encoder. After linear mapping, $\boldsymbol{\hat{k}}$ will be
used as one of the inputs to the time-frequency Transformer module.

###  2.3 Time-Frequency Transformer Module

Time-Frequency Transformer Module aims to aggregate the local emotion encoding
in time and frequency domains generated by Time Transformer and Frequency
Transformer into the global emotion representation in time-frequency domain.
Therefore, the main function of this module is to use the time-frequency
domain features of the speech that have been weighted by the MSA mechanism to
use the Multi-head Attention(MA) mechanism [13] to further weight, so that the
model pays more attention to the emotion-related segments in time-frequency
domain, thereby improving accuracy of speech emotion recognition.

First, We utilize the 2D convolution operation $C(\cdot)$ to encode
$\boldsymbol{x}$ in the time-frequency domain. After that, we normalize and
activate the feature to obtain $\boldsymbol{\hat{x}}\in\mathbb{R}^{b\times
c1\times(f/4)\times(d/4)}$ . The process can be represented as

$\boldsymbol{\hat{x}}=Act(BN(C(Act(BN(C(\boldsymbol{x}))))))$ (7)

Then, we take the mean value of $\boldsymbol{\hat{x}}$ in the convolution
output channel dimension to get
$\boldsymbol{v}\in\mathbb{R}^{b\times(f/4)\times(d/4)}$. We linearly map
$\boldsymbol{\hat{q}}$ and $\boldsymbol{\hat{k}}$ to get
$\boldsymbol{q}\in\mathbb{R}^{b\times(f/4)\times(d/4)}$ and
$\boldsymbol{k}\in\mathbb{R}^{b\times(f/4)\times(d/4)}$ respectively. The
$\boldsymbol{q}$, $\boldsymbol{k}$ and $\boldsymbol{v}$ are used as the input
of the CO-Transformer encoder. Compared with an original Transformer encoder,
we replace the MSA sub-layer in the encoder with a MA sub-layer to obtain a
CO-Transformer encoder. A CO-Transformer encoder is composed of a MA sub-layer
and a feed-forward neural network using residual connection.The main
difference between MSA and MA is that when doing attention calculations, the
inputs $\boldsymbol{Q}$ (Query Vector), $\boldsymbol{K}$ (Keyword Vector), and
$\boldsymbol{V}$ (Value Vector) of MSA are the same, but the inputs
$\boldsymbol{Q}$ ,$\boldsymbol{K}$, and $\boldsymbol{V}$ of MA are different.
The process can be represented as

$\boldsymbol{p}=LN(MA(\boldsymbol{q},\boldsymbol{k},\boldsymbol{v})+\boldsymbol{v})$
(8) $\boldsymbol{y}=LN(MLP(\boldsymbol{p})+\boldsymbol{p})$ (9)

where $MA(\cdot)$ is MA and $\boldsymbol{p}$ is the output of the input
features after the MA and layer normalization of the CO-Transformer encoder.
Besides, $\boldsymbol{y}\in\mathbb{R}^{b\times(f/4)\times(d/4)}$ is the output
of the CO-Transformer encoder. The obtained $\boldsymbol{y}$ is the input of
the classifier and finally obtain the predicted emotional category.

The classifier consists of a pooling layer and a fully connected layer. The
main function of the pooling layer is to reduce the feature dimension. The
pooling layer takes the mean and standard deviation in the frequency-domain
dimension of $\boldsymbol{y}$ and concatenates them to get
$\boldsymbol{\hat{y}}\in\mathbb{R}^{b\times(d/2)}$, which can be represented
as

$\boldsymbol{\hat{y}}=Pool(\boldsymbol{y})$ (10)

where $Pool(\cdot)$ is the pooling operation.

We calculate the prediction probability of all emotions through a fully
connected layer and ultimately obtain
$\boldsymbol{\hat{y}}^{\prime}\in\mathbb{R}^{b\times c}$, where _c_ is the
number of emotion categories of the corpus. We take the emotion with the
largest prediction probability as the predicted emotion of the model and
optimize the model by reducing the cross-entropy loss _Loss_ between the
predicted emotion label
$\boldsymbol{\hat{y}}^{\prime\prime}\in\mathbb{R}^{b\times c}$ and the true
emotion label $\boldsymbol{z}\in\mathbb{R}^{b\times c}$. The operations can be
represented as

$\boldsymbol{\hat{y}}^{\prime}=FC(\boldsymbol{\hat{y}})$ (11)
$\boldsymbol{\hat{y}}^{\prime\prime}=Softmax(\boldsymbol{\hat{y}}^{\prime})$
(12)
$Loss=CrossEntropyLoss(\boldsymbol{\hat{y}}^{\prime\prime},\boldsymbol{z}))$
(13)

where $FC(\cdot)$, $Softmax(\cdot)$ and $CrossEntropyLoss(\cdot)$ is the fully
connected layer operation, Softmax function [14] and Cross-Entropy loss
function [15] respectively.

##  3 Experiments

###  3.1 Experimental Databases

Extensive experiments are conducted on two well-known speech emotion
databases, _i.e._ , IEMOCAP [16] and CASIA [17].

  * •

IEMOCAP is an audio-visual database released by the Sail Laboratory of the
University of Southern California. This database consists of five dyadic
sessions, and each session is performed by a male actor and a female actor in
improvised and scripted scenarios to obtain various emotions (angry, happy,
sad, neutral, frustrated, excited, fearful, surprised, disgusted, and others).
We select the audio samples in improvised scenario, including 2280 sentences
belonging to four emotions (angry, happy, sad, neutral) for experiments.

  * •

CASIA is a mandarin database collected by the Institute of Automation of the
Chinese Academy of Sciences. Four speakers are required to perform six
different emotions, _e.g._ , happiness, anger, fear, sadness, neutrality and
surprise. A total of 1200 sentences are utilized in this experiment. The
sampling rate of both databases is 16 kHz. The details of the above two
databases are reported in Table 1.

Table 1: Experimental Database Description Database | Language | Samples of Each Emotion | Total Samples
---|---|---|---
IEMOCAP | English |  |  _angry_ (289) _happy_ (284)
---
_neutral_ (1099) _sad_ (608)
2280
CASIA | Mandarin |  |  _angry_ (200) _fear_ (200)
---
_happy_ (200) _neutral_ (200)
_sad_ (200) _surprise_ (200)
1200

###  3.2 Experimental Protocol

In the experiment, we follow the same protocol of the previous research [10]
and adopt the Leave-One-Speaker-Out (LOSO) cross-validation for evaluation.

Specifically, for CASIA, when one speaker’s samples are served as the testing
data, the remaining three speakers’ samples are used for training. Similarly,
for IEMOCAP, we use one speaker’s samples as the testing data and other
speakers’ samples as the training data. Moreover, since IEMOCAP contains 5
sessions, the leave-one-session-out cross-validation protocol (one session’s
samples as the testing data and four sessions’ samples as the training data)
is also a common way for evaluation [18]. Therefore, in Table 4, we also
choose some methods using this protocol to compare with our proposed method.

In this paper, we choose the weighted average recall (WAR) [6] and the
unweighted average recall (UAR) [5], which are widely-used SER evaluation
indicators, to effectively measure the performance of the proposed method.

###  3.3 Experimental Setting

Before the feature extraction, we preprocess the audio samples by dividing
them into small segments of 80 frames (20ms per frame). With this operation,
samples are not only augmented, but also maintain the integrity of speech
emotions. After that, we pre-emphasize the speech segments and the pre-
emphasis coefficient is 0.97. Then, we use a 20ms Hamming window with a frame
shift of 10ms to extract log-Mel-spectrogram, where the number of points of
Fast Fourier Transform (FFT) and bands of Mel-filter are 512 and 80
respectively. Finally, the model input features of _b_ =64, _c_ =1, _f_ =80,
_d_ =80 are obtained.

Besides, the parameters of $C_{t}(\cdot)$, $C_{f}(\cdot)$, $C(\cdot)$ are
shown in Table 2. The $BN(\cdot)$ and $Act(\cdot)$ denote the BatchNorm
function [19] and the ReLU function [14], respectively. The parameters of the
Transformer encoder used in our model are shown in Table 3. The proposed
method is implemented by Pytorch [20] with NVIDIA A10 Tensor Core GPUs, which
is trained from scratch with 1000 epochs and optimized by Adam optimizer [21]
with the initialized learning rate of 0.001.

Table 2: CNN Parameters Settings  Operation |  Out Channels |  Kernel Size |  Stride |  Padding
---|---|---|---|---
$C_{t}(\cdot)$ |  64 |  (5,1) |  (2,1) |  (2,0)
$C_{f}(\cdot)$ |  64 |  (1,5) |  (1,2) |  (0,2)
$C(\cdot)$ |  64 |  (5,5) |  (2,2) |  (2,2)
Table 3: Transformer Encoder Parameters Settings Module | Embed Dimension | Feed-forward Dimension | Attention Heads
---|---|---|---
Time Transformer | 20 | 512 | 2
Frequency Transformer | 20 | 512 | 2
Time-Frequency Transformer | 20 | 1024 | 4
Table 4: Experimental Results on IEMOCAP and CASIA Database | Experimental Protocol | Comparison Method | Accuracy(%)
---|---|---|---
WAR | UAR
IEMOCAP |  Leave One Session/Speaker Out (LOSO) (5 Sessions or 10 Speakers) | MSA-AL [18] | 72.34 | 58.31
CNN-LSTM [11] | 68.80 | 59.40
STC-Attention [22] | 61.32 | 60.43
DNN-SALi [23] | 62.28 | 58.02
GRU-CNN-SeqCap [12] | 72.73 | 59.71
DNN-BN [24] | 59.7 | 61.4
Ours | 74.43 | 62.90
CASIA |  Leave One Speaker Out (LOSO) (4 Speakers) | GA-BEL [25] | 39.50 | 39.50
ELM-DNN [26] | 41.17 | 41.17
LoHu [27] | 43.50 | 43.50
DCNN-DTPM [28] | 45.42 | 45.42
ATFNN [10] | 48.75 | 48.75
Ours | 53.17 | 53.17

(a) Confusion matrix on IEMOCAP

(b) Confusion matrix on CASIA

Figure 2: Confusion matrices on IEMOCAP and CASIA.

###  3.4 Experimental Results and Analysis

####  3.4.1 Results on IEMOCAP

We selected some state-of-the-art methods for performance comparison with the
proposed method, _i.e._ , a model based on MSA that fuses acoustic and
linguistic features (MSA-AL) [18] , a model that combines CNN with Long Short
Term Memory (LSTM) and uses spectrogram as the input features (CNN-LSTM) [11],
spectro-temporal and CNN with attention model(STC-Attention) [22], a Deep
Neural Network (DNN) method combining Gaussian Mixture Models (GMM) and Hidden
Markov Models (HMM) using subspace alignment strategy (DNN-SAli) [23], a
method of using Gated Recurrent Unit (GRU) in CNN layers and combining with
sequential Capsules (GRU-CNN-SeqCap) [12], and a DNN method with Bottleneck
features (DNN-BN) [24].

The experimental results are shown in Table 4. From the results, we can find
something interesting. Firstly, our proposed Time-Frequency Transformer
achieves the best performance on both WAR (74.43%) and UAR (62.90%) compared
to other mentioned methods. Moreover, compared to all methods using Leave-One-
Speaker-Out protocol, our proposed method achieves a promising increase over
10% in term of WAR and 1.5% in UAR.

The confusion matrix of IEMOCAP is shown in Figure. 2(a). What we can observe
first is that the proposed method exhibits a excellent performance in
classifying specific emotions, _e.g._ , _angry_ , _neutral_ and _sad_.
However, it is difficult for the proposed model to correctly recognize the
emotion _happy_. As shown in the figure, 72% _happy_ samples are misclassified
as _neutral_ and only 17% are correctly classified. Obviously, it cannot be
caused by the reason that _happy_ is more close to _neural_ than other
emotions (negative emotions: _anger_ and _sad_) since the possibility of
_neutral_ samples being misclassified as _happy_ is only 0.73%. This situation
lead us to consider the other reason which is the unbalanced sample size.
Since the number of _happy_ samples in IEMOCAP is only 284 which is the
smallest among all emotions, the model cannot learn the unique emotional
characteristics of _happy_ well. It may lead to this situation that the
_happy_ samples are more likely to be mistaken for _neutral_.

####  3.4.2 Results on CASIA

Some state-of-the-art methods that also use the LOSO protocol are used for
comparison with the proposed method, including Genetic Algorithm (GA) combined
with Brain Emotional Learning (BEL) model (GA-BEL) [25], Extreme Learning
Machine (ELM) combined with DNN (ELM-DNN) [26], weighted spectral features
based on Local Hu moments (LoHu) [27], Deep CNN (DCNN) combined with a
Discriminant Temporal Pyramid Matching (DTPM) strategy (DCNN-DTPM) [28] and an
Attentive Time-Frequency Neural Network (ATFNN) [10].

The results on the CASIA database is shown in Table 4. It is obvious that our
method achieves state-of-the-art performance among all algorithms.
Specifically, our method obtains the best result on WAR (53.17%) and UAR
(53.17%) than all comparison methods. Since the sample numbers of the 6
emotions of CASIA used in the experiment are balanced, WAR and UAR are equal.
Besides, our results are not only the best, but also far superior to other
methods. Even compared to ATFNN which is the second best method, the proposed
method still obtain a over 4% performance increase.

From the confusion matrix of CASIA in Figure. 2(b), it is obvious that the
proposed method has a high recognition rate in four types of emotions(_angry_
, _fear_ , _neutral_ , _sad_), but the recognition effect on _happy_ and
_surprise_ is poor. Since _happy_ is easily misclassified as _sad_ , it may be
caused by the pendulum effect [29] in psychology. Human emotions are
characterized by multiplicity and bipolarity under the influence of external
stimuli. Beside that, _surprise_ is always confused with _fear_. Due to the
similar arousal [30] of the two emotions, it may lead to them inducing each
other.

Table 5: Ablation experiments of different architectures for our model on IEMOCAP and CASIA, where ’✓’ or ’✗’ represents the network with or without the corresponding module. ’T-Trans’, ’F-Trans’, and ’TF-Trans’ are the modules of Time Transformer, Frequency Transformer, and Time-frequency Transformer, respectively. Architecture | Ablation Experiments |  IEMOCAP(%) |  CASIA(%)
---|---|---|---
T-Trans | F-Trans | TF-Trans |  WAR |  UAR |  WAR |  UAR
T$+$F | ✓ | ✓ | ✗ | 70.12 | 58.77 | 40.32 | 40.32
T$+$TF | ✓ | ✗ | ✓ | 70.96 | 60.34 | 48.91 | 48.91
F$+$TF | ✗ | ✓ | ✓ | 71.47 | 60.59 | 49.26 | 49.26
T$+$F$+$TF (ours) | ✓ | ✓ | ✓ | 74.43 | 62.90 | 53.17 | 53.17

####  3.4.3 Ablation Experiments

We verified the effectiveness of our method by removing some modules of the
proposed method. The experimental results are shown in Table 5, where
’T-Trans’, ’F-Trans’, and ’TF-Trans’ are the modules of Time Transformer,
Frequency Transformer, and Time-frequency Transformer, respectively. According
to the results of the ablation experiments, the TF-Trans can effectively make
the model focus on the emotion-related segments in the time-frequency domain
and improve the emotional discrimination of the model. In addition, the effect
of removing T-Trans is better than removing F-Trans, indicating that the
frequency domain information of speech is of great significance for emotion
recognition. Moreover, it is easy to observe that model_1 achieves the worst
result, particular on CASIA. Compared to model_2 and model_3, model_1 has a
significant performance degradation over 8% of both WAR and UAR on CASIA. This
phenomenon indicates the effectiveness of the Time-Frequency Transformer
module. If we remove this part, the local time-domain and frequency-domain
emotion features are not fully utilized to mine the time-frequency emotional
correlations. Thus, the model cannot learn more discriminative global acoustic
emotional feature representations.

(a) Spectrogram

(b) T-Trans Attention

(c) F-Trans Attention

(d) TF-Trans Attention

Figure 3: Visualization of log-Mel-spectrogram, T-Trans Attention, F-Trans
Attention, and TF-Trans Attention (the emotion category of this sample is
_sad_ , which belongs to IEMOCAP).

####  3.4.4 Visualization of Attention

In order to further investigate whether the proposed method focuses on
frequency bands with specific energy activations and emotional key speech
frames in the speech signal, we visualize the attention of Time Transformer,
Frequency Transformer and Time-Frequency Transformer to the log-Mel-
spectrogram, as shown in Figure. 3. Form the visualization of Time Transformer
in Figure. 3(b), we can observe that there are strong activation values in
$15^{th}-20^{th}$ frames and $50^{th}-60^{th}$ frames, which indicates that
the emotion correlations between these frames is important to represent speech
emotions. And these frames correspond to the positions with richer semantic
information in Figure. 3(a). The visualization of Frequency Transformer in
Figure. 3(c) shows that the activation of the middle and low frequency bands
is more obvious, demonstrating that the middle and low frequency bands are key
for the _sad_ emotion representation. From the results of time-frequency
attention in Figure. 3(d), we can see that the larger activation value (i. e.,
the salient patches) corresponds to the regions where the semantic information
is more concentrated in Figure. 3(a). Therefore, our proposed Time-Frequency
Transformer can fully capture the time-frequency regions highly correlated
with emotions while ensuring the complete modeling of local emotion
information in the time and frequency domains to obtain discriminative speech
emotion features.

##  4 Conclusion

In this paper, we propose a novel Transformer-based time frequency domain
joint learning method for SER, i.e., Time-Frequency Transformer. It can
effectively model local emotion correlations between frames and frequency
bands through Time Frequency and Frequency Transformer. Then these local
emotion features are aggregated into more discriminative global emotion
representations by a Time-Frequency Transformer. However, the MSA operation in
Transformer is aiming at model global long-range discrepancy, which is easily
disturbed by noisy frames or frequency bands in speech. Therefore, our Future
research will focus on sparse MSA for speech emotion representations.

## References

  * [1] Schuller, B., Batliner, A.: Computational paralinguistics: emotion, affect and personality in speech and language processing. John Wiley & Sons (2013)
  * [2] Schuller, B.W.: Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends. Communications of the ACM 61(5), 90–99 (2018)
  * [3] Akçay, M.B., Oğuz, K.: Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers. Speech Communication 116, 56–76 (2020)
  * [4] Lu, C., Zong, Y., Zheng, W., Li, Y., Tang, C., Schuller, B.W.: Domain invariant feature learning for speaker-independent speech emotion recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30, 2217–2230 (2022)
  * [5] Stuhlsatz, A., Meyer, C., Eyben, F., Zielke, T., Meier, G., Schuller, B.: Deep neural networks for acoustic emotion recognition: Raising the benchmarks. In: 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 5688–5691. IEEE (2011)
  * [6] Schuller, B., Vlasenko, B., Eyben, F., Rigoll, G., Wendemuth, A.: Acoustic emotion recognition: A benchmark comparison of performances. In: 2009 IEEE Workshop on Automatic Speech Recognition & Understanding. pp. 552–557. IEEE (2009)
  * [7] Abbaschian, B.J., Sierra-Sosa, D., Elmaghraby, A.: Deep learning techniques for speech emotion recognition, from databases to models. Sensors 21(4), 1249 (2021)
  * [8] Wang, J., Xue, M., Culhane, R., Diao, E., Ding, J., Tarokh, V.: Speech emotion recognition with dual-sequence lstm architecture. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6474–6478. IEEE (2020)
  * [9] Mao, Q., Dong, M., Huang, Z., Zhan, Y.: Learning salient features for speech emotion recognition using convolutional neural networks. IEEE transactions on multimedia 16(8), 2203–2213 (2014)
  * [10] Lu, C., Zheng, W., Lian, H., Zong, Y., Tang, C., Li, S., Zhao, Y.: Speech emotion recognition via an attentive time–frequency neural network. IEEE Transactions on Computational Social Systems (2022)
  * [11] Satt, A., Rozenberg, S., Hoory, R., et al.: Efficient emotion recognition from speech using deep learning on spectrograms. In: Interspeech. pp. 1089–1093 (2017)
  * [12] Wu, X., Liu, S., Cao, Y., Li, X., Yu, J., Dai, D., Ma, X., Hu, S., Wu, Z., Liu, X., et al.: Speech emotion recognition using capsule networks. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6695–6699. IEEE (2019)
  * [13] Chaudhari, S., Mithal, V., Polatkan, G., Ramanath, R.: An attentive survey of attention models. ACM Transactions on Intelligent Systems and Technology (TIST) 12(5), 1–32 (2021)
  * [14] Dubey, S.R., Singh, S.K., Chaudhuri, B.B.: Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing (2022)
  * [15] Zhang, Z., Sabuncu, M.: Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems 31 (2018)
  * [16] Busso, C., Bulut, M., Lee, C.C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J.N., Lee, S., Narayanan, S.S.: Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation 42, 335–359 (2008)
  * [17] Zhang, J., Jia, H.: Design of speech corpus for mandarin text to speech. In: The blizzard challenge 2008 workshop (2008)
  * [18] Bhosale, S., Chakraborty, R., Kopparapu, S.K.: Deep encoded linguistic and acoustic cues for attention based end to end speech emotion recognition. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 7189–7193. IEEE (2020)
  * [19] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning. pp. 448–456. pmlr (2015)
  * [20] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019)
  * [21] Adam, K.D.B.J., et al.: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 1412 (2014)
  * [22] Guo, L., Wang, L., Xu, C., Dang, J., Chng, E.S., Li, H.: Representation learning with spectro-temporal-channel attention for speech emotion recognition. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6304–6308. IEEE (2021)
  * [23] Mao, S., Tao, D., Zhang, G., Ching, P., Lee, T.: Revisiting hidden markov models for speech emotion recognition. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6715–6719. IEEE (2019)
  * [24] Kim, E., Shin, J.W.: Dnn-based emotion recognition based on bottleneck acoustic features and lexical features. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6720–6724. IEEE (2019)
  * [25] Liu, Z.T., Xie, Q., Wu, M., Cao, W.H., Mei, Y., Mao, J.W.: Speech emotion recognition based on an improved brain emotion learning model. Neurocomputing 309, 145–156 (2018)
  * [26] Han, K., Yu, D., Tashev, I.: Speech emotion recognition using deep neural network and extreme learning machine. In: Interspeech 2014 (2014)
  * [27] Sun, Y., Wen, G., Wang, J.: Weighted spectral features based on local hu moments for speech emotion recognition. Biomedical signal processing and control 18, 80–90 (2015)
  * [28] Zhang, S., Zhang, S., Huang, T., Gao, W.: Speech emotion recognition using deep convolutional neural network and discriminant temporal pyramid matching. IEEE Transactions on Multimedia 20(6), 1576–1590 (2017)
  * [29] Wegner, D.M., Ansfield, M., Pilloff, D.: The putt and the pendulum: Ironic effects of the mental control of action. Psychological Science 9(3), 196–199 (1998)
  * [30] Hanjalic, A., Xu, L.Q.: Affective video content representation and modeling. IEEE transactions on multimedia 7(1), 143–154 (2005)


author date

# Estimation of acoustic wave non-linearity in ultrasonic measurement systems

Leander Claes  Measurement Engineering Group, Paderborn University,
Warburger Straße 100, 33098 Paderborn, Germany  Carolin Steidl  Measurement
Engineering Group, Paderborn University,
Warburger Straße 100, 33098 Paderborn, Germany  Tim Hetkämper  Measurement
Engineering Group, Paderborn University,
Warburger Straße 100, 33098 Paderborn, Germany  Bernd Henning  Measurement
Engineering Group, Paderborn University,
Warburger Straße 100, 33098 Paderborn, Germany

###### Abstract

Most measurement methods based on ultrasound, such as sound velocity,
absorption or flow measurement systems, require that the acoustic wave
propagation is linear. In many cases, linear wave propagation is assumed due
to small signal amplitudes or verified, for example, by analysing the received
signal spectra for the generation of harmonic frequency components. In this
contribution, we present an approach to quantify occurrence of non-linear
effects of acoustic wave propagation in ultrasonic measurement systems based
on the evaluation of the acoustic Reynolds number. One parameter required for
the determination of the acoustic Reynolds number is the particle velocity of
the acoustic wave, which is not trivially obtained in most measurement
systems. We thus present a model-based approach to estimate the particle
velocity of an acoustic wave by identifying a Mason model from electrical
impedance measurements of a given transducer. The Mason model is then used to
determine the transducer’s velocity output for a given electrical signal,
allowing for an evaluation of the acoustic Reynolds number for different
target media.

##  1 Motivation

Due to the character of the differential equations that describe the behaviour
of fluids, such as the Navier-Stokes-Equation and the equation of state for
the respective fluid, all acoustic wave propagation is non-linear. When
deriving the differential equation for an acoustic wave, one assumes the
amplitude of the acoustic wave to be sufficiently small, so that the non-
linear terms that exist in constituting equations are negligible [1]. Thus,
for applications of acoustic waves that are assumed to be linear, it has to be
determined if the amplitude of the acoustic wave created by a given transducer
satisfies the aforementioned condition. One option to verify if the acoustic
signal’s particle velocity amplitude $v_{0}$ is sufficiently small is to
consider the acoustic Reynolds number $N_{\mathrm{Re}}$ [2]:

$N_{\mathrm{Re}}=\frac{v_{0}c\rho}{\mu\omega},$ (1)

where $c$ denotes the sound velocity and $\rho$ denotes the density of the
medium. $\omega$ is the angular frequency of the acoustic wave and $\mu$ are
the combined linear thermal and viscous losses in the fluid:

$\mu=\frac{4}{3}\mu_{\mathrm{s}}+\mu_{\mathrm{v}}+\frac{c_{\mathrm{p}}-c_{\mathrm{v}}}{c_{\mathrm{p}}\cdot
c_{\mathrm{v}}}\nu,$ (2)

with the shear viscosity $\mu_{\mathrm{s}}$ and the volume viscosity
$\mu_{\mathrm{v}}$. $c_{\mathrm{p}}$ and $c_{\mathrm{v}}$ are the isobaric and
isochoric specific heat capacities and $\nu$ is the thermal conductivity of
the fluid. The acoustic Reynolds number $N_{\mathrm{Re}}$ describes the
relation of the accumulation of non-linear effects to the effects of linear
absorption caused by the thermal and viscous losses $\mu$. Thus, for
$N_{\mathrm{Re}}\gg 1$ non-linear effects are predominant, while for
$N_{\mathrm{Re}}\ll 1$, linear absorption dominates the properties of acoustic
wave propagation [2]. For practical applications this relationship implies
that the sound propagation tends to be more linear if the losses $\mu$ in the
medium are high. This can be expressed as a lower bound for $\mu$ if the
requirement $N_{\mathrm{Re}}\ll 1$ is to be satisfied:

$\mu\gg\frac{v_{0}c\rho}{\omega}=\frac{v_{0}Z}{\omega}.$ (3)

In equation 1 and equation 3 the product of sound velocity $c$ and density
$\rho$ can be replaced by the specific acoustic impedance of the medium $Z$.

While the values of the sound velocity and the density of the medium, as well
as the angular frequency $\omega$ of the acoustic wave, are usually available
and the losses $\mu$ can be estimated, the particle velocity amplitude $v_{0}$
is not trivially obtainable. As the acoustic wave amplitude in a simplified
acoustic measurement system usually has its maximum at the transmitting
transducer’s active surface, one can assume $v_{0}$ to be the normal velocity
of said surface. While this velocity can be measured by means of laser
vibrometry, the surface has to be loaded with the target medium during the
measurement. Moreover, this requires the medium to be transparent and the
transducer’s active surface to be optically accessible. Thus, a more general
approach to estimate the transducer’s surface velocity is implemented by
identifying a three-port Mason model using impedance measurements.

##  2 Transducer modelling

To model the electromechanical behaviour of a transducer in the spectral range
around its thickness resonance frequency, the established Mason model [3] is
applied. The model uses a three-port network with two mechanical ports (with
force $F_{i}$ and velocity $v_{i}$) and one electrical port (with voltage $u$
and current $i$). The mechanical ports represent the faces of a given
piezoelectric transducer. The three-port can be described mathematically by a
$3\times 3$ matrix $\mathbf{M}$:

$\begin{bmatrix}F_{1}\\\ F_{2}\\\
u\end{bmatrix}=\begin{bmatrix}Z_{\mathrm{m,t}}\coth\left(\gamma
t\right)&Z_{\mathrm{m,t}}\operatorname{csch}\left(\gamma
t\right)&h_{\mathrm{t}}/(\mathrm{j}\omega)\\\
Z_{\mathrm{m,t}}\operatorname{csch}\left(\gamma
t\right)&Z_{\mathrm{m,t}}\coth\left(\gamma
t\right)&h_{\mathrm{t}}/(\mathrm{j}\omega)\\\
h_{\mathrm{t}}/(\mathrm{j}\omega)&h_{\mathrm{t}}/(\mathrm{j}\omega)&1/(\mathrm{j}\omega
C_{\mathrm{t}})\\\ \end{bmatrix}\cdot\begin{bmatrix}v_{1}\\\ v_{2}\\\
i\end{bmatrix}$ (4)

with parameters

$\displaystyle Z_{\mathrm{m,t}}=$
$\displaystyle\rho_{\mathrm{t}}c_{\mathrm{t}}A_{\mathrm{t}},\hskip
28.45274pt\gamma$ $\displaystyle=\mathrm{j}\frac{\omega}{c_{\mathrm{t}}},$ (5)
$\displaystyle h_{\mathrm{t}}=$ $\displaystyle
c_{\mathrm{t}}\sqrt{\tfrac{k_{\mathrm{t}}^{2}\rho_{\mathrm{t}}}{\varepsilon_{\mathrm{t}}}},\hskip
28.45274ptC_{\mathrm{t}}$
$\displaystyle=\tfrac{\varepsilon_{\mathrm{t}}A_{\mathrm{t}}}{t}.$

Here, $Z_{\mathrm{m,t}}$ is the mechanical impedance of the transducer’s
mechanical port, defined by the product of density $\rho_{\mathrm{t}}$ and
sound velocity $c_{\mathrm{t}}$ of the transducer’s material and the area
$A_{\mathrm{t}}$ of the transducer. The parameter $h_{\mathrm{t}}$ couples the
electrical and the mechanical properties of the model and is determined using
the piezoelectric coupling factor $k_{\mathrm{t}}$ and the permittivity
$\varepsilon_{\mathrm{t}}$. Finally, the electrical capacitance of the
transducer is represented by $C_{\mathrm{t}}$, which depends on the thickness
$t$ of the transducer.

$\mathbf{M}$$v_{1}$$v_{2}$$i$$F_{1}$$F_{2}$$u$$Z_{\mathrm{m,1}}$$Z_{\mathrm{m,2}}$
Figure 1: Three-port Mason model with the mechanical ports terminated by the
mechanical impedance of the adjacent medium.

For the identification of a given transducer, the mechanical ports of the
models are terminated using the mechanical impedance of the adjacent medium
(figure 1). Setting $F_{i}=-v_{i}Z_{\mathrm{m,}}{}_{i}$, this enables to solve
equation 4 for the frequency-dependent electrical impedance
$Z_{\mathrm{el}}=u/i$ of the transducer model. In an inverse procedure, the
parameters of the Mason model for a given ultrasonic transducer are identified
by comparing the electrical impedance of the model $Z_{\mathrm{el}}$ with the
electrical impedance $Z_{\mathrm{meas}}$ of a physical transducer [4]. As the
physical transducer is to be used in an acoustic absorption measurement system
based on an established system for sound velocity measurement (e.g. applied by
Javed et al. [5]), it consists of a piezoelectric disc surrounded on both
sides by the target medium. For the identification of the transducer,
measurements in air are performed assuming the acoustic impedance of air
($412\text{\,}\mathrm{kg}\text{\,}{\mathrm{m}}^{-2}\text{\,}{\mathrm{s}}^{-1}$)
multiplied by the area of the transducer to terminate the mechanical ports of
the model. For the modelling of other transducers, the specific acoustic
impedance of the backing material can be used for the termination of one
mechanical port with the value estimated or identified in the subsequent
optimization process. The transducer identified here consists of a hard lead
zirconate titanate ceramic (PIC181, _PI Ceramic GmbH_), has a thickness of
$0.2\text{\,}\mathrm{mm}$ and a radius of $8\text{\,}\mathrm{mm}$. The density
of the material is measured gravimetrically with the result conforming with
the manufacturer’s value of
$7800\text{\,}\mathrm{kg}\text{\,}{\mathrm{m}}^{-3}$. This leaves only three
parameters of the terminated Mason model to be identified: The sound velocity
of the transducer’s material $c_{\mathrm{t}}$, the piezoelectric coupling
factor $k_{\mathrm{t}}$, and the permittivity $\varepsilon_{\mathrm{t}}$.

$9$$9.5$$10$$10.5$$11$$11.5$$12$$10^{-2}$$10^{-1}$$10^{0}$$10^{1}$$10^{2}$$10^{3}$Frequency
$f/$\mathrm{MHz}$$Electrical impedance in
$\mathrm{\SIUnitSymbolOhm}$Measurement ($Z_{\mathrm{meas}}$)Mason model
($Z_{\mathrm{el}}$) Figure 2: Magnitude of the electrical impedance of a
physical transducer and the impedance of the identified Mason model.

As a cost function for the inverse procedure, the difference in the frequency-
dependent magnitude of model and measured impedance is weighted with an
arctangent function for robustness. Then, the sum of the squares of this
expression is minimized using a Trust Region Reflective algorithm [6]. This
optimization process yields the following parameters for the identified Mason
model of the transducer:
$c_{\mathrm{t}}=$4380\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$$,
$k_{\mathrm{t}}=$0.450\text{\,}$$, and
$\varepsilon_{\mathrm{t}}=$5.49\text{⋅}{10}^{-9}\text{\,}\mathrm{A}\text{\,}\mathrm{s}\text{\,}{\mathrm{V}}^{-1}\text{\,}{\mathrm{m}}^{-1}$$.
The resulting impedance of the model matches the measured impedance closely
(figure 2), with only the areas close to the resonance and antiresonance
frequency showing significant deviation. This is due to the low resp. high
impedance values close to these frequencies, which result in an increased
noise and uncertainty in the measurement with the impedance analyser (E4990A,
_Keysight Technologies_) used. As PIC181 is a hard piezoelectric material,
pronounced resonance and antiresonance frequencies are, however, expected. The
measurement also shows superimposed influence of radial modes that the Mason
model cannot represent as it is based on one-dimensional considerations.

##  3 Estimation of non-linearity

With the Mason model identified in the previous section, it is possible to
model the electromechanical behaviour of the transducer. This allows to
estimate the velocity of the faces of the transducer for a given voltage.
Changing the terminating mechanical impedance also allows to estimate the
velocity for changing target media. Assuming that the transducer is terminated
with the same mechanical impedance ($Z_{\mathrm{m}}=A_{\mathrm{t}}Z$) at both
mechanical ports as before, solving equation 4 for $v/u$ yields the frequency
response of the transducer in transmission mode:

$\frac{v}{u}=G_{\mathrm{t}}(\mathrm{j}\omega)={\left(\frac{Z_{\mathrm{m}}}{C_{\mathrm{t}}h_{\mathrm{t}}}-\frac{Z_{\mathrm{m,t}}}{C_{\mathrm{t}}h_{\mathrm{t}}}\left(\coth\left(\gamma
t\right)+\operatorname{csch}\left(\gamma
t\right)\right)-\frac{2h_{\mathrm{t}}}{\mathrm{j}\omega}\right)}^{-1}.$ (6)

Note that $\gamma$ also depends on the angular frequency $\omega$ (equation
5). For continuous, monofrequent excitation of acoustic waves, one can apply
equation 6 directly by setting $u$ to the electrical signal’s amplitude and
solving for the velocity $v$. The absolute value of $v$ can then be used as an
estimate for the particle velocity close to the transducers surface $v_{0}$,
allowing to determine the acoustic Reynolds number $N_{\mathrm{Re}}$ using
equation 1. In physical measurement systems, however, signals are typically
limited in the temporal regime and thus have a finitely small bandwidth. As
equation 6 models the frequency-dependent behaviour of the transducer, it
describes how an electrical voltage signal translates into a velocity signal
in the frequency domain. Fourier transform ($\mathcal{F}\left\\{\right\\}$)
and inverse Fourier transform ($\mathcal{F}^{-1}\left\\{\right\\}$) then allow
to model the influence of the transducer on a transient signal $u(t)$:

$v(t)=\mathcal{F}^{-1}\left\\{G_{\mathrm{t}}(\mathrm{j}\omega)\mathcal{F}\left\\{u(t)\right\\}\right\\}.$
(7)

The resulting transient velocity $v(t)$ of the transducer’s faces can then be
evaluated for its maximum as an estimate for $v_{0}$:

$v_{0}\approx\max(v(t)).$ (8)

Similar to the physical setup [5], the electrical excitation signal $u(t)$ is
modelled as a Gaussian modulated sinusoidal pulse with a centre frequency of
$10.5\text{\,}\mathrm{MHz}$, a relative bandwidth of $0.1\text{\,}$, and a
peak voltage of $1\text{\,}\mathrm{V}$. As the setup utilizes the pulse-echo
technique, the centre frequency is chosen between the resonance and
antiresonance frequency (figure 2) of the transducer to enable transmitting
and receiving operation. The setup is used for a variety of different fluids,
so the maximum of the velocity $v_{0}$ is evaluated dependent on the specific
acoustic impedance $Z$ of the fluid. The results are then inserted in equation
3 to determine the lower bound of the linear losses $\mu$ the fluid to be
analysed needs to have for the acoustic Reynolds number to be less than one,
resulting in predominately linear wave propagation. To analyse a wide range of
the specific acoustic impedance, the results are presented with logarithmic
scales (figure 3), showing that the minimal losses $\mu$ for linear sound
propagation increase with the specific acoustic impedance $Z$ of the target
fluid. At values for the specific acoustic impedance of the fluid that
approach and exceed the specific acoustic impedance of the transducer’s
material, the minimal losses for linear sound propagation show a constant
value. Note that these results are only valid for the setup and transducer
described before with an excitation signal voltage of $1\text{\,}\mathrm{V}$.

As a reference, the specific acoustic impedances and losses of several fluids
at $293\text{\,}\mathrm{K}$ and $100\text{\,}\mathrm{kPa}$ are included in
figure 3 as well [7]. The losses $\mu$ depicted are low estimates, as they
only include the influence of shear viscosity and thermal conductivity
($\mu=\frac{4}{3}\mu_{\mathrm{s}}+\frac{c_{\mathrm{p}}-c_{\mathrm{v}}}{c_{\mathrm{p}}\cdot
c_{\mathrm{v}}}\nu$), omitting the additional loss caused by the relatively
unexplored volume viscosity. Thus, in a physical setup, the difference between
the minimal losses necessary for linear sound propagation and the actual
losses in the respective fluid is expected to be more significant.

The sound propagation in all fluids used for the comparison is expected to be
predominantly linear, if the identified transducer and the excitation signal
is used. The distance to the lower boundary for the losses is significant for
the gases used for comparison (helium, air, argon and xenon), showing that the
acoustic Reynolds number of these transducer-fluid combinations is
significantly smaller than one. For the depicted liquids (n-hexane, methanol,
ethanol and water), the distance to the depicted graph is smaller. Especially
if n-Hexane is analysed with the identified transducer, the acoustic Reynolds
number approaches one. In this case, non-linear wave propagation may occur and
measures to prevent or detect these non-linear effects, such as lowering the
signal voltage or analysing the acoustic signal spectrum for higher harmonics,
should be taken. It should be noted that changing the thermodynamic state of
the fluids will result in different properties ($Z$ and $\mu$) which could
potentially fail to satisfy equation 3. Also note that these considerations
constitute a worst-case assessment, as additional dissipative effects that may
prevent non-linear wave propagation in the fluid, such as the effects of
losses in the transducer’s material and additional linear absorption due to
volume viscosity, are neglected.

$10^{2}$$10^{3}$$10^{4}$$10^{5}$$10^{6}$$10^{7}$$10^{-7}$$10^{-6}$$10^{-5}$$10^{-4}$$10^{-3}$$10^{-2}$WaterEthanolMethanolArgonHeliumXenonAirn-
HexaneSpecific acoustic impedance
$Z/$\mathrm{kg}\text{\,}{\mathrm{m}}^{-2}\text{\,}{\mathrm{s}}^{-1}$$Thermal
and viscous losses
$\mu/$\mathrm{Pa}\text{\,}\mathrm{s}$$$\mu=\frac{v_{0}Z}{\omega}$ Figure 3:
Minimum value for losses $\mu$ dependent on the specific acoustic impedance of
the target medium.

##  4 Conclusions

A means to assess whether acoustic wave propagation can be assumed as linear
in a given medium is the acoustic Reynolds number. The evaluation of this
parameter, however, requires quantitative information about the particle
velocity. This velocity can be estimated using a Mason model for a given
transducer, which can be identified by a measurement of the transducer’s
frequency-dependent electrical impedance. The procedure requires no direct
measurement of the particle velocity or other acoustic quantities and is thus
easy to realize experimentally for a variety of application scenarios.

The approach may be further expanded by applying more in-depth models for the
transducers, using e.g. chain matrices for the modelling of matching layers
[8] or complete finite-element simulations. As the results of these
consideration describe a worst-case scenario (if the aim is to have linear
sound propagation), edge cases ($N_{\mathrm{Re}}\approx 1$) should be reviewed
by evaluating the acoustic signal spectrum for the existence of harmonic
frequencies caused by non-linearity.

## References

  * [1] L.D. Landau and E.M. Lifshitz  “Fluid Mechanics”  Elsevier Science, 2013
  * [2] Oleg Vladimirovič Rudenko, Stepan Ivanovich Solujan and Robert T. Beyer  “Theoretical foundations of nonlinear acoustics”, Studies in Soviet science Physical sciences  New York: Consultants Bureau, 1977
  * [3] W.. Mason  “An Electromechanical Representation of a Piezoelectric Crystal Used as a Transducer”  In _Bell System Technical Journal_ 14.4, 1935, pp. 718–723  DOI: 10.1002/j.1538-7305.1935.tb00713.x
  * [4] Nadine Feldmann et al.  “An inverse approach to the characterisation of material parameters of piezoelectric discs with triple-ring-electrodes”  In _tm - Technisches Messen_ 86.2, 2019, pp. 59–65  DOI: 10.1515/teme-2018-0066
  * [5] Muhammad Ali Javed, Elmar Baumhögger and Jadran Vrabec  “Thermodynamic Speed of Sound Data for Liquid and Supercritical Alcohols”  In _Journal of Chemical & Engineering Data_ 64.3 American Chemical Society (ACS), 2019, pp. 1035–1044  DOI: 10.1021/acs.jced.8b00938
  * [6] Mary Ann Branch, Thomas F. Coleman and Yuying Li  “A Subspace, Interior, and Conjugate Gradient Method for Large-Scale Bound-Constrained Minimization Problems”  In _SIAM Journal on Scientific Computing_ 21.1 Society for Industrial & Applied Mathematics (SIAM), 1999, pp. 1–23
  * [7] E.. Lemmon, M.. Huber and M.. McLinden  “REFPROP 9.1”  In _NIST Standard Reference Database_ 23, 2013
  * [8] M. Webersen, F. Bause, J. Rautenberg and B. Henning  “B1.2 - Identification of Temperature-Dependent Model Parameters of Ultrasonic Piezo-Composite Transducers”  In _AMA Conferences 2015_ AMA Service GmbH, Germany, 2015  DOI: 10.5162/SENSOR2015/B1.2