# Transfer Learning of Lexical Semantic Families
for Argumentative Discourse Units Identification

João A. Rodrigues  Ruben Branco  António Branco
University of Lisbon
NLX – Natural Language and Speech Group, Department of Informatics
Faculdade de Ciências, Campo Grande, 1749-016 Lisboa, Portugal
{jarodrigues, rmbranco<EMAIL_ADDRESS>
###### Abstract

Argument mining tasks require an informed range of low to high complexity
linguistic phenomena and commonsense knowledge. Previous work has shown that
pre-trained language models are highly effective at encoding syntactic and
semantic linguistic phenomena when applied with transfer learning techniques
and built on different pre-training objectives. It remains an issue of how
much the existing pre-trained language models encompass the complexity of
argument mining tasks. We rely on experimentation to shed light on how
language models obtained from different lexical semantic families leverage the
performance of the identification of argumentative discourse units task.
Experimental results show that transfer learning techniques are beneficial to
the task and that current methods may be insufficient to leverage commonsense
knowledge from different lexical semantic families.

##  1 Introduction

The language technology landscape has been changed by pre-trained language
models (LMs) obtained from semantic spaces of words [Mikolov et al., 2013],
sentences [Kiros et al., 2015] and now shifting towards contextual words
[Devlin et al., 2019]. Although these pre-trained LMs have brought huge
improvements, the encoded knowledge and transferability [Caruana, 1997] are
still to be fully understood.

Argument mining [Lawrence and Reed, 2020], the automatic identification of
argumentative structures within natural language expressions, has improved and
gained new insights resorting to pre-trained LMs. Recent work has incorporated
pre-trained LMs using transfer learning techniques to predict argument
structure [Pathak et al., 2016, Li et al., 2017], convincingness [Simpson and
Gurevych, 2018], or identify argumentative component relations [Cocarascu and
Toni, 2017, Rocha et al., 2018]. Transfer learning techniques have also been
successful across-domains for the classification of argumentative discourse
units [Al-Khatib et al., 2016, Ajjour et al., 2017] and with multi-task
transfer [Eger et al., 2017, Potash et al., 2017, Stab et al., 2018, Choi and
Lee, 2018, Schulz et al., 2018, Jo et al., 2019].

We extend prior work by addressing argument mining linguistic complexity with
different lexical semantic families. We resort to three distinct data sets and
perform a range of analysis regarding the identification of argumentative
discourse units (ADUs)[Peldszus and Stede, 2013] task. We aim to answer the
question: What is the impact of models pre-trained with different lexical
semantic families on ADUs identification? We address this question by
evaluating several classifiers pre-trained with different semantic spaces and
provide insights over the linguistic knowledge that encompasses the task.

##  2 Methodology

Task and Data sets \- We address the identification of ADUs with the following
data sets: UKP Sentential Argument Mining Corpus (UKPS) [Stab et al., 2018],
Webis-Debate-16 (WEBIS) [Al-Khatib et al., 2016] and AraucariaDB (ARAUC) [Reed
et al., 2008]. These mainstream data sets are annotated with different and
somewhat competing criteria. The data sets were adapted for the identification
of ADUs as a binary task: classifying sentences as argument or non-argument.
UKPS was obtained with crowd-sourced annotations. WEBIS was automatically
obtained from debate forums. ARAUC was annotated following a comprehensive
argumentation theoretical background [Walton, 2009]. We discarded the topic
that accompanied the UKPS arguments and retrieved the ARAUC corpus as in
[Rooney et al., 2012]. ARAUC and WEBIS were under-sampled to balance the
classes.111The appendices contain details regarding the task, data sets and
classifiers necessary for reproduction.

Pre-trained semantic spaces \- Different lexical semantic families are
conceived with different semantic relations and objectives, using them as pre-
trained LMs may capture idiosyncratic linguistic knowledge useful for the
identification of ADUs. We considered three major lexical semantic families:
semantic networks, feature-based models and semantics spaces.

Semantic networks, as the WordNet [Fellbaum, 1998], represent lexical
semantics with graphs where the nodes are the lexical units and the labeled
edges represent semantic relations. Feature-based models, as the Small World
of Words (SWOW) [De Deyne et al., 2019], represent lexical semantics with hash
tables where the keys are the lexical units and the keys values represent the
semantic relations. Semantic spaces, as the Word2vec [Mikolov et al., 2013],
represent lexical semantics with vectors of real value and the semantic
relations are extracted using the cosine distance between lexical units.
Different semantic theories guide the creation of these lexical semantic
families. WordNet was obtained with the elaborated work of linguists that
reflect on the semantic properties of words. SWOW was obtained by asking
common native speakers of a language to refer to three words they associate
when exposed to a cue word. Word2vec and semantic spaces in general are
obtained from corpora, extracting semantic knowledge using the co-occurrence
of words.

Both semantic networks and feature-based models encode commonsense knowledge
in their relations. This makes them candidates for obtaining improvements in a
task such as ADUs that requires commonsense knowledge. Semantic spaces are the
typical go-to lexical models given the state-of-art results found across many
natural language processing tasks.

As semantic spaces, we used the original Word2vec as also enriched semantic
spaces, namely: GloVe, FastText, Dependency, BERT and T5. GloVe [Pennington et
al., 2014] is a count-based method that trains on aggregated global word-word
co-occurrence statistics from a corpus. The FastText [Joulin et al., 2016]
model uses morphological information. The Dependency [Levy and Goldberg, 2014]
uses the dependency parse context to guide the co-occurrence train. BERT
[Devlin et al., 2019] processes words in relation to all the other words in a
sentence. T5 [Raffel et al., 2019] uses a unified text-to-text-format for the
input/output resorting to an encoder-decoder architecture.

To compare the different semantic families we converted a semantic network
(WordNet) and a feature-based model (SWOW) to separate semantic spaces. The
WordNet and SWOW were converted into a raw corpus using a similar method as in
[Agirre et al., 2014], using a projection of the graph and associations. With
the raw corpus, we trained the models applying the Skip-gram algorithm
[Mikolov et al., 2013]. We fine-tuned the models with the average score
obtained from a semantic similarity and relatedness tasks, SimLex-999 [Hill et
al., 2015] and WordSim353-Rel [Agirre et al., 2009] respectively.

Classifiers \- We used a BiLSTM [Hochreiter and Schmidhuber, 1997] neural
network, a simple yet effective model widely used in previous studies. The
semantic spaces were transferred to the neural network input layer, mapping
lexical units to semantic vectors. For the contextualized representations we
used a fine-tuned BERT [Devlin et al., 2019] and T5 [Raffel et al.,
2019].222Using a different classifier, a Transformer architecture [Vaswani et
al., 2017], for the contextualized representations weakens the comparability
across pre-trained models but the alternative technique of feature extraction
from BERT or T5 would undermine the model by tilting it to a specific layer
and thus capturing a narrow range of linguistic phenomena. For baselines we
used a linear model, support-vector machine (SVM) [Cortes and Vapnik, 1995],
and a randomly initialized BiLSTM neural network.

##  3 Results and Discussion

Model | UKPS | WEBIS | ARAUC
---|---|---|---
P | R | F1 | F* | OOV | P | R | F1 | F* | OOV | P | R | F1 | F* | OOV
SVM | .73 | .73 | .72 |  | .03 | .70 | .69 | .69 |  | .05 | .54 | .54 | .53 |  | .09
Random | .67 | .78 | .72 | .72 | .04 | .65 | .70 | .68 | .70 | .06 | .52 | .64 | .56 | .55 | .11
WordNet | .67 | .79 | .71 | .69 | .22 | .54 | .71 | .61 | .64 | .22 | .51 | .62 | .55 | .56 | .26
SWOW | .71 | .76 | .73 | .72 | .08 | .60 | .84 | .70 | .68 | .09 | .50 | .63 | .55 | .54 | .16
Word2vec | .73 | .78 | .75 | .72 | .19 | .63 | .78 | .70 | .68 | .19 | .49 | .65 | .56 | .54 | .23
GloVe | .73 | .74 | .74 | .72 | .03 | .69 | .77 | .73 | .73 | .06 | .52 | .67 | .58 | .57 | .11
Fasttext | .73 | .78 | .75 | .72 | .04 | .71 | .75 | .73 | .71 | .06 | .51 | .62 | .56 | .55 | .11
Dep. | .71 | .77 | .74 | .72 | .06 | .69 | .80 | .74 | .72 | .06 | .51 | .64 | .57 | .54 | .12
BERT | .87 | .70 | .77 |  |  | .89 | .95 | .92 |  |  | .70 | .70 | .68 |  |
T5 | .74 | .80 | .77 |  |  | .98 | .98 | .98 |  |  | .81 | .67 | .74 |  |
Table 1: Results for all the experiments. P=Precision; R=Recall; F1=F1
measure; F*=F1 measure with a trainable semantic space (weights are not
frozen). OOV=Out-of-vocabulary words.

The experiments results can be found in Table 1. Several models surpassed the
baselines (F1). BERT and T5 surpassed all models. Although obtaining top
performance was not the aim of this work, the results are in line with state-
of-the-art results found in [Schulz et al., 2018] for UKPS, [Al-Khatib et al.,
2016] for WEBIS and [Moens et al., 2007, Palau and Moens, 2009, Rooney et al.,
2012] for ARAUC.

When comparing the different semantic families there is no dominance of one of
these models over the others, with one notable exception, contextual semantic
spaces. It is worth mentioning that WordNet and SWOW have a smaller training
data set and subsequently a higher OOV. Nevertheless, they still obtain
results close to the other models, suggesting that larger data sets may
increase the scores. The scores also indicate that converting semantic
networks (WordNet) and feature-based models (SWOW) fine-tuned to semantic
similarity and relatedness tasks works for the identifications of ADUs.
Although enriched semantic spaces encode a higher level of linguistic
complexity these only improved slightly over the vanilla Word2vec model. The
examples available for ARAUC were insufficient to obtain convergence with the
BiLSTM models. The results also show that, in general, training with a fixed
semantic space versus a trainable semantic space (F1 vs F*) improves the
scores corroborating that transfer learning the knowledge of semantic spaces
leverage the score.

##  4 Analysis

To analyze: a) the impact of the data size, we report on a profile study; b)
the diverseness of the data sets constituent structure, we perform a cross-
train; c) the interpretability of the lexical features, we analyze by proxy
the bag-of-words used as features in the SVM; d) the complementary of the
models, we evaluate a range of ensembles; and finally, e) the task linguistic
complexity, we evaluate the models with downstream tasks encompassing
different fine distinctions of lexical and linguistic organization.

Profiling analysis \- Evaluating the models with incremental training sizes
shows that in general, all models improve with more data (F1). The exception
goes for all non-contextual models trained on ARAUC. When using all the ARAUC
data the scores are two points below the top-performing score. When comparing
the train with half versus full UKPS data the scores only improved two points.
This points out to a possible plateau and the need for more extrinsic
knowledge.

Cross-training \- We experimented training iteratively with a data set and
testing on the left-out data sets. All models obtained a random accuracy,
except for the models trained with an SVM/BERT on UKPS and evaluated on WEBIS.
Although this suggests that UKPS data can be generalized to WEBIS, the results
show a high degree of heterogeneity across data sets.

BoW analysis \- To perform a bag-of-words analysis we intersected the SVM
features obtained across the data sets (unigrams and bigrams). As expected, a
quantitative analysis of the content versus functional unigrams shows that
content words are five times more frequent relevant features. We expected to
find signal words but no obvious patterns or data bias were identified across
the data sets. This seems to indicate that grammatical structure and context
play a major role in solving the identification of ADUs.

Ensemble \- We experimented with several model ensembles using the largest
data set (UKPS). All the ensembles obtained worst results except a weighted
ensemble with marginal improvements.333Ensemble obtained with BERT, SVM and
the Dependency model improving +0.003 F1. We hypothesized that knowledge from
different semantic families as WordNet or SWOW would complement other semantic
spaces but this was not shown on the ensemble scores.

Figure 1: Linguistic complexity evaluation: Performance of models with
different semantic spaces (bars) over four different GLUE downstream tasks
(groups), measured in different evaluation metrics projected to a common range
[0, 100]. The reported scores are the average of three runs.

Parallel with fine-grained downstream tasks \- To assert a possible range of
linguistic complexity addressed by the semantic spaces we resort to the
evaluation of parallel downstream tasks. These tasks benefit in different
degrees of fine-grained information regarding the lexical knowledge required
to address them, some with a fine distinction between the meaning of the
different words. We resorted to four tasks of GLUE [Wang et al., 2018], a
standard benchmark for natural language understanding tasks. For the
classifier, we adopted the BiLSTM encoder from [Wang et al., 2019] for
comparability with existing semantic spaces evaluated on the tasks, except
with the contextualized models where we used the fine-tuned version for each
data set. The following downstream tasks were used: CoLA, a language
membership task (grammatical acceptability); RTE, an entailment task with the
data from RTE1-RTE5 (natural language inference); MRPC, a paraphrase task
(semantic equivalence) and SST-2, a sentiment analysis task. The results are
plotted in Figure 1. Arranging the performance of the tasks from worst to
best, as in Figure 1, provides several insights: higher linguistic complexity
downstream tasks are harder to solve, lexical semantic spaces achieve high
performance on low linguistic complexity but fail to address harder tasks and
contextualized semantic spaces are successful across tasks.

In the CoLA task, some models perform below or at best on a par with a random
semantic space. The task is very hard as it relies on rich information about
the grammatical structure of the sentences and to categorically decide
sentence membership in the language defined by the grammar. Hence, the
evaluation scores seem to indicate that the signal from the lexical
information encoded in pre-trained models, from whatever lexical theory or
empirical source, has a somewhat marginal impact. While the fine-tuned
contextual models can address CoLA, the same parallel can be observed with the
ADUs task, this correlation suggests that highly complex linguistic
information plays a major role to solve the task.

##  5 Conclusion

This paper addresses a clear point, to measure the impact of models pre-
trained with different lexical semantic families on ADUs identification. We
presented a focused contribution resorting to three different and
heterogeneous data sets, with the transfer learning of semantic spaces from
different semantic families. Using transfer learning techniques proved to be
beneficial to the task. When proposing that in some way or the other the
correlation of downstream tasks mirrors the linguistic complexity found in the
ADUs task, the results obtained suggest that ARAUC encodes higher linguistic
complexity followed by UKPS and finally by an almost completely lexical
dependent WEBIS data set. We expected the contribution of the commonsense
knowledge encoded in different lexical semantic models. The somewhat negative
results using different lexical semantic families and the lack of improvements
using ensembles points the future work to alternatives for the training-
objectives of current LMs architectures, for example exploring neuro-symbolic
approaches guided with feature-based models and semantic networks.

## References

  * [Agirre et al., 2009]  Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa.  2009\.  A Study On Similarity and Relatedness Using Distributional and Wordnet-Based Approaches.  In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 19–27. Association for Computational Linguistics (ACL).
  * [Agirre et al., 2014]  Eneko Agirre, Oier López de Lacalle, and Aitor Soroa.  2014\.  Random Walks for Knowledge-Based Word Sense Disambiguation.  Computational Linguistics, 40(1):57–84.
  * [Ajjour et al., 2017]  Yamen Ajjour, Wei-Fan Chen, Johannes Kiesel, Henning Wachsmuth, and Benno Stein.  2017\.  Unit Segmentation of Argumentative Texts.  In Proceedings of the 4th Workshop on Argument Mining, pages 118–128.
  * [Al-Khatib et al., 2016]  Khalid Al-Khatib, Henning Wachsmuth, Matthias Hagen, Jonas Köhler, and Benno Stein.  2016\.  Cross-Domain Mining of Argumentative Text Through Distant Supervision.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 1395–1404.
  * [Caruana, 1997]  Rich Caruana.  1997\.  Multitask Learning.  Machine learning, 28(1):41–75.
  * [Choi and Lee, 2018]  HongSeok Choi and Hyunju Lee.  2018\.  GIST at SemEval-2018 Task 12: A Network Transferring Inference Knowledge to Argument Reasoning Comprehension Task.  In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 773–777.
  * [Cocarascu and Toni, 2017]  Oana Cocarascu and Francesca Toni.  2017\.  Identifying attack and support argumentative relations using deep learning.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1374–1379.
  * [Cortes and Vapnik, 1995]  Corinna Cortes and Vladimir Vapnik.  1995\.  Support-vector networks.  Machine learning, 20(3):273–297.
  * [De Deyne et al., 2019]  Simon De Deyne, Danielle J Navarro, Amy Perfors, Marc Brysbaert, and Gert Storms.  2019\.  The “small world of words” english word association norms for over 12,000 cue words.  Behavior research methods, 51(3):987–1006.
  * [Devlin et al., 2019]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2019\.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.
  * [Eger et al., 2017]  Steffen Eger, Johannes Daxenberger, and Iryna Gurevych.  2017\.  Neural End-To-End Learning for Computational Argumentation Mining.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 11–22.
  * [Fellbaum, 1998]  Christiane Fellbaum, editor.  1998\.  Wordnet: An Electronic Lexical Database.  MIT Press.
  * [Hill et al., 2015]  Felix Hill, Roi Reichart, and Anna Korhonen.  2015\.  Simlex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation.  Computational Linguistics, 41(4):665–695.
  * [Hochreiter and Schmidhuber, 1997]  Sepp Hochreiter and Jürgen Schmidhuber.  1997\.  Long short-term memory.  Neural computation, 9(8):1735–1780.
  * [Jo et al., 2019]  Yohan Jo, Jacky Visser, Chris Reed, and Eduard Hovy.  2019\.  A cascade model for proposition extraction in argumentation.  In Proceedings of the 6th Workshop on Argument Mining, pages 11–24. Association for Computational Linguistics.
  * [Joulin et al., 2016]  Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov.  2016\.  Fasttext.zip: Compressing text classification models.  arXiv preprint arXiv:1612.03651.
  * [Kiros et al., 2015]  Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  2015\.  Skip-thought vectors.  In Advances in neural information processing systems, pages 3294–3302.
  * [Lawrence and Reed, 2020]  John Lawrence and Chris Reed.  2020\.  Argument mining: A survey.  Computational Linguistics, 45(4):765–818.
  * [Levy and Goldberg, 2014]  Omer Levy and Yoav Goldberg.  2014\.  Dependency-based word embeddings.  In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 2, pages 302–308.
  * [Li et al., 2017]  Lei Li, Liyuan Mao, and Moye Chen.  2017\.  Word embedding and topic modeling enhanced multiple features for content linking and argument/sentiment labeling in online forums.  In Proceedings of the MultiLing 2017 Workshop on Summarization and Summary Evaluation Across Source Types and Genres, pages 32–36.
  * [Mikolov et al., 2013]  Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  2013\.  Distributed representations of words and phrases and their compositionality.  In Advances in Neural Information Processing Systems, pages 3111–3119.
  * [Moens et al., 2007]  Marie-Francine Moens, Erik Boiy, Raquel Mochales Palau, and Chris Reed.  2007\.  Automatic Detection of Arguments in Legal Texts.  In Proceedings of the 11th International Conference on Artificial Intelligence and Law, pages 225–230. ACM.
  * [Palau and Moens, 2009]  Raquel Mochales Palau and Marie-Francine Moens.  2009\.  Argumentation Mining: the Detection, Classification and Structure of Arguments in Text.  In Proceedings of the 12th International Conference on Artificial Intelligence and Law, pages 98–107. ACM.
  * [Pathak et al., 2016]  Arkanath Pathak, Pawan Goyal, and Plaban Bhowmick.  2016\.  A two-phase approach towards identifying argument structure in natural language.  In Proceedings of the 3rd Workshop on Natural Language Processing Techniques for Educational Applications, pages 11–19.
  * [Peldszus and Stede, 2013]  Andreas Peldszus and Manfred Stede.  2013\.  From Argument Diagrams to Argumentation Mining in Texts: A Survey.  International Journal of Cognitive Informatics and Natural Intelligence (IJCINI), 7(1):1–31.
  * [Pennington et al., 2014]  Jeffrey Pennington, Richard Socher, and Christopher Manning.  2014\.  Glove: Global Vectors for Word Representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
  * [Potash et al., 2017]  Peter Potash, Alexey Romanov, and Anna Rumshisky.  2017\.  Here’S My Point: Joint Pointer Architecture for Argument Mining.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1364–1373.
  * [Raffel et al., 2019]  Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  2019\.  Exploring the limits of transfer learning with a unified text-to-text transformer.  arXiv preprint arXiv:1910.10683.
  * [Reed et al., 2008]  Chris Reed, Raquel Mochales Palau, Glenn Rowe, and Marie-Francine Moens.  2008\.  Language Resources for Studying Argument.  In Proceedings of the International Conference on Language Resources and Evaluation.
  * [Řehůřek and Sojka, 2010]  Radim Řehůřek and Petr Sojka.  2010\.  Software Framework for Topic Modelling with Large Corpora.  In Proceedings of the Language Resources and Evaluation Conference 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Valletta, Malta, May. ELRA.  http://is.muni.cz/publication/884893/en.
  * [Rocha et al., 2018]  Gil Rocha, Christian Stab, Henrique Lopes Cardoso, and Iryna Gurevych.  2018\.  Cross-lingual argumentative relation identification: from English to Portuguese.  In Proceedings of the 5th Workshop on Argument Mining, pages 144–154.
  * [Rooney et al., 2012]  Niall Rooney, Hui Wang, and Fiona Browne.  2012\.  Applying Kernel Methods to Argumentation Mining.  In Twenty-Fifth International FLAIRS Conference.
  * [Schulz et al., 2018]  Claudia Schulz, Steffen Eger, Johannes Daxenberger, Tobias Kahse, and Iryna Gurevych.  2018\.  Multi-Task Learning for Argumentation Mining in Low-Resource Settings.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 35–41.
  * [Simpson and Gurevych, 2018]  Edwin Simpson and Iryna Gurevych.  2018\.  Finding convincing arguments using scalable bayesian preference learning.  Transactions of the Association of Computational Linguistics, 6:357–371.
  * [Stab et al., 2018]  Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna Gurevych.  2018\.  Cross-topic argument mining from heterogeneous sources using attention-based neural networks.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3664–3674.
  * [Vaswani et al., 2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008.
  * [Walton, 2009]  Douglas Walton.  2009\.  Argumentation theory: A very short introduction.  In Argumentation in Artificial Intelligence, pages 1–22. Springer.
  * [Wang et al., 2018]  Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  2018\.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355.
  * [Wang et al., 2019]  Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, et al.  2019\.  Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4465–4476.

The following appendices include information that allow the reproduction of
the paper along with non-essential detailed information about the data sets
(Appendix A), conversion process (Appendix B), semantic spaces (Appendix C),
classifiers (Appendix D) and experiments results (Appendix E).

##  Appendix A Data sets

Next, it follows a detailed description of each data set used in the
evaluation process. An example of each data set is shown in Table 2. In Table
3 the number of sentences and tokens are quantified. We split the data sets in
70% for the train, 10% for the development and 20% for the testing.

\- UKP Sentential Argument Mining Corpus (UKPS) [Stab et al., 2018]. UKPS was
created from eight controversial topics and across domains: news reports,
editorials, blogs, debate forums, and encyclopedia articles. Each sentence in
the corpus was manually annotated (concerning the topic) as argument or non-
argument. The corpus has approximately 25k sentences, 10k arguments and 15k
non-arguments. An argument is defined as a span of text that conveys evidence
or reasoning that can be used to either support or oppose a given topic.

\- Webis Debate 16 (WEBIS) [Al-Khatib et al., 2016]. WEBIS was created from an
online-debate platform and obtained with a distant supervision algorithm. The
meta-data from the debates was mapped to units of argument or non-argument
discourse. Meta-data referring to the debate introduction was labeled as non-
argument, the meta-data for points for or against and counterpoints were
labeled as arguments. The corpus has approximately 30k sentences, 24k
arguments and 5k non-arguments. We under-sampled the data set to balanced it,
resulting in a total of 11,112 examples.

\- AraucariaDB (ARAUC) [Reed et al., 2008]. ARAUC was created from different
domains: news editorials, parliamentary records, judicial summaries and
discussion boards. The corpus uses the Walton [Walton, 2009] argumentative
scheme set theory and is originally stored in a graph structure. As in [Rooney
et al., 2012], all the sentences not containing part of a premise or
conclusion were considered a non-argument. The corpus was under-sampled to
balanced it, it resulted in 3k sentences.

Data set |  Example |  Label
---|---|---
UKP |  We need a safe, genuinely sustainable, global and green solution to our energy needs, not a dangerous diversion like nuclear power. |  argument
There are many notable authors of books and articles that render scientific findings available in lay language to a wider public. |  non-argument
WEBIS |  Having the whole of an ethnicity within one state will help prevent misappropriation of culture and history by another state. |  argument
Since the end of the AU’s peace and security council has had responsibility for maintaining peace in Africa. |  non-argument
ARAUC |  To build a more hopeful America we must help our children reach as far as their vision and character can take them. |  argument
To emphasize the priority of youth affairs the king answered questions from three college students via an international webcast. |  non-argument
Table 2: Examples of what is labeled as argument or non-argument for each of the data sets. Data set | Sentences | Tokens
---|---|---
UKPS | 25,492 | 609,740 (23.92)
WEBIS | 11,112 | 297,635 (26.51)
ARAUC | 3,000 | 53,091 (17.70)
Table 3: Argument identification data sets (after undersampling). The
Sentences column show the total of arguments/non-argument. The Tokens column
show the total of tokens and tokens per sentence.

##  Appendix B Lexical semantic families conversion to semantic spaces

The typical representation factors for each of the lexical semantic families
can be seen in Table 4. For the conversion from the WordNet, a Semantic
Network, to a semantic space the nodes were projected to a raw corpus. Each
node was written to the corpus (lemma) followed by all the nodes with
corresponding relations. For the conversion from SWOW, a Feature-based model,
to a semantic space the keys lexical represention (word) were projected to a
raw corpus. Each key was written to the corpus followed by all the key’s
association cues.

We used the Skip-gram algorithm to create a semantic space for each corpora.
We resorted to the Gensim Skip-gram implementation [Řehůřek and Sojka, 2010].
We fixed the vector dimension at 300 units and performed a grid-search on four
hyper-parameters: window (3, 10, 20), negative samples (5, 25, 100), learning
rate (0.1, 0.01, 0.001) and epochs (1, 5, 10). The final semantic spaces for
WordNet and SWOW obtained 0.5389 and 0.5261 Spearman rank-order correlation
coefficient in Simlex, and with WordSim352-Rel obtained 0.6719 and 0.4541
Spearman rank-order correlation coefficient. For comparison, the word2vec
obtains 0.4361 and 0.6989 Spearman rank-order correlation coefficient in
Simlex and WordSim352-Rel respectively.

Semantic Family | Representation | Lexical Unit | Semantic Relations | Example
---|---|---|---|---
Semantic Networks | graph | node | edge | WordNet
Feature-based models | hash tables | key | key values | SWOW
Semantic spaces | vector space | vector | cosine distance | word2vec
Table 4: Lexical semantic families and their typical representation factors.
The Semantic relations column refers to how the semantic similarity of two
different words is measured.

##  Appendix C Semantic spaces

Table 5 summarizes the semantic spaces (except contextual semantic spaces). We
used the original Word2Vec model [Mikolov et al., 2013] and dependency model
[Levy and Goldberg, 2014]. For GloVe we used the 6 Billion token model trained
with the Wikipedia 2014 and the Gigaword 5 corpora. For Fasttext we used the 1
million word vectors trained on Wikipedia 2017, UMBC web base corpus and
statmt.org news data set.

Model | Domain | #Vocab. | #Dim
---|---|---|---
Word2Vec | News | 3,000,000 | 300
GloVe | Wiki. & News | 400,000 | 300
Fasttext | Wiki. & News | 999,994 | 300
Dependency | Wiki. | 174,015 | 300
Table 5: Summary of semantic spaces models regarding the domain of the data
used for training, the resulting vocabulary of the model and the word vector
dimension.

For the contextual semantic space we used the original BERT models fine-tuned
to the task data sets. We fine-tuned the BERT-Base Uncased
(uncased_L-12_H-768_A-12), on each of the three data sets (UKPS, WEBIS and
ARAUC). We performed a grid-search on 2 hyper-parameters: epoch (10, 8, 5, 3,
1) and learning rate (0.002, 0.0002, 0.00002, 0.000002, 0.000002). We used a
maximum sequence length of 128 units and a 32 train batch size for all of the
models. Given that BERT and T5 use a word segmentation method we can assume
that there are no out-of-vocabulary words.

##  Appendix D Classifiers

The hyper-parameters were obtained from a grid-search using a random (vectors)
semantic space baseline (a strong baseline given that it was the only model,
along with contextual models where the hyper-parameters were tuned). For the
vocabulary of the baseline we used the 1M vocabulary extracted from Fasttext
(only lexical units, random vectors were generated). For the UKPS and WEBIS
development set the top performing hyper-parameters were: 10 epochs, a batch
size of 64, a sequence length of 30, a learning rate of 0.01, a dropout of 0.8
and 48 LSTM units. For the ARAUC development set the top performing hyper-
parameters were: 20 epochs, batch size of 16, a sequence length of 15, a
learning rate of 0.001, a dropout of 0.7 and 1 LSTM unit. All the models were
trained 10 times and the scores presented are the average of all the runs.

##  Appendix E Results

The values for the extrinsic evaluation and used for plotting Figure 1 are
more detailed in Table 6.

| CoLA | RTE | MRPC | SST-2
---|---|---|---|---
Random | 12.37 $\pm$ 0.65 | 57.90 $\pm$ 2.35 | 78.02 $\pm$ 0.78 | 65.70 $\pm$ 0.95
WordNet | 14.13 $\pm$ 0.26 | 54.07 $\pm$ 0.74 | 78.47 $\pm$ 0.77 | 76.37 $\pm$ 0.77
SWOW | 14.97 $\pm$ 1.17 | 55.37 $\pm$ 2.39 | 77.60 $\pm$ 1.07 | 76.20 $\pm$ 0.37
Dependency | 07.80 $\pm$ 2.41 | 58.47 $\pm$ 2.32 | 76.50 $\pm$ 0.43 | 71.17 $\pm$ 1.22
Word2Vec | 11.33 $\pm$ 0.68 | 56.70 $\pm$ 0.0 | 78.40 $\pm$ 1.21 | 78.73 $\pm$ 0.25
fastText | 16.43 $\pm$ 2.25 | 57.73 $\pm$ 1.56 | 79.92 $\pm$ 0.48 | 81.87 $\pm$ 0.76
Glove | 14.40 $\pm$ 0.78 | 61.47 $\pm$ 2.56 | 81.30 $\pm$ 0.23 | 84.57 $\pm$ 0.58
BERT | 52.10 | 66.40 | 86.85 | 93.50
T5 | 51.10 | 80.10 | 89.10 | 95.20
Table 6: Extrinsic evaluation: Performance over five GLUE downstream tasks
(columns) of models with different input layer word embeddings (rows). For the
task CoLA, performance is measured with Matthews correlation. For MRPC, an
average of accuracy and F1 is reported. For the remaining tasks, accuracy is
reported. The evaluation scores were projected to a [0-100] common scale
(higher is better), with bold denoting top results. Each score is the average
of the results from three runs with the random seeds 1147, 1256 and 1179. To
enhance the readability of eventual data patterns, the content of this table
is rendered in Figure 1.


# Rheology of dense suspensions under shear rotation

Frédéric Blanc  François Peters  Université Côte d’Azur, CNRS, Institut de
Physique de Nice (INPHYNI), France  Jurriaan J.J. Gillissen  The Technology
Partnership, Science Park, Melbourn, UK.  Michael E. Cates  DAMTP, Centre for
Mathematical Sciences, University of Cambridge, Wilberforce Road, Cambridge
CB3 0WA, UK  Sandra Bosio  Camille Benarroche  Université Côte d’Azur, CNRS,
Institut de Physique de Nice (INPHYNI), France  Romain Mari  Univ. Grenoble
Alpes, CNRS, LIPhy, 38000 Grenoble, France

###### Abstract

Dense non-Brownian suspensions exhibit a spectacular and abrupt drop in
viscosity under change of shear direction, as revealed by shear inversions
(reversals) or orthogonal superposition. Here, we introduce an experimental
setup to systematically explore their response to shear rotations, where one
suddenly rotates the principal axes of shear by an angle $\theta$, and measure
the shear stresses with a bi-axial force sensor. Our measurements confirm the
genericness of the transient decrease of the resistance to shear under
unsteady conditions. Moreover, the orthogonal shear stress, which vanishes in
steady state, takes non-negligible values with a rich $\theta$-dependence,
changing qualitatively with solid volume fraction $\phi$, and resulting in a
force that tends to reduce or enhance the direction of flow for small or large
$\phi$. These experimental findings are confirmed and rationalized by
particle-based numerical simulations and a recently proposed constitutive
model. We show that the rotation angle dependence of the orthogonal stress
results from a $\phi$-dependent interplay between hydrodynamic and contact
stresses.

Suspensions of non-Brownian hard particles form a large class of complex
fluids [1, 2]. They are dense when solid and fluid are mixed in roughly equal
proportion. Their widespread use in industry calls for proper constitutive
characterization and modelling to enable reliable process design.

In steady state, their viscosity is either deformation-rate-independent [3, 4,
5] or slightly shear thinning [6, 7, 8, 9, 10]. Shear thickening is also
observed when particles are repulsive beyond pure hard-core forces [11, 12],
but we here focus on the strictly hard-sphere case. The viscosity increases
with the solid volume fraction $\phi$, and diverges at the jamming volume
fraction $\phi_{\mathrm{J}}$ [3, 4]. However, dense suspensions exhibit
striking unsteady behaviors, e.g. the sharp viscosity drop in orthogonal
superposition [13, 14, 15, 16, 17, 18, 19]. In shear reversal, where a
suspension initially sheared in steady state under a deformation rate
$\dot{\gamma}$ is suddenly sheared with a rate $-\dot{\gamma}$, the viscosity
drops suddenly at reversal, passes through a minimum value and climbs up to
its steady-state value after a few strain units [20, 21, 22, 23, 24, 25, 26,
27].

Under shear, suspensions develop an anisotropic micro-structure which takes up
most of the stress at large concentrations [28, 29, 30, 11, 31, 32, 5, 24] and
is built in a finite strain [20]. Upon shear reversal, the micro-structure is
initially not compliant with the new direction of shear, leading to a
viscosity dip which ends when the micro-structure is rebuilt in the new
orientation [20]. This behavior may be a vestige of the fragility of jammed
suspensions that can be made to flow by a change of applied load direction
[33, 34, 35].

Characterization of the mechanical response to sudden changes of the strain
axes, beyond shear reversal (which is the extreme case, as the compressional
and elongational axes are swapped) [25, 26], is however absent. Here we fill
this gap by considering the response to _shear rotations_ , i.e. rotations of
the strain axes by an arbitrary angle $\theta$ about the gradient direction.
We perform shear rotations in experiments, with a specifically designed
rheometer; simulations, using discrete element method (DEM); and the
Gillissen-Wilson (GW) constitutive model [36, 37]. This gives us access to the
viscosity drop as a function of angle $\theta$ and post-rotation strain
$\gamma$. This also unveils a new non-Newtonian phenomenon: following a shear
rotation, the shear viscosity orthogonal to the flow direction, $\eta_{32}$,
is transiently finite, reaching up to $50\text{\,}\mathrm{\char 37\relax}$ of
the usual shear viscosity $\eta_{12}$ for large $\phi$. Moreover, we show that
the nature of angular dependence of $\eta_{32}$ depends on $\phi$: while at
moderate $\phi$, $\eta_{32}$ shows a change of sign in $\theta\in[0,\pi]$,
associated to a force resisting shear rotation for small $\theta$ values, for
the largest $\phi$ values it keeps a constant sign. We show that this is due
to the decreasing relative contribution of hydrodynamic stresses versus
contact stresses when $\phi$ increases.

## Experimental setup

Figure 1: Sketch of the setup. (a) View along the flow-gradient direction. Two
translation stages (light green) independently move an upper plate (orange)
and a lower plate (green gray), between which the suspension (light gray) is
sheared. (b) Shear plane view. A force sensor measures the tangential
stresses. (c) View along the flow-gradient direction of the trajectory (thick
white line) of the top plate relative to the bottom plate during a shear
rotation by an angle $\theta$. A transient force
$\bm{F}=F_{\parallel}\bm{e}_{1}+F_{\perp}\bm{e}_{3}$ is recorded after the
shear rotation.

We designed a _cross rheometer_ [38, 39], sketched in Fig. 1, made with two
parallel plates mounted on two motorized linear stages of 25mm stroke (Newport
MFA-CC) acting in perpendicular directions, allowing for arbitrary relative
parallel motion and therefore arbitrary simple shear with velocity gradient
orthogonal to the plates. We apply a simple shear (Fig. 1(b)) with velocity
gradient $\bm{L}=\dot{\gamma}\bm{e}_{1}\bm{e}_{2}$ (with $\bm{e}_{1}$,
$\bm{e}_{2}$ and $\bm{e}_{3}$ respectively the flow, gradient and vorticity
directions), from which we define the strain-rate tensor
$\bm{E}=\dot{\gamma}\hat{\bm{E}}\equiv(\bm{L}+\bm{L}^{\mathrm{T}})/2$. The
shear rate $\dot{\gamma}$ is related to the velocity of the top plane relative
to the bottom plane $\bm{v}=\dot{\gamma}g\bm{e}_{1}$,
$g=$1\text{\,}\mathrm{mm}$$ being the gap width between the two plates.

A force sensor (AMTI HE6x6-1) measures the tangential force
$\vec{F}=\vec{F}_{\parallel}+\vec{F}_{\perp}$ exerted on the lower plate (and
thus by the upper plate on the suspension), with $\vec{F}_{\parallel}$ and
$\vec{F}_{\perp}$ respectively along $\bm{e}_{1}$ and $\bm{e}_{3}$ (Fig.
1(b)-(c)). We define the shear viscosity
$\eta_{12}\equiv\bm{\Sigma}:\bm{e}_{\mathrm{1}}\bm{e}_{\mathrm{2}}/\dot{\gamma}=\vec{F}_{\parallel}\cdot\bm{e}_{1}/(S\dot{\gamma})$,
with $S$ the area of the upper plate, as well as the “orthogonal” shear
viscosity
$\eta_{32}\equiv\bm{\Sigma}:\bm{e}_{\mathrm{3}}\bm{e}_{\mathrm{2}}/\dot{\gamma}=\vec{F}_{\perp}\cdot\bm{e}_{3}/(S\dot{\gamma})$.
Both have opposite $\theta$-parities: $\eta_{12}$ is even,
$\eta_{12}(\gamma,\theta)=\eta_{12}(\gamma,-\theta)$, while $\eta_{32}$ is
odd, $\eta_{32}(\gamma,\theta)=-\eta_{32}(\gamma,-\theta)$ (in particular this
enforces that $\eta_{32}(\gamma,0)$ and $\eta_{32}(\gamma,\pi)$ vanish).

A preshear is first applied over 10 strain units, which is enough to reach
steady state, followed by a shear rotation where we rotate the flow direction
$\bm{e}_{1}$ and vorticity direction $\bm{e}_{3}$ by an angle
$\theta\in[-\pi,\pi]$ around the gradient direction $\bm{e}_{2}$ (thick white
line in Fig. 1(c)). The imposed shear rate is set at
$\dot{\gamma}=0.4\,\mathrm{s}^{-1}$ for all presented experimental data.

The viscosities $\eta_{12}(\gamma,\theta)$ and $\eta_{32}(\gamma,\theta)$ are
recorded via a Data Acquisition System (USB-1608FS-PLUS, MCCDAQ) as a function
of the subsequent strain $\gamma<10$. The strain resolution is
$\approx$3\text{\times}{10}^{-3}$$, the lowest available strain is
$\approx$1\text{\times}{10}^{-2}$$, and $\theta$ is sampled every $\pi/18$.

The suspension particles are polystyrene spheres with a diameter of
$40\text{\,}\mathrm{\SIUnitSymbolMicro m}$ (Microbeads TS40). They are
dispersed in poly(ethylene glycol-ran-propylene glycol) (Sigma-Aldrich,
viscosity $\eta_{0}=$38.4\text{\,}\mathrm{Pa}\text{\,}\mathrm{s}$$ at
$25\text{\,}\mathrm{\SIUnitSymbolCelsius}$, $M_{\mathrm{n}}\approx$12\,000$$)
for suspensions at $\phi=0.45$ or silicone oil (M1000, Roth,
$\eta_{0}=$0.98\text{\,}\mathrm{Pa}\text{\,}\mathrm{s}$$ at
$25\text{\,}\mathrm{\SIUnitSymbolCelsius}$) at $\phi=0.55$ and $0.57$.

## Numerics & model

We perform the same protocol in DEM simulations of a suspension of $N=2000$
frictional particles subject to lubrication and contact forces in a tri-
periodic configuration, using a method described in [39, 31]. The suspension
is bidisperse (with a size ratio $1:1.4$) and the friction coefficient is
$\mu_{\mathrm{p}}=0.5$, in order to match the viscosity values observed
experimentally.

We also compare our results to the predictions of the GW model, a model
capturing the features of shear reversal [36, 40]. A detailed derivation and
discussion of the underlying assumptions of the GW model can be found in [37].
The GW model considers the strain evolution of a fabric tensor
$\langle\bm{n}\bm{n}\rangle$ where $\bm{n}$ is the unit separation vector
between pairs of particles in a near interaction via contact or lubrication
forces

$\partial_{\gamma}\langle\bm{n}\bm{n}\rangle=\hat{\bm{L}}\cdot\langle\bm{n}\bm{n}\rangle+\langle\bm{n}\bm{n}\rangle\cdot\hat{\bm{L}}^{\mathrm{T}}-2\hat{\bm{L}}:\langle\bm{nnnn}\rangle\\\
-\beta\left[\hat{\bm{E}}_{\mathrm{e}}:\langle\bm{nnnn}\rangle+\frac{\color[rgb]{0,0,0}\phi\color[rgb]{0,0,0}}{15}\left(2\hat{\bm{E}}_{\mathrm{c}}+\operatorname{Tr}(\hat{\bm{E}}_{\mathrm{c}})\bm{\delta}\right)\right]\,.$
(1)

The top line of Eq. (1) describes that particle pairs rotate like dumbbells
with the velocity gradient $\hat{\bm{L}}=\bm{L}/\dot{\gamma}$ and the bottom
line describes the association and dissociation of particle pairs due to the
compressive part $\hat{\bm{E}}_{\mathrm{c}}$ and the extensive part
$\hat{\bm{E}_{\mathrm{e}}}$ of the strain rate tensor $\hat{\bm{E}}$ which
pushes particles together and pulls them apart, respectively. The pair
association (and dissociation) rate $\beta$ is a tuneable parameter. The
tensor $\langle\bm{nnnn}\rangle$ is approximated in terms of
$\langle\bm{n}\bm{n}\rangle$ with the Hinch & Leal closure [41]. Furthermore,
the GW model decomposes the stress
$\bm{\Sigma}=\bm{\Sigma}^{\mathrm{H}}+\bm{\Sigma}^{\mathrm{C}}+2\eta_{0}\bm{E}$
in contributions from hydrodynamics, $\bm{\Sigma}^{\mathrm{H}}$, and contacts,
$\bm{\Sigma}^{\mathrm{C}}$,

$\displaystyle\frac{\bm{\Sigma}^{\mathrm{H}}}{\eta_{\mathrm{s}}\dot{\gamma}}=\frac{\alpha_{0}\hat{\bm{E}}:\langle\bm{nnnn}\rangle}{\left(1-\phi/\phi_{\mathrm{RCP}}\right)^{2}},$
$\displaystyle\frac{\bm{\Sigma}^{\mathrm{C}}}{\eta_{\mathrm{s}}\dot{\gamma}}=\frac{\chi_{0}\hat{\bm{E}}_{\mathrm{c}}:\langle\bm{nnnn}\rangle}{\left(1-\xi/\xi_{\mathrm{J}}\right)^{2}},$
(2)

where $\alpha_{0}$ and $\chi_{0}$ are tuneable parameters.
$\bm{\Sigma}^{\mathrm{H}}$ diverges when $\phi$ approaches the random close
packing volume fraction $\phi_{\mathrm{RCP}}=0.65$ and
$\bm{\Sigma}^{\mathrm{C}}$ diverges when the ‘jamming coordinate’
$\xi=-\langle\bm{n}\bm{n}\rangle:\bm{E}_{\mathrm{c}}|\bm{E}_{\mathrm{c}}|^{-1}$,
a proxy for the coordination number, approaches the jamming value
$\xi_{\mathrm{J}}$. By demanding that, in steady shear flow,
$\bm{\Sigma}^{\mathrm{C}}$ diverges when $\phi$ approaches the friction-
dependent jamming volume fraction $\phi_{\mathrm{J}}=0.58$, we have previously
shown that:
$\xi_{\mathrm{J}}=\phi_{\mathrm{J}}\left(213\beta^{2}-234\beta+2080\right)\left[15\left(9\beta^{2}+54\beta+416\right)\right]^{-1}$
[37]. In the SI we argue our choices for $\alpha_{0}=2.4$, $\chi_{0}=2.3$ and
$\beta=7$.

## Results

Figure 2: (a) Shear viscosity $\eta_{12}$ normalized by the steady-state value
$\eta_{12}^{\mathrm{SS}}$ as a function of strain $\gamma$ in experiments for
$\phi=0.45$, for several values of $\theta\in[0,\pi]$, increasing from light
to dark. The minimum values of the viscosity
$\eta_{12}^{\mathrm{min}}/\eta_{12}^{\mathrm{SS}}$ for each $\theta$ are
circled. (b) and (c) $\eta_{12}^{\mathrm{min}}/\eta_{12}^{\mathrm{SS}}$ and
orthogonal viscosity just after rotation
$\eta_{32}^{0^{+}}/\eta_{12}^{\mathrm{SS}}$ as a function of shear rotation
angle $\theta$ for $\phi=0.45,0.55$ and $0.57$. (d) and (e) Polar
representation of $\eta_{12}(\gamma,\theta)$ and $\eta_{32}(\gamma,\theta)$,
normalized by $\eta_{12}^{\mathrm{SS}}$, for $\phi=0.45$, in experiments (top
halves) and numerical simulations (bottom halves). Symmetries impose that
$\eta_{12}$ is even, $\eta_{12}(\gamma,\theta)=\eta_{12}(\gamma,-\theta)$, and
$\eta_{32}$ is odd, $\eta_{32}(\gamma,\theta)=-\eta_{32}(\gamma,-\theta)$. (f)
and (g) Same, but for $\phi=0.57$.

In Fig. 2(a), we show the viscosity $\eta_{12}(\gamma,\theta)$ measured in
experiments, for a moderately dense suspension at $\phi=0.45$. It decreases at
low strain values, then passes through a minimum before increasing back to its
steady-state value. The minimum is located at a strain $\gamma_{\mathrm{min}}$
weakly dependent on $\theta$, from $\gamma_{\mathrm{min}}\approx 0.15$ for
$\theta\approx\pi/2$ to $\gamma_{\mathrm{min}}\approx 0.35$ for shear reversal
($\theta=\pi$). As shown in Fig. 2(b), the minimum value
$\eta_{12}^{\mathrm{min}}$ gradually decreases when $\theta$ increases, to
reach its lowest value for shear reversal. Once normalized by the steady-state
value $\eta_{12}^{\mathrm{SS}}$,
$\eta_{12}^{\mathrm{min}}/\eta_{12}^{\mathrm{SS}}$ for a given $\theta$
decreases when $\phi$ increases, as is already known for shear reversal [25].
Interestingly, for the lowest $\phi=0.45$,
$\eta_{12}^{\mathrm{min}}\approx\eta_{12}^{\mathrm{SS}}$ for
$\theta\lesssim\pi/4$: the suspension seems oblivious to the shear rotation at
small angles. We will see that this is not quite true when considering
$\eta_{32}$.

We compare these data with the DEM ones in a radial representation
$\eta_{12}(\gamma,\theta)$ in Fig. 2(d), with experiments in the top half and
numerics in the bottom half. The agreement is good, besides simulations
predicting a quicker relaxation to steady state than actually observed.

We turn in Fig. 2(e) to $\eta_{32}$, again comparing experiments in the top
half and numerics in the bottom half. Both datasets are in excellent agreement
and reveal a structure mixing first and second order odd circular harmonics
(respectively $\propto\sin\theta$ and $\propto\sin 2\theta$) with similar
amplitudes. For $0<\theta\lesssim\pi/2$, we find $\eta_{32}<0$ for
$\gamma\lesssim 1$, i.e. the suspension exerts on the top plate a “restoring”
force in the direction of decreasing $\theta$ values. In a force control setup
where one sets the upper plate force $\vec{F}_{\parallel}$ rather than its
displacement, the suspension would thus be stable with respect to shear
rotations, by rotating the velocity of the top plate towards lower $\theta$
values. By contrast, for $\pi/2\lesssim\theta<\pi$, we find $\eta_{32}>0$ for
$\gamma\lesssim 2$, which can be interpreted as the suspension tending to
rotate the trajectory of the top plate towards larger $\theta$ values.

In Fig. 2(f),(g), we show $\eta_{12}$ and $\eta_{32}$ for $\phi=0.57$. Both
experimental and numerical data show that the relaxation to steady state is
quicker than at $\phi=0.45$, with a smaller
$\eta_{12}^{\mathrm{min}}/\eta_{12}^{\mathrm{SS}}$ value [25, 27]. More
importantly, the first harmonic of $\eta_{32}$ dominates. For $0<\theta<\pi$,
we find $\eta_{32}>0$: the suspension always tends to push the top plate to
move towards larger $\theta$ values, that is, the response is no longer
stabilizing for small shear rotations. In Fig. 2(c), we highlight this
qualitative change by showing the $\theta$ dependence of the values just after
rotation $\eta_{32}^{0^{+}}$.

Our simulations also show that the fabric evolution after shear rotation does
not mimic the full stress response but only its contact contribution [39].
Notably, the fabric evolution does not exhibit any qualitative change upon
increase of $\phi$ that we could correlate to the change of behavior of
$\eta_{32}$.

Figure 3: Contributions to the orthogonal viscosity from contacts (a) and
hydrodynamics (b), in the GW model in the top halves and DEM simulations in
the bottom halves, for $\phi=0.45$. We chose $\phi_{\mathrm{RCP}}=0.65$,
$\phi_{\mathrm{J}}=0.58$, $\alpha_{0}=2.3$ $\chi_{0}=2.4$, and $\beta=7$,
based on earlier comparisons with DEM simulations [37, 39]. We recall that
$\eta_{32}$ is odd, $\eta_{32}(\gamma,\theta)=-\eta_{32}(\gamma,-\theta)$. (c)
A particle during initial shear at $t=0^{-}$ has more near contacts in
compressional quadrants (red) than elongational quadrants (blue). (d)–(e)
Looking down from the gradient direction, just after shear rotation by
$\theta=\pi/2$ the new compressional and elongational quadrants are
respectively below and above the dashed lines. Contact forces come from the
new compressional, and are dominated by the more numerous contacts in the old
compressional (red arrow), leading to $\eta_{32}^{\mathrm{C}}>0$ (d).
Hydrodynamic forces have symmetric contributions from new compressional and
elongational quadrants, leading to $\eta_{32}^{\mathrm{H}}=0$ (e).

To understand the origin of the $\eta_{32}$ behavior, we interrogate the GW
model and its predictions for the contact and hydrodynamic contributions to
the stress response. As shown in Fig. 3(a), the contact contribution
$\eta^{\mathrm{C}}_{32}$ has a dominant first harmonic, with
$\eta^{\mathrm{C}}_{32}>0$ for $0<\theta<\pi$. The large second harmonic of
$\eta_{32}$ is instead due to the hydrodynamic component
$\eta^{\mathrm{H}}_{32}$ (Fig. 3(b)), which at $\phi=0.45$ still accounts for
a substantial part of the total stress [32]. This is confirmed by numerical
simulations, which compare well to the predictions of the model.

The difference between contact and hydrodynamic contributions can be
qualitatively understood. In Fig. 3(c), we sketch in the shear plane a
particle during preshear. It shows a fore-aft asymmetry: it has more near
interactions (lubricated and in contact) in the compressional quadrants (in
red) than in the elongational one (in blue). The same particle is seen from
the gradient direction $\bm{e}_{2}$ right after a shear rotation with
$\theta=\pi/2$ in Fig. 3(d),(e). After rotation, the fore-aft asymmetry
accumulated in preshear is a “left-right” asymmetry, and fore-aft symmetry is
temporarily restored. Post-rotation contact stresses (Fig. 3(d)) stem from
contacts in the post-rotation compressional quadrant, below the dashed line,
and due to the left-right asymmetry, are dominated by contacts that carry over
from the pre-rotation in the intersect of pre- and post-rotation compressional
quadrants. Contact forces $\vec{F}_{\mathrm{C}}$ in this overlap region (red
vector) are such that $\bm{e}_{3}\cdot\vec{F}_{\mathrm{C}}>0$, giving a
positive contribution to $\eta_{32}$. By contrast, in Fig. 3(d), all
interactions contribute hydrodynamic forces, and the fore-aft symmetry ensures
that the hydrodynamic contribution from the post-rotation elongational and
compressional quadrants share the same $\bm{e}_{1}$ component, but have
opposite $\bm{e}_{3}$ component. This results in a net-zero hydrodynamic
contribution to $\eta_{32}$.

Whereas this reasoning can be extended to show that $\eta^{\mathrm{C}}_{32}>0$
for $\theta\in]0,\pi[$, the sign of $\eta^{\mathrm{H}}_{32}$ for
$\theta\neq\pi/2$ depends on aspects of the distribution of near interactions
that cannot be deduced from symmetry considerations. The GW model however
gives us a quantitative picture alongside a microstructural insight. Calling
$\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}$ the steady-state fabric in pre-
shear, we get the following contributions to $\eta_{32}$ at $\gamma=0^{+}$
[39]

$\displaystyle\frac{\eta^{\mathrm{H}}_{32}}{\eta_{\mathrm{s}}}$
$\displaystyle=\frac{\alpha(\phi)}{14}\sin
2\theta\left(\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{xx}-\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{zz}\right)$
(3) $\displaystyle\frac{\eta^{\mathrm{C}}_{32}}{\eta_{\mathrm{s}}}$
$\displaystyle=\frac{\chi(\xi)}{7}\left[\frac{\sin
2\theta}{4}\left(\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{xx}-\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{zz}\right)-\sin\theta\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{xy}\right]\,,$

with

$\xi=\big{[}\cos^{2}\theta\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{xx}+\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{yy}+\sin^{2}\theta\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{zz}\\\
-2\cos\theta\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{xy}\big{]}/2\,.$

For the contact contribution, the first harmonic dominates for the values of
$\phi$ and $\beta$ investigated. It is such that $\eta^{\mathrm{C}}_{32}>0$
for $0<\theta<\pi$. By contrast, the hydrodynamic contribution only has a
second harmonic. Therefore, when $\alpha(\phi)/\chi(\xi)$ is large enough, the
four-lobed hydrodynamic response dominates, which happens at moderate $\phi$.
However, since contact stresses diverge for $\phi\to\phi_{\mathrm{J}}$ while
hydrodynamic stresses diverge for
$\phi\to\phi_{\mathrm{RCP}}>\phi_{\mathrm{J}}$, the two-lobed contact response
takes over when $\phi$ moves closer to $\phi_{J}$.

The sign of $\eta^{\mathrm{H}}_{32}$ is not obvious, as it is set by
$\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{xx}-\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{zz}$.
With our value of $\beta=7$, we always find
$\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{xx}-\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{zz}<0$
in the GW model. For small $\theta$ values, it therefore “stabilizes” the
microstructure, as
$\operatorname{sgn}\eta^{\mathrm{H}}_{32}=-\operatorname{sgn}\theta$: it
provides a restoring force acting against the rotation of the flow direction.
In simulations,
$\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{xx}-\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{zz}$
is measured tiny [27]. To test the GW model predictions, from our DEM
simulations we compute $\langle\bm{n}\bm{n}\rangle$ based on particle pairs
separated by at most a gap of $\epsilon_{\mathrm{c}}=0.05$ times the average
radius of the pair. We find
$\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{xx}-\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{zz}<0$,
albeit decreasing in amplitude when $\phi$ increases. Interestingly, the value
of
$\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{xx}-\langle\bm{n}\bm{n}\rangle^{\mathrm{ss}}_{zz}$
becomes positive for small enough $\epsilon_{\mathrm{c}}$, which highlights
how subtle the hydrodynamic stabilization is.

We performed shear rotations experimentally, numerically, and in a
constitutive model, measuring both the shear viscosity $\eta_{12}$ and the
orthogonal viscosity $\eta_{32}$ which is not measurable in a conventional
rotational rheometer. It revealed a rich phenomenology. The shear viscosity
exhibits a dip (except at small $\theta$ for the smallest $\phi$ explored
here, $\phi=0.45$) on strain scales of order unity or less, and its amplitude
increases upon increase of $|\theta|$ or $\phi$. Remarkably, we find that
$\eta_{32}\neq 0$ during the post-rotation transient, with $|\eta_{32}|$
reaching up to $50\text{\,}\mathrm{\char 37\relax}$ of $\eta_{12}$ at
$\phi=0.57$, and $10\text{\,}\mathrm{\char 37\relax}$ of
$\eta_{12}^{\mathrm{SS}}$. The qualitative angular structure of
$\eta_{32}(\gamma,\theta)$ depends on $\phi$, which is explained by the
predominance of either hydrodynamic or contact stresses. For $\phi=0.45$, the
second harmonic of $\eta(\gamma,\theta)$ is large as hydrodynamic stresses are
significant, while for $\phi=0.57$ it is small as contact stresses dominate.
Consequently, for small $\theta$, $\eta_{32}$ produces a force that acts to
reduce (stabilize) $\theta$ at smaller $\phi$ while it increases
(destabilizes) $\theta$ at larger $\phi$.

In an actual non-uniform or unsteady flow where strain axes rotation occur
over a strain $\gamma_{\mathrm{unsteady}}$, we can define a Deborah-like
number $\mathrm{De}=\gamma_{\mathrm{min}}/\gamma_{\mathrm{unsteady}}$ [42,
43], such that one should expect to observe the transient effects described
here when $\mathrm{De}\gtrsim 1$. The decrease of $\eta_{12}$ under shear
rotations has already been used to suggest energy-saving flow strategies [16,
17, 19], however the behavior of $\eta_{32}$ has so far been overlooked.
Whereas in this work we impose the deformation and measure $\eta_{32}$, in
many cases one imposes the force or stress. A finite $\eta_{32}$ may lead to
non-trivial trajectories, e.g. during the pulling or the sedimentation of an
object in a dense suspension, especially if the suspension is not stable
against shear rotations.

We have focused on a subset of the possible flow changes in a complex
geometry. One would also need to characterize changes from simple shear to
extensional flows and non-uniform flows, which are known to induce migration
phenomena [44]. While characterizing all flow histories relevant for
applications (or even a carefully selected subset) is a major task, our
results show that it would certainly reveal non-trivial yet possibly important
stress responses. The GW model captures the salient features of the stress
response under shear rotation and in non-uniform flows [45], and could also
prove an efficient design tool in this endeavour.

## Acknowledgements.

Work funded in part by the European Research Council under the Horizon 2020
Programme, ERC grant agreement number 740269.

## References

  * Denn and Morris [2014] M. M. Denn and J. F. Morris, Annual Review of Chemical and Biomolecular Engineering 5, 203 (2014).
  * Ness  _et al._ [2022] C. Ness, R. Seto, and R. Mari, Annual Review of Condensed Matter Physics 13, 97 (2022).
  * Ovarlez  _et al._ [2006] G. Ovarlez, F. Bertrand, and S. Rodts, Journal of Rheology (1978-present) 50, 259 (2006).
  * Boyer  _et al._ [2011] F. Boyer, E. Guazzelli, and O. Pouliquen, Physical Review Letters 107, 188301 (2011).
  * Guy  _et al._ [2015] B. M. Guy, M. Hermes, and W. C. K. Poon, Physical Review Letters 115, 088304 (2015).
  * Zarraga  _et al._ [2000] I. E. Zarraga, D. A. Hill, and D. T. L. Jr, Journal of Rheology (1978-present) 44, 185 (2000).
  * Dai  _et al._ [2013] S.-C. Dai, E. Bertevas, F. Qi, and R. I. Tanner, Journal of Rheology (1978-present) 57, 493 (2013).
  * Dbouk  _et al._ [2013] T. Dbouk, L. Lobry, and E. Lemaire, Journal of Fluid Mechanics 715, 239 (2013).
  * Vázquez-Quesada  _et al._ [2016] A. Vázquez-Quesada, R. I. Tanner, and M. Ellero, Physical Review Letters 117, 108001 (2016).
  * Lobry  _et al._ [2019] L. Lobry, E. Lemaire, F. Blanc, S. Gallier, and F. Peters, Journal of Fluid Mechanics 860, 682 (2019).
  * Seto  _et al._ [2013] R. Seto, R. Mari, J. F. Morris, and M. M. Denn, Physical Review Letters 111, 218301 (2013).
  * Wyart and Cates [2014] M. Wyart and M. E. Cates, Physical Review Letters 112, 098302 (2014).
  * Ovarlez  _et al._ [2010] G. Ovarlez, Q. Barral, and P. Coussot, Nature materials 9, 115 (2010).
  * Barral [2011] Q. Barral,  _Superposition d’écoulements orthogonaux dans des fluides complexes: mise en place de l’expérience, application aux suspensions et aux fluides à seuil_ , Ph.D. thesis, Université Paris-Est (2011).
  * Blanc  _et al._ [2014] F. Blanc, E. Lemaire, and F. Peters, Journal of Fluid Mechanics 746, 10.1017/jfm.2014.160 (2014).
  * Lin  _et al._ [2016] N. Y. C. Lin, C. Ness, M. E. Cates, J. Sun, and I. Cohen, Proceedings of the National Academy of Sciences 113, 10774 (2016).
  * Ness  _et al._ [2018] C. Ness, R. Mari, and M. E. Cates, Science Advances 4, eaar3296 (2018).
  * Ramaswamy  _et al._ [2022] M. Ramaswamy, I. Griniasty, J. P. Sethna, B. Chakraborty, and I. Cohen, Incorporating tunability into a universal scaling framework for shear thickening (2022), arXiv:2205.02184 [cond-mat] .
  * Acharya and Trulsson [2023] P. Acharya and M. Trulsson, Optimum dissipation by cruising in dense suspensions (2023), arXiv:2302.08810 [cond-mat] .
  * Gadala‐Maria and Acrivos [1980] F. Gadala‐Maria and A. Acrivos, Journal of Rheology 24, 799 (1980), publisher: The Society of Rheology.
  * Narumi  _et al._ [2002] T. Narumi, H. See, Y. Honma, T. Hasegawa, T. Takahashi, and N. Phan-Thien, Journal of Rheology (1978-present) 46, 295 (2002).
  * Kolli  _et al._ [2002] V. G. Kolli, E. J. Pollauf, and F. Gadala-Maria, Journal of Rheology (1978-present) 46, 321 (2002).
  * Blanc  _et al._ [2011a] F. Blanc, F. Peters, and E. Lemaire, Journal of Rheology (1978-present) 55, 835 (2011a).
  * Lin  _et al._ [2015] N. Y. C. Lin, B. M. Guy, M. Hermes, C. Ness, J. Sun, W. C. K. Poon, and I. Cohen, Physical Review Letters 115, 228304 (2015).
  * Peters  _et al._ [2016] F. Peters, G. Giovanni, S. Gallier, F. Blanc, E. Lemaire, and L. Lobry, Journal of Rheology 60, 715 (2016).
  * Ness and Sun [2016] C. Ness and J. Sun, Physical Review E 93, 012604 (2016).
  * Chacko  _et al._ [2018] R. N. Chacko, R. Mari, S. M. Fielding, and M. E. Cates, Journal of Fluid Mechanics 847, 700 (2018).
  * Lootens  _et al._ [2005] D. Lootens, H. van Damme, Y. Hémar, and P. Hébraud, Physical Review Letters 95, 268302 (2005).
  * Blanc  _et al._ [2011b] F. Blanc, F. Peters, and E. Lemaire, Physical Review Letters 107, 208302 (2011b).
  * Blanc  _et al._ [2013] F. Blanc, E. Lemaire, A. Meunier, and F. Peters, Journal of Rheology (1978-present) 57, 273 (2013).
  * Mari  _et al._ [2014] R. Mari, R. Seto, J. F. Morris, and M. M. Denn, Journal of Rheology (1978-present) 58, 1693 (2014).
  * Gallier  _et al._ [2014] S. Gallier, E. Lemaire, F. Peters, and L. Lobry, Journal of Fluid Mechanics 757, 514 (2014).
  * Cates  _et al._ [1998] M. E. Cates, J. P. Wittmer, J.-P. Bouchaud, and P. Claudin, Physical Review Letters 81, 1841 (1998).
  * Seto  _et al._ [2019] R. Seto, A. Singh, B. Chakraborty, M. M. Denn, and J. F. Morris, Granular Matter 21, 82 (2019).
  * Giusteri and Seto [2021] G. G. Giusteri and R. Seto, Physical Review Letters 127, 138001 (2021).
  * Gillissen and Wilson [2018] J. J. J. Gillissen and H. J. Wilson, Physical Review E 98, 033119 (2018).
  * Gillissen  _et al._ [2020] J. J. J. Gillissen, C. Ness, J. D. Peterson, H. J. Wilson, and M. E. Cates, Journal of Rheology 64, 353 (2020), publisher: The Society of Rheology.
  * Lin  _et al._ [2014] N. Y. C. Lin, J. H. McCoy, X. Cheng, B. Leahy, J. N. Israelachvili, and I. Cohen, Review of Scientific Instruments 85, 033905 (2014).
  * [39] See Supplemental Material at [URL will be inserted by publisher] for the measured steady-state rheology, validation of the cross-rheometer, details on the Gillissen-Wilson model, calculations leading to Eq. (3), DEM implementation and simulation results for the post-rotation fabric evolution, which includes Refs. [45-51].
  * Gillissen  _et al._ [2019] J. J. J. Gillissen, C. Ness, J. D. Peterson, H. J. Wilson, and M. E. Cates, Physical Review Letters 123, 214504 (2019).
  * Hinch and Leal [1976] E. J. Hinch and L. G. Leal, J. Fluid Mech. 76, 187 (1976).
  * Macosko [1994] C. W. Macosko,  _Rheology: Principles, Measurements, and Applications_ , Advances in Interfacial Engineering Series (VCH, New York, 1994).
  * Poole [2012] R. Poole, Rheol. Bull 53, 32 (2012).
  * Guazzelli and Pouliquen [2018] É. Guazzelli and O. Pouliquen, Journal of Fluid Mechanics 852, P1 (2018).
  * Gillissen and Ness [2020] J. J. J. Gillissen and C. Ness, Physical Review Letters 125, 184503 (2020).
  * Jeffrey and Onishi [1984] D. J. Jeffrey and Y. Onishi, J. Fluid Mech. 139, 261 (1984).
  * Ball and Melrose [1997] R. C. Ball and J. R. Melrose, Physica A 247, 444 (1997).
  * Cundall and Strack [1979] P. A. Cundall and O. D. L. Strack, Geotechnique 29, 47 (1979).
  * Brady and Bossis [1988] J. F. Brady and G. Bossis, Ann. Rev. Fluid Mech. 20, 111 (1988).
  * Arshad  _et al._ [2021] M. Arshad, A. Maali, C. Claudet, L. Lobry, F. Peters, and E. Lemaire, Soft Matter 17, 6088 (2021).
  * Bounoua  _et al._ [2019] S. N. Bounoua, P. Kuzhir, and E. Lemaire, Journal of Rheology 63, 785 (2019).