# Paralinguistics-Enhanced Large Language Modeling of
Spoken Dialogue

###### Abstract

Large Language Models (LLMs) have demonstrated superior abilities in tasks
such as chatting, reasoning, and question-answering. However, standard LLMs
may ignore crucial paralinguistic information, such as sentiment, emotion, and
speaking style, which are essential for achieving natural, human-like spoken
conversation, especially when such information is conveyed by acoustic cues.
We therefore propose Paralinguistics-enhanced Generative Pretrained
Transformer (ParalinGPT), an LLM utilizes text and speech modality to better
model the linguistic content and paralinguistic attribute of spoken response.
The model takes the conversational context of text, speech embeddings, and
paralinguistic attributes as input prompts within a serialized multitasking
multi-modal framework. Specifically, our framework serializes tasks in the
order of current paralinguistic attribute prediction, response paralinguistic
attribute prediction, and response text generation with autoregressive
conditioning. We utilize the Switchboard-1 corpus, including its sentiment
labels to be the paralinguistic attribute, as our spoken dialogue dataset.
Experimental results indicate the proposed serialized multitasking method
outperforms typical sequence classification techniques on current and response
sentiment classification. Furthermore, leveraging conversational context and
speech embeddings significantly improves both response text generation and
sentiment prediction. Our proposed framework achieves relative improvements of
6.7%, 12.0%, and 3.5% in current sentiment accuracy, response sentiment
accuracy, and response text BLEU score, respectively.

Index Terms—  Spoken dialogue modeling, large language models,
paralinguistics, speech sentiment analysis.

##  1 Introduction

Large language models (LLMs) [1] have enabled substantial recent progress in
dialogue generation, language understanding, and reasoning, mostly operating
on text. However, the most natural form of communication for human-human
interaction is speech. Speech signals contain a wealth of information related
to human communication, encompassing linguistic aspects (such as words,
phonetics, syntax, and semantics), paralinguistic elements (including
emotions, sentiments, and speaker characteristics), and pragmatic factors
(like sarcasm and attitude).

Fig. 1: Speech dialogue scenario. <> denotes paralinguistic sentiment
associated with an utterance. “Context History” refers to conversational turns
before the “Current Turn”, and “Response Turn” is what follows the current
turn.

In human-machine interaction environments, deployment of LLMs that solely rely
on lexical aspects of speech thus fail to capture vital information present in
human spoken dialogue. Consequently, LLMs struggle to perceive and
differentiate between speaking styles, often responding to users in a
monotonous manner [2]. Furthermore, there are instances where subtle
differences in speaking styles can alter the semantic meaning of an utterance,
such as with sarcasm [3]. Consider the example shown in Figure 1: when the
current turn is ”Umm … I think it wasn’t a bad game” conveyed with a negative
tone, the textual information still leans towards a positive or neutral
sentiment. By ignoring the speech signal the actual sentiment of the speaker
would be misintepreted.

Despite the existence of prior work that leverages sentiment or emotion labels
to enhance dialogue modeling [4, 5, 6, 7, 8, 9, 2], these studies are
primarily based on textual dialogue data, with sentiment or emotion labels
being inferred from the word alone. On the other hand, prior work in modeling
spoken dialogue has also shown that prosodic information is helpful for
language modeling, i.e., predicting the words [10, 11].

In this work, alongside the linguistic information extracted from text, we
introduce ParalinGPT, a language model capable of incorporating speech
embeddings from a self-supervised speech encoder as input prompts, within a
serialized multitasking multi-modal framework designed to predict upcoming
text- and speech-based paralinguistic information. We employ the Switchboard-1
dataset, the largest collection of human-human spontaneous spoken dialogues
with annotated speech sentiments, as our primary spoken dialogue data source.

Fig. 2: ParalinGPT and serialized multitasking: The history context (including
text, speech, and sentiment labels), current text, and current speech encoding
are the input prompt for the ParalinGPT LLM. The prediction target is the
current sentiment label, the response sentiment label, and the response text
via autoregression.

To our knowledge, this is a novel attempt to employ a multi-modal language
model for multiple tasks involving both response text generation and
prediction of response and current sentiment labels, particularly within the
context of real-world spoken dialogues. Our specific contributions include:

  * •

Use of LLM to infer current sentiment, predict response sentiment and use both
to predict the textual response, using an unified serialized multitasking
framework.

  * •

Demonstration that utilizing speech features and context history sentiment
labels improves current and response sentiment prediction, as well as response
text generation, in natural human-human speech dialogue.

  * •

Results showing that the framework achieves superior performance compared to
sequence classification baselines, with 6.7% and 12.0% accuracy improvement
for current and response sentiment, respectively, while seeing 3.5% relative
improvement in BLEUd score for response text generation.

##  2 Related Work

###  2.1 Dialogue Modeling with Sentiment and Emotion

There are several prior works aiming to model the sentiment [4, 5, 12, 13, 14,
15] or emotion [12, 6, 7, 8, 9] in dialogue. Most of this work concentrates on
current-turn sentiment/emotion as a pure classification task. However,
previous studies have also shown that using multi-modal approaches are crucial
for sentiment and emotion analysis [16, 17, 18, 19, 20, 21].

There have been fewer attempts at response modeling where the goal is to
predict the response sentiment [4] or emotion [12] label; others are
leveraging the sentiment [15, 5] or emotion label [6, 7, 8, 9, 2] to guide the
response text generation. It is worth noting that Varshney et al. [8] proposed
a multi-tasking framework with context-turn emotion recognition and response
generation, showing improved emotional relevancy of the generated response,
but they do not predict the response emotion. Compared to this prior work, our
proposal is novel in the following aspects: (i) we use multi-modal input
(speech and text) with multi-turn dialog context on real-world spoken
dialogue, and (ii) we model current and response sentiment, as well as
response text as conditioned on the paralinguistic labels.

###  2.2 Language Modeling for Speech

Much recent research uses spoken language modeling on discrete speech/audio
units [22, 23, 24, 25, 26]. [22, 23, 24, 25] use a decoder-only LM to perform
speech continuation. AudioPaLM [26] and SpeechGPT [27] leverage paired,
aligned transcripts and speech for ASR and TTS tasks. However, discrete
audio/speech units mainly capture content information and lack prosodic and
paralinguistic information. pGSLM [23] explicitly models extra pitch and
duration units to improve speech naturalness. Notably, dGSLM [24] is the only
study of two-channel language modeling for spoken dialogue, but the generated
speech is still lacking in semantic coherence. Different from the above-
mentioned studies, we focus on equipping a strong text LM with continuous
speech embeddings to capture the paralinguistic signal in spoken dialogue.

##  3 Method

###  3.1 The Serialized Multitasking Framework

In the course of spontaneous natural human conversation, when the next speaker
prepares to respond, a natural progression involves understanding the ongoing
turn (including current sentiment), selecting an appropriate response style
(response sentiment prediction), and subsequently crafting the response
content (response text generation).

Inspired by the above and to move beyond isolated predictions, we propose a
sequential conditioning mechanism, encompassing current sentiment prediction,
response sentiment prediction, and response text generation, all joined in an
autoregressive chain; we call this framework serialized multitasking since the
model’s input and output streams encode multiple tasks. The proposed framework
is illustrated in Figure 2.

We formulate the problem as follows: given a multi-turn spoken dialog with
historical context turns $H$ and window size $w$, comprising speech
$\\{S_{n-w},..,S_{n-1}\\}$, text $\\{T_{n-w},..,T_{n-1}\\}$, and sentiment
labels $\\{L_{n-w},..,L_{n-1}\\}$, a current turn $C$ comprising speech
$S_{n}$ and $T_{n}$, our prediction target $O$ is the current sentiment
$L_{n}$, response sentiment $L_{n+1}$, and response text $T_{n+1}$. The
overall objective is as follows:

$\displaystyle P(O|H,C)$ $\displaystyle=$ $\displaystyle
P(L_{n}|H,C)P(L_{n+1}|H,C,L_{n})$ (1) $\displaystyle\quad
P(T_{n+1}|H,C,L_{n},L_{n+1}),$

where
$H=\\{F_{t}(T_{n-w}),F_{t}(L_{n-w}),F_{s}(S_{n-w}),...,F_{t}(T_{n-1}),\\\
F_{t}(L_{n-1}),F_{s}(S_{n-1})\\}$, and $C=\\{F_{t}(T_{n}),F_{s}(S_{n})\\}$.
$F_{s}$ is a utterance-wise speech encoder, $F_{t}$ is the subword embedding.
The sentiment labels, $L$, are incorporated into the text using special
markers <$L$>. We treat the concatenation of $(H,C)$ as the input prompt, and
directly feed the continuous embedding into the LM input space.

The training objective is the typical cross-entropy loss for language modeling
with teacher forcing. During testing, we generate $O$ by autoregressively
feeding earlier predicted tokens back into the LLM inputs to predict a target
sequence that contains current sentiment, response sentiment, and response
text all presented as text. Note that the text and sentiment label in the
history $H$, as well as the current text tokens $T_{n}$ are ground truth.
Future work can look at using ASR output and erroneous versions of historical
sentiment labels. After sequence generation, we use the special marker <> to
decode sentiment predictions for evaluation.

###  3.2 Language Model

Due to the recent success of autoregressive LLMs, we focus on the decoder-only
transformer model for our purposes. We utilize DialoGPT [28] as our pretrained
LM, which is a GPT2 model fine-tuned on large amounts of text dialogue (Reddit
posts), with human-level dialogue generation capability.

Method # Current Context History Prediction Target Modality Modality Sentiment
Curr UA ($\uparrow$) Res UA ($\uparrow$) BLEU ($\uparrow$) Lu et al. [29] 1
E2E ASR ✗ ✗ 62.4 - - Shon et al. [30] 2 T ✗ ✗ 64.5 - - Human performance [29]
3 ✗ ✗ ✗ 84.6 - - Random baseline 4 ✗ ✗ ✗ 33.3 33.3 - LM only 5 T ✗ ✗ - - 14.2
6 T T ✗ - - 14.5 Sequence Classification Baseline∗ 7 T ✗ ✗ 66.3 36.1 - 8 S ✗ ✗
65.1 34.8 - 9 T + S ✗ ✗ 68.1 37.4 - Proposed Serialized Multitasking Framework
10 T ✗ ✗ 68.3 38.0 14.0 11 T T ✗ 69.3 40.5 14.4 12 T T ✓ 70.3 41.4 14.5 13 T +
S ✗ ✗ 70.7 38.6 14.1 14 T + S T + S ✗ 71.3 40.9 14.7 15 T + S T + S ✓ 71.7
41.9 14.9

Table 1: Experimental results: “Curr” and “Res” denote current and response
sentiment, respectively. In the “Modality” column, “T” indicates text, and “S”
denotes speech. “✓” indicates presence and “✗” indicates absence. ∗As a
sequence classification baseline, we train separate models for current
sentiment and response sentiment.

###  3.3 Speech Encoder

We use the wav2vec 2.0-large [31] self-supervised model pretrained on
Switchboard, Fisher, LibriLight [32] as the speech encoder. Wav2vec 2.0 has
been shown to encode rich prosodic information useful for speech sentiment and
sarcasm detection [20]. We use the middle layer (the 12th of 24 layers) of the
wav2vec 2.0 model for frame-wise feature extraction, since the middle layer
achieved best results in our initial linear probing experiments with sentiment
analysis, aligned with the finding in [33]. We apply mean-pooling and a linear
feature projector (1.04M parameters, 0.3% of all LM parameters) to extract
utterance-wise embeddings. To reduce computational cost, we freeze the
parameters of wav2vec 2.0, and only update the linear projector during
training.

##  4 Experiments

###  4.1 Data and metrics

We use the Switchboard-1 corpus [34] (LDC97S62), along with speech sentiment
annotations (LDC2020T14) [35]. The sentiment labels were annotated through a
three-way human agreement process, categorizing utterances into positive,
neutral or negative. This dataset encompasses a total of 140 hours of audio.
By utilizing the temporal information for the starts and ends of audio
segments, we are able to retrieve the context history of each conversational
turn. Since the data split from [35] is not released, we randomly sample 5% of
conversations without speaker overlap as holdout validation and testing sets.
It is important to emphasize that our dataset division adheres to a strict
dialogue-level separation without speaker overlap. The details of data
statistics are listed in Table 2. We normalize the text data to lowercase and
remove nonlexical markers such as laughter and noises.

Train Dev Test # samples 43600 1158 1319 # dialogues 1124 56 56 # speakers 385
86 72 Class 13,074/7,317/23,209 349/222/587 316/190/813

Table 2: Dataset statistics. The “class” row lists the distribution of
positive/negative/neutral classes.

For evaluating sentiment classification, similar to [35], we use unweighted
accuracy (UA) for both current and response sentiment prediction, to account
for the sentiment class imbalance in the dataset. To evaluate response text
generation, we use the BLEU (4-gram) score to measure the content similarity
of the generated text response to the ground truth.

###  4.2 Implementation Details

We utilize the pretrained DialoGPT with 345M
parameters111https://huggingface.co/microsoft/DialoGPT-medium as LM and
wav2vec 2.0 Large as the speech
encoder222https://huggingface.co/facebook/wav2vec2-large-robust. For the
context history, we use window size $w=4$ and truncate the input if the length
exceeds 320 tokens during training. We use a batch size of 32 and Adam
optimizer with learning rate 0.0001, and train for 100k steps. During
training, the parameters of the speech encoder are frozen and only the
parameters of the LM and speech linear projector are updated. We select the
best model according to the validation set, and report the performance on the
test set.

###  4.3 Baseline approaches

We consider several baselines to benchmark our proposed model for each task,
based on prior published results.
Response text generation: We fine-tune DialoGPT on Switchboard data using
typical LM objective and multi-turn context history as a baseline.
Sentiment classification: We use three baseline systems to benchmark sentiment
classification against our proposed technique. (i) Random baseline, since ours
is the first work targeting response sentiment prediction on the Switchboard
corpus. (ii) Sequence classification: for speech, we fine-tune wav2vec-2.0
with a classifier head; for text, we fine-tune DialoGPT for simple
classification; for multi-modal approaches combining text and speech we
augment DialoGPT with input text concatenated with wav2vec 2.0 mean-pooled
embeddings. (iii) Prior published results on the Switchboard corpus for
utterance classification.

##  5 Results

###  5.1 Baselines

From prior work on current-utterance sentiment analysis, we quote the
performance reported in [29, 30], while noting differences in terms of the
data split. For response sentiment, we utilize a random (chance) baseline,
since there is no prior work on Switchboard to compare to. The sequence
classification baseline is used to assess the task difficulty and for
comparison to our proposed serialized multitasking method.

From the results in Table 1, the Text+Speech (row 9) approach consistently
yields superior performance compared to the Text-only (row 7) and Speech-only
(row 8) approaches. This suggests that for both current and response sentiment
prediction, speech and text modalities complement each other. For response
sentiment prediction, the performance stands at 37.0% UA, which is
considerably lower than the performance for current sentiment classification,
as the latter represents a more challenging task (since the corresponding
words and speech are not known yet). Overall, our sentiment classification
baseline reaches comparable and even better performance than the prior
studies, and yields nontrivial response sentiment prediction compared to the
random baseline.

For evaluations of response text generation, we use a domain-adapted DialoGPT
model as baseline (rows 5 and 6 in Table 1). It achieves a BLEU score of 14.2
and 14.5 without and with multi-turn context, respectively.

###  5.2 Serialized multitasking framework

Comparing the proposed method (Table 1, row 10) to the classification baseline
(row 7), we observe better performance for both current and response
sentiment, with 3.0% relative UA improvement for current sentiment and 2.4%
relative improvement for response sentiment. These results suggest that joint
sentiment and language modeling, in the correct order, can lead to
improvements beyond relying on sequence classification alone. However, for
response text generation, we see a slightly worse BLEU score compared to the
LM-only baseline (rows 10 and 5). One possible reason for this is that the
response sentiment prediction is noisy which can affect the subsequent text
generation.

We also experiment with varying the ordering of tasks in the serialized
multitasking framework, as shown in Table 3. We find that predicting the
current sentiment followed by response sentiment and finally the response word
tokens yields the best results. This suggests that response sentiment can be
useful to guide the generation of response text. The response sentiment
prediction also benefits by conditioning on the current-utterance sentiment
prediction. Besides, when current sentiment prediction is absent, we see a
slight performance drop compared to the proposed method, which shows that
understanding current sentiment is helpful to generate proper responses.

###  5.3 Multi-modal serialized multitasking framework

Additionally, incorporating speech embeddings improves both current and
response sentiment classification, with a 3.5% and 1.6% relative UA
improvement, respectively. These results suggest that the inclusion of the
speech modality provides valuable nonlexical information, such as laughter and
expressive speaking styles, which complement the text modality effectively.

Table 3: Results breakdown with respect to task ordering for the text
modality. $L_{n}$: current sentiment label, $L_{n+1}$ is the response
sentiment label, and $T_{n+1}$ is the response text.

Task order Curr UA Res UA BLEU $L_{n}\rightarrow L_{n+1}\rightarrow T_{n+1}$
(Proposed) 70.3 41.4 14.5 $L_{n}\rightarrow T_{n+1}\rightarrow L_{n+1}$ 64.6
38.8 13.1 $L_{n+1}\rightarrow T_{n+1}$ - 40.9 14.4

Fig. 3: Effect of context length on response sentiment classification and text
generation with the proposed method.

###  5.4 Role of Multi-turn Context

Further, we leverage multi-turn context history in the form of text, speech
and sentiment information (see column ”Context History” in Table 1).
Textual context: Text context is crucial for both the LM-only baseline and the
proposed approach. We observe a 2.1% relative BLEU score improvement in the
LM-only baseline. Moreover, for the text serialized multitasking method,
textual context also contributes to improved sentiment prediction. Besides a
2.9% relative BLEU score gain, both current and response sentiment prediction
is improved by 1.5% and 3.4% relative UA, respectively. As for the context
history length, from Figure 3, we tried smaller context window sizes ($w=2,0$)
and observed that performance drops gradually across all prediction targets,
highlighting the importance of longer contexts.
Multi-modal context: Beyond text context, we can incorporate speech embeddings
for contextual turns to provide more acoustic clues for interpretation and
prediction. Comparing text (Table 1, row 11) with Text+Speech context (row
14), we observe further improvements of 1.4%, 0.5%, and 2.1% in relative terms
for current sentiment, response sentiment, and BLEU score, respectively.
Sentiment context: Lastly, using contextual sentiment labels also enhances the
overall performance of the proposed method. For the text modality, there are
relative gains of 1.4%, 1.7%, and 0.7% in current sentiment, response
sentiment, and response text, respectively. Overall, the best-performing
multi-modal serialized method (row 15) achieves a current sentiment accuracy
of 71.7%, response sentiment accuracy of 41.9%, and a response text BLEU score
of 14.9. This result implies that contextual speech embeddings and sentiment
information are complementary. We hypothesize that the rich information in
speech, such as speaker identity, helps to disambiguate and understand the
spoken dialogue effectively.

##  6 Conclusion

We introduced ParalinGPT, a modeling framework for spoken dialogues that
integrates paralinguistic and linguistic information in a single
autoregressive model. This framework enables the joint generation of response
text and current and response sentiment prediction. Through our experiments on
a corpus of human-human conversations, we show the effectiveness of the
proposed serialized multitasking method compared to separate classification or
language models. Additionally, we demonstrate that incorporating self-
supervised speech features and context history further improves performance.
There are several potential avenues for future research, including using the
latest and large-scale language models and adding more tasks under serialized
multi-tasking, such as turn taking prediction and emotion recognition. Another
important direction for future research involves the exploration of
paralinguistic speech embeddings at various time resolutions and pooling
methods.

## References

  * [1] OpenAI,  “Gpt-4 technical report,” 2023.
  * [2] Weixiang Zhao, Yanyan Zhao, Xin Lu, Shilong Wang, Yanpeng Tong, and Bing Qin,  “Is chatgpt equipped with emotional dialogue capabilities?,”  arXiv preprint arXiv:2304.09582, 2023.
  * [3] Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria,  “Towards multimodal sarcasm detection (an _obviously_ perfect paper),”  in ACL 2019, 2019, pp. 4619–4629.
  * [4] Chandrakant Bothe, Sven Magg, Cornelius Weber, and Stefan Wermter,  “Dialogue-based neural learning to estimate the sentiment of a next upcoming utterance,”  in Proc. of ICANN. Springer, 2017, pp. 477–485.
  * [5] Isabel Dias, Ricardo Rei, Patrícia Pereira, and Luisa Coheur,  “Towards a sentiment-aware conversational agent,”  in Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents, 2022, pp. 1–3.
  * [6] Zhenqiao Song et al.,  “Generating responses with a specific emotion in dialog,”  in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019, pp. 3685–3695, Association for Computational Linguistics.
  * [7] Mauajama Firdaus, Hardik Chauhan, Asif Ekbal, and Pushpak Bhattacharyya,  “Emosen: Generating sentiment and emotion controlled responses in a multimodal dialogue system,”  IEEE Transactions on Affective Computing, vol. 13, no. 3, pp. 1555–1566, 2020.
  * [8] Deeksha Varshney, Asif Ekbal, and Pushpak Bhattacharyya,  “Modelling context emotions using multi-task learning for emotion controlled dialog generation,”  in Proc. of EMNLP, 2021, pp. 2919–2931.
  * [9] Ruibo Liu et al.,  “Modulating language models with emotions,”  in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online, Aug. 2021, pp. 4332–4339, Association for Computational Linguistics.
  * [10] Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer,  “Dialogue act modeling for automatic tagging and recognition of conversational speech,”  Computational linguistics, vol. 26, no. 3, pp. 339–373, 2000.
  * [11] Nigel G. Ward, Alejandro Vega, and Timo Baumann,  “Prosodic and temporal features for language modeling for dialog,”  Speech Communication, vol. 54, no. 2, pp. 161–174, 2012.
  * [12] Xiang Li and Ming Zhang,  “Emotion analysis for the upcoming response in open-domain human-computer conversation,”  in Web and Big Data: APWeb-WAIM 2018 International Workshops: MWDA, BAH, KGMA, DMMOOC, DS, Macau, China, July 23–25, 2018, Revised Selected Papers 2. Springer, 2018, pp. 352–367.
  * [13] Sarik Ghazarian, Behnam Hedayatnia, Alexandros Papangelis, Yang Liu, and Dilek Hakkani-Tur,  “User response and sentiment prediction for automatic dialogue evaluation,”  arXiv preprint arXiv:2111.08808, 2021.
  * [14] Sarik Ghazarian, Behnam Hedayatnia, Alexandros Papangelis, Yang Liu, and Dilek Hakkani-Tur,  “What is wrong with you?: Leveraging user sentiment for automatic dialog evaluation,”  in Findings of ACL 2022, 2022, pp. 4194–4204.
  * [15] Mauajama Firdaus, Asif Ekbal, and Pushpak Bhattacharyya,  “Polise: Reinforcing politeness using user sentiment for customer care response generation,”  in Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 6165–6175.
  * [16] Bryan Li, Dimitrios Dimitriadis, and Andreas Stolcke,  “Acoustic and lexical sentiment analysis for customer service calls,”  in ICASSP 2019. IEEE, 2019, pp. 5876–5880.
  * [17] Sara Motamed, Saeed Setayeshi, Azam Rabiee, and Arash Sharifi,  “Speech emotion recognition based on fusion method,”  Journal of Information Systems and Telecommunication, vol. 5, pp. 50–56, 2017.
  * [18] Jingwei Yan, Wenming Zheng, Zhen Cui, Chuangao Tang, Tong Zhang, Yuan Zong, and Ning Sun,  “Multi-clue fusion for emotion recognition in the wild,”  in Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016, pp. 458–463.
  * [19] Soujanya Poria, Haiyun Peng, Amir Hussain, Newton Howard, and Erik Cambria,  “Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis,”  Neurocomputing, vol. 261, pp. 217–230, 2017.
  * [20] Guan-Ting Lin, Chi-Luen Feng, Wei-Ping Huang, Yuan Tseng, Tzu-Han Lin, Chen-An Li, Hung-yi Lee, and Nigel G Ward,  “On the utility of self-supervised models for prosody-related tasks,”  in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 1104–1111.
  * [21] Egor Lakomkin, Mohammad Ali Zamani, Cornelius Weber, Sven Magg, and Stefan Wermter,  “Incorporating end-to-end speech recognition models for sentiment analysis,”  in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 7976–7982.
  * [22] Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al.,  “On generative spoken language modeling from raw audio,”  Transactions of the Association for Computational Linguistics, vol. 9, pp. 1336–1354, 2021.
  * [23] Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, et al.,  “Text-free prosody-aware generative spoken language modeling,”  in ACL 2022, 2022, pp. 8666–8681.
  * [24] Tu Anh Nguyen et al.,  “Generative spoken dialogue language modeling,”  Transactions of the Association for Computational Linguistics, vol. 11, pp. 250–266, 2023.
  * [25] Zalán Borsos et al.,  “Audiolm: a language modeling approach to audio generation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
  * [26] Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al.,  “Audiopalm: A large language model that can speak and listen,”  arXiv preprint arXiv:2306.12925, 2023.
  * [27] Dong Zhang, Shimin Li, et al.,  “Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,”  arXiv preprint arXiv:2305.11000, 2023.
  * [28] Yizhe Zhang, Siqi Sun, et al.,  “Dialogpt: Large-scale generative pre-training for conversational response generation,”  in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020, pp. 270–278.
  * [29] Zhiyun Lu, Liangliang Cao, Yu Zhang, Chung-Cheng Chiu, and James Fan,  “Speech sentiment analysis via pre-trained features from end-to-end asr models,”  in ICASSP 2020. IEEE, 2020, pp. 7149–7153.
  * [30] Suwon Shon, Pablo Brusco, Jing Pan, Kyu J. Han, and Shinji Watanabe,  “Leveraging Pre-Trained Language Model for Speech Sentiment Analysis,”  in Proc. Interspeech 2021, 2021, pp. 3420–3424.
  * [31] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.
  * [32] Wei-Ning Hsu et al.,  “Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training,”  arXiv preprint arXiv:2104.01027, 2021.
  * [33] Yuanchao Li, Yumnah Mohamied, Peter Bell, and Catherine Lai,  “Exploration of a self-supervised speech model: A study on emotional corpora,”  in 2022 IEEE SLT. IEEE, 2023, pp. 868–875.
  * [34] John J Godfrey, Edward C Holliman, and Jane McDaniel,  “Switchboard: Telephone speech corpus for research and development,”  in Acoustics, speech, and signal processing, ieee international conference on. IEEE Computer Society, 1992, vol. 1, pp. 517–520.
  * [35] Eric Chen, Zhiyun Lu, Hao Xu, Liangliang Cao, Yu Zhang, and James Fan,  “A large scale speech sentiment corpus,”  in Proceedings of the Twelfth Language Resources and Evaluation Conference, 2020, pp. 6549–6555.


# Position: The Causal Revolution Needs Scientific Pragmatism

Joshua R. Loftus

###### Abstract

Causal models and methods have great promise, but their progress has been
stalled. Proposals using causality get squeezed between two opposing
worldviews. Scientific perfectionism–an insistence on only using “correct”
models–slows the adoption of causal methods in knowledge generating
applications. Pushing in the opposite direction, the academic discipline of
computer science prefers algorithms with no or few assumptions, and
technologies based on automation and scalability are often selected for
economic and business applications. We argue that these system-centric
inductive biases should be replaced with a human-centric philosophy we refer
to as scientific pragmatism. The machine learning community must strike the
right balance to make space for the causal revolution to prosper.

Causality, causal inference, scientific method, pragmatism, value pluralism

##  1 Introduction

There is a causal revolution underway in empirical sciences. From its century-
old roots in controlled experiments, in recent decades it has made inroads in
observational health and social sciences. As a research topic among
methodologists, causal models have gained some popularity in economics,
computer science, and statistics, and may be spreading into applications in
other research fields and industries. However, there is a risk that the causal
revolution will stall, and that humanity will miss out on the potential
benefits of its success. To realize these benefits, we argue that a philosophy
of scientific pragmatism is necessary for the causal revolution to flourish.
This philosophy views causal models pragmatically as tools for hypothetical
reasoning.

For the sake of simplicity we discuss only two types of models: causal and
predictive. Strictly speaking, causal models are a subset of predictive
models, so here we will use “predictive” as shorthand for “predictive models
which are not also causal.” If we think of modeling as a process of making
assumptions and drawing conclusions then _causal models make more assumptions
than predictive models_. These additional assumptions afford causal models
additional conclusions, but also open them up to more criticism.

In the scenario of Figure 1, if we only judge a model based on its predictions
of some important variable $Y$ then the same model may work equally well
regardless of the direction of causality. If we expect a model to also tell us
what happens when we intervene on variables, then these models with equal
predictive accuracy are no longer equivalent. _We become responsible to choose
one of them_.

$X$$Y$$X$$Y$$X$$Z$$Y$ Figure 1: Structural causal models (SCMs) represented as
directed acyclic graphs (DAGs). Variables are the nodes of the graph and
causal effects are represented by arrows. An intervention is an operation that
modifies the graph in some way. Intervening on a variable means we erase the
arrows pointing into that variable, set the value of that variable
arbitrarily, and then propagate the new value along directed pathways pointing
out of that variable. On the left, $X$ is a cause of $Y$, so intervening on
$X$ will result in a change in $Y$. In the other cases, intervening on $X$
results in no change in $Y$. It is possible that the accuracy of some function
$f(X)$ in predicting $Y$ is equal in all cases.

We can use causal models to answer important questions that predictive models
cannot illuminate. For example, if we want to know how we might change $Y$,
then the model on the left of Figure 1 says we can intervene on $X$, the model
on the right says we can intervene on $Z$ (and that $X$ would also change in
that case), and the model in the middle says we would have to intervene on $Y$
directly (which changes $X$ as well).

The differences between these cases have important consequences when we apply
models like these to reason about the real world. Suppose that $Y$ is some
variable we would like to change, but due to real world constraints we can
only intervene on $X$. We have data and sophisticated predictive models that
predict $Y$ from $X$ with state of the art (SOTA) accuracy. Does this mean we
should then spend resources on an intervention that changes $X$? The
predictive model alone cannot answer this, we have to choose a causal model.
If we choose the model on the left of Figure 1, but the real world is better
described by the one on the right, then we may end up wasting resources,
harming people, or pursuing a research program that will end in some type of
failure.

Causal models have more assumptions and more implications. They force us to
make more choices. As a result, there are more ways our choices can go wrong,
but also more potential applications. To access the benefits of a causal
revolution we have to be willing to pay the costs. We must learn to strike the
right balance when judging proposals that involve causal modeling.

###  1.1 Published Context

Before elaborating our positions, we provide some context about the status of
the causal revolution in empirical sciences. We would like to know if
scientists care about causality and whether they are modeling and writing
about it in a justified way. Fortunately for our purposes, meta-scientists
have been hard at work studying these questions. Some indicative findings:

  * •

In observational studies from 4 leading journals of obesity and nutrition (in
2006), 31% of papers had inappropriate language implying causality in the
titles or abstracts (Cofield et al., 2010).

  * •

Among all observational studies in the top 4 medical journals in 2010, authors
recommend a medical practice in 56% (and only 14% of these recommended an RCT
should be done to confirm their observational results) (Prasad et al., 2013).

  * •

An NLP model applied to “about 38,000 observational studies in PubMed” showed
about 32% used direct causal language, and most of these instances occurred
only in the conclusion sections (i.e. not in abstracts or titles) (Yu et al.,
2019).

  * •

A study found “spin” in 84% of abstracts “of all non-randomized studies
assessing an intervention published in the BioMed Central Medical Series
journals between January 1, 2011 and December 31, 2013” and the most common
kind of spin was causal language, occurring in 53% of abstracts (Lazarus et
al., 2015).

  * •

A study of non-RCTs in “1,170 articles from 18 high-profile medical/public
health/epidemiology journals (65 per journal) published from 2010–2019” found
“few studies explicitly declared an interest in estimating causal effects, the
majority used language that moderately or strongly implied causality” and
“action recommendations were identified in 60.3% […] of discussion sections,
about twice that in abstracts” (Haber et al., 2022).

  * •

In a study of the 50 health studies that were most shared on social media in
2015, “34% percent of academic studies and 48% of media articles used language
that [human expert] reviewers considered too strong for their strength of
causal inference” (Haber et al., 2018).

  * •

A systematic review of 199 systematic reviews of observational studies
(published in 2019) found most (57%) were explicitly addressing causal issues,
and among those which were not explicit about causal intent roughly half (51%)
used causal language (Han et al., 2022).

The available research on “spin” and unjustified causal language is
concentrated in health/medical sciences, but we believe the problem is much
more widespread than directly indicated by the literature above. The health
sciences have established traditions of systematic reviews, meta-analysis, and
(pre-)registered trials. There are reasons to think the problem would only be
worse in other fields where these accountability mechanisms have yet to catch
on. For example, other fields have been shown to suffer from irreproducibility
in their published literature (Open Science Collaboration, 2015), and failure
to properly model causality may be one of the reasons that replication studies
fail.

##  2 Positions

In this section we sometimes refer to the concept of an equivalent predictive
model. Given any causal model, we can simply “forget” the additional causal
assumptions regarding interventions and directionality and obtain an
equivalent predictive model.

Positions 1-4 should prove relatively uncontroversial since they have stronger
logical and/or evidential support. We still find it valuable to state and
argue for them because, even though they may be uncontroversial, they may
still be under-appreciated. Positions 5-6 are more speculative. We do not have
evidence to cite for them. The reader will have to judge them based on their
own experiences.

######  Position 1 (Proving too much).

For any argument against using a causality in general, or a specific causal
method in particular, we should check that this argument does not also apply
to an equivalent predictive model.

This is a simple logical point but may too often be forgotten.

There are many ways that probabilistic or mathematical models in general can
be criticized. Sometimes people rediscover an old, general criticism and apply
it to the newest wave of research. This can be a good thing. It is worth
noticing when a new modeling approach is not immune to a long-standing general
problem. But we should also take care to not penalize any specific class of
models based on a general critique.

As an example, consider using a causal model to analyze fairness and
discrimination, and suppose we have a variable in this model that represents a
social category like race or gender. It is reasonable to use the concept of
construct validity (Jacobs & Wallach, 2021) to criticize any model including
such a variable. So it would be selective if that critique was only applied to
causal models. On the other hand, it _is_ reasonable to criticize causal
models specifically with an argument that focuses on the experimental non-
manipulability of variables like race and gender (Sen & Wasow, 2016; Kohler-
Hausmann, 2018; Hu, 2023; Hu & Kohler-Hausmann, 2020). And in response to good
criticisms, it may be possible to adapt and improve causal models for those
purposes (Bynum et al., 2021, 2023, 2024).

######  Position 2 (People want causality).

People usually want to reach causal conclusions. Even when they use predictive
models, they often take actions or discuss the results as if the model were
causal.

We saw evidence for this position already in the literature cited in Section
1.1. Pushing beyond the evidence, we believe this problem is more prevalent
and impactful than indicated by peer reviewed scientific work. This is because
many scientists are aware that “association (e.g. correlation) is not
causation,” and hence may avoid using causal language when describing their
predictive models. However, they may still be thinking about their work
causally and taking actions as if the conclusions are causal. For example,
they may choose to study a question that is only interesting or useful if the
conclusions are causal, but they may do so using only predictive models.

######  Position 3 (Balanced standards).

Predictive models are often used in place of causal models because they are
held to too low a standard. Conversely, causal models–and methods or papers
that use them–are often held to a higher standard. This is usually wrong.

An author choosing to use a causal model makes additional explicit assumptions
relative to an equivalent predictive model. At the stage of peer review, a
referee may criticize any of these additional assumptions. And since causal
models entail additional conclusions or implications, if the paper is
published it is possible some subsequent research will falsify one of those
additional conclusions, thereby calling the paper into question. Not wanting
to risk it, scientists may avoid these costs of causality and opt for a model
with fewer assumptions.

The principle of parsimony (or simplicity) states “ _other things being equal_
, simpler theories are better [emphasis added]” (Baker, 2022). Now, it is true
that a causal model has more assumptions than an equivalent predictive model,
so in a sense the predictive model is simpler. But it is _not_ true that these
two models are otherwise equal. The causal version implies more.

There are some good reasons to apply different standards to causal and
predictive models. If an assumption of a causal model is difficult or
impossible to test via any experiment, we should treat that assumption with
caution and transparently acknowledge the limitation of any conclusions based
on that model. All further conclusions or models built on that assumption must
never forget the untested part of their foundation. Similarly, when reporting
results of a predictive model we must acknowledge they do not necessarily
imply any causal relationships.

Achieving a level playing field between causal and predictive models requires
applying balanced standards. If a research project proposes using predictive
models it should be questioned whether the desired goals of the inquiry only
include predictive conclusions. If the work does not pay the price of making
explicit assumptions required for causal conclusions it should not be allowed
to suggest those conclusions for free. On the other hand, if causal
conclusions are desired, then it is wrong to penalize the additional
assumptions necessary for the model to able to answer our actual questions.

######  Position 4 (Falsifiability and the scientific method).

Causal models can be falsified in more ways than predictive models. This is
usually good.

Falsificationism, one of the most well-known philosophies about the scientific
method, states that science proceeds not by proving theories but rather by
falsifying them (Thornton, 2023). Suppose someone does an experiment in a
scenario described by Figure 1 that manipulates $X$ and finds no significant
change in $Y$. This falsifies the causal model $X\to Y$, but does not falsify
the predictive model. If the status quo in this field is to use predictive
models, then this experiment may not result in any scientific progress. Other
studies will continue to support the predictive model’s accuracy, and the
field can remain conflicted.

We have already discussed how causality involves more assumptions and
conclusions. Any of these that can be tested provide another target for
potential falsification. As a result, the scientist in our publishing story
above decided not to use a causal model. Irrespective of their individual
success in publishing, is this a good thing for science in general?

It is a misconception about science that the goal of designing a model or
theory is to have the closest possible fit to reality in general, or to be
immune to any criticism or disproof. A scientific theory can have great
explanatory power and application potential even if it is known to be
incorrect. For example, suppose someone builds an algorithm for autonomous
vehicle control that uses classical mechanics. It would not be interesting or
useful to point out that classical mechanics is “wrong” _unless_ , say, that
algorithm will be used for rockets traveling at relativistic speeds.

Falsification does not need to be a blindly adversarial process reaching for
any conclusion that can be disproved. It can be done selectively, with
attention given to those parts of a model that are consequential in the
intended application. As an example consider causal analysis of
discrimination. The interested parties may only be disputing the strength of
causal determination along one pathway in a causal graph, and are not
interested in testing every other falsifiable prediction of the model.

Assumptions can enable or block scientific progress. They can block progress
if they are never questioned, which seems unlikely given the incentives to
publish novel results. They cannot enable progress if they are never
considered. And that may be a real risk if the scientific and machine learning
communities do not learn to tolerate more assumptions in some applications.

######  Position 5 (Outsourcing responsibility).

A preference for predictive models is often a sign of avoiding responsibility.

If a work’s value derives mainly from possible causal conclusions, but the
author does not make transparent causal assumptions, they are leaving it to
readers to reach the intended conclusion.

Researchers sometimes avoid making transparent causal assumptions out of fear.
They do not want to be responsible for more modeling choices because each
choice opens up another potential criticism. They prefer whichever methods
seem to work with the least number of choices, automatically, on the default
settings. And this may not be all wrong, in fact, except that the same
researchers often want to interpret the results causally. If they have taken a
statistics course they may know to avoid using the word “cause” when talking
about the results, but they might get away with suggesting or hinting at
causality with weasel words like $X$ “drives,” “predicts,” or “is linked to”
$Y$. The responsibility for making causal conclusions falls to their audience,
who, due to their distance from the work, may be less familiar with the
limitations.

Using causal models can be a more honest and transparent way to answer the
questions we truly want to ask. Instead of pretending that we are only
interested in prediction, we can admit that we want to change the world. We
are almost always going to use predictions for some purpose that involves
taking some actions to try to change some outcome. So we can and should use a
causal model to elaborate how we think this is going to work.

######  Position 6 (Relevance to machine learning).

The machine learning community is uniquely well-positioned to integrate
causality, but also has a disciplinary bias against it.

The overlap between machine learning and causality is growing in absolute
terms (see Makhlouf et al. (2020); Yao et al. (2021); Schölkopf (2022);
Kaddour et al. (2022) for some recent surveys). This is natural since
applications with causal models usually involve learning functions from data.
Machine learning has developed many powerful algorithms for estimating
functions and learning about structure, and it is often only some additional
assumptions required to transform any of these into explicitly causal methods.

However, there are reasons to be concerned about the prospects of causality in
machine learning.

First, computer science as a discipline prefers general purpose algorithms,
i.e. ones which are “assumption free” or “model free.” This makes sense for
building robust systems, since an algorithm that relies on specific
assumptions can break down if conditions change so that the assumptions no
longer hold. There is also an economic logic: a technological service that is
more automatic and scalable has more potential customers. Algorithms using
causal models are less automatic and scalable because they require more
context or task-specific input. Decisions about causal directions might
require domain knowledge, for example.

Second, for many years, the “secret sauce” of machine learning’s success
(Donoho, 2017) has been the Common Task Framework: a competition between
algorithms on a fixed, well-defined task (Liberman, 2010). These tasks are
usually about predictive accuracy or a set of predictive benchmarks. And,
usually, the SOTA algorithms that gain the most success and attention in the
field are highly complex predictive models where “the goal is not
interpretability” (Breiman, 2001). It makes sense from the viewpoint of
constrained optimization that if we evaluate models only on predictive
accuracy then any constraints–for example, belonging to a class of certain
interpretable causal models–will likely result in worse evaluations.

On the plus side, pragmatism in computer science does sometimes make room for
works that pose or reformulate problems differently. This may allow some
exceptional works to use causal modeling without assuming the goal of learning
a “correct” model (Janzing et al., 2023).

#### Speculation.

Allow us this brief departure from more evidenced argument. We speculate that
the high degree of competition in machine learning conferences–with their low
acceptance rates and focus on SOTA performance in predictive tasks–exacerbates
some of the challenges for causal methods we have outlined. In particular,
when referees enter a review process that they know has a low acceptance rate,
they may be looking for anything they can criticize, and the additional
assumptions of a causal model are a prime target.

##  3 Principles

###  3.1 Scientific Pragmatism

Pragmatism is a philosophical tradition that emphasizes usefulness (Legg &
Hookway, 2021). Continuing with this tradition, we advocate thinking of
scientific theories–and models in general–as tools. Instead of asking if a
model is right, correct, or accurate, we should ask if it is _useful for a
certain purpose_. Thus, the fitness of a theory or model depends on the goals
of the people using it.

Recall our previous example of the autonomous vehicle control algorithm. If
this system will only be used for automobiles on the surface of our planet,
then classical mechanics is a good fit. Quantum mechanics has not been
falsified under any conditions, but it would be a bad fit for the design of
such an algorithm.

Causal models can be useful as tools for a variety of purposes. They can serve
as a notation and language for reasoning and conveniently communicating about
relationships between sets of variables (Imbens, 2020) and, hence, for
interpretability and explainability (Blöbaum & Shimizu, 2017; Sani et al.,
2020; Frye et al., 2020; Heskes et al., 2020; Zhao & Hastie, 2021; Loftus et
al., 2023). Causality can be used to analyze algorithms for discrimination or
fairness (Kilbertus et al., 2017; Kusner et al., 2017; Nabi & Shpitser, 2018;
Zhang & Bareinboim, 2018; Chiappa, 2019; Yang et al., 2021).

Machine learning researchers are likely already comfortable with the idea of
selecting the right tool for a task. It is important that they remember there
are tasks other than prediction with SOTA accuracy. And in some application
domains, the principle of scientific pragmatism supports our Positions 3 and 4
as an important counterweight to scientific perfectionism.

The greatest challenge for a scientific pragmatist is the danger of “cargo
cult science” (Feynman, 1998) or its relative “cargo cult programming”
(Raymond & Steele, 1997). These terms describe failures that occur when people
select tools or attempt to build them without understanding how they work,
based only on appearances of success. In the classic example of a cargo cult
as described by Feynman, people attempt to build an aircraft, for example, by
fashioning objects with similar appearances. The problem is that such an
aircraft, built without the knowledge and tools of engineering, cannot
function. While the problem is obvious in this example, it can be more subtle
in cargo cult science where a model or method may appear to function. The
problem of cargo cult machine learning may be worse still because machine
learning tasks are highly standardized and its tools automated. Students of
machine learning can rapidly achieve the appearance of success without
understanding the tools they are using, knowing about their limitations, or
developing skills to judge or build new tools.

Pragmatism can serve as a descriptive theory about practices that are already
standard. We simply advocate that this is done consciously and honestly,
acknowledging our purposes. When we select tools for predictive accuracy we
should not expect this to necessarily increase scientific knowledge if, for
example, causal interpretability is sacrificed in the process.

###  3.2 Value Pluralism

When selecting tools and judging their usefulness, we may not be able to
reduce this judgment to the application of a single value (Mason, 2023). We
have a plurality of values that are important to us, and the relevance of
these values may change depending on our purposes. The challenge increases
when our values are in conflict. Readers may be thinking that multi-objective
optimization or multi-task learning can simply “solve” this problem for us,
but different values may be incommensurable or not quantifiable (Schroeder,
2021). For example, we may desire both predictive accuracy and
interpretability, and we may not have any quantified measures of
interpretability.

Value pluralism is in strong conflict with machine learning’s “secret sauce”
as described in Position 5. The field has become more pluralistic by widening
the set of standard tasks, and this is good work that should continue.
However, there may still be limits on which values can be included for tasks
that are standardized, or in systems built using automation and scaling
(Nguyen, forthcoming; Finnveden et al., 2022). Some values may require
something closer to case-by-case human judgments. For example, the usefulness
of a particular causal model may not be fully determined by the available data
and algorithms, but also depend on a subjective judgment involving its
interpretation.

###  3.3 Humanism or Anthropocentrism

When we build infrastructure and systems using automation, selecting tools for
their scalability and plug-and-play fitness within the existing system, it is
not just causal models that are at risk of being pushed out of the picture.
Another risk is evident in the terms “human-in-the-loop” or “human feedback,”
the very existence of these terms implying that there are alternatives which
somehow do not involve humans.

We assert that humans are the ultimate source and arbiters of values.111We are
not opposed to a view that includes other forms of biological life, but we
leave that for another discussion. By bringing the focus to humans using
tools, pragmatism could shed light on various perennially confusing issues in
machine learning. It can help us remember that the “agents” are humans, and
that software processes started by humans are tools being used for a purpose.

In this human-centric view, it does not matter as much that causal methods may
be less automatic or scalable. It does not matter as much when a particular
algorithm is the SOTA for a certain task. We always have other purposes and
values which are not codified in that task, and may not even be readily
quantifiable. For example, we may sometimes wish to use a causal model within
the context of a broad scientific question or business plan that involves
various subjective factors. We may be motivated by those factors to make
assumptions in that causal model which are not decidable in an automated way
based on the available data. We can remember that this introduces limitations,
and we can decide that the choice still fits with our values.

##  4 Conclusion

###  4.1 Related Ideas

Inspired by some earlier debates in statistics, Tukey (1960) proposed we
distinguish between two different goals or modes of statistical analysis:
conclusions vs decisions. Conclusions (the Fisherian mode) should be final,
authoritative, well-tested, and we should rarely need to revise them, if ever.
As such, it makes sense to have high standards of evidence for conclusions. On
the other hand, decisions (the Neyman-Pearson mode) are temporary, we often
make them just so we can proceed to act, and we are not surprised if we need
to try something else. We speculate that our Positions 2 and 3 are often
related to a failure to make the distinction Tukey recommends here. In
particular, we can make the decision to use a causal model without taking that
model as conclusive.

Echoing through history, the pragmatic statistician George Box reminds us:
“All models are wrong, but some are useful,” and “Since all models are wrong
the scientist must be alert to what is _importantly wrong_ [emphasis added].
It is inappropriate to be concerned about safety from mice when there are
tigers abroad” (Box, 1976).

Preceding the development of structural causal models with their formal
definition of intervention, philosopher of science Hacking (1983) makes a case
that the practice of science involves intervention rather than representation.
This makes scientists pragmatic by necessity.

In the early days of machine learning, Breiman (2001) argued in favor of its
single-minded focus on predictive accuracy, and, more recently, Rodu &
Baiocchi (2023) described conditions when such “outcome reasoning” may succeed
or may be inappropriate.

Much recent work has focused on the problem of distribution shift or the
ability of predictive models to generalize out-of-distribution. Schölkopf et
al. (2021) showed causal models can be a natural fit for this task.
Interestingly, Richens & Everitt (2024) established something like a converse
of this result: if a predictive model can succeed under sufficiently many
distribution shifts then that model entails an approximate causal model of the
data generating process. This implies that if someone cares about maintaining
predictive accuracy across a variety of different settings then, in a sense,
they must care about causality.

###  4.2 Significance

Should anyone who does not personally have any uses for causal models care
about challenges holding back the causal revolution? Yes, if they want
humanity to continue unlocking the benefits of scientific progress, or if they
want human values that are not in harmony with automated systems to be able to
flourish.

A pragmatic use of causal models could revolutionize science. It has the
potential to become a new standard language for the communication of
scientific results, replacing the less interesting significance tests for
differences between groups or for regression coefficients. For the sake of
simplicity let’s take the view that a scientific discovery occurs when someone
falsifies a hypothesis. If that hypothesis is uninteresting, then it is not
much of a discovery. Causal models are an ideal framework for specifying and
communicating _interesting_ hypotheses.

It is an empirical fact that humans have many different values. Systems using
machine learning apparently grow most rapidly when they are built based on
automation, scalability, and a narrow focus on standardized tasks. Such
systems risk crowding out values which are not included in the set of tasks,
or which are at odds with automation and scale.

###  4.3 Summary

We have argued that methods based on causality face some challenges which can
put them at a disadvantage. Within machine learning specifically, this occurs
because of a narrow focus on SOTA performance on prediction tasks. Most of
those tasks do not encode subjective human uses like causal interpretation,
for example. We advocate a philosophy combining scientific pragmatism, value
pluralism, and humanism, where models and tools can be selected based on a
more inclusive set of values which may not be easily quantifiable or
standardized. With these views, the machine learning community can make room
for a greater variety of interesting applications and methods, including more
that focus on causality, and promote the flourishing of more values aside from
predictive accuracy. The causal revolution needs scientific pragmatism in
order to proceed beyond its first steps, and human-centric value pluralism can
help us apply scientific pragmatism wisely.

## Acknowledgements

The author would like to thank Pooja Loftus, Ricardo Silva, and David Watson
for helpful conversations and feedback on this work.

## Impact Statement

This paper presents positions intended to advance the field of Machine
Learning. There are many potential societal consequences, some of which were
discussed throughout the text. We do not feel we should specifically highlight
any other potential impacts here.

## References

  * Baker (2022) Baker, A.  Simplicity.  In Zalta, E. N. (ed.), _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, Summer 2022 edition, 2022.
  * Blöbaum & Shimizu (2017) Blöbaum, P. and Shimizu, S.  Estimation of interventional effects of features on prediction.  In _2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP)_ , pp. 1–6. IEEE, 2017.
  * Box (1976) Box, G. E.  Science and statistics.  _Journal of the American Statistical Association_ , 71(356):791–799, 1976.
  * Breiman (2001) Breiman, L.  Statistical modeling: The two cultures (with comments and a rejoinder by the author).  _Statistical science_ , 16(3):199–231, 2001.
  * Bynum et al. (2021) Bynum, L., Loftus, J., and Stoyanovich, J.  Disaggregated Interventions to Reduce Inequality.  In _Equity and Access in Algorithms, Mechanisms, and Optimization_ , pp. 1–13. Association for Computing Machinery, New York, NY, USA, October 2021.  ISBN 978-1-4503-8553-4.  URL https://doi.org/10.1145/3465416.3483286.
  * Bynum et al. (2023) Bynum, L., Loftus, J., and Stoyanovich, J.  Counterfactuals for the future.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2023.
  * Bynum et al. (2024) Bynum, L. E., Loftus, J. R., and Stoyanovich, J.  A new paradigm for counterfactual reasoning in fairness and recourse.  In _Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI)_ , 2024.
  * Chiappa (2019) Chiappa, S.  Path-specific counterfactual fairness.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 7801–7808, 2019.
  * Cofield et al. (2010) Cofield, S. S., Corona, R. V., and Allison, D. B.  Use of causal language in observational studies of obesity and nutrition.  _Obesity facts_ , 3(6):353–356, 2010.
  * Donoho (2017) Donoho, D.  50 years of data science.  _Journal of Computational and Graphical Statistics_ , 26(4):745–766, 2017.
  * Feynman (1998) Feynman, R. P.  Cargo cult science.  In _The art and science of analog circuit design_ , pp. 55–61. Elsevier, 1998.
  * Finnveden et al. (2022) Finnveden, L., Riedel, J., and Shulman, C.  AGI and Lock-In.  https://forum.effectivealtruism.org/posts/KqCybin8rtfP3qztq/agi-and-lock-in, 2022.
  * Frye et al. (2020) Frye, C., Rowat, C., and Feige, I.  Asymmetric shapley values: incorporating causal knowledge into model-agnostic explainability.  _Advances in Neural Information Processing Systems_ , 33:1229–1239, 2020.
  * Haber et al. (2018) Haber, N., Smith, E. R., Moscoe, E., Andrews, K., Audy, R., Bell, W., Brennan, A. T., Breskin, A., Kane, J. C., Karra, M., et al.  Causal language and strength of inference in academic and media articles shared in social media (claims): A systematic review.  _PloS one_ , 13(5):e0196346, 2018.
  * Haber et al. (2022) Haber, N. A., Wieten, S. E., Rohrer, J. M., Arah, O. A., Tennant, P. W., Stuart, E. A., Murray, E. J., Pilleron, S., Lam, S. T., Riederer, E., et al.  Causal and associational language in observational health research: a systematic evaluation.  _American journal of epidemiology_ , 191(12):2084–2097, 2022.
  * Hacking (1983) Hacking, I.  _Representing and intervening: Introductory topics in the philosophy of natural science_.  Cambridge university press, 1983.
  * Han et al. (2022) Han, M. A., Leung, G., Storman, D., Xiao, Y., Srivastava, A., Talukdar, J. R., El Dib, R., Morassut, R. E., Zeraatkar, D., Johnston, B. C., et al.  Causal language use in systematic reviews of observational studies is often inconsistent with intent: a systematic survey.  _Journal of Clinical Epidemiology_ , 148:65–73, 2022.
  * Heskes et al. (2020) Heskes, T., Sijben, E., Bucur, I. G., and Claassen, T.  Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models.  _Advances in neural information processing systems_ , 33:4778–4789, 2020.
  * Hu (2023) Hu, L.  What is “race” in algorithmic discrimination on the basis of race?  _Journal of Moral Philosophy_ , 1(aop):1–26, 2023.
  * Hu & Kohler-Hausmann (2020) Hu, L. and Kohler-Hausmann, I.  What’s sex got to do with machine learning?  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pp. 513–513, 2020.
  * Imbens (2020) Imbens, G. W.  Potential outcome and directed acyclic graph approaches to causality: Relevance for empirical practice in economics.  _Journal of Economic Literature_ , 58(4):1129–1179, 2020.
  * Jacobs & Wallach (2021) Jacobs, A. Z. and Wallach, H.  Measurement and fairness.  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , pp. 375–385, 2021.
  * Janzing et al. (2023) Janzing, D., Faller, P. M., and Vankadara, L. C.  Reinterpreting causal discovery as the task of predicting unobserved joint statistics.  _arXiv preprint arXiv:2305.06894_ , 2023.
  * Kaddour et al. (2022) Kaddour, J., Lynch, A., Liu, Q., Kusner, M. J., and Silva, R.  Causal machine learning: A survey and open problems.  _arXiv preprint arXiv:2206.15475_ , 2022.
  * Kilbertus et al. (2017) Kilbertus, N., Rojas Carulla, M., Parascandolo, G., Hardt, M., Janzing, D., and Schölkopf, B.  Avoiding discrimination through causal reasoning.  _Advances in neural information processing systems_ , 30, 2017.
  * Kohler-Hausmann (2018) Kohler-Hausmann, I.  Eddie Murphy and the dangers of counterfactual causal thinking about detecting racial discrimination.  _Nw. UL Rev._ , 113:1163, 2018.
  * Kusner et al. (2017) Kusner, M. J., Loftus, J., Russell, C., and Silva, R.  Counterfactual Fairness.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html.
  * Langley (2000) Langley, P.  Crafting papers on machine learning.  In Langley, P. (ed.), _Proceedings of the 17th International Conference on Machine Learning (ICML 2000)_ , pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann.
  * Lazarus et al. (2015) Lazarus, C., Haneef, R., Ravaud, P., and Boutron, I.  Classification and prevalence of spin in abstracts of non-randomized studies evaluating an intervention.  _BMC medical research methodology_ , 15:1–8, 2015.
  * Legg & Hookway (2021) Legg, C. and Hookway, C.  Pragmatism.  In Zalta, E. N. (ed.), _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, Summer 2021 edition, 2021.
  * Liberman (2010) Liberman, M.  Obituary: Fred jelinek.  _Computational Linguistics_ , 36(4):595–599, 2010.
  * Loftus et al. (2023) Loftus, J. R., Bynum, L. E., and Hansen, S.  Causal dependence plots.  _arXiv preprint arXiv:2303.04209_ , 2023.
  * Makhlouf et al. (2020) Makhlouf, K., Zhioua, S., and Palamidessi, C.  Survey on causal-based machine learning fairness notions.  _arXiv preprint arXiv:2010.09553_ , 2020.
  * Mason (2023) Mason, E.  Value Pluralism.  In Zalta, E. N. and Nodelman, U. (eds.), _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, Summer 2023 edition, 2023.
  * Nabi & Shpitser (2018) Nabi, R. and Shpitser, I.  Fair inference on outcomes.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Nguyen (forthcoming) Nguyen, C. T.  Value capture.  _Journal of Ethics and Social Philosophy_ , forthcoming.
  * Open Science Collaboration (2015) Open Science Collaboration.  Estimating the reproducibility of psychological science.  _Science_ , 349(6251):aac4716, 2015.
  * Prasad et al. (2013) Prasad, V., Jorgenson, J., Ioannidis, J. P., and Cifu, A.  Observational studies often make clinical practice recommendations: an empirical evaluation of authors’ attitudes.  _Journal of clinical epidemiology_ , 66(4):361–366, 2013.
  * Raymond & Steele (1997) Raymond, E. S. and Steele, G. L.  _The Jargon File, Version 4.0. 0, 24 Jul 1996_.  Project Gutenberg, 1997.
  * Richens & Everitt (2024) Richens, J. and Everitt, T.  Robust agents learn causal world models.  _International Conference on Learning Representations_ , 2024.
  * Rodu & Baiocchi (2023) Rodu, J. and Baiocchi, M.  When black box algorithms are (not) appropriate.  _Observational Studies_ , 9(2):79–101, 2023.
  * Sani et al. (2020) Sani, N., Malinsky, D., and Shpitser, I.  Explaining the behavior of black-box prediction algorithms with causal learning.  _arXiv preprint arXiv:2006.02482_ , 2020.
  * Schölkopf (2022) Schölkopf, B.  Causality for machine learning.  In _Probabilistic and Causal Inference: The Works of Judea Pearl_ , pp. 765–804. 2022.
  * Schölkopf et al. (2021) Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y.  Toward causal representation learning.  _Proceedings of the IEEE_ , 109(5):612–634, 2021.
  * Schroeder (2021) Schroeder, M.  Value Theory.  In Zalta, E. N. (ed.), _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, Fall 2021 edition, 2021.
  * Sen & Wasow (2016) Sen, M. and Wasow, O.  Race as a bundle of sticks: Designs that estimate effects of seemingly immutable characteristics.  _Annual Review of Political Science_ , 19:499–522, 2016.
  * Thornton (2023) Thornton, S.  Karl Popper.  In Zalta, E. N. and Nodelman, U. (eds.), _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, Winter 2023 edition, 2023.
  * Tukey (1960) Tukey, J. W.  Conclusions vs decisions.  _Technometrics_ , 2(4):423–433, 1960.
  * Yang et al. (2021) Yang, K., Loftus, J. R., and Stoyanovich, J.  Causal Intersectionality and Fair Ranking.  In Ligett, K. and Gupta, S. (eds.), _2nd Symposium on Foundations of Responsible Computing (FORC 2021)_ , volume 192 of _Leibniz International Proceedings in Informatics (LIPIcs)_ , pp. 7:1–7:20, Dagstuhl, Germany, 2021. Schloss Dagstuhl – Leibniz-Zentrum für Informatik.  ISBN 978-3-95977-187-0.  doi: 10.4230/LIPIcs.FORC.2021.7.  URL https://drops.dagstuhl.de/opus/volltexte/2021/13875.  ISSN: 1868-8969.
  * Yao et al. (2021) Yao, L., Chu, Z., Li, S., Li, Y., Gao, J., and Zhang, A.  A survey on causal inference.  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ , 15(5):1–46, 2021.
  * Yu et al. (2019) Yu, B., Li, Y., and Wang, J.  Detecting causal language use in science findings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 4664–4674, 2019.
  * Zhang & Bareinboim (2018) Zhang, J. and Bareinboim, E.  Fairness in decision-making—the causal explanation formula.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Zhao & Hastie (2021) Zhao, Q. and Hastie, T.  Causal interpretations of black-box models.  _Journal of Business & Economic Statistics_, 39(1):272–281, 2021.  doi: 10.1080/07350015.2019.1624293.  URL https://doi.org/10.1080/07350015.2019.1624293.