# Large-scale Language Model Rescoring on Long-form Data

###### Abstract

In this work, we study the impact of Large-scale Language Models (LLM) on
Automated Speech Recognition (ASR) of YouTube videos, which we use as a source
for long-form ASR. We demonstrate up to 8% relative reduction in Word Error
Eate (WER) on US English (en-us) and code-switched Indian English (en-in)
long-form ASR test sets and a reduction of up to 30% relative on Salient Term
Error Rate (STER) over a strong first-pass baseline that uses a maximum-
entropy based language model. Improved lattice processing that results in a
lattice with a proper (non-tree) digraph topology and carrying context from
the 1-best hypothesis of the previous segment(s) results in significant wins
in rescoring with LLMs. We also find that the gains in performance from the
combination of LLMs trained on vast quantities of available data (such as C4
[1]) and conventional neural LMs is additive and significantly outperforms a
strong first-pass baseline with a maximum entropy LM.

Index Terms: Large-scale language models, N-best rescoring, Fine-tuning

##  1 Introduction

Large-scale language models (LLM), such as as BERT [2], T5 [3], GPT-3 [4], and
PaLM [5], have proven to be successful in natural language processing (NLP)
tasks such as, Question Answering, Text Summarization, and other Zero Shot
learning applications. These models are trained on vast amounts of text data
and have yielded state-of-the-art results across several NLP and search tasks.
However, there is very limited work on the use of these LLMs in Automated
Speech Recognition (ASR).

Recent research has focused on fine-tuning GPT, GPT-2 and BERT models with
small amounts of in-domain data showing that they tend to outperform the
performance of conventional Neural LMs such as transformer LMs trained on the
same data [6]. The authors in [7] propose the use of pseudo-likelihood scores
and show that rescoring N-best hypotheses from an ASR model can yield
significant wins on Librispeech but there is always a trade-off between in-
domain modeling and fine-tuning a model trained with far more text. An
alternate approach to directly predict the oracle hypothesis was originally
proposed in [8] and used in [9] to re-rank the N-best hypothesis using scores
from BERT.

In this paper, we scale the use of LLMs to ASR on YouTube videos, which we use
as a source for long-form ASR. We show the importance of lattice quality and
contextual augmentation for long-form ASR and compare the performance of LLMs
with other neural and maximum entropy based LMs using two metrics: Word Error
Rate (WER) and Salient Term Error Rate (STER).

##  2 Related Work

Several methods to incorporate LMs in end-to-end sequence models have been
proposed in the literature. Decoding algorithms [10, 11, 12] employ fusion
strategies, such as shallow [13], cold [14], deep [15] and component [16]
fusion. However, the wins from incorporating LMs in this fashion have been
relatively small for large scale ASR [17]. The Hybrid Autoregressive
Transducer (HAT) model introduced in [18] for encoder-decoder models, allowed
for the computation of an internal language model component that can be
quantified and appropriately interpolated with an external language model
(ELM). The density ratio method proposed in [19] offers a theoretically
grounded solution to leverage an external language model while separating out
the acoustic likelihood score and the internal LM score on the source domain.
This modular framework lends itself to principled approaches of LM rescoring
and adaptation thus overcoming some of the shortcomings of the aforementioned
LM integration strategies [18, 20].

ASR systems perform best when the training data is matched to the target
domain. However, end-to-end ASR models are trained on large quantities of
available speech data and the LM is trained on the limited text data available
in the target domain, thus enabling cross-domain transfer. Alternatively,
Large LMs are trained on vast quantities of text and subsequently fine tuned
on target domain text. In both scenarios, finding an optimal combination of
the end-to-end ASR model, with its implicitly trained internal LM and the
external LM, is critical for best performance in the target domain. Neural
Oracle Search leverages HAT factorization for LM rescoring with an external LM
to directly pick the oracle hypothesis [8], while others have explored on-
device neural and biasing LM integration [21] and compared rescoring and
deliberation [22], demonstrating wins across all tasks.

In this paper, we study the impact of LLMs within the HAT framework for long-
form ASR. Using data from two different sources, US English (en-us) and Indian
English (en-in) which is heavily code-switched with Hindi and other Indian
languages, we show that wins of up to 8% relative can be obtained in long-form
ASR while achieving a reduction of up to 30% relative on Salient Term Error
Rate (STER) over a strong first-pass baseline that uses a maximum-entropy
based language model. We also demonstrate the importance of improved lattice
quality that results in a lattice with a proper (non-tree) digraph topology
and carrying context from the 1-best hypothesis of the previous segment(s)
obtain best performance with LLMs. We find that both Text-to-Text Transfer
Transformer (T5) [3] and its multilingual counterpart, MT5 [23] are
complementary to conventional neural LMs and outperform a strong first-pass
baseline that utilizes a maximum entropy LM.

##  3 Large Language Models

Several LLMs have been proposed to date with significant improvements on
varied NLP tasks. In this work, we mainly focus on two LLMs, T5 and PaLM,
ranging in size from 3B to 540B parameters, summarized in Table 1. The
conventional neural LM used for comparisons is a conformer LM described in
Section 4.4 and comprising of 70M parameters.

Conventional LMs | Size |  T5 [3] | Size |  MT5 [23] | Size |  PaLM [5] | Size
---|---|---|---|---|---|---|---
Neural LM | 70M |  S | 60M |  |  |  S | 8B
MaxEnt | 4.5B |  M | 220M |  |  |  M | 62B
|  |  L | 770M |  |  |  L | 540B
|  |  XL | 3B |  XL | 3.7B |  |
|  |  XXL | 11B |  XXL | 13B |  |
Table 1: Comparison of LM sizes.

###  3.1 T5 and PaLM

Built on an encoder-decoder transformer-based architecture, T5 optimizes the
log-likelihood of the target text given input to learn a mapping from the
input to target. While T5 is pretrained on the span corruption task, LM and
Prefix LM are two fine-tuning tasks used for language modeling. The LM task
predicts the target sequence with null context input while the prefix LM task
randomly splits the text into two halves, using the first half as the input to
predict the second half. These fine-tuning tasks enable direct computation of
log-likelihood of the target text, instead of the estimation of a pseudo log-
likelihood as proposed initially in [2] for masked LMs. Thus, given a text
sequence $Y$, similar to the LM task, we can compute its T5 score
$S_{\text{T5}}(Y)$ by using an empty string $\epsilon$ as input and the text
sequence $Y$ as target, with the following equation:

$S_{\text{T5}}(Y)=\log P_{\text{T5}}(Y|\epsilon;\Theta_{\text{T5}}).$ (1)

For longer sequences, we can make better use of the previous context and
compute the score in a semi-autoregressive fashion. Therefore, $Y$ can be
split into multiple segments $Y_{1}\dots Y_{S}$ and the log-likelihood of the
current segment can be computed using the previous segment’s context:

$S_{\text{T5}}(Y)=\sum_{s=1}^{S}\log
P_{\text{T5}}(Y_{s}|Y_{s-1};\Theta_{\text{T5}}),\vspace{-0.15cm}$ (2)

where $Y_{0}$ being $\epsilon$.

PaLM is an autoregressive LM with a decoder-only architecture. Hence the score
of a text sequence can be computed straight-forwardly.

###  3.2 Integration with ASR Models

In this work, we use a first-pass model based on the conformer architecture
[24] that uses HAT factorization [18]. Not only does HAT model provide a
posterior score $S_{\text{HAT}}(Y|X)$, but it also estimates the internal LM
(ILM) score. As mentioned in Section 2, when interpolating an external LM
during rescoring or shallow fusion, estimating and subtracting the internal LM
score yields wins. Thus, inference search maximizes:

$S(Y,X)=S_{\text{HAT}}(Y|X)-\mu S_{\text{ILM}}(Y)+\nu S_{\text{ELM}}(Y),$ (3)

where $\mu$ and $\nu$ are tunable hyperparameters.

##  4 Experiments

###  4.1 Data

We conduct experiments with data from two language locales, en-us and en-in.
The multi-domain ASR model used in this paper is trained on several thousand
hours of long-form utterances derived from YouTube videos[25] and short-form
utterances that are anonymized, hand-transcribed and are representative of
Google’s Voice Search traffic [26]. The test sets contain long-form utterances
derived from 30-minute-long YouTube videos. We set aside a subset containing
5% of the test utterances as the development test to tune the hyperparameters.

The pre-training corpus used to train T5 is the publicly available, Colossal
Clean Crawled Corpus(C4), while MT5 is pre-trained on the multilingual
variant, MC4 [23]. To address code-switching seen in en-in [27], text data
consisting of Indian English and Hindi Wikipedia and CCNet [28] collectively
referred to as WEBDOC, is used. This corpus consists of 170M sentences
yielding 2.9B word tokens. We use 90% data for training and 10% data for
validation. All data in mixed writing systems is transliterated to Latin to be
consistent with ASR model training data used for en-in.

###  4.2 Training Large Language Models

We experimented with T5 and MT5 models of sizes XL and XXL. Both T5 and MT5
models were pre-trained for 1M steps using the span corruption task and then
fine-tuned for 100K steps using the prefix LM task on C4/MC4. To address the
heavy code-switching prevalent in en-in and the lack of Hindi data in MC4
corpus, we fine-tune MT5 on the LM task for an additional 300k steps on the
WEBDOC corpus. PaLM models with three different sizes were trained as
described in [5] for the en-us task. The corpus used to train these models
consisted of filtered web pages, books, Wikipedia, news articles, source code,
and social media conversations. We use these pre-trained models as-is with no
additional fine-tuning.

###  4.3 ASR Models

We use a first-pass ASR model based on the conformer architecture [24] that
uses HAT factorization [18]. The encoder consists of a convolution subsampling
layer and 17-layers of conformer blocks. A conformer block is composed of a
feed-forward module, multi-headed self-attention with relative positional
encoding module, a convolution and a final feed-forward module, stacked
together. The configuration used in this work has an encoder dimension of 512,
8 attention heads, a convolution kernel size of 32 and a decoder dimension of
640 [24]. The decoder at label $y_{u}$ is only conditioned on the previous two
labels $y_{u-1}$ and $y_{u-2}$, with their embeddings concatenated and
projected [29]. The models are trained on 80-dimensional log-mel filter bank
coefficients and predict word-piece targets (4096 for en-us and 8192 for en-
in). The choice of these parameters was determined by sweeping for best
performance within the expected model size.

###  4.4 Neural and Maximum-Entropy based Language Models

In order to better understand the value of LLMs in ASR, we trained two state-
of-the-art LMs, a conventional neural LM and a Maximum Entropy based LM. The
conventional Neural LM is a small, unidirectional, conformer LM (CLM) with 70M
parameters, originally designed for on-device rescoring [21]. It consists of
12 causal conformer layers, each with a dimension of 384, a feedforward layer
dimension of 2048, a convolution kernel of size 15. We use 4-headed self
attention with a left context size 31. The model is trained on the same data
as the LLMs to predict the same word-piece targets as the first-pass ASR
model. Thus, for en-us, we trained it on C4 and for en-in, we trained it on
WEBDOC to match the fine-tuning corpus of MT5. The Maximum Entropy based
(MaxEnt) LM [30, 31] is a log linear model based on N-gram and skip-gram word
contexts, with a size of 4.5B parameters and is comparable to the size of the
T5/MT5 XL models. It is also trained on the same data as the conventional
Neural LM.

###  4.5 Decoding and Rescoring

Decoding is performed by a time-synchronous beam search using the breadth-
search expansion strategy [32] where the number of active hypotheses at each
frame is bounded by a beam size $k$. A VAD-based segmenter [33] runs in
parallel to the beam-search decoder. When the decoder receives an end-of-
segment signal from the segmenter, a segment lattice is generated from the
currently active hypotheses. If present, a rescoring LM is applied to this
segment lattice, with the 1-best hypotheses from previous segments optionally
provided as context. Only the best hypothesis in the lattice (eventually after
rescoring) is carried forward in the beam-search for the next segment. The
final utterance lattice is obtained by concatenating all the segment lattices.

When using an ASR model with unlimited label context, each hypothesis within
the beam encodes the full history from the beginning of the utterance. Hence,
the segment lattice is a trie with a total number of paths (e.g. hypotheses)
bounded by the beam size $k$.

When using an ASR model where the label context is bound by $n$ [34], beam-
search hypotheses sharing the same label context of length $n$ will correspond
to the same state in the segment lattice. This results in lattice with a
proper (non-tree) digraph topology where the number of paths can grow up to
exponentially in the number of states. This was shown to lead to a significant
improvement in lattice quality: lattice diversity improvement and oracle WER
reduction [34].

The ASR models described in section 4.3 used limited label context with $n=2$.
However when combining these models with the conformer LMs from section 4.4
during the beam search using HAT fusion results in dramatic increase of the
label context limit making the resulting combined model to effectively have
unlimited label context.

##  5 Results

###  5.1 Lattice Quality

dev | Oracle WER | WER | #paths/segment
---|---|---|---
| en-us | en-in | en-us | en-in | en-us | en-in
Baseline | 7.3 | 12.8 | 12.2 | 17.2 | 4e20 | 4e13
No state merging | 8.8 | 13.1 | 12.2 | 17.2 | 5.7 | 5.8
Neural LM fusion | 8.4 | 11.0 | 11.6 | 15.6 | 5.2 | 5.7
Table 2: Lattice quality on the en-us and en-in dev sets.

The success of a rescoring approach crucially depends on the quality of the
hypotheses of the first-pass beam-search decoder. To assess the lattice
quality, we computed metrics such as the $N$-best oracle WER and the average
number of paths/hypotheses per segment for our baseline systems on the en-us
and en-in development sets as reported in Table 2.

As the contribution to first-pass model’s posterior and internal LM at label
$y_{u}$ depends only on the previous two labels, our baseline systems can
leverage the state merging benefits of limited context models described in
Section 4.5 as demonstrated by the relatively low oracle WER and high number
of paths per segments.

Lattice quality can be improved by improving first-pass modeling by
integrating a neural LM in the beam-search decoding using HAT fusion. Table 2
shows this results in a significant improvement in 1-best WER. However, this
causes the loss of the state merging benefits and results in an increase of
oracle WER in en-us. However, this is still an significant improvement
compared to disabling state merging in the baseline systems.

###  5.2 Comparison of LMs

In this Section, we consider the impact of LM integration on the en-us task.
Table 3 demonstrates the value of providing longer context to Large LMs. Each
row contains the result of rescoring with the T5 XXL model when carrying over
contexts of different lengths, i.e., of carrying over the 1-best hypotheses
from different number of previous segments. We observe that carrying over
previous context outperforms no context. However, longer contexts do not seem
to provide additional wins. The rest of this paper thus uses contextual
information from just the previous segment.

WER | dev
---|---
Baseline | 12.2
\+ T5 rescoring, carrying 0 segment | 11.6
\+ T5 rescoring, carrying 1 segment | 11.5
\+ T5 rescoring, carrying 2 segments | 11.5
Table 3: WER comparison on the en-us test set for different lengths of carried
over context

Table 4 presents the rescoring and fusion results on the en-us development and
evaluation test sets for various LMs. First we observe that a small Neural LM
edges out over the performance of a Maxent LM. Moreover, though the T5 S
model, whose size is slightly smaller than the NLM, was slightly behind NLM,
increasing the size of T5 leads to better results. It is also interesting to
note that the NLM and T5 XXL models are complementary, as fusion can give a
better 1-best WER. In addition, we experimented with more enormous PaLM LMs
and they are able to brings the power of larger capacity and large amounts of
training text, yielding better results than T5.

WER | dev | eval
---|---|---
Baseline | 12.2 | 16.1
\+ MaxEnt rescoring | 12.2 | 16.4
\+ NLM rescoring | 11.8 | 15.8
\+ T5 S rescoring | 11.9 | 15.9
\+ T5 M rescoring | 11.7 | 15.8
\+ T5 XL rescoring | 11.6 | 15.7
\+ T5 XXL rescoring | 11.5 | 15.7
\+ PaLM S rescoring | 11.5 | 15.5
\+ PaLM M rescoring | 11.3 | 15.4
\+ PaLM L rescoring | 11.3 | -
\+ NLM fusion | 11.6 | 15.6
\+ NLM fusion & T5 XXL rescoring | 11.4 | 15.5
Table 4: en-us WER comparison between T5 and other LMs

###  5.3 Code-switching Task

In this Section, we present the performance of LLMs on a more challenging en-
in task dominated by heavy code-switching. Although MT5 is meant to be a
multilingual LM, the amount of training data from the different languages is
unbalanced. The training data consists of 5.67% English, but only 1.21% is
Hindi in the Devanagari script [23]. This imbalance between en-in and Hindi
fails to capture the frequent code switches between English and Hindi
predominant in the en-in test sets. To address this issue, we finetune both XL
and XXL MT5 models on the WEBDOC corpra with the LM task. We evaluate the raw
MT5 model and these fine-tuned models on the en-in development set to study
the effect of fine-tuning. These results are tabulated in Table 5.

en-in dev | MT5 XL | MT5 XXL
---|---|---
Baseline | 17.2
Raw | 16.6 | 16.8
Fine-tuned | 16.1 | 16.3
Table 5: WER comparison on en-in dev set with raw and fine-tuned MT5 models of
sizes XL and XXL

It can be seen that rescoring with the fine tuned models outperforms rescoring
with the raw MT5 model. This can be attributed to the lack of sufficient Hindi
data in the MC4 corpus which can be fixed with data balanced fine-tuning. When
compared to en-us, the wins from LLMs on en-in are less. We hypothesize that
this could be related to the small size of the WEBDOC corpus compared to MC4,
in line with the data-hungry nature of LLMs [35, 36].

###  5.4 Comparison of LMs on the code-switching task

WER | dev | eval
---|---|---
Baseline | 17.2 | 16.4
\+ MaxEnt rescoring | 16.5 | 15.9
\+ NLM rescoring | 16.2 | 15.4
\+ MT5 XL rescoring | 16.1 | 15.2
\+ NLM fusion | 15.6 | 15.0
\+ NLM fusion & MT5 XL rescoring | 15.4 | 14.6
Table 6: en-in WER comparison between MT5 and other LMs

Table 6 presents the rescoring results from various LMs. The MT5 XL model is
the best performing model with a WER reduction of 7.3% relative on the
evaluation test set. On the other hand, the Conformer LM when used in shallow
fusion in the first-pass shows additional wins. Since we fine-tuned MT5 on the
same training data as Conformer LM, we also report the perplexity of MT5 and
Conformer LM on the 10% validation part of WEBDOC. MT5 has a log perplexity
per word of 4.15, slightly higher than the Conformer LM at 2.98 and MaxEnt at
3.69. We observe that the Conformer LM and MT5 are complementary and the
combination results in a best WER reduction of 8% relative.

##  6 Error Analysis

To analyze the effectiveness of large LM, we select unigrams and bigrams with
the highest Term Frequency Inverse Document Frequency (TF-IDF) values from the
evaluation test sets (salient terms) for the two languages studied in this
paper. In general, such terms capture the topic presented in the video. On the
one hand, they are important for indexing or information retrieval; on the
other hand, they are more difficult to be recognized compared to frequently
occurring function words (such as, ”the”, ”of”, etc.). We analyzed the
performance of the baseline and the various large LMs on these salient terms
to study the impact on rare words. The Salient Term Error Rate (STER) is
reported in Table 7, defined as the number of deletion and substitution errors
on the salient terms divided by the total number of salient terms. Out of a
total of 600K words, approximately, 10% words are tagged as salient terms for
en-in and 5% for en-us. First we observe that almost all rescoring and fusion
can reduce the error made on these salient terms. In en-us, as reflected by
the WER reported in Table 4, T5 outperforms other LMs. In en-in, however, NLM
fusion in the first pass has a bigger impact on the salient terms than any
rescoring method similar to what has been reported in [37]. Although MT5 has
been fine tuned on the same data as the NLM, we find that it is less impactful
by itself on the salient terms in en-in. Although MT5 has been fine tuned on
the same data as the neural LM, we find that it is less impactful by itself on
the salient terms. However, in both languages, the combination of these two
LMs through interpolation is additive (last row in Table 6) resulting in the
best performance. As noted in [35, 36] scaling to larger and larger datasets
is only beneficial when the data is high-quality and larger models require
larger data sets. This can explain some of the differences seen between these
two relatively high resource languages.

STER | en-us | en-in
---|---|---
Baseline | 28.8 | 20.0
\+ MaxEnt rescoring | 28.8 | 17.4
\+ NLM rescoring | 27.4 | 16.7
\+ T5/MT5 rescoring | 26.7 | 17.6
\+ NLM fusion | 27.2 | 15.4
\+ NLM fusion & T5/MT5 rescoring | 26.4 | 12.1
Table 7: Errors analysis on salient terms of en-us and en-in.

##  7 Conclusion

In this study, we presented the impact of LLMs (up to 350B parameters) on
long-form ASR. We demonstrated up to 8% relative reduction in Word Error Rate
(WER) on US English (en-us) and code-switched Indian English (en-in) long-form
ASR test sets and a reduction of up to 30% relative on Salient Term Error Rate
(STER) over a strong first-pass baseline that uses a maximum-entropy based
language model. We also find that the gains in performance from the
combination of LLMs trained on vast quantities of available data (such as C4
[1]) and conventional neural LMs is additive and significantly outperforms a
strong first-pass baseline with a maximum entropy LM. To the best of our
knowledge, this is the first study that scales LLMs to long-form ASR.

## References

  * [1] C. Raffel _et al._ , “Exploring the limits of transfer learning with a unified text-to-text transformer.” _J. Mach. Learn. Res._ , vol. 21, no. 140, pp. 1–67, 2020.
  * [2] A. Wang and K. Cho, “Bert has a mouth, and it must speak: Bert as a markov random field language model,” _arXiv preprint arXiv:1902.04094_ , 2019.
  * [3] C. Raffel _et al._ , “Exploring the limits of transfer learning with a unified text-to-text transformer,” _Journal of Machine Learning Research_ , vol. 21, no. 140, pp. 1–67, 2020.
  * [4] T. Brown _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [5] A. Chowdhery _et al._ , “Palm: Scaling language modeling with pathways,” _arXiv preprint arXiv:2204.02311_ , 2022.
  * [6] X. Zheng, C. Zhang, and P. C. Woodland, “Adapting gpt, gpt-2 and bert language models for speech recognition,” in _2021 IEEE ASRU_ , 2021, pp. 162–168.
  * [7] J. Salazar, D. Liang, T. Q. Nguyen, and K. Kirchhoff, “Masked language model scoring,” in _2020 ACL_ , Jul. 2020.
  * [8] E. Variani _et al._ , “Neural oracle search on n-best hypotheses,” in _ICASSP_ , 2020, pp. 7824–7828.
  * [9] S.-H. Chiu and B. Chen, “Innovative bert-based reranking language models for speech recognition,” in _SLT_. IEEE, 2021, pp. 266–271.
  * [10] T. Hori, Y. Kubo, and A. Nakamura, “Real-time one-pass decoding with recurrent neural network language model for speech recognition,” in _ICASSP_. IEEE, 2014, pp. 6364–6368.
  * [11] J. Chorowski and N. Jaitly, “Towards better decoding and language model integration in sequence to sequence models,” _arXiv preprint arXiv:1612.02695_ , 2016.
  * [12] T. Hori, J. Cho, and S. Watanabe, “End-to-end speech recognition with word-based rnn language models,” in _SLT_. IEEE, 2018, pp. 389–396.
  * [13] C. Peyser _et al._ , “Improving tail performance of a deliberation e2e asr model using a large text corpus,” _arXiv preprint arXiv:2008.10491_ , 2020\.
  * [14] A. Sriram, H. Jun, S. Satheesh, and A. Coates, “Cold fusion: Training seq2seq models together with language models,” _arXiv preprint arXiv:1708.06426_ , 2017.
  * [15] C. Gulcehre _et al._ , “On using monolingual corpora in neural machine translation,” _arXiv preprint arXiv:1503.03535_ , 2015.
  * [16] C. Shan _et al._ , “Component fusion: Learning replaceable language model component for end-to-end speech recognition system,” in _ICASSP_. IEEE, 2019, pp. 5361–5635.
  * [17] A. Kannan _et al._ , “An analysis of incorporating an external language model into a sequence-to-sequence model,” in _ICASSP_ , 2018, pp. 1–5828.
  * [18] E. Variani, D. Rybach, C. Allauzen, and M. Riley, “Hybrid autoregressive transducer (hat),” in _ICASSP_ , 2020, pp. 6139–6143.
  * [19] E. McDermott, H. Sak, and E. Variani, “A density ratio approach to language model fusion in end-to-end automatic speech recognition,” in _ASRU_ , 2019, pp. 434–441.
  * [20] C. Allauzen, E. Variani, M. Riley, D. Rybach, and H. Zhang, “A hybrid seq-2-seq ASR design for on-device and server applications,” in _Interspeech 2021_ , 2021, pp. 4044–4048.
  * [21] T. N. Sainath _et al._ , “An efficient streaming non-recurrent on-device end-to-end model with improvements to rare-word modeling,” in _Interspeech_ , 2021, pp. 1777–1781.
  * [22] K. Hu _et al._ , “Improving deliberation by text-only and semi-supervised training,” _arXiv preprint arXiv:2206.14716_ , 2022.
  * [23] L. Xue _et al._ , “mT5: A massively multilingual pre-trained text-to-text transformer,” in _2021 NAACL: Human Language Technologies_ , Jun. 2021.
  * [24] A. Gulati _et al._ , “Conformer: Convolution-augmented transformer for speech recognition,” _arXiv preprint arXiv:2005.08100_ , 2020.
  * [25] H. Liao, E. McDermott, and A. Senior, “Large scale deep neural network acoustic modeling with semi-supervised training data for youtube video transcription,” in _ASRU_ , 2013, pp. 368–373.
  * [26] A. Narayanan _et al._ , “Recognizing long-form speech using streaming end-to-end models,” in _ASRU_ , 2019, pp. 920–927.
  * [27] J. Emond, B. Ramabhadran, B. Roark, P. Moreno, and M. Ma, “Transliteration based approaches to improve code-switched speech recognition performance,” in _SLT_. IEEE, 2018, pp. 448–455.
  * [28] G. Wenzek _et al._ , “Ccnet: Extracting high quality monolingual datasets from web crawl data,” _arXiv preprint arXiv:1911.00359_ , 2019.
  * [29] R. Botros _et al._ , “Tied & reduced rnn-t decoder,” _arXiv preprint arXiv:2109.07513_ , 2021.
  * [30] F. Biadsy, K. Hall, P. Moreno, and B. Roark, “Backoff inspired features for maximum entropy language models,” 2014.
  * [31] F. Biadsy, M. Ghodsi, and D. Caseiro, “Effectively building tera scale maxent language models incorporating non-linguistic signals,” 2017.
  * [32] A. Tripathi, H. Lu, H. Sak, and H. Soltau, “Monotonic recurrent neural network transducer and decoding strategies,” in _ASRU_ , 2019, pp. 944––948.
  * [33] R. Zazo, T. N. Sainath, G. Simko, and C. Parada, “Feature learning with raw-waveform cldnns for voice activity detection,” in _Interspeech_ , 2016, pp. 3668–3672.
  * [34] R. Prabhavalkar _et al._ , “Less is more: Improved rnn-t decoding using limited label context and path merging,” in _ICASSP_ , 2021, pp. 5659–5663.
  * [35] J. Kaplan _et al._ , “Scaling laws for neural language models,” _arXiv preprint arXiv:2001.08361_ , 2020.
  * [36] J. Hoffmann _et al._ , “Training compute-optimal large language models,” _arXiv preprint arXiv:2203.15556_ , 2022.
  * [37] V. Ravi _et al._ , “Improving accuracy of rare words for rnn-transducer through unigram shallow fusion,” _arXiv preprint arXiv:2012.00133_ , 2020\.


# A Perspective on Individualized Treatment Effects Estimation from Time-
series Health Data

Ghadeer O. Ghosheh  Institute of Biomedical Engineering, University of Oxford
Moritz Gögl  Institute of Biomedical Engineering, University of Oxford
Tingting Zhu  Institute of Biomedical Engineering, University of Oxford

## Abstract

The burden of diseases is rising worldwide, with unequal treatment efficacy
for patient populations that are underrepresented in clinical trials.
Healthcare, however, is driven by the average population effect of medical
treatments and, therefore, operates in a “one-size-fits-all” approach, not
necessarily what best fits each patient. These facts suggest a pressing need
for methodologies to study individualized treatment effects (ITE) to drive
personalized treatment. Despite the increased interest in machine-learning-
driven ITE estimation models, the vast majority focus on tabular data with
limited review and understanding of methodologies proposed for time-series
electronic health records (EHRs). To this end, this work provides an overview
of ITE works for time-series data and insights into future research. The work
summarizes the latest work in the literature and reviews it in light of
theoretical assumptions, types of treatment settings, and computational
frameworks. Furthermore, this work discusses challenges and future research
directions for ITEs in a time-series setting. We hope this work opens new
directions and serves as a resource for understanding one of the exciting yet
under-studied research areas.

##  1 Introduction

Medical treatment and drug budgets are the highest burdens on governments and
medical institutions. Despite these high costs, only about 90% of drugs work
for 30-50% of the population [27]. These statistics suggest a pressing need to
identify patient subgroups where personalized treatments can be prescribed.
Studying treatment effects has gained much attention over the past years,
where various tools and approaches have been proposed to help mitigate the
financial cost and optimize the effectiveness and efficacy of prescribed
treatments. One of the fast-growing applications in clinical Machine Learning
(ML) is studying Individualized Treatment Effects (ITEs) [5], where ML
capabilities, in many cases, have superseded those of state-of-the-art
clinical and pharmaceutical advancements. Using the wealth of observational
Electronic Health Records (EHRs) data, individualized patient response to
various treatments can be estimated [5].

While there has been increased attention to ITE works from static
observational EHR data [49, 9], much less attention has been given to those
from EHR-data measured over time, often referred to as time-series EHRs.
Despite similarities in the concepts for treatment estimation between static
and time-series data, challenges related to the time-varying nature of
covariates make many existing works not directly applicable to time-series
data. To this end, we aim to provide an overview of the main concepts and
ideas for estimating ITE from time-series data. Starting with introducing the
concepts of Randomized Controlled Trials (RCTs) and observational data, then
discussing topics such as efficacy and effectiveness of treatments and
challenges in ITE estimation. This work aims to bridge the theoretical
assumptions from causal inference and machine learning works to real-world EHR
implications. To the best of our knowledge, this is the first piece of work
comprehensively reviewing challenges and methods in ITE estimation for time-
series data. We summarize the latest work in the literature and group them
based on theoretical assumptions, types of treatment settings and
computational frameworks. Lastly, this work discusses challenges and future
research directions for ITEs in a time-series setting. We hope this work opens
new directions and serves as a resource for understanding one of the exciting
yet under-studied research areas.

###  1.1 Randomized Controlled Trial Data

RCT data are considered the gold standard for studying treatment effects. This
mainly stems from the random assignment of participants to treatment groups,
which eliminates confounding bias effectively. The randomness and control
measures employed in RCTs make them a lucrative option for studying the effect
of treatments; an unbiased estimate of the average treatment effects (ATE) can
be directly computed from the data [31]. While RCTs offer methodological
strengths, various challenges hinder their full and optimal usage for studying
treatment effects [6]. Firstly, despite the power of "randomness" to eliminate
confounding, certain biases may remain in RCTs. This bias does not come from
the treatment assignment level but from representatives of samples presented
in the study, denoted as sample selection bias hereafter. Most RCTs tend to
employ stringent exclusion criteria of enrolled participants, which might
introduce bias in the results for the unrepresented population members [30,
23, 42, 6, 45]. This becomes a bigger issue when a significant percentage of
treated patients in real-world data belong to the population excluded from the
RCT. For example, the ageing population is rarely enrolled in diabetes RCTs
due to their old age and multi-morbid health conditions, despite constituting
a large proportion of diabetic patients [22]. Such factors might introduce
bias that makes RCTs’ generalizability and external validity questionable [17,
6].

The generalizability of RCT results is further limited by the fact that
current RCTs are typically designed to only measure the ATE. In other words,
they estimate how the "average patient" will respond to a given treatment. For
a more personalized approach, the patient’s unique characteristics would be
needed to predict an individual patient’s response to a given treatment, and
it may differ significantly from the average response of a population.
Furthermore, RCTs tend to be financially costly to design and implement [19,
40] and their data tend to be hard to share due to privacy concerns [41].
Additional limitations exist in terms of their relatively small sample size
[17], ethical issues [15], and short lengths of follow-ups which might miss
out on the long-term effects of medications [6]. For example, the effects of
oral contraceptives were not quantified until the presence of long-term data,
which were not captured in RCTs [44].

###  1.2 Observational Data: From Efficacy to Effectiveness

Despite all of the stated challenges, RCTs are still essential for determining
the Efficacy for treatments [25] but are not necessarily optimal for studying
treatment Effectiveness. Making a clear distinction between these two terms
will help one to understand when data-driven and statistical approaches can
improve the use of RCT data. Efficacy refers to the effect of interventions
under ideal "theoretical" circumstances, while Effectiveness means an effect
is detected not under ideal but under real-world conditions [25].
Observational EHR data presents a good candidate to better test for treatment
effectiveness. Specifically, longitudinal observational data collected in EHRs
typically includes a diverse patient cohort with no strict exclusion criteria,
making it more representative of the real or targeted patient population.
Observational data is also less expensive to collect when compared to RCTs and
captures long-term outcomes [6, 34]. Moreover, with the widespread use of EHR
systems worldwide, observational data can allow for estimating treatment
effects from diverse clinical settings such as low-middle-income countries
(LMICs), where performing RCTs would not be feasible due to high costs. To
this end, observational data is a promising resource for studying treatment
effects with more inclusive estimations for various patient groups while
maintaining low cost and learning from real-world evidence.

###  1.3 Challenges of Treatment Effects Estimation

Treatment effect estimation is a subfield of causal inference, and as such
suffers from the fundamental problem that counterfactual outcomes are never
observed [38]. A counterfactual outcome refers to the hypothetical outcome
that would have been observed if a different treatment than the actual
(factual) one had been given [5]. Of course, for treatments that were not
administered, it is not possible to extract the ground truths of individual
patients’ outcomes directly, whether from observational or clinical trial
data. In RCTs, this problem is resolved by estimating the ATE across the
entire study population but not the ITE of an individual. Randomization of
treatment allocation ensures that the underlying distribution of the patient
characteristics is similar in both the treatment and control groups. This
allows us to compare the average outcomes in both groups and thus determine
the ATE.

Omitting the randomized control measures used in RCTs and relying on
observational data precludes the direct computation of treatment effects. This
is because the treatment assignment is not random, but biased, as treatment
selection in observational data is often driven by the patient’s
characteristics, such as treatment allocation flowcharts in clinical practice
guidelines [46]. Therefore, clinical practice recorded in real-world
observational data results in systematic differences in the characteristics of
treated and untreated patients. To be able to estimate treatment effects from
observational data, it is essential to remove the confounding bias, introduced
by the non-random treatment assignment. A confounder is a variable that
influences both the intervention and the outcome, potentially leading to a
spurious association between them [1]. For example, in clinical practice,
patients with more severe health conditions might be more likely to be given
stronger medications while still being expected to have poorer outcomes.
However, it would be wrong to conclude that stronger medication leads to
poorer patient outcomes. Failing to adjust for the severity of a health
condition as a confounding factor in this case would lead to measuring a
spurious association that does not reflect the actual treatment effect.

###  1.4 From Average to Individualized Treatment Effects Estimation

The modern understanding of estimating treatment effects is highly attributed
to the work of Neyman-Rubin’s "potential outcomes framework" [39]. In the
Neyman-Rubin model, the ITE between a treatment $A$ and a treatment $B$ is
defined as the difference between the two potential outcomes (e.g., blood
pressure) after administering a treatment $A$ or $B$ to a given patient.
Subject to certain assumptions [5], the ATE can be computed directly from RCT
data by calculating the difference between the average outcomes in both
treatment groups. However, due to the absence of counterfactuals in real-world
data, ITEs cannot be calculated directly but must be estimated through the use
of models. Based on statistical methods, but also driven by recent
developments in machine and deep learning approaches, various models have been
proposed to estimate treatment effects on a personalized level [49, 14, 4,
32]. Overall, most of these models estimate the potential outcome for an
individual patient by learning the underlying effects and interactions between
patients’ characteristics, treatment and outcome. The patient’s ITE can then
be calculated as the difference between the predicted potential outcomes with
and without treatment. Furthermore, various methods were proposed to address
the problem of confounding bias introduced in observational studies as a
result of unobserved confounders [37, 3, 26], paving the way for ITE
estimations for personalized medicine.

##  2 Estimating Individualized Treatment Effects from Time-series Data

Various works for estimating ITE using observational EHR data have been
proposed recently [49, 43, 14]. Despite the plethora of works, most of them
estimate the treatment effect using static data, where each patient is
represented as a snapshot of covariates at the exposure of the treatment.
While useful, using only static data and disregarding the time component have
many limitations for estimating the impact of treatment over time.
Furthermore, the ITE estimation from static data limits the opportunities for
learning when to stop or change the treatments when the outcome is dynamic and
varies over time, a critical clinical application for ITE. Additionally, in
the static treatment effect estimation setup, the treatments are assigned at a
single time point and often remain static over time. On the other hand, using
time-series data for treatment effect estimation would allow for monitoring
and changing the treatment plan dynamically while simultaneously observing the
effect of time-varying treatment on patient covariates and outcomes of
interest. A typical example of a time-series ITE problem is in the cancer
application, where a patient’s treatment option (e.g., radiation or
chemotherapy) is adjusted according to his or her clinical response (tumor
size) over time. In the static setting, such a problem cannot be addressed due
to the absence of the time factor.

Despite the promise and potential of treatment effect estimation from time-
series data, the major problem lies in the time-varying or temporal
confounders. Similar to the static confounders, a temporal confounder is
typically a time-varying variable that impacts both the treatment assignment
and outcome. For example, consider that Angiotensin receptor blockers (ARBs)
(treatment 1) are given when a hypertension patient’s blood pressure
(covariate) is outside the normal range value. Suppose also that this
patient’s covariate was affected by the past administration of ACE inhibitor
(treatment 0), another type of treatment for uncontrolled blood pressure.
Estimating the effect of a different sequence of treatments on the patient
outcome would require adjusting for the bias at the current step (treatment 1)
and the bias introduced by the previous application of ACE inhibitors
(treatment 0). Adjusting for time-varying confounders remains a major
challenge hindering the direct application of methodologies developed for
static treatment effects tasks to dynamic problem settings. Here we have
reviewed works in the literature for estimating treatment effects from time-
series data, including the estimation frameworks, model architectures and
assumptions used. An overview of the existing work on ITE from time-series
data is presented in Table 1. They are categorized into two main groups: (i)
outcome estimation methods which focus on inferring the ITE by estimating the
potential outcomes of different treatments, and (ii) deconfounder methods,
which estimate the ITE in the presence of hidden confounders. More details are
presented in Sections 2.2 and 2.3, covering ITE outcome estimation and
deconfounder methods for time-series data, respectively. In Figure 1, we show
an example causal model that underlies a dynamic ITE estimation setting with
time-varying treatments and covariates.

Table 1: A summary of ITE works for time-series data.

| Proposed Methods | Estimation Framework | Assumptions | Model Architecture | Validation Data
---|---|---|---|---|---
Outcome Estimation Methods | MSM [37] | MSM | C/P/SSI | LR | NA
RMSN [29] | MSM | C/P/SSI | LSTM | simulated tumor dynamics data
CRN [4] | Balanced Representation | C/P/SSI | LSTM | simulated tumor dynamics data, ICU data
G-Net [28] | G-formula | C/P/SSI | LSTM |  | simulated tumor dynamics data,
---
simulated cardiovascular data
Causal Transformer [32] | Balanced Representation | C/P/SSI | Transformer |  | simulated tumor dynamics data,
---
(semi-synthetic) ICU data
Deconfounder Methods | Time Series Deconfounder [3] | Latent Factor Model | C/P/SSSI | RNN Factor Model | simulated data, ICU data
Sequential Deconfounder [16] | Latent Factor Model | C/P/TIUC | GPLVM | simulated data, ICU data
Deconfounding Temporal AutoEncoder [26] | Noisy Proxies | C/P | AutoEncoder | simulated data, ICU data

*The abbreviations in full form. (C): Consistency, (SO): Sequential Overlap, (SSI): Sequential Strong Ignorability, (SSSI): Sequential Single Strong Ignorability, (TIUC): Time-Invariant Unobserved Confounding, (LR): Logistic Regression, (LSTM): Long Short-Term Memory, (RNN): Recurrent Neural Network, (GPLVM): Gaussian Process Latent Variable Model.

Figure 1: Illustration of a causal model underlying a dynamic ITE estimation
setting with time-varying treatments and covariates. The arrows indicate
causal dependencies between variables. Here, the observed covariates are blood
pressure (BP) readings that act as time-varying confounders affecting all
subsequent treatment decisions (ACE inhibitor vs. ARB) and the outcome of
interest (risk of stroke). We also showcase an example of an unobserved
"hidden" confounder, namely potential comorbidity that is not directly
represented in the data. All connections from covariates and unobserved
variables to the treatment variables are depicted in red, and would need to be
accounted for to remove confounding bias effectively.

###  2.1 Assumptions

Estimating treatment effects from time-series data relies on the potential
outcome framework [39] and its extensions to the time-varying setting [36]. In
the potential outcome framework, an ITE is the difference between the
potential outcomes for a specific individual given different treatments. Three
main assumptions are typically required for treatment effects to be
identifiable from time-series data. We explain each in lay terms and provide
examples from real-world clinical problems. For interested readers, we provide
references to works that explain and include mathematical notations.

  1. 1.

Consistency. This assumption states that the potential outcome of a patient
should be consistent with his/her "factual" outcome if the same treatment plan
is applied. For example, consider a patient who has been on a diabetes
treatment plan $A$ for several years, where blood glucose is considered an
outcome monitored over time. If a model is built to estimate this patient’s
potential outcome given the same treatment plan $A$, such outcome would be
equivalent to the outcome observed for that patient (i.e., blood glucose). The
mathematical notation and theoretical basis for this consistency are explained
in [4, 32].

  2. 2.

Sequential Overlap or Positivity. This assumption means that each treatment
option has a non-zero probability of being given to the patient at each
timestep. For example, let us consider a cancer treatment where options are
either radiotherapy or chemotherapy. If a patient was given radiotherapy last
month, a doctor might give this patient chemotherapy or radiotherapy this
month, and both have a non-zero probability of being given to the patient.

  3. 3.

Sequential Strong Ignorability This assumption means that conditioned on the
observed patient history, the current treatment assignment is independent of
the potential outcome. In some works, this assumption is referred to as
sequential exchangeability or "no unobserved confounders". In other words, no
unobserved confounders affect both treatment and outcome. While useful, this
assumption can’t be tested in practice since various factors may impact the
treatment and the outcome, yet they might not be recorded or observed. For
this purpose, various works have been proposed to relax this assumption (see
referenced works for estimation with hidden confounders described in Section
2.3). Example variants include Sequential Single Strong Ignorability, where
the assumption is limited to no hidden single cause confounders [3]. Another
example is Time-Invariant Unobserved Confounding, where confounders exist with
the condition that they are the same random variable at each time step [16].

###  2.2 ITE Outcome Estimation Methods in time-series data

####  2.2.1 Marginal Structural Models (MSM)

Various approaches from epidemiology have been proposed for accounting for
time-varying confounders, one of which is inverse probability of treatment
weighting (IPTW) [8]. The main idea behind IPTW involves assigning weights
that will redistribute or balance the population such that the effect of time-
varying confounding is removed. The weights are derived based on the inverse
probability of receiving the patient’s treatment at each respective time point
conditional on their covariate’s history. The IPTW setup creates a pseudo
population set where getting treatment assignment is independent of the
underlying patient characteristics and previous treatment assignments [2]. One
way to implement IPTW in the time-series setting is via marginal structural
models (MSMs) [37]. MSMs focus on controlling the effects of time-varying
confounders affected by previous treatment exposure. The name "Marginal"
refers to the approach of estimating the marginal distribution of the
treatment over time with respect to the outcome [48]. Similarly, the name
"structural" refers to the approach of causal relationship exploration
inspired by econometrics [48]. An MSM first calculates the weights, most
commonly via a regression-based IPTW model, and assigns such weights to each
observation. The estimated weights indicate whether each of the observations
in confounders is under-represented or over-represented in the sample for a
target population [48]. The use of sample reweighing aims to remove the
imbalance and bias caused by the uneven distribution of time-varying
confounders across treatment groups. Finally, the treatment effect can be
estimated using the calculated weights.

While powerful and useful, MSMs have limitations when dealing with high-
dimensional and complicated data dynamics. This is because the treatment
effects predictions are calculated using linear or logistic regression models.
To address this, [29] proposed recurrent marginal structural networks (RMSMs)
where recurrent neural networks [10] were used to estimate the inverse
probability of treatment weights and the counterfactual treatment outcomes.
Similar to the standard MSM two-stage approach, RMSM has two main networks.
The first network calculates the treatment probability weights used for the
IPTW. The second network, on the other hand, is the prediction used to
determine the treatment response given a sequence of treatments and the
calculated weights [29]. Despite the promise shown by the statistical and deep
learning approaches, MSMs can be unstable if the IPTW results in extreme
weights, leading to model misspecification [4].

####  2.2.2 G-formula

Another method for estimating treatment effects in time-varying confounders is
G-formula. G-formula was first described by Robins et al. [35], where the
author proposed a method for generalization of standardization to time-varying
treatments and confounders and referred to it as the G-computation algorithm
formula. Most of the works from epidemiology use the term G-formula or
G-computation to refer to the same method proposed by [35]. The key assumption
for the G-computation formula is that the treatment received at each time was
allocated conditional on the observed past treatment and covariate history
[35]. G-formula works by estimating the conditional distribution of relevant
covariates given covariate and treatment history at each time point, then
producing Monte Carlo estimates of counterfactual outcomes by simulating
forward patient trajectories under treatment strategies of interest [33]. In
most statistical works, the estimation of patient trajectories and outcomes is
done via simplistic regression estimators. While useful, it is important to
remember that simple regression models fail to capture complex dependencies
over time when dealing with high-dimensional time-varying data. In terms of
implementation, there are no well-established G-formula implementations in
statistical packages, which limits its applicability when compared to MSMs.
Recently, [28] proposed G-NEt, the first deep-learning work that estimates
ITEs via an LSTM-based G-formula model. The G-Net results showed improved
performance when compared to those estimated using a logistic regression
estimator and other deep-learning-based models [29, 4].

####  2.2.3 Balanced Representations

Unlike MSM and G-formula-based estimations, a new class of estimation from
deep learning evolved based on learning representations that balance the
distribution of the treatment and control groups. The original works for
learning balanced representations were first proposed for static settings
[20], several studies then used deep learning architecture to learn treatment
invariant representation for each time step to remove the association between
the patient history and treatment assignment. For example, the counterfactual
recurrent network (CRN) [4] is the first work to use a sequence-to-sequence
model to learn balanced representations via adversarial training. In their
proposed work, CRN aims to learn representations not predictive of treatment
assignments yet achieve the highest performance in predicting the outcome.
Another related work is that of [32], where the authors proposed Causal
Transformer, which is a transformer-based model that aims to learn treatment
invariant balanced representations to estimate ITE over time. To do so, the
Causal Transformer comprises three transformer sub-networks for processing the
time-varying covariates, treatments and outcomes, all of which are combined
via a joint network with cross-attentions.

###  2.3 ITE Deconfounding methods for time-series data

####  2.3.1 Latent Factor Models

All of the aforementioned studies for estimating ITE in time-series data focus
on settings where all confounders are observed, or in other words; they
require sequential strong ignorability assumption to hold. Despite the
potential of such works, the sequential ignorability is not testable in
practice. To this end, several studies have proposed approaches where
sequential ignorability is relaxed to account for settings where forms of
hidden confounders exist in the data. For example, the first work to propose a
deep learning model for deconfounding time-series data was the Time Series
Deconfounder [3]. In their proposed work, the authors focus on addressing a
specific type of hidden confounder which they refer to as multi-cause hidden
confounders. The Time Series Deconfounder builds on a factor model to learn
the distribution of treatments over time. By leveraging the dependence between
multiple treatment options at each given time step, the factor model infers
substitutes for unobserved confounders at each time step. The assumption of
sequential strong ignorability is relaxed to sequential single strong
ignorability where the assumption is limited to no hidden single-cause
confounders [3]. The Times Series Deconfounder can be applied to the datasets
before passing the deconfounded data to other outcome estimation models such
as RMSM [29] or CRN [4].

While the results show great promise, the Time Series Deconfounder only works
when there are multiple treatment options and fails when there’s a single
treatment option at each time step. This limitation of the Time Series
Deconfounder is related to its use on the dependence between multiple
treatment options to infer substitutes of hidden confounders. Sequential
Deconfounder, on the other hand, is a method that deconfounds time-series data
for ITE by fitting a Gaussian Process (GP) latent variable model to capture
any sequential dependence between the assigned treatments [16], without the
limitation of depending on multiple treatments. The GP-based latent variable
model aims to control for the substitutes and uses them in conjunction with
outcome estimation models such as RMSM [29] or CRN [4] to estimate treatment
effects over time. While the Sequential Deconfounder does not require multiple
treatments at each time step, it requires a special case of the ignorability
assumption, which they refer to as Time-Invariant Unobserved confounding [16].
This assumption imposes a requirement that the hidden confounder is the same
random variable at each time step.

####  2.3.2 Noisy Proxies

Most aforementioned studies utilize latent models to infer the hidden
confounder in time-series ITE by capturing sequential dependencies. Recently,
Deconfounding Temporal AutoEncoder (DTA) is an autorencoder-based model that
utilizes noise proxies as an alternative to latent factor models to learn
hidden embeddings that resemble the true hidden confounders [26]. The main
assumption in DTA builds on the fact that the observed covariates are not
necessarily true confounders and assumes that the observed covariates are
noisy proxies of the true confounders. DTA aims to learn a hidden embedding
for which the ITE is the same when hidden confounders are present and when
Sequential Strong Ignorbaility applies. To do so, DTA optimizes over a special
loss that is referred to as a cause regularization penalty to yield outcomes
and treatment assignments that are conditionally independent for each hidden
embedding [26].

##  3 Datasets and Evaluation

Most studies found in the literature make use of simulated datasets to
evaluate their methods for ITE estimation from time-series data. Unlike real-
world data, where only the factual outcome is observed, simulations provide
ground truths for all potential outcomes. Some simulated datasets used in the
literature [3, 16, 26] do not aim to mimic specific medical scenarios;
instead, they are based on purely mathematical modeling of time-varying
covariates, hidden confounders, treatments, and outcomes. In contrast, models
such as the pharmacokinetic-pharmacodynamic (PK-PD) model by Geng et al. [13],
which simulates cancer dynamics, strive to provide a realistic perspective on
actual medical processes. Simulated "observational" cancer growth data,
derived from the PK-PD model, is widely utilized in the literature [29, 4, 28,
32] for evaluation purposes. The model has been adapted to simulate the change
in tumor volume over time under the influence of different treatment options,
such as chemotherapy and radiation. Time-dependent confounding can be
incorporated into the model by expressing the probabilities of administering
chemotherapy and radiation as a function of tumor size [29].

In addition to the PK-PD model, Li et al. [28] evaluate the performance of
G-Net on longitudinal data, simulated using Heldt et al.’s [18] CVSim. CVSim
provides a mechanistic model of the human cardiovascular system and enables
the simulation of trajectories of outcome parameters such as the mean arterial
pressure (MAP) or the central venous pressure (CVP) under interventions such
as different administration strategies of fluid or vasopressors.

Moreover, Melnychuk et al. [32] evaluate their Causal Transformer on both
semi-synthetic and real-world datasets of ICU patient trajectories that are
based on the MIMIC-III dataset [21]. For their semi-synthetic data, they
combine real-world covariates from the MIMIC-extract by Wang et al. (2020)
[47] with simulated trajectories of control outcomes. Treated outcomes are
obtained by simulating synthetic binary treatments, incorporating confounding,
and applying those treatments to the control outcomes [32]. For their
experiments on real-world data, they use the same patient’s covariates from
MIMIC-III again and consider the effect of vasopressors and mechanical
ventilation on blood pressure. However, since counterfactual outcomes are not
available for real-world data, they can only report the performance in
predicting the factual outcomes.

The fundamental problem of causal inference and the resulting reliance on
(semi-) simulated data sets for comprehensive validation poses a major
challenge to developing models for ITE estimation. Although models such as
CVSim or the PK-PD model can offer valuable insights from a medical
standpoint, their data generation process is less complex with simple
assumptions, which could result in lower performance when compared to real-
world applications.

##  4 Future Outlook

###  4.1 Irregular Sampling and Missingness

All of the studies included in this work have shown great promise in
estimating ITE in discrete time-series data. Transforming multi-variate time-
series data to one with discrete time steps requires covariate alignment.
Since most time-series covariates are measured at irregular time-steps where
treatment is also administered at various time-steps, missing values are
inevitably produced in the data. Despite the increasing number of proposed
approaches for estimating ITEs and deconfounding the data, none of the
previous works have investigated the impact of missing values on
counterfactual predictions and model performance. With a wide variety of
methods proposed for handling time-series missingness with various underlying
assumptions on the nature of missingness [12, 7, 11], we believe that new
research directions should investigate the impact of such assumptions on the
ITE assumptions and models. This is particularly important when data
missingness can be related to the underlying patient’s health state, which can
lead to a lack of follow-up, making it relevant patient information for the
ITE estimation.

###  4.2 ITE and Dynamic Reinforcement Learning

Another line of work that receives increased attention from the ITE research
community is that of dynamic reinforcement learning (DRL)-based treatment
regimes. While the objectives may appear similar, DRL and ITE have different
underlying assumptions that are important to note. For instance, ITE makes
strong assumptions about the nature of the data concerning confounders and
treatment effects, which is not the case in DRL. DRL also assumes a Markovian
state-data generation mechanism [24], which is not necessarily required in
ITE. Understanding the differences in the related work in literature can help
one in understanding the limitations of integrating emerging works into real-
world problems.

## References

  * [1] Sarah C Anoke, Sharon-Lise Normand and Corwin M Zigler  “Approaches to treatment effect heterogeneity in the presence of confounding”  In _Statistics in medicine_ 38.15 Wiley Online Library, 2019, pp. 2797–2815
  * [2] Peter C Austin and Elizabeth A Stuart  “Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies”  In _Statistics in medicine_ 34.28 Wiley Online Library, 2015, pp. 3661–3679
  * [3] Ioana Bica, Ahmed Alaa and Mihaela Van Der Schaar  “Time series deconfounder: Estimating treatment effects over time in the presence of hidden confounders”  In _International Conference on Machine Learning_ , 2020, pp. 884–895  PMLR
  * [4] Ioana Bica, Ahmed M Alaa, James Jordon and Mihaela Schaar  “Estimating counterfactual treatment outcomes over time through adversarially balanced representations”  In _arXiv preprint arXiv:2002.04083_ , 2020
  * [5] Ioana Bica, Ahmed M Alaa, Craig Lambert and Mihaela Van Der Schaar  “From real-world patient data to individualized treatment effects using machine learning: current and future methods to address underlying challenges”  In _Clinical Pharmacology & Therapeutics_ 109.1 Wiley Online Library, 2021, pp. 87–100
  * [6] Nick Black  “Why we need observational studies to evaluate the effectiveness of health care”  In _Bmj_ 312.7040 British Medical Journal Publishing Group, 1996, pp. 1215–1218
  * [7] Wei Cao et al.  “Brits: Bidirectional recurrent imputation for time series”  In _Advances in neural information processing systems_ 31, 2018
  * [8] Nicholas C Chesnaye et al.  “An introduction to inverse probability of treatment weighting in observational research”  In _Clinical Kidney Journal_ 15.1 Oxford University Press, 2022, pp. 14–20
  * [9] Hugh A Chipman, Edward I George and Robert E McCulloch  “BART: Bayesian additive regression trees”  In _The Annals of Applied Statistics_ 4.1 Institute of Mathematical Statistics, 2010, pp. 266–298
  * [10] Jerome T Connor, R Douglas Martin and Les E Atlas  “Recurrent neural networks and robust time series prediction”  In _IEEE transactions on neural networks_ 5.2 IEEE, 1994, pp. 240–254
  * [11] Wenjie Du, David Côté and Yan Liu  “Saits: Self-attention-based imputation for time series”  In _Expert Systems with Applications_ 219 Elsevier, 2023, pp. 119619
  * [12] Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch and Stephan Mandt  “Gp-vae: Deep probabilistic time series imputation”  In _International conference on artificial intelligence and statistics_ , 2020, pp. 1651–1661  PMLR
  * [13] Changran Geng, Harald Paganetti and Clemens Grassberger  “Prediction of Treatment Response for Combined Chemo- and Radiation Therapy for Non-Small Cell Lung Cancer Patients Using a Bio-Mathematical Model”  In _Scientific Reports_ 7.1 Springer ScienceBusiness Media LLC, 2017  DOI: 10.1038/s41598-017-13646-z
  * [14] Shantanu Ghosh, Christina Boucher, Jiang Bian and Mattia Prosperi  “Propensity score synthetic augmentation matching using generative adversarial networks (PSSAM-GAN)”  In _Computer methods and programs in biomedicine update_ 1 Elsevier, 2021, pp. 100020
  * [15] Cory E Goldstein et al.  “Ethical issues in pragmatic randomized controlled trials: a review of the recent literature identifies gaps in ethical argumentation”  In _BMC medical ethics_ 19.1 BioMed Central, 2018, pp. 1–10
  * [16] Tobias Hatt and Stefan Feuerriegel  “Sequential deconfounding for causal inference with unobserved confounders”  In _arXiv preprint arXiv:2104.09323_ , 2021
  * [17] RJ Hayes and S Bennett  “Simple sample size calculation for cluster-randomized trials.”  In _International journal of epidemiology_ 28.2, 1999, pp. 319–326
  * [18] Thomas Heldt, Ramakrishna Mukkamala, George B Moody and Roger G Mark  “CVSim: An open-source cardiovascular simulator for teaching and research”  In _Open Pacing Electrophysiol. Ther. J._ 3, 2010, pp. 45–54
  * [19] Susan D Horn et al.  “Another look at observational studies in rehabilitation research: going beyond the holy grail of the randomized controlled trial”  In _Archives of Physical Medicine and Rehabilitation_ 86.12 Elsevier, 2005, pp. 8–15
  * [20] Fredrik Johansson, Uri Shalit and David Sontag  “Learning representations for counterfactual inference”  In _International conference on machine learning_ , 2016, pp. 3020–3029
  * [21] Alistair E.W. Johnson et al.  “MIMIC-III, a freely accessible critical care database”  In _Scientific Data_ 3.1 Springer ScienceBusiness Media LLC, 2016  DOI: 10.1038/sdata.2016.35
  * [22] Rita R Kalyani, Sherita H Golden and William T Cefalu  “Diabetes and aging: unique considerations and goals of care”  In _Diabetes Care_ 40.4 Am Diabetes Assoc, 2017, pp. 440–443
  * [23] Alison M Kim, Candace M Tingen and Teresa K Woodruff  “Sex bias in trials and treatment must end”  In _Nature_ 465.7299 Nature Publishing Group, 2010, pp. 688–689
  * [24] Matthieu Komorowski et al.  “The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care”  In _Nature medicine_ 24.11 Nature Publishing Group US New York, 2018, pp. 1716–1720
  * [25] Helena Chmura Kraemer  “Pitfalls of multisite randomized clinical trials of efficacy and effectiveness”  In _Schizophrenia Bulletin_ 26.3 Oxford University Press, 2000, pp. 533–541
  * [26] Milan Kuzmanovic, Tobias Hatt and Stefan Feuerriegel  “Deconfounding Temporal Autoencoder: estimating treatment effects over time using noisy proxies”  In _Machine Learning for Health_ , 2021, pp. 143–155  PMLR
  * [27] The Lancet  “Personalised medicine in the UK.”  In _Lancet (London, England)_ 391.10115, 2018, pp. e1
  * [28] Rui Li et al.  “G-net: a recurrent network approach to g-computation for counterfactual prediction under a dynamic treatment regime”  In _Machine Learning for Health_ , 2021, pp. 282–299  PMLR
  * [29] Bryan Lim  “Forecasting treatment responses over time using recurrent marginal structural networks”  In _Advances in neural information processing systems_ 31, 2018
  * [30] WA Lindsay, MM Murphy, DS Almghairbi and IK Moppett  “Age, sex, race and ethnicity representativeness of randomised controlled trials in peri-operative medicine”  In _Anaesthesia_ 75.6 Wiley Online Library, 2020, pp. 809–815
  * [31] Jared K Lunceford and Marie Davidian  “Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study”  In _Statistics in medicine_ 23.19 Wiley Online Library, 2004, pp. 2937–2960
  * [32] Valentyn Melnychuk, Dennis Frauen and Stefan Feuerriegel  “Causal transformer for estimating counterfactual outcomes”  In _International Conference on Machine Learning_ , 2022, pp. 15293–15329  PMLR
  * [33] Ashley I Naimi, Stephen R Cole and Edward H Kennedy  “An introduction to g methods”  In _International journal of epidemiology_ 46.2 Oxford University Press, 2017, pp. 756–762
  * [34] Simon J Newsome, Ruth H Keogh and Rhian M Daniel  “Estimating long-term treatment effects in observational data: A comparison of the performance of different methods under real-world uncertainty”  In _Statistics in medicine_ 37.15 Wiley Online Library, 2018, pp. 2367–2390
  * [35] James Robins  “A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect”  In _Mathematical modelling_ 7.9-12 Elsevier, 1986, pp. 1393–1512
  * [36] James Robins and Miguel Hernan  “Estimation of the causal effects of time-varying exposures”  In _Chapman & Hall/CRC Handbooks of Modern Statistical Methods_ ChapmanHall/CRC, 2008, pp. 553–599
  * [37] James M Robins, Miguel Angel Hernan and Babette Brumback  “Marginal structural models and causal inference in epidemiology”  In _Epidemiology_ JSTOR, 2000, pp. 550–560
  * [38] Paul R Rosenbaum and Donald B Rubin  “The central role of the propensity score in observational studies for causal effects”  In _Biometrika_ 70.1 Oxford University Press, 1983, pp. 41–55
  * [39] Donald B Rubin  “Causal inference using potential outcomes: Design, modeling, decisions”  In _Journal of the American Statistical Association_ 100.469 Taylor & Francis, 2005, pp. 322–331
  * [40] Robert William Sanson-Fisher, Billie Bonevski, Lawrence W Green and Cate D’Este  “Limitations of the randomized controlled trial in evaluating population-based health interventions”  In _American journal of preventive medicine_ 33.2 Elsevier, 2007, pp. 155–161
  * [41] Roosmarijn MC Schelvis et al.  “Evaluation of occupational health interventions using a randomized controlled trial: challenges and alternative research designs”  In _Scandinavian journal of work, environment & health_ JSTOR, 2015, pp. 491–503
  * [42] Kenneth F Schulz, Iain Chalmers, Richard J Hayes and Douglas G Altman  “Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials”  In _Jama_ 273.5 American Medical Association, 1995, pp. 408–412
  * [43] Uri Shalit, Fredrik D Johansson and David Sontag  “Estimating individual treatment effect: generalization bounds and algorithms”  In _International Conference on Machine Learning_ , 2017, pp. 3076–3085  PMLR
  * [44] RW Shaw  “Adverse long-term effects of oral contraceptives: a review”  In _British journal of obstetrics and gynaecology_ 94.8, 1987, pp. 724–730
  * [45] Charles A Stiller  “Centralised treatment, entry to trials and survival”  In _British journal of cancer_ 70.2 Nature Publishing Group, 1994, pp. 352–362
  * [46] Sheldon W Tobe, Diane Hua and Patrick Twohig  “Clinical practice guidelines”  Future Medicine, 2013
  * [47] Shirly Wang et al.  “MIMIC-Extract”  In _Proceedings of the ACM Conference on Health, Inference, and Learning_ ACM, 2020  DOI: 10.1145/3368555.3384469
  * [48] Tyler Williamson and Pietro Ravani  “Marginal structural models in clinical research: when and how to use them?”  In _Nephrology Dialysis Transplantation_ 32.suppl_2 Oxford University Press, 2017, pp. ii84–ii90
  * [49] Jinsung Yoon, James Jordon and Mihaela Schaar  “GANITE: Estimation of individualized treatment effects using generative adversarial nets”  In _International Conference on Learning Representations_ , 2018