# Exact and Asymptotic Tests for Sufficient Followup in Censored Survival Data

Ross Mallera, Sidney Resnickb and Soudabeh Shemehsavara,b aResearch School of
Finance, Actuarial Studies & Statistics, Australian National University,
<EMAIL_ADDRESS>
bSchool of Operations Research and Information Engineering, Cornell
University<EMAIL_ADDRESS>
cSchool of Mathematics, Statistics & Computer Sciences, University of Tehran,
<EMAIL_ADDRESS>(corresponding author)

###### Abstract

The existence of immune or cured individuals in a population and whether there
is sufficient follow-up in a sample of censored observations on their
lifetimes to be confident of their presence are questions of major importance
in medical survival analysis. So far only a few candidates have been put
forward as possible test statistics for the existence of sufficient follow-up
in a sample. Here we investigate one such statistic, $Q_{n}$, and give a
detailed analysis assuming independence between survival and censoring times.
We obtain an exact finite sample as well as asymptotic distributions for
$Q_{n}$, and use these to calculate the power of the test as a function of the
follow-up in the sample. A particularly useful finding is that the asymptotic
distribution of the test statistic is parameter free in the null case when
follow-up is insufficient. The methods are illustrated with detailed schematic
and real data sets, and the effect of dependence between survival and
censoring is considered via a copula model and simulations.

Keywords Sufficient follow-up; censored survival data; cure model.

##  1 Introduction

There is a large and growing interest in the analysis of censored survival
data from a population which may contain immune or cured individuals, that is,
individuals who will not experience the event of interest no matter how long
follow-up may be. A systematic formulation and treatment of this kind of
problem is in the book by Maller and Zhou (1996) which contains many practical
examples from medicine, criminology and various other fields, of this kind of
data. For other reviews and applications, see for example Amica and Van
Keilegom (2018), Escobar-Bach et al. (2021), Legrand and Bertrand (2019), Ma
(2009), Othus et al. (2012), Peng and Taylor (2014), Peng and Yu (2021).

In a sample of data of the kind mentioned, we have observations on the time to
an event of interest (we refer to them as “lifetimes”), possibly right-
censored, of individuals from a population which contains some who are
“susceptible” to suffering the event under consideration, and possibly also
some who are “immune to”, “cured” of it, or are “longterm survivors”. For
“susceptibles”, the issue is to infer properties of their lifetime
distribution from the sample. For “immunes”, the main questions of interest
are whether they are in fact present in the population, and if so in what
proportion. Herein, we concentrate on the first question: based on the sample
information, how confident can we be that immunes are in fact present in the
population? Currently developed methods for assessing this ultimately rely in
some way on the amount of follow-up in the sample.

We do not know whether a particular censored lifetime in the sample is from an
immune or cured individual (uncensored lifetimes are obviously not from
immunes); but, in aggregate, the presence of cured individuals may be
signalled by an interval of constancy of the Kaplan-Meier estimator (Kaplan
and Meier (1958) (KME)) at its right hand end; that is, the interval
containing the censored lifetimes exceeding the largest uncensored lifetime.
The length of that interval and the number of censored survival times larger
than the largest uncensored survival time are important indicators for the
presence of cured individuals, and for whether there is sufficient follow-up
in the sample to be confident of their presence.

Ways of testing for sufficient follow-up are still in a very undeveloped
state. One such test statistic, $Q_{n}$, is suggested in Maller and Zhou
(1994) and Maller and Zhou (1996), p.81. Since then, there have been only two
other definite approaches that we know of, namely those of Shen (2000) (his
statistic is denoted by $\widetilde{\alpha}_{n}$) and Klebanov and Yakovlev
(2007). We discuss these approaches further in Section 9, but otherwise
restrict discussion and analysis to $Q_{n}$.

The joint distribution of the largest uncensored and the largest survival time
in the sample is given in alternative ways in Maller and Resnick (2021) and
Maller et al. (2022). In the present paper we apply the foundational results
in Maller et al. (2022) to obtain exact finite sample as well as asymptotic
distributions for $Q_{n}$, which can be used to assess whether follow-up is
sufficient in a sample. The methods are illustrated with schematic and real
data sets.

An important additional point is that statistics such as $Q_{n}$ and
$\widetilde{\alpha}_{n}$ can be used not only to test for sufficient follow-up
but also to provide measures of how much follow-up there is in a sample. Both
these aspects are prominent in a paper by Liu et al. (2018) where testing for
and measurement of sufficient follow-up in the TCGA pan-cancer clinical data
resource are done on a very extensive scale in order to provide
recommendations to cancer researchers wishing to assess the adequacy of
clinical follow-up in a medical situation. Liu et al. (2018) processed follow-
up data files for 11,160 patients across 33 cancer types, calculating median
follow-up times as well as median times to event (or censorship) based on the
observed times for four endpoints (overall survival, disease-specific
survival, disease-free interval, or progression-free interval). They used
$Q_{n}$ and $\widetilde{\alpha}_{n}$ to classify all $33\times 4$ resulting
KMEs as having sufficient or insufficient follow-up (or noted cases in which
tests were inconclusive) in order to give endpoint usage recommendations for
each cancer type. The analyses we present here help to validate the
application of these tests in the data analysed in Liu et al. (2018).

##  2 Test Statistics for Sufficient follow-up

###  2.1 Notation and distributional setup

For the distributional results to follow we use the notation in Maller et al.
(2022), which should be read in conjunction with the present paper. We assume
a general independent censoring model (“the iid censoring model”) with right
censoring. A sample of size $n$ consists of observations on the sequence of
iid (independent and identically distributed) 2-vectors
$\big{(}T_{i}=T_{i}^{*}\wedge U_{i},C_{i}={\bf 1}(T_{i}^{*}\leq U_{i});\,1\leq
i\leq n\big{)}$. The $T_{i}^{*}$ with continuous cumulative distribution
function (cdf) $F^{*}$ on $[0,\infty)$ represent the times of occurrence of an
event under study, such as the death of a person, the onset of a disease, the
recurrence of a disease, the arrest of a person charged with a crime, the re-
arrest of an individual released from prison, etc. The $U_{i}$ with continuous
cdf $G$ on $[0,\infty)$ are censoring random variables, independent of the
$T_{i}^{*}$. In a sample of data from a population containing long-term
survivors we observe the random variables $T_{i}=T_{i}^{*}\wedge U_{i}$, these
being potential lifetimes censored at a limit of follow-up represented for
individual $i$ by the random variable $U_{i}$. The random variables
$C_{i}={\bf 1}(T_{i}^{*}\leq U_{i})$ are censor indicators. Let
$M(n):=\max_{1\leq i\leq n}T_{i}$ be the largest observed survival time and
let $M_{u}(n)$ be the largest observed uncensored survival time.

The censoring distribution $G$ of the $U_{i}$ is always assumed proper (total
mass 1), but we allow the possibility that the distribution $F^{*}$ of the
$T_{i}^{*}$ is improper. We assume $F^{*}$ to be of the form

$F^{*}(t)=pF(t),$ (2.1)

where $0<p\leq 1$ and $F$ is a proper distribution. We think of $F$ as being
the distribution of susceptible individuals in the population. Only
susceptibles can experience the event of interest and have a potentially
uncensored failure time. The remainder of the population is immune to the
event of interest or cured of it. The presence of immunes is signalled by a
value of $p<1$, in which case the distribution $F^{*}$ is improper, with total
mass $p$. Then $1-p$ is the proportion of immune or cured individuals in the
population. Observations on immunes are always censored; those on susceptibles
may or may not be according as the corresponding $T_{i}^{*}>U_{i}$ or not.

Let $\overline{F}^{*}(t)=1-F^{*}(t)$, $t\geq 0$, denote the survival function
(tail function) of $F^{*}$, and similarly for $\overline{F}$ and
$\overline{G}$. Let $\tau_{F^{*}}=\inf\\{t>0:F^{*}(t)=1\\}$ (with the inf of
the empty set equal to $\infty$) be the right extreme of the survival
distribution $F^{*}$, and similarly $\tau_{F}$ and $\tau_{G}$ are the right
extremes of $F$ and $G$. Let $H(t):=P(T_{1}\leq t)$ be the distribution of the
observed survival times $T_{i}=T_{i}^{*}\wedge U_{i}$, with tail
$\overline{H}(t)=1-H(t)=P(T_{i}^{*}\wedge
U_{i}>t)=\overline{F}^{*}(t)\overline{G}(t)$, $t\geq 0$, and right extreme
$\tau_{H}=\tau_{F^{*}}\wedge\tau_{G}$. We always have $H(\tau_{H})=1$,
$G(\tau_{G})=1$ and $F(\tau_{F})=1$. When $p=1$, so that $F^{*}\equiv F$,
$F^{*}$ has total mass 1 and $\tau_{F^{*}}=\tau_{F}$; when $p<1$ we have
$\tau_{F^{*}}=\infty$, and $\tau_{F}\leq\tau_{F^{*}}$, with the possibility
that $\tau_{F}<\tau_{F^{*}}$.

###  2.2 Test statistics and procedure

As test statistic for sufficient follow-up we focus on the statistic $Q_{n}$
proposed in Maller and Zhou (1994). This is defined as follows. Consider a
sample of size $n$ with all survival times necessarily in $[0,M(n)]$, a number
$N_{u}(n)$ of uncensored survival times, necessarily in $[0,M_{u}(n)]$, a
number $N_{c}^{<}(n)$ of censored survival times in $[0,M_{u}(n))$, and a
number $N_{c}^{>}(n)$ of censored survival times in $(M_{u}(n),M(n)]$, thus
with a total of $N_{c}(n)=N_{c}^{<}(n)+N_{c}^{>}(n)=n-N_{u}(n)$ censored
survival times in the sample. Set $\Delta_{n}:=2M_{u}(n)-M(n)$. As in Maller
and Zhou (1996), p.81 we define

$Q_{n}=\frac{1}{n}\\#\\{{\rm uncensored\ observations\ in}\
[\Delta_{n},M_{u}(n))\\}.$ (2.2)

(Note that we exclude $M_{u}(n)$ itself when counting the number of uncensored
observations greater than $\Delta_{n}$.) The statistic $Q_{n}$ is the
proportion of uncensored observations in the interval
$[2M_{u}(n)-M(n),M_{u}(n)]$, relative to the sample size $n$. It measures the
length of the interval exceeding $M_{u}(n)$ but in a proportional rather than
absolute way. A rationale for the definition (2.2) is given in Maller and Zhou
(1996), p.84.

The distribution of $Q_{n}$ was unavailable when Maller and Zhou (1996) was
written and had to be simulated to get quantiles. Our intention here is to get
exact formulae (in Theorem 4.1 below) for the distribution of $Q_{n}$ under
the iid censoring model. With these we can calculate asymptotic distributions
(in Section 5) and percentage points when estimates of $F$ and $G$ are made
from data (an example is in Section 8).

Our test procedure will be as follows. We have at hand survival data with
hypothesized cured individuals present and wish to test for sufficient follow-
up. This is specified in Maller and Zhou (1996), p.81, to be the parametric
condition $\tau_{F}\leq\tau_{G}$. (For a rationale for this condition, see
Sections 2.2 and 2.3 of Maller and Zhou (1996).) We proceed by assuming the
contrapositive hypothesis, $H_{0}:\tau_{G}<\tau_{F}$. If $H_{0}$ is true the
probability of seeing a large value of the test statistic $Q_{n}$ is small. So
we will reject $H_{0}$ and conclude that follow-up is sufficient if the
observed value of the test statistic exceeds a nominated quantile of its
distribution under $H_{0}$. A test based on large values of $Q_{n}$ will
reject the hypothesis of insufficient follow-up with probability approaching 1
as sample size tends to infinity; this follows from the asymptotic results in
Theorem 5.1. In order to understand the behaviour of $Q_{n}$, we first
consider its finite sample properties.

##  3 Understanding the sample properties of $Q_{n}$

The value of $Q_{n}$ depends in a complicated way on the numbers of censored
and uncensored observations, the way they happen to occur below or above
$M_{u}(n)$, and on the relative magnitudes of $M_{u}(n)$ and $M(n)$. In order
to calculate its distribution under the iid censoring model we need to
understand how it varies with these things. To do this we consider
hypothetical sample situations, vary the mentioned quantities and see how the
value of $Q_{n}$ changes.

We begin by considering possible values of $\Delta_{n}$. We always have
$\Delta_{n}=2M_{u}(n)-M(n)\leq 2M_{u}(n)-M_{u}(n)=M_{u}(n)$. Possible values
of $\Delta_{n}$ range from $\Delta_{n}=-M(n)$ if $M_{u}(n)=0$, equivalently,
if all observations are censored, to $\Delta_{n}=M_{u}(n)=M(n)$ if
$M_{u}(n)=M(n)$, equivalently, if the largest observation is uncensored. We
have $\Delta_{n}=0$ if it happens that $M_{u}(n)=M(n)/2$. Thus we may have
$\Delta_{n}<0$, $\Delta_{n}=0$, or $\Delta_{n}>0$. When $\Delta_{n}\leq 0$
then $[\Delta_{n},M_{u}(n))\supseteq[0,M_{u}(n))$ and (2.2) gives
$nQ_{n}=N_{u}(n)-1$. At the other extreme, the interval
$[\Delta_{n},M_{u}(n))$ may be empty, and this is certainly so when
$\Delta_{n}=M_{u}(n)$. Whenever this occurs we set $Q_{n}=0$.

Now think of the way $Q_{n}$ changes if we rearrange the conformation of the
censored observations less than or greater than $M_{u}(n)$, by keeping $M(n)$
and $N_{u}(n)>0$ fixed and varying $N_{c}^{<}(n)$ and $N_{c}^{>}(n)$. It helps
to visualise the various situations with schematic KME diagrams in the
different cases, as we show in Figures 1(a) – 1(d).

(a) All censored observations $>M_{u}(n)$

(b) 2 censored observations $>M_{u}(n)$

(c) 1 censored observation $>M_{u}(n)$

(d) Largest observation uncensored

Figure 1: Schematic KME diagrams

We start with an extreme case.
Case 1: $N_{c}^{<}(n)=0$, $N_{c}^{>}(n)>0$ (see Fig. 1(a)). In this
conformation all the censored observations in the sample form a level stretch
of the KME between $M_{u}(n)$ and $M(n)$. In this case $M_{u}(n)$ takes the
minimum possible value for the sample under this kind of rearrangement,
$M(n)-M_{u}(n)$ takes the maximum possible value,
$\Delta_{n}=2M_{u}(n)-M(n)=M_{u}(n)-(M(n)-M_{u}(n))$ takes the minimum
possible value, and $Q_{n}$ takes the maximum possible value under this kind
of rearrangement for the sample. We reject $H_{0}:\tau_{G}<\tau_{F}$ and
conclude there is sufficient follow-up if we observe large values of $Q_{n}$,
so this arrangement accords with our intuition that a (long) level stretch on
the KME between $M_{u}(n)$ and $M(n)$ indicates there is sufficient follow-up.
Case 2: $N_{c}^{<}(n)>0$, $N_{c}^{>}(n)>0$ (see Fig. 1(b)). As censored
observations are moved to the left of $M_{u}(n)$, $M_{u}(n)$ tends to increase
and $M(n)-M_{u}(n)$ tends to decrease (it cannot increase). So $\Delta_{n}$
will tend to increase and consequently $Q_{n}$ will tend to decrease. This
accords with our intuition that a decrease in the number of censored
observations above $M_{u}(n)$ and in the length of the level stretch of the
KME between $M_{u}(n)$ and $M(n)$ makes it less likely to reject $H_{0}$, the
hypothesis of insufficient follow-up.

Ultimately, continuing this process, we reach:
Case 3: $N_{c}^{<}(n)>0$, $N_{c}^{>}(n)=1$ (see Fig. 1(c)). The one censored
observation above $M_{u}(n)$ is $M(n)$ itself and $\Delta_{n}=2M_{u}(n)-M(n)$
will be close to or equal to $M_{u}(n)$. The interval $[\Delta_{n},M_{u}(n))$
is small and $Q_{n}$ is small, possibly equal to $0$ (this certainly occurs
when $\Delta_{n}=M_{u}(n)$). This accords with our intuition that a short
level stretch of the KME between $M_{u}(n)$ and $M(n)$ indicates via a small
value of $Q_{n}$ that there is insufficient follow-up.

In these scenarios, $Q_{n}$ decreases monotonically from a sufficient follow-
up situation to one with insufficient follow-up.

The actual values taken on by $Q_{n}$ in these scenarios depend on the
relative magnitudes of $M_{u}(n)$ and $M(n)$. The possibilities are as
follows. Note that since $N_{u}(n)>0$, we have $M_{u}(n)>0$.

(a) When $0<M_{u}(n)\leq\tfrac{1}{2}M(n)$, then $\Delta_{n}\leq 0$ and
$[\Delta_{n},M_{u}(n))\supseteq[0,M_{u}(n))$. In this case

$Q_{n}=\frac{1}{n}\\#\\{{\rm uncensored\ observations\ other\ than}\
M_{u}(n)\\}\ =\frac{N_{u}(n)-1}{n}.$

This is the largest value $Q_{n}$ can take for a given sample.

(b) When $\tfrac{1}{2}M(n)<M_{u}(n)<M(n)$, then $\Delta_{n}>0$ and the
interval $[\Delta_{n},M_{u}(n))$ contains, say, $k$ observations. We have
$k\geq 0$ and $k\leq n-1$ since there is at least one censored observation
greater than $M_{u}(n)$, namely, $M(n)$. So we can write

$Q_{n}=\frac{k}{n}=\frac{1}{n}\\#\\{{\rm uncensored\ observations\ in\
[\Delta_{n},M_{u}(n))\\}},$ (3.1)

where $k$ decreases from its maximum value when $M_{u}(n)$ is near
$\tfrac{1}{2}M(n)$, reaching 0 when $M_{u}(n)$ is near $M(n)$.

There are also two other extreme cases to consider.

(c) When $N_{c}^{>}(n)=0$, then $M_{u}(n)=M(n)$, and the largest observation
is uncensored (see Fig. 1(d)). Then $\Delta_{n}=M_{u}(n)$, the interval
$[\Delta_{n},M_{u}(n))$ is empty, and $Q_{n}=0$. Here the level stretch has
length 0 and the low $Q_{n}$ value correctly reflects sufficient follow-up.
(This case includes also the possibility that all observations are uncensored,
corresponding to $N_{u}(n)=n$, and $k=n$.) But this Case (c) means there is no
evidence of immunes and hence no issue of sufficient or insufficient follow-
up. We condition on the non-occurrence of this event when calculating the
distribution of $Q_{n}$.

(d) When $N_{u}(n)=0$, all observations are censored, and, formally,
$Q_{n}=0$. This anomalous or ambiguous case is of no interest and we condition
on its non-occurrence also, when calculating the distribution of $Q_{n}$.

Figure 2: Possible Values for $Q_{n}$

##  4 Finite Sample Distribution of $Q_{n}$

Given the discussion in the previous section, in calculating the distribution
of $Q_{n}$ we will eliminate the cases $M_{u}(n)=0$ and $M_{u}(n)=M(n)$, and
partition the event of interest, $\\{0<M_{u}(n)<M(n)\\}$, as
$\\{0<M_{u}(n)\leq\tfrac{1}{2}M(n)\\}\cup\\{\tfrac{1}{2}M(n)<M_{u}(n)<M(n)\\}$.
So we condition on $\\{M_{u}(n)=t,M(n)=x\\}$, where $0<t<x\leq\tau_{H}$, and
consider the cases $0<t\leq\tfrac{1}{2}x$ and $\tfrac{1}{2}x<t<x$ separately.

Equivalently, we consider Case A: $2t-x\leq 0$, and Case B:
$0<2t-x\leq\tau_{H}$, with $0<t<x\leq\tau_{H}$ in both cases. Since we know
the joint distribution of $M(n)$ and $M_{u}(n)$ from Theorem 2.3 of Maller et
al. (2022), we can integrate to obtain the distribution of $Q_{n}$ conditional
on $\\{0<M_{u}(n)<M(n)\\}$. One such calculation is carried out in the proof
of Theorem 4.1, and a large-sample version is in Theorem 5.1.

We set out some further preliminaries to Theorem 4.1. Recall the formula (3.1)
for $Q_{n}$. Throughout we keep $n>2$, $0<t<x\leq\tau_{H}$ and $1\leq r\leq
n-1$, and will begin by conditioning on the event
$\\{M_{u}(n)=t,M(n)=x,N_{c}^{>}(M_{u}(n))=r\\}$. We need some separate
notation in Cases A and B. For Case A define

$\pi^{A}(t):=\frac{P(0<T_{1}^{*}\leq t,T_{1}^{*}\leq U_{1})}{P(T_{1}^{*}\wedge
U_{1}\leq t)}=\frac{\int_{0}^{t}\overline{G}(y){\rm d}F^{*}(y)}{H(t)}$ (4.1)

(which does not depend on $x$) and for Case B define

$\pi^{B}(t,x):=\frac{P(2t-x<T_{1}^{*}\leq t,T_{1}^{*}\leq
U_{1})}{P(T_{1}^{*}\wedge U_{1}\leq
t)}=\frac{\int_{2t-x}^{t}\overline{G}(y){\rm d}F^{*}(y)}{H(t)}.$ (4.2)

Define also the probability

$p_{c}^{>}(t,x)=\frac{\int_{y=t}^{x}\overline{F}^{*}(y){\rm
d}G(y)}{\int_{y=t}^{x}\overline{F}^{*}(y){\rm d}G(y)+H(t)},$ (4.3)

and let

$\rho^{A}(t,x):=(1-p_{c}^{>}(t,x))\pi^{A}(t)\ {\rm and}\
\rho^{B}(t,x):=(1-p_{c}^{>}(t,x))\pi^{B}(t,x).$ (4.4)

Define integers $I_{n}^{<}:=\\{i\in\mathbb{N}:T_{i}<M_{u}(n)\\}$ and let
$\sigma^{<}$ be the smallest $\sigma$-field making $(T_{i},C_{i})_{i\in
I_{n}^{<}}$ measurable. Likewise let
$I_{n}^{>}:=\\{i\in\mathbb{N}:T_{i}>M_{u}(n)\\}$ and let $\sigma^{>}$ be the
smallest $\sigma$-field making $(T_{i},C_{i})_{i\in I_{n}^{>}}$ measurable. A
key result from Theorem 2.1 and Corollary 5 of Maller et al. (2022) is that,
conditional on the event $\\{M_{u}(n)=t,M(n)=x,N_{c}^{>}(M_{u}(n))=r\\}$, or,
equivalently, conditional on the event
$\\{M_{u}(n)=t,N_{c}^{>}(M_{u}(n))=r\\}$, the $\sigma$-fields $\sigma^{<}$ and
$\sigma^{>}$ are independent, and the conditional probability of an event
$A^{<}$ in $\sigma^{<}$ can be calculated by substituting truncated rvs
$(T_{i}(t))$ having the distribution of $T_{i}$ given $T_{i}\leq t$ for the
$(T_{i})$ in $A^{<}$.

With this setup we can now state Lemma 4.1. (Proofs of the lemma and the
subsequent Theorems 4.1 and 5.1 are in the supplementary material.)

######  Lemma 4.1.

Part (i): We have for $1\leq r\leq n-1$, $0<t<x\leq\tau_{H}$,

$P\big{(}N_{c}^{>}(M_{u}(n))=r\big{|}M_{u}(n)=t,M(n)=x\big{)}=P\big{(}Bin(n-2,\,p_{c}^{>}(t,x))=r-1\big{)}.$
(4.5)

(With the lefthand side taken as 0 when $r=0$).

Part (ii): For $0\leq k\leq n-2$,

$P\big{(}nQ_{n}=k\big{|}M_{u}(n)=t,M(n)=x\big{)}=P\big{(}Bin(n-2,\,\rho(t,x))=k\big{)},$
(4.6)

where $\rho(t,x)=\rho^{A}(t,x)$ in Case A and $\rho(t,x)=\rho^{B}(t,x)$ in
Case B (see (4.4)).

We need one more formula: by Eq. (2.14) of Maller et al. (2022) we have

$\displaystyle P_{n}({\rm d}t,{\rm d}x):=P\big{(}M_{u}(n)\in{\rm
d}t,M(n)\in{\rm d}x\big{)}$ (4.7)
$\displaystyle=n(n-1)\Big{(}\int_{y=t}^{x}\overline{F}^{*}(y){\rm
d}G(y)+H(t)\Big{)}^{n-2}\,\overline{G}(t){\rm
d}F^{*}(t)\,\overline{F}^{*}(x){\rm d}G(x).\ \ $ (4.8)

Next we can state Theorem 4.1.

######  Theorem 4.1.

Assume the iid censoring model in Subsection 2.1. Then for $n>2$,
$k=0,1,2,\ldots,n-2$,

$P\big{(}nQ_{n}=k\big{|}0<M_{u}(n)<M(n)\big{)}=\frac{A_{n}(k)+B_{n}(k)}{D_{n}},$
(4.9)

where

$A_{n}(k)=\int_{t=0}^{\tau_{H}/2}\int_{x=2t}^{\tau_{H}}P\big{(}Bin(n-2,\,\rho^{A}(t,x))=k\big{)}P_{n}({\rm
d}t,{\rm d}x)$ (4.10)

and

$B_{n}(k)=\Big{[}\int_{t=0}^{\tau_{H}/2}\int_{x=t}^{2t}+\int_{t=\tau_{H}/2}^{\tau_{H}}\int_{x=t}^{\tau_{H}}\Big{]}P\big{(}Bin(n-2,\,\rho^{B}(t,x))=k\big{)}P_{n}({\rm
d}t,{\rm d}x)$ (4.11)

(recall (4.7) for $P_{n}({\rm d}t,{\rm d}x)$.) The denominator in (4.9) is

$\displaystyle D_{n}$ $\displaystyle=$ $\displaystyle
P\big{(}0<M_{u}(n)<M(n)\big{)}$ (4.12) $\displaystyle=$ $\displaystyle
1-\left(\int_{t=0}^{\tau_{H}}\overline{F}^{*}(z){\rm
d}G(z)\right)^{n}-n\int_{t=0}^{\tau_{H}}H^{n-1}(t)\overline{G}(t){\rm
d}F^{*}(t).$ (4.13)

###  4.1 Probability mass functions of $Q_{n}$

Figure 3 shows graphs of the probability mass functions (pmfs) of $nQ_{n}$
calculated from (4.9) for various scenarios with $F$ exponential, $G$ uniform,
and $n=50,100,150$. For small $n$ the pmfs are bimodal, reflecting the two
components on the RHS of (4.9). The bimodality is least prominent when
censoring is heavy and disappears altogether as $n\to\infty$. The pmfs in
Figure 3 are consistent with those obtained by simulation in Section 4.3 of
Maller and Zhou (1996). The tables in Maller and Zhou (1996) are based on
exponential survival distributions but remain relevant also for a certain
scale family of distributions; see the Supplement for details.

Figure 3: Probability mass functions for $nQ_{n}$. $F=exp(1)$, $p=0.8$.
Left column: $G=U[0,3]$; Right column: $G=U[0,6]$.
Top, middle, bottom row: $n=50,100,150$.

We mentioned in connection with Liu et al. (2018) in Section 1 that $Q_{n}$
can be used not only to test for sufficient follow-up but also to provide a
measure of how much follow-up there is in a sample. In this respect the
bimodality evident in some of the pmfs in Figure 3 is an unsatisfactory
feature but as we pointed out it disappears as $n\to\infty$ and samples of
survival data are often of many thousands of individuals, as is the case in
Liu et al. (2018).

We go on to give the asymptotic distribution of $Q_{n}$ in the next section.

##  5 Asymptotic Distribution of $Q_{n}$

In this section we give the large sample distribution of $Q_{n}$ in situations
both of insufficient (the main case of interest) and sufficient follow-up. We
assume $G$ has a finite right endpoint $\tau_{G}$ and $\overline{G}$ behaves
linearly near $\tau_{G}$. We also impose mild regularity conditions on $F$.
These conditions are satisfied when $G$ is uniform on $[0,\tau_{G}]$ and $F$
is exponential, for example. The asymptotic distribution of $Q_{n}$ is shown
to be geometric when $\tau_{G}<\tau_{F}$ and normal when $\tau_{F}<\tau_{G}$,
under these conditions. Our main result for this section is:

######  Theorem 5.1.

[Asymptotic distribution of $Q_{n}$] Assume the iid censoring model and
$0<p\leq 1$ throughout. We have the following limiting distributions in cases
of interest.
Case 1: Assume $\tau_{G}<\tau_{F}\leq\infty$, so $\tau_{H}=\tau_{G}<\infty$.
Suppose also

$\overline{G}(\tau_{G}-z)=a_{G}(1+o(1))z\ {\rm as}\ z\downarrow 0,$ (5.1)

for a constant $a_{G}>0$, and, in addition, $F$ has a density in a
neighbourhood of $\tau_{G}$ which is positive and continuous at $\tau_{G}$.
Then

$\lim_{n\to\infty}P\big{(}nQ_{n}=k\big{)}=\frac{1}{4}\Big{(}\frac{3}{4}\Big{)}^{k},\
k=0,1,2,\ldots,$ (5.2)

so $nQ_{n}$ is asymptotically geometric with parameter $1/4$.

Case 2: Assume $\tau_{F}<\tau_{G}<2\tau_{F}$, so that
$\tau_{H}=\tau_{F}<\infty$. Suppose also that (5.1) holds and in addition

$\overline{F}(\tau_{F}-z)=a_{F}(1+o(1))z,$ (5.3)

for a constant $a_{F}>0$. Then as $n\to\infty$

$\frac{\sqrt{n}(Q_{n}-\nu^{B})}{\sqrt{\nu^{B}(1-\nu^{B})}}\stackrel{{\scriptstyle\mathrm{D}}}{{\longrightarrow}}N(0,1),$
(5.4)

where the parameter
$\nu^{B}=p\int_{2\tau_{F}-\tau_{G}}^{\tau_{F}}\overline{G}(y)\,{\rm
d}F(y)/(1-p\overline{G}(\tau_{F}))\in(0,1)$.
Case 3: Assume $2\tau_{F}<\tau_{G}<\infty$, so that
$\tau_{H}=\tau_{F}<\infty$. Suppose also that (5.1) and (5.3) hold. Then as
$n\to\infty$

$\frac{\sqrt{n}(Q_{n}-\nu^{A})}{\sqrt{\nu^{A}(1-\nu^{A})}}\stackrel{{\scriptstyle\mathrm{D}}}{{\longrightarrow}}N(0,1),$
(5.5)

where $\nu^{A}=p\int_{0}^{\tau_{F}}\overline{G}(y)\,{\rm
d}F(y)/(1-p\overline{G}(\tau_{F}))$.

Remarks. (i) Case 1 with $\tau_{G}<\tau_{F}$ is a situation of insufficient
follow-up, and in it $nQ_{n}$ has asymptotically a finite nondegenerate limit
(a geometric rv). Hence in this situation
$Q_{n}\stackrel{{\scriptstyle\mathrm{P}}}{{\longrightarrow}}0$ as
$n\to\infty$, showing that the hypothesis of insufficient follow-up will be
accepted in large samples (with probability approaching 1 as $n\to\infty$)
when it is true. When follow-up is sufficient, i.e, Cases 2 and 3, $Q_{n}$ is
ultimately normally distributed around positive levels $\nu^{A}$ or $\nu^{B}$
in large samples, and, depending on sample size, the hypothesis of
insufficient follow-up will be rejected, as it should be. The specific
formulae for the distributions in (5.2), (5.4) and (5.5), enable the power
calculations presented in the next section.

(ii) We remark that conditions (5.1) and (5.3) are special cases of those
imposed in Theorem 3.1 of Maller et al. (2022), where more generally, a
regularly varying function is allowed in place of the linear factors in (5.1)
and (5.3). However our present result is general enough for wide
applicability.

##  6 Power of the $Q_{n}$ Test

In this section we use the asymptotic distributions of $Q_{n}$ found under the
assumptions (5.1) and the density condition on $F$ in Section 5, to calculate
the power of the $Q_{n}$ test as the parameter $\tau_{G}$, reflecting the
amount of follow-up, changes. In view of (5.2), it is more convenient to use
$nQ_{n}$ than $Q_{n}$.

We proceed by calculating the 95-th quantile $K_{0.95}$ of the asymptotic
distribution of $nQ_{n}$ from (5.2), assuming the hypothesis
$H_{0}:\tau_{G}<\tau_{F}$ (insufficient follow-up) is true. From (5.2) we can
find $K_{0.95}$ explicitly as

$K_{0.95}=K_{0.95}(p,\tau_{G})=\frac{\log(0.05)}{\log(3/4)}-1=9.41.$ (6.1)

Thus, under $H_{0}$, we have $P(nQ_{n}>K_{0.95})\approx 0.05$, for large $n$.
Then we successively increase $\tau_{G}$ above $\tau_{F}$, hence in the region
of the alternate hypothesis, and use (5.4) and (5.5) to calculate the
corresponding values of $P(nQ_{n}>K_{0.95})$. Thus when
$\tau_{F}<\tau_{G}<2\tau_{F}$, according to (5.4) we set

$\displaystyle P(nQ_{n}>K_{0.95})$ $\displaystyle=$ $\displaystyle
P\left(\frac{nQ_{n}-n\nu^{B}}{\sqrt{n\nu^{B}(1-\nu^{B})}}>\frac{K_{0.95}-n\nu^{B}}{\sqrt{n\nu^{B}(1-\nu^{B})}}\right)$
(6.2) $\displaystyle\approx$ $\displaystyle
P\left(N(0,1)>\frac{K_{0.95}-n\nu^{B}}{\sqrt{n\nu^{B}(1-\nu^{B})}}\right),$
(6.3)

where $\nu^{B}=p\int_{2\tau_{F}-\tau_{G}}^{\tau_{F}}\overline{G}(y)\,{\rm
d}F(y)/(1-p\overline{G}(\tau_{F})))$; and when $2\tau_{F}<\tau_{G}$, according
to (5.5) we replace $\nu^{B}$ in (6.2) by
$\nu^{A}=p\int_{0}^{\tau_{F}}\overline{G}(y)\,{\rm
d}F(y)/(1-p\overline{G}(\tau_{F}))$.

Using $\nu$ to denote $\nu^{A}$ or $\nu^{B}$ as appropriate, we will use the
approximation (6.2) only when $n\nu$ is large, and take the function of
$\tau_{G}$ defined by

$P(\tau_{G};\nu):=P\left(N(0,1)>\min\left(\frac{K_{0.95}-n\nu}{\sqrt{n\nu(1-\nu)}},1.58\right)\right)$
(6.4)

as an approximation to the power of the test. Keep $\tau_{G}>\tau_{F}$, and,
at first, $\tau_{F}<\tau_{G}<2\tau_{F}$. As $\tau_{G}$ increases above
$\tau_{F}$, $\nu^{B}$ increases and $P(\tau_{G};\nu^{B})$ increases. (Notice
that $K_{0.95}$ no longer depends on $\tau_{G}$ for values of
$\tau_{G}>\tau_{F}$.) When $\nu^{B}=K_{0.95}/n$ then $P(\tau_{G};\nu^{B})$
reaches $0.50$, and once $\tau_{G}$ reaches $2\tau_{F}$ then
$\nu^{B}=p\int_{0}^{\tau_{F}}\overline{G}(y)\,{\rm
d}F(y)/(1-p\overline{G}(\tau_{F}))$. For $\tau_{G}$ values greater than this
$\nu^{B}$ is replaced in (6.4) by
$\nu^{A}=p\int_{0}^{\tau_{F}}\overline{G}(y)\,{\rm
d}F(y)/(1-p\overline{G}(\tau_{F}))$ and we note that $\nu^{A}=\nu^{B}$ at the
transition. For larger values of $\tau_{G}$, $P(\tau_{G}:\nu^{A})$ stays
constant at a value which approaches 1 as $n\to\infty$.

In summary, the power function of the test appears to behave very well. We
assume for illustration a sample size of $n=100$, for $G$ a
Uniform$[0,\tau_{G}]$ distribution, and for $F$ a unit exponential
distribution truncated at a finite value $\tau_{F}=5$. Since the probability
in the tail of $F$ above 5 is less than $0.01$, this is effectively assuming a
unit exponential distribution for susceptible lifetimes. A graph of
$P(\tau_{G}:\nu)$ for these parameter values is in Fig. 4.

Figure 4: Power as a function of $\tau_{G}$ for $Q_{n}$, with $F\sim exp(1)$,
truncated at $\tau_{F}=5$; $G\sim[0,\tau_{G}]$; $n=100$.

##  7 Dependent Censoring

The distributional results derived so far have been based on the assumption of
independence between the times of occurrence of the event under study and the
censoring mechanism. This assumption may not be tenable in some situations and
there have been a number of studies where it has been relaxed. See for example
the competing dependent risks of leukaemia relapse and graft versus host
disease analysed in Kalbfleisch and Prentice (2003) and Kovar et al. (2018).
In order to assess the robustness of our results to departures from
independence we consider the distribution of $Q_{n}$ in a model where there is
dependence between the survival and censoring distributions.

We assume a sample consists of observations on the 2-vectors

$\big{(}T_{i}=T_{i}^{*}\wedge U_{i},\ C_{i}={\bf 1}(T_{i}^{*}\leq
U_{i});\,1\leq i\leq n\big{)},$

where now the $T_{i}^{*}$ and $C_{i}$ are dependent with a joint continuous
distribution $H$ having marginal distributions $F^{*}$ and $G$ on
$[0,\infty)$. We model the dependence between $F^{*}$ and $G$ using a copula
to connect the marginal distributions with the joint distribution.

Numerous copulas are defined and described in Nelsen (2006), to which we refer
for background. For our bivariate setup we have 2 uniform random variables
$W_{1},W_{2}$, whose joint distribution function is specified as

$J(w_{1},w_{2},\theta):=P(W_{1}\leq{w_{1}},W_{2}\leq{w_{2}}),$ (7.1)

for a copula parameter $\theta$ which quantifies the dependence between
$W_{1}$ and $W_{2}$. We restrict our discussion to the class of Archimedean
copulas which contains many subfamilies capable of representing different
dependency structures. The distribution function of an Archimedean copula is
written as:

$J(w_{1},w_{2})=\Phi^{-1}\big{(}\Phi(w_{1})+\Phi(w_{2})\big{)},\ 0\leq
w_{1},w_{2}\leq 1,$ (7.2)

where the function $\Phi$ is the generator function  of the copula.

We consider two generators which give rise to two Archimedean copulas: the
Frank and Ali-Mikhail-Haq (AMH) copulas. Each copula has an analytical
expression that links its parameters to its related Kendall $\tau$ as a
measure of association (Salvadori et al. (2007)).

The Frank copula (Frank (1979)) has generator

$\Phi(t)=-\log\frac{e^{-\theta
t}-1}{e^{-\theta}-1},\;\;\;\theta\in\mathbb{R}\setminus\\{0\\},$

giving rise to the copula function

$J_{{\rm Frank}}(w_{1},w_{2})=-\frac{1}{\theta}\log\Big{(}1+\frac{(e^{-\theta
w_{1}}-1)(e^{-\theta_{2}}-1)}{e^{-\theta}-1}\Big{)}.$

The Ali-Mikhail-Haq copula (Ali et al. (1978)) has generator

$\Phi(t)=\log\Big{(}\frac{1-\theta(1-t)}{t}\Big{)},\;\;\;\;\theta\in[-1,1],$

giving rise to the copula function

$C_{{\rm AMH}}(w_{1},w_{2})=\frac{w_{1}w_{2}}{1-\theta(1-w_{1})(1-w_{2})}.$

We proceed by assuming that $H$ is a bivariate distribution with specified
continuous marginals $F^{*}$ and $G$. By virtue of Sklar’s theorem (Sklar
(1959)), $H$ can be expressed in a unique way via a 2-copula $J$. In order to
simulate an observation on $(T_{i}^{*},U_{i})\sim H$, it is sufficient to
simulate a vector $(W_{1},W_{2})\sim J$ with values $w_{1}$ and $w_{2}$ where
the r.v.’s $W_{1}$ and $W_{2}$ are uniform on $[0,1]$. Then

$t^{*}=F^{*,\leftarrow}(w_{1}),\;\;u=G^{\leftarrow}(w_{2}),$

is an observation on $(T^{*},U)$ having the required joint distribution. (see
Salvadori et al. (2007), Appendix A).

For our robustness analysis we simulated samples of size $n=50,100,150$, from
a $J$ based on the Frank and AMH copulas for various values of $\theta$. We
took $F^{*}=pF$, where $F$ is exponential with parameter 1, $G=U[0,6]$ and
$p=0.8$. In each sample we calculated the value of $Q_{n}$ and repeated this
$N=10000$ times to draw up the pmfs of $Q_{n}$ (Figures 5 and 6). In each
figure the pmf for $\theta=0$ corresponds to independence. Viewing from the
centre panel left we see that introducing negative dependence tends to
concentrate the mass near small values of $Q_{n}$; viewing from the centre
panel right shows that posiative dependence tends shifts the pmfs closer to
normal. Percentage points calculated from these distributions could be used
for correction if dependence is assumed or detected in a sample.

Figure 5:  Probability mass functions for $nQ_{n}$ with Frank copula for
dependence. $F=exp(1)$, $p=0.8$, $G=U[0,6]$. Top, middle, bottom panel:
$n=50,100,150$. Left to right: $\theta=300,6,0,-6,-300$.

Figure 6:  Probability mass functions for $nQ_{n}$ with AMH copula for
dependence. $F=exp(1)$, $p=0.8$, $G=U[0,6]$. Top, middle, bottom panel:
$n=50,100,150$. Left to right: $\theta=1,0.5,0,-0.5,-1$.

##  8 Data Example

We calculated KMEs for the survival and censoring distributions from data for
glioma (brain cancer) patients contained in the SEER (2019) database (Figures
7(a) and 7(b)). This is observational rather than from a randomised clinical
trial but we use it here just to illustrate how consideration of followup with
the $Q_{n}$ test can add value to an analysis of censored survival data.

The data was subdivided into two classes: Type 9380 (4248 patient records, of
which 2075 are censored) and Other Types111Types 9381, 9382, 9383, 9384, 9392,
9401, 9430, 9451, 9440, 9441, 9442, as used in Yang et al. (2018). (54375
patient records, 20482 censored) according to the SEER classification scheme.

(a) Survival KMEs for Glioma Types with fitted generalised gamma
distributions.

(b) Censor KMEs for Glioma Types with fitted generalised gamma distributions.

The maximum values of the KMEs are 0.68 for Type 9380 and 0.80 for Other
Types, suggesting cured probabilities of 0.32 for Type 9380 and 0.20 for Other
Types. These are significantly different from 0 based on Greenwood’s formula
for the variance of the KME, or using the tables in Maller and Zhou (1996).
But is followup sufficient for us to be confident in concluding cured
components? For Type 9380 we note a level stretch at the end of the KME at 503
months, with the largest uncensored observation at 435 months. Measuring back
$503-435=68$ months we come to 367 months, and the number of uncensored
observations in $[367,435)$ is found to be 3. So $nQ_{n}=3$ for this data.
Since the sample size is large we can use Case 1 of Theorem 5.1 to calculate
the p-value of the statistic for testing $H_{0}:\tau_{G}<\tau_{F}$ as
$(3/4)^{3}=0.42$ which is far from significance. So we do not reject $H_{0}$
and decide that followup is insufficient for this data. The same conclusion
follows using the tables in Maller and Zhou (1996). We can see that, despite
apparent levellings in Figure 7(a), the curves continue to rise, with late
deaths still occurring up till 400 months after diagnosis.

This test is nonparametric. But having failed to reject
$H_{0}:\tau_{G}<\tau_{F}$, Cases 2 and 3 of Theorem 5.1 become relevant, and
we can also consider the possibility that $\tau_{G}=\tau_{F}=\infty$. The
latter case is not included in Theorem 5.1 as the corresponding result
requires assumptions on the asymptotic behaviour of $(M_{u}(n),M(n))$, but it
can be shown that $Q_{n}$ is asymptotically normal under reasonable
assumptions. In order to apply Theorem 5.1, estimates of the distributions of
$F$ and $G$ are needed. Generalised gamma distributions with a cured component
as discussed in Jackson (2016) and Amdahl (2020)222For alternative
nonparametric fitting see López-Cheda et al. (2021). fit the data quite well
(Figures 7(a) and 7(b)) and can be used for further analysis. Knowledge of the
exact distribution allows for simulation via MCMC, for example.

For the Other glioma types the value of $nQ_{n}$ is 13 and the p-value using
Case 1 of Theorem 5.1 is 0.02. For this data we reject
$H_{0}:\tau_{G}<\tau_{F}$ and decide that followup is sufficient. There is
only a small level stretch at the end of the KME, but the conformation of
censored and uncensored observations mean that it is significant. Further
details of the data and analysis are in the Supplement.

This brief discussion is not meant to be a substantial analysis of this data,
at all, but it highlights the valuable information present in the observations
at the right hand end of a KME, and how the $Q_{n}$ statistic can guide us in
interpreting it.

##  9 Discussion

Calculation of exact distributions (under the iid censoring model) makes
unnecessary the need for simulations of percentage points, though in practice
the unknown distributions must be estimated or postulated. This includes
making assumptions about their right hand endpoints, and, especially, whether
they are finite or not.

Most practical is to assume $\tau_{G}<\infty$ since observation must always
cease at some finite point. In many cases the assumption $\tau_{F}<\infty$ may
also be natural. Certainly in real survival data no individual lives forever,
but we would have $\tau_{F}=\infty$ for example when studying the occurrence
of an infectious disease where an immune individual would never contract the
disease no matter how long the follow-up. (For just such an analysis with
children immune to malaria, see Cairns et al. (2013).) Regardless of the
situation, in modelling exercises it is not uncommon to use an exponential,
Weibull, lognormal, or Gumbel, with infinite right endpoints, as the lifetime
distribution. In doing so we accept that the probability of seeing an
extremely long lifetime under the assumed model is negligible, so the
theoretical approximation is good enough for practical purposes.

It is natural to base tests for sufficient follow-up on the length of the
interval $(M_{u}(n),M(n)]$, or the number of censored survival times larger
than the largest uncensored survival time, or some combination or variant of
these. Kaplan-Meier plots provide very strong intuition in this respect; see
for example the very evocative plots in Powles et al. (2021), who are
expressly concerned with “determin(ing) which patients $\ldots$ are cured
after surgery.”

As other test statistics we might use the difference between the extremes,
$M(n)-M_{u}(n)$, or a standardised version of this such as
$R_{n}=1-M_{u}(n)/M(n)$, which is in $(0,1)$. Formulae for their distributions
are in Maller et al. (2022). These variables measure the absolute or relative
length of the level stretch of the KME rather than a proportion of
observations related to them, as $Q_{n}$ does. At present their properties
remain to be investigated in detail. We note that they, like $Q_{n}$, are very
sensitive to the occurrence of one or a few failures in the righthand end of
the KME. This is a robustness issue such as has to be addressed in any
statistical analysis. A test for outliers in the iid model is in Maller and
Zhou (1994). The Shen (2000) statistic $\widetilde{\alpha}_{n}$ is suggested
by similar arguments to the way $Q_{n}$ was obtained in Maller and Zhou
(1996), though the rationale seems not so obvious and the sample properties
not so clear as for $Q_{n}$. Shen (2000) does not give a formula for the
distribution of $\widetilde{\alpha}_{n}$. We expect that one could be obtained
as a modification of the way we calculated the distribution of $Q_{n}$ in
Theorem 4.1, but the computations are more difficult. Shen reports, based on
some limited simulations, that $\widetilde{\alpha}_{n}$ sometimes performed
better than $Q_{n}$ in terms of Type 1 error and power. A more extensive
investigation of this is warranted.

Klebanov and Yakovlev (2007) give a succinct overview of the $Q_{n}$ statistic
as presented in Maller and Zhou (1996) and discuss some of its properties.
They draw attention to a perceived deficiency of the statistic, as follows.
Suppose follow-up in a sample were hypothetically extended beyond what’s
currently there. It’s possible then that a long-lived susceptible individual
with currently censored lifetime may die during the extended follow-up period,
and that the $Q_{n}$ value calculated on the extended sample then decreases
from its former value, possibly even to 0. Klebanov and Yakovlev see this non-
monotone behavior as problematic. But there is really no contradiction here.
On the extended sample, with its late failure, the new, low, $Q_{n}$ is
correctly registering that there is insufficient follow-up. If in this
hypothetical situation we continue to increase follow-up, susceptible
individuals will continue to die (all do, eventually), while those who are
immune will remain so, and with extended follow-up. The $Q_{n}$ value may well
fluctuate, but eventually only cured individuals will be left and the
continued follow-up will give rise to increasing values of $Q_{n}$, till it
reaches its maximum value for the sample. At each stage $Q_{n}$ is correctly
(according to its constitution) indicating the extent of follow-up. There is
no reason why $Q_{n}$ should be monotone in a hypothetical situation of
increasing follow-up.

Nevertheless the quite different approach in Klebanov and Yakovlev (2007)
provides a potentially useful perspective on the problem. Unfortunately
however their proposed statistic is technically complex and very non-intuitive
and its application would likely be restricted to specialist statisticians,
whereas an approach based on the length of the level stretch at the end of the
KME is highly visible and interpretable, and easily understood by
practitioners. Shen’s statistic is also constructed in this way. That there is
a need and a desire for a summary statistic with these properties is well
exemplified by the Liu et al. (2018) analysis.

Understanding how $Q_{n}$ depends on sample properties of censored data, and
the formulae for the exact and asymptotic distributions of $Q_{n}$ we have
obtained, open the way to its more general use in the analysis of survival
data with immune or cured individuals. We note that under
$H_{0}:\tau_{G}<\tau_{F}$, the hypothesis of insufficient follow-up, with some
reasonable side conditions, the asymptotic distribution of $Q_{n}$ is
completely non-parametric (cf. (5.2)). Future directions of research could
usefully include issues of sufficient follow-up in competing risks analysis,
and in multivariate survival analysis with cured individuals. For the latter,
see Chatterjee and Shih (2001), Chatterjee and Shih (2003) and Coelho-Barros
et al. (2016).

Shen (2000) and Klebanov and Yakovlev (2007) quote Maller and Zhou (1996) to
the effect that $Q_{n}$ is by no means the last word on the subject, and this
is worth stressing again here. Having formulae for the exact and asymptotic
distributions of $Q_{n}$ in the iid censoring model is a big step forward, but
there is much still to be explored in the analysis of sufficient follow-up and
cure models in general.

Acknowledgments We are pleased to thank Muzhi Zhao for helpful comments.

## References

  * Ali et al. (1978) M.M. Ali, N. Mikhail, and M. Haq.  A class of bivariate distributions including the bivariate logistic.  _Journal of Multivariate Analysis_ , 8:405–412, 1978.
  * Amdahl (2020) J. Amdahl.  flexsurvcure: Flexible parametric mixture and non-mixture cure models for time-to-event data.  _
https://cran.r-project.org/web/packages/flexsurvcure/flexsurvcure.pdf_, 2020.

  * Amica and Van Keilegom (2018) M. Amica and I. Van Keilegom.  Cure models in survival analysis.  _Ann. Rev. Statist. Appl._ , 5:311–342, 2018.
  * Cairns et al. (2013) M.E. Cairns, K.P. Asante, S. Owusu-Agyei, D. Chandramohan, B.M. Greenwood, and P.J. Milligan.  Analysis of partial and complete protection in malaria cohort studies.  _Malaria J._ , 12:355, 2013.
  * Chatterjee and Shih (2001) N. Chatterjee and Shih.  A bivariate cure-mixture approach for modeling familial association in diseases.  _Biometrics_ , 57:779–786, 2001.
  * Chatterjee and Shih (2003) N. Chatterjee and Shih.  On use of bivariate survival models with cure fraction.  _Biometrics_ , 59:1184–1185, 2003.
  * Coelho-Barros et al. (2016) E. Coelho-Barros, J.A. Achcar, and J. Mazucheli.  Bivariate Weibull distributions derived from copula functions in the presence of cure fraction and censored data.  _J. Data Science_ , 14:295–316, 2016.
  * Escobar-Bach et al. (2021) M. Escobar-Bach, R.A. Maller, I. Van Keilegom, and M. Zhao.  Estimation of the cure rate for distributions in the Gumbel maximum domain of attraction under insufficient follow-up.  _Biometrika, to appear_ , 2021.
  * Frank (1979) M. J. Frank.  On the simultaneous associativity of f(x,y) and x+ y-f(x,y).  _Aequationes Mathematicae_ , 19:194–226, 1979.
  * Jackson (2016) C.H. Jackson.  flexsurv: A platform for parametric survival modeling in R.  _Journal of Statistical Software_ , 70:1–33, 2016.
  * Kalbfleisch and Prentice (2003) J. D. Kalbfleisch and R. L. Prentice.  _The Statistical Analysis of Failure Time Data_.  Wiley, New York, 2003.
  * Kaplan and Meier (1958) E.L. Kaplan and P. Meier.  Nonparametric estimation from incomplete observations.  _J. Amer. Statist. Assoc._ , 53:457–481, 1958.
  * Klebanov and Yakovlev (2007) L.B. Klebanov and A.Y. Yakovlev.  A new approach to testing for sufficient follow-up in cure-rate analysis.  _J. Statist. Plan. Inf._ , 137:3557–3569, 2007.
  * Kovar et al. (2018) R. Kovar, I. Mala, and F. Habarta.  Dependent censoring in survival regression models.  _12th International Days of Statistics and Economics, Prague, September 6-8_ , 2018.
  * Legrand and Bertrand (2019) C. Legrand and A. Bertrand.  Cure models in oncology clinical trials.  _LIDAM Reprints ISBA 2019043, Université Catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA)_ , 2019.
  * Liu et al. (2018) J. Liu, T. Lichtenberg, K.A. Hoadley, L.M. Poisson, A.J. Lazar, A.D. Cherniack, A.J. Kovatich, C.C. Benz, D.A. Levine, A.V. Lee, L. Omberg, D.M. Wolf, C.D. Shriver, V. Thorsson, and H. Hu.  An integrated tcga pan-cancer clinical data resource to drive high-quality survival outcome analytics.  _Cell_ , 173:400–416, 2018.
  * López-Cheda et al. (2021) Ana López-Cheda, M. Amalia Jácome, and Ignacio López de Ullibarri.  npcure: An R package for nonparametric inference in mixture cure models.  _The R Journal_ , 13(1):21–41, 2021.  doi: 10.32614/RJ-2021-027.  URL https://doi.org/10.32614/RJ-2021-027.
  * Ma (2009) S. Ma.  Cure model with current status data.  _Statist. Sinica_ , 19:233–249, 2009.
  * Maller and Resnick (2021) R.A. Maller and S.I Resnick.  Extremes of censored and uncensored lifetimes in survival data.  _Extremes_ , to appear, 2021.
  * Maller and Zhou (1994) R.A. Maller and X. Zhou.  Testing for sufficient followup and outliers in survival data.  _J. Amer. Statist. Assoc._ , 89:1499–1506, 1994.
  * Maller and Zhou (1996) R.A. Maller and X. Zhou.  _Survival Analysis with Longterm Survivors_.  Wiley, Chichester, first edition, 1996.
  * Maller et al. (2022) R.A. Maller, S.I. Resnick, and S. Shemehsavar.  Splitting the sample at the largest uncensored observation.  _Bernoulli_ , Forthcoming, 2022.
  * Nelsen (2006) R.B. Nelsen.  _An Introduction to Copulas_.  Springer, New York, 2006.
  * Othus et al. (2012) M. Othus, B. Barlogie, M.L. LeBlanc, and J.J. Crowley.  Cure models as a useful statistical tool for analyzing survival.  _Clinical Cancer Research_ , 18:311–342, 2012.
  * Peng and Taylor (2014) Y. Peng and J.M.G. Taylor.  Cure models.  _In: Klein, J., van Houwelingen, H., Ibrahim, J. G., and Scheike, T. H., editors, Handbook of Survival Analysis, Handbooks of Modern Statistical Methods series, chapter 6. Chapman & Hall, Boca Raton, FL, USA._, pages 113–134, 2014.
  * Peng and Yu (2021) Y. Peng and B. Yu.  Cure models: Methods, applications, and implementation.  _Chapman and Hall/CRC_ , 2021.
  * Powles et al. (2021) T. Powles, Z.J. Assaf, N. Davarpanah, R. Banchereau, B.E. Szabados, K.C. Yuen, P. Grivas, M. Hussain, S. Oudard, J.E. Gschwend, P. Albers, D. Castellano, H. Nishiyama, S. Daneshmand, S. Sharma, B.G. Zimmermann, H. Sethi, A. Aleshin, M. Perdicchio, J. Zhang, D.S. Shames, V. Degaonkar, X. Shen, C. Carter, C. Bais, J. Bellmunt, and S. Mariathasan.  ctDNA guiding adjuvant immunotherapy in urothelial carcinoma.  _Nature_ , https://doi.org/10.1038/s41586-021-03642-9, 2021.
  * Salvadori et al. (2007) G. Salvadori, C. DeMichele, N.T. Kottegoda, and R. Rosso.  _Extremes in Nature: An Approach using Copulas_.  Springer Nature, 2007.
  * SEER (2019) DCCPS SEER, National Cancer Institute.  Surveillance research program, released april 2019, based on the november 2018 submission.  2019\.
  * Shen (2000) Pao-sheng Shen.  Testing for sufficient follow-up in survival data.  _Statistics & Probability Letters_, 49:313–322, 2000\.
  * Sklar (1959) A. Sklar.  Fonctions de répartition à $n$ dimensions et leurs marges.  _Publ. Inst. Statist. Univ. Paris_ , 8:229–231, 1959.
  * Yang et al. (2018) W Yang, T Xu, T Garzon-Muvdi, C Jiang, J Huang, and K Chaichana.  Survival of ventricular and periventricular high-grade gliomas: A surveillance, epidemiology, and end results program-based study.  _World Neurosurgery_ , 11:e323–e334, 2018.


# TI-ASU: Toward Robust Automatic Speech Understanding through Text-to-speech
Imputation Against Missing Speech Modality

Tiantian Feng1, Xuan Shi1, Rahul Gupta2, Shrikanth S. Narayanan1
1University of Southern California 2Amazon AGI
<EMAIL_ADDRESS>
###### Abstract

Automatic Speech Understanding (ASU) aims at human-like speech interpretation,
providing nuanced intent, emotion, sentiment, and content understanding from
speech and language (text) content conveyed in speech. Typically, training a
robust ASU model relies heavily on acquiring large-scale, high-quality speech
and associated transcriptions. However, it is often challenging to collect or
use speech data for training ASU due to concerns such as privacy. To approach
this setting of enabling ASU when speech (audio) modality is missing, we
propose TI-ASU, using a pre-trained text-to-speech model to impute the missing
speech. We report extensive experiments evaluating TI-ASU on various missing
scales, both multi- and single-modality setting, and the use of LLMs. Our
findings show that TI-ASU yields substantial benefits to improve ASU in
scenarios where even up to 95% of training speech is missing. Moreover, we
show that TI-ASU is adaptive to dropout training, improving model robustness
in addressing missing speech during inference.

##  1 Introduction

Speech understanding is fundamental to human communication as the rich
information conveyed through speech allows efficient and explicit ways to
exchange thoughts, emotions, and ideas. Within the development of
conversational AI, automatic speech understanding (ASU) aims to accurately and
comprehensively interpret input speech using advanced ML techniques, including
bridging potential communication gaps between people from diverse backgrounds
and language capabilities. With the advances in mobile computing, one popular
application of ASU has been virtual assistants, such as Amazon Alexa and Apple
Siri.

In the deep learning era, the success of data-driven ASU relies heavily on the
quality and diversity of the data used for training the models. To ensure that
the ASU model reaches a fair convergence, an ideal corpus is expected to
contain an adequate amount of complete audio samples that cover diverse speech
attributes, such as emotion, intonation, and speaker demographics [9].
However, multiple practical constraints–both technological and human–hinge on
the collection of the ASU datasets, such as hardware instability, imbalances
in data resources, and the need for privacy protection. To deal with
limitations to access speech samples, it is hence necessary to modify the ASU
toward robustness to ‘missing’ speech.

In this work, we focus on investigating the ASU when speech is increasingly
missing from the corpus, and in some extreme cases, only the label of the
speech is provided during the training. Previously, [40, 17] approached this
problem by projecting multi-modal data to a unified feature space from which
the missing modality can be reconstructed during inference. Inspired by the
recent surge of generative AI, some attempts [12, 11, 38] incorporate
synthetic data in the training to boost the model performance in zero-shot
setting. Moreover, [31, 8, 2] have demonstrated that synthetic speech can
serve as data augmentation to boost speech recognition.

In this work, we propose TI-ASU, a Text-to-speech Imputation approach for
Automatic Speech Understanding that addresses missing modality challenges in
ASU applications. The core idea of TI-ASU is to impute the missing speech
modality with TTS models from text transcriptions. Extensive experimental
results demonstrate the effectiveness of TI-ASU in various missing scale,
single- and multi-modality scenarios.

Figure 1: Problem formulation of missing modalities in this work with ASU. The
missing speech modality includes cases in training data alone or any data
(both training data and testing data).

##  2 Related Works

Text-To-Speech (TTS): TTS aims to generate human-like speech from a given text
input. Modern TTS models typically include two modules: 1) an acoustic module
that maps intermediate acoustic features from the input text (Tacotrons [34,
27], FastSpeechs [26, 25], and et al.); and, 2) a waveform module that
transforms the acoustic feature to the audio (WavNet [21], NSF [33], HiFi-GAN
[13], DiffWave [14], and et al.) With the rise of foundation model in language
and speech, SpeechT5 [1] and VALL-E(X) [32, 39] learns aligned speech and
language representations with self-supervised learning to synthesis speech.

Missing Modality: How to leverage a data-driven model with fragmented input? A
straightforward method is to remove the incomplete samples [6, 20]. Instead of
removal, another direction is to reconstruct the missing modality from joint
representation space [23, 40] or bayesian-meta-learning [19]. With the
prevalence of the transformer in multi-model, multiple threads modifications
are proposed to extend the transformer for the missing modality input, such as
feature-reconstruction [36], tag-based encoding [37], multi-task modeling
[18], and prompt learning [15]. However, existing literature studies this
issue from the glass-half-empty perspective, while TI-ASU reframes this
problem from the glass-half-full perspective with the assistance of generative
AI to boost low-resource speech training.

##  3 Problem Definition

The present work focuses on studying multi-modal learning for ASU with missing
speech modality, as shown in Figure 1 111The figure in this paper uses images
from https://openmoji.org/. Our motivation to study scenarios with missing
speech arises from the fact that speech data carry sensitive information about
an individual, such as biometric fingerprints and demographic and health
status. In such instances, a common assumption is that the text information
may remain accessible by deploying efficient ASR models on the edge, similar
to Figure 2. However, the speech data is not allowed to egress out of the user
devices due to privacy risks, preventing them from being used for training ASU
models. Overall, we study two distinct scenarios of missing speech modality:
speech missing in training or from any data setting, both in training and
testing.

###  3.1 Speech-Missing in Training Data

Our investigation starts with a relatively trivial case where missing speech
only occurs in the training data while testing data are with complete speech-
text data. Following the notation conventions for missing modalities in [15],
we define the set of complete training data samples and text-only training
samples as $\mathcal{D}_{train}^{C}$ and $\mathcal{D}_{train}^{T}$,
respectively. Specifically, we represent
$\mathcal{D}_{train}^{C}=\\{x_{i}^{T},x_{i}^{S},y_{i}\\}$ and
$\mathcal{D}_{train}^{T}=\\{x_{i}^{T},y_{i}\\}$, where $i\in\mathbb{N}$, $T$
and $S$ represent text and speech modalities, respectively. In the end, the
training dataset can be expressed as
$\mathcal{D}_{train}=\\{\mathcal{D}_{train}^{C},\mathcal{D}_{train}^{T}\\}$.
Without loss of generality, we introduce the speech modality missing ratio
presented in training data, denoted as $p$. Namely, $p=0\%$ indicates the
special case that no speech samples are missing.

Figure 2: Illustration of edge devices performing ASR services, where text
modality is always present. Figure 3: Learning framework of TI-ASU: Imputing
missing speech modality with synthetic speech content through text-to-speech
transformer models for robust automatic speech understanding.

###  3.2 Speech-Missing in Any Data

Apart from exploring the scenario with speech missing in training data, we
further study cases where speech data can be missing in both training and
testing data. Previous studies on multimodal learning have highlighted that
multimodal models, such as multimodal transformers, are frequently sensitive
to modality missing during inference, resulting in substantial performance
drops when test samples contain missing modalities. On the other hand, missing
modality can be regarded as a robustness challenge in multimodal learning, as
the missing modality is a unique case of data perturbation. In summary, we can
denote the test dataset as
$\mathcal{D}_{test}=\\{\mathcal{D}_{test}^{C},\mathcal{D}_{test}^{T}\\}$
following our notation established in the previous subsection. We define the
speech modality missing ratio in testing data as $q$, with $q=100\%$
representing the scenario where only text modality is present in the testing
set.

##  4 TI-ASU Framework

###  4.1 Pre-trained Models

This work experiments with widely adopted pre-trained models as the backbone
for modeling ASU, specifically choosing WavLM [7] and RoBERTa [16] as the
speech and text encoders, respectively. The WavLM, a model trained with self-
supervised learning, demonstrates promising performance in a wide range of
speech-centered tasks. This model is trained with multiple learning
objectives, such as frame prediction, speech enhancement, and speech
prediction. Our experiments employ the WavLM Base Plus model, consisting of 12
encoding layers with approximately 90M parameters. On the other hand, RoBERTa
is a transformer model for text classification based on BERT, benefiting from
using a much larger training corpus and modified training objectives than
BERT.

###  4.2 End-to-End Downstream Modeling

Our modeling draws inspiration from [10], highlighting the effectiveness of
combining speech and text information for ASU.

Speech: Similarly to [22], our speech downstream model utilizes learnable
weight parameters to combine hidden outputs from all encoder layers. The
combined speech output is then fed into two 1D pointwise convolutional layers
with a filter size of 256. Finally, global average pooling is applied to the
convolutional layer output, resulting in an output vector of size 256.

Text: In contrast to speech modeling, we apply a 2-layer bidirectional GRU of
a hidden size of 256 to the last hidden output from the RoBERTa model. The
output from the GRU layers is averaged to obtain an output embedding of size
256.

Multimodal Fusion: In the last stage, we concatenate the text embedding and
speech embedding into a multimodal embedding, which is subsequently fed into
two fully connected layers for the ASU classification tasks. We want to
highlight that speech-only or text-only models respectively feed speech
embedding and text embeddings into the classifiers without the fusion.

###  4.3 Speech Data Generation

The core idea behind TI-ASU is to impute the missing speech modality (as shown
in Figure 3) with transformer-based TTS models. To do so, we prompt the TTS
models with transcriptions to synthesize speech data. Prior research on visual
recognition [28, 12] has demonstrated that diversity is needed in training
data generation. For example, researchers propose to include multi-domain
knowledge to enrich the diversity of image generation by augmenting the domain
"photo" to other domains such as drawing and painting. In contrast to the
diversity enhancement approaches in image generation, our proposed TI-ASU
enriches the generation diversity by employing multiple TTS models in the
generation. This differs from image generation, where diversity is associated
with the prompt message. In speech generation, however, the input
transcription to the TTS model remains the same. Here, we deploy 3 TTS:
Speech-T5 [1], MMS-En [24], and Vall-E X [39].

###  4.4 Training with Speech Imputation

The multi-modal learning framework in TI-ASU is presented in Figure 3. Without
loss of generality, we define the generated speech dataset as
$\mathcal{D}_{train}^{S^{\prime}}=\\{{x}_{j}^{S^{\prime}},y_{j}\\}$, where
${x}_{j}^{S^{\prime}}$ is the generated speech from the text $x_{j}^{T}$ and
$S^{\prime}$ denotes the generated speech. More concretely, given a TTS model
$\mathcal{G}$, we can define ${x}_{j}^{S^{\prime}}=\mathcal{G}(x_{j}^{T})$. We
impute each text-only data during each training epoch by randomly selecting a
generated speech from the generation set involving three TTS models.
Consequently, we obtain a modality-complete dataset
$\hat{\mathcal{D}}=\\{\mathcal{D}^{C},\hat{\mathcal{D}}^{T}\\}$ with speech
data imputation, where
$\hat{\mathcal{D}}^{T}=\\{x_{i}^{T},x_{i}^{S^{\prime}},y_{i}\\}$. Finally, we
perform multi-modal training with the imputed dataset $\hat{\mathcal{D}}$. It
is worth noting that TI-ASU can integrate with other multi-modal learning
algorithms, such as dropout training and prompt learning, as shown in Figure
3.

##  5 Datasets

Table 1: Summary of dataset statistics used in this work. Datasets | Speaker # |  Classes |  Utterance #
---|---|---|---
IEMOCAP | 10 | 4 | 5,531
MSP-Improv | 12 | 4 | 7,798
SLURP | 177 | 46 | 72,277
SLUE | 140 | 3 | 7,231

We utilize four datasets – IEMOCAP [4], MSP-Improv [5], SLURP [3], and SLUE-
Voxceleb [29] – to evaluate TI-ASU across three ASU-related tasks: speech
emotion recognition, spoken language understanding, and speech sentiment
analysis. Notably, the first two datasets are employed for the identical task.

##  6 Experimental Details

###  6.1 Speech Data Generation

As mentioned previously, we apply 3 TTS experts to generate speech samples:
Speech-T5, MMS-En, and Vall-E X. We generate three speech samples for each
text-only sample using these TTS models. In particular, in the context of
emotion recognition, we generate speech samples by introducing emotion styles
associated with the training data when utilizing the Vall-E X model. We would
emphasize that Vall-E X derives the emotion styles from datasets distinct from
IEMOCAP and MSP-Improv datasets, avoiding introducing in-domain knowledge in
the generation process. Most importantly, we choose not to impute missing data
in the test set to prevent introducing artifacts in the evaluation.

###  6.2 Evaluation and Hyperparameters

Evaluation: We use unweighted average recall (UAR) to evaluate emotion
recognition, while we utilize the F1 score to evaluate intent and sentiment
classifications. We conducted the experiments with 5-fold and 6-fold cross-
validation on the IEMOCAP and MSP-Improv datasets, respectively. Moreover, we
performed the training with 3 random seeds and reported the average
performance on the remaining datasets. For the Slurp dataset, we use standard
splits for training, validation, and testing, while for the SLUE-Voxceleb
dataset, we split 20% of the default training data for validation. We report
performance on the default validation set, given that the label of the test
set is unavailable from the public SLUE-Voxceleb data.

Hyperparameters: In all experiments, including fine-tuning for natural and
imputed datasets, we set the batch size as 64. Specifically, we set the
learning rate at 0.0005 and the maximum training epoch as 30 for training all
speech models. We apply a learning rate of 0.0001 and a maximum training epoch
of 20 for text and multimodal training, as we observed faster training
convergence with text modality. The experiments are conducted on a high-
performance computing server with A40 GPUs. We use the checkpoints of each
pre-trained model from HuggingFace [35].

##  7 Can TI-ASU Improve Data Efficiency with Speech-Missing in Training?

In this section, we present the results of ASU with speech missing in the
training. Specifically, we implement the following approaches for comparisons:

  * •

Text Training:  Given that a substantial portion of speech is missing from our
experiments, one natural baseline is to rely solely on the text for ASU. This
baseline is essentially a text-only model.

  * •

Speech Training:  In addition to the text-only model, another baseline is to
train a speech model with the speech data available. Given the missing ratio
$p$, we train the model with $100-p\%$ speech.

  * •

Multimodal Training:  Another extension to speech training with limited speech
is adding the paired text. Specifically, given speech missing ratio $p$, we
train with $100-p\%$ speech-text data.

  * •

Multi-modal Training with Zero-filling Imputation:  The approaches above use a
portion of available data. To utilize all the available data, we impute the
missing images by filling them with zeros as in [15]. For example, we fill
$p\%$ speech data with zero in the multi-modal training.

  * •

TI-ASU-S:  Speech-only training based on TI-ASU.

  * •

TI-ASU-MM: Multimodal training based on TI-ASU.

Table 2: Comparisons among text, speech, and multi-modal models across different datasets. We compare different models with $p=0\%$, where $p$ indicates speech missing ratio in training. | Text-Only | Speech-Only | Multi-modal
---|---|---|---
IEMOCAP | $62.8$ | $69.2$ | 71.4
MSP-Improv | $52.7$ | $63.7$ | 65.6
Slurp | $78.8$ | $59.2$ | 80.3
SLUE-Voxceleb | 51.4 | $45.1$ | $50.0$

###  7.1 Would text alone be enough for ASU?

Text modality alone is effective in diverse language-related tasks. Therefore,
our investigation begins with whether text data alone achieves competitive ASU
performance. To answer this, we compare the text model with speech and multi-
modal models trained with complete data, as shown in Table 2. The results
demonstrate that multimodal models consistently yield benefits in emotion
recognition, while text data exhibits competitive or even better performances
compared to multimodal models in other tasks. However, we find that speech
models often underperform text and multimodal models. In summary, our
experiments highlight, not surprisingly, that speech information can benefit
emotion recognition (leveraging the acoustic variation in emotion expression)
but does not necessarily contribute to sentiment and intent classifications
(which are largely driven by language information).

Table 3: Comparisons of speech training and TI-ASU-S. Here, TI-ASU-S is trained with pure synthetic data ($p=100\%$), and speech training uses complete real speech for training ($p=0\%$). $p$ indicates the speech missing ratio in training data. | Speech Training | TI-ASU-S
---|---|---
| $\mathbf{p=0\%}$ | $\mathbf{p=100\%}$
IEMOCAP | $69.2$ | $52.7$
MSP-Improv | $63.7$ | $44.4$
Slurp | $59.2$ | $55.5$
SLUE-Voxceleb | $45.1$ | $41.2$
Figure 4: Comparisons among using single TTS generation and multiple
generation in TI-ASU. Here, the training set is entirely based on synthetic
speech data.

###  7.2 Can TI-ASU-S Provide Competitive Zero-shot Performance Without Real
Speech?

Zero-shot Performance with Synthetic Speech: While speech data may not
consistently benefit multimodal ASU, as shown in Table 2, it is worth
investigating whether relying solely on synthetic speech can result in a
better-performed speech model. Accordingly, this involves a speech missing
ratio $p=100\%$ in training. Table 3 compares speech training without missing
and TI-ASU-S with $p=100\%$. The results reveal that real speech exhibits
substantial advantages in emotion recognition, while synthetic speech alone
can yield competitive speech models for intent and sentiment classification.
Even though synthetic speech data cannot replace real speech data, our
findings suggest their potential as a valuable training resource.

Multiple TTS in Speech Generation: We further explore the effectiveness of our
proposed method in enhancing generation diversity through multiple TTS for ASU
training. We compare the TI-ASU-S using individual TTS generation versus our
previously mentioned combined approach. The comparisons in Figure 4 suggest
that augmenting speech generation by deploying multiple TTS models brings
substantial benefits to ASU training, leading to consistent performance
increases compared to single TTS generation. Therefore, the remaining results
of TI-ASU are based on Multiple TTS generations.

Table 4: Comparisons of speech models trained using real speech and TI-ASU-S. Here, we present real speech training and TI-ASU-S with $p=50\%$ and $p=95\%$. $p$ indicates the speech missing ratio in training data. | $\mathbf{p=95\%}$ | $\mathbf{p=50\%}$
---|---|---
|  Speech | TI-ASU-S |  Speech | TI-ASU-S
IEMOCAP | $52.5$ | $\mathbf{54.8}$ | $65.7$ | $\mathbf{65.8}$
MSP-Improv | $44.4$ | $\mathbf{48.3}$ | $\mathbf{61.5}$ | $60.5$
Slurp | $<5\%$ | $\mathbf{56.0}$ | $50.4$ | $\mathbf{58.0}$
SLUE-Voxceleb | $34.4$ | $\mathbf{41.2}$ | $40.4$ | $\mathbf{45.7}$
Table 5: Comparing TI-ASU-MM with models trained using complete text and multimodal data. Here, TI-ASU-MM is with $p=100\%$, meaning no real speech is used in training but synthetic speech. | Text | Multimodal | TI-ASU-MM
---|---|---|---
| $\mathbf{p=100\%}$
IEMOCAP | $62.8$ | $71.4$ | $67.1$
MSP-Improv | $52.7$ | $65.6$ | $58.0$
Slurp | $78.8$ | $80.3$ | $79.4$
SLUE | $51.4$ | $50.0$ | $51.0$

###  7.3 Can TI-ASU-S Improve Model Performance with Limited Real Speech?

Table 3 shows the importance of real speech data in training ASU, leading us
to study training ASU models with limited available speech data. $p$ indicates
the speech-missing ratio in training.

TI-ASU-S with $\mathbf{p=95\%}$ (extreme missing condition): To begin with, we
study severe speech-missing conditions, including cases where the speech-
missing ratio is $p=95\%$. Table 4 compares the performance between speech
training and TI-ASU-S at $p=95\%$. The results indicate that our proposed TI-
ASU-S can consistently outperform real-speech training, where $95\%$ of speech
is missing. We identify that this performance increase is substantial in
intent classification, where the speech training fails to converge.

TI-ASU-S with $\mathbf{p=50\%}$ (moderate missing condition): In addition to
cases where a significant portion of the speech data is missing, we extend our
investigation to less severe scenarios where only half of the speech is
unavailable. The results in Table 4 demonstrate that even with more real
speech, TI-ASU-S can yield better performances compared to real speech
training in most datasets. Similar to results in cases where $p=95\%$, we
identify that TI-ASU-S provides notable performance improvements in intent
classification. However, the performance benefit is smaller at $p=50\%$
compared to $p=95\%$.

Table 6: Comparing TI-ASU-MM with multimodal learning using available speech-text pairs and zero-filling imputation on missing speech. Here, we experiment with the condition where $p=95\%$. $p$ indicates the speech missing ratio in training data. | MM | MM Zero Filling | TI-ASU-MM
---|---|---|---
| Imputation
IEMOCAP | $52.8$ | $63.9$ | $\mathbf{67.1}$
MSP-Improv | $32.8$ | $53.2$ | $\mathbf{57.9}$
Slurp | $45.9$ | $78.7$ | $\mathbf{80.4}$
SLUE | $42.5$ | $\mathbf{51.4}$ | $50.6$
Figure 5: Comparisons of multimodal training between GTI-ASU and zero-filling
imputation on missing speech. Speech missing ratio in training data
$p\in\\{50\%,70\%,90\%,95\%\\}$

###  7.4 Can TI-ASU-MM Provide Competitive Multimodal Models with No Real
Speech?

Apart from assessing TI-ASU in assisting speech training for ASU, we further
investigate multimodal learning, incorporating text modality into ASU
training. Similar to TI-ASU-S experiments, we begin with TI-ASU-MMwhere speech
missing ratio $p=100\%$, indicating that no real speech is available in the
training. Table 5 compares TI-ASU-MM, involving pure synthetic speech with
text, against text-only and multimodal training. The latter two are with
complete data in training. The results indicate that TI-ASU-MM can improve
emotion recognition compared to text-only training, leading to 5-6%
performance increase. However, TI-ASU-MM underperforms multimodal learning in
emotion recognition. On the other hand, we find that TI-ASU-MM does not
improve intent or sentiment classification, where the text-only model provides
the best performances.

###  7.5 Can TI-ASU-MM Improve Multimodal Performance with Limited Real
Speech?

TI-ASU-MM with $\mathbf{p=95\%}$ (extreme missing condition): Similar to the
TI-ASU experiments where we compared TI-ASU with speech training of limited
speech data, we compare TI-ASU-MM with multimodal training with data that both
modalities are present. In addition, we compare TI-ASU-MM with multimodal
training using zero-filling imputation on missing speech samples as shown in
Table 6. Similar to findings from speech training, we observe that TI-ASU-MM
benefits emotion recognition in multimodal learning the most. In addition,
this advantage is also exhibited in intent classification but not in sentiment
classification. Overall, the results support that TI-ASU enhances ASU
performance in multimodal learning by imputing missing speech with TTS models.

TI-ASU-MM with Lower $\mathbf{p}$: Moreover, we assess TI-ASU-MM where limited
real speech is accessible. Given our findings suggesting marginal or no
benefits of TI-ASU in intent and sentiment classification, we focus this
analysis on emotion recognition. We plot the comparisons between TI-ASU-MM and
multimodal learning with zero-filling imputation on IEMOCAP and MSP-Improv, as
shown in Figure 5. The plot highlights that TI-ASU-MM is more effective in
enhancing model performance when the speech missing ratio is significant
(e.g., $p>70\%$), and multimodal learning with zero-filling imputation
delivers comparable performances to TI-ASU-MM when $50\%$ of speech is
available.

##  8 Can TI-ASU Enhance Robustness with Speech-Missing in Train and Test?

This section extends our previous experiment from the speech missing in
training data to any data. Dropout training [18] has been widely applied to
enhance the robustness of the multimodal models against missing modalities
during inference. In the dropout training, a random portion of the selected
modality is replaced with zeros for each batch. In this paper, we perform
dropout on speech modality. Specifically, we propose TI-ASU Dropout, combing
the dropout training with TI-ASU, where we randomly fill the data in the
imputed dataset
$\hat{\mathcal{D}}=\\{\mathcal{D}^{C},\hat{\mathcal{D}}^{T}\\}$ with zero.
Moreover, our baseline is MM-Dropout, multimodal dropout training with
complete data, which is the baseline in [15, 18]. The dropout training uses
the dropout rate equal to the test speech missing ratio $q$. We evaluate the
model with the missing ratio $q=\\{50\%,70\%,90\%\\}$.

Table 7: Comparing MM-Dropout (Multimodal-Dropout) with TI-ASU Dropout. $p$ and $q$ are training and testing speech missing ratios, respectively. Dataset | Dropout | $\mathbf{p}$ | Test Missing Ratio ($q$)
---|---|---|---
50 | 70 | 90
IEMOCAP | MM | $0$ | $\mathbf{66.6}$ | $64.5$ | $63.8$
TI-ASU | $50$ | $66.4$ | $\mathbf{64.6}$ |  $\mathbf{64.1}$
$95$ | $63.9$ | $63.2$ | $63.3$
Slurp | MM | $0$ | $79.0$ | $78.3$ | $78.6$
TI-ASU | $50$ | $78.4$ | $78.1$ |  $78.2$
$95$ | $\mathbf{79.4}$ | $\mathbf{78.8}$ | $\mathbf{79.0}$
SLUE | MM | $0$ | $51.0$ | $49.3$ | $50.4$
TI-ASU | $50$ | $\mathbf{51.4}$ | $50.6$ |  $\mathbf{51.3}$
$95$ | $51.4$ | $\mathbf{51.1}$ | $51.1$
Table 8: Comparisons of TI-ASU using original transcript and LLM-assisted transcript augmentation. Here, we present both TI-ASU-S and TI-ASU-MM with $p=95\%$. $p$ indicates the speech missing ratio in training data. | Speech | TI-ASU-S | TI-ASU-S (LLM) | MM Zero-Filling | TI-ASU-MM | TI-ASU-MM (LLM)
---|---|---|---|---|---|---
| Imputation
IEMOCAP | $52.5$ | $\mathbf{54.8}$ | $54.5$ | $63.9$ | $\mathbf{67.1}$ | $65.4$
Slurp | $<5\%$ | $\mathbf{56.0}$ | $50.4$ | $78.7$ | $\mathbf{80.4}$ | $76.2$
SLUE | $34.4$ | $41.2$ | $\mathbf{41.7}$ | ${51.4}$ | $50.6$ | $\mathbf{53.0}$

Can TI-ASU Dropout outperform MM-Dropout when $p=95\%$ (extreme missing
condition)? Here, we train TI-ASU with $p=95\%$, meaning that only 5% of real
speech is available. The comparisons between MM-Dropout and TI-ASU Dropout are
listed in Table 7. The results show that TI-ASU with dropout training is more
robust than the multi-modal dropout training in intent and sentiment
classifications. However, we observe this difference is marginal. In contrast,
MM dropout outperforms TI-ASU Dropout in emotion recognition when $q$ is low.

Can TI-ASU Dropout outperform MM-Dropout when $p=50\%$ (moderate missing
condition)? We further investigate TI-ASU Dropout with more real data in
training. The comparison in Table 7 shows that TI-ASU Dropout at $p=50\%$
yields competitive emotion recognition performance compared to MM-dropout at
different $q$. In addition, we find marginal differences between TI-ASU
Dropout and multimodal dropout in the remaining tasks. Overall, these results
suggest that, at a lower $p$, TI-ASU Dropout enhances model robustness against
test speech missing.

##  9 Can LLM-Assisted Speech Generation Apply to TI-ASU?

###  9.1 Generation Approach

In the previous section, we investigated speech imputation using speech
generated from transcriptions. We argue that the diversity of this generation
process is limited due to the constrained language content from the original
speech. To augment existing speech generation, we propose to augment the
transcriptions using the recently released LLM, LLaMa2-70B [30]. In
particular, we adopt the following prompting message to augment the text
transcriptions:

Don’t repeat my instructions. Rephrase the following sentence: TRANSCRIPTIONS

Specifically, for the text-only training set
$\mathcal{D}_{train}^{T}=\\{x_{i}^{T},y_{i}\\}$, LLM model $\mathcal{F}$, and
prompt function $prompt(\cdot)$, we create augmented text set
$\mathcal{D}_{train}^{T_{aug}}=\\{x_{i}^{T_{aug}},y_{i}\\}$, where
$x_{i}^{T_{aug}}=\mathcal{F}(prompt(x_{i}^{T}))$. Moreover, we perform a
similar approach to generate speech set
$\hat{\mathcal{D}}^{T_{aug}}=\\{x_{i}^{T_{aug}},x_{i}^{S^{\prime}},y_{i}\\}$.
Finally, we combine $\hat{\mathcal{D}}^{T_{aug}}$ and $\hat{\mathcal{D}}^{T}$
to form the imputed multimodal set $\hat{\mathcal{D}}^{T^{\prime}}$ with
speech missing. During training, we randomly select a pair of text and speech
from $\hat{\mathcal{D}}^{T^{\prime}}$ as the data for training ASU. In this
way, we enrich the generation diversity by employing multiple TTS models and
language content enhancement using LLM.

###  9.2 Generation Examples

Here, we provide generation examples from IEMOCAP, Slurp, and SLUE-Voxceleb
datasets.

Dataset - IEMOCAP

Raw: Out there somewhere there is this huge mass of silverfish headed this way

Augmented: A large school of shimmering silver fish is making its way towards
us.

Dataset - Slurp

Raw: i did not want you to send that text yet wait until i say send

Augmented: I wanted you to hold off on sending the text message until I gave
you the green light, rather than sending it right away.

SLUE-Voxceleb

Raw: well it’s hard to get the the right the proper finance money to

Augmented: Securing adequate financial resources can be a challenge.

###  9.3 Performance of LLM-assisted TI-ASU

We compare the LLM-assisted TI-ASU with TI-ASU that relies on the original
transcript. We conduct the experiments using speech and multimodal training
with $p=95\%$. Particularly, we perform the LLM augmentation on IEMOCAP,
Slurp, and SLUE-Voxceleb datasets, as demonstrated in Table 8. The results
indicate that LLM augmentation does not always improve the fine-tuning
performances, yielding decreased performance compared to TI-ASU in IEMOCAP and
Slurp datasets. We notice this performance drop exists in both speech and
multimodal training. However, we observe that the LLM-assisted augmentation
can benefit sentiment classification, improving performance on SLUE datasets
in both speech and multimodal training. Overall, the findings suggest that LLM
holds promise for augmenting speech samples. However, there is a need to
enhance the quality of the rephrased transcription utilized for speech
generation. This can be achieved through designing more effective prompt
strategies or employing more advanced models such as ChatGPT.

##  10 Conclusion

In this work, we proposed TI-ASU, a TTS imputation approach, for multi-modal
ASU learning to address the challenges caused by missing speech modality. Our
experiments demonstrate that TI-ASU provides robust multi-modal solutions
against severe missing speech modality settings in training data or testing
data. Crucially, increasing the diversity through multiple TTS speech
generation enhances TI-ASU performance.

Limitations and Future Works: While TI-ASU is effective in imputing speech
data using raw transcript for ASU training, it encounters challenges in
combining LLM-assisted augmentations, implying the potential issues in text
generation. For example, we identify that LLM frequently extends the spoken
language content, making it unlikely to be used in daily communication.
Moreover, enriching speaker diversity in speech generation remains a
challenge. Our future work plans to explore more advanced LLMs to improve the
quality of the text augmentation. Moreover, we plan to include human
inspection to evaluate generation quality.

## References

  * [1] Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei.  Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, pages 5723–5738. Association for Computational Linguistics, 2022.
  * [2] Martijn Bartelds, Nay San, Bradley McDonnell, Dan Jurafsky, and Martijn Wieling.  Making more of little data: Improving low-resource automatic speech recognition using data augmentation.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 715–729. Association for Computational Linguistics, 2023.
  * [3] Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser.  SLURP: A spoken language understanding resource package.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7252–7262. Association for Computational Linguistics, 2020.
  * [4] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan.  Iemocap: Interactive emotional dyadic motion capture database.  Language resources and evaluation, 42:335–359, 2008.
  * [5] Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost.  Msp-improv: An acted corpus of dyadic interactions to study emotion perception.  IEEE Transactions on Affective Computing, 8(1):67–80, 2016.
  * [6] Jiayi Chen and Aidong Zhang.  Hgmf: Heterogeneous graph-based fusion for multimodal data with incompleteness.  KDD ’20, page 1295–1305. Association for Computing Machinery, 2020.
  * [7] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022.
  * [8] Amin Fazel, Wei Yang, Yulan Liu, Roberto Barra-Chicote, Yixiong Meng, Roland Maas, and Jasha Droppo.  Synthasr: Unlocking synthetic data for speech recognition.  arXiv preprint arXiv:2106.07803, 2021.
  * [9] Tiantian Feng, Rajat Hebbar, Nicholas Mehlman, Xuan Shi, Aditya Kommineni, Shrikanth Narayanan, et al.  A review of speech-centric trustworthy machine learning: Privacy, safety, and fairness.  APSIPA Transactions on Signal and Information Processing, 12(3), 2023.
  * [10] Tiantian Feng and Shrikanth Narayanan.  Foundation model assisted automatic speech emotion recognition: Transcribing, annotating, and augmenting.  arXiv preprint arXiv:2309.08108, 2023.
  * [11] Tiantian Feng, Daniel Yang, Digbalay Bose, and Shrikanth Narayanan.  Can text-to-image model assist multi-modal learning for visual recognition with visual modality missing?  arXiv preprint arXiv:2402.09036, 2024.
  * [12] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi.  Is synthetic data from generative models ready for image recognition?  arXiv preprint arXiv:2210.07574, 2022.
  * [13] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  Advances in Neural Information Processing Systems, 33:17022–17033, 2020.
  * [14] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  In International Conference on Learning Representations, ICLR, 2021.
  * [15] Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Lee.  Multimodal prompting with missing modalities for visual recognition.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [16] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [17] Wei Luo, Mengying Xu, and Hanjiang Lai.  Multimodal reconstruct and align net for missing modality problem in sentiment analysis.  In International Conference on Multimedia Modeling, pages 411–422. Springer, 2023.
  * [18] Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng.  Are multimodal transformers robust to missing modality?  In Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, pages 18177–18186, 2022.
  * [19] Mengmeng Ma, Jian Ren, Long Zhao, S. Tulyakov, Cathy Wu, and Xi Peng.  Smil: Multimodal learning with severely missing modality.  ArXiv, abs/2103.05677, 2021.
  * [20] Jianmo Ni, Larry Muhlstein, and Julian McAuley.  Modeling heart rate and activity data for personalized fitness recommendation.  The World Wide Web Conference, 2019.
  * [21] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  arXiv preprint arXiv:1609.03499, 2016.
  * [22] Leonardo Pepino, Pablo Riera, and Luciana Ferrer.  Emotion recognition from speech using wav2vec 2.0 embeddings.  arXiv preprint arXiv:2104.03502, 2021.
  * [23] Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos.  Found in translation: Learning robust joint representations by cyclic translations between modalities.  ArXiv, abs/1812.07809, 2018.
  * [24] Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al.  Scaling speech technology to 1,000+ languages.  arXiv preprint arXiv:2305.13516, 2023.
  * [25] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  arXiv preprint arXiv:2006.04558, 2020.
  * [26] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech: Fast, robust and controllable text to speech.  Advances in neural information processing systems, 32, 2019.
  * [27] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al.  Natural tts synthesis by conditioning wavenet on mel spectrogram predictions.  In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4779–4783. IEEE, 2018.
  * [28] Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes.  Diversity is definitely needed: Improving model-agnostic zero-shot classification via stable diffusion.  2023\.
  * [29] Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu J Han.  Slue: New benchmark tasks for spoken language understanding evaluation on natural speech.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7927–7931. IEEE, 2022.
  * [30] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [31] Sei Ueno, Masato Mimura, Shinsuke Sakai, and Tatsuya Kawahara.  Data augmentation for asr using tts via a discrete representation.  In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 68–75. IEEE, 2021.
  * [32] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al.  Neural codec language models are zero-shot text to speech synthesizers.  arXiv preprint arXiv:2301.02111, 2023.
  * [33] Xin Wang, Shinji Takaki, and Junichi Yamagishi.  Neural source-filter waveform models for statistical parametric speech synthesis.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:402–415, 2019.
  * [34] Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous.  Tacotron: Towards end-to-end speech synthesis.  In Proc. Interspeech 2017, pages 4006–4010. ISCA, 2017.
  * [35] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  arXiv preprint arXiv:1910.03771, 2019.
  * [36] Ziqi Yuan, Wei Li, Hua Xu, and Wenmeng Yu.  Transformer-based feature reconstruction network for robust multimodal sentiment analysis.  Proceedings of the 29th ACM International Conference on Multimedia, 2021.
  * [37] Jiandian Zeng, Tianyi Liu, and Jiantao Zhou.  Tag-assisted multimodal sentiment analysis under uncertain missing modalities.  Proc. of the 45th International ACM SIGIR Conf. on Research and Development in Information Retrieval, 2022.
  * [38] Tuo Zhang, Tiantian Feng, Samiul Alam, Mi Zhang, Shrikanth S Narayanan, and Salman Avestimehr.  Gpt-fl: Generative pre-trained model-assisted federated learning.  arXiv preprint arXiv:2306.02210, 2023.
  * [39] Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al.  Speak foreign languages with your own voice: Cross-lingual neural codec language modeling.  arXiv preprint arXiv:2303.03926, 2023.
  * [40] Jinming Zhao, Ruichen Li, and Qin Jin.  Missing modality imagination network for emotion recognition with uncertain missing modalities.  In Annual Meeting of the Association for Computational Linguistics, 2021.