# Functional Causal Inference with Time-to-Event Data

Xiyuan Gao1 Jiayi Wang2,∗ Guanyu Hu3 Jianguo Sun4

1<EMAIL_ADDRESS>
2,∗<EMAIL_ADDRESS>
3<EMAIL_ADDRESS>
4<EMAIL_ADDRESS>

1,3,4 Department of Statistics  University of Missouri-Columbia
2,∗ Department of Mathematical Sciences  University of Texas at Dallas

###### Abstract

Functional data is a powerful tool for capturing and analyzing complex
patterns and relationships in a variety of fields, allowing for more precise
modeling, visualization, and decision-making. For example, in healthcare,
functional data such as medical images can help doctors make more accurate
diagnoses and develop more effective treatment plans. However, understanding
the causal relationships between functional predictors and time-to-event
outcomes remains a challenge. To address this, we propose a functional causal
framework including a functional accelerated failure time (FAFT) model and
three causal approaches. The regression adjustment approach is based on
conditional FAFT with subsequent confounding marginalization, while the
functional-inverse-probability-weighting approach is based on marginal FAFT
with well-defined functional propensity scores. The double robust approach
combines the strengths of both methods and achieves a balance condition
through the weighted residuals between imputed observations and regression
adjustment outcomes. Our approach can accurately estimate causality, predict
outcomes, and is robust to different censoring rates. We demonstrate the power
of our framework with simulations and real-world data from the Alzheimer’s
Disease Neuroimaging Initiative (ADNI) study. Our findings provide more
precise subregions of the hippocampus that align with medical research,
highlighting the power of this work for improving healthcare outcomes.

Keywords: Accelerated failure time ; Functional treatment; Functional
propensity score; Double robust estimator.

##  1 Introduction

Alzheimer’s disease (AD) is a debilitating and fatal neurodegenerative
disorder that affects millions of people worldwide, which has become the 6th
leading cause of death among US adults (CDC2022), with increasing death rates
and high costs of medical expenses (AD2022). However, the pathogenesis of AD
remains unclear, and there is currently no effective cure(ashleigh2023role).
Early detection of mild cognitive impairment (MCI) and accurate assessment of
progression from MCI to AD (MCI-AD) are crucial for the effective management
of the disease (potashman2023psychometric). Magnetic resonance imaging (MRI)
has emerged as a valuable tool in the diagnosis and prognosis of AD,
particularly in the analysis of the hippocampus which is a detectable brain
atrophy susceptible to the impact of AD and frequently used as an MCI-AD
progression biomarker. As a result, increasing attention has been given to
exploring the use of MRI imaging data on the hippocampus to determine its
relationship with MCI-AD and predict the time of conversion from MCI to AD
(warren2023functional).

By treating the MRI on the hippocampus as a functional predictor, recent
advancements including 2018_Kong and 2021_Yang investigated its conditional
effects on the hazard ratio while assuming that the effects of other
covariates are held constant in the functional linear Cox regression model,
but left the causal relationship as an open question. In an AD observational
study such as Alzheimer’s Disease Neuroimaging Initiative (ADNI), it is
challenging to interpret the fitted models if the clinical measurements are
confounded by other factors such as age, education, and ADAS-Cog score. In
order to explore the causal relationship between the hippocampus and MRI-AD,
MRI imaging is regarded as a treatment throughout the paper to be in line with
the causal inference language.

The existing causal inference methods considering time-to-event data primarily
focused on binary (cao2023adjusting), multi-level (corder2022utilizing),
continuous (cui2021instrumental), and time-varying treatments
(yang2020semiparametric), and have not addressed the complexities of
functional treatments. Unlike time-varying treatments that are measured
sequentially at a sparse time grid in a longitudinal analysis, functional
treatments measure the entire trajectory synchronously over the whole domain
at one time point, without a dynamic component. They are featured by the high-
dimension, continuous nature, and discrete observations, resulting in a non-
sparse effect on outcomes. By focusing on continuous responses, 2021_Xiaoke
proposed the functional weights, which cannot be straightforwardly adapted to
time-to-event data due to the existence of censoring. Therefore, new methods
are needed to account for unique features of functional treatment and
determine its causal effect on time-to-event data.

The well-known Cox regression model has been widely investigated with a
functional variable (2018_Kong; 2021_Yang), whose clinical interpretation of
hazard ratios (HR) is limited by the proportional hazard (PH) assumption.
However, even when the PH is plausible, the HR is not causally interpretable
unless there is no treatment effect or an untestable and unrealistic
assumption holds (2022_Marti). As an alternative, the accelerated failure time
(AFT) model was proposed and was able to provide a direct interpretation of
the mean survival time (1990_Ritov). It has been widely studied for a small
number of predictors (saikia2017review) but has not yet been generalized to
functional data. In order to fill the research gap, this paper aims to develop
a novel functional causal framework that can effectively handle the functional
treatment and provide a meaningful causal interpretation with time-to-event
data.

The novel functional causal survival framework consists of the functional AFT
(FAFT) model and three functional causal estimation approaches. Distinguishing
from previous research, this paper makes several new contributions. The FAFT
is proposed by incorporating functional linear regression into AFT for
modeling the relationship between survival outcomes and functional treatment.
We define the functional estimand as the coefficient of functional treatment
in FAFT, representing the causal differences and variations over a domain. In
fact this idea and our proposed approach are widely applicable to general
time-to-event data with functional predictors. To deal with the nonparametric
functional treatment and corresponding coefficient, the functional principle
component analysis (FPCA) is employed to achieve dimension reduction and avoid
computation difficulty in regularization methods (sang2022functional). To
adjust the confounding effects, three causal inference approaches are proposed
for time-to-event data, including the regression adjustment approach, the
functional inverse-probability-of-weighting (FIPW) approach, and the double
robust approach.

The remainder of the paper is organized as follows. In Section 2, we first
review the FPCA on a functional treatment, and then introduce the required
assumptions and propose the FAFT model. Details of three causal inference
approaches are presented in Section 3. To facilitate computation, Section 4
develops the corresponding algorithms used for FAFT estimation, regression
adjustment, FIPW, and double robust approach. Extensive simulation studies are
conducted in Section 5 and an application on AD is included in Section 6. In
addition, the discussion can be found in Section 7. For ease of exposition,
additional technical and numerical results are given in supplementary
material.

##  2 Model and assumptions

###  2.1 Notation and FPCA

The functional treatment $X(s)$ is defined over a compact set
$\mathcal{S}\subseteq\mathbb{R}$, where $s$ denotes the continuous domain of
$X(\cdot)$. However, in practice, $X(s)$ only can be observed on a discrete
set of points within $\mathcal{S}$. The basic idea behind functional data
analysis is to express discrete observed points in the form of a function,
representing the entire measurements as a single observation. The functional
linear regression considers a scalar response $Y$ and models its relationship
with $X(\cdot)$ by using as follows:

$Y=\alpha+\int_{\mathcal{S}}\beta(s)X(s)\mathrm{\leavevmode\nobreak\
d}s+\epsilon,$ (1)

where $\alpha$ is an unknown intercept, $\beta(\cdot)$ is an unknown
coefficient with a functional form, and $\epsilon$ is an error term
independent of $X(\cdot)$. The variability of the functional coefficient
$\beta(s)$ over $\mathcal{S}$ characterizes how the effect of a predictor
variable changes smoothly over the entire domain. It is also possible to
interpret $\beta(\cdot)$ at specific points or regions. In the AD example,
$\beta(\cdot)$ allows us to study how different brain sub-regions impact the
MCI-AD progression. By examining how the coefficient varies over the entire
brain, we can gain insights into which sub-regions have the most significant
impact and which are most vulnerable to progression.

Unlike scalar values, $X(\cdot)$ often exhibits complex structures such as
high dimensionality, nonlinearity, and non-stationarity, making it difficult
to visualize and analyze. To overcome these challenges, this paper adopts FPCA
due to its ability to represent functional data in the most parsimonious way
(jiao2021filtrated), and then performs statistical analysis on the
coefficients of these representations.

Explicitly, the Karhunen-Loève theorem allows for the representation
$X(s)=\sum_{k=1}^{\infty}A_{k}\phi_{k}(s),\allowbreak s\in\mathcal{S},$ where
$\\{\phi_{k}(\cdot):1\leqslant k<\infty\\}$ are eigenfunctions and
$\\{A_{k}=\int_{\mathcal{S}}X(s)\phi_{k}(s)\mathrm{d}s\\}$ are functional
principal component scores (FPCS), with the properties
$E(A_{k})=0,\operatorname{Var}\left(A_{k}\right)=\lambda_{k}$, and
$\operatorname{Cov}\left(A_{k_{1}},A_{k_{2}}\right)=0$ for any $k_{1}\neq
k_{2}$. The corresponding eigenvalues $\\{\lambda_{k}:1\leqslant k<\infty\\}$
satisfy that $\lambda_{k}\geqslant 0$ in a decreasing order and
$\sum_{k=1}^{\infty}\lambda_{k}<\infty$. Based on the span of
$\phi_{k}(\cdot)$’s, the projection of $\beta(\cdot)$ corresponding to
$X(\cdot)$ is also identifiable, as shown below:

$\int_{\mathcal{S}}\beta(s)X(s)\mathrm{\leavevmode\nobreak\
d}s=\int_{\mathcal{S}}\beta(s)\left(\sum_{k=1}^{\infty}A_{k}\phi_{k}(s)\right)\mathrm{\leavevmode\nobreak\
d}s=\sum_{k=1}^{\infty}A_{k}\left(\int_{\mathcal{S}}\beta(s)\phi_{k}(s)\mathrm{\leavevmode\nobreak\
d}s\right).$

Let
$\beta_{k}=\int_{\mathcal{S}}\beta(s)\phi_{k}(s)\mathrm{\leavevmode\nobreak\
d}s$, the expanded functional coefficient is
$\beta(s)=\sum_{k=1}^{\infty}\beta_{k}\phi_{k}(s),s\in\mathcal{S}.$

In practice, the number of FPCS used in modeling the data is usually truncated
at some number $K$ due to computational constraints and to avoid overfitting.
The selection of $K$ is typically based on the percentage of variation
explained (PVE), which measures the proportion of the total variability in the
data that is captured by the first $K$ FPCS. In a sample of size $n$, the
selected threshold is denoted as $K_{n}$ and calculated as
PVE$(K_{n})=\sum_{k=1}^{K_{n}}\lambda_{k}/\sum_{k=1}^{\infty}\lambda_{k}$.
Therefore, $X(s)\approx\sum_{k=1}^{K_{n}}A_{k}\phi_{k}(s),s\in\mathcal{S},$
and the accuracy of such approximation has been proved to increase
asymptotically as $n\to\infty$ (wang2016functional). Common choices of
PVE$(K_{n})$ range from $70\%$ to $99\%$, depending on the complexity of the
data and the research question of interest (2018_Kong).

###  2.2 Causality identification

When observing a time-to-event response, take $T$ and $C$ as the failure time
and censoring time, respectively. The observed event time is denoted as
$\widetilde{T}=\min(T,C)$ and the corresponding censoring indicator is
determined by $\delta=\operatorname{I}\allowbreak(T\leqslant C)$, which equals
to 1 if observing a failure and 0 otherwise. To remove the non-negativity
restriction on event times, we define the response on its $\log$ scale, i.e.,
$Y=\log T$. In addition, a $p-$dimensional vector of observed covariates is
represented by the vector $\mathbf{Z}$.

In accordance with the potential outcomes framework (rubin1974estimating),
consider a potential treatment $x:=x(s)\in
L^{2}=\\{f:\int_{\mathcal{S}}f^{2}(s)\mathrm{\leavevmode\nobreak\
d}s<\infty\\}$. Let $Y(x)$ be the potential outcome and $\beta(\cdot)$ be the
functional coefficient representing the causal effect curve. Our objective is
to model $\mathbb{E}\left[Y(x)\right]$ and estimate $\beta(\cdot|x)$, in terms
of the observed data
$\mathcal{O}=\Bigl{\\{}\left(X_{i}(s),\mathbf{Z}_{i},\delta_{i},\widetilde{T}_{i}\right),\
s\in\mathcal{S},\allowdisplaybreaks i=1,...,n\Bigr{\\}}$. To link the observed
data to counterfactual data, the following assumptions are required to
identify the causal effect.

  * Assumption 1. _Ignorability_

$\quad X\perp\\!\\!\\!\perp Y(x)\ |\ \mathbf{Z},\ \forall x\in L^{2}.$

This is also known as conditional exchangeability, i.e.,
$\mathbb{E}\left[Y(x)|X,Z\right]=\mathbb{E}\left[Y(x)|\mathbf{Z}\right]$. In a
randomized clinical trial (RCT), it always hold since the treatment allocation
mechanism provides no information on the counterfactual outcomes. In an
observational study, this is generally not true and is said to hold provided
that the treatment is randomized within strata of the recorded covariates.
Such condition is generally not verifiable empirically and their plausibility
should be justified based on subject matter knowledge in practice. In general,
it holds if the study guarantees it by design (e.g., stratified randomized
trial) or sufficient potential confounders have been collected.

The following two assumptions are associated with the propensity score for a
functional treatment. Since the conditional density function of a functional
treatment does not exist, the rank-$K$ functional propensity score is defined
as $\pi_{\mathbf{Z}}(\mathbf{A})=P(\mathbf{A}=\mathbf{a}|\mathbf{Z})$ by using
the selected FPCS vector $\mathbf{A}=\left\\{A_{1},...,A_{K}\right\\}$
(2021_Xiaoke).

  * Assumption 2. _Consistency_

$\
\sum_{k=1}^{K}A_{k}\phi_{k}(s)=\sum_{k=1}^{K}a_{k}\phi_{k}(s),s\in\mathcal{S}\Rightarrow
Y=Y(x).$

  * Assumption 3. _Positivity_

$\quad\ 0<\pi_{\mathbf{Z}}(\mathbf{A})<1\text{ with probability
}1,\quad\forall\mathbf{a}\in\mathbb{R}^{K}.$

Assumption 2 states that potential outcomes are uniquely defined by an
subject’s own treatment level, no interference with others and no different
versions of treatment. This ensures that the causal effect of $X(\cdot)$ on an
subject is not influenced by external factors. By implying that every subject
has a positive chance of receiving any level of treatment, regardless of their
covariates, Assumption 3 ensures that the treatment effect estimates are
derived from a representative sample of the population, rather than a biased
subgroup of subjects. It is important to point out that how to address the
positivity violation remains an open problem for continuous treatments
(zhao2020propensity), let alone functional treatments, where the issue of
positivity violation is likely to be more severe.

For the censoring mechanism, we assume independent censoring between the
actual survival time and the censoring time.

  * Assumption 4. _Noninformative censoring_

$\ C\perp\\!\\!\\!\perp Y(x)|\left(\mathbf{Z},X\right)\Rightarrow
C\perp\\!\\!\\!\perp Y(X)|\left(\mathbf{Z},X\right).$

Given Assumptions 1–4, based on the observed data, $\mathbb{E}[Y(x)]$ can be
identified via

$\mathbb{E}[Y(x)]=\int_{\mathbf{Z}}\mathbb{E}\left[Y(x)|\mathbf{Z}=\mathbf{z}\right]\mathrm{\leavevmode\nobreak\
d}f_{\mathbf{Z}}(\mathbf{z})=\int_{\mathbf{Z}}\mathbb{E}\left[Y|X=x,\mathbf{Z}=\mathbf{z}\right]\mathrm{\leavevmode\nobreak\
d}f_{\mathbf{Z}}(\mathbf{z}),$ (2)

where $f_{\mathbf{Z}}(\mathbf{z})$ is the joint density function of all
covariates.

###  2.3 FAFT model

In RCTs, the causal effect estimation is straightforward, since RCTs may
achieve sufficient control over confounding factors provided a good design,
proper conduction, and enough enrollment. We propose a FAFT model that
incorporates the functional linear regression and classical accelerated
failure time model. Specifically, for subject $i$, the FAFT has the form

$Y_{i}=\log(T_{i})=\alpha+\int_{\mathcal{S}}\beta(s)X_{i}(s)\mathrm{\leavevmode\nobreak\
d}s+\epsilon_{i}.$ (3)

For simplicity, assume that $X_{i}(s)$ can be fully observed on grid points
$\\{s_{m}\in\mathcal{S},1\leqslant m\leqslant M\\}$ without any measurement
error. The generalization to situations with measurement error can be found in
2018_Kong. With FPCA, the FAFT can be rewritten and approximated as

$Y_{i}=\log
T_{i}=\alpha+\sum_{k=1}^{\infty}A_{ik}\beta_{k}+\epsilon_{i}\approx\alpha+\sum_{k=1}^{K_{n}}A_{ik}\beta_{k}+\epsilon_{i}.$
(4)

Such approximation reduces the estimation of FAFT with an unknown curve to a
classical AFT regression model with finite-dimensional predictors. Estimation
details will be discussed in Section 4.1 and the resulting estimates include
$\hat{\alpha}$ and $\\{\hat{\beta}_{1},...,\hat{\beta}_{K_{n}}\\}$. The
resulted $\hat{\beta}(\cdot)$ is the estimated functional coefficient in the
model (3) and intuitively represents the causality between $Y$ and $X(\cdot)$
in RCTs.

However, this is usually not true in observational studies due to the possible
disparities in confounders on both $X(\cdot)$ and $Y$. We propose three
approaches to address confounding effects and denote the resulting estimators
with extra subscripts, including (1) regression adjustment approach,
$\Bigl{\\{}\hat{\alpha}_{\text{RegAdj}},\hat{\beta}_{\text{RegAdj},1},...,\hat{\beta}_{\text{RegAdj},K_{n}}\Bigr{\\}}$
and $\hat{\beta}_{\text{RegAdj}}(\cdot)$; (2) FIPW approach,
$\Bigl{\\{}\hat{\alpha}_{\text{FIPW}},\allowbreak\hat{\beta}_{\text{FIPW},1},...,\hat{\beta}_{\text{FIPW},K_{n}}\Bigr{\\}}$
and $\hat{\beta}_{\text{FIPW}}(\cdot)$; and (3) double robust approach,
$\Bigl{\\{}\hat{\alpha}_{\text{DR}},\hat{\beta}_{\text{DR},1},...,\allowbreak\hat{\beta}_{\text{DR},K_{n}}\Bigr{\\}}$
and $\hat{\beta}_{\text{DR}}(\cdot)$.

##  3 Causal inference approaches

###  3.1 Regression adjustment approach

In order to marginalize the confounding effects, the regression adjustment
approach provides consistent estimates of contrasts (e.g. differences, ratios)
by regressing $Y_{i}$ conditional on $X_{i}$ and $\mathbf{Z}_{i}$ followed by
adjusting confounding effects. By considering a full FAFT,

$Y_{i}=\log
T_{i}=\alpha+\int_{\mathcal{S}}\beta(s)X_{i}(s)\mathrm{\leavevmode\nobreak\
d}s+\boldsymbol{\gamma}^{\top}\mathbf{Z}_{i}+\epsilon_{i}\approx\alpha+\sum_{k=1}^{K_{n}}A_{ik}\beta_{k}+\boldsymbol{\gamma}^{\top}\mathbf{Z}_{i}+\epsilon_{i}.$
(5)

the association between $X(\cdot)$ and $Y$ conditioning on $\mathbf{Z}$ is
revealed. This conditional mean survival outcome is the conditional
expectation $\mathbb{E}\left[Y|X=x,\mathbf{Z}=\mathbf{z}\right]$ in equation
(2).

Suppose the fitted FAFT model gives parameter estimates
$(\hat{\alpha},\hat{\beta}_{1},...,\hat{\beta}_{K_{n}},\hat{\gamma}_{1},...,\hat{\gamma}_{p})$,
the confounding adjustment proceeds in the following two steps. For each
subject $i$, construct new responses adjusted by empirical mean of all
confounders,

$\widehat{Y}_{i}=\frac{1}{n}\sum_{j=1}^{n}\left(\hat{\alpha}+\int_{\mathcal{S}}\beta(s)X_{i}(s)\mathrm{\leavevmode\nobreak\
d}s+\hat{\boldsymbol{\gamma}}^{\top}\mathbf{Z}_{j}\right)\approx\hat{\alpha}+\sum_{k=1}^{K_{n}}A_{ik}\hat{\beta}_{k}+\frac{1}{n}\sum_{j=1}^{n}\hat{\boldsymbol{\gamma}}^{\top}\mathbf{Z}_{j}.$
(6)

Based on $\left(X_{i},\widehat{Y}_{i},i=1,...,n\right)$, refit model (4) and
the estimates
$\Bigl{\\{}\hat{\alpha}_{\text{RegAdj}},\hat{\beta}_{\text{RegAdj},1},...,\allowbreak\hat{\beta}_{\text{RegAdj},K_{n}}\Bigr{\\}}$
can be used to reconstruct the estimated functional coefficient, via
$\hat{\beta}(\cdot)_{\text{RegAdj}}=\sum_{k=1}^{K_{n}}\hat{\beta}_{\text{RegAdj},k}\phi_{k}(\cdot)$.

###  3.2 FIPW approach

Instead of modeling survival outcomes conditionally as discussed in Section
3.1, the causal estimator can also be identified by direct marginal modeling.
By defining the true weights to be $\mathrm{w}=f(\mathbf{Z})/f(\mathbf{Z}|X)$,
the FIPW approach created a weighted pseudo sample and the marginal mean of
the potential outcome within it is equal to the adjusted conditional mean of
the actually observed population.

  * Proposition 1.

Under Assumptions 1-3, if the sample is re-weighted by using the weights
defined as $\mathrm{w}=f(\mathbf{Z})/f(\mathbf{Z}|X)$, then in the created
pseudo-sample,

$\mathbb{E}\Bigl{[}\mathrm{w}Y|X(s)=x(s)\Bigr{]}=\mathbb{E}[Y(x)]\Bigr{]}.$
(7)

The left side of the equation is the expectation over the weighted pseudo-
sample and is the logic behind FIPW, while the right side is the expectation
after marginalization over $\mathbf{Z}$ and is the rationale behind the
regression adjustment. Proof details are included in Section 1.1 of the
supplementary material.

Unlike categorical treatments whose propensity scores are defined in terms of
conditional densities and are usually estimated directly, directly estimating
the weights via the definition of $w$ can be very challenging as the
(conditional) densities ($f(Z)$ and $f(Z\mid X$)) are difficult to estimate
and $Z$ is often multi-dimensional. Inspired by 2021_Xiaoke, the functional
propensity score (FPS) and corresponding weight are defined as

$s_{i}=f_{\mathbf{A}\mid\mathbf{Z}}(\mathbf{a_{i}}\mid\mathbf{z_{i}}),\quad
w_{i}=\frac{f_{\mathbf{A}}\left(\mathbf{A}_{i}\right)}{s_{i}}=\frac{f_{\mathbf{A}}\left(\mathbf{A}_{i}\right)}{f_{\mathbf{A}\mid\mathbf{Z}}(\mathbf{a_{i}}\mid\mathbf{z_{i}})},$

where $\mathbf{A}=(A_{1},...,A_{K^{*}})$ is the selected FPCS vector from FPCA
and $K^{*}$ can be different from $K$. $f_{\mathbf{A}\mid\mathbf{Z}}(\cdot)$
and $f_{\mathbf{A}}(\cdot)$ are the conditional and marginal probability
densities of $\mathbf{A}$. This way to calculate weights is the same as the
way to define $\mathrm{w}$, as derivation shown in Section 1.2 of the
supplementary material.

  * Proposition 2.

If $X(s)=\sum_{k=1}^{K^{*}}A_{i}\phi_{k}(s),s\in\mathcal{S}$, then
$w_{i}=\mathrm{w}_{i}$ for subject $i$.

For computation convenience, the standardized FPS $s_{i}^{*}$ and standardized
weights $w_{i}^{*}$ are usually utilized. Define
$s_{i}^{*}=f_{\mathbf{A}^{*}\mid\mathbf{Z}^{*}}(\mathbf{a_{i}}^{*}\mid\mathbf{z_{i}}^{*})$
and $\ w_{i}^{*}=f_{\mathbf{A}^{*}}\left(\mathbf{A}_{i}^{*}\right)/s_{i}^{*}$
with standardized covariates
$\mathbf{Z}_{i}^{*}=\boldsymbol{\Gamma}_{\mathbf{Z}}^{-1/2}\mathbf{Z}_{i},\boldsymbol{\Gamma}_{\mathbf{Z}}=E\left(\mathbf{ZZ}^{\top}\right)$
and standardized FPCS
$\mathbf{A}_{i}^{*}=\left(A_{i1}^{*},\ldots,A_{iK_{n}^{*}}^{*}\right)^{\top},A_{ik_{n}}^{*}=\lambda_{k}^{-1/2}A_{ik_{n}},\
k=1,\ldots,K_{n}^{*}$. Apparently, the standardized FPS automatically
satisfies the positivity and consistency assumption. To satisfy the
conditional exchangeability assumption, the weight vector $\boldsymbol{w}^{*}$
should be able to achieve the covariate balance condition in the sense that
minimizing the weighted correlation between $\mathbf{A}^{*}$ and
$\mathbf{Z}^{*}$, i.e.,
$\mathbb{E}\left(\boldsymbol{w}^{*}\mathbf{A}^{*}\mathbf{Z}^{*\top}\right)=0$.
The optimization details will be discussed in Section 4.1. With the estimated
weights $\boldsymbol{\hat{w}}$, FAFT regression is then performed on
$X_{i}(\cdot)$ the weighted survival outcome
$\hat{w}_{i}Y_{i}^{\text{Imputed}}$, where $Y_{i}^{\text{Imputed}}$ is
obtained based on equation (9), resulting in
$\hat{\beta}_{\text{FIPW}}(\cdot)$.

###  3.3 Double robust approach

The regression adjustment approach discussed in Section 3.1 requires the
correct regression model specification, while the FIPW approach proposed in
Section 3.2 is unstable when outliers exist. Therefore, the double robust
approach is proposed and the advantage of being less sensitive to model
misspecifications and data outliers, and attaining faster rates of convergence
when both models are consistently estimated (van2003unified). After the
regression adjustment, for subject-$i$, let $Y_{i}^{\text{Imputed}}$ be the
imputed outcome obtained during the estimation process of the full AFT and
$\hat{Y}_{\text{RegAdj},i}$ the fitted causal outcome. We construct a new
adjusted pseudo outcome as follows,

$\widetilde{Y}_{i}=\hat{Y}_{\text{RegAdj},i}+\hat{w}_{i}\left(Y_{i}^{\text{Imputed}}-\hat{Y}_{\text{RegAdj},i}\right),$
(8)

By applying $\left\\{\widetilde{Y}_{1},...,\widetilde{Y}_{n}\right\\}$ on
model (4), we can obtain the $\hat{\beta}_{\text{DR}}(\cdot)$.

##  4 Implementations of the approaches

The objective of Sections 2 and 3 is to present a comprehensive framework for
identifying the causal relationship between a functional treatment and a
survival outcome. In this section, we will provide in-depth information on the
estimation process and summarizes the algorithms utilized within this
functional causal survival framework.

###  4.1 FAFT estimation

With simpler notation, let $\boldsymbol{\theta}$ represent all unknown
parameters and $\mathbf{D}_{i}$ predictors, respectively. For example,
$\boldsymbol{\theta}=(\alpha,\beta_{1},...,\beta_{k})^{\top}$ in FAFT (4) and
$\boldsymbol{\theta}=(\alpha,\beta_{1},...,\beta_{k},\gamma_{1},...,\gamma_{p})^{\top}$
in full FAFT (5). There, we write models as
$Y_{i}=\boldsymbol{\theta}^{\top}\mathbf{D}_{i}+\epsilon_{i}$.

To fit the FAFT, we generalize the idea of the least squares method due to its
stability for a large number of predictors (jin2006least). To deal with the
right censoring, each response $Y_{i}$ is imputed by its conditional
expectation,

$Y_{i}^{\text{Imputed}}(\boldsymbol{\theta})=\delta_{i}Y_{i}+\left(1-\delta_{i}\right)\mathbb{E}_{\boldsymbol{\theta}}\left[Y_{i}\mid
Y_{i}\geqslant\log C_{i}\right],$ (9)

where the expectation is evaluated based on the Kaplan-Meier estimator
$\hat{F}(e)$. Let $r_{i}(\boldsymbol{\theta})=\widetilde{T}_{i}\ -\
\mathbb{E}_{\boldsymbol{\theta}}\left[Y|X_{i},\mathbf{Z}_{i}\right]$, then
$\hat{F}(e)=1-\prod_{\\{i:\mathbf{r}(i)\leqslant
e\\}}\left(\frac{n-i}{n-i+1}\right)^{\delta_{i}},$ where $\mathbf{r}(i)$ are
the ordered $r_{i}(\boldsymbol{\theta})$’s. Therefore the expectation in (9)
is estimated as

$\hat{\mathbb{E}}\left(Y_{i}\mid Y_{i}\geqslant\log
C_{i}\right)=\mathbb{E}_{\boldsymbol{\theta}}\left[Y|X_{i},\mathbf{Z}_{i}\right]+\hat{\mathbb{E}}\left[e_{i}\mid
e_{i}\geqslant r_{i}(\boldsymbol{\theta})\right],$ (10)

where $\hat{\mathbb{E}}\left[e_{i}\mid e_{i}\geqslant
r_{i}(\boldsymbol{\theta})\right]=\int_{r_{i}(\boldsymbol{\theta})}^{\infty}\frac{s}{1-\hat{F}\left(r_{i}(\boldsymbol{\theta})\right)}\mathrm{\leavevmode\nobreak\
d}\hat{F}(s)$. Given initial values $\hat{\boldsymbol{\theta}}^{(0)}$, the
least square estimator is the solution of the estimating equation
$U_{n}(\boldsymbol{\theta},\hat{\boldsymbol{\theta}}^{(0)})=\sum_{i=1}^{n}\left(\mathbf{D}_{i}-\bar{\mathbf{D}}\right)^{\top}\Bigl{(}Y_{i}^{*}(\hat{\boldsymbol{\theta}}^{(0)})-\allowbreak\boldsymbol{\theta}^{\top}\mathbf{D}_{i}\Bigr{)}\allowbreak=0.$
The estimation procedure can be proceeded iteratively by solving
$\hat{\boldsymbol{\theta}}^{(m)}_{n}=L_{n}\Bigl{(}\hat{\boldsymbol{\theta}}^{(m-1)}_{n}\Bigr{)},\allowbreak\\\
m\geqslant 1$, and

$L_{n}(\boldsymbol{\theta})=\left[\sum_{i=1}^{n}\left(\mathbf{D}_{i}-\bar{\mathbf{D}}\right)^{\top}\left(\mathbf{D}_{i}-\bar{\mathbf{D}}\right)\right]^{-1}\left[\sum_{i=1}^{n}\left(\mathbf{D}_{i}-\bar{\mathbf{D}}\right)^{\top}\left(Y_{i}^{\text{Imputed}}(\boldsymbol{\theta})-\bar{Y}_{i}^{\text{Imputed}}(\boldsymbol{\theta})\right)\right],$
(11)

where
$\bar{Y}_{i}^{\text{Imputed}}(\boldsymbol{\theta})=\frac{\sum_{i=1}^{n}Y_{i}^{\text{Imputed}}(\boldsymbol{\theta})}{n}$.
The whole estimation procedure is briefly summarized in Algorithm 1 in the
supplementary material.

In practical implementation, when the initial estimator is consistent and
asymptotically normal such as the induced smoothing Gehan estimator,
$\hat{\boldsymbol{\theta}}$ is also consistent and asymptotically normal
(jin2006least). The multiplier resampling can be applied to approximate the
variance of the resulting estimator.

###  4.2 Causal effect estimation

####  4.2.1 Regression adjustment approach

As details shown in Section 3.1, after fitting the full outcome regression
FAFT model, adjusting the confounding effects takes two more steps: firstly
reconstruct adjusted responses and then refit the FAFT. Algorithm 2 in the
supplementary material summarizes the whole procedure.

####  4.2.2 FIPW approach

Ideally, the functional weights are expected to achieve the balance condition
for each observed subject, i.e.,

$\mathbb{E}\left(w_{i}^{*}\mathbf{A}_{i}^{*}\mathbf{Z}_{i}^{*\top}\right)=0,\quad
i=1,...,n.$ (12)

The optimization process involves the selection of a tuning parameter, denoted
as $\rho$, which represents the level of tolerance for unbalancing in order to
resolve the non-convexity issue arising in equation (12). The recommended
value is $\rho_{0}=0.1/N$. For further details, refer to 2021_Xiaoke. After
getting $\hat{w}_{i}^{*}$ ’s, according to equation (7), a weighted pseudo-
sample, which is created by
$\boldsymbol{w}\circ\mathbf{Y}=\Bigl{\\{}w_{1}Y_{1}^{\text{Imputed}},...,w_{n}Y_{n}^{\text{Imputed}}\Bigr{\\}}$
with $Y_{i}^{\text{Imputed}}$ being calculated from equation (9), and will be
used to fit FAFT. Algorithm 3 in the supplementary material is a summary of
causal effect estimation.

####  4.2.3 Double robust approach

Since the double robust approach is a combination of regression adjustment and
FIPW, the corresponding algorithm also involves the implementation of
Algorithm 2 and Algorithm 3. The details are shown in Algorithm 4 in the
supplementary material.

##  5 A simulation study

To evaluate the finite-sample performance of the proposed methods, we conduct
simulation studies under two different scenarios, representing light and
strong effects of confounding variables. In both scenarios, we consider three
different censoring rates.

For the $i$-th subject, the functional treatment $X_{i}(\cdot)$ is given by
$X_{i}(s)=\sum_{k=1}^{K=6}A_{ik}\phi_{k}(s),s\in[0,1].$ The six eigenfunctions
are defined as $\phi_{2k-1}(s)=\sin{(2\pi ks)},\phi_{2k}(s)=\cos{(2\pi
ks)},k=1,2,3$. The corresponding FPCS are assumed to be
$A_{i1}=\sqrt{16}W_{i1},A_{i2}=\sqrt{12}W_{i2},A_{i3}=\sqrt{8}W_{i3},A_{i4}=\sqrt{4}W_{i4},A_{i5}=W_{i5}$,
and $A_{i6}=W_{i6}/\sqrt{2}$ , where $W_{i1},...,W_{i6}$ are simulated from a
multivariate normal distribution with mean $\mathbf{0}$ and covariance matrix
$\text{diag}(\sqrt{K},\dots,\sqrt{K})$ with $K=6$. Suppose that a three-
dimensional covariate vector
$\mathbf{Z}_{i}=\left(Z_{i1},Z_{i2},Z_{i3}\right)^{\top}$ can be observed from
each subject, simulating $\mathbf{Z}_{i}$ as
$Z_{i1}=W_{i1}+e_{i1},Z_{i2}=0.2W_{i2}+e_{i2}$, and $Z_{i3}=0.2W_{i3}+e_{i3}$
where $e_{i1}\sim\operatorname{N}(0,0.5)$ and
$e_{i2},e_{i3}\overset{\text{i.i.d}}{\sim}\operatorname{N}(0,1)$. The log-
transformed failure times $Y_{i}=\log T_{i}$ for all subjects are
independently generated under two distinct scenarios:

  * •

Scenario 1: $Y_{i}=\log
T_{i}=1+\int_{0}^{1}\beta_{0}(s)X_{i}(s)\mathrm{\leavevmode\nobreak\
d}s+2Z_{i1}+e_{i},\ e_{i}\sim\operatorname{N}(0,0.5)$;

  * •

Scenario 2: $Y_{i}=\log
T_{i}=1+\int_{0}^{1}\beta_{0}(s)X_{i}(s)\mathrm{\leavevmode\nobreak\
d}s+2Z_{i1}+2Z_{i1}^{2}\times A_{i1}+e_{i},\
e_{i}\sim\operatorname{N}(0,0.5)$;

in which the true functional coefficient is assume to be
$\beta_{0}(s)=2\sin(2\pi s)+\cos(2\pi s)+\sin(4\pi s)/2+\cos(4\pi
s)/2,s\in[0,1]$, representing the true causal effect of $X(\cdot)$. The
survival times are then transformed back to the original scale. The right
censoring is introduced by independently generating $C_{i}$ from
$\text{Uniform}(a,b)$ with different $a$ and $b$ being considered to determine
different right censoring rates. The observed survival outcome
$\widetilde{T}_{i}$ is decided by $\min\\{T_{i},C_{i}\\}$ and the censoring
indicator is $\delta_{i}=I(T_{i}\leqslant C_{i})$. When the sample size of the
study is $N$, the observed data are
$\mathcal{O}=\Big{\\{}\widetilde{T}_{i},\delta_{i},X_{i}(s),Z_{i1},Z_{i2},Z_{i3},\quad
i=1,...,N\Big{\\}}.$

The analysis goal is to get the estimated causal effect $\hat{\beta}(\cdot)$
using $\mathcal{O}$. The naive way fits the model (4) directly, assuming no
confounding effect. However, in both scenarios considered here, among the
three generated covariates, $Z_{i1}$ is directly related to both
$X_{i}(\cdot)$ and $Y_{i}$, while $Z_{i2}$ and $Z_{i3}$ are only directly
related to $X_{i}(\cdot)$ but not $Y_{i}$. Such existence of confounding
variables may bring bias to the estimates of the treatment effect if not
appropriately accounted for. Especially in scenario 2, the non-linear
relationship between $Z_{i1}$ and $Y_{i}$ may pose additional challenges for
modeling and estimation. We will apply the proposed methods to adjust
confoundings and compare them with the naive approach.

Each scenario considered three levels of right censoring (20%, 40%, and 60%)
and two sample sizes ($N=200$ and $400$), and each setup is repeated 500
times. The selection criterion for both $K_{n}$ and $K_{n}^{*}$ is set to be
PVE=95%.

###  5.1 Evaluation measures

To assess the accuracy and validity of each estimator and provide insights
into their strengths and weaknesses in handling linear and non-linear
confounding effects, as well as right censoring, we consider the following
measures.

  * •

Define the relative mean square error as
$\operatorname{RMSE}=\Bigl{(}\int_{\mathcal{S}}(\hat{\beta}(s)-\beta_{0}(s))^{2}\mathrm{\leavevmode\nobreak\
d}s\Bigr{)}/\allowbreak\Bigl{(}\int_{\mathcal{S}}\beta_{0}^{2}(s)\mathrm{\leavevmode\nobreak\
d}s\Bigr{)},$ where $\hat{\beta}(\cdot)$ is the estimated mean coefficient
function over all simulation runs. A lower value indicates a more accurate
estimate to the true functional causal effect.

  * •

Define the integrated squared bias as
$\operatorname{ISB}=\left(\sum_{\mathcal{S}}(\hat{\beta}(s)-\beta_{0}(s))^{2}\right)/|\mathcal{S}|,$
where $|\mathcal{S}|$ is the $L_{2}$ distance of the domain and
$|\mathcal{S}|=1$ in our situation. It characterizes the accuracy relative to
the domain and a lower ISB indicates better accuracy.

  * •

The averaged integrated squared error (AISE) is the mean of the integrated
squared error (ISE) over all simulation runs. In the $r$-th simulation run,
$\operatorname{ISE}_{\text{sim-r}}=\Bigl{(}\int_{\mathcal{S}}(\hat{\beta}_{\text{sim-r}}(s)-\beta_{0}(s))^{2}\allowbreak\mathrm{\leavevmode\nobreak\
d}s\Bigr{)}/|\mathcal{S}|,$ where $\hat{\beta}_{\text{sim-r}}(\cdot)$ is the
estimated functional coefficient. The AISE illustrates the accuracy over all
simulation runs and its standard error (SE) measures the estimation variation.
By contrast, the median ISE (MISE) provides an additional summary of the
accuracy, which is less sensitive to outliers than AISE.

To examine the prediction performance, the 80% / 20% splitting rule was
applied on each generated dataset for training and testing. In the $r$-th
simulation run, the root mean squared error is calculated as

$\operatorname{Root-
MSE}_{\text{sim-r}}=\left(\sqrt{\sum_{i=1}^{n}(\hat{Y}_{\text{sim-r},i,\text{pred}}-Y_{\text{sim-r},i,\text{causal}})^{2}/n}\right),$

where $\hat{Y}_{\text{sim-r},i,\text{pred}}$ is the fitted value and
$Y_{\text{sim-r},i,\text{causal}}$ is the generated true causal outcome for
subject $i$. This measure assesses the accuracy of the predicted outcome
compared to the true causal outcome. We report its mean and three quantiles
($q_{.25},q_{.50}$, and $q_{.75}$) over all runs.

###  5.2 Simulation results

The proposed three causal approaches gave rise to five causal estimators,
namely RegAdj, FIPW.para (FIPW with weights estimated parametrically), FIPW.np
(FIPW with weights estimated nonparametrically), DR.para (DR with weights
estimated parametrically), and DR.np (DR with weights estimated
nonparametrically). The results presented below are based on $N=400$, with
similar results for $N=200$ included in Section 3 of the supplementary
material.

####  5.2.1 Results of estimation

Table 1 summarizes four evaluation measures that assess the precision of the
estimated functional coefficient $\hat{\beta}(\cdot)$ under two scenarios,
each with three levels of censoring rates. In scenario 1, where the
confounding effect is weak and the dependence relationship is linear, all
proposed causal estimators exhibit smaller errors than the naive estimator.
When it comes to scenario 2 where the confounding effect is stronger and the
correlation is more complex, the naive approach introduces greater bias to
causal estimation. Instead, the proposed methods and algorithms successfully
handle the situation. The much smaller values in means of RMSE, AISE, MISE,
and ISB indicate a substantial improvement in estimation accuracy, and the
significant reduction in SE of AISE demonstrates the stability of the proposed
algorithms.

The estimated mean functional coefficients are displayed in Figure 1, with
different estimators represented by variations in line colors and point
shapes. The shaded area represents the confidence interval (CI) for
$\hat{\beta}(\cdot)$ over all replications. As shown in Figure 1, the naive
estimator (the dark yellow line with dimond shapes) deviates away from the
truth (the black line with squares) and has a 95% CI that does not cover the
truth in scenario 1, which is far worse in scenario 2. By contrast, the five
newly proposed estimators are shown to be close to $\beta_{0}(\cdot)$. In
scenario 1, three estimators (DR.para in bright yellow, DR.np in dark blue,
and RegAdj in orange) overlap with the truth, indicating less estimation bias
than FIPW.para and FIPW.np. This is because the assumed model in the
regression adjustment approach is similar to the true model used to generate
our data. When such an assumption is heavily violated by the fact, which is
the scenario 2, the FIPW approach (FIPW.para and FIPW.np) appears to have the
advantage due to its ability to handle model misspecification. Another finding
is that the FIPW approach yields a larger SE of AISE among five new estimators
under scenario 1, while it becomes much smaller than others under scenario 2.
This difference is due to the possibility of extreme estimated weights, which
add instability to the estimation process. When the imbalance among
confounding variables is weak (as in scenario 1), this instability contributes
a larger proportion to the total variation, leading to a larger SE than
others. In contrast, when the imbalance among confounding variables is
stronger (as in scenario 2), the extreme estimated weights have less impact on
the total variation, resulting in a smaller SE than others.

When looking at the DR approach, we can find that it performs between the
regression adjustment and the FIPW approach, reflecting its "double robust"
property. This suggests that the DR approach outperforms the regression
approach when the regression model is misspecified and is more stable when the
estimated weights exist with extreme values.

####  5.2.2 Results of prediction

Table 2 includes in-sample and out-sample Root-MSE measures, displayed as the
mean, 25%, 50%, and 75% quantiles ($q_{.25},q_{.50}$, and $q_{.75}$). Compared
to the naive estimator, the proposed approaches show better accuracy and
efficiency in predicting causal outcomes, illustrated by a lower mean of Root-
MSE and less variation among quantiles. Even though all estimators decrease in
performance with more complex confounding effects and higher censoring rates,
the proposed methods always outperform the naive approach in a more robust
way. When comparing among five proposed estimators, once again, the double
robust (DR.para and DR.np) and regression adjustment approaches show more
accurate predictions in scenario 1, while the FIPW approach (FIPW.para and
FIPW.np) performs better in scenario 2, as discussed in Section 5.2.1.

####  5.2.3 Different right censoring rates

Higher right censoring rates consistently lower estimation and prediction
accuracy across scenarios and estimators due to information loss resulting
from censoring. This creates uncertainty during imputation, leading to
increased bias and variation in causal estimates and predictions.

As observed in Table 1, the SE of AISE for all estimators increases with
increasing censoring rates under both scenarios. While the mean of AISE
remains relatively stable under scenario 1, it exhibits an obvious increasing
trend under scenario 2, particularly there is a big jump in AISE and its SE in
the transition from a 40% to a 60% censoring rate. However, in comparison to
the naive estimator, the proposed methods exhibit greater reliability and
robustness. In particular, the FIPW demonstrates superiority over the DR in
scenario 2 as the censoring rate increases. This is due to the increased bias
that arises from utilizing the imputed outcomes of censored subjects as their
true observations during the final step of estimating the DR estimators. This
bias becomes more substantial and contributes to a decrease in estimation
accuracy in the presence of a misspecified regression model in scenario 2. A
similar pattern can also be observed in Table 2 with increasing censoring
rates.

##  6 An application on AD

We implemented the proposed functional causal framework to analyze the data
from the ADNI observational study, which comprises survival information about
AD diagnosis, clinical measurements, and MRI records. In contrast to previous
studies, we aimed to identify the causal relationship between subregions
within the hippocampus and the time of progression from MCI to AD, to
contribute to a better understanding of the mechanisms underlying AD
progression, and to provide insight into potential targets for early
intervention.

For our purpose, 373 MCI subjects in ADNI-1 study along with their imaging and
clinical measures are considered in the analysis. Among them, 161 MCI subjects
progressed to AD during the study and the remaining 212 MCI subjects did not
convert to AD prior to the study’s end. Thus, the time of conversion from MCI
to AD should be treated as time-to-event data with a censoring rate of 56.9%.
The clinical characteristics of participants include age, education length,
ADAS-Cog score, gender (0=male; 1=female), handedness (0=right; 1=left),
marital status (0=single; 1=married), retirement (0=no; 1=yes), and
apolipoprotein E (APOE) genetic covariates. The APOE is defined based on two
single nucleotide polymorphisms (SNPs), rs429358 and rs7412, and produced a
3-allele haplotype, resulting in $\epsilon 2,\epsilon 3,$ and $\epsilon 4$
variants. Descriptive statistics were summarized in Table 4, after
categorizing the 373 MRI subjects into four groups based on their observed
survival outcomes. The hippocampus image data was treated as a functional
treatment and was represented as a matrix with dimension $(N,p)=(373,30000)$
after being preprocessed (2018_Kong). Each row contained an subject’s
hippocampal radial distances of 30,000 surface points on the left and right
hippocampus surfaces, which is a summary statistic of the hippocampal shape
and size and defined as the distance between the medial core of the
hippocampus and the corresponding vertex.

Specifically, we applied FPCA to the treatment with the top 12 FPCS
($K_{n}$=12, PVE=70%) selected to summarize the image data. To investigate
potential correlations between the functional treatment and each clinical
measurement, we calculated the absolute Pearson correlation between each of
the top 13 FPCs and each continuous covariate, as well as the absolute Point-
Bserial correlation for each categorical covariate. We reported the weighted
absolute Pearson and Point-Bserial correlation to assess the performance of
the estimated weights in improving covariate balance. We calculated weights
both parametrically, and nonparametrically using three different values of the
tuning parameter: $\rho_{0}=0.1/N$ (default), $\rho_{1}=1/N$ (more tolerance
of imbalance), and $\rho_{2}=0.01/N$ (lower tolerance). The results are
denoted as unweighted, weighted.para, weighted.np.0, weighted.np.1, and
weighted.np.2 in Figure 2 (a). As shown in all eight plots, the treatment is
correlated with all covariates. The usage of weights is able to reduce the
correlation over the top 12 FPCs in general, but it is obvious to see that the
nonparametrically estimated weights perform better in improving covariate
balance conditions. In addition, nonparametric weighting performs pretty
robustly to different $\rho$’s. For age, handedness, and retirement,
nonparametric weighting effectively lowers down the correlation value of the
first FPC and keeps the correlation of the second to twelfth FPCs the lowest.
In contrast, weights.para reduces the imbalance less, possibly due to
misspecified Gaussian distributions when calculating the weights
parametrically.

We evaluated the causal effect using all proposed estimators. When using three
different tuning parameters in FIPW.np estimation, the results are pretty
close to each other. The FIPW.para also gives similar estimations. However,
the regression adjustment approach yields different estimators from the FIPW
approach and the double robust approach performs between them. This might be
because they fit FAFT twice and thus rely more on the imputed outcomes. Since
the censoring rate is pretty high in the study, the resulting estimators are
influenced, as shown in the simulation study. We included all causal
estimators in Section 3 of the supplementary material and presented the
FIPW.np.0 (FIPW.np using $\rho_{0}=0.1/N$) estimator in Figure 2 (b) and (c).
It can be clearly seen that the subfield of CA1 on both hippocampus has a
negative effect on the survival time, indicating that the thicker these areas
on the hippocampus are, the shorter the time is to convert to AD. Compared to
the work of 2018_Kong and 2021_Yang, our work identified more precise and
refined subregions. Specifically, the previous works focused on the
conditional effect on general subfields as shown in panel (d). As shown in
panel (e), our work examined the causal effect and gave more precise
subregions, which is in line with the findings in the medical research
(bienkowski2018integration). The reason for such a causal effect lies in the
accumulation of neurofibrillary tangles (NFTs) first in the CA1 area and then
gradually in the subiculum, CA2, CA3, and DG (rao2022hippocampus), which might
be considered as potential targets for early intervention cure.

##  7 Discussion

The main contribution of this paper is the development of the FAFT model and a
causal inference framework for observational studies involving time-to-event
data and infinite-dimensional functional treatments. Due to the lack of a
direct functional survival model, we propose a FAFT model to incorporate
infinite-dimensional imaging data for a survival outcome. Three causal
inference approaches were developed to adjust confounding effects and result
in causal estimators. The results of the simulation study indicated the
appealing numerical performance of the proposed methods in balancing
confounding variables, estimating causal effects, and predicting survival
outcomes, and demonstrated the robustness with respect to different right
censoring rates. In addition, we successfully identified more precise
subregions of the hippocampus brain area and made the first endeavor to study
its causal effect on conversion time from MCI to AD.

The proposed methods may be generalized to other applications. For example, it
is almost directly applicable to handling multidimensional continuous
treatments (e.g., kong2019multi), multidimensional functional treatments, and
functional/categorical outcomes. With slight modifications, the proposed
methods can be used to study the interval-censored data and other survival
models such as the generalized transformation model. We list several
directions for future work. The proposed weights are only capable of
controlling the correlation between treatments and covariates. In order to
improve the stability of weights and the performance of weighted estimators,
the weights could be constructed to control the balancing error of the
treatments and any outcome function on confounders. A possible solution could
be to estimate weights via the kernel method based on a reproducing kernel
Hilbert space (wang2021estimation) Another direction is to incorporate a
longitudinal assessments of the functional treatment, such as multiple MRI
images over 5 years for one subject. In this situation, a shared confounding
structure is needed in order to emphasize the identifiability and estimation
of causal effects. (kong2022identifiability). Thus, it is possible to
generalize our method by including the latent confounding structure for a
better understanding of disease progression and higher accuracy of survival
prediction.

Figure 1: Simulation results for evaluating the estimated functional causal
effect curve under two scenarios with three censoring rates when sample size
N=400. Different estimators are represented by line color and point shape with
the shading area as the corresponding CI. Figure 2: ADNI data analysis
results: (a) The covariate balancing comparison between the first thirteen
FPCS of the image and each covariate when using equal weights (unweighted),
parametric weights (weighted.para), and nonparametric weights with
$\rho_{0}=0.1/N$ (weighted.np.0), $\rho_{1}=1/N$ (weighted.np.1), and
$\rho_{2}=0.01/N$ (weighted.np.2). The weighted absolute Pearson and Point-
Biserial correlation is calculated for continuous and categorical covariates,
respectively. (b) The color bar illustration. (c) The estimated coefficient
function $\hat{\beta}_{\text{FIPW.np.0}}(\cdot)$. (d) The hippocampal
subfields from 2018_Kong. (e) Newly found refined organizational hippocampal
subfields by bienkowski2018integration. Table 1: Simulation results for
estimation accuracy for the functional causal curve $\beta(\cdot)$ under two
scenarios with three censoring rates when the sample size is $N=400$.

Different right censoring rates 20% 40% 60% RMSE AISE (SE) MISE ISB RMSE AISE
(SE) MISE ISB RMSE AISE (SE) MISE ISB Scenario 1 Naive 0.18 0.50(0.036) 0.50
0.49 0.18 0.50(0.040) 0.50 0.49 0.18 0.50(0.056) 0.50 0.49 RegAdj 0.00
0.00(0.002) 0.00 0.00 0.00 0.00(0.002) 0.00 0.00 0.00 0.00(0.003) 0.00 0.00
FIPW-para 0.02 0.23(0.564) 0.12 0.06 0.01 0.22(0.590) 0.12 0.03 0.01
0.30(0.645) 0.15 0.02 FIPW-np 0.02 0.09(0.065) 0.08 0.05 0.03 0.13(0.077) 0.11
0.08 0.04 0.16(0.091) 0.15 0.10 DR.para 0.00 0.00(0.004) 0.00 0.00 0.00
0.00(0.004) 0.00 0.00 0.00 0.01(0.004) 0.00 0.00 DR.np 0.00 0.00(0.002) 0.00
0.00 0.00 0.00(0.002) 0.00 0.00 0.00 0.00(0.003) 0.00 0.00 Scenario 2 Naive
28.90 83.15(30.207) 77.74 79.19 49.37 142.67(57.488) 131.52 135.29 92.51
268.10(115.493) 239.73 253.50 RegAdj 4.38 14.74(9.766) 12.79 12.01 9.94
31.82(19.470) 26.84 27.25 21.35 67.11(42.056) 55.76 58.52 FIPW-para 2.63
7.92(3.035) 7.30 7.20 3.09 9.49(4.770) 8.44 8.47 4.27 13.64(8.566) 11.72 11.70
FIPW-np 1.18 3.51(1.809) 3.11 3.22 1.32 3.96(2.199) 3.41 3.61 1.30 4.11(2.933)
3.27 3.55 DR.para 1.64 6.40(4.824) 5.28 4.49 5.72 19.53(13.151) 16.64 15.68
14.44 48.66(34.866) 39.58 39.57 DR.np 2.09 6.90(5.405) 5.70 5.71 6.58
20.93(13.842) 17.66 18.02 16.14 51.58(34.104) 42.33 44.23

Table 2: Simulation results for prediction accuracy in terms of Root-MSE for
the survival outcome $Y=\log T$ under two scenarios, both in-sample and out-
sample, and with three censoring rates when sample size being $N=400$.

Different right censoring rates 20% 40% 60% Mean $q_{.25}$ $q_{.50}$ $q_{.75}$
Mean $q_{.25}$ $q_{.50}$ $q_{.75}$ Mean $q_{.25}$ $q_{.50}$ $q_{.75}$ Scenario
1 In sample Naive 2.07 2.00 2.07 2.13 2.06 2.00 2.06 2.13 2.07 1.98 2.06 2.15
RegAdj 0.52 0.50 0.51 0.53 0.52 0.50 0.52 0.54 0.53 0.51 0.52 0.54 FIPW-para
1.35 0.92 1.20 1.59 1.45 1.02 1.26 1.72 1.96 1.32 1.63 2.23 FIPW-np 1.03 0.86
1.03 1.18 1.17 0.99 1.17 1.33 1.37 1.17 1.37 1.54 DR.para 0.52 0.50 0.52 0.54
0.53 0.50 0.52 0.54 0.53 0.51 0.53 0.55 DR.np 0.52 0.50 0.52 0.53 0.52 0.50
0.52 0.54 0.53 0.51 0.52 0.55 Out sample Naive 2.04 1.92 2.04 2.15 2.04 1.91
2.04 2.16 2.05 1.91 2.03 2.17 RegAdj 0.53 0.49 0.52 0.56 0.53 0.49 0.52 0.56
0.53 0.50 0.53 0.57 FIPW-para 1.36 0.94 1.23 1.61 1.47 1.03 1.29 1.76 1.97
1.31 1.66 2.28 FIPW-np 1.05 0.87 1.03 1.20 1.18 1.01 1.18 1.34 1.38 1.17 1.36
1.56 DR.para 0.53 0.49 0.52 0.56 0.53 0.50 0.53 0.57 0.54 0.50 0.53 0.57 DR.np
0.53 0.49 0.52 0.56 0.53 0.50 0.52 0.56 0.54 0.50 0.53 0.57 Scenario 2 In
sample Naive 25.56 21.98 25.09 28.27 34.32 28.97 33.29 38.02 51.35 42.90 49.05
58.01 RegAdj 10.28 7.98 10.02 12.11 17.82 13.82 17.02 20.46 33.73 27.12 31.51
38.44 FIPW-para 7.93 6.85 7.73 8.83 8.71 7.18 8.43 9.85 10.86 8.51 10.36 12.65
FIPW-np 5.24 4.44 5.07 5.87 5.55 4.58 5.35 6.25 5.52 4.24 5.20 6.50 DR.para
6.73 5.10 6.39 8.06 14.23 10.88 13.73 16.56 27.99 22.52 26.16 31.96 DR.np 7.06
5.20 6.80 8.51 14.27 10.99 13.49 16.54 27.95 22.44 26.59 31.88 Out sample
Naive 25.35 22.08 24.72 28.35 34.11 28.93 33.00 38.46 51.15 42.37 49.32 57.07
RegAdj 10.14 7.87 9.81 11.85 17.73 13.95 16.96 20.37 33.66 27.18 31.98 38.45
FIPW-para 7.84 6.82 7.66 8.71 8.63 7.10 8.35 9.78 10.82 8.37 10.32 12.62 FIPW-
np 5.19 4.31 4.99 5.89 5.50 4.53 5.30 6.28 5.48 4.15 5.13 6.44 DR.para 6.66
4.95 6.34 7.96 14.17 11.01 13.47 16.52 27.94 22.13 26.41 31.79 DR.np 6.97 5.14
6.76 8.48 14.20 11.03 13.68 16.58 27.89 22.18 26.44 31.91

Table 3: Simulation results of prediction accuracy in terms of Root-MSE for
the survival outcome $Y=\log T$ under two scenarios, both in-sample and out-
sample, and with three censoring rates when sample size being $N=200$.

Different right censoring rates 20% 40% 60% Mean $q_{.25}$ $q_{.50}$ $q_{.75}$
Mean $q_{.25}$ $q_{.50}$ $q_{.75}$ Mean $q_{.25}$ $q_{.50}$ $q_{.75}$ Scenario
1 In sample Naive 2.08 1.98 2.06 2.17 2.08 1.97 2.07 2.18 2.09 1.97 2.07 2.21
RegAdj 0.53 0.50 0.53 0.56 0.54 0.50 0.53 0.57 0.55 0.51 0.55 0.59 FIPW.para
1.35 0.94 1.22 1.53 1.48 1.02 1.28 1.74 1.92 1.29 1.66 2.20 FIPW.np 1.25 1.01
1.25 1.46 1.41 1.14 1.41 1.67 1.67 1.36 1.63 1.94 DR.para 0.54 0.51 0.53 0.57
0.55 0.51 0.54 0.58 0.56 0.52 0.55 0.59 DR.np 0.54 0.50 0.53 0.57 0.54 0.51
0.54 0.58 0.56 0.52 0.55 0.59 Out sample Naive 2.03 1.85 2.03 2.20 2.03 1.86
2.03 2.21 2.04 1.84 2.03 2.23 RegAdj 0.55 0.49 0.54 0.59 0.56 0.50 0.54 0.60
0.57 0.51 0.56 0.62 FIPW.para 1.37 0.94 1.24 1.59 1.51 1.02 1.33 1.82 1.94
1.31 1.66 2.27 FIPW.np 1.27 1.02 1.24 1.51 1.44 1.14 1.42 1.70 1.68 1.33 1.61
1.96 DR.para 0.56 0.50 0.55 0.60 0.56 0.51 0.55 0.61 0.57 0.51 0.56 0.62 DR.np
0.55 0.50 0.55 0.60 0.56 0.50 0.55 0.61 0.57 0.51 0.56 0.62 Scenario 2 In
sample Naive 25.91 20.79 24.76 29.82 34.75 26.92 32.80 41.22 53.31 40.10 50.56
63.61 RegAdj 10.59 7.59 9.84 12.82 18.10 13.05 16.75 21.40 35.31 25.77 33.07
41.68 FIPW.para 7.83 6.52 7.58 8.94 8.58 6.75 8.31 9.89 10.76 7.54 10.01 12.87
FIPW.np 4.71 3.46 4.42 5.77 4.98 3.69 4.65 6.09 4.81 3.05 4.25 6.08 DR.para
7.21 4.94 6.64 8.74 14.65 10.56 13.49 17.37 29.78 21.35 27.39 35.37 DR.np 7.38
4.93 6.76 9.08 14.63 10.38 13.51 17.37 29.82 21.45 27.68 35.51 Out sample
Naive 25.46 20.38 24.57 29.29 34.36 26.60 33.00 40.44 53.01 39.86 50.25 62.31
RegAdj 10.35 7.33 9.75 12.48 17.99 13.00 16.57 21.56 35.34 25.37 32.94 42.39
FIPW.para 7.66 6.37 7.36 8.70 8.44 6.72 8.05 9.73 10.67 7.65 9.76 12.49
FIPW.np 4.60 3.39 4.34 5.63 4.87 3.59 4.53 5.85 4.71 2.97 4.16 5.88 DR.para
7.12 4.86 6.63 8.68 14.61 10.68 13.35 17.47 29.79 21.44 27.85 35.79 DR.np 7.22
4.92 6.74 8.99 14.53 10.29 13.34 17.50 29.79 21.27 27.78 35.66

Table 4: Summary statistics for clinical measures after categorizing 373 MRI
individuals into four groups according to their observed survival outcome,
either the progression time from MRI to AD or the right censored time. The
sample mean and standard error are reported for continuous variables. Sample
frequency and proportion are reported for categorical variables.

Continuous: mean(se) Categorical: frequency(proportion) allele.1 allele.2
Total age edu adas male right married retire geno.2 geno.3 geno.4 geno.3
geno.4 $\widetilde{T}\leqslant$ 1 yr 57 74(7.3) 15(3.5) 15(4.8) 40(70%)
51(89%) 46(81%) 12(21%) 0(0%) 52(91%) 5(9%) 22(39%) 35(61%) 1 yr
$<\widetilde{T}\leqslant$ 2 yr 117 75(7.2) 16(2.9) 12(4.0) 69(59%) 107(91%)
92(79%) 15(13%) 8(7%) 90(77%) 19(16%) 47(40%) 70(60%) 2 yr
$<\widetilde{T}\leqslant$ 3 yr 91 75(7.5) 16(2.7) 10(4.4) 58(64%) 86(95%)
73(80%) 24(26%) 10(11%) 68(75%) 13(14%) 40(44%) 51(56%) $\widetilde{T}>$ 3 yr
108 75(7.4) 16(3.0) 10(3.9) 70(65%) 98(91%) 89(82%) 19(18%) 8(7%) 90(83%)
10(9%) 60(56%) 48(44%) 373 75(7.3) 16(3.0) 12(4.5) 237(64%) 342(92%) 300(80%)
70(19%) 26(7%) 300(80%) 47(13%) 169(45%) 204(55%)


# A Study of Designing Compact AUDIO-VISUAL WAKE WORD SPOTTING SYSTEM BASED ON
ITERATIVE FINE-TUNING IN NEURAL NETWORK PRUNING

###### Abstract

Audio-only based wake word spotting (WWS) is challenging under noisy
conditions due to the environmental interference in signal transmission. In
this paper, we investigate on designing a compact audio-visual WWS system by
utilizing the visual information to alleviate the degradation. Specifically,
in order to use visual information, we first encode the detected lips to
fixed-size vectors with MobileNet and concatenate them with acoustic features
followed by the fusion network for WWS. However, the audio-visual model based
on neural network requires a large footprint and a high computational
complexity. To meet the application requirements, we introduce neural network
pruning strategy via the lottery ticket hypothesis in an iterative fine-tuning
manner (LTH-IF), to the single-modal and multi-modal models, respectively.
Tested on our in-house corpus for audio-visual WWS in a home TV scene, the
proposed audio-visual system achieves significant performance improvements
over the single-modality (audio-only or video-only) system under different
noisy conditions. Moreover, LTH-IF pruning can largely reduce the network
parameters and computations with no degradation of WWS performance, leading to
a potential product solution for the TV wake-up scenario.

Index Terms—  Wake word spotting, noisy environments, audio-visual, LTH
pruning, iterative fine-tuning

##  1 Introduction

Wake word spotting (WWS) can be considered as a specific case of keyword
spotting (KWS), concerning the identification of pre-defined wake word(s) in
utterances, often used for the wake-up of speech-enabled devices, such as “Hey
Siri” in iPhone, “Alexa” in Amazon Echo, and “Ok Google” in Google Home [1, 2,
3, 4], etc. In order to activate the interactions between devices and users, a
standby wake word detection module is particularly important [5].

Traditional approaches in keyword spotting tasks involve the keyword/filler
hidden Markov model (HMM) [6, 7], namely training an HMM for the keyword and a
filler HMM for the non-keyword segments, respectively. Recently, deep learning
based keyword spotting have attracted much attention. Chen et al. proposed a
simple discriminative keyword spotting approach based on deep neural networks
which have improved the performance of system [8]. The first attempt to use
convolutional neural networks (CNNs) for keyword spotting, by Sainath and
Parada [9], was recently improved by jointly integrating deep residual
learning and dilated convolutions [10]. Arik et al. [11] also applied the
convolutional recurrent neural network (CRNN) architecture to single English
keyword detection. With the achievements of Transformer [12] in the field of
deep learning, several variants of Transformers for wake word detection are
explored in [13]. Besides, more efficient networks have been also investigated
by leveraging recent advances in differentiable neural architecture search
[14].

Despite the above research progress, KWS is still a challenging task and has
attracted the attention of speech researchers. On the one hand, the KWS
systems usually perform very well under clean speech conditions. However, the
systems suffer from sharp performance degradation under noisy environments.
The authors in [15] propose integrating multiple beamformed signals and
leveraging the attention mechanism to dynamically tune the model’s attention
to the reliable input sources to improve the KWS performance under noisy and
far-field conditions. The data-efficient solutions are presented in [16] to
improve the model robustness in WWS modeling under noisy conditions. A multi-
task network that performs KWS and speaker verification (SV) simultaneously is
also proposed in [17] to fully utilize the interrelated domain information
aiming at performance improvement in challenging conditions. In addition, the
authors in [18] have developed a novel tuple-based loss function along with a
training strategy for noise-robust keyword spotting.

Fig. 1: The architecture of proposed audio-visual wake word spotting with
neural network pruning using LTH-IF.

On the other hand, the WWS system usually runs on smart devices, it’s critical
to design the model with a small footprint and low computational power. The
application of multi-scale temporal modeling to the small-footprint keyword
spotting task has been explored in [19]. The authors also explore the latency
and accuracy of KWS models in streaming and non-streaming modes for
simplifying model deployment on mobile devices [20]. In [21], the researchers
designed different models and neural architectures for small footprint keyword
spotting. Different loss functions for the training of a small-footprint KWS
system have also been explored in [22]. Recently, in order to optimize towards
memory footprint and execution time, power-consuming audio preprocessing and
data transfer steps are eliminated in [23] by directly classifying from raw
audio.

In order to alleviate the performance degradation of WWS under noisy
conditions, in this paper, we investigate an audio-visual WWS system by
utilizing the visual information. First, the detected lips are encoded to
fixed-size vectors with MobileNet, and concatenated with acoustic features.
Next, a neural network pruning strategy, i.e the lottery ticket hypothesis
based iterative fine-tuning (LTH-IF) is introduced to the WWS systems.
Finally, tested on our in-house corpus for audio-visual WWS in a home TV
scene, the proposed audio-visual system achieves significant performance
improvements over the single-modality system under different noisy conditions.
Moreover, LTH-IF pruning can largely reduce the network parameters and
computations with no degradation of WWS performance, leading to a potential
product solution for the TV wake-up scenario.

##  2 PROPOSED APPROACH

###  2.1 Audio-Visual Model for Wake Word Spotting

Inspired by the work in [24], we design the proposed audio-visual wake word
spotting (WWS) architecture in an end-to-end manner. The main difference from
[24] lies on that we have the only one wake word, so we do not need to use the
phonetic sequence or calculate the similarity matrix. Accordingly we directly
adopt the classification task through the fully connected (FC) layers. The
overall flowchart of proposed audio-visual WWS architecture is shown in Fig.
1, which mainly consists of three parts: audio stream, video stream and fusion
stream. The details will be elaborated in the following subsections.

####  2.1.1 Audio Stream

For the audio stream, the acoustic features extracted frame by frame are
selected as input features. Here, we employ 40-dimensional filter bank (FBank)
features normalized by global mean and variance. Given the raw input audio
data ${I}_{\text{A}}$, we can calculate normalized FBank features
${F}_{\text{A}}$ through the FBank extractor $f_{\text{A}}$:

${F}_{\text{A}}=f_{\text{A}}({I}_{\text{A}})$ (1)

####  2.1.2 Video Stream

For the video stream, considering practicality and lightweight, we select a
combination of MobileNetV2 [25] as our video embedding extractor. In addition,
we replace bidirectional long short-term memory (BLSTM) with LSTM to further
reduce the model size and latency. In this study, 13 linear bottlenecks are
first adopted as a lip feature extractor $f_{\text{V}}$. Given the input image
${I}_{\text{V}}$, the lip feature ${F}_{\text{V}}$ is calculated through
$f_{\text{V}}$:

${F}_{\text{V}}=f_{\text{V}}({I}_{\text{V}})$ (2)

The gray scale lip ${F}_{\text{V}}$ reshaped to 88 $\times$ 88 is used as the
MobileNetV2 input, and the output is an encoded vector by using average
pooling. The high-level video embeddings ${E}_{\text{V}}$ can be obtained by
MobileNetV2 $f_{\text{Mobile}}$:

${E}_{\text{V}}=f_{\text{Mobile}}({F}_{\text{V}})$ (3)

For more details, please refer to [25].

####  2.1.3 Fusion Stream

For the audio-visual fusion stream, a direct concatenation for audio-visual
fusion at the encoder is first considered to integrate information from two
sources ${F}_{\text{A}}$ and ${E}_{\text{V}}$:

${F}_{\text{AV}}=[{F}_{\text{A}},{E}_{\text{V}}]$ (4)

Then a mixture of convolutional layers, LSTM layers and FC layers are designed
to generate the final output as shown in Fig. 1. For the single-modality
network, the above audio-visual features ${F}_{\text{AV}}$ are replaced by
audio acoustic features ${F}_{\text{A}}$ and visual embeddings
${E}_{\text{V}}$, respectively. In this study, 5 epochs ($E=5$) are selected
to train these three WWS models, namely audio-only model, video-only model,
and audio-visual model. The final output of these models is compared with the
preset threshold after sigmoid operation, ‘1’ indicates that the current
sample contains wake word, and ‘0’ indicates the opposite. Given a sample, the
model ($G$) outputs a probability $p(y=1|\Theta)$ representing the possibility
that the wake word is included, where $\Theta$ represents the model parameter
set. The optimisation objective is a binary cross-entropy loss between this
prediction and the ground truth label $y^{*}$:

$\ L_{\text{WWS}}\\!=\\!-y^{*}\log\
p(y\\!=\\!1|\Theta)\\!-\\!(1\\!\\!-\\!y^{*})\
\log(1\\!\\!-p(y\\!=\\!1|\Theta))\ $ (5)

Algorithm 1 LTH with Iterative Fine-tuning

1:A model, $G_{0}$

2:Randomly initialize weights $(\Theta_{0})$

3:Initialize model: $G_{0}(\Theta_{0})\rightarrow G_{1}$

4:For $t=1,\cdots,T:\\#\ Pruning\;searching\;iterations$

5: If $t=1$:

6: For $e=1,\cdots,E:\\#\;Training\;epochs$

7: $\Theta_{e}\rightarrow\Theta$ : Train $G_{t}$ for its final weights
$(\Theta_{t})$

8: Else:

9: For $e=1:\\#\;Training\;epoch$

10: $\Theta_{e}\rightarrow\Theta$ : Train $G_{t}$ for its final weights
$(\Theta_{t})$

11: If $t<T:\\#\ LTH\;pruning\;strategy$

12: Mask$(\Theta_{t})$ to get a pruned graph $G_{p}$ from $G_{t-1}$

13: Load weights $\Theta_{p}\in\Theta_{t}$ from $G_{p}$

14: Update target model $G_{p}(\Theta_{p})\rightarrow G_{t+1}$

15:A well-trained pruned model $G_{T}(\Theta_{T})$

###  2.2 Audio-Visual Model Pruning Using LTH-IF

The recent “Lottery Ticket Hypothesis” [26] showed an encouraging phenomenon
that some subnetworks (winning tickets) could be obtained by pruning the
original network through specific methods, and then they can be trained to
achieve the performance equal to or better than the untrimmed original model.
Although convolutional neural networks have shown to be effective to the
small-footprint WWS problem, they still need hundreds of thousands of
parameters to achieve good performance [27]. Although LTH-based low-complexity
neural models have proven competitive prediction performance on several image
classification tasks, machine translation [28, 29] and acoustic scene
classification [30], and recently have been supported with some theoretical
findings [31] related to overparameterization, the effect of LTH on our
multimodal task of audio-visual WWS is not unknown. In this study, with a high
demand of designing a compact audio-visual WWS model with low-latency for real
applications, we investigate on neural network pruning based on LTH with an
iterative fine-tuning strategy.

LTH-IF Algorithm Design: In Algorithm 1, we detail our approach: First, our
WWS model with its original neural architecture $G_{0}$ is initialized with
the weights parameters $(\Theta_{0})$. Different from [26], the complete
number of iterations ($E=5$) is selected to train the model only before
pruning (i.e. $t=1$). At the end of each training phase, a pruning iteration
is started if the current iteration $t$ is less than $T$. And the final
weights $\Theta_{T}$ are used for the new initialization to fine-tune the
model. The LTH pruning searches for a low-complexity model from steps (10) to
(13).

For the audio-only WWS model, we prune the whole network directly based on
LTH-IF. However for video-only WWS model, unlike audio-only WWS, it includes
an additional lip encoder, and we also prune it jointly with the back-end
module based on LTH-IF.

Interestingly, for the audio-visual WWS model, we found that separate pruning
for the lip encoder first will yield better performance than pruning the whole
model directly at the same degree of pruning. Therefore, for the audio-visual
WWS model, we first prune the lip encoder using LTH-IF. Then the lip encoder
is fixed and initialized using the weights and mask obtained above. Finally,
the model is trained for pruning back-end network of fusion stream based on
LTH-IF.

##  3 EXPERIMENTS

###  3.1 Databases and Implementation Details

We conduct experiments on an audio-visual dataset collected in smart home TV
scenes. There are 250 speakers in total, with a male-to-female ratio of 1:1.
The wake word is “Xiao T Xiao T”. The speakers in the training set,
development set and test set do not overlap, which are 210, 20 and 20
respectively. For the training set, in addition to the original positive and
negative samples, we also added several types of noises for data augmentation.
Our final training set includes 50 hours of positive samples and 50 hours of
negative samples, respectively. For the development/test set, in order to
facilitate comparison, we only add noise with three signal-to-noise ratios
(i.e. -5dB, 0dB, 5dB), including 2 hours of positive samples and 2 hours of
negative samples, respectively. Each positive sample contains only one wake
word. The duration of each sample is 1.3 seconds.

We use false reject rate (FRR) and false alarm rate (FAR) on test set as the
criterion of the WWS performance. Suppose the test set consists of
$N_{\text{wake}}$ examples with wake word and $N_{\text{non-wake}}$ examples
without wake word, FRR and FAR are defined as follows:

$\ \text{FRR}=\frac{N_{\text{FR}}}{N_{\text{wake}}}\ $ (6)

$\ \text{FAR}=\frac{N_{\text{FA}}}{N_{\text{non-wake}}}\ $ (7)

where $N_{\text{FR}}$ denotes the number of examples including wake-up word
but the WWS system gives a negative decision. $N_{\text{FA}}$ is the number of
examples without wake-up word but the WWS system gives a positive decision.

We employ pytorch to train all models and minimize the loss function using the
Adam optimization method. The batch size is 64 for audio-only WWS system and
16 for video-only and audio-visual systems. The learning rates are set to
0.0001, 0.0002, 0.0002 for audio-only, video-only and audio-visual systems
respectively.

###  3.2 Results on Audio-Visual Wake Word Spotting

First we evaluate the performance of the single-modal systems. We train the
audio-only and the video-only system respectively, corresponding to the audio
stream and video stream of the audio-visual model shown in Fig. 1. Our results
are presented in Table 1. We can observe that the better results were achieved
by audio-only system compared with video-only especially in less-noisy
environments. In low-SNR adverse acoustic environments, video-only system
achieves better FAR performance, which indicates potential of audio-visual
system that integrates the advantages of audio-only and video-only systems.
Moreover, a direct concatenation for audio-visual fusion at the encoder part
is further implemented, which yielded remarkable improvements compared with
the single-modal system under various signal-to-noise ratios. For example, the
performance gap for -5dB is 4.78% of FAR between audio-only model and audio-
visual model.

Table 1: Test set performance comparison of different systems. Modality | 1-FRR (%) | FAR (%)
---|---|---
-5dB | 0dB | 5dB
Audio | 98.78 | 8.03 | 2.95 | 1.60
Video | 98.78 | 6.92 | 6.92 | 6.92
Audio-visual | 98.78 | 3.25 | 1.06 | 0.56
Table 2: Parameter statistics of different systems. Network | Param. (M) | FLOPs (M)
---|---|---
Audio | 0.35 | 11.29
Lip Encoder | 0.39 | 642.56
Video | 1.19 | 651.68
Audio-Visual | 1.55 | 656.48

The statistics of the parameters and FLOPs of these three models are shown in
Table 2. We can observe that after adding video modality, the number of
parameters and FLOPs of the audio-visual WWS model is greatly increased. In
particular, the parameters and FLOPs of lip encoder exceed those of audio-only
network, which also promotes us to explore more effective pruning methods.

Table 3: Test set performance of single-modalities and multi-modality systems under different pruning degrees. [1-FRR : 98.78%] Modality | Method | FAR (%) | Pruned (%)
---|---|---|---
-5dB | 0dB | 5dB
Audio | No Pruning | 8.03 | 2.95 | 1.60 | 0.00
LTH [26] | 34.00 | 23.67 | 18.3 | 71.90
LTH-IF | 7.71 | 2.26 | 1.15 | 80.21
Video | No Pruning | 6.92 | 6.92 | 6.92 | 0.00
LTH [26] | 16.11 | 16.11 | 16.11 | 29.86
LTH-IF | 6.89 | 6.89 | 6.89 | 55.65
Audio-Visual | No Pruning | 3.25 | 1.06 | 0.56 | 0.00
LTH-IF | 2.29 | 0.84 | 0.53 | 42.52

###  3.3 Results on LTH-IF Pruning

We first implemented LTH-based pruning on single-modal systems described in
[26] with the results shown in Table 3. When LTH-based pruning in [26] is
firstly applied to the single-modal models, the performance degrades rapidly
in both audio and video modalities. Compared to original unpruned model, using
LTH with iterative fine-tuning (LTH-IF) achieves better performance especially
in the audio modality even though over 80% model parameters are pruned, which
demonstrates the effectiveness of the iterative fine-tuning strategy.

Fig. 2: Test set performance on audio-visual WWS model with the iterative
pruning during the training process.

Based on these positive results, we next apply LTH-IF pruning to the audio-
visual WWS model, and the results are shown in last row of Table 3. According
to Table 2, most of the parameters and FLOPs in audio-visual system come from
the lip encoder. Thus we design an experiment by only applying LTH-IF pruning
for the lip encoder. The pruned model achieves better performance than the
audio-visual unpruned model after pruning about 80% of the parameters in lip
encoder. Fig. 2 shows the results comparison of applying LTH-IF pruning (blue)
to the audio-visual WWS model and original unpruned model (red). According to
the experimental results above, the system performance gradually deteriorates
after about 80% of the model parameters are pruned. Finally, we initialize the
mask in LTH-IF according to the lip encoder result, and then apply LTH-IF to
the whole audio-visual model for pruning. According to Table 3, after pruning
42.52% of the parameters, we achieve better performance compared to the model
without pruning under all three signal-to-noise ratios.

We randomly select a specific iteration (e.g. $t=20$) and list the pruned
parameters of different layers of the model which is shown in Table 4. It can
be observed that for both single-modality and multi-modality systems, the
pruning proportions of different types of layers are similar.

However, during the pruning process, we observe that the performance of video-
only system is often unstable (degradation with potentially important nodes
pruned) and more sensitive to the pruning proportion setting compared with the
audio-visual system. So the audio-visual system seems more robust to pruning,
which might be explained by that the audio-visual fusion leads to the
selection of more suitable nodes without being pruned.

Table 4: The comparison of pruned parameters after the same iterations for single-modalities and multi-modality systems. [$t$=20] Modality | Conv layers(%) | LSTM(%) | FC layers(%)
---|---|---|---
Audio | 65.54 | 65.24 | 65.72
Video | 65.54 | 65.86 | 65.73
Audio-Visual | 65.53 | 65.89 | 65.70

##  4 CONCLUSION

In this paper, we investigate on designing a compact audio-visual WWS system
under noisy conditions by utilizing video information to alleviate the
performance degradation. Tested on our in-house corpus for audio-visual WWS in
a home TV scene, the proposed audio-visual system achieves significant
performance improvements over the single-modality system under different noisy
conditions. Furthermore, a neural network pruning strategy via LTH in an
iterative fine-tuning manner is introduced which can largely reduce the
network parameters and computations with no degradation of WWS performance.

##  5 Acknowledgement

This work was supported in part by the National Natural Science Foundation of
China under Grant No. 62171427, and Alibaba Group.

## References

  * [1] David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, and Joseph Dureau,  “Federated learning for keyword spotting,”  in ICASSP, 2019, pp. 6341–6345.
  * [2] Iván López-Espejo, Zheng-Hua Tan, and Jesper Jensen,  “Improved external speaker-robust keyword spotting for hearing assistive devices,”  IEEE/ACM Transactions on Audio, Speech and Language Processing, vol. 28, pp. 1233–1247, 2020.
  * [3] Yiteng Arden Huang, Turaj Z. Shabestary, and Alexander Gruenstein,  “Hotword cleaner: Dual-microphone adaptive noise cancellation with deferred filter coefficients for robust keyword spotting,”  in ICASSP, 2019, pp. 6346–6350.
  * [4] Christin Jose, Yuriy Mishchenko, Thibaud Sénéchal, Anish Shah, Alex Escott, and Shiv Naga Prasad Vitaladevuni,  “Accurate detection of wake word start and end using a CNN,”  in INTERSPEECH, 2020, pp. 3346–3350.
  * [5] Xiong Wang, Sining Sun, Changhao Shan, Jingyong Hou, Lei Xie, Shen Li, and Xin Lei,  “Adversarial examples for improving end-to-end attention-based small-footprint keyword spotting,”  in ICASSP, 2019, pp. 6366–6370.
  * [6] R.C. Rose and D.B. Paul,  “A hidden markov model based keyword recognition system,”  in ICASSP, 1990, pp. 129–132.
  * [7] Ashish Shrivastava, Arnav Kundu, Chandra Dhir, Devang Naik, and Oncel Tuzel,  “Optimize what matters: Training dnn-hmm keyword spotting model using end metric,”  in ICASSP, 2021, pp. 4000–4004.
  * [8] Guoguo Chen, Carolina Parada, and Georg Heigold,  “Small-footprint keyword spotting using deep neural networks,”  in ICASSP, 2014, pp. 4087–4091.
  * [9] Tara Sainath and Carolina Parada,  “Convolutional neural networks for small-footprint keyword spotting,”  in INTERSPEECH, 2015, p. 1478–1482.
  * [10] Raphael Tang and Jimmy Lin,  “Deep residual learning for small-footprint keyword spotting,”  in ICASSP, 2018, pp. 5484–5488.
  * [11] Sercan Ömer Arik, Markus Kliegl, Rewon Child, Joel Hestness, Andrew Gibiansky, Christopher Fougner, Ryan Prenger, and Adam Coates,  “Convolutional recurrent neural networks for small-footprint keyword spotting,”  in INTERSPEECH, 2017, pp. 1606–1610.
  * [12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in Advances in neural information processing systems, 2017, pp. 5998–6008.
  * [13] Yiming Wang, Hang Lv, Daniel Povey, Lei Xie, and Sanjeev Khudanpur,  “Wake word detection with streaming transformers,”  in ICASSP, 2021, pp. 5864–5868.
  * [14] Bo Zhang, Wenfeng Li, Qingyuan Li, Weiji Zhuang, Xiangxiang Chu, and Yujun Wang,  “Autokws: Keyword spotting with differentiable architecture search,”  in ICASSP, 2021, pp. 2830–2834.
  * [15] Xuan Ji, Meng Yu, Jie Chen, Jimeng Zheng, Dan Su, and Dong Yu,  “Integration of multi-look beamformers for multi-channel keyword spotting,”  in ICASSP, 2020, pp. 7464–7468.
  * [16] Yixin Gao, Yuriy Mishchenko, Anish Shah, Spyros Matsoukas, and Shiv Vitaladevuni,  “Towards data-efficient modeling for wake word spotting,”  in ICASSP, 2020, pp. 7479–7483.
  * [17] Myunghun Jung, Youngmoon Jung, Jahyun Goo, and Hoirin Kim,  “Multi-task network for noise-robust keyword spotting and speaker verification using ctc-based soft VAD and global query attention,”  in INTERSPEECH, 2020, pp. 931–935.
  * [18] Iván López-Espejo, Zheng-Hua Tan, and Jesper Jensen,  “A novel loss function and training strategy for noise-robust keyword spotting,”  IEEE/ACM Transactions on Audio, Speech and Language Processing, vol. 29, pp. 2254–2266, 2021.
  * [19] Ximin Li, Xiaodong Wei, and Xiaowei Qin,  “Small-footprint keyword spotting with multi-scale temporal convolution,”  in INTERSPEECH, 2020, pp. 1987–1991.
  * [20] Oleg Rybakov, Natasha Kononenko, Niranjan Subrahmanya, Mirkó Visontai, and Stella Laurenzo,  “Streaming keyword spotting on mobile devices,”  in INTERSPEECH, 2020, pp. 2277–2281.
  * [21] Théodore Bluche and Thibault Gisselbrecht,  “Predicting detection filters for small footprint open-vocabulary keyword spotting,”  in INTERSPEECH, 2020, pp. 2552–2556.
  * [22] Bin Liu, Shuai Nie, Yaping Zhang, Shan Liang, Zhanlei Yang, and Wenju Liu,  “Loss and double-edge-triggered detector for robust small-footprint keyword spotting,”  in ICASSP, 2019, pp. 6361–6365.
  * [23] Simon Mittermaier, Ludwig Kürzinger, Bernd Waschneck, and Gerhard Rigoll,  “Small-footprint keyword spotting on raw audio data with sinc-convolutions,”  in ICASSP, 2020, pp. 7454–7458.
  * [24] L Momeni, T Afouras, T Stafylakis, S Albanie, and A Zisserman,  “Seeing wake words: Audio-visual keyword spotting,”  in BMVA, 2020.
  * [25] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen,  “Mobilenetv2: Inverted residuals and linear bottlenecks,”  in CVPR, 2018, pp. 4510–4520.
  * [26] Jonathan Frankle and Michael Carbin,  “The lottery ticket hypothesis: Finding sparse, trainable neural networks,”  in ICLR, 2018.
  * [27] Menglong Xu and Xiao-Lei Zhang,  “Depthwise separable convolutional resnet with squeeze-and-excitation blocks for small-footprint keyword spotting,”  in INTERSPEECH, 2020, pp. 2547–2551.
  * [28] Alex Renda, Jonathan Frankle, and Michael Carbin,  “Comparing rewinding and fine-tuning in neural network pruning,”  in ICLR, 2019.
  * [29] Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, and Zhangyang Wang,  “The elastic lottery ticket hypothesis,”  arXiv preprint arXiv:2103.16547, 2021.
  * [30] Chao-Han Huck Yang, Hu Hu, Sabato Marco Siniscalchi, Qing Wang, Yuyang Wang, Xianjun Xia, Yuanjun Zhao, Yuzhong Wu, Yannan Wang, Jun Du, et al.,  “A lottery ticket hypothesis framework for low-complexity device-robust neural acoustic scene classification,”  arXiv preprint arXiv:2107.01461, 2021.
  * [31] Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir,  “Proving the lottery ticket hypothesis: Pruning is all you need,”  in ICML. PMLR, 2020, pp. 6682–6691.