11institutetext: Department of Computer Science, University of Waikato, New
Zealand 11email<EMAIL_ADDRESS>
# Predicting COVID-19 Patient Shielding: A Comprehensive Study

Vithya Yogarajan  0000-0002-6054-9543 Jacob Montiel  0000-0003-2245-0718 Tony
Smith  0000-0003-0403-7073 Bernhard Pfahringer  0000-0002-3732-5787

###### Abstract

There are many ways machine learning and big data analytics are used in the
fight against the COVID-19 pandemic, including predictions, risk management,
diagnostics, and prevention. This study focuses on predicting COVID-19 patient
shielding — identifying and protecting patients who are clinically extremely
vulnerable from coronavirus. This study focuses on techniques used for the
multi-label classification of medical text. Using the information published by
the United Kingdom NHS and the World Health Organisation, we present a novel
approach to predicting COVID-19 patient shielding as a multi-label
classification problem. We use publicly available, de-identified ICU medical
text data for our experiments. The labels are derived from the published
COVID-19 patient shielding data. We present an extensive comparison across 12
multi-label classifiers from the simple binary relevance to neural networks
and the most recent transformers. To the best of our knowledge this is the
first comprehensive study, where such a range of multi-label classifiers for
medical text are considered. We highlight the benefits of various approaches,
and argue that, for the task at hand, both predictive accuracy and processing
time are essential.

###### Keywords:

COVID-19 Multi-label Neural Networks Transformers Medical Text

##  1 Introduction

The Coronavirus disease 2019 (COVID-19) pandemic has presented a considerable
challenge to the world health care system, and the management of COVID-19 is
an ongoing struggle. The ability to identify and protect high-risk groups is
debated by the scientific community [14]. COVID-19 patient shielding refers to
identifying and protecting patients who are clinically extremely vulnerable
from coronavirus. Patients in these categories include those people who have
been identified by health professionals before the pandemic as being
clinically extremely vulnerable, and those identified through the COVID-19
Population Risk Assessment model [5]. Such patients present with co-morbidity
[5] and hence the need to use multi-label classification techniques. The
clinical risk prediction model (QCOVID) [5] was developed by the United
Kingdom NHS based on a cohort study of the population using data from over 6
million adults. It provides guidelines for patients who are considered high
risk (clinically extremely vulnerable); for example, patients with a certain
type of cancer, patients who have had organ transplants, and patients having a
serious heart condition while also being pregnant.

QCOVID provides information on medical codes which can be used in hospital
databases to identify patients who are at high risk and fall under the
criteria of COVID-19 shielding. Due to privacy and legal issues, obtaining
current patient records from hospitals is not possible [6]. However,
electronic health records (EHRs) from publicly available data such as Medical
Information Mart for Intensive Care (MIMIC-III) [15, 11] and electronic
Intensive Care Unit (eICU) [11, 25] present a realistic set of EHRs obtained
and de-identified from hospitals in the United States. This research presents
a novel approach of considering COVID-19 patient shielding as a multi-label
problem (see Figure 1 for details). Multi-label classification assigns a set
of labels to an instance. From a collection of labels, each record will be
assigned relevant medical codes/labels. Using the medical codes presented in
QCOVID and the code conversions presented by the WHO, we identify and predict
patients with one or more high-risk medical codes associated with EHRs in
MIMIC-III and eICU.

Figure 1: Flow chat forming labels and instances for multi-label
classification of predicting COVID-19 patient shielding.

This research presents an extensive comparative study across 12 multi-label
classifiers starting from the simplest binary relevance with logistic
regression to neural networks and several transformers. We highlight the
advantages and disadvantages of these classifiers when applied to predicting
COVID-19 patient shielding. We present overall predictive accuracy as well as
label F1-scores where each label is a high-risk COVID-19 medical code. We show
that, in addition to the overall predictive accuracy, processing time plays a
big role in decision making.111The code to recreate the experiments and
evaluations described in this paper is accessible at:
https://github.com/vithyayogarajan/COVID-19-Patient-Shielding

The contributions of this work are:

  1. 1.

we present a novel approach of considering predicting COVID-19 patient
shielding as a multi-label problem;

  2. 2.

we analyse the effectiveness of 12 different multi-label classifiers including
several neural networks and transformers for predicting medical codes
associated with COVID-19 patient shielding from medical text;

  3. 3.

we show both overall prediction accuracy and processing time plays a role in
selecting a multi-label approach;

##  2 Related Work

The advancement of machine learning-based predictions motivates researchers to
consider ways to help in the fight against COVID-19, and there are numerous
ways in which this can be achieved [1]. The year 2020 has seen many examples
of machine learning approaches applied to various aspects of the COVID-19
pandemic. Some examples of machine learning approaches include:

  * •

a hybrid machine learning technique for pandemic predictions based on data
from Hungary [24]

  * •

a deep convolutional neural network model named CoroNet for predicting
COVID-19 from chest X-ray images[16]

  * •

support vector machines for predicting radiological findings consistent with
COVID-19 from radiology text reports [21]

  * •

classifying clinical reports using logistic regression and multinomial Naive
Bayes [17]

  * •

spectrometric data analysis for a diagnosis based study [8]

These examples are only the beginning of the endless possibilities of machine
learning approaches for the COVID-19 pandemic. Several machine learning
methods are used for outbreak predictive studies worldwide to make decisions
and enforce control measures. The systems mentioned above are only a small
subset of examples of such studies. This reflects a need for extensive
comparison among machine learning models, especially for predictions from
medical text.

Yogarajan et al. (2020) [30] show domain-specific skip-gram pre-trained
fastText embeddings perform better that general text pre-trained embeddings.
Hence, 100 and 300-dimensional domain-specific fastText pre-trained embeddings
[29, 30] are used for this research. A collection of neural networks and
transformer models are chosen for the predictive study in this research, based
on literature where such models have been used for multi-label medical code
prediction and have achieved state-of-the-art results [31, 23, 22].

##  3 Data and Labels

Table 1: Summary of Data for COVID-19 patient shielding, for EHRs with
available relevant medical codes is included. Percentage frequency of
occurrences of labels for MIMIC-III and eICU is also presented.

MIMIC-III | eICU
---|---
Critical care units at the | The Philips eICU program.
Beth Israel Deaconess Medical | Admitted to ICUs in 2014 and 2015
Center between 2001 - 2012. | across the United States.
Free-form medical text. Eg: | Semi-structured medical text. Eg:
“59M w HepC cirrhosis c/b grade | “infectious diseases $|$ medications $|$
I/II esophageal varices and portal | therapeutic antibacterials $|$ cardio $|$
gastropathy, p/w coffee-ground” | inotropic agent $|$ norepinephrine”
35,458 hospital admissions with | 34,387 patients with EHRs.
discharge summaries. |
Text length: 60 - 9500 tokens. | Text length: 50 - 1400 tokens.
42 medical codes (labels) | 25 medical codes (labels)
|
ICD-9 code examples: | ICD-9 code examples:
Code 285 (35%) Other and unspecified anemias | Code 491 (40.3%) Chronic bronchitis
Code 996 (16%) Complications peculiar to | Code 288 (17.5%) Diseases of white blood cells
certain specified procedures |

ICD (standards for International statistical Classification of Diseases and
related health problems) codes are used to classify diseases, symptoms, signs,
and causes of diseases. For this research, we use ICD-9 because the available
data pnly includes labels for ICD-9 codes. A summary of MIMIC-III and eICU,
with details of number of instances and labels, label frequencies and examples
of EHRs and ICD-9 codes are presented in Table 1.

##  4 Multi-Label Classifiers

(a)

(b)

(c)

Figure 2: (2(a)) Flowchart for traditional multi-label classifiers; (2(b))
CNNText [18] architecture; and (2(c)) Example of BERT-base model for multi-
label classification.  Table 2: Summary of Transformer Models. Except for
general text pre-trained models Longformer and TransformerXL, all other models
included in this research can only handle a sequence length of at most 512
tokens. Continuous training approach would initialize with the standard BERT
model, pre-trained using Wikipedia and BookCorpus. It then continues the pre-
training process with masked language modelling and next sentence prediction
using domain-specific data. Similarly for BioMed-RoBERTa.

Transformers | Training Data | Model Details
---|---|---
BERT [9] | Books + Wiki | 12-layers with a hidden size of 768, 12 self-attention
|  | heads, 110M parameter neural network architecture.
ClinicalBERT [2] | MIMIC-III | continuous training of BERT-base
PubMedBERT [12] | PubMed | BERT-base model architecture
|  | Trained from scratch
RoBERTa-base [20] | Web crawl | robustly optimised BERT approach
|  | improved training methodology
BioMed-RoBERTa [13] | Scientific papers | continuous training of RoBERTa-base
TransformerXL [7] | Can handle documents with $>512$ tokens. Models longer-term dependency
| by combining recurrence and relative positional encoding.
| Training Data: general text including Wiki
Longformer [3] | Can handle documents with $>512$ tokens.
| Provides computational and memory efficiency.
| Training Data: Books + Wiki + Realnews + Stories

The first and simplest multi-label classification algorithm is binary
relevance (BR) [10]. A separate binary classification model is created for
each label, such that any text with that label is a positive instance, and all
other records form the negative instances. BR models make their predictions
independently. However, for multi-label problems where there is a strong
correlation between labels, a model could benefit from the result of another
label when making its predictions. BR models can be ‘chained’ together into a
sequence such that the predictions made by earlier classifiers are made
available as additional features for the later classifiers. Such a
configuration is called a classifier chain (CC) [26]. Ensembles of classifier
chains (ECC) built with diverse random chaining orders can help mitigate the
effect of chaining order. In this research, logistic regression (LR) is used
as the base classifier for both BR and ECC. Figure 2(a) provides a flow chart
for BR and ECC, where document embeddings for each clinical document are used
as features. Document level embeddings are computed by obtaining the vector
sum of the embeddings for each word in the document and then normalising the
sum to have length one, to ensure that documents of different lengths have
representations of similar magnitudes.

CNNText [18] combines one-dimensional convolutions with a max-over-time
pooling layer and a fully connected layer. Each word or token in the clinical
document is mapped to its $k$-dimensional embeddings using the lookup table
for a given pre-trained embeddings model. A new feature is produced using the
convolution operation, and max-over-time pooling is applied over the feature
map to capture the most important feature value. Multiple filters are used
with varying window sizes. The final prediction is made by computing a
weighted combination of the pooled values and applying a sigmoid function. A
simple architecture of CNNText is presented in Figure 2(b).

Gated Recurrent Units (GRU) [4] are a type of recurrent neural networks which
are designed to better handle vanishing gradients. Bidirectional GRU (Bi-GRU)
considers both sequences from left to right and the reverse order. As with
CNNText, clinical documents are mapped to their embeddings, and a sigmoid
activation function is used.

Mullenbach et al. (2018) [23] present Convolutional Attention for Multi-Label
classification (CAML), which achieves SOTA results for predicting ICD-9 codes
from MIMIC III data [22]. CAML combines convolution networks with an attention
mechanism. Simultaneously, a second module is used to learn embeddings of the
descriptions of ICD-9 codes to improve the predictions of less frequent
labels. A regularising objective with a trade-off hyperparameter was added to
the loss function of CAML. This variant is called Description Regularized-CAML
(DR-CAML) [23].

This research includes several variations of the most recent developments of
NLP transformers [28]. In this paper, we perform transfer learning of pre-
trained transformer models by fine tuning for predicting COVID-19 patient
shielding. All parameters are fine-tuned end-to-end. A summary of Transformer
models used in this research is presented in Table 2. BERT-base variations and
RoBERTa-base variations can only handle a maximum sequence length of 512,
whereas Longformer and TransformerXL can handle longer sequences. Continuous
training of transformers refers to existing general text pre-trained models
that are further trained with a masked language model and next sentence
prediction using domain-specific data. The vocabulary is the same as the
original BERT model, which is considered a disadvantage for domain-specific
tasks [12]. Alternatively, training from scratch indicates that the models are
pre-trained with domain-specific data only and the vocabulary of these models
is domain-specific. Figure 2(c) provides an example of a BERT-base model.

##  5 Experimental Setup

Neural network models presented in this research are implemented using
PyTorch. All evaluations were done using sklearn metrics and the transformer
implementations are based on the open-source PyTorch-transformer repository.
For multi-label classification MEKA [27], an open-source Java system
specifically designed to support multi-label classification experiments, is
used for BR and ECC. Domain-specific fastText embeddings [29, 30] of
100-dimensions are used for neural networks and 300-dimensions for BR and ECC.

Transformer models were fine-tuned on all layers without freezing. Adam [19],
an adaptive learning rate optimisation algorithm, is used as the optimiser for
all neural networks. A non-linear sigmoid function $f(z)=\frac{1}{1+e^{-z}}$,
is used as the activation function. All experiments using ECC and BR are
validated through 10-fold cross-validation. Due to resource restrictions, all
neural network results, including transformers, use random seed train-test
hold-out set validation where the results are averaged over three runs.
Experiments for neural networks and transformers were run on a 12 core
Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz, and a GPU device GV100GL [Quadro
GV100]. Experiments for BR and ECC were run on a 4 core Intel i7-6700K CPU @
4.00GHz with 64GB of RAM.

For MIMIC-III data, discharge summaries are used as the free-form medical text
and each hospital admission is treated as an instance. For ECC and BR, all
available discharge summaries for each hospital admission are treated as one
document, and 300 dimensional features are obtained from fastText pre-trained
domain-specific embeddings. For embedding-based neural networks CNNtext,
BiGRU, CAML and DRCAML, the pre-processed tokens from discharge summaries are
truncated to 3,000 words. For BERT and RoBERTa based transformers, the input
sequence is truncated to 512 tokens, and for Longformer and TransformerXL the
input sequence is truncated to 3,000 and 3,072 tokens respectively. Discharge
summaries were pre-processed by removing tokens that contain non alphabetic
characters and special characters, and tokens that appear in fewer than three
training documents. The extensively pre-processed eICU data was used ‘as is’
for all experiments.

##  6 Results

Table 3: Comparison of micro-F1 and macro-F1 of ICD-9 codes for various multi-
label classifiers for both MIMIC-III and eICU data. Time required per run is
also presented. Bold is used to indicate the best results among the groups,
and underline is used for overall best results. Results are averaged over
three runs for neural networks and transformers. 10-fold cross validation is
used for BR-LR and ECC-LR.

| MIMIC-III |  | eICU
---|---|---|---
Classifiers | Micro -F1 | Macro-F1 | Total Time |  | Micro -F1 | Macro-F1 | Total Time
|  |  | (per run) |  |  |  | (per run)
BR-LR | 0.39 | 0.26 | 12 min |  | 0.54 | 0.28 | 7 min
ECC-LR | 0.45 | 0.27 | 38 min |  | 0.51 | 0.28 | 34 min
CNNText | 0.58 | 0.42 | 46 min |  | 0.59 | 0.36 | 45 min
BiGRU | 0.59 | 0.31 | 216 min |  | 0.59 | 0.35 | 210 min
CAML | 0.61 | 0.40 | 49 min |  | 0.60 | 0.32 | 48 min
DRCAML | 0.60 | 0.39 | 64 min |  | 0.61 | 0.32 | 60 min
BERT-base | 0.50 | 0.44 | 10 hr |  | 0.60 | 0.36 | 11 hr
ClinicalBERT | 0.51 | 0.45 | 16 hr |  | 0.60 | 0.36 | 11 hr
BioMed-RoBERTa | 0.53 | 0.45 | 12 hr |  | 0.61 | 0.37 | 11 hr
PubMedBERT | 0.54 | 0.48 | 16 hr |  | 0.64 | 0.39 | 14 hr
Longformer | 0.58 | 0.50 | 82 hr |  | 0.61 | 0.40 | 49 hr
TransformerXL | 0.65 | 0.51 | 206 hr |  | 0.63 | 0.40 | 53 hr

(a) Data: MIMIC-III

(b) Data: eICU

Figure 3: Critical difference plots of label F1 scores for COVID-19 patient
shielding with MIMIC-III (top) and eICU (bottom). Nemenyi post-hoc test (95%
confidence level), identifying statistical differences between multi-label
classifiers presented in Table 3. Critical difference is calculated for
individual label F1 scores.

Table 3 presents a summary of overall micro and macro F1 scores for 12 multi-
label classifiers and the total time per run of each experiment for both
MIMIC-III and eICU data. Figure 3 presents critical difference plots based on
the label F1 scores for MIMIC-III and eICU to help identify the statistical
difference between the multi-label classifiers. Compared to the basic BR-LR,
which requires the least time for experiments, there is a significant
improvement in overall micro and macro F1 measures for MIMIC-III data. This
observation is also reflected on the critical difference plots where BR-LR is
the classifier with the worst ranking. Among transformer models, the micro and
macro F1 scores of TransformerXL and Longformer are better than the scores of
the BERT-base and RoBERTa-base models. One of the main reasons for this is
that BERT-base and RoBERTa-base models are restricted by the maximum sequence
length of $512$ tokens, while TransformerXL and Longformer can handle longer
sequences. Since MIMIC-III data are long with an average of $1,500$ tokens,
the ability to encode more extended input data improves accuracy. However, the
total time required for the experiments using TransformerXL and Longformer is
exceptionally high. On the CD plot, a bold line connects PubMedBERT to the
above two transformers indicating no statistically significant difference in
average ranking. Compared to PubMedBERT, Longformer requires 5 times more
processing time and TransformerXL almost 13 times more processing time. Among
the embedding-based neural networks, CNNText and CAML are the best performing
models, and they require less than an hour in total for an experiment.
Micro-F1 scores of embeddings-based neural networks are better than those of
most transformer models, except for TransformerXL.

Figure 4: Macro F1 scores of transformer models for COVID-19 patient shielding
over number of epochs for MIMIC-III data. Due to resource restrictions,
TransformerXL was only run for 15 epochs.

Figure 5: Best F1 scores and corresponding multi-label classifiers for
individual labels with MIMIC-III (top) and eICU (bottom). Codes 235 and 142
for MIMIC-III data and code 501 for eICU are omitted, as all models fail to
predict them.

For eICU data, as observed in MIMIC-III, BR-LR and ECC-LR are the two worst-
ranking classifiers in the CD-plot, with other classifiers performing better
than ECC and BR overall. The micro-F1 score of PubMedBERT is the best among
the multi-label classifiers, with a minimal difference in macro-F1 compared to
the macro-F1 score of Longformer. TransformerXL requires 3.8 times more
processing time than PubMedBERT and 71.6 times more processing time than
CNNText. The Critical difference plot indicates no statistically significant
difference between most classifiers, except BR-LR and ECC-LR (worst
performers).

Figure 4 presents a comparison of macro F1 scores after each epoch for a range
of 1 to 25 for transformer models presented in Table 3 for COVID-19 patient
shielding using MIMIC-III data. For TransformerXL, the experiments were only
performed for 15 epochs as the required computational resources for these
experiments are very high. For macro-F1 scores, there is a steady increase in
the number of epochs, where more are required before measures stabilise. For
this research, macro-F1 scores were prioritised, as macro-F1 is an average
over F1 scores of each label. In the case of predicting COVID-19 patient
shielding, an increase in macro-F1 will indicate an increase in individual F1
score labels, where each label refers to a medical code associated with
patient at high risk of COVID-19. The total training times presented in Table
3 are calculated for 20 epochs for all transformer models, except for
TransformerXL.

Figure 5 presents the best F1-scores for each label, and different colours are
used to differentiate the corresponding classifiers. For MIMIC-III data,
TransformerXL is the clear winner with the highest number of best F1 scores.
For eICU data, PubMedBERT is marginally better than TransformerXL and CNNText.

##  7 Discussion

There are many examples of the use of machine learning approaches in the fight
against COVID-19. We propose a novel multi-label approach to predicting
COVID-19 patient shielding from EHRs using publicly available information on
COVID-19 and publicly available data. We present an extensive study on 12
different multi-label approaches to predicting medical codes where we include
simple methods such as BR and ECC, examples of traditional word embeddings
based neural networks, and the most recent transformer models. If overall
predictive accuracy is the only deciding factor TransformerXL is the best
option. However, given pandemic situations where time is a significant factor,
the predictive accuracy of models is not the only factor to consider in
selecting an approach. To the best of our knowledge, this is the first study
to consider such a range of multi-label text classification models to enable a
better understanding of the potential to help fight against COVID-19.

## References

  * [1] Alimadadi, A., Aryal, S., Manandhar, I., Munroe, P.B., Joe, B., Cheng, X.: Artificial intelligence and machine learning to fight covid-19 (2020)
  * [2] Alsentzer, E., Murphy, J., Boag, W., Weng, W.H., Jindi, D., Naumann, T., McDermott, M.: Publicly available clinical BERT embeddings. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop. pp. 72–78 (2019)
  * [3] Beltagy, I., Peters, M., Cohan, A.: Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020)
  * [4] Cho, K., van Merrienboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: Encoder-decoder approaches. In: Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8), 2014 (2014)
  * [5] Clift, A.K., Coupland, C.A.C., Keogh, R.H., Diaz-Ordaz, K., Williamson, E., Harrison, E.M., Hayward, A., Hemingway, H., Horby, P., Mehta, N., Benger, J., Khunti, K., Spiegelhalter, D., Sheikh, A., Valabhji, J., Lyons, R.A., Robson, J., Semple, M.G., Kee, F., Johnson, P., Jebb, S., Williams, T., Hippisley-Cox, J.: Living risk prediction algorithm (qcovid) for risk of hospital admission and mortality from coronavirus 19 in adults: national derivation and validation cohort study. BMJ 371 (2020). https://doi.org/10.1136/bmj.m3731, https://www.bmj.com/content/371/bmj.m3731
  * [6] Cosgriff, C. V.and Ebner, D.K., Celi, L.A.: Data sharing in the era of COVID-19. (2020)
  * [7] Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., Salakhutdinov, R.: Transformer-XL: Attentive language models beyond a fixed-length context. In: ACL (2019)
  * [8] Delafiori, J., Navarro, L.C., Siciliano, R.F., De Melo, G.C., Busanello, E.N.B., Nicolau, J.C., Sales, G.M., De Oliveira, A.N., Val, F.F.A., De Oliveira, D.N., et al.: Covid-19 automated diagnosis and risk assessment through metabolomics and machine learning. Analytical Chemistry 93(4), 2471–2479 (2021)
  * [9] Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: NAACL-HLT (2019)
  * [10] Godbole, S., Sarawagi, S.: Discriminative methods for multi-labeled classification. In: Pacific-Asia conference on knowledge discovery and data mining. pp. 22–30. Springer (2004)
  * [11] Goldberger, A.L., Amaral, L.A., Glass, L., Hausdorff, J.M., Ivanov, P.C., Mark, R.G., Mietus, J.E., Moody, G.B., Peng, C.K., Stanley, H.E.: PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 101(23), e215–e220 (2000)
  * [12] Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H.: Domain-specific language model pretraining for biomedical natural language processing. arXiv preprint arXiv:2007.15779 (2020)
  * [13] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., Smith, N.A.: Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In: Proceedings of ACL (2020)
  * [14] Ioannidis, J.P.: Precision shielding for covid-19: metrics of assessment and feasibility of deployment. BMJ Global Health 6(1), e004614 (2021)
  * [15] Johnson, A.E., Pollard, T.J., Shen, L., Li-wei, H.L., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L.A., Mark, R.G.: MIMIC-III, a freely accessible critical care database. Scientific data 3, 160035 (2016)
  * [16] Khan, A.I., Shah, J.L., Bhat, M.M.: Coronet: A deep neural network for detection and diagnosis of covid-19 from chest x-ray images. Computer Methods and Programs in Biomedicine 196, 105581 (2020)
  * [17] Khanday, A.M.U.D., Rabani, S.T., Khan, Q.R., Rouf, N., Din, M.M.U.: Machine learning based approaches for detecting covid-19 using clinical text data. International Journal of Information Technology 12(3), 731–739 (2020)
  * [18] Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1746–1751. Association for Computational Linguistics (2014)
  * [19] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)
  * [20] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * [21] López-Úbeda, P., Díaz-Galiano, M.C., Martín-Noguerol, T., Luna, A., Ureña-López, L.A., Martín-Valdivia, M.T.: Covid-19 detection in radiological text reports integrating entity recognition. Computers in Biology and Medicine 127, 104066 (2020)
  * [22] Moons, E., Khanna, A., Akkasi, A., Moens, M.F.: A comparison of deep learning methods for icd coding of clinical records. Applied Sciences 10(15), 5262 (2020)
  * [23] Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., Eisenstein, J.: Explainable prediction of medical codes from clinical text. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1. ACL: New Orleans, LA, USA (2018)
  * [24] Pinter, G., Felde, I., Mosavi, A., Ghamisi, P., Gloaguen, R.: Covid-19 pandemic prediction for hungary; a hybrid machine learning approach. Mathematics 8(6), 890 (2020)
  * [25] Pollard, T.J., Johnson, A.E.W., Raffa, J.D., Celi, L.A., Mark, R.G., Badawi, O.: The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Scientific Data 5, 180178 (2018)
  * [26] Read, J., Pfahringer, B., Holmes, G., Frank, E.: Classifier chains for multi-label classification. Machine learning 85(3), 333 (2011)
  * [27] Read, J., Reutemann, P., Pfahringer, B., Holmes, G.: MEKA: A Multi-label/Multi-target Extension to WEKA. Journal of Machine Learning Research 17(21), 1–5 (2016), http://jmlr.org/papers/v17/12-164.html
  * [28] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30, 5998–6008 (2017)
  * [29] Yogarajan, V., Gouk, H., Smith, T., Mayo, M., Pfahringer, B.: Comparing High Dimensional Word Embeddings Trained on Medical Text to Bag-of-Words For Predicting Medical Codes. In: Asian Conference on Intelligent Information and Database Systems. pp. 97–108. Springer (2020)
  * [30] Yogarajan, V., Montiel, J., Smith, T., Pfahringer, B.: Seeing the whole patient: Using multi-label medical text classification techniques to enhance predictions of medical codes. arXiv preprint arXiv:2004.00430 (2020)
  * [31] Yogarajan, V., Montiel, J., Smith, T., Pfahringer, B.: Transformers for multi-label classification of medical text: An empirical comparison. In: Tucker, A., Henriques Abreu, P., Cardoso, J., Pereira Rodrigues, P., Riaño, D. (eds.) Artificial Intelligence in Medicine. pp. 114–123. Springer International Publishing, Cham (2021)


#  ProActive: Self-Attentive Temporal Point Process Flows
for Activity Sequences

Vinayak Gupta  IIT DelhiNew DelhiIndia<EMAIL_ADDRESS>and
Srikanta Bedathur  IIT DelhiNew DelhiIndia<EMAIL_ADDRESS>
(2022)

###### Abstract.

Any human activity can be represented as a temporal sequence of actions
performed to achieve a certain goal. Unlike machine-made time series, these
action sequences are highly disparate as the time taken to finish a similar
action might vary between different persons. Therefore, understanding the
dynamics of these sequences is essential for many downstream tasks such as
activity length prediction, goal prediction, _etc._ Existing neural approaches
that model an activity sequence are either limited to visual data or are task-
specific, _i.e._ , limited to next action or goal prediction. In this paper,
we present ProActive, a neural marked temporal point process (MTPP) framework
for modeling the continuous-time distribution of actions in an activity
sequence while simultaneously addressing three high-impact problems – next
action prediction, sequence-goal prediction, and end-to-end sequence
generation. Specifically, we utilize a self-attention module with temporal
normalizing flows to model the influence and the inter-arrival times between
actions in a sequence. Moreover, for time-sensitive prediction, we perform an
early detection of sequence goal via a constrained margin-based optimization
procedure. This in-turn allows ProActive to predict the sequence goal using a
limited number of actions. Extensive experiments on sequences derived from
three activity recognition datasets show the significant accuracy boost of
ProActive over the state-of-the-art in terms of action and goal prediction,
and the first-ever application of end-to-end action sequence generation.

marked temporal point process; continuous time sequences; activity modeling;
goal prediction; sequence generation

††journalyear: 2022††copyright: acmlicensed††conference: Proceedings of the
28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
14–18, 2022; Washington, DC, USA††booktitle: Proceedings of the 28th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22), August
14–18, 2022, Washington, DC, USA††price: 15.00††doi:
10.1145/3534678.3539477††isbn: 978-1-4503-9385-0/22/08††ccs: Information
systems Data mining

##  1\. Introduction

A majority of data generated via human activities, _e.g._ , running, playing
basketball, cooking, _etc._ , can be represented as a sequence of actions over
a continuous-time. These actions denote a step taken by a user towards
achieving a certain goal and vary in their start and completion times,
depending on the user and the surrounding environment(Mehrasa et al., 2017;
Kuehne et al., 2014; Mehrasa et al., 2019b). Therefore, unlike synthetic time
series, these continuous-time action sequences (CTAS) can vary significantly
even if they consist of the same set of actions. For _e.g._ , one person
making omelets may take a longer time to cook eggs while another may prefer to
cook for a short time111https://bit.ly/3F5aEwX (Accessed January 2022); or
Xavi may make a quicker pass than Pirlo in a football game – although in both
cases the goals are same, and they all have to perform the same sequence of
actions. In addition, modeling the dynamics of these CTAS is increasingly
challenging due to the limited ability of modern recurrent and self-attention-
based approaches in capturing continuous action times (Vaswani et al., 2017;
Kang and McAuley, 2018). This situation is further exacerbated due to the
asynchronous occurrence of actions and the large variance in action-times and
types. Therefore, the problem of modeling a CTAS has been overlooked by the
past literature.

In recent years, neural marked temporal point processes (MTPP) have shown a
significant promise in modeling a variety of continuous-time sequences in
healthcare (Rizoiu et al., 2018, 2017), finance (Zhang et al., 2020; Bacry et
al., 2015), education (Yao et al., 2021), and social networks (Zhao et al.,
2015; Mei and Eisner, 2017; Zuo et al., 2020). However, standard MTPP have a
limited modeling ability for CTAS as: (i) they assume a homogeneity among
sequences, _i.e._ , they cannot distinguish between two sequences of similar
actions but with different time duration; (ii) in a CTAS, an action may finish
before the start of the next action and thus, to model this empty time
interval an MTPP must introduce a new action type, _i.e._ , NULL or end-action
which may lead to an unwarranted increase in the types of actions to be
modeled; and (iii) they cannot encapsulate the additional features associated
with an action, for _e.g._ , minimum time for completion, necessary previous
actions, or can be extended to sequence generation.

###  1.1. Our Contribution

In this work, we present ProActive (Point Process flows for Activity
Seqeunces), a normalizing flow-based neural MTPP framework, designed
specifically to model the dynamics of a CTAS. Specifically, ProActive
addresses three challenging problems – (i) action prediction; (ii) goal
detection; and (iii) the first-of-its-kind task of end-to-end sequence
generation. We learn the distribution of actions in a CTAS using temporal
normalizing flows (NF) (Rezende and Mohamed, 2015; Mehrasa et al., 2019a)
conditioned on the dynamics of the sequence as well as the action-features
(_e.g._ , minimum completion time, _etc._). Such a flow-based formulation
provides ProActive flexibility over other similar models (Sharma et al., 2021;
Shchur et al., 2020) to better model the inter-action dynamics within and
across sequences. Moreover, our model is designed for early detection of the
sequence goal, _i.e._ , identifying the result of a CTAS without traversing
the complete sequence. We achieve this via a time-bounded optimization, _i.e._
, by incrementally increasing the probability of identifying the true goal
using a margin-based and a weighted factor-based learning (Ma et al., 2016;
Hoai and la Torre, 2012). Such an optimization procedure allows ProActive to
model the goal-action hierarchy, _i.e._ , the necessary set of actions in a
CTAS towards achieving a particular goal, and simultaneously, the order of
actions in CTAS with similar goals.

To the best of our knowledge, in this paper we present the first-ever
application of MTPP via end-to-end action sequence generation. Specifcally,
given the resultant goal, ProActive can generate a CTAS with the necessary set
of actions and their occurrence times. Such a novel ability for MTPP models
can reinforce their usage in applications related to bio-signals (Haradal et
al., 2018), sensor-data (Alzantot et al., 2017), _etc._ , and overcome the
modeling challenge due to scarcity of activity data (Donahue et al., 2020; Luo
et al., 2020; Gupta and Bedathur, 2022). Buoyed by the success of attention
models in sequential applications (Vaswani et al., 2017), we use a self-
attention architecture in ProActive to model the inter-action influences in a
CTAS. In summary, the key contributions we make in this paper via ProActive
are:

  1. (1)

We propose ProActive, a novel temporal flow-based MTPP framework designed
specifically for modeling human activities with a time-bounded optimization
framework for early detection of CTAS goal.

  2. (2)

Our normalizing flow-based modeling framework incorporates the sequence and
individual action dynamics along with the action-goal hierarchy. Thus,
ProActive introduces the first-of-its-kind MTPP application of end-to-end
action CTAS generation with just the sequence-goal as input.

  3. (3)

Finally, we empirically show that ProActive outperforms the state-of-the-art
models for all three tasks – action prediction, goal detection, and sequence
generation.

###  1.2. Organization

We present a problem formulation and a background on necessary techniques in
Section 2. Section 3 gives an overview followed by an detailed development of
all components in ProActive. Section 4 contains in-depth experimental analysis
and qualitative studies over all datasets. Lastly, Section 5 reviews a few
relevant works before concluding in Section 6.

##  2\. Preliminaries

In this section, we present a background of MTPP and normalizing flows, and
then present the problems addressed in this paper.

###  2.1. Background

Marked Temporal Point Processes. MTPP(Hawkes, 1971) are probabilistic
generative models for continuous-time event sequences. An MTPP can be
represented as a probability distribution over sequences of variable length
belonging to a time interval $[0,T]$. Equivalently, they can be described
using a counting process, say $N(t)$, and are characterized by the underlying
conditional intensity function, $\lambda^{*}(t)$ which specifies the
likelihood of the next event, conditioned on the history of events. The
intensity function $\lambda^{*}(t)$ computes the infinitesimal probability
that an event will happen in the time window $(t,t+dt]$ conditioned on the
history as:

(1) $\mathbb{P}\big{(}dN(t)=N(t+dt)-N(t)=1\big{)}=\lambda^{*}(t),$

Here, $*$ denotes a dependence on history. Given the conditional intensity
function, we obtain the probability density function as:

(2)
$p^{*}(\Delta_{t,i})=\lambda^{*}(t_{i-1}+\Delta_{t,i})\exp\bigg{(}-\int_{0}^{\Delta_{t,i}}\lambda^{*}(t_{i-1}+r)dr\bigg{)},$

where, $\Delta_{t,i}$ denotes the inter-event time interval, _i.e._ ,
$t_{i}-t_{i-1}$. In contrast to other neural MTPP models that rely on the
intensity function (Du et al., 2016; Mei and Eisner, 2017; Zhang et al., 2020;
Zuo et al., 2020) we replace the intensity function with a _log-normal_ flow.
Such a formulation facilitates closed-form and faster sampling as well as more
accurate prediction than the intensity-based models (Shchur et al., 2020;
Mehrasa et al., 2019a).

Normalizing Flows. Normalizing flows(Rezende and Mohamed, 2015; Shchur et al.,
2020; Mehrasa et al., 2019a) are generative models that are used for density
estimation as well as event sampling. They work by mapping simple
distributions to complex ones using multiple bijective functions. In detail,
if our goal is to estimate the density function $p_{\boldsymbol{X}}$ of a
random vector $\boldsymbol{X}\in\mathbb{R}^{D}$, then the normalizing flows
assign a new distribution $\boldsymbol{X}=g_{\phi}(\boldsymbol{Z})$, with
$g_{\phi}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{D}$ is a bijective function,
and $\boldsymbol{Z}\in\mathbb{R}^{D}$ is a random vector samped from a simple
density function $p_{\boldsymbol{Z}}$. Using NFs, sampling an event from
$p_{\boldsymbol{X}}$ is done by a two step procedure of first sampling from
the simple distribution $\boldsymbol{z}\sim p_{\boldsymbol{Z}}$ and then
applying the bijective function $\boldsymbol{x}=g_{\phi}(\boldsymbol{z})$.
Such a procedure supports closed form sampling. Moreover, modern
approaches(Kingma et al., 2016; Kingma and Dhariwal, 2018; Rezende and
Mohamed, 2015) represent the function $g_{\phi}(\bullet)$ via a neural
network.

###  2.2. Problem Formulation

As mentioned in Section 1, we represent an activity via a continuous-time
action sequence, _i.e._ , a series of actions undertaken by users and their
corresponding time of occurrences. We derive each CTAS from annotated frames
of videos consisting of individuals performing certain activities.
Specifically, for every video, we have a sequence of activity labels being
performed in the video along with timestamps for each activity. Therefore,
each CTAS used in our dataset is derived from these sequences of a video.
Formally, we provide a detailed description of a CTAS in Definition 2.1.

###### Definition 0 (Continuous Time Action Sequence).

We define a continuous-time action sequence (CTAS) as a series of action
events taken by a user to achieve a particular goal. Specifically, we
represent a CTAS as
$\mathcal{S}_{k}=\\{e_{i}=(c_{i},t_{i})|i\in[k],t_{i}<t_{i+1}\\}$, where
$t_{i}\in\mathbb{R}^{+}$ is the start-time of the action,
$c_{i}\in\mathcal{C}$ is the discrete category or mark of the $i$-th action,
$\mathcal{C}$ is the set of all categories, $\Delta_{t,i}=t_{i}-t_{i-1}$ as
the inter-action time, and $\mathcal{S}_{k}$ denotes the sequence of first $k$
actions. Each CTAS has an associated result, $g\in\mathcal{G}$, that signifies
the goal of the CTAS. Here, $\mathcal{G}$ denotes the set of all possible
sequence goals.

To highlight the relationship between sequence goal and actions consider the
example of a CTAS with the goal of ‘making-coffee’, would comprise of actions
– ‘take-a-cup’, ‘pour-milk’, ‘add-coffee-powder’, ‘add-sugar’, and ‘stir’ – at
different time intervals. Given the aforementioned definitions, we formulate
the tasks of next action, sequence goal prediction, and sequence generation
as:

Input. A CTAS of all actions, $\mathcal{S}_{k}$, consisting of categories and
times of different actions that lead to a goal $g$.

Output. A probabilistic prediction model with three distinct tasks – (i) to
estimate the likelihood of the next action $e_{k+1}$ along with the action
category and occurrence time; (ii) to predict the goal of the CTAS being
modeled, _i.e._ , $\widehat{g}$; and (iii) a generative model to sample a
sequence of actions, $\widehat{\mathcal{S}}$ given the true sequence goal,
$g$.

##  3\. ProActive Model

In this section, we first present a high-level overview of the ProActive model
and then describe the neural parameterization of each component in detail.
Lastly, we provide a detailed description of its optimization and sequence
generation procedure.

###  3.1. High Level Overview

We use an MTPP denoted by $p_{\theta}(\cdot)$, to learn the generative
mechanism of a continuous-time action sequence. Moreover, we design the
sequence modeling framework of $p_{\theta}(\cdot)$ using a self-attention
based encoder-decoder model (Vaswani et al., 2017). Specifically, we embed the
actions in a CTAS, _i.e._ , $\mathcal{S}_{k}$, to a vector embedding, denoted
by $\boldsymbol{s}_{k}$, using a weighted aggregation of all past actions.
Therefore, $\boldsymbol{s}_{k}$ signifies a compact neural representation of
the sequence history, _i.e._ , all actions till the $k$-th index and their
marks and occurrence times. Recent research (Zuo et al., 2020; Zhang et al.,
2020) has shown that an attention-based modeling choice can better capture the
long-term dependencies as compared to RNN-based MTPP models (Du et al., 2016;
Mei and Eisner, 2017; Shchur et al., 2020; Omi et al., 2019). A detailed
description of the embedding procedure is given in Section 3.2.

We use our MTPP $p_{\theta}(\cdot)$ to estimate the generative model for the
$(k+1)$-th action conditioned on the past, _i.e._ , $p(e_{k+1})$ as:

(3)
$p_{\theta}(e_{k+1}|\boldsymbol{s}_{k})=\mathbb{P}_{\theta}(c_{k+1}|\boldsymbol{s}_{k})\cdot\rho_{\theta}(\Delta_{t,k+1}|\boldsymbol{s}_{k}),$

where, $\mathbb{P}_{\theta}(\cdot)$ and $\rho_{\theta}(\cdot)$ denote the
probability distribution of marks and the density function for inter-action
arrival times respectively. Note that both the functions are conditioned on
$\boldsymbol{s}_{k}$ and thus ProActive requires a joint optimizing procedure
for both – action time and mark prediction. Next, we describe the mechanism
used in ProActive to predict the next action and goal detection in a CTAS.

Next Action Prediction. We determine the most probable mark and time of the
next action, using $p_{\theta}(\cdot)$ via standard sampling techniques over
$\mathbb{P}_{\theta}(\cdot)$ and $\rho_{\theta}(\cdot)$ respectively (Du et
al., 2016; Shchur et al., 2020).

(4) $\widehat{e_{k+1}}\sim p_{\theta}(e_{k+1}|\boldsymbol{s}_{k}),$

In addition, to keep the history embedding up-to-date with the all past
actions, we iteratively update $\boldsymbol{s}_{k}$ to $\boldsymbol{s}_{k+1}$
by incorporating the details of action $e_{k+1}$.

Goal Detection. Since the history embedding, $\boldsymbol{s}_{k}$, represents
an aggregation of all past actions in a sequence, it can also be used to
capture the influences between actions and thus, can be extended to detect the
goal of the CTAS. Specifically, to detect the CTAS goal, we use a non-linear
transformation over $\boldsymbol{s}_{k}$ as:

(5) $\widehat{g}\sim\mathbb{P}_{g^{\prime}\in\mathcal{G}}(\Phi(s_{k})),$

where, $\mathbb{P}_{\bullet}$ denotes the distribution over all sequence goals
and $\Phi(\cdot)$ denotes the transformation via a fully-connected MLP layer.

###  3.2. Neural Parametrization

Here, we present a detailed description of the neural architecture of our
MTPP, $p_{\theta}(\cdot)$, and the optimization procedure in ProActive.
Specifically, we realize $p_{\theta}(\cdot)$ using a three layer architecture:

Input Layer. As mentioned in Section 2, each action $e_{i}\in\mathcal{S}_{k}$
is represented by a mark $c_{i}$ and time $t_{i}$. Therefore, we embed each
action as a combination of all these features as:

(6)
$\boldsymbol{y}_{i}=\boldsymbol{w}_{y,c}c_{i}+\boldsymbol{w}_{y,t}t_{i}+\boldsymbol{w}_{y,\Delta}\Delta_{t,i}+\boldsymbol{b}_{y},$

where $\boldsymbol{w}_{\bullet,\bullet},\boldsymbol{b}_{\bullet}$ are
trainable parameters and $\boldsymbol{y}_{i}\in\mathbb{R}^{D}$ denotes the
vector embedding for the action $e_{i}$ respectively. In other sections as
well, we denote weight and bias as $\boldsymbol{w}_{\bullet,\bullet}$ and
$\boldsymbol{b}_{\bullet,\bullet}$ respectively.

Self-Attention Layer. We use a masked self-attention layer to embed the past
actions to $\boldsymbol{s}_{k}$ and to interpret the influence between the
past and the future actions. In detail, we follow the standard attention
procedure (Vaswani et al., 2017) and first add a trainable positional
encoding, $\boldsymbol{p}_{i}$, to the action embedding, _i.e._ ,
$\boldsymbol{y}_{i}\leftarrow\boldsymbol{y}_{i}+\boldsymbol{p}_{i}$. Such
trainable encodings are shown to be more scalable and robust for long sequence
lengths as compared to those based on a fixed function (Kang and McAuley,
2018; Li et al., 2020). Later, to calculate an attentive aggregation of all
actions in the past, we perform three independent linear transformations on
the action representation to get the query, key, and value embeddings, _i.e._
,

(7)
$\boldsymbol{q}_{i}=\boldsymbol{W}^{Q}\boldsymbol{y}_{i},\quad\boldsymbol{k}_{i}=\boldsymbol{W}^{K}\boldsymbol{y}_{i},\quad\boldsymbol{v}_{i}=\boldsymbol{W}^{V}\boldsymbol{y}_{i},$

where,
$\boldsymbol{q}_{\bullet},\boldsymbol{k}_{\bullet},\boldsymbol{v}_{\bullet}$
denote the query, key, and value vectors respectively. Following standard
self-attention model, we represent $\boldsymbol{W}^{Q}$, $\boldsymbol{W}^{K}$
and $\boldsymbol{W}^{V}$ as trainable Query, Key, and Value matrices
respectively. Finally, we compute $\boldsymbol{s}_{k}$ conditioned on the
history as:

(8)
$\boldsymbol{s}_{k}=\sum_{i=1}^{k}\frac{\exp\left(\boldsymbol{q}_{k}^{\top}\boldsymbol{k}_{i}/\sqrt{D}\right)}{\sum_{i^{\prime}=1}^{k}\exp\left(\boldsymbol{q}_{k}^{\top}\boldsymbol{k}_{i^{\prime}}/\sqrt{D}\right)}\boldsymbol{v}_{i},$

where, $D$ denotes the number of hidden dimensions. Here, we compute the
attention weights via a softmax over the interactions between the query and
key embeddings of each action in the sequence and perform a weighted sum of
the value embeddings.

Now, given the representation $\boldsymbol{s}_{k}$, we use the attention
mechanism in Eqn. (8) and apply a feed-forward neural network to incorporate
the necessary non-linearity to the model as:

$\boldsymbol{s}_{k}\leftarrow\sum_{i=1}^{k}\big{[}\boldsymbol{w}_{s,m}\odot\textsc{ReLU}(\boldsymbol{s}_{i}\odot\boldsymbol{w}_{s,n}+\boldsymbol{b}_{s,n})+\boldsymbol{b}_{s,m}\big{]},$

where, $\boldsymbol{w}_{s,m},\boldsymbol{b}_{s,m}$ and
$\boldsymbol{w}_{s,n},\boldsymbol{b}_{s,n}$ are trainable parameters of the
outer and inner layer of the point-wise feed-forward layer.

To support faster convergence and training stability, we employ: (i) layer
normalization; (ii) stacking multiple self-attention blocks; and (iii) multi-
head attention. Since these are standard techniques (Ba et al., 2016; Vaswani
et al., 2017), we omit their mathematical descriptions in this paper.

Output Layer. At every index $k$, ProActive outputs the next action and the
most probable goal of the CTAS. We present the prediction procedure for each
of them as follows:

Action Prediction: We use the output of the self-attention layer,
$\boldsymbol{s}_{k}$ to estimate the mark distribution and time density of the
next event, _i.e._ , $\mathbb{P}_{\theta}(e_{k+1})$ and
$\rho_{\theta}(e_{k+1})$ respectively. Specifically, we model the
$\mathbb{P}_{\theta}(\cdot)$ as a softmax over all other marks as:

(9)
$\mathbb{P}_{\theta}(c_{k+1})=\frac{\exp\left(\boldsymbol{w}_{c,s}^{\top}\boldsymbol{s}_{i}+\boldsymbol{b}_{c,s}\right)}{\sum_{c^{\prime}=1}^{|\mathcal{C}|}\exp\left(\boldsymbol{w}_{c^{\prime},s}^{\top}\boldsymbol{s}_{i}+\boldsymbol{b}_{c^{\prime},s}\right)},$

where, $\boldsymbol{w}_{\bullet,\bullet}$ and
$\boldsymbol{b}_{\bullet,\bullet}$ are trainable parameters.

In contrast to standard MTPP approaches that rely on an intensity-based model
(Du et al., 2016; Mei and Eisner, 2017; Zuo et al., 2020; Zhang et al., 2020),
we capture the inter-action arrival times via a temporal normalizing flow
(NF). In detail, we use a LogNormal flow to model the temporal density
$\rho_{\theta}(\Delta_{t,k+1})$. Moreover, standard flow-based approaches
(Shchur et al., 2020; Mehrasa et al., 2019a) utilize a common NF for all
events in a sequence, _i.e._ , the arrival times of each event are determined
from a single or mixture of flows trained on all sequences. We highlight that
such an assumption restricts the ability to model the dynamics of a CTAS, as
unlike standard events, an action has three distinguishable characteristics –
(i) every action requires a minimum time for completion; (ii) the time taken
by a user to complete an action would be similar to the times of another user;
and (iii) similar actions require similar times to complete. For example, the
time taken to complete the action ‘add-coffee’ would require a certain minimum
time of completion and these times would be similar for all users.
Intuitively, the time for completing the action ‘add-coffee’ would be similar
to those for the action ‘add-sugar’.

To incorporate these features in ProActive, we identify actions with similar
completion times and model them via independent temporal flows. Specifically,
we cluster all actions $c_{i}\in\mathcal{C}$ into $\mathcal{M}$ non-
overlapping clusters based on the mean of their times of completion and for
each cluster we define a trainable embedding
$\boldsymbol{z}_{r}\in\mathbb{R}^{D}\,\forall r\in\\{1,\cdots,\mathcal{M}\\}$.
Later, we sample the start-time of the future action by conditioning our
temporal flows on the cluster of the current action as:

(10)
$\widehat{\Delta_{t,k+1}}\sim\textsc{LogNormal}\left(\boldsymbol{\mu}_{k},\boldsymbol{\sigma}^{2}_{k}\right),$

where, $[\boldsymbol{\mu}_{k},\boldsymbol{\sigma}^{2}_{k}]$, denote the mean
and variance of the log-normal temporal flow and are calculated via the
sequence embedding and the cluster embedding as:

(11)
$\boldsymbol{\mu}_{k}=\sum_{r=1}^{\mathcal{M}}\mathcal{R}(e_{k},r)\big{(}\boldsymbol{w}_{\mu}\left(\boldsymbol{s}_{k}\odot\boldsymbol{z}_{c,i}\right)+\boldsymbol{b}_{\mu}\big{)},$
(12)
$\boldsymbol{\sigma}^{2}_{k}=\sum_{r=1}^{\mathcal{M}}\mathcal{R}(e_{k},r)\big{(}\boldsymbol{w}_{\sigma}\left(\boldsymbol{s}_{k}\odot\boldsymbol{z}_{c,i}\right)+\boldsymbol{b}_{\sigma}\big{)},$

where $\boldsymbol{w}_{\bullet},\boldsymbol{b}_{\bullet}$ are trainable
parameters, $\mathcal{R}(e_{k},r)$ is an indicator function that determines if
event $e_{k}$ belongs to the cluster $r$ and $\boldsymbol{z}_{r}$ denotes the
corresponding cluster embedding. Such a cluster-based formulation facilitates
the ability of model to assign similar completion times for events in a same
cluster. To calculate the time of next action, we add the sampled time
difference to the time of the previous action $e_{k}$, _i.e._ ,

(13) $\widehat{t_{k+1}}=t_{k}+\widehat{\Delta_{t,k+1}}$

where, $\widehat{t_{k+1}}$ denotes the predicted time for the action
$e_{k+1}$.

Goal Detection: In contrast to other MTPP approaches (Du et al., 2016; Mei and
Eisner, 2017; Zuo et al., 2020; Zhang et al., 2020; Shchur et al., 2020), an
important feature of ProActive is identifying the goal of a sequence, _i.e._ ,
a hierarchy on top of the actions in a sequence, based on the past sequence
dynamics. To determine the goal of a CTAS, we utilize the history embedding
$\boldsymbol{s}_{k}$ as it encodes the inter-action relationships of all
actions in the past. Specifically, we use a non-linear transformation via a
feed-forward network, denoted as $\Phi(\cdot)$ over $\boldsymbol{s}_{k}$ and
apply a softmax over all possible goals.

(14)
$\Phi(\boldsymbol{s}_{k})=\textsc{ReLU}(\boldsymbol{w}_{\Phi,s}\boldsymbol{s}_{k}+\boldsymbol{b}_{\Phi,s}),$

where, $\boldsymbol{w}_{\bullet,\bullet},\boldsymbol{b}_{\bullet,\bullet}$ are
trainable parameters. We sample the most probable goal as in Eqn. (5).

We highlight that we predict the CTAS goal at each interval, though a CTAS has
only one goal. This is to facilitate early goal detection in comparison to
detecting the goal after traversing the entire CTAS. More details are given in
Section 3.3 and Section 3.4.

###  3.3. Early Goal Detection and Action Hierarchy

Here, we highlight the two salient features of ProActive– early goal detection
and modeling the goal-action hierarchy.

Early Goal Detection. Early detection of sequence goals has many applications
ranging from robotics to vision (Ryoo, 2011; Hoai and la Torre, 2012). To
facilitate early detection of the goal of a CTAS in ProActive, we devise a
ranking loss that forces the model to predict a non-decreasing detection score
for the correct goal category. Specifically, the detection score of the
correct goal at the $k$-th index of the sequence, denoted by
$p_{k}(g|\boldsymbol{s}_{k},\Phi)$, must be more than the scores assigned the
correct goal in the past. Formally, we define the ranking loss as:

(15)
$\mathcal{L}_{k,g}=\max\big{(}0,p^{*}_{k}(g)-p_{k}(g|\boldsymbol{s}_{k},\Phi)\big{)},$

where $p^{*}_{k}(g)$ denotes the maximum probability score given to the
correct goal in all past predictions.

(16) $p^{*}_{k}(g)=\max_{j\in\\{1,k-1\\}}p_{j}(g|\boldsymbol{s}_{j},\Phi),$

where $p_{k}(g)$ denotes the probability score for the correct goal at index
$k$. Intuitively, the ranking loss $\mathcal{L}_{k,g}$ would penalize the
model for predicting a smaller detection score for the correct CTAS goal than
any previous detection score for the same goal.

Action Hierarchy. Standard MTPP approaches assume the category of marks as
independent discrete variables, _i.e._ , the probability of an upcoming mark
is calculated independently (Du et al., 2016; Mei and Eisner, 2017; Zhang et
al., 2020; Zuo et al., 2020). Such an assumption restricts the predictive
ability while modeling CTAS, as in the latter case, there exists a hierarchy
between goals and actions that lead to the specific goal. Specifically,
actions that lead to a common goal may have similar dynamics and it is also
essential to model the relationships between the actions of different CTAS
with a common goal. We incorporate this hierarchy in ProActive along with our
next action prediction via an action-based ranking loss. In detail, we devise
a loss function similar to Eqn. 15 where we restrict the model to assign non-
decreasing probabilities to all actions leading to the goal of CTAS under
scrutiny.

(17)
$\mathcal{L}_{k,c}=\sum_{c^{\prime}\in\mathcal{C}^{*}_{g}}\max\big{(}0,p^{*}_{k}(c^{\prime})-p_{k}(c^{\prime}|\boldsymbol{s}_{k})\big{)},$

where $\mathcal{C}^{*}_{g},p_{k}(c^{\prime}|\boldsymbol{s}_{k})$ denote a set
of all actions in CTAS with the goal $g$ and the probability score for the
action $c^{\prime}\in\mathcal{C}^{*}_{g}$ at index $k$ respectively. Here,
$p^{*}_{k}(c^{\prime})$ denotes the maximum probability score given to action
$c^{\prime}$ in all past predictions and is calculated similar to Eqn. 16. We
regard $\mathcal{L}_{k,g}$ and $\mathcal{L}_{k,c}$ as margin losses, as they
aim to increase the difference between two prediction probabilities.

###  3.4. Optimization

We optimize the trainable parameters in ProActive, _i.e._ , the weight and
bias tensors ($\boldsymbol{w}_{\bullet,\bullet}$ and
$\boldsymbol{b}_{\bullet,\bullet}$) for our MTPP $p_{\theta}(\cdot)$, using a
two channels of training consisting of action and goal prediction.
Specifically, to optimize the ability of ProActive for predicting the next
action, we maximize the the joint likelihood for the next action and the log-
normal density distribution of the temporal flows.

(18)
$\mathscr{L}=\sum_{k=1}^{|\mathcal{S}|}\log\big{(}\mathbb{P}_{\theta}(c_{k+1}|\boldsymbol{s}_{k})\cdot\rho_{\theta}(\Delta_{t,k+1}|\boldsymbol{s}_{k})\big{)},$

where $\mathscr{L}$ denotes the joint likelihood, which we represent as the
sum of the likelihoods for all CTAS. In addition to action prediction, we
optimize the ProActive parameters for early goal detection via a temporally
weighted cross entropy (CE) loss over all sequence goals. Specifically, we
follow a popular reinforcement recipe of using a time-varying discount factor
over the prediction loss as:

(19)
$\mathcal{L}_{g}=\sum_{k=1}^{|\mathcal{S}|}\gamma^{k}\cdot\mathcal{L}_{\textsc{CE}}\big{(}p_{k}(g|\boldsymbol{s}_{k})\big{)},$

where
$\gamma\in[0,1],\mathcal{L}_{\textsc{CE}}\big{(}p_{k}(g|\boldsymbol{s}_{k})\big{)}$
denote the decaying factor and a standard softmax-cross-entropy loss
respectively. Such a recipe is used exhaustively for faster convergence of
reinforcement learning models (Sutton and Barto, 2018; Pitis, 2019). Here, the
discount factor penalizes the model for taking longer times for detecting the
CTAS goal by decreasing the gradient updates to the loss.

Margin Loss. In addition, we minimize the margin losses given in Section 3.3
with the current optimization procedure. Specifically, we minimize the
following loss:

(20)
$\mathcal{L}_{m}=\sum_{k=1}^{|\mathcal{S}|}\mathcal{L}_{k,g}+\mathcal{L}_{k,c},$

where $\mathcal{L}_{k,g}$ and $\mathcal{L}_{k,c}$ are margin losses defined in
Eqn. 15 and Eqn. 17 respectively. We learn the parameters of ProActive using
an Adam (Kingma and Ba, 2015) optimizer for both likelihood and prediction
losses.

###  3.5. Sequence Generation

A crucial contribution of this paper via ProActive is an end-to-end generation
of action sequences. Specifically, given the CTAS goal as input, we can
generate a most probable sequence of actions that may lead to that specific
goal. Such a feature has a range of applications from sports analytics
(Mehrasa et al., 2017), forecasting (Deshpande et al., 2021), identifying the
duration of an activity (Mehrasa et al., 2019b), _etc._

A standard approach for training a sequence generator is to sample future
actions in a sequence and then compare with the true actions (Yoon et al.,
2019). However, such a procedure has multiple drawbacks as it is susceptible
to noises during training and deteriorates the scalability of the model.
Moreover, we highlight that such a sampling based training cannot be applied
to a self-attention-based model as it requires a fixed sized sequence as input
(Vaswani et al., 2017). Therefore, we resort to a two-step generation
procedure that is defined below:

  1. (1)

Pre-Training: The first step requires training all ProActive parameters for
action prediction and goal detection. This step is necessary to model the
relationships between actions and goals and we represent the set of optimized
parameters as $\theta^{*}$ and the corresponding MTPP as
$p_{\theta^{*}}(\cdot)$ respectively.

  2. (2)

Iterative Sampling: We iteratively sample events and update parameters via our
trained MTPP till the model predicts the correct goal for CTAS or we encounter
an <EOS> action. Specifically, using $p_{\theta^{*}}(\cdot)$ and the first
real action ($e_{1}$) as input, we calculate the detection score for the
correct goal, _i.e._ , $p_{1}(g|\boldsymbol{s}_{k})$ and while its value is
highest among all probable goals, we sample the mark and time of next action
using Eqn. 9 and Eqn. 10 respectively.

Such a generation procedure harnesses the fast sampling of temporal
normalizing flows and simultaneously is conditioned on the action and goal
relationships. A detailed pseudo-code of sequence generation procedure used in
ProActive is given in Algorithm 1.

1 Input: $g$: Goal of CTAS

2 $e_{1}$: First Action

3 $p_{\theta^{*}}(\cdot)$: Trained MTPP

4 Output: $\widehat{S}$: Generated CTAS $\mathcal{S}_{1}\leftarrow e_{1}$

5 $k=1$

6 while  _$k <\mathtt{max\\_len}$_ do

7 Sample the mark of next action:
$\widehat{c_{k+1}}\sim\mathbb{P}_{\theta^{*}}(\boldsymbol{s}_{k})$

8 Sample the time of next action:
$\widehat{t_{k+1}}\sim\rho_{\theta^{*}}(\boldsymbol{s}_{k})$

9 Add to CTAS: $\mathcal{S}_{k+1}\leftarrow\mathcal{S}_{k}+e_{k+1}$

10 Update the MTPP parameters $\boldsymbol{s}_{k+1}\leftarrow
p(\boldsymbol{s}_{k},e_{k+1})$

11 Calculate most probable goal: $\widehat{g}_{k}=\max_{\forall
g^{\prime}}\big{(}p_{k}(g^{\prime}|\boldsymbol{s}_{k})\big{)}$

12 if  _$\widehat{g_{i}}!=g\,\mathrm{or}\,\widehat{c_{k+1}}==\mathtt{ <EOS>}$_
then

13 Add EOS mark:
$\widehat{\mathcal{S}}\leftarrow\mathcal{S}_{k+1}+\mathtt{<EOS>}$

14 Exit the sampling procedure: Break

15

16 Increment iteration: $k\leftarrow k+1$

17

18 Return generated CTAS: return $\widehat{\mathcal{S}}$

Algorithm 1 Sequence Generation with ProActive

##  4\. Experiments

In this section, we present the experimental setup and the empirical results
to validate the efficacy of ProActive. Through our experiments we aim to
answer the following research questions:

  * RQ1

What is the action-mark and time prediction performance of ProActive in
comparison to the state-of-the-art baselines?

  * RQ2

How accurately and quickly can ProActive identify the goal of an activity
sequence?

  * RQ3

How effectively can ProActive generate an action sequence?

  * RQ4

How does the action prediction performance of ProActive vary with different
hyperparameters values?

Table 1. Performance of all the methods in terms of action prediction accuracy (APA) and mean absolute error (MAE) across all datasets. Bold fonts and underline indicate the best performer and the best baseline respectively. Results marked $\dagger$ are statistically significant (_i.e._ , two-sided $t$-test with $p\leq 0.1$) over the best baseline. | Action Prediction Accuracy (APA) | Mean Absolute Error (MAE)
---|---|---
| Breakfast | Multi-THUMOS | Activity-Net | Breakfast | Multi-THUMOS | Activity-Net
NHP (Mei and Eisner, 2017) |  0.528$\pm$0.024 |  0.272$\pm$0.019 |  0.684$\pm$0.034 |  0.411$\pm$0.019 | 0.017$\pm$0.002 |  0.796$\pm$0.045
AVAE (Mehrasa et al., 2019b) |  0.533$\pm$0.028 |  0.279$\pm$0.022 |  0.678$\pm$0.036 |  0.417$\pm$0.021 |  0.018$\pm$0.002 |  0.803$\pm$0.049
RMTPP (Du et al., 2016) |  0.542$\pm$0.022 |  0.274$\pm$0.017 |  0.683$\pm$0.034 | 0.403$\pm$0.018 | 0.017$\pm$0.002 | 0.791$\pm$0.046
SAHP (Zhang et al., 2020) |  0.547$\pm$0.031 |  0.287$\pm$0.023 |  0.688$\pm$0.042 |  0.425$\pm$0.031 |  0.019$\pm$0.003 |  0.820$\pm$0.072
THP (Zuo et al., 2020) | 0.559$\pm$0.028 | 0.305$\pm$0.018 | 0.693$\pm$0.038 |  0.413$\pm$0.023 |  0.019$\pm$0.002 |  0.806$\pm$0.061
ProAct-c |  0.561$\pm$0.027 |  0.297$\pm$0.020 |  0.698$\pm$0.038 |  0.415$\pm$0.027 |  0.015$\pm$0.002 |  0.774$\pm$0.054
ProAct-t |  0.579$\pm$0.025 |  0.306$\pm$0.018 |  0.722$\pm$0.035 |  0.407$\pm$0.025 |  0.015$\pm$0.002 |  0.783$\pm$0.058
ProActive | 0.583$\pm$0.027$\dagger$ | 0.316$\pm$0.019 | 0.728$\pm$0.037$\dagger$ | 0.364$\pm$0.028$\dagger$ | 0.013$\pm$0.002$\dagger$ | 0.742$\pm$0.059$\dagger$

###  4.1. Datasets

To evaluate ProActive, we need time-stamped action sequences and their goals.
Therefore, we derive CTAS from three activity modeling datasets sourced from
different real-world applications – cooking, sports, and collective activity.
The datasets vary significantly in terms of origin, sparsity, and sequence
lengths. We highlight the details of each of these datasets below:

  * •

Breakfast (Kuehne et al., 2014). This dataset contains CTAS derived from 1712
videos of different people preparing breakfast. The actions in a CTAS and
sequence goals can be classified into 48 and 9 classes respectively. These
actions are performed by 52 different individuals in 18 different kitchens.

  * •

Multi-THUMOS (Yeung et al., 2015). A sports activity dataset that is designed
for action recognition in videos. We derive the CTAS using 400 videos of
individuals involved in different sports such as discus throw, baseball,
_etc._ The actions and goals can be classified into 65 and 9 classes
respectively and on average, there are 10.5 action class labels per video.

  * •

Activity-Net (Fabian Caba Heilbron and Niebles, 2015). This dataset comprises
of activity categories collected from 591 YouTube videos with a total of 49
action labels and 14 goals.

We highlight that in Activity-Net, many of the videos are shot by amateurs in
many uncontrolled environments, the variances within the CTAS of the same goal
are often large, and the lengths of CTAS vary and are often long and complex.

###  4.2. Baselines

We compare the action prediction performance of ProActive with the following
state-of-the-art methods:

  * RMTPP (Du et al., 2016): A recurrent neural network that models time differences to learn a representation of the past events.

  * NHP (Mei and Eisner, 2017): Models an MTPP using continuous-time LSTMs for capturing the temporal evolution of sequences.

  * AVAE (Mehrasa et al., 2019b): A variational auto-encoder based MTPP framework designed specifically for activities in a sequence.

  * SAHP (Zhang et al., 2020): A self-attention model to learn the temporal dynamics using an aggregation of historical events.

  * THP (Zuo et al., 2020): Extends the transformer model (Vaswani et al., 2017) to include the conditional intensity of event arrival and the inter-mark influences.

We omit comparison with other continuous-time models (Omi et al., 2019; Shchur
et al., 2020; Xiao et al., 2017a, b; Hawkes, 1971) as they have already been
outperformed by these approaches.

###  4.3. Evaluation Criteria

Given the dataset $\mathcal{D}$ of $N$ action sequences, we split them into
training and test set based on the goal of the sequence. Specifically, for
each goal $g\in\mathcal{G}$, we consider 80% of the sequences as the training
set and the other last 20% as the test set. We evaluate ProActive and all
baselines on the test set in terms of (i) mean absolute error (MAE) of
predicted times of action, and (ii) action prediction accuracy (APA) described
as:

(21)
$\mathrm{MAE}=\frac{1}{|\mathcal{S}|}\sum_{e_{i}\in\mathcal{S}}[|t_{i}-\widehat{t}_{i}|],\quad\mathrm{APA}=\frac{1}{|\mathcal{S}|}\sum_{e_{i}\in\mathcal{S}}\\#(c_{i}=\widehat{c}_{i}),$

where, $\widehat{t_{i}}$ and $\widehat{c_{i}}$ are the predicted time and type
the $i$-th action in test set. Moreover, we follow a similar protocol to
evaluate the sequence generation ability of ProActive and other models. For
goal prediction, we report the results in terms of accuracy (ratio) calculated
across all sequences. We calculate confidence intervals across 5 independent
runs.

###  4.4. Experimental Setup

All our implementations and datasets are publicly available at:
https://github.com/data-iitd/proactive/.

System Configuration. All our experiments were done on a server running Ubuntu
16.04. CPU: Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz , RAM: 125GB and GPU:
NVIDIA Tesla T4 16GB DDR6.

Parameter Settings. For our experiments, we set $D=16$, $\mathcal{M}=8$,
$\gamma=0.9$ and weigh the margin loss $\mathcal{L}_{m}$ by 0.1. In addition,
we set a $l_{2}$ regularizer over the parameters with coefficient value
$0.001$.

(a) Multi-THUMOS

(b) Activity-Net

Figure 1.  Real life true and predicted inter-arrival times $\Delta_{t,k}$ of
different events $e_{k}$ for (a) Multi-THUMOS and (b) Activity-Net datasets.
The results show that the true arrival times match with the times predicted by
ProActive.

(a) Breakfast

(b) Multi-THUMOS

(c) Activity-Net

Figure 2. Sequence goal prediction performance of ProActive, its variants –
ProAct-m and ProAct-$\gamma$, and other baseline models. The results show that
ProActive can effectively detect the CTAS goal even with smaller test
sequences as input.

###  4.5. Action Prediction (RQ1)

We report on the performance of action prediction of different methods across
all our datasets in Table 1. In addition, we include two variants of
ProActive– (i) ProAct-c, represents our model without the goal-action
hierarchy loss (Eqn. 17) and cluster-based flows (Eqn. 11 and 12), and (ii)
ProAct-t, represents our model without cluster-based flows. From Table 1, we
note the following:

  * $\bullet$

ProActive consistently yields the best prediction performance on all the
datasets. In particular, it improves over the strongest baselines by 8-27% for
time prediction and by 2-7% for action prediction. These results signify the
drawbacks of using standard sequence approaches for modeling a temporal action
sequence.

  * $\bullet$

RMTPP (Du et al., 2016) is the second-best performer in terms of MAE of time
prediction in almost all the datasets. We also note that for Activity-
Netdataset, THP (Zuo et al., 2020) outperforms RMTPP for action category
prediction. However, ProActive still significantly outperforms these models
across all metrics.

  * $\bullet$

Neural MTPP methods that deploy a self-attention for modeling the distribution
of action – namely THP, SAHP, and ProActive, achieve better performance in
terms of category prediction.

  * $\bullet$

Despite AVAE (Mehrasa et al., 2019b) being a sequence model designed
specifically for activity sequences, other neural methods that incorporate
complex structures using self-attention or normalizing flows easily outperform
it.

To sum up, our empirical analysis suggests that ProActive can better model the
underlying dynamics of a CTAS as compared to all other baseline models.
Moreover, the performance gain over ProAct-c and ProAct-t highlights the need
for modeling action hierarchy and cluster-based flows.

Qualitative Assessment. We also perform a qualitative analysis to highlight
the ability of ProActive for modeling the inter-arrival times for action
prediction. Specifically, we plot the actual inter-action time differences and
the time-difference predicted by ProActive in Figure 1 for Multi-THUMOS and
Activity-Net datasets. From the results, we note that the predicted inter-
arrival times closely match with the true inter-arrival times and ProActive is
even able to capture large time differences (peaks). For brevity, we omitted
the results for Activity-Net dataset.

###  4.6. Goal Prediction (RQ2)

Here, we evaluate the goal detection performance of ProActive along with other
baselines. To highlight the early goal detection ability of our model, we
report the results across different variants of the test set, _i.e._ , with
the initial 30% and 60% of the actions in the CTAS in terms of goal prediction
accuracy (GPA). In addition, we introduce two novel baselines, LSTM-c, and
LSTM-t, that detect the CTAS goal using just the types and the times of
actions respectively. We also compare with the two best-performing MTPP
baselines – RMTPP and THP which we extend for the task of goal detection by a
$k$-means clustering algorithm. In detail, we obtain the sequence embedding,
say $\boldsymbol{s}_{k}$ using the MTPP models and then cluster them into
$|\mathcal{G}|$ clusters based on their cosine similarities and perform a
maximum polling across each cluster, _i.e._ , predict the most common goal for
each cluster as the goal for all CTAS in the same cluster. In addition, we
introduce two new variants of our model to analyze the benefits of early goal
detection procedures in ProActive– (i) ProActive-m, represents our model
without the goal-based margin loss given in Eqn. 15 and (ii)
ProActive-$\gamma$, is our model without the discount-factor weight in Eqn.
19. We also report the results for the complete model ProActive.

The results for goal detection in Figure 2, show that the complete design of
ProActive achieves the best performance among all other models. We also note
that the performance of MTPP based models deteriorates significantly for this
new task which shows the unilateral nature of the prediction prowess of MTPP
models, unlike ProActive. Interestingly, the variant of ProActive without the
margin loss ProActive-m performs poorly as compared to the one without the
discount-factor, ProActive-$\gamma$. This could be attributed to better
convergence guarantees with a margin-based loss over the latter. Finally, we
observe that standard LSTM models are easily outperformed by our model, thus
reinforcing the need for joint training of types and action times.

(a) Breakfast

(b) Multi-THUMOS

(c) Activity-Net

Figure 3.  Sequence Generation results for ProActive and other baselines in
terms of APA for action prediction.

(a) Breakfast

(b) Multi-THUMOS

(c) Activity-Net

Figure 4.  Sequence Generation results for ProActive and other baselines in
terms of MAE for time prediction.

###  4.7. Sequence Generation (RQ3)

Here, we evaluate the sequence generation ability of ProActive. Specifically,
we generate all the sequences in the test set by giving the true goal of the
CTAS and the first action as input to the procedure described in Section 3.5.
However, there may be differences in lengths of the generated and true
sequences, _i.e._ , the length of generated sequences is usually greater than
the true CTAS. Therefore, we compare the actions in the true sequence with the
initial $|\mathcal{S}|$ generated actions. Such an evaluation procedure
provides us the flexibility of comparing with other MTPP models such as RMTPP
(Du et al., 2016) and THP (Zuo et al., 2020). As these models cannot be used
for end-to-end sequence generation, we alter their underlying model for
forecasting future actions given the first action and then iteratively update
and sample from the MTPP parameters. We report the results in terms of APA and
MAE for action and time prediction in Figure 3 and Figure 4 respectively. The
results show that ProActive can better capture the generative dynamics of a
CTAS in comparison to other MTPP models. We also note that the prediction
performance deteriorates significantly in comparison to the results given in
Table 1. This could be attributed to the error that gets compounded in farther
predictions made by the model. Interestingly, the performance advantage that
ProActive has over THP and RMTPP is further widened during sequence
generation.

Length Comparison. Here, we report the results for length comparison of the
generated sequence and the true sequence. Specifically, we identify the count
of instances where ProActive was able to effectively capture the generative
mechanism of a sequence as:

(22)
$\mathrm{CL}=\frac{1}{N}\sum_{\forall\mathcal{S}}\\#(|\mathcal{S}|=|\widehat{\mathcal{S}}|),$

where CL denotes the Correct-Length ratio with values $0.21$, $0.11$, and
$0.16$ for datasets Breakfast, Multi-THUMOS, and Activity-Net respectively. On
a coarse level these results might seem substandard, however, given the
difficulty associated with the problem of sequence generation using just the
CTAS goal, we believe these values are satisfactory. Moreover, we believe that
the sequence generation procedure of ProActive opens up new frontiers for
generating action sequences.

(a) Across $D$

(b) Across $\mathcal{M}$

(c) Across $\gamma$

Figure 5.  ProActive sensitivity for Activity-Net dataset with different
hyper-parameter values.

###  4.8. Parameter Sensitivity (RQ4)

Finally, we perform the sensitivity analysis of ProActive over key parameters:
(i) $D$, the dimension of embeddings; (ii) $\mathcal{M}$, no. clusters for
log-normal flow; and (iii) $\gamma$, the discount factor described in Eqn.19.
For brevity purposes, we only report results on Activity dataset, but other
datasets displayed a similar behavior. From Figure 5 we show the performance
of ProActive across different hyper-parameter values. We note that as we
increase the embedding dimension the performance first increases since it
leads to better modeling. However, beyond a point the complexity of the model
increases requiring more training to achieve good results, and hence we see
its performance deteriorating. We see a similar trend for $\mathcal{M}$, as
increasing the number of clusters leads to better results before saturating at
a certain point. We found $\mathcal{M}=5$ to be the optimal point across
datasets in our experiments. Finally across $\gamma$, we notice that smaller
values for gamma penalize the loss function for detecting the goal late,
however, it deteriorates the action prediction performance of ProActive.
Therefore, we found $\gamma=0.9$ as the best trade-off between goal and action
prediction.

Scalability. For all datasets, the runtimes for training ProActive are within
1 hour and thus are within the practical range for deployment in real-world
scenarios. These running times further reinforce our design choice of using a
neural MTPP due to their faster learning and closed-form sampling (Shchur et
al., 2020; Mehrasa et al., 2019a).

##  5\. Related Work

In this section, we introduce key related work for this paper. It mainly falls
into – activity prediction and temporal point processes.

###  5.1. Activity Prediction

Activity modeling in videos is a widely used application with recent
approaches focusing on frame-based prediction. Lan et al. (2014) predicts the
future actions via hierarchical representations of short clips, Tahmida Mahmud
and Roy-Chowdhury (2017) jointly predicts future activity ad the starting time
by capturing different sequence features and a similar procedure is adopted by
(Yazan Abu Farha and Gall, 2018) that predicts the action categories of a
sequence of future activities as well as their starting and ending time. Ma et
al. (2016) propose a method for early classification of a sequence of frames
extracted from a video by maximizing the margin-based loss between the correct
and the incorrect categories, however, it is limited to visual data and cannot
incorporate the action-times. This limits its ability for use in CTAS, and
sequence generation. A recent approach (Mehrasa et al., 2019b) proposed to
model the dynamics of action sequences using a variational auto-encoder built
on top of a temporal point process. We consider their work as most relevant to
ProActive as it also addressed the problem of CTAS modeling. However, as shown
in our experiments ProActive was able to easily outperform it across all
metrics. This could be attributed to the limited modeling capacity of VAE over
normalizing flows. Moreover, their sampling procedure could not be extended to
sequence generation. Therefore, in contrast to the past literature, ProActive
is the first application of MTPP models for CTAS modeling and end-to-end
sequence generation.

###  5.2. Temporal Point Processes

In recent years neural Marked Temporal Point Processes (MTPP) have shown a
significant promise in modeling a variety of continuous-time sequences in
healthcare (Rizoiu et al., 2018; Gupta et al., 2022), finance (Zhang et al.,
2020; Bacry et al., 2015), education (Yao et al., 2021), and social networks
(Zhao et al., 2015; Mei and Eisner, 2017; Zuo et al., 2020; Gupta et al.,
2021; Gupta and Bedathur, 2021; Likhyani et al., 2020). However, due to the
limitations of traditional MTPP models, in recent years, neural enhancements
to MTPP models have significantly enhanced the predictive power of these
models. Specifically, they combine the continuous-time approach from the point
process with deep learning approaches and thus, can better capture complex
relationships between events. The most popular approaches (Du et al., 2016;
Mei and Eisner, 2017; Shchur et al., 2020; Zhang et al., 2020; Zuo et al.,
2020) use different methods to model the time- and mark-distribution via
neural networks. Specifically, Du et al. (2016) embeds the event history to a
vector representation via a recurrent encoder that updates its state after
parsing each event in a sequence; Mei and Eisner (2017) modified the LSTM
architecture to employ a continuous-time state evolution; Shchur et al. (2020)
replaced the intensity function with a mixture of log-normal flows for closed-
form sampling; Zhang et al. (2020) utilized the transformer architecture
Vaswani et al. (2017) to capture the long-term dependencies between events in
the history embedding and Zuo et al. (2020) used the transformer architecture
for sequence embedding but extended it to graph settings as well. However,
these models are not designed to capture the generative distribution of future
events in human-generated sequences.

##  6\. Conclusion

Standard deep-learning models are not designed for modeling sequences of
actions localized in continuous time. However, neural MTPP models overcome
this drawback but have limited ability to model the events performed by a
human. Therefore, in this paper, we developed a novel point process flow-based
architecture called ProActive for modeling the dynamics of a CTAS. ProActive
solves the problems associated with action prediction, goal prediction, and
for the first time, we extend MTPP for end-to-end CTAS generation. Our
experiments on three large-scale diverse datasets reveal that ProActive can
significantly improve over the state-of-the-art baselines across all metrics.
Moreover, the results also reinforce the novel ability of ProActive to
generate a CTAS. We hope that such an application will open many horizons for
using MTPP in a wide range of tasks. As a future work, we plan to incorporate
a generative adversarial network (Goodfellow et al., 2014; Xiao et al., 2017a)
with action sampling and train the generator and the MTPP model
simultaneously.

###### Acknowledgements.

This work was partially supported by a DS Chair of AI fellowship to Srikanta
Bedathur.

## References

  * (1)
  * Alzantot et al. (2017) Moustafa Alzantot, Supriyo Chakraborty, and Mani Srivastava. 2017\.  Sensegen: A deep learning architecture for synthetic sensor data generation. In _PerCom (Workshops)_.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016\.  Layer Normalization.  _arXiv preprint arXiv:1607.06450_ (2016).
  * Bacry et al. (2015) Emmanuel Bacry, Iacopo Mastromatteo, and Jean-François Muzy. 2015\.  Hawkes processes in finance.  _arXiv preprint arXiv:1502.04592_ (2015).
  * Deshpande et al. (2021) Prathamesh Deshpande, Kamlesh Marathe, Abir De, and Sunita Sarawagi. 2021.  Long Horizon Forecasting With Temporal Point Processes. In _WSDM_.
  * Donahue et al. (2020) Seth R Donahue, Li Jin, and Michael E Hahn. 2020.  User Independent Estimations of Gait Events With Minimal Sensor Data.  _IEEE Journal of Biomedical and Health Informatics_ 25, 5 (2020), 1583–1590.
  * Du et al. (2016) Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. 2016\.  Recurrent marked temporal point processes: Embedding event history to vector. In _KDD_.
  * Fabian Caba Heilbron and Niebles (2015) Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. 2015\.  ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding. In _CVPR_.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative Adversarial Networks. In _NeurIPS_.
  * Gupta and Bedathur (2021) Vinayak Gupta and Srikanta Bedathur. 2021.  Region Invariant Normalizing Flows for Mobility Transfer. In _CIKM_.
  * Gupta and Bedathur (2022) Vinayak Gupta and Srikanta Bedathur. 2022.  Doing More with Less: Overcoming Data Scarcity for POI Recommendation via Cross-Region Transfer.  _arXiv preprint arXiv:2201.06095_ (2022).
  * Gupta et al. (2021) Vinayak Gupta, Srikanta Bedathur, Sourangshu Bhattacharya, and Abir De. 2021.  Learning Temporal Point Processes with Intermittent Observations. In _AISTATS_.
  * Gupta et al. (2022) Vinayak Gupta, Srikanta Bedathur, and Abir De. 2022.  Learning Temporal Point Processes for Efficient Retrieval of Continuous Time Event Sequences. In _AAAI_.
  * Haradal et al. (2018) Shota Haradal, Hideaki Hayashi, and Seiichi Uchida. 2018\.  Biosignal data augmentation based on generative adversarial networks. In _EMBC_.
  * Hawkes (1971) Alan G Hawkes. 1971\.  Spectra of some self-exciting and mutually exciting point processes.  _Biometrika_ 58, 1 (1971), 83–90.
  * Hoai and la Torre (2012) Minh Hoai and Fernando De la Torre. 2012.  Max-margin early event detectors. In _CVPR_.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-Attentive Sequential Recommendation. In _ICDM_.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _ICLR_.
  * Kingma and Dhariwal (2018) Durk P Kingma and Prafulla Dhariwal. 2018.  Glow: Generative flow with invertible 1x1 convolutions. In _NeurIPS_.
  * Kingma et al. (2016) Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016\.  Improved variational inference with inverse autoregressive flow. In _NeurIPS_.
  * Kuehne et al. (2014) H. Kuehne, A. B. Arslan, and T. Serre. 2014.  The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities. In _CVPR_.
  * Lan et al. (2014) Tian Lan, Tsung-Chuan Chen, and Silvio Savarese. 2014\.  A hierarchical representation for future action prediction. In _ECCV_.
  * Li et al. (2020) Jiacheng Li, Yujie Wang, and Julian McAuley. 2020.  Time Interval Aware Self-Attention for Sequential Recommendation. In _WSDM_.
  * Likhyani et al. (2020) Ankita Likhyani, Vinayak Gupta, PK Srijith, P Deepak, and Srikanta Bedathur. 2020.  Modeling Implicit Communities from Geo-tagged Event Traces using Spatio-Temporal Point Processes. In _WISE_.
  * Luo et al. (2020) Yue Luo, Sarah M Coppola, Philippe C Dixon, Song Li, Jack T Dennerlein, and Boyi Hu. 2020\.  A database of human gait performance on irregular and uneven surfaces collected by wearable sensors.  _Scientific data_ 7, 1 (2020), 1–9.
  * Ma et al. (2016) Shugao Ma, Leonid Sigal, and Stan Sclaroff. 2016.  Learning Activity Progression in LSTMs for Activity Detection and Early Detection. In _CVPR_.
  * Mehrasa et al. (2019a) Nazanin Mehrasa, Ruizhi Deng, Mohamed Osama Ahmed, Bo Chang, Jiawei He, Thibaut Durand, Marcus Brubaker, and Greg Mori. 2019a.  Point Process Flows.  _arXiv preprint arXiv:1910.08281_ (2019).
  * Mehrasa et al. (2019b) Nazanin Mehrasa, Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. 2019b.  A Variational Auto-Encoder Model for Stochastic Point Processes. In _CVPR_.
  * Mehrasa et al. (2017) Nazanin Mehrasa, Yatao Zhong, Frederick Tung, Luke Bornn, and Greg Mori. 2017.  Learning person trajectory representations for team activity analysis.  _arXiv preprint arXiv:1706.00893_ (2017).
  * Mei and Eisner (2017) Hongyuan Mei and Jason M Eisner. 2017.  The neural hawkes process: A neurally self-modulating multivariate point process. In _NeurIPS_.
  * Omi et al. (2019) Takahiro Omi, Naonori Ueda, and Kazuyuki Aihara. 2019\.  Fully Neural Network based Model for General Temporal Point Processes. In _NeurIPS_.
  * Pitis (2019) Silviu Pitis. 2019\.  Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach. In _AAAI_.
  * Rezende and Mohamed (2015) Danilo Rezende and Shakir Mohamed. 2015.  Variational inference with normalizing flows. In _ICML_.
  * Rizoiu et al. (2018) Marian-Andrei Rizoiu, Swapnil Mishra, Quyu Kong, Mark Carman, and Lexing Xie. 2018\.  SIR-Hawkes: on the Relationship Between Epidemic Models and Hawkes Point Processes. In _WWW_.
  * Rizoiu et al. (2017) Marian-Andrei Rizoiu, Lexing Xie, Scott Sanner, Manuel Cebrian, Honglin Yu, and Pascal Van Hentenryck. 2017.  Expecting to be hip: Hawkes intensity processes for social media popularity. In _WWW_.
  * Ryoo (2011) M. Ryoo. 2011.  Human activity prediction: Early recognition of ongoing activities from streaming videos. In _ICCV_.
  * Sharma et al. (2021) Karishma Sharma, Yizhou Zhang, Emilio Ferrara, and Yan Liu. 2021\.  Identifying Coordinated Accounts on Social Media through Hidden Influence and Group Behaviours. In _KDD_.
  * Shchur et al. (2020) Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. 2020\.  Intensity-Free Learning of Temporal Point Processes. In _ICLR_.
  * Sutton and Barto (2018) Richard S. Sutton and Andrew G. Barto. 2018.  _Reinforcement Learning: An Introduction_.
  * Tahmida Mahmud and Roy-Chowdhury (2017) Mahmudul Hasan Tahmida Mahmud and Amit K. Roy-Chowdhury. 2017.  Joint prediction of activity labels and starting times in untrimmed videos. In _ICCV_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _NeurIPS_.
  * Xiao et al. (2017a) Shuai Xiao, Mehrdad Farajtabar, Xiaojing Ye, Junchi Yan, Le Song, and Hongyuan Zha. 2017a.  Wasserstein Learning of Deep Generative Point Process Models. In _NeurIPS_.
  * Xiao et al. (2017b) Shuai Xiao, Junchi Yan, Xiaokang Yang, Hongyuan Zha, and Stephen M Chu. 2017b.  Modeling the Intensity Function of Point Process Via Recurrent Neural Networks. In _AAAI_.
  * Yao et al. (2021) M. Yao, S. Zhao, S. Sahebi, and R. Feyzi Behnagh. 2021\.  Stimuli-Sensitive Hawkes Processes for Personalized Student Procrastination Modeling. In _WWW_.
  * Yazan Abu Farha and Gall (2018) Alexander Richard Yazan Abu Farha and Juergen Gall. 2018.  When Will You Do What? - Anticipating Temporal Occurrences of Activities. In _CVPR_.
  * Yeung et al. (2015) Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, and Li Fei-Fei. 2015\.  Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos.  _arXiv preprint arXiv:1507.05738_ (2015).
  * Yoon et al. (2019) Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. 2019\.  Time-series Generative Adversarial Networks. In _NeurIPS_.
  * Zhang et al. (2020) Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. 2020\.  Self-attentive Hawkes processes. In _ICML_.
  * Zhao et al. (2015) Q. Zhao, M. Erdogdu, H. He, A. Rajaraman, and J. Leskovec. 2015.  SEISMIC: A Self-Exciting Point Process Model for Predicting Tweet Popularity. In _KDD_.
  * Zuo et al. (2020) Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. 2020.  Transformer Hawkes Process. In _ICML_.