# More Effective Software Repository Mining

Adam Tutko<EMAIL_ADDRESS>0003-2412-5409 University of
TennesseeKnoxvilleTN37996 ,  Austin Z. Henley<EMAIL_ADDRESS>0003-1069-2795
University of TennesseeKnoxvilleTN37996 and  Audris Mockus<EMAIL_ADDRESS>0002-7987-7598 University of TennesseeKnoxvilleTN37996

###### Abstract.

Background: Data mining and analyzing of public Git software repositories is a
growing research field. The tools used for studies that investigate a single
project or a group of projects have been refined, but it is not clear whether
the results obtained on such “convenience samples” generalize. Aims: This
paper aims to elucidate the difficulties faced by researchers who would like
to ascertain the generalizability of their findings by introducing an
interface that addresses the issues with obtaining representative samples.
Results: To do that we explore how to exploit the World of Code system to make
software repository sampling and analysis much more accessible. Specifically,
we present a resource for Mining Software Repository researchers that is
intended to simplify data sampling and retrieval workflow and, through that,
increase the validity and completeness of data. Conclusions: This system has
the potential to provide researchers a resource that greatly eases the
difficulty of data retrieval and addresses many of the currently standing
issues with data sampling.

##  1\. Introduction

When developing software, it is a common occurrence for developers to rely on
the Git version control system111https://git-scm.com/. This is done primarily
to make collaboration between developers easier and because it provides a
fail-safe against catastrophic errors. This reliance on Git means that the
software development process generates a lot of publicly available data. This
data could be immensely useful to Mining Software Repository (MSR) researchers
in facilitating creation of tools to support developers. MSR researchers have
realized this potential and the field of research has experienced rapid growth
in recent years. However, due to the vastness of the field, there are a lot of
difficulties faced when first retrieving data.

Due to the immensity of the Git ecosystem, one’s ability to do any form of
analysis often requires sampling of the data. However, if researchers are not
careful when sampling, the data could easily be flawed. Due to the vastness of
the dataset, it is easy to retrieve bad data or heterogeneous data
(Trautsch2016MSR, ). This process is made even more difficult for MSR
researchers focused on Git data because the retrieval process requires using
difficult APIs or manual crawling of Git repositories. Many MSR papers
currently use these methods to perform data retrieval and the difficulties
with the retrieval process is well documented (Kolovos2019MSR, ),
(Sakamoto2012IWESE, ).

Alongside this, many MSR papers perform their sampling by manually selecting
one or more projects. As seen in (Yang2016ICMSR, ), (Silva2017MSR, ),
(Yu2015MSR, ) the project selection can be based on a list of different
criteria. (Silva2017MSR, ) attempts to separate which projects to analyze by
randomly selecting a project from a list of 20 randomly retrieved Java
repositories hosted on GitHub that contain a Maven project file (pom.xml). On
the other hand, (Yang2016ICMSR, ) specifically selects five large scale
repositories (OpenStack, LibreOffice, AOSP, QT, and Eclipse) that are
integrated with Gerritt and Git. Similarly, there are many ways of performing
the data extraction and often they do not follow a set process. As noted in
(Yang2016ICMSR, ), the process can change even within one’s own workflow since
the process of scraping one repository is not always applicable to another.
This lack of conformity in sampling and data extraction practices between
researchers provides a potential for bias and errors in data retrieval.

Many of the publicly available repositories are hosted on
GitHub222https://github.com (a popular code hosting and version control
platform). As noted above, in order to select a sample of projects for
analysis MSR researchers typically filter by the metadata available on GitHub.
GitHub categorizes these projects based on metadata such as stars (a measure
for users to keep track of repositories they like) or languages used.
Currently, GitHub is the only well known system for sampling of projects and
it is quite often the first choice of MSR researchers. For example,
researchers might sample projects with more than three stars and/or further
refine the search by using the language attribute, the project description,
project creation data, or other metadata provided by GitHub API.
Unfortunately, this process is time consuming, error prone (sometimes metadata
is absent or incorrectly specified), and it is only possible to sample
projects based on the very limited set of attributes provided by GitHub API.
For example, there is no way to retrieve projects that contain a certain
number of source code files from a specific language or to determine all
projects that a specific developer has worked on.

The World of Code (WoC) (Ma2019MSR, ) aims to provide an interface that
simplifies the data retrieval process by allowing mass retrieval of git data
that resides across all open source repositories such as commits, files,
authors, projects, and blobs. WoC allows for rapid cross-data retrieval due to
a set of key-value mappings between data types (e.g. commit to author of the
commit). More information on the initial implementation of this interface can
be found in (Ma2019MSR, ). Such capabilities are particularly suited to
support representative sampling and, more generally, research on software
ecosystems. In this paper we describe how we used WoC to support sampling of
repositories based on specific criteria that are not available via, for
example, GitHub API (e.g. number of commits, number of committers, lines of
code, languages used, age of the repository, library dependencies).
Specifically, we compiled databases to provide these sampling capabilities
based on activities in git repositories or by authors making commits to all
public git repositories.

The compiled datasets are stored in a MongoDB collection to make extraction
and analysis easier. For MSR researchers interested in analyzing a single
project, WoC can be used for data retrieval beyond the scope of the project.
Due to the key-value mappings between Git data types, it is possible to easily
retrieve information about developer activities outside of the project being
analyzed.

##  2\. Creation of the Databases for Sampling

Using the World of Code, a set of publicly available datasets was created to
allow for more exact sampling of git project/author data. To make data
retrieval less challenging for users not accustomed to the WoC system, the
datasets are stored in MongoDB collections labeled as metadata. While these
datasets contain useful information, they are not an exhaustive collection of
all the possible data that can be derived from Git. The World of Code is a
system that stores Git data in an easily accessed form. Alongside this, the
system is not restricted solely to one source code hosting site. Instead, it
saves data from the greater git ecosystem. Both of these metadata datasets
were produced by iterating over the entirety of the projects/authors contained
within the World of Code. Utilizing the base-mappings between datatypes, it
was possible to retrieve and store all the desired data for these collections.
However, considering the size of data contained with WoC, the process of
creating these datasets takes about 24 hours for authors and roughly 48 hours
for projects.

Figure 1. System Layout for resources within the World of Code

###  2.1. The types of MSR analyses that could benefit from better sampling

To determine if the World of Code provides a system unique in its ability to
be used for certain MSR research, we considered a set of usage examples.
Furthermore, we recruited 3 external researchers to perform/provide research
tasks.

In the development process, a potentially large factor for developers when
considering how to market their software is coding language popularity ( e.g.
which language should the product be implemented in, which language should be
provided for others to interface with a component, etc). One way to determine
popularity using the World of Code is by plotting language usage over time. By
assessing the file extensions of each file it is possible to determine which
files are related to which coding languages. Afterwards, using the mapping
from files to commits, the timestamp related to each commit can be used to
determine usage per year.

Another avenue of research the World of Code offers is analyzing developer
ecosystems. When analyzing the developers in git, it is quite troublesome to
retrieve accurate and specific data of developers names. Names or emails of
developers are frequently misspelled, incomplete, or completely missing. This
makes performing any research on developer networks quite difficult. However,
the World of Code provides some options for enhancing accuracy measures when
attempting to disambiguate author identities. To perform any form of
disambiguation algorithm it is necessary to determine common patterns of data
irregularity. Fortunately, the World of Code contains a near complete
collection of author ids (e.g. John Doe<EMAIL_ADDRESS>within Git. This makes
the dataset much more viable for studying and removing such irregularities.
Furthermore, using WoC it is possible to even further disambiguate based on
other factors of similarity. Such similarity measures include time patterns of
commits, writing style within commits, and author id comparison within file
changes.

Another form of research that WoC can help perform is analysis of
sustainability of open source ecosystems. By analyzing the existing projects
within a particular ecosystem, for example Python’s PyPI packages, it is
possible to determine how often these projects achieve ”feature completeness”
(achieved intended usage and requires no further maintenance) versus how often
projects are abandoned. This can be done once again using the file to commit
mapping and by assessing the imports within each file.

Other research areas include analysis of developers across project ecosystems,
file cloning across ecosystems, repository filtering, the popularity and
relationship between NPM packages, and investigating what influences adopting
of software.

###  2.2. A Use Case for project/author sampling

Figure 2. Sampling/Data Retrieval Workflow without the World of Code Figure 3.
Sampling/Data Retrieval Workflow without the World of Code

To further elucidate how Git data retrieval can be tedious, consider a
hypothetical researcher Sam who wants to start repository analysis. Sam wants
to analyze the frequency of changes to coding language files in projects.
Considering that the average number of lines of code in C files is much
greater than in Python files, Sam wants to determine whether C files are
changed more often than Python files. To proceed with this research Sam needs
to have access to all changes made to previous files over a timeframe in order
to be able to determine which (if any) coding language experiences more
frequent changes. Sam’s sampling and data retrieval workflow is illustrated in
Figure 2.

To achieve this goal, Sam wants to sample 20 disparate projects that contain
at least 20 Python files each and a minimum of 5 committing authors. Sam also
wants 20 projects that contains at least 20 C files and at least 5 committing
authors. They also want these files to contain at least 50 lines of code for
the Python files and over 100 lines for the C files. Sam has used GitHub in
the past and believes that it must host many such projects. Thus, they want to
use it when performing the project sampling. They do a quick search and finds
that they can get a list of all currently trending C/Python projects. However,
despite having a list of projects, they have no way to know which of these
projects satisfy all of the requirements without directly assessing each
project.

Sam hopes to avoid such a time consuming process and wants to automate some of
the work. Thus, they do another quick search and realizes that GitHub has a
way to further restrict the resulting repositories based on the project
metadata GitHub collects. Unfortunately, they quickly realize that some of the
restrictions they need have not been implemented. Sam has no way to
specifically restrict the results by the number of coding language files
within each project. Furthermore, there does not seem to be a way to restrict
the files to a number of lines.

Thus, they need to decide how else they would like to restrict the search or
choose to manually search through each project. To make their work more
expedient, they decide to modify the search field slightly by simply choosing
projects that have a large number of stars and fall into either the C or
Python category. This will hopefully satisfy the requirements regarding the
number of files and committing authors. Unfortunately, the workload of
manually looking for 40 projects is still too time consuming. Thus, Sam
decides to use the first 10 of each language found.

After determining the projects Sam would like to analyze, they have to
determine how to go about retrieving the desired data. They can either decide
to manually scrape the data from each project’s webpage or look for a system
in place that will do it for them. They do a quick search and find that there
are publicly available API that will allow them to interface with the GitHub
project data. However, all of the API will take time to learn and it is hard
to determine if the system will have all the desired data.

Thus, Sam must decide whether to learn how to use one of these systems or
implement a temporary system to personally scrape the data. To avoid
potentially retrieving bad data by performing manual scraping, they decide to
learn one of the available systems and attempt to retrieve the information.
Now Sam must determine how to go about assessing the rate of change of each
file. They know that they can look at different commits in Git and determine
if there were changes made. Thus, they choose an API that allows them to
retrieve that information and after a little time and effort they finally
retrieve the desired data.

Had Sam known about the World of Code, they could have saved much time and
effort. Sam’s hypothetical workflow using the World of Code is illustrated in
Figure 3. Using the World of Code they would be able to perform their project
sampling in the exact way they originally desired and could have performed
their analysis/data retrieval while performing the sampling. Since the World
of Code has access to the files, committing authors, and lines of code in each
project, they would have been able to perform all of their work quickly and
within one system. Since the World of Code contains basemappings between data-
types, performing the sampling is a simple matter.

This hypothetical process could have been solved in its entirety using the
World of Code. Had Sam used the World of Code, Sam would have needed to
retrieve all the files associated with a project. Then, to perform a count of
the specific language files contained within the project, they could have
simply assessed the file extensions of each file. Once a project with the
desired number of files had been found, they could have retrieved the number
of committing authors of that project by utilizing the direct mapping between
projects and authors. This basemapping would return a list of authors and
whether there were 5 or more authors in the project would be easily discerned.
Finally, they could determine whether the coding files contain the desired
number of lines by using the mapping between projects and blobs. Blobs in WoC
contain the actual contents of each file. Thus, counting the lines of the
blobs can discern if the project satisfies the set requirements. Furthermore,
once a project had been found that satisfies the requirements, it is possible
to directly perform the analysis. Since the World of Code contains all the
commits associated with each project and every blob is linked to a commit, it
is possible to determine how often one file has been changed. New blobs are
not created unless there were changes made to the file. Thus, counting the
blobs of each of the language files will quickly retrieve the desired
information, and would have solved Sam’s problem. Not only is the workflow
simpler to figure out when using the World of Code, but it also only requires
the user understand one system.

Further, had Sam desired to research individual committing authors within Git,
the workflow without the World of Code would have been quite a bit more
difficult. Currently, there is no known system that allows for sampling of
authors and thus the research would have required personal implementation for
any automation of the process. To make matters even more difficult, many of
the publicly available API lack the requisite data regarding authors to
perform in depth research. This would have made their workflow quite a bit
more difficult and almost certainly would have forced them to manually
implement any scraping of data. However, because the World of Code contains
basemappings for authors similar to projects, performing a sampling of
specific authors based on certain restrictions is still easily performed.
Additionally, the analysis can also be done within WoC when researching
authors.

Considering much of these examples are focused around making researchers’
workflow easier, it is worth noting that much of the above stated work is
possible using the metadata datasets compiled using the World of Code as well.
While it is not possible to determine lines of code using the metadata
datasets, performing the sampling based on number of language files and
authors within the projects is easily accomplished using MongoDBs built-in
restrictions. This would allow researchers to fully avoid having a firm grasp
on Perl, Python, or the Unix Command Line which is necessary to perform the
sampling using the World of Code. Then the later data retrieval/analysis
process could be performed using their preferred API.

###  2.3. Project Metadata Extracted from Code Commits

The project dataset is stored in a MongoDB collection titled
proj_metadata followed by the current version of WoC
(e.g. proj_metadata.Q). The collection stores the total number of authors,
commits, and files associated with the project. It also stores an activity
range for each project based on the Unix timestamp linked to the first and
last commit. Alongside this, the data includes coding language usage based on
the file extensions of each file in the project. Due to the prevalence of
forked projects in Git, if WoC determines a project to be a fork then the
original location of the forked project is included. When the project is
hosted on GitHub and has a stars rating, the collection includes information
on the stars rating of the project.

###  2.4. Author Metadata

Like the project dataset, the author dataset is stored in a MongoDB collection
titled auth_metadata followed by the current version of WoC (e.g
auth_metadata.Q). It includes the total number of commits, blobs, files, and
projects the author has participated in. It includes a time frame the author
was active based on the Unix timestamps of the first and last commit. Lastly,
it includes coding language usage based on file extension.

##  3\. Challenges and Limitations of the World of Code

_World of Code Versioning:_ Since the World of Code contains so much
information, mass-updates of information must be done in increments. Due to
this fact, updating the version of WoC often happens months apart. This is
necessary because updating the basemaps requires computationally intensive
work that takes a non-trivial amount of time. This versioning of WoC means the
data is subject to the latest WoC update. Thus, information on currently
active repositories/authors is also restricted by this update system.
Timeframe analysis based upon the first and last commit time is therefore not
suitable when the data must be truly current.

_Language Inclusion:_ There are restrictions to the languages included in the
metadata datasets. The languages included are Ada, C/C++, COBOL, CSharp,
Erlang, Fml, Fortran, Go, Java, Javascript, JL, Lisp, Lua, Perl, PHP, Python,
R, Ruby, Rust, Scala, SQL, and Swift. When counting the files, languages were
determined using the file extensions on the filenames. If the filename does
not have one of the extensions for these languages it is not counted as a
program file. Alongside this restriction, languages that do not require a
specific file extension will be ignored by the algorithm and counted as a
regular file. To analyze languages not included in this set, researchers may
need to generate a personal dataset that includes the language, or request the
language be included in the next iteration.

_Forks:_ There are limitations to the fork information provided in the
metadata collections. This information is based on an arbitrary clustering
method developed for determining forks. This was done partly using commits
that are associated with many projects. Since there is a timestamp for each
commit, it is theoretically possible to find the earliest commit in a project,
then see what project that timestamp is associated with, and then claim the
current project was forked from that project. However, if such a process is
followed, the total number of ”un-forked” projects becomes much smaller than
is reasonable. This is because many projects have dependencies on widely used
projects. Thus, the clustering method that was used had to make determinations
on whether the project truly is a fork or not.

##  4\. Related Work

Retrieving Git data can be a convoluted process and, as is discussed in
(Yang2016ICMSR, ), the use of domain specific API can be difficult.
(Yang2016ICMSR, ) describes a framework meant to make mining of code review
repositories on Gerrit easier. This framework retrieves code review
information from publicly available repositories and stores it in a more
easily accessed format. Desired information is compiled by scripts that query
the Gerrit API and then parse the returned JSON object into a more easily used
format. The parsed data is then stored in a relational database so that
interested parties can access the data more easily. Similar to the data
compiled by the World of Code, this data is intended for researchers
interested in repository analysis. However, unlike WoC, their collected data
is targeted at Code Review repositories and thus is not applicable to general
repository analysis.

A common practice to expand the research area is to analyze data pulled from
general Git projects. Due to its dominant presence as a source code host,
GitHub tends to be one of the primary targets for data retrieval and analysis.
There is a publicly available API for GitHub that leverages the REST API to
return JSON objects with the requested information. However, as was discussed
in (Gousios2013MSR, ), using the GitHub API can be difficult, is restricted to
specific fields of research, and may lead to biased results if used
incorrectly. The API’s limitations include restrictions on the total number of
requests that can be made per hour and the difficulties of parsing the
results. On top of the restrictions already in place from the API, GitHub also
only provides a subset of the total Git ecosystem. Despite its prevalence,
GitHub only makes up a fraction of the total number of Git repositories. Thus,
the data collected there fails to include many projects.

Also presented in (Gousios2013MSR, ), GHTorrent, a project meant to mirror the
GitHub event timeline and store the raw JSON for retrieval, is another
framework that is often used for repository analysis. The system has been
picked up by many researchers because it removes the restriction on number of
requests per hour and also attempts to provide the results in an already
parsed form. This data is stored in a relational database and a set of MongoDB
collections meant for querying. However, because it is based strictly off the
GitHub ecosystem, it still has an issue with restricting the data to GitHub
alone. It also introduces a set of new issues which are discussed in
(Gousios2013MSR, ).

##  5\. Conclusions

In this paper, we presented the World of Code and the metadata datasets that
were compiled within. Further, we outlined the limitations of WoC and the
metadata in regards to data integrity. This system has potential to become an
excellent sampling resource for researchers interested in MSR and for data
analysis of Git projects and Git authors. The World of Code has potential for
much easier Git data retrieval and could be used to drastically simplify the
MSR research workflow. Furthermore, the system can be used to create similar
metadata datasets in the future to make sampling of data for researchers much
simpler.

## References

  * [1] Georgios Gousios.  The ghtorent dataset and tool suite.  In 2013 10th Working Conference on Mining Software Repositories (MSR), pages 233–236. IEEE, 2013.
  * [2] Dimitris Kolovos, Patrick Neubauer, Konstantinos Barmpis, Nicholas Matragkas, and Richard Paige.  Crossflow: a framework for distributed mining of software repositories.  In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pages 155–159. IEEE, 2019.
  * [3] Yuxing Ma, Chris Bogart, Sadika Amreen, Russell Zaretzki, and Audris Mockus.  World of code: An infrastructure for mining the universe of open source vcs data.  In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pages 143–154. IEEE, 2019.
  * [4] Yasutaka Sakamoto, Shinsuke Matsumoto, and Masahide Nakamura.  Integrating service oriented msr framework and google chart tools for visualizing software evolution.  In 2012 Fourth International Workshop on Empirical Software Engineering in Practice, pages 35–39. IEEE, 2012.
  * [5] Danilo Silva and Marco Tulio Valente.  Refdiff: detecting refactorings in version histories.  In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), pages 269–279. IEEE, 2017.
  * [6] Fabian Trautsch, Steffen Herbold, Philip Makedonski, and Jens Grabowski.  Adressing problems with external validity of repository mining studies through a smart data platform.  In Proceedings of the 13th International Conference on Mining Software Repositories, pages 97–108, 2016.
  * [7] Xin Yang, Raula Gaikovina Kula, Norihiro Yoshida, and Hajimu Iida.  Mining the modern code review repositories: A dataset of people, process and product.  In Proceedings of the 13th International Conference on Mining Software Repositories, pages 460–463, 2016.
  * [8] Yue Yu, Huaimin Wang, Vladimir Filkov, Premkumar Devanbu, and Bogdan Vasilescu.  Wait for it: Determinants of pull request evaluation latency on github.  In 2015 IEEE/ACM 12th working conference on mining software repositories, pages 367–371. IEEE, 2015.


# Deep conditional transformation models for
survival analysis

Gabriele Campanella1,2,*, Lucas Kook3,4,*, Ida Häggström5,6,
Torsten Hothorn3, Thomas J. Fuchs1,2 ††Authors contributed equally.
Corresponding author<EMAIL_ADDRESS>
Preprint. Version October 20, 2022. Licensed under CC-BY.

(1Department of AI and Human Health, Icahn School of Medicine at Mount Sinai,
New York, 10029, USA
2Hasso Plattner Institute for Digital Health at Mount Sinai, New York, 10029,
USA
3Epidemiology, Biostatistics & Prevention Institute, University of Zurich,
CH-8001, Switzerland
4Institute for Data Analysis and Process Design, Zurich University of Applied
Sciences, CH-8400, Switzerland
5Chalmers University of Technology, Department of Electrical Engineering,
41296, Sweden
6Memorial Sloan Kettering Cancer Center, Department of Radiology, New York,
10065, USA
)

###### Abstract

An every increasing number of clinical trials features a time-to-event outcome
and records non-tabular patient data, such as magnetic resonance imaging or
text data in the form of electronic health records. Recently, several neural-
network based solutions have been proposed, some of which are binary
classifiers. Parametric, distribution-free approaches which make full use of
survival time and censoring status have not received much attention. We
present deep conditional transformation models (DCTMs) for survival outcomes
as a unifying approach to parametric and semiparametric survival analysis.
DCTMs allow the specification of non-linear and non-proportional hazards for
both tabular and non-tabular data and extend to all types of censoring and
truncation. On real and semi-synthetic data, we show that DCTMs compete with
state-of-the-art DL approaches to survival analysis.

##  1 Introduction

Arguably one of the most important aspects of health and medical research is
being able to understand and predict patient outcome in order to improve
patient management and ultimately extend their life span or time in remission
(Hosny et al., 2018). Survival analysis is used for these purposes to study
time-to-event information relating to for example death, response to
treatment, adverse treatment effects, disease relapse, and the development of
new disease (Collett, 2015). Traditional approaches, such as Cox Proportional
Hazards (cf. Section 3), relied on tabular features and are not amenable to
analyze high-dimensional non-tabular data such as medical images. With recent
advances in computer vision and deep leaning, there has been increasingly more
interest in performing survival analysis directly from high-dimensional data
in order to automatically learn patterns that stratify patients based on their
outcome without the need for feature engineering.

In this paper we present DCTM, a framework for parametric and semiparametric
survival analysis rooted in statistical modeling. DCTMs allow the
specification of non-linear and non-proportional hazards for both tabular and
non-tabular (image or text) data. We will describe in detail the formalization
of our proposed models and apply them to a real dataset of medical images.
Additionally, we describe how DCTMs can be used as generative models for the
generation of semi-synthetic data. To the best of our knowledge, this paper is
the first to cover deep survival regression from the DCTM point of view.

##  2 Deep transformation models for survival outcomes

In the following, we introduce deep conditional transformation models (DCTMs)
for survival outcomes. We briefly recap survival analysis and conditional
transformation models. Then we describe how to setup, fit, evaluate and sample
from our proposed models. Fig. 1 is an overview of the proposed class of
models.

##### Survival analysis

Survival analysis characterizes the distribution of the positive real-valued
event time $T$ conditional on covariates $X$, usually on the scale of the
survivor function, $S_{T\mid X=x}(t)=1-F_{T\mid X=x}(t)$, where $F_{T\mid
X=x}$ denotes the cumulative distribution function (CDF) of $T\mid X=x$. One
of the most popular choices is the Cox proportional hazards (Cox PH) model
(Cox, 1972)

$\displaystyle F_{T\mid X=x}(t)=1-\exp(-\Lambda(t\mid
x))=1-\exp(-\Lambda_{0}(t)\exp(x^{\top}\beta)),$ (1)

where $\Lambda(t\mid x)$ denotes the positive, monotone increasing cumulative
hazard function. The hazard function is assumed to be decomposable into a
baseline hazard $\Lambda_{0}$, independent of the covariates, and a hazard
ratio $\exp(x^{\top}\beta)$. The hazard ratio models the influence of the
covariates on the survivor function. The hazard function $\lambda(t\mid
x)=\frac{\mathrm{d}}{\mathrm{d}t}\Lambda(t\mid x)$ measures the instantaneous
risk of an event after time $t$ conditional on the covariates $x$ and having
survived until $t$ (Collett, 2015).

The Cox PH model is commonly estimated using the maximum partial likelihood
obtained from profiling out $\Lambda_{0}$ (Cox, 1975). The connection to
transformation models for distributional regression is drawn next.

##### Conditional transformation models

CTMs are parametric distributional regression models of the form (Hothorn et
al., 2014)

$\displaystyle F_{T\mid X=x}(t)=F_{Z}(h(t\mid x)),$ (2)

where the conditional distribution of the response $T\mid X=x$ is decomposed
into an _a priori_ chosen and parameter-free target distribution $F_{Z}$ and a
conditional transformation function $h$, which depends on the input data $x$.
In order for $F_{T}$ to be a valid CDF, $h$ needs to be monotonically
increasing in $t$ (Hothorn et al., 2018).

CTMs can be estimated via maximum likelihood and allow various kinds of
responses and uninformative censoring. The model in (2) suggests a close
connection to the Cox PH model in (1), namely for $F_{Z}(z)=1-\exp(-\exp(z))$
(the minimum extreme value distribution) and $h(t\mid x)=\log\Lambda(t\mid
x)$, the two models coincide.

##### DCTMs for survival analysis

In DCTMs, the transformation function is parameterized via (deep) neural
networks. For instance, let $\phi:\mathcal{X}\to\mathbb{R}^{d}$ denote a
feature extractor, which maps the input $x$ to a feature vector of dimension
$d$. We can then choose different parameterizations for the transformation
function, depending on the desired complexity of the model. For example,

$\displaystyle h(t\mid
x;\phi)=\alpha+\beta\log(t)+\phi(x)^{\top}w,\quad\beta>0,$ (3)

together with $F_{Z}(z)=1-\exp(-\exp(z))$ is a Weibull proportional hazards
model with non-linear log hazard-ratios depending on the input $x$ (for
instance, medical images). However, also non-proportional (time-varying)
hazards can be realized in DCTMs, via

$\displaystyle h(t\mid x;\phi)=\alpha+g(\phi(x)^{\top}w)\log(t),$ (4)

where $g:\mathbb{R}\to\mathbb{R}_{+}$, e.g., the soft-plus function, ensures a
valid CDF. In Section 4.1, we describe the parameterization of the DCTMs used
in this paper. An overview of the proposed method is given in Figure 1.

A parametric version of the Cox PH model can be estimated in the DCTM
framework. Here, the log cumulative baseline hazard function is estimated as a
smooth basis expansion $a(t)^{\top}\vartheta$, instead of using the non-
parametric estimate. A common choice are Bernstein polynomials, which are
easily constrained to be monotonically increasing (see Section 4).

Figure 1: Proposed DCTM model architecture. The inputs to the model consist of
tabular or non-tabular explanatory variables associated with survival data in
terms of exact or right-censored times to event. The explanatory variables can
be mapped to a latent feature space via a feature extractor. In the case of
non-tabular data such as images, the feature extractor will consist of a non
linear mapping such as a convolutional neural network. The extracted features
are then fed to the transformation layer. The network is optimized by
minimizing the negative log-likelihood (NLL). In the figure we show the
transformation function along with the cumulative density function (CDF),
probability density function (PDF) and NLL for an exact event and a right-
censored example.

##### Fitting DCTMs

DCTMs are fitted by optimizing the empirical negative log-likelihood (NLL) via
stochastic gradient descent (SGD). The likelihood contribution of a single
observation $(t,x)$ can be expressed in terms of the general transformation
model

$\displaystyle\mathcal{L}(h;t,x)=\begin{cases}f_{Z}(h(t\mid
x))h^{\prime}(t\mid x)&t\mbox{ exact event},\\\ 1-F_{Z}(h(t\mid x))&t\mbox{
right censored}.\end{cases}$ (5)

However, also interval- and left-censored observations can be handled with the
proposed method (see e.g., Hothorn et al., 2018). For a general choice of the
transformation function $h(t\mid x)$ and error distribution $F=\sigma$, we
have the following likelihood function

$\displaystyle\mathcal{L}(h;t,x)=\begin{cases}\sigma(h(t\mid
x))(1-\sigma(h(t\mid x))h^{\prime}(t\mid x)&t\mbox{ exact event},\\\
1-\sigma(h(t\mid x))&t\mbox{ right censored}.\end{cases}$ (6)

Lastly, the NLL

$\displaystyle\operatorname{NLL}=-\sum_{i=1}^{n_{t}}\log\mathcal{L}(h;t_{i},x_{i}),$
(7)

is minimized via SGD, where $n_{t}$ denotes the training sample size.

##### Evaluating DCTMs

After fitting a DCTM, the conditional survivor function of a test observation
can be computed from the estimated parameters via $\hat{F}_{T\mid
X=x}(t)=F_{Z}(\hat{h}(t\mid x))$. Now, all evaluation metrics can be computed
from (some form of) the predicted conditional distribution.

Evaluating models that predict conditional distribution of survival times are
not straight-forward to evaluate, due to censoring and their probabilistic
nature (Collett, 2015). For instance, mean squared error or median absolute
deviation are insensible for a right-censored observation $((t,+\infty],x)$
and prediction $\hat{t}$, for instance the conditional median survival time.

The c-index (Harrell, 1982)

$\displaystyle
c=\frac{\sum_{i,j}\mathds{1}_{T_{j}<T_{i}}\cdot\mathds{1}_{\eta_{j}>\eta_{i}}\cdot\delta_{j}}{\sum_{i,j}\mathds{1}_{T_{j}<T_{i}}\cdot\delta_{j}},$
(8)

overcomes at least the issue of censoring. However, the c-index does not
encourage faithful probabilistic predictions, because it is not a proper
scoring rule (Blanche et al., 2018).

Proper scoring rules (Gneiting and Raftery, 2005, 2007) are explicitly
designed to evaluate probabilistic predictions and can inherently deal with
censoring. A score is proper, if its average w.r.t. the data-generating
distribution is minimized when predicting exactly this distribution, i.e., if
$T\sim\mathbb{P}_{T\mid X=x}$,

$\displaystyle\mathbb{E}_{\mathbb{P}_{T\mid
X=x}}[S(p;T)]\leq\mathbb{E}_{\mathbb{P}_{T\mid X=x}}[S(q;T)].$ (9)

Here, $\mathbb{P}_{T\mid X=x}$ denotes the data-generating probability
distribution with density $p$ and $q$ is another predicted density. A score is
called _strictly proper_ , when the above inequality holds if and only if
$p=q$.

The c-index does not fulfill (9), whereas the negative log-likelihood

$\displaystyle
S_{\mathrm{NLL}}(\hat{F},t)=-\delta\log\hat{f}(t)-(1-\delta)\log\hat{S}(t)$
(10)

does (Good, 1952). Here, $\delta\in\\{0,1\\}$ denotes an exact ($\delta=1$) or
a censored ($\delta=0$) survival time. In fact, the NLL is the only strictly
proper local scoring rule (up to affine transformations, Bröcker and Smith,
2007). However, since DCTMs are fitted by minimizing the empirical NLL, a
comparison based solely on this score could be deemed unfair, if the competing
method optimizes a different score, say the c-index. For this reason, we also
compute the continuous ranked probability score (see e.g., Avati et al., 2020)

$\displaystyle
S_{\mathrm{CRPS}}(\hat{F},t,\delta)=\int_{0}^{t}\hat{F}^{2}(u)\;\mathrm{d}u+\delta\int_{t}^{\infty}(1-\hat{F}(u))^{2}\;\mathrm{d}u.$
(11)

##### DCTMs as generative models

DCTMs model the entire conditional distribution of $T\mid X=x$. Consequently,
one can sample from the estimated distribution $\hat{F}_{T\mid X=x}(t)$, e.g.,
via inversion sampling (see Figure 2). In more detail, given a real sample and
a fitted DCTM, the sample’s CDF can be generated. Next, random uniform
probability values can be drawn and mapped back to time via the inverse CDF
(Figure 2b). The usefulness of sampling from DCTMs is illustrated in our semi-
synthetic experiments in Section 5.

Figure 2: Generation of semi-synthetic data via inversion sampling for DCTMs.
a) Fitting of real data with three DCTM parametrizations of increasing
complexity. The more flexible models reach a lower NLL. b) Example of a
patient’s fitted CDF and one sampled synthetic time. c) Distribution of 1,000
sampled times for the previous CDF. d) Generated semi-synthetic datasets based
on the three DCTM parametrizations compared to the real data.

##  3 Related work

We summarize the concurrent literature on deep learning for survival analysis
and transformation models.

##### Deep learning for survival analysis

We distinguish Bayesian and frequentist approaches and the loss function
involved in fitting the models.

Several DL approaches to survival analysis are inspired by the Cox PH model.
One of the earliest approaches using (shallow) neural networks for survival
analysis is Liestbl et al. (1994). The authors use the partial likelihood as a
loss function and even describe extensions to piece-wise constant, or non-
proportional hazards and non-linear effects. A direct extension to deep Cox PH
models is “DeepSurv” (Katzman et al., 2018), which optimizes the
$\ell_{2}$-regularized log partial-likelihood. Also, Nagpal et al. (2021)
proposes a mixture of Cox models together with an expectation maximization
(EM) algorithm to optimize the partial likelihood and the evidence lower bound
(ELBO) of a variational auto-encoder for the features.

In contrast to the semi-parametric approaches derived from the Cox PH model, a
fully parametric framework for survival analysis was given in Nagpal et al.
(2020). The authors model the survivor function as a mixture distribution and
fit the model using an ELBO for exact and censored responses. Ranganath et al.
(2016) present a generative (Bayesian) Weibull model for survival times. Some
approaches based on pseudo-values exist (Zhao and Feng, 2020; Rahman et al.,
2020). Fornili et al. (2014) employ piece-wise exponential hazard functions in
a penalized likelihood loss.

“DeepHazard” proposed by Rava and Bradic (2020), uses a squared error loss
based on the underlying counting process, which takes into account censoring.
The model allows for time-varying covariates and non-linear effects, as well
as non-proportional hazards.

A different branch of literature treats survival regression as a
classification problem. Fotso (2018) introduce “neural multi-task logistic
regression” (N-MTLR), which splits the positive real line into $K$ sub-
intervals and fits a softmax last-layer model to the event indicators. The
authors evaluate their model using proper scoring rules, namely the weighted
and integrated Brier score (which is equivalent to the CRPS). Lee et al.
(2019) propose “dynamic DeepHit”, which is based on “DeepHit” (Lee et al.,
2018) and again uses a softmax last-layer activation. (Dynamic) DeepHit is
able to deal with competing risks and non-linear effects.

DL for survival analysis has been shown to be useful in medical applications.
For instance, Lao et al. (2017) use DL together with radiomics features for
prediction of glioblastoma multiforme survival. A comparison between a deep
learning and the classical Cox PH can be found in (Matsuo et al., 2019).
DeepSurv has been applied and compared against the Cox PH model and survival
random forests for oral cancer (Kim et al., 2019) and non-metastatic clear
cell renal cell carcinoma (Byun et al., 2021) survival prediction.

##### Deep conditional transformation models

Our proposed class of models belongs to the family of DCTMs. DCTMs extend the
flexible class of CTMs Hothorn et al. (2014, 2018) with deep neural networks
and SGD to handle non-tabular data, such as images. TMs are distributional
regression models, which capture the entire conditional distribution instead
of a single or few moments thereof (Kneib et al., 2021). More detail on TMs in
given in Section 2.

DCTMs were first introduced by Sick et al. (2021), where the authors used a
series of nested functions to model the transformation function and the
standard normal CDF as the target distribution. Baumann et al. (2021) extended
the work on DCTMs for continuous distributions with exact observations. Kook
et al. (2022) treated ordinal regression from the DCTM point of view. The
authors view the ordinal response as an interval censored version of a latent
logistic distribution. By imposing structural assumptions on the
transformation function, such as additivity, DCTMs retain interpretability of
certain parts of the models, e.g., parameters for a single data modality.
DCTMs were also developed for time series, i.e., serially correlated data
(Rügamer et al., 2021). DCTMs are all fitted by minimizing the empirical NLL
via some form of SGD.

##  4 Experimental setup

In the following, we describe the details of the experimental setup including
parameterization of the DCTMs and the use of ensemble estimates for improving
prediction performance.

###  4.1 Model parametrizations

We employ different parameterizations of the transformation function, namely a
linear shift, a linear scale, and general shift and scale models with a non-
linear function in $t$ using Bernstein polynomials. For all experiments, we
choose the standard logistic CDF $\sigma(z)=(1+\exp(-z))^{-1}$ or the standard
minimum extreme value CDF (also called Gompertz)
$F_{\mathrm{MEV}}(z)=1-\exp(-\exp(z))$ as the target distribution.
Consequently, the transformation function can be interpreted on the log-odds
or log-hazard scale, respectively.

For the linear shift parameterization, the transformation function is
parameterized as

$\begin{split}h(t\mid x;\phi)=a+b\log t+\phi(x)^{\top}w,\mbox{ where}\\\
\phi:\mathcal{X}\to\mathbb{R}^{P},a\in\mathbb{R},b\in\mathbb{R},w\in\mathbb{R}^{P},\end{split}$
(12)

where $h$ denotes the transformation function, with a linear basis in $t$
parameterized via $a$ and $b$, and a feature extractor $\phi$ representing a
neural network with linear last-layer activation and weights $w$. Here, the
extracted features enter linearly on the scale of the transformation function.
In turn, the conditional distribution function changes only in terms of
location, but not in terms of scale or any higher moments. That is, $F_{T\mid
X=x}$ is restricted to the same family of distributions as the target
distribution $F$. Hence, this model is not distribution-free. For
$F_{Z}=F_{\rm MEV}$, the linear shift model is equivalent to a Weibull model.
Similarly, $F_{Z}=\operatorname{expit}$, yields a log-logistic model.

For the linear scale model, we have

$\begin{split}h(t\mid
x;\phi)=a+\operatorname{softplus}(\phi(x)^{\top}w)\cdot\log t,\mbox{ where}\\\
\phi:\mathcal{X}\to\mathbb{R}^{P},a\in\mathbb{R},w\in\mathbb{R}^{P},\end{split}$
(13)

in which the neural network models the scale of $F_{T\mid X=x}$, which allows
for linear non-proportional hazards. Still, the model is not distribution-
free, the resulting distributions are again Weibull or log-logistic.

The general shift transformation model parameterizes the transformation
function using Bernstein polynomials,

$\begin{split}h(t\mid x;\phi)=b(\log t)^{\top}\vartheta+\phi(x)^{\top}w,\mbox{
where},\\\
\phi:\mathcal{X}\to\mathbb{R}^{P},b:[0,1]\to\mathbb{R}^{K+1},\end{split}$ (14)

which requires $\vartheta$ to be increasing, i.e.,
$\vartheta_{k+1}>\vartheta_{k}$, $k=1,\dots,K$, for all $x$ to ensure
monotonicity of $h(t\mid x;\phi)$ in $t$ (Hothorn et al., 2014). Monotonicity
can be ensured by transforming

$\displaystyle\vartheta=g(\gamma)=\left(\gamma_{1},\gamma_{1}+\operatorname{softplus}(\gamma_{2}),\dots,\gamma_{1}+\sum_{k=2}^{K+1}\operatorname{softplus}(\gamma_{k})\right).$

The general shift transformation model allows a flexible baseline hazard
function. The transformation function influences all higher moments of
$F_{T\mid X=x}$ and the model is distribution-free. Thus, the general shift
model can be viewed as a parametric version of the Cox proportional hazards
model.

The shift-scale transformation model,

$\begin{split}h(t\mid
x;\phi)=\operatorname{softplus}(\phi(x)^{\top}\beta)\cdot b(\log
t)^{\top}\vartheta+\phi(x)^{\top}w,\mbox{ where}\\\
\phi:\mathcal{X}\to\mathbb{R}^{P},b:[0,1]\to\mathbb{R}^{K+1},\end{split}$ (15)

allows for non-proportional hazards, explicitly modelling the scale of
$F_{T\mid X=x}$. The model is distribution-free (Siegfried et al., 2022).

Allowing the parameters of the Bernstein polynomials to fully flexibly depend
on $x$, we arrive at the most flexible transformation model

$\begin{split}h(t\mid x;\vartheta)=b(\log t)^{\top}\vartheta(x),\mbox{
where}\\\
\phi:\mathcal{X}\to\mathbb{R}^{P},b:[0,1]\to\mathbb{R}^{K+1}\end{split}$ (16)

It is important to note that for all of the above $u=\log t$ is scaled to
$\tilde{u}=\frac{u-a}{b-a}$ where $a,b$ are chosen appropriately, e.g., min
and max.

###  4.2 Ensemble predictions

The linear shift and linear scale parameterizations induce strong
distributional assumptions on $T\mid X=x$. Nevertheless, these distributional
assumptions can help prevent overfitting and lead to more stable predictions.
Using the Bernstein polynomial basis loosens assumptions, but is prone to
overfitting. However, deep ensembles have been shown to improve prediction
performance in both cases (Lakshminarayanan et al., 2017).

Let $\hat{F}_{1},\dots,\hat{F}_{M}$ be $M$ estimates of the conditional
distribution of $T\mid X=x$, e.g., obtained by optimizing $M$ instances of the
same DCTM via SGD and early stopping on different bootstrapped data samples.
The ensemble estimate is the point-wise average of those $M$ estimates, and
denoted by $\bar{F}_{M}$,

$\displaystyle\bar{F}_{M}=M^{-1}\sum_{m=1}^{M}\hat{F}_{m}.$ (17)

The ensemble distribution is then used for evaluation. When having access to
$B>M$ estimates and selecting $M$ based on their validation loss, we refer to
$\bar{F}_{M}$ as the top-$M$ ensemble estimate.

###  4.3 Data

####  4.3.1 Real data

The real data used in this study comprised 959 stage 4 lung cancer patients
from Memorial Sloan Kettering Cancer Center (MSKCC) diagnosed from Oct 2010
through Mar 2018. The data used included pre-treatment PET/CT images of the
subjects, overall survival time and censoring information. 768 of the patients
(80%) experienced a terminal event, while 191 (20%), were right-censored. The
patients comprise a real-world, clinical grade cohort collected over many
years on a wide selection of scanners, with cases reviewed by many different
clinicians, and patients prescribed varying treatment regimens. The
retrospective data use was approved by the local ethics committee at MSKCC,
and informed consent was waived since the study was deemed minimum risk.

####  4.3.2 Semi-synthetic data

The DCTM architecture was used to model the conditional time to event
distribution of the real data described above. Three different DCTM
parametrizations were used of increasing complexity: linear shift, linear
scale, and fully flexible Bernstein. The entire real data dataset was used to
train each DCTM model for a predetermined number of epochs. The fitted models
were then used to generate new semi-synthetic time-to-event data. If a time
was predicted beyond the real data’s range, it was right-censored to the
maximum value. For each patient we sampled 10 new data points, resulting in
datasets 10 times larger than the original (see Figure 2d).

###  4.4 Computational details

All experiments were performed on the high-performance computing (HPC) cluster
at MSKCC. Training scripts were written in python using PyTorch (Paszke et
al., 2017) for model definition and optimization. Model optimization was
performed with SGD and training learning rates were chosen via a
hyperparameter grid search on a single data training/validation partition. For
the DCTM models, different learning rates were used for the CNN feature
extractor and the DCTM survival layer. The full list of learning rates used
for all experiments is presented in 1.

Table 1: List of learning rates used to train models for all experiments in this study. Parametrization | CDF | lr CNN | lr DCTM
---|---|---|---
Deepsurv |  | 0.0005 |
Baseline | Sigmoid |  | 0.1
Linear shift | Sigmoid | 0.001 | 0.01
Linear scale | Sigmoid | 0.001 | 0.1
Bernstein shift | Sigmoid | 0.001 | 0.1
Bernstein shift/scale | Sigmoid | 0.001 | 0.01
Bernstein | Sigmoid | 0.001 | 0.1
Baseline | Gompertz |  | 0.01
Linear shift | Gompertz | 0.001 | 0.01
Linear scale | Gompertz | 0.001 | 0.01
Bernstein shift | Gompertz | 0.001 | 0.01
Bernstein shift/scale | Gompertz | 0.001 | 0.01
Bernstein | Gompertz | 0.001 | 0.1

##  5 Results

###  5.1 Real Data

Experiments were carried out to test the performance of DCTM models of
increasing complexity. As a baseline a Bernstein non conditional model, where
the same output is associated to every sample, was used. DCTM results were
also compared with DeepSurv. Since DeepSurv does not compute the full
likelihood, only the c-index can be compared in this case. Figure 3a reports
the distributions of validation NLL for each experimental condition in order
of complexity, starting with the baseline at the top and finishing with the
fully flexible Bernstein model at the bottom. It can be observed that, for
both CDF functions, the NLL decreases with model complexity, as expected. In
addition, it is apparent that the Gompertz experiments resulted in wider NLL
distributions, highlighting the underlying training instability we observed
for this subset of experiments. Figure 3b reports the c-index results. The
baseline results are by definition 0.5, since all patients are assigned the
same output time-to-event. It is interesting to note that while the NLL
decreased with model complexity, a slight trend of decreasing c-index can be
observed. Compared to DeepSurv, the DCTM models perform on par or better,
except in one condition. Finally, Figure 3c reports the results in terms of
CRPS. In concordance with the NLL results, a slight trend of decreasing CRPS
with model complexity can be noted.

Figure 3: Experimental results for the real data. The performance of different
parametrizations of DCTM are shown compared to DeepSurv where possible. For
DCTM, the sigmoid and Gompertz functions were tested. a) Distribution of
validation NLL. b) Distribution of validation c-index. c) Distribution of
validation CRPS.

###  5.2 Simulated Data

To analyze the behavior of DCTMs when the interaction between inputs and
survival comes from different distributions, three semi-synthetic datasets of
increasing complexity were generated as described earlier. In Figure 4 are
shown the results of the experiments performed on the semi-synthetic data.
Figure 4a shows the distribution of validation resuts in terms of NLL for the
three datasets fitted with various DCTM parametrizations. In all the
experiments, the sigmoid function was used as CDF. While it is expected that
data of lower complexity would be easier to fit, the opposite effect can be
observed, where the linear shift generated data resulted in a higher NLL,
while the flexible bernstein generated data resulted in lower NLLs.
Additionally, unlike our results on the real data, the simplest DCTM
parametrization resulted in the lowest NLL in each of the three datasets.
Figures 4b,c show similar trends in terms of c-index and CRPS respectively. In
figure 4b it can be noted that DCTM achieves similar performance as DeepSurv
on the three datasets.

Figure 4: Results of the experiments performed on the semi-synthetic data. For
each of the three datasets, the performance of different parametrizations of
DCTM are shown compared to DeepSurv where possible. a) Distribution of
validation NLL. b) Distribution of validation c-index. c) Distribution of
validation CRPS.

##  6 Discussion

In this work we have introduced DCTMs as a novel framework for parametric and
semiparametric survival analysis that can leverage the power of deep learning
to learn patient stratification strategies directly from high-dimensional non-
tabular data. Techniques such as this are necessary in the medical domain to
advance the understanding of disease generation and progression. Modelling
time-to-event outcomes directly from non-tabular medical data such as
radiology images, as presented here, has the potential to shed light in the
biological processes underlying complex diseases such as cancer, and change
how patients are treated ultimately improving patient outcomes. Extensions to
time-varying covariates and competing risks exist for transformation models
with tabular data (Fokas et al., 2017) and can be carried over to DCTMs, which
we leave for future work.

## Acknowledgments

The authors are grateful for the generous computational support given by the
Warren Alpert foundation, and the project management support of Christina M.
Virgo. GC, IH, and TF were supported in part through the NIH/NCI Cancer Center
Support Grant (grant number P30 CA008748). The research of LK was supported by
Novartis Research Foundation (FreeNovation 2019) and by the Swiss National
Science Foundation (grant no. S-86013-01-01 and S-42344-04-01). TH was
supported by the Swiss National Science Foundation (SNF) under the project “A
Lego System for Transformation Inference” (grant no. 200021_184603). TF is a
founder, equity owner, and Chief Scientific Officer of Paige.AI.

##  Appendix A Notation

$T\mid X=x$ denotes a positive real-valued random variable conditional on
covariates $X\in\mathcal{X}$. By $\phi_{\theta}:\mathcal{X}\to\mathbb{R}^{d}$,
we denote a feature extractor, e.g., a convolutional neural network. We write
$\phi$ and suppress the dependency of $\phi_{\theta}$ on network parameters
$\theta$. $F_{T\mid X=x}$ denotes the conditional cumulative distribution
function of $T\mid X=x$, and $F_{Z}$ the target distribution in a
transformation model. $F_{T\mid X=x}(t)=F(h(t\mid x))$ then denotes a general
transformation model with transformation function $h$. For $F_{Z}$ we commonly
chose the standard logistic distribution, $\sigma(z)=(1-\exp(-z))^{-1}$, or
the standard minimum extreme value distribution
$F_{\text{MEV}}(z)=1-\exp(-\exp(z))$. For hazard functions we reserve
$\lambda(t\mid x)$, and $\Lambda(t\mid x)=\int_{0}^{t}\lambda(u\mid
x)~{}\mathrm{d}u$ for cumulative hazard functions.

We denote $n$ observations by $\\{(t_{i},x_{i},\delta_{i})\\}_{i=1}^{n}$,
where $t_{i}$ are i.i.d. realizations of $T\mid X=x_{i}$, with event indicator
$\delta_{i}\in\\{0,1\\}$, where 0 represents a right-censored and 1 an exact
event time. The likelihood function is denoted by $\mathcal{L}(h;t,x,\delta)$,
and the log-liklihood by $\ell(h;t,x,\delta)$. Depending on the
parameterization of the transformation $h$, we substitute the parameters of
$h$ as the argument of the (log-)likelihood function, e.g.,
$\ell(a,b,w,\phi;t,x,\delta)$ for $h(t\mid x)=a+bt+\phi(x)^{\top}w$.

## References

  * Avati et al. [2020] Anand Avati, Tony Duan, Sharon Zhou, Kenneth Jung, Nigam H. Shah, and Andrew Y. Ng.  Countdown regression: Sharp and calibrated survival predictions.  In Ryan P. Adams and Vibhav Gogate, editors, _Proceedings of The 35th Uncertainty in Artificial Intelligence Conference_ , volume 115 of _Proceedings of Machine Learning Research_ , pages 145–155. PMLR, 2020.  URL https://proceedings.mlr.press/v115/avati20a.html.
  * Baumann et al. [2021] Philipp F. M. Baumann, Torsten Hothorn, and David Rügamer.  Deep conditional transformation models.  In _Machine Learning and Knowledge Discovery in Databases. Research Track_ , pages 3–18. Springer-Verlag, 2021.  doi: 10.1007/978-3-030-86523-8_1.
  * Blanche et al. [2018] Paul Blanche, Michael W Kattan, and Thomas A Gerds.  The $c$-index is not proper for the evaluation of $t$-year predicted risks.  _Biostatistics_ , 20(2):347–357, 2018.  doi: 10.1093/biostatistics/kxy006.
  * Bröcker and Smith [2007] Jochen Bröcker and Leonard A Smith.  Scoring probabilistic forecasts: The importance of being proper.  _Weather and Forecasting_ , 22(2):382–388, 2007\.  doi: 10.1175/waf966.1.
  * Byun et al. [2021] Seok-Soo Byun, Tak Sung Heo, Jeong Myeong Choi, Yeong Seok Jeong, Yu Seop Kim, Won Ki Lee, and Chulho Kim.  Deep learning based prediction of prognosis in nonmetastatic clear cell renal cell carcinoma.  _Scientific Reports_ , 11(1), 2021.  doi: 10.1038/s41598-020-80262-9.
  * Collett [2015] David Collett.  _Modelling Survival Data in Medical Research_.  CRC press, 2015.  doi: 10.1201/b18041.
  * Cox [1972] D. R. Cox.  Regression Models and Life-Tables.  _Journal of the Royal Statistical Society B_ , 34(2):187–202, 1972.  doi: 10.1111/j.2517-6161.1972.tb00899.x.
  * Cox [1975] D. R. Cox.  Partial Likelihood.  _Biometrika_ , 62(2):269–276, 1975.  doi: 10.1093/biomet/62.2.269.
  * Fokas et al. [2017] Emmanouil Fokas, Philipp Ströbel, Rainer Fietkau, Michael Ghadimi, Torsten Liersch, Gerhard G Grabenbauer, Arndt Hartmann, Marco Kaufmann, Rolf Sauer, Ullrich Graeven, Hans Hoffmanns, Hans-Rudolf Raab, Torsten Hothorn, Christian Wittekind, Claus Rödel, and on behalf of the German Rectal Cancer Study Group.  Tumor Regression Grading After Preoperative Chemoradiotherapy as a Prognostic Factor and Individual-Level Surrogate for Disease-Free Survival in Rectal Cancer.  _JNCI: Journal of the National Cancer Institute_ , 109(12), 2017.  doi: 10.1093/jnci/djx095.
  * Fornili et al. [2014] Marco Fornili, Federico Ambrogi, Patrizia Boracchi, and Elia Biganzoli.  Piecewise Exponential Artificial Neural Networks (PEANN) for Modeling Hazard Function with Right Censored Data.  _Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , 8452 LNBI:125–136, 2014.  doi: 10.1007/978-3-319-09042-9_9.
  * Fotso [2018] Stephane Fotso.  Deep Neural Networks for Survival Analysis Based on a Multi-Task Framework.  _arXiv Preprint_ , 2018.  doi: 10.1109/ijcnn52387.2021.9533314.
  * Gneiting and Raftery [2005] Tilmann Gneiting and Adrian E Raftery.  Weather forecasting with ensemble methods.  _Science_ , 310(5746):248–249, 2005.  doi: 10.1126/science.1115255.
  * Gneiting and Raftery [2007] Tilmann Gneiting and Adrian E Raftery.  Strictly proper scoring rules, prediction, and estimation.  _Journal of the American Statistical Association_ , 102(477):359–378, 2007.  doi: 10.1198/016214506000001437.
  * Good [1952] I. J. Good.  Rational decisions.  _Journal of the Royal Statistical Society. Series B (Methodological)_ , 14(1):107–114, 1952.  doi: 10.1111/j.2517-6161.1952.tb00104.x.
  * Harrell [1982] Frank E. Harrell.  Evaluating the yield of medical tests.  _JAMA: The Journal of the American Medical Association_ , 247(18):2543, 1982.  doi: 10.1001/jama.1982.03320430047030.
  * Hosny et al. [2018] Ahmed Hosny, Chintan Parmar, Thibaud P. Coroller, Patrick Grossmann, Roman Zeleznik, Avnish Kumar, Johan Bussink, Robert J. Gillies, Raymond H. Mak, and Hugo J. W. L. Aerts.  Deep learning for lung cancer prognostication: A retrospective multi-cohort radiomics study.  _PLOS Medicine_ , 15(11):e1002711, 2018.  doi: 10.1371/journal.pmed.1002711.
  * Hothorn et al. [2014] Torsten Hothorn, Thomas Kneib, and Peter Bühlmann.  Conditional Transformation Models.  _Journal of the Royal Statistical Society. Series B: Statistical Methodology_ , 76(1):3–27, 2014.  doi: 10.1111/rssb.12017.
  * Hothorn et al. [2018] Torsten Hothorn, Lisa Möst, and Peter Bühlmann.  Most Likely Transformations.  _Scandinavian Journal of Statistics_ , 45(1):110–134, 2018.  doi: 10.1111/sjos.12291.
  * Katzman et al. [2018] Jared L. Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger.  DeepSurv: Personalized treatment recommender system using a Cox proportional hazards deep neural network.  _BMC Medical Research Methodology_ , 18(1), 2018.  doi: 10.1186/s12874-018-0482-1.
  * Kim et al. [2019] Dong Wook Kim, Sanghoon Lee, Sunmo Kwon, Woong Nam, In-Ho Cha, and Hyung Jun Kim.  Deep learning-based survival prediction of oral cancer patients.  _Scientific Reports_ , 9(1), 2019.  doi: 10.1038/s41598-019-43372-7.
  * Kneib et al. [2021] Thomas Kneib, Alexander Silbersdorff, and Benjamin Säfken.  Rage against the mean – a review of distributional regression approaches.  _Econometrics and Statistics_ , 2021.  ISSN 2452-3062.  doi: 10.1016/j.ecosta.2021.07.006.
  * Kook et al. [2022] Lucas Kook, Lisa Herzog, Torsten Hothorn, Oliver Dürr, and Beate Sick.  Deep and interpretable regression models for ordinal outcomes.  _Pattern Recognition_ , 122:108263, 2022.  doi: 10.1016/j.patcog.2021.108263.
  * Lakshminarayanan et al. [2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf.
  * Lao et al. [2017] Jiangwei Lao, Yinsheng Chen, Zhi Cheng Li, Qihua Li, Ji Zhang, Jing Liu, and Guangtao Zhai.  A Deep Learning-Based Radiomics Model for Prediction of Survival in Glioblastoma Multiforme.  _Scientific Reports_ , 7(1):1–8, 2017.  doi: 10.1038/s41598-017-10649-8.
  * Lee et al. [2018] Changhee Lee, William Zame, Jinsung Yoon, and Mihaela van der Schaar.  Deephit: A deep learning approach to survival analysis with competing risks.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1), 2018.  doi: 10.1609/aaai.v32i1.11842.
  * Lee et al. [2019] Changhee Lee, Jinsung Yoon, Mihaela van der Schaar, and Mihaela Van Der Schaar.  Dynamic-Deephit: A Deep Learning Approach for Dynamic Survival Analysis with Competing Risks Based on Longitudinal Data.  _IEEE Transactions on Biomedical Engineering_ , 67(1):122–133, 2019.  doi: 10.1109/tbme.2019.2909027.
  * Liestbl et al. [1994] Knut Liestbl, Per Kragh Andersen, and Ulrich Andersen.  Survival Analysis and Neural Nets.  _Statistics in Medicine_ , 13(12):1189–1200, 1994\.  doi: 10.1002/sim.4780131202.
  * Matsuo et al. [2019] Koji Matsuo, Sanjay Purushotham, Bo Jiang, Rachel S. Mandelbaum, Tsuyoshi Takiuchi, Yan Liu, and Lynda D. Roman.  Survival Outcome Prediction in Cervical Cancer: Cox Models vs Deep-Learning Model.  _American Journal of Obstetrics and Gynecology_ , 220(4):381.e1–381.e14, 2019.  doi: 10.1016/j.ajog.2018.12.030.
  * Nagpal et al. [2020] Chirag Nagpal, Xinyu Li, and Artur Dubrawski.  Deep Survival Machines: Fully Parametric Survival Regression and Representation Learning for Censored Data with Competing Risks.  _arXiv Preprint_ , 2020.  doi: 10.1109/jbhi.2021.3052441.
  * Nagpal et al. [2021] Chirag Nagpal, Steve Yadlowsky, Negar Rostamzadeh, and Katherine Heller.  Deep Cox Mixtures for Survival Regression.  _arXiv Preprint_ , 2021.  doi: 10.7717/peerj.9052/table-6.
  * Paszke et al. [2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in PyTorch.  In _Proceedings of the 31st Conference on Neural Information Processing Systems_ , 2017.
  * Rahman et al. [2020] Md Mahamudur Rahman, Shinya Matsuzaki, Koji Matsuo, and Sanjay Purushotham.  Deeppseudo : A deep learning approach based on pseudo values for competing risk analysis, 2020.  URL https://par.nsf.gov/biblio/10218876.
  * Ranganath et al. [2016] Rajesh Ranganath, Adler Perotte, Noémie Elhadad, and David Blei.  Deep survival analysis.  In Finale Doshi-Velez, Jim Fackler, David Kale, Byron Wallace, and Jenna Wiens, editors, _Proceedings of the 1st Machine Learning for Healthcare Conference_ , volume 56 of _Proceedings of Machine Learning Research_ , pages 101–114, Northeastern University, Boston, MA, USA, 2016. PMLR.  URL https://proceedings.mlr.press/v56/Ranganath16.html.
  * Rava and Bradic [2020] Denise Rava and Jelena Bradic.  DeepHazard: Neural Network for Time-Varying Risks.  _arXiv Preprint_ , 2020.  doi: 10.1109/icmlc.2007.4370761.
  * Rügamer et al. [2021] David Rügamer, Philipp FM Baumann, Thomas Kneib, and Torsten Hothorn.  Transforming autoregression: Interpretable and expressive time series forecast.  _arXiv Preprint_ , 2021.  doi: 10.1007/978-3-030-86523-8_1.
  * Sick et al. [2021] Beate Sick, Torsten Hathorn, and Oliver Durr.  Deep transformation models: Tackling complex regression problems with neural network based transformation models.  In _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021.  doi: 10.1109/icpr48806.2021.9413177.
  * Siegfried et al. [2022] Sandra Siegfried, Lucas Kook, and Torsten Hothorn.  Distribution-free location-scale regression.  _arXiv Preprint arXiv:2208.05302_ , 2022.  doi: 10.48550/arxiv.2208.05302.
  * Zhao and Feng [2020] Lili Zhao and Dai Feng.  Deep neural networks for survival analysis using pseudo values.  _IEEE Journal of Biomedical and Health Informatics_ , 24(11):3308–3314, 2020.  doi: 10.1109/jbhi.2020.2980204.