Visual Acuity Prediction on Real-Life Patient Data Using a Machine Learning Based Multistage System

1,⁎]Tobias Schlosser
1]Frederik Beuth
1]Trixy Meyer
1]Arunodhayan Sampath Kumar
2]Gabriel Stolze
2]Olga Furashova
2]Katrin Engelmann
1]Danny Kowerko
[1]
Junior Professorship of Media Computing,
Chemnitz University of Technology,
09107 Chemnitz, Germany

[2]
Department of Ophthalmology,
Klinikum Chemnitz gGmbH,
09116 Chemnitz, Germany

In ophthalmology, intravitreal operative medication therapy (IVOM) is widespread treatment for diseases such as the age-related macular degeneration (AMD), the diabetic macular edema (DME), as well as the retinal vein occlusion (RVO). However, in real-world settings, patients often suffer from loss of vision on time scales of years despite therapy, whereas the prediction of the visual acuity (VA) and the earliest possible detection of deterioration under real-life conditions is challenging due to heterogeneous and incomplete data. In this contribution, we present a workflow for the development of a research-compatible data corpus fusing different IT systems of the department of ophthalmology of a German maximum care hospital. The extensive data corpus allows predictive statements of the expected progression of a patient and his or her VA in each of the three diseases. Within our proposed multistage system, we classify the VA progression into the three groups of therapy “winners”, “stabilizers”, and “losers” (WSL scheme). Our OCT biomarker classification using an ensemble of deep neural networks results in a classification accuracy (F1-score) of over $98$ %, enabling us to complete incomplete OCT documentations while allowing us to exploit them for a more precise VA modelling process. Our VA prediction requires at least four VA examinations and optionally OCT biomarkers from the same time period to predict the VA progression within a forecasted time frame. While achieving a prediction accuracy of up to $69$ % (macro average F1-score) when considering all three WSL-based progression groups, this corresponds to an improvement by $11$ % in comparison to our ophthalmic expertise ($58$ %).

§ INTRODUCTION AND MOTIVATION

High-resolution imaging of the central retina utilizing optical coherence tomography (OCT) plays a key role in the diagnosis and monitoring of the most common macular diseases such as age-related macular degeneration (AMD), diabetic macular edema (DME), and retinal vein occlusion (RVO) <cit.>. Furthermore, detailed analysis of different biomarkers on OCT scans is now the basis for treatment decisions as several biological markers provide not only information on diagnosis of these particular eye conditions, but also play an important role in predicting the treatment response. With the increasing amount of available data for therapeutic strategies, the identification of biomarkers with predictive values, as well as different medication options such as intravitreal operative medication therapies (IVOM), it is challenging for ophthalmologists to individualize the therapy for each patient. Artificial intelligence (AI) based algorithms should in the future help to find optimal individual therapeutic strategies for each patient. Real-world studies have already demonstrated how important the early treatment begin and the compliance with the therapy are <cit.>.

The study group of Gerding et al. was one of the first ones to classify patients with AMD into three treatment groups based on visual acuity (VA) and central retinal thickness: therapy “winners”, “stabilizers”, and “losers” (WSL scheme) <cit.>. This interdisciplinary cooperation between IT specialists and ophthalmologists aims at analyzing the patients' data according to the WSL scheme while identifying predictive values for several OCT biomarkers. The results should help ophthalmologists to better define their therapy strategy for each patient in everyday practice. Moreover, Schmidt-Erfurth et al. recently reported the potential of AI-based approaches for targeted optimization of diagnosis and therapy for eye diseases <cit.>. In their contributions, they furthermore describe the impact of deep learning (DL) for the prediction of patient progressions in the earlier stages of AMD utilizing OCT biomarkers <cit.>.

With medical doctors for medical doctors as well as researchers and users developed patient progression visualization and modelling dashboard with general patient information and VA, OCT biomarkers, diagnoses, and medications. Besides the shown patient-wise information, ophthalmic doctors are furthermore enabled to on-site annotate their predictions regarding the expected VA progression.

§.§ Related work

In the following sections, we discuss ophthalmic research on data mining from clinical IT systems (section <ref>), text and OCT image processing (section <ref>), and the use of the processed data for patient progression modelling (section <ref>).

§.§.§ Availability of ophthalmic data in research

In recent years, more and more ophthalmic data sets are being released <cit.>. A recent review article identified 94 open access data sets containing 507 724 images and 125 videos from 122 364 patients with diabetic retinopathy, glaucoma, and AMD being disproportionately over-represented in comparison to other eye diseases. However, the documentation of demographic characteristics such as age, sex, and ethnicity was reported to be of poor data quality even at the aggregate level <cit.>.

In 2017, we proposed a prototypical workflow to aggregate ophthalmic text and image data of all patients from the Department of Ophthalmology of the maximum care hospital Klinikum Chemnitz gGmbH in Chemnitz, Germany. We combined data mining and basic natural language processing utilizing the interface of the clinic's practice management software Turbomed (CompuGroup Medical) and extracted a set of preliminary diagnostic patient data in order to determine the ratio of patients with VA improvement, stabilization, and deterioration <cit.>.

§.§.§ Ophthalmic text and OCT image processing

While widely being used for general text processing, natural language processing systems have recently been demonstrated to robustly extract medication information from clinical notes to study VA, intraocular pressure, and medication outcomes of cataract and glaucoma surgeries <cit.> to develop predictive models for low-vision prognosis <cit.> as well as to predict glaucoma progressions <cit.>.

Following De Fauw et al., especially machine learning (ML) and deep learning based approaches enable a more precise progression modelling as recent advances prove their applicability and capabilities within the domain <cit.>. Moreover, recognition of OCT biomarkers <cit.> allows further VA and treatment based medical forecasts, including ensemble-based solutions to OCT segmentation <cit.> that enable the completion of incomplete OCT documentations. However, automated thresholding algorithms can yield an improved reproducibility of OCT parameters while allowing a more sensitive diagnosis of pathologies when, e.g., discriminating between healthy and impaired macular perfusion <cit.>.

§.§.§ Patient progressing modelling

Schmidt-Erfurth et al. investigate the influence of hyperreflective foci as OCT biomarkers during the progression of geographic atrophy within the context of AMD <cit.>. While utilizing deep neural networks (DNN) for OCT segmentation <cit.>, they identify and localize occurrences given a data set of 87 eyes from 54 different patients. Following Schmidt-Erfurth et al.'s contribution, Waldstein et al. propose a further developed system while evaluating it with 8 529 OCT volumes of 512 different patients <cit.>. Moreover, time-dependent sequence modelling using recurrent neural networks (RNN) <cit.> constitutes a promising approach to treatment prediction <cit.>. At this point it is also noted that approaches to patient progression modelling exist that explore conventional models. These include mathematical models, e.g., to determine the effect of the anti-vascular endothelial growth factor on VA via medication concentration and tolerance <cit.>, as well as regression-based approaches, e.g., to predict VA in diabetic retinopathy via the foveal avascular zone area <cit.>.

§.§ Contribution of this work

In this contribution, we present an IT system architecture that aggregates patient information for more than 49 000 patients from different categories of various multimedia data in the form of text and images within multiple heterogeneous ophthalmic data resources.

The resulting data corpus enables predictive statements to be made about the expected progression of a patient's VA after at least four VA examinations in each of the three diseases AMD, DME, and RVO. A more fine-grained analysis is conducted to reveal the influence of medical co-existing factors such as other diseases in this real-world setting.

Within our proposed multistage system, an ensemble of deep neural networks allows the completion of incomplete or missing OCT documentations. In order to conduct a patient progression modelling, we define a fundamental use case for the prediction of the VA by aggregating different patient information as input of the subsequent VA forecast after a given time period. We present an evaluation formalism and discuss the resulting predictions in comparison to those of a human annotator as our ophthalmic expertise (an experienced ophthalmologist).

In order to enable ophthalmic doctors to annotate their predictions regarding the patient-wise expected VA progression, our proposed patient progression visualization and modelling dashboard gives an overview over our aggregated data with patient-wise information of, e.g., general patient information and VA, OCT biomarkers, diagnosis, and medication information (Fig. <ref>).

d/.style=draw, diamond,
e/.style=draw, ellipse,
p/.style=draw, trapezium, trapezium left angle=75, trapezium right angle=105,
r/.style=draw, rectangle, minimum height=0.55cm]
[r, minimum width=3cm] (l1n1) at (-4, 0) EHR system;
[r, minimum width=3cm] (l1n2) at ( 0, 0) CIS system;
[r, minimum width=3cm] (l1n3) at ( 4, 0) OCT system;

[r, minimum width=3cm, align=center] (l2n1) at (-4, -1.5) BDT-based data
(*.BDT interface);
[r, minimum width=3cm, align=center] (l2n2) at ( 0, -1.5) IVOM-based data
(*.CSV interface);
[r, minimum width=3cm, align=center] (l2n3) at ( 4, -1.5) E2E-based data
(*.E2E interface);

[r, minimum width=4cm] (l3n1) at (0, -3) Patient-based data merging;
[r, minimum width=4cm] (l3n2) at (6, -3) Dashboard visualization;
[r, minimum width=4cm, draw=none] (l3n0) at (-6, -3) ;

[r, minimum width=3cm] (l4n1) at (-4, -4.5) Anamnesis;
[r, minimum width=3cm] (l5n1) at (-4, -5.5) Disease;
[r, minimum width=3cm] (l6n1) at (-4, -6.5) IVOM;
[r, minimum width=3cm] (l7n1) at (-4, -7.5) VA;
[draw, inner sep=0.2cm, fit=(l4n1) (l5n1) (l6n1) (l7n1)] (f1) ;
[above] (f1l) at (f1.north) Category-centered tables;

[r, minimum width=3cm] (l4n2) at (0, -4.5) Global statistics;
[r, minimum width=3cm] (l5n2) at (0, -5.5) Anamnesis statistics;
[r, minimum width=3cm] (l6n2) at (0, -6.5) Disease statistics;
[r, minimum width=3cm] (l7n2) at (0, -7.5) IVOM statistics;
[draw, inner sep=0.2cm, fit=(l4n2) (l5n2) (l6n2) (l7n2)] (f2) ;
[above] (f2l) at (f2.north) Predictive statistics;

[->] (l4n1.east) – (l5n2.west);
[->] (l5n1.east) – (l6n2.west);
[->] (l6n1.east) – (l7n2.west);

[r, minimum width=3cm] (l4n3) at (4, -4.5) Local OCT class.;
[r, minimum width=3cm] (l5n3) at (4, -5.5) Global OCT class.;
[r, minimum width=3cm] (l6n3) at (4, -6.5) VA prediction;
[r, minimum width=3cm] (l7n3) at (4, -7.5) Treatment adjustment;
[draw, inner sep=0.2cm, fit=(l4n3) (l5n3) (l6n3) (l7n3)] (f3) ;
[above] (f3l) at (f3.north) Classification and prediction;

[->] (l4n3) – (l5n3);
[->] (l5n3) – (l6n3);

[r, minimum width=4cm] (l8n1) at (0, -9) Treatment recommendation;

[->] (l1n1) – (l2n1);
[->] (l1n2) – (l2n2);
[->] (l1n3) – (l2n3);

[->] (l2n1) – (l3n1);
[->] (l2n2) – (l3n1);
[->] (l2n3) – (l3n1);

[->] (l3n1) – (l3n2);
[->] (l3n1) – (f1l.north);
[->] (l3n1) – (f2l.north);
[->] (l3n1) – (f3l.north);

[->] (f1.south) – (l8n1);
[->] (f3.south) – (l8n1);

Proposed medical text and image data mining workflow for the following descriptive statistics and VA modelling based on our three base systems of the electronic health record system (EHR or Turbomed system), the clinical information system (CIS system), and the OCT system. Our patient progression visualization and modelling dashboard is shown in Fig. <ref>. Currently, the classification and prediction step of treatment adjustment is still in development. However, given all data, a treatment recommendation can be realized.

§ METHODS AND IMPLEMENTATION

This section provides the methods and their implementation related to the patient progression modelling. Firstly, the proposed system's architecture for data acquisition, preprocessing, analysis, and prediction is introduced (section <ref>). To allow a unified data processing, the role of the related terminologies for medical application (ophthalmic ontology) is addressed (section <ref>), whereas the data fusion and cleaning is introduced (section <ref>). Following, descriptive statistics are derived (section <ref>). Finally, the principles of the patient progression modelling within the context of our ML- and DL-based approaches are introduced (section <ref>).

The related implemented models and approaches to patient progression modelling as well as OCT biomarker classification are provided via open science from the within this contribution leveraged machine learning framework[<https://github.com/TSchlosser13/Hexnet/blob/master/_ML/models/contrib>].

§.§ System architecture

The from data as well as text mining obtained ophthalmic core data set is based on the categories of general patient information (G), VA-based patient information (V), OCT scans and biomarkers (O), diseases (D), as well as treatments and medications (T). These categories are obtained from different base systems, the electronic health record system (EHR or Turbomed system), the clinical information system (CIS system), and an OCT system (Heidelberg system) (see also Fig. <ref>). While all three systems contain basic patient information, the OCT system provides first and foremost OCT scans (categories G and O). The EHR system consists of one large database which contains all relevant patient information within the categories of G, V, D, and T. As the treatment and medication information within the EHR system is not always complete, the clinic's CIS system is additionally utilized in order to retrieve missing medication information (category CIS). Following the retrieved data of all three base systems in the form of BDT (EHR system), CSV (CIS system), as well as E2E files (OCT system), all relevant information for a patient-based data merging is processed. Finally, all patient information is chronologically synchronized, whereas sensitive patient information is pseudo-anonymized. These results are then presented via our patient progression visualization and modelling dashboard (Fig. <ref>).

§.§ Text processing and ontology

The challenge of the ophthalmic text processing is given by the heterogeneity and incompleteness of the underlying data itself, which are in turn documented by many different medical doctors. For text processing and ontology creation, the programming language Python with the Natural Language Toolkit (NLTK) <cit.> was utilized. Our implementation encompasses a set of category-specific rules handling abbreviations, negations, and synonyms as well as orthography and grammar variants and mistakes. For this purpose, recognized text strings are mapped into a unique ontology via a dictionary-based stemming <cit.>. The results have been qualitatively inspected for reliability and plausibility. A thorough quantitative evaluation of our text processing algorithms is still subject to further investigations and will be part of our future research. Therefore, it is beyond the scope of this contribution to further explain the underlying (German) text processing in detail. In the following, we demonstrate some arbitrary strings from the EHR system's diagnosis with customized (German) abbreviations and how they are mapped to diseases, whereby “$*$” denotes a placeholder for any string that is not handled otherwise. The mapping is case-insensitive and was specifically adjusted for German doctors (Table <ref>).

Disease     Rules

`feucht', `feuchte', `exsudativ', `exsudative', *     $\rightarrow$     True

`trocken', `trockene'                                 $\rightarrow$     False

`diabetisch', `diabetisches'     $\rightarrow$     True

*                                $\rightarrow$     False

`ast', `retinal', `zentral'     $\rightarrow$     True

*                               $\rightarrow$     False

Exemplary (German) ophthalmic text processing rules for AMD, DME, and RVO.

§.§ Data fusion and cleaning

Our practice management software supports database exports via the BDT file format. Here, we extracted electronic health records from over 10 years ranging from 2010 to 2020, including over 49 000 patients and more than 130 000 examinations. Our six main categories (G, V, O, D, T, and OP) currently span a total of more than 30 subcategories. More than 18 000 IVOMs are available from 2013 to 2020 after a fusion of the EHR and CIS systems' data. All data were linked to the over 12 000 OCT volumes exported from the OCT system via the patient ID (see also Fig. <ref>, patient-based data merging).

Subsequently, the obtained merged data table is further processed and cleaned. Firstly, it is cleaned by mapping the arbitrary text strings from the medical letters to diseases. The medical letters contain several terms and abbreviations for specific diseases as different letters stem from several different doctors. Secondly, unspecific and invalid terms are revised. For example, the entry “eye side” of treatments is only allowed to have the entries “left” or “right”, while invalid entries such as “-” are removed.

§.§ Category-centered data organization and descriptive statistics

After the data cleaning and fusion, the data are focused towards description of i) anamnesis, ii) intravitreal operative medications (IVOMs), iii) diseases, as well as iv) visual acuity. The main table is changed towards a patient-centered description (Fig. <ref>). The aim is to record the start and (if available) the end dates of anamnesis entries, diseases, and IVOM therapy cycles for each patient. We designed our tables via two lines per patient as the diseases are to a large degree eye independent, hence we include both eyes as separate lines. For the medical doctors, the category-centered tables allow an easier filtering, sorting, and a more compact view of the data for particular medical information.

Finally, all tables are visualized in the statistical-description module. In addition to illustrating a single data set, the tables can also be combined to show cross-referenced data correlations. From all available visualization, including up to 30 combinations of visual acuity and disease statistics, we illustrate in this work due to space constraints the statistics of the aforementioned diseases as well as a disease statics under comorbidities.

[Exudative AMD]

ybar, bar width=0.15cm, enlarge x limits=0.25, symbolic x coords=<1, 1-3, 3-6, >6,
width=0.3, height=0.3,
title=$\mathbf{N = 885}$ (737 patients), xlabel=Time window [years], ylabel=Relative frequency,
ymin=0.0, ymax=0.8,
ytick=0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, yticklabels=0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8,

xtick=data, ymajorgrids=true,
legend columns=-1, legend style=column sep=1ex]
[draw=LimeGreen,   fill=LimeGreen]   coordinates (<1, 0.28788) (1-3, 0.20243) (3-6, 0.18391) (>6, 0.16766);
[draw=custom_blue, fill=custom_blue] coordinates (<1, 0.45454) (1-3, 0.32186) (3-6, 0.23371) (>6, 0.20959);
[draw=Red,         fill=Red]         coordinates (<1, 0.25758) (1-3, 0.47571) (3-6, 0.58238) (>6, 0.62275);

W, S, L

ybar, bar width=0.15cm, enlarge x limits=0.25, symbolic x coords=<1, 1-3, 3-6, >6,
width=0.3, height=0.3,
title=$\mathbf{N = 320}$ (312 patients), xlabel=Time window [years], ylabel=,
ymin=0.0, ymax=0.8,
ytick=0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, yticklabels=,

xtick=data, ymajorgrids=true]
[draw=LimeGreen,   fill=LimeGreen]   coordinates (<1, 0.50000) (1-3, 0.39098) (3-6, 0.30556) (>6, 0.42308);
[draw=custom_blue, fill=custom_blue] coordinates (<1, 0.30769) (1-3, 0.40601) (3-6, 0.31944) (>6, 0.23077);
[draw=Red,         fill=Red]         coordinates (<1, 0.19231) (1-3, 0.20301) (3-6, 0.37500) (>6, 0.34615);

ybar, bar width=0.15cm, enlarge x limits=0.25, symbolic x coords=<1, 1-3, 3-6, >6,
width=0.3, height=0.3,
title=$\mathbf{N = 647}$ (530 patients), xlabel=Time window [years], ylabel=,
ymin=0.0, ymax=0.8,
ytick=0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, yticklabels=,

xtick=data, ymajorgrids=true]
[draw=LimeGreen,   fill=LimeGreen]   coordinates (<1, 0.39252) (1-3, 0.33607) (3-6, 0.21714) (>6, 0.29787);
[draw=custom_blue, fill=custom_blue] coordinates (<1, 0.38318) (1-3, 0.43852) (3-6, 0.30857) (>6, 0.32979);
[draw=Red,         fill=Red]         coordinates (<1, 0.22430) (1-3, 0.22541) (3-6, 0.47429) (>6, 0.37234);

[DME with
epiretinal membrane]

ybar, bar width=0.15cm, enlarge x limits=0.25, symbolic x coords=<1, 1-3, 3-6, >6,
width=0.3, height=0.3,
title=$\mathbf{N = 80}$ (73 patients), xlabel=Time window [years], ylabel=,
ymin=0.0, ymax=0.8,
ytick=0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, yticklabels=,

xtick=data, ymajorgrids=true]
[draw=LimeGreen,   fill=LimeGreen]   coordinates (<1, 0.33333) (1-3, 0.43750) (3-6, 0.36364) (>6, 0.21429);
[draw=custom_blue, fill=custom_blue] coordinates (<1, 0.33334) (1-3, 0.25000) (3-6, 0.22727) (>6, 0.21428);
[draw=Red,         fill=Red]         coordinates (<1, 0.33333) (1-3, 0.31250) (3-6, 0.40909) (>6, 0.57143);

(a–c) Real-world therapy winner (W), stabilizer (S), and loser (L) distribution for exudative AMD, RVO, and DME. (d) Distribution for disease DME under exemplary medical co-factor of epiretinal membrane. The shown results are based on our disease statistics (Fig. <ref>), whereas $\mathbf{N}$ denotes the number of eyes for the given number of patients.

§.§ Patient progression modelling

For the OCT biomarker classification and patient progression modelling via VA prediction, a set of prominent as well as conventional approaches from machine learning and deep learning are adapted by utilizing the machine learning library TensorFlow with Keras as its front end. Furthermore, scikit-learn is deployed for all machine learning based models and evaluation procedures. Since extracted medical data from patients' EHRs originate from ophthalmologists' documentations, they are defined as our ground truth and serve as training as well as test data.

§.§.§ OCT biomarker classification

In order to complete incomplete OCT biomarker documentations, our multistage system consists of an ensemble of different models for the local (slice-wise) OCT classification, which in turn enables a global (scan-wise) OCT biomarker classification. Within our proposed approach, OCT biomarkers are firstly classified using the provided OCT images of the OCT scans' slices (OCT B-scans). For the following scan-wise OCT biomarker classification, the slice-wise classifications are combined by utilizing the respectively obtained classification confidences. Based on the obtained OCT biomarker classifications, the subsequent VA modelling is realized as a time series prediction using, among others, different ML- and DL-based models such as multilayer perceptrons (MLP) <cit.> and recurrent neural networks <cit.>.

group style=
group size=1 by 3,
vertical sep=2pt,
xlabels at=edge bottom,
xticklabels at=edge bottom
width=0.7, height=0.33,
xlabel=VA measurement number $i$,
y label style=at=(axis description cs:-0.1,0.5),
yticklabel style=/pgf/number format/.cd, precision=1, fixed, fixed zerofill,
legend pos=outer north east, legend cell align=left,
xmin=-1, xmax=19]
[ylabel=VA [logMAR], ymin=0.45, ymax=1.05, y dir=reverse, ytick=0.5, 0.6, 0.7, 0.8, 0.9, 1.0]

[mark=square*, mark options=draw=black!80, fill=custom_blue] coordinates
(0, 0.6) (1, 0.6) (2, 0.8) (3, 0.5) (4, 0.8) (5, 0.5) (6, 0.5) (7, 0.5) (8, 0.5) (9, 0.6) (10, 0.6) (11, 0.6) (12, 0.6) (13, 0.8) (14, 0.8) (15, 0.8) (16, 0.8) (17, 0.8) (18, 1.0)
Ground truth

[mark=*, mark options=scale=1.75, draw=black!80, fill=custom_blue] coordinates
(4, 0.6) (5, 0.6) (6, 0.6) (7, 0.55) (8, 0.55) (9, 0.55) (10, 0.6) (11, 0.6) (12, 0.6) (13, 0.75) (14, 0.75) (15, 0.75) (16, 0.75) (17, 0.75) (18, 0.75)
Predictions model

[mark=o, mark options=scale=1.75, draw=black!80] coordinates
(4, 0.5) (5, 0.5) (6, 0.5) (7, 0.5) (8, 0.5) (9, 0.6) (10, 0.6) (11, 0.6) (12, 0.6) (13, 0.8) (14, 0.8) (15, 0.8) (16, 0.8) (17, 0.8) (18, 0.8)
Predictions ophthalmic expert

[ylabel=$\Delta\text{VA [logMAR]}$, ymin=-0.35, ymax=0.35, y dir=reverse, ytick=-0.3, -0.2, -0.1, 0.0, 0.1, 0.2, 0.3, extra y ticks=-0.1, 0.1, extra y tick labels=, extra y tick style=grid=major]

at (axis cs:1,-0.2) Winners;
at (axis cs:1,-0.0) Stabilizers;
at (axis cs:1, 0.2) Losers;

[mark=square*, mark options=draw=black!80, fill=custom_blue] coordinates
(4, 0.3) (5, -0.3) (6, 0.0) (7, 0.0) (8, 0.0) (9, 0.1) (10, 0.0) (11, 0.0) (12, 0.0) (13, 0.2) (14, 0.0) (15, 0.0) (16, 0.0) (17, 0.0) (18, 0.2)
Ground truth

[mark=*, mark options=scale=1.75, draw=black!80, fill=custom_blue] coordinates
(4, 0.0) (5, 0.0) (6, 0.0) (7, -0.05) (8, 0.0) (9, 0.0) (10, 0.05) (11, 0.0) (12, 0.0) (13, 0.15) (14, 0.0) (15, 0.0) (16, 0.0) (17, 0.0) (18, 0.0)
Predictions model

[mark=o, mark options=scale=1.75, draw=black!80] coordinates
(4, 0.0) (5, 0.0) (6, 0.0) (7, 0.0) (8, 0.0) (9, 0.1) (10, 0.0) (11, 0.0) (12, 0.0) (13, 0.2) (14, 0.0) (15, 0.0) (16, 0.0) (17, 0.0) (18, 0.0)
Predictions ophthalmic expert

[ylabel=WSL prediction, ymin=-0.5, ymax=2.5, ytick=0, 1, 2, yticklabels=L, S, W]

[only marks, mark=*, mark options=scale=1.75, draw=black!80, fill=LimeGreen, y filter/.expression=y-0.125] coordinates
(6, 1) (7, 1) (8, 1) (9, 1) (10, 1) (11, 1) (12, 1) (13, 0) (14, 1) (15, 1) (16, 1) (17, 1)
Correct prediction: model

[only marks, mark=o, mark options=scale=1.75, draw=LimeGreen, y filter/.expression=y+0.125] coordinates
(6, 1) (7, 1) (8, 1) (9, 1) (10, 1) (11, 1) (12, 1) (13, 0) (14, 1) (15, 1) (16, 1) (17, 1)
Correct prediction: oph. expert

[only marks, mark=*, mark options=scale=1.25, draw=black!80, fill=Red, y filter/.expression=y-0.125] coordinates
(4, 1) (5, 1) (18, 1)
Incorrect prediction: model

[only marks, mark=o, mark options=scale=1.25, draw=Red, y filter/.expression=y+0.125] coordinates
(4, 1) (5, 1) (18, 1)
Incorrect prediction: oph. expert

Visual acuity (VA) modelling and evaluation principles. Top: VA progression of our exemplary patient from Fig. <ref>. Predictions of the VA are conducted by our MLP-LDA model starting with the 5th VA value. In addition, we ask an ophthalmic expert to predict the VA. As shown in our visualization, both the predictions of our ophthalmic expert as well as our proposed model lie close to each other, which in turn emphasizes their prediction capabilities within the context of our ground truth data. Middle: Difference of the VA from the last to the current data point, showing the improvement, stabilization, or aggravation of the VA from the last examination. A threshold of $0.1$ logMAR units divides the patient into three groups: $\Delta\textit{VA}_i < -0.1$ into winners (W), $-0.1 \leq \Delta\textit{VA}_i \leq 0.1$ into stabilizers (S), and $\Delta\textit{VA}_i > 0.1$ into losers (L), whereby $i$ denotes the data point with $\Delta\textit{VA}_i = \textit{VA}_i - \textit{VA}_{i - 1}$. The first four data points of the ground truth are omitted within the visualization. Bottom: Prediction quality of our model and our ophthalmic expert. Shown are the predictions of the patient for our approach to VA prediction and our ophthalmic expert, whereas correct (•) and incorrect (•) WSL predictions are highlighted. The underlying ophthalmic feature vector data organization for VA prediction is depicted in Table <ref>.

§.§.§ Visual acuity prediction

To allow a WSL-based grouping of VA values, the logMAR score of each VA value is derived via $\textit{VA}_\textit{logMAR} = -\log_{10}\textit{VA}_\textit{dec}$ <cit.>. We define a decimal VA range of $0.04$ to $2.0$, corresponding to a logMAR range of $1.4$ to $-0.3$. For the local VA $\Delta\textit{VA}_i$ of the examination $i$, two adjacent VA values are compared via $\Delta\textit{VA}_i = \textit{VA}_i - \textit{VA}_{i - 1}$. A threshold of $0.1$ logMAR units divides examination $i$: $\Delta\textit{VA}_i < -0.1$ into winners, $-0.1 \leq \Delta\textit{VA}_i \leq 0.1$ into stabilizers, and $\Delta\textit{VA}_i > 0.1$ into losers (see also Fig. <ref>). Table <ref> gives an overview over our ophthalmic data set with its data organization for VA prediction. To predict the VA at a given date, a window with previous patient information of minimum window size 4 and maximum window size 10 is utilized (a). For this purpose, the shown data organization from lines 1 to 10 is deployed, whereas the data within the fields “treatment”, “OCT biomarker”, as well as “additional data” is further detailed in (b). Here, “l/r” denotes the availability of data for both eyes. When translating the information from (a) and (b), factorized data vectors are obtained (c) which are in turn applied to the model learning and evaluation procedures. In (c), “$-1$” denotes a numeric placeholder when no information is present within the respective data fields. For the sake of simplicity, the processing step of data factorization was omitted. For example, the data entry of the patient's sex can result in three different values: $0$ for male, $1$ for female, and $2$ for patients of diverse sex. Shown are the values for the first window of size 4 in Fig. <ref>.

1c|     5c||Slice-wise F1-score [%]     Scan-wise F1-score [%]

OCT biomarker     Logistic Regression <cit.>     Random Forest Classifier <cit.>     Multilayer Perceptron Classifier <cit.>     DenseNet201 <cit.>     ResNet152V2 <cit.>     Random Forest Classifier <cit.>

ELM                     $85.4 \pm 2.1$     $92.1 \pm 1.4$     $95.3 \pm 0.9$     $96.2 \pm 0.6$     $\mathbf{98.3 \pm 0.5}$     $\mathbf{99.9 \pm 0.1}$

Ellipsoid               $82.7 \pm 1.7$     $93.8 \pm 1.4$     $96.1 \pm 0.7$     $97.3 \pm 1.2$     $\mathbf{98.7 \pm 1.1}$     $\mathbf{99.9 \pm 0.1}$

Foveal depression       $56.0 \pm 3.4$     $69.4 \pm 0.9$     $77.3 \pm 2.5$     $75.9 \pm 2.3$     $\mathbf{77.5 \pm 2.4}$     $\mathbf{99.9 \pm 0.1}$

Scars                   $63.8 \pm 0.9$     $72.2 \pm 1.0$     $74.5 \pm 1.3$     $77.2 \pm 1.2$     $\mathbf{77.6 \pm 1.0}$     $\mathbf{98.3 \pm 1.2}$

PED                     $50.4 \pm 1.4$     $63.6 \pm 0.9$     $64.0 \pm 1.1$     $\mathbf{69.0 \pm 1.0}$     $67.3 \pm 0.9$     $\mathbf{94.8 \pm 0.5}$

Subretinal fibrosis     $69.3 \pm 1.5$     $70.6 \pm 0.6$     $74.5 \pm 0.4$     $\mathbf{75.1 \pm 0.2}$     $74.6 \pm 0.3$     $\mathbf{96.2 \pm 0.9}$

Mean     $67.9$     $77.0$     $80.3$     $81.8$     $\mathbf{82.3}$     $\mathbf{98.2}$

OCT biomarker classification results for the slice-wise as well as the scan-wise OCT biomarker classification. We conducted our test runs with a randomized data set split ratio of 80/10/10 for training, validation, and test set averaged over five runs. For our OCT biomarker data set overview, see Table <ref>.

§ TEST RESULTS, EVALUATION, AND DISCUSSION

The following sections provide our evaluation regarding the predictive statistics of therapy winners, stabilizers, and losers (section <ref>). To allow the inclusion of OCT biomarkers into the patient progression and VA modelling process, incomplete or missing OCT documentations are completed (section <ref>). Finally, given each patient's medical data, the following patient progression and VA prediction utilizes, i.a., the resulting OCT biomarker completions (section <ref>).

§.§ Predictive statistics

For the following predictive statistics, a statistical analysis was conducted for the aforementioned diseases, whereby the progression of the VA was classified into therapy winners, stabilizers, and losers via the first and the last VA measurement of a patient (section  <ref>).

The size of our data corpus and its harmonisation as described in sections <ref> to <ref> allows different kind of statistical surveys, e.g., separated according to disease, time periods, and comorbidities. The data originate from a maximum care medical hospital and thus indicate effects of real-world scenarios for the aforementioned ascertainments. Our statistical investigations consequently allow us to make statistical predictions under real-life conditions for questions such as “If a patient has exudative AMD, what are the future prognoses for this patient?”. For the three aforementioned diseases, the outcomes are shown in Fig. <ref>. We observed that in average especially for AMD a deterioration of the visual acuity might be expected. A more fine-grained analysis reveals effects over time. Here, the time window refers to the time difference (in years) between the first and the last VA measurement of individual patients. The data of the $\mathbf{N} = \text{885}$ eyes of AMD patients shown in Fig. <ref> are now divided into 4 substatistics with different time windows where the percentage of losers in an observation time window of under 1 year is about $25~\%$. With regard to longer observation time windows, the proportion of losers increases further till $>60~\%$ for time windows of >6 years and longer. Note that each of the four subpopulations consist of $\mathbf{N} = \text{150--250}$ eyes, and patients / eyes in the <1 year statistics do not occur in the other three substatistics (>1 year), and vice versa. Statistical significance tests are currently not the focus of this work. However, we believe that the data allow us to interpret the increase in the population of the therapy loser group as real effect. This means that more and more patients are experiencing deterioration of vision over longer periods of time. The representation of comorbidities is shown in Fig. <ref> as a proof of concept. Even here, if DME and epiretinal membrane (ERM) occur simultaneously, it becomes apparent that only about 12–25 patients are included in each substatistic and the direct comparison with the DME-only group would probably not yet stand up to statistical tests. More data will be needed in the future, e.g., by merging several ophthalmic hospitals into a common research data infrastructure.

§.§ OCT biomarker classification

Our 6 OCT biomarkers of interest are separated into two states, physiological and pathological, defining two distinct classes. These take the values interrupted (pathological) vs. preserved (physiological) for external limiting membrane (ELM) and ellipsoid zone (ellipsoid) as well as present (pathological) vs. not present (physiological) for foveal depression, scars, pigment epithelial detachment (PED), and subretinal fibrosis, respectively. An OCT biomarker data set overview with the available OCT B-scans per OCT biomarker and related classes is shown in Table <ref>. For our OCT biomarker OCT slice extraction, we determined an intermediate subset of slices with a slice range of 8 to 18 from 25 in total. Since different OCT scans may possess different original image resolutions, their image resolution was scaled to an initial size of $256 \times 256$ pixels.

Table <ref> shows our classification results for the slice-wise and the scan-wise OCT classification using different prominent approaches from machine learning and deep learning averaged over five runs. In comparison, the selected models DenseNet-201 <cit.> and ResNet-152V2 <cit.> show the best classification accuracies in F1-score with mean classification accuracies of $81.8$ and $82.3$ % for our 6 OCT biomarkers. When comparing the results for the different OCT biomarkers, ELM and ellipsoid show the best scan-wise accuracies, whereas subretinal fibrosis and PED represent the more challenging OCT biomarkers. For the following scan-wise OCT classification, a random forest classifier <cit.> is already sufficient in order to achieve high classification accuracies of up to $99.9$ %. Finally, we obtain the resulting mean classification accuracies in F1-score of $82.3$ (slice-wise) and $98.2$ % (scan-wise).

1c|     2c|Macro average F1-score [%]

Model     Feature vectors containing VA values only     Feature vectors containing additional medical data

3|c|Model predictions

Statistical estimator                                  $32.8 \pm 0.4$     /

MA estimator                                           $29.6 \pm 0.3$     /

Weighted MA estimator                                  $22.6 \pm 0.4$     /

Bagging regressor <cit.>                    $\mathbf{40.8 \pm 0.2}$     $41.3 \pm 0.5$

Random forest regressor <cit.>              $39.9 \pm 0.1$     $40.7 \pm 0.4$

RNN with LSTM <cit.>     $37.6 \pm 0.3$     $39.0 \pm 0.3$

RNN with GRU <cit.>           $38.1 \pm 0.3$     $39.1 \pm 0.1$

MLP <cit.>                                  $\mathbf{40.2 \pm 0.4}$     $\mathbf{44.6 \pm 0.6}$

MLP-LDA <cit.>                  $\mathbf{44.9 \pm 4.5}$     $\mathbf{69.0 \pm 5.2}$

MLP / MLP-LDA w/o annot. OCT biom.     $42.9 \pm 0.4$     $63.1 \pm 4.7$

MLP / MLP-LDA w/o class. OCT biom.     $42.7 \pm 0.6$     $62.8 \pm 3.1$

Ophthalmic expertise     2c|$\mathbf{58.0}$

VA modelling results overview for our approaches with feature vectors containing VA values only / additional medical data (Table <ref>b) as well as our annotations (ophthalmic expertise) averaged over five runs. “/” denotes model predictions without additional medical data and VA values only. For our VA modelling data set overview with data explanations and vectors, see Table <ref>. Figures <ref> and <ref> as well as Table <ref> show our evaluation results in further detail.

§.§ Visual acuity prediction

The principles of VA prediction are illustrated in Fig. <ref> for an exemplary patient whose first diagnosis was cataract in both eyes and DME in the right eye in 02/2014. Shown is the VA progression over a timespan of 3 years, in which the patient had 6 IVOMs with Eylea and Lucentis. Given a minimum window size of 4, the VA predictions for the 5th to the last examination are based on the previous examinations with their underlying medical patient data. For evaluation, the horizontal lines indicate our thresholds for therapy winners, stabilizers, and losers (see also section <ref>).

§.§.§ Evaluation principles

Out of 49 000 patients, 7 878 patients with VA series of length $\geq$ 5 exist within our data, resulting in over 100 000 separate VA series of length 5. With the three diseases AMD, DME, and RVO, 1 496 patients with VA series of length $\geq$ 5 exist, resulting in 14 026 separate VA series of length 5.

For the evaluation, all changes within all consecutive VA-based predictions are considered. Hence, all predictions as shown in Fig. <ref> are utilized in order to formulate a prediction capability assessment. The VA-based prediction accuracy is calculated via all VA predictions and related local $\Delta\textit{VA}_i$. For the modelling process, the completed OCT documentations (section <ref>) and the related additional data is retrieved. For this purpose, Table <ref> gives an extensive overview over the leveraged data set as well as the related data organization.

§.§.§ Training and test setup

Our training setup for VA prediction includes: the Glorot initializer <cit.> for weight initialization, the Adam optimizer <cit.> with a standard learning rate of $0.001$ and exponential decay rates of $0.9$ and $0.999$, as well as a batch size of $32$. We conducted our test runs with a randomized data set split ratio of 80/10/10 for training, validation, and test set.

ybar, enlarge x limits=0.1,
symbolic x coords=Winner (P), Winner (R), Loser (P), Loser (R), Stabilizer (P), Stabilizer (R),
width=0.9, height=8cm,
xlabel=Metric, ylabel=Result [%],
ymin=0, ymax=100,

xtick=data, ymajorgrids=true,
legend columns=-1, legend style=at=(1, 1.05), anchor=south east, column sep=1ex]
[draw=bblue,   fill=bblue]   coordinates (Winner (P), 38.7) (Winner (R), 17.1) (Loser (P), 75.0) (Loser (R), 50.6) (Stabilizer (P), 85.1) (Stabilizer (R), 94.9);
[draw=ggreen,  fill=ggreen]  coordinates (Winner (P), 16.3) (Winner (R), 37.7) (Loser (P), 38.2) (Loser (R), 29.0) (Stabilizer (P), 83.0) (Stabilizer (R), 73.3);
[draw=ppurple, fill=ppurple] coordinates (Winner (P), 84.2) (Winner (R), 84.2) (Loser (P), 87.5) (Loser (R), 73.7) (Stabilizer (P), 63.6) (Stabilizer (R), 73.7);

Ophthalmic expertise, MLP, MLP-LDA

VA modelling results' precision (P) and recall (R) plot for our annotations (ophthalmic expertise) as well as our approaches to VA prediction with MLP and MLP-LDA based on our WSL scheme (see also Table <ref>).

§.§.§ Visual acuity prediction results

Table <ref> shows our prediction results using different statistical approaches as well as prominent approaches from machine learning and deep learning averaged over five runs. For this purpose, our statistical estimator predicts VA progressions utilizing the statistical distribution of our WSL scheme within our train set. The moving average (MA) estimator averages the given window of VA values, whereas the weighted MA estimator weights recent VA values more strongly in order to formulate a baseline prediction.

Whereas our baseline approaches result in prediction accuracies of up to $40.8$ % in macro average F1-score (bagging regressor <cit.>), a more realistic setting includes the completed OCT documentations, but also the in Table <ref> shown additional data. Our MLP-based predictor results in the second highest prediction accuracy, whereas the addition of OCT biomarkers allows their inclusion in the VA modelling process, resulting in a prediction accuracy of $44.6$ % with an improvement by $4.4$ %. Therefore, the inclusion of OCT biomarkers allows their modelling as crucial information and influential visual factors when no OCT classifications are provided. Following our MLP model, a meta model was realized which classifies the predicted VA values via our WSL scheme. While the MLP predicts the VA values, our so-called MLP-LDA model utilizes these predictions by further classifying them via linear discriminant analysis, whereas the obtained results are cross-validated. We obtain a final prediction accuracy of $69$ % ($+24.4$ %), while the predictions of our ophthalmic expertise reach $58$ %. Figure <ref> gives an overview over our obtained main results in precision and recall for therapy winners, stabilizers, and losers. Considering the class-wise scores, MLP-LDA strikes a balance between all three progression groups, whereas a trade-off between therapy winners and losers with therapy stabilizers is observable. Finally, Table <ref> depicts an extensive VA modelling results overview. In comparison to MLP and MLP-LDA, our annotations show increased scores for therapy stabilizers while achieving the highest general accuracy in true positives. However, MLP-LDA achieves a by $11$ % increased macro average F1-score while showing only a by $5.1$ % reduced accuracy in true positives. As therapy stabilizers are predominantly present within our data corpus, we conclude that in general a high class-wise prediction accuracy, especially for the critical progression group of therapy losers, has to be enforced in order to realize a (semi-)automated recommender system.

§ CONCLUSION AND OUTLOOK

In this contribution, we developed an IT system architecture that aggregates patient-wise information for more than 49 000 patients from different categories of various multimedia data in the form of text and images within multiple heterogeneous ophthalmic data resources stemming from a German hospital of maximum care. As the prediction of a patient's progression is challenging within this real-world setting, our realized workflow allows a first processing of medical patient data to enable a visual acuity prediction, OCT biomarker classification, as well as a general statistical evaluation and visualization.

The resulting data corpus allows predictive statements of the expected progression of a patient and his or her visual acuity in each of the three diseases of AMD, DME, and RVO. Our data revealed that especially exudative AMD results in a notable high amount of therapy “losers” ($56$ % regarding a time span of 3 to 6 years). A more fine-grained analysis is able to reveal the influence of medical co-existing factors such as other diseases. As proof of concept, we showed exemplary DME with an epiretinal membrane, yet, in some cases statistical surveys of comorbidities in combination with different observation time windows, the data situation is still too weak to derive reliable correlations.

For the following visual acuity based treatment progression modelling, incomplete OCT documentations are completed by classifying the OCT scans' slices (OCT B-scans), which in turn allows the classification of OCT scans when only single OCT slices are provided. Based on the obtained OCT slice classifications, a scan-wise OCT classification of the OCT biomarkers ELM, ellipsoid zone, foveal depression, scars, PED, and subretinal fibrosis resulted in an overall classification accuracy (F1-score) of over $98$ %. Finally, the completed OCT documentations are combined with additional medical data, defining our ophthalmic feature vectors for visual acuity prediction. In comparison to different approaches from machine learning and deep learning, we achieve an overall prediction accuracy of up to $69$ % (macro average F1-score), whereas our ophthalmic expertise reaches an overall prediction accuracy of $58$ % ($+11$ %).

However, as the influence of the OCT biomarkers is not yet fully understood, further investigations have to be conducted. Future contributions can build on these initial results in order to determine an optimal time for a change in medication or therapy. In addition, patient-related data of the different categories available and their relevance for the modelling process have to be further investigated in order to allow the realization of a (semi-)automated recommender system.

§ ACKNOWLEDGMENT

This research is partially funded by the European Social Fund for Germany as well as the Novartis Pharma GmbH.

§ APPENDIX

§.§ Classification definitions and terminology

The macro average F1-score is calculated via the class-wise F1-scores $f$, where $F$ denotes the set of all class-wise F1-scores following our WSL scheme: $\textit{macro average F1-score} = \sfrac{1}{|F|} \cdot \sum_{f \in F} f$, whereas the class-wise F1-scores are in turn calculated via $\textit{F1-score} = 2 \cdot \sfrac{(\textit{precision} \cdot \textit{recall})}{(\textit{precision} + \textit{recall})}$ with $\textit{precision} = \sfrac{\textit{TP}}{\textit{TP} + \textit{FP}}$ and $\textit{recall} = \sfrac{\textit{TP}}{\textit{TP} + \textit{FN}}$ (TP = classification true positives, FP = false positives, and FN = false negatives).

1c     2|c|Class     1c

OCT biomarker     Physiological     Pathological     Ratio [$\sfrac{\text{phys.}}{\text{path.}}$]

ELM                     $\hphantom{0}7\,660$     $\hphantom{00\,}407$     $18.8$

Ellipsoid               $\hphantom{0}7\,377$     $\hphantom{00\,}308$     $24.0$

Foveal depression       $40\,087$     $10\,923$     $\hphantom{0}3.7$

Scars                   $\hphantom{0}9\,088$     $\hphantom{0}4\,687$     $\hphantom{0}1.9$

PED                     $\hphantom{0}7\,866$     $10\,635$     $\hphantom{0}0.7$

Subretinal fibrosis     $\hphantom{0}8\,481$     $\hphantom{0}5\,261$     $\hphantom{0}1.6$

Ophthalmic data set overview for our 6 different OCT biomarkers. Shown are the available OCT slices per OCT biomarker with their related classes of physiological as well as pathological image samples.

[Ophthalmic expertise]

xlabel=Predictions, ylabel=Ground truth,
xticklabels=Winner, Loser, Stabilizer, xtick=0, 1, 2, xtick style=draw=none,
yticklabels=Stabilizer, Loser, Winner, ytick=0, 1, 2, ytick style=draw=none,
width=0.32, height=0.32,
enlargelimits=false, axis on top,
colormap=summapcolor=(white) color=(custom_blue!50!black),
nodes near coords, nodes near coords align=center,

nodes near coords black white/.style=
small value/.style=text=black,
large value/.style=text=white,
every node near coord/.style=
/pgf/number format/.cd, precision=3, fixed, fixed zerofill, /tikz/.cd,
check for zero/.code=
=/pgfplots/small value
/pgfplots/large value

check for zero

nodes near coords black white=0.24]

[matrix plot*, point meta=explicit] file

xticklabels=Winner, Loser, Stabilizer, xtick=0, 1, 2, xtick style=draw=none,
yticklabels=, ytick style=draw=none,
width=0.32, height=0.32,
enlargelimits=false, axis on top,
colormap=summapcolor=(white) color=(custom_blue!50!black),
nodes near coords, nodes near coords align=center,

nodes near coords black white/.style=
small value/.style=text=black,
large value/.style=text=white,
every node near coord/.style=
/pgf/number format/.cd, precision=3, fixed, fixed zerofill, /tikz/.cd,
check for zero/.code=
=/pgfplots/small value
/pgfplots/large value

check for zero

nodes near coords black white=0.24]

[matrix plot*, point meta=explicit] file

xticklabels=Winner, Loser, Stabilizer, xtick=0, 1, 2, xtick style=draw=none,
yticklabels=, ytick style=draw=none,
width=0.32, height=0.32,
enlargelimits=false, axis on top,
colormap=summapcolor=(white) color=(custom_blue!50!black),
nodes near coords, nodes near coords align=center,

nodes near coords black white/.style=
small value/.style=text=black,
large value/.style=text=white,
every node near coord/.style=
/pgf/number format/.cd, precision=3, fixed, fixed zerofill, /tikz/.cd,
check for zero/.code=
=/pgfplots/small value
/pgfplots/large value

check for zero

nodes near coords black white=0.24]

[matrix plot*, point meta=explicit] file

VA modelling results' globally normalized confusion matrices for our annotations (ophthalmic expertise) as well as our approaches to VA prediction with MLP and MLP-LDA based on our WSL scheme.

[linewidth=15cm, language=Python, basicstyle=, breaklines=true, frame=single, numbers=left, tabsize=1, gobble=3]
date_j,            date_j+1,            date_j+2,            ..., date_i
age_j,             age_j+1,             age_j+2,             ..., age_i
VA_j,              VA_j+1,              VA_j+2,              ..., VA_i
treatment_j,       treatment_j+1,       treatment_j+2,       ..., treatment_i
OCT biomarker_j,1, OCT biomarker_j+1,1, OCT biomarker_j+2,1, ..., OCT biomarker_i,1
OCT biomarker_j,2, OCT biomarker_j+1,2, OCT biomarker_j+2,2, ..., OCT biomarker_i,2
OCT biomarker_j,3, OCT biomarker_j+1,3, OCT biomarker_j+2,3, ..., OCT biomarker_i,3
...,                 ...,                   ...,                   ..., ...
OCT biomarker_j,n, OCT biomarker_j+1,n, OCT biomarker_j+2,n, ..., OCT biomarker_i,n
additional data_j, additional data_j+1, additional data_j+2, ..., additional data_i

[Data organization]

[Data explanations]

Entry     Explanation

1.     Current date in days since the patient's birth

2.     The patient's year of birth

3.     The patient's sex

4.     VA (l/r)

5.     Medication (l/r): Eylea or Lucentis

6.–8.     Related information: apoplexy, blood thinning, and myocardial infarction

2|c|OCT scans

9.–14.     OCT biomarkers (l/r): ELM, ellipsoid, foveal depression, scars, PED, and subretinal fibrosis

15.–17.     Related information (l/r): central retinal thickness as well as intraretinal and subretinal fluid

2|c|Additional data

18.–24.     Diseases (l/r): AMD, DME, and RVV as well as cataract, diabetic retinopathy, ERM, and pseudophakia

[linewidth=4cm, language=Python, basicstyle=, breaklines=true, frame=single, numbers=left, tabsize=1, gobble=3]
20461, 20495, 20523, 20551
1960, 1960, 1960, 1960
0, 0, 0, 0
0.8, 0.5, 0.5, 0.5
-1, -1, -1, -1
-1, -1, -1, -1
1, -1, -1, -1
-1, -1, -1, -1
-1, -1, -1, -1
-1, -1, -1, -1
431, -1, -1, -1
-1, -1, -1, -1
-1, -1, -1, -1
-1, -1, -1, -1
0, -1, -1, -1
1, -1, -1, -1
-1, -1, -1, -1
-1, -1, -1, -1
-1, -1, -1, -1
-1, -1, -1, -1
-1,  1,  1,  1
-1,  1,  1,  1
-1,  0,  0,  0
-1,  0,  0,  0

[Data vectors]

Ophthalmic data set overview with utilized data organization for VA prediction. To predict the VA at date $i$, a window of maximum window size 10 ($i - j$) is utilized (a). For this purpose, the shown data organization from lines 1 to 10 is deployed, whereas the data within the fields “treatment”, “OCT biomarker”, as well as “additional data” is further detailed in (b). Here, “l/r” denotes the availability of data for both eyes. When translating the information from (a) and (b), factorized data vectors are obtained (c) which are in turn applied to the model learning and evaluation procedures. In (c), “$-1$” denotes a numeric placeholder when no information is present within the respective data fields. For the sake of simplicity, the processing step of data factorization was omitted. For example, the data entry of the patient's sex can result in three different values: $0$ for male, $1$ for female, and $2$ for patients of diverse sex. Shown are the values for the first window of size 4 in Fig. <ref>.

9cOphthalmic expertise

2c     3|c|Predictions     4c

2c|     Winner     Loser     Stabilizer     Total     Pro rata [%]     Precision     Recall

3*Ground truth     Winner     $12$     $\hphantom{0}2$     $\hphantom{0}56$     $\hphantom{0}70$     $\hphantom{0}9.7$     $38.7$     $17.1$

Loser     $\hphantom{0}2$     $42$     $\hphantom{0}39$     $\hphantom{0}83$     $11.5$     $75.0$     $50.6$

Stabilizer     $17$     $12$     $543$     $572$     $78.9$     $85.1$     $94.9$

1c|     True positives     3c|$597$ ($\mathbf{82.3}$ %)     $725$     Mean     $66.3$     $54.2$

5c|     2c|Macro average F1-score     2c|$58.0$

2c     3|c|Predictions     4c

2c|     Winner     Loser     Stabilizer     Total     Pro rata [%]     Precision     Recall

3*Ground truth     Winner     $\hphantom{0}46$     $\hphantom{0}3$     $\hphantom{0}73$     $\hphantom{0\,}122$     $\hphantom{0}9.1$     $16.3$     $37.7$

Loser     $\hphantom{0}15$     $42$     $\hphantom{0}88$     $\hphantom{0\,}145$     $10.8$     $38.2$     $29.0$

Stabilizer     $222$     $65$     $788$     $1\,075$     $80.1$     $83.0$     $73.3$

1c|     True positives     3c|$876$ ($81.5$ %)     $1\,342$     Mean     $45.8$     $46.7$

5c|     2c|Macro average F1-score     2c|$44.5$

2c     3|c|Predictions     4c

2c|     Winner     Loser     Stabilizer     Total     Pro rata [%]     Precision     Recall

3*Ground truth     Winner     $70$     $\hphantom{0}0$     $25$     $\hphantom{0}95$     $33.3$     $84.2$     $84.2$

Loser     $\hphantom{0}0$     $80$     $15$     $\hphantom{0}95$     $33.3$     $87.5$     $73.7$

Stabilizer     $10$     $15$     $70$     $\hphantom{0}95$     $33.3$     $63.6$     $73.7$

1c|     True positives     3c|$220$ ($77.2$ %)     $285$     Mean     $78.4$     $77.2$

5c|     2c|Macro average F1-score     2c|$\mathbf{77.5}$

Extensive VA modelling results overview for our baseline approaches without additional data (VA values only), ML- and DL-based approaches with additional data, as well as annotations (ophthalmic expertise) of one single run.