# A comprehensive and easy-to-use multi-domain multi-task medical imaging
meta-dataset (MedIMeta)

Stefano Woerner  Cluster of Excellence Machine Learning: New Perspectives for
Science, University of Tübingen, Germany  Arthur Jaques  Cluster of Excellence
Machine Learning: New Perspectives for Science, University of Tübingen,
Germany  Christian F. Baumgartner  Cluster of Excellence Machine Learning: New
Perspectives for Science, University of Tübingen, Germany  Faculty of Health
Sciences and Medicine, University of Lucerne, Switzerland

###### Abstract

While the field of medical image analysis has undergone a transformative shift
with the integration of machine learning techniques, the main challenge of
these techniques is often the scarcity of large, diverse, and well-annotated
datasets. Medical images vary in format, size, and other parameters and
therefore require extensive preprocessing and standardization, for usage in
machine learning. Addressing these challenges, we introduce the Medical
Imaging Meta-Dataset (MedIMeta), a novel multi-domain, multi-task meta-
dataset. MedIMeta contains 19 medical imaging datasets spanning 10 different
domains and encompassing 54 distinct medical tasks, all of which are
standardized to the same format and readily usable in PyTorch or other ML
frameworks. We perform a technical validation of MedIMeta, demonstrating its
utility through fully supervised and cross-domain few-shot learning baselines.

Figure 1: Example images of all MedIMeta datasets.

## Background & Summary

In recent years, the field of medical image analysis has undergone a
transformative shift with the integration of machine learning (ML) techniques
fundamentally expanding the landscape of diagnostic and therapeutic
strategies. The advancement of this field hinges on the availability of large,
diverse, and well-annotated datasets, which are pivotal for training robust
and effective ML models. However, the process of collecting raw medical images
and bringing them to a format suitable for ML applications is often complex
and fraught with challenges. Medical images vary in format, size, and other
parameters and therefore require extensive preprocessing and standardization,
a task that becomes increasingly complex when working with multiple datasets
from multiple domains that need to be integrated into a cohesive standardised
format.

Another challenge is the scarcity of annotated datasets in medical imaging,
particularly for rare diseases or specific conditions. This limitation has
sparked the exploration of few-shot learning (FSL) methodologies. These try to
learn from few examples and are designed to make predictions based on a
limited number of training examples. While humans are adept at learning and
making accurate predictions from minimal information, achieving comparable
levels of performance in machines remains a significant challenge.

Cross-domain few-shot learning (CD-FSL) allows models to leverage the
information learned for a task in one domain and apply it to another task in
another domain, potentially reducing the need for extensive data in every new
task. This ability is of immense value in these contexts, especially for rare
conditions. We define a domain as a particular type of subject, i.e. the
studied anatomical region in the medical context, combined with a particular
imaging modality. Despite its potential, the cross-domain transfer poses
significant challenges, given the inherent variability in imaging modalities,
disease presentations, and data characteristics across different medical
fields and clinics. In addition to the variability in domains, the nature of
tasks involved in medical image analysis varies considerably, encompassing a
range of classification types such as binary, multi-class, or multi-label, and
differing in the number of target labels or classes. Developing algorithms
capable of navigating these complexities is essential for effective knowledge
transfer. In order to facilitate the development of such algorithms, good
preprocessing and standardization across the different domains are even more
critical in this context.

Addressing these challenges, we introduce the Medical Imaging Meta-Dataset
(MedIMeta), a novel multi-domain, multi-task meta-dataset designed to
facilitate the development and standardized evaluation of ML models and cross-
domain FSL algorithms for medical image classification. MedIMeta contains 19
medical imaging datasets spanning 10 different domains and encompassing 54
distinct medical tasks, offering opportunities for both single-task and multi-
task training. Many of the tasks are diagnostic tasks or are tasks immediately
relevant to a diagnosis. Additionally, MedIMeta contains auxiliary tasks (such
as gender prediction) that may not have immediate clinical relevance but may
nevertheless be of interest. Furthermore, they have relevance for multi-task
training or for training FSL algorithms which benefit from having a large
amount of tasks.

For improved practicality, each dataset within the MedIMeta dataset is
standardized to a size of $224\times 224$ pixels which matches image size
commonly used in pre-trained models. Furthermore, the dataset comes with pre-
made splits to ensure ease of use and standardized benchmarking. We
meticulously preprocessed the data and release a user-friendly Python package
[33] to directly load images for use in PyTorch.

This makes MedIMeta exceptionally accessible to ML researchers. This ease of
access to a diverse set of realistic medical tasks with no need for additional
preprocessing can serve as a bridge between medical professionals and the ML
community, fostering interdisciplinary collaboration. Moreover, MedIMeta is an
ideal platform for investigating cross-domain few-shot learning in medical
imaging. The rich array of tasks and domains presents an excellent opportunity
to study and develop cross-domain few-shot learning techniques.

In addition to presenting the meta-dataset, this paper also presents a
technical validation of MedIMeta, demonstrating its utility through fully
supervised and CD-FSL baselines. The validation confirms the dataset’s
reliability and robustness, establishing it as a credible benchmark for
research in ML for medical image analysis.

### Related datasets

Existing meta-datasets can be divided into two categories: those consisting of
multiple datasets from a single domain, and those comprising data from
multiple domains. An overview is shown in Table 1.

#### Single-domain meta-datasets

Single domain meta-datasets offer an easy way to benchmark few-shot learning
techniques such as meta-learning. One of the first meta-datasets in this
category was Omniglot [18], which consists of handwritten characters from a
wide range of alphabets. More challenging meta-datasets derived from natural
images were obtained by subsampling the widely used ImageNet [26], or CIFAR
datasets [16]. Examples include the MiniImageNet [30] and the TieredImageNet
[25] datasets, as well as CIFAR-FS [2] and FC100 [22].

#### Multi-domain meta-datasets

While single-domain meta-datasets offer an easy standardized way to evaluate
few-shot learning techniques they often lack realism. In reality data are
rarely from a single domain in FSL problems. This realization has led to the
release of several multi-domain meta-datasets.

The visual decathlon dataset [24] is one of the first multi-domain datasets.
It consists of 10 datasets from different visual tasks including traffic sign
and flower recognition. It also includes Omniglot [18] and CIFAR [16] datasets
discussed earlier.

Later [27] released another collection of 10 datasets coined the “Meta-
Dataset” [27]. While it partially overlaps with the visual decathlon dataset,
the “Meta-Dataset” was specifically designed to benchmark few-shot learning
algorithms on multiple domains. However, it does not contain any medical
datasets.

With a similar motivation [40] released the Visual Task Adaptation Benchmark
(VTAB) [40]. This meta-dataset consists of 19 tasks, again, partially
subsuming the previous datasets. A particular property of the VTAB benchmark
is the inclusion of datasets from three different domains covering natural
image understanding, structured scene understanding, and specialized tasks.
The VTAB dataset is also to our knowledge the first meta-dataset to include
medical images.

[7] later unified VTAB [40] and Meta-Dataset [27] into a larger Few-Shot
Classification Benchmark. [10] collected multiple previously available
datasets from multiple domains for benchmarking CD-FSL methods. Meta-Album
[29] is a collection of 40 different dataset from multiple domains and follows
a similar goal to our work. While it contains more datasets than MedIMeta, it
does not contain any medical domains except for microscopy.

MedMNISTv2 [39] is a collection of 12 medical imaging datasets from 9
different domains. It additionally contains 3D datasets. It is similar in
spirit to our work, but we go significantly beyond MedMNIST in the number of
tasks and task realism. The images in MedMNISTv2 have a very low resolution of
$28\times 28$ pixels that obscures fine details that may be clinically
relevant. In contrast, we process all images at high resolution and make them
available with an image size of $224\times 224$ pixels, which allows to detect
more clinically relevant features and is the typical resolution used in
pretrained neural networks. Additionally, MedMNISTv2 does not contain any
multitask datasets. In contrast, MedIMetacontains a wide variety of tasks
including binary, multi-class, and multi-label classification as well as
ordinal regression.

In some of the overlapping datasets, we found significant problems with
MedMNIST’s preprocessing, which we improve upon. Specifically, we found that
some of the center-cropped images in MedMNIST had their relevant part, i.e.,
the part of the image showing the disease, cropped away. We fix this problem
by instead zero-padding these datasets. Additionally, some datasets in
MedMNISTv2 may have training-test splits that put images of the same subject
in multiple splits. We instead generated splits by taking subject information
into account.

Table 1: Comparison of different datasets

Name |  # domains |  # datasets |  # medical datasets |  task types * |  # tasks |  # images |  image size |  multi-domain |  different task types |  multi-task |  contains medical data |  uniform image size
---|---|---|---|---|---|---|---|---|---|---|---|---
Omniglot [18] | 1 | 1 | 0 | MC | 1 | 32 000 | $105\times 105$ | – | – | – | – | ✓
_mini_ ImageNet [30] | 1 | 1 | 0 | MC | 1 | 60 000 | $84\times 84$ | – | – | – | – | ✓
_tiered_ ImageNet [25] | 1 | 1 | 0 | MC | 1 | 779 165 |  $84\times 84$ ** | – | – | – | – | ✓
CIFAR-FS [2] | 1 | 1 | 0 | MC | 1 | 60 000 | $32\times 32$ | – | – | – | – | ✓
FC100 [22] | 1 | 1 | 0 | MC | 1 | 60 000 | $32\times 32$ | – | – | – | – | ✓
Visual Decathlon [24] | 1 | 10 | 0 | MC, (B) | 10 | 1 580 558 | $\min(h,w)=72$ | ✓ | – | – | – | $\sim$
Meta-Dataset [27] | 7 | 10 | 0 | MC, (B) | 10 | 53 068 000 |  | ✓ | – | – | – | –
VTAB [40] | 3 | 19 | 1 | MC, (B) | 19 | 2 244 000 |  | ✓ | – | – | $\sim$ | –
Meta-Album _Mini_ [29] | 10 | 40 | 2 | MC | 40 | 220 950 | $128\times 128$ | ✓ | – | – | $\sim$ | ✓
MedMNIST v2 (2D) [39] | 9 | 12 | 12 | MC, B, ML, O | 12 | 708 069 | $28\times 28$ | ✓ | ✓ | – | ✓ | ✓
MedIMeta (ours) | 10 | 19 | 19 | MC, B, ML, O | 54 | 399 742 | $224\times 224$ | ✓ | ✓ | ✓ | ✓ | ✓

* MC: multi-class, B: binary, ML: multi-label, O: ordinal. Brackets indicate that a task can be interpreted as that task type but is not defined as such in the respective benchmark dataset.
** default settings.

While Meta-Dataset, VTAB, Meta-Album and MedMNIST contain different domains,
only VTAB and MedMNIST contain a variety of different tasks. MedIMeta is the
only data collection which contains datasets with multiple tasks, providing
users of MedIMeta with the option to train multi-task algorithms. Out of these
benchmarks, only MedMNIST contains a significant number of _medical_ tasks.
Our proposed MedIMeta contains 54 medical tasks and is easily extensible to
tasks from other fields, like natural images. Furthermore, we provide
utilities for converting other datasets (e.g. ImageNet) into the MedIMeta
format allowing to easily integrate MedIMeta with other data sources.

## Methods

We release the MedIMeta dataset, a novel, highly standardized meta-dataset
comprised of 19 publicly available datasets containing a total of 54 tasks. In
the following, we describe the source datasets and the data generation in
detail.

### Dataset

All datasets included in the MedIMeta dataset have either been previously
published under an open license that allows redistribution under a CC-BY-SA or
CC-BY-SA-NC license, or we obtained an explicit permission to do so. In
addition to having an open license, we selected these datasets based on three
criteria: suitability for defining a minimum of one classification task on the
data, image size suitable for rescaling to our target size without producing
noticeable artifacts, and a minimum of 100 images. All images in MedIMeta were
standardized to an image size of $224\times 224$ pixels. We provide pre-
defined training, validation and testing splits for all 19 datasets. If data
splits were already defined in the source data, we used the pre-existing
splits. Otherwise we generated our own. Most datasets include more than one
classification task. Typically, there is one main diagnostic task and several
auxiliary tasks. Most of these tasks were already present in the source
datasets. In some instances, we created additional tasks not present in the
source data. Table 2 gives an overview of all dataset, tasks, and their key
properties. In the following, we describe in detail all datasets that MedIMeta
is comprised of using the dataset ID, as well as the full dataset name.

aml, AML Cytomorphology:

Morphological dataset of leukocytes with expert-labeled single-cell images
from peripheral blood smears of patients with acute myeloid leukemia (AML) and
patients without signs of hematological malignancy, derived from the _Munich
AML Morphology Dataset_ [21]. The 18,365 original images were resized to
$224\times 224$ pixels using bi-cubic interpolation (no images were up-scaled)
and converted to RGB format by removing the transparency channel. We adopted
the original multi-class classification task with 15 morphological classes
from the source dataset.

bus, Breast Ultrasound:

Dataset of breast ultrasound images of women between 25 and 75 years old,
derived from the _Breast Ultrasound Images Dataset_ [6]. The 780 original
images were converted to grayscale and their masks to binary format. Images
and masks were zero-padded to a square shape and resized to $224\times 224$
pixels using bi-cubic interpolation and nearest neighbor interpolation
respectively (no images were up-scaled). The multi-class tumor classification
task with normal, benign, and malignant examples was adopted without
modifications from the source dataset. We additionally defined a binary
classification task between malignant tumors and other images.

crc, Colorectal Cancer:

Dataset of image patches from hematoxylin & eosin (H&E) stained histological
images of human colorectal cancer (CRC) and healthy tissue, derived from the
_NCT-CRC-HE-100K_ and _CRC-VAL-HE-7K_ datasets [12]. The 107,180 original
images from the training and validation sets were not modified, as they
already had the right shape and size for MedIMeta. We adopted the multi-class
tissue classification task with 9 labels from the source dataset without
modifications.

cxr, Chest X-ray Multi-disease:

Dataset of frontal-view X-ray chest images, derived from the _ChestX-ray14_
dataset [31]. The 112,120 original images were resized to $224\times 224$
pixels using bi-cubic interpolation (no images were up-scaled); the 519 images
that were originally in RGBA format were converted to grayscale. We provide a
multi-label thorax disease classification task with 14 labels adopted without
modifications from the source dataset. We additionally provide a binary
classification task of the patient sex derived from the labels present in the
original data.

derm, Dermatoscopy:

Dataset of dermatoscopic images of common pigmented skin lesions from
different populations acquired and stored by different modalities, derived
from the _HAM10000_ dataset [28]. The 11,720 original images were center-
cropped to a square shape and resized to $224\times 224$ pixels using bi-cubic
interpolation (no images were up-scaled). We provide the multi-class disease
classification task with 7 labels defined in the challenge hosted by the
International Skin Imaging Collaboration (ISIC) [4].

dr_regular, Diabetic Retinopathy (Regular Fundus):

Dataset of fundus images with diabetic retinopathy grades and image quality
annotations, derived from the _DeepDRiD_ dataset [20]. The 2,000 original
images were center-cropped to a square shape and resized to $224\times 224$
pixels using bi-cubic interpolation (no images were up-scaled). Following the
annotations present in the original data, we provide 5 tasks: diabetic
retinopathy grade (ordinal regression task with 5 labels), sufficient image
quality for gradability (binary classification task), strength of artifact
(ordinal regression task with 6 labels), image clarity (ordinal regression
task with 5 labels), and field definition (ordinal regression task with 5
labels).

dr_uwf, Diabetic Retinopathy (Ultra-widefield Fundus):

Dataset of ultra-widefield fundus images with annotations for diabetic
retinopathy grading, derived from the _DeepDRiD_ dataset [20]. Only the 250
original images without missing labels were kept. They were center-cropped to
a square shape and resized to $224\times 224$ pixels using bi-cubic
interpolation (no images were up-scaled). We adopted the DR grading task
(ordinal regression) with 5 labels from the source dataset without
modifications.

fundus, Fundus Multi-disease:

Multi-disease retinal fundus dataset of images captured using three different
fundus cameras with 45 conditions annotated through adjudicated consensus of
two senior retinal experts as well as an overall disease presence label,
derived from the _Retinal Fundus Multi-disease Image Dataset_ [23]. The 3,200
images were center-cropped to a square shape and resized to $224\times 224$
pixels using bi-cubic interpolation (no images were up-scaled). The original
disease presence binary classification task and disease multi-label
classification task with 45 labels were directly derived from the annotations
provided by the original dataset.

glaucoma, Glaucoma-specific fundus images:

Glaucoma-specific Indian ethnicity retinal fundus dataset of images acquired
using three devices, where five expert ophthalmologists provided annotations
on whether the subject is suspect for glaucoma or not, derived from the
_Chákṣu_ dataset [17]. The 1,345 original images and their masks were zero-
padded to a square shape and resized to $224\times 224$ pixels using bi-cubic
interpolation and nearest neighbor interpolation respectively (no images were
up-scaled). We used the glaucoma suspect majority vote annotation to derive a
binary classification task.

mammo_calc, Mammography (Calcifications):

Dataset of cropped regions of interest (calcifications), derived from the
_Curated Breast Imaging Subset of the Digital Database for Screening
Mammography_ (CBIS-DDSM) [19]. The 1,872 images were obtained by extending the
regions of interest (bounding boxes) to a square shape with a minimum size of
$224\times 224$ pixels, and extracting the resulting region crops from the
full original images. The region crops were then resized to $224\times 224$
pixels using bi-cubic interpolation. From the annotations, 3 tasks were
derived: pathology presence (binary classification task), calcification type
(multi-label classification task with 14 labels), and calcification
distribution (multi-label classification task with 5 labels).

mammo_mass, Mammography (Masses):

Dataset of cropped regions of interest (masses) from CBIS-DDSM. The 1,696
images were preprocessed as described for _Mammography (Calcifications)_. From
the annotations, 3 tasks were derived: pathology presence (binary
classification task), mass shape (multi-label classification task with 8
labels), and mass margins (multi-label classification task with 5 labels).

oct, OCT:

Dataset of validated Optical Coherence Tomography (OCT) images labeled for
disease classification, derived from [14]. The 84,484 original images were
center-cropped to a square shape and resized to $224\times 224$ pixels using
bi-cubic interpolation (no images were up-scaled). The original dataset
contains a multi-class disease classification task, with three different
diseases and a healthy class, which we adopt without modifications.
Additionally, we provide a binary task for whether the image warrants urgent
referral to a specialist based on the annotations present in the original
data.

organs_axial, Axial Organ Slices:

Dataset of axial image slices of 11 different organs, extracted from the
_Liver Tumor Segmentation Benchmark_ (LiTS) dataset [3] and the corresponding
organ bounding box annotations from [38]. We derived a multi-class organ
classification task with 11 labels by extracting a cropped image of each
individual organ in each of the CT volumes using the bounding box annotations.
We obtained a total of 1,645 organ images images. We removed 106 images for
which the voxel size information was missing. The axes on one image were
permuted to bring it to the same format as the other images. The images and
masks were sliced from the original 3D volumes by taking the center of the
organ bounding box in the axial plane. The Hounsfield-Units of the images were
transformed into grayscale images by applying a window with a width of 400 and
a level of 50, which are typical values for abdominal CT imaging. The images
and masks were cropped to a square size in the physical space, by centering at
the center of the bounding box and expanding the smaller side. The resulting
images and masks were resized to $224\times 224$ pixels using bi-cubic and
nearest neighbor interpolation, respectively. For visualization purposes, we
additionally provide images averaged over the 10% central slices with the
projected bounding boxes of all organs extracted from the image drawn on top.

organs_coronal, Coronal Organ Slices:

Dataset of coronal image slices of 11 different organs, extracted from the
LiTS dataset. The images were processed the same as described for the _Axial
Organ Slices_ dataset, except that the coronal projections were used.

organs_sagittal, Sagittal Organ Slices:

Dataset of sagittal image slices of 11 different organs, extracted from the
LiTS dataset. The images were processed the same as described for the _Axial
Organ Slices_ dataset, except that the sagittal projections were used.

pbc, Peripheral Blood Cells:

Dataset of microscopic peripheral blood cell images of individual normal
cells, captured from individuals without infection, with hematologic or
oncologic disease and free of any pharmacologic treatment at the moment of
blood collection, derived from [1]. The 17,092 original images were center-
cropped to a square shape and resized to $224\times 224$ pixels using bi-cubic
interpolation (no images were up-scaled). We adopted the original multi-class
blood cell classification task with 8 labels from the source dataset without
modifications to the annotations.

pneumonia, Pediatric Pneumonia:

Dataset of pediatric chest X-ray images labeled for pneumonia classification,
derived from [14]. The 5,856 original images were zero-padded to a square
shape and resized to $224\times 224$ pixels using bi-cubic interpolation (some
images were up-scaled); the 283 images that were originally in RGB format were
converted to grayscale. From the original annotations, we derived a binary
classification task for pneumonia presence as well as a multi-class task
differentiating between normal images, bacterial pneumonia and viral
pneumonia.

skinl_photo, Skin Lesion Evaluation (Clinical Photography):

A dataset containing clinical color photography images of skin lesions, along
with corresponding labels for seven different evaluation criteria and the
diagnosis, derived from [13]. The images were zero-padded to obtain a square
image and then resized to $224\times 224$ pixels using bi-cubic interpolation.
We adopted an overall diagnostic multi-class task, as well as separate
classification tasks for each of the seven diagnostic criteria from the source
dataset. Tasks containing infrequent labels have additional grouped versions
which bundle the infrequent labels together into more frequent labels. This
grouping is provided by the source dataset.

skinl_derm, Skin Lesion Evaluation (Dermoscopy):

A dataset containing dermoscopic color images of skin lesions, along with
corresponding labels for seven different evaluation criteria and the
diagnosis, derived from [13]. This dataset contains the same subjects as Skin
Lesion Evaluation (Clinical Photography) and images were preprocessed in the
same manner. The tasks were also identical to Skin Lesion Evaluation (Clinical
Photography).

Table 2: All MedIMeta tasks. Dataset Name |  Domain |  # Train Images |  # Val Images |  # Test Images | Task Name | Task Target |  # Labels
---|---|---|---|---|---|---|---
AML Cytomorphology | Microscopy | 12 855 | 1 836 | 3 674 | morphological class | multi-class classification | 15
Breast Ultrasound | Breast ultrasound | 546 | 78 | 156 | case category | multi-class classification | 3
malignancy | binary classification | 2
Colorectal Cancer Histopathology | Histopathology | 85 000 | 15 000 | 7 180 | tissue class | multi-class classification | 9
Chest X-ray Multi-disease | Chest X-ray | 73 421 | 13 103 | 25 596 | disease labels | multi-label classification | 14
patient sex | binary classification | 2
Dermatoscopy | Dermatoscopy | 10 015 | 193 | 1512 | disease category | multi-class classification | 7
Diabetic Retinopathy (Regular Fundus) | Retinal fundus | 1 200 | 400 | 400 | DR level | ordinal regression | 5
Overall quality | binary classification | 2
Artifact | ordinal regression | 6
Clarity | ordinal regression | 5
Field definition | ordinal regression | 5
Diabetic Retinopathy (Ultra-widefield Fundus) |  |  |  |  |  |  |
Retinal fundus | 150 | 50 | 50 | DR level | ordinal regression | 5
Fundus Multi-disease | Retinal fundus | 1 920 | 640 | 640 | disease presence | binary classification | 2
disease labels | multi-label classification | 45
Glaucoma-specific fundus images | Retinal fundus | 857 | 152 | 336 | Glaucoma suspect | binary classification | 2
Mammography (Calcifications) | Mammography | 1 332 | 214 | 326 | pathology | binary classification | 2
calc type | multi-label classification | 14
calc distribution | multi-label classification | 5
Mammography (Masses) | Mammography | 1 126 | 192 | 378 | pathology | binary classification | 2
mass shape | multi-label classification | 8
mass margins | multi-label classification | 5
OCT | OCT | 91 615 | 16 694 | 1 000 | disease class | multi-class classification | 4
urgent referral | binary classification | 2
Axial Organ Slices | Abdominal CT | 871 | 156 | 618 | organ label | multi-class classification | 11
Coronal Organ Slices | Abdominal CT | 871 | 156 | 618 | organ label | multi-class classification | 11
Sagittal Organ Slices | Abdominal CT | 871 | 156 | 618 | organ label | multi-class classification | 11
Peripheral Blood Cells | Microscopy | 11 964 | 1 709 | 3 419 | cell class | multi-class classification | 8
Pediatric Pneumonia | Chest X-ray | 4 415 | 817 | 624 | pneumonia presence | binary classification | 2
disease class | multi-class classification | 3
Skin Lesion Evaluation (Clinical) | Clinical skin imaging | 413 | 203 | 395 | Diagnosis | multi-class classification | 15
Diagnosis grouped | multi-class classification | 5
Pigment Network | multi-class classification | 3
Blue Whitish Veil | binary classification | 2
Vascular Structures | multi-class classification | 8
Vascular Structures grouped | multi-class classification | 3
Pigmentation | multi-class classification | 5
Pigmentation grouped | multi-class classification | 3
Streaks | multi-class classification | 3
Dots and Globules | multi-class classification | 3
Regression Structures | multi-class classification | 4
Regression Structures grouped | binary classification | 2
Skin Lesion Evaluation (Dermoscopy) | Dermatoscopy | 413 | 203 | 395 | Diagnosis | multi-class classification | 15
Diagnosis grouped | multi-class classification | 5
Pigment Network | multi-class classification | 3
Blue Whitish Veil | binary classification | 2
Vascular Structures | multi-class classification | 8
Vascular Structures grouped | multi-class classification | 3
Pigmentation | multi-class classification | 5
Pigmentation grouped | multi-class classification | 3
Streaks | multi-class classification | 3
Dots and Globules | multi-class classification | 3
Regression Structures | multi-class classification | 4
Regression Structures grouped | binary classification | 2

We open-source all code for creating the above datasets from their respective
source materials [36]. Our published source code contains easy-to-use utility
functions to extend MedIMeta with additional datasets. As examples, we provide
several additional pipelines to create more datasets in the same format from
other medical data which is publicly available but is not published under a
license which allows redistribution of derivative work.

### Python package

We release a Python package called medimeta that enables data loading. This
package is installable via pip install medimeta. Users can load data as single
datasets or as cross-domain batches. It also supports loading few-shot tasks
with support and query sets. Additionally, it is fully compatible with
TorchCross [34], a library for cross-domain and few-shot learning.

## Data Records

We have organized and made available the data files of the MedIMeta dataset on
Zenodo in [37]. The data for all datasets within MedIMeta can be accessed via
the provided DOI. Each dataset is packaged in a single zip file for
convenience and can be downloaded either separately or in one batch with all
datasets.

Each zip file contains a structured and organized collection of data, designed
to facilitate ease of use and comprehensive understanding of the dataset. The
content of these zip files is as follows:

images

This folder contains all the image files for the dataset as uint8 TIFFs, named
sequentially (e.g., 000000.tiff, 000001.tiff, etc.), ensuring easy access and
exploration for human viewers.

splits

This directory includes text files (train.txt, val.txt, and test.txt) listing
the image paths belonging to each respective split.

original_splits

For source datasets with pre-existing split definitions, this directory
contains those original splits, allowing users to adhere to the original data
segmentation if desired. The format is the same as in the splits directory.

task_labels

Each task within the dataset is accompanied by an .npy file named with the
respective task’s name (e.g., task_name_1.npy, task_name_2.npy, etc.)
contained in this folder. Each .npy file contains a single NumPy array that
represents the labels associated with that specific task.

annotations.csv

This file provides a comprehensive set of annotations, such as the patient id
or imaging plane, for the images in the dataset, offering detailed insights
and data points for those interested. It also contains all task labels to make
them more readily accessible to human readers.

images.hdf5

This file contains all the images from the folder images formatted as a single
dataset with dimensions $N\times C\times H\times W$, where N represents the
number of images, C the number of channels, H the height, and W the width. The
HDF5 file is useful for reading data in machine learning applications.

info.yaml

This file contains all relevant information about the dataset, including its
ID, name and description, the number of images in each split, domain
identifier, task definitions, and attribution information.

LICENSE

Each dataset is accompanied by its specific license file.

## Technical Validation

In order to validate our proposed dataset, we used it in two distinct learning
scenarios. First, we performed simple supervised training on each of the
datasets on its primary tasks. Secondly, we investigated the utility of our
dataset for CD-FSL. In the following, we describe the experiment setups for
the two scenarios in more detail.

### Supervised learning experiments

For the supervised experiments we trained ResNet-18 and ResNet-50 models [11]
on the primary task for each dataset. All networks were initialized with pre-
trained weights from ImageNet [26]. Early stopping was performed using the
AUROC on the respective validation sets.

### Cross-domain few-shot learning (CD-FSL) experiments

In this evaluation, we compare 5-shot performance using three different CD-FSL
approaches (described below). We evaluated the performance for each dataset in
a leave-one-out fashion using one task as target task, and the other tasks
from all datasets with no domain overlap or subject overlap as source tasks to
transfer knowledge from. The knowledge transfer was achieved by first training
a common backbone on the source tasks, and then fine-tuning the network to the
target task using a small support set of 5 labeled examples per class.
Evaluation was performed on a distinct query set consisting of 10 samples from
the same task. Figure 2 illustrates the training and evaluation procedure.

Figure 2: An overview of the CD-FSL scenario: The few-shot learner is first
trained on the meta-dataset of highly diverse training data. It is then
adapted to a new task from a new domain using the labeled examples from the
support set of a few-shot task. Performance is assessed using a query set from
the same task.

Because performance can vary substantially between runs due to the quality of
the 5 labeled examples, we reran the experiments 100 times for each target
task to obtain a more robust estimation of the performance.

#### Baseline CD-FSL algorithms

##### ImageNet pre-training (IM-PT):

The simplest baseline we investigated is simply initializing the common
backbone network with ImageNet weights, and then directly fine-tuning to the
target task.

##### Multi-domain multi-task pre-training (mm-PT):

Pre-training using ImageNet lacks specificity to the medical domain.
Incrementally pre-training on a series of available datasets may offer a
strategy to learn from many related datasets rather than just one [5].
However, incremental pre-training may suffer from catastrophic forgetting of
earlier tasks [15]. To address this issue, we propose a multi-domain multi-
task pre-training schedule, where for each model update we sample a batch from
a random source task. This strategy may facilitate learning representations
suitable for a wide range of tasks. The algorithm is summarized in the
supplemental materials.

##### Multi-domain Multi-task MAML (mm-MAML):

Model-agnostic meta-learning [8] has been shown to be a promising strategy for
CD-FSL [9]. MAML first “learns to learn” from a set of training tasks before
learning the desired test task. However, MAML assumes identical task types and
number of classes for each task, which is not realistic in practical settings.
Here, we employed our previously proposed Multi-domain Multi-task MAML (mm-
MAML) strategy [35] where we used an individual classification layer for each
class that is simply initialized with zeros. The algorithm is summarized in
the supplemental materials.

### Results

Table 3 shows the results for the single-domain baselines. It can be seen that
the models were able to achieve a high performances in terms of AUROC for most
datasets. The comparatively low performance on the diabetic retinopathy
detection from ultra-wide field images (dr_uwf) reflects the difficulty of the
task. As expected, the more complex ResNet 50 model achieved slightly higher
performance for most datasets compared to the ResNet 18.

Table 3: AUROC (%) on the test set for the fully supervised baselines.

| aml | bus | crc | cxr | derm |  | dr_
---
regular
| dr_
---
uwf
fundus | glaucoma |  | mammo_
---
calc
ResNet 18 | $99.1$ | $96.0$ | $99.5$ | $78.4$ | $94.0$ | $84.9$ | $58.1$ | $97.1$ | $74.5$ | $76.8$
ResNet 50 | $98.4$ | $96.6$ | $99.5$ | $80.5$ | $96.2$ | $86.5$ | $58.3$ | $97.2$ | $86.6$ | $75.0$

|  | mammo_
---
mass
oct |  | organs_
---
axial
| organs_
---
coronal
| organs_
---
sagittal
pbc | pneumonia |  | skinl_
---
photo
| skinl_
---
derm
ResNet 18 | $73.0$ | $99.7$ | $97.8$ | $97.1$ | $96.7$ | $99.9$ | $98.7$ | $75.0$ | $82.9$
ResNet 50 | $72.3$ | $99.8$ | $98.6$ | $97.9$ | $96.8$ | $100.0$ | $99.3$ | $78.0$ | $90.1$

Table 4: AUROC (%) for the CD-FSL baselines averaged across 100 5-shot
episodes using a ResNet18 and ResNet50.

|  | aml | bus | crc | cxr | derm |  | dr_
---
regular
| dr_
---
uwf
fundus | glaucoma |  | mammo_
---
calc
Training Type | Backbone |  |  |  |  |  |  |  |  |  |
ImageNet | ResNet18 | $82.1\pm 0.6$ | $71.4\pm 1.8$ | $94.2\pm 0.4$ | $58.5\pm 0.6$ | $74.4\pm 1.0$ | $69.4\pm 1.1$ | $53.0\pm 1.6$ | $75.1\pm 2.6$ | $55.0\pm 2.9$ | $61.4\pm 2.6$
ResNet50 | $82.0\pm 0.7$ | $72.9\pm 1.7$ | $95.4\pm 0.3$ | $57.9\pm 0.6$ | $74.9\pm 0.8$ | $71.3\pm 0.9$ | $54.0\pm 1.6$ | $77.0\pm 2.4$ | $53.0\pm 2.8$ | $64.2\pm 2.8$
MM-PFT | ResNet18 | $77.9\pm 0.8$ | $66.2\pm 1.7$ | $85.8\pm 0.6$ | $55.5\pm 0.5$ | $69.7\pm 0.8$ | $65.1\pm 1.1$ | $51.2\pm 1.5$ | $64.2\pm 2.9$ | $56.2\pm 3.1$ | $60.5\pm 2.8$
ResNet50 | $79.6\pm 0.7$ | $69.7\pm 1.7$ | $89.7\pm 0.5$ | $57.1\pm 0.6$ | $72.1\pm 1.0$ | $67.9\pm 1.0$ | $52.3\pm 1.6$ | $69.8\pm 2.5$ | $53.4\pm 2.6$ | $60.1\pm 3.1$
MM-MAML | ResNet18 | $70.3\pm 0.7$ | $61.5\pm 1.8$ | $82.4\pm 0.6$ | $54.4\pm 0.5$ | $68.6\pm 0.8$ | $63.3\pm 1.1$ | $51.1\pm 1.5$ | $58.4\pm 2.8$ | $52.2\pm 2.8$ | $58.3\pm 2.8$

|  |  | mammo
---
mass
oct |  | organs_
---
axial
| organs_
---
coronal
| organs_
---
sagittal
pbc | pneumonia |  | skinl_
---
photo
| skinl_
---
derm
Training Type | Backbone |  |  |  |  |  |  |  |  |
ImageNet | ResNet18 | $54.3\pm 3.0$ | $73.0\pm 1.2$ | $96.5\pm 0.3$ | $92.9\pm 0.4$ | $89.0\pm 0.5$ | $89.6\pm 0.6$ | $95.3\pm 1.1$ | $60.4\pm 1.8$ | $69.5\pm 1.7$
ResNet50 | $55.5\pm 2.8$ | $74.9\pm 1.3$ | $94.6\pm 0.3$ | $88.8\pm 0.4$ | $85.6\pm 0.5$ | $94.4\pm 0.5$ | $94.9\pm 1.1$ | $60.5\pm 1.6$ | $70.8\pm 1.8$
MM-PFT | ResNet18 | $55.4\pm 2.9$ | $68.9\pm 1.3$ | $92.1\pm 0.3$ | $87.0\pm 0.4$ | $84.8\pm 0.5$ | $85.0\pm 0.8$ | $88.5\pm 1.8$ | $63.0\pm 1.7$ | $67.5\pm 1.8$
ResNet50 | $54.2\pm 3.2$ | $67.0\pm 1.4$ | $91.3\pm 0.4$ | $86.9\pm 0.5$ | $85.9\pm 0.5$ | $88.5\pm 0.8$ | $91.4\pm 1.6$ | $59.7\pm 1.7$ | $65.4\pm 1.8$
MM-MAML | ResNet18 | $53.6\pm 2.8$ | $63.6\pm 1.2$ | $88.6\pm 0.4$ | $82.4\pm 0.5$ | $81.6\pm 0.5$ | $83.7\pm 0.7$ | $90.9\pm 1.4$ | $56.6\pm 1.5$ | $67.5\pm 1.7$

Table 4 shows 5-shot results for the CD-FSL baselines described earlier.
Surprisingly, simple fine-tuning from pre-trained ImageNet weights performed
as well or better than fine-tuning from the two pre-fine-tuning and MAML
baselines. This can mean that the pre-fine-tuning method we chose is too
simple to bring a meaningful benefit. At the same time it is also apparent
that simple methods for few-shot learning such as these do not achieve
performance close to fully supervised training. We therefore conclude that
MedIMeta offers high enough complexity for evaluating future few-shot methods,
with the fully supervised results setting an upper bar for future few-shot
methods to achieve.

## Usage Notes

All datasets contained in the MedIMeta dataset [37] can be downloaded from
Zenodo. Using the code provided in our data loaders repository [33], all tasks
in MedIMeta can easily be loaded as PyTorch datasets for single-domain, cross-
domain and few-shot scenarios. No further pre-processing is required, but it
is possible to provide any TorchVision transforms to the dataset class when
initializing it. A simple step-by-step example of how to load a single dataset
is as follows.

  1. 1.

Download the zip file for the dataset you would like to use (e.g. OCT) from
the Zenodo record [37], and extract it to a directory of choice. Let us use
./data/MedIMeta here.

  2. 2.

pip install medimeta

  3. 3.

The data can now be used as a PyTorch dataset. The following code snippet
would instantiate the dataset for the Disease task of the OCT dataset,
assuming the data is stored in the data/MedIMeta directory.

        from medimeta import MedIMeta
    dataset = MedIMeta("data/MedIMeta", "oct", "Disease")

The repository also contains a directory “examples” which contains several
usage examples for single-domain training, cross-domain training, and few-shot
fine-tuning.

## Code availability

The algorithms, methodologies, and procedures used in this paper are fully
documented in our accompanying code repositories. All code has been developed
in Python and builds on widely used libraries. Specific dependencies and their
versions are documented as requirements in the respective repositories,
ensuring easy setup and reproducibility.

MedIMeta dataset scripts:

The source code for creating datasets from their respective materials contains
all scripts needed for the creation of the 19 MedIMeta datasets, utility
functions for extending the MedIMeta with additional datasets and all
parameters used, such as image size. Code available under [36].

MedIMeta data loaders:

Easy-to-use code for data loading from MedIMeta is provided both for the
single-domain and the cross-domain (few-shot) scenarios. The code is available
at [33]. The package can be directly installed using pip install medimeta.

Experiments:

The code for our experiments includes scripts for preprocessing, data
augmentation, model training, evaluation, and other necessary utilities. It
utilizes our data loader code described above. The code is available under
[32].

TorchCross library:

A general PyTorch few-shot learning and cross-domain learning library which is
used in the MedIMeta data loaders code. The code is available under [34], and
the package can be directly installed using pip install torchcross.

## References

  * [1] Andrea Acevedo, Anna Merino, Santiago Alférez, Ángel Molina, Laura Boldú and José Rodellar  “A dataset of microscopic peripheral blood cell images for development of automatic recognition systems”  In _Data in Brief_ 30, 2020, pp. 105474  DOI: 10.1016/j.dib.2020.105474
  * [2] Luca Bertinetto, Joao F Henriques, Philip HS Torr and Andrea Vedaldi  “Meta-learning with differentiable closed-form solvers”  In _arXiv preprint arXiv:1805.08136_ , 2018
  * [3] Patrick Bilic et al.  “The liver tumor segmentation benchmark (lits)”  In _Medical Image Analysis_ 84 Elsevier, 2023, pp. 102680
  * [4] Noel Codella et al.  “Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic)”  In _arXiv preprint arXiv:1902.03368_ , 2019
  * [5] Guy Davidson and Michael C Mozer  “Sequential mastery of multiple visual tasks: Networks naturally learn to learn and forget to forget”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9282–9293
  * [6] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled and Aly Fahmy  “Dataset of breast ultrasound images”  In _Data in Brief_ 28, 2020, pp. 104863  DOI: 10.1016/j.dib.2019.104863
  * [7] Vincent Dumoulin, Neil Houlsby, Utku Evci, Xiaohua Zhai, Ross Goroshin, Sylvain Gelly and Hugo Larochelle  “Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark”  arXiv, 2021  DOI: 10.48550/ARXIV.2104.02638
  * [8] Chelsea Finn, Pieter Abbeel and Sergey Levine  “Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks”  In _Proceedings of the 34th International Conference on Machine Learning_ 70 PMLR, 2017, pp. 1126–1135
  * [9] Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing and Rogerio Feris  “A broader study of cross-domain few-shot learning”  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16_ , 2020, pp. 124–141  Springer
  * [10] Yunhui Guo, Noel C. Codella, Leonid Karlinsky, James V. Codella, John R. Smith, Kate Saenko, Tajana Rosing and Rogerio Feris  “A Broader Study of Cross-Domain Few-Shot Learning”  arXiv, 2019  DOI: 10.48550/ARXIV.1912.07200
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep Residual Learning for Image Recognition”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [12] Jakob Nikolas Kather, Niels Halama and Alexander Marx  “100,000 Histological Images Of Human Colorectal Cancer And Healthy Tissue” Type: dataset  Zenodo, 2018  DOI: 10.5281/ZENODO.1214456
  * [13] Jeremy Kawahara, Sara Daneshvar, Giuseppe Argenziano and Ghassan Hamarneh  “Seven-Point Checklist and Skin Lesion Classification Using Multitask Multimodal Neural Nets”  In _IEEE Journal of Biomedical and Health Informatics_ 23.2, 2019, pp. 538–546  DOI: 10.1109/JBHI.2018.2824327
  * [14] Daniel S. Kermany et al.  “Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning”  In _Cell_ 172.5, 2018, pp. 1122–1131.e9  DOI: 10.1016/j.cell.2018.02.010
  * [15] James Kirkpatrick et al.  “Overcoming catastrophic forgetting in neural networks”  In _Proceedings of the national academy of sciences_ 114.13 National Acad Sciences, 2017, pp. 3521–3526
  * [16] Alex Krizhevsky  “Learning multiple layers of features from tiny images”, 2009
  * [17] JR Harish Kumar, Chandra Sekhar Seelamantula, JH Gagan, Yogish S Kamath, Neetha IR Kuzhuppilly, U Vivekanand, Preeti Gupta and Shilpa Patil  “Chákṣu: A glaucoma specific fundus image database”  In _Scientific data_ 10.1 Nature Publishing Group UK London, 2023, pp. 70
  * [18] Brenden M Lake, Ruslan Salakhutdinov and Joshua B Tenenbaum  “Human-level concept learning through probabilistic program induction”  In _Science_ 350.6266 American Association for the Advancement of Science, 2015, pp. 1332–1338
  * [19] Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy and Daniel L. Rubin  “A curated mammography data set for use in computer-aided detection and diagnosis research”  In _Scientific Data_ 4.1, 2017, pp. 170177  DOI: 10.1038/sdata.2017.177
  * [20] Ruhan Liu et al.  “Deepdrid: Diabetic retinopathy—grading and image quality estimation challenge”  In _Patterns_ 3.6 Elsevier, 2022, pp. 100512
  * [21] Christian Matek, Simone Schwarz, Karsten Spiekermann and Carsten Marr  “Human-level recognition of blast cells in acute myeloid leukaemia with convolutional neural networks”  In _Nature Machine Intelligence_ 1.11, 2019, pp. 538–544  DOI: 10.1038/s42256-019-0101-9
  * [22] Boris Oreshkin, Pau Rodríguez López and Alexandre Lacoste  “Tadam: Task dependent adaptive metric for improved few-shot learning”  In _Advances in neural information processing systems_ 31, 2018
  * [23] Samiksha Pachade, Prasanna Porwal, Dhanshree Thulkar, Manesh Kokare, Girish Deshmukh, Vivek Sahasrabuddhe, Luca Giancardo, Gwenolé Quellec and Fabrice Mériaudeau  “Retinal fundus multi-disease image dataset (rfmid): A dataset for multi-disease detection research”  In _Data_ 6.2 MDPI, 2021, pp. 14
  * [24] Sylvestre-Alvise Rebuffi, Hakan Bilen and Andrea Vedaldi  “Learning multiple visual domains with residual adapters”  In _Advances in neural information processing systems_ 30, 2017
  * [25] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle and Richard S. Zemel  “Meta-Learning for Semi-Supervised Few-Shot Classification”, 2018  arXiv:1803.00676 [cs.LG]
  * [26] Olga Russakovsky et al.  “ImageNet Large Scale Visual Recognition Challenge”  In _International Journal of Computer Vision (IJCV)_ 115.3, 2015, pp. 211–252  DOI: 10.1007/s11263-015-0816-y
  * [27] Eleni Triantafillou et al.  “Meta-dataset: A dataset of datasets for learning to learn from few examples”  In _arXiv preprint arXiv:1903.03096_ , 2019
  * [28] Philipp Tschandl, Cliff Rosendahl and Harald Kittler  “The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions”  In _Scientific data_ 5.1 Nature Publishing Group, 2018, pp. 1–9
  * [29] Ihsan Ullah et al.  “Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification”  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022  URL: https://meta-album.github.io/
  * [30] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray and Daan Wierstra  “Matching Networks for One Shot Learning”  In _Advances in Neural Information Processing Systems_ 29 Curran Associates, Inc., 2016
  * [31] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri and Ronald M. Summers  “ChestX-ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017
  * [32] Stefano Woerner  “MedIMeta Experiments”, 2024  URL: https://github.com/StefanoWoerner/medimeta-experiments
  * [33] Stefano Woerner  “MedIMeta for PyTorch”, 2024  URL: https://github.com/StefanoWoerner/medimeta-pytorch
  * [34] Stefano Woerner  “TorchCross”, 2024  URL: https://github.com/StefanoWoerner/torchcross
  * [35] Stefano Woerner and Christian F. Baumgartner  “Strategies for Meta-Learning with Diverse Tasks”  In _Medical Imaging with Deep Learning_ , 2022
  * [36] Stefano Woerner and Arthur Jaques  “MedIMeta Dataset Scripts”, 2024  URL: https://github.com/StefanoWoerner/medimeta-dataset-scripts
  * [37] Stefano Woerner, Arthur Jaques and Christian F. Baumgartner  “A comprehensive and easy-to-use multi-domain multi-task medical imaging meta-dataset (MedIMeta)”, 2024  DOI: 10.5281/zenodo.7884735
  * [38] Xuanang Xu, Fugen Zhou, Bo Liu, Dongshan Fu and Xiangzhi Bai  “Efficient multiple organ localization in CT image using 3D region proposal network”  In _IEEE transactions on medical imaging_ 38.8 IEEE, 2019, pp. 1885–1898
  * [39] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister and Bingbing Ni  “MedMNIST v2-A large-scale lightweight benchmark for 2D and 3D biomedical image classification”  In _Scientific Data_ 10.1 Nature Publishing Group UK London, 2023, pp. 41
  * [40] Xiaohua Zhai et al.  “A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark”  arXiv, 2019  DOI: 10.48550/ARXIV.1910.04867

## Acknowledgements

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy – EXC number 2064/1 – Project
number 390727645. The authors thank the International Max Planck Research
School for Intelligent Systems (IMPRS-IS) for supporting Stefano Woerner.

## Author contributions statement

SW and CB conceived the dataset and the experiments, SW and AJ pre-processed
and prepared the data, SW conducted the experiments. All authors reviewed the
manuscript.

## Competing interests

The authors have no competing interests to declare.