# ALBench: A Framework for Evaluating Active Learning in Object Detection

Zhanpeng Feng Lighthouse  Shiliang Zhang School of Computer Science, Peking
University  Rinyoichi Takezoe School of Computer Science, Peking University
Wenze Hu Lighthouse  Manmohan Chandraker University of California, San Diego
Li-Jia Li Stanford University  Vijay K. Narayanan ServiceNow  Xiaoyu Wang
Lighthouse

###### Abstract

Active learning is an important technology for automated machine learning
systems. In contrast to Neural Architecture Search (NAS) which aims at
automating neural network architecture design, active learning aims at
automating training data selection. It is especially critical for training a
long-tailed task, in which positive samples are sparsely distributed. Active
learning alleviates the expensive data annotation issue through incrementally
training models powered with efficient data selection. Instead of annotating
all unlabeled samples, it iteratively selects and annotates the most valuable
samples. Active learning has been popular in image classification, but has not
been fully explored in object detection. Most of current approaches on object
detection are evaluated with different settings, making it difficult to fairly
compare their performance. To facilitate the research in this field, this
paper contributes an active learning benchmark framework named as ALBench for
evaluating active learning in object detection. Developed on an automatic deep
model training system, this ALBench framework is easy-to-use, compatible with
different active learning algorithms, and ensures the same training and
testing protocols. We hope this automated benchmark system help researchers to
easily reproduce literature’s performance and have objective comparisons with
prior arts. The code will be release through
Github111https://github.com/modelai/ALBench.

##  1 Introduction

An automated machine learning system involves multiple components, including
automated model design, training data selection, _etc_. Currently, a large
body of works focus on the model design. Related works include neural
architecture search, meta-learning, and hyper-parameter optimization, _etc._
The research on the other component, _i.e._ , automated data selection has not
been as intensive. It is partially attributed to the complication of
performance evaluation. The resulting model output by a model design strategy
can be easily evaluated using standard classification/detection/segmentation
datasets, following commonly used training/testing paradigms without any
ambiguity. Differently, the data automation process could involve sampling
strategies with extra setups, such as the number of initial training samples,
the number of samples added in each iteration, when to stop, _etc._
Unfortunately, different works propose different practices which makes direct
apple-to-apple comparison difficult. It obviously does not benefit the
prosperity of the community.

Active learning is one of the most important techniques for data selection
automation. It (Kovashka et al.,, 2016; Li et al.,, 2007; Grauman and
Belongie,, 2014) has been proposed to address the challenge of expensive data
annotation encountered by AI models, especially deep models (He et al.,, 2016)
which could consume millions or even billions of samples. Instead of
annotating every sample available and training the model all at once, active
learning trains models in multiple steps. Each step tends to select and
annotate a small portion of samples, which are most valuable for subsequent
performance enhancement. Combined with incremental learning (Kading et al.,,
2016), active learning has shown great potentials in image classification,
_e.g._ , it significantly reduced the annotation cost (60%) while achieving
performance on par with that of using full data (Caramalau et al.,, 2021).
Given the fact that academic datasets used for evaluating active learning have
been dedicatedly prepared with sample selection and noise filtering, the
improvement from active learning could be even more substantial for open world
scenarios, which commonly contain more noises and redundancies.

Method |  Training Set | Initial Size | Added Size | Stop Size | Testset
---|---|---|---|---|---
Aghdam et al., (2019) |  Caltech Pedestrian, CityPersons, BDD100K | 500 | 500 | 7.5K | Caltech Pedestrian testset
Haussmann et al., (2020) |  self collected | 100K | 200K | 700K | self collected
Yoo and Kweon, (2019) |  VOC07, VOC12 | 1K | 1K | 10K | VOC07 testset
|  VOC07, VOC12 | 1K | 1K | 10K | VOC07 testset
Choi et al., (2021) |  VOC07 | 2K | 1K | 4K | VOC07 testset
|  COCO14 | 5k | 1k | 7k | COCO17 VAL
|  VOC12 | 500 | 200 | 3.5K | VOC12 testset
Kao et al., (2018) |  VOC07 | 500 | 200 | 3.5K | VOC07 testset
|  COCO14 | 5K | 1K | 9K | COCO14 val
Feng et al., (2019) |  KITTI | 1K | 200 | 12K | KITTI self-divided
Brust et al., (2018) |  VOC12 | 50 | 50 | 250 | VOC12 self-divided
|  VOC07 | 500 | 500 | 3.5K | VOC07 testset
Yu et al., (2021) |  VOC12 | 500 | 500 | 3.5K | VOC12 testset
|  COCO14 | 5K | 1K | 9K | COCO14 VAL
Roy et al., (2018) |  VOC07, VOC12 | 1655 | 827 | 7447 | VOC07 testset
Yuan et al., (2021) |  VOC07, VOC12 | 827 | 413 | 3.31K | VOC07 testset
|  COCO14 | 2340 | 2340 | 11.7K | COCO14 VAL
Table 1: Training and testing setups utilized in existing active learning
works for object detection, where “Added Size” denotes the number of selected
samples at each iteration. The training stops when the size of augmented
training set reaches the “Stop Size”. It shows that current methods employ
various benchmark setups which prevents objective comparison among them.

Research on active learning for object detection has not been as popular as
that for image classification. This is partially because it involves more
training setups than traditional learning strategies, making the algorithm
implementation, repeat, and comparison difficult. Besides that, existing
active learning algorithms for detection follow different evaluation
paradigms, making their performance not directly comparable. We summarize the
training and test setups in existing works in Table 1. The listed studies
could use:

  * •

different datasets. For instance, the algorithm proposed by Yoo and Kweon,
(2019) is only evaluated on the PASCAL VOC (Everingham et al.,, 2010) dataset,
and is not tested on the more challenging COCO dataset (Lin et al.,, 2014).
Some approaches are tested on autonomous driving datasets, rather than generic
object detection datasets.

  * •

different initialization settings. Both Yoo and Kweon, (2019) and Kao et al.,
(2018) perform experiments on the PASCAL VOC dataset, but use different number
of samples in their initial training sets, respectively.

  * •

different sampling settings for each iteration. The method (Haussmann et al.,,
2020) selects 200k images at each training step, substantially more than those
determined by other approaches (Yoo and Kweon,, 2019). Introducing more
samples at each step helps to boost the model performance, but leads to higher
annotation cost.

  * •

different stop criteria. Both (Yuan et al.,, 2021) and (Yu et al.,, 2021) are
tested on the MS COCO dataset, but those two methods end up with 11.7K and 9K
training samples, respectively. Stopping with more data definitely benefits
the final performance.

Figure 1: Illustration of active learning procedure implemented by ALBench
based on YMIR (Huang et al.,, 2021), where “AL” denotes the active learning
model provided by authors. “Train” denotes the training action using the
augmented training data, each training action typically produces a model. The
automated evaluation system collects all the models obtained from training
actions in each iteration and produces performance profiling for the
corresponding model (Perf0, Perf1, _etc._). The comprehensive profiling
results are used to compare with those of other methods, and generate the
leaderboard accordingly. The whole process is automatic once the author
provides the training and active learning code for a single iteration.

As the performance of active learning is closely related to both training and
test settings, those observations motivated us to construct a benchmark
framework for the community that is easy to reproduce and fairly compare the
performance of different algorithms. To this end, we contribute a public
benchmark framework named ALBench constructed based on our automated deep
model training system called YMIR (Huang et al.,, 2021). YMIR could
automatically train deep models on given training samples. Based on YMIR,
ALBench implements an incremental training pipeline, which first trains
initial detection models on a small annotated training set, then iteratively
selects samples using active learning models. Selected samples at each step
are adopted for updating deep models and active learning models with YMIR. As
illustrated in Fig. 1, this procedure is iteratively repeated by ALBench until
the stop criterion of active learning is met.

ALBench is a plug-and-play framework. The compatibility of various active
learning algorithms with ALBench is implemented through docker images which
package active learning models to fit into our training and mining framework.
Within ALBench, different active learning algorithms share the same training
setup, _e.g._ , training dataset, number of selected samples at each
iteration,_etc._ The subsequent testing stage tests different active learning
algorithms with the same setup. We illustrate the training and testing
procedures in Fig. 2, which effectively ensure an objective comparison among
different active learning algorithms. Moreover, it is straightforward to study
the model behavior using different settings once the related training and
active learning docker images are provided. Performance on a setting can be
obtained even the original work did not explore the corresponding study. We
believe those characteristics of the proposed system will facilitate the
research for the active learning community.

We tested ALBench with its default detector and dataset. Different active
learning algorithms including baseline random sampling, entropy Roy et al.,
(2018), ALDD Aghdam et al., (2019), and CALD Yu et al., (2021) are compared.
Besides showing the general comparison, ALBench also illustrates the per-
category performance, where many interesting observations can be observed.
ALBench is a flexible and allows to involve more testing setups, detectors,
and active learning algorithms. It will be publicly released to facility the
research community. To the best of our knowledge, this is an original
contribution on the benchmark of active learning in object detection.

##  2 Background: Active Learning for Object Detection

Different active learning algorithms share similar objective of selecting the
most valuable training samples. Some algorithms has been adopted to train deep
models for image classification. As shown in previous literature (Caramalau et
al.,, 2021), iteratively selecting training samples saves considerable
annotation cost. Object detection is another important vision task, and also
suffers from the difficulty of data annotation. Existing active learning works
for object detection propose different criteria to select samples. For
instance, (Roy et al.,, 2018) proposes a query by committee paradigm and uses
the disagreement among convolution layers in the detector backbone to select
images. (Brust et al.,, 2018) evaluates different aggregation metrics for
uncertainty-based active learning methods. (Kao et al.,, 2018) first considers
both the classification and localization results of the unlabeled images for
sample selection by defining localization tightness and localization
stability. (Aghdam et al.,, 2019) calculates the uncertainty of background
pixels to select informative samples. (Yu et al.,, 2021) leverages a
consistency-based metric to consider the information of box regression and
classification simultaneously. (Yuan et al.,, 2021) treats unlabeled images as
instance bags and estimates image uncertainty by re-weighting instances in a
multiple instance learning fashion. (Haussmann et al.,, 2020) studies the
scalability of active learning, and builds a scalable production system for
autonomous driving.

A fair comparison among those algorithms is critical for future studies. As
illustrated in Table 1, current works apply different setups, making a fair
comparison difficult. One of related works (Aghdam et al.,, 2019) focuses on
the pedestrian detection task, and is not evaluated on general object
detection datasets like COCO (Lin et al.,, 2014) and VOC (Everingham et al.,,
2010). Similarly, (Feng et al.,, 2019) focuses on the object detection in
autonomous driving scenarios. Existing approaches also select different
numbers of samples at each iteration. For instance, (Haussmann et al.,, 2020)
uses 100k images as the initial training set, and selects 200k images at each
iteration. Another work (Brust et al.,, 2018) trains the YOLO (Redmon et al.,,
2016) detector and considers the category imbalance issue during training. It
only selects 50 images at each iteration. Annotating 200k images at each
iteration is expensive for real applications. Selecting too few samples
degrades the speed of performance enhancement, leading to a time-consuming
training process. Therefore, it not reasonable to select either too many or
too few samples for active learning.

Existing approaches also follow different testing setups and some approaches
suggest contradictory conclusions. Some approaches like (Yoo and Kweon,, 2019)
only report their performance on the VOC dataset and are not evaluated on the
more challenging COCO. A recent work (Choi et al.,, 2021) trains the SSD (Liu
et al.,, 2016) detector and uses both COCO and VOC in experiments. However, it
does not report iteration details on COCO and the performance is inferior to
commonly used ensemble and dropout methods. Kao et al., (2018) and Yu et al.,
(2021) provide comprehensive studies on VOC and COCO. Kao et al., (2018) use
random selection as the baseline and the proposed method is only applicable to
2-stage detectors. Both RetinaNet (Lin et al.,, 2017) and FasterRCNN (Ren et
al.,, 2015) are trained with active learning algorithms proposed by (Yu et
al.,, 2021). Experiments in (Yu et al.,, 2021) show that the LS/C method
proposed in (Kao et al.,, 2018) is not as good as the random selection
baseline, which contradicts with the conclusions in (Kao et al.,, 2018).

To summarize, current active learning algorithms for object detection follow
different training and testing setups. Some works indicate that, baseline
algorithms like random selection perform similarly with more complicated
algorithms. It is also interesting to find that, some works draw contradictory
conclusions. Those phenomena could be attributed to multiple reasons. Most
academic datasets contain high quality samples thanks to their manual
annotation and pre-selection procedure, making randomly selected samples
present reasonably good quality. Contradictory conclusions can be attributed
to different testing and training setups. It is important to build a benchmark
platform to 1) Simplify the process to benchmark a method, 2) provide unified
training and testing setups, and 3) provide comprehensive evaluations, _e.g._
, show the performance on training and testing sets with different qualities
and difficulties.

Figure 2: Illustration about how the authors interact with ALBench as well as
detailed process inside ALBench performing the benchmark task. Authors’ Code:
Implementation provided by authors. Training with Active Learning: Model
training powered by YMIR with the provided model training code and the active
learning code. Evaluation: Performance evaluation for the resulting model as
well as all intermediate models. Authors could dismiss the "Training Docker
Image" part if they employ a standard training methodology that has already
been adopted by YMIR. The brown arrows show the looping training cycle.

##  3 Benchmark Framework Design

This part proceeds to introduce the design and details of ALBench. An overview
of ALBench is first presented, followed by detailed introductions on
individual components.

###  3.1 Overview

Fig. 2 presents the framework of ALBench, which is an automatic training and
evaluation system requiring only a few necessary running interfaces from the
authors. In ALBench, different active learning algorithms can stick to their
own optimal settings and hyper-parameters to conduct sample selection. ALBench
incrementally trains object detection models with augmented training sets
obtained from the active learning process, and compares their object detection
performance. It provides standard setups for subsequent detection model
training and testing to ensure an objective comparison. The final results will
be reported on the leaderboard.

ALBench is constructed based on the automated deep model training system
called YMIR (Huang et al.,, 2021). YMIR is an open source platform to support
the rapid training of deep models at scale. It puts the efficient data
development at the center of the machine learning development process,
integrates active learning methods, data and model version control, and uses
concepts such as projects to enable fast iterations of multiple task specific
datasets in parallel. It abstracts the development process into core states
and operations, and integrates third party tools via open APIs as
implementations of the operations. More details of YMIR can be found in (Huang
et al.,, 2021).

As illustrated in Fig. 2, ALBench is composed of three components including
the Input Interface, Training with Active Learning, and Evaluation,
respectively.

###  3.2 Open Interface

The open interface (right blue box in Fig. 2) defines how the users interact
with the benchmark system. It defines protocols for packaging active learning
algorithms into AL Docker Images, which are hence hosted by YMIR to run sample
selection. Without providing detector training code, ALBench has the
capability of training the detector with default deep learning models.
However, some active learning methodologies are highly coupled with specific
detector architectures. To evaluate those active learning algorithms, ALBench
allows the users to provide their own object detection training code through a
docker image. The docker image will replace the standard training methodology
in YMIR to train object detectors and update active learning models. It is
fairly simple for using the benchmark system.

Figure 3: Illustration of the Training with Active Learning module of ALBench.

###  3.3 Training with Active Learning

The Training with Active Learning module takes docker images as input,
iteratively runs sample selection, and outputs the augmented training set
after each active learning iteration. With docker images as input, the
Training with Active Learning module iteratively conducts following steps on a
dataset:

  * •

Determine new training samples with active learning algorithms

  * •

Update object detection models on the augmented training set

  * •

Update active learning models with corresponding updated detectors and the
provided AL algorithm.

ALBench iteratively repeats above steps till the predefined stop criteria were
met. Fig. 3 illustrates the computational flow chart of the Training with
Active Learning module. The training docker image and AL docker image are
hosted by the YMIR system. The training docker image is instantiated to train
detection models and active learning models. Note that some active learning
method requires a training process while others do not. Thus the active
learning model training part does not have to be necessary. The AL docker
image is instantiated to perform sample selection which gives each input
sample a ranking probability for subsequent sample selection. The sampling
procedure is implemented by YMIR, simply choosing the top K samples. Different
AL algorithms can stick to their optimal settings by packing their code and
hyper-parameters in the docker image. ALBench has no dependency on these
settings. Meanwhile, all AL methods share identical training setups such as
initial training data, number of samples to output for each iteration, stop
criteria, _etc._ , to ensure an objective performance comparison.

###  3.4 Evaluation

Fig. 4 shows the detailed structure of the Evaluation module, which provides a
unified testing setup for different active learning algorithms. After each
training iteration, the Training with Active Learning module outputs the
updated training set and updated object detection models to the Evaluation
module. The Evaluation module proceeds to evaluate the active learning
algorithm by comparing the performance of object detectors trained using their
selected samples. To fairly compare the quality of their selected samples, the
Evaluation module will also train default object detectors with intermediate
augmented training sets in each iteration. Some active learning algorithms are
coupled with specifically designed detectors and training algorithms. Besides
training those detectors with standard methodology provided by ALBench, the
Evaluation module also trains them with their specific training algorithms
provided by users. In other words, some detectors are trained by both commonly
followed methodology and their specific ones to distinguish the contribution
from sample selection with active learning from detectors themselves.

The performance evaluation is conducted after each active learning iteration
and is reported on a leaderboard as shown in Fig. 5. Note that, ALBench allows
to flexibly add new testing setups and state-of-the art object detectors.
Different testing setups and detectors can be adjusted or added through a user
interface.

Figure 4: Illustration of the detailed structure of the automated evaluation
module of ALBench.
Figure 5: Illustration of the leaderboard.

##  4 Benchmark Showcase

This section compares different active learning algorithms for object
detection with ALBench. It first introduces training and testing setups, then
shows the performance comparison, followed by discussions.

###  4.1 Default Setups

With the automated training pipeline, training and testing setups can be
flexibly defined in ALBench, _e.g_., to include new datasets and train new
detectors. This paper simply tests ALBench with default setups to show its
validity.

Datasets: ALBench uses two widely used object detection datasets Pascal VOC
2007 and VOC 2012 (Everingham et al.,, 2010) as the default training set. For
VOC2007, we use its trainval set for training and sample mining, and test on
its testset. On VOC2012, we use its train set for training and sample mining,
and test on the validation set. We randomly select 1000 images as the initial
training set, and sample 1000 images in each iteration. The training stops
after 4 active learning iterations.

Algorithms: ALBench adopts the SSD300 (Liu et al.,, 2016) implemented with
Vgg16 (Simonyan and Zisserman,, 2015) as the default object detection model.
Codes for training are acquired from the mxnet model zoo. Each active learning
iteration trains for 100 epochs and picks the best model with highest mAP on
the testset for sample selection. Default active learning algorithms include
two commonly used baselines, _i.e_., Entropy (Roy et al.,, 2018) and Random
Selection (Choi et al.,, 2021), and two recent approaches ALDD Aghdam et al.,
(2019) and CALD Yu et al., (2021), respectively.

Dataset | Method | baseline | iter1 | iter2 | iter3 | iter4
---|---|---|---|---|---|---
VOC07 | SSD+ALDD | 52.50 | (+2.29) 54.79 | (+5.94) 60.73 | (+4.07) 64.80 | (+2.13) 66.93
SSD+CALD | 52.50 | (+7.47) 59.97 | (+4.19) 64.16 | (+2.78) 66.94 | (+1.56) 68.50
SSD+RANDOM | 52.50 | (+4.28) 56.78 | (+5.33) 62.11 | (+2.67) 64.78 | (+1.70) 66.48
SSD+ENTROPY | 52.50 | (+3.85) 56.35 | (+5.22) 61.57 | (+3.91) 65.48 | (+1.09) 66.57
VOC12 | SSD+ALDD | 48.29 | (+3.96) 52.25 | (+5.20) 57.45 | (+2.84) 60.29 | (+2.19) 62.48
SSD+CALD | 48.29 | (+7.64) 55.93 | (+2.28) 58.21 | (+3.36) 61.57 | (+0.94) 62.51
SSD+RANDOM | 48.29 | (+5.19) 53.48 | (+4.53) 58.01 | (+2.35) 60.36 | (+1.73) 62.09
SSD+ENTROPY | 48.29 | (+5.68) 53.97 | (+3.50) 57.47 | (+3.18) 60.65 | (+1.80) 62.24
Table 2: Performance of AL methods evaluated through our system

###  4.2 Results and Discussions

Table 2 presents the performance comparison among default active learning
algorithms in ALBench. The recent CALD outperforms ALDD and baselines on VOC07
and VOC12. At iter1, CLAD substantial outperforms others. As more training
samples are selected, those AL algorithms start to show similar performance.
This is because the training sets of VOC07 and VOC12 only contain about 5K and
5.7K images, respectively, which minish the importance and contribution of AL
algorithms as more samples are selected. Another interesting observation is
that ALDD does not perform as good as simple baseline AL algorithms at iter1
and iter2. This could be because VOC training set generally contains high
quality samples, making random selection achieves reasonable good performance
as claimed in many studies Yu et al., (2021).

ALBench could also show a detailed per-category performance comparison as
illustrated in Fig. 6, where CALD does not always perform the best. For
example, on the horse, cat, and sheet categories of VOC12 as well as the dog
category of VOC07, CALD performs worse than random selection. In Fig. 6, many
AL algorithms are not stable, _e.g._ , substantial performance degradation
occurs for CALD and ALDD as more samples are selected on many categories. Fig.
6 also indicates that, low baseline detector performance does not
substantially degrade the effectiveness of subsequent sample selection on
VOC07 and VOC12. For instance, the baseline detector only achieves about 26%
AP on chair category. AL algorithms effectively boost the AP by about 16% on
both datasets. This could also be attributed to the fact that VOC datasets
have been dedicatedly prepared with sample selection and noise filtering,
which simplifies high quality sample selection. Using realistic datasets
containing larger portions of noises, redundancies, and long-tailed
catregories has potential to substantially evaluate the value and performance
of active learning algorithms.

Figure 6: Illustration of a detailed per-category performance comparison.

##  5 Limitation and Broader Impact

ALBench currently focuses on object detection task. Further work will be
conducted to extend ALBench to more vision and machine learning tasks. Besides
default setups, more active learning algorithms, larger and realistic training
and testing sets will be included in ALBench to make more comprehensive
comparisons. As the first open source benchmark platform for active learning,
ALBench is expected to bring broad impact to the field of active learning.

##  6 Conclusions

This paper proposes an active learning benchmark framework named as ALBench to
simplify the implementation and comparison of active learning algorithms in
object detection. ALBench is powered by an automatic deep model training
system, which applies unified training and testing setups for different active
learning algorithms. ALBench framework is easy-to-use, compatible with
different active learning algorithms, and ensures their objective comparison.
ALBench also allows to flexibly add more training and testing setups, and
detectors. As an original contribution on the benchmark of active learning,
ALBench is expected to benefit the active learning community.

## References

  * Aghdam et al., (2019) Aghdam, H. H., Gonzales-Garcia, A., Lopez, A. M., and Weijer, J. v. d. (2019).  Active learning for deep detection neural networks.  In ICCV.
  * Brust et al., (2018) Brust, C. A., Kading, C., and Denzler, J. (2018).  Active learning for deep object detection.  arXiv preprint arXiv:1809.09875.
  * Caramalau et al., (2021) Caramalau, R., Bhattarai, B., and Kim, T. K. (2021).  Sequential graph convolutional network for active learning.  In CVPR.
  * Choi et al., (2021) Choi, J., Elezi, I., Lee, H. J., Farabet, C., and Alvarez, J. M. (2021).  Active learning for deep object detection via probabilistic modeling.  In CVPR.
  * Everingham et al., (2010) Everingham, M., L., V. G., Williams, C. K. I., Winn, J., and Zisserman, A. (2010).  The pascal visual object classes (voc) challenge.  International Journal of Computer Vision, 88(2):303–338.
  * Feng et al., (2019) Feng, D., Wei, X., Rosenbaum, L., Maki, A., and Dietmayer, K. (2019).  Deep active learning for efficient training of a lidar 3d object detector.  In IEEE Intelligent Vehicles Symposium.
  * Grauman and Belongie, (2014) Grauman, K. and Belongie, S. (2014).  Editorial: Special issue on active and interactive methods in computer vision.  IJCV.
  * Haussmann et al., (2020) Haussmann, E., Fenzi, M., Chitta, K., Ivanecky, J., Xu, H., Roy, D., Mittel, A., Koumchatzky, N., Farabet, C., and M. Alvarez, J. (2020).  Scalable active learning for object detection.  arXiv preprint arXiv:2004.04699.
  * He et al., (2016) He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep residual learning for image recognition.  In CVPR.
  * Huang et al., (2021) Huang, P. X., Hu, W., Brendel, W., Manmohan, C., Li, L.-J., and Wang, X. (2021).  Ymir: A rapid data-centric development platform for vision applications.  In Proceedings of the Data-Centric AI Workshop at NeurIPS.
  * Kading et al., (2016) Kading, C., Rodner, E., Freytag, A., and Denzler, J. (2016).  Fine-tuning deep neural networks in continuous learning scenarios.  In ACCV-WS.
  * Kao et al., (2018) Kao, C. C., Lee, T. Y., Sen, P., and Liu, M. Y. (2018).  Localization-aware active learning for object detection.  In ACCV.
  * Kovashka et al., (2016) Kovashka, A., Russakovsky, O., Fei-Fei, L., and Grauman, K. (2016).  Crowdsourcing in computer vision.  Foundations and Trends in Computer Graphics and Vision.
  * Li et al., (2007) Li, L., Wang, G., and Li, F. (2007).  Optimol: automatic online picture collection via incremental model learning.  CVPR.
  * Lin et al., (2017) Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollar, P. (2017).  Focal loss for dense object detection.  In ICCV.
  * Lin et al., (2014) Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014).  Microsoft coco: Common objects in context.  In ECCV.
  * Liu et al., (2016) Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A. C. (2016).  Ssd: Single shot multibox detector.  In ECCV.
  * Redmon et al., (2016) Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016).  You only look once: Unified, real-time object detection.  In CVPR.
  * Ren et al., (2015) Ren, S., He, K., Girshick, R., and Sun, J. (2015).  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NeurIPS.
  * Roy et al., (2018) Roy, S., Unmesh, A., and Namboodiri, V. P. (2018).  Deep active learning for object detection.  In BMVC.
  * Simonyan and Zisserman, (2015) Simonyan, K. and Zisserman, A. (2015).  Very deep convolutional networks for large-scale image recognition.  In ICLR.
  * Yoo and Kweon, (2019) Yoo, D. and Kweon, I. S. (2019).  Learning loss for active learning.  In CVPR.
  * Yu et al., (2021) Yu, W., Zhu, S., Yang, T., Chen, C., and Liu, M. (2021).  Consistency-based active learning for object detection.  arXiv preprint arXiv:2103.10374.
  * Yuan et al., (2021) Yuan, T., Wan, F., Fu, M., Liu, J., Xu, S., Ji, X., and Ye, Q. (2021).  Multiple instance active learning for object detection.  In CVPR.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

ICPM 2023: ICPM Doctoral Consortium and Demo Track 2023, October 23–27, 2023,
Rome, Italy

[orcid=000-0003-1350-6732<EMAIL_ADDRESS>url=https://istvank.eu, ] [1]

[orcid=0000-0001-8954-4925<EMAIL_ADDRESS>url=https://niklasadams.com/, ]

[orcid=0000-0002-3279-4795<EMAIL_ADDRESS>url=https://www.alessandroberti.it/, ]

[orcid=0000-0002-0955-6940<EMAIL_ADDRESS>url=https://vdaalst.com/, ]

[1]Corresponding author.

# OCEL 2.0 Resources – www.ocel-standard.org

István Koren  Chair of Process and Data Science, RWTH Aachen University,
Germany  Jan Niklas Adams  Alessandro Berti  Wil M.P. van der Aalst

(2023)

###### Abstract

Process mining has become a cornerstone of process analysis and improvement
over the last few years. With the widespread adoption of process mining tools
and libraries, the limitations of traditional process mining to deal with
event data with multiple case identifiers, i.e., object-centric event data,
have become apparent. As a response, the subfield of object-centric process
mining has formed, including a file format standardization attempt in the form
of OCEL 1.0, unifying the insights of previous developments in capturing
object-centric event data. However, discussions among researchers and
practitioners have shown that the proposed OCEL 1.0 standard does not go far
enough. OCEL 2.0 has been proposed as an advanced refinement, including
normative and explicit object-to-object relationships, qualifiers for object-
to-object and event-to-object relationships, and evolving object attribute
values. This demonstration presents the OCEL 2.0 website available under the
URL https://www.ocel-standard.org as a one-stop shop for the detailed
specification, example event logs, and broad tool support to facilitate the
adoption of the format.

###### keywords:

Process Mining Object-Centric Event Logs OCEL 2.0 Simulation Models Developer
Support

##  1 Introduction

Process mining has permeated many organizations over the last few years. A
wide range of dedicated tools and functionalities within off-the-shelf
software systems can perform process analysis. Although process mining can
yield powerful insights, there are currently some limiting factors. Often,
only case-based analysis is offered. Therefore, for each new perspective on a
process or related processes, new extractions and transformations from the
information systems have to be performed. Traditional event logs stored in the
XES standard format can only store and provide case-based event data. Object-
centric process mining addresses this limitation [1, 2], broadening the
horizon of many enterprises to look at intertwined processes within and across
their business units and organizations. The OCEL (1.0) format [3] has been
initially introduced as a standardization attempt to store and exchange
object-centric event data. However, industry and academia have found that this
format is not expressive enough and should be moved closer to reality, e.g.,
recording object changes and giving detailed information about object
relationships [4]. These limitations are the motivation for the OCEL 2.0
standard, which refines the original format introducing _object-to-object
relationships_ , _evolving object attribute values_ , and qualifiers for
_object-to-object_ and _event-to-object relationships_ altogether with an
increased interpretability of its specifications while maintaining backward
compatibility to OCEL 1.0 and graph-based object-centric storage formats. In
this paper, we specifically introduce the OCEL 2.0 website, a one-stop shop
for the OCEL 2.0 standard and its resources, featuring the specification of
the standard, multiple simulation models, event logs, and tool support to
immediately benefit from the new constructs. This comprehensive set of
training materials is collected on the website https://www.ocel-standard.org.

##  2 Website

The website serves as an exhaustive resource for both researchers and
practitioners. It provides access to the latest specification along with a
scientific formalization of the standard, and a number of event logs.
Additionally, it offers an array of libraries and examples to get started.

###  2.1 Specification

Figure 1 shows a screenshot of the website and the OCEL 2.0 metamodel. The
greatest strength of the format is that various relationships and their
qualifiers are now directly incorporated. Events now have a qualified
relationship to objects, which means that the nature of the relationship is
explicitly annotated. Similarly, object-to-object relationships can be
explicitly stored and supplemented with a qualifier. Both reinforce the
benefit of flexibility by storing various intertwined processes without having
to refer back to source data. In addition, changes to object attribute values
can now be tracked. We provide three file storage formats. The _XML_ variant
is human-readable and structured according to industry XML best practices. The
_JSON_ variant features a lightweight structure that can be natively used in
web environments. The _SQLite_ variant allows for efficient SQL querying,
eliminating the necessity to parse the entire event log.

Figure 1: The OCEL 2.0 website and metamodel.

###  2.2 Event Logs

The website initially provides four new event logs that target and demonstrate
new features of OCEL 2.0, available on the research data platform Zenodo.

The Logistics [5] log describes the processes of a company selling goods
overseas. After receiving an order, the shipment of the goods is scheduled.
According to the schedule, the goods are picked up from the local production
site and brought to a terminal, where a service provider receives and ships
them.

The Order Management [6] simulation and event log deals with an imaginary
e-commerce enterprise. It comprises both the registration and payment of
incoming orders, as well as the process of packing and shipping them. For
these tasks, the company deploys staff in their sales, warehousing, and
shipment departments.

The Procure-to-Pay [7] log is a simulated event log that has been replicated
with domain knowledge of a real SAP system. It handles scenarios such as
lengthy approvals and duplicate payments. Resource availability is
realistically mapped based on business hours, and the capacity of human
resources.

The Angular GitHub Commits [8] is a real-world event log containing
information on the commits in the GitHub repository of the Angular project.
Additionally, we converted some event logs that have already been made
available for OCEL 1.0.

###  2.3 Tool Support

Several tools and libraries have been developed to accommodate the OCEL 2.0
format:

  * •

_OC $\pi$_ [9]: A standalone tool for object-centric process discovery,
variant analysis, and filtering. OC$\pi$ supports object-centric variants and
filters based on frequency of behavior across object types.

  * •

_Object-Centric Process Mining (OCPM)_ [10]: A JavaScript web application that
provides capabilities such as process discovery and visualization of object-
centric models.

  * •

_Ocelot.pm_ : A web-based event log inspector for OCEL 2.0 files. It
visualizes event and object types as models and shows events and objects in
searchable data tables.

  * •

_Oracle EBS Connector_ : A transformer connecting to Oracle databases,
targeting tables related to the Purchase-to-Pay process, and facilitating OCEL
2.0 export.

  * •

_Celonis OCDM Connector_ : A tool designed for the Celonis platform, enabling
import/export of object-centric event logs in OCEL 2.0 to the Celonis _Object-
Centric Data Model_.

  * •

ocpa _library_ [11]: Focuses on processing object-centric event logs, process
discovery, model evaluation, performance analysis, and predictive monitoring.

  * •

_pm4js library_ : For JavaScript environments, it supports tasks like log
flattening, statistics generation, process discovery, and more.

  * •

_pm4py library_ [12]: A prominent Python library in process mining that has
incorporated OCEL 2.0 support, compatible with the XML, JSON, and relational
storage formats.

##  3 Conclusion

Using object-centric process mining, analysts can get a holistic overview of
relationships between different processes and their objects. The new OCEL 2.0
specification overcomes the limitations of the OCEL 1.0 format by providing
qualifiers, object-to-object relationships, and changes to object attribute
values. Our website builds a comprehensive one-stop shop with all the content
necessary to get started implementing connectors and custom process mining
analyses. In the future, we plan to extend the website with further content,
such as additional event logs and tools/libraries. We are open to external
contributions, particularly those that include real-life event logs.

###### Acknowledgements.

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy - EXC-2023 Internet of
Production - 390621612. We also thank the Alexander von Humboldt (AvH)
Stiftung for supporting our research.

## References

  * van der Aalst [2019] W. M. P. van der Aalst,  Object-Centric Process Mining: Dealing with Divergence and Convergence in Event Data,  in: SEFM, Springer, 2019, pp. 3–25. doi:10.1007/978-3-030-30446-1_1.
  * Fahland [2022] D. Fahland,  Process Mining over Multiple Behavioral Dimensions with Event Knowledge Graphs,  in: Process Mining Handbook, Springer, 2022, pp. 274–319. doi:10.1007/978-3-031-08848-3_9.
  * Ghahfarokhi et al. [2020] A. Ghahfarokhi, G. Park, A. Berti, W. M. P. van der Aalst,  OCEL Standard (2020). URL: https://www.ocel-standard.org/1.0/specification.pdf.
  * Wynn et al. [2021] M. T. Wynn, J. Lebherz, W. M. P. van der Aalst, R. Accorsi, C. D. Ciccio, L. Jayarathna, H. M. W. Verbeek,  Rethinking the Input for Process Mining: Insights from the XES Survey and Workshop,  in: ICPM Workshops, Springer, 2021, pp. 3–16. doi:10.1007/978-3-030-98581-3_1.
  * Knopp and Graves [2023] B. Knopp, N. Graves, Container Logistics Object-Centric Event Log, 2023\. doi:10.5281/ZENODO.8289899.
  * Knopp and van der Aalst [2023] B. Knopp, W. M. P. van der Aalst, Order Management Object-Centric Event Log in OCEL 2.0 standard, 2023. doi:10.5281/zenodo.8337463.
  * Park and Tacke Genannt Unterberg [2023] G. Park, L. Tacke Genannt Unterberg, Procure-to-Payment (P2P) Object-Centric event log in OCEL 2.0 Standard, 2023. doi:10.5281/zenodo.8412919.
  * Pegoraro [2023] M. Pegoraro, Angular GitHub Commits Object-Centric Event Log, 2023. doi:10.5281/zenodo.8430331.
  * Adams and van der Aalst [2022] J. N. Adams, W. M. P. van der Aalst,  OC($\pi$): Object-centric Process Insights,  in: L. Bernardinello, L. Petrucci (Eds.), Application and Theory of Petri Nets and Concurrency - 43rd International Conference, PETRI NETS 2022, Bergen, Norway, June 19-24, 2022, Proceedings, volume 13288 of Lecture notes in computer science, Springer, 2022, pp. 139–150. doi:10.1007/978-3-031-06653-5_8.
  * Berti and van der Aalst [2023] A. Berti, W. M. P. van der Aalst,  OC-PM: analyzing object-centric event logs and process models,  Int. J. Softw. Tools Technol. Transf. 25 (2023) 1–17. doi:10.1007/s10009-022-00668-w.
  * Adams et al. [2022] J. N. Adams, G. Park, W. M. P. van der Aalst,  ocpa: A Python library for object-centric process analysis,  Softw. Impacts 14 (2022) 100438. doi:10.1016/j.simpa.2022.100438.
  * Berti et al. [2023] A. Berti, S. van Zelst, D. Schuster,  PM4Py: A process mining library for Python,  Software Impacts 17 (2023) 100556. doi:10.1016/j.simpa.2023.100556.