# PyCPL: The ESO Common Pipeline Library in Python v1.0

Mrunmayi S. Deshpande,1 Nuria P. F. Lorente,1 Anthony Horton,1 Brent
Miszalski,1 Ralf Palsa,2 Lars Lundin,2 Anthony Heng,1 Aidan Farrell1

###### Abstract

PyCPL provides full access to ESO’s Common Pipeline Library (CPL) for
astronomical data reduction within a Python environment. Not only does it
offer a Python interface to the robust CPL library, but it also lets users and
developers fully utilise the rest of the scientific Python ecosystem. We have
written a C++ layer to CPL and with pybind11 (a third-party library) created a
Pythonic API to CPL. Since CPL has been around for so long, it has been
thoroughly tested and understood. In 2003 it was developed in C due to its
efficiency and speed of execution. With the community however moving away from
C/C++ programming and embracing Python for data processing tasks, there is a
need to provide access to the CPL utilities within a Python environment. With
the latest version being released users can now install PyCPL to run existing
CPL recipes (written in C) and access the results from Python. It also
provides the ability to create new recipes in Python using the functionality
provided by CPL.

1Australian Astronomical Optics, Macquarie University, Sydney, Australia;
<EMAIL_ADDRESS>
2European Southern Observatory, Garching, Germany

##  1 Introduction

###  1.1 The Common Pipeline Library

The Common Pipeline Library (CPL) is a collection of ISO-C libraries that
offer a strong, fast, and extensive software toolkit for creating astronomical
data-reduction tasks, or recipes. These can either be executed manually by the
user or initiated by an automated framework (called "pipelines").

The Common Pipeline Library (Jung et al. 2004) was created to standardise the
building process of VLT instrument pipelines, reducing the duration of their
development and simplifying their maintenance. The CPL code has been
engineered to limit external third party dependencies and make the library
portable and adaptable, and it can be used outside the VLT context for similar
applications.

CPL provides a host of functionalities, presented in a clear, generic and
uniform manner. It is used at ESO for a quick look at data processing at the
observatory. CPL is also used for data product creation from the ESO archive
facility, and monitoring of the status of VLT instruments (Ballester 2011).
Since its initial release in 2003, CPL has been an essential tool for
developing data reduction pipelines for ESO VLT instruments. It will continue
to be used as the foundation for pipelines involving ELT instruments (CPL Team
2011). CPL also forms the basis for ESO’s High-level Data Reduction Library
(HDRL) which implements more complex or specialised data processing
algorithms.

##  2 PyCPL

As the science and engineering community are moving away from the C/C++
languages and adopting Python for data processing tasks, we have created
PyCPL, a library that enables using CPL in a more Pythonic way. PyCPL offers
Python language bindings for the CPL tool kit API, including the CPL plugin
interface (CPL Team 2011). With PyCPL one can run existing CPL recipes and
access the results from Python. The user can also create new recipes in Python
by using the functionalities in CPL together with other Python libraries (e.g.
Astropy). Through PyCPL one may develop Python recipes that comply with ESO
standards and execute the recipes of currently available ESO instrument
pipeline packages from Python scripts or the interpreter prompt. A command
line program, PyEsoRex, can be used analogously with the C-based EsoRex and
execute pipeline recipes that are already developed using the CPL C API, as
well as those that are implemented using the PyCPL Python API. This
possibility of accessing CPL through Python was initially investigated for the
MUSE instrument on ESO VLT using the PYTHON-CPL module, which made it possible
to run previously produced CPL recipes (Streicher & Weilbacher 2012).

###  2.1 Bindings

Python bindings to CPL are achieved by means of an intermediary C++ layer with
the use of pybind11111 https://github.com/pybind/pybind11 , a lightweight
header-only library that exposes C++ types in Python and vice versa (Jakob et
al. 2017). The bindings provide the necessary user-facing Python interface to
our C++ layer (see Figure 1). Python provides an easy platform to build and
manage recipes, especially when it comes to changing recipe parameters for
simulations.

With the advent of Python bindings, we can improve the accessibility of CPL.
This ranges from the most basic length() and dump() functions to some more
complex functions including array based access to cpl.core.Images. In some
special cases we can even make use of Numpy masked arrays that allow us to
record invalid elements efficiently.

###  2.2 Three Layer Model

Figure 1 shows the three layer model used to create PyCPL. Even with the
addition of pybind11, straight translation from CPL to PyCPL does not occur
automatically. We have created a three-layer model that wraps the existing CPL
ISO-C code. Creating an object-oriented C++ layer is the first step. All the
CPL library’s features are included in the PyCPL libraries, divided into four
smaller modules (Horton et al. 2023):

  1. 1.

The CPL library itself, written in ISO-C in a pseudo object-oriented style
(structs with sets of associated functions). The CPL library contains C
object-like constructs.

     * •

cpl.core: Classes containing the core modules of image, mask, table, matrix,
vector, metadata, statistics, arithmetic, I/O and conversion operations for
these classes, function fitting and exceptions.

     * •

cpl.dfs: This module contains data flow system related functions for
integrating data products into the ESO Data Flow System.

     * •

cpl.drs: Classes Containing functions for data reduction software, higher
level tools for WCS, plate solving, geometric transformations, wavelength
calibration.

     * •

cpl.ui: This module consists of classes representing recipes, parameters, data
frames, and lists of data frames.

  2. 2.

The second layer consists of the C++ binding layer. This translates C structs
and functions to C++ classes and methods as an object-oriented wrapper for the
CPL C library. The C++ layer’s goal is to turn CPL structs and the functions
that go along with them like tables, properties, and images into C++ classes,
for a better integration with the pybind11 Python binding generator.

  3. 3.

The Python Interface which is the third layer, provides access to the CPL
library in Python. The C++ classes and methods are mapped to natural Python
classes and methods using the Python bindings.

Figure 1.: PyCPL architecture layout with three layers namely CPL modules,
bindings layers and python environment.

###  2.3 Testing

The ESO CPL library is robust, well-tested and has been extensively used since
2003. With PyCPL we have adopted and expanded the CPL tests, creating unit
tests for each PyCPL function. The PyCPL source repository therefore includes
a comprehensive set of unit, validation and regression tests.

##  3 PyCPL version 1.0

PyCPL version 1.0 was released on 6th November 2023. It comes with numerous
API changes as compared to the previous versions. It can be obtained from the
ESO Common Pipeline Library web page 222
https://www.eso.org/sci/software/pycpl/.

##  4 PyCPL Applications

PyCPL provides Pythonic interfaces and the advantage of utilising a well
understood, robust C library in Python. Our first application of PyCPL,
developed separately at the AAO, is an extensible data reduction framework
that uses PyCPL and PyEsoRex to manage parameters and run a user-specified
list of recipes. The regular framework provided by PyCPL substantially reduces
the amount of code required to keep track of recipe input and output products,
thus minimising maintenance and likelihood of errors. The framework currently
includes MUSE data reduction workflows, and we plan to include more pipelines
in the near future.

We encourage our community to consider using PyCPL for data reduction and
software pipeline development, and use it to prototype recipes and algorithms.

## References

  * Ballester (2011) Ballester, P. 2011, Data Flow for VLT/VLTI Instruments Deliverables Specification, Tech. Rep. VLT-SPE-ESO-19000-1618, European Southern Observatory
  * CPL Team (2011) CPL Team 2011, Common Pipeline Library User Manual, ESO VLT-MAN-ESO-19500-2720. http://www.eso.org/sci/software/cpl/
  * Horton et al. (2023) Horton, A., Heng, A., Farrell, A., Lorente, N. P. F., Palsa, R., Lundin, L., & Sealey, K. 2023, in Astronomical Data Analysis Software and Systems XXXI, vol. 534 of Astronomical Society of the Pacific Conference Series. In press
  * Jakob et al. (2017) Jakob, W., Rhinelander, J., & Moldovan, D. 2017, Pybind11 – Seamless operability between C++ 11 and Python. URL https://github.com/pybind/pybind11
  * Jung et al. (2004) Jung, Y., Ballester, P., Banse, K., Hummel, W., Izzo, C., McKay, D. J., Kiesgen, M., Lundin, L. K., Modigliani, A., Palsa, R. M., & Sabet, C. 2004, in Astronomical Data Analysis Software and Systems (ADASS) XIII, edited by F. Ochsenbein, M. G. Allen, & D. Egret, vol. 314 of Astronomical Society of the Pacific Conference Series, 764
  * Streicher & Weilbacher (2012) Streicher, O., & Weilbacher, P. M. 2012, in Astronomical Data Analysis Software and Systems XXI, edited by P. Ballester, D. Egret, & N. P. F. Lorente, vol. 461 of Astronomical Society of the Pacific Conference Series, 853