openeo-processes-dask is a collection of Python implementations of OpenEO processes based on the xarray/dask ecosystem. It is intended to be used alongside with openeo-pg-parser-networkx, which handles the parsing and execution of OpenEO process graphs. There you'll also find a tutorial on how to register process implementations from an arbitrary source (e.g. this repo) to the registry of available processes.
If you already have GDAL installed on your system (or in a conda/micromamba env), always install the matching Python bindings first, then this package:
Install this project via pip:
pip install "gdal==$(gdal-config --version)" openeo-processes-daskNote that by default this only installs the JSON process specs.
In order to install the actual implementations, add the implementations extra:
pip install "gdal==$(gdal-config --version)" openeo-processes-dask[implementations]This ensures that the Python bindings link against your system GDAL libraries and avoids pip pulling a mismatched GDAL wheel.
System packages (Ubuntu/Debian):
sudo apt-get install gdal-bin libgdal-dev python3-gdal
pip install "gdal==$(gdal-config --version)" openeo-processes-dask[implementations]Conda (recommended for most users):
conda create -n openeo_processes_dask -c conda-forge python=3.12 gdal
conda activate openeo_processes_dask
pip install openeo-processes-dask[implementations]Micromamba (lightweight alternative to conda):
micromamba create -n openeo_processes_dask -c conda-forge python=3.12 gdal
micromamba activate openeo_processes_dask
pip install openeo-processes-dask[implementations]A subset of process implementations with heavy or unstable dependencies are hidden behind these extras:
-
ML processes:
pip install openeo-processes-dask[ml]
-
Experimental processes:
pip install openeo-processes-dask[experimental]
implementations, ml, experimental) may trigger installation of packages that depend on GDAL.
To avoid version conflicts, make sure you have installed GDAL first (via conda/micromamba or system packages) and then install the extras as shown above.
openeo-processes-dask requires poetry >1.2, see their docs for installation instructions.
Clone the repository with --recurse-submodules to also fetch the process specs:
git clone --recurse-submodules [email protected]:Open-EO/openeo-processes-dask.gitTo setup the python venv and install this project into it run:
poetry install --all-extraspoetry install --all-extras, Poetry will attempt to install GDAL via pip, which may pull the latest GDAL wheels and cause conflicts with system libraries.
It is strongly recommended to create a conda/micromamba environment with GDAL preinstalled before running poetry install. For example:
conda create -n openeo_processes_dask_dev -c conda-forge python=3.12 gdal
conda activate openeo_processes_dask_dev
poetry install --all-extrasTo add a new core dependency run:
poetry add some_new_dependencyTo add a new development dependency run:
poetry add some_new_dependency --group devTo run the test suite run:
poetry run python -m pytestNote that you can also use the virtual environment that's generated by poetry as the kernel for the ipynb notebooks.
This repo makes use of pre-commit hooks to enforce linting & a few sanity checks. In a fresh development setup, install the hooks using poetry run pre-commit install. These will then automatically be checked against your changes before making the commit.
The json specs for the individual processes are tracked as a git submodule in openeo_processes_dask/specs/openeo-processes.
The raw json for a specific process can be imported using from openeo_processes_dask.specs import reduce_dimension.
To bump these specs to a later version use:
git -C openeo_processes_dask/specs/openeo-processes checkout <tag>
git add openeo_processes_dask/specs/openeo-processes