Skip to content

Releases: zenml-io/zenml

0.91.2

19 Nov 06:31
@schustmi schustmi
0.91.2
608fde4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.91.2 Latest
Latest

New Features

Kubernetes Deployer

  • New deployer implementation that allows you to deploy your pipelines on Kubernetes

Other

  • Support for Mlflow 3.0

Bug Fixes

  • The S3 artifact store now works again with custom backends
  • Passing inline SSL certificates for the RestZenStore now works
  • The Weights & Biases experiment tracker now doesn't fail if the pipeline run name exceeds the maximum tag length

What's Changed

  • Add version 0.91.0 to legacy docs by @github-actions[bot] in #4164
  • Fix missing snapshot in remote deployments by @stefannica in #4167
  • Add 0.91.1 to the migration tests by @github-actions[bot] in #4166
  • Add success comments to changelog validation workflow by @strickvl in #4169
  • Fix applying default kubernetes resources by @schustmi in #4171
  • Fix bug in deploy staging workspace workflow by @htahir1 in #4176
  • Fix release docker build workflow by @schustmi in #4168
  • Fix breaking CI checks (Linting & Integration tests) by @Json-Andriopoulos in #4143
  • Improvements to CV for FiftyOne (lineage / dataset versioning) by @strickvl in #4147
  • Add custom commands, reviewer subagents and Claude workflows by @strickvl in #4172
  • Fix documentation errors in containerization and Vertex orchestrator by @strickvl in #4187
  • Fix code upload logic for dynamic pipelines without step templates by @schustmi in #4181
  • Fix UnboundLocalError in SSL certificate validation by @htahir1 in #4188
  • Cache key index for faster cache lookup by @schustmi in #4178
  • Add docs for dynamic pipelines experimental feature by @htahir1 in #4173
  • Support MLflow 3.0 by @htahir1 in #4160
  • Give Claude permission to run the formatting script by @strickvl in #4195
  • Fix S3 artifact store endpoint configuration for custom S3 backends by @strickvl in #4186
  • Fix Claude GitHub Actions workflow configuration by @strickvl in #4200
  • Changes to issue templates by @bcdurak in #4199
  • Improve detecting of local images during multi-step-build by @schustmi in #4194
  • Sanitize wandb tags by @schustmi in #4189
  • Add workflow to close stale PRs after 4 weeks of inactivity by @htahir1 in #4192
  • Add Kubernetes Deployer for Pipeline Deployments by @safoinme in #4127
  • Fix deletion of dynamic pipeline runs by @schustmi in #4191
  • Include project ID in Kubernetes labels by @schustmi in #4177
  • Rename method of dynamic orchestrator by @schustmi in #4206
  • Implemented detail view for artifact version by @AlexejPenner in #4203

Full Changelog: 0.91.1...0.91.2

Contributors

strickvl, stefannica, and 6 other contributors
Assets 2
Loading

0.91.1

11 Nov 12:43
@schustmi schustmi
0.91.1
5e4d0e0

Choose a tag to compare

View all tags
0.91.1

New Features

Huggingface Deployer

  • New deployer implementation that allows you to deploy your pipelines on Huggingface

Kubernetes orchestrator

  • Additional configuration option to specify the container security context
  • Additional configuration option that can be used skip creating owner references

Other

  • New authentication options for the HashiCorp Vault secret store
  • Support for newer Databricks versions
  • Introduced v1 of dynamic pipelines. This feature is highly experimental and not meant for production, and we are seeking early feedback. Read these docs for more details.

Bug Fixes

  • Fix port reuse for local deployments
  • Fix parallel deployment invocations
  • Fix keyboard interrupt handling while monitoring a synchronous run
  • Fix case-sensitivity issue when updating the name of some entities like stack components

Known issues

  • When a kubernetes orchestrator is configured with pass_zenml_token_as_secret=True and orchestrator_pod_settings.env does not contain any values, the orchestrator fails to correctly pass the secret. As a workaround in this release, you can either
    • set pass_zenml_token_as_secret to False, or
    • configure a dummy environment variable (e.g. using the following CLI command zenml orchestrator update --orchestrator_pod_settings='{"env": [{"name": "DUMMY", "value": ""}]}')

What's Changed

Full Changelog: 0.91.0...0.91.1

Contributors

strickvl, stefannica, and 5 other contributors
Loading

0.91.0

25 Oct 03:08
@schustmi schustmi
0.91.0
fa9283b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.91.0

This release brings major deployment enhancements (Local Deployer, fully customizable deployment server, and deployment visualizations), powerful caching controls (file/object-based cache invalidation, cache expiration, and custom cache functions), Python 3.13 support, an MLX array materializer, and a new option to customize image tags during builds. It also deprecates legacy model deployers and drops Python 3.9 support.

⚠️ Breaking Changes

  • Drop Python 3.9 support

Deprecations

  • Model deployers are deprecated; use deployers to deploy entire pipelines instead

New Features

Deployment

  • Local Deployer for deploying pipelines locally
  • Deployment server is fully customizable via settings
  • Attach custom visualizations to deployments

Caching

  • Specify files or Python objects that invalidate a step's cache
  • Cache expiration to bound cache lifetime
  • Custom cache function for advanced invalidation logic

Other

  • MLX array materializer
  • Python 3.13 support
  • Customize the image tag of built Docker images

Bug Fixes

  • Fix incompatibility of print capturing when using numba
  • Fix the mount point configuration in the Hashicorp Vault secrets store

What's Changed

New Contributors

Full Changelog: 0.90.0...0.91.0

Contributors

strickvl, stefannica, and 7 other contributors
Loading

0.90.0

02 Oct 07:02
@schustmi schustmi
0.90.0
77fa781

Choose a tag to compare

View all tags
0.90.0

This release introduces pipeline snapshots and pipeline deployments, refactors the base package dependencies, and adds runtime environment variable support.

⚠️ Breaking Changes

  • Client-Server Compatibility: Not compatible with previous versions. Client and server must be upgraded simultaneously.
  • Run Templates: Existing run templates need to be recreated as they contain previous Client versions in their container images.
  • Base Package: The zenml package no longer includes local database dependencies. Install zenml[local] if you want to use ZenML without a local/remote server.

Deprecations

  • Run Templates: Deprecated in favor of pipeline snapshots. Snapshots can be triggered from server/dashboard and deployed as endpoints.
  • ZenML Pro On-Prem deployments only: The ZENML_SERVER_MAX_CONCURRENT_TEMPLATE_RUNS server environment variable is deprecated. Use ZENML_SERVER_MAX_CONCURRENT_SNAPSHOT_RUNS instead.
  • Stack Component API: The StackComponent.prepare_pipeline_deployment(...) method has been removed and will not be called anymore. This might be relevant if you've implemented custom stack components that rely on this method being called.

New Features

Pipeline Snapshots

  • Capture immutable snapshots of pipeline code, configuration, and container images
  • Execute snapshots from server/dashboard without local code environment
  • Can be deployed

Pipeline Deployments

  • Add a new Deployer stack component and Docker, AWS and GCP implementations
  • Deploy pipelines as HTTP endpoints for online inference or agentic usecases
  • Add pipeline init/cleanup hooks for quicker deployment runs

Runtime Environment Variables

  • Configure environment variables when running pipelines
  • Environment variables can either be defined directly or from ZenML secrets

Dependency Management

  • Move local database dependencies to local package extra
  • Add support for latest Pydantic and CloudPickle versions
  • Reduce base package dependencies

Jax Support

  • Add materializer for Jax arrays

What's Changed

New Contributors

Full Changelog: 0.85.0...0.90.0

Contributors

strickvl, stefannica, and 5 other contributors
Loading

0.85.0

12 Sep 11:24
@schustmi schustmi
0.85.0
cb407b2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.85.0

The 0.85.0 release delivers powerful pipeline execution enhancements and caching improvements that provide users with greater control over pipeline behavior.

⚠️ Breaking Changes

  • Local Orchestrator Behavior: The local orchestrator will now continue executing steps after some steps fail (instead of stopping execution immediately)
  • Docker Package Installer: Default Python package installer in Docker settings switched from pip to uv
  • Log Endpoint Format: Log endpoints return a different format - affects manual API access but not regular pipeline operations
  • Fetching Runs with old Client: Using a client of a previous release (<0.85.0) to fetch pipeline runs created by a client with version >=0.85.0 can cause an error if the run is in
    the new provisioning status.

📢 Upcoming Breaking Change

Important Notice for Future Release: In the next release, the base zenml package will no longer include dependencies for running ZenML connected to a local database.. Users will only be able to connect to deployed ZenML servers with the base package.
To continue using ZenML locally with a SQLite database, install with the local extra: pip install 'zenml[local]'. If you're using ZenML with a local server, you're already installing zenml[server] and this change will not affect you.

🚀 New Features

Pipeline Execution Modes

  • Flexible Failure Handling: Configure what happens to a pipeline run when any step fails, providing fine-grained control over pipeline execution behavior

Advanced Caching System

  • Value-Based Caching: Materializers now support caching artifacts based on their actual content/value rather than just artifact ID, enabling more intelligent cache reuse
  • Cache Policies: New cache policy system allows users to specify precisely when a step should be cached, providing granular control over caching behavior

Airflow 3.0 support

  • Airflow 3.0 Compatibility: Support for Apache Airflow 3.0, ensuring compatibility with the latest Airflow features and improvements

What's Changed

New Contributors

Full Changelog: 0.84.3...0.85.0

Contributors

strickvl, htahir1, and 3 other contributors
Loading

0.84.3

27 Aug 15:10
@schustmi schustmi
0.84.3
0712cee
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.84.3

The 0.84.3 release introduces ZenML Pro service account authentication support and includes important Kubernetes integration fixes. This release enhances authentication flexibility for automated workflows and improves the reliability of Kubernetes-based orchestration.

🚀 New Features

ZenML Pro Service Account Authentication

  • CLI Login Support: Added ability to authenticate with ZenML Pro using service account API keys via zenml login --api-key
  • Programmatic Access: Service account API keys can now be used for programmatic access to ZenML Pro workspaces
  • Organization-Level Access: Service accounts provide access to all workspaces within an organization for automated workflows

🛠️ Fixes

Kubernetes Integration

  • Name Sanitization: Fixed Kubernetes resource name sanitization to properly handle special characters and ensure compliance with Kubernetes naming requirements

Dependencies

  • Click Version: Relaxed Click dependency version constraints to improve compatibility with other packages

What's Changed

Full Changelog: 0.84.2...0.84.3

Contributors

strickvl, stefannica, and 2 other contributors
Loading

0.84.2

06 Aug 19:18
@schustmi schustmi
0.84.2
d626435
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.84.2

The 0.84.2 release brings significant performance improvements and reliability enhancements that strengthen ZenML's orchestration capabilities. This release features a major refactor of the Kubernetes orchestrator with enhanced restart capabilities, faster pipeline compilation for large pipelines, critical security fixes, and improved logging performance.

Kubernetes Orchestrator

  • Enhanced Robustness: Complete rework to use Jobs instead of raw pods for orchestrator execution with state reconstruction support, enabling automatic restarts for a much more robust orchestration experience
  • Deprecated Settings: Several settings attributes are now deprecated due to the shift from pods to Jobs, streamlining the configuration interface
  • Updated Logging Behavior: The orchestrator pod no longer streams logs from step pods directly, improving performance and resource usage

🚀 Improvements

  • Faster Pipeline Compilation: Significantly improved pipeline compilation performance for large pipelines

🛠️ Fixes

  • Run Creation Deadlock: Fixed deadlock that could happen if two steps try to create a run at the same time
  • Security Enhancement: Enhanced path materializer to properly validate symlinks and hardlinks to prevent path traversal attacks
  • Logging Performance: Improved logging thread performance by avoiding unnecessary sleep when shutdown is requested
  • WandB Integration: Fixed WandB experiment tracker flavor initialization when the integration is not installed
  • Type Annotations: Fixed type annotations for experiment_tracker and step_operator parameters to accept both string and boolean values

What's Changed

  • Add version 0.84.0 to legacy docs by @github-actions[bot] in #3863
  • Add 0.84.1 to the migration tests by @github-actions[bot] in #3865
  • Fix wandb flavor needing integration to be installed by @stefannica in #3871
  • Verify symlinks and hardlinks in the path materializer by @stefannica in #3870
  • Don't sleep in logging thread if shutdown was requested by @schustmi in #3872
  • Fix run creation deadlock by @schustmi in #3876
  • Fix nonlinear scaling of pipeline compilation time by @schustmi in #3873
  • Fix type annotations by @schustmi in #3878
  • Orchestrator pod restarts by @schustmi in #3869

Full Changelog: 0.84.1...0.84.2

Contributors

stefannica and schustmi
Loading

0.84.1

30 Jul 10:09
@schustmi schustmi
0.84.1
acaebfd
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.84.1

The 0.84.1 release delivers important stability improvements and feature enhancements that strengthen ZenML's orchestration capabilities and developer experience. This release focuses on enhanced Kubernetes orchestrator management with schedule support and better error handling, improved step exception collection for debugging, external service account support, dynamic fan-out/fan-in patterns for run templates, and critical fixes for Vertex step operators and logging reliability.

🚀 Improvements

  • Step Exception Handling: Improved collection of step run exception information for better debugging
  • External Service Accounts: Added support for external service accounts for improved flexibility

Kubernetes Orchestrator

  • Enhanced Schedule Management: Added schedule management capabilities (updating and deleting) for the Kubernetes orchestrator
  • Better Error Handling: Added explicit Kubernetes failure reasons to logs and improved pod monitoring
  • Secret Cleanup: Fixed cleanup of Kubernetes secrets when orchestrator pods fail to start

🛠️ Fixes

  • Vertex step operator credential refresh: Fixed retry logic and credential refresh for the Vertex step operator
  • Logging Fixes: Resolved race conditions in logging for better reliability

🛠️ Documentation & Examples

  • Agent Examples: Updated README and added comprehensive agent examples
  • Template Updates: Bumped LLM template version for latest improvements
  • Deprecation Cleanup: Removed deprecation warnings in quickstart for cleaner user experience
  • Dynamic Fan-out/Fan-in: Added support for dynamic fan-out/fan-in patterns with run templates for more flexible pipeline architectures

What's Changed

Full Changelog: 0.84.0...0.84.1

Contributors

strickvl, stefannica, and 5 other contributors
Loading

0.84.0

11 Jul 13:43
@schustmi schustmi
0.84.0
6aac5a7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.84.0

The 0.84.0 release delivers significant architectural improvements and orchestration enhancements that boost ZenML's reliability, performance, and developer experience. This release focuses on pipeline execution resilience with step retries and early stopping capabilities, enhanced Kubernetes orchestrator features, and improved server-side processing for better scalability.

🚨 Breaking Changes

Kubernetes Orchestrator Compatibility

  • Client-Orchestrator Version Compatibility: This release introduces changes to the Kubernetes orchestrator that make the new client version incompatible with previous versions. When using the Kubernetes orchestrator, the ZenML version in both the client and orchestrator pod must match exactly. This means that if you upgrade your ZenML client to 0.84.0, you must also ensure that the orchestrator pod images are also using ZenML 0.84.0 to avoid compatibility issues.

🚀 Orchestration Enhancements

Pipeline Control & Reliability

  • Early Pipeline Stopping: Added ability to stop pipelines early with Kubernetes orchestrator implementation for better resource management
  • Step Retries: Implemented configurable step retry mechanisms for improved pipeline resilience
  • Step Status Refresh: Enhanced step status monitoring with real-time refresh capabilities

Kubernetes Orchestrator Improvements

  • Run Steps using Jobs: Leverage Kubernetes jobs for better retry behavior when running steps
  • Enhanced Pod Management: Always build pipeline images for Kubernetes orchestrator and improved pod caching
  • Pod Logs Access: Added orchestrator pod logs functionality for better debugging and monitoring
  • Kubernetes Model Validation: Added warnings for invalid Kubernetes model data

⚡ Performance Improvements

  • Threadsafe RestZenStore: Implemented thread-safe operations for improved concurrent access
  • Server-side Processing: Moved parent step computation and cascading tags processing to server-side for better performance
  • Improved Fetching: Enhanced pipeline and step run fetching mechanisms for faster response times

🔧 Developer Experience

CLI & Login Improvements

  • New ZenML Login: Redesigned login experience with improved CLI styling and user interface

Configuration & Compatibility

  • Easier Configuration: Simplified step operator and experiment tracker configuration for steps
  • Unified Imports: Standardized config imports to match documentation examples

🔒 Infrastructure & Security

  • Helm Chart Improvements: Added PVC creation option for local configuration path
  • Service Connector Security: Hidden service connector secrets as internal implementation details

🛠️ Fixes

  • Model Version Fetching: Fixed fetching model versions by UUID passed as string
  • Visualization Handling: Ensured visualizations are committed before potential transaction rollbacks
  • SageMaker Configuration: Made SagemakerOrchestratorSettings.processor_tags optional
  • Data Artifact Access: Fixed data artifact fetching issues
  • Pydantic Validation: Added support for step/parameter names with leading underscores in run templates
  • Path Sanitization: Added missing remote path and Docker tag sanitizations

📚 Documentation

  • Best Practices: Updated documentation with minimum permissions for cloud stack components
  • Concurrent Execution: Added guidance for handling concurrent pipeline execution in separate containers
  • User Guides: Enhanced user guides with llm-complete-guide project and improved examples
  • ZenML Pro: Updated URLs in documentation to use zenml.io/pro
  • VSCode Integration: Added VSCode tutorial pipeline testing to CI

What's Changed

Full Changelog: 0.83.1...0.84.0

Contributors

jsuchome, strickvl, and 6 other contributors
Loading

0.83.1

23 Jun 13:40
@schustmi schustmi
0.83.1
e941c5e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.83.1

The 0.83.1 release delivers significant performance improvements and optimizations that enhance ZenML's efficiency and responsiveness. This release focuses on architectural improvements including step configuration separation from deployment, idempotent POST requests with request caching, and enhanced artifact loading performance, alongside important bug fixes and documentation updates.

⚡ Performance Improvements

  • Architectural Enhancement: Separated step configurations from deployment for improved modularity and reduced overhead
  • Request Optimization: Implemented idempotent POST requests with intelligent request caching for faster API responses
  • Artifact Loading: Enhanced artifact loading performance by using the active artifact store as cache

Fixes

  • Fixed potential race condition during pipeline run status updates
  • Fixed missing artifact nodes in DAG visualization
  • Fixed run template fetching issues that could cause errors during pipeline execution
  • Fixed missing pipeline substitutions for legacy steps
  • Reset active project after deleting local database

Documentation

  • Updated Kubernetes configuration documentation with clearer guidance
  • Enhanced run template documentation with better examples and explanations
  • Added shared CSS guidance for HTML visualizations
  • Updated kaniko project status documentation
  • Fixed authentication confusion in ZenML Pro API documentation

What's Changed

New Contributors

Full Changelog: 0.83.0...0.83.1

Contributors

strickvl, stefannica, and 5 other contributors
Loading