Schedule inconsistent state

[Json-Andriopoulos]

Contact Details [Optional]

No response

System Information

No response

What happened?

Problem Statement

1. If the user provides an invalid cron expression when creating a schedule the record in not validated in our code and is stored in the database. Later on, in the use case of Kubernetes orchestrator the expression is considered invalid and the cron job is never created. Therefore we have an inconsistent state where users can see schedules but the schedules are not running.
2. If the user attempts to fix the error by updating or deleting the schedule the CLI commands fail as they first attempt to update/delete the cron job (which doesn't exist). So, the user has a dangling DB record - visible on list operations.
3. Even if the user provides a correct cron expression they can always by accident delete the cron job on their infrastructure. Then the same DB/cron job inconsistency appears.

What should happen

1. Invalid CRON expressions should fail as soon as possible. Introducing cron_expression validation on the Pydantic model level should do the trick.
2. Infrastructure update operations (delete/update cron job) should gracefully check if the cron job exists and fail/succeed accordingly. If I delete a schedule with missing cron job I would expect to skip the operation on Kubernetes level and delete the DB record. If I update a schedule with a missing cron job I would try to re-create the cron-job with the updated expression.
3. Schedule creation should (ideally) happen on the DB after the cron job is created in atomic fashion (if cron job creation fails, schedule creation should fail).

Reproduction steps

...

Relevant log output

Code of Conduct

• I agree to follow this project's Code of Conduct