A few TODO for state machine operator

[vsoch]

These are from my personal notes - not high priority so putting them here.

• We need a way to pause the workflow manager, or issue a command to it, or reload job configs.
• We also need to test different kinds of manual deletions / changes and see how it recovers.
• Test flux as an executor with entire workflow (Mummi) with containers first
• When a job fails, cancel / delete objects in the cluster so the state machine will trigger new entities.

For the last, we can provision the container or require the user to.

[vsoch]

There is no concept of a max number of tries for the overall state machine. E.g., "I want 10 completions of cganalysis, and when an entire machine fails launch a new one." If there is some fundamental issue with the pipeline you might never get one result, e.g., this I just simulated:

We won't have this in our study now, but it's something we will want to add - a more global failure or stopping condition to the entire orchestration.