2022 Cylc Meetings

[hjoliver]

Following on from #139 ...

[hjoliver]

17/18 January (UK/NZ)

Apologies

• Tim P

Any new Cylc 8 issues?

• JR: "UI safe mode" e.g. confirmation dialog for "pause workflow"
  • see mutations: interactive prompts cylc-ui#674
• DM: users might struggle with adapting Cylc 7 working practices around "state reset" etc
  • see how it goes; consider adding use-case specific instructions to migration guide

Team Members: work done/ongoing, any related issues

DS

• n > 0 db lookups.
• ZeroMQ GraphQL subscriptions
• CLI via UIS

OS

• UID!

RD

• profiling - cylc-flow#141
• UI: add status of task/family/cycle-point to top of mutations menu - cylc-ui#769

HO

• (holiday leave ~3 weeks)
• finished off several small PRs (e.g. fixed clock-trigger offset for inexact intervals)
• reviewing, testing, merging

MH

• Holiday leave 2 weeks
• Autodocumenting UIServer traitilets (pr incoming shortly)
• reviewing
• Small UI pr, changing "development" to hostname

TP

• (Holiday/Sick ~2½ weeks)
• Assorted documentation changes
• Moved tutorial workflows to cylc-flow after prolonged "discussion".
• Improved support for users working out what platforms are available.
• Retired ROSE_SITE, and fixed some bugs I found in Cylc Rose.

AC

• (just back at work yesterday)

Discussion Topics

8.0rc1 release?

• In NZ, DM and AC are back at work for the rest of January; HO on leave tomorrow, back at work Tues-Thurs 25-27 Jan, then away for another 5 days
• UK - only DM has significant leave during remainder of Jan
• Aim for rc1 release 25-27 Jan?

Profiling results so far - cylc-flow#141

• memory use? (imports? zmq client? datastore ~10MB complex suite?)
• memory leak? (datetime example going up slowly for ~2month run, 8x daily)
• wall clock time?
  • HO: scheduler should have: smaller task pool, more efficient scheduling algorithm, but more database access, different sleeps, different datastore etc.
  • RD: complex suite Cylc 8 50% less CPU, although 60% longer run time
  • DS: longer load time for ZMQ libs, even for job clients
  • maybe the wall clock increase only relevant for quick busy test workflows?

Historical data and reflow in the datastore and the UI

• batch DB call for prerequistes etc., flow numbers
• currently loads historical regardless of flow
  • need at least to show flow nums in the UI to make this understandable
• future: flow-specific n-windows; overlapping flows: need to think about the best we can do, in the datastore and UI, without flow number in the UID (Cylc 9)

Need another reviewer for AC's UI PR: cylc/cylc-ui#861

• UK team to assign

Extant Issues

cylc-flow

• 8.0rc1 open issues

  • ONLY 3 🎉 - only DS datastore one is big

• 8.0 open issues

cylc-ui

• 1.0 milestone

cylc-uiserver

• 1.0 milestone

AOB?

Open Questions

• HO still to check the old rose symlink config issue - does NIWA still need it? DM says no way to do it with Cylc 8 at the moment.

Next video conference

TBD: early-mid Feb?

[wxtim]

15/16 February (UK/NZ)

Apologies

Any new Cylc 8 issues?

• Weird scan-induced scheduler health-check => shutdown bug?
  • reports contact file corrupted or missing; then missing when tries to delete it on shutdown
  • TODO: check filehandle closed

Team Members: work done/ongoing, any related issues

DS

• Coordinated PUB/SUB and REQ/REP, and GraphQL subscriptions at the Scheduler.
• Reviews?

OS

• Trigger bugs.
• Release stuff.

RD

• Lots of small things
• Discovered some bugs
• Reviewed universal ID and trigger + queue bugfixes, amongst others
• Show Cylc object ID, type and status at top of mutations menu cylc-ui#891

HO

• fixing various 8.0rc1 triggering and polling bugs
• ongoing discussion re exactly how trigger tasks and flows interact, inside and outside of the n=0 task pool

MH

• Auto documenting UI Server traitlets
• Reviewing
• Some updates to Rose Tutorials
• Some other small things including bug investigation and small prs
• Covid leave 😷

TP

• Taking Cylc (and Rose) tutorials and fixing things.
• Failing to have automatic spellchecking working on Cylc - I wasn't happy with the libraries I could find for spellchecking in Sphinx. IMO we'd be better writing something from scratch using Ispell, but it's not a priority.
  • We should probably decide on ["parameterize", "parametrize" or "parameterise"] - I'm in favour of "parametrize", not because I like it but because it's the spelling used by pytest.
• But finding lots of typos in the process.
• Fixing bugfixes which have come up in testing:
  • A breakage to Cylc 7 Review.
  • ``cylc play --stopcp=` bug
  • Convoluted discussion of whether cylc install a/b installs to ~/cylc-run/b or ~/cylc-run/a/b (we went for the latter).
  • --stopcp check routine done after database initialized bug.

Question
Talking to Met Office weather Science Comms group about Cylc?

AC

• Tackling small UI bugs
• Working on changes to the single run collapsable gsuite component MR

DM

• Wrapper script enhancements cylc-flow#4655 (rc2)
• Testing trigger + set-outputs

Discussion Topics

8.0rc1 release?

• can we do it this week?

Where are we at now with tutorials and training material?

• pretty much done! (for a new users)
• (still need a "what's new in Cylc" / 7-8 migration tutorial)

Post 8.0rc1 release:

• pushing it out to users
• training and support
• bug fixes
• optimization

ongoing development priorities

• certain important not-exactly-new features for rc2,3,...
  • e.g. refinement of triggering behavior (including triggering)
  • scheduler GraphQL subscriptions, TUI performance (OS: 8.1? ... some work rc-x possibly)
  • cross-user via the UI
    • prob need a J-Hub service to look at the authenticated users config?
  • inferred prerequisite status (OK for future tasks - even if previous flows; that) (Just historical tasks needed)
  • use TUI to show reflow number (first cut)
• UI priorities?
  • (gscan state propagation done?)
  • AC has ideas on table view column customization
  • user preferences: storage; memory (e.g. of table columns; view layout)
    • J-Hub service to store prefs under .cylc
    • can use browser storage as a first cut?

Extant Issues

cylc-flow

• 
• 
• 

cylc-ui

• 
• 

cylc-uiserver

• 
• 

cylc-doc

• 

AOB?

Open Questions

Debrief on BOM Cylc 8 seminar

Next video conference

TBD: End Feb?

[wxtim]

2/3 March (UK/NZ)

Apologies

Any new issues?

• problem identified with Rose fcm_make apps

Team Members: work done/ongoing, any related issues

DS

• (mostly operations work since last meeting)

OS

• Bits and bobs.
• Cylc trigger discussions.
• 8.0rc1 deployment
• uis: fixing workflows in stuck states

RD

• Dealt with markupsafe packaging problem - cylc-flow#4703, cylc-flow-feedstock#37
• Stop traceback for ParsecErrors during config load when running workflow - cylc-flow#4720
• Investigated integration test hangs
• Working with Tim on enabling mutation menu when clicking on workflow icons in GScan sidebar - cylc-ui#928

HO

• local HPC installation of rc1, + user group
• tutorials revision
• trigger + reflow behaviour and defaults

MH

• Serialise/Deserialise flow_nums
• Cylc Documentation - Quick Summary Changes refactor
• Some reviewing
• Work in progress: Making file installation rsync configurable for tests/f: remote/01 find a better way to simulate slow rsync cylc-flow#4696

TP

• Added support for Cylc Clean (and potentially other commands using ProcPoolExecutor) in Uiserver Add Cylc Clean Support cylc-uiserver#323.
• Work (In Progress) on adding menus to GScan icons. Add mutations to Gscan workflow icons. cylc-ui#928 - useful as a training exercise.
• Reviewing Documentation.
• Getting my head around the conda-forge release process (I was not previously au-fait with this).

AC

• most of last week consumed with SDT work, half of Thursday, Friday and Yesterday debugging tests on my MR for collapsing single task workflows

DM

Discussion Topics

BOM (JR):

• new HPC in acceptance testing
• will not have Python 2 (or Cylc 7)
• so aiming for Cylc 8 deployment
• JR to install and test Cylc 8 somewhere on site v. soon

UK:

• rc1 deployed; wrapper deployed
• Mike T testing with real workflows...
• ... Rose fcm_make app issue identified, needs quick rc2 after fixing (to re-release the rose plugin)
• markupsafe Jinja2 dependency issue: need to pin to older version for pip install; we can do this for rc2

Tutorials and training material

• more or less done?
• plans for delivery?
• UK: seminars, followed by various tutorial sessions, will begin planning soon
• HO: similar here; TODO: pass latest seminar slides on in case useful

8.0rc2 release priorities

• rc2 cylc-flow open questions
  • (some questions resolved and de-labelled)
• Primarily: agree and implement final trigger changes: trigger: generalisation of triggering approaches cylc-flow#4686
  • bump to rc3; the Rose App problem is now more urgent, as a blocker for UM workflow uptake
• Other?
• Release ETA?
  • mid-March for triggering etc. (rc3?)
  • rc2 soon as possible for Rose App fix

Stop-cycle-point, config, CLI override

• DM and RD to discuss and propose solution on the issue

Jinja2 v3:

• need to test urgently
• concern about breakage affect Cylc 7 back-compat; parallel deployments of Cylc 8 might be needed for this

Tomek's environment override issue

• HO to revisit and make a decision - probably keep the status quo and document it

Triggering and flows

• use --flow=all not --flow=any, for triggered tasks, with or without waiting
• general agreement reached on trigger behaviour and defaults 🎉
• set-outputs and remove commands: need revisiting in light of this (use cases and behaviour)

cylc 7 release

• Another one needed now, for cylc review and job polling fixes
• TODO: also check Cylc 7 milestones - did I forgot to create one for the last release?

Extant Issues

cylc-flow

• 
• 

cylc-ui

• 

cylc-uiserver

• 

cylc-doc

• 

AOB?

Open Questions

Next video conference

~2 weeks: mid March

[hjoliver]

17/18 March (UK/NZ)

Apologies

• DS
• TP - No, I'm alive
• AC

Any new issues?

Another final Cylc 7 release: 7.8.11/7.9.6

• DONE

Team Members: work done/ongoing, any related issues

DS

OS

RD

• Make rose host-select use bash login shell to pass through env vars
• Docs: Fix nightly cylc-doc#425
• A bunch of smallish things

HO

• 7.8.11 / 7.9.6 release
• Implemented final agreed flow triggering
  • Flow triggers 1/2: no-wait cases cylc-flow#4738
  • Flow trigger CLI and defaults  cylc-flow#4739
• And documented it:
  • Redo the User Guide Reflow section. cylc-doc#427
• Flow stop fix:
  • Stop a specific flow with non-active n=0 tasks present cylc-flow#4743

MH

• Fcm Make fix (Fcm make fix for remote and subshell platforms cylc-rose#120, Fcm make fix for remote and subshell platforms metomi/rose#2557)
• Reviewing
• Configurable rsync (Rsync configurable cylc-flow#4736)

TP

• Small Bugfixes
• Documentation
• Covid 🤒
• Leave

AC

DM

Discussion Topics

Rose fcm_make app fix? (UM uptake blocker)

8.0rc2 release priorities

• rc2 cylc-flow open questions

• Agreed final trigger changes: trigger: generalisation of triggering approaches cylc-flow#4686

  • (Needs code review; --wait PR not quite finished)

• Are we going to do this jinja2: move to version 3 cylc-flow#4702 (TP)?

  • any testing done yet?

• Release ETA

  • RC2 next week, with Rose App fix, TP and MH to do

  • RC3 early April, for triggering etc.?

  • Other?

Tomek's environment override issue:

• 8.0 or 8.x? (handle as 7-8 migration issue or non-default config change?)
• Not worth delaying 8.0, so punt it forward for now (DONE)

Triggering and flows

• implemented (see HO list above)
  • pending final tweaks on 2/2 DONE
  • DB outputs table? DONE
• documented (see HO list above) DONE

Stopping flows with cylc remove? cylc/cylc-flow#4728 (DM)

Spawn on demand / developer / admin documentation (DM)

• Need for future reference
• The old SOD proposal is now out of date in some ways
• Where to put it?
• Raise again next meeting

Jinja2 v3

• RD to test on real workflows, e.g. Nesting Suite and Rose Stem.

General Doc restructure would be good

• Raise again next meeting

Extant Issues

cylc-flow

• 
• 

cylc-ui

• 

cylc-uiserver

• 

cylc-doc

• 

rose

• 

cylc-rose

• 

AOB?

Open Questions

Next video conference

End of March, switch to NZ night time.

[hjoliver]

31 March

Apologies

• AC

News

• RC2 done 🎉 - thanks MH and TP

Any new issues?

New rose mpi-launch problem identified

• not sure why it affects only Cylc 8
• accidental interface change in rose date? Needs new isodattime release and cylc-flow PR to use it

Reload not resulting in new remote installation (note NOT cylc install)

• Issue up, need to do it

Washup of release process copied from Element (for the record) - MH and TP

• It's a nicely documented and automated process.
• Don't have Github break during release process.
• Make sure everyone is au-fait with the process - It'll get faster with practice
• Pairing it speeds it up - reduces risk of things needing correction and means that the first review is effectively done by the paired person.

Team Members: work done/ongoing, any related issues

DS

• Ready for review:
  -- Node (Network) Coordinated Subscriptions (part 1/3) - Centralise REP and PUB into single runtime server cylc-flow#4274
  -- fix graphql node state and remove ghosts filters cylc-flow#4791 & remove ghosts from graphql args cylc-ui#980
• Investigate: data store: issues when reloading workflows cylc-flow#4784
• Continue GraphQL subs at scheduler (part 2/3)

OS

RD

• Handle missing optional dependencies better cylc-flow#4745
• Fixing slightly nasty little bug in UI: Fix mutations-load race condition cylc-ui#979
• Change next release from 2.1 to 3.0 metomi/isodatetime#205
• Various GH Actions fixes

HO

• Finished off the flow triggering PRs for RC3 (needs review)
  • Flow triggers 1/2: no-wait cases cylc-flow#4738
  • Flow trigger CLI and defaults  cylc-flow#4739
• Fix cylc play --set CLI for remote run hosts Escape quotes in remote command args. cylc-flow#4769
• Stop a specific flow with non-active n=0 tasks present cylc-flow#4743
• (a bunch of small fixes and reviews)
• (local deployment of RC2, user comms etc.)

MH

• Release of 2.0rc2
• FCM Make fix
• Reviewing
• Small tidy PRs e.g Remove cylc hub button in single user mode cylc-ui#969, Pass gui mode to ui. cylc-uiserver#331, Fix Error Message for RunN Workflow IDs cylc-flow#4789
• Rose 2019 release process with @wxtim

TP

• Worked with @datamel to do the Cylc 8 release process:
  • Made only very small edits to the Cylc Admin issue template which highlight just how good the process seems to be.
  • It was slow because we were being careful && GH was having problems.
• Working on releasing Rose 2019.01.6 - safety features to stop Rose interacting with Cylc 8 workflows. (also highlights slickness of Cylc 8 &c automated processes).
• Worked with @datamel on FCM make fix
• Investigating a pile of related Cylc-Rose bugs thrown up by @dpmatthews testing.
• Investigation Cylc 7 Review issue reported Add "preparing" to list of filterable states. cylc-flow#4782

AC

DM

• deployed (and re-deplyed RC2)
• identified rose mpi-launch issue (rose mpi-launch breaks if ROSE_LAUNCHER contains spaces metomi/rose#2573)

Discussion Topics

Release ETA

• URGENT RC3 early-mid April, for triggering etc.?
• Yes, should be feasible, by Easter

Is there a problem with GraphiQL and subscriptions, or not? (OS)

• subscriptions hang for OS
• HO has seen this but it worked again after nuking the environment and starting anew
• pip freeze etc. -> OS? in case its a dependency issue

Note graphql-ws and/or? graphen-tornado increasingly looking like abandonware. At some point we might have to contribute or absorb the code into cylc. Just adaptor classes, not too complex, but knowledge of graphene and graphql internals needed.

Stopping flows with cylc remove? cylc/cylc-flow#4728 (DM)

• nothing more yet

Spawn on demand / developer / admin documentation (DM)

• Agreed last meeting; now: where to put it.
• consider splitting content by usage cylc-doc#449

Jinja2 v3: Any progress on testing with real workflows (RD?)

• Mike T testing the global suite
• Andy M testing rose stem
• 0-prefixed integers mainly a problem for rose-suite.conf (i.e. cylc-rose plugin), not suite.rc
• v3 could be an 8.x change; can't afford to delay 8.0
• could we use cylc install to install the non-default version into the workflow environment? See issue on extending the scheduler environment; but not feasible by 8.0.

General Doc restructure would be good

• proposal coming

Tomek's environment variable issue?

• a simple validator (post inheritance) to check for the order problem, would be easy to do and useful - RC4?

Upcoming Milestones

cylc-flow

• 

cylc-ui

• 

cylc-uiserver

• 

cylc-doc

• 

Rose

• 

cylc-rose

• 

isodatetime

• 

AOB?

Open Questions

Next video conference

Mid-April, NZ night time.

[hjoliver]

11 April

Apologies

News

• several early adopters at NIWA. One needs sub-workflows already.

Any new issues?

Reload not resulting in new remote installation (note NOT cylc install)

• Issue up, need to do it

Work done or ongoing, and related issues

DS

OS

RD

• cylc-flow now updated to use isodatetime 3.0: Update to isodatetime 3.0 cylc-flow#4554
• Improve cycling docs cylc-doc#454
• Tidy & homogenise usage of CylcOptionParser cylc-flow#4809

HO

• sub-workflows
• reviews, and review responses
• bunch of small tweak PRs
• (started spawn-on-demand developer/expert documentation)

MH

TP

• Reviewing. Reviewing. Reviewing.
• Poking Add Cylc Clean Support cylc-uiserver#323
• Qustions about how Cylc Rose handles CLI defined vars (Fix.do not allow multiple templating sections cylc-rose#133, https://github.com, Make test_pre_configure[template engine conflict] less specific cylc-rose#131, Pre configure not handling rose template variables properly cylc-rose#132, Ensure that fileinstall doesn't fail given the status of asyncio loop cylc-rose#130)
• Fix broken things in Rose found by testers:
  • Mpi launch fix metomi/rose#2574
  • Rose env cat fix metomi/rose#2578
  • update usage of isodatetime methods metomi/rose#2579

AC

DM

Discussion Topics

Release ETA

• URGENT RC3 early-mid April, for triggering etc.?

Tomek's environment variable issue?

• (was an issue raised for the validator?)

Project Issue Summary

Open Questions

AOB?

Next video conference

Late April, NZ night time.

[oliver-sanders]

Wednesday 4 May

Apologies: DS, AC?; HO will be a bit late

Work done or ongoing, and related issues

HO

• (family holiday)

• (local deployment and 2022/3 funding)

• PRs: https://github.com/cylc/cylc-flow/pulls?q=is%3Apr+author%3Ahjoliver+sort%3Aupdated-desc; e.g.:

  • a couple of scan fixes
  • disallow direct validation (etc.) of flow.cylc
  • nested workflow visibility
  • support for sub-workflows
  • datastore prerequisites

• TP

  • Rose Stem Fix - Rose stem broke because of it's coupling with cylc install interface.
  • Chasing assorted issues with platforms
  • Holidays
  • Priorities
    • Investigate bahaviour of section merging in platform definitions.
    • Update Cylc Review once Log dir tidy cylc-flow#4836 is in, then get @hjoliver to do another Cylc 7 release and have @dpmatthews deploy it. Question - might we just want to glob the log directory and offer it up to review users to make it safer against lots of versions?
  • Questions
    • How urgent are the two Rose Tickets marked RC3

RD

• Prevent Cylc 7 from running suite.rc workflow that was previously run with Cylc 8 cylc-flow#4838
• Improve flow.cylc reference docs cylc-flow#4846
• Remove --directory option for cylc install cylc-flow#4823
• Move processing of stop cycle point to config.py cylc-flow#4827
• aotf: fix mutations in workflow toolbar cylc-ui#999

Discussion Topics

• NIWA priorities

  • 8.0rc3 release, for training
  • sub-workflows: now have two teams wanting this

• n > 0 prerequisites - Keep prerequisites for the datastore. cylc-flow#4849

• Post 8.0.0 release plan.

  • Should we keep jumping major versions (FireFox style) or should we go for a longer string of 8.x releases with deprecations on route (Python style).
  • Basically should we aim to be more waterfall or more agile once we have got Cylc 8 out of the way?
  • This is relevant for deprecation notices, some currently say Cylc 9 which is long way off, perhaps they should say a future 8.x release with the "x" filled in later when decided.

• Altair: upcoming project

Project Issue Summary

Open Questions

AOB?

Next video conference

[MetRonnie]

23 May

Apologies: OS

Work done or ongoing, and related issues

RD

• Involved in release
• Update Jinja2 to 3.0.* cylc-flow#4877
• Fix task matching bug cylc-flow#4881
• Raise error if there is a relative path in global.cylc[install]source dirs cylc-flow#4887
• Looked into & removed unnecessary cylc-ui dependencies
• Fix bug where mutations menu would show wrong workflow cylc-ui#1011
• Allow cylc clean to use runN cylc-flow#4872

HO

• rc3 release
• misc PRs closed: https://github.com/cylc/cylc-flow/issues?q=assignee%3Ahjoliver+is%3Aclosed e.g.
  • scan fixes
  • src in run-dir: sub-worklows
  • past-task prerequisites
  • fixed some confusing logging (succeeded(runahead) tasks; fake state changes from datastore task proxies)
• cylc install symlink-dirs and cylc-rose bug Check symlink target. cylc-flow#4890

DS

• PRs ready:
  -- Node (Network) Coordinated Subscriptions (part 1/3) - Centralise REP and PUB into single runtime server cylc-flow#4274
  -- Fix data-store DB job load cylc-flow#4891
• Start 2/3 graphql subs (preferably after above is in).
• CLI via UIS (look into post notebook => server)
• Assign me something to review/do (^.^)

MH

• Reviewing
• Log dir tidy cylc-flow#4836

TP

• Config Changes (😞 at this stage)
  • Expand Global Config the way @dpmatthews and @oliver-sanders expected. (Polished Oliver's work)
  • Warn about using owner
• Fixing tutorial workflows (Mixed bag of issues)
• Rose / Rose date / FCM Make fixes . (TLDR %Z is horrible)

Discussion Topic

• UM User Workshop 15 min presentation?
  • HO to do it again. Consider it a "trailer" for upcoming longer seminars.
• Next UI/UIServer maintenance release soon-ish, try to fix these bugs: https://github.com/cylc/cylc-ui/milestone/16
  • consider another quick full-release, to pick up Fix data-store DB job load cylc-flow#4891
  • (we were planning to do a bunch of quick-fire rc releases, although releasing has turned out to be quick a long process)
• Platforms: default directives (esp. for test battery)
• cylc install symlink target exists?
  • HO to investigate effect on remote platforms
• "open questions" question?
  • we should try to flag a few of these issues before each meeting, because addressing them "cold" on the day often isn't fruitful
  • UK team trying to do this in local meetings, but NZ can raise issues too if needed
• what to do with the old cylc and cylc-ui conda packages?
  • Dead-ended Cylc conda packages #155
• DM to look at HO's sub-workflow example when time allows
• OS has shared new seminar and workshop slide repos with HO
• HO to complete SOD developer docs ASAP
  • SoD: "how it works" cylc-doc#426

Project Issue Summary

Open Questions

AOB?

Next video conference

N.B:

• UK has public holidays on the 2nd and 3rd June
• NZ on the 6th
• OS away for 2 weeks
• UM User workshop week of 13-17 June

... suggest next meeting 21 June

[hjoliver]

27 June Meeting

Work done or ongoing, and related issues

RD

• UI: making mutation form validation work properly
• Contributed a bit to Log dir tidy cylc-flow#4836
• Preparing for workshop on Cylc 7/8 interoperability & upgrading to Cylc 8
• Docs:
  • Improve config reference docs cylc-flow#4916
  • Working on improving back-compat page in migration guide, and Migration guide: optional outputs cylc-doc#484
• 7.8.x: Fix bug in remote tidy error handling cylc-flow#4894

OS

HO

• Final? spawn-on-demand tweak Fix spawning of mixed-parentage tasks. cylc-flow#4906
• UM workshop presentation
• local Cylc 8 deployment and support, across two HPC platforms (NIWA and NeSI)
  • e.g. NeSI J-hub interactive Slurm sessions, w/o dedicated Cylc servers.
  • planning upcoming training

DS

MH

• Cycl gui command takes a workflow name (Add gui workflow arg cylc-uiserver#370)
• Log dir tidy work (issue with auto-restart logging - bumping to an issue.)
• Fix unintended directory stripping during cylc install cylc-flow#4931
• Seminar prep

TP

Prepping seminars
Cylc lint

Discussion Topics

1. Roadmap to 8.0 release by mid-late July?

   • one more rc, or merge rc4,5 milestones into 8.0?
   • concentrate on existing bug fix PRs, and 728 syntax checker?
     • (aim for week starting 18 July)
     • (728 not super critical as most will start with back-compat mode: release where it's at, improve over time)
     • (no more rcs, str8 to 8.0)
     • (MH's log dir tidy is essential)

2. MO seminar debrief

3. Cylc 8 workshop progress and plans

4. Another Cylc 7 release needed (for cylc review)

   • (Needed shortly after 8.0 comes out, for cylc review - and maybe another bug fix around polling and restart?)

5. Dead-ended Cylc packages on Conda Forge

   • (forgot to raise this 😵‍💫 )

6. Any pre-considered Open Questions

   • (ALL: prioritize must-do 8.0 issues in coming days)

Project Issue Summary

Open Questions

AOB?

Next video conference

(We'll line up a meeting on the fly before end of July IF anything critical comes up associated with 8.0 release)

[hjoliver]

11 August

Recent activity

RD

• Improve migration guide cylc-doc#490
• Various bugfixes pre-8.0.0
• Trim various unused cylc-ui dependencies pre-8.0.0
• Release lead for 8.0.0
• Various bugfixes post-8.0.0

OS

HO

• Local Cylc 8 config, documentation, and training (in prep) for NIWA and NeSI HPC
  • scheduler run hosts, ssh command config, run-dir symlinks, interactive Slurm sessions, job platforms
• Isolate ICP and FCP graphs using the runahead limit cylc-flow#5036
• Db store force triggered cylc-flow#5023 (incl. ditch a bit of redundant DB interaction)
• (reviews and various bits of fluff)

DS

• Figure out how to migrate to Graphene v3
• CLI(/TUI) via UIS GraphQL
• Scheduler subscriptions (might tie into the first point)
• w/e else (reviews/fixes)

MH

• Working on Add gui workflow arg cylc-uiserver#370. New implementation using Jupyter Server pid file rather than maintaining our own.
• Off project on leave and Met Office Science event commitments.
• Working on rotating log for uiserver - new log on restarting uiserver, last 5 logs will be archived. logging: rollover on restart cylc-uiserver#352, pr incoming soon.

TP

• Cylc Linter
• Workshops at Met Office.
• Assorted bugfixes, notably the horrid polling bug
• Documenting the pre conf/post install plugin entry points

AC

Discussion Topics

1. 8.0 release debrief
   • went fine, pretty quick with the release team colocated
2. User feedback
   • SW @ BOM (incl, contact file issue below)
     • Jinja2 in global config came in handy
     • need to think about how to deal with server down for a long period, this would have the same effect the interactive Slurm session problem (in that case we may be able to scrape the ssh rejection message to infer that the workflow is not running)
   • laggy mutations menu pop-up (system or network load?) (Could this be Investigate slow performance of mutations menu after UI running for a while cylc-ui#913 ? - RD)
     • need to keep a watch on this and investigate
3. Site activity (seminars, training etc.)
   • MO seminars done for now; users not in a hurry to migrate
   • NIWA seminars start next week
4. Should HO go ahead now with another final Cylc 7 release?
   • still awaiting final cylc review changes; can go ahead after that
5. MO team immediate priorities (Rose?)
   • will need to start on rose edit and rosie UIs early 2023
6. UI Quick Start Guide needs some love. Can we reuse the recent MO UI "workshop" material?
   • ideally, use the actual components to generate pics and animations automatically via cypress tests, rather than static screen shots. Allegedly this won't be too difficult.
7. Pre-considered Open Questions and Issues
   • whats new page cylc-doc#314
     • approach agreed on, as per the issue
   • task metadata view cylc-ui#1071
     • agreed to use existing tab interface (hopefully with new tabs on top, across the board); and extend to comprehensive task - and possibly task instance - info as needed; similarly for workflow info
   • cylc scan --ping not removing contact files cylc-flow#5051
     • (comments above)
   • Dead-ended Cylc conda packages #155
     • we should be careful not to close off the cylc and cylc-ui packages from future use
     • HO to ask on Conda Forge gitter what the best approach is [DONE, on Element]
   • Add Spellchecking to Cylc Doc cylc-doc#501
     • decision: UK/NZ in the docs, US in the code
   • Use task icons in gscan, not job icons. cylc-ui#1050
     • agreed we need to resolve the current inconsistent use of job icons, but exactly what to do instead needs some thought
8. 8.0.1 release:
   • Soon, possibly with UIS too (they can be released individually)
9. Post-8.0 major priorities

• Graph view
• Job Log view
• task metadata view
• hub and cross-user operation
• gscan sorting (small?)
• cylc gui (almost done)
• stable Tui
• cylc-ui workflow events stream

13. Post-8.0 secondary priorities

• save tab layout
• trigger edit (may be tricky); need runtime config in the schema - also for task info view
• ditch vuetify in favour of "element"? - this is blocking migration to Vue3, which has performance advantages

Links to Issues

Project Issue Summary

Open Questions

AOB?

(workflow analytics, may be implement by another MO team)

Next video conference

mid-September

[hjoliver]

15 Sept

Recent activity

RD

• Fixing bugs

OS

• Graph view.
• UI data store work.
• Cylc 8 teething issues and support.

HO

• NIWA migration training, NeSI webinar, NZRSE conference demo
• various smallish PRs:
  • Fix post-reload trigger. cylc-flow#5104
  • Isolated start-up and shut-down graphs. cylc-flow#5090
  • parsec: keep whitespace inside quoted single-line values cylc-flow#5085
  • Tweak triggering when no flows present cylc-flow#5084
  • Fix deprecation warning format. cylc-flow#5119
  • fix install w/ .cylcignore cylc-flow#5066
  • Datastore fix for removed taskdef. cylc-flow#5067
• fixed bug in JupyterHub 🎉
• (various reviews)

DS

• Runtime config provision (with broadcast deltas).
• Node Edges bug
• CLI via UIS (only prototype, need client refactor done first)
• Reviews & other bug fixes.
• Many hours to spend/catch-up on.

MH

• Reviewing
• Small pr bug investigation rsync (which has surfaced in Cylc 7 too)
• Documentation
• Started looking at log view (ui-server end) work

TP

Features

• cylc vip cylc-flow#5121 - I'm waiting for this to get further through the reviewing process, but I think that the other compound CLI items should be easier than the first.
• pyproject.toml: cylc lint settings cylc-flow#5086 (I had some complaints from early adopters about the inability to ignore style complaints)

Maintaince and incremental improvements

• Rose Stem (Investigation and fixing bugs Rose Stem Fixes cylc-rose#172, feature: stem installs to workflow name not rose-stem cylc-rose#180, Improve rose stem coverage cylc-rose#181, Review rose stem docs cylc-rose#179)
• Cylc Rose related stuff (Don't pass rose variables with state ! or !! to Cylc. cylc-rose#171, reinstall changes to rose-suite.conf cylc-flow#5125)
• Horrible Rose Dependency issue (use any python>=2.7 for rsync check metomi/rose#2611)

AC

• drawer resizing
• gscan auto-collapse and summary icons
• comments on Vue.js and testing

Discussion Topics

8.0.2 released

cylc hub currently broken: need a UIS release pronto

final Cylc 7 release (rsync, cylc review...) - can we do it yet?

OS: update on graph view investigations

File-driven workflow example

BOM new HPC? (if JR present)

Quick review of priorities - can we assign people?

Links to Issues

Project Issue Summary

Open Questions

AOB?

(workflow analytics, may be implement by another MO team)

Next video conference

mid-September

[wxtim]

12 October

Recent activity

RD

• Beginning work on the new "edit runtime" broadcast-based replacement for trigger-edit.
• GH Actions stuff
• Identified restart incompatibility problem: Db upgrade 8.0.3 cylc-flow#5187
• Table view: use natural sort order for cycle point column cylc-ui#1107

OS

• Graph view & UI data store

HO

• a couple of SOD-related scheduler bug fixes and improvements
• reviews
• local C8 support and bug-chasing
• (school holidays; NIWA + Altair plans; BOM visit plans)

DS

MH

• Update on Log Work: Have logs streaming to the ui (pr nearly ready for ui-server). Currently looking at work related to ui end.
• Platform documentation
• Reviews etc..

TP

Features

• cylc vip cylc-flow#5121 - I'm waiting for this to get further through the reviewing process, but I think that the other compound CLI items should be easier than the first.
  complaints)
• Get Cylc template variables from database for validation. cylc-flow#5189 - Validating workflows that have/are played/playing using template vars stored in database.

Maintaince and incremental improvements

• Helping a couple of new users with C8
• Messing with Rose Stem
• Upping coverage of Cylc-Rose

AC

Discussion Topics

• HO planning to start looking at multi-user and hub issues
• Release 8.0.3 end of week, get as many bug fixes merged as we can in next day or two
• HO to ask JT/TAG about extending funding for DS
• cylc remove: can remaining use cases be replaced by manual task expiration? (Do we have a "remove" issue already?)

Links to Issues

AOB?

Next video conference

Week of 14 Nov