Watcher event wait

[kramvan1]

Includes the following changes

• Rule lock retries
  • new Option RuleWatcherLockRetries
    • When the rule source is the Watcher, it will optionally retry getting the lock
    • RuleWatcherLockRetries(tries uint, wait time.Duration)
  • new lock label for watcher lock attempt metrics
• Watcher log tweak
• Lock session log tweak

ENSURE THE FOLLOWING ARE MET:

Below is a brief summary of PR requirements (full list).

• Your PR title summarizes the changes at a high level (not simply "updated X" or "changes Y")
• Your PR description links to the issue/design. If not available, includes a full description for the change.
• Your code contains relevant unit tests.

By opening this PR for review, the author has agreed that these criteria must be met.

By approving this PR, the reviewers have also agreed these criteria have been met and it is ready to be merged.