[router] make a distinction between fatal errors, warnings and unknown as healthcheck result

[DaanHoogland]

Description

This PR is a small functional change inspired by #9800, #9852 and #9473. More related issues may exist. The intention is to introduce the concept of non fatal failures in the router health check scripts. In the past only success and failure were reported. Now also the statuses warning and unknown are available.

The scripts,

• cpu_usage_check.py,
• dhcp_check.py and
• diskspace_check.py
  were changed, but the machanism may be applied to other scripts as well.

monitor.py has been changed to relay the extra possible statuses, and the backend and UI have been altered to store them and display them to the user.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

1. create a VPC
2. create a tier
3. create a VM in the new tier
4. wait for the VM to come up
5. delete the VM
6. .... (wait) once the vr has shut the tier down the VR won't have a webservice for the tier anymore and will report a failure which is a false positive. After the change it should show as UNKNOWN instead of FAILED

How did you try to break this feature and the system with this change?

added some fake tests to test the UI:

[codecov]

Codecov Report

❌ Patch coverage is 13.48684% with 263 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.39%. Comparing base (3d6ec29) to head (39fdd8d).
⚠️ Report is 56 commits behind head on main.

Files with missing lines	Patch %	Lines
...ork/router/VirtualNetworkApplianceManagerImpl.java	4.82%	134 Missing and 4 partials ⚠️
...tack/engine/orchestration/NetworkOrchestrator.java	9.52%	57 Missing ⚠️
...a/com/cloud/network/router/CommandSetupHelper.java	0.00%	22 Missing and 2 partials ⚠️
...loud/network/lb/LoadBalancingRulesManagerImpl.java	0.00%	13 Missing ⚠️
...src/main/java/com/cloud/api/ApiResponseHelper.java	0.00%	9 Missing ⚠️
.../api/response/RouterHealthCheckResultResponse.java	0.00%	8 Missing ⚠️
.../src/main/java/com/cloud/configuration/Config.java	50.00%	8 Missing ⚠️
...m/cloud/network/dao/RouterHealthCheckResultVO.java	0.00%	2 Missing ⚠️
.../apache/cloudstack/framework/config/ConfigKey.java	66.66%	2 Missing ⚠️
...oud/network/lb/ElasticLoadBalancerManagerImpl.java	0.00%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #10710      +/-   ##
============================================
+ Coverage     17.36%   17.39%   +0.03%     
- Complexity    15237    15281      +44     
============================================
  Files          5888     5890       +2     
  Lines        525741   526155     +414     
  Branches      64164    64233      +69     
============================================
+ Hits          91274    91528     +254     
- Misses       424167   424282     +115     
- Partials      10300    10345      +45     

Flag	Coverage Δ
uitests	3.62% <ø> (-0.01%)	⬇️
unittests	18.44% <13.48%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[Pearl1594]

We probably need to raise this against 4.19?

[harikrishna-patnala]

code LGTM

[vishesh92]

clgtm

[DaanHoogland]

this has been (dev-)tested but needs an upgrade test as most important validation.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15063

[blueorangutan]

[SF] Trillian test result (tid-14379)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 54023 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10710-t14379-kvm-ol8.zip
Smoke tests completed. 145 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_scale_kubernetes_cluster	Failure	1.17	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	Failure	27.82	test_kubernetes_clusters.py
test_01_deployVMInSharedNetwork	Failure	410.15	test_network.py

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 15111

[harikrishna-patnala]

@DaanHoogland is this failure related "test_01_deployVMInSharedNetwork" ?

[weizhouapache]

@DaanHoogland is this failure related "test_01_deployVMInSharedNetwork" ?

maybe just a intermittent failure

@blueorangutan package

[blueorangutan]

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15123

[weizhouapache]

@blueorangutan test

[blueorangutan]

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-14410)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 52362 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10710-t14410-kvm-ol8.zip
Smoke tests completed. 147 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[DaanHoogland]

@vishesh92 @harikrishna-patnala @weizhouapache , there is a slight backwards incompatibility with this change; success can now be true, false or not present. In addition to result there is status, which can be “success”, “failure”, “warning” or “unknown”. So the incompatibilty is that result may not be present. I think this is not grave as it would be a false positive in the past. What do you think?

Other than this we are good to merge (@vladimirpetrov tested but forgot to give his opinion here;)

[blueorangutan]

[SF] Trillian test result (tid-14415)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50207 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10710-t14415-kvm-ol8.zip
Smoke tests completed. 147 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[harikrishna-patnala]

@vishesh92 @harikrishna-patnala @weizhouapache , there is a slight backwards incompatibility with this change; success can now be true, false or not present. In addition to result there is status, which can be “success”, “failure”, “warning” or “unknown”. So the incompatibilty is that result may not be present. I think this is not grave as it would be a false positive in the past. What do you think?

Other than this we are good to merge (@vladimirpetrov tested but forgot to give his opinion here;)

as we are only updating the value of the response parameter properly I think this is fine and moreover more detailed 'status' is now present.