CNDB-8641: Add a metric to count all request errors

[cbornet]

What is the issue

https://github.com/riptano/cndb/issues/8641

PR in CNDB: https://github.com/riptano/cndb/pull/15306

What does this PR fix and why was it fixed

This pull request introduces new metrics for tracking invalid and other error requests for CQL statements, and integrates these metrics into the query failure notification logic. The main changes are the addition of the AllRequestsMetrics class, updates to the ClientRequestsMetrics class to include these new metrics, and modifications to the query event notification methods to record errors using the new metrics.

Metrics added:

• org.apache.cassandra.metrics.ClientRequest.Timeouts.All
• org.apache.cassandra.metrics.ClientRequest.Unavailables.All
• org.apache.cassandra.metrics.ClientRequest.Failures.All
• org.apache.cassandra.metrics.ClientRequest.Invalid.All
• org.apache.cassandra.metrics.ClientRequest.OtherErrors.All

Note: only requests for which a tenant can be identified are counted.

[github-actions]

Checklist before you submit for review

• This PR adheres to the Definition of Done
• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the DataStax copyright header instead of the Apache License one

[JeremiahDJordan]

What about CQL statements with no keyspace specified? Or where the reason it's an IRE is the keyspace name was wrong? Will those be captured somewhere else?

[cbornet]

I think they will be captured in ClientMetrics.unknownException with all the other exceptions (with keyspace or not).
If we don't have a keyspace, I don't think we can identify a tenant ?

[eolivelli]

we cannot identify the tenant for sure
this feature is also for non-CNDB users

we should add tests cases to cover this case

[JeremiahDJordan]

Is this going to double count all the other exceptions like timeouts or other validation errors besides IRE?

[JeremiahDJordan]

We want to be able to put these metrics onto the grafana graph with the other errors that are happening. So we need to make sure things we collect make sense when viewed as a whole on the error chart.

[cbornet]

I added logic to count separately timeouts, unavailable and failures.
But note that these are counted both there and in read/write metrics.
Basically, allRequestsMetrics.timeouts = readMetrics.timeouts + writeMetrics.timeouts
It should be possible to have a dashboard from allRequestsMetrics alone.

[cbornet]

Note that some errors are already counted twice in the dashboard.
For instance a read timeout increments both coordinator_read_requests_timeouts_total and coordinator_cas_read_requests_timeouts_total.

[JeremiahDJordan]

Needs tests, and need to make sure the metrics make sense when viewed as a whole with the other exception metrics that are captured. But I think this is a good starting point.
You might make a dedicated tenant in astra dev and deploy this so that you can try it out and make sure the metrics make it into the grafana there.

[cbornet]

Yes. Tests are on the TODO. I just wanted to be sure that it's going in the right direction (so much for TDD...)

[cbornet]

@JeremiahDJordan I added some tests.

[JeremiahDJordan]

Looks good. Thanks!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
86.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cassci-bot]

✔️ Build ds-cassandra-pr-gate/PR-1983 approved by Butler

---

Approved by Butler
See build details here