CNDB-14207: Don't mark the index non-queryable if flush fails (#1770)

[djatnieks]

If the index fails to build, mark the index non-queryable, but don't fail the C* flush. This way the node can continue to run the other queries.

[github-actions]

Checklist before you submit for review

• This PR adheres to the Definition of Done
• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the DataStax copyright header instead of the Apache License one

[djatnieks]

@pkolaczk After porting CNDB-14207 to main-5.0 branch there are a couple of test failures in tests that come from Apache C* 5.0. I am hoping you would have more context and could recommend how to resolve these - do we change the test expectations, or should the code be adjusted for these test cases?

https://jenkins-stargazer.aws.dsinternal.org/job/ds-cassandra-pr-gate/job/PR-1970/1/testReport/org.apache.cassandra.distributed.test.sai/ImportIndexedSSTablesTest/tests_stage_1___jvm_distributed___testIndexBuildingFailureDuringImport__jdk11/

https://jenkins-stargazer.aws.dsinternal.org/job/ds-cassandra-pr-gate/job/PR-1970/1/testReport/org.apache.cassandra.distributed.test.sai/IndexStreamingFailureTest/tests_stage_1___jvm_distributed___testAvailabilityAfterFailedNonEntireFileStreaming__jdk11/

[driftx]

I guess ticket the tests?

[djatnieks]

I guess ticket the tests?

I'll see what @pkolaczk can suggest - waiting for Jenkins to be available again - but, yeah, if needed we can do that.

[djatnieks]

I investigated the failing tests and found they are Apache C* tests that were not expecting the change in behavior made by this ticket. I have adjusted the tests to now expect a IndexNotAvailableException following the errors that are injected during the test.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-1970 rejected by Butler

---

2 regressions found
See build details here

---

Found 2 new test failures

Test	Explanation	Runs	Upstream
o.a.c.distributed.test.sai.ImportIndexedSSTablesTest.testIndexBuildingFailureDuringImport	REGRESSION	🔵🔴	0 / 2
o.a.c.distributed.test.sai.IndexStreamingFailureTest.testAvailabilityAfterFailedNonEntireFileStreaming	REGRESSION	🔵🔴	0 / 2

---

Found 13 known test failures