CNDB-15919: Optimize SAI NOT queries, push logic into posting lists

[michaeljmarshall]

What is the issue

Fixes: https://github.com/riptano/cndb/issues/15919
Test PR: https://github.com/riptano/cndb/pull/15949

What does this PR fix and why was it fixed

In the original implementation for #820, we introduced the PrimaryKeyMapIterator to iterate all primary keys in an sstable and then do an anti-join on the result of an equality query. That design works, but requires some additional reads from disk to get primary keys that are unnecessary.

There are two possible solutions:

1. We can use row ids (either sstable or segment) to do the complement of the resulting posting lists. This will be the most performant, since it avoids object allocations. The main issue with this solution is that it is much more complicated to implement and had unaddressed edge cases.
2. We can use the primaryKeyFromRowId that takes primary key bounds and then uses a row id, when rows are from the same sstable. This will be worse that solution 1 because it creates an object per key and requires comparing sstable ids before comparing sstable row ids, but it is a significant improvement over the current solution, which hits disk to load the primary key.

When testing on my local machine and reviewing the JMH benchmarks, I can see that the current solution is about 16x worse than the minimum solution (2) and 32x worse than the optimal (1) solution. Given that the benchmarks in question are highly specific to the use case, I do no think we have sufficient motivation to introduce the exceedingly complex (1) solution.

Note that the ideal solution to 1, that would have much less complexity, is to convert posting lists into a single iterator of sstable row ids, and then to take the complement of them.

[github-actions]

Checklist before you submit for review

• This PR adheres to the Definition of Done
• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the DataStax copyright header instead of the Apache License one

[pkolaczk]

In addition to the other minor things I mentioned, we need a test for verifying how NEQ works together with token range / partition key restriction.

[michaeljmarshall]

@pkolaczk - I added a benchmark and discovered that in the most extreme cases, we can get most of the benefit for an extremely small code change, compared to what I had proposed before. I think we should go with this one for now and leave a deeper re-write for later, if there is demand.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-2112 rejected by Butler

---

1 regressions found
See build details here

---

Found 1 new test failures

Test	Explanation	Runs	Upstream
o.a.c.index.sai.cql.datamodels.QueryWriteLifecycleWithCompositePartitionKeyTest.terminated successfully	NEW	🔴⚪	0 / 17

---

Found 1 known test failures