DSP-24600 & CASSANDRA-20485 Fixes & test additions for handling of UDTs across versions

[djatnieks]

All big formats now have implicitly frozen tuples. Originally this was only an issue up to me, but C*'s na and nb formats are also implicitly frozen, are compatibility against them is required. Fix language around hasExplicitlyFrozenTuples versus hasImplicitlyFrozenTuples in TrieIndexFormat.

Extends SSTableHeaderFix to only run when dropped multi-cell columns are detected. Add a StartupCheck preventing upgrades from <3.0.25 and <3.11.11, only used by HCD, to reduce the liklihood of unfixed sstable headers being introduced into a running cluster (it is best if the operator manually fixes these headers before the upgrade to HCD 1.x (CC4 / C* 4.0 based).

Additional DSE tries legacy sstable data has been added in addition to the backport described below.

Backport of CASSANDRA-20485, as follows:
Add LegacySSTableTest data for past sstable formats

Also add test method testVerifyOldTupleSSTables for frozen tuples and dropping of them.

The test data also adds different versions of me format sstables from C* version 3.0.25 and 3.11.11 as they do (unfortunately) differ. Test sstable files have had their sequence generation ids changed to represent the server version used to create them, while this shouldn't be necessary it is for debug purposes when sstable formats have changed between versions.

patch by Roxana Neophytou, Mick Semb Wever; reviewed by Dmitry Konstantinov for CASSANDRA-20485

[github-actions]

Checklist before you submit for review

• This PR adheres to the Definition of Done
• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the DataStax copyright header instead of the Apache License one

[djatnieks]

Lot of merge conflicts in LegacySSTableTest that I ended up comparing main and main-5.0 versions line by line.

Two of these tests are failing: testVerifyOldDroppedTupleSSTables, which was already failing on main-5.0 and testInaccurateSSTableMinMax which afaict only fails due to the first failing test - e.g. running testInaccurateSSTableMinMax alone passes and marking testVerifyOldDroppedTupleSSTables with @Ignore and running the entire class also will also pass testInaccurateSSTableMinMax .

[driftx]

I see merge markers here

[djatnieks]

I see merge markers here

Oh, thanks. I fixed that and then I saw rat-check complaining about .keep files, so I updated build-rat.xml to ignore those.

I pushed an updated branch now.

[sonarqubecloud]

Quality Gate failed

Failed conditions
79.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-1980 rejected by Butler

---

5 regressions found
See build details here

---

Found 5 new test failures

Test	Explanation	Runs	Upstream
o.a.c.distributed.test.hostreplacement.HostReplacementTest.replaceAliveHost	REGRESSION	🔴🔵	0 / 2
o.a.c.distributed.test.hostreplacement.HostReplacementTest.replaceDownedHost	REGRESSION	🔴🔵	0 / 2
o.a.c.distributed.test.hostreplacement.HostReplacementTest.retryingFailedReplaceWithNodeInHibernateState	REGRESSION	🔴🔵	0 / 2
o.a.c.distributed.test.hostreplacement.HostReplacementTest.seedGoesDownBeforeDownHost	REGRESSION	🔴🔵	0 / 2
o.a.c.io.sstable.LegacySSTableTest.testInaccurateSSTableMinMax (compression)	REGRESSION	🔴🔴	0 / 2

---

Found 13 known test failures

[driftx]

Heh, well now the test is complaining about it: java.lang.IllegalArgumentException: Invalid sstable file .keep: the name doesn't look like a supported sstable file name

[djatnieks]

Heh, well now the test is complaining about it: java.lang.IllegalArgumentException: Invalid sstable file .keep: the name doesn't look like a supported sstable file name

Right, that's what I was describing in #1980 (comment)

testInaccurateSSTableMinMax gets that error about the .keep file, but only when the entire test class runs and testVerifyOldDroppedTupleSSTables runs and fails first.

And testVerifyOldDroppedTupleSSTables is a known failure on main-5.0 already. I'm pretty sure when that is fixed, then testInaccurateSSTableMinMax will pass too.

[djatnieks]

I created https://github.com/riptano/cndb/issues/15292 for the testVerifyOldDroppedTupleSSTables that is already failing on main-5.0.