CDNB-15253: make UCS shard progression smoother when num_shards and min sstable size are configured #2031

jasonstack · 2025-10-01T03:26:08Z

What is the issue

https://github.com/riptano/cndb/issues/15253 large shard count jump caused HCD-130

What does this PR fix and why was it fixed

Replace power-of-two shard progression with factorization-based smooth growth when num_shards is explicitly configured. This prevents problematic large jumps that can cause data loss (e.g., HCD-130). For example, with num_shards 1000 which is 5^3* 2^3:

Before: 1 → 2 → 8 → 1000 (125x jump causes data loss due to hcd-130)
After: 1 → 5 → 25 → 125 → 250 → 500 → 1000 (max 5x jump)

new behavior can be disabled via -Dunified_compaction.use_factorization_shard_count_growth

github-actions · 2025-10-01T03:26:29Z

blambov · 2025-10-01T10:40:26Z

src/java/org/apache/cassandra/db/compaction/unified/Controller.java

+    static final boolean USE_FACTORIZATION_SHARD_COUNT_GROWTH = Boolean.parseBoolean(getSystemProperty("use_factorization_shard_count_growth", "true"));
+
+    /** Cache for factorized shard sequences to avoid repeated calculations for the same target values */
+    private static final Map<Integer, List<Integer>> shardSequenceCache = new ConcurrentHashMap<>();


Instead of making this a static cache, we can calculate it for the given baseShardCount during initialization (the count is final, so we can make the calculation final as well).

The cost of calculating the factors is not negligible (i.e. we could not do it on every getNumShards call), but is not as high as to prohibit it being done on every Controller construction, and it will save the map lookup on every getNumShards.

good idea, updated to private final Optional<int[]> factorizedShardSequence

blambov · 2025-10-01T10:41:49Z

src/java/org/apache/cassandra/db/compaction/unified/Controller.java

+    @VisibleForTesting
+    boolean useFactorizationShardCountGrowth()
+    {
+        return USE_FACTORIZATION_SHARD_COUNT_GROWTH && sstableGrowthModifier == 1.0 && baseShardCount > 0;


We should always use this calculation whenever baseShardCount is not a power of two, regardless of sstableGrowthModifier.

The only reason to not use it for powers of two is that the existing formula provides the same result quicker.

changed to USE_FACTORIZATION_SHARD_COUNT_GROWTH && baseShardCount > 0 && Integer.bitCount(baseShardCount) != 1

blambov · 2025-10-01T10:43:21Z

src/java/org/apache/cassandra/db/compaction/unified/Controller.java

+                }
+                else
+                {
+                    // Keep original power-of-two logic for dynamic growth mode


In dynamic growth mode, we also respect baseShardCount and it makes no sense to not apply the smoother logic there as well.

removed this comment..

blambov · 2025-10-01T10:46:29Z

src/java/org/apache/cassandra/db/compaction/unified/Controller.java

+        int result = 1;
+        for (int shards : shardSequence)
+        {
+            if (shards <= currentCountBasedOnDensity)


Why not a binary search?

updated to binary search. Initially I thought iterating array would be as fast because of cpu cache

blambov · 2025-10-01T10:49:25Z

src/java/org/apache/cassandra/db/compaction/unified/Controller.java

+
+        // Check odd factors up to sqrt(num)
+        int sqrtNum = (int) Math.sqrt(num);
+        for (int factor = 3; factor <= sqrtNum; factor += 2)


We can do
for (int factor = 3; factor*factor <= num; factor += 2)
to avoid the square root and also finish quicker by taking into account the reduction of num.

updated to for (int factor = 3; factor <= num / factor; factor += 2). this is safer..

You are right, the multiplication will overflow when the limit is e.g. Integer.MAX_VALUE.

I'd personally still use 1L * factor * factor <= num which does not involve division, but it's a moot point since we are performing the modulus for the same division on the next line anyway, and a compiler could optimize one of them away.

blambov · 2025-10-01T10:55:23Z

src/java/org/apache/cassandra/db/compaction/unified/Controller.java

+        List<Integer> primes = primeFactors(target);
+
+        // 2) Sort factors in descending order
+        primes.sort(Comparator.reverseOrder());


You can use primes.reversed() or just do the loop by decreasing index.

updated to loop by decreasing index

blambov · 2025-10-01T10:55:49Z

src/java/org/apache/cassandra/db/compaction/unified/Controller.java

+        primes.sort(Comparator.reverseOrder());
+
+        // 3) Cumulative product to form the chain.
+        List<Integer> divisors = new ArrayList<>();


I would use int[] (we know the size at this point).

updated to int[]

blambov · 2025-10-02T08:31:37Z

src/java/org/apache/cassandra/db/compaction/unified/Controller.java

+
+        // insertion point
+        int insertionPoint = -idx - 1;
+        int candidateIndex = Math.max(0, insertionPoint - 1);


Nit: max is not needed here (-(-1) -1 is 0). Generally it could result in sequence.length, but that will happen only if the count is larger than the base shard count, which won't end up in this function.

Right, -idx - 1 is always positive. It's searching for the value before insertion point via insertionPoint - 1, so it uses max when insertionPoint is 0.

blambov · 2025-10-02T08:56:38Z

test/unit/org/apache/cassandra/db/compaction/unified/ControllerTest.java

+        // Test 3200 = 5^2 * 2^7
+        int[] expected3200 = new int[]{ 1, 5, 25, 50, 100, 200, 400, 800, 1600, 3200 };
+        assertArrayEquals(expected3200, Controller.factorizedSmoothShardSequence(3200));
+    }


Let's add a test with Integer.MAX_VALUE as well.

blambov · 2025-10-02T08:58:30Z

src/java/org/apache/cassandra/db/compaction/unified/Controller.java

+
+        // Check odd factors up to sqrt(num)
+        int sqrtNum = (int) Math.sqrt(num);
+        for (int factor = 3; factor <= sqrtNum; factor += 2)


You are right, the multiplication will overflow when the limit is e.g. Integer.MAX_VALUE.

I'd personally still use 1L * factor * factor <= num which does not involve division, but it's a moot point since we are performing the modulus for the same division on the next line anyway, and a compiler could optimize one of them away.

blambov · 2025-10-02T08:59:51Z

src/java/org/apache/cassandra/db/compaction/unified/Controller.java

+     * use prime factorization to create smooth sequences like 1→5→25→125→250→500→1000 for num_shards=1000.
+     * This prevents the large jumps that can cause data loss issues (e.g., HCD-130).
+     * <p>
+     * Only applies when num_shards is explicitly configured


This line is no longer correct, and maybe the first line of the doc should also be changed to base_shard_count not being a power of two.

src/java/org/apache/cassandra/db/compaction/unified/Controller.java

…nt growth when num_shards and min sstable size are configured * Replace power-of-two shard progression with factorization-based smooth growth when num_shards is explicitly configured. This prevents problematic large jumps that can cause data loss (e.g., HCD-130). For example, with num_shards 1000 which is 5^3* 2^3: Before: 1 → 2 → 8 → 1000 (125x jump causes data loss) After: 1 → 5 → 25 → 125 → 250 → 500 → 1000 (max 5x jump) * new behavior can be disabled via `-Dunified_compaction.use_factorization_shard_count_growth`

sonarqubecloud · 2025-10-03T01:42:47Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
97.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

cassci-bot · 2025-10-03T01:48:10Z

❌ Build ds-cassandra-pr-gate/PR-2031 rejected by Butler

2 regressions found
See build details here

Found 2 new test failures

Test	Explanation	Runs	Upstream
o.a.c.distributed.test.FullRepairCoordinatorTimeoutTest.prepareRPCTimeout[SEQUENTIAL/false]	REGRESSION	🔴🔵	0 / 7
o.a.c.index.sai.cql.VectorKeyRestrictedOnPartitionTest.partitionRestrictedWidePartitionBqCompressedTest[ec] (compression)	REGRESSION	🔴🔵	0 / 7

Found 1 known test failures

…shards and min sstable size are configured (#2031) riptano/cndb#15253 large shard count jump was involved in HCD-130 Replace power-of-two shard progression with factorization-based smooth growth when base shard count is not power of 2. This prevents problematic large jumps. For example, with num_shards 1000 which is 5^3* 2^3: - Before: 1 → 2 → 8 → 1000 (125x jump causes data loss due to hcd-130) - After: 1 → 5 → 25 → 125 → 250 → 500 → 1000 (max 5x jump) * new behavior can be disabled via `-Dunified_compaction.use_factorization_shard_count_growth`

jasonstack requested a review from blambov October 1, 2025 03:26

blambov reviewed Oct 1, 2025

View reviewed changes

jasonstack requested a review from blambov October 2, 2025 03:47

jasonstack force-pushed the cndb-15253-main branch from 0e0a45e to 8fff3f4 Compare October 2, 2025 04:03

blambov approved these changes Oct 2, 2025

View reviewed changes

jasonstack added 3 commits October 3, 2025 08:56

Apply review feedback

9118425

Apply review feedback

49cca51

jasonstack force-pushed the cndb-15253-main branch from d66ce6f to 49cca51 Compare October 3, 2025 00:56

jasonstack merged commit 48eb797 into main Oct 3, 2025
487 of 495 checks passed

jasonstack deleted the cndb-15253-main branch October 3, 2025 08:21

CDNB-15253: make UCS shard progression smoother when num_shards and min sstable size are configured #2031

CDNB-15253: make UCS shard progression smoother when num_shards and min sstable size are configured #2031

Uh oh!

Conversation

jasonstack commented Oct 1, 2025

What is the issue

What does this PR fix and why was it fixed

Uh oh!

github-actions bot commented Oct 1, 2025 • edited by jasonstack Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist before you submit for review

Uh oh!

blambov Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jasonstack Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jasonstack Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud bot commented Oct 3, 2025

Quality Gate passed

Uh oh!

cassci-bot commented Oct 3, 2025

❌ Build ds-cassandra-pr-gate/PR-2031 rejected by Butler

Found 2 new test failures

Found 1 known test failures

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Oct 1, 2025 •

edited by jasonstack

Loading

blambov Oct 1, 2025 •

edited

Loading

jasonstack Oct 2, 2025 •

edited

Loading

jasonstack Oct 2, 2025 •

edited

Loading