Add restore validation feature: restores to special keyspace allowing validating backup/restore in single cluster (space willing) #12573

saintstack · 2025-11-21T20:49:11Z

This is an implementation of a Neethu design (design is included in the PR).

Add restore validation feature with simplified backup gap fix

Implements restore validation using audit_storage to verify backup/restore
correctness. Includes a minimal fix for the backup gap bug.

Key components:
- ValidateRestore audit type: compares source keys against restored keys
  at \xff\x02/rlog/ prefix in storage server
- DD audit fixes: propagate validation errors, handle DD failover correctly
- RestoreValidation and BackupAndRestoreValidation workloads for testing
- Simplified backup gap fix: prevent snapshot from finishing in the same
  iteration it dispatches the last tasks (single flag + one check)

Backup gap bug fix (FileBackupAgent.actor.cpp):
The original dispatcher marks ranges as DONE when selecting them for dispatch,
then immediately checks if all ranges are done. This causes snapshots to finish
before the dispatched tasks complete, creating gaps in backup coverage.

The fix adds a dispatchedInThisIteration flag. If tasks were dispatched in
this iteration, the completion check is skipped, ensuring at least one full
loop between dispatch and completion. This minimal change prevents premature
snapshot completion without complex state tracking.

Here is running the new simulation included here 100k times:

20251121-184138-stack-1458b890ad727389 compressed=True data_size=55635346 duration=3717292 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:44:53 sanity=False started=100000 stopped=20251121-192631 submitted=20251121-184138 timeout=5400 username=stack

I ran all tests 100k times and looks like hangs on the end at 99975 or so. Looking to see if related.

Also verified the feature manually.

foundationdb-ci · 2025-11-21T21:13:15Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 2526933
Duration 0:23:53
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T21:19:06Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 2526933
Duration 0:29:44
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /opt/homebrew/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T21:34:36Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 2526933
Duration 0:45:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T21:35:17Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 2526933
Duration 0:45:54
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T21:51:51Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 5c53ce8
Duration 0:04:04
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T21:52:06Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 5c53ce8
Duration 0:04:22
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T21:52:10Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 5c53ce8
Duration 0:04:23
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T21:52:11Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 5c53ce8
Duration 0:04:23
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T21:52:15Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 5c53ce8
Duration 0:04:29
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-11-21T22:06:02Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 2526933
Duration 1:16:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-11-21T22:29:22Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 761b593
Duration 0:27:25
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T22:48:40Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 761b593
Duration 0:46:43
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T22:59:30Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 2526933
Duration 2:10:08
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T23:06:41Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 2526933
Duration 2:17:20
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-21T23:13:03Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 761b593
Duration 1:11:06
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-11-22T00:15:56Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 761b593
Duration 2:14:02
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-22T00:28:43Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 761b593
Duration 2:26:47
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-22T01:14:09Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: dcca4b7
Duration 0:24:30
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-22T01:25:28Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: dcca4b7
Duration 0:35:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-22T01:35:19Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: dcca4b7
Duration 0:45:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-22T01:42:57Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: dcca4b7
Duration 0:53:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-22T01:58:49Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: dcca4b7
Duration 1:09:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-22T02:03:52Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: dcca4b7
Duration 1:14:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-11-22T03:07:42Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: dcca4b7
Duration 2:18:06
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-23T06:16:52Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 77618b6
Duration 0:04:25
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-23T06:16:55Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 77618b6
Duration 0:04:27
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-11-23T06:16:56Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 77618b6
Duration 0:04:25
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-16T22:41:08Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: ac07a89
Duration 0:22:57
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-16T22:51:48Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: ac07a89
Duration 0:33:36
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-16T23:01:35Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: ac07a89
Duration 0:43:21
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-16T23:08:34Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: ac07a89
Duration 0:50:27
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-16T23:10:24Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: ac07a89
Duration 0:52:11
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-16T23:10:45Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: ac07a89
Duration 0:52:33
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-17T00:18:12Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: ac07a89
Duration 2:00:01
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-12-17T20:07:09Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 9057bd0
Duration 0:24:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-17T20:27:51Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 9057bd0
Duration 0:45:35
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-17T20:51:46Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 9057bd0
Duration 1:09:32
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-17T20:53:04Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 9057bd0
Duration 1:10:51
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-17T21:40:20Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 9057bd0
Duration 1:58:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

neethuhaneesha · 2025-12-18T20:36:19Z

fdbserver/storageserver.actor.cpp

+		} else {
+			// Extra key in restored data (skip it, as per design doc: one-directional comparison)
+			// Log the extra key for visibility without failing the validation
+			TraceEvent(SevInfo, "SSAuditRestoreExtraKey", thisServerID)


Let's add it ass 'TraceEvent(SevInfo, "SSAuditRestoreExtraKey", just to be in-sync with other failures.
Let also push it in errors.push_back(error);
We want the user to know that there are some keys and it's not valid data. Lets treat this error as other errors found.

No problem. Done.

neethuhaneesha · 2025-12-18T20:42:11Z

fdbserver/storageserver.actor.cpp

+	}
+
+	// Check for any remaining source keys that are missing from restored data
+	if (errors.empty() && sourceIdx < sourceReply.data.size() && !sourceReply.more) {


Why this !sourceReply.more ?

We are checking end of the src and dest... Want to see if one side has more than the other and error if so (But above should be better.... I added && !restoredReply.more and only then error)

I did no understand, why it it not an error if we have sourceReply.more is true ? Is it still be an error?

* If extra key in restored data, report as error rather than as info. * Tighten up check on end where one side -- src or dest -- might have more data... if so, error.

foundationdb-ci · 2025-12-19T22:44:59Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 2b740ad
Duration 0:28:40
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-19T23:01:49Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 2b740ad
Duration 0:45:34
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-19T23:27:23Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 2b740ad
Duration 1:11:07
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-19T23:34:28Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 2b740ad
Duration 1:18:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-20T00:19:43Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 2b740ad
Duration 2:03:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

neethuhaneesha · 2025-12-20T00:37:19Z

fdbserver/storageserver.actor.cpp

+			// Extra key in restored data
+			std::string error =
+			    format("Extra key in restored data: %s", Traceable<StringRef>::toString(restoredKV.key).c_str());
+			TraceEvent(SevError, "SSAuditRestoreExtraKey", thisServerID)


Is it intentional to use SSAuditRestoreExtraKey instead of SSAuditRestoreError

neethuhaneesha · 2025-12-20T00:38:04Z

fdbserver/storageserver.actor.cpp

+	// Check for any remaining restored keys that don't have matching source keys
+	if (errors.empty() && restoredIdx < restoredReply.data.size() && !restoredReply.more) {
+		// Log the extra keys found in restored data
+		TraceEvent(SevInfo, "SSAuditRestoreExtraRestoredKeys", thisServerID)


Shall we change this also to SevError and push them into errors?
SSAuditRestoreExtraRestoredKeys->SSAuditRestoreError

neethuhaneesha · 2025-12-20T00:43:08Z

fdbserver/storageserver.actor.cpp

+		tr.setVersion(version);
+		tr.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
+		tr.setOption(FDBTransactionOptions::LOCK_AWARE);
+		state RangeResult restoredData = wait(tr.getRange(restoredRange, limit, Snapshot::False, Reverse::False));


In the scenario of limitBytes for reply reaches, does the getRange(restoreRange) will give less keys than getRange(rangeToRead) because of bigger key size in restoreRange?

If yes, how is it handled in the code?

foundationdb-ci · 2025-12-23T01:52:57Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 3872f35
Duration 0:22:31
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-23T02:04:42Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 3872f35
Duration 0:34:14
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-23T02:14:05Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 3872f35
Duration 0:43:41
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-23T02:19:13Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 3872f35
Duration 0:48:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-23T02:32:21Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 3872f35
Duration 1:01:55
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-23T02:33:58Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 3872f35
Duration 1:03:32
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-23T03:31:06Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 3872f35
Duration 2:00:39
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

saintstack requested review from akankshamahajan15 and neethuhaneesha November 21, 2025 20:49

saintstack force-pushed the restore-validation-simple branch from dcca4b7 to 77618b6 Compare November 23, 2025 06:12

saintstack force-pushed the restore-validation-simple branch from cafb643 to ac07a89 Compare December 16, 2025 22:18

Address review comments

9057bd0

neethuhaneesha reviewed Dec 18, 2025

View reviewed changes

Address review comments.

2b740ad

* If extra key in restored data, report as error rather than as info. * Tighten up check on end where one side -- src or dest -- might have more data... if so, error.

neethuhaneesha reviewed Dec 20, 2025

View reviewed changes

Address Neethu Feedback

3872f35

Add restore validation feature: restores to special keyspace allowing validating backup/restore in single cluster (space willing) #12573

Are you sure you want to change the base?

Add restore validation feature: restores to special keyspace allowing validating backup/restore in single cluster (space willing) #12573

Uh oh!

Conversation

saintstack commented Nov 21, 2025

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 21, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 22, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 22, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 22, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 22, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Nov 22, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 22, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Nov 22, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 22, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 22, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 23, 2025