* Add restore validation feature: restores to special keyspace allowing validating backup/restore in single cluster (space willing) (#12573)
* Add restore validation feature with simplified backup gap fix
Implements restore validation using audit_storage to verify backup/restore
correctness. Includes a minimal fix for the backup gap bug.
Key components:
- ValidateRestore audit type: compares source keys against restored keys
at \xff\x02/rlog/ prefix in storage server
- DD audit fixes: propagate validation errors, handle DD failover correctly
- RestoreValidation and BackupAndRestoreValidation workloads for testing
- Simplified backup gap fix: prevent snapshot from finishing in the same
iteration it dispatches the last tasks (single flag + one check)
* No faultInjection in 7.4
* Make it so no encryption when the restore validation test runs
* Add error_code_audit_storage_task_outdated to bypass list rather than do special-case handling
---------
Co-authored-by: michael stack <stack@duboce.com>
Redwood complains DB is invalid for this unit test:
Assertion keyProvider.isValid() || db.isValid() failed @ /root/src/foundationdb/fdbserver/VersionedBTree.actor.cpp 8029:
This test validates that in-flight commit to the storage engine is properly
handled. As found in https://github.com/apple/foundationdb/pull/10714, an
engine could misses in-flight data and cause data corruptions.
The test case is modeled after the above corruption: insert data, then clear
the data in the next commit to the storage engine, and finally varify that the
data is cleared.
The only case when a coordination server should fail with file_not_found
is when the DiskQueue only finds 1 out of its 2 queue files. This either
means data has been lost, or the initial creation of the two files failed
partway through. In either case, it's sensible to crash the process and
refuse to start.
The existing code was catching the file_not_found error and letting the
FDBD process continue after the coordination server failed. This resulted
in clients attempting to contact the coordination server on a well-known
endpoint and never getting a response, because Flow transport silently
dropped the request when there was no receiver for it. By terminating the
process instead, we effectively notify waiting clients that they won't
get a response from this coordinator.
Michael Stack