* Add restore validation feature: restores to special keyspace allowing validating backup/restore in single cluster (space willing) (#12573)
* Add restore validation feature with simplified backup gap fix
Implements restore validation using audit_storage to verify backup/restore
correctness. Includes a minimal fix for the backup gap bug.
Key components:
- ValidateRestore audit type: compares source keys against restored keys
at \xff\x02/rlog/ prefix in storage server
- DD audit fixes: propagate validation errors, handle DD failover correctly
- RestoreValidation and BackupAndRestoreValidation workloads for testing
- Simplified backup gap fix: prevent snapshot from finishing in the same
iteration it dispatches the last tasks (single flag + one check)
* No faultInjection in 7.4
* Make it so no encryption when the restore validation test runs
* Add error_code_audit_storage_task_outdated to bypass list rather than do special-case handling
---------
Co-authored-by: michael stack <stack@duboce.com>
readLogData() may encounter an error but the caller is not waiting on the Future.
As a result, the caller waiting on the PromiseStream will become stuck. The fix
is to send the error to the PromiseStream.
Another fix is to call taskBucket->keepRunning() in writeMutations(), which is
part of the work in _execute().
20251009-164341-jzhou-9217f131839ff7a4
* Enable Backup Encryption in Simulation Tests
* Enable encryption in BackupNew/OldRestore test and add fix
* Enable encryption in BackupAndParallelRestoreCorrectness workload
* Enable in RestoreBackup workload
* Add additional workloads
* Fix for Snap restart where second is not able to find if first run has
encryption key or not.
* Change the encryption probability to 50% in worloads
* Format
* Disable it in test and re-enable it later
* Separate backup and restore into two workloads
This allows more flexible testing as well as cleaner code.
* Keep sim backup agents after the Backup workload
Otherwise, the Restore workload doesn't have agents, thus can't make progress.
100k partitioned restore tests, i.e., BackupAndRestore.toml and
BackupCorrectnessPartitioned.toml:
20250311-200411-jzhou-9d34d22d5225d6fe
* Remove TransformPartitionedLog parameter from restore()
Instead, infer the flag from backup description.
* Add usePartitionedLog option for Backup workload
This allows us to specify old or new style of backup to be used. Added two
tests that switch between them and randomly choose one backup to restore.
20250314-034057-jzhou-c27ca23b6c69cecf
* Enable buggify in these backup tests
20250314-034541-jzhou-13ed090d0b111474
* Disable backup encryptions for two tests
Because we separate backup and restore into two workloads, they may not choose
the same encryption option, i.e., one encrypted and the other unencrypted.
20250320-013757-jzhou-12b4c8e4504ffd96
* A small refactor
20250321-222324-jzhou-fdcd6f145f3ac0f8
* Fix backup agent being randomly paused
This can cause subsequent backup and restore workload to fail.
20250322-040333-jzhou-15c32299d18f4456
100k backup tests:
20250322-040453-jzhou-2bdb4e0ddc265632
* Refactor RestoreLogDataPartitionedTaskFunc::_execute
* Batch multiple versions together when applying backup mutations
If done version by version, it is inefficient and causes the task to be
interrupted in simulation, thus never finishing the RestoreLogDataPartitionedTaskFunc.
20250324-040712-jzhou-cd8501d3890a6b56
100k backup tests:
20250324-040751-jzhou-8cec93182e6d3acb
* Correct how many log bytes are written during restore
20250326-182125-jzhou-375d243c097c3b5a
20250325-221139-jzhou-5dac71c4525d414c
100k backup tests:
20250326-162525-jzhou-f08e3fc12887a3e9
* Skip consistency check and Quiescence for backup tests
Do these checks in the last [[test]] specified in TOML file.
* Disable backup workers when no active partitioned backups
Currently, when submitting backup, backup workers will be enabled for partitioned backups.
However, we didn't clear the backup worker setting if no partitioned backup is active,
which will cause backup workers to be recruited, but doing nothing.
This PR changes the behavior so that when submitting, aborting, or discontinuing
backups, we'll disable backup workers if there is no active partitioned backup jobs.
20250517-162642-jzhou-4966348e89f1794d
20250517-044345-jzhou-f02f7defca3ea010
* Fix a test assertion failure
The continuous log end version could be less than min restorable version, when
the snapshot is a single version. I.e., min and max restorable versions are the
same.
20250519-042009-jzhou-88a2e0c67e8bed92
100k backup tests 20250519-155545-jzhou-c0aeaaf4a933cff9
* Fix tests where backup agents are not resumed
Also consolidate key updates for pausing backups.
20250520-175022-jzhou-10106ade7e0ad74f
100k backup 20250520-175208-jzhou-15255334e0c57eec
Found by simulation:
seed: -f tests/slow/ApiCorrectnessAtomicRestore.toml -s 177856328 -b on
Commit: 51ad8428e0
Compiler: clang++
Env: Rhel9 okteto
applyMutations() has processed version 801400000-803141392, and before calling sendCommitTransactionRequest(),
which was going to update apply begin version to 803141392. But DID NOT wait for the transaction commit.
Then there is an update on the apply end version to 845345760, which picks up the PREVIOUS apply begin version 801400000.
Thus started another applyMutation() with version range 801400000-845345760. Note because previous
applyMutation() has finished and didn't wait for the transaction commit, thus the starting version
is wrong. As a result, this applyMutation() re-processed version range 801400000-803141392.
The test failed during re-processing, because mutations are missing for the overlapped range.
The fix is to wait for the transaction to commit in sendCommitTransactionRequest().
This bug probably affects DR as well.
See rdar://146877552
20250317-162835-jzhou-ff4c4d6d7c51bfed
* New restore consolidated commit
This change adds RestoreDispatchPartitionedTaskFunc to restore
from partitioned-format backup.
* ArenaBlock::totalSize parameter pass by ref
* Fix format issues identified by CI
This PR includes a few stability fixes for Backup Worker
* Fixed memory bookkeeping issue in Backup Worker. Previously
it didn't release flow lock correctly when erasing messages.
* Added TLogServer fix to return 0 from poppedVersion() for
unrecognized log router tags.
Description
Given Configurable encryption has been checked in and being tested via
simulation for more than a month and also to avoid penalty of accessing
KNOBS in inline commit path, patch retires the KNOB and make
ConfigurationEncryption default EaR mode for FDB.
BlobCipher still supports the old format header and encryption semantics,
will remove the dead code as a followup PR.
Testing
devRunCorrectness - 100K
* EaR: Update ApiWorkload to validate encryption at-rest guarantees
Description
FDB encryption data at-rest guarantees if cluster is configured with feature
enabled, all data written to persistent disks shall be "encrypted". Given FDB
maintains multiple persistent storages during lifecycle of the data, the patch
proposes a scheme to validate the invariant via "simulation testing"
Patch proposes updating ApiCorrectness workload to do the following:
1. Client supplied params and/randomly enable the validation feature.
2. Validation when enabled, allows injecting a known "marker string"
to workload generated Key and Value data patterns.
3. On shutdown, if the validation is enabled, all test files are
scanned for the known "marker" pattern.
Simulation tests are already capable of doing the following:
1. Randomly select TenantMode (disabled/optional/required)
2. Randomly select EncryptionAtRestMode (cluster_aware/domain_aware)
Hence, the updates test all possible combinations are validated. Also,
'defaultTenant' is present to cover 'domain_aware' encryption use cases.
Testing
devRunCorrectness
devRetryCorrectness - ApiCorrectness & EncryptedBackupCorrectness
* EaR: Update encryption methods to make 'cipherHeaderKey' optional
Description
diff-1: Address review comments
Major changes includes:
1. Update BlobCipher Encrypt/Decrypt classes to make 'headerCipher' optional
2. Update GetEncryptionCipherKeys actor methods to make 'headerCipherKey' optional
3. Update the usage across all encryption participant methods
Testing
BlobCipherUnitTest
EnryptedBackupCorrecctness
BlobGranuleCorrectness*
devRunCorrectness - 100K
Commit proxy needs to fetch additional cipher keys post-resolution, since tenant ids for raw access requests and cross-tenant clear ranges are calculated after resolution.
Michael Stack