Increase the buffer size for backup workers and rename the knob.
Seed: -f ./tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml -s 3120109021 -b on
* Fix backup worker assertion failure on memory usage
pullAsyncData() can be cancelled thus not take() some of the memory used by
the in memory queue.
20250320-171737-jzhou-752fd8c45fdadd4a
100k tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250320-172251-jzhou-3bc7db8e7ce5e1de
* Refactor such that messages in the queue have to reserve memory first
Thus, when we release the memory, it must have already been reserved.
100k ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250320-234748-jzhou-a159972dd0a72e03
20250320-235252-jzhou-54a6ae8c67873b59
* Fix a backup worker assertion failure
The pop version obtained from GRV proxy replies may go backwards, which can
cause assertion failure when pulling mutations, i.e., missing mutations. Fix
this bug and add an assertion that popVersion can go lower.
100k 20250321-012019-jzhou-01eb56938cf0a3fc
100k tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250321-012144-jzhou-68181bb51aedbb5d
* Fix assertion failure of triggered version
The triggered version could be the same as popVersion.
20250321-025429-jzhou-a088c3f119331b93
100k tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250321-025532-jzhou-77e2dbd5fd035157
* Update fdbserver/BackupWorker.actor.cpp
Co-authored-by: Syed Paymaan Raza <1238752+spraza@users.noreply.github.com>
---------
Co-authored-by: Syed Paymaan Raza <1238752+spraza@users.noreply.github.com>
lock->release(toRelease) can trigger assertion failure that release size is larger
than the active size. This is because lock->take() has not returned yet. So the
memory released has not been reserved. Fix by breaking release into two steps.
After the first release(), the lock->take() will be unblocked, thus the second
release() does the correct accounting.
gcc correctness:
20250319-002950-jzhou-a9ecc09dd1c8812a
Previously I used arena size for accouting, which has problems such as arena
memory usage changes and arena memory block circular reference. And we are
acquiring flow lock multiple times, even though the cursor has already fetched
data into memory.
This change simplifies the accounting to just use message sizes, reducing the
number of flow lock acquisitions.
Otherwise, the pop version can become larger than the actual saved version when
switching to the regular pulling mode. Because the pop version is larger,
mutations larger than saved version could be popped and no long available.
* New restore consolidated commit
This change adds RestoreDispatchPartitionedTaskFunc to restore
from partitioned-format backup.
* ArenaBlock::totalSize parameter pass by ref
* Fix format issues identified by CI
This PR includes a few stability fixes for Backup Worker
* Fixed memory bookkeeping issue in Backup Worker. Previously
it didn't release flow lock correctly when erasing messages.
* Added TLogServer fix to return 0 from poppedVersion() for
unrecognized log router tags.
* EaR: Configurable encryption support for TLog mutations
Description
diff-1 : Address review comments
Major changes includes:
1. Update the code involved in ensuring Tlog mutation encryption to be
compliant with "configurable encryption" feature.
2. Update ENABLE_CONFIGURABLE_ENCRYPTION flag to be 'true' by default
and BUGGIFY it.
Testing
devRunCorrectness - 100K
Commit proxy needs to fetch additional cipher keys post-resolution, since tenant ids for raw access requests and cross-tenant clear ranges are calculated after resolution.
The number of released bytes exceeds the number of acquired bytes in locks.
This is because the bytes counted towards release is calculated after a "wait",
when more bytes could be allocated.
Adding the following metrics:
* BlobCipherKeyCache hit/miss
* EKP: KMS requests latencies
* For each component that using encryption, they now need to pass a UsageType enum to the encryption helper methods (GetEncryptCipherKeys/GetLatestEncryptCipherKey/encrypt/decrypt) and those methods will help to log get cipher key latency samples and encryption/decryption cpu times accordingly.
* proof of concept
* use code-probe instead of test
* code probe working on gcc
* code probe implemented
* renamed TestProbe to CodeProbe
* fixed refactoring typo
* support filtered output
* print probes at end of simulation
* fix missed probes print
* fix deduplication
* Fix refactoring issues
* revert bad refactor
* make sure file paths are relative
* fix more wrong refactor changes
This PR add support for TLog encryption through commit proxy. The encryption is done on per-mutation basis. As CP writes mutations to TLog, it inserts encryption header alongside encrypted mutations. Storage server (and other consumers of TLog such as storage cache and backup worker) decrypts the mutations as they peek TLog.
neethuhaneesha