Previously we have to modify SimulatedCluster source code to
enable/disable RocksDB storage engine in simulations. Now it is doable
by using cmake flags/ccmake gui by setting the OPTION
ENABLE_ROCKSDB_IN_SIMULATION
ON/OFF.
* Ensuring HTTP ports are stable for the same servers, and adding a test http server that ensures no other users accidentally talk to it
* fixing warning with werror
* more werror fixes
* Passes existing tests
* adding http unit test for wrong md5 sum
* Added new HTTPKeyValueStore workload to test long-running http clients
* fixing warnings
* Enable fault injection;
Ignore MAX_COORDINATOR_SNAPSHOT_FAULT_TOLERANCE in simulation
Fix the issue when there's not fdb.cluster file and disable randomMoveKeys in SnapTest workload
* Move snap tests to from_7.3.0
* Add comments for the change
* Change SevError to SevWarn when Tlog process is not found in the worker map
* EaR: Configurable encryption support for TLog mutations
Description
diff-1 : Address review comments
Major changes includes:
1. Update the code involved in ensuring Tlog mutation encryption to be
compliant with "configurable encryption" feature.
2. Update ENABLE_CONFIGURABLE_ENCRYPTION flag to be 'true' by default
and BUGGIFY it.
Testing
devRunCorrectness - 100K
Disable sharded rocks storage for downgrade tests where we
Need to keep knob "shard_encode_location_metadata" so that downgrade tests
can pass the second phase.
The PR is updating storage server and Redwood to enable encryption based on the encryption mode in DB config, which was previously controlled by a knob. High level changes are
1. Passing encryption mode in DB config to storage server
1.1 If it is a new storage server, pass the encryption mode through `InitializeStorageRequest`. the encryption mode is pass to Redwood for initialization
1.2 If it is an existing storage server, on restart the storage server will send `GetStorageServerRejoinInfoRequest` to commit proxy, and commit proxy will return the current encryption mode, which it get from DB config on its own initialization. Storage server will compare the DB config encryption mode to the local storage encryption mode, and fail if they don't match
2. Adding a new `encryptionMode()` method to `IKeyValueStore`, which return a future of local encryption mode of the KV store instance. A KV store supporting encryption would need to persist its own encryption mode, and return the mode via the API.
3. Redwood accepts encryption mode from its constructor. For a new Redwood instance, caller has to specific the encryption mode, which will be stored in Redwood per-instance file header. For existing instance, caller is supposed to not passing the encryption mode, and let Redwood find it out from its own header.
4. Refactoring in Redwood to accommodate the above changes.
* Print an asan heap profile on OOM
* Use 32KiB stacks for boost coro
* Print 100%, 10 max contexts for asan OOM
* Lower machineCount to 30 in DataLossRecovery test
* Add asanMachineCount override to control ASAN memory usage
* Fix Redwood tree height overgrowth when EaR and tenant page split are enabled, by removing the buildNewSubtree() logic.
* Fixing incorrect page upper bound for the last page created by writePages() without the buildNewSubtree() logic.
* Enable tenant page split if encryption mode is domain-aware encryption.
* Related test fixes:
- In simulation, pass encryption mode to storage/Redwood via knobs. This is a workaround to enable testing with Redwood encryption before we correctly pass the encryption mode via db config. Also temporarily disable tenant page split for restart tests.
- Disable raw access in FuzzApiCorrectness test if domain-aware encryption is enabled, to avoid test timeout
- Disable encryption for DrUpgradeRestart test, which is likely to fail due to a rare EKP deadlock issue blocking recovery. Will re-enable after the deadlock issue is fixed.
Xiaoge Su