The PR is updating storage server and Redwood to enable encryption based on the encryption mode in DB config, which was previously controlled by a knob. High level changes are
1. Passing encryption mode in DB config to storage server
1.1 If it is a new storage server, pass the encryption mode through `InitializeStorageRequest`. the encryption mode is pass to Redwood for initialization
1.2 If it is an existing storage server, on restart the storage server will send `GetStorageServerRejoinInfoRequest` to commit proxy, and commit proxy will return the current encryption mode, which it get from DB config on its own initialization. Storage server will compare the DB config encryption mode to the local storage encryption mode, and fail if they don't match
2. Adding a new `encryptionMode()` method to `IKeyValueStore`, which return a future of local encryption mode of the KV store instance. A KV store supporting encryption would need to persist its own encryption mode, and return the mode via the API.
3. Redwood accepts encryption mode from its constructor. For a new Redwood instance, caller has to specific the encryption mode, which will be stored in Redwood per-instance file header. For existing instance, caller is supposed to not passing the encryption mode, and let Redwood find it out from its own header.
4. Refactoring in Redwood to accommodate the above changes.
* Allow multiple keyranges in CheckpointRequest.
Include DataMove ID in CheckpointMetaData.
* Use UID dataMoveId instead of Optional<UID>.
* Implemented ShardedRocks::checkpoint().
* Implementing createCheckpoint().
* Attempted to change getCheckpointMetaData*() for a single keyrange.
* Added getCheckpointMetaDataForRange.
* Minor fixes for NativeAPI.actor.cpp.
* Replace UID CheckpointMetaData::ssId with std::vector<UID>
CheckpointMetaData::src;
* Implemented getCheckpointMetaData() and completed checkpoint creation
and fetch in test.
* Refactoring CheckpointRequest and CheckpointMetaData
rename `dataMoveId` as `actionId` and make it Optional.
* Fixed ctor of CheckpointMetaData.
* Implemented ShardedRocksDB::restore().
* Tested checkpoint restore, and added range check for restore, so that
the target ranges can be a subset of the checkpoint ranges.
* Added test to partially restore a checkpoint.
* Refactor: added checkpointRestore().
* Sort ranges for comparison.
* Cleanups.
* Check restore ranges are empty; Add ranges in main thread.
* Resolved comments.
* Fixed GetCheckpointMetaData range check issue.
* Refactor CheckpointReader for CF checkpoint.
* Added CheckpointAsKeyValues as a parameter for newCheckpointReader.
* PhysicalShard::restoreKvs().
* Added `ranges` in fetchCheckpoint.
* Added RocksDBCheckpointKeyValues::ranges.
* Added ICheckpointIterator and implemented for RocksDBCheckpointReader.
* Refactored OpenAction for CheckpointReader, handled failure cases.
* Use RocksDBCheckpointIterator::end() in readRange.
* Set CheckpointReader timout and other Rocks read options.
* Implementing fetchCheckpointRange().
* Added more CheckpointReader tests.
* Cleanup.
* More cleanup.
* Added fetchCheckpointRanges test.
* Implemented restore of kv-based checkpoint.
* Improved CheckpointRestore test for kv, non-kv, as well as partial
restore.
* Added test of merge.
* Fixed merge test.
* Cleanup.
* Resolved comments.
Co-authored-by: He Liu <heliu@apple.com>
If a log directory is specified in the "db_log_dir" option, RocksDB will add the whole file path in the log file name, which can exceed the 255 byte limit of Linux file system, causing storage server failures.
See
c3f720c60d/include/rocksdb/options.h (L674-L679)
* Allow multiple keyranges in CheckpointRequest.
Include DataMove ID in CheckpointMetaData.
* Use UID dataMoveId instead of Optional<UID>.
* Implemented ShardedRocks::checkpoint().
* Implementing createCheckpoint().
* Attempted to change getCheckpointMetaData*() for a single keyrange.
* Added getCheckpointMetaDataForRange.
* Minor fixes for NativeAPI.actor.cpp.
* Replace UID CheckpointMetaData::ssId with std::vector<UID>
CheckpointMetaData::src;
* Implemented getCheckpointMetaData() and completed checkpoint creation
and fetch in test.
* Refactoring CheckpointRequest and CheckpointMetaData
rename `dataMoveId` as `actionId` and make it Optional.
* Fixed ctor of CheckpointMetaData.
* Implemented ShardedRocksDB::restore().
* Tested checkpoint restore, and added range check for restore, so that
the target ranges can be a subset of the checkpoint ranges.
* Added test to partially restore a checkpoint.
* Refactor: added checkpointRestore().
* Sort ranges for comparison.
* Cleanups.
* Check restore ranges are empty; Add ranges in main thread.
* Resolved comments.
* Fixed GetCheckpointMetaData range check issue.
* Refactor CheckpointReader for CF checkpoint.
* Added CheckpointAsKeyValues as a parameter for newCheckpointReader.
* PhysicalShard::restoreKvs().
* Added `ranges` in fetchCheckpoint.
* Added RocksDBCheckpointKeyValues::ranges.
* Added ICheckpointIterator and implemented for RocksDBCheckpointReader.
* Refactored OpenAction for CheckpointReader, handled failure cases.
* Use RocksDBCheckpointIterator::end() in readRange.
* Set CheckpointReader timout and other Rocks read options.
* Implementing fetchCheckpointRange().
* Added more CheckpointReader tests.
* Cleanup.
* More cleanup.
* Resolved comments.
Co-authored-by: He Liu <heliu@apple.com>
* Allow multiple keyranges in CheckpointRequest.
Include DataMove ID in CheckpointMetaData.
* Use UID dataMoveId instead of Optional<UID>.
* Implemented ShardedRocks::checkpoint().
* Implementing createCheckpoint().
* Attempted to change getCheckpointMetaData*() for a single keyrange.
* Added getCheckpointMetaDataForRange.
* Minor fixes for NativeAPI.actor.cpp.
* Replace UID CheckpointMetaData::ssId with std::vector<UID>
CheckpointMetaData::src;
* Implemented getCheckpointMetaData() and completed checkpoint creation
and fetch in test.
* Refactoring CheckpointRequest and CheckpointMetaData
rename `dataMoveId` as `actionId` and make it Optional.
* Fixed ctor of CheckpointMetaData.
* Implemented ShardedRocksDB::restore().
* Tested checkpoint restore, and added range check for restore, so that
the target ranges can be a subset of the checkpoint ranges.
* Added test to partially restore a checkpoint.
* Refactor: added checkpointRestore().
* Sort ranges for comparison.
* Cleanups.
* Check restore ranges are empty; Add ranges in main thread.
* Resolved comments.
* Fixed GetCheckpointMetaData range check issue.
* Fixed error description.
Co-authored-by: He Liu <heliu@apple.com>
Yi Wu