Delete encryption at rest in accordance with plans circulated in PR #12400.
Development of this PR was mostly straightforward. Several issues of note:
Upgrade tests which use Redwood want to use the old "encrypt by 0xFF XOR" test-only encoding/encryption algorithm. I wanted to delete that in main and did so. Upgrade tests have been modified not to use Redwood (storage engine 3, mentioned in various storageEngineExcludeTypes test options updates). 7.1 did not define storageEngineExcludeTypes. After some discussion we have decided to delete 7.1-based upgrade tests. 7.3- and 7.4-based upgrade tests remain in place.
Cross-version compatibility (such as it is, I assume in client library startup) remains super easy to break when deleting code. Tips: (A) removing unnecessary arguments to functions is bread and butter code editing when removing code, but if you find yourself removing arguments from a serializer() invocation, you might be breaking a protocol. (B) run ctest -R upgrade early and often.
An implication of these changes is that we are going to rely on the community to make us aware of any {upgrade+Redwood}-specific bugs. In other words, we are declining to continue to go out of our way to test functionality that we have no plans to use.
On the plus side this reclaims 15,000 more lines of code that we don't have to look at or think about, including in common areas such as commit proxy, backups, and generic storage server code.
Testing:
20260129-230241-gglass-15694f5f80af6932 compressed=True data_size=34905446 duration=4335357 ended=100000 fail_fast=1000 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=6:21:47 sanity=False started=100000 stopped=20260130-052428 submitted=20260129-230241 timeout=5400 username=gglass
20260202-214159-gglass-69b90c779cf8ec68 compressed=True data_size=35007141 duration=4612639 ended=100000 fail_fast=1000 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=3:00:25 sanity=False started=100000 stopped=20260203-004224 submitted=20260202-214159 timeout=5400 username=gglass
* Checkpoint file removals and code edits for removing encryption at rest. Have not tried to compile this yet.
* Checkpoint some improvements. Still does not compile.
* Merging with upstream changes in copyright updates resurrected previously deleted files, so delete them again
* Checkpoint incremental progress towards getting this to compile
* Checkpoint more intermediate changes getting encryption at rest deleted. Still doesnt compile but getting closer.
* Changes sufficient to get things to compile with removal of encryption at rest. NOT TESTED.
* Delete encryptModes from toml files run by current fdbserver binaries; restarting tests using <= 7.4 binaries do need encryptModes to say disabled
* Avoid using MAX_ENCODING_VALUE for random purposes for which other solutions are more clear and generally better
* Stop using XOREncryption_TestOnly because that no longer works. Also I see no need to support it as it requires unneeded interfaces and test fixtures in order to actually work.
* Remove more page encryption stuff, and address some TODO(gglass) comments.
* debugging redwood failures
* Fix some misc simulation failures. Notably, disable storage engine type 3 (redwood) on upgrade tests, as it writes databases with encoding type 1 which is no longer supported
* Add a comment discussion deprecation options for fields in persistent metadata, and explain why we are merely renaming the member as deprecated and nothing anything else.
* Remove 7.1 upgrade tests. These tests enable Redwood and write databases with "0xFF XOR encryption" style encoding (encoding 1), which is removed in main.
* Improve comments
* Obligatory f3f commit. Format The Effin Source Files
* Delete more unneeded encryption stuff
* Put back about 1% of deleted code in a desperate attempt to unbreak broken protocol compatibility
* formatting
* Remove mentions of encryption at rest in backup related APIs
* Address misc review comments. Remove --encrypt-files backup option.
ControlService: Implements a subset of fdbcli commands
Implements following commands from fdbcli:
- GetCoordinators
- ChangeCoordinators
- GetStatus
- GetWorkers
- Include
- Kill
'fdbctl' aims to implement control plane layer for FoundationDB.
This includes general operations related to cluster management,
getting service health etc. This is all exposed via ControlService
with interface described in 'control_service.proto'.
Eventually 'fdbcli' and 'ControlService' can both reuse 'fdbctl'
in library much of what 'fdbcli' does is within scope of this
component.
Trevor Clinkenbeard