This parallel restore feature has been slated for removal for at least a year. This PR is closely based on earlier PR #12107.
This blog post explains some of the problems with the parallel restore feature: https://medium.com/@jingyuzhou/why-foundationdb-restore-is-slow-and-what-can-be-done-about-it-e73a821fdd33
As far as large feature removal changes go, this one is very straightforward, with most relevant files and test cases simply being deleted. There is one knob rename where storageserver.actor.cpp was using a knob with FASTRESTORE in the name. Other than that, changes to shared files mainly involve removing fastrestore-specific CLI and role support.
In progress:
20260330-222511-gglass-5ee0142213471b70 compressed=True data_size=35343375 duration=4611964 ended=100000 fail=1 fail_fast=1000 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:58:23 sanity=False started=100000 stopped=20260330-232334 submitted=20260330-222511 timeout=5400 username=gglass
The one failure was in SwizzledCycleTest.toml with too many lines of output and a timeout. I kind of suspect it's unrelated but haven't looked further.
* Remove parallel restore feature. This is based on earlier PR 12107. Compiles but untested.
* AI generated commit:
⏺ The fix restores a single if block that was accidentally deleted when removing the FASTRESTORE_TOOL code:
if (!restoreSystemKeys && !restoreUserKeys && backupKeys.empty()) {
addDefaultBackupRanges(backupKeys);
}
When no explicit key ranges are specified on the command line and neither --user-data nor --system-metadata flags are set, this populates backupKeys with the default backup ranges
(essentially all user data). Without it, backupKeys stays empty and hits the ASSERT(!backupRanges.empty()) in submitBackup().
* Remove a believed-to-be-dead code path, and update .gitignore
* Remove duplicate definition of restoreRequestDoneKey
* Initiate deletion of storage cache feature. This is rough and is mostly done by commenting out code in case backtracking is needed. Compiles. Not tested.
* fix some test errors about cache consistency check options which we no longer care about
* design/feature-status.md: Storage Cache status updated to `has been deleted`.
* Delete it for real
* disable BackupS3BlobCorrectness.toml because it fails a lot
* In the interest of a single-purpose, clean diff, put back a removed dumb warning that generates compile error noise
* fix formatting
* Add TODO comment to remove tagLocalityLogRouter
* fix typo
* Recruit new singleton for consistency checker.
* Recruit the consistency checker only if enabled.
* Add a yield in monitorConsistencyChecker().
* Minor fixes.
* Consistency check workload enhancements.
* Minor fixes and clarifications.
* clang format
* Clang format.
* Minor fixes, cleanup, debug tracing.
* Misc.
* Move the consistency scan information from dbconfig to a key backed object.
* Move consistency scan config out of db cofig to a state object and feature rename.
* ConsistencyCheck workload refactor.
* devFormat
* Update fdbcli/ConsistencyScanCommand.actor.cpp
* Review Comments.
Co-authored-by: negoyal <neelam.goyal@gmail.com>
Co-authored-by: Ata E Husain Bohra <ata.husain@snowflake.com>
Major changes includes:
1. Add a new FDB role responsible- EncyrptKeyProxy. The role is
responsible to expose APIs to fetch encyrption keys interacting
with external Encryption KeyManager interface.
2. The process is a FDB singleton process following similar recruitment
rules as other singleton processes in the system.
3. Code to recruit the worker process; given the encryption keys are
needed during recovery (decode TLog records), for now the process
is co-located in same datacenter as ClusterController.
4. Skeleton process actor code; more functionality will be added in
subsequent PRs.
NOTE: The code is protected under a SERVER_KNOB with the default
value as 'false' for now.
Add support for more than 1 blob worker in simulated cluster.
We now have at least 2, and up to 7 (randomly).
Also removed the ability for blob workers to onboard onto
unset processes.
if the first GRV proxy or resolver is forced to share a process, it should prefer to share with the commit proxy so that the commit proxy has more potential options it can share with
Add a new role for ratekeeper.
Remove StorageServerChanges from data distribution.
Ratekeeper monitors storage servers, which borrows the idea from
DataDistribution.
Let cluster controller to start a new data distributor role by sending a
message to a chosen worker.
Change MasterInterface usage in DataDistribution to masterId
Add DataDistributor rejoin handling.
This allows the data distributor to tell the new cluster controller of its
existence so that the controller doesn't spawn a new one. I.e., there should
be only ONE data distributor in the cluster.
If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries
to recruit one as DD. CC also monitors DD and restarts one if it failed.
The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for
the new DD.
Add GetRecoveryInfo RPC to master server, which is called by data distributor
to obtain the recovery Transaction version from the master server.
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
The previous order of fitness was
BestFit > GoodFit > BestOtherFit > ...
which is baffling. It's now:
BestFit > GoodFit > OkayFit > ...
which won't break anyone's expectations.
Trevor Clinkenbeard