expireData() can silently roll the expiration point back to a log file begin version when the requested version falls in the middle of that log file, since a log file cannot be partially deleted.
The CLI always printed the originally requested version regardless, making it look like more data was deleted than actually was.
ExpireProgress now tracks both the requested and actual end version so the CLI can print a note when they differ, in either direction (rolled back, or already expired past the request).
Four small, independent changes surfaced while running a manual
backup/restore demo end to end:
- fdbrestore now accepts -C and --cluster-file as aliases for
--dest-cluster-file. Every other tool (fdbcli, fdbbackup, fdbserver,
backup_agent) already pairs -C with --cluster-file; fdbrestore was
the lone outlier and forced scripts to special-case it.
- fdbcli now accepts --logdir as an alias for --log-dir. Every other
tool spells the flag --logdir (no hyphen); the --log-dir form in
fdbcli was the outlier here.
- fdbbackup status no longer prints "BulkLoad Compatible: no" when
Snapshot Mode is rangefile. In that mode the answer is tautologically
derivable from the line above it (no bulkdump task ever runs), so
the line carries no information. The field still prints in modes
bulkdump and both, where it can flip during an in-progress backup.
- Correct the misleading comment in FileBackupAgent.cpp that claimed
submitBulkLoadJob validates cluster preconditions server-side. It
does not. A bulkload restore against a cluster missing
shard_encode_location_metadata / enable_read_lock_on_range, or with
a storage engine that cannot ingest SSTs, succeeds at submission and
then silently stalls in "State: running, Tasks: 0/0" forever. Replace
the comment with a faithful description of the failure mode and add
a matching TODO in submitBulkLoadJob explaining where the real
validation needs to live (somewhere with cluster-side knob
visibility) and why it cannot be done from fdbclient.
* Refactor blob store classes to use interface
* update to make some methods non-pure virtual
* Complete IBlobStoreEndpoint interface refactoring
Move doRequest, connect, returnConnection, updateSecret, writeEntireFile,
listObjects, and deleteRecursively from S3BlobStoreEndpoint to the shared
IBlobStoreEndpoint base class. Introduce provider hook methods
(setRequestHeaders, normalizeResourceForRequest, simulateRequestFailure,
processRequestFailure, preRetryCheck, extractCredentialFields) so that
the request loop is provider-agnostic.
Rename s3_backup_test.sh to backup_restore_test.sh and update CMake
to reflect the provider-agnostic naming.
Fix double proxy absolute-form rewrite bug where both
S3BlobStoreEndpoint::normalizeResourceForRequest and the base doRequest
would prepend http://host:port for proxy connections.
Fix getResourceURL duplication by having S3's override delegate to the
base class for shared knob-params/extra-headers logic.
* resolve review comments
* remove gcs auto region block
* clang format
* Enable +x bit.
* clang tidy
This parallel restore feature has been slated for removal for at least a year. This PR is closely based on earlier PR #12107.
This blog post explains some of the problems with the parallel restore feature: https://medium.com/@jingyuzhou/why-foundationdb-restore-is-slow-and-what-can-be-done-about-it-e73a821fdd33
As far as large feature removal changes go, this one is very straightforward, with most relevant files and test cases simply being deleted. There is one knob rename where storageserver.actor.cpp was using a knob with FASTRESTORE in the name. Other than that, changes to shared files mainly involve removing fastrestore-specific CLI and role support.
In progress:
20260330-222511-gglass-5ee0142213471b70 compressed=True data_size=35343375 duration=4611964 ended=100000 fail=1 fail_fast=1000 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:58:23 sanity=False started=100000 stopped=20260330-232334 submitted=20260330-222511 timeout=5400 username=gglass
The one failure was in SwizzledCycleTest.toml with too many lines of output and a timeout. I kind of suspect it's unrelated but haven't looked further.
* Remove parallel restore feature. This is based on earlier PR 12107. Compiles but untested.
* AI generated commit:
⏺ The fix restores a single if block that was accidentally deleted when removing the FASTRESTORE_TOOL code:
if (!restoreSystemKeys && !restoreUserKeys && backupKeys.empty()) {
addDefaultBackupRanges(backupKeys);
}
When no explicit key ranges are specified on the command line and neither --user-data nor --system-metadata flags are set, this populates backupKeys with the default backup ranges
(essentially all user data). Without it, backupKeys stays empty and hits the ASSERT(!backupRanges.empty()) in submitBackup().
* Remove a believed-to-be-dead code path, and update .gitignore
* Remove duplicate definition of restoreRequestDoneKey
Trevor Clinkenbeard