Four small, independent changes surfaced while running a manual
backup/restore demo end to end:
- fdbrestore now accepts -C and --cluster-file as aliases for
--dest-cluster-file. Every other tool (fdbcli, fdbbackup, fdbserver,
backup_agent) already pairs -C with --cluster-file; fdbrestore was
the lone outlier and forced scripts to special-case it.
- fdbcli now accepts --logdir as an alias for --log-dir. Every other
tool spells the flag --logdir (no hyphen); the --log-dir form in
fdbcli was the outlier here.
- fdbbackup status no longer prints "BulkLoad Compatible: no" when
Snapshot Mode is rangefile. In that mode the answer is tautologically
derivable from the line above it (no bulkdump task ever runs), so
the line carries no information. The field still prints in modes
bulkdump and both, where it can flip during an in-progress backup.
- Correct the misleading comment in FileBackupAgent.cpp that claimed
submitBulkLoadJob validates cluster preconditions server-side. It
does not. A bulkload restore against a cluster missing
shard_encode_location_metadata / enable_read_lock_on_range, or with
a storage engine that cannot ingest SSTs, succeeds at submission and
then silently stalls in "State: running, Tasks: 0/0" forever. Replace
the comment with a faithful description of the failure mode and add
a matching TODO in submitBulkLoadJob explaining where the real
validation needs to live (somewhere with cluster-side knob
visibility) and why it cannot be done from fdbclient.
submitParallelRestore enforced an empty-destination precheck for all
restore modes by reading the first row of each target range. This is
correct for rangefile restore but wrong for bulkload: bulkload owns
the target range via an exclusive range lock and the storage server
clears each shard before SST ingestion, so a non-empty destination is
expected. Gate the precheck on useRangeFileRestore so bulkload skips
it.
Without this fix, "fdbrestore start --mode bulkload" against a cluster
with existing data fails with restore_destination_not_empty (2370).
The previous workaround was a manual clearrange before invoking
restore, which at large scale (>=4TB) overwhelms storage servers with
tombstones and stalls the cluster for hours.
Also document the per-shard clear-then-ingest sequence in bulkload.rst
and clarify in bulkload-user.rst that the clearrange in the quickstart
example is illustrative, not required.
Validated end-to-end at 100M scale (2026-06-04): bulkload from S3,
backup, rangefile baseline restore into validation prefix, bulkload
main restore into normal keyspace, marker-key data verification all
passed.
* Improve backup/restore/bulkload observability and reliability
Restore progress tracking:
- Add sub-phase counts (Submitted/Triggered/Running/TotalTasks) to
fdbrestore status output so users can see task submission progress
- Add getBulkLoadTaskProgress() to scan task states during restore
- Replace monitorBulkLoadJobCompletion with progress-tracking variant
that updates RestoreConfig counters every 5 seconds
Backup mode=BOTH fixes:
- Set bulkDumpJobId on BackupConfig so status shows BulkLoad Compatible
- Add bulkDumpSnapshotEndVersion for proper getLatestRestorableVersion
- Fix firstSnapshotEndVersion: only set from rangefile in mode=BOTH
- Skip empty BulkDump snapshots (totalSize=0) in rangefile restore
BulkLoad restore reliability:
- Abort restore immediately when no bulkdump data found (not retry forever)
- Add monitorBulkLoadModeAndSpawnActors so DD picks up bulkload jobs
submitted after DD initialization (required for restore workflow)
Audit validate_restore fixes:
- Read source data via database transaction instead of local SS to
avoid missing keys at shard boundaries after bulkload restore
- Add fast-path detection for completely empty baseline or source
- Retry on server_overloaded errors during audit
- Add AUDIT_RESTORE_BATCH_KEY_LIMIT and AUDIT_PROGRESS_PERSIST_BYTES_INTERVAL
knobs for tuning audit performance
CLI cleanup:
- Combine redundant task lines in bulkload/bulkdump status output
- Remove misleading health score and optimization recommendations
- Raise bulk task stall threshold from 60s to 600s (SST downloads
from blobstore routinely take 5-10 minutes)
* Address PR review: incrementalBackup check for mode=BOTH, snapshot type label, retry logging
- Add incrementalBackup fallback in getLatestRestorableVersion for mode==2
(was missing unlike modes 0 and 1)
- Note snapshot type heuristic as imperfect in describe output
- Log retries in getBulkLoadTaskProgress
* Parse snapshot type from filename instead of inferring from heuristic
KeyspaceSnapshotFile now has a snapshotType field parsed from the
snapshot filename (format: snapshot,begin,end,size,type). The type
suffix was already written but discarded during parsing. Now captured
and used for reliable bulkdump vs rangefile identification in
fdbbackup describe output.
* Simplify getBulkLoadTaskProgress: always set LOCK_AWARE, drop lockAware parameter
* Use Transaction instead of ReadYourWritesTransaction in getBulkLoadTaskProgress
Read-only function does not need write tracking overhead.
Uses Transaction* overload of krmGetRanges.
This parallel restore feature has been slated for removal for at least a year. This PR is closely based on earlier PR #12107.
This blog post explains some of the problems with the parallel restore feature: https://medium.com/@jingyuzhou/why-foundationdb-restore-is-slow-and-what-can-be-done-about-it-e73a821fdd33
As far as large feature removal changes go, this one is very straightforward, with most relevant files and test cases simply being deleted. There is one knob rename where storageserver.actor.cpp was using a knob with FASTRESTORE in the name. Other than that, changes to shared files mainly involve removing fastrestore-specific CLI and role support.
In progress:
20260330-222511-gglass-5ee0142213471b70 compressed=True data_size=35343375 duration=4611964 ended=100000 fail=1 fail_fast=1000 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:58:23 sanity=False started=100000 stopped=20260330-232334 submitted=20260330-222511 timeout=5400 username=gglass
The one failure was in SwizzledCycleTest.toml with too many lines of output and a timeout. I kind of suspect it's unrelated but haven't looked further.
* Remove parallel restore feature. This is based on earlier PR 12107. Compiles but untested.
* AI generated commit:
⏺ The fix restores a single if block that was accidentally deleted when removing the FASTRESTORE_TOOL code:
if (!restoreSystemKeys && !restoreUserKeys && backupKeys.empty()) {
addDefaultBackupRanges(backupKeys);
}
When no explicit key ranges are specified on the command line and neither --user-data nor --system-metadata flags are set, this populates backupKeys with the default backup ranges
(essentially all user data). Without it, backupKeys stays empty and hits the ASSERT(!backupRanges.empty()) in submitBackup().
* Remove a believed-to-be-dead code path, and update .gitignore
* Remove duplicate definition of restoreRequestDoneKey
* Add BulkDump/BulkLoad observability for backup/restore progress tracking
Progress tracking APIs:
- getBulkDumpProgress() / getBulkLoadProgress() - track tasks, bytes, throughput, ETA
- getBulkDumpOwner() - track job ownership (e.g., which backup owns a bulkdump)
- Owner registration system for backup agents
CLI improvements:
- 'bulkdump status' and 'bulkload status' commands with human-readable output
- Progress display with throughput and ETA
Backup agent integration:
- Register backup as owner of bulkdump jobs
- Show bulkdump/bulkload progress in backup status output
Utility functions (flow/Util.h):
- formatBytesHumanReadable() - "1.23 TB", "456 MB"
- formatDurationHumanReadable() - "2 hours 30 minutes"
- formatBytesProgress(), formatThroughputLine(), formatETALine()
Test coverage:
- verifyBulkDumpObservability() / verifyBulkLoadObservability() in BackupS3BlobCorrectness
Bug fixes:
- Fix format string mismatch crash in FileBackupAgent progress display
- Add empty result guards for krmGetRanges() calls to prevent size_t underflow
* Formatting
* Rebase and formatting
* * cmake/CompileRocksDB.cmake
Preserve version when find_packge doesn't turn up rocksdb version
* fdbbackup/tests/backup_tests_common.sh
Wait when mode is 'both' for snapshot to be written.
* fdbclient/BulkLoading.cpp
Remove extra '/'.
* Output time of bulkload in bulkload history fdcli output as iso8601 instead of seconds since epoch
* Print start time as iso8601 -- readable -- instead of seconds since epoch -- unreadable... And don't crash if not a UUID
* Remove cute graphics.. and bullet characters
* Throw exception if not able to parse backup 'mode'
Trevor Clinkenbeard