* Improve backup/restore/bulkload observability and reliability
Restore progress tracking:
- Add sub-phase counts (Submitted/Triggered/Running/TotalTasks) to
fdbrestore status output so users can see task submission progress
- Add getBulkLoadTaskProgress() to scan task states during restore
- Replace monitorBulkLoadJobCompletion with progress-tracking variant
that updates RestoreConfig counters every 5 seconds
Backup mode=BOTH fixes:
- Set bulkDumpJobId on BackupConfig so status shows BulkLoad Compatible
- Add bulkDumpSnapshotEndVersion for proper getLatestRestorableVersion
- Fix firstSnapshotEndVersion: only set from rangefile in mode=BOTH
- Skip empty BulkDump snapshots (totalSize=0) in rangefile restore
BulkLoad restore reliability:
- Abort restore immediately when no bulkdump data found (not retry forever)
- Add monitorBulkLoadModeAndSpawnActors so DD picks up bulkload jobs
submitted after DD initialization (required for restore workflow)
Audit validate_restore fixes:
- Read source data via database transaction instead of local SS to
avoid missing keys at shard boundaries after bulkload restore
- Add fast-path detection for completely empty baseline or source
- Retry on server_overloaded errors during audit
- Add AUDIT_RESTORE_BATCH_KEY_LIMIT and AUDIT_PROGRESS_PERSIST_BYTES_INTERVAL
knobs for tuning audit performance
CLI cleanup:
- Combine redundant task lines in bulkload/bulkdump status output
- Remove misleading health score and optimization recommendations
- Raise bulk task stall threshold from 60s to 600s (SST downloads
from blobstore routinely take 5-10 minutes)
* Address PR review: incrementalBackup check for mode=BOTH, snapshot type label, retry logging
- Add incrementalBackup fallback in getLatestRestorableVersion for mode==2
(was missing unlike modes 0 and 1)
- Note snapshot type heuristic as imperfect in describe output
- Log retries in getBulkLoadTaskProgress
* Parse snapshot type from filename instead of inferring from heuristic
KeyspaceSnapshotFile now has a snapshotType field parsed from the
snapshot filename (format: snapshot,begin,end,size,type). The type
suffix was already written but discarded during parsing. Now captured
and used for reliable bulkdump vs rangefile identification in
fdbbackup describe output.
* Simplify getBulkLoadTaskProgress: always set LOCK_AWARE, drop lockAware parameter
* Use Transaction instead of ReadYourWritesTransaction in getBulkLoadTaskProgress
Read-only function does not need write tracking overhead.
Uses Transaction* overload of krmGetRanges.
* Add BulkDump/BulkLoad observability for backup/restore progress tracking
Progress tracking APIs:
- getBulkDumpProgress() / getBulkLoadProgress() - track tasks, bytes, throughput, ETA
- getBulkDumpOwner() - track job ownership (e.g., which backup owns a bulkdump)
- Owner registration system for backup agents
CLI improvements:
- 'bulkdump status' and 'bulkload status' commands with human-readable output
- Progress display with throughput and ETA
Backup agent integration:
- Register backup as owner of bulkdump jobs
- Show bulkdump/bulkload progress in backup status output
Utility functions (flow/Util.h):
- formatBytesHumanReadable() - "1.23 TB", "456 MB"
- formatDurationHumanReadable() - "2 hours 30 minutes"
- formatBytesProgress(), formatThroughputLine(), formatETALine()
Test coverage:
- verifyBulkDumpObservability() / verifyBulkLoadObservability() in BackupS3BlobCorrectness
Bug fixes:
- Fix format string mismatch crash in FileBackupAgent progress display
- Add empty result guards for krmGetRanges() calls to prevent size_t underflow
* Formatting
* Rebase and formatting
* * cmake/CompileRocksDB.cmake
Preserve version when find_packge doesn't turn up rocksdb version
* fdbbackup/tests/backup_tests_common.sh
Wait when mode is 'both' for snapshot to be written.
* fdbclient/BulkLoading.cpp
Remove extra '/'.
* Output time of bulkload in bulkload history fdcli output as iso8601 instead of seconds since epoch
* Print start time as iso8601 -- readable -- instead of seconds since epoch -- unreadable... And don't crash if not a UUID
* Remove cute graphics.. and bullet characters
* Throw exception if not able to parse backup 'mode'
Trevor Clinkenbeard