* Improve backup/restore/bulkload observability and reliability
Restore progress tracking:
- Add sub-phase counts (Submitted/Triggered/Running/TotalTasks) to
fdbrestore status output so users can see task submission progress
- Add getBulkLoadTaskProgress() to scan task states during restore
- Replace monitorBulkLoadJobCompletion with progress-tracking variant
that updates RestoreConfig counters every 5 seconds
Backup mode=BOTH fixes:
- Set bulkDumpJobId on BackupConfig so status shows BulkLoad Compatible
- Add bulkDumpSnapshotEndVersion for proper getLatestRestorableVersion
- Fix firstSnapshotEndVersion: only set from rangefile in mode=BOTH
- Skip empty BulkDump snapshots (totalSize=0) in rangefile restore
BulkLoad restore reliability:
- Abort restore immediately when no bulkdump data found (not retry forever)
- Add monitorBulkLoadModeAndSpawnActors so DD picks up bulkload jobs
submitted after DD initialization (required for restore workflow)
Audit validate_restore fixes:
- Read source data via database transaction instead of local SS to
avoid missing keys at shard boundaries after bulkload restore
- Add fast-path detection for completely empty baseline or source
- Retry on server_overloaded errors during audit
- Add AUDIT_RESTORE_BATCH_KEY_LIMIT and AUDIT_PROGRESS_PERSIST_BYTES_INTERVAL
knobs for tuning audit performance
CLI cleanup:
- Combine redundant task lines in bulkload/bulkdump status output
- Remove misleading health score and optimization recommendations
- Raise bulk task stall threshold from 60s to 600s (SST downloads
from blobstore routinely take 5-10 minutes)
* Address PR review: incrementalBackup check for mode=BOTH, snapshot type label, retry logging
- Add incrementalBackup fallback in getLatestRestorableVersion for mode==2
(was missing unlike modes 0 and 1)
- Note snapshot type heuristic as imperfect in describe output
- Log retries in getBulkLoadTaskProgress
* Parse snapshot type from filename instead of inferring from heuristic
KeyspaceSnapshotFile now has a snapshotType field parsed from the
snapshot filename (format: snapshot,begin,end,size,type). The type
suffix was already written but discarded during parsing. Now captured
and used for reliable bulkdump vs rangefile identification in
fdbbackup describe output.
* Simplify getBulkLoadTaskProgress: always set LOCK_AWARE, drop lockAware parameter
* Use Transaction instead of ReadYourWritesTransaction in getBulkLoadTaskProgress
Read-only function does not need write tracking overhead.
Uses Transaction* overload of krmGetRanges.
Trevor Clinkenbeard