* Improve backup/restore/bulkload observability and reliability
Restore progress tracking:
- Add sub-phase counts (Submitted/Triggered/Running/TotalTasks) to
fdbrestore status output so users can see task submission progress
- Add getBulkLoadTaskProgress() to scan task states during restore
- Replace monitorBulkLoadJobCompletion with progress-tracking variant
that updates RestoreConfig counters every 5 seconds
Backup mode=BOTH fixes:
- Set bulkDumpJobId on BackupConfig so status shows BulkLoad Compatible
- Add bulkDumpSnapshotEndVersion for proper getLatestRestorableVersion
- Fix firstSnapshotEndVersion: only set from rangefile in mode=BOTH
- Skip empty BulkDump snapshots (totalSize=0) in rangefile restore
BulkLoad restore reliability:
- Abort restore immediately when no bulkdump data found (not retry forever)
- Add monitorBulkLoadModeAndSpawnActors so DD picks up bulkload jobs
submitted after DD initialization (required for restore workflow)
Audit validate_restore fixes:
- Read source data via database transaction instead of local SS to
avoid missing keys at shard boundaries after bulkload restore
- Add fast-path detection for completely empty baseline or source
- Retry on server_overloaded errors during audit
- Add AUDIT_RESTORE_BATCH_KEY_LIMIT and AUDIT_PROGRESS_PERSIST_BYTES_INTERVAL
knobs for tuning audit performance
CLI cleanup:
- Combine redundant task lines in bulkload/bulkdump status output
- Remove misleading health score and optimization recommendations
- Raise bulk task stall threshold from 60s to 600s (SST downloads
from blobstore routinely take 5-10 minutes)
* Address PR review: incrementalBackup check for mode=BOTH, snapshot type label, retry logging
- Add incrementalBackup fallback in getLatestRestorableVersion for mode==2
(was missing unlike modes 0 and 1)
- Note snapshot type heuristic as imperfect in describe output
- Log retries in getBulkLoadTaskProgress
* Parse snapshot type from filename instead of inferring from heuristic
KeyspaceSnapshotFile now has a snapshotType field parsed from the
snapshot filename (format: snapshot,begin,end,size,type). The type
suffix was already written but discarded during parsing. Now captured
and used for reliable bulkdump vs rangefile identification in
fdbbackup describe output.
* Simplify getBulkLoadTaskProgress: always set LOCK_AWARE, drop lockAware parameter
* Use Transaction instead of ReadYourWritesTransaction in getBulkLoadTaskProgress
Read-only function does not need write tracking overhead.
Uses Transaction* overload of krmGetRanges.
* Refactor blob store classes to use interface
* update to make some methods non-pure virtual
* Complete IBlobStoreEndpoint interface refactoring
Move doRequest, connect, returnConnection, updateSecret, writeEntireFile,
listObjects, and deleteRecursively from S3BlobStoreEndpoint to the shared
IBlobStoreEndpoint base class. Introduce provider hook methods
(setRequestHeaders, normalizeResourceForRequest, simulateRequestFailure,
processRequestFailure, preRetryCheck, extractCredentialFields) so that
the request loop is provider-agnostic.
Rename s3_backup_test.sh to backup_restore_test.sh and update CMake
to reflect the provider-agnostic naming.
Fix double proxy absolute-form rewrite bug where both
S3BlobStoreEndpoint::normalizeResourceForRequest and the base doRequest
would prepend http://host:port for proxy connections.
Fix getResourceURL duplication by having S3's override delegate to the
base class for shared knob-params/extra-headers logic.
* resolve review comments
* remove gcs auto region block
* clang format
* Enable +x bit.
* clang tidy
akankshamahajan15