* Fix read-side overflow and simplify append API for large snapshot manifests
Follow-up to #13349, which fixed the write-side overflow when a snapshot manifest exceeds ~2 GB but left the read side and the API untidy. This PR addresses both.
### Changes
**Simpler append API**
There were two `append()` methods — one taking `int`, one `size_t` — and which ran depended on the argument type, which is easy to get wrong. Replaced with a single
public `append()` that safely chunks any size, plus a clearly-named backend hook `appendImpl()` that each storage backend implements. No more overload ambiguity.
**Read side fix**
`readKeyspaceSnapshot` read the manifest into a buffer whose length is an `int`, so a manifest larger than 2 GB could truncate and crash on restore. It now reads into a
`std::string` (which can exceed 2 GB) in chunks, matching the write side, and drops a redundant full copy of the manifest.
**Knob rename**
`BACKUP_MANIFEST_WRITE_CHUNK_SIZE` → `BACKUP_MANIFEST_CHUNK_SIZE`, since it now controls chunk size for both reads and writes.
**Test**
Added a unit test that reads a manifest back in many small chunks and verifies all range files and key ranges round-trip correctly.
### Notes
- Range and log files are unaffected — they're already streamed in small blocks on both read and write.
* Addressed comments
* Fix clang tidy errors
expireData() can silently roll the expiration point back to a log file begin version when the requested version falls in the middle of that log file, since a log file cannot be partially deleted.
The CLI always printed the originally requested version regardless, making it look like more data was deleted than actually was.
ExpireProgress now tracks both the requested and actual end version so the CLI can print a note when they differ, in either direction (rolled back, or already expired past the request).
* Fix overflow bug for large json documents in writeKeyspaceSnapshotFile
* Move chunked append into non-virtual overload to fix the same overflow issue for all append calls and add additional test case
* Improve backup/restore/bulkload observability and reliability
Restore progress tracking:
- Add sub-phase counts (Submitted/Triggered/Running/TotalTasks) to
fdbrestore status output so users can see task submission progress
- Add getBulkLoadTaskProgress() to scan task states during restore
- Replace monitorBulkLoadJobCompletion with progress-tracking variant
that updates RestoreConfig counters every 5 seconds
Backup mode=BOTH fixes:
- Set bulkDumpJobId on BackupConfig so status shows BulkLoad Compatible
- Add bulkDumpSnapshotEndVersion for proper getLatestRestorableVersion
- Fix firstSnapshotEndVersion: only set from rangefile in mode=BOTH
- Skip empty BulkDump snapshots (totalSize=0) in rangefile restore
BulkLoad restore reliability:
- Abort restore immediately when no bulkdump data found (not retry forever)
- Add monitorBulkLoadModeAndSpawnActors so DD picks up bulkload jobs
submitted after DD initialization (required for restore workflow)
Audit validate_restore fixes:
- Read source data via database transaction instead of local SS to
avoid missing keys at shard boundaries after bulkload restore
- Add fast-path detection for completely empty baseline or source
- Retry on server_overloaded errors during audit
- Add AUDIT_RESTORE_BATCH_KEY_LIMIT and AUDIT_PROGRESS_PERSIST_BYTES_INTERVAL
knobs for tuning audit performance
CLI cleanup:
- Combine redundant task lines in bulkload/bulkdump status output
- Remove misleading health score and optimization recommendations
- Raise bulk task stall threshold from 60s to 600s (SST downloads
from blobstore routinely take 5-10 minutes)
* Address PR review: incrementalBackup check for mode=BOTH, snapshot type label, retry logging
- Add incrementalBackup fallback in getLatestRestorableVersion for mode==2
(was missing unlike modes 0 and 1)
- Note snapshot type heuristic as imperfect in describe output
- Log retries in getBulkLoadTaskProgress
* Parse snapshot type from filename instead of inferring from heuristic
KeyspaceSnapshotFile now has a snapshotType field parsed from the
snapshot filename (format: snapshot,begin,end,size,type). The type
suffix was already written but discarded during parsing. Now captured
and used for reliable bulkdump vs rangefile identification in
fdbbackup describe output.
* Simplify getBulkLoadTaskProgress: always set LOCK_AWARE, drop lockAware parameter
* Use Transaction instead of ReadYourWritesTransaction in getBulkLoadTaskProgress
Read-only function does not need write tracking overhead.
Uses Transaction* overload of krmGetRanges.
* Refactor blob store classes to use interface
* update to make some methods non-pure virtual
* Complete IBlobStoreEndpoint interface refactoring
Move doRequest, connect, returnConnection, updateSecret, writeEntireFile,
listObjects, and deleteRecursively from S3BlobStoreEndpoint to the shared
IBlobStoreEndpoint base class. Introduce provider hook methods
(setRequestHeaders, normalizeResourceForRequest, simulateRequestFailure,
processRequestFailure, preRetryCheck, extractCredentialFields) so that
the request loop is provider-agnostic.
Rename s3_backup_test.sh to backup_restore_test.sh and update CMake
to reflect the provider-agnostic naming.
Fix double proxy absolute-form rewrite bug where both
S3BlobStoreEndpoint::normalizeResourceForRequest and the base doRequest
would prepend http://host:port for proxy connections.
Fix getResourceURL duplication by having S3's override delegate to the
base class for shared knob-params/extra-headers logic.
* resolve review comments
* remove gcs auto region block
* clang format
* Enable +x bit.
* clang tidy
Akanksha Mahajan