* Add BulkDump/BulkLoad observability for backup/restore progress tracking
Progress tracking APIs:
- getBulkDumpProgress() / getBulkLoadProgress() - track tasks, bytes, throughput, ETA
- getBulkDumpOwner() - track job ownership (e.g., which backup owns a bulkdump)
- Owner registration system for backup agents
CLI improvements:
- 'bulkdump status' and 'bulkload status' commands with human-readable output
- Progress display with throughput and ETA
Backup agent integration:
- Register backup as owner of bulkdump jobs
- Show bulkdump/bulkload progress in backup status output
Utility functions (flow/Util.h):
- formatBytesHumanReadable() - "1.23 TB", "456 MB"
- formatDurationHumanReadable() - "2 hours 30 minutes"
- formatBytesProgress(), formatThroughputLine(), formatETALine()
Test coverage:
- verifyBulkDumpObservability() / verifyBulkLoadObservability() in BackupS3BlobCorrectness
Bug fixes:
- Fix format string mismatch crash in FileBackupAgent progress display
- Add empty result guards for krmGetRanges() calls to prevent size_t underflow
* Formatting
* Rebase and formatting
* * cmake/CompileRocksDB.cmake
Preserve version when find_packge doesn't turn up rocksdb version
* fdbbackup/tests/backup_tests_common.sh
Wait when mode is 'both' for snapshot to be written.
* fdbclient/BulkLoading.cpp
Remove extra '/'.
* Output time of bulkload in bulkload history fdcli output as iso8601 instead of seconds since epoch
* Print start time as iso8601 -- readable -- instead of seconds since epoch -- unreadable... And don't crash if not a UUID
* Remove cute graphics.. and bullet characters
* Throw exception if not able to parse backup 'mode'
* Integrate BulkDump/BulkLoad with backup/restore system
This commit adds the ability to use BulkDump for creating backup snapshots
and BulkLoad for restoring them, providing faster backup/restore operations
for large databases.
Key changes:
- Add BulkDumpTaskFunc to create SST file snapshots during backup
- Add BulkLoadRestoreTaskFunc to restore from BulkDump snapshots
- Store bulkDumpJobId in snapshot metadata for restore coordination
- Add snapshotMode parameter (0=RANGEFILE, 1=BULKDUMP) to control backup type
- Add useRangeFileRestore parameter to control restore method
- Add CLIENT_KNOBS for configurable job timeouts
- Add test assertions to verify BulkDump/BulkLoad execution
- Check for existing running jobs to avoid conflicts when multiple agents run
- Properly scope state variables for error handling in Flow actors
New test: tests/slow/BackupS3BlobBulkLoadRestore.toml
* Update design/bulkload-restore-integration.md
* Add multiparting to s3client.
Fix boost::urls::parse_uri 's dislike of credentialed blobstore urls.
* fdbclient/BulkLoading.cpp
Add blobstore regex to extract credentials before feeding the boost
parse_uri.
* fdbclient/include/fdbclient/S3BlobStore.h
* fdbclient/S3BlobStore.actor.cpp
Add cleanup of failed multipart -- abortMultiPartUpload l(s3 will do
this in the background eventually but lets clean up after ourselves).
Also add getObjectRangeMD5 so can do multipart checksumming.
* fdbclient/S3Client.actor.cpp
Change upload file and download file to do multipart always.
Retry too.
* fdbclient/S3Client_cli.actor.cpp
Add command line to trace rather than output.
* Address Zhe review
* More logging around part upload and download
* Undo assert that proved incorrect; restore the old length math
doing copy in readObject.
Cleanup around TraceEvents in HTTTP.actor.
* Undo commented out cleanup -- for debugging
* formatting
---------
Co-authored-by: stack <stack@duboce.com>
* bulkload support general engine and fix bugs
* add comments
* improve test coverage and fix bug
* nits and address comments
* nit
* nits
* fix data inconsistency bug due to bulkload metadata
* fix ss bulkload task metadata bugs
* nit and fix CI issue
* fix bugs of restore ss bulkload metadata
* use ssBulkLoadMetadata for fetchKey and general kv engine
* cleanup bulkload file for fetchkey
* fix CI issue
* fix simulation stuck due to repeated re-recruitment of unfit dd
* randomly do available space check when finding the dest team for bulkload in simulation
* address conflict
* code clean up
* update BulkDumping.toml same to BulkLoading.toml
* consolidate ss fetchkey and fetchshard failed to read bulkload task metadata
* fix DD bulkload job busy loop bug which causes segfault and test terminate unexpectedly in joshua test
* nit
* fix ss busy loop for bulkload in fetchkey
* use sqlite for bulkload ctest
* fix bulkload ctest stuck issue due to merge and change storage engine to ssd
* fix comments for CC recruit DD
* address comments
* address comments
* add comments
* fix ci format issue
* address comments
* add comments
It was disabled until we made it so the SS could
talk to s3, included in this PR.
Also finished the bulkload test. It only had the
bulkdump portion. bulkload support was recentlty
added so finish off the test here by adding bulkload
of the bulkdump and then verifying all data present.
Added passing knobs to the fdb cluster so available to the
fdbserver when it goes to talk to s3. Also added passing
SS count to start in fdb cluster.
* fdbclient/tests/fdb_cluster_fixture.sh
Add ability to pass multiple knobs to fdb cluster
and to specify more than just one SS.
* fdbserver/fdbserver.actor.cpp
Add --blob-server option and processing of FDB_BLOB_CREDENTIALS
if present (hijacked the unused, unadvertised --
blob-credentials-file).
* tests/loopback_cluster/run_custom_cluster.sh
Allow passing more than just one knob.
* fdbclient/BulkLoading.cpp
* fdbclient/include/fdbclient/BulkLoading.h
Added getPath
* fdbclient/S3BlobStore.actor.cpp
Fix bug where we were doubling up the first '/' on a path if
it had a root '/' already (s3 treats /a/b as distinct from
/a//b).
* fdbclient/S3Client.actor.cpp
Fix up of traceevent Types.
* fdbclient/tests/bulkload_test.sh
Enable being able to use s3 if available.
Pick up jobid when bulkdumping. Feed it to new bulkload
method. Add verification all data present post-bulkload.
* fdbserver/BulkLoadUtil.actor.cpp
Add support for blobstore.
* tests/loopback_cluster/run_custom_cluster.sh
Bug fix -- we were only able to pass in one knob. Allow
passing multiple.
* add bulkload job framework and fix bugs
* add BulkLoadChecksum, fix CI issue
* nits
* nits
* address comments
* mitigate perpetual wiggle to make sure DD can select a valid team to inject data
* fix submitBulkDumpJob and submitBulkLoadJob
* change remoteRoot to jobRoot
* add comments
* Improve BulkLoad/Dump implementation
* make bulkload test data folder inside simfdb folder
* simplify code
* use manifest in bulkdump metadata
* use manifest in bulkload
* apply bulkload fileset to bulkload and fix bugs of bytesampling value generation
* remove BulkDumpFileFullPathSet
* address comments
* address comments
* address comments
Trevor Clinkenbeard