* Integrate BulkDump/BulkLoad with backup/restore system
This commit adds the ability to use BulkDump for creating backup snapshots
and BulkLoad for restoring them, providing faster backup/restore operations
for large databases.
Key changes:
- Add BulkDumpTaskFunc to create SST file snapshots during backup
- Add BulkLoadRestoreTaskFunc to restore from BulkDump snapshots
- Store bulkDumpJobId in snapshot metadata for restore coordination
- Add snapshotMode parameter (0=RANGEFILE, 1=BULKDUMP) to control backup type
- Add useRangeFileRestore parameter to control restore method
- Add CLIENT_KNOBS for configurable job timeouts
- Add test assertions to verify BulkDump/BulkLoad execution
- Check for existing running jobs to avoid conflicts when multiple agents run
- Properly scope state variables for error handling in Flow actors
New test: tests/slow/BackupS3BlobBulkLoadRestore.toml
* Update design/bulkload-restore-integration.md
- Add chaos injection support to BulkDumping workload
- Added MockS3ServerChaos include to BulkDumping.actor.cpp
- Added chaos configuration options (errorRate, throttleRate, delayRate, corruptionRate, maxDelay)
- Enhanced _setup() to support both regular and chaos MockS3 servers
- Added detection for already-registered servers to avoid conflicts
- Configure chaos rates per-test to allow progressive intensity testing
- Created BulkDumpingS3WithChaos.toml test suite with 4 test scenarios:
* Stable (0% chaos) - baseline
* Light chaos (5-10%) - minor disruptions
* Medium chaos (15-20%) - moderate disruptions
* Heavy chaos (30-40%) - severe disruptions
This mirrors the S3ClientWorkloadWithChaos design and allows testing
BulkDumping resilience to S3 failures in simulation.
The validateBulkLoadJobHistory function incorrectly asserted that every
job in history had the same jobId, when it should have been searching
for a specific job. This caused crashes when multiple jobs existed in
history from cancellations/retries during chaos testing.
* Fix bulk load hangs with empty ranges and improve chaos test resilience
Empty Range Handling Fixes:
1. BulkLoadUtil.actor.cpp (bulkLoadDownloadTaskFileSets):
- Fixed root cause where empty ranges were skipped entirely with 'continue',
never added to localFileSets, causing FetchKeys to hang forever
- Now creates empty BulkLoadFileSet entries for empty ranges to track them
2. BulkLoadUtil.actor.cpp (bulkLoadTransportBlobstore_impl):
- Added hasDataFile() check before attempting S3 downloads
- Skip data file download gracefully for empty ranges
3. storageserver.actor.cpp (tryGetRangeForBulkLoad):
- Added hasDataFile() check to prevent calling getDataFileFullPath() on empty fileSets
- Added empty result signaling: when all ranges are empty, send empty RangeResult
with end_of_stream() to properly complete FetchKeys operations
4. storageserver.actor.cpp (bulk load SST ingestion):
- Added hasDataFile() checks before accessing data file paths during SST processing
5. storageserver.actor.cpp (restoreDurableState):
- Improved bulk load recovery robustness by using intersectingRanges() instead
of exact range match during storage server recovery
- Handle range boundary mismatches that can occur due to splits/chaos
- Added comprehensive error logging for debugging recovery issues
Performance and Retry Improvements:
1. Added BULKLOAD_DOWNLOAD_RETRY_DELAY knob (2.0s for both simulation and production)
2. Added BULKLOAD_DOWNLOAD_MAX_RETRIES knob (20 for both simulation and production)
- Total retry window: 20 retries × 2s = 40 seconds maximum
3. Reduced BLOBSTORE_MAX_DELAY_RETRYABLE_ERROR from 60s to 20s in production
4. Added retry limits to prevent infinite retry loops in bulk load downloads
* add trace event to notify users when giving up sst file ingestion due to task range not aligned with manifests
* Testing bulkdump per range
* address comments
* use retryable
* dedup code in getBulkLoadJobFileManifestEntryFromJobManifestFile
* * fdbclient/S3Client.actor.cpp
Change field names so capitialized (convention)
Add duration as field to traces.
* fdbserver/BulkLoadUtil.actor.cpp
When the job-manifest is big, processing blocks
so much getBulkLoadJobFileManifestEntryFromJobManifestFile
fails.
* Make bulkload file reads and writes async and memory parsimonious.
In tests at scale, processing a large job-manifest.txt was blocking
and causing the bulk job to fail. This is part 1 of two patches.
The second is to address data copy added in the below when we
made methods ACTORs (ACTOR doesn't allow passing by reference).
* fdbserver/BulkDumpUtil.actor.cpp
Removed writeStringToFile and buldDumpFileCopy in favor of new methods
in BulkLoadUtil. Made hosting functions ACTORs so could wait on
async calls.
* fdbserver/BulkLoadUtil.actor.cpp
Added async read and write functions.
* fdbserver/DataDistribution.actor.cpp
Making uploadBulkDumpJobManifestFile async made it so big bulkloads
work.
* fix memory corruption in writeBulkFileBytes and fix read options in getBulkLoadJobFileManifestEntryFromJobManifestFile
* If read or write < 1MB, do it in a single read else do multiple read/writes
* * packaging/docker/fdb-aws-s3-credentials-fetcher/fdb-aws-s3-credentials-fetcher.go
Just be blunt and write out the credentials. Trying to figure when the
blob credentials have expired is error prone.
Co-authored-by: michael stack <stack@duboce.com>
Co-authored-by: Zhe Wang <zhe.wang@wustl.edu>
* improve bulkload code
* address CI
* disable audit storage replica check and distributed consistency check in bulkload and bulkdump simulation test
* fix ci
* disable waitForQuiescence in bulkload and bulkdump tests
* bulkload support general engine and fix bugs
* add comments
* improve test coverage and fix bug
* nits and address comments
* nit
* nits
* fix data inconsistency bug due to bulkload metadata
* fix ss bulkload task metadata bugs
* nit and fix CI issue
* fix bugs of restore ss bulkload metadata
* use ssBulkLoadMetadata for fetchKey and general kv engine
* cleanup bulkload file for fetchkey
* fix CI issue
* fix simulation stuck due to repeated re-recruitment of unfit dd
* randomly do available space check when finding the dest team for bulkload in simulation
* address conflict
* code clean up
* update BulkDumping.toml same to BulkLoading.toml
* consolidate ss fetchkey and fetchshard failed to read bulkload task metadata
* fix DD bulkload job busy loop bug which causes segfault and test terminate unexpectedly in joshua test
* nit
* fix ss busy loop for bulkload in fetchkey
* use sqlite for bulkload ctest
* fix bulkload ctest stuck issue due to merge and change storage engine to ssd
* fix comments for CC recruit DD
* address comments
* address comments
* add comments
* fix ci format issue
* address comments
* add comments
It was disabled until we made it so the SS could
talk to s3, included in this PR.
Also finished the bulkload test. It only had the
bulkdump portion. bulkload support was recentlty
added so finish off the test here by adding bulkload
of the bulkdump and then verifying all data present.
Added passing knobs to the fdb cluster so available to the
fdbserver when it goes to talk to s3. Also added passing
SS count to start in fdb cluster.
* fdbclient/tests/fdb_cluster_fixture.sh
Add ability to pass multiple knobs to fdb cluster
and to specify more than just one SS.
* fdbserver/fdbserver.actor.cpp
Add --blob-server option and processing of FDB_BLOB_CREDENTIALS
if present (hijacked the unused, unadvertised --
blob-credentials-file).
* tests/loopback_cluster/run_custom_cluster.sh
Allow passing more than just one knob.
* fdbclient/BulkLoading.cpp
* fdbclient/include/fdbclient/BulkLoading.h
Added getPath
* fdbclient/S3BlobStore.actor.cpp
Fix bug where we were doubling up the first '/' on a path if
it had a root '/' already (s3 treats /a/b as distinct from
/a//b).
* fdbclient/S3Client.actor.cpp
Fix up of traceevent Types.
* fdbclient/tests/bulkload_test.sh
Enable being able to use s3 if available.
Pick up jobid when bulkdumping. Feed it to new bulkload
method. Add verification all data present post-bulkload.
* fdbserver/BulkLoadUtil.actor.cpp
Add support for blobstore.
* tests/loopback_cluster/run_custom_cluster.sh
Bug fix -- we were only able to pass in one knob. Allow
passing multiple.
* add bulkload job framework and fix bugs
* add BulkLoadChecksum, fix CI issue
* nits
* nits
* address comments
* mitigate perpetual wiggle to make sure DD can select a valid team to inject data
* fix submitBulkDumpJob and submitBulkLoadJob
* change remoteRoot to jobRoot
* add comments
* Improve BulkLoad/Dump implementation
* make bulkload test data folder inside simfdb folder
* simplify code
* use manifest in bulkdump metadata
* use manifest in bulkload
* apply bulkload fileset to bulkload and fix bugs of bytesampling value generation
* remove BulkDumpFileFullPathSet
* address comments
* address comments
* address comments
Jingyu Zhou