- Add chaos injection support to BulkDumping workload
- Added MockS3ServerChaos include to BulkDumping.actor.cpp
- Added chaos configuration options (errorRate, throttleRate, delayRate, corruptionRate, maxDelay)
- Enhanced _setup() to support both regular and chaos MockS3 servers
- Added detection for already-registered servers to avoid conflicts
- Configure chaos rates per-test to allow progressive intensity testing
- Created BulkDumpingS3WithChaos.toml test suite with 4 test scenarios:
* Stable (0% chaos) - baseline
* Light chaos (5-10%) - minor disruptions
* Medium chaos (15-20%) - moderate disruptions
* Heavy chaos (30-40%) - severe disruptions
This mirrors the S3ClientWorkloadWithChaos design and allows testing
BulkDumping resilience to S3 failures in simulation.
The validateBulkLoadJobHistory function incorrectly asserted that every
job in history had the same jobId, when it should have been searching
for a specific job. This caused crashes when multiple jobs existed in
history from cancellations/retries during chaos testing.
* Fix bulk load hangs with empty ranges and improve chaos test resilience
Empty Range Handling Fixes:
1. BulkLoadUtil.actor.cpp (bulkLoadDownloadTaskFileSets):
- Fixed root cause where empty ranges were skipped entirely with 'continue',
never added to localFileSets, causing FetchKeys to hang forever
- Now creates empty BulkLoadFileSet entries for empty ranges to track them
2. BulkLoadUtil.actor.cpp (bulkLoadTransportBlobstore_impl):
- Added hasDataFile() check before attempting S3 downloads
- Skip data file download gracefully for empty ranges
3. storageserver.actor.cpp (tryGetRangeForBulkLoad):
- Added hasDataFile() check to prevent calling getDataFileFullPath() on empty fileSets
- Added empty result signaling: when all ranges are empty, send empty RangeResult
with end_of_stream() to properly complete FetchKeys operations
4. storageserver.actor.cpp (bulk load SST ingestion):
- Added hasDataFile() checks before accessing data file paths during SST processing
5. storageserver.actor.cpp (restoreDurableState):
- Improved bulk load recovery robustness by using intersectingRanges() instead
of exact range match during storage server recovery
- Handle range boundary mismatches that can occur due to splits/chaos
- Added comprehensive error logging for debugging recovery issues
Performance and Retry Improvements:
1. Added BULKLOAD_DOWNLOAD_RETRY_DELAY knob (2.0s for both simulation and production)
2. Added BULKLOAD_DOWNLOAD_MAX_RETRIES knob (20 for both simulation and production)
- Total retry window: 20 retries × 2s = 40 seconds maximum
3. Reduced BLOBSTORE_MAX_DELAY_RETRYABLE_ERROR from 60s to 20s in production
4. Added retry limits to prevent infinite retry loops in bulk load downloads
Simulation found a case data movement takes longer to complete.
To repro on release-7.4 clang:
commit: ef0e70739f
seed: -f ./tests/fast/InventoryTestAlmostReadOnly.toml -s 583837717 -b off
This is the first experimental feature to be deleted in the list published at PR #12400.
There is more code here than I anticipated. It is about 40,000 lines total, of which about three quarters are in dedicated files which I am deleting, and about one quarter is in shared files. That means about 10k lines in shared files, which is the stuff we tend to notice day to day (that plus the test failures on heretofore not-yet-disabled test cases, which I am now deleting).
I ran 3 million simulations, mostly against 692df86 or very similar code (differing by one TraceEvent). This was prior to syncing with upstream/main, which had no conflicts and from which I don't expect problems. The number of failures in these runs was about 8. We looked at them and believe there is a high likelihood that these are existing issues not related to the changes in this PR. More details on these failures can be found in docs linked from here: https://quip-apple.com/MN7gAyXLjgyn
* change Long Term status for unowned features for "scheduled for deletion" where applicable
* Relax wording about scheduled for deletion features
* Delete blob granule feature. WIP. Does not compile.
* more incremental hacking to remove / comment out blob granule related code
* more hacking to remove blob granule related code, e.g. blob manager and blob migrator roles
* delete more blob granule stuff
* more hacking
* more hacking
* more hacking
* More changes to remove blob granule related code. IT COMPILES NOW
* dont try to run AuthzSecurity tests as we have deleted that workload as part of this effort
* delete more stuff that matches, abbreviates, or smells like blob granule related
* EncryptKeyProxy: dont do blobMetadata stuff, because that is not used and support is being removed
* delete more references to blob granule stuff
* SimulationConfig::setEncryptionAtRestMode: always use DISABLED; also disable EncryptKeyProxyTest.toml
* format code
* manual update to bindings/java/src/tests.cmake to remove a deleted file
* fix compile errors. I guess by default I dont build Java bindings
* remove unneeded blob granule functions rather than #if..#endif them out
* remove more code in #if..#endif
* remove more code in #if 0..#endif
* revert changes to fdb_c.h in preparation for marking removed API calls as removed
* rework C API declarations to in preparation for marking blob granule APIs as removed
* deprecate removed glob granule related API functions as of version 740 (and add a comment to request a justification of this convention)
* make progress on broken ctests. E.g. 1) python does not need to do blob granule stuff. 2) authz tests seemingly not needed
* remove blob granule stuff from Java and Python APIs and fix test runner stuff so that ctests pass
* reformat comments to fix compile error. FIXME: why is this error not happening on the default compile commands we use
* hacks all the way down to try to fix the Mac build
* add pointed comment about the perceived pointlessness of the API deprecation scheme embodied in this source file
* really serious about the C++ style comments, arent we
* remove commented-out code from prior iterative efforts
* put back undeleted code in original order
* delete commented-out code
* update feature-status.md to say blob granule is mostly deleted
* upgrade `mostly deleted` to `has been deleted`
Currently, when submitting backup, backup workers will be enabled for partitioned backups.
However, we didn't clear the backup worker setting if no partitioned backup is active,
which will cause backup workers to be recruited, but doing nothing.
This PR changes the behavior so that when submitting, aborting, or discontinuing
backups, we'll disable backup workers if there is no active partitioned backup jobs.
20250517-162642-jzhou-4966348e89f1794d
20250517-044345-jzhou-f02f7defca3ea010
This is a rewrite of BUGGIFY function/macros. Seems the performance
improved a lot during the simulation, e.g.
fdbserver -r simulation -b on -f ../CycleTest.toml -s 99438
Without this patch:
Unseed: 54646
Elapsed: 494.091327 simsec, 14.586831 real seconds
With this patch:
Unseed: 54646
Elapsed: 494.091327 simsec, 12.580612 real seconds
I expected the improvement but did not expect a ~13% improvement.
Found test failure due to 30s gate being too small:
./fdbserver.6.3.16 -r simulation -f ./tests/restarting/from_6.3.13_until_7.2.0/DrUpgradeRestart-1.txt -s 3218805329 -b on --logsize 1GB
-f ./tests/restarting/from_6.3.13_until_7.2.0/DrUpgradeRestart-2.txt --restarting -s 3218805330 -b on
clang build
commit 7d5d1e082
* added operational metrics and some polish
* moving consistency scan enablement in simulation tests to main tester workflow
* more stats and throttling polish
An 'attribute_not_found' error was slipping through the cracks
when attempting to get the max storage queue size for a storage
server which has been removed. This change explicitly logs the
error where it occurs, making it easier to debug.
When remote DC is down, the remote team collection of DD can initializing
waiting for the remote to recover (all_tlog_recruited state). However, the
getTeam request can already be served by the remote team collection. So, for
a RelocateShard (data movement such as split, move), it will get a team for
the remote DC. But the data movement can't make progress on the remote team
because the remote DC hasn't recovered yet. Because of the stuck of data
movement, the primary cannot reach the "storage_recovered" state and stay in
accepting_commit state.
The specifc test failure: slow/ApiCorrectness.toml -s 339026305 -b on
at commit: 0edd899d65
In this test, primary DC has 1 SS killed, remote DC has 2 TLog and 2 SS killed.
So the remote is dead, the remaining 2 SSes can't make progress because of the
loss of 2 TLogs. The repairDeadDatacenter() can't reach the "storage_recovered"
state due to DD's failure of moving shards away from the killed SS in the
primary.
The fix is to exclude all remote in repairDeadDatacenter() so that tells DD to
mark all SSes in the remote as unhealthy. Another fix is to return empty
results for getTeam request if the remote team collection is not ready. This
will allow the data movement to continue, essentially remote team is not changed
for the data movement.
* throttle the cluster when blob workers fall behind
* do not throttle on blob workers if they are not enabled
* remove an unnecessary actor
* fixed a compile error
* fetch blob worker metrics at the same interval as the rate is updated, avoid fetching the complete blob worker list too frequently
* fixed another compilation bug
* added a 5 second delay before bw throttling to prevent false positives caused by the 100e6 version jump during recovery. Lower the throttling thresholds to react much quicker to bw lag.
* fixed a number of problems
* changed the minBlobVersionRequest to look at storage server versions since this will be a lot more efficient
* fix: do not let desired go backwards
* fix: track the version of notAtLatest changefeeds for throttling
* ratekeeper now throttled blob workers by estimating the transaction per second throughput of the blob workers
* added metrics for blob worker change feeds
* added a knob to disable bw throttling
* fixed the transaction options in blob manager
* Shard based move.
* Clean up.
* Clear results on retry in getInitialDataDistribution.
* Remove assertion on SHARD_ENCODE_LOCATION_METADATA for compatibility.
* Resolved comments.
Co-authored-by: He Liu <heliu@apple.com>
QuietDatabase will now fail by itself after 1000 seconds
instead of relying on the general simulation timeout.
Additionally it will print a more human friendly error.
* Initialize cluster version at wall-clock time
Previously, new clusters would begin at version 0. After this change,
clusters will initialize at a version matching wall-clock time. Instead
of using the Unix epoch (or Windows epoch), FDB clusters will use a new
epoch, defaulting to January 1, 2010, 01:00:00+00:00. In the future,
this base epoch will be modifiable through fdbcli, allowing
administrators to advance the cluster version.
Basing the version off of time allows different FDB clusters to share
data without running into version issues.
* Send version epoch to master
* Cleanup
* Update fdbserver/storageserver.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
* Jump directly to expected version if possible
* Fix initial version issue on storage servers
* Add random recovery offset to start version in simulation
* Type fixes
* Disable reference time by default
Enable on a cluster using the fdbcli command `versionepoch add 0`.
* Use correct recoveryTransactionVersion when recovering
* Allow version epoch to be adjusted forwards (to decrease the version)
* Set version epoch in simulation
* Add quiet database check to ensure small version offset
* Fix initial version issue on storage servers
* Disable reference time by default
Enable on a cluster using the fdbcli command `versionepoch add 0`.
* Add fdbcli command to read/write version epoch
* Cause recovery when version epoch is set
* Handle optional version epoch key
* Add ability to clear the version epoch
This causes version advancement to revert to the old methodology whereas
versions attempt to advance by about a million versions per second,
instead of trying to match the clock.
* Update transaction access
* Modify version epoch to use microseconds instead of seconds
* Modify fdbcli version target API
Move commands from `versionepoch` to `targetversion` top level command.
* Add fdbcli tests for
* Temporarily disable targetversion cli tests
* Fix version epoch fetch issue
* Fix Arena issue
* Reduce max version jump in simulation to 1,000,000
* Rework fdbcli API
It now requires two commands to fully switch a cluster to using the
version epoch. First, enable the version epoch with `versionepoch
enable` or `versionepoch set <versionepoch>`. At this point, versions
will be given out at a faster or slower rate in an attempt to reach the
expected version. Then, run `versionepoch commit` to perform a one time
jump to the expected version. This is essentially irreversible.
* Temporarily disable old targetversion tests
* Cleanup
* Move version epoch buggify to sequencer
This will cause some issues with the QuietDatabase check for the version
offset - namely, it won't do anything, since the version epoch is not
being written to the txnStateStore in simulation. This will get fixed in
the future.
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
Instead, in order to enforce the maximum fault tolerance for snapshots,
update getStorageWorkers to return the number of unavailable storage
servers (instead of throwing an error when unavailable storage servers
exist).
Jingyu Zhou