* fix: Non-storage processes were not being checked for locality exclusions
fix: Data distribution when not detect a newly added process was locality excluded
fix: RemoveServerSafely did not wait for processes to be excluded before killing them when excluding localities
* fix: do not allow locality based excludes if they cannot exclude the required addresses
* Remove duplicate getRange() for DB handles and update existing GetRange to accept DB handles.
* Initial progress checkpoint on new ConsistencyScan role.
* Updated TODOs, finished most if not all state updates.
* placeholder
* Add more TODOs, documentation and comment improvements.
* Checkpoint round state to avoid advancing progress if commit fails.
* Bug fix, check is supposed to be for overlap, not lack of overlap.
* Added more TODO's and added faked read results / exceptions and faked DB size retrieval to prove the consistencyScanCore logic works.
* Update JSON schemas and command help.
* Add comment about lifetime stats reset.
* More TODO comments and some renames for clarity, some bug fixes.
* properly stopping consistency scan in simulation so that it doesn't run forever and cause quiet database to fail
* removing trailing comma from consistency_scan json schema
* Making CC inconsistency not an error if it's intentional tss corruption
* consistency scan actually reads storage locations
* added check that consistency scan actually completes a round in simulation, fixed bug and added debugging around consistency scan getting stuck
* made consistency scan properly fetch database size
* refactoring data check to be used in both consistency scan and consistency check
* checking that consistency scan always completes at least one round and doesn't get stuck
* cleanup
* fixing ide build
* consistencyscan fdbcli command wasn't actually changing db state
* consistencyscan fdbcli command always said enabled even when it wasn't
---------
Co-authored-by: Steve Atherton <steve.atherton@snowflake.com>
* Passes existing tests
* adding http unit test for wrong md5 sum
* Added new HTTPKeyValueStore workload to test long-running http clients
* fixing warnings
CheckExtraDataStores reboots or kills storage servers with extra data stores.
Since this occurs during a consistency check, the expectation is that the database
is quiet and not in the midst of recovery. This was done with a 10-second delay,
but it's possible during simulation tests that it takes longer than 10 seconds
to recruit a new master, so this assumption is invalid and can cause a test failure
when the consistency checks proceed.
Instead of a delay, we run an empty transaction through the system and explicitly
wait for the cluster to return to a fully-recovered state.
When VV is enabled, the waitForVersion() may use the oldest version in the MVCC window as the actual read version, which may often result in the transaction_too_old if the MVCC moved, especially during a readRange(). The change here is to try using a higher read version that is still valid, but not likely to cause transaction_too_old error.
* fix: the log router could see popped() versions between recoveredAt and the recovery txn version which cause remote tlogs to be stopped at a version in this range, which causes storage servers to read at versions in this range
* fix: a spurious correctness bug resulted from the consistency check running during a recovery
* Recruit new singleton for consistency checker.
* Recruit the consistency checker only if enabled.
* Add a yield in monitorConsistencyChecker().
* Minor fixes.
* Consistency check workload enhancements.
* Minor fixes and clarifications.
* clang format
* Clang format.
* Minor fixes, cleanup, debug tracing.
* Misc.
* Move the consistency scan information from dbconfig to a key backed object.
* Move consistency scan config out of db cofig to a state object and feature rename.
* ConsistencyCheck workload refactor.
* devFormat
* Update fdbcli/ConsistencyScanCommand.actor.cpp
* Review Comments.
Co-authored-by: negoyal <neelam.goyal@gmail.com>
Co-authored-by: Ata E Husain Bohra <ata.husain@snowflake.com>
* proof of concept
* use code-probe instead of test
* code probe working on gcc
* code probe implemented
* renamed TestProbe to CodeProbe
* fixed refactoring typo
* support filtered output
* print probes at end of simulation
* fix missed probes print
* fix deduplication
* Fix refactoring issues
* revert bad refactor
* make sure file paths are relative
* fix more wrong refactor changes
Previously to get around the issue that EKP is not present when restart test switching encryption from on to off and read encrypted data, EKP was made to start in simulation regardless of encryption knob. This PR revert that change, and instead force restart test not to change encryption knob, by passing previous encryption knob through restartInfo.ini file. Also since we don't allow downgrading an encrypted cluster to previous version, disable encryption in downgrade tests.
Also adding an assert to allow reading encrypted mutations only if encryption knob is on. We may reconsider allowing switching encryption on/off for existing cluster, but for now we don't allow it.
* Shard based move.
* Clean up.
* Clear results on retry in getInitialDataDistribution.
* Remove assertion on SHARD_ENCODE_LOCATION_METADATA for compatibility.
* Resolved comments.
Co-authored-by: He Liu <heliu@apple.com>
Evan Tschannen