With single storage replication, a rapid recovery-generation handoff during
initial database bring-up can orphan the sole seed storage server before it
makes the seed data durable: the superseding recovery pops the previous
generation's TLogs, the still-catching-up storage server hits PeekPoppedTLogData
and is removed (its KVS deleted), and because there is only one replica no
storage server remains and none is re-seeded. The database is then permanently
unavailable and the test's starting-configuration change times out
(TestFailure 'Unable to set starting configuration', e.g. seed 327963237).
This is a pre-existing single-replica recovery race, unrelated to the range-lock
feature the test exercises. Forcing at least double redundancy keeps the
database available across the handoff, matching how other startup-sensitive
tests (e.g. LowLatency) pin minimumReplication.
Ports apple/foundationdb#13297 to release-7.4.
The Cycle workload's built-in minimum-rate check (expectedRate, default 0.7)
conflates a throughput measurement with the cycle-invariant correctness check,
so Cycle false-fails with "Rate below desired rate" whenever it runs under load
(Attrition, RandomClogging, kills) even though the invariant holds. Remove that
check from Cycle and move dedicated throughput testing into a new
MinimumThroughput workload that runs only in error-free conditions.
- fdbserver/workloads/Cycle.actor.cpp: drop minExpectedTransactionsPerSecond,
the expectedRate option, and the "Rate below desired rate" TestFailure.
- fdbserver/workloads/MinimumThroughput.cpp (new) + tests/fast/MinimumThroughput.toml
(new) + CMakeLists registration: dedicated throughput workload that disables
all failure injection and runs with buggify off.
- Strip the now-dead expectedRate option from every Cycle-workload test
(136 occurrences across 93 tomls/txts). expectedRate is retained where the
Increment workload still consumes it (IncrementTest.toml,
SwizzledRollbackTimeLapseIncrement.toml); #13297 does not touch Increment.
Layout adaptations from main: Cycle.cpp -> Cycle.actor.cpp; MinimumThroughput
includes fdbserver/TesterInterface.actor.h + fdbserver/workloads/workloads.actor.h
(plus flow/Coroutines.h for co_await) rather than main's fdbserver/core and
fdbserver/tester paths. The expectedRate strip additionally covers 7.4-only
Cycle tests absent on main (older restarting/ dirs, legacy .txt tests) so no
test is left with an unconsumed option (which fdbserver treats as a failure).
Testing: Release build (clang, WERROR). fdbserver -r simulation:
- CycleTest.toml seeds 1-5 (buggify on): all PASS; the same seeds failed 5/5
on "Rate below desired rate" (SevError-free) before this change.
- MoveKeysCycle, CycleWithKills, CloggedCycleTest: PASS, zero
"Not all options consumed" events (expectedRate strips verified correct).
- MinimumThroughput.toml seeds 1-2: PASS.
- IncrementTest still fails its own (unchanged) Increment rate check under
buggify -- pre-existing, out of scope for #13297; its expectedRate is
consumed (no unconsumed-option error).
Joshua validation pending.
* Do not check for PROXY_USE_RESOLVER_PRIVATE_MUTATIONS in rangeLockEnabled
* Dont modify knob proxy_use_resolver_private_mutations in range lock tests
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* disable version vector with range lock tests
* turn off rangeLock if versionvector is on (#11747)
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
Co-authored-by: Zhe Wang <zhe.wang@wustl.edu>
* range lock framework
* improve the framework
* persist to txnStateStore
* fix bugs
* code clean
* code clean
* bug fix
* address comments
* add complex test workload and fix bugs found by the workload
* add workload correctness check and fix bugs
* code clean up
* add random range lock injection
* fix bugs in RandomRangeLock.actor.cpp
* enable random range lock injection in general workloads
* add rangelockcycle test
* disable random range lock in backup workloads
* nits
* add range lock ownership concept
* enable lock ownership to rangeLock
* api deal with tenant
* fix CI
* add test for multiple rangeLock owners
* nits
* address comments and renaming
* address comments
Syed Paymaan Raza