* Fix GcGenerations test: block trackRecoveryReq when disableTLogRecoveryFinish is set
disableTLogRecoveryFinish only blocked TLogRecoveryFinishedRequest but
not TrackTLogRecoveryRequest. This allowed recoveredVersion to advance
and purgeOldRecoveredGenerationsCoreState to GC generations during the
accumulation phase, causing the test assertion to fire when oldTLogs
shrank instead of grew. Also fix the wait loop predicate to use <=
instead of == so it only exits on strict growth.
* Fix GcGenerations test: skip remote DC masters and cleanup on timeout
When the remote DC is clogged, the CC may recruit the master there.
Rebooting a remote DC master stalls recovery because it cannot
communicate with primary DC processes. Only reboot primary DC masters
and retry if the master is in the remote DC.
Add destructor cleanup so that if the workload times out, simulator
state (clog, connection failures, disableTLogRecoveryFinish) is
restored, preventing the Cycle workload check from failing on a
permanently degraded cluster.
Tested: 3 failures in 100k joshua runs (all pre-existing: 2x
TracedTooManyLines from waitForQuietDatabase hang, 1x LogRouter
segfault). Zero GcGenerations workload assertion failures.
* Formatting
* Revert <= back to == in generation wait loop
GC should not reduce oldTLogs.size() while disableTLogRecoveryFinish
is true, so the defensive <= check is unnecessary and could mask a
real problem.
* Add suppressFor(60) to WaitingToBeUnblocked trace event
Without suppression, each blocked TLog emits a trace line every 10s.
Across many TLogs and recovery generations this hits TracedTooManyLines
causing the process to abort, which is the source of the test timeouts
and SIGTERM failures (10 out of 19k runs).
* Reboot master after clearing disableTLogRecoveryFinish to unstick GC
The trackRecoveryReq blocking during generation accumulation prevents
TLogs from reporting their recovered state. After the flag is cleared,
the current recovery tracking is stale: FinalUpdate never fires because
the blocked TLogs missed their reporting window. Without FinalUpdate,
purgeOldRecoveredGenerationsCoreState never runs, and oldTLogs stays
at 15 forever.
Fix by rebooting the master after clearing the flag. The fresh recovery
starts with clean tracking state, all TLogs respond normally, FinalUpdate
fires, and generation GC proceeds.
* Fix peek cursor assertion crash and GC timeout in GcGenerations test
Two fixes for remaining GcGenerations test failures:
1. enableConnectionFailures instead of extendConnectionFailures:
extendConnectionFailures only pushes out connectionFailureDisableTime
without resetting connectionFailureEnableTime. After enough loop
iterations, the peek cursor assertion window at
LogSystemPeekCursor.actor.cpp:358 opens while clogged pairs are still
active, causing ASSERT_WE_THINK crashes.
2. GC retry loop with rebootPrimaryMaster helper:
Replace single master reboot + generationReduced wait with a loop that
periodically reboots the master until oldTLogs.size() <= 1. GC may need
multiple recovery cycles because remote TLogs must catch up from old
generations before remoteRecoveredVersion advances past their recoverAt,
and purgeOldRecoveredGenerationsCoreState only purges below that.
Summary of all changes from the original test (commit 673a9fce5b):
- Destructor: original had no cleanup on timeout. Timeout left
clog/connection-failures/disableTLogRecoveryFinish active, poisoning
the Cycle check.
- rebootPrimaryMaster helper: original rebooted whatever master it found,
including remote DC masters that can't coordinate recovery when the
remote DC is clogged.
- for -> while with remote DC guard in accumulation loop: same issue,
original unconditionally rebooted the master.
- extendConnectionFailures -> enableConnectionFailures: extendConnectionFailures
doesn't reset connectionFailureEnableTime. After enough iterations, the
peek cursor assertion window opens while clogged pairs are still active.
- trackRecoveryReq blocking in TLogServer.actor.cpp: original didn't block
this. Without it, disableTLogRecoveryFinish doesn't actually prevent GC,
so generations get GC'd during accumulation.
- Master reboot after clearing disableTLogRecoveryFinish: after blocking
trackRecoveryReq, the current recovery's tracking is stale. Need a fresh
recovery for GC to proceed.
- GC retry loop replacing single generationReduced wait: original assumed
GC completes in one recovery cycle. Remote TLogs need to catch up through
multiple generations, so remoteRecoveredVersion may need multiple recovery
cycles to advance past all recoverAt versions.
* Reboot remote DC masters in dbAvailable to prevent indefinite blocking
When the master is elected in the clogged remote DC, recovery can never
reach ACCEPTING_COMMITS because the master cannot communicate with primary
DC processes. dbAvailable() would block forever waiting for a state that
can never be reached, consuming the entire 1000s workload budget.
Fix by checking the master DC on each dbInfo update inside dbAvailable.
If recovery has not reached ACCEPTING_COMMITS and the master is in the
remote DC, reboot it immediately to force the CC to elect a primary DC
master. Extract isMasterInRemoteDc helper to share the check.
* Fix GC phase stuck on remote DC master after unclogging
After unclogging the remote DC and entering the GC phase,
rebootPrimaryMaster refused to reboot remote DC masters. But with the
clog removed, a remote DC master can coordinate recovery just fine.
The loop spun every 5s for the entire remaining test budget, always
seeing the same remote DC master, until timeout.
Fix by rebooting whatever master is present in the GC phase (no DC
guard needed since the remote DC is unclogged). Remove the now-unused
rebootPrimaryMaster helper. The primary-DC-only guard remains in the
accumulation loop where the remote DC is still clogged.
Move disableTLogRecoveryFinish=false before generationReduced() wait.
This commit:
commit 774c792e14
Author: Jingyu Zhou <jingyu_zhou@apple.com>
Date: Mon Jun 30 16:21:45 2025 -0700
Fix a race that can cause recovery to be stuck (#12212)
* Fix a race that can causes recovery to be stuck
changed the generation GC ordering to prevent a race condition where:
1. In-memory oldLogData was purged first
2. Recovery could happen before coordinator state was written
3. TLogs could be displaced, causing recovery to get stuck
The above now updates coordinator state first, then syncs in-memory state,
This test was blocking TLog recovery throughout, then waiting for
generation reduction while TLogs remained blocked. The 774c792e14
fix correctly prevents GC in this state. After the above change, the
test would timeout. Previous to the above, this test was Gc'ing generations
BEFORE the update to coordinator state based off in-memory state. The test
was passing. It started to fail after 774c792e14.
This fix maintains the test's intent:
- Block TLogs during generation creation to create recovery stress
- Unblock TLogs before generation reduction to allow safety criteria evaluation
- Verify that old generations are properly cleaned up
Co-authored-by: stack <stack@duboce.com>
Trevor Clinkenbeard