Not a fix. Just an improvement in failure reporting: rather than run out the clock on a hang, fail fast with why.
Symptom
-------
Sideband.toml with seed 2881621233 (buggify on) ran the simulator to
sim-time ~21494s and died on TracedTooManyLines (1,000,001 trace lines).
The "too many lines" abort is only the simulator's dead-man switch; it is
not the real failure.
Actual issue
------------
waitForQuietDatabase() blocks indefinitely. After the workload and
consistency check, the End phase enters its pre-check recovery wait:
while (dbInfo->get().recoveryState != RecoveryState::FULLY_RECOVERED)
co_await dbInfo->onChange();
For this seed the cluster never returns to FULLY_RECOVERED. dbInfo keeps
updating (onChange fires ~70 times during the wait), so this is not a
stalled AsyncVar -- the recoveryState condition is simply never satisfied
because the cluster is genuinely wedged:
* Config is usable_regions=2 with satellites (one_satellite_triple).
* A recovery reached all_logs_recruited (RecoveryState 7) at t=128.8 and
never advanced. ClusterRecovery's trackTlogRecovery only reaches
STORAGE_RECOVERED / FULLY_RECOVERED once oldTLogData drains.
* The old TLog generation never drains because ~10 remote-region log
routers are permanently unable to find a primary peek location:
NoPrimaryPeekLocationForLR / LogRouterSlowPeek fire continuously from
t=368 to the end of the run (LogID="[not set]"). peekLogRouter keeps
returning a cursor with bestServer < 0, so getPrimaryPeekLocation() is
empty. No peek -> old generation can't pop -> recovery wedged.
* All DCs still have live processes at the end, so no region "died" --
this is a peek-resolution deadlock, not lost quorum.
Because the recovery wait has no deadline, the test hangs for ~21000s and
is finally killed by the trace-line cap with a useless message instead of
failing at the real problem.
This change (harness fix)
-------------------------
Bound the recovery wait by the same maxDDRunTime budget the quiet-database
checks already enforce, and on timeout fail with the same ddGotStuck
semantics: in simulation assert (fast, loud, named failure); on a real
cluster log SevWarnAlways and fall through to keep retrying. The added
delay() keeps the deadline responsive and also covers a genuinely frozen
dbInfo.
Result for this seed: clean termination at t=4134 (~5x faster, trace
properly closed) with
QuietDatabaseNeverFullyRecovered Phase="End" RecoveryState="7"
FailedAssertion="!g_network->isSimulated()" QuietDatabase.cpp
which points straight at "recovery never advanced past all_logs_recruited"
instead of burying it under a 21000s trace avalanche.
What still needs fixing (NOT addressed here)
--------------------------------------------
This only makes the failure fast and diagnosable; it does not make the
seed pass, because the underlying product bug remains: in usable_regions=2,
a remote-region log router that cannot find a primary peek location can
wedge recovery indefinitely (old generation never drains). LogRouter.cpp
already flags this as needing manual intervention ("The LR may become a
bottleneck on the system and need to be excluded"), but in simulation there
is no operator. The real fix is for the cluster controller / recovery to
detect a persistently-stuck log router and automatically exclude and
re-recruit it (or otherwise avoid depending on it forever) so recovery can
complete without manual intervention.
Problem: Data Distribution has no global cap on total tracked moves. The total is an emergent property of several independent per-server and per-phase limits. Burst failures can generate thousands of relocations that overwhelm DD on restart.
Solution: New knob DD_MAX_PIPELINE_MOVES (default 1000, BUGGIFY 5) caps total moves DD tracks at any time (gate-pending + queued + in-flight). A pipelineGateActor sits between the raw relocation input stream and DDQueue's main loop, holding relocations when the pipeline is full and forwarding them when room opens. Back-pressure is event-driven via an AsyncVar<bool> pipelineFull signal.
Other notes:
Exemptions — Cancellations always pass through (they reduce metadata). Moves with priority >= PRIORITY_TEAM_UNHEALTHY (700) bypass the gate, which covers failure-recovery
priorities as well as shard splits.
Simulation quiescence — waitForQuietDatabase() permanently disables the gate via a global disableDDPipelineControl() flag so DD can drain all pending moves before consistency
checks.
Safety margin — Production clusters have been observed doing 25-30GB in flight (~100 shards), well within the 1000 default.
Observability:
Pragmatically, grepping for DDPipe-prefixed messages suffices to tell if this control has been active.
Testing:
20260503-052118-gglass-93c982cafc028215 compressed=True data_size=37065700 duration=5090249 ended=100000 fail=1 fail_fast=10 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:56:56 sanity=False started=100000 stopped=20260503-061814 submitted=20260503-052118 timeout=5400 username=gglass
(see comments below about prior runs)
* Attempt to enforce a "pipeline max" number of relocations in DD
* formatting
* dial back logging frequency for new code
* Add a way to disable pipeline control to let simulation unblock pending moves prior to quiescence
* update comment on default value of DD_MAX_PIPELINE_MOVES
* update comment justifying DD_MAX_PIPELINE_MOVES of 1000
* Update pipeline accounting to account for the new stage we are adding. Compiles but not tested
* Address second Codex review comment
* add self->
Michael Stack