* Extend PEEK_REPLY_TIMEOUT to parallel peek path (LogRouter fix)
The sender-side peek timeout added in 446079cc (PR #13248) only covers
the non-parallel peek path (serverPeekGetMoreImpl), used by storage
servers. LogRouters use the parallel peek path (serverPeekParallelGetMoreImpl)
via peekLogRouter/SetPeekCursor, which still had the old wait-forever
behavior.
Add the same PEEK_REPLY_TIMEOUT race branch to serverPeekParallelGetMoreImpl.
When the timeout fires, clear the pending futures and re-send the peek
(same retry-without-throw approach as the non-parallel path).
Extract peekReplyTimeout() helper used by both paths to compute the
timeout future consistently. Both sites cross-reference each other in
comments.
This fixes the live-TLog LogRouter hang where a TLog is re-recruited
with new endpoint tokens. The LogRouter's peek request is silently
dropped (same root cause as the dead-TLog hang), but the parallel path
never detected it.
Seed 408543238 (TxnStateStoreCycleTest.toml, buggify=on) reproduces on
x86_64 linux. Passes with this fix (206s vs 24000+ hang).
Also fix fdb_c_wiggle_only hang on shutdown due to stats timer race.
The fdb_c_api_tester process hangs on shutdown after all workloads
complete, causing the test harness to SIGKILL it and report failure.
Root cause: schedulePrintStatistics() re-schedules itself inside its
timer callback. When the last workload finishes, workloadDone() calls
statsTimer->cancel() then scheduler->stop(). But if the timer fires
and the callback creates a new timer between cancel() and stop() (or
just after stop()), the new timer keeps io_ctx.run() alive
indefinitely since the work guard has already been released but there
is still an outstanding async operation.
This race is more likely to trigger now because the sender-side peek
timeout -- this and #13248 -- changes recovery timing during the cluster
wiggle, causing more transient transaction errors (connection_failed,
commit_unknown_result, not_committed). Workloads take longer to drain
their in-flight transactions after receiving STOP, giving the 5-second
stats timer more opportunities to fire and race with shutdown.
Fix: add an atomic flag checked by the timer callback before
re-scheduling. Set the flag before cancelling the timer to guarantee
no new timer is created after shutdown begins.
* Fix dead TLog hang with sender-side peek timeout
BEHAVIOR CHANGE: Non-parallel TLog peek requests now have a sender-side
timeout (PEEK_REPLY_TIMEOUT knob, default 30s). Previously, a peek
would wait indefinitely for a reply or an interface change. Now, if no
reply arrives within PEEK_REPLY_TIMEOUT seconds, the peek is re-sent.
Set PEEK_REPLY_TIMEOUT to 0 to disable and restore the old wait-forever
behavior.
Problem: when a TLog is killed with data destruction (KillType=6) and
reboots at the same address, FlowTransport silently drops requests to
old endpoint tokens. The sender (LogRouter/SS/consistency checker) waits
forever because:
- The connection is healthy (same address, process is alive)
- No ENDPOINT_NOT_FOUND is sent (non-stream unknown endpoints are
silently discarded by scanPackets in FlowTransport.cpp)
- The interface does not change (same address)
This manifests as simulation timeouts (5400s) in multi-region configs
with single replication after datacenter kills. Seen in recent gcc
nightlies (tests: AtomicOps.toml seed 309473476, DataLossRecovery.toml
seed 3721859193, MaxGrvQueueDelay.toml seed 3023304950,
TxnStateStoreCycleTest.toml seed 2843566457) and in local reproduction.
Fix: in LogSystemPeekCursor::serverPeekGetMoreImpl, the existing
race(peekReply, interf->onChange()) now includes a third branch:
delay(PEEK_REPLY_TIMEOUT). When the timeout fires, the loop continues
(re-sends the peek request) rather than throwing.
Why retry (continue) instead of throw: throwing timed_out() propagated
up and killed the storage server (SSUpdateError/StorageServerFailed),
causing false failures in clog tests (ClogRemoteTLog, GcGenerations)
where TLogs are intentionally unreachable for extended periods.
Why this approach over receiver-side ENDPOINT_NOT_FOUND: a receiver-
side fix was attempted (sending ENDPOINT_NOT_FOUND for unknown tokens)
but abandoned because the receiver cannot distinguish "dead endpoint"
from "clogged endpoint" -- both appear as packets to an unknown token.
This caused widespread false failures (71/6760 runs, 1%) in tests
that inject network clogging (ClogRemoteTLog, DcLag, GcGenerations).
How the fix resolves the dead TLog hang:
1. Peek request sent to dead TLog -> silently dropped
2. 30s timeout fires -> peek re-sent (new request, same dead endpoint)
3. Also silently dropped -> timeout fires again
4. Meanwhile, cluster controller detects the dead TLog
5. Interface updated -> interf->onChange() fires -> cursor gets new
TLog endpoint -> recovery/consistency check proceeds
Tested: 100,000 Joshua correctness runs, 99,997 passed. 3 failures
all unrelated (RocksDB file lock bug, TracedTooManyLines in long test).
* Formatted
* Address review feedback
* Formatting
Signed-off-by: spraza
Signed-off-by: ploxiln
Signed-off-by: sbodagala
Trevor Clinkenbeard