* feat(tent): improve RailMonitor recovery with exponential backoff and
configurable parameters
- Introduce exponential backoff cooldown with a configurable upper bound
(kMaxCooldown)
- Support dynamic configuration of error threshold, error window, and cooldown
via Config
- Call markRecovered on successful transfer completion to un-pause rails
promptly
- Reset error count and cooldown on recovery to prevent accumulated doubling
- Cache target_machine_id in RdmaSlice to avoid segment lookup on hot path
- Add unit tests for recovery behavior, cooldown reset, and best-device mapping
Signed-off-by: staryxchen <staryxchen@tencent.com>
* test(engine): add end-to-end failover tests and test-only transport swap hook
- Add`swapTransportForTest`method to`TransferEngineImpl`
for test-only transport injection.
- Introduce`engine_failover_e2e_test.cpp`
with scenarios: status corruption failover, budget exhaustion, mixed faults,
per-task independence, and boundary conditions for
`max_failover_attempts`.
- Register new test target`tent_engine_failover_e2e_test`in CMakeLists.
Signed-off-by: staryxchen <staryxchen@tencent.com>
* docs(tent): add failover design doc
Describe TENT's two-layer failure handling: cross-transport failover in
TransferEngineImpl and per-rail cooldown recovery in RailMonitor.
Covers fault model, state machines (with code refs), config knobs,
observability, testing, and known gaps (submit-stage failover, cooldown
reset on recovery, no CI coverage).
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(tent): cache RailMonitor pointer on slice to avoid hot-path string lookup
Each RdmaSlice carried the target machine_id as std::string and
asyncPollCq did a hash+strcmp lookup on worker.rails per completion.
Resolve the RailMonitor once during generatePostPath and stash the
pointer on the slice; the completion path becomes a single deref.
WorkerContext::rails now stores values via unique_ptr so a rehash
only moves the pointer slot and does not invalidate pointers already
held by in-flight slices.
disableEndpoint no longer needs a segment lookup to mark the rail
failed either -- it uses the cached pointer.
Addresses gemini-code-assist review on PR #1984.
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
5ef889dd11