* feat(tent): improve RailMonitor recovery with exponential backoff and
configurable parameters
- Introduce exponential backoff cooldown with a configurable upper bound
(kMaxCooldown)
- Support dynamic configuration of error threshold, error window, and cooldown
via Config
- Call markRecovered on successful transfer completion to un-pause rails
promptly
- Reset error count and cooldown on recovery to prevent accumulated doubling
- Cache target_machine_id in RdmaSlice to avoid segment lookup on hot path
- Add unit tests for recovery behavior, cooldown reset, and best-device mapping
Signed-off-by: staryxchen <staryxchen@tencent.com>
* test(engine): add end-to-end failover tests and test-only transport swap hook
- Add`swapTransportForTest`method to`TransferEngineImpl`
for test-only transport injection.
- Introduce`engine_failover_e2e_test.cpp`
with scenarios: status corruption failover, budget exhaustion, mixed faults,
per-task independence, and boundary conditions for
`max_failover_attempts`.
- Register new test target`tent_engine_failover_e2e_test`in CMakeLists.
Signed-off-by: staryxchen <staryxchen@tencent.com>
* docs(tent): add failover design doc
Describe TENT's two-layer failure handling: cross-transport failover in
TransferEngineImpl and per-rail cooldown recovery in RailMonitor.
Covers fault model, state machines (with code refs), config knobs,
observability, testing, and known gaps (submit-stage failover, cooldown
reset on recovery, no CI coverage).
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(tent): cache RailMonitor pointer on slice to avoid hot-path string lookup
Each RdmaSlice carried the target machine_id as std::string and
asyncPollCq did a hash+strcmp lookup on worker.rails per completion.
Resolve the RailMonitor once during generatePostPath and stash the
pointer on the slice; the completion path becomes a single deref.
WorkerContext::rails now stores values via unique_ptr so a rehash
only moves the pointer slot and does not invalidate pointers already
held by in-flight slices.
disableEndpoint no longer needs a segment lookup to mark the rail
failed either -- it uses the cached pointer.
Addresses gemini-code-assist review on PR #1984.
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
* feat(metrics): add TENT metrics system with HTTP server and Prometheus
integration
- Add comprehensive metrics system based on yalantinglibs for monitoring data
transfer performance
- Implement HTTP server with endpoints for Prometheus, JSON, and human-readable
metrics
- Add compile-time and runtime performance optimization with zero-overhead when
disabled
- Integrate metrics into TransferEngine with automatic latency tracking
- Add configuration loader supporting config files and environment variables
- Include example application demonstrating metrics usage
- Add documentation for metrics system configuration and usage
Signed-off-by: staryxchen <staryxchen@tencent.com>
* Update docs/source/design/tent/metrics.md
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* refactor(metrics): simplify config loading with explicit priority
- Replace indirect environment config loading with direct parsing
- Implement clear priority: file config > environment variables > defaults
- Add validation for environment variable values
- Remove redundant default value comparisons
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(transfer_engine): extract metrics recording logic into dedicated method
- Add recordTaskCompletionMetrics method to TransferEngineImpl class
- Replace duplicate metrics recording code in getTransferStatus methods with
calls to new method
- Centralize task completion metrics logic for better maintainability
Signed-off-by: staryxchen <staryxchen@tencent.com>
* build(metrics): improve yalantinglibs dependency handling
- Change warning to fatal error when TENT_METRICS_ENABLED is ON but
yalantinglibs is missing
- Provide clearer warning message when metrics are disabled
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(metrics): replace manual JSON construction with nlohmann/json library
- Use nlohmann/json for cleaner and more maintainable JSON serialization
- Remove manual string stream manipulation and formatting
- Improve code readability and reduce error-prone manual concatenation
Signed-off-by: staryxchen <staryxchen@tencent.com>
* style: reformat code with clang-format
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(config): centralize parsing utilities in ConfigHelper
- Move parsing functions from MetricsConfigLoader to ConfigHelper
- Add applyEnvironmentOverrides method to reduce code duplication
- Update includes and comments to reflect new structure
Signed-off-by: staryxchen <staryxchen@tencent.com>
* test: add unit tests for metrics config loader and reorganize test structure
- Move examples directory to tests directory in CMakeLists.txt
- Add comprehensive unit tests for MetricsConfigLoader functionality
- Include tests for config parsing, environment variable loading, and validation
- Rename and relocate tent_metrics_example.cpp to tests directory
Signed-off-by: staryxchen <staryxchen@tencent.com>
* style: reformat code lines for better readability
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix(build): remove redundant Asio dependency from metrics CMakeLists
- Remove Asio dependency search and linking as yalantinglibs bundles it
internally
- Add clarifying comment about bundled Asio in yalantinglibs
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Stary