* fix: correct TP zero-copy put semantics
* fix: apply clang-format to store binding
* fix: default batch TP-from results to errors
* fix: return success for batch TP put-from
Ensure batch_put_tensor_with_tp_impl reports zero on fully successful shard writes so TP zero-copy Python tests don't fail with false INVALID_PARAMS results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* tests: align TP zero-copy benchmark with get API signature
Keep the TP zero-copy benchmark focused on put-from semantics by removing an unsupported split_dim argument from batch_get_tensor_with_tp_into.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* trigger ci
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
* [Store] Add hard pin mechanism for eviction-protected objects
Objects created with ReplicateConfig.with_hard_pin=true are never
evicted by the eviction policy, providing guaranteed persistence
for model weights in RL and model management workloads.
Changes:
- ReplicateConfig: add with_hard_pin field (default false)
- ObjectMetadata: add hard_pinned boolean, set at creation via PutStart
- BatchEvict: skip hard-pinned objects in all eviction passes
- Serialization: persist hard_pinned in snapshots (backward compatible
with old format that lacks the field)
- Tests: verify hard-pinned objects survive eviction, coexist with
soft pin, and can still be explicitly removed
When building with USE_EFA=ON, auto_discover is disabled to prevent
RDMA transport installation (QP creation fails on EFA devices). This
means TCP transport is also not installed automatically. Add explicit
TCP transport installation for non-EFA protocols in the EFA build path.
Documentation changes:
- build.md: Add USE_EFA option and clarify USE_CUDA default/purpose
- supported-protocols.md: Add EFA as a supported protocol
- efa_transport.md: Add USE_CUDA=ON to build command, document GPU
memory requirement
Co-authored-by: whn09 <whn09@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
* add indexer api design doc
* remove some api & add explain info for tiered storage
* rename tired storage level name
* add overview architecture
* define the input of indexer kvevent
* add example for indexer api output
* fix some issue from code reviews
* add explain for different medium
* delete the confusing note
* add more Introduction for indexer output value
* change some code based on the code review
* use medium name directly
* fix some format problem
* support multi-tenant & cache salt and rename engine_name
* [TE] Add AWS EFA transport using libfabric
Add EfaTransport as a new transport backend for AWS Elastic Fabric
Adapter (EFA) devices. EFA exposes RDMA-like NICs but does not support
the full ibverbs QP API, so this transport uses libfabric's FI_EP_RDM
(Reliable Datagram Message) endpoint type instead.
Architecture (per EFA device):
EfaTransport → EfaContext → EfaEndPoint
- EfaContext: owns fabric/domain/AV/CQ/MR resources
- EfaEndPoint: one RDM endpoint per peer, with address-vector addressing
- Dedicated CQ poller thread per device for responsive completion draining
Key design decisions:
- FI_THREAD_SAFE requested from provider; per-endpoint spinlock on
fi_write as safety net for concurrent submission threads
- Atomic CAS reservation of CQ and WR capacity before posting fi_write
to prevent CQ overflow under high concurrency
- CQ error path drains all queued errors (fi_cq_readerr loop) before
returning, per libfabric semantics
- Retry-with-backoff on CQ/WR full instead of immediate slice failure
- Thread-safe endpoint creation via atomic getOrInsert to prevent
duplicate endpoints for the same peer
- Handshake exchanges EFA endpoint addresses via dedicated efa_addr
field in HandShakeDesc
Build: cmake -DUSE_EFA=ON (requires libfabric from AWS EFA installer)
Tested on p6-b200.48xlarge (8 EFA devices, 8×400 Gbps): 59.72 GB/s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [TE] Add EFA unit tests and bench tool support
Add efa_transport_test with 5 test cases:
- InstallTransport: verify EFA transport installation
- LoopbackWrite: basic loopback write operation
- WriteAndRead: write then read with data integrity check
- MultiWrite: batch write (16 requests)
- StressMultipleBatches: stress test (20 batches × 8 requests)
Add --protocol=efa support to transfer_engine_bench with manual
topology discovery (EFA needs explicit discover() since
TransferEngine(false) skips auto-discovery).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Docs] Add EFA transport documentation
Add comprehensive EFA transport documentation covering:
- Prerequisites and build instructions
- Usage with vLLM (prefill/decode disaggregation)
- Unit test descriptions and environment variables
- Benchmark results on p6-b200.48xlarge: 59.72 GB/s (EFA) vs
9.5 GB/s (TCP iperf3) vs 0.11 GB/s (Mooncake TCP transport)
- EFA vs RoCE RDMA comparison table
- Thread safety design notes
- Troubleshooting guide
Add EfaTransport to the transfer-engine index toctree and supported
transport lists.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [TE] Address PR review: ifdef EFA fields, use find_package for libfabric
- Wrap efa_addr in HandShakeDesc with #ifdef USE_EFA in transfer_metadata.h
- Wrap efa_addr serialization/deserialization with #ifdef USE_EFA in transfer_metadata.cpp
- Replace hardcoded /opt/amazon/efa paths with find_path/find_library in common.cmake
- Remove redundant hardcoded EFA paths from all CMakeLists.txt files
- Fix git clone URL in efa-transport.md to use official kvcache-ai/Mooncake repo
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Docs] Update EFA benchmark results with tuned parameters (170 GB/s)
Update benchmark documentation with comprehensive parameter tuning results
from cross-machine testing on p6-b200.48xlarge instances. Key finding:
MC_SLICE_SIZE=262144 nearly doubles EFA throughput from ~70 to ~170 GB/s,
reaching 88% of RoCE RDMA performance.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix clang-format violation in transfer_metadata.h
Remove extra space before comment on efa_addr field to satisfy
clang-format-20 style check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Docs] Add EFA latency benchmark script, rename efa doc to underscore
- Add efa_latency_bench.py: automated benchmark script that measures
EFA throughput for tuned/default configs via SSH and plots
Latency vs Cache Size chart
- Add efa_latency_bench.png: benchmark results chart
- Rename efa-transport.md -> efa_transport.md to match naming
convention of other transport docs (ascend_transport.md, etc.)
- Update toctree reference in index.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-25-79.us-east-2.compute.internal>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-22-204.us-east-2.compute.internal>
* [Store]: add task executor feature with unit and executor test
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: add some optimizations to task executor
Refactor the task_executor into the client_service and add a
task structure on the client side. Additionally, remove the
existence check in the execute function;
only retrying should be performed if replica allocation fails.
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: get source replica from copyStart or moveStart api
* [Store]: call move or copy end if the target replica already exist to complete the replication task
* [Store]: use the max_retry_attempts in master side
* [Store]: add client integration test and set default max_retry_attempts to 10
* [Doc] update task api introduction
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store] Set copy and move as private methods
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Doc]: change the default task max_retry_attempts to 10
* [Doc]: fix some description error
* [Store]: allocate the buffer size to be a multiple of 16MB
* [Store]: validate the replica is in local and directly construct slices from replica buffer address instead of copy the data to local buffer
* [Store]: change the validate logic to directly use transfer engine endpoint or local_hostname_.
* [Store] add e2e ci test for copy and move api
* [Store] refactor the client move and copy function
* [Store] fix the e2e test
* [Store] remove unused code
* [Store] add source field when build replica copy payload in the task_manager_test
* [Store] rename back to snake case for split_into_slices function and also remove hard code for client poll count
* [Store] revert mis deleted field when resolve conflicts
* [Store] change test to validate the real behaviour
* [Store] change the default task fetch size to 16
* [Doc]: change the replica copy/move sequence diagram
* [Store] add new split_into_slice method
* [Store] change the real client to use split_to_slice with buffer handle parameters
---------
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
Co-authored-by: Vincent Gao <vincentbo@linux.alibaba.com>
Add MC_HANDSHAKE_MAX_LENGTH environment variable to configure the maximum
handshake message length in P2P mode.
## Problem
When using P2P handshake mode with a single RDMA instance that registers
many memory buffers (>10,000), the serialized segment metadata JSON can
exceed the hardcoded 1MB limit, causing handshake failures with error:
"readString: too large length from socket: <length>"
Each registered buffer adds ~96 bytes to the JSON payload:
- 1,000 buffers ≈ 94KB
- 5,000 buffers ≈ 469KB
- 10,000 buffers ≈ 938KB (near 1MB limit)
- 15,000 buffers ≈ 1.37MB (exceeds limit)
## Solution
- Add getHandshakeMaxLength() function that reads MC_HANDSHAKE_MAX_LENGTH
- Value is in bytes, valid range: 1MB to 128MB
- Default remains 1MB (1048576 bytes) for backward compatibility
- Logs custom value when set, warns on invalid values
## Usage
```bash
# Set to 4MB to support ~40,000 buffers
export MC_HANDSHAKE_MAX_LENGTH=4194304
```
* Add early mem backend detection method in NVLINK_allocator
Add early detection method for sglang NVLINK_allocator to avoid CuMemCreate
Use enumerate type to indicate mem backend type
format check use pre-commit
* Change enumerate type in allocator.py for different mem backend
* Isolate nvlink intraNode transport from nvlink_transport and modify corresponding transfer_engine_bench
* IntraNode transport isolation to be compatible with SGlang
* Modify the code style using pre-commit check
* isolate intraNode nvlink from previous nvlink_transport
* [TE] Revert to previous nvlink_transport
* Delete extra log and refine the code format
* Discard revert due to new related PR has been submitted
* Change the Code format to align with main branch
* Change to clang-format
* Modify transfer_engine_bench to be compatible with protocol --nvlink_intra
* Delete useless function in intranode_nvlink.cpp
* Used for rerun CI
* Modify the intraNode isolation to be compatible with transfer_bench and transfer_engine_impl
* isolate intraNode header file from mnnvl.h
* Delete specific instaince type for intraNode
---------
Co-authored-by: 百麒 <yaozhong.lyz@alibaba-inc.com>
* feat(metrics): add TENT metrics system with HTTP server and Prometheus
integration
- Add comprehensive metrics system based on yalantinglibs for monitoring data
transfer performance
- Implement HTTP server with endpoints for Prometheus, JSON, and human-readable
metrics
- Add compile-time and runtime performance optimization with zero-overhead when
disabled
- Integrate metrics into TransferEngine with automatic latency tracking
- Add configuration loader supporting config files and environment variables
- Include example application demonstrating metrics usage
- Add documentation for metrics system configuration and usage
Signed-off-by: staryxchen <staryxchen@tencent.com>
* Update docs/source/design/tent/metrics.md
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* refactor(metrics): simplify config loading with explicit priority
- Replace indirect environment config loading with direct parsing
- Implement clear priority: file config > environment variables > defaults
- Add validation for environment variable values
- Remove redundant default value comparisons
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(transfer_engine): extract metrics recording logic into dedicated method
- Add recordTaskCompletionMetrics method to TransferEngineImpl class
- Replace duplicate metrics recording code in getTransferStatus methods with
calls to new method
- Centralize task completion metrics logic for better maintainability
Signed-off-by: staryxchen <staryxchen@tencent.com>
* build(metrics): improve yalantinglibs dependency handling
- Change warning to fatal error when TENT_METRICS_ENABLED is ON but
yalantinglibs is missing
- Provide clearer warning message when metrics are disabled
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(metrics): replace manual JSON construction with nlohmann/json library
- Use nlohmann/json for cleaner and more maintainable JSON serialization
- Remove manual string stream manipulation and formatting
- Improve code readability and reduce error-prone manual concatenation
Signed-off-by: staryxchen <staryxchen@tencent.com>
* style: reformat code with clang-format
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(config): centralize parsing utilities in ConfigHelper
- Move parsing functions from MetricsConfigLoader to ConfigHelper
- Add applyEnvironmentOverrides method to reduce code duplication
- Update includes and comments to reflect new structure
Signed-off-by: staryxchen <staryxchen@tencent.com>
* test: add unit tests for metrics config loader and reorganize test structure
- Move examples directory to tests directory in CMakeLists.txt
- Add comprehensive unit tests for MetricsConfigLoader functionality
- Include tests for config parsing, environment variable loading, and validation
- Rename and relocate tent_metrics_example.cpp to tests directory
Signed-off-by: staryxchen <staryxchen@tencent.com>
* style: reformat code lines for better readability
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix(build): remove redundant Asio dependency from metrics CMakeLists
- Remove Asio dependency search and linking as yalantinglibs bundles it
internally
- Add clarifying comment about bundled Asio in yalantinglibs
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* feat(metrics): add task completion latency tracking and detailed metrics reporting
- Add histogram metric for tracking transfer task completion latency with
configurable buckets
- Record task start time in submitTransfer and calculate latency upon completion
- Enhance metrics reporting thread to include latency distribution statistics
- Add skip_metrics parameter to getBatchTransferStatus to avoid double counting
- Update task structure to include start_time field for latency calculation
Signed-off-by: staryxchen <staryxchen@tencent.com>
* feat: add metrics support for submitTransferWithNotify
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(metrics): improve metrics recording logic with early returns
- Replace nested if statements with early returns using goto for cleaner flow
- Add boundary checks for task_id and validate start_time before recording
- Only record metrics for COMPLETED status and reset start_time to prevent
duplicates
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(metrics): simplify bucket count tracking logic
- Remove conditional checks for vector bounds by ensuring prev_bucket_counts_
size matches bucket_counts
- Consolidate previous snapshot update into a single assignment
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix(metrics): conditionally enable metrics collection
- Add metrics_enabled_ flag check before collecting metrics
- Wrap metrics collection logic with flag condition to avoid unnecessary
operations
- Ensure metrics are only recorded when explicitly enabled
Signed-off-by: staryxchen <staryxchen@tencent.com>
* docs: clarify Transfer Engine metrics limitation with TENT
- Add note about MC_TE_METRIC not being supported when using Transfer Engine
TENT in deployment guide
- Add same note to Python API reference for MC_TE_METRIC environment variable
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
fatSheep