Immediately completing the batch on transfer failures will result in
use-after-free problems because other tasks in the batch might be still
in processing.
Signed-off-by: Chen Jinlong <chenjinlong.cjl@alibaba-inc.com>
When MC_STORE_MEMCPY is not explicitly set, auto-detect based on
installed transports: enable memcpy only when TCP is the sole
transport (no RDMA, NVLink, etc.), since TCP loopback is less
efficient than direct memcpy for same-host transfers. In RDMA
environments the default remains disabled, as RDMA is more
resource-efficient.
Add TransferEngine::isTcpOnly() API that checks whether TCP is the
only installed transport via MultiTransport::transport_map_. This
is future-proof: any new transport registered via installTransport()
is automatically accounted for without maintaining a protocol list.
TENT path returns false unconditionally since TENT already rejects
TCP loopback without MC_STORE_MEMCPY.
Signed-off-by: Tianchen Ding <dtcccc@linux.alibaba.com>
* feat(efa): auto-split large MR registrations exceeding max_mr_size
Buffers larger than the EFA device's max_mr_size are now transparently
split into chunks, each registered as a separate MR. This fixes the
silent truncation bug where only the first max_mr_size bytes were
registered, causing transfers to unregistered regions to fail at runtime.
Key changes:
- Query EFA device max_mr_size via ibverbs during init (libfabric does
not expose this) and clamp globalConfig accordingly
- Auto-split buffers > (max_mr_size - 1GB) into chunks in
registerLocalMemoryInternal, each with its own BufferDesc metadata
- Track chunk mappings for proper cleanup in unregisterLocalMemory
- Change lkey()/rkey() from exact-match to range lookup (matching
mrDesc() pattern) so key lookups work for any address within a chunk
- Replace silent truncation in EfaContext with a hard error as safety net
- Remove preTouchMemory truncation to touch the full buffer
Tested on P5EN (16 EFA NIC, max_mr_size=192GB): 191GB single-chunk
registration succeeds. Auto-split triggers correctly for larger buffers
(200GB splits into 191GB + 9GB). Total registerable size per buffer is
bounded by system pinned_vm limits (~191GB on 16-NIC P5EN).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): per-NIC partition for large buffer MR registration
Each EFA NIC can only register up to max_mr_size total. The previous
auto-split approach registered every chunk on ALL NICs, hitting the
per-NIC limit for buffers > max_mr_size. This change assigns each
chunk to a disjoint subset of NICs, enabling registration of buffers
up to max_mr_size × num_NICs (e.g. ~1.5TB on P5EN with 16 NICs).
Key changes:
- chunk_limit = max_mr_size / 2 (was max_mr_size - 1GB) for headroom
- NIC assignment: chunks distributed evenly across available NICs
- selectDevice(): skips NICs with rkey=0 (unassigned for that chunk)
- Striping path: filters by lkey!=0 to avoid unregistered NICs
- Unregister: only deregisters from assigned NICs per chunk
- ChunkRegistration struct tracks per-chunk NIC assignments
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): PTE-aware auto-split replaces conservative max_mr_size/2 threshold
The previous chunk_limit of max_mr_size/2 (~96GB on P5EN) caused unnecessary
buffer splitting even when hugepages were available. This detects the actual
backing page size via /proc/self/smaps and computes the PTE-based limit:
- 4KB pages: 22M PTEs × 4KB = 88GB (genuine hardware constraint)
- 2MB hugepages: 22M PTEs × 2MB = 44TB (effectively max_mr_size)
Verified on P5EN (H200, 16 EFA): 100GB pool with hugepages no longer splits,
restoring full 16-NIC throughput (108 GB/s vs 46 GB/s with the old threshold).
MR registration time dropped from 302s to 1.6s.
Adds MC_EFA_MAX_PTE_ENTRIES env var for override (default 22M).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add KV cache prefix transfer benchmark script
Python benchmark to measure EFA transfer performance for LLM prefix
cache hit scenarios. Tests different pool sizes (10GB-500GB) and prefix
lengths (4K-32K tokens) to evaluate per-NIC partition impact on
transfer latency and throughput.
Default KV bytes/token matches GLM-5.1 (754B MoE, MLA attention):
(kv_lora_rank=512 + qk_rope_head_dim=64) * 2 * 78 layers = 88KB/token
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(bench): add connection warmup before benchmark loop
The first prefix size's measurements were skewed by EFA connection
establishment (openSegment, endpoint creation). Add 3 small transfers
before entering the benchmark loop to warm up the connection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(bench): per-offset warmup and p50 throughput reporting
Warmup now exercises all offsets the benchmark will measure, eliminating
first-access TLB/page-fault outliers (4K token p99 dropped from 80ms to
3.5ms). Throughput is reported from p50 latency instead of avg for more
stable numbers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(bench): add --threads option for concurrent prefix transfer
Adds optional multi-threaded transfer support. Each thread transfers a
chunk of the prefix in parallel via separate transfer_sync_read calls.
Default is 1 (single transfer, same as before). Testing shows threads=2
matches single-thread throughput (~108 GB/s), while higher values add
scheduling overhead with no benefit since EFA already stripes internally.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): full NIC coverage for multi-chunk MR registration
When a buffer exceeds max_mr_size and must be split into multiple chunks,
register every chunk on ALL NICs instead of disjoint per-NIC partition,
as long as total PTE usage per NIC fits within the PTE budget.
With hugepages (2MB), 500GB buffer uses only 250K PTE/NIC (budget: 24M),
so all 16 NICs cover every address. Falls back to disjoint partition when
PTE budget is exceeded (e.g. 4KB pages with large buffers).
500GB pool throughput: 35 GB/s (disjoint, 5-6 NIC) → 108 GB/s (full, 16 NIC).
Registration time unchanged (~9.5s) due to parallel MR registration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(bench): add batch memory registration benchmark
Test script for registering multiple independent memory blocks
(e.g. multi-tenant KV cache pools). Supports both per-block
register_memory and batch_register_memory APIs. Target mode
allocates and registers N blocks; initiator mode transfers
data and measures throughput.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply clang-format to EFA transport files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): expose discoverTopology C API and Rust bindings
Add discoverTopology() to the C API and discover_topology()/install_transport()
to the Rust bindings, enabling EFA transport initialization from C/Rust callers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(efa): add C API test for discoverTopology and EFA transport
Verify that the new discoverTopology() C API correctly populates the
device list, enabling installTransport("efa") and memory registration
via the pure C interface (used by Rust/Go bindings).
All 4 tests passed on p5en.48xlarge (16 EFA NICs).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): O(log n) MR lookup and chunk registration rollback
- Replace unordered_map with std::map for mr_map_ and use upper_bound
for O(log n) range lookups in rkey/lkey/mrDesc instead of O(n) scan
- Add rollbackChunks lambda to unregister already-registered chunks on
failure in registerLocalMemoryInternal, preventing MR leaks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply clang-format to efa_c_api_test.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: rename thr_tag to thread_tag to pass typos spell check
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): round-robin multi-chunk MR and large MR registration test
When a buffer splits into more chunks than available NICs, round-robin
assign chunks across NICs with per-NIC PTE budget validation, instead
of hard-failing with "Buffer requires N chunks but only M NICs".
Add efa_single_nic_large_mr_test: tests single-NIC and all-NIC large
MR registration with hugepages. Supports --chunk_gb for multi-buffer
mode (e.g. 200×2GB).
Verified on P5EN (16 EFA NIC, 2MB hugepages):
- 1 NIC × 200×2GB (400GB): 53.7s
- 16 NICs × 200×2GB (400GB): 64.8s
- 1 NIC × 200GB single buffer: 2.7s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add cross-node transfer test for multi-buffer MR registration
Tests 200x2GB buffer registration on all NICs with actual data transfer
between two P5EN nodes. Supports target/initiator modes with single-read
and multi-block batch benchmarks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): avoid CUDA context leak when built without GPU support
libfabric 2.4's EFA provider dlopens libcudart/libcuda during
fi_getinfo/fi_domain to detect HMEM support, even though the caller
never touches GPU memory. This creates a CUDA primary context on
GPU 0 and permanently holds ~616 MiB of device memory.
When Mooncake is built with USE_CUDA=OFF (and USE_HIP=OFF), set
FI_HMEM=system before fi_getinfo and drop FI_MR_HMEM from the domain
hints so the provider skips GPU hmem initialization entirely.
* feat(efa): multi-thread initiator and wildcard location for 16-NIC coverage
Register target buffers with wildcard location "*" instead of "cpu:0"
so initiator-side remote NIC selection distributes evenly across all 16
NICs (both NUMA nodes). With "cpu:0", selectDevice only picked NUMA-0's
8 NICs, leaving NUMA-1 idle — throughput capped at ~107 GB/s instead of
~149 GB/s on P5EN (16×EFA 200Gbps).
Also convert the initiator from single-threaded to multi-threaded
(--threads flag) and print the actual P2P handshake address.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add p6-b300 bandwidth results (752 GB/s GPU, 230 GB/s CPU)
- GPU-to-GPU peak 752 GB/s write / 713 GB/s read at ~94% line rate
(16×400 Gbps = 800 GB/s theoretical)
- CPU-to-CPU peak 230 GB/s, DRAM-limited on Xeon 8559C
(NUMA-0 NICs 90 Gbps vs NUMA-1 NICs 53 Gbps)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(rust): customer_pattern sample for per-NUMA multi-MR registration
Adds a Rust binary mirroring the customer's EFA usage pattern: target
registers many MRs per NUMA (e.g. 16 NICs x N x 10GB via cpu:<numa>
locations); initiator reads/writes a specific (numa, buffer_index)
tuple, using --source-numa to pick which local NIC set is exercised.
Also makes rust/build.rs robust against non-default transfer_engine
build configurations: etcd-cpp-api is opt-in via MOONCAKE_WITH_ETCD=1,
CUDA linking is opt-in via MOONCAKE_WITH_CUDA=1 with CUDART_LIB_DIR
override, and libbase.a + libasio.so + libfabric paths are picked up
whether the repo uses the standalone or top-level CMake layout.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(rust): drop customer_pattern sample from upstream
Revert Cargo.toml [[bin]] additions and remove the bench demo source so
the public tree no longer ships it. Keep it gitignored locally so it can
be iterated on without accidental re-adds.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(efa): eager endpoint warmup for segments to remove first-batch stall
libfabric FI_EP_RDM endpoints resolve peer addresses lazily — the first
submitTransfer() to a new segment serializes fi_av_insert + handshake over
every (local_ctx, peer_nic) pair. On 16-NIC instances this is a ~4 s stall
on the first 100 × 0.5 MB batch (measured on p6-B300).
Add EfaTransport::warmupSegment(name) that pre-connects all pairs
concurrently via std::async, plus C wrapper warmupEfaSegment() (guarded by
USE_EFA) and Rust binding TransferEngine::warmup_efa_segment(). Idempotent,
safe to re-run after metadata changes. RDMA/TCP paths untouched.
Measured on p6-B300 (16 × 16 endpoints, dual-NUMA initiator):
first-batch: 4043 ms -> 13.5 ms (~300x)
steady-state: 141 GB/s -> 230 GB/s
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(efa): enable auto-split when max_mr_size is not configured
When MC_EFA_MAX_MR_SIZE is unset, max_mr was 0 and chunk_limit collapsed
to 0, bypassing the PTE-aware split entirely. Large 4KB-paged buffers
then hit the per-NIC PTE ceiling at registration time.
Fall back to pte_limit so splitting kicks in based on the PTE budget
alone when no explicit max_mr_size is provided.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* style: apply clang-format to EFA transport/tests and PEP8 to kvcache bench
Pure formatting changes to satisfy CI format hook (clang-format-20) and
address Gemini review comment on kvcache_prefix_bench.py indentation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-13-185.us-east-2.compute.internal>
Co-authored-by: whn09 <whn09@github.com>
* feat(tent): add FaultProxyTransport for fault injection testing
Introduce a decorator Transport that wraps any real Transport and
injects configurable faults (submit failures, status corruption,
artificial latency) according to a FaultPolicy. This enables
integration testing of the failover state machine without hardware.
New files:
- fault_proxy_transport.h: header-only FaultProxyTransport + FaultPolicy
- fault_proxy_test.cpp: 8 GTest cases covering unit, failover, and
policy mutation scenarios
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix(tent): fix off-by-one in ExhaustAllTransports test
The test had 3 failover attempts matching kMaxAttempts=3, so
failover_count == kMaxAttempts (not greater). Add a 4th attempt
to properly exceed the limit, matching resubmitTransferTask logic
which uses `++count > max` (strictly greater).
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor: format code style in fault proxy transport
- Adjust line breaks and indentation for consistent formatting
- No functional changes, only code style improvements
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor: use thread-safe random number generation in fault proxy transport
- Replace instance RNG with thread-local static method for thread safety
- Simplify fault injection logic and remove redundant atomic operations
Signed-off-by: staryxchen <staryxchen@tencent.com>
* feat(transport): add null check and override memory management methods in
FaultProxyTransport
- Add assert to ensure real transport is not null in constructor
- Implement override methods for addMemoryBuffer, allocateLocalMemory,
freeLocalMemory, and warmupMemory to delegate to real transport
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
PutToLocalFile and FileStorage::OffloadObjects crash with SIGSEGV when
slice.ptr points to GPU device memory, because CPU memcpy cannot access
GPU virtual addresses. The RDMA memory-replica path is unaffected.
Add synchronous Device-to-Host staging via PinnedBufferPool before data
reaches the disk-write paths:
- New gpu_staging_utils.h: shared IsDevicePointer/CopyDeviceToHost/
SetDevice helpers with cross-platform support (CUDA/HIP/MUSA/MACA/
Ascend CANN)
- New PinnedBufferPool: thread-safe pinned host memory pool with
max capacity limit (default 32) and O(1) swap-pop acquire
- PutToLocalFile: sync D2H on calling thread, PutRevoke on failure
- OffloadObjects: D2H staging before BatchOffload; on per-slice failure
the entire object is skipped to prevent partial/corrupt data
- CMakeLists: auto-detect CUDAToolkit/HIP/Ascend independently of
global USE_CUDA flag, with explicit PRIVATE compile definitions
- CI: add -lcudart to Go test CGO_LDFLAGS when CUDA is present
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: expose batch_replica_clear in Python binding
Add batch_replica_clear(keys, segment_name) to PyClient/RealClient/DummyClient.
Allows explicit deletion of replicas for lease-expired keys.
Also adds Client::getClientId() public getter.
---------
Signed-off-by: hnts03-moreh <geonwoo.choi@moreh.io>
* fix(efa): add fi_read support and endpoint eviction to EFA transport
1. EFA submitPostSend() now branches on slice->opcode to call fi_read
or fi_write, matching RDMA transport behavior. Previously only
fi_write was implemented, breaking remote-read workloads on EFA.
2. EfaEndpointStore now tracks endpoint activity and evicts stale
endpoints when approaching AV capacity. This prevents AV exhaustion
in long-running services communicating with many transient peers.
- Endpoints marked active on access, inactive on set_active(false)
- Configurable inactive timeout (default 5 min)
- evictStale() removes endpoints inactive beyond timeout
- removeDisconnected() cleans up broken connections
- Eviction triggered automatically in getOrInsert() at capacity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* perf(efa): eliminate per-slice overhead with NIC-count-based striping
Replace fixed 64KB slicing with NIC-count-based chunking in EFA
submitTransferTask. For large transfers (>128KB), creates one slice
per active NIC instead of thousands of small slices, dramatically
reducing spinlock, atomic, and allocation overhead.
- Large transfers: stripe across all active NICs (1 chunk per NIC)
- Small transfers: single slice on one NIC, no sub-slicing
- Fallback: per-slice retry for edge cases (unregistered memory)
Example: 240MB transfer on 32 NICs now creates 32 slices vs 3840,
matching NIXL approach of one fi_write per rail per descriptor.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): add FI_MR_HMEM to mr_mode hints for GPU memory registration
EFA provider requires FI_MR_HMEM in domain mr_mode hints to support
heterogeneous (GPU) memory registration via fi_mr_reg. Without this
flag, fi_mr_reg fails with EFAULT (Bad address) when registering
CUDA device memory buffers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style(efa): apply clang-format-20 formatting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): restore lost changes from merge — endpoint eviction, batched WR, docs
Restores 5 commits that were lost during the upstream merge:
- AV entry removal on disconnect to prevent target degradation
- Endpoint reuse for same physical peer (normalizeNicPath)
- Batched WR submission in submitPostSend
- Pre-resolved remote peer info for striped transfers
- Documentation: GPU-to-GPU (313 GB/s), CPU-to-CPU (222 GB/s) benchmarks,
CPU-only build instructions, B300/P5 instance types
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): use dependencies.sh for build deps, add EFA-specific extras
Replace manual package list with dependencies.sh script reference.
Add libgflags-dev as EFA-specific extra dependency.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add tip to use CUDA_VISIBLE_DEVICES for CPU-to-CPU benchmarks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add DLAMI troubleshooting for CUDA env and Python activation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add LIBRARY_PATH for CUDA libs, remove unneeded CUDAToolkit_ROOT
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add block_size=1MB to all benchmark configurations
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): merge clone step into build dependencies section
dependencies.sh already runs git submodule update --init --recursive,
so the separate clone section was redundant. Move git clone into the
Prerequisites section and renumber build steps.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add SGLang usage, Docker troubleshooting, reorder sections
- Move Usage with vLLM/SGLang after benchmark results (was too early)
- Add Usage with SGLang section (EFA patch, env vars, Docker example)
- Add libfabric version mismatch in Docker troubleshooting
- Reorder p5en results: GPU-to-GPU before CPU-to-CPU (consistent with B200)
- Update p5en single-instance CPU results: Write 179 GB/s, Read 185 GB/s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): use GlobalConfig max_ep_per_ctx (65536) as default, round-robin CQ assignment
- Change EfaEndpointStore and EfaContext::construct default max_endpoints
from 256 to 65536, consistent with GlobalConfig.max_ep_per_ctx
- Add round-robin CQ assignment (nextCq) for new endpoints instead of
always using cq_list_[0], distributing load across CQs
- Pass CQ outstanding counter directly to endpoint construct
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style(efa): clang-format, move striping threshold to GlobalConfig
- Run clang-format on all EFA transport files and common.h
- Move MC_EFA_STRIPING_THRESHOLD from inline std::getenv to GlobalConfig
mechanism (loaded centrally in loadGlobalConfig), per reviewer feedback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: add 'wqs' to typos ignore list (EFA API function name)
efadv_query_qp_wqs is a libfabric EFA API symbol, not a typo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add per-transfer latency benchmark script
Automated script that measures single-transfer latency across block sizes
by SSHing into target/initiator nodes and running transfer_engine_bench
with threads=1, batch_size=1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add --threads and --batch_size args to latency bench script
Allow configurable threads/batch_size for multi-thread bandwidth scaling tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add --env flag to pass env vars to remote bench via SSH
Allows setting MC_EFA_STRIPING_THRESHOLD and other env vars on the
remote initiator without manual SSH config.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: fix typos check by adding wqs to extend-words
The extend-ignore-words key is not recognized by typos v1.30.2 used
in CI. Add wqs (EFA API function name efadv_query_qp_wqs) to the
[default.extend-words] section which is the correct format.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: revert formatting-only changes in common.h
Revert pointer/reference style changes (`&`/`*` placement) that were
introduced by clang-format but are unnecessary for this PR. Keep only
the normalizeNicPath() function addition.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): clarify that MC_SLICE_SIZE does not apply to EFA transport
EFA transport uses NIC-count-based striping (since commit 84d7dc5),
not fixed-size slicing. block_size directly determines per fi_write/
fi_read size for transfers below the striping threshold. Update docs
to reflect this and correct outdated tuning advice.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): address review issues - nextCq guard, error path double-free, GPU MR registration
1. nextCq(): add empty guard to prevent division-by-zero when cq_list_ is empty
2. submitTransferTask error paths: replace getSliceCache().deallocate(s) with
s->markFailed() to prevent double-free (TransferTask destructor also deallocates)
and ensure tasks complete with FAILED status instead of hanging
3. registerMemoryRegionInternal(): use fi_mr_regattr() with explicit FI_HMEM_CUDA
iface and device ordinal for GPU memory, since EFA provider fi_mr_reg()
hardcodes iface=FI_HMEM_SYSTEM
4. Make total_bytes updates atomic (__sync_fetch_and_add) for thread safety
Tested: unit tests (5/5), cross-node GPU benchmark (write 250 GB/s, read 269 GB/s)
on p5en.48xlarge.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style(efa): apply clang-format to efa_context.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-8-212.us-west-1.compute.internal>
Co-authored-by: whn09 <whn09@github.com>
* [PG][TENT]: fix hang bug
* [fix]: code format
* Always kWildcardLocation for CPU and make MooncakeBarrierWorkCuda use BackoffWaiter
* CUDAStreamPool for TENT and non-blocking enq_stream for PG.
* Temporary workaround for deprecated getLocalTopology.
* [tent]: update getMachineID to use /proc/sys/kernel/random/boot_id to
verify
* code format
---------
Co-authored-by: caozhanhao <cao2013zh@163.com>
* build(tent): introduce tent_link_group for library linking
- Add tent_link_group interface library to manage tent and related dependencies
- Update CMake targets to use tent_link_group instead of tent for linking
- Implement link group with -Wl,--start-group and -Wl,--end-group to resolve
circular dependencies
Signed-off-by: staryxchen <staryxchen@tencent.com>
* style: reformat CMakeLists files with consistent indentation and spacing
- Apply consistent indentation (2 spaces) across all modified CMakeLists files
- Normalize conditional statement formatting (if/else/endif)
- Improve line wrapping for better readability in long commands
- Standardize target_link_libraries and other CMake command formatting
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
* feat(tent): replace raw RdmaEndPoint* with weak_ptr for lifecycle safety
RdmaSlice::ep_weak_ptr was a raw pointer despite its name, creating a
dangling-pointer risk when endpoints are evicted from the EndpointStore
cache while slices are still in-flight.
Changes:
- RdmaEndPoint now inherits std::enable_shared_from_this (endpoints are
already managed as shared_ptr in FIFOEndpointStore/SIEVEEndpointStore)
- RdmaSlice::ep_weak_ptr changed from RdmaEndPoint* to
std::weak_ptr<RdmaEndPoint>
- submitSlices() assigns via shared_from_this() instead of raw this
- All dereference sites in workers.cpp now call .lock() and gracefully
handle nullptr (endpoint already destroyed) by marking slices FAILED
- Add endpoint_lifecycle_test with 7 test cases verifying weak_ptr
semantics: lock-while-alive, expire-after-release,
shared_from_this, slice access patterns, multi-slice, and reset
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* perf(tent): hoist shared_from_this() out of submitSlices loop
Move the shared_from_this() call before the loop so we create a single
shared_ptr and assign it to each slice's weak_ptr, avoiding N redundant
atomic refcount increment/decrement pairs per batch submission.
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix(tent): decrement inflight_slices when endpoint expired on timeout
When a slice times out and its endpoint weak_ptr has already expired,
the inflight_slices counter was not decremented, causing the worker to
never enter suspension and skewing load balancing decisions.
Add fetch_sub(1) in the null-endpoint timeout path to keep the counter
consistent with the actual number of in-flight slices.
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously, NUMA-aware global segment allocation was gated behind
`!ipc_socket_path_.empty()`, meaning it only activated in standalone
(Dummy+Real Client) mode. In practice, the vast majority of
deployments (mooncake_store_service, sglang/vLLM integration, Python
store.setup()) use RealClient-only mode where ipc_socket_path is
always empty, so NUMA optimization never triggered.
Remove the ipc_socket_path condition so that all RDMA deployments
benefit from NUMA-segmented memory allocation and NUMA-aware NIC
routing via selectDevice.
* feat: enhance memory registration with transport type support
- Add transport type filtering to getSupportedTransports for specific
type requests instead of always returning all transports
- Return early with InvalidArgument when no transport is available
during registerLocalMemory
- Improve same_machine detection in getTransportType to handle
LOCAL_SEGMENT_ID and empty machine_id cases
- Simplify benchmark to use MemoryOptions with default UNSPEC type,
removing unnecessary branching
Signed-off-by: staryxchen <staryxchen@tencent.com>
* Refine TENT same-machine segment lookup
* refactor: improve code formatting for same_machine condition
- Align logical operators for better readability
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
* [TransferEngine] Wire up cross-transport failover with safety limits and observability
The existing resubmitTransferTask() had complete failover logic (increment
xport_priority, resolve next transport, resubmit) but was never called.
This commit activates the failover path and adds production safeguards:
- Wire getTransferStatus() to call resubmitTransferTask() on FAILED tasks
- Add failover_count to TaskInfo with configurable max_failover_attempts (default 3)
- Add transportTypeName() helper and structured LOG(INFO) for failover events
- Add tent_transport_failover_total Prometheus counter metric
- Add unit tests for failover state machine, config loading, and limit checks
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix: prevent failover from overwriting permanent FAILED in batch status
Move the failover attempt before status aggregation so that a
successfully resubmitted task appears as PENDING to the existing
if/else-if chain. Previously the code unconditionally set
overall_status.s = PENDING after a successful resubmit, which could
overwrite a permanent FAILED from another task in the same batch.
Now the `else if (task_status.s != PENDING)` branch naturally skips
PENDING tasks, and only truly-failed tasks set overall_status to
FAILED.
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add ROCm HIP support to the Mooncake Python package
Expose HIP as the Python-facing AMD GPU transport, wire HIP transport into the
build and runtime selection paths, and fix sticky peer-access errors so
repeated connector initialization works reliably. Update wheel packaging and
container validation to cover ROCm HIP usage in vllm-omni.
Signed-off-by: Zejian Wang <zejianwang@sjtu.edu.cn>
* [Bugfix] Decouple HIP transport from NVLink branch
HIP IPC is intra-node only and should coexist with RDMA, not replace it.
Signed-off-by: Zejian Wang <zejianwang.sjtu.edu.cn>
Signed-off-by: Zejian Wang <zejianwang@sjtu.edu.cn>
* Abandon manual transport
Signed-off-by: Zejian Wang <zejianwang.sjtu.edu.cn>
Signed-off-by: Zejian Wang <zejianwang@sjtu.edu.cn>
* style: apply clang-format to changed files
Signed-off-by: Zejian Wang <zejianwang@sjtu.edu.cn>
Made-with: Cursor
Signed-off-by: Zejian Wang <zejianwang@sjtu.edu.cn>
---------
Signed-off-by: Zejian Wang <zejianwang@sjtu.edu.cn>
Signed-off-by: Zejian Wang <zejianwang.sjtu.edu.cn>
Co-authored-by: Zejian Wang <zejianwang@sjtu.edu.cn>
* [TransferEngine] Add retry, async execution, and graceful shutdown for TENT TCP transport
- Replace volatile with std::atomic for thread-safe TcpTask status tracking
- Add configurable exponential backoff retry with interruptible sleep
- Introduce ThreadPool-based async transfer dispatch
- Add graceful shutdown via atomic flag (following RDMA transport pattern)
- Clear notification callback before shutdown to prevent use-after-free
- Remove dead timeout_ns config field from TCP section
- Use size_t for count/size params in TcpParams for type safety
Signed-off-by: staryxchen <staryxchen@tencent.com>
* [TransferEngine] Add unit tests for TCP transport robustness
Cover TcpParams defaults, TcpTask atomic semantics, TcpSubBatch
pointer stability, config round-tripping, cross-thread visibility,
and exponential backoff calculation.
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
* [Store] add get_into_range and src_offset plumbing
Add single-key range read support to Mooncake Store:
- get_into_range: read [src_offset, src_offset+size) from an object
into (buffer + dst_offset)
- submitRangeRead in TransferSubmitter with src_offset support
- Get() overload and TransferReadRange in Client
- DummyClient RPC path via get_into_range_dummy_helper
- PyClient virtual interface for get_into_range
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Vladislav Nosivskoy