* [TransferEngine] feat(efa): add MC_EFA_LOOPBACK_PREFER_EMULATED to recover same-host loopback throughput
After #2041 ([TE] fix(efa): request libfabric API 1.18 so device RDMA is the default on all EFA generations) the EFA provider unconditionally enables device RDMA on every supported EFA hardware. This is the right default for cross-host transfers -- it is what unlocks the 300+ GB/s benchmarks documented in this file -- but it regresses any Mooncake Store deployment that runs producer and consumer as separate processes on the same host (single-machine development, single-host benchmarks, co-located workers).
EFA NICs have no hardware loopback short-circuit: a "loopback" fi_write still drives a real DMA round-trip through the device (PCIe out, NIC SRD packet processing, PCIe back), so a same-host transfer pays full per-packet NIC overhead instead of taking the memcpy fast path that libfabric's emulated RDMA provider applies for same-host endpoints.
Measured on p5.48xlarge (1 NIC, 80 MB transfer, two Mooncake Store clients on the same host, put_from):
FI_EFA_USE_DEVICE_RDMA=1 (default after #2041): ~830 ms / call
FI_EFA_USE_DEVICE_RDMA=0 (emulated): ~390 ms / call
The 2.1x ratio is reproducible across runs; the emulated number is on par with what we measure for the same transfer cross-host with device RDMA on (~340 ms), once single-host memory-bandwidth contention is accounted for, confirming the slow path is NIC loopback rather than anything in the Mooncake Store layers above.
Add MC_EFA_LOOPBACK_PREFER_EMULATED as an explicit opt-in. When set to 1/true/yes/on, EfaContext::construct sets FI_EFA_USE_DEVICE_RDMA=0 before fi_getinfo so the EFA provider takes the emulated path. The env is opt-in, not auto-detect, because a single EfaTransport instance may serve a mix of loopback and cross-host peers, and FI_EFA_USE_DEVICE_RDMA is a provider-level flag resolved at fi_getinfo time -- flipping it disables device RDMA for every transfer in the process, including cross-host ones, which is exactly the wrong behavior for production fan-out. We use setenv(..., 0) so an explicit FI_EFA_USE_DEVICE_RDMA set by the user still wins over the opt-in.
Cross-host benchmarks are unaffected unless the env is also set on the cross-host process; the default behavior of this code path is unchanged.
A real fix (per-transfer same-host memcpy or cross-process zero-copy via process_vm_writev) is tracked in a follow-up issue. This change is the minimal mitigation users need today to avoid silently giving up half their single-host throughput.
* [TransferEngine] refactor(efa): address review feedback on MC_EFA_LOOPBACK_PREFER_EMULATED
Per maintainer review on #2222:
1. Register MC_EFA_LOOPBACK_PREFER_EMULATED in the Environ singleton
(mooncake-common/{include/environ.h,src/environ.cpp}) instead of
parsing it ad-hoc at the call site, so it shows up in the same
inventory as every other tunable and goes through the existing
GetBool() helper (which already handles 1/true/TRUE/on/yes).
2. Guard the setenv("FI_EFA_USE_DEVICE_RDMA", "0", 0) block with
!std::getenv("FI_EFA_USE_DEVICE_RDMA"). This fixes two bugs:
- We no longer log "-> FI_EFA_USE_DEVICE_RDMA=0" when the user
has already set the env explicitly (setenv is a no-op there,
so the old log line was misleading).
- EfaContext::construct runs once per NIC (up to 32 times on
p5.48xlarge); the getenv check causes the first NIC to set
the env and subsequent NICs to skip the block entirely, so
we log exactly once.
3. Drop the hand-rolled std::transform + ::tolower entirely (which
was UB on signed char anyway -- flagged by Copilot and gemini)
by delegating to Environ::GetBool. Removes <algorithm>, adds
<cstdlib> for std::getenv/setenv.
Behavior is unchanged for the same-host case in the verification
table; this is purely structural cleanup.
* [TransferEngine] docs(efa): correct transfer size in verification table
The verification table in PR #2222 cited "80 MB transfer" as the
per-call payload, but the actual measurement was per ~1.2 GiB
(1218.8 MiB) ref blob (see ref_extractor log: blob_bytes=1218.8MiB
put=489.88ms). Update both the docs section and the in-code
comment to reflect the real transfer size. Latency numbers
(~830 ms / ~390 ms / ~340 ms) are unchanged -- they were always
measured on the 1.2 GiB blob.
Also collapse two single-statement multi-line getters/initializers
in mooncake-common to single-line form to match the existing
convention in environ.{h,cpp} (all other GetX() accessors are
single-line). No behavior change.
* [TransferEngine] docs(efa): drop MC_EFA_LOOPBACK_PREFER_EMULATED wrapper, document FI_EFA_USE_DEVICE_RDMA=0 directly
Per review feedback on #2222: the EFA user base is already familiar with
FI_EFA_USE_DEVICE_RDMA (it is documented by the EFA installer and
appears in every libfabric/EFA tuning guide), so wrapping it in a
Mooncake-namespaced alias does not pay for itself. The wrapper was a
literal one-to-one alias with no defaulting or transform.
Revert the Environ registration and the efa_context.cpp setenv block
(net code change for this PR becomes zero). Keep the diagnosis and the
verification table in efa_transport.md, but rewrite the recommendation
to point at FI_EFA_USE_DEVICE_RDMA=0 directly with the same
per-process / mixed-traffic caveat.
The long-term fix for same-host loopback (routing same-host
different-process transfers through process_vm_writev as a new
TransferStrategy::CROSS_PROCESS_MEMCPY, bypassing the NIC entirely)
remains tracked as #2223.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [TENT] Add configuration-driven transport selector
Add TransportSelector for flexible, configuration-driven transport selection
policy while maintaining full backward compatibility.
Key features:
- Configuration-driven transport selection via JSON policy rules
- Support for segment_type filtering, device allocation, and transport priority
- Legacy mode option (use_legacy_transport_selection) for exact original behavior
- Default policies match original hardcoded behavior
Changes:
- Add TransportSelector class with SelectionContext, SelectionPolicy, SelectionResult
- Integrate TransportSelector into TransferEngineImpl
- Add legacy mode support to preserve original code path
- Restore TaskInfo fields (xport_priority, failover_count) for backward compatibility
- Add max_failover_attempts configuration option
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [TENT] Add comprehensive unit tests for TransportSelector
Add transport_selector_test.cpp with test coverage for:
- Default policies matching original behavior (File/Memory segments)
- Transport type name parsing
- Legacy mode enable/disable
- Transport availability based on capabilities
- Priority offset for fallback scenarios
- Device mask handling
- NVLINK same-machine constraint
- ROCm memory type support
- GPU-to-GPU, CPU-to-CPU, CPU-to-GPU, GPU-to-CPU transfers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [TENT] Fix FakeTransport to use protected caps member
Fix compilation error by accessing Transport::caps (protected)
through helper methods instead of a separate public member.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix
* fix format issues
* Add priority-based rule
* Reformat
* Fix review comments
* Fix memory leak in endpoint_store_integration_test
When ibv_get_device_list returns a non-NULL list but num_devices == 0,
we need to call ibv_free_device_list before returning to avoid leaking
the allocated memory.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Trigger CI
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revise implementation
* Reformat
* fix fallback logic
* Add QoS APIs and docs
* Reformat
* Add QoS starvation prevention & bugfixes
* Reformat
* update docs
* remove tl_caller_id
* Fix worker distribution for multi-threaded submissions
When multiple threads submit slices simultaneously, they all start
distributing from worker 0, causing contention. Use thread_local
offset to distribute starting worker across threads, ensuring
each thread begins from a different worker.
Also rename submit_slices to next_worker_idx for clarity.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Introduce`USE_MLX5DV`CMake option and link against`libmlx5`when enabled
- Add`MC_MLX5_QP_UDP_SPORTS`
environment variable to specify comma-separated UDP source ports for ECMP/LAG
path diversification
- Add`MC_MLX5_QP_LAG_PORT_BALANCE`
environment variable to enable automatic QP distribution across bonded LAG ports
- Update`RdmaContext`to query and expose the number of LAG ports via
`mlx5dv_query_device`
- Implement QP modification logic in`RdmaEndPoint::doSetupConnection`
to apply UDP source port and LAG port pinning
- Extend configuration parsing and logging to support the new mlx5-specific
options
- Document the new runtime options in the design documentation
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix(file_storage): correct GC interval default fallback in FromEnvironment
The GetEnvOr fallback for client_buffer_gc_interval_seconds was
incorrectly using config.heartbeat_interval_seconds (10s) instead of
config.client_buffer_gc_interval_seconds (1s). This copy-paste bug
caused the GC thread to run every 10 seconds instead of every 1 second
when MOONCAKE_OFFLOAD_CLIENT_BUFFER_GC_INTERVAL_SECONDS is not set,
significantly delaying zombie buffer reclamation.
Fixeskvcache-ai/Mooncake#2119
* docs: update GC interval default value
* [Doc] Update p6-b300 EFA throughput numbers post-#1944
Re-measured on a fresh main build between two p6-b300.48xlarge
instances. GPU-to-GPU peak 780 GB/s write (~97.5% of 800 Gbps line
rate, up from 752); CPU-to-CPU peak 283 GB/s write / 270 GB/s read
(up from 230/180). Drops the "predates the refactor" caveat.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* [Doc] Update EFA transport doc: SGLang section + vllm-router + p5.48xlarge bench
- Rewrite "Usage with SGLang" to use sglang PR #25083 (MOONCAKE_PROTOCOL=efa
flows natively) and mirror the vLLM section structure (Prefill / Decode /
Router subsections). Drop unrelated GLOO_SOCKET_IFNAME and NVSHMEM bits.
Note the cross-host router trap: PREFILL_HOST must be reachable from decode.
- Add Router subsection to "Usage with vLLM" using vllm-router with
--kv-connector mooncake.
- Drop the SGLang Docker subsection (redundant).
- Add benchmark section "4. p5.48xlarge (H100, 32 EFA × 100 Gbps)" with
GPU-to-GPU and CPU-to-CPU sweeps. Peak GPU 389 GB/s write / 382 GB/s read
(~97% of 400 GB/s line rate). CPU plateaus at ~64 GB/s, bounded by
DDR4-3200 on EPYC 7R13.
- Update Tuning Tips for the 32-NIC WR cap (8192 vs 4096 on 16-NIC hosts)
and the read-batch difference between p5en and p5.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat: add Hygon DCU/DTK and Iluvatar CoreX platform support
Add build system and runtime support for two CUDA-compatible domestic
accelerator platforms:
- Hygon DCU with DTK SDK (USE_HYGON, /opt/dtk/cuda/cuda-11/)
- Iluvatar CoreX SDK (USE_COREX, /usr/local/corex/)
Both platforms expose CUDA-compatible APIs, so the integration follows
the same pattern as existing CUDA-like platforms (MUSA, MACA): add the
new macros to all platform guard chains and register SDK paths in CMake.
---------
Co-authored-by: KarmaD7 <KarmaD7@users.noreply.github.com>
Exposes the existing TransferMetadata::sendProbe C++ method through the
TransferEngine pybind module as engine.send_probe(peer_server_name).
This enables SGLang's MooncakeKVManager to issue lightweight JSON-RPC
probes against peers, used to test whether a previously-blacklisted
mooncake_session_id has become reachable again so it can be removed
from the failed_sessions set.
Returns 0 on success, non-zero on failure (matching the C++ contract).
No behavior change for existing engine.* methods.
Tested:
- New Python unit tests in transfer_engine_initiator_test.py covering
both the reachable-peer and unknown-peer cases.
- Manually validated end-to-end against SGLang's MooncakeKVManager.
* support engram
Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
* add test case for engram
Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
* add docs for engram
Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [Store] Add lock-free MmapArena allocator for buffer mmap path
Replace per-allocation mmap() syscalls in allocate_buffer_mmap_memory()
with a lock-free atomic bump allocator (MmapArena). Pre-allocates a
configurable pool (default 64GB) and serves allocations via CAS loop,
reducing allocation latency from ~1us (mmap syscall) to ~50ns (atomic).
Allocation lifecycle is static: all callers (ClientBufferAllocator,
global segments in RealClient::setup_internal) allocate at startup and
free at shutdown. The arena outlives all allocations, so the bump-only
(no individual free) design is correct for this usage pattern.
Feature-flagged via gflags:
--use_mmap_arena_allocator (default: true)
--mmap_arena_pool_size (default: 64GB)
Falls back to direct mmap() when arena is disabled, fails to init,
or is exhausted.
Cherry-picked from flow-ipc-poc branch (utils.cpp perf path only).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Store] Fix three correctness issues in MmapArena
1. Honor caller's alignment contract: allocate() now accepts a
per-call alignment parameter and uses max(arena default, caller
request). allocate_buffer_mmap_memory() forwards its alignment
argument to the arena. Previously, the caller's alignment was
silently ignored — the arena always used 64-byte alignment
regardless of what the caller requested.
2. Remove MAP_POPULATE from arena pool mmap: the default pool is
64GB but callers typically use only a fraction (e.g. 4GB of
segments). MAP_POPULATE would pre-fault all 64GB of pages upfront,
causing seconds of startup delay and potentially triggering OOM
on machines with less physical memory. Pages now fault on demand.
3. Make alignment_ atomic and store it BEFORE the CAS on pool_base_:
previously alignment_ was a plain size_t written AFTER the release
CAS, so the store was not in the happens-before relationship
established by the acquire-release pair on pool_base_. Now both
alignment_ and pool_size_ are stored before the CAS with the
release fence guaranteeing their visibility to readers.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(tent): improve RailMonitor recovery with exponential backoff and
configurable parameters
- Introduce exponential backoff cooldown with a configurable upper bound
(kMaxCooldown)
- Support dynamic configuration of error threshold, error window, and cooldown
via Config
- Call markRecovered on successful transfer completion to un-pause rails
promptly
- Reset error count and cooldown on recovery to prevent accumulated doubling
- Cache target_machine_id in RdmaSlice to avoid segment lookup on hot path
- Add unit tests for recovery behavior, cooldown reset, and best-device mapping
Signed-off-by: staryxchen <staryxchen@tencent.com>
* test(engine): add end-to-end failover tests and test-only transport swap hook
- Add`swapTransportForTest`method to`TransferEngineImpl`
for test-only transport injection.
- Introduce`engine_failover_e2e_test.cpp`
with scenarios: status corruption failover, budget exhaustion, mixed faults,
per-task independence, and boundary conditions for
`max_failover_attempts`.
- Register new test target`tent_engine_failover_e2e_test`in CMakeLists.
Signed-off-by: staryxchen <staryxchen@tencent.com>
* docs(tent): add failover design doc
Describe TENT's two-layer failure handling: cross-transport failover in
TransferEngineImpl and per-rail cooldown recovery in RailMonitor.
Covers fault model, state machines (with code refs), config knobs,
observability, testing, and known gaps (submit-stage failover, cooldown
reset on recovery, no CI coverage).
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(tent): cache RailMonitor pointer on slice to avoid hot-path string lookup
Each RdmaSlice carried the target machine_id as std::string and
asyncPollCq did a hash+strcmp lookup on worker.rails per completion.
Resolve the RailMonitor once during generatePostPath and stash the
pointer on the slice; the completion path becomes a single deref.
WorkerContext::rails now stores values via unique_ptr so a rehash
only moves the pointer slot and does not invalidate pointers already
held by in-flight slices.
disableEndpoint no longer needs a segment lookup to mark the rail
failed either -- it uses the cached pointer.
Addresses gemini-code-assist review on PR #1984.
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
* [TENT][Sunrise] Add sunrise_link transport, platform support, and UT coverage
Integrate Sunrise platform/transport wiring across TENT runtime and examples, add SunriseLink end-to-end unit tests, and fix RDMA error logging pointer formatting to avoid crash during registration failure paths.
Made-with: Cursor
* [CI] Update pre-commit hook versions
Bump pre-commit hook revisions to current releases so local checks and CI use newer lint/format toolchains consistently.
Made-with: Cursor
* [TENT][Sunrise] Resolve review issues and drop stale bench target
Address review feedback in SunriseLink transport/platform paths (stream/device context, registration map synchronization, safer probe/allocator handling, and cache-refresh strategy), and remove the obsolete transfer_engine_sunrise_bench CMake target now that its source no longer exists.
* [TENT][sunrise] add engnish doc for sunrise_link
* [TENT][sunrise] restore the memory free logic in the bench, and restore config files
---------
Co-authored-by: liujialai <liujialai@sunrise-ai.com>
* [Store] add tensor object metadata and TP upsert APIs
Introduce explicit tensor object metadata for tensor read/write paths, add TP-aware tensor upsert wrappers, and update tests/docs for the new serialized layout.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [transfer_engine] feat: make RDMA QP pkey_index configurable via MC_PKEY_INDEX
Previously the QP attr.pkey_index was hardcoded to 0 during connection
setup, which prevented use of non-default partition keys. Add a
pkey_index field to GlobalConfig (default 0) that can be overridden
through the MC_PKEY_INDEX environment variable, and apply it when
transitioning the QP to INIT state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* patch
* add test and also update doc
* Use try catch blocks instead
---------
Co-authored-by: Yan Huang <yan.huang@Yan-Huangs-MacBook-Pro.local>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [transfer_engine] fix: add periodic endpoint reclaim from monitorWorker
reclaimEndpoint() is currently invoked only from RdmaContext::endpoint()
after a new insertion. Under healthy load, insertions and evictions are
1:1 so this works. Under failure load -- many error completions trigger
deleteEndpoint(), but new-insertion traffic stalls because the dead peer
isn't generating new connection paths -- waiting_list_ grows without
bound and QPs never get destroyed.
Add a 1Hz reclaimEndpoints() call from monitorWorker on the existing
1-second context heartbeat. This decouples reclaim cadence from
insertion traffic.
See issue #1845.
* [transfer_engine] test: endpoint_store reclaim coverage for #1845
Adds unit + integration coverage for the periodic reclaim fix.
endpoint_store_test (5 tests, no RDMA device, runs under ctest):
- reclaim drains quiescent entries on its own
- reclaim leaves active entries alone (gate preserved)
- reclaim is idempotent when empty
- leak manifests without reclaim call (1118-eviction mirror of reporter)
- reclaim works without active map (guard against insert/reclaim coupling)
endpoint_store_integration_test (requires RDMA device, not auto-registered):
- Verifies WorkerPool::monitorWorker actually calls reclaimEndpoints at
~1 Hz by constructing a real RdmaContext and waiting for the tick to
drain injected entries. Confirms the end-to-end fix wiring.
Supporting changes:
- EndpointStore::waitingListSize() accessor (diagnostics + tests)
- SIEVEEndpointStore::testOnlyInsertWaiting() for test injection
- RdmaContext::endpointStore() accessor (diagnostics + tests)
* [docs] note periodic reclaim behavior and #1845 symptom
- design/transfer-engine: add a sentence to Endpoint Management explaining
that waiting_list_ drains both on insertion and on the monitorWorker
heartbeat, so accumulated reclaim does not stall under failure load.
- troubleshooting: extend the "Failed to create QP: Cannot allocate
memory" entry with a bullet pointing at issue #1845 so operators
seeing the symptom find the cause and the fix.
* [transfer_engine] fix: guard FIFOEndpointStore::waitingListSize with atomic counter
Per PR #1952 review: FIFO variant returned waiting_list_.size() on
std::unordered_set without holding endpoint_map_lock_, racing
concurrent modification. Mirror the SIEVE pattern with an atomic
waiting_list_len_ incremented in delete/evict, decremented in reclaim.
* [transfer_engine] test: suppress intentional RdmaTransport leak under LSAN
CI build (3.10/3.12) runs with -DENABLE_ASAN=ON and LSAN flagged the
5 × 288 byte allocation the test fixture intentionally leaks
(~RdmaTransport dereferences a null metadata_ unless install() ran).
Gate on __SANITIZE_ADDRESS__ / __has_feature and mark the pointer with
__lsan_ignore_object so real leaks are still caught.
* [transfer_engine] fix: widen waiting_list_len_ atomic to size_t
waitingListSize() returns size_t but the underlying counter was atomic<int>,
which quietly narrowed on load. Promote to atomic<size_t> in both FIFO and
SIEVE so the getter is a clean pass-through with no implicit conversion.
* [transfer_engine] docs: pin reclaimEndpoint lock contract on base interface
monitorWorker now calls reclaimEndpoint() via RdmaContext; it already
acquired endpoint_map_lock_ internally, but nothing declared that. Codify
the precondition on the base so future callers know not to hold the lock.
RWSpinlock is non-reentrant, so recursive acquisition would deadlock.
* [transfer_engine] refactor: narrow RdmaContext endpoint store test surface
Previously exposed a raw EndpointStore* via RdmaContext::endpointStore()
for the integration test. A raw pointer is easy to misuse outside of
tests and couples the caller to the concrete store via dynamic_cast.
Replace with two narrow methods on RdmaContext: waitingListSize() (value
return) and testOnlyInsertWaiting(shared_ptr<RdmaEndPoint>). The latter
is lifted onto the EndpointStore base interface and implemented on both
FIFO and SIEVE, so the integration test no longer downcasts.
* [transfer_engine] test: register endpoint_store_integration_test with ctest
Integration test was previously unregistered and invoked manually. Now
self-skips via GTEST_SKIP when no RDMA device is present, so it runs
cleanly on CI runners without RDMA (skips) and on rxe/mlx5 hosts
(executes). Labeled "rdma" for ctest -L filtering.
* [transfer_engine] perf: short-circuit FIFO reclaim when waiting list is empty
monitorWorker now drives reclaim at ~1 Hz regardless of activity. On FIFO
this grabbed endpoint_map_lock_ as WriteGuard every tick even in the
common steady-state case where waiting_list_ is empty. Add the same
counter-check short-circuit SIEVE already has.
* [transfer_engine] test: skip integration test when RdmaContext::construct fails
GHA ubuntu-22.04 runners enumerate a phantom mlx5_0 via ibv_get_device_list
without a working port/GID, so pickRdmaDevice() returns a non-empty name
and the earlier GTEST_SKIP on empty device list doesn't fire. Then
construct() fails with ERR_CONTEXT and the hard ASSERT_EQ fails the test.
Convert the assertion to a GTEST_SKIP on construct failure. Matches the
"attempt setup, skip on failure" convention used elsewhere in the repo
(e.g., client_local_hot_cache_test.cpp:794-799).
* feat(efa): idempotent warmupSegment for repeated calls
Adds EfaContext::peekEndpoint() — a non-creating, normalized-key lookup
— and uses it in EfaTransport::warmupSegment() to short-circuit when
every (local_ctx, peer_nic) pair is already connected. Repeated warmup
calls on the same segment now return immediately instead of firing a
256-thread std::async fan-out each time.
Verified on p6-B300 (16 NICs × 16 peer NICs = 256 endpoints):
iter 0: 7.95s (full handshake)
iter 1: 0.000s (short-circuit hit)
iter 2: 0.000s (short-circuit hit)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(efa): evict-on-ENOMEM + drop-on-failure for endpoint store
Recover from QP exhaustion (fi_enable returns -FI_ENOMEM at the device
cap, 768/device on p6-B300) and prevent dead endpoints from piling up
under callers that drift endpoint keys (e.g. keys carrying a timestamp).
Three places:
* EfaContext::endpoint (slow path): on construct failure, evict stale
endpoints once and retry. Eviction only scans when we actually ran
out of room, so normal traffic pays nothing.
* EfaTransport::warmupSegment: if setupConnectionsByActive fails, drop
the endpoint via deleteEndpoint(normalizedKey) instead of leaving a
dead fid_ep holding its QP slot.
* EfaContext::submitPostSend: if submitPostSend returns non-zero and
the endpoint is no longer connected, drop it for the same reason.
Reproduced on B300: without the fix, injecting ":drift<iter>" into the
peer_nic_path exhausts fi_enable at iter 48 (768/16 peer NICs). With
the fix, 325 iterations complete with RSS bounded at ~6 GB.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(efa): SRD shared-endpoint refactor — drop per-peer fid_ep model
Under SRD (FI_EP_RDM), a single fid_ep per local NIC can address any number
of peers via fi_av_insert. The old model created 16 per-peer endpoints per
connection (one per local NIC), consuming 16 QPs each — hitting the 768 QP
cap after only 48 peers and forcing an LRU + evict-on-ENOMEM dance.
This change:
- EfaContext owns a single shared fid_ep + peer address vector (`peer_map_`)
- EfaEndPoint becomes a thin per-peer fi_addr_t holder (no fi_endpoint,
no fi_enable) — handshake is RPC + fi_av_insert only
- submitPostSend delegates to EfaContext::submitSlicesOnPeer, using the
peer's AV index against the shared endpoint
- setPeerNicPath detaches stale AV slot on peer reconnect so drift scenarios
re-handshake cleanly
- Deletes EfaEndpointStore + LRU/ENOMEM eviction machinery — no longer needed
- warmupSegment short-circuit updated for the shared-endpoint flow
P5EN drift stress (1 target ↔ 1 initiator, customer shape 328 GB, 16 NIC,
180s target lifetime):
- iter 0 (fresh peer) first_batch: 33.6 ms (was ~8.95 s → 266× faster)
- iter 1–3 (drift) first_batch: 260–340 ms (was ~8.95 s → ~26× faster)
- steady: ~215–220 GB/s, 0 failures across 204 M completed ops
- QP growth per added peer: 0 (was 16) — no more 48-peer cliff
See mooncake-transfer-engine/example/efa_srd_refactor_validation/ for raw
CSV and target-teardown timings.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(efa): expose warmup_efa_segment in Python binding
The C API (warmupEfaSegment) and Rust FFI (warmup_efa_segment) already
pre-connect every (local_ctx, peer_nic) pair so the first submitTransfer
does not stall on handshake RPC + fi_av_insert. The pybind TransferEngine
did not expose this, so Python-driven integrations (vLLM / SGLang) could
not benefit from the first-call latency improvement from the SRD shared-
endpoint refactor (iter 0 first_batch 33.6 ms vs 8.95 s baseline).
Mirror the C API shape: thin pass-through that dynamic_casts the installed
"efa" transport and calls EfaTransport::warmupSegment. No-op on non-EFA
builds or when the EFA transport is not installed.
Tested: built with USE_EFA=ON on p5en, imported engine.TransferEngine, and
confirmed `warmup_efa_segment(segment_name: str) -> int` is bound.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(efa): apply code_format.sh (clang-format-20)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(efa): drop internal drift_stress/validation dirs + stale gflags doc note
- Remove mooncake-transfer-engine/example/drift_stress/ — depends on
the out-of-tree customer_pattern Rust binary and isn't usable by
upstream contributors.
- Remove mooncake-transfer-engine/example/efa_srd_refactor_validation/
— validation artifacts for the SRD refactor; the headline numbers
live in the PR description instead.
- Remove the libgflags-dev note from docs/.../efa_transport.md — it
is pulled in transitively by dependencies.sh and no longer needs a
manual install step.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(efa): close shared endpoint before dropping peer map in teardown
Calling fi_av_remove() on peer addresses after the shared fid_ep was
still open — or calling it at all during context teardown when the AV
itself is about to be fi_av_close()d — trips a libfabric EFA-provider
assertion, causing the efa_transport_test unit tests to segfault on
engine destruction.
Reorder EfaContext::deconstruct() to close shared_ep_ first, then
detach every peer handle via a new markDetachedForTeardown() that just
resets the AV-slot field without calling fi_av_remove(). fi_av_close()
below invalidates every slot in one shot, so no per-slot removal is
required on the teardown path.
Verified: all 5 EFATransportTest cases (InstallTransport, LoopbackWrite,
WriteAndRead, MultiWrite, StressMultipleBatches) now pass on P5EN.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* test(efa): add coverage for warmupSegment, batch register, large xfer, re-open
Five additional EFATransportTest cases, filling gaps in the existing
suite (which only exercised install + 4 single-buffer write/read paths):
- WarmupSegmentLoopback — covers EfaTransport::warmupSegment() and
its idempotent short-circuit on the second call, plus the empty /
self-name no-op path.
- WarmupSegmentNotFound — warmupSegment must fail cleanly (not hang)
for an unknown segment name.
- RegisterMemoryBatch — registerLocalMemoryBatch +
unregisterLocalMemoryBatch round-trip.
- LargeTransfer — 128 MB buffer, 64 x 1 MB slices; exercises
WR / CQ pacing in EfaContext::submitSlicesOnPeer well past the
16 x 64 KB MultiWrite ceiling.
- RepeatedOpenSegment — openSegment() on the same peer repeatedly
must keep transferring correctly; guards the setPeerNicPath detach
path that target-restart drift depends on.
docs/.../efa_transport.md's Unit Tests table updated to match.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): restrict CTest invocation to the efa tests
Running the full ctest suite on an EFA-only host trips etcd /
mooncake_master requirements in the TCP / metadata / store tests, which
can read as "EFA broke the build". The EFA doc should not recommend
that invocation — filter to the two EFA binaries and call out the
reason in a note.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): drop redundant MC_METADATA_SERVER / MC_LOCAL_SERVER_NAME block
The defaults (P2PHANDSHAKE and 127.0.0.1:12345) are what the loopback
unit tests actually want, and nothing in the current test suite needs
an override, so documenting these "env var exports" next to the test
invocation just makes readers wonder whether they are mandatory.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): drop obsolete CUDA_VISIBLE_DEVICES tip for CPU-to-CPU runs
The tip existed because libfabric's EFA provider used to dlopen
libcudart at fi_getinfo time, creating a CUDA context even for
CPU-only benchmarks. EfaContext::construct now sets FI_HMEM=system
when the build is not GPU-enabled, which already prevents that
context from being created — the CUDA_VISIBLE_DEVICES="" workaround
is no longer needed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): lead with a bench command that hits line rate
The previous "Initiator Node" snippet used block_size=65536 /
threads=8 / buf=1GB, which reaches only ~50 GB/s on an 8x400G host
and reads as "EFA is broken" to anyone copy-pasting it. Replace with
block=1MB / threads=32 / buf=4GB — the same values that produce the
GB/s numbers quoted in the Benchmark Results tables — and add a short
sentence telling readers exactly why block_size matters. Also clarify
that the MC_SLICE_SIZE no-effect note covers both CPU-to-CPU and
GPU-to-GPU paths (verified: no EFA code reads globalConfig().slice_size,
only rdma_transport and kunpeng_transport do).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): drop MC_SLICE_SIZE references, sync target --buffer_size
EFA transport never reads globalConfig().slice_size — the only readers
are rdma_transport and kunpeng_transport — so the env-var table row,
the "MC_SLICE_SIZE has no effect" tuning-tip bullet, and the
pre-optimization tuning-history details block were all documentation
debt rather than user guidance. Drop all three.
Also: the target command defaulted to --buffer_size=1GB (the flag's
default), but the recommended initiator command now uses 4GB. Since
the target allocates and registers exactly FLAGS_buffer_size bytes and
the initiator writes into offsets [0, buffer_size), they have to
match. Add --buffer_size=4294967296 to the target snippet and a short
line calling out the constraint.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(efa): skip preTouchMemory for GPU buffers, clarify target flags
preTouchMemory() does a CPU-side store per page, which was gated only
by "chunk >= 4 GiB". That trips a segfault the moment a user registers
a >=4 GiB cudaMalloc buffer — exactly what the recommended benchmark
config does. Restrict the pre-touch to host (cpu:*) memory so VRAM
registration stays a pure libfabric call.
Also amend the benchmark docs:
- call out that --use_vram / --gpu_id apply to the target too, and
that GPU-to-GPU runs need --gpu_id=-1 on both sides
- mention --use_vram=false as the alternative to a -DUSE_CUDA=OFF
build for CPU-to-CPU runs
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): default benchmark snippet to gpu_id=-1 + threads=16
Align the benchmark quick-start with what actually reaches line rate
on a p5en.48xlarge pair (≈350 GB/s write, verified). The previous
snippet recommended --threads=32 and no --gpu_id override, which:
- ran on one GPU (default gpu_id=0), so buffers sat on one NUMA
node and half the NICs were cross-NUMA — settled at ≈56 GB/s
- ran into the SRD shared-endpoint WR cap (16 × 256 = 4096 WRs
vs 32 × 128 = 4096 outstanding slices, zero headroom) and
triggered "timed out waiting for CQ drain"
Rework the section: lead with the two knobs that matter (--gpu_id=-1
on both sides, --block_size=1048576), add --gpu_id=-1 to the target
snippet for symmetry, switch the initiator to --threads=16, and
explain in a note why 16 is the ceiling (so readers know where the
limit comes from, not just that we picked it).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): complete the Key Parameters table
The table was missing several flags that the bench snippet in the
same section actually uses (--mode, --protocol, --metadata_server,
--segment_id, --use_vram), and some of the flags it did list had
defaults / descriptions that did not match the binary (e.g. default
operation is read, not write). Sync every row to DEFINE_* in
transfer_engine_bench.cpp and add explicit "what this is for on EFA"
hints next to the two flags (--block_size, --gpu_id) that most
determine whether the run hits line rate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): reorder benchmark sections by peak throughput
Lead with p6-b300 (752 GB/s peak), then p5en (347 GB/s), then
p6-b200 (313 GB/s). Readers skim the first table to decide whether
EFA transport is fast enough for them; putting the best-performing
platform first avoids undersell.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): reorder benchmarks by hardware generation, not raw GB/s
Previous reorder put p5en (H200) above p6-b200 (B200) because the
p5en rows happen to show a higher headline number, but B200 is a
newer generation than H200 and the p6-b200 numbers there are an
older snapshot. Fix the ordering to reflect hardware recency:
p6-b300 → p6-b200 → p5en. Same change applied to the
Cross-Transport Comparison table.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): refresh p5en bench results (366 GB/s write / 304 GB/s read)
Re-swept on P5EN (H200, 16×200G) with the SRD shared-endpoint build
and --buffer_size=4GB per GPU. New peaks:
- Write: 365.66 GB/s @ threads=16, batch=128 (up from 347 GB/s)
- Read: 303.78 GB/s @ threads=16, batch=32 (up from 308 GB/s,
but now with a clear optimal config)
The new table preserves enough rows to show the two interesting
axes — write wants big batches, read wants small ones — and the
Cross-Transport Comparison is updated to 366 GB/s (~91% of the
400 GB/s line rate). B200/B300 rows are unchanged (no hardware
available to re-sweep).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): mark B200/B300 bench rows stale, refresh p5en CPU, drop cross-transport table
- p5en CPU-to-CPU re-swept on current SRD shared-endpoint build:
peak write 213.57 GB/s @ threads=48/batch=32, peak read 212.18 GB/s
@ threads=16/batch=32 (vs old 192 GB/s / 182 GB/s). Table shows a
few representative points — throughput is essentially flat across
the parameter grid because the run is DRAM-bound.
- B200 and B300 sections tagged as predating the SRD refactor and the
recent tuning work; a re-sweep is pending hardware availability.
- Drop the "Cross-Transport Comparison" subsection and the
"EFA vs RoCE RDMA" paragraph — both were derived numbers that
quickly rot and were already redundant with the per-platform
tables above them.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): add block_size + buffer_size sweep tables as evidence
Two sweeps on p5en GPU-to-GPU (SRD shared-endpoint build, --gpu_id=-1)
at the peak config (threads=16, batch=128), backing up the tuning
claims elsewhere in this file:
- block_size: 64 KB default reaches only 26% of peak. Write
throughput climbs steeply through 512 KB, plateaus between 1 MB
and 2 MB. 2 MB is MC_EFA_STRIPING_THRESHOLD (a different code
path). 1 MB is the recommended value — within 4% of the 2 MB
peak and safely below the striping boundary.
- buffer_size: only needs to clear block × batch × threads (2 GB
in the peak config). 2 GB vs 4 GB differs by ~3% on write, read
is flat within noise. The example commands use 4 GB purely as
a generous default, not because smaller fails.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): rewrite Tuning Tips around the SRD shared-endpoint model
Two stale claims under Tuning Tips were written for the old per-peer
endpoint code:
- "Increase threads to 32-48": under the shared endpoint each NIC
has a 256 WR cap, and threads × batch past 16×256 = 4096 trips
the CQ-drain timeout. The sweep on p5en shows 16 threads is the
ceiling, not the floor.
- "Write peaks at threads=32, read at threads=16 / buffer=2GB max
per GPU": both taken from the pre-refactor B300 snapshot.
Current p5en peaks are both at threads=16; the 2GB/GPU limit
was the VRAM preTouch segfault that is now fixed.
Rewrite the bullets to describe what the SRD shared-endpoint
actually constrains: block_size sweet spot, threads × batch WR
cap, the write-vs-read batch-size split, and --gpu_id=-1 on both
sides. Add an explicit line that buffer_size only needs to clear
block × batch × threads (pointing readers at the sweep table).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(efa): remove broken per-request striping across NICs
submitTransferTask had a branch that, when request.length exceeded
MC_EFA_STRIPING_THRESHOLD (default 2 MB), split the request into
num_nics slices and dispatched one per NIC. The intent was to speed
up large single transfers by paralleling across all NICs.
A sweep on p5en (SRD shared-endpoint build, 16 NICs x 200G) shows
the branch is a ~20x negative optimization in every bench config:
block=1 MB, threads=16, batch=128: ON 16 GB/s vs OFF 366 GB/s
block=8 MB, threads=16, batch=16: ON 18 GB/s vs OFF 355 GB/s
block=32 MB, threads=16, batch=4: ON 18 GB/s vs OFF 328 GB/s
Even the designed-for scenario (threads=1, batch=1, single huge
request) only wins by 1.2x (27 vs 23 GB/s) — far below the 16x
expected from parallel dispatch, because the per-slice post_lock /
peer lookup still serialize the dispatch. And that "win" is only
reachable with threads=1 — any realistic caller with multiple
threads can parallelize on its own without this mechanism.
Since the branch silently degrades every common bench / workload
configuration above the 2 MB threshold and delivers no real win
even in its best case, delete it:
- efa_transport.cpp: drop the LARGE TRANSFER if-branch and the
kStripingThreshold read.
- config.h / config.cpp: drop the efa_striping_threshold field
and the MC_EFA_STRIPING_THRESHOLD env var.
- docs/.../efa_transport.md: drop the Environment Variable row,
the "Note on EFA slicing" striping explanation, and the
"striping off" callouts in the benchmark tables and tuning
tips.
Verified: all 10 EFATransportTest cases pass, and the peak p5en
bench run is unchanged (352.81 GB/s write @ threads=16, batch=128).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): number the sequential subsections for easier skimming
Four groups of level-3 / level-4 headings in this doc read as
step-by-step instructions (do A first, then B), but only two of
them were numbered while the others relied on order alone. Result:
readers had to count positions to tell "how far through" they were.
Number every such group consistently:
- Performance Benchmark: Target Node (1) / Initiator Node (2)
- Usage with vLLM: Prefill Instance (1) / Decode Instance (2)
- Benchmark Results: p6-b300 (1) / p6-b200 (2) / p5en (3)
Reference-style sections (Key Parameters, Tuning Tips, Warmup,
Technical Details, Troubleshooting) stay unnumbered — those aren't
ordered and numbering them would imply a reading path that
doesn't exist.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* test(efa): add efa_first_submit_probe for warmup / first-submit timing
A standalone cross-node probe that measures, separately:
- warmupSegment() latency, and
- per-submit latency for the first N single-transfer submits
so we can isolate the handshake / av_insert cost (warmup=OFF makes
it land on submit #0) and compare pre-refactor vs post-refactor
code without having to squint at 10s throughput averages.
Not enabled by default ctest — only built when USE_EFA=ON. Meant
for manual two-host runs alongside transfer_engine_bench.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): replace warmup numbers with a cross-branch p5en measurement
The old "Eager endpoint warmup" numbers (4 s stall / 13.5 ms warmed)
were from a B300 snapshot that predated the SRD shared-endpoint
refactor, so they described the OLD code. Replace with a fresh
cross-node measurement on p5en that compares both branches head to
head using efa_first_submit_probe:
- Warmup itself: 1.1 s (SRD shared endpoint) vs 17 s (per-peer
fid_ep on upstream main) — ~15x faster and much more stable
(SRD runs were all 1.13-1.14 s; old code swung 9-17 s across
three reps).
- Cold first submit with NO warmup: 26 ms (SRD) vs 99 ms (old) —
~4x faster, because the shared endpoint removes the per-peer
fi_endpoint / fi_enable step that used to dominate the first
send.
The prior text's "first-batch stall" framing also reads as if the
whole first submitTransfer always pays the full handshake cost.
That's not quite how it works — the initiator round-robins across
local NICs, so only a handful of pairs warm on each call, and the
stall amortizes across the first several submits. Fix the wording.
Add a pointer to the probe source (example/efa_first_submit_probe.cpp)
so the measurement is reproducible.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): reframe "first-request latency" around two audiences
The previous version led with a long explanation of warmupSegment and
its internals, which made it sound like callers had to opt in to get
any first-request speedup from this PR. That isn't the case: the SRD
shared-endpoint refactor cuts the cold submit from 99 ms to 26 ms
automatically, and vLLM / SGLang (which don't call warmupSegment) see
that 4x win without any integration work on their side. The explicit
warmupSegment API is an additional 15x improvement on top of that,
but it matters mainly to direct Mooncake callers.
Restructure the section accordingly:
- Lead with a 2x2 table showing both numbers side by side.
- Frame the ~4x cold-submit win as the default, no-code-change
benefit everyone gets.
- Frame warmupSegment as the opt-in path for callers that want
sub-10 ms first-request latency.
- Explicitly note that vLLM / SGLang currently don't call it.
- Keep the peer handshake-daemon bottleneck note so readers
understand why the warmup scales linearly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): reference #1944 explicitly instead of "this PR"
"This PR" dangles once the doc lands on main — it's fine in review
context, less fine afterwards. Use the concrete issue number so
readers (including future us) can follow the link.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): unwrap soft-wrapped paragraphs, let the renderer flow them
Several blockquote notes, tuning-tip bullets, and the "first-request
latency" section used ~70-column soft wraps, which look fine rendered
but make the raw markdown look ragged — the line after 'Peer
addressing resolves lazily:' used to stop before the line was
actually full, which reads as a formatting bug in plain text. Let
each logical paragraph / bullet be a single line and defer wrapping
to the browser / sphinx renderer. No rendered change.
Also update the PR body on #1944 the same way.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): fix post-refactor stale references (architecture, thread safety, comparison)
Four places still described the pre-#1944 code:
1. docs/.../efa_transport.md "EFA Transport Architecture" diagram —
showed fid_ep inside EfaEndpoint, which is wrong under the
shared-endpoint model. fid_ep lives on EfaContext (shared_ep_);
EfaEndPoint only holds the peer's fi_addr_t AV slot. Redraw.
2. docs/.../efa_transport.md "Thread Safety" — said the transport
"adds per-endpoint spinlocks" (plural, per-endpoint). In the
new code there's exactly one post_lock_ per EfaContext guarding
the shared fid_ep. Reword.
3. docs/.../efa_transport.md "EFA vs RoCE RDMA" comparison rows —
used the pre-refresh p5en GPU number (347 GB/s) and the stale
B200 row (313 GB/s). Replace with the current p5en peak from
the benchmark tables above (365 GB/s GPU, 213 GB/s CPU), drop
the row pointing at a now-absent B200 re-sweep.
4. efa_context.cpp:589 — "Fast path: peer info pre-resolved by
submitTransferTask's striping path". The striping path was
removed in an earlier commit in this PR; the fast path now
just catches slices whose peer_nic_path the caller already
filled in. Reword without referencing the removed mechanism.
No behavioral change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(efa): address pr review — drop unused params, sysconf once, drop friend-class, expand probe comment
Addresses trivial review feedback on #1944:
- EfaContext::construct() no longer takes num_comp_channels / port /
gid_index. These three parameters were plumbed through from the
original RDMA-era signature but the EFA/libfabric path never read
them; the previous commit just silenced warnings with (void) casts.
Remove them from the signature and the one caller in
EfaTransport::installSglangTransport.
- preTouchMemory() was calling sysconf(_SC_PAGESIZE) twice (once
for the >0 check and once for the value). Cache the result in a
local.
- `friend class EfaEndPoint` in efa_context.h was unnecessary —
EfaEndPoint only touches public methods on EfaContext
(nicPath, localEpAddr, engine, insertPeerAddr, removePeerAddr,
submitSlicesOnPeer). Drop the friend decl.
- Expand the header comment on efa_first_submit_probe.cpp to say
what problem it exists to measure and why transfer_engine_bench
alone isn't sufficient — reviewer asked what the example is for.
No behavioral change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* perf(efa): faster hex<->binary AV address handling
Reviewer pointed out two sources of overhead in the handshake path:
1. localEpAddr() built the hex string via std::ostringstream with
std::hex / std::setw / std::setfill, which pulls in stream
formatting for every byte. Replace with a direct table-driven
encoder writing into a pre-sized std::string.
2. insertPeerAddr() decoded the hex using substr() + strtol() for
every byte (a temporary std::string and a libc base-16 parse per
pair). Replace with a single pass using a hexNibble() helper and
a pre-sized buffer.
3. Loopback called insertPeerAddr(localEpAddr()), encoding the
bytes to hex and then immediately decoding back. Expose the raw
bytes via localEpAddrBytes() and add an insertPeerAddrBytes()
overload; loopback now skips the round-trip entirely.
Handshake path is not the dominant cost in steady-state serving, but
warmupSegment() fires 256 of these on a fresh 16x16 topology and
every one of them was doing the extra work. The new path also avoids
the per-byte std::string allocation (strtol path).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(efa): use std::atomic for wr_depth / cq outstanding, fix O(N^2) erase
Replace `volatile int` + __sync_* builtins with std::atomic<int> on the
shared-endpoint pacing counters (EfaContext::wr_depth_ and
EfaCq::outstanding). `volatile` never implied atomicity under the C++
memory model, and mixing it with __sync builtins was technically UB; the
atomic version is both portable and more explicit about the intended
memory ordering (acq_rel on the mutating ops, relaxed on the speculative
load used for CAS).
Also fix an O(N^2) batch consumption pattern in submitSlicesOnPeer:
`slice_list.erase(begin, begin + batch_count)` shifted the tail on every
iteration. Consume via a moving cursor instead, and splice retry slices
in place when -FI_EAGAIN forces a partial post.
Behavior is unchanged on the happy path; the retry path now preserves
the original slice order at the cursor, matching the previous insert-at-
front semantics.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(efa): drop stale efa_latency_bench.{py,png}, document efa_first_submit_probe
The `efa_latency_bench.py` script and its `efa_latency_bench.png` output
shipped with the original EFA transport PR (#1509) two years ago. No
markdown doc references them, the script hard-codes old private IPs as
defaults, and nothing guarantees the chart is still representative.
Remove both.
Expand the First-request latency section of efa_transport.md to actually
document `efa_first_submit_probe`: what it measures (cold-submit +
eager-warmup cost, which `transfer_engine_bench`'s 10s average hides),
how to run it (target / initiator commands), the expected output layout,
all flags, and the two situations where it is worth running (deciding
whether your app needs `warmupSegment()`, and comparing PR branches on
the same hardware). Marked it explicitly as EFA-specific — RDMA/TCP
transports do not have an equivalent warmup concept, so folding this into
`transfer_engine_bench` would muddy the generic benchmark for no gain.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-8-212.us-west-1.compute.internal>
* feat(efa): auto-split large MR registrations exceeding max_mr_size
Buffers larger than the EFA device's max_mr_size are now transparently
split into chunks, each registered as a separate MR. This fixes the
silent truncation bug where only the first max_mr_size bytes were
registered, causing transfers to unregistered regions to fail at runtime.
Key changes:
- Query EFA device max_mr_size via ibverbs during init (libfabric does
not expose this) and clamp globalConfig accordingly
- Auto-split buffers > (max_mr_size - 1GB) into chunks in
registerLocalMemoryInternal, each with its own BufferDesc metadata
- Track chunk mappings for proper cleanup in unregisterLocalMemory
- Change lkey()/rkey() from exact-match to range lookup (matching
mrDesc() pattern) so key lookups work for any address within a chunk
- Replace silent truncation in EfaContext with a hard error as safety net
- Remove preTouchMemory truncation to touch the full buffer
Tested on P5EN (16 EFA NIC, max_mr_size=192GB): 191GB single-chunk
registration succeeds. Auto-split triggers correctly for larger buffers
(200GB splits into 191GB + 9GB). Total registerable size per buffer is
bounded by system pinned_vm limits (~191GB on 16-NIC P5EN).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): per-NIC partition for large buffer MR registration
Each EFA NIC can only register up to max_mr_size total. The previous
auto-split approach registered every chunk on ALL NICs, hitting the
per-NIC limit for buffers > max_mr_size. This change assigns each
chunk to a disjoint subset of NICs, enabling registration of buffers
up to max_mr_size × num_NICs (e.g. ~1.5TB on P5EN with 16 NICs).
Key changes:
- chunk_limit = max_mr_size / 2 (was max_mr_size - 1GB) for headroom
- NIC assignment: chunks distributed evenly across available NICs
- selectDevice(): skips NICs with rkey=0 (unassigned for that chunk)
- Striping path: filters by lkey!=0 to avoid unregistered NICs
- Unregister: only deregisters from assigned NICs per chunk
- ChunkRegistration struct tracks per-chunk NIC assignments
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): PTE-aware auto-split replaces conservative max_mr_size/2 threshold
The previous chunk_limit of max_mr_size/2 (~96GB on P5EN) caused unnecessary
buffer splitting even when hugepages were available. This detects the actual
backing page size via /proc/self/smaps and computes the PTE-based limit:
- 4KB pages: 22M PTEs × 4KB = 88GB (genuine hardware constraint)
- 2MB hugepages: 22M PTEs × 2MB = 44TB (effectively max_mr_size)
Verified on P5EN (H200, 16 EFA): 100GB pool with hugepages no longer splits,
restoring full 16-NIC throughput (108 GB/s vs 46 GB/s with the old threshold).
MR registration time dropped from 302s to 1.6s.
Adds MC_EFA_MAX_PTE_ENTRIES env var for override (default 22M).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add KV cache prefix transfer benchmark script
Python benchmark to measure EFA transfer performance for LLM prefix
cache hit scenarios. Tests different pool sizes (10GB-500GB) and prefix
lengths (4K-32K tokens) to evaluate per-NIC partition impact on
transfer latency and throughput.
Default KV bytes/token matches GLM-5.1 (754B MoE, MLA attention):
(kv_lora_rank=512 + qk_rope_head_dim=64) * 2 * 78 layers = 88KB/token
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(bench): add connection warmup before benchmark loop
The first prefix size's measurements were skewed by EFA connection
establishment (openSegment, endpoint creation). Add 3 small transfers
before entering the benchmark loop to warm up the connection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(bench): per-offset warmup and p50 throughput reporting
Warmup now exercises all offsets the benchmark will measure, eliminating
first-access TLB/page-fault outliers (4K token p99 dropped from 80ms to
3.5ms). Throughput is reported from p50 latency instead of avg for more
stable numbers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(bench): add --threads option for concurrent prefix transfer
Adds optional multi-threaded transfer support. Each thread transfers a
chunk of the prefix in parallel via separate transfer_sync_read calls.
Default is 1 (single transfer, same as before). Testing shows threads=2
matches single-thread throughput (~108 GB/s), while higher values add
scheduling overhead with no benefit since EFA already stripes internally.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): full NIC coverage for multi-chunk MR registration
When a buffer exceeds max_mr_size and must be split into multiple chunks,
register every chunk on ALL NICs instead of disjoint per-NIC partition,
as long as total PTE usage per NIC fits within the PTE budget.
With hugepages (2MB), 500GB buffer uses only 250K PTE/NIC (budget: 24M),
so all 16 NICs cover every address. Falls back to disjoint partition when
PTE budget is exceeded (e.g. 4KB pages with large buffers).
500GB pool throughput: 35 GB/s (disjoint, 5-6 NIC) → 108 GB/s (full, 16 NIC).
Registration time unchanged (~9.5s) due to parallel MR registration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(bench): add batch memory registration benchmark
Test script for registering multiple independent memory blocks
(e.g. multi-tenant KV cache pools). Supports both per-block
register_memory and batch_register_memory APIs. Target mode
allocates and registers N blocks; initiator mode transfers
data and measures throughput.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply clang-format to EFA transport files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): expose discoverTopology C API and Rust bindings
Add discoverTopology() to the C API and discover_topology()/install_transport()
to the Rust bindings, enabling EFA transport initialization from C/Rust callers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(efa): add C API test for discoverTopology and EFA transport
Verify that the new discoverTopology() C API correctly populates the
device list, enabling installTransport("efa") and memory registration
via the pure C interface (used by Rust/Go bindings).
All 4 tests passed on p5en.48xlarge (16 EFA NICs).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): O(log n) MR lookup and chunk registration rollback
- Replace unordered_map with std::map for mr_map_ and use upper_bound
for O(log n) range lookups in rkey/lkey/mrDesc instead of O(n) scan
- Add rollbackChunks lambda to unregister already-registered chunks on
failure in registerLocalMemoryInternal, preventing MR leaks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply clang-format to efa_c_api_test.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: rename thr_tag to thread_tag to pass typos spell check
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): round-robin multi-chunk MR and large MR registration test
When a buffer splits into more chunks than available NICs, round-robin
assign chunks across NICs with per-NIC PTE budget validation, instead
of hard-failing with "Buffer requires N chunks but only M NICs".
Add efa_single_nic_large_mr_test: tests single-NIC and all-NIC large
MR registration with hugepages. Supports --chunk_gb for multi-buffer
mode (e.g. 200×2GB).
Verified on P5EN (16 EFA NIC, 2MB hugepages):
- 1 NIC × 200×2GB (400GB): 53.7s
- 16 NICs × 200×2GB (400GB): 64.8s
- 1 NIC × 200GB single buffer: 2.7s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add cross-node transfer test for multi-buffer MR registration
Tests 200x2GB buffer registration on all NICs with actual data transfer
between two P5EN nodes. Supports target/initiator modes with single-read
and multi-block batch benchmarks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): avoid CUDA context leak when built without GPU support
libfabric 2.4's EFA provider dlopens libcudart/libcuda during
fi_getinfo/fi_domain to detect HMEM support, even though the caller
never touches GPU memory. This creates a CUDA primary context on
GPU 0 and permanently holds ~616 MiB of device memory.
When Mooncake is built with USE_CUDA=OFF (and USE_HIP=OFF), set
FI_HMEM=system before fi_getinfo and drop FI_MR_HMEM from the domain
hints so the provider skips GPU hmem initialization entirely.
* feat(efa): multi-thread initiator and wildcard location for 16-NIC coverage
Register target buffers with wildcard location "*" instead of "cpu:0"
so initiator-side remote NIC selection distributes evenly across all 16
NICs (both NUMA nodes). With "cpu:0", selectDevice only picked NUMA-0's
8 NICs, leaving NUMA-1 idle — throughput capped at ~107 GB/s instead of
~149 GB/s on P5EN (16×EFA 200Gbps).
Also convert the initiator from single-threaded to multi-threaded
(--threads flag) and print the actual P2P handshake address.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add p6-b300 bandwidth results (752 GB/s GPU, 230 GB/s CPU)
- GPU-to-GPU peak 752 GB/s write / 713 GB/s read at ~94% line rate
(16×400 Gbps = 800 GB/s theoretical)
- CPU-to-CPU peak 230 GB/s, DRAM-limited on Xeon 8559C
(NUMA-0 NICs 90 Gbps vs NUMA-1 NICs 53 Gbps)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(rust): customer_pattern sample for per-NUMA multi-MR registration
Adds a Rust binary mirroring the customer's EFA usage pattern: target
registers many MRs per NUMA (e.g. 16 NICs x N x 10GB via cpu:<numa>
locations); initiator reads/writes a specific (numa, buffer_index)
tuple, using --source-numa to pick which local NIC set is exercised.
Also makes rust/build.rs robust against non-default transfer_engine
build configurations: etcd-cpp-api is opt-in via MOONCAKE_WITH_ETCD=1,
CUDA linking is opt-in via MOONCAKE_WITH_CUDA=1 with CUDART_LIB_DIR
override, and libbase.a + libasio.so + libfabric paths are picked up
whether the repo uses the standalone or top-level CMake layout.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(rust): drop customer_pattern sample from upstream
Revert Cargo.toml [[bin]] additions and remove the bench demo source so
the public tree no longer ships it. Keep it gitignored locally so it can
be iterated on without accidental re-adds.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(efa): eager endpoint warmup for segments to remove first-batch stall
libfabric FI_EP_RDM endpoints resolve peer addresses lazily — the first
submitTransfer() to a new segment serializes fi_av_insert + handshake over
every (local_ctx, peer_nic) pair. On 16-NIC instances this is a ~4 s stall
on the first 100 × 0.5 MB batch (measured on p6-B300).
Add EfaTransport::warmupSegment(name) that pre-connects all pairs
concurrently via std::async, plus C wrapper warmupEfaSegment() (guarded by
USE_EFA) and Rust binding TransferEngine::warmup_efa_segment(). Idempotent,
safe to re-run after metadata changes. RDMA/TCP paths untouched.
Measured on p6-B300 (16 × 16 endpoints, dual-NUMA initiator):
first-batch: 4043 ms -> 13.5 ms (~300x)
steady-state: 141 GB/s -> 230 GB/s
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(efa): enable auto-split when max_mr_size is not configured
When MC_EFA_MAX_MR_SIZE is unset, max_mr was 0 and chunk_limit collapsed
to 0, bypassing the PTE-aware split entirely. Large 4KB-paged buffers
then hit the per-NIC PTE ceiling at registration time.
Fall back to pte_limit so splitting kicks in based on the PTE budget
alone when no explicit max_mr_size is provided.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* style: apply clang-format to EFA transport/tests and PEP8 to kvcache bench
Pure formatting changes to satisfy CI format hook (clang-format-20) and
address Gemini review comment on kvcache_prefix_bench.py indentation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-13-185.us-east-2.compute.internal>
Co-authored-by: whn09 <whn09@github.com>
* fix(efa): add fi_read support and endpoint eviction to EFA transport
1. EFA submitPostSend() now branches on slice->opcode to call fi_read
or fi_write, matching RDMA transport behavior. Previously only
fi_write was implemented, breaking remote-read workloads on EFA.
2. EfaEndpointStore now tracks endpoint activity and evicts stale
endpoints when approaching AV capacity. This prevents AV exhaustion
in long-running services communicating with many transient peers.
- Endpoints marked active on access, inactive on set_active(false)
- Configurable inactive timeout (default 5 min)
- evictStale() removes endpoints inactive beyond timeout
- removeDisconnected() cleans up broken connections
- Eviction triggered automatically in getOrInsert() at capacity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* perf(efa): eliminate per-slice overhead with NIC-count-based striping
Replace fixed 64KB slicing with NIC-count-based chunking in EFA
submitTransferTask. For large transfers (>128KB), creates one slice
per active NIC instead of thousands of small slices, dramatically
reducing spinlock, atomic, and allocation overhead.
- Large transfers: stripe across all active NICs (1 chunk per NIC)
- Small transfers: single slice on one NIC, no sub-slicing
- Fallback: per-slice retry for edge cases (unregistered memory)
Example: 240MB transfer on 32 NICs now creates 32 slices vs 3840,
matching NIXL approach of one fi_write per rail per descriptor.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): add FI_MR_HMEM to mr_mode hints for GPU memory registration
EFA provider requires FI_MR_HMEM in domain mr_mode hints to support
heterogeneous (GPU) memory registration via fi_mr_reg. Without this
flag, fi_mr_reg fails with EFAULT (Bad address) when registering
CUDA device memory buffers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style(efa): apply clang-format-20 formatting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): restore lost changes from merge — endpoint eviction, batched WR, docs
Restores 5 commits that were lost during the upstream merge:
- AV entry removal on disconnect to prevent target degradation
- Endpoint reuse for same physical peer (normalizeNicPath)
- Batched WR submission in submitPostSend
- Pre-resolved remote peer info for striped transfers
- Documentation: GPU-to-GPU (313 GB/s), CPU-to-CPU (222 GB/s) benchmarks,
CPU-only build instructions, B300/P5 instance types
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): use dependencies.sh for build deps, add EFA-specific extras
Replace manual package list with dependencies.sh script reference.
Add libgflags-dev as EFA-specific extra dependency.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add tip to use CUDA_VISIBLE_DEVICES for CPU-to-CPU benchmarks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add DLAMI troubleshooting for CUDA env and Python activation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add LIBRARY_PATH for CUDA libs, remove unneeded CUDAToolkit_ROOT
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add block_size=1MB to all benchmark configurations
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): merge clone step into build dependencies section
dependencies.sh already runs git submodule update --init --recursive,
so the separate clone section was redundant. Move git clone into the
Prerequisites section and renumber build steps.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add SGLang usage, Docker troubleshooting, reorder sections
- Move Usage with vLLM/SGLang after benchmark results (was too early)
- Add Usage with SGLang section (EFA patch, env vars, Docker example)
- Add libfabric version mismatch in Docker troubleshooting
- Reorder p5en results: GPU-to-GPU before CPU-to-CPU (consistent with B200)
- Update p5en single-instance CPU results: Write 179 GB/s, Read 185 GB/s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): use GlobalConfig max_ep_per_ctx (65536) as default, round-robin CQ assignment
- Change EfaEndpointStore and EfaContext::construct default max_endpoints
from 256 to 65536, consistent with GlobalConfig.max_ep_per_ctx
- Add round-robin CQ assignment (nextCq) for new endpoints instead of
always using cq_list_[0], distributing load across CQs
- Pass CQ outstanding counter directly to endpoint construct
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style(efa): clang-format, move striping threshold to GlobalConfig
- Run clang-format on all EFA transport files and common.h
- Move MC_EFA_STRIPING_THRESHOLD from inline std::getenv to GlobalConfig
mechanism (loaded centrally in loadGlobalConfig), per reviewer feedback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: add 'wqs' to typos ignore list (EFA API function name)
efadv_query_qp_wqs is a libfabric EFA API symbol, not a typo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add per-transfer latency benchmark script
Automated script that measures single-transfer latency across block sizes
by SSHing into target/initiator nodes and running transfer_engine_bench
with threads=1, batch_size=1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add --threads and --batch_size args to latency bench script
Allow configurable threads/batch_size for multi-thread bandwidth scaling tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add --env flag to pass env vars to remote bench via SSH
Allows setting MC_EFA_STRIPING_THRESHOLD and other env vars on the
remote initiator without manual SSH config.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: fix typos check by adding wqs to extend-words
The extend-ignore-words key is not recognized by typos v1.30.2 used
in CI. Add wqs (EFA API function name efadv_query_qp_wqs) to the
[default.extend-words] section which is the correct format.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: revert formatting-only changes in common.h
Revert pointer/reference style changes (`&`/`*` placement) that were
introduced by clang-format but are unnecessary for this PR. Keep only
the normalizeNicPath() function addition.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): clarify that MC_SLICE_SIZE does not apply to EFA transport
EFA transport uses NIC-count-based striping (since commit 84d7dc5),
not fixed-size slicing. block_size directly determines per fi_write/
fi_read size for transfers below the striping threshold. Update docs
to reflect this and correct outdated tuning advice.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): address review issues - nextCq guard, error path double-free, GPU MR registration
1. nextCq(): add empty guard to prevent division-by-zero when cq_list_ is empty
2. submitTransferTask error paths: replace getSliceCache().deallocate(s) with
s->markFailed() to prevent double-free (TransferTask destructor also deallocates)
and ensure tasks complete with FAILED status instead of hanging
3. registerMemoryRegionInternal(): use fi_mr_regattr() with explicit FI_HMEM_CUDA
iface and device ordinal for GPU memory, since EFA provider fi_mr_reg()
hardcodes iface=FI_HMEM_SYSTEM
4. Make total_bytes updates atomic (__sync_fetch_and_add) for thread safety
Tested: unit tests (5/5), cross-node GPU benchmark (write 250 GB/s, read 269 GB/s)
on p5en.48xlarge.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style(efa): apply clang-format to efa_context.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-8-212.us-west-1.compute.internal>
Co-authored-by: whn09 <whn09@github.com>
Suryxin