* feat(transfer-engine): extract Device API and IBGDA transport layer
Move IBGDA files from mooncake-ep to mooncake-transfer-engine (git mv):
- 6 headers: mooncake_ibgda/ → transport/device/ibgda/
- 1 source: mlx5gda.cpp → transport/device/mlx5gda.cpp
Add new Device API layer under transport/device/:
- device_transport.h: P2pTransport + RdmaTransport interfaces
- device_ops.cuh: DeviceOps function pointer table (bottom IR)
- comm_device.cuh, p2p_device.cuh, ibgda_device.cuh: device contexts
- cuda_ops.cuh, musa_ops.cuh: platform DeviceOps implementations
- ibgda_device_transport.cpp: RdmaTransport IBGDA implementation
- p2p_device_transport.cpp: P2pTransport NVLink/MTLink implementation
Modify transfer-engine:
- transfer_engine.h/impl: add getOrCreateP2pTransport/RdmaTransport
- gpu_vendor/musa.h: add MUSA API aliases
- Build: link mlx5, add device/ subdirectory
Minimal EP changes (include paths + build system only):
- Update include paths to new transport/device/ibgda/ location
- Remove mlx5gda.cpp from EP sources
- Forward EP_USE_MUSA env var in BuildEpExt.cmake
Root CMakeLists.txt: guard CUDAToolkit with USE_CUDA
* feat(example): add device_transport_example for P2P Device API
Two-rank example demonstrating the full Device API lifecycle:
- Host side: P2pTransport for IPC handle exchange and peer mapping
- Device side: CommCtx + mc_route_put + mc_signal for GPU-initiated
P2P data transfer and notification
Uses file-based IPC handle exchange (no external dependencies).
Requires 2 GPUs with P2P access (NVLink or PCIe).
* fix(example): handle missing TORCH_CUDA_ARCH_LIST
* fix(example): convert torch arch format to CMake CUDA format
* fix(example): construct CommCtx manually on host side
* fix(example): add using namespace mooncake::device in kernels
* fix(example): pass CommCtx by value, add barrier before cleanup
- Pass CommCtx by value to kernel (CUDA copies to param space) instead of
dereferencing a device pointer on host (caused segfault).
- Add file-based barrier so rank 0 waits for rank 1 to finish before
freeing its GDR buffer (IPC handle only valid while allocation exists).
- Change default metadata_server to P2PHANDSHAKE (no etcd dependency).
* fix: address review feedback — fence ordering, error checks, QP leak
- musa_ops.cuh: fix fence ordering for acquire/release semantics
(fence after load for acquire, before store/atomic for release)
- ibgda_device_transport.cpp: check cudaMalloc return values,
add num_qps >= num_ranks guard, destroy QP on rst2init failure
- p2p_device_transport.cpp: add device_count > 0 guard
- device_transport_example.cu: validate kDataBytes % 16 == 0
and kDataBytes <= kSignalWordOffset
* style: apply clang-format to Device API files
* fix: guard device transport code with USE_CUDA/USE_MUSA macros
The device transport accessors (getOrCreateP2pTransport,
getOrCreateRdmaTransport) and their member variables were not guarded
by USE_CUDA/USE_MUSA preprocessor macros. When building with
USE_CUDA=OFF (the default), the device transport source files aren't
compiled but the headers and implementations still reference them,
causing linker errors in CI build-flags and build jobs.
* fix: set CMAKE_CUDA_STANDARD 20 and make ibgda PUBLIC
- common.cmake: add CMAKE_CUDA_STANDARD 20 so nvcc compiles host code
in C++20 mode, matching CMAKE_CXX_STANDARD. Fixes "starts_with is
not a member of std::string" when compiling .cu files that indirectly
include common.h.
- transport/CMakeLists.txt: change ibgda from PRIVATE to PUBLIC so
mlx5gda_* symbols are visible to downstream consumers (Go p2p store
via transfer_engine). Fixes undefined reference errors for
mlx5gda_destroy_qp, mlx5dv_devx_umem_reg, etc.
* fix: compile mlx5gda into device_transport and link mlx5 for Go consumers
The previous attempt (PUBLIC ibgda) did not work because the Go p2p-store
and mooncake-store binaries link libtransfer_engine.a via hand-written cgo
ldflags, which bypass CMake's target_link_libraries propagation entirely.
- transport/device: compile mlx5gda.cpp directly into the device_transport
OBJECT library (like every other transport module) instead of a separate
ibgda STATIC lib, so mlx5gda_* symbols flow into libtransfer_engine.a and
are visible to all consumers regardless of how they link.
- transport: link libmlx5 (PUBLIC) since ibgda_device_transport.cpp /
mlx5gda.cpp call mlx5dv_devx_* / mlx5dv_init_obj directly.
- p2p-store/build.sh, mooncake-store/go/build.sh, ci.yml: add -lmlx5 to the
hand-written cgo ldflags so the DevX symbols resolve.
- example: set CUDA_STANDARD 20 on device_transport_example so nvcc compiles
common.h (std::string::starts_with) in C++20 mode.
* fix: CUDA_EXTENSIONS OFF for example, add -lm for Go consumers
Follow-up to compiling mlx5gda.cpp into device_transport:
- example: nvcc has no gnu++20 dialect, so CUDA_STANDARD 20 with the default
CUDA_EXTENSIONS=ON fails at CMake generate ("does not know the compile
flags"). Set CUDA_EXTENSIONS OFF to request plain -std=c++20.
- p2p-store/build.sh, mooncake-store/go/build.sh: mlx5gda.cpp uses log2/ceil
(<cmath>); now that its object lives in libtransfer_engine.a, the hand-
written cgo ldflags need -lm to resolve log2@GLIBC_2.29. (ci.yml already
had -lm.)
* fix(ci): gate Device API GPU example off by default, link mlx5 for Rust
The Docker build failed at CMake generate because device_transport_example
needs the CUDA20 dialect (transfer_engine.h -> common.h uses C++20
std::string::starts_with), which the older CMake in the CI image cannot
enable. CUDA_EXTENSIONS OFF did not help since the limitation is the CMake
version, not the dialect flavor. Gate this manual, 2-GPU example behind a
new BUILD_DEVICE_TRANSPORT_EXAMPLE option (default OFF) so the default build
no longer requires CUDA20.
Also link mlx5 in the mooncake-store Rust build script: the IBGDA device
transport (mlx5 DevX) is now compiled into transfer_engine, so the Rust
test link step needs -lmlx5 to resolve mlx5dv_devx_* symbols.
* [Common] Harden Environ parsing, fix opendir leak, support .yml config
Fixes#2315
1. Fix: Replace opendir() with stat()+S_ISDIR() in config.cpp to fix
DIR* handle leak.
2. Robustness: Replace atoi() with strtol() in Environ::GetInt with
endptr/errno validation. Invalid values now fall back to default
with a warning instead of silently returning 0.
3. Robustness: Add leading '-' rejection in Environ::GetSizeT to
prevent strtoull negative wrapping (e.g. MC_SLICE_SIZE=-1 yielding
ULLONG_MAX).
4. Feature: Support .yml extension in DefaultConfig::Load().
5. Tests: Add 26 unit tests for Environ::GetInt/GetSizeT/GetBool/
GetString covering valid, invalid, missing, overflow, negative,
and trailing garbage inputs. Tests call the real production code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Common] Use strtoll instead of strtoull in GetSizeT for robust negative handling
Address Gemini review: val[0]=='-' check missed leading-whitespace
cases like " -1". Using strtoll catches negatives regardless of
whitespace, and also guards against 32-bit SIZE_MAX truncation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Common] Address review: restore comments, use std::filesystem
- Restore helper method comments stripped when moving to public
- Switch from stat()+S_ISDIR() to std::filesystem::is_directory()
per reviewer suggestion (C++20 project, already used elsewhere)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Common] Fix clang-format violation in environ.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Common] Address Copilot review: non-throwing is_directory, fix stray PUBLIC
- Use std::error_code overload of std::filesystem::is_directory to
avoid throwing on permission errors (EACCES). Matches original
opendir() non-throwing behavior.
- Remove pre-existing stray PUBLIC keyword in tests/CMakeLists.txt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1. multi_transport.cpp: Delete Transport object when install() fails.
The raw pointer was leaked on the error return path.
2. transfer_engine_c.cpp: Add null check after malloc and early return
when size is 0 in getNotifsFromEngine(). Prevents null dereference
in memset when malloc fails, and avoids implementation-defined
behavior of malloc(0).
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Cache engine and target in QueryBatch to avoid per-poll metadata lookups,
propagate route failures only to batches already in pending_batches with
a single disconnect, and treat NOT_CONNECTED as success on auto_connect
disconnect.
Co-authored-by: Developer user <youxiao@huawei.com>
* fix: route EFA same-process loopback transfers through local copy
The EFA provider's SHM intra-node path performs a host memcpy into
FI_HMEM_CUDA device buffers and segfaults on the first same-host
transfer (loopback self-transfer), e.g. checkpoint-engine P2P weight
update on a single TP=8 node. See ofiwg/libfabric#12328.
Detect same-process self-loopback in EfaContext::submitPostSend (peer
NIC path equals our own nicPath(), whose server_name embeds the
per-process RPC port, so the match guarantees the peer is this very
process on this device) and satisfy the copy locally with a GPU-aware
cudaMemcpy (cudaMemcpyDefault), bypassing EFA entirely. Same-host
cross-process peers carry a different port, never match, and still go
through EFA.
The copy direction honors the slice opcode, mirroring fi_read/fi_write:
WRITE copies source_addr -> dest_addr, READ copies dest_addr ->
source_addr (the two are distinct local buffers, so it is not a
symmetric self-copy).
This mirrors how the RDMA transport already treats loopback as a
special case (rdma_endpoint.cpp self-connected QP); RDMA relies on NIC
hardware loopback and is unaffected by the libfabric SHM bug.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: add EFA GPU loopback test for FI_HMEM_CUDA same-host transfers
Add efa_gpu_loopback_test.cpp, the CUDA-device-memory counterpart of
efa_transport_test.cpp (which only covers host/numa loopback). It
reproduces the EFA SHM intra-node segfault on FI_HMEM_CUDA buffers
(ofiwg/libfabric#12328) and validates EfaContext::tryLoopbackCopy:
* GpuLoopbackWrite — same-host GPU WRITE must not crash.
* GpuLoopbackWriteThenRead — WRITE then READ with byte-accurate
verification, exercising both copy directions.
* GpuLoopbackMultiWrite — batched GPU writes through the per-slice
loopback short-circuit.
Self-skips when no EFA device or no CUDA GPU is present. Registered
under `USE_EFA AND USE_CUDA`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* efa: log fabric name and document same-host GPU loopback segfault
Add fabric_attr->name to the EFA device init log so the active fabric
(efa-rdm vs efa-direct) is visible at runtime — both share the same
provider name, FI_EP_RDM type, and <device>-rdm domain name, so the
fabric name is the only field that distinguishes them.
Revise the "Single-host loopback" doc section:
- Correct the same-host fast-path attribution: the memcpy fast path is
supplied by the SHM provider (FI_EFA_ENABLE_SHM_TRANSFER, default on),
not by FI_EFA_USE_DEVICE_RDMA. Confirmed at runtime: a default
(device-RDMA-enabled) config still reports "Opened fabric: shm".
- Add a warning that the default SHM path host-memcpy's into FI_HMEM_CUDA
destinations and segfaults on GPU buffers (ofiwg/libfabric#12328);
document the same-process tryLoopbackCopy short-circuit and the
FI_EFA_ENABLE_SHM_TRANSFER=0 workaround for cross-process GPU peers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Separate the tebench benchmark from the TENT build so it can be compiled
with only the classic Transfer Engine backend.
- Gate the benchmark subdirectory solely on BUILD_BENCHMARK instead of
also requiring USE_TENT.
- Drop tent_backend.cpp from the sources and only link tent_link_group /
define USE_TENT when USE_TENT is enabled; otherwise expose just the
tent/include header path for the header-only helpers used by the
classic backend.
- Guard the TENT backend include and runner in main.cpp, returning an
error when a TENT-only backend is requested in a non-TENT build.
The TCP transport previously hardcoded a 64KB (65536 bytes) slice size
for splitting large transfers into socket read/write operations. This
commit makes it configurable via the MC_TCP_SLICE_SIZE environment
variable, consistent with the RDMA transport's MC_SLICE_SIZE naming.
Usage: export MC_TCP_SLICE_SIZE=1048576 # 1MB slices
Default: 65536 (64KB, unchanged)
* [TENT] Add policy name binding to transport selector
Add ability to bind a request to a specific transport policy by name,
making the policy's "name" field in configuration actually useful.
Changes:
- Add optional `policy_name` field to Request struct (types.h)
- Add optional `policy_name` field to SelectionContext (transport_selector.h)
- Modify matchesPolicy() to prioritize exact policy name matching
when context.policy_name is specified
- Pass policy_name from request to context in transfer_engine_impl.cpp
When a request specifies policy_name, the selector will only match
the policy with that exact name, ignoring other matching conditions
(segment_type, priority, memory_pattern, etc.).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reformat
* Update mooncake-transfer-engine/tent/src/runtime/transport_selector.cpp
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Reformat
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update intraNode nvlink transport from MemcpuAsync to BatchAsync
* Code format update for intraNode nvlink
* Change addr to base_addr for register and unregister
---------
Co-authored-by: 百麒 <yaozhong.lyz@alibaba-inc.com>
* Initial plan
* Add MC_RDMA_BIND_ADDRESS support for dual-NIC P2PHANDSHAKE setups
In dual-NIC environments where TCP and RDMA use separate interfaces,
P2PHANDSHAKE mode previously required using a single IP for both
TCP handshake and RDMA NIC paths, causing conflicts.
This change adds MC_RDMA_BIND_ADDRESS env var support:
- When set, RDMA NIC paths use the RDMA-reachable IP
- TCP P2P routing continues using the local_server_name IP
- Segment descriptors carry rdma_server_name for consistent NIC
path construction on both sides
- P2P metadata exchange caches RDMA->TCP address mapping so
subsequent handshakes resolve to TCP-routable addresses
* Changes before error encountered
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/fc2826eb-a0ae-450f-b1f4-4ab94269d97a
* Apply dual-NIC (MC_RDMA_BIND_ADDRESS) support to TENT transport and update Chinese docs
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* [TransferEngine][ROCm] Add HIP dmabuf MR registration for AMD GPUs
Fixes#751.
Adds a parallel `#elif defined(USE_HIP)` branch in
RdmaContext::registerMemoryRegionInternal that mirrors the existing CUDA
dmabuf path (added by #704) using ROCm's `hsa_amd_portable_export_dmabuf()`
instead of `cuMemGetHandleForAddressRange(...DMA_BUF_FD...)`. This lets
Mooncake register AMD GPU memory for RDMA without requiring an
nvidia-peermem-equivalent kernel module — the path UCX's ROCm backend
(uct/rocm/base/rocm_base.c) already uses successfully.
Same host-vs-device split as the CUDA branch: `hipPointerGetAttributes`
detects host memory and falls back to `ibv_reg_mr`; device/managed memory
goes through the dmabuf path. `hipMemGetAddressRange` is used to get the
true allocation base because `addr` may sit at an offset within a larger
hipMalloc block (caching allocators pack tensors).
CMake: added `hsa-runtime64` to the HIP link line in
mooncake-transfer-engine/src/CMakeLists.txt.
Validation:
- Standalone dmabuf probe verified PASS on:
* AMD MI355X (gfx950) + Pensando ionic + ROCm 7.2.2
* AMD MI300X (gfx942) + Broadcom Thor2 (bnxt_re) + ROCm 7.0.2
Probe source + container recipe:
https://github.com/andyluo7/dynamo/blob/amd-poc-consumer-polish/amd-mi355x-poc/advanced/debug-probes/dmabuf_register_probe.cpp
- Standalone compile check confirms all HIP/HSA/ibverbs symbols in the
new branch resolve and link cleanly with hsa-runtime64 + libibverbs.
End-to-end SGLang+Mooncake disagg validation (T3) on MI355X+ionic will
follow in a comment once a full Mooncake build with submodules completes.
CC @misterwilliam @stmatengss @alogfans (active on #751)
Closes#751
---------
Signed-off-by: Andy Luo <anluo@amd.com>
Signed-off-by: Andy Luo <andy.luo@amd.com>
Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
* [EFA] Add MC_EFA_CQ_THREADS env var and reduce idle CPU spin
Two changes to EFA transport CQ polling:
1. Add MC_EFA_CQ_THREADS environment variable to cap the number of CQ
polling threads. When running multiple EFA consumers (e.g. KV transfer
+ DeepEP all-to-all) in the same process, each creates threads per
context. This allows limiting contention.
2. Replace std::this_thread::yield() with sleep_for(10us) in the idle
path of workerThreadFunc. yield() on Linux compiles to sched_yield()
which busy-spins at 100% CPU when there is no CQ work, wasting cores
that could serve other EFA consumers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: move MC_EFA_CQ_THREADS to Environ singleton, revert yield change
Address reviewer feedback:
- Register MC_EFA_CQ_THREADS in Environ with GetInt (default 0 = unset)
- Use Environ::Get().GetEfaCqThreads() instead of raw std::getenv/stoull
- Revert yield() -> sleep_for() change (keep original yield behavior)
- Update comment to explain when/why the cap is useful
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add MC_EFA_CQ_THREADS documentation to EFA transport guide
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: change MC_EFA_CQ_THREADS default to 1 to reduce idle CPU spin
Benchmarks on p5.48xlarge show cap=1 reaches 99.93% of peak GPU-to-GPU
throughput (386.22 vs 386.48 GB/s) while freeing 31 cores from busy-spin.
Set MC_EFA_CQ_THREADS=0 to restore the legacy one-poller-per-context behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Ubuntu <ubuntu@ip-10-0-2-68.ec2.internal>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(transfer-engine): eliminate race condition in rePublishRpcMetaEntry
Remove the redundant storage_plugin_->remove() call before set().
All storage backends (HTTP PUT, Redis SET, Etcd put) have upsert
semantics, so remove-then-set creates a window where concurrent
get() returns empty / 404, causing transfer failures (-800).
Also change Json::UInt64 to Json::UInt for rpc_port to ensure
existing == desired comparison works correctly after JSON parse.
* fix: adjust eviction thread initial timing to prevent race in CI SSD tests
Start last_discard_time with an already-elapsed window so the first loop
iteration triggers DiscardExpiredProcessingReplicas immediately. Without
this, a task admitted shortly after thread startup can survive the first
reaper cycle and not be cleaned until ~2s later, causing promotion-on-hit
tests that sleep for 2s to flake.
* [TENT] Add configuration-driven transport selector
Add TransportSelector for flexible, configuration-driven transport selection
policy while maintaining full backward compatibility.
Key features:
- Configuration-driven transport selection via JSON policy rules
- Support for segment_type filtering, device allocation, and transport priority
- Legacy mode option (use_legacy_transport_selection) for exact original behavior
- Default policies match original hardcoded behavior
Changes:
- Add TransportSelector class with SelectionContext, SelectionPolicy, SelectionResult
- Integrate TransportSelector into TransferEngineImpl
- Add legacy mode support to preserve original code path
- Restore TaskInfo fields (xport_priority, failover_count) for backward compatibility
- Add max_failover_attempts configuration option
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [TENT] Add comprehensive unit tests for TransportSelector
Add transport_selector_test.cpp with test coverage for:
- Default policies matching original behavior (File/Memory segments)
- Transport type name parsing
- Legacy mode enable/disable
- Transport availability based on capabilities
- Priority offset for fallback scenarios
- Device mask handling
- NVLINK same-machine constraint
- ROCm memory type support
- GPU-to-GPU, CPU-to-CPU, CPU-to-GPU, GPU-to-CPU transfers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [TENT] Fix FakeTransport to use protected caps member
Fix compilation error by accessing Transport::caps (protected)
through helper methods instead of a separate public member.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix
* fix format issues
* Add priority-based rule
* Reformat
* Fix review comments
* Fix memory leak in endpoint_store_integration_test
When ibv_get_device_list returns a non-NULL list but num_devices == 0,
we need to call ibv_free_device_list before returning to avoid leaking
the allocated memory.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Trigger CI
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add missing mappings in gpu_vendor/maca.h:
- CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR -> mcMemHandleTypePosixFileDescriptor
- CU_POINTER_ATTRIBUTE_DEVICE_ORDINAL -> mcPointerAttributeDevice
These aliases complement #2227 and ensure full coverage of CUDA-like symbols
used in nvlink_allocator.cpp and related components when building with
-DUSE_MACA=ON.
Add the missing CUDA-like type alias in maca.h so nvlink_allocator.cpp
can compile when building with -DUSE_MACA=ON.
Co-authored-by: Cursor <cursoragent@cursor.com>
- mooncake-transfer-engine: add parentheses around && within ||, add
static_cast for narrowing, mark unused function [[maybe_unused]]
- mooncake-store: fix member reorder warnings, add std::ignore for
unused results, fix missing field initializers, mark unused variables
- mooncake-integration: fix sign-compare comparison, mark unused
functions [[maybe_unused]]
- All fixes are semantic-preserving (no behavior changes)
* Revise implementation
* Reformat
* fix fallback logic
* Add QoS APIs and docs
* Reformat
* Add QoS starvation prevention & bugfixes
* Reformat
* update docs
* remove tl_caller_id
* Fix worker distribution for multi-threaded submissions
When multiple threads submit slices simultaneously, they all start
distributing from worker 0, causing contention. Use thread_local
offset to distribute starting worker across threads, ensuring
each thread begins from a different worker.
Also rename submit_slices to next_worker_idx for clarity.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Introduce`USE_MLX5DV`CMake option and link against`libmlx5`when enabled
- Add`MC_MLX5_QP_UDP_SPORTS`
environment variable to specify comma-separated UDP source ports for ECMP/LAG
path diversification
- Add`MC_MLX5_QP_LAG_PORT_BALANCE`
environment variable to enable automatic QP distribution across bonded LAG ports
- Update`RdmaContext`to query and expose the number of LAG ports via
`mlx5dv_query_device`
- Implement QP modification logic in`RdmaEndPoint::doSetupConnection`
to apply UDP source port and LAG port pinning
- Extend configuration parsing and logging to support the new mlx5-specific
options
- Document the new runtime options in the design documentation
Signed-off-by: staryxchen <staryxchen@tencent.com>
* IntraNode NVLink async transfer
* Async intranode nvlink transfer
* Solve pytorch and Memcpy race condition by introducing sync event to make sure pytorch data prepared before transfer
---------
Co-authored-by: 百麒 <yaozhong.lyz@alibaba-inc.com>
* [CI/Build] Switch WITH_NVIDIA_PEERMEM to env variable
* [CI/Build] Switch WITH_NVIDIA_PEERMEM to runtime env variable
Instead of a cmake build-time option, check the WITH_NVIDIA_PEERMEM
environment variable at runtime in rdma_context.cpp and
rdma_transport.cpp to switch between ibv_reg_mr() and
ibv_reg_dmabuf_mr().
- Remove option(WITH_NVIDIA_PEERMEM) and add_compile_definitions() from
common.cmake (no more compile-time flag)
- Update CMakeLists.txt to use GPU toolkit presence instead of cmake var
for nvlink-allocator build condition
- Add withNvidiaPeermem() runtime helper reading WITH_NVIDIA_PEERMEM env
var; default false (dmabuf path, no nvidia-peermem required)
- Replace #if !defined(WITH_NVIDIA_PEERMEM) && defined(USE_CUDA) guards
with runtime if (!withNvidiaPeermem()) checks
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/05c94a1b-d4d6-4b44-be25-3b98d9b01f1b
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Add WITH_NVIDIA_PEERMEM to Environ; use Environ::Get() in rdma files
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/66e6cabb-c473-4a9d-9711-aebe468fcee2
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix linker error: link transfer_engine against mooncake_common for Environ::Get()
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/29d46e31-6dc1-4fee-be33-4a603537b827
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix linker error in Go CGO builds: add -lmooncake_common to build.sh scripts
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/ab73f098-d408-4ed0-95bb-77f9ce9f71ae
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Add USE_MACA to nvlink-allocator conditions to cover all GPU cases
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/251d2c35-d12a-43ed-9958-40526e0420d5
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Go CGO linker path: add mooncake-common/src to library search paths
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/5a91e4b9-660d-463b-a8f2-d3fb0118fc34
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Rust build.rs: add mooncake_common link and CUDA stubs search paths
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/e0d1e4fd-2cc5-4fe9-9268-5c205e3fc0f5
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Rust build.rs: remove CUDA stubs from search_dirs to prevent runtime libcuda.so.1 dep
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/cd938ceb-c822-4439-b617-03f065015d4c
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Rust build.rs: remove CUDA stubs from early rustc-link-search to prevent libcuda.so.1 runtime dep
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/055de577-7669-4039-a89a-d6066493e2d0
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix CI: create libcuda.so.1 symlink and set LD_LIBRARY_PATH for cargo test --lib
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/76131e72-de6f-485b-98ab-342b112f3968
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* [TENT] Refactor RDMA transport with unidirectional endpoint lifecycle
This commit aligns TENT RDMA transport with the source implementation,
focusing on endpoint lifecycle management, configuration consolidation,
and code organization improvements.
**Endpoint Lifecycle (Unidirectional)**
- Rename enum Status -> EndpointState to avoid ambiguity with mooncake:⛺:Status
- Define explicit state machine: EP_UNINIT -> EP_HANDSHAKING -> EP_READY -> EP_DESTROYING -> EP_DESTROYED
- Remove redundant active_/inactive_time_ members, use status_ for all state judgment
- Implement unidirectional lifecycle: endpoints never reset or reuse
- Add resetConnection() for marking failed endpoints for destruction
- Deprecate reset() to prevent accidental endpoint reuse
**Two-Phase QP Destruction**
- beginDestroy(): Mark endpoint as EP_DESTROYING, transition QPs to ERR state
- finishDestroy(): Wait for inflight WRs to drain, then destroy QPs
- Add destroy_start_time_ for timeout enforcement (30s default)
- Fix deconstructUnlocked() to maintain EP_DESTROYED state (no rollback to EP_UNINIT)
**Endpoint Store Cleanup**
- Unify remove() and removeRef() into single remove(RdmaEndPoint*) method
- Add terminal state checking in getOrInsert() - auto-remove and recreate
- Fix evictOne() to call beginDestroy() before moving to waiting_list
- Fix reclaim() to use finishDestroy() instead of getInflightSlices()
- Add waiting_list_len_ early return check in FIFO.reclaim()
**Configuration Management**
- Migrate PCIe Relaxed Ordering from environment variables to config
- Add backward compatibility mappings for legacy MC_* environment variables:
- MC_NUM_CQ_PER_CTX, MC_NUM_COMP_CHANNELS_PER_CTX, MC_IB_PORT
- MC_GID_INDEX, NCCL_IB_GID_INDEX, MC_MAX_CQE_PER_CTX
- MC_MAX_EP_PER_CTX, MC_NUM_QP_PER_EP, MC_MAX_SGE, MC_MAX_WR
- MC_MAX_INLINE, MC_PKEY_INDEX, MC_MTU, MC_IB_TC
- MC_IB_PCI_RELAXED_ORDERING, MC_WORKERS_PER_CTX
- MC_SLICE_SIZE, MC_RETRY_CNT, MC_DISABLE_GPU_DIRECT_RDMA
- Add RdmaTransport::config() public accessor for config-driven decisions
**Code Quality**
- Update state checks from CONNECTED -> EP_READY
- Simplify status checks by removing active_ dependency
- Improve logging for state transitions
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Reformat code
* Avoid use RdmaEndpoint::reset()
* Fix code issues
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add Hygon DCU/DTK and Iluvatar CoreX platform support
Add build system and runtime support for two CUDA-compatible domestic
accelerator platforms:
- Hygon DCU with DTK SDK (USE_HYGON, /opt/dtk/cuda/cuda-11/)
- Iluvatar CoreX SDK (USE_COREX, /usr/local/corex/)
Both platforms expose CUDA-compatible APIs, so the integration follows
the same pattern as existing CUDA-like platforms (MUSA, MACA): add the
new macros to all platform guard chains and register SDK paths in CMake.
---------
Co-authored-by: KarmaD7 <KarmaD7@users.noreply.github.com>
* [TENT] Fix batch getTransferStatus premature FAILED aggregation
Previously, one permanently-FAILED task would latch overall_status to
FAILED even while other tasks were still PENDING (mid-failover). This
caused lazyFreeBatch to teardown the batch while retries were in-flight.
Now the batch reports FAILED only when ALL tasks reach a terminal state
(success_tasks + failed_tasks == total_tasks). A task still in PENDING
(e.g. resubmitted on a secondary transport) keeps the batch in PENDING.
Add unit tests covering the new aggregation logic: FAILED+PENDING →
PENDING, FAILED+COMPLETED → FAILED, all COMPLETED → COMPLETED, and
derived-task skipping.
Signed-off-by: Yuxin Chen <grityxchen@gmail.com>
* [TENT] Replace hardcoded transport type tests with dynamic sentinel check
The old AllEnumValuesDistinct and SupportedCount tests hardcoded the
enum list and expected count, breaking whenever a new transport type
(like SUNRISE_LINK) was added. Replace them with UnspecIsSentinel
which verifies invariants independent of how many transports exist.
Signed-off-by: Yuxin Chen <grityxchen@gmail.com>
* style(test): reformat lambda expression in failover_test
- Adjust line break for lambda assignment to improve readability
Signed-off-by: Yuxin Chen <grityxchen@gmail.com>
* fix(transfer-engine): correct worst failure tracking in getTransferStatus
- Introduce severity-based comparison to ensure `worst_failure` reflects the most severe status, preventing overwrites with lower severity.
Signed-off-by: Yuxin Chen <grityxchen@gmail.com>
---------
Signed-off-by: Yuxin Chen <grityxchen@gmail.com>
Co-authored-by: Yuxin Chen <grityxchen@gmail.com>
* [TransferEngine][Integration] feat: add MACA/MetaX GPU support and fix RDMA dmabuf registration
- Fix MACA compatibility macros: correct CUdeviceptr alias, add missing
CUDA memory type and pointer attribute macros, implement inline
cuGetErrorString wrapper
- Fix RDMA dmabuf memory registration for GPU memory: use allocation
base address for cuMemGetHandleForAddressRange and compute proper
offset for ibv_reg_dmabuf_mr (fixes#1975, #1965)
- Add USE_MACA guard alongside existing USE_MLU/USE_CUDA guards for
dmabuf-based memory registration path
- Support remote_request_id in mooncake connector v1 for cross-request
KV cache transfer between prefiller and decoder
---------
Co-authored-by: zhangxin <zhangxin@zhangxins-MacBook-Air.local>
* [TransferEngine] Fix GPU dependency in transfer_engine_bench
Problem: transfer_engine_bench crashes (exit code 247) when running with
--use_vram=false in environments without GPU, even though it only uses
CPU memory (DRAM).
Root cause: freeMemoryPool() calls cudaPointerGetAttributes() with
checkCudaError(), which exits the program if CUDA fails.
Solution: Use graceful error handling like transfer engine core library
(memory_location.cpp). When CUDA query fails, assume CPU memory and
use numa_free().
Impact:
- Enables CPU-only RDMA bandwidth testing without GPU
- Consistent behavior with mooncake_client
- No impact on existing GPU-enabled scenarios
Test: Verified RDMA bandwidth testing works in CPU-only pods and
achieves 10+ GB/s throughput on 200G RDMA network.
Signed-off-by: jibxie <jibxie@ebay.com>
* [TransferEngine] Optimize memory deallocation logic in transfer_engine_bench
Check FLAGS_use_vram before calling cudaPointerGetAttributes to avoid
unnecessary CUDA calls when memory is explicitly allocated on CPU.
- When FLAGS_use_vram is false, memory is guaranteed to be allocated
via numa_alloc_onnode, so we can directly call numa_free without
checking CUDA pointer attributes
- This avoids confusing WARNING logs on systems without GPU when users
explicitly choose to use DRAM
- Change log level from WARNING to ERROR when FLAGS_use_vram is true
but cudaPointerGetAttributes fails, for consistency with
memory_location.cpp
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
---------
Signed-off-by: jibxie <jibxie@ebay.com>
Co-authored-by: jibxie <jibxie@ebay.com>
Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
* [TENT] fallback to per-task cudaMemcpyAsync when driver lacks batch support
Fixes the case where containers ship CUDA Toolkit 12.8 but the host
driver only supports 12.2
* remove unconditionally overwrites the err
* style: format nvlink_transport.cpp with clang-format-20
---------
Co-authored-by: jinke15 <jinke15@jd.com>
This commit introduces a standalone maca_transport following the same
pattern as hip_transport, instead of polluting nvlink_transport with
MACA-specific workarounds.
Key points:
- Sync mcMemcpy with device-context guard (save/restore device before
each copy to avoid mcErrorContextIsDestroyed / SIGSEGV).
- Base-pointer registration via cuMemGetAddressRange for correct IPC
handle semantics with framework caching allocators.
- IPC-only path; fabric memory is not reliably supported on MACA 3.5.3.
- P2P access enabled in constructor with original device restoration.
Glue changes:
- multi_transport.cpp: register "maca" protocol
- transfer_engine_impl.cpp: auto-install maca transport under USE_MACA
- transfer_metadata.cpp: add "maca" to encode/decode protocol whitelist
- transfer_engine_validator.cpp: support --protocol=maca
- maca.h: add missing CU_POINTER_ATTRIBUTE_*, cuGetErrorString macros
Verified on Metax C500 (2-GPU) with transfer_engine_validator:
Data validation passed, throughput ~6.9 GB/s
* Add reference counting to RdmaTask to prevent UAF
- Convert RdmaSubBatch::task_list from value to pointer storage
- Add atomic reference counting to RdmaTask with Slab allocator integration
- Properly dereference tasks in freeSubBatch cleanup path
- Each slice holds a reference to its parent task
Author: Feng Ren <alogfans@gmail.com>
* Add paired ref/deref
* remove task->ref_count assignment
Mooncake's fi_getinfo() requests FI_VERSION(1, 14). The EFA provider's efa_rdm_get_use_device_rdma() keeps a legacy compatibility branch for callers on API < 1.18 that hardcodes the default for FI_EFA_USE_DEVICE_RDMA based on vendor_part_id: efa0/efa1 → false, everything newer → true. Under that legacy branch we silently disabled device RDMA on Nitro v4 EFA hardware (p5.48xlarge, p5e.48xlarge — vendor_part_id 0xefa1) while leaving it enabled on Nitro v5+ (p5en and later). Cross-node transfer_engine_bench reproduces the split: p5en runs fine out of the box, p5/p5e segfault inside libfabric.so during fi_cq_read once the handshake wave finishes and the first real fi_writes start. The crashing stack has a concurrent fi_av_insert in flight on the same EfaContext, and the crashing memcpy is in the provider's emulated-RDMA CQE reconstruction path — i.e. a thread-safety regression in libfabric 2.4.0's emulated RDMA data path that only ever runs when device RDMA is off.
Request API 1.18. With the 1.18+ code path the default becomes hw_support (unconditionally true on every EFA hardware that supports RDMA, which is every Mooncake target platform starting from p4d), so p5/p5e pick up the same device-RDMA default p5en already has, the emulated path is never entered, and the segfault is gone. Applications that still want the emulated path can opt out with FI_EFA_USE_DEVICE_RDMA=0. 1.18 is from March 2023 and is the oldest libfabric shipped with EFA installer 1.26 and up; every Mooncake EFA deployment today runs libfabric ≥ 2.0, so bumping the requested API costs nothing in compatibility.
Verified by transfer_engine_bench on two p5.48xlarge nodes (libfabric 2.4.0amzn1.0, 32 EFA NICs, 8 × H100 per node):
- Before this patch, no env: target SIGSEGV inside libfabric.so on first real transfer.
- Before this patch, FI_EFA_USE_DEVICE_RDMA=1: 377.74 GB/s, stable.
- After this patch, no env: 377.93 GB/s, stable — matches the explicit env case.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Say we have 3 transfer engine instances T0, T1, T2, in the following
case with P2PHANDSHAKE, they will form a circular dead lock:
T0.listener is handling connection request from T1:
-> setupConnectionsByPassive()
-> getSegmentDescByName(T1)
-> exchangeMetadata(T1)
-> wait for T1.listener processing
T1.listener is handling connection request from T2:
-> setupConnectionsByPassive()
-> getSegmentDescByName(T2)
-> exchangeMetadata(T2)
-> wait for T2.listener processing
T2.listener is handling connection request from T0:
-> setupConnectionsByPassive()
-> getSegmentDescByName(T0)
-> exchangeMetadata(T0)
-> wait for T0.listener processing
T0 -> T1 -> T2 -> T0
To fix this, we can remove the calling to getSegmentDescByName
completely from the connection establish process, and exchange
necessary connection information through HandShakeDesc. This can
also significantly simplify the connection process.
Signed-off-by: Chen Jinlong <chenjinlong.cjl@alibaba-inc.com>
Share allocator build scaffolding across nvlink and ubshmem backends.
- expose a common allocator probe/malloc/free ABI while keeping legacy symbols
- reuse shared Python helper logic for allocator loading and probing
- centralize allocator CMake and shell build helpers for packaging and builds
Xun Sun