* [TE] Harden TCP transport: validate remote addresses, fix idle cleanup, add TCP_NODELAY
Fixes#2313
1. Security: ServerSession now validates remote-supplied memory addresses
against registered local buffers before use, preventing arbitrary
memory read/write from malicious peers.
2. Performance: Set TCP_NODELAY on all server-accept and client-connect
paths to eliminate Nagle-induced latency on small control messages.
3. Correctness: Call io_context.restart() after exception in worker loop
to prevent busy-spin when io_context enters stopped state.
4. Correctness: Fix cleanupIdleConnections to scan the full deque instead
of only the back, so idle connections anywhere in the pool get cleaned.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [TE] Fix clang-format violations in TCP transport
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [TE] Fix lock leak, missing transport_, and empty entries UB
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix pre-existing clang-format violations in dump.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Fix freeaddrinfo leak on ERR_MALFORMED_JSON early returns in
sendNotify, sendProbe, send, and exchangeMetadata.
- Add null check for getSegmentDescByID in sendNotifyByID to
prevent null pointer dereference on invalid segment ID.
- Guard readString against zero-length network input to prevent
OOB access on empty buffer.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Remove exit(EXIT_FAILURE) on invalid MC_MTU so the process
continues with the default IBV_MTU_4096 instead of crashing.
- Wrap MC_HANDSHAKE_LISTEN_BACKLOG std::stoi in try-catch to
match MC_PKEY_INDEX / MC_IB_TC pattern and prevent crash on
non-numeric input.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [TE] Fix data race, double-close, uninit, off-by-one in RDMA transport
- Remove 'static' from access_rights in registerLocalMemoryInternal
to eliminate data race under concurrent registerLocalMemoryBatch.
- Add event_fd_ = -1 after close(event_fd_) on 3 error paths in
RdmaContext::construct() to prevent double-close in destructor.
- Value-initialize comp_channel_ array to zero so partial-failure
cleanup in deconstruct() sees nullptr instead of garbage pointers.
- Fix off-by-one: change > to >= in doSetupConnection bounds check
to prevent OOB access when qp_index equals qp_list_.size().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [TE] Value-initialize wr_depth_list_ to fix sibling uninit bug
Address review: wr_depth_list_ has the same uninitialized-array
bug as comp_channel_. Partial QP creation failure leaves garbage
values that corrupt the CQ outstanding counter in destructor.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Init wr_depth_list_ to nullptr, guard deconstructLocked()
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(transfer-engine): extract Device API and IBGDA transport layer
Move IBGDA files from mooncake-ep to mooncake-transfer-engine (git mv):
- 6 headers: mooncake_ibgda/ → transport/device/ibgda/
- 1 source: mlx5gda.cpp → transport/device/mlx5gda.cpp
Add new Device API layer under transport/device/:
- device_transport.h: P2pTransport + RdmaTransport interfaces
- device_ops.cuh: DeviceOps function pointer table (bottom IR)
- comm_device.cuh, p2p_device.cuh, ibgda_device.cuh: device contexts
- cuda_ops.cuh, musa_ops.cuh: platform DeviceOps implementations
- ibgda_device_transport.cpp: RdmaTransport IBGDA implementation
- p2p_device_transport.cpp: P2pTransport NVLink/MTLink implementation
Modify transfer-engine:
- transfer_engine.h/impl: add getOrCreateP2pTransport/RdmaTransport
- gpu_vendor/musa.h: add MUSA API aliases
- Build: link mlx5, add device/ subdirectory
Minimal EP changes (include paths + build system only):
- Update include paths to new transport/device/ibgda/ location
- Remove mlx5gda.cpp from EP sources
- Forward EP_USE_MUSA env var in BuildEpExt.cmake
Root CMakeLists.txt: guard CUDAToolkit with USE_CUDA
* feat(example): add device_transport_example for P2P Device API
Two-rank example demonstrating the full Device API lifecycle:
- Host side: P2pTransport for IPC handle exchange and peer mapping
- Device side: CommCtx + mc_route_put + mc_signal for GPU-initiated
P2P data transfer and notification
Uses file-based IPC handle exchange (no external dependencies).
Requires 2 GPUs with P2P access (NVLink or PCIe).
* fix(example): handle missing TORCH_CUDA_ARCH_LIST
* fix(example): convert torch arch format to CMake CUDA format
* fix(example): construct CommCtx manually on host side
* fix(example): add using namespace mooncake::device in kernels
* fix(example): pass CommCtx by value, add barrier before cleanup
- Pass CommCtx by value to kernel (CUDA copies to param space) instead of
dereferencing a device pointer on host (caused segfault).
- Add file-based barrier so rank 0 waits for rank 1 to finish before
freeing its GDR buffer (IPC handle only valid while allocation exists).
- Change default metadata_server to P2PHANDSHAKE (no etcd dependency).
* fix: address review feedback — fence ordering, error checks, QP leak
- musa_ops.cuh: fix fence ordering for acquire/release semantics
(fence after load for acquire, before store/atomic for release)
- ibgda_device_transport.cpp: check cudaMalloc return values,
add num_qps >= num_ranks guard, destroy QP on rst2init failure
- p2p_device_transport.cpp: add device_count > 0 guard
- device_transport_example.cu: validate kDataBytes % 16 == 0
and kDataBytes <= kSignalWordOffset
* style: apply clang-format to Device API files
* fix: guard device transport code with USE_CUDA/USE_MUSA macros
The device transport accessors (getOrCreateP2pTransport,
getOrCreateRdmaTransport) and their member variables were not guarded
by USE_CUDA/USE_MUSA preprocessor macros. When building with
USE_CUDA=OFF (the default), the device transport source files aren't
compiled but the headers and implementations still reference them,
causing linker errors in CI build-flags and build jobs.
* fix: set CMAKE_CUDA_STANDARD 20 and make ibgda PUBLIC
- common.cmake: add CMAKE_CUDA_STANDARD 20 so nvcc compiles host code
in C++20 mode, matching CMAKE_CXX_STANDARD. Fixes "starts_with is
not a member of std::string" when compiling .cu files that indirectly
include common.h.
- transport/CMakeLists.txt: change ibgda from PRIVATE to PUBLIC so
mlx5gda_* symbols are visible to downstream consumers (Go p2p store
via transfer_engine). Fixes undefined reference errors for
mlx5gda_destroy_qp, mlx5dv_devx_umem_reg, etc.
* fix: compile mlx5gda into device_transport and link mlx5 for Go consumers
The previous attempt (PUBLIC ibgda) did not work because the Go p2p-store
and mooncake-store binaries link libtransfer_engine.a via hand-written cgo
ldflags, which bypass CMake's target_link_libraries propagation entirely.
- transport/device: compile mlx5gda.cpp directly into the device_transport
OBJECT library (like every other transport module) instead of a separate
ibgda STATIC lib, so mlx5gda_* symbols flow into libtransfer_engine.a and
are visible to all consumers regardless of how they link.
- transport: link libmlx5 (PUBLIC) since ibgda_device_transport.cpp /
mlx5gda.cpp call mlx5dv_devx_* / mlx5dv_init_obj directly.
- p2p-store/build.sh, mooncake-store/go/build.sh, ci.yml: add -lmlx5 to the
hand-written cgo ldflags so the DevX symbols resolve.
- example: set CUDA_STANDARD 20 on device_transport_example so nvcc compiles
common.h (std::string::starts_with) in C++20 mode.
* fix: CUDA_EXTENSIONS OFF for example, add -lm for Go consumers
Follow-up to compiling mlx5gda.cpp into device_transport:
- example: nvcc has no gnu++20 dialect, so CUDA_STANDARD 20 with the default
CUDA_EXTENSIONS=ON fails at CMake generate ("does not know the compile
flags"). Set CUDA_EXTENSIONS OFF to request plain -std=c++20.
- p2p-store/build.sh, mooncake-store/go/build.sh: mlx5gda.cpp uses log2/ceil
(<cmath>); now that its object lives in libtransfer_engine.a, the hand-
written cgo ldflags need -lm to resolve log2@GLIBC_2.29. (ci.yml already
had -lm.)
* fix(ci): gate Device API GPU example off by default, link mlx5 for Rust
The Docker build failed at CMake generate because device_transport_example
needs the CUDA20 dialect (transfer_engine.h -> common.h uses C++20
std::string::starts_with), which the older CMake in the CI image cannot
enable. CUDA_EXTENSIONS OFF did not help since the limitation is the CMake
version, not the dialect flavor. Gate this manual, 2-GPU example behind a
new BUILD_DEVICE_TRANSPORT_EXAMPLE option (default OFF) so the default build
no longer requires CUDA20.
Also link mlx5 in the mooncake-store Rust build script: the IBGDA device
transport (mlx5 DevX) is now compiled into transfer_engine, so the Rust
test link step needs -lmlx5 to resolve mlx5dv_devx_* symbols.
* [Common] Harden Environ parsing, fix opendir leak, support .yml config
Fixes#2315
1. Fix: Replace opendir() with stat()+S_ISDIR() in config.cpp to fix
DIR* handle leak.
2. Robustness: Replace atoi() with strtol() in Environ::GetInt with
endptr/errno validation. Invalid values now fall back to default
with a warning instead of silently returning 0.
3. Robustness: Add leading '-' rejection in Environ::GetSizeT to
prevent strtoull negative wrapping (e.g. MC_SLICE_SIZE=-1 yielding
ULLONG_MAX).
4. Feature: Support .yml extension in DefaultConfig::Load().
5. Tests: Add 26 unit tests for Environ::GetInt/GetSizeT/GetBool/
GetString covering valid, invalid, missing, overflow, negative,
and trailing garbage inputs. Tests call the real production code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Common] Use strtoll instead of strtoull in GetSizeT for robust negative handling
Address Gemini review: val[0]=='-' check missed leading-whitespace
cases like " -1". Using strtoll catches negatives regardless of
whitespace, and also guards against 32-bit SIZE_MAX truncation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Common] Address review: restore comments, use std::filesystem
- Restore helper method comments stripped when moving to public
- Switch from stat()+S_ISDIR() to std::filesystem::is_directory()
per reviewer suggestion (C++20 project, already used elsewhere)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Common] Fix clang-format violation in environ.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Common] Address Copilot review: non-throwing is_directory, fix stray PUBLIC
- Use std::error_code overload of std::filesystem::is_directory to
avoid throwing on permission errors (EACCES). Matches original
opendir() non-throwing behavior.
- Remove pre-existing stray PUBLIC keyword in tests/CMakeLists.txt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1. multi_transport.cpp: Delete Transport object when install() fails.
The raw pointer was leaked on the error return path.
2. transfer_engine_c.cpp: Add null check after malloc and early return
when size is 0 in getNotifsFromEngine(). Prevents null dereference
in memset when malloc fails, and avoids implementation-defined
behavior of malloc(0).
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Cache engine and target in QueryBatch to avoid per-poll metadata lookups,
propagate route failures only to batches already in pending_batches with
a single disconnect, and treat NOT_CONNECTED as success on auto_connect
disconnect.
Co-authored-by: Developer user <youxiao@huawei.com>
* fix: route EFA same-process loopback transfers through local copy
The EFA provider's SHM intra-node path performs a host memcpy into
FI_HMEM_CUDA device buffers and segfaults on the first same-host
transfer (loopback self-transfer), e.g. checkpoint-engine P2P weight
update on a single TP=8 node. See ofiwg/libfabric#12328.
Detect same-process self-loopback in EfaContext::submitPostSend (peer
NIC path equals our own nicPath(), whose server_name embeds the
per-process RPC port, so the match guarantees the peer is this very
process on this device) and satisfy the copy locally with a GPU-aware
cudaMemcpy (cudaMemcpyDefault), bypassing EFA entirely. Same-host
cross-process peers carry a different port, never match, and still go
through EFA.
The copy direction honors the slice opcode, mirroring fi_read/fi_write:
WRITE copies source_addr -> dest_addr, READ copies dest_addr ->
source_addr (the two are distinct local buffers, so it is not a
symmetric self-copy).
This mirrors how the RDMA transport already treats loopback as a
special case (rdma_endpoint.cpp self-connected QP); RDMA relies on NIC
hardware loopback and is unaffected by the libfabric SHM bug.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: add EFA GPU loopback test for FI_HMEM_CUDA same-host transfers
Add efa_gpu_loopback_test.cpp, the CUDA-device-memory counterpart of
efa_transport_test.cpp (which only covers host/numa loopback). It
reproduces the EFA SHM intra-node segfault on FI_HMEM_CUDA buffers
(ofiwg/libfabric#12328) and validates EfaContext::tryLoopbackCopy:
* GpuLoopbackWrite — same-host GPU WRITE must not crash.
* GpuLoopbackWriteThenRead — WRITE then READ with byte-accurate
verification, exercising both copy directions.
* GpuLoopbackMultiWrite — batched GPU writes through the per-slice
loopback short-circuit.
Self-skips when no EFA device or no CUDA GPU is present. Registered
under `USE_EFA AND USE_CUDA`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* efa: log fabric name and document same-host GPU loopback segfault
Add fabric_attr->name to the EFA device init log so the active fabric
(efa-rdm vs efa-direct) is visible at runtime — both share the same
provider name, FI_EP_RDM type, and <device>-rdm domain name, so the
fabric name is the only field that distinguishes them.
Revise the "Single-host loopback" doc section:
- Correct the same-host fast-path attribution: the memcpy fast path is
supplied by the SHM provider (FI_EFA_ENABLE_SHM_TRANSFER, default on),
not by FI_EFA_USE_DEVICE_RDMA. Confirmed at runtime: a default
(device-RDMA-enabled) config still reports "Opened fabric: shm".
- Add a warning that the default SHM path host-memcpy's into FI_HMEM_CUDA
destinations and segfaults on GPU buffers (ofiwg/libfabric#12328);
document the same-process tryLoopbackCopy short-circuit and the
FI_EFA_ENABLE_SHM_TRANSFER=0 workaround for cross-process GPU peers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Separate the tebench benchmark from the TENT build so it can be compiled
with only the classic Transfer Engine backend.
- Gate the benchmark subdirectory solely on BUILD_BENCHMARK instead of
also requiring USE_TENT.
- Drop tent_backend.cpp from the sources and only link tent_link_group /
define USE_TENT when USE_TENT is enabled; otherwise expose just the
tent/include header path for the header-only helpers used by the
classic backend.
- Guard the TENT backend include and runner in main.cpp, returning an
error when a TENT-only backend is requested in a non-TENT build.
The TCP transport previously hardcoded a 64KB (65536 bytes) slice size
for splitting large transfers into socket read/write operations. This
commit makes it configurable via the MC_TCP_SLICE_SIZE environment
variable, consistent with the RDMA transport's MC_SLICE_SIZE naming.
Usage: export MC_TCP_SLICE_SIZE=1048576 # 1MB slices
Default: 65536 (64KB, unchanged)
* [TENT] Add policy name binding to transport selector
Add ability to bind a request to a specific transport policy by name,
making the policy's "name" field in configuration actually useful.
Changes:
- Add optional `policy_name` field to Request struct (types.h)
- Add optional `policy_name` field to SelectionContext (transport_selector.h)
- Modify matchesPolicy() to prioritize exact policy name matching
when context.policy_name is specified
- Pass policy_name from request to context in transfer_engine_impl.cpp
When a request specifies policy_name, the selector will only match
the policy with that exact name, ignoring other matching conditions
(segment_type, priority, memory_pattern, etc.).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reformat
* Update mooncake-transfer-engine/tent/src/runtime/transport_selector.cpp
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Reformat
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update intraNode nvlink transport from MemcpuAsync to BatchAsync
* Code format update for intraNode nvlink
* Change addr to base_addr for register and unregister
---------
Co-authored-by: 百麒 <yaozhong.lyz@alibaba-inc.com>
* Initial plan
* Add MC_RDMA_BIND_ADDRESS support for dual-NIC P2PHANDSHAKE setups
In dual-NIC environments where TCP and RDMA use separate interfaces,
P2PHANDSHAKE mode previously required using a single IP for both
TCP handshake and RDMA NIC paths, causing conflicts.
This change adds MC_RDMA_BIND_ADDRESS env var support:
- When set, RDMA NIC paths use the RDMA-reachable IP
- TCP P2P routing continues using the local_server_name IP
- Segment descriptors carry rdma_server_name for consistent NIC
path construction on both sides
- P2P metadata exchange caches RDMA->TCP address mapping so
subsequent handshakes resolve to TCP-routable addresses
* Changes before error encountered
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/fc2826eb-a0ae-450f-b1f4-4ab94269d97a
* Apply dual-NIC (MC_RDMA_BIND_ADDRESS) support to TENT transport and update Chinese docs
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* [TransferEngine][ROCm] Add HIP dmabuf MR registration for AMD GPUs
Fixes#751.
Adds a parallel `#elif defined(USE_HIP)` branch in
RdmaContext::registerMemoryRegionInternal that mirrors the existing CUDA
dmabuf path (added by #704) using ROCm's `hsa_amd_portable_export_dmabuf()`
instead of `cuMemGetHandleForAddressRange(...DMA_BUF_FD...)`. This lets
Mooncake register AMD GPU memory for RDMA without requiring an
nvidia-peermem-equivalent kernel module — the path UCX's ROCm backend
(uct/rocm/base/rocm_base.c) already uses successfully.
Same host-vs-device split as the CUDA branch: `hipPointerGetAttributes`
detects host memory and falls back to `ibv_reg_mr`; device/managed memory
goes through the dmabuf path. `hipMemGetAddressRange` is used to get the
true allocation base because `addr` may sit at an offset within a larger
hipMalloc block (caching allocators pack tensors).
CMake: added `hsa-runtime64` to the HIP link line in
mooncake-transfer-engine/src/CMakeLists.txt.
Validation:
- Standalone dmabuf probe verified PASS on:
* AMD MI355X (gfx950) + Pensando ionic + ROCm 7.2.2
* AMD MI300X (gfx942) + Broadcom Thor2 (bnxt_re) + ROCm 7.0.2
Probe source + container recipe:
https://github.com/andyluo7/dynamo/blob/amd-poc-consumer-polish/amd-mi355x-poc/advanced/debug-probes/dmabuf_register_probe.cpp
- Standalone compile check confirms all HIP/HSA/ibverbs symbols in the
new branch resolve and link cleanly with hsa-runtime64 + libibverbs.
End-to-end SGLang+Mooncake disagg validation (T3) on MI355X+ionic will
follow in a comment once a full Mooncake build with submodules completes.
CC @misterwilliam @stmatengss @alogfans (active on #751)
Closes#751
---------
Signed-off-by: Andy Luo <anluo@amd.com>
Signed-off-by: Andy Luo <andy.luo@amd.com>
Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
* [EFA] Add MC_EFA_CQ_THREADS env var and reduce idle CPU spin
Two changes to EFA transport CQ polling:
1. Add MC_EFA_CQ_THREADS environment variable to cap the number of CQ
polling threads. When running multiple EFA consumers (e.g. KV transfer
+ DeepEP all-to-all) in the same process, each creates threads per
context. This allows limiting contention.
2. Replace std::this_thread::yield() with sleep_for(10us) in the idle
path of workerThreadFunc. yield() on Linux compiles to sched_yield()
which busy-spins at 100% CPU when there is no CQ work, wasting cores
that could serve other EFA consumers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: move MC_EFA_CQ_THREADS to Environ singleton, revert yield change
Address reviewer feedback:
- Register MC_EFA_CQ_THREADS in Environ with GetInt (default 0 = unset)
- Use Environ::Get().GetEfaCqThreads() instead of raw std::getenv/stoull
- Revert yield() -> sleep_for() change (keep original yield behavior)
- Update comment to explain when/why the cap is useful
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add MC_EFA_CQ_THREADS documentation to EFA transport guide
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: change MC_EFA_CQ_THREADS default to 1 to reduce idle CPU spin
Benchmarks on p5.48xlarge show cap=1 reaches 99.93% of peak GPU-to-GPU
throughput (386.22 vs 386.48 GB/s) while freeing 31 cores from busy-spin.
Set MC_EFA_CQ_THREADS=0 to restore the legacy one-poller-per-context behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Ubuntu <ubuntu@ip-10-0-2-68.ec2.internal>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(transfer-engine): eliminate race condition in rePublishRpcMetaEntry
Remove the redundant storage_plugin_->remove() call before set().
All storage backends (HTTP PUT, Redis SET, Etcd put) have upsert
semantics, so remove-then-set creates a window where concurrent
get() returns empty / 404, causing transfer failures (-800).
Also change Json::UInt64 to Json::UInt for rpc_port to ensure
existing == desired comparison works correctly after JSON parse.
* fix: adjust eviction thread initial timing to prevent race in CI SSD tests
Start last_discard_time with an already-elapsed window so the first loop
iteration triggers DiscardExpiredProcessingReplicas immediately. Without
this, a task admitted shortly after thread startup can survive the first
reaper cycle and not be cleaned until ~2s later, causing promotion-on-hit
tests that sleep for 2s to flake.
* [TENT] Add configuration-driven transport selector
Add TransportSelector for flexible, configuration-driven transport selection
policy while maintaining full backward compatibility.
Key features:
- Configuration-driven transport selection via JSON policy rules
- Support for segment_type filtering, device allocation, and transport priority
- Legacy mode option (use_legacy_transport_selection) for exact original behavior
- Default policies match original hardcoded behavior
Changes:
- Add TransportSelector class with SelectionContext, SelectionPolicy, SelectionResult
- Integrate TransportSelector into TransferEngineImpl
- Add legacy mode support to preserve original code path
- Restore TaskInfo fields (xport_priority, failover_count) for backward compatibility
- Add max_failover_attempts configuration option
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [TENT] Add comprehensive unit tests for TransportSelector
Add transport_selector_test.cpp with test coverage for:
- Default policies matching original behavior (File/Memory segments)
- Transport type name parsing
- Legacy mode enable/disable
- Transport availability based on capabilities
- Priority offset for fallback scenarios
- Device mask handling
- NVLINK same-machine constraint
- ROCm memory type support
- GPU-to-GPU, CPU-to-CPU, CPU-to-GPU, GPU-to-CPU transfers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [TENT] Fix FakeTransport to use protected caps member
Fix compilation error by accessing Transport::caps (protected)
through helper methods instead of a separate public member.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix
* fix format issues
* Add priority-based rule
* Reformat
* Fix review comments
* Fix memory leak in endpoint_store_integration_test
When ibv_get_device_list returns a non-NULL list but num_devices == 0,
we need to call ibv_free_device_list before returning to avoid leaking
the allocated memory.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Trigger CI
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add missing mappings in gpu_vendor/maca.h:
- CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR -> mcMemHandleTypePosixFileDescriptor
- CU_POINTER_ATTRIBUTE_DEVICE_ORDINAL -> mcPointerAttributeDevice
These aliases complement #2227 and ensure full coverage of CUDA-like symbols
used in nvlink_allocator.cpp and related components when building with
-DUSE_MACA=ON.
Add the missing CUDA-like type alias in maca.h so nvlink_allocator.cpp
can compile when building with -DUSE_MACA=ON.
Co-authored-by: Cursor <cursoragent@cursor.com>
- mooncake-transfer-engine: add parentheses around && within ||, add
static_cast for narrowing, mark unused function [[maybe_unused]]
- mooncake-store: fix member reorder warnings, add std::ignore for
unused results, fix missing field initializers, mark unused variables
- mooncake-integration: fix sign-compare comparison, mark unused
functions [[maybe_unused]]
- All fixes are semantic-preserving (no behavior changes)
* Revise implementation
* Reformat
* fix fallback logic
* Add QoS APIs and docs
* Reformat
* Add QoS starvation prevention & bugfixes
* Reformat
* update docs
* remove tl_caller_id
* Fix worker distribution for multi-threaded submissions
When multiple threads submit slices simultaneously, they all start
distributing from worker 0, causing contention. Use thread_local
offset to distribute starting worker across threads, ensuring
each thread begins from a different worker.
Also rename submit_slices to next_worker_idx for clarity.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Introduce`USE_MLX5DV`CMake option and link against`libmlx5`when enabled
- Add`MC_MLX5_QP_UDP_SPORTS`
environment variable to specify comma-separated UDP source ports for ECMP/LAG
path diversification
- Add`MC_MLX5_QP_LAG_PORT_BALANCE`
environment variable to enable automatic QP distribution across bonded LAG ports
- Update`RdmaContext`to query and expose the number of LAG ports via
`mlx5dv_query_device`
- Implement QP modification logic in`RdmaEndPoint::doSetupConnection`
to apply UDP source port and LAG port pinning
- Extend configuration parsing and logging to support the new mlx5-specific
options
- Document the new runtime options in the design documentation
Signed-off-by: staryxchen <staryxchen@tencent.com>
* IntraNode NVLink async transfer
* Async intranode nvlink transfer
* Solve pytorch and Memcpy race condition by introducing sync event to make sure pytorch data prepared before transfer
---------
Co-authored-by: 百麒 <yaozhong.lyz@alibaba-inc.com>
* [CI/Build] Switch WITH_NVIDIA_PEERMEM to env variable
* [CI/Build] Switch WITH_NVIDIA_PEERMEM to runtime env variable
Instead of a cmake build-time option, check the WITH_NVIDIA_PEERMEM
environment variable at runtime in rdma_context.cpp and
rdma_transport.cpp to switch between ibv_reg_mr() and
ibv_reg_dmabuf_mr().
- Remove option(WITH_NVIDIA_PEERMEM) and add_compile_definitions() from
common.cmake (no more compile-time flag)
- Update CMakeLists.txt to use GPU toolkit presence instead of cmake var
for nvlink-allocator build condition
- Add withNvidiaPeermem() runtime helper reading WITH_NVIDIA_PEERMEM env
var; default false (dmabuf path, no nvidia-peermem required)
- Replace #if !defined(WITH_NVIDIA_PEERMEM) && defined(USE_CUDA) guards
with runtime if (!withNvidiaPeermem()) checks
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/05c94a1b-d4d6-4b44-be25-3b98d9b01f1b
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Add WITH_NVIDIA_PEERMEM to Environ; use Environ::Get() in rdma files
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/66e6cabb-c473-4a9d-9711-aebe468fcee2
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix linker error: link transfer_engine against mooncake_common for Environ::Get()
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/29d46e31-6dc1-4fee-be33-4a603537b827
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix linker error in Go CGO builds: add -lmooncake_common to build.sh scripts
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/ab73f098-d408-4ed0-95bb-77f9ce9f71ae
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Add USE_MACA to nvlink-allocator conditions to cover all GPU cases
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/251d2c35-d12a-43ed-9958-40526e0420d5
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Go CGO linker path: add mooncake-common/src to library search paths
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/5a91e4b9-660d-463b-a8f2-d3fb0118fc34
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Rust build.rs: add mooncake_common link and CUDA stubs search paths
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/e0d1e4fd-2cc5-4fe9-9268-5c205e3fc0f5
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Rust build.rs: remove CUDA stubs from search_dirs to prevent runtime libcuda.so.1 dep
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/cd938ceb-c822-4439-b617-03f065015d4c
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Rust build.rs: remove CUDA stubs from early rustc-link-search to prevent libcuda.so.1 runtime dep
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/055de577-7669-4039-a89a-d6066493e2d0
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix CI: create libcuda.so.1 symlink and set LD_LIBRARY_PATH for cargo test --lib
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/76131e72-de6f-485b-98ab-342b112f3968
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* [TENT] Refactor RDMA transport with unidirectional endpoint lifecycle
This commit aligns TENT RDMA transport with the source implementation,
focusing on endpoint lifecycle management, configuration consolidation,
and code organization improvements.
**Endpoint Lifecycle (Unidirectional)**
- Rename enum Status -> EndpointState to avoid ambiguity with mooncake:⛺:Status
- Define explicit state machine: EP_UNINIT -> EP_HANDSHAKING -> EP_READY -> EP_DESTROYING -> EP_DESTROYED
- Remove redundant active_/inactive_time_ members, use status_ for all state judgment
- Implement unidirectional lifecycle: endpoints never reset or reuse
- Add resetConnection() for marking failed endpoints for destruction
- Deprecate reset() to prevent accidental endpoint reuse
**Two-Phase QP Destruction**
- beginDestroy(): Mark endpoint as EP_DESTROYING, transition QPs to ERR state
- finishDestroy(): Wait for inflight WRs to drain, then destroy QPs
- Add destroy_start_time_ for timeout enforcement (30s default)
- Fix deconstructUnlocked() to maintain EP_DESTROYED state (no rollback to EP_UNINIT)
**Endpoint Store Cleanup**
- Unify remove() and removeRef() into single remove(RdmaEndPoint*) method
- Add terminal state checking in getOrInsert() - auto-remove and recreate
- Fix evictOne() to call beginDestroy() before moving to waiting_list
- Fix reclaim() to use finishDestroy() instead of getInflightSlices()
- Add waiting_list_len_ early return check in FIFO.reclaim()
**Configuration Management**
- Migrate PCIe Relaxed Ordering from environment variables to config
- Add backward compatibility mappings for legacy MC_* environment variables:
- MC_NUM_CQ_PER_CTX, MC_NUM_COMP_CHANNELS_PER_CTX, MC_IB_PORT
- MC_GID_INDEX, NCCL_IB_GID_INDEX, MC_MAX_CQE_PER_CTX
- MC_MAX_EP_PER_CTX, MC_NUM_QP_PER_EP, MC_MAX_SGE, MC_MAX_WR
- MC_MAX_INLINE, MC_PKEY_INDEX, MC_MTU, MC_IB_TC
- MC_IB_PCI_RELAXED_ORDERING, MC_WORKERS_PER_CTX
- MC_SLICE_SIZE, MC_RETRY_CNT, MC_DISABLE_GPU_DIRECT_RDMA
- Add RdmaTransport::config() public accessor for config-driven decisions
**Code Quality**
- Update state checks from CONNECTED -> EP_READY
- Simplify status checks by removing active_ dependency
- Improve logging for state transitions
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Reformat code
* Avoid use RdmaEndpoint::reset()
* Fix code issues
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add Hygon DCU/DTK and Iluvatar CoreX platform support
Add build system and runtime support for two CUDA-compatible domestic
accelerator platforms:
- Hygon DCU with DTK SDK (USE_HYGON, /opt/dtk/cuda/cuda-11/)
- Iluvatar CoreX SDK (USE_COREX, /usr/local/corex/)
Both platforms expose CUDA-compatible APIs, so the integration follows
the same pattern as existing CUDA-like platforms (MUSA, MACA): add the
new macros to all platform guard chains and register SDK paths in CMake.
---------
Co-authored-by: KarmaD7 <KarmaD7@users.noreply.github.com>
* [TENT] Fix batch getTransferStatus premature FAILED aggregation
Previously, one permanently-FAILED task would latch overall_status to
FAILED even while other tasks were still PENDING (mid-failover). This
caused lazyFreeBatch to teardown the batch while retries were in-flight.
Now the batch reports FAILED only when ALL tasks reach a terminal state
(success_tasks + failed_tasks == total_tasks). A task still in PENDING
(e.g. resubmitted on a secondary transport) keeps the batch in PENDING.
Add unit tests covering the new aggregation logic: FAILED+PENDING →
PENDING, FAILED+COMPLETED → FAILED, all COMPLETED → COMPLETED, and
derived-task skipping.
Signed-off-by: Yuxin Chen <grityxchen@gmail.com>
* [TENT] Replace hardcoded transport type tests with dynamic sentinel check
The old AllEnumValuesDistinct and SupportedCount tests hardcoded the
enum list and expected count, breaking whenever a new transport type
(like SUNRISE_LINK) was added. Replace them with UnspecIsSentinel
which verifies invariants independent of how many transports exist.
Signed-off-by: Yuxin Chen <grityxchen@gmail.com>
* style(test): reformat lambda expression in failover_test
- Adjust line break for lambda assignment to improve readability
Signed-off-by: Yuxin Chen <grityxchen@gmail.com>
* fix(transfer-engine): correct worst failure tracking in getTransferStatus
- Introduce severity-based comparison to ensure `worst_failure` reflects the most severe status, preventing overwrites with lower severity.
Signed-off-by: Yuxin Chen <grityxchen@gmail.com>
---------
Signed-off-by: Yuxin Chen <grityxchen@gmail.com>
Co-authored-by: Yuxin Chen <grityxchen@gmail.com>
* [TransferEngine][Integration] feat: add MACA/MetaX GPU support and fix RDMA dmabuf registration
- Fix MACA compatibility macros: correct CUdeviceptr alias, add missing
CUDA memory type and pointer attribute macros, implement inline
cuGetErrorString wrapper
- Fix RDMA dmabuf memory registration for GPU memory: use allocation
base address for cuMemGetHandleForAddressRange and compute proper
offset for ibv_reg_dmabuf_mr (fixes#1975, #1965)
- Add USE_MACA guard alongside existing USE_MLU/USE_CUDA guards for
dmabuf-based memory registration path
- Support remote_request_id in mooncake connector v1 for cross-request
KV cache transfer between prefiller and decoder
---------
Co-authored-by: zhangxin <zhangxin@zhangxins-MacBook-Air.local>
* [TransferEngine] Fix GPU dependency in transfer_engine_bench
Problem: transfer_engine_bench crashes (exit code 247) when running with
--use_vram=false in environments without GPU, even though it only uses
CPU memory (DRAM).
Root cause: freeMemoryPool() calls cudaPointerGetAttributes() with
checkCudaError(), which exits the program if CUDA fails.
Solution: Use graceful error handling like transfer engine core library
(memory_location.cpp). When CUDA query fails, assume CPU memory and
use numa_free().
Impact:
- Enables CPU-only RDMA bandwidth testing without GPU
- Consistent behavior with mooncake_client
- No impact on existing GPU-enabled scenarios
Test: Verified RDMA bandwidth testing works in CPU-only pods and
achieves 10+ GB/s throughput on 200G RDMA network.
Signed-off-by: jibxie <jibxie@ebay.com>
* [TransferEngine] Optimize memory deallocation logic in transfer_engine_bench
Check FLAGS_use_vram before calling cudaPointerGetAttributes to avoid
unnecessary CUDA calls when memory is explicitly allocated on CPU.
- When FLAGS_use_vram is false, memory is guaranteed to be allocated
via numa_alloc_onnode, so we can directly call numa_free without
checking CUDA pointer attributes
- This avoids confusing WARNING logs on systems without GPU when users
explicitly choose to use DRAM
- Change log level from WARNING to ERROR when FLAGS_use_vram is true
but cudaPointerGetAttributes fails, for consistency with
memory_location.cpp
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
---------
Signed-off-by: jibxie <jibxie@ebay.com>
Co-authored-by: jibxie <jibxie@ebay.com>
Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
* [TENT] fallback to per-task cudaMemcpyAsync when driver lacks batch support
Fixes the case where containers ship CUDA Toolkit 12.8 but the host
driver only supports 12.2
* remove unconditionally overwrites the err
* style: format nvlink_transport.cpp with clang-format-20
---------
Co-authored-by: jinke15 <jinke15@jd.com>
This commit introduces a standalone maca_transport following the same
pattern as hip_transport, instead of polluting nvlink_transport with
MACA-specific workarounds.
Key points:
- Sync mcMemcpy with device-context guard (save/restore device before
each copy to avoid mcErrorContextIsDestroyed / SIGSEGV).
- Base-pointer registration via cuMemGetAddressRange for correct IPC
handle semantics with framework caching allocators.
- IPC-only path; fabric memory is not reliably supported on MACA 3.5.3.
- P2P access enabled in constructor with original device restoration.
Glue changes:
- multi_transport.cpp: register "maca" protocol
- transfer_engine_impl.cpp: auto-install maca transport under USE_MACA
- transfer_metadata.cpp: add "maca" to encode/decode protocol whitelist
- transfer_engine_validator.cpp: support --protocol=maca
- maca.h: add missing CU_POINTER_ATTRIBUTE_*, cuGetErrorString macros
Verified on Metax C500 (2-GPU) with transfer_engine_validator:
Data validation passed, throughput ~6.9 GB/s
* Add reference counting to RdmaTask to prevent UAF
- Convert RdmaSubBatch::task_list from value to pointer storage
- Add atomic reference counting to RdmaTask with Slab allocator integration
- Properly dereference tasks in freeSubBatch cleanup path
- Each slice holds a reference to its parent task
Author: Feng Ren <alogfans@gmail.com>
* Add paired ref/deref
* remove task->ref_count assignment
Jiangtian Feng