* feat(transfer-engine): extract Device API and IBGDA transport layer
Move IBGDA files from mooncake-ep to mooncake-transfer-engine (git mv):
- 6 headers: mooncake_ibgda/ → transport/device/ibgda/
- 1 source: mlx5gda.cpp → transport/device/mlx5gda.cpp
Add new Device API layer under transport/device/:
- device_transport.h: P2pTransport + RdmaTransport interfaces
- device_ops.cuh: DeviceOps function pointer table (bottom IR)
- comm_device.cuh, p2p_device.cuh, ibgda_device.cuh: device contexts
- cuda_ops.cuh, musa_ops.cuh: platform DeviceOps implementations
- ibgda_device_transport.cpp: RdmaTransport IBGDA implementation
- p2p_device_transport.cpp: P2pTransport NVLink/MTLink implementation
Modify transfer-engine:
- transfer_engine.h/impl: add getOrCreateP2pTransport/RdmaTransport
- gpu_vendor/musa.h: add MUSA API aliases
- Build: link mlx5, add device/ subdirectory
Minimal EP changes (include paths + build system only):
- Update include paths to new transport/device/ibgda/ location
- Remove mlx5gda.cpp from EP sources
- Forward EP_USE_MUSA env var in BuildEpExt.cmake
Root CMakeLists.txt: guard CUDAToolkit with USE_CUDA
* feat(example): add device_transport_example for P2P Device API
Two-rank example demonstrating the full Device API lifecycle:
- Host side: P2pTransport for IPC handle exchange and peer mapping
- Device side: CommCtx + mc_route_put + mc_signal for GPU-initiated
P2P data transfer and notification
Uses file-based IPC handle exchange (no external dependencies).
Requires 2 GPUs with P2P access (NVLink or PCIe).
* fix(example): handle missing TORCH_CUDA_ARCH_LIST
* fix(example): convert torch arch format to CMake CUDA format
* fix(example): construct CommCtx manually on host side
* fix(example): add using namespace mooncake::device in kernels
* fix(example): pass CommCtx by value, add barrier before cleanup
- Pass CommCtx by value to kernel (CUDA copies to param space) instead of
dereferencing a device pointer on host (caused segfault).
- Add file-based barrier so rank 0 waits for rank 1 to finish before
freeing its GDR buffer (IPC handle only valid while allocation exists).
- Change default metadata_server to P2PHANDSHAKE (no etcd dependency).
* fix: address review feedback — fence ordering, error checks, QP leak
- musa_ops.cuh: fix fence ordering for acquire/release semantics
(fence after load for acquire, before store/atomic for release)
- ibgda_device_transport.cpp: check cudaMalloc return values,
add num_qps >= num_ranks guard, destroy QP on rst2init failure
- p2p_device_transport.cpp: add device_count > 0 guard
- device_transport_example.cu: validate kDataBytes % 16 == 0
and kDataBytes <= kSignalWordOffset
* style: apply clang-format to Device API files
* fix: guard device transport code with USE_CUDA/USE_MUSA macros
The device transport accessors (getOrCreateP2pTransport,
getOrCreateRdmaTransport) and their member variables were not guarded
by USE_CUDA/USE_MUSA preprocessor macros. When building with
USE_CUDA=OFF (the default), the device transport source files aren't
compiled but the headers and implementations still reference them,
causing linker errors in CI build-flags and build jobs.
* fix: set CMAKE_CUDA_STANDARD 20 and make ibgda PUBLIC
- common.cmake: add CMAKE_CUDA_STANDARD 20 so nvcc compiles host code
in C++20 mode, matching CMAKE_CXX_STANDARD. Fixes "starts_with is
not a member of std::string" when compiling .cu files that indirectly
include common.h.
- transport/CMakeLists.txt: change ibgda from PRIVATE to PUBLIC so
mlx5gda_* symbols are visible to downstream consumers (Go p2p store
via transfer_engine). Fixes undefined reference errors for
mlx5gda_destroy_qp, mlx5dv_devx_umem_reg, etc.
* fix: compile mlx5gda into device_transport and link mlx5 for Go consumers
The previous attempt (PUBLIC ibgda) did not work because the Go p2p-store
and mooncake-store binaries link libtransfer_engine.a via hand-written cgo
ldflags, which bypass CMake's target_link_libraries propagation entirely.
- transport/device: compile mlx5gda.cpp directly into the device_transport
OBJECT library (like every other transport module) instead of a separate
ibgda STATIC lib, so mlx5gda_* symbols flow into libtransfer_engine.a and
are visible to all consumers regardless of how they link.
- transport: link libmlx5 (PUBLIC) since ibgda_device_transport.cpp /
mlx5gda.cpp call mlx5dv_devx_* / mlx5dv_init_obj directly.
- p2p-store/build.sh, mooncake-store/go/build.sh, ci.yml: add -lmlx5 to the
hand-written cgo ldflags so the DevX symbols resolve.
- example: set CUDA_STANDARD 20 on device_transport_example so nvcc compiles
common.h (std::string::starts_with) in C++20 mode.
* fix: CUDA_EXTENSIONS OFF for example, add -lm for Go consumers
Follow-up to compiling mlx5gda.cpp into device_transport:
- example: nvcc has no gnu++20 dialect, so CUDA_STANDARD 20 with the default
CUDA_EXTENSIONS=ON fails at CMake generate ("does not know the compile
flags"). Set CUDA_EXTENSIONS OFF to request plain -std=c++20.
- p2p-store/build.sh, mooncake-store/go/build.sh: mlx5gda.cpp uses log2/ceil
(<cmath>); now that its object lives in libtransfer_engine.a, the hand-
written cgo ldflags need -lm to resolve log2@GLIBC_2.29. (ci.yml already
had -lm.)
* fix(ci): gate Device API GPU example off by default, link mlx5 for Rust
The Docker build failed at CMake generate because device_transport_example
needs the CUDA20 dialect (transfer_engine.h -> common.h uses C++20
std::string::starts_with), which the older CMake in the CI image cannot
enable. CUDA_EXTENSIONS OFF did not help since the limitation is the CMake
version, not the dialect flavor. Gate this manual, 2-GPU example behind a
new BUILD_DEVICE_TRANSPORT_EXAMPLE option (default OFF) so the default build
no longer requires CUDA20.
Also link mlx5 in the mooncake-store Rust build script: the IBGDA device
transport (mlx5 DevX) is now compiled into transfer_engine, so the Rust
test link step needs -lmlx5 to resolve mlx5dv_devx_* symbols.
* fix(ci): move sccache --show-stats to after build steps
The "Run sccache stat for check" step was running before any build
step, so it always reported 0 cache hits and 0 cache misses. Move it
to after the build so it shows actual sccache statistics.
* fix(ci): move sccache stat after nvlink_allocator build in first job
In the first `build` job, the sccache stats step was placed after `Build
project` but before `Build nvlink_allocator.so`, missing the nvlink_allocator
compilation from the cache statistics. Move it after `Build nvlink_allocator.so`
to match the ordering in the `build-flags` job and capture all compilation steps.
Keep debug symbols enabled by default for local developer builds. CI test
workflows pass -DENABLE_DEBUG_SYMBOLS=OFF to reduce binary sizes during
testing. Release workflows are unchanged and retain debug symbols.
* [CI/Build] Switch WITH_NVIDIA_PEERMEM to env variable
* [CI/Build] Switch WITH_NVIDIA_PEERMEM to runtime env variable
Instead of a cmake build-time option, check the WITH_NVIDIA_PEERMEM
environment variable at runtime in rdma_context.cpp and
rdma_transport.cpp to switch between ibv_reg_mr() and
ibv_reg_dmabuf_mr().
- Remove option(WITH_NVIDIA_PEERMEM) and add_compile_definitions() from
common.cmake (no more compile-time flag)
- Update CMakeLists.txt to use GPU toolkit presence instead of cmake var
for nvlink-allocator build condition
- Add withNvidiaPeermem() runtime helper reading WITH_NVIDIA_PEERMEM env
var; default false (dmabuf path, no nvidia-peermem required)
- Replace #if !defined(WITH_NVIDIA_PEERMEM) && defined(USE_CUDA) guards
with runtime if (!withNvidiaPeermem()) checks
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/05c94a1b-d4d6-4b44-be25-3b98d9b01f1b
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Add WITH_NVIDIA_PEERMEM to Environ; use Environ::Get() in rdma files
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/66e6cabb-c473-4a9d-9711-aebe468fcee2
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix linker error: link transfer_engine against mooncake_common for Environ::Get()
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/29d46e31-6dc1-4fee-be33-4a603537b827
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix linker error in Go CGO builds: add -lmooncake_common to build.sh scripts
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/ab73f098-d408-4ed0-95bb-77f9ce9f71ae
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Add USE_MACA to nvlink-allocator conditions to cover all GPU cases
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/251d2c35-d12a-43ed-9958-40526e0420d5
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Go CGO linker path: add mooncake-common/src to library search paths
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/5a91e4b9-660d-463b-a8f2-d3fb0118fc34
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Rust build.rs: add mooncake_common link and CUDA stubs search paths
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/e0d1e4fd-2cc5-4fe9-9268-5c205e3fc0f5
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Rust build.rs: remove CUDA stubs from search_dirs to prevent runtime libcuda.so.1 dep
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/cd938ceb-c822-4439-b617-03f065015d4c
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix Rust build.rs: remove CUDA stubs from early rustc-link-search to prevent libcuda.so.1 runtime dep
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/055de577-7669-4039-a89a-d6066493e2d0
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
* Fix CI: create libcuda.so.1 symlink and set LD_LIBRARY_PATH for cargo test --lib
Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/76131e72-de6f-485b-98ab-342b112f3968
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
- Remove CUDA-related flags and environment variables
- Add new test step for TENT unit tests
Signed-off-by: Yuxin Chen <grityxchen@gmail.com>
Co-authored-by: Yuxin Chen <grityxchen@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
PyTorch 2.12.0 no longer publishes cu128 wheels (only cu130 and cu126).
Add a cmake branch in SetupPyTorchEnv.cmake that routes torch 2.12.0+
on CUDA 12 hosts to the cu126 wheel index.
Changes:
- EP_TORCH_VERSIONS: 2.9.0;2.9.1;2.10.0;2.11.0 → 2.9.1;2.10.0;2.11.0;2.12.0
- SetupPyTorchEnv.cmake: add version >= 2.12.0 branch → cu126
* [Store] Add lock-free MmapArena allocator for buffer mmap path
Replace per-allocation mmap() syscalls in allocate_buffer_mmap_memory()
with a lock-free atomic bump allocator (MmapArena). Pre-allocates a
configurable pool (default 64GB) and serves allocations via CAS loop,
reducing allocation latency from ~1us (mmap syscall) to ~50ns (atomic).
Allocation lifecycle is static: all callers (ClientBufferAllocator,
global segments in RealClient::setup_internal) allocate at startup and
free at shutdown. The arena outlives all allocations, so the bump-only
(no individual free) design is correct for this usage pattern.
Feature-flagged via gflags:
--use_mmap_arena_allocator (default: true)
--mmap_arena_pool_size (default: 64GB)
Falls back to direct mmap() when arena is disabled, fails to init,
or is exhausted.
Cherry-picked from flow-ipc-poc branch (utils.cpp perf path only).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Store] Fix three correctness issues in MmapArena
1. Honor caller's alignment contract: allocate() now accepts a
per-call alignment parameter and uses max(arena default, caller
request). allocate_buffer_mmap_memory() forwards its alignment
argument to the arena. Previously, the caller's alignment was
silently ignored — the arena always used 64-byte alignment
regardless of what the caller requested.
2. Remove MAP_POPULATE from arena pool mmap: the default pool is
64GB but callers typically use only a fraction (e.g. 4GB of
segments). MAP_POPULATE would pre-fault all 64GB of pages upfront,
causing seconds of startup delay and potentially triggering OOM
on machines with less physical memory. Pages now fault on demand.
3. Make alignment_ atomic and store it BEFORE the CAS on pool_base_:
previously alignment_ was a plain size_t written AFTER the release
CAS, so the store was not in the happens-before relationship
established by the acquire-release pair on pool_base_. Now both
alignment_ and pool_size_ are stored before the CAS with the
release fence guaranteeing their visibility to readers.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The pull_request_target trigger introduced in PR #1989 was incorrectly
preserved during the merge of PR #1992 due to conflict resolution.
Remove it along with the fork-routing conditions in ascend-test and
integration-test, since the mirror URL is now hardcoded in ci_ascend.yml
and vars/secrets access is no longer needed.
Signed-off-by: staryxchen <staryxchen@tencent.com>
Fork PRs trigger pull_request events where GitHub Actions withholds
vars and secrets. This causes vars.ASCEND_GITHUB_MIRROR_URLS to resolve
as empty on the self-hosted Ascend runner, blocking mirror-based checkout.
Add pull_request_target as a complementary trigger and route both
ascend-test and integration-test by fork origin: non-fork PRs continue
on pull_request, fork PRs are handled by pull_request_target where vars
and secrets are available. Duplicate runs are avoided by skipping the
opposite event for each case.
Signed-off-by: staryxchen <staryxchen@tencent.com>
PutToLocalFile and FileStorage::OffloadObjects crash with SIGSEGV when
slice.ptr points to GPU device memory, because CPU memcpy cannot access
GPU virtual addresses. The RDMA memory-replica path is unaffected.
Add synchronous Device-to-Host staging via PinnedBufferPool before data
reaches the disk-write paths:
- New gpu_staging_utils.h: shared IsDevicePointer/CopyDeviceToHost/
SetDevice helpers with cross-platform support (CUDA/HIP/MUSA/MACA/
Ascend CANN)
- New PinnedBufferPool: thread-safe pinned host memory pool with
max capacity limit (default 32) and O(1) swap-pop acquire
- PutToLocalFile: sync D2H on calling thread, PutRevoke on failure
- OffloadObjects: D2H staging before BatchOffload; on per-slice failure
the entire object is skipped to prevent partial/corrupt data
- CMakeLists: auto-detect CUDAToolkit/HIP/Ascend independently of
global USE_CUDA flag, with explicit PRIVATE compile definitions
- CI: add -lcudart to Go test CGO_LDFLAGS when CUDA is present
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Changes:
- Add job dependencies in ci.yml (build jobs depend on spell-check/clang-format)
- Convert ci_ascend.yml and integration-test.yml to workflow_run trigger
- Add paths-filter to avoid running on docs-only PRs
- Fix context variables in integration-test.yml for workflow_run
- Add paths filtering to ci_cu13.yml
* build: add memory-aware compile/link parallelism
Auto-detect available memory and CPU cores at configure time to calculate
safe parallel job limits. With Ninja generator, creates separate job pools
for compilation (~1.5GB/job) and linking (~4GB/job) so high-core machines
can compile fast without OOM during linking.
Changes:
- New mooncake-common/limit_jobs.cmake module
- Include from common.cmake for all build modes
- Switch Dockerfile and CI workflows to Ninja
- User can override via -DPARALLEL_COMPILE_JOBS / -DPARALLEL_LINK_JOBS
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix(ci): remove sudo from Ascend CI install step
The Ascend CI job runs inside a Docker container as root, where sudo
is not available. This caused exit code 127 after a successful build.
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [Store] feat: CXL storage full features.
* fix(ci): resolve cxl test failure and code format
* fix(ci): add CXL protocol support and fix code format issues
* [Store] feat: CXL storage full features, reset and rm extern/pybind
* [TE] Support TCP fallback in EFA build and improve EFA documentation (#1523)
When building with USE_EFA=ON, auto_discover is disabled to prevent
RDMA transport installation (QP creation fails on EFA devices). This
means TCP transport is also not installed automatically. Add explicit
TCP transport installation for non-EFA protocols in the EFA build path.
Documentation changes:
- build.md: Add USE_EFA option and clarify USE_CUDA default/purpose
- supported-protocols.md: Add EFA as a supported protocol
- efa_transport.md: Add USE_CUDA=ON to build command, document GPU
memory requirement
Co-authored-by: whn09 <whn09@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
* [Store] Optimize BucketStorageBackend for reduced lock contention and add delete safety (#1456)
* fix(ci): resolve cxl test failure and code format
* [Store] Add Local Cache Mechanism for Mooncake Store Client (#1226)
* feat(Store): add local hot cache for client
* feat(Store): add client local hot cache log to show performance
* fix: local hot cache initialize bug
* fix(Store): Mooncake put slice is max 16MB， so make local hot cache block 16MB
* feat(Store): move local hot cache initialization to Client::Create
* feat(Store): local hot cache remove unused small block implementation
* feat(Store): add client local hot cache unit test
* fix(Store): modify client local hot cache suit with v0.3.7
* feat(Store): change local hot cache unit tes
* fix: initialize local hot cache with negative value
* feat: use in process master and metadata fro local hot cache unit test.
* feat: update local hot cache to one replica one slice version
* fix: local hot cache unit test use in process master service
* fix: code style fix
* fix: fix dirty read when client wants to read a previously hitted hot block but the hot block is modified by incoming put actions
* fix: local hot cache unit test use in process master service
* fix: code format fix
* fix: fix comment problems for
* feat: add local hot asynchronous queue size limit
* fix: local hot cache task involves the block so that there is no memcpy operation when inserting local hot cache
* fix: code check fix
* fix: update block in_use prop to reference count
---------
Co-authored-by: shichangzhang064 <zhangshichang@h-partners.com>
* fix(ci): add CXL protocol support and fix code format issues
* fix: address comments from code review
* fix(ci): resolve cxl test failure
* fix(ci): resolve ci error
---------
Co-authored-by: 王鹤男 <wanghenan09@gmail.com>
Co-authored-by: whn09 <whn09@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
Co-authored-by: Mahesh Bapatu <153306023+maheshrbapatu@users.noreply.github.com>
Co-authored-by: Shichang Zhang <77728761+Shichang-Zhang@users.noreply.github.com>
Co-authored-by: shichangzhang064 <zhangshichang@h-partners.com>
* [Store]: add task executor feature with unit and executor test
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: add some optimizations to task executor
Refactor the task_executor into the client_service and add a
task structure on the client side. Additionally, remove the
existence check in the execute function;
only retrying should be performed if replica allocation fails.
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: get source replica from copyStart or moveStart api
* [Store]: call move or copy end if the target replica already exist to complete the replication task
* [Store]: use the max_retry_attempts in master side
* [Store]: add client integration test and set default max_retry_attempts to 10
* [Doc] update task api introduction
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store] Set copy and move as private methods
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Doc]: change the default task max_retry_attempts to 10
* [Doc]: fix some description error
* [Store]: allocate the buffer size to be a multiple of 16MB
* [Store]: validate the replica is in local and directly construct slices from replica buffer address instead of copy the data to local buffer
* [Store]: change the validate logic to directly use transfer engine endpoint or local_hostname_.
* [Store] add e2e ci test for copy and move api
* [Store] refactor the client move and copy function
* [Store] fix the e2e test
* [Store] remove unused code
* [Store] add source field when build replica copy payload in the task_manager_test
* [Store] rename back to snake case for split_into_slices function and also remove hard code for client poll count
* [Store] revert mis deleted field when resolve conflicts
* [Store] change test to validate the real behaviour
* [Store] change the default task fetch size to 16
* [Doc]: change the replica copy/move sequence diagram
* [Store] add new split_into_slice method
* [Store] change the real client to use split_to_slice with buffer handle parameters
---------
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
Co-authored-by: Vincent Gao <vincentbo@linux.alibaba.com>
* feat: use setuptools_scm for more elegant version and support MOONCAKE_LOCAL_VERSION env
* feat: more compatible
* feat: add fetch-tags for build ci
* feat: manually add Fetch git tags actions
* feat: use fetch-depth: 0 and filter: tree:0
---------
Co-authored-by: huangweixiao <huangweixiao@moonshot.cn>
* [Store] feat: add cxl storage for mooncake store
* Update extern/pybind11 to match main
* fix: use fake cxl device to bypass ci-test error
* Fix code formatting in segment.cpp
---------
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
* test
* Test
* Apply suggestions from code review
---------
Co-authored-by: Ke Yang <yangke@approaching.ai>
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
- Add conditional checkout with different fetch-depth for PR vs push events
- Implement changed files detection for both PR and push scenarios
- Run clang-format only on changed C++ files instead of entire codebase
- Add skip step when no C++ files are modified
Signed-off-by: staryxchen <staryxchen@tencent.com>
Aoi