mooncake-track
/
Mooncake
0
0
Fork 12
Code Issues Pull Requests 13 Actions Packages Projects Releases Wiki Activity

[CCF Archive] Store object type eviction policy submission #3

Closed
kancel wants to merge 382 commits from kancel:ccf-archive-pr2746 into main
pull from: kancel:ccf-archive-pr2746
merge into: mooncake-track:main
Conversation 0 Commits 382 Files Changed 654 +115043 -15224
654 changed files with 115043 additions and 15224 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

51
.ci/run_test.sh
View File

@ -1,51 +0,0 @@
set -ex
TEST_DIR=$1
OUTPUT_DIR=$2
run_test(){
echo "$OUTPUT_DIR"
./"$1"
LOCAL_SUCCESS=$?
if [[ $LOCAL_SUCCESS == 0 ]]; then
echo "PASSED : $1 " >> "$OUTPUT_DIR"/results.txt
else
echo "FAILED : $1 " >> "$OUTPUT_DIR"/results.txt
fi
}
pushd $TEST_DIR
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://10.0.0.1:2379 &
export MC_GID_INDEX=1
run_test transport_uint_test
run_test tcp_transport_test
export MC_GID_INDEX=1
# sudo pkill etcd
# sleep 1
# etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://10.0.0.1:2379 &
# sudo fuser -k 14345/tcp
export MC_GID_INDEX=1 && ./rdma_transport_test --mode=target --metadata_server=127.0.0.1:2379 --local_server_name=127.0.0.2:14345 --device_name=erdma_0 &
export MC_GID_INDEX=1 && ./rdma_transport_test --metadata_server=127.0.0.1:2379 --segment_id=127.0.0.2:14345 --local_server_name=127.0.0.3:14346 --device_name=erdma_1
LOCAL_SUCCESS=$?
if [[ $LOCAL_SUCCESS == 0 ]]; then
echo "PASSED : rdma_transport_test " >> "$OUTPUT_DIR"/results.txt
else
echo "FAILED : rdma_transport_test " >> "$OUTPUT_DIR"/results.txt
fi
# sudo pkill etcd
# sleep 1
# etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://10.0.0.1:2379 &
# export MC_GID_INDEX=1
# sudo fuser -k 15345/tcp
# export MC_GID_INDEX=1 && ./rdma_transport_test --mode=target --metadata_server=127.0.0.1:2379 --local_server_name=127.0.0.2:14345 --device_name=erdma_0 &
export MC_GID_INDEX=1 && ./rdma_transport_test2 --metadata_server=127.0.0.1:2379 --segment_id=127.0.0.2:14345 --local_server_name=127.0.0.3:14346 --device_name=erdma_1
LOCAL_SUCCESS=$?
if [[ $LOCAL_SUCCESS == 0 ]]; then
echo "PASSED : rdma_transport_test2 " >> "$OUTPUT_DIR"/results.txt
else
echo "FAILED : rdma_transport_test2 " >> "$OUTPUT_DIR"/results.txt
fi
popd

42
.claude-plugin/marketplace.json Normal file
View File

@ -0,0 +1,42 @@
{
"$schema": "https://json.schemastore.org/claude-code-marketplace.json",
"name": "mooncake",
"owner": {
"name": "Mooncake Team"
},
"metadata": {
"description": "Built-in Claude Code skills for working with Mooncake: deployment troubleshooting, local CI validation, and the Python API."
},
"plugins": [
{
"name": "mooncake-troubleshoot",
"source": {
"source": "git-subdir",
"url": "https://github.com/kvcache-ai/Mooncake.git",
"path": ".claude/skills/mooncake-troubleshoot"
},
"description": "Systematically diagnose Mooncake deployment and runtime issues (services, RDMA, env vars, connectivity, logs) and propose fixes.",
"homepage": "https://kvcache-ai.github.io/Mooncake/skills/mooncake-troubleshoot.html"
},
{
"name": "mooncake-ci-local",
"source": {
"source": "git-subdir",
"url": "https://github.com/kvcache-ai/Mooncake.git",
"path": ".claude/skills/mooncake-ci-local"
},
"description": "Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.",
"homepage": "https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html"
},
{
"name": "mooncake-api",
"source": {
"source": "git-subdir",
"url": "https://github.com/kvcache-ai/Mooncake.git",
"path": ".claude/skills/mooncake-api"
},
"description": "Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.",
"homepage": "https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html"
}
]
}

12
.claude/skills/mooncake-api/.claude-plugin/plugin.json Normal file
View File

@ -0,0 +1,12 @@
{
"$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
"name": "mooncake-api",
"description": "Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.",
"author": {
"name": "Mooncake Team"
},
"homepage": "https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html",
"repository": "https://github.com/kvcache-ai/Mooncake",
"license": "Apache-2.0",
"keywords": ["mooncake", "python", "api", "transfer-engine", "store"]
}

5
.claude/skills/mooncake-api/SKILL.md
View File

@ -1,4 +1,7 @@
<!-- Let's perfect this skill together. -->
---
name: mooncake-api
description: Help users work with the Mooncake Python APIs for distributed storage and high-performance data transfer. Use when working with Mooncake Store (distributed KV cache), Transfer Engine (RDMA/TCP transfers), service setup (master, metadata server), PyTorch tensors in the Store, zero-copy/buffer management, batch operations and replication, or Mooncake EP / Backend (Expert Parallelism). Trigger on questions about MooncakeDistributedStore, TransferEngine, put/get, put_tensor, register_buffer, ReplicateConfig, or the mooncake.store / mooncake.engine / mooncake.pg Python modules.
---
# Mooncake Python API Skill

12
.claude/skills/mooncake-ci-local/.claude-plugin/plugin.json Normal file
View File

@ -0,0 +1,12 @@
{
"$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
"name": "mooncake-ci-local",
"description": "Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.",
"author": {
"name": "Mooncake Team"
},
"homepage": "https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html",
"repository": "https://github.com/kvcache-ai/Mooncake",
"license": "Apache-2.0",
"keywords": ["mooncake", "ci", "validation", "pre-pr"]
}

383
.claude/skills/mooncake-ci-local/SKILL.md
View File

@ -1,266 +1,159 @@
---
name: mooncake-ci-local
description: Run Mooncake CI test suite locally — maps GitHub Actions CI steps to local commands. Use this skill whenever the user wants to run tests locally, reproduce a CI failure, check if their changes break tests, or run any subset of the CI test suite (C++ unit tests via ctest, Python integration tests, code format checks, or the full test pipeline). Trigger on phrases like "run tests", "run CI locally", "reproduce CI failure", "check my changes", "test before PR", "run ctest", "run python tests", "run all tests".
description: Run Mooncake pre-PR local validation through scripts/run_ci_test.sh. Use this skill whenever the user wants to validate a branch before opening or submitting a PR, run local CI, run ci test, check changes before PR, reproduce GitHub Actions locally, or force a full pre-submit verification. Trigger on phrases like "提交 PR 前验证", "run ci test", "run local CI", "check my branch", "test before PR", "pre-submit validation", and "reproduce CI locally".
---
# Mooncake CI Local Test Runner
# Mooncake Pre-PR Local Validation
You help users run the Mooncake CI test suite locally. The CI has three test layers. Map what the user wants to the right layer, check prerequisites, and run the tests.
Use `bash scripts/run_ci_test.sh` as the default entry point. This is the single local lane for PR-before-submit validation, and it already coordinates the reproducible parts of GitHub Actions.
## CI Test Layers
## Default Entry Point
### Layer 1 — C++ Unit Tests (ctest)
**CI equivalent:** `build` job in `ci.yml` — "Test (in build env) with coverage"
When the user asks for any of the following, run the repo script first instead of reconstructing the workflow by hand:
**Prerequisite services:**
```bash
# 1. etcd (port 2379)
etcd --advertise-client-urls http://127.0.0.1:2379 --listen-client-urls http://127.0.0.1:2379 &
sleep 2
etcdctl --endpoints=http://127.0.0.1:2379 endpoint health # verify
- 提交 PR 前本地验证
- run ci test
- run local CI
- check my branch before PR
- reproduce CI locally
# 2. HTTP metadata server (port 8080)
cd mooncake-transfer-engine/example/http-metadata-server-python
pip install aiohttp
python ./bootstrap_server.py &
cd -
```
**Run:**
```bash
cd build
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
MC_METADATA_SERVER=http://127.0.0.1:8080/metadata DEFAULT_KV_LEASE_TTL=500 ctest -j --output-on-failure
```
**Run specific test:**
```bash
cd build
MC_METADATA_SERVER=http://127.0.0.1:8080/metadata DEFAULT_KV_LEASE_TTL=500 ctest -R <test_name_pattern> --output-on-failure
# List all available tests: ctest -N
```
### Layer 2 — Python Integration Tests
**CI equivalent:** `test-wheel-ubuntu` job — `run_tests.sh`
**Prerequisite:** Mooncake wheel must be installed (either via `pip install` or via `make install` after build).
**Check install:**
```bash
python -c "import mooncake; print('OK')"
which mooncake_master # must NOT be /usr/local/bin (must be from Python package)
```
**Run full suite:**
```bash
# Start metadata server first
mooncake_http_metadata_server --port 8080 &
sleep 1
cd mooncake-wheel/tests
MC_METADATA_SERVER=http://127.0.0.1:8080/metadata DEFAULT_KV_LEASE_TTL=500 MC_FORCE_TCP=true \
bash ../../scripts/run_tests.sh
```
**Individual Python tests** (all require metadata server + mooncake_master on port 50051):
```bash
# Setup shared services
mooncake_http_metadata_server --port 8080 &
mooncake_master --default_kv_lease_ttl=500 &
sleep 2
cd mooncake-wheel/tests
export MC_METADATA_SERVER=http://127.0.0.1:8080/metadata
export DEFAULT_KV_LEASE_TTL=500
export MC_FORCE_TCP=true
# Pick any test:
python test_distributed_object_store.py
python test_replicated_distributed_object_store.py
python test_put_get_tensor.py # requires torch + numpy
python test_safetensor_functions.py # requires safetensors
python test_dummy_client.py
python test_cli.py
python test_distributed_object_store_cxl.py # requires CXL build
```
**Transfer engine tests specifically:**
```bash
cd mooncake-wheel/tests
MC_METADATA_SERVER=http://127.0.0.1:8080/metadata MC_FORCE_TCP=true python transfer_engine_target.py &
TARGET_PID=$!
MC_METADATA_SERVER=http://127.0.0.1:8080/metadata MC_FORCE_TCP=true python transfer_engine_initiator_test.py
kill $TARGET_PID
```
**Scripts-based tests** (from `test-wheel-ubuntu` job):
```bash
# Tensor API perf test
export MOONCAKE_MASTER="127.0.0.1:50051"
export MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata"
export MOONCAKE_PROTOCOL="tcp"
export LOCAL_HOSTNAME="127.0.0.1"
python scripts/test_tensor_api.py -n 1
python scripts/test_async_store.py
python scripts/test_copy_move_api.py
```
### Layer 3 — Static Checks (no services needed)
**CI equivalent:** `clang-format` and `spell-check` jobs
**Code format (changed files vs main):**
```bash
./scripts/code_format.sh --check --base origin/main
# Auto-fix:
./scripts/code_format.sh --base origin/main
```
**Spell check:**
```bash
# Requires typos tool: cargo install typos-cli
typos
```
**Pre-commit (runs all hooks):**
```bash
pip install pre-commit
pre-commit run --all-files
# Or just on staged files:
pre-commit run
```
## Build Configurations (from CI)
If the user needs to build first, here are the CI-equivalent cmake flags:
**Standard build with coverage (mirrors `build` job):**
```bash
mkdir build && cd build
cmake -G Ninja .. -DUSE_HTTP=ON -DUSE_CXL=ON -DUSE_ETCD=ON -DSTORE_USE_ETCD=ON -DENABLE_ASAN=ON -DCMAKE_BUILD_TYPE=Debug
cmake --build .
sudo cmake --install .
```
**All features ON (mirrors `build-flags` job):**
```bash
mkdir build && cd build
cmake -G Ninja .. -DUSE_ETCD=ON -DUSE_CXL=ON -DUSE_REDIS=ON -DUSE_HTTP=ON -DWITH_STORE=ON -DWITH_P2P_STORE=ON -DWITH_METRICS=ON -DBUILD_UNIT_TESTS=ON -DBUILD_EXAMPLES=ON
cmake --build .
sudo cmake --install .
```
**Transfer engine only:**
```bash
cd mooncake-transfer-engine
mkdir build && cd build
cmake -G Ninja .. -DUSE_ETCD=OFF -DUSE_CXL=ON -DUSE_REDIS=ON -DUSE_HTTP=ON -DBUILD_UNIT_TESTS=ON -DBUILD_EXAMPLES=ON
cmake --build .
```
## Workflow: Diagnosing and Running Tests
### Step 1 — Understand what the user wants
Ask (or infer from context):
- All tests, or a specific subset?
- Did a specific CI job fail? Which one?
- Is the build already done, or do they need to build first?
### Step 2 — Check and Fix Prerequisites
**One-command setup** — this script checks all prerequisites and auto-fixes issues:
Default command:
```bash
bash .claude/skills/mooncake-ci-local/scripts/check-prerequisites.sh
bash scripts/run_ci_test.sh
```
**What it checks:**
1. ✓ Build directory exists
2. ✓ mooncake package installed (auto-installs via cmake --install if missing)
3. ✓ ctest available
4. ✓ Restarts all services (etcd, metadata server) in clean state
5. ✓ Verifies all services are healthy
What this script already covers:
**If you need to build first:**
```bash
mkdir build && cd build
cmake -G Ninja .. -DUSE_HTTP=ON -DUSE_ETCD=ON -DUSE_CXL=ON -DSTORE_USE_ETCD=ON -DCMAKE_BUILD_TYPE=Debug
cmake --build .
sudo cmake --install .
```
- GitHub-like `paths-filter` against `origin/main`
- `typos`
- `scripts/code_format.sh --check`
- default CMake configure/build/install in `build-ci-local`
- `ctest`
- wheel build in `build-wheel-local`
- wheel installation validation
- `scripts/run_tests.sh`
- selected Python API and integration tests
- per-stage summary and logs under `local_test/run-ci-logs/<timestamp>/`
**If script fails:**
- Build issues: See "Build Configurations" section below
- mooncake install fails: Try `pip install mooncake-wheel/dist/*.whl` manually
- etcd install fails: Download from https://github.com/etcd-io/etcd/releases
## Standard Agent Workflow
### Step 3 — Run and report
1. Run `bash scripts/run_ci_test.sh` from the repo root unless the user explicitly asks for a narrower subset.
2. Read the stage summary instead of dumping raw terminal output.
3. Report these items back to the user:
- passed stages
- failed stages
- blocked stages
- unsupported stages
- whether `paths-filter` skipped downstream stages
- the log directory under `local_test/run-ci-logs/...`
4. If there is a failure, inspect the corresponding stage log and summarize the root cause.
Run the relevant test layer. On failure:
1. Show the exact error message
2. Check if it's a service/env issue (most common) vs a real test failure
3. Suggest the fix (see common issues below)
## Common Options
## Common Local Test Issues
**"mooncake_master found in /usr/local/bin" error in run_tests.sh:**
The test expects mooncake_master to come from the Python package, not a system install.
```bash
# Remove the system-installed binary:
sudo rm /usr/local/bin/mooncake_master
# Or use the wheel-installed one:
pip install mooncake-wheel/dist/*.whl
```
**etcd port conflict:**
```bash
pkill etcd && sleep 1
etcd --advertise-client-urls http://127.0.0.1:2379 --listen-client-urls http://127.0.0.1:2379 &
```
**Metadata server port conflict:**
```bash
pkill -f bootstrap_server.py
pkill -f mooncake_http_metadata_server
```
**Tests hang (master not responding):**
```bash
pkill mooncake_master
sleep 2
mooncake_master --default_kv_lease_ttl=500 &
sleep 1
```
**torch/numpy not installed for tensor tests:**
```bash
pip install torch numpy safetensors packaging
```
**ctest shows no tests found:**
```bash
# Rebuild with unit tests enabled:
cd build
cmake .. -DBUILD_UNIT_TESTS=ON
cmake --build .
```
## Quick One-Liners
Force a full lane even if `paths-filter` would skip downstream stages:
```bash
# Run ALL C++ tests (after building with etcd + metadata server running):
# Note: full suite takes 5-15 minutes depending on hardware
cd build && MC_METADATA_SERVER=http://127.0.0.1:8080/metadata DEFAULT_KV_LEASE_TTL=500 ctest -j --output-on-failure
# Run only fast tests (skip slow integration tests):
cd build && MC_METADATA_SERVER=http://127.0.0.1:8080/metadata DEFAULT_KV_LEASE_TTL=500 ctest -j --output-on-failure --exclude-regex "etcd|ha_test|redis"
# Run ALL Python tests:
mooncake_http_metadata_server --port 8080 & sleep 1 && cd mooncake-wheel/tests && MC_METADATA_SERVER=http://127.0.0.1:8080/metadata MC_FORCE_TCP=true bash ../../scripts/run_tests.sh
# Check code format (changed files only):
./scripts/code_format.sh --check --base origin/main
# Full pre-commit check:
pre-commit run --all-files
bash scripts/run_ci_test.sh --skip-path-filter
```
Use another base ref:
```bash
bash scripts/run_ci_test.sh --base origin/main
```
Auto-install missing dependencies:
```bash
bash scripts/run_ci_test.sh --install-deps
```
Keep services running for follow-up debugging:
```bash
bash scripts/run_ci_test.sh --keep-services
```
## Minimal Example
User prompt:
- 提交 PR 前，帮我跑一遍本地 CI 验证当前分支。
Expected action:
```bash
bash scripts/run_ci_test.sh
```
If the user wants to ignore changed-path optimization and force the full lane:
```bash
bash scripts/run_ci_test.sh --skip-path-filter
```
See also `.claude/skills/mooncake-ci-local/examples/minimal.md`.
## How To Interpret Results
- `passed`: the stage succeeded locally.
- `failed`: the stage reproduced a real local failure and needs investigation.
- `blocked`: local environment or dependency issue prevented execution.
- `unsupported`: intentionally not run in the local lane because it needs external platforms, special hardware, or a non-default build.
If `paths-filter` skips downstream stages, explain that the current branch changed only non-source paths relative to the selected base.
## Current Local Coverage
Included by default:
- spell check
- code format check
- default ASan CMake lane in `build-ci-local`
- `ctest`
- wheel build and installation test
- `scripts/run_tests.sh`
- selected Python API tests
Unsupported by design in the default local lane:
- Ascend jobs
- T-one integration jobs
- MUSA jobs
- Docker image build jobs
- CUDA 13 wheel jobs
- PG-backend tests absent from the default wheel build
- Python drain-http API stage in the local ASan lane
## Targeted Reruns For Debugging
Use targeted reruns only after the full script identifies a failing area, or when the user explicitly asks for a smaller scope.
Rerun a specific C++ test pattern:
```bash
cd build-ci-local
MC_METADATA_SERVER=http://127.0.0.1:8080/metadata DEFAULT_KV_LEASE_TTL=500 ctest -R <pattern> --output-on-failure
```
Rerun the Python wheel integration lane:
```bash
source test_env/bin/activate
MC_STORE_MEMCPY=false TEST_SSD_OFFLOAD_IN_EVICT=true ./scripts/run_tests.sh
```
Rerun the safetensor unittest:
```bash
source test_env/bin/activate
python -m unittest mooncake-wheel.tests.test_safetensor_functions
```
## Notes For The Agent
- Prefer the repo script over rebuilding the CI workflow step by step.
- Preserve the separation between `build-ci-local` and `build-wheel-local`.
- Summarize failing stages from their logs instead of pasting raw output.
- If the user only asks whether the branch is safe before opening a PR, the default answer path is `bash scripts/run_ci_test.sh`.

30
.claude/skills/mooncake-ci-local/examples/minimal.md Normal file
View File

@ -0,0 +1,30 @@
# Minimal Example
Goal: validate the current branch before opening or submitting a PR.
User prompt:
- 提交 PR 前，帮我跑一遍本地 CI 验证当前分支。
Expected action:
```bash
bash scripts/run_ci_test.sh
```
Force a full rerun even if `paths-filter` would skip downstream stages:
```bash
bash scripts/run_ci_test.sh --skip-path-filter
```
Result location:
- `local_test/run-ci-logs/<timestamp>/`
Typical report format:
- passed stages
- failed or blocked stages
- unsupported stages
- first actionable failure and its log path

12
.claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json Normal file
View File

@ -0,0 +1,12 @@
{
"$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
"name": "mooncake-troubleshoot",
"description": "Systematically diagnose Mooncake deployment and runtime issues (services, RDMA, env vars, connectivity, logs) and propose fixes.",
"author": {
"name": "Mooncake Team"
},
"homepage": "https://kvcache-ai.github.io/Mooncake/skills/mooncake-troubleshoot.html",
"repository": "https://github.com/kvcache-ai/Mooncake",
"license": "Apache-2.0",
"keywords": ["mooncake", "troubleshooting", "rdma", "deployment"]
}

3
.devcontainer/Dockerfile
View File

@ -12,7 +12,7 @@ RUN apt-get install -y libibverbs-dev \
libjsoncpp-dev \
libnuma-dev \
libpython3-dev \
libboost-all-dev \
libboost-dev \
libssl-dev \
libgrpc-dev \
libgrpc++-dev \
@ -26,6 +26,7 @@ RUN apt-get install -y libibverbs-dev \
libzstd-dev \
libmsgpack-dev \
libgflags-dev \
libxxhash-dev \
pkg-config \
patchelf

6
.devcontainer/devcontainer.json
View File

@ -1,6 +1,8 @@
{
"name": "Mooncake Dev",
"dockerFile": "Dockerfile",
"build": {
"dockerfile": "Dockerfile"
},
"runArgs": [
"--cap-add=SYS_PTRACE",
"--cap-add=NET_RAW",
@ -18,7 +20,7 @@
"vscode": {
"settings": {
"terminal.integrated.shell.linux": "/bin/bash",
"bazel.buildifierFixOnFormat": true,
"bazel.buildifierFixOnFormat": true
}
}
}

1
.github/CODEOWNERS vendored
View File

@ -15,6 +15,7 @@
/mooncake-store @ykwd @stmatengss @XucSh @YiXR
/mooncake-store/*/ha/ @Libotry @YiXR @00fish0
/mooncake-transfer-engine @alogfans @doujiang24 @chestnut-Q
/mooncake-transfer-engine/tent @alogfans @doujiang24 @chestnut-Q @staryxchen @00fish0 @dtcccc
/mooncake-transfer-engine/*/transport/hip_transport/ @alogfans @amd-arozanov
/mooncake-transfer-engine/*/transport/ascend_transport/ @alogfans @ascend-direct-dev
/mooncake-transfer-engine/*/transport/efa_transport/ @alogfans @whn09

40
.github/labeler.yml vendored
View File

@ -22,3 +22,43 @@ Mooncake EP:
Installation:
- changed-files:
- any-glob-to-any-file: 'mooncake-wheel/**/*'
P2P Store:
- changed-files:
- any-glob-to-any-file: 'mooncake-p2p-store/**/*'
Integration:
- changed-files:
- any-glob-to-any-file: 'mooncake-integration/**/*'
Common:
- changed-files:
- any-glob-to-any-file: 'mooncake-common/**/*'
CI/Build:
- changed-files:
- any-glob-to-any-file:
- '.github/**/*'
- 'CMakeLists.txt'
- '.pre-commit-config.yaml'
- 'dependencies.sh'
Documentation:
- changed-files:
- any-glob-to-any-file:
- 'docs/**/*'
- '*.md'
Tests:
- changed-files:
- any-glob-to-any-file:
- 'scripts/test_*'
- 'mooncake-wheel/tests/**/*'
- 'scripts/tone_tests/**/*'
Ascend/NPU:
- changed-files:
- any-glob-to-any-file:
- '**/ascend*/**'
- 'scripts/ascend/**'
- '.github/workflows/*ascend*'

36
.github/pull_request_template.md vendored
View File

@ -7,10 +7,11 @@
- [ ] Transfer Engine (`mooncake-transfer-engine`)
- [ ] Mooncake Store (`mooncake-store`)
- [ ] Mooncake EP (`mooncake-ep`)
- [ ] Mooncake PG (`mooncake-pg`)
- [ ] Integration (`mooncake-integration`)
- [ ] P2P Store (`mooncake-p2p-store`)
- [ ] Python Wheel (`mooncake-wheel`)
- [ ] PyTorch Backend (`mooncake-pg`)
- [ ] Common (`mooncake-common`)
- [ ] Mooncake RL (`mooncake-rl`)
- [ ] CI/CD
- [ ] Docs
@ -23,15 +24,38 @@
- [ ] Refactor
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Other
## How Has This Been Tested?
<!-- Please describe the tests you've run to verify your changes. -->
<!-- Describe the tests you've run. Include commands and output if applicable. -->
**Test commands:**
```bash
# Example: bash scripts/run_ci_test.sh
```
**Test results:**
- [ ] Unit tests pass
- [ ] Integration tests pass (if applicable)
- [ ] Manual testing done (describe below)
## Checklist
- [ ] I have performed a self-review of my own code.
- [ ] I have formatted my own code using `./scripts/code_format.sh` before submitting.
- [ ] I have updated the documentation.
- [ ] I have added tests to prove my changes are effective.
- [ ] I have performed a self-review of my own code
- [ ] I have formatted my code using `./scripts/code_format.sh`
- [ ] I have run `pre-commit run --all-files` and all hooks pass
- [ ] I have updated the documentation (if applicable)
- [ ] I have added tests to prove my changes are effective
- [ ] For changes >500 LOC: I have filed an RFC issue
## AI Assistance Disclosure
<!-- If AI tools were used, please disclose. This helps reviewers focus attention. -->
- [ ] No AI tools were used
- [ ] AI tools were used (specify below)
<!-- If AI tools were used, briefly describe which tool and what it helped with.
The human submitter is responsible for understanding and defending all changes. -->

60
.github/workflows/assistant.yml vendored Normal file
View File

@ -0,0 +1,60 @@
name: Qoder Assistant
on:
issue_comment:
types: [created]
pull_request_review_comment:
types: [created]
jobs:
qoder-assistant:
if: |
contains(github.event.comment.body, '@qoder') &&
!endsWith(github.event.comment.user.login, '[bot]')
runs-on: ubuntu-latest
permissions:
contents: read
issues: write
pull-requests: write
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Build Arguments
id: build_args
run: |
ARGS="REPO: ${{ github.repository }}
REQUEST_SOURCE: ${{ github.event_name }}
THREAD_ID: ${{ github.event.comment.node_id }}
COMMENT_ID: ${{ github.event.comment.id }}
AUTHOR: ${{ github.event.comment.user.login }}
BODY: ${{ github.event.comment.body }}
URL: ${{ github.event.comment.html_url }}
IS_PR: ${{ github.event.issue.pull_request != null || github.event_name == 'pull_request_review_comment' }}
ISSUE_OR_PR_NUMBER: ${{ github.event.issue.number || github.event.pull_request.number }}"
if [ -n "${{ github.event.comment.pull_request_review_id }}" ]; then
ARGS="$ARGS
REVIEW_ID: ${{ github.event.comment.pull_request_review_id }}"
fi
if [ -n "${{ github.event.comment.in_reply_to_id }}" ]; then
ARGS="$ARGS
REPLY_TO_COMMENT_ID: ${{ github.event.comment.in_reply_to_id }}"
fi
echo "args<<EOF" >> $GITHUB_OUTPUT
echo "$ARGS" >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
- name: Run Qoder Assistant
uses: QoderAI/qoder-action@v0
with:
qoder_personal_access_token: ${{ secrets.QODER_PERSONAL_ACCESS_TOKEN }}
prompt: |
/assistant
${{ steps.build_args.outputs.args }}

284
.github/workflows/ci.yml vendored
View File

@ -43,6 +43,9 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Install and start etcd
run: |
wget https://github.com/etcd-io/etcd/releases/download/v3.6.1/etcd-v3.6.1-linux-amd64.tar.gz
@ -72,6 +75,11 @@ jobs:
sudo apt-get update
sudo apt-get install -y lcov gcovr ninja-build
- name: Test HugeTLB sizing helper
run: |
python3 scripts/test_hicache_hugepage_requirements.py
shell: bash
- name: Set up coverage compilation flags
run: |
echo "Setting up coverage compilation flags..."
@ -90,17 +98,13 @@ jobs:
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Run sccache stat for check
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Configure project with coverage support
run: |
sudo apt update -y
sudo bash -x dependencies.sh -y
mkdir build
cd build
cmake -G Ninja .. -DUSE_HTTP=ON -DUSE_CXL=ON -DUSE_UB=ON -DUSE_ETCD=ON -DSTORE_USE_ETCD=ON -DENABLE_ASAN=ON -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Debug
cmake -G Ninja .. -DUSE_HTTP=ON -DUSE_CXL=ON -DUSE_UB=ON -DUSE_ETCD=ON -DSTORE_USE_ETCD=ON -DENABLE_ASAN=ON -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Debug -DENABLE_DEBUG_SYMBOLS=OFF
shell: bash
- name: Build project
@ -118,6 +122,11 @@ jobs:
bash build.sh ../../build/mooncake-transfer-engine/nvlink-allocator/
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Start Metadata Server
run: |
cd mooncake-transfer-engine/example/http-metadata-server-python
@ -125,6 +134,39 @@ jobs:
python ./bootstrap_server.py &
shell: bash
- name: Run Mooncake Store Rust smoke test and benchmark
run: |
$GITHUB_WORKSPACE/build/mooncake-store/src/mooncake_master \
--eviction_high_watermark_ratio=0.95 \
--cluster_id=ci_rust_test_cluster \
--port 50051 &
MASTER_PID=$!
sleep 3
cd mooncake-store/rust
export LD_LIBRARY_PATH=$GITHUB_WORKSPACE/build/mooncake-asio:$GITHUB_WORKSPACE/build/mooncake-store/src:$GITHUB_WORKSPACE/build/mooncake-store/src/cachelib_memory_allocator:$GITHUB_WORKSPACE/build/mooncake-transfer-engine/src:$GITHUB_WORKSPACE/build/mooncake-transfer-engine/src/common/base:$GITHUB_WORKSPACE/build/mooncake-common/etcd:$LD_LIBRARY_PATH
export MOONCAKE_BUILD_DIR=$GITHUB_WORKSPACE/build
export MOONCAKE_STORE_LIB_DIR=$GITHUB_WORKSPACE/build/mooncake-store/src
export MOONCAKE_STORE_INCLUDE_DIR=$GITHUB_WORKSPACE/mooncake-store/include
# This job builds Mooncake with -DENABLE_ASAN=ON, so the C++ libraries
# the Rust package links against carry undefined __asan_* references. Opt
# in to linking the ASan runtime; build.rs emits -lasan first, which
# keeps libasan first in the initial library list as ASan requires.
# Non-sanitized builds leave this unset and link without ASan.
export MOONCAKE_LINK_ASAN=1
export MC_METADATA_SERVER=http://127.0.0.1:8080/metadata
export MC_RUST_STORE_RUN_INTEGRATION=true
export MC_RUST_STORE_MASTER_ADDR=127.0.0.1:50051
export MC_RUST_STORE_LOCAL_HOSTNAME=127.0.0.1
export MC_RUST_STORE_PROTOCOL=tcp
export MC_RUST_STORE_DEVICE_NAME=
cargo test --test minimal_smoke -- --nocapture
MC_RUST_BENCH_ITERATIONS=4 \
MC_RUST_BENCH_VALUE_SIZE=4096 \
MC_RUST_BENCH_WARMUP=1 \
cargo run --release --example store_benchmark
kill $MASTER_PID 2>/dev/null || true
shell: bash
- name: Run Go store binding integration tests
run: |
$GITHUB_WORKSPACE/build/mooncake-store/src/mooncake_master \
@ -137,7 +179,7 @@ jobs:
export LD_LIBRARY_PATH=$GITHUB_WORKSPACE/build/mooncake-common:$GITHUB_WORKSPACE/build/mooncake-store/src:$GITHUB_WORKSPACE/build/mooncake-transfer-engine/src:$GITHUB_WORKSPACE/build/mooncake-transfer-engine/src/common/base:$GITHUB_WORKSPACE/build/mooncake-common/etcd
export CGO_ENABLED=1
export CGO_CFLAGS="-I$GITHUB_WORKSPACE/mooncake-store/include -I$GITHUB_WORKSPACE/mooncake-transfer-engine/include"
export CGO_LDFLAGS="-L$GITHUB_WORKSPACE/build/mooncake-store/src -L$GITHUB_WORKSPACE/build/mooncake-store/src/cachelib_memory_allocator -L$GITHUB_WORKSPACE/build/mooncake-transfer-engine/src -L$GITHUB_WORKSPACE/build/mooncake-transfer-engine/src/common/base -L$GITHUB_WORKSPACE/build/mooncake-common -L$GITHUB_WORKSPACE/build/mooncake-common/etcd -lmooncake_store -lcachelib_memory_allocator -ltransfer_engine -lbase -lasio -letcd_wrapper -lstdc++ -lnuma -lglog -lgflags -libverbs -ljsoncpp -lzstd -lcurl -luring -lasan -lm -lgcov"
export CGO_LDFLAGS="-L$GITHUB_WORKSPACE/build/mooncake-store/src -L$GITHUB_WORKSPACE/build/mooncake-store/src/cachelib_memory_allocator -L$GITHUB_WORKSPACE/build/mooncake-transfer-engine/src -L$GITHUB_WORKSPACE/build/mooncake-transfer-engine/src/common/base -L$GITHUB_WORKSPACE/build/mooncake-common -L$GITHUB_WORKSPACE/build/mooncake-common/etcd -lmooncake_store -lcachelib_memory_allocator -ltransfer_engine -lbase -lasio -letcd_wrapper -lstdc++ -lnuma -lglog -lgflags -libverbs -lmlx5 -ljsoncpp -lzstd -lcurl -luring -lasan -lm -lgcov -lxxhash -lyaml-cpp"
# Link cudart if CUDA is available (needed for D2H staging in mooncake_store)
if [ -d /usr/local/cuda/lib64 ]; then export CGO_LDFLAGS="$CGO_LDFLAGS -L/usr/local/cuda/lib64 -lcudart"; fi
ASAN_OPTIONS=detect_leaks=0:verify_asan_link_order=0 MC_METADATA_SERVER=http://127.0.0.1:8080/metadata go test -v ./tests/...
@ -262,7 +304,7 @@ jobs:
bash -x dependencies.sh -y
mkdir build
cd build
cmake -G Ninja .. -DUSE_MUSA=ON -DUSE_MNNVL=ON -DUSE_ETCD=ON -DSTORE_USE_ETCD=ON -DUSE_CXL=ON -DUSE_TCP=ON -DBUILD_UNIT_TESTS=OFF -DBUILD_EXAMPLES=OFF
cmake -G Ninja .. -DUSE_MUSA=ON -DUSE_MNNVL=ON -DUSE_ETCD=ON -DSTORE_USE_ETCD=ON -DUSE_CXL=ON -DUSE_TCP=ON -DBUILD_UNIT_TESTS=OFF -DBUILD_EXAMPLES=OFF -DENABLE_DEBUG_SYMBOLS=OFF
shell: bash
- name: Build project
@ -273,6 +315,106 @@ jobs:
cmake --install .
shell: bash
build-arm64:
needs: [spell-check, clang-format, check-paths]
if: >-
(needs.check-paths.outputs.should-run-downstream == 'true' ||
github.event_name == 'workflow_dispatch') &&
(github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
github.event.action == 'opened' ||
contains(github.event.pull_request.labels.*.name, 'run-ci'))
runs-on: ubuntu-22.04-arm
strategy:
matrix:
python-version: ['3.10', '3.12']
env:
CI: "true"
CU13_BUILD: "1"
TORCH_CUDA_ARCH_LIST: "9.0"
SCCACHE_GHA_ENABLED: "true"
CUDA_HOME: "/usr/local/cuda"
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
- name: Install CUDA Toolkit 13.0 (arm64 SBSA)
run: |
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y cuda-toolkit-13-0
echo "/usr/local/cuda/bin" >> $GITHUB_PATH
/usr/local/cuda/bin/nvcc --version
shell: bash
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
- name: Configure sccache
uses: actions/github-script@v7
with:
script: |
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Configure project
run: |
sudo apt update -y
sudo bash -x dependencies.sh -y
mkdir build
cd build
cmake -G Ninja .. -DUSE_HTTP=ON -DUSE_CUDA=ON -DUSE_MNNVL=ON -DWITH_EP=OFF -DWITH_STORE_RUST=OFF -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release
shell: bash
- name: Build project
run: |
export CUDA_HOME=/usr/local/cuda
for dir in /usr/local/cuda/lib64/stubs /usr/local/cuda/targets/*/lib/stubs; do
if [ -d "$dir" ]; then
export LIBRARY_PATH="$dir:${LIBRARY_PATH:-}"
fi
done
cd build
cmake --build .
sudo cmake --install .
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_arm64
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
shell: bash
- name: Build Python wheel
run: |
PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-py${{ steps.generate_tag_arm64.outputs.python_version_tag }} ./scripts/build_wheel.sh
shell: bash
- name: Upload wheel artifact
uses: actions/upload-artifact@v4
with:
name: wheel-arm64-cuda13-py${{ steps.generate_tag_arm64.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_arm64.outputs.python_version_tag }}/*.whl
test-wheel-ubuntu:
needs: [spell-check, clang-format, build-flags]
if: >-
@ -430,7 +572,7 @@ jobs:
MC_FORCE_TCP: "true"
run: |
source test_env/bin/activate
python -m unittest mooncake-wheel.tests.test_mooncake_backend_cpu
python mooncake-pg/tests/test_pg_collectives.py
shell: bash
- name: Test Safetensor Functions
@ -458,6 +600,7 @@ jobs:
BUILD_WITH_EP: "1"
TORCH_CUDA_ARCH_LIST: "8.0;9.0"
SCCACHE_GHA_ENABLED: "true"
PIP_NO_CACHE_DIR: "1"
steps:
- uses: actions/checkout@v4
@ -496,10 +639,6 @@ jobs:
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Run sccache stat for check
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Install dependencies
run: |
sudo apt update -y
@ -518,7 +657,7 @@ jobs:
cd build
export PATH=/usr/local/nvidia/bin:/usr/local/nvidia/lib64:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LD_LIBRARY_PATH
cmake -G Ninja .. -DUSE_ETCD=OFF -DUSE_CXL=ON -DUSE_REDIS=ON -DUSE_HTTP=ON -DWITH_METRICS=ON -DBUILD_UNIT_TESTS=ON -DBUILD_EXAMPLES=ON -DENABLE_SCCACHE=ON -DUSE_CUDA=OFF -DUSE_MNNVL=OFF -DUSE_UB=OFF -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs"
cmake -G Ninja .. -DUSE_ETCD=OFF -DUSE_CXL=ON -DUSE_REDIS=ON -DUSE_HTTP=ON -DWITH_METRICS=ON -DBUILD_UNIT_TESTS=ON -DBUILD_EXAMPLES=ON -DENABLE_SCCACHE=ON -DUSE_CUDA=OFF -DUSE_MNNVL=OFF -DUSE_UB=OFF -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs" -DENABLE_DEBUG_SYMBOLS=OFF
cmake --build .
sudo cmake --install .
df -h
@ -528,7 +667,7 @@ jobs:
run: |
mkdir build
cd build
cmake -G Ninja .. -DUSE_ETCD=ON -DUSE_CXL=ON -DUSE_REDIS=ON -DUSE_HTTP=ON -DWITH_STORE=ON -DWITH_P2P_STORE=ON -DWITH_METRICS=ON -DBUILD_UNIT_TESTS=ON -DBUILD_EXAMPLES=ON -DENABLE_SCCACHE=ON -DUSE_CUDA=ON -DUSE_MNNVL=OFF -DUSE_UB=OFF -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs"
cmake -G Ninja .. -DUSE_ETCD=ON -DUSE_CXL=ON -DUSE_REDIS=ON -DUSE_HTTP=ON -DWITH_STORE=ON -DWITH_P2P_STORE=ON -DWITH_METRICS=ON -DBUILD_UNIT_TESTS=ON -DBUILD_EXAMPLES=ON -DENABLE_SCCACHE=ON -DUSE_CUDA=ON -DUSE_MNNVL=OFF -DUSE_UB=OFF -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs" -DENABLE_DEBUG_SYMBOLS=OFF
shell: bash
# TODO: lack USE_NVMEOF,USE_MNNVL
@ -545,7 +684,7 @@ jobs:
- name: Configure project with unit tests and examples
run: |
cd build
cmake -G Ninja .. -DBUILD_UNIT_TESTS=ON -DBUILD_EXAMPLES=ON -DWITH_STORE_RUST=ON -DENABLE_SCCACHE=ON
cmake -G Ninja .. -DBUILD_UNIT_TESTS=ON -DBUILD_EXAMPLES=ON -DWITH_STORE_RUST=ON -DENABLE_SCCACHE=ON -DENABLE_DEBUG_SYMBOLS=OFF
shell: bash
- name: Build project with unit tests and examples
@ -557,19 +696,28 @@ jobs:
sudo cmake --install .
shell: bash
- name: Check Mooncake Store Rust bindings and example
- name: Check Mooncake Store Rust bindings, examples, and tests
run: |
# libcuda.so.1 (SONAME of the CUDA stub) must be findable at runtime.
# The toolkit stubs dir only ships libcuda.so; create the versioned symlink.
if [ -f /usr/local/cuda/lib64/stubs/libcuda.so ] && \
[ ! -e /usr/local/cuda/lib64/stubs/libcuda.so.1 ]; then
sudo ln -s libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so.1
fi
export LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LD_LIBRARY_PATH
cd mooncake-store/rust
export MOONCAKE_BUILD_DIR=$GITHUB_WORKSPACE/build
cargo test --lib
MOONCAKE_STORE_LIB_DIR=$GITHUB_WORKSPACE/build/mooncake-store/src \
MOONCAKE_STORE_INCLUDE_DIR=$GITHUB_WORKSPACE/mooncake-store/include \
cargo check --example basic_usage --tests
cargo test --examples --tests --no-run
shell: bash
- name: Configure project
run: |
cd build
rm -r */tests
cmake -G Ninja .. -DBUILD_UNIT_TESTS=OFF -DBUILD_EXAMPLES=OFF -DUSE_HTTP=ON -DENABLE_SCCACHE=ON -DUSE_CXL=ON -DWITH_EP=ON -DEP_TORCH_VERSIONS="2.9.0;2.9.1;2.10.0;2.11.0"
cmake -G Ninja .. -DBUILD_UNIT_TESTS=OFF -DBUILD_EXAMPLES=OFF -DUSE_HTTP=ON -DENABLE_SCCACHE=ON -DUSE_CXL=ON -DWITH_EP=ON -DEP_TORCH_VERSIONS="2.9.1;2.10.0;2.11.0;2.12.0;2.12.1" -DENABLE_DEBUG_SYMBOLS=OFF
shell: bash
- name: Build project
@ -581,6 +729,26 @@ jobs:
sudo cmake --install .
shell: bash
- name: Configure project with TENT
run: |
mkdir build-tent
cd build-tent
cmake -G Ninja .. -DUSE_TENT=ON -DUSE_HTTP=ON -DENABLE_SCCACHE=ON -DBUILD_UNIT_TESTS=ON -DBUILD_EXAMPLES=ON -DENABLE_DEBUG_SYMBOLS=OFF
shell: bash
- name: Build project with TENT
run: |
cd build-tent
cmake --build .
sudo cmake --install .
shell: bash
- name: Test (TENT)
run: |
cd build-tent
ctest --test-dir mooncake-transfer-engine/tent/tests -j --output-on-failure
shell: bash
- name: Build nvlink_allocator.so
run: |
mkdir -p build/mooncake-transfer-engine/nvlink-allocator
@ -591,6 +759,11 @@ jobs:
bash build.sh ../../build/mooncake-transfer-engine/nvlink-allocator/
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_flags
run: |
@ -638,10 +811,10 @@ jobs:
spell-check:
name: Spell Check with Typos
if: >-
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
github.event.action == 'opened' ||
contains(github.event.pull_request.labels.*.name, 'run-ci')
(github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
github.event.action == 'opened' ||
contains(github.event.pull_request.labels.*.name, 'run-ci'))
runs-on: ubuntu-22.04
steps:
- name: Checkout Actions Repository
@ -654,10 +827,10 @@ jobs:
clang-format:
name: Check code format
if: >-
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
github.event.action == 'opened' ||
contains(github.event.pull_request.labels.*.name, 'run-ci')
(github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
github.event.action == 'opened' ||
contains(github.event.pull_request.labels.*.name, 'run-ci'))
runs-on: ubuntu-22.04
steps:
- name: Checkout Actions Repository
@ -680,7 +853,7 @@ jobs:
echo "Error: code_format.sh not found or not executable"
exit 1
fi
# Determine base ref for comparison
if [ "${{ github.event_name }}" == "pull_request" ]; then
# For PRs: compare against the target branch
@ -701,6 +874,48 @@ jobs:
shell: bash
docs-check:
name: Check Sphinx docs build
if: >-
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
github.event.action == 'opened' ||
contains(github.event.pull_request.labels.*.name, 'run-ci')
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 2
persist-credentials: false
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
docs:
- 'docs/**'
- 'requirements_docs.txt'
- name: Set up Python
if: steps.filter.outputs.docs == 'true'
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install documentation dependencies
if: steps.filter.outputs.docs == 'true'
run: |
python -m pip install --upgrade pip
pip install -r requirements_docs.txt
- name: Build docs with strict mode
if: steps.filter.outputs.docs == 'true'
run: |
cd docs
make html SPHINXOPTS=-W
shell: bash
check-paths:
if: >-
github.event_name == 'push' ||
@ -746,6 +961,18 @@ jobs:
uses: ./.github/workflows/ci_cu13.yml
secrets: inherit
build-wheel-efa:
needs: [spell-check, clang-format, check-paths]
if: >-
(needs.check-paths.outputs.should-run-downstream == 'true' ||
github.event_name == 'workflow_dispatch') &&
(github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
github.event.action == 'opened' ||
contains(github.event.pull_request.labels.*.name, 'run-ci'))
uses: ./.github/workflows/ci_efa.yml
secrets: inherit
ascend-test:
needs: [build, check-paths]
if: needs.check-paths.outputs.should-run-downstream == 'true'
@ -764,12 +991,15 @@ jobs:
needs:
- spell-check
- clang-format
- docs-check
- build
- build-musa
- build-arm64
- build-flags
- build-docker
- test-wheel-ubuntu
- build-wheel-cu13
- build-wheel-efa
- ascend-test
- integration-test
runs-on: ubuntu-latest

12
.github/workflows/ci_ascend.yml vendored
View File

@ -4,7 +4,7 @@ on:
workflow_call:
inputs:
checkout_ref:
description: 'Git ref to checkout (PR head SHA for pull_request_target)'
description: 'Git ref to checkout (PR head SHA)'
required: false
type: string
@ -49,7 +49,7 @@ jobs:
if: steps.checkout_code.outcome == 'failure'
shell: bash
env:
ASCEND_GITHUB_MIRROR_URLS: ${{ vars.ASCEND_GITHUB_MIRROR_URLS }}
ASCEND_GITHUB_MIRROR_URLS: 'https://ghfast.top/'
CHECKOUT_REF: ${{ inputs.checkout_ref || github.sha }}
run: |
set -euo pipefail
@ -108,7 +108,7 @@ jobs:
- name: Configure CMake
shell: bash
env:
ASCEND_GITHUB_MIRROR_URLS: ${{ vars.ASCEND_GITHUB_MIRROR_URLS }}
ASCEND_GITHUB_MIRROR_URLS: 'https://ghfast.top/'
run: |
source /usr/local/Ascend/cann-9.0.0/set_env.sh
pwd
@ -163,7 +163,9 @@ jobs:
cmake .. \
-DUSE_ASCEND_DIRECT=ON \
-DBUILD_EXAMPLES=OFF \
-DBUILD_UNIT_TESTS=OFF
-DBUILD_UNIT_TESTS=OFF \
-DENABLE_DEBUG_SYMBOLS=OFF \
-DBUILD_BENCHMARK=OFF
- name: Build
shell: bash
@ -171,7 +173,7 @@ jobs:
source /usr/local/Ascend/cann-9.0.0/set_env.sh
echo "Building..."
cd build
cmake --build . -j$(nproc)
cmake --build . -j48
cmake --install .
echo "Mooncake installed successfully."

14
.github/workflows/ci_cu13.yml vendored
View File

@ -52,10 +52,6 @@ jobs:
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Run sccache stat for check
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Install dependencies
run: |
sudo apt update -y
@ -75,7 +71,7 @@ jobs:
-DWITH_STORE=ON \
-DWITH_P2P_STORE=ON \
-DWITH_EP=ON \
-DEP_TORCH_VERSIONS="2.9.0;2.9.1;2.10.0;2.11.0" \
-DEP_TORCH_VERSIONS="2.9.1;2.10.0;2.11.0;2.12.0;2.12.1" \
-DWITH_METRICS=ON \
-DBUILD_UNIT_TESTS=OFF \
-DBUILD_EXAMPLES=ON \
@ -83,7 +79,8 @@ jobs:
-DBUILD_BENCHMARK=ON \
-DUSE_CUDA=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs"
-DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs" \
-DENABLE_DEBUG_SYMBOLS=OFF
shell: bash
- name: Build project
@ -106,6 +103,11 @@ jobs:
bash build.sh ../../build/mooncake-transfer-engine/nvlink-allocator/
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag
run: |

154
.github/workflows/ci_efa.yml vendored Normal file
View File

@ -0,0 +1,154 @@
name: 'Build Wheel (AWS EFA)'
on:
workflow_call: {}
# Builds the AWS EFA (libfabric) wheel variants on a stock ubuntu runner.
# No EFA hardware is required to *build*: USE_EFA only needs the libfabric
# headers/lib to compile and link. auditwheel later excludes libfabric/libefa
# from the wheel so they resolve to the user's system EFA install
# (/opt/amazon/efa/lib) at runtime. The distro libfabric (1.x) is ABI-forward-
# compatible with the AWS EFA libfabric (2.x) that loads at runtime; the EFA
# transport only uses long-stable fi_* core APIs.
#
# Two variants, since the EFA transport's memory path is CUDA-aware
# (FI_HMEM_CUDA / GPUDirect under USE_CUDA=ON, FI_HMEM=system otherwise):
# efa USE_CUDA=ON (GPU)
# efa-non-cuda USE_CUDA=OFF (CPU/DRAM)
# PR validation builds one python version per variant to keep CI cheap;
# the release workflow builds the full python matrix.
jobs:
build-wheel-efa:
runs-on: ubuntu-22.04
strategy:
matrix:
include:
- variant: cuda
use_cuda: "ON"
build_env: "EFA_BUILD"
python-version: "3.12"
- variant: non-cuda
use_cuda: "OFF"
build_env: "EFA_NON_CUDA_BUILD"
python-version: "3.10"
env:
TORCH_CUDA_ARCH_LIST: "8.0;9.0"
SCCACHE_GHA_ENABLED: "true"
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo rm -rf /usr/local/lib/android
df -h
- name: Install CUDA Toolkit
if: matrix.use_cuda == 'ON'
uses: Jimver/cuda-toolkit@v0.2.24
with:
cuda: '12.8.1'
method: 'network'
sub-packages: '["nvcc", "nvrtc-dev"]'
non-cuda-sub-packages: '["libcusparse-dev", "libcublas-dev", "libcusolver-dev"]'
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
- name: Configure sccache
uses: actions/github-script@v7
with:
script: |
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Install dependencies
run: |
sudo apt update -y
sudo apt install -y ninja-build libfabric-dev libfabric1
sudo bash -x dependencies.sh -y
df -h
shell: bash
- name: Configure project
run: |
mkdir build
cd build
EXTRA_FLAGS=""
if [ "${{ matrix.use_cuda }}" = "ON" ]; then
EXTRA_FLAGS="-DCMAKE_EXE_LINKER_FLAGS=-L/usr/local/cuda/lib64/stubs"
fi
cmake -G Ninja .. \
-DUSE_ETCD=ON \
-DUSE_HTTP=ON \
-DWITH_STORE=ON \
-DWITH_METRICS=ON \
-DBUILD_UNIT_TESTS=OFF \
-DBUILD_EXAMPLES=ON \
-DENABLE_SCCACHE=ON \
-DBUILD_BENCHMARK=ON \
-DUSE_EFA=ON \
-DUSE_CUDA=${{ matrix.use_cuda }} \
-DLIBFABRIC_INCLUDE_DIR=/usr/include \
-DLIBFABRIC_LIBRARY=/usr/lib/x86_64-linux-gnu/libfabric.so \
-DCMAKE_BUILD_TYPE=Release \
-DENABLE_DEBUG_SYMBOLS=OFF \
$EXTRA_FLAGS
shell: bash
- name: Build project
run: |
if [ "${{ matrix.use_cuda }}" = "ON" ]; then
export LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LD_LIBRARY_PATH
export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LIBRARY_PATH
fi
cd build
cmake --build .
sudo cmake --install .
df -h
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
shell: bash
- name: Build Python wheel
run: |
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export ${{ matrix.build_env }}=1
PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-py${{ steps.generate_tag.outputs.python_version_tag }} ./scripts/build_wheel.sh
shell: bash
- name: Verify libfabric is excluded from the wheel
run: |
WHL=$(ls mooncake-wheel/dist-py${{ steps.generate_tag.outputs.python_version_tag }}/*.whl | head -1)
echo "Inspecting $WHL"
if unzip -l "$WHL" | grep -iE 'libfabric|libefa'; then
echo "::error::libfabric/libefa must NOT be bundled in the EFA wheel"
exit 1
fi
echo "OK: libfabric/libefa correctly excluded (resolve to system EFA at runtime)"
shell: bash
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: mooncake-wheel-efa-${{ matrix.variant }}-ubuntu-py${{ steps.generate_tag.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag.outputs.python_version_tag }}/*.whl

29
.github/workflows/code-review.yml vendored Normal file
View File

@ -0,0 +1,29 @@
name: Qoder Auto Code Review
on:
pull_request:
types: [opened, synchronize, reopened]
jobs:
qoder-review:
# Skip fork and cross-repo PRs (head repo must match this repository)
if: github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Run Qoder Code Review
uses: QoderAI/qoder-action@v0
with:
qoder_personal_access_token: ${{ secrets.QODER_PERSONAL_ACCESS_TOKEN }}
prompt: |
/review-pr
REPO:${{ github.repository }} PR_NUMBER:${{ github.event.pull_request.number }}

71
.github/workflows/issue-bot.yml vendored Normal file
View File

@ -0,0 +1,71 @@
name: Issue Bot
on:
issues:
types: [opened]
schedule:
# Daily at 03:00 UTC
- cron: "0 3 * * *"
workflow_dispatch:
permissions:
issues: write
jobs:
welcome-and-assign:
if: github.event_name == 'issues'
runs-on: ubuntu-latest
steps:
- name: Post welcome comment
uses: actions/github-script@v7
with:
script: |
const issue = context.payload.issue;
const body = [
`Thanks for opening this issue, @${issue.user.login}!`,
'',
'| Field | Value |',
'|-------|-------|',
`| **Issue** | #${issue.number} |`,
`| **GitHub user ID** | \`${issue.user.id}\` |`,
`| **Reporter** | @${issue.user.login} |`,
'',
'A maintainer will triage this when possible. To help us respond faster, please include:',
'',
'- Mooncake version or commit SHA',
'- Environment (OS, CUDA/driver, RDMA stack if relevant)',
'- Steps to reproduce and expected vs. actual behavior',
'',
`Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/${context.repo.owner}/${context.repo.repo}/blob/main/CONTRIBUTING.md)`,
'',
'> This message was posted automatically by the issue bot.',
].join('\n');
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issue.number,
body,
});
close-stale-issues:
if: github.event_name != 'issues'
runs-on: ubuntu-latest
steps:
- name: Close inactive issues (3+ months)
uses: actions/stale@v9
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
days-before-stale: 90
days-before-close: 7
stale-issue-label: stale
close-issue-label: auto-closed
stale-issue-message: >
This issue has had no activity for 90 days and will be closed in 7 days
if there is no further activity. Please comment or react if it should
stay open.
close-issue-message: >
Closing due to 3 months of inactivity. If this is still relevant,
please comment and we can reopen.
exempt-issue-labels: pinned,keep-open,security
operations-per-run: 100

31
.github/workflows/pr-tidy.yml vendored Normal file
View File

@ -0,0 +1,31 @@
name: PR Tidy
on:
pull_request_target:
types: [opened]
permissions:
pull-requests: write
jobs:
tidy:
runs-on: ubuntu-latest
steps:
- name: Clean PR description
uses: actions/github-script@v7
with:
script: |
const pr = context.payload.pull_request;
if (!pr.body) return;
// Remove HTML comment blocks (template instructions)
let body = pr.body.replace(/<!--[\s\S]*?-->/g, '').trim();
if (body !== pr.body) {
await github.rest.pulls.update({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: pr.number,
body: body
});
}

549
.github/workflows/pre-release.yaml vendored Normal file
View File

@ -0,0 +1,549 @@
name: Pre-Release
# Dry-run of the release pipelines: build wheels like Release / Release Non-CUDA /
# Release CUDA 13, validate artifacts, but do not create a GitHub Release or publish to PyPI.
#
# Trigger by pushing a pre-release tag, for example:
# git tag v1.0.0-rc1 && git push origin v1.0.0-rc1
on:
push:
tags:
- 'v*-rc*'
- 'v*-alpha*'
- 'v*-beta*'
- 'v*-pre*'
env:
SCCACHE_GHA_ENABLED: "true"
jobs:
build-cuda:
name: Build (CUDA 12)
runs-on: ubuntu-22.04
permissions:
contents: read
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12', '3.13']
env:
BUILD_WITH_EP: "1"
TORCH_CUDA_ARCH_LIST: "8.0;9.0"
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Set version from tag
run: echo "VERSION=${GITHUB_REF_NAME#v}" >> $GITHUB_ENV
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo rm -rf /usr/local/lib/android
df -h
- name: Install CUDA Toolkit
uses: Jimver/cuda-toolkit@v0.2.24
with:
cuda: '12.8.1'
linux-local-args: '["--toolkit"]'
method: 'network'
sub-packages: '["nvcc", "nvrtc-dev"]'
non-cuda-sub-packages: '["libcusparse-dev", "libcublas-dev", "libcusolver-dev"]'
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
- name: Configure sccache
uses: actions/github-script@v7
with:
script: |
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Configure project
run: |
sudo apt update -y
sudo bash -x dependencies.sh -y
mkdir build
cd build
cmake .. -DBUILD_UNIT_TESTS=OFF -DUSE_HTTP=ON -DUSE_ETCD=ON -DUSE_CUDA=ON -DWITH_EP=ON -DEP_TORCH_VERSIONS="2.9.1;2.10.0;2.11.0;2.12.0;2.12.1" -DSTORE_USE_ETCD=ON -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release
shell: bash
- name: Build project
run: |
export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LIBRARY_PATH
cd build
make -j
sudo -E make install
shell: bash
- name: Build nvlink_allocator.so
run: |
export PATH=/usr/local/nvidia/bin:/usr/local/nvidia/lib64:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LD_LIBRARY_PATH
export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LIBRARY_PATH
mkdir -p build/mooncake-transfer-engine/nvlink-allocator
cd mooncake-transfer-engine/nvlink-allocator
bash build.sh ../../build/mooncake-transfer-engine/nvlink-allocator/
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_release
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
shell: bash
- name: Build Python wheel
run: |
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-py${{ steps.generate_tag_release.outputs.python_version_tag }} ./scripts/build_wheel.sh
env:
VERSION: ${{ env.VERSION }}
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: pre-release-cuda-py${{ steps.generate_tag_release.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
build-non-cuda:
name: Build (Non-CUDA)
runs-on: ubuntu-22.04
permissions:
contents: read
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12', '3.13']
env:
BUILD_WITH_EP: "0"
NON_CUDA_BUILD: "1"
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Set version from tag
run: echo "VERSION=${GITHUB_REF_NAME#v}" >> $GITHUB_ENV
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
- name: Configure sccache
uses: actions/github-script@v7
with:
script: |
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Configure project
run: |
sudo apt update -y
sudo bash -x dependencies.sh -y
mkdir build
cd build
cmake .. -DUSE_HTTP=ON -DUSE_ETCD=ON -DUSE_CUDA=OFF -DWITH_EP=OFF -DSTORE_USE_ETCD=ON -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release
shell: bash
- name: Build project
run: |
cd build
make -j
sudo make install
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_release
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
shell: bash
- name: Build Python wheel
run: |
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-py${{ steps.generate_tag_release.outputs.python_version_tag }} ./scripts/build_wheel.sh
env:
VERSION: ${{ env.VERSION }}
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: pre-release-non-cuda-py${{ steps.generate_tag_release.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
build-cuda13:
name: Build (CUDA 13)
runs-on: ubuntu-22.04
permissions:
contents: read
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12', '3.13']
env:
BUILD_WITH_EP: "1"
CU13_BUILD: "1"
TORCH_CUDA_ARCH_LIST: "8.0;9.0"
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Set version from tag
run: echo "VERSION=${GITHUB_REF_NAME#v}" >> $GITHUB_ENV
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo rm -rf /usr/local/lib/android
df -h
- name: Install CUDA Toolkit 13
uses: Jimver/cuda-toolkit@v0.2.29
with:
cuda: '13.0.2'
linux-local-args: '["--toolkit"]'
method: 'network'
sub-packages: '["nvcc", "nvrtc-dev"]'
non-cuda-sub-packages: '["libcusparse-dev", "libcublas-dev", "libcusolver-dev"]'
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
- name: Configure sccache
uses: actions/github-script@v7
with:
script: |
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Configure project
run: |
sudo apt update -y
sudo bash -x dependencies.sh -y
mkdir build
cd build
cmake .. -DBUILD_UNIT_TESTS=OFF -DUSE_HTTP=ON -DUSE_ETCD=ON -DUSE_CUDA=ON -DWITH_EP=ON -DEP_TORCH_VERSIONS="2.9.1;2.10.0;2.11.0;2.12.0;2.12.1" -DSTORE_USE_ETCD=ON -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release
shell: bash
- name: Build project
run: |
export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LIBRARY_PATH
cd build
make -j
sudo make install
shell: bash
- name: Build nvlink_allocator.so
run: |
export PATH=/usr/local/nvidia/bin:/usr/local/nvidia/lib64:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LD_LIBRARY_PATH
export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LIBRARY_PATH
mkdir -p build/mooncake-transfer-engine/nvlink-allocator
cd mooncake-transfer-engine/nvlink-allocator
bash build.sh ../../build/mooncake-transfer-engine/nvlink-allocator/
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_release
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
shell: bash
- name: Build Python wheel
run: |
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-py${{ steps.generate_tag_release.outputs.python_version_tag }} ./scripts/build_wheel.sh
env:
VERSION: ${{ env.VERSION }}
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: pre-release-cuda13-py${{ steps.generate_tag_release.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
build-cuda-arm64:
name: Build (CUDA 12, arm64)
runs-on: ubuntu-22.04-arm
permissions:
contents: read
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12', '3.13']
env:
TORCH_CUDA_ARCH_LIST: "9.0"
CUDA_HOME: "/usr/local/cuda"
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Set version from tag
run: echo "VERSION=${GITHUB_REF_NAME#v}" >> $GITHUB_ENV
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
- name: Install CUDA Toolkit 12.8 (arm64 SBSA)
run: |
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y cuda-toolkit-12-8
echo "/usr/local/cuda/bin" >> $GITHUB_PATH
/usr/local/cuda/bin/nvcc --version
shell: bash
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
- name: Configure sccache
uses: actions/github-script@v7
with:
script: |
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Configure project
run: |
sudo apt update -y
sudo bash -x dependencies.sh -y
mkdir build
cd build
cmake -G Ninja .. -DUSE_HTTP=ON -DUSE_CUDA=ON -DUSE_MNNVL=ON -DWITH_EP=OFF -DWITH_STORE_RUST=OFF -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release
shell: bash
- name: Build project
run: |
export CUDA_HOME=/usr/local/cuda
for dir in /usr/local/cuda/lib64/stubs /usr/local/cuda/targets/*/lib/stubs; do
if [ -d "$dir" ]; then
export LIBRARY_PATH="$dir:${LIBRARY_PATH:-}"
fi
done
cd build
cmake --build .
sudo cmake --install .
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_arm64
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
shell: bash
- name: Build Python wheel
run: |
export CUDA_HOME=/usr/local/cuda
PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-py${{ steps.generate_tag_arm64.outputs.python_version_tag }} ./scripts/build_wheel.sh
env:
VERSION: ${{ env.VERSION }}
shell: bash
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: pre-release-cuda-arm64-py${{ steps.generate_tag_arm64.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_arm64.outputs.python_version_tag }}/*.whl
build-cuda13-arm64:
name: Build (CUDA 13, arm64)
runs-on: ubuntu-22.04-arm
permissions:
contents: read
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12', '3.13']
env:
CU13_BUILD: "1"
TORCH_CUDA_ARCH_LIST: "9.0"
CUDA_HOME: "/usr/local/cuda"
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Set version from tag
run: echo "VERSION=${GITHUB_REF_NAME#v}" >> $GITHUB_ENV
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
- name: Install CUDA Toolkit 13.0 (arm64 SBSA)
run: |
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y cuda-toolkit-13-0
echo "/usr/local/cuda/bin" >> $GITHUB_PATH
/usr/local/cuda/bin/nvcc --version
shell: bash
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
- name: Configure sccache
uses: actions/github-script@v7
with:
script: |
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Configure project
run: |
sudo apt update -y
sudo bash -x dependencies.sh -y
mkdir build
cd build
cmake -G Ninja .. -DUSE_HTTP=ON -DUSE_CUDA=ON -DUSE_MNNVL=ON -DWITH_EP=OFF -DWITH_STORE_RUST=OFF -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release
shell: bash
- name: Build project
run: |
export CUDA_HOME=/usr/local/cuda
for dir in /usr/local/cuda/lib64/stubs /usr/local/cuda/targets/*/lib/stubs; do
if [ -d "$dir" ]; then
export LIBRARY_PATH="$dir:${LIBRARY_PATH:-}"
fi
done
cd build
cmake --build .
sudo cmake --install .
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_arm64
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
shell: bash
- name: Build Python wheel
run: |
export CUDA_HOME=/usr/local/cuda
PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-py${{ steps.generate_tag_arm64.outputs.python_version_tag }} ./scripts/build_wheel.sh
env:
VERSION: ${{ env.VERSION }}
shell: bash
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: pre-release-cuda13-arm64-py${{ steps.generate_tag_arm64.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_arm64.outputs.python_version_tag }}/*.whl
validate-release:
name: Validate release artifacts
needs: [build-cuda, build-non-cuda, build-cuda13, build-cuda-arm64, build-cuda13-arm64]
runs-on: ubuntu-22.04
permissions:
contents: read
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Download all pre-release wheel artifacts
uses: actions/download-artifact@v4
with:
path: mooncake-wheel/dist-all
pattern: pre-release-*
- name: Prepare wheels for validation
run: |
mkdir -p mooncake-wheel/dist-release
find mooncake-wheel/dist-all -name "*.whl" -exec cp {} mooncake-wheel/dist-release/ \;
echo "Pre-release tag: ${GITHUB_REF_NAME}"
echo "Collected wheels:"
ls -la mooncake-wheel/dist-release/
wheel_count=$(find mooncake-wheel/dist-release -name "*.whl" | wc -l)
echo "wheel_count=${wheel_count}" >> "$GITHUB_ENV"
if [ "${wheel_count}" -lt 20 ]; then
echo "Expected at least 20 wheels (x86: 4 Python versions x 3 variants; arm64: 4 Python versions x 2 CUDA variants), found ${wheel_count}"
exit 1
fi
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Validate wheels with twine
run: |
pip install twine
twine check mooncake-wheel/dist-release/*.whl
- name: Upload validated wheels as workflow artifacts
uses: actions/upload-artifact@v4
with:
name: pre-release-wheels-${{ github.ref_name }}
path: mooncake-wheel/dist-release/*.whl
retention-days: 14
- name: Pre-release validation summary
run: |
echo "## Pre-Release validation passed" >> "$GITHUB_STEP_SUMMARY"
echo "" >> "$GITHUB_STEP_SUMMARY"
echo "- Tag: \`${GITHUB_REF_NAME}\`" >> "$GITHUB_STEP_SUMMARY"
echo "- Wheels built: ${wheel_count}" >> "$GITHUB_STEP_SUMMARY"
echo "- PyPI publish: skipped (pre-release dry run)" >> "$GITHUB_STEP_SUMMARY"
echo "- GitHub Release upload: skipped (pre-release dry run)" >> "$GITHUB_STEP_SUMMARY"

142
.github/workflows/release-cuda13.yaml vendored
View File

@ -10,6 +10,7 @@ env:
jobs:
build:
runs-on: ubuntu-22.04
container: pytorch/manylinux2_28-builder:cuda13.0
permissions:
contents: write
strategy:
@ -23,27 +24,15 @@ jobs:
- name: Checkout source
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Mark workspace safe for git (container runs as root)
run: git config --global --add safe.directory '*'
- name: Free up disk space
- name: Select Python ${{ matrix.python-version }} from manylinux image
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo rm -rf /usr/local/lib/android
df -h
- name: Install CUDA Toolkit 13
uses: Jimver/cuda-toolkit@v0.2.29
with:
cuda: '13.0.2'
linux-local-args: '["--toolkit"]'
method: 'network'
sub-packages: '["nvcc", "nvrtc-dev"]'
non-cuda-sub-packages: '["libcusparse-dev", "libcublas-dev", "libcusolver-dev"]'
PYV_NODOT=$(echo "${{ matrix.python-version }}" | tr -d '.')
PYBIN="/opt/python/cp${PYV_NODOT}-cp${PYV_NODOT}/bin"
echo "$PYBIN" >> "$GITHUB_PATH"
"$PYBIN/pip" install --quiet "cmake<4" setuptools wheel
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
@ -55,17 +44,13 @@ jobs:
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Run sccache stat for check
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Configure project
run: |
sudo apt update -y
sudo bash -x dependencies.sh -y
bash -x dependencies.sh -y
echo "/usr/local/go/bin" >> "$GITHUB_PATH"
mkdir build
cd build
cmake .. -DBUILD_UNIT_TESTS=OFF -DUSE_HTTP=ON -DUSE_ETCD=ON -DUSE_CUDA=ON -DWITH_EP=ON -DEP_TORCH_VERSIONS="2.9.0;2.9.1;2.10.0;2.11.0" -DSTORE_USE_ETCD=ON -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release
cmake .. -DBUILD_UNIT_TESTS=OFF -DUSE_HTTP=ON -DUSE_ETCD=ON -DUSE_CUDA=ON -DWITH_EP=ON -DEP_TORCH_VERSIONS="2.9.1;2.10.0;2.11.0;2.12.0;2.12.1" -DSTORE_USE_ETCD=ON -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release -DPython3_EXECUTABLE="$(which python3)"
shell: bash
- name: Build project
@ -73,7 +58,7 @@ jobs:
export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LIBRARY_PATH
cd build
make -j
sudo make install
make install
shell: bash
- name: Build nvlink_allocator.so
@ -86,6 +71,11 @@ jobs:
bash build.sh ../../build/mooncake-transfer-engine/nvlink-allocator/
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_release
run: |
@ -106,8 +96,102 @@ jobs:
name: mooncake-wheel-cuda13-py${{ steps.generate_tag_release.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
build-arm64:
if: ${{ !contains(github.ref_name, '-') }}
runs-on: ubuntu-22.04-arm
permissions:
contents: write
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12', '3.13']
env:
CU13_BUILD: "1"
TORCH_CUDA_ARCH_LIST: "9.0"
CUDA_HOME: "/usr/local/cuda"
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
- name: Install CUDA Toolkit 13.0 (arm64 SBSA)
run: |
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y cuda-toolkit-13-0
echo "/usr/local/cuda/bin" >> $GITHUB_PATH
/usr/local/cuda/bin/nvcc --version
shell: bash
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
- name: Configure sccache
uses: actions/github-script@v7
with:
script: |
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Configure project
run: |
sudo apt update -y
sudo bash -x dependencies.sh -y
mkdir build
cd build
cmake -G Ninja .. -DUSE_HTTP=ON -DUSE_CUDA=ON -DUSE_MNNVL=ON -DWITH_EP=OFF -DWITH_STORE_RUST=OFF -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release
shell: bash
- name: Build project
run: |
export CUDA_HOME=/usr/local/cuda
for dir in /usr/local/cuda/lib64/stubs /usr/local/cuda/targets/*/lib/stubs; do
if [ -d "$dir" ]; then
export LIBRARY_PATH="$dir:${LIBRARY_PATH:-}"
fi
done
cd build
cmake --build .
sudo cmake --install .
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_release
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
shell: bash
- name: Build Python wheel
run: |
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:-}:/usr/local/lib"
PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-py${{ steps.generate_tag_release.outputs.python_version_tag }} ./scripts/build_wheel.sh
shell: bash
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: mooncake-wheel-cuda13-arm64-py${{ steps.generate_tag_release.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
publish-release:
needs: build
if: ${{ !contains(github.ref_name, '-') }}
needs: [build, build-arm64]
runs-on: ubuntu-22.04
permissions:
contents: write
@ -120,7 +204,7 @@ jobs:
uses: actions/download-artifact@v4
with:
path: mooncake-wheel/dist-all
pattern: mooncake-wheel-cuda13-py*
pattern: mooncake-wheel-cuda13*
- name: Prepare wheels for release
run: |

155
.github/workflows/release-efa-non-cuda.yaml vendored Normal file
View File

@ -0,0 +1,155 @@
name: Release EFA Non-CUDA
on:
push:
tags:
- 'v*'
# Publishes the AWS EFA (libfabric) non-CUDA wheel variant:
# mooncake-transfer-engine-efa-non-cuda USE_EFA=ON USE_CUDA=OFF (CPU/DRAM only)
# The CUDA variant (mooncake-transfer-engine-efa) is built by release-efa.yaml
# — split into its own workflow because each PyPI package publishes from a
# dedicated release workflow (trusted publisher / artifact pattern is
# per-package), mirroring release.yaml vs release-non-cuda.yaml.
#
# No EFA hardware is needed to build: USE_EFA only needs libfabric headers/lib
# to compile/link. auditwheel excludes libfabric/libefa from the wheel so they
# resolve to the user's system AWS EFA install (/opt/amazon/efa/lib) at
# runtime. The distro libfabric (1.x) used to build is ABI-forward-compatible
# with the AWS EFA libfabric (2.x) loaded at runtime; the EFA transport uses
# only long-stable fi_* core APIs.
env:
SCCACHE_GHA_ENABLED: "true"
jobs:
build:
runs-on: ubuntu-22.04
permissions:
contents: write
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12', '3.13']
env:
EFA_NON_CUDA_BUILD: "1"
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo rm -rf /usr/local/lib/android
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
- name: Configure sccache
uses: actions/github-script@v7
with:
script: |
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Configure project
run: |
sudo apt update -y
sudo apt install -y libfabric-dev libfabric1
sudo bash -x dependencies.sh -y
mkdir build
cd build
cmake .. \
-DUSE_HTTP=ON \
-DUSE_ETCD=ON \
-DUSE_CUDA=OFF \
-DWITH_EP=OFF \
-DSTORE_USE_ETCD=ON \
-DUSE_EFA=ON \
-DLIBFABRIC_INCLUDE_DIR=/usr/include \
-DLIBFABRIC_LIBRARY=/usr/lib/x86_64-linux-gnu/libfabric.so \
-DENABLE_SCCACHE=ON \
-DCMAKE_BUILD_TYPE=Release
shell: bash
- name: Build project
run: |
cd build
make -j
sudo make install
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_release
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
shell: bash
- name: Build Python wheel
run: |
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-py${{ steps.generate_tag_release.outputs.python_version_tag }} ./scripts/build_wheel.sh
env:
VERSION: ${{ env.VERSION }}
- name: Verify libfabric is excluded from the wheel
run: |
WHL=$(ls mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl | head -1)
echo "Inspecting $WHL"
if unzip -l "$WHL" | grep -iE 'libfabric|libefa'; then
echo "::error::libfabric/libefa must NOT be bundled in the EFA wheel"
exit 1
fi
echo "OK: libfabric/libefa correctly excluded"
shell: bash
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: mooncake-wheel-efa-non-cuda-py${{ steps.generate_tag_release.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
publish-release:
if: ${{ !contains(github.ref_name, '-') }}
needs: build
runs-on: ubuntu-22.04
permissions:
contents: write
id-token: write
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Download all wheel artifacts
uses: actions/download-artifact@v4
with:
path: mooncake-wheel/dist-all
pattern: mooncake-wheel-efa-non-cuda-py*
- name: Prepare wheels for release
run: |
mkdir -p mooncake-wheel/dist-release
find mooncake-wheel/dist-all -name "*.whl" -exec cp {} mooncake-wheel/dist-release/ \;
echo "Collected wheels for release:"
ls -la mooncake-wheel/dist-release/
- name: Upload wheels to GitHub Release
uses: softprops/action-gh-release@v1
with:
files: mooncake-wheel/dist-release/*.whl
- name: Publish package to PyPI
if: github.repository == 'kvcache-ai/Mooncake'
uses: pypa/gh-action-pypi-publish@release/v1
with:
packages-dir: mooncake-wheel/dist-release/
password: ${{ secrets.PYPI_API_TOKEN }}

166
.github/workflows/release-efa.yaml vendored Normal file
View File

@ -0,0 +1,166 @@
name: Release EFA
on:
push:
tags:
- 'v*'
# Publishes the AWS EFA (libfabric) CUDA wheel variant:
# mooncake-transfer-engine-efa USE_EFA=ON USE_CUDA=ON (GPU, GPUDirect/FI_HMEM_CUDA)
# The non-CUDA variant (mooncake-transfer-engine-efa-non-cuda) is built by
# release-efa-non-cuda.yaml — split into its own workflow because each PyPI
# package publishes from a dedicated release workflow (trusted publisher /
# artifact pattern is per-package), mirroring release.yaml vs release-non-cuda.yaml.
#
# No EFA hardware is needed to build: USE_EFA only needs libfabric headers/lib
# to compile/link. auditwheel excludes libfabric/libefa from the wheel so they
# resolve to the user's system AWS EFA install (/opt/amazon/efa/lib) at
# runtime. The distro libfabric (1.x) used to build is ABI-forward-compatible
# with the AWS EFA libfabric (2.x) loaded at runtime; the EFA transport uses
# only long-stable fi_* core APIs.
env:
SCCACHE_GHA_ENABLED: "true"
jobs:
build:
runs-on: ubuntu-22.04
permissions:
contents: write
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12', '3.13']
env:
EFA_BUILD: "1"
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo rm -rf /usr/local/lib/android
- name: Install CUDA Toolkit
uses: Jimver/cuda-toolkit@v0.2.24
with:
cuda: '12.8.1'
method: 'network'
sub-packages: '["nvcc", "nvrtc-dev"]'
non-cuda-sub-packages: '["libcusparse-dev", "libcublas-dev", "libcusolver-dev"]'
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
- name: Configure sccache
uses: actions/github-script@v7
with:
script: |
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Configure project
run: |
sudo apt update -y
sudo apt install -y libfabric-dev libfabric1
sudo bash -x dependencies.sh -y
mkdir build
cd build
cmake .. \
-DUSE_HTTP=ON \
-DUSE_ETCD=ON \
-DUSE_CUDA=ON \
-DWITH_EP=OFF \
-DSTORE_USE_ETCD=ON \
-DUSE_EFA=ON \
-DLIBFABRIC_INCLUDE_DIR=/usr/include \
-DLIBFABRIC_LIBRARY=/usr/lib/x86_64-linux-gnu/libfabric.so \
-DENABLE_SCCACHE=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_EXE_LINKER_FLAGS=-L/usr/local/cuda/lib64/stubs
shell: bash
- name: Build project
run: |
export LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LD_LIBRARY_PATH
export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LIBRARY_PATH
cd build
make -j
sudo make install
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_release
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
shell: bash
- name: Build Python wheel
run: |
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-py${{ steps.generate_tag_release.outputs.python_version_tag }} ./scripts/build_wheel.sh
env:
VERSION: ${{ env.VERSION }}
- name: Verify libfabric is excluded from the wheel
run: |
WHL=$(ls mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl | head -1)
echo "Inspecting $WHL"
if unzip -l "$WHL" | grep -iE 'libfabric|libefa'; then
echo "::error::libfabric/libefa must NOT be bundled in the EFA wheel"
exit 1
fi
echo "OK: libfabric/libefa correctly excluded"
shell: bash
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: mooncake-wheel-efa-py${{ steps.generate_tag_release.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
publish-release:
if: ${{ !contains(github.ref_name, '-') }}
needs: build
runs-on: ubuntu-22.04
permissions:
contents: write
id-token: write
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Download all wheel artifacts
uses: actions/download-artifact@v4
with:
path: mooncake-wheel/dist-all
pattern: mooncake-wheel-efa-py*
- name: Prepare wheels for release
run: |
mkdir -p mooncake-wheel/dist-release
find mooncake-wheel/dist-all -name "*.whl" -exec cp {} mooncake-wheel/dist-release/ \;
echo "Collected wheels for release:"
ls -la mooncake-wheel/dist-release/
- name: Upload wheels to GitHub Release
uses: softprops/action-gh-release@v1
with:
files: mooncake-wheel/dist-release/*.whl
- name: Publish package to PyPI
if: github.repository == 'kvcache-ai/Mooncake'
uses: pypa/gh-action-pypi-publish@release/v1
with:
packages-dir: mooncake-wheel/dist-release/
password: ${{ secrets.PYPI_API_TOKEN }}

161
.github/workflows/release-musa.yaml vendored Normal file
View File

@ -0,0 +1,161 @@
name: Release MUSA
on:
push:
tags:
- 'v*'
jobs:
build:
if: ${{ !contains(github.ref_name, '-') }}
runs-on: ubuntu-22.04
container: mthreads/musa:rc4.3.0-devel-ubuntu22.04-amd64
permissions:
contents: write
strategy:
max-parallel: 2
matrix:
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
env:
BUILD_WITH_EP: "0"
MUSA_BUILD: "1"
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Mark repository as safe
shell: bash
run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Set Python bin
shell: bash
run: |
set -euo pipefail
PYTHON_BIN="$(command -v python${{ matrix.python-version }})"
"$PYTHON_BIN" --version
"$PYTHON_BIN" -m pip --version
echo "PYTHON_BIN=${PYTHON_BIN}" >> "$GITHUB_ENV"
- name: Install dependencies
shell: bash
run: |
set -eo pipefail
export PYTHONPATH="${PYTHONPATH:-}"
export CMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH:-}"
bash -x dependencies.sh -y
echo "PATH=/usr/local/go/bin:/usr/local/musa/bin:${PATH}" >> "$GITHUB_ENV"
echo "LD_LIBRARY_PATH=/usr/local/musa/lib:${LD_LIBRARY_PATH:-}" >> "$GITHUB_ENV"
echo "LIBRARY_PATH=/usr/local/musa/lib:${LIBRARY_PATH:-}" >> "$GITHUB_ENV"
- name: Configure project
shell: bash
run: |
set -eo pipefail
export PYTHONPATH="${PYTHONPATH:-}"
export CMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH:-}"
rm -rf build
mkdir build
cd build
cmake_args=(
-DUSE_MUSA=ON
-DUSE_HTTP=ON
-DUSE_ETCD=ON
-DSTORE_USE_ETCD=ON
-DBUILD_UNIT_TESTS=OFF
-DENABLE_DEBUG_SYMBOLS=OFF
-DCMAKE_BUILD_TYPE=Release
-DPython3_EXECUTABLE="${PYTHON_BIN}"
)
cmake -G Ninja .. "${cmake_args[@]}"
- name: Build project
shell: bash
run: |
set -eo pipefail
export PYTHONPATH="${PYTHONPATH:-}"
export CMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH:-}"
cd build
cmake --build . -j"$(nproc)"
- name: Build MUSA allocator
shell: bash
run: |
set -eo pipefail
mkdir -p build/mooncake-transfer-engine/nvlink-allocator
cd mooncake-transfer-engine/nvlink-allocator
bash build.sh --use-mcc ../../build/mooncake-transfer-engine/nvlink-allocator/
- name: Install project
shell: bash
run: |
set -eo pipefail
cd build
cmake --install .
- name: Generate Python version tag
id: generate_tag_release
shell: bash
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
- name: Build Python wheel
shell: bash
run: |
set -eo pipefail
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:-}:/usr/local/lib:/usr/local/musa/lib"
MUSA_BUILD=1 PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-musa-py${{ steps.generate_tag_release.outputs.python_version_tag }} ./scripts/build_wheel.sh
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: mooncake-wheel-musa-x86_64-py${{ steps.generate_tag_release.outputs.python_version_tag }}
path: mooncake-wheel/dist-musa-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
publish-release:
if: ${{ !contains(github.ref_name, '-') }}
needs: build
runs-on: ubuntu-22.04
environment: pypi
permissions:
contents: write
id-token: write
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Download all wheel artifacts
uses: actions/download-artifact@v4
with:
path: mooncake-wheel/dist-all
pattern: mooncake-wheel-musa-*
- name: Prepare wheels for release
run: |
mkdir -p mooncake-wheel/dist-release
find mooncake-wheel/dist-all -name "*.whl" -exec cp {} mooncake-wheel/dist-release/ \;
echo "Collected wheels for release:"
ls -la mooncake-wheel/dist-release/
- name: Upload wheels to GitHub Release
uses: softprops/action-gh-release@v1
with:
files: mooncake-wheel/dist-release/*.whl
- name: Publish package to PyPI
if: github.repository == 'kvcache-ai/Mooncake'
uses: pypa/gh-action-pypi-publish@release/v1
with:
packages-dir: mooncake-wheel/dist-release/

11
.github/workflows/release-non-cuda.yaml vendored
View File

@ -43,10 +43,6 @@ jobs:
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Run sccache stat for check
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Configure project
run: |
sudo apt update -y
@ -63,6 +59,11 @@ jobs:
sudo make install
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_release
run: |
@ -84,6 +85,7 @@ jobs:
path: mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
publish-release:
if: ${{ !contains(github.ref_name, '-') }}
needs: build
runs-on: ubuntu-22.04
permissions:
@ -120,4 +122,3 @@ jobs:
with:
packages-dir: mooncake-wheel/dist-release/
password: ${{ secrets.PYPI_API_TOKEN }}

196
.github/workflows/release-npu.yaml vendored Normal file
View File

@ -0,0 +1,196 @@
name: Release Ascend NPU
on:
push:
tags:
- 'v*'
workflow_dispatch:
inputs:
tag:
description: 'Release tag (e.g. v0.1.0)'
required: true
type: string
jobs:
build:
if: ${{ (github.event_name == 'push' && !contains(github.ref_name, '-')) || (github.event_name == 'workflow_dispatch' && startsWith(inputs.tag, 'v') && !contains(inputs.tag, '-')) }}
strategy:
max-parallel: 2
matrix:
arch: [aarch64, x86_64]
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
include:
- arch: aarch64
runner: ubuntu-22.04-arm
cann_arch: aarch64
- arch: x86_64
runner: ubuntu-22.04
cann_arch: x86_64
runs-on: ${{ matrix.runner }}
permissions:
contents: write
env:
BUILD_WITH_EP: "0"
NPU_BUILD: "1"
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ github.event_name == 'workflow_dispatch' && format('refs/tags/{0}', inputs.tag) || github.ref }}
fetch-depth: 0
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Set Python bin
shell: bash
run: |
set -euo pipefail
PYTHON_BIN="$(command -v python${{ matrix.python-version }})"
"$PYTHON_BIN" --version
"$PYTHON_BIN" -m pip --version
echo "PYTHON_BIN=${PYTHON_BIN}" >> "$GITHUB_ENV"
- name: Install CANN Toolkit
shell: bash
run: |
set -euo pipefail
CANN_BASE_URL="https://ascend.devcloud.huaweicloud.com/cann/run/software"
CANN_VERSION=$(curl -s "${CANN_BASE_URL}/" | grep -oP '[0-9]+\.[0-9]+\.[0-9]+(?=/)' | sort -V | tail -1)
echo "Latest CANN version: ${CANN_VERSION}"
CANN_URL="${CANN_BASE_URL}/${CANN_VERSION}/${{ matrix.cann_arch }}/Ascend-cann-toolkit_${CANN_VERSION}_linux-${{ matrix.cann_arch }}.run"
echo "Downloading CANN ${CANN_VERSION} from ${CANN_URL}"
wget -q --show-progress -O /tmp/cann_toolkit.run "${CANN_URL}"
chmod +x /tmp/cann_toolkit.run
sudo /tmp/cann_toolkit.run --install --install-for-all --install-path=/usr/local/Ascend --quiet
rm -f /tmp/cann_toolkit.run
- name: Build and install HIXL
shell: bash
run: |
set -eo pipefail
export PYTHONPATH="${PYTHONPATH:-}"
export CMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH:-}"
source /usr/local/Ascend/cann/set_env.sh
sudo ${PYTHON_BIN} -m pip install setuptools
cd /tmp
git clone https://gitcode.com/cann/hixl.git
cd hixl
bash build.sh -j$(nproc)
sudo ./build_out/cann-hixl_*.run --full --quiet --pylocal --install-path=/usr/local/Ascend
- name: Install dependencies
shell: bash
run: |
set -eo pipefail
export PYTHONPATH="${PYTHONPATH:-}"
export CMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH:-}"
source /usr/local/Ascend/cann/set_env.sh
sudo bash -x dependencies.sh -y
sudo bash scripts/ascend/dependencies_ascend_installation.sh
echo "PATH=/usr/local/go/bin:${PATH}" >> "$GITHUB_ENV"
- name: Configure project
shell: bash
run: |
set -eo pipefail
export PYTHONPATH="${PYTHONPATH:-}"
export CMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH:-}"
source /usr/local/Ascend/cann/set_env.sh
rm -rf build
mkdir build
cd build
cmake_args=(
-DUSE_ASCEND_DIRECT=ON
-DUSE_ETCD=ON
-DSTORE_USE_ETCD=ON
-DBUILD_UNIT_TESTS=OFF
-DCMAKE_BUILD_TYPE=Release
-DPython3_EXECUTABLE="${PYTHON_BIN}"
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=OFF
-DCMAKE_BUILD_WITH_INSTALL_RPATH=OFF
-DCMAKE_SKIP_INSTALL_RPATH=ON
)
cmake .. "${cmake_args[@]}"
- name: Build project
shell: bash
run: |
set -eo pipefail
export PYTHONPATH="${PYTHONPATH:-}"
export CMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH:-}"
source /usr/local/Ascend/cann/set_env.sh
export PATH="/usr/local/go/bin:${PATH}"
cd build
cmake --build . -j"$(nproc)"
sudo cmake --install .
- name: Generate Python version tag
id: generate_tag_release
shell: bash
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
- name: Build Python wheel
shell: bash
run: |
set -eo pipefail
export PYTHONPATH="${PYTHONPATH:-}"
export CMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH:-}"
source /usr/local/Ascend/cann/set_env.sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
NPU_BUILD=1 PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-npu-py${{ steps.generate_tag_release.outputs.python_version_tag }} ./scripts/build_wheel.sh
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: mooncake-wheel-npu-${{ matrix.arch }}-py${{ steps.generate_tag_release.outputs.python_version_tag }}
path: mooncake-wheel/dist-npu-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
publish-release:
if: ${{ (github.event_name == 'push' && !contains(github.ref_name, '-')) || (github.event_name == 'workflow_dispatch' && startsWith(inputs.tag, 'v') && !contains(inputs.tag, '-')) }}
needs: build
runs-on: ubuntu-22.04
environment: pypi
permissions:
contents: write
id-token: write
steps:
- name: Checkout source
uses: actions/checkout@v4
with:
ref: ${{ github.event_name == 'workflow_dispatch' && format('refs/tags/{0}', inputs.tag) || github.ref }}
- name: Download all wheel artifacts
uses: actions/download-artifact@v4
with:
path: mooncake-wheel/dist-all
pattern: mooncake-wheel-npu-*
- name: Prepare wheels for release
run: |
mkdir -p mooncake-wheel/dist-release
find mooncake-wheel/dist-all -name "*.whl" -exec cp {} mooncake-wheel/dist-release/ \;
echo "Collected wheels for release:"
ls -la mooncake-wheel/dist-release/
- name: Upload wheels to GitHub Release
uses: softprops/action-gh-release@v1
with:
tag_name: ${{ github.event_name == 'workflow_dispatch' && inputs.tag || github.ref_name }}
files: mooncake-wheel/dist-release/*.whl
- name: Publish package to PyPI
if: github.repository == 'kvcache-ai/Mooncake'
uses: pypa/gh-action-pypi-publish@release/v1
with:
packages-dir: mooncake-wheel/dist-release/

142
.github/workflows/release.yaml vendored
View File

@ -9,7 +9,10 @@ env:
SCCACHE_GHA_ENABLED: "true"
jobs:
build:
# Skip semver pre-release tags (e.g. v1.0.0-rc1); those are handled by pre-release.yaml.
if: ${{ !contains(github.ref_name, '-') }}
runs-on: ubuntu-22.04
container: pytorch/manylinux2_28-builder:cuda12.8
permissions:
contents: write
strategy:
@ -22,27 +25,15 @@ jobs:
- name: Checkout source
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Mark workspace safe for git (container runs as root)
run: git config --global --add safe.directory '*'
- name: Free up disk space
- name: Select Python ${{ matrix.python-version }} from manylinux image
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo rm -rf /usr/local/lib/android
df -h
- name: Install CUDA Toolkit
uses: Jimver/cuda-toolkit@v0.2.24
with:
cuda: '12.8.1'
linux-local-args: '["--toolkit"]'
method: 'network'
sub-packages: '["nvcc", "nvrtc-dev"]'
non-cuda-sub-packages: '["libcusparse-dev", "libcublas-dev", "libcusolver-dev"]'
PYV_NODOT=$(echo "${{ matrix.python-version }}" | tr -d '.')
PYBIN="/opt/python/cp${PYV_NODOT}-cp${PYV_NODOT}/bin"
echo "$PYBIN" >> "$GITHUB_PATH"
"$PYBIN/pip" install --quiet "cmake<4" setuptools wheel
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
@ -54,25 +45,21 @@ jobs:
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Run sccache stat for check
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Configure project
run: |
sudo apt update -y
sudo bash -x dependencies.sh -y
bash -x dependencies.sh -y
echo "/usr/local/go/bin" >> "$GITHUB_PATH"
mkdir build
cd build
cmake .. -DBUILD_UNIT_TESTS=OFF -DUSE_HTTP=ON -DUSE_ETCD=ON -DUSE_CUDA=ON -DWITH_EP=ON -DEP_TORCH_VERSIONS="2.9.0;2.9.1;2.10.0;2.11.0" -DSTORE_USE_ETCD=ON -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release
cmake .. -DBUILD_UNIT_TESTS=OFF -DUSE_HTTP=ON -DUSE_ETCD=ON -DUSE_CUDA=ON -DWITH_EP=ON -DEP_TORCH_VERSIONS="2.9.1;2.10.0;2.11.0;2.12.0;2.12.1" -DSTORE_USE_ETCD=ON -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release -DPython3_EXECUTABLE="$(which python3)"
shell: bash
- name: Build project
run: |
export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:LIBRARY_PATH
export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LIBRARY_PATH
cd build
make -j
sudo make install
make install
shell: bash
- name: Build nvlink_allocator.so
@ -85,6 +72,11 @@ jobs:
bash build.sh ../../build/mooncake-transfer-engine/nvlink-allocator/
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_release
run: |
@ -105,8 +97,100 @@ jobs:
name: mooncake-wheel-py${{ steps.generate_tag_release.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
build-arm64:
if: ${{ !contains(github.ref_name, '-') }}
runs-on: ubuntu-22.04-arm
permissions:
contents: write
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12', '3.13']
env:
TORCH_CUDA_ARCH_LIST: "9.0"
CUDA_HOME: "/usr/local/cuda"
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
- name: Install CUDA Toolkit 12.8 (arm64 SBSA)
run: |
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y cuda-toolkit-12-8
echo "/usr/local/cuda/bin" >> $GITHUB_PATH
/usr/local/cuda/bin/nvcc --version
shell: bash
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
- name: Configure sccache
uses: actions/github-script@v7
with:
script: |
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
- name: Configure project
run: |
sudo apt update -y
sudo bash -x dependencies.sh -y
mkdir build
cd build
cmake -G Ninja .. -DUSE_HTTP=ON -DUSE_CUDA=ON -DUSE_MNNVL=ON -DWITH_EP=OFF -DWITH_STORE_RUST=OFF -DENABLE_SCCACHE=ON -DCMAKE_BUILD_TYPE=Release
shell: bash
- name: Build project
run: |
export CUDA_HOME=/usr/local/cuda
for dir in /usr/local/cuda/lib64/stubs /usr/local/cuda/targets/*/lib/stubs; do
if [ -d "$dir" ]; then
export LIBRARY_PATH="$dir:${LIBRARY_PATH:-}"
fi
done
cd build
cmake --build .
sudo cmake --install .
shell: bash
- name: Run sccache stat for check
if: ${{ env.SCCACHE_PATH != '' }}
shell: bash
run: ${SCCACHE_PATH} --show-stats
- name: Generate Python version tag
id: generate_tag_release
run: |
echo "python_version_tag=$(echo ${{ matrix.python-version }} | tr -d '.')" >> $GITHUB_OUTPUT
shell: bash
- name: Build Python wheel
run: |
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:-}:/usr/local/lib"
PYTHON_VERSION=${{ matrix.python-version }} OUTPUT_DIR=dist-py${{ steps.generate_tag_release.outputs.python_version_tag }} ./scripts/build_wheel.sh
shell: bash
- name: Upload Python wheel artifact
uses: actions/upload-artifact@v4
with:
name: mooncake-wheel-arm64-py${{ steps.generate_tag_release.outputs.python_version_tag }}
path: mooncake-wheel/dist-py${{ steps.generate_tag_release.outputs.python_version_tag }}/*.whl
publish-release:
needs: build
needs: [build, build-arm64]
runs-on: ubuntu-22.04
permissions:
contents: write

41
.github/workflows/stale.yml vendored Normal file
View File

@ -0,0 +1,41 @@
name: Close Stale Issues and PRs
on:
schedule:
- cron: '37 2 * * *'
workflow_dispatch: {}
permissions:
issues: write
pull-requests: write
jobs:
stale:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v9
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
stale-issue-message: >
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed in 30 days if no further activity occurs.
If this issue is still relevant, please comment to keep it open.
stale-pr-message: >
This PR has been automatically marked as stale because it has not had
recent activity. It will be closed in 14 days if no further activity occurs.
Please update the PR or comment to keep it open.
close-issue-message: >
This issue was closed because it has been inactive for 120 days.
Feel free to reopen if this is still relevant.
close-pr-message: >
This PR was closed because it has been inactive for 104 days.
Feel free to reopen if you'd like to continue the work.
days-before-stale: 120
days-before-close: 30
days-before-pr-stale: 120
days-before-pr-close: 30
stale-issue-label: 'stale'
stale-pr-label: 'stale'
exempt-issue-labels: 'pinned,security,RFC'
exempt-pr-labels: 'pinned,work-in-progress'
operations-per-run: 100

6
.gitignore vendored
View File

@ -206,3 +206,9 @@ _codeql_detected_source_root
# CodeBuddy Memory
.codebuddy/
# core dumps
core_*
# MacOS
.DS_Store
.envrc

8
.typos.toml
View File

@ -1,13 +1,17 @@
[default]
extend-ignore-words = ["CANN", "ASO", "fre", "wqs"]
extend-ignore-words = ["CANN", "ASO", "fre", "wqs", "hsa"]
[default.extend-words]
CANN = "CANN"
ASO = "ASO"
fre = "fre"
wqs = "wqs"
# AMD HSA runtime symbol prefix (hsa_*, hsaRes, hsaErr, etc.) — used by the
# ROCm dmabuf MR registration path.
hsa = "hsa"
HPE = "HPE"
[files]
extend-exclude = [
"mooncake-transfer-engine/tent/include/tent/thirdparty/nlohmann/json.h",
]
]

16
CMakeLists.txt
View File

@ -19,6 +19,7 @@ option(WITH_P2P_STORE "build p2p store library and sample code" OFF)
option(WITH_RUST_EXAMPLE "build the Rust interface and sample code for the transfer engine" OFF)
option(WITH_STORE_RUST "build the Rust bindings for the Mooncake Store" ON)
option(WITH_EP "build mooncake with expert parallelism support" OFF)
option(USE_NOF "build mooncake store with NoF SSD pool support" OFF)
include(${CMAKE_CURRENT_SOURCE_DIR}/mooncake-common/SetupPython.cmake)
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/extern/pybind11)
@ -56,6 +57,12 @@ if (STORE_USE_K8S_LEASE)
add_compile_definitions(STORE_USE_K8S_LEASE)
endif()
if (USE_NOF)
add_compile_definitions(USE_NOF)
else()
message(STATUS "USE_NOF=OFF, NoF SSD pool support is disabled")
endif()
option(STORE_USE_JEMALLOC "Use jemalloc in mooncake store master" OFF)
# Define ASIO macros before building targets that include ASIO headers.
@ -63,6 +70,7 @@ add_compile_definitions(ASIO_SEPARATE_COMPILATION ASIO_DYN_LINK)
add_subdirectory(mooncake-common)
include_directories(mooncake-common/etcd)
include_directories(mooncake-common/k8s-lease)
include_directories(mooncake-common/include)
if (WITH_TE)
@ -94,8 +102,10 @@ if (WITH_EP)
include_directories(mooncake-pg/include)
else ()
message(STATUS "WITH_EP enabled: building Mooncake EP and PG Python extensions")
find_package(CUDAToolkit REQUIRED)
message(STATUS "Detected CUDA version: ${CUDAToolkit_VERSION}")
if(USE_CUDA)
find_package(CUDAToolkit REQUIRED)
message(STATUS "Detected CUDA version: ${CUDAToolkit_VERSION}")
endif()
# EP_TORCH_VERSIONS: semicolon-separated list of PyTorch versions to build for.
# Can be set via -DEP_TORCH_VERSIONS="2.9.1;2.8.0" or the EP_TORCH_VERSIONS env var.
@ -136,6 +146,7 @@ if (WITH_EP)
"-DTORCH_CUDA_ARCH_LIST=${_torch_cuda_arch_list_pipe}"
"-DSTAGING_DIR=${EP_PG_STAGING_DIR}"
"-DENGINE_SO_PATH=$<TARGET_FILE:engine>"
"-DEP_USE_MUSA=$<IF:$<BOOL:${USE_MUSA}>,1,0>"
-P "${CMAKE_CURRENT_SOURCE_DIR}/mooncake-ep/BuildEpExt.cmake"
COMMENT "Building Mooncake EP Python extension(s)"
DEPENDS engine
@ -152,6 +163,7 @@ if (WITH_EP)
"-DTORCH_CUDA_ARCH_LIST=${_torch_cuda_arch_list_pipe}"
"-DSTAGING_DIR=${EP_PG_STAGING_DIR}"
"-DENGINE_SO_PATH=$<TARGET_FILE:engine>"
"-DEP_USE_MUSA=$<IF:$<BOOL:${USE_MUSA}>,1,0>"
-P "${CMAKE_CURRENT_SOURCE_DIR}/mooncake-pg/BuildPgExt.cmake"
COMMENT "Building Mooncake PG Python extension(s)"
DEPENDS engine mooncake_ep_ext

2
CONTRIBUTING.md
View File

@ -70,7 +70,7 @@ git commit -m "chore: pre-commit autoupdate"
If clang-format is missing, install it (Ubuntu example):
```bash
sudo apt-get update && sudo apt-get install -y clang-format
sudo apt-get update && sudo apt-get install -y clang-format-20
```
You can temporarily skip hooks:

17
FAST25-release/README.md Normal file
View File

@ -0,0 +1,17 @@
```json
{
"timestamp": 27482,
"input_length": 6955,
"output_length": 52,
"hash_ids": [46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 2353, 2354]
}
{
"timestamp": 30535,
"input_length": 6472,
"output_length": 26,
"hash_ids": [46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 2366]
}
```
The above presents two samples from our trace dataset. The trace includes the timing of request arrivals, the number of input tokens, the number of output tokens, and the remapped block hash. To protect our customers' privacy, we applied several mechanisms to remove user-related information while preserving the dataset's utility for simulated evaluation. More descriptions of the trace (e.g., up to 50% cache hit ratio) can be found in Section 4 of the technical report.
**_Update [Feb 21, 2025]: The updated [traces](./traces) used in our FAST'25 paper have been released! Please refer to the paper's appendix (found [here](Mooncake-FAST25.pdf)) for more details._**

2
MAINTAINERS.md
View File

@ -12,6 +12,6 @@ Current list of codeowners on this project:
| ---------------------------- | ---------------------------- | ---------------------------- | ---------------------------- |
| <img src="image/partners/approaching_ai_logo.png" width="120"/> | <img src="image/partners/ant_group_logo.png" width="120"/> | <img src="image/partners/huawei_logo.png" width="120"/> | <img src="image/partners/nvidia_logo.png" width="120"/> |
| <img src="image/partners/moore_thread_logo.jpg" width="120"/> | <img src="image/partners/tencent_logo.png" width="120"/> | <img src="image/partners/volcengine_logo.png" width="120"/> | <img src="image/partners/amd_logo.png" width="120"/> |
| <img src="image/partners/ieitsystems_logo.png" width="120"/> | | | |
| <img src="image/partners/ieitsystems_logo.png" width="120"/> | <img src="image/partners/sunrise_logo.png" width="120"/> | <img src="image/partners/hygon_logo.png" width="120"/> | |
Want to include your company logo? Just open a Pull Request!

365
README.md
View File

@ -6,34 +6,41 @@
<a href="https://www.usenix.org/system/files/fast25-qin.pdf" target="_blank"><strong>Paper</strong></a>
| <a href="https://www.usenix.org/system/files/fast25_slides-qin.pdf" target="_blank"><strong>Slides</strong></a>
| <a href="FAST25-release/traces" target="_blank"><strong>Traces</strong></a>
| <a href="https://arxiv.org/abs/2407.00079" target="_blank"><strong>Technical Report</strong></a>
| <a href="https://kvcache-ai.github.io/Mooncake/" target="_blank"><strong>Blog</strong></a>
| <a href="https://kvcache-ai.github.io/Mooncake/" target="_blank"><strong>Documentation</strong></a>
| <a href="https://kvcache.ai/" target="_blank"><strong>Blog</strong></a>
| <a href="https://join.slack.com/t/mooncake-project/shared_invite/zt-3qx4x35ea-zSSTqTHItHJs9SCoXLOSPA" target="_blank"><strong>Slack</strong></a>
<br />
<br />
[![Docs](https://img.shields.io/badge/docs-live-brightgreen)](https://kvcache-ai.github.io/Mooncake/)
[![PyPI](https://img.shields.io/pypi/v/mooncake-transfer-engine)](https://pypi.org/project/mooncake-transfer-engine)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/mooncake-transfer-engine)](https://pypi.org/project/mooncake-transfer-engine)
[![CUDA <=12.9](https://img.shields.io/static/v1?label=CUDA&message=%3C%3D12.9&color=76B900)](https://pypi.org/project/mooncake-transfer-engine)
[![CUDA 13.0/13.1](https://img.shields.io/static/v1?label=CUDA&message=13.0%2F13.1&color=76B900)](https://pypi.org/project/mooncake-transfer-engine-cuda13)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/mooncake-transfer-engine)](https://pypi.org/project/mooncake-transfer-engine)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/kvcache-ai/Mooncake)
[![PyPI - Downloads](https://static.pepy.tech/badge/mooncake-transfer-engine?period=month)](https://pypi.org/project/mooncake-transfer-engine)
[![GitHub commit activity](https://img.shields.io/github/commit-activity/w/kvcache-ai/Mooncake)](https://github.com/kvcache-ai/Mooncake/graphs/commit-activity)
[![license](https://img.shields.io/github/license/kvcache-ai/mooncake.svg)](https://github.com/kvcache-ai/Mooncake/blob/main/LICENSE-APACHE)
[![Docker](https://img.shields.io/docker/v/kvcacheai/mooncake?label=docker&logo=docker&logoColor=white&color=2496ED)](https://hub.docker.com/r/kvcacheai/mooncake)
<br />
[![PyPI](https://img.shields.io/pypi/v/mooncake-transfer-engine)](https://pypi.org/project/mooncake-transfer-engine)
[![PyPI CUDA <=12.9](https://img.shields.io/static/v1?label=pypi&message=CUDA%20%3C%3D12.9&color=76B900)](https://pypi.org/project/mooncake-transfer-engine)
[![PyPI CUDA 13.0/13.1](https://img.shields.io/static/v1?label=pypi&message=CUDA%2013.0%2F13.1&color=76B900)](https://pypi.org/project/mooncake-transfer-engine-cuda13)
[![PyPI Non-CUDA](https://img.shields.io/static/v1?label=pypi&message=non-CUDA&color=00BFFF)](https://pypi.org/project/mooncake-transfer-engine-non-cuda/)
[![PyPI NPU](https://img.shields.io/static/v1?label=pypi&message=NPU&color=F87171)](https://pypi.org/project/mooncake-transfer-engine-npu/)
</div>
<br/>
Mooncake is the serving platform for <a href="https://kimi.ai/"><img src="image/kimi.png" alt="icon" style="height: 16px; vertical-align: middle;"> Kimi</a>, a leading LLM service provided by <a href="https://www.moonshot.cn/"><img src="image/moonshot.jpg" alt="icon" style="height: 16px; vertical-align: middle;"> Moonshot AI</a>.
Now both the Transfer Engine and Mooncake Store are open-sourced!
This repository also hosts its technical report and the open-sourced traces.
Under real workloads, Mooncake’s innovative architecture enables Kimi to handle 75% more requests while adhering to SLOs.
<h2 id="updates">🔄 Updates</h2>
- **May 7, 2026**: 🚀 [vLLM officially features Mooncake Store](https://vllm.ai/blog/mooncake-store) — a deep dive into how Mooncake's distributed KVCache engine supercharges vLLM inference with high-throughput, memory-efficient, cross-instance KV cache sharing!
- **Apr 29, 2026**: SGLang introduces [RDMA-based P2P weight transfer for large-scale distributed RL](https://lmsys.org/blog/2026-04-29-p2p-update/) using Mooncake TransferEngine, achieving 7x faster weight updates for the 1T-parameter Kimi-K2 model (53s → 7.2s) with zero-copy RDMA transfer across thousands of GPUs.
- **Mar 19, 2026**: [TorchSpec: Speculative Decoding Training at Scale](https://pytorch.org/blog/torchspec-speculative-decoding-training-at-scale) is [open sourced](https://github.com/torchspec-project/TorchSpec), using Mooncake to decouple inference and training via efficient hidden states management.
- **Mar 5, 2026**: [LightX2V](https://github.com/ModelTC/LightX2V/pull/893) now supports disaggregated deployment based on Mooncake, enabling encoder/transformer service decoupling with Mooncake Transfer Engine for high-performance cross-device and cross-machine data transfer.
- **Feb 25, 2026**: [SGLang](https://github.com/sgl-project/sglang) merged [Encoder Global Cache Manager](https://github.com/sgl-project/sglang/pull/16137), introducing a Mooncake-powered global multimodal embedding cache that enables cross-instance sharing of ViT embeddings to avoid redundant GPU computation.
<details>
<summary>More</summary>
- **Feb 24, 2026**: [vLLM-Omni](https://docs.vllm.ai/projects/vllm-omni/en/latest/design/feature/disaggregated_inference/) introduces disaggregated inference connectors with support for both `MooncakeStoreConnector` and `MooncakeTransferEngineConnector` for multi-node omni-modality pipelines.
- **Feb 12, 2026**: [Mooncake Joins PyTorch Ecosystem](https://pytorch.org/blog/mooncake-joins-pytorch-ecosystem/) We are thrilled to announce that Mooncake has officially joined the PyTorch Ecosystem!
- **Jan 28, 2026**: [FlexKV](https://github.com/taco-project/FlexKV), a distributed KV store and cache system from Tencent and NVIDIA in collaboration with the community, now supports [distributed KVCache reuse](https://github.com/taco-project/FlexKV/blob/main/docs/dist_reuse/README_en.md) with the Mooncake Transfer Engine.
@ -63,127 +70,136 @@ This repository also hosts its technical report and the open-sourced traces.
- **June 27, 2024**: We present a series of Chinese blogs with more discussions on <a href="https://zhuanlan.zhihu.com/p/705754254">zhihu 1</a>, <a href="https://zhuanlan.zhihu.com/p/705910725">2</a>, <a href="https://zhuanlan.zhihu.com/p/706204757">3</a>, <a href="https://zhuanlan.zhihu.com/p/707997501">4</a>, <a href="https://zhuanlan.zhihu.com/p/9461861451">5</a>, <a href="https://zhuanlan.zhihu.com/p/1939988652114580803">6</a>, <a href="https://zhuanlan.zhihu.com/p/1959366095443064318">7</a>.
- **June 26, 2024**: Initial technical report release.
</details>
<h2 id="overview">🎉 Overview</h2>
Mooncake features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated KVCache pool.
![architecture](image/architecture.png)
The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges in highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake’s innovative architecture enables <a href="https://kimi.ai/">Kimi</a> to handle 75% more requests.
<h2 id="components">🧩 Components</h2>
<!-- ![components](image/components.png) -->
<img src=image/components.png width=74% />
<div align="center">
<img src=image/components.png width=74% />
</div>
**Mooncake Core Component: Transfer Engine (TE)**
The core of Mooncake is the Transfer Engine (TE), which provides a unified interface for batched data transfer across various storage devices and network links. Supporting multiple protocols including TCP, RDMA, CXL/shared-memory, and NVMe over Fabric (NVMe-of), TE is designed to enable fast and reliable data transfer for AI workloads. Compared to Gloo (used by Distributed PyTorch) and traditional TCP, TE achieves significantly lower I/O latency, making it a superior solution for efficient data transmission.
Mooncake is an infrastructure project for large-scale LLM inference and training. It features a KV cache-centric disaggregated architecture that separates prefill and decode clusters, while leveraging otherwise underutilized CPU, DRAM, and SSD resources in GPU clusters to build a disaggregated KV cache pool.
**P2P Store and Mooncake Store**
Both P2P Store and Mooncake Store are built on the Transfer Engine and provide key/value caching for different scenarios. P2P Store focuses on sharing temporary objects (e.g., checkpoint files) across nodes in a cluster, preventing bandwidth saturation on a single machine. Mooncake Store, on the other hand, supports distributed pooled KVCache, specifically designed for XpYd disaggregation to enhance resource utilization and system performance.
**Mooncake Integration with Leading LLM Inference Systems**
Mooncake has been seamlessly integrated with several popular large language model (LLM) inference systems. Through collaboration with the vLLM and SGLang teams, Mooncake now officially supports prefill-decode disaggregation. By leveraging the high-efficiency communication capabilities of RDMA devices, Mooncake significantly improves inference efficiency in prefill-decode disaggregation scenarios, providing robust technical support for large-scale distributed inference tasks.
In addition, Mooncake has been successfully integrated with SGLang's Hierarchical KV Caching, vLLM's prefill serving, and LMCache, augmenting KV cache management capabilities across large-scale inference scenarios.
**Elastic Expert Parallelism Support**
Mooncake adds elasticity and fault tolerance support for MoE model inference, enabling inference systems to remain responsive and recoverable in the event of GPU failures or changes in resource configuration. This functionality includes automatic faulty rank detection and can work with the EPLB module to dynamically route tokens to healthy ranks during inference.
**Tensor-Centric Ecosystem**
Mooncake establishes a full-stack, Tensor-oriented AI infrastructure where Tensors serve as the fundamental data carrier. The ecosystem spans from the Transfer Engine, which accelerates Tensor data movement across heterogeneous storage (DRAM/VRAM/NVMe), to the P2P Store and Mooncake Store for distributed management of Tensor objects (e.g., Checkpoints and KVCache), up to the Mooncake Backend enabling Tensor-based elastic distributed computing. This architecture is designed to maximize Tensor processing efficiency for large-scale model inference and training.
Mooncake includes a high-performance Transfer Engine for low-latency data movement across heterogeneous networks and accelerators; Mooncake Store for distributed KV cache and model-weight management; and Mooncake EP & PG for elastic MoE serving. Deeply integrated with ecosystems such as SGLang and vLLM, Mooncake helps LLM systems improve cache reuse, reduce serving latency, and scale efficiently across multi-node clusters.
<h2 id="show-cases">🔥 Show Cases</h2>
### Use Transfer Engine Standalone ([Guide](https://kvcache-ai.github.io/Mooncake/design/transfer-engine/index.html))
### Transfer Engine (TE)
Transfer Engine is a high-performance data transfer framework. Transfer Engine provides a unified interface to transfer data from DRAM, VRAM or NVMe, while the technical details related to hardware are hidden. Transfer Engine supports multiple communication protocols including TCP, RDMA (InfiniBand/RoCEv2/eRDMA/NVIDIA GPUDirect), NVMe over Fabric (NVMe-of), NVLink, HIP, CXL, and Ascend. When built with the corresponding runtime, Transfer Engine can also detect and route accelerator memory on CUDA, MUSA, HIP, and Cambricon MLU devices. For a complete list of supported protocols and configuration guide, see the [Supported Protocols Documentation](https://kvcache-ai.github.io/Mooncake/getting_started/supported-protocols.html).
The core of Mooncake is the Transfer Engine (TE), a high-performance data transfer framework. TE offers a unified interface for batched data movement across diverse storage, network, and accelerator environments. By supporting multiple transport protocols, topology-aware routing, multi-NIC bandwidth aggregation, and automatic failover, TE delivers low-latency, scalable, and robust data transmission for distributed AI workloads. See the [Transfer Engine guide](https://kvcache-ai.github.io/Mooncake/design/transfer-engine/index.html) for details.
<details>
<summary>Highlights</summary>
#### Highlights
- **Efficient use of multiple RDMA NIC devices.** Transfer Engine supports the use of multiple RDMA NIC devices to achieve the *aggregation of transfer bandwidth*.
- **Topology aware path selection.** Transfer Engine can *select optimal devices* based on the location (NUMA affinity, etc.) of both source and destination.
- **Topology-aware path selection.** Transfer Engine can *select optimal devices* based on the location (NUMA affinity, etc.) of both source and destination.
- **More robust against temporary network errors.** Once transmission fails, Transfer Engine will try to use alternative paths for data delivery automatically.
- **Robust against temporary network errors.** Once transmission fails, Transfer Engine will try to use alternative paths for data delivery automatically.
#### Performance
With 40 GB of data (equivalent to the size of the KVCache generated by 128k tokens in the LLaMA3-70B model), Mooncake Transfer Engine delivers up to **87 GB/s** and **190 GB/s** of bandwidth in 4×200 Gbps and 8×400 Gbps RoCE networks respectively, which are about **2.4x and 4.6x faster** than the TCP protocol.
- **Superior performance at scale.** With 40 GB of data (equivalent to the size of the KVCache generated by 128k tokens in the LLaMA3-70B model), Mooncake Transfer Engine delivers up to **87 GB/s** and **190 GB/s** of bandwidth in 4×200 Gbps and 8×400 Gbps RoCE networks respectively, which are about **2.4x and 4.6x faster** than the TCP protocol.
<!-- ![transfer-engine-performance.png](image/transfer-engine-performance.png) -->
<img src=image/transfer-engine-performance.png width=75% />
### P2P Store ([Guide](https://kvcache-ai.github.io/Mooncake/design/p2p-store.html))
P2P Store is built on the Transfer Engine and supports sharing temporary objects between peer nodes in a cluster. P2P Store is ideal for scenarios like checkpoint transfer, where data needs to be rapidly and efficiently shared across a cluster.
**P2P Store has been used in the checkpoint transfer service of Moonshot AI.**
- **Broad support for heterogeneous transports and accelerators.** Transfer Engine provides unified data transfer across diverse protocols, including TCP, RDMA, AWS EFA, NVMe-oF, NVLink, HIP, Barex, CXL, and Ascend-family transports. When built with the corresponding runtime, Transfer Engine can detect accelerator memory and select suitable transport paths for efficient data movement across CUDA, MUSA, HIP, MACA, Cambricon MLU, and Ascend-enabled environments. For a complete list of supported protocols and configuration guide, see the [Supported Protocols Documentation](https://kvcache-ai.github.io/Mooncake/getting_started/supported-protocols.html).
#### Highlights
- **Decentralized architecture.** P2P Store leverages a pure client-side architecture with global metadata managed by the etcd service.
- **Widely adopted across the LLM ecosystem.** TE is used in production inference stacks such as [SGLang](https://github.com/sgl-project/sglang), [vLLM](https://github.com/vllm-project/vllm), [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [vLLM-Ascend](https://github.com/vllm-project/vllm-ascend), [checkpoint-engine](https://github.com/MoonshotAI/checkpoint-engine), and [NIXL](https://github.com/ai-dynamo/nixl), among others, to efficiently transfer KV cache, embeddings, model weights, and other data.
- **Efficient data distribution.** Designed to enhance the efficiency of large-scale data distribution, P2P Store *avoids bandwidth saturation* issues by allowing replicated nodes to share data directly. This reduces the CPU/RDMA NIC pressures of data providers (e.g., trainers).
</details>
<!-- #### Performance
Thanks to the high performance of Transfer Engine, P2P Stores can also distribute objects with full utilization of *hardware incoming bandwidth* (e.g., A 25Gbps NIC was used in the following figure, and the throughput of get replica is about 3.1 GB/s). -->
### Mooncake Store
<!-- ![p2p-store.gif](image/p2p-store.gif) -->
Mooncake Store is a high-performance distributed key-value cache storage engine designed for LLM inference. Built on the Transfer Engine, it stores and manages reusable KV caches and model weights across inference clusters, with support for efficient object storage, replication, eviction, and high-bandwidth data transfer. See the [Mooncake Store guide](https://kvcache-ai.github.io/Mooncake/design/mooncake-store.html) for details.
### Mooncake Store ([Guide](https://kvcache-ai.github.io/Mooncake/design/mooncake-store.html))
Mooncake Store is a distributed KVCache storage engine specialized for LLM inference based on Transfer Engine. It is the central component of the KVCache-centric disaggregated architecture. The goal of Mooncake Store is to store the reusable KV caches across various locations in an inference cluster. Mooncake Store has been supported in [SGLang's Hierarchical KV Caching](https://lmsys.org/blog/2025-09-10-sglang-hicache/), [vLLM's prefill serving](https://docs.vllm.ai/en/latest/features/disagg_prefill.html) and is now integrated with [LMCache](https://kvcache-ai.github.io/Mooncake/getting_started/examples/lmcache-integration.html) to provide enhanced KVCache management capabilities.
<details>
<summary>Highlights</summary>
#### Highlights
- **Multi-replica support**: Mooncake Store supports storing multiple data replicas for the same object, effectively alleviating hotspots in access pressure.
- **High bandwidth utilization.** Mooncake Store supports large-object striping, parallel I/O, and end-to-end zero-copy data transfer, fully utilizing aggregated bandwidth across multiple NICs.
- **High bandwidth utilization**: Mooncake Store supports striping and parallel I/O transfer of large objects, fully utilizing multi-NIC aggregated bandwidth for high-speed data reads and writes.
- **Multi-tier cache hierarchy**. Mooncake Store supports a multi-level cache design across DRAM and SSD/NVMe, enabling larger cache capacity.
### SGLang Integration ([Guide](https://kvcache-ai.github.io/Mooncake/getting_started/examples/sglang-integration/hicache-integration-v1.html))
- **Elastic and disaggregated storage.** Mooncake Store decouples KVCache storage from inference engines, allowing storage nodes to be dynamically added or removed while keeping cached data independent from engine restarts, upgrades, and scheduling decisions.
SGLang officially supports Mooncake Store as a [HiCache storage backend](https://lmsys.org/blog/2025-09-10-sglang-hicache/). This integration enables scalable KV cache retention and high-performance access for large-scale LLM serving scenarios.
- **Programmatic object management.** Mooncake Store allows applications to control object placement and lifecycle through per-object policies, including replica counts, preferred segments, soft pin, and hard pin. These controls help inference systems protect important KV caches and model weights while guiding replication, placement, and eviction behavior.
- **Broad ecosystem adoption.** Mooncake Store is used across the LLM systems ecosystem as a high-performance distributed storage backend for KV caches, hidden states, and model weights. It supports integrations with [SGLang's Hierarchical KV Caching](https://lmsys.org/blog/2025-09-10-sglang-hicache/), [vLLM's prefill serving](https://docs.vllm.ai/en/latest/features/disagg_prefill.html), and [LMCache](https://kvcache-ai.github.io/Mooncake/getting_started/examples/lmcache-integration.html), and has been adopted by systems such as [TorchSpec](https://pytorch.org/blog/torchspec-speculative-decoding-training-at-scale/) and [TransferQueue](https://github.com/Ascend/TransferQueue) to decouple inference, training, and reinforcement-learning workloads through efficient state management and asynchronous data movement.
</details>
### Mooncake EP and Process Group (PG)
Mooncake EP and Mooncake PG extend Mooncake from high-performance data movement to fault-tolerant distributed execution for large-scale MoE inference. Mooncake EP adapts DeepEP-style expert-parallel dispatch and combine operations with rank activeness awareness, while Mooncake PG provides a PyTorch distributed process-group backend with collective communication primitives that can detect failed ranks, report failures to upper layers, and recover ranks without restarting the entire inference service. See the [Mooncake EP & Backend guide](https://kvcache-ai.github.io/Mooncake/python-api-reference/ep-backend.html) for details.
<details>
<summary>Highlights</summary>
- **Fault-tolerant expert parallelism.** Mooncake EP adds `active_ranks` awareness to expert-parallel dispatch and combine APIs, allowing MoE inference systems to route around failed ranks and continue serving with healthy experts.
- **DeepEP-compatible programming model.** Mooncake EP keeps the API largely consistent with DeepEP's low-latency mode, making it easier for inference engines to adopt fault-tolerant expert parallelism without rewriting their MoE communication stack.
- **PyTorch ProcessGroup integration.** Mooncake PG can be registered as a `torch.distributed` backend, enabling standard collective APIs such as `all_gather` while using Mooncake's communication and failure-reporting mechanisms underneath.
- **Elastic rank recovery.** Mooncake PG exposes recovery-oriented primitives such as peer-state polling and rank recovery, allowing replacement processes to rejoin existing process groups and helping inference services recover from partial failures.
- **SGLang integration for production MoE serving.** Mooncake's collective backend and expert-parallel kernels are integrated into SGLang to support fault-tolerant expert-parallel inference for large MoE models, including Elastic Expert Parallel serving scenarios.
</details>
### Tensor-Centric Ecosystem
Mooncake establishes a full-stack, Tensor-oriented AI infrastructure where Tensors serve as the fundamental data carrier. The ecosystem spans from the Transfer Engine, which accelerates Tensor data movement across heterogeneous storage (DRAM/VRAM/NVMe), to Mooncake Store for distributed management of Tensor objects (e.g., KVCache and model weight), up to the Mooncake Backend enabling Tensor-based elastic distributed computing. This architecture is designed to maximize Tensor processing efficiency for large-scale model inference and training.
### SGLang Integration ([Guide](https://kvcache-ai.github.io/Mooncake/getting_started/examples/sglang-integration/index.html))
Mooncake is deeply integrated into [SGLang](https://github.com/sgl-project/sglang/) as a high-performance communication and storage backend. These integrations enable efficient KV cache transfer in PD-disaggregated serving, scalable multi-level KV caching through HiCache, fault-tolerant expert-parallel inference, high-performance multimodal pipeline data movement, and fast RDMA-based weight synchronization for large-scale RL training. Together, Mooncake and SGLang provide a production-oriented foundation for building elastic, high-throughput, and resource-efficient LLM and multimodal serving systems.
<details>
<summary>Details</summary>
- **PD Disaggregated Serving:** SGLang officially supports Mooncake Transfer Engine as a backend for disaggregated serving and KV cache transfer, enabling prefill and decode workers to exchange KV cache data efficiently across devices and machines.
#### Highlights
- **Hierarchical KV Caching**: Mooncake Store serves as an external storage backend in SGLang's HiCache system, extending RadixAttention with multi-level KV cache storage across device, host, and remote storage layers.
- **Flexible Cache Management**: Supports multiple cache policies including write-through, write-through-selective, and write-back modes, with intelligent prefetching strategies for optimal performance.
- **Comprehensive Optimizations**: Features advanced data plane optimizations including page-first memory layout for improved I/O efficiency, zero-copy mechanisms for reduced memory overhead, GPU-assisted I/O kernels delivering fast CPU-GPU transfers, and layer-wise overlapping for concurrent KV cache loading while computation executes.
- **Elastic Expert Parallel**: Mooncake's collective communication backend and expert parallel kernels are integrated into SGLang to enable fault-tolerant expert parallel inference ([sglang#11657](https://github.com/sgl-project/sglang/pull/11657)).
- **Significant Performance Gains**: The multi-turn benchmark demonstrates substantial performance improvements over the non-HiCache setting. See our [benchmark report](https://kvcache-ai.github.io/Mooncake/performance/sglang-hicache-benchmark-results-v1.html) for more details.
- **Community Feedback**: Effective KV caching significantly reduces TTFT by eliminating redundant and costly re-computation. Integrating SGLang HiCache with the Mooncake service enables scalable KV cache retention and high-performance access. In our evaluation, we tested the DeepSeek-R1-671B model under PD-disaggregated deployment using in-house online requests sampled from a general QA scenario. On average, cache hits achieved an 84% reduction in TTFT compared to full re-computation. – Ant Group
### vLLM Integration ([Guide v0.2](https://kvcache-ai.github.io/Mooncake/getting_started/examples/vllm-integration/vllm-integration-v0.2.html))
To optimize LLM inference, the vLLM community is working on supporting [disaggregated prefilling (PR 10502)](https://github.com/vllm-project/vllm/pull/10502). This feature allows separating the **prefill** phase from the **decode** phase in different processes. The vLLM uses `nccl` and `gloo` as the transport layer by default, but currently it cannot efficiently decouple both phases in different machines.
- **Elastic Expert Parallel**: Mooncake's collective communication backend and expert parallel kernels are integrated into SGLang to enable fault-tolerant expert parallel inference ([Elastic EP](https://www.lmsys.org/blog/2026-03-25-eep-partial-failure-tolerance/)).
We have implemented vLLM integration, which uses Transfer Engine as the network layer instead of `nccl` and `gloo`, to support **inter-node KVCache transfer** [(PR 10884)](https://github.com/vllm-project/vllm/pull/10884). Transfer Engine provides simpler interfaces and more efficient use of RDMA devices.
- **Cloud-Native SGLang HiCache Deployment with RBG**: The [RBG](https://github.com/sgl-project/rbg) + SGLang HiCache + Mooncake integration provides a role-based, out-of-the-box cloud-native deployment solution that is elastic, scalable, and optimized for high-performance inference workloads.
We will soon release the new vLLM integration based on Mooncake Store, which supports xPyD prefill/decode disaggregation.
- **Encode-Prefill-Decode Disaggregation for Multimodal Serving**: SGLang introduces Encode-Prefill-Decode disaggregation with Mooncake as a transfer backend. This enables compute-intensive multimodal encoders, such as Vision Transformers, to be decoupled from language model workers while transferring large embeddings efficiently through Mooncake’s RDMA-based engine.
**_Update[Dec 16, 2024]: Here is the latest vLLM Integration ([Guide v0.2](https://kvcache-ai.github.io/Mooncake/getting_started/examples/vllm-integration/vllm-integration-v0.2.html)) that is based on vLLM's main branch._**
- **SGLang-Omni Multi-Stage Pipeline Data Transfer**: [SGLang-Omni](https://github.com/sgl-project/sglang-omni) integrates Mooncake as a relay backend for efficient cross-stage tensor and blob transfer in multimodal serving pipelines. This enables high-performance data movement between heterogeneous components such as thinker, talker, codec, and vocoder stages.
#### Performance
By supporting Topology Aware Path Selection and multi-card bandwidth aggregation, Mean TTFT of vLLM with Transfer Engine is up to 25% lower than traditional TCP-based transports.
In the future, we will further improve TTFT through GPUDirect RDMA and zero-copy.
- **RDMA-Based P2P Weight Transfer for Distributed RL**: SGLang adopts Mooncake TransferEngine for RDMA-based peer-to-peer weight transfer in large-scale distributed reinforcement learning. This enables zero-copy weight updates across thousands of GPUs and significantly accelerates synchronization for trillion-parameter models.
| Backend/Setting | Output Token Throughput (tok/s) | Total Token Throughput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms)|
|---------------------------------------------------------|---------------------------------|--------------------------------|----------------|------------------|---------------|
| Transfer Engine (RDMA) | 12.06 | 2042.74 | 1056.76 | 635.00 | 4006.59 |
| TCP | 12.05 | 2041.13 | 1414.05 | 766.23 | 6035.36 |
</details>
- Click [here](https://kvcache-ai.github.io/Mooncake/performance/vllm-benchmark-results-v0.2.html) to access detailed benchmark results.
### vLLM Integration ([Guide](https://kvcache-ai.github.io/Mooncake/getting_started/examples/vllm-integration/index.html))
**More advanced features are coming soon, so stay tuned!**
Mooncake integrates with [vLLM](https://github.com/vllm-project/vllm) to accelerate large language model serving through high-performance KV cache transfer and distributed KV cache storage. The integration supports both disaggregated prefill-decode serving and cross-instance KV cache sharing, helping vLLM deployments reduce TTFT, improve cache reuse, and scale more efficiently across multi-node inference clusters.
<h2 id="quick-start">🚀 Quick Start</h2>
<details>
<summary>Details</summary>
### Before using Mooncake
- **Disaggregated prefill-decode serving**: Mooncake enables vLLM to split prefill and decode workloads across different nodes. Through MooncakeConnector, vLLM transfers KV cache blocks from prefill workers to decode workers using Mooncake’s high-performance transfer engine, allowing prefill and decode resources to scale independently while keeping cross-node KV transfer overhead low.
Mooncake is designed and optimized for high-speed RDMA networks. Though Mooncake supports TCP-only data transfer, we **strongly** recommend users to evaluate the functionality and performance of Mooncake with RDMA network support.
- **Distributed KV cache pooling and sharing**: [Mooncake Store extends vLLM](https://vllm.ai/blog/2026-05-06-mooncake-store) from isolated per-instance KV caches to a shared, cluster-level KV cache pool. Through MooncakeStoreConnector, multiple vLLM instances can store, retrieve, and reuse KV cache blocks based on hash-based prefix caching, reducing redundant prefill computation and improving cache efficiency for workloads with repeated prefixes, especially agentic and multi-turn serving scenarios.
The following need to be installed before running any component of Mooncake:
- RDMA Driver & SDK, such as Mellanox OFED.
- Python 3.10, virtual environment is recommended.
- CUDA 12.1 and above, including NVIDIA GPUDirect Storage Support, if the package is built with `-DUSE_CUDA` (disabled by default). *You may install them from [here](https://developer.nvidia.com/cuda-downloads)*.
- Cambricon Neuware, if the package is built with `-DUSE_MLU`. By default Mooncake looks for Neuware under `NEUWARE_HOME` or `/usr/local/neuware`.
- **vLLM-Omni stage communication**: Mooncake also integrates with [vLLM-Omni](https://github.com/vllm-project/vllm-omni) through `MooncakeTransferEngineConnector` and `MooncakeStoreConnector`, enabling efficient cross-node data exchange between vLLM-Omni stages.
### Use Python package
The simplest way to use Mooncake Transfer Engine is using `pip`:
</details>
**For CUDA-enabled systems:**
<h2 id="supported-hardware">🖥️ Supported Hardware</h2>
Mooncake supports hardware backends across accelerator vendors, cloud fabrics, and standard datacenter interconnects, as listed below. See the [supported protocols](https://kvcache-ai.github.io/Mooncake/getting_started/supported-protocols.html) and [Transfer Engine design docs](https://kvcache-ai.github.io/Mooncake/design/transfer-engine/index.html) for details.
| <img src="image/partners/nvidia_logo.png" width="120" alt="NVIDIA"/> | <img src="image/partners/huawei_logo.png" width="120" alt="Huawei"/> | <img src="image/partners/amd_logo.png" width="120" alt="AMD"/> | <img src="image/hardwares/cambricon_logo.png" width="120" alt="Cambricon"/> | <img src="image/partners/moore_thread_logo.jpg" width="120" alt="Moore Threads"/> | <img src="image/partners/aws-logo.png" width="120" alt="AWS"/> |
| --- | --- | --- | --- | --- | --- |
| <img src="image/hardwares/MetaX_logo.png" width="120" alt="MetaX"/> | <img src="image/hardwares/T-Head_logo.png" width="120" alt="T-Head"/> | <img src="image/partners/aliyun_logo.png" width="120" alt="Alibaba Cloud"/> | <img src="image/partners/sunrise_logo.png" width="120" alt="Sunrise"/> | <img src="image/partners/hygon_logo.png" width="120" alt="Hygon"/> | |
<h2 id="quick-start">🚀 Getting Started</h2>
Install Mooncake using `pip`. The `mooncake-transfer-engine` package includes Mooncake Transfer Engine, Mooncake Store, Mooncake EP and PG:
- CUDA < 13.0
```bash
@ -194,108 +210,84 @@ pip install mooncake-transfer-engine
pip install mooncake-transfer-engine-cuda13
```
**For non-CUDA systems:**
```bash
pip install mooncake-transfer-engine-non-cuda
In addition to CUDA, Mooncake also supports other accelerator backends, along with flexible installation and deployment options. See the guides below for details:
- [Quick Start](https://kvcache-ai.github.io/Mooncake/getting_started/quick-start.html)
- [Build from Source](https://kvcache-ai.github.io/Mooncake/getting_started/build.html)
- [Deployment Guide](https://kvcache-ai.github.io/Mooncake/deployment/mooncake-store-deployment-guide.html)
### Skills for AI Assistants
Mooncake ships a set of **built-in skills** under [`.claude/skills`](.claude/skills) — reusable, task-focused playbooks that an AI coding assistant (such as Claude Code) invokes automatically when your request matches, or that you can run as a slash command.
<details>
<summary>Details</summary>
| Skill | Description |
|-------|-------------|
| `/mooncake-troubleshoot` | Diagnose Mooncake deployment and runtime issues (services, RDMA, env vars, logs). |
| `/mooncake-ci-local` | Run pre-PR local validation via `scripts/run_ci_test.sh`. |
| `/mooncake-api` | Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs. |
Install them without cloning the repository via the [Claude Code plugin marketplace](https://code.claude.com/docs/en/plugin-marketplaces):
```text
/plugin marketplace add kvcache-ai/Mooncake --sparse .claude-plugin
/plugin install mooncake-troubleshoot@mooncake
/plugin install mooncake-ci-local@mooncake
/plugin install mooncake-api@mooncake
```
> [!IMPORTANT]
> - The CUDA version (`mooncake-transfer-engine`) includes Mooncake-EP and GPU topology detection, requiring CUDA 12.1+.
> - The non-CUDA version (`mooncake-transfer-engine-non-cuda`) is for environments without CUDA dependencies.
> - MLU support is currently available through source builds with `-DUSE_MLU=ON`; there is no dedicated prebuilt MLU wheel yet.
> - If users encounter problems such as missing `lib*.so`, they should uninstall the package they installed and build the binaries manually.
The `--sparse .claude-plugin` flag fetches only the marketplace catalog, and each plugin is published as a `git-subdir` source, so installing one fetches only that single skill directory — never the whole repo. If you are already working inside a Mooncake checkout, the skills under `.claude/skills/` load automatically with no setup.
### Use Docker image
Mooncake supports Docker-based deployment, see [Build Guide](https://kvcache-ai.github.io/Mooncake/getting_started/build.html) in detail.
</details>
To produce an image that compiles Mooncake from source, builds the wheel via `scripts/build_wheel.sh`, and installs that wheel inside the container, use `build-wheel.dockerfile`:
<h2 id="trace">📦 Open Source Traces and Tools </h2>
```bash
docker build -f docker/mooncake.Dockerfile \
--build-arg PYTHON_VERSION=3.10 \
--build-arg EP_TORCH_VERSIONS="2.9.1" \
-t mooncake:from-source .
```
We open-source anonymized request traces containing request arrival times, input and output token counts, and remapped block hashes. These traces are designed to support reproducible simulation and evaluation of caching behavior while preserving user privacy. The released traces and related details are available in [FAST25-release](FAST25-release).
The resulting image already has a virtual environment at `/opt/venv` with the freshly built wheel installed. Launch it with GPU/RDMA access as needed, for example:
```bash
docker run --gpus all --network host -it mooncake:from-source /bin/bash
```
> [!NOTE]
> Make sure you build the image from the repository root so that Git metadata and submodules are available inside the build context.
### Build and use binaries
The following are additional dependencies for building Mooncake:
- Build essentials, including gcc, g++ (9.4+) and cmake (3.16+).
- Go 1.20+, if you want to build with `-DWITH_P2P_STORE`, `-DUSE_ETCD` (enabled by default to use etcd as metadata servers), or `-DSTORE_USE_ETCD` (use etcd for the failover of the store master).
- CUDA 12.1 and above, including NVIDIA GPUDirect Storage Support, if the package is built with `-DUSE_CUDA`. *This is NOT included in the `dependencies.sh` script. You may install them from [here](https://developer.nvidia.com/cuda-downloads)*.
- Cambricon Neuware, if you want to build with `-DUSE_MLU`. *This is NOT included in the `dependencies.sh` script.* Mooncake resolves it from `NEUWARE_HOME` or `/usr/local/neuware` by default, and also supports overriding `MLU_INCLUDE_DIR` / `MLU_LIB_DIR` during CMake configure.
- [Optional] Rust Toolchain, if you want to build with `-DWITH_RUST_EXAMPLE`. *This is NOT included in the `dependencies.sh` script.*
- [Optional] `hiredis`, if you want to build with `-DUSE_REDIS` to use Redis instead of etcd as metadata servers.
- [Optional] `curl`, if you want to build with `-DUSE_HTTP` to use HTTP instead of etcd as metadata servers.
The build and installation steps are as follows:
1. Retrieve source code from GitHub repo
```bash
git clone https://github.com/kvcache-ai/Mooncake.git
cd Mooncake
```
2. Install dependencies
```bash
bash dependencies.sh
```
3. Compile Mooncake and examples
```bash
mkdir build
cd build
cmake ..
make -j
sudo make install # optional, make it ready to be used by vLLM/SGLang
```
For Cambricon MLU builds, configure CMake with `-DUSE_MLU=ON`. For example:
```bash
mkdir build
cd build
cmake .. -DUSE_MLU=ON -DNEUWARE_ROOT=/usr/local/neuware
make -j
```
<h2 id="milestones"> 🛣️ Incoming Milestones</h2>
- [x] First release of Mooncake and integrate with latest vLLM
- [ ] Share KV caches across multiple serving engines
- [ ] User and developer documentation
<h2 id="trace">📦 Open Source Trace</h2>
```json
{
"timestamp": 27482,
"input_length": 6955,
"output_length": 52,
"hash_ids": [46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 2353, 2354]
}
{
"timestamp": 30535,
"input_length": 6472,
"output_length": 26,
"hash_ids": [46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 2366]
}
```
The above presents two samples from our trace dataset. The trace includes the timing of request arrivals, the number of input tokens, the number of output tokens, and the remapped block hash. To protect our customers' privacy, we applied several mechanisms to remove user-related information while preserving the dataset's utility for simulated evaluation. More descriptions of the trace (e.g., up to 50% cache hit ratio) can be found in Section 4 of the technical report.
**_Update[Feb 21, 2025]: The updated [traces](FAST25-release/traces) used in our FAST'25 paper have been released! Please refer to the paper's appendix (found [here](FAST25-release/Mooncake-FAST25.pdf)) for more details._**
Together with the released traces, we also provide two KV cache analysis tools: a [KV Cache Size Calculator](https://kvcache.ai/tools/kv-cache-size-calculator/) for calculating cache capacity across popular LLM model families, and a [KV Cache Hit Rate Simulator](https://kvcache.ai/tools/kv-cache-hit-rate-simulator/) for analyzing KV cache hit rates and planning cache capacity under different workloads and models. These tools help users better understand KV cache storage costs and caching effectiveness when analyzing or reproducing serving workloads. The tools are open-sourced [here](https://github.com/kvcache-ai/kvcache-blog).
<h2 id="citation">📑 Citation</h2>
Please kindly cite our paper if you find the paper or the traces are useful:
Please kindly cite our papers if you find the papers or the traces are useful:
```bibtex
@inproceedings{qin2025mooncake,
author = {Ruoyu Qin and Zheming Li and Weiran He and Jialei Cui and Feng Ren and Mingxing Zhang and Yongwei Wu and Weimin Zheng and Xinran Xu},
title = {Mooncake: Trading More Storage for Less Computation {\textemdash} A {KVCache-centric} Architecture for Serving {LLM} Chatbot},
booktitle = {23rd USENIX Conference on File and Storage Technologies (FAST 25)},
year = {2025},
isbn = {978-1-939133-45-8},
address = {Santa Clara, CA},
pages = {155--170},
url = {https://www.usenix.org/conference/fast25/presentation/qin},
publisher = {USENIX Association},
month = {feb},
}
```
<details>
<summary>More</summary>
```bibtex
@misc{ren2026tentdeclarativeslicespraying,
title = {TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving},
author = {Feng Ren and Ruoyu Qin and Teng Ma and Shangming Cai and Zheng Liu and Chao Lei and Dejiang Zhu and Ke Yang and Zheming Li and Jialei Cui and Weixiao Huang and Yikai Zhao and Yineng Zhang and Hao Wu and Xiang Gao and Yuhao Fu and Jinlei Jiang and Yongwei Wu and Mingxing Zhang},
year = {2026},
eprint = {2604.00368},
archivePrefix = {arXiv},
primaryClass = {cs.DC},
url = {https://arxiv.org/abs/2604.00368},
}
@article{sun2026survivingpartialrankfailures,
title = {Surviving Partial Rank Failures in Wide Expert-Parallel MoE Inference},
author = {Xun Sun and Shaoyuan Chen and Pingchuan Ma and Yue Chen and Ziwei Yuan and Zhanhao Cao and Han Han and Shangming Cai and Teng Ma and Xuchun Shang and Xinpeng Zhao and Ke Yang and Junlin Wei and Lianzhi Lin and Yuji Liu and Feng Ren and Haoran Hu and Cheng Wan and Yingdi Shan and Yongwei Wu and Mingxing Zhang},
year = {2026},
url = {https://arxiv.org/abs/2605.10670},
}
@article{qin2025mooncake_tos,
author = {Qin Ruoyu and Li Zheming and He Weiran and Cui Jialei and Tang Heyi and Ren Feng and Ma Teng and Cai Shangming and Zhang Yineng and Zhang Mingxing and Wu Yongwei and Zheng Weimin and Xu Xinran},
title = {Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving},
@ -310,19 +302,6 @@ Please kindly cite our paper if you find the paper or the traces are useful:
keywords = {Machine learning system, LLM serving, KVCache},
}
@inproceedings{qin2025mooncake,
author = {Ruoyu Qin and Zheming Li and Weiran He and Jialei Cui and Feng Ren and Mingxing Zhang and Yongwei Wu and Weimin Zheng and Xinran Xu},
title = {Mooncake: Trading More Storage for Less Computation {\textemdash} A {KVCache-centric} Architecture for Serving {LLM} Chatbot},
booktitle = {23rd USENIX Conference on File and Storage Technologies (FAST 25)},
year = {2025},
isbn = {978-1-939133-45-8},
address = {Santa Clara, CA},
pages = {155--170},
url = {https://www.usenix.org/conference/fast25/presentation/qin},
publisher = {USENIX Association},
month = {feb},
}
@article{qin2024mooncake_arxiv,
title = {Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving},
author = {Ruoyu Qin and Zheming Li and Weiran He and Mingxing Zhang and Yongwei Wu and Weimin Zheng and Xinran Xu},
@ -330,3 +309,5 @@ Please kindly cite our paper if you find the paper or the traces are useful:
url = {https://arxiv.org/abs/2407.00079},
}
```
</details>

41
benchmarks/storage_benchmark_v1/__init__.py Normal file
View File

@ -0,0 +1,41 @@
"""
Mooncake KVCache Storage Benchmark Library
"""
__version__ = "2.0.0"
from benchmark import (
main,
run_benchmark,
print_results,
StorageBenchmark,
TraceReplay,
KVCacheRequest,
)
from storage import Storage, KVKey, KVValue, DiskHashTable, SSDStorage
from layout import (
KVLayout, MLALayout, KVEntry, MLA_MODEL_CONFIG, get_model_config, create_layout,
)
__all__ = [
# Main
'main',
'run_benchmark',
'print_results',
'StorageBenchmark',
'TraceReplay',
'KVCacheRequest',
# Storage
'Storage',
'KVKey',
'KVValue',
'DiskHashTable',
'SSDStorage',
# Layout
'KVLayout',
'MLALayout',
'KVEntry',
'MLA_MODEL_CONFIG',
'get_model_config',
'create_layout',
]

11
benchmarks/storage_benchmark_v1/__main__.py Normal file
View File

@ -0,0 +1,11 @@
"""
Entry point for running the benchmark as a module
Usage:
python -m storage_benchmark --model=glm5 --scenario=toolagent --max-requests=100
"""
from benchmark import main
if __name__ == '__main__':
main()

486
benchmarks/storage_benchmark_v1/benchmark.py Normal file
View File

@ -0,0 +1,486 @@
#!/usr/bin/env python3
"""
Mooncake KVCache Storage Benchmark
Complete benchmark tool with CLI interface.
"""
import argparse
import json
import sys
import time
import statistics
import signal
from dataclasses import dataclass
from pathlib import Path
from typing import Iterator, List, Dict, Any
from storage import DiskHashTable
from layout import get_model_config, create_layout
# ============================================================================
# Data Structures
# ============================================================================
@dataclass
class KVCacheRequest:
"""KVCache request from trace"""
timestamp: float
hash_ids: List[int]
input_length: int
output_length: int
# ============================================================================
# Trace Replay
# ============================================================================
class TraceReplay:
"""Trace replay handler"""
def __init__(self, trace_path: str):
self.trace_path = trace_path
def load_all(self) -> List[KVCacheRequest]:
"""Load all requests from trace file"""
requests = []
with open(self.trace_path, 'r', encoding='utf-8') as f:
for line in f:
line = line.strip()
if line:
req = json.loads(line)
requests.append(KVCacheRequest(
timestamp=req.get('timestamp', 0),
hash_ids=req.get('hash_ids', []),
input_length=req.get('input_length', 0),
output_length=req.get('output_length', 0),
))
return requests
# ============================================================================
# Storage Benchmark
# ============================================================================
class StorageBenchmark:
"""KVCache storage benchmark
Processes KVCache requests using layout-generated access patterns.
"""
def __init__(self, storage_dir: str, model_config: dict,
page_size_tokens: int = 512,
max_pages: int = 100000,
fsync_mode: str = 'none', fsync_batch_size: int = 100):
"""Initialize benchmark
Args:
storage_dir: Directory for storage files
model_config: Model configuration dict
page_size_tokens: Tokens per page (default: 512)
max_pages: Maximum number of pages
fsync_mode: When to fsync ('none', 'batch', 'always', 'end')
fsync_batch_size: Number of writes between fsync in batch mode
"""
self.model_config = model_config
self.layout = create_layout(model_config, page_size_tokens)
self.page_size_bytes = self.layout.value_size_bytes
# Initialize storage
self.storage = DiskHashTable(
storage_dir=storage_dir,
page_size=self.page_size_bytes,
max_pages=max_pages,
fsync_mode=fsync_mode,
fsync_batch_size=fsync_batch_size
)
# Statistics
self.stats = {
'total_requests': 0,
'total_tokens': 0,
'read_pages': 0,
'write_pages': 0,
'page_hits': 0,
'request_latencies_ms': [],
}
def process_request(self, req: KVCacheRequest) -> float:
"""Process a KVCache request
Args:
req: KVCache request
Returns:
Total latency in milliseconds
"""
self.stats['total_requests'] += 1
self.stats['total_tokens'] += req.input_length + req.output_length
total_latency = 0.0
# Process each access requirement from layout
for access in self.layout.get_operations(req):
if self.storage.exists(access.page_id):
# Page exists, perform READ
total_latency += self.storage.read(
access.page_id,
offset_in_page=access.offset_in_page,
length=access.length
)
self.stats['read_pages'] += 1
self.stats['page_hits'] += 1
else:
# Page doesn't exist, perform WRITE
total_latency += self.storage.write(
access.page_id,
offset_in_page=access.offset_in_page,
length=access.length
)
self.stats['write_pages'] += 1
latency_ms = total_latency if total_latency > 0 else 0.0
if latency_ms > 0:
self.stats['request_latencies_ms'].append(latency_ms)
return latency_ms
def get_stats(self) -> Dict:
"""Get statistics"""
storage_stats = self.storage.get_stats()
request_latencies = self.stats['request_latencies_ms']
if request_latencies:
sorted_latencies = sorted(request_latencies)
n = len(sorted_latencies)
def get_percentile(p: float) -> float:
idx = int(n * p)
return sorted_latencies[idx] if idx < n else sorted_latencies[-1]
latency_stats = {
'avg_ms': statistics.mean(request_latencies),
'p50_ms': sorted_latencies[n // 2],
'p95_ms': get_percentile(0.95),
'p99_ms': get_percentile(0.99),
}
else:
latency_stats = {'avg_ms': 0, 'p50_ms': 0, 'p95_ms': 0, 'p99_ms': 0}
total_pages = self.stats['read_pages'] + self.stats['write_pages']
return {
'total_requests': self.stats['total_requests'],
'total_tokens': self.stats['total_tokens'],
'total_pages': total_pages,
'read_pages': self.stats['read_pages'],
'write_pages': self.stats['write_pages'],
'page_hits': self.stats['page_hits'],
'page_hit_rate': self.stats['read_pages'] / total_pages if total_pages > 0 else 0,
'write_ratio': self.stats['write_pages'] / total_pages if total_pages > 0 else 0,
'latency': latency_stats,
'storage': storage_stats,
}
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.close()
return False
def close(self, force_sync: bool = True):
self.storage.close(force_sync=force_sync)
# ============================================================================
# Benchmark Runner
# ============================================================================
def get_max_page_id(requests: List[KVCacheRequest]) -> int:
max_id = 0
for req in requests:
if req.hash_ids:
max_id = max(max_id, max(req.hash_ids))
return max_id
def run_benchmark(trace_path: str, storage_dir: str, model_config: dict,
max_requests: int = None, max_pages: int = None,
page_size_tokens: int = 512,
fsync_mode: str = 'none', fsync_batch_size: int = 100) -> Dict:
"""Run benchmark
Args:
trace_path: Trace file path
storage_dir: Storage directory
model_config: Model configuration
max_requests: Maximum number of requests (None = all)
max_pages: Maximum number of pages (None = auto-calculate)
page_size_tokens: Tokens per page
fsync_mode: When to fsync
fsync_batch_size: Number of writes between fsync
Returns:
Benchmark results dictionary
"""
print(f"\n{'='*80}")
print(f"Running: {Path(trace_path).name}")
print(f"Model: {model_config['name']}")
print(f"Layers: {model_config['num_layers']}")
print(f"Page size: {page_size_tokens} tokens")
print(f"{'='*80}")
# Load trace
replay = TraceReplay(trace_path)
requests = replay.load_all()
if max_requests:
requests = requests[:max_requests]
print(f"Loaded {len(requests)} requests")
# Find max page_id from trace
max_page_id = get_max_page_id(requests)
max_pages_needed = max_page_id + 1 # page_id is 0-based
# Create layout to get page size
layout = create_layout(model_config, page_size_tokens)
page_size_bytes = layout.value_size_bytes
# Determine max_pages
if max_pages is None:
# Use max page_id from trace
max_pages = max_pages_needed
# Round up to next thousand for cleaner numbers
max_pages = ((max_pages + 999) // 1000) * 1000
else:
# User specified max_pages
pass
max_size_gb = max_pages * page_size_bytes / (1024**3)
trace_size_gb = max_pages_needed * page_size_bytes / (1024**3)
print(f"\n[Storage Configuration]")
print(f" Max page_id in trace: {max_page_id:,}")
print(f" Pages needed (trace): {max_pages_needed:,}")
print(f" Trace storage size: {trace_size_gb:.2f} GB")
print(f" Max pages configured: {max_pages:,}")
print(f" Max storage available: {max_size_gb:.2f} GB")
if max_pages_needed > max_pages:
shortfall = max_pages_needed - max_pages
shortfall_gb = shortfall * page_size_bytes / (1024**3)
compression_ratio = max_pages / max_pages_needed
print(f"\n ⚠️ Storage insufficient: {shortfall:,} pages shortfall ({shortfall_gb:.2f} GB)")
print(f" ⚠️ Consider increasing --max-pages to at least {max_pages_needed:,} for full simulation")
else:
surplus = max_pages - max_pages_needed
surplus_pct = (surplus / max_pages) * 100 if max_pages > 0 else 0
print(f" ✓ Direct mapping: all {max_pages_needed:,} logical pages uniquely mapped")
# Run benchmark
with StorageBenchmark(
storage_dir=storage_dir,
model_config=model_config,
page_size_tokens=page_size_tokens,
max_pages=max_pages,
fsync_mode=fsync_mode,
fsync_batch_size=fsync_batch_size
) as benchmark:
start_time = time.perf_counter()
try:
for i, req in enumerate(requests):
benchmark.process_request(req)
# Print progress for each request
elapsed = time.perf_counter() - start_time
qps = (i + 1) / elapsed if elapsed > 0 else 0
stats = benchmark.get_stats()
storage = stats.get('storage', {})
read_latency = storage.get('read', {}).get('avg_ms', 0)
write_latency = storage.get('write', {}).get('avg_ms', 0)
read_mb = storage.get('read', {}).get('mb', 0)
write_mb = storage.get('write', {}).get('mb', 0)
read_time = storage.get('read', {}).get('time_s', 0)
write_time = storage.get('write', {}).get('time_s', 0)
read_mbps = read_mb / read_time if read_time > 0 else 0
write_mbps = write_mb / write_time if write_time > 0 else 0
print(f" [{i+1:5d}/{len(requests)}] ids={len(req.hash_ids):3d} "
f"tokens={req.input_length+req.output_length:6d} | "
f"QPS={qps:7.2f} | "
f"R={stats['read_pages']:6d} ({read_latency:6.2f}ms, {read_mbps:6.1f}MB/s) | "
f"W={stats['write_pages']:6d} ({write_latency:6.2f}ms, {write_mbps:6.1f}MB/s)")
except KeyboardInterrupt:
print(f"\n\n{'='*80}")
print(f"Interrupted! Showing partial results:")
print(f"{'='*80}")
elapsed = time.perf_counter() - start_time
stats = benchmark.get_stats()
print_results([{
'trace_file': Path(trace_path).name,
'total_requests': i + 1,
'io_time_s': elapsed,
'requests_per_second': (i + 1) / elapsed if elapsed > 0 else 0,
'model': model_config['name'],
'fsync_mode': fsync_mode,
**stats,
}])
sys.exit(0)
elapsed = time.perf_counter() - start_time
stats = benchmark.get_stats()
return {
'trace_file': Path(trace_path).name,
'total_requests': len(requests),
'io_time_s': elapsed,
'requests_per_second': len(requests) / elapsed if elapsed > 0 else 0,
'model': model_config['name'],
'fsync_mode': fsync_mode,
**stats,
}
# ============================================================================
# Output Formatting
# ============================================================================
def format_storage_stats(stats: Dict, title: str = "Storage"):
"""Format storage statistics with clear read/write separation"""
storage = stats.get('storage', {})
read_stats = storage.get('read', {})
write_stats = storage.get('write', {})
output = []
output.append(f"\n[{title}]")
# General info
output.append(f"\n[General]")
output.append(f" Model: {stats.get('model', 'N/A')}")
output.append(f" Requests: {stats.get('total_requests', 0):,}")
output.append(f" Tokens: {stats.get('total_tokens', 0):,}")
output.append(f" Total I/O Time: {stats.get('io_time_s', 0):.3f} s")
output.append(f" QPS: {stats.get('requests_per_second', 0):.2f}")
output.append(f" Hit Rate: {stats.get('page_hit_rate', 0):.2%}")
# Read Stats
output.append(f"\n[Read Operations]")
output.append(f" Count: {read_stats.get('count', 0):,}")
output.append(f" Data Volume: {read_stats.get('mb', 0):.2f} MB")
read_time = read_stats.get('time_s', 0)
read_mbps = read_stats.get('mb', 0) / read_time if read_time > 0 else 0
output.append(f" Total Time: {read_time:.3f} s")
output.append(f" Bandwidth: {read_mbps:.2f} MB/s")
output.append(f" Latency:")
output.append(f" Avg: {read_stats.get('avg_ms', 0):.3f} ms")
output.append(f" P50: {read_stats.get('p50_ms', 0):.3f} ms")
output.append(f" P95: {read_stats.get('p95_ms', 0):.3f} ms")
output.append(f" P99: {read_stats.get('p99_ms', 0):.3f} ms")
# Write Stats
output.append(f"\n[Write Operations]")
output.append(f" Count: {write_stats.get('count', 0):,}")
output.append(f" Data Volume: {write_stats.get('mb', 0):.2f} MB")
write_time = write_stats.get('time_s', 0)
write_mbps = write_stats.get('mb', 0) / write_time if write_time > 0 else 0
output.append(f" Total Time: {write_time:.3f} s")
output.append(f" Bandwidth: {write_mbps:.2f} MB/s")
output.append(f" Latency:")
output.append(f" Avg: {write_stats.get('avg_ms', 0):.3f} ms")
output.append(f" P50: {write_stats.get('p50_ms', 0):.3f} ms")
output.append(f" P95: {write_stats.get('p95_ms', 0):.3f} ms")
output.append(f" P99: {write_stats.get('p99_ms', 0):.3f} ms")
# Storage Info
output.append(f"\n[Storage Info]")
output.append(f" Max Pages: {storage.get('max_pages', 0):,}")
output.append(f" Written Pages: {storage.get('written_pages', 0):,}")
output.append(f" Sync Count: {storage.get('sync_count', 0):,}")
return "\n".join(output)
def print_results(results: List[Dict]):
"""Print benchmark results"""
for i, r in enumerate(results, 1):
print(f"\n{'='*80}")
print(f" [{i}/{len(results)}] {r['trace_file']}")
print(f"{'='*80}")
print(format_storage_stats(r))
# ============================================================================
# CLI Entry Point
# ============================================================================
def main():
"""Main entry point"""
parser = argparse.ArgumentParser(
description='Mooncake KVCache Storage Benchmark',
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument('--trace-dir', type=str, default='../../FAST25-release/traces',
help='Trace files directory')
parser.add_argument('--scenario', type=str, choices=['conversation', 'synthetic', 'toolagent', 'all'],
default='toolagent', help='Test scenario')
parser.add_argument('--storage-dir', type=str, default='/tmp/mooncake_bench',
help='Storage directory')
parser.add_argument('--model', type=str, default='glm5', choices=['glm5', 'kimi-k2.6'],
help='Model preset')
parser.add_argument('--page-size-tokens', type=int, default=512,
help='Page size in tokens (default: 512)')
parser.add_argument('--max-requests', type=int, default=None,
help='Maximum number of requests')
parser.add_argument('--max-pages', type=int, default=2000,
help='Maximum number of pages')
parser.add_argument('--fsync-mode', type=str, choices=['batch', 'always', 'end', 'none'],
default='none', help='When to fsync')
parser.add_argument('--fsync-batch-size', type=int, default=100,
help='Number of writes between fsync')
args = parser.parse_args()
print(f"\n{'='*80}")
print(f"{'Mooncake KVCache Storage Benchmark':^80}")
print(f"{'='*80}")
model_config = get_model_config(args.model)
print(f"Model: {args.model} ({model_config['num_layers']} layers)")
# Determine scenarios
scenarios = ['conversation', 'synthetic', 'toolagent'] if args.scenario == 'all' else [args.scenario]
trace_files = {
'conversation': 'conversation_trace.jsonl',
'synthetic': 'synthetic_trace.jsonl',
'toolagent': 'toolagent_trace.jsonl'
}
# Run benchmarks
results = []
for scenario in scenarios:
trace_path = Path(args.trace_dir) / trace_files[scenario]
if trace_path.exists():
result = run_benchmark(
str(trace_path),
str(Path(args.storage_dir) / scenario),
model_config,
args.max_requests,
args.max_pages,
args.page_size_tokens,
args.fsync_mode,
args.fsync_batch_size
)
results.append(result)
else:
print(f"Warning: Trace file not found: {trace_path}")
# Print results
if results:
print_results(results)
else:
print("Error: No trace files were successfully processed.", file=sys.stderr)
sys.exit(1)
if __name__ == '__main__':
main()

121
benchmarks/storage_benchmark_v1/doc/README.md Normal file
View File

@ -0,0 +1,121 @@
# KVCache Storage Benchmark v1
## Overview
The KVCache Storage Benchmark is a tool for evaluating storage performance of KVCache workloads. It simulates real-world cache access patterns using trace replay and measures storage I/O performance with detailed statistics.
## Usage
### Basic Usage
```bash
cd benchmarks/storage_benchmark_v1
python benchmark.py --scenario conversation \
--trace-dir /path/to/Mooncake/FAST25-release/traces \
--storage-dir /path/to/test/drive
```
### Command Line Options
| Option | Default | Description |
|--------|---------|-------------|
| `--trace-dir` | `../../FAST25-release/traces` | Directory containing trace files |
| `--scenario` | `toolagent` | Test scenario: `conversation`, `synthetic`, `toolagent`, or `all` |
| `--storage-dir` | `/tmp/mooncake_bench` | Directory for storage files |
| `--model` | `glm5` | Model preset: `glm5` or `kimi-k2.6` |
| `--page-size-tokens` | `512` | Page size in tokens |
| `--max-requests` | `None` | Maximum number of requests to process |
| `--max-pages` | `2000` | Maximum number of pages (creates modulo mapping if trace is larger) |
| `--fsync-mode` | `none` | When to fsync: `none`, `batch`, `always`, or `end` |
| `--fsync-batch-size` | `100` | Number of writes between fsync in batch mode |
## Output Format
### Progress Output
During execution, each request displays real-time statistics:
```
[ 10/12031] ids= 35 tokens= 18060 | QPS= 2.45 | R= 36 ( 22.01ms, 2435.2MB/s) | W= 963 ( 19.35ms, 2770.1MB/s)
```
Fields:
- `[N/Total]`: Current request progress
- `ids=N`: Number of hash_ids in this request
- `tokens=N`: Total tokens (input + output)
- `QPS=X`: Queries per second (overall)
- `R=N (latency, bandwidth)`: Read count, average latency, bandwidth
- `W=N (latency, bandwidth)`: Write count, average latency, bandwidth
### Final Results
```
================================================================================
[1/1] toolagent_trace.jsonl
================================================================================
[General]
Model: glm5
Requests: 12031
Tokens: 123456789
Total I/O Time: 245.123 s
QPS: 49.07
Hit Rate: 3.25%
[Read Operations]
Count: 390
Data Volume: 20919.62 MB
Total Time: 8.590 s
Bandwidth: 2435.67 MB/s
Latency:
Avg: 22.032 ms
P50: 21.456 ms
P95: 28.912 ms
P99: 35.234 ms
[Write Operations]
Count: 11641
Data Volume: 654321.45 MB
Total Time: 236.533 s
Bandwidth: 2765.89 MB/s
Latency:
Avg: 20.312 ms
P50: 19.876 ms
P95: 25.123 ms
P99: 31.456 ms
[Storage Info]
Max Pages: 2000
Written Pages: 2000
Sync Count: 0
```
## Modulo Mapping
When the trace requires more pages than `--max-pages`, modulo mapping is enabled:
```
physical_page_id = logical_page_id % max_pages
```
This allows simulating large traces (millions of pages) with limited storage (thousands of pages). The first-hit/read-then-write logic is preserved by tracking written logical pages in memory.
**Example**: With `--max-pages 2000`, logical page IDs 0-1999 map directly to physical pages 0-1999. Logical page 2000 maps to physical page 0, logical page 2001 maps to physical page 1, etc.
## Graceful Interruption
Press `Ctrl-C` at any time to stop the benchmark and view partial results. The output will display all statistics collected up to the interruption point, using the same format as final results.
## Troubleshooting
### Insufficient Storage Warning
```
⚠️ Modulo mapping ENABLED (limited storage)
⚠️ Storage insufficient: 14,257,620 pages shortfall (762.34 GB)
```
Increase `--max-pages` to reduce modulo mapping effects.
### Low Bandwidth or High Latency
- Check fsync mode (`--fsync-mode none` for best performance)
- Verify disk performance with `fio` or `dd`
- Verify storage device health

17
benchmarks/storage_benchmark_v1/layout/__init__.py Normal file
View File

@ -0,0 +1,17 @@
"""
KVCache Layout Module
Provides layout interface and implementations for different model architectures.
"""
from .interface import KVLayout, StorageAccess
from .mla import MLALayout, MLA_MODEL_CONFIG, get_model_config, create_layout
__all__ = [
'KVLayout',
'StorageAccess',
'MLALayout',
'MLA_MODEL_CONFIG',
'get_model_config',
'create_layout',
]

55
benchmarks/storage_benchmark_v1/layout/interface.py Normal file
View File

@ -0,0 +1,55 @@
"""
KVCache Layout Interface
Defines the abstract interface for KVCache storage layouts.
Layout layer converts normalized requests into storage access requirements.
"""
from abc import ABC, abstractmethod
from typing import Iterator, Any
from dataclasses import dataclass
@dataclass
class StorageAccess:
"""Storage access requirement
Represents a need to access a page. Whether to READ or WRITE is determined
by the upper layer based on whether the page already exists.
"""
page_id: int # Page ID (hash_id from trace)
offset_in_page: int = 0 # Offset within the page (default: 0)
length: int = None # Number of bytes (default: entire page)
def __repr__(self):
if self.offset_in_page == 0 and self.length is None:
return f"Access(page_id={self.page_id})"
else:
return f"Access(page_id={self.page_id}, offset={self.offset_in_page}, length={self.length})"
class KVLayout(ABC):
"""Abstract interface for KVCache storage layout
Converts normalized KVCache requests into storage access requirements.
Request format:
- hash_ids: List[int] - chunk/page identifiers
- input_length: int - input token count
- output_length: int - output token count
Output:
- Iterator of StorageAccess (page access requirements)
"""
@abstractmethod
def get_operations(self, request: Any) -> Iterator[StorageAccess]:
"""Generate storage access requirements for a request
Args:
request: KVCache request with hash_ids, input_length, output_length
Yields:
StorageAccess: Page access requirements (READ vs WRITE decided by upper layer)
"""
pass

174
benchmarks/storage_benchmark_v1/layout/mla.py Normal file
View File

@ -0,0 +1,174 @@
"""
MLA (Multi-head Latent Attention) KVCache Layout
Implements the KVLayout interface for MLA architecture.
"""
from typing import Iterator, Any
from .interface import KVLayout, StorageAccess
# ============================================================================
# MLA Model Configurations
# ============================================================================
# MLA Model configurations
# Source: https://kvcache.ai/tools/kv-cache-calculator/
# MLA architecture: hash_id -> {layer_0: [pages], layer_1: [pages], ...}
MLA_MODEL_CONFIG = {
# GLM-5: 78 layers, 64 tokens/page
# KV: 78 layers × 64 tokens × (512+64+128) × 2 = 90,112 bytes/page
# Per token: 1,408 bytes
"glm5": {
"name": "glm5",
"num_layers": 78,
"kv_lora_rank": 512,
"qk_rope_head_dim": 64,
"index_head_dim": 128,
"kv_precision_bytes": 2, # BF16
"indexer_precision_bytes": 2, # BF16
},
# Kimi-K2.6: 61 layers, 64 tokens/page
# KV: 61 layers × 64 tokens × (512+64) × 2 = 73,728 bytes/page
# Per token: 1,152 bytes
"kimi-k2.6": {
"name": "kimi-k2.6",
"num_layers": 61,
"kv_lora_rank": 512,
"qk_rope_head_dim": 64,
"index_head_dim": 0, # Kimi doesn't use separate indexer
"kv_precision_bytes": 2, # BF16
"indexer_precision_bytes": 0,
},
}
class MLALayout(KVLayout):
"""MLA (Multi-head Latent Attention) KVCache layout
MLA Architecture:
- Each hash_id corresponds to a 512-token chunk
- Each hash_id maps to ONE complete entry containing all layers
- Entry contains KV + Indexer data for all layers for 512 tokens
- Value size is fixed per hash_id (includes all layers)
Value Size Calculation (per hash_id entry):
per_layer_size = 512 × (kv_lora_rank + qk_rope_head_dim + index_head_dim) × precision_bytes
value_size = per_layer_size × num_layers
For GLM-5 with 512 tokens per entry:
per_layer_size = 512 × (512 + 64 + 128) × 2 = 720,896 bytes
value_size = 720,896 × 78 = 56,229,888 bytes = 53.6 MiB
Key Pattern: hash_id → single entry (all layers included)
Total Keys = len(hash_ids)
Used in: GLM-5, Kimi-K2.6
"""
def __init__(self, num_layers: int, kv_lora_rank: int, qk_rope_head_dim: int,
index_head_dim: int, precision_bytes: int, page_size_tokens: int = 512):
"""Initialize MLA layout
Args:
num_layers: Number of transformer layers
kv_lora_rank: KV LoRA rank dimension
qk_rope_head_dim: QK rope head dimension
index_head_dim: Indexer head dimension
precision_bytes: Precision in bytes (BF16=2, INT8=1, INT4=0.5)
page_size_tokens: Tokens per page (default: 512)
"""
self.num_layers = num_layers
self.kv_lora_rank = kv_lora_rank
self.qk_rope_head_dim = qk_rope_head_dim
self.index_head_dim = index_head_dim
self.precision_bytes = precision_bytes
self.page_size_tokens = page_size_tokens
# Calculate fixed value size per entry (per hash_id)
# Each entry contains KV + Indexer for all layers for page_size_tokens
# Per layer: page_size_tokens × (kv_lora_rank + qk_rope_head_dim + index_head_dim) × precision_bytes
# Total: per_layer_size × num_layers
per_layer_size = page_size_tokens * (kv_lora_rank + qk_rope_head_dim + index_head_dim) * precision_bytes
self.value_size_bytes = per_layer_size * num_layers
# Store page_size for backward compatibility
self.page_size = self.value_size_bytes
def get_operations(self, request: Any) -> Iterator[StorageAccess]:
"""Generate storage access requirements for a request
For MLA architecture:
- Each hash_id corresponds to one complete page (512 tokens, all layers)
- Generate one access requirement per hash_id
Args:
request: KVCache request with hash_ids, input_length, output_length
Yields:
StorageAccess: Page access requirements
"""
for hash_id in request.hash_ids:
yield StorageAccess(
page_id=hash_id,
offset_in_page=0,
length=self.value_size_bytes
)
# ============================================================================
# Utility Functions
# ============================================================================
def get_model_config(model_name: str) -> dict:
"""Get MLA model configuration by name
Args:
model_name: Model identifier (e.g., 'glm5', 'kimi-k2.6')
Returns:
dict: Model configuration
Raises:
KeyError: If model name is not found
"""
if model_name not in MLA_MODEL_CONFIG:
available = ", ".join(MLA_MODEL_CONFIG.keys())
raise KeyError(f"Unknown model: {model_name}. Available: {available}")
return MLA_MODEL_CONFIG[model_name].copy()
def create_layout(model_config: dict, page_size_tokens: int = 64) -> MLALayout:
"""Create an MLALayout instance from model configuration
Args:
model_config: Model configuration dictionary with fields:
- num_layers: Number of transformer layers
- kv_lora_rank: KV LoRA rank dimension
- qk_rope_head_dim: QK rope head dimension
- index_head_dim: Indexer head dimension
- kv_precision_bytes: Precision in bytes (BF16=2, INT8=1)
page_size_tokens: Tokens per page (default 64)
Returns:
MLALayout: Layout instance for MLA architecture
Raises:
ValueError: If model configuration is invalid
"""
required_fields = ['num_layers', 'kv_lora_rank', 'qk_rope_head_dim',
'index_head_dim', 'kv_precision_bytes']
for field in required_fields:
if field not in model_config:
raise ValueError(f"Missing required field: {field}")
return MLALayout(
num_layers=model_config['num_layers'],
kv_lora_rank=model_config['kv_lora_rank'],
qk_rope_head_dim=model_config['qk_rope_head_dim'],
index_head_dim=model_config['index_head_dim'],
precision_bytes=model_config['kv_precision_bytes'],
page_size_tokens=page_size_tokens,
)

13
benchmarks/storage_benchmark_v1/storage/__init__.py Normal file
View File

@ -0,0 +1,13 @@
"""
KVCache Storage Module
Provides storage backend implementations for KVCache systems.
"""
from .interface import Storage
from .disk import DiskHashTable
__all__ = [
'Storage',
'DiskHashTable',
]

328
benchmarks/storage_benchmark_v1/storage/disk.py Normal file
View File

@ -0,0 +1,328 @@
"""
Simple SSD Hash Table Storage
Each key maps to a complete page entry.
"""
import os
import time
from pathlib import Path
from typing import Dict, List, Optional, Any
from .interface import Storage
def calc_percentiles(data):
"""Calculate latency percentiles"""
if not data:
return {'avg_ms': 0, 'p50_ms': 0, 'p95_ms': 0, 'p99_ms': 0}
import statistics
sorted_data = sorted(data)
n = len(sorted_data)
def get_percentile(p):
idx = int(n * p / 100)
if idx >= n: idx = n - 1
return sorted_data[idx]
return {
'avg_ms': statistics.mean(data),
'p50_ms': get_percentile(50),
'p95_ms': get_percentile(95),
'p99_ms': get_percentile(99),
}
class DiskHashTable(Storage):
"""Simple disk-based hash table
File Layout:
Page 0: offset = 0 * page_size
Page 1: offset = 1 * page_size
Page 2: offset = 2 * page_size
...
Page N: offset = N * page_size
When max_pages < actual page_id range, uses modulo mapping:
actual_page_id = page_id % max_pages
This allows simulating large traces with limited storage.
"""
def __init__(self, storage_dir: str, page_size: int,
max_pages: int = 100000,
fsync_mode: str = 'batch', fsync_batch_size: int = 100):
"""Initialize disk hash table
Args:
storage_dir: Storage directory
page_size: Size of each entry (page) in bytes
max_pages: Maximum number of entries (creates circular mapping if trace is larger)
fsync_mode: When to fsync ('batch', 'always', 'end', 'none')
fsync_batch_size: Writes between fsync
"""
self.storage_dir = Path(storage_dir)
self.page_size = page_size
self.max_pages = max_pages
self.max_page_id = max_pages - 1
self.fsync_mode = fsync_mode
self.fsync_batch_size = fsync_batch_size
self.storage_dir.mkdir(parents=True, exist_ok=True)
self.storage_file = self.storage_dir / "data.bin"
self._allocate_file()
self.fd = None
self._buffer = os.urandom(page_size)
self.stats = {
'read_count': 0,
'write_count': 0,
'read_bytes': 0,
'write_bytes': 0,
'read_latencies_ms': [],
'write_latencies_ms': [],
'read_time_s': 0.0,
'write_time_s': 0.0,
'sync_count': 0,
'hit': 0,
'miss': 0,
}
self._pending_syncs = 0
self._written_pages: set = set()
def _allocate_file(self):
"""Pre-allocate disk space efficiently"""
file_size = self.max_pages * self.page_size
if not self.storage_file.exists():
print(f" [Storage] Creating file: {self.storage_file}")
print(f" [Storage] Requested size: {file_size / (1024**3):.2f} GB ({self.max_pages:,} pages × {self.page_size} bytes = {file_size:,} bytes)")
# Use fallocate for efficient preallocation (Linux)
fd = os.open(self.storage_file, os.O_RDWR | os.O_CREAT | os.O_TRUNC, 0o644)
try:
# Try fallocate first (Linux specific, much faster)
try:
import fcntl
fcntl.fallocate(fd, 0, file_size)
method = "fallocate"
except (ImportError, AttributeError, OSError):
# Fallback to seek+write method
os.lseek(fd, file_size - 1, os.SEEK_SET)
os.write(fd, b'\0')
os.fsync(fd)
method = "seek+write"
finally:
os.close(fd)
actual_size = self.storage_file.stat().st_size if self.storage_file.exists() else 0
print(f" [Storage] Pre-allocated {actual_size / (1024**3):.2f} GB using {method}")
else:
actual_size = self.storage_file.stat().st_size
actual_pages = actual_size // self.page_size
print(f" [Storage] Reusing existing file: {self.storage_file}")
print(f" [Storage] Current file size: {actual_size / (1024**3):.2f} GB ({actual_pages:,} pages × {self.page_size} bytes = {actual_size:,} bytes)")
def _get_fd(self):
if self.fd is None:
print(f" [Storage] Opening file: {self.storage_file}")
self.fd = os.open(self.storage_file, os.O_RDWR | os.O_CREAT, 0o644)
if self.storage_file.exists():
actual_size = self.storage_file.stat().st_size
actual_pages = actual_size // self.page_size
print(f" [Storage] File size: {actual_size / (1024**3):.2f} GB ({actual_pages:,} pages × {self.page_size} bytes = {actual_size:,} bytes)")
return self.fd
# ========================================================================
# Core operations
# ========================================================================
def _map_page_id(self, page_id: int) -> int:
"""Map logical page_id to physical page_id using modulo
This allows simulating large traces with limited storage space.
Args:
page_id: Logical page_id (from trace)
Returns:
Physical page_id in storage (0 to max_pages-1)
"""
return page_id % self.max_pages
def read(self, page_id: int, offset_in_page: int = 0, length: int = None) -> float:
"""Read entry from disk
Args:
page_id: Logical page ID (hash_id from trace)
offset_in_page: Offset within the page (default: 0)
length: Number of bytes to read (default: entire page)
Returns:
Read latency in ms
"""
if length is None:
length = self.page_size - offset_in_page
# Validate parameters
if offset_in_page < 0 or offset_in_page >= self.page_size:
raise ValueError(f"offset_in_page {offset_in_page} out of range [0, {self.page_size})")
if length <= 0 or offset_in_page + length > self.page_size:
raise ValueError(f"length {length} invalid with offset_in_page {offset_in_page} (page_size={self.page_size})")
# Map to physical page_id and calculate offset
physical_page_id = self._map_page_id(page_id)
offset = physical_page_id * self.page_size + offset_in_page
start = time.perf_counter()
try:
fd = self._get_fd()
os.pread(fd, length, offset)
latency = (time.perf_counter() - start) * 1000.0
self.stats['read_count'] += 1
self.stats['read_bytes'] += length
self.stats['read_latencies_ms'].append(latency)
self.stats['read_time_s'] += latency / 1000.0
self.stats['hit'] += 1
return latency
except OSError as e:
print(f"Read error (page_id={page_id}, physical_page_id={physical_page_id}): {e}")
return 0.0
def write(self, page_id: int, offset_in_page: int = 0, length: int = None) -> float:
"""Write entry to disk
Args:
page_id: Logical page ID (hash_id from trace)
offset_in_page: Offset within the page (default: 0)
length: Number of bytes to write (default: entire page)
Returns:
Write latency in ms
"""
if length is None:
length = self.page_size - offset_in_page
# Validate parameters
if offset_in_page < 0 or offset_in_page >= self.page_size:
raise ValueError(f"offset_in_page {offset_in_page} out of range [0, {self.page_size})")
if length <= 0 or offset_in_page + length > self.page_size:
raise ValueError(f"length {length} invalid with offset_in_page {offset_in_page} (page_size={self.page_size})")
# Map to physical page_id and calculate offset
physical_page_id = self._map_page_id(page_id)
offset = physical_page_id * self.page_size + offset_in_page
start = time.perf_counter()
try:
fd = self._get_fd()
# Use corresponding portion of buffer
os.pwrite(fd, self._buffer[:length], offset)
write_done = time.perf_counter()
# Fsync
if self.fsync_mode == 'always':
os.fsync(fd)
self.stats['sync_count'] += 1
self._pending_syncs = 0
latency = (time.perf_counter() - start) * 1000.0
elif self.fsync_mode == 'batch':
self._pending_syncs += 1
if self._pending_syncs >= self.fsync_batch_size:
os.fsync(fd)
self.stats['sync_count'] += 1
self._pending_syncs = 0
latency = (write_done - start) * 1000.0
else:
latency = (write_done - start) * 1000.0
self._written_pages.add(page_id)
self.stats['write_count'] += 1
self.stats['write_bytes'] += length
self.stats['write_latencies_ms'].append(latency)
self.stats['write_time_s'] += latency / 1000.0
self.stats['miss'] += 1
return latency
except OSError as e:
print(f"Write error (page_id={page_id}, offset_in_page={offset_in_page}, length={length}): {e}")
return 0.0
# ========================================================================
# Storage interface methods
# ========================================================================
def exists(self, page_id: int) -> bool:
"""Check if logical page has been written
Args:
page_id: Logical page ID (hash_id from trace)
Returns:
True if this logical page has been written before
"""
return page_id in self._written_pages
def delete(self, page_id: int) -> bool:
"""Delete page (no-op in direct mapping)
Args:
page_id: Page ID to delete
Returns:
True (always succeeds in direct mapping)
"""
# No-op since we don't track which pages have been written
return True
def get_stats(self) -> Dict[str, Any]:
"""Get statistics"""
import statistics
def calc_stats(latencies):
if not latencies:
return {'avg_ms': 0, 'p50_ms': 0, 'p95_ms': 0, 'p99_ms': 0}
return {
'avg_ms': statistics.mean(latencies),
**calc_percentiles(latencies),
}
return {
'read': {
'count': self.stats['read_count'],
'mb': self.stats['read_bytes'] / 1024 / 1024,
'time_s': self.stats['read_time_s'],
**calc_stats(self.stats['read_latencies_ms'])
},
'write': {
'count': self.stats['write_count'],
'mb': self.stats['write_bytes'] / 1024 / 1024,
'time_s': self.stats['write_time_s'],
**calc_stats(self.stats['write_latencies_ms'])
},
'sync_count': self.stats['sync_count'],
'max_pages': self.max_pages,
'written_pages': len(self._written_pages),
'page_hits': self.stats['hit'],
'page_misses': self.stats['miss'],
}
# ========================================================================
# Resource management
# ========================================================================
def close(self, force_sync: bool = True):
"""Close file"""
if force_sync and self.fsync_mode in ['end', 'batch']:
if self.fd is not None:
try:
os.fsync(self.fd)
self.stats['sync_count'] += 1
except OSError:
pass
if self.fd is not None:
try:
os.close(self.fd)
except OSError:
pass
self.fd = None
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.close()
return False

93
benchmarks/storage_benchmark_v1/storage/interface.py Normal file
View File

@ -0,0 +1,93 @@
"""
KVCache Storage Interface
Simplified storage interface for KVCache benchmark.
Key = page_id (int), Value = fixed-size bytes.
"""
from abc import ABC, abstractmethod
from typing import Dict, Any
class Storage(ABC):
"""Abstract base class for KVCache storage
Simplified design:
- Key: page_id (int, hash_id from trace)
- Value: fixed-size bytes (page_size)
- Direct mapping: page_id -> offset -> payload
Core operations: read, write, exists, delete.
"""
@abstractmethod
def read(self, page_id: int, offset_in_page: int = 0, length: int = None) -> float:
"""Read page from disk
Args:
page_id: Page ID (hash_id from trace)
offset_in_page: Offset within the page (default: 0)
length: Number of bytes to read (default: entire page)
Returns:
Read latency in milliseconds
"""
pass
@abstractmethod
def write(self, page_id: int, offset_in_page: int = 0, length: int = None) -> float:
"""Write page to disk
Args:
page_id: Page ID (hash_id from trace)
offset_in_page: Offset within the page (default: 0)
length: Number of bytes to write (default: entire page)
Returns:
Write latency in milliseconds
"""
pass
@abstractmethod
def exists(self, page_id: int) -> bool:
"""Check if page exists
Args:
page_id: Page ID (hash_id from trace)
Returns:
True if page_id is within valid range
"""
pass
@abstractmethod
def delete(self, page_id: int) -> bool:
"""Delete page (no-op in direct mapping)
Args:
page_id: Page ID to delete
Returns:
True (always succeeds in direct mapping)
"""
pass
@abstractmethod
def get_stats(self) -> Dict[str, Any]:
"""Get storage statistics
Returns:
Dictionary containing storage statistics
"""
pass
def close(self):
"""Close the storage and release resources"""
pass
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.close()
return False

258
dependencies.sh
View File

@ -24,6 +24,7 @@ NC="\033[0m" # No Color
REPO_ROOT=`pwd`
GITHUB_PROXY=${GITHUB_PROXY:-"https://github.com"}
GOVER=1.25.9
OS_RELEASE_FILE=${OS_RELEASE_FILE:-/etc/os-release}
# Function to print section headers
print_section() {
@ -48,23 +49,57 @@ check_success() {
fi
}
read_os_release_value() {
local key="$1"
awk -F= -v key="$key" '
$1 == key {
value = $0
sub(/^[^=]*=/, "", value)
gsub(/^"|"$/, "", value)
print value
exit
}
' "$OS_RELEASE_FILE"
}
# Function to detect OS
detect_os() {
if [ -f "$OS_RELEASE_FILE" ]; then
ID=$(read_os_release_value ID)
VERSION_ID=$(read_os_release_value VERSION_ID)
OS=$(echo "$ID" | tr '[:upper:]' '[:lower:]')
OS_VERSION=$VERSION_ID
elif [ -f /etc/redhat-release ]; then
OS="centos"
else
print_error "Cannot detect OS. Supported OS: Ubuntu, Debian, CentOS, RHEL, Rocky, AlmaLinux, EulerOS, and openEuler."
fi
echo -e "${GREEN}Detected OS: $OS ${OS_VERSION:-unknown}${NC}"
}
if [ $(id -u) -ne 0 ]; then
print_error "Require root permission, try sudo ./dependencies.sh"
fi
# Parse command line arguments
SKIP_CONFIRM=false
INSTALL_SPDK=false
for arg in "$@"; do
case $arg in
-y|--yes)
SKIP_CONFIRM=true
;;
--with-spdk)
INSTALL_SPDK=true
;;
-h|--help)
echo -e "${YELLOW}Mooncake Dependencies Installer${NC}"
echo -e "Usage: ./dependencies.sh [OPTIONS]"
echo -e "\nOptions:"
echo -e " -y, --yes Skip confirmation and install all dependencies"
echo -e " -h, --help Show this help message and exit"
echo -e " -y, --yes Skip confirmation and install all dependencies"
echo -e " --with-spdk Install SPDK for NVMe-oF support"
echo -e " -h, --help Show this help message and exit"
exit 0
;;
esac
@ -77,6 +112,9 @@ echo -e "The following components will be installed:"
echo -e " - System packages (build tools, libraries)"
echo -e " - Git submodules (including pybind11 and yalantinglibs)"
echo -e " - Go $GOVER"
if [ "$INSTALL_SPDK" = true ]; then
echo -e " - SPDK (for NVMe-oF support)"
fi
echo
# Ask for confirmation unless -y flag is used
@ -89,51 +127,100 @@ if [ "$SKIP_CONFIRM" = false ]; then
fi
fi
# Detect OS
detect_os
# Update package lists
print_section "Updating package lists"
apt-get update
check_success "Failed to update package lists"
if [ "$OS" = "ubuntu" ] || [ "$OS" = "debian" ]; then
apt-get update
check_success "Failed to update package lists"
elif [ "$OS" = "centos" ] || [ "$OS" = "rhel" ] || [ "$OS" = "rocky" ] || [ "$OS" = "almalinux" ] || [ "$OS" = "euleros" ] || [ "$OS" = "openeuler" ]; then
yum install -y dnf-plugins-core epel-release || true
yum config-manager --set-enabled powertools || yum config-manager --set-enabled crb || true
yum clean all
yum makecache
check_success "Failed to update package lists"
else
print_error "Unsupported OS: $OS"
fi
# Install system packages
print_section "Installing system packages"
echo -e "${YELLOW}This may take a few minutes...${NC}"
SYSTEM_PACKAGES="build-essential \
cmake \
ninja-build \
git \
wget \
unzip \
libibverbs-dev \
libgoogle-glog-dev \
libgtest-dev \
libjsoncpp-dev \
libunwind-dev \
libnuma-dev \
libpython3-dev \
libboost-all-dev \
libssl-dev \
libgrpc-dev \
libgrpc++-dev \
libprotobuf-dev \
libyaml-cpp-dev \
protobuf-compiler-grpc \
libcurl4-openssl-dev \
libhiredis-dev \
liburing-dev \
libjemalloc-dev \
libmsgpack-dev \
libzstd-dev \
libasio-dev \
libxxhash-dev \
pkg-config \
patchelf \
libc6-dev \
libc-bin"
if [ "$OS" = "ubuntu" ] || [ "$OS" = "debian" ]; then
SYSTEM_PACKAGES="build-essential \
cmake \
ninja-build \
git \
wget \
unzip \
libibverbs-dev \
libgoogle-glog-dev \
libgtest-dev \
libjsoncpp-dev \
libunwind-dev \
libnuma-dev \
libpython3-dev \
libboost-all-dev \
libssl-dev \
libgrpc-dev \
libgrpc++-dev \
libprotobuf-dev \
libyaml-cpp-dev \
protobuf-compiler-grpc \
libcurl4-openssl-dev \
libhiredis-dev \
liburing-dev \
libjemalloc-dev \
libmsgpack-dev \
libzstd-dev \
libasio-dev \
libxxhash-dev \
pkg-config \
patchelf \
libc6-dev \
libc-bin"
apt-get install -y $SYSTEM_PACKAGES
check_success "Failed to install system packages"
elif [ "$OS" = "centos" ] || [ "$OS" = "rhel" ] || [ "$OS" = "rocky" ] || [ "$OS" = "almalinux" ] || [ "$OS" = "euleros" ] || [ "$OS" = "openeuler" ]; then
SYSTEM_PACKAGES="@development \
cmake \
git \
wget \
rdma-core-devel \
glog-devel \
gflags-devel \
gtest-devel \
jsoncpp-devel \
libunwind-devel \
numactl-devel \
python3-devel \
boost1.78-devel \
openssl-devel \
protobuf-devel \
yaml-cpp-devel \
libcurl-devel \
hiredis-devel \
liburing-devel \
jemalloc-devel \
msgpack-devel \
libzstd-devel \
pkgconf-pkg-config \
elfutils-libelf-devel \
patchelf \
xxhash-devel \
libbsd-devel"
yum install -y $SYSTEM_PACKAGES
check_success "Failed to install system packages"
else
print_error "Unsupported OS: $OS"
fi
apt-get install -y $SYSTEM_PACKAGES
check_success "Failed to install system packages"
print_success "System packages installed successfully"
# Initialize and update git submodules
@ -185,15 +272,16 @@ cd "${REPO_ROOT}"
print_section "Verifying essential build tools"
# Verify getconf and ldd (required for glibc version detection in build_wheel.sh)
# Both are provided by libc-bin, which is included in SYSTEM_PACKAGES
if ! command -v getconf >/dev/null 2>&1; then
print_error "getconf not found after installing system packages. This should not happen."
if [ "$OS" = "ubuntu" ] || [ "$OS" = "debian" ]; then
if ! command -v getconf >/dev/null 2>&1; then
print_error "getconf not found after installing system packages. This should not happen."
fi
if ! command -v ldd >/dev/null 2>&1; then
print_error "ldd not found after installing system packages. This should not happen."
fi
print_success "getconf found: $(getconf --version 2>&1 | head -1)"
print_success "ldd found: $(ldd --version 2>&1 | head -1)"
fi
if ! command -v ldd >/dev/null 2>&1; then
print_error "ldd not found after installing system packages. This should not happen."
fi
print_success "getconf found: $(getconf --version 2>&1 | head -1)"
print_success "ldd found: $(ldd --version 2>&1 | head -1)"
print_section "Installing Go $GOVER"
@ -224,8 +312,6 @@ install_go() {
echo "Downloading Go $GOVER from ${url}..."
if wget -q --show-progress --timeout=30 --tries=2 -O "${GO_TARBALL}" "${url}"; then
DOWNLOAD_SUCCESS=true
# If the official source (go.dev) failed and we fell back to a CN mirror,
# it likely means the network has restricted access to international sites.
if [[ "$url" != "https://go.dev/dl/${GO_TARBALL}" ]]; then
USED_CN_MIRROR=true
fi
@ -241,19 +327,16 @@ install_go() {
print_error "Failed to download Go $GOVER from all mirrors"
fi
# Install Go
echo "Installing Go $GOVER..."
tar -C /usr/local -xzf "${GO_TARBALL}"
check_success "Failed to install Go $GOVER"
# Clean up downloaded file
rm -f "${GO_TARBALL}"
check_success "Failed to clean up Go installation file"
print_success "Go $GOVER installed successfully"
}
# Check if Go is already installed
if command -v go &> /dev/null; then
GO_VERSION=$(go version | awk '{print $3}')
if [[ "$GO_VERSION" == "go$GOVER" ]]; then
@ -273,8 +356,7 @@ if ! grep -q "export PATH=\$PATH:/usr/local/go/bin" ~/.bashrc; then
echo -e "${YELLOW}Please run 'source ~/.bashrc' or start a new terminal to use Go${NC}"
fi
# Set GOPROXY only if Go download fell back to a CN mirror, indicating restricted
# network access to international sites. Skip if user already configured GOPROXY.
# Set GOPROXY only if Go download fell back to a CN mirror
if [ "$USED_CN_MIRROR" = true ] && [ -z "$GOPROXY" ]; then
export GOPROXY=https://goproxy.cn,https://goproxy.io,direct
echo -e "${YELLOW}Detected restricted network (Go was downloaded from a CN mirror).${NC}"
@ -287,6 +369,69 @@ elif [ -n "$GOPROXY" ]; then
echo -e "${GREEN}GOPROXY already set to: ${GOPROXY}${NC}"
fi
# Install SPDK if requested
if [ "$INSTALL_SPDK" = true ]; then
print_section "Installing SPDK"
cd "${REPO_ROOT}/extern"
check_success "Failed to change to extern directory"
# Remove existing SPDK if present
if [ -d "spdk" ]; then
echo -e "${YELLOW}SPDK directory already exists. Removing for fresh install...${NC}"
rm -rf spdk
check_success "Failed to remove existing SPDK directory"
fi
# Clone SPDK
echo "Cloning SPDK from ${GITHUB_PROXY}/spdk/spdk.git..."
git clone ${GITHUB_PROXY}/spdk/spdk.git
check_success "Failed to clone SPDK"
cd spdk
check_success "Failed to change to SPDK directory"
# Checkout specific version
echo "Checking out SPDK version v23.01.1..."
git checkout v23.01.1
check_success "Failed to checkout SPDK version v23.01.1"
# Initialize submodules
echo "Initializing SPDK submodules..."
git submodule update --init
check_success "Failed to initialize SPDK submodules"
# Install SPDK dependencies
echo "Installing SPDK dependencies..."
./scripts/pkgdep.sh
check_success "Failed to install SPDK dependencies"
# Configure SPDK with RDMA support
echo "Configuring SPDK with RDMA support..."
./configure --with-rdma
check_success "Failed to configure SPDK"
# Build SPDK
echo "Building SPDK (using $(nproc) cores)..."
make -j$(nproc)
check_success "Failed to build SPDK"
# Install SPDK
echo "Installing SPDK..."
make install
check_success "Failed to install SPDK"
# Copy DPDK libraries to system library path
if ls dpdk/build/lib/*.a >/dev/null 2>&1; then
echo "Copying DPDK libraries to /usr/local/lib..."
cp dpdk/build/lib/*.a /usr/local/lib/
check_success "Failed to copy DPDK libraries"
fi
print_success "SPDK installed successfully"
cd "${REPO_ROOT}"
fi
# Return to the repository root
cd "${REPO_ROOT}"
@ -298,6 +443,13 @@ echo -e " ${GREEN}✓${NC} System packages"
echo -e " ${GREEN}✓${NC} yalantinglibs"
echo -e " ${GREEN}✓${NC} Git submodules"
echo -e " ${GREEN}✓${NC} Go $GOVER"
if [ "$INSTALL_SPDK" = true ]; then
echo -e " ${GREEN}✓${NC} SPDK (v23.01.1)"
fi
echo
echo -e "You can now build and run Mooncake."
echo -e "${YELLOW}Note: You may need to restart your terminal or run 'source ~/.bashrc' to use Go.${NC}"
if [ "$INSTALL_SPDK" = true ]; then
echo -e "${YELLOW}Note: SPDK requires hugepages and RDMA configuration. Please refer to SPDK documentation for setup.${NC}"
fi

1
docker/mooncake.Dockerfile
View File

@ -116,6 +116,7 @@ RUN apt-get update && \
# Copy wheels produced in builder stage and install them via pip
COPY --from=builder /workspace/mooncake-wheel/dist /tmp/mooncake-wheel
COPY --chmod=755 scripts/check_hicache_hugepage_requirements.py /usr/local/bin/mooncake-hicache-sizing
RUN python${PYTHON_VERSION} -m pip install --no-cache-dir /tmp/mooncake-wheel/*.whl && rm -rf /tmp/mooncake-wheel /root/.cache/pip
CMD ["/bin/bash"]

16
docs/source/api-reference/cpp/index.md Normal file
View File

@ -0,0 +1,16 @@
# C/C++ API Reference
| Module | Description |
|--------|-------------|
| [Transfer Engine C++ API](../../design/transfer-engine/cpp-api) | `TransferEngine` class — memory registration, batch transfer, segment management, RDMA transport |
| [TENT C++ API](../../design/tent/cpp-api) | `mooncake::tent::TransferEngine` — next-gen transfer engine with automatic transport selection and fault tolerance |
| [Mooncake Store Client C++ API](mooncake-store) | `Client` class — `Put`/`Get`/`Remove`/`Replicate` operations, `BufferAllocatorBase` interface |
:::{toctree}
:maxdepth: 1
:hidden:
../../design/transfer-engine/cpp-api
../../design/tent/cpp-api
mooncake-store
:::

178
docs/source/api-reference/cpp/mooncake-store.md Normal file
View File

@ -0,0 +1,178 @@
# Mooncake Store C++ API Reference
## Client C++ API
### Constructor and Initialization `Init`
```C++
ErrorCode Init(const std::string& local_hostname,
const std::string& metadata_connstring,
const std::string& protocol,
void** protocol_args,
const std::string& master_server_entry);
```
Initializes the Mooncake Store client. The parameters are as follows:
- `local_hostname`: The `IP:Port` of the local machine or an accessible domain name (default value used if port is not included)
- `metadata_connstring`: The address of the metadata service (e.g., etcd/Redis) required for Transfer Engine initialization
- `protocol`: The protocol supported by the Transfer Engine, including RDMA and TCP
- `protocol_args`: Protocol parameters required by the Transfer Engine
- `master_server_entry`: The address information of the Master (`IP:Port` for default mode and `etcd://IP:Port;IP:Port;...;IP:Port` for high availability mode)
### Get
```C++
tl::expected<void, ErrorCode> Get(const std::string& object_key,
std::vector<Slice>& slices);
```
`Get` retrieves the value of `object_key` into the provided `slices`. The returned data is guaranteed to be complete and correct. Each slice must reference local DRAM/VRAM memory that has been pre-registered with `registerLocalMemory(addr, len)` (not the global segments that contribute to the distributed memory pool). When persistence is enabled and the requested data is not found in the distributed memory pool, `Get` will fall back to loading the data from SSD.
### Put
```C++
tl::expected<void, ErrorCode> Put(const ObjectKey& key,
std::vector<Slice>& slices,
const ReplicateConfig& config);
```
`Put` stores the value associated with `key` in the distributed memory pool. The `config` parameter allows specifying the required number of replicas as well as the preferred segment for storing the value. When persistence is enabled, `Put` also asynchronously triggers a persistence operation to SSD.
**Replication Guarantees and Best Effort Behavior:**
- Each slice of an object is guaranteed to be replicated to different segments, ensuring distribution across separate storage nodes
- Different slices from different objects may be placed in the same segment
- Replication operates on a best-effort basis: if insufficient space is available for all requested replicas, the object will still be written with as many replicas as possible
The data structure details of `ReplicateConfig` are as follows:
```C++
struct ReplicateConfig {
size_t replica_num{1}; // Total number of replicas for the object
bool with_soft_pin{false}; // Whether to enable soft pin mechanism for this object
bool with_hard_pin{false}; // Whether to enable hard pin (never evicted)
std::string preferred_segment{}; // Preferred segment for allocation
};
```
### Upsert
```C++
tl::expected<void, ErrorCode> Upsert(const ObjectKey& key,
std::vector<Slice>& slices,
const ReplicateConfig& config);
std::vector<tl::expected<void, ErrorCode>> BatchUpsert(
const std::vector<ObjectKey>& keys,
std::vector<std::vector<Slice>>& batched_slices,
const ReplicateConfig& config);
```
`Upsert` inserts `key` if it does not exist and updates the existing object if
it does. It uses the same replication configuration model as `Put`, while
allowing the store to reuse existing placement for in-place updates when the
current layout permits it. `BatchUpsert` performs the same operation for
multiple keys using a shared replication configuration.
### Remove
```C++
tl::expected<void, ErrorCode> Remove(const ObjectKey& key);
```
Used to delete the object corresponding to the specified key. This interface marks all data replicas associated with the key in the storage engine as deleted, without needing to communicate with the corresponding storage node (Client).
### CreateCopyTask
```C++
tl::expected<UUID, ErrorCode> CreateCopyTask(
const std::string& key,
const std::vector<std::string>& targets);
```
`CreateCopyTask` creates an asynchronous copy task that will be executed by the client's task execution system. This is useful when you want to submit multiple copy operations without waiting for each one to complete. The task is submitted to the master service, assigned a unique task ID, and executed asynchronously by an available client. The task status can be queried using `QueryTask`.
**Task Execution and Result Reporting:**
1. **Task Assignment**: The master service assigns the task to an available client during the client's periodic ping operation
2. **Task Execution**: The assigned client executes the copy operation asynchronously in a background thread pool
3. **Result Reporting**: Upon completion (success or failure), the client automatically reports the result to the master service via `MarkTaskToComplete`:
- On success: `status = SUCCESS`, `message = "Task completed successfully"`
- On failure: `status = FAILED`, `message = <error description>`
4. **Status Query**: You can query the task status at any time using `QueryTask` to monitor progress
### CreateMoveTask
```C++
tl::expected<UUID, ErrorCode> CreateMoveTask(
const std::string& key,
const std::string& source,
const std::string& target);
```
`CreateMoveTask` creates an asynchronous move task that will be executed by the client's task execution system. This is useful when you want to submit multiple move operations without waiting for each one to complete. The task is submitted to the master service, assigned a unique task ID, and executed asynchronously by an available client. The task status can be queried using `QueryTask`.
**Task Execution and Result Reporting:**
1. **Task Assignment**: The master service assigns the task to an available client during the client's periodic ping operation
2. **Task Execution**: The assigned client executes the move operation asynchronously in a background thread pool
3. **Result Reporting**: Upon completion (success or failure), the client automatically reports the result to the master service via `MarkTaskToComplete`:
- On success: `status = SUCCESS`, `message = "Task completed successfully"`
- On failure: `status = FAILED`, `message = <error description>`
4. **Status Query**: You can query the task status at any time using `QueryTask` to monitor progress
### QueryTask
```C++
tl::expected<QueryTaskResponse, ErrorCode> QueryTask(const UUID& task_id);
```
`QueryTask` queries the status of an asynchronous task (copy or move). This allows you to monitor the progress of task-based operations. The response includes task status, type, creation time, last update time, assigned client, and status message.
The data structure details of `QueryTaskResponse` are as follows:
```C++
struct QueryTaskResponse {
UUID id; // Task UUID
TaskType type; // Task type (REPLICA_COPY or REPLICA_MOVE)
TaskStatus status; // Task status (PENDING, PROCESSING, SUCCESS, or FAILED)
int64_t created_at_ms_epoch; // Task creation timestamp in milliseconds
int64_t last_updated_at_ms_epoch; // Last update timestamp in milliseconds
UUID assigned_client; // UUID of the client assigned to execute the task
std::string message; // Status message or error description
};
```
### BatchQueryIp
```C++
tl::expected<std::unordered_map<UUID, std::vector<std::string>, boost::hash<UUID>>, ErrorCode>
BatchQueryIp(const std::vector<UUID>& client_ids);
```
Used to batch query the IP addresses for multiple client IDs. For each client ID in the input list, this interface retrieves the unique IP addresses from all segments mounted by that client. The operation is performed on the Master Service and returns a map from client ID to their IP address lists. Only client IDs that have successfully mounted segments are included in the result map. This is useful for discovering the network locations of storage nodes in the cluster.
### BatchReplicaClear
```C++
tl::expected<std::vector<std::string>, ErrorCode>
BatchReplicaClear(const std::vector<std::string>& object_keys,
const UUID& client_id,
const std::string& segment_name);
```
Used to batch clear replicas for multiple object keys belonging to a specific client ID. This interface allows clearing replicas either on a specific segment or across all segments. If segment_name is empty, all replicas of the specified objects are cleared (the objects are deleted entirely). If segment_name is provided, only replicas located on that specific segment are cleared. The operation is performed on the Master Service and returns a list of object keys that were successfully cleared. Only objects that belong to the specified `client_id`, have expired leases, and meet the clearing criteria are processed. This is useful for managing storage resources and cleaning up data on specific storage nodes.
### QueryByRegex
```C++
tl::expected<std::unordered_map<std::string, std::vector<Replica::Descriptor>>, ErrorCode>
QueryByRegex(const std::string& str);
```
Used to query the replica information for all objects whose keys match the given regular expression. This is useful for batch operations or for retrieving a group of related objects. The operation is performed on the Master and returns a map of keys to their replica lists.
### RemoveByRegex
```C++
tl::expected<long, ErrorCode> RemoveByRegex(const ObjectKey& str);
```
Used to delete all objects from the store whose keys match the specified regular expression. This provides a powerful way to perform bulk deletions. The command returns the number of objects that were successfully removed.

14
docs/source/api-reference/http/index.md Normal file
View File

@ -0,0 +1,14 @@
# HTTP
| Module | Description |
|--------|-------------|
| [HTTP Service](../../http-api-reference/http-service) | RESTful endpoints for cluster management, metrics, and data inspection |
| [Conductor Indexer API](../../design/conductor/indexer-api-design) | Cache-aware routing: service registration, KV event subscription, prefix cache-hit query APIs |
:::{toctree}
:maxdepth: 1
:hidden:
../../http-api-reference/http-service
../../design/conductor/indexer-api-design
:::

18
docs/source/api-reference/python/index.md Normal file
View File

@ -0,0 +1,18 @@
# Python
| Module | Description |
|--------|-------------|
| [Mooncake Store](../../python-api-reference/mooncake-store) | Distributed KV cache storage client — `put`/`get`/`remove`/`replicate` operations |
| [DataProto Structured Object Transfer](../../python-api-reference/dataproto-structured-object-transfer) | Structured-object helpers for storing and retrieving DataProto-like payloads |
| [Transfer Engine](../../python-api-reference/transfer-engine) | High-performance RDMA/TCP data transfer between nodes |
| [EP Backend](../../python-api-reference/ep-backend) | Expert-parallel backend for large MoE model deployment |
:::{toctree}
:maxdepth: 1
:hidden:
../../python-api-reference/mooncake-store
../../python-api-reference/dataproto-structured-object-transfer
../../python-api-reference/transfer-engine
../../python-api-reference/ep-backend
:::

15
docs/source/api-reference/rust/index.md Normal file
View File

@ -0,0 +1,15 @@
# Rust
| Module | Description |
|--------|-------------|
| [Mooncake Store](./mooncake-store) | Safe Rust bindings for the distributed KV-cache store (`mooncake_store`) |
| [Transfer Engine](./transfer-engine) | Rust bindings for Transfer Engine (FFI wrapper used by the Rust example crate) |
```{toctree}
:maxdepth: 1
:hidden:
mooncake-store
transfer-engine
```

223
docs/source/api-reference/rust/mooncake-store.md Normal file
View File

@ -0,0 +1,223 @@
# Mooncake Store Rust API
This page documents the Rust crate `mooncake_store` (located at `mooncake-store/rust`).
It is a **safe wrapper** around the Mooncake Store C API (`store_c.h`).
For deployment and service prerequisites, also see:
- Mooncake Store deployment guide: `deployment/mooncake-store-deployment-guide`
- Error code reference: `troubleshooting/error-code`
## Build & runtime prerequisites
The Rust crate links against the C++ Mooncake build outputs.
- **Build**:
- Build Mooncake with Store + Rust enabled via CMake: `-DWITH_STORE=ON -DWITH_STORE_RUST=ON`
- Or build with Cargo after exporting the CMake build directory / include paths (see `mooncake-store/rust/README.md`).
- **Runtime**:
- Dynamic linker must find Mooncake shared libraries (typically via `LD_LIBRARY_PATH` pointing at the CMake build outputs).
- The store client requires:
- a **metadata server** (HTTP metadata or etcd, depending on your deployment)
- `mooncake_master`
## Quick start (copy-paste)
```rust
use mooncake_store::MooncakeStore;
fn main() -> Result<(), mooncake_store::StoreError> {
let store = MooncakeStore::new()?;
store.setup(
"127.0.0.1",
"http://127.0.0.1:8080/metadata",
512 << 20, // global_segment_size
128 << 20, // local_buffer_size
"tcp",
"",
"127.0.0.1:50051",
)?;
store.put("hello", b"world", None)?;
let value = store.get("hello")?;
assert_eq!(value, b"world");
store.remove("hello", false)?;
Ok(())
}
```
## API reference
### `MooncakeStore`
#### `new() -> Result<MooncakeStore, StoreError>`
Allocate a new store handle (uninitialised). You must call `setup()` before any data operations.
#### `setup(...) -> Result<(), StoreError>`
Initialise the store client and establish connections.
Parameters:
- `local_hostname`: IP/hostname for this node.
- `metadata_server`: metadata URI, for example:
- HTTP: `"http://127.0.0.1:8080/metadata"`
- etcd: `"etcd://127.0.0.1:2379"`
- `global_segment_size`: per-segment size in bytes.
- `local_buffer_size`: local staging buffer size in bytes.
- `protocol`: transport protocol string (for example `"tcp"` / `"rdma"`).
- `device_name`: device selector; empty string means auto-select (when supported by the backend).
- `master_server_addr`: `mooncake_master` address, e.g. `"127.0.0.1:50051"`.
Returns `Ok(())` on success, otherwise `StoreError::OperationFailed(code)`.
#### `health_check() -> Result<(), StoreError>`
Connectivity health check. Returns `Ok(())` when the backend is reachable.
#### `put(key, value, config) -> Result<(), StoreError>`
Store `value` under `key`. This is a **copying** API: `value` is copied into store-managed buffers.
- `config`: optional replication settings (`ReplicateConfig`).
#### `get(key) -> Result<Vec<u8>, StoreError>`
Retrieve the full value for `key` into a newly allocated `Vec<u8>`.
Notes:
- Internally calls `get_size()` to allocate an exact-sized buffer, then `get_into()` to fill it.
- A missing key or backend failure can surface as `OperationFailed(...)` because the C API does not provide a distinct NotFound code in all paths.
#### `unsafe get_into(key, buffer, size) -> Result<i64, StoreError>`
Retrieve the value for `key` into a caller-provided buffer.
- **Returns**: number of bytes written on success.
- **Safety**: `buffer` must point to at least `size` bytes of writable valid memory.
#### `is_exist(key) -> Result<bool, StoreError>`
Existence check.
- `Ok(true)` if exists, `Ok(false)` if missing.
- Any other return code becomes `StoreError::OperationFailed(code)`.
#### `get_size(key) -> Result<i64, StoreError>`
Get the stored value size in bytes.
Important limitation:
- The underlying C API uses a single negative return code for multiple error conditions, so Rust surfaces errors as `OperationFailed(raw_code)` without distinguishing NotFound.
#### `get_hostname() -> Result<String, StoreError>`
Returns the hostname (and potentially port) that the store client is registered under.
#### `remove(key, force) -> Result<(), StoreError>`
Remove a key.
- If `force = true`, the key is removed even if another client is reading it.
#### `remove_by_regex(pattern, force) -> Result<i64, StoreError>`
Remove keys matching a regex pattern. Returns number of removed keys.
#### `remove_all(force) -> Result<i64, StoreError>`
Remove **all** keys. Returns number of removed keys.
### Zero-copy APIs (advanced)
The Rust wrapper exposes zero-copy APIs that map directly to the underlying RDMA-capable C++ store.
#### `unsafe register_buffer(buffer, size) -> Result<(), StoreError>`
Register a memory region for zero-copy operations.
- **Safety**: `buffer` must remain valid and pinned until `unregister_buffer()` is called.
- This is required before calling `put_from()` or other registered-memory operations.
#### `unsafe unregister_buffer(buffer) -> Result<(), StoreError>`
Deregister a previously registered buffer.
#### `unsafe put_from(key, buffer, size, config) -> Result<(), StoreError>`
Store from a registered buffer.
- **Safety**: `buffer` must have been registered via `register_buffer()` and be at least `size` bytes.
### Batch APIs
Batch forms are useful when you want to amortize RPC overhead.
#### `unsafe batch_put_from(keys, buffers, sizes, config) -> Result<Vec<i32>, StoreError>`
Batch version of `put_from()`.
- **Returns**: per-key result codes (0 = success, non-zero = error code for that key).
- **Safety**: each `buffers[i]` must be registered and valid for `sizes[i]` bytes.
#### `unsafe batch_get_into(keys, buffers, sizes) -> Result<Vec<i64>, StoreError>`
Batch version of `get_into()`.
- **Returns**: per-key bytes written (≥ 0) or error code (< 0).
- **Safety**: each destination buffer must be writable and at least `sizes[i]` bytes.
#### `batch_is_exist(keys) -> Result<Vec<bool>, StoreError>`
Batch existence check. Errors are returned as `OperationFailed(code)`.
### `ReplicateConfig`
Replication settings for write operations (`put`, `put_from`, `batch_put_from`).
Fields:
- `replica_num`: number of replicas (0 means “use server default”).
- `with_soft_pin`: prefer retaining the object in memory (soft pin).
- `with_hard_pin`: never evict (hard pin).
- `preferred_segments`: whitelist of segment names that should host a replica.
Example:
```rust
use mooncake_store::{MooncakeStore, ReplicateConfig};
fn write_with_replication(store: &MooncakeStore) -> Result<(), mooncake_store::StoreError> {
let cfg = ReplicateConfig {
replica_num: 2,
with_soft_pin: true,
with_hard_pin: false,
preferred_segments: vec!["seg-a".to_string(), "seg-b".to_string()],
};
store.put("k", b"v", Some(&cfg))?;
Ok(())
}
```
### `StoreError`
Errors returned by the Rust wrapper.
- `NullHandle`: store handle allocation failed.
- `InvalidString`: input string contained an interior `\0` (cannot form C string).
- `OperationFailed(i32)`: underlying C layer returned a non-zero / negative code.
- `NotFound`: convenience for consumers that implement a NotFound check externally.
- `InvalidArgument(String)`: wrapper-level argument validation failure (e.g. mismatched array lengths).
## Safety & thread-safety
- `MooncakeStore` is `Send + Sync` (the underlying C object is internally synchronised).
- Methods that accept raw pointers are marked `unsafe`:
- You must uphold Rust aliasing and lifetime rules for buffers passed to FFI.
- For zero-copy operations, buffers must be registered and remain valid until unregistered.

207
docs/source/api-reference/rust/transfer-engine.md Normal file
View File

@ -0,0 +1,207 @@
# Transfer Engine Rust API
This page documents the Rust bindings living under `mooncake-transfer-engine/rust`.
At the time of writing, the crate (`transfer_engine_rust`) is primarily used as a **Rust-side binding + example binary**. The public Rust types are implemented in `src/transfer_engine.rs` and wrap the Transfer Engine C API (`transfer_engine_c.h`).
For Transfer Engine design docs and non-Rust APIs, see:
- Transfer Engine design docs: `design/transfer-engine/index`
- Transfer Engine C++ API: `design/transfer-engine/cpp-api`
## Build & runtime prerequisites
The Rust crate uses bindgen + CMake to link against the Transfer Engine C/C++ build outputs.
- **Build**:
- Requires Rust toolchain and libclang (bindgen).
- The crate has `build.rs` that expects to find / build the native library via CMake.
- **Runtime**:
- Dynamic linker must find Transfer Engine shared libraries.
- You need a metadata server backend (commonly etcd) and a reachable peer segment registry.
## Mental model
The Transfer Engine operates on **segments** and **transfer batches**:
- You create a `TransferEngine` bound to:
- `metadata_uri` (for example, etcd endpoint)
- `local_server_name` (this node's address/name)
- `rpc_port` (RPC listener port)
- You register local memory regions as RDMA-capable buffers.
- You open a remote segment to obtain a `segment_id` (an integer handle).
- You allocate a batch id for a fixed number of transfer requests.
- You submit a batch of `TransferRequest`.
- You poll status per task id inside the batch, then free the batch id.
## API reference
### Enums
#### `OpcodeEnum`
- `OpcodeEnum::Read`
- `OpcodeEnum::Write`
Used by `TransferRequest.opcode`.
#### `TransferStatusEnum`
Status values returned by the C layer. Common values you will check for:
- `Completed`
- `Failed`
- `Timeout`
### Structs
#### `TransferRequest`
One transfer operation inside a batch.
Fields:
- `opcode: OpcodeEnum`
- `source: *mut c_void`: local source/destination pointer (depends on opcode).
- `target_id: i32`: segment id returned by `open_segment()`.
- `target_offset: u64`: byte offset inside the target segment.
- `length: u64`: transfer length in bytes.
#### `BufferEntry`
Used for batch memory registration:
- `addr: *mut c_void`
- `length: u64`
### `TransferEngine`
#### `new(metadata_uri, local_server_name, rpc_port) -> anyhow::Result<TransferEngine>`
Create a new engine handle.
Notes:
- `metadata_uri` and `local_server_name` are passed through `CString`; interior `\0` bytes will error.
- The wrapper currently disables `auto_discover` in the underlying C call.
#### `discover_topology() -> anyhow::Result<()>`
Trigger topology discovery.
#### `install_transport(proto) -> anyhow::Result<()>`
Install a transport by name (e.g. `"tcp"`, `"rdma"`, `"efa"` depending on build/runtime support).
#### `register_local_memory(addr, length, location) -> anyhow::Result<()>`
Register a local memory region for zero-copy transfers.
- `addr`: pointer to the memory region.
- `length`: size in bytes.
- `location`: location string such as `"cpu:0"`.
#### `unregister_local_memory(addr) -> anyhow::Result<()>`
Unregister a previously registered memory region.
#### `register_local_memory_batch(buffer_list, location) -> anyhow::Result<()>`
Batch register multiple local buffers.
- No-op when `buffer_list` is empty.
#### `unregister_local_memory_batch(buffer_list) -> anyhow::Result<()>`
Batch unregister multiple local buffers.
#### `open_segment(name: String) -> anyhow::Result<i32>`
Open a remote segment by name and get a segment id.
#### `close_segment(segment_id: i32) -> anyhow::Result<()>`
Close a previously opened segment.
#### `warmup_efa_segment(name: &str) -> anyhow::Result<()>`
Eagerly establish EFA endpoints so the first `submit_transfer()` does not pay the serial connection setup cost.
- No-op on non-EFA transports.
- Call after `open_segment()` and after the metadata server has published the peer's NIC list.
#### `sync_segment_cache() -> anyhow::Result<()>`
Synchronize segment cache from metadata.
#### `allocate_batch_id(batch_size) -> anyhow::Result<u64>`
Allocate a batch id for `batch_size` transfer requests.
You must call `free_batch_id(batch_id)` after all tasks are done.
#### `submit_transfer(batch_id, requests) -> anyhow::Result<()>`
Submit a batch transfer request list.
- No-op when `requests` is empty.
- The wrapper converts each `TransferRequest` into the C representation (`transfer_request_t`).
#### `get_transfer_status(batch_id, task_id) -> anyhow::Result<(i32, u64)>`
Get status for one task in a batch.
- `task_id` is an index inside the batch, typically `0..batch_size`.
- Returns `(status_code, transferred_bytes)`.
The `status_code` maps to values in `TransferStatusEnum` (represented as `i32`).
#### `free_batch_id(batch_id) -> anyhow::Result<()>`
Free a previously allocated batch id.
## Minimal usage example (pseudo-code)
The crate's `src/main.rs` contains a full benchmark-style example. The following sketch shows the typical control flow:
```rust
use std::ffi::c_void;
use transfer_engine_rust::transfer_engine::{OpcodeEnum, TransferEngine, TransferRequest};
fn main() -> anyhow::Result<()> {
let engine = TransferEngine::new("127.0.0.1:2379", "127.0.0.1", 12345)?;
engine.discover_topology()?;
engine.install_transport("tcp")?;
// Register local memory (example only; you must allocate and pin memory appropriately).
let mut buffer = vec![0u8; 4096];
engine.register_local_memory(buffer.as_mut_ptr() as *mut c_void, buffer.len(), "cpu:0")?;
let seg_id = engine.open_segment("target-seg".to_string())?;
let batch_id = engine.allocate_batch_id(1)?;
let mut reqs = [TransferRequest {
opcode: OpcodeEnum::Write,
source: buffer.as_mut_ptr() as *mut c_void,
target_id: seg_id,
target_offset: 0,
length: buffer.len() as u64,
}];
engine.submit_transfer(batch_id, &mut reqs)?;
let (status, bytes) = engine.get_transfer_status(batch_id, 0)?;
println!("status={status}, bytes={bytes}");
engine.free_batch_id(batch_id)?;
engine.close_segment(seg_id)?;
engine.unregister_local_memory(buffer.as_mut_ptr() as *mut c_void)?;
Ok(())
}
```
## Safety & thread-safety
- The wrapper marks `TransferEngine` as `Send + Sync`, but it owns an FFI handle (`transfer_engine_t`).
- All pointer-based arguments must satisfy Rust’s aliasing and lifetime rules.
- You must ensure registered memory remains valid until it is unregistered.

21
docs/source/conf.py
View File

@ -41,35 +41,28 @@ extensions = [
"sphinx.ext.linkcode",
"sphinx.ext.intersphinx",
"sphinx_copybutton",
"autodoc2",
"myst_parser",
"sphinxarg.ext",
"sphinx_design",
"sphinx_togglebutton",
"sphinxcontrib.mermaid",
]
myst_enable_extensions = [
"colon_fence",
"fieldlist",
]
autodoc2_packages = [
{
"path": "../../Mooncake",
"exclude_dirs": ["__pycache__", "third_party"],
},
]
autodoc2_output_dir = "api"
autodoc2_render_plugin = "myst"
autodoc2_hidden_objects = ["dunder", "private", "inherited"]
autodoc2_sort_names = True
autodoc2_index_template = None
myst_fence_as_directive = ["mermaid"]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns: list[str] = ["**/*.template.md", "**/*.inc.md"]
exclude_patterns: list[str] = [
"**/*.template.md",
"**/*.inc.md",
"zh_archive/**",
]
# Exclude the prompt "$" when copying code
copybutton_prompt_text = r"\$ "

841
docs/source/deployment/mooncake-store-deployment-guide.md
View File

@ -1,81 +1,228 @@
# Mooncake Store Deployment & Operations Guide
# Mooncake Store Deployment & Tuning Guide
This page summarizes useful flags, environment variables, and HTTP endpoints to help advanced users tune Mooncake Master and observe metrics.
This guide covers minimal deployment, and operational tuning of Mooncake Store.
## Master Startup Flags (with defaults)
## Architecture Overview
- RPC Related
- `--rpc_port` (int, default 50051): RPC listen port.
- `--rpc_thread_num` (int, default min(4, CPU cores)): RPC worker threads. If not set, uses `--max_threads` (default 4) capped by CPU cores.
- `--rpc_address` (str, default `0.0.0.0`): RPC bind address.
- `--rpc_interface` (str, default empty): Network interface used to resolve the final RPC address. When set, Mooncake Master resolves the interface's current IPv4 address at startup and uses it as the final `rpc_address`. This overrides `--rpc_address`.
- `--rpc_conn_timeout_seconds` (int, default `0`): RPC idle connection timeout; `0` disables.
- `--rpc_enable_tcp_no_delay` (bool, default `true`): Enable TCP_NODELAY.
- Metrics
- `--enable_metric_reporting` (bool, default `true`): Periodically log master metrics to INFO.
- `--metrics_port` (int, default `9003`): HTTP port for `/metrics` endpoints.
![architecture](../image/mooncake-store-preview.png)
- HTTP Metadata Server For Mooncake Transfer Engine
- `--enable_http_metadata_server` (bool, default `false`): Enable embedded HTTP metadata server.
- `--http_metadata_server_host` (str, default `0.0.0.0`): Metadata bind host.
- `--http_metadata_server_port` (int, default `8080`): Metadata TCP port.
**Master Service** (`mooncake_master`): The central coordinator. It manages cluster membership, allocates object storage across client nodes, and enforces eviction/placement policies. Runs as a standalone process.
- Allocation Strategy
- `--allocation_strategy` (str, default `random`): Memory allocation strategy for replica placement. Available options:
- `random`: Pure random selection across segments (baseline, fastest).
- `free_ratio_first`: Free-ratio-first strategy. Samples multiple candidates and selects those with highest free space ratio for better load balancing.
**Client Node**: Each node contributes DRAM (and optionally VRAM/SSD) to form the distributed cache pool. Clients communicate with the master over RPC for control operations (`Put`/`Get`/`Remove`), but transfer actual data directly between each other via the Transfer Engine — the master is never in the data path.
- Eviction and TTLs
- `--default_kv_lease_ttl` (duration, default `5000` ms): Default lease TTL for KV objects. The default unit is milliseconds, so `5000` means `5000ms`. Duration strings such as `5000ms`, `5s`, `30m`, or `1h` are also supported.
- `--default_kv_soft_pin_ttl` (duration, default `1800000` ms): Soft pin TTL (30 minutes). The default unit is milliseconds, so `1800000` means `1800000ms`. Duration strings such as `1800000ms`, `30m`, or `1h` are also supported.
- `--allow_evict_soft_pinned_objects` (bool, default `true`): Allow evicting soft-pinned objects.
- `--eviction_ratio` (double, default `0.05`): Fraction evicted when hitting high watermark.
- `--eviction_high_watermark_ratio` (double, default `0.95`): Usage ratio to trigger eviction.
**Metadata Service**: A separate service (etcd, Redis, or HTTP) used by the Transfer Engine for peer discovery and configuration. The master's embedded HTTP metadata server can replace an external etcd/Redis for simple deployments. We also provide a P2P handshake mechanism (`P2PHANDSHAKE`) that enables decentralized metadata management by storing metadata locally on each node, eliminating the need for a centralized service — this is the simplest metadata handshake method and the recommended starting point (see [Quick Start](#quick-start)).
- High Availability (optional)
- `--enable_ha` (bool, default `false`): Enable HA (requires etcd).
- `--etcd_endpoints` (str, default empty unless HA config): etcd endpoints, semicolon separated.
- `--client_ttl` (int64, default `10` s): Client alive TTL after last ping (HA mode).
- `--cluster_id` (str, default `mooncake_cluster`): Cluster ID for persistence in HA mode.
For a detailed design discussion, see the [Mooncake Store Design](../design/mooncake-store.md).
- Task Manager (optional)
- `--max_total_finished_tasks` (uint32, default `10000`): Maximum number of finished tasks to keep in memory. When this limit is reached, the oldest finished tasks will be pruned from memory.
- `--max_total_pending_tasks` (uint32, default `10000`): Maximum number of pending tasks that can be queued in memory. When this limit is reached, new task submissions will fail with `TASK_PENDING_LIMIT_EXCEEDED` error.
- `--max_total_processing_tasks` (uint32, default `10000`): Maximum number of tasks that can be processing simultaneously. When this limit is reached, no new tasks will be popped from the pending queue until some processing tasks complete.
- `--max_retry_attempts` (uint32, default `10`): Maximum number of retry attempts for failed tasks. Tasks that fail with `NO_AVAILABLE_HANDLE` error will be retried up to this many times before being marked as failed.
---
- DFS Storage (optional)
- `--root_fs_dir` (str, default empty): DFS mount directory for storage backend, used in Multi-layer Storage Support.
- `--global_file_segment_size` (int64, default `int64_max`): Maximum available space for DFS segments.
## Quick Start
- Snapshot / Restore (optional)
- `--enable_snapshot` (bool, default `false`): Enable periodic snapshot of master metadata data (effective when using the `offset` memory allocator).
- `--snapshot_interval_seconds` (uint64, default `600`): Interval in seconds between periodic snapshots of master data.
- `--snapshot_child_timeout_seconds` (uint64, default `300`): Timeout in seconds for each snapshot child process.
- `--snapshot_retention_count` (uint32, default `2`): Number of recent snapshots to keep. Older snapshots beyond this limit will be automatically deleted.
- `--snapshot_backend_type` (str, required when snapshot enabled): Snapshot storage backend type: `local` for local filesystem, `s3` for S3 storage.
- `--snapshot_backup_dir` (str, default empty): Optional local directory for snapshot backup. If empty (default), local backup is disabled. When set, it serves two purposes: (1) during snapshot persistence, data will be saved locally as a fallback if uploading to the backend fails; (2) during restore, downloaded metadata will also be saved to this directory as a local backup.
- `--enable_snapshot_restore` (bool, default `false`): Enable restore from the latest snapshot at master startup.
- **Environment variable** `MOONCAKE_SNAPSHOT_LOCAL_PATH` (**required** when `--snapshot_backend_type=local`): Persistent directory path for local snapshot storage. This variable **must** be set before starting the master; there is no default value. Example: `export MOONCAKE_SNAPSHOT_LOCAL_PATH=/data/mooncake_snapshots`.
Deploy a minimal single-node Mooncake Store in three steps.
> **Warning: Managed Directory**
>
> The snapshot storage path (`MOONCAKE_SNAPSHOT_LOCAL_PATH` for local backend, or S3 bucket for S3 backend) is a **managed directory** exclusively controlled by the Mooncake snapshot system. **DO NOT store other files or data in this directory.** Old snapshots exceeding `--snapshot_retention_count` will be automatically and permanently deleted during cleanup. Use a dedicated, isolated directory for snapshot storage to avoid accidental data loss.
### 1. Start the Metadata Service
Example (enable embedded HTTP metadata and metrics):
This quick start uses **P2P handshake** — the simplest option, with **nothing to start**: each node exchanges and stores Transfer Engine metadata locally during connection setup. You just pass the literal string `P2PHANDSHAKE` as the client's `metadata_server` (step 3).
For large or long-lived clusters, use the master's embedded HTTP metadata server or an external etcd/Redis instead — see [Deployment Scenarios](#deployment-scenarios).
### 2. Start the Master Service
With P2P handshake the master needs no metadata-server flags:
```bash
mooncake_master
```
On success the master logs a single line like:
```
Master service started on port 50051, max_threads=4, ...
```
The master's default RPC port is `50051`. (To embed an HTTP metadata server instead of using P2P, add `--enable_http_metadata_server=true --http_metadata_server_port=8080`.)
(start-a-store-client)=
### 3. Start a Store Client
A client contributes DRAM (and optionally SSD) to the cluster. The simplest way is to embed Mooncake in a Python process and call `store.setup(...)` with `metadata_server="P2PHANDSHAKE"`:
```python
from mooncake.store import MooncakeDistributedStore
store = MooncakeDistributedStore()
store.setup(
local_hostname="localhost",
metadata_server="P2PHANDSHAKE", # decentralized; no metadata service
global_segment_size=3200 * 1024 * 1024, # DRAM contributed to the cluster
local_buffer_size=512 * 1024 * 1024, # Transfer Engine buffer
protocol="tcp",
rdma_devices="", # keyword is rdma_devices (not device_name)
master_server_addr="127.0.0.1:50051", # keyword is master_server_addr
)
```
There are **three ways** to run a client — programmatic (above), a standalone `mooncake_store_service` process (configured via `MOONCAKE_*`), and the `mooncake_client` real-client RPC process. See [Reference: Client Configuration & Tuning](#reference-client-configuration-tuning) for all three, with full parameter/env tables.
**What just happened:**
1. The client registered itself with the master via RPC.
2. The master allocated a 3.2 GB segment on this node and added it to the cluster's memory pool.
3. The client is now ready to serve `Put`/`Get`/`Remove` requests.
### Run the Stress Benchmark
Mooncake Store includes sample programs for validating C++ and Python integrations. The [stress benchmark script](gh-file:mooncake-store/tests/stress_cluster_benchmark.py) can be used to verify a two-role prefill/decode setup.
Configure the script with command-line flags (run with `--help` for the full list):
- `--local-hostname`: the local machine's reachable IP address or hostname.
- `--metadata-server`: the Transfer Engine metadata service, e.g. `P2PHANDSHAKE`, `http://127.0.0.1:8080/metadata`, or an etcd address.
- `--master-server`: the Mooncake Store master address. Use `IP:Port` in default mode, or `etcd://IP:Port;IP:Port;...;IP:Port` in etcd-backed HA mode.
- `--protocol`: transport, `tcp` / `rdma` / `cxl` / `ascend` (defaults to `rdma`).
Then start the roles:
```bash
python3 mooncake-store/tests/stress_cluster_benchmark.py --role prefill
python3 mooncake-store/tests/stress_cluster_benchmark.py --role decode
```
For RDMA, topology auto-discovery and NIC filters can be passed through environment variables:
```bash
MC_MS_AUTO_DISC=1 MC_MS_FILTERS="mlx5_1,mlx5_2" python3 mooncake-store/tests/stress_cluster_benchmark.py --role prefill
MC_MS_AUTO_DISC=1 MC_MS_FILTERS="mlx5_1,mlx5_2" python3 mooncake-store/tests/stress_cluster_benchmark.py --role decode
```
The absence of errors indicates successful data transfer.
### Verify Installed Examples
For a Python integration check, run `mooncake-store/tests/distributed_object_store_provider.py` after starting the metadata service and `mooncake_master`.
For a C++ integration check, run `build/mooncake-store/tests/client_integration_test` after building tests and starting the required services.
### Verify
```bash
# Health check — master metrics endpoint
curl -s http://localhost:9003/metrics/summary
# List registered clients
# (exposed through the store's Python API or RPC)
```
---
## Deployment Scenarios
### Single-Node (TCP) — Development / Quick Evaluation
The simplest deployment, as shown in [Quick Start](#quick-start). A single `mooncake_master` orchestrates clients over TCP. Suitable for development, testing, and single-host evaluation.
```bash
mooncake_master \
--enable_http_metadata_server=true \
--http_metadata_server_host=0.0.0.0 \
--http_metadata_server_port=8080 \
--rpc_thread_num=64 \
--metrics_port=9003 \
--enable_metric_reporting=true
--http_metadata_server_port=8080
```
Example (resolve the master RPC address from a stable interface name in a container):
Limitation: the master is a single point of failure. If it crashes, cluster operations pause until it is restored.
---
### High-Availability (etcd) — Production HA
Runs a cluster of master instances coordinated through etcd. If the leader fails, the remaining instances elect a new leader automatically.
```bash
# Start each master instance with:
mooncake_master \
--enable_ha=true \
--etcd_endpoints="10.0.0.1:2379;10.0.0.2:2379;10.0.0.3:2379" \
--rpc_address=10.0.0.1
```
Each instance must specify its own reachable `--rpc_address`. The etcd cluster used for HA can be shared with or separate from the Transfer Engine's metadata etcd.
**Client addressing:** to reach an HA cluster, clients must use the `etcd://` master-address form (so they can discover the current leader) instead of a single `IP:Port` — set `master_server_addr` (Method A) / `MOONCAKE_MASTER` (Method B) / `--master_server_address` (Method C) to `etcd://10.0.0.1:2379;10.0.0.2:2379;...`.
---
### High-Availability (Redis) — Alternative HA Backend
Same HA semantics but using Redis instead of etcd for leader election:
```bash
mooncake_master \
--enable_ha=true \
--ha_backend_type=redis \
--ha_backend_connstring="redis://127.0.0.1:6379" \
--rpc_address=10.0.0.1
```
**Client addressing:** clients reach a Redis-backed HA cluster with the `redis://connstring` master-address form (e.g. `redis://127.0.0.1:6379`) for `master_server_addr` / `MOONCAKE_MASTER` / `--master_server_address`, instead of a single `IP:Port`.
---
### Snapshot & Restore — Backup / Disaster Recovery
```{caution}
Metadata Snapshot And Restore is experimental feature.
```
Periodically persist master metadata to local disk or S3, enabling recovery from a recent snapshot after a crash.
```bash
export MOONCAKE_SNAPSHOT_LOCAL_PATH=/data/mooncake_snapshots
mooncake_master \
--enable_snapshot=true \
--snapshot_interval_seconds=300 \
--snapshot_retention_count=5 \
--snapshot_object_store_type=local \
--enable_snapshot_restore=true
```
---
### Tiered Storage with SSD Offload — Cost-Effective Capacity
Extends the cache pool from DRAM to SSD while keeping normal reads and writes on the distributed memory path. With `--enable_offload=true`, completed memory writes are queued for asynchronous SSD persistence through the master control plane. Set `--offload_on_evict=true` to defer that SSD write until the memory eviction path selects an object for reclamation. When `--promotion_on_hit=true`, SSD-only objects can be promoted back to DRAM after repeated reads; admission is gated by `--promotion_admission_threshold`.
```bash
mooncake_master \
--enable_offload=true \
--offload_on_evict=true \
--promotion_on_hit=true \
--promotion_admission_threshold=2 \
--root_fs_dir=/mnt/ssd_cache \
--enable_http_metadata_server=true \
--http_metadata_server_port=8080
```
---
### CXL-Aware Allocation — Memory Tiering
When the host has CXL-attached memory, the master can preferentially allocate new objects on the CXL tier, reserving local DRAM for latency-sensitive operations.
```bash
mooncake_master \
--enable_cxl=true \
--cxl_path=/dev/dax0.0 \
--cxl_size=17179869184 \
--allocation_strategy=cxl
```
---
### Container / Dynamic Network Interface
When the master runs in a container with a dynamic IP, use `--rpc_interface` to resolve the RPC address from a stable interface name:
```bash
mooncake_master \
@ -85,86 +232,558 @@ mooncake_master \
--http_metadata_server_port=8080
```
This resolves the current IPv4 address of `eth0` at startup and uses it as the final `rpc_address`.
The master resolves the current IPv4 address of `eth0` at startup and uses it as the advertised RPC address.
Example (use free-ratio-first allocation strategy for better load balancing):
```bash
mooncake_master \
--allocation_strategy=free_ratio_first \
--enable_http_metadata_server=true \
--http_metadata_server_port=8080
```
**Tips:**
In addition to command-line flags, the Master also supports configuration via JSON and YAML files. For example:
```bash
mooncake_master \
--config_path=mooncake-store/conf/master.yaml
```
For config files, the equivalent setting is:
```yaml
rpc_interface: "eth0"
rpc_port: 50051
```
---
## Metrics Endpoints
The master exposes Prometheus-style metrics over HTTP on `--metrics_port`:
- `GET /metrics` — Prometheus format (`text/plain; version=0.0.4`).
- `GET /metrics/summary` — Human-readable summary.
Examples:
The master exposes Prometheus-style metrics on `--metrics_port`:
```bash
# Prometheus format
curl -s http://<master_host>:9003/metrics
# Human-readable summary
curl -s http://<master_host>:9003/metrics/summary
```
## Client/Engine Tuning (Env Vars, with defaults)
When tenant quota is enabled, `/metrics` also includes per-tenant quota gauges and quota counters:
- Topology discovery (Store Client → Transfer Engine)
- `MC_MS_AUTO_DISC` (default `1`): Auto-discover NIC/GPU topology. Set `0` to disable and provide `rdma_devices` manually.
- `MC_MS_FILTERS` (default empty): Optional comma-separated NIC whitelist when auto-discovery is enabled (e.g., `mlx5_0,mlx5_2`).
- If `MC_MS_AUTO_DISC=0`, pass `rdma_devices` (comma-separated) to the Python `setup(...)` call.
- `mooncake_tenant_quota_requested_bytes{tenant_id}`
- `mooncake_tenant_quota_effective_bytes{tenant_id}`
- `mooncake_tenant_quota_used_bytes{tenant_id}`
- `mooncake_tenant_quota_reserved_bytes{tenant_id}`
- `mooncake_tenant_quota_committed_count{tenant_id}`
- `mooncake_tenant_quota_metadata_object_count{tenant_id}`
- `mooncake_tenant_quota_over_quota{tenant_id}`
- `mooncake_tenant_quota_explicit_policy{tenant_id}`
- `mooncake_tenant_quota_reject_total{tenant_id,reason}`
- `mooncake_tenant_evict_bytes_total{tenant_id}`
- `mooncake_tenant_quota_allocatable_capacity_bytes`
- `mooncake_tenant_quota_requested_bytes_sum`
- `mooncake_tenant_quota_effective_bytes_sum`
- Transfer Engine metrics (disabled by default)
- `MC_TE_METRIC` (default `0`/unset): Set to `1` to enable periodic engine metrics logging. **Note:** Not supported when using Transfer Engine TENT.
- `MC_TE_METRIC_INTERVAL_SECONDS` (default `5`): Positive integer seconds between reports (effective only if metrics enabled).
---
- Client metrics (enabled by default)
- `MC_STORE_CLIENT_METRIC` (default `1`): Client-side metrics on by default; set `0` to disable entirely.
- `MC_STORE_CLIENT_METRIC_INTERVAL` (default `0`): Reporting interval in seconds; `0` collects but does not periodically report.
## Tenant Quota Management
- Local memcpy optimization (Store transfer path)
- `MC_STORE_MEMCPY` (default `0`/false): Set to `1` to prefer local memcpy when source/destination are on the same client.
Tenant quota admission is disabled by default. Enable strict multi-tenant mode on the master when you want memory writes admitted against connector-managed per-tenant quota:
## Set the Log Level for yalantinglibs coro_rpc and coro_http
By default, the log level is set to warning. You can customize it using the following environment variable:
```bash
mooncake_master \
--enable_multi_tenants=true \
--tenant_quota_connector_type=file \
--tenant_quota_connector_uri=/etc/mooncake/tenant_quotas.yaml
```
`export MC_YLT_LOG_LEVEL=info`
The v1 connector is a writable YAML file. The file must use schema version `1`; tenant names must be non-empty, unique, must not start with `_`, and must not contain NUL or control characters; quotas must be positive integers with optional `B`, `KB`, `MB`, `GB`, or `TB` units:
This sets the log level for yalantinglibs (including coro_rpc and coro_http) to info.
```yaml
version: 1
Available log levels: trace, debug, info, warn (or warning), error, and critical.
tenants:
- name: tenant-a
quota: 200GB
- name: tenant-b
quota: 500GB
```
When strict multi-tenant mode is enabled, write requests must include a registered tenant. The `default` tenant is not special unless it is explicitly registered in the connector policy.
The same HTTP port used for metrics exposes the tenant quota admin API:
```bash
# List tenant quota snapshots
curl -s http://<master_host>:9003/api/v1/tenant_quotas
# Query one tenant
curl -s "http://<master_host>:9003/api/v1/tenant_quotas?tenant_id=tenant-a"
# Upsert an explicit policy. Explicit tenant policies must be positive.
curl -s -X PUT "http://<master_host>:9003/api/v1/tenant_quotas?tenant_id=tenant-a" \
-H 'Content-Type: application/json' \
-d '{"requested_quota_bytes":2147483648}'
# Delete an explicit policy. The tenant must not own objects or quota usage.
curl -s -X DELETE "http://<master_host>:9003/api/v1/tenant_quotas?tenant_id=tenant-a"
```
Each tenant quota snapshot returns:
```json
{
"success": true,
"data": {
"tenant_id": "tenant-a",
"requested_quota_bytes": 2147483648,
"effective_quota_bytes": 2147483648,
"used_bytes": 0,
"reserved_bytes": 0,
"committed_count": 0,
"metadata_object_count": 0,
"over_quota": false,
"has_explicit_policy": true
}
}
```
In HA mode, quota admin requests are served only by the active master service. Standby, candidate, or inactive services return HTTP 503. If strict multi-tenant mode is disabled, the quota admin API returns HTTP 409 with `UNAVAILABLE_IN_CURRENT_MODE`. Deleting a non-empty tenant returns HTTP 409 with `TENANT_NOT_EMPTY`.
---
## Quick Tips
- Scale `--rpc_thread_num` with available CPU cores and workload.
- Start with default eviction settings; adjust `--eviction_high_watermark_ratio` and `--eviction_ratio` based on memory pressure and object churn.
- Use `/metrics/summary` during bring-up; integrate `/metrics` with Prometheus/Grafana for production.
- For detailed SSD offload configuration (storage backends, eviction policies, io_uring), see the [SSD Offload guide](ssd-offload).
- For NVMe-oF SSD pool configuration see the [NVMe-oF SSD Pool Deployment Guide](nvmf-ssd-deployment-guide)
- For experimental 3FS (USRBIO) integration as a persistent storage backend, see the [3FS USRBIO Plugin guide](../getting_started/plugin-usage/3FS-USRBIO-Plugin).
- For detailed monitoring and observation see [Observability](../getting_started/observability)
:::{toctree}
:maxdepth: 1
:hidden:
ssd-offload
NvMe-Of SSD Pool<nvmf-ssd-deployment-guide>
HF3FS Plugin (Experimental)<../getting_started/plugin-usage/3FS-USRBIO-Plugin>
../getting_started/observability
:::
---
:::{toctree}
:caption: Advanced Topics
:maxdepth: 1
## Reference: Master Startup Flags
ssd-offload
:::
### RPC
| Flag | Default | Description |
|------|---------|-------------|
| `--rpc_port` | `0` → effective `50051` | RPC listen port. The literal default is `0`, which falls back to the deprecated `--port` (default `50051`) |
| `--rpc_thread_num` | `0` → effective `min(4, CPU cores)` | RPC worker threads. The literal default is `0`, which falls back to the deprecated `--max_threads` → `min(4, CPU cores)` |
| `--rpc_address` | `0.0.0.0` | RPC bind address |
| `--rpc_interface` | empty | Network interface to resolve RPC address at startup (overrides `--rpc_address`) |
| `--rpc_conn_timeout_seconds` | `0` | Idle connection timeout; `0` disables |
| `--rpc_enable_tcp_no_delay` | `true` | Enable TCP_NODELAY |
### Logging
The master uses glog. When `--log_dir` is set, all severities are merged into a single journal file in that directory (`mooncake_master.INFO.<date>-<time>.<pid>`), reachable through the stable `mooncake_master.INFO` symlink.
glog's standard flags (`--log_dir`, `--max_log_size`, `--logtostderr`, ...) control the rest.
### Metrics
| Flag | Default | Description |
|------|---------|-------------|
| `--enable_metric_reporting` | `true` | Periodically log master metrics |
| `--metrics_port` | `9003` | HTTP port for `/metrics` endpoints |
### HTTP Metadata Server (Embedded)
| Flag | Default | Description |
|------|---------|-------------|
| `--enable_http_metadata_server` | `false` | Enable embedded HTTP metadata server |
| `--http_metadata_server_host` | `0.0.0.0` | Metadata bind host |
| `--http_metadata_server_port` | `8080` | Metadata TCP port |
| `--enable_metadata_cleanup_on_timeout` | `false` | Delete a client's stale HTTP metadata (`mooncake/[<cluster>/]ram/<segment>` and `mooncake/[<cluster>/]rpc_meta/<segment>`) when its heartbeat times out (see below) |
### Stale Metadata Cleanup on Client Timeout
When a client crashes or is force-killed (`kill -9`, OOM, node failure), it cannot
run its normal cleanup, leaving stale entries on the HTTP metadata server
(`mooncake/[<cluster>/]ram/<segment>` and `mooncake/[<cluster>/]rpc_meta/<segment>`).
The HTTP metadata server has no heartbeat of its own, so these entries linger and
can mislead nodes that later connect or restart with different RDMA parameters.
With `--enable_metadata_cleanup_on_timeout=true`, the Master Service reuses its
existing client-heartbeat monitor: when a client's `--client_ttl` expires, in
addition to unmounting the segment it also removes that client's `ram/` and
`rpc_meta/` keys from the HTTP metadata server. It supports both deployment
topologies:
- **Co-located** (`--enable_http_metadata_server=true`): the master removes the
keys via a direct in-process call (no network overhead).
- **Separately deployed** HTTP metadata server: the master derives the metadata
server address from the cluster's existing configuration and removes the keys
via HTTP `DELETE`. The address is read, in priority order, from:
1. the `MOONCAKE_TE_META_DATA_SERVER` environment variable (the same Transfer
Engine metadata connection string the clients use, e.g.
`http://host:8080/metadata`), then
2. the `metadata_server` field of the JSON file pointed to by
`MOONCAKE_CONFIG_PATH`.
Notes:
- Only `http(s)` metadata servers are supported; `etcd`/`redis`/`P2PHANDSHAKE`
backends are not cleaned up (a warning is logged and cleanup stays disabled).
- The feature is opt-in and best-effort: if no co-located server is enabled and
no HTTP metadata address can be derived, the master logs a warning and
disables cleanup. Remote `DELETE` failures are logged but never block the
client-monitor thread or the main process.
- Respects `MC_METADATA_CLUSTER_ID` for custom key prefixes (matching the
Transfer Engine).
```bash
# Co-located metadata server
mooncake_master \
--enable_http_metadata_server=true \
--enable_metadata_cleanup_on_timeout=true \
--client_ttl=10
# Separately-deployed HTTP metadata server (address derived from the env var)
export MOONCAKE_TE_META_DATA_SERVER=http://metadata-host:8080/metadata
mooncake_master \
--enable_metadata_cleanup_on_timeout=true \
--client_ttl=10
```
### Memory Allocator
| Flag | Default | Description |
|------|---------|-------------|
| `--memory_allocator` | `offset` | Memory allocator: `offset` (default) or `cachelib` |
### Allocation Strategy
| Flag | Default | Description |
|------|---------|-------------|
| `--allocation_strategy` | `random` | `random` (pure random, fastest), `free_ratio_first` (best load balance), or `cxl` (prefer CXL memory) |
### PutStart Timeouts
| Flag | Default | Description |
|------|---------|-------------|
| `--put_start_discard_timeout_sec` | `30` | Seconds before an uncompleted `PutStart` is discarded |
| `--put_start_release_timeout_sec` | `600` (10 min) | Seconds before `PutStart`-allocated space is released
### Eviction & TTLs
| Flag | Default | Description |
|------|---------|-------------|
| `--default_kv_lease_ttl` | `5000` ms | Lease TTL for KV objects. Supports `5000ms`, `5s`, `30m`, `1h` |
| `--default_kv_soft_pin_ttl` | `1800000` ms | Soft pin TTL (30 min) |
| `--allow_evict_soft_pinned_objects` | `true` | Allow evicting soft-pinned objects |
| `--eviction_ratio` | `0.05` | Fraction evicted at high watermark |
| `--eviction_high_watermark_ratio` | `0.95` | Usage ratio triggering eviction |
| `--client_ttl` | `10` s | Seconds before a silent client is considered disconnected |
### Tenant Quota
| Flag | Default | Description |
|------|---------|-------------|
| `--enable_multi_tenants` | `false` | Enable strict tenant registration and per-tenant memory quota admission |
| `--tenant_quota_connector_type` | `file` | Tenant quota policy connector type |
| `--tenant_quota_connector_uri` | empty | Connector URI; for `file`, the writable YAML policy path |
### High Availability
**Master Node High Availability**
| Flag | Default | Description |
|------|---------|-------------|
| `--enable_ha` | `false` | Enable HA mode |
| `--ha_backend_type` | `etcd` | HA backend: `etcd`, `redis`, or `k8s` |
| `--ha_backend_connstring` | empty | HA backend connection string |
| `--etcd_endpoints` | empty | etcd endpoints, semicolon separated (when `--ha_backend_type=etcd`) |
| `--cluster_id` | `mooncake_cluster` | Cluster ID for HA persistence |
```{caution}
Metadata Snapshot And Restore is experimental feature.
```
**Metadata Snapshot And Restore**
| Flag | Default | Description |
|------|---------|-------------|
| `--enable_snapshot` | `false` | Enable periodic metadata snapshot |
| `--snapshot_interval_seconds` | `600` (10 min) | Interval between snapshots |
| `--snapshot_child_timeout_seconds` | `300` (5 min) | Timeout per snapshot child process |
| `--snapshot_retention_count` | `2` | Number of recent snapshots retained |
| `--snapshot_object_store_type` | required | Object store: `local` or `s3` |
| `--snapshot_catalog_store_type` | empty | Catalog store: `embedded` or `redis` |
| `--snapshot_catalog_store_connstring` | empty | Catalog store connection string (required for `redis`) |
| `--snapshot_backup_dir` | empty | Optional local backup directory |
| `--enable_snapshot_restore` | `false` | Restore from latest snapshot at startup |
**Environment variable:** `MOONCAKE_SNAPSHOT_LOCAL_PATH` (required when `--snapshot_object_store_type=local`) — persistent directory for local snapshots.
```{warning}
The snapshot storage path is a **managed directory** exclusively controlled by Mooncake. Old snapshots exceeding `--snapshot_retention_count` are automatically deleted. Use a dedicated directory to avoid data loss.
```
### Task Manager
| Flag | Default | Description |
|------|---------|-------------|
| `--max_total_finished_tasks` | `10000` | Max finished tasks kept in memory |
| `--max_total_pending_tasks` | `10000` | Max queued pending tasks |
| `--max_total_processing_tasks` | `10000` | Max simultaneously processing tasks |
| `--pending_task_timeout_sec` | `300` (5 min) | Timeout for pending tasks (`0` = no timeout) |
| `--processing_task_timeout_sec` | `300` (5 min) | Timeout for processing tasks (`0` = no timeout) |
| `--max_retry_attempts` | `10` | Max retries for failed tasks (`NO_AVAILABLE_HANDLE`) |
### Offload / Tiered Storage
Flags for controlling data movement between DRAM and SSD.
| Flag | Default | Description |
|------|---------|-------------|
| `--enable_offload` | `false` | Enable offload from DRAM to SSD |
| `--offload_on_evict` | `false` | Defer offload to eviction time rather than at `Put` |
| `--offload_force_evict` | `false` | Force-evict objects exceeding capacity without offload |
| `--offloading_queue_limit` | `50000` | Max number of objects allowed in the offloading queue per local disk segment. Increase to allow more objects to be offloaded to SSD before force-eviction kicks in |
| `--offload_cap_ratio` | `0.5` | Per-cycle offload cap as a fraction of `offloading_queue_limit` (range `[0.0, 1.0]`). Controls how many objects can be queued for offload in a single eviction cycle before falling back to force-evict |
| `--promotion_on_hit` | `false` | Promote SSD-resident keys to DRAM on read hit |
| `--promotion_admission_threshold` | `2` | Min CountMinSketch count to allow promotion (`1` = disable gating) |
| `--promotion_max_per_heartbeat` | `1` | Max promotion tasks handed to a single client per heartbeat. Each task is a synchronous SSD-read + RDMA-write on the client; serializing them avoids blocking past the client-liveness window |
| `--promotion_queue_limit` | `50000` | Max in-flight promotion tasks |
| `--quota_bytes` | `0` (90% of capacity) | Storage quota in bytes |
| `--enable_disk_eviction` | `true` | Enable disk eviction |
Start with `--enable_offload=true` for eager asynchronous SSD persistence after `Put` completion. Add `--offload_on_evict=true` when you want SSD writes to happen only when memory pressure selects an object for eviction. Add `--promotion_on_hit=true` to allow hot SSD-only data to be promoted back to DRAM, and tune `--promotion_admission_threshold` to control how many observed reads are required before promotion is queued.
When `--offload_on_evict=true` is active, each `BatchEvict` cycle can queue at most `offloading_queue_limit * offload_cap_ratio` objects for SSD offload (default: `50000 * 0.5 = 25000`); objects exceeding this cap fall back to force-evict (discard) if `--offload_force_evict=true`, otherwise they remain in memory. For SSD-heavy workloads where NVMe bandwidth is underutilized while the KV-cache hit rate suffers, raise both `--offloading_queue_limit` and `--offload_cap_ratio` so more objects per cycle are actually persisted to SSD instead of discarded. Example: `--offloading_queue_limit=500000 --offload_cap_ratio=0.8` yields a per-cycle cap of `400000` (vs the default `25000`).
### CXL Memory
| Flag | Default | Description |
|------|---------|-------------|
| `--enable_cxl` | `false` | Enable CXL memory support |
| `--cxl_path` | `/dev/dax0.0` | DAX device path for CXL memory |
| `--cxl_size` | `8GB` (`8589934592`) | CXL memory size in bytes |
When `--allocation_strategy=cxl` is set alongside `--enable_cxl=true`, the master preferentially allocates new objects on CXL memory.
### DFS Storage
| Flag | Default | Description |
|------|---------|-------------|
| `--root_fs_dir` | empty | DFS mount directory for multi-layer storage backend |
| `--global_file_segment_size` | `INT64_MAX` (unlimited) | Max available space for DFS segments; default does not cap DFS usage |
### NoF (NVMe-oF SSD Pool)
```{caution}
NVMe-oF SSD Pool (NoF) is an experimental feature.
```
Master-side flags for the NVMe-oF SSD pool. They control eviction within the NoF SSD tier and the heartbeat used to detect and unmount unresponsive NoF segments. For the client-side NoF I/O tuning (`MC_NOF_*`), see the [NVMe-oF SSD Pool Deployment Guide](nvmf-ssd-deployment-guide.md).
| Flag | Default | Description |
|------|---------|-------------|
| `--nof_eviction_ratio` | `0.05` | Fraction of objects evicted when NoF SSD space is full |
| `--nof_eviction_high_watermark_ratio` | `0.95` | Usage ratio that triggers eviction in the NoF SSD tier |
| `--nof_heartbeat_interval_sec` | `10` | How often the master probes each mounted NoF segment |
| `--nof_heartbeat_probe_timeout_ms` | `1000` | Timeout for a single NoF heartbeat probe |
| `--nof_heartbeat_failures_threshold` | `3` | Consecutive NoF heartbeat failures before a segment is unmounted |
### Master Configuration File
In addition to CLI flags, the master accepts JSON/YAML config files:
```bash
mooncake_master --config_path=mooncake-store/conf/master.yaml
```
```yaml
rpc_interface: "eth0"
rpc_port: 50051
```
---
(reference-client-configuration-tuning)=
## Reference: Client Configuration & Tuning
A client is configured through one of the **methods** introduced in [Start a Store Client](#start-a-store-client), plus a shared family of engine-tuning variables:
- **Method A — Programmatic (`setup()` arguments)**: you pass configuration as explicit Python arguments. `MOONCAKE_*` variables are **not** read in this method.
- **Method B — Service / Integration (`MOONCAKE_*` + CLI)**: `mooncake.mooncake_store_service` and the vLLM/SGLang connectors read `MOONCAKE_*` environment variables (via `MooncakeConfig`).
- **Method C — Resource-owning real client (`mooncake_client`)**: configured through `mooncake_client` CLI flags (see the **Method C** subsection below).
- **Engine runtime tuning (`MC_*`)**: low-level variables read by the C++ Transfer Engine / store client at runtime. They are orthogonal to the above and **apply to all methods**.
The Method A arguments and the `MOONCAKE_*` variables are the **same logical fields in two forms** (Method B maps onto Method A); note that the `mooncake_client` CLI (Method C) uses yet another spelling for some of them (e.g. `--device_names`, `--master_server_address`).
### Method A — Programmatic (`setup()` arguments)
Arguments of `MooncakeDistributedStore.setup(...)`:
| Argument | Type | Default | Description |
|----------|------|---------|-------------|
| `local_hostname` | str | required | This node's hostname / IP |
| `metadata_server` | str | required | `P2PHANDSHAKE` / `http://…:8080/metadata` / etcd address |
| `global_segment_size` | int (bytes) | required | DRAM contributed to the cluster (the sample uses 3.2 GB) |
| `local_buffer_size` | int (bytes) | required | Transfer Engine buffer |
| `protocol` | str | required | `tcp` / `rdma` / `efa` / `cxl` / `ascend` |
| `rdma_devices` | str | required | RDMA NIC(s), comma-separated (pass `""` for non-RDMA). **Keyword is `rdma_devices`, not `device_name`** |
| `master_server_addr` | str | required | Master `host:port`. **Keyword is `master_server_addr`, not `master_server_address`** |
| `engine` | TransferEngine | `None` | *(advanced)* Reuse an existing Transfer Engine instance instead of creating one |
| `enable_ssd_offload` | bool | `false` | *(advanced)* Enable client-side SSD offload |
| `ssd_offload_path` | str | empty | *(advanced)* SSD offload directory |
| `tenant_id` | str | `default` | *(advanced)* Tenant identifier |
```{note}
The first seven arguments have **no Python default** — the C++ defaults are not exposed by the pybind binding, so they must all be supplied (a bare `setup(local_hostname, metadata_server)` raises `TypeError`). Only `engine` / `enable_ssd_offload` / `ssd_offload_path` / `tenant_id` are optional. Also, in Method A only the `MC_*` engine variables below have any effect — `MOONCAKE_*` are ignored.
```
### Method B — Service / Integration (`MOONCAKE_*` + CLI)
`python -m mooncake.mooncake_store_service` (and the vLLM/SGLang connectors) build their configuration through `MooncakeConfig`, resolved in this order:
1. `--config <path>` CLI argument → load from that JSON file.
2. Otherwise `MOONCAKE_CONFIG_PATH` (if set) → load from that file; else read the `MOONCAKE_*` variables below.
3. `-D key=value` CLI overrides individual fields (keys must match the `MooncakeConfig` field names, e.g. `-Dmaster_server_address=...`).
```{note}
The store service CLI only accepts `--config`, `-D/--define`, `--port`, and `--max-wait-time`. There are **no** `--local_hostname` / `--metadata_server` / `--master_server` flags — use the `MOONCAKE_*` variables (or `-D`) instead.
```
| Variable | Maps to (`setup()` arg) | Default | Description |
|----------|-------------------------|---------|-------------|
| `MOONCAKE_MASTER` | `master_server_addr` | — (required unless `MOONCAKE_CONFIG_PATH`) | Master `host:port` |
| `MOONCAKE_TE_META_DATA_SERVER` | `metadata_server` | `P2PHANDSHAKE` | `P2PHANDSHAKE` / `http://…:8080/metadata` / etcd address |
| `MOONCAKE_PROTOCOL` | `protocol` | `tcp` | `tcp` / `rdma` / `efa` / `cxl` / `ascend` |
| `MOONCAKE_DEVICE` | `rdma_devices` | empty | RDMA/EFA device(s), comma-separated; `auto-discovery` supported |
| `MOONCAKE_GLOBAL_SEGMENT_SIZE` | `global_segment_size` | `3355443200` (3.125 GiB) | DRAM contributed; accepts byte integer **or** suffixed form like `500gb` |
| `MOONCAKE_LOCAL_BUFFER_SIZE` | `local_buffer_size` | `1073741824` (1 GiB) | Transfer Engine buffer; same parsing as above |
| `MOONCAKE_LOCAL_HOSTNAME` | `local_hostname` | `localhost` | |
| `MOONCAKE_OFFLOAD_ENABLED` | `enable_ssd_offload` | `false` | Client-side SSD offload |
| `MOONCAKE_OFFLOAD_FILE_STORAGE_PATH` | `ssd_offload_path` | empty | Offload directory |
| `MOONCAKE_CONFIG_PATH` | — | unset | Path to a JSON config file (takes precedence over the variables above) |
```{note}
`MooncakeConfig` (Method B) defaults `global_segment_size`/`local_buffer_size` to 3.125 GiB / 1 GiB. A direct `setup()` (Method A) has **no** default for these — they are required arguments. Unlike `MC_STORE_LOCAL_HOT_CACHE_SIZE` (raw bytes only), `MOONCAKE_GLOBAL_SEGMENT_SIZE` / `MOONCAKE_LOCAL_BUFFER_SIZE` accept human-readable suffixes (`kb`/`mb`/`gb`/…) because they are parsed by `MooncakeConfig`.
```
**Launch examples:**
```bash
# P2P handshake
MOONCAKE_MASTER=127.0.0.1:50051 \
MOONCAKE_TE_META_DATA_SERVER=P2PHANDSHAKE \
python -m mooncake.mooncake_store_service
# HTTP metadata server
MOONCAKE_MASTER=127.0.0.1:50051 \
MOONCAKE_TE_META_DATA_SERVER=http://127.0.0.1:8080/metadata \
python -m mooncake.mooncake_store_service
```
Or via a JSON config file. The service also exposes a lightweight HTTP API (on `--port`, default `8080`) for manual `Get`/`Put` debugging:
```json
{
"local_hostname": "localhost",
"metadata_server": "http://127.0.0.1:8080/metadata",
"global_segment_size": 268435456,
"local_buffer_size": 268435456,
"protocol": "tcp",
"device_name": "",
"master_server_address": "127.0.0.1:50051"
}
```
```bash
python -m mooncake.mooncake_store_service --config=<config_path> --port=8081
```
### Method C — Resource-owning Real Client (`mooncake_client`)
Run the `mooncake_client` binary as a standalone RPC process that owns storage resources; application processes (vLLM / SGLang) use lightweight **dummy clients** to forward requests to it. It connects to the master and listens on port `50052` by default.
```bash
mooncake_client \
--global_segment_size="4GB" \
--master_server_address="127.0.0.1:50051" \
--metadata_server="http://127.0.0.1:8080/metadata"
```
| Flag | Default | Description |
|------|---------|-------------|
| `--host` | `0.0.0.0` | Client service bind host |
| `--port` | `50052` | Client service listen port |
| `--global_segment_size` | `4 GB` | Global segment size contributed by the client |
| `--master_server_address` | `127.0.0.1:50051` | Master service address |
| `--metadata_server` | `http://127.0.0.1:8080/metadata` | Transfer Engine metadata service |
| `--protocol` | `tcp` | Transfer protocol |
| `--device_names` | empty | Transfer device name(s), comma-separated |
| `--threads` | `1` | Client worker thread count |
| `--enable_offload` | `false` | Enable client-side SSD offload |
| `--start_offload_rpc_server` | `true` | Start the offload RPC server for dummy clients |
### Engine Runtime Tuning (`MC_*`)
The following `MC_*` variables are read directly by the engine/client at runtime and **apply to all methods (A, B, and C)**.
#### Runtime Protocol
| Variable | Default | Description |
|----------|---------|-------------|
| `MC_RPC_PROTOCOL` | `tcp` | RPC transport protocol between master and clients: `tcp` or `rdma` |
| `MC_RPC_TIMEOUT_MS` | `30000` | Per-request deadline (ms) for all client→master RPCs. Applies uniformly to every RPC method. A negative value disables the timeout. On expiry the call returns `RPC_TIMEOUT` |
| `MC_RPC_CONNECT_TIMEOUT_MS` | `30000` | Connection-establishment timeout (ms) for the master RPC client |
| `MC_USE_TENT` / `MC_USE_TEV1` | unset | Set to any value to enable the TENT (next-gen) transfer engine |
| `MC_STORE_CLUSTER_ID` | unset | Cluster ID label attached to client metrics |
#### Topology Discovery
| Variable | Default | Description |
|----------|---------|-------------|
| `MC_MS_AUTO_DISC` | unset | Auto-discover NIC/GPU topology. Set `1` to force on, `0` to provide `rdma_devices` manually. When unset, auto-discovery is **off** except for `rdma`/`efa` protocols when no `rdma_devices` are given, where it defaults **on**. Ignored when TENT is enabled |
| `MC_MS_FILTERS` | empty | Comma-separated NIC whitelist (e.g., `mlx5_0,mlx5_2`) |
When `MC_MS_AUTO_DISC=0`, pass `rdma_devices` (comma-separated) to the Python `setup()` call.
#### Transfer Engine Metrics (disabled by default)
| Variable | Default | Description |
|----------|---------|-------------|
| `MC_TE_METRIC` | `0` | Set to `1` to enable engine metrics. Not supported with TENT |
| `MC_TE_METRIC_INTERVAL_SECONDS` | `5` | Seconds between reports |
#### Client Metrics (enabled by default)
| Variable | Default | Description |
|----------|---------|-------------|
| `MC_STORE_CLIENT_METRIC` | `1` | Set `0` to disable |
| `MC_STORE_CLIENT_METRIC_INTERVAL` | `0` | Reporting interval; `0` collects but does not periodically report |
| `MC_STORE_CLIENT_MIN_PORT` | `12300` | Min local port for client connections |
| `MC_STORE_CLIENT_MAX_PORT` | `14300` | Max local port for client connections |
#### Local Hot Cache
Local hot cache provides a DRAM read cache on top of SSD-resident objects for faster access.
| Variable | Default | Description |
|----------|---------|-------------|
| `MC_STORE_LOCAL_HOT_CACHE_SIZE` | unset | Size of the local hot cache **in raw bytes** (decimal integer, e.g., `8589934592` for 8 GB). Suffixed forms like `"8gb"` are **not** parsed. Set to a positive value to enable the hot cache |
| `MC_STORE_LOCAL_HOT_BLOCK_SIZE` | `16777216` (16 MB) | Block size for hot cache **in raw bytes** (decimal integer, e.g., `2097152` for 2 MB). Suffixed forms like `"2mb"` are **not** parsed. Only read when the hot cache is enabled |
| `MC_STORE_LOCAL_HOT_CACHE_USE_SHM` | unset | Set `1` to use memfd-backed shared memory |
| `MC_STORE_LOCAL_HOT_ADMISSION_THRESHOLD` | unset | Minimum CountMinSketch count before a key is admitted to hot cache |
#### Local Memory Optimization
| Variable | Default | Description |
|----------|---------|-------------|
| `MC_STORE_MEMCPY` | auto | Prefer local memcpy when source/destination are on the same client. When unset, auto-detected by transport: **enabled** in a TCP-only environment, **disabled** when an RDMA/other transport is available. Accepts `1`/`true`/`yes`/`on` or `0`/`false`/`no`/`off` to override |
| `MC_STORE_CLIENT_SETUP_RETRIES` | `20` | Number of times to retry client registration on failure |
| `MC_CXL_DEV_SIZE` | unset | CXL device size in raw bytes for client-side allocation. **Required when `protocol="cxl"`** — the client aborts at startup if it is missing |
#### MMap Buffer & HugePages
| Variable | Default | Description |
|----------|---------|-------------|
| `MC_STORE_USE_HUGEPAGE` | unset | Set `1` to request HugeTLB-backed `mmap()` |
| `MC_STORE_HUGEPAGE_SIZE` | `2MB` | Supported: `2MB`, `1GB` |
| `MC_MMAP_ARENA_POOL_SIZE` | unset | Pre-allocated arena pool size (e.g., `8gb`). Explicitly set to enable the arena |
| `MC_DISABLE_MMAP_ARENA` | unset | Disable arena, fall back to per-call `mmap()`. Accepts `1`/`true`/`yes`/`on` (or `0`/`false`/`no`/`off`) |
#### yalantinglibs Log Level
```bash
export MC_YLT_LOG_LEVEL=info
```
Available: `trace`, `debug`, `info`, `warn` (or `warning`), `error`, `critical`. When unset (or set to an unrecognized value), the level defaults to `warn`.

286
docs/source/deployment/nvmf-ssd-deployment-guide.md Normal file
View File

@ -0,0 +1,286 @@
# Mooncake NVMe-oF SSD Pool Deployment Guide
## Overview
This guide shows how to attach an NVMe-oF SSD pool to Mooncake Store. The
deployment has two main phases:
- Start Mooncake services built with NoF support enabled.
- Create SPDK NVMe-oF targets on SSD pool nodes and register their namespaces
with the Mooncake master.
After registration, the master reports the registered NVMe-oF namespaces as a
remote SSD pool in its metrics, and clients can place NoF replicas through
Mooncake Store.
## 1. Build Mooncake with NoF Support
Follow the "Build with NVMe-oF SSD Pool" section in the
[Build Guide](../getting_started/build.md) to install SPDK dependencies and
build Mooncake with `-DUSE_NOF=ON`.
## 2. Deploy Mooncake Services
### 2.1 Node Topology
- **Mooncake service node**: 192.168.65.81. This node runs the master, metadata, and store services.
- **SSD pool nodes**: 192.168.65.56 and 192.168.65.57. These nodes provide SSD storage resources.
### 2.2 Deploy the Master Service
```bash
mooncake_master --rpc_address=192.168.65.81
```
### 2.3 Deploy the Metadata Service
```bash
python3 -m mooncake.http_metadata_server --host=192.168.65.81 --port=8080
```
If an aiohttp-related error occurs during startup, install aiohttp:
```bash
pip3 install aiohttp
```
### 2.4 Deploy the Store Service
#### Configure `store_service.json`
Create `store_service.json` under `/home`:
```json
{
"local_hostname": "localhost",
"metadata_server": "http://192.168.65.81:8080/metadata",
"master_server_address": "192.168.65.81:50051",
"protocol": "rdma",
"device_name": "mlx5_0",
"global_segment_size": "50gb",
"local_buffer_size": 0
}
```
**Notes**:
- `device_name`: Run `ibv_devices` on node 192.168.65.81 to check the RDMA device name.
#### Start the Service
The store service initializes the SPDK environment during startup. Configure hugepages on the store service node, 192.168.65.81:
```bash
echo 512 > /proc/sys/vm/nr_hugepages
```
Only a small number of hugepages is required during startup. In most cases, 512 hugepages are sufficient.
Start the store service:
```bash
python3 -m mooncake.mooncake_store_service --config=/home/store_service.json --port=8081
```
If a timeout error occurs during startup, check whether a proxy is configured on node 192.168.65.81. If a proxy is configured, unset the proxy configuration and try again.
## 3. Deploy the NVMe-oF SSD Pool
### 3.1 Prerequisites
1. Configure passwordless SSH login from the Mooncake node, 192.168.65.81, to the SSD pool nodes, 192.168.65.56 and 192.168.65.57.
See [OpenSSH key-based authentication](https://help.ubuntu.com/community/SSH/OpenSSH/Keys).
2. Build SPDK on each SSD pool node in advance.
See [SPDK build instructions](https://github.com/spdk/spdk/blob/master/README.md#build).
### 3.2 Install SSH Dependencies
```bash
python3 -m pip install "paramiko>=3.4.0"
```
### 3.3 Deploy the SSD Pool
#### Deployment Command
```bash
python3 -m mooncake.spdk_tgt_create \
--spdk_target_info="ip:192.168.65.56 path:/home/spdk pci:0000:01:00.0,0000:02:00.0" \
--spdk_target_info="ip:192.168.65.57 path:/home/spdk" \
--core-mask=0xff \
--transport-type=RDMA \
--max-queue-depth=128 \
--max-io-qpairs-per-ctrlr=127 \
--max-io-size=4096 \
--in-capsule-data-size=131072 \
--io-unit-size=131072 \
--max-aq-depth=128 \
--num-shared-buffers=4096 \
--buf-cache-size=32
```
#### Parameters
| Parameter | Description |
|-----------|-------------|
| `ip` | IP address of the target node. |
| `path` | SPDK installation path on the target node. |
| `pci` | PCI addresses of SSDs to register with the target. Use commas to separate multiple PCI addresses. If this field is omitted, SPDK-ready or unmounted NVMe devices on the target node are registered. |
| `--core-mask` | CPU core mask used to start `nvmf_tgt` with `-m`. The default value is `0xff`. |
**Tip**: Run `/path/scripts/setup.sh status` on a target node to list available PCI addresses.
#### Transport Options
The transport options are passed to the SPDK `nvmf_create_transport` RPC. If an option is not specified, the tool uses the default value listed below.
| Option | Default | Description |
|--------|---------|-------------|
| `--transport-type` | `RDMA` | NVMe-oF transport type. |
| `--max-queue-depth` | `128` | Maximum number of outstanding I/O operations per queue. |
| `--max-io-qpairs-per-ctrlr` | `127` | Maximum number of I/O queue pairs per controller. |
| `--max-io-size` | `4096` | Maximum I/O size, in bytes. |
| `--in-capsule-data-size` | `131072` | Maximum in-capsule data size, in bytes. |
| `--io-unit-size` | `131072` | I/O unit size, in bytes. |
| `--max-aq-depth` | `128` | Maximum number of admin commands per admin queue. |
| `--num-shared-buffers` | `4096` | Number of pooled data buffers available to the transport. |
| `--buf-cache-size` | `32` | Number of shared buffers reserved for each poll group. |
## 4. Register the NVMe-oF SSD Pool
### 4.1 Register All SSDs
```bash
python3 -m mooncake.mooncake_ssd_register \
--master_server_address=192.168.65.81:50051 \
--spdk_target_info="ip:192.168.65.56 path:/home/spdk" \
--spdk_target_info="ip:192.168.65.57 path:/root/spdk"
```
#### Parameters
| Parameter | Description |
|-----------|-------------|
| `--master_server_address` | IP address and port of the master service node. The default port is 50051. |
| `--spdk_target_info` | Target node information, including `ip`, the node IP address, and `path`, the SPDK installation path. |
| `--username` | SSH username used to connect to target nodes. The default value is `root`. |
| `--port` | SSH port used to connect to target nodes. The default value is `22`. |
| `--password` | SSH password used to connect to target nodes. |
| `--key-file` | SSH private key file used to connect to target nodes. |
## 5. Unregister the NVMe-oF SSD Pool
### 5.1 Unregister a Specific SSD
```bash
python3 -m mooncake.mooncake_ssd_unregister \
--master_server_address=192.168.65.81:50051 \
--spdk_target_info="ip:192.168.65.56 ns:1 nqn:nqn.2016-06.io.spdk:cnode1"
```
#### Parameters
| Parameter | Description |
|-----------|-------------|
| `--master_server_address` | IP address and port of the master service node. The default port is 50051. |
| `--spdk_target_info` | Disk information to unregister, including `ip`, the node IP address, `ns`, the namespace ID, and `nqn`, the subsystem NQN. |
| `--username` | SSH username used to connect to target nodes. The default value is `root`. |
| `--port` | SSH port used to connect to target nodes. The default value is `22`. |
| `--password` | SSH password used to connect to target nodes. |
| `--key-file` | SSH private key file used to connect to target nodes. |
### 5.2 Get Target Disk Information
Enter the SPDK directory on the target node and run the following commands.
1. Show subsystem information, including NQN and namespace IDs:
```bash
./scripts/rpc.py nvmf_get_subsystems
```
2. Show disk details, including block size and PCI address:
```bash
./scripts/rpc.py bdev_get_bdevs
```
## 6. Performance Tests
### 6.1 Use the Built-in Benchmark Tool
```bash
./build/mooncake-store/benchmarks/nof_worker_pool_bench \
--endpoints='traddr:192.168.65.56 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1 trtype:RDMA adrfam:IPv4 ns:1, traddr:192.168.65.56 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1 trtype:RDMA adrfam:IPv4 ns:2' \
--op=read \
--io_size=1048576 \
--iodepth=8 \
--warmup_sec=3 \
--duration_sec=30
```
#### Parameters
| Parameter | Description |
|-----------|-------------|
| `--endpoints` | Disk information used for the test. |
| `--op` | I/O operation type, either `read` or `write`. |
| `--io_size` | Block size in bytes. |
| `--iodepth` | Read/write queue depth. |
| `--warmup_sec` | Warmup duration in seconds. |
| `--duration_sec` | Test duration in seconds. |
### 6.2 Use NoF with vLLM + LMCache
For the general VLLM + LMCache + Mooncake deployment flow, see
[vLLM V1 Disaggregated Serving with Mooncake Store and LMCache](../getting_started/examples/vllm-integration/vllmv1-lmcache-integration.md).
After the NVMe-oF SSD pool is registered with Mooncake, add the NoF-specific
Mooncake configuration below.
#### NoF Environment Variables
```bash
export LMCACHE_CONFIG_FILE="/path/vllm-lmcache-mooncake-config.yaml"
export MC_NOF_WORKERS=4
export MC_NOF_SUBMIT_CHUNK_BYTES=$((1 << 17)) # 128KB
export MC_NOF_INFLIGHT_BYTES_LIMIT=$((1 << 25)) # 32MB
```
#### NoF LMCache Configuration
```yaml
chunk_size: 256
remote_url: "mooncakestore://192.168.65.81:50051/"
remote_serde: "naive"
local_cpu: True
max_local_cpu_size: 8
enable_mooncake_nof_pool: True
extra_config:
local_hostname: "localhost"
metadata_server: "http://192.168.65.81:8080/metadata"
master_server_address: "192.168.65.81:50051"
global_segment_size: 0
local_buffer_size: 1073741824
protocol: "rdma"
device_name: "mlx5_0"
```
**Notes**:
- `enable_mooncake_nof_pool=True` enables writing KV cache objects to the
registered NoF pool.
- `global_segment_size: 0` means the inference process does not contribute a
memory segment to the Mooncake cluster.
- Keep `local_buffer_size` non-zero because the client still needs local
staging buffers for Mooncake transfers.
- The parameters in `extra_config` should use the same Mooncake master,
metadata server, protocol, and RDMA device as the store service.
| Environment Variable | Description | Default |
|----------------------|-------------|---------|
| `MC_NOF_WORKERS` | Number of worker threads used to process SPDK NoF I/O operations. | 4 |
| `MC_NOF_SUBMIT_CHUNK_BYTES` | Size of each I/O operation submitted to SPDK. | 128KB |
| `MC_NOF_INFLIGHT_BYTES_LIMIT` | Maximum number of in-flight I/O bytes allowed in the system. | 32MB |
These three parameters together provide QoS control for SPDK NoF I/O.

82
docs/source/deployment/ssd-offload.md
View File

@ -4,7 +4,14 @@
Mooncake Store supports offloading KV cache objects from distributed memory to local SSD. When memory pressure is high, the master instructs clients to persist selected objects to disk. On a cache miss, the client automatically falls back to reading from SSD.
SSD offload is currently **only available in Real Client mode**. The real client is a standalone process that communicates with the application (e.g., SGLang) via RPC. All SSD reads and writes happen within this process.
For measured TTFT and throughput impact in multi-turn workloads, see [Mooncake SSD Offload Benchmark](../performance/ssd-offload-benchmark-results.md).
SSD offload requires the **Real Client** and supports two deployment modes:
- **Mode A: Embedded Real Client** — the Python process embeds the Real Client, and SSD offload runs inside the Python process.
- **Mode B: Standalone Real Client + DummyClient** — a standalone `mooncake_client` process runs SSD offload, and the Python process connects via a DummyClient.
In both modes, all SSD reads and writes happen within the Real Client (embedded or standalone).
## Startup Steps
@ -19,12 +26,33 @@ mkdir -p /nvme/mooncake_offload
```bash
mooncake_master \
--rpc_port=50051 \
--enable-offload true
--enable_offload=true
```
### Step 3: Start the real client with SSD offload enabled
### Step 3A (Mode A): Start the application with embedded Real Client
Use the `--enable_offload` flag to enable SSD offload, and set environment variables to specify the storage path and backend:
Use the `--enable_ssd_offload` flag to enable SSD offload, and set environment variables to specify the storage path and backend:
```python
from mooncake.store import MooncakeDistributedStore
store = MooncakeDistributedStore()
store.setup(
local_hostname="<machine IP>",
metadata_server="P2PHANDSHAKE",
global_segment_size=4 * 1024 * 1024 * 1024, # 4 GB
local_buffer_size=512 * 1024 * 1024, # 512 MB
protocol="rdma",
rdma_devices="eth0",
master_server_addr="127.0.0.1:50051",
enable_ssd_offload=True,
ssd_offload_path="/nvme/mooncake_offload"
)
```
### Step 3B (Mode B): Start the standalone real client with SSD offload enabled and connect it with the dummy client
Set the same SSD offload environment variables as in Mode A.
```bash
export MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/nvme/mooncake_offload
@ -41,33 +69,28 @@ mooncake_client \
--metadata_server="P2PHANDSHAKE"
```
> **Note:** On startup, the real client automatically scans existing SSD data and reports it to the master. No manual recovery is needed.
### Step 4: Connect the application to the real client
The application (e.g., SGLang) connects to the real client via the `MooncakeDistributedStore` Python SDK. SSD offload and fallback loading are handled transparently.
The application (e.g., SGLang) connects to the standalone real client via the `MooncakeDistributedStore` Python SDK. The Python process acts as a DummyClient; SSD offload and fallback loading are handled by the standalone real client process.
```python
from mooncake.store import MooncakeDistributedStore
store = MooncakeDistributedStore()
store.setup(
local_hostname="<machine IP>",
metadata_server="P2PHANDSHAKE",
global_segment_size=4 * 1024 * 1024 * 1024, # 4 GB
local_buffer_size=512 * 1024 * 1024, #512MB
protocol="rdma",
device_name="eth0",
master_server_address="127.0.0.1:50051",
store.setup_dummy(
mem_pool_size=4 * 1024 * 1024 * 1024, # 4 GB
local_buffer_size=512 * 1024 * 1024, # 512 MB
server_address="<machine IP>:50052" # mooncake_client RPC address from Step 3
)
```
> **Note:** When using `bucket_storage_backend` or `file_per_key_storage_backend`, the real client scans existing SSD metadata on startup and reports it to the master automatically. `offset_allocator_storage_backend` is the exception: it truncates its data file during initialization and does not recover previously offloaded objects after a restart.
---
## Real Client Parameters
| Flag | Default | Description |
|------|---------|-------------|
| `--metadata_server` | `http://127.0.0.1:8080/metadata` | Metadata server connection string |
| `--master_server_address` | `127.0.0.1:50051` | Master address |
| `--host` | `0.0.0.0` | This machine's externally reachable IP |
| `--port` | `50052` | Real client RPC listening port |
@ -92,6 +115,8 @@ store.setup(
| `MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES` | `2199023255552` (2 TB) | Maximum disk usage |
| `MOONCAKE_OFFLOAD_TOTAL_KEYS_LIMIT` | `10000000` | Maximum number of objects on disk |
| `MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS` | `10` | Interval for offload heartbeat to master (seconds) |
| `MOONCAKE_OFFLOAD_CLIENT_BUFFER_GC_INTERVAL_SECONDS` | `1` | Interval for reclaiming expired offload buffers; defaults to the heartbeat interval in the current implementation |
| `MOONCAKE_OFFLOAD_CLIENT_BUFFER_GC_TTL_MS` | `5000` | Lease time for buffers returned by `batch_get_offload_object` before GC reclaims them |
| `MOONCAKE_OFFLOAD_USE_URING` | `false` | Enable io_uring for async file I/O |
### Bucket backend settings
@ -103,7 +128,16 @@ Applies when `MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend
| `MOONCAKE_OFFLOAD_BUCKET_SIZE_LIMIT_BYTES` | `268435456` (256 MB) | Max size per bucket |
| `MOONCAKE_OFFLOAD_BUCKET_KEYS_LIMIT` | `500` | Max keys per bucket |
| `MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE` | `0` | Eviction threshold in bytes. When set to `0`, the backend uses **90% of the physical disk capacity** as the quota — it does not mean unlimited. Set an explicit value to control disk usage precisely. |
| `MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY` | `none` | Eviction policy: `none` / `fifo` / `lru` |
| `MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY` | `fifo` | Eviction policy: `none` / `fifo` / `lru` |
### File-per-key backend settings
Applies when `MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=file_per_key_storage_backend`.
| Environment Variable | Default | Description |
|---|---|---|
| `MOONCAKE_OFFLOAD_FSDIR` | `file_per_key_dir` | Subdirectory name created under `MOONCAKE_OFFLOAD_FILE_STORAGE_PATH` |
| `ENABLE_EVICTION` | `true` | Enables local-storage eviction logic for this backend |
---
@ -176,7 +210,8 @@ The following example starts a master and a real client on a single machine.
```bash
mooncake_master \
--rpc_port=50051
--rpc_port=50051 \
--enable_offload=true
```
### Start the real client (new terminal)
@ -202,7 +237,7 @@ mooncake_client \
## Notes
- `MOONCAKE_OFFLOAD_FILE_STORAGE_PATH` must be an absolute path to an existing, writable directory. Symbolic links and paths containing `..` are rejected.
- On real client restart, the backend automatically scans existing SSD files and reports them to the master, so previously offloaded objects remain accessible.
- On real client restart, `bucket_storage_backend` and `file_per_key_storage_backend` scan existing SSD metadata and report it to the master, so previously offloaded objects remain accessible. `offset_allocator_storage_backend` does not support restart recovery.
- Eviction only notifies the master and deletes local files; objects replicated on other nodes are unaffected.
- Each machine requires its own real client process. In multi-node deployments, ensure `--host` and `--port` are correctly set so nodes can reach each other.
@ -210,7 +245,7 @@ mooncake_client \
```bash
# Node A — runs the master and its own real client
mooncake_master --rpc_port=50051 --enable-offload true &
mooncake_master --rpc_port=50051 --enable_offload=true &
export MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/nvme/mooncake_offload
mooncake_client \
@ -244,8 +279,8 @@ mooncake_client \
### SSD offload is not triggering
- Confirm `--enable_offload=true` is passed to `mooncake_client` and `--enable-offload true` is passed to `mooncake_master`.
- Check that `MOONCAKE_OFFLOAD_FILE_STORAGE_PATH` points to an existing, writable directory. The client will fail silently if the path is invalid.
- Confirm `--enable_offload=true` is passed to both `mooncake_client` and `mooncake_master`. Both binaries define the same gflags boolean; hyphenated aliases may work, but the underscored spelling matches the source.
- Check that `MOONCAKE_OFFLOAD_FILE_STORAGE_PATH` points to an existing, writable directory. If the path is invalid, real client setup fails during `FileStorageConfig::Validate()`.
- Verify memory pressure is actually high enough for the master to trigger offload. If the memory pool (`--global_segment_size`) is large relative to the data written, offload may never activate.
### "Permission denied" or "No such file or directory" on the storage path
@ -253,6 +288,7 @@ mooncake_client \
- Ensure the directory exists before starting the client: `mkdir -p <path>`.
- Confirm the process user has read/write access to the directory.
- Symbolic links and paths containing `..` are rejected — use an absolute, canonical path.
- The path must already exist and must be a directory. Passing a regular file path also causes setup to fail.
### "Failed to register buffer with UringFile" warning in logs

2
docs/source/design/architecture.md
View File

@ -12,7 +12,7 @@ Mooncake:
## Architectural Overview
![architecture](../image/mooncake-store.png)
- Mooncake provides object-level operations, i.e. `Get/Put/List/Del`, and also supports dynamically configurating replication strategies (`Replicate` operations);
- Mooncake provides object-level operations, i.e. `Get/Put/List/Del`, and also supports dynamically configuring replication strategies (`Replicate` operations);
- Mooncake supports zero-copy and multi-NIC data transfer over VRAM/DRAM/NVMe SSD. This feature is supported by Transfer Engine, which has been open-sourced;
- **The master node** centrally manages the mappings of objects to VRAM/DRAM/NVM buffers. The master node also drives **managed pool buffer nodes** to achieve data transfer by calling Transfer Engine's APIs;
- **Managed pool buffer nodes** mainly provide DRAM space for storing objects.

182
docs/source/design/conductor/conductor-architecture-design.md Normal file
View File

@ -0,0 +1,182 @@
# Mooncake Conductor Architecture
## Overview
Mooncake Conductor is the kv-cache indexer used by cache-aware routers. It
subscribes to KV cache events from inference engines or storage backends,
normalizes those events, maintains a global prefix cache table, and exposes
HTTP APIs for dynamic service registration and cache-hit queries.
The design goal is to let routers answer a simple scheduling question:
for this request prefix, which registered instance has the best reusable KV
cache locality, and on which cache tiers is that prefix available?
## Architecture
```mermaid
flowchart TD
subgraph Router["Gateway or router"]
HTTPClient["HTTP client"]
end
subgraph Conductor["Mooncake Conductor"]
EventManager["EventManager"]
ZMQClient["ZMQClient"]
KVEventHandler["KVEventHandler"]
PrefixCacheTable["PrefixCacheTable"]
end
subgraph Publishers["vLLM, SGLang, Mooncake, or cache daemons"]
InferenceService["Inference or cache service"]
KVEvent["KV event stream"]
end
EventManager -->|creates and manages| ZMQClient
ZMQClient -->|registers handler| KVEventHandler
InferenceService -->|publishes events| KVEvent
KVEvent -->|ZMQ SUB or DEALER| ZMQClient
ZMQClient -->|decoded batches| KVEventHandler
KVEventHandler -->|normalizes events| PrefixCacheTable
HTTPClient -->|/query, /register, /unregister| EventManager
EventManager -->|cache-hit computation| PrefixCacheTable
PrefixCacheTable -->|hit result by instance| EventManager
EventManager -->|HTTP response| HTTPClient
```
## Components
| Component | Responsibility |
|---|---|
| `EventManager` | Owns the Conductor lifecycle, HTTP server, dynamic registration, active service map, and tenant-to-instance map. |
| `ZMQClient` | Connects to publisher endpoints, consumes event frames, decodes event batches, tracks sequence numbers, and requests replay after reconnects. |
| `KVEventHandler` | Adapts decoded engine events into Conductor store/remove events enriched with registration metadata. |
| `PrefixCacheTable` | Maintains model-context-specific prefix maps, engine-hash to conductor-hash mappings, medium metadata, DP-rank metadata, and query-time hit computation. |
## Data model
The prefix index is scoped by `ModelContext`:
```text
(tenant_id, model_name, lora_name, block_size, additional_salt, instance_id)
```
Within each context, Conductor stores:
- a mapping from engine-provided block hash to Conductor prefix hash;
- a prefix hash map that records replica count, medium set, DP-rank set, and
per-instance access metadata;
- a DP-rank set used to report rank-level hit information.
The current implementation computes complete-block prefix hashes from token IDs
and ignores trailing partial blocks during `/query`.
## Event flow
1. A service is registered statically from `conductor_config.json` or
dynamically through `POST /register`.
2. `EventManager` creates one `ZMQClient` per `(instance_id, tenant_id,
dp_rank)` service key.
3. `ZMQClient` subscribes to the publisher endpoint and consumes frames in the
form `[topic, sequence, payload]`.
4. The payload is decoded into a batch of engine events. Today, the implemented
parser supports vLLM `BlockStored` and `BlockRemoved` msgpack events.
5. `KVEventHandler` enriches events with registration metadata such as model,
LoRA, tenant, instance, block size, and additional salt.
6. `PrefixCacheTable` updates the prefix map for stored or removed blocks.
7. If a reconnect detects missed sequence numbers, the `replay endpoint` can be
used to request missed events.
## Query flow
1. A router obtains prompt token IDs, usually from an engine tokenizer endpoint.
2. The router calls `POST /query` with `model`, `token_ids`, `block_size`, and
optional `tenant_id`, `instance_id`, `lora_name`, and `cache_salt`.
3. Conductor computes complete-block prefix hashes for the request.
4. The prefix table is scanned in order. The first miss terminates the scan so
prefix continuity is preserved.
5. Conductor returns per-instance `longest_matched`, medium hit counts, and
DP-rank hit counts.
6. The router selects the best target instance and forwards the request.
## Dynamic registration
Conductor supports runtime registration so routers or control planes can add
and remove KV event publishers without restarting the process.
```json
{
"endpoint": "tcp://127.0.0.1:5557",
"replay_endpoint": "tcp://127.0.0.1:5558",
"type": "vLLM",
"modelname": "qwen2.5",
"lora_name": "",
"tenant_id": "default",
"instance_id": "vllm-prefill-node1",
"block_size": 128,
"dp_rank": 0,
"additionalsalt": ""
}
```
Static configuration uses the same fields under `kvevent_instance`:
```json
{
"http_server_port": 13333,
"kvevent_instance": {
"vllm-prefill-node1": {
"endpoint": "tcp://127.0.0.1:5557",
"replay_endpoint": "tcp://127.0.0.1:5558",
"type": "vLLM",
"modelname": "qwen2.5",
"lora_name": "",
"tenant_id": "default",
"instance_id": "vllm-prefill-node1",
"block_size": 128,
"dp_rank": 0,
"additionalsalt": ""
}
}
}
```
## Environment variables
| Variable | Default | Description |
|---|---|---|
| `CONDUCTOR_LOG_LEVEL` | `INFO` | Log level: `DEBUG`, `INFO`, `WARN`, or `ERROR`. |
| `CONDUCTOR_CONFIG_PATH` | `~/.mooncake/conductor_config.json` | Path to the static configuration file. |
| `CONDUCTOR_SEED` | random | Legacy seed option for hash computation experiments. |
## Build and run
```bash
cd mooncake-conductor/conductor-ctrl
go mod tidy
go build -o mooncake_conductor .
```
```bash
export CONDUCTOR_CONFIG_PATH=../example/conductor_config.json
export CONDUCTOR_LOG_LEVEL=INFO
./mooncake_conductor
```
## Project structure
```text
mooncake-conductor/
+-- conductor-ctrl/
| +-- common/ # shared types, helpers, and sync map
| +-- kvevent/ # EventManager and KVEventHandler
| +-- prefixindex/ # prefix cache table and hit computation
| +-- zmq/ # ZMQ client, event decoding, event types
| +-- main.go # process entry point
+-- example/ # demo config and cache-aware proxy
+-- build.sh
+-- CMakeLists.txt
```
See [Indexer API](./indexer-api-design.md) for the HTTP API and KV Events wire
format.

536
docs/source/design/conductor/indexer-api-design.md
View File

@ -1,224 +1,362 @@
# Mooncake Conductor Indexer
# Mooncake Conductor Indexer API
## Introduction
The Mooncake Conductor Indexer is a specialized service designed to efficiently track and report token hit counts across various caching levels for different model instances. It provides a list of APIs that allow users to query token hit statistics based on token ID or chunked token hash, thereby facilitating optimized the performance of LLM inference.The figure below illustrates the architecture of Mooncake KVindexer: ![Mooncake KVindexer](../../image/conductor/architecture.png)
## Overview
## tiered storage & pools
We drew inspiration from the definition of [KVBM components](https://github.com/ai-dynamo/dynamo/blob/main/docs/kvbm/kvbm_components.md) and divided the KV cache into three levels: G1, G2, and G3. The detailed introduction is as follows:
Mooncake Conductor is a KV cache indexer used by routers and gateways to make
cache-aware scheduling decisions. It consumes KV cache events from inference
engines or storage backends, maintains prefix-hit metadata across cache tiers,
and exposes HTTP APIs for service registration and cache-hit queries.
- **Device Pool(G1)**: Device-resident KV block pool. Allocates mutable device blocks, registers completed blocks (immutable), serves lookups by sequence hash, and is the target for onboarding (Host→Device, Disk→Device).
- **Host Pool(G2)**: Mooncake registered memory KV pool. Receives Device offloads (Device→Host), can onboard to Device (Host→Device), and offloads to Disk. For high-performance, zero-copy data transfers, it utilizes the Mooncake Transfer-Engine.
- **Disk Pool(G3)**: SSD NVMe-backed KV pool. Receives Host offloads (Host→Disk), and provides large space for storing KV.
This document incorporates the latest API direction from:
- [RFC #1403: Mooncake KV-Store Indexer API Standardization](https://github.com/kvcache-ai/Mooncake/issues/1403)
- [RFC #1527: KV Events API Standardization](https://github.com/kvcache-ai/Mooncake/issues/1527)
## Indexer API
The Conductor can serve multiple model groups in one process. Each query is
scoped by model identity, block size, LoRA identity, tenant isolation, and the
registered instance that can receive traffic.
## Concepts
### Storage tiers
The indexer tracks KV cache availability across three logical tiers:
- **G1, Device Pool**: Device-resident KV blocks, such as GPU, NPU, HBM, or
other accelerator memory owned by inference engines.
- **G2, Host Pool**: CPU or host DRAM KV blocks, including Mooncake registered
memory pools.
- **G3, Disk Pool**: SSD, 3FS, DFS, NFS, or other disk-backed KV storage.
The `medium` field identifies the concrete tier or device type. Common values
are `gpu`, `cpu`, and `disk`. Engines may add other values as new media are
supported.
### Identity dimensions
KV cache hits are interpreted under the following dimensions:
| Dimension | Description |
|---|---|
| `model_name` or `model` | Model identifier. KV blocks from different models are incompatible. |
| `block_size` | Number of tokens per KV block. Different block sizes produce different token-to-block mappings. |
| `additional_salt` | Opaque salt used to separate hash namespaces for quantization, model revision, tenant isolation, or other deployment-specific dimensions. |
|`cache_salt`| Ensure cached data blocks are kept separate for different customers|
| `lora_name` | LoRA adapter name. Empty or `null` means the base model. |
| `tenant_id` | Upstream tenant or customer identity. Used for isolation and to keep query output bounded. |
| `instance_id` | Routable API server or engine instance returned by the Indexer API. Routers use this value as the scheduling target. |
| `backend_id` | KV Events identity for the entity that owns the KV blocks. It may be an inference worker, a Mooncake storage daemon, or another cache backend. |
| `medium` | Cache medium where the blocks are present. |
| `dp_rank` | Data-parallel rank that owns or can serve the blocks. |
`instance_id` and `backend_id` intentionally have different meanings.
`instance_id` is the router-facing target in the Indexer API. `backend_id` is
the event-facing cache owner in the KV Events API. In deployments where cache
storage is decoupled from inference workers, `backend_id` can identify a cache
daemon while `instance_id` still identifies the engine endpoint that receives
requests.
## Hashing standard
The standardized event contract recommends **XXH3-64 with seed `S`**.
- **Local block hash**:
`XXH3(token_bytes_le, S)`, where tokens are little-endian `u32` values
concatenated for one block.
- **Rolling sequence hash**:
The first block uses `seq_hash[0] = local_block_hash[0]`. Each subsequent
block uses:
```text
seq_hash[i] = XXH3(seq_hash[i-1]_le || local_block_hash[i]_le, S)
```
Here `||` means byte concatenation, not a logical OR.
All hashes used by the standardized KV Events API are rolling sequence hashes.
A `seq_hash` identifies the whole prefix up to that block depth, so equal
prefixes produce equal hashes until the first differing block.
If an engine does not follow the standardized hashing scheme, it must provide
`token_ids` in `stored` events so the consumer can recompute the indexer's hash
representation.
## HTTP APIs
### `POST /register`
Registers a KV event publisher and starts consuming events from it.
```json
{
"endpoint": "tcp://1.1.1.1:5557",
"replay_endpoint": "tcp://1.1.1.1:5558",
"type": "vLLM",
"modelname": "deepseek",
"lora_name": "sql-adapter",
"tenant_id": "default",
"instance_id": "vllm-prefill-node1",
"block_size": 128,
"dp_rank": 0,
"additionalsalt": "w8a8"
}
```
| Field | Required | Description |
|---|---|---|
| `endpoint` | Yes | ZMQ KV event publisher endpoint. |
| `replay_endpoint` | No | ZMQ replay endpoint used to recover missed events. |
| `type` | Yes | Publisher type, such as `vLLM`, `SGLang`, or `Mooncake`. |
| `modelname` | Yes | Model name for this publisher. This is the HTTP API wire name for `model_name`. |
| `lora_name` | No | LoRA adapter name. Empty or omitted means base model. |
| `tenant_id` | No | Tenant identity. Defaults to `default`. |
| `instance_id` | Yes | Router-facing engine or API server instance identity. |
| `block_size` | Yes | KV block size in tokens. |
| `dp_rank` | Yes | Data-parallel rank for this publisher. |
| `additionalsalt` | No | HTTP API wire name for `additional_salt`. Defaults to an empty string. |
Successful response:
```json
{
"status": "registered successfully",
"instance_id": "vllm-prefill-node1"
}
```
### `POST /unregister`
Stops consuming events for a registered publisher.
```json
{
"type": "vLLM",
"modelname": "deepseek",
"lora_name": "sql-adapter",
"tenant_id": "default",
"instance_id": "vllm-prefill-node1",
"block_size": 128,
"dp_rank": 0
}
```
`tenant_id` defaults to `default`. The current implementation removes the
subscription identified by `(instance_id, tenant_id, dp_rank)`.
Successful response:
```json
{
"status": "unregistered successfully",
"removed_instances": ["vllm-prefill-node1|default|0"]
}
```
### `POST /query`
query token hit count.
- **Input**:
- **Body** (JSON):
```json
{
"model": "deepseek",
"lora_name": "xx-adapter",
"lora_id": 12, // defined for backward compatibility and should not be used together with `lora_name`
"token_ids": [1, 15, 100],
"tenant_id": None,
"cache_salt": None,
}
```
- **Parameter Description**:
- `model`: (required, string) model name
- `lora_name`: (optional, string) The name of the LoRA adapter, default is `None`(indicating no LoRA adapter is used)
- `lora_id`: (optional, int) The ID of the LoRA adapter. This parameter is defined for backward compatibility and should not be used together with `lora_name`(Only one of them can be specified). Default is `-1`(indicating no LoRA adapter is used)
- `token_ids`: (required, [int]) prompt token id list
- `tenant_id`: (optional, int) In a multi-tenant architecture, tenant_id is the key identifier for distinguishing and isolating data from different tenants (such as different companies or user groups). All data operations are logically isolated based on this ID. If you provide it, the indexer will only return the token hit information for this tenant. Default is None(meaning there is only one tenant)
- `cache_salt`: (optional, int) An optional salt value to ensure cached data blocks are kept separate for different customers. This prevents one customer's kv-index data from being served to another. Default is None, meaning no salt is used.
- **example**:
```json
{
"model": "deepseek-v3",
"lora_name": "sql_adapter",
"token_ids": [101, 15, 100, 55, 89],
}
```
- **Output**:
```json
{
"data": {
"tenant_id": {
"api_server_unique_name": {
"longest_matched": 100, // the number of longest prefix matched token among multiple DPs(if there are)
"GPU": 20,
"DP": {
0: 10,
1: 20
},
"CPU": 60,
"DISK": 10
},
... // other engine instance
},
... // other tenant
}
}
```
- **Parameter Description**
- `tenant_id`: tenant id, only used in multi-tenant scenario.
- `api_server_unique_name`: it is a unique name for a LLM API server endpoint in the engine side. For example, two service instances are currently started separately by running the `vllm server` command, and they are registered in the indexer with different names(such as vllm-1,vllm-2)
- `longest_matched`: the number of longest prefix matched token among G1/G2/G3. Indexer will sequentially query the hit status of each token-block according to the prefix order. If it hits, count the situation of this token-block at each level; If it missed, terminate the query (ensuring prefix continuity).
- `GPU`, `CPU`, `DISK`: token ids hit count for each tiered storage medium. The Indexer will track the storage status of KV-cache across various media. This requires different KV publishers to inform the Indexer of the actual storage medium type via kv-events. The following examples list several common names, such as using GPU or NPU to represent the Device Pool, using CPU to represent the Host Pool, and using DISK to represent the Disk Pool.
- `DP`: token ids hit count for each DP rank.
- **example**:
Assume the input token_ids are [101, 15, 100, 55, 89, 63], the block_size is 2, and the dp2 strategy is enabled. There are three block hashes to match [H1, H2, H3], where H1 hits in GPU (dp0, dp1), CPU, and DISK; H2 hits in GPU (dp0) and CPU; and H3 hits in DISK.
```json
{
"vllm-1": {
"longest_matched": 6,
"GPU": 4,
"DP": {
0: 4,
1: 2
},
"CPU": 4,
"DISK": 4
}
}
```
Query cache hits by token IDs.
```json
{
"model": "deepseek",
"lora_name": "sql-adapter",
"token_ids": [101, 15, 100, 55, 89],
"tenant_id": "default",
"instance_id": "vllm-prefill-node1",
"block_size": 64,
"cache_salt": "w8a8"
}
```
| Field | Required | Description |
|---|---|---|
| `model` | Yes | Model name. |
| `lora_name` | No | LoRA adapter name. Empty or omitted means base model. |
| `lora_id` | No | Deprecated compatibility field. Do not use together with `lora_name`. |
| `token_ids` | Yes | Prompt token IDs. Only complete blocks are considered. |
| `tenant_id` | No | Tenant identity. Defaults to `default`. |
| `instance_id` | No | If set, query one instance. If omitted, query all instances registered under the tenant. |
| `block_size` | Yes | KV block size in tokens. |
| `cache_salt` | No | Query-side hash namespace salt. Corresponds to the event `additional_salt` concept. |
Response:
```json
{
"default": {
"vllm-prefill-node1": {
"longest_matched": 256,
"GPU": 128,
"DP": {
"0": 128,
"1": 256
},
"CPU": 256,
"DISK": 0
}
}
}
```
| Field | Description |
|---|---|
| `longest_matched` | Longest continuous prefix hit in tokens across all tracked media and DP ranks for this instance. |
| `GPU`, `CPU`, `DISK` | Matched prefix tokens available on each medium. Names are examples; future media can be added. |
| `DP` | Matched prefix tokens grouped by data-parallel rank. |
### `POST /query_by_hash`
query token hit count by chunked_token hash key. Each model service uses its own independent page_size, `longest_matched = page_size * matched hash_key`
- **Input**:
- **Body** (JSON):
```json
{
"model": "deepseek",
"lora_name": "xx-adapter",
"lora_id": 12, // defined for backward compatibility and should not be used together with `lora_name`
"block_hash": ["hash_key_by_chunked_tokens"],
"tenant_id": None,
"cache_salt": None,
}
```
- **Parameter Description**:
- `model`: (required, string) model name
- `lora_name`: (optional, string) The name of the LoRA adapter, default is `None`(indicating no LoRA adapter is used)
- `lora_id`: (optional, int) The ID of the LoRA adapter. This parameter is defined for backward compatibility and should not be used together with `lora_name`(Only one of them can be specified). Default is `-1`(indicating no LoRA adapter is used)
- `block_hash`: (required, [int]) chunk_token hash list
- `tenant_id`: (optional, int) In a multi-tenant architecture, tenant_id is the key identifier for distinguishing and isolating data from different tenants (such as different companies or user groups). All data operations are logically isolated based on this ID. If you provide it, the indexer will only return the token hit information for this tenant. Default is None(meaning there is only one tenant)
- `cache_salt`: (optional, int) An optional salt value to ensure cached data blocks are kept separate for different customers. This prevents one customer's kv-index data from being served to another. Default is None, meaning no salt is used.
- **Output**:
```json
{
"data": {
"tenant_id": {
"api_server_unique_name": {
"longest_matched": 100, // the number of longest prefix matched token among multiple DPs(if there are)
"GPU": 20,
"DP": {
0: 10,
1: 20
},
"CPU": 60,
"DISK": 10
},
... // other engine instance
},
... // other tenant
}
}
```
- **Parameter Description**: The output result is same as `/query` api.
Queries cache hits by precomputed rolling sequence hashes. This API avoids
sending long token lists over the network.
## Indexer KVEvents Structure
Typically, the device pool is used for loading model weights, with the remaining space registered for KV blocks by the model inference service runtime, the host pool and disk pool are managed uniformly by the Mooncake Store. There is a difference in the management unit for KV data between the two: the device pool uses blocks as the smallest unit for KV data, while the host pool and disk pool use Mooncake Store Objects as the smallest unit for KV storage. In practice, users may split a complete KV block into multiple Mooncake Store Objects for maintenance according to parallel strategies such as tensor parallelism (tp) and context parallelism (cp).
### G1 KVEvents
[vLLM](https://github.com/vllm-project/vllm/blob/main/vllm/distributed/kv_events.py) and [SGLang](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/disaggregation/kv_events.py) publishes events using the `EventBatch` structure, with each batch containing three types of events:
- `BlockStored`：Adds a single KV block.
- `BlockRemoved`：Removes a single KV block.
- `AllBlocksCleared`：Clears all KV blocks.
```py
EventBatch：
```json
{
ts: float， # timestamp
events：list[BlockStored | BlockRemoved | AllBlocksCleared]，
data_parallel_rank: int | None = None, # vLLM use this to indicate dp rank
attn_dp_rank: int | None = None, # SGLang use this to indicate dp rank
"model": "deepseek",
"lora_name": "sql-adapter",
"seq_hashes": [1234567890, 9876543210],
"tenant_id": "default",
"instance_id": "vllm-prefill-node1",
"block_size": 64,
"cache_salt": "w8a8"
}
BlockStored:
{
block_hashes: list[int]
parent_block_hash: int | None
token_ids: list[int]
block_size: int
lora_id: int | None
"""Deprecated: use `lora_name` for KV block key hash.
Retained for backward compatibility.
"""
medium: str | None
"""KV cache is categorized by tier. Currently, the following types are supported:
set "GPU", "NPU" for device pool(G1),
set "CPU" for host pool(G2),
set "DISK" for disk pool(G3).
In the future, more medium types can be supported for each tier. For example, "TPU" and "AMD" could be added for device pool.
"""
lora_name: str | None
}
BlockRemoved:
{
block_hashes: list[int]
lora_name: str | None
}
```
### G2/G3 KVEvents
Mooncake Store is a distributed key-value (KV) store. To ensure system consistency, a timestamp is assigned to each KVEvent for maintenance.
For compatibility with earlier drafts, clients may call the hash list
`block_hash`, but new clients should use `seq_hashes` to make it explicit that
the values are rolling sequence hashes rather than local block hashes.
Mooncake publishes events using the `EventBatch` structure, with each batch containing three types of events:
- `BlockStoreEvent`：Adds a single Mooncake Store Object.
- `BlockUpdateEvent`：Updates a single Mooncake Store Object.
- `RemoveAllEvent`：Removes all Mooncake Store Objects.
The response shape is the same as `/query`. For this API,
`longest_matched = block_size * matched_hash_count`.
```cpp
EventBatch {
std::vector<std::variant<BlockStoreEvent, BlockUpdateEvent, RemoveAllEvent>> events,
}
BlockStoreEvent {
"BlockStoreEvent", // Event type identifier, string type
float ts, // timestamp
std::string mooncake_key,
std::vector<std::string> addr_list, // Storage location of each replica
## KV Events API
uint64_t block_hash,
uint64_t parent_block_hash,
std::vector<uint32_t> token_id,
uint32_t block_size,
The KV Events API is the wire contract between cache owners and indexers.
Conductor normalizes engine-specific events into this model.
std::string model_name,
std::string lora_name,
uint32_t lora_id, // Retained for backward compatibility
}
### Event envelope
BlockUpdateEvent {
"BlockUpdateEvent",
float ts, // timestamp
mooncake_key,
std::vector<std::string> addr_list,
}
Every standardized event carries the same envelope:
RemoveAllEvent {
"RemoveAllEvent"
```json
{
"event_id": 42,
"timestamp": 1739145600000,
"event_type": "stored",
"model_name": "llama-3.1-8b",
"block_size": 64,
"additional_salt": null,
"lora_name": null,
"tenant_id": "default",
"backend_id": "worker-0",
"medium": "gpu",
"dp_rank": 0
}
```
| Field | Type | Description |
|---|---|---|
| `event_id` | `u64` | Monotonically increasing sequence number scoped by the event stream dimensions. Authoritative for ordering. |
| `timestamp` | `u64 or null` | Unix epoch milliseconds. Informational only, not used for ordering. |
| `event_type` | `string` | One of `stored`, `removed`, or `cleared`. |
| `model_name` | `string or null` | Model identifier. |
| `block_size` | `u32 or null` | Tokens per block. |
| `additional_salt` | `string or null` | Opaque deployment salt or namespace. |
| `lora_name` | `string or null` | LoRA adapter name, or `null` for the base model. |
| `tenant_id` | `string` | Tenant or customer identity. |
| `backend_id` | `string` | Entity that owns the KV blocks. This can be an engine worker or a decoupled cache daemon. |
| `medium` | `string or null` | Cache medium such as `gpu`, `cpu`, or `disk`. |
| `dp_rank` | `u32 or null` | Data-parallel rank. |
Events must be processed in consecutive `event_id` order within each stream
identified by `(model_name, block_size, additional_salt, lora_name, tenant_id,
backend_id, medium, dp_rank)`.
### `stored`
Published when one or more consecutive blocks are committed to a KV cache.
```json
{
"event_id": 42,
"timestamp": 1739145600000,
"event_type": "stored",
"model_name": "llama-3.1-8b",
"block_size": 64,
"additional_salt": null,
"lora_name": null,
"tenant_id": "default",
"backend_id": "worker-0",
"medium": "gpu",
"dp_rank": 0,
"seq_hashes": [1234567890, 9876543210, 1122334455],
"base_block_idx": 5,
"parent_hash": 9999999999,
"token_ids": null
}
```
| Field | Type | Description |
|---|---|---|
| `seq_hashes` | `u64[]` | Rolling sequence hashes of consecutive stored blocks. |
| `base_block_idx` | `u32 or null` | Zero-based depth of the first block in this event. |
| `parent_hash` | `u64 or null` | Rolling sequence hash at depth `base_block_idx - 1`; `null` at the root. |
| `token_ids` | `u32[] or null` | Tokens across all blocks in this event. Required when the publisher does not use the standardized hash. |
At least one of `base_block_idx` or `parent_hash` must be present so the
consumer can locate the blocks in the sequence.
### `removed`
Published when one or more blocks are evicted.
```json
{
"event_id": 43,
"timestamp": 1739145601000,
"event_type": "removed",
"model_name": "llama-3.1-8b",
"block_size": 64,
"additional_salt": null,
"lora_name": null,
"tenant_id": "default",
"backend_id": "worker-0",
"medium": "gpu",
"dp_rank": 0,
"seq_hashes": [1122334455],
"base_block_idx": 7
}
```
`seq_hashes` is required. `base_block_idx` is optional but recommended for
collision detection and observability.
### `cleared`
Published when all blocks for the event stream dimensions are purged.
```json
{
"event_id": 44,
"timestamp": 1739145602000,
"event_type": "cleared",
"model_name": "llama-3.1-8b",
"block_size": 64,
"additional_salt": null,
"lora_name": null,
"tenant_id": "default",
"backend_id": "worker-0",
"medium": "gpu",
"dp_rank": 0
}
```
No additional payload fields are required.
## Compatibility notes
The current Conductor implementation consumes vLLM ZMQ msgpack batches and
normalizes `BlockStored` and `BlockRemoved` into the internal prefix index.
Registration metadata supplies fields such as `modelname`, `tenant_id`,
`instance_id`, `block_size`, and `additionalsalt` when the engine event does
not carry the full standardized envelope.

144
docs/source/design/engram.md Normal file
View File

@ -0,0 +1,144 @@
# EngramStore Backend
Mooncake provides EngramStore as the storage backend for Engram embedding tables.
The scope is intentionally narrow:
- the caller defines the physical table layout
- the caller uploads one table per head
- the caller provides precomputed row ids with shape `[B, L, H]`
- Mooncake returns the selected rows as `[B, L, H, D]`
Mooncake does not implement tokenizer compression, N-gram hashing, query logic,
or any other model-side Engram algorithm.
## Current Backend Boundary
The current implementation is intentionally conservative. It keeps EngramStore
on top of the existing Store interfaces and does not depend on:
- transfer scatter read
- grouped transfer task
- `get_into_range`
- `batch_query`
- local direct mapping
- query cache
- remote gather control-plane changes
Those optimizations are deferred to follow-up PRs so that the EngramStore backend can
land first as a small, reviewable unit.
## Configuration
`EngramStoreConfig` contains the physical layout for one EngramStore layer:
- `table_vocab_sizes`: per-head table sizes `[N_0, N_1, ..., N_{H-1}]`
- `embedding_dim`: row width `D`
For `layer_id`, Mooncake generates one store key per head:
```text
engram:l{layer_id}:h{head_idx}
```
Each key stores a `float32` table with shape `[N_h, D]`.
## Public Interface
Python:
- `EngramStore(layer_id, config, store=None)`
- `populate(embedding_buffers)`
- `lookup(row_ids)`
- `remove_from_store(force=False)`
- `get_table_vocab_sizes()`
- `get_store_keys()`
- `get_num_heads()`
- `get_embedding_dim()`
The Python `store` argument accepts the existing `MooncakeDistributedStore`
wrapper, or `None` for metadata-only construction.
C++:
- constructor `EngramStore(int layer_id, const EngramStoreConfig&, std::shared_ptr<PyClient>)`
- `populate(...)`
- `lookup_rows(...)`
- `lookup_rows_contiguous(...)`
- `remove_from_store(...)`
- metadata getters matching the Python surface
## Data Contract
Populate expects one NumPy `float32` array per head:
```text
embedding_buffers[h].shape == [N_h, D]
```
Lookup accepts either:
- nested Python lists with logical shape `[B, L, H]`, or
- a contiguous NumPy `int64` array with shape `[B, L, H]`
Lookup returns:
```text
output.shape == [B, L, H, D]
```
## Populate Flow
Populate follows the existing Store write path:
1. validate that exactly one table is provided for each head
2. validate that every table matches `[N_h, D]`
3. verify that the target head-table keys do not already exist
4. register each embedding table buffer
5. upload all head tables with `batch_put_from(...)`
6. unregister the staging buffers
`populate(...)` is defined as a create-only operation for one EngramStore layer. To
reuse a `layer_id`, first remove the old tables with `remove_from_store(...)`.
If upload fails after some head tables have already been written, or if publish
finishes but post-write buffer cleanup fails, the backend best-effort removes
the keys written by the failed populate attempt before returning an error.
## Lookup Flow
Each lookup follows the same simplified backend flow:
1. validate the `row_ids` shape and bounds
2. build per-head byte ranges for the requested rows
3. issue one `get_into_ranges(...)` call to materialize those rows into the output buffer
For NumPy `row_ids`, the binding uses a contiguous fast path and builds ranges
directly from the input tensor without first converting the entire input into a
nested C++ container.
## Validation
The backend enforces these invariants:
- `table_vocab_sizes` is non-empty and every entry is positive
- `embedding_dim` is positive
- `populate(...)` receives exactly one table per head
- every populated table matches `[N_h, D]`
- `lookup(...)` receives a non-empty `[B, L, H]` input
- every row id satisfies `0 <= row_ids[..., h] < N_h`
## Validation Status
This backend is covered by:
- correctness tests in `scripts/test_engram_store.py`
- benchmark coverage in `scripts/bench_engram_store_27b.py`
`scripts/test_engram_store.py` can run against an existing Mooncake deployment through
`MOONCAKE_CONFIG_PATH` / `MOONCAKE_MASTER`, or it can start a local
`mooncake_master` instance automatically for a self-contained TCP test run.
By default, the benchmark exercises `engram_store.populate(...)` directly. Its
fallback populate paths are gated behind `ENGRAM_ALLOW_POPULATE_FALLBACK=1` so
they do not silently mask regressions in the current implementation.

224
docs/source/design/mooncake-backend-pg.md Normal file
View File

@ -0,0 +1,224 @@
# Mooncake Backend (PG) Design
Mooncake Backend is a `torch.distributed` ProcessGroup backend for Mooncake. It
provides collective and point-to-point communication primitives, rank-health
tracking, and elastic recovery hooks for inference systems that need to keep
serving after partial rank failures.
This document is intended for developers who maintain Mooncake PG itself or
integrate it into higher-level serving systems.
## Goals
Mooncake Backend is designed to:
- integrate with PyTorch through the standard ProcessGroup extension mechanism;
- expose `mooncake` for accelerator tensors and `mooncake-cpu` for CPU tensors;
- support common collective APIs used by inference engines;
- track active and inactive ranks so collectives can continue after failures;
- allow replacement ranks to publish metadata, join an existing group, and be
activated by healthy ranks;
- reuse Mooncake Transfer Engine and topology information for data movement.
Non-goals:
- It is not a drop-in replacement for every NCCL/Gloo behavior. Validate each
collective, dtype, and topology required by the application.
- Elastic recovery is an explicit protocol. The backend does not silently add a
new process to all collectives without application coordination.
## Relationship with `torch.distributed`
Mooncake registers two PyTorch backends when the PG extension module is imported:
- `mooncake-cpu`, registered for CPU devices;
- `mooncake`, registered for accelerator devices such as CUDA or MUSA depending
on the build.
The backend class itself derives from `c10d::ProcessGroup`. Applications use
regular PyTorch APIs such as `dist.init_process_group()`, `dist.all_reduce()`,
`dist.new_group()`, and `dist.batch_isend_irecv()`.
Point-to-point dispatch in PyTorch expects a `c10d::Backend` object. Mooncake PG
therefore includes a lightweight P2P shim that delegates `send` and `recv` calls
back to the owning `MooncakeBackend` instance.
## Main runtime objects
### `MooncakeBackendOptions`
`MooncakeBackendOptions` carries Mooncake-specific process-group configuration:
| Field | Meaning |
| --- | --- |
| `activeRanks_` | Rank-health tensor exposed to collectives and user code. |
| `isExtension_` | Whether this process is a joining/replacement rank. |
| `maxWorldSize_` | Optional reserved capacity for future ranks. |
The `activeRanks_` tensor must be `torch.int32`. It must be on CPU for
`mooncake-cpu` and on the accelerator device for `mooncake`. When
`maxWorldSize_` is set, `activeRanks_` must be sized to `maxWorldSize_` so the
backend can reserve inactive rank slots.
### Transfer group metadata
Each backend owns shared metadata for:
- the current rank and backend index;
- current capacity (`size`) and visible active size (`activeSize`);
- host and device active-rank masks;
- peer connection state;
- rank-local and rank-global mapping;
- store handles and extension state used by recovery;
- P2P proxy and connection poller state.
`size` is the reserved capacity. `activeSize` is the visible group size returned
by `dist.get_world_size()`. With `max_world_size`, `size` may be larger than
`activeSize`; inactive slots are masked out by the active-rank state.
### Transfer Engine ownership
By default, Mooncake Backend initializes its own Transfer Engine. Advanced
integrations may call `pg.set_transfer_engine(engine)` before
`init_process_group()` to inject an external Transfer Engine. In that mode, the
caller owns the engine and must keep it alive until all Mooncake process groups
using it are destroyed.
## Initialization lifecycle
The initialization flow is:
1. Python imports `mooncake.pg`, which loads a PyTorch-version-specific native
extension.
2. The extension registers `mooncake` / `mooncake-cpu` with PyTorch.
3. `dist.init_process_group()` invokes the backend factory with PyTorch
distributed options and optional `MooncakeBackendOptions`.
4. The backend initializes active-rank masks and reserved rank slots.
5. Non-extension ranks publish local peer metadata and wait until current peers
are connected.
6. Extension ranks enter local-only mode and wait for the explicit join protocol.
The important distinction is that reserving capacity does not automatically make
future ranks active. New ranks are masked until `recover_ranks()` activates them.
## Active ranks and dynamic world size
Mooncake PG tracks two related concepts:
- **Reserved capacity** (`size`): how many rank slots the backend knows about.
- **Visible active size** (`activeSize`): the current group size visible through
PyTorch APIs.
When `max_world_size` is larger than the initial `world_size`, Mooncake reserves
extra slots but marks them inactive. This lets healthy ranks poll for joiner
metadata and activate the joiners later without reconstructing the process group.
`extend_group_size_to(size)` can also increase capacity. Newly extended ranks
start inactive; the application must call `get_peer_state()` and
`recover_ranks()` before they participate in collectives.
## Elastic recovery protocol
Mooncake PG uses a two-phase protocol for recovery and scale-up:
```mermaid
sequenceDiagram
participant H as Healthy ranks
participant J as Joining rank
participant S as Store / metadata
H->>S: init_process_group(world_size=M, max_world_size=N)
J->>S: init_process_group(world_size=N, is_extension=True)
J->>S: publish local peer metadata
H->>S: get_peer_state(join_ranks)
H->>S: recover_ranks(join_ranks)
S-->>J: extension state
J->>J: join_group() returns
H->>J: collectives include recovered ranks
```
Healthy rank responsibilities:
1. Reserve capacity with `max_world_size` or `extend_group_size_to()`.
2. Poll `get_peer_state(backend, ranks)` from all healthy ranks in a consistent
order.
3. Call `recover_ranks(backend, ranks)` once candidate ranks are connected.
4. Refresh higher-level components, such as Mooncake EP buffers, if they cache
transport metadata.
Joining rank responsibilities:
1. Initialize the process group with `is_extension=True`.
2. Publish local peer metadata through the backend initialization path.
3. Call `join_group(backend)` and block until healthy ranks publish extension
state.
4. Re-enter normal collectives after `join_group()` returns.
## Subgroup semantics
Mooncake PG follows PyTorch process-group ordering requirements. All processes
that participate in a parent group should call `dist.new_group()` in a consistent
order. This is especially important for elastic subgroups because healthy ranks
and joining ranks must agree on store prefixes and backend indices.
For split-rank elastic patterns, create subgroups using the current membership
on healthy ranks and the eventual membership on joining ranks, while preserving
the same creation order. The PG elastic tests contain executable examples of this
pattern.
## Collective behavior
Collectives use the backend active-rank state to skip inactive ranks. The exact
implementation varies by operation and device type, but the high-level contract
is:
- active ranks participate in the collective;
- inactive ranks are not waited on;
- if communication detects a rank failure, active-rank state can be updated;
- user code can read the current mask with `pg.get_active_ranks(backend)`.
The backend currently implements common collective APIs including all-reduce,
broadcast, all-gather, reduce-scatter, all-to-all, barrier, reduce, gather,
scatter, and single-tensor P2P send/recv.
## Failure and recovery boundaries
Mooncake PG exposes low-level recovery primitives; higher-level systems are
responsible for policy decisions such as:
- which ranks are safe to replace;
- when to stop routing traffic to a failed rank;
- how to recreate model state on a replacement process;
- when to refresh EP, scheduler, or application-level metadata;
- how to coordinate subgroup recovery.
Avoid assuming that `recover_ranks()` alone reconstructs all higher-level state.
It activates the process-group communication path; the application still owns
model weights, KV-cache state, routing policy, and request scheduling.
## Testing checklist for PG changes
When modifying PG internals, run at least:
```bash
# CPU functional tests
python -m unittest discover -s mooncake-pg/tests -k CPU -v
# CUDA functional tests, when GPUs are available
python -m unittest discover -s mooncake-pg/tests -k CUDA -v
# Collective benchmark smoke test
PYTHONPATH=mooncake-pg \
python mooncake-pg/benchmark/pgbench.py \
--collective all_reduce --backend mooncake --device cuda -g 2 -b 8 -e 1M -f 2
```
Also run elastic tests for changes that touch active ranks, metadata polling,
subgroups, `extend_group_size_to()`, `get_peer_state()`, `recover_ranks()`, or
`join_group()`.
## Related documentation
- [Mooncake EP design](mooncake-ep.md)
- [Python API reference](../python-api-reference/ep-backend.md)
- [PG/EP troubleshooting](../troubleshooting/pg-ep-troubleshooting.md)

247
docs/source/design/mooncake-ep.md Normal file
View File

@ -0,0 +1,247 @@
# Mooncake EP Design
Mooncake EP is the expert-parallel communication runtime used for MoE token
dispatch and combine. It follows the DeepEP low-latency programming model while
adding Mooncake transport integration and rank activeness awareness.
This document explains the runtime at a level useful for developers maintaining
Mooncake EP or integrating it into inference engines.
## Goals
Mooncake EP is designed to:
- provide low-latency dispatch/combine operations for expert-parallel MoE
inference;
- keep the Python programming model close to DeepEP low-latency mode;
- use Mooncake device transports for fast intra-node and inter-node movement;
- detect failed source ranks through timeout-aware kernels;
- interoperate with Mooncake Backend (PG) for bootstrap metadata exchange and
rank-health state.
## High-level data flow
MoE inference with Mooncake EP has three phases:
1. **Dispatch**: each rank sends token hidden states to the ranks that own the
selected experts. The receiver packs tokens by local expert.
2. **Expert compute**: each rank runs its local experts over the packed inputs.
3. **Combine**: expert outputs are routed back to the original token owners and
reduced with routing weights.
```mermaid
flowchart LR
A[Local tokens x + topk_idx] --> B[dispatch]
B --> C[Packed local expert inputs]
C --> D[Local expert kernels]
D --> E[Packed expert outputs]
E --> F[combine]
F --> G[Combined local token outputs]
```
## Relationship with Mooncake Backend (PG)
Mooncake EP is constructed from a `torch.distributed` process group:
```python
from mooncake.mooncake_ep_buffer import Buffer
buffer = Buffer(group, num_ep_buffer_bytes)
```
The group should normally be a Mooncake Backend process group. EP uses it to
exchange RDMA memory-region metadata, QP information, GID/LID information, and
IPC handles. The backend active-rank state is also used by the fallback path and
by peer synchronization.
After PG recovery changes membership, call `Buffer.update_ep_member()` to refresh
EP peer metadata and QPs before relying on the recovered ranks for dispatch and
combine.
## Runtime objects
### Python wrapper
`mooncake.mooncake_ep_buffer.Buffer` is the user-facing wrapper. It owns:
- the process group;
- the native EP buffer runtime;
- the fallback flag;
- Python fallback buffers used when fast path is unavailable.
`Buffer.connect()` exchanges peer metadata. It first tries the RDMA/IBGDA path,
then exchanges IPC handles for intra-node P2P. If neither fast path is usable, it
falls back to a Python implementation based on PyTorch collectives.
### Native buffer
The native `MooncakeEpBuffer` owns or references:
- rank and world-size metadata;
- a GDR workspace buffer;
- P2P and RDMA device transports;
- a communication stream;
- temporary workspace used by dispatch/combine kernels.
If an external Transfer Engine is supplied by a higher layer, EP can reference
device transports owned by that engine; otherwise EP creates and owns the
transports itself.
### Buffer layout
EP allocates paired send/receive buffers for double-buffered dispatch/combine.
Each pair contains:
- RDMA send signal buffer;
- RDMA receive signal buffer;
- RDMA send data buffer;
- RDMA receive data buffer.
The workspace size returned by `Buffer.get_ep_buffer_size_hint()` is derived from
`num_max_dispatch_tokens_per_rank`, `hidden`, `num_ranks`, and `num_experts`.
Size this for peak dispatch demand, not the average request.
## Fast paths and fallback
Mooncake EP has three broad execution modes:
| Mode | Purpose | Notes |
| --- | --- | --- |
| IBGDA / RDMA fast path | Inter-node GPU memory movement. | Requires RDMA-capable environment and successful metadata/QP setup. |
| P2P / IPC fast path | Intra-node peer access such as NVLink. | Requires peer accessibility and IPC handle exchange. |
| Python fallback | Functional fallback for unsupported environments. | Slower; useful for correctness and limited testing. |
The runtime reports whether IBGDA is disabled and whether the fast path is usable
through native helper methods surfaced in Python. The Python wrapper updates its
fallback flag after metadata exchange.
## Metadata exchange
During `Buffer.connect()`, ranks exchange:
- RDMA memory-region address and key;
- local QP numbers;
- LID and GID information;
- subnet prefix and interface ID;
- CUDA IPC handles for local peer access;
- current active-rank mask from the backend.
The exchange uses `dist.all_gather()` and `dist.all_to_all()` on the process
group. For this reason, the process group must already be initialized and healthy
before the EP buffer is constructed or refreshed.
## Dispatch internals
`dispatch()` takes local token hidden states and selected expert IDs. It sends
tokens to expert-owner ranks and packs received tokens into local-expert-major
layout.
Important inputs:
- `x`: `[num_tokens, hidden]` token hidden states;
- `topk_idx`: `[num_tokens, top_k]` global expert IDs;
- `active_ranks`: `[num_ranks]` int32 rank-health tensor;
- `num_max_dispatch_tokens_per_rank`: receive capacity per source rank;
- `num_experts`: global expert count, divisible by `num_ranks`;
- `timeout_us`: failure-detection timeout.
Important outputs:
- packed local-expert input tensor;
- per-local-expert receive counts;
- source/layout metadata handle used by `combine()`;
- event/hook synchronization helpers.
When `use_fp8=True`, dispatch returns packed FP8 data and FP32 scales. The local
expert path must either consume that format directly or dequantize before expert
compute.
## Combine internals
`combine()` sends local expert outputs back to token-owner ranks and applies
routing weights. It consumes the handle produced by the matching `dispatch()`
call.
For zero-copy combine, call `get_next_combine_buffer(handle)`, write expert
outputs into that buffer, then call `combine(..., zero_copy=True)` with the same
handle. Do not reuse a handle across unrelated dispatch/combine pairs.
## Rank activeness and timeout behavior
Mooncake EP receives an `active_ranks` tensor in both dispatch and combine. The
native kernels poll receive signals from source ranks. If `timeout_us` is not
`-1` and a source rank does not make progress before the timeout, the kernel can
mark `active_ranks[src_rank] = 0` and skip that source.
This EP-level tensor is rank-level and should have shape `[num_ranks]`. It is
related to, but not automatically identical to, the backend-level active-rank
mask passed through `MooncakeBackendOptions`. Integrations should propagate
health updates consistently between scheduling logic, PG state, and EP buffers.
## Stream synchronization model
Mooncake EP operations return an `EventOverlap` object and, optionally, a hook:
- If `return_recv_hook=False`, call `event.current_stream_wait()` before using
the output tensors on the current stream.
- If `return_recv_hook=True`, call the returned `hook()` at the chosen overlap
point.
- If `async_finish=True`, the wrapper records extra tensors in the event helper
to keep lifetimes safe for asynchronous use and CUDA graph scenarios.
Always make synchronization explicit when composing EP with custom expert
kernels, CUDA graphs, or application-level streams.
## Recovery integration
When a rank fails, multiple layers must be updated:
1. PG active-rank state must stop collectives from waiting for the failed rank.
2. Scheduler / MoE routing should stop assigning tokens to unavailable experts.
3. Replacement ranks should join through the PG elastic protocol.
4. EP buffers should refresh peer metadata with `update_ep_member()` after the
process group activates recovered ranks.
EP timeout detection can mark a rank inactive in the EP-level tensor, but higher
layers still need to coordinate recovery and routing decisions.
## Performance considerations
- Prefer fast path operation with working RDMA/IBGDA or P2P peer access.
- Size `num_max_dispatch_tokens_per_rank` for the worst expected per-rank token
count to avoid overflow.
- Use `return_recv_hook=True` or `async_finish=True` only when the application
deliberately overlaps communication and compute.
- `use_fp8=True` reduces dispatch bandwidth but requires FP8-aware expert code
or explicit dequantization.
- Repeatedly reconstructing EP buffers is expensive; refresh membership only
when PG membership changes.
## Developer test checklist
For EP changes, run correctness across:
- BF16 and FP8 dispatch;
- zero-copy and non-zero-copy combine;
- synchronous event wait and hook-based synchronization;
- fallback and fast path when available;
- failure simulation with finite `timeout_us`;
- multi-rank topologies where `num_experts % num_ranks == 0`.
Useful entry points:
```bash
# EP grid correctness test
python mooncake-ep/tests/test_ep_grid.py
# Wheel-level EP smoke test
python mooncake-wheel/tests/test_mooncake_ep.py
```
Adapt launch commands to the target environment and number of GPUs.
## Related documentation
- [Mooncake Backend (PG) design](mooncake-backend-pg.md)
- [Python API reference](../python-api-reference/ep-backend.md)
- [PG/EP troubleshooting](../troubleshooting/pg-ep-troubleshooting.md)

523
docs/source/design/mooncake-store.md
View File

@ -6,10 +6,11 @@ Mooncake Store is a high-performance **distributed key-value (KV) cache storage
Unlike traditional caching systems such as Redis or Memcached, Mooncake Store is positioned as **a distributed KV cache rather than a generic caching system**. The key difference is that in the latter, the key is derived from the value through hashing, so value is immutable after inserting (although the key/value pair may be evicted).
Mooncake Store provides low-level object storage and management capabilities, including configurable caching and eviction strategies that offers high memory efficiency and is specifically designed to accelerate LLM inference performance.
Mooncake Store provides low-level object storage and management capabilities, including configurable caching and eviction strategies that offer high memory efficiency and is specifically designed to accelerate LLM inference performance.
Key features of Mooncake Store include:
- **Object-level storage operations**: Mooncake Store provides simple and easy-to-use object-level APIs, including `Put`, `Get`, and `Remove` operations.
- **Optional object grouping**: Related objects can carry an optional group ID so that the Master can route their metadata to the same shard and apply best-effort shared lifecycle behavior.
- **Multi-replica support**: Mooncake Store supports storing multiple data replicas for the same object, effectively alleviating hotspots in access pressure. Each slice within an object is guaranteed to be placed in different segments, while different objects' slices may share segments. Replication operates on a best-effort basis.
- **Strong consistency**: Mooncake Store guarantees that `Get` operations always return correct and complete data. Once an object has been successfully `Put`, it remains immutable until removal, ensuring that all subsequent `Get` requests retrieve the most recent value.
- **Zero-copy, bandwidth-saturating transfers**: Powered by the Transfer Engine, Mooncake Store eliminates redundant memory copies and exploits multi-NIC GPUDirect RDMA pooling to drive data across the network at full line rate while keeping CPU overhead negligible.
@ -48,198 +49,33 @@ If the current leader fails or becomes partitioned from the network, the remaini
In both modes, the leader monitors the health of all client nodes through periodic heartbeats. If a client crashes or becomes unreachable, the leader quickly detects the failure and takes appropriate action. When a client node recovers or reconnects, it can automatically rejoin the cluster without manual intervention.
(client-c-api)=
## Client C++ API
### Constructor and Initialization `Init`
The `Client` class provides the primary interface for Mooncake Store operations:
```C++
ErrorCode Init(const std::string& local_hostname,
const std::string& metadata_connstring,
const std::string& protocol,
void** protocol_args,
const std::string& master_server_entry);
```
| API | Description |
|-----|-------------|
| `Init` | Initialize the client with metadata server, protocol, and master address |
| `Get` | Retrieve object data into pre-registered local memory slices |
| `Put` | Store object data with configurable replication and persistence |
| `Upsert` / `BatchUpsert` | Insert or update with existing placement reuse |
| `Remove` | Delete an object and all its replicas |
| `CreateCopyTask` / `CreateMoveTask` | Asynchronous cross-node data transfer |
| `QueryTask` | Monitor the status of async copy/move tasks |
| `BatchQueryIp` | Discover network locations of storage nodes |
| `BatchReplicaClear` | Batch clear replicas on specific segments |
| `QueryByRegex` / `RemoveByRegex` | Query or delete objects matching a regex |
Initializes the Mooncake Store client. The parameters are as follows:
- `local_hostname`: The `IP:Port` of the local machine or an accessible domain name (default value used if port is not included)
- `metadata_connstring`: The address of the metadata service (e.g., etcd/Redis) required for Transfer Engine initialization
- `protocol`: The protocol supported by the Transfer Engine, including RDMA and TCP
- `protocol_args`: Protocol parameters required by the Transfer Engine
- `master_server_entry`: The address information of the Master (`IP:Port` for default mode and `etcd://IP:Port;IP:Port;...;IP:Port` for high availability mode)
For full API signatures, parameter details, and usage examples, see the [Mooncake Store C++ API Reference](../api-reference/cpp/mooncake-store.md).
### Get
```C++
tl::expected<void, ErrorCode> Get(const std::string& object_key,
std::vector<Slice>& slices);
```
![mooncake-store-simple-get](../image/mooncake-store-simple-get.png)
`Get` retrieves the value of `object_key` into the provided `slices`. The returned data is guaranteed to be complete and correct. Each slice must reference local DRAM/VRAM memory that has been pre-registered with `registerLocalMemory(addr, len)` (not the global segments that contribute to the distributed memory pool). When persistence is enabled and the requested data is not found in the distributed memory pool, `Get` will fall back to loading the data from SSD.
### Put
```C++
tl::expected<void, ErrorCode> Put(const ObjectKey& key,
std::vector<Slice>& slices,
const ReplicateConfig& config);
```
![mooncake-store-simple-put](../image/mooncake-store-simple-put.png)
`Put` stores the value associated with `key` in the distributed memory pool. The `config` parameter allows specifying the required number of replicas as well as the preferred segment for storing the value. When persistence is enabled, `Put` also asynchronously triggers a persistence operation to SSD.
**Replication Guarantees and Best Effort Behavior:**
- Each slice of an object is guaranteed to be replicated to different segments, ensuring distribution across separate storage nodes
- Different slices from different objects may be placed in the same segment
- Replication operates on a best-effort basis: if insufficient space is available for all requested replicas, the object will still be written with as many replicas as possible
The data structure details of `ReplicateConfig` are as follows:
```C++
struct ReplicateConfig {
size_t replica_num{1}; // Total number of replicas for the object
bool with_soft_pin{false}; // Whether to enable soft pin mechanism for this object
bool with_hard_pin{false}; // Whether to enable hard pin (never evicted)
std::string preferred_segment{}; // Preferred segment for allocation
};
```
### Upsert
```C++
tl::expected<void, ErrorCode> Upsert(const ObjectKey& key,
std::vector<Slice>& slices,
const ReplicateConfig& config);
std::vector<tl::expected<void, ErrorCode>> BatchUpsert(
const std::vector<ObjectKey>& keys,
std::vector<std::vector<Slice>>& batched_slices,
const ReplicateConfig& config);
```
`Upsert` inserts `key` if it does not exist and updates the existing object if
it does. It uses the same replication configuration model as `Put`, while
allowing the store to reuse existing placement for in-place updates when the
current layout permits it. `BatchUpsert` performs the same operation for
multiple keys using a shared replication configuration.
### Remove
```C++
tl::expected<void, ErrorCode> Remove(const ObjectKey& key);
```
Used to delete the object corresponding to the specified key. This interface marks all data replicas associated with the key in the storage engine as deleted, without needing to communicate with the corresponding storage node (Client).
### CreateCopyTask
```C++
tl::expected<UUID, ErrorCode> CreateCopyTask(
const std::string& key,
const std::vector<std::string>& targets);
```
![mooncake-store-create-copy-task](../image/mooncake-store-client-create-copy-task.png)
`CreateCopyTask` creates an asynchronous copy task that will be executed by the client's task execution system. This is useful when you want to submit multiple copy operations without waiting for each one to complete. The task is submitted to the master service, assigned a unique task ID, and executed asynchronously by an available client. The task status can be queried using `QueryTask`.
**Task Execution and Result Reporting:**
1. **Task Assignment**: The master service assigns the task to an available client during the client's periodic ping operation
2. **Task Execution**: The assigned client executes the copy operation asynchronously in a background thread pool
3. **Result Reporting**: Upon completion (success or failure), the client automatically reports the result to the master service via `MarkTaskToComplete`:
- On success: `status = SUCCESS`, `message = "Task completed successfully"`
- On failure: `status = FAILED`, `message = <error description>`
4. **Status Query**: You can query the task status at any time using `QueryTask` to monitor progress
### CreateMoveTask
```C++
tl::expected<UUID, ErrorCode> CreateMoveTask(
const std::string& key,
const std::string& source,
const std::string& target);
```
![mooncake-store-create-move-task](../image/mooncake-store-client-create-move-task.png)
`CreateMoveTask` creates an asynchronous move task that will be executed by the client's task execution system. This is useful when you want to submit multiple move operations without waiting for each one to complete. The task is submitted to the master service, assigned a unique task ID, and executed asynchronously by an available client. The task status can be queried using `QueryTask`.
**Task Execution and Result Reporting:**
1. **Task Assignment**: The master service assigns the task to an available client during the client's periodic ping operation
2. **Task Execution**: The assigned client executes the move operation asynchronously in a background thread pool
3. **Result Reporting**: Upon completion (success or failure), the client automatically reports the result to the master service via `MarkTaskToComplete`:
- On success: `status = SUCCESS`, `message = "Task completed successfully"`
- On failure: `status = FAILED`, `message = <error description>`
4. **Status Query**: You can query the task status at any time using `QueryTask` to monitor progress
### QueryTask
```C++
tl::expected<QueryTaskResponse, ErrorCode> QueryTask(const UUID& task_id);
```
`QueryTask` queries the status of an asynchronous task (copy or move). This allows you to monitor the progress of task-based operations. The response includes task status, type, creation time, last update time, assigned client, and status message.
The data structure details of `QueryTaskResponse` are as follows:
```C++
struct QueryTaskResponse {
UUID id; // Task UUID
TaskType type; // Task type (REPLICA_COPY or REPLICA_MOVE)
TaskStatus status; // Task status (PENDING, PROCESSING, SUCCESS, or FAILED)
int64_t created_at_ms_epoch; // Task creation timestamp in milliseconds
int64_t last_updated_at_ms_epoch; // Last update timestamp in milliseconds
UUID assigned_client; // UUID of the client assigned to execute the task
std::string message; // Status message or error description
};
```
### BatchQueryIp
```C++
tl::expected<std::unordered_map<UUID, std::vector<std::string>, boost::hash<UUID>>, ErrorCode>
BatchQueryIp(const std::vector<UUID>& client_ids);
```
Used to batch query the IP addresses for multiple client IDs. For each client ID in the input list, this interface retrieves the unique IP addresses from all segments mounted by that client. The operation is performed on the Master Service and returns a map from client ID to their IP address lists. Only client IDs that have successfully mounted segments are included in the result map. This is useful for discovering the network locations of storage nodes in the cluster.
### BatchReplicaClear
```C++
tl::expected<std::vector<std::string>, ErrorCode>
BatchReplicaClear(const std::vector<std::string>& object_keys,
const UUID& client_id,
const std::string& segment_name);
```
Used to batch clear replicas for multiple object keys belonging to a specific client ID. This interface allows clearing replicas either on a specific segment or across all segments. If segment_name is empty, all replicas of the specified objects are cleared (the objects are deleted entirely). If segment_name is provided, only replicas located on that specific segment are cleared. The operation is performed on the Master Service and returns a list of object keys that were successfully cleared. Only objects that belong to the specified `client_id`, have expired leases, and meet the clearing criteria are processed. This is useful for managing storage resources and cleaning up data on specific storage nodes.
### QueryByRegex
```C++
tl::expected<std::unordered_map<std::string, std::vector<Replica::Descriptor>>, ErrorCode>
QueryByRegex(const std::string& str);
```
Used to query the replica information for all objects whose keys match the given regular expression. This is useful for batch operations or for retrieving a group of related objects. The operation is performed on the Master and returns a map of keys to their replica lists.
### RemoveByRegex
```C++
tl::expected<long, ErrorCode> RemoveByRegex(const ObjectKey& str);
```
Used to delete all objects from the store whose keys match the specified regular expression. This provides a powerful way to perform bulk deletions. The command returns the number of objects that were successfully removed.
### Master Service
## Master Service
The cluster's available resources are viewed as a large resource pool, managed centrally by a Master process for space allocation and guiding data replication
**Note: The Master Service does not take over any data flow, only providing corresponding metadata information.**
#### Snapshot & Restore
### Snapshot & Restore
To reduce cache warm-up time after a master restart, the Master Service supports periodic snapshots of its in-memory metadata and recovery from these snapshots.
@ -255,7 +91,47 @@ To reduce cache warm-up time after a master restart, the Master Service supports
>
> The snapshot storage location is **exclusively managed** by the Mooncake snapshot system. Old snapshots are automatically deleted during cleanup. **DO NOT store other files in this location.** Use a dedicated, isolated storage for snapshots.
#### Master Service APIs
### Tenant Quota
The Master Service can optionally enforce strict multi-tenant memory quota admission. This feature is disabled by default. When `enable_multi_tenants=false`, request tenant IDs are ignored for object placement, all objects use the `default` namespace, and tenant quota management requests return `UNAVAILABLE_IN_CURRENT_MODE`.
When strict multi-tenant mode is enabled, the tenant quota policy is loaded from the configured connector. The v1 connector is a writable YAML file configured by `tenant_quota_connector_type=file` and `tenant_quota_connector_uri=<path>`. Tenants must be explicitly present in that connector policy before they can write. Missing tenants, empty tenants, and an unregistered `default` tenant are rejected with `TENANT_NOT_REGISTERED`.
The YAML policy uses schema version `1`:
```yaml
version: 1
tenants:
- name: tenant-a
quota: 200GB
```
Tenant names must be non-empty, unique, must not start with `_`, and must not contain NUL or control characters. Quotas must be positive integers and may use `B`, `KB`, `MB`, `GB`, or `TB` units.
Effective quota is recomputed from the current registered memory capacity:
- If explicit tenant requests fit within the registered memory capacity, tenants receive their requested quotas and remaining capacity stays unallocated.
- If explicit tenant requests exceed registered memory capacity, explicit tenants receive quota scaled proportionally by request size.
- Remainders are assigned deterministically by tenant ID, so repeated recomputes produce stable results.
- Tenants present in restored metadata but missing from the connector policy become in-memory orphans with requested quota `0`, effective quota `0`, and `over_quota=true` while they still own metadata. Reads and removals are allowed so operators can clean them up; writes remain blocked until the tenant is re-registered or emptied.
`PutStart` and size-changing `UpsertStart` charge quota before memory is allocated. If the first reservation fails, the master performs tenant-scoped memory eviction for the target tenant and retries the reservation. The retry is bounded to two eviction attempts. Tenant quota eviction scans only the target tenant, skips hard-pinned objects, honors soft-pin eviction configuration, and preserves grouped-object lease safety checks.
Admin policy changes are persisted before the final in-memory policy is applied. `PUT` writes the connector first and then applies the policy in memory. `DELETE` first marks the tenant unregistered in memory to block concurrent writes, verifies the tenant is empty, writes the connector, and rolls back the in-memory mark if the connector write fails. The admin HTTP API exposes:
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/tenant_quotas` | List quota snapshots for active or explicit tenants |
| `GET` | `/api/v1/tenant_quotas?tenant_id=<tenant>` | Query one tenant quota snapshot |
| `PUT` | `/api/v1/tenant_quotas?tenant_id=<tenant>` | Create or update a tenant quota policy |
| `DELETE` | `/api/v1/tenant_quotas?tenant_id=<tenant>` | Delete an empty tenant quota policy |
Tenant quota snapshots include `tenant_id`, `requested_quota_bytes`, `effective_quota_bytes`, `used_bytes`, `reserved_bytes`, `committed_count`, `metadata_object_count`, `over_quota`, and `has_explicit_policy`.
Snapshots restore object runtime state only. Tenant quota policy is always loaded from the connector after metadata restore, then usage and effective quota are rebuilt from restored metadata and current registered capacity. If the connector cannot be loaded in strict multi-tenant mode, startup fails.
### Master Service APIs
The protobuf definition between Master and Client is as follows:
@ -498,7 +374,7 @@ message UnMountSegmentResponse {
When the space needs to be released, this interface is used to remove the previously mounted resources from the Master Service.
#### Object Information Maintenance
### Object Information Maintenance
The Master Service needs to maintain mappings related to buffer allocators and object metadata to efficiently manage memory resources and precisely control replica states in multi-replica scenarios. Additionally, the Master Service uses read-write locks to protect critical data structures, ensuring data consistency and security in multi-threaded environments. The following are the interfaces maintained by the Master Service for storage space information:
@ -590,7 +466,25 @@ tl::expected<long, ErrorCode> RemoveByRegex(const std::string& str);
The Client requests the Master Service to delete all replicas corresponding to the specified key or for all object keys that match the specified regular expression.
### Buffer Allocator
### Optional Object Groups
Object groups are intended for workloads where one logical cache entry is split into multiple Mooncake Store objects, such as separate K/V tensors, parallel shards, or auxiliary index objects. Without grouping, these objects are managed independently, so lease refresh and memory eviction may affect different parts of the same logical entry at different times.
Mooncake Store remains an object-oriented KV cache: objects are still put, queried, and removed by key. For workloads where one logical cache unit is represented by multiple physical objects, callers may attach optional group metadata through `ReplicateConfig::group_ids` during `Put`, `BatchPut`, `Upsert`, or `BatchUpsert`.
For single-object writes, `group_ids` contains one entry. For batch writes, it must have the same length as the key list, and entry `i` is the group ID for key `i`. An empty string stores that key as ungrouped, and leaving the field unset preserves the legacy ungrouped behavior. For an existing object, group membership is immutable: `Upsert` may preserve the existing group, but it cannot move the object to another group or clear its group while the object exists.
On the Master side, group state is tenant-scoped. Objects with a non-empty group ID are routed to the metadata shard selected by `hash(group_id)`, and the Master keeps a tenant-scoped object-to-group routing index so existing key-based APIs can still locate grouped objects. The Master tracks only the current member set of each group; it does not require an expected member count, a member index, or a commit protocol for group completeness.
Group metadata affects lifecycle behavior on a best-effort basis:
- `ExistKey` and `GetReplicaList` refresh the lease, and the soft-pin timeout if present, for the current members of the group.
- Memory eviction expands a grouped candidate to the group's current members and then applies the existing per-object safety checks. Members with active leases, hard pins, soft pins when soft-pin eviction is disabled, incomplete writes, busy replicas, or unavailable replica states are skipped.
- Object removal APIs, copy/move tasks, and NoF eviction keep their existing object-level semantics. Group routing and membership metadata are cleaned up when objects are removed.
This design is intentionally lightweight and backward compatible. Grouping should be treated as a lifecycle hint for related objects, not as a transactional guarantee that all members are created, made visible, or evicted atomically.
## Buffer Allocator
The buffer allocator serves as a low-level memory management component within the Mooncake Store system, primarily responsible for efficient memory allocation and deallocation. It builds upon underlying memory allocators to perform its functions.
@ -600,6 +494,8 @@ Mooncake Store provides two concrete implementations of `BufferAllocatorBase`:
**OffsetBufferAllocator (default and recommended)**: This allocator is derived from [OffsetAllocator](https://github.com/sebbbi/OffsetAllocator), which uses a custom bin-based allocation strategy that supports fast hard realtime `O(1)` offset allocation with minimal fragmentation. Mooncake Store optimizes this allocator based on the specific memory usage characteristics of LLM inference workloads, thereby enhancing memory utilization in LLM scenarios.
For measured utilization and allocation latency across LLM-style workloads, see [Allocator Performance](../performance/allocator-benchmark-result.md).
**CachelibBufferAllocator (deprecated)**: This allocator leverages Facebook's [CacheLib](https://github.com/facebook/CacheLib) to manage memory using a slab-based allocation strategy. It provides efficient memory allocation with good fragmentation resistance and is well-suited for high-performance scenarios. However, in our modified version, it does not handle workloads with highly variable object sizes effectively, so it is currently marked as deprecated.
Users can choose the allocator that best matches their performance and memory usage requirements through the `--memory-allocator` startup parameter of `master_service`.
@ -620,7 +516,15 @@ class BufferAllocatorBase {
3. **`deallocate` Function**: This function is automatically triggered by the `BufHandle` destructor. It calls the internal allocator to release the associated memory and updates the handle’s status to `BufStatus::UNREGISTERED`.
### AllocationStrategy
### Client Local Buffer and Python BufferPool
Each Store client can also create a setup-time local buffer through `local_buffer_size`. This memory is registered once with the Transfer Engine and managed by `ClientBufferAllocator` for short-lived client-side staging work.
The Python `BufferPool` reuses this existing local buffer instead of allocating a second registered arena. A pool lease is a sub-allocation from `client_buffer_allocator_`, so the common path avoids per-lease `register_buffer()` and `unregister_buffer()` calls. The pool still keeps the Python-facing lease API, memoryview lifetime checks, blocking acquire semantics, and optional `max_regions` concurrency limiting.
This is a soft-isolation policy: internal Store paths and external Python leases share the local registered buffer, allowing bursty external usage when memory is available rather than reserving a hard partition. If the local buffer is temporarily exhausted, `BufferPool` can allocate and register a short-lived overflow buffer so bursts do not immediately surface as upper-layer errors; that overflow region is unregistered as soon as the lease is released. If callers need to cap long-lived external pressure, they should use pool-level controls such as `max_regions`, `max_bytes`, or acquire timeouts.
## AllocationStrategy
AllocationStrategy is a strategy class for efficiently managing memory resource allocation and replica storage location selection in a distributed environment. It is mainly used in the following scenarios:
- Determining the allocation locations for object storage replicas.
- Selecting suitable read/write paths among multiple replicas.
@ -630,7 +534,7 @@ AllocationStrategy is used in conjunction with the Master Service and the underl
- Master Service: Determines the target locations for replica allocation via `AllocationStrategy`.
- Buffer Allocator: Executes the actual memory allocation and release tasks.
#### APIs
### APIs
`Allocate`: Finds suitable storage segments from available storage resources to allocate space of a specified size for multiple replicas. Uses best-effort semantics, meaning it allocates as many replicas as possible even if the full requested count cannot be satisfied.
@ -652,7 +556,7 @@ virtual tl::expected<std::vector<Replica>, ErrorCode> Allocate(
- On success: vector of allocated replicas (may be fewer than requested due to resource constraints, but at least 1)
- On failure: ErrorCode::NO_AVAILABLE_HANDLE if no replicas can be allocated, ErrorCode::INVALID_PARAMS for invalid configuration
#### Allocation Strategies
### Allocation Strategies
Mooncake Store provides multiple built-in allocation strategies to control how storage space is distributed across segments. Users can select a strategy via the `--allocation_strategy` flag when starting the master service:
@ -662,7 +566,7 @@ Mooncake Store provides multiple built-in allocation strategies to control how s
Valid values are: `random` (default), `free_ratio_first`, `cxl` (case-sensitive).
##### How to Choose
#### How to Choose
| Strategy | Best For | Trade-off |
|---|---|---|
@ -678,7 +582,9 @@ Valid values are: `random` (default), `free_ratio_first`, `cxl` (case-sensitive)
**Use `cxl`** only when your hardware includes CXL (Compute Express Link) memory devices and you want to allocate data exclusively on CXL segments.
##### Strategy Details
For benchmark data comparing `random` and `free_ratio_first` across segment counts, replica counts, and skewed capacities, see [AllocationStrategy Performance](../performance/allocation-strategy-benchmark-result.md).
#### Strategy Details
**`random` — RandomAllocationStrategy**
@ -715,21 +621,25 @@ Specialized for CXL (Compute Express Link) memory hardware. Unlike the other str
Limitations: This strategy only supports single-replica allocation (does not distribute across multiple segments) and does not support the `AllocateFrom()` interface.
### Eviction Policy
## Eviction Policy
When a `PutStart` request fails due to insufficient memory, or when the eviction thread detects that space usage has reached the configured high watermark (95% by default, configurable via `-eviction_high_watermark_ratio`), an eviction task is triggered to free up space by evicting a portion of objects (5% by default, configurable via `-eviction_ratio`). Similar to `Remove`, evicted objects are simply marked as deleted, with no data transfer required.
Currently, an approximate LRU policy is adopted, where the least recently used objects are preferred for eviction. To avoid data races and corruption, objects currently being read or written by clients should not be evicted. For this reason, objects that have leases or have not been marked as complete by `PutEnd` requests will be ignored by the eviction task.
### Lease
For grouped objects, memory eviction resolves the current group membership and attempts to reclaim eligible members together.
## Lease
To avoid data conflicts, a per-object lease is granted whenever an `ExistKey` request or a `GetReplicaListRequest` request succeeds. While the lease is active, the object is protected from `Remove`, `RemoveAll`, and `Eviction` operations. Specifically, a `Remove` request targeting a leased object will fail, and a `RemoveAll` request will only delete objects without an active lease. This ensures that the object’s data can be safely read as long as the lease has not expired.
For grouped objects, a successful `ExistKey` or `GetReplicaList` refreshes the lease for the current members of the group, so recently accessed members are less likely to be separated by memory eviction.
However, if the lease expires before a `Get` operation finishes reading the data, the operation will be considered failed, and no data will be returned, in order to prevent potential data corruption.
The default lease TTL is 5 seconds and is configurable via a startup parameter of `master_service`.
### Soft Pin
## Soft Pin
For important and frequently used objects, such as system prompts, Mooncake Store provides a soft pin mechanism. When putting an object, it can be configured to enable soft pin. During eviction, objects that are not soft pinned are prioritized for eviction. Soft pinned objects are only evicted when memory is insufficient and no other objects are eligible for eviction.
@ -743,7 +653,7 @@ There are two startup parameters in `master_service` related to the soft pin mec
Notably, soft pinned objects can still be removed using APIs such as `Remove` or `RemoveAll`.
### Hard Pin
## Hard Pin
For objects that must never be evicted under any circumstances (e.g., model weights, critical metadata), Mooncake Store provides a hard pin mechanism. Unlike soft pin, hard-pinned objects are permanently protected from eviction — they will never be selected as eviction candidates regardless of memory pressure.
@ -755,23 +665,23 @@ Key differences from soft pin:
- Hard-pinned objects are completely skipped during eviction. Soft-pinned objects may still be evicted when no other candidates are available.
- Hard pin is immutable once set. Soft pin status is automatically refreshed on access.
### Zombie Object Cleanup
## Zombie Object Cleanup
If a Client crashes or experiences a network failure after sending a `PutStart` request but before it can send the corresponding `PutEnd` or `PutRevoke` request to the Master, the object initiated by `PutStart` enters a "zombie" state—rendering it neither usable nor deletable. The existence of such "zombie objects" not only consumes storage space but also prevents subsequent `Put` operations on the same keys. To mitigate these issues, the Master records the start time of each `PutStart` request and employs two timeout thresholds—`put_start_discard_timeout` and `put_start_release_timeout`—to clean up zombie objects.
#### `PutStart` Preemption
### `PutStart` Preemption
If an object receives neither a `PutEnd` nor a `PutRevoke` request within `put_start_discard_timeout` (default: 30 seconds) after its `PutStart`, any subsequent `PutStart` request for the same object will be allowed to "preempt" the previous `PutStart`. This enables the new request to proceed with writing the object, thereby preventing a single faulty Client from permanently blocking access to that object. Note that during such preemption, the storage space allocated by the old `PutStart` is not reused; instead, new space is allocated for the preempting `PutStart`. The space previously allocated by the old `PutStart` will be reclaimed via the mechanism described below.
#### Space Reclaim
### Space Reclaim
Replica space allocated during a `PutStart` is considered releasable by the Master if the write operation is neither completed (via `PutEnd`) nor canceled (via `PutRevoke`) within `put_start_release_timeout` (default: 10 minutes) after the `PutStart`. When object eviction is triggered—either due to allocation failures or because storage utilization exceeds the configured threshold—these releasable replica spaces are prioritized for release to reclaim storage capacity.
### Preferred Segment Allocation
## Preferred Segment Allocation
Mooncake Store provides a **preferred segment allocation** feature that allows users to specify a preferred storage segment (node) for object allocation. This feature is particularly useful for optimizing data locality and reducing network overhead in distributed scenarios.
#### How It Works
### How It Works
The preferred segment allocation feature is implemented through the `AllocationStrategy` system and is controlled via the `preferred_segment` field in the `ReplicateConfig` structure:
@ -795,23 +705,23 @@ When a `Put` operation is initiated with a non-empty `preferred_segment` value,
- **Data Locality**: By preferring local segments, applications can reduce network traffic and improve access performance for frequently used data.
- **Load Balancing**: Applications can distribute data across specific nodes to achieve better load distribution.
### Multi-layer Storage Support
## Multi-layer Storage Support
This system provides support for a hierarchical cache architecture, enabling efficient data access through a combination of in-memory caching and persistent storage. Data is initially stored in memory cache and asynchronously backed up to a Distributed File System (DFS), forming a two-tier "memory-SSD persistent storage" cache structure.
#### Enabling Persistence Functionality
### Enabling Persistence Functionality
When the user specifies `--root_fs_dir=/path/to/dir` when starting the master, and this path is a valid DFS-mounted directory on all machines where the clients reside, Mooncake Store's tiered caching functionality will work properly. Additionally, during master initialization, a `cluster_id` is loaded. This ID can be specified during master initialization (`--cluster_id=xxxx`). If not specified, the default value `mooncake_cluster` will be used. Subsequently, the root directory for client persistence will be `<root_fs_dir>/<cluster_id>`.
​Note​​: When enabling this feature, the user must ensure that the DFS-mounted directory (`root_fs_dir=/path/to/dir`) is valid and consistent across all client hosts. If some clients have invalid or incorrect mount paths, it may cause abnormal behavior in Mooncake Store.
#### Persistent Storage Space Configuration​
### Persistent Storage Space Configuration​
Mooncake provides configurable DFS available space. Users can specify `--global_file_segment_size=1048576` when starting the master, indicating a maximum usable space of 1MB on DFS.
The current default setting is the maximum value of int64 (as we generally do not restrict DFS storage usage), which is displayed as `infinite` in `mooncake_maseter`'s console logs.
**Notice** The DFS cache space configuration must be used together with the `--root_fs_dir` parameter. Otherwise, you will observe that the `SSD Storage` usage consistently shows: `0 B / 0 B`
**Notice** The capability for file eviction on DFS has not been provided yet
#### Data Access Mechanism
### Data Access Mechanism
The persistence feature also follows Mooncake Store's design principle of separating control flow from data flow. The read/write operations of kvcache objects are completed on the client side, while the query and management functions of kvcache objects are handled on the master side. In the file system, the key -> kvcache object index information is maintained by a fixed indexing mechanism, with each file corresponding to one kvcache object (the filename serves as the associated key name).
@ -820,24 +730,33 @@ After enabling the persistence feature:
- For each `Put` or `BatchPut` operation, both a synchronous memory pool write operation and an asynchronous DFS persistence operation will be initiated.
- For each `Get` or `BatchGet` operation, if the corresponding kvcache is not found in the memory pool, the system will attempt to read the file data from DFS and return it to the user.
#### 3FS USRBIO Plugin
### 3FS USRBIO Plugin (Experimental)
```{note}
This integration is **experimental** and incomplete; see the plugin page for details before relying on it.
```
If you need to use 3FS's native API (USRBIO) to achieve high-performance persistent file reads and writes, you can refer to the configuration instructions in this document [3FS USRBIO Plugin](../getting_started/plugin-usage/3FS-USRBIO-Plugin.md).
### Builtin Metadata Server
## Builtin Metadata Server
Mooncake Store provides a built-in HTTP metadata server as an alternative to etcd for storing cluster metadata. This feature is particularly useful for development environments or scenarios where etcd is not available.
#### Configuration Parameters
### Configuration Parameters
The HTTP metadata server can be configured using the following parameters:
- **`enable_http_metadata_server`** (boolean, default: `false`): Enables the built-in HTTP metadata server instead of using etcd. When set to `true`, the master service will start an embedded HTTP server that handles metadata operations.
- **`http_metadata_server_port`** (integer, default: `8080`): Specifies the TCP port on which the HTTP metadata server will listen for incoming connections. This port must be available and not conflict with other services.
- **`http_metadata_server_host`** (string, default: `"0.0.0.0"`): Specifies the host address for the HTTP metadata server to bind to. Use `"0.0.0.0"` to listen on all available network interfaces, or specify a specific IP address for security purposes.
#### Environment Variables
### Environment Variables
- MC_STORE_CLUSTER_ID: Identify the metadata when multiple cluster share the same master, default 'mooncake'.
- MC_STORE_MEMCPY: Enables or disables local memcpy optimization, set to 1/true to enable, 0/false to disable.
- MC_STORE_CLIENT_METRIC: Enables client metric reporting, enabled by default; set to 0/false to disable.
- MC_STORE_CLIENT_METRIC_INTERVAL: Reporting interval in seconds, default 0 (collects but does not report).
- MC_STORE_CLIENT_MIN_PORT: Minimum local port for client connections (default 12300). Must be in range 1024–32767 or 61000–65535; falls back to default on invalid input.
- MC_STORE_CLIENT_MAX_PORT: Maximum local port for client connections (default 14300). Same range constraints; must be ≥ MC_STORE_CLIENT_MIN_PORT.
- MC_STORE_USE_HUGEPAGE: Enables huge page support, disabled by default.
- MC_STORE_HUGEPAGE_SIZE: Specifies the page size of the huge page to use, default 2M.
#### Usage Example
- MC_MMAP_ARENA_POOL_SIZE: Size of the pre-allocated arena pool for mmap buffer allocations. Accepts human-readable sizes (e.g., `"8gb"`, `"20gb"`). Providing this variable explicitly enables the arena; when enabled via gflag without an env override, the default pool size is `8gb`. The arena is allocated once at first use and serves subsequent allocations via lock-free atomic bump pointer (~50ns per allocation vs ~1000ns for direct mmap).
- MC_DISABLE_MMAP_ARENA: Set to `1` to disable the arena allocator and fall back to per-call `mmap()`, even if the arena was explicitly requested. Also accepts `true`, `yes`, or `on`. This must be set before the first Mooncake mmap-buffer allocation in the process. Useful for debugging or memory-constrained environments where pre-allocating a pool is not desirable.
### Usage Example
To start the master service with the HTTP metadata server enabled:
```bash
./build/mooncake-store/src/mooncake_master \
@ -848,195 +767,15 @@ To start the master service with the HTTP metadata server enabled:
When enabled, the HTTP metadata server will start automatically and provide metadata services for the Mooncake Store cluster. This eliminates the need for an external etcd deployment, simplifying the setup process for development and testing environments.
Note that the HTTP metadata server is designed for single-node deployments and does not provide the high availability features that etcd offers. For production environments requiring high availability, etcd is still the recommended choice.
For detailed guidance on monitoring master metrics, Prometheus endpoints, and health checks, see the [Observability guide](../getting_started/observability.md).
## Mooncake Store Python API
**Complete Python API Documentation**: [https://kvcache-ai.github.io/Mooncake/python-api-reference/mooncake-store.html](https://kvcache-ai.github.io/Mooncake/python-api-reference/mooncake-store.html)
## Compilation and Usage
Mooncake Store is compiled together with other related components (such as the Transfer Engine).
For default mode:
```
mkdir build && cd build
cmake .. # default mode
make
sudo make install # Install Python interface support package
```
High availability mode:
```
mkdir build && cd build
cmake .. -DSTORE_USE_ETCD # compile etcd wrapper that depends on go
make
sudo make install # Install Python interface support package
```
**Note:** To use high availability mode, only `-DSTORE_USE_ETCD` is required. `-DUSE_ETCD` is a compilation option for the **Transfer Engine** and is **not related** to the high availability mode.
### Starting the Transfer Engine's Metadata Service
Mooncake Store uses the Transfer Engine as its core transfer engine, so it is necessary to start the metadata service (etcd/redis/http). The startup and configuration of the `metadata` service can be referred to in the relevant sections of [Transfer Engine](./transfer-engine/index.md). **Special Note**: For the etcd service, by default, it only provides services for local processes. You need to modify the listening options (IP to 0.0.0.0 instead of the default 127.0.0.1). You can use commands like curl to verify correctness.
### Starting the Master Service
The Master Service runs as an independent process, provides gRPC interfaces externally, and is responsible for the metadata management of Mooncake Store (note that the Master Service does not reuse the metadata service of the Transfer Engine). The default listening port is `50051`. After compilation, you can directly run `mooncake_master` located in the `build/mooncake-store/src/` directory. After starting, the Master Service will output the following content in the log:
```
Starting Mooncake Master Service
Port: 50051
Max threads: 4
Master service listening on 0.0.0.0:50051
```
**High availability mode**:
HA mode relies on an etcd service for coordination. If Transfer Engine also uses etcd as its metadata service, the etcd cluster used by Mooncake Store can either be shared with or separate from the one used by Transfer Engine.
HA mode allows deployment of multiple master instances to eliminate the single point of failure. Each master instance must be started with the following parameters:
```
--enable-ha: enables high availability mode
--etcd-endpoints: specifies endpoints for etcd service, separated by ';'
--rpc-address: the RPC address of this instance. Note that the address specified here should be accessible to the client.
```
For example:
```
./build/mooncake-store/src/mooncake_master \
--enable-ha=true \
--etcd-endpoints="0.0.0.0:2379;0.0.0.0:2479;0.0.0.0:2579" \
--rpc-address=10.0.0.1
```
### Starting the Sample Program
Mooncake Store provides various sample programs, including interface forms based on C++ and Python. Below is an example of how to run using `stress_cluster_benchmark`.
1. Open `stress_cluster_benchmark.py` and update the initialization settings based on your network environment. Pay particular attention to the following fields:
`local_hostname`: the IP address of the local machine
`metadata_server`: the address of the Transfer Engine metadata service
`master_server_address`: the address of the Master Service
**Note**: The format of `master_server_address` depends on the deployment mode. In default mode, use the format `IP:Port`, specifying the address of a single master node. In HA mode, use the format `etcd://IP:Port;IP:Port;...;IP:Port`, specifying the addresses of the etcd cluster endpoints.
For example:
```python
import os
import time
from distributed_object_store import DistributedObjectStore
store = DistributedObjectStore()
# Protocol used by the transfer engine, optional values are "rdma" or "tcp"
protocol = os.getenv("PROTOCOL", "tcp")
# Device name used by the transfer engine
device_name = os.getenv("DEVICE_NAME", "ibp6s0")
# Hostname of this node in the cluster, port number is randomly selected from (12300-14300)
local_hostname = os.getenv("LOCAL_HOSTNAME", "localhost")
# Metadata service address of the Transfer Engine, here etcd is used as the metadata service
metadata_server = os.getenv("METADATA_ADDR", "127.0.0.1:2379")
# The size of the Segment mounted by each node to the cluster, allocated by the Master Service after mounting, in bytes
global_segment_size = 3200 * 1024 * 1024
# Local buffer size registered with the Transfer Engine, in bytes
local_buffer_size = 512 * 1024 * 1024
# Address of the Master Service of Mooncake Store
master_server_address = os.getenv("MASTER_SERVER", "127.0.0.1:50051")
# Data length for each put()
value_length = 1 * 1024 * 1024
# Total number of requests sent
max_requests = 1000
# Initialize Mooncake Store Client
retcode = store.setup(
local_hostname,
metadata_server,
global_segment_size,
local_buffer_size,
protocol,
device_name,
master_server_address,
)
```
2. Run `ROLE=prefill python3 ./stress_cluster_benchmark.py` on one machine to start the Prefill node.
For "rdma" protocol, you can also enable topology auto discovery and filters, e.g., `ROLE=prefill MC_MS_AUTO_DISC=1 MC_MS_FILTERS="mlx5_1,mlx5_2" python3 ./stress_cluster_benchmark.py`.
3. Run `ROLE=decode python3 ./stress_cluster_benchmark.py` on another machine to start the Decode node.
For "rdma" protocol, you can also enable topology auto discovery and filters, e.g., `ROLE=decode MC_MS_AUTO_DISC=1 MC_MS_FILTERS="mlx5_1,mlx5_2" python3 ./stress_cluster_benchmark.py`.
The absence of error messages indicates successful data transfer.
### Starting the Client as Standalone Process and accessing via RPC
To start a RPC type **real** `Client` as a standalone process, you can use the following command:
```bash
./build/mooncake-store/src/mooncake_client \
--global_segment_size="4GB" \
--master_server_address="localhost:50051" \
--metadata_server="http://localhost:8080/metadata"
```
Next, a **real** `Client` instance is created and connected to the Master Service. The **real** `Client` instance is listening on port 50052 as default.
If you want to send requests to it, a **dummy** `Client` should be used in the application process (e.g., vLLM, SGLang). You can start a **dummy** `Client`
with specific parameters defined in the application.
The **real** `Client` can be configured using the following parameters:
- **`host`**: (string, default: "0.0.0.0"): The hostname of the client.
- **`port`**: (int, default: 50052): The port number the client service listens on.
- **`global_segment_size`**: (string, default: "4GB"): The size of the global segment to be allocated by the client.
- **`master_server_address`**: (string, default: "localhost:50051"): The address of the Master Service.
- **`metadata_server`**: (string, default: "http://localhost:8080/metadata"): The address of the metadata service.
- **`protocol`**: (string, default: "tcp"): The protocol used by the Transfer Engine.
- **`device_name`**: (string, default: ""): The device name used by the Transfer Engine.
- **`threads`**: (int, default: 1): The number of threads used by the client.
### Starting the Client as Standalone Process and accessing via HTTP
Use `mooncake-wheel/mooncake/mooncake_store_service.py` to start a **real** `Client` as a standalone process and accessing via HTTP.
First, create and save a configuration file in JSON format. For example:
```
{
"local_hostname": "localhost",
"metadata_server": "http://localhost:8080/metadata",
"global_segment_size": 268435456,
"local_buffer_size": 268435456,
"protocol": "tcp",
"device_name": "",
"master_server_address": "localhost:50051"
}
```
Then run `mooncake_store_service.py`. This program starts an HTTP server alongside the **real** `Client`. Through this server, users can manually perform operations such as `Get` and `Put`, which is useful for debugging.
The main startup parameters include:
* `config`: Path to the configuration file.
* `port`: Port number for the HTTP server.
Suppose the `mooncake_transfer_engine` wheel package is already installed, the following command starts the program:
```bash
python -m mooncake.mooncake_store_service --config=[config_path] --port=8081
```
### Set the Log Level for yalantinglibs coro_rpc and coro_http
By default, the log level is set to warning. You can customize it using the following environment variable:
`export MC_YLT_LOG_LEVEL=info`
This sets the log level for yalantinglibs (including coro_rpc and coro_http) to info.
Available log levels: trace, debug, info, warn (or warning), error, and critical.
## Example Code
#### Python Usage Example
We provide a reference example `distributed_object_store_provider.py`, located in the `mooncake-store/tests` directory. To check if the related components are properly installed, you can run etcd and Master Service (`mooncake_master`) in the background on the same server, and then execute this Python program in the foreground. It should output a successful test result.
#### C++ Usage Example
The C++ API of Mooncake Store provides more low-level control capabilities. We provide a reference example `client_integration_test`, located in the `mooncake-store/tests` directory. To check if the related components are properly installed, you can run etcd and Master Service (`mooncake_master`) on the same server, and then execute this C++ program (located in the `build/mooncake-store/tests` directory). It should output a successful test result.
## Version Management Policy
The current version of Mooncake Store is defined in [`CMakeLists.txt`](../../mooncake-store/CMakeLists.txt) as `project(MooncakeStore VERSION 2.0.0)`.
The current version of Mooncake Store is defined in [`CMakeLists.txt`](gh-file:mooncake-store/CMakeLists.txt) as `project(MooncakeStore VERSION 2.0.0)`.
When to bump the version:

152
docs/source/design/ssd-free-ratio-first-allocation.md Normal file
View File

@ -0,0 +1,152 @@
# SSD Free-Ratio-First Allocation Design
## Overview
Mooncake Store distributes KV cache objects across multiple memory segments hosted on different nodes. The master's allocation strategy decides which segment receives each new object replica. When using DDR-only strategies such as `random` or `free_ratio_first`, the allocator ignores SSD state entirely. In deployments where some segments have SSD offload enabled and others do not, this blind allocation can concentrate traffic on a small subset of segments whose SSD capacity is quickly exhausted, while segments with ample SSD headroom remain underutilized.
This document describes `SsdFreeRatioFirstAllocationStrategy`, an allocation strategy that ranks candidate segments by their SSD free ratio and preferentially allocates to segments with the most available SSD space. It integrates with the existing allocation framework and adds SSD usage tracking to `LocalDiskSegment` so that the master can make informed placement decisions.
---
## Architecture
```
Allocation Request
│
▼
┌───────────────────────┐
│ Sample candidate │
│ segments (up to │
│ 6 * replica_num) │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Compute SSD free │
│ ratio per candidate │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Sort by SSD free │
│ ratio descending │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Allocate from top │
│ candidates │
└───────────┬───────────┘
│
┌──────────┴──────────┐
│ Remaining replicas? │
└──────┬──────┬───────┘
│ │
Yes ◄┘ └► No → Done
│
▼
┌───────────────────────┐
│ Fallback: random │
│ allocation for │
│ remaining replicas │
└───────────────────────┘
```
The flow follows the same high-level structure as the existing `FreeRatioFirstAllocationStrategy`: sample a subset of candidates, compute a ranking metric, sort, and allocate from the top. The key difference is that the ranking metric is SSD free ratio rather than DRAM free ratio.
`MasterService` only passes an `SsdMetricsProvider` when the effective allocation strategy is `SSD_FREE_RATIO_FIRST`. Non-SSD strategies receive `nullptr`, so they avoid unnecessary local-disk segment access.
---
## Core Algorithm
### Candidate sampling
For each allocation request requesting `replica_num` replicas, the strategy samples `min(6 * replica_num, total_segments)` candidate segments. This bounded sampling keeps the sorting cost predictable regardless of cluster size while still providing a statistically diverse candidate set.
### SSD free ratio
For each sampled segment, the SSD free ratio is computed as:
```
ssd_free_ratio = (ssd_total_capacity - ssd_used_bytes) / ssd_total_capacity
```
A segment with 1 TB total SSD capacity and 200 GB used has an SSD free ratio of 0.80. A segment whose SSD is full has a ratio of 0.0.
Before calculating the ratio, `ssd_used_bytes` is clamped to `[0, ssd_total_capacity]`. This keeps transient concurrent accounting drift from producing a negative free ratio or a value greater than 1.0. If no SSD metrics provider is available, or if the reported total capacity is not positive, the strategy treats the segment as fully free.
### Sorting
Candidates are sorted by SSD free ratio in descending order. Segments with more available SSD space appear first and are preferred for allocation.
### Preferred segments
As with other allocation strategies, segments marked as preferred by the caller are handled first. Preferred segments bypass the SSD free ratio ranking and are allocated immediately if they have sufficient capacity.
### Fallback to random allocation
After allocating from the SSD-ranked candidates, any remaining replicas that could not be satisfied are allocated using the standard random strategy as a fallback. This ensures that allocation succeeds even when SSD metrics are unavailable (for example, on segments without SSD offload configured).
---
## SSD Usage Tracking
### `ssd_used_bytes` counter
`LocalDiskSegment` maintains `ssd_total_capacity_bytes`, updated by `ReportSsdCapacity`, and an atomic counter `ssd_used_bytes` that tracks the total number of bytes currently occupied by offloaded replicas on the segment's SSD. `ssd_used_bytes` is updated alongside metadata changes:
- **Increment**: `NotifyOffloadSuccess` increments `ssd_used_bytes` by the object size only after the master successfully adds a `LOCAL_DISK` replica to the object entry. If the object has already disappeared from metadata, the notification is ignored and the counter is not changed.
- **Decrement**: The master decrements `ssd_used_bytes` via `ReleaseLocalDiskUsage` whenever a `LOCAL_DISK` replica is removed from metadata — on full object deletion (`EraseMetadata`), and on partial replica removal through the shared erase helper (`EraseReplicasWithCacheTotalAccounting`, used by batch clear, revoke, and stale-handle cleanup) and the local-disk eviction path. `ReleaseLocalDiskUsage` iterates only `LOCAL_DISK` replicas and is a no-op for other replica types, so it is safe to call on any replica set.
The counter is atomic to allow concurrent updates from multiple RPC handler threads without requiring a separate lock. The allocation strategy treats it as an eventually consistent placement signal and clamps it before computing the free ratio.
### `SsdMetricsProvider` interface
`ScopedLocalDiskSegmentAccess` implements the `SsdMetricsProvider` interface, which exposes two methods:
| Method | Return type | Description |
|--------|-------------|-------------|
| `getSsdTotalCapacity` | `int64_t` | Total SSD capacity configured for the segment, in bytes |
| `getSsdUsedBytes` | `int64_t` | Current SSD usage, read from `ssd_used_bytes` |
The allocation strategy queries these methods through the `SsdMetricsProvider` interface, keeping the strategy decoupled from the concrete segment implementation.
---
## Configuration Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `allocation_strategy` | string | `"random"` | Set to `"ssd_free_ratio_first"` to enable SSD-ratio-based load balancing |
The parameter is passed as a gflag to the master process at startup.
---
## Code Structure
| File | Change |
|------|--------|
| `mooncake-store/include/types.h` | Add `SSD_FREE_RATIO_FIRST` enum value to the allocation strategy enum |
| `mooncake-store/include/allocation_strategy.h` | Add `SsdMetricsProvider` interface and `SsdFreeRatioFirstAllocationStrategy` class |
| `mooncake-store/include/segment.h` | Add `ssd_total_capacity_bytes` and `ssd_used_bytes` fields to `LocalDiskSegment`; inherit `SsdMetricsProvider` |
| `mooncake-store/src/segment.cpp` | Implement `getSsdTotalCapacity` and `getSsdUsedBytes` |
| `mooncake-store/src/master_service.cpp` | Pass SSD metrics only to `SSD_FREE_RATIO_FIRST`; update `ssd_used_bytes` after successful `NotifyOffloadSuccess` metadata insertion; release usage when `LOCAL_DISK` replicas are erased |
---
## Usage Example
Start the master with the SSD free-ratio-first strategy:
```bash
./mooncake_master --allocation_strategy=ssd_free_ratio_first
```
With this configuration:
1. The master samples up to `6 * replica_num` candidate segments for each allocation request.
2. Candidates are ranked by SSD free ratio (descending).
3. Allocation proceeds from the top-ranked candidates.
4. Any remaining replicas fall back to random allocation.

6
docs/source/design/ssd-offload.md
View File

@ -6,6 +6,8 @@ Mooncake Store supports offloading KV cache objects from distributed memory to l
SSD offload is implemented as a background subsystem within the **real client** process. It is transparent to the application: a `Put` that would otherwise be evicted from memory is persisted to disk, and a `Get` that finds no memory replica automatically falls back to reading from SSD.
For multi-turn conversation benchmark results, see [Mooncake SSD Offload Benchmark](../performance/ssd-offload-benchmark-results.md).
---
## Architecture
@ -242,4 +244,6 @@ To prevent `io_uring`'s `FOLL_LONGTERM` page pinning from failing on systems wit
## Metadata Recovery on Restart
On startup, `FileStorage::Init` calls `StorageBackend::ScanMeta`, which reads all on-disk metadata and invokes a callback for each discovered object. The callback calls `MasterClient::NotifyOffloadSuccess` to re-register the objects with the master. This restores the full disk-replica view without any application-level intervention.
On startup, `FileStorage::Init` calls `StorageBackend::ScanMeta`, which reads on-disk metadata and invokes a callback for each discovered object. The callback calls `MasterClient::NotifyOffloadSuccess` to re-register the objects with the master. This restores the full disk-replica view without any application-level intervention for the backends that preserve restart metadata, namely `BucketStorageBackend` and the file-per-key backend.
`OffsetAllocatorStorageBackend` is the exception. It truncates its pre-allocated data file during initialization and clears its in-memory metadata, so previously offloaded objects are not recoverable after a real client restart.

11
docs/source/design/tent/cpp-api.md
View File

@ -189,6 +189,9 @@ struct Request {
SegmentID target_id;
uint64_t target_offset;
size_t length;
int priority = PRIO_HIGH;
std::optional<std::string> policy_name;
TransportType transport_hint = UNSPEC;
};
```
@ -197,6 +200,9 @@ struct Request {
- `target_id`: Segment ID obtained from `openSegment`.
- `target_offset`: Offset within the target segment.
- `length`: Number of bytes to transfer.
- `priority`: Scheduling priority. Used by the QoS layer; see [qos.md](qos.md).
- `policy_name`: Optional. When set, the request matches the named entry instead of the first-matching policy.
- `transport_hint`: Optional. `UNSPEC` (default) defers to `TransportSelector`. Any other `TransportType` pins this request onto that transport for its first try.
#### TransferStatus
@ -522,7 +528,8 @@ Location strings identify device affinity: `"cpu:0"`, `"cuda:0"`, `"cuda:1"`, et
```cpp
enum TransportType {
RDMA = 0,
UNSPEC = 0,
RDMA,
MNNVL,
SHM,
NVLINK,
@ -530,7 +537,7 @@ enum TransportType {
IOURING,
TCP,
AscendDirect,
UNSPEC
SUNRISE_LINK,
};
```

197
docs/source/design/tent/failover.md Normal file
View File

@ -0,0 +1,197 @@
# TENT Failover
TENT hides transfer failures from the application by recovering inside the data path.
This document describes how the recovery works, which knobs control it, and how it is tested.
The design has two layers:
1. **Cross-transport failover** in `TransferEngineImpl`. When a transport fails a task at the completion stage, the engine moves that task to the next available transport (for example RDMA → TCP). Submit-stage failures are not retried today; see Known Gaps.
2. **Intra-RDMA rail recovery** in `RailMonitor`. When a specific (local NIC, remote NIC) rail keeps failing, the monitor pauses it with exponential cooldown; a successful transfer or the cooldown expiry brings it back.
Application code submits a batch and polls `getTransferStatus`. It never sees a `FAILED` task as long as any healthy path remains and the failover budget is not exhausted.
## Fault Model
TENT focuses on three kinds of transient faults:
| Fault | Surface | Recovery action |
|-------|---------|-----------------|
| Work request completion error (WC error) | RDMA worker sees a bad completion | Rail-level `markFailed` + task-level resubmit |
| QP / endpoint failure | `submitTransferTasks` returns non-OK | *Not retried today*: task surfaces as `FAILED`. See Known Gaps. |
| Peer disconnect mid-transfer | `getTransferStatus` returns `FAILED` | Cross-transport failover |
Permanent or application-visible errors (invalid arguments, out-of-memory, segment not found) are *not* retried; they are returned to the caller as-is.
## Architecture
```
+-------------------------+
submitTransfer | TransferEngineImpl |
---------------> classify by TransportType
| submitTransferTasks |----failure----+
+-------------------------+ |
| v
| resubmitTransferTask
| (bump priority, pick next
| transport, resubmit)
v
+---------------------------+
| RdmaTransport / workers |
| +---------------------+ |
| | RailMonitor | |
| | per-rail state | |
| | cooldown / recover | |
| +---------------------+ |
+---------------------------+
```
* Each request is owned by one `TaskInfo`. `type` names the transport currently executing the task; `xport_priority` is the index into the ranked fallback list; `failover_count` caps how many times we may re-resolve the transport.
* The ranked fallback list comes from `getTransportType(req, priority)`. Priority 0 yields the best available transport; increasing priority walks down the list; `UNSPEC` means no transport left.
* RDMA rail state lives in `RailMonitor`. Its lifecycle is independent of the task-level state machine: a rail can be paused while tasks keep flowing on other rails.
## State Machine
### Cross-transport failover
`resubmitTransferTask` is the single entry point that promotes a failing task to the next transport:
```
++task.failover_count
if failover_count > max_failover_attempts -> return error (exhausted)
task.xport_priority++
type = resolveTransport(task.request, task.xport_priority)
if type == UNSPEC -> return error (no transport)
transport_list_[type]->submitTransferTasks(...)
```
It has two callers, one per recoverable failure surface:
1. **Completion-stage failure.** `getTransferStatus(batch_id, task_id, status)` and the batch-form overload call `resubmitTransferTask` once per `FAILED` completion. On success the task is re-marked `PENDING` so the aggregated batch status does not latch to `FAILED` because of a task that is actually retrying.
2. **Exhaustion.** When the budget is hit, `resubmitTransferTask` sets the returned status to `InvalidEntry("Failover limit exceeded, all transports exhausted")`. Callers leave `task.type` unchanged; the task then reports `FAILED` through the normal status flow.
Submit-stage failures (`submitTransferTasks` returning non-OK) are **not** retried today. They mark the task as `UNSPEC`, and `getTransferStatus` short-circuits to `FAILED`. See Known Gaps for why.
### RDMA rail recovery
Inside `RdmaTransport`, each completion drives the rail monitor:
* Bad completion → `rail.markFailed(local_nic, remote_nic)`
* Good completion → `rail.markRecovered(local_nic, remote_nic)`
`markFailed` bumps `error_count` inside `error_window_`. Once the count hits `error_threshold_` the rail is paused until `now + cooldown_`; the cooldown doubles on every repeat failure up to `kMaxCooldown` (300 s).
`markRecovered` clears the error count, un-pauses the rail, and resets the exponential-backoff memory so the next failure cycle starts from the initial cooldown. A fast path returns without work when the rail is already healthy, which is the common case on the completion hot path.
`available(local, remote)` is the gate every work request passes through before posting. If the cooldown has expired, `available` itself resets all backoff state (error count, resume time, cooldown) and logs `Rail recovered: ... (cooldown expired)`. Otherwise it returns false and the scheduler picks another rail via `findBestRemoteDevice`.
This produces two independent recovery signals — cooldown expiry and live success — so a flaky rail does not stall forever if no other rail is posted to, and a recovered rail returns to service at the first good completion instead of waiting for the full cooldown.
## Configuration
All knobs live in the top-level `transfer-engine.json`. Defaults are safe for production; tune only if you have evidence.
| Key | Default | Meaning |
|-----|---------|---------|
| `enable_auto_failover_on_poll` | `true` | Controls whether `getTransferStatus` automatically resubmits tasks that report a recoverable `FAILED` completion. Set to `false` to make status polling observational only; internal completion paths can still trigger failover/resubmit. |
| `max_failover_attempts` | `3` | Upper bound on `resubmitTransferTask` calls per task. `0` disables cross-transport failover entirely. `1` allows exactly one switch. |
| `transports/rdma/rail_error_threshold` | `3` | Number of failures inside `rail_error_window_secs` that trips a rail into the paused state. |
| `transports/rdma/rail_error_window_secs` | `10` | Sliding window for counting rail errors. A failure older than the window resets `error_count` to 1. |
| `transports/rdma/rail_cooldown_secs` | `30` | Initial cooldown after tripping. Doubles on each repeat failure, capped at 300 s. |
The RDMA keys are read by `RailMonitor::load`. Example:
```json
{
"enable_auto_failover_on_poll": true,
"max_failover_attempts": 3,
"transports": {
"rdma": {
"rail_error_threshold": 3,
"rail_error_window_secs": 10,
"rail_cooldown_secs": 30
}
}
}
```
## Observability
### Metric
`tent_transport_failover_total` is a counter incremented once per successful transport switch inside `resubmitTransferTask`. A non-zero rate means the engine is actively recovering; a sudden jump usually points at a single bad link or flaky peer.
The counter is only built when TENT is compiled with `-DTENT_METRICS_ENABLED=ON` (see `metrics.md`). Without that flag the macro is a no-op.
### Log keywords
| Keyword | Interpretation |
|---------|----------------|
| `Transport failover: X -> Y (attempt N/M)` | A task has successfully switched transports. |
| `Task failover limit reached (M), last transport=X` | Task exhausted its budget and will surface `FAILED`. |
| `No more transports available after X failed` | `resolveTransport` returned `UNSPEC`; no further fallback exists for this request. |
| `Rail recovered: local_nic=... remote_nic=... (cooldown expired)` | Cooldown elapsed and the rail is back in service. |
| `Rail recovered: ... (un-paused by successful transfer)` | Live success on a previously paused rail brought it back early. |
## Testing
Real hardware faults are hard to stage, so TENT tests the failover machinery with decorator-style fault injection.
### FaultProxyTransport
`FaultProxyTransport` wraps any `Transport` and injects four policy-driven faults:
* `submit_fail_rate` — probability that `submitTransferTasks` returns an error.
* `status_corrupt_rate` — probability that `getTransferStatus` flips `COMPLETED → FAILED`.
* `fail_after_n_submits` — deterministic variant: succeed the first N submits, then always fail.
* `fail_install` — make `install()` fail, simulating a transport that cannot come up.
Because it implements the `Transport` interface, the engine sees an ordinary transport. All failover paths (`submitTransfer`, `getTransferStatus`, `resubmitTransferTask`) run unmodified.
### Test-only injection hook
`TransferEngineImpl::swapTransportForTest` replaces the transport in one slot after `construct()`. This is the only way the end-to-end test can wrap the real transport with `FaultProxyTransport` without bypassing `resolveTransport` or `resubmitTransferTask`. Production code never calls it.
### End-to-end suite
The end-to-end failover test suite drives the real `TransferEngineImpl` with fake transports (`FakeTransport`) wrapped in `FaultProxyTransport`. It uses a `p2p` metadata backend on `127.0.0.1` so no external services are required — the whole suite is self-contained.
Current cases:
| Test | What it exercises |
|------|-------------------|
| `StatusCorruptionTriggersFailoverToSecondary` | Primary reports `FAILED` in `getTransferStatus`; engine must resubmit on the secondary. |
| `BothTransportsFailExhaustsFailoverBudget` | Both transports fail at the completion stage; task must surface `FAILED` once the budget is drained. |
| `MixedFaultsAcrossManySubmissions` | 10 one-request batches with 30% completion corruption on RDMA; every task must end `COMPLETED`, and submit-counter math must hold. |
| `MaxFailoverAttemptsZeroDisablesFailover` | `max_failover_attempts = 0` → the first completion fault is permanent, TCP is never touched. |
| `MaxFailoverAttemptsOneAllowsSingleFailover` | `max_failover_attempts = 1` → one switch allowed; RDMA fault → TCP success. |
| `PerTaskFailoverCountsAreIndependent` | A failing task must not consume another task's budget; `failover_count` is strictly per-task. |
A test-local `PerRequestFaultProxy` (in the same file) subclasses `FaultProxyTransport` to take a `std::function` predicate, remembers which sub-task ids it marked as "poisoned" at submit time, and flips only those completions from `COMPLETED` to `FAILED` at status-query time.
### Running manually
The TENT tests are **not** in CI today (the upstream workflow builds with `USE_TENT=OFF`). Run them locally:
```bash
cmake -S . -B build-tent -DUSE_TENT=ON -DUSE_CUDA=OFF
cmake --build build-tent --target tent_failover_test tent_engine_failover_e2e_test -j
./build-tent/mooncake-transfer-engine/tent/tests/tent_failover_test
./build-tent/mooncake-transfer-engine/tent/tests/tent_engine_failover_e2e_test
```
Setting `USE_CUDA=OFF` forces `CpuPlatform`, which always reports `MTYPE_CPU`. With `USE_CUDA=ON` on a host without a GPU, `cudaPointerGetAttributes` fails, `getMemoryType` returns `MTYPE_UNKNOWN`, every transport reports unavailable, and `resolveTransport` returns `UNSPEC` before the fault injection ever runs.
Companion unit tests cover the rail monitor and related building blocks: `tent_rail_monitor_test`, `tent_failover_test`, `tent_fault_proxy_test`.
## Known Gaps
* **Submit-stage failures do not trigger failover.** When `submitTransferTasks` returns non-OK, every task in that call is marked `UNSPEC` and surfaces as `FAILED`. A naive retry loop here is unsafe for two reasons:
1. **Merged requests.** When `merge_requests` is enabled (default), `task_id_list[type]` contains both the real merged task and its derived aliases. Resubmitting per task-id re-posts one logical transfer multiple times on the fallback transport, breaking the deduplication the merge pass established.
2. **Partial enqueue.** Some transports (for example `ShmTransport::submitTransferTasks`, `NVLinkTransport::submitTransferTasks`) enqueue or start work for earlier requests in `request_list` before returning an error on a later one. The return status alone does not tell us which tasks partially succeeded, so a blanket resubmit would duplicate already-started transfers.
A safe submit-stage recovery needs either (a) a transport-level "atomic submit" capability flag plus per-task skip of derived ids, or (b) per-request status returned from `submitTransferTasks`. Neither exists today.
* `markRecovered` (and cooldown expiry in `available`) clears the exponential-backoff memory entirely. A rail that flaps repeatedly therefore does not accumulate a growing cooldown across recovery cycles. If this becomes a problem the fix is to decay rather than reset.
* Cross-transport failover is driven purely by return status; there is no latency-based "this transport is healthy but too slow, try another" signal. That belongs to the scheduler, not this document.
* TENT tests are not exercised by CI. A follow-up can add a CI job that builds with `-DUSE_TENT=ON -DUSE_CUDA=OFF` and runs the `tent_*` test targets; none of the code in this document changes in that case.

40
docs/source/design/tent/overview.md
View File

@ -75,11 +75,7 @@ The design favors predictable behavior and operational simplicity over manual tu
## TENT C++ API Reference
:::{toctree}
:maxdepth: 1
cpp-api
:::
[TENT C++ API Reference](cpp-api.md)
## TENT Metrics System
@ -87,4 +83,36 @@ cpp-api
:maxdepth: 1
metrics
:::
:::
## TENT Transport Selection
:::{toctree}
:maxdepth: 1
transport-selector
:::
## TENT Quality of Service
:::{toctree}
:maxdepth: 1
qos
:::
## TENT Slice Spraying
:::{toctree}
:maxdepth: 1
slice-spraying
:::
## TENT Failover
:::{toctree}
:maxdepth: 1
failover
:::

396
docs/source/design/tent/qos.md Normal file
View File

@ -0,0 +1,396 @@
# TENT Quality of Service (QoS)
## Overview
TENT provides Quality of Service (QoS) support to ensure that high-priority requests receive preferential treatment in multi-tenant and multi-workload environments. This document describes TENT's QoS architecture and configuration.
## Background
In shared RDMA clusters, different types of transfers have different priority requirements:
1. **Metadata and Control Messages**: Require low latency, small size
2. **Interactive Queries**: Require low to medium latency, medium size
3. **Bulk Data Transfer**: Can tolerate higher latency, large size
Without QoS, low-priority bulk transfers can monopolize bandwidth and cause high tail latency for critical requests.
TENT addresses this through:
- **Per-worker priority queues** for intra-process isolation
- **Global time-sliced coordination** for inter-process isolation
- **Priority-aware device filtering** for NUMA-aware scheduling
## Architecture
### Priority Levels
TENT supports three priority levels:
| Priority | Value | Description | Use Cases |
|----------|-------|-------------|-----------|
| `PRIO_HIGH` | 0 | High-priority requests | Metadata, control messages, latency-sensitive operations |
| `PRIO_MEDIUM` | 1 | Medium-priority requests | Interactive queries, serving workloads |
| `PRIO_LOW` | 2 | Low-priority requests | Bulk data transfer, background jobs |
### Per-Worker Priority Queues
Each worker thread maintains separate queues for each priority level:
```
┌─────────────────────────────────────┐
│ Worker Thread │
├─────────────────────────────────────┤
│ PRIO_HIGH Queue │ │
│ PRIO_MEDIUM Queue │ │
│ PRIO_LOW Queue │ │
├─────────────────────────────────────┤
│ Dequeue Priority: HIGH→MEDIUM→LOW │
└─────────────────────────────────────┘
```
**Scheduling Logic**:
1. Always drain HIGH priority queue first
2. Only process MEDIUM when HIGH is empty
3. Only process LOW when both HIGH and MEDIUM are empty
**Priority Promotion (Anti-Starvation)**:
To prevent low-priority requests from starving indefinitely, TENT implements timeout-based priority promotion:
- MEDIUM priority requests are promoted to HIGH after waiting too long
- LOW priority requests are promoted to MEDIUM after waiting too long
- Promotion checks run periodically (every 1ms by default)
This ensures that:
- High-priority requests normally never wait behind lower-priority work
- Low-priority requests eventually get serviced even under continuous high-priority load
### Global Slot Coordination
For multi-process environments, TENT implements global time-sliced coordination using shared memory:
```
Time slices rotate every N milliseconds:
Slot 0 (0-Nms): Only HIGH priority requests allowed
Slot 1 (N-2Nms): MEDIUM + HIGH priority requests allowed
Slot 2 (2N-3Nms): All priorities allowed
...repeats...
```
**Default Configuration**: 2ms per slot (6ms full cycle)
This mechanism ensures that:
- High-priority requests get dedicated service windows
- No process can monopolize bandwidth indefinitely
- Fair access across process boundaries
### Shared Memory Structure
The global slot state is maintained in shared memory:
```cpp
struct SharedHeader {
uint64_t magic; // Magic number for validation
int32_t version; // Format version
std::atomic<int> current_slot; // Current global slot (0, 1, or 2)
pthread_mutex_t global_mutex; // For synchronization (robust)
};
```
**Operations**:
- Background thread rotates slot every N milliseconds
- Workers check `canSend()` before processing requests
- Only requests with priority ≤ slot level are processed
## Configuration
### Priority Filtering
```json
{
"transports": {
"rdma": {
"enable_priority_filtering": true,
"local_rotation_interval_us": 200,
"priority_promotion_timeout_us": 10000
}
}
}
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `enable_priority_filtering` | bool | `true` | Enable priority-based device filtering |
| `local_rotation_interval_us` | int | `200` | Local device priority rotation interval (microseconds) |
| `priority_promotion_timeout_us` | int | `10000` | Timeout for priority promotion (microseconds) |
### Global Coordination
```json
{
"transports": {
"rdma": {
"slot_rotation_interval_ms": 2,
"shared_quota_shm_path": "/mooncake_rdma_slots"
}
}
}
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `slot_rotation_interval_ms` | int | `2` | Global slot rotation interval (milliseconds) |
| `shared_quota_shm_path` | string | `""` | Shared memory path for multi-process coordination |
**Note**: Leave `shared_quota_shm_path` empty to disable global coordination (single-process mode).
## Usage Examples
### Example 1: Latency-Critical Workload
For workloads where high-priority requests must have minimal latency:
```json
{
"transports": {
"rdma": {
"enable_priority_filtering": true,
"slot_rotation_interval_ms": 1,
"shared_quota_shm_path": "/mooncake_rdma_slots"
}
}
}
```
**Effect**: High-priority requests get dedicated windows every 1ms.
### Example 2: Single-Process Mode
For single-process deployments where global coordination is not needed:
```json
{
"transports": {
"rdma": {
"enable_priority_filtering": true,
"shared_quota_shm_path": ""
}
}
}
```
**Effect**: Per-worker priority queues only, no cross-process coordination.
### Example 3: Bulk-Friendly Configuration
For workloads where low-priority bulk transfers should not be starved:
```json
{
"transports": {
"rdma": {
"slot_rotation_interval_ms": 10,
"shared_quota_shm_path": "/mooncake_rdma_slots"
}
}
}
```
**Effect**: Longer slots allow more low-priority work to complete.
## Priority Assignment
### Setting Request Priority
Priority is assigned when creating `Request` objects. The default priority is `PRIO_HIGH`.
### C++ API
```cpp
// In C++
#include "tent/transfer_engine.h"
using namespace mooncake::tent;
// Create request with default priority (HIGH)
Request req;
req.opcode = Request::OpCode::READ;
req.source = buffer;
req.target_id = segment_id;
req.target_offset = 0;
req.length = size;
// req.priority is PRIO_HIGH by default
// Or specify priority explicitly
req.priority = PRIO_MEDIUM; // or PRIO_HIGH, PRIO_LOW
// Submit the request
engine.submitTransfer(batch_id, {req});
```
### Python API
```python
# In Python
import tent
# Create request with default priority (HIGH)
req = tent.Request(
opcode=tent.OpCode.READ,
source=buffer_addr,
target_id=segment_id,
target_offset=0,
length=size
)
# req.priority is tent.PRIO_HIGH by default
# Or specify priority in constructor
req = tent.Request(
opcode=tent.OpCode.READ,
source=buffer_addr,
target_id=segment_id,
target_offset=0,
length=size,
priority=tent.PRIO_LOW # or PRIO_HIGH, PRIO_MEDIUM
)
# Or set after creation
req.priority = tent.PRIO_MEDIUM
# Submit the request
engine.submit_transfer(batch_id, [req])
```
### C API
```c
// In C
#include "tent/transfer_engine.h"
// Create request with priority
tent_request_t req = {
.opcode = OPCODE_READ,
.source = buffer,
.target_id = segment_id,
.target_offset = 0,
.length = size,
.priority = 0 // 0=HIGH, 1=MEDIUM, 2=LOW
};
// Submit the request
tent_submit(engine, batch_id, &req, 1);
```
## Performance Considerations
### Trade-offs
| Configuration | High-Priority Latency | Low-Priority Throughput | Fairness |
|---------------|----------------------|------------------------|----------|
| Short slot interval (1ms) | Excellent | Poor | High |
| Default slot interval (2ms) | Good | Fair | High |
| Long slot interval (10ms) | Fair | Good | Medium |
| No global coordination | Variable | Excellent | Low (per-process only) |
### Starvation Prevention
TENT prevents starvation through two mechanisms:
1. **Global slot mechanism**:
- **HIGH priority**: Never starved (always allowed in slot 0)
- **MEDIUM priority**: Never starved (allowed in slots 1 and 2)
- **LOW priority**: Never starved (always allowed in slot 2)
2. **Priority promotion timeout**:
- Low-priority requests waiting longer than `priority_promotion_timeout_us` are promoted
- MEDIUM → HIGH promotion ensures medium priority gets service
- LOW → MEDIUM promotion ensures low priority eventually gets service
- Configurable via `priority_promotion_timeout_us` (default 10ms)
### Tuning Guidelines
1. **Start with default settings** (2ms slot interval)
2. **Measure tail latency** for each priority level
3. **Adjust slot interval** based on observations:
- If HIGH priority latency is too high: decrease interval
- If LOW priority throughput is too low: increase interval
## Troubleshooting
### Problem: High-priority requests have high latency
**Symptoms**: `PRIO_HIGH` requests experiencing unexpected delays
**Possible causes**:
1. Slot interval too long
2. Global coordination not enabled
3. Worker threads blocked on LOW priority work
**Solution**:
```json
{
"slot_rotation_interval_ms": 1,
"enable_priority_filtering": true
}
```
### Problem: Low-priority transfers starved
**Symptoms**: `PRIO_LOW` requests making no progress
**Possible causes**:
1. HIGH priority load is continuous
2. Slot interval too short
**Solution**: Increase slot interval to give LOW priority more time:
```json
{
"slot_rotation_interval_ms": 10
}
```
### Problem: Shared memory creation fails
**Symptoms**: Error messages about `/mooncake_rdma_slots`
**Possible causes**:
1. Permission issues (need write access to `/dev/shm`)
2. Stale shared memory from previous run
**Solution**:
```bash
# Remove stale shared memory
rm -f /dev/shm/mooncake_rdma_slots
# Or use a different path
{
"shared_quota_shm_path": "/mooncake_rdma_slots_v2"
}
```
## Monitoring
### Traffic Statistics
Monitor per-device traffic distribution:
```cpp
device_selector_->printTrafficStats();
```
Output example:
```
=== Device Traffic Statistics ===
Dev 0: Total=10.5 GB, EWMA BW=45.23 Gbps, Inflight=0 bytes
Dev 1: Total=8.2 GB, EWMA BW=42.18 Gbps, Inflight=0 bytes
Dev 2: Total=0.5 GB, EWMA BW=38.91 Gbps, Inflight=0 bytes
Dev 3: Total=0.3 GB, EWMA BW=39.12 Gbps, Inflight=0 bytes
```
### Priority Statistics
Monitor queue depths for each priority level (requires instrumentation).
## References
- [TENT Overview](overview.md)
- [TENT Slice Spraying](slice-spraying.md)
- [TENT C++ API](cpp-api.md)

398
docs/source/design/tent/slice-spraying.md Normal file
View File

@ -0,0 +1,398 @@
# TENT Slice Spraying
## Overview
This document describes TENT's Slice Spraying mechanism, which enables efficient data movement in multi-rail RDMA environments through intelligent device selection and adaptive load balancing.
## Background
In multi-rail RDMA environments, naive round-robin striping leads to suboptimal performance because:
1. **NUMA Effects**: Cross-NUMA access incurs additional latency and reduces effective bandwidth
2. **Load Imbalance**: Static striping cannot adapt to dynamic load conditions
3. **Heterogeneous Link Quality**: Different rails may have different effective bandwidth due to congestion or hardware characteristics
TENT addresses these issues through:
- **NUMA-aware device selection** with configurable penalties
- **EWMA-based bandwidth estimation** for adaptive load balancing
- **Dynamic multi-path allocation** for large transfers
## Architecture
### Device Selector
The `DeviceSelector` component is responsible for choosing which RDMA device(s) to use for each transfer request. It operates in two modes:
#### Baseline Mode (Round-Robin)
When `enable_smart_scheduling = false`, the selector uses simple round-robin within the highest-priority device tier (typically local NUMA devices):
```
For each request:
1. Find first non-empty device tier (local NUMA preferred)
2. Select devices round-robin within that tier
3. Ignore lower-priority tiers
```
**Characteristics**:
- Deterministic behavior
- No runtime overhead for tracking
- Consistent with original TE behavior
- Does not adapt to load conditions
#### Smart Mode (EWMA-Based Selection)
When `enable_smart_scheduling = true`, the selector uses an EWMA-based algorithm:
```
For each request:
1. Calculate predicted completion time for each device:
predicted_time = (inflight_bytes + slice_bytes) / ewma_bandwidth
2. Apply NUMA penalty based on tier:
score = predicted_time × numa_tier_weights[tier]
3. Select device(s) with minimum score:
- Single slice: best device only
- Multiple slices: weighted distribution across devices
4. Update EWMA bandwidth on completion:
ewma_bandwidth = α × ewma_bandwidth + (1 - α) × observed_bandwidth
where α = bandwidth_learning_rate
```
**Characteristics**:
- Adapts to changing load conditions
- Prefers local NUMA devices
- Spreads load across multiple rails
- Higher runtime overhead
### NUMA-Aware Selection
Devices are organized into tiers based on NUMA distance:
| Tier | Description | Default Penalty |
|------|-------------|-----------------|
| Rank 0 | Local NUMA | 1.0 (baseline) |
| Rank 1 | Remote NUMA (tier 1) | 5.0 |
| Rank 2 | Remote NUMA (tier 2) | 10.0 |
The penalty is applied as a multiplier to predicted completion time, making remote devices less attractive unless local devices are heavily loaded.
### EWMA Bandwidth Estimation
Each device maintains an EWMA (Exponentially Weighted Moving Average) of its effective bandwidth:
```
initial_value = theoretical_bandwidth
on_transfer_complete:
observed_bandwidth = transfer_size / transfer_time
ewma_bandwidth = α × ewma_bandwidth + (1 - α) × observed_bandwidth
ewma_bandwidth = clamp(ewma_bandwidth,
0.1 × theoretical,
10.0 × theoretical)
```
where `α = bandwidth_learning_rate`.
**Note on terminology**: The EWMA formula uses α as the coefficient for the old value. Therefore:
- **Lower α** (closer to 0) → more weight on new observations → **faster adaptation**
- **Higher α** (closer to 1) → more weight on old value → **slower adaptation**
Examples:
- α = 0: `ewma_bandwidth = observed_bandwidth` (full adaptation, always use new value)
- α = 1: `ewma_bandwidth = ewma_bandwidth` (no learning, never update)
- α = 0.01: `ewma_bandwidth = 0.01 × old + 0.99 × new` (default, gradual adaptation)
The EWMA provides:
- **Memory**: Recent observations have more influence than old ones
- **Stability**: Smooths out transient fluctuations
- **Adaptability**: Tracks gradual changes in link quality
### Multi-Path Allocation
For large transfers, TENT distributes slices across multiple devices:
**Single Path** (small requests):
- All slices go to the single best device
- Minimizes coordination overhead
**Multi Path** (large requests):
- **Normal mode** (99% of calls): Slices distributed proportionally to device capacity
- Each device gets: `(device_weight / total_weight) × num_slices`
- Remaining slices assigned to best device
- **Probe mode** (1% of calls, every 100th call): Slices distributed round-robin
- Purpose: Ensure all devices are continuously sampled for EWMA updates
- Prevents EWMA starvation for less-used devices
### Request Flow
```
┌──────────────┐
│ Application │
└──────┬───────┘
│ submitTransfer()
▼
┌──────────────────────────────────────┐
│ RdmaTransport::submitTransferTasks │
│ - Split large requests into slices │
│ - Call DeviceSelector for allocation │
│ - Only if num_slices >= max_slice_count/2 │
└──────┬───────────────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ DeviceSelector::allocate │
│ ┌────────────────────────────────┐ │
│ │ smart_selection_enabled? │ │
│ └────┬──────────────────────┬────┘ │
│ │ Yes │ No │
│ ▼ ▼ │
│ ┌─────────┐ ┌─────────┐ │
│ │ Smart │ │ Baseline│ │
│ │ Mode │ │ Mode │ │
│ └────┬────┘ └────┬────┘ │
│ │ │ │
│ └────────┬───────────┘ │
│ ▼ │
│ ┌────────────────────────────────┐ │
│ │ Return slice_dev_ids │ │
│ └────────────────────────────────┘ │
└──────────────────────────────────────┘
```
## Configuration
All slice spraying parameters are configurable via the configuration file:
### Core Scheduling
```json
{
"transports": {
"rdma": {
"enable_smart_scheduling": true
}
}
}
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `enable_smart_scheduling` | bool | `true` | Enable EWMA-based selection (false = round-robin) |
### NUMA Penalties
```json
{
"transports": {
"rdma": {
"numa_penalties": [1.0, 5.0, 10.0]
}
}
}
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `numa_penalties` | array[float] | `[1.0, 5.0, 10.0]` | Penalty multipliers for each NUMA tier |
**Guidelines**:
- Higher values = stronger preference for local devices
- Set all to `1.0` to disable NUMA awareness
- Increase remote penalties if cross-NUMA latency is high
### Bandwidth Estimation
```json
{
"transports": {
"rdma": {
"bandwidth_learning_rate": 0.01,
"ewma_min_bandwidth_multiplier": 0.1,
"ewma_max_bandwidth_multiplier": 10.0
}
}
}
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `bandwidth_learning_rate` | float | `0.01` | EWMA learning rate (0.0 = full adaptation, 1.0 = no learning) |
| `ewma_min_bandwidth_multiplier` | float | `0.1` | Minimum bandwidth as fraction of theoretical |
| `ewma_max_bandwidth_multiplier` | float | `10.0` | Maximum bandwidth as fraction of theoretical |
**Guidelines**:
- Lower α (e.g., 0.001) → faster adaptation, more volatile → responds quickly to changes
- Higher α (e.g., 0.1) → slower adaptation, more stable → smooths out transient fluctuations
- Default α = 0.01 provides balanced adaptation
- Multipliers constrain EWMA to reasonable range [0.1×, 10.0×] of theoretical bandwidth
### Device Selection Scoring
```json
{
"transports": {
"rdma": {
"score_jitter_range": 1e-9,
"score_epsilon": 1e-12
}
}
}
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `score_jitter_range` | float | `1e-9` | Random jitter range to avoid deterministic selection |
| `score_epsilon` | float | `1e-12` | Small value to prevent division by zero |
### Bandwidth Constants
```json
{
"transports": {
"rdma": {
"default_bandwidth_gbps": 400.0,
"min_bandwidth_gbps": 10.0,
"max_bandwidth_gbps": 800.0
}
}
}
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `default_bandwidth_gbps` | float | `400.0` | Default NIC bandwidth when topology info unavailable |
| `min_bandwidth_gbps` | float | `10.0` | Minimum valid NIC bandwidth (Gbps) |
| `max_bandwidth_gbps` | float | `800.0` | Maximum valid NIC bandwidth (Gbps) |
**Notes**:
- These constants define the valid range and default for device bandwidth
- Used in EWMA calculations and theoretical bandwidth estimation
- If a device's reported bandwidth is outside [min, max], default_bandwidth is used
## Usage Examples
### Example 1: Latency-Sensitive Workload
For latency-sensitive queries where local NUMA access is critical:
```json
{
"transports": {
"rdma": {
"enable_smart_scheduling": true,
"numa_penalties": [1.0, 100.0, 1000.0],
"bandwidth_learning_rate": 0.001
}
}
}
```
**Effect**: Strongly prefers local devices, slow adaptation for stability.
### Example 2: Bulk Data Transfer
For bulk transfers where throughput is more important than latency:
```json
{
"transports": {
"rdma": {
"enable_smart_scheduling": true,
"numa_penalties": [1.0, 2.0, 3.0],
"bandwidth_learning_rate": 0.1
}
}
}
```
**Effect**: Allows cross-NUMA transfers, fast adaptation to load.
### Example 3: Baseline Mode
For deterministic performance matching original TE:
```json
{
"transports": {
"rdma": {
"enable_smart_scheduling": false
}
}
}
```
**Effect**: Round-robin within local NUMA tier, no adaptation, minimal overhead.
## Performance Considerations
### Overhead Comparison
| Mode | CPU Overhead | Adaptability | NUMA Awareness |
|------|--------------|--------------|----------------|
| Baseline | Minimal | None | Tier-based (static) |
| Smart | Moderate | EWMA-based | Dynamic + penalty |
### When to Use Each Mode
**Use Baseline Mode when**:
- Workload is uniform and predictable
- Deterministic performance is required
- CPU overhead must be minimized
- All devices are in same NUMA node
**Use Smart Mode when**:
- Workload is heterogeneous
- Link quality varies over time
- NUMA effects are significant
- Maximum throughput is desired
### Tuning Guidelines
1. **Start with baseline mode** to establish performance baseline
2. **Enable smart mode** with conservative parameters:
- `numa_penalties = [1.0, 2.0, 5.0]`
- `bandwidth_learning_rate = 0.01`
3. **Monitor performance** and adjust based on observations:
- If cross-NUMA transfers are too frequent: increase remote penalties
- If adaptation is too slow (EWMA not keeping up with load changes): decrease α
- If performance is unstable (too much fluctuation): increase α
## Troubleshooting
### Problem: All requests go to cross-NUMA devices
**Symptoms**: Poor performance, high latency
**Diagnosis**:
```cpp
device_selector_->printTrafficStats();
```
**Solution**: Check `numa_penalties` configuration. Ensure local devices have lowest penalty (1.0).
### Problem: Performance worse than baseline
**Symptoms**: Smart mode slower than baseline mode
**Possible causes**:
1. Learning rate too high (volatile decisions)
2. NUMA penalties too low (not preferring local)
3. Score jitter too large (too much randomness)
**Solution**: Use more conservative:
```json
{
"bandwidth_learning_rate": 0.001,
"numa_penalties": [1.0, 10.0, 100.0],
"score_jitter_range": 1e-12
}
```
## References
- [TENT Overview](overview.md)
- [TENT QoS](qos.md)
- [TENT C++ API](cpp-api.md)

246
docs/source/design/tent/transport-selector.md Normal file
View File

@ -0,0 +1,246 @@
# TENT Transport Selector
## Overview
The Transport Selector is responsible for choosing the optimal transport and devices for each transfer request based on configuration policies.
## Request Priority
TENT uses a unified priority system across all components. See [QoS.md](qos.md) for detailed description of priority levels and their usage throughout the system.
Quick reference:
- `"high"` / `0` - High-priority requests (metadata, control, latency-sensitive)
- `"medium"` / `1` - Medium-priority requests (interactive queries, serving)
- `"low"` / `2` - Low-priority requests (bulk transfer, background jobs)
## Configuration-Based Transport Selection
Transport selection is driven by configuration with pattern-based rules.
### Configuration Example
```json
{
"policy": [
{
"name": "high_prio_fast",
"segment_type": "memory",
"priority": "high",
"devices": ["mlx5_0", "mlx5_1", "mlx5_2"],
"transports": ["nvlink", "rdma", "shm"]
},
{
"name": "low_prio_slow",
"segment_type": "memory",
"priority": "low",
"devices": ["mlx5_0"],
"transports": ["rdma", "tcp"]
},
{
"name": "file_storage",
"segment_type": "file",
"transports": ["gds", "io_uring", "rdma"]
}
]
}
```
### Policy Fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `name` | string | Yes | Policy identifier (for logging) |
| `segment_type` | string | Yes | `"memory"` or `"file"` |
| `priority` | string or int | No | Match only requests with this priority: `"high"` (0), `"medium"` (1), `"low"` (2) |
| `devices` | array[string] | No | List of allowed device names (empty = all devices) |
| `transports` | array[string] | No | Transport preference list (evaluated in order) |
### Memory Type Filters
For `memory` segments, you can filter by source/destination memory type:
| Pattern | Matches |
|---------|---------|
| `"cuda"` | CUDA GPU memory |
| `"cpu"` | CPU/host memory |
| `"hip"` | ROCm/HIP GPU memory |
| `"npu"` | Ascend NPU memory |
| `"*"` | Any memory type |
### Size Filters
Restrict policies to specific transfer sizes:
```json
{
"name": "small_transfers",
"segment_type": "memory",
"min_size": 0,
"max_size": 1048576,
"transports": ["shm", "nvlink"]
}
```
## Device Mask
The `devices` field in a policy creates a bitmask that restricts which NICs can be used:
```json
{
"name": "use_nic_0_only",
"segment_type": "memory",
"devices": ["mlx5_0"],
"transports": ["rdma"]
}
```
This is translated internally to a 64-bit bitmask where each bit represents one device:
- `devices: ["mlx5_0"]` → `device_mask = 0x0001` (bit 0 set)
- `devices: ["mlx5_1", "mlx5_2"]` → `device_mask = 0x0006` (bits 1 and 2 set)
- `devices: []` (empty) → `device_mask = ~0ULL` (all devices)
## Transport Fallback
When multiple transports are listed in the `transports` array, they act as fallback options:
```json
{
"transports": ["rdma", "tcp"]
}
```
This means:
1. Try RDMA first
2. If RDMA is unavailable or fails, fall back to TCP
### Fallback with `transport_index`
For programmatic control, the `transport_index` parameter selects which transport to use:
```cpp
// transport_index = 0 → First transport (rdma)
// transport_index = 1 → Second transport (tcp)
// transport_index = 2 → Third transport (if exists)
auto result = selector.select(context, transports, transport_index);
```
## Per-request override (`transport_hint`)
`Request::transport_hint` lets a caller bypass the policy lookup for one request at a time. It's the per-request analogue of `transport_index` and sits on top of the configured policies, so callers can keep the global policy config as the default while still pinning specific requests.
```cpp
Request r{};
r.opcode = Request::WRITE;
r.source = local_ptr;
r.target_id = seg;
r.target_offset = 0;
r.length = 4096;
r.transport_hint = TransportType::TCP; // UNSPEC (default) defers to policy
engine.submitTransfer(batch_id, {r});
```
### Semantics
* **`UNSPEC` (default)**: unchanged — `TransportSelector::select(...)` runs as documented above.
* **Pinned**: the request's first attempt uses the hinted transport. On failover (when enabled), the engine asks the selector for the next candidate **with the hint excluded**, so failover never loops back onto the transport that already failed.
## Default Behavior
If no `policy` is configured, TENT falls back to original behavior:
| Segment Type | Default Transport Order |
|--------------|-------------------------|
| File | GDS → IOURING → RDMA |
| Memory | Uses `buffer_transports` order from buffer registration |
## Complete Example
```json
{
"policy": [
{
"name": "high_priority_local",
"segment_type": "memory",
"same_machine": true,
"priority": "high",
"transports": ["nvlink", "shm"]
},
{
"name": "high_priority_remote",
"segment_type": "memory",
"same_machine": false,
"priority": "high",
"local_memory": "cuda",
"remote_memory": "cuda",
"devices": ["mlx5_0", "mlx5_1", "mlx5_2"],
"transports": ["rdma"]
},
{
"name": "medium_priority",
"segment_type": "memory",
"priority": "medium",
"transports": ["rdma"]
},
{
"name": "bulk_transfer",
"segment_type": "memory",
"priority": "low",
"min_size": 104857600,
"transports": ["rdma", "tcp"]
},
{
"name": "file_ops",
"segment_type": "file",
"transports": ["gds", "io_uring"]
}
]
}
```
## Unified Priority System
The priority value propagates through multiple layers:
```
Request.priority (application layer)
↓
TransportSelector policy matching
↓
DeviceSelector allocation (QoS filtering)
↓
Worker thread scheduling queues
```
**Layer interactions**:
1. **TransportSelector**: Policy's `priority` field filters which requests match
2. **DeviceSelector**: `priority` parameter controls device eligibility (QoS mode)
3. **Workers**: Separate queues per priority level with strict draining order
See [QoS.md](qos.md) for details on worker scheduling and global slot coordination.
## Data Flow
```
Request with priority
↓
TransportSelector.select(context, transports, transport_index)
↓
Match policy by:
- segment_type (file/memory)
- priority (exact match if specified in policy)
- location constraints
- size constraints
↓
Build device_mask from policy.devices
↓
Select transport from policy.transports[transport_index]
↓
Return SelectionResult { transport, device_mask }
↓
RdmaTransport.submitTransferTasks(batch, requests)
↓
DeviceSelector.allocate(..., request.priority, batch.device_mask)
↓
Worker scheduling (separate queues per priority)
```

6
docs/source/design/transfer-engine/ascend_direct_transport.md
View File

@ -48,7 +48,7 @@ The following environment variables can be configured to control Ascend Direct T
| Variable | Description | Default Value | Example |
|----------|-------------|---------------|---------|
| `ASCEND_AUTO_CONNECT` | Enable automatic connection management | 0 (disabled) | `ASCEND_AUTO_CONNECT=1` |
| `ASCEND_AUTO_CONNECT` | Enable automatic connection management | 1 (enabled) | `ASCEND_AUTO_CONNECT=0` |
| `ASCEND_ENABLE_USE_FABRIC_MEM` | Enable fabric memory transfer mode in Mooncake Store (A3 only) | 0 (disabled) | `ASCEND_ENABLE_USE_FABRIC_MEM=1` |
| `ASCEND_USE_ASYNC_TRANSFER` | Enable asynchronous transfer mode | 0 (disabled) | `ASCEND_USE_ASYNC_TRANSFER=1` |
| `ASCEND_GLOBAL_RESOURCE_CONFIG` | Global resource configuration | - | `ASCEND_GLOBAL_RESOURCE_CONFIG="{\"fabric_memory.max_capacity\":32}"` |
@ -64,7 +64,7 @@ The following environment variables can be configured to control Ascend Direct T
**Detailed Descriptions:**
- **ASCEND_AUTO_CONNECT**: Requires CANN 9.0 or later. Default is 0, recommended to enable on supported versions: link can be automatically disconnected when the remote end goes offline abnormally.
- **ASCEND_AUTO_CONNECT**: Requires CANN 9.0 or later. Default is 1: link can be automatically disconnected when the remote end goes offline abnormally. Set `ASCEND_AUTO_CONNECT=0` to disable.
- **ASCEND_ENABLE_USE_FABRIC_MEM**: Requires CANN 9.0+ and HDK 26.0+. Recommended when using Mooncake Store on supported A3 platforms: it can significantly improve transmission performance.
- **ASCEND_USE_ASYNC_TRANSFER**: Requires CANN 8.5+. Enables HIXL asynchronous transfer mode, defaults to synchronous mode.
- **ASCEND_GLOBAL_RESOURCE_CONFIG**: Configures HIXL global resources. Refer to HIXL documentation for `OPTION_GLOBAL_RESOURCE_CONFIG` settings.
@ -100,4 +100,4 @@ The following environment variables can be configured to control Ascend Direct T
12. **Fabric Memory mode**: On the A3, with the latest drivers and CANN installed, when using Mooncake store, the ASCEND_ENABLE_USE_FABRIC_MEM environment variable can be set to enable fabric memory transfer mode (which allows direct access remote HOST memory).
13. **Auto Connect**: The auto connect feature can be enabled by configuring the `ASCEND_AUTO_CONNECT` environment variable. The default value is 0 (disabled).
13. **Auto Connect**: The auto connect feature is enabled by default (`ASCEND_AUTO_CONNECT=1`). Set `ASCEND_AUTO_CONNECT=0` to disable.

2
docs/source/design/transfer-engine/cpp-api.md
View File

@ -337,7 +337,7 @@ The HTTP server should implement three following RESTful APIs, while the metadat
2. `PUT /metadata?key=$KEY`: Update the metadata corresponding to `$KEY` to the value of the request body.
3. `DELETE /metadata?key=$KEY`: Delete the metadata corresponding to `$KEY`.
For specific implementation, refer to the demo service implemented in Golang at [mooncake-transfer-engine/example/http-metadata-server](../../../mooncake-transfer-engine/example/http-metadata-server).
For specific implementation, refer to the demo service implemented in Golang at [mooncake-transfer-engine/example/http-metadata-server](gh-dir:mooncake-transfer-engine/example/http-metadata-server).
### Initialization

588
docs/source/design/transfer-engine/efa_transport.md
View File

@ -4,6 +4,7 @@ This document describes how to build and use Mooncake with AWS Elastic Fabric Ad
## Prerequisites
(efa-prerequisites-driver)=
### 1. AWS EFA Driver and libfabric
EFA driver and libfabric should be pre-installed on AWS instances with EFA support (e.g., p6-b300.48xlarge, p6-b200.48xlarge, p5en.48xlarge, p5e.48xlarge, p5.48xlarge).
@ -32,14 +33,23 @@ sudo ./dependencies.sh -y
This installs all system packages, git submodules (including pybind11 and yalantinglibs), and Go.
**Additional EFA-specific dependencies** (not covered by `dependencies.sh`):
> **Note:** The EFA driver and libfabric are **not** installed by `dependencies.sh`. They must be pre-installed on the instance (see section 1 above).
## Installing from PyPI (recommended)
Pre-built EFA wheels are published to PyPI by the official release pipeline, so most users do not need to build from source. The EFA transport's memory path is CUDA-aware, so two variants are published:
```bash
# gflags is needed by transfer_engine_bench and EFA unit tests
sudo apt-get install -y libgflags-dev
# GPU memory transfers (e.g., KV cache in vLLM) — built with USE_CUDA=ON
pip install mooncake-transfer-engine-efa
# CPU/DRAM-only transfers — built with USE_CUDA=OFF
pip install mooncake-transfer-engine-efa-non-cuda
```
> **Note:** The EFA driver and libfabric are **not** installed by `dependencies.sh`. They must be pre-installed on the instance (see section 1 above).
> **Note:** These wheels deliberately do **not** bundle `libfabric`/`libefa` (see the runtime note in [Building a Distributable Wheel](#efa-distributable-wheel)). They resolve to the system AWS EFA installation at runtime, so the EFA driver and libfabric from the [Prerequisites](#efa-prerequisites-driver) must still be present on the instance. Make sure `/opt/amazon/efa/lib` is on `LD_LIBRARY_PATH`.
To build from source instead (for development, an unreleased revision, or a custom configuration), follow the sections below.
## Building Mooncake with EFA Support
@ -87,6 +97,31 @@ cp mooncake-common/libasio.so ../mooncake-wheel/mooncake/
pip install -e ../mooncake-wheel --no-build-isolation
```
(efa-distributable-wheel)=
### 3. Building a Distributable Wheel (optional)
To produce a relocatable wheel for distribution (instead of the editable install above), use `scripts/build_wheel.sh`, which runs `auditwheel repair` to bundle non-system dependencies:
```bash
# After the cmake/make build above completes:
PYTHON_VERSION=3.13 BUILD_DIR=build bash scripts/build_wheel.sh 3.13 dist
pip install dist/mooncake_transfer_engine-*.whl
```
To produce a wheel whose package name matches the published variants (`mooncake-transfer-engine-efa` / `-efa-non-cuda`), set the corresponding build-variant environment variable — this is exactly what the release pipeline does:
```bash
# GPU build (cmake was configured with USE_CUDA=ON):
EFA_BUILD=1 PYTHON_VERSION=3.13 BUILD_DIR=build bash scripts/build_wheel.sh 3.13 dist
# CPU build (cmake was configured with USE_CUDA=OFF):
EFA_NON_CUDA_BUILD=1 PYTHON_VERSION=3.13 BUILD_DIR=build bash scripts/build_wheel.sh 3.13 dist
```
> **CI/CD:** EFA wheels are built and published automatically — see `.github/workflows/ci_efa.yml` (per-PR build validation) and `.github/workflows/release-efa.yaml` (tagged release to GitHub Release + PyPI). No EFA hardware is required to *build* the wheel: only the libfabric headers/library are needed to compile and link, which the CI runner obtains from the distro `libfabric-dev` package.
> **Important (EFA builds):** `auditwheel repair` excludes `libfabric` and `libefa` from the wheel so they resolve to the system EFA installation (`/opt/amazon/efa/lib`) at runtime. This is required because the in-process `aws-ofi-nccl` plugin (loaded by NCCL) links the **same** system `libfabric`. If the wheel bundled its own copy, the process would load two independent libfabric instances — Mooncake's bundled one and NCCL's system one — and whichever initializes first claims the EFA device, leaving the other with an empty provider list (`fi_getinfo: provider efa output empty list`). NCCL then silently falls back to the TCP provider and cross-node collectives such as `all_gather_object` hang. Excluding libfabric/libefa (see `scripts/build_wheel.sh`) keeps a single shared libfabric in the process. If you are on an older Mooncake build whose wheel still bundles libfabric, force the system copy with `export LD_PRELOAD=/opt/amazon/efa/lib/libfabric.so.1` as a workaround.
## Verification
Test EFA transport initialization:
@ -99,7 +134,7 @@ result = te.initialize('127.0.0.1', 'P2PHANDSHAKE', 'efa', '')
print(f'Initialize result: {result}') # Should be 0
# You should see logs like:
# EFA device (libfabric): rdmap79s0, domain: rdmap79s0-rdm, provider: efa
# EFA device (libfabric): rdmap79s0, domain: rdmap79s0-rdm, fabric: efa, provider: efa
```
## Unit Tests
@ -119,34 +154,43 @@ The test suite includes:
| `WriteAndRead` | Write then read with data integrity check |
| `MultiWrite` | Batch write (16 requests) |
| `StressMultipleBatches` | Stress test (20 batches x 8 requests) |
| `WarmupSegmentLoopback` | `warmupSegment()` handshake path + idempotent re-call |
| `WarmupSegmentNotFound` | `warmupSegment()` fails cleanly for an unknown segment |
| `RegisterMemoryBatch` | `registerLocalMemoryBatch` / `unregisterLocalMemoryBatch` round-trip |
| `LargeTransfer` | 128 MB buffer, 64 x 1 MB slices — exercises WR / CQ pacing |
| `RepeatedOpenSegment` | `openSegment()` on the same peer repeatedly still transfers correctly |
You can also run all unit tests via CTest:
You can also run the EFA tests via CTest:
```bash
cd build && ctest --output-on-failure
cd build && ctest --output-on-failure -R 'efa'
```
Environment variables for test configuration:
```bash
export MC_METADATA_SERVER=P2PHANDSHAKE # default
export MC_LOCAL_SERVER_NAME=127.0.0.1:12345 # default
```
> **Note:** `ctest --output-on-failure` without a filter runs every test in the build, including TCP / metadata / master-service suites that require an etcd server or a running `mooncake_master`. Those will fail or hang on a machine that is only provisioned for EFA testing — the failures are not EFA-specific. Use `-R 'efa'` to restrict the run to the EFA tests.
## Performance Benchmark
Use `transfer_engine_bench` to measure EFA transport throughput between two nodes.
### Target Node (receiver)
The following commands are the GPU-to-GPU configuration that produces the headline numbers in the [Benchmark Results](#benchmark-results) tables (≈ 350 GB/s write on a p5en.48xlarge pair, ≈ 302 GB/s on p6-b200.48xlarge). Two things matter the most:
- `--gpu_id=-1` on **both** sides — this fans buffers across every GPU, which in turn lets both NUMA nodes' NICs saturate. Pinning a single GPU (the default `--gpu_id=0`) halves throughput because half the NICs end up cross-NUMA.
- `--block_size=1048576` (1MB, not the 64 KB default) — each block becomes one `fi_write` / `fi_read`, so larger blocks amortize per-op overhead and are the main knob for hitting line rate.
### 1. Target Node (receiver)
```bash
./build/mooncake-transfer-engine/example/transfer_engine_bench \
--mode=target \
--protocol=efa \
--metadata_server=P2PHANDSHAKE
--metadata_server=P2PHANDSHAKE \
--buffer_size=4294967296 \
--gpu_id=-1
```
### Initiator Node (sender)
`--buffer_size` must be at least as large as the initiator's `--buffer_size` — the initiator writes into offsets `[0, buffer_size)` on the target, so keep these in sync.
### 2. Initiator Node (sender)
```bash
./build/mooncake-transfer-engine/example/transfer_engine_bench \
@ -156,45 +200,78 @@ Use `transfer_engine_bench` to measure EFA transport throughput between two node
--segment_id=<target_hostname>:<target_port> \
--operation=write \
--duration=10 \
--threads=8 \
--block_size=65536 \
--threads=16 \
--block_size=1048576 \
--batch_size=128 \
--buffer_size=1073741824 \
--buffer_size=4294967296 \
--gpu_id=-1 \
--report_unit=GB
```
> **Tip:** For CPU-to-CPU benchmarks, prepend `CUDA_VISIBLE_DEVICES=""` to prevent the CUDA runtime from being initialized. Without it, `nvidia-smi` may show GPU memory usage (due to CUDA context initialization) even though the benchmark only uses DRAM.
Replace `<target_hostname>:<target_port>` with the target node's address shown in the target's startup log (e.g., `ip-172-31-29-226:12345`).
> **CPU-to-CPU** (no GPUs): build with `-DUSE_CUDA=OFF`, **or** pass `--use_vram=false` to a CUDA-enabled binary. Drop `--gpu_id=-1` in that case — the bench will spread buffers across NUMA nodes instead.
> **Why `threads=16` and not 32:** the SRD shared endpoint caps outstanding WRs per NIC (default 256 — see `MC_MAX_WR`). With `threads × batch ≤ NICs × max_wr` the CQ never saturates; going higher triggers backoff and times out. 32 threads × 128 batch = 4096 slices chasing 16 × 256 = 4096 WRs has no headroom, so the steady-state config settles at 16 threads.
### Key Parameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `--block_size` | 65536 | Bytes per transfer request |
| `--mode` | initiator | `initiator` (sender) or `target` (receiver) |
| `--protocol` | rdma | Transport protocol; use `efa` here |
| `--metadata_server` | `192.168.3.77:2379` | etcd address or `P2PHANDSHAKE` for standalone use |
| `--segment_id` | `192.168.3.76` | Initiator only: `<target_host>:<target_port>` from the target's startup log |
| `--operation` | read | `read` or `write` |
| `--block_size` | 65536 | Bytes per transfer request; **1 MB (1048576) is the main knob for EFA throughput** |
| `--batch_size` | 128 | Requests per batch |
| `--threads` | 12 | Concurrent submission threads |
| `--buffer_size` | 1 GB | Total buffer size (per GPU when `--gpu_id=-1`) |
| `--threads` | 12 | Concurrent submission threads (initiator) |
| `--buffer_size` | 1 GB | Buffer size (per GPU when `--gpu_id=-1`, otherwise total) |
| `--duration` | 10 | Test duration in seconds |
| `--operation` | write | `read` or `write` |
| `--report_unit` | GB | `GB\|GiB\|Gb\|MB\|MiB\|Mb` |
| `--gpu_id` | 0 | GPU device ID; `-1` to use all GPUs (requires `-DUSE_CUDA=ON`) |
| `--report_unit` | GB | `GB\|GiB\|Gb\|MB\|MiB\|Mb\|KB\|KiB\|Kb` |
| `--use_vram` | true | Allocate from GPU VRAM (requires `-DUSE_CUDA=ON`); pass `--use_vram=false` for CPU-to-CPU on a CUDA build |
| `--gpu_id` | 0 | GPU device ID when `--use_vram=true`; `-1` fans buffers across every GPU and is what actually saturates all NICs in a GPU-to-GPU run |
| `--init_mem` | true | Zero-fill the allocated buffer; rarely needs to change |
| `--auto_discovery` | false | Auto-discover topology on init; off for reproducible runs |
| Environment Variable | Default | Description |
|---------------------|---------|-------------|
| `MC_SLICE_SIZE` | 65536 | Slice size for RDMA transport. **Not used by EFA transport** (see note below). |
| `MC_EFA_STRIPING_THRESHOLD` | 2097152 | Transfers larger than this (bytes) are striped across all NICs |
> **Note on EFA slicing:** Unlike RDMA transport which splits every transfer into fixed `MC_SLICE_SIZE` chunks, EFA transport uses a different strategy: transfers ≤ `MC_EFA_STRIPING_THRESHOLD` (default 2MB) are sent as a **single `fi_write`/`fi_read`** whose size equals `block_size`; transfers larger than the threshold are striped across all NICs (one chunk per NIC). This means **`block_size` directly determines per-operation size** and is the key tuning parameter for EFA, while `MC_SLICE_SIZE` has no effect.
> **Note on EFA slicing:** EFA transport does not split each transfer into fixed-size slices the way RDMA transport does — each transfer is sent as a single `fi_write` / `fi_read` whose size equals `block_size`, round-robin'd across NICs per request. **`block_size` is the key tuning parameter** for EFA throughput.
> **Note:** `buffer_size` must be >= `block_size * batch_size * threads`. The benchmark auto-adjusts if too small.
(benchmark-results)=
### Benchmark Results
#### p6-b200.48xlarge (B200, 8 EFA × 400 Gbps)
#### 1. p6-b300.48xlarge (B300, 16 EFA × 400 Gbps)
Tested on two p6-b300.48xlarge instances (Intel Xeon Platinum 8559C, 8× B300, 16 EFA devices) in the same AWS placement group. Numbers below are post-SRD-shared-endpoint (#1944) on a fresh `main` build with the DLAMI pytorch env (CUDA 13).
**GPU-to-GPU** (build with `-DUSE_CUDA=ON`, `--gpu_id=-1` for all 8 GPUs, `--buffer_size=2147483648`):
| Configuration | Write | Read |
|---------------|-------|------|
| block=1MB, threads=16, batch=128 | 758.88 GB/s | 720.35 GB/s |
| block=1MB, threads=32, batch=64 | 753.32 GB/s | **755.78 GB/s** |
| **block=1MB, threads=32, batch=32** | **780.33 GB/s** | - |
| block=1MB, threads=64, batch=32 | 780.23 GB/s | - |
> **Peak: 780 GB/s write**, reaching ~97.5% of the 800 GB/s theoretical line rate (16×400 Gbps). GPUDirect RDMA bypasses DRAM entirely (HBM3e → PCIe switch → NIC), so performance is not bottlenecked by CPU memory bandwidth.
**CPU-to-CPU** (build with `-DUSE_CUDA=OFF`, or `--use_vram=false` on a CUDA build, `--buffer_size=4294967296`):
| Configuration | Write | Read |
|---------------|-------|------|
| **block=1MB, threads=32, batch=128** | **282.93 GB/s** | **270.47 GB/s** |
| block=1MB, threads=16, batch=128 | 282.04 GB/s | 249.67 GB/s |
| block=1MB, threads=32, batch=64 | 282.84 GB/s | 256.26 GB/s |
> CPU-to-CPU is bounded by DRAM bandwidth on the Xeon 8559C — write throughput is essentially flat across thread/batch combinations (~282 GB/s), confirming DRAM controller saturation rather than a NIC or in-flight-WR limit.
#### 2. p6-b200.48xlarge (B200, 8 EFA × 400 Gbps)
Tested on two p6-b200.48xlarge instances in the same AWS placement group.
> **Note:** numbers below predate the SRD shared-endpoint refactor (#1944) and current EFA tuning work. They are a lower bound for the current code; we will re-sweep and update when a B200 pair is available again.
**GPU-to-GPU** (build with `-DUSE_CUDA=ON`, `--gpu_id=-1` for all 8 GPUs):
| Configuration | Write | Read |
@ -208,121 +285,223 @@ Tested on two p6-b200.48xlarge instances in the same AWS placement group.
|---------------|-------|------|
| block=1MB, threads=32, batch=128, buf=4GB | **222 GB/s** (stable over 6 runs) | **226 GB/s** |
<details>
<summary>CPU Parameter Tuning History (p6-b200)</summary>
Earlier CPU-to-CPU tuning results (before EFA striping optimization, when `MC_SLICE_SIZE` was still used by EFA):
| block_size | threads | batch_size | MC_SLICE_SIZE | Throughput |
|-----------|---------|------------|---------------|-----------|
| 64KB | 8 | 128 | default (64KB) | 69.47 GB/s |
| 128KB | 32 | 128 | default | 92.33 GB/s |
| 128KB | 32 | 128 | 256KB | 156.18 GB/s |
| 128KB | 48 | 128 | 256KB | 160.34 GB/s |
> **Note:** These results predate the EFA striping optimization. With the current code, `MC_SLICE_SIZE` no longer affects EFA performance. Use `--block_size=1048576` (1MB) instead, which achieves 222 GB/s.
</details>
#### p6-b300.48xlarge (B300, 16 EFA × 400 Gbps)
Tested on two p6-b300.48xlarge instances (Intel Xeon Platinum 8559C, 8× B300, 16 EFA devices) in the same AWS placement group.
**GPU-to-GPU** (build with `-DUSE_CUDA=ON`, `--gpu_id=-1` for all 8 GPUs, `--buffer_size=2147483648`):
| Configuration | Write | Read |
|---------------|-------|------|
| block=1MB, threads=16, batch=128 | 701 GB/s | **697 GB/s** |
| **block=1MB, threads=32, batch=64** | **752 GB/s** | 713 GB/s |
| block=1MB, threads=32, batch=32 | 751 GB/s | - |
| block=1MB, threads=64, batch=32 | 728 GB/s | - |
> **Peak: 752 GB/s write**, reaching ~94% of the 800 GB/s theoretical line rate (16×400 Gbps). GPUDirect RDMA bypasses DRAM entirely (HBM3e → PCIe switch → NIC), so performance is not bottlenecked by CPU memory bandwidth.
**CPU-to-CPU** (build with `-DUSE_CUDA=OFF`):
| Configuration | Write | Read |
|---------------|-------|------|
| **block=1MB, threads=32, batch=128, buf=4GB** | **230 GB/s** | 180 GB/s |
| block=16MB, threads=32, batch=8, buf=8GB (striping off) | 233 GB/s | - |
> CPU-to-CPU is bounded by DRAM bandwidth (~250 GB/s/socket on Xeon 8559C). Per-NIC sampling shows NUMA-0 NICs at 90 Gbps and NUMA-1 NICs at 53 Gbps, confirming DRAM controller saturation rather than NIC limit.
#### p5en.48xlarge (H200, 16 EFA × 200 Gbps)
#### 3. p5en.48xlarge (H200, 16 EFA × 200 Gbps)
Tested on two p5en.48xlarge instances (Intel Xeon 8488C, 8× H200 141GB, 16 EFA devices) in the same AWS placement group.
**GPU-to-GPU** (build with `-DUSE_CUDA=ON`, `--gpu_id=-1` for all 8 GPUs):
**GPU-to-GPU** (build with `-DUSE_CUDA=ON`, `--gpu_id=-1` for all 8 GPUs, `--buffer_size=4294967296`):
| Configuration | Write | Read |
|---------------|-------|------|
| block=1MB, threads=8, batch=128, buf=1GB/GPU | 236 GB/s | 271 GB/s |
| block=1MB, threads=16, batch=128, buf=2GB/GPU | 271 GB/s | **297-308 GB/s** |
| **block=1MB, threads=32, batch=64, buf=2GB/GPU** | **337-347 GB/s** | 274 GB/s |
| block=1MB, threads=8, batch=128 | 318.71 GB/s | 277.06 GB/s |
| **block=1MB, threads=16, batch=128** | **365.66 GB/s** | 284.22 GB/s |
| **block=1MB, threads=16, batch=32** | 297.23 GB/s | **303.78 GB/s** |
| block=1MB, threads=32, batch=64 | 357.12 GB/s | 279.61 GB/s |
| block=1MB, threads=32, batch=128 | 364.21 GB/s | 250.90 GB/s |
| block=1MB, threads=48, batch=64 | 363.47 GB/s | 268.43 GB/s |
> GPU HBM bandwidth (>3 TB/s) eliminates the memory bottleneck, allowing full EFA utilization. Write and read have different optimal thread counts: write peaks at 32 threads, read peaks at 16 threads.
> **Peak write: 365 GB/s** at `threads=16, batch=128` — ~91% of the 400 GB/s theoretical line rate (16×200 Gbps). Write saturates on batch size, so `batch=128` outperforms smaller batches as long as `threads × batch ≤ 16 × 256 = 4096` (the shared-endpoint WR cap). **Peak read: 304 GB/s** at `threads=16, batch=32` — reads tolerate smaller in-flight queues, and throughput drops as batch grows.
> **Note:** EFA memory region registration (fi_mr_reg) for GPU memory segfaults at 4GB+ per GPU. Use `--buffer_size=2147483648` (2GB) as the maximum per-GPU buffer.
**CPU-to-CPU** (build with `-DUSE_CUDA=OFF`):
**CPU-to-CPU** (build with `-DUSE_CUDA=OFF`, or `--use_vram=false` on a CUDA build, `--buffer_size=4294967296`):
| Configuration | Write | Read |
|---------------|-------|------|
| Single instance (block=1MB, threads=32, batch=128, buf=4GB) | 179 GB/s | 185 GB/s |
| NUMA-split (block=1MB, 2 instances, 8 NICs each, threads=16, buf=2GB) | **192 GB/s** | **182 GB/s** |
| block=1MB, threads=8, batch=128 | 210.76 GB/s | 209.93 GB/s |
| block=1MB, threads=16, batch=128 | 212.71 GB/s | 211.21 GB/s |
| **block=1MB, threads=16, batch=32** | 211.67 GB/s | **212.18 GB/s** |
| block=1MB, threads=32, batch=128 | 212.99 GB/s | 210.33 GB/s |
| **block=1MB, threads=48, batch=32** | **213.57 GB/s** | 206.92 GB/s |
> CPU-to-CPU throughput is bottlenecked by DRAM bandwidth (~155 GB/s per NUMA node, measured with STREAM Copy).
> CPU-to-CPU is DRAM-bound — throughput is essentially flat (~205–214 GB/s) across every thread / batch combination that doesn't hit the WR cap. Peak write 213.57 GB/s, peak read 212.18 GB/s.
#### Cross-Transport Comparison
**block_size sweep** (p5en GPU-to-GPU, `threads=16 batch=128 buf=4GB --gpu_id=-1`):
| Transport | Throughput | Notes |
|-----------|-----------|-------|
| **EFA GPU-to-GPU (B300)** | **752 GB/s** | p6-b300.48xlarge, 16×400G, block=1MB, ~94% line rate |
| **EFA GPU-to-GPU (H200)** | **347 GB/s** | p5en.48xlarge, 16×200G, block=1MB |
| **EFA GPU-to-GPU (B200)** | **313 GB/s** | p6-b200.48xlarge, 8×400G, block=1MB |
| **EFA CPU-to-CPU (B300)** | **230 GB/s** | p6-b300.48xlarge, 16×400G, block=1MB, DRAM-limited |
| **EFA CPU-to-CPU (B200)** | **222 GB/s** | p6-b200.48xlarge, 8×400G, block=1MB, DRAM-limited |
| **EFA CPU-to-CPU (H200)** | **192 GB/s** | p5en.48xlarge, block=1MB, NUMA-split, DRAM-limited |
| EFA (default params) | 69.47 GB/s | Default block=64KB |
| TCP (iperf3 baseline) | 9.5 GB/s | Kernel TCP stack, 8 parallel streams |
| block | Write | Read |
|-------|-------|------|
| 64 KB (default) | 97.11 GB/s | 96.87 GB/s |
| 128 KB | 190.82 GB/s | 203.88 GB/s |
| 256 KB | 324.90 GB/s | 296.19 GB/s |
| 512 KB | 357.62 GB/s | 289.47 GB/s |
| **1 MB (recommended)** | **352.59 GB/s** | **301.14 GB/s** |
| 2 MB | 366.87 GB/s | 302.25 GB/s |
**EFA vs RoCE RDMA**: On comparable 8×400 Gbps RoCE networks, Mooncake's RDMA transport achieves ~190 GB/s. Tuned EFA **exceeds** RoCE performance with GPU memory (313-347 GB/s) and on CPU-to-CPU (222 GB/s).
> The 64 KB default only reaches ~26% of peak. Write throughput climbs steeply up to ~512 KB and plateaus between 1 MB and 2 MB; read saturates at ~256 KB. 1 MB is the recommended value — within a few percent of the 2 MB peak with more headroom for `batch_size` under the shared-endpoint WR cap.
**buffer_size sweep** (p5en GPU-to-GPU, `threads=16 batch=128 block=1MB --gpu_id=-1`):
| buffer_size | Write | Read |
|-------------|-------|------|
| 2 GB (min: `block × batch × threads`) | 353.51 GB/s | 297.03 GB/s |
| 4 GB | 364.86 GB/s | 293.79 GB/s |
> `buffer_size` only needs to satisfy `buffer_size ≥ block_size × batch_size × threads` (the bench auto-adjusts if smaller, but silently). Anything larger than that minimum does not change throughput — 2 GB vs 4 GB differs by ~3% on write, read is flat within noise. The example commands use 4 GB because it is safe for any reasonable threads/batch combination without having to recompute the minimum.
#### 4. p5.48xlarge (H100, 32 EFA × 100 Gbps)
Tested on two p5.48xlarge instances (AMD EPYC 7R13, 8× H100 80GB, 32 EFA devices) in the same AWS placement group. Per-NIC line rate is half of p5en's, but with twice the NIC count the aggregate ceiling is the same 400 GB/s. The shared-endpoint WR cap scales with NIC count: `32 NICs × 256 = 8192` in-flight slots, so `threads × batch_size ≤ 8192` (vs 4096 on p5en).
**GPU-to-GPU** (build with `-DUSE_CUDA=ON`, `--gpu_id=-1` for all 8 GPUs, `--buffer_size=4294967296`):
| Configuration | Write | Read |
|---------------|-------|------|
| block=1MB, threads=8, batch=128 | 335.11 GB/s | - |
| block=1MB, threads=16, batch=128 | 388.52 GB/s | 379.10 GB/s |
| block=1MB, threads=32, batch=64 | 388.83 GB/s | 379.78 GB/s |
| **block=1MB, threads=32, batch=128** | **388.90 GB/s** | **381.64 GB/s** |
| block=1MB, threads=16, batch=32 | - | 356.94 GB/s |
| block=1MB, threads=32, batch=32 | - | 380.66 GB/s |
> **Peak write: 389 GB/s** at `threads=32, batch=128` — ~97% of the 400 GB/s theoretical line rate (32×100 Gbps). The plateau is wide: any `(threads, batch)` between `(16, 128)` and `(32, 128)` lands within 0.1% of peak. **Peak read: 382 GB/s** at `threads=32, batch=128` — unlike p5en, reads on this host scale with batch size up to 128 because the wider 32-NIC fabric absorbs larger in-flight queues without backoff. `(32, 256)` and `(64, 128)` (both at the 8192 WR cap) fail with no headroom for retries.
**CPU-to-CPU** (build with `-DUSE_CUDA=OFF`, or `--use_vram=false` on a CUDA build, `--buffer_size=4294967296`):
| Configuration | Write | Read |
|---------------|-------|------|
| block=1MB, threads=16, batch=128 | 39.83 GB/s | 40.43 GB/s |
| block=1MB, threads=32, batch=64 | 47.31 GB/s | 48.06 GB/s |
| block=1MB, threads=32, batch=128 | 47.75 GB/s | 48.62 GB/s |
| block=1MB, threads=32, batch=32 | 55.66 GB/s | 56.68 GB/s |
| **block=1MB, threads=48, batch=16** | **63.60 GB/s** | 63.00 GB/s |
| **block=1MB, threads=64, batch=16** | 63.39 GB/s | 63.05 GB/s |
| block=1MB, threads=32, batch=16 | 63.21 GB/s | 60.82 GB/s |
| block=1MB, threads=96, batch=32 | 57.61 GB/s | 59.63 GB/s |
> **Peak: ~64 GB/s** on both write and read — far below the GPU-to-GPU number despite identical NIC count. The bottleneck is DDR4-3200 DRAM bandwidth on the EPYC 7R13 (Milan): `batch=16` consistently wins because larger in-flight queues only deepen DRAM contention without unlocking new NIC capacity. p5.48xlarge CPU-to-CPU runs around **3× slower than p5en** (DDR5 Xeon 8488C, ~213 GB/s) at the same NIC aggregate. For PD KV transfer, the GPU-to-GPU path is the relevant one.
### Single-host loopback
EFA NICs have no hardware loopback short-circuit: when a transfer's source and destination resolve to the same host, the data does not go out on the wire as GPUDirect/device RDMA. libfabric handles the same-host case in software, and there are **two distinct provider knobs** that select how:
- **`FI_EFA_ENABLE_SHM_TRANSFER`** (default `1`, on): when on, the EFA provider routes same-host peers through the **`shm` provider** — verifiable at runtime, where libfabric reports `Opened fabric: shm` alongside `Opened fabric: efa` even on a default (device-RDMA-enabled) configuration. This SHM path is the one that supplies the same-host memcpy fast path; it is active **by default**, independent of `FI_EFA_USE_DEVICE_RDMA`.
- **`FI_EFA_USE_DEVICE_RDMA`** (default `1` after #2041): controls whether the EFA RDM data path uses device RDMA vs libfabric's emulated RDM path. It is a provider-level flag resolved at `fi_getinfo` time; Mooncake does not wrap it.
```{warning}
**GPU (FI_HMEM_CUDA) buffers — known segfault.** The default same-host **SHM**
path (`FI_EFA_ENABLE_SHM_TRANSFER=1`) performs a **host `memcpy` into the
destination buffer** during SHM SAR reassembly, *without* honoring an
`FI_HMEM_CUDA` destination's iface. On a GPU buffer this writes host memory
straight into a device pointer and **segfaults** on the first same-host
transfer — `__memcpy_avx_unaligned` ← `ofi_copy_to_mr_iov` ← `smr_copy_from_sar`
← `efa_rdm_cq_readfrom` ← `fi_cq_read`. Reported upstream as
[ofiwg/libfabric#12328](https://github.com/ofiwg/libfabric/issues/12328).
- **Same-process self-loopback** (e.g. a TP-colocated rank reading its own
registered GPU weights — checkpoint-engine p2p weight update) is handled
inside Mooncake: `EfaContext::tryLoopbackCopy` detects a same-process peer
(matched on `local_server_name`, which embeds this process's unique RPC
port) and satisfies the transfer with a local `cudaMemcpy` instead of routing
it over EFA, so it never reaches the broken SHM path.
- **Same-host cross-process GPU transfers** are *not* short-circuited (the
peer is a different process / address space). Until libfabric#12328 is fixed,
set `FI_EFA_ENABLE_SHM_TRANSFER=0` on such processes — same-host transfers
then fall back to device RDMA, which is GPU-aware and correct.
```
For **host (DRAM) buffers** the SHM memcpy path is safe (host→host copy) and is the same-host fast path the measurements below exercise.
Measured on p5.48xlarge (1 NIC, ~1.2 GiB per `put_from` call, host DRAM buffer, same-host producer/consumer in **separate processes**):
| same-host path | per-write latency |
|---|---:|
| device RDMA (NIC round-trip, no fast-path for loopback) | ~830 ms |
| SHM memcpy fast path (default) | ~390 ms |
For reference, a cross-host `put_from` of the same payload (device RDMA, 1 NIC) is ~340 ms — i.e., driving a same-host loopback through the NIC is *slower* than going over the wire to another host, because the NIC has no fast-path for loopback. Cross-host transfers always use device RDMA and are unaffected by `FI_EFA_ENABLE_SHM_TRANSFER`: leave it at its default on any process that also talks to remote peers.
### Tuning Tips
- **Use `--block_size=1048576` (1MB)** — this is the most important tuning parameter for EFA. Each `block_size`-sized transfer becomes a single `fi_write`/`fi_read` call, so larger blocks amortize per-operation overhead. 1MB gives ~2× throughput over the 64KB default.
- `MC_SLICE_SIZE` has **no effect** on EFA transport (it only applies to RDMA transport). Use `block_size` instead.
- Increase `--threads` to 32-48 to saturate multiple EFA devices (2-4 threads per device is a good starting point)
- For **CPU-to-CPU**: use `--block_size=1048576` (1MB) with NUMA-split (separate instances per NUMA node) for best results
- For **GPU-to-GPU**: use `--block_size=1048576` (1MB), `--gpu_id=-1` (all GPUs), and `--buffer_size=2147483648` (2GB max per GPU). Write peaks at threads=32, read at threads=16
- Keep `--batch_size` such that `block_size * batch_size * threads <= buffer_size`
- Allocate buffers on both NUMA nodes for balanced NIC utilization (the bench tool does this by default for CPU mode)
- On 16-NIC instances (p5en), writes are NUMA-sensitive: 8 local-NUMA NICs reach 90 Gbps each, while 8 cross-NUMA NICs only reach ~20 Gbps without NUMA-split
- **Use `--block_size=1048576` (1MB)** — the single most important knob. The 64 KB default reaches only ~26% of peak. 1 MB is within a few percent of the 2 MB plateau while leaving headroom for `batch_size` under the shared-endpoint WR cap.
- **Keep `threads × batch_size ≤ num_nics × max_wr`** — under the SRD shared endpoint each NIC carries one `fid_ep` with a 256 WR cap (`MC_MAX_WR`), giving `16 NICs × 256 = 4096` in-flight slots on a 16-NIC host (b300, b200, p5en) and `32 NICs × 256 = 8192` on p5. Exceeding this trips "timed out waiting for CQ drain". `threads=16, batch=128` is a solid baseline on 16-NIC hosts; on 32-NIC p5, `threads=32, batch=128` works the same way.
- **Write vs read:** write benefits from larger batches (peak at `batch=128`); on 16-NIC p5en read prefers smaller queues (peak at `batch=32`), but on 32-NIC p5 reads scale up to `batch=128` because the wider fabric absorbs larger in-flight queues.
- For **GPU-to-GPU**: pass `--gpu_id=-1` on **both** sides so buffers fan out across every GPU. Pinning a single GPU halves throughput because half the NICs end up cross-NUMA.
- For **CPU-to-CPU**: DRAM bandwidth is the ceiling. NUMA-split (separate initiator/target instances per NUMA node) can help reduce contention when one instance can't saturate both nodes.
- `--buffer_size` only needs `≥ block × batch × threads`; larger values do not improve throughput. The example commands use 4 GB because that is safe for any reasonable config.
### Eager endpoint warmup (first-request latency)
### First-request latency
libfabric `FI_EP_RDM` endpoints resolve peer addresses lazily: `fi_av_insert()` and the metadata handshake fire on the first send to each `(local_ctx, peer_nic)` pair. On 16-NIC instances that gives `16 × N_peer_NICs` serial handshakes inside the first `submitTransfer`, which shows up as a single-digit-second first-batch stall (measured ~4 s on p6-B300 for a 100 × 0.5 MB batch; the first batch runs at <0.1 GB/s while the CQ drains, steady-state afterwards is unaffected).
Peer addressing resolves lazily: `fi_av_insert()` and the metadata handshake fire on the first send to each `(local_NIC, peer_NIC)` pair. On 16-NIC hosts, the first few `submitTransfer` calls carry this cost before steady state.
Mooncake exposes an explicit eager-warmup API to eliminate the stall:
**Measured on p5en (16 × 16 NICs, cross-node, 1 MB write, 3 reps, median):**
- C++: `EfaTransport::warmupSegment(const std::string& segment_name)`
- C: `int warmupEfaSegment(transfer_engine_t engine, const char *segment_name)`
- Rust: `TransferEngine::warmup_efa_segment(name: &str)`
Call it once per peer segment, right after `openSegment` (or after any metadata change that adds a new peer). Every `(local_ctx, peer_nic)` endpoint is connected concurrently via `std::async`; the critical path becomes `max(handshake RTT)` instead of `sum(handshake RTT)`. The call is idempotent — safe to re-run.
Measured on p6-B300 (16 local NICs × 16 peer NICs, dual-NUMA initiator, 100 × 0.5 MB batch):
| | first-batch latency | steady-state |
| | cold submit #0 (no warmup) | `warmupSegment()` (all 256 pairs) |
|---|---:|---:|
| No warmup | 4,043 ms | 141 GB/s |
| `warmup_efa_segment` (256 endpoints connected in 4.1 s) | **13.5 ms** (~300×) | 230 GB/s |
| SRD shared endpoint (#1944) | **26 ms** | **1.1 s** |
| Per-peer `fid_ep` (upstream main) | 99 ms | 17 s |
| Speedup | **~4×** | **~15×** |
The warmup call itself takes roughly the same wall time as the stall it replaces — the win is that it's a one-time setup cost decoupled from the critical path of the first real transfer, not paid inside your latency budget.
The SRD shared-endpoint refactor (#1944) speeds up first-request latency two different ways:
- **Without any code change from callers** — the cold `submitTransfer` is ~4× faster (26 ms vs 99 ms), because the shared endpoint removes the per-peer `fi_endpoint` / `fi_enable` that used to dominate. This is what existing Mooncake callers (vLLM, SGLang, etc.) will see.
- **For callers that want sub-10 ms first-request latency**, an explicit eager-warmup API lets you pay the handshake cost up front, outside the critical path:
- C++: `EfaTransport::warmupSegment(const std::string& segment_name)`
- C: `int warmupEfaSegment(transfer_engine_t engine, const char *segment_name)`
- Rust: `TransferEngine::warmup_efa_segment(name: &str)`
- Python: `engine.warmup_efa_segment(segment_name)`
Call once per peer right after `openSegment`. The call is idempotent. Under this refactor `warmupSegment` itself is ~15× faster than the pre-#1944 code (1.1 s vs 17 s), bounded by the peer's single-threaded handshake RPC daemon (`accept` + JSON parse serialized on one thread), so it scales linearly with the number of fresh NIC pairs.
vLLM and SGLang do not currently call `warmupSegment` — they go through the generic `TransferEngine` interface and pick up the 4× cold-submit speedup automatically. The API is there for direct Mooncake callers that want the larger win.
#### Reproducing the numbers — `efa_first_submit_probe`
`mooncake-transfer-engine/example/efa_first_submit_probe.cpp` is a two-host probe that isolates the two latency costs `transfer_engine_bench` hides inside its 10-second throughput average:
1. **Cold first-submit** — the handshake + `fi_av_insert()` that fires on the first send to each `(local_NIC, peer_NIC)` pair.
2. **Eager warmup** — how much of that is paid up front by an explicit `warmupSegment()` call.
It is EFA-specific (there is no `warmupSegment` on the RDMA / TCP transports — they establish connections at `connect` time, so "pre-warming" has no meaning there) and intentionally not wired into `ctest`: it needs two hosts.
**Run**:
```bash
# On the target host:
./build/mooncake-transfer-engine/example/efa_first_submit_probe \
--mode=target \
--metadata_server=P2PHANDSHAKE \
--local_server_name=$(hostname):12345
# Note the "[target] ready, addr=<ip>:<port>" line, then on the initiator:
./build/mooncake-transfer-engine/example/efa_first_submit_probe \
--mode=initiator \
--metadata_server=P2PHANDSHAKE \
--local_server_name=$(hostname):12346 \
--segment_id=<target_ip>:<target_port> \
--warmup=1 \
--iters=5
```
**What it prints**:
```
warmup: 1094.74 ms (rc=0) # present only with --warmup=1
submit #0: 6.95 ms # first user-visible submit
submit #1: 5.45 ms # steady state
submit #2: 5.92 ms
submit #3: 5.67 ms
submit #4: 0.09 ms
```
- `warmup:` is the wall time of `EfaTransport::warmupSegment()` — it opens every `(local_NIC, peer_NIC)` pair (16 × 16 = 256 on p5en) up front.
- `submit #0` through `#N-1` are the timings of single 1 MB `submitTransfer + poll`. With `--warmup=1` they are all in the steady-state regime. With `--warmup=0`, `#0` pays the handshake cost, `#1+` are steady state.
**Flags**:
| Flag | Default | What it controls |
|---|---|---|
| `--warmup` | `true` | Call `warmupSegment()` before the first submit |
| `--iters` | `5` | How many timed submits after warmup |
| `--xfer_size` | `1<<20` (1 MB) | Bytes per submit |
| `--buffer_size` | `1<<30` (1 GB) | Registered buffer size |
| `--local_server_name` | `hostname:12345` | Local metadata advertise name |
**Use cases**:
- **Deciding whether your application needs `warmupSegment()`**: if `submit #0` in the `--warmup=0` run is acceptable for your use case, you don't need to call `warmupSegment` at all.
- **Comparing PR branches**: run the probe on the same hardware on this PR's branch vs `main` (or whatever upstream you're benchmarking against) to see per-NIC-pair handshake cost directly, without having the number drowned in a 10-second throughput average.
## Usage with vLLM
### Prefill Instance
### 1. Prefill Instance
```bash
VLLM_MOONCAKE_BOOTSTRAP_PORT=8998 \
@ -332,7 +511,7 @@ vllm serve <model_path> -tp 8 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_connector_extra_config":{"mooncake_protocol":"efa"}}'
```
### Decode Instance
### 2. Decode Instance
```bash
vllm serve <model_path> -tp 8 \
@ -341,105 +520,166 @@ vllm serve <model_path> -tp 8 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"mooncake_protocol":"efa"}}'
```
### 3. Router
Front the prefill / decode pair with `vllm-router`. **`PREFILL_HOST` / `DECODE_HOST` must be each node's reachable private IP, not `127.0.0.1`** — the router forwards these addresses to the peer for the KV handshake; localhost will fail with `Connection refused` and stall traffic.
```bash
vllm-router --policy round_robin \
--vllm-pd-disaggregation \
--prefill http://<prefill_ip>:8010 \
--decode http://<decode_ip>:8020 \
--kv-connector mooncake \
--host 0.0.0.0 --port 30000
```
> **Do not add `--intra-node-data-parallel-size` here.** The prefill / decode instances above are launched with `-tp 8` (pure tensor parallelism, data parallel size = 1), so there is no intra-node DP to advertise. Only pass `--intra-node-data-parallel-size N` when your instances actually run `N`-way data parallelism per node (e.g. you launched them with `--data-parallel-size N`); setting it to match `-tp 8` is wrong and will misroute requests.
## Usage with SGLang
SGLang's Mooncake integration currently hardcodes the `"rdma"` protocol. To use EFA transport, apply the provided patch and set environment variables.
SGLang's PD-disaggregation Mooncake integration reads the transport from `MOONCAKE_PROTOCOL`. Set it to `efa` and select the Mooncake backend with `--disaggregation-transfer-backend mooncake`.
### 1. Apply EFA Patch
### 1. Apply EFA Patch (only if SGLang version predates PR #25083)
SGLang's transfer engine initialization needs to be patched to read the protocol from an environment variable instead of using hardcoded `"rdma"`. Use the [patch script](https://github.com/whn09/kimi-k2-sglang):
Older SGLang releases hardcode `"rdma"` in the transfer engine init. [SGLang PR #25083](https://github.com/sgl-project/sglang/pull/25083) has been **merged into SGLang `main`**, so the protocol is now read from `MOONCAKE_PROTOCOL`. If your SGLang build includes that PR (any recent `main` or release built after it), **this step is unnecessary** — skip to step 2.
Only if you are pinned to an older release that predates PR #25083, apply the [patch script](https://github.com/whn09/kimi-k2-sglang/blob/main/patch_sglang_efa.sh):
```bash
bash patch_sglang_efa.sh
```
This is idempotent and safe to rerun.
The script is idempotent and safe to rerun.
### 2. Environment Variables
Only one Mooncake-specific env is required:
```bash
export MOONCAKE_PROTOCOL=efa
```
If your container does not already export libfabric/EFA paths in its `Dockerfile`, also set:
```bash
export FI_PROVIDER=efa
export FI_EFA_USE_DEVICE_RDMA=1
export GLOO_SOCKET_IFNAME=enp71s0 # adjust to your instance's primary interface
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:$LD_LIBRARY_PATH
```
For multi-node expert parallelism (EP) deployments, also set:
> **Note on additional `MC_*` knobs:** `MC_NUM_CQ_PER_CTX`, `MC_MAX_WR`, `MC_MAX_CQE_PER_CTX`, `MC_SLICE_SIZE`, and `MC_EFA_STRIPING_THRESHOLD` are **not** required at typical PD-disagg loads — the SRD shared-endpoint refactor (#1944) makes them redundant up to high concurrency on 1k/1k traffic. Treat them as emergency switches for CQ-overflow or long-running drift symptoms.
> **`MC_EFA_CQ_THREADS`** — caps the number of CQ polling threads spawned by the EFA transport. Default is `1`, which reaches 99.93% of peak GPU-to-GPU throughput while saving CPU for other workloads. Set to `0` to disable the cap (one poller per EFA context — the legacy behavior). Higher values (e.g., `MC_EFA_CQ_THREADS=4`) are available as an escape hatch for throughput tuning but rarely help in practice.
>
> ```bash
> export MC_EFA_CQ_THREADS=1 # default: single CQ poller (recommended)
> export MC_EFA_CQ_THREADS=0 # disable cap: one poller per EFA context (legacy)
> ```
>
> If the value exceeds the number of EFA contexts, it is safely ignored (no excess threads are created).
### 3. Prefill Instance
```bash
export NVSHMEM_REMOTE_TRANSPORT=libfabric
export NVSHMEM_LIBFABRIC_PROVIDER=efa
MOONCAKE_PROTOCOL=efa \
sglang serve <model_path> \
--trust-remote-code \
--tp 8 --dp 2 --enable-dp-attention --enable-dp-lm-head \
--host 0.0.0.0 --port 8010 \
--disaggregation-mode prefill \
--disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998
```
> **Warning:** Do **not** set NVSHMEM variables on single-node deployments — doing so causes segmentation faults.
### 3. Docker Launch Example
### 4. Decode Instance
```bash
docker run -d --name sglang \
--runtime=nvidia --gpus all --network host \
--privileged --shm-size=600g \
--device=/dev/infiniband \
-e MOONCAKE_PROTOCOL=efa \
-e FI_PROVIDER=efa \
-e FI_EFA_USE_DEVICE_RDMA=1 \
<image> bash start.sh
MOONCAKE_PROTOCOL=efa \
sglang serve <model_path> \
--trust-remote-code \
--tp 8 --dp 2 --enable-dp-attention --enable-dp-lm-head \
--host 0.0.0.0 --port 8020 \
--disaggregation-mode decode \
--disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998
```
> **Note:** Ensure the Docker image's libfabric version matches the host's EFA driver. If not, mount the host's EFA libraries into the container (see [Troubleshooting](#libfabric-version-mismatch-in-docker)).
### 5. Router
Front the pair with `sglang_router` from the prefill host. **`PREFILL_HOST` must be the prefill node's reachable IP, not `127.0.0.1`** — the router forwards this address to the decode node for the bootstrap_room handshake; localhost will fail with `Connection refused` and stall traffic at 0/N.
```bash
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--prefill "http://<prefill_ip>:8010" 8998 \
--decode "http://<decode_ip>:8020" \
--policy round_robin \
--host 0.0.0.0 --port 8000
```
The trailing `8998` after `--prefill` must match the prefill's `--disaggregation-bootstrap-port`.
## Technical Details
### Why libfabric instead of ibverbs?
AWS EFA exposes RDMA-like devices through the ibverbs interface, but does not support the full ibverbs API. Specifically:
- Queue Pair (QP) creation fails with "Operation not supported" (error 95)
- EFA requires using libfabric's `FI_EP_RDM` (Reliable Datagram Message) endpoint type
AWS EFA exposes an RDMA-capable device through the ibverbs interface, but it does **not** implement the full ibverbs API. In particular, EFA only supports **SRD** (Scalable Reliable Datagram) and **UD** (Unreliable Datagram) queue pairs — it does **not** support the **RC** (Reliable Connection) queue pairs that Mooncake's RDMA (`rdma`) transport is built on. Attempting to create an RC QP on an EFA device fails (`EOPNOTSUPP`), and SRD has no one-sided RC-style `ibv_post_send(RDMA_WRITE)` verb in the public ibverbs API.
The portable way to drive EFA's SRD transport is libfabric, whose EFA provider exposes SRD through the `FI_EP_RDM` (Reliable Datagram Message) endpoint type and implements `fi_write` / `fi_read` (one-sided RMA) on top of it. Mooncake's EFA transport therefore targets libfabric directly rather than ibverbs.
### EFA Transport Architecture
Under the SRD shared-endpoint model every peer is addressed through one `fid_ep` per local NIC — peers are AV-slot entries, not separate endpoints.
```
┌─────────────────────────────────────────────────────┐
│ EfaTransport │
├─────────────────────────────────────────────────────┤
│ EfaContext (per device) │
│ ├── fi_info (fabric info) │
│ ├── fid_fabric (fabric handle) │
│ ├── fid_domain (protection domain) │
│ ├── fid_av (address vector for peer lookup) │
│ ├── fid_cq (completion queues) │
│ └── fid_mr (memory regions) │
├─────────────────────────────────────────────────────┤
│ EfaEndpoint (per connection) │
│ ├── fid_ep (RDM endpoint) │
│ ├── fi_addr_t (peer address) │
│ └── local_addr (local endpoint address) │
└─────────────────────────────────────────────────────┘
┌───────────────────────────────────────────────────────────┐
│ EfaTransport │
├───────────────────────────────────────────────────────────┤
│ EfaContext (per local NIC) │
│ ├── fid_fabric (fabric handle) │
│ ├── fid_domain (protection domain) │
│ ├── fid_av (address vector — one slot per peer) │
│ ├── fid_cq (completion queues) │
│ ├── fid_mr (memory regions) │
│ ├── shared_ep_ (the single fid_ep that serves every │
│ │ peer via fi_addr_t lookup in the AV) │
│ └── peer_map_ (full "host:port@nic" path -> │
│ EfaEndPoint; the RPC port is NOT │
│ stripped — see note below) │
├───────────────────────────────────────────────────────────┤
│ EfaEndPoint (per peer) │
│ └── peer_fi_addr_ (AV slot index for this peer; sends │
│ route through the owning context's │
│ shared_ep_ with this fi_addr_t) │
└───────────────────────────────────────────────────────────┘
```
> **Peer-map keying.** `peer_map_` is keyed by the **full** `host:port@nic` path, *not* a port-stripped form. Under SGLang DP > 1 each DP worker on a peer host is a separate process with its own Mooncake `TransferEngine` and its own P2PHANDSHAKE RPC port; they share host + NIC but have distinct EFA addresses. Normalizing the port away would collapse every DP worker on that host onto one `EfaEndPoint`, so each arriving handshake would look like a "peer reconnected" to the previous holder and trigger `fi_av_remove` + `fi_av_insert` churn on every KV transfer. Keeping the port in the key costs nothing in steady state (the port is stable for a worker's lifetime).
### Thread Safety
The EFA transport requests `FI_THREAD_SAFE` from the libfabric provider and adds per-endpoint spinlocks to serialize `fi_write`/`fi_read` calls. This is necessary because:
The EFA transport requests `FI_THREAD_SAFE` at the domain level and guards the shared endpoint with a single `post_lock_` spinlock (one per `EfaContext`, i.e. one per local NIC) to serialize `fi_write`/`fi_read` calls. This is necessary because:
- Multiple submission threads may route slices to the same endpoint concurrently
- libfabric RDM endpoints default to `FI_THREAD_UNSPEC` (no thread safety guarantees)
- Concurrent `fi_write`/`fi_read` without serialization corrupts provider internals, causing completions to silently vanish
- Multiple submission threads may route slices through the same shared endpoint concurrently.
- libfabric's EFA RDM endpoints are not thread-safe for concurrent `fi_write`/`fi_read` even under `FI_THREAD_SAFE` at the domain level — concurrent posts corrupt provider internals and completions silently vanish.
CQ completion queues are polled by dedicated worker threads (one per EFA device) that run independently of submission threads.
CQ completion queues are polled by dedicated worker threads that run independently of submission threads. The poller count is `min(MC_EFA_CQ_THREADS, num_EFA_devices)`; `MC_EFA_CQ_THREADS` defaults to `1`, so a single poller round-robins every context's CQ (which already reaches ~99.9% of peak — see the SGLang env-var note above). Set `MC_EFA_CQ_THREADS=0` to lift the cap and spawn one poller per EFA device (the legacy behavior).
### EFA vs RoCE RDMA
| Feature | EFA (libfabric SRD) | RoCE (ibverbs) |
|---------|--------------------|--------------------|
| Protocol | Scalable Reliable Datagram | RDMA over Converged Ethernet |
| Endpoint type | `FI_EP_RDM` (message-based) | Queue Pairs (true RDMA) |
| Write operation | Software-emulated via messages + ACKs | Hardware-offloaded one-sided RDMA |
| CPU overhead | Moderate (provider processes ACKs) | Minimal (NIC handles everything) |
| Throughput CPU-to-CPU (8×400G) | 222 GB/s (tuned) | ~190 GB/s |
| Throughput GPU-to-GPU (16×200G) | 347 GB/s (tuned) | N/A |
| Throughput GPU-to-GPU (8×400G) | 313 GB/s (tuned) | N/A |
| Protocol | Scalable Reliable Datagram (SRD) | RDMA over Converged Ethernet |
| QP type | SRD / UD (no RC) | RC (Reliable Connection) |
| Endpoint type | `FI_EP_RDM` (connectionless) | Queue Pairs (connection-oriented) |
| Reliability / ordering | Reliable delivery, **unordered** (SRD sprays across paths) | Reliable, in-order |
| Write operation | One-sided `fi_write` over SRD; device-RDMA-offloaded by default (`FI_EFA_USE_DEVICE_RDMA=1`), falls back to libfabric's emulated RMA only if device RDMA is disabled | Hardware-offloaded one-sided RDMA |
| Throughput GPU-to-GPU (16×200G, p5en) | 365 GB/s (tuned) | N/A |
| Throughput CPU-to-CPU (16×200G, p5en) | 213 GB/s (tuned) | — |
| AWS availability | All EFA-enabled instances | Not available on AWS |
> Mooncake requests libfabric API ≥ 1.18 at `fi_getinfo`, which makes `FI_EFA_USE_DEVICE_RDMA=1` the default on every supported EFA generation (p5/p5e included). On this path `fi_write` / `fi_read` are hardware-offloaded one-sided RMA over SRD — the host CPU is not in the data path. The software-emulated RMA path only applies if you explicitly set `FI_EFA_USE_DEVICE_RDMA=0`.
### Supported AWS Instance Types
- p6-b300.48xlarge (16 EFA devices × 400 Gbps = 6,400 Gbps, `rdmap*` naming)

46
docs/source/design/transfer-engine/index.md
View File

@ -13,6 +13,7 @@ As shown in the diagram, each specific client corresponds to a `TransferEngine`,
Mooncake Transfer Engine provides interfaces through the `TransferEngine` class (located in `mooncake-transfer-engine/include/transfer_engine.h`), where the specific data transfer functions for different backends are implemented by the `Transport` class, currently supporting `TcpTransport`, `RdmaTransport`, `EfaTransport`, `NVMeoFTransport`, `NvlinkTransport`, `IntraNodeNvlinkTransport`, and `HipTransport`.
(segment)=
### Segment
Segment represents a collection of source address ranges and target address ranges available during the data transfer process in Transfer Engine. That is, all local and remote addresses involved in `BatchTransfer` requests must be within the valid segment range. Transfer Engine supports the following two types of Segments.
@ -28,6 +29,7 @@ In addition, Transfer Engine also supports registering some **local DRAM areas**
#### 2. NVMeof Segment
Transfer Engine also leverages the NVMeof protocol to support direct data transfer from files on NVMe to DRAM/VRAM via PCIe, without going through the CPU and achieving zero-copy. Users need to follow the instructions to mount remote storage nodes locally and use the `openSegment` interface for reference to complete data read/write operations.
(batchtransfer)=
### BatchTransfer
With the help of Transfer Engine, Mooncake Store can achieve local DRAM/VRAM reading and writing of specified parts in valid segments through TCP, (GPUDirect) RDMA, NVMe-of protocols, etc.
@ -75,6 +77,8 @@ If a connection fails due to link errors, it is removed from the
endpoint pools on both sides and re-established during the
next data transfer attempt.
Evicted and deleted endpoints are moved to an internal `waiting_list_` and reclaimed asynchronously once their outstanding slices drain. Reclaim runs on every new endpoint insertion, and additionally on a ~1 Hz heartbeat from the per-context `monitorWorker`, so the waiting list drains even under failure load where new insertions stall while evictions continue.
### Fault Handling
In a multi-NIC environment, one common failure scenario is the temporary unavailability of a specific NIC, while other routes may still connect two nodes.
Mooncake Store is designed to adeptly manage such temporary
@ -146,7 +150,7 @@ After successfully compiling Transfer Engine, the test program `transfer_engine_
The initiator node can also configure the following test parameters: `--operation` (can be `"read"` or `"write"`), `batch_size`, `block_size`, `duration`, `threads`, etc.
> [!NOTE]
> If an exception occurs during execution, it is usually due to incorrect parameter settings. It is recommended to refer to the [troubleshooting document](troubleshooting.md) for preliminary troubleshooting.
> If an exception occurs during execution, it is usually due to incorrect parameter settings. It is recommended to refer to the [troubleshooting document](../../troubleshooting/troubleshooting.md) for preliminary troubleshooting.
### Sample Run
@ -165,7 +169,7 @@ Transfer Engine provides batch-based read/write transfers between segments (DRAM
### Multi-Transport Management
The `TransferEngine` class internally manages multiple backend `Transport` classes.
And it will discover the toplogy between CPU/CUDA and RDMA devices automatically
And it will discover the topology between CPU/CUDA and RDMA devices automatically
(more device types are working in progress, feedbacks are welcome when the automatic discovery mechanism is not accurate),
and it will install `Transport` automatically based on the topology.
@ -250,7 +254,7 @@ The HTTP server should implement three following RESTful APIs, while the metadat
2. `PUT /metadata?key=$KEY`: Update the metadata corresponding to `$KEY` to the value of the request body.
3. `DELETE /metadata?key=$KEY`: Delete the metadata corresponding to `$KEY`.
For specific implementation, refer to the demo service implemented in Golang at [mooncake-transfer-engine/example/http-metadata-server](../../../mooncake-transfer-engine/example/http-metadata-server).
For specific implementation, refer to the demo service implemented in Golang at [mooncake-transfer-engine/example/http-metadata-server](gh-dir:mooncake-transfer-engine/example/http-metadata-server).
## Using Transfer Engine to Your Projects
@ -263,7 +267,7 @@ To support the operational needs of P2P Store, Transfer Engine provides a Golang
When compiling the project, enable the `-DWITH_P2P_STORE=ON` option to compile the P2P Store example program at the same time.
### Using Rust Interface
Under `mooncake-transfer-engine/rust`, the Rust interface implementation of TransferEngine is provided, and a Rust version of the benchmark is implemented based on the interface, similar to [transfer_engine_bench.cpp](../../../mooncake-transfer-engine/example/transfer_engine_bench.cpp). To compile the rust example, you need to install the Rust SDK and add `-DWITH_RUST_EXAMPLE=ON` in the cmake command.
Under `mooncake-transfer-engine/rust`, the Rust interface implementation of TransferEngine is provided, and a Rust version of the benchmark is implemented based on the interface, similar to [transfer_engine_bench.cpp](gh-file:mooncake-transfer-engine/example/transfer_engine_bench.cpp). To compile the rust example, you need to install the Rust SDK and add `-DWITH_RUST_EXAMPLE=ON` in the cmake command.
## Advanced Runtime Options
For advanced users, TransferEngine provides the following advanced runtime options, all of which can be passed in through **environment variables**.
@ -272,8 +276,12 @@ For advanced users, TransferEngine provides the following advanced runtime optio
- `MC_NUM_COMP_CHANNELS_PER_CTX` The number of Completion Channel created per device instance, default value 1
- `MC_IB_PORT` The IB port number used per device instance, default value 1
- `MC_IB_TC` Adjust RDMA NIC Traffic Class when switch/NIC defaults differ or for traffic planning. Default value -1
- `MC_IB_SL` Set the InfiniBand Service Level (0-15) of RDMA QPs. The switch maps SL to a Virtual Lane for QoS isolation, e.g. to steer KV-cache traffic into a different VL than Expert-Parallel all-to-all traffic that shares the same NIC. -1 keeps the default (0). Default value -1
- `MC_IB_PCI_RELAXED_ORDERING` Setting the PCIe ordering to relaxed for the network adapter sometimes results in better performance. Can set 1 to enable RO function. Default value 0
- `MC_MLX5_QP_UDP_SPORTS` Comma-separated list of UDP source ports (0-65535) used to override the RoCEv2 UDP source port of each QP, for spreading traffic across different ECMP/LAG paths. QP at index *i* uses `list[i % size]`. Default empty (driver chooses). **Requires** an mlx5 NIC + RoCEv2, and the binary built with `-DUSE_MLX5DV=ON`. Recommend ports in the dynamic range 49152-65535. Example: `MC_MLX5_QP_UDP_SPORTS="49152,49153,49154,49155"`
- `MC_MLX5_QP_LAG_PORT_BALANCE` Set to `1` or `true` to enable automatic LAG port balancing across bonded physical ports. QP at index *i* is pinned to port `(i % num_lag_ports) + 1`; the number of LAG ports is queried from hardware via `mlx5dv_query_device` at startup and printed in the device log. If the device is not in LAG mode the setting is a no-op. Default: disabled. **Requires** the binary built with `-DUSE_MLX5DV=ON`. Example: `MC_MLX5_QP_LAG_PORT_BALANCE=1`
- `MC_GID_INDEX` The GID index used per device instance, default value 3 (or the maximum value supported by the platform)
- `MC_PKEY_INDEX` The QP `pkey_index` (partition key table index) used when transitioning the QP to the INIT state. Valid range: 0 to 65535. Default value 0. Set this when the partition key required for your fabric is not at index 0 of the HCA's pkey table
- `MC_MAX_CQE_PER_CTX` The CQ buffer size per device instance, default value 4096
- `MC_MAX_EP_PER_CTX` The maximum number of active EndPoint per device instance, default value 65536. **Note:** For versions prior to 0.3.7.post1, the default value is 256, and it cannot be manually set to 65536. The maximum supported value is 65535!
- `MC_NUM_QP_PER_EP` The number of QPs per EndPoint, the more the number, the better the fine-grained I/O performance, default value 2
@ -284,9 +292,11 @@ For advanced users, TransferEngine provides the following advanced runtime optio
- `MC_WORKERS_PER_CTX` The number of asynchronous worker threads corresponding to each device instance
- `MC_SLICE_SIZE` The segmentation granularity of user requests in Transfer Engine
- `MC_RETRY_CNT` The maximum number of retries in Transfer Engine
- `MC_AUTO_GID_MAX_RETRIES` The maximum number of automatic local GID reprobe retries during classic RDMA handshake recovery. Default value 2. Set to 0 to disable automatic GID retry.
- `MC_LOG_LEVEL` This option can be set as `TRACE`/`INFO`/`WARNING`/`ERROR` (see [glog doc](https://github.com/google/glog/blob/master/docs/logging.md)), and more detailed logs will be output during runtime
- `MC_DISABLE_METACACHE` Disable local meta cache to prevent transfer failure due to dynamic memory registrations, which may downgrades the performance
- `MC_HANDSHAKE_LISTEN_BACKLOG` The backlog size of socket listening for handshaking, default value is 128
- `MC_HANDSHAKE_CONNECT_TIMEOUT` Connect timeout in seconds for outbound handshake-port requests (QP handshake, probe, notify, metadata exchange), default value is 5. Bounds the stall when the peer address is unreachable; without it, a connect to an unroutable address (e.g. a removed node) blocks for the kernel's full TCP SYN retry cycle, which can take minutes
- `MC_HANDSHAKE_MAX_LENGTH` The maximum handshake message length in bytes for P2P mode. Valid range: 1MB to 128MB. Default value is 1MB (1048576 bytes). Increase this value when using a single RDMA instance with many registered memory buffers (>10,000) to avoid handshake failures. Example: set to 10485760 for 10MB
- `MC_LOG_DIR` Specify the directory path for log redirection files. If invalid, log to stderr instead.
- `MC_REDIS_PASSWORD` The password for Redis storage plugin, only takes effect when Redis is specified as the metadata server. If not set, no authentication will be attempted to log in to the Redis.
@ -298,19 +308,26 @@ For advanced users, TransferEngine provides the following advanced runtime optio
- `MC_FORCE_MNNVL` Force to use Multi-Node NVLink as the active transport regardless whether RDMA devices are installed.
- `MC_INTRA_NVLINK` Enable intra-node NVLINK transport, and cannot be used together with MC_FORCE_MNNVL.
- `MC_FORCE_TCP` Force to use TCP as the active transport regardless whether RDMA devices are installed.
- `MC_MIN_PRC_PORT` Specifies the minimum port number for RPC service. The default value is 15000.
- `MC_MAX_PRC_PORT` Specifies the maximum port number for RPC service. The default value is 17000.
- `MC_MIN_RPC_PORT` Specifies the minimum port number for RPC service. The default value is 15000.
- `MC_MAX_RPC_PORT` Specifies the maximum port number for RPC service. The default value is 17000.
- `MC_PATH_ROUNDROBIN` Use round-robin mode in the RDMA path selection. This may be beneficial for transferring large bulks.
- `MC_TE_FILTERS` Optional comma-separated whitelist of IB device names (e.g. `mlx5_0,mlx5_2`) for legacy Transfer Engine topology discovery. When unset, all available devices are discovered.
- `WITH_NVIDIA_PEERMEM` When set to `1`, `ON`, or `TRUE`, Mooncake uses `ibv_reg_mr()` directly for GPU memory registration (requires the `nvidia-peermem` kernel module). By default (unset or `0`), Mooncake uses the DMA-BUF path which does not require `nvidia-peermem`.
- `MC_ENDPOINT_STORE_TYPE` Choose FIFO Endpoint Store (`FIFO`) or Sieve Endpoint Store (`SIEVE`), default is `SIEVE`.
- `MC_TCP_ENABLE_CONNECTION_POOL` Enable TCP Connection Pool to avoid excessive sockets.
- `MC_TCP_ENABLE_CONNECTION_POOL` Enable TCP Connection Pool to avoid excessive sockets.
- `MC_TCP_SLICE_SIZE` The segmentation granularity (in bytes) of TCP transport for splitting large transfers into socket read/write operations. Corresponds to `MC_SLICE_SIZE` for RDMA. Default value 65536 (64KB).
## C++ API Reference
::::{toctree}
For the complete C++ API reference, see [Transfer Engine C++ API](../../api-reference/cpp/index).
## Supported Protocols
:::{toctree}
:maxdepth: 1
cpp-api
::::
../../getting_started/supported-protocols
:::
## EFA Transport (AWS)
@ -330,6 +347,15 @@ ascend_transport
heterogeneous_ascend
:::
## Sunrise Link Transport Component
:::{toctree}
:maxdepth: 1
kunpeng_ub_transport
sunrise_link_transport
:::
## Benchmark and Tuning Guide
:::{toctree}

2
docs/source/design/transfer-engine/kunpeng_ub_transport.md
View File

@ -50,7 +50,7 @@ sudo apt-get install -y \
libjsoncpp-dev \
libnuma-dev \
libibverbs-dev \
libboost-all-dev \
libboost-dev \
libcurl4-openssl-dev \
libgtest-dev \
libmsgpack-dev \

129
docs/source/design/transfer-engine/sunrise_link_transport.md Normal file
View File

@ -0,0 +1,129 @@
# Sunrise Link Transport
## Overview
Sunrise Link is a GPU-to-GPU direct communication interconnect developed by Sunrise. In Mooncake, it is integrated as a GPU transport backend in the TENT transfer framework. It relies on Tang Runtime (`tangrt`) for device memory allocation, pointer attribute queries, peer copy, and IPC handle operations, and is registered as `SUNRISE_LINK` in TENT.
At runtime, `TransferEngineImpl` loads Sunrise Link Transport when `USE_SUNRISE` is enabled at build time and `transports/sunrise_link/enable=true` is set in config.
### Transfer Pipeline
1. **Init**: `dlopen` `libptml_shared.so` + `libtangrt_shared.so`, initialize PTML.
2. **Topology**: `ptmlPtlinkPhytopoDetect` discovers inter-chip ports, builds C2C mapping (`local_chipid, remote_chipid → local_port`).
3. **Registration**: `tangIpcGetMemHandle` generates IPC handles for GPU buffers.
4. **Execution**: Same device → `tangMemcpy`; Cross-device → `tangMemcpyPeer` / `tangMemcpyPeer_v2`, fallback to `tangDeviceGetPeerPointer` + C2C; Remote → `tangIpcOpenMemHandle` + peer copy; Host ↔ Device → `tangMemcpy` with direction.
5. **Async**: When size exceeds `async_memcpy_threshold`, uses `tangMemcpyAsync` with per-device stream.
---
## Additional Dependencies
In addition to Mooncake's base dependencies, Sunrise Link Transport requires Tang Runtime:
- **Header path**: `/usr/local/tangrt/include`
- **Library path**: `/usr/local/tangrt/lib/linux-x86_64`
- **Shared library**: `libtangrt_shared.so` (loaded from the default installation path at runtime)
Make sure runtime libraries related to `tangrt` and `ptml` can be found by the dynamic linker (for example, via `LD_LIBRARY_PATH` or system library search paths).
---
## Build and Compile
**Prerequisites**
- Tang Runtime is installed and available (default path: `/usr/local/tangrt`)
- Build environment can access Mooncake and its base dependencies
**CMake Configuration**
```bash
# Clone Mooncake
git clone https://github.com/kvcache-ai/Mooncake.git
cd Mooncake
# Enable TENT + Sunrise Link
mkdir build && cd build
cmake .. -DUSE_TENT=ON -DUSE_SUNRISE=ON
# Build
make -j$(nproc)
```
If Tang Runtime is installed at a non-default location:
```bash
cmake .. -DUSE_TENT=ON -DUSE_SUNRISE=ON -DMC_TANGRT_ROOT=/opt/tangrt
```
---
## Run and Test
`transfer_engine_bench` supports the `sunrise_link` protocol for basic connectivity and performance validation.
```bash
# Terminal 1: target
./transfer_engine_bench \
--mode=target \
--protocol=sunrise_link \
--local_server_name=10.0.0.2 \
--metadata_server=P2PHANDSHAKE \
--gpu_id=0
# Terminal 2: initiator
./transfer_engine_bench \
--mode=initiator \
--protocol=sunrise_link \
--metadata_server=P2PHANDSHAKE \
--segment_id=10.0.0.2:$PORT \
--gpu_id=0 \
--block_size=8388608 \
--batch_size=32
```
> Note: when `metadata_server=P2PHANDSHAKE`, the target node may listen on a dynamically assigned port. Replace `$PORT` in `--segment_id` with the actual port printed in target logs.
### Using `tebench` (TENT Backend)
Role is determined by `--target_seg_name`: empty → target, otherwise → initiator.
```bash
# Terminal 1: target
./tebench --backend=tent --metadata_type=p2p --xport_type=sunrise_link \
--seg_type=VRAM --local_gpu_id=0
# Terminal 2: initiator
./tebench --backend=tent --metadata_type=p2p --xport_type=sunrise_link \
--seg_type=VRAM --target_seg_name=192.168.172.52:168 \
--local_gpu_id=1 --target_gpu_id=0
```
---
## Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `MC_TANGRT_ROOT` | Tang Runtime root directory | `/usr/local/tangrt` |
| `MC_TANGRT_LIB_DIR` | Direct path to `.so` dir. Overrides `MC_TANGRT_ROOT` + arch lookup. | (unset) |
| `MC_TANGRT_LIB_ARCH` | Architecture subdirectory under `<root>/lib/` | `linux-x86_64` |
---
## Configuration Options (TENT)
Sunrise Link behavior can be tuned through config file options:
- `transports/sunrise_link/enable`: enable or disable the transport (default: `true`)
- `transports/sunrise_link/async_memcpy_threshold`: threshold for async memcpy in MiB, 0 to disable (default: `0`)
- `transports/sunrise_link/enable_port_striping`: enable multi-port striping for large cross-device copies >4 MiB (default: `false`)
Tune these options based on workload characteristics and hardware topology.
---
## Important Notes
1. **Device Context**: Internally manages `tangSetDevice`. When calling Tang API directly, set the correct device before querying pointer attributes.
2. **IPC Handle Lifecycle**: IPC handles are cached and released in `uninstall()`.
3. **Concurrent Peer Copy**: Per-device mutexes protect context-bound ops. Cross-device copies acquire source/destination mutexes in fixed order (lower device ID first) to prevent deadlocks.
4. **Mapped Host Pointer Normalization**: `tangHostAlloc` host pointers with corresponding device pointers are auto-normalized for correct peer copy path selection.

405
docs/source/design/unified-parallel-tensor-io.md Normal file
View File

@ -0,0 +1,405 @@
# Unified Parallel Tensor IO
## Goal
This document defines the single source of truth for Mooncake's next-generation tensor IO API. The purpose is to keep implementation aligned around one explicit design and avoid drifting into ad hoc TP/EP/DP/PP-specific interfaces.
The target outcome is:
- one unified write API family
- one unified read API family
- one unified upsert API family
- TP-specific APIs retained only as compatibility wrappers
- minimal caller-facing inputs, with layout and planning details derived internally
- one stable abstraction that can cover TP / DP / EP / PP combinations without exploding the public API surface
## Design principles
1. **One API family, not one family per parallelism type.**
Public APIs should not branch into separate long-term `*_with_tp`, `*_with_ep`, `*_with_pp`, and `*_with_dp` families.
2. **Parallelism is modeled as explicit axes.**
`mixed` must not be a vague enum value. A shard should be described by the exact axis coordinates that identify it.
3. **Reads must encode caller intent explicitly.**
A read request must say whether the caller wants the stored view, a target shard/view, or the reconstructed full tensor.
4. **Only require true caller intent.**
If Mooncake can derive a field from the input tensor, stored metadata, or planning context, it should not be required in the public API.
5. **Planner/runtime complexity stays internal.**
Byte ranges, payload offsets, reconstruction plans, and lowering to `get_into_ranges(...)` are runtime concerns, not public API concerns.
6. **Compatibility wrappers stay thin.**
Existing TP methods should lower to the same unified implementation path.
## Core model
### Parallelism is an axis list
A tensor object may be identified by one axis or by multiple orthogonal axes.
Examples:
- pure TP shard: `[TP]`
- DP + TP shard: `[DP, TP]`
- PP + TP shard: `[PP, TP]`
- DP + PP + TP shard: `[DP, PP, TP]`
- DP + PP + EP + TP shard: `[DP, PP, EP, TP]`
So the design should not use:
```python
kind = "mixed"
```
Instead it should use:
```python
class ParallelAxis:
kind: Literal["tp", "dp", "ep", "pp"]
rank: int
size: int
split_dim: Optional[int] = None
expert_id: Optional[int] = None
stage_id: Optional[int] = None
```
```python
class TensorParallelism:
axes: list[ParallelAxis]
```
Single-axis cases are just special cases of this model:
- TP only: `axes=[TP(...)]`
- EP only: `axes=[EP(...)]`
- PP + TP: `axes=[PP(...), TP(...)]`
## Public API shape
### Write / upsert side
```python
def put_tensor_with_parallelism(
self,
key: str,
tensor: torch.Tensor,
parallelism: TensorParallelism | None = None,
replica: ReplicateConfig | None = None,
) -> int
```
```python
def batch_put_tensor_with_parallelism(
self,
keys: list[str],
tensors: list[torch.Tensor],
parallelisms: list[TensorParallelism | None] | None = None,
replica: ReplicateConfig | None = None,
) -> list[int]
```
```python
def upsert_tensor_with_parallelism(
self,
key: str,
tensor: torch.Tensor,
parallelism: TensorParallelism | None = None,
replica: ReplicateConfig | None = None,
) -> int
```
```python
def batch_upsert_tensor_with_parallelism(
self,
keys: list[str],
tensors: list[torch.Tensor],
parallelisms: list[TensorParallelism | None] | None = None,
replica: ReplicateConfig | None = None,
) -> list[int]
```
### Read side
Read-side requests need one extra structure because the request must encode not only the target parallel coordinates, but also the materialization mode.
```python
class ReadTarget:
mode: Literal["as_stored", "shard", "full"]
parallelism: TensorParallelism | None = None
```
```python
def get_tensor_with_parallelism(
self,
key: str,
target: ReadTarget | None = None,
) -> torch.Tensor
```
```python
def batch_get_tensor_with_parallelism(
self,
keys: list[str],
targets: list[ReadTarget | None] | None = None,
) -> list[torch.Tensor]
```
Optional zero-copy forms follow the same model:
```python
def get_tensor_with_parallelism_into(
self,
key: str,
buffer_ptr: int,
size: int,
target: ReadTarget | None = None,
) -> torch.Tensor
```
```python
def batch_get_tensor_with_parallelism_into(
self,
keys: list[str],
buffer_ptrs: list[int],
sizes: list[int],
targets: list[ReadTarget | None] | None = None,
) -> list[torch.Tensor]
```
## Caller inputs vs internal derived fields
### Caller must provide
These are true caller intent and belong in the public API:
- the tensor itself
- whether the write target is full or shard-like
- the shard identity axes when writing a shard object
- the read target mode: `as_stored`, `shard`, or `full`
- the target axis coordinates when requesting a target shard/view
- `ReplicateConfig` when the caller wants replication / publish-like behavior
### Mooncake should derive internally
These should not be mandatory public inputs when they are derivable:
- logical shape
- local shard shape
- metadata encoding/version details
- payload offsets
- source and destination byte ranges
- reconstruction plans
- lowering to `get_into_ranges(...)`
## Why `ReadTarget` is required
`parallelism` alone is not enough to describe a read.
For example, if a tensor is stored as TP shards, a request that carries TP axis coordinates is ambiguous unless it also says whether the caller wants:
- the stored local shard
- a target shard/view
- the reconstructed full tensor
So reads must explicitly encode:
```python
ReadTarget(mode="as_stored")
ReadTarget(mode="shard", parallelism=...)
ReadTarget(mode="full")
```
This distinction must not be guessed from the axis metadata.
## Mixed-parallel scenarios in training and inference
`mixed` is not a mode. It means the shard identity needs more than one axis coordinate to be uniquely described.
### Scenario table
| Scenario | Typical axis list | Meaning |
|---|---|---|
| TP training | `[TP]` | one TP slice of a logical tensor |
| DP + TP training | `[DP, TP]` | one TP slice within one DP replica/group |
| PP + TP training | `[PP, TP]` | one TP slice owned by one pipeline stage |
| DP + PP + TP training | `[DP, PP, TP]` | one TP slice in one stage in one DP replica |
| DP + PP + EP + TP training | `[DP, PP, EP, TP]` | one expert-local TP slice in one stage and one DP replica |
| TP inference | `[TP]` | one TP slice used by one inference rank |
| PP + TP inference | `[PP, TP]` | one TP slice owned by one inference pipeline stage |
| EP inference | `[EP]` or `[EP, TP]` | one expert shard, optionally further TP-sliced |
| multi-replica serving inference | `[DP, TP]` or `[DP, PP, TP]` | one shard scoped to a serving replica plus model-parallel axes |
### Important distinctions
- TP and EP often affect the tensor's physical layout directly.
- PP and serving-replica / DP often act more like ownership or scope tags, even when they do not themselves change the byte layout inside the local shard.
## Write-side matrix
The unified write family should be driven by the identity of the object being written, not by method-name proliferation.
| Caller holds | Wants to store | API | Required `parallelism` |
|---|---|---|---|
| full tensor | full tensor | `put_tensor_with_parallelism(..., parallelism=None)` | none |
| full tensor | TP shard | `put_tensor_with_parallelism(...)` | `axes=[TP(rank,size,split_dim)]` |
| full tensor | DP-scoped shard/replica | `put_tensor_with_parallelism(...)` | `axes=[DP(rank,size)]` plus layout axis if actually sharded |
| full tensor | EP shard | `put_tensor_with_parallelism(...)` | `axes=[EP(rank,size,expert_id)]` plus `split_dim` if needed |
| full tensor | PP stage shard | `put_tensor_with_parallelism(...)` | `axes=[PP(rank,size,stage_id)]` |
| full tensor | combined shard | `put_tensor_with_parallelism(...)` | explicit axis list such as `[PP(...), TP(...)]` |
| shard tensor | shard object | `put_tensor_with_parallelism(...)` | explicit axis list describing that shard identity |
For TP-containing **multi-axis** layouts, the write semantic is now: the caller may pass the **full source tensor**, and the provided TP rank/layout tells Mooncake which uniform shard to materialize and persist. Callers no longer need to pre-split the tensor themselves for `dp_tp` / `pp_tp` / `ep_tp` style writes.
Single-axis TP compatibility wrappers and the preserved plain-TP `with_parallelism` behavior still accept shard input rather than auto-materializing from a full tensor.
Pure DP still does not invent a split axis on its own. If the stored object is actually sharded, the request must still include the layout axis that defines the shard shape.
The same matrix applies to `upsert_tensor_with_parallelism(...)`.
## Read-side matrix
The unified read family should be driven by `ReadTarget`.
| Stored layout | Caller wants | API | `ReadTarget` |
|---|---|---|---|
| full tensor | stored full tensor | `get_tensor_with_parallelism(...)` | `None` or `mode="as_stored"` |
| TP shard object | stored shard | `get_tensor_with_parallelism(...)` | `mode="as_stored"` |
| TP shard set | target TP shard | `get_tensor_with_parallelism(...)` | `mode="shard", parallelism=TP(...)` |
| TP shard set | full tensor | `get_tensor_with_parallelism(...)` | `mode="full"` |
| EP shard set | target expert shard | `get_tensor_with_parallelism(...)` | `mode="shard", parallelism=EP(...)` |
| EP shard set | full tensor | `get_tensor_with_parallelism(...)` | `mode="full"` |
| PP + TP shard set | target PP+TP shard | `get_tensor_with_parallelism(...)` | `mode="shard", parallelism=[PP(...), TP(...)]` |
| mixed shard set | full tensor | `get_tensor_with_parallelism(...)` | `mode="full"` |
## Source-layout to target-layout matrix
The planner must eventually cover these conversions, but the public API should remain the same across all of them.
| Stored source layout | Requested target layout | Support model |
|---|---|---|
| TP | TP | direct shard fetch or shard-local fast path |
| TP | full | reconstruct full tensor |
| TP | EP / PP / DP / mixed | planner-driven remap |
| EP | EP | direct shard fetch or shard-local fast path |
| EP | full | reconstruct full tensor |
| EP | TP / PP / DP / mixed | planner-driven remap |
| PP | PP | direct shard fetch |
| PP | full | reconstruct full tensor when meaningful |
| PP | TP / EP / DP / mixed | planner-driven remap |
| mixed | mixed | direct fetch if exact match, else planner-driven remap |
| mixed | full | reconstruct full tensor |
| mixed | TP / EP / PP / DP | planner-driven remap |
The important point is that these combinations must not create public API explosion.
## Compatibility wrappers
Existing TP APIs remain compatibility wrappers.
Conceptually:
```python
put_tensor_with_tp(key, tensor, tp_rank, tp_size, split_dim)
```
lowers to:
```python
put_tensor_with_parallelism(
key,
tensor,
TensorParallelism(axes=[TP(rank=tp_rank, size=tp_size, split_dim=split_dim)]),
)
```
and:
```python
get_tensor_with_tp(key, tp_rank, tp_size, split_dim)
```
lowers to:
```python
get_tensor_with_parallelism(
key,
ReadTarget(
mode="shard",
parallelism=TensorParallelism(
axes=[TP(rank=tp_rank, size=tp_size, split_dim=split_dim)]
),
),
)
```
Existing TP behavior stays stable, but implementation should flow through the unified path.
## Runtime lowering direction
Internally, reads should be planner-driven.
When a request can be lowered to explicit:
- key
- src offset
- dst offset
- size
ranges, the runtime should reuse `get_into_ranges(...)` to assemble the result directly into the output buffer.
Otherwise it can fall back to the simpler whole-object path.
This keeps the public interface stable while allowing future planner work to add DP / TP / EP / PP remapping and optimized reconstruction without another API redesign.
## Current implementation status
The current `store_py.cpp` implementation now reflects the main shape of this design:
- unified write APIs are exposed as `put_tensor_with_parallelism(...)` and `batch_put_tensor_with_parallelism(...)`
- unified read APIs are exposed as `get_tensor_with_parallelism(...)` and `batch_get_tensor_with_parallelism(...)`
- unified upsert APIs are exposed as `upsert_tensor_with_parallelism(...)` and `batch_upsert_tensor_with_parallelism(...)`
- zero-copy `_into` and `_from` variants exist for the unified API family
- TP-specific APIs remain available as compatibility wrappers and should not be treated as the long-term surface area
### Implemented write-side convenience: `writer_partitions`
Batch write and batch upsert paths also support `writer_partitions` as a convenience input for full tensors that should be written as stored shards.
This is intentionally narrower than the full `TensorParallelism` model:
- it is a write-side convenience, not a replacement for `TensorParallelism`
- it is primarily for batch full-tensor writes where the caller already knows rank / size / split_dim per item
- it should not change the unified read-side abstraction
`writer_partitions` remains a separate explicit route. The newer TP-containing `parallelism` write semantic now overlaps with it for the common case of “full tensor in, store one requested shard”, but `writer_partitions` is still useful when the caller wants a lighter write-side request shape without constructing `TensorParallelism` objects.
### Implemented read-side behavior
Read-side support includes:
- returning the stored local object (`mode="as_stored"`)
- returning a target shard (`mode="shard"`)
- reconstructing the full tensor (`mode="full"`)
- lowering reconstruction-oriented paths onto existing runtime helpers such as `get_into_ranges(...)` where appropriate
### Compatibility boundary
The project should continue to preserve this boundary:
- keep old TP APIs functional
- keep them thin
- do not expand the old TP-specific family as the primary interface
- document and evolve the unified `*_with_parallelism` family instead
## Scope still intentionally limited
This document describes the stable public API direction, but not every theoretical source-layout to target-layout remap is fully implemented.
In particular, the design should continue to avoid over-promising planner coverage for arbitrary remaps across all DP / TP / EP / PP combinations until those paths are explicitly implemented and tested.
The safe documented contract today is:
1. define the public API structures and signatures clearly
2. align `store_py.cpp` with `axes + ReadTarget`
3. keep TP wrappers working by lowering into the unified path
4. do not introduce a new long-term `kind="mixed"` model
5. do not push planner internals such as byte ranges and derived shapes into public arguments
6. do not expand into many parallelism-specific public methods
7. reuse existing runtime helpers like `get_into_ranges(...)` rather than inventing a parallel reconstruction path from scratch

107
docs/source/getting_started/build.md
View File

@ -17,10 +17,10 @@ pip install mooncake-transfer-engine-non-cuda
```
📦 **Package Details**: [https://pypi.org/project/mooncake-transfer-engine-non-cuda/](https://pypi.org/project/mooncake-transfer-engine-non-cuda/)
> **Note**: The CUDA version includes Mooncake-EP and GPU topology detection, requiring CUDA 12.1+. The non-CUDA version is for environments without CUDA dependencies.
> **Note**: The CUDA version includes Mooncake-EP and GPU topology detection, requiring CUDA 12.1+. The non-CUDA version is for environments without CUDA dependencies, but it still requires the system runtime libraries used by the transfer stack. On Ubuntu, install them with `sudo apt-get update && sudo apt-get install -y libcurl4 libibverbs1 rdma-core librdmacm1 libnuma1 liburing2`.
> **Note**: MLU support is currently source-build only. If you need Cambricon MLU memory support, install Neuware and build with `-DUSE_MLU=ON`.
## Automatic
## Automatic Build
### Recommended Version
- OS: Ubuntu 22.04 LTS+
@ -30,7 +30,7 @@ pip install mooncake-transfer-engine-non-cuda
### Steps
1. Install dependencies, stable Internet connection is required:
```bash
bash dependencies.sh
sudo bash dependencies.sh
```
2. In the root directory of this project, run the following commands:
@ -45,7 +45,25 @@ pip install mooncake-transfer-engine-non-cuda
sudo make install
```
## Manual
**Build with NVMe-oF SSD Pool**
To enable the NVMe-oF SSD pool, install the SPDK dependencies and build
Mooncake with `USE_NOF` enabled:
```bash
sudo bash dependencies.sh --with-spdk
mkdir build
cd build
cmake .. -DUSE_NOF=ON
make -j
sudo make install
```
`-DUSE_NOF=ON` builds the NoF registration APIs and deployment tools. Use
`-DUSE_NOF=OFF` or omit the option when the NVMe-oF SSD pool is not needed.
## Manual Build
### Recommended Version
- cmake: 3.22.x
@ -70,7 +88,7 @@ pip install mooncake-transfer-engine-non-cuda
libnuma-dev \
libunwind-dev \
libpython3-dev \
libboost-all-dev \
libboost-dev \
libssl-dev \
pybind11-dev \
libcurl4-openssl-dev \
@ -100,12 +118,15 @@ pip install mooncake-transfer-engine-non-cuda
```
2. If you want to compile the GPUDirect support module, first follow the instructions in https://docs.nvidia.com/cuda/cuda-installation-guide-linux/ to install CUDA (ensure to enable `nvidia-fs` for proper `cuFile` module compilation). After that:
1) Follow Section 3.7 in https://docs.nvidia.com/cuda/gpudirect-rdma/ to install `nvidia-peermem` for enabling GPU-Direct RDMA
2) Configure `LIBRARY_PATH` and `LD_LIBRARY_PATH` to ensure linking of `cuFile`, `cudart`, and other libraries during compilation:
1) Configure `LIBRARY_PATH` and `LD_LIBRARY_PATH` to ensure linking of `cuFile`, `cudart`, and other libraries during compilation:
```bash
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/cuda/lib64
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
```
```{admonition} GPU-Direct RDMA
:class: note
Mooncake could use the DMA-BUF path for GPU-Direct RDMA, which does **not** require the `nvidia-peermem` kernel module. If you prefer the DMA-BUF path, please set the runtime environment variable `WITH_NVIDIA_PEERMEM=0` before starting Mooncake. If you prefer the legacy `ibv_reg_mr` path (which requires `nvidia-peermem`), set the runtime environment variable `WITH_NVIDIA_PEERMEM=1`. See Section 3.7 of https://docs.nvidia.com/cuda/gpudirect-rdma/ for instructions on installing `nvidia-peermem`.
```
3. If you want to compile the Moore Mthreads GPUDirect support module, first follow the instructions in https://docs.mthreads.com/musa-sdk/musa-sdk-doc-online/install_guide to install MUSA. After that:
1) Install `mthreads-peermem` for enabling GPU-Direct RDMA
@ -149,7 +170,20 @@ pip install mooncake-transfer-engine-non-cuda
- `-DMACA_LIB_DIR=/path/to/maca/lib64`
- `-DMACA_RUNTIME_LIBS="mcruntime;mxc-runtime64;rt"` (semicolon-separated CMake list)
6. Install yalantinglibs
6. If you want to compile Huawei Ascend NPU support, first install the Ascend CANN Toolkit following the instructions at https://www.hiascend.com/document. After that:
1) Source `set_env.sh` in the CANN installation directory to configure the build environment (no need to manually set `ASCEND_HOME_PATH` or other related environment variables).
2) Mooncake provides two Ascend NPU transport paths, choose one as needed:
- `-DUSE_ASCEND_DIRECT=ON` (**recommended**): Ascend Direct transport based on the ADXL engine. (refer to [Version Compatibility Guide](https://gitcode.com/cann/hixl/wiki/Mooncake%20+%20HIXL%20%E5%BF%AB%E9%80%9F%E4%B8%8A%E6%89%8B%E6%8C%87%E5%8D%97.md) for details).
- `-DUSE_UBSHMEM=ON`: Shared memory transport based on CANN VMM APIs (requires CANN >= 9.0.0, driver >= 26.0.0, Lingqu >= 1.5).
Example for building with Ascend NPU:
```bash
source /usr/local/Ascend/cann/set_env.sh
cmake .. -DUSE_ASCEND_DIRECT=ON
make -j
```
7. Install yalantinglibs
```bash
git clone https://github.com/alibaba/yalantinglibs.git
cd yalantinglibs
@ -159,7 +193,7 @@ pip install mooncake-transfer-engine-non-cuda
make install
```
7. In the root directory of this project, run the following commands:
8. In the root directory of this project, run the following commands:
```bash
mkdir build
cd build
@ -167,23 +201,56 @@ pip install mooncake-transfer-engine-non-cuda
make -j
```
8. Install Mooncake python package and mooncake_master executable
9. Install Mooncake python package and mooncake_master executable
```bash
make install
```
## Use Mooncake in Docker Containers
Mooncake supports Docker-based deployment. What you need is to get Docker image by `docker pull alogfans/mooncake`.
Mooncake supports Docker-based deployment. You can either build the image from
this repository with `docker/mooncake.Dockerfile` or substitute a published
tag that matches the release you want to run.
For the container to use the host's network resources, you need to add the `--device` option when starting the container. The following is an example.
```
# In host
sudo docker run --net=host --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm --ulimit memlock=-1 -t -i mooncake:v0.9.0 /bin/bash
sudo docker build -f docker/mooncake.Dockerfile -t mooncake:from-source .
sudo docker run --net=host --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm --ulimit memlock=-1 -t -i mooncake:from-source /bin/bash
# Run transfer engine in container
cd /Mooncake-main/build/mooncake-transfer-engine/example
./transfer_engine_bench --device_name=ibp6s0 --metadata_server=10.1.101.3:2379 --mode=target --local_server_name=10.1.100.3
```
For SGLang HiCache deployments inside Docker, reserve HugeTLB pages on the host before starting the container and pass the allocator settings through the container environment:
```bash
python3 scripts/check_hicache_hugepage_requirements.py \
--tp-size 4 \
--hicache-size 64gb \
--global-segment-size 8gb \
--arena-pool-size 56gb \
--available-hugetlb 512gb
sudo sysctl -w vm.nr_hugepages=262144
grep -E 'HugePages_Total|HugePages_Free|Hugepagesize' /proc/meminfo
sudo docker run --gpus all \
--net=host \
--ipc=host \
--ulimit memlock=-1 \
--shm-size=128g \
--device=/dev/infiniband/uverbs0 \
--device=/dev/infiniband/rdma_cm \
-e MC_STORE_USE_HUGEPAGE=1 \
-e MC_STORE_HUGEPAGE_SIZE=2MB \
-e MOONCAKE_GLOBAL_SEGMENT_SIZE=8gb \
-e MC_MMAP_ARENA_POOL_SIZE=56gb \
-t -i mooncake:from-source /bin/bash
```
The `64gb` / `56gb` values above are tuned examples for large HiCache deployments, not defaults. The arena remains disabled unless you explicitly enable it, and if you enable it via gflag without an env override the default pool size is `8gb`. On smaller hosts, start with `8gb` or `16gb` and size upward with the helper. When you want the baseline direct-`mmap()` path instead of the arena, set `MC_DISABLE_MMAP_ARENA=1` (also accepts `true`, `yes`, or `on`) and omit `MC_MMAP_ARENA_POOL_SIZE`. Set it before the first Mooncake mmap-buffer allocation in the process. If you build the image from source with `docker/mooncake.Dockerfile`, that source-built image also installs the helper as `mooncake-hicache-sizing`.
Without `MC_STORE_USE_HUGEPAGE=1`, the arena may opportunistically try hugepages and then retry on regular pages if HugeTLB is unavailable. When `MC_STORE_USE_HUGEPAGE=1` is set, both the arena path and the direct-`mmap()` fallback path require HugeTLB pages. Mooncake will not silently degrade that explicit hugepage request to regular pages.
## Advanced Compile Options
The following options can be used during `cmake ..` to specify whether to compile certain components of Mooncake.
- `-DUSE_CUDA=[ON|OFF]`: Enable GPU memory support (GPUDirect RDMA, NVMe-oF, and GPU-aware TCP transport). **Default: OFF.** Required when transferring GPU memory (e.g., KV cache in vLLM disaggregated serving), even when using TCP protocol.
@ -195,6 +262,14 @@ The following options can be used during `cmake ..` to specify whether to compil
- `-DMACA_LIB_DIR=/path/to/lib64`: Override MACA library directory when `-DUSE_MACA=ON`.
- `-DMACA_RUNTIME_LIBS="mcruntime;mxc-runtime64;rt"`: Override MACA runtime libraries linked by `transfer_engine`.
- `-DUSE_HIP=[ON|OFF]`: Enable AMD GPU support via HIP/ROCm
- `-DUSE_HYGON=[ON|OFF]`: Enable Hygon DCU support via DTK SDK. **Default: OFF.** Uses CUDA-compatible runtime.
- `-DDTK_ROOT=/path/to/dtk`: Override the default DTK SDK root used when `-DUSE_HYGON=ON`. If unset, Mooncake uses `DTK_HOME` or `/opt/dtk`.
- `-DDTK_INCLUDE_DIR=/path/to/include`: Override the DTK include directory when `-DUSE_HYGON=ON`.
- `-DDTK_LIB_DIR=/path/to/lib64`: Override the DTK library directory when `-DUSE_HYGON=ON`.
- `-DUSE_COREX=[ON|OFF]`: Enable Iluvatar CoreX GPU support. **Default: OFF.** Uses CUDA-compatible runtime.
- `-DCOREX_ROOT=/path/to/corex`: Override the default CoreX SDK root used when `-DUSE_COREX=ON`. If unset, Mooncake uses `COREX_HOME` or `/usr/local/corex`.
- `-DCOREX_INCLUDE_DIR=/path/to/include`: Override the CoreX include directory when `-DUSE_COREX=ON`.
- `-DCOREX_LIB_DIR=/path/to/lib`: Override the CoreX library directory when `-DUSE_COREX=ON`.
- `-DUSE_MLU=[ON|OFF]`: Enable Cambricon MLU memory support via Neuware. **Default: OFF.** Supports MLU memory detection, topology discovery, and RDMA registration for Transfer Engine.
- `-DNEUWARE_ROOT=/path/to/neuware`: Override the default Neuware SDK root used when `-DUSE_MLU=ON`. If unset, Mooncake uses `NEUWARE_HOME` or `/usr/local/neuware`.
- `-DMLU_INCLUDE_DIR=/path/to/include`: Override the Neuware include directory when `-DUSE_MLU=ON`.
@ -204,12 +279,16 @@ The following options can be used during `cmake ..` to specify whether to compil
- `-DUSE_CXL=[ON|OFF]`: Enable CXL support
- `-DWITH_STORE=[ON|OFF]`: Build Mooncake Store component
- `-DWITH_P2P_STORE=[ON|OFF]`: Enable Golang support and build P2P Store component, require go 1.23+
- `-DWITH_WITH_RUST_EXAMPLE=[ON|OFF]`: Enable Rust support
- `-DWITH_RUST_EXAMPLE=[ON|OFF]`: Build the Transfer Engine Rust interface and sample code. **Default: OFF.**
- `-DWITH_STORE_RUST=[ON|OFF]`: Build Mooncake Store Rust bindings and CMake Rust targets. **Default: ON.**
- `-DWITH_EP=[ON|OFF]`: Build the EP (Expert Parallelism) and PG Python extensions for CUDA. Requires CUDA toolkit and PyTorch. Use `-DEP_TORCH_VERSIONS="2.9.1"` (semicolon-separated) to build for specific PyTorch versions, or leave empty to use the currently-installed torch. The CUDA version is detected automatically. **Default: OFF.**
- `-DUSE_REDIS=[ON|OFF]`: Enable Redis-based metadata service
- `-DUSE_REDIS=[ON|OFF]`: Enable Redis-based metadata service for the Transfer Engine, require hiredis
- `-DUSE_HTTP=[ON|OFF]`: Enable Http-based metadata service
- `-DUSE_ETCD=[ON|OFF]`: Enable etcd-based metadata service, require go 1.23+
- `-DSTORE_USE_ETCD=[ON|OFF]`: Enable etcd-based failover for Mooncake Store, require go 1.23+. **Note:** `-DUSE_ETCD` and `-DSTORE_USE_ETCD` are two independent options. Enabling `-DSTORE_USE_ETCD` does **not** depend on `-DUSE_ETCD`
- `-DSTORE_USE_REDIS=[ON|OFF]`: Enable Redis-based failover for Mooncake Store, require hiredis. **Default: OFF.** **Note:** `-DUSE_REDIS` and `-DSTORE_USE_REDIS` are two independent options. Enabling `-DSTORE_USE_REDIS` does **not** depend on `-DUSE_REDIS`.
- `-DBUILD_SHARED_LIBS=[ON|OFF]`: Build Transfer Engine as shared library, default is OFF
- `-DBUILD_UNIT_TESTS=[ON|OFF]`: Build unit tests, default is ON
- `-DBUILD_EXAMPLES=[ON|OFF]`: Build examples, default is ON
- `-DUSE_ASCEND_DIRECT=[ON|OFF]`: Enable Ascend Direct transport and HCCS support via the ADXL engine (**recommended**).
- `-DUSE_UBSHMEM=[ON|OFF]`: Enable Huawei Ascend NPU shared memory transport via CANN VMM APIs.

11
docs/source/getting_started/examples/lmcache-integration.md
View File

@ -1,4 +1,6 @@
# Mooncake x LMCache: Unite to Pioneer KVCache-Centric LLM Serving System
# Mooncake x LMCache Integration
Mooncake and LMCache unite to pioneer KVCache-centric LLM serving systems.
Mooncake and LMCache have announced a strategic collaboration aimed at pioneering a KVCache-centric Large Language Model (LLM) serving system. This partnership seeks to significantly enhance the efficiency, scalability, and responsiveness of LLM applications.
@ -23,7 +25,12 @@ This strategic alliance delivers substantial mutual technological advancements:
For a complete deployment guide with step-by-step instructions, see:
👉 **[vLLM V1 Disaggregated Serving with Mooncake Store and LMCache](vllm-integration/vllmv1-lmcache-integration.md)**
:::{toctree}
:maxdepth: 1
../examples/vllm-integration/vllmv1-lmcache-integration
../examples/vllm-integration/vllmv1-lmcache-mp-integration
:::
## Performance Benchmarking and Results

10
docs/source/getting_started/examples/lmdeploy-integration-v0.9.md
View File

@ -16,17 +16,17 @@ pip install mooncake-transfer-engine
Note:
- If any `.so` file is missing, uninstall the pip package with `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually from source following the [build instructions](https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/build.md).
- If any `.so` file is missing, uninstall the pip package with `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually from source following the [build instructions](../build.md).
### Install the latest version of LMDeploy
##### 1. Clone LMDeploy from the official repo
#### 1. Clone LMDeploy from the official repo
```bash
git clone https://github.com/InternLM/lmdeploy.git
```
##### 2. Build
#### 2. Build
##### 2.1 Build from source
@ -74,7 +74,7 @@ lmdeploy serve api_server Qwen/Qwen3-8B \
- The `--role` parameter sets the node role in the disaggregated system (`Prefill` for token embedding and KV cache generation).
- The `--proxy-url` parameter connects the worker instance back to the proxy for coordination.
- The `--backend` parameter specifies the model execution backend (e.g., `pytorch`, `turbomind`).
- The `--migration-backend` parameter defines the KV cache transport mechanism (e.g., `Mooncake` and `DlSllime`).
- The `--migration-backend` parameter defines the KV cache transport mechanism (e.g., `Mooncake` and `DLSlime`).
#### Decode:
@ -164,4 +164,4 @@ curl -X POST "http://192.168.0.147:8000/v1/completions" \
- You can specify multiple prefill or decode instances with distinct `--server-port` and different GPUs using `CUDA_VISIBLE_DEVICES`.
- MooncakeTransferEngine supports both intra-node (PCIe) and inter-node (RDMA/CXL) transfer, and device selection is automatic or customizable via config.
- When using HF models that timeout during prefill, consider setting model path to `~/Qwen3-8B` to accelerate loading from localhost.
- Use `--log-level DEBUG` to get detailed runtime logs for troubleshooting.
- Use `--log-level DEBUG` to get detailed runtime logs for troubleshooting.

293
docs/source/getting_started/examples/sglang-integration-v1.md
View File

@ -1,175 +1,206 @@
# SGLang Disaggregated Serving with MooncakeTransferEngine
## Overview
This is the latest version of the MooncakeTransferEngine integration doc with the SGLang project based on [PR 4654](https://github.com/sgl-project/sglang/pull/4654) and [PR 4880](https://github.com/sgl-project/sglang/pull/4880) to support KVCache transfer for intra-node and inter-node disaggregated serving scenarios.
SGLang uses Mooncake's Transfer Engine to enable disaggregated prefill-decode (PD) serving across nodes via RDMA, with support for EP and EPD backends. This integration is based on [PR 4654](https://github.com/sgl-project/sglang/pull/4654) and [PR 4880](https://github.com/sgl-project/sglang/pull/4880).
**_Please note that this is still an experimental version and will be modified anytime based on feedback from the SGLang community._**
In benchmarks, PD disaggregation with Mooncake achieves **~30% lower ITL** while maintaining comparable throughput ([details](../../performance/sglang-benchmark-results-v1)).
```
+-----------+ Transfer Engine (RDMA) +-----------+
| SGLang | ◄━━━━━━━━━━━━━━━━━━━━━━► | SGLang |
| Prefill | KV cache blocks | Decode |
+-----------+ +-----------+
```
## Prerequisites
## Installation
### Prerequisite
```bash
pip3 install mooncake-transfer-engine
```
Note: If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](build.md).
If you encounter missing `lib*.so` errors, uninstall and build from source instead:
### Install the latest version of SGLang
#### 1. Clone SGLang from official repo
```bash
git clone git@github.com:sgl-project/sglang.git
pip3 uninstall mooncake-transfer-engine
# Build from source — see build instructions
```
#### 2. Build
##### 2.1 Build from source
### Install SGLang
It is recommended to use uv for faster installation:
```bash
cd sglang
pip install --upgrade pip
pip install -e "python[all]" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python
pip install uv
uv pip install sglang
```
##### If running on AMD GPU, tried below steps to install sglang for rocm or just run in rocm docker from https://hub.docker.com/r/rocm/sgl-dev directly
The major version of Cuda is 13 by default. To install sglang under Cuda 12 with pip or uv, please try the following commands:
```bash
pip install --upgrade pip
cd sgl-kernel
python setup_rocm.py install
cd ..
pip install -e "python[all_hip]"
or
docker run -it --rm --network=host \
--device=/dev/kfd --device=/dev/dri \
--ipc=host --shm-size 16G \
--group-add video \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v /home/workspace:/workspace \
rocm/sgl-dev:20250707
pip3 install mooncake-transfer-engine
sudo apt update
sudo apt install libibverbs1 libibverbs-dev -y
sudo apt-get install lsof net-tools iputils-ping -y
pip install uv
uv pip install sglang
uv pip install --force-reinstall torch==2.11.0 torchaudio==2.11.0 torchvision --index-url https://download.pytorch.org/whl/cu129
uv pip install --force-reinstall sglang-kernel --index-url https://docs.sglang.ai/whl/cu129/
uv pip install --force-reinstall sgl-deep-gemm --index-url https://docs.sglang.ai/whl/cu129/ --no-deps
```
- If you encounter any problems that you cannot solve, please refer to the [SGLang official compilation guide](https://docs.sglang.ai/start/install.html).
See the [SGLang official compilation guide](https://docs.sglang.ai/start/install.html) if you encounter issues.
## Configuration
- **Update(Apr 10, 2025)** Good news: The configuration file requirement has been removed since [PR 5460](https://github.com/sgl-project/sglang/pull/5460). There is no need to prepare the _mooncake.json_ file anymore.
Key arguments:
| Argument | Description |
|----------|-------------|
| `--disaggregation-mode` | `prefill` or `decode` — node role |
| `--disaggregation-ib-device` | RDMA device(s). Auto-detected, comma-separated for multi-NIC |
| `--tp-size` | Tensor parallelism size (optional) |
| `--base-gpu-id` | Starting GPU index for same-node deployments |
| `--host` / `--port` | SGLang service address |
## Run PD Disaggregation
### To run prefill instance and decode instance on different node
### Multi-Node
Prefill:
```bash
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --disaggregation-mode prefill --port 30000 --host 192.168.0.137 --tp-size 2
```
- The `--model-path` parameter specifies the model to use.
- The `--host` parameter specifies the SGLang service host.
- The `--port` parameter specifies the SGLang service port on which to listen.
- The `--disaggregation-mode` is the node's role, either 'prefill' or 'decode'.
- The `--disaggregation-ib-device` is the device to be used for data transmission, it is optional since we will detect this config automatically. Or you can still explicitly specify devices if needed. If multiple NIC devices are used, they can be separated by commas, such as "erdma_0,erdma_1". Please note that there are no spaces between them.
- Option `--tp-size` is supported. Example: append `--tp-size 2` to the run command to run SGLang with multiple GPUs.
# Prefill node (192.168.0.137)
python -m sglang.launch_server \
--model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--disaggregation-mode prefill \
--port 30000 --host 192.168.0.137 --tp-size 2
Decode:
```bash
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --disaggregation-mode decode --port 30001 --host 192.168.0.140 --tp-size 2
```
# Decode node (192.168.0.140)
python -m sglang.launch_server \
--model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--disaggregation-mode decode \
--port 30001 --host 192.168.0.140 --tp-size 2
Proxy:
```bash
python3 -m sglang.srt.disaggregation.mini_lb --prefill http://192.168.0.137:30000 --decode http://192.168.0.140:30001 --host 0.0.0.0 --port 8000
# Router
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--prefill "http://192.168.0.137:30000" 8998 \
--decode "http://192.168.0.140:30001" \
--policy round_robin \
--host 0.0.0.0 --port 8000
```
Test:
```bash
curl -X POST http://127.0.0.1:8000/generate -H "Content-Type: application/json" -d '{
"text": "Let me tell you a long story ",
"sampling_params": {
"temperature": 0
}
}'
curl -X POST http://127.0.0.1:8000/generate \
-H "Content-Type: application/json" \
-d '{"text": "Tell me a long story", "sampling_params": {"temperature": 0}}'
```
### To run prefill instance and decode instance on the same node
### Single-Node
Prefill:
```bash
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --disaggregation-mode prefill --port 30000 --host 192.168.0.137 --tp-size 2
```
Decode:
```bash
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --disaggregation-mode decode --port 30001 --base-gpu-id 2 --host 192.168.0.137 --tp-size 2
```
- The function of `--base-gpu-id` is similar to env var`CUDA_VISIBLE_DEVICES`, which is used to avoid reusing the 0th GPU card. The difference is that it is used to specify the starting number of the GPU. If it is set to 2, the first and second cards will be skipped, and the third card will be used directly.
Proxy:
```bash
python3 -m sglang.srt.disaggregation.mini_lb --prefill http://192.168.0.137:30000 --decode http://192.168.0.137:30001 --host 0.0.0.0 --port 8000
```
Test:
```bash
curl -X POST http://127.0.0.1:8000/generate -H "Content-Type: application/json" -d '{
"text": "Let me tell you a long story ",
"sampling_params": {
"temperature": 0
}
}'
```
Note:
- TP is supported but not required, you can remove `--tp-size 2` if you want.
- The `--disaggregation-ib-device` is the device to be used for data transmission, it is optional since we will detect this config automatically. Or you can still explicitly specify devices if needed. If multiple NIC devices are used, they can be separated by commas, such as "erdma_0,erdma_1". Please note that there are no spaces between them.
- XpYd is supported. It is ok to run one prefill and multiple decode instances on the same node, however, multiple prefill instances on the same node are not supported due to the port conflict of bootstrap server.
- e.g., `python3 -m sglang.srt.disaggregation.mini_lb --prefill http://192.168.0.137:30000,http://192.168.0.140:30000 --decode http://192.168.0.137:30001,http://192.168.0.140:30001 --host 0.0.0.0 --port 8000`
- HuggingFace timeout can be addressed by `export SGLANG_USE_MODELSCOPE=true`
### To enable Mooncake EP Backend
Prefill:
```bash
python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3-0324 --disaggregation-mode prefill --port 30000 --host 192.168.0.137 --tp-size 8 --dp-size 8 --elastic-ep-backend mooncake --mooncake-ib-device <mooncake-ib-device> --moe-a2a-backend mooncake
```
Decode:
```bash
python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3-0324 --disaggregation-mode decode --port 30001 --host 192.168.0.140 --tp-size 8 --dp-size 8 --elastic-ep-backend mooncake --mooncake-ib-device <mooncake-ib-device> --moe-a2a-backend mooncake
```
- Set `--elastic-ep-backend` and `--moe-a2a-backend` to "mooncake" to enable Mooncake EP Backend.
- The value of `--mooncake-ib-device` should be the same as `--disaggregation-ib-device`.
### To enable Mooncake EPD Backend
Encoder:
```bash
# Prefill
python -m sglang.launch_server \
--model-path $MODEL \
--encoder-only \
--encoder-transfer-backend mooncake \
--port $PORT
```
--model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--disaggregation-mode prefill \
--port 30000 --host 192.168.0.137 --tp-size 2
Prefill:
```bash
# Decode (skip first 2 GPUs, use cards 2+)
python -m sglang.launch_server \
--model-path $MODEL \
--disaggregation-mode prefill \
--disaggregation-transfer-backend mooncake \
--encoder-transfer-backend mooncake \
--tp $TP \
--mem-fraction-static $MEM_FRACTION \
--chunked-prefill-size $CHUNK_SIZE \
--language-only \
--encoder-urls http://127.0.0.1:30002 http://127.0.0.1:30003 http://127.0.0.1:30004 http://127.0.0.1:30005 http://127.0.0.1:30006 http://127.0.0.1:30007 \
--port $PORT
--model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--disaggregation-mode decode \
--port 30001 --base-gpu-id 2 --host 192.168.0.137 --tp-size 2
# Router
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--prefill "http://192.168.0.137:30000" 8998 \
--decode "http://192.168.0.137:30001" \
--policy round_robin \
--host 0.0.0.0 --port 8000
# Router for multi node.
# Here is an example for 2 decode node running on 192.168.0.137 and 192.168.0.140
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--prefill "http://192.168.0.137:30000" 8998 \
--decode "http://192.168.0.137:30001,http://192.168.0.140:30001" \
--policy round_robin \
--host 0.0.0.0 --port 8000
```
- Set `--encoder-transfer-backend` to "mooncake" to enable Mooncake Backend.
TP is supported but not required — omit `--tp-size 2` for single-GPU setups.
Multiple decode instances per prefill are supported. Multiple prefills on the same node are not supported due to bootstrap server port conflicts.
```{tip}
If you encounter HuggingFace timeouts, set `export SGLANG_USE_MODELSCOPE=true`.
```
## Advanced Backends
### EP Backend — Expert Parallelism for MoE Models
For Mixture-of-Experts models (e.g., DeepSeek-V3), different nodes hold different expert weights. During inference, Mooncake transfers expert activations between prefill and decode nodes via RDMA, replacing the NCCL-based all-to-all with a faster, disaggregation-aware transfer path.
```bash
# Prefill
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3-0324 \
--disaggregation-mode prefill \
--port 30000 --host 192.168.0.137 \
--tp-size 8 --dp-size 8 \
--elastic-ep-backend mooncake \
--moe-a2a-backend mooncake
# Decode
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3-0324 \
--disaggregation-mode decode \
--port 30001 --host 192.168.0.140 \
--tp-size 8 --dp-size 8 \
--elastic-ep-backend mooncake \
--moe-a2a-backend mooncake
```
Set `--mooncake-ib-device` to the same value as `--disaggregation-ib-device` if explicit device specification is needed.
### EPD Backend — Encoder-Prefill-Decode for Multimodal Models
For multimodal models (e.g., LLaVA, InternVL), the **encoder** (Vision Transformer) processes images, the **prefill** node handles text + encoded visual tokens, and the **decode** node generates output. Mooncake transfers encoder outputs (visual embeddings) between these nodes via RDMA.
The three node roles are:
| Role | Flag | Responsibility |
|------|------|----------------|
| Encoder | `--encoder-only` | Processes images, produces visual embeddings. Mooncake sends embeddings to prefill via `--encoder-transfer-backend mooncake` |
| Prefill | `--disaggregation-mode prefill` | Receives encoder embeddings, runs prefill with text + visual context. Mooncake transfers KV cache to decode |
| Decode | `--disaggregation-mode decode` | Receives KV cache from prefill, generates output tokens |
```bash
# Encoder-only node
python -m sglang.launch_server \
--model-path $MODEL \
--encoder-only \
--encoder-transfer-backend mooncake \
--port 30002
# Prefill node
python -m sglang.launch_server \
--model-path $MODEL \
--disaggregation-mode prefill \
--disaggregation-transfer-backend mooncake \
--encoder-transfer-backend mooncake \
--tp $TP \
--mem-fraction-static $MEM_FRACTION \
--chunked-prefill-size $CHUNK_SIZE \
--language-only \
--encoder-urls http://127.0.0.1:30002 http://127.0.0.1:30003 \
--port $PORT
# Decode node
python -m sglang.launch_server \
--model-path $MODEL \
--disaggregation-mode decode \
--disaggregation-transfer-backend mooncake \
--port $PORT
```

189
docs/source/getting_started/examples/sglang-integration/hicache-integration-v1.md
View File

@ -6,22 +6,6 @@ This document describes how to use Mooncake as the storage backend for SGLang Hi
## Introduction
### About Mooncake
Mooncake aims to enhance the inference efficiency of large language models (LLMs), especially in slow object storage environments, by constructing a multi-level caching pool on high-speed interconnected DRAM/SSD resources. Compared to traditional caching systems, Mooncake utilizes (GPUDirect) RDMA technology to transfer data directly in a zero-copy manner, while maximizing the use of multi-NIC resources on a single machine.
For more details about Mooncake, please refer to [Mooncake project](https://github.com/kvcache-ai/Mooncake) and [Mooncake documents](https://kvcache-ai.github.io/Mooncake/).
### About SGLang HiCache
SGLang HiCache is a hierarchical KV caching system that extends SGLang's RadixAttention with advanced multi-tier memory management capabilities. It implements a scalable hierarchical storage architecture that spans GPU memory, CPU memory, and external storage layers, delivering significant performance improvements for large language model inference.
HiCache introduces a **HiRadixTree** that acts as a page table for referencing KV caches across different memory tiers: **GPU Memory (L1)**, **CPU Memory (L2)**, **Mooncake and other Storage Backends (L3)**.
The system includes an intelligent cache controller that automatically manages data movement between tiers, implementing optimized prefetching strategies and multiple write policies (write-through, write-through-selective, and write-back).
For more details about SGLang HiCache, please refer to [HiCache system design document](https://docs.sglang.ai/advanced_features/hicache_design.html) and [this blog](https://lmsys.org/blog/2025-09-10-sglang-hicache/).
### Mooncake & SGLang HiCache
Mooncake serves as a high-performance L3 storage backend for SGLang HiCache, enabling distributed KV cache storage across multiple servers with RDMA-accelerated data transfer. This integration addresses the capacity limitations of traditional GPU-only or GPU+CPU caching by providing virtually unlimited cache storage through a distributed memory pool.
@ -42,21 +26,26 @@ This integration is particularly valuable for production deployments involving l
### Install SGLang
1. Clone SGLang from official repo
It is recommended to use uv for faster installation:
```bash
git clone git@github.com:sgl-project/sglang.git
```
2. Build
```bash
cd sglang
pip install --upgrade pip
pip install -e "python[all]"
pip install uv
uv pip install sglang
```
For more details, please refer to [SGLang official installation guide](https://docs.sglang.ai/get_started/install.html).
The major version of Cuda is 13 by default. To install sglang under Cuda 12 with pip or uv, please try the following commands:
```bash
pip install --upgrade pip
pip install uv
uv pip install sglang
uv pip install --force-reinstall torch==2.11.0 torchaudio==2.11.0 torchvision --index-url https://download.pytorch.org/whl/cu129
uv pip install --force-reinstall sglang-kernel --index-url https://docs.sglang.ai/whl/cu129/
uv pip install --force-reinstall sgl-deep-gemm --index-url https://docs.sglang.ai/whl/cu129/ --no-deps
```
See the [SGLang official compilation guide](https://docs.sglang.ai/start/install.html) if you encounter issues.
### Install Mooncake
@ -66,37 +55,7 @@ For more details, please refer to [SGLang official installation guide](https://d
pip install mooncake-transfer-engine
```
**Method 2: from source**
Clone Mooncake project:
```bash
git clone https://github.com/kvcache-ai/Mooncake --recursive
```
Install dependencies:
```bash
cd Mooncake
bash dependencies.sh
```
Build the project:
```bash
mkdir build
cd build
cmake ..
make -j
```
Install Mooncake:
```bash
sudo make install
```
For more details, please refer to [Mooncake official installation guide](https://kvcache-ai.github.io/Mooncake/getting_started/build.html).
If you want to build from source or using some other advanced features which not contained in prebuilt pip package, please refer to [Mooncake official installation guide](https://kvcache-ai.github.io/Mooncake/getting_started/build.html).
## Deployment
@ -365,54 +324,12 @@ python -m sglang.launch_server \
### Prefill/Decode Disaggregation
In **PD disaggregation**, the configurations for the `metadata service`, `mooncake master`, and the optional `store service` remain the same as described above. The difference is that SGLang introduces three distinct roles: `prefill worker`, `decode worker`, and `router`.
Mooncake HiCache works with SGLang's **PD disaggregation** mode. The `master service`, `metadata service`, and optional `store service` configurations are the same as described above.
Among these, the `prefill worker` supports enabling **HiCache**. To run with PD disaggregation, start from the [PD configuration](https://kvcache-ai.github.io/Mooncake/getting_started/examples/sglang-integration-v1.html), and add the HiCache-related parameters (as previously described for the `SGLang server`) to the `prefill worker`.
1. Follow the [PD Disaggregation Guide](../sglang-integration-v1) to set up the prefill, decode, and router workers.
2. Add the HiCache-related parameters (`--enable-hierarchical-cache`, `--hicache-storage-backend mooncake`, `--hicache-storage-prefetch-policy`, etc.) to the **prefill worker** only, as described in the HiCache sections above.
In the example below, one `prefill worker`, one `decode worker`, and one `router` are launched. HiCache is enabled on the `prefill worker` to optimize prefill performance.
**Prefill worker**:
```bash
MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata" \
MOONCAKE_MASTER=127.0.0.1:50051 \
MOONCAKE_PROTOCOL="rdma" \
MOONCAKE_DEVICE="mlx5_1" \
MOONCAKE_GLOBAL_SEGMENT_SIZE=4294967296 \
python -m sglang.launch_server \
--model-path [model_path] \
--page-size 64 \
--enable-hierarchical-cache \
--hicache-storage-prefetch-policy timeout \
--hicache-storage-backend mooncake \
--disaggregation-mode prefill \
--disaggregation-ib-device "mlx5_1" \
--base-gpu-id 0 \
--port 30000
```
**Decode worker**:
```bash
python -m sglang.launch_server \
--model-path [model_path] \
--page-size 64 \
--disaggregation-mode decode \
--disaggregation-ib-device "mlx5_1" \
--base-gpu-id 1 \
--port 30001
```
**Router**:
```bash
python -m sglang_router.launch_router \
--pd-disaggregation \
--prefill "http://127.0.0.1:30000" \
--decode "http://127.0.0.1:30001" \
--host 0.0.0.0 \
--port 8000
```
The Mooncake and HiCache configuration (environment variables or JSON config) is applied identically to the prefill worker — no changes are needed on the decode worker or router.
## Troubleshooting
@ -429,6 +346,72 @@ If the model is small but the GPU memory is large — especially in multi-TP (te
In such cases, you should manually configure an appropriate L2 cache size based on your hardware. This can be done by setting `--hicache-ratio` or `--hicache-size`.
**HugeTLB Bring-up Checklist:**
Before enabling HugeTLB-backed HiCache memory, size the host's hugepage pool against your planned `--hicache-size`, `MOONCAKE_GLOBAL_SEGMENT_SIZE`, and `MC_MMAP_ARENA_POOL_SIZE` values.
From a source checkout:
```bash
python3 scripts/check_hicache_hugepage_requirements.py \
--tp-size 4 \
--hicache-size 64gb \
--global-segment-size 8gb \
--arena-pool-size 56gb \
--available-hugetlb 512gb
```
From the source-built Docker image:
```bash
mooncake-hicache-sizing \
--tp-size 4 \
--hicache-size 64gb \
--global-segment-size 8gb \
--arena-pool-size 56gb
```
The `64gb` / `56gb` values above are tuned examples for large multi-GPU HiCache deployments, not defaults. Arena enablement remains opt-in, and the default pool is `8gb` only when you enable the arena via gflag without an explicit env override. On smaller hosts, start with `8gb` or `16gb` and size upward with the helper.
The helper reports two numbers:
* **Baseline floor** = `hicache-size + MOONCAKE_GLOBAL_SEGMENT_SIZE` per rank. Falling below this usually means startup or allocation failure.
* **Clean arena target** = baseline floor + `MC_MMAP_ARENA_POOL_SIZE` per rank. Reaching this target makes arena-backed launches less likely to spill onto the regular-page fallback path.
Treat the helper as a planning tool rather than a hard proof: runs can still succeed between the baseline floor and the clean arena target if some arena allocations fall back to regular pages.
Reserve HugeTLB pages on the host before launching SGLang:
```bash
sudo sysctl -w vm.nr_hugepages=262144
grep -E 'HugePages_Total|HugePages_Free|Hugepagesize' /proc/meminfo
```
With `2 MiB` pages, `262144` pages equals `512 GiB`; `49152` pages equals `96 GiB`. Persist the setting with `/etc/sysctl.d/90-mooncake-hugepages.conf` if you need it across reboots.
**Memory Allocator Tuning:**
Mooncake's mmap arena is opt-in for HiCache host KV allocations. Setting `MC_MMAP_ARENA_POOL_SIZE` explicitly enables the arena and sizes the pool; the arena then pre-allocates a hugepage-backed pool and serves subsequent allocations via atomic bump pointer, reducing per-allocation latency from ~1000ns to ~50ns. The `56gb` example below is a benchmark-scale tuning value, not the allocator default.
For HugeTLB-backed runs, export the hugepage and allocator settings together:
```bash
export MC_STORE_USE_HUGEPAGE=1
export MC_STORE_HUGEPAGE_SIZE=2MB
export MOONCAKE_GLOBAL_SEGMENT_SIZE=8gb
export MC_MMAP_ARENA_POOL_SIZE=56gb
```
To disable the arena and fall back to direct `mmap()` while keeping the hugepage-backed baseline path, set the flag before the first Mooncake mmap-buffer allocation in the process:
```bash
export MC_DISABLE_MMAP_ARENA=1
```
Without `MC_STORE_USE_HUGEPAGE=1`, the arena may opportunistically try hugepages and then retry on regular pages if HugeTLB is unavailable. If `MC_STORE_USE_HUGEPAGE=1` is set, Mooncake treats hugepages as a hard requirement for both the arena path and the direct-`mmap()` baseline path. It will not silently retry those host-buffer allocations on regular pages.
If the helper reports `baseline_fits_arena_may_fallback`, either increase `vm.nr_hugepages` or reduce `MC_MMAP_ARENA_POOL_SIZE`. For containerized launches, pass the same environment variables through `docker run -e ...` and use `--ipc=host --ulimit memlock=-1 --shm-size=128g`.
**More Information:**
Additional troubleshooting information can be found [here](https://kvcache-ai.github.io/Mooncake/troubleshooting/troubleshooting.html).

31
docs/source/getting_started/examples/sglang-integration/hicache-quick-start.md
View File

@ -1,6 +1,6 @@
# Quick Start: SGLang HiCache with Mooncake Backend
Follow this streamlined workflow to get SGLang HiCache running with Mooncake as the L3 storage backend.
Follow this streamlined workflow to get SGLang HiCache running with Mooncake as the L3 storage backend. In benchmarks, pre-populated Mooncake achieves **best TTFT** across all tiers, maintaining high cache hit rates as conversation rounds grow ([details](../../../performance/sglang-hicache-benchmark-results-v1)).
> Need more background or tuning options? See the [Complete Guide](hicache-integration-v1.md).
@ -10,21 +10,43 @@ Before you begin, make sure that:
- SGLang is installed with HiCache support on the machine hosting your SGLang server. Refer to the [official installation guide](https://docs.sglang.ai/get_started/install.html) if needed.
- Mooncake is installed and accessible as the hierarchical cache backend. Detailed build steps live in the [Mooncake documentation](https://kvcache-ai.github.io/Mooncake/getting_started/build.html).
- If you want HugeTLB-backed host KV memory, reserve the hugepages on the host before launching SGLang. The helper `python3 scripts/check_hicache_hugepage_requirements.py ...` from a source checkout, or `mooncake-hicache-sizing ...` inside the source-built Docker image, can size the required budget from your `--hicache-size`, `MOONCAKE_GLOBAL_SEGMENT_SIZE`, and `MC_MMAP_ARENA_POOL_SIZE` values.
- The router package is installed to provide the `sglang_router` entrypoint:
```bash
pip install sglang-router
```
### 1. Launch the Mooncake master service
### 1. Size and reserve HugeTLB on the host
```bash
python3 scripts/check_hicache_hugepage_requirements.py \
--tp-size 4 \
--hicache-size 64gb \
--global-segment-size 8gb \
--arena-pool-size 56gb \
--available-hugetlb 512gb
sudo sysctl -w vm.nr_hugepages=262144
grep -E 'HugePages_Total|HugePages_Free|Hugepagesize' /proc/meminfo
```
`262144` is an example for `512 GiB` of `2 MiB` pages. The `64gb` / `56gb` sizing inputs above are tuned examples for large multi-GPU runs, not defaults. Adjust them, and the resulting `vm.nr_hugepages`, to match your planned HiCache budget. On smaller hosts, start with an arena pool of `8gb` or `16gb`.
### 2. Launch the Mooncake master service
```bash
mooncake_master --enable_http_metadata_server=true
```
### 2. Launch SGLang with Mooncake L3 storage
### 3. Launch SGLang with Mooncake L3 storage
```bash
export MC_STORE_USE_HUGEPAGE="1"
export MC_STORE_HUGEPAGE_SIZE="2MB"
export MOONCAKE_GLOBAL_SEGMENT_SIZE="8gb"
export MC_MMAP_ARENA_POOL_SIZE="56gb"
MOONCAKE_MASTER=127.0.0.1:50051 python -m sglang.launch_server \
--model-path [model_path] \
--page-size 64 \
@ -35,6 +57,8 @@ MOONCAKE_MASTER=127.0.0.1:50051 python -m sglang.launch_server \
**Key flag:** `--hicache-storage-prefetch-policy {best_effort,wait_complete,timeout}` determines when prefetching from storage should stop. `timeout` usually offers the best balance when Mooncake is the backend.
**Memory allocator:** Mooncake's mmap arena is opt-in. The `56gb` example above is a benchmark-scale tuning value, not the default. Setting `MC_MMAP_ARENA_POOL_SIZE` enables it and chooses the pool size; if you enable it via gflag instead, the default pool size is `8gb`. To force the baseline direct-`mmap()` path, set `MC_DISABLE_MMAP_ARENA=1` (also accepts `true`, `yes`, or `on`) before the first Mooncake mmap-buffer allocation in the process. See the [Complete Guide](hicache-integration-v1.md) for the full HugeTLB sizing and troubleshooting flow.
## Prefill/Decode Disaggregation
The disaggregated setup runs three processes—prefill worker, decode worker, and router. Launch each command below in its own terminal window.
@ -94,6 +118,7 @@ curl -X POST http://127.0.0.1:8000/generate \
- `--disaggregation-ib-device` is optional—SGLang autodetects devices, but you can set it explicitly (comma-separated, no spaces) when multiple NICs are available.
- Use `--tp-size` to enable tensor-parallel execution across GPUs if required.
- For Docker launches, use `--ipc=host`, `--ulimit memlock=-1`, and a large `--shm-size`, then pass the same `MC_STORE_USE_HUGEPAGE`, `MC_STORE_HUGEPAGE_SIZE`, `MOONCAKE_GLOBAL_SEGMENT_SIZE`, and `MC_MMAP_ARENA_POOL_SIZE` values through `docker run -e ...`.
- Optional flags to experiment with once you need them (some still have compatibility gaps):
- `--disaggregation-decode-enable-offload-kvcache` writes the decode worker's outputs back into Mooncake; enable it when you want decoded KV to persist in L3.
- Launch dedicated Mooncake `store service` nodes when you want to scale L3 capacity beyond what the SGLang servers contribute.

48
docs/source/getting_started/examples/sglang-integration/index.md
View File

@ -1,8 +1,54 @@
# SGLang HiCache with Mooncake Backend
# Mooncake x SGLang Integration
Mooncake integrates with SGLang through two paths — **PD Disaggregation** for cross-instance KV cache transfer via the Transfer Engine, and **HiCache L3 Backend** for hierarchical KV cache storage with Mooncake Store.
---
## PD Disaggregation
SGLang uses Mooncake's Transfer Engine for direct zero-copy KV cache transfer between prefill and decode instances over RDMA, with support for EP and EPD backends. In benchmarks, PD disaggregation with Mooncake achieves **~30% lower ITL** while maintaining comparable throughput.
```
+-----------+ Transfer Engine (RDMA) +------------+
| SGLang | ◄━━━━━━━━━━━━━━━━━━━━━━► | SGLang |
| Prefill | KV cache blocks | Decode |
+-----------+ +------------+
```
**Related:** [Full PD Disaggregation Guide](../sglang-integration-v1) — installation, cross-node/same-node setup, XpYd topology, EP backend for MoE models, and EPD backend for multimodal models.
**Benchmark:** [PD Disaggregation Performance](../../../performance/sglang-benchmark-results-v1) — compares 1P1D disaggregation with regular SGLang instances.
---
## HiCache with Mooncake Store
HiCache extends SGLang's RadixAttention with three memory tiers, using Mooncake Store as the distributed L3 backend. When local cache misses, HiCache automatically prefetches KV blocks from remote storage via RDMA.
```
+----------------------------------------------+
| SGLang + HiCache |
| ┌─────────┐ ┌─────────┐ ┌──────────────┐ |
| │ L1(GPU) │ │ L2(CPU) │ │ L3(Mooncake) │ |
| └─────────┘ └─────────┘ └──────┬───────┘ |
+-----------------------------------+----------+
|
+--------+---------+
| Mooncake Store |
| Distributed Pool |
+------------------+
```
**Related:**
- [HiCache Quick Start](hicache-quick-start) — minimal setup steps with hugepage sizing
- [HiCache Complete Guide](hicache-integration-v1) — full deployment, prefetch strategies, memory tuning, and architecture deep dive
- [SGLang Performance Benchmarks](../../../performance/sglang/index) — benchmark overview for PD disaggregation and HiCache with Mooncake
::::{toctree}
:maxdepth: 1
:hidden:
../sglang-integration-v1
hicache-quick-start
hicache-integration-v1
::::

337
docs/source/getting_started/examples/vllm-integration/disagg-prefill-decode.md Normal file
View File

@ -0,0 +1,337 @@
# Disaggregated Prefill-Decode with MooncakeConnector
## Overview
This guide demonstrates how to use `MooncakeConnector` with vLLM for disaggregated Prefill-Decode (PD) serving. `MooncakeConnector` enables direct cross-node KV cache transfer between prefill and decode instances using RDMA technology, achieving up to **142.25 GB/s** peak bandwidth (71.1% utilization of 8x RoCE).
For more details about Mooncake, please refer to [Mooncake project](https://github.com/kvcache-ai/Mooncake) and [Mooncake documents](https://kvcache-ai.github.io/Mooncake/).
---
## Choose Your vLLM Backend
| Backend | vLLM Version | Status | Guide |
|---------|-------------|--------|-------|
| **vLLM V1** | Latest | Recommended | [Jump to V1 guide](#using-vllm-v1-recommended) |
| **vLLM V0** | ≤ v0.6.4.post1 | Legacy | [Jump to V0 guide](#using-vllm-v0-legacy) |
```{admonition} New Users
:class: tip
If you are starting a new deployment, use the **vLLM V1** backend. V0 support is maintained for existing deployments only.
```
---
## Using vLLM V1 (Recommended)
This section covers `MooncakeConnector` integration with vLLM V1 backend. The integration enables efficient cross-node KV cache transfer via RDMA.
### Installation
#### Prerequisites
Install mooncake-transfer-engine through pip:
```bash
pip install mooncake-transfer-engine
```
```{note}
If you encounter problems such as missing `lib*.so`, uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](../../build.md).
```
#### Install vLLM
Refer to [vLLM official installation guide](https://docs.vllm.ai/en/latest/getting_started/installation.html) for the latest installation instructions.
### Usage
#### Basic Setup (Different Nodes)
**Prefiller Node** (192.168.0.2):
```bash
vllm serve Qwen/Qwen2.5-7B-Instruct \
--port 8010 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer"}'
```
**Decoder Node** (192.168.0.3):
```bash
vllm serve Qwen/Qwen2.5-7B-Instruct \
--port 8020 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer"}'
```
**Proxy Server:**
```bash
# In vllm root directory.
python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
--prefiller-host 192.168.0.2 --prefiller-port 8010 \
--decoder-host 192.168.0.3 --decoder-port 8020
```
> NOTE: The Mooncake Connector currently uses the proxy from nixl_integration. This will be replaced with a self-developed proxy in the future.
Now you can send requests to the proxy server through port 8000.
**Test:**
```bash
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [
{"role": "user", "content": "Tell me a long story about artificial intelligence."}
]
}'
```
#### Advanced Configuration
**With Tensor Parallelism:**
Prefiller:
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
vllm serve Qwen/Qwen2.5-7B-Instruct \
--port 8010 \
--tensor-parallel-size 8 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer"}'
```
Decoder:
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
vllm serve Qwen/Qwen2.5-7B-Instruct \
--port 8020 \
--tensor-parallel-size 8 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer"}'
```
#### Configuration Parameters
- `--kv-transfer-config`: JSON string to configure the KV transfer connector
- `kv_connector`: Set to `"MooncakeConnector"`
- `kv_role`: Role of the instance
- `kv_producer`: For prefiller instances that generate KV caches
- `kv_consumer`: For decoder instances that consume KV caches
- `kv_both`: Enables symmetric functionality (experimental)
- `num_workers`: Thread pool size in each prefiller worker to send kvcache (default 10)
### Environment Variables
- `VLLM_MOONCAKE_BOOTSTRAP_PORT`: Port for Mooncake bootstrap server (default: 8998)
- Required only for prefiller instances
- Each vLLM worker needs a unique port on its host
- For TP/DP deployments, each worker's port is computed as: `base_port + dp_rank * tp_size + tp_rank`
- `VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT`: Timeout (in seconds) for automatically releasing KV cache (default: 480)
- Used when a request is aborted to prevent holding resources indefinitely
### Performance
For detailed performance benchmarks and results, see the [vLLM Benchmark](../../../performance/vllm-v1-support-benchmark.md) documentation.
---
## Using vLLM V0 (Legacy)
```{admonition} Legacy Backend
:class: warning
This section is for vLLM V0 backend (≤ v0.6.4.post1). For new deployments, use the [V1 backend](#using-vllm-v1-recommended) above.
```
This integration is based on [PR 10502](https://github.com/vllm-project/vllm/pull/10502) and [PR 10884](https://github.com/vllm-project/vllm/pull/10884). Preview benchmark results are available at [vLLM Benchmark Results V0.2](../../../performance/vllm-benchmark-results-v0.2.md).
### Installation
#### Prerequisite
```bash
pip3 install mooncake-transfer-engine
```
```{note}
- If you encounter problems such as missing `lib*.so`, uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](../../build.md).
- For vLLM version ≤ v0.8.4, it requires `mooncake-transfer-engine ≤ 0.3.3.post2`. In the latest release, the interface `mooncake_vllm_adaptor` has been deprecated.
```
#### Install vLLM
**1. Clone vLLM from official repo:**
```bash
git clone git@github.com:vllm-project/vllm.git
```
**2. Build from source (Include C++ and CUDA code):**
```bash
cd vllm
pip3 uninstall vllm -y
pip3 install -e .
```
```{tip}
If the build fails, try upgrading cmake: `pip3 install cmake --upgrade`.
```
If you encounter any problems, refer to the [vLLM official compilation guide](https://docs.vllm.ai/en/v0.6.4.post1/getting_started/installation.html#install-the-latest-code).
### Configuration
#### Prepare configuration file over RDMA
Create a `mooncake.json` file for both Prefill and Decode instances. Use the identical config file on both sides.
```json
{
"prefill_url": "192.168.0.137:13003",
"decode_url": "192.168.0.139:13003",
"metadata_server": "192.168.0.139:2379",
"metadata_backend": "etcd",
"protocol": "rdma",
"device_name": "erdma_0"
}
```
- `prefill_url`: The IP address and port of the Prefill node (port is used to communicate with metadata server).
- `decode_url`: The IP address and port of the Decode node. If running prefill and decode on the same node, set a different port (at least 50 apart from `prefill_url` port) to avoid conflicts.
- `metadata_server`: The metadata server address. Supports `etcd`, `redis`, and `http` backends. Example: `"etcd://192.168.0.137:2379"`, `"redis://192.168.0.137:6379"`, `"http://192.168.0.137:8080/metadata"`.
- `metadata_backend`: Currently supports `"etcd"`, `"redis"`, and `"http"`. If absent and `metadata_server` has no prefix, defaults to `"etcd"`. This parameter will be deprecated in a future version.
- `protocol`: `"rdma"` or `"tcp"`.
- `device_name`: Required when protocol is `"rdma"`. Multiple NICs can be separated by commas (`"erdma_0,erdma_1"`).
#### Prepare configuration file over TCP
```json
{
"prefill_url": "192.168.0.137:13003",
"decode_url": "192.168.0.139:13003",
"metadata_server": "192.168.0.139:2379",
"metadata_backend": "etcd",
"protocol": "tcp",
"device_name": ""
}
```
### Run Example
Change the IP addresses and ports according to your environment.
```bash
# Begin from root of your cloned repo!
# 1. Start the etcd server
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379
# 2. Run on the prefilling side (producer role)
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8100 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":2e9}'
# 3. Run on the decoding side (consumer role)
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8200 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":2e9}'
```
**Key parameters:**
- `MOONCAKE_CONFIG_PATH`: Path to the mooncake.json configuration file.
- `VLLM_USE_MODELSCOPE`: Optional. Remove if you have HuggingFace access.
- `--kv-transfer-config`: Connector configuration
- `kv_connector`: `"MooncakeConnector"`
- `kv_role`: `"kv_producer"` or `"kv_consumer"`
- `kv_rank`: 0 for producer, 1 for consumer
- `kv_parallel_size`: Fixed to 2 currently
- `kv_buffer_size`: KVCache lookup buffer size; increase for longer prompts. If OOM occurs, decrease `--gpu-memory-utilization`.
- `kv_ip` and `kv_port`: Used to specify the IP address and port of the master node for `"PyNcclConnector"` distributed setup. Not used for `"MooncakeConnector"` currently. Instead, `"MooncakeConnector"` uses a config file to set up the distributed connection.
- `--tensor-parallel-size` / `-tp`: Supported. If running on the same node, set different `CUDA_VISIBLE_DEVICES`.
```{note}
If running prefill and decode on the same node, set a different port for `decode_url`. To avoid port conflicts, ensure the decode port differs by at least 50 from the `prefill_url` port (e.g., `"decode_url": "192.168.0.137:13103"`). If the same URL is set for both, the port of `decode_url` will be automatically incremented by 100.
```
**Proxy Server:**
```bash
python3 proxy_server.py
```
```python
# proxy_server.py
import os
import aiohttp
from quart import Quart, make_response, request
AIOHTTP_TIMEOUT = aiohttp.ClientTimeout(total=6 * 60 * 60)
app = Quart(__name__)
async def forward_request(url, data):
async with aiohttp.ClientSession(timeout=AIOHTTP_TIMEOUT) as session:
headers = {"Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}"}
async with session.post(url=url, json=data, headers=headers) as response:
if response.status == 200:
async for chunk_bytes in response.content.iter_chunked(1024):
yield chunk_bytes
@app.route('/v1/completions', methods=['POST'])
async def handle_request():
try:
original_request_data = await request.get_json()
prefill_request = original_request_data.copy()
prefill_request['max_tokens'] = 1 # prefill only
async for _ in forward_request('http://localhost:8100/v1/completions', prefill_request):
continue
generator = forward_request('http://192.168.0.139:8200/v1/completions', # Change IP
original_request_data)
response = await make_response(generator)
response.timeout = None
return response
except Exception as e:
import sys, traceback
exc_info = sys.exc_info()
print("Error occurred in disagg prefill proxy server")
print(e)
print("".join(traceback.format_exception(*exc_info)))
if __name__ == '__main__':
app.run(host="0.0.0.0", port=8000)
```
> Be sure to change the IP address in the proxy server code.
### Test
```bash
curl -s http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4",
"prompt": "San Francisco is a",
"max_tokens": 1000
}'
```
---
## Troubleshooting
- If you encounter connection issues, check that:
- All nodes can reach each other over the network
- Firewall rules allow traffic on the specified ports
- RDMA devices are properly configured and listed in `device_name`
- For missing library errors, rebuild `mooncake-transfer-engine` from source
- Enable debug logging with `VLLM_LOGGING_LEVEL=DEBUG` for detailed diagnostics
- For production deployments, consider using a more robust proxy solution

53
docs/source/getting_started/examples/vllm-integration/index.md
View File

@ -1,12 +1,55 @@
# vLLM Disaggregated Serving
# Mooncake x vLLM Integration
## Overview
Mooncake integrates with vLLM to accelerate large language model serving through high-performance KV cache transfer and shared storage. The integration supports two primary scenarios:
- **Disaggregated Prefill-Decode Serving**: Seamlessly split prefill and decode across nodes using `MooncakeConnector`, with RDMA-powered cross-node KV cache transfer achieving up to **142.25 GB/s** peak bandwidth (71.1% utilization of 8x RoCE). Transfer overhead is negligible &mdash; for 32K-token prompts (4.50 GB of KV data), transfer takes only **31.65 ms**, accounting for just **4.2%** of total TTFT.
- **KV Cache Storage & Sharing**: Extend effective KV cache capacity via `MooncakeStore` / `MooncakeStoreConnector`, with hash-based prefix caching that enables multiple vLLM instances to share cached KV blocks. Supports CPU/Disk offloading and dynamic XpYd topologies at runtime.
| Scenario | Guide | vLLM Backend |
|----------|-------|-------------|
| PD Disaggregation (KV transfer) | [Disaggregated Prefill-Decode](disagg-prefill-decode) | V1 ✅ / V0 ⚠️ |
| KV Cache Storage & Sharing | [KV Cache Storage with MooncakeStore](kv-cache-storage) | V1 ✅ / V0 ⚠️ |
For detailed benchmark coverage across these scenarios, see [vLLM Integration Performance Benchmarks](../../../performance/vllm/index).
```{admonition} New to Mooncake + vLLM?
:class: tip
Start with the V1 guides above. Legacy V0 documentation is available for existing deployments only.
```
---
## Getting Started
### Disaggregated Prefill-Decode
Direct KV cache transfer between prefill and decode nodes via `MooncakeConnector` using RDMA.
::::{toctree}
:maxdepth: 1
vllmv1-lmcache-integration
vllm-integration-v0.2
vllm-integration-v0.3
vllm-integration-v1.0
disagg-prefill-decode
::::
### KV Cache Storage & Sharing
Distributed KV cache storage via `MooncakeStore` / `MooncakeStoreConnector` for offloading, prefix caching, and cross-instance sharing.
::::{toctree}
:maxdepth: 1
kv-cache-storage
::::
---
## Archived Documentation
The following pages are from earlier versions of the integration and are no longer maintained. All content has been consolidated into the scenario-based guides above.
- [MooncakeStoreConnector (Original)](vllm-mooncakestoreconnector)
- [vLLM V0 PD Disaggregation Demo (Original)](vllm-integration-v0.2)
- [vLLM V0 MooncakeStore (Original)](vllm-integration-v0.3)
- [vLLM V1 PD Disaggregation (Original)](vllm-integration-v1.0)

409
docs/source/getting_started/examples/vllm-integration/kv-cache-storage.md Normal file
View File

@ -0,0 +1,409 @@
# KV Cache Storage & Sharing with MooncakeStore
## Overview
This guide demonstrates how to use `MooncakeStore` / `MooncakeStoreConnector` with vLLM to build a distributed KV cache storage pool. It enables KV cache offloading to CPU/SSD, hash-based prefix caching across multiple vLLM instances, and flexible XpYd disaggregated deployment — where you can dynamically adjust prefill and decode group sizes at runtime.
Compared to Redis-based backends, MooncakeStore achieves significantly lower TTFT (e.g., **~32% improvement** in mean TTFT for 2P2D tp=2 under RDMA). See [benchmark results](../../../performance/vllm-benchmark-results-v1.md) for details.
---
## Choose Your vLLM Backend
| Backend | Connector | vLLM Version | Status | Guide |
|---------|-----------|-------------|--------|-------|
| **vLLM V1** | `MooncakeStoreConnector` | Latest | Recommended | [Jump to V1 guide](#using-vllm-v1-recommended) |
| **vLLM V0** | `MooncakeStore` | ≤ v0.6.4.post1 | Legacy | [Jump to V0 guide](#using-vllm-v0-legacy) |
```{admonition} New Users
:class: tip
If you are starting a new deployment, use the **vLLM V1** backend with `MooncakeStoreConnector`. V0 support is maintained for existing deployments only.
```
Key differences from v0.x to v1:
- **XpYd support and orchestration**: Dynamically change the population of prefill and decode groups
- **More stable and fault-tolerant**: A sudden crash of a single vLLM instance is tolerable; instance-to-instance connections are removed, so each instance works as a vanilla vLLM instance capable of handling requests independently
---
## Using vLLM V1 (Recommended)
This section covers `MooncakeStoreConnector` — the new vLLM KV connector that uses `MooncakeDistributedStore` as a shared KV cache pool. It enables:
- **CPU/Disk offloading**: Extend effective KV cache capacity by offloading to CPU memory or SSD via Mooncake's transfer engine.
- **Hash-based prefix caching across instances**: Multiple vLLM instances share cached KV blocks through the store using block-hash deduplication.
- **Flexible deployment**: Works as a single-node KV cache extension (`kv_both`), or in disaggregated prefill-decode setups (`kv_producer` / `kv_consumer`).
### Deployment
#### 1. Prerequisites
- [vLLM](https://github.com/vllm-project/vllm) is installed
- [Mooncake](https://github.com/kvcache-ai/Mooncake) is installed
Refer to the [vLLM official repository](https://github.com/vllm-project/vllm) and [Mooncake official repository](https://github.com/kvcache-ai/Mooncake) for installation instructions and building from source.
#### 2. Mooncake Master Server
**Start:**
```shell
mooncake_master --port 50063
```
**Configure Mooncake**: Create a JSON configuration file (e.g., `mooncake_config.json`):
```json
{
"metadata_server": "http://127.0.0.1:8092/metadata",
"master_server_address": "127.0.0.1:50063",
"global_segment_size": "0",
"local_buffer_size": "2147483648",
"protocol": "rdma",
"device_name": ""
}
```
**Set environment variable:**
```shell
export MOONCAKE_CONFIG_PATH=/path/to/mooncake_config.json
```
#### 3. Usage
**3.1 Single-Node KV Cache Offloading** (`kv_both`):
```shell
MOONCAKE_CONFIG_PATH=mooncake_config.json \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both"}'
```
**3.2 XpYd Disaggregated Prefill-Decode** (`kv_producer/kv_consumer`):
Prefill Node:
```shell
MOONCAKE_CONFIG_PATH=mooncake_config.json \
VLLM_MOONCAKE_BOOTSTRAP_PORT=50052 \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--port 8100 \
--kv-transfer-config '{
"kv_connector": "MultiConnector",
"kv_role": "kv_producer",
"kv_connector_extra_config": {
"connectors": [
{
"kv_connector": "MooncakeConnector",
"kv_role": "kv_producer"
},
{
"kv_connector": "MooncakeStoreConnector",
"kv_role": "kv_producer"
}
]
}
}'
```
Decode Node:
```shell
MOONCAKE_CONFIG_PATH=mooncake_config.json \
VLLM_MOONCAKE_BOOTSTRAP_PORT=50053 \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--port 8200 \
--kv-transfer-config '{
"kv_connector": "MultiConnector",
"kv_role": "kv_consumer",
"kv_connector_extra_config": {
"connectors": [
{
"kv_connector": "MooncakeConnector",
"kv_role": "kv_consumer"
},
{
"kv_connector": "MooncakeStoreConnector",
"kv_role": "kv_consumer"
}
]
}
}'
```
Proxy:
```shell
python examples/disaggregated/disaggregated_serving/mooncake_connector/mooncake_connector_proxy.py \
--prefill http://192.168.0.2:8100 \
--decode http://192.168.0.3:8200
```
> When running with data parallelism, set a fixed `PYTHONHASHSEED` so that block hashes are consistent across DP ranks:
>
> ```shell
> PYTHONHASHSEED=0 vllm serve ...
> ```
>
> Without this, identical prompts may produce different block hashes on different DP ranks, preventing cross-instance prefix cache hits.
---
## Using vLLM V0 (Legacy)
```{admonition} Legacy Backend
:class: warning
This section is for vLLM V0 backend with `MooncakeStore`. For new deployments, use the [V1 backend with `MooncakeStoreConnector`](#using-vllm-v1-recommended) above.
```
This integration is based on [PR 10502](https://github.com/vllm-project/vllm/pull/10502) and [PR 12957](https://github.com/vllm-project/vllm/pull/12957) to support KVCache transfer for intra-node and inter-node disaggregated serving.
### Installation
#### Prerequisite
```bash
pip3 install mooncake-transfer-engine
```
```{note}
- If you encounter problems such as missing `lib*.so`, uninstall by `pip3 uninstall mooncake-transfer-engine`, and build manually according to the [instructions](../../build.md).
- For vLLM version ≤ v0.8.4, it requires `mooncake-transfer-engine ≤ 0.3.3.post2`. The interface `mooncake_vllm_adaptor` has been deprecated in the latest release.
```
#### Install vLLM
**1. Clone vLLM:**
```bash
git clone git@github.com:vllm-project/vllm.git
```
**2. Build from source:**
```bash
cd vllm
pip3 install -e .
```
If you encounter problems, refer to the [vLLM official compilation guide](https://docs.vllm.ai/en/latest/getting_started/installation/index.html).
### Configuration
#### Prepare configuration for RDMA
Create a `mooncake.json` file:
```json
{
"local_hostname": "192.168.0.137",
"metadata_server": "etcd://192.168.0.137:2379",
"protocol": "rdma",
"device_name": "erdma_0",
"master_server_address": "192.168.0.137:50001"
}
```
- `local_hostname`: The IP address of the current node. All prefill and decode instances on the same node can share this config.
- `metadata_server`: The metadata server. Supports `etcd`, `redis`, and `http` backends.
- `protocol`: `"rdma"` or `"tcp"`.
- `device_name`: Required for RDMA. Multiple NICs separated by commas (`"erdma_0,erdma_1"`).
- `master_server_address`: The IP address and port of the MooncakeStore master daemon.
#### Prepare configuration for TCP
```json
{
"local_hostname": "192.168.0.137",
"metadata_server": "etcd://192.168.0.137:2379",
"protocol": "tcp",
"device_name": "",
"master_server_address": "192.168.0.137:50001"
}
```
### Run Example
Change the IP addresses and ports according to your environment. `VLLM_USE_V1=0` is required for vLLM V0 backend.
```bash
# Begin from root of your cloned repo!
# 1. Start the etcd server
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379
# You may need to terminate other etcd processes before running the above command
# 2. Start the mooncake_master server
mooncake_master --port 50001
# If some vllm instances exit unexpectedly, some connection metadata will be
# corrupted since they are not properly cleaned. In that case, we recommend
# you restart the mooncake_master before running another test.
# 3. Run multiple vllm instances
# kv_producer role
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8100 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=1 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8101 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8102 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=3 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8103 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
# kv_consumer role
CUDA_VISIBLE_DEVICES=4 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8200 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=5 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8201 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=6 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8202 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=7 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8203 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
```
**Key parameters:**
- `MOONCAKE_CONFIG_PATH`: Path to the mooncake.json configuration file.
- `VLLM_USE_MODELSCOPE`: Optional. Remove if you have HuggingFace access.
- `VLLM_USE_V1=0`: Required since the disaggregated feature is currently only supported on V0 vLLM. You can also `export` this configuration to the env instead of putting it in front of every command.
- `--model`: The model to use.
- `--port`: The vllm service port on which to listen.
- `--max-model-len`: The maximum length of the model.
- `--tensor-parallel-size` / `-tp`: Supported. All instances should have the same tensor_parallel_size. If running prefill and decode on the same node, set different `CUDA_VISIBLE_DEVICES` (e.g., `CUDA_VISIBLE_DEVICES=0,1` for prefill and `CUDA_VISIBLE_DEVICES=2,3` for decode).
- `--kv-transfer-config`: Set `kv_connector` to `"MooncakeStoreConnector"`, `kv_role` to `"kv_producer"`, `"kv_consumer"`, or `"kv_both"`.
- If some vLLM instances exit unexpectedly, connection metadata may be corrupted. Restart `mooncake_master` before another test.
```bash
# 5. Start the proxy server
cd vllm
python3 examples/online_serving/disagg_examples/disagg_proxy_demo.py \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--prefill localhost:8100 localhost:8101 \
--decode localhost:8200 localhost:8201 \
--port 8000
```
- `--model`: The model and tokenizer used by the proxy server.
- `--port`: The proxy server port on which to listen.
- `--prefill` / `-p`: IP and port of the vllm prefill instances.
- `--decode` / `-d`: IP and port of the vllm decode instances.
#### Dynamic XpYd Adjustment
To dynamically adjust prefill and decode instances at runtime:
```bash
export ADMIN_API_KEY="xxxxxxxx"
# or add it before the command:
ADMIN_API_KEY="xxxxxxxx" python3 vllm/examples/online_serving/disagg_examples/disagg_demo.py \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--prefill localhost:8100 localhost:8101 \
--decode localhost:8200 localhost:8201 \
--port 8000 \
--scheduling round_robin
# Add instances to groups dynamically
curl -X POST "http://localhost:8000/instances/add" \
-H "Content-Type: application/json" \
-H "X-API-Key: $ADMIN_API_KEY" \
-d '{"type": "prefill", "instance": "localhost:8102"}'
curl -X POST "http://localhost:8000/instances/add" \
-H "Content-Type: application/json" \
-H "X-API-Key: $ADMIN_API_KEY" \
-d '{"type": "prefill", "instance": "localhost:8103"}'
curl -X POST "http://localhost:8000/instances/add" \
-H "Content-Type: application/json" \
-H "X-API-Key: $ADMIN_API_KEY" \
-d '{"type": "decode", "instance": "localhost:8202"}'
curl -X POST "http://localhost:8000/instances/add" \
-H "Content-Type: application/json" \
-H "X-API-Key: $ADMIN_API_KEY" \
-d '{"type": "decode", "instance": "localhost:8203"}'
# Get proxy status
curl localhost:8000/status | jq
```
```{note}
Mooncake team provides this simple round-robin proxy as a demo. In production, you can implement custom global proxy strategies.
```
**Be sure to change the IP address in the commands.**
### Test
```bash
curl -s http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4",
"prompt": "San Francisco is a",
"max_tokens": 1000
}'
```
- If you are not testing on the proxy server, change `localhost` to the IP address of the proxy server.
---
## Performance
| Scenario | Document |
|----------|----------|
| V1 MooncakeStoreConnector vs Redis | [Benchmark V1](../../../performance/vllm-benchmark-results-v1.md) |
| V0 MooncakeStore vs Redis | [Benchmark V0](../../../performance/vllm-benchmark-results-v0.2.md) |
---
## Troubleshooting
- If you encounter connection issues, check that:
- All nodes can reach each other over the network
- Firewall rules allow traffic on the specified ports
- RDMA devices are properly configured
- `mooncake_master` is running and reachable
- For missing library errors, rebuild `mooncake-transfer-engine` from source
- If vLLM instances exit unexpectedly, restart `mooncake_master` to clean up corrupted metadata
- Enable debug logging with `VLLM_LOGGING_LEVEL=DEBUG` for detailed diagnostics

21
docs/source/getting_started/examples/vllm-integration/vllm-integration-v0.2.md
View File

@ -1,5 +1,10 @@
# vLLM V0 Disaggregated Serving Demo
```{admonition} Archived
:class: warning
This page has been **consolidated** into the unified [Disaggregated Prefill-Decode](disagg-prefill-decode) guide (see the V0 Legacy section). Please use that guide for up-to-date information.
```
## Overview
This is the latest version of mooncake-transfer-engine integration doc with the vLLM project based on [PR 10502](https://github.com/vllm-project/vllm/pull/10502) and [PR 10884](https://github.com/vllm-project/vllm/pull/10884) (vllm version: v0.6.4.post1/main) to accelerate KVCache transfer for inter-node disaggregated serving scenario. We have run some experiments to obtain some [preview benchmark results](../../../performance/vllm-benchmark-results-v0.2.md). More benchmark results will be released in due time.
@ -16,7 +21,7 @@ pip3 install mooncake-transfer-engine
```
Note:
- If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](build.md).
- If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](../../build.md).
- For vLLM version <= v0.8.4, it requires mooncake-transfer-engine <= 0.3.3.post2. In the latest release, interface `mooncake_vllm_adaptor` has been deprecated.
### Install the latest version of vLLM
@ -89,10 +94,20 @@ etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://loc
# You may need to terminate other etcd processes before running the above command
# 2. Run on the prefilling side (producer role)
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8100 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":2e9}'
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8100 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":2e9}'
# 3. Run on the decoding side (consumer role)
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8200 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":2e9}'
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8200 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":2e9}'
```
- `MOONCAKE_CONFIG_PATH` is the path to the mooncake.json configuration file.

76
docs/source/getting_started/examples/vllm-integration/vllm-integration-v0.3.md
View File

@ -1,5 +1,10 @@
# vLLM V0 Disaggregated Serving with MooncakeStore
```{admonition} Archived
:class: warning
This page has been **consolidated** into the unified [KV Cache Storage & Sharing](kv-cache-storage) guide (see the V0 Legacy section). Please use that guide for up-to-date information.
```
## Overview
This is the latest version of the MooncakeStore integration doc with the vLLM project based on [PR 10502](https://github.com/vllm-project/vllm/pull/10502) and [PR 12957](https://github.com/vllm-project/vllm/pull/12957) to support KVCache transfer for intra-node and inter-node disaggregated serving scenario. Benchmark results will be released soon.
@ -24,7 +29,7 @@ pip3 install mooncake-transfer-engine
```
Note:
- If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](build.md).
- If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](../../build.md).
- For vLLM version <= v0.8.4, it requires mooncake-transfer-engine <= 0.3.3.post2. In the latest release, interface `mooncake_vllm_adaptor` has been deprecated.
### Install the latest version of vLLM
@ -91,22 +96,62 @@ mooncake_master --port 50001
# 3. Run multiple vllm instances
# kv_producer role
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8100 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8100 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=1 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8101 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=1 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8101 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8102 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8102 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=3 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8103 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=3 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8103 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
# kv_consumer role
CUDA_VISIBLE_DEVICES=4 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8200 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=4 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8200 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=5 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8201 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=5 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8201 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=6 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8202 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=6 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8202 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=7 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8203 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=7 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--port 8203 \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
```
- `MOONCAKE_CONFIG_PATH` is the path to the mooncake.json configuration file.
@ -127,7 +172,11 @@ CUDA_VISIBLE_DEVICES=7 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python
```bash
# 4. Start the proxy server
cd vllm
python3 examples/online_serving/disagg_examples/disagg_proxy_demo.py --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --prefill localhost:8100 localhost:8101 --decode localhost:8200 localhost:8201 --port 8000
python3 examples/online_serving/disagg_examples/disagg_proxy_demo.py \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--prefill localhost:8100 localhost:8101 \
--decode localhost:8200 localhost:8201 \
--port 8000
```
- The `--model` parameter specifies the model to use, also specifies the tokenizer used by the proxy server.
@ -139,7 +188,12 @@ python3 examples/online_serving/disagg_examples/disagg_proxy_demo.py --model Qwe
# If you want to dynamically adjust the instances of p-nodes and d-nodes during runtime, you need to configure this environment variables.
export ADMIN_API_KEY="xxxxxxxx"
# or add it before the command:
ADMIN_API_KEY="xxxxxxxx" python3 vllm/examples/online_serving/disagg_examples/disagg_demo.py --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --prefill localhost:8100 localhost:8101 --decode localhost:8200 localhost:8201 --port 8000 --scheduling round_robin
ADMIN_API_KEY="xxxxxxxx" python3 vllm/examples/online_serving/disagg_examples/disagg_demo.py \
--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
--prefill localhost:8100 localhost:8101 \
--decode localhost:8200 localhost:8201 \
--port 8000 \
--scheduling round_robin
# Then use this command to add instances into prefill group or decode group
curl -X POST "http://localhost:8000/instances/add" -H "Content-Type: application/json" -H "X-API-Key: $ADMIN_API_KEY" -d '{"type": "prefill", "instance": "localhost:8102"}'

11
docs/source/getting_started/examples/vllm-integration/vllm-integration-v1.0.md
View File

@ -1,5 +1,10 @@
# vLLM v1 backend Disaggregated Serving with MooncakeConnector
```{admonition} Archived
:class: warning
This page has been **consolidated** into the unified [Disaggregated Prefill-Decode](disagg-prefill-decode) guide. Please use that guide for up-to-date information.
```
## Overview
This guide demonstrates how to use the MooncakeConnector with vLLM v1 backend for disaggregated serving in Prefill-Decode separation architecture. The integration enables efficient cross-node KV cache transfer using RDMA technology.
@ -16,7 +21,7 @@ Install mooncake-transfer-engine through pip:
pip install mooncake-transfer-engine
```
Note: If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](../build.md).
Note: If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](../../build.md).
### Install vLLM
@ -100,7 +105,7 @@ vllm serve Qwen/Qwen2.5-7B-Instruct \
- `kv_producer`: For prefiller instances that generate KV caches
- `kv_consumer`: For decoder instances that consume KV caches
- `kv_both`: Enables symmetric functionality (experimental)
- `num_workers`: Thread pool size in each prefiller worker to send kvcache (default 10)
- `num_workers`: Thread pool size in each prefiller worker to send kvcache (default 10)
## Environment Variables
@ -118,7 +123,7 @@ The following environment variables can be used to customize Mooncake behavior:
## Performance
For detailed performance benchmarks and results, see the [vLLM Benchmark](../../performance/vllm-v1-support-benchmark.md) documentation.
For detailed performance benchmarks and results, see the [vLLM Benchmark](../../../performance/vllm-v1-support-benchmark.md) documentation.
## Notes

135
docs/source/getting_started/examples/vllm-integration/vllm-mooncakestoreconnector.md Normal file
View File

@ -0,0 +1,135 @@
# Guide: vLLM MooncakeStoreConnector
```{admonition} Archived
:class: warning
This page has been **consolidated** into the unified [KV Cache Storage & Sharing](kv-cache-storage) guide (see the V1 Recommended section). Please use that guide for up-to-date information.
```
## Overview
This document describes how to deploy vLLM's `MooncakeStoreConnector`. `MooncakeStoreConnector` is a new vLLM's KV connector that uses `MooncakeDistributedStore` as a shared KV cache pool. It enables:
* **CPU/Disk offloading**: Extend effective KV cache capacity by offloading to CPU memory or SSD via Mooncake's transfer engine.
* **Hash-based prefix caching across instances**: Multiple vLLM instances share cached KV blocks through the store using block-hash deduplication.
* **Flexible deployment**: Works as a single-node KV cache extension (`kv_both`), or in disaggregated prefill-decode setups (`kv_producer` / `kv_consumer`).
## Deployment
### 1. Prerequisites
Before you begin, make sure that:
* [vLLM](https://github.com/vllm-project/vllm) is installed, [Mooncake](https://github.com/kvcache-ai/Mooncake) is installed. Refer to the [vLLM official repository](https://github.com/vllm-project/vllm) and [Mooncake official repository](https://github.com/kvcache-ai/Mooncake) for more installation instructions and building from source.
### 2. Mooncake Master Server
**Start:**
```shell
mooncake_master --port 50063
```
**Configure Mooncake** : Create a JSON configuration file (e.g., `mooncake_config.json`):
```json
{
"metadata_server": "http://127.0.0.1:8092/metadata",
"master_server_address": "127.0.0.1:50063",
"global_segment_size": "0",
"local_buffer_size": "2147483648",
"protocol": "rdma",
"device_name": "",
}
```
**Set environment variable:**
```shell
export MOONCAKE_CONFIG_PATH=/path/to/mooncake_config.json
```
### 3. Usage
**3.1** **Single-Node KV Cache Offloading** (i.e., `kv_both`)
```shell
MOONCAKE_CONFIG_PATH=mooncake_config.json \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both"}'
```
**3.2** **XpYd Disaggregated Prefill-Decode** (i.e., `kv_producer/kv_consumer`)
Prefill Node:
```shell
MOONCAKE_CONFIG_PATH=mooncake_config.json \
VLLM_MOONCAKE_BOOTSTRAP_PORT=50052 \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--port 8100 \
--kv-transfer-config '{
"kv_connector": "MultiConnector",
"kv_role": "kv_producer",
"kv_connector_extra_config": {
"connectors": [
{
"kv_connector": "MooncakeConnector",
"kv_role": "kv_producer"
},
{
"kv_connector": "MooncakeStoreConnector",
"kv_role": "kv_producer"
}
]
}
}'
```
Decode Node:
```shell
MOONCAKE_CONFIG_PATH=mooncake_config.json \
VLLM_MOONCAKE_BOOTSTRAP_PORT=50053 \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--port 8200 \
--kv-transfer-config '{
"kv_connector": "MultiConnector",
"kv_role": "kv_consumer",
"kv_connector_extra_config": {
"connectors": [
{
"kv_connector": "MooncakeConnector",
"kv_role": "kv_consumer"
},
{
"kv_connector": "MooncakeStoreConnector",
"kv_role": "kv_consumer"
}
]
}
}'
```
Proxy:
```shell
python examples/disaggregated/disaggregated_serving/mooncake_connector/mooncake_connector_proxy.py \
--prefill http://192.168.0.2:8100 \
--decode http://192.168.0.3:8200
```
> When running with data parallelism, set a fixed `PYTHONHASHSEED` so that block hashes are consistent across DP ranks:
>
> ```shell
> PYTHONHASHSEED=0 vllm serve ...
> ```
>
> Without this, identical prompts may produce different block hashes on different DP ranks, preventing cross-instance prefix cache hits.

146
docs/source/getting_started/examples/vllm-integration/vllmv1-lmcache-integration.md
View File

@ -2,43 +2,73 @@
## Overview
The vLLM v1 version has been released with support for PD disaggregation. The detailed design document can be found [here](https://docs.google.com/document/d/1uPGdbEXksKXeN4Q9nUm9hzotqEjQhYmnpAhidLuAsjk). LMCache immediately implemented the corresponding connector to support storage, transmission, and loading of KVCache, enabling collaborative operation with PD nodes. Mooncake, as LMCache's backend storage engine, has undergone extensive optimizations in usability, performance, and stability. This document explains how to deploy a PD disaggregated serving demo using LMCache + Mooncake.
This guide shows a two-machine 1-prefill/1-decode deployment using vLLM V1,
LMCache's non-MP `LMCacheConnectorV1`, and Mooncake Store as LMCache's remote
storage backend.
LMCache supports both non-MP mode and MP mode with Mooncake Store. This page
covers the non-MP path, where each vLLM instance loads an LMCache YAML config
through `LMCACHE_CONFIG_FILE` and connects directly to Mooncake Store with
`remote_url: "mooncakestore://..."`. For the LMCache multiprocess server path,
see [vLLM V1 Disaggregated Serving with Mooncake Store and LMCache \[MP\]](vllmv1-lmcache-mp-integration.md).
The examples below use:
- Machine A: Mooncake master and vLLM decoder
- Machine B: vLLM prefiller
- Mooncake master RPC address: `{IP of Machine A}:50051`
- Mooncake HTTP metadata endpoint: `http://{IP of Machine A}:8080/metadata`
- RDMA device: `{RDMA device}`
Replace these placeholders with the IP addresses, hostname, and RDMA device for
your environment.
## Prerequisites
Install Mooncake, vLLM, and LMCache on both machines. For installation details,
refer to the official documentation of each project:
- [Mooncake build guide](../../build.md)
- [LMCache installation](https://docs.lmcache.ai/getting_started/installation.html)
- [vLLM installation](https://docs.vllm.ai/en/latest/getting_started/installation/)
## Deployment
1. First, you need to prepare two GPU-equipped machines, which we will refer to as Machine A and Machine B. Install [vLLM](https://docs.vllm.ai/en/latest/getting_started/quickstart.html), [Mooncake](https://kvcache-ai.github.io/Mooncake/getting_started/build.html) and [LMCache](https://docs.lmcache.ai/getting_started/installation.html) on both Machine A and Machine B. For specific installation instructions, please refer to the official documentation of each repository.
### 1. Start Mooncake Master on Machine A
2. Start the Mooncake Master node on Machine A:
```bash
mooncake_master -port 50052 -max_threads 64 -metrics_port 9004 \
mooncake_master -v=1 \
--rpc_port=50051 \
--metrics_port=9003 \
--enable_http_metadata_server=true \
--http_metadata_server_host=0.0.0.0 \
--http_metadata_server_port=8080
```
3. Launch the Decoder instance on machine A
- Modify the vllm/examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh file.
### 2. Configure and Start the vLLM Decoder on Machine A
Modify the vLLM disaggregated prefill launcher to use a Mooncake-backed LMCache
config for the decoder.
The decoder should continue to use `LMCacheConnectorV1` with `kv_role` set to
`kv_consumer`.
```diff
diff --git a/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh b/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
index 831ef0bb5..a2ff0744c 100644
--- a/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
+++ b/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
@@
elif [[ $1 == "decoder" ]]; then
# Decoder listens on port 8200
- decode_config_file=$SCRIPT_DIR/configs/lmcache-decoder-config.yaml
+ decode_config_file=$SCRIPT_DIR/configs/mooncake-decoder-config.yaml
UCX_TLS=cuda_ipc,cuda_copy,tcp \
LMCACHE_CONFIG_FILE=$decode_config_file \
LMCACHE_USE_EXPERIMENTAL=True \
VLLM_ENABLE_V1_MULTIPROCESSING=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
CUDA_VISIBLE_DEVICES=1 \
```
- Add the `mooncake-decoder-config.yaml` file
Create `configs/mooncake-decoder-config.yaml`:
```yaml
chunk_size: 256
remote_url: "mooncakestore://{IP of Machine A}:50052/"
remote_url: "mooncakestore://{IP of Machine A}:50051/"
remote_serde: "naive"
local_cpu: False
max_local_cpu_size: 100
@ -47,38 +77,43 @@ extra_config:
local_hostname: "{IP of Machine A}"
metadata_server: "http://{IP of Machine A}:8080/metadata"
protocol: "rdma"
device_name: "mlx5_0" # Multiple RDMA devices can be specified as comma-separated list
master_server_address: "{IP of Machine A}:50052"
device_name: "{RDMA device}"
master_server_address: "{IP of Machine A}:50051"
global_segment_size: 32212254720 # 30GB
local_buffer_size: 1073741824 # 1GB
transfer_timeout: 1
save_chunk_meta: False
```
- Launch the Decoder instance using command
Launch the decoder:
```bash
bash disagg_vllm_launcher.sh decoder Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
```
4. Launch the Prefiller instance on machine B
- Modify the vllm/examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh file.
### 3. Configure and Start the vLLM Prefiller on Machine B
Modify the launcher to use a Mooncake-backed LMCache config for the prefiller.
The prefiller should continue to use `LMCacheConnectorV1` with `kv_role` set to
`kv_producer`.
```diff
diff --git a/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh b/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
index 831ef0bb5..9e5a3f044 100644
--- a/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
+++ b/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
@@ -18,12 +18,14 @@ fi
@@
if [[ $1 == "prefiller" ]]; then
# Prefiller listens on port 8100
- prefill_config_file=$SCRIPT_DIR/configs/lmcache-prefiller-config.yaml
+ prefill_config_file=$SCRIPT_DIR/configs/mooncake-prefiller-config.yaml
```
- Add the `mooncake-prefiller-config.yaml` file
Create `configs/mooncake-prefiller-config.yaml`:
```yaml
chunk_size: 256
remote_url: "mooncakestore://{IP of Machine A}:50052/"
remote_url: "mooncakestore://{IP of Machine A}:50051/"
remote_serde: "naive"
local_cpu: False
max_local_cpu_size: 100
@ -87,33 +122,70 @@ extra_config:
local_hostname: "{IP of Machine B}"
metadata_server: "http://{IP of Machine A}:8080/metadata"
protocol: "rdma"
device_name: "mlx5_0" # Multiple RDMA devices can be specified as comma-separated list
master_server_address: "{IP of Machine A}:50052"
device_name: "{RDMA device}"
master_server_address: "{IP of Machine A}:50051"
global_segment_size: 32212254720 # 30GB
local_buffer_size: 1073741824 # 1GB
transfer_timeout: 1
save_chunk_meta: False
```
- Launch the Prefiller instance using command
Launch the prefiller:
```bash
bash disagg_vllm_launcher.sh prefiller Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
```
5. Prepare the router `disagg_proxy_server`
### 4. Start the Disaggregated Proxy
We use the [disagg_proxy_server](https://github.com/LMCache/LMCache/blob/dev/examples/disagg_prefill/disagg_proxy_server.py) provided by LMCache. According to [LMCache/LMCache#1342](https://github.com/LMCache/LMCache/issues/1342), when using Mooncake Store as the backend, you need to comment out `wait_decode_kv_ready(req_id)` in the proxy code.
6. Launch the `disagg_proxy_server` using command
Use the LMCache [`disagg_proxy_server.py`](https://github.com/LMCache/LMCache/blob/dev/examples/disagg_prefill/disagg_proxy_server.py) to route requests between the prefiller and decoder. According to [LMCache/LMCache#1342](https://github.com/LMCache/LMCache/issues/1342), when using Mooncake Store as the backend, comment out the `wait_decode_kv_ready(...)` call in the proxy before starting it.
```bash
python3 disagg_proxy_server.py --host localhost --port 9000 --prefiller-host IP_of_Machine_B --prefiller-port 8100 --decoder-host IP_of_Machine_A --decoder-port 8200
python3 disagg_proxy_server.py \
--host 0.0.0.0 \
--port 9000 \
--prefiller-host {IP of Machine B} \
--prefiller-port 8100 \
--decoder-host {IP of Machine A} \
--decoder-port 8200
```
7. Now we can send the requests to the `disagg_proxy_server` to test PD disaggregated serving.
### 5. Send a Test Request
Send traffic to the proxy, not directly to either vLLM instance.
```bash
curl -N http://{Proxy IP}:9000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4",
"messages": [
{
"role": "user",
"content": "Explain how KV cache reuse helps long-context serving."
}
],
"max_tokens": 128,
"temperature": 0.7
}'
```
## Port and Configuration Checklist
When changing ports away from these defaults, update all dependent settings
together:
- Mooncake master `--rpc_port` must match `remote_url` and
`extra_config.master_server_address`.
- `extra_config.metadata_server` must point to the Mooncake HTTP metadata
endpoint when HTTP metadata is used.
- Decoder and prefiller `device_name`, `protocol`, `global_segment_size`, and
`local_buffer_size` should be set for the local hardware and workload.
- Proxy `--prefiller-port` and `--decoder-port` must match the two vLLM
instance ports.
## Additional Resources
* [Mooncake x LMCache: Unite to Pioneer KVCache-Centric LLM Serving System](../../../getting_started/examples/lmcache-integration.md)
* [Mooncake x LMCache: Unite to Pioneer KVCache-Centric LLM Serving System](../lmcache-integration.md)
* [Using Mooncake in LMCache](https://docs.lmcache.ai/kv_cache/storage_backends/mooncake.html)
* [Using LMCache in vLLM](https://github.com/vllm-project/vllm/tree/main/examples/others/lmcache)
* [Using LMCache in vLLM](https://github.com/vllm-project/vllm/tree/main/examples/others/lmcache)

193
docs/source/getting_started/examples/vllm-integration/vllmv1-lmcache-mp-integration.md Normal file
View File

@ -0,0 +1,193 @@
# vLLM V1 Disaggregated Serving with Mooncake Store and LMCache [MP]
## Overview
This guide shows a single-machine 1-prefill/1-decode deployment using vLLM V1,
LMCache's multiprocess server, and Mooncake Store as the LMCache L2 backend.
LMCache supports both non-MP mode and MP mode with Mooncake Store. This page
covers the MP path, where vLLM instances connect to an LMCache server through
`LMCacheMPConnector`, and the LMCache server connects to Mooncake Store through
the `mooncake_store` L2 adapter. For the non-MP `LMCacheConnectorV1` path, see
[vLLM V1 Disaggregated Serving with Mooncake Store and LMCache](vllmv1-lmcache-integration.md).
In this setup, one machine runs Mooncake master, one LMCache MP server, the
disaggregated proxy, the prefiller vLLM instance, and the decoder vLLM
instance. The prefiller and decoder should use different GPUs.
This example uses `"metadata_server":"P2PHANDSHAKE"` for Mooncake transfer
metadata, so the Mooncake HTTP metadata server is not needed. If you switch to
HTTP metadata, remember that Mooncake's HTTP metadata server also defaults to
`8080`, which conflicts with LMCache's HTTP API on a single host.
## Prerequisites
Install Mooncake, vLLM, and LMCache on the machine. The example assumes an RDMA
deployment and uses:
- local host address: `{IP of Machine}`
- RDMA device: `{RDMA device}`
- LMCache checkout path: `/path/to/LMCache`
Replace these values with the local hostname/IP, RDMA device, and LMCache
checkout path for your environment.
LMCache requirement: This example requires LMCache v0.4.5 or later. LMCache
must also be built from source with Mooncake support enabled, because the
`mooncake_store` MP L2 adapter depends on the optional
`lmcache.lmcache_mooncake` C++ extension.
The standard prebuilt LMCache wheels currently include the Python Mooncake
adapter files, but do not include the optional `lmcache.lmcache_mooncake`
native extension.
```bash
BUILD_MOONCAKE=1 \
MOONCAKE_INCLUDE_DIR=/path/to/mooncake/include \
MOONCAKE_LIB_DIR=/path/to/mooncake/lib \
pip install -e /path/to/LMCache --verbose
```
## Deployment
### 1. Start Mooncake Master
```bash
mooncake_master -v=1 \
--rpc_port=50051 \
--metrics_port=9003
```
### 2. Start the LMCache Multiprocess Server
Start one LMCache MP server and configure Mooncake Store as the L2 adapter.
```bash
lmcache server \
--host 127.0.0.1 \
--port 5555 \
--http-host 127.0.0.1 \
--http-port 8080 \
--l1-size-gb 32 \
--eviction-policy LRU \
--no-l1-use-lazy \
--l2-adapter '{
"type": "mooncake_store",
"local_hostname": "{IP of Machine}",
"metadata_server": "P2PHANDSHAKE",
"protocol": "rdma",
"rdma_devices": "{RDMA device}",
"global_segment_size": 32212254720,
"local_buffer_size": 1073741824,
"master_server_addr": "127.0.0.1:50051"
}'
```
### 3. Start the Disaggregated Proxy
The proxy receives client requests, sends prefill requests to the prefiller,
sends decode requests to the decoder, and receives LMCache request telemetry
from the prefiller.
```bash
python /path/to/LMCache/examples/disagg_prefill_mp/disagg_proxy_server.py \
--host 127.0.0.1 \
--port 8000 \
--prefiller-host 127.0.0.1 \
--prefiller-port 8100 \
--decoder-host 127.0.0.1 \
--decoder-port 8200 \
--telemetry-port 5768
```
### 4. Start the vLLM Prefiller
The prefiller reports request telemetry back to the proxy so the proxy knows
when KV cache storage has completed.
```bash
CUDA_VISIBLE_DEVICES=0 \
LMCACHE_REQUEST_TELEMETRY_TYPE=fastapi \
LMCACHE_REQUEST_TELEMETRY_ENDPOINT=http://127.0.0.1:5768/api/v1/telemetry \
vllm serve Qwen/Qwen3-4B \
--host 127.0.0.1 \
--port 8100 \
--gpu-memory-utilization 0.8 \
--no-enable-log-requests \
--no-enable-prefix-caching \
--kv-transfer-config '{
"kv_connector": "LMCacheMPConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"lmcache.mp.host": "tcp://127.0.0.1",
"lmcache.mp.port": 5555
}
}'
```
### 5. Start the vLLM Decoder
The decoder connects to the same local LMCache MP server. It does not need
request telemetry environment variables; only the prefiller reports the "KV
cache is stored" event back to the proxy.
```bash
CUDA_VISIBLE_DEVICES=1 \
vllm serve Qwen/Qwen3-4B \
--host 127.0.0.1 \
--port 8200 \
--gpu-memory-utilization 0.8 \
--no-enable-log-requests \
--no-enable-prefix-caching \
--kv-transfer-config '{
"kv_connector": "LMCacheMPConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"lmcache.mp.host": "tcp://127.0.0.1",
"lmcache.mp.port": 5555
}
}'
```
### 6. Send a Test Request
Send traffic to the proxy, not directly to either vLLM instance.
```bash
curl -N http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-4B",
"messages": [
{
"role": "user",
"content": "Explain how KV cache reuse helps long-context serving."
}
],
"max_tokens": 128,
"temperature": 0.7
}'
```
## Port and Configuration Checklist
When changing ports away from these defaults, update all dependent settings
together:
- Mooncake master `--rpc_port` must match LMCache `master_server_addr`.
- The vLLM prefiller and decoder should connect to the local LMCache MP server
via `kv_connector_extra_config.lmcache.mp.host` and
`kv_connector_extra_config.lmcache.mp.port`.
- Proxy `--prefiller-port` and `--decoder-port` must match the two vLLM
`--port` values.
- `LMCACHE_REQUEST_TELEMETRY_ENDPOINT` on the prefiller must point to the proxy
telemetry endpoint.
- If `metadata_server` is changed from `P2PHANDSHAKE` to an HTTP metadata URL,
enable Mooncake HTTP metadata server and make sure its port does not conflict
with LMCache `--http-port`.
## Additional Resources
* [Mooncake x LMCache: Unite to Pioneer KVCache-Centric LLM Serving System](../lmcache-integration.md)
* [LMCache MP `mooncake_store` L2 adapter](https://docs.lmcache.ai/mp/l2_storage.html#mooncake-store-mooncake-store-native-connector)
* [LMCache multiprocess disaggregated prefill example](https://github.com/LMCache/LMCache/tree/dev/examples/disagg_prefill_mp)

167
docs/source/getting_started/observability.md Normal file
View File

@ -0,0 +1,167 @@
# Observability
This document describes how to monitor and observe a running Mooncake Store deployment.
## Master Metrics Log
When started, the Mooncake master periodically prints a metrics summary log every 10 seconds (configurable via `kMetricReportIntervalSeconds`). This log provides a comprehensive snapshot of the master's runtime state.
### Log Format
```
I0512 15:03:30.321475 239489 rpc_service.cpp:269] Master Admin Metrics: role=leader, state=serving, service_ready=true, master={...}, ha={...}, leader=127.0.0.1:50051, view_version=1
```
Each log line contains:
- **GLog header**: timestamp, thread ID, source file and line number
- **role**: HA role — `leader` or `standby`
- **state**: HA runtime state — `serving`, `starting`, `stopping`, etc.
- **service_ready**: whether the gRPC service is accepting requests
- **master**: master metrics block (see below)
- **ha**: HA metrics block
- **leader**: (only when available) the current leader address and view version
### Master Metrics Block
A typical `master={...}` block looks like this:
```
Mem Storage: 94.09 MB / 100.00 MB (94.1%) | SSD Storage: 0 B / 0 B | Keys: 16058 (soft-pinned: 0) | Clients: 1 | Requests (Success/Total per sec): PutStart=0.00/0.00, PutEnd=0.00/0.00, PutRevoke=0.00/0.00, Get=0.00/0.00, Exist=0.00/0.00, Del=0.00/0.00, DelAll=0.00/0.00, Ping=1.00/1.00, CopyStart=0.00/0.00, CopyEnd=0.00/0.00, CopyRevoke=0.00/0.00, MoveStart=0.00/0.00, MoveEnd=0.00/0.00, MoveRevoke=0.00/0.00, EvictDiskReplica=0.00/0.00 | Batch Requests (per sec, Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0.00/0.00/0.00, Item=0.00/0.00), PutEnd:(Req=0.00/0.00/0.00, Item=0.00/0.00), PutRevoke:(Req=0.00/0.00/0.00, Item=0.00/0.00), Get:(Req=0.00/0.00/0.00, Item=0.00/0.00), ExistKey:(Req=0.00/0.00/0.00, Item=0.00/0.00), QueryIp:(Req=0.00/0.00/0.00, Item=0.00/0.00), Clear:(Req=0.00/0.00/0.00, Item=0.00/0.00), CreateMoveTask:(Req=0.00/0.00), CreateCopyTask:(Req=0.00/0.00), QueryTask:(Req=0.00/0.00), FetchTasks:(Req=0.00/0.00), MarkTaskToComplete:(Req=0.00/0.00) | Eviction: Success/Attempts=0/0, AllocFail=0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B | Snapshots: Success=0, Fail=0
```
Request counters are reported as **rates per second** over the time window between two consecutive log outputs (10 seconds by default). Real-time state values (storage, key count, client count, discard staging size) are not rate-limited and reflect the current value at log time.
The metrics block consists of the following sections:
#### Storage
| Field | Description |
|-------|-------------|
| `Mem Storage` | Current memory usage / total memory capacity, with percentage |
| `SSD Storage` | Current SSD-backed storage usage / total SSD capacity |
#### Keys and Clients
| Field | Description |
|-------|-------------|
| `Keys` | Total number of keys managed by the master |
| `soft-pinned` | Number of keys with active soft-pin leases (protected from eviction) |
| `Clients` | Number of currently connected clients |
#### Requests (Success/Total per sec)
Rate counters for individual (non-batch) RPC requests over the last time window. Each shows `<success_rate>/<total_rate>` in requests per second:
| Counter | Description |
|---------|-------------|
| `PutStart` | Put object allocation requests |
| `PutEnd` | Put object commit requests |
| `PutRevoke` | Put object cancellation requests |
| `Get` | Get replica list requests |
| `Exist` | Key existence check requests |
| `Del` | Single key deletion requests |
| `DelAll` | Delete-all objects requests |
| `Ping` | Client heartbeat/ping requests |
| `CopyStart` | Copy object allocation requests |
| `CopyEnd` | Copy object commit requests |
| `CopyRevoke` | Copy object cancellation requests |
| `MoveStart` | Move object allocation requests |
| `MoveEnd` | Move object commit requests |
| `MoveRevoke` | Move object cancellation requests |
| `EvictDiskReplica` | Evict disk replica requests |
#### Batch Requests (per sec)
Batch operations aggregate multiple items into a single RPC. Rates are per second over the last time window. Format: `Req=<success>/<partial_success>/<total>`, `Item=<success_items>/<total_items>`:
| Counter | Description |
|---------|-------------|
| `PutStart` | Batch put object allocation requests |
| `PutEnd` | Batch put object commit requests |
| `PutRevoke` | Batch put object cancellation requests |
| `Get` | Batch get replica list requests |
| `ExistKey` | Batch key existence check requests |
| `QueryIp` | Batch query IP requests |
| `Clear` | Batch replica clear requests |
A request is considered "partial success" when it succeeds for some items but not all.
#### Task Operations
| Counter | Description |
|---------|-------------|
| `CreateMoveTask` | Move task creation requests |
| `CreateCopyTask` | Copy task creation requests |
| `QueryTask` | Task status query requests |
| `FetchTasks` | Pending task fetch requests (polled by store clients) |
| `MarkTaskToComplete` | Task completion acknowledgement requests |
#### Eviction & Discard
Eviction counters are **deltas** between two consecutive log outputs — they show what happened in the time window, not cumulative totals.
| Field | Description |
|-------|-------------|
| `Eviction: Success/Attempts` | Eviction rounds that succeeded at least partially vs. total attempts in this window |
| `AllocFail` | Number of PutStart/UpsertStart failures caused by replica allocation failure (triggers eviction) in this window |
| `keys` | Number of keys evicted in this window |
| `size` | Total size of evicted data in this window |
| `Discard: Released/Total` | Released (cleaned up) vs. total discarded PutStart staging replicas (live values) |
| `StagingSize` | Current size of discarded but not-yet-released staging buffers (live value) |
## Prometheus Metrics Endpoint
Mooncake master exposes Prometheus-format metrics at the HTTP admin endpoint. This allows integration with Prometheus, Grafana, or any Prometheus-compatible monitoring stack.
### Endpoints
The admin HTTP server runs on `metrics_port` (default: **9003**) and exposes the following endpoints:
| Endpoint | Content-Type | Description |
|----------|-------------|-------------|
| `GET /metrics` | `text/plain; version=0.0.4` | All metrics in Prometheus exposition format |
| `GET /metrics/summary` | `text/plain; version=0.0.4` | Human-readable summary (same content as the periodic log) |
| `GET /health` | `application/json` | Health check with role, HA state, and service readiness |
| `GET /role` | `text/plain` | Current HA role (`leader` / `standby`) |
| `GET /ha_status` | `text/plain` | Current HA runtime state (`serving` / `starting` / etc.) |
### Usage
**Scrape the /metrics endpoint with Prometheus:**
Add a scrape config to your `prometheus.yml`:
```yaml
scrape_configs:
- job_name: 'mooncake-master'
static_configs:
- targets: ['<master-host>:9003']
metrics_path: '/metrics'
```
**Quick check with curl:**
```bash
# Get Prometheus metrics
curl http://<master-host>:9003/metrics
# Get human-readable summary
curl http://<master-host>:9003/metrics/summary
# Check health
curl http://<master-host>:9003/health
```
### Configuration
The admin HTTP server is configured in the master config file (`master.json` or `master.yaml`):
```text
{
"enable_metric_reporting": true,
"metrics_port": 9003,
...
}
```
Set `enable_metric_reporting` to `false` to disable the periodic metrics log. HTTP endpoints (`/metrics`, `/health`, etc.) remain available regardless of this setting.

16
docs/source/getting_started/plugin-usage/3FS-USRBIO-Plugin.md
View File

@ -1,4 +1,8 @@
# Mooncake HF3FS Plugin
# Mooncake HF3FS Plugin (Experimental)
```{warning}
**Experimental / incomplete.** The HF3FS (3FS USRBIO) integration is under development and is not yet considered production-ready. Behavior, build flags, and configuration may change without notice. Use only for evaluation and testing.
```
This plugin implements 3FS native API (USRBIO) as a high-performance storage backend for Mooncake.
@ -6,9 +10,9 @@ This plugin implements 3FS native API (USRBIO) as a high-performance storage bac
### 1. 3FS Installation
- Build and install [3FS](https://github.com/deepseek-ai/3FS/)
- Required library: `libhf3fs_api_shared.so` (Default location: `3FS_PATH/build/src/lib/api`)
- Required library: `libhf3fs_api_shared.so` (Default location: `3FS_PATH/build/src/lib/api`)
→ Install to: `/usr/lib/`
- Required header: `hf3fs_usrbio.h` (Default location: `3FS_PATH/src/lib/api`)
- Required header: `hf3fs_usrbio.h` (Default location: `3FS_PATH/src/lib/api`)
→ Install to: `/usr/include/`
### 2. Mooncake Configuration
@ -27,10 +31,10 @@ Start master server and specify the 3FS mount point:
```bash
./build/mooncake-store/src/mooncake_master \
--root_fs_dir=/path/to/3fs_mount_point
--root_fs_dir=/path/to/3fs_mount_point
```
### Important Notes
1. The specified directory **must** be a 3FS mount point
1. The specified directory **must** be a 3FS mount point
- If not, the system will automatically fall back to POSIX API
2. For optimal performance:
- Ensure proper permissions on the 3FS mount point
@ -40,4 +44,4 @@ Start master server and specify the 3FS mount point:
```bash
ROLE=prefill MOONCAKE_STORAGE_ROOT_DIR=/mnt/3fs python3 ./stress_cluster_benchmark.py
```
```

Some files were not shown because too many files have changed in this diff Show More