mooncake-track
/
Mooncake
0
0
Fork 12
Code Issues Pull Requests 13 Actions Packages Projects Releases Wiki Activity

[Doc] Merge doc to docs (#1153) 

* merge doc/en/ to docs/

* move chinese documents to an archive directory
This commit is contained in:
Jianxin 2025-12-15 12:08:49 +08:00 committed by GitHub
parent 771510fb3a
commit df201882b2
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
45 changed files with 92 additions and 2476 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
README.md
View File

@ -32,11 +32,11 @@ This repository also hosts its technical report and the open sourced traces.
- **Sept 10, 2025**: SGLang officially supports Mooncake Store as a [hierarchical KV caching storage backend](https://lmsys.org/blog/2025-09-10-sglang-hicache/). The integration extends RadixAttention with multi-tier KV cache storage across device, host, and remote storage layers.
- **Sept 10, 2025**: The official & high-performance version of Mooncake P2P Store is open-sourced as [checkpoint-engine](https://github.com/MoonshotAI/checkpoint-engine/). It has been successfully applied in K1.5 and K2 production training, updating Kimi-K2 model (1T parameters) across thousands of GPUs in ~20s.
- **Aug 23, 2025**: [xLLM](https://github.com/jd-opensource/xllm) high-performance inference engine builds hybrid KV cache management based on Mooncake, supporting global KV cache management with intelligent offloading and prefetching.
- **Aug 18, 2025**: vLLM-Ascend [integrates Mooncake Transfer Engine](https://github.com/vllm-project/vllm-ascend/blob/main/examples/disaggregated_prefill_v1/mooncake_connector_deployment_guide.md) for KV cache register and disaggregate prefill, enabling efficient distributed inference on Ascend NPUs.
- **Aug 18, 2025**: vLLM-Ascend [integrates Mooncake Transfer Engine](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/multi_node_pd_disaggregation_mooncake.html) for KV cache register and disaggregate prefill, enabling efficient distributed inference on Ascend NPUs.
- **Jul 20, 2025**: Mooncake powers [the deployment of Kimi K2](https://lmsys.org/blog/2025-07-20-k2-large-scale-ep/) on 128 H200 GPUs with PD disaggregation and large-scale expert parallelism, achieving 224k tokens/sec prefill throughput and 288k tokens/sec decode throughput.
- **Jun 20, 2025**: Mooncake becomes a PD disaggregation [backend](https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/lmdeploy-integration-v0.9.md) for LMDeploy.
- **Jun 20, 2025**: Mooncake becomes a PD disaggregation [backend](https://kvcache-ai.github.io/Mooncake/getting_started/examples/lmdeploy-integration-v0.9.html) for LMDeploy.
- **May 9, 2025**: NIXL officially supports Mooncake Transfer Engine as [a backend plugin](https://github.com/ai-dynamo/nixl/blob/main/src/plugins/mooncake/README.md).
- **May 8, 2025**: Mooncake x LMCache <a href="doc/en/lmcache-integration.md" target="_blank">unite</a> to pioneer KVCache-centric LLM serving system.
- **May 8, 2025**: [Mooncake x LMCache](https://kvcache-ai.github.io/Mooncake/getting_started/examples/lmcache-integration.html) unite to pioneer KVCache-centric LLM serving system.
- **May 5, 2025**: Supported by Mooncake Team, SGLang release <a href="https://lmsys.org/blog/2025-05-05-large-scale-ep/" target="_blank">guidance</a> to deploy DeepSeek with PD Disaggregation on 96 H100 GPUs.
- **Apr 22, 2025**: LMCache officially supports Mooncake Store as a <a href="https://blog.lmcache.ai/2025-04-22-tencent/" target="_blank">remote connector</a>.
- **Apr 10, 2025**: SGLang officially supports Mooncake Transfer Engine for disaggregated prefilling and KV cache transfer.
@ -78,7 +78,7 @@ Mooncake adds elasticity and fault tolerance support for MoE model inference, en
<h2 id="show-cases">🔥 Show Cases</h2>
### Use Transfer Engine Standalone ([Guide](doc/en/transfer-engine.md))
### Use Transfer Engine Standalone ([Guide](https://kvcache-ai.github.io/Mooncake/design/transfer-engine/index.html))
Transfer Engine is a high-performance data transfer framework. Transfer Engine provides a unified interface to transfer data from DRAM, VRAM or NVMe, while the technical details related to hardware are hidden. Transfer Engine supports TCP, RDMA (InfiniBand/RoCEv2/eRDMA/NVIDIA GPUDirect) and NVMe over Fabric (NVMe-of) protocols.
@ -95,7 +95,7 @@ With 40 GB of data (equivalent to the size of the KVCache generated by 128k toke
<!-- ![transfer-engine-performance.png](image/transfer-engine-performance.png) -->
<img src=image/transfer-engine-performance.png width=75% />
### P2P Store ([Guide](doc/en/p2p-store.md))
### P2P Store ([Guide](https://kvcache-ai.github.io/Mooncake/design/p2p-store.html))
P2P Store is built on the Transfer Engine and supports sharing temporary objects between peer nodes in a cluster. P2P Store is ideal for scenarios like checkpoint transfer, where data needs to be rapidly and efficiently shared across a cluster.
**P2P Store has been used in the checkpoint transfer service of Moonshot AI.**
@ -109,8 +109,8 @@ Thanks to the high performance of Transfer Engine, P2P Stores can also distribut
<!-- ![p2p-store.gif](image/p2p-store.gif) -->
### Mooncake Store ([Guide](doc/en/mooncake-store.md))
Mooncake Store is a distributed KVCache storage engine specialized for LLM inference based on Transfer Engine. It is the central component of the KVCache-centric disaggregated architecture. The goal of Mooncake Store is to store the reusable KV caches across various locations in an inference cluster. Mooncake Store has been supported in [SGLang's Hierarchical KV Caching](https://lmsys.org/blog/2025-09-10-sglang-hicache/), [vLLM's prefill serving](https://docs.vllm.ai/en/latest/features/disagg_prefill.html) and is now integrated with [LMCache](doc/en/lmcache-integration.md) to provide enhanced KVCache management capabilities.
### Mooncake Store ([Guide](https://kvcache-ai.github.io/Mooncake/design/mooncake-store.html))
Mooncake Store is a distributed KVCache storage engine specialized for LLM inference based on Transfer Engine. It is the central component of the KVCache-centric disaggregated architecture. The goal of Mooncake Store is to store the reusable KV caches across various locations in an inference cluster. Mooncake Store has been supported in [SGLang's Hierarchical KV Caching](https://lmsys.org/blog/2025-09-10-sglang-hicache/), [vLLM's prefill serving](https://docs.vllm.ai/en/latest/features/disagg_prefill.html) and is now integrated with [LMCache](https://kvcache-ai.github.io/Mooncake/getting_started/examples/lmcache-integration.html) to provide enhanced KVCache management capabilities.
#### Highlights
- **Multi-replica support**: Mooncake Store supports storing multiple data replicas for the same object, effectively alleviating hotspots in access pressure.
@ -129,14 +129,14 @@ SGLang officially supports Mooncake Store as a [HiCache storage backend](https:/
- **Significant Performance Gains**: The multi-turn benchmark demonstrates substantial performance improvements over the non-HiCache setting. See our [benchmark report](https://kvcache-ai.github.io/Mooncake/performance/sglang-hicache-benchmark-results-v1.html) for more details.
- **Community Feedback**: Effective KV caching significantly reduces TTFT by eliminating redundant and costly re-computation. Integrating SGLang HiCache with the Mooncake service enables scalable KV cache retention and high-performance access. In our evaluation, we tested the DeepSeek-R1-671B model under PD-disaggregated deployment using in-house online requests sampled from a general QA scenario. On average, cache hits achieved an 84% reduction in TTFT compared to full re-computation. – Ant Group
### vLLM Integration ([Guide v0.2](doc/en/vllm-integration-v0.2.md))
### vLLM Integration ([Guide v0.2](https://kvcache-ai.github.io/Mooncake/getting_started/examples/vllm-integration/vllm-integration-v0.2.html))
To optimize LLM inference, the vLLM community is working on supporting [disaggregated prefilling (PR 10502)](https://github.com/vllm-project/vllm/pull/10502). This feature allows separating the **prefill** phase from the **decode** phase in different processes. The vLLM uses `nccl` and `gloo` as the transport layer by default, but currently it cannot efficiently decouple both phases in different machines.
We have implemented vLLM integration, which uses Transfer Engine as the network layer instead of `nccl` and `gloo`, to support **inter-node KVCache transfer** [(PR 10884)](https://github.com/vllm-project/vllm/pull/10884). Transfer Engine provides simpler interfaces and more efficient use of RDMA devices.
We will soon release the new vLLM integration based on Mooncake Store, which supports xPyD prefill/decode disaggregation.
**_Update[Dec 16, 2024]: Here is the latest vLLM Integration ([Guide v0.2](doc/en/vllm-integration-v0.2.md)) that is based on vLLM's main branch._**
**_Update[Dec 16, 2024]: Here is the latest vLLM Integration ([Guide v0.2](https://kvcache-ai.github.io/Mooncake/getting_started/examples/vllm-integration/vllm-integration-v0.2.html)) that is based on vLLM's main branch._**
#### Performance
By supporting Topology Aware Path Selection and multi-card bandwidth aggregation, Mean TTFT of vLLM with Transfer Engine is up to 25% lower than traditional TCP-based transports.
@ -147,7 +147,7 @@ In the future, we will further improve TTFT through GPUDirect RDMA and zero-copy
| Transfer Engine (RDMA) | 12.06 | 2042.74 | 1056.76 | 635.00 | 4006.59 |
| TCP | 12.05 | 2041.13 | 1414.05 | 766.23 | 6035.36 |
- Click [here](doc/en/vllm-benchmark-results-v0.2.md) to access detailed benchmark results.
- Click [here](https://kvcache-ai.github.io/Mooncake/performance/vllm-benchmark-results-v0.2.html) to access detailed benchmark results.
**More advanced features will coming soon, so stay tuned!**
@ -181,7 +181,7 @@ pip install mooncake-transfer-engine-non-cuda
> - If users encounter problems such as missing `lib*.so`, they should uninstall the package they installed and build the binaries manually.
### Use Docker image
Mooncake supports Docker-based deployment, see [Build Guide](doc/en/build.md) in detail.
Mooncake supports Docker-based deployment, see [Build Guide](https://kvcache-ai.github.io/Mooncake/getting_started/build.html) in detail.
### Build and use binaries
The following are additional dependencies for building Mooncake:

20
doc/en/architecture.md
View File

@ -1,20 +0,0 @@
# Mooncake Architecture
Mooncake aims to enhance the inference efficiency of large language models (LLMs), especially in slow object storage environments, by constructing a multi-level caching pool on high-speed interconnected DRAM/SSD resources. Compared to traditional caching system, Mooncake utilizes (GPUDirect) RDMA technology to transfer data directly from the initiator's DRAM/VRAM to the target's DRAM/VRAM in a zero-copy manner, while maximizing the use of multi-NIC resources on a single machine.
Mooncake:
- provides object-level data storage services
- supports data replication in the cache layer with slice-level placement guarantees and best-effort allocation, with a lightweight design due to not guaranteeing high availability
- ensures the atomicity of object write operations, meaning a `Get` operation will always read one consistent version, but not necessarily the latest one
- supports striping and parallel I/O transfer for larger objects to utilize the aggregated bandwidth of multiple network cards
- supports multiple modes for flushing slow object storage
- supports dynamic addition and removal of cache resources
## Architectural Overview
![architecture](../../image/mooncake-store.png)
- Mooncake provides object-level operations, i.e. `Get/Put/List/Del`, and also supports dynamically configurating replication strategies (`Replicate` operations);
- Mooncake supports zero-copy and multi-NIC data transfer over VRAM/DRAM/NVMe SSD. This feature is supported by Transfer Engine, which has been open-sourced;
- **The master node** centrally manages the mappings of objects to VRAM/DRAM/NVM buffers. The master node also drives **managed pool buffer nodes** to achieve data transfer by calling Transfer Engine's APIs;
- **Managed pool buffer nodes** mainly provide DRAM space for storing objects.
> Mooncake has open-sourced the Transfer Engine subsystem, and updates are forthcoming!

167
doc/en/build.md
View File

@ -1,167 +0,0 @@
# Build Guide
This document describes how to build Mooncake from source.
- Note:
- we have prebuild the PyPI wheel for Ubuntu 22.04/24.04, you can simply install mooncake with pip/pip3.
```bash
pip3 install mooncake-transfer-engine --upgrade
```
- Starting from version 0.3.7, the mooncake-transfer-engine wheel package from PyPI requires that CUDA be installed in the user environment. If your environment reports the following error:
```bash
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory
```
Please switch to the following PyPI wheel:
```bash
pip install mooncake-transfer-engine-non-cuda
```
Or you can include `-DUSE_CUDA=OFF` and compile from the source code.
## Automatic
### Recommended Version
- OS: Ubuntu 22.04 LTS+
- cmake: 3.20.x
- gcc: 9.4+
### Steps
1. Install dependencies, stable Internet connection is required:
```bash
bash dependencies.sh
```
2. In the root directory of this project, run the following commands:
```bash
mkdir build
cd build
cmake ..
make -j
```
3. Install Mooncake python package and mooncake_master executable
```bash
sudo make install
```
## Manual
### Recommended Version
- cmake: 3.22.x
- boost-devel: 1.66.x
- googletest: 1.12.x
- gcc: 10.2.1
- go: 1.22+
- hiredis
- curl
### Steps
1. Install dependencies from system software repository:
```bash
# For debian/ubuntu
apt-get install -y build-essential \
cmake \
libibverbs-dev \
libgoogle-glog-dev \
libgtest-dev \
libjsoncpp-dev \
libnuma-dev \
libunwind-dev \
libpython3-dev \
libboost-all-dev \
libssl-dev \
pybind11-dev \
libcurl4-openssl-dev \
libhiredis-dev \
pkg-config \
patchelf
# For centos/alibaba linux os
yum install cmake \
gflags-devel \
glog-devel \
libibverbs-devel \
numactl-devel \
gtest \
gtest-devel \
boost-devel \
openssl-devel \
hiredis-devel \
libcurl-devel
```
NOTE: You may need to install gtest, glog, gflags from source code:
```bash
git clone https://github.com/gflags/gflags
git clone https://github.com/google/glog
git clone https://github.com/abseil/googletest.git
```
2. If you want to compile the Nvidia GPUDirect support module, first follow the instructions in https://docs.nvidia.com/cuda/cuda-installation-guide-linux/ to install CUDA (ensure to enable `nvidia-fs` for proper `cuFile` module compilation). After that:
1) Follow Section 3.7 in https://docs.nvidia.com/cuda/gpudirect-rdma/ to install `nvidia-peermem` for enabling GPU-Direct RDMA
2) Configure `LIBRARY_PATH` and `LD_LIBRARY_PATH` to ensure linking of `cuFile`, `cudart`, and other libraries during compilation:
```bash
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/cuda/lib64
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
```
3. If you want to compile the Moore Mthreads GPUDirect support module, first follow the instructions in https://docs.mthreads.com/musa-sdk/musa-sdk-doc-online/install_guide to install MUSA. After that:
1) Install `mthreads-peermem` for enabling GPU-Direct RDMA
2) Configure `LIBRARY_PATH` and `LD_LIBRARY_PATH` to ensure linking of `musart`, and other libraries during compilation:
```bash
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/musa/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/musa/lib
```
3. Install yalantinglibs
```bash
git clone https://github.com/alibaba/yalantinglibs.git
cd yalantinglibs
mkdir build && cd build
cmake .. -DBUILD_EXAMPLES=OFF -DBUILD_BENCHMARK=OFF -DBUILD_UNIT_TESTS=OFF
make -j$(nproc)
make install
```
4. In the root directory of this project, run the following commands:
```bash
mkdir build
cd build
cmake ..
make -j
```
5. Install Mooncake python package and mooncake_master executable
```bash
make install
```
## Use Mooncake in Docker Containers
Mooncake supports Docker-based deployment. What you need is to get Docker image by `docker pull alogfans/mooncake`.
For the the container to use the host's network resources, you need to add the `--device` option when starting the container. The following is an example.
```
# In host
sudo docker run --net=host --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm --ulimit memlock=-1 -t -i mooncake:v0.9.0 /bin/bash
# Run transfer engine in container
cd /Mooncake-main/build/mooncake-transfer-engine/example
./transfer_engine_bench --device_name=ibp6s0 --metadata_server=10.1.101.3:2379 --mode=target --local_server_name=10.1.100.3
```
## Advanced Compile Options
The following options can be used during `cmake ..` to specify whether to compile certain components of Mooncake.
- `-DUSE_CUDA=[ON|OFF]`: Enable GPU Direct RDMA and NVMe-of support
- `-DUSE_MUSA=[ON|OFF]`: Enable Moore Threads GPU support via MUSA
- `-DUSE_HIP=[ON|OFF]`: Enable AMD GPU support via HIP/ROCm
- `-DUSE_CXL=[ON|OFF]`: Enable CXL support
- `-DWITH_STORE=[ON|OFF]`: Build Mooncake Store component
- `-DWITH_P2P_STORE=[ON|OFF]`: Enable Golang support and build P2P Store component, require go 1.23+
- `-DWITH_WITH_RUST_EXAMPLE=[ON|OFF]`: Enable Rust support
- `-DUSE_REDIS=[ON|OFF]`: Enable Redis-based metadata service
- `-DUSE_HTTP=[ON|OFF]`: Enable Http-based metadata service
- `-DUSE_ETCD=[ON|OFF]`: Enable etcd-based metadata service, require go 1.23+
- `-DSTORE_USE_ETCD=[ON|OFF]`: Enable etcd-based failover for Mooncake Store, require go 1.23+. **Note:** `-DUSE_ETCD` and `-DSTORE_USE_ETCD` are two independent options. Enabling `-DSTORE_USE_ETCD` does **not** depend on `-DUSE_ETCD`
- `-DBUILD_SHARED_LIBS=[ON|OFF]`: Build Transfer Engine as shared library, default is OFF
- `-DBUILD_UNIT_TESTS=[ON|OFF]`: Build unit tests, default is ON
- `-DBUILD_EXAMPLES=[ON|OFF]`: Build examples, default is ON

65
doc/en/error-code.md
View File

@ -1,65 +0,0 @@
# Error Code Explanation
## Mooncake TransferEngine
TransferEngine may generate various types of errors during execution. For most APIs, the return value indicates the error reason. For details, refer to `mooncake-transfer-engine/include/error.h`.
| Group | Return Value | Description |
|------------|-----------------------------|-----------------------------------------------------------------------------------------------------------|
| Normal | 0 | Normal execution |
| Error Args | ERR_INVALID_ARGUMENT | Input parameters are incorrect (and cannot be detailed into other items in this group) |
| | ERR_TOO_MANY_REQUESTS | The number of requests passed by the user when calling the `SubmitTransfer` interface exceeds the maximum value specified by the allocated `BatchID` |
| | ERR_ADDRESS_NOT_REGISTERED | The source address and/or target address in the request initiated by the user are not registered (including the situation where it has been registered locally but not uploaded to the metadata server) |
| | ERR_BATCH_BUSY | Reclaim a `BatchID` that is currently executing a request |
| | ERR_DEVICE_NOT_FOUND | No available RDMA device to execute the user's request |
| | ERR_ADDRESS_OVERLAPPED | Registering overlapping memory areas multiple times |
| Handshake | ERR_DNS | Local server name is not a valid DNS hostname or IP address, preventing other nodes from handshaking with this node |
| | ERR_SOCKET | Errors related to TCP Socket during the handshake process |
| | ERR_MALFORMED_JSON | Data format error during handshake exchange |
| | ERR_REJECT_HANDSHAKE | Peer rejects the handshake due to peer's errors |
| Other | ERR_METADATA | Failed to communicate with metadata server |
| | ERR_ENDPOINT | Exceptions during the creation and use of `RdmaEndPoint` objects |
| | ERR_NUMA | The system does not support numa interface |
| | ERR_CLOCK | The system does not support `clock_gettime` interface |
| | ERR_MEMORY | Out of memory |
## Mooncake Store
Mooncake Store may generate various types of errors during execution. For most APIs, the return value indicates the error reason. For details, refer to `mooncake-store/include/types.h`.
| Group | Return Value | Description |
|--------------------------|--------------------------------|-----------------------------------------------------------------------------------------------------------|
| Normal | 0 | Operation successful |
| Internal | INTERNAL_ERROR (-1) | Internal error occurred |
| Buffer Allocation | BUFFER_OVERFLOW (-10) | Insufficient buffer space |
| Segment Selection | SHARD_INDEX_OUT_OF_RANGE (-100)| Shard index is out of bounds |
| | SEGMENT_NOT_FOUND (-101) | No available segments found |
| | SEGMENT_ALREADY_EXISTS (-102) | Segment already exists |
| Handle Selection | NO_AVAILABLE_HANDLE (-200) | Memory allocation failed due to insufficient space |
| Version | INVALID_VERSION (-300) | Invalid version |
| Key | INVALID_KEY (-400) | Invalid key |
| Engine | WRITE_FAIL (-500) | Write operation failed |
| Parameter | INVALID_PARAMS (-600) | Invalid parameters |
| Engine Operation | INVALID_WRITE (-700) | Invalid write operation |
| | INVALID_READ (-701) | Invalid read operation |
| | INVALID_REPLICA (-702) | Invalid replica operation |
| Object | REPLICA_IS_NOT_READY (-703) | Replica is not ready |
| | OBJECT_NOT_FOUND (-704) | Object not found |
| | OBJECT_ALREADY_EXISTS (-705) | Object already exists |
| | OBJECT_HAS_LEASE (-706) | Object has lease |
| | LEASE_EXPIRED (-707) | Lease expired before data transfer completed |
| Transfer | TRANSFER_FAIL (-800) | Transfer operation failed |
| RPC | RPC_FAIL (-900) | RPC operation failed |
| High Availability | ETCD_OPERATION_ERROR (-1000) | etcd operation failed |
| | ETCD_KEY_NOT_EXIST (-1001) | Key not found in etcd |
| | ETCD_TRANSACTION_FAIL (-1002) | etcd transaction failed |
| | ETCD_CTX_CANCELLED (-1003) | etcd context cancelled |
| | UNAVAILABLE_IN_CURRENT_STATUS (-1010) | Request cannot be done in current status |
| | UNAVAILABLE_IN_CURRENT_MODE (-1011) | Request cannot be done in current mode |
| File | FILE_NOT_FOUND (-1100) | File not found |
| | FILE_OPEN_FAIL (-1101) | Error opening file or writing to an existing file |
| | FILE_READ_FAIL (-1102) | Error reading file |
| | FILE_WRITE_FAIL (-1103) | Error writing file |
| | FILE_INVALID_BUFFER (-1104) | File buffer is wrong |
| | FILE_LOCK_FAIL (-1105) | File lock operation failed |
| | FILE_INVALID_HANDLE (-1106) | Invalid file handle |

59
doc/en/lmcache-integration.md
View File

@ -1,59 +0,0 @@
# Mooncake x LMCache: Unite to Pioneer KVCache-Centric LLM Serving System
Mooncake and LMCache have announced a strategic collaboration aimed at pioneering a KVCache-centric Large Language Model (LLM) serving system. This partnership seeks to significantly enhance the efficiency, scalability, and responsiveness of LLM applications.
By combining LMCache’s advanced KVCache management techniques with Mooncake’s powerful and optimized backend infrastructure, the collaboration aims to improve the capability of LLM serving systems to efficiently handle diverse workloads and meet demanding latency requirements.
## About Mooncake
Mooncake originated as the serving platform for Kimi, a prominent LLM chatbot service from Moonshot AI. It utilizes a KVCache-centric, disaggregated architecture that substantially boosts throughput by effectively separating Prefill and Decode operations and optimizing KVCache reuse. After integrating contributions from multiple infrastructure providers, Mooncake has become an open-source platform, integrated with popular systems such as vLLM (via LMCache) and SGLang, establishing itself as a critical component within the open-source LLM serving ecosystem.
## About LMCache
LMCache is an open-source Knowledge Delivery Network (KDN), specifically designed to accelerate LLM applications by up to eightfold while also substantially reducing costs. It achieves this by storing Key-Value (KV) caches of reusable texts across multiple storage layers, facilitating effective reuse of caches across various serving instances. This strategy significantly cuts down GPU resource usage and reduces response times, particularly in long-context applications.
## Collaboration Highlights
This strategic alliance delivers substantial mutual technological advancements:
* **LMCache Integrates Mooncake for Enhanced Performance**: LMCache leverages Mooncake’s advanced transfer engine and KVCache storage solutions, significantly enhancing data distribution efficiency and overall system performance.
* **Mooncake Integrates LMCache for Enhanced Functionality**: Mooncake now includes LMCache as its KVCache management layer, benefiting from LMCache’s flexible cache control mechanisms and future-oriented functionalities, such as CacheBlend. This integration enables Mooncake to offer more sophisticated features and improved operational flexibility.
## Performance Benchmarking and Results
To illustrate the benefits of this collaboration, a comprehensive performance evaluation of the integrated vLLM, LMCache, and Mooncake Store was conducted. The selected experimental conditions simulate realistic LLM deployment scenarios, specifically comparing the initial cold start with subsequent cache-hit performance to demonstrate the advantages of KVCache reuse.
### Experimental Setup
* **Hardware:** 8 × H800 GPUs
* **Model:** Qwen2.5-72B-Instruct (non-quantized)
* **Workload:** 50 requests, each with an input of 9,728 tokens and generating 64 tokens per request, without concurrency limits.
* **Cache Configuration:** LMCache’s local CPU cache was disabled to ensure a direct assessment of Mooncake Store’s effectiveness.
### Performance Metrics and Improvements
The test revealed significant performance enhancements under cache-hit conditions:
| Metric | Cold Start (First Round) | Cache Hit (Second Round) | **Improvement** |
|------------------------------|--------------------------|--------------------------|-----------------|
| Average TTFT | 21,707.62 ms | 6,708.39 ms | **↓ 69.1%** |
| P50 TTFT | 22,102.51 ms | 7,253.38 ms | **↓ 67.2%** |
| P90 TTFT | 38,170.54 ms | 11,128.26 ms | **↓ 70.9%** |
| Average TPOT | 368.12 ms | 140.17 ms | **↓ 61.9%** |
| P50 TPOT | 362.08 ms | 132.98 ms | **↓ 63.3%** |
| P90 TPOT | 632.90 ms | 221.93 ms | **↓ 64.9%** |
| Request Throughput (req/s) | 1.11 | 3.23 | **↑ 191.0%** |
| Output Token Throughput (tok/s)| 71.24 | 202.91 | **↑ 184.8%** |
| Total Token Throughput (tok/s)| 10,899.84 | 31,665.01 | **↑ 190.5%** |
These results clearly illustrate how the collaborative integration of LMCache and Mooncake **substantially improves latency, throughput, and overall system efficiency** through **KVCache reuse**.
## Future Developments
Moving forward, LMCache and Mooncake plan to collaborate closely on several key improvements highly valued by the community, including:
* Developing **optimized KVCache eviction and placement strategies** to maximize throughput.
* Implementing **asynchronous KVCache scheduling** and **zero-copy KVCache transferring mechanisms** to minimize costs through effective prefetching and offloading to cheaper storage tiers.
* **Expanding caching strategies beyond simple prefix matching**, enhancing KVCache reusability by supporting more flexible matching patterns.
This strategic partnership represents a significant advancement toward fully realizing the potential of next-generation LLM serving architectures.

134
doc/en/lmcacheV1-deployment.md
View File

@ -1,134 +0,0 @@
# Mooncake Store x LMCache for vLLM V1
The vLLM v1 version has been released with support for PD separation. The detailed design document can be found here: https://docs.google.com/document/d/1uPGdbEXksKXeN4Q9nUm9hzotqEjQhYmnpAhidLuAsjk. LMCache immediately implemented the corresponding connector to support storage, transmission, and loading of KVCache, enabling collaborative operation with PD nodes. Mooncake, as LMCache's backend storage engine, has undergone extensive optimizations in usability, performance, and stability. This document explains how to deploy a complete vLLM V1 PD separation instance using LMCache + Mooncake.
## Deployment
1. First, you need to prepare two GPU-equipped machines, which we will refer to as Machine A and Machine B. Install vLLM, LMCache, and Mooncake on both Machine A and Machine B. For specific installation instructions, please refer to the official documentation of each repository.
2. Start the Mooncake Master node on Machine A using the following command:
`cd Mooncake/build && ./mooncake-store/src/mooncake_master -v=1 -port=50052 -max_threads 64 -metrics_port 9004`
3. Start etcd on Machine A with the command:
`etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379`
4. Launch D endpoint on machine A
- Modify the vllm/examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh file.
```diff
diff --git a/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh b/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
index 831ef0bb5..a2ff0744c 100644
--- a/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
+++ b/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
@@ -24,6 +24,8 @@ if [[ $1 == "prefiller" ]]; then
LMCACHE_CONFIG_FILE=$prefill_config_file \
LMCACHE_USE_EXPERIMENTAL=True \
VLLM_ENABLE_V1_MULTIPROCESSING=1 \
+ VLLM_USE_MODELSCOPE=True \
+ MOONCAKE_CONFIG_PATH=./mooncake.json \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
CUDA_VISIBLE_DEVICES=0 \
vllm serve $MODEL \
@@ -36,11 +38,13 @@ if [[ $1 == "prefiller" ]]; then
elif [[ $1 == "decoder" ]]; then
# Decoder listens on port 8200
- decode_config_file=$SCRIPT_DIR/configs/lmcache-decoder-config.yaml
+ decode_config_file=$SCRIPT_DIR/decode.yaml
UCX_TLS=cuda_ipc,cuda_copy,tcp \
LMCACHE_CONFIG_FILE=$decode_config_file \
LMCACHE_USE_EXPERIMENTAL=True \
+ VLLM_USE_MODELSCOPE=True \
+ MOONCAKE_CONFIG_PATH=./mooncake.json \
VLLM_ENABLE_V1_MULTIPROCESSING=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
CUDA_VISIBLE_DEVICES=1 \
```
- Add `decode.yaml` and 'mooncake.json' file
```yaml
chunk_size: 256
local_device: "cpu"
remote_url: "mooncakestore://{IP of Machine A}:50052/"
remote_serde: "naive"
pipelined_backend: False
local_cpu: False
max_local_cpu_size: 100
```
```json
{
"local_hostname": "{IP of Machine A}",
"metadata_server": "etcd://{IP of Machine A}:2379",
"protocol": "rdma",
"device_name": "erdma_0, erdma_1",
"global_segment_size": 3355443200,
"local_buffer_size": 1073741824,
"master_server_address": "{IP of Machine A}:50052"
}
```
- Launch D endpoint using command `bash disagg_vllm_launcher.sh decoder Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4`
5. Launch P endpoint on machine B
- Modify the vllm/examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh file.
```diff
diff --git a/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh b/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
index 831ef0bb5..9e5a3f044 100644
--- a/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
+++ b/examples/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
@@ -18,12 +18,14 @@ fi
if [[ $1 == "prefiller" ]]; then
# Prefiller listens on port 8100
- prefill_config_file=$SCRIPT_DIR/configs/lmcache-prefiller-config.yaml
+ prefill_config_file=$SCRIPT_DIR/prefill.yaml
UCX_TLS=cuda_ipc,cuda_copy,tcp \
LMCACHE_CONFIG_FILE=$prefill_config_file \
LMCACHE_USE_EXPERIMENTAL=True \
VLLM_ENABLE_V1_MULTIPROCESSING=1 \
+ VLLM_USE_MODELSCOPE=True \
+ MOONCAKE_CONFIG_PATH=./mooncake.json \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
CUDA_VISIBLE_DEVICES=0 \
vllm serve $MODEL \
@@ -42,6 +44,8 @@ elif [[ $1 == "decoder" ]]; then
LMCACHE_CONFIG_FILE=$decode_config_file \
LMCACHE_USE_EXPERIMENTAL=True \
VLLM_ENABLE_V1_MULTIPROCESSING=1 \
+ VLLM_USE_MODELSCOPE=True \
+ MOONCAKE_CONFIG_PATH=./mooncake.json \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
CUDA_VISIBLE_DEVICES=1 \
vllm serve $MODEL \
```
- Add `prefill.yaml` and `mooncake.json` file
```yaml
chunk_size: 256
local_device: "cpu"
remote_url: "mooncakestore://{IP of Machine A}:50052/"
remote_serde: "naive"
pipelined_backend: False
local_cpu: False
max_local_cpu_size: 100
```
```json
{
"local_hostname": "{IP of Machine B}",
"metadata_server": "etcd://{IP of Machine A}:2379",
"protocol": "rdma",
"device_name": "erdma_0, erdma_1",
"global_segment_size": 3355443200,
"local_buffer_size": 1073741824,
"master_server_address": "{IP of Machine A}:50052"
}
```
- Launch P endpoint using command `bash disagg_vllm_launcher.sh prefiller Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4`
6. Launch the LoadBalance endpoint using command
```bash
python3 disagg_proxy_server.py --host localhost --port 9000 --prefiller-host IP_of_Machine_B --prefiller-port 8100 --decoder-host IP_of_Machine_A --decoder-port 8200
```
7. Now we can send the requests to LoadBalance to test PD separation.

167
doc/en/lmdeploy-integration-v0.9.md
View File

@ -1,167 +0,0 @@
# LMDeploy Disaggregated Serving with MooncakeTransferEngine
## Overview
This is the latest version of the MooncakeTransferEngine integration doc with the LMDeploy project based on [PR 3304](https://github.com/InternLM/lmdeploy/pull/3304#issue-2940383503) and [PR 3620](https://github.com/InternLM/lmdeploy/pull/3620) to support KVCache transfer for intra-node and inter-node disaggregated serving scenarios.
***Please note that this is still an experimental version and will be modified anytime based on feedback from the LMDeploy community.***
## Installation
### Prerequisites
```bash
pip install mooncake-transfer-engine
```
Note:
- If any `.so` file is missing, uninstall the pip package with `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually from source following the [build instructions](https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/build.md).
### Install the latest version of LMDeploy
##### 1. Clone LMDeploy from the official repo
```bash
git clone https://github.com/InternLM/lmdeploy.git
```
##### 2. Build
##### 2.1 Build from source
```bash
cd LMDeploy
pip install -e .
```
- If you encounter any problems that you cannot solve, please refer to the [LMDeploy official installation guide](https://lmdeploy.readthedocs.io/en/latest/get_started/installation.html).
------
## Configuration
### Running prefill and decode instances on different nodes
#### Proxy:
```bash
lmdeploy serve proxy \
--server-name <IP1> \
--server-port 8000 \
--routing-strategy "min_expected_latency" \
--serving-strategy DistServe \
--log-level INFO
```
- The `--server-name` and `--server-port` parameters specify the LMDeploy proxy service host and listening port.
- The `--routing-strategy` parameter determines how requests are routed to prefill/decode servers (choose from`min_expected_latency`, `min_observed_latency` and `random`).
- The `--serving-strategy` parameter sets the serving mode; `DistServe` enables disaggregated serving. Default mode of `Hybrid` will colocate prefill and decode.
- The `--log-level` parameter controls the verbosity of runtime logging.
#### Prefill:
```bash
lmdeploy serve api_server Qwen/Qwen3-8B \
--server-name <IP1> \
--server-port 23333 \
--role Prefill \
--proxy-url http://<IP1>:8000 \
--backend pytorch \
--migration-backend Mooncake
```
- The `--role` parameter sets the node role in the disaggregated system (`Prefill` for token embedding and KV cache generation).
- The `--proxy-url` parameter connects the worker instance back to the proxy for coordination.
- The `--backend` parameter specifies the model execution backend (e.g., `pytorch`, `turbomind`).
- The `--migration-backend` parameter defines the KV cache transport mechanism (e.g., `Mooncake` and `DlSllime`).
#### Decode:
```bash
lmdeploy serve api_server Qwen/Qwen3-8B \
--server-name <IP2> \
--server-port 23334 \
--role Decode \
--proxy-url http://<IP1>:8000 \
--backend pytorch \
--migration-backend Mooncake
```
#### Test Inference:
```bash
curl -X POST "http://<IP1>:8000/v1/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-8B",
"temperature": 0,
"prompt": "Shanghai is a city that ",
"max_tokens": 16,
"stream": false
}'
```
------
### Running prefill and decode Instances on the same nodes
#### Proxy:
```bash
lmdeploy serve proxy \
--server-name <IP> \
--server-port 8000 \
--routing-strategy "min_expected_latency" \
--serving-strategy DistServe \
--log-level INFO
```
#### Prefill:
```bash
CUDA_VISIBLE_DEVICES=0 \
lmdeploy serve api_server Qwen/Qwen3-8B \
--server-name <IP> \
--server-port 23333 \
--role Prefill \
--proxy-url http://<IP>:8000 \
--backend pytorch \
--migration-backend Mooncake
```
- The `CUDA_VISIBLE_DEVICES=0` specify the available GPU for prefill service since generally one GPU may not have enough VRAM to run both prefill and decode process.
#### Decode:
```bash
CUDA_VISIBLE_DEVICES=1 \
lmdeploy serve api_server Qwen/Qwen3-8B \
--server-name <IP> \
--server-port 23334 \
--role Decode \
--proxy-url http://<IP>:8000 \
--backend pytorch \
--migration-backend Mooncake
```
#### Test Inference:
```bash
curl -X POST "http://<IP>:8000/v1/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-8B",
"temperature": 0,
"prompt": "Shanghai is a city that ",
"max_tokens": 16,
"stream": false
}'
```
## Notes
- You can specify multiple prefill or decode instances with distinct `--server-port` and different GPUs using `CUDA_VISIBLE_DEVICES`.
- MooncakeTransferEngine supports both intra-node (PCIe) and inter-node (RDMA/CXL) transfer, and device selection is automatic or customizable via config.
- When using HF models that timeout during prefill, consider setting model path to `~/Qwen3-8B` to accelerate loading from localhost.
- Use `--log-level DEBUG` to get detailed runtime logs for troubleshooting.

99
doc/en/p2p-store.md
View File

@ -1,99 +0,0 @@
# P2P Store
## Overview
P2P Store is built on [Transfer Engine](transfer-engine.md) and supports the temporary sharing of objects between peer nodes in a cluster, with typical scenarios including checkpoint distribution, etc. P2P Store is a client-only architecture with no centralized master node; global metadata is maintained by the metadata service. P2P Store is now used in Moonshot AI's checkpoint transfer service.
P2P Store provides several interfaces like `Register` and `GetReplica`. A `Register` is equivalent to seeding in BitTorrent, where a local file is registered with the global metadata without any data transfer occurring; it merely registers metadata in etcd. A `GetReplica` searches metadata and clones data from other machines that have called Register or Get (unless explicitly calling `Unregister` or `DeleteReplica` to stop pulling files from the local machine), and it can also act as a data source to improve the efficiency of data transfer for other nodes. This approach can increase the efficiency of large-scale data distribution and avoid the outbound bandwidth saturation.
## P2P Store Sample Program
After compiling P2P Store successfully by following the compilation guide with `cmake .. -DWITH_P2P_STORE=ON && make -j`, a test program `p2p-store-example` will be produced in the `build/mooncake-p2p-store` directory. This tool demonstrates the usage of P2P Store, simulating the process of migrating model data from training nodes to a large number of inference nodes after the training task is completed. Currently, it only supports the RDMA protocol.
1. **Start the `etcd` service.** This is consistent with the method described in Transfer Engine Bench.
2. **Start the simulated training node.** This node will create a simulated model file and make it public within the cluster.
```bash
# This is 10.0.0.2
export MC_GID_INDEX=n # NOTE that n is integer
./p2p-store-example --cmd=trainer \
--metadata_server=10.0.0.1:2379 \
--local_server_name=10.0.0.2:12345 \
```
3. **Start the simulated inference node.** This node will pull data from the simulated training node or other simulated inference nodes.
```bash
# This is 10.0.0.3
export MC_GID_INDEX=n
./p2p-store-example --cmd=inferencer \
--metadata_server=10.0.0.1:2379 \
--local_server_name=10.0.0.3:12346 \
```
The test is completed with the display of "ALL DONE".
In the above process, the simulated inference nodes search for data sources, which is done by P2P Store, so there is no need for users to provide the IP address of the training node. Similarly, it is necessary to ensure that other nodes can access this machine using the local machine's hostname or the `--local_server_name` filled in during the creation of the node.
## P2P Store API
Mooncake P2P Store currently implements the following interfaces in Golang:
```go
func NewP2PStore(metadataUri string, localSegmentName string) (*P2PStore, error)
```
Creates an instance of `P2PStore`, which internally starts a Transfer Engine service.
- `metadataUri`: The hostname or IP address of the metadata server/etcd service.
- `localSegmentName`: The local server name (hostname/IP address:port), ensuring uniqueness within the cluster.
- Return value: If successful, returns a pointer to the `P2PStore` instance, otherwise returns `error`.
```go
func (store *P2PStore) Close() error
```
Closes the P2PStore instance.
```go
type Buffer struct {
addr uintptr
size uint64
}
func (store *P2PStore) Register(ctx context.Context, name string, addrList []uintptr, sizeList []uint64, maxShardSize uint64, location string) error
```
Registers a local file to the cluster, making it downloadable by other peers. Ensure that the data in the specified address range is not modified or unmapped before calling `Unregister`.
- `ctx`: Golang Context reference.
- `name`: The file registration name, ensuring uniqueness within the cluster.
- `addrList` and `sizeList`: These two arrays represent the memory range of the file, with `addrList` indicating the starting address and `sizeList` indicating the corresponding length. The file content corresponds logically to the order in the arrays.
- `maxShardSize`: The internal data sharding granularity, with a recommended value of 64MB.
- `location`: The device name corresponding to this memory segment.
```go
func (store *P2PStore) Unregister(ctx context.Context, name string) error
```
Remove the registration of a local file to the entire cluster. After calling this function, it is safe to modify/delete the memory region reserved for this file.
- `ctx`: Golang Context reference.
- `name`: The file registration name, ensuring uniqueness within the cluster.
```go
type PayloadInfo struct {
Name string // Full name of the Checkpoint file
MaxShardSize uint64 // The maxShardSize passed into Register
TotalSize uint64 // The total length of the sizeList passed into Register
SizeList []uint64 // The sizeList passed into Register
}
func (store *P2PStore) List(ctx context.Context, namePrefix string) ([]PayloadInfo, error)
```
Obtains a list of files registered in the cluster, with the ability to filter by file name prefix.
- `ctx`: Golang Context reference.
- `namePrefix`: The file name prefix; if empty, it indicates enumeration of all files.
```go
func (store *P2PStore) GetReplica(ctx context.Context, name string, addrList []uintptr, sizeList []uint64) error
```
Pulls a copy of a file to a specified local memory area, while allowing other nodes to pull the file from this copy. Ensure that the data in the corresponding address range is not modified or unmapped before calling `DeleteReplica`. A file can only be pulled once on the same P2PStore instance.
- `ctx`: Golang Context reference.
- `name`: The file registration name, ensuring uniqueness within the cluster.
- `addrList` and `sizeList`: These two arrays represent the memory range of the file, with `addrList` indicating the starting address and `sizeList` indicating the corresponding length. The file content corresponds logically to the order in the arrays.
```go
func (store *P2PStore) DeleteReplica(ctx context.Context, name string) error
```
Stops other nodes from pulling the file from the local node. After calling this function, it is safe to modify/delete the memory region reserved for this file.
- `ctx`: Golang Context reference.
- `name`: The file registration name, ensuring uniqueness within the cluster.

26
doc/en/sglang-benchmark-results-v1.md
View File

@ -1,26 +0,0 @@
## PD Disaggregation Performance
We evaluated the current implementation on two A10 servers. By comparing the performance of a 1P1D configuration with that of two regular (non-disaggregated) instances, we observed that P/D disaggregation achieves approximately 30% lower ITL while maintaining comparable total throughput. This aligns with findings from the Mooncake paper, which highlighted that P/D disaggregation is effective in reducing TBT/ITL under similar throughput conditions—or conversely, in enabling higher throughput under stricter ITL/TBT SLOs.
Moreover, we anticipate even greater benefits in larger-scale clusters where both the number of prefill and decode nodes (x and y in xPyD configurations) increase, offering enhanced scheduling flexibility and resource efficiency.
## Traffic Request Rate: 1.0
* model: Qwen2.5-7B-Instruct-GPTQ-Int4
* TP: 4
* random_input_len=8192, random_output_len=512
* num prompt=50
| Configuration | Output Token Throughput (tok/s) | Mean E2E Latency (ms) |Total Token Throughput (tok/s) | Mean TTFT (ms) | P99 TTFT (ms) | Mean ITL (ms) | P99 ITL (ms) |
|----------------|----------------------------------|-----------------------|----------------|---------------|---------------|--------------|---------------------------------|
| 1P1D | 407.59 | 3413.86 |7084.46 | 732.54 | 2952.57 | 7.23 | 10.76 |
| 2 Regular | 427.65 | 4586.54 |7433.27 | 767.18 | 1264.88 | 10.30 | 12.73 |
## Traffic Request Rate: 4.0
* model: Qwen2.5-7B-Instruct-GPTQ-Int4
* TP: 2
* random_input_len=2048, random_output_len=512
* num prompt=200
| Configuration | Output Token Throughput (tok/s) | Mean E2E Latency (ms) | Total Token Throughput (tok/s) | Mean TTFT (ms) |P99 TTFT (ms) | Mean ITL (ms) | P99 ITL (ms) |
|---------------|---------------------------------|-------------|--------------------------------|----------------|---------------|----------------|--------------|
| 1P1D | 1215.17 | 11519.24 | 6161.43 | 1111.94 | 2725.89 | 17.06 | 19.72 |
| 2 Regular | 1223.03 | 11683.15 | 6201.29 | 310.01 | 720.91 | 25.74 | 294.89 |

181
doc/en/sglang-hicache-benchmark-results-v1.md
View File

@ -1,181 +0,0 @@
# SGLang HiCache with Mooncake Backend Benchmark
We evaluated the performance of SGLang HiCache using a multi-turn conversation benchmark designed to simulate realistic user interactions. The benchmark spawns concurrent clients, each engaging in multi-round conversations. For every client, starting from the second round, the input consists of the concatenation of the input and output from the preceding round.
Our evaluation covers four different configurations:
* **GPU only**: KV cache stored in GPU memory.
* **(HiCache L1) + L2**: KV cache stored across both device and host memory.
* **(HiCache L1 + L2) + Mooncake backend**: KV cache stored across device, host, and Mooncake storage.
* **Pre-populated Mooncake**: Same as *L1 + L2 + Mooncake*, except that the KV cache is preloaded into Mooncake storage before testing.
Since HiCache is currently used to accelerate the prefill stage, this benchmark focuses on two key prefill metrics: Time-To-First-Token (TTFT) and input token throughput.
## Benchmark Result
![overall performance](../../docs/source/image/hicache_multi_turn_overall.png)
We first evaluated the overall performance on a cluster consisting 3 servers, each has 2 NVIDIA A10 GPUs and 2 100Gbps eRDMA NICs.
As shown in the figure, in terms of prefill performance, `populated Mooncake` achieves the best results, followed by `+Mooncake`, `+L2`, and finally `GPU only`.
![overall performance](../../docs/source/image/hicache_multi_turn_per_turn.png)
Next, we take a closer look at the sources of performance differences. We recorded the TTFT and cache hit rate for each conversation round. The experiment was conducted on a server equipped with 8 × H800 GPUs and 8 × mlx5 RDMA NICs. To minimize interference from the decode stage and highlight prefill performance differences, we set the output length to 1.
From the figure, we observe that the KV cache hit rate has a substantial impact on prefill performance. When the cache is hit, the TTFT is significantly lower compared to the cache-miss case.
Across all rounds, `+L2` outperforms `GPU only`, as host memory provides additional capacity for storing the KV cache.
In the first three rounds, `+Mooncake` and `+L2` exhibit the same cache hit rate. During this phase, `+L2` is slightly faster than `+Mooncake` since it avoids the overhead of fetching data from remote storage. However, as the number of rounds increases and the KV cache size exceeds the `+L2`'s memory capacity, `+L2`’s hit rate gradually decreases, leading to a noticeable increase in TTFT. In contrast, Mooncake maintains a high hit rate, and its TTFT grows only very slowly.
In practical deployment, Mooncake aggregates memory across the entire cluster into a large distributed memory pool, enabling the caching of a vast number of KV entries. Each cached KV can be shared by all SGLang instances, which substantially improves the cache hit rate under the same memory budget. As a result, Mooncake reduces inference latency and increases throughput for large-scale model serving.
## Benchmark Setup
### A10 Cluster
**Experimental Environment**
- Hardware Configuration: 2 × NVIDIA A10, 2 × 100Gbps eRDMA NICs.
- Model: Qwen3-14B
**Cluster Deployment:**
The benchmark was conducted across a 3-node cluster connected via eRDMA:
- Machine A: SGLang server, Mooncake master service, Mooncake client (30GB memory)
- Machine B: Mooncake client (60GB memory)
- Machine C: Mooncake client (60GB memory)
- Total Distributed Memory Pool: 150GB
**Benchmark Script:**
```bash
python3 benchmark/hicache/bench_multiturn.py \
--model-path $MODEL_PATH \
--disable-random-sample \
--output-length 16 \
--request-length 2048 \
--num-clients 20 \
--num-rounds 10 \
--max-parallel 4 \
--request-rate 20 \
--ready-queue-policy random \
--disable-auto-run \
--seed 42
```
**GPU Only:**
```bash
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--tp-size 2 \
--page-size 64
```
**HiCache L1 + L2:**
```bash
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--tp-size 2 \
--page-size 64 \
--enable-hierarchical-cache \
--hicache-write-policy write_through \
--hicache-ratio 2 --hicache-size 0
```
**L1 + L2 + Mooncake:**
```bash
MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1/metadata" \
MOONCAKE_GLOBAL_SEGMENT_SIZE=0 \
MOONCAKE_PROTOCOL="rdma" \
MOONCAKE_DEVICE="$DEVICE_LIST" \
MOONCAKE_MASTER=127.0.0.1:50051 \
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--tp-size 2 \
--page-size 64 \
--enable-hierarchical-cache \
--hicache-write-policy write_through \
--hicache-ratio 2 --hicache-size 0 \
--hicache-storage-prefetch-policy wait_complete \
--hicache-storage-backend mooncake
```
**Populated Mooncake**
`Populated Mooncake` uses the same parameters as `L1 + L2 + Mooncake`, except that the KV cache is preloaded into Mooncake storage before testing.
### H800 Server
**Experimental Environment**
- Hardware Configuration: 8 × H800 GPUs, 8 × mlx5 RDMA NICs, 2 NUMA nodes.
- Model: Qwen3-235B-A22B-Instruct-2507
**Benchmark Script:**
We used SGLang's [multiturn benchmark](https://github.com/sgl-project/sglang/blob/main/benchmark/hicache/bench_multiturn.py) for the evaluation.
```bash
python3 benchmark/hicache/bench_multiturn.py \
--model-path $MODEL_PATH \
--dataset-path $DATASET_PATH \
--disable-random-sample \
--output-length 1 \
--request-length 2048 \
--num-clients 80 \
--num-rounds 10 \
--max-parallel 4 \
--request-rate 16 \
--ready-queue-policy random \
--disable-auto-run \
--enable-round-barrier
```
**GPU Only:**
```bash
numactl --cpunodebind=0,1 --interleave=0,1 \
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--tp 8 \
--page-size 64
```
**HiCache L1 + L2:**
```bash
numactl --cpunodebind=0,1 --interleave=0,1 \
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--tp 8 \
--page-size 64 \
--enable-hierarchical-cache \
--hicache-write-policy write_through \
--hicache-size 30 # 8*30 = 240GB
```
**L1 + L2 + Mooncake:**
```bash
# Total Distributed Memory Pool: 760GB
numactl --cpunodebind=0,1 --interleave=0,1 \
MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata" \
MOONCAKE_GLOBAL_SEGMENT_SIZE=816043786240 \
MOONCAKE_PROTOCOL="rdma" \
MOONCAKE_DEVICE="$DEVICE_LIST" \
MOONCAKE_MASTER=127.0.0.1:50051 \
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--tp 8 \
--page-size 64 \
--enable-hierarchical-cache \
--hicache-write-policy write_through \
--hicache-ratio 2 \
--hicache-storage-prefetch-policy timeout \
--hicache-storage-backend mooncake
```

132
doc/en/sglang-integration-v1.md
View File

@ -1,132 +0,0 @@
# SGLang Disaggregated Serving with MooncakeTransferEngine
## Overview
This is the latest version of the MooncakeTransferEngine integration doc with the SGLang project based on [PR 4654](https://github.com/sgl-project/sglang/pull/4654) and [PR 4880](https://github.com/sgl-project/sglang/pull/4880) to support KVCache transfer for intra-node and inter-node disaggregated serving scenarios.
**_Please note that this is still an experimental version and will be modified anytime based on feedback from the SGLang community._**
## Installation
### Prerequisite
```bash
pip3 install mooncake-transfer-engine
```
Note: If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](build.md).
### Install the latest version of SGLang
#### 1. Clone SGLang from official repo
```bash
git clone git@github.com:sgl-project/sglang.git
```
#### 2. Build
##### 2.1 Build from source
```bash
cd sglang
pip install --upgrade pip
pip install -e "python[all]"
```
##### If running on AMD GPU, tried below steps to install sglang for rocm or just run in rocm docker from https://hub.docker.com/r/rocm/sgl-dev directly
```bash
pip install --upgrade pip
cd sgl-kernel
python setup_rocm.py install
cd ..
pip install -e "python[all_hip]"
or
docker run -it --rm --network=host \
--device=/dev/kfd --device=/dev/dri \
--ipc=host --shm-size 16G \
--group-add video \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v /home/workspace:/workspace \
rocm/sgl-dev:20250707
pip3 install mooncake-transfer-engine
sudo apt update
sudo apt install libibverbs1 libibverbs-dev -y
sudo apt-get install lsof net-tools iputils-ping -y
```
- If you encounter any problems that you cannot solve, please refer to the [SGLang official compilation guide](https://docs.sglang.ai/start/install.html).
## Configuration
- **Update(Apr 10, 2025)** Good news: The configuration file requirement has been removed since [PR 5460](https://github.com/sgl-project/sglang/pull/5460). There is no need to prepare the _mooncake.json_ file anymore.
### To run prefill instance and decode instance on different node
Prefill:
```bash
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --disaggregation-mode prefill --port 30000 --host 192.168.0.137 --tp-size 2 --page-size 32
```
- The `--model-path` parameter specifies the model to use.
- The `--host` parameter specifies the SGLang service host.
- The `--port` parameter specifies the SGLang service port on which to listen.
- The `--disaggregation-mode` is the node's role, either 'prefill' or 'decode'.
- The `--disaggregation-ib-device` is the device to be used for data transmission, it is optional since we will detect this config automatically. Or you can still explicitly specify devices if needed. If multiple NIC devices are used, they can be separated by commas, such as "erdma_0,erdma_1". Please note that there are no spaces between them.
- The `--page-size` is the page size for the radix tree, a larger page size will help reduce kvcache indice fragmentation.
- Option `--tp-size` is supported. Example: append `--tp-size 2` to the run command to run SGLang with multiple GPUs.
Decode:
```bash
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --disaggregation-mode decode --port 30001 --host 192.168.0.140 --tp-size 2 --page-size 32
```
Proxy:
```bash
python3 -m sglang.srt.disaggregation.mini_lb --prefill http://192.168.0.137:30000 --decode http://192.168.0.140:30001 --host 0.0.0.0 --port 8000
```
Test:
```bash
curl -X POST http://127.0.0.1:8000/generate -H "Content-Type: application/json" -d '{
"text": "Let me tell you a long story ",
"sampling_params": {
"temperature": 0
}
}'
```
### To run prefill instance and decode instance on the same node
Prefill:
```bash
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --disaggregation-mode prefill --port 30000 --host 192.168.0.137 --tp-size 2 --page-size 32
```
Decode:
```bash
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --disaggregation-mode decode --port 30001 --base-gpu-id 2 --host 192.168.0.137 --tp-size 2 --page-size 32
```
- The function of `--base-gpu-id` is similar to env var`CUDA_VISIBLE_DEVICES`, which is used to avoid reusing the 0th GPU card. The difference is that it is used to specify the starting number of the GPU. If it is set to 2, the first and second cards will be skipped, and the third card will be used directly.
Proxy:
```bash
python3 -m sglang.srt.disaggregation.mini_lb --prefill http://192.168.0.137:30000 --decode http://192.168.0.137:30001 --host 0.0.0.0 --port 8000
```
Test:
```bash
curl -X POST http://127.0.0.1:8000/generate -H "Content-Type: application/json" -d '{
"text": "Let me tell you a long story ",
"sampling_params": {
"temperature": 0
}
}'
```
Note:
- TP is supported but not required, you can remove `--tp-size 2` if you want.
- The `--disaggregation-ib-device` is the device to be used for data transmission, it is optional since we will detect this config automatically. Or you can still explicitly specify devices if needed. If multiple NIC devices are used, they can be separated by commas, such as "erdma_0,erdma_1". Please note that there are no spaces between them.
- XpYd is supported. It is ok to run multiple prefill and multiple decode instances on the same node, but you need to set up `--prefill-bootstrap-ports` when running multiple prefill instances on the same node.
- e.g., `python3 -m sglang.srt.disaggregation.mini_lb --prefill http://192.168.0.137:30000 http://192.168.0.140:30000 --decode http://192.168.0.137:30001 http://192.168.0.140:30001 --host 0.0.0.0 --port 8000`
- `python3 -m sglang.srt.disaggregation.mini_lb --prefill http://192.168.0.137:30000 http://192.168.0.137:30100 --decode http://192.168.0.137:30200 http://192.168.0.137:30300 --host 0.0.0.0 --port 8000 --prefill-bootstrap-ports 8998 8999`
- HuggingFace timeout can be addressed by `export SGLANG_USE_MODELSCOPE=true`

431
doc/en/transfer-engine-python.md
View File

@ -1,431 +0,0 @@
# Transfer Engine Python API Documentation
## Overview
The Transfer Engine Python API provides a high-level interface for efficient data transfer between distributed systems using RDMA (Remote Direct Memory Access) and other transport protocols. It enables fast, low-latency data movement between nodes in a cluster.
## Installation
```python
from mooncake.engine import TransferEngine
```
## Class: TransferEngine
The main class that provides all transfer engine functionality.
### Constructor
```python
TransferEngine()
```
Creates a new TransferEngine instance with default settings.
### Initialization Methods
#### initialize()
```python
initialize(local_hostname, metadata_server, protocol, device_name)
```
Initializes the transfer engine with basic configuration.
**Parameters:**
- `local_hostname` (str): The hostname and port of the local server (e.g., "127.0.0.1:12345")
- `metadata_server` (str): The metadata server connection string (e.g., "127.0.0.1:2379" or "etcd://127.0.0.1:2379")
- `protocol` (str): The transport protocol to use ("rdma", "tcp", etc.)
- `device_name` (str): Comma-separated list of device names to filter, or empty string for all devices
**Returns:**
- `int`: 0 on success, negative value on failure
#### initialize_ext()
```python
initialize_ext(local_hostname, metadata_server, protocol, device_name, metadata_type)
```
Initializes the transfer engine with extended configuration including metadata type specification.
**Parameters:**
- `local_hostname` (str): The hostname and port of the local server
- `metadata_server` (str): The metadata server connection string
- `protocol` (str): The transport protocol to use
- `device_name` (str): Comma-separated list of device names to filter
- `metadata_type` (str): The type of metadata server ("etcd", "p2p", etc.)
**Returns:**
- `int`: 0 on success, negative value on failure
### Network Information
#### get_rpc_port()
```python
get_rpc_port()
```
Gets the RPC port that the transfer engine is listening on.
**Returns:**
- `int`: The RPC port number
### Buffer Management
#### allocate_managed_buffer()
```python
allocate_managed_buffer(length)
```
Allocates a managed buffer of the specified size using a buddy allocation system for efficient memory management.
**Parameters:**
- `length` (int): The size of the buffer to allocate in bytes
**Returns:**
- `int`: The memory address of the allocated buffer as an integer, or 0 on failure
#### free_managed_buffer()
```python
free_managed_buffer(buffer_addr, length)
```
Frees a previously allocated managed buffer.
**Parameters:**
- `buffer_addr` (int): The memory address of the buffer to free
- `length` (int): The size of the buffer in bytes
**Returns:**
- `int`: 0 on success, negative value on failure
#### get_first_buffer_address()
```python
get_first_buffer_address(segment_name)
```
Gets the address of the first buffer in a specified segment.
**Parameters:**
- `segment_name` (str): The name of the segment
**Returns:**
- `int`: The memory address of the first buffer in the segment
### Data Transfer Operations
#### transfer_sync_write()
```python
transfer_sync_write(target_hostname, buffer, peer_buffer_address, length)
```
Performs a synchronous write operation to transfer data from local buffer to remote buffer.
**Parameters:**
- `target_hostname` (str): The hostname of the target server
- `buffer` (int): The local buffer address
- `peer_buffer_address` (int): The remote buffer address
- `length` (int): The number of bytes to transfer
**Returns:**
- `int`: 0 on success, negative value on failure
#### transfer_sync_read()
```python
transfer_sync_read(target_hostname, buffer, peer_buffer_address, length)
```
Performs a synchronous read operation to transfer data from remote buffer to local buffer.
**Parameters:**
- `target_hostname` (str): The hostname of the target server
- `buffer` (int): The local buffer address
- `peer_buffer_address` (int): The remote buffer address
- `length` (int): The number of bytes to transfer
**Returns:**
- `int`: 0 on success, negative value on failure
#### transfer_sync()
```python
transfer_sync(target_hostname, buffer, peer_buffer_address, length, opcode)
```
Performs a synchronous transfer operation with specified opcode.
**Parameters:**
- `target_hostname` (str): The hostname of the target server
- `buffer` (int): The local buffer address
- `peer_buffer_address` (int): The remote buffer address
- `length` (int): The number of bytes to transfer
- `opcode` (TransferOpcode): The transfer operation type (READ or WRITE)
**Returns:**
- `int`: 0 on success, negative value on failure
#### transfer_submit_write()
```python
transfer_submit_write(target_hostname, buffer, peer_buffer_address, length)
```
Submits an asynchronous write operation and returns immediately.
**Parameters:**
- `target_hostname` (str): The hostname of the target server
- `buffer` (int): The local buffer address
- `peer_buffer_address` (int): The remote buffer address
- `length` (int): The number of bytes to transfer
**Returns:**
- `int`: Batch ID for tracking the operation, or negative value on failure
#### transfer_check_status()
```python
transfer_check_status(batch_id)
```
Checks the status of an asynchronous transfer operation.
**Parameters:**
- `batch_id` (int): The batch ID returned from transfer_submit_write()
**Returns:**
- `int`:
- 1: Transfer completed successfully
- 0: Transfer still in progress
- -1: Transfer failed
- -2: Transfer timed out
### Buffer I/O Operations
#### write_bytes_to_buffer()
```python
write_bytes_to_buffer(dest_address, src_ptr, length)
```
Writes bytes from a Python bytes object to a buffer at the specified address.
**Parameters:**
- `dest_address` (int): The destination buffer address
- `src_ptr` (bytes): The source bytes to write
- `length` (int): The number of bytes to write
**Returns:**
- `int`: 0 on success, negative value on failure
#### read_bytes_from_buffer()
```python
read_bytes_from_buffer(source_address, length)
```
Reads bytes from a buffer at the specified address and returns them as a Python bytes object.
**Parameters:**
- `source_address` (int): The source buffer address
- `length` (int): The number of bytes to read
**Returns:**
- `bytes`: The bytes read from the buffer
### Memory Registration (Experimental)
#### register_memory()
```python
register_memory(buffer_addr, capacity)
```
Registers a memory region for RDMA access (experimental feature).
**Parameters:**
- `buffer_addr` (int): The memory address to register
- `capacity` (int): The size of the memory region in bytes
**Returns:**
- `int`: 0 on success, negative value on failure
#### unregister_memory()
```python
unregister_memory(buffer_addr)
```
Unregisters a previously registered memory region.
**Parameters:**
- `buffer_addr` (int): The memory address to unregister
**Returns:**
- `int`: 0 on success, negative value on failure
### Enums
#### TransferOpcode
```python
TransferOpcode.READ # Read operation
TransferOpcode.WRITE # Write operation
```
## Environment Variables
The Transfer Engine respects the following environment variables:
- `MC_TRANSFER_TIMEOUT`: Sets the transfer timeout in seconds (default: 30)
- `MC_METADATA_SERVER`: Default metadata server address
- `MC_LEGACY_RPC_PORT_BINDING`: Enables legacy RPC port binding behavior
- `MC_TCP_BIND_ADDRESS`: Specifies the TCP bind address
- `MC_CUSTOM_TOPO_JSON`: Path to custom topology JSON file
- `MC_TE_METRIC`: Enables metrics reporting (set to "1", "true", "yes", or "on")
- `MC_TE_METRIC_INTERVAL_SECONDS`: Sets metrics reporting interval in seconds
## Usage Examples
### Basic Setup and Data Transfer
```python
from mooncake.engine import TransferEngine
import os
# Create transfer engine instance
engine = TransferEngine()
# Initialize with basic configuration
ret = engine.initialize(
local_hostname="127.0.0.1:12345",
metadata_server="127.0.0.1:2379",
protocol="rdma",
device_name=""
)
if ret != 0:
raise RuntimeError(f"Initialization failed with code {ret}")
# Allocate and initialize client buffer (1MB)
client_buffer = np.ones(1024 * 1024, dtype=np.uint8) # Fill with ones
buffer_data = client_buffer.ctypes.data
buffer_data_len = client_buffer.nbytes
# Prepare data
data = b"Hello, Transfer Engine!"
data_len = len(data)
engine.register(buffer_data, buffer_data_len)
# Get Remote Addr from ZMQ or upper-layer inference framework
remote_addr = ??
# Transfer data to remote node
ret = engine.transfer_sync_write(
target_hostname="127.0.0.1:12346",
buffer=data,
peer_buffer_address=remote_addr,
length=data_len
)
if ret == 0:
print("Data transfer completed successfully")
else:
print(f"Data transfer failed with code {ret}")
engine.deregister(data)
```
### Asynchronous Transfer
```python
# Submit asynchronous write
batch_id = engine.transfer_submit_write(
target_hostname="127.0.0.1:12346",
buffer=local_addr,
peer_buffer_address=remote_addr,
length=data_len
)
if batch_id < 0:
print(f"Failed to submit transfer with code {batch_id}")
else:
# Poll for completion
while True:
status = engine.transfer_check_status(batch_id)
if status == 1:
print("Transfer completed successfully")
break
elif status == -1:
print("Transfer failed")
break
elif status == -2:
print("Transfer timed out")
break
# Transfer still in progress, continue polling
import time
time.sleep(0.001) # Small delay to avoid busy waiting
```
### Managed Buffer Allocation
```python
# Allocate managed buffer
buffer_size = 1024 * 1024 # 1MB
buffer_addr = engine.allocate_managed_buffer(buffer_size)
if buffer_addr == 0:
print("Failed to allocate buffer")
else:
# Use the buffer
test_data = b"Test data for managed buffer"
engine.write_bytes_to_buffer(buffer_addr, test_data, len(test_data))
# Read back
read_data = engine.read_bytes_from_buffer(buffer_addr, len(test_data))
print(f"Read data: {read_data}")
# Free the buffer when done
engine.free_managed_buffer(buffer_addr, buffer_size)
```
## Error Handling
All methods return integer status codes:
- `0`: Success
- Negative values: Error codes indicating various failure conditions
Common error scenarios:
- Network connectivity issues
- Invalid buffer addresses
- Memory allocation failures
- Transfer timeouts
- Metadata server connection problems
## Performance Considerations
1. **Buffer Reuse**: Reuse allocated buffers when possible to avoid frequent allocation/deallocation overhead
2. **Batch Operations**: Use `transfer_submit_write()` and `transfer_check_status()` for better throughput when multiple transfers are needed
3. **Memory Alignment**: Ensure buffers are properly aligned for optimal RDMA performance
4. **Timeout Configuration**: Adjust `MC_TRANSFER_TIMEOUT` based on your network characteristics and data sizes
## Thread Safety
The Transfer Engine Python API is thread-safe for most operations. However, it's recommended to:
- Use separate TransferEngine instances for different threads when possible
- Avoid concurrent modifications to the same buffer addresses
- Use proper synchronization when sharing buffer addresses between threads
## Troubleshooting
1. **Initialization Failures**: Check metadata server connectivity and network configuration
2. **Transfer Failures**: Verify target hostname is correct and network connectivity is established
3. **Memory Issues**: Ensure sufficient system memory and proper buffer alignment
4. **Performance Issues**: Check RDMA device configuration and network topology

107
doc/en/troubleshooting.md
View File

@ -1,107 +0,0 @@
# Troubleshooting
This document lists common errors that may occur when using Mooncake Store and provides troubleshooting and resolution measures.
> **CheckList**
> - [ ] `connectable_name` is not the local machine's LAN/WAN address, such as the loopback address (`127.0.0.1`/`localhost`) and address of other machines.
> - [ ] Incorrect MTU and GID configurations. Use the environment variables MC_MTU and MC_GID_INDEX.
> - [ ] Incorrect RDMA device name and connection status is not active.
> - [ ] etcd is not started normally and it is not bind with `0.0.0.0`.
## Metadata and Out-of-Band Communication
1. At startup, a `TransferMetadata` object is constructed according to the incoming `metadata_server` parameter. During program execution, this object is used to communicate with the etcd server to maintain internal data required for connection.
2. At startup, the current node is registered with the cluster according to the incoming `connectable_name` parameter and `rpc_port` parameter, and the TCP port specified by the `rpc_port` parameter is listened. Before other nodes send the first read/write request to the current node, they will use the above information, resolve DNS, and initiate a connection through socket's `connect()` method.
Errors in this part usually indicate that the error occurred within the `mooncake-transfer-engine/src/transfer_metadata.cpp` file.
### Recommended Troubleshooting Directions
1. The incoming `metadata_server` parameter is not a valid and reachable etcd server address (or a group of addresses). In this case, an `Error from etcd client` error will be displayed. It is recommended to investigate from the following aspects:
- After installing the etcd service, the default listening IP is 127.0.0.1, which other nodes cannot use. Therefore, the actual listening IP should be determined in conjunction with the network environment. In the experimental environment, 0.0.0.0 can be used. For example, the following command line can be used to start the required service:
```bash
# This is 10.0.0.1
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://10.0.0.1:2379
```
You can verify this on other nodes using `curl <metadata_server>`.
- HTTP proxies need to be disabled before starting the program.
```bash
unset http_proxy
unset https_proxy
```
2. Other nodes cannot establish a socket out-of-band communication with the current node through the incoming `connectable_name` parameter and `rpc_port` parameter to implement connection establishment operations. The types of errors in this case may include:
- A `bind address already in use` error is displayed when starting the process, usually because the port number corresponding to the `rpc_port` parameter is occupied. Try using another port number.
- Another node displays a `connection refused` type of error when initiating a Batch Transfer to the current node, which requires a focus on checking the correctness of these two parameters when the current node is created.
- `connectable_name` must be a non-Loopback IP address of the current node or a valid Hostname (with records in DNS or `/etc/hosts`), and other nodes in the cluster can use `connectable_name` and `rpc_port` parameters to connect to the current node.
- There may be firewall mechanisms in some networks that need to be added to the whitelist in advance.
- If the correspondence between `local_server_name` and `connectable_name`/`rpc_port` parameters changes and various errors occur, you can try clearing the etcd database and restarting the cluster.
## RDMA Resource Initialization
1. At startup, all RDMA network cards corresponding to the incoming `nic_priority_matrix` parameter are initialized, including device contexts and other internal objects.
2. Before other nodes send the first read/write request to the current node, they will exchange GID, LID, QP Num, etc., through out-of-band communication mechanisms and complete the establishment of RDMA reliable connection paths.
Errors in this part usually indicate that the error occurred within the `mooncake-transfer-engine/src/transport/rdma_transport/rdma_*.cpp` files.
### Recommended Troubleshooting Directions
1. If the error `No matched device found` is displayed, check if there are any network card names in the `nic_priority_matrix` parameter that do not exist on the machine. You can use the `ibv_devinfo` command to view the list of installed network cards on the machine.
2. If the error `Device XXX port not active` is displayed, it indicates that the default RDMA Port of the corresponding device (RDMA device port, to be distinguished from the `rpc_port` TCP port) is not in the ACTIVE state. This is usually due to RDMA cables not being installed properly or the driver not being configured correctly. You can use the `MC_IB_PORT` environment variable to change the default RDMA Port used.
3. If both the error `Worker: Cannot make connection for endpoint` and `Failed to exchange handshake description` are displayed, it indicates that the two parties cannot establish a reliable RDMA connection path. In most cases, it is usually due to incorrect configuration on one side or the inability of both parties to reach each other. First, use tools like `ib_send_bw` to confirm the reachability of the two nodes and pay attention to the output of GID, LID, MTU, and other parameter information. Then, analyze the possible error points based on the error message:
1. After starting, the log output usually includes several lines of log information like `RDMA device: XXX, LID: XXX, GID: (X) XX:XX:XX:...`. If the displayed GID address is all 0 (the bracket indicates GID Index), you need to choose the correct GID Index according to the network environment and specify it at startup using the `MC_GID_INDEX` environment variable.
2. If the error `Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num` is displayed, first determine which party the error occurred on. If there is no prefix `Handshake request rejected by peer endpoint: `, it indicates that the problem comes from the party displaying the error. According to the error message, you need to check the MTU length configuration (adjust using the `MC_MTU` environment variable), whether your own and the other party's GID addresses are valid, etc. At the same time, if the two nodes cannot achieve physical connection, it may also be hang/interrupted at this step, please pay attention.
4. If the error `Failed to create QP: Cannot allocate memory` is displayed, it typically caused by too many QP have been created, reaching the driver limit. You can use `rdma resource` to trace how many QP is created. One possible way to resolve this issue:
- Update Mooncake to version v0.3.5 or later
- Set the environment variable `MC_ENABLE_DEST_DEVICE_AFFINITY=1` before starting the application
## RDMA Transfer Period
### Recommended Troubleshooting Directions
If the network state is unstable, some requests may not be delivered, displaying errors like `Worker: Process failed for slice`. Transfer Engine can avoid problems by reselecting paths, etc. In some complex cases, if a large number of such errors are output continuously, it is recommended to search for the cause of the problem according to the string prompt of the last field.
Note: In most cases, the errors output, except for the first occurrence, are `work request flushed error`. This is because when the first error occurs, the RDMA driver sets the connection to an unavailable state, so tasks in the submission queue are blocked from execution and subsequent errors are reported. Therefore, it is recommended to locate the first occurrence of the error and check it.
In addition, if the error `Failed to get description of XXX` is displayed, it indicates that the Segment name input by the user when calling the `openSegment` interface cannot be found in the etcd database. For memory read/write scenarios, the Segment name needs to strictly match the `local_hostname` field filled in by the other node during initialization.
## SGLang Common Questions
### Do I need RDMA to run SGLang and Mooncake?
When using Mooncake for KV cache transfer in SGLang PD disaggregation deployments, GPUDirect RDMA (GDR) is required.
When using Mooncake as a KV cache storage backend in SGLang HiCache, RDMA is recommended for better performance.
However, if RDMA NICs are not available, the TCP protocol is also supported.
### How to make sure GPUDirect RDMA (GDR) is supported
1. Verify the presence of an RDMA-capable NIC (e.g., Mellanox, ERDMA) and drivers.
```
ibv_devices
lspci | grep rdma
lsmod | grep -E 'ib_core|mlx4_core|mlx5_core|nvidia_peer_mem'
```
If no RDMA devices appear: (1) Confirm physical NIC presence via lspci
(2) Install vendor-specific drivers (e.g., Mellanox MLNX_OFED)
2. check GDR driver is ready, and peer_memory module (part of MLNX_OFED) should be installed
```
# Check peer_memory module (from MLNX_OFED)
lsmod | grep peer_mem
# Verify NVIDIA peer memory module
lsmod | grep nvidia_peer_mem
```
3. If you use container to run SGLang, please make sure RDMA and GDR driver are installed in the container and run container in privileged mode. Requirements: (1) privileged mode must be enabled. (2) RDMA devices/NVIDIA devices mounted into container
4. Check the connectivity
Benchmark end-to-end performance using ib_write_bw.
```
apt install perftest
# server side
ib_write_bw -d [rdma_device] -R -x gdr
# client side
ib_write_bw -d [rdma_device] -R -x gdr [server_ip]
```
Expected Output:
Successful bidirectional transfer with "BW peak" reported
Errors with -x gdr indicate GDR setup failures

49
doc/en/vllm-benchmark-results-v0.2.md
View File

@ -1,49 +0,0 @@
# Benchmark performance on NVIDIA A10 (V0.2)
Here are some preview mooncake benchmark results on A10 with up to 2 RDMA NICs. We are currently having some trouble benchmarking `PyNcclConnector` now. For some unknown reasons, it crashes a lot for inter-node disaggregated scenarios. So the benchmark results haven't included the `PyNcclConnector` yet.
In addition, we are also coordinating resources to integrate some machines with more RDMA NICs and more advanced GPUs. The official benchmark results will be released in due time.
### Varying tp (input length = 1024, qps = 2, output length =6)
| Setting | num_rdma_nic | Successful Requests | Duration (s) | Total Input Tokens | Total Generated Tokens | Req Throughput (req/s) | Output Token Throughput (tok/s) | Total Token Throughput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) | Mean ITL (ms) | Median ITL (ms) | P99 ITL (ms) |
|-----------------|--------------|---------------------|--------------|--------------------|------------------------|------------------------|---------------------------------|-------------------------------|----------------|-----------------|--------------|---------------|------------------|--------------|--------------|----------------|-------------|
| tp = 1 | 2 | 200 | 99.47 | 201995 | 1200 | 2.01 | 12.06 | 2042.74 | 1056.76 | 635.00 | 4006.59 | 97.08 | 26.94 | 781.91 | 97.01 | 14.05 | 2205.51 |
| tp = 2 | 2 | 200 | 98.98 | 201995 | 1200 | 2.02 | 12.12 | 2052.95 | 314.87 | 231.20 | 949.40 | 25.65 | 15.56 | 129.60 | 25.62 | 15.48 | 288.06 |
| tp = 4 | 2 | 200 | 98.76 | 201995 | 1200 | 2.03 | 12.15 | 2057.44 | 198.10 | 160.03 | 461.61 | 23.52 | 18.93 | 94.38 | 23.50 | 18.01 | 187.79 |
| tp = 1 | 1 | 200 | 99.44 | 201995 | 1200 | 2.01 | 12.07 | 2043.39 | 1071.12 | 631.56 | 4361.02 | 83.93 | 26.93 | 794.75 | 83.86 | 14.13 | 1932.66 |
| tp = 2 | 1 | 200 | 98.96 | 201995 | 1200 | 2.02 | 12.13 | 2053.35 | 335.26 | 258.30 | 997.93 | 28.84 | 15.56 | 144.82 | 28.80 | 15.42 | 397.56 |
| tp = 4 | 1 | 200 | 98.78 | 201995 | 1200 | 2.02 | 12.15 | 2057.03 | 201.68 | 162.85 | 456.33 | 22.31 | 16.74 | 94.76 | 22.29 | 16.73 | 189.13 |
| tp = 1 | TCP | 200 | 99.55 | 201995 | 1200 | 2.01 | 12.05 | 2041.13 | 1414.05 | 766.23 | 6035.36 | 155.01 | 35.28 | 1191.24 | 154.91 | 14.32 | 3148.99 |
| tp = 2 | TCP | 200 | 98.97 | 201995 | 1200 | 2.02 | 12.12 | 2053.03 | 333.74 | 251.32 | 954.63 | 28.74 | 15.49 | 161.24 | 28.70 | 15.35 | 393.52 |
| tp = 4 | TCP | 200 | 98.78 | 201995 | 1200 | 2.02 | 12.15 | 2056.94 | 205.37 | 162.92 | 463.70 | 21.54 | 16.51 | 94.04 | 21.51 | 16.56 | 170.54 |
### Varying qps (length = 1024, tp = 4, output length =6)
|Setting | num_rdma_nic | Successful Requests | Duration (s) | Total Input Tokens | Total Generated Tokens | Req Throughput (req/s) | Output Token Throughput (tok/s) | Total Token Throughput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) | Mean ITL (ms) | Median ITL (ms) | P99 ITL (ms) |
|-----------------|--------------|---------------------|--------------|--------------------|------------------------|------------------------|---------------------------------|-------------------------------|----------------|-----------------|--------------|---------------|------------------|--------------|--------------|----------------|-------------|
| qps = 2 | 2 | 200 | 98.77 | 201995 | 1200 | 2.02 | 12.15 | 2057.33 | 200.64 | 156.62 | 478.22 | 22.63 | 17.35 | 99.61 | 22.60 | 17.08 | 186.25 |
| qps = 4 | 2 | 200 | 49.75 | 201995 | 1200 | 4.02 | 24.12 | 4084.03 | 341.88 | 240.68 | 1430.54 | 38.36 | 18.39 | 313.45 | 38.31 | 17.17 | 588.80 |
| qps = 6 | 2 | 200 | 33.44 | 201995 | 1200 | 5.98 | 35.88 | 6075.54 | 851.15 | 501.59 | 3239.89 | 102.51 | 47.67 | 606.77 | 102.34 | 18.35 | 1704.79 |
| qps = 8 | 2 | 200 | 27.16 | 201995 | 1200 | 7.36 | 44.19 | 7482.52 | 4835.08 | 5733.45 | 8846.27 | 1276.59 | 1150.11 | 4401.23 | 1274.43 | 48.34 | 20682.35 |
| qps = 2 | 1 | 200 | 98.77 | 201995 | 1200 | 2.02 | 12.15 | 2057.31 | 201.77 | 161.53 | 473.44 | 22.13 | 16.52 | 96.18 | 22.11 | 16.51 | 190.40 |
| qps = 4 | 1 | 200 | 49.76 | 201995 | 1200 | 4.02 | 24.12 | 4083.83 | 337.31 | 243.38 | 1395.85 | 39.95 | 17.61 | 325.39 | 39.88 | 17.06 | 838.68 |
| qps = 6 | 1 | 200 | 33.44 | 201995 | 1200 | 5.98 | 35.88 | 6075.99 | 820.53 | 458.84 | 3169.52 | 83.92 | 30.50 | 663.07 | 83.78 | 17.85 | 1306.32 |
| qps = 8 | 1 | 200 | 27.19 | 201995 | 1200 | 7.36 | 44.14 | 7473.44 | 5291.91 | 6160.55 | 9596.56 | 1190.36 | 1040.63 | 4418.66 | 1188.33 | 47.61 | 20815.23 |
| qps = 2 | TCP | 200 | 98.76 | 201995 | 1200 | 2.03 | 12.15 | 2057.42 | 207.22 | 160.81 | 511.01 | 22.17 | 16.59 | 94.96 | 22.15 | 16.59 | 181.82 |
| qps = 4 | TCP | 200 | 49.79 | 201995 | 1200 | 4.02 | 24.10 | 4081.06 | 355.43 | 252.63 | 1554.91 | 40.15 | 16.92 | 314.28 | 40.09 | 16.66 | 708.50 |
| qps = 6 | TCP | 200 | 33.49 | 201995 | 1200 | 5.97 | 35.83 | 6067.71 | 907.74 | 514.85 | 3253.93 | 122.75 | 45.51 | 648.40 | 122.56 | 18.09 | 2282.92 |
| qps = 8 | TCP | 200 | 28.39 | 201995 | 1200 | 7.04 | 42.26 | 7156.09 | 6714.57 | 7885.09 | 11787.51 | 1116.06 | 408.32 | 4645.25 | 1114.29 | 46.87 | 21898.03 |
### Varying input length (tp = 4, qps = 2, output length =6)
| Setting | num_rdma_nic | Successful Requests | Duration (s) | Total Input Tokens | Total Generated Tokens | Req Throughput (req/s) | Output Token Throughput (tok/s) | Total Token Throughput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) | Mean ITL (ms) | Median ITL (ms) | P99 ITL (ms) |
|-----------------|--------------|---------------------|--------------|--------------------|------------------------|------------------------|---------------------------------|-------------------------------|----------------|-----------------|--------------|---------------|------------------|--------------|--------------|----------------|-------------|
| 1024 | 2 | 200 | 98.77 | 201995 | 1200 | 2.02 | 12.15 | 2057.32 | 195.47 | 151.55 | 482.84 | 22.83 | 19.27 | 96.55 | 22.81 | 18.12 | 158.16 |
| 2048 | 2 | 200 | 99.22 | 406707 | 1200 | 2.02 | 12.09 | 4110.95 | 723.76 | 488.67 | 2941.96 | 67.25 | 18.93 | 632.73 | 67.20 | 17.49 | 1209.54 |
| 4096 | 2 | 200 | 117.42 | 818415 | 1200 | 1.70 | 10.22 | 6979.90 | 14616.48 | 18323.82 | 23191.04 | 8042.84 | 7593.16 | 19851.11 | 8040.02 | 65.43 | 93511.26 |
| 8192 | 2 | 200 | 247.77 | 1636065 | 1200 | 0.81 | 4.84 | 6608.10 | 75783.36 | 79331.60 | 147544.42 | 16961.27 | 15140.11 | 39278.98 | 16958.32 | 90.01 | 186151.61 |
| 1024 | 1 | 200 | 98.77 | 201995 | 1200 | 2.02 | 12.15 | 2057.31 | 201.77 | 161.53 | 473.44 | 22.13 | 16.52 | 96.18 | 22.11 | 16.51 | 190.40 |
| 2048 | 1 | 200 | 99.25 | 406707 | 1200 | 2.02 | 12.09 | 4109.96 | 719.43 | 482.02 | 3208.13 | 61.92 | 17.64 | 681.26 | 61.86 | 16.83 | 978.90 |
| 4096 | 1 | 200 | 111.88 | 818415 | 1200 | 1.79 | 10.73 | 7326.16 | 20362.10 | 22807.05 | 31853.55 | 5915.16 | 4521.51 | 18739.12 | 5913.18 | 67.03 | 81600.29 |
| 8192 | 1 | 200 | 270.01 | 1636065 | 1200 | 0.74 | 4.44 | 6063.79 | 103355.40 | 106546.65 | 172025.11 | 12894.35 | 11027.66 | 35110.13 | 12892.85 | 64.84 | 151774.68 |
| 1024 | TCP | 200 | 98.81 | 201995 | 1200 | 2.02 | 12.14 | 2056.44 | 203.32 | 160.83 | 460.90 | 21.81 | 16.96 | 95.27 | 21.78 | 16.91 | 171.80 |
| 2048 | TCP | 200 | 99.27 | 406707 | 1200 | 2.01 | 12.09 | 4108.98 | 731.60 | 484.78 | 3213.69 | 68.55 | 17.88 | 639.93 | 68.49 | 17.33 | 1257.45 |
| 4096 | TCP | 200 | 118.37 | 818415 | 1200 | 1.69 | 10.14 | 6923.89 | 23735.69 | 27101.97 | 36573.47 | 6386.62 | 5102.00 | 20032.26 | 6384.71 | 69.57 | 92811.27 |
| 8192 | TCP | 200 | 278.12 | 1636065 | 1200 | 0.72 | 4.31 | 5886.95 | 106873.23 | 109941.33 | 179781.64 | 13360.87 | 12155.24 | 36022.96 | 13359.20 | 68.01 | 156716.38 |

22
doc/en/vllm-benchmark-results-v1.md
View File

@ -1,22 +0,0 @@
# Benchmark performance on NVIDIA A10 (V1)
Here are some preview MooncakeStore benchmark results on A10 with "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4".
### Varying PD ratio (input length = 1024, qps = 2, output length =6, num of requests = 200)
| Configuration | Backend | Duration (s) | Output Token Throughput (tok/s) | Total Token Throughput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) | Mean ITL (ms) | Median ITL (ms) | P99 ITL (ms) |
|----------------------------|----------------------|--------------|---------------------------------|-------------------------------|----------------|-----------------|--------------|---------------|------------------|--------------|--------------|----------------|-------------|
| 2P2D tp = 1 | Redis | 99.47 | 12.06 | 2042.75 | 844.28 | 666.84 | 2270.91 | 16.88 | 11.57 | 104.83 | 16.84 | 11.56 | 239.67 |
| | MooncakeStore (TCP) | 99.44 | 12.07 | 2043.30 | 817.43 | 639.48 | 1969.89 | 12.49 | 11.55 | 45.52 | 12.46 | 11.55 | 15.31 |
| | MooncakeStore (RDMA) | 99.33 | 12.08 | 2045.57 | 763.58 | 604.22 | 2030.34 | 12.43 | 11.53 | 43.02 | 12.39 | 11.52 | 15.40 |
| 2P2D tp = 2 | Redis | 98.92 | 12.13 | 2054.12 | 397.20 | 352.37 | 782.44 | 9.00 | 8.05 | 36.06 | 8.97 | 8.03 | 13.94 |
| | MooncakeStore (TCP) | 98.81 | 12.14 | 2056.43 | 327.91 | 309.38 | 573.36 | 8.33 | 8.04 | 17.62 | 8.30 | 8.03 | 11.23 |
| | MooncakeStore (RDMA) | 98.74 | 12.15 | 2057.79 | 271.25 | 250.11 | 532.00 | 8.34 | 8.12 | 14.70 | 8.31 | 8.10 | 11.12 |
| 3P3D (1 remote P, 1 remote D) tp = 2 qps = 2 | Redis | 98.89 | 12.13 | 2054.80 | 382.73 | 358.18 | 659.31 | 8.07 | 8.02 | 8.86 | 8.04 | 7.99 | 10.14 |
| | MooncakeStore (TCP) | 98.71 | 12.16 | 2058.47 | 298.71 | 302.74 | 512.84 | 8.06 | 8.04 | 8.54 | 8.03 | 8.02 | 8.88 |
| | MooncakeStore (RDMA) | 98.69 | 12.16 | 2058.88 | 269.73 | 252.96 | 543.38 | 8.13 | 8.04 | 10.49 | 8.10 | 8.02 | 11.29 |
| 4P2D (2 remote P) tp = 2 | Redis | 98.85 | 12.14 | 2055.66 | 350.39 | 339.15 | 506.78 | 8.54 | 8.01 | 26.42 | 8.51 | 7.99 | 11.43 |
| | MooncakeStore (TCP) | 98.76 | 12.15 | 2057.56 | 312.32 | 307.50 | 475.79 | 8.29 | 8.03 | 19.87 | 8.25 | 8.01 | 9.51 |
| | MooncakeStore (RDMA) | 98.71 | 12.16 | 2058.59 | 259.87 | 251.23 | 461.96 | 8.20 | 8.05 | 10.20 | 8.17 | 8.03 | 11.50 |
| 2P4D (2 remote D) tp = 2 | Redis | 98.88 | 12.14 | 2054.90 | 381.91 | 338.25 | 722.00 | 8.07 | 8.05 | 8.55 | 8.04 | 8.02 | 9.15 |
| | MooncakeStore (TCP) | 98.78 | 12.15 | 2057.11 | 317.42 | 304.53 | 521.66 | 8.07 | 8.03 | 8.75 | 8.04 | 8.02 | 9.62 |
| | MooncakeStore (RDMA) | 98.73 | 12.15 | 2058.02 | 275.13 | 251.57 | 487.43 | 8.18 | 8.06 | 9.19 | 8.15 | 8.05 | 10.53 |

191
doc/en/vllm-integration-v0.2.md
View File

@ -1,191 +0,0 @@
# vLLM Disaggregated Serving Demo
## Overview
This is the latest version of mooncake-transfer-engine integration doc with the vLLM project based on [PR 10502](https://github.com/vllm-project/vllm/pull/10502) and [PR 10884](https://github.com/vllm-project/vllm/pull/10884) (vllm version: v0.6.4.post1/main) to accelerate KVCache transfer for inter-node disaggregated serving scenario. We have run some experiments to obtain some [preview benchmark results](vllm-benchmark-results-v0.2.md). More benchmark results will be released in due time.
**_Please note that this is still an experimental version and will be modified anytime based on feedback from the vLLM community._**
- **Update(Apr 10, 2025)**: We are working on the vLLM v1 integration now. Stay tuned.
## Installation
### Prerequisite
```bash
pip3 install mooncake-transfer-engine
```
Note:
- If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](build.md).
- For vLLM version <= v0.8.4, it requires mooncake-transfer-engine <= 0.3.3.post2. In the latest release, interface `mooncake_vllm_adaptor` has been deprecated.
### Install the latest version of vLLM
#### 1. Clone vLLM from official repo
```bash
git clone git@github.com:vllm-project/vllm.git
```
#### 2. Build
##### 2.1 Build from source (Include C++ and CUDA code)
```bash
cd vllm
pip3 uninstall vllm -y
pip3 install -e .
```
- **If the build fails, try upgrading the version of cmake through `pip3 install cmake --upgrade`.**
- If you encounter any problems that you cannot solve, please refer to the [vLLM official compilation guide](https://docs.vllm.ai/en/v0.6.4.post1/getting_started/installation.html#install-the-latest-code).
## Configuration
### Prepare configuration file to Run Example over RDMA
- Prepare a _**mooncake.json**_ file for both Prefill and Decode instances
- **You don't need to change the `prefill_url` and `decode_url` of the config file in the decode side, please use the identical config file.**
```json
{
"prefill_url": "192.168.0.137:13003",
"decode_url": "192.168.0.139:13003",
"metadata_server": "192.168.0.139:2379",
"metadata_backend": "etcd",
"protocol": "rdma",
"device_name": "erdma_0"
}
```
- "prefill_url": The IP address and port of the Prefill node.
- The port in the URL is used to communicate with metadata server.
- "decode_url": The IP address and port of the Decode node.
- The port in the URL is used to communicate with metadata server.
- **_If you want to run the prefill instance and decode instance on the same node, please set up a different port for the `decode_url`. To avoid port conflicts, ensure that the port number differs by at least 50 from the port number in `prefill_url`. For example, "decode_url": "192.168.0.137:13103". Please note that if you set up the same URL for both instances, we will automatically add 100 to the port of the `decode_url`._**
- "metadata_server": The metadata server of the mooncake transfer engine. For examples,
- Use `etcd` as backend: `"192.168.0.137:2379"`, `"etcd://192.168.0.137:2379"` or `"etcd://192.168.0.137:2379,192.168.0.138:2379"`
- Use `redis` as backend: `"redis://192.168.0.137:6379"`
- Use `http` as backend: `"http://192.168.0.137:8080/metadata"`
- "metadata_backend": Currently we support "etcd", "redis", and "http" backends. If this parameter is absent and the `metadata_server` is not a complete URL with the backend prefix, the mooncake transfer engine will use "etcd" automatically. Please note that this parameter will be deprecated in the next version, we recommend you provide a complete URL for `metadata_server` with the prefix of the target backend.
- "protocol": The protocol to be used for data transmission. ("rdma/tcp")
- "device_name": The device to be used for data transmission, it is required when "protocol" is set to "rdma". If multiple NIC devices are used, they can be separated by commas such as "erdma_0,erdma_1". Please note that there are no spaces between them.
### Prepare configuration file to Run Example over TCP
- Prepare a _**mooncake.json**_ file for both Prefill and Decode instances
```json
{
"prefill_url": "192.168.0.137:13003",
"decode_url": "192.168.0.139:13003",
"metadata_server": "192.168.0.139:2379",
"metadata_backend": "etcd",
"protocol": "tcp",
"device_name": ""
}
```
Note: we will support auto-detect in the next version when the `protocol` is absent in the config file.
## Run Example
- Please change the IP addresses and ports in the following guide according to your env.
```bash
# Begin from `root` of your cloned repo!
# 1. Start the etcd server
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379
# You may need to terminate other etcd processes before running the above command
# 2. Run on the prefilling side (producer role)
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8100 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":2e9}'
# 3. Run on the decoding side (consumer role)
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8200 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":2e9}'
```
- `MOONCAKE_CONFIG_PATH` is the path to the mooncake.json configuration file.
- `VLLM_USE_MODELSCOPE` is optional, if you have access to huggingface, please remove it.
- The `--model` parameter specifies the model to use.
- The `--port` parameter specifies the vllm service port on which to listen.
- The `--max-model-len` parameter specifies the maximum length of the model.
- Option `--tensor_parallel_size` \ `-tp` is supported now. Example: append `-tp 2` to the run command to run vllm with multiple GPUs.
- If you want to run the prefill instance and decode instance on the same node, please set up different `CUDA_VISIBLE_DEVICES`. For example, `CUDA_VISIBLE_DEVICES=0,1` for the prefill instance and `CUDA_VISIBLE_DEVICES=2,3` for the decode instance.
- The `--kv-transfer-config` parameter specifies the connector and its config to be used.
- Please set up `kv_connector` to `MooncakeConnector`.
- `kv_role` is the node's role, either 'kv_producer' or 'kv_consumer'.
- `kv_rank` is the rank of the instance. Currently, `kv_producer`'s rank is 0, `kv_consumer`'s rank is 1.
- `kv_parallel_size` is fixed to 2 currently.
- `kv_buffer_size` is the size of the KVCache lookup buffer, if the average `input_len` of the prompt is large, please increase the buffer size. If the OOM still occurs, please decrease the ratio of `--gpu-memory-utilization`.
- `kv_ip` and `kv_port` are used to specify the IP address and port of the master node for "PyNcclConnector" distributed setup. It is not used for "MooncakeConnector" currently. Instead, "MooncakeConnector" uses a config file to set up the distributed connection. Therefore, you don't need to set these params for "MooncakeConnector" currently.
```bash
# 4. Start the proxy server on one node (Let's take the prefill node as an example)
python3 proxy_server.py
```
The implementation of `proxy_server.py`
```python
import os
import aiohttp
from quart import Quart, make_response, request
AIOHTTP_TIMEOUT = aiohttp.ClientTimeout(total=6 * 60 * 60)
app = Quart(__name__)
async def forward_request(url, data):
async with aiohttp.ClientSession(timeout=AIOHTTP_TIMEOUT) as session:
headers = {
"Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}"
}
async with session.post(url=url, json=data,
headers=headers) as response:
if response.status == 200:
if True:
async for chunk_bytes in response.content.iter_chunked(
1024):
yield chunk_bytes
else:
content = await response.read()
yield content
@app.route('/v1/completions', methods=['POST'])
async def handle_request():
try:
original_request_data = await request.get_json()
prefill_request = original_request_data.copy()
# change max_tokens = 1 to let it only do prefill
prefill_request['max_tokens'] = 1
# finish prefill
async for _ in forward_request('http://localhost:8100/v1/completions',
prefill_request):
continue
# return decode
generator = forward_request('http://192.168.0.139:8200/v1/completions', # Be sure to change the IP address for your machine
original_request_data)
response = await make_response(generator)
response.timeout = None
return response
except Exception as e:
import sys
import traceback
exc_info = sys.exc_info()
print("Error occurred in disagg prefill proxy server")
print(e)
print("".join(traceback.format_exception(*exc_info)))
if __name__ == '__main__':
app.run(host="0.0.0.0",port=8000)
```
**_Be sure to change the IP address in the code._**
## Test with openai compatible request
```
curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
"model": "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4",
"prompt": "San Francisco is a",
"max_tokens": 1000
}'
```
- If you are not testing on the proxy server, please change the `localhost` to the IP address of the proxy server.

167
doc/en/vllm-integration-v1.md
View File

@ -1,167 +0,0 @@
# vLLM Disaggregated Serving with MooncakeStore
## Overview
This is the latest version of the MooncakeStore integration doc with the vLLM project based on [PR 10502](https://github.com/vllm-project/vllm/pull/10502) and [PR 12957](https://github.com/vllm-project/vllm/pull/12957) to support KVCache transfer for intra-node and inter-node disaggregated serving scenario. Benchmark results will be released soon.
Main changes from v0.x to v1:
- XpYd support and orchestration
- dynamic changing the population of prefill group and decode group
- More stable and more fault-tolerant
- The sudden crash of a single vllm instance is tolerable
- Since instance-to-instance connections are removed, each instance works as a vanilla vllm instance, which means it can serve the requests that are not from the proxy and finish them normally
**_Please note that this is still an experimental version and will be modified anytime based on feedback from the vLLM community._**
- **Update(Apr 10, 2025)**: We are working on the vLLM v1 integration now. Stay tuned.
## Installation
### Prerequisite
```bash
pip3 install mooncake-transfer-engine
```
Note:
- If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](build.md).
- For vLLM version <= v0.8.4, it requires mooncake-transfer-engine <= 0.3.3.post2. In the latest release, interface `mooncake_vllm_adaptor` has been deprecated.
### Install the latest version of vLLM
#### 1. Clone vLLM from official repo
```bash
git clone git@github.com:vllm-project/vllm.git
```
#### 2. Build
##### 2.1 Build from source
```bash
cd vllm
pip3 install -e .
```
- If you encounter any problems that you cannot solve, please refer to the [vLLM official compilation guide](https://docs.vllm.ai/en/latest/getting_started/installation/index.html).
## Configuration
### Prepare configuration file to Run Example over RDMA
- Prepare a _**mooncake.json**_ file for both Prefill and Decode instances
```json
{
"local_hostname": "192.168.0.137",
"metadata_server": "etcd://192.168.0.137:2379",
"protocol": "rdma",
"device_name": "erdma_0",
"master_server_address": "192.168.0.137:50001"
}
```
- "local_hostname": The IP address of the current node used to communicate with the metadata server.
- **_All prefill instances and decode instances can share this config file on the same node._**
- "metadata_server": The metadata server of the mooncake transfer engine. For example,
- Use `etcd` as backend: `"192.168.0.137:2379"`, `"etcd://192.168.0.137:2379"` or `"etcd://192.168.0.137:2379,192.168.0.138:2379"`
- Use `redis` as backend: `"redis://192.168.0.137:6379"`
- Use `http` as backend: `"http://192.168.0.137:8080/metadata"`
- "protocol": The protocol to be used for data transmission. ("rdma/tcp")
- "device_name": The device to be used for data transmission, it is required when "protocol" is set to "rdma". If multiple NIC devices are used, they can be separated by commas such as "erdma_0,erdma_1". Please note that there are no spaces between them.
- "master_server_address": The IP address and the port of the master daemon process of MooncakeStore.
### Prepare configuration file to Run Example over TCP
- Prepare a _**mooncake.json**_ file for both Prefill and Decode instances
```json
{
"local_hostname": "192.168.0.137",
"metadata_server": "etcd://192.168.0.137:2379",
"protocol": "tcp",
"device_name": "",
"master_server_address": "192.168.0.137:50001"
}
```
## Run Example
- Please change the IP addresses and ports in the following guide according to your env.
```bash
# Begin from `root` of your cloned repo!
# 1. Start the etcd server
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379
# You may need to terminate other etcd processes before running the above command
# 2. Start the mooncake_master server
mooncake_master --port 50001
# If some vllm instances exit unexpectedly, some connection metadata will be corrupted since they are not properly cleaned. In that case, we recommend you restart the mooncake_master before running another test.
# 3. Run multiple vllm instances
# kv_producer role
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8100 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=1 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8101 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8102 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
CUDA_VISIBLE_DEVICES=3 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8103 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
# kv_consumer role
CUDA_VISIBLE_DEVICES=4 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8200 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=5 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8201 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=6 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8202 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
CUDA_VISIBLE_DEVICES=7 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8203 --max-model-len 10000 --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
```
- `MOONCAKE_CONFIG_PATH` is the path to the mooncake.json configuration file.
- `VLLM_USE_MODELSCOPE` is optional, if you have access to huggingface, please remove it.
- `VLLM_USE_V1=0` is required since the disaggregated feature is currently only supported on V0 vLLM.
- You can also `export` this configuration to the env, instead of putting it in front of every single command.
- The `--model` parameter specifies the model to use.
- The `--port` parameter specifies the vllm service port on which to listen.
- The `--max-model-len` parameter specifies the maximum length of the model.
- Option `--tensor_parallel_size` \ `-tp` is supported. Example: append `-tp 2` to the run command to run vllm with multiple GPUs.
- Note: All instances should have the same tensor_parallel_size.
- If you want to run the prefill instance and decode instance on the same node, please set up different `CUDA_VISIBLE_DEVICES`. For example, `CUDA_VISIBLE_DEVICES=0,1` for the prefill instance and `CUDA_VISIBLE_DEVICES=2,3` for the decode instance.
- The `--kv-transfer-config` parameter specifies the connector and its config to be used.
- Please set up `kv_connector` to `MooncakeStoreConnector`.
- `kv_role` is the node's role, either 'kv_producer', 'kv_consumer' or 'kv_both'.
```bash
# 4. Start the proxy server
cd vllm
python3 examples/online_serving/disagg_examples/disagg_proxy_demo.py --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --prefill localhost:8100 localhost:8101 --decode localhost:8200 localhost:8201 --port 8000
```
- The `--model` parameter specifies the model to use, also specifies the tokenizer used by the proxy server.
- The `--port` parameter specifies the vllm service port on which to listen.
- The `--prefill` or `-p` specifies the ip and port of the vllm prefill instances.
- The `--decode` or `-d` specifies the ip and port of the vllm decode instances.
```bash
# If you want to dynamically adjust the instances of p-nodes and d-nodes during runtime, you need to configure this environment variables.
export ADMIN_API_KEY="xxxxxxxx"
# or add it before the command:
ADMIN_API_KEY="xxxxxxxx" python3 vllm/examples/online_serving/disagg_examples/disagg_demo.py --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --prefill localhost:8100 localhost:8101 --decode localhost:8200 localhost:8201 --port 8000 --scheduling round_robin
# Then use this command to add instances into prefill group or decode group
curl -X POST "http://localhost:8000/instances/add" -H "Content-Type: application/json" -H "X-API-Key: $ADMIN_API_KEY" -d '{"type": "prefill", "instance": "localhost:8102"}'
curl -X POST "http://localhost:8000/instances/add" -H "Content-Type: application/json" -H "X-API-Key: $ADMIN_API_KEY" -d '{"type": "prefill", "instance": "localhost:8103"}'
curl -X POST "http://localhost:8000/instances/add" -H "Content-Type: application/json" -H "X-API-Key: $ADMIN_API_KEY" -d '{"type": "decode", "instance": "localhost:8202"}'
curl -X POST "http://localhost:8000/instances/add" -H "Content-Type: application/json" -H "X-API-Key: $ADMIN_API_KEY" -d '{"type": "decode", "instance": "localhost:8203"}'
# Use this command to get the proxy status
curl localhost:8000/status | jq
```
Mooncake team implements this simple disagg_proxy based on round-robin as a demo. In the production stage, service providers and users can also implement corresponding global proxy strategies according to their needs.
**_Be sure to change the IP address in the commands._**
## Test with openai compatible request
```
curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
"model": "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4",
"prompt": "San Francisco is a",
"max_tokens": 1000
}'
```
- If you are not testing on the proxy server, please change the `localhost` to the IP address of the proxy server.

0
doc/en/ep-backend.md → docs/source/design/ep-backend.md
View File

29
docs/source/design/mooncake-store.md
View File

@ -607,6 +607,12 @@ When the user specifies `--root_fs_dir=/path/to/dir` when starting the master, a
​Note​​: When enabling this feature, the user must ensure that the DFS-mounted directory (`root_fs_dir=/path/to/dir`) is valid and consistent across all client hosts. If some clients have invalid or incorrect mount paths, it may cause abnormal behavior in Mooncake Store.
#### Persistent Storage Space Configuration​
Mooncake provides configurable DFS available space. Users can specify `--global_file_segment_size=1048576` when starting the master, indicating a maximum usable space of 1MB on DFS.
The current default setting is the maximum value of int64 (as we generally do not restrict DFS storage usage), which is displayed as `infinite` in `mooncake_maseter`'s console logs.
**Notice** The DFS cache space configuration must be used together with the `--root_fs_dir` parameter. Otherwise, you will observe that the `SSD Storage` usage consistently shows: `0 B / 0 B`
**Notice** The capability for file eviction on DFS has not been provided yet
#### Data Access Mechanism
The persistence feature also follows Mooncake Store's design principle of separating control flow from data flow. The read/write operations of kvcache objects are completed on the client side, while the query and management functions of kvcache objects are handled on the master side. In the file system, the key -> kvcache object index information is maintained by a fixed indexing mechanism, with each file corresponding to one kvcache object (the filename serves as the associated key name).
@ -619,6 +625,29 @@ After enabling the persistence feature:
#### 3FS USRBIO Plugin
If you need to use 3FS's native API (USRBIO) to achieve high-performance persistent file reads and writes, you can refer to the configuration instructions in this document [3FS USRBIO Plugin](https://kvcache-ai.github.io/Mooncake/getting_started/plugin-usage/3FS-USRBIO-Plugin.html).
### Builtin Metadata Server
Mooncake Store provides a built-in HTTP metadata server as an alternative to etcd for storing cluster metadata. This feature is particularly useful for development environments or scenarios where etcd is not available.
#### Configuration Parameters
The HTTP metadata server can be configured using the following parameters:
- **`enable_http_metadata_server`** (boolean, default: `false`): Enables the built-in HTTP metadata server instead of using etcd. When set to `true`, the master service will start an embedded HTTP server that handles metadata operations.
- **`http_metadata_server_port`** (integer, default: `8080`): Specifies the TCP port on which the HTTP metadata server will listen for incoming connections. This port must be available and not conflict with other services.
- **`http_metadata_server_host`** (string, default: `"0.0.0.0"`): Specifies the host address for the HTTP metadata server to bind to. Use `"0.0.0.0"` to listen on all available network interfaces, or specify a specific IP address for security purposes.
#### Environment Variables
- MC_STORE_CLUSTER_ID: Identify the metadata when multiple cluster share the same master, default 'mooncake'.
- MC_STORE_MEMCPY: Enables or disables local memcpy optimization, set to 1/true to enable, 0/false to disable.
- MC_STORE_CLIENT_METRIC: Enables client metric reporting, enabled by default; set to 0/false to disable.
- MC_STORE_CLIENT_METRIC_INTERVAL: Reporting interval in seconds, default 0 (collects but does not report).
#### Usage Example
To start the master service with the HTTP metadata server enabled:
```bash
./build/mooncake-store/src/mooncake_master \
--enable_http_metadata_server=true \
--http_metadata_server_port=8080 \
--http_metadata_server_host=0.0.0.0
```
When enabled, the HTTP metadata server will start automatically and provide metadata services for the Mooncake Store cluster. This eliminates the need for an external etcd deployment, simplifying the setup process for development and testing environments.
Note that the HTTP metadata server is designed for single-node deployments and does not provide the high availability features that etcd offers. For production environments requiring high availability, etcd is still the recommended choice.
## Mooncake Store Python API
**Complete Python API Documentation**: [https://kvcache-ai.github.io/Mooncake/python-api-reference/mooncake-store.html](https://kvcache-ai.github.io/Mooncake/python-api-reference/mooncake-store.html)

435
docs/source/design/transfer-engine.md
View File

@ -1,435 +0,0 @@
# Transfer Engine
## Overview
Mooncake Transfer Engine is a high-performance, zero-copy data transfer library designed around two core abstractions: Segment and BatchTransfer.
- [**Segment**](#segment) represents a contiguous address space that can be remotely read and written, which can be either non-persistent storage provided by DRAM or VRAM, known as **RAM Segment**, or persistent storage provided by NVMeof, known as **NVMeof Segment**.
- [**BatchTransfer**](#batchtransfer) encapsulates operation requests, specifically responsible for synchronizing data between a set of non-contiguous data spaces in one Segment and the corresponding spaces in another set of Segments, supporting Read/Write in both directions, thus acting like an asynchronous and more flexible AllScatter/AllGather.
![transfer_engine](../image/transfer-engine.png)
As shown in the diagram, each specific client corresponds to a `TransferEngine`, which not only includes a RAM Segment but also integrates management for high-speed transfers across multiple threads and network cards. The RAM Segment, in principle, corresponds to the entire virtual address space of this `TransferEngine`, but in reality, only parts of it (known as a `Buffer`) are registered for (GPUDirect) RDMA Read/Write. Each Buffer can have separate permissions (corresponding to RDMA `rkey`, etc.) and network card affinity (e.g., preferred NICs for different types of memory).
Mooncake Transfer Engine provides interfaces through the `TransferEngine` class (located in `mooncake-transfer-engine/include/transfer_engine.h`), where the specific data transfer functions for different backends are implemented by the `Transport` class, currently supporting `TcpTransport`, `RdmaTransport`, and `NVMeoFTransport`.
### Segment
Segment represents a collection of source address ranges and target address ranges available during the data transfer process in Transfer Engine. That is, all local and remote addresses involved in `BatchTransfer` requests must be within the valid segment range. Transfer Engine supports the following two types of Segments.
#### 1. RAM Segment in Memory Address Space (DRAM, VRAM)
When each process starts, Transfer Engine automatically creates a segment named after its own `local_hostname` (see the initialization function of `TransferEngine`, which needs to be globally unique), which logically covers the entire memory address space, including storage media such as DRAM/VRAM. When using the `BatchTransfer` interface for transfer tasks, Transfer Engine automatically determines the corresponding hardware information to choose the best transfer method. **Each process has and only has one segment.** Other processes can reference the segment and complete read/write operations by calling the `openSegment` interface and passing the correct name.
In actual deployment, application systems usually only use part of the memory address space for data transfer, so Transfer Engine further divides the segment into multiple buffers internally. Each Buffer represents a contiguous address space located on the same device, and when users use the `BatchTransfer` interface to complete read/write operations, if referencing a RAM Segment, each read/write task must be within one of the valid buffers.
The memory range within a segment does not need to be contiguous, which means that multiple DRAM/VRAM address spaces can be allocated and included in the same segment.
In addition, Transfer Engine also supports registering some **local DRAM areas**, which are merely used as the local side storage space for data operations, such as the DRAM PageCache area of vLLM. It is also considered a part of the effective RAM Segment in the current process but cannot be referenced by other processes by calling the `openSegment` interface.
#### 2. NVMeof Segment
Transfer Engine also leverages the NVMeof protocol to support direct data transfer from files on NVMe to DRAM/VRAM via PCIe, without going through the CPU and achieving zero-copy. Users need to follow the instructions to mount remote storage nodes locally and use the `openSegment` interface for reference to complete data read/write operations.
### BatchTransfer
With the help of Transfer Engine, Mooncake Store can achieve local DRAM/VRAM reading and writing of specified parts in valid segments through TCP, (GPUDirect) RDMA, NVMe-of protocols, etc.
| Remote ↓ Local → | DRAM | VRAM |
|----------|------|------|
| DRAM | ✓ | ✓ |
| VRAM | ✓ | ✓ |
| NVMe-of | ✓ | ✓ |
- Local memcpy: If the target Segment is actually in the local DRAM/VRAM, direct data copy interfaces such as memcpy, cudaMemcpy are used.
- TCP: Supports data transfer between local DRAM and remote DRAM.
- RDMA: Supports data transfer between local DRAM/VRAM and remote DRAM. It supports multi-network card pooling and retry functions in implementation.
- cuFile (GPUDirect Storage): Implements data transfer between local DRAM/VRAM and Local/Remote NVMeof.
The BatchTransfer API uses an array of requests, which specify the operation type (READ or WRITE), data length, and local and remote memory addresses. The transfer operation is applicable to DRAM and GPU VRAM. The completion of these operations can be asynchronously monitored through the `getTransferStatus` API.
### Topology Aware Path Selection
Modern inference servers often consist of multiple CPU sockets, DRAM, GPUs, and RDMA NIC devices. Although it's technically possible to transfer data from local DRAM or VRAM to a remote location using any RDMA NIC, these transfers can be limited by the bandwidth constraints of the Ultra Path Interconnect (UPI) or PCIe Switch. To overcome these limitations, Transfer Engine implements a topology-aware path selection algorithm.
Before processing requests, each server generates a topology matrix and broadcasts it across the cluster.
This matrix categorizes network interface cards (NICs) into preferred and secondary lists for various types of memory, which types are specified during memory registration.
Under normal conditions, a NIC from the preferred list is selected for transfers, facilitating RDMA operations within the local NUMA or GPU Direct RDMA through the local PCIe switch only.
In case of failures, NICs from both lists may be utilized.
The process involves identifying the appropriate local and target NICs based on the memory addresses, establishing a connection, and executing the data transfer.
![topology-matrix](../image/topology-matrix.png)
For instance, as illustrated in figure above, to transfer data from buffer 0 (assigned to cpu:0) in the local node to buffer 1 (assigned to cpu:1) in the target node, the engine first identifies the preferred NICs for cpu:0 using the local server's topology matrix and selects one, such as mlx5_1, as the local NIC. Similarly, the target NIC, such as mlx5_3, is selected based on the target memory address. This setup enables establishing an RDMA connection from mlx5_1@local to mlx5_3@target to carry out RDMA read and write operations.
To further maximize bandwidth utilization, if a single request's transfer is internally divided into multiple slices if its length exceeds 64KB.
Each slice might use a different path, enabling collaborative work among all RDMA NICs.
### Endpoint Management
Mooncake Store employs a pair of endpoints to represent the connection between a local RDMA
NIC and a remote RDMA NIC. In practice, each endpoint
includes one or more RDMA queue pair objects. Connections in Mooncake Store are established in an on demand manner;
endpoints remain unpaired until the first request is made.
To prevent a large number of endpoints from slowing down
request processing, Mooncake Store employs endpoint pooling,
which caps the maximum number of active connections. We
use the SIEVE algorithm to manage endpoint eviction.
If a connection fails due to link errors, it is removed from the
endpoint pools on both sides and re-established during the
next data transfer attempt.
### Fault Handling
In a multi-NIC environment, one common failure scenario is the temporary unavailability of a specific NIC, while other routes may still connect two nodes.
Mooncake Store is designed to adeptly manage such temporary
failures effectively. If a connection is identified as unavailable,
Mooncake Store automatically identifies an alternative, reachable
path and resubmits the request to a different RDMA NIC
device. Furthermore, Mooncake Store is capable of detecting problems with other RDMA resources, including RDMA contexts
and completion queues. It temporarily avoids using these
resources until the issue, such as a downed link, is resolved.
## Example: Transfer Engine Bench
The sample program provided in `mooncake-transfer-engine/example/transfer_engine_bench.cpp` demonstrates the basic usage of Transfer Engine by repeatedly reading/writing data blocks from the DRAM of the target node to the initiator node through the Transfer Engine interface. It can also be used to measure read and write throughput. Currently, the Transfer Engine Bench tool supports RDMA and TCP protocols.
After successfully compiling Transfer Engine, the test program `transfer_engine_bench` can be found in the `build/mooncake-transfer-engine/example` directory.
1. **Start the `metadata` service.** This service is used for the centralized highly available management of various metadata for Mooncake, including the internal connection status of Transfer Engine. It is necessary to ensure that both the initiator and target nodes can smoothly access this metadata service, so pay attention to:
- The listening IP of the metadata service should not be 127.0.0.1; it should be determined in conjunction with the network environment. In the experimental environment, 0.0.0.0 can be used.
- On some platforms, if the initiator and target nodes have set the `http_proxy` or `https_proxy` environment variables, it will also affect the communication between Transfer Engine and the metadata service.
Transfer Engine support multiple kinds of metadata services, including `etcd`, `redis`, and `http`. The following describes how to start the metadata service using `etcd` and `http` as examples.
1.1. **`etcd`**
For example, the following command line can be used to start the etcd service:
```bash
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://10.0.0.1:2379
```
1.2. **`http`**
For example, you can use the `http` service in the `mooncake-transfer-engine/example/http-metadata-server` example:
```bash
# cd mooncake-transfer-engine/example/http-metadata-server
go run . --addr=:8080
```
2. **Start the target node.**
```bash
./transfer_engine_bench --mode=target \
--metadata_server=etcd://10.0.0.1:2379 \
[--local_server_name=TARGET_NAME] \
[--device_name=erdma_0 | --auto-discovery]
```
The meanings of the various parameters are as follows:
- `--mode=target` indicates the start of the target node. The target node does not initiate read/write requests; it passively supplies or writes data as required by the initiator node.
> [!NOTE]
> In actual applications, there is no need to distinguish between target nodes and initiator nodes; each node can freely initiate read/write requests to other nodes in the cluster.
- `--metadata_server` is the address of the metadata server. Its form is `[proto]://[hostname:port]`. For example, the following addresses are VALID:
- Use `etcd` as metadata storage: `"10.0.0.1:2379"`, `"etcd://10.0.0.1:2379"` or `"etcd://10.0.0.1:2379,10.0.0.2:2379"`
- Use `redis` as metadata storage: `"redis://10.0.0.1:6379"`
- Use `http` as metadata storage: `"http://10.0.0.1:8080/metadata"`
- `--local_server_name` represents the segment name of current node, which does not need to be set in most cases. If this option is not set, the value is equivalent to the hostname of this machine (i.e., `hostname(2)`). This should keep unique among the cluster.
- `--device_name` indicates the name of the RDMA network card used in the transfer process (separated by commas without space). You can also specify `--auto_discovery` to enable discovery topology automatically, which generates a network card priority matrix based on the operating system configuration.
- In network environments that only support TCP, the `--protocol=tcp` parameter can be used; in this case, there is no need to specify the `--device_name` parameter.
1. **Start the initiator node.**
```bash
./transfer_engine_bench --metadata_server=etcd://10.0.0.1:2379 \
--segment_id=TARGET_NAME \
[--local_server_name=INITIATOR_NAME] \
[--device_name=erdma_1 | --auto-discovery]
```
The meanings of the various parameters are as follows (the rest are the same as before):
- `--segment_id` is the segment name of target node. It needs to be consistent with the value passed to `--local_server_name` when starting the target node (if any).
Under normal circumstances, the initiator node will start the transfer operation, wait for 10 seconds, and then display the "Test completed" message, indicating that the test is complete.
The initiator node can also configure the following test parameters: `--operation` (can be `"read"` or `"write"`), `batch_size`, `block_size`, `duration`, `threads`, etc.
> [!NOTE]
> If an exception occurs during execution, it is usually due to incorrect parameter settings. It is recommended to refer to the [troubleshooting document](troubleshooting.md) for preliminary troubleshooting.
### Sample Run
The following video shows a normal run as described above, with the Target on the right and the Initiator on the left, at the end of the test the Initiator reports the test duration (10 seconds), IOPS (379008 requests/s), and throughput (19.87 GiB/s). The throughput here exceeds the maximum throughput supported by a single card on the host computer used.
![transfer-engine-running](../image/transfer-engine-running.gif)
## Transfer Engine C/C++ API
Transfer Engine provides interfaces through the `TransferEngine` class (located in `mooncake-transfer-engine/include/transfer_engine.h`), where the specific data transfer functions for different backends are implemented by the `Transport` class, currently supporting `TcpTransport`, `RdmaTransport` and `NVMeoFTransport`.
### Data Transfer
#### TransferEngine::TransferRequest
The core API provided by Mooncake Transfer Engine is submitting a group of asynchronous `TransferRequest` tasks through the `submitTransfer` interface, and querying their status through the `getTransferStatus` interface. Each `TransferRequest` specifies reading or writing a continuous data space of `length` starting from the local starting address `source`, to the position starting at `target_offset` in the segment corresponding to `target_id`.
The `TransferRequest` structure is defined as follows:
```cpp
using SegmentID = int32_t;
struct TransferRequest
{
enum OpCode { READ, WRITE };
OpCode opcode;
void *source;
SegmentID target_id; // The ID of the target segment, which may correspond to local or remote DRAM/VRAM/NVMeof, with the specific routing logic hidden
size_t target_offset;
size_t length;
};
```
- `opcode` takes the values `READ` or `WRITE`. `READ` indicates that data is copied from the target address indicated by `<target_id, target_offset>` to the local starting address `source`; `WRITE` indicates that data is copied from `source` to the address indicated by `<target_id, target_offset>`.
- `source` represents the DRAM/VRAM buffer managed by the current `TransferEngine`, which must have been registered in advance by the `registerLocalMemory` interface.
- `target_id` represents the segment ID of the transfer target. The segment ID is obtained using the `openSegment` interface. Segments are divided into the following types:
- RAM space type, covering DRAM/VRAM. As mentioned earlier, there is only one segment under the same process (or `TransferEngine` instance), which contains various types of Buffers (DRAM/VRAM). In this case, the segment name passed to the `openSegment` interface is equivalent to the server hostname. `target_offset` is the virtual address of the target server.
- NVMeOF space type, where each file corresponds to a segment. In this case, the segment name passed to the `openSegment` interface is equivalent to the unique identifier of the file. `target_offset` is the offset of the target file.
- `length` represents the amount of data transferred. TransferEngine may further split this into multiple read/write requests internally.
#### TransferEngine::allocateBatchID
```cpp
BatchID allocateBatchID(size_t batch_size);
```
Allocates a `BatchID`. A maximum of `batch_size` `TransferRequest`s can be submitted under the same `BatchID`.
- `batch_size`: The maximum number of `TransferRequest`s that can be submitted under the same `BatchID`;
- Return value: If successful, returns `BatchID` (non-negative); otherwise, returns a negative value.
#### TransferEngine::submitTransfer
```cpp
int submitTransfer(BatchID batch_id, const std::vector<TransferRequest> &entries);
```
Submits new `TransferRequest` tasks to `batch_id`. The task is asynchronously submitted to the background thread pool. The total number of `entries` accumulated under the same `batch_id` should not exceed the `batch_size` defined at creation.
- `batch_id`: The `BatchID` it belongs to;
- `entries`: Array of `TransferRequest`;
- Return value: If successful, returns 0; otherwise, returns a negative value.
#### TransferEngine::getTransferStatus
```cpp
enum TaskStatus
{
WAITING, // In the transfer phase
PENDING, // Not supported
INVALID, // Ilvalid parameters
CANCELED, // Not supported
COMPLETED, // Transfer completed
TIMEOUT, // Not supported
FAILED // Transfer failed even after retries
};
struct TransferStatus {
TaskStatus s;
size_t transferred; // How much data has been successfully transferred (not necessarily an accurate value, but it is a lower bound)
};
int getTransferStatus(BatchID batch_id, size_t task_id, TransferStatus &status)
```
Obtains the running status of the `TransferRequest` with `task_id` in `batch_id`.
- `batch_id`: The `BatchID` it belongs to;
- `task_id`: The sequence number of the `TransferRequest` to query;
- `status`: Output Transfer status;
- Return value: If successful, returns 0; otherwise, returns a negative value.
#### TransferEngine::freeBatchID
```cpp
int freeBatchID(BatchID batch_id);
```
Recycles `BatchID`, and subsequent operations on `submitTransfer` and `getTransferStatus` are undefined. If there are still `TransferRequest`s pending completion in the `BatchID`, the operation is refused.
- `batch_id`: The `BatchID` it belongs to;
- Return value: If successful, returns 0; otherwise, returns a negative value.
### Multi-Transport Management
The `TransferEngine` class internally manages multiple backend `Transport` classes.
And it will discover the toplogy between CPU/CUDA and RDMA devices automatically
(more device types are working in progress, feedbacks are welcome when the automatic discovery mechanism is not accurate),
and it will install `Transport` automatically based on the topology.
### Space Registration
For the RDMA transfer process, the source pointer `TransferRequest::source` must be registered in advance as an RDMA readable/writable Memory Region space, that is, included as part of the RAM Segment of the current process. Therefore, the following functions are needed:
#### TransferEngine::registerLocalMemory
```cpp
int registerLocalMemory(void *addr, size_t size, string location, bool remote_accessible);
```
Registers a space starting at address `addr` with a length of `size` on the local DRAM/VRAM.
- `addr`: The starting address of the registration space;
- `size`: The length of the registration space;
- `location`: The `device` corresponding to this memory segment, such as `cuda:0` indicating the GPU device, `cpu:0` indicating the CPU socket, by matching with the network card priority order table (see `installTransport`), the preferred network card is identified. You can also use `*`, Transfer Engine will try to automatically recognize the `device` corresponding to `addr`, if it fails to recognize the device, it will print a `WARNING` level log and use all network cards, no preferred network cards.
- `remote_accessible`: Indicates whether this memory can be accessed by remote nodes.
- Return value: If successful, returns 0; otherwise, returns a negative value.
#### TransferEngine::unregisterLocalMemory
```cpp
int unregisterLocalMemory(void *addr);
```
Unregisters the region.
- `addr`: The starting address of the registration space;
- Return value: If successful, returns 0; otherwise, returns a negative value.
### Segment Management and Metadata Format
TransferEngine provides the `openSegment` function, which obtains a `SegmentHandle` for subsequent `Transport` transfers.
```cpp
SegmentHandle openSegment(const std::string& segment_name);
```
- `segment_name`: The unique identifier of the segment. For RAM Segment, this needs to be consistent with the `server_name` filled in by the peer process when initializing the TransferEngine object.
- Return value: If successful, returns the corresponding `SegmentHandle`; otherwise, returns a negative value.
```cpp
int closeSegment(SegmentHandle segment_id);
```
- `segment_id`: The unique identifier of the segment.
- Return value: If successful, returns 0; otherwise, returns a negative value.
<details>
<summary><strong>Metadata Format</strong></summary>
```
// Used to find the communicable address and exposed rpc port based on server_name.
// Created: when calling TransferEngine::init().
// Deleted: when TransferEngine is destructed.
Key = mooncake/rpc_meta/[server_name]
Value = {
'ip_or_host_name': 'node01'
'rpc_port': 12345
}
// For segments, the key naming method of mooncake/[proto]/[segment_name] is used, and the segment name can use the Server Name.
// A segment corresponds to a machine, and a buffer corresponds to different segments of memory or different files or different disks on the machine. Different buffers of the same segment are in the same fault domain.
// RAM Segment, used by RDMA Transport to obtain transfer information.
// Created: command line tool register.py, at this time buffers are empty, only fill in the information that can be known in advance.
// Modified: TransferEngine at runtime through register / unregister to add or delete Buffer.
Key = mooncake/ram/[segment_name]
Value = {
'server_name': server_name,
'protocol': rdma,
'devices': [
{ 'name': 'mlx5_2', 'lid': 17, 'gid': 'fe:00:...' },
{ 'name': 'mlx5_3', 'lid': 22, 'gid': 'fe:00:...' }
],
'priority_matrix': {
"cpu:0": [["mlx5_2"], ["mlx5_3"]],
"cpu:1": [["mlx5_3"], ["mlx5_2"]],
"cuda:0": [["mlx5_2"], ["mlx5_3"]],
},
'buffers': [
{
'name': 'cpu:0',
'addr': 0x7fa16bdf5000,
'length': 1073741824,
'rkey': [1fe000, 1fdf00, ...], // The length is the same as the number of elements in the 'devices' field
},
],
}
// Created: command line tool register.py, determine the file path that can be mounted.
// Modified: command line tool mount.py, add a mapping of the machine mounting the file to the file path on the mounting machine to the buffers.local_path_map.
Key = mooncake/nvmeof/[segment_name]
Value = {
'server_name': server_name,
'protocol': nvmeof,
'buffers':[
{
'length': 1073741824,
'file_path': "/mnt/nvme0" // The file path on this machine
'local_path_map': {
"node01": "/mnt/transfer_engine/node01/nvme0", // The machine mounting the file -> The file path on the mounting machine
.....
},
}，
{
'length': 1073741824,
'file_path': "/mnt/nvme1",
'local_path_map': {
"node02": "/mnt/transfer_engine/node02/nvme1",
.....
},
}
]
}
```
</details>
### HTTP Metadata Server
The HTTP server should implement three following RESTful APIs, while the metadata server configured to `http://host:port/metadata` as an example:
1. `GET /metadata?key=$KEY`: Get the metadata corresponding to `$KEY`.
2. `PUT /metadata?key=$KEY`: Update the metadata corresponding to `$KEY` to the value of the request body.
3. `DELETE /metadata?key=$KEY`: Delete the metadata corresponding to `$KEY`.
For specific implementation, refer to the demo service implemented in Golang at [mooncake-transfer-engine/example/http-metadata-server](../../../mooncake-transfer-engine/example/http-metadata-server).
### Initialization
TransferEngine needs to initializing by calling the `init` method before further actions:
```cpp
TransferEngine();
int init(const std::string &metadata_conn_string,
const std::string &local_server_name);
```
- `metadata_conn_string`: Connecting string of metadata storage servers, i.e., the IP address/hostname of `etcd`/`redis` or the URI of the http service.
The general form is `[proto]://[hostname:port]`. For example, the following metadata server addresses are legal:
- Using `etcd` as a metadata storage service: `“10.0.0.1:2379”` or `“etcd://10.0.0.1:2379”`.
- Using `redis` as a metadata storage service: `“redis://10.0.0.1:6379”`
- Using `http` as a metadata storage service: `“http://10.0.0.1:8080/metadata”`
- `local_server_name`: The local server name, ensuring uniqueness within the cluster. It also serves as the name of the RAM Segment that other nodes refer to the current instance (i.e., Segment Name).
```cpp
~TransferEngine();
```
Reclaims all allocated resources and also deletes the global meta data server information.
## Using Transfer Engine to Your Projects
### Using C/C++ Interface
After compiling Mooncake Store, you can move the compiled static library file `libtransfer_engine.a` and the C header file `transfer_engine_c.h` into your own project. There is no need to reference other files under `src/transfer_engine`.
### Using Golang Interface
To support the operational needs of P2P Store, Transfer Engine provides a Golang interface wrapper, see `mooncake-p2p-store/src/p2pstore/transfer_engine.go`.
When compiling the project, enable the `-DWITH_P2P_STORE=ON` option to compile the P2P Store example program at the same time.
### Using Rust Interface
Under `mooncake-transfer-engine/rust`, the Rust interface implementation of TransferEngine is provided, and a Rust version of the benchmark is implemented based on the interface, similar to [transfer_engine_bench.cpp](../../../mooncake-transfer-engine/example/transfer_engine_bench.cpp). To compile the rust example, you need to install the Rust SDK and add `-DWITH_RUST_EXAMPLE=ON` in the cmake command.
## Advanced Runtime Options
For advanced users, TransferEngine provides the following advanced runtime options, all of which can be passed in through **environment variables**.
- `MC_NUM_CQ_PER_CTX` The number of CQs created per device instance, default value 1
- `MC_NUM_COMP_CHANNELS_PER_CTX` The number of Completion Channel created per device instance, default value 1
- `MC_IB_PORT` The IB port number used per device instance, default value 1
- `MC_GID_INDEX` The GID index used per device instance, default value 3 (or the maximum value supported by the platform)
- `MC_MAX_CQE_PER_CTX` The CQ buffer size per device instance, default value 4096
- `MC_MAX_EP_PER_CTX` The maximum number of active EndPoint per device instance, default value 256
- `MC_NUM_QP_PER_EP` The number of QPs per EndPoint, the more the number, the better the fine-grained I/O performance, default value 2
- `MC_MAX_SGE` The maximum number of SGEs supported per QP, default value 4 (or the highest value supported by the platform)
- `MC_MAX_WR` The maximum number of Work Request supported per QP, default value 256 (or the highest value supported by the platform)
- `MC_MAX_INLINE` The maximum Inline write data volume (bytes) supported per QP, default value 64 (or the highest value supported by the platform)
- `MC_MTU` The MTU length used per device instance, can be 512, 1024, 2048, 4096, default value 4096 (or the maximum length supported by the platform)
- `MC_WORKERS_PER_CTX` The number of asynchronous worker threads corresponding to each device instance
- `MC_SLICE_SIZE` The segmentation granularity of user requests in Transfer Engine
- `MC_RETRY_CNT` The maximum number of retries in Transfer Engine
- `MC_LOG_LEVEL` This option can be set as `TRACE`/`INFO`/`WARNING`/`ERROR` (see [glog doc](https://github.com/google/glog/blob/master/docs/logging.md)), and more detailed logs will be output during runtime
- `MC_DISABLE_METACACHE` Disable local meta cache to prevent transfer failure due to dynamic memory registrations, which may downgrades the performance
- `MC_HANDSHAKE_LISTEN_BACKLOG` The backlog size of socket listening for handshaking, default value is 128

0
doc/en/ascend_direct_transport.md → docs/source/design/transfer-engine/ascend_direct_transport.md
View File

0
doc/en/ascend_transport.md → docs/source/design/transfer-engine/ascend_transport.md
View File

0
doc/en/heterogeneous_ascend.md → docs/source/design/transfer-engine/heterogeneous_ascend.md
View File

29
doc/en/transfer-engine.md → docs/source/design/transfer-engine/index.md
View File

@ -7,7 +7,7 @@ Mooncake Transfer Engine is a high-performance, zero-copy data transfer library
- [**BatchTransfer**](#batchtransfer) encapsulates operation requests, specifically responsible for synchronizing data between a set of non-contiguous data spaces in one Segment and the corresponding spaces in another set of Segments, supporting Read/Write in both directions, thus acting like an asynchronous and more flexible AllScatter/AllGather.
![transfer_engine](../../image/transfer-engine.png)
![transfer_engine](../image/transfer-engine.png)
As shown in the diagram, each specific client corresponds to a `TransferEngine`, which not only includes a RAM Segment but also integrates management for high-speed transfers across multiple threads and network cards. The RAM Segment, in principle, corresponds to the entire virtual address space of this `TransferEngine`, but in reality, only parts of it (known as a `Buffer`) are registered for (GPUDirect) RDMA Read/Write. Each Buffer can have separate permissions (corresponding to RDMA `rkey`, etc.) and network card affinity (e.g., preferred NICs for different types of memory).
@ -54,7 +54,7 @@ Under normal conditions, a NIC from the preferred list is selected for transfers
In case of failures, NICs from both lists may be utilized.
The process involves identifying the appropriate local and target NICs based on the memory addresses, establishing a connection, and executing the data transfer.
![topology-matrix](../../image/topology-matrix.png)
![topology-matrix](../image/topology-matrix.png)
For instance, as illustrated in figure above, to transfer data from buffer 0 (assigned to cpu:0) in the local node to buffer 1 (assigned to cpu:1) in the target node, the engine first identifies the preferred NICs for cpu:0 using the local server's topology matrix and selects one, such as mlx5_1, as the local NIC. Similarly, the target NIC, such as mlx5_3, is selected based on the target memory address. This setup enables establishing an RDMA connection from mlx5_1@local to mlx5_3@target to carry out RDMA read and write operations.
@ -151,7 +151,7 @@ After successfully compiling Transfer Engine, the test program `transfer_engine_
The following video shows a normal run as described above, with the Target on the right and the Initiator on the left, at the end of the test the Initiator reports the test duration (10 seconds), IOPS (379008 requests/s), and throughput (19.87 GiB/s). The throughput here exceeds the maximum throughput supported by a single card on the host computer used.
![transfer-engine-running](../../image/transfer-engine-running.gif)
![transfer-engine-running](../image/transfer-engine-running.gif)
## Transfer Engine C/C++ API
Transfer Engine provides interfaces through the `TransferEngine` class (located in `mooncake-transfer-engine/include/transfer_engine.h`), where the specific data transfer functions for different backends are implemented by the `Transport` class, currently supporting `TcpTransport`, `RdmaTransport` and `NVMeoFTransport`.
@ -376,7 +376,7 @@ The HTTP server should implement three following RESTful APIs, while the metadat
2. `PUT /metadata?key=$KEY`: Update the metadata corresponding to `$KEY` to the value of the request body.
3. `DELETE /metadata?key=$KEY`: Delete the metadata corresponding to `$KEY`.
For specific implementation, refer to the demo service implemented in Golang at [mooncake-transfer-engine/example/http-metadata-server](../../mooncake-transfer-engine/example/http-metadata-server).
For specific implementation, refer to the demo service implemented in Golang at [mooncake-transfer-engine/example/http-metadata-server](../../../mooncake-transfer-engine/example/http-metadata-server).
### Initialization
@ -402,6 +402,7 @@ The general form is `[proto]://[hostname:port]`. For example, the following meta
Reclaims all allocated resources and also deletes the global meta data server information.
## Using Transfer Engine to Your Projects
### Using C/C++ Interface
After compiling Mooncake Store, you can move the compiled static library file `libtransfer_engine.a` and the C header file `transfer_engine_c.h` into your own project. There is no need to reference other files under `src/transfer_engine`.
@ -411,7 +412,7 @@ To support the operational needs of P2P Store, Transfer Engine provides a Golang
When compiling the project, enable the `-DWITH_P2P_STORE=ON` option to compile the P2P Store example program at the same time.
### Using Rust Interface
Under `mooncake-transfer-engine/rust`, the Rust interface implementation of TransferEngine is provided, and a Rust version of the benchmark is implemented based on the interface, similar to [transfer_engine_bench.cpp](../../mooncake-transfer-engine/example/transfer_engine_bench.cpp). To compile the rust example, you need to install the Rust SDK and add `-DWITH_RUST_EXAMPLE=ON` in the cmake command.
Under `mooncake-transfer-engine/rust`, the Rust interface implementation of TransferEngine is provided, and a Rust version of the benchmark is implemented based on the interface, similar to [transfer_engine_bench.cpp](../../../mooncake-transfer-engine/example/transfer_engine_bench.cpp). To compile the rust example, you need to install the Rust SDK and add `-DWITH_RUST_EXAMPLE=ON` in the cmake command.
## Advanced Runtime Options
For advanced users, TransferEngine provides the following advanced runtime options, all of which can be passed in through **environment variables**.
@ -445,3 +446,21 @@ For advanced users, TransferEngine provides the following advanced runtime optio
- `MC_MAX_PRC_PORT` Specifies the maximum port number for RPC service. The default value is 17000.
- `MC_PATH_ROUNDROBIN` Use round-robin mode in the RDMA path selection. This may be beneficial for transferring large bulks.
- `MC_ENDPOINT_STORE_TYPE` Choose FIFO Endpoint Store (`FIFO`) or Sieve Endpoint Store (`SIEVE`), default is `SIEVE`.
## Ascend Transport Component
:::{toctree}
:maxdepth: 1
ascend_direct_transport
ascend_transport
heterogeneous_ascend
:::
## Benchmark and Tuning Guide
:::{toctree}
:maxdepth: 1
transfer-engine-bench-tuning
:::

0
doc/en/transfer-engine-bench-tuning.md → docs/source/design/transfer-engine/transfer-engine-bench-tuning.md
View File

15
docs/source/getting_started/build.md
View File

@ -23,7 +23,7 @@ pip install mooncake-transfer-engine-non-cuda
### Recommended Version
- OS: Ubuntu 22.04 LTS+
- cmake: 3.16.x
- cmake: 3.20.x
- gcc: 9.4+
### Steps
@ -106,7 +106,14 @@ pip install mooncake-transfer-engine-non-cuda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
```
3. Install yalantinglibs
3. If you want to compile the Moore Mthreads GPUDirect support module, first follow the instructions in https://docs.mthreads.com/musa-sdk/musa-sdk-doc-online/install_guide to install MUSA. After that:
1) Install `mthreads-peermem` for enabling GPU-Direct RDMA
2) Configure `LIBRARY_PATH` and `LD_LIBRARY_PATH` to ensure linking of `musart`, and other libraries during compilation:
```bash
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/musa/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/musa/lib
4. Install yalantinglibs
```bash
git clone https://github.com/alibaba/yalantinglibs.git
cd yalantinglibs
@ -116,7 +123,7 @@ pip install mooncake-transfer-engine-non-cuda
make install
```
4. In the root directory of this project, run the following commands:
5. In the root directory of this project, run the following commands:
```bash
mkdir build
cd build
@ -124,7 +131,7 @@ pip install mooncake-transfer-engine-non-cuda
make -j
```
5. Install Mooncake python package and mooncake_master executable
6. Install Mooncake python package and mooncake_master executable
```bash
make install
```

2
docs/source/getting_started/examples/vllm-integration/vllm-integration-v0.2.md
View File

@ -17,7 +17,7 @@ pip3 install mooncake-transfer-engine
Note:
- If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](build.md).
- For vLLM version <= v0.8.4, you must build from source since the earlier mooncake_vllm_adaptor interface is not contained in the pip wheel and will be deprecated in near future.
- For vLLM version <= v0.8.4, it requires mooncake-transfer-engine <= 0.3.3.post2. In the latest release, interface `mooncake_vllm_adaptor` has been deprecated.
### Install the latest version of vLLM
#### 1. Clone vLLM from official repo

2
docs/source/getting_started/examples/vllm-integration/vllm-integration-v1.md
View File

@ -25,7 +25,7 @@ pip3 install mooncake-transfer-engine
Note:
- If you encounter problems such as missing `lib*.so`, you should uninstall this package by `pip3 uninstall mooncake-transfer-engine`, and build the binaries manually according to the [instructions](build.md).
- For vLLM version <= v0.8.4, you must build from source since the earlier mooncake_vllm_adaptor interface is not contained in the pip wheel.
- For vLLM version <= v0.8.4, it requires mooncake-transfer-engine <= 0.3.3.post2. In the latest release, interface `mooncake_vllm_adaptor` has been deprecated.
### Install the latest version of vLLM
#### 1. Clone vLLM from official repo

5
docs/source/index.md
View File

@ -93,12 +93,13 @@ http-api-reference/http-service
:::{toctree}
:caption: Design Documents
:maxdepth: 1
:maxdepth: 2
design/architecture
design/mooncake-store
design/p2p-store
design/transfer-engine
design/transfer-engine/index
design/ep-backend
design/hicache-design
:::

4
docs/source/troubleshooting/troubleshooting.md
View File

@ -63,6 +63,10 @@ Errors in this part usually indicate that the error occurred within the `mooncak
**Solution:**
Ensure that the total memory registration does not exceed the device's upper limit. You may need to reduce the amount of memory being registered or split large memory regions into smaller chunks that fit within the device's `max_mr_size` limit.
5. If the error `Failed to create QP: Cannot allocate memory` is displayed, it typically caused by too many QP have been created, reaching the driver limit. You can use `rdma resource` to trace how many QP is created. One possible way to resolve this issue:
- Update Mooncake to version v0.3.5 or later
- Set the environment variable `MC_ENABLE_DEST_DEVICE_AFFINITY=1` before starting the application
## RDMA Transfer Period
### Recommended Troubleshooting Directions

8
docs/source/zh_archive/README.md Normal file
View File

@ -0,0 +1,8 @@
# 中文文档归档说明
此目录包含mooncake项目的中文文档归档。
## 当前状态
- 已归档，不再活跃维护
- 网站当前只显示英文文档
- 如需查看/编辑最新文档，请访问docs目录

0
doc/zh/architecture.md → docs/source/zh_archive/architecture.md
View File

0
doc/zh/ascend_direct_transport.md → docs/source/zh_archive/ascend_direct_transport.md
View File

0
doc/zh/ascend_transport.md → docs/source/zh_archive/ascend_transport.md
View File

0
doc/zh/build.md → docs/source/zh_archive/build.md
View File

0
doc/zh/error-code.md → docs/source/zh_archive/error-code.md
View File

0
doc/zh/heterogeneous_ascend.md → docs/source/zh_archive/heterogeneous_ascend.md
View File

0
doc/zh/mooncake-store.md → docs/source/zh_archive/mooncake-store.md
View File

0
doc/zh/p2p-store.md → docs/source/zh_archive/p2p-store.md
View File

0
doc/zh/run-examples.md → docs/source/zh_archive/run-examples.md
View File

0
doc/zh/transfer-engine-python.md → docs/source/zh_archive/transfer-engine-python.md
View File

0
doc/zh/transfer-engine.md → docs/source/zh_archive/transfer-engine.md
View File

0
doc/zh/troubleshooting.md → docs/source/zh_archive/troubleshooting.md
View File

0
doc/zh/vllm-integration-v0.1.md → docs/source/zh_archive/vllm-integration-v0.1.md
View File

0
doc/zh/vllm-integration-v1.md → docs/source/zh_archive/vllm-integration-v1.md
View File