[Store]doc: add dummy client support for SGLang hicache integration (#1299)

Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
This commit is contained in:
EkiRui 2025-12-29 16:50:26 +08:00 committed by GitHub
parent 0b149e6c13
commit 70488aae3f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 87 additions and 0 deletions

View File

@ -776,6 +776,8 @@ The **real** `Client` can be configured using the following parameters:
- **`host`**: (string, default: "0.0.0.0"): The hostname of the client.
- **`port`**: (int, default: 50052): The port number the client service listens on.
- **`global_segment_size`**: (string, default: "4GB"): The size of the global segment to be allocated by the client.
- **`master_server_address`**: (string, default: "localhost:50051"): The address of the Master Service.

View File

@ -280,6 +280,89 @@ Distributed deployment of Mooncake is straightforward. Similar to the single-nod
Mooncake also supports high availability mode. This mode enhances fault tolerance by running the `master service` as a cluster of multiple master nodes coordinated through an `etcd` cluster. The master nodes use `etcd` to elect a leader, which is responsible for handling client requests. For more details about how to deploy in this mode, please refer to our [documents](https://kvcache-ai.github.io/Mooncake/).
### Deployment with Dummy Client (Experimental)
In addition to the standard deployment where SGLang acts as a full Mooncake node, you can use the **Dummy Client** mode. In this mode, SGLang connects to a local **Mooncake Store Service** (Real Client) via RPC/IPC. This decouples the SGLang process from the heavy RDMA and memory management, potentially improving stability and allowing the cache to persist even if the SGLang process restarts.
**Architecture:**
* **Mooncake Master**: Manages the cluster topology (same as standard).
* **Mooncake Store Service (Real Client)**: Manages the actual memory pool and RDMA connections. Must be running locally.
* **SGLang Server (Dummy Client)**: Connects to the local Store Service to access the cache.
#### 1. Launch Services (Master & Store)
First, start the `master service` and the `store service`. The `store service` acts as the Real Client.
**Start Master:**
```bash
mooncake_master --eviction_high_watermark_ratio=0.95
```
**Start Store Service (Real Client):** Crucially, the default port (50052) is used for internal RPC, which the Dummy Client will connect to.
```bash
mooncake_client --global_segment_size=4GB
```
**Parameter Explanation:**
- **`host`**: (string, default: "0.0.0.0"): The hostname of the client.
- **`port`**: (int, default: 50052): The port number the client service listens on.
- **`global_segment_size`**: (string, default: "4GB"): The size of the global segment to be allocated by the client.
- **`master_server_address`**: (string, default: "localhost:50051"): The address of the Master Service.
- **`metadata_server`**: (string, default: "http://localhost:8080/metadata"): The address of the metadata service.
- **`protocol`**: (string, default: "tcp"): The protocol used by the Transfer Engine.
- **`device_name`**: (string, default: ""): The device name used by the Transfer Engine.
- **`threads`**: (int, default: 1): The number of threads used by the client.
#### 2. Launch SGLang (Dummy Client)
Configure SGLang to connect to the Real Client using the client_server_address parameter.
**Using extra-config of sglang arguments to configure Mooncake**
```bash
python -m sglang.launch_server \
--enable-hierarchical-cache \
--hicache-storage-backend mooncake \
--model-path [model_path] \
--hicache-storage-backend-extra-config '{"standalone_storage": true, "client_server_address": "127.0.0.1:50052"}'
```
**Using JSON file to configure Mooncake**
SGLang server can load Mooncake config from `SGLANG_HICACHE_MOONCAKE_CONFIG_PATH`.
```bash
export SGLANG_HICACHE_MOONCAKE_CONFIG_PATH=/sgl-workspace/sglang/benchmark/hicache/mooncake_config.json
echo '{
"standalone_storage": true,
"client_server_address": "127.0.0.1:50052"
}' > ${SGLANG_HICACHE_MOONCAKE_CONFIG_PATH}
python -m sglang.launch_server \
--enable-hierarchical-cache \
--hicache-storage-backend mooncake \
--model-path [model_path]
```
**Using env variables to configure Mooncake**
```bash
MOONCAKE_STANDALONE_STORAGE=1
MOONCAKE_REAL_CLIENT_ADDRESS="127.0.0.1:50052"
python -m sglang.launch_server \
--enable-hierarchical-cache \
--hicache-storage-backend mooncake \
--model-path [model_path]
```
### Prefill/Decode Disaggregation
In **PD disaggregation**, the configurations for the `metadata service`, `mooncake master`, and the optional `store service` remain the same as described above. The difference is that SGLang introduces three distinct roles: `prefill worker`, `decode worker`, and `router`.

View File

@ -796,6 +796,8 @@ retcode = store.setup(
- **`host`**: (字符串, 默认: "0.0.0.0": client 的主机名。
- **`port`**: (整型, 默认: 50052: client 监听的端口。
- **`global_segment_size`**: (字符串, 默认: "4GB": client 向集群中挂载的 Segment 大小。
- **`master_server_address`**: (字符串, 默认: "localhost:50051": Master 服务的地址。