From 70488aae3f1b3e975cc417a5b208ca12bad9c59b Mon Sep 17 00:00:00 2001 From: EkiRui <374389385@qq.com> Date: Mon, 29 Dec 2025 16:50:26 +0800 Subject: [PATCH] [Store]doc: add dummy client support for SGLang hicache integration (#1299) Signed-off-by: Xingrui Yi --- docs/source/design/mooncake-store.md | 2 + .../hicache-integration-v1.md | 83 +++++++++++++++++++ docs/source/zh_archive/mooncake-store.md | 2 + 3 files changed, 87 insertions(+) diff --git a/docs/source/design/mooncake-store.md b/docs/source/design/mooncake-store.md index d8385322..08677b60 100644 --- a/docs/source/design/mooncake-store.md +++ b/docs/source/design/mooncake-store.md @@ -776,6 +776,8 @@ The **real** `Client` can be configured using the following parameters: - **`host`**: (string, default: "0.0.0.0"): The hostname of the client. +- **`port`**: (int, default: 50052): The port number the client service listens on. + - **`global_segment_size`**: (string, default: "4GB"): The size of the global segment to be allocated by the client. - **`master_server_address`**: (string, default: "localhost:50051"): The address of the Master Service. diff --git a/docs/source/getting_started/examples/sglang-integration/hicache-integration-v1.md b/docs/source/getting_started/examples/sglang-integration/hicache-integration-v1.md index 806a67c7..3b963987 100644 --- a/docs/source/getting_started/examples/sglang-integration/hicache-integration-v1.md +++ b/docs/source/getting_started/examples/sglang-integration/hicache-integration-v1.md @@ -280,6 +280,89 @@ Distributed deployment of Mooncake is straightforward. Similar to the single-nod Mooncake also supports high availability mode. This mode enhances fault tolerance by running the `master service` as a cluster of multiple master nodes coordinated through an `etcd` cluster. The master nodes use `etcd` to elect a leader, which is responsible for handling client requests. For more details about how to deploy in this mode, please refer to our [documents](https://kvcache-ai.github.io/Mooncake/). +### Deployment with Dummy Client (Experimental) + +In addition to the standard deployment where SGLang acts as a full Mooncake node, you can use the **Dummy Client** mode. In this mode, SGLang connects to a local **Mooncake Store Service** (Real Client) via RPC/IPC. This decouples the SGLang process from the heavy RDMA and memory management, potentially improving stability and allowing the cache to persist even if the SGLang process restarts. + +**Architecture:** +* **Mooncake Master**: Manages the cluster topology (same as standard). +* **Mooncake Store Service (Real Client)**: Manages the actual memory pool and RDMA connections. Must be running locally. +* **SGLang Server (Dummy Client)**: Connects to the local Store Service to access the cache. + +#### 1. Launch Services (Master & Store) + +First, start the `master service` and the `store service`. The `store service` acts as the Real Client. + +**Start Master:** +```bash +mooncake_master --eviction_high_watermark_ratio=0.95 +``` + +**Start Store Service (Real Client):** Crucially, the default port (50052) is used for internal RPC, which the Dummy Client will connect to. +```bash +mooncake_client --global_segment_size=4GB +``` + +**Parameter Explanation:** + +- **`host`**: (string, default: "0.0.0.0"): The hostname of the client. + +- **`port`**: (int, default: 50052): The port number the client service listens on. + +- **`global_segment_size`**: (string, default: "4GB"): The size of the global segment to be allocated by the client. + +- **`master_server_address`**: (string, default: "localhost:50051"): The address of the Master Service. + +- **`metadata_server`**: (string, default: "http://localhost:8080/metadata"): The address of the metadata service. + +- **`protocol`**: (string, default: "tcp"): The protocol used by the Transfer Engine. + +- **`device_name`**: (string, default: ""): The device name used by the Transfer Engine. + +- **`threads`**: (int, default: 1): The number of threads used by the client. + +#### 2. Launch SGLang (Dummy Client) +Configure SGLang to connect to the Real Client using the client_server_address parameter. + +**Using extra-config of sglang arguments to configure Mooncake** + +```bash +python -m sglang.launch_server \ + --enable-hierarchical-cache \ + --hicache-storage-backend mooncake \ + --model-path [model_path] \ + --hicache-storage-backend-extra-config '{"standalone_storage": true, "client_server_address": "127.0.0.1:50052"}' +``` + +**Using JSON file to configure Mooncake** + +SGLang server can load Mooncake config from `SGLANG_HICACHE_MOONCAKE_CONFIG_PATH`. + +```bash +export SGLANG_HICACHE_MOONCAKE_CONFIG_PATH=/sgl-workspace/sglang/benchmark/hicache/mooncake_config.json + +echo '{ + "standalone_storage": true, + "client_server_address": "127.0.0.1:50052" +}' > ${SGLANG_HICACHE_MOONCAKE_CONFIG_PATH} + +python -m sglang.launch_server \ + --enable-hierarchical-cache \ + --hicache-storage-backend mooncake \ + --model-path [model_path] +``` + +**Using env variables to configure Mooncake** + +```bash +MOONCAKE_STANDALONE_STORAGE=1 +MOONCAKE_REAL_CLIENT_ADDRESS="127.0.0.1:50052" +python -m sglang.launch_server \ + --enable-hierarchical-cache \ + --hicache-storage-backend mooncake \ + --model-path [model_path] +``` + ### Prefill/Decode Disaggregation In **PD disaggregation**, the configurations for the `metadata service`, `mooncake master`, and the optional `store service` remain the same as described above. The difference is that SGLang introduces three distinct roles: `prefill worker`, `decode worker`, and `router`. diff --git a/docs/source/zh_archive/mooncake-store.md b/docs/source/zh_archive/mooncake-store.md index fdde058b..78cb296f 100644 --- a/docs/source/zh_archive/mooncake-store.md +++ b/docs/source/zh_archive/mooncake-store.md @@ -796,6 +796,8 @@ retcode = store.setup( - **`host`**: （字符串, 默认: "0.0.0.0"）: client 的主机名。 +- **`port`**: （整型, 默认: 50052）: client 监听的端口。 + - **`global_segment_size`**: （字符串, 默认: "4GB"）: client 向集群中挂载的 Segment 大小。 - **`master_server_address`**: （字符串, 默认: "localhost:50051"）: Master 服务的地址。