diff --git a/doc/en/troubleshooting.md b/doc/en/troubleshooting.md index 10cad63a..bd7d5bf9 100644 --- a/doc/en/troubleshooting.md +++ b/doc/en/troubleshooting.md @@ -64,6 +64,13 @@ In addition, if the error `Failed to get description of XXX` is displayed, it in ## SGLang Common Questions +### Do I need RDMA to run SGLang and Mooncake? + +When using Mooncake for KV cache transfer in SGLang PD disaggregation deployments, GPUDirect RDMA (GDR) is required. + +When using Mooncake as a KV cache storage backend in SGLang HiCache, RDMA is recommended for better performance. +However, if RDMA NICs are not available, the TCP protocol is also supported. + ### How to make sure GPUDirect RDMA (GDR) is supported 1. Verify the presence of an RDMA-capable NIC (e.g., Mellanox, ERDMA) and drivers. @@ -84,7 +91,7 @@ lsmod | grep peer_mem lsmod | grep nvidia_peer_mem ``` -3. If you use container to run SGLang, please make sure RDMA and GDR driver are installed in the container and run container in previledge mode. Requirements: (1) privileged mode must be enabled. (2) RDMA devices/NVIDIA devices mounted into container +3. If you use container to run SGLang, please make sure RDMA and GDR driver are installed in the container and run container in privileged mode. Requirements: (1) privileged mode must be enabled. (2) RDMA devices/NVIDIA devices mounted into container 4. Check the connectivity Benchmark end-to-end performance using ib_write_bw. @@ -93,7 +100,6 @@ apt install perftest # server side ib_write_bw -d [rdma_device] -R -x gdr # client side -# server side ib_write_bw -d [rdma_device] -R -x gdr [server_ip] ``` Expected Output: diff --git a/docs/source/getting_started/examples/sglang-integration/hicache-integration-v1.md b/docs/source/getting_started/examples/sglang-integration/hicache-integration-v1.md index 02d9cde1..806a67c7 100644 --- a/docs/source/getting_started/examples/sglang-integration/hicache-integration-v1.md +++ b/docs/source/getting_started/examples/sglang-integration/hicache-integration-v1.md @@ -102,28 +102,43 @@ For more details, please refer to [Mooncake official installation guide](https:/ **Mooncake** is a distributed system that efficiently aggregates memory resources across multiple servers. It can also be deployed on a single server for simpler setups. -When integrated with **SGLang**, the system conceptually consists of four key components: `the master service`, `metadata service`, `store service`, and the `SGLang server`. Among them, the `master service` and `metadata service` are responsible for object and metadata maintenance. The `store service` manages a contiguous memory segment that contributes to the distributed KV cache, making its memory accessible to both local and remote `SGLang servers`. Data transfer occurs directly between the `store service` and `SGLang servers`, bypassing the `master service`. +When integrated with **SGLang**, the system conceptually consists of four key components: `the master service`, `metadata service` (Optional), `store service` (Optional), and the `SGLang server`. Among them, the `master service` and `metadata service` are responsible for object and metadata maintenance. The `store service` manages a contiguous memory segment that contributes to the distributed KV cache, making its memory accessible to both local and remote `SGLang servers`. Data transfer occurs directly between the `store service` and `SGLang servers`, bypassing the `master service`. ### Single Server Deployment -**Launch Mooncake `metadata service`:** +**Launch Mooncake `metadata service` (Optional):** ```bash python -m mooncake.http_metadata_server ``` +This service is responsible for centralized metadata management including internal connection status and related metadata. + +Deployment of the `metadata service` can be skipped in the following cases: +* Mooncake supports non-centralized metadata management via a P2P handshake mechanism to exchange metadata. When using this mode, deployment of the `metadata service` can be skipped. +* Mooncake also supports embedding `mededata service` into `master service`. In this case, only the `master service` needs to be started. + **Launch Mooncake `master service`:** +The `master service` orchestrates the logical storage space pool across the entire cluster, managing KV cache space allocation and eviction. + +To start `mooncake_master`: + ```bash mooncake_master --eviction_high_watermark_ratio=0.95 ``` +To start `mooncake_master` with embedded `metadata service` (so that a separate `metadata service` deployment can be skipped): + +```bash +mooncake_master --enable_http_metadata_server=true --http_metadata_server_port=8080 --eviction_high_watermark_ratio=0.95 +``` + **Understanding `eviction_high_watermark_ratio`:** When a `PutStart` request fails due to insufficient memory, or when the eviction thread detects that space usage has reached the configured high watermark ratio, an eviction task is triggered to free up space by evicting a portion of objects. -Due to memory fragmentation, allocation failures may occur even when memory usage has not yet reached 100%. The actual threshold depends on the workload. This [benchmark document](https://kvcache-ai.github.io/Mooncake/performance/allocator-benchmark-result.html) - provides memory allocation efficiency results under different scenarios. if excessive allocation failures are observed, consider lowering this parameter accordingly. +Due to memory fragmentation, allocation failures may occur even when memory usage has not yet reached 100%. The actual threshold depends on the workload. This [benchmark document](https://kvcache-ai.github.io/Mooncake/performance/allocator-benchmark-result.html) provides memory allocation efficiency results under different scenarios. if excessive allocation failures are observed, consider lowering this parameter accordingly. **Launch Mooncake `store service` (Optional):** @@ -132,72 +147,71 @@ First, create and save a configuration file in JSON format. For example: ```json { "local_hostname": "localhost", - "metadata_server": "http://localhost:8080/metadata", - "master_server_address": "localhost:50051", + "metadata_server": "http://127.0.0.1:8080/metadata", + "master_server_address": "127.0.0.1:50051", "protocol": "rdma", - "device_name": "mlx5_0,mlx5_1", - "global_segment_size": 2684354560, + "device_name": "", + "global_segment_size": "4gb", "local_buffer_size": 0 } ``` -Parameter Explanation: +Note: If the `metadata service` is not deployed, set this field to: -* `local_hostname`: The hostname of the `store service`. -* `metadata_server`: The network address of the `metadata service`. The default port is 8080. -* `master_server_address`: The network address of the `master service`. The default port is 50051. -* `protocol`: The protocol used by the Mooncake. Supported values are `"rdma"` or `"tcp"`. For optimal performance, `"rdma"` is recommended. -* `device_name`: The RDMA devices used by Mooncake. This parameter is required only when the protocol is set to `"rdma"`. Available devices can be listed using the `ibv_devices` command. -* `global_segment_size`: The amount of memory (in bytes) contributed to the global memory pool. A larger value allows Mooncake to cache more KV tensors. -* `local_buffer_size`: Local buffer is used to do request operations such as `Get` or `Put`. In this case, it is set to 0 because the instance functions solely as a storage server, contributing memory to the global pool without issuing any request operations. +```json + "metadata_server": "P2PHANDSHAKE", +``` Then start the `store service`: ```bash -python -m mooncake.mooncake_store_service --config=[config_path] +python -m mooncake.mooncake_store_service --config=[config_path] --port=8081 ``` Mooncake `store service` configuration can also be provided via environment variables: ```bash +MOONCAKE_LOCAL_HOSTNAME="localhost" \ MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata" \ -MOONCAKE_GLOBAL_SEGMENT_SIZE=4294967296 \ +MOONCAKE_MASTER="127.0.0.1:50051" \ MOONCAKE_PROTOCOL="rdma" \ -MOONCAKE_DEVICE="erdma_0,erdma_1" \ -MOONCAKE_MASTER=127.0.0.1:50051 \ -python -m mooncake.mooncake_store_service +MOONCAKE_DEVICE="" \ +MOONCAKE_GLOBAL_SEGMENT_SIZE="4gb" \ +MOONCAKE_LOCAL_BUFFER_SIZE=0 \ +python -m mooncake.mooncake_store_service --port=8081 ``` +**Parameter Explanation:** + +* `local_hostname`, `MOONCAKE_LOCAL_HOSTNAME`: The hostname of the `store service`. +* `metadata_server`, `MOONCAKE_TE_META_DATA_SERVER` : The network address of the `metadata service`. The default port is 8080. If the `metadata service` is not deployed, set this field to: `"metadata_server": "P2PHANDSHAKE"`. +* `master_server_address`, `MOONCAKE_MASTER`: The network address of the `master service`. The default port is 50051. +* `protocol`, `MOONCAKE_PROTOCOL`: The protocol used by Mooncake. Supported values are `"rdma"` or `"tcp"`. For optimal performance, `"rdma"` is recommended. +* `device_name`, `MOONCAKE_DEVICE`: The RDMA devices used by Mooncake. This field can usually be left empty, as Mooncake automatically discovers available NICs by default. This parameter is required only when the protocol is set to `"rdma"` **and** a specific set of NICs needs to be used. Example: `"device_name": "mlx5_0,mlx5_1"`. To list available devices, run `ibv_devices`. **Note:** If the environment variable `MC_MS_AUTO_DISC` is set to `1`, any `device_name` or `MOONCAKE_DEVICE` configuration will be overridden, and Mooncake will switch to auto-discovery mode. +* `global_segment_size`, `MOONCAKE_GLOBAL_SEGMENT_SIZE`: The amount of memory contributed to the global memory pool. Accepts either bytes (integer) or a string with the `gb` suffix, e.g., `"4294967296"` or `"4gb"`. A larger value allows Mooncake to cache more KV tensors. +* `local_buffer_size`, `MOONCAKE_LOCAL_BUFFER_SIZE`: Local buffer is used to do request operations such as `Get` or `Put`. In this case, it is set to 0 because the instance functions solely as a storage server, contributing memory to the global pool without issuing any request operations. + +**Important: Understanding Global Segment Size** + +`global_segment_size` and `MOONCAKE_GLOBAL_SEGMENT_SIZE`: This parameter specifies the amount of memory each instance contributes to the distributed memory pool. The total memory available for KV cache storage across the cluster is the sum of the memory contributed by all instances. + +Adjust this value according to system’s available memory and expected cache requirements. Note: If `MOONCAKE_GLOBAL_SEGMENT_SIZE` is set to a non-zero value when starting the `SGLang server`, launching the `store service` can be skipped. In this case, the `SGLang server` also takes on the role of the `store service`, which simplifies deployment but couples the two components together. Users can choose the deployment approach that best fits their needs. **Start the `SGLang server` with Mooncake enabled:** -Mooncake configuration can be provided via environment variables. Note that, for optimal performance, the Mooncake backend currently supports only the `page_first` layout (which optimizes memory access patterns for KV cache operations). +There are three ways to configure Mooncake: -There are two ways to configure Mooncake: 1. Using environment variables; 2. Using extra-config of sglang arguments. +1. Via extra configuration passed through sglang parameters +2. Using JSON configuration files +3. Using environment variables -**Using env variables to configure Mooncake** +Mooncake loads configuration in the following priority order: -```bash -MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata" \ -MOONCAKE_MASTER=127.0.0.1:50051 \ -MOONCAKE_PROTOCOL="rdma" \ -MOONCAKE_DEVICE="mlx5_0,mlx5_1" \ -MOONCAKE_GLOBAL_SEGMENT_SIZE=4294967296 \ -python -m sglang.launch_server \ - --enable-hierarchical-cache \ - --hicache-storage-backend mooncake\ - --model-path [model_path] -``` - -Parameter Explanation: - -* `MOONCAKE_TE_META_DATA_SERVER`: The network address of the `metadata service`. The default port is 8080. -* `MOONCAKE_MASTER`: The network address of the `master service`. The default port is 50051. -* `MOONCAKE_PROTOCOL`: The protocol used by Mooncake. Supported values are `"rdma"` or `"tcp"`. For optimal performance, `"rdma"` is recommended. -* `MOONCAKE_DEVICE`: The RDMA devices used by Mooncake. This parameter is required only when the protocol is set to `"rdma"`. Available devices can be listed using the `ibv_devices` command. -* `MOONCAKE_GLOBAL_SEGMENT_SIZE`: The amount of memory (in bytes) contributed to the global memory pool. If at least one `store service` is launched, then this value could be set to `0`. In this case, the `SGLang server` will not contribute any memory to the system. Note that KV tensors cached in the contributed memory will be lost once this process terminates; however, this will not cause any system errors. +1. If Mooncake-specific options are provided in `--hicache-storage-backend-extra-config`, they are used first. +2. If not, Mooncake checks whether the environment variable `DEFAULT_MOONCAKE_CONFIG_PATH_ENV` is set, and loads the JSON config file from that path. +3. If neither of the above is provided, Mooncake falls back to environment variables. **Using extra-config of sglang arguments to configure Mooncake** @@ -206,14 +220,52 @@ python -m sglang.launch_server \ --enable-hierarchical-cache \ --hicache-storage-backend mooncake \ --model-path [model_path] \ - --hicache-storage-backend-extra-config '{"master_server_address": "127.0.0.1:50051", "local_hostname": "localhost", "metadata_server": "http://127.0.0.1:8080/metadata", "global_segment_size": 4294967296, "local_buffer_size": 16777216, "protocol": "rdma", "device_name": "mlx5_0,mlx5_1"}' + --hicache-storage-backend-extra-config '{"master_server_address": "127.0.0.1:50051", "local_hostname": "localhost", "metadata_server": "http://127.0.0.1:8080/metadata", "global_segment_size": "4gb", "protocol": "rdma", "device_name": ""}' ``` -**Important: Understanding Global Segment Size** +**Using JSON file to configure Mooncake** -`global_segment_size` for `store service` and `MOONCAKE_GLOBAL_SEGMENT_SIZE` for `SGLang service`: This parameter specifies the amount of memory each instance contributes to the distributed memory pool. The total memory available for KV cache storage across the cluster is the sum of the memory contributed by all instances. +SGLang server can load Mooncake config from `SGLANG_HICACHE_MOONCAKE_CONFIG_PATH`. -Adjust this value according to system’s available memory and expected cache requirements. +```bash +export SGLANG_HICACHE_MOONCAKE_CONFIG_PATH=/sgl-workspace/sglang/benchmark/hicache/mooncake_config.json + +echo '{ + "local_hostname": "localhost", + "metadata_server": "http://127.0.0.1:8080/metadata", + "master_server_address": "127.0.0.1:50051", + "protocol": "rdma", + "device_name": "", + "global_segment_size": "4gb" +}' > ${SGLANG_HICACHE_MOONCAKE_CONFIG_PATH} + +python -m sglang.launch_server \ + --enable-hierarchical-cache \ + --hicache-storage-backend mooncake \ + --model-path [model_path] +``` + +**Using env variables to configure Mooncake** + +```bash +MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata" \ +MOONCAKE_MASTER="127.0.0.1:50051" \ +MOONCAKE_PROTOCOL="rdma" \ +MOONCAKE_DEVICE="" \ +MOONCAKE_GLOBAL_SEGMENT_SIZE="4gb" \ +python -m sglang.launch_server \ + --enable-hierarchical-cache \ + --hicache-storage-backend mooncake\ + --model-path [model_path] +``` + +**Parameter Explanation:** + +The Mooncake parameters used here are essentially the same as those configured for the `store service`. + +In particular, for the `global segment size`, if at least one `store service` instance is running, this value can be set to `0`. In this case, the SGLang server will not contribute any memory to the system. Note that KV tensors stored in this contributed memory will be lost when the process exits; however, this will **not** cause any system errors. + +**Important:** when `tp > 1`, each Tensor Parallel (TP) rank launches its own Mooncake backend instance and contributes `1/global_segment_size` memory. Therefore, the total memory consumption equals `global segment size`. **HiCache Related Parameters for SGLang Server** @@ -281,11 +333,23 @@ python -m sglang_router.launch_router \ ## Troubleshooting -**RDMA Registration Failure**: +**RDMA Registration Failure:** * In some environments, RDMA registration may require root privileges. In this case, try running the program as root. * In certain environments (e.g., eRDMA), there is an upper limit on the total amount of RDMA memory that can be registered. Once this limit is exceeded, registration will fail. To resolve this, you can lower the value of `MOONCAKE_GLOBAL_SEGMENT_SIZE`, or reduce the host memory allocated to HiCache in the `SGLang server` (since this memory is fully registered with RDMA to enable zero-copy). +**HiCache CPU Memory Usage:** + +When using HiCache, the default L2 host DRAM (CPU memory) size for KV cache is **2 times** the size of the L1 device memory (GPU memory) for KV cache. + +If the model is small but the GPU memory is large — especially in multi-TP (tensor parallel) setups — this may cause the L1 KV cache to become very large, which in turn can consume excessive CPU DRAM. + +In such cases, you should manually configure an appropriate L2 cache size based on your hardware. This can be done by setting `--hicache-ratio` or `--hicache-size`. + +**More Information:** + +Additional troubleshooting information can be found [here](https://kvcache-ai.github.io/Mooncake/troubleshooting/troubleshooting.html). + ## Test Mooncake Store This test is intended for developers to quickly verify that the MooncakeStore class interfaces are functioning correctly. @@ -296,7 +360,6 @@ First, start the `metadata service` and `master service`. Then run the `test_moo MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata" \ MOONCAKE_MASTER=127.0.0.1:50051 \ MOONCAKE_PROTOCOL="rdma" \ -MOONCAKE_DEVICE="mlx5_0,mlx5_1" \ MOONCAKE_GLOBAL_SEGMENT_SIZE=16777216 \ python3 [path of test_mooncake_store.py] ``` diff --git a/docs/source/troubleshooting/troubleshooting.md b/docs/source/troubleshooting/troubleshooting.md index 3f04a7dd..e6c50f59 100644 --- a/docs/source/troubleshooting/troubleshooting.md +++ b/docs/source/troubleshooting/troubleshooting.md @@ -71,3 +71,47 @@ If the network state is unstable, some requests may not be delivered, displaying Note: In most cases, the errors output, except for the first occurrence, are `work request flushed error`. This is because when the first error occurs, the RDMA driver sets the connection to an unavailable state, so tasks in the submission queue are blocked from execution and subsequent errors are reported. Therefore, it is recommended to locate the first occurrence of the error and check it. In addition, if the error `Failed to get description of XXX` is displayed, it indicates that the Segment name input by the user when calling the `openSegment` interface cannot be found in the etcd database. For memory read/write scenarios, the Segment name needs to strictly match the `local_hostname` field filled in by the other node during initialization. + +## SGLang Common Questions + +### Do I need RDMA to run SGLang and Mooncake? + +When using Mooncake for KV cache transfer in SGLang PD disaggregation deployments, GPUDirect RDMA (GDR) is required. + +When using Mooncake as a KV cache storage backend in SGLang HiCache, RDMA is recommended for better performance. +However, if RDMA NICs are not available, the TCP protocol is also supported. + +### How to make sure GPUDirect RDMA (GDR) is supported + +1. Verify the presence of an RDMA-capable NIC (e.g., Mellanox, ERDMA) and drivers. +``` +ibv_devices +lspci | grep rdma +lsmod | grep -E 'ib_core|mlx4_core|mlx5_core|nvidia_peer_mem' +``` +If no RDMA devices appear: (1) Confirm physical NIC presence via lspci +(2) Install vendor-specific drivers (e.g., Mellanox MLNX_OFED) + +2. check GDR driver is ready, and peer_memory module (part of MLNX_OFED) should be installed +``` +# Check peer_memory module (from MLNX_OFED) +lsmod | grep peer_mem + +# Verify NVIDIA peer memory module +lsmod | grep nvidia_peer_mem +``` + +3. If you use container to run SGLang, please make sure RDMA and GDR driver are installed in the container and run container in privileged mode. Requirements: (1) privileged mode must be enabled. (2) RDMA devices/NVIDIA devices mounted into container + +4. Check the connectivity +Benchmark end-to-end performance using ib_write_bw. +``` +apt install perftest +# server side +ib_write_bw -d [rdma_device] -R -x gdr +# client side +ib_write_bw -d [rdma_device] -R -x gdr [server_ip] +``` +Expected Output: +Successful bidirectional transfer with "BW peak" reported +Errors with -x gdr indicate GDR setup failures diff --git a/mooncake-store/src/pybind_client.cpp b/mooncake-store/src/pybind_client.cpp index 06a00dc1..9ac7ca51 100644 --- a/mooncake-store/src/pybind_client.cpp +++ b/mooncake-store/src/pybind_client.cpp @@ -193,6 +193,8 @@ tl::expected PyClient::setup_internal( client_buffer_allocator_ = ClientBufferAllocator::create(local_buffer_size, this->protocol); if (local_buffer_size > 0) { + LOG(INFO) << "Registering local memory: " << local_buffer_size + << " bytes"; auto result = client_->RegisterLocalMemory( client_buffer_allocator_->getBase(), local_buffer_size, kWildcardLocation, false, true); diff --git a/mooncake-wheel/mooncake/mooncake_config.py b/mooncake-wheel/mooncake/mooncake_config.py index 7ff330b2..8a0328cd 100644 --- a/mooncake-wheel/mooncake/mooncake_config.py +++ b/mooncake-wheel/mooncake/mooncake_config.py @@ -10,7 +10,7 @@ from typing import Optional DEFAULT_GLOBAL_SEGMENT_SIZE = 3355443200 # 3.125 GiB DEFAULT_LOCAL_BUFFER_SIZE = 1073741824 # 1.0 GiB -def _parse_global_segment_size(value) -> int: +def _parse_segment_size(value) -> int: if isinstance(value, int): return value if isinstance(value, str): @@ -19,7 +19,7 @@ def _parse_global_segment_size(value) -> int: num = s[:-2].strip() if not num: raise ValueError( - "Invalid global_segment_size: missing number before 'gb'" + "Invalid segment size: missing number before 'gb'" ) return int(num) * 1024 * 1024 * 1024 return int(s) @@ -73,11 +73,12 @@ class MooncakeConfig: return MooncakeConfig( local_hostname=config.get("local_hostname"), metadata_server=config.get("metadata_server"), - global_segment_size=_parse_global_segment_size( + global_segment_size=_parse_segment_size( config.get("global_segment_size", DEFAULT_GLOBAL_SEGMENT_SIZE) ), - local_buffer_size=config.get("local_buffer_size", - DEFAULT_LOCAL_BUFFER_SIZE), + local_buffer_size=_parse_segment_size( + config.get("local_buffer_size", DEFAULT_LOCAL_BUFFER_SIZE) + ), protocol=config.get("protocol", "tcp"), device_name=config.get("device_name", ""), master_server_address=config.get("master_server_address"), @@ -96,13 +97,14 @@ class MooncakeConfig: if not os.getenv("MOONCAKE_MASTER"): raise ValueError("Neither the environment variable 'MOONCAKE_CONFIG_PATH' nor 'MOONCAKE_MASTER' is set.") return MooncakeConfig( - local_hostname=os.getenv("LOCAL_HOSTNAME", "localhost"), + local_hostname=os.getenv("MOONCAKE_LOCAL_HOSTNAME", "localhost"), metadata_server=os.getenv("MOONCAKE_TE_META_DATA_SERVER", "P2PHANDSHAKE"), - global_segment_size=_parse_global_segment_size( + global_segment_size=_parse_segment_size( os.getenv("MOONCAKE_GLOBAL_SEGMENT_SIZE", DEFAULT_GLOBAL_SEGMENT_SIZE) ), - # Zero copy interface does not need local buffer - local_buffer_size=DEFAULT_LOCAL_BUFFER_SIZE, + local_buffer_size=_parse_segment_size( + os.getenv("MOONCAKE_LOCAL_BUFFER_SIZE", DEFAULT_LOCAL_BUFFER_SIZE) + ), protocol=os.getenv("MOONCAKE_PROTOCOL", "tcp"), device_name=os.getenv("MOONCAKE_DEVICE", ""), master_server_address=os.getenv("MOONCAKE_MASTER"),