diff --git a/README.md b/README.md index 5969a094..edabe735 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ Mooncake is the serving platform for icon Kimi, a leading LLM service provided by icon Moonshot AI. Now both the Transfer Engine and Mooncake Store are open-sourced! -This repository also hosts its technical report and the open sourced traces. +This repository also hosts its technical report and the open-sourced traces.

🔄 Updates

@@ -47,23 +47,23 @@ This repository also hosts its technical report and the open sourced traces. - **May 5, 2025**: Supported by Mooncake Team, SGLang release guidance to deploy DeepSeek with PD Disaggregation on 96 H100 GPUs. - **Apr 22, 2025**: LMCache officially supports Mooncake Store as a remote connector. - **Apr 10, 2025**: SGLang officially supports Mooncake Transfer Engine for disaggregated prefilling and KV cache transfer. - - **Mar 7, 2025**: We open sourced the Mooncake Store, a distributed KVCache based on Transfer Engine. vLLM's xPyD disaggregated prefilling & decoding based on Mooncake Store will be released soon. +- **Mar 7, 2025**: We open-sourced the Mooncake Store, a distributed KVCache based on Transfer Engine. vLLM's xPyD disaggregated prefilling & decoding based on Mooncake Store will be released soon. - **Feb 25, 2025**: Mooncake receives the **Best Paper Award** at **FAST 2025**! - **Feb 21, 2025**: The updated traces used in our FAST'25 paper have been released. - **Dec 16, 2024**: vLLM officially supports Mooncake Transfer Engine for disaggregated prefilling and KV cache transfer. - - **Nov 28, 2024**: We open sourced the Transfer Engine, the central component of Mooncake. We also provide two demonstrations of Transfer Engine: a P2P Store and vLLM integration. - - **July 9, 2024**: We open sourced the trace as a jsonl file. +- **Nov 28, 2024**: We open-sourced the Transfer Engine, the central component of Mooncake. We also provide two demonstrations of Transfer Engine: a P2P Store and vLLM integration. +- **July 9, 2024**: We open-sourced the trace as a JSONL file. - **June 27, 2024**: We present a series of Chinese blogs with more discussions on zhihu 1, 2, 3, 4, 5, 6, 7. - **June 26, 2024**: Initial technical report release.

🎉 Overview

-Mooncake features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache. +Mooncake features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated KVCache pool. ![architecture](image/architecture.png) -The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs) requirements. Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake’s innovative architecture enables Kimi to handle 75% more requests. +The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges in highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake’s innovative architecture enables Kimi to handle 75% more requests.

🧩 Components

@@ -81,7 +81,7 @@ Mooncake has been seamlessly integrated with several popular large language mode In addition, Mooncake has been successfully integrated with SGLang's Hierarchical KV Caching, vLLM's prefill serving, and LMCache, augmenting KV cache management capabilities across large-scale inference scenarios. **Elastic Expert Parallelism Support** -Mooncake adds elasticity and fault tolerance support for MoE model inference, enabling inference systems to remain responsive and recoverable in the event of GPU failures or changes in resource configuration. This functionality includes automatic faulty rank detection and can incorporate with the EPLB module to dynamically route tokens to healthy ranks during inference. +Mooncake adds elasticity and fault tolerance support for MoE model inference, enabling inference systems to remain responsive and recoverable in the event of GPU failures or changes in resource configuration. This functionality includes automatic faulty rank detection and can work with the EPLB module to dynamically route tokens to healthy ranks during inference. **Tensor-Centric Ecosystem** Mooncake establishes a full-stack, Tensor-oriented AI infrastructure where Tensors serve as the fundamental data carrier. The ecosystem spans from the Transfer Engine, which accelerates Tensor data movement across heterogeneous storage (DRAM/VRAM/NVMe), to the P2P Store and Mooncake Store for distributed management of Tensor objects (e.g., Checkpoints and KVCache), up to the Mooncake Backend enabling Tensor-based elastic distributed computing. This architecture is designed to maximize Tensor processing efficiency for large-scale model inference and training. @@ -97,7 +97,7 @@ Transfer Engine is a high-performance data transfer framework. Transfer Engine p - **Topology aware path selection.** Transfer Engine can *select optimal devices* based on the location (NUMA affinity, etc.) of both source and destination. -- **More robust on temporary network error.** Once transmission fails, Transfer Engine will try to use alternative paths for data delivery automatically. +- **More robust against temporary network errors.** Once transmission fails, Transfer Engine will try to use alternative paths for data delivery automatically. #### Performance With 40 GB of data (equivalent to the size of the KVCache generated by 128k tokens in the LLaMA3-70B model), Mooncake Transfer Engine delivers up to **87 GB/s** and **190 GB/s** of bandwidth in 4×200 Gbps and 8×400 Gbps RoCE networks respectively, which are about **2.4x and 4.6x faster** than the TCP protocol. @@ -159,7 +159,7 @@ In the future, we will further improve TTFT through GPUDirect RDMA and zero-copy - Click [here](https://kvcache-ai.github.io/Mooncake/performance/vllm-benchmark-results-v0.2.html) to access detailed benchmark results. -**More advanced features will coming soon, so stay tuned!** +**More advanced features are coming soon, so stay tuned!**

🚀 Quick Start

@@ -167,13 +167,13 @@ In the future, we will further improve TTFT through GPUDirect RDMA and zero-copy Mooncake is designed and optimized for high-speed RDMA networks. Though Mooncake supports TCP-only data transfer, we **strongly** recommend users to evaluate the functionality and performance of Mooncake with RDMA network support. -The following needs to be installed before running any component of Mooncake: +The following need to be installed before running any component of Mooncake: - RDMA Driver & SDK, such as Mellanox OFED. - Python 3.10, virtual environment is recommended. -- CUDA 12.1 and above, including NVIDIA GPUDirect Storage Support, if the package is build with `-DUSE_CUDA` (disabled by default). *You may install them from [here](https://developer.nvidia.com/cuda-downloads)*. +- CUDA 12.1 and above, including NVIDIA GPUDirect Storage Support, if the package is built with `-DUSE_CUDA` (disabled by default). *You may install them from [here](https://developer.nvidia.com/cuda-downloads)*. ### Use Python package -The most simple way to use Mooncake Transfer Engine is using `pip`: +The simplest way to use Mooncake Transfer Engine is using `pip`: **For CUDA-enabled systems:** ```bash @@ -202,7 +202,7 @@ The following are additional dependencies for building Mooncake: - [Optional] `hiredis`, if you want to build with `-DUSE_REDIS` to use Redis instead of etcd as metadata servers. - [Optional] `curl`, if you want to build with `-DUSE_HTTP` to use HTTP instead of etcd as metadata servers. -The building and installation steps are the following: +The build and installation steps are as follows: 1. Retrieve source code from GitHub repo ```bash git clone https://github.com/kvcache-ai/Mooncake.git @@ -254,31 +254,37 @@ The above presents two samples from our trace dataset. The trace includes the ti Please kindly cite our paper if you find the paper or the traces are useful: ```bibtex -@article{qin2024mooncake, - title={Mooncake: A kvcache-centric disaggregated architecture for llm serving}, - author={Qin, Ruoyu and Li, Zheming and He, Weiran and Cui, Jialei and Tang, Heyi and Ren, Feng and Ma, Teng and Cai, Shangming and Zhang, Yineng and Zhang, Mingxing and others}, - journal={ACM Transactions on Storage}, - year={2024}, - publisher={ACM New York, NY} +@article{qin2025mooncake_tos, + author = {Qin Ruoyu and Li Zheming and He Weiran and Cui Jialei and Tang Heyi and Ren Feng and Ma Teng and Cai Shangming and Zhang Yineng and Zhang Mingxing and Wu Yongwei and Zheng Weimin and Xu Xinran}, + title = {Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving}, + year = {2025}, + publisher = {Association for Computing Machinery}, + address = {New York, NY, USA}, + issn = {1553-3077}, + url = {https://doi.org/10.1145/3773772}, + doi = {10.1145/3773772}, + journal = {ACM Trans. Storage}, + month = {nov}, + keywords = {Machine learning system, LLM serving, KVCache}, } -@article{qin2024mooncake, - title = {Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving}, - author = {Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu}, - year = {2024}, - url = {https://arxiv.org/abs/2407.00079} +@inproceedings{qin2025mooncake, + author = {Ruoyu Qin and Zheming Li and Weiran He and Jialei Cui and Feng Ren and Mingxing Zhang and Yongwei Wu and Weimin Zheng and Xinran Xu}, + title = {Mooncake: Trading More Storage for Less Computation {\textemdash} A {KVCache-centric} Architecture for Serving {LLM} Chatbot}, + booktitle = {23rd USENIX Conference on File and Storage Technologies (FAST 25)}, + year = {2025}, + isbn = {978-1-939133-45-8}, + address = {Santa Clara, CA}, + pages = {155--170}, + url = {https://www.usenix.org/conference/fast25/presentation/qin}, + publisher = {USENIX Association}, + month = {feb}, } -@inproceedings {qin2025mooncake, - author = {Ruoyu Qin and Zheming Li and Weiran He and Jialei Cui and Feng Ren and Mingxing Zhang and Yongwei Wu and Weimin Zheng and Xinran Xu}, - title = {Mooncake: Trading More Storage for Less Computation {\textemdash} A {KVCache-centric} Architecture for Serving {LLM} Chatbot}, - booktitle = {23rd USENIX Conference on File and Storage Technologies (FAST 25)}, - year = {2025}, - isbn = {978-1-939133-45-8}, - address = {Santa Clara, CA}, - pages = {155--170}, - url = {https://www.usenix.org/conference/fast25/presentation/qin}, - publisher = {USENIX Association}, - month = feb +@article{qin2024mooncake_arxiv, + title = {Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving}, + author = {Ruoyu Qin and Zheming Li and Weiran He and Mingxing Zhang and Yongwei Wu and Weimin Zheng and Xinran Xu}, + year = {2024}, + url = {https://arxiv.org/abs/2407.00079}, } ``` diff --git a/docs/source/index.md b/docs/source/index.md index 394ea473..04663377 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -23,7 +23,7 @@ Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. Now both the Transfer Engine and Mooncake Store are open-sourced! -This repository also hosts its technical report and the open sourced traces. +This repository also hosts its technical report and the open-sourced traces.

🔄 Updates

@@ -45,12 +45,12 @@ This repository also hosts its technical report and the open sourced traces. - **May 5, 2025**: Supported by Mooncake Team, SGLang release guidance to deploy DeepSeek with PD Disaggregation on 96 H100 GPUs. - **Apr 22, 2025**: LMCache officially supports Mooncake Store as a remote connector. - **Apr 10, 2025**: SGLang officially supports Mooncake Transfer Engine for disaggregated prefilling and KV cache transfer. - - **Mar 7, 2025**: We open sourced the Mooncake Store, a distributed KVCache based on Transfer Engine. vLLM's xPyD disaggregated prefilling & decoding based on Mooncake Store will be released soon. + - **Mar 7, 2025**: We open-sourced the Mooncake Store, a distributed KVCache based on Transfer Engine. vLLM's xPyD disaggregated prefilling & decoding based on Mooncake Store will be released soon. - **Feb 25, 2025**: Mooncake receives the **Best Paper Award** at **FAST 2025**! - **Feb 21, 2025**: The updated traces used in our FAST'25 paper have been released. - **Dec 16, 2024**: vLLM officially supports Mooncake Transfer Engine for disaggregated prefilling and KV cache transfer. - - **Nov 28, 2024**: We open sourced the Transfer Engine, the central component of Mooncake. We also provide two demonstrations of Transfer Engine: a P2P Store and vLLM integration. - - **July 9, 2024**: We open sourced the trace as a jsonl file. + - **Nov 28, 2024**: We open-sourced the Transfer Engine, the central component of Mooncake. We also provide two demonstrations of Transfer Engine: a P2P Store and vLLM integration. + - **July 9, 2024**: We open-sourced the trace as a JSONL file. - **June 27, 2024**: We present a series of Chinese blogs with more discussions on zhihu 1, 2, 3, 4, 5, 6, 7. - **June 26, 2024**: Initial technical report release.