diff --git a/docs/source/design/transfer-engine/efa_transport.md b/docs/source/design/transfer-engine/efa_transport.md index b6fee161..317b7e5a 100644 --- a/docs/source/design/transfer-engine/efa_transport.md +++ b/docs/source/design/transfer-engine/efa_transport.md @@ -68,14 +68,14 @@ mkdir build && cd build cmake .. \ -DUSE_EFA=ON \ - -DWITH_TE=ON \ - -DWITH_STORE=ON \ - -DBUILD_UNIT_TESTS=ON \ + -DUSE_CUDA=ON \ -DCMAKE_BUILD_TYPE=RelWithDebInfo make -j$(nproc) ``` +> **Note:** `-DUSE_CUDA=ON` is required when transferring GPU memory (e.g., KV cache in vLLM). Without it, the TCP transport (used as fallback when `mooncake_protocol` is set to `"tcp"`) cannot detect GPU memory and will fail with "Bad address" (EFAULT) errors. + ### 3. Install Python Package ```bash diff --git a/docs/source/getting_started/build.md b/docs/source/getting_started/build.md index 3fa92b53..3ec9133c 100644 --- a/docs/source/getting_started/build.md +++ b/docs/source/getting_started/build.md @@ -150,10 +150,11 @@ cd /Mooncake-main/build/mooncake-transfer-engine/example ## Advanced Compile Options The following options can be used during `cmake ..` to specify whether to compile certain components of Mooncake. -- `-DUSE_CUDA=[ON|OFF]`: Enable GPU Direct RDMA and NVMe-of support +- `-DUSE_CUDA=[ON|OFF]`: Enable GPU memory support (GPUDirect RDMA, NVMe-oF, and GPU-aware TCP transport). **Default: OFF.** Required when transferring GPU memory (e.g., KV cache in vLLM disaggregated serving), even when using TCP protocol. - `-DUSE_MNNVL=[ON|OFF]`: Enable Multi-Node NVLink transport support, default is OFF. **Note:** `-DUSE_CUDA` is required when `-DUSE_MNNVL` is on. - `-DUSE_MUSA=[ON|OFF]`: Enable Moore Threads GPU support via MUSA - `-DUSE_HIP=[ON|OFF]`: Enable AMD GPU support via HIP/ROCm +- `-DUSE_EFA=[ON|OFF]`: Enable AWS Elastic Fabric Adapter transport via libfabric. **Default: OFF.** See [EFA Transport](../design/transfer-engine/efa_transport.md) for details. - `-DUSE_INTRA_NVLINK=[ON|OFF]`: Enable intranode nvlink transport - `-DUSE_CXL=[ON|OFF]`: Enable CXL support - `-DWITH_STORE=[ON|OFF]`: Build Mooncake Store component diff --git a/docs/source/getting_started/supported-protocols.md b/docs/source/getting_started/supported-protocols.md index cca70f85..00e4c3b5 100644 --- a/docs/source/getting_started/supported-protocols.md +++ b/docs/source/getting_started/supported-protocols.md @@ -8,6 +8,7 @@ Mooncake Transfer Engine supports multiple communication protocols for data tran |----------|-------------------|----------|-------------------| | **tcp** | Standard network | General purpose, works everywhere | ✅ Primary | | **rdma** | RDMA-capable NIC | High-performance, low-latency | ✅ Primary | +| **efa** | AWS EFA-capable instance | High-performance on AWS (libfabric SRD) | ✅ Primary | | **nvmeof** | NVMe-oF capable storage | Direct NVMe storage access | ⚠️ Advanced | | **nvlink** | NVIDIA MNNVL | Inter-node GPU communication | ⚠️ Advanced | | **nvlink_intra** | NVIDIA NVLink | Intra-node GPU communication | ⚠️ Advanced | @@ -122,6 +123,45 @@ ibv_devices # List InfiniBand/RDMA devices - Configure proper NUMA affinity for optimal performance - See [Transfer Engine Benchmark Tuning](../design/transfer-engine/transfer-engine-bench-tuning.md) for detailed optimization +### EFA (AWS Elastic Fabric Adapter) + +**Description:** AWS EFA transport using libfabric's Scalable Reliable Datagram (SRD) protocol, providing high-bandwidth RDMA-like performance on AWS instances without traditional RDMA support. + +**Use When:** +- Running on AWS EFA-enabled instances (e.g., p5e.48xlarge, p6-b200.48xlarge, p4d.24xlarge) +- High-performance networking is required on AWS +- Traditional RDMA (ibverbs QP) is not supported by the hardware + +**Configuration:** +```python +# Python API +engine.initialize( + hostname="localhost", + metadata_server="P2PHANDSHAKE", + protocol="efa", + device_name="" +) +``` + +**Build Requirements:** +```bash +cmake .. -DUSE_EFA=ON -DUSE_CUDA=ON +``` + +> **Note:** `-DUSE_CUDA=ON` is required when transferring GPU memory. Without it, fallback to TCP protocol will fail with "Bad address" errors on GPU buffers. + +**Advantages:** +- High throughput (~170 GB/s with 8 EFA devices, tuned) +- Bypasses kernel network stack +- Available on all AWS EFA-enabled instances + +**Limitations:** +- AWS-only +- Software-emulated RDMA writes (higher CPU overhead than true RDMA) +- ~88% of RoCE RDMA throughput + +**Documentation:** See [EFA Transport](../design/transfer-engine/efa_transport.md) for build instructions, benchmarks, and tuning. + ## Advanced Protocols (C++ Transfer Engine) The following protocols are available at the C++ Transfer Engine level for specialized use cases. They are not commonly used through the Python API. @@ -277,6 +317,7 @@ export MOONCAKE_LOCAL_HOSTNAME="node1" |----------|---------------------|-------| | Development/Testing | tcp | Simple setup, no special hardware | | Production Inference | rdma | Best performance and latency | +| AWS Cloud (EFA instances) | efa | High performance on p5e, p6-b200, p4d, etc. | | Cloud Environments | tcp or rdma (if available) | Check cloud provider support | | Multi-tier Storage | rdma + nvmeof | Combine protocols for different layers | | AMD GPU Clusters | rdma + hip | Use HIP for local GPU communication | diff --git a/mooncake-integration/transfer_engine/transfer_engine_py.cpp b/mooncake-integration/transfer_engine/transfer_engine_py.cpp index da45791d..b2816913 100644 --- a/mooncake-integration/transfer_engine/transfer_engine_py.cpp +++ b/mooncake-integration/transfer_engine/transfer_engine_py.cpp @@ -205,6 +205,18 @@ int TransferEnginePy::initializeExt(const char *local_hostname, return -1; } LOG(INFO) << "EFA transport installed successfully"; + } else { + // For non-EFA protocols (e.g. TCP), manually install TCP transport + // since auto_discover is disabled to prevent RDMA installation + // (RDMA QP creation fails on EFA devices). + LOG(INFO) + << "Installing TCP transport (auto_discover disabled in EFA build)"; + auto transport = engine_->installTransport("tcp", nullptr); + if (!transport) { + LOG(ERROR) << "Failed to install TCP transport"; + return -1; + } + LOG(INFO) << "TCP transport installed successfully"; } #endif