[TE] Support TCP fallback in EFA build and improve EFA documentation (#1523)

When building with USE_EFA=ON, auto_discover is disabled to prevent
RDMA transport installation (QP creation fails on EFA devices). This
means TCP transport is also not installed automatically. Add explicit
TCP transport installation for non-EFA protocols in the EFA build path.

Documentation changes:
- build.md: Add USE_EFA option and clarify USE_CUDA default/purpose
- supported-protocols.md: Add EFA as a supported protocol
- efa_transport.md: Add USE_CUDA=ON to build command, document GPU
  memory requirement

Co-authored-by: whn09 <whn09@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>

docs/source/design/transfer-engine/efa_transport.md

@@ -68,14 +68,14 @@ mkdir build && cd build
 
 cmake .. \
     -DUSE_EFA=ON \
-    -DWITH_TE=ON \
-    -DWITH_STORE=ON \
-    -DBUILD_UNIT_TESTS=ON \
+    -DUSE_CUDA=ON \
     -DCMAKE_BUILD_TYPE=RelWithDebInfo
 
 make -j$(nproc)
 ```
 
+> **Note:** `-DUSE_CUDA=ON` is required when transferring GPU memory (e.g., KV cache in vLLM). Without it, the TCP transport (used as fallback when `mooncake_protocol` is set to `"tcp"`) cannot detect GPU memory and will fail with "Bad address" (EFAULT) errors.
+
 ### 3. Install Python Package
 
 ```bash

docs/source/getting_started/build.md

@@ -150,10 +150,11 @@ cd /Mooncake-main/build/mooncake-transfer-engine/example
 
 ## Advanced Compile Options
 The following options can be used during `cmake ..` to specify whether to compile certain components of Mooncake.
-- `-DUSE_CUDA=[ON|OFF]`: Enable GPU Direct RDMA and NVMe-of support
+- `-DUSE_CUDA=[ON|OFF]`: Enable GPU memory support (GPUDirect RDMA, NVMe-oF, and GPU-aware TCP transport). **Default: OFF.** Required when transferring GPU memory (e.g., KV cache in vLLM disaggregated serving), even when using TCP protocol.
 - `-DUSE_MNNVL=[ON|OFF]`: Enable Multi-Node NVLink transport support, default is OFF. **Note:** `-DUSE_CUDA` is required when `-DUSE_MNNVL` is on.
 - `-DUSE_MUSA=[ON|OFF]`: Enable Moore Threads GPU support via MUSA
 - `-DUSE_HIP=[ON|OFF]`: Enable AMD GPU support via HIP/ROCm
+- `-DUSE_EFA=[ON|OFF]`: Enable AWS Elastic Fabric Adapter transport via libfabric. **Default: OFF.** See [EFA Transport](../design/transfer-engine/efa_transport.md) for details.
 - `-DUSE_INTRA_NVLINK=[ON|OFF]`: Enable intranode nvlink transport
 - `-DUSE_CXL=[ON|OFF]`: Enable CXL support
 - `-DWITH_STORE=[ON|OFF]`: Build Mooncake Store component

docs/source/getting_started/supported-protocols.md

@@ -8,6 +8,7 @@ Mooncake Transfer Engine supports multiple communication protocols for data tran
 |----------|-------------------|----------|-------------------|
 | **tcp** | Standard network | General purpose, works everywhere | ✅ Primary |
 | **rdma** | RDMA-capable NIC | High-performance, low-latency | ✅ Primary |
+| **efa** | AWS EFA-capable instance | High-performance on AWS (libfabric SRD) | ✅ Primary |
 | **nvmeof** | NVMe-oF capable storage | Direct NVMe storage access | ⚠️ Advanced |
 | **nvlink** | NVIDIA MNNVL | Inter-node GPU communication | ⚠️ Advanced |
 | **nvlink_intra** | NVIDIA NVLink | Intra-node GPU communication | ⚠️ Advanced |
@@ -122,6 +123,45 @@ ibv_devices  # List InfiniBand/RDMA devices
 - Configure proper NUMA affinity for optimal performance
 - See [Transfer Engine Benchmark Tuning](../design/transfer-engine/transfer-engine-bench-tuning.md) for detailed optimization
 
+### EFA (AWS Elastic Fabric Adapter)
+
+**Description:** AWS EFA transport using libfabric's Scalable Reliable Datagram (SRD) protocol, providing high-bandwidth RDMA-like performance on AWS instances without traditional RDMA support.
+
+**Use When:**
+- Running on AWS EFA-enabled instances (e.g., p5e.48xlarge, p6-b200.48xlarge, p4d.24xlarge)
+- High-performance networking is required on AWS
+- Traditional RDMA (ibverbs QP) is not supported by the hardware
+
+**Configuration:**
+```python
+# Python API
+engine.initialize(
+    hostname="localhost",
+    metadata_server="P2PHANDSHAKE",
+    protocol="efa",
+    device_name=""
+)
+```
+
+**Build Requirements:**
+```bash
+cmake .. -DUSE_EFA=ON -DUSE_CUDA=ON
+```
+
+> **Note:** `-DUSE_CUDA=ON` is required when transferring GPU memory. Without it, fallback to TCP protocol will fail with "Bad address" errors on GPU buffers.
+
+**Advantages:**
+- High throughput (~170 GB/s with 8 EFA devices, tuned)
+- Bypasses kernel network stack
+- Available on all AWS EFA-enabled instances
+
+**Limitations:**
+- AWS-only
+- Software-emulated RDMA writes (higher CPU overhead than true RDMA)
+- ~88% of RoCE RDMA throughput
+
+**Documentation:** See [EFA Transport](../design/transfer-engine/efa_transport.md) for build instructions, benchmarks, and tuning.
+
 ## Advanced Protocols (C++ Transfer Engine)
 
 The following protocols are available at the C++ Transfer Engine level for specialized use cases. They are not commonly used through the Python API.
@@ -277,6 +317,7 @@ export MOONCAKE_LOCAL_HOSTNAME="node1"
 |----------|---------------------|-------|
 | Development/Testing | tcp | Simple setup, no special hardware |
 | Production Inference | rdma | Best performance and latency |
+| AWS Cloud (EFA instances) | efa | High performance on p5e, p6-b200, p4d, etc. |
 | Cloud Environments | tcp or rdma (if available) | Check cloud provider support |
 | Multi-tier Storage | rdma + nvmeof | Combine protocols for different layers |
 | AMD GPU Clusters | rdma + hip | Use HIP for local GPU communication |

mooncake-integration/transfer_engine/transfer_engine_py.cpp

@@ -205,6 +205,18 @@ int TransferEnginePy::initializeExt(const char *local_hostname,
             return -1;
         }
         LOG(INFO) << "EFA transport installed successfully";
+    } else {
+        // For non-EFA protocols (e.g. TCP), manually install TCP transport
+        // since auto_discover is disabled to prevent RDMA installation
+        // (RDMA QP creation fails on EFA devices).
+        LOG(INFO)
+            << "Installing TCP transport (auto_discover disabled in EFA build)";
+        auto transport = engine_->installTransport("tcp", nullptr);
+        if (!transport) {
+            LOG(ERROR) << "Failed to install TCP transport";
+            return -1;
+        }
+        LOG(INFO) << "TCP transport installed successfully";
     }
 #endif