docs: implement documentation plan for missing Mooncake features

Agent-Logs-Url: https://github.com/kvcache-ai/Mooncake/sessions/3432c219-559e-4d6a-b137-ce729304fc49

Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>

docs/source/deployment/ha-hot-standby.md

@@ -0,0 +1,146 @@
+# Mooncake Store HA Hot Standby
+
+Mooncake Store supports a **Hot Standby** mode that keeps a passive replica of the master metadata in sync with the primary. When the primary fails, the standby can be promoted to take over with minimal downtime and no data loss.
+
+## Architecture
+
+```
+  ┌─────────────────────┐          OpLog stream           ┌─────────────────────┐
+  │   Primary Master    │ ──────────────────────────────▶ │   Standby Master    │
+  │  (MasterService)    │                                  │ (HotStandbyService) │
+  │                     │  ①  Snapshot bootstrap (once)   │                     │
+  │  Oplog  ─────────── │ ──────────────────────────────▶ │  ─── apply oplogs   │
+  │  (OpLogManager)     │  ②  Oplog replication (steady)  │  (OpLogApplier)     │
+  └─────────────────────┘                                  └─────────────────────┘
+           ▲                                                        │
+           │           Leader Election (etcd / Redis)              │  promotion
+           └────────────────────────────────────────────────────────┘
+```
+
+### Phase 1 — Snapshot Bootstrap
+
+On startup, the standby optionally downloads the latest snapshot produced by the primary (see [Snapshot / Restore flags](mooncake-store-deployment-guide.md)). This baseline allows the standby to catch up quickly without replaying the entire oplog history.
+
+Enable snapshot bootstrap via `HotStandbyConfig.enable_snapshot_bootstrap = true`.
+
+### Phase 2 — Oplog Replication
+
+After the snapshot is applied, the standby enters **steady-state replication**: it continuously polls the primary's `OpLogReplicator` for new oplog entries and applies them locally through `OpLogApplier`. The lag is bounded by `max_replication_lag_entries` (default 1000 entries).
+
+### Leader Election
+
+Mooncake uses either **etcd** or **Redis** as the distributed coordination backend for leader election:
+
+- **etcd** (`STORE_USE_ETCD`): Build with `-DSTORE_USE_ETCD=ON`. Set `--etcd_endpoints` on the master.
+- **Redis** (`STORE_USE_REDIS`): Build with `-DSTORE_USE_REDIS=ON`. Set connection details via environment variables (see [Redis HA Backend](mooncake-store-deployment-guide.md#redis-ha-backend)).
+
+When the primary is unresponsive for longer than `--client_ttl` seconds, the standby acquires the leader lease and promotes itself.
+
+### Promotion
+
+Promotion is handled by `StandbyStateMachine`. On promotion the standby:
+1. Stops polling the (now-dead) primary.
+2. Registers itself in the metadata service under the primary's endpoint address.
+3. Begins accepting client RPCs.
+
+Clients that retry their connections will automatically reconnect to the promoted standby without any application-level change.
+
+## Configuration Reference
+
+### `HotStandbyConfig` Fields
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `standby_id` | — | Unique identifier for this standby instance |
+| `primary_address` | — | `ip:port` of the primary master's RPC endpoint |
+| `replication_port` | 0 (auto) | Port for the oplog replication channel |
+| `verification_interval_sec` | 30 | How often to verify sync status |
+| `max_replication_lag_entries` | 1000 | Alert threshold for oplog lag |
+| `enable_verification` | true | Enable periodic lag verification |
+| `enable_snapshot_bootstrap` | false | Download latest snapshot before replicating |
+| `enable_oplog_following` | true | Enable steady-state oplog following |
+| `oplog_store_type` | default | Where to persist the local oplog copy |
+| `oplog_store_root_dir` | default | Root directory for local oplog storage |
+| `oplog_poll_interval_ms` | default | Polling interval for new oplog entries |
+
+### Master Startup Flags (HA-related)
+
+The following flags from [Mooncake Store Deployment Guide](mooncake-store-deployment-guide.md) apply to HA setups:
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--enable_ha` | `false` | Enable HA mode (requires etcd or Redis) |
+| `--etcd_endpoints` | — | Semicolon-separated etcd endpoints |
+| `--client_ttl` | `10` s | How long a client (or standby) has to re-ping before considered dead |
+| `--cluster_id` | `mooncake_cluster` | Cluster ID used for persistence keys in HA mode |
+
+## Deployment Example
+
+### Step 1: Start the Primary Master (with HA enabled)
+
+```bash
+mooncake_master \
+    --rpc_port=50051 \
+    --enable_ha=true \
+    --etcd_endpoints="http://etcd-0:2379;http://etcd-1:2379;http://etcd-2:2379" \
+    --cluster_id=prod-cluster \
+    --client_ttl=15 \
+    --enable_snapshot=true \
+    --snapshot_backend_type=local \
+    --snapshot_interval_seconds=300
+```
+
+### Step 2: Start the Standby Master
+
+The standby is a separate `mooncake_master` process with identical flags **plus** the standby-specific ones:
+
+```bash
+MOONCAKE_SNAPSHOT_LOCAL_PATH=/data/mooncake_snapshots \
+mooncake_master \
+    --rpc_port=50052 \
+    --enable_ha=true \
+    --etcd_endpoints="http://etcd-0:2379;http://etcd-1:2379;http://etcd-2:2379" \
+    --cluster_id=prod-cluster \
+    --client_ttl=15 \
+    --enable_snapshot_restore=true \
+    --snapshot_backend_type=local
+```
+
+The standby will:
+1. Detect that another master holds the leader lease via etcd.
+2. Start the `HotStandbyService` with snapshot bootstrap enabled.
+3. Download the latest snapshot from the primary and apply it.
+4. Begin following the oplog in steady state.
+
+### Step 3: Verify Sync Status
+
+The standby logs its sync status periodically. Look for lines like:
+
+```
+[HotStandbyService] applied_seq_id=12345 lag=0 entries
+[HotStandbyService] verification OK: primary_seq_id=12345 standby_seq_id=12345
+```
+
+A non-zero lag that is growing indicates the standby cannot keep up with the primary write rate — consider reducing the primary write load or increasing `oplog_poll_interval_ms`.
+
+### Step 4: Simulate a Failover
+
+Kill the primary master process. Within `--client_ttl` seconds, the standby will:
+
+1. Detect the primary is down (etcd lease expiry).
+2. Acquire the leader lease.
+3. Promote itself and start serving client RPCs on its own `--rpc_port`.
+
+Point clients to the standby's address or use a DNS/load-balancer alias.
+
+## Tuning Tips
+
+- **`max_replication_lag_entries`**: Lower values trigger earlier alerts but may cause false positives during write bursts. The default of 1000 is conservative for most workloads.
+- **`verification_interval_sec`**: Increase to reduce overhead if the standby's oplog applier is a bottleneck.
+- **`enable_snapshot_bootstrap`**: Always enable in production to reduce time-to-sync after a standby restart.
+- **`client_ttl`**: Set this equal to or slightly higher than your deployment's network heartbeat interval. A value too low causes spurious failovers; too high delays recovery.
+
+## See Also
+
+- [Mooncake Store Deployment Guide](mooncake-store-deployment-guide) — snapshot flags, S3 backend, Redis HA
+- [Mooncake Store Design](../design/mooncake-store)

docs/source/deployment/monitoring.md

@@ -0,0 +1,130 @@
+# Monitoring Mooncake with Prometheus and Grafana
+
+Mooncake Master exposes a Prometheus-compatible `/metrics` endpoint. The `monitoring/` directory in the repository contains a ready-to-use Docker Compose stack that wires Prometheus and Grafana together with a pre-built dashboard for `mooncake_master`.
+
+## Quick Start
+
+### Prerequisites
+
+- Docker and Docker Compose installed on the monitoring host.
+- `mooncake_master` accessible from the monitoring host.
+
+### Step 1: Start the Monitoring Stack
+
+```bash
+cd monitoring
+docker-compose up -d
+```
+
+This starts two containers:
+- **Prometheus** — scrapes metrics from `mooncake_master` every 15 s.
+- **Grafana** — pre-configured with a Prometheus data source and a sample dashboard.
+
+### Step 2: Open the UIs
+
+| UI | URL | Credentials |
+|----|-----|-------------|
+| Prometheus | <http://localhost:9090> | — |
+| Grafana | <http://localhost:3000> | `admin` / `admin` |
+
+Navigate to **Grafana → Dashboards** to find the pre-built `mooncake_master` dashboard.
+
+### Step 3: Start `mooncake_master` with Metrics Enabled
+
+```bash
+./build/mooncake_master \
+    --metrics_port=9003 \
+    --enable_metric_reporting=true \
+    --rpc_port=50051
+```
+
+Verify the metrics endpoint is live:
+
+```bash
+curl -s http://localhost:9003/metrics | head -20
+```
+
+You should see Prometheus-format lines such as:
+
+```
+# HELP mooncake_master_kv_object_count Total number of KV objects in the store
+# TYPE mooncake_master_kv_object_count gauge
+mooncake_master_kv_object_count 4096
+```
+
+Check **Prometheus → Status → Targets** — the `mooncake-master` job should show `UP`.
+
+## Configuration Files
+
+| File | Description |
+|------|-------------|
+| `monitoring/docker-compose.yml` | Service definitions for Prometheus and Grafana |
+| `monitoring/prometheus/prometheus.yml` | Prometheus scrape configuration |
+| `monitoring/grafana/` | Grafana provisioning (data source + dashboard JSON) |
+
+### Prometheus Scrape Target
+
+By default, `prometheus.yml` scrapes `host.docker.internal:9003`. On Linux, `host.docker.internal` may not be available — add the following to the `prometheus` service in `docker-compose.yml`:
+
+```yaml
+extra_hosts:
+  - "host.docker.internal:host-gateway"
+```
+
+To scrape a remote `mooncake_master`, change the target in `prometheus/prometheus.yml`:
+
+```yaml
+scrape_configs:
+  - job_name: mooncake-master
+    static_configs:
+      - targets:
+          - "10.0.0.1:9003"   # replace with actual master host
+```
+
+## Available Metrics
+
+The `/metrics/summary` endpoint (human-readable) and `/metrics` endpoint (Prometheus format) expose the following categories:
+
+| Category | Example Metric | Description |
+|----------|---------------|-------------|
+| KV objects | `mooncake_master_kv_object_count` | Total objects in the store |
+| Memory | `mooncake_master_segment_free_bytes` | Free bytes per segment |
+| Eviction | `mooncake_master_eviction_total` | Eviction events |
+| Tasks | `mooncake_master_pending_tasks` | Pending transfer tasks |
+| RPC | `mooncake_master_rpc_requests_total` | Total RPC requests served |
+
+Browse all available metrics via:
+
+```bash
+curl -s http://localhost:9003/metrics/summary
+```
+
+## Grafana Alerts (Optional)
+
+To set up alerts in Grafana:
+
+1. Open the dashboard and click the panel you want to alert on.
+2. Choose **Edit → Alert → Create alert rule**.
+3. Example: alert when `mooncake_master_segment_free_bytes` drops below 10 % of total.
+
+## Multi-Master Monitoring
+
+To monitor multiple masters in one Prometheus instance, add multiple targets:
+
+```yaml
+scrape_configs:
+  - job_name: mooncake-masters
+    static_configs:
+      - targets:
+          - "master-0:9003"
+          - "master-1:9003"
+          - "master-2:9003"
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: instance
+```
+
+## See Also
+
+- [Mooncake Store Deployment Guide](mooncake-store-deployment-guide) — master startup flags including `--metrics_port`
+- [Mooncake Store HA Hot Standby](ha-hot-standby) — monitoring standby lag

docs/source/deployment/mooncake-store-deployment-guide.md

@@ -153,6 +153,126 @@ This sets the log level for yalantinglibs (including coro_rpc and coro_http) to
 
 Available log levels: trace, debug, info, warn (or warning), error, and critical.
 
+## S3 Snapshot Backend
+
+When `--snapshot_backend_type=s3` is set, Mooncake Master stores snapshots in an S3-compatible object store.
+
+### Required Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `AWS_ACCESS_KEY_ID` | S3 access key (or IAM role credential) |
+| `AWS_SECRET_ACCESS_KEY` | S3 secret key |
+| `AWS_DEFAULT_REGION` | AWS region (e.g., `us-east-1`) |
+| `MOONCAKE_SNAPSHOT_S3_BUCKET` | Target S3 bucket name |
+| `MOONCAKE_SNAPSHOT_S3_PREFIX` | (Optional) Key prefix inside the bucket |
+
+For IAM role-based authentication (recommended in production), omit `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` — the SDK will use the instance profile automatically.
+
+### Bucket Naming Conventions
+
+- Create a dedicated bucket for Mooncake snapshots (e.g., `my-org-mooncake-snapshots`).
+- Enable versioning on the bucket for additional safety.
+- Set a lifecycle rule to expire old snapshot objects beyond the `--snapshot_retention_count` limit.
+
+> **Warning:** The S3 bucket is a managed directory. Do not store other data under the same key prefix as Mooncake snapshots — old snapshots are deleted automatically during cleanup.
+
+### Example Startup Command (S3 backend)
+
+```bash
+export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
+export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
+export AWS_DEFAULT_REGION=us-east-1
+export MOONCAKE_SNAPSHOT_S3_BUCKET=my-org-mooncake-snapshots
+export MOONCAKE_SNAPSHOT_S3_PREFIX=prod-cluster/
+
+mooncake_master \
+    --rpc_port=50051 \
+    --enable_snapshot=true \
+    --snapshot_backend_type=s3 \
+    --snapshot_interval_seconds=300 \
+    --snapshot_retention_count=3 \
+    --enable_snapshot_restore=true
+```
+
+### Restore Procedure
+
+On master restart with `--enable_snapshot_restore=true`, the master:
+1. Lists snapshots in the configured S3 bucket/prefix.
+2. Downloads the latest snapshot.
+3. Applies the snapshot to restore in-memory metadata.
+4. Resumes serving client RPCs.
+
+If `--snapshot_backup_dir` is also set, the downloaded snapshot is additionally saved locally as a fallback.
+
+---
+
+## Redis HA Backend
+
+Mooncake Store HA mode can use **Redis** instead of etcd for distributed leader election and cluster coordination.
+
+### Build Requirement
+
+Redis HA support must be enabled at compile time:
+
+```bash
+cmake .. \
+    -DSTORE_USE_REDIS=ON \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+make -j$(nproc)
+```
+
+This adds the `STORE_USE_REDIS` compile definition and links `hiredis`.
+
+### Connection Configuration
+
+| Environment Variable | Description |
+|----------------------|-------------|
+| `MC_REDIS_PASSWORD` | Redis AUTH password (omit if no auth is configured) |
+| `MC_REDIS_DB_INDEX` | Redis database index (default: `0`) |
+
+The Redis server URL is passed via `--etcd_endpoints` using the `redis://` scheme:
+
+```bash
+mooncake_master \
+    --rpc_port=50051 \
+    --enable_ha=true \
+    --etcd_endpoints="redis://10.0.0.10:6379" \
+    --cluster_id=prod-cluster \
+    --client_ttl=15
+```
+
+For a Redis Cluster or Sentinel setup:
+
+```bash
+# Redis Cluster (semicolon-separated nodes)
+--etcd_endpoints="redis://10.0.0.10:6379;redis://10.0.0.11:6379;redis://10.0.0.12:6379"
+```
+
+### Key Hash-Tag Conventions
+
+Mooncake uses Redis hash tags to ensure that all keys for a given cluster land on the same Redis cluster slot. The tag is derived from `--cluster_id` and sanitised via `SanitizeHashTagComponent()` (which strips characters that are illegal inside `{}`). For example:
+
+- `--cluster_id=prod-cluster` → Redis keys use `{prod-cluster}` as the hash tag.
+- All leader election keys, oplog entries, and session heartbeats share this tag.
+
+Do not use the same Redis instance / keyspace for other applications without ensuring their keys use different hash tags.
+
+### ConnectRedis() Helper
+
+The internal `ha::backends::redis::ConnectRedis()` helper reads `MC_REDIS_PASSWORD` and `MC_REDIS_DB_INDEX` from the environment automatically. No additional code changes are needed; configure these variables before starting the master.
+
+### Redis vs etcd
+
+| Feature | Redis | etcd |
+|---------|-------|------|
+| Build flag | `STORE_USE_REDIS=ON` | `STORE_USE_ETCD=ON` |
+| URL scheme | `redis://` | `etcd://` or bare `host:port` |
+| Cluster support | Redis Cluster / Sentinel | etcd cluster |
+| Recommended for | Environments already running Redis | New deployments |
+
+---
+
 ## Quick Tips
 
 - Scale `--rpc_thread_num` with available CPU cores and workload.
@@ -167,4 +287,7 @@ Available log levels: trace, debug, info, warn (or warning), error, and critical
 :maxdepth: 1
 
 ssd-offload
+ha-hot-standby
+monitoring
+multi-tier-storage
 :::

docs/source/deployment/multi-tier-storage.md

@@ -0,0 +1,140 @@
+# Multi-Tier Storage
+
+Mooncake Store supports a **multi-tier storage hierarchy** that extends the in-memory KV cache with a persistent distributed filesystem (DFS) layer. This enables workloads that cannot fit entirely in GPU/CPU memory to spill objects to a fast DFS backend (e.g., 3FS or any POSIX-compatible filesystem).
+
+## Tier Overview
+
+| Tier | Storage Medium | Access Path | Typical Latency |
+|------|---------------|-------------|----------------|
+| **G1** | GPU VRAM (on each client node) | Direct VRAM read/write | < 1 µs |
+| **G2** | CPU DRAM (on each client node) | RDMA or local memcpy | 1–10 µs |
+| **G3** | DFS (shared filesystem) | POSIX / 3FS USRBIO | 100 µs–ms |
+
+Objects flow down the hierarchy as memory pressure increases (G1 → G2 → G3) and are promoted back up on a cache hit.
+
+## Architecture
+
+```
+  Client Node 1                         Shared DFS
+ ┌──────────────────┐                  ┌──────────────┐
+ │ G1: GPU VRAM     │                  │ G3: 3FS /    │
+ │ G2: CPU DRAM     │ ── POSIX/USRBIO──│    NFS /     │
+ └──────────────────┘                  │    GPFS …    │
+                                        └──────────────┘
+  Client Node 2
+ ┌──────────────────┐
+ │ G1: GPU VRAM     │
+ │ G2: CPU DRAM     │
+ └──────────────────┘
+
+   ↑ All clients share G3 via the DFS mount point
+```
+
+The master (`mooncake_master`) manages the G3 segment as a special **DFS segment** registered at startup. Client nodes write KV objects to the DFS when instructed by the master's eviction policy.
+
+## Configuration
+
+### Master Flags
+
+The following flags enable and configure G3 DFS storage (from [Mooncake Store Deployment Guide](mooncake-store-deployment-guide)):
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--root_fs_dir` | — (empty, G3 disabled) | DFS mount directory. When set, the master registers a DFS-backed segment at startup. |
+| `--global_file_segment_size` | `INT64_MAX` | Maximum bytes the DFS segment may occupy. Set to the available DFS capacity. |
+
+### Client Flags
+
+Client nodes do not need additional flags for DFS; the master instructs clients to write to the DFS segment automatically during eviction. Ensure the DFS mount is accessible from every client node at the path specified in `--root_fs_dir`.
+
+## Deployment Example
+
+### Step 1: Mount the DFS on All Nodes
+
+```bash
+# Example: mount 3FS (see docs/source/getting_started/plugin-usage/3FS-USRBIO-Plugin.md)
+mount -t 3fs <3fs_server>:/kvcache /mnt/3fs
+
+# Example: mount NFS
+mount -t nfs <nfs_server>:/kvcache /mnt/kvcache
+```
+
+### Step 2: Start the Master with G3 Enabled
+
+```bash
+mooncake_master \
+    --rpc_port=50051 \
+    --enable_http_metadata_server=true \
+    --http_metadata_server_port=8080 \
+    --root_fs_dir=/mnt/3fs/mooncake \
+    --global_file_segment_size=107374182400   # 100 GB
+```
+
+The master will create the DFS segment directory under `--root_fs_dir` and register it as a G3 segment.
+
+### Step 3: Start Client Nodes (unchanged)
+
+```bash
+mooncake_client \
+    --master_server_address=<master_ip>:50051 \
+    --host=<client_ip> \
+    --protocol=rdma \
+    --device_names=mlx5_0 \
+    --global_segment_size="32GB" \
+    --metadata_server="P2PHANDSHAKE"
+```
+
+### Step 4: Connect the Application
+
+```python
+from mooncake.store import MooncakeDistributedStore
+
+store = MooncakeDistributedStore()
+store.setup(
+    local_hostname="<client_ip>",
+    metadata_server="P2PHANDSHAKE",
+    global_segment_size=32 * 1024 ** 3,
+    local_buffer_size=4 * 1024 ** 3,
+    protocol="rdma",
+    device_name="mlx5_0",
+    master_server_address="<master_ip>:50051",
+)
+```
+
+G3 eviction and promotion are fully transparent to the application.
+
+## 3FS USRBIO Integration
+
+For maximum DFS throughput, Mooncake supports the **3FS USRBIO** (User-space Block IO) interface, which bypasses the kernel page cache and achieves near-NVMe throughput from user space.
+
+To enable:
+1. Build Mooncake with `USE_3FS=ON` and ensure the 3FS USRBIO plugin is installed (see [3FS USRBIO Plugin Guide](../getting_started/plugin-usage/3FS-USRBIO-Plugin)).
+2. Mount 3FS with USRBIO support enabled.
+3. Pass `--root_fs_dir` pointing to the 3FS mount.
+
+When USRBIO is not available, Mooncake falls back to standard POSIX `read`/`write` calls.
+
+## Eviction Policy
+
+The master controls which objects are evicted from G1/G2 to G3 using the same watermark-based policy as SSD offload:
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--eviction_high_watermark_ratio` | `0.95` | Memory usage fraction that triggers eviction |
+| `--eviction_ratio` | `0.05` | Fraction of objects evicted per cycle |
+| `--allow_evict_soft_pinned_objects` | `true` | Whether soft-pinned objects can be evicted to G3 |
+
+Hard-pinned objects are never evicted to G3.
+
+## Performance Tips
+
+- **Place the DFS close to clients**: Use a high-bandwidth interconnect (e.g., InfiniBand / 100GbE) between clients and the DFS storage nodes.
+- **Use 3FS USRBIO** for best throughput when writing large KV cache tensors.
+- **Size `--global_file_segment_size`** conservatively: set it to 80–90 % of actual available DFS capacity to leave room for snapshots and other data.
+- **Monitor G3 usage** via the `/metrics` endpoint: `mooncake_master_dfs_segment_used_bytes`.
+
+## See Also
+
+- [SSD Offload](ssd-offload) — local NVMe offload (single-node)
+- [3FS USRBIO Plugin](../getting_started/plugin-usage/3FS-USRBIO-Plugin) — 3FS setup and USRBIO configuration
+- [Mooncake Store Deployment Guide](mooncake-store-deployment-guide) — all master flags

docs/source/design/mooncake-pg.md

@@ -0,0 +1,135 @@
+# Mooncake Process Group (mooncake-pg)
+
+## Overview
+
+`mooncake-pg` is a custom PyTorch distributed process-group backend built on top of the Mooncake Transfer Engine. It provides `torch.distributed` collective and point-to-point (P2P) communication primitives that exploit high-speed interconnects (RDMA, NVLink, etc.) while remaining fully compatible with the standard `torch.distributed` API.
+
+It was designed for large-scale expert-parallelism and disaggregated inference scenarios such as those described in the [Kimi K2 deployment blog post](https://lmsys.org/blog/2025-07-20-k2-large-scale-ep/), where Mooncake replaced NCCL for all-to-all expert routing across 128 H200 GPUs.
+
+## Architecture
+
+```
+┌──────────────────────────────────────────────────────────────────┐
+│                       Python / torch.distributed                  │
+│  dist.send / dist.recv / dist.broadcast / dist.all_reduce / …    │
+└───────────────────────────────┬──────────────────────────────────┘
+                                │
+                    ┌───────────▼────────────┐
+                    │    MooncakeBackend     │  (c10d::Backend subclass)
+                    │  (mooncake_backend.h)  │
+                    └───┬──────────┬─────────┘
+                        │          │
+          ┌─────────────▼──┐  ┌───▼────────────────┐
+          │ ConnectionPoller│  │  MooncakeWorker     │
+          │(connection_     │  │  (mooncake_worker.  │
+          │  poller.h)      │  │   cu / .cuh)        │
+          └────────────────┘  └───────┬─────────────┘
+                                      │
+                           ┌──────────▼──────────┐
+                           │    P2PProxy          │
+                           │    (p2p_proxy.h)     │
+                           └──────────┬──────────┘
+                                      │
+                           ┌──────────▼──────────┐
+                           │  TransferEngine      │
+                           │  (RDMA / NVLink /    │
+                           │   TCP …)             │
+                           └─────────────────────┘
+```
+
+### MooncakeBackend
+
+`MooncakeBackend` is registered with PyTorch as two backends:
+- `"mooncake"` — for CUDA tensors.
+- `"mooncake-cpu"` — for CPU tensors.
+
+It subclasses `c10d::Backend` and overrides all collective operations: `send`, `recv`, `broadcast`, `allreduce`, `allgather`, `alltoall`, `scatter`, `reduce`, `gather`, `barrier`, and `batch_isend_irecv`.
+
+Under the hood it maintains:
+- A singleton `TransferEngine` instance shared across all backend instances within the same process, initialized with the RDMA/NVLink device configuration.
+- A `MooncakeWorkerManager` (one per backend instance) that owns a per-GPU CUDA worker thread queue.
+- A `P2PProxy` that serialises and routes send/recv payloads through the Transfer Engine.
+- A `ConnectionPoller` that continuously polls Transfer Engine for completed transfers and signals waiting Work objects.
+
+`MooncakeBackendOptions` carries an `activeRanks_` tensor (a boolean mask) that enables **elastic group membership**: ranks can be added dynamically via `extendGroupSizeTo()`, queried with `getActiveRanks()`, or recovered after a failure with `recoverRanks()`.
+
+### ConnectionPoller
+
+`ConnectionPoller` runs on a dedicated thread. It polls `TransferEngine::getTransferStatus()` for pending batch IDs and sets the `std::atomic<bool> completed` flag on the associated `MooncakeP2PWork` object, which unblocks `Work::wait()` in the calling Python thread.
+
+### MooncakeWorker (CUDA Worker)
+
+Each GPU rank spawns a `MooncakeWorker` CUDA thread (via a CUDA stream / host thread) to execute device-side work items — primarily memory copies involving GPU tensors (`cudaMemcpyAsync`, slicing, casting). This separation keeps the Python GIL thread free during GPU-side operations.
+
+### P2PProxy
+
+`P2PProxy` translates high-level send/recv requests (tensor + rank + tag) into one or more `TransferRequest` entries that the Transfer Engine can process. It handles:
+- Buffer registration (delegated to `TransferEngine::registerLocalMemory`).
+- Multi-slice splitting for large tensors.
+- Tag-based demultiplexing when multiple P2P streams are in flight.
+
+## Supported Operations
+
+| Operation | Supported | Notes |
+|-----------|-----------|-------|
+| `send` / `recv` | ✅ | Single-tensor P2P |
+| `batch_isend_irecv` | ✅ | Async batch P2P |
+| `broadcast` | ✅ | Rank 0 → all |
+| `allreduce` | ✅ | SUM only |
+| `allgather` | ✅ | |
+| `allgather_into_tensor` | ✅ | |
+| `alltoall` | ✅ | Equal-size exchange |
+| `alltoall_base` | ✅ | Variable-size exchange |
+| `scatter` | ✅ | |
+| `gather` | ✅ | |
+| `reduce` | ✅ | SUM only |
+| `barrier` | ✅ | Via allreduce on a dummy tensor |
+| Sparse tensors | ❌ | Not supported |
+| Non-SUM reduce ops | ❌ | Only SUM is implemented |
+
+## Elastic Group Membership
+
+`MooncakeBackend` exposes several extension APIs beyond the standard `c10d::Backend` interface:
+
+| Function | Description |
+|----------|-------------|
+| `getActiveRanks()` | Returns a boolean `torch.Tensor` of size `world_size` |
+| `getNumSyncedRanks()` | Number of currently synced (alive) ranks |
+| `extendGroupSizeTo(size)` | Grow the process group without restart |
+| `getPeerState(ranks)` | Check liveness of a list of ranks |
+| `recoverRanks(ranks)` | Re-admit previously failed ranks |
+| `getPreferredHca(location)` | Query optimal RDMA HCA for a memory location |
+
+These enable fault-tolerant and elastic training scenarios where nodes may join, leave, or fail during a run.
+
+## Build
+
+`mooncake-pg` requires a CUDA-capable build. It is an optional component enabled via the main CMake configuration:
+
+```bash
+mkdir build && cd build
+cmake .. \
+    -DWITH_PG=ON \
+    -DUSE_CUDA=ON \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+make -j$(nproc) mooncake_pg
+```
+
+The build produces `mooncake_pg.cpython-*.so`, which can be installed with pip via the wheel:
+
+```bash
+pip install -e ../mooncake-wheel --no-build-isolation
+```
+
+## Use Cases
+
+- **Expert parallelism (MoE)**: All-to-all expert token routing between GPU ranks using RDMA/NVLink instead of NCCL, as demonstrated in the Kimi K2 deployment.
+- **Pipeline parallelism**: Low-latency activation transfers between pipeline stages.
+- **Disaggregated prefill/decode**: Offloading KV-cache transfers from NCCL to Mooncake's topology-aware engine.
+- **Elastic training**: Dynamic rank management without process restart.
+
+## See Also
+
+- [Mooncake PG Usage Guide](../getting_started/examples/mooncake-pg-usage)
+- [Transfer Engine Architecture](transfer-engine/index)
+- [Kimi K2 deployment blog post](https://lmsys.org/blog/2025-07-20-k2-large-scale-ep/)

docs/source/design/transfer-engine/barex_transport.md

@@ -0,0 +1,81 @@
+# Barex Transport
+
+The Barex transport (`BarexTransport`) is a **bare-metal RDMA extension** that provides an alternative RDMA data path with different queue pair management and flow-control characteristics compared to the standard `RdmaTransport`.
+
+## Overview
+
+While the standard `RdmaTransport` targets high-throughput, multi-NIC environments with endpoint pooling and topology-aware path selection, `BarexTransport` is designed for scenarios that require:
+
+- **Dedicated queue pairs per connection** rather than shared endpoint pools.
+- **Fine-grained flow control** via a countdown-latch mechanism that gates completion acknowledgements.
+- **Simplified connection lifecycle** for bare-metal or HPC environments where connections are long-lived and the overhead of dynamic endpoint management is undesirable.
+
+Both transports use the same ibverbs interface (`infiniband/verbs.h`) and can be compiled into the same binary; the choice is made at runtime via the protocol string.
+
+## Hardware Requirements
+
+- RDMA-capable NIC (InfiniBand or RoCE), same as `RdmaTransport`.
+- `libibverbs` installed on the host.
+
+## Build
+
+```bash
+cmake .. \
+    -DUSE_BAREX=ON \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+make -j$(nproc)
+```
+
+`USE_BAREX` links `barex_transport` into the `transfer_engine` shared library. It can be combined with `USE_CUDA`, `USE_MNNVL`, and other feature flags.
+
+## Usage
+
+### Protocol String
+
+Use `"barex"` as the protocol string:
+
+```python
+from mooncake.engine import TransferEngine
+
+te = TransferEngine()
+te.initialize("node1:12345", "P2PHANDSHAKE", "barex", "mlx5_0")
+```
+
+### When to Use Barex Instead of rdma
+
+| Scenario | Recommended Transport |
+|----------|-----------------------|
+| High-throughput, many concurrent connections, topology-aware routing | `rdma` |
+| Long-lived dedicated connections, HPC / bare-metal, fixed topology | `barex` |
+| AWS EFA | `efa` |
+
+## Internal Design
+
+### BarexContext
+
+Each `BarexContext` object manages the ibverbs resources for a single local RDMA NIC:
+- Protection Domain (`ibv_pd`)
+- Completion Queue (`ibv_cq`)
+- Memory Regions (`ibv_mr`) for each registered buffer
+
+### Queue Pair Management
+
+Unlike `RdmaTransport` which uses endpoint pooling with the SIEVE eviction algorithm, `BarexTransport` allocates a dedicated Queue Pair (QP) per connection pair. QPs are transitioned to the `RTS` (Ready to Send) state during the connection handshake and remain open for the lifetime of the Transfer Engine instance.
+
+### CountDownLatch
+
+`BarexTransport` uses a `CountDownLatch` synchronisation primitive to track in-flight send operations. When a batch of transfer requests is submitted, the latch count is set to the number of outstanding operations. Each CQ completion decrements the count; the initiating thread blocks until the count reaches zero.
+
+This approach simplifies correctness at the cost of some throughput compared to the fully asynchronous pipeline in `RdmaTransport`.
+
+## Limitations
+
+- Not designed for environments with many dynamically appearing / disappearing peers (the static QP model does not scale to thousands of short-lived connections).
+- Does not support topology-aware multi-NIC path selection.
+- Requires `USE_BAREX=ON` at compile time; cannot be selected at runtime if not compiled in.
+
+## See Also
+
+- [Transfer Engine Architecture](index) — standard RDMA transport
+- [Supported Protocols](../../getting_started/supported-protocols)
+- [EFA Transport](efa_transport) — alternative for AWS EFA

docs/source/design/transfer-engine/cxl_transport.md

@@ -0,0 +1,97 @@
+# CXL Transport
+
+The CXL transport (`CxlTransport`) enables memory transfers over **Compute Express Link (CXL)**, a high-speed interconnect that allows CPUs and accelerators to share a unified memory pool across the PCIe bus.
+
+## Overview
+
+CXL memory pooling disaggregates physical DRAM from compute nodes: a CXL memory expander can be mounted into the address space of one or more host CPUs, making remote memory appear as local DRAM. `CxlTransport` leverages this to move data between CXL-attached memory regions across nodes without involving a network NIC.
+
+Typical use cases include:
+- **Memory disaggregation**: Offload KV cache tensors to a large, shared CXL memory pool that multiple inference nodes can access.
+- **Bandwidth aggregation**: Use CXL memory expanders to increase the total memory bandwidth available to a single node.
+
+## Hardware Requirements
+
+- CXL 2.0 or later capable host CPU (e.g., Intel Xeon Scalable 4th Gen / AMD EPYC 9004).
+- CXL memory expander (e.g., Samsung CMM-D, Micron CZ120, Ayar Labs).
+- CXL device must be mounted and visible as a NUMA node (verify with `numactl -H`).
+
+## Build
+
+```bash
+cmake .. \
+    -DUSE_CXL=ON \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+make -j$(nproc)
+```
+
+`USE_CXL` compiles `CxlTransport` into the `transfer_engine` library. No additional runtime library is required beyond the standard Linux kernel CXL driver (enabled by default in Linux 5.15+).
+
+## Usage
+
+### Protocol String
+
+Use `"cxl"` as the protocol string:
+
+```python
+from mooncake.engine import TransferEngine
+
+te = TransferEngine()
+te.initialize("localhost:12345", "P2PHANDSHAKE", "cxl", "")
+```
+
+### Memory Registration
+
+CXL memory regions are registered with a `cpu:N` location tag where `N` is the NUMA node corresponding to the CXL device:
+
+```bash
+# Find the CXL memory NUMA node
+numactl -H | grep -A5 "node distances"
+```
+
+```python
+import ctypes
+
+# Allocate on CXL NUMA node (e.g., node 2)
+buf = ctypes.create_string_buffer(256 * 1024 * 1024)
+te.register_memory(ctypes.addressof(buf), len(buf), "cpu:2")
+```
+
+### Base Address
+
+`CxlTransport` exposes a `getCxlBaseAddr()` method that returns the virtual base address of the mapped CXL region. This is useful when constructing scatter-gather lists that reference CXL memory directly.
+
+### CXL Segment Allocation
+
+`CxlTransport` maintains an internal segment table with a fixed number of local segments (controlled by `allocateLocalSegmentID()`). Each registered CXL buffer occupies one slot. The segment limit is set at compile time; contact the Mooncake team if you need to increase it.
+
+## Limitations
+
+- CXL memory regions must be on the same host or connected through a CXL switch; this transport does not use a network NIC.
+- CXL 1.1 devices (non-pooled) appear as standard DRAM NUMA nodes — no special transport is needed for them.
+- CXL 2.0 pooling (shared memory across multiple hosts) requires a CXL switch and OS-level support (Linux 6.6+).
+
+## Troubleshooting
+
+### CXL device not visible as NUMA node
+
+```bash
+ls /sys/bus/cxl/devices/
+daxctl list
+```
+
+If the device appears as a DAX device but not a NUMA node, use `daxctl reconfigure-device` to online it as system RAM.
+
+### Permission denied on CXL device file
+
+```bash
+sudo chmod 660 /dev/dax0.0
+sudo chown root:$(id -gn) /dev/dax0.0
+```
+
+## See Also
+
+- [Transfer Engine Architecture](index)
+- [Supported Protocols](../../getting_started/supported-protocols)
+- [SSD Offload](../../deployment/ssd-offload) — multi-tier storage with NVMe
+- [Multi-Tier Storage](../../deployment/multi-tier-storage) — G1/G2/G3 tier configuration

docs/source/design/transfer-engine/hip_transport.md

@@ -0,0 +1,81 @@
+# HIP Transport (AMD ROCm)
+
+The HIP transport (`HipTransport`) enables GPU-to-GPU data transfers on **AMD ROCm** platforms, using either IPC handles or Shareable handles for intra-node transfers between AMD GPUs.
+
+## Overview
+
+`HipTransport` is the AMD equivalent of the CUDA-based transfer paths. It is designed for **intra-node** GPU communication: moving data between AMD GPU VRAM buffers on the same host without routing through system memory or a network NIC.
+
+The transport uses the HIP runtime (`hip_runtime.h`) and supports two handle types for mapping peer GPU memory:
+
+| Handle Type | Use Case |
+|-------------|----------|
+| **IPC handle** (`hipIpcMemHandle_t`) | Mapping GPU memory from another process on the same host |
+| **Shareable handle** | Used when IPC handles are unavailable or for cross-device mappings |
+
+## Hardware Requirements
+
+- AMD GPU with ROCm 5.0 or later (e.g., MI200 / MI300 series).
+- Peer access must be supported between the source and destination GPUs (`hipDeviceCanAccessPeer`).
+
+## Build
+
+```bash
+cmake .. \
+    -DUSE_HIP=ON \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+make -j$(nproc)
+```
+
+The `USE_HIP` flag enables HIPification of the Transfer Engine sources (`hipify_files`) and links `hip::host` and the ROCm runtime library.
+
+> **Note:** `USE_HIP` and `USE_CUDA` are mutually exclusive. Do not enable both in the same build.
+
+## Usage
+
+### Protocol String
+
+Use `"hip"` as the protocol string:
+
+```python
+from mooncake.engine import TransferEngine
+
+te = TransferEngine()
+te.initialize("localhost:12345", "P2PHANDSHAKE", "hip", "")
+```
+
+### Memory Registration
+
+Register AMD GPU memory using the standard `TransferEngine::registerLocalMemory` API. Specify the location as `"cuda:N"` (the HIP transport reuses the same location tag convention as the CUDA transport, since HIP mirrors the CUDA device numbering):
+
+```python
+import ctypes, mooncake
+
+te = TransferEngine()
+te.initialize("localhost:12345", "P2PHANDSHAKE", "hip", "")
+
+# Allocate 256 MB on GPU 0 (HIP device 0)
+buf = mooncake.allocate_gpu_memory(256 * 1024 * 1024, device=0)
+te.register_memory(buf, 256 * 1024 * 1024, "cuda:0")
+```
+
+### Environment Variables
+
+`HipTransport` respects the same topology and configuration environment variables as the RDMA transport:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `MC_MS_AUTO_DISC` | `1` | Auto-discover GPU topology |
+| `MC_MS_FILTERS` | — | NIC / device whitelist |
+
+## Limitations
+
+- Intra-node only: `HipTransport` does not support transfers across network links. Combine with `"rdma"` (via the standard RDMA transport) for cross-node transfers in a heterogeneous setup.
+- IPC handles require both processes to be on the same OS instance and the same AMD GPU driver version.
+- Inter-GPU peer access must be enabled at the OS / driver level.
+
+## See Also
+
+- [Supported Protocols](../../getting_started/supported-protocols) — protocol selection guide
+- [Transfer Engine Architecture](index)
+- [ROCm Documentation](https://rocm.docs.amd.com/)

docs/source/design/transfer-engine/index.md

@@ -330,6 +330,20 @@ ascend_transport
 heterogeneous_ascend
 :::
 
+## Accelerator and Specialized Transports
+
+:::{toctree}
+:maxdepth: 1
+
+nvlink_transport
+nvlink_intra_transport
+hip_transport
+cxl_transport
+barex_transport
+mlu_transport
+maca_transport
+:::
+
 ## Benchmark and Tuning Guide
 
 :::{toctree}

docs/source/design/transfer-engine/maca_transport.md

@@ -0,0 +1,131 @@
+# MACA Transport (MetaX / Muxi)
+
+Mooncake Transfer Engine supports **MetaX (Muxi) MACA** GPU accelerators. MACA (MetaX Architecture for Computing Acceleration) is the GPU compute platform from MetaX Integrated Circuits.
+
+## Overview
+
+MACA support in Mooncake mirrors the CUDA code path: memory allocations on MetaX GPUs are registered with the Transfer Engine and transferred via the RDMA NIC that is closest to the device in the PCIe topology. MACA-specific runtime libraries replace the CUDA runtime.
+
+## Hardware Requirements
+
+- MetaX (Muxi) GPU accelerator.
+- MACA runtime installed at `MACA_HOME` (default `/opt/maca`).
+- RDMA-capable NIC in the same server.
+
+## Build
+
+```bash
+cmake .. \
+    -DUSE_MACA=ON \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+make -j$(nproc)
+```
+
+### Environment Variables (build-time)
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `MACA_HOME` | `/opt/maca` | Root of the MACA installation (headers and libraries) |
+
+CMake resolves `MACA_INCLUDE_DIR` as `${MACA_HOME}/include`.
+
+### Runtime Libraries
+
+By default, `USE_MACA=ON` links:
+
+```cmake
+mcruntime      # MACA compute runtime
+mxc-runtime64  # MetaX cross-platform runtime
+rt             # POSIX real-time library
+```
+
+Override the library list at configure time if your MACA installation uses different names:
+
+```bash
+cmake .. \
+    -DUSE_MACA=ON \
+    -DMACA_RUNTIME_LIBS="mcruntime;mxc-runtime64;rt"
+```
+
+## Usage
+
+### Protocol String
+
+Use `"rdma"` as the protocol string. MACA support extends the RDMA path with MACA-aware memory registration:
+
+```python
+from mooncake.engine import TransferEngine
+
+te = TransferEngine()
+te.initialize("node1:12345", "P2PHANDSHAKE", "rdma", "mlx5_0")
+```
+
+Register MACA GPU memory:
+
+```python
+# Allocate MACA GPU memory via mcMalloc (or framework allocator)
+maca_ptr = mc_malloc(256 * 1024 * 1024)
+
+# Register with Transfer Engine using "cuda:N" location tag
+# (MACA devices follow the same device-numbering convention as CUDA)
+te.register_memory(maca_ptr, 256 * 1024 * 1024, "cuda:0")
+```
+
+### Topology Discovery
+
+When `USE_MACA=ON`, the topology discovery module enumerates MetaX GPUs via the MACA runtime and maps them to their nearest RDMA NIC(s). The resulting `priority_matrix` entries use `"cuda:N"` location tags (shared convention with CUDA devices).
+
+## Environment Variables (run-time)
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `MC_MS_AUTO_DISC` | `1` | Auto-discover MACA GPU topology. Set `0` to disable. |
+| `MC_MS_FILTERS` | — | Comma-separated NIC whitelist |
+| `MACA_HOME` | `/opt/maca` | Used at runtime if the MACA shared libraries are not on `LD_LIBRARY_PATH` |
+
+Ensure the MACA runtime libraries are on `LD_LIBRARY_PATH`:
+
+```bash
+export LD_LIBRARY_PATH=$MACA_HOME/lib:$LD_LIBRARY_PATH
+```
+
+## Build with Both MACA and Other Features
+
+`USE_MACA=ON` can be combined with other flags such as `USE_EFA`, `USE_CXL`, or `USE_BAREX`. It cannot be combined with `USE_CUDA=ON` (both define the same CUDA-like runtime aliases).
+
+## Limitations
+
+- Cross-node MACA GPU transfers require an RDMA NIC; direct MACA peer-to-peer transfers between nodes are not supported.
+- MACA and CUDA cannot be enabled in the same build.
+
+## Troubleshooting
+
+### MACA runtime library not found
+
+```
+error while loading shared libraries: libmcruntime.so
+```
+
+Add the MACA library directory to `LD_LIBRARY_PATH`:
+
+```bash
+export LD_LIBRARY_PATH=/opt/maca/lib:$LD_LIBRARY_PATH
+```
+
+### Incorrect `MACA_HOME`
+
+```
+CMake Error: MACA_INCLUDE_DIR not found
+```
+
+Set `MACA_HOME` explicitly:
+
+```bash
+cmake .. -DUSE_MACA=ON -DMACA_HOME=/path/to/maca
+```
+
+## See Also
+
+- [Transfer Engine Architecture](index) — RDMA transport and topology-aware path selection
+- [Supported Protocols](../../getting_started/supported-protocols)
+- [Build Guide](../../getting_started/build) — `USE_MACA` flag in the build matrix

docs/source/design/transfer-engine/mlu_transport.md

@@ -0,0 +1,121 @@
+# MLU Transport (Cambricon)
+
+Mooncake Transfer Engine supports **Cambricon MLU** (Machine Learning Unit) accelerators through an extension to the standard `RdmaTransport`. There is no separate `mlu` protocol string; MLU support adds MLU-aware memory registration and topology discovery on top of the normal RDMA data path.
+
+## Overview
+
+Cambricon MLU devices expose their on-device memory via the **DMA-BUF** kernel interface, the same mechanism used by NVIDIA GPUDirect RDMA. When `USE_MLU=ON` is enabled, Mooncake:
+
+1. Detects MLU devices during topology discovery.
+2. Registers MLU VRAM buffers with the RDMA NIC using DMA-BUF handles.
+3. Performs RDMA read/write operations directly to/from MLU VRAM — no intermediate CPU bounce buffer is needed.
+
+The result is a zero-copy transfer path between MLU device memory and remote DRAM / another MLU.
+
+## Hardware Requirements
+
+- Cambricon MLU accelerator (MLU370, MLU590, etc.).
+- Cambricon Neuware SDK installed (`neuware` package).
+- RDMA-capable NIC (InfiniBand or RoCE) in the same server.
+- Linux kernel with DMA-BUF support (5.6+).
+
+## Build
+
+```bash
+cmake .. \
+    -DUSE_MLU=ON \
+    -DUSE_CUDA=OFF \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+make -j$(nproc)
+```
+
+### Environment Variables (build-time)
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `NEUWARE_HOME` | `/usr/local/neuware` | Path to the Neuware installation |
+| `NEUWARE_ROOT` | — | Alternative Neuware root (checked if `NEUWARE_HOME` is unset) |
+
+The CMake scripts use these to locate `libcnrt`, `libcndrv`, and the MLU headers.
+
+### Runtime Libraries
+
+`USE_MLU=ON` links the following Neuware libraries into `transfer_engine`:
+
+| Library | Purpose |
+|---------|---------|
+| `cnrt` | Cambricon Runtime (memory allocation, device management) |
+| `cndrv` | Cambricon Driver (low-level device access, DMA-BUF export) |
+
+## Usage
+
+### Protocol String
+
+Use the standard `"rdma"` protocol string. MLU support is transparent at the API level:
+
+```python
+from mooncake.engine import TransferEngine
+
+te = TransferEngine()
+te.initialize("node1:12345", "P2PHANDSHAKE", "rdma", "mlx5_0")
+```
+
+To register MLU memory:
+
+```python
+import ctypes
+
+# Allocate MLU memory via cnrt (or use a framework allocator)
+mlu_ptr = cnrt_malloc(256 * 1024 * 1024)
+
+# Register with Transfer Engine — specify the MLU device location
+te.register_memory(mlu_ptr, 256 * 1024 * 1024, "mlu:0")
+```
+
+The location tag `"mlu:N"` tells the topology engine to associate the buffer with MLU device N and select an RDMA NIC that has direct PCIe connectivity to that device.
+
+### Topology Discovery
+
+When `USE_MLU=ON` and MLU devices are present, the Transfer Engine topology module:
+- Enumerates all MLU devices and their PCIe BDF addresses.
+- Maps each MLU to the nearest RDMA NIC(s) based on PCIe topology.
+- Populates the `priority_matrix` with `"mlu:N"` location entries.
+
+This ensures that DMA-BUF transfers use the NIC closest to the MLU, minimising PCIe switch hops.
+
+## Limitations
+
+- MLU support uses the RDMA data path; it does not add a dedicated protocol endpoint.
+- Cross-node MLU-to-MLU transfers require an RDMA NIC on each node.
+- Intra-node MLU-to-MLU transfers use the RDMA loopback path (not a direct NVLink / PCIe peer copy); direct peer-to-peer MLU copy is not yet implemented in Mooncake.
+- `USE_MLU=ON` and `USE_CUDA=ON` can coexist in the same build (the DMA-BUF registration code is independent), but has not been tested extensively in mixed-accelerator environments.
+
+## Troubleshooting
+
+### DMA-BUF export fails
+
+```
+cnDrvMemExportToDmaBuf: CNDRV_ERROR_NO_SUPPORT
+```
+
+Ensure the Cambricon driver version supports DMA-BUF export (driver ≥ 4.9) and that the kernel DMA-BUF interface is enabled:
+
+```bash
+zcat /proc/config.gz | grep DMA_BUF
+# CONFIG_DMA_BUF=y
+```
+
+### MLU not detected in topology
+
+```bash
+cnrt-diagnose   # should list MLU devices
+lspci | grep Cambricon
+```
+
+Verify that `NEUWARE_HOME` points to the correct Neuware installation.
+
+## See Also
+
+- [Transfer Engine Architecture](index) — RDMA transport and topology-aware path selection
+- [Supported Protocols](../../getting_started/supported-protocols) — `rdma` protocol usage with MLU
+- [Cambricon Neuware Documentation](https://developer.cambricon.com)

docs/source/design/transfer-engine/nvlink_intra_transport.md

@@ -0,0 +1,68 @@
+# Intra-Node NVLink Transport
+
+The Intra-Node NVLink transport (`IntraNodeNvlinkTransport`) enables **zero-copy GPU-to-GPU data transfers within a single host** using the NVLink high-speed interconnect, without going through the PCIe bus or system memory.
+
+## Overview
+
+Modern NVIDIA servers (DGX A100/H100/H200) connect all GPUs on the same node via NVLink. For within-node transfers this provides substantially higher bandwidth and lower latency than PCIe-based copies or RDMA loopback.
+
+`IntraNodeNvlinkTransport` uses CUDA IPC handles and the **UBShmem Fabric Allocator** to map peer GPU memory directly into the local GPU's address space, then issues `cudaMemcpyAsync` on the NVLink path.
+
+## Hardware Requirements
+
+- NVIDIA GPUs with intra-node NVLink (e.g., A100 / H100 / H200 SXM variants in DGX systems).
+- CUDA 11.0+ with peer access support (`cudaDeviceEnablePeerAccess`).
+
+## Build
+
+```bash
+cmake .. \
+    -DUSE_INTRA_NVLINK=ON \
+    -DUSE_CUDA=ON \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+make -j$(nproc)
+```
+
+`USE_INTRA_NVLINK` enables the `IntraNodeNvlinkTransport` class in the `transfer_engine` library.
+
+## Usage
+
+### Protocol String
+
+Use `"nvlink_intra"` as the protocol string:
+
+```python
+from mooncake.engine import TransferEngine
+
+te = TransferEngine()
+te.initialize("localhost:12345", "P2PHANDSHAKE", "nvlink_intra", "cuda:0")
+```
+
+### Automatic Selection
+
+When `MooncakeDistributedStore` is used with `protocol="rdma"` and both the source and destination are on the same node, the topology-aware path selector may automatically prefer the intra-node NVLink path over RDMA when `USE_INTRA_NVLINK` is enabled and peer access is available.
+
+## Memory Registration
+
+`IntraNodeNvlinkTransport` uses the **UBShmem Fabric Allocator**, which:
+1. Creates a shareable CUDA IPC memory handle via `cudaIpcGetMemHandle`.
+2. Exports the handle to other CUDA contexts on the same host via a shared-memory segment.
+3. Maps the remote handle into the local GPU's address space with `cudaIpcOpenMemHandle`.
+
+Only memory in the VRAM of GPUs that are NVLink-connected can be mapped this way. CPU (DRAM) transfers fall through to the standard memcpy path.
+
+## Comparison with Inter-Node NVLink
+
+| Feature | Intra-Node (`nvlink_intra`) | Inter-Node (`nvlink`) |
+|---------|-----------------------------|-----------------------|
+| Scope | Single host, multiple GPUs | Multiple hosts (MNNVL fabric) |
+| Build flag | `USE_INTRA_NVLINK=ON` | `USE_MNNVL=ON` |
+| Memory mechanism | CUDA IPC handles | CUDA Virtual Memory Management + MNNVL fabric export |
+| Bandwidth | Full NVLink bandwidth (e.g., 900 GB/s total on H100 SXM) | Full MNNVL bandwidth |
+| Hardware needed | Any DGX / server with NVLink switch | MNNVL-capable nodes |
+
+## See Also
+
+- [NVLink Transport (MNNVL / Inter-Node)](nvlink_transport) — cross-node NVLink transfers
+- [Supported Protocols](../../getting_started/supported-protocols) — protocol selection guide
+- [Transfer Engine Architecture](index)

docs/source/design/transfer-engine/nvlink_transport.md

@@ -0,0 +1,83 @@
+# NVLink Transport (MNNVL / Inter-Node)
+
+The NVLink transport (`NvlinkTransport`) enables high-bandwidth, low-latency GPU-to-GPU data transfers across nodes using **NVIDIA Multi-Node NVLink (MNNVL)**. It bypasses the PCIe bus and the network stack entirely, delivering bandwidth that approaches the raw NVLink fabric speed.
+
+## Overview
+
+MNNVL is available on NVIDIA platforms that physically link GPUs across multiple nodes via NVLink cables (e.g., DGX SuperPOD / DGX GB200 NVL72). Within such a fabric, all GPUs share a unified address space exposed through the NVLink Allocator, making remote GPU memory directly addressable from any node in the fabric.
+
+`NvlinkTransport` registers GPU memory into the NVLink fabric via `cuMemCreate` / `cuMemMap` (CUDA Virtual Memory Management) and performs transfers through direct remote memory writes — no RDMA NIC is involved.
+
+## Hardware Requirements
+
+- NVIDIA MNNVL-capable hardware (e.g., H100 / H200 / B200 in a NVL72 configuration).
+- CUDA 12.0+ with NVLink fabric support (`cuMemGetAllocationGranularity` with `CU_MEM_ALLOC_GRANULARITY_RECOMMENDED`).
+- NVLink fabric initialised by the system firmware before the application starts.
+
+## Build
+
+Enable MNNVL support at CMake configure time:
+
+```bash
+cmake .. \
+    -DUSE_MNNVL=ON \
+    -DUSE_CUDA=ON \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+make -j$(nproc)
+```
+
+The `USE_MNNVL` flag compiles `NvlinkTransport` into the `transfer_engine` library.
+
+## Usage
+
+### Protocol String
+
+Use `"nvlink"` as the protocol string when initialising the Transfer Engine:
+
+```python
+from mooncake.engine import TransferEngine
+
+te = TransferEngine()
+te.initialize("node1:12345", "P2PHANDSHAKE", "nvlink", "")
+```
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `MC_FORCE_MNNVL` | `false` | Force NVLink even when RDMA NICs are present. When RDMA HCAs are detected and `MC_FORCE_MNNVL` is not set, the engine prefers RDMA. Set `MC_FORCE_MNNVL=true` to override. |
+
+### When to Use
+
+- Use `"nvlink"` only when the deployment hardware has MNNVL connectivity and you want maximum GPU-to-GPU bandwidth between nodes.
+- On clusters without MNNVL hardware, fall back to `"rdma"` or `"tcp"`.
+- For **intra-node** NVLink transfers (within a single host), use `"nvlink_intra"` instead (see [Intra-Node NVLink Transport](nvlink_intra_transport)).
+
+## Memory Registration
+
+`NvlinkTransport` uses a custom **NVLink Allocator** that:
+1. Allocates GPU memory via CUDA Virtual Memory Management (`cuMemCreate`).
+2. Exports the allocation handle through the NVLink fabric (`cuMulticastAddMemory` / `cuMemExportToShareableHandle`).
+3. Registers the allocation with the Transfer Engine metadata service so remote peers can map it.
+
+All memory registered through `TransferEngine::registerLocalMemory` with a `cuda:N` location tag is automatically handled by the NVLink Allocator when `NvlinkTransport` is active.
+
+## Tuning Tips
+
+- **Buffer alignment**: Allocate buffers in multiples of the NVLink granularity returned by `cuMemGetAllocationGranularity`. Misaligned allocations will be rounded up internally.
+- **Pinning**: Keep GPU buffers pinned for the lifetime of a Transfer Engine session; frequent re-registration degrades performance.
+- **Topology**: MNNVL works best when the NVLink fabric is fully populated. Partial fabric configurations will fall back to slower paths for non-adjacent GPUs.
+
+## Test
+
+```bash
+./build/mooncake-transfer-engine/tests/nvlink_transport_test
+```
+
+The test suite requires MNNVL hardware. On systems without MNNVL, the test will skip automatically.
+
+## See Also
+
+- [Intra-Node NVLink Transport](nvlink_intra_transport) — within-node NVLink transfers
+- [Supported Protocols](../../getting_started/supported-protocols) — protocol selection guide
+- [Transfer Engine Architecture](index)

docs/source/getting_started/examples/mooncake-pg-usage.md

@@ -0,0 +1,253 @@
+# Mooncake Process Group — Usage Guide
+
+This guide shows how to build `mooncake-pg`, register it with PyTorch distributed, and use it as a drop-in replacement for NCCL in collective and point-to-point workloads.
+
+## Prerequisites
+
+| Requirement | Version |
+|-------------|---------|
+| CUDA | 11.8 or later |
+| PyTorch | 2.1 or later |
+| RDMA hardware | InfiniBand / RoCE / NVLink (TCP fallback available) |
+| Mooncake | Built with `-DWITH_PG=ON -DUSE_CUDA=ON` |
+
+## Installation
+
+### Step 1: Build Mooncake with PG support
+
+```bash
+git clone https://github.com/kvcache-ai/Mooncake.git
+cd Mooncake
+git submodule update --init --recursive
+
+mkdir build && cd build
+cmake .. \
+    -DWITH_PG=ON \
+    -DUSE_CUDA=ON \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+make -j$(nproc)
+```
+
+### Step 2: Install the Python wheel
+
+```bash
+# Copy built extension into the wheel directory
+cp mooncake-pg/mooncake_pg.cpython-*.so ../mooncake-wheel/mooncake/
+
+# Install with pip
+cd ..
+pip install -e mooncake-wheel --no-build-isolation
+```
+
+### Step 3: Verify the import
+
+```python
+import mooncake_pg   # no error = installed correctly
+import torch.distributed as dist
+```
+
+## Initializing the Process Group
+
+`mooncake-pg` registers two backends with PyTorch:
+- `"mooncake"` for CUDA tensors.
+- `"mooncake-cpu"` for CPU tensors.
+
+Use the standard `torch.distributed.init_process_group` call:
+
+```python
+import torch
+import torch.distributed as dist
+
+# Standard torchrun / mpirun launch sets MASTER_ADDR, MASTER_PORT,
+# RANK, WORLD_SIZE automatically.
+dist.init_process_group(
+    backend="mooncake",   # or "mooncake-cpu" for CPU-only
+    init_method="env://",
+)
+
+rank = dist.get_rank()
+world_size = dist.get_world_size()
+print(f"[rank {rank}/{world_size}] process group initialized")
+```
+
+The process group creation triggers `MooncakeBackend` construction, which:
+1. Initialises the shared `TransferEngine` singleton (first call only, subsequent calls reuse it).
+2. Registers local GPU memory buffers.
+3. Starts the `ConnectionPoller` background thread.
+
+## Example: All-Reduce
+
+```python
+import torch
+import torch.distributed as dist
+
+dist.init_process_group(backend="mooncake", init_method="env://")
+rank = dist.get_rank()
+
+# Create a tensor on the local GPU
+device = torch.device(f"cuda:{rank}")
+tensor = torch.ones(1024, device=device) * rank
+
+# In-place allreduce (SUM)
+dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
+
+# On rank 0 with world_size=4: tensor == [0+1+2+3] * 1024 == 6144
+print(f"[rank {rank}] allreduce result[0] = {tensor[0].item()}")
+dist.destroy_process_group()
+```
+
+## Example: Point-to-Point (P2P)
+
+```python
+import torch
+import torch.distributed as dist
+
+dist.init_process_group(backend="mooncake", init_method="env://")
+rank = dist.get_rank()
+world_size = dist.get_world_size()
+
+device = torch.device(f"cuda:{rank}")
+N = 1024 * 1024  # 4 MB
+
+if rank == 0:
+    tensor = torch.arange(N, dtype=torch.float32, device=device)
+    dist.send(tensor, dst=1, tag=42)
+    print("[rank 0] sent tensor")
+elif rank == 1:
+    tensor = torch.empty(N, dtype=torch.float32, device=device)
+    dist.recv(tensor, src=0, tag=42)
+    print(f"[rank 1] received tensor[0]={tensor[0].item()}")
+
+dist.destroy_process_group()
+```
+
+## Example: All-to-All (Expert Parallelism)
+
+This pattern is the primary use case for `mooncake-pg` in MoE (Mixture-of-Experts) models, where tokens are routed to expert GPUs.
+
+```python
+import torch
+import torch.distributed as dist
+
+dist.init_process_group(backend="mooncake", init_method="env://")
+rank = dist.get_rank()
+world_size = dist.get_world_size()
+
+device = torch.device(f"cuda:{rank}")
+tokens_per_rank = 128
+hidden_dim = 4096
+
+# Each rank sends a different chunk to every other rank
+input_tensor = torch.randn(world_size * tokens_per_rank, hidden_dim, device=device)
+output_tensor = torch.empty_like(input_tensor)
+
+dist.all_to_all_single(output_tensor, input_tensor)
+print(f"[rank {rank}] all_to_all complete, output shape={output_tensor.shape}")
+
+dist.destroy_process_group()
+```
+
+## Example: Batch Async P2P (batch_isend_irecv)
+
+```python
+import torch
+import torch.distributed as dist
+
+dist.init_process_group(backend="mooncake", init_method="env://")
+rank = dist.get_rank()
+device = torch.device(f"cuda:{rank}")
+N = 1024
+
+peer = (rank + 1) % dist.get_world_size()
+send_tensor = torch.ones(N, device=device) * rank
+recv_tensor = torch.zeros(N, device=device)
+
+ops = [
+    dist.P2POp(dist.isend, send_tensor, peer),
+    dist.P2POp(dist.irecv, recv_tensor, peer),
+]
+works = dist.batch_isend_irecv(ops)
+for w in works:
+    w.wait()
+
+print(f"[rank {rank}] received from {peer}: {recv_tensor[0].item()}")
+dist.destroy_process_group()
+```
+
+## Launch Commands
+
+### torchrun (recommended)
+
+```bash
+# 4 GPUs on 1 node
+torchrun \
+    --nproc_per_node=4 \
+    --master_addr=127.0.0.1 \
+    --master_port=29500 \
+    your_script.py
+
+# 16 GPUs across 2 nodes (run on each node)
+torchrun \
+    --nproc_per_node=8 \
+    --nnodes=2 \
+    --node_rank=0 \          # 1 on second node
+    --master_addr=10.0.0.1 \
+    --master_port=29500 \
+    your_script.py
+```
+
+### mpirun
+
+```bash
+mpirun -np 8 \
+    -x MASTER_ADDR=10.0.0.1 \
+    -x MASTER_PORT=29500 \
+    python your_script.py
+```
+
+## Elastic Group Membership
+
+`mooncake-pg` exposes Python helpers for fault-tolerant setups via `mooncake_pg`:
+
+```python
+import mooncake_pg
+import torch.distributed as dist
+
+dist.init_process_group(backend="mooncake", init_method="env://")
+pg = dist.group.WORLD
+
+# Check which ranks are alive
+active = mooncake_pg.get_active_ranks(pg)           # torch.BoolTensor
+n_synced = mooncake_pg.get_num_synced_ranks(pg)      # int
+
+# Dynamically grow the group
+mooncake_pg.extend_group_size_to(pg, new_size=16)
+
+# Check specific peer liveness
+states = mooncake_pg.get_peer_state(pg, [2, 5, 7])  # list[bool]
+
+# Re-admit a recovered rank
+mooncake_pg.recover_ranks(pg, [5])
+```
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `MOONCAKE_MASTER` | — | Mooncake master address (`ip:port`) |
+| `MOONCAKE_PROTOCOL` | `rdma` | Transfer protocol (`rdma`, `tcp`, `nvlink`, …) |
+| `MOONCAKE_DEVICE` | auto | RDMA/HCA device name (e.g., `mlx5_0`) |
+| `MC_METADATA_SERVER` | `P2PHANDSHAKE` | Metadata server URL |
+| `MC_FORCE_MNNVL` | `false` | Force MNNVL even when RDMA NICs are present |
+
+## Limitations
+
+- Only `SUM` is supported for reduce operations (`allreduce`, `reduce`).
+- Sparse tensors are not supported.
+- Each `send`/`recv` call must transfer a single tensor; use `batch_isend_irecv` for multiple tensors.
+
+## See Also
+
+- [Mooncake PG Design](../../design/mooncake-pg)
+- [Transfer Engine](../../design/transfer-engine/index)
+- [Supported Protocols](../supported-protocols)

docs/source/getting_started/examples/rl-training-integration.md

@@ -0,0 +1,156 @@
+# Reinforcement Learning Training Integration
+
+`mooncake-rl` demonstrates how to use `MooncakeDistributedStore` as a zero-copy data bus between **rollout engines** (inference workers) and **training engines** (gradient workers) in a distributed reinforcement-learning setup.
+
+The key idea is to decouple the rollout and training phases: rollout engines write experience batches into Mooncake Store under a well-known key, and training engines fetch those batches concurrently — without any explicit synchronisation barrier or shared memory.
+
+## Motivation
+
+In a typical RL-from-human-feedback (RLHF) or online RL pipeline:
+
+```
+ Rollout Engines                    Training Engines
+ (inference, e.g. SGLang/vLLM)      (gradient step, e.g. PyTorch)
+ ┌──────────────────┐                ┌─────────────────────┐
+ │  generate()      │ ─── RDMA ───▶  │  train(key)         │
+ │  put_tensor(key) │                │  get_tensor(key)    │
+ └──────────────────┘                └─────────────────────┘
+```
+
+Data flows over RDMA (or TCP fallback) via `MooncakeDistributedStore`, bypassing the CPU for GPU-to-GPU transfers and avoiding Python serialisation overhead.
+
+## Example: `rl_samples.py`
+
+The reference implementation lives in [`mooncake-rl/examples/rl_samples.py`](../../../../mooncake-rl/examples/rl_samples.py). It provides a minimal, runnable mock of the full RL loop:
+
+| Class | Role |
+|-------|------|
+| `RolloutEngine` | Generates random `(obs, action, reward)` samples |
+| `RolloutController` | Manages dataset state and connects to Mooncake Store |
+| `RolloutManager` | Orchestrates rollout engines; writes samples to the Store |
+| `TrainActor` | Consumes samples and performs a dummy training step |
+| `TrainGroup` | Coordinates training actors; reads samples from the Store |
+
+### Data Flow Walk-Through
+
+1. **RolloutManager.generate(rollout_id)** — each rollout engine produces a sample dict `{obs, action, reward}`. All samples for this rollout step are assembled into a list and written into the Store:
+
+   ```python
+   # RolloutController uses an RDMA-initialised Store client
+   self.rollout_client.put_tensor(str(rollout_id), rollout_samples)
+   ```
+
+2. **TrainGroup.train(rollout_id, key)** — the training side fetches those samples and distributes them across training actors:
+
+   ```python
+   samples = self.training_client.get_tensor(rollout_key)
+   for actor, sample in zip(self.actor_handlers, samples):
+       actor.train(sample)
+   ```
+
+3. **Training loop** — the top-level `train()` function alternates generation, training, evaluation, and checkpoint saving:
+
+   ```python
+   for rollout_id in range(start, num_rollout):
+       key = rollout_manager.generate(rollout_id)  # write to Store
+       actor_model.train(rollout_id, key)           # read from Store
+       actor_model.update_weights()                 # sync weights to rollout
+   ```
+
+## Running the Example
+
+### Prerequisites
+
+- A running Mooncake metadata server (e.g., the HTTP server or etcd).
+- A running `mooncake_master` (for distributed Store coordination).
+- Mooncake Python wheel installed (`pip install mooncake-transfer-engine`).
+
+### Start the Metadata Server
+
+```bash
+cd mooncake-transfer-engine/example/http-metadata-server-python
+pip install aiohttp
+python bootstrap_server.py &
+```
+
+### Start the Master
+
+```bash
+./build/mooncake_master \
+    --rpc_port=50051 \
+    --enable_http_metadata_server=true \
+    --http_metadata_server_host=0.0.0.0 \
+    --http_metadata_server_port=8080
+```
+
+### Run the RL Example
+
+```bash
+python mooncake-rl/examples/rl_samples.py \
+    --num_rollout=10 \
+    --num_train_actor=2 \
+    --num_rollout_actor=2 \
+    --save_interval=5 \
+    --eval_interval=5 \
+    --model_path=./checkpoints
+```
+
+## Connecting to a Real Metadata Server
+
+The example initialises two `MooncakeDistributedStore` clients — one on the rollout side and one on the training side — with different local hostnames and NIC device names:
+
+```python
+# Rollout client (on rollout node / process)
+self.rollout_client = MooncakeDistributedStore()
+self.rollout_client.setup(
+    "localhost:12346",                     # local_hostname:port
+    "http://localhost:8080/metadata",      # metadata_server URL
+    512 * 1024 * 1024,                     # global_segment_size (512 MB)
+    128 * 1024 * 1024,                     # local_buffer_size (128 MB)
+    "rdma",                                # protocol
+    "erdma_0",                             # NIC device (e.g. mlx5_0)
+    "localhost:50051",                     # mooncake_master address
+)
+
+# Training client (on training node / process)
+self.training_client = MooncakeDistributedStore()
+self.training_client.setup(
+    "localhost:12345",
+    "http://localhost:8080/metadata",
+    512 * 1024 * 1024,
+    128 * 1024 * 1024,
+    "rdma",
+    "erdma_1",
+    "localhost:50051",
+)
+```
+
+In a real deployment:
+- Replace `localhost` with actual hostnames or IPs.
+- Replace `erdma_0` / `erdma_1` with the NIC names from `ibv_devices`.
+- Use separate metadata server and master addresses accessible by both sets of nodes.
+
+## Key/Value Conventions
+
+By default, the example uses the rollout ID as the string key:
+
+```python
+key = str(rollout_id)               # e.g. "0", "1", "42"
+self.rollout_client.put_tensor(key, rollout_samples)
+```
+
+In production, use a structured key scheme to avoid collisions across concurrent runs:
+
+```python
+key = f"run:{run_id}/rollout:{rollout_id}"
+```
+
+## Integration with Real RL Frameworks
+
+This pattern is directly applicable to frameworks such as [THUDM/slime](https://github.com/THUDM/slime), [veRL](https://github.com/volcengine/verl), and [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF). Replace the dummy `generate()` and `train()` implementations with real policy inference (SGLang / vLLM) and gradient update (PyTorch FSDP / DeepSpeed) calls, keeping the `put_tensor` / `get_tensor` calls as the data handoff point.
+
+## See Also
+
+- [MooncakeDistributedStore Python API](../../python-api-reference/mooncake-store)
+- [Mooncake Store Deployment Guide](../../deployment/mooncake-store-deployment-guide)
+- [SSD Offload](../../deployment/ssd-offload)

docs/source/getting_started/rust-bindings.md

@@ -0,0 +1,195 @@
+# Rust Bindings
+
+Mooncake provides Rust bindings for two components:
+
+| Crate | Path | Purpose |
+|-------|------|---------|
+| `mooncake_store` | `mooncake-store/rust/` | Distributed KV cache store client |
+| Transfer Engine (Rust bench) | `mooncake-transfer-engine/rust/` | Transfer Engine benchmark / integration |
+
+## Mooncake Store Rust Bindings
+
+### Overview
+
+`mooncake_store` is a Rust crate that wraps the Mooncake Store C ABI (`store_c.h`) using `bindgen`-generated bindings. It exposes a safe, idiomatic Rust API for storing and retrieving byte slices (KV cache objects) over RDMA.
+
+### Build Requirements
+
+The crate requires the compiled Mooncake Store shared library and its public headers. These are produced by the standard CMake build:
+
+```bash
+mkdir build && cd build
+cmake .. \
+    -DWITH_STORE_RUST=ON \   # ON by default
+    -DUSE_CUDA=ON \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+cmake --build . --target build_mooncake_store_rust
+```
+
+The CMake target sets the necessary Rust environment variables and runs `cargo build` automatically.
+
+If you want to build the crate independently (e.g., in your own workspace after a CMake install), set:
+
+```bash
+export MOONCAKE_STORE_LIB_DIR=/path/to/build/mooncake-store
+export MOONCAKE_STORE_INCLUDE_DIR=/path/to/Mooncake/mooncake-store/include
+cargo build
+```
+
+### Adding as a Dependency
+
+After building, you can depend on the crate from a local path:
+
+```toml
+# Cargo.toml
+[dependencies]
+mooncake_store = { path = "/path/to/Mooncake/mooncake-store/rust" }
+```
+
+### API Reference
+
+```rust
+use mooncake_store::{MooncakeStore, ReplicateConfig, StoreError};
+
+// 1. Create a store handle
+let store = MooncakeStore::new()?;
+
+// 2. Connect to a running Mooncake master
+store.setup(
+    "node1",                           // local_hostname
+    "http://10.0.0.1:8080/metadata",   // metadata_server
+    512 << 20,                         // global_segment_size (512 MiB)
+    128 << 20,                         // local_buffer_size  (128 MiB)
+    "rdma",                            // protocol: "tcp" | "rdma" | …
+    "mlx5_0",                          // device_name ("" for auto)
+    "10.0.0.1:50051",                  // mooncake_master address
+)?;
+
+// 3. Store a value
+store.put("my-key", b"hello, mooncake!", None)?;
+
+// 4. Store with replication options
+let config = ReplicateConfig {
+    replica_num: 2,
+    with_soft_pin: true,
+    with_hard_pin: false,
+    preferred_segments: vec!["seg-0".into()],
+};
+store.put("replicated-key", b"replicated value", Some(&config))?;
+
+// 5. Check existence
+let exists: bool = store.is_exist("my-key")?;
+
+// 6. Get size
+let size: u64 = store.get_size("my-key")?;
+
+// 7. Retrieve value
+let data: Vec<u8> = store.get("my-key")?;
+assert_eq!(data, b"hello, mooncake!");
+
+// 8. Remove
+store.remove("my-key", /*force=*/false)?;
+```
+
+### Error Handling
+
+All fallible operations return `Result<T, StoreError>`. The main variants are:
+
+| Variant | Meaning |
+|---------|---------|
+| `StoreError::OperationFailed(code)` | The underlying C library returned a non-zero status code |
+| `StoreError::SetupError(msg)` | Store setup / connection failed |
+| `StoreError::InvalidArgument(msg)` | A null pointer or invalid argument was passed |
+
+A missing key returns `StoreError::OperationFailed(code)` where `code` matches the Mooncake error table in [`mooncake-store/include/types.h`](../../../mooncake-store/include/types.h).
+
+### Running the Example
+
+```bash
+# Start metadata server
+cd mooncake-transfer-engine/example/http-metadata-server-python
+pip install aiohttp && python bootstrap_server.py &
+
+# Start mooncake_master
+./build/mooncake_master \
+    --enable_http_metadata_server=true \
+    --http_metadata_server_port=8080 \
+    --rpc_port=50051 &
+
+# Run the bundled example
+cd build
+cargo run --example basic_usage --manifest-path ../mooncake-store/rust/Cargo.toml
+```
+
+Or via CMake:
+
+```bash
+cd build
+cmake --build . --target build_mooncake_store_rust
+# The basic_usage binary is at:
+./mooncake-store/rust/target/debug/examples/basic_usage
+```
+
+---
+
+## Transfer Engine Rust Bindings
+
+### Overview
+
+The Transfer Engine Rust bindings live under `mooncake-transfer-engine/rust/`. They provide a Rust interface to `TransferEngine` and include a Rust port of the `transfer_engine_bench` benchmark tool.
+
+### Build
+
+```bash
+mkdir build && cd build
+cmake .. \
+    -DWITH_RUST_EXAMPLE=ON \
+    -DUSE_CUDA=ON \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo
+cmake --build . --target rust_transfer_engine_bench
+```
+
+### Running the Benchmark
+
+The Rust benchmark mirrors `transfer_engine_bench.cpp` in behaviour:
+
+```bash
+# Target node (receiver)
+./build/mooncake-transfer-engine/rust/target/release/transfer_engine_bench \
+    --mode=target \
+    --protocol=rdma \
+    --metadata_server=P2PHANDSHAKE
+
+# Initiator node (sender) — replace <target>:<port> from the target log
+./build/mooncake-transfer-engine/rust/target/release/transfer_engine_bench \
+    --mode=initiator \
+    --protocol=rdma \
+    --metadata_server=P2PHANDSHAKE \
+    --segment_id=<target>:<port> \
+    --operation=write \
+    --duration=10 \
+    --threads=8
+```
+
+---
+
+## CI Integration
+
+The Mooncake CI pipeline validates both Rust crates on every pull request:
+
+```yaml
+# Relevant CI steps (simplified from .github/workflows/ci.yml)
+- uses: dtolnay/rust-toolchain@stable
+- run: |
+    export MOONCAKE_STORE_LIB_DIR=$BUILD_DIR/mooncake-store
+    export MOONCAKE_STORE_INCLUDE_DIR=$SRC_DIR/mooncake-store/include
+    cargo check --manifest-path mooncake-store/rust/Cargo.toml
+```
+
+Both crates must pass `cargo check` (and `cargo clippy`) before merging.
+
+## See Also
+
+- [Mooncake Store Python API](../python-api-reference/mooncake-store)
+- [Transfer Engine C++ API](../design/transfer-engine/cpp-api)
+- [Mooncake Store Deployment Guide](../deployment/mooncake-store-deployment-guide)

docs/source/index.md

@@ -66,12 +66,15 @@ This repository also hosts its technical report and the open-sourced traces.
 getting_started/build
 getting_started/quick-start
 getting_started/supported-protocols
+getting_started/rust-bindings
 getting_started/plugin-usage/3FS-USRBIO-Plugin
 getting_started/examples/lmcache-integration
 getting_started/examples/lmdeploy-integration-v0.9
 getting_started/examples/sglang-integration-v1
 getting_started/examples/sglang-integration/index
 getting_started/examples/vllm-integration/index
+getting_started/examples/rl-training-integration
+getting_started/examples/mooncake-pg-usage
 :::
 
 % Making the most out of Mooncake
@@ -110,10 +113,12 @@ python-api-reference/ep-backend
 design/architecture
 design/mooncake-store
 design/p2p-store
+design/mooncake-pg
 design/transfer-engine/index
 design/tent/overview
 design/tent/tebench
 design/hicache-design
+design/conductor/indexer-api-design
 :::
 
 % Q&A for Mooncake
@@ -133,6 +138,9 @@ troubleshooting/troubleshooting
 :maxdepth: 2
 
 deployment/mooncake-store-deployment-guide
+deployment/ha-hot-standby
+deployment/monitoring
+deployment/multi-tier-storage
 :::
 
 % Community