[Docs] Add SSD offload documentation (#1647)
This commit is contained in:
zhangzuo21
GitHub
parent
41b4fbaff0
commit
f40407eb59
|
|
@ -138,3 +138,13 @@ Available log levels: trace, debug, info, warn (or warning), error, and critical
|
|||
- Scale `--rpc_thread_num` with available CPU cores and workload.
|
||||
- Start with default eviction settings; adjust `--eviction_high_watermark_ratio` and `--eviction_ratio` based on memory pressure and object churn.
|
||||
- Use `/metrics/summary` during bring-up; integrate `/metrics` with Prometheus/Grafana for production.
|
||||
|
||||
|
||||
---
|
||||
|
||||
:::{toctree}
|
||||
:caption: Advanced Topics
|
||||
:maxdepth: 1
|
||||
|
||||
ssd-offload
|
||||
:::
|
||||
|
|
|
|||
|
|
@ -0,0 +1,274 @@
|
|||
# SSD Offload
|
||||
|
||||
## Overview
|
||||
|
||||
Mooncake Store supports offloading KV cache objects from distributed memory to local SSD. When memory pressure is high, the master instructs clients to persist selected objects to disk. On a cache miss, the client automatically falls back to reading from SSD.
|
||||
|
||||
SSD offload is currently **only available in Real Client mode**. The real client is a standalone process that communicates with the application (e.g., SGLang) via RPC. All SSD reads and writes happen within this process.
|
||||
|
||||
## Startup Steps
|
||||
|
||||
### Step 1: Create the SSD storage directory
|
||||
|
||||
```bash
|
||||
mkdir -p /nvme/mooncake_offload
|
||||
```
|
||||
|
||||
### Step 2: Start the master
|
||||
|
||||
```bash
|
||||
mooncake_master \
|
||||
--rpc_port=50051 \
|
||||
--enable-offload true
|
||||
```
|
||||
|
||||
### Step 3: Start the real client with SSD offload enabled
|
||||
|
||||
Use the `--enable_offload` flag to enable SSD offload, and set environment variables to specify the storage path and backend:
|
||||
|
||||
```bash
|
||||
export MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/nvme/mooncake_offload
|
||||
export MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend
|
||||
|
||||
mooncake_client \
|
||||
--master_server_address=127.0.0.1:50051 \
|
||||
--host=<machine IP> \
|
||||
--protocol="rdma" \
|
||||
--device_names=<NIC name, e.g. eth0> \
|
||||
--port=50052 \
|
||||
--global_segment_size="4 GB" \
|
||||
--enable_offload=true \
|
||||
--metadata_server="P2PHANDSHAKE"
|
||||
```
|
||||
|
||||
> **Note:** On startup, the real client automatically scans existing SSD data and reports it to the master. No manual recovery is needed.
|
||||
|
||||
### Step 4: Connect the application to the real client
|
||||
|
||||
The application (e.g., SGLang) connects to the real client via the `MooncakeDistributedStore` Python SDK. SSD offload and fallback loading are handled transparently.
|
||||
|
||||
```python
|
||||
from mooncake.store import MooncakeDistributedStore
|
||||
|
||||
store = MooncakeDistributedStore()
|
||||
store.setup(
|
||||
local_hostname="<machine IP>",
|
||||
metadata_server="P2PHANDSHAKE",
|
||||
global_segment_size=4 * 1024 * 1024 * 1024, # 4 GB
|
||||
local_buffer_size=512 * 1024 * 1024, #512MB
|
||||
protocol="rdma",
|
||||
device_name="eth0",
|
||||
master_server_address="127.0.0.1:50051",
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Real Client Parameters
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--master_server_address` | `127.0.0.1:50051` | Master address |
|
||||
| `--host` | `0.0.0.0` | This machine's externally reachable IP |
|
||||
| `--port` | `50052` | Real client RPC listening port |
|
||||
| `--device_names` | ` ` | NIC name(s), e.g. `eth0` or `mlx5_0` |
|
||||
| `--protocol` | `tcp` | Transport protocol: `tcp` or `rdma` |
|
||||
| `--global_segment_size` | `4 GB` | Memory pool size allocated for this node |
|
||||
| `--enable_offload` | `false` | **Must be set to `true` to enable SSD offload** |
|
||||
| `--threads` | `1` | Number of RPC server threads |
|
||||
|
||||
---
|
||||
|
||||
## SSD Offload Configuration
|
||||
|
||||
### Core settings
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `MOONCAKE_OFFLOAD_FILE_STORAGE_PATH` | `/data/file_storage` | Absolute path to the SSD storage directory |
|
||||
| `MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR` | `bucket_storage_backend` | Storage backend type (see below) |
|
||||
| `MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES` | `1342177280` (1.25 GB) | Client-side staging buffer size |
|
||||
| `MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES` | `2199023255552` (2 TB) | Maximum disk usage |
|
||||
| `MOONCAKE_OFFLOAD_TOTAL_KEYS_LIMIT` | `10000000` | Maximum number of objects on disk |
|
||||
| `MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS` | `10` | Interval for offload heartbeat to master (seconds) |
|
||||
| `MOONCAKE_USE_URING` | `false` | Enable io_uring for async file I/O |
|
||||
|
||||
### Bucket backend settings
|
||||
|
||||
Applies when `MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend`.
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `MOONCAKE_OFFLOAD_BUCKET_SIZE_LIMIT_BYTES` | `268435456` (256 MB) | Max size per bucket |
|
||||
| `MOONCAKE_OFFLOAD_BUCKET_KEYS_LIMIT` | `500` | Max keys per bucket |
|
||||
| `MOONCAKE_BUCKET_MAX_TOTAL_SIZE` | `0` | Eviction threshold in bytes. When set to `0`, the backend uses **90% of the physical disk capacity** as the quota — it does not mean unlimited. Set an explicit value to control disk usage precisely. |
|
||||
| `MOONCAKE_BUCKET_EVICTION_POLICY` | `none` | Eviction policy: `none` / `fifo` / `lru` |
|
||||
|
||||
---
|
||||
|
||||
## Storage Backends
|
||||
|
||||
### `bucket_storage_backend` (recommended)
|
||||
|
||||
Groups multiple objects into bucket files. Reduces filesystem overhead, supports efficient batch I/O, and supports FIFO and LRU eviction.
|
||||
|
||||
**File layout:**
|
||||
```
|
||||
/nvme/mooncake_offload/
|
||||
├── 1710000000000-0.bucket # data file (multiple KV pairs)
|
||||
├── 1710000000000-0.meta # metadata file
|
||||
├── 1710000000001-0.bucket
|
||||
└── ...
|
||||
```
|
||||
|
||||
Best for: general-purpose use, large-scale deployments.
|
||||
|
||||
### `file_per_key_storage_backend`
|
||||
|
||||
Stores each object in an individual file. Simple and easy to inspect, but generates many small files at scale.
|
||||
|
||||
Best for: debugging or small-scale deployments.
|
||||
|
||||
### `offset_allocator_storage_backend`
|
||||
|
||||
Pre-allocates a single large file and manages offset-based allocation within it. Highest concurrency via 1024-shard metadata.
|
||||
|
||||
> **Warning:** This backend does **not** support metadata recovery on restart. On initialization, the data file is truncated and all in-memory metadata is cleared. Any previously offloaded objects become inaccessible after a process restart.
|
||||
|
||||
**Capacity:** `MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES` is used directly as the pre-allocated file size (100%, no safety margin). Unlike `bucket_storage_backend`, there is no separate quota variable — this is the sole disk usage control. Set it below the physical disk capacity to avoid filling the disk; writes are rejected once usage reaches this limit.
|
||||
|
||||
Best for: high-concurrency scenarios with many small objects where restart durability is not required.
|
||||
|
||||
---
|
||||
|
||||
## Eviction (Bucket Backend Only)
|
||||
|
||||
When `MOONCAKE_BUCKET_MAX_TOTAL_SIZE` is set, the backend automatically evicts buckets before writing new ones if total disk usage would exceed the limit.
|
||||
|
||||
| Policy | Behavior |
|
||||
|--------|----------|
|
||||
| `none` | No eviction (default); writes fail when disk is full |
|
||||
| `fifo` | Evict the oldest bucket first |
|
||||
| `lru` | Evict the least recently read bucket first |
|
||||
|
||||
Eviction is two-phase: the bucket is removed from metadata and master is notified first, then in-flight reads are drained before files are deleted.
|
||||
|
||||
---
|
||||
|
||||
## Example
|
||||
|
||||
The following example starts a master and a real client on a single machine.
|
||||
|
||||
### Environment
|
||||
|
||||
- Machine IP: `192.168.1.10`
|
||||
- NIC: `eth0`
|
||||
- SSD mount point: `/nvme`
|
||||
- Memory pool size: 4 GB (smaller than the total data written, to trigger offload)
|
||||
|
||||
### Start the master
|
||||
|
||||
```bash
|
||||
mooncake_master \
|
||||
--rpc_port=50051
|
||||
```
|
||||
|
||||
### Start the real client (new terminal)
|
||||
|
||||
```bash
|
||||
export MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/nvme/mooncake_offload
|
||||
export MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend
|
||||
export MOONCAKE_BUCKET_MAX_TOTAL_SIZE=$((200 * 1024 * 1024 * 1024)) # 200 GB
|
||||
export MOONCAKE_BUCKET_EVICTION_POLICY=lru
|
||||
|
||||
mooncake_client \
|
||||
--master_server_address="192.168.1.10:50051" \
|
||||
--host="192.168.1.10" \
|
||||
--device_names="eth0" \
|
||||
--port=50052 \
|
||||
--protocol="rdma" \
|
||||
--global_segment_size="4GB" \
|
||||
--enable_offload="true"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- `MOONCAKE_OFFLOAD_FILE_STORAGE_PATH` must be an absolute path to an existing, writable directory. Symbolic links and paths containing `..` are rejected.
|
||||
- On real client restart, the backend automatically scans existing SSD files and reports them to the master, so previously offloaded objects remain accessible.
|
||||
- Eviction only notifies the master and deletes local files; objects replicated on other nodes are unaffected.
|
||||
- Each machine requires its own real client process. In multi-node deployments, ensure `--host` and `--port` are correctly set so nodes can reach each other.
|
||||
|
||||
**2-node example:** suppose Node A (`192.168.1.10`) runs the master and Node B (`192.168.1.11`) is a second worker. Both real clients must point to the same master and advertise their own externally reachable IP:
|
||||
|
||||
```bash
|
||||
# Node A — runs the master and its own real client
|
||||
mooncake_master --rpc_port=50051 --enable-offload true &
|
||||
|
||||
export MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/nvme/mooncake_offload
|
||||
mooncake_client \
|
||||
--master_server_address="192.168.1.10:50051" \
|
||||
--host="192.168.1.10" \ # externally reachable IP of Node A
|
||||
--device_names="eth0" \
|
||||
--protocol="rdma" \
|
||||
--metadata_server="P2PHANDSHAKE" \
|
||||
--port=50052 \
|
||||
--global_segment_size="4GB" \
|
||||
--enable_offload="true"
|
||||
```
|
||||
|
||||
```bash
|
||||
# Node B — real client only; points to the same master on Node A
|
||||
export MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/nvme/mooncake_offload
|
||||
mooncake_client \
|
||||
--master_server_address="192.168.1.10:50051" \
|
||||
--host="192.168.1.11" \ # externally reachable IP of Node B, NOT 127.0.0.1
|
||||
--device_names="eth0" \
|
||||
--protocol="rdma" \
|
||||
--metadata_server="P2PHANDSHAKE" \
|
||||
--port=50052 \
|
||||
--global_segment_size="4GB" \
|
||||
--enable_offload="true"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### SSD offload is not triggering
|
||||
|
||||
- Confirm `--enable_offload=true` is passed to `mooncake_client` and `--enable-offload true` is passed to `mooncake_master`.
|
||||
- Check that `MOONCAKE_OFFLOAD_FILE_STORAGE_PATH` points to an existing, writable directory. The client will fail silently if the path is invalid.
|
||||
- Verify memory pressure is actually high enough for the master to trigger offload. If the memory pool (`--global_segment_size`) is large relative to the data written, offload may never activate.
|
||||
|
||||
### "Permission denied" or "No such file or directory" on the storage path
|
||||
|
||||
- Ensure the directory exists before starting the client: `mkdir -p <path>`.
|
||||
- Confirm the process user has read/write access to the directory.
|
||||
- Symbolic links and paths containing `..` are rejected — use an absolute, canonical path.
|
||||
|
||||
### "Failed to register buffer with UringFile" warning in logs
|
||||
|
||||
This warning appears when `MOONCAKE_USE_URING=true` and the io_uring fixed-buffer registration fails. The most common cause is that `MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES` exceeds the process's locked-memory limit (`RLIMIT_MEMLOCK`). io_uring requires the registered buffer to be pinned in physical memory, which counts against this limit.
|
||||
|
||||
Check the current limit:
|
||||
|
||||
```bash
|
||||
ulimit -l # in KB; "unlimited" means no cap
|
||||
```
|
||||
|
||||
To raise it for the current session:
|
||||
|
||||
```bash
|
||||
ulimit -l unlimited
|
||||
```
|
||||
|
||||
To raise it permanently, add the following to `/etc/security/limits.conf`:
|
||||
|
||||
```
|
||||
* soft memlock unlimited
|
||||
* hard memlock unlimited
|
||||
```
|
||||
|
||||
Alternatively, reduce `MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES` to a value within the existing limit. Note that the warning does not abort startup — the client falls back to non-fixed-buffer I/O — but performance may be lower than expected.
|
||||
|
|
@ -975,3 +975,13 @@ When to bump the version:
|
|||
* **Major version (X.0.0)**: For breaking API changes, major architectural changes, or significant new features that affect backward compatibility
|
||||
* **Minor version (0.X.0)**: For new features, API additions, or notable improvements that maintain backward compatibility
|
||||
* **Patch version (0.0.X)**: For bug fixes, performance optimizations, or minor improvements that don't affect the API
|
||||
|
||||
|
||||
---
|
||||
|
||||
:::{toctree}
|
||||
:caption: Related Design Docs
|
||||
:maxdepth: 1
|
||||
|
||||
ssd-offload
|
||||
:::
|
||||
|
|
|
|||
|
|
@ -0,0 +1,245 @@
|
|||
# SSD Offload Design
|
||||
|
||||
## Overview
|
||||
|
||||
Mooncake Store supports offloading KV cache objects from distributed memory to local SSD. This extends the effective cache capacity beyond DRAM limits at lower cost, while preserving the performance characteristics of the hot path through zero-copy RDMA-based memory transfers.
|
||||
|
||||
SSD offload is implemented as a background subsystem within the **real client** process. It is transparent to the application: a `Put` that would otherwise be evicted from memory is persisted to disk, and a `Get` that finds no memory replica automatically falls back to reading from SSD.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ Application (vLLM, etc.) │
|
||||
└──────────────────────────┬──────────────────────────────┘
|
||||
│ MooncakeDistributedStore API
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ Real Client │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────┐ │
|
||||
│ │ FileStorage │ │
|
||||
│ │ ┌────────────┐ ┌──────────────────────────┐ │ │
|
||||
│ │ │ Heartbeat │ │ ClientBuffer (staging) │ │ │
|
||||
│ │ │ Thread │ └──────────────────────────┘ │ │
|
||||
│ │ └─────┬──────┘ │ │
|
||||
│ │ │ offload / load │ │
|
||||
│ │ ▼ │ │
|
||||
│ │ ┌─────────────────────────────────────────┐ │ │
|
||||
│ │ │ StorageBackendInterface │ │ │
|
||||
│ │ │ ┌───────────┐ ┌──────────┐ ┌────────┐ │ │ │
|
||||
│ │ │ │ Bucket │ │FilePerKey│ │Offset │ │ │ │
|
||||
│ │ │ │ Backend │ │ Backend │ │Alloc. │ │ │ │
|
||||
│ │ │ └───────────┘ └──────────┘ └────────┘ │ │ │
|
||||
│ │ └─────────────────────────────────────────┘ │ │
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────┐ │
|
||||
│ │ In-memory distributed KV cache │ │
|
||||
│ │ (Transfer Engine / RDMA) │ │
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
│
|
||||
Local SSD / NVMe
|
||||
```
|
||||
|
||||
The key components are:
|
||||
|
||||
- **FileStorage**: The top-level coordinator. It owns the storage backend, a staging buffer (`ClientBuffer`), and background threads for heartbeating and buffer garbage collection.
|
||||
- **StorageBackendInterface**: An abstract interface implemented by three backends (see below). Responsible for the actual on-disk layout and I/O.
|
||||
- **Heartbeat thread**: Periodically contacts the master. The master returns a list of objects to offload; the heartbeat thread writes them to SSD and notifies the master of completion.
|
||||
- **ClientBuffer**: A pre-registered, O_DIRECT-aligned staging area used for zero-copy reads from SSD back into application memory.
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Offload (memory → SSD)
|
||||
|
||||
The offload path is driven entirely by the heartbeat thread inside `FileStorage`. No write path from the application is involved.
|
||||
|
||||
```
|
||||
Heartbeat Thread Master Local Memory Segment
|
||||
│ │ │
|
||||
│─OffloadObjectHB ───▶│ │
|
||||
│◀─ {key→size} map ───│ (objects to evict from │
|
||||
│ │ memory to SSD) │
|
||||
│ │ │
|
||||
│─ BatchQuery(keys) ───────────────────────────────▶│
|
||||
│◀─ {key→Slice} ────────────────────────────────────│
|
||||
│ │ │
|
||||
│ [PrepareEviction: remove old buckets, notify master via BatchEvictDiskReplica]
|
||||
│─ BatchEvictDiskReplica(evicted_keys) ────────────▶│ (master removes stale replicas)
|
||||
│ [FinalizeEviction: delete evicted files]
|
||||
│ │ │
|
||||
│ BatchOffload(slices) → StorageBackend → SSD │
|
||||
│ │ │
|
||||
│─ NotifyOffloadSuccess(keys, metadata) ───────────▶│
|
||||
│ │ (master adds LOCAL_DISK │
|
||||
│ │ replica to object entry) │
|
||||
```
|
||||
|
||||
Step by step:
|
||||
|
||||
1. **Heartbeat**: The heartbeat thread wakes up every `MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS` seconds and calls `client_->OffloadObjectHeartbeat(enable_offloading_, offloading_objects)`. The master replies with a map of `{key → size}` for objects it has selected to evict from memory.
|
||||
2. **Read slices from memory**: `FileStorage::OffloadObjects` groups the keys into buckets (for `BucketStorageBackend`) and calls `BatchQuerySegmentSlices` to obtain `{key → Slice}` from the local memory segment via `client_->BatchQuery`.
|
||||
3. **Eviction** (if capacity limit is set): Before writing, `PrepareEviction` removes old buckets from metadata under the exclusive lock and collects their keys. The `eviction_handler` callback calls `client_->BatchEvictDiskReplica` to notify the master in a single RPC. `FinalizeEviction` then deletes the corresponding files.
|
||||
4. **Write to SSD**: `StorageBackend::BatchOffload` serializes and writes the key-value data to disk.
|
||||
5. **Notify master**: On success, the `complete_handler` calls `client_->NotifyOffloadSuccess(keys, metadatas)`. The master adds a `LOCAL_DISK` replica entry (carrying the real client's RPC address as `transport_endpoint`) to the object's replica list.
|
||||
|
||||
### Load (SSD → memory)
|
||||
|
||||
The load path involves three parties: the **requesting client**, the **target client** that holds the SSD data, and the **Transfer Engine** for zero-copy data movement.
|
||||
|
||||
```
|
||||
Requesting Client Target Client Master
|
||||
│ │ │
|
||||
│─ BatchGet(keys) ──────────────────────────────────────────▶│
|
||||
│◀─ QueryResult {replicas: [LOCAL_DISK(rpc_addr)]} ──────────│
|
||||
│ │ │
|
||||
│ (no memory replica available) │ │
|
||||
│─ batch_get_offload_object(keys, sizes) ───────────────────▶│
|
||||
│ │ │
|
||||
│ FileStorage::BatchGet │
|
||||
│ → StorageBackend::BatchLoad │
|
||||
│ → read from SSD into ClientBuffer │
|
||||
│ │ │
|
||||
│◀─ BatchGetOffloadObjectResponse ───│ │
|
||||
│ {batch_id, pointers[], transfer_engine_addr, gc_ttl_ms} │
|
||||
│ │ │
|
||||
│─ Transfer Engine: BatchGetOffloadObject ──────────────────▶│
|
||||
│ (RDMA/TCP: pull data from ClientBuffer into app memory) │
|
||||
│◀─ done ────────────────────────────│ │
|
||||
│ │ │
|
||||
│─ release_offload_buffer(batch_id) ────────────────────────▶│
|
||||
│ │ (free ClientBuffer slot)│
|
||||
```
|
||||
|
||||
Step by step:
|
||||
|
||||
1. **Query master**: The requesting client calls `client_->BatchGet(keys, ...)` to query the master for replica locations. If the object has been offloaded, the master returns a `LOCAL_DISK` replica descriptor containing the target client's RPC address (`transport_endpoint`).
|
||||
2. **RPC to target client**: The requesting client calls `batch_get_offload_object(keys, sizes)` on the target client identified by `transport_endpoint`. The target client calls `FileStorage::BatchGet`, which allocates slots in `ClientBuffer` and reads the requested objects from SSD via `StorageBackend::BatchLoad`.
|
||||
3. **Response with buffer pointers**: The target client returns a `BatchGetOffloadObjectResponse` containing `batch_id`, a list of buffer `pointers` (addresses within `ClientBuffer`), the Transfer Engine address, and `gc_ttl_ms` (the buffer lease TTL).
|
||||
4. **Zero-copy transfer**: The requesting client invokes `client_->BatchGetOffloadObject(transfer_engine_addr, keys, pointers, slices)`, which uses the Transfer Engine (RDMA or TCP) to pull the data directly from the target client's `ClientBuffer` into the application's target memory (DRAM or VRAM). No intermediate copy is made on the requesting client side.
|
||||
5. **Release buffer**: After the transfer completes, the requesting client immediately calls `release_offload_buffer(batch_id)` on the target client to free the `ClientBuffer` slots. If the transfer takes longer than `gc_ttl_ms`, the buffer GC thread reclaims the slot automatically as a fallback.
|
||||
|
||||
---
|
||||
|
||||
## Storage Backends
|
||||
|
||||
### BucketStorageBackend (default)
|
||||
|
||||
Objects are grouped into **buckets** before being written to disk. Each bucket produces two files:
|
||||
|
||||
- **`.bucket`** — binary data file containing serialized key-value records
|
||||
- **`.meta`** — metadata file describing the keys and byte offsets within the data file
|
||||
|
||||
Bucket IDs are monotonically increasing timestamps with a sequence suffix, so `buckets_` (a `std::map<int64_t, BucketMetadata>`) is always ordered by creation time.
|
||||
|
||||
**Grouping strategy** (`GroupOffloadingKeysByBucket`): objects are accumulated into a bucket until either `bucket_size_limit` (default 256 MB) or `bucket_keys_limit` (default 500) is reached. Objects that do not fill a complete bucket are held in `ungrouped_offloading_objects_` and retried on the next heartbeat.
|
||||
|
||||
**In-flight read tracking**: A `BucketReadGuard` RAII object increments `BucketMetadata::inflight_reads_` on construction and decrements it on destruction. This allows safe deletion of bucket files even when concurrent reads are in progress.
|
||||
|
||||
### StorageBackendAdaptor (FilePerKey)
|
||||
|
||||
Each object is stored as an individual file. The file path is derived from the key via a two-level hash-sharded directory structure to avoid large flat directories. This backend is simple and easy to inspect but does not scale well to millions of objects.
|
||||
|
||||
### OffsetAllocatorStorageBackend
|
||||
|
||||
A single pre-allocated file (`kv_cache.data`) is shared by all objects. Space within the file is managed by an `OffsetAllocator`. Metadata is sharded across 1024 independent maps to reduce lock contention under high concurrency. Records follow the layout `[key_len: u32 | value_len: u32 | key | value]`.
|
||||
|
||||
---
|
||||
|
||||
## Eviction (BucketStorageBackend)
|
||||
|
||||
When `MOONCAKE_BUCKET_MAX_TOTAL_SIZE` is set, the backend evicts existing buckets to make room before writing a new one. Eviction is disabled by default (`BucketEvictionPolicy::NONE`).
|
||||
|
||||
### Policies
|
||||
|
||||
| Policy | Candidate selection |
|
||||
|--------|---------------------|
|
||||
| `FIFO` | `buckets_.begin()` — always the oldest bucket, since `buckets_` is ordered by bucket ID |
|
||||
| `LRU` | `std::min_element` over `BucketMetadata::last_access_ns_` — the bucket with the smallest last-read timestamp |
|
||||
|
||||
`last_access_ns_` is an atomic `int64_t` updated on every `BatchLoad` with relaxed ordering. Buckets that have never been read have `last_access_ns_ == 0` and are therefore always evicted first under LRU, giving FIFO-among-unread semantics.
|
||||
|
||||
### Two-phase eviction protocol
|
||||
|
||||
Eviction is split into two phases to ensure that the master is notified before files are deleted, and that no in-flight reads are interrupted.
|
||||
|
||||
**Phase 1 — `PrepareEviction(required_size)`** (called under exclusive lock):
|
||||
|
||||
1. Repeatedly call `SelectEvictionCandidate()` until `total_size_ + required_size <= max_total_size`.
|
||||
2. For each selected bucket: remove it from `buckets_` and `object_bucket_map_`, subtract its size from `total_size_`.
|
||||
3. Collect all evicted keys and bucket metadata into a `PendingEviction` struct and return it — no file I/O at this point.
|
||||
|
||||
**Between phases** — notify master:
|
||||
|
||||
The caller invokes the `eviction_handler` callback with the full list of evicted keys. The handler calls `MasterClient::BatchEvictDiskReplica`, which sends a single RPC to the master to remove the disk replicas for all evicted keys atomically.
|
||||
|
||||
**Phase 2 — `FinalizeEviction(pending)`** (called after master notification):
|
||||
|
||||
For each evicted bucket:
|
||||
1. Spin-wait (with a 10-second timeout) until `inflight_reads_ == 0`.
|
||||
2. Evict any stale file-handle cache entries.
|
||||
3. Delete the `.bucket` and `.meta` files.
|
||||
|
||||
This ordering guarantees:
|
||||
- The master never serves a stale disk-replica location for a file that has already been deleted.
|
||||
- Ongoing reads complete successfully before their files are removed.
|
||||
- Freed disk space is available for the incoming write before `WriteBucket` is called.
|
||||
|
||||
---
|
||||
|
||||
## io_uring File I/O
|
||||
|
||||
When `MOONCAKE_USE_URING=true`, the storage backends replace POSIX `pread`/`pwrite` calls with an io_uring-based implementation (`UringFile`). The design prioritizes eliminating inter-thread lock contention, which was the dominant latency source in the previous global-ring approach.
|
||||
|
||||
### Thread-local rings (`SharedUringRing`)
|
||||
|
||||
Each thread owns exactly one `io_uring` ring, stored in `thread_local` storage. This means:
|
||||
|
||||
- **No mutex between threads.** Each ring is accessed only by its owning thread, so concurrent I/O from multiple threads is fully parallel with zero synchronization overhead.
|
||||
- **Within-thread batching.** Multiple SQEs can be enqueued before calling `io_uring_submit_and_wait`, exposing NVMe queue depth > 1 within a single thread. `batch_read` exploits this to issue up to `QUEUE_DEPTH` (32) independent reads in one submission.
|
||||
- **File-descriptor registration is omitted.** The per-I/O `fdget()` overhead (~50 ns) is negligible compared to the lock contention (> 1 ms) the old global ring imposed, so `IOSQE_FIXED_FILE` is not used.
|
||||
|
||||
Rings are initialized lazily on first use and destroyed when the thread exits. If ring initialization fails (e.g., kernel too old), the backend falls back gracefully to POSIX I/O.
|
||||
|
||||
### Fixed-buffer registration
|
||||
|
||||
The `ClientBuffer` (the staging buffer used for SSD reads) is registered with io_uring as a **fixed buffer** via `io_uring_register_buffers`. When a read destination falls within the registered region, the backend uses `io_uring_prep_read_fixed` instead of `io_uring_prep_read`, which avoids a per-I/O `mmap`/`munmap` in the kernel and reduces system-call overhead.
|
||||
|
||||
Buffer registration is global but applied **lazily per thread**: `g_buf` stores the base address and length atomically; each thread-local ring calls `ensure_buf_registered()` on its first I/O and registers the buffer independently. This avoids a global barrier at startup.
|
||||
|
||||
To prevent `io_uring`'s `FOLL_LONGTERM` page pinning from failing on systems with Transparent Huge Pages (THP) enabled, `MADV_NOHUGEPAGE` is applied to the buffer region before registration. This forces the kernel to back the range with 4 KB pages, making long-term pinning reliable regardless of system THP policy.
|
||||
|
||||
### O_DIRECT and alignment
|
||||
|
||||
`UringFile` supports an optional `O_DIRECT` mode. When enabled:
|
||||
|
||||
- All file descriptors are opened with `O_DIRECT`.
|
||||
- Buffers, lengths, and offsets must be aligned to 4 KB (`ALIGNMENT_ = 4096`).
|
||||
- For unaligned writes (e.g., metadata serialized into a `std::string`), the backend allocates a temporary aligned bounce buffer via `posix_memalign`, copies the data, performs the aligned write, and frees the bounce buffer.
|
||||
- `read_aligned` and `write_aligned` are the primary I/O paths; they assert alignment constraints and delegate directly to the ring.
|
||||
|
||||
### I/O operations
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `read` / `write` | Contiguous read or write, chunked into up to `QUEUE_DEPTH` SQEs per submission |
|
||||
| `read_aligned` / `write_aligned` | Same as above but with alignment preconditions for O_DIRECT |
|
||||
| `batch_read` | Submits multiple independent reads (different offsets) in batches of up to `QUEUE_DEPTH`, maximizing NVMe queue utilization |
|
||||
| `vector_read` / `vector_write` | Scatter/gather I/O: one SQE per `iovec`, submitted in batches |
|
||||
| `datasync` | Issues `IORING_FSYNC_DATASYNC` and waits for completion |
|
||||
|
||||
### Integration with storage backends
|
||||
|
||||
- **BucketStorageBackend**: uses `UringFile` for both bucket data files and metadata files when `use_uring_` is set. A file-handle cache (`file_cache_`) avoids repeated `open`/`close` for hot buckets. On eviction, the cache entry is explicitly removed before the file is deleted to prevent stale handles.
|
||||
- **OffsetAllocatorStorageBackend**: opens the single pre-allocated data file with `O_DIRECT` and `UringFile`, and uses `GetFileInstance()` to expose the file handle for external buffer registration.
|
||||
- **StorageBackendAdaptor** (FilePerKey): uses `UringFile` for reads when `use_uring_` is set; writes use POSIX paths.
|
||||
|
||||
## Metadata Recovery on Restart
|
||||
|
||||
On startup, `FileStorage::Init` calls `StorageBackend::ScanMeta`, which reads all on-disk metadata and invokes a callback for each discovered object. The callback calls `MasterClient::NotifyOffloadSuccess` to re-register the objects with the master. This restores the full disk-replica view without any application-level intervention.
|
||||
|
|
@ -128,7 +128,7 @@ troubleshooting/troubleshooting
|
|||
|
||||
:::{toctree}
|
||||
:caption: Deployment
|
||||
:maxdepth: 1
|
||||
:maxdepth: 2
|
||||
|
||||
deployment/mooncake-store-deployment-guide
|
||||
:::
|
||||
|
|
|
|||
Loading…
Reference in New Issue