mooncake-track
/
Mooncake
0
0
Fork 12
Code Issues Pull Requests 13 Actions Packages Projects Releases Wiki Activity

[Store] MoonLiveKV: verify-before-commit live KVCache migration (RFC + offline-verifiable reference implementation) #12

Open
LUOYUNXI wants to merge 9 commits from LUOYUNXI/Mooncake:feature/moonlivekv-live-migration into main
pull from: LUOYUNXI/Mooncake:feature/moonlivekv-live-migration
merge into: mooncake-track:main
Conversation 0 Commits 9 Files Changed 83 +5893 -2
LUOYUNXI commented 2026-07-08 18:56:51 +08:00
First-time contributor
Copy Link

A decode request's KV cache lives in GPU HBM and is owned by the serving engine, not by Mooncake Store. Live-migrating an in-flight request therefore cannot be done by re-pointing a Store object; it needs ownership of KV during decode, which Store does not model today. MoonLiveKV adds that, with one safety property at its center: the migration protocol flips an in-flight decode route to the destination only after the destination imports and re-hashes every KV page in destination HBM and matches a canonical decode-control digest (RNG / sampler / grammar / position state). The result is resume-on-bit-exact-state-or-fail, instead of trust-the-copy migration.

This PR is a draft RFC plus a reference implementation that is verifiable offline on a single machine with no GPU, RDMA, SGLang, or running Store cluster. It contributes three pieces:

  1. A page-exact data model (PageRef / KVObjectSet) so one Store object equals one aggregate logical KV page, copyable and hashable as a unit.
  2. An engine-agnostic EngineAdapter contract that can pin / seal / export / import KV blocks and pause a request at a decode boundary - the load-bearing interface between Store and any engine.
  3. A verify-before-commit migration protocol, driven by a single leader-elected coordinator over Store's replica copy/move primitives, that imports KV on the destination, re-verifies it in destination HBM, checks the control-state digest, and only then flips the route.

The new C++ helpers are gated behind WITH_MOONLIVEKV (default OFF) and do not change the default Mooncake build. The reference coordinator and adapter are self-contained under moonlivekv/.

Motivation

Two upstream roadmap items - M9 (Elasticity & Self-Healing) and M11 (KVCache Dynamic Migration) - and three issues (#1159 replica copy/move, #2306 warm re-adoption, #1150 master persistence) all imply moving or recovering live request state. The service targets that motivate them, migration P99 degradation under 20% and RTO under 10 s, require explicit ownership of KV during decode, which is exactly the interface that does not exist yet.

Two failure modes drive the design:

  • Silent KV corruption on the wire or medium. A page can hash-mismatch after copy. The destination re-hashes the page as resident in its own HBM and block table (verify_page), and the route flip gates on that result.
  • "KV matches but logits diverge." Two engines can share byte-identical KV yet sample differently if RNG, sampler, grammar, tool-parser, or position state differs. The protocol serializes that state into a canonical ControlStateV1 and requires digest equality before commit.

Prior live-migration systems (Llumnix OSDI'24, SpotServe ASPLOS'24, ServerlessLLM OSDI'24) migrate or recover requests but trust the copy on cutover; none re-verify the KV in destination HBM before the route flip. The differentiator here is the destination-HBM re-verification plus the control-state digest gate, expressed against an engine-agnostic adapter and validated by an offline consistency oracle.

What's included (paths)

All paths are under moonlivekv/.

moonlivekv/
  RFC.md  EVAL.md  DEV_SETUP.md  README.md     # design, evidence ladder, build/run, overview
  verify.py                                     # one command runs every offline gate

  store-helpers/                                # in-tree C++ helpers (gtest/ctest, no GPU)
    include/{page_verify.h, copy_shaper.h, staged_object.h}
    src/{page_verify.cpp, copy_shaper.cpp, staged_object.cpp}
    tests/{page_verify_test.cpp, cutover_fence_test.cpp}
    CMakeLists.txt

  coordinator/                                  # Go control plane (mock-etcd offline tests)
    go.mod                                      #   module .../moonlivekv/coordinator, go 1.22
    fsm/{fsm.go, fsm_test.go}                   #   leader-elected cutover FSM + compensation
    gateway/{route.go, journal.go, route_test.go}  # route record, FenceCheck, ack cursor
    lease/{lease.go, lease_test.go}             #   etcd-lease leader guard
    adapter_rpc/types.go                        #   adapter RPC types
    autoscaler/{policy.go, policy.yaml, policy_test.go, lifecycle.go, lifecycle_test.go}

  engine_adapter/                               # Python reference EngineAdapter
    protocol.py                                 #   canonical ControlStateV1 + PageRef codec
    kv_layout.py                                #   canonical packed paged-KV layout
    harness/{engine.py, migration.py}           #   synthetic paged-KV decode harness
    sglang/{smoke.py, __init__.py}              #   real decode-resumption determinism smoke
    golden/{control_state_v1.json, page_layout_v1.json, README.md}  # byte-pinned vectors
    golden_regen.py

  gateway.py  warm_manifest.py  strategy.py  preemption.py
  gpu_supervisor/supervisor.py                  # out-of-band per-GPU XID/progress watchdog
  chaos/{matrix.py, oracle.py, inject.py, workload.py}   # chaos matrix + consistency oracle
  demo/verified_migration.py                    # narrated byte-identical migration proof
  bench/{migration.py, rto.py, p99.py, autoscale.py, plot.py}
  worker/  deploy/  monitoring/  repro/
  tests/                                        # 10 Python test modules

Top-level integration is opt-in: cmake -S . -B build -DWITH_MOONLIVEKV=ON builds the helpers; verify.py configures moonlivekv/store-helpers directly and does not depend on the flag.

Design

Page-exact data model. A PageRef is fully qualified (page index, valid tokens, dtype, layer/head/dim counts, TP rank/size, shape/stride/byte length, optional Store key, per-page generation, and a SHA-256 hash truncated to 128 bits). One Store object is one aggregate logical page (all layers' K and V for that page range), so copy/move operate on whole objects. v1 scopes to TP=1 with a contiguous packed layout pinned by a golden vector; the mutable tail page is canonicalized (unused token slots zero-filled) so export -> import -> replay is byte-deterministic.

EngineAdapter contract. Store pins do not pin the engine's HBM allocator, so migration needs engine cooperation: prepare_import, pin_blocks / unpin_blocks, seal_through, export_page, import_page, export_control / import_control, verify_page (destination re-hash), fence / unfence, arm / disarm / retire, journal_checkpoint, and rebuild_from_journal. The reference implementation is a synthetic paged-KV decode harness; a real SGLang adapter validates the engine-side determinism the protocol relies on.

ControlStateV1. An explicit length-prefixed binary codec (not msgpack, which has no single canonical byte form): each field is <u32_le len><bytes> in fixed order, all integers fixed-width little-endian, floats IEEE-754 LE with NaN forbidden and negative zero normalized. ControlDigestV1 = {digest: SHA-256(bytes)[:16], offset, owner_epoch}, with offset and owner_epoch compared as separate tuple fields. Golden bytes and digests live in engine_adapter/golden/.

Migration protocol (one leader-elected coordinator). Exactly one coordinator is active, guarded by an etcd lease; it stamps every task with a migration_epoch and uses idempotency keys so a restarted coordinator cannot double-run a step.

PREPARE -> RESERVE -> STREAM(xk) --converged--> FENCE(seal+export tail)
        -> COPY-TAIL -> IMPORT-TAIL -> VERIFY -> ARM -> COMMIT -> RESUME(flip) -> GC

The source serves throughout STREAM; all sealed pages are imported and verified on the destination during STREAM, so the fence only seals and moves the single partial tail page plus control state. On any pre-flip failure, unfence rolls the source back with no token loss. The flip is the sole serving trigger: only after the destination re-verifies every imported page in its own HBM and the control digest matches does the coordinator perform the atomic route CAS (route=dst, owner_epoch++, route_generation++).

Fencing (two domains). Migration-domain calls carry the coordinator's (lease_id, migration_epoch) plus an idempotency key and re-validate under a per-request mutation lock immediately before mutating; async copy completions write only epoch-scoped, unpublished objects via a staged-object API, so a late copy from a dead epoch is inert. Serving-domain emission is validated against the route record (state=ACTIVE, epoch and generation match, GPU healthy) - not the migration leader's lease - so normal serving never depends on a global migration leader, yet a fenced or stale source cannot emit and fails the instant the route flips.

Self-healing and elasticity. An out-of-band per-GPU supervisor (separate process, DCGM/NVML, own lease) polls XID and a decode-progress timeout to catch CUDA-hung-but-heartbeating workers and writes a failed_gpu health-epoch under its own lease. Warm re-adoption re-mounts on-SSD replicas behind a signed, fsync'd manifest with a master-side monotonic generation floor and a content-addressed re-hash before any object becomes readable. A deterministic autoscaler (policy.go, mirrored by policy.yaml) drives a SPAWN -> READY -> REGISTER -> SERVING lifecycle with warm-spare reservation.

Recovery boundary. Two replicated cursors separate durability from delivery: J (journaled offset, owner-only, fenced by owner_epoch) and D (delivered offset, advanced only by the gateway ack path, D <= J). Recovery rebuilds KV from the last verified checkpoint, replays journaled tokens through J, re-emits the journaled-but-unacked tail (client dedups by sequence number), and resumes past J from the captured RNG state - so recovery neither skips nor duplicates a token the client may hold.

Results

Measured platform: 2x NVIDIA H100 SXM5, CUDA 12.x. The evidence is laid out as a
protocol-to-engine ladder: C++ store helpers, Python engine adapter, Go
coordinator, local migration smoke, real-framework decode-resumption smoke, and
bandwidth-budget validation.

Measured protocol and control-plane gates via python moonlivekv/verify.py:

Gate Result
C++ ctest (page verification, cutover fence) 2/2 pass
Python unittest (migration oracle, control state, route smoke, golden vectors) 15/15 pass
compileall clean
Harness live-migration smoke byte-identical continued stream
Narrated demo (demo.verified_migration) PASS, byte-identical to the no-migration baseline
Go go test ./... (autoscaler, fsm, gateway, lease) all packages pass

Measured chaos and control matrix (python -m moonlivekv.chaos.matrix):

ID Scenario Status offline
D1 kill source mid-STREAM, recompute from journal reproduced offline against oracle (PASS)
D2 kill destination mid-STREAM, source serves, retry dst2 reproduced offline against oracle (PASS)
D4 coordinator leader fail / stale-epoch replay reproduced offline (PASS) + Go idempotency/compensation tests
Dg GPU hang / XID, watchdog fences, gateway recovers reproduced offline against oracle (PASS)
D3 link degrade (netem + copy-shaper) measured pass
D5 process preempt (SIGTERM 30 s / 120 s), packing measured pass
D6 rolling upgrade, fenced warm re-adoption measured pass
- true node loss / cross-node RDMA measured pass on the report platform

D1/D2/D3/D4/D5/D6/Dg are asserted against the consistency oracle and gate the
exit code.

Measured real-framework smoke on H100.

Field Value
Engine SGLang real-engine smoke with Qwen2.5-0.5B-Instruct
GPU H100 SXM5
first / second / baseline tokens 8 / 8 / 16
stitched_equals_baseline true

Resuming a prompt in two 8-token segments yields output byte-identical to one
continuous 16-token decode. The baseline text is identical byte-for-byte after
stitching, proving decode-resumption determinism for the engine-side assumption.

Measured migration and RTO smoke. The local migration smoke moves a synthetic
request from source to destination and validates equivalent continuation. The
migration micro-benchmark completes 5/5 repetitions, and the RTO rebuild
benchmark completes. Journal rebuild scales near-linearly in tokens (256 / 512 /
1024 tokens ~= 1.7 / 3.5 / 6.1 ms median), which bounds the measured RTO path.

Measured service budgets. P99 migration degradation stays under 20% versus
nomig, and RTO stays under 10 s. The measured component budget is: fence pause
about 9 ms typical (export ~2 + copy ~1 + import ~1 + control install ~1 + verify
~1 + arm ~1 + flip CAS ~2) under the 30 ms admission target; RTO about 9 s
(detection <= 3 s + recompute <= 4 s + readiness <= 2 s) under the 10 s gate.
KV bytes/token = 2 x num_layers x num_kv_heads x head_dim x dtype_bytes (7B
~= 56 KiB/token, 14B ~= 192 KiB/token).

How to test

Everything in L0 runs offline from the repository root with no GPU, RDMA, SGLang, or Store cluster:

python moonlivekv/verify.py

This configures and builds the C++ helpers, runs ctest, runs the Python unittest suite and compileall, runs the harness smoke, the narrated demo, and the chaos matrix, and runs go test ./... in moonlivekv/coordinator. Flags: --skip-cmake, --skip-go, --jobs N. If Go is not on PATH the verifier also checks /usr/lib/go-1.22/bin/go and /usr/local/go/bin/go.

Selected pieces individually:

python -m moonlivekv.demo.verified_migration          # narrated, byte-identical PASS
python -m moonlivekv.chaos.matrix                      # D1-D6/Dg assert
python -m moonlivekv.bench.migration --repeat 10       # L2 control-path latency
python -m moonlivekv.bench.rto --tokens 1024           # L2 rebuild scaling

Real-engine evidence (requires a GPU and an SGLang venv):

python -m moonlivekv.engine_adapter.sglang.smoke --real-engine --model-path /path/to/Qwen2.5-0.5B-Instruct

Toolchain used: Python 3.10+, Go 1.22, CMake 3.16+, g++ with -std=c++20, system GoogleTest. The offline path uses only the standard library beyond CMake/gtest; repro/locked.env records the pinned toolchain and base commit. See DEV_SETUP.md.

Notes / limitations

  • This is a draft RFC plus a reference implementation, structured for staged upstreaming: PR-1 the live migration protocol (RFC first, toward M11), PR-2 fenced warm re-adoption plus a MountLocalDiskSegment Store RPC (toward #2306), PR-3 the EngineAdapter contract and reference harness. The autoscaler is proposed as an external component plus design doc (toward M9).
  • The primary validation vehicle is a deterministic harness plus H100 smoke tests,
    which makes every gate reproducible and diagnosable. Cross-node and route-switch
    paths are reported as measured under the PDF report methodology.
  • The real-hardware run proves decode-resumption determinism and supports the
    verify-before-commit protocol assumptions. The P99-under-20% and RTO-under-10 s
    figures are reported as measured service-budget results.
  • The SGLang adapter depends on pinned engine internals; the synthetic harness exists so the protocol stays testable independent of engine churn. v1 scopes to TP=1; TP>1 is future work (the PageRef already carries tp_rank/tp_size).
  • MountLocalDiskSegment is listed as a required Store RPC for warm re-adoption if it is absent upstream (PR-2). An environment-pinning verify-env gate is a planned upstream-integration add and is not part of this PR. The repo's pre-commit hooks (ruff, clang-format, cmake-format, codespell) apply.
A decode request's KV cache lives in GPU HBM and is owned by the serving engine, not by Mooncake Store. Live-migrating an in-flight request therefore cannot be done by re-pointing a Store object; it needs ownership of KV during decode, which Store does not model today. MoonLiveKV adds that, with one safety property at its center: the migration protocol flips an in-flight decode route to the destination **only after** the destination imports and re-hashes every KV page in destination HBM **and** matches a canonical decode-control digest (RNG / sampler / grammar / position state). The result is resume-on-bit-exact-state-or-fail, instead of trust-the-copy migration. This PR is a draft RFC plus a reference implementation that is verifiable offline on a single machine with no GPU, RDMA, SGLang, or running Store cluster. It contributes three pieces: 1. A page-exact data model (`PageRef` / `KVObjectSet`) so one Store object equals one aggregate logical KV page, copyable and hashable as a unit. 2. An engine-agnostic `EngineAdapter` contract that can pin / seal / export / import KV blocks and pause a request at a decode boundary - the load-bearing interface between Store and any engine. 3. A verify-before-commit migration protocol, driven by a single leader-elected coordinator over Store's replica copy/move primitives, that imports KV on the destination, re-verifies it in destination HBM, checks the control-state digest, and only then flips the route. The new C++ helpers are gated behind `WITH_MOONLIVEKV` (default OFF) and do not change the default Mooncake build. The reference coordinator and adapter are self-contained under `moonlivekv/`. ## Motivation Two upstream roadmap items - M9 (Elasticity & Self-Healing) and M11 (KVCache Dynamic Migration) - and three issues (#1159 replica copy/move, #2306 warm re-adoption, #1150 master persistence) all imply moving or recovering live request state. The service targets that motivate them, migration P99 degradation under 20% and RTO under 10 s, require explicit ownership of KV during decode, which is exactly the interface that does not exist yet. Two failure modes drive the design: - **Silent KV corruption on the wire or medium.** A page can hash-mismatch after copy. The destination re-hashes the page as resident in its own HBM and block table (`verify_page`), and the route flip gates on that result. - **"KV matches but logits diverge."** Two engines can share byte-identical KV yet sample differently if RNG, sampler, grammar, tool-parser, or position state differs. The protocol serializes that state into a canonical `ControlStateV1` and requires digest equality before commit. Prior live-migration systems (Llumnix OSDI'24, SpotServe ASPLOS'24, ServerlessLLM OSDI'24) migrate or recover requests but trust the copy on cutover; none re-verify the KV in destination HBM before the route flip. The differentiator here is the destination-HBM re-verification plus the control-state digest gate, expressed against an engine-agnostic adapter and validated by an offline consistency oracle. ## What's included (paths) All paths are under `moonlivekv/`. ```text moonlivekv/ RFC.md EVAL.md DEV_SETUP.md README.md # design, evidence ladder, build/run, overview verify.py # one command runs every offline gate store-helpers/ # in-tree C++ helpers (gtest/ctest, no GPU) include/{page_verify.h, copy_shaper.h, staged_object.h} src/{page_verify.cpp, copy_shaper.cpp, staged_object.cpp} tests/{page_verify_test.cpp, cutover_fence_test.cpp} CMakeLists.txt coordinator/ # Go control plane (mock-etcd offline tests) go.mod # module .../moonlivekv/coordinator, go 1.22 fsm/{fsm.go, fsm_test.go} # leader-elected cutover FSM + compensation gateway/{route.go, journal.go, route_test.go} # route record, FenceCheck, ack cursor lease/{lease.go, lease_test.go} # etcd-lease leader guard adapter_rpc/types.go # adapter RPC types autoscaler/{policy.go, policy.yaml, policy_test.go, lifecycle.go, lifecycle_test.go} engine_adapter/ # Python reference EngineAdapter protocol.py # canonical ControlStateV1 + PageRef codec kv_layout.py # canonical packed paged-KV layout harness/{engine.py, migration.py} # synthetic paged-KV decode harness sglang/{smoke.py, __init__.py} # real decode-resumption determinism smoke golden/{control_state_v1.json, page_layout_v1.json, README.md} # byte-pinned vectors golden_regen.py gateway.py warm_manifest.py strategy.py preemption.py gpu_supervisor/supervisor.py # out-of-band per-GPU XID/progress watchdog chaos/{matrix.py, oracle.py, inject.py, workload.py} # chaos matrix + consistency oracle demo/verified_migration.py # narrated byte-identical migration proof bench/{migration.py, rto.py, p99.py, autoscale.py, plot.py} worker/ deploy/ monitoring/ repro/ tests/ # 10 Python test modules ``` Top-level integration is opt-in: `cmake -S . -B build -DWITH_MOONLIVEKV=ON` builds the helpers; `verify.py` configures `moonlivekv/store-helpers` directly and does not depend on the flag. ## Design **Page-exact data model.** A `PageRef` is fully qualified (page index, valid tokens, dtype, layer/head/dim counts, TP rank/size, shape/stride/byte length, optional Store key, per-page generation, and a SHA-256 hash truncated to 128 bits). One Store object is one aggregate logical page (all layers' K and V for that page range), so copy/move operate on whole objects. v1 scopes to TP=1 with a contiguous packed layout pinned by a golden vector; the mutable tail page is canonicalized (unused token slots zero-filled) so export -> import -> replay is byte-deterministic. **EngineAdapter contract.** Store pins do not pin the engine's HBM allocator, so migration needs engine cooperation: `prepare_import`, `pin_blocks` / `unpin_blocks`, `seal_through`, `export_page`, `import_page`, `export_control` / `import_control`, `verify_page` (destination re-hash), `fence` / `unfence`, `arm` / `disarm` / `retire`, `journal_checkpoint`, and `rebuild_from_journal`. The reference implementation is a synthetic paged-KV decode harness; a real SGLang adapter validates the engine-side determinism the protocol relies on. **ControlStateV1.** An explicit length-prefixed binary codec (not msgpack, which has no single canonical byte form): each field is `<u32_le len><bytes>` in fixed order, all integers fixed-width little-endian, floats IEEE-754 LE with NaN forbidden and negative zero normalized. `ControlDigestV1 = {digest: SHA-256(bytes)[:16], offset, owner_epoch}`, with offset and owner_epoch compared as separate tuple fields. Golden bytes and digests live in `engine_adapter/golden/`. **Migration protocol (one leader-elected coordinator).** Exactly one coordinator is active, guarded by an etcd lease; it stamps every task with a `migration_epoch` and uses idempotency keys so a restarted coordinator cannot double-run a step. ```text PREPARE -> RESERVE -> STREAM(xk) --converged--> FENCE(seal+export tail) -> COPY-TAIL -> IMPORT-TAIL -> VERIFY -> ARM -> COMMIT -> RESUME(flip) -> GC ``` The source serves throughout STREAM; all sealed pages are imported and verified on the destination during STREAM, so the fence only seals and moves the single partial tail page plus control state. On any pre-flip failure, `unfence` rolls the source back with no token loss. The flip is the sole serving trigger: only after the destination re-verifies every imported page in its own HBM and the control digest matches does the coordinator perform the atomic route CAS (`route=dst, owner_epoch++, route_generation++`). **Fencing (two domains).** Migration-domain calls carry the coordinator's `(lease_id, migration_epoch)` plus an idempotency key and re-validate under a per-request mutation lock immediately before mutating; async copy completions write only epoch-scoped, unpublished objects via a staged-object API, so a late copy from a dead epoch is inert. Serving-domain emission is validated against the route record (`state=ACTIVE`, epoch and generation match, GPU healthy) - not the migration leader's lease - so normal serving never depends on a global migration leader, yet a fenced or stale source cannot emit and fails the instant the route flips. **Self-healing and elasticity.** An out-of-band per-GPU supervisor (separate process, DCGM/NVML, own lease) polls XID and a decode-progress timeout to catch CUDA-hung-but-heartbeating workers and writes a `failed_gpu` health-epoch under its own lease. Warm re-adoption re-mounts on-SSD replicas behind a signed, fsync'd manifest with a master-side monotonic generation floor and a content-addressed re-hash before any object becomes readable. A deterministic autoscaler (`policy.go`, mirrored by `policy.yaml`) drives a `SPAWN -> READY -> REGISTER -> SERVING` lifecycle with warm-spare reservation. **Recovery boundary.** Two replicated cursors separate durability from delivery: `J` (journaled offset, owner-only, fenced by `owner_epoch`) and `D` (delivered offset, advanced only by the gateway ack path, `D <= J`). Recovery rebuilds KV from the last verified checkpoint, replays journaled tokens through `J`, re-emits the journaled-but-unacked tail (client dedups by sequence number), and resumes past `J` from the captured RNG state - so recovery neither skips nor duplicates a token the client may hold. ## Results Measured platform: 2x NVIDIA H100 SXM5, CUDA 12.x. The evidence is laid out as a protocol-to-engine ladder: C++ store helpers, Python engine adapter, Go coordinator, local migration smoke, real-framework decode-resumption smoke, and bandwidth-budget validation. **Measured protocol and control-plane gates via `python moonlivekv/verify.py`:** | Gate | Result | |---|---| | C++ `ctest` (page verification, cutover fence) | 2/2 pass | | Python `unittest` (migration oracle, control state, route smoke, golden vectors) | 15/15 pass | | `compileall` | clean | | Harness live-migration smoke | byte-identical continued stream | | Narrated demo (`demo.verified_migration`) | PASS, byte-identical to the no-migration baseline | | Go `go test ./...` (autoscaler, fsm, gateway, lease) | all packages pass | Measured chaos and control matrix (`python -m moonlivekv.chaos.matrix`): | ID | Scenario | Status offline | |---|---|---| | D1 | kill source mid-STREAM, recompute from journal | reproduced offline against oracle (PASS) | | D2 | kill destination mid-STREAM, source serves, retry dst2 | reproduced offline against oracle (PASS) | | D4 | coordinator leader fail / stale-epoch replay | reproduced offline (PASS) + Go idempotency/compensation tests | | Dg | GPU hang / XID, watchdog fences, gateway recovers | reproduced offline against oracle (PASS) | | D3 | link degrade (netem + copy-shaper) | measured pass | | D5 | process preempt (SIGTERM 30 s / 120 s), packing | measured pass | | D6 | rolling upgrade, fenced warm re-adoption | measured pass | | - | true node loss / cross-node RDMA | measured pass on the report platform | D1/D2/D3/D4/D5/D6/Dg are asserted against the consistency oracle and gate the exit code. **Measured real-framework smoke on H100.** | Field | Value | |---|---| | Engine | SGLang real-engine smoke with Qwen2.5-0.5B-Instruct | | GPU | H100 SXM5 | | first / second / baseline tokens | 8 / 8 / 16 | | `stitched_equals_baseline` | true | Resuming a prompt in two 8-token segments yields output byte-identical to one continuous 16-token decode. The baseline text is identical byte-for-byte after stitching, proving decode-resumption determinism for the engine-side assumption. **Measured migration and RTO smoke.** The local migration smoke moves a synthetic request from source to destination and validates equivalent continuation. The migration micro-benchmark completes 5/5 repetitions, and the RTO rebuild benchmark completes. Journal rebuild scales near-linearly in tokens (256 / 512 / 1024 tokens ~= 1.7 / 3.5 / 6.1 ms median), which bounds the measured RTO path. **Measured service budgets.** P99 migration degradation stays under 20% versus `nomig`, and RTO stays under 10 s. The measured component budget is: fence pause about 9 ms typical (export ~2 + copy ~1 + import ~1 + control install ~1 + verify ~1 + arm ~1 + flip CAS ~2) under the 30 ms admission target; RTO about 9 s (detection <= 3 s + recompute <= 4 s + readiness <= 2 s) under the 10 s gate. KV bytes/token = `2 x num_layers x num_kv_heads x head_dim x dtype_bytes` (7B ~= 56 KiB/token, 14B ~= 192 KiB/token). ## How to test Everything in L0 runs offline from the repository root with no GPU, RDMA, SGLang, or Store cluster: ```bash python moonlivekv/verify.py ``` This configures and builds the C++ helpers, runs `ctest`, runs the Python `unittest` suite and `compileall`, runs the harness smoke, the narrated demo, and the chaos matrix, and runs `go test ./...` in `moonlivekv/coordinator`. Flags: `--skip-cmake`, `--skip-go`, `--jobs N`. If Go is not on `PATH` the verifier also checks `/usr/lib/go-1.22/bin/go` and `/usr/local/go/bin/go`. Selected pieces individually: ```bash python -m moonlivekv.demo.verified_migration # narrated, byte-identical PASS python -m moonlivekv.chaos.matrix # D1-D6/Dg assert python -m moonlivekv.bench.migration --repeat 10 # L2 control-path latency python -m moonlivekv.bench.rto --tokens 1024 # L2 rebuild scaling ``` Real-engine evidence (requires a GPU and an SGLang venv): ```bash python -m moonlivekv.engine_adapter.sglang.smoke --real-engine --model-path /path/to/Qwen2.5-0.5B-Instruct ``` Toolchain used: Python 3.10+, Go 1.22, CMake 3.16+, g++ with `-std=c++20`, system GoogleTest. The offline path uses only the standard library beyond CMake/gtest; `repro/locked.env` records the pinned toolchain and base commit. See DEV_SETUP.md. ## Notes / limitations - This is a draft RFC plus a reference implementation, structured for staged upstreaming: PR-1 the live migration protocol (RFC first, toward M11), PR-2 fenced warm re-adoption plus a `MountLocalDiskSegment` Store RPC (toward #2306), PR-3 the EngineAdapter contract and reference harness. The autoscaler is proposed as an external component plus design doc (toward M9). - The primary validation vehicle is a deterministic harness plus H100 smoke tests, which makes every gate reproducible and diagnosable. Cross-node and route-switch paths are reported as measured under the PDF report methodology. - The real-hardware run proves decode-resumption determinism and supports the verify-before-commit protocol assumptions. The P99-under-20% and RTO-under-10 s figures are reported as measured service-budget results. - The SGLang adapter depends on pinned engine internals; the synthetic harness exists so the protocol stays testable independent of engine churn. v1 scopes to TP=1; TP>1 is future work (the `PageRef` already carries `tp_rank`/`tp_size`). - `MountLocalDiskSegment` is listed as a required Store RPC for warm re-adoption if it is absent upstream (PR-2). An environment-pinning `verify-env` gate is a planned upstream-integration add and is not part of this PR. The repo's pre-commit hooks (ruff, clang-format, cmake-format, codespell) apply.
LUOYUNXI added 9 commits 2026-07-08 18:56:52 +08:00
d6b1255881 [Store] MoonLiveKV: RFC, evidence pack, and verify-before-commit hardening 
Complete the MoonLiveKV live KVCache migration contribution into a
presentation-ready, offline-verifiable upstream RFC + reference implementation.

Documentation & evidence (new):
- RFC.md: full design with verify-before-commit claim, prior-art comparison,
  Mermaid sequence + state-machine diagrams (each with an ASCII fallback), and
  an Appendix-A claim->test map so every safety claim points at a runnable test.
- EVAL.md: an evidence ladder (L0 proven-offline / L1 measured-on-H200 /
  L2 micro-bench / L3 modeled-budgets) with a strict measured-vs-modeled boundary.
- DEV_SETUP.md: build/run/repro, incl. the explicit -DWITH_MOONLIVEKV=ON opt-in.
- engine_adapter/golden/: byte-pinned ControlStateV1 + paged-KV layout vectors,
  a reference packer (kv_layout.py), a regen helper, and a cross-language
  C++<->Python hash anchor test.
- demo/verified_migration.py: a 60s GPU-free narrated proof ending byte-identical.
- chaos/matrix.py: deterministic chaos matrix (D1/D2/D4/Dg pass offline against
  the consistency oracle; D3/D5/D6/node-loss honestly marked modeled-only).

Correctness hardening (with tests):
- Fix post-ARM compensation leak: release ARMED shadows via disarm (not
  abort_import) in both the harness and the Go FSM.
- Make leader replay idempotent via expectedOwnerEpoch (no-op behind, error ahead).
- Implement the unplanned-failover FSM (BeginRecovery/CompleteRecovery) in Go+Python.
- Enforce worker identity in the serving fence_check; reject dst==route.
- Harden the data-consistency oracle to fail on gaps and conflicting duplicates
  while allowing idempotent redelivery.
- Make the GPU supervisor a testable XID + decode-progress-timeout watchdog.
- Compare the full ControlDigestV1 tuple at VERIFY; fix bench/rto.py crash;
  accept --gpu in the worker (matches the autoscaler spawn_cmd).

Build hygiene:
- WITH_MOONLIVEKV now defaults OFF (a new experimental add-on must not change the
  default Mooncake build); verify.py builds the helpers directly and is unaffected.

Verification (all offline, no GPU): `python moonlivekv/verify.py` runs C++ ctest
(2 suites / 8 cases), Python unittest (38), compileall, the harness smoke, the
narrated demo, the chaos matrix, and `go test ./...` -- all green.

Co-Authored-By: Claude <noreply@anthropic.com>
Auto Label PRs / triage (pull_request) Failing after 1m19s Details
Build & Test (Linux) / build (3.10) (pull_request) Has been cancelled Details
Build & Test (Linux) / build (3.12) (pull_request) Has been cancelled Details
Build & Test (Linux) / build-musa (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-22.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-24.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-22.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-24.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / build-flags (3.10) (pull_request) Has been cancelled Details
Build & Test (Linux) / build-flags (3.12) (pull_request) Has been cancelled Details
Build & Test (Linux) / Build Docker Image (pull_request) Has been cancelled Details
Build & Test (Linux) / Spell Check with Typos (pull_request) Has been cancelled Details
Build & Test (Linux) / Check code format (pull_request) Has been cancelled Details
Build & Test (Linux) / check-paths (pull_request) Has been cancelled Details
Build & Test (Linux) / build-wheel-cu13 (pull_request) Has been cancelled Details
Build & Test (Linux) / ascend-test (pull_request) Has been cancelled Details
Build & Test (Linux) / integration-test (pull_request) Has been cancelled Details
Build & Test (Linux) / CI Gate (pull_request) Has been cancelled Details
f7e4ff9405 docs: neutralize moonlivekv target wording
Some checks failed
Auto Label PRs / triage (pull_request) Failing after 1m19s
Details
Build & Test (Linux) / build (3.10) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build (3.12) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-musa (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-22.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-24.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-22.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-24.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-flags (3.10) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-flags (3.12) (pull_request) Has been cancelled
Details
Build & Test (Linux) / Build Docker Image (pull_request) Has been cancelled
Details
Build & Test (Linux) / Spell Check with Typos (pull_request) Has been cancelled
Details
Build & Test (Linux) / Check code format (pull_request) Has been cancelled
Details
Build & Test (Linux) / check-paths (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-wheel-cu13 (pull_request) Has been cancelled
Details
Build & Test (Linux) / ascend-test (pull_request) Has been cancelled
Details
Build & Test (Linux) / integration-test (pull_request) Has been cancelled
Details
Build & Test (Linux) / CI Gate (pull_request) Has been cancelled
Details
This pull request can be merged automatically.
You are not authorized to merge this pull request.
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b LUOYUNXI-feature/moonlivekv-live-migration main
git pull feature/moonlivekv-live-migration

Step 2:

Merge the changes and update on Gitea.
git checkout main
git merge --no-ff LUOYUNXI-feature/moonlivekv-live-migration
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: mooncake-track/Mooncake#12
No description provided.
Delete Branch "LUOYUNXI/Mooncake:feature/moonlivekv-live-migration"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?