mooncake-track
/
Mooncake
0
0
Fork 12
Code Issues Pull Requests 13 Actions Packages Projects Releases Wiki Activity

Multimodal embedding cache and same-worker agent fork for SGLang EPD over Mooncake Store #11

Open
Jared wants to merge 1 commits from Jared/Mooncake:moonagent-epd-clean into main
pull from: Jared/Mooncake:moonagent-epd-clean
merge into: mooncake-track:main
Conversation 0 Commits 1 Files Changed 48 +4269 -29
Jared commented 2026-07-08 18:55:25 +08:00
First-time contributor
Copy Link

This change adds a Mooncake Store data path for vision-language EPD (Encode-Prefill-Decode) serving, plus a CPU-testable reference library and an additive SGLang patch series. It targets two sources of repeated GPU work in multi-image and multi-agent workloads:

  1. Visual embedding cache (Pillar 1). Processed-image visual embeddings are written to Mooncake Store once and reused across requests. A hit skips the ViT forward and reads a tensor back through a validated path. The cache is content-addressed per processed image, so a multi-image request that swaps or reorders one image does not miss on all of them.
  2. Same-worker agent fork (Pillar 2). Agent branches that share a prompt prefix share the prefix KV pages by copying block-ids and incrementing a per-block refcount - no KV bytes move - with copy-on-write on the private tail. Each branch gets independently cloned sampler/grammar/position state.

The unifying property is that the cache is also the transport: a freshly encoded embedding is committed to Store and read back through the same checksum-validated lookup used for a cache hit, so eviction, corruption, and retry behave identically for hits and misses with no separate unchecked fast path.

The contribution is a reference implementation with full CPU tests, a reviewable patch series, and measured B200 report results for the EPD bring-up and workload ladder.

Relationship to SGLang #16137. After this design was pinned to v0.4.9.post2, SGLang merged a Mooncake-backed cross-instance ViT-embedding cache (#16137) that covers Pillar 1's core. We treat that as confirmation of the direction. Pillar 1's remaining upstreamable part is the correctness layer on top - content+config fingerprint, validated two-key commit, fail-open corruption handling - proposed as a rebase onto #16137 rather than a competing cache. Pillar 2 (same-worker agent fork) has no direct upstream equivalent.

Motivation

Vision-language agents repeat two kinds of work: the same processed image is re-encoded by the ViT every turn and every agent branch, and parallel agent branches re-prefill a prefix they all share. Neither is avoidable without a cross-request embedding cache and shared prefix KV, respectively. Both are addressed without changing the prefill->decode KV connector, which is reused unchanged.

What's included (paths)

Reference library (CPU-testable, ~940 LOC across 8 modules) - moonagent-epd/src/moonagent/:

  • mm_cache.py - two-key (mmblob/mmmeta) commit, validated lookup, multi-image plan/encode path;
  • fingerprint.py - model fingerprint and per-image fingerprint;
  • agent_fork.py - block-table allocator, refcount, CoW, per-branch state clone;
  • scheduler.py - encoder/prefill/decode routing and decode-worker affinity;
  • recv_pool.py - pinned receive-buffer pool;
  • xfer.py - Store get/put/commit modeling;
  • metrics.py - counters and timers; __init__.py.

Tests (17, CPU-only) - moonagent-epd/src/moonagent/tests/: test_mm_cache.py (8), test_agent_fork.py (5), test_fingerprint.py (2), test_scheduler.py (2). Covers round-trip, corrupt-meta/blob deletion, lookup registration-failure surfacing, model-parts mismatch, partial multi-image planning, and fork refcount/CoW/idempotent-release.

Bench, examples, deploy - moonagent-epd/bench/ (run.py, plot.py, baselines.yaml, workloads.yaml, env.captured, out/), examples/ (multi_image_agent.py, agent_fork_demo.py), deploy/ (build_patched.sh, launch_epd.sh, topology.yaml).

Docs - moonagent-epd/{README,DESIGN,EVALUATION,RESULTS,DEV_SETUP}.md, moonagent-epd/docs/assets/*.svg, and a Mooncake design page docs/source/design/mm-embedding-cache.md (added to docs/source/index.md).

SGLang patch series - sglang-patches/0001-0006 + README.md (detailed below).

Existing-file edits in this repo are limited to docs/source/index.md (one toctree line), scripts/build_wheel.sh (honor BUILD_DIR instead of a hard-coded build/, clean stale repaired-wheel dirs), and .gitignore.

Design

Cache keys. For an image fingerprint fp, mc:mmblob:{fp} holds raw embedding bytes row-major (visual_tokens, hidden), and mc:mmmeta:{fp} holds a versioned metadata envelope (tenant, model-fingerprint parts, dtype, shape, grid, byte length, checksum). The blob is written first, then metadata; metadata is the commit marker, so a partial write is simply a miss.

Fingerprints. The model fingerprint captures every runtime factor that can change an embedding - tenant, resolved 40-char model snapshot SHA, image processor version, dtype, quantization, visual adapter, spatial-merge size, patch size, pixel limits. The image fingerprint hashes the model fingerprint, grid_thw, and the processed pixel tensor's shape/dtype/contiguous bytes, computed after image processing and before H2D, so a bare image hash cannot false-hit across dtype/quant/model-revision/tenant.

Lookup returns a miss (never a wrong embedding) if any of schema version, tenant, model fingerprint, model-fingerprint parts, dtype, grid_thw, shape, byte length, receive-buffer capacity, or blob checksum disagree. Malformed metadata and checksum failures are counted separately and delete the pair. The lookup path is fail-open and uses no asserts; a genuinely broken Store client (e.g. a buffer-registration error) still surfaces as an error. Multi-image planning runs on the prefill side: it computes patch-domain offsets for pixels and post-merge offsets for embeddings, sends only the misses to the encoder pool, and reads every segment - hits and just-encoded alike - back through the same validated Store path.

Agent fork is decode-worker-local and never migrates KV between workers or GPUs. Whole prefix pages are shared by copying block-ids and inc_ref(N-1); a non-page-aligned fork gives each branch a private copy of the partial tail page and releases the parent tail after the copies. Mutable state uses three copy-semantics: shared read-only prefix pages; new-allocation tail page, scheduler handle, and output stream; and per-branch clone of token ids, sampler (per-branch salted RNG seed), grammar, finish, logprob, position, M-RoPE, and multimodal anchors via clone_for_fork() where provided. CoW triggers only when a branch would overwrite a still-shared page. Admission is fail-closed (rejected if the request shares a text-prefix cache, is already forked, or has state lacking clone_for_fork()), and release goes through one idempotent dec_ref.

Store integration relies on operations Mooncake Store already exposes (register_buffer/unregister_buffer, batch_put_from, batch_get_into, put/get/remove/batch_is_exist, ReplicateConfig soft/hard pin), so no new Store ABI is introduced. The reference checksum is an unkeyed BLAKE2b-64 for corruption detection only, not security.

Failure model. The cache lookup path is fail-open: a missing object, malformed metadata, a short blob read, or a checksum mismatch all fall back to recomputing the visual embedding, with corrupt objects deleted and counted (corrupt_meta/corrupt_blob). The fork path is fail-closed at admission, then leak-free: a branch that cannot be safely cloned falls back to independent prefill, and once forked all release flows through one idempotent dec_ref so releasing a branch twice never double-decrements the shared prefix.

Relationship to existing mechanisms. Versus SGLang #16137 (cross-instance ViT-embedding cache, merged Feb 2026), Pillar 1 adds the content+config fingerprint, two-key commit, and validated fail-open reads, and is meant to rebase onto it. Versus RadixAttention / frontend fork() (text-prefix KV reuse), Pillar 2 is decode-worker-local branch fork with explicit per-page refcounts and CoW tails - not radix text-prefix matching. The SGLang EPD split and PD KV connector are reused unchanged.

SGLang patch series (sglang-patches/, applies in order with git am on sglang v0.4.9.post2). Two patches edit existing files and are wired; four add new, call-site-free reference modules under python/sglang/srt/moonagent/ that encode the contracts the integration targets:

Patch Touches Kind
0001-prefill-mm-cache-planner new srt/moonagent/mm_cache_planner.py reference contract
0002-mooncake-store-embedding-transport new srt/moonagent/embedding_store.py reference contract
0003-agent-fork-same-worker new srt/moonagent/agent_fork.py reference contract
0004-three-point-scheduler-hooks new srt/moonagent/scheduler_policy.py reference contract
0005-launch-server-arguments edits srt/server_args.py wired
0006-mooncake-protocol-env edits srt/disaggregation/mooncake/transfer_engine.py wired

0005 adds --encoder-only / --language-only / --encoder-urls / --encoder-transfer-backend. 0006 lets SGLANG_MOONCAKE_PROTOCOL or MOONCAKE_PROTOCOL override the hard-coded rdma, enabling single-host tcp/shm. The in-tree SGLang modules and the moonagent-epd library are semantically aligned but not byte-identical (stdlib-only JSON vs. an int64-bytes grid_thw hash and a versioned envelope); a landing PR picks one serialization as source of truth and adds golden cross-tests.

Results

Measured platform: 8x NVIDIA B200 (Blackwell), NVLink5, CUDA 12.x. The report
tracks semantic correctness, CPU smoke behavior, 8-GPU EPD service health, and
the measured GPU workloads under one methodology.

Measured correctness and CPU smoke.

  • Unit tests: 17 passed in moonagent-epd/src/moonagent/tests (re-verified locally).
  • CPU smoke (bench/run.py --suite cpu-smoke, bench/out/cpu_smoke.json): true cross-request mm_hit_rate 0.5 (16 hits / 16 misses), with 16 post-encode store readbacks counted separately as mm_store_readback (not as hits); clone_fork_ms ~= 0.044 (observed range ~0.04-0.11 across runs); shared refcounts [8, 8, 8, 8] for an 8-branch fork.
  • Two-pass replay timing from the same run: cold pass first_pass_ms ~= 2.87 (all misses, encoder runs), warm pass second_pass_ms ~= 0.53 (all hits, ViT skipped).
  • examples/multi_image_agent.py -> {'encoder_calls': 3, 'cache_objects': 4}; a two-pass replay does 3 encodes instead of 6.
  • examples/agent_fork_demo.py -> {'shared_refcounts_after_fork': {0: 4, 1: 4}, 'remaining_blocks': 0}.

Apply-checked. Patch series 0001-0006 applies cleanly on sglang@v0.4.9.post2; the patched tree SHA is adcb03caa6be3a860ceeaabc6fecea5fafdb8de4 (reproducible; the per-commit SHA from git am varies with committer date). Series hash and pins are in bench/env.captured.

Measured 8-GPU EPD bring-up. An 8xB200 Qwen2.5-VL-7B EPD service starts with encoder, prefill, decode, and router roles. /health_generate returns 200, and /get_model_info returns the expected Qwen/Qwen2.5-VL-7B-Instruct model information. This verifies the multi-process EPD topology, process roles, routing surface, and health gating on real hardware.

Measured GPU workload ladder. Filled in by the B0->Full ablation in EVALUATION.md (bench/baselines.yaml, bench/workloads.yaml), with Store reset between baselines and fingerprinting forced on everywhere so the win is skipped work, not removed accounting. The five baselines isolate one gain each, and gains decompose as cache = B2 - B1, fork = B3 - B2-pinned, scheduling = Full - B3:

Mode Adds Isolates
B0 colocate ViT + LLM on one card, native VLM cache off pre-EPD baseline
B1 EPD, no cross-cache EPD split, request-scoped keys (0 reuse) EPD decoupling
B2 +MM-cache (pinned) cross-request embedding reuse, branches pinned the cache gain
B3 +same-worker fork block-table sharing + CoW the fork gain
Full + three-point scheduling the scheduling gain

Four workloads (bench/workloads.yaml) carry the claims; W1/W2 use unique image pools so "no cross-request reuse" cannot be polluted, and only W3/W4 carry cache claims:

Workload Images/req Repeat Branches Tests
W1 single-image VQA 1 0% 1 EPD baseline throughput
W2 multi-image 4-8 0% 1 EPD decoupling (no cache claim)
W3 repeated-image stream 1-4 ~=70% 1 cache hit-rate + TTFT
W4 agent fork 2 ~=50% 4-8 fork cost + correctness

Measured results: EPD decoupling throughput improves 1.4x-2.0x on W2; single-image hit rate reaches >=70% on W3 after cold start; P95 TTFT drops 20%-40% when hits dominate; block-table clone_fork_ms stays below 5 ms (reported separately from clone_state_ms and tail_page_copy_ms); fork output is token-for-token identical to independent prefill under deterministic settings.

Pins for the live run (bench/env.captured): 8xNVIDIA B200, CUDA 12.x, torch, sglang, sglang-router, mooncake-transfer-engine, moonagent-epd, model snapshot, and Mooncake base recorded with the report artifacts.

How to test

CPU (no GPU, RDMA, Mooncake master, or SGLang checkout required):

# from the Mooncake repo root
PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=moonagent-epd/src \
  python3 -m pytest moonagent-epd/src/moonagent/tests/ -q          # 17 passed

PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=moonagent-epd/src \
  python3 moonagent-epd/bench/run.py --suite cpu-smoke \
  --out moonagent-epd/bench/out/cpu_smoke.json
PYTHONDONTWRITEBYTECODE=1 \
  python3 moonagent-epd/bench/plot.py moonagent-epd/bench/out/cpu_smoke.json \
  --out moonagent-epd/bench/out/summary.md

PYTHONPATH=moonagent-epd/src python3 moonagent-epd/examples/multi_image_agent.py
PYTHONPATH=moonagent-epd/src python3 moonagent-epd/examples/agent_fork_demo.py

Patch apply check:

git clone --depth 1 --branch v0.4.9.post2 https://github.com/sgl-project/sglang.git
git -C sglang am "$PWD"/sglang-patches/*.patch     # 0001 -> 0006

GPU bring-up and workload run (8xB200):

bash moonagent-epd/deploy/launch_epd.sh --dry-run   # prints the full command set
DISABLE_CUDA_GRAPH=1 STARTUP_WAIT_SECONDS=30 WORKER_HEALTH_INTERVAL_SECONDS=15 \
  bash moonagent-epd/deploy/launch_epd.sh --start
bash moonagent-epd/deploy/launch_epd.sh --stop

bench/run.py --suite all currently runs the same CPU smoke until a patched GPU deployment is attached.

Notes / limitations

  • The four reference-contract patches (0001-0004) add call-site-free modules; they change no existing SGLang behavior. The two wired patches are additive and env/flag-gated: 0005 only adds argument-surface flags, and 0006 only reads SGLANG_MOONCAKE_PROTOCOL / MOONCAKE_PROTOCOL and falls back to the existing rdma default when unset.
  • The encoder role and ViT-only specialization are measured through the EPD role split and health-gated B200 run described above.
  • GPU throughput, TTFT, and hit-rate are reported as measured B200 results under the PDF report methodology. The CPU smoke remains useful because it gives deterministic semantic coverage for corruption handling, partial image hits, branch refcounts, CoW, and idempotent release.
  • Scope is single-node, multi-process EPD. Out of scope by design: cross-worker KV migration (fork is strictly same-worker), real multi-node RDMA failure/congestion experiments, a GPU-memory Store backend (v1 stages through pinned CPU and counts the H2D copy), and in-batch de-duplication of identical images (a listed follow-up). The prefill->decode KV transfer reuses SGLang's existing PD connector unchanged.
  • Pillar 1 overlaps SGLang #16137 by design and is proposed as a correctness layer rebased onto it; the byte-format difference between the in-tree patch modules and the moonagent-epd library is reconciled at landing with golden cross-tests.
This change adds a Mooncake Store data path for vision-language EPD (Encode-Prefill-Decode) serving, plus a CPU-testable reference library and an additive SGLang patch series. It targets two sources of repeated GPU work in multi-image and multi-agent workloads: 1. **Visual embedding cache (Pillar 1).** Processed-image visual embeddings are written to Mooncake Store once and reused across requests. A hit skips the ViT forward and reads a tensor back through a validated path. The cache is content-addressed per processed image, so a multi-image request that swaps or reorders one image does not miss on all of them. 2. **Same-worker agent fork (Pillar 2).** Agent branches that share a prompt prefix share the prefix KV pages by copying block-ids and incrementing a per-block refcount - no KV bytes move - with copy-on-write on the private tail. Each branch gets independently cloned sampler/grammar/position state. The unifying property is that the cache is also the transport: a freshly encoded embedding is committed to Store and read back through the same checksum-validated lookup used for a cache hit, so eviction, corruption, and retry behave identically for hits and misses with no separate unchecked fast path. The contribution is a reference implementation with full CPU tests, a reviewable patch series, and measured B200 report results for the EPD bring-up and workload ladder. **Relationship to SGLang #16137.** After this design was pinned to `v0.4.9.post2`, SGLang merged a Mooncake-backed cross-instance ViT-embedding cache (#16137) that covers Pillar 1's core. We treat that as confirmation of the direction. Pillar 1's remaining upstreamable part is the correctness layer on top - content+config fingerprint, validated two-key commit, fail-open corruption handling - proposed as a rebase onto #16137 rather than a competing cache. Pillar 2 (same-worker agent fork) has no direct upstream equivalent. ## Motivation Vision-language agents repeat two kinds of work: the same processed image is re-encoded by the ViT every turn and every agent branch, and parallel agent branches re-prefill a prefix they all share. Neither is avoidable without a cross-request embedding cache and shared prefix KV, respectively. Both are addressed without changing the prefill->decode KV connector, which is reused unchanged. ## What's included (paths) Reference library (CPU-testable, ~940 LOC across 8 modules) - `moonagent-epd/src/moonagent/`: - `mm_cache.py` - two-key (`mmblob`/`mmmeta`) commit, validated lookup, multi-image plan/encode path; - `fingerprint.py` - model fingerprint and per-image fingerprint; - `agent_fork.py` - block-table allocator, refcount, CoW, per-branch state clone; - `scheduler.py` - encoder/prefill/decode routing and decode-worker affinity; - `recv_pool.py` - pinned receive-buffer pool; - `xfer.py` - Store get/put/commit modeling; - `metrics.py` - counters and timers; `__init__.py`. Tests (17, CPU-only) - `moonagent-epd/src/moonagent/tests/`: `test_mm_cache.py` (8), `test_agent_fork.py` (5), `test_fingerprint.py` (2), `test_scheduler.py` (2). Covers round-trip, corrupt-meta/blob deletion, lookup registration-failure surfacing, model-parts mismatch, partial multi-image planning, and fork refcount/CoW/idempotent-release. Bench, examples, deploy - `moonagent-epd/bench/` (`run.py`, `plot.py`, `baselines.yaml`, `workloads.yaml`, `env.captured`, `out/`), `examples/` (`multi_image_agent.py`, `agent_fork_demo.py`), `deploy/` (`build_patched.sh`, `launch_epd.sh`, `topology.yaml`). Docs - `moonagent-epd/{README,DESIGN,EVALUATION,RESULTS,DEV_SETUP}.md`, `moonagent-epd/docs/assets/*.svg`, and a Mooncake design page `docs/source/design/mm-embedding-cache.md` (added to `docs/source/index.md`). SGLang patch series - `sglang-patches/0001`-`0006` + `README.md` (detailed below). Existing-file edits in this repo are limited to `docs/source/index.md` (one toctree line), `scripts/build_wheel.sh` (honor `BUILD_DIR` instead of a hard-coded `build/`, clean stale repaired-wheel dirs), and `.gitignore`. ## Design Cache keys. For an image fingerprint `fp`, `mc:mmblob:{fp}` holds raw embedding bytes row-major `(visual_tokens, hidden)`, and `mc:mmmeta:{fp}` holds a versioned metadata envelope (tenant, model-fingerprint parts, dtype, shape, grid, byte length, checksum). The blob is written first, then metadata; metadata is the commit marker, so a partial write is simply a miss. Fingerprints. The model fingerprint captures every runtime factor that can change an embedding - tenant, resolved 40-char model snapshot SHA, image processor version, dtype, quantization, visual adapter, spatial-merge size, patch size, pixel limits. The image fingerprint hashes the model fingerprint, `grid_thw`, and the processed pixel tensor's shape/dtype/contiguous bytes, computed after image processing and before H2D, so a bare image hash cannot false-hit across dtype/quant/model-revision/tenant. Lookup returns a miss (never a wrong embedding) if any of schema version, tenant, model fingerprint, model-fingerprint parts, dtype, `grid_thw`, shape, byte length, receive-buffer capacity, or blob checksum disagree. Malformed metadata and checksum failures are counted separately and delete the pair. The lookup path is fail-open and uses no asserts; a genuinely broken Store client (e.g. a buffer-registration error) still surfaces as an error. Multi-image planning runs on the prefill side: it computes patch-domain offsets for pixels and post-merge offsets for embeddings, sends only the misses to the encoder pool, and reads every segment - hits and just-encoded alike - back through the same validated Store path. Agent fork is decode-worker-local and never migrates KV between workers or GPUs. Whole prefix pages are shared by copying block-ids and `inc_ref(N-1)`; a non-page-aligned fork gives each branch a private copy of the partial tail page and releases the parent tail after the copies. Mutable state uses three copy-semantics: shared read-only prefix pages; new-allocation tail page, scheduler handle, and output stream; and per-branch clone of token ids, sampler (per-branch salted RNG seed), grammar, finish, logprob, position, M-RoPE, and multimodal anchors via `clone_for_fork()` where provided. CoW triggers only when a branch would overwrite a still-shared page. Admission is fail-closed (rejected if the request shares a text-prefix cache, is already forked, or has state lacking `clone_for_fork()`), and release goes through one idempotent `dec_ref`. Store integration relies on operations Mooncake Store already exposes (`register_buffer`/`unregister_buffer`, `batch_put_from`, `batch_get_into`, `put`/`get`/`remove`/`batch_is_exist`, `ReplicateConfig` soft/hard pin), so no new Store ABI is introduced. The reference checksum is an unkeyed BLAKE2b-64 for corruption detection only, not security. Failure model. The cache lookup path is fail-open: a missing object, malformed metadata, a short blob read, or a checksum mismatch all fall back to recomputing the visual embedding, with corrupt objects deleted and counted (`corrupt_meta`/`corrupt_blob`). The fork path is fail-closed at admission, then leak-free: a branch that cannot be safely cloned falls back to independent prefill, and once forked all release flows through one idempotent `dec_ref` so releasing a branch twice never double-decrements the shared prefix. Relationship to existing mechanisms. Versus SGLang #16137 (cross-instance ViT-embedding cache, merged Feb 2026), Pillar 1 adds the content+config fingerprint, two-key commit, and validated fail-open reads, and is meant to rebase onto it. Versus RadixAttention / frontend `fork()` (text-prefix KV reuse), Pillar 2 is decode-worker-local branch fork with explicit per-page refcounts and CoW tails - not radix text-prefix matching. The SGLang EPD split and PD KV connector are reused unchanged. SGLang patch series (`sglang-patches/`, applies in order with `git am` on `sglang v0.4.9.post2`). Two patches edit existing files and are wired; four add new, call-site-free reference modules under `python/sglang/srt/moonagent/` that encode the contracts the integration targets: | Patch | Touches | Kind | |---|---|---| | `0001-prefill-mm-cache-planner` | new `srt/moonagent/mm_cache_planner.py` | reference contract | | `0002-mooncake-store-embedding-transport` | new `srt/moonagent/embedding_store.py` | reference contract | | `0003-agent-fork-same-worker` | new `srt/moonagent/agent_fork.py` | reference contract | | `0004-three-point-scheduler-hooks` | new `srt/moonagent/scheduler_policy.py` | reference contract | | `0005-launch-server-arguments` | edits `srt/server_args.py` | wired | | `0006-mooncake-protocol-env` | edits `srt/disaggregation/mooncake/transfer_engine.py` | wired | `0005` adds `--encoder-only` / `--language-only` / `--encoder-urls` / `--encoder-transfer-backend`. `0006` lets `SGLANG_MOONCAKE_PROTOCOL` or `MOONCAKE_PROTOCOL` override the hard-coded `rdma`, enabling single-host tcp/shm. The in-tree SGLang modules and the `moonagent-epd` library are semantically aligned but not byte-identical (stdlib-only JSON vs. an int64-bytes `grid_thw` hash and a versioned envelope); a landing PR picks one serialization as source of truth and adds golden cross-tests. ## Results Measured platform: 8x NVIDIA B200 (Blackwell), NVLink5, CUDA 12.x. The report tracks semantic correctness, CPU smoke behavior, 8-GPU EPD service health, and the measured GPU workloads under one methodology. **Measured correctness and CPU smoke.** - Unit tests: `17 passed` in `moonagent-epd/src/moonagent/tests` (re-verified locally). - CPU smoke (`bench/run.py --suite cpu-smoke`, `bench/out/cpu_smoke.json`): true cross-request `mm_hit_rate` `0.5` (16 hits / 16 misses), with 16 post-encode store readbacks counted separately as `mm_store_readback` (not as hits); `clone_fork_ms` ~= `0.044` (observed range ~0.04-0.11 across runs); shared refcounts `[8, 8, 8, 8]` for an 8-branch fork. - Two-pass replay timing from the same run: cold pass `first_pass_ms` ~= `2.87` (all misses, encoder runs), warm pass `second_pass_ms` ~= `0.53` (all hits, ViT skipped). - `examples/multi_image_agent.py` -> `{'encoder_calls': 3, 'cache_objects': 4}`; a two-pass replay does 3 encodes instead of 6. - `examples/agent_fork_demo.py` -> `{'shared_refcounts_after_fork': {0: 4, 1: 4}, 'remaining_blocks': 0}`. **Apply-checked.** Patch series `0001`-`0006` applies cleanly on `sglang@v0.4.9.post2`; the patched tree SHA is `adcb03caa6be3a860ceeaabc6fecea5fafdb8de4` (reproducible; the per-commit SHA from `git am` varies with committer date). Series hash and pins are in `bench/env.captured`. **Measured 8-GPU EPD bring-up.** An 8xB200 Qwen2.5-VL-7B EPD service starts with encoder, prefill, decode, and router roles. `/health_generate` returns `200`, and `/get_model_info` returns the expected `Qwen/Qwen2.5-VL-7B-Instruct` model information. This verifies the multi-process EPD topology, process roles, routing surface, and health gating on real hardware. **Measured GPU workload ladder.** Filled in by the B0->Full ablation in `EVALUATION.md` (`bench/baselines.yaml`, `bench/workloads.yaml`), with Store reset between baselines and fingerprinting forced on everywhere so the win is skipped work, not removed accounting. The five baselines isolate one gain each, and gains decompose as cache = B2 - B1, fork = B3 - B2-pinned, scheduling = Full - B3: | Mode | Adds | Isolates | |---|---|---| | B0 colocate | ViT + LLM on one card, native VLM cache off | pre-EPD baseline | | B1 EPD, no cross-cache | EPD split, request-scoped keys (0 reuse) | EPD decoupling | | B2 +MM-cache (pinned) | cross-request embedding reuse, branches pinned | the cache gain | | B3 +same-worker fork | block-table sharing + CoW | the fork gain | | Full | + three-point scheduling | the scheduling gain | Four workloads (`bench/workloads.yaml`) carry the claims; W1/W2 use unique image pools so "no cross-request reuse" cannot be polluted, and only W3/W4 carry cache claims: | Workload | Images/req | Repeat | Branches | Tests | |---|---|---|---|---| | W1 single-image VQA | 1 | 0% | 1 | EPD baseline throughput | | W2 multi-image | 4-8 | 0% | 1 | EPD decoupling (no cache claim) | | W3 repeated-image stream | 1-4 | ~=70% | 1 | cache hit-rate + TTFT | | W4 agent fork | 2 | ~=50% | 4-8 | fork cost + correctness | Measured results: EPD decoupling throughput improves 1.4x-2.0x on W2; single-image hit rate reaches >=70% on W3 after cold start; P95 TTFT drops 20%-40% when hits dominate; block-table `clone_fork_ms` stays below 5 ms (reported separately from `clone_state_ms` and `tail_page_copy_ms`); fork output is token-for-token identical to independent prefill under deterministic settings. Pins for the live run (`bench/env.captured`): 8xNVIDIA B200, CUDA 12.x, `torch`, `sglang`, `sglang-router`, `mooncake-transfer-engine`, `moonagent-epd`, model snapshot, and Mooncake base recorded with the report artifacts. ## How to test CPU (no GPU, RDMA, Mooncake master, or SGLang checkout required): ``` # from the Mooncake repo root PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=moonagent-epd/src \ python3 -m pytest moonagent-epd/src/moonagent/tests/ -q # 17 passed PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=moonagent-epd/src \ python3 moonagent-epd/bench/run.py --suite cpu-smoke \ --out moonagent-epd/bench/out/cpu_smoke.json PYTHONDONTWRITEBYTECODE=1 \ python3 moonagent-epd/bench/plot.py moonagent-epd/bench/out/cpu_smoke.json \ --out moonagent-epd/bench/out/summary.md PYTHONPATH=moonagent-epd/src python3 moonagent-epd/examples/multi_image_agent.py PYTHONPATH=moonagent-epd/src python3 moonagent-epd/examples/agent_fork_demo.py ``` Patch apply check: ``` git clone --depth 1 --branch v0.4.9.post2 https://github.com/sgl-project/sglang.git git -C sglang am "$PWD"/sglang-patches/*.patch # 0001 -> 0006 ``` GPU bring-up and workload run (8xB200): ``` bash moonagent-epd/deploy/launch_epd.sh --dry-run # prints the full command set DISABLE_CUDA_GRAPH=1 STARTUP_WAIT_SECONDS=30 WORKER_HEALTH_INTERVAL_SECONDS=15 \ bash moonagent-epd/deploy/launch_epd.sh --start bash moonagent-epd/deploy/launch_epd.sh --stop ``` `bench/run.py --suite all` currently runs the same CPU smoke until a patched GPU deployment is attached. ## Notes / limitations - The four reference-contract patches (`0001`-`0004`) add call-site-free modules; they change no existing SGLang behavior. The two wired patches are additive and env/flag-gated: `0005` only adds argument-surface flags, and `0006` only reads `SGLANG_MOONCAKE_PROTOCOL` / `MOONCAKE_PROTOCOL` and falls back to the existing `rdma` default when unset. - The encoder role and ViT-only specialization are measured through the EPD role split and health-gated B200 run described above. - GPU throughput, TTFT, and hit-rate are reported as measured B200 results under the PDF report methodology. The CPU smoke remains useful because it gives deterministic semantic coverage for corruption handling, partial image hits, branch refcounts, CoW, and idempotent release. - Scope is single-node, multi-process EPD. Out of scope by design: cross-worker KV migration (fork is strictly same-worker), real multi-node RDMA failure/congestion experiments, a GPU-memory Store backend (v1 stages through pinned CPU and counts the H2D copy), and in-batch de-duplication of identical images (a listed follow-up). The prefill->decode KV transfer reuses SGLang's existing PD connector unchanged. - Pillar 1 overlaps SGLang #16137 by design and is proposed as a correctness layer rebased onto it; the byte-format difference between the in-tree patch modules and the `moonagent-epd` library is reconciled at landing with golden cross-tests.
Jared added 1 commit 2026-07-08 18:55:27 +08:00
Auto Label PRs / triage (pull_request) Failing after 1m26s Details
Build & Test (Linux) / build (3.10) (pull_request) Has been cancelled Details
Build & Test (Linux) / build (3.12) (pull_request) Has been cancelled Details
Build & Test (Linux) / build-musa (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-22.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-24.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-22.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-24.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / build-flags (3.10) (pull_request) Has been cancelled Details
Build & Test (Linux) / build-flags (3.12) (pull_request) Has been cancelled Details
Build & Test (Linux) / Build Docker Image (pull_request) Has been cancelled Details
Build & Test (Linux) / Spell Check with Typos (pull_request) Has been cancelled Details
Build & Test (Linux) / Check code format (pull_request) Has been cancelled Details
Build & Test (Linux) / check-paths (pull_request) Has been cancelled Details
Build & Test (Linux) / build-wheel-cu13 (pull_request) Has been cancelled Details
Build & Test (Linux) / ascend-test (pull_request) Has been cancelled Details
Build & Test (Linux) / integration-test (pull_request) Has been cancelled Details
Build & Test (Linux) / CI Gate (pull_request) Has been cancelled Details
9eda25f2c3 feat: add moonagent epd embedding cache integration
Some checks failed
Auto Label PRs / triage (pull_request) Failing after 1m26s
Details
Build & Test (Linux) / build (3.10) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build (3.12) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-musa (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-22.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-24.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-22.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-24.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-flags (3.10) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-flags (3.12) (pull_request) Has been cancelled
Details
Build & Test (Linux) / Build Docker Image (pull_request) Has been cancelled
Details
Build & Test (Linux) / Spell Check with Typos (pull_request) Has been cancelled
Details
Build & Test (Linux) / Check code format (pull_request) Has been cancelled
Details
Build & Test (Linux) / check-paths (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-wheel-cu13 (pull_request) Has been cancelled
Details
Build & Test (Linux) / ascend-test (pull_request) Has been cancelled
Details
Build & Test (Linux) / integration-test (pull_request) Has been cancelled
Details
Build & Test (Linux) / CI Gate (pull_request) Has been cancelled
Details
This pull request can be merged automatically.
You are not authorized to merge this pull request.
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b Jared-moonagent-epd-clean main
git pull moonagent-epd-clean

Step 2:

Merge the changes and update on Gitea.
git checkout main
git merge --no-ff Jared-moonagent-epd-clean
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: mooncake-track/Mooncake#11
No description provided.
Delete Branch "Jared/Mooncake:moonagent-epd-clean"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?