mooncake-track
/
Mooncake
0
0
Fork 12
Code Issues Pull Requests 13 Actions Packages Projects Releases Wiki Activity

EPD Hidden State path and Store type-aware eviction #14

Open
kancel wants to merge 1 commits from kancel/Mooncake:ccf-track4-submission into main
pull from: kancel/Mooncake:ccf-track4-submission
merge into: mooncake-track:main
Conversation 0 Commits 1 Files Changed 64 +23520
kancel commented 2026-07-09 13:53:03 +08:00
First-time contributor
Copy Link

CCF Mooncake Track 4 Submission

This PR archives the CCF Mooncake Track 4 submission materials under
mooncake-track4/.

The submission targets two gaps in multimodal EPD serving:

  1. Pillar 1: EPD Hidden State data path. It adds a vLLM-side
    MooncakeStoreECConnector implementation so Encoder Hidden State can be
    written to Mooncake Store and consumed by Prefill workers. This complements
    the existing Prefill -> Decode KV Cache path.
  2. Pillar 2: mixed-object cache governance. It adds Store-level object type
    semantics and type-aware eviction policies so Hidden State and KV Cache can
    be managed differently under the same memory pressure.
  3. Architecture follow-up. It includes a hierarchical eviction policy
    abstraction patch that responds to Mooncake review feedback about policy
    logic accumulating inside master_service.

The full reproduction commands, environment variables, and troubleshooting
notes are kept in the README files under each subdirectory.

Motivation

Multimodal EPD splits one request into Encoder, Prefill, and Decode stages.
Mooncake and vLLM already provide a KV Cache transfer path for Prefill ->
Decode, but Vision Encoder outputs also need to cross process boundaries.
Without treating Hidden State as a first-class Store object, EPD can split the
process topology but cannot reuse Encoder computation across instances.

Once Hidden State enters Mooncake Store, the Store becomes a mixed-object cache
instead of a KV-only cache. KV Cache objects dominate bytes and are reused
frequently. Hidden State objects are smaller in total bytes, but they are more
expensive to recompute and are reused over longer intervals. A single global
lease-time / LRU-like eviction rule can evict valuable Hidden State too early,
while simply protecting Hidden State can hurt the KV hot path. This motivates
type-aware scoring and per-type capacity guardrails.

Relationship To Upstream

The implementation is designed as an additive extension on top of existing
upstream abstractions:

  • vLLM side: MooncakeStoreECConnector reuses the V1 ECConnectorBase
    Scheduler / Worker lifecycle. It does not modify the core scheduler loop,
    worker execution loop, or multimodal model code.
  • KV path: Prefill -> Decode still uses the existing vLLM
    MooncakeStoreConnector.
  • Mooncake Store side: type-aware eviction is optional. With no object-type
    policy configured, the default global eviction behavior is preserved.
  • Architecture follow-up: the hierarchical policy patch is a response to review
    feedback from Mooncake PR #2746 and the follow-up discussion in issue #2791.

Related upstream work:

  • vLLM Hidden State ECConnector PR: vllm-project/vllm#47302
  • Mooncake Store type-aware eviction PR series: kvcache-ai/Mooncake#2743 to
    #2746, plus related follow-up #2689
  • Hierarchical eviction abstraction discussion:
    https://github.com/kvcache-ai/Mooncake/issues/2791

What's Included

mooncake-track4/
  README.md                         # submission entrypoint and shortest test paths
  DESIGN.md                         # design details, interfaces, data flow, tradeoffs
  EVALUATION.md                     # environment, baselines, full results, analysis
  epd-vllm-demo/                    # EPD end-to-end reproduction package
    mooncake/epd_vllm_v1_proxy_server.py  # EPD control-plane proxy
    configs/                        # environment templates
    scripts/                        # preflight, correctness matrix, benchmark scripts
    tests/                          # local tests for proxy and helper scripts
    workloads/                      # OpenAI-compatible multimodal payloads
    results/                        # archived EPD results and metrics
  object-type-eviction-bench/       # Store-level type-aware eviction benchmark
    benchmarks/                     # continuous multimodal session workload
    scripts/                        # single-case smoke and full policy suite runners
    results/                        # archived representative policy results
  patches/
    vllm/                           # vLLM MooncakeStoreECConnector patches
    mooncake/                       # Mooncake Store policy and architecture patches

Key artifacts:

  • patches/vllm/patches/vllm-epd-hidden-ec-connector-b4482f0a1-full-feature.patch
    archives the vLLM-side MooncakeStoreECConnector.
  • epd-vllm-demo/mooncake/epd_vllm_v1_proxy_server.py archives the EPD demo
    proxy used for request splitting and transfer metadata forwarding.
  • patches/mooncake/0001-store-object-type-aware-eviction-policy.patch
    archives the Store object type, type-aware scoring, per-type budget, and
    eviction_grace changes.
  • patches/mooncake/0002-store-hierarchical-eviction-policy-abstraction.patch
    archives the policy abstraction follow-up.

Design

1. EPD Hidden State Data Path

MooncakeStoreECConnector follows the vLLM EC Connector split between
Scheduler-side planning and Worker-side data movement. The Scheduler side
performs lookup, load/save planning, and connector metadata construction. The
Worker side performs the actual Mooncake Store reads and writes. The EPD proxy
is only a control-plane component; it forwards request metadata and does not
carry tensor payloads.

client
  -> EPD proxy
    -> encoder: MooncakeStoreECConnector, ec_producer
    -> prefill: MooncakeStoreECConnector + MooncakeStoreConnector,
                ec_consumer + kv_producer
    -> decode:  MooncakeStoreConnector, kv_consumer

Hidden State uses a separate key namespace that includes model, vision config,
parallel layout, storage layout, and tensor format information. This prevents
incorrect reuse across incompatible configurations. The stored object format is
a fixed metadata header plus a contiguous tensor payload, matching Mooncake
Store's buffer-registration and range-read capabilities.

2. Failure Model And Compatibility

The submission explicitly separates cache-fill failure, cache-load failure, and
Store API compatibility:

  • If an Encoder-side Hidden State save fails, the object is not reported as
    finished. Later requests naturally fall back to the miss / recompute path.
  • Prefill-side Hidden State loading is a pre-execution dependency. A load
    failure does not populate local encoder_cache; the error is surfaced rather
    than allowing an invalid tensor to continue through inference.
  • The Store client wrapper handles Mooncake Python binding differences and
    prefers the buffer-registration path when available.
  • The EPD proxy only forwards control metadata. Tensor payloads stay on the
    Mooncake Store data path.
  • Store type-aware eviction keeps default behavior compatible when no policy is
    configured.

3. Store Type-Aware Eviction

The Store patch adds ObjectDataType.HIDDEN_STATE and extends BatchEvict
candidate selection with object type information. The policy knobs are:

  • reuse_scale: adjusts cross-type victim ranking to reflect recomputation
    value.
  • eviction_grace: adds an extra protection window for object types whose reuse
    interval is longer.
  • budget_ratio: provides a per-type capacity guardrail so one type does not
    unboundedly squeeze out another.

The ranking uses type-adjusted age:

adjusted_age = max(0, rank_reference_time - lease_timeout - eviction_grace)
             * soft_pin_weight / reuse_scale

budget_ratio is a logical budget rather than a physical partition. The Store
keeps a shared memory pool, while over-budget types contribute victims first
under pressure.

4. Hierarchical Eviction Abstraction

The follow-up patch separates policy planning from master_service metadata
mutation. MasterService still scans candidates, revalidates metadata, and
performs the actual deletion. The policy module only works on candidate
references and scope indexes.

The abstraction includes:

  • Assign: assign eviction responsibility by type, tenant, or future priority
    scopes.
  • Evict: select victim candidates within a scope.
  • Fallback: compensate from the parent scope when a child scope cannot meet
    its target.

The current patch validates a type-level single-layer abstraction. A full
tenant -> type -> priority recursive executor is future work.

Results

Detailed environment settings and raw result summaries are in
mooncake-track4/EVALUATION.md.

1. EPD Correctness

The EPD experiment was run on a single-node, three-GPU TCP-only setup with
Qwen/Qwen2.5-VL-7B-Instruct. The result is correctness and reproducibility
evidence, not an RDMA / SHM performance upper bound.

correctness matrix: 6/6 passed
required hidden evidence cases: 2/2 passed
encoder_hidden_store_put: 2
prefill_hidden_store_scheduler_hit: 2
prefill_hidden_store_get: 2
prefill_hidden_store_scheduler_miss: 0
output comparison cases: 6/6 passed
exact output match cases: 6/6

2. TCP-Only Transfer Baseline

successful requests: 6/6
request throughput: 0.667639 req/s
TTFT p50: 835.323 ms
TTFT p95: 2666.689 ms
latency p50: 1180.717 ms
latency p95: 2950.505 ms
Path Bytes Time Bandwidth
Hidden E->Store save 32,062,464 44.033 ms 694.414 MiB/s
Hidden Store->P load 32,062,464 33.520 ms 912.206 MiB/s
KV P->Store save_put 206,438,400 623.245 ms 315.887 MiB/s

3. Type-Aware Eviction Ablation

The Store benchmark simulates 600 continuous multimodal sessions. KV Cache
accounts for about 90.88% of unique bytes, while Hidden State accounts for about
9.12%. KV is reused more frequently; Hidden State is reused more sparsely but
has higher recomputation cost.

Policy Hidden hit KV hit Saving Time Rate Main takeaway
baseline default 33.94% 96.65% 87.80% original global policy
hidden grace only 67.89% 95.84% 92.42% Hidden +33.95 pp, KV -0.81 pp
hidden reuse scale only 84.71% 94.98% 94.38% Hidden +50.77 pp, KV -1.67 pp
balanced type-aware policy 75.54% 95.74% 94.26% Hidden +41.60 pp, KV -0.91 pp

Balanced configuration:

HIDDEN_STATE reuse_scale = 8.0
HIDDEN_STATE eviction_grace = 10000
KVCACHE budget_ratio = 0.80
HIDDEN_STATE budget_ratio = 0.12

4. Hierarchical Eviction Validation

The hierarchical eviction patch is based on official Mooncake c9896684. It
adds a standalone policy module and focused unit test coverage for type-level
Assign / Evict / Fallback planning, candidate-index isolation from metadata
mutation, and compatibility with the default eviction semantics.
eviction_policy_test passes; full commands are in
mooncake-track4/patches/mooncake/README.md.

How To Test

The test entrypoints are grouped from minimal smoke to full end-to-end
reproduction. Full build and environment details are in the subdirectory
README files.

From the GitLink repository root:

export SUBMISSION_ROOT="$PWD"
export TRACK4_ROOT="$SUBMISSION_ROOT/mooncake-track4"

1. Minimal Smoke: Benchmark Harness

No GPU or EPD service is required.

cd "$TRACK4_ROOT/object-type-eviction-bench"
python3 -m unittest benchmarks/test_continuous_session_policy_bench.py -v

2. Store Policy Module

Apply patches/mooncake/0001-store-object-type-aware-eviction-policy.patch to
official Mooncake f4f7fd4a03539974fc39acbd045b1ee4ba0ed967, then build
mooncake_master and the Python binding. See
object-type-eviction-bench/README.md for the exact build commands.

Single policy case:

cd "$TRACK4_ROOT/object-type-eviction-bench"
MOONCAKE_ROOT=/tmp/mooncake-object-type-runtime \
OUT_DIR=/tmp/mooncake_single_policy_case \
bash scripts/run_single_policy_case.sh

Full policy suite:

MOONCAKE_ROOT=/tmp/mooncake-object-type-runtime \
OUT_DIR=/tmp/mooncake_scale_0006_policy_suite \
bash scripts/run_pr2_pr4_scale_0006_suite.sh

3. Hierarchical Eviction Patch

git clone https://github.com/kvcache-ai/Mooncake.git ../mooncake-hier-eviction
cd ../mooncake-hier-eviction
git checkout c9896684
git config user.name reviewer
git config user.email reviewer@example.com
git am "$TRACK4_ROOT/patches/mooncake/0002-store-hierarchical-eviction-policy-abstraction.patch"

cmake -S . -B build-hier-tests \
  -DWITH_STORE=ON -DWITH_TE=ON -DWITH_EP=OFF \
  -DWITH_STORE_GO=OFF -DWITH_STORE_RUST=OFF -DWITH_P2P_STORE=OFF \
  -DBUILD_UNIT_TESTS=ON -DBUILD_EXAMPLES=OFF -DBUILD_BENCHMARK=OFF

cmake --build build-hier-tests --target eviction_policy_test -j2
cd build-hier-tests
ctest -R eviction_policy_test --output-on-failure

4. EPD End-To-End

This requires a three-GPU environment and a patched vLLM runtime. If the server
does not already provide a patched vLLM wheel, apply the vLLM patch documented
in patches/vllm/README.md.

cd "$TRACK4_ROOT/epd-vllm-demo"
cp configs/server_env.example.sh configs/server_env.local.sh
vim configs/server_env.local.sh
source configs/server_env.local.sh
bash "$EVAL_ROOT/scripts/check_server_env.sh"
FULL_EPD_MATRIX=1 bash "$EVAL_ROOT/scripts/run_full_epd_mooncake_pd_smoke.sh"

Notes And Limitations

  • The EPD experiment is TCP-only correctness and reproducibility evidence. It
    is not an RDMA / SHM performance upper bound.
  • The Store eviction benchmark is a Store-level isolated experiment. It measures
    cache retention behavior, not end-to-end model latency.
  • The object-type eviction patch must be applied on the documented baseline
    f4f7fd4a03539974fc39acbd045b1ee4ba0ed967; current official main should
    not be assumed to accept it directly.
  • The hierarchical eviction abstraction currently validates a type-level
    single-layer policy boundary. A full multi-layer executor is future work.
  • The EPD proxy is a demo orchestration component, not part of the upstream vLLM
    connector itself.
  • The current EPD validation focuses on a single-node, three-GPU, single-rank
    Hidden State object path. Multi-node RDMA, sharded Hidden State, and richer
    cross-request cache policies are future work.
# CCF Mooncake Track 4 Submission This PR archives the CCF Mooncake Track 4 submission materials under `mooncake-track4/`. The submission targets two gaps in multimodal EPD serving: 1. **Pillar 1: EPD Hidden State data path.** It adds a vLLM-side `MooncakeStoreECConnector` implementation so Encoder Hidden State can be written to Mooncake Store and consumed by Prefill workers. This complements the existing Prefill -> Decode KV Cache path. 2. **Pillar 2: mixed-object cache governance.** It adds Store-level object type semantics and type-aware eviction policies so Hidden State and KV Cache can be managed differently under the same memory pressure. 3. **Architecture follow-up.** It includes a hierarchical eviction policy abstraction patch that responds to Mooncake review feedback about policy logic accumulating inside `master_service`. The full reproduction commands, environment variables, and troubleshooting notes are kept in the README files under each subdirectory. ## Motivation Multimodal EPD splits one request into Encoder, Prefill, and Decode stages. Mooncake and vLLM already provide a KV Cache transfer path for Prefill -> Decode, but Vision Encoder outputs also need to cross process boundaries. Without treating Hidden State as a first-class Store object, EPD can split the process topology but cannot reuse Encoder computation across instances. Once Hidden State enters Mooncake Store, the Store becomes a mixed-object cache instead of a KV-only cache. KV Cache objects dominate bytes and are reused frequently. Hidden State objects are smaller in total bytes, but they are more expensive to recompute and are reused over longer intervals. A single global lease-time / LRU-like eviction rule can evict valuable Hidden State too early, while simply protecting Hidden State can hurt the KV hot path. This motivates type-aware scoring and per-type capacity guardrails. ## Relationship To Upstream The implementation is designed as an additive extension on top of existing upstream abstractions: - vLLM side: `MooncakeStoreECConnector` reuses the V1 `ECConnectorBase` Scheduler / Worker lifecycle. It does not modify the core scheduler loop, worker execution loop, or multimodal model code. - KV path: Prefill -> Decode still uses the existing vLLM `MooncakeStoreConnector`. - Mooncake Store side: type-aware eviction is optional. With no object-type policy configured, the default global eviction behavior is preserved. - Architecture follow-up: the hierarchical policy patch is a response to review feedback from Mooncake PR #2746 and the follow-up discussion in issue #2791. Related upstream work: - vLLM Hidden State ECConnector PR: `vllm-project/vllm#47302` - Mooncake Store type-aware eviction PR series: `kvcache-ai/Mooncake#2743` to `#2746`, plus related follow-up `#2689` - Hierarchical eviction abstraction discussion: `https://github.com/kvcache-ai/Mooncake/issues/2791` ## What's Included ```text mooncake-track4/ README.md # submission entrypoint and shortest test paths DESIGN.md # design details, interfaces, data flow, tradeoffs EVALUATION.md # environment, baselines, full results, analysis epd-vllm-demo/ # EPD end-to-end reproduction package mooncake/epd_vllm_v1_proxy_server.py # EPD control-plane proxy configs/ # environment templates scripts/ # preflight, correctness matrix, benchmark scripts tests/ # local tests for proxy and helper scripts workloads/ # OpenAI-compatible multimodal payloads results/ # archived EPD results and metrics object-type-eviction-bench/ # Store-level type-aware eviction benchmark benchmarks/ # continuous multimodal session workload scripts/ # single-case smoke and full policy suite runners results/ # archived representative policy results patches/ vllm/ # vLLM MooncakeStoreECConnector patches mooncake/ # Mooncake Store policy and architecture patches ``` Key artifacts: - `patches/vllm/patches/vllm-epd-hidden-ec-connector-b4482f0a1-full-feature.patch` archives the vLLM-side `MooncakeStoreECConnector`. - `epd-vllm-demo/mooncake/epd_vllm_v1_proxy_server.py` archives the EPD demo proxy used for request splitting and transfer metadata forwarding. - `patches/mooncake/0001-store-object-type-aware-eviction-policy.patch` archives the Store object type, type-aware scoring, per-type budget, and `eviction_grace` changes. - `patches/mooncake/0002-store-hierarchical-eviction-policy-abstraction.patch` archives the policy abstraction follow-up. ## Design ### 1. EPD Hidden State Data Path `MooncakeStoreECConnector` follows the vLLM EC Connector split between Scheduler-side planning and Worker-side data movement. The Scheduler side performs lookup, load/save planning, and connector metadata construction. The Worker side performs the actual Mooncake Store reads and writes. The EPD proxy is only a control-plane component; it forwards request metadata and does not carry tensor payloads. ```text client -> EPD proxy -> encoder: MooncakeStoreECConnector, ec_producer -> prefill: MooncakeStoreECConnector + MooncakeStoreConnector, ec_consumer + kv_producer -> decode: MooncakeStoreConnector, kv_consumer ``` Hidden State uses a separate key namespace that includes model, vision config, parallel layout, storage layout, and tensor format information. This prevents incorrect reuse across incompatible configurations. The stored object format is a fixed metadata header plus a contiguous tensor payload, matching Mooncake Store's buffer-registration and range-read capabilities. ### 2. Failure Model And Compatibility The submission explicitly separates cache-fill failure, cache-load failure, and Store API compatibility: - If an Encoder-side Hidden State save fails, the object is not reported as finished. Later requests naturally fall back to the miss / recompute path. - Prefill-side Hidden State loading is a pre-execution dependency. A load failure does not populate local `encoder_cache`; the error is surfaced rather than allowing an invalid tensor to continue through inference. - The Store client wrapper handles Mooncake Python binding differences and prefers the buffer-registration path when available. - The EPD proxy only forwards control metadata. Tensor payloads stay on the Mooncake Store data path. - Store type-aware eviction keeps default behavior compatible when no policy is configured. ### 3. Store Type-Aware Eviction The Store patch adds `ObjectDataType.HIDDEN_STATE` and extends `BatchEvict` candidate selection with object type information. The policy knobs are: - `reuse_scale`: adjusts cross-type victim ranking to reflect recomputation value. - `eviction_grace`: adds an extra protection window for object types whose reuse interval is longer. - `budget_ratio`: provides a per-type capacity guardrail so one type does not unboundedly squeeze out another. The ranking uses type-adjusted age: ```text adjusted_age = max(0, rank_reference_time - lease_timeout - eviction_grace) * soft_pin_weight / reuse_scale ``` `budget_ratio` is a logical budget rather than a physical partition. The Store keeps a shared memory pool, while over-budget types contribute victims first under pressure. ### 4. Hierarchical Eviction Abstraction The follow-up patch separates policy planning from `master_service` metadata mutation. `MasterService` still scans candidates, revalidates metadata, and performs the actual deletion. The policy module only works on candidate references and scope indexes. The abstraction includes: - `Assign`: assign eviction responsibility by type, tenant, or future priority scopes. - `Evict`: select victim candidates within a scope. - `Fallback`: compensate from the parent scope when a child scope cannot meet its target. The current patch validates a type-level single-layer abstraction. A full tenant -> type -> priority recursive executor is future work. ## Results Detailed environment settings and raw result summaries are in `mooncake-track4/EVALUATION.md`. ### 1. EPD Correctness The EPD experiment was run on a single-node, three-GPU TCP-only setup with `Qwen/Qwen2.5-VL-7B-Instruct`. The result is correctness and reproducibility evidence, not an RDMA / SHM performance upper bound. ```text correctness matrix: 6/6 passed required hidden evidence cases: 2/2 passed encoder_hidden_store_put: 2 prefill_hidden_store_scheduler_hit: 2 prefill_hidden_store_get: 2 prefill_hidden_store_scheduler_miss: 0 output comparison cases: 6/6 passed exact output match cases: 6/6 ``` ### 2. TCP-Only Transfer Baseline ```text successful requests: 6/6 request throughput: 0.667639 req/s TTFT p50: 835.323 ms TTFT p95: 2666.689 ms latency p50: 1180.717 ms latency p95: 2950.505 ms ``` | Path | Bytes | Time | Bandwidth | | --- | ---: | ---: | ---: | | Hidden E->Store save | 32,062,464 | 44.033 ms | 694.414 MiB/s | | Hidden Store->P load | 32,062,464 | 33.520 ms | 912.206 MiB/s | | KV P->Store save_put | 206,438,400 | 623.245 ms | 315.887 MiB/s | ### 3. Type-Aware Eviction Ablation The Store benchmark simulates 600 continuous multimodal sessions. KV Cache accounts for about 90.88% of unique bytes, while Hidden State accounts for about 9.12%. KV is reused more frequently; Hidden State is reused more sparsely but has higher recomputation cost. | Policy | Hidden hit | KV hit | Saving Time Rate | Main takeaway | | --- | ---: | ---: | ---: | --- | | baseline default | 33.94% | 96.65% | 87.80% | original global policy | | hidden grace only | 67.89% | 95.84% | 92.42% | Hidden +33.95 pp, KV -0.81 pp | | hidden reuse scale only | 84.71% | 94.98% | 94.38% | Hidden +50.77 pp, KV -1.67 pp | | balanced type-aware policy | 75.54% | 95.74% | 94.26% | Hidden +41.60 pp, KV -0.91 pp | Balanced configuration: ```text HIDDEN_STATE reuse_scale = 8.0 HIDDEN_STATE eviction_grace = 10000 KVCACHE budget_ratio = 0.80 HIDDEN_STATE budget_ratio = 0.12 ``` ### 4. Hierarchical Eviction Validation The hierarchical eviction patch is based on official Mooncake `c9896684`. It adds a standalone policy module and focused unit test coverage for type-level `Assign / Evict / Fallback` planning, candidate-index isolation from metadata mutation, and compatibility with the default eviction semantics. `eviction_policy_test` passes; full commands are in `mooncake-track4/patches/mooncake/README.md`. ## How To Test The test entrypoints are grouped from minimal smoke to full end-to-end reproduction. Full build and environment details are in the subdirectory README files. From the GitLink repository root: ```bash export SUBMISSION_ROOT="$PWD" export TRACK4_ROOT="$SUBMISSION_ROOT/mooncake-track4" ``` ### 1. Minimal Smoke: Benchmark Harness No GPU or EPD service is required. ```bash cd "$TRACK4_ROOT/object-type-eviction-bench" python3 -m unittest benchmarks/test_continuous_session_policy_bench.py -v ``` ### 2. Store Policy Module Apply `patches/mooncake/0001-store-object-type-aware-eviction-policy.patch` to official Mooncake `f4f7fd4a03539974fc39acbd045b1ee4ba0ed967`, then build `mooncake_master` and the Python binding. See `object-type-eviction-bench/README.md` for the exact build commands. Single policy case: ```bash cd "$TRACK4_ROOT/object-type-eviction-bench" MOONCAKE_ROOT=/tmp/mooncake-object-type-runtime \ OUT_DIR=/tmp/mooncake_single_policy_case \ bash scripts/run_single_policy_case.sh ``` Full policy suite: ```bash MOONCAKE_ROOT=/tmp/mooncake-object-type-runtime \ OUT_DIR=/tmp/mooncake_scale_0006_policy_suite \ bash scripts/run_pr2_pr4_scale_0006_suite.sh ``` ### 3. Hierarchical Eviction Patch ```bash git clone https://github.com/kvcache-ai/Mooncake.git ../mooncake-hier-eviction cd ../mooncake-hier-eviction git checkout c9896684 git config user.name reviewer git config user.email reviewer@example.com git am "$TRACK4_ROOT/patches/mooncake/0002-store-hierarchical-eviction-policy-abstraction.patch" cmake -S . -B build-hier-tests \ -DWITH_STORE=ON -DWITH_TE=ON -DWITH_EP=OFF \ -DWITH_STORE_GO=OFF -DWITH_STORE_RUST=OFF -DWITH_P2P_STORE=OFF \ -DBUILD_UNIT_TESTS=ON -DBUILD_EXAMPLES=OFF -DBUILD_BENCHMARK=OFF cmake --build build-hier-tests --target eviction_policy_test -j2 cd build-hier-tests ctest -R eviction_policy_test --output-on-failure ``` ### 4. EPD End-To-End This requires a three-GPU environment and a patched vLLM runtime. If the server does not already provide a patched vLLM wheel, apply the vLLM patch documented in `patches/vllm/README.md`. ```bash cd "$TRACK4_ROOT/epd-vllm-demo" cp configs/server_env.example.sh configs/server_env.local.sh vim configs/server_env.local.sh source configs/server_env.local.sh bash "$EVAL_ROOT/scripts/check_server_env.sh" FULL_EPD_MATRIX=1 bash "$EVAL_ROOT/scripts/run_full_epd_mooncake_pd_smoke.sh" ``` ## Notes And Limitations - The EPD experiment is TCP-only correctness and reproducibility evidence. It is not an RDMA / SHM performance upper bound. - The Store eviction benchmark is a Store-level isolated experiment. It measures cache retention behavior, not end-to-end model latency. - The object-type eviction patch must be applied on the documented baseline `f4f7fd4a03539974fc39acbd045b1ee4ba0ed967`; current official `main` should not be assumed to accept it directly. - The hierarchical eviction abstraction currently validates a type-level single-layer policy boundary. A full multi-layer executor is future work. - The EPD proxy is a demo orchestration component, not part of the upstream vLLM connector itself. - The current EPD validation focuses on a single-node, three-GPU, single-rank Hidden State object path. Multi-node RDMA, sharded Hidden State, and richer cross-request cache policies are future work.
kancel added 1 commit 2026-07-09 13:53:05 +08:00
Auto Label PRs / triage (pull_request) Failing after 1m19s Details
Build & Test (Linux) / build (3.10) (pull_request) Has been cancelled Details
Build & Test (Linux) / build (3.12) (pull_request) Has been cancelled Details
Build & Test (Linux) / build-musa (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-22.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-24.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-22.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-24.04) (pull_request) Has been cancelled Details
Build & Test (Linux) / build-flags (3.10) (pull_request) Has been cancelled Details
Build & Test (Linux) / build-flags (3.12) (pull_request) Has been cancelled Details
Build & Test (Linux) / Build Docker Image (pull_request) Has been cancelled Details
Build & Test (Linux) / Spell Check with Typos (pull_request) Has been cancelled Details
Build & Test (Linux) / Check code format (pull_request) Has been cancelled Details
Build & Test (Linux) / check-paths (pull_request) Has been cancelled Details
Build & Test (Linux) / build-wheel-cu13 (pull_request) Has been cancelled Details
Build & Test (Linux) / ascend-test (pull_request) Has been cancelled Details
Build & Test (Linux) / integration-test (pull_request) Has been cancelled Details
Build & Test (Linux) / CI Gate (pull_request) Has been cancelled Details
1b1605c351 Add CCF Track 4 submission materials
Some checks failed
Auto Label PRs / triage (pull_request) Failing after 1m19s
Details
Build & Test (Linux) / build (3.10) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build (3.12) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-musa (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-22.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.10, ubuntu-24.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-22.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / test-wheel-ubuntu (3.12, ubuntu-24.04) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-flags (3.10) (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-flags (3.12) (pull_request) Has been cancelled
Details
Build & Test (Linux) / Build Docker Image (pull_request) Has been cancelled
Details
Build & Test (Linux) / Spell Check with Typos (pull_request) Has been cancelled
Details
Build & Test (Linux) / Check code format (pull_request) Has been cancelled
Details
Build & Test (Linux) / check-paths (pull_request) Has been cancelled
Details
Build & Test (Linux) / build-wheel-cu13 (pull_request) Has been cancelled
Details
Build & Test (Linux) / ascend-test (pull_request) Has been cancelled
Details
Build & Test (Linux) / integration-test (pull_request) Has been cancelled
Details
Build & Test (Linux) / CI Gate (pull_request) Has been cancelled
Details
This pull request can be merged automatically.
You are not authorized to merge this pull request.
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b kancel-ccf-track4-submission main
git pull ccf-track4-submission

Step 2:

Merge the changes and update on Gitea.
git checkout main
git merge --no-ff kancel-ccf-track4-submission
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: mooncake-track/Mooncake#14
No description provided.
Delete Branch "kancel/Mooncake:ccf-track4-submission"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?