mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,944 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

183 Commits

Author SHA1 Message Date
Yan Ru Pei bddaaa2658
feat(kv-router): pluggable scheduling policy for router queue [DYN-2454] (#7260) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-03-13 04:16:49 +00:00
Hongkuan Zhou cd4773fb31
feat: ForwardPassMetrics dynamo event plane integration (#7250) 
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
 2026-03-12 20:58:01 +00:00
Michael Feil fca0a801dc
feat: follow up of optinal, batching-of-client-side-events (#6741) 
Signed-off-by: michaelfeil <63565275+michaelfeil@users.noreply.github.com>
Signed-off-by: Michael Feil <michaelfeil@users.noreply.github.com>
Signed-off-by: Michael Feil <63565275+michaelfeil@users.noreply.github.com>
 2026-03-11 23:30:51 +00:00
Yan Ru Pei 2e29620df1
chore(kv-router): move benches to lib/bench to break circular dep [OPS-3752] (#7013) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-03-06 23:24:03 +00:00
Graham King abc02c689f
fix: llm/mocker: Remove the llm -> mocker crate dependency, move config (#6998) 
Signed-off-by: Graham King <grahamk@nvidia.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-03-06 09:41:39 -08:00
Yan Ru Pei f934744fb9
fix: restore --enforce-disagg to reject requests before prefill router activates (#6957) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-03-06 04:39:28 +00:00
GuanLuo 90d7463721
fix: properly setup and register vLLM worker for external / hybrid load balancing. Update launch script (#6695) 
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
 2026-03-03 14:00:52 -08:00
Yan Ru Pei 8055b7dd2c
fix(kv-router): plumb expected_osl through scheduler queue path (#6812) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-03-03 10:57:42 -08:00
Yan Ru Pei 07742cc27f
feat: move standalone KV indexer into kv-router crate with HTTP integration tests (#6569) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 10:34:21 -08:00
Michael Feil 713c96d26d
feat: dynamic batching of client side events (#6733) 
Signed-off-by: michaelfeil <63565275+michaelfeil@users.noreply.github.com>
 2026-02-28 18:27:57 +00:00
Yan Ru Pei 55c0a769b4
feat: default router_event_threads to 4 (#6672) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-27 22:14:47 +00:00
ishandhanani f1d5c95a4e
feat: wire nvext.cache_control TTL-based pinning through Dynamo router (#6213) 
Co-authored-by: Claude <noreply@anthropic.com>
 2026-02-27 01:03:27 +00:00
Yan Ru Pei ebaf048d16
chore: plumb allowed_worker_ids through RoutingHints (#6580) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-25 12:12:53 -08:00
atchernych c916cd42ff
feat: Support epp's "pods" interface in Dynamo fixes [DEP-424] (#6302) 
Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
 2026-02-25 01:54:38 +00:00
Biswa Panda 75fea78797
feat: make KV cache events and routing LoRA-aware (#6517) 2026-02-24 18:35:08 +00:00
Yan Ru Pei c9ff623538
chore: replace --enforce-disagg with --decode-fallback, default to enforcing disagg (#6515) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-24 09:24:11 -08:00
Tzu-Ling Kan 80cac7c14b
feat: Remove Component from public (#6403) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-23 22:58:38 +00:00
Thomas Montfort cb55766c47
feat(runtime): add hierarchical Model/WorkerSet architecture for multi-namespace support (#6054) 
Signed-off-by: tmontfort <tmontfort@nvidia.com>
 2026-02-23 22:44:38 +00:00
Yan Ru Pei 038b50d2b6
feat: add standalone KV indexer with query endpoint [DYN-2164] (#6446) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-23 12:12:25 -08:00
Yan Ru Pei fc229004b5
chore: Remove ZmqKvEventListener binding and rework standalone TRT-LLM example to use native Python ZMQ (#6164) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-19 04:15:13 +00:00
zhongdaor-nv 815b129126
feat: mm aware routing for vllm (#6235) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
 2026-02-13 21:16:06 -08:00
Yan Ru Pei 14eceb43df
chore: rename KvPushRouter to KvRouter in python + more bindings removal (#6238) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-12 23:44:52 -08:00
Yan Ru Pei bc514fbee6
feat: router priority queue (#6010) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2026-02-13 02:43:17 +00:00
Karen Chung cd6984b951
feat: use RNG when dp routing targets are tied; override no-assume-kv-reuse for decode requests (#6253) 
Signed-off-by: Karen Chung <karenc@nvidia.com>
 2026-02-12 15:59:37 -08:00
Yan Ru Pei 937398cf61
feat: Flash Indexer (#5785) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: jthomson04 <jothomson@nvidia.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: Janelle Cai <jcai18@mit.edu>
Co-authored-by: jthomson04 <jwillthomson19@gmail.com>
Co-authored-by: Janelle Cai <jcai18@mit.edu>
 2026-02-11 22:26:56 -08:00
Yan Ru Pei 6a728d1044
chore: remove kv indexers bindings (#6159) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-11 20:23:48 +00:00
Yan Ru Pei f46720c996
feat: add router-level Prometheus metrics and centralize request tracking (#6146) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-11 07:36:08 +00:00
Graham King 4f99451bb0
feat(frontend): Use vllm for pre and post processing (#5544) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-02-10 22:39:11 +00:00
Yan Ru Pei 6783bdcaa9
chore: enable local indexers by default, and use normal event plane by default (not jetstream) (#5941) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-08 20:48:02 +00:00
Jacky 1ffa489ea1
refactor: Move --migration-limit flag from backend to frontend (#5918) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2026-02-06 20:50:55 +00:00
Yan Ru Pei 3e41702211
chore: remove and unify bindings in kv.rs (#6016) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-06 18:44:46 +00:00
Keiven C cb7ebdd7e8
chore: remove unused NIM specific code (part 2) (#5893) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-02-05 12:21:10 -08:00
Yan Ru Pei 91fb78cddf
chore: make mocker it's own crate (#5958) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-05 19:35:42 +00:00
Hongkuan Zhou 0c0336e6ea
feat: Add per-worker Prometheus metrics for router load monitoring (#5842) 
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
 2026-02-04 11:51:43 -08:00
Yan Ru Pei a337113aa9
feat: prefill tokens threshold based on max num batched tokens frac (#5867) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-03 03:08:25 +00:00
Yan Ru Pei 902eabd9ec
feat: per dp rank gap detection (#5873) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-03 03:07:57 +00:00
Biswa Panda 373e76c1ae
feat(lora): Add lora_name tracking to scheduling and sequence management (#5875) 2026-02-03 00:59:09 +00:00
ishandhanani c5f5ab6056
fix: enable DP attention KV events for multi-node deployments (#5589) 
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: huitianbai <huitianbai@gmail.com>
Co-authored-by: Huitianqi Bai <huitianbai@users.noreply.github.com>
 2026-02-02 07:20:29 +00:00
Biswa Panda 1174c819cb
feat: remove deprecated events api (#5778) 2026-01-29 05:48:16 +00:00
Yan Ru Pei fc92fc187b
chore: clean ups in kv-router (#5771) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-29 03:20:04 +00:00
Yan Ru Pei feb6d272ac
chore: nuke ForwardPassMetrics (#5531) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-23 16:26:01 +00:00
Graham King ca31b3aa6b
chore: Remove binding OAIChatPreprocessor (#5516) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-22 08:43:02 +08:00
Yan Ru Pei 050906b593
feat: track output tokens / blocks in the Router (optional) (#5452) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-18 09:30:25 +08:00
Graham King a6e7348400
chore(bindings): Remove 'Backend' binding (#5458) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-16 12:29:50 -05:00
Graham King c5458b9954
chore(router): Remove deprecated best_worker_id and metrics_labels (#5412) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-14 16:20:42 +00:00
Yan Ru Pei 7fdc742e17
feat: allow router not assuming decode kv reuse (#5350) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-13 07:33:47 +08:00
Tushar Sharma cf433e6825
chore: update all copyright headers in repo to 2026 (#5130) 
Signed-off-by: Tushar Sharma <tusharma@nvidia.com>
 2026-01-02 22:08:23 +00:00
Yan Ru Pei 0b33c1dfa0
fix: sglang disagg routing fixes and optimizations [DYN-1692] (#5106) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
Co-authored-by: Sean SH Choi <sechoi@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2025-12-31 10:55:44 -08:00
Yan Ru Pei f4245c9929
chore: clean up PreprocessedRequest (#5040) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-19 23:15:59 +00:00
milesial d2faf0e6f5
feat: Runtime media decoder config (#5011) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-12-19 05:34:09 +00:00