mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,250 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

155 Commits

Author SHA1 Message Date
Yan Ru Pei 6783bdcaa9
chore: enable local indexers by default, and use normal event plane by default (not jetstream) (#5941) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-08 20:48:02 +00:00
Jacky 1ffa489ea1
refactor: Move --migration-limit flag from backend to frontend (#5918) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2026-02-06 20:50:55 +00:00
Yan Ru Pei 3e41702211
chore: remove and unify bindings in kv.rs (#6016) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-06 18:44:46 +00:00
Keiven C cb7ebdd7e8
chore: remove unused NIM specific code (part 2) (#5893) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-02-05 12:21:10 -08:00
Yan Ru Pei 91fb78cddf
chore: make mocker it's own crate (#5958) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-05 19:35:42 +00:00
Hongkuan Zhou 0c0336e6ea
feat: Add per-worker Prometheus metrics for router load monitoring (#5842) 
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
 2026-02-04 11:51:43 -08:00
Yan Ru Pei a337113aa9
feat: prefill tokens threshold based on max num batched tokens frac (#5867) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-03 03:08:25 +00:00
Yan Ru Pei 902eabd9ec
feat: per dp rank gap detection (#5873) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-03 03:07:57 +00:00
Biswa Panda 373e76c1ae
feat(lora): Add lora_name tracking to scheduling and sequence management (#5875) 2026-02-03 00:59:09 +00:00
ishandhanani c5f5ab6056
fix: enable DP attention KV events for multi-node deployments (#5589) 
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: huitianbai <huitianbai@gmail.com>
Co-authored-by: Huitianqi Bai <huitianbai@users.noreply.github.com>
 2026-02-02 07:20:29 +00:00
Biswa Panda 1174c819cb
feat: remove deprecated events api (#5778) 2026-01-29 05:48:16 +00:00
Yan Ru Pei fc92fc187b
chore: clean ups in kv-router (#5771) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-29 03:20:04 +00:00
Yan Ru Pei feb6d272ac
chore: nuke ForwardPassMetrics (#5531) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-23 16:26:01 +00:00
Graham King ca31b3aa6b
chore: Remove binding OAIChatPreprocessor (#5516) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-22 08:43:02 +08:00
Yan Ru Pei 050906b593
feat: track output tokens / blocks in the Router (optional) (#5452) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-18 09:30:25 +08:00
Graham King a6e7348400
chore(bindings): Remove 'Backend' binding (#5458) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-16 12:29:50 -05:00
Graham King c5458b9954
chore(router): Remove deprecated best_worker_id and metrics_labels (#5412) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-14 16:20:42 +00:00
Yan Ru Pei 7fdc742e17
feat: allow router not assuming decode kv reuse (#5350) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-13 07:33:47 +08:00
Tushar Sharma cf433e6825
chore: update all copyright headers in repo to 2026 (#5130) 
Signed-off-by: Tushar Sharma <tusharma@nvidia.com>
 2026-01-02 22:08:23 +00:00
Yan Ru Pei 0b33c1dfa0
fix: sglang disagg routing fixes and optimizations [DYN-1692] (#5106) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
Co-authored-by: Sean SH Choi <sechoi@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2025-12-31 10:55:44 -08:00
Yan Ru Pei f4245c9929
chore: clean up PreprocessedRequest (#5040) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-19 23:15:59 +00:00
milesial d2faf0e6f5
feat: Runtime media decoder config (#5011) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-12-19 05:34:09 +00:00
zhongdaor-nv 481dc6363b
feat: add multimodal support to KV router with standalone trtllm example (#4577) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
 2025-12-18 21:01:45 -08:00
Keiven C c22280ccf4
refactor: frontend/grpc tests to use dynamic ports (#4992) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-12-18 16:21:28 -08:00
Yan Ru Pei 85b5c808ac
feat: unify per-request metric tracking (#5004) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-18 23:59:25 +00:00
Graham King da0f2fb84a
feat(frontend): First part of Python request handling (#4999) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-12-17 20:03:39 -05:00
Sean SH Choi 1c39e4835c
feat: add frontend based prefill request routing for sglang (#4635) 
Signed-off-by: Sean Choi <sechoi@nvidia.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2025-12-17 09:51:46 -08:00
milesial 74fcd4a9b9
feat: video decoder in the frontend (#4719) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-12-16 14:28:06 -08:00
Yan Ru Pei c2a29f8045
chore: enable local indexers for sglang and trtllm (#4932) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-12 11:41:37 -08:00
Karen Chung 332499458b
feat: worker-local KvIndexer in KvEventPublisher (#4519) 
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
 2025-12-11 23:18:34 +00:00
Yan Ru Pei 10b01b4590
feat: early rejection based on active prefill tokens (#4837) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-11 23:16:02 +00:00
Biswa Panda 71f94eda23
feat: lora - centralize lora cache key, restructure folders, s3 resiliency (#4644) 2025-12-02 08:57:25 +00:00
Biswa Panda 6a0e67ed78
feat: add LoRA common APIs and implementation for lora management (#4464) 2025-11-26 20:29:25 +00:00
Graham King 0fc5273c11
refactor(llm): Rename EngineConfig::Static to InProcess (#4585) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-25 12:22:09 -05:00
Yan Ru Pei c61e0dd3cc
chore: merge KvIndexer and ApproxKvIndexer (#4500) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-21 09:09:11 -08:00
Kris Hung 2f18b23ee7
chore: Use general engine source for KVBM KV events (#4515) 2025-11-20 14:47:48 -08:00
Waël Boukhobza 2cf427ed8f
fix(router): change prune channel to be mpsc instead of watch, increase max size (#4351) 
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
 2025-11-14 18:19:20 -05:00
Yan Ru Pei ce83398366
chore: better error handling in prefill router (#4286) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-13 22:29:26 +00:00
Yan Ru Pei 8379b0cd10
feat: kv router should route to available instances (#4225) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-13 02:34:58 +00:00
Waël Boukhobza 3ea22fcf39
feat(router): max tree size based pruning (#4057) 
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
 2025-11-12 08:33:49 -08:00
Graham King e1af3af6ee
chore: Remove static mode (#4235) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-11 19:25:12 +00:00
milesial 14af074ecd
feat: Media decoder and fetcher options in the MDC (#4094) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-11-08 02:17:39 +00:00
Graham King 794c0a4431
feat(keyvalue): Filesystem backed KeyValueStore (#4138) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-07 21:11:33 +00:00
Yan Ru Pei 427ca9aba4
chore: no need for explicit Python::with_gil (#4075) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-05 05:41:49 +00:00
Michael Feil defe5de74b
feat: release gil for publisher (#3747) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
 2025-11-05 02:17:06 +00:00
Graham King dadf0e2247
chore: Remove old DisaggregatedRouter, making etcd presence optional (#4011) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-03 10:10:13 -05:00
Richard Huo 6afa679c5d
chore: KVBM pip wheel (#3826) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>
 2025-10-31 14:19:30 -07:00
Kris Hung 1898601025
feat: Add KV event consolidator for KVBM (vllm) and router integration (#3725) 
Signed-off-by: krishung5 <krish@nvidia.com>
 2025-10-30 17:11:42 +00:00
Yan Ru Pei cc4c3516bb
feat: mocker disagg (#3833) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-28 20:36:02 +00:00
Graham King c78b590134
chore(runtime): Do not expose etcd lease ID (#3915) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-28 16:20:25 +00:00