mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,557 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

1109 Commits

Author SHA1 Message Date
Alec e10787894d
fix: add periodic vLLM engine stats logging and fix log routing (#6566) 
Signed-off-by: alec-flowers <aflowers@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 23:45:34 +00:00
Schwinn Saereesitthipitak a6d970e961
feat: refactor GMS client memory manager with tiered API (#6549) 2026-02-25 23:24:17 +00:00
daiyaanarfeen 651ef5b506
feat: throughput-metrics-source for SLA planner + GlobalPlanner disagg scaling (#6500) 
Signed-off-by: Daiyaan <darfeen@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-02-25 22:58:24 +00:00
Nikita 21fce9ba0d
feat: Tiktoken support (#6460) 
Signed-off-by: Nikita Sukharev <kaonael@gmail.com>
 2026-02-25 12:46:47 -08:00
Yan Ru Pei ebaf048d16
chore: plumb allowed_worker_ids through RoutingHints (#6580) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-25 12:12:53 -08:00
Nikita d1bd210f1b
feat: Kimi K2/K2.5 tool and reasoning parsers (#6407) 
Signed-off-by: Nikita Sukharev <kaonael@gmail.com>
 2026-02-25 18:19:14 +00:00
Keiven C ff06b17e7f
fix: guarantee RouterRequestMetrics availability & documentation updates (#6558) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-25 10:18:01 -08:00
dagil-nvidia e0373bd725
docs: alphabetize backends (SGLang, TensorRT-LLM, vLLM) (#6537) 
Signed-off-by: Dan Gil <dagil@nvidia.com>
Signed-off-by: dagil-nvidia <dagil@nvidia.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-25 10:07:59 -06:00
Yongming Ding 5d9586234e
feat(mocker): add KV cache transfer latency for disagg serving (#6504) 
Signed-off-by: Yongming Ding <yongmingd@nvidia.com>
 2026-02-24 23:53:28 -08:00
Ran Rubin e5f97343fd
ci: Accelerate dynamo frontend image build (#6479) 2026-02-25 08:56:22 +02:00
John Pohl 44d43d0cf6
feat: MJPEG video streaming via /v1/videos/stream (experimental) (#6487) 
Signed-off-by: Ryan McCormick <mccormick.codes@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2026-02-25 03:04:39 +00:00
Yan Ru Pei 65dc451db9
feat: add ZMQ KV event publishing to mocker [DYN-2221] (#6528) 
Signed-off-by: Ru Pei <rupei@nvidia.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-24 18:57:28 -08:00
Ryan McCormick 4bd4c1657b
feat: Log discovery backend type (#6559) 2026-02-25 02:25:44 +00:00
atchernych c916cd42ff
feat: Support epp's "pods" interface in Dynamo fixes [DEP-424] (#6302) 
Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
 2026-02-25 01:54:38 +00:00
ishandhanani 6642e23e0f
feat: sglang to 0.5.9 + updated docs (#6518) 
Co-authored-by: baihuitian <baihuitian.bht@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-24 21:48:25 +00:00
Keiven C 1df620b4d0
feat: router metrics with dynamo_router_* {worker_id=...}. Update docs (#6227) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-24 21:34:14 +00:00
Yan Ru Pei efa89448f5
chore: de-async scheduler read paths and unblock decode output tracking (#6510) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-24 21:00:36 +00:00
Biswa Panda 6fab12be19
test: add tests for lora aware kv cache events and routing (#6523) 2026-02-24 12:38:06 -08:00
Yan Ru Pei bc00ef3879
chore: split JetStream subscriber into dedicated module and deprecate durable_kv_events [DYN-2203] (#6477) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-24 11:30:38 -08:00
Biswa Panda 75fea78797
feat: make KV cache events and routing LoRA-aware (#6517) 2026-02-24 18:35:08 +00:00
Vladislav Nosivskoy 432fae67b1
feat: add error_type label to reqs metric for fine-grained error classification (#5568) 
Signed-off-by: Vladislav Nosivskoy <vladnosiv@gmail.com>
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-24 10:31:47 -08:00
Yuewei Na 6654a57de6
fix: prevent device_blocks double-add in TRT-LLM KVBM connector (#6406) 
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Co-authored-by: Yuewei Na <nv-yna@users.noreply.github.com>
 2026-02-24 10:11:34 -08:00
Yan Ru Pei c9ff623538
chore: replace --enforce-disagg with --decode-fallback, default to enforcing disagg (#6515) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-24 09:24:11 -08:00
Jacky c8276cd28b
feat: Standardized Dynamo Error Type (#6303) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2026-02-23 20:23:58 -08:00
Yan Ru Pei 0cb1d73383
fix: fold scheduling into queue so backpressure actually works (#6470) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-23 18:33:03 -08:00
MatejKosec a9e0696023
fix(api): preserve interleaved reasoning order for KV cache correctness (#6442) 
Signed-off-by: Matej Kosec <mkosec@nvidia.com>
 2026-02-24 02:20:54 +00:00
Yan Ru Pei ddeee88b05
feat: expose predicted KV hit rate as Prometheus histogram (#6507) 
Signed-off-by: Pea Brane <peabrane@peabrane.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-24 00:54:14 +00:00
Yan Ru Pei 32622b1c74
docs: add multiturn_bench README with speculative prefill demo (#6502) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-23 15:52:07 -08:00
Tzu-Ling Kan 80cac7c14b
feat: Remove Component from public (#6403) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-23 22:58:38 +00:00
Thomas Montfort cb55766c47
feat(runtime): add hierarchical Model/WorkerSet architecture for multi-namespace support (#6054) 
Signed-off-by: tmontfort <tmontfort@nvidia.com>
 2026-02-23 22:44:38 +00:00
Anant Sharma f391553e77
fix: remove unused deps for kvbm bindings (#6496) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2026-02-23 16:59:13 -05:00
Yan Ru Pei 038b50d2b6
feat: add standalone KV indexer with query endpoint [DYN-2164] (#6446) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-23 12:12:25 -08:00
Yan Ru Pei e66f326789
fix: derive kv-router discovery name from target worker component (#6475) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-23 10:28:18 -08:00
Ryan Olson 61c6780469
feat: add kvbm-kernels crate and upgrade cudarc to 0.19 (#6309) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
 2026-02-23 16:15:28 +00:00
Alec d3ee09d7ba
fix: gracefully skip NIXL UCX tests when backend unavailable (#6484) 
Signed-off-by: alec-flowers <aflowers@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-22 13:44:47 -08:00
Tzu-Ling Kan 42d6980546
feat: Remove public uses of CancellationToken (#6405) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-20 21:01:13 -08:00
Ryan Olson 7a6283e927
chore: cargo machete - remove unused dependencies (#6453) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
 2026-02-20 19:11:52 +00:00
Yongming Ding 9b2e8f3dd5
refactor(mocker): modularize into common/scheduler/kv_manager/cache (#6440) 
Signed-off-by: Yongming Ding <yongmingd@nvidia.com>
 2026-02-20 10:47:59 -08:00
MatejKosec d82b0050f9
feat: interleaved thinking support in reasoning parser (#6422) 
Signed-off-by: Matej Kosec <mkosec@nvidia.com>
 2026-02-20 10:33:02 -08:00
Ayush Agarwal 6dd3ce2ea0
feat: vllm omni text to video generation pipeline (#6104) 
Signed-off-by: ayushag <ayushag@nvidia.com>
 2026-02-20 04:23:44 +00:00
zhongdaor-nv 23de4e86ae
feat: e2e mm aware kv cache routing support for trtllm backend (#5480) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
Signed-off-by: Zhongdao Ren <zhongdaor@zhongdaor-mlt.client.nvidia.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Zhongdao Ren <zhongdaor@zhongdaor-mlt.client.nvidia.com>
 2026-02-19 20:52:00 -07:00
Tzu-Ling Kan 0ce3461a9e
feat: Add runtime.endpoint() method to eliminate namespace chaining (#6386) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-19 16:02:37 -05:00
Yan Ru Pei 1549c338d5
chore: derive router_id and cancellation_token from component instead of plumbing (#6419) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-19 19:48:21 +00:00
Ziqi Fan 832bf80a15
feat: add NVTX annotation to KVBM (#6334) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2026-02-19 17:23:33 +00:00
Yan Ru Pei fc229004b5
chore: Remove ZmqKvEventListener binding and rework standalone TRT-LLM example to use native Python ZMQ (#6164) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-19 04:15:13 +00:00
Yan Ru Pei c7986b3540
refactor: move per-item stream tracking into RequestGuard (#6355) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-18 19:26:30 -08:00
Ayush Agarwal 33d71102b1
feat: glm47 tool parser (#5897) 
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: Matej Kosec <mkosec@nvidia.com>
Signed-off-by: Marko Kosec <mkosec@nvidia.com>
Co-authored-by: Matej Kosec <mkosec@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2026-02-18 23:17:23 +00:00
Tzu-Ling Kan bc8f1170ce
feat: Move ModelDeploymentCard to _internal (#6378) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-18 23:12:18 +00:00
Yan Ru Pei cf51a0c4fd
chore: gate plotters dependency behind bench feature (#6380) 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-18 15:27:19 -05:00
Anant Sharma aa16ccf545
fix: remove default-members in workspace (#6279) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2026-02-18 12:39:17 -05:00