mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,557 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

704 Commits

Author SHA1 Message Date
Nikita 21fce9ba0d
feat: Tiktoken support (#6460) 
Signed-off-by: Nikita Sukharev <kaonael@gmail.com>
 2026-02-25 12:46:47 -08:00
Yan Ru Pei ebaf048d16
chore: plumb allowed_worker_ids through RoutingHints (#6580) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-25 12:12:53 -08:00
Nikita d1bd210f1b
feat: Kimi K2/K2.5 tool and reasoning parsers (#6407) 
Signed-off-by: Nikita Sukharev <kaonael@gmail.com>
 2026-02-25 18:19:14 +00:00
Keiven C ff06b17e7f
fix: guarantee RouterRequestMetrics availability & documentation updates (#6558) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-25 10:18:01 -08:00
Yongming Ding 5d9586234e
feat(mocker): add KV cache transfer latency for disagg serving (#6504) 
Signed-off-by: Yongming Ding <yongmingd@nvidia.com>
 2026-02-24 23:53:28 -08:00
John Pohl 44d43d0cf6
feat: MJPEG video streaming via /v1/videos/stream (experimental) (#6487) 
Signed-off-by: Ryan McCormick <mccormick.codes@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2026-02-25 03:04:39 +00:00
Yan Ru Pei 65dc451db9
feat: add ZMQ KV event publishing to mocker [DYN-2221] (#6528) 
Signed-off-by: Ru Pei <rupei@nvidia.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-24 18:57:28 -08:00
atchernych c916cd42ff
feat: Support epp's "pods" interface in Dynamo fixes [DEP-424] (#6302) 
Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
 2026-02-25 01:54:38 +00:00
ishandhanani 6642e23e0f
feat: sglang to 0.5.9 + updated docs (#6518) 
Co-authored-by: baihuitian <baihuitian.bht@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-24 21:48:25 +00:00
Keiven C 1df620b4d0
feat: router metrics with dynamo_router_* {worker_id=...}. Update docs (#6227) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-24 21:34:14 +00:00
Yan Ru Pei efa89448f5
chore: de-async scheduler read paths and unblock decode output tracking (#6510) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-24 21:00:36 +00:00
Biswa Panda 6fab12be19
test: add tests for lora aware kv cache events and routing (#6523) 2026-02-24 12:38:06 -08:00
Yan Ru Pei bc00ef3879
chore: split JetStream subscriber into dedicated module and deprecate durable_kv_events [DYN-2203] (#6477) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-24 11:30:38 -08:00
Biswa Panda 75fea78797
feat: make KV cache events and routing LoRA-aware (#6517) 2026-02-24 18:35:08 +00:00
Vladislav Nosivskoy 432fae67b1
feat: add error_type label to reqs metric for fine-grained error classification (#5568) 
Signed-off-by: Vladislav Nosivskoy <vladnosiv@gmail.com>
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-24 10:31:47 -08:00
Yan Ru Pei c9ff623538
chore: replace --enforce-disagg with --decode-fallback, default to enforcing disagg (#6515) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-24 09:24:11 -08:00
Jacky c8276cd28b
feat: Standardized Dynamo Error Type (#6303) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2026-02-23 20:23:58 -08:00
Yan Ru Pei 0cb1d73383
fix: fold scheduling into queue so backpressure actually works (#6470) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-23 18:33:03 -08:00
MatejKosec a9e0696023
fix(api): preserve interleaved reasoning order for KV cache correctness (#6442) 
Signed-off-by: Matej Kosec <mkosec@nvidia.com>
 2026-02-24 02:20:54 +00:00
Yan Ru Pei ddeee88b05
feat: expose predicted KV hit rate as Prometheus histogram (#6507) 
Signed-off-by: Pea Brane <peabrane@peabrane.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-24 00:54:14 +00:00
Thomas Montfort cb55766c47
feat(runtime): add hierarchical Model/WorkerSet architecture for multi-namespace support (#6054) 
Signed-off-by: tmontfort <tmontfort@nvidia.com>
 2026-02-23 22:44:38 +00:00
Yan Ru Pei 038b50d2b6
feat: add standalone KV indexer with query endpoint [DYN-2164] (#6446) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-23 12:12:25 -08:00
Yan Ru Pei e66f326789
fix: derive kv-router discovery name from target worker component (#6475) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-23 10:28:18 -08:00
Alec d3ee09d7ba
fix: gracefully skip NIXL UCX tests when backend unavailable (#6484) 
Signed-off-by: alec-flowers <aflowers@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-22 13:44:47 -08:00
Ryan Olson 7a6283e927
chore: cargo machete - remove unused dependencies (#6453) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
 2026-02-20 19:11:52 +00:00
Yongming Ding 9b2e8f3dd5
refactor(mocker): modularize into common/scheduler/kv_manager/cache (#6440) 
Signed-off-by: Yongming Ding <yongmingd@nvidia.com>
 2026-02-20 10:47:59 -08:00
Ayush Agarwal 6dd3ce2ea0
feat: vllm omni text to video generation pipeline (#6104) 
Signed-off-by: ayushag <ayushag@nvidia.com>
 2026-02-20 04:23:44 +00:00
Yan Ru Pei 1549c338d5
chore: derive router_id and cancellation_token from component instead of plumbing (#6419) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-19 19:48:21 +00:00
Yan Ru Pei fc229004b5
chore: Remove ZmqKvEventListener binding and rework standalone TRT-LLM example to use native Python ZMQ (#6164) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-19 04:15:13 +00:00
Yan Ru Pei c7986b3540
refactor: move per-item stream tracking into RequestGuard (#6355) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-18 19:26:30 -08:00
Yan Ru Pei c5c6a55190
feat: more flash indexer optimizations (#6305) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-18 09:13:37 -08:00
jh-nv 56379dd584
chore: standardizes Rust-side Dynamo environment variable names (#6358) 2026-02-18 11:39:03 -05:00
Graham King 9a93eb75cd
fix: Remove http temp cancel token, use real one (#6344) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-02-18 08:48:17 -07:00
Ayush Agarwal 2ace5a4a3d
chore: multimodal endpoint registration via cli modality (#6270) 
Signed-off-by: ayushag <ayushag@nvidia.com>
 2026-02-18 07:59:39 +00:00
Ryan Olson 86892a051f
refactor: dedup memory module with dynamo-memory; re-exports preserve current t… (#6345) 
Co-authored-by: Ryan Olson <rolson@nvidia.com>
Co-authored-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
 2026-02-18 07:39:08 +00:00
ishandhanani 8245633a49
feat: add MiniMax-M2 reasoning parser and fix tool call parsing (#6294) 
Co-authored-by: Claude <noreply@anthropic.com>
 2026-02-18 06:44:04 +00:00
ishandhanani b6603d90d6
feat: add Anthropic Messages API endpoint (/v1/messages) (#6231) 
Signed-off-by: Matej Kosec <mkosec@nvidia.com>
Signed-off-by: Marko Kosec <mkosec@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Matej Kosec <mkosec@nvidia.com>
 2026-02-18 06:11:52 +00:00
Yan Ru Pei ee0218b566
feat: plumb request priority through sglang and vllm handlers (#6348) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-18 04:48:52 +00:00
MatejKosec b8c7015f3c
fix: wire reasoning parser from worker runtime config into frontend (#6300) 
Signed-off-by: Matej Kosec <mkosec@nvidia.com>
 2026-02-18 00:03:51 +00:00
Yan Ru Pei e361d6facc
fix: local block hash consistency in mooncake bench (#6310) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-16 14:06:16 -08:00
Yan Ru Pei 4ede59a269
feat: speculative prefill (#6230) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Janelle Cai <jcai18@mit.edu>
 2026-02-15 09:27:34 +00:00
zhongdaor-nv 815b129126
feat: mm aware routing for vllm (#6235) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
 2026-02-13 21:16:06 -08:00
Michael Feil 0e55e82128
fix: tool call validation and stop words. (#5504) 
Signed-off-by: Michael Feil <63565275+michaelfeil@users.noreply.github.com>
 2026-02-14 01:21:10 +00:00
Tzu-Ling Kan 5624d14481
Rename fetch_llm to fetch_model (#6268) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-13 23:22:35 +00:00
Yan Ru Pei e30696e5aa
chore: allow frontend + mockers to run on macos (#6282) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-13 19:33:45 +00:00
mohammedabdulwahhab a289695c37
fix: consolidate dyn_discovery_backend and dyn_kv_store (#6167) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
 2026-02-13 19:07:11 +00:00
Yan Ru Pei 5a3c52ab70
chore: more mocker rusty cleanups (#6271) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-13 08:20:53 -08:00
Yan Ru Pei f6c34e07de
chore: otel tracings for routing overheads and metrics (#6194) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-13 09:34:46 +00:00
ishandhanani 2be83be2d8
feat: add video generation support (T2V) (#5793) 2026-02-13 09:09:56 +00:00
atchernych 5227176057
feat: Decomposed pipeline for EPP integration [DEP-730] (#5446) 
Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
 2026-02-13 06:18:29 +00:00