mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,944 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

489 Commits

Author SHA1 Message Date
Yan Ru Pei c9ff623538
chore: replace --enforce-disagg with --decode-fallback, default to enforcing disagg (#6515) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-24 09:24:11 -08:00
Jacky c8276cd28b
feat: Standardized Dynamo Error Type (#6303) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2026-02-23 20:23:58 -08:00
Tzu-Ling Kan 80cac7c14b
feat: Remove Component from public (#6403) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-23 22:58:38 +00:00
Thomas Montfort cb55766c47
feat(runtime): add hierarchical Model/WorkerSet architecture for multi-namespace support (#6054) 
Signed-off-by: tmontfort <tmontfort@nvidia.com>
 2026-02-23 22:44:38 +00:00
Yan Ru Pei 038b50d2b6
feat: add standalone KV indexer with query endpoint [DYN-2164] (#6446) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-23 12:12:25 -08:00
Ryan Olson 61c6780469
feat: add kvbm-kernels crate and upgrade cudarc to 0.19 (#6309) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
 2026-02-23 16:15:28 +00:00
Tzu-Ling Kan 42d6980546
feat: Remove public uses of CancellationToken (#6405) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-20 21:01:13 -08:00
Ryan Olson 7a6283e927
chore: cargo machete - remove unused dependencies (#6453) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
 2026-02-20 19:11:52 +00:00
zhongdaor-nv 23de4e86ae
feat: e2e mm aware kv cache routing support for trtllm backend (#5480) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
Signed-off-by: Zhongdao Ren <zhongdaor@zhongdaor-mlt.client.nvidia.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Zhongdao Ren <zhongdaor@zhongdaor-mlt.client.nvidia.com>
 2026-02-19 20:52:00 -07:00
Tzu-Ling Kan 0ce3461a9e
feat: Add runtime.endpoint() method to eliminate namespace chaining (#6386) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-19 16:02:37 -05:00
Yan Ru Pei fc229004b5
chore: Remove ZmqKvEventListener binding and rework standalone TRT-LLM example to use native Python ZMQ (#6164) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-19 04:15:13 +00:00
Tzu-Ling Kan bc8f1170ce
feat: Move ModelDeploymentCard to _internal (#6378) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-18 23:12:18 +00:00
Yan Ru Pei cf51a0c4fd
chore: gate plotters dependency behind bench feature (#6380) 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-18 15:27:19 -05:00
Anant Sharma aa16ccf545
fix: remove default-members in workspace (#6279) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2026-02-18 12:39:17 -05:00
Yan Ru Pei c5c6a55190
feat: more flash indexer optimizations (#6305) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-18 09:13:37 -08:00
Ayush Agarwal 2ace5a4a3d
chore: multimodal endpoint registration via cli modality (#6270) 
Signed-off-by: ayushag <ayushag@nvidia.com>
 2026-02-18 07:59:39 +00:00
zhongdaor-nv 815b129126
feat: mm aware routing for vllm (#6235) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
 2026-02-13 21:16:06 -08:00
Tzu-Ling Kan 5624d14481
Rename fetch_llm to fetch_model (#6268) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-13 23:22:35 +00:00
Yan Ru Pei e30696e5aa
chore: allow frontend + mockers to run on macos (#6282) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-13 19:33:45 +00:00
mohammedabdulwahhab a289695c37
fix: consolidate dyn_discovery_backend and dyn_kv_store (#6167) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
 2026-02-13 19:07:11 +00:00
ishandhanani 2be83be2d8
feat: add video generation support (T2V) (#5793) 2026-02-13 09:09:56 +00:00
Yan Ru Pei 14eceb43df
chore: rename KvPushRouter to KvRouter in python + more bindings removal (#6238) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-12 23:44:52 -08:00
atchernych 5227176057
feat: Decomposed pipeline for EPP integration [DEP-730] (#5446) 
Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
 2026-02-13 06:18:29 +00:00
Yan Ru Pei bc514fbee6
feat: router priority queue (#6010) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2026-02-13 02:43:17 +00:00
Yongming Ding 2d517e7780
feat(mocker): improve mocker's perf timing accuracy (#6100) 
Signed-off-by: Yongming Ding <yongmingd@nvidia.com>
 2026-02-12 17:33:43 -08:00
Karen Chung cd6984b951
feat: use RNG when dp routing targets are tied; override no-assume-kv-reuse for decode requests (#6253) 
Signed-off-by: Karen Chung <karenc@nvidia.com>
 2026-02-12 15:59:37 -08:00
Graham King bbe82f182a
chore: Remove dynamo-run and mistral-rs engine (#6203) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-02-12 21:50:58 +00:00
Yan Ru Pei 937398cf61
feat: Flash Indexer (#5785) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: jthomson04 <jothomson@nvidia.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: Janelle Cai <jcai18@mit.edu>
Co-authored-by: jthomson04 <jwillthomson19@gmail.com>
Co-authored-by: Janelle Cai <jcai18@mit.edu>
 2026-02-11 22:26:56 -08:00
Jonathan Tong 39d645e586
docs: migrate Fern docs from fern/ into docs/ (#6206) 
Signed-off-by: Jont828 <jt572@cornell.edu>
 2026-02-11 16:22:27 -08:00
Yan Ru Pei 6a728d1044
chore: remove kv indexers bindings (#6159) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-11 20:23:48 +00:00
Graham King 73dc3be81a
chore: Merge bindings client and client2 functions (#6158) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-02-11 11:51:01 -05:00
Yan Ru Pei f46720c996
feat: add router-level Prometheus metrics and centralize request tracking (#6146) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-11 07:36:08 +00:00
Keiven C e18840cef3
feat: add Prometheus auto and custom label injection for engine metrics (#5989) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-02-10 21:22:53 -08:00
Graham King 4f99451bb0
feat(frontend): Use vllm for pre and post processing (#5544) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-02-10 22:39:11 +00:00
Graham King f1bcb17542
feat: Add metric tokenizer_latency_ms (#6092) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-02-10 08:52:31 -08:00
Keiven C 027d2653a5
feat: expose Python Prometheus metric via DynamoComponentMetrics (#5817) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-02-09 17:10:37 -08:00
Ayush Agarwal 9f76d0606c
feat: text to image vLLM Omni (#5912) 
Signed-off-by: ayushag <ayushag@nvidia.com>
 2026-02-09 18:19:27 -05:00
Yan Ru Pei 6783bdcaa9
chore: enable local indexers by default, and use normal event plane by default (not jetstream) (#5941) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-08 20:48:02 +00:00
Yongming Ding 7c25f70291
feat(mocker): add optional KV cache allocation/eviction trace (#6052) 
Signed-off-by: Yongming Ding <yongmingd@nvidia.com>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
 2026-02-07 03:02:28 +00:00
Jacky 1ffa489ea1
refactor: Move --migration-limit flag from backend to frontend (#5918) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2026-02-06 20:50:55 +00:00
Anant Sharma 3842b24479
fix: update bytes crate version to latest (#6041) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2026-02-06 15:11:04 -05:00
Yan Ru Pei 3e41702211
chore: remove and unify bindings in kv.rs (#6016) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-06 18:44:46 +00:00
Biswa Panda 953e5d7beb
feat(lora): implement HRW-based LoRA allocation system (#5992) 2026-02-05 14:56:15 -08:00
Keiven C cb7ebdd7e8
chore: remove unused NIM specific code (part 2) (#5893) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-02-05 12:21:10 -08:00
Yan Ru Pei 91fb78cddf
chore: make mocker it's own crate (#5958) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-05 19:35:42 +00:00
Hongkuan Zhou 0c0336e6ea
feat: Add per-worker Prometheus metrics for router load monitoring (#5842) 
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
 2026-02-04 11:51:43 -08:00
Konrad Nowicki 7e970d44c9
feat: image diffusion with SGLang diffusion (#5609) 
Signed-off-by: Konrad Nowicki <knowicki@nvidia.com>
Co-authored-by: dagil-nvidia <dagil@nvidia.com>
 2026-02-03 21:19:21 +00:00
milesial 9bff03f237
feat: vLLM backend with frontend media decoding (#5781) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2026-02-03 09:19:26 -08:00
Yan Ru Pei a337113aa9
feat: prefill tokens threshold based on max num batched tokens frac (#5867) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-03 03:08:25 +00:00
Yan Ru Pei 902eabd9ec
feat: per dp rank gap detection (#5873) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-03 03:07:57 +00:00
Biswa Panda 373e76c1ae
feat(lora): Add lora_name tracking to scheduling and sequence management (#5875) 2026-02-03 00:59:09 +00:00
Janelle Cai 18d9d1fa4f
feat(mocker): pre-fetch model and staggered launches (#5871) 2026-02-03 00:46:23 +00:00
ishandhanani c5f5ab6056
fix: enable DP attention KV events for multi-node deployments (#5589) 
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: huitianbai <huitianbai@gmail.com>
Co-authored-by: Huitianqi Bai <huitianbai@users.noreply.github.com>
 2026-02-02 07:20:29 +00:00
Biswa Panda 1174c819cb
feat: remove deprecated events api (#5778) 2026-01-29 05:48:16 +00:00
Yan Ru Pei fc92fc187b
chore: clean ups in kv-router (#5771) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-29 03:20:04 +00:00
Keiven C 5a00a7d643
fix: prevent duplicate Prometheus metrics from Python expfmt callbacks (#5761) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-01-28 23:34:33 +00:00
Keiven C 8e72fb69cf
docs: DYN-1967 update metrics docs after kvstats removal (#5704) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-01-27 17:24:12 -08:00
Dmitry Tokarev 2d6babe5fc
chore: Bump rust lib oneshot to 0.1.13 (#5694) 
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
 2026-01-27 17:26:11 -05:00
Pavithra Vijayakrishnan 66c369964a
chore: version update for 0.9.0 (#5661) 
Signed-off-by: pvijayakrish <pvijayakrish@nvidia.com>
 2026-01-26 17:18:05 -08:00
Qi Wang 89e135b9ce
chore: remove legacy KVBM pythong static type checks and test (#5486) 2026-01-26 12:46:49 -08:00
Abhishek Gupta 7de80960dc
feat: add __all__ exports and __repr__ methods for better debugging (#5606) 2026-01-23 12:44:03 -08:00
Keiven C 7361de4208
refactor: reduce Python API to only Prometheus Exposition Format callback (#5594) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-01-23 10:06:06 -08:00
Yan Ru Pei feb6d272ac
chore: nuke ForwardPassMetrics (#5531) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-23 16:26:01 +00:00
Nate Mailhot 6dba119d7a
chore: update nixl to 0.9.0 (#5528) 2026-01-22 15:25:03 -08:00
Graham King ca31b3aa6b
chore: Remove binding OAIChatPreprocessor (#5516) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-22 08:43:02 +08:00
Michael Feil fa66f65437
feat: typing - Add py.typed support for typing (#4913) 
Signed-off-by: Michael Feil <63565275+michaelfeil@users.noreply.github.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>
 2026-01-20 15:28:44 +00:00
Schwinn Saereesitthipitak d953f9d038
feat: add endpoint instance registration/unregistration for sleep/wake (#5203) 
Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-20 12:17:58 +00:00
Yan Ru Pei 050906b593
feat: track output tokens / blocks in the Router (optional) (#5452) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-18 09:30:25 +08:00
Graham King a6e7348400
chore(bindings): Remove 'Backend' binding (#5458) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-16 12:29:50 -05:00
Graham King c5458b9954
chore(router): Remove deprecated best_worker_id and metrics_labels (#5412) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-14 16:20:42 +00:00
Neal Vaidya a936ba5ad2
fix: read store_kv arg from env when using decorator (#5421) 
Signed-off-by: Neal Vaidya <nealv@nvidia.com>
 2026-01-14 03:27:34 +00:00
Yan Ru Pei 7fdc742e17
feat: allow router not assuming decode kv reuse (#5350) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-13 07:33:47 +08:00
Graham King f6ae58e349
fix(llm): Send HF URLs to frontend not full paths (#5290) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-12 12:05:49 -05:00
Biswa Panda 0d3ff44000
fix: enable toggling kv events pub/sub (currently nats based) with --no-kv-events flag (#5237) 2026-01-09 06:36:51 +00:00
Neelay Shah d22d9e761e
fix: support truthy values for OTEL_EXPORT_ENABLED in bindings layer [DYN-1690][DYN-1646] (#5129) 
Co-authored-by: Claude <noreply@anthropic.com>
 2026-01-02 23:39:09 +00:00
Tushar Sharma cf433e6825
chore: update all copyright headers in repo to 2026 (#5130) 
Signed-off-by: Tushar Sharma <tusharma@nvidia.com>
 2026-01-02 22:08:23 +00:00
Yan Ru Pei 0b33c1dfa0
fix: sglang disagg routing fixes and optimizations [DYN-1692] (#5106) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
Co-authored-by: Sean SH Choi <sechoi@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2025-12-31 10:55:44 -08:00
Phillippe Siclait ac8c9023e4
fix: Truncate HttpError message to first 8192 characters (#5020) 
Signed-off-by: Phillippe Siclait <546332+siclait@users.noreply.github.com>
 2025-12-22 12:23:35 -08:00
Yan Ru Pei f4245c9929
chore: clean up PreprocessedRequest (#5040) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-19 23:15:59 +00:00
Jacky e6a6a1f244
feat: Request Migration Metrics (#5029) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2025-12-19 12:22:36 -08:00
Nate Mailhot 910b587db1
chore: update nixl to 0.8.0 and ucx 1.20 (#5007) 
Signed-off-by: Nate Mailhot <nmailhot@nvidia.com>
 2025-12-19 10:55:40 -08:00
Anant Sharma 7436e3bc03
chore: update versions for 0.8.0 release (#5031) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2025-12-19 10:21:52 -08:00
milesial d2faf0e6f5
feat: Runtime media decoder config (#5011) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-12-19 05:34:09 +00:00
zhongdaor-nv 481dc6363b
feat: add multimodal support to KV router with standalone trtllm example (#4577) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
 2025-12-18 21:01:45 -08:00
Keiven C c22280ccf4
refactor: frontend/grpc tests to use dynamic ports (#4992) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-12-18 16:21:28 -08:00
Yan Ru Pei 85b5c808ac
feat: unify per-request metric tracking (#5004) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-18 23:59:25 +00:00
Biswa Panda 18b64e90b6
feat: switch to tcp as default request plane (#4845) 2025-12-18 23:05:38 +00:00
Anant Sharma 5b82b8b096
chore: update versions to match latest release (#4966) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2025-12-18 08:45:44 -08:00
Graham King da0f2fb84a
feat(frontend): First part of Python request handling (#4999) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-12-17 20:03:39 -05:00
Sean SH Choi 1c39e4835c
feat: add frontend based prefill request routing for sglang (#4635) 
Signed-off-by: Sean Choi <sechoi@nvidia.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2025-12-17 09:51:46 -08:00
milesial 74fcd4a9b9
feat: video decoder in the frontend (#4719) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-12-16 14:28:06 -08:00
KrishnanPrash 24ac6c1588
chore: bump minijinja to 2.14.0 (#4949) 
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
 2025-12-15 09:58:52 -08:00
Yan Ru Pei c2a29f8045
chore: enable local indexers for sglang and trtllm (#4932) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-12 11:41:37 -08:00
Karen Chung 332499458b
feat: worker-local KvIndexer in KvEventPublisher (#4519) 
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
 2025-12-11 23:18:34 +00:00
Yan Ru Pei 10b01b4590
feat: early rejection based on active prefill tokens (#4837) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-11 23:16:02 +00:00
Dmitry Tokarev 2ef0f5fbab
fix: relax pydantic requirements (#4865) 
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
 2025-12-11 19:32:19 +00:00
Vladislav Nosivskoy f8bb53c0ed
feat: add cached tokens prometheus metric (#4534) 
Signed-off-by: Vladislav Nosivskoy <vladnosiv@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-12-11 08:03:12 +03:00
Jacky 00f8615e47
test: Ensure Request Cancellation, Migration, Rejection Work with TCP transport (#4875) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2025-12-11 02:21:17 +00:00
J Wyman 96fe63fe2e
feat: nixl_connect: Improve Concurrency Support (#4433) 
Signed-off-by: J Wyman <jwyman@nvidia.com>
 2025-12-09 17:29:22 -05:00
Keiven C 1e37c1033f
fix: add timeout to tests before lora to unblock (#4808) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-12-08 20:50:33 -08:00