mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,795 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

476 Commits

Author SHA1 Message Date
Ryan Olson 63d7c01c9c
feat: velo-backend (#6547) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
 2026-03-09 23:26:25 +00:00
Yan Ru Pei 2e29620df1
chore(kv-router): move benches to lib/bench to break circular dep [OPS-3752] (#7013) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-03-06 23:24:03 +00:00
Yifan Jiang 100819299f
feat(trtllm): add additional metrics for dynamo-trtllm (#6668) 
Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-06 14:34:38 -08:00
jh-nv 2831bfecde
chore: add mypy typing to vllm (#6858) 2026-03-06 21:45:43 +00:00
Graham King abc02c689f
fix: llm/mocker: Remove the llm -> mocker crate dependency, move config (#6998) 
Signed-off-by: Graham King <grahamk@nvidia.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-03-06 09:41:39 -08:00
Yan Ru Pei f934744fb9
fix: restore --enforce-disagg to reject requests before prefill router activates (#6957) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-03-06 04:39:28 +00:00
jh-nv 6398d453b1
chore: add mypy typing to sglang (#6859) 2026-03-05 21:07:33 +00:00
jh-nv f6b4da0846
chore: mypy typing for python bindings (#6857) 2026-03-05 18:26:13 +00:00
jh-nv 869e733682
chore: add mypy typing for components/src/dynamo/common (#6721) 2026-03-05 04:48:56 +00:00
jh-nv 2128d48ae9
chore: add mypy typing to frontend (#6861) 2026-03-04 22:03:25 +00:00
jh-nv a96f5d7398
chore: add mypy for router (#6864) 2026-03-04 20:54:22 +00:00
GuanLuo 90d7463721
fix: properly setup and register vLLM worker for external / hybrid load balancing. Update launch script (#6695) 
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
 2026-03-03 14:00:52 -08:00
Anant Sharma 9254e3d4d2
chore: update nixl version to 0.10.1 (#6826) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2026-03-03 16:47:55 -05:00
Yan Ru Pei 8055b7dd2c
fix(kv-router): plumb expected_osl through scheduler queue path (#6812) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-03-03 10:57:42 -08:00
Ryan Olson 57bdbdf1ee
feat: upgrade rust to v1.93.0 (#6802) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
 2026-03-03 17:08:35 +00:00
Ryan Olson 7604295938
feat(kvbm): add testing utilities to kvbm-logical and kvbm-physical (#6691) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
 2026-03-02 21:18:32 -07:00
Yan Ru Pei 073fb4371d
feat(kv-router): add router queue depth Prometheus metric and nvext field (#6786) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-03-02 17:04:51 -08:00
Yan Ru Pei 07742cc27f
feat: move standalone KV indexer into kv-router crate with HTTP integration tests (#6569) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 10:34:21 -08:00
Neal Vaidya ece08dc926
docs: restructure docs directory and move fern config to fern/ (#6700) 
Signed-off-by: Neal Vaidya <nealv@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-01 19:58:22 -08:00
Ryan Olson 9ab148dcb9
feat: kvbm-physical (#6490) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
 2026-03-01 09:43:03 +00:00
Michael Feil 713c96d26d
feat: dynamic batching of client side events (#6733) 
Signed-off-by: michaelfeil <63565275+michaelfeil@users.noreply.github.com>
 2026-02-28 18:27:57 +00:00
Yan Ru Pei e5850e2385
feat(kv-router): add ActiveSequences benchmark and extract common bench utils (#6633) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 02:55:20 +00:00
Yan Ru Pei 55c0a769b4
feat: default router_event_threads to 4 (#6672) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-27 22:14:47 +00:00
Nate Mailhot 64082ccd3a
chore: upgrade nixl to 0.10.0 (#6685) 2026-02-27 13:01:07 -08:00
ishandhanani f1d5c95a4e
feat: wire nvext.cache_control TTL-based pinning through Dynamo router (#6213) 
Co-authored-by: Claude <noreply@anthropic.com>
 2026-02-27 01:03:27 +00:00
Harrison Saturley-Hall db88c9530b
chore: bump versions to 1.0.0 (#6613) 
Signed-off-by: Harrison Saturley-Hall <hsaturleyhal@nvidia.com>
 2026-02-25 22:42:33 -05:00
Tanmay Verma 4a71926d74
chore: Decouple TLLM_LOG_LEVEL with DYN_LOG (#6606) 2026-02-25 17:31:51 -08:00
Yan Ru Pei 3f99cf217f
chore: move ActiveSequences to kv-router and add unit tests (#6600) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-26 00:53:04 +00:00
Alec e10787894d
fix: add periodic vLLM engine stats logging and fix log routing (#6566) 
Signed-off-by: alec-flowers <aflowers@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 23:45:34 +00:00
daiyaanarfeen 651ef5b506
feat: throughput-metrics-source for SLA planner + GlobalPlanner disagg scaling (#6500) 
Signed-off-by: Daiyaan <darfeen@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-02-25 22:58:24 +00:00
Nikita 21fce9ba0d
feat: Tiktoken support (#6460) 
Signed-off-by: Nikita Sukharev <kaonael@gmail.com>
 2026-02-25 12:46:47 -08:00
Yan Ru Pei ebaf048d16
chore: plumb allowed_worker_ids through RoutingHints (#6580) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-25 12:12:53 -08:00
Keiven C ff06b17e7f
fix: guarantee RouterRequestMetrics availability & documentation updates (#6558) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-25 10:18:01 -08:00
atchernych c916cd42ff
feat: Support epp's "pods" interface in Dynamo fixes [DEP-424] (#6302) 
Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
 2026-02-25 01:54:38 +00:00
Keiven C 1df620b4d0
feat: router metrics with dynamo_router_* {worker_id=...}. Update docs (#6227) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-24 21:34:14 +00:00
Yan Ru Pei bc00ef3879
chore: split JetStream subscriber into dedicated module and deprecate durable_kv_events [DYN-2203] (#6477) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-24 11:30:38 -08:00
Biswa Panda 75fea78797
feat: make KV cache events and routing LoRA-aware (#6517) 2026-02-24 18:35:08 +00:00
Yan Ru Pei c9ff623538
chore: replace --enforce-disagg with --decode-fallback, default to enforcing disagg (#6515) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-24 09:24:11 -08:00
Jacky c8276cd28b
feat: Standardized Dynamo Error Type (#6303) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2026-02-23 20:23:58 -08:00
Tzu-Ling Kan 80cac7c14b
feat: Remove Component from public (#6403) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-23 22:58:38 +00:00
Thomas Montfort cb55766c47
feat(runtime): add hierarchical Model/WorkerSet architecture for multi-namespace support (#6054) 
Signed-off-by: tmontfort <tmontfort@nvidia.com>
 2026-02-23 22:44:38 +00:00
Yan Ru Pei 038b50d2b6
feat: add standalone KV indexer with query endpoint [DYN-2164] (#6446) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-23 12:12:25 -08:00
Ryan Olson 61c6780469
feat: add kvbm-kernels crate and upgrade cudarc to 0.19 (#6309) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
 2026-02-23 16:15:28 +00:00
Tzu-Ling Kan 42d6980546
feat: Remove public uses of CancellationToken (#6405) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-20 21:01:13 -08:00
Ryan Olson 7a6283e927
chore: cargo machete - remove unused dependencies (#6453) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
 2026-02-20 19:11:52 +00:00
zhongdaor-nv 23de4e86ae
feat: e2e mm aware kv cache routing support for trtllm backend (#5480) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
Signed-off-by: Zhongdao Ren <zhongdaor@zhongdaor-mlt.client.nvidia.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Zhongdao Ren <zhongdaor@zhongdaor-mlt.client.nvidia.com>
 2026-02-19 20:52:00 -07:00
Tzu-Ling Kan 0ce3461a9e
feat: Add runtime.endpoint() method to eliminate namespace chaining (#6386) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-19 16:02:37 -05:00
Yan Ru Pei fc229004b5
chore: Remove ZmqKvEventListener binding and rework standalone TRT-LLM example to use native Python ZMQ (#6164) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-19 04:15:13 +00:00
Tzu-Ling Kan bc8f1170ce
feat: Move ModelDeploymentCard to _internal (#6378) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2026-02-18 23:12:18 +00:00
Yan Ru Pei cf51a0c4fd
chore: gate plotters dependency behind bench feature (#6380) 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-02-18 15:27:19 -05:00