mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,987 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

144 Commits

Author SHA1 Message Date
Yan Ru Pei b5e762b2b8
feat: remove stale workers on snapshot + some refactoring (#3589) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-13 21:36:09 +00:00
mohammedabdulwahhab 1f92dd547e
feat: OTEL Exporter and Tempo Visualization (#3307) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
 2025-10-13 18:39:09 +00:00
Yan Ru Pei c3fcfdd6e3
chore: consolidations of KvPushRouter bindings and usage examples (#3543) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-10-11 05:47:25 +00:00
Keiven C 65cc533702
feat: implement custom backend metrics for NIM (#3266) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-10 20:28:58 -07:00
Ziqi Fan 4224e57da0
feat: rm the old KVBM metrics | update G2 to G3 metrics collection (#3561) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-11 01:48:09 +00:00
Keiven C 55e458d899
feat: add SGLang and vLLM passthrough metrics on Dynamo backend worker (#3539) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-10 17:59:35 -07:00
Keiven C a4746ab65f
fix: callback registration, fix metric name access, ensure ordered vec, etc... (#3541) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-11 00:03:24 +00:00
Graham King 0e0218ff02
chore: Remove model_config from LocalModel (#3558) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-10 22:01:30 +00:00
Keiven C f2ba58e568
feat: add Rust-to-Python const code generator for prometheus_names.py (#3425) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-10 11:20:36 -07:00
Ziqi Fan ca67409888
feat: add KVBM host to disk metrics | clean up dashboard (#3534) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-10 18:17:34 +00:00
Ziqi Fan b94ecd1627
feat: add disk offloading filtering in KVBM (#3532) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2025-10-10 09:51:55 -07:00
Graham King e9a710095b
chore: Remove clear_namespace script (#3519) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-09 13:43:59 -04:00
Ziqi Fan 9975cb9ffe
feat: enable KVBM to support PD disagg in Dynamo vLLM (#3352) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-08 13:45:36 -07:00
Richard Huo cf83794ae0
feat: DIS-678 kvbm modularity: standalone metrics endpoint (#3433) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-10-08 10:18:33 -07:00
Keiven C 0c4c4d1de3
feat: add Python MetricsRegistry Python metrics registration (#3341) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-07 21:24:25 -07:00
Graham King bdad6f1a50
fix: Make planner VirtualConnectorClient also use v1/ prefix. (#3468) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-07 22:34:30 +00:00
Graham King a5371bfc50
feat(etcd): Version the etcd keys (#3458) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-07 16:48:26 +00:00
Graham King 81162dfeb9
chore(discovery): Watch/publish ModelDeploymentCard instead of ModelEntry (#3350) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-07 10:02:02 -04:00
Yan Ru Pei 30610e7371
feat: use KvPushRouter for prefill router (#3401) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-03 18:21:48 -07:00
Olga Andreeva d2e3b66e65
feat: Transition to FullyContiguous Host and Disk layouts (#3090) 
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: oandreeva-nv <oandreeva-nv@nvidia.com>
 2025-10-01 16:26:39 -07:00
Richard Huo 713e9e481b
fix: DIS-706 skip offloading the G1 matched blocks during offloading (#3299) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-10-01 10:28:06 -07:00
Yan Ru Pei 9b9536d0d7
feat: make prefill router general (#3329) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-30 20:38:47 -07:00
Keiven C f4a3a6b66a
refactor: standardize Prometheus metric naming conventions (part 1) (#3035) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-09-30 19:01:39 -07:00
Michael Feil 5b457b70a6
feat: python add abi compatability for cross-platform builds + add a unit test to HttpServer (#3044) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Signed-off-by: Michael Feil <63565275+michaelfeil@users.noreply.github.com>
Signed-off-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Signed-off-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
 2025-09-30 14:42:42 -07:00
Yan Ru Pei d354763c40
fix: python bindings for router should register to etcd as well (#3302) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-30 12:38:56 -07:00
Keiven C cacac9b9f4
feat: add Python const for Prometheus metric names (#3244) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-09-30 10:28:47 -07:00
Yan Ru Pei 3aa3077808
fix: more fixes for stable router benchmarking (#3264) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-29 23:08:56 +00:00
Ziqi Fan e21dcf6cab
feat: enable KVBM emit metrics in Dynamo TRTLLM (#3254) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-09-29 08:48:51 -07:00
Graham King 7ebbd001c0
chore: Remove etcd from python bindings (#3238) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-26 12:58:30 -04:00
Alec 5bb7490448
chore: bump vllm version to 0.10.2 (#3180) 
Signed-off-by: Alec <aflowers@nvidia.com>
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Co-authored-by: Kris Hung <krish@nvidia.com>
 2025-09-26 04:01:57 +00:00
Neelay Shah f2e2e935b2
feat: Add distributed tracing context support to Python bindings (#3160) 
Signed-off-by: nnshah1 <neelays@nvidia.com>
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
 2025-09-25 17:26:35 -07:00
Graham King c03e2f6bbe
chore: Migrate planner virtual_connector internals into bindings (#3205) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-25 18:04:58 -04:00
GuanLuo 6ba64c31f5
feat: tensor type for generic inference. (#2746) 
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-09-24 07:31:09 +00:00
Yan Ru Pei 031590fc14
feat: vllm prefill router (#3155) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-22 16:38:56 -07:00
Graham King 7a5a0bd6cd
chore: Upgrade Rust to 1.90 (#3147) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-19 18:54:29 -04:00
Yan Ru Pei 5b19a39bab
feat: allow router to not track active blocks (prefill), and to not track cached blocks (decode) (#3135) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-19 20:33:07 +00:00
Graham King 3865a94148
feat: Port vllm port allocator to Rust in bindings (#3125) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-19 18:34:52 +00:00
Olga Andreeva 2d39f1b1bb
feat: KVBM connector : enabling vectorized copy from pinned memory to device memory and vice versa (#2989) 
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: oandreeva-nv <oandreeva-nv@nvidia.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: oandreeva-nv <oandreeva-nv@nvidia.com>
 2025-09-19 09:35:07 -07:00
Graham King b6595e2484
chore(bindings): Provide a binding to clear etcd namespace (#3094) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-18 11:49:08 -04:00
Yan Ru Pei 78a3fedab9
fix: hook up worker removals for indexer (#3095) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-17 21:33:25 +00:00
Graham King f88d7dc74b
chore(bindings): Remove NatsQueue (#3086) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-17 12:46:54 -04:00
Graham King 9060ce12ce
feat: Make part of discovery re-usable (#3073) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-17 10:43:24 -04:00
Tzu-Ling Kan 08cb08c1bc
feat: Canary Health Check. (#2903) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2025-09-17 03:31:56 +00:00
Graham King 723f2da74b
chore: Remove more extended Apache headers (#3063) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-16 15:48:51 -04:00
Graham King 87e6e0529d
fix: Interactive inputs actually stops, does not ignore stop token (#3057) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-16 14:39:05 -04:00
Ziqi Fan 55659eae70
fix: early stop if CPU or disk space not set when using KVBM (#2997) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-09-15 10:14:01 -07:00
blarson-b10 4000097653
feat: adds kv indexer metrics (#2905) 
Signed-off-by: Brian Larson <brian.larson@baseten.co>
 2025-09-10 21:33:15 +00:00
Graham King cb5a657a6a
fix: Load the tokenizer JSON once for chat and completions. (#2910) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-05 16:48:16 -04:00
Olga Andreeva 27fad26faf
refactor: Split ModelType to ModelInput for request and response type; ModelType for the supported workloads (#2714) 
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Guan Luo <gluo@nvidia.com>
Co-authored-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
 2025-09-03 22:22:37 +00:00
KrishnanPrash c920cbd9dc
feat: Add --custom-jinja-template argument to pass a custom chat template for vLLM (#2829) 
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
 2025-09-03 14:22:15 -07:00