mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,944 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

279 Commits

Author SHA1 Message Date
Waël Boukhobza 3ea22fcf39
feat(router): max tree size based pruning (#4057) 
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
 2025-11-12 08:33:49 -08:00
Graham King fbad28602e
chore: Remove the python bindings for port allocation (#4237) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-11 20:35:36 +00:00
Graham King e1af3af6ee
chore: Remove static mode (#4235) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-11 19:25:12 +00:00
zhongdaor-nv ec7af93953
fix: Extend add_tensor_model so that ModelDeploymentCard can be correctly picked up (#4169) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
 2025-11-10 13:59:40 -08:00
Keiven C afccc9d434
refactor: consolidate Observability files (e.g. OTEL docker-compose, md files) (#4173) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-11-10 20:19:34 +00:00
Graham King cf630bf7b6
refactor: Make the Runtime and DistributedRuntime fields private (#4193) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-10 15:56:22 +00:00
milesial 14af074ecd
feat: Media decoder and fetcher options in the MDC (#4094) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-11-08 02:17:39 +00:00
Graham King 794c0a4431
feat(keyvalue): Filesystem backed KeyValueStore (#4138) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-07 21:11:33 +00:00
Jacky 6e2b22eaef
feat: ETCD high availability client failover - lease watch resilience (#3950) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-11-05 17:21:09 -08:00
Yan Ru Pei 427ca9aba4
chore: no need for explicit Python::with_gil (#4075) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-05 05:41:49 +00:00
Michael Feil defe5de74b
feat: release gil for publisher (#3747) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
 2025-11-05 02:17:06 +00:00
Graham King dadf0e2247
chore: Remove old DisaggregatedRouter, making etcd presence optional (#4011) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-03 10:10:13 -05:00
Yan Ru Pei 6ff49edbb7
feat: efficient serving of multiple mockers (#3997) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-01 01:24:55 +00:00
Richard Huo 6afa679c5d
chore: KVBM pip wheel (#3826) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>
 2025-10-31 14:19:30 -07:00
Kris Hung 1898601025
feat: Add KV event consolidator for KVBM (vllm) and router integration (#3725) 
Signed-off-by: krishung5 <krish@nvidia.com>
 2025-10-30 17:11:42 +00:00
Yan Ru Pei cc4c3516bb
feat: mocker disagg (#3833) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-28 20:36:02 +00:00
Graham King c78b590134
chore(runtime): Do not expose etcd lease ID (#3915) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-28 16:20:25 +00:00
Kyle McGill 0028cdf43d
feat: Use ForwardPassCallback api from TRTLLM to register end of forward pass callback to enable cuda graphs (#3297) 
Signed-off-by: Kyle McGill <kmcgill@nvidia.com>
 2025-10-27 15:12:04 -07:00
Keiven C cbe0b177ab
refactor: redesign the metrics API from Trait to composition to make the code cleaner and easier to understand (#3687) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-24 14:34:29 -07:00
zhongdaor-nv 9e5407f20d
feat: python binding for kserve grpc frontend (#3739) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-10-24 11:20:20 -07:00
Graham King 7731b0245c
chore: Use KeyValueStoreManager instead of etcd::Client (#3822) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-23 14:09:58 -04:00
Yan Ru Pei 41ff394f51
chore: restructure mocker cli args handling, to include prefill/decode (#3847) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-23 10:41:23 -07:00
Richard Huo 94aa2a7bbb
refactor: kvbm modularity DIS-657 Eliminate ETCD from the leader-worker initialization (#3202) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-10-23 00:10:09 -07:00
Michael Feil 681951d4e6
feat: python gil release for radix tree + dump_tree_as_events in python (#3748) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
 2025-10-22 01:41:35 +00:00
Yan Ru Pei e01c6e99bd
feat: bake prefill router into frontend, supporting vllm for now (#3762) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-21 15:43:02 -07:00
mohammedabdulwahhab 8a04d7b252
fix: revert placement of logging init if otel exports is disabled (#3797) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
 2025-10-21 17:36:31 -04:00
Ziqi Fan d5840cda65
feat: enable KVBM GPU offload to Disk bypassing CPU (#3510) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-21 08:49:54 -07:00
Graham King f6ed01b14c
chore: Replace ServiceConfigBuilder with add_stats_service (#3736) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-20 16:47:05 -04:00
Yan Ru Pei 4b7a806c7a
feat: add prefill workers to discovery (#3709) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-18 00:59:32 +00:00
Yan Ru Pei 4c207e0c8c
chore: remove kv metrics scraping and aggregation (#3701) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-17 10:58:48 -07:00
Yan Ru Pei ae4e96a2bc
fix: mocker engines should ignore downloading weights from hf (again) (#3664) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-16 02:48:40 +00:00
Jacky 734d2f87a8
feat: Enable cancellation during or before a stream is established (#3635) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-10-15 19:22:55 -07:00
Yan Ru Pei f978f4d1d6
feat: dp rank routing (#3597) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-16 01:48:27 +00:00
Ziqi Fan f5854e1784
fix: avoid offload redundant prefill blocks | fix cuda graph hanging (#3632) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-15 10:38:35 -07:00
Graham King ab0da5821e
feat: Python binding to download a model. (#3593) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-15 12:26:02 -04:00
Yan Ru Pei b5e762b2b8
feat: remove stale workers on snapshot + some refactoring (#3589) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-13 21:36:09 +00:00
mohammedabdulwahhab 1f92dd547e
feat: OTEL Exporter and Tempo Visualization (#3307) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
 2025-10-13 18:39:09 +00:00
Yan Ru Pei c3fcfdd6e3
chore: consolidations of KvPushRouter bindings and usage examples (#3543) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-10-11 05:47:25 +00:00
Keiven C 65cc533702
feat: implement custom backend metrics for NIM (#3266) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-10 20:28:58 -07:00
Ziqi Fan 4224e57da0
feat: rm the old KVBM metrics | update G2 to G3 metrics collection (#3561) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-11 01:48:09 +00:00
Keiven C 55e458d899
feat: add SGLang and vLLM passthrough metrics on Dynamo backend worker (#3539) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-10 17:59:35 -07:00
Keiven C a4746ab65f
fix: callback registration, fix metric name access, ensure ordered vec, etc... (#3541) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-11 00:03:24 +00:00
Graham King 0e0218ff02
chore: Remove model_config from LocalModel (#3558) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-10 22:01:30 +00:00
Keiven C f2ba58e568
feat: add Rust-to-Python const code generator for prometheus_names.py (#3425) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-10 11:20:36 -07:00
Ziqi Fan ca67409888
feat: add KVBM host to disk metrics | clean up dashboard (#3534) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-10 18:17:34 +00:00
Ziqi Fan b94ecd1627
feat: add disk offloading filtering in KVBM (#3532) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2025-10-10 09:51:55 -07:00
Graham King e9a710095b
chore: Remove clear_namespace script (#3519) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-09 13:43:59 -04:00
Ziqi Fan 9975cb9ffe
feat: enable KVBM to support PD disagg in Dynamo vLLM (#3352) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-08 13:45:36 -07:00
Richard Huo cf83794ae0
feat: DIS-678 kvbm modularity: standalone metrics endpoint (#3433) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-10-08 10:18:33 -07:00
Keiven C 0c4c4d1de3
feat: add Python MetricsRegistry Python metrics registration (#3341) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-07 21:24:25 -07:00
Graham King bdad6f1a50
fix: Make planner VirtualConnectorClient also use v1/ prefix. (#3468) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-07 22:34:30 +00:00
Graham King a5371bfc50
feat(etcd): Version the etcd keys (#3458) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-07 16:48:26 +00:00
Graham King 81162dfeb9
chore(discovery): Watch/publish ModelDeploymentCard instead of ModelEntry (#3350) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-07 10:02:02 -04:00
Yan Ru Pei 30610e7371
feat: use KvPushRouter for prefill router (#3401) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-03 18:21:48 -07:00
Olga Andreeva d2e3b66e65
feat: Transition to FullyContiguous Host and Disk layouts (#3090) 
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: oandreeva-nv <oandreeva-nv@nvidia.com>
 2025-10-01 16:26:39 -07:00
Richard Huo 713e9e481b
fix: DIS-706 skip offloading the G1 matched blocks during offloading (#3299) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-10-01 10:28:06 -07:00
Yan Ru Pei 9b9536d0d7
feat: make prefill router general (#3329) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-30 20:38:47 -07:00
Keiven C f4a3a6b66a
refactor: standardize Prometheus metric naming conventions (part 1) (#3035) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-09-30 19:01:39 -07:00
Michael Feil 5b457b70a6
feat: python add abi compatability for cross-platform builds + add a unit test to HttpServer (#3044) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Signed-off-by: Michael Feil <63565275+michaelfeil@users.noreply.github.com>
Signed-off-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Signed-off-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
 2025-09-30 14:42:42 -07:00
Yan Ru Pei d354763c40
fix: python bindings for router should register to etcd as well (#3302) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-30 12:38:56 -07:00
Keiven C cacac9b9f4
feat: add Python const for Prometheus metric names (#3244) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-09-30 10:28:47 -07:00
Yan Ru Pei 3aa3077808
fix: more fixes for stable router benchmarking (#3264) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-29 23:08:56 +00:00
Ziqi Fan e21dcf6cab
feat: enable KVBM emit metrics in Dynamo TRTLLM (#3254) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-09-29 08:48:51 -07:00
Graham King 7ebbd001c0
chore: Remove etcd from python bindings (#3238) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-26 12:58:30 -04:00
Alec 5bb7490448
chore: bump vllm version to 0.10.2 (#3180) 
Signed-off-by: Alec <aflowers@nvidia.com>
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Co-authored-by: Kris Hung <krish@nvidia.com>
 2025-09-26 04:01:57 +00:00
Neelay Shah f2e2e935b2
feat: Add distributed tracing context support to Python bindings (#3160) 
Signed-off-by: nnshah1 <neelays@nvidia.com>
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
 2025-09-25 17:26:35 -07:00
Graham King c03e2f6bbe
chore: Migrate planner virtual_connector internals into bindings (#3205) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-25 18:04:58 -04:00
GuanLuo 6ba64c31f5
feat: tensor type for generic inference. (#2746) 
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-09-24 07:31:09 +00:00
Yan Ru Pei 031590fc14
feat: vllm prefill router (#3155) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-22 16:38:56 -07:00
Graham King 7a5a0bd6cd
chore: Upgrade Rust to 1.90 (#3147) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-19 18:54:29 -04:00
Yan Ru Pei 5b19a39bab
feat: allow router to not track active blocks (prefill), and to not track cached blocks (decode) (#3135) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-19 20:33:07 +00:00
Graham King 3865a94148
feat: Port vllm port allocator to Rust in bindings (#3125) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-19 18:34:52 +00:00
Olga Andreeva 2d39f1b1bb
feat: KVBM connector : enabling vectorized copy from pinned memory to device memory and vice versa (#2989) 
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: oandreeva-nv <oandreeva-nv@nvidia.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: oandreeva-nv <oandreeva-nv@nvidia.com>
 2025-09-19 09:35:07 -07:00
Graham King b6595e2484
chore(bindings): Provide a binding to clear etcd namespace (#3094) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-18 11:49:08 -04:00
Yan Ru Pei 78a3fedab9
fix: hook up worker removals for indexer (#3095) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-17 21:33:25 +00:00
Graham King f88d7dc74b
chore(bindings): Remove NatsQueue (#3086) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-17 12:46:54 -04:00
Graham King 9060ce12ce
feat: Make part of discovery re-usable (#3073) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-17 10:43:24 -04:00
Tzu-Ling Kan 08cb08c1bc
feat: Canary Health Check. (#2903) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2025-09-17 03:31:56 +00:00
Graham King 723f2da74b
chore: Remove more extended Apache headers (#3063) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-16 15:48:51 -04:00
Graham King 87e6e0529d
fix: Interactive inputs actually stops, does not ignore stop token (#3057) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-16 14:39:05 -04:00
Ziqi Fan 55659eae70
fix: early stop if CPU or disk space not set when using KVBM (#2997) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-09-15 10:14:01 -07:00
blarson-b10 4000097653
feat: adds kv indexer metrics (#2905) 
Signed-off-by: Brian Larson <brian.larson@baseten.co>
 2025-09-10 21:33:15 +00:00
Graham King cb5a657a6a
fix: Load the tokenizer JSON once for chat and completions. (#2910) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-05 16:48:16 -04:00
Olga Andreeva 27fad26faf
refactor: Split ModelType to ModelInput for request and response type; ModelType for the supported workloads (#2714) 
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Guan Luo <gluo@nvidia.com>
Co-authored-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
 2025-09-03 22:22:37 +00:00
KrishnanPrash c920cbd9dc
feat: Add --custom-jinja-template argument to pass a custom chat template for vLLM (#2829) 
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
 2025-09-03 14:22:15 -07:00
Biswa Panda c6becbc859
feat: dynamo namespace isolation (#2394) 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
 2025-09-03 15:47:06 +00:00
Yan Ru Pei 383e3b3a52
feat: don't modify kv scheduler states on query + more python binding (#2798) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-02 17:09:43 -07:00
Ayush Agarwal 87a721a83e
feat: added parser name bindings (#2808) 
Signed-off-by: Ayush Agarwal <ayushag@nvidia.com>
 2025-09-02 20:37:29 +00:00
Jacky 6c539fbdac
feat: FT Request Cancellation feature and test for 0.5.0 (#2500) 2025-09-02 08:26:21 -07:00
Yan Ru Pei 7fabe7bfe2
fix: do not delete KV events jetstream (#2800) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-01 21:25:49 +00:00
Yan Ru Pei 488c87095c
feat: Router warm restarts via durable KV event consumers and radix snapshotting (#2756) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-08-30 23:42:57 +00:00
Richard Huo a68c2f8f12
feat: DIS-373 dynamo KVBM connector API integration with TRTLLM (#2544) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-08-29 19:27:15 -07:00
Keiven C 15539fd093
feat: add Prometheus metrics integration for KvStats (#2704) 
Signed-off-by: Keiven C <213854356+keivenchang@users.noreply.github.com>
 2025-08-28 22:12:22 -07:00
Yan Ru Pei f08729ae15
feat: python bindings for the entire KvPushRouter + per-request router configs (#2658) 2025-08-25 22:27:10 +00:00
Ziqi Fan b39382ba68
feat: add initial batch of KVBM metrics on match, offload and onboard (#2673) 2025-08-25 09:28:25 -07:00
Ayush Agarwal cbe854fc5f
feat: [vLLM] implement cli args for tool and reasoning parsers (#2619) 2025-08-22 20:29:18 +00:00
Ziqi Fan b658ba6139
feat: enable dynamo metrics on KVBM (#2626) 2025-08-22 19:58:05 +00:00
Graham King 6a358f7c8c
chore(llm): Rename protocols::Endpoint to EndpointId (#2615) 2025-08-22 15:07:34 +00:00
Michael Feil 174389e6d3
fix: Httpengine sync-enable-endpoint (#2591) 2025-08-21 18:29:52 -04:00
Tzu-Ling Kan 57728909cf
feat: Add model label for vllm backend metrics (#2474) 
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-08-21 17:32:34 +00:00
Michael Feil 626d7e182d
feat(request cancellation): pycontext, propagating the `is_stopped` into python land. (#2158) 2025-08-20 12:06:13 -07:00
Ryan Olson 07cfc3a11b
feat: kvbm + connector (#2258) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
Co-authored-by: Olga Andreeva <oandreeva@nvidia.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: John Thompson <jothomson@nvidia.com>
Co-authored-by: Richard Huo <rihuo@nvidia.com>
Co-authored-by: Zicheng Ma <zichengm@nvidia.com>
 2025-08-19 12:36:53 -07:00
suzu c5d9d26703
feat(frontend): support setting HTTP host via CLI (--http-host) (#2523) 2025-08-19 09:01:04 -04:00
Yan Ru Pei 85d8310806
feat: router-level request rejection (#2465) 2025-08-19 01:51:00 -07:00
Graham King a4bbe49228
feat(http): TLS support (#2492) 2025-08-18 16:06:29 -04:00
Keiven C acbdabc464
feat(metrics): add NATS client metrics to prometheus_metrics_fmt (#2292) 
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-08-14 19:33:53 -07:00
Tzu-Ling Kan 3a3f5bf275
feat: Add a "model" label to Component metrics (#2389) 2025-08-14 13:48:25 -05:00
Jorge António d0a6363584
feat: add RuntimeConfig to ModelEntry (#2311) 
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
 2025-08-14 10:19:53 -07:00
Graham King 72ec5f5c7b
feat: Allow an endpoint to serve multiple models (#2418) 2025-08-13 10:48:42 -04:00
Yan Ru Pei 5166a3dd44
feat: Router replicas with state-sharing (#2264) 2025-08-07 22:57:57 +00:00
Graham King 1954fcfa09
chore: Remove service_name from ModelDeploymentCard (#2349) 2025-08-07 11:02:41 -04:00
Graham King 6a1a801c2d
feat: Support static workers, run without etcd. (#2281) 2025-08-06 09:51:43 -04:00
Hongkuan Zhou 36c4ef5eb2
feat: migrate requests when planner shutdown decode engine (vllm) (#2280) 
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: hhzhang16 <54051230+hhzhang16@users.noreply.github.com>
 2025-08-05 12:24:07 -07:00
Jacky 347620a1ff
feat: Allow Python Engine to end stream before final (#2270) 2025-08-05 10:44:18 -07:00
Chi 433f60121a
feat: Pass user_data to register_llm for LoRA support (#2286) 2025-08-05 10:16:43 -04:00
Yan Ru Pei bae25dc6d4
feat: skip downloading model weights if using mocker (only tokenizer) (#2213) 2025-07-31 18:15:57 +00:00
Jacky 1f07dab7bd
feat: Add migration to LLM requests (#1930) 2025-07-18 20:04:20 +00:00
Graham King fc12436048
feat(frontend): router-mode settings (#2001) 2025-07-18 18:52:57 +00:00
Graham King 182d3b5dc7
chore(bindings): Remove mistralrs / llama.cpp (#1970) 2025-07-16 16:12:40 -04:00
Yan Ru Pei f31732a22d
feat: integrate mocker with dynamo-run and python cli (#1927) 2025-07-16 18:22:15 +00:00
Graham King 2bf27924a1
feat(python): Python bindings for the Dynamo CLI tools (#1799) 2025-07-08 21:40:36 +00:00
Yan Ru Pei 84e71e27d3
feat: predictive active blocks for routing without load metrics (#1731) 
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
 2025-07-08 00:18:22 -07:00
jain-ria 439e977d9c
feat: vllm speculative decoding metrics (#1549) 
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
 2025-07-07 23:40:43 +00:00
Jacky b4ddca99a0
feat: Failure Detection while Responses are returning (#1671) 2025-07-07 14:00:17 -07:00
Alec 0a32b3443f
fix: default to None initialization of routing config (#1713) 2025-07-01 10:55:05 -07:00
jthomson04 6365a015b3
fix: Fix main (#1712) 2025-06-30 22:08:06 -07:00
jthomson04 aaf283bbb8
feat: Approximate KV Routing (#1636) 2025-06-30 20:34:08 -07:00
Graham King 92f06b0e7f
chore(dynamo-run): Refactor to library (#1687) 
Move much of what was in the `dynamo-run` crate into `dynamo-llm` so that everyone can use it.

Example usage:

1. Create a `LocalModel`:

```
    let local_model = LocalModelBuilder::default()
	.model_path("Qwen/Qwen3-0.6B")
	.http_port(8080)
	.build().await?;
```

2. Make an engine:

```
    let engine_config = EngineConfig::StaticFull {
	engine: dynamo_engine_mistralrs::make_engine(&local_model).await?,
	model: Box::new(local_model),
    };
```

3. Connect it to an input and run it

```
    dynamo_llm::entrypoint::input::run_input(Input::Http, runtime, engine_config).await?;
```

For https://github.com/ai-dynamo/dynamo/issues/1647

Code Rabbit summary, thanks:
  * Introduced a flexible builder pattern for local model configuration, allowing advanced customization and easier initialization.
  * Added new input modes and unified input handling, supporting interactive chat, HTTP server, batch file, and distributed endpoint modes.
  * Centralized engine configuration and routing, enabling more extensible and maintainable engine management.
  * Simplified and modularized the codebase by moving input and engine logic into dedicated modules.
  * Replaced direct model construction with an asynchronous builder for improved clarity and extensibility.
  * Streamlined configuration and validation for flags and router settings.
  * Added validation to prevent incompatible input and output combinations in endpoint and dynamic modes.
 2025-06-30 21:06:24 +00:00
Yan Ru Pei 8392e7a190
feat: Unnormalize waiting requests + predictive load updates for Python router (mirroring Rust) + softmax sampling to reduce thrashing (#1638) 2025-06-27 09:01:59 +00:00
Yan Ru Pei 13a99b7f76
feat: Standalone Router (#1409) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: jain-ria <riajain@NVIDIA.com>
 2025-06-14 19:02:31 +00:00
Graham King 3f6a74723f
chore: Remove PreprocessedRequest alias BackendInput (#1307) 
It was confusing to have two names for one type.

This tidy up started in #1064 , is now complete.
 2025-06-02 11:22:27 -04:00
Alec 2f8da9ad1d
refactor: rename KvMetricsPublisher to WorkerMetricsPublisher (#1284) 2025-05-30 17:23:32 +00:00
jthomson04 9210a26d90
refactor: Refactor kv event publishers (#1287) 2025-05-30 09:04:28 -07:00
Alec f67dc38b28
fix: Renamed event publisher classes and configuration (#1273) 2025-05-29 13:00:06 -07:00
Jacky 7677f74f37
feat: KVBM async Python bindings and Layer class (#1141) 2025-05-29 17:49:05 +00:00
Alec 0df6d462d7
feat: add KV Event Publishing to vLLM v1 (#1181) 2025-05-29 07:56:49 -07:00
Graham King 0a1d1fbe3d
feat(dynamo-llm): Remove bring-your-own-engine (#1216) 
It was removed from the docs in 0.2.1 and replaced with writing a [standalone Python engine](https://github.com/ai-dynamo/dynamo/blob/main/docs/guides/dynamo_run.md#writing-your-own-engine-in-python).

Also remove the associated `dynamo-run` feature `python`.

Releasing this in 0.3.0 will resolve #784 and #1109.
 2025-05-28 18:52:27 +00:00
Graham King 183f2b3286
feat(dynamo-run): Allow setting KV cache block size (#1175) 
Example:
```
dynamo-run out=<engine> <model> --kv-cache-block-size 64
```

In a distributed system this goes on the worker node and is propagated to ingress via the model deployment card.

Previously hard coded to 16, which is now the default.

- Load context_length from model. Closes #1172
- Store context length and KV cache block size in Model Deployment Card #1170
 2025-05-22 14:55:36 -07:00
Graham King 3e8e38a942
fix(llmctl): Use ModelWatcher instead of direct etcd operations (#1150) 2025-05-21 19:06:56 +00:00
Ryan Olson 80256acf15
feat: adding outer dimension to isolate k/v blocks (#1126) 2025-05-20 08:51:41 -04:00
Graham King aeb79e6277
feat: Support multiple models on single ingress node (#1127) 
We can now do this:

- Node 1:

```
dynamo-run in=http out=dyn
```

- Node 2 and 3, two instances of component 'backend' in the nemotron_ultra pipeline:

```
dynamo-run in=dyn://nemotron_ultra.backend.generate out=vllm /data/models/NemotronUltra
```

- Node 4 and 5, two instances of the 'backend' component in nemotron_super pipeline:

```
dynamo-run in=dyn://nemotron_super.backend.generate out=vllm /data/models/NemotronSuper
```

The ingress node will discover all four instances and route correctly. We have been planning for this for a long time now.

As part of this auto-discovery is now always `out=dyn`, with no extra URL parts. Previously it could only route to a single pipeline.

Also:
- Refactor endpoint / instance naming now that I understand them
- Fix removing models when their instance stops.
 2025-05-19 18:41:07 -04:00
Jacky 437cae0ad0
feat: KV Block Manager Python bindings (#1022) 2025-05-19 12:33:41 -04:00
Tom O'Brien 73fdfb8ab8
feat: Add OpenAI Embeddings interface in rust lib (#1110) 
Implements OpenAI embeddings (interface only).

- Adds ModelType::Embedding
- Adds OpenAI embedding request/response structs
- Adds support for embedding model discovery
 2025-05-19 11:59:46 -04:00
Graham King 2981350880
feat(dynamo-run): KV-aware routing (#1064) 
Router:
```
dynamo-run in=http out=dyn://dynamo.endpoint.generate --router-mode kv
```

Worker (* N):
```
dynamo-run in=dyn://dynamo.endpoint.generate out=vllm /data/llms/Qwen/Qwen3-4B
```

You need patched vllm and the C bindings `.so`. Full docs in the updated guide: `docs/guides/dynamo_run.md`.

This gives us a pure-Rust ingress node: OpenAI compliant HTTP server + Pre-processor + KV-aware router.
 2025-05-14 15:21:46 -04:00
Hongkuan Zhou 466b8e5fb7
refactor: use primary lease + self-contained graceful shutdown trigged by SIGINT/SIGTERM (#1001) 2025-05-08 16:53:39 -07:00
Hongkuan Zhou a590d10317
feat: cleanup EtcdKvCache and PrefillQueue before and after launch (#925) 2025-05-07 16:35:55 -07:00
Graham King 92bbbc3923
fix: Fix vllm/sglang engine model name if using HF repo (#986) 
Signed-off-by: Graham King <graham@gkgk.org>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-05-07 23:20:51 +00:00
jthomson04 c4213899fe
feat: Migrate NATS Queue to Rust (#669) (#961) 2025-05-06 15:36:55 -07:00
Graham King 99cd9d85a9
feat: dynamo-run <-> python interop (#934) 
Adding this to a Python script makes it register on the network so that `dynamo-run` can discover it and send it requests:
```
from dynamo.llm import register_llm

MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
await register_llm(endpoint, MODEL, 3)
```

Full vllm example, with pre-processing in dynamo:
- `dynamo-run in=text out=dyn://dynamo.backend.generate`
- `cd lib/bindings/python/examples/hello_world`
- `python server_vllm.py`

This builds on top of the work to move pre-processor to ingress side. It means we can decouple Rust and Python using NATS as the bus.

The `register_llm` call does this:

- Download the model from HF if necessary
- Load the model deployment card from the HF folder or extract from GGUF
- Push the tokenizer config etc into NATS object store so ingress can access it from a different machine
- Publish the model deployment card to ETCD
 2025-05-05 20:49:19 -04:00
Hongkuan Zhou 9d643f1eed
fix: use primary lease for NixlMetadataStore (#928) 2025-05-05 08:38:52 -07:00