b94ecd1627
feat: add disk offloading filtering in KVBM ( #3532 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-10-10 09:51:55 -07:00
e9a710095b
chore: Remove clear_namespace script ( #3519 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-09 13:43:59 -04:00
ef04a0869e
feat: Add OpenAPI routes to serve HTTP server API documentation ( #3480 )
2025-10-08 13:52:19 -07:00
9975cb9ffe
feat: enable KVBM to support PD disagg in Dynamo vLLM ( #3352 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
2025-10-08 13:45:36 -07:00
0aa0768f53
chore: Remove llama.cpp engine ( #3499 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-08 16:44:53 -04:00
cf83794ae0
feat: DIS-678 kvbm modularity: standalone metrics endpoint ( #3433 )
...
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
2025-10-08 10:18:33 -07:00
1b1265e6fc
chore: Remove GGUF support ( #3488 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-08 11:42:24 -04:00
0c4c4d1de3
feat: add Python MetricsRegistry Python metrics registration ( #3341 )
...
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
2025-10-07 21:24:25 -07:00
bdad6f1a50
fix: Make planner VirtualConnectorClient also use v1/ prefix. ( #3468 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-07 22:34:30 +00:00
68bbe8a9a5
feat: Connect DYN_LOG level to TLLM_LOG_LEVEL ( #3451 )
...
Signed-off-by: Tanmay Verma <tanmay2592@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-10-07 21:02:35 +00:00
a5371bfc50
feat(etcd): Version the etcd keys ( #3458 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-07 16:48:26 +00:00
81162dfeb9
chore(discovery): Watch/publish ModelDeploymentCard instead of ModelEntry ( #3350 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-07 10:02:02 -04:00
30610e7371
feat: use KvPushRouter for prefill router ( #3401 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-03 18:21:48 -07:00
d2e3b66e65
feat: Transition to FullyContiguous Host and Disk layouts ( #3090 )
...
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: oandreeva-nv <oandreeva-nv@nvidia.com>
2025-10-01 16:26:39 -07:00
713e9e481b
fix: DIS-706 skip offloading the G1 matched blocks during offloading ( #3299 )
...
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
2025-10-01 10:28:06 -07:00
9b9536d0d7
feat: make prefill router general ( #3329 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-30 20:38:47 -07:00
f4a3a6b66a
refactor: standardize Prometheus metric naming conventions (part 1) ( #3035 )
...
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
2025-09-30 19:01:39 -07:00
af4c0cadcd
feat: add Context class to Python type stubs ( #3327 )
...
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
2025-09-30 17:10:02 -07:00
5b457b70a6
feat: python add abi compatability for cross-platform builds + add a unit test to HttpServer ( #3044 )
...
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Signed-off-by: Michael Feil <63565275+michaelfeil@users.noreply.github.com>
Signed-off-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Signed-off-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
2025-09-30 14:42:42 -07:00
d354763c40
fix: python bindings for router should register to etcd as well ( #3302 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-30 12:38:56 -07:00
cacac9b9f4
feat: add Python const for Prometheus metric names ( #3244 )
...
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
2025-09-30 10:28:47 -07:00
3aa3077808
fix: more fixes for stable router benchmarking ( #3264 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-29 23:08:56 +00:00
e21dcf6cab
feat: enable KVBM emit metrics in Dynamo TRTLLM ( #3254 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
2025-09-29 08:48:51 -07:00
67e1f6ee11
feat: enable parallel tool calling and add testing ( #3188 )
...
Signed-off-by: Elyas Mehtabuddin <emehtabuddin@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-09-29 15:30:42 +00:00
7ebbd001c0
chore: Remove etcd from python bindings ( #3238 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-26 12:58:30 -04:00
5bb7490448
chore: bump vllm version to 0.10.2 ( #3180 )
...
Signed-off-by: Alec <aflowers@nvidia.com>
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Co-authored-by: Kris Hung <krish@nvidia.com>
2025-09-26 04:01:57 +00:00
f2e2e935b2
feat: Add distributed tracing context support to Python bindings ( #3160 )
...
Signed-off-by: nnshah1 <neelays@nvidia.com>
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
2025-09-25 17:26:35 -07:00
c03e2f6bbe
chore: Migrate planner virtual_connector internals into bindings ( #3205 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-25 18:04:58 -04:00
980727bba1
chore: bump versions ahead of 0.5.1 release ( #3209 )
...
Signed-off-by: Harrison Saturley-Hall <hsaturleyhal@nvidia.com>
2025-09-24 18:26:38 -04:00
6ba64c31f5
feat: tensor type for generic inference. ( #2746 )
...
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-09-24 07:31:09 +00:00
c63cceaa77
feat: JailedStream ( #3034 )
...
Signed-off-by: ayushag <ayushag@nvidia.com>
2025-09-23 22:24:17 +00:00
031590fc14
feat: vllm prefill router ( #3155 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-22 16:38:56 -07:00
31c78df781
refactor: PyTest subprocess for cancellation unit tests ( #3127 )
...
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: michaelfeil <me@michaelfeil.eu>
2025-09-22 10:56:37 -07:00
7a5a0bd6cd
chore: Upgrade Rust to 1.90 ( #3147 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-19 18:54:29 -04:00
5b19a39bab
feat: allow router to not track active blocks (prefill), and to not track cached blocks (decode) ( #3135 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-19 20:33:07 +00:00
3865a94148
feat: Port vllm port allocator to Rust in bindings ( #3125 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-19 18:34:52 +00:00
2d39f1b1bb
feat: KVBM connector : enabling vectorized copy from pinned memory to device memory and vice versa ( #2989 )
...
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: oandreeva-nv <oandreeva-nv@nvidia.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: oandreeva-nv <oandreeva-nv@nvidia.com>
2025-09-19 09:35:07 -07:00
a8fd127130
feat: Request Cancellation unary request support ( #3004 )
...
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
2025-09-18 22:26:37 -07:00
31f5ed3ce7
feat: add finish reason = tool_calls for stream=False and phi-4 detect token start fix ( #3087 )
...
Signed-off-by: Elyas Mehtabuddin <emehtabuddin@nvidia.com>
2025-09-18 21:45:52 +00:00
b6595e2484
chore(bindings): Provide a binding to clear etcd namespace ( #3094 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-18 11:49:08 -04:00
86ce03f428
fix: Update the KVBM <> TRT-LLM integration interface to match the latest TRT-LLM connector API ( #2979 )
...
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
2025-09-17 15:33:13 -07:00
78a3fedab9
fix: hook up worker removals for indexer ( #3095 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-17 21:33:25 +00:00
f88d7dc74b
chore(bindings): Remove NatsQueue ( #3086 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-17 12:46:54 -04:00
9060ce12ce
feat: Make part of discovery re-usable ( #3073 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-17 10:43:24 -04:00
08cb08c1bc
feat: Canary Health Check. ( #2903 )
...
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
2025-09-17 03:31:56 +00:00
960dc8967e
fix: replace hard coded dynamo namespace with env var ( #3048 )
...
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
2025-09-16 14:56:51 -07:00
723f2da74b
chore: Remove more extended Apache headers ( #3063 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-16 15:48:51 -04:00
87e6e0529d
fix: Interactive inputs actually stops, does not ignore stop token ( #3057 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-16 14:39:05 -04:00
435803ea16
chore: add additional param support for multimodal models ( #3042 )
...
Signed-off-by: Ryan Lempka <rlempka@nvidia.com>
2025-09-16 16:26:39 +00:00
55659eae70
fix: early stop if CPU or disk space not set when using KVBM ( #2997 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
2025-09-15 10:14:01 -07:00
bd835b03b6
fix: change KVBM Connector build_connector_meta due to vLLM ToT change ( #2887 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
2025-09-15 16:22:19 +00:00
c8ecc4025f
ci: fix completions race ( #3010 )
...
Signed-off-by: alec-flowers <aflowers@nvidia.com>
2025-09-11 19:46:59 +00:00
4000097653
feat: adds kv indexer metrics ( #2905 )
...
Signed-off-by: Brian Larson <brian.larson@baseten.co>
2025-09-10 21:33:15 +00:00
de5ab2c4af
docs: Python runtime Request Cancellation examples ( #2893 )
...
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
2025-09-10 19:23:30 +00:00
6f14e94159
feat: Add a checksum to ModelDeploymentCard fields ( #2934 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-09 16:12:42 +00:00
cb5a657a6a
fix: Load the tokenizer JSON once for chat and completions. ( #2910 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-05 16:48:16 -04:00
27fad26faf
refactor: Split ModelType to ModelInput for request and response type; ModelType for the supported workloads ( #2714 )
...
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Guan Luo <gluo@nvidia.com>
Co-authored-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
2025-09-03 22:22:37 +00:00
c920cbd9dc
feat: Add --custom-jinja-template argument to pass a custom chat template for vLLM ( #2829 )
...
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
2025-09-03 14:22:15 -07:00
c6becbc859
feat: dynamo namespace isolation ( #2394 )
...
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
2025-09-03 15:47:06 +00:00
383e3b3a52
feat: don't modify kv scheduler states on query + more python binding ( #2798 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-02 17:09:43 -07:00
87a721a83e
feat: added parser name bindings ( #2808 )
...
Signed-off-by: Ayush Agarwal <ayushag@nvidia.com>
2025-09-02 20:37:29 +00:00
561ecb98a2
chore: bump version numbers ahead of 0.5.0 release ( #2812 )
...
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
2025-09-02 16:13:28 -04:00
6c539fbdac
feat: FT Request Cancellation feature and test for 0.5.0 ( #2500 )
2025-09-02 08:26:21 -07:00
7fabe7bfe2
fix: do not delete KV events jetstream ( #2800 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-01 21:25:49 +00:00
488c87095c
feat: Router warm restarts via durable KV event consumers and radix snapshotting ( #2756 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-08-30 23:42:57 +00:00
a68c2f8f12
feat: DIS-373 dynamo KVBM connector API integration with TRTLLM ( #2544 )
...
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
2025-08-29 19:27:15 -07:00
43a26958d9
feat: add logits processor support for trtllm backend ( #2702 )
...
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-08-30 01:06:03 +00:00
15539fd093
feat: add Prometheus metrics integration for KvStats ( #2704 )
...
Signed-off-by: Keiven C <213854356+keivenchang@users.noreply.github.com>
2025-08-28 22:12:22 -07:00
95ce83d59a
feat: Integrate Model Express Client into Dynamo Model Downloads ( #2574 )
...
Signed-off-by: Kavin Krishnan <kavink@nvidia.com>
Co-authored-by: KavinKrishnan <kavin.krishnan@nvidia.com>
2025-08-28 16:09:04 -07:00
91a459c038
feat: KServe gRPC support ( #2638 )
2025-08-26 22:57:31 -07:00
ef535edb98
feat: Trtllm metric_labels. ( #2666 )
2025-08-27 05:50:05 +00:00
f08729ae15
feat: python bindings for the entire KvPushRouter + per-request router configs ( #2658 )
2025-08-25 22:27:10 +00:00
3036e60b1e
feat: add gpt oss reasoning parser through harmony ( #2656 )
...
- couple of refactors
- added a new dependency, openai-harmony
- implemented the gpt oss parser
2025-08-25 17:13:38 +00:00
b39382ba68
feat: add initial batch of KVBM metrics on match, offload and onboard ( #2673 )
2025-08-25 09:28:25 -07:00
9b9f2ce4f6
fix: pytest robustness and parsing error ( #2676 )
2025-08-24 10:13:41 -07:00
cbe854fc5f
feat: [vLLM] implement cli args for tool and reasoning parsers ( #2619 )
2025-08-22 20:29:18 +00:00
b658ba6139
feat: enable dynamo metrics on KVBM ( #2626 )
2025-08-22 19:58:05 +00:00
b92a805edb
feat: add BaseLogitsProcessor core interface ( #2613 )
...
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
2025-08-22 13:41:30 -04:00
6a358f7c8c
chore(llm): Rename protocols::Endpoint to EndpointId ( #2615 )
2025-08-22 15:07:34 +00:00
174389e6d3
fix: Httpengine sync-enable-endpoint ( #2591 )
2025-08-21 18:29:52 -04:00
57728909cf
feat: Add model label for vllm backend metrics ( #2474 )
...
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
2025-08-21 17:32:34 +00:00
626d7e182d
feat(request cancellation): pycontext, propagating the `is_stopped` into python land. ( #2158 )
2025-08-20 12:06:13 -07:00
49958435eb
chore: Remove async-openai-macros ( #2554 )
2025-08-20 11:33:10 -07:00
d319abf3a3
feat: upload/download rust structs directly through NATs object store ( #2540 )
2025-08-20 17:17:21 +00:00
9a02188531
chore: Bumped Dynamo version to 0.4.1 ( #2545 )
2025-08-19 21:55:37 -04:00
177d662f86
fix: Dockerfile.sglang - Fixed sglang and dynamo wheels installaiton in ru… ( #2537 )
2025-08-19 20:56:12 -04:00
199b9a30f4
chore: Bring async-openai into repo as request starter ( #2520 )
...
Co-authored-by: Graham King <grahamk@nvidia.com>
2025-08-19 17:47:01 -04:00
07cfc3a11b
feat: kvbm + connector ( #2258 )
...
Signed-off-by: Ryan Olson <rolson@nvidia.com>
Co-authored-by: Olga Andreeva <oandreeva@nvidia.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: John Thompson <jothomson@nvidia.com>
Co-authored-by: Richard Huo <rihuo@nvidia.com>
Co-authored-by: Zicheng Ma <zichengm@nvidia.com>
2025-08-19 12:36:53 -07:00
a33033b7f6
feat: task scheduler ( #2406 )
...
Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com>
2025-08-19 12:35:31 -06:00
c5d9d26703
feat(frontend): support setting HTTP host via CLI (--http-host) ( #2523 )
2025-08-19 09:01:04 -04:00
85d8310806
feat: router-level request rejection ( #2465 )
2025-08-19 01:51:00 -07:00
a4bbe49228
feat(http): TLS support ( #2492 )
2025-08-18 16:06:29 -04:00
0444217339
fix: replace metrics callback with background scraping to prevent tim… ( #2480 )
...
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
2025-08-18 12:38:50 -07:00
ffae72b7d2
fix: remove kvmanager feature from python 3.12 ai-dynamo-runtime wheel ( #2456 )
2025-08-15 16:43:55 -04:00
acbdabc464
feat(metrics): add NATS client metrics to prometheus_metrics_fmt ( #2292 )
...
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
2025-08-14 19:33:53 -07:00
3a3f5bf275
feat: Add a "model" label to Component metrics ( #2389 )
2025-08-14 13:48:25 -05:00
d0a6363584
feat: add RuntimeConfig to ModelEntry ( #2311 )
...
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
2025-08-14 10:19:53 -07:00
c12c25787f
fix: upgrade cudarc to 0.17.1 ( #2341 )
...
Signed-off-by: Dan Aloni <dan.aloni@vastdata.com>
Co-authored-by: Tushar Sharma <tusharma@nvidia.com>
2025-08-13 15:45:08 -04:00
72ec5f5c7b
feat: Allow an endpoint to serve multiple models ( #2418 )
2025-08-13 10:48:42 -04:00
9b87c89c41
feat: multi-modal example with vLLM v1 and UX v2 ( #2040 )
...
Co-authored-by: krishung5 <krish@nvidia.com>
2025-08-12 23:55:31 +00:00
5166a3dd44
feat: Router replicas with state-sharing ( #2264 )
2025-08-07 22:57:57 +00:00
bd4fe1a7d1
feat: cross process instrumentation ( #2243 )
...
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
2025-08-07 15:32:48 +00:00
1954fcfa09
chore: Remove service_name from ModelDeploymentCard ( #2349 )
2025-08-07 11:02:41 -04:00
dbe48a1d2b
chore: Bump mistral.rs, llama.cpp and tokenizers deps ( #2338 )
2025-08-06 16:23:54 -04:00
b2aa504b47
fix: upgrade axum to 0.8 and etcd-client to 0.16 ( #2317 )
...
Signed-off-by: Dan Aloni <dan.aloni@vastdata.com>
2025-08-06 18:25:49 +00:00
6a1a801c2d
feat: Support static workers, run without etcd. ( #2281 )
2025-08-06 09:51:43 -04:00
36c4ef5eb2
feat: migrate requests when planner shutdown decode engine (vllm) ( #2280 )
...
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: hhzhang16 <54051230+hhzhang16@users.noreply.github.com>
2025-08-05 12:24:07 -07:00
347620a1ff
feat: Allow Python Engine to end stream before final ( #2270 )
2025-08-05 10:44:18 -07:00
433f60121a
feat: Pass user_data to register_llm for LoRA support ( #2286 )
2025-08-05 10:16:43 -04:00
4cbd4f3881
fix: show raw sglang logs given env var ( #2287 )
2025-08-05 06:11:44 +00:00
625578c3bb
chore: update nixl version to 0.4.1 ( #2221 )
2025-07-31 17:46:21 -04:00
bae25dc6d4
feat: skip downloading model weights if using mocker (only tokenizer) ( #2213 )
2025-07-31 18:15:57 +00:00
4c90b1b924
chore: Version bump to 0.4.0 ( #2179 )
2025-07-30 02:07:31 +00:00
222245e2a0
refactor: Move engine and publisher from dynamo.llm.tensorrt_llm to dynamo.trtllm ( #2128 )
2025-07-26 02:23:17 +00:00
19a77ae7d5
chore(dynamo-run): Remove out=sglang|vllm|trtllm ( #1920 )
2025-07-23 17:22:31 -07:00
66b7d2c75d
fix: updates versions and adds ahashmap to BPE ( #2072 )
2025-07-23 20:27:18 +00:00
e0a51940d1
feat: Connect Library ( #1478 )
2025-07-23 14:16:34 -04:00
9f2356cb9e
chore: Remove unused portion of kv bindings test ( #2052 )
2025-07-22 19:12:42 +00:00
1f07dab7bd
feat: Add migration to LLM requests ( #1930 )
2025-07-18 20:04:20 +00:00
fc12436048
feat(frontend): router-mode settings ( #2001 )
2025-07-18 18:52:57 +00:00
182d3b5dc7
chore(bindings): Remove mistralrs / llama.cpp ( #1970 )
2025-07-16 16:12:40 -04:00
f31732a22d
feat: integrate mocker with dynamo-run and python cli ( #1927 )
2025-07-16 18:22:15 +00:00
aba60996f2
perf(router): Remove lock from router hot path ( #1963 )
2025-07-16 12:26:32 -04:00
7b9182fdb9
chore: Move examples/cli to lib/bindings/examples/cli ( #1952 )
2025-07-15 16:33:07 -04:00
d975761b8b
chore: update nixl to 0.4.0 release ( #1860 ) ( #1886 )
2025-07-11 12:45:44 -04:00
1704b126f3
build: Revert "chore: update nixl to 0.4.0 release" ( #1880 )
2025-07-10 23:30:28 +00:00
f242b4552b
refactor: vllm v1 examples ( #1756 )
...
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Co-authored-by: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com>
Co-authored-by: Hongkuan Zhou <tedzhouhk@gmail.com>
2025-07-10 22:30:16 +00:00
5fa4cddab9
chore: update nixl to 0.4.0 release ( #1860 )
2025-07-10 17:08:34 -04:00
ece76a6201
feat: Build DistributedRuntime-level HTTP server with /health /metrics ( #1656 )
2025-07-08 22:49:28 +00:00
2bf27924a1
feat(python): Python bindings for the Dynamo CLI tools ( #1799 )
2025-07-08 21:40:36 +00:00
84e71e27d3
feat: predictive active blocks for routing without load metrics ( #1731 )
...
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
2025-07-08 00:18:22 -07:00
439e977d9c
feat: vllm speculative decoding metrics ( #1549 )
...
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
2025-07-07 23:40:43 +00:00
c4935b3497
chore: update versions for 0.3.2 release ( #1793 )
2025-07-07 17:21:02 -04:00
b4ddca99a0
feat: Failure Detection while Responses are returning ( #1671 )
2025-07-07 14:00:17 -07:00
a9241b61f0
chore: update nixl to latest 0.3.1 commit ( #1762 )
2025-07-03 15:31:47 -07:00
4ab476174a
chore(engines): Upgrade mistralrs to 0.6.0 ( #1767 )
2025-07-03 12:45:58 -07:00
0a32b3443f
fix: default to None initialization of routing config ( #1713 )
2025-07-01 10:55:05 -07:00
dfbd741de2
feat: Support for Responses API ( #1694 )
2025-07-01 12:08:15 -04:00
edf00c5c71
fix(bindings): Default router config in bindings ( #1716 )
...
* Added a default temperature value for text generation requests when no temperature is specified.
* Improved handling of missing configuration values to prevent errors during model initialization.
2025-07-01 14:53:32 +00:00
6365a015b3
fix: Fix main ( #1712 )
2025-06-30 22:08:06 -07:00
aaf283bbb8
feat: Approximate KV Routing ( #1636 )
2025-06-30 20:34:08 -07:00
92f06b0e7f
chore(dynamo-run): Refactor to library ( #1687 )
...
Move much of what was in the `dynamo-run` crate into `dynamo-llm` so that everyone can use it.
Example usage:
1. Create a `LocalModel`:
```
let local_model = LocalModelBuilder::default()
.model_path("Qwen/Qwen3-0.6B")
.http_port(8080)
.build().await?;
```
2. Make an engine:
```
let engine_config = EngineConfig::StaticFull {
engine: dynamo_engine_mistralrs::make_engine(&local_model).await?,
model: Box::new(local_model),
};
```
3. Connect it to an input and run it
```
dynamo_llm::entrypoint::input::run_input(Input::Http, runtime, engine_config).await?;
```
For https://github.com/ai-dynamo/dynamo/issues/1647
Code Rabbit summary, thanks:
* Introduced a flexible builder pattern for local model configuration, allowing advanced customization and easier initialization.
* Added new input modes and unified input handling, supporting interactive chat, HTTP server, batch file, and distributed endpoint modes.
* Centralized engine configuration and routing, enabling more extensible and maintainable engine management.
* Simplified and modularized the codebase by moving input and engine logic into dedicated modules.
* Replaced direct model construction with an asynchronous builder for improved clarity and extensibility.
* Streamlined configuration and validation for flags and router settings.
* Added validation to prevent incompatible input and output combinations in endpoint and dynamic modes.
2025-06-30 21:06:24 +00:00
82eae1fdf5
refactor: Upgrade async-openai ( #1693 )
2025-06-30 13:35:10 -04:00
8392e7a190
feat: Unnormalize waiting requests + predictive load updates for Python router (mirroring Rust) + softmax sampling to reduce thrashing ( #1638 )
2025-06-27 09:01:59 +00:00
250ed7334b
fix: Fix NIXL 0.3.1 build ( #1561 )
2025-06-17 14:30:10 -07:00
13a99b7f76
feat: Standalone Router ( #1409 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: jain-ria <riajain@NVIDIA.com>
2025-06-14 19:02:31 +00:00
99e67e607a
chore: update dynamo and nixl versions for 0.3.1 ( #1517 )
2025-06-13 21:13:43 +00:00
0e7d4d8298
docs: DIS-133 and DIS-134 plus copyediting ( #1439 )
...
Signed-off-by: Kristen Kelleher <kkelleher@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-06-12 12:10:01 -07:00
75d7c3b933
fix: remove unused bentoml references ( #1412 )
2025-06-09 17:32:02 -07:00
e61f1c8a40
chore: Remove nats-py dependency ( #1387 )
2025-06-05 15:50:36 -07:00
568eb1009e
fix: Use Rust Ingress (dynamo-run) for the Frontend ( #1391 )
2025-06-04 22:27:05 -07:00
5e9370d3aa
docs: fix sphinx errors admonitions adobe config ( #1179 )
...
Signed-off-by: Kristen Kelleher <kkelleher@nvidia.com>
- Content, format, and structural changes to the Dynamo docs for 0.3.0.
- Includes copyediting and the first batch of changes from the DMO review.
2025-06-04 08:16:34 -07:00
3f6a74723f
chore: Remove PreprocessedRequest alias BackendInput ( #1307 )
...
It was confusing to have two names for one type.
This tidy up started in #1064 , is now complete.
2025-06-02 11:22:27 -04:00
2f8da9ad1d
refactor: rename KvMetricsPublisher to WorkerMetricsPublisher ( #1284 )
2025-05-30 17:23:32 +00:00
9210a26d90
refactor: Refactor kv event publishers ( #1287 )
2025-05-30 09:04:28 -07:00
f67dc38b28
fix: Renamed event publisher classes and configuration ( #1273 )
2025-05-29 13:00:06 -07:00
7d0c938690
feat: Initial Granite support ( #1271 )
...
- Add Granite to our tokenizer
- Fix pre-processor to load context length correctly
- Add strftime_now Jinja function for prompt templates
- Update llama.cpp
- Handle trtllm errors when not using trtllm
Support depends on the engine:
- `mistral.rs`, our default engine, doesn't support Granite yet.
- `llama.cpp` does and works very well:
```
dynamo-run out=llamacpp ~/llms/granite-3.3-2b-instruct-Q4_K_M.gguf --context-length 16384
```
- `vllm` also works very well:
```
dynamo-run in=http out=vllm ~/llms/granite-3.3-2b-instruct --context-length 16384
```
- `sglang` mostly works, but it doesn't catch the stop token, so we do in the HTTP ingress, and log an error. The Text ingress doesn't catch it because I disabled it to make the raw echo engine work. A bit of work to do here.
Closes : #1245
2025-05-29 15:25:07 -04:00
7677f74f37
feat: KVBM async Python bindings and Layer class ( #1141 )
2025-05-29 17:49:05 +00:00
9d9a1d9b74
chore: update dynamo and nixl versions for 0.3.0 ( #1240 )
2025-05-29 17:30:17 +00:00
0df6d462d7
feat: add KV Event Publishing to vLLM v1 ( #1181 )
2025-05-29 07:56:49 -07:00
0a1d1fbe3d
feat(dynamo-llm): Remove bring-your-own-engine ( #1216 )
...
It was removed from the docs in 0.2.1 and replaced with writing a [standalone Python engine](https://github.com/ai-dynamo/dynamo/blob/main/docs/guides/dynamo_run.md#writing-your-own-engine-in-python ).
Also remove the associated `dynamo-run` feature `python`.
Releasing this in 0.3.0 will resolve #784 and #1109 .
2025-05-28 18:52:27 +00:00
1b1e089a45
feat: Enable dynamo-run out=trtllm ( #1223 )
2025-05-28 10:59:41 -07:00
cc40af704b
fix: dynamo-run pass proper args using register-llm ( #1230 )
2025-05-28 09:21:53 -07:00
31ff23704b
feat: adding arena allocator for storage objects ( #1178 )
2025-05-23 08:41:42 -06:00
183f2b3286
feat(dynamo-run): Allow setting KV cache block size ( #1175 )
...
Example:
```
dynamo-run out=<engine> <model> --kv-cache-block-size 64
```
In a distributed system this goes on the worker node and is propagated to ingress via the model deployment card.
Previously hard coded to 16, which is now the default.
- Load context_length from model. Closes #1172
- Store context length and KV cache block size in Model Deployment Card #1170
2025-05-22 14:55:36 -07:00
f992a6a2ba
docs: Fix broken link in python bindings documentation ( #1163 )
...
Co-authored-by: Suman Tatiraju <statiraju@statiraju-mlt.client.nvidia.com>
2025-05-22 03:13:23 +00:00
3e8e38a942
fix(llmctl): Use ModelWatcher instead of direct etcd operations ( #1150 )
2025-05-21 19:06:56 +00:00
8d636ebdbe
docs: Add sphinx-theme based userguides ( #528 )
...
Signed-off-by: Suman Tatiraju <167138127+statiraju@users.noreply.github.com>
Signed-off-by: Anant Sharma <anants@nvidia.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>
Co-authored-by: Dmitry Tokarev <dtokarev@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: Kristen Kelleher <kkelleher@nvidia.com>
Co-authored-by: Suman Tatiraju <statiraju@statiraju-mlt.client.nvidia.com>
Co-authored-by: Hannah Zhang <hannahz@nvidia.com>
2025-05-21 22:41:34 +08:00
80256acf15
feat: adding outer dimension to isolate k/v blocks ( #1126 )
2025-05-20 08:51:41 -04:00
7e452a2e22
fix: Disable block manager by default in Python bindings ( #1128 )
2025-05-19 16:28:12 -07:00
aeb79e6277
feat: Support multiple models on single ingress node ( #1127 )
...
We can now do this:
- Node 1:
```
dynamo-run in=http out=dyn
```
- Node 2 and 3, two instances of component 'backend' in the nemotron_ultra pipeline:
```
dynamo-run in=dyn://nemotron_ultra.backend.generate out=vllm /data/models/NemotronUltra
```
- Node 4 and 5, two instances of the 'backend' component in nemotron_super pipeline:
```
dynamo-run in=dyn://nemotron_super.backend.generate out=vllm /data/models/NemotronSuper
```
The ingress node will discover all four instances and route correctly. We have been planning for this for a long time now.
As part of this auto-discovery is now always `out=dyn`, with no extra URL parts. Previously it could only route to a single pipeline.
Also:
- Refactor endpoint / instance naming now that I understand them
- Fix removing models when their instance stops.
2025-05-19 18:41:07 -04:00
74221fd716
feat: Add support for SSD offloading in block manager ( #1115 )
2025-05-19 15:35:00 -07:00
437cae0ad0
feat: KV Block Manager Python bindings ( #1022 )
2025-05-19 12:33:41 -04:00
73fdfb8ab8
feat: Add OpenAI Embeddings interface in rust lib ( #1110 )
...
Implements OpenAI embeddings (interface only).
- Adds ModelType::Embedding
- Adds OpenAI embedding request/response structs
- Adds support for embedding model discovery
2025-05-19 11:59:46 -04:00
34f3fc6d12
test: Add doc tests to Rust CI ( #1102 )
2025-05-16 07:56:52 -07:00
2981350880
feat(dynamo-run): KV-aware routing ( #1064 )
...
Router:
```
dynamo-run in=http out=dyn://dynamo.endpoint.generate --router-mode kv
```
Worker (* N):
```
dynamo-run in=dyn://dynamo.endpoint.generate out=vllm /data/llms/Qwen/Qwen3-4B
```
You need patched vllm and the C bindings `.so`. Full docs in the updated guide: `docs/guides/dynamo_run.md`.
This gives us a pure-Rust ingress node: OpenAI compliant HTTP server + Pre-processor + KV-aware router.
2025-05-14 15:21:46 -04:00
4564a38711
feat: kv block manager ( #965 )
2025-05-09 21:05:24 +00:00
24e2cbf516
docs: Example Chat sglang engine ( #1015 )
...
Example of how to connect a Python sglang engine to the message bus (NATS/etc). I
In this example sglang does the pre/post processing. There is already an example where Dynamo does it.
The examples teach this:
- Be a chat completions engine, do your own pre-processing:
```
await register_llm(ModelType.Chat, endpoint, config.model)
```
- Have Dynamo do pre-processing. It will register us under both Chat and Completions endpoints, because that's handled before a Backend engine gets the request:
```
await register_llm(ModelType.Backend, endpoint, config.model)
```
2025-05-09 20:45:03 +00:00
c7bb1e8346
fix(bindings): serve_endpoint no longer takes a lease ( #1014 )
2025-05-09 16:18:19 +00:00
e9cb035ac7
chore: bump versions and NIXL dependencies for 0.2.1 ( #1012 )
2025-05-09 15:35:01 +00:00
b2e401bcae
feat: allow adding auth to etcd ( #980 )
...
Allow both password or TLS auth, if none of these is provided fallback to no auth
Closes #657
2025-05-09 08:36:41 -04:00
5d5235bc51
feat(sglang): aggregated support ( #937 )
...
Co-authored-by: ishandhanani <ishandhananai@gmail.com>
2025-05-09 06:01:40 +00:00
466b8e5fb7
refactor: use primary lease + self-contained graceful shutdown trigged by SIGINT/SIGTERM ( #1001 )
2025-05-08 16:53:39 -07:00
ceaeba3e38
feat: Qwen3, Gemma3 and Llama4 support ( #1002 )
...
. New mistralrs and llamacpp version
. mistralrs: Handle Gemma 3 and Llama 4 as vision models
. Update the dynamo-run docs to use Qwen 3
. Our pre-processor now supports Llama 4's newer multi-modal `config.json`
. Upgrade minijinja to handle Qwen 3's prompt template
For Llama 4 we'll need to limit the max seq len. vllm says:
> To serve at least one request with the models's max seq len (10485760), (240.00 GiB KV cache is needed,...
I was able to run Llama 4 with llamacpp and a quantized GGUF, with Dynamo doing the pre-processing.
2025-05-08 18:18:40 -04:00
a590d10317
feat: cleanup EtcdKvCache and PrefillQueue before and after launch ( #925 )
2025-05-07 16:35:55 -07:00
92bbbc3923
fix: Fix vllm/sglang engine model name if using HF repo ( #986 )
...
Signed-off-by: Graham King <graham@gkgk.org>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-05-07 23:20:51 +00:00
c4213899fe
feat: Migrate NATS Queue to Rust ( #669 ) ( #961 )
2025-05-06 15:36:55 -07:00
28fd481c9a
feat(dynamo-run): vllm and sglang subprocess engines ( #954 )
...
New vllm and sglang engines that run in a sub-process. Will hopefully replace the existing embedded python engines.
Why?
- Pure Python, does not require knowing Rust to work on it. Much simpler to maintain.
- No embedded Python interpreter which avoids linking libpython and avoids the MacOS virtualenv issues.
- Should have better performance as it's "native" vllm / sglang.
- Works with any version of vllm (including v1!) and sglang. Less upgrade struggle.
2025-05-06 16:03:31 -04:00
99cd9d85a9
feat: dynamo-run <-> python interop ( #934 )
...
Adding this to a Python script makes it register on the network so that `dynamo-run` can discover it and send it requests:
```
from dynamo.llm import register_llm
MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
await register_llm(endpoint, MODEL, 3)
```
Full vllm example, with pre-processing in dynamo:
- `dynamo-run in=text out=dyn://dynamo.backend.generate`
- `cd lib/bindings/python/examples/hello_world`
- `python server_vllm.py`
This builds on top of the work to move pre-processor to ingress side. It means we can decouple Rust and Python using NATS as the bus.
The `register_llm` call does this:
- Download the model from HF if necessary
- Load the model deployment card from the HF folder or extract from GGUF
- Push the tokenizer config etc into NATS object store so ingress can access it from a different machine
- Publish the model deployment card to ETCD
2025-05-05 20:49:19 -04:00
9d643f1eed
fix: use primary lease for NixlMetadataStore ( #928 )
2025-05-05 08:38:52 -07:00
2d2a102727
chore(dynamo-llm): Move the pre-processor to ingress side ( #903 )
...
Part of https://github.com/ai-dynamo/dynamo/issues/743
2025-05-01 19:55:10 +00:00
a1a10365a7
chore: Split PushRouter from Client ( #817 )
...
In a distributed system we don't know if the remote workers need pre-processing done ingress-side or not. Previously Client required us to decide this before discovering the remote endpoints, which was fine because pre-processing was worker-side.
As part of moving pre-processing back to ingress-side we need to split this into two steps:
- Client discovers the endpoints, and (later PR) will fetch their Model Deployment Card.
- PushRouter will use the Model Deployment Card to decide if they need pre-processing or not, which affects the types of the generic parameters.
Part of #743
2025-04-29 14:39:56 +00:00
974201c869
refactor: move logging config to runtime ( #863 )
2025-04-28 14:22:33 -07:00
7d5d6f8c08
feat: local planner for 0.2.0 release ( #398 )
...
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: ishandhanani <ishandhanani@gmail.com>
Co-authored-by: Ubuntu <ubuntu@dev-inst-2w1vokvyuts83rzn4n1k7mnzew9.us-central1-a.c.brevdevprod.internal>
Co-authored-by: Biswa Panda <biswa.panda@gmail.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>
2025-04-26 00:05:54 +00:00
0715d4691f
chore: bump NIXL version and package versions ( #836 )
...
Signed-off-by: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com>
2025-04-25 19:24:38 -04:00
d346782c7c
chore: Publish Model Deployment Card to NATS ( #799 )
...
This will allow an ingress-side pre-processor to see it without needing a model checkout.
Currently pre-processing is done in the worker, which has access to the model deployment card ("MDC") files (`config.json`, `tokenizer.json` and `tokenizer_config.json`) locally. We want to move the pre-processor to the ingress side to support KV routing. That requires ingress side (i.e the HTTP server), on a different machine than the worker to be able to see those three files.
To support that this PR makes the worker upload the contents of those files to the NATS object store, and publishes the MDC with those NATS urls to the key-value store.
The key-value store has an interface so any store (nats, etcd, redis, etc) can be supported. Implementations for memory and NATS are provided.
Fetching the MDC from the store, doing pre-processing ingress side, and publishing a card backed by a GGUF, are all for a later commit.
Part of #743
2025-04-25 08:52:59 -04:00
420b7a82be
fix: Fix cancellation flow in python component graph ( #765 )
2025-04-21 16:07:49 -04:00
c392c3419b
feat: add custom lease to worker components ( #748 )
2025-04-21 17:43:02 +00:00
4c38680eac
feat: gracefully shutdown endpoint by revoking etcd lease + python binding ( #730 )
...
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2025-04-18 18:36:50 +00:00
50aa390b11
feat: configure logger with detail info ( #654 )
...
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2025-04-17 19:36:22 +00:00
f4780e855c
docs: Remove outdated python-wheels directory reference ( #719 )
2025-04-16 23:22:10 -07:00
08fd28978c
feat: ETCD prefix watcher + python binding + runtime reconfiguration for router and disagg router ( #581 )
...
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
2025-04-11 18:15:42 -07:00
447840c223
docs: add docstring for llm.rs ( #267 )
2025-04-10 22:40:09 -04:00
fa7ee14c2a
chore: update versions to 0.1.1 ( #552 )
2025-04-09 09:50:30 -04:00
4b6cfc1be0
feat: KV recorder for dumping router events into a jsonl ( #505 )
2025-04-04 15:31:32 -07:00
88ad3425c4
feat: Python decorator dynamo_worker takes optional `static` parameter without etcd ( #494 )
...
Adds `@dynamo_worker(static = True)` to create a static worker which has a predictable name and hence does not require discovery or `etcd` to be running. There can only be a single static worker per namespace / component / endpoint trio.
This contrasts with the default dynamic `dynamo_worker` endpoints we have now, which get a unique random name (based on namespace/component/endpoint), and are discovered by ingress components using etcd.
Also change the hello_world example to use `dynamo_worker(static = True)` so that it is exercised and demonstrated somewhere.
For NIM.
2025-04-04 09:26:52 -04:00
84985d3f1d
refactor: migrate engines to standalone crates ( #453 )
...
Moved all of `lib/llm/src/engines` to their own crates as e.g. `lib/engines/mistralrs`. This will allow publishing of the `dynamo-llm` crate as it won't have any github dependencies.
The only engines in dynamo-llm will be the demo `echo` ones.
Co-authored-by: Graham King <grahamk@nvidia.com>
2025-04-03 21:53:54 +00:00
6795e645d0
fix: adding missing file ( #501 )
2025-04-03 08:29:33 -04:00
c4106e6a27
feat: kv aware router executable ( #399 )
2025-04-02 17:58:21 +02:00
5b682f4839
feat: unified logging ( #472 )
2025-04-01 19:49:20 +00:00
de2905377c
refactor: prometheus upgrade ( #452 )
2025-03-31 13:03:36 -06:00
8621d914e3
feat: dynamo deploy hello world example to k8s ( #205 )
2025-03-28 20:51:48 +00:00
c7067fc2ec
feat: Build pre-processor from GGUF ( #344 )
...
This lets us do:
```
dynamo-run out=llamacpp <gguf_file>
```
Previously a `--model-config <hf-repo>` was also required, to configure our tokenizer.
2025-03-24 11:59:41 -04:00
7c3fd5c94c
chore: Don't depend on openssl ( #292 )
...
This makes the Rust parts all use ring / rustls library instead of local install of openssl. It's a step on the journey to being statically linked.
Pieces:
- `tokenizers` and `mistralrs` now support rustls (mistralrs by default, tokenizers with feature flag).
- Move shared dependencies up into workspace
- New `rand` crate has some renames for future rust
- Ensure the dependency doesn't creep back in by enforcing it with cargo deny.
2025-03-19 14:16:19 -04:00
548578f4a9
docs: fix links in docs ( #256 )
...
Co-authored-by: Anant Sharma <anants@nvidia.com>
2025-03-18 03:40:31 -04:00
6e09681e0e
feat: expose Python binding for KVEventPublisher. Use event pub/sub trait for KV events ( #169 )
2025-03-17 00:17:33 -07:00
7f136e29c1
feat: update deploy api & sdk ( #74 )
...
Co-authored-by: Julien Mancuso <jmancuso@nvidia.com>
Co-authored-by: Hannah Zhang <hannahz@nvidia.com>
Co-authored-by: Biswa Panda <biswa.panda@gmail.com>
Co-authored-by: Maksim Khadkevich <mkhadkevich@nvidia.com>
2025-03-15 18:32:57 -07:00
c1c22703c5
build: reorganize python packaging to build new wheels ( #118 )
2025-03-14 13:43:29 -04:00
f04359cf69
feat: global kv block manager ( #45 )
2025-03-13 20:57:28 -06:00
3d2928510c
build: add top level rust workspace ( #137 )
2025-03-13 17:22:19 -04:00
3f84cdadfa
feat: add new metrics and simple router cost fn ( #88 )
2025-03-11 11:53:21 -07:00
dd6208254d
feat: add openai http service ( #82 )
2025-03-10 17:27:22 -07:00
fc4da34502
chore: update wheel name and reset versions ( #73 )
2025-03-10 17:41:13 -04:00
989bb3d59c
feat: make block_size input for indexer, router, publisher ( #66 )
2025-03-09 15:58:27 -07:00
dd31a322ae
chore: stragglers rename ( #69 )
...
Co-authored-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
2025-03-09 14:09:11 -07:00
678cffb4e5
chore: left over renaming ( #67 )
...
Co-authored-by: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com>
Co-authored-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
2025-03-09 14:09:04 -04:00
6ba39b0932
chore: address comments for #35 ( #53 )
2025-03-09 07:44:45 -07:00
19844fc07e
feat: kv aware router + disagg router + prefill queue ( #11 )
...
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: hongkuan <hongkuanz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz <ptarasiewicz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz Nvidia <ptarasiewicznv@Piotrs-MacBook-Pro.local>
Co-authored-by: alec-flowers <aflowers@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
2025-03-08 17:09:11 -08:00
b4d56a5789
chore: Renamed Triton Distributed to Dynamo ( #56 )
2025-03-08 09:42:09 -08:00
602352ce19
chore: rename dynamo ( #44 )
...
Co-authored-by: Biswa Panda <biswa.panda@gmail.com>
2025-03-08 01:42:40 -08:00
dcecc47de9
test: add tests for kv bindings ( #35 )
2025-03-07 16:01:28 -08:00
12714d9080
feat: Python bring-your-own-engine with our tokenizer ( #47 )
...
Instead of using `out=pystr:<my.py>` we can now do this:
```
dynemo-run out=pytok:/home/graham/my_python_engine.py --model-path <hf-repo-checkout>
```
That engine will receive and respond with tokens. Here's an example engine file:
```
import asyncio
async def generate(request):
yield {"token_ids":[791]}
await asyncio.sleep(0.1)
yield {"token_ids":[6864]}
await asyncio.sleep(0.1)
yield {"token_ids":[315]}
await asyncio.sleep(0.1)
yield {"token_ids":[9822]}
await asyncio.sleep(0.1)
yield {"token_ids":[374]}
await asyncio.sleep(0.1)
yield {"token_ids":[12366]}
await asyncio.sleep(0.1)
yield {"token_ids":[13]}
```
Also reduce duplication by making the bindings engine use the llm lib engine.
2025-03-07 10:48:37 -05:00
e159e53fe6
feat: expose KV routing components for easier router customization ( #15 )
2025-03-05 16:06:19 -08:00
1af7433bff
refactor: rename triton_distributed to dynemo ( #22 )
...
Co-authored-by: Graham King <grahamk@nvidia.com>
2025-03-05 09:15:46 -08:00
a32cdad622
feat: add python binding for rust llm modules ( #13 )
2025-03-04 14:52:09 -08:00
3a5fe17db9
feat: nixl metadata store and retrieved from etcd ( #6 )
...
Co-authored-by: hongkuanz <hongkuanz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz <ptarasiewicz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz Nvidia <ptarasiewicznv@Piotrs-MacBook-Pro.local>
Co-authored-by: Neelay Shah <neelays@ipp2-0493.ipp2u1.colossus.nvidia.com>
Co-authored-by: Neelay Shah <neelays@ipp1-1941.ipp1a1.colossus.nvidia.com>
Co-authored-by: ishandhanani <ishandhanani@gmail.com>
Co-authored-by: Neelay Shah <neelays@4u8g-gen-0078.ipp3a2.colossus.nvidia.com>
Co-authored-by: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com>
2025-03-04 13:10:16 -08:00
2d906fb4c9
fix: Install specific toolchain ( #329 )
...
`cargo build --locked` won't let you use "1.85.0" if you only have "stable" installed, even if those are the same thing right now.
2025-03-03 11:53:14 -05:00
057f8f4700
feat: TensorRT-LLM engine ( #317 )
...
Engine, `tio` support and docs.
Proof of concept / experimental.
2025-02-28 17:09:01 -05:00
11a36651a6
[fix] KV Router Example fixes ( #314 )
...
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-02-28 12:20:47 -08:00
85cc7b67c6
refactor: service/endpoint stats_handler ( #282 )
2025-02-27 11:30:18 -07:00
ea401e3bc0
ci: build wheel from root directory ( #274 )
2025-02-27 11:56:51 -05:00
86aff237ab
refactor: using async_openai
...
Co-authored-by: Graham King <grahamk@nvidia.com>
2025-02-26 13:59:39 -08:00
e97493eb00
feat: sglang backend for tio ( #271 )
...
- Setup venv
```
uv venv
source .venv/bin/activate
uv pip install pip
uv pip install sgl-kernel --force-reinstall --no-deps
uv pip install "sglang[all]==0.4.2" --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer/
```
- Build: `cargo build --release --features sglang`
- Run single node (make sure you're in the venv): `./tio out=sglang ~/llm_models/my_model`
- Run Deepseek multi-gpu / multi-node:
Node 1:
```
tio in=http out=sglang --model-path ~/llm_models/DeepSeek-R1-Distill-Llama-70B/ --tensor-parallel-size 8 --num-nodes 2 --node-rank 0 --dist-init-addr 10.217.98.122:9876
```
Node 2:
```
tio in=none out=sglang --model-path ~/llm_models/DeepSeek-R1-Distill-Llama-70B/ --tensor-parallel-size 8 --num-nodes 2 --node-rank 1 --dist-init-addr 10.217.98.122:9876
```
2025-02-25 17:04:08 -05:00
c70de37fcb
chore: updating docs after restructure
2025-02-25 12:37:25 -08:00
b760c5694d
feat: Add completion endpoint to http server and llmctl ( #230 )
...
Co-authored-by: aflowers <aflowers@nvidia.com>
2025-02-25 12:32:09 -08:00
861c50982b
feat: enable metrics polling
...
Signed-off-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
Co-authored-by: Biswa Panda <biswapanda@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-02-25 11:12:54 -08:00
c06b95ffdb
ci: Add rust checks to missing directories ( #239 )
...
Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
2025-02-25 07:42:33 -08:00
08fcd7e93b
refactor: move libs to lib dir
...
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-02-24 18:21:02 -08:00
Ziqi Fan