f08729ae15
feat: python bindings for the entire KvPushRouter + per-request router configs ( #2658 )
2025-08-25 22:27:10 +00:00
3036e60b1e
feat: add gpt oss reasoning parser through harmony ( #2656 )
...
- couple of refactors
- added a new dependency, openai-harmony
- implemented the gpt oss parser
2025-08-25 17:13:38 +00:00
b39382ba68
feat: add initial batch of KVBM metrics on match, offload and onboard ( #2673 )
2025-08-25 09:28:25 -07:00
9b9f2ce4f6
fix: pytest robustness and parsing error ( #2676 )
2025-08-24 10:13:41 -07:00
cbe854fc5f
feat: [vLLM] implement cli args for tool and reasoning parsers ( #2619 )
2025-08-22 20:29:18 +00:00
b658ba6139
feat: enable dynamo metrics on KVBM ( #2626 )
2025-08-22 19:58:05 +00:00
b92a805edb
feat: add BaseLogitsProcessor core interface ( #2613 )
...
Signed-off-by: Bhuvan Agrawal <11240550+bhuvan002@users.noreply.github.com>
2025-08-22 13:41:30 -04:00
6a358f7c8c
chore(llm): Rename protocols::Endpoint to EndpointId ( #2615 )
2025-08-22 15:07:34 +00:00
174389e6d3
fix: Httpengine sync-enable-endpoint ( #2591 )
2025-08-21 18:29:52 -04:00
57728909cf
feat: Add model label for vllm backend metrics ( #2474 )
...
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
2025-08-21 17:32:34 +00:00
626d7e182d
feat(request cancellation): pycontext, propagating the `is_stopped` into python land. ( #2158 )
2025-08-20 12:06:13 -07:00
49958435eb
chore: Remove async-openai-macros ( #2554 )
2025-08-20 11:33:10 -07:00
d319abf3a3
feat: upload/download rust structs directly through NATs object store ( #2540 )
2025-08-20 17:17:21 +00:00
9a02188531
chore: Bumped Dynamo version to 0.4.1 ( #2545 )
2025-08-19 21:55:37 -04:00
177d662f86
fix: Dockerfile.sglang - Fixed sglang and dynamo wheels installaiton in ru… ( #2537 )
2025-08-19 20:56:12 -04:00
199b9a30f4
chore: Bring async-openai into repo as request starter ( #2520 )
...
Co-authored-by: Graham King <grahamk@nvidia.com>
2025-08-19 17:47:01 -04:00
07cfc3a11b
feat: kvbm + connector ( #2258 )
...
Signed-off-by: Ryan Olson <rolson@nvidia.com>
Co-authored-by: Olga Andreeva <oandreeva@nvidia.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: John Thompson <jothomson@nvidia.com>
Co-authored-by: Richard Huo <rihuo@nvidia.com>
Co-authored-by: Zicheng Ma <zichengm@nvidia.com>
2025-08-19 12:36:53 -07:00
a33033b7f6
feat: task scheduler ( #2406 )
...
Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com>
2025-08-19 12:35:31 -06:00
c5d9d26703
feat(frontend): support setting HTTP host via CLI (--http-host) ( #2523 )
2025-08-19 09:01:04 -04:00
85d8310806
feat: router-level request rejection ( #2465 )
2025-08-19 01:51:00 -07:00
a4bbe49228
feat(http): TLS support ( #2492 )
2025-08-18 16:06:29 -04:00
0444217339
fix: replace metrics callback with background scraping to prevent tim… ( #2480 )
...
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
2025-08-18 12:38:50 -07:00
ffae72b7d2
fix: remove kvmanager feature from python 3.12 ai-dynamo-runtime wheel ( #2456 )
2025-08-15 16:43:55 -04:00
acbdabc464
feat(metrics): add NATS client metrics to prometheus_metrics_fmt ( #2292 )
...
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
2025-08-14 19:33:53 -07:00
3a3f5bf275
feat: Add a "model" label to Component metrics ( #2389 )
2025-08-14 13:48:25 -05:00
d0a6363584
feat: add RuntimeConfig to ModelEntry ( #2311 )
...
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
2025-08-14 10:19:53 -07:00
c12c25787f
fix: upgrade cudarc to 0.17.1 ( #2341 )
...
Signed-off-by: Dan Aloni <dan.aloni@vastdata.com>
Co-authored-by: Tushar Sharma <tusharma@nvidia.com>
2025-08-13 15:45:08 -04:00
72ec5f5c7b
feat: Allow an endpoint to serve multiple models ( #2418 )
2025-08-13 10:48:42 -04:00
9b87c89c41
feat: multi-modal example with vLLM v1 and UX v2 ( #2040 )
...
Co-authored-by: krishung5 <krish@nvidia.com>
2025-08-12 23:55:31 +00:00
5166a3dd44
feat: Router replicas with state-sharing ( #2264 )
2025-08-07 22:57:57 +00:00
bd4fe1a7d1
feat: cross process instrumentation ( #2243 )
...
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
2025-08-07 15:32:48 +00:00
1954fcfa09
chore: Remove service_name from ModelDeploymentCard ( #2349 )
2025-08-07 11:02:41 -04:00
dbe48a1d2b
chore: Bump mistral.rs, llama.cpp and tokenizers deps ( #2338 )
2025-08-06 16:23:54 -04:00
b2aa504b47
fix: upgrade axum to 0.8 and etcd-client to 0.16 ( #2317 )
...
Signed-off-by: Dan Aloni <dan.aloni@vastdata.com>
2025-08-06 18:25:49 +00:00
6a1a801c2d
feat: Support static workers, run without etcd. ( #2281 )
2025-08-06 09:51:43 -04:00
36c4ef5eb2
feat: migrate requests when planner shutdown decode engine (vllm) ( #2280 )
...
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: hhzhang16 <54051230+hhzhang16@users.noreply.github.com>
2025-08-05 12:24:07 -07:00
347620a1ff
feat: Allow Python Engine to end stream before final ( #2270 )
2025-08-05 10:44:18 -07:00
433f60121a
feat: Pass user_data to register_llm for LoRA support ( #2286 )
2025-08-05 10:16:43 -04:00
4cbd4f3881
fix: show raw sglang logs given env var ( #2287 )
2025-08-05 06:11:44 +00:00
625578c3bb
chore: update nixl version to 0.4.1 ( #2221 )
2025-07-31 17:46:21 -04:00
bae25dc6d4
feat: skip downloading model weights if using mocker (only tokenizer) ( #2213 )
2025-07-31 18:15:57 +00:00
4c90b1b924
chore: Version bump to 0.4.0 ( #2179 )
2025-07-30 02:07:31 +00:00
222245e2a0
refactor: Move engine and publisher from dynamo.llm.tensorrt_llm to dynamo.trtllm ( #2128 )
2025-07-26 02:23:17 +00:00
19a77ae7d5
chore(dynamo-run): Remove out=sglang|vllm|trtllm ( #1920 )
2025-07-23 17:22:31 -07:00
66b7d2c75d
fix: updates versions and adds ahashmap to BPE ( #2072 )
2025-07-23 20:27:18 +00:00
e0a51940d1
feat: Connect Library ( #1478 )
2025-07-23 14:16:34 -04:00
9f2356cb9e
chore: Remove unused portion of kv bindings test ( #2052 )
2025-07-22 19:12:42 +00:00
1f07dab7bd
feat: Add migration to LLM requests ( #1930 )
2025-07-18 20:04:20 +00:00
fc12436048
feat(frontend): router-mode settings ( #2001 )
2025-07-18 18:52:57 +00:00
182d3b5dc7
chore(bindings): Remove mistralrs / llama.cpp ( #1970 )
2025-07-16 16:12:40 -04:00
Yan Ru Pei