mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,557 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

448 Commits

Author SHA1 Message Date
KrishnanPrash 24ac6c1588
chore: bump minijinja to 2.14.0 (#4949) 
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
 2025-12-15 09:58:52 -08:00
Yan Ru Pei c2a29f8045
chore: enable local indexers for sglang and trtllm (#4932) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-12 11:41:37 -08:00
Karen Chung 332499458b
feat: worker-local KvIndexer in KvEventPublisher (#4519) 
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
 2025-12-11 23:18:34 +00:00
Yan Ru Pei 10b01b4590
feat: early rejection based on active prefill tokens (#4837) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-11 23:16:02 +00:00
Dmitry Tokarev 2ef0f5fbab
fix: relax pydantic requirements (#4865) 
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
 2025-12-11 19:32:19 +00:00
Vladislav Nosivskoy f8bb53c0ed
feat: add cached tokens prometheus metric (#4534) 
Signed-off-by: Vladislav Nosivskoy <vladnosiv@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-12-11 08:03:12 +03:00
Jacky 00f8615e47
test: Ensure Request Cancellation, Migration, Rejection Work with TCP transport (#4875) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2025-12-11 02:21:17 +00:00
J Wyman 96fe63fe2e
feat: nixl_connect: Improve Concurrency Support (#4433) 
Signed-off-by: J Wyman <jwyman@nvidia.com>
 2025-12-09 17:29:22 -05:00
Keiven C 1e37c1033f
fix: add timeout to tests before lora to unblock (#4808) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-12-08 20:50:33 -08:00
Biswa Panda 5588d77074
test: add timeout for lora utils tests (#4781) 2025-12-06 02:23:20 +00:00
William Arnold 3bc129dc37
feat: Add python-configurable engine routes for sglang profiling (#4617) 
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
 2025-12-05 10:28:36 -08:00
Neal Vaidya 118323f26e
fix: skip HuggingFace download for non-llms (#4686) 
Signed-off-by: Neal Vaidya <nealv@nvidia.com>
 2025-12-04 10:23:24 -08:00
Biswa Panda 71f94eda23
feat: lora - centralize lora cache key, restructure folders, s3 resiliency (#4644) 2025-12-02 08:57:25 +00:00
Biswa Panda 6a0e67ed78
feat: add LoRA common APIs and implementation for lora management (#4464) 2025-11-26 20:29:25 +00:00
Graham King fcb91e4b3d
refactor(storage): Remove the stuttering from key_value_store. (#4604) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-25 22:59:29 +00:00
milesial 0a06591826
feat: decoded media via NIXL (#3988) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
Signed-off-by: Ayush Agarwal <ayushag@nvidia.com>
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
Co-authored-by: Ayush Agarwal <ayushag@nvidia.com>
Co-authored-by: Krishnan Prashanth <kprashanth@nvidia.com>
Co-authored-by: KrishnanPrash <140860868+KrishnanPrash@users.noreply.github.com>
 2025-11-25 19:15:04 +00:00
Graham King 0fc5273c11
refactor(llm): Rename EngineConfig::Static to InProcess (#4585) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-25 12:22:09 -05:00
Graham King 27904535fb
fix(storage): Correctly encoding FileStore keys (#4539) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-21 17:01:40 -05:00
Yan Ru Pei c61e0dd3cc
chore: merge KvIndexer and ApproxKvIndexer (#4500) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-21 09:09:11 -08:00
Yan Ru Pei d821a8b9f7
chore: parallelize planner profile tests + bindings test cleanup (#4532) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-21 09:21:43 +00:00
Kris Hung 2f18b23ee7
chore: Use general engine source for KVBM KV events (#4515) 2025-11-20 14:47:48 -08:00
Graham King 8a14f9edc1
chore: Remove DistributedRuntime::etcd_client (#4489) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-20 12:35:10 -05:00
Biswa Panda 781331c6f3
feat: unregister discovery instance (#4459) 2025-11-19 20:25:44 +00:00
Graham King 69797b5ab3
feat: Only monitor NATS metrics if using NATS request plane (#4442) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-19 17:48:49 +00:00
Biswa Panda 684107c461
feat: generate deterministic lora id from lora name (#4457) 2025-11-19 02:26:00 +00:00
Graham King 2f1778c13d
fix(storage): File store shouldn't notify on metdata changes (#4434) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-18 10:24:04 -08:00
J Wyman 0814653104
feat: nixl_connect: Change the way `bytes` are passed to NIXL (#4314) 
Signed-off-by: J Wyman <jwyman@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-11-18 18:05:46 +00:00
Yan Ru Pei fc2ad4eb1e
feat: mocker can use planner profile data (#4422) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Hongkuan Zhou <tedzhouhk@gmail.com>
 2025-11-18 08:06:50 +00:00
Keiven C 0e77d3442e
refactor: centralize environment variable constants (#4083) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-11-17 11:16:38 -08:00
Graham King 886506c12d
feat: Command line flag to set request plane mode: tcp, http or nats (#4365) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-17 09:02:36 -08:00
Waël Boukhobza 2cf427ed8f
fix(router): change prune channel to be mpsc instead of watch, increase max size (#4351) 
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
 2025-11-14 18:19:20 -05:00
Yan Ru Pei ce83398366
chore: better error handling in prefill router (#4286) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-13 22:29:26 +00:00
Biswa Panda 06b0ebef88
feat: transport agnostic request plane for dynamo - natless (#4246) 2025-11-13 17:54:54 +00:00
Yan Ru Pei 8379b0cd10
feat: kv router should route to available instances (#4225) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-13 02:34:58 +00:00
Yan Ru Pei 38242c8d7b
chore: use notify instead of inotify (#4250) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-12 16:53:39 +00:00
Waël Boukhobza 3ea22fcf39
feat(router): max tree size based pruning (#4057) 
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
 2025-11-12 08:33:49 -08:00
Graham King fbad28602e
chore: Remove the python bindings for port allocation (#4237) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-11 20:35:36 +00:00
Graham King e1af3af6ee
chore: Remove static mode (#4235) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-11 19:25:12 +00:00
Aryan Bagade 8f868986e8
feat: Add output token counter to frontend metrics (#4202) 
Signed-off-by: Aryan Bagade <aryan@aryanbagade.com>
 2025-11-11 05:09:07 +00:00
mohammedabdulwahhab 7afb5431ff
fix: add kube impl for discovery and add metadata endpoint (#4136) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
Co-authored-by: tmontfort <tmontfort@nvidia.com>
 2025-11-10 22:42:04 +00:00
zhongdaor-nv ec7af93953
fix: Extend add_tensor_model so that ModelDeploymentCard can be correctly picked up (#4169) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
 2025-11-10 13:59:40 -08:00
Anant Sharma 23660bc5c3
chore: update versions for 0.7.0 release (#4217) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2025-11-10 21:40:06 +00:00
Keiven C afccc9d434
refactor: consolidate Observability files (e.g. OTEL docker-compose, md files) (#4173) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-11-10 20:19:34 +00:00
Graham King cf630bf7b6
refactor: Make the Runtime and DistributedRuntime fields private (#4193) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-10 15:56:22 +00:00
mohammedabdulwahhab 09b26bf6b3
fix: refactor to use service discovery (#4092) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
 2025-11-08 18:05:18 +00:00
milesial 14af074ecd
feat: Media decoder and fetcher options in the MDC (#4094) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-11-08 02:17:39 +00:00
Keiven C 6bccf099f8
feat: deprecate DYN_SYSTEM_ENABLED in favor of DYN_SYSTEM_PORT (#4082) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-11-07 15:31:51 -08:00
Graham King 794c0a4431
feat(keyvalue): Filesystem backed KeyValueStore (#4138) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-07 21:11:33 +00:00
KrishnanPrash 3d036fc4a4
feat: Reject unsupported parameters with 400 Bad Request (`/v1/completions`) (#4140) 
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
 2025-11-05 17:32:50 -08:00
Jacky 6e2b22eaef
feat: ETCD high availability client failover - lease watch resilience (#3950) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-11-05 17:21:09 -08:00
KrishnanPrash b73c571f0f
feat: Add base64 and HTTP image URL support to vLLM workers (#4114) 
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
 2025-11-05 16:47:06 -08:00
Yan Ru Pei 427ca9aba4
chore: no need for explicit Python::with_gil (#4075) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-05 05:41:49 +00:00
Michael Feil defe5de74b
feat: release gil for publisher (#3747) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
 2025-11-05 02:17:06 +00:00
Graham King dadf0e2247
chore: Remove old DisaggregatedRouter, making etcd presence optional (#4011) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-03 10:10:13 -05:00
Yan Ru Pei 6ff49edbb7
feat: efficient serving of multiple mockers (#3997) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-01 01:24:55 +00:00
Richard Huo 6afa679c5d
chore: KVBM pip wheel (#3826) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>
 2025-10-31 14:19:30 -07:00
Anant Sharma 8bd37c96d6
refactor: move backend deploy, launch and slurm files from components to examples (#3849) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2025-10-31 13:09:46 -04:00
Tushar Sharma 1da9d70ab1
chore: bump nixl verison to 0.7.0 (#3936) 
Signed-off-by: Tushar Sharma <tusharma@nvidia.com>
 2025-10-30 10:28:17 -07:00
Kris Hung 1898601025
feat: Add KV event consolidator for KVBM (vllm) and router integration (#3725) 
Signed-off-by: krishung5 <krish@nvidia.com>
 2025-10-30 17:11:42 +00:00
milesial da10299c19
fix: NIXL connect header strip (#3959) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-10-29 16:11:35 -07:00
jthomson04 3998fdcb28
feat: KVBM V2 Initial Migration (#3861) 
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
 2025-10-28 16:08:14 -07:00
Yan Ru Pei cc4c3516bb
feat: mocker disagg (#3833) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-28 20:36:02 +00:00
milesial 76f2ef9112
feat: b64 encoding for nixl_connect metadata (#3843) 2025-10-28 11:22:50 -07:00
Graham King c78b590134
chore(runtime): Do not expose etcd lease ID (#3915) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-28 16:20:25 +00:00
Kyle McGill 0028cdf43d
feat: Use ForwardPassCallback api from TRTLLM to register end of forward pass callback to enable cuda graphs (#3297) 
Signed-off-by: Kyle McGill <kmcgill@nvidia.com>
 2025-10-27 15:12:04 -07:00
Tushar Sharma 9bb1af3376
chore: Update version to 0.6.1 (#3916) 
Signed-off-by: Tushar Sharma <tusharma@nvidia.com>
 2025-10-27 17:17:26 -04:00
Keiven C cbe0b177ab
refactor: redesign the metrics API from Trait to composition to make the code cleaner and easier to understand (#3687) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-24 14:34:29 -07:00
zhongdaor-nv 9e5407f20d
feat: python binding for kserve grpc frontend (#3739) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-10-24 11:20:20 -07:00
Graham King 7731b0245c
chore: Use KeyValueStoreManager instead of etcd::Client (#3822) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-23 14:09:58 -04:00
Yan Ru Pei 41ff394f51
chore: restructure mocker cli args handling, to include prefill/decode (#3847) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-23 10:41:23 -07:00
Richard Huo 94aa2a7bbb
refactor: kvbm modularity DIS-657 Eliminate ETCD from the leader-worker initialization (#3202) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-10-23 00:10:09 -07:00
Keiven C f93b619ad9
feat: add TensorRT-LLM Prometheus metrics support with prefixing + filtering (#3676) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-22 11:25:13 -07:00
Michael Feil 681951d4e6
feat: python gil release for radix tree + dump_tree_as_events in python (#3748) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
 2025-10-22 01:41:35 +00:00
Yan Ru Pei e01c6e99bd
feat: bake prefill router into frontend, supporting vllm for now (#3762) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-21 15:43:02 -07:00
mohammedabdulwahhab 8a04d7b252
fix: revert placement of logging init if otel exports is disabled (#3797) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
 2025-10-21 17:36:31 -04:00
Ziqi Fan d5840cda65
feat: enable KVBM GPU offload to Disk bypassing CPU (#3510) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-21 08:49:54 -07:00
Graham King 9ae98ed7f9
refactor(runtime): Replace std::sync::Mutex with parking_lot::Mutex (#3740) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-21 12:38:51 +00:00
Graham King f6ed01b14c
chore: Replace ServiceConfigBuilder with add_stats_service (#3736) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-20 16:47:05 -04:00
Michael Feil af7b73c045
feat: python package build with abi (cross py3.10+) (#3571) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
 2025-10-20 12:04:38 -07:00
Yan Ru Pei 4b7a806c7a
feat: add prefill workers to discovery (#3709) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-18 00:59:32 +00:00
Yan Ru Pei 4c207e0c8c
chore: remove kv metrics scraping and aggregation (#3701) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-17 10:58:48 -07:00
Anish 598cbbb73b
docs: reorganizing documentation to make things clearer (#3658) 
Signed-off-by: athreesh <anish.maddipoti@utexas.edu>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-16 23:59:59 +00:00
Yan Ru Pei ae4e96a2bc
fix: mocker engines should ignore downloading weights from hf (again) (#3664) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-16 02:48:40 +00:00
Jacky 734d2f87a8
feat: Enable cancellation during or before a stream is established (#3635) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-10-15 19:22:55 -07:00
Yan Ru Pei f978f4d1d6
feat: dp rank routing (#3597) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-16 01:48:27 +00:00
Ziqi Fan f5854e1784
fix: avoid offload redundant prefill blocks | fix cuda graph hanging (#3632) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-15 10:38:35 -07:00
Graham King ab0da5821e
feat: Python binding to download a model. (#3593) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-15 12:26:02 -04:00
Harrison Saturley-Hall cd2389baef
chore: pre-0.6.0 activities (#3592) 
Signed-off-by: Harrison Saturley-Hall <hsaturleyhal@nvidia.com>
 2025-10-13 17:45:19 -04:00
Yan Ru Pei b5e762b2b8
feat: remove stale workers on snapshot + some refactoring (#3589) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-13 21:36:09 +00:00
mohammedabdulwahhab 1f92dd547e
feat: OTEL Exporter and Tempo Visualization (#3307) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
 2025-10-13 18:39:09 +00:00
Alec 90dc758938
chore: bump vllm to 0.11.0 (#3422) 
Signed-off-by: alec-flowers <aflowers@nvidia.com>
 2025-10-12 14:26:45 -07:00
Yan Ru Pei c3fcfdd6e3
chore: consolidations of KvPushRouter bindings and usage examples (#3543) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-10-11 05:47:25 +00:00
Keiven C 65cc533702
feat: implement custom backend metrics for NIM (#3266) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-10 20:28:58 -07:00
Ziqi Fan 4224e57da0
feat: rm the old KVBM metrics | update G2 to G3 metrics collection (#3561) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-11 01:48:09 +00:00
Keiven C 55e458d899
feat: add SGLang and vLLM passthrough metrics on Dynamo backend worker (#3539) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-10 17:59:35 -07:00
Keiven C a4746ab65f
fix: callback registration, fix metric name access, ensure ordered vec, etc... (#3541) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-11 00:03:24 +00:00
Graham King 0e0218ff02
chore: Remove model_config from LocalModel (#3558) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-10 22:01:30 +00:00
Richard Huo 5c69c11930
chore: upgrade nixl to 0.6.0 (#3550) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-10-10 13:18:21 -07:00
Keiven C f2ba58e568
feat: add Rust-to-Python const code generator for prometheus_names.py (#3425) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-10 11:20:36 -07:00
Ziqi Fan ca67409888
feat: add KVBM host to disk metrics | clean up dashboard (#3534) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-10 18:17:34 +00:00
Ziqi Fan b94ecd1627
feat: add disk offloading filtering in KVBM (#3532) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2025-10-10 09:51:55 -07:00
Graham King e9a710095b
chore: Remove clear_namespace script (#3519) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-09 13:43:59 -04:00
Ryan McCormick ef04a0869e
feat: Add OpenAPI routes to serve HTTP server API documentation (#3480) 2025-10-08 13:52:19 -07:00
Ziqi Fan 9975cb9ffe
feat: enable KVBM to support PD disagg in Dynamo vLLM (#3352) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-10-08 13:45:36 -07:00
Graham King 0aa0768f53
chore: Remove llama.cpp engine (#3499) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-08 16:44:53 -04:00
Richard Huo cf83794ae0
feat: DIS-678 kvbm modularity: standalone metrics endpoint (#3433) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-10-08 10:18:33 -07:00
Graham King 1b1265e6fc
chore: Remove GGUF support (#3488) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-08 11:42:24 -04:00
Keiven C 0c4c4d1de3
feat: add Python MetricsRegistry Python metrics registration (#3341) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-07 21:24:25 -07:00
Graham King bdad6f1a50
fix: Make planner VirtualConnectorClient also use v1/ prefix. (#3468) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-07 22:34:30 +00:00
Tanmay Verma 68bbe8a9a5
feat: Connect DYN_LOG level to TLLM_LOG_LEVEL (#3451) 
Signed-off-by: Tanmay Verma <tanmay2592@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2025-10-07 21:02:35 +00:00
Graham King a5371bfc50
feat(etcd): Version the etcd keys (#3458) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-07 16:48:26 +00:00
Graham King 81162dfeb9
chore(discovery): Watch/publish ModelDeploymentCard instead of ModelEntry (#3350) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-07 10:02:02 -04:00
Yan Ru Pei 30610e7371
feat: use KvPushRouter for prefill router (#3401) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-03 18:21:48 -07:00
Olga Andreeva d2e3b66e65
feat: Transition to FullyContiguous Host and Disk layouts (#3090) 
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: oandreeva-nv <oandreeva-nv@nvidia.com>
 2025-10-01 16:26:39 -07:00
Richard Huo 713e9e481b
fix: DIS-706 skip offloading the G1 matched blocks during offloading (#3299) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-10-01 10:28:06 -07:00
Yan Ru Pei 9b9536d0d7
feat: make prefill router general (#3329) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-30 20:38:47 -07:00
Keiven C f4a3a6b66a
refactor: standardize Prometheus metric naming conventions (part 1) (#3035) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-09-30 19:01:39 -07:00
Keiven C af4c0cadcd
feat: add Context class to Python type stubs (#3327) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-09-30 17:10:02 -07:00
Michael Feil 5b457b70a6
feat: python add abi compatability for cross-platform builds + add a unit test to HttpServer (#3044) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Signed-off-by: Michael Feil <63565275+michaelfeil@users.noreply.github.com>
Signed-off-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Signed-off-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
 2025-09-30 14:42:42 -07:00
Yan Ru Pei d354763c40
fix: python bindings for router should register to etcd as well (#3302) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-30 12:38:56 -07:00
Keiven C cacac9b9f4
feat: add Python const for Prometheus metric names (#3244) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-09-30 10:28:47 -07:00
Yan Ru Pei 3aa3077808
fix: more fixes for stable router benchmarking (#3264) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-29 23:08:56 +00:00
Ziqi Fan e21dcf6cab
feat: enable KVBM emit metrics in Dynamo TRTLLM (#3254) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-09-29 08:48:51 -07:00
Elyas Mehtabuddin 67e1f6ee11
feat: enable parallel tool calling and add testing (#3188) 
Signed-off-by: Elyas Mehtabuddin <emehtabuddin@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-09-29 15:30:42 +00:00
Graham King 7ebbd001c0
chore: Remove etcd from python bindings (#3238) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-26 12:58:30 -04:00
Alec 5bb7490448
chore: bump vllm version to 0.10.2 (#3180) 
Signed-off-by: Alec <aflowers@nvidia.com>
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Co-authored-by: Kris Hung <krish@nvidia.com>
 2025-09-26 04:01:57 +00:00
Neelay Shah f2e2e935b2
feat: Add distributed tracing context support to Python bindings (#3160) 
Signed-off-by: nnshah1 <neelays@nvidia.com>
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
 2025-09-25 17:26:35 -07:00
Graham King c03e2f6bbe
chore: Migrate planner virtual_connector internals into bindings (#3205) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-25 18:04:58 -04:00
Harrison Saturley-Hall 980727bba1
chore: bump versions ahead of 0.5.1 release (#3209) 
Signed-off-by: Harrison Saturley-Hall <hsaturleyhal@nvidia.com>
 2025-09-24 18:26:38 -04:00
GuanLuo 6ba64c31f5
feat: tensor type for generic inference. (#2746) 
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-09-24 07:31:09 +00:00
Ryan Olson c63cceaa77
feat: JailedStream (#3034) 
Signed-off-by: ayushag <ayushag@nvidia.com>
 2025-09-23 22:24:17 +00:00
Yan Ru Pei 031590fc14
feat: vllm prefill router (#3155) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-22 16:38:56 -07:00
Jacky 31c78df781
refactor: PyTest subprocess for cancellation unit tests (#3127) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: michaelfeil <me@michaelfeil.eu>
 2025-09-22 10:56:37 -07:00
Graham King 7a5a0bd6cd
chore: Upgrade Rust to 1.90 (#3147) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-19 18:54:29 -04:00
Yan Ru Pei 5b19a39bab
feat: allow router to not track active blocks (prefill), and to not track cached blocks (decode) (#3135) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-19 20:33:07 +00:00
Graham King 3865a94148
feat: Port vllm port allocator to Rust in bindings (#3125) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-19 18:34:52 +00:00
Olga Andreeva 2d39f1b1bb
feat: KVBM connector : enabling vectorized copy from pinned memory to device memory and vice versa (#2989) 
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: oandreeva-nv <oandreeva-nv@nvidia.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: oandreeva-nv <oandreeva-nv@nvidia.com>
 2025-09-19 09:35:07 -07:00
Jacky a8fd127130
feat: Request Cancellation unary request support (#3004) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2025-09-18 22:26:37 -07:00
Elyas Mehtabuddin 31f5ed3ce7
feat: add finish reason = tool_calls for stream=False and phi-4 detect token start fix (#3087) 
Signed-off-by: Elyas Mehtabuddin <emehtabuddin@nvidia.com>
 2025-09-18 21:45:52 +00:00
Graham King b6595e2484
chore(bindings): Provide a binding to clear etcd namespace (#3094) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-18 11:49:08 -04:00
Richard Huo 86ce03f428
fix: Update the KVBM <> TRT-LLM integration interface to match the latest TRT-LLM connector API (#2979) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-09-17 15:33:13 -07:00
Yan Ru Pei 78a3fedab9
fix: hook up worker removals for indexer (#3095) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-17 21:33:25 +00:00
Graham King f88d7dc74b
chore(bindings): Remove NatsQueue (#3086) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-17 12:46:54 -04:00
Graham King 9060ce12ce
feat: Make part of discovery re-usable (#3073) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-17 10:43:24 -04:00
Tzu-Ling Kan 08cb08c1bc
feat: Canary Health Check. (#2903) 
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
 2025-09-17 03:31:56 +00:00
Biswa Panda 960dc8967e
fix: replace hard coded dynamo namespace with env var (#3048) 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
 2025-09-16 14:56:51 -07:00
Graham King 723f2da74b
chore: Remove more extended Apache headers (#3063) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-16 15:48:51 -04:00
Graham King 87e6e0529d
fix: Interactive inputs actually stops, does not ignore stop token (#3057) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-16 14:39:05 -04:00
ryan-lempka 435803ea16
chore: add additional param support for multimodal models (#3042) 
Signed-off-by: Ryan Lempka <rlempka@nvidia.com>
 2025-09-16 16:26:39 +00:00
Ziqi Fan 55659eae70
fix: early stop if CPU or disk space not set when using KVBM (#2997) 
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
 2025-09-15 10:14:01 -07:00