cc4c3516bb
feat: mocker disagg ( #3833 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-28 20:36:02 +00:00
c78b590134
chore(runtime): Do not expose etcd lease ID ( #3915 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-28 16:20:25 +00:00
0028cdf43d
feat: Use ForwardPassCallback api from TRTLLM to register end of forward pass callback to enable cuda graphs ( #3297 )
...
Signed-off-by: Kyle McGill <kmcgill@nvidia.com>
2025-10-27 15:12:04 -07:00
7731b0245c
chore: Use KeyValueStoreManager instead of etcd::Client ( #3822 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-23 14:09:58 -04:00
41ff394f51
chore: restructure mocker cli args handling, to include prefill/decode ( #3847 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-23 10:41:23 -07:00
94aa2a7bbb
refactor: kvbm modularity DIS-657 Eliminate ETCD from the leader-worker initialization ( #3202 )
...
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
2025-10-23 00:10:09 -07:00
681951d4e6
feat: python gil release for radix tree + dump_tree_as_events in python ( #3748 )
...
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
2025-10-22 01:41:35 +00:00
e01c6e99bd
feat: bake prefill router into frontend, supporting vllm for now ( #3762 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-21 15:43:02 -07:00
d5840cda65
feat: enable KVBM GPU offload to Disk bypassing CPU ( #3510 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
2025-10-21 08:49:54 -07:00
4c207e0c8c
chore: remove kv metrics scraping and aggregation ( #3701 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-17 10:58:48 -07:00
ae4e96a2bc
fix: mocker engines should ignore downloading weights from hf (again) ( #3664 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-16 02:48:40 +00:00
f978f4d1d6
feat: dp rank routing ( #3597 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-16 01:48:27 +00:00
f5854e1784
fix: avoid offload redundant prefill blocks | fix cuda graph hanging ( #3632 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
2025-10-15 10:38:35 -07:00
ab0da5821e
feat: Python binding to download a model. ( #3593 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-15 12:26:02 -04:00
b5e762b2b8
feat: remove stale workers on snapshot + some refactoring ( #3589 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-13 21:36:09 +00:00
c3fcfdd6e3
chore: consolidations of KvPushRouter bindings and usage examples ( #3543 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-10-11 05:47:25 +00:00
65cc533702
feat: implement custom backend metrics for NIM ( #3266 )
...
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
2025-10-10 20:28:58 -07:00
4224e57da0
feat: rm the old KVBM metrics | update G2 to G3 metrics collection ( #3561 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
2025-10-11 01:48:09 +00:00
0e0218ff02
chore: Remove model_config from LocalModel ( #3558 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-10 22:01:30 +00:00
ca67409888
feat: add KVBM host to disk metrics | clean up dashboard ( #3534 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
2025-10-10 18:17:34 +00:00
b94ecd1627
feat: add disk offloading filtering in KVBM ( #3532 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-10-10 09:51:55 -07:00
9975cb9ffe
feat: enable KVBM to support PD disagg in Dynamo vLLM ( #3352 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
2025-10-08 13:45:36 -07:00
cf83794ae0
feat: DIS-678 kvbm modularity: standalone metrics endpoint ( #3433 )
...
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
2025-10-08 10:18:33 -07:00
30610e7371
feat: use KvPushRouter for prefill router ( #3401 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-03 18:21:48 -07:00
d2e3b66e65
feat: Transition to FullyContiguous Host and Disk layouts ( #3090 )
...
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: oandreeva-nv <oandreeva-nv@nvidia.com>
2025-10-01 16:26:39 -07:00
713e9e481b
fix: DIS-706 skip offloading the G1 matched blocks during offloading ( #3299 )
...
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
2025-10-01 10:28:06 -07:00
9b9536d0d7
feat: make prefill router general ( #3329 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-30 20:38:47 -07:00
d354763c40
fix: python bindings for router should register to etcd as well ( #3302 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-30 12:38:56 -07:00
3aa3077808
fix: more fixes for stable router benchmarking ( #3264 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-29 23:08:56 +00:00
e21dcf6cab
feat: enable KVBM emit metrics in Dynamo TRTLLM ( #3254 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
2025-09-29 08:48:51 -07:00
5bb7490448
chore: bump vllm version to 0.10.2 ( #3180 )
...
Signed-off-by: Alec <aflowers@nvidia.com>
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Co-authored-by: Kris Hung <krish@nvidia.com>
2025-09-26 04:01:57 +00:00
6ba64c31f5
feat: tensor type for generic inference. ( #2746 )
...
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-09-24 07:31:09 +00:00
031590fc14
feat: vllm prefill router ( #3155 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-22 16:38:56 -07:00
7a5a0bd6cd
chore: Upgrade Rust to 1.90 ( #3147 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-19 18:54:29 -04:00
5b19a39bab
feat: allow router to not track active blocks (prefill), and to not track cached blocks (decode) ( #3135 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-19 20:33:07 +00:00
2d39f1b1bb
feat: KVBM connector : enabling vectorized copy from pinned memory to device memory and vice versa ( #2989 )
...
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: oandreeva-nv <oandreeva-nv@nvidia.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: oandreeva-nv <oandreeva-nv@nvidia.com>
2025-09-19 09:35:07 -07:00
78a3fedab9
fix: hook up worker removals for indexer ( #3095 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-17 21:33:25 +00:00
f88d7dc74b
chore(bindings): Remove NatsQueue ( #3086 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-17 12:46:54 -04:00
9060ce12ce
feat: Make part of discovery re-usable ( #3073 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-17 10:43:24 -04:00
723f2da74b
chore: Remove more extended Apache headers ( #3063 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-16 15:48:51 -04:00
87e6e0529d
fix: Interactive inputs actually stops, does not ignore stop token ( #3057 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-16 14:39:05 -04:00
55659eae70
fix: early stop if CPU or disk space not set when using KVBM ( #2997 )
...
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
2025-09-15 10:14:01 -07:00
4000097653
feat: adds kv indexer metrics ( #2905 )
...
Signed-off-by: Brian Larson <brian.larson@baseten.co>
2025-09-10 21:33:15 +00:00
cb5a657a6a
fix: Load the tokenizer JSON once for chat and completions. ( #2910 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-09-05 16:48:16 -04:00
c920cbd9dc
feat: Add --custom-jinja-template argument to pass a custom chat template for vLLM ( #2829 )
...
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
2025-09-03 14:22:15 -07:00
c6becbc859
feat: dynamo namespace isolation ( #2394 )
...
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
2025-09-03 15:47:06 +00:00
383e3b3a52
feat: don't modify kv scheduler states on query + more python binding ( #2798 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-02 17:09:43 -07:00
7fabe7bfe2
fix: do not delete KV events jetstream ( #2800 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-01 21:25:49 +00:00
488c87095c
feat: Router warm restarts via durable KV event consumers and radix snapshotting ( #2756 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-08-30 23:42:57 +00:00
a68c2f8f12
feat: DIS-373 dynamo KVBM connector API integration with TRTLLM ( #2544 )
...
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
2025-08-29 19:27:15 -07:00
Yan Ru Pei