Yan Ru Pei
|
8379b0cd10
|
feat: kv router should route to available instances (#4225)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-11-13 02:34:58 +00:00 |
|
Waël Boukhobza
|
3ea22fcf39
|
feat(router): max tree size based pruning (#4057)
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
|
2025-11-12 08:33:49 -08:00 |
|
Graham King
|
fbad28602e
|
chore: Remove the python bindings for port allocation (#4237)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-11-11 20:35:36 +00:00 |
|
Graham King
|
e1af3af6ee
|
chore: Remove static mode (#4235)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-11-11 19:25:12 +00:00 |
|
zhongdaor-nv
|
ec7af93953
|
fix: Extend add_tensor_model so that ModelDeploymentCard can be correctly picked up (#4169)
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
|
2025-11-10 13:59:40 -08:00 |
|
Keiven C
|
afccc9d434
|
refactor: consolidate Observability files (e.g. OTEL docker-compose, md files) (#4173)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-11-10 20:19:34 +00:00 |
|
Graham King
|
cf630bf7b6
|
refactor: Make the Runtime and DistributedRuntime fields private (#4193)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-11-10 15:56:22 +00:00 |
|
milesial
|
14af074ecd
|
feat: Media decoder and fetcher options in the MDC (#4094)
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
|
2025-11-08 02:17:39 +00:00 |
|
Graham King
|
794c0a4431
|
feat(keyvalue): Filesystem backed KeyValueStore (#4138)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-11-07 21:11:33 +00:00 |
|
Jacky
|
6e2b22eaef
|
feat: ETCD high availability client failover - lease watch resilience (#3950)
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
|
2025-11-05 17:21:09 -08:00 |
|
Yan Ru Pei
|
427ca9aba4
|
chore: no need for explicit Python::with_gil (#4075)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-11-05 05:41:49 +00:00 |
|
Michael Feil
|
defe5de74b
|
feat: release gil for publisher (#3747)
Signed-off-by: michaelfeil <me@michaelfeil.eu>
|
2025-11-05 02:17:06 +00:00 |
|
Graham King
|
dadf0e2247
|
chore: Remove old DisaggregatedRouter, making etcd presence optional (#4011)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-11-03 10:10:13 -05:00 |
|
Yan Ru Pei
|
6ff49edbb7
|
feat: efficient serving of multiple mockers (#3997)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-11-01 01:24:55 +00:00 |
|
Richard Huo
|
6afa679c5d
|
chore: KVBM pip wheel (#3826)
Signed-off-by: Anant Sharma <anants@nvidia.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>
|
2025-10-31 14:19:30 -07:00 |
|
Kris Hung
|
1898601025
|
feat: Add KV event consolidator for KVBM (vllm) and router integration (#3725)
Signed-off-by: krishung5 <krish@nvidia.com>
|
2025-10-30 17:11:42 +00:00 |
|
Yan Ru Pei
|
cc4c3516bb
|
feat: mocker disagg (#3833)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-28 20:36:02 +00:00 |
|
Graham King
|
c78b590134
|
chore(runtime): Do not expose etcd lease ID (#3915)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-28 16:20:25 +00:00 |
|
Kyle McGill
|
0028cdf43d
|
feat: Use ForwardPassCallback api from TRTLLM to register end of forward pass callback to enable cuda graphs (#3297)
Signed-off-by: Kyle McGill <kmcgill@nvidia.com>
|
2025-10-27 15:12:04 -07:00 |
|
Keiven C
|
cbe0b177ab
|
refactor: redesign the metrics API from Trait to composition to make the code cleaner and easier to understand (#3687)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-24 14:34:29 -07:00 |
|
zhongdaor-nv
|
9e5407f20d
|
feat: python binding for kserve grpc frontend (#3739)
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
|
2025-10-24 11:20:20 -07:00 |
|
Graham King
|
7731b0245c
|
chore: Use KeyValueStoreManager instead of etcd::Client (#3822)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-23 14:09:58 -04:00 |
|
Yan Ru Pei
|
41ff394f51
|
chore: restructure mocker cli args handling, to include prefill/decode (#3847)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-23 10:41:23 -07:00 |
|
Richard Huo
|
94aa2a7bbb
|
refactor: kvbm modularity DIS-657 Eliminate ETCD from the leader-worker initialization (#3202)
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
|
2025-10-23 00:10:09 -07:00 |
|
Michael Feil
|
681951d4e6
|
feat: python gil release for radix tree + dump_tree_as_events in python (#3748)
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
|
2025-10-22 01:41:35 +00:00 |
|
Yan Ru Pei
|
e01c6e99bd
|
feat: bake prefill router into frontend, supporting vllm for now (#3762)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-21 15:43:02 -07:00 |
|
mohammedabdulwahhab
|
8a04d7b252
|
fix: revert placement of logging init if otel exports is disabled (#3797)
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
|
2025-10-21 17:36:31 -04:00 |
|
Ziqi Fan
|
d5840cda65
|
feat: enable KVBM GPU offload to Disk bypassing CPU (#3510)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-21 08:49:54 -07:00 |
|
Graham King
|
f6ed01b14c
|
chore: Replace ServiceConfigBuilder with add_stats_service (#3736)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-20 16:47:05 -04:00 |
|
Yan Ru Pei
|
4b7a806c7a
|
feat: add prefill workers to discovery (#3709)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-18 00:59:32 +00:00 |
|
Yan Ru Pei
|
4c207e0c8c
|
chore: remove kv metrics scraping and aggregation (#3701)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-17 10:58:48 -07:00 |
|
Yan Ru Pei
|
ae4e96a2bc
|
fix: mocker engines should ignore downloading weights from hf (again) (#3664)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-16 02:48:40 +00:00 |
|
Jacky
|
734d2f87a8
|
feat: Enable cancellation during or before a stream is established (#3635)
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
|
2025-10-15 19:22:55 -07:00 |
|
Yan Ru Pei
|
f978f4d1d6
|
feat: dp rank routing (#3597)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-16 01:48:27 +00:00 |
|
Ziqi Fan
|
f5854e1784
|
fix: avoid offload redundant prefill blocks | fix cuda graph hanging (#3632)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-15 10:38:35 -07:00 |
|
Graham King
|
ab0da5821e
|
feat: Python binding to download a model. (#3593)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-15 12:26:02 -04:00 |
|
Yan Ru Pei
|
b5e762b2b8
|
feat: remove stale workers on snapshot + some refactoring (#3589)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-13 21:36:09 +00:00 |
|
mohammedabdulwahhab
|
1f92dd547e
|
feat: OTEL Exporter and Tempo Visualization (#3307)
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
|
2025-10-13 18:39:09 +00:00 |
|
Yan Ru Pei
|
c3fcfdd6e3
|
chore: consolidations of KvPushRouter bindings and usage examples (#3543)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
|
2025-10-11 05:47:25 +00:00 |
|
Keiven C
|
65cc533702
|
feat: implement custom backend metrics for NIM (#3266)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-10 20:28:58 -07:00 |
|
Ziqi Fan
|
4224e57da0
|
feat: rm the old KVBM metrics | update G2 to G3 metrics collection (#3561)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-11 01:48:09 +00:00 |
|
Keiven C
|
55e458d899
|
feat: add SGLang and vLLM passthrough metrics on Dynamo backend worker (#3539)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-10 17:59:35 -07:00 |
|
Keiven C
|
a4746ab65f
|
fix: callback registration, fix metric name access, ensure ordered vec, etc... (#3541)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-11 00:03:24 +00:00 |
|
Graham King
|
0e0218ff02
|
chore: Remove model_config from LocalModel (#3558)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-10 22:01:30 +00:00 |
|
Keiven C
|
f2ba58e568
|
feat: add Rust-to-Python const code generator for prometheus_names.py (#3425)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-10 11:20:36 -07:00 |
|
Ziqi Fan
|
ca67409888
|
feat: add KVBM host to disk metrics | clean up dashboard (#3534)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-10 18:17:34 +00:00 |
|
Ziqi Fan
|
b94ecd1627
|
feat: add disk offloading filtering in KVBM (#3532)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
|
2025-10-10 09:51:55 -07:00 |
|
Graham King
|
e9a710095b
|
chore: Remove clear_namespace script (#3519)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-09 13:43:59 -04:00 |
|
Ziqi Fan
|
9975cb9ffe
|
feat: enable KVBM to support PD disagg in Dynamo vLLM (#3352)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-08 13:45:36 -07:00 |
|
Richard Huo
|
cf83794ae0
|
feat: DIS-678 kvbm modularity: standalone metrics endpoint (#3433)
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
|
2025-10-08 10:18:33 -07:00 |