Yan Ru Pei
|
cc4c3516bb
|
feat: mocker disagg (#3833)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-28 20:36:02 +00:00 |
|
Graham King
|
c78b590134
|
chore(runtime): Do not expose etcd lease ID (#3915)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-28 16:20:25 +00:00 |
|
Kyle McGill
|
0028cdf43d
|
feat: Use ForwardPassCallback api from TRTLLM to register end of forward pass callback to enable cuda graphs (#3297)
Signed-off-by: Kyle McGill <kmcgill@nvidia.com>
|
2025-10-27 15:12:04 -07:00 |
|
Keiven C
|
cbe0b177ab
|
refactor: redesign the metrics API from Trait to composition to make the code cleaner and easier to understand (#3687)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-24 14:34:29 -07:00 |
|
zhongdaor-nv
|
9e5407f20d
|
feat: python binding for kserve grpc frontend (#3739)
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
|
2025-10-24 11:20:20 -07:00 |
|
Graham King
|
7731b0245c
|
chore: Use KeyValueStoreManager instead of etcd::Client (#3822)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-23 14:09:58 -04:00 |
|
Yan Ru Pei
|
41ff394f51
|
chore: restructure mocker cli args handling, to include prefill/decode (#3847)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-23 10:41:23 -07:00 |
|
Richard Huo
|
94aa2a7bbb
|
refactor: kvbm modularity DIS-657 Eliminate ETCD from the leader-worker initialization (#3202)
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
|
2025-10-23 00:10:09 -07:00 |
|
Michael Feil
|
681951d4e6
|
feat: python gil release for radix tree + dump_tree_as_events in python (#3748)
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
|
2025-10-22 01:41:35 +00:00 |
|
Yan Ru Pei
|
e01c6e99bd
|
feat: bake prefill router into frontend, supporting vllm for now (#3762)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-21 15:43:02 -07:00 |
|
mohammedabdulwahhab
|
8a04d7b252
|
fix: revert placement of logging init if otel exports is disabled (#3797)
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
|
2025-10-21 17:36:31 -04:00 |
|
Ziqi Fan
|
d5840cda65
|
feat: enable KVBM GPU offload to Disk bypassing CPU (#3510)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-21 08:49:54 -07:00 |
|
Graham King
|
f6ed01b14c
|
chore: Replace ServiceConfigBuilder with add_stats_service (#3736)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-20 16:47:05 -04:00 |
|
Yan Ru Pei
|
4b7a806c7a
|
feat: add prefill workers to discovery (#3709)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-18 00:59:32 +00:00 |
|
Yan Ru Pei
|
4c207e0c8c
|
chore: remove kv metrics scraping and aggregation (#3701)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-17 10:58:48 -07:00 |
|
Yan Ru Pei
|
ae4e96a2bc
|
fix: mocker engines should ignore downloading weights from hf (again) (#3664)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-16 02:48:40 +00:00 |
|
Jacky
|
734d2f87a8
|
feat: Enable cancellation during or before a stream is established (#3635)
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
|
2025-10-15 19:22:55 -07:00 |
|
Yan Ru Pei
|
f978f4d1d6
|
feat: dp rank routing (#3597)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-16 01:48:27 +00:00 |
|
Ziqi Fan
|
f5854e1784
|
fix: avoid offload redundant prefill blocks | fix cuda graph hanging (#3632)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-15 10:38:35 -07:00 |
|
Graham King
|
ab0da5821e
|
feat: Python binding to download a model. (#3593)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-15 12:26:02 -04:00 |
|
Yan Ru Pei
|
b5e762b2b8
|
feat: remove stale workers on snapshot + some refactoring (#3589)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-13 21:36:09 +00:00 |
|
mohammedabdulwahhab
|
1f92dd547e
|
feat: OTEL Exporter and Tempo Visualization (#3307)
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
|
2025-10-13 18:39:09 +00:00 |
|
Yan Ru Pei
|
c3fcfdd6e3
|
chore: consolidations of KvPushRouter bindings and usage examples (#3543)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
|
2025-10-11 05:47:25 +00:00 |
|
Keiven C
|
65cc533702
|
feat: implement custom backend metrics for NIM (#3266)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-10 20:28:58 -07:00 |
|
Ziqi Fan
|
4224e57da0
|
feat: rm the old KVBM metrics | update G2 to G3 metrics collection (#3561)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-11 01:48:09 +00:00 |
|
Keiven C
|
55e458d899
|
feat: add SGLang and vLLM passthrough metrics on Dynamo backend worker (#3539)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-10 17:59:35 -07:00 |
|
Keiven C
|
a4746ab65f
|
fix: callback registration, fix metric name access, ensure ordered vec, etc... (#3541)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-11 00:03:24 +00:00 |
|
Graham King
|
0e0218ff02
|
chore: Remove model_config from LocalModel (#3558)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-10 22:01:30 +00:00 |
|
Keiven C
|
f2ba58e568
|
feat: add Rust-to-Python const code generator for prometheus_names.py (#3425)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-10 11:20:36 -07:00 |
|
Ziqi Fan
|
ca67409888
|
feat: add KVBM host to disk metrics | clean up dashboard (#3534)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-10 18:17:34 +00:00 |
|
Ziqi Fan
|
b94ecd1627
|
feat: add disk offloading filtering in KVBM (#3532)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
|
2025-10-10 09:51:55 -07:00 |
|
Graham King
|
e9a710095b
|
chore: Remove clear_namespace script (#3519)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-09 13:43:59 -04:00 |
|
Ziqi Fan
|
9975cb9ffe
|
feat: enable KVBM to support PD disagg in Dynamo vLLM (#3352)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-08 13:45:36 -07:00 |
|
Richard Huo
|
cf83794ae0
|
feat: DIS-678 kvbm modularity: standalone metrics endpoint (#3433)
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
|
2025-10-08 10:18:33 -07:00 |
|
Keiven C
|
0c4c4d1de3
|
feat: add Python MetricsRegistry Python metrics registration (#3341)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-07 21:24:25 -07:00 |
|
Graham King
|
bdad6f1a50
|
fix: Make planner VirtualConnectorClient also use v1/ prefix. (#3468)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-07 22:34:30 +00:00 |
|
Graham King
|
a5371bfc50
|
feat(etcd): Version the etcd keys (#3458)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-07 16:48:26 +00:00 |
|
Graham King
|
81162dfeb9
|
chore(discovery): Watch/publish ModelDeploymentCard instead of ModelEntry (#3350)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-07 10:02:02 -04:00 |
|
Yan Ru Pei
|
30610e7371
|
feat: use KvPushRouter for prefill router (#3401)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-03 18:21:48 -07:00 |
|
Olga Andreeva
|
d2e3b66e65
|
feat: Transition to FullyContiguous Host and Disk layouts (#3090)
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: oandreeva-nv <oandreeva-nv@nvidia.com>
|
2025-10-01 16:26:39 -07:00 |
|
Richard Huo
|
713e9e481b
|
fix: DIS-706 skip offloading the G1 matched blocks during offloading (#3299)
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
|
2025-10-01 10:28:06 -07:00 |
|
Yan Ru Pei
|
9b9536d0d7
|
feat: make prefill router general (#3329)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-09-30 20:38:47 -07:00 |
|
Keiven C
|
f4a3a6b66a
|
refactor: standardize Prometheus metric naming conventions (part 1) (#3035)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-09-30 19:01:39 -07:00 |
|
Michael Feil
|
5b457b70a6
|
feat: python add abi compatability for cross-platform builds + add a unit test to HttpServer (#3044)
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Signed-off-by: Michael Feil <63565275+michaelfeil@users.noreply.github.com>
Signed-off-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Signed-off-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
|
2025-09-30 14:42:42 -07:00 |
|
Yan Ru Pei
|
d354763c40
|
fix: python bindings for router should register to etcd as well (#3302)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-09-30 12:38:56 -07:00 |
|
Keiven C
|
cacac9b9f4
|
feat: add Python const for Prometheus metric names (#3244)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-09-30 10:28:47 -07:00 |
|
Yan Ru Pei
|
3aa3077808
|
fix: more fixes for stable router benchmarking (#3264)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-09-29 23:08:56 +00:00 |
|
Ziqi Fan
|
e21dcf6cab
|
feat: enable KVBM emit metrics in Dynamo TRTLLM (#3254)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-09-29 08:48:51 -07:00 |
|
Graham King
|
7ebbd001c0
|
chore: Remove etcd from python bindings (#3238)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-09-26 12:58:30 -04:00 |
|
Alec
|
5bb7490448
|
chore: bump vllm version to 0.10.2 (#3180)
Signed-off-by: Alec <aflowers@nvidia.com>
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Co-authored-by: Kris Hung <krish@nvidia.com>
|
2025-09-26 04:01:57 +00:00 |