Tushar Sharma
|
cf433e6825
|
chore: update all copyright headers in repo to 2026 (#5130)
Signed-off-by: Tushar Sharma <tusharma@nvidia.com>
|
2026-01-02 22:08:23 +00:00 |
|
Yan Ru Pei
|
0b33c1dfa0
|
fix: sglang disagg routing fixes and optimizations [DYN-1692] (#5106)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
Co-authored-by: Sean SH Choi <sechoi@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
|
2025-12-31 10:55:44 -08:00 |
|
Yan Ru Pei
|
f4245c9929
|
chore: clean up PreprocessedRequest (#5040)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-12-19 23:15:59 +00:00 |
|
milesial
|
d2faf0e6f5
|
feat: Runtime media decoder config (#5011)
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
|
2025-12-19 05:34:09 +00:00 |
|
zhongdaor-nv
|
481dc6363b
|
feat: add multimodal support to KV router with standalone trtllm example (#4577)
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
|
2025-12-18 21:01:45 -08:00 |
|
Keiven C
|
c22280ccf4
|
refactor: frontend/grpc tests to use dynamic ports (#4992)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-12-18 16:21:28 -08:00 |
|
Yan Ru Pei
|
85b5c808ac
|
feat: unify per-request metric tracking (#5004)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-12-18 23:59:25 +00:00 |
|
Graham King
|
da0f2fb84a
|
feat(frontend): First part of Python request handling (#4999)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-12-17 20:03:39 -05:00 |
|
Sean SH Choi
|
1c39e4835c
|
feat: add frontend based prefill request routing for sglang (#4635)
Signed-off-by: Sean Choi <sechoi@nvidia.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
|
2025-12-17 09:51:46 -08:00 |
|
milesial
|
74fcd4a9b9
|
feat: video decoder in the frontend (#4719)
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
|
2025-12-16 14:28:06 -08:00 |
|
Yan Ru Pei
|
c2a29f8045
|
chore: enable local indexers for sglang and trtllm (#4932)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-12-12 11:41:37 -08:00 |
|
Karen Chung
|
332499458b
|
feat: worker-local KvIndexer in KvEventPublisher (#4519)
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
|
2025-12-11 23:18:34 +00:00 |
|
Yan Ru Pei
|
10b01b4590
|
feat: early rejection based on active prefill tokens (#4837)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-12-11 23:16:02 +00:00 |
|
Biswa Panda
|
71f94eda23
|
feat: lora - centralize lora cache key, restructure folders, s3 resiliency (#4644)
|
2025-12-02 08:57:25 +00:00 |
|
Biswa Panda
|
6a0e67ed78
|
feat: add LoRA common APIs and implementation for lora management (#4464)
|
2025-11-26 20:29:25 +00:00 |
|
Graham King
|
0fc5273c11
|
refactor(llm): Rename EngineConfig::Static to InProcess (#4585)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-11-25 12:22:09 -05:00 |
|
Yan Ru Pei
|
c61e0dd3cc
|
chore: merge KvIndexer and ApproxKvIndexer (#4500)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-11-21 09:09:11 -08:00 |
|
Kris Hung
|
2f18b23ee7
|
chore: Use general engine source for KVBM KV events (#4515)
|
2025-11-20 14:47:48 -08:00 |
|
Waël Boukhobza
|
2cf427ed8f
|
fix(router): change prune channel to be mpsc instead of watch, increase max size (#4351)
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
|
2025-11-14 18:19:20 -05:00 |
|
Yan Ru Pei
|
ce83398366
|
chore: better error handling in prefill router (#4286)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-11-13 22:29:26 +00:00 |
|
Yan Ru Pei
|
8379b0cd10
|
feat: kv router should route to available instances (#4225)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-11-13 02:34:58 +00:00 |
|
Waël Boukhobza
|
3ea22fcf39
|
feat(router): max tree size based pruning (#4057)
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
|
2025-11-12 08:33:49 -08:00 |
|
Graham King
|
e1af3af6ee
|
chore: Remove static mode (#4235)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-11-11 19:25:12 +00:00 |
|
milesial
|
14af074ecd
|
feat: Media decoder and fetcher options in the MDC (#4094)
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
|
2025-11-08 02:17:39 +00:00 |
|
Graham King
|
794c0a4431
|
feat(keyvalue): Filesystem backed KeyValueStore (#4138)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-11-07 21:11:33 +00:00 |
|
Yan Ru Pei
|
427ca9aba4
|
chore: no need for explicit Python::with_gil (#4075)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-11-05 05:41:49 +00:00 |
|
Michael Feil
|
defe5de74b
|
feat: release gil for publisher (#3747)
Signed-off-by: michaelfeil <me@michaelfeil.eu>
|
2025-11-05 02:17:06 +00:00 |
|
Graham King
|
dadf0e2247
|
chore: Remove old DisaggregatedRouter, making etcd presence optional (#4011)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-11-03 10:10:13 -05:00 |
|
Richard Huo
|
6afa679c5d
|
chore: KVBM pip wheel (#3826)
Signed-off-by: Anant Sharma <anants@nvidia.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>
|
2025-10-31 14:19:30 -07:00 |
|
Kris Hung
|
1898601025
|
feat: Add KV event consolidator for KVBM (vllm) and router integration (#3725)
Signed-off-by: krishung5 <krish@nvidia.com>
|
2025-10-30 17:11:42 +00:00 |
|
Yan Ru Pei
|
cc4c3516bb
|
feat: mocker disagg (#3833)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-28 20:36:02 +00:00 |
|
Graham King
|
c78b590134
|
chore(runtime): Do not expose etcd lease ID (#3915)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-28 16:20:25 +00:00 |
|
Kyle McGill
|
0028cdf43d
|
feat: Use ForwardPassCallback api from TRTLLM to register end of forward pass callback to enable cuda graphs (#3297)
Signed-off-by: Kyle McGill <kmcgill@nvidia.com>
|
2025-10-27 15:12:04 -07:00 |
|
Graham King
|
7731b0245c
|
chore: Use KeyValueStoreManager instead of etcd::Client (#3822)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-23 14:09:58 -04:00 |
|
Yan Ru Pei
|
41ff394f51
|
chore: restructure mocker cli args handling, to include prefill/decode (#3847)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-23 10:41:23 -07:00 |
|
Richard Huo
|
94aa2a7bbb
|
refactor: kvbm modularity DIS-657 Eliminate ETCD from the leader-worker initialization (#3202)
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
|
2025-10-23 00:10:09 -07:00 |
|
Michael Feil
|
681951d4e6
|
feat: python gil release for radix tree + dump_tree_as_events in python (#3748)
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
|
2025-10-22 01:41:35 +00:00 |
|
Yan Ru Pei
|
e01c6e99bd
|
feat: bake prefill router into frontend, supporting vllm for now (#3762)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-21 15:43:02 -07:00 |
|
Ziqi Fan
|
d5840cda65
|
feat: enable KVBM GPU offload to Disk bypassing CPU (#3510)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-21 08:49:54 -07:00 |
|
Yan Ru Pei
|
4c207e0c8c
|
chore: remove kv metrics scraping and aggregation (#3701)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-17 10:58:48 -07:00 |
|
Yan Ru Pei
|
ae4e96a2bc
|
fix: mocker engines should ignore downloading weights from hf (again) (#3664)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-16 02:48:40 +00:00 |
|
Yan Ru Pei
|
f978f4d1d6
|
feat: dp rank routing (#3597)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-16 01:48:27 +00:00 |
|
Ziqi Fan
|
f5854e1784
|
fix: avoid offload redundant prefill blocks | fix cuda graph hanging (#3632)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-15 10:38:35 -07:00 |
|
Graham King
|
ab0da5821e
|
feat: Python binding to download a model. (#3593)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-15 12:26:02 -04:00 |
|
Yan Ru Pei
|
b5e762b2b8
|
feat: remove stale workers on snapshot + some refactoring (#3589)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
|
2025-10-13 21:36:09 +00:00 |
|
Yan Ru Pei
|
c3fcfdd6e3
|
chore: consolidations of KvPushRouter bindings and usage examples (#3543)
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
|
2025-10-11 05:47:25 +00:00 |
|
Keiven C
|
65cc533702
|
feat: implement custom backend metrics for NIM (#3266)
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
|
2025-10-10 20:28:58 -07:00 |
|
Ziqi Fan
|
4224e57da0
|
feat: rm the old KVBM metrics | update G2 to G3 metrics collection (#3561)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-11 01:48:09 +00:00 |
|
Graham King
|
0e0218ff02
|
chore: Remove model_config from LocalModel (#3558)
Signed-off-by: Graham King <grahamk@nvidia.com>
|
2025-10-10 22:01:30 +00:00 |
|
Ziqi Fan
|
ca67409888
|
feat: add KVBM host to disk metrics | clean up dashboard (#3534)
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
|
2025-10-10 18:17:34 +00:00 |