ebaf048d16
chore: plumb allowed_worker_ids through RoutingHints ( #6580 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2026-02-25 12:12:53 -08:00
c916cd42ff
feat: Support epp's "pods" interface in Dynamo fixes [DEP-424] ( #6302 )
...
Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
2026-02-25 01:54:38 +00:00
75fea78797
feat: make KV cache events and routing LoRA-aware ( #6517 )
2026-02-24 18:35:08 +00:00
80cac7c14b
feat: Remove Component from public ( #6403 )
...
Signed-off-by: tzulingk@nvidia.com <tzulingk@nvidia.com>
2026-02-23 22:58:38 +00:00
038b50d2b6
feat: add standalone KV indexer with query endpoint [DYN-2164] ( #6446 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:12:25 -08:00
fc229004b5
chore: Remove ZmqKvEventListener binding and rework standalone TRT-LLM example to use native Python ZMQ ( #6164 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2026-02-19 04:15:13 +00:00
815b129126
feat: mm aware routing for vllm ( #6235 )
...
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
2026-02-13 21:16:06 -08:00
14eceb43df
chore: rename KvPushRouter to KvRouter in python + more bindings removal ( #6238 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2026-02-12 23:44:52 -08:00
bc514fbee6
feat: router priority queue ( #6010 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-02-13 02:43:17 +00:00
cd6984b951
feat: use RNG when dp routing targets are tied; override no-assume-kv-reuse for decode requests ( #6253 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com>
2026-02-12 15:59:37 -08:00
6a728d1044
chore: remove kv indexers bindings ( #6159 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2026-02-11 20:23:48 +00:00
f46720c996
feat: add router-level Prometheus metrics and centralize request tracking ( #6146 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-11 07:36:08 +00:00
6783bdcaa9
chore: enable local indexers by default, and use normal event plane by default (not jetstream) ( #5941 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2026-02-08 20:48:02 +00:00
3e41702211
chore: remove and unify bindings in kv.rs ( #6016 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2026-02-06 18:44:46 +00:00
0c0336e6ea
feat: Add per-worker Prometheus metrics for router load monitoring ( #5842 )
...
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
2026-02-04 11:51:43 -08:00
902eabd9ec
feat: per dp rank gap detection ( #5873 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2026-02-03 03:07:57 +00:00
373e76c1ae
feat(lora): Add lora_name tracking to scheduling and sequence management ( #5875 )
2026-02-03 00:59:09 +00:00
c5f5ab6056
fix: enable DP attention KV events for multi-node deployments ( #5589 )
...
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: huitianbai <huitianbai@gmail.com>
Co-authored-by: Huitianqi Bai <huitianbai@users.noreply.github.com>
2026-02-02 07:20:29 +00:00
1174c819cb
feat: remove deprecated events api ( #5778 )
2026-01-29 05:48:16 +00:00
fc92fc187b
chore: clean ups in kv-router ( #5771 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2026-01-29 03:20:04 +00:00
feb6d272ac
chore: nuke ForwardPassMetrics ( #5531 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2026-01-23 16:26:01 +00:00
c5458b9954
chore(router): Remove deprecated best_worker_id and metrics_labels ( #5412 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2026-01-14 16:20:42 +00:00
cf433e6825
chore: update all copyright headers in repo to 2026 ( #5130 )
...
Signed-off-by: Tushar Sharma <tusharma@nvidia.com>
2026-01-02 22:08:23 +00:00
0b33c1dfa0
fix: sglang disagg routing fixes and optimizations [DYN-1692] ( #5106 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
Co-authored-by: Sean SH Choi <sechoi@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2025-12-31 10:55:44 -08:00
f4245c9929
chore: clean up PreprocessedRequest ( #5040 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-12-19 23:15:59 +00:00
481dc6363b
feat: add multimodal support to KV router with standalone trtllm example ( #4577 )
...
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
2025-12-18 21:01:45 -08:00
85b5c808ac
feat: unify per-request metric tracking ( #5004 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-12-18 23:59:25 +00:00
c2a29f8045
chore: enable local indexers for sglang and trtllm ( #4932 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-12-12 11:41:37 -08:00
332499458b
feat: worker-local KvIndexer in KvEventPublisher ( #4519 )
...
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
2025-12-11 23:18:34 +00:00
c61e0dd3cc
chore: merge KvIndexer and ApproxKvIndexer ( #4500 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-11-21 09:09:11 -08:00
2f18b23ee7
chore: Use general engine source for KVBM KV events ( #4515 )
2025-11-20 14:47:48 -08:00
2cf427ed8f
fix(router): change prune channel to be mpsc instead of watch, increase max size ( #4351 )
...
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
2025-11-14 18:19:20 -05:00
8379b0cd10
feat: kv router should route to available instances ( #4225 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-11-13 02:34:58 +00:00
3ea22fcf39
feat(router): max tree size based pruning ( #4057 )
...
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
2025-11-12 08:33:49 -08:00
427ca9aba4
chore: no need for explicit Python::with_gil ( #4075 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-11-05 05:41:49 +00:00
defe5de74b
feat: release gil for publisher ( #3747 )
...
Signed-off-by: michaelfeil <me@michaelfeil.eu>
2025-11-05 02:17:06 +00:00
cc4c3516bb
feat: mocker disagg ( #3833 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-28 20:36:02 +00:00
c78b590134
chore(runtime): Do not expose etcd lease ID ( #3915 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-28 16:20:25 +00:00
7731b0245c
chore: Use KeyValueStoreManager instead of etcd::Client ( #3822 )
...
Signed-off-by: Graham King <grahamk@nvidia.com>
2025-10-23 14:09:58 -04:00
681951d4e6
feat: python gil release for radix tree + dump_tree_as_events in python ( #3748 )
...
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
2025-10-22 01:41:35 +00:00
e01c6e99bd
feat: bake prefill router into frontend, supporting vllm for now ( #3762 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-21 15:43:02 -07:00
4c207e0c8c
chore: remove kv metrics scraping and aggregation ( #3701 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-17 10:58:48 -07:00
f978f4d1d6
feat: dp rank routing ( #3597 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-16 01:48:27 +00:00
b5e762b2b8
feat: remove stale workers on snapshot + some refactoring ( #3589 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-13 21:36:09 +00:00
c3fcfdd6e3
chore: consolidations of KvPushRouter bindings and usage examples ( #3543 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-10-11 05:47:25 +00:00
30610e7371
feat: use KvPushRouter for prefill router ( #3401 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-10-03 18:21:48 -07:00
9b9536d0d7
feat: make prefill router general ( #3329 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-30 20:38:47 -07:00
d354763c40
fix: python bindings for router should register to etcd as well ( #3302 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-30 12:38:56 -07:00
5bb7490448
chore: bump vllm version to 0.10.2 ( #3180 )
...
Signed-off-by: Alec <aflowers@nvidia.com>
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Co-authored-by: Kris Hung <krish@nvidia.com>
2025-09-26 04:01:57 +00:00
031590fc14
feat: vllm prefill router ( #3155 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
2025-09-22 16:38:56 -07:00
Yan Ru Pei