mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,557 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

448 Commits

Author SHA1 Message Date
Yan Ru Pei 3e41702211
chore: remove and unify bindings in kv.rs (#6016) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-06 18:44:46 +00:00
Biswa Panda 953e5d7beb
feat(lora): implement HRW-based LoRA allocation system (#5992) 2026-02-05 14:56:15 -08:00
Keiven C cb7ebdd7e8
chore: remove unused NIM specific code (part 2) (#5893) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-02-05 12:21:10 -08:00
Yan Ru Pei 91fb78cddf
chore: make mocker it's own crate (#5958) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-05 19:35:42 +00:00
Hongkuan Zhou 0c0336e6ea
feat: Add per-worker Prometheus metrics for router load monitoring (#5842) 
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
 2026-02-04 11:51:43 -08:00
Konrad Nowicki 7e970d44c9
feat: image diffusion with SGLang diffusion (#5609) 
Signed-off-by: Konrad Nowicki <knowicki@nvidia.com>
Co-authored-by: dagil-nvidia <dagil@nvidia.com>
 2026-02-03 21:19:21 +00:00
milesial 9bff03f237
feat: vLLM backend with frontend media decoding (#5781) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2026-02-03 09:19:26 -08:00
Yan Ru Pei a337113aa9
feat: prefill tokens threshold based on max num batched tokens frac (#5867) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-03 03:08:25 +00:00
Yan Ru Pei 902eabd9ec
feat: per dp rank gap detection (#5873) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-02-03 03:07:57 +00:00
Biswa Panda 373e76c1ae
feat(lora): Add lora_name tracking to scheduling and sequence management (#5875) 2026-02-03 00:59:09 +00:00
Janelle Cai 18d9d1fa4f
feat(mocker): pre-fetch model and staggered launches (#5871) 2026-02-03 00:46:23 +00:00
ishandhanani c5f5ab6056
fix: enable DP attention KV events for multi-node deployments (#5589) 
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: huitianbai <huitianbai@gmail.com>
Co-authored-by: Huitianqi Bai <huitianbai@users.noreply.github.com>
 2026-02-02 07:20:29 +00:00
Biswa Panda 1174c819cb
feat: remove deprecated events api (#5778) 2026-01-29 05:48:16 +00:00
Yan Ru Pei fc92fc187b
chore: clean ups in kv-router (#5771) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-29 03:20:04 +00:00
Keiven C 5a00a7d643
fix: prevent duplicate Prometheus metrics from Python expfmt callbacks (#5761) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-01-28 23:34:33 +00:00
Keiven C 8e72fb69cf
docs: DYN-1967 update metrics docs after kvstats removal (#5704) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-01-27 17:24:12 -08:00
Dmitry Tokarev 2d6babe5fc
chore: Bump rust lib oneshot to 0.1.13 (#5694) 
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
 2026-01-27 17:26:11 -05:00
Pavithra Vijayakrishnan 66c369964a
chore: version update for 0.9.0 (#5661) 
Signed-off-by: pvijayakrish <pvijayakrish@nvidia.com>
 2026-01-26 17:18:05 -08:00
Qi Wang 89e135b9ce
chore: remove legacy KVBM pythong static type checks and test (#5486) 2026-01-26 12:46:49 -08:00
Abhishek Gupta 7de80960dc
feat: add __all__ exports and __repr__ methods for better debugging (#5606) 2026-01-23 12:44:03 -08:00
Keiven C 7361de4208
refactor: reduce Python API to only Prometheus Exposition Format callback (#5594) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2026-01-23 10:06:06 -08:00
Yan Ru Pei feb6d272ac
chore: nuke ForwardPassMetrics (#5531) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-23 16:26:01 +00:00
Nate Mailhot 6dba119d7a
chore: update nixl to 0.9.0 (#5528) 2026-01-22 15:25:03 -08:00
Graham King ca31b3aa6b
chore: Remove binding OAIChatPreprocessor (#5516) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-22 08:43:02 +08:00
Michael Feil fa66f65437
feat: typing - Add py.typed support for typing (#4913) 
Signed-off-by: Michael Feil <63565275+michaelfeil@users.noreply.github.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>
 2026-01-20 15:28:44 +00:00
Schwinn Saereesitthipitak d953f9d038
feat: add endpoint instance registration/unregistration for sleep/wake (#5203) 
Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-20 12:17:58 +00:00
Yan Ru Pei 050906b593
feat: track output tokens / blocks in the Router (optional) (#5452) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-18 09:30:25 +08:00
Graham King a6e7348400
chore(bindings): Remove 'Backend' binding (#5458) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-16 12:29:50 -05:00
Graham King c5458b9954
chore(router): Remove deprecated best_worker_id and metrics_labels (#5412) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-14 16:20:42 +00:00
Neal Vaidya a936ba5ad2
fix: read store_kv arg from env when using decorator (#5421) 
Signed-off-by: Neal Vaidya <nealv@nvidia.com>
 2026-01-14 03:27:34 +00:00
Yan Ru Pei 7fdc742e17
feat: allow router not assuming decode kv reuse (#5350) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-01-13 07:33:47 +08:00
Graham King f6ae58e349
fix(llm): Send HF URLs to frontend not full paths (#5290) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2026-01-12 12:05:49 -05:00
Biswa Panda 0d3ff44000
fix: enable toggling kv events pub/sub (currently nats based) with --no-kv-events flag (#5237) 2026-01-09 06:36:51 +00:00
Neelay Shah d22d9e761e
fix: support truthy values for OTEL_EXPORT_ENABLED in bindings layer [DYN-1690][DYN-1646] (#5129) 
Co-authored-by: Claude <noreply@anthropic.com>
 2026-01-02 23:39:09 +00:00
Tushar Sharma cf433e6825
chore: update all copyright headers in repo to 2026 (#5130) 
Signed-off-by: Tushar Sharma <tusharma@nvidia.com>
 2026-01-02 22:08:23 +00:00
Yan Ru Pei 0b33c1dfa0
fix: sglang disagg routing fixes and optimizations [DYN-1692] (#5106) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
Co-authored-by: Sean SH Choi <sechoi@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2025-12-31 10:55:44 -08:00
Phillippe Siclait ac8c9023e4
fix: Truncate HttpError message to first 8192 characters (#5020) 
Signed-off-by: Phillippe Siclait <546332+siclait@users.noreply.github.com>
 2025-12-22 12:23:35 -08:00
Yan Ru Pei f4245c9929
chore: clean up PreprocessedRequest (#5040) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-19 23:15:59 +00:00
Jacky e6a6a1f244
feat: Request Migration Metrics (#5029) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2025-12-19 12:22:36 -08:00
Nate Mailhot 910b587db1
chore: update nixl to 0.8.0 and ucx 1.20 (#5007) 
Signed-off-by: Nate Mailhot <nmailhot@nvidia.com>
 2025-12-19 10:55:40 -08:00
Anant Sharma 7436e3bc03
chore: update versions for 0.8.0 release (#5031) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2025-12-19 10:21:52 -08:00
milesial d2faf0e6f5
feat: Runtime media decoder config (#5011) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-12-19 05:34:09 +00:00
zhongdaor-nv 481dc6363b
feat: add multimodal support to KV router with standalone trtllm example (#4577) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
 2025-12-18 21:01:45 -08:00
Keiven C c22280ccf4
refactor: frontend/grpc tests to use dynamic ports (#4992) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-12-18 16:21:28 -08:00
Yan Ru Pei 85b5c808ac
feat: unify per-request metric tracking (#5004) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-18 23:59:25 +00:00
Biswa Panda 18b64e90b6
feat: switch to tcp as default request plane (#4845) 2025-12-18 23:05:38 +00:00
Anant Sharma 5b82b8b096
chore: update versions to match latest release (#4966) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2025-12-18 08:45:44 -08:00
Graham King da0f2fb84a
feat(frontend): First part of Python request handling (#4999) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-12-17 20:03:39 -05:00
Sean SH Choi 1c39e4835c
feat: add frontend based prefill request routing for sglang (#4635) 
Signed-off-by: Sean Choi <sechoi@nvidia.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
 2025-12-17 09:51:46 -08:00
milesial 74fcd4a9b9
feat: video decoder in the frontend (#4719) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-12-16 14:28:06 -08:00
KrishnanPrash 24ac6c1588
chore: bump minijinja to 2.14.0 (#4949) 
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
 2025-12-15 09:58:52 -08:00
Yan Ru Pei c2a29f8045
chore: enable local indexers for sglang and trtllm (#4932) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-12 11:41:37 -08:00
Karen Chung 332499458b
feat: worker-local KvIndexer in KvEventPublisher (#4519) 
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
 2025-12-11 23:18:34 +00:00
Yan Ru Pei 10b01b4590
feat: early rejection based on active prefill tokens (#4837) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-12-11 23:16:02 +00:00
Dmitry Tokarev 2ef0f5fbab
fix: relax pydantic requirements (#4865) 
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
 2025-12-11 19:32:19 +00:00
Vladislav Nosivskoy f8bb53c0ed
feat: add cached tokens prometheus metric (#4534) 
Signed-off-by: Vladislav Nosivskoy <vladnosiv@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-12-11 08:03:12 +03:00
Jacky 00f8615e47
test: Ensure Request Cancellation, Migration, Rejection Work with TCP transport (#4875) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2025-12-11 02:21:17 +00:00
J Wyman 96fe63fe2e
feat: nixl_connect: Improve Concurrency Support (#4433) 
Signed-off-by: J Wyman <jwyman@nvidia.com>
 2025-12-09 17:29:22 -05:00
Keiven C 1e37c1033f
fix: add timeout to tests before lora to unblock (#4808) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-12-08 20:50:33 -08:00
Biswa Panda 5588d77074
test: add timeout for lora utils tests (#4781) 2025-12-06 02:23:20 +00:00
William Arnold 3bc129dc37
feat: Add python-configurable engine routes for sglang profiling (#4617) 
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
 2025-12-05 10:28:36 -08:00
Neal Vaidya 118323f26e
fix: skip HuggingFace download for non-llms (#4686) 
Signed-off-by: Neal Vaidya <nealv@nvidia.com>
 2025-12-04 10:23:24 -08:00
Biswa Panda 71f94eda23
feat: lora - centralize lora cache key, restructure folders, s3 resiliency (#4644) 2025-12-02 08:57:25 +00:00
Biswa Panda 6a0e67ed78
feat: add LoRA common APIs and implementation for lora management (#4464) 2025-11-26 20:29:25 +00:00
Graham King fcb91e4b3d
refactor(storage): Remove the stuttering from key_value_store. (#4604) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-25 22:59:29 +00:00
milesial 0a06591826
feat: decoded media via NIXL (#3988) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
Signed-off-by: Ayush Agarwal <ayushag@nvidia.com>
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
Co-authored-by: Ayush Agarwal <ayushag@nvidia.com>
Co-authored-by: Krishnan Prashanth <kprashanth@nvidia.com>
Co-authored-by: KrishnanPrash <140860868+KrishnanPrash@users.noreply.github.com>
 2025-11-25 19:15:04 +00:00
Graham King 0fc5273c11
refactor(llm): Rename EngineConfig::Static to InProcess (#4585) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-25 12:22:09 -05:00
Graham King 27904535fb
fix(storage): Correctly encoding FileStore keys (#4539) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-21 17:01:40 -05:00
Yan Ru Pei c61e0dd3cc
chore: merge KvIndexer and ApproxKvIndexer (#4500) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-21 09:09:11 -08:00
Yan Ru Pei d821a8b9f7
chore: parallelize planner profile tests + bindings test cleanup (#4532) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-21 09:21:43 +00:00
Kris Hung 2f18b23ee7
chore: Use general engine source for KVBM KV events (#4515) 2025-11-20 14:47:48 -08:00
Graham King 8a14f9edc1
chore: Remove DistributedRuntime::etcd_client (#4489) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-20 12:35:10 -05:00
Biswa Panda 781331c6f3
feat: unregister discovery instance (#4459) 2025-11-19 20:25:44 +00:00
Graham King 69797b5ab3
feat: Only monitor NATS metrics if using NATS request plane (#4442) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-19 17:48:49 +00:00
Biswa Panda 684107c461
feat: generate deterministic lora id from lora name (#4457) 2025-11-19 02:26:00 +00:00
Graham King 2f1778c13d
fix(storage): File store shouldn't notify on metdata changes (#4434) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-18 10:24:04 -08:00
J Wyman 0814653104
feat: nixl_connect: Change the way `bytes` are passed to NIXL (#4314) 
Signed-off-by: J Wyman <jwyman@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-11-18 18:05:46 +00:00
Yan Ru Pei fc2ad4eb1e
feat: mocker can use planner profile data (#4422) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Hongkuan Zhou <tedzhouhk@gmail.com>
 2025-11-18 08:06:50 +00:00
Keiven C 0e77d3442e
refactor: centralize environment variable constants (#4083) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-11-17 11:16:38 -08:00
Graham King 886506c12d
feat: Command line flag to set request plane mode: tcp, http or nats (#4365) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-17 09:02:36 -08:00
Waël Boukhobza 2cf427ed8f
fix(router): change prune channel to be mpsc instead of watch, increase max size (#4351) 
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
 2025-11-14 18:19:20 -05:00
Yan Ru Pei ce83398366
chore: better error handling in prefill router (#4286) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-13 22:29:26 +00:00
Biswa Panda 06b0ebef88
feat: transport agnostic request plane for dynamo - natless (#4246) 2025-11-13 17:54:54 +00:00
Yan Ru Pei 8379b0cd10
feat: kv router should route to available instances (#4225) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-13 02:34:58 +00:00
Yan Ru Pei 38242c8d7b
chore: use notify instead of inotify (#4250) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-11-12 16:53:39 +00:00
Waël Boukhobza 3ea22fcf39
feat(router): max tree size based pruning (#4057) 
Signed-off-by: Wael Boukhobza <wawa_wael@live.fr>
 2025-11-12 08:33:49 -08:00
Graham King fbad28602e
chore: Remove the python bindings for port allocation (#4237) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-11 20:35:36 +00:00
Graham King e1af3af6ee
chore: Remove static mode (#4235) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-11 19:25:12 +00:00
Aryan Bagade 8f868986e8
feat: Add output token counter to frontend metrics (#4202) 
Signed-off-by: Aryan Bagade <aryan@aryanbagade.com>
 2025-11-11 05:09:07 +00:00
mohammedabdulwahhab 7afb5431ff
fix: add kube impl for discovery and add metadata endpoint (#4136) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
Co-authored-by: tmontfort <tmontfort@nvidia.com>
 2025-11-10 22:42:04 +00:00
zhongdaor-nv ec7af93953
fix: Extend add_tensor_model so that ModelDeploymentCard can be correctly picked up (#4169) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
 2025-11-10 13:59:40 -08:00
Anant Sharma 23660bc5c3
chore: update versions for 0.7.0 release (#4217) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
 2025-11-10 21:40:06 +00:00
Keiven C afccc9d434
refactor: consolidate Observability files (e.g. OTEL docker-compose, md files) (#4173) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-11-10 20:19:34 +00:00
Graham King cf630bf7b6
refactor: Make the Runtime and DistributedRuntime fields private (#4193) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-10 15:56:22 +00:00
mohammedabdulwahhab 09b26bf6b3
fix: refactor to use service discovery (#4092) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
 2025-11-08 18:05:18 +00:00
milesial 14af074ecd
feat: Media decoder and fetcher options in the MDC (#4094) 
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
 2025-11-08 02:17:39 +00:00
Keiven C 6bccf099f8
feat: deprecate DYN_SYSTEM_ENABLED in favor of DYN_SYSTEM_PORT (#4082) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-11-07 15:31:51 -08:00
Graham King 794c0a4431
feat(keyvalue): Filesystem backed KeyValueStore (#4138) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-07 21:11:33 +00:00
KrishnanPrash 3d036fc4a4
feat: Reject unsupported parameters with 400 Bad Request (`/v1/completions`) (#4140) 
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
 2025-11-05 17:32:50 -08:00
Jacky 6e2b22eaef
feat: ETCD high availability client failover - lease watch resilience (#3950) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-11-05 17:21:09 -08:00