mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,346 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

42 Commits

Author SHA1 Message Date
Graham King e1af3af6ee
chore: Remove static mode (#4235) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-11 19:25:12 +00:00
zhongdaor-nv ec7af93953
fix: Extend add_tensor_model so that ModelDeploymentCard can be correctly picked up (#4169) 
Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
 2025-11-10 13:59:40 -08:00
mohammedabdulwahhab 09b26bf6b3
fix: refactor to use service discovery (#4092) 
Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
 2025-11-08 18:05:18 +00:00
Graham King 794c0a4431
feat(keyvalue): Filesystem backed KeyValueStore (#4138) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-11-07 21:11:33 +00:00
Richard Huo 6afa679c5d
chore: KVBM pip wheel (#3826) 
Signed-off-by: Anant Sharma <anants@nvidia.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>
 2025-10-31 14:19:30 -07:00
Michael Feil 681951d4e6
feat: python gil release for radix tree + dump_tree_as_events in python (#3748) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
 2025-10-22 01:41:35 +00:00
Yan Ru Pei 4c207e0c8c
chore: remove kv metrics scraping and aggregation (#3701) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-17 10:58:48 -07:00
Jacky 734d2f87a8
feat: Enable cancellation during or before a stream is established (#3635) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-10-15 19:22:55 -07:00
Yan Ru Pei f978f4d1d6
feat: dp rank routing (#3597) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-10-16 01:48:27 +00:00
Keiven C 0c4c4d1de3
feat: add Python MetricsRegistry Python metrics registration (#3341) 
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-10-07 21:24:25 -07:00
Graham King 81162dfeb9
chore(discovery): Watch/publish ModelDeploymentCard instead of ModelEntry (#3350) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-10-07 10:02:02 -04:00
Michael Feil 5b457b70a6
feat: python add abi compatability for cross-platform builds + add a unit test to HttpServer (#3044) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Signed-off-by: Michael Feil <63565275+michaelfeil@users.noreply.github.com>
Signed-off-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Signed-off-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeil2-dev-pod-b200-0.michaelfeil2-dev-pod-b200.baseten.svc.cluster.local>
Co-authored-by: root <root@michaelfeildns-dev-pod-h100-0.michaelfeildns-dev-pod-h100.baseten.svc.cluster.local>
 2025-09-30 14:42:42 -07:00
GuanLuo 6ba64c31f5
feat: tensor type for generic inference. (#2746) 
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-09-24 07:31:09 +00:00
Jacky 31c78df781
refactor: PyTest subprocess for cancellation unit tests (#3127) 
Signed-off-by: michaelfeil <me@michaelfeil.eu>
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: michaelfeil <me@michaelfeil.eu>
 2025-09-22 10:56:37 -07:00
Graham King 3865a94148
feat: Port vllm port allocator to Rust in bindings (#3125) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-19 18:34:52 +00:00
Jacky a8fd127130
feat: Request Cancellation unary request support (#3004) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2025-09-18 22:26:37 -07:00
Graham King b6595e2484
chore(bindings): Provide a binding to clear etcd namespace (#3094) 
Signed-off-by: Graham King <grahamk@nvidia.com>
 2025-09-18 11:49:08 -04:00
Alec c8ecc4025f
ci: fix completions race (#3010) 
Signed-off-by: alec-flowers <aflowers@nvidia.com>
 2025-09-11 19:46:59 +00:00
Jacky de5ab2c4af
docs: Python runtime Request Cancellation examples (#2893) 
Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
 2025-09-10 19:23:30 +00:00
Ayush Agarwal 87a721a83e
feat: added parser name bindings (#2808) 
Signed-off-by: Ayush Agarwal <ayushag@nvidia.com>
 2025-09-02 20:37:29 +00:00
Jacky 6c539fbdac
feat: FT Request Cancellation feature and test for 0.5.0 (#2500) 2025-09-02 08:26:21 -07:00
Yan Ru Pei 488c87095c
feat: Router warm restarts via durable KV event consumers and radix snapshotting (#2756) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-08-30 23:42:57 +00:00
Ryan Olson 07cfc3a11b
feat: kvbm + connector (#2258) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
Co-authored-by: Olga Andreeva <oandreeva@nvidia.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: John Thompson <jothomson@nvidia.com>
Co-authored-by: Richard Huo <rihuo@nvidia.com>
Co-authored-by: Zicheng Ma <zichengm@nvidia.com>
 2025-08-19 12:36:53 -07:00
Harrison Saturley-Hall ffae72b7d2
fix: remove kvmanager feature from python 3.12 ai-dynamo-runtime wheel (#2456) 2025-08-15 16:43:55 -04:00
Ryan McCormick 9f2356cb9e
chore: Remove unused portion of kv bindings test (#2052) 2025-07-22 19:12:42 +00:00
jain-ria 439e977d9c
feat: vllm speculative decoding metrics (#1549) 
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
 2025-07-07 23:40:43 +00:00
jthomson04 aaf283bbb8
feat: Approximate KV Routing (#1636) 2025-06-30 20:34:08 -07:00
Yan Ru Pei 13a99b7f76
feat: Standalone Router (#1409) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: jain-ria <riajain@NVIDIA.com>
 2025-06-14 19:02:31 +00:00
Alec 2f8da9ad1d
refactor: rename KvMetricsPublisher to WorkerMetricsPublisher (#1284) 2025-05-30 17:23:32 +00:00
Jacky 7677f74f37
feat: KVBM async Python bindings and Layer class (#1141) 2025-05-29 17:49:05 +00:00
Ryan Olson 80256acf15
feat: adding outer dimension to isolate k/v blocks (#1126) 2025-05-20 08:51:41 -04:00
Jacky 437cae0ad0
feat: KV Block Manager Python bindings (#1022) 2025-05-19 12:33:41 -04:00
Graham King 88ad3425c4 feat: Python decorator dynamo_worker takes optional `static` parameter without etcd (#494) 
Adds `@dynamo_worker(static = True)` to create a static worker which has a predictable name and hence does not require discovery or `etcd` to be running. There can only be a single static worker per namespace / component / endpoint trio.

This contrasts with the default dynamic `dynamo_worker` endpoints we have now, which get a unique random name (based on namespace/component/endpoint), and are discovered by ingress components using etcd.

Also change the hello_world example to use `dynamo_worker(static = True)` so that it is exercised and demonstrated somewhere.

For NIM.
 2025-04-04 09:26:52 -04:00
GuanLuo 6e09681e0e feat: expose Python binding for KVEventPublisher. Use event pub/sub trait for KV events (#169) 2025-03-17 00:17:33 -07:00
Alec 3f84cdadfa feat: add new metrics and simple router cost fn (#88) 2025-03-11 11:53:21 -07:00
Alec 989bb3d59c feat: make block_size input for indexer, router, publisher (#66) 2025-03-09 15:58:27 -07:00
GuanLuo 6ba39b0932 chore: address comments for #35 (#53) 2025-03-09 07:44:45 -07:00
Neelay Shah 602352ce19 chore: rename dynamo (#44) 
Co-authored-by: Biswa Panda <biswa.panda@gmail.com>
 2025-03-08 01:42:40 -08:00
GuanLuo dcecc47de9 test: add tests for kv bindings (#35) 2025-03-07 16:01:28 -08:00
Neelay Shah 1af7433bff refactor: rename triton_distributed to dynemo (#22) 
Co-authored-by: Graham King <grahamk@nvidia.com>
 2025-03-05 09:15:46 -08:00
Neelay Shah 3a5fe17db9 feat: nixl metadata store and retrieved from etcd (#6) 
Co-authored-by: hongkuanz <hongkuanz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz <ptarasiewicz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz Nvidia <ptarasiewicznv@Piotrs-MacBook-Pro.local>
Co-authored-by: Neelay Shah <neelays@ipp2-0493.ipp2u1.colossus.nvidia.com>
Co-authored-by: Neelay Shah <neelays@ipp1-1941.ipp1a1.colossus.nvidia.com>
Co-authored-by: ishandhanani <ishandhanani@gmail.com>
Co-authored-by: Neelay Shah <neelays@4u8g-gen-0078.ipp3a2.colossus.nvidia.com>
Co-authored-by: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com>
 2025-03-04 13:10:16 -08:00
Neelay Shah 08fcd7e93b refactor: move libs to lib dir 
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-02-24 18:21:02 -08:00