3f84cdadfa
feat: add new metrics and simple router cost fn ( #88 )
2025-03-11 11:53:21 -07:00
dd6208254d
feat: add openai http service ( #82 )
2025-03-10 17:27:22 -07:00
989bb3d59c
feat: make block_size input for indexer, router, publisher ( #66 )
2025-03-09 15:58:27 -07:00
19844fc07e
feat: kv aware router + disagg router + prefill queue ( #11 )
...
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: hongkuan <hongkuanz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz <ptarasiewicz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz Nvidia <ptarasiewicznv@Piotrs-MacBook-Pro.local>
Co-authored-by: alec-flowers <aflowers@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
2025-03-08 17:09:11 -08:00
602352ce19
chore: rename dynamo ( #44 )
...
Co-authored-by: Biswa Panda <biswa.panda@gmail.com>
2025-03-08 01:42:40 -08:00
12714d9080
feat: Python bring-your-own-engine with our tokenizer ( #47 )
...
Instead of using `out=pystr:<my.py>` we can now do this:
```
dynemo-run out=pytok:/home/graham/my_python_engine.py --model-path <hf-repo-checkout>
```
That engine will receive and respond with tokens. Here's an example engine file:
```
import asyncio
async def generate(request):
yield {"token_ids":[791]}
await asyncio.sleep(0.1)
yield {"token_ids":[6864]}
await asyncio.sleep(0.1)
yield {"token_ids":[315]}
await asyncio.sleep(0.1)
yield {"token_ids":[9822]}
await asyncio.sleep(0.1)
yield {"token_ids":[374]}
await asyncio.sleep(0.1)
yield {"token_ids":[12366]}
await asyncio.sleep(0.1)
yield {"token_ids":[13]}
```
Also reduce duplication by making the bindings engine use the llm lib engine.
2025-03-07 10:48:37 -05:00
e159e53fe6
feat: expose KV routing components for easier router customization ( #15 )
2025-03-05 16:06:19 -08:00
1af7433bff
refactor: rename triton_distributed to dynemo ( #22 )
...
Co-authored-by: Graham King <grahamk@nvidia.com>
2025-03-05 09:15:46 -08:00
a32cdad622
feat: add python binding for rust llm modules ( #13 )
2025-03-04 14:52:09 -08:00
3a5fe17db9
feat: nixl metadata store and retrieved from etcd ( #6 )
...
Co-authored-by: hongkuanz <hongkuanz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz <ptarasiewicz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz Nvidia <ptarasiewicznv@Piotrs-MacBook-Pro.local>
Co-authored-by: Neelay Shah <neelays@ipp2-0493.ipp2u1.colossus.nvidia.com>
Co-authored-by: Neelay Shah <neelays@ipp1-1941.ipp1a1.colossus.nvidia.com>
Co-authored-by: ishandhanani <ishandhanani@gmail.com>
Co-authored-by: Neelay Shah <neelays@4u8g-gen-0078.ipp3a2.colossus.nvidia.com>
Co-authored-by: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com>
2025-03-04 13:10:16 -08:00
11a36651a6
[fix] KV Router Example fixes ( #314 )
...
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-02-28 12:20:47 -08:00
85cc7b67c6
refactor: service/endpoint stats_handler ( #282 )
2025-02-27 11:30:18 -07:00
b760c5694d
feat: Add completion endpoint to http server and llmctl ( #230 )
...
Co-authored-by: aflowers <aflowers@nvidia.com>
2025-02-25 12:32:09 -08:00
861c50982b
feat: enable metrics polling
...
Signed-off-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
Co-authored-by: Biswa Panda <biswapanda@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-02-25 11:12:54 -08:00
c06b95ffdb
ci: Add rust checks to missing directories ( #239 )
...
Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
2025-02-25 07:42:33 -08:00
08fcd7e93b
refactor: move libs to lib dir
...
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-02-24 18:21:02 -08:00
Alec