mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,557 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

448 Commits

Author SHA1 Message Date
Ryan McCormick f4780e855c
docs: Remove outdated python-wheels directory reference (#719) 2025-04-16 23:22:10 -07:00
Hongkuan Zhou 08fd28978c
feat: ETCD prefix watcher + python binding + runtime reconfiguration for router and disagg router (#581) 
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
 2025-04-11 18:15:42 -07:00
Cole 447840c223 docs: add docstring for llm.rs (#267) 2025-04-10 22:40:09 -04:00
Anant Sharma fa7ee14c2a chore: update versions to 0.1.1 (#552) 2025-04-09 09:50:30 -04:00
Yan Ru Pei 4b6cfc1be0 feat: KV recorder for dumping router events into a jsonl (#505) 2025-04-04 15:31:32 -07:00
Graham King 88ad3425c4 feat: Python decorator dynamo_worker takes optional `static` parameter without etcd (#494) 
Adds `@dynamo_worker(static = True)` to create a static worker which has a predictable name and hence does not require discovery or `etcd` to be running. There can only be a single static worker per namespace / component / endpoint trio.

This contrasts with the default dynamic `dynamo_worker` endpoints we have now, which get a unique random name (based on namespace/component/endpoint), and are discovered by ingress components using etcd.

Also change the hello_world example to use `dynamo_worker(static = True)` so that it is exercised and demonstrated somewhere.

For NIM.
 2025-04-04 09:26:52 -04:00
Ryan Olson 84985d3f1d refactor: migrate engines to standalone crates (#453) 
Moved all of `lib/llm/src/engines` to their own crates as e.g. `lib/engines/mistralrs`. This will allow publishing of the `dynamo-llm` crate as it won't have any github dependencies.

The only engines in dynamo-llm will be the demo `echo` ones.

Co-authored-by: Graham King <grahamk@nvidia.com>
 2025-04-03 21:53:54 +00:00
Ryan Olson 6795e645d0 fix: adding missing file (#501) 2025-04-03 08:29:33 -04:00
Ryan Olson c4106e6a27 feat: kv aware router executable (#399) 2025-04-02 17:58:21 +02:00
Ryan Olson 5b682f4839 feat: unified logging (#472) 2025-04-01 19:49:20 +00:00
Ryan Olson de2905377c refactor: prometheus upgrade (#452) 2025-03-31 13:03:36 -06:00
Biswa Panda 8621d914e3 feat: dynamo deploy hello world example to k8s (#205) 2025-03-28 20:51:48 +00:00
Graham King c7067fc2ec feat: Build pre-processor from GGUF (#344) 
This lets us do:
```
dynamo-run out=llamacpp <gguf_file>
```

Previously a `--model-config <hf-repo>` was also required, to configure our tokenizer.
 2025-03-24 11:59:41 -04:00
Graham King 7c3fd5c94c chore: Don't depend on openssl (#292) 
This makes the Rust parts all use ring / rustls library instead of local install of openssl. It's a step on the journey to being statically linked.

Pieces:
- `tokenizers` and `mistralrs` now support rustls (mistralrs by default, tokenizers with feature flag).
- Move shared dependencies up into workspace
- New `rand` crate has some renames for future rust
- Ensure the dependency doesn't creep back in by enforcing it with cargo deny.
 2025-03-19 14:16:19 -04:00
Dmitry Tokarev 548578f4a9 docs: fix links in docs (#256) 
Co-authored-by: Anant Sharma <anants@nvidia.com>
 2025-03-18 03:40:31 -04:00
GuanLuo 6e09681e0e feat: expose Python binding for KVEventPublisher. Use event pub/sub trait for KV events (#169) 2025-03-17 00:17:33 -07:00
April Yang 7f136e29c1 feat: update deploy api & sdk (#74) 
Co-authored-by: Julien Mancuso <jmancuso@nvidia.com>
Co-authored-by: Hannah Zhang <hannahz@nvidia.com>
Co-authored-by: Biswa Panda <biswa.panda@gmail.com>
Co-authored-by: Maksim Khadkevich <mkhadkevich@nvidia.com>
 2025-03-15 18:32:57 -07:00
Anant Sharma c1c22703c5 build: reorganize python packaging to build new wheels (#118) 2025-03-14 13:43:29 -04:00
Ryan Olson f04359cf69 feat: global kv block manager (#45) 2025-03-13 20:57:28 -06:00
Anant Sharma 3d2928510c build: add top level rust workspace (#137) 2025-03-13 17:22:19 -04:00
Alec 3f84cdadfa feat: add new metrics and simple router cost fn (#88) 2025-03-11 11:53:21 -07:00
Biswa Panda dd6208254d feat: add openai http service (#82) 2025-03-10 17:27:22 -07:00
Anant Sharma fc4da34502 chore: update wheel name and reset versions (#73) 2025-03-10 17:41:13 -04:00
Alec 989bb3d59c feat: make block_size input for indexer, router, publisher (#66) 2025-03-09 15:58:27 -07:00
Neelay Shah dd31a322ae chore: stragglers rename (#69) 
Co-authored-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
 2025-03-09 14:09:11 -07:00
Neelay Shah 678cffb4e5 chore: left over renaming (#67) 
Co-authored-by: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com>
Co-authored-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
 2025-03-09 14:09:04 -04:00
GuanLuo 6ba39b0932 chore: address comments for #35 (#53) 2025-03-09 07:44:45 -07:00
Hongkuan Zhou 19844fc07e feat: kv aware router + disagg router + prefill queue (#11) 
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: hongkuan <hongkuanz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz <ptarasiewicz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz Nvidia <ptarasiewicznv@Piotrs-MacBook-Pro.local>
Co-authored-by: alec-flowers <aflowers@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
 2025-03-08 17:09:11 -08:00
Dmitry Tokarev b4d56a5789 chore: Renamed Triton Distributed to Dynamo (#56) 2025-03-08 09:42:09 -08:00
Neelay Shah 602352ce19 chore: rename dynamo (#44) 
Co-authored-by: Biswa Panda <biswa.panda@gmail.com>
 2025-03-08 01:42:40 -08:00
GuanLuo dcecc47de9 test: add tests for kv bindings (#35) 2025-03-07 16:01:28 -08:00
Graham King 12714d9080 feat: Python bring-your-own-engine with our tokenizer (#47) 
Instead of using `out=pystr:<my.py>` we can now do this:
```
dynemo-run out=pytok:/home/graham/my_python_engine.py --model-path <hf-repo-checkout>
```

That engine will receive and respond with tokens. Here's an example engine file:
```
import asyncio

async def generate(request):
    yield {"token_ids":[791]}
    await asyncio.sleep(0.1)
    yield {"token_ids":[6864]}
    await asyncio.sleep(0.1)
    yield {"token_ids":[315]}
    await asyncio.sleep(0.1)
    yield {"token_ids":[9822]}
    await asyncio.sleep(0.1)
    yield {"token_ids":[374]}
    await asyncio.sleep(0.1)
    yield {"token_ids":[12366]}
    await asyncio.sleep(0.1)
    yield {"token_ids":[13]}
```

Also reduce duplication by making the bindings engine use the llm lib engine.
 2025-03-07 10:48:37 -05:00
GuanLuo e159e53fe6 feat: expose KV routing components for easier router customization (#15) 2025-03-05 16:06:19 -08:00
Neelay Shah 1af7433bff refactor: rename triton_distributed to dynemo (#22) 
Co-authored-by: Graham King <grahamk@nvidia.com>
 2025-03-05 09:15:46 -08:00
Biswa Panda a32cdad622 feat: add python binding for rust llm modules (#13) 2025-03-04 14:52:09 -08:00
Neelay Shah 3a5fe17db9 feat: nixl metadata store and retrieved from etcd (#6) 
Co-authored-by: hongkuanz <hongkuanz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz <ptarasiewicz@nvidia.com>
Co-authored-by: Piotr Tarasiewicz Nvidia <ptarasiewicznv@Piotrs-MacBook-Pro.local>
Co-authored-by: Neelay Shah <neelays@ipp2-0493.ipp2u1.colossus.nvidia.com>
Co-authored-by: Neelay Shah <neelays@ipp1-1941.ipp1a1.colossus.nvidia.com>
Co-authored-by: ishandhanani <ishandhanani@gmail.com>
Co-authored-by: Neelay Shah <neelays@4u8g-gen-0078.ipp3a2.colossus.nvidia.com>
Co-authored-by: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com>
 2025-03-04 13:10:16 -08:00
Graham King 2d906fb4c9 fix: Install specific toolchain (#329) 
`cargo build --locked` won't let you use "1.85.0" if you only have "stable" installed, even if those are the same thing right now.
 2025-03-03 11:53:14 -05:00
Graham King 057f8f4700 feat: TensorRT-LLM engine (#317) 
Engine, `tio` support and docs.

Proof of concept / experimental.
 2025-02-28 17:09:01 -05:00
Alec 11a36651a6 [fix] KV Router Example fixes (#314) 
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-02-28 12:20:47 -08:00
Ryan Olson 85cc7b67c6 refactor: service/endpoint stats_handler (#282) 2025-02-27 11:30:18 -07:00
Anant Sharma ea401e3bc0 ci: build wheel from root directory (#274) 2025-02-27 11:56:51 -05:00
Paul Hendricks 86aff237ab refactor: using async_openai 
Co-authored-by: Graham King <grahamk@nvidia.com>
 2025-02-26 13:59:39 -08:00
Graham King e97493eb00 feat: sglang backend for tio (#271) 
- Setup venv

```
uv venv
source .venv/bin/activate
uv pip install pip
uv pip install sgl-kernel --force-reinstall --no-deps
uv pip install "sglang[all]==0.4.2" --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer/
```

- Build: `cargo build --release --features sglang`

- Run single node (make sure you're in the venv): `./tio out=sglang ~/llm_models/my_model`

- Run Deepseek multi-gpu / multi-node:

Node 1:
```
tio in=http out=sglang --model-path ~/llm_models/DeepSeek-R1-Distill-Llama-70B/ --tensor-parallel-size 8 --num-nodes 2 --node-rank 0 --dist-init-addr 10.217.98.122:9876
```

Node 2:
```
tio in=none out=sglang --model-path ~/llm_models/DeepSeek-R1-Distill-Llama-70B/ --tensor-parallel-size 8 --num-nodes 2 --node-rank 1 --dist-init-addr 10.217.98.122:9876
```
 2025-02-25 17:04:08 -05:00
Neelay Shah c70de37fcb chore: updating docs after restructure 2025-02-25 12:37:25 -08:00
Alec b760c5694d feat: Add completion endpoint to http server and llmctl (#230) 
Co-authored-by: aflowers <aflowers@nvidia.com>
 2025-02-25 12:32:09 -08:00
GuanLuo 861c50982b feat: enable metrics polling 
Signed-off-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
Co-authored-by: Biswa Panda <biswapanda@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-02-25 11:12:54 -08:00
Ryan McCormick c06b95ffdb ci: Add rust checks to missing directories (#239) 
Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-02-25 07:42:33 -08:00
Neelay Shah 08fcd7e93b refactor: move libs to lib dir 
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
 2025-02-24 18:21:02 -08:00