Signed-off-by: Kristen Kelleher <kkelleher@nvidia.com>
- Content, format, and structural changes to the Dynamo docs for 0.3.0.
- Includes copyediting and the first batch of changes from the DMO review.
- Add Granite to our tokenizer
- Fix pre-processor to load context length correctly
- Add strftime_now Jinja function for prompt templates
- Update llama.cpp
- Handle trtllm errors when not using trtllm
Support depends on the engine:
- `mistral.rs`, our default engine, doesn't support Granite yet.
- `llama.cpp` does and works very well:
```
dynamo-run out=llamacpp ~/llms/granite-3.3-2b-instruct-Q4_K_M.gguf --context-length 16384
```
- `vllm` also works very well:
```
dynamo-run in=http out=vllm ~/llms/granite-3.3-2b-instruct --context-length 16384
```
- `sglang` mostly works, but it doesn't catch the stop token, so we do in the HTTP ingress, and log an error. The Text ingress doesn't catch it because I disabled it to make the raw echo engine work. A bit of work to do here.
Closes: #1245
Example:
```
dynamo-run out=<engine> <model> --kv-cache-block-size 64
```
In a distributed system this goes on the worker node and is propagated to ingress via the model deployment card.
Previously hard coded to 16, which is now the default.
- Load context_length from model. Closes#1172
- Store context length and KV cache block size in Model Deployment Card #1170
We can now do this:
- Node 1:
```
dynamo-run in=http out=dyn
```
- Node 2 and 3, two instances of component 'backend' in the nemotron_ultra pipeline:
```
dynamo-run in=dyn://nemotron_ultra.backend.generate out=vllm /data/models/NemotronUltra
```
- Node 4 and 5, two instances of the 'backend' component in nemotron_super pipeline:
```
dynamo-run in=dyn://nemotron_super.backend.generate out=vllm /data/models/NemotronSuper
```
The ingress node will discover all four instances and route correctly. We have been planning for this for a long time now.
As part of this auto-discovery is now always `out=dyn`, with no extra URL parts. Previously it could only route to a single pipeline.
Also:
- Refactor endpoint / instance naming now that I understand them
- Fix removing models when their instance stops.
Router:
```
dynamo-run in=http out=dyn://dynamo.endpoint.generate --router-mode kv
```
Worker (* N):
```
dynamo-run in=dyn://dynamo.endpoint.generate out=vllm /data/llms/Qwen/Qwen3-4B
```
You need patched vllm and the C bindings `.so`. Full docs in the updated guide: `docs/guides/dynamo_run.md`.
This gives us a pure-Rust ingress node: OpenAI compliant HTTP server + Pre-processor + KV-aware router.
Example of how to connect a Python sglang engine to the message bus (NATS/etc). I
In this example sglang does the pre/post processing. There is already an example where Dynamo does it.
The examples teach this:
- Be a chat completions engine, do your own pre-processing:
```
await register_llm(ModelType.Chat, endpoint, config.model)
```
- Have Dynamo do pre-processing. It will register us under both Chat and Completions endpoints, because that's handled before a Backend engine gets the request:
```
await register_llm(ModelType.Backend, endpoint, config.model)
```
. New mistralrs and llamacpp version
. mistralrs: Handle Gemma 3 and Llama 4 as vision models
. Update the dynamo-run docs to use Qwen 3
. Our pre-processor now supports Llama 4's newer multi-modal `config.json`
. Upgrade minijinja to handle Qwen 3's prompt template
For Llama 4 we'll need to limit the max seq len. vllm says:
> To serve at least one request with the models's max seq len (10485760), (240.00 GiB KV cache is needed,...
I was able to run Llama 4 with llamacpp and a quantized GGUF, with Dynamo doing the pre-processing.
New vllm and sglang engines that run in a sub-process. Will hopefully replace the existing embedded python engines.
Why?
- Pure Python, does not require knowing Rust to work on it. Much simpler to maintain.
- No embedded Python interpreter which avoids linking libpython and avoids the MacOS virtualenv issues.
- Should have better performance as it's "native" vllm / sglang.
- Works with any version of vllm (including v1!) and sglang. Less upgrade struggle.
Adding this to a Python script makes it register on the network so that `dynamo-run` can discover it and send it requests:
```
from dynamo.llm import register_llm
MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
await register_llm(endpoint, MODEL, 3)
```
Full vllm example, with pre-processing in dynamo:
- `dynamo-run in=text out=dyn://dynamo.backend.generate`
- `cd lib/bindings/python/examples/hello_world`
- `python server_vllm.py`
This builds on top of the work to move pre-processor to ingress side. It means we can decouple Rust and Python using NATS as the bus.
The `register_llm` call does this:
- Download the model from HF if necessary
- Load the model deployment card from the HF folder or extract from GGUF
- Push the tokenizer config etc into NATS object store so ingress can access it from a different machine
- Publish the model deployment card to ETCD
In a distributed system we don't know if the remote workers need pre-processing done ingress-side or not. Previously Client required us to decide this before discovering the remote endpoints, which was fine because pre-processing was worker-side.
As part of moving pre-processing back to ingress-side we need to split this into two steps:
- Client discovers the endpoints, and (later PR) will fetch their Model Deployment Card.
- PushRouter will use the Model Deployment Card to decide if they need pre-processing or not, which affects the types of the generic parameters.
Part of #743
This will allow an ingress-side pre-processor to see it without needing a model checkout.
Currently pre-processing is done in the worker, which has access to the model deployment card ("MDC") files (`config.json`, `tokenizer.json` and `tokenizer_config.json`) locally. We want to move the pre-processor to the ingress side to support KV routing. That requires ingress side (i.e the HTTP server), on a different machine than the worker to be able to see those three files.
To support that this PR makes the worker upload the contents of those files to the NATS object store, and publishes the MDC with those NATS urls to the key-value store.
The key-value store has an interface so any store (nats, etcd, redis, etc) can be supported. Implementations for memory and NATS are provided.
Fetching the MDC from the store, doing pre-processing ingress side, and publishing a card backed by a GGUF, are all for a later commit.
Part of #743
Adds `@dynamo_worker(static = True)` to create a static worker which has a predictable name and hence does not require discovery or `etcd` to be running. There can only be a single static worker per namespace / component / endpoint trio.
This contrasts with the default dynamic `dynamo_worker` endpoints we have now, which get a unique random name (based on namespace/component/endpoint), and are discovered by ingress components using etcd.
Also change the hello_world example to use `dynamo_worker(static = True)` so that it is exercised and demonstrated somewhere.
For NIM.
Moved all of `lib/llm/src/engines` to their own crates as e.g. `lib/engines/mistralrs`. This will allow publishing of the `dynamo-llm` crate as it won't have any github dependencies.
The only engines in dynamo-llm will be the demo `echo` ones.
Co-authored-by: Graham King <grahamk@nvidia.com>
This makes the Rust parts all use ring / rustls library instead of local install of openssl. It's a step on the journey to being statically linked.
Pieces:
- `tokenizers` and `mistralrs` now support rustls (mistralrs by default, tokenizers with feature flag).
- Move shared dependencies up into workspace
- New `rand` crate has some renames for future rust
- Ensure the dependency doesn't creep back in by enforcing it with cargo deny.
Co-authored-by: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com>
Co-authored-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Instead of using `out=pystr:<my.py>` we can now do this:
```
dynemo-run out=pytok:/home/graham/my_python_engine.py --model-path <hf-repo-checkout>
```
That engine will receive and respond with tokens. Here's an example engine file:
```
import asyncio
async def generate(request):
yield {"token_ids":[791]}
await asyncio.sleep(0.1)
yield {"token_ids":[6864]}
await asyncio.sleep(0.1)
yield {"token_ids":[315]}
await asyncio.sleep(0.1)
yield {"token_ids":[9822]}
await asyncio.sleep(0.1)
yield {"token_ids":[374]}
await asyncio.sleep(0.1)
yield {"token_ids":[12366]}
await asyncio.sleep(0.1)
yield {"token_ids":[13]}
```
Also reduce duplication by making the bindings engine use the llm lib engine.
Tanmay Verma