* Added a default temperature value for text generation requests when no temperature is specified.
* Improved handling of missing configuration values to prevent errors during model initialization.
We can now do this:
- Node 1:
```
dynamo-run in=http out=dyn
```
- Node 2 and 3, two instances of component 'backend' in the nemotron_ultra pipeline:
```
dynamo-run in=dyn://nemotron_ultra.backend.generate out=vllm /data/models/NemotronUltra
```
- Node 4 and 5, two instances of the 'backend' component in nemotron_super pipeline:
```
dynamo-run in=dyn://nemotron_super.backend.generate out=vllm /data/models/NemotronSuper
```
The ingress node will discover all four instances and route correctly. We have been planning for this for a long time now.
As part of this auto-discovery is now always `out=dyn`, with no extra URL parts. Previously it could only route to a single pipeline.
Also:
- Refactor endpoint / instance naming now that I understand them
- Fix removing models when their instance stops.
Router:
```
dynamo-run in=http out=dyn://dynamo.endpoint.generate --router-mode kv
```
Worker (* N):
```
dynamo-run in=dyn://dynamo.endpoint.generate out=vllm /data/llms/Qwen/Qwen3-4B
```
You need patched vllm and the C bindings `.so`. Full docs in the updated guide: `docs/guides/dynamo_run.md`.
This gives us a pure-Rust ingress node: OpenAI compliant HTTP server + Pre-processor + KV-aware router.
New vllm and sglang engines that run in a sub-process. Will hopefully replace the existing embedded python engines.
Why?
- Pure Python, does not require knowing Rust to work on it. Much simpler to maintain.
- No embedded Python interpreter which avoids linking libpython and avoids the MacOS virtualenv issues.
- Should have better performance as it's "native" vllm / sglang.
- Works with any version of vllm (including v1!) and sglang. Less upgrade struggle.
Adding this to a Python script makes it register on the network so that `dynamo-run` can discover it and send it requests:
```
from dynamo.llm import register_llm
MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
await register_llm(endpoint, MODEL, 3)
```
Full vllm example, with pre-processing in dynamo:
- `dynamo-run in=text out=dyn://dynamo.backend.generate`
- `cd lib/bindings/python/examples/hello_world`
- `python server_vllm.py`
This builds on top of the work to move pre-processor to ingress side. It means we can decouple Rust and Python using NATS as the bus.
The `register_llm` call does this:
- Download the model from HF if necessary
- Load the model deployment card from the HF folder or extract from GGUF
- Push the tokenizer config etc into NATS object store so ingress can access it from a different machine
- Publish the model deployment card to ETCD
Tzu-Ling Kan