* Added a default temperature value for text generation requests when no temperature is specified.
* Improved handling of missing configuration values to prevent errors during model initialization.
We can now do this:
- Node 1:
```
dynamo-run in=http out=dyn
```
- Node 2 and 3, two instances of component 'backend' in the nemotron_ultra pipeline:
```
dynamo-run in=dyn://nemotron_ultra.backend.generate out=vllm /data/models/NemotronUltra
```
- Node 4 and 5, two instances of the 'backend' component in nemotron_super pipeline:
```
dynamo-run in=dyn://nemotron_super.backend.generate out=vllm /data/models/NemotronSuper
```
The ingress node will discover all four instances and route correctly. We have been planning for this for a long time now.
As part of this auto-discovery is now always `out=dyn`, with no extra URL parts. Previously it could only route to a single pipeline.
Also:
- Refactor endpoint / instance naming now that I understand them
- Fix removing models when their instance stops.
Router:
```
dynamo-run in=http out=dyn://dynamo.endpoint.generate --router-mode kv
```
Worker (* N):
```
dynamo-run in=dyn://dynamo.endpoint.generate out=vllm /data/llms/Qwen/Qwen3-4B
```
You need patched vllm and the C bindings `.so`. Full docs in the updated guide: `docs/guides/dynamo_run.md`.
This gives us a pure-Rust ingress node: OpenAI compliant HTTP server + Pre-processor + KV-aware router.
Example of how to connect a Python sglang engine to the message bus (NATS/etc). I
In this example sglang does the pre/post processing. There is already an example where Dynamo does it.
The examples teach this:
- Be a chat completions engine, do your own pre-processing:
```
await register_llm(ModelType.Chat, endpoint, config.model)
```
- Have Dynamo do pre-processing. It will register us under both Chat and Completions endpoints, because that's handled before a Backend engine gets the request:
```
await register_llm(ModelType.Backend, endpoint, config.model)
```
New vllm and sglang engines that run in a sub-process. Will hopefully replace the existing embedded python engines.
Why?
- Pure Python, does not require knowing Rust to work on it. Much simpler to maintain.
- No embedded Python interpreter which avoids linking libpython and avoids the MacOS virtualenv issues.
- Should have better performance as it's "native" vllm / sglang.
- Works with any version of vllm (including v1!) and sglang. Less upgrade struggle.
Tzu-Ling Kan