Signed-off-by: Ryan Olson <rolson@nvidia.com>
Co-authored-by: Olga Andreeva <oandreeva@nvidia.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: John Thompson <jothomson@nvidia.com>
Co-authored-by: Richard Huo <rihuo@nvidia.com>
Co-authored-by: Zicheng Ma <zichengm@nvidia.com>
Move much of what was in the `dynamo-run` crate into `dynamo-llm` so that everyone can use it.
Example usage:
1. Create a `LocalModel`:
```
let local_model = LocalModelBuilder::default()
.model_path("Qwen/Qwen3-0.6B")
.http_port(8080)
.build().await?;
```
2. Make an engine:
```
let engine_config = EngineConfig::StaticFull {
engine: dynamo_engine_mistralrs::make_engine(&local_model).await?,
model: Box::new(local_model),
};
```
3. Connect it to an input and run it
```
dynamo_llm::entrypoint::input::run_input(Input::Http, runtime, engine_config).await?;
```
For https://github.com/ai-dynamo/dynamo/issues/1647
Code Rabbit summary, thanks:
* Introduced a flexible builder pattern for local model configuration, allowing advanced customization and easier initialization.
* Added new input modes and unified input handling, supporting interactive chat, HTTP server, batch file, and distributed endpoint modes.
* Centralized engine configuration and routing, enabling more extensible and maintainable engine management.
* Simplified and modularized the codebase by moving input and engine logic into dedicated modules.
* Replaced direct model construction with an asynchronous builder for improved clarity and extensibility.
* Streamlined configuration and validation for flags and router settings.
* Added validation to prevent incompatible input and output combinations in endpoint and dynamic modes.
Example:
```
dynamo-run out=<engine> <model> --kv-cache-block-size 64
```
In a distributed system this goes on the worker node and is propagated to ingress via the model deployment card.
Previously hard coded to 16, which is now the default.
- Load context_length from model. Closes#1172
- Store context length and KV cache block size in Model Deployment Card #1170
We can now do this:
- Node 1:
```
dynamo-run in=http out=dyn
```
- Node 2 and 3, two instances of component 'backend' in the nemotron_ultra pipeline:
```
dynamo-run in=dyn://nemotron_ultra.backend.generate out=vllm /data/models/NemotronUltra
```
- Node 4 and 5, two instances of the 'backend' component in nemotron_super pipeline:
```
dynamo-run in=dyn://nemotron_super.backend.generate out=vllm /data/models/NemotronSuper
```
The ingress node will discover all four instances and route correctly. We have been planning for this for a long time now.
As part of this auto-discovery is now always `out=dyn`, with no extra URL parts. Previously it could only route to a single pipeline.
Also:
- Refactor endpoint / instance naming now that I understand them
- Fix removing models when their instance stops.
Yan Ru Pei