b2aa504b47
fix: upgrade axum to 0.8 and etcd-client to 0.16 ( #2317 )
...
Signed-off-by: Dan Aloni <dan.aloni@vastdata.com>
2025-08-06 18:25:49 +00:00
6a1a801c2d
feat: Support static workers, run without etcd. ( #2281 )
2025-08-06 09:51:43 -04:00
36c4ef5eb2
feat: migrate requests when planner shutdown decode engine (vllm) ( #2280 )
...
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: hhzhang16 <54051230+hhzhang16@users.noreply.github.com>
2025-08-05 12:24:07 -07:00
347620a1ff
feat: Allow Python Engine to end stream before final ( #2270 )
2025-08-05 10:44:18 -07:00
433f60121a
feat: Pass user_data to register_llm for LoRA support ( #2286 )
2025-08-05 10:16:43 -04:00
4cbd4f3881
fix: show raw sglang logs given env var ( #2287 )
2025-08-05 06:11:44 +00:00
625578c3bb
chore: update nixl version to 0.4.1 ( #2221 )
2025-07-31 17:46:21 -04:00
bae25dc6d4
feat: skip downloading model weights if using mocker (only tokenizer) ( #2213 )
2025-07-31 18:15:57 +00:00
4c90b1b924
chore: Version bump to 0.4.0 ( #2179 )
2025-07-30 02:07:31 +00:00
222245e2a0
refactor: Move engine and publisher from dynamo.llm.tensorrt_llm to dynamo.trtllm ( #2128 )
2025-07-26 02:23:17 +00:00
19a77ae7d5
chore(dynamo-run): Remove out=sglang|vllm|trtllm ( #1920 )
2025-07-23 17:22:31 -07:00
66b7d2c75d
fix: updates versions and adds ahashmap to BPE ( #2072 )
2025-07-23 20:27:18 +00:00
e0a51940d1
feat: Connect Library ( #1478 )
2025-07-23 14:16:34 -04:00
9f2356cb9e
chore: Remove unused portion of kv bindings test ( #2052 )
2025-07-22 19:12:42 +00:00
1f07dab7bd
feat: Add migration to LLM requests ( #1930 )
2025-07-18 20:04:20 +00:00
fc12436048
feat(frontend): router-mode settings ( #2001 )
2025-07-18 18:52:57 +00:00
182d3b5dc7
chore(bindings): Remove mistralrs / llama.cpp ( #1970 )
2025-07-16 16:12:40 -04:00
f31732a22d
feat: integrate mocker with dynamo-run and python cli ( #1927 )
2025-07-16 18:22:15 +00:00
aba60996f2
perf(router): Remove lock from router hot path ( #1963 )
2025-07-16 12:26:32 -04:00
7b9182fdb9
chore: Move examples/cli to lib/bindings/examples/cli ( #1952 )
2025-07-15 16:33:07 -04:00
d975761b8b
chore: update nixl to 0.4.0 release ( #1860 ) ( #1886 )
2025-07-11 12:45:44 -04:00
1704b126f3
build: Revert "chore: update nixl to 0.4.0 release" ( #1880 )
2025-07-10 23:30:28 +00:00
f242b4552b
refactor: vllm v1 examples ( #1756 )
...
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Co-authored-by: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com>
Co-authored-by: Hongkuan Zhou <tedzhouhk@gmail.com>
2025-07-10 22:30:16 +00:00
5fa4cddab9
chore: update nixl to 0.4.0 release ( #1860 )
2025-07-10 17:08:34 -04:00
ece76a6201
feat: Build DistributedRuntime-level HTTP server with /health /metrics ( #1656 )
2025-07-08 22:49:28 +00:00
2bf27924a1
feat(python): Python bindings for the Dynamo CLI tools ( #1799 )
2025-07-08 21:40:36 +00:00
84e71e27d3
feat: predictive active blocks for routing without load metrics ( #1731 )
...
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
2025-07-08 00:18:22 -07:00
439e977d9c
feat: vllm speculative decoding metrics ( #1549 )
...
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
2025-07-07 23:40:43 +00:00
c4935b3497
chore: update versions for 0.3.2 release ( #1793 )
2025-07-07 17:21:02 -04:00
b4ddca99a0
feat: Failure Detection while Responses are returning ( #1671 )
2025-07-07 14:00:17 -07:00
a9241b61f0
chore: update nixl to latest 0.3.1 commit ( #1762 )
2025-07-03 15:31:47 -07:00
4ab476174a
chore(engines): Upgrade mistralrs to 0.6.0 ( #1767 )
2025-07-03 12:45:58 -07:00
0a32b3443f
fix: default to None initialization of routing config ( #1713 )
2025-07-01 10:55:05 -07:00
dfbd741de2
feat: Support for Responses API ( #1694 )
2025-07-01 12:08:15 -04:00
edf00c5c71
fix(bindings): Default router config in bindings ( #1716 )
...
* Added a default temperature value for text generation requests when no temperature is specified.
* Improved handling of missing configuration values to prevent errors during model initialization.
2025-07-01 14:53:32 +00:00
6365a015b3
fix: Fix main ( #1712 )
2025-06-30 22:08:06 -07:00
aaf283bbb8
feat: Approximate KV Routing ( #1636 )
2025-06-30 20:34:08 -07:00
92f06b0e7f
chore(dynamo-run): Refactor to library ( #1687 )
...
Move much of what was in the `dynamo-run` crate into `dynamo-llm` so that everyone can use it.
Example usage:
1. Create a `LocalModel`:
```
let local_model = LocalModelBuilder::default()
.model_path("Qwen/Qwen3-0.6B")
.http_port(8080)
.build().await?;
```
2. Make an engine:
```
let engine_config = EngineConfig::StaticFull {
engine: dynamo_engine_mistralrs::make_engine(&local_model).await?,
model: Box::new(local_model),
};
```
3. Connect it to an input and run it
```
dynamo_llm::entrypoint::input::run_input(Input::Http, runtime, engine_config).await?;
```
For https://github.com/ai-dynamo/dynamo/issues/1647
Code Rabbit summary, thanks:
* Introduced a flexible builder pattern for local model configuration, allowing advanced customization and easier initialization.
* Added new input modes and unified input handling, supporting interactive chat, HTTP server, batch file, and distributed endpoint modes.
* Centralized engine configuration and routing, enabling more extensible and maintainable engine management.
* Simplified and modularized the codebase by moving input and engine logic into dedicated modules.
* Replaced direct model construction with an asynchronous builder for improved clarity and extensibility.
* Streamlined configuration and validation for flags and router settings.
* Added validation to prevent incompatible input and output combinations in endpoint and dynamic modes.
2025-06-30 21:06:24 +00:00
82eae1fdf5
refactor: Upgrade async-openai ( #1693 )
2025-06-30 13:35:10 -04:00
8392e7a190
feat: Unnormalize waiting requests + predictive load updates for Python router (mirroring Rust) + softmax sampling to reduce thrashing ( #1638 )
2025-06-27 09:01:59 +00:00
250ed7334b
fix: Fix NIXL 0.3.1 build ( #1561 )
2025-06-17 14:30:10 -07:00
13a99b7f76
feat: Standalone Router ( #1409 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: jain-ria <riajain@NVIDIA.com>
2025-06-14 19:02:31 +00:00
99e67e607a
chore: update dynamo and nixl versions for 0.3.1 ( #1517 )
2025-06-13 21:13:43 +00:00
0e7d4d8298
docs: DIS-133 and DIS-134 plus copyediting ( #1439 )
...
Signed-off-by: Kristen Kelleher <kkelleher@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
2025-06-12 12:10:01 -07:00
75d7c3b933
fix: remove unused bentoml references ( #1412 )
2025-06-09 17:32:02 -07:00
e61f1c8a40
chore: Remove nats-py dependency ( #1387 )
2025-06-05 15:50:36 -07:00
568eb1009e
fix: Use Rust Ingress (dynamo-run) for the Frontend ( #1391 )
2025-06-04 22:27:05 -07:00
5e9370d3aa
docs: fix sphinx errors admonitions adobe config ( #1179 )
...
Signed-off-by: Kristen Kelleher <kkelleher@nvidia.com>
- Content, format, and structural changes to the Dynamo docs for 0.3.0.
- Includes copyediting and the first batch of changes from the DMO review.
2025-06-04 08:16:34 -07:00
3f6a74723f
chore: Remove PreprocessedRequest alias BackendInput ( #1307 )
...
It was confusing to have two names for one type.
This tidy up started in #1064 , is now complete.
2025-06-02 11:22:27 -04:00
2f8da9ad1d
refactor: rename KvMetricsPublisher to WorkerMetricsPublisher ( #1284 )
2025-05-30 17:23:32 +00:00
Dan Aloni