mooncake-track
/
dynamo
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,607 Commits 2 Branches 0 Tags 76 MiB
Commit Graph

96 Commits

Author SHA1 Message Date
Olga Andreeva 27fad26faf
refactor: Split ModelType to ModelInput for request and response type; ModelType for the supported workloads (#2714) 
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Guan Luo <gluo@nvidia.com>
Co-authored-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
 2025-09-03 22:22:37 +00:00
KrishnanPrash c920cbd9dc
feat: Add --custom-jinja-template argument to pass a custom chat template for vLLM (#2829) 
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
 2025-09-03 14:22:15 -07:00
Biswa Panda c6becbc859
feat: dynamo namespace isolation (#2394) 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
 2025-09-03 15:47:06 +00:00
Yan Ru Pei 383e3b3a52
feat: don't modify kv scheduler states on query + more python binding (#2798) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-02 17:09:43 -07:00
Ayush Agarwal 87a721a83e
feat: added parser name bindings (#2808) 
Signed-off-by: Ayush Agarwal <ayushag@nvidia.com>
 2025-09-02 20:37:29 +00:00
Jacky 6c539fbdac
feat: FT Request Cancellation feature and test for 0.5.0 (#2500) 2025-09-02 08:26:21 -07:00
Yan Ru Pei 7fabe7bfe2
fix: do not delete KV events jetstream (#2800) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-09-01 21:25:49 +00:00
Yan Ru Pei 488c87095c
feat: Router warm restarts via durable KV event consumers and radix snapshotting (#2756) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2025-08-30 23:42:57 +00:00
Richard Huo a68c2f8f12
feat: DIS-373 dynamo KVBM connector API integration with TRTLLM (#2544) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-08-29 19:27:15 -07:00
Keiven C 15539fd093
feat: add Prometheus metrics integration for KvStats (#2704) 
Signed-off-by: Keiven C <213854356+keivenchang@users.noreply.github.com>
 2025-08-28 22:12:22 -07:00
Yan Ru Pei f08729ae15
feat: python bindings for the entire KvPushRouter + per-request router configs (#2658) 2025-08-25 22:27:10 +00:00
Ziqi Fan b39382ba68
feat: add initial batch of KVBM metrics on match, offload and onboard (#2673) 2025-08-25 09:28:25 -07:00
Ayush Agarwal cbe854fc5f
feat: [vLLM] implement cli args for tool and reasoning parsers (#2619) 2025-08-22 20:29:18 +00:00
Ziqi Fan b658ba6139
feat: enable dynamo metrics on KVBM (#2626) 2025-08-22 19:58:05 +00:00
Graham King 6a358f7c8c
chore(llm): Rename protocols::Endpoint to EndpointId (#2615) 2025-08-22 15:07:34 +00:00
Michael Feil 174389e6d3
fix: Httpengine sync-enable-endpoint (#2591) 2025-08-21 18:29:52 -04:00
Tzu-Ling Kan 57728909cf
feat: Add model label for vllm backend metrics (#2474) 
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-08-21 17:32:34 +00:00
Michael Feil 626d7e182d
feat(request cancellation): pycontext, propagating the `is_stopped` into python land. (#2158) 2025-08-20 12:06:13 -07:00
Ryan Olson 07cfc3a11b
feat: kvbm + connector (#2258) 
Signed-off-by: Ryan Olson <rolson@nvidia.com>
Co-authored-by: Olga Andreeva <oandreeva@nvidia.com>
Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Co-authored-by: John Thompson <jothomson@nvidia.com>
Co-authored-by: Richard Huo <rihuo@nvidia.com>
Co-authored-by: Zicheng Ma <zichengm@nvidia.com>
 2025-08-19 12:36:53 -07:00
suzu c5d9d26703
feat(frontend): support setting HTTP host via CLI (--http-host) (#2523) 2025-08-19 09:01:04 -04:00
Yan Ru Pei 85d8310806
feat: router-level request rejection (#2465) 2025-08-19 01:51:00 -07:00
Graham King a4bbe49228
feat(http): TLS support (#2492) 2025-08-18 16:06:29 -04:00
Keiven C acbdabc464
feat(metrics): add NATS client metrics to prometheus_metrics_fmt (#2292) 
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
 2025-08-14 19:33:53 -07:00
Tzu-Ling Kan 3a3f5bf275
feat: Add a "model" label to Component metrics (#2389) 2025-08-14 13:48:25 -05:00
Jorge António d0a6363584
feat: add RuntimeConfig to ModelEntry (#2311) 
Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
 2025-08-14 10:19:53 -07:00
Graham King 72ec5f5c7b
feat: Allow an endpoint to serve multiple models (#2418) 2025-08-13 10:48:42 -04:00
Yan Ru Pei 5166a3dd44
feat: Router replicas with state-sharing (#2264) 2025-08-07 22:57:57 +00:00
Graham King 1954fcfa09
chore: Remove service_name from ModelDeploymentCard (#2349) 2025-08-07 11:02:41 -04:00
Graham King 6a1a801c2d
feat: Support static workers, run without etcd. (#2281) 2025-08-06 09:51:43 -04:00
Hongkuan Zhou 36c4ef5eb2
feat: migrate requests when planner shutdown decode engine (vllm) (#2280) 
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: hhzhang16 <54051230+hhzhang16@users.noreply.github.com>
 2025-08-05 12:24:07 -07:00
Jacky 347620a1ff
feat: Allow Python Engine to end stream before final (#2270) 2025-08-05 10:44:18 -07:00
Chi 433f60121a
feat: Pass user_data to register_llm for LoRA support (#2286) 2025-08-05 10:16:43 -04:00
Yan Ru Pei bae25dc6d4
feat: skip downloading model weights if using mocker (only tokenizer) (#2213) 2025-07-31 18:15:57 +00:00
Jacky 1f07dab7bd
feat: Add migration to LLM requests (#1930) 2025-07-18 20:04:20 +00:00
Graham King fc12436048
feat(frontend): router-mode settings (#2001) 2025-07-18 18:52:57 +00:00
Graham King 182d3b5dc7
chore(bindings): Remove mistralrs / llama.cpp (#1970) 2025-07-16 16:12:40 -04:00
Yan Ru Pei f31732a22d
feat: integrate mocker with dynamo-run and python cli (#1927) 2025-07-16 18:22:15 +00:00
Graham King 2bf27924a1
feat(python): Python bindings for the Dynamo CLI tools (#1799) 2025-07-08 21:40:36 +00:00
Yan Ru Pei 84e71e27d3
feat: predictive active blocks for routing without load metrics (#1731) 
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
 2025-07-08 00:18:22 -07:00
jain-ria 439e977d9c
feat: vllm speculative decoding metrics (#1549) 
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
 2025-07-07 23:40:43 +00:00
Jacky b4ddca99a0
feat: Failure Detection while Responses are returning (#1671) 2025-07-07 14:00:17 -07:00
Alec 0a32b3443f
fix: default to None initialization of routing config (#1713) 2025-07-01 10:55:05 -07:00
jthomson04 6365a015b3
fix: Fix main (#1712) 2025-06-30 22:08:06 -07:00
jthomson04 aaf283bbb8
feat: Approximate KV Routing (#1636) 2025-06-30 20:34:08 -07:00
Graham King 92f06b0e7f
chore(dynamo-run): Refactor to library (#1687) 
Move much of what was in the `dynamo-run` crate into `dynamo-llm` so that everyone can use it.

Example usage:

1. Create a `LocalModel`:

```
    let local_model = LocalModelBuilder::default()
	.model_path("Qwen/Qwen3-0.6B")
	.http_port(8080)
	.build().await?;
```

2. Make an engine:

```
    let engine_config = EngineConfig::StaticFull {
	engine: dynamo_engine_mistralrs::make_engine(&local_model).await?,
	model: Box::new(local_model),
    };
```

3. Connect it to an input and run it

```
    dynamo_llm::entrypoint::input::run_input(Input::Http, runtime, engine_config).await?;
```

For https://github.com/ai-dynamo/dynamo/issues/1647

Code Rabbit summary, thanks:
  * Introduced a flexible builder pattern for local model configuration, allowing advanced customization and easier initialization.
  * Added new input modes and unified input handling, supporting interactive chat, HTTP server, batch file, and distributed endpoint modes.
  * Centralized engine configuration and routing, enabling more extensible and maintainable engine management.
  * Simplified and modularized the codebase by moving input and engine logic into dedicated modules.
  * Replaced direct model construction with an asynchronous builder for improved clarity and extensibility.
  * Streamlined configuration and validation for flags and router settings.
  * Added validation to prevent incompatible input and output combinations in endpoint and dynamic modes.
 2025-06-30 21:06:24 +00:00
Yan Ru Pei 8392e7a190
feat: Unnormalize waiting requests + predictive load updates for Python router (mirroring Rust) + softmax sampling to reduce thrashing (#1638) 2025-06-27 09:01:59 +00:00
Yan Ru Pei 13a99b7f76
feat: Standalone Router (#1409) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yan Ru Pei <yanrpei@gmail.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: jain-ria <riajain@NVIDIA.com>
 2025-06-14 19:02:31 +00:00
Graham King 3f6a74723f
chore: Remove PreprocessedRequest alias BackendInput (#1307) 
It was confusing to have two names for one type.

This tidy up started in #1064 , is now complete.
 2025-06-02 11:22:27 -04:00
Alec 2f8da9ad1d
refactor: rename KvMetricsPublisher to WorkerMetricsPublisher (#1284) 2025-05-30 17:23:32 +00:00
jthomson04 9210a26d90
refactor: Refactor kv event publishers (#1287) 2025-05-30 09:04:28 -07:00