67329d1023
Wrap wait_for_deployment_ready() in try/except TimeoutError for both prefill and decode profiling sweeps On timeout: log error, record via add_profiling_error(), clean up the timed-out deployment, and continue to the next parallelization mapping Previously, a single deployment timeout would crash the entire profiler job |
||
|---|---|---|
| .. | ||
| burstgpt_loadgen | ||
| incluster | ||
| llm | ||
| multimodal | ||
| nat_trace | ||
| prefix_data_generator | ||
| profiler | ||
| router | ||
| sin_load_generator | ||
| utils | ||
| README.md | ||
| pyproject.toml | ||
README.md
Benchmarks
This directory contains benchmarking scripts and tools for performance evaluation of Dynamo deployments. The benchmarking framework is a wrapper around aiperf that makes it easy to benchmark DynamoGraphDeployments or other deployments with exposed endpoints.
Quick Start
Benchmark a Dynamo Deployment
First, deploy your DynamoGraphDeployment using the deployment documentation, then:
# Port-forward your deployment to http://localhost:8000
kubectl port-forward -n <namespace> svc/<frontend-service-name> 8000:8000 > /dev/null 2>&1 &
# Run benchmark
python3 -m benchmarks.utils.benchmark \
--benchmark-name my-benchmark \
--endpoint-url http://localhost:8000 \
--model "<your-model>"
# Generate plots
python3 -m benchmarks.utils.plot --data-dir ./benchmarks/results
# Or plot only specific benchmark experiments
python3 -m benchmarks.utils.plot --data-dir ./benchmarks/results --benchmark-name my-benchmark
Features
Benchmark any HTTP endpoints! The benchmarking framework supports:
Flexible Configuration:
- User-defined benchmark names using
--benchmark-nameflag - Support for single endpoint benchmarking with
--endpoint-urlflag - Customizable concurrency levels (configurable via CONCURRENCIES env var), sequence lengths, and models
- Automated performance plot generation with custom benchmark names
Supported Backends:
- DynamoGraphDeployments with port-forwarded endpoints
- External HTTP endpoints (for comparison with non-Dynamo backends or platforms)
Installation
This is already included as part of the Dynamo container images. To install locally or standalone:
pip install -e .
Data Generation Tools
This directory also includes lightweight tools for:
- Analyzing prefix-structured data (
datagen analyze) - Synthesizing structured data customizable for testing purposes (
datagen synthesize)
Detailed information is provided in the prefix_data_generator directory.
Comprehensive Guide
For detailed documentation, configuration options, and advanced usage, see the complete benchmarking guide.