dynamo/tests

Dynamo Testing Framework

Overview

This document outlines the testing framework for the Dynamo runtime system, including test discovery, organization, and best practices.

Directory Structure

tests/
├── serve/              # E2E tests
│   ├── conftest.py     # test fixtures as needed for specific test area
├── conftest.py         # Shared fixtures and configuration
├── utils               # Common utils accross tests
└── README.md           # This file

Test Discovery

Pytest automatically discovers tests based on their naming convention. All test files must follow this pattern:

test_<component_or_flow>.py

Where:

• component_or_flow: The component or flow being tested (e.g., planner, kv_router)
  • For e2e tests, this could be the API or simply "dynamo"

Running Tests

To run all tests:

pytest

To run only specific tests:

# Run only vLLM tests
pytest -v -m vllm

# Run only e2e tests
pytest -v -m e2e

# Run tests for a specific component
pytest -v -m planner

# Run with print statements visible
pytest -s

Test Markers

Markers help control which tests run under different conditions. Add these decorators to your test functions:

Frequency-based markers

• @pytest.mark.nightly - Tests run nightly
• @pytest.mark.weekly - Tests run weekly
• @pytest.mark.pre_merge - Tests run before merging PRs

Role-based markers

• @pytest.mark.e2e - End-to-end tests
• @pytest.mark.integration - Integration tests
• @pytest.mark.unit - Unit tests
• @pytest.mark.stress - Stress/load tests
• @pytest.mark.benchmark - Performance benchmark tests

Component-specific markers

• @pytest.mark.vllm - Framework tests
• @pytest.mark.sglang - Framework tests
• @pytest.mark.trtllm_marker - Framework tests
• @pytest.mark.planner - Planner component tests
• @pytest.mark.kv_router - KV Router component tests
• etc.

Execution-related markers

• @pytest.mark.slow - Tests that take a long time to run
• @pytest.mark.skip(reason="Example: KV Manager is under development") - Skip these tests
• @pytest.mark.xfail(reason="Expected to fail because...") - Tests expected to fail

Environment Setup

Tests are designed to run in the appropriate framework container built via ./container/build.sh --framework X and run via ./container/run.sh --mount-workspace -it -- pytest.

Environment Variables

• HF_TOKEN - Your HuggingFace API token to avoid rate limits
  • Get a token from HuggingFace Settings
  • Set it before running tests: export HF_TOKEN=your_token_here

Model Download Cache

The tests will automatically use a local cache at ~/.cache/huggingface to avoid repeated downloads of model files. This cache is shared across test runs to improve performance.

Running tests locally outside of a container

To run tests outside of the development container, ensure that you have properly setup your environment and have installed the following dependencies in your venv:

uv pip install pytest-mypy
uv pip install pytest-asyncio