forked from ccf-ai-infra/TileOPs-Metax
22 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Cao Ying
|
ef5d248f2f
|
[Refactor][Benchmark] Extract ManifestBenchmark base class and shared helpers (#915)
## Summary Extract a generic `ManifestBenchmark` base class and shared helpers (`roofline_vars`, `workloads_to_params`) into `benchmarks/benchmark.py`, so manifest-driven bench files only specify the op name and baseline function per test. This eliminates ~300 lines of duplication across 5 benchmark files. Closes #904 ## Changes - **benchmarks/benchmark.py** — Add `ManifestBenchmark(op_name)` subclass of `BenchmarkBase` with `roofline_vars(workload)` and `workloads_to_params(op_name)` public helpers - **benchmarks/ops/bench_{softmax,reduce,argreduce,logical_reduce,vector_norm}.py** — Refactor to use `ManifestBenchmark`, replacing 19 per-op benchmark classes - **scripts/validate_manifest.py** — Teach AST validator to recognize `ManifestBenchmark` subclasses as satisfying `eval_roofline` requirement - **tests/test_validate_manifest.py** — Add tests for indirect bench-validator AST paths and a regression test documenting the subclass override limitation ## Test plan - [x] AC-1: All 5 bench files use `ManifestBenchmark` instead of per-op classes - [x] AC-2: `_roofline_vars` and `_workloads_to_params` exist only in `benchmarks/benchmark.py`, not duplicated - [x] AC-3: `pytest --collect-only` passes for all 5 bench files (63 tests collected) - [x] AC-4: `validate_manifest.py --check-op` produces zero `[bench]` warnings for all 19 affected ops - [x] 89 validator tests pass, lint clean ## Follow-up - #917 — Add import alias and attribute access support to bench AST validator - #918 — Type-narrow roofline_vars parameter from WorkloadBase to typed Protocol --------- Co-authored-by: Ibuki 🍃 — a wind born from Claude Opus <Ibuki-wind@users.noreply.github.com> |
|
|
Cao Ying
|
d81ec1f5ef
|
[Test][Softmax] Add non-aligned edge-case shapes for softmax-family ops (#914)
## Summary Add targeted edge-case test shapes that exercise the M×N non-aligned combination for softmax-family kernels, covering single-tile and multi-tile (large-N) paths. Closes #906 ## Changes **Tests** (`tests/ops/test_softmax.py`): - 3 new shapes per fixture (SoftmaxFixture, LogSoftmaxFixture, LogSumExpFixture): - `(33, 300)` fp32 — both M and N non-aligned, single-tile path - `(33, 33000)` fp16 — both M and N non-aligned, multi-tile with masked loads - `(33, 32768)` fp16 — non-aligned M with large-N tiled path **Scripts** (`scripts/test_node_delta.py`): - Fix base-ref collection: write temp file in original directory instead of `/tmp/` so pytest can resolve project imports **Foundry** (`.foundry/mold/pre-create-checks.md`): - Add test node delta as pre-create check #3 (soft gate) with correct `--base upstream/main` command ## Test plan - [x] AC-1: All 126 tests pass (`pytest tests/ops/test_softmax.py`) - [x] AC-2: New shapes exercise non-aligned M×N paths (single-tile and multi-tile) ## Test node delta ``` File Base HEAD Delta -------------------------------------------------- tests/ops/test_softmax.py 117 126 +9 -------------------------------------------------- TOTAL 117 126 +9 Growth: +7.7% ``` Justification: 9 new `pytest.mark.full` nodes (3 shapes × 3 fixtures). Each shape targets a distinct non-aligned code path not covered by existing aligned shapes. No cross-dtype expansion (existing fixtures already cover fp32/fp16/bf16 on aligned shapes; the non-aligned path shares the same tiling/masked-load logic across fp16 and bf16). ## Follow-up No follow-up issues or suggestions. --------- Co-authored-by: Ibuki 🍃 — a wind born from Claude Opus <Ibuki-wind@users.noreply.github.com> |
|
|
Cao Ying
|
0ce134bb24
|
[Refactor][Manifest] Enforce status as required and validate kernel_map schema (#888)
## Summary Update the manifest validator to enforce `status` as required and validate `kernel_map` schema. - `status` added to `_REQUIRED_TOP` — entries missing it produce a schema error - `status` must be a string (`"implemented"` or `"spec-only"`); null and non-string values rejected - `source.kernel_map` validated when present: must be `dict[str, str]` - Missing `kernel_map` for `status: implemented` entries produces a **warning** (not error) — full enforcement deferred to #887 which populates kernel_map for all existing ops - `_is_spec_only()` defaults to `True` on missing/non-string status (safe for `--levels` bypass) - Integration test validates against the real checked-in manifest ## Changed files - `scripts/validate_manifest.py` — status required, kernel_map validation, _is_spec_only safe default - `tests/test_validate_manifest.py` — 9 new tests, updated helper, integration test ## Test plan - [x] `pytest tests/test_validate_manifest.py` — 84 passed, 1 xfailed - [x] `python scripts/validate_manifest.py` exits 0 on real manifest (25 kernel_map warnings) - [x] Pre-commit hooks pass Closes #886 ## Follow-up - #887 — Populate kernel_map for all implemented ops (converts warnings to clean pass) No additional follow-up issues or suggestions. --------- Co-authored-by: Ibuki 🍃 — a wind born from Claude Opus <Ibuki-wind@users.noreply.github.com> |
|
|
Cao Ying
|
c85712ec45
|
[Refactor][Manifest] Rename keys to PascalCase and add ref_api field (#869)
## Summary
Rename all manifest keys from snake_case to PascalCase class names in
`{Name}{Direction}Op` format and add the required `ref_api` field to
every op entry. Validator enforces exact `cls.__name__ == manifest_key`
identity — no heuristic resolution. No op/kernel/benchmark
implementation code is modified.
**Note**: After this PR merges, benchmark callers that use snake_case op
names (e.g. `load_workloads("rmsnorm_fwd")`) will break. PR-B (#866)
fixes all callers as part of the class rename.
Closes #865
## Test plan
- [x] All manifest keys are PascalCase class names in
`{Name}{Direction}Op` format
- [x] `ref_api` field required for all ops (`"none"` when no external
API counterpart)
- [x] Validator L0 schema enforces `ref_api` as required string
- [x] Validator requires exact `cls.__name__ == manifest_key` — no
single-candidate bypass, no heuristic fallback
- [x] No changes to any file under `tileops/ops/`, `tileops/kernels/`,
or `benchmarks/`
- [x] Integration test `xfail` until PR-B renames Op classes
## Changed files
- `tileops/ops_manifest.yaml` — renamed all 52 keys, added `ref_api` to
every entry
- `tileops/manifest.py` — canonical PascalCase keys only (no legacy
alias resolution)
- `scripts/validate_manifest.py` — `ref_api` required in L0 schema,
exact class-name identity enforcement
- `tests/test_ops_manifest.py` — updated test fixtures, canonical key
tests
- `tests/test_validate_manifest.py` — updated validator tests,
exact-match enforcement tests
---------
Co-authored-by: Ibuki 🍃 — a wind born from Claude Opus <Ibuki-wind@users.noreply.github.com>
|
|
|
Cao Ying
|
dbf58a10b1
|
[Doc][Manifest] Clarify same_as(ref) as identity constraint with validator enforcement (#864)
## Summary Clarify \`same_as(ref)\` as a dtype-only identity constraint in manifest spec, and add validator enforcement. **Docs:** - \`docs/manifest.md\`: R3/R3a define \`same_as(ref)\` as exact dtype identity (not "dependent types"). R5/R8 clarify \`same_as\` is dtype-only — shape must be explicitly declared. R13 status default changed to \`spec-only\`. Overall doc reduced by ~27% (470→342 lines). - \`.claude/domain-rules/manifest-spec.md\`: matching updates — \`same_as\` dtype identity, shape explicit, status default \`spec-only\`. **Validator:** - \`_check_dtype_combos_same_as_identity()\`: enforces that \`dtype_combos\` entries assign the same dtype to \`same_as\`-bound tensors. Rejects partial combos where a bound tensor appears without its reference. - \`_is_spec_only()\`: defaults to \`spec-only\` when \`status\` is absent, matching R13. **Tests:** 5 new test cases — dtype_combos pass, mismatch, multi-binding mismatch, partial-combo rejection, missing-status default. Closes #863 ## Changes - \`docs/manifest.md\` — rewritten: concise, no "dependent types", \`same_as\` dtype-only, status default spec-only - \`.claude/domain-rules/manifest-spec.md\` — matching rule updates - \`scripts/validate_manifest.py\` — \`_build_same_as_map()\`, \`_check_dtype_combos_same_as_identity()\`, \`_is_spec_only()\` default fix - \`tests/test_validate_manifest.py\` — 5 new tests (81 total, all pass) ## Test plan - [x] AC-1: \`docs/manifest.md\` R3/R3a explicitly defines \`same_as(ref)\` as dtype identity constraint - [x] AC-2: Validator flags \`dtype_combos\` with mismatched or partial \`same_as\`-bound entries - [x] AC-3: Full manifest validation passes (\`python scripts/validate_manifest.py\` → 0 errors) ## Follow-up No follow-up issues or suggestions. --------- Co-authored-by: Ibuki 🍃 — a wind born from Claude Opus <Ibuki-wind@users.noreply.github.com> |
|
|
Cao Ying
|
a93643998e
|
[Refactor][Validator] Deduplicate PascalCase resolution loops in _resolve_op_class (#836)
## Summary Consolidates the duplicated resolution heuristics in `_resolve_op_class` into a single strategy loop and adds ambiguity detection to all matching stages (not just the terminal fallback). - Three separate PascalCase loops (stripped_pascal, full_pascal, suffix_fwd_bwd) replaced with single `for`-loop over an ordered strategies list - Suffix matching (`_fwd`/`_bwd`) now emits an ambiguity warning when multiple candidates match, consistent with the pattern established in PR #830 - One new test (`test_suffix_match_ambiguity_emits_warning`) covers the multi-candidate suffix case Closes #811 ## Acceptance Criteria | ID | Criterion | Status | |----|-----------|--------| | AC-1 | Single resolution loop replaces the duplicated PascalCase loops | PASS | | AC-2 | _fwd/_bwd suffix matching emits ambiguity warning when multiple candidates match | PASS | | AC-3 | All existing TestResolveOpClass tests pass unchanged | PASS | | AC-4 | New test covers suffix-match ambiguity (multiple fwd candidates) | PASS | | AC-5 | Full manifest validation passes | PASS | ## Test plan - [x] 25 TestResolveOpClass tests pass (24 pre-existing + 1 new) — 2.93s - [x] Full test suite: 75 passed, 0 failed - [x] `python scripts/validate_manifest.py` — all manifest checks passed - [x] pre-commit lint passes (ruff, codespell, mdformat) ## Follow-up No follow-up issues or suggestions. --------- Co-authored-by: Ibuki 🍃 — a wind born from Claude Opus <Ibuki-wind@users.noreply.github.com> |
|
|
Cao Ying
|
e814e01bf9
|
[Fix][Validator] Emit warning on ambiguous op class resolution instead of silent fallback (#830)
## Summary When `_resolve_op_class` finds multiple candidate classes but no naming heuristic matches, it previously fell back silently to `candidates[0]` (alphabetical order). This could produce wrong validation results without any diagnostic. This PR replaces the silent fallback with a `UserWarning` that identifies the op name, all candidate class names, and the file path, then returns an unresolved result (`cls=None`) so the caller can handle it explicitly. Closes #810 ## Test plan - [x] AC-1: When all heuristics fail, a warning or error is emitted identifying the ambiguous resolution - [x] AC-2: Full manifest validation passes (no existing ops broken) — `python scripts/validate_manifest.py` -> All checks passed - [x] AC-3: Unit test covers the ambiguous-fallback code path — `test_ambiguous_fallback_returns_none_with_warning` added **Test results**: 79 tests passed, 0 failed ## Follow-up - #811 — Consolidate resolution heuristics + fix `_fwd`/`_bwd` suffix-match silent fallback Suggestions: strengthen ambiguous-fallback test to match full diagnostic fields (op_name, candidate_names, op_file) --------- Co-authored-by: Ibuki 🍃 — a wind born from Claude Opus <Ibuki-wind@users.noreply.github.com> |
|
|
Cao Ying
|
2a9b0fad56
|
[Fix][Validator] Resolve op classes in multi-class files by PascalCase convention (#807)
## Summary Fix `_resolve_op_class` to correctly resolve op names to classes when source files contain multiple Op classes (e.g., `reduce.py` with 8 ops). The previous substring heuristic (`all parts in cls_lower`) was too permissive — `var_fwd` matched `AmaxOp` instead of `VarOp` because no class contained `"fwd"`, and the alphabetical fallback picked the wrong candidate. Closes #803 ## Changes ### Resolver heuristic rewrite (`scripts/validate_manifest.py`) - **Removed** the loose substring-based multi-candidate heuristic (`all(p in cls_lower for p in parts)`) - **Added** exact PascalCase match as the primary multi-candidate strategy: strip `_fwd`/`_bwd` suffix, convert remainder to PascalCase + `"Op"` (e.g., `var_fwd` → `VarOp`, `var_mean_fwd` → `VarMeanOp`) - **Added** full-name PascalCase fallback for Fwd/Bwd-suffixed class naming conventions (e.g., `batchnorm_fwd` → `BatchNormFwdOp`) - Retained suffix matching (`fwd`/`bwd` in class name) as final fallback ### Test coverage (`tests/test_validate_manifest.py`) - Added `TestResolveOpClass` class with 21 test cases: - 8 parametrized tests for `reduce.py` multi-class resolution (SumOp, MeanOp, AmaxOp, AminOp, ProdOp, VarOp, StdOp, VarMeanOp) - 11 parametrized tests for all single-file reduction ops (argmax, argmin, all, any, count_nonzero, l1_norm, l2_norm, inf_norm, softmax, log_softmax, logsumexp) - Single-class file resolution, import error handling, no-op-class handling ### Review fixes - Removed duplicate test method flagged by Copilot review - Fixed misleading comment example (`"batchnorm_fwd" → "BatchnormFwdOp"` → `"sum_fwd" → "SumFwdOp"`) ## Test plan - [x] **AC-1**: `_resolve_op_class` returns correct class for all 19 reduction ops (8 multi-class in `reduce.py` + 11 single-file) — verified by parametrized tests - [x] **AC-2**: Existing resolver behavior unchanged for single-class files and Fwd/Bwd-suffixed classes — verified by `test_single_class_file_resolves` - [x] **AC-3**: Unit tests cover multi-class resolution scenarios — 21 test cases in `TestResolveOpClass` **Test results:** 72 passed, 0 failed ## Follow-up - #810 — Emit warning on silent `candidates[0]` fallback - #811 — Deduplicate PascalCase resolution loops --------- Co-authored-by: Ibuki 🍃 — a wind born from Claude Opus <Ibuki-wind@users.noreply.github.com> |
|
|
Cao Ying
|
f1f9ba1517
|
[Feat][Manifest] Add --check-op flag for spec-only validation override (#780)
## Summary Add a `--check-op <name>` CLI flag to the manifest validator that forces all validation levels (L0-L4) on the specified op, bypassing the `status: spec-only` gate. Default (unflagged) behavior is unchanged. Closes #779 ## Test plan | AC | Description | Status | |----|-------------|--------| | AC-1 | `python scripts/validate_manifest.py --check-op softmax_fwd` runs L0-L4 on `softmax_fwd` even when `status: spec-only` | pass | | AC-2 | `python scripts/validate_manifest.py` (without flag) behavior unchanged -- spec-only ops still skip L1-L4 | pass | | AC-3 | Unit test covers both behaviors (with/without flag, nonexistent op, single-op scoping, variant_of regression) | pass | | AC-4 | Modified files pass existing tests (45/45 passed) | pass | ## Validation - 45 tests passed, 0 failed - Pre-commit hooks pass (ruff, codespell, mdformat, gitleaks) - Files changed: `scripts/validate_manifest.py`, `tests/test_validate_manifest.py` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Cao Ying
|
ce6da052ae
|
[Chore][Manifest] Strict L1 signature: check __init__() + forward() param union (#768)
## Summary Make L1 validator strictly require that all manifest-declared params for `status: implemented` ops appear in the Op's `__init__()` + `forward()` union signature. This ensures manifest declarations stay in sync with actual code interfaces. - Update `check_l1_signature()` to inspect both `__init__` and `forward` param names - Audit and fix all 23 implemented ops that failed strict L1 - Update `docs/manifest.md` L1 description to reflect the stricter rule Closes #765 ## Test plan - [x] **AC-1**: check_l1_signature() checks manifest params against __init__ + forward union, not just forward - [x] **AC-2**: All status: implemented ops pass strict L1 in CI - [x] **AC-3**: docs/manifest.md L1 description updated to reflect strict matching rule - [x] **AC-4**: Modified files pass existing tests **Test results**: 47/47 passed (24 validator + 23 manifest), 0 failed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Ang Gao
|
89708ee424
|
[Fix][Benchmark] Add serial validation to correct noisy autotune configs (#773)
## Summary
- Parallel warmup (`-n 8`) causes autotuner to select suboptimal configs
due to GPU contention noise, which get cached and reused by serial
benchmarks
- Add a Phase 2 serial validation pass in `warmup_kernel_cache.py` that
re-tunes on a quiet GPU with `.so` cache hits (compilation instant, only
profiling re-runs)
- Add autotune config column to `BenchmarkReport` so nightly reports
show which config was selected
## How it works
| Phase | Compilation cache (`.so`) | Autotune cache (`best_config`) |
|---|---|---|
| Phase 1: parallel warmup (`-n 8`) | Write (parallel, fast) | Write
(noisy — GPU contention) |
| Phase 2: serial validation | Read (cache hit, instant) | Overwrite
(accurate — exclusive GPU) |
`conftest_warmup.py` supports a new `TILEOPS_WARMUP_VALIDATE=1` mode
that patches `AutoTuner._load_result_from_disk` to return `None`,
forcing re-tune while preserving `.so` cache.
## Evidence
Same kernel (`avg_pool2d vision-3x3-s2`) selects different configs under
contention vs serial:
- **Parallel (8 workers)**: `{'block_m': 128, 'block_c': 64, 'threads':
256}`
- **Serial (3/3 runs consistent)**: `{'block_m': 64, 'block_c': 32,
'threads': 256}`
Fixes #772
## Test plan
- [x] Full warmup + validation pass completes on all benchmark files
- [x] Serial validation correctly overwrites noisy autotune cache
- [x] `BenchmarkReport` includes config column for tileops entries
- [x] Verified on H200 with `tileops-runner:latest`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
ChongLi
|
cddd14e7d3
|
[CI] Add gpu smoke workflow summary report (#764)
Closes #763 ## Summary - add a JUnit-XML-based GPU smoke report generator for workflow summaries - publish GPU smoke summary and artifacts from the gpu-smoke workflow ## Test plan - [x] `python -m py_compile scripts/gpu_smoke_report.py` - [x] generated a sample report from a minimal JUnit XML fixture ## Additional context - `Gpu-smoke ops number` reports testcase count, per issue clarification. - The workflow still uses existing pytest metadata from `tests/conftest.py`; no test selection policy changed. |
|
|
Cao Ying
|
20170a617d
|
[Chore][Testing] Add unit-test policy and test node growth detection script (#761)
## Summary Define a unit-test policy for operator tests and provide a local script (`scripts/test_node_delta.py`) that reports test node count delta for touched files in a PR, making growth visible and justifiable. Closes #760 ## Changes **M1 — Unit-test policy** - Added Unit-Test Policy section to `docs/testing.md` covering: allowed UT purposes, dtype risk class rule, shape coverage rule, manifest separation, growth justification - Added `.claude/rules/testing-budget.md` with concise agent-enforceable version - Verified policy is consistent with existing smoke/full tier system in `tests/conftest.py` **M2 — Growth-detection script** - Created `scripts/test_node_delta.py` that compares test node counts between current branch and main - Added usage instructions to `docs/testing.md` - Added note in `.claude/rules/testing-budget.md` about running script and justifying growth ## Test plan - [x] **AC-1-1**: docs/testing.md contains a Unit-Test Policy section with dtype risk class, shape coverage, manifest separation, and growth justification rules - [x] **AC-1-2**: .claude/rules/testing-budget.md exists and is loadable by Claude Code - [x] **AC-1-3**: Policy does not contradict existing tier enforcement in tests/conftest.py - [x] **AC-2-1**: Script runs successfully on the current repo and reports node counts for at least 3 existing test files - [x] **AC-2-2**: Script correctly detects delta when a test file has more nodes than its main branch version - [x] **AC-2-3**: Script exits 0 regardless of delta (non-blocking) - [x] **AC-2-4**: Script handles new test files (no main branch version) without error - [x] **AC-2-5**: docs/testing.md documents how to run the script 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Cao Ying
|
6c0d687439
|
[Chore][Manifest] Add manifest validator with CI integration (#745)
Closes #738 ## Summary - Add `scripts/validate_manifest.py` implementing manifest validation for schema, `Op.forward()` signature consistency, `shape_rules` syntax, dtype conformance, and benchmark manifest usage - Integrate the validator into the preflight workflow with a dedicated `validate-manifest` job - Add MoE manifest entries and migrate the MoE benchmarks in this PR to manifest-driven workloads via `load_workloads()` and `eval_roofline()` - Make benchmark strictness explicit with `source.bench_manifest_driven: true` for migrated entries, while leaving legacy non-migrated benchmarks as warnings - Clarify in `docs/manifest.md` that `workloads` are for nightly/performance benchmark coverage only, not unit-test coverage - Add merge-ready validator coverage in `tests/test_validate_manifest.py` - Add a developer-agent rule in `.claude/rules/manifest-validator.md` forbidding validator edits as a way to bypass manifest constraints ## Test plan - [x] pre-commit passed (all hooks green) - [x] `pytest -q tests/test_validate_manifest.py` -- 20 passed - [x] `pytest -q tests/ops/test_moe_permute.py` -- 9 passed - [x] `python scripts/validate_manifest.py` passes on current codebase (exit 0) ## Structural Readiness SKIP -- primary deliverable is infrastructure (manifest validator + CI integration), not a new kernel/op. ## Additional context **Validation checks:** - **Schema**: YAML structure, required fields, and field types - **Signature**: `Op.forward()` parameters must match manifest inputs, plus any accepted manifest-declared runtime params, in order - **Shape**: `shape_rules` must parse as valid Python expressions - **Dtype**: manifest dtype strings and workload dtypes must be recognized - **Benchmark**: migrated benchmark files must import and call `load_workloads` / `eval_roofline` with the correct op name **Current enforcement model:** - `status: spec-only` entries receive schema-only validation - L4 benchmark enforcement is hard-fail only for entries that explicitly opt in with `source.bench_manifest_driven: true` - Legacy benchmark files that have not yet migrated remain warnings, not CI failures --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Ang Gao
|
857d875d19
|
[Bench] Add FA3/FlashInfer baselines across all attention benchmarks (#734)
## Summary - **Fix FA3 import**: `bench_mha.py` used `import flash_attn_interface` (not installed), silently falling back to `torch-sdpa`. Fixed to `from flash_attn import flash_attn_func`. - **Add FA3 baselines**: `bench_mha_decode`, `bench_gqa_decode`, `bench_mha_decode_paged`, `bench_gqa_decode_paged` — previously only had `torch-sdpa`/`torch-ref`. - **Add FlashInfer baselines**: All 8 attention benchmark files now include FlashInfer baselines using batched APIs (`BatchPrefillWithRaggedKVCacheWrapper` / `BatchDecodeWithPagedKVCacheWrapper`). - **Fairness fix**: Replaced per-batch `single_prefill_with_kv_cache` loops with batched wrappers. CUPTI sums all kernel durations — per-batch loops launch B separate kernels that can't exploit cross-batch parallelism. - **bf16 fix**: FlashInfer `wrapper.plan()` defaults `q_data_type='float16'`. Added explicit `q_data_type=q.dtype` to all plan() calls. Note: the decode wrapper's `data_type` param does NOT propagate to `q_data_type`. - **Multi-baseline report**: `conftest.py` writes all baselines to JUnit XML with tag-prefixed properties. `nightly_report.py` parses and displays multiple baselines per config. ## Scope | File | Changes | |:-----|:--------| | `bench_mha.py` | Fix FA3 import + add FlashInfer fwd | | `bench_mha_decode.py` | Add FA3 + FlashInfer | | `bench_gqa.py` | FlashInfer: per-batch loop to batched API | | `bench_gqa_decode.py` | Add FA3 + FlashInfer | | `bench_mha_decode_paged.py` | Add FA3 (page>=256) + FlashInfer paged | | `bench_gqa_decode_paged.py` | Add FA3 (page>=256) + FlashInfer paged | | `bench_gqa_sliding_window_fwd.py` | Add FlashInfer (wr<0 only) | | `bench_gqa_sliding_window_varlen_fwd.py` | Add FlashInfer ragged (wr<0 only) | | `conftest.py` | Multi-baseline JUnit XML properties | | `nightly_report.py` | Multi-baseline parsing, display, alerts | ## Skipped (expected) - **bwd**: FlashInfer has no backward API - **window_right >= 0**: FlashInfer only supports `window_left` - **FA3 paged with page_size < 256**: FA3 requires page_block_size multiple of 256 ## Test plan - [x] 36/36 benchmarks passed in `tileops-runner:latest` on H200 (GPU1, 1830 MHz locked) - [x] bf16 configs verified: FlashInfer data present for all non-bwd, non-window_right configs - [x] Multi-baseline XML properties correctly written (fa3 + flashinfer tags with latency/ratio) - [ ] Nightly CI should now report fa3 and flashinfer baselines across all attention ops --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Ang Gao
|
8bccfa30e7
|
[CI] Fix warmup OOM, enable autotuner caching, skip baselines (#685)
## Summary Three fixes to the nightly warmup phase: 1. **Fix GPU OOM causing warmup to hang** — pytest-xdist workers accumulated GPU memory over their lifetime; idle workers held ~140GB while the last worker starved and hung at 98%. Added `gc.collect()` + `torch.cuda.empty_cache()` after each test. 2. **Enable autotuner result caching** — Removed `TILELANG_AUTO_TUNING_DISABLE_CACHE=1` and the `do_bench` mock patch. Warmup now runs real GPU profiling and saves `best_config.json` to disk. Subsequent warmup and benchmark runs hit the autotuner cache (~2ms per kernel instead of minutes). 3. **Skip baseline profiling during warmup** — Patched `BenchmarkBase.profile` to return dummy results for non-Op functors (torch-cublas, FA3, etc.). Baseline measurements serve no purpose during warmup. Also bumped default pytest-xdist workers from 8 to 16. **Result**: warmup dropped from 38min (hanging at 98%) to 9min (912/912 passed). ## Changes - `scripts/warmup_kernel_cache.py` — Remove `TILELANG_AUTO_TUNING_DISABLE_CACHE=1`, default `-n 16` - `scripts/conftest_warmup.py` — Remove `do_bench` mock, add GPU memory cleanup hook, add baseline skip - `.github/workflows/nightly.yml` — Bump warmup to `-n 16` ## Test plan - [x] Warmup completes without OOM (912/912 passed, GPU memory stable at ~20-30GB) - [x] Warmup time: 38min (hanging) → 9min with caches warm - [x] Autotuner cache populated (best_config.json written to disk) - [ ] Verify all 4 nightly phases complete via `workflow_dispatch` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Ang Gao
|
3471d9adfd
|
[Fix][CI] Install external baseline libraries in nightly benchmarks (#673)
## Summary Closes #670. Nightly benchmarks were missing external baseline libraries (flash-attn, fla, vllm, sgl-kernel, native_sparse_attention), causing benchmarks to fall back to slow PyTorch reference implementations and inflating speedup numbers. - Add `bench` extras group in `pyproject.toml` with pinned versions (flash-attn 2.8.3, flash-linear-attention 0.4.2, vllm 0.18.0, sgl-kernel 0.3.21) - Relax torch upper bound from `<2.10.0` to `<2.11.0` for vllm 0.18.0 compatibility (tilelang 0.1.8 verified compatible with torch 2.10.0) - Add "Install benchmark baseline libraries" step in nightly Phase 1 (cache-warmup) and Phase 2 (benchmark), always executed regardless of `VENV_REUSED` — pip is idempotent and completes in seconds when versions are unchanged - Exclude `bench` group from `dep_hash.py` so bench version bumps do not trigger full venv rebuilds or invalidate kernel compile caches - Pin `native_sparse_attention` to commit `bd67af59` (not on PyPI); gracefully skip on GitHub network errors - Add `install-bench` Makefile target for local development ## Test plan - [x] `python scripts/dep_hash.py` produces identical hash before and after adding `bench` group (confirms exclusion works) - [x] `pip install --dry-run` resolves all bench deps + core deps without conflicts (torch 2.10.0, tilelang 0.1.8, vllm 0.18.0, flash-attn 2.8.3, flash-linear-attention 0.4.2, sgl-kernel 0.3.21) - [ ] Next nightly run should show benchmarks using real baselines (fa3, fla, etc.) instead of torch fallbacks 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
|
|
Ang Gao
|
a49f6aeed2
|
[Fix][Bench] Fix 11 nightly benchmark failures and report omission (#635)
## Summary - **Backward baselines fail under `no_grad()`**: `BenchmarkBase.profile()` wraps calls in `torch.no_grad()`, breaking autograd-based torch SDPA backward baselines when FA3 is unavailable. Fixed by adding `@torch.enable_grad()` to baseline functions in `bench_engram_bwd`, `bench_gqa`, and `bench_mha`. - **mha_decode `result_idx` OOB**: When autotune selects `num_split>1` but `seqlen_kv` is too small, the runtime fallback to `num_split=1` recompiles `mha_decode_no_split` (5 params) with `out_idx=[7]` (resolved from the 8-param split function). Fixed by guarding the split kernel's `logsum` division against zero (empty splits) and removing the runtime `num_split` fallback. - **Nightly report silently drops benchmark failures**: `aggregate_bench_results()` filters `outcome != "passed"`, so failed benchmarks never appear. Added `collect_bench_failures()`, a "Benchmark Failures" section, summary row, and health indicator integration. Closes #634 ## Test plan - [x] `bench_engram_bwd` — all 4 cases pass locally (no FA3) - [x] `bench_gqa::test_gqa_bwd_bench` — all 3 cases pass locally (no FA3) - [x] `bench_mha::test_mha_bwd_bench` — all 3 cases pass locally (no FA3) - [x] `bench_mha_decode::test_mha_decode_bench[short-kv-tail]` — passes locally - [x] `test_mha_decode` unit tests — all 3 cases pass (including `seqlen_kv=5`) - [x] Nightly report shows benchmark failures in synthetic test 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Ang Gao
|
51fb1ef0bb
|
[Fix][Bench] Fix nightly report: broken summary, formatting, missing baselines (#626)
## Summary Closes #625 Four fixes for the nightly report pipeline: ### A. Remove broken benchmark step summary The Phase 2 bash XML parser failed on single-line XML (`<testsuites>` and `<testsuite>` on same line → regex never matched → all counters 0). Removed entirely since Phase 3 nightly report already covers this. ### B. Improve nightly report aesthetics - Emoji status indicators, color-coded ratio circles - `<details>` collapsible sections for long tables (120+ correctness, 600+ benchmark rows) - Right-aligned numeric columns, bold Op names, `code` module paths - Emoji unicode centralized as module-level constants ### C. Annotate strategy benchmark entries 54 `test_fused_gated_strategy_bench` entries had blank Via column. Added `tileops_variant` property in conftest → report shows `strategy: direct` / `strategy: explicit_parallel`. ### D. Add missing baseline to `bench_engram_bwd.py` 4 entries had no baseline at all. Added `test.ref_program` with `tag="torch"`. ## Report Preview **Header & Summary (all healthy):** ``` # ✅ TileOPs Nightly Report > **2026-03-23 04:30** | `14e2cbc` | NVIDIA H200 | | | |---|---| | **Correctness** | ✅ (788/788 tests across 123 ops) | | **Benchmarked Ops** | 129 | | **Regressions** (vs 14-day best) | ✅ None | | **Baseline Alerts** (< 80%) | ⚠️ 58 | ``` **Header & Summary (with failures):** ``` # ❌ TileOPs Nightly Report | **Correctness** | ❌ 1 failed (791/793 tests across 124 ops) | | **Regressions** (vs 14-day best) | ⚠️ 1 | | **Improvements** (vs 14-day best) | 🎉 1 | ``` **Regression table:** ``` ## ⚠️ Performance Regressions (vs 14-day best) | Op | Config | Best (ms) | Current (ms) | Delta | TFLOPS | |:---|:-------|----------:|-----------:|------:|-------:| | **SoftmaxOp** | test_softmax_bench[mainstream-fp16] | 0.0050 | 0.0161 | +222.0% | 1.04 | ``` **Strategy annotation in Via column:** ``` | | gelu_and_mul_strategy | ...-direct] | 0.0173 | ... | strategy: direct | - | | | gelu_and_mul_strategy | ...-explicit_parallel] | 0.0109 | ... | strategy: explicit_parallel | - | ``` ## Test plan - [ ] Verify nightly report renders correctly on next nightly run - [ ] Verify engram bwd baseline rows appear in bench_results.xml - [ ] Verify strategy entries show `strategy: xxx` in Via column 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|
|
Ang Gao
|
14e2cbceae
|
[CI] Add Op-level nightly report with correctness, performance, and regression tracking (#620)
## Summary
- Inject Op metadata (class name, module path, error metrics, perf data)
into pytest JUnit XML `<property>` elements
- Add `scripts/nightly_report.py` to generate an Op-granularity markdown
report from JUnit XML
- Integrate report generation into nightly CI workflow with
`perf_history.json` artifact for 14-day regression tracking
## Changes
### Data collection (pytest side)
- **`tests/test_base.py`**: `check()` now records
`op.__class__.__name__`, `op.__class__.__module__`, `max_abs_err` via
logging + thread-local
- **`tests/conftest.py`**: `pytest_runtest_call` hookwrapper writes Op
info to `item.user_properties` → JUnit XML `<property>`
- **`benchmarks/benchmark.py`**: `record()` accepts Op objects
(backward-compatible), accumulates tileops + baseline data
- **`benchmarks/conftest.py`**: hookwrapper writes perf data + baseline
ratio to `item.user_properties`
- **`benchmarks/ops/bench_*.py`** (52 files): `record("name", ...)` →
`record(op, ...)`
### Report generation
- **`scripts/nightly_report.py`**: Parses JUnit XML, aggregates by Op,
detects regressions vs 14-day best, baseline alerts (ratio < 80%),
generates markdown
- **`.github/workflows/nightly.yml`**: Downloads artifacts, runs report
script, posts to `$GITHUB_STEP_SUMMARY`, manages `perf_history.json`
artifact
### JUnit XML output format
```xml
<!-- Correctness test -->
<testcase name="test_gqa_fwd[smoke-fwd-fp16]">
<property name="op" value="GroupQueryAttentionFwdOp"/>
<property name="op_module" value="tileops.ops.gqa"/>
<property name="max_abs_err" value="2.44e-04"/>
</testcase>
<!-- Benchmark -->
<testcase name="test_gqa_fwd_bench[throughput-fp16]">
<property name="op" value="GroupQueryAttentionFwdOp"/>
<property name="tileops_latency_ms" value="0.0104"/>
<property name="tileops_tflops" value="206.26"/>
<property name="baseline_tag" value="FA3"/>
<property name="baseline_latency_ms" value="0.35"/>
<property name="baseline_ratio" value="0.83"/>
</testcase>
```
## Test plan
- [x] `pytest tests/ops/test_gqa.py -m smoke --junit-xml=test_out.xml` →
XML contains `op`, `op_module`, `max_abs_err` properties
- [x] `pytest benchmarks/ops/bench_gqa.py -k prefill
--junit-xml=bench_out.xml` → XML contains perf properties
- [x] `python scripts/nightly_report.py --test-xml test_out.xml
--bench-xml bench_out.xml --output report.md` → generates valid markdown
report
- [x] All pre-commit hooks pass (ruff, codespell, etc.)
- [ ] Nightly workflow dispatch to verify CI integration
Closes #618
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
Ang Gao
|
27c6688808
|
[BugFix][CI] Fix warmup error handling, monkeypatch cleanup, and portable cache paths (#548)
## Summary
- Fix warmup script to propagate infrastructure errors (pytest exit
codes 2-5) while still tolerating test failures (exit code 1)
- Restore original `ThreadPoolExecutor` in `pytest_unconfigure` to
prevent global state leaks
- Replace all hardcoded `/home/ci-runner/` paths with `${HOME}/` for
runner portability
Closes #545
Closes #546
## Changes
### 1. Warmup exit code handling (#545)
```diff
- sys.exit(0) # unconditional
+ if exit_code in (0, 1):
+ sys.exit(0) # tests passed or failed — compilation succeeded
+ else:
+ sys.exit(exit_code) # infrastructure error — surface it
```
| Exit Code | Meaning | Before | After |
|-----------|---------|--------|-------|
| 0 | All passed | exit 0 | exit 0 |
| 1 | Test failures | exit 0 | exit 0 (compilation still worked) |
| 2 | Interrupted / bad args | exit 0 (swallowed) | **exit 2** |
| 3 | Internal error | exit 0 (swallowed) | **exit 3** |
| 4 | Usage error | exit 0 (swallowed) | **exit 4** |
| 5 | No tests collected | exit 0 (swallowed) | **exit 5** |
### 2. ThreadPoolExecutor cleanup (#545)
```diff
def pytest_unconfigure(config):
patcher = getattr(config, "_warmup_patcher", None)
if patcher is not None:
patcher.stop()
+
+ orig_pool = getattr(config, "_warmup_orig_pool", None)
+ if orig_pool is not None:
+ concurrent.futures.ThreadPoolExecutor = orig_pool
```
### 3. Portable cache paths (#546)
```diff
- TILELANG_CACHE_DIR="/home/ci-runner/.tilelang/cache"
+ TILELANG_CACHE_DIR="${HOME}/.tilelang/cache"
```
Applied to all 4 jobs (cache-warmup, benchmark, op_test, packaging). 14
occurrences replaced.
## Test plan
- [ ] Verify warmup exits non-zero when pytest encounters infrastructure
errors
- [ ] Verify warmup exits 0 when tests fail but compilation completed
- [ ] Verify no hardcoded `/home/ci-runner/` paths remain
- [ ] Verify cache paths resolve correctly on CI runner
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
Ang Gao
|
7b94393188
|
[CI] Separate kernel compilation from profiling and parallelize compilation in nightly (#542)
## Summary - Add a `cache-warmup` job to the nightly workflow that pre-compiles all benchmark kernel variants before the benchmark job runs, separating compilation (CPU-bound) from profiling (GPU-bound) - Fix venv cache hash to only include dependency fields from `pyproject.toml`, preventing unnecessary rebuilds from linting/style config changes - Restructure nightly job flow: `cache-warmup → benchmark → op_test → packaging` ## Background In [nightly run #23161046010](https://github.com/tile-ai/TileOPs/actions/runs/23161046010), the benchmark job hit the 90-minute timeout at only 48% progress. Analysis showed ~80-90% of the time was spent on kernel compilation (`g++`/`cc1plus` still running at cancellation), not GPU profiling. Root causes: 1. **Venv prefix rename** in #530 caused `VENV_REUSED=false` → pip reinstalled tilelang → `libtilelang.so` mtime changed → **all kernel caches invalidated** (tilelang cache keys include `.so` mtime) 2. **`tune=True` benchmarks** (GEMM 288 configs, MHA, etc.) compile hundreds of kernel variants through TIR → CUDA → nvcc/g++ pipeline, all serially during pytest execution ## Changes ### 1. Kernel cache warmup (`scripts/warmup_kernel_cache.py`) - Monkeypatches `tilelang.profiler.do_bench` to return dummy value → compilation happens normally, GPU profiling skipped - Sets `TILELANG_AUTO_TUNING_DISABLE_CACHE=1` → dummy results not persisted to autotuner cache - Two levels of parallelism: - **pytest-xdist** (`-n 8`): 8 worker processes running different benchmark test cases simultaneously - **ThreadPoolExecutor** (`--max-workers 64`): within each `tune=True` op, autotuner compiles up to 64 config variants in parallel - **Zero maintenance**: new ops with benchmarks are automatically discovered and pre-compiled - Supports `--shard`/`--total-shards` for future parallelization across runners ### 2. Venv hash fix (Closes #540) ```diff - sha256sum pyproject.toml + python3 scripts/dep_hash.py # hashes only dependencies, optional-deps, requires-python, build-requires ``` Applied to all 4 jobs (cache-warmup, benchmark, op_test, packaging). Changes to ruff rules, codespell config, pytest markers etc. no longer trigger venv rebuild. ### 3. Job dependency restructure & timeout adjustments ``` cache-warmup (60 min) → benchmark (120 min) → op_test (180 min) → packaging ``` GPU jobs (benchmark, op_test) remain serial for measurement accuracy. Compilation cost is moved to the warmup phase. ## Local benchmark results (H200) Tested on a clean tilelang cache to simulate a worst-case cold start: | Phase | Duration | |-------|----------| | Warmup (n=8, --max-workers 64, cold cache) | 24 min | | Benchmark (warm cache, first run) | 73 min | | **Total** | **97 min** | ### Comparison with the previous approach | | Before | After (this PR) | |---|---|---| | Compilation | Serial, interleaved with profiling. 90 min timeout at 48% — estimated **3h+** to complete | Warmup job: 8 pytest workers × 64 compile threads, **24 min** | | Benchmark | Blocked behind compilation, never finished | All kernels pre-compiled, **73 min** (first run) | | **Total** | **> 3h (timeout, incomplete)** | **97 min (complete)** | ### Expected time in subsequent nightly runs (steady state) Both the kernel cache and autotuner result cache are stored in persistent directories on the self-hosted runner (`TILELANG_CACHE_DIR`). As long as the venv is not rebuilt (i.e., dependencies unchanged), these caches persist across nightly runs: | Phase | First run (cold cache) | Subsequent runs (warm cache) | |-------|----------------------|---------------------------| | Warmup | 24 min (full compilation) | **~2-3 min** (all cache hits, no actual compilation) | | Benchmark | 73 min (autotuner profiles all configs) | **~30-40 min** (autotuner cache hit → skip profiling 288 configs per op, only benchmark the best config) | | **Total** | **97 min** | **~35-45 min** | ### Additional notes - **Deepseek kernel issues**: Several Deepseek-related kernels currently have compilation or runtime issues (6 consecutive failures at ~48% progress). Once fixed, the time spent on their failed compilation/retry will be recovered. - **Warmup failures are non-blocking**: The warmup job uses `continue-on-error: true` and always exits 0. Any kernel that fails to compile during warmup will simply be compiled on-demand during the benchmark job (same behavior as before this PR). ## Test plan - [x] Verify warmup script compiles kernels and populates cache (local H200) - [x] Verify benchmark completes with warm cache (local H200, 73 min) - [x] Verify n=8 xdist parallelism works (24 min vs n=4 at 32 min) - [ ] Verify nightly completes within timeout on CI runner - [ ] Verify venv hash is stable across non-dependency pyproject.toml changes Closes #540 Closes #541 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |