FrRay
/
TileOPs-Metax
forked from ccf-ai-infra/TileOPs-Metax
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: FrRay:main
Branches Tags
FrRay:main
FrRay:summer-camp-2026
FrRay:feat/group-count
FrRay:backup/summer-camp-2026-before-sync-20260803
FrRay:stats
FrRay:nightly-bench
ccf-ai-infra:summer-camp-2026
ccf-ai-infra:stats
ccf-ai-infra:main
ccf-ai-infra:nightly-bench
...
pull from: FrRay:nightly-bench
Branches Tags
FrRay:summer-camp-2026
FrRay:feat/group-count
FrRay:backup/summer-camp-2026-before-sync-20260803
FrRay:stats
FrRay:main
FrRay:nightly-bench
ccf-ai-infra:summer-camp-2026
ccf-ai-infra:stats
ccf-ai-infra:main
ccf-ai-infra:nightly-bench

No commits in common. "main" and "nightly-bench" have entirely different histories.
main ... nightly-be

571 changed files with 272 additions and 157147 deletions
Show Stats Expand all files Collapse all files

105
.claude/conventions/types.sh
View File

@ -1,105 +0,0 @@
#!/bin/bash
# types.sh — Single source of truth for TileOPs commit/PR/issue type conventions.
#
# Usage:
# source .claude/conventions/types.sh
# # Now use $COMMIT_MSG_PATTERN, $BRANCH_NAME_PATTERN, TYPE_TO_LABEL, etc.
#
# This file is sourced by validation scripts, CI workflows, and skill docs.
# Update HERE first, then all consumers pick up the change automatically.
# ---------------------------------------------------------------------------
# Commit / PR types
# ---------------------------------------------------------------------------
COMMIT_PR_TYPES="Bench|BugFix|Chore|CI|Design|Doc|Enhancement|Feat|Fix|Maintain|Perf|Refactor|Style|Test"
# Full commit message / PR title regex
# [Type] description or [Type][Scope] description
COMMIT_MSG_PATTERN="^\[(${COMMIT_PR_TYPES})\](\[[a-zA-Z0-9_-]+\])? .+"
# ---------------------------------------------------------------------------
# Branch naming
# ---------------------------------------------------------------------------
BRANCH_PREFIXES="bench|chore|design|doc|feat|fix|maintain|perf|refactor|style|test"
BRANCH_NAME_PATTERN="^(${BRANCH_PREFIXES})/[a-z0-9._-]+/[a-z0-9._-]+$"
# ---------------------------------------------------------------------------
# Type → GitHub label mapping
# ---------------------------------------------------------------------------
declare -A TYPE_TO_LABEL=(
[Bench]=bench
[BugFix]=fix
[Chore]=chore
[CI]=ci
[Design]=design
[Doc]=docs
[Enhancement]=enhancement
[Feat]=feature
[Fix]=fix
[Maintain]=maintain
[Perf]=perf
[Refactor]=refactor
[Style]=style
[Test]=test
)
# All type-related labels (used for stale-label cleanup)
ALL_TYPE_LABELS="bench bug chore ci design docs enhancement feature fix maintain perf refactor style test"
# ---------------------------------------------------------------------------
# Issue types (ALL CAPS, used in issue titles)
# ---------------------------------------------------------------------------
ISSUE_TYPES="BENCHMARK|BUG|DESIGN|DOCS|FEAT|MAINTAIN|META|PERF|REFACTOR|STYLE|TEST"
# Issue type → GitHub label (for auto-label workflow)
declare -A ISSUE_TYPE_TO_LABEL=(
[BENCHMARK]=bench
[BUG]=bug
[DESIGN]=design
[DOCS]=docs
[FEAT]=feature
[MAINTAIN]=maintain
[META]=chore
[PERF]=perf
[REFACTOR]=refactor
[STYLE]=style
[TEST]=test
)
# Issue type → commit/PR type prefix
declare -A ISSUE_TO_COMMIT_TYPE=(
[BENCHMARK]=Bench
[BUG]=BugFix
[DESIGN]=Design
[DOCS]=Doc
[FEAT]=Feat
[MAINTAIN]=Maintain
[META]=Chore
[PERF]=Enhancement
[REFACTOR]=Refactor
[STYLE]=Style
[TEST]=Test
)
# Issue type → branch prefix
declare -A ISSUE_TO_BRANCH_PREFIX=(
[BENCHMARK]=bench
[BUG]=fix
[DESIGN]=design
[DOCS]=doc
[FEAT]=feat
[MAINTAIN]=maintain
[META]=chore
[PERF]=perf
[REFACTOR]=refactor
[STYLE]=style
[TEST]=test
)
# Default for unrecognized issue types
ISSUE_DEFAULT_COMMIT_TYPE="Fix"
ISSUE_DEFAULT_BRANCH_PREFIX="fix"

14
.claude/domain-rules/benchmark.md
View File

@ -1,14 +0,0 @@
## Boundary
- **OWNS**: `benchmarks/`
- **MUST NOT WRITE**: `tileops/ops/`, `tileops/kernels/`, `tests/`, `workloads/`, `tileops/manifest/`
- **MUST NOT** (oracle-leakage rule): import oracle/ref functions from `tests/` or `workloads/`. Reads of any other file are unrestricted.
→ [trust-model.md §Benchmark](../../docs/design/trust-model.md#benchmark) | [testing.md §Benchmarks](../../docs/design/testing.md#benchmarks)
______________________________________________________________________
- Every benchmark records ≥1 non-`tileops` baseline. If the external baseline is conditional, add a local torch fallback.
- Tag names: lowercase, hyphen-separated. Tags starting with `tileops` are TileOPs entries; everything else is a baseline.
- `calculate_flops()` / `calculate_memory()` return `None` to omit the metric.
- Benchmark shapes reflect real DNN workloads (LLaMA-family by default). Annotate shape constants with the model/scenario; never use arbitrary flat numbers (262K, 1M, 4M).

2
.claude/domain-rules/design-docs.md
View File

@ -1,2 +0,0 @@
- `docs/design/*.md` records top-level design decisions only — no file enumerations, line counts, or other implementation snapshots.
- When a reviewer asks to add implementation detail (e.g. "list the current files"), push back unless the design itself changed. Test: would this wording survive the next 5 implementations?

56
.claude/domain-rules/manifest-spec.md
View File

@ -1,56 +0,0 @@
## Boundary
- **OWNS**: `tileops/manifest/`
- **MUST NOT WRITE**: `tileops/ops/`, `tileops/kernels/`, `tests/`, `benchmarks/`
- Manifest changes require human review in a separate PR.
→ [trust-model.md §Manifest](../../docs/design/trust-model.md#manifest)
______________________________________________________________________
- Manifest key must equal the Op `cls.__name__` exactly. Class-naming convention: see [ops-design.md](ops-design.md).
- `ref_api` (required): the external API the signature mirrors (e.g. `torch.nn.functional.rms_norm`); `"none"` if none. Validator enforces presence + string type only; semantics not checked.
- `inputs`, `outputs`, `params` are ordered dicts — key order is signature position. Don't reorder.
- Op signatures must match PyTorch's public API (names, set, semantics); include every supported parameter even if the kernel only honors the default. Default to `__init__` kwargs (lifetime-fixed); use `forward()` only when the reference API requires it or the value is per-batch — justify in the introducing issue.
- `dtype` syntax: `|` for alternatives. `same_as(ref)` is dtype-only identity (matches `ref` at runtime, no extra axis in `dtype_combos`, never used for shape).
- `dtype_combos` only when the supported set is a strict subset of the Cartesian product. Omit when all combinations are valid.
- Output shapes are fully specified by `shape` and/or `shape_rules`. `shape` present → fixed rank, names become roofline variables; `shape` absent on inputs → arbitrary rank, use `params` + `shape_rules`. Shared dim names across tensors → sizes must match.
- `shape_rules` are Python expressions describing shape relationships. For reduction-dim validation, use the canonical predicates / extractors in `tileops.manifest.shape_rules` (callable by bare name from any rule body); never silently wrap out-of-range indices with `% x.ndim`. Inline string expressions are a transitional fallback only.
- **Reduction `dim` authoring contract.** When `dim` accepts an integer or a sequence (`list[int]` / `tuple[int, ...]`), declare three `shape_rules` in this order:
1. **Range validity.** Every axis in `[-x.ndim, x.ndim)`. For ops accepting `None`: `"dim is None or all(-x.ndim <= d < x.ndim for d in ([dim] if isinstance(dim, int) else dim))"`. Drop the `dim is None or` prefix when the op does not accept `None`.
1. **Normalize negatives.** Downstream rules apply `% x.ndim` only after step 1, producing the canonical axis set `{d % x.ndim for d in dim}`.
1. **Uniqueness (sequence only).** `"isinstance(dim, (int, type(None))) or len({d % x.ndim for d in dim}) == len(dim)"`.
Empty-sequence semantics is per-op:
- Ops accepting `dim=None` (`sum`, `mean`, `amax`, `amin`, `var`, `std`, `var_mean`, `all`, `any`, `count_nonzero`, `linalg.vector_norm` variants): empty sequence ≡ full reduction; formulas use `set(range(x.ndim))` as fallback.
- Ops without `dim=None` (e.g. `logsumexp`): empty sequence is invalid; declare `"isinstance(dim, int) or len(dim) > 0"`.
- Roofline `vars` maps variable names to Python expressions over tensor shapes and params. Required for arbitrary-rank ops.
- `status` is required: `implemented` or `spec-only`.
- `torch_compile_fullgraph`: literal `true` only; omit for no promise; invalid on `spec-only`. Declare only ops with a registered cold `fullgraph=True` compile test. Semantics: [manifest.md](../../docs/design/manifest.md#torch_compile_fullgraph).
- No `Optional[Tensor]` in manifest. Conditional inputs split into variant entries linked by `variant_of` (single-level, no chaining). Variants share `source.kernel` and `source.op`; each carries its own `signature`, `workloads`, `roofline`.
- Tensor layout defaults to contiguous row-major. Non-default needs an explicit `layout` field; `shape` dim names reflect memory order.
- `source.kernel_map` is the Op→Kernel dispatch registration table (`dispatch_key: KernelClassName`). It declares what an Op uses, not how dispatch picks.
- Never modify manifest to match non-conforming code. Code drift → `status: spec-only` and fix code in a follow-up PR. Never remove `params`, roofline `vars`, or `shape_rules` to silence validator errors.
- `ref_api` is the spec oracle for the manifest signature, not an Op-layer dispatch target at forward time.
- **Manifest comment policy.** Comments may carry technical content the DSL can't express (schema clarifications, edge cases, conventions, file headers); they MUST NOT carry process metadata bound to a specific issue, PR, commit, or round. Keep only if meaningful after every issue/PR is renumbered; otherwise move to commit message, PR description, or follow-up issue.
Discovery scan: `grep -rnE '#[0-9]{3,}|[Ff]ollow.?up|AC-[0-9]+' tileops/manifest/*.yaml`

5
.claude/domain-rules/manifest-validator.md
View File

@ -1,5 +0,0 @@
- Validator failures mean the op, manifest, or benchmark is wrong — **not** the validator. Do not edit validator code, downgrade errors to warnings, add skip paths, or special-case a new op to get CI green.
- Manifest-key match is direct string equality on `cls.__name__`. Do not add normalization, casing fallbacks, or alias paths.
- Validator-related infrastructure: `scripts/validate_manifest.py`, `tests/test_validate_manifest.py`, `docs/design/manifest.md`. Only modify these when the PR is explicitly about manifest schema, trust-model policy, or validator infrastructure — and document the change in the PR body.
- If a new op cannot satisfy the current validator: stop and surface the mismatch. Do not "fix" it in the validator.
- Full spec: [docs/design/manifest.md](../../docs/design/manifest.md) (Manifest Validation section).

28
.claude/domain-rules/ops-design.md
View File

@ -1,28 +0,0 @@
## Boundary
- **OWNS**: `tileops/ops/`, `tileops/kernels/`
- **MUST NOT WRITE**: `tests/`, `benchmarks/`, `workloads/`, `tileops/manifest/`
→ [trust-model.md §Implementation](../../docs/design/trust-model.md#implementation) | [ops-design.md](../../docs/design/ops-design.md)
______________________________________________________________________
- Class names: PascalCase `{Name}{Direction}Op` (Op layer) or `{Name}{Direction}Kernel` (Kernel layer); direction suffix mandatory. Manifest author chooses `{Name}`. Builder functions stay snake_case.
- `kernel_map` is the Op→Kernel dispatch registration table: snake_case dispatch keys (decoupled from class names) → Kernel class names. Manifest declares it; agents implement the listed Kernels. See [ops-design-reference.md § Kernel Dispatch](../../docs/design/ops-design-reference.md#kernel-dispatch-kernel_map).
- Op `__init__` is keyword-only (`def __init__(self, *, ...)`). Parameter names come from the manifest: `shape` dim names (fixed-rank), `static_dims` keys (arbitrary-rank), `params` keys. Only manifest-declared information belongs in `__init__`.
- Arbitrary-rank ops declare construction-time values via manifest `static_dims`. Each entry is a single-axis reference `<tensor>.shape[<const_or_param>]`; other dims come from tensors at forward time. See [manifest.md R20](../../docs/design/manifest.md).
- Update `docs/design/ops-design.md` whenever you add/modify an intermediate base class, change a kernel-dispatch pattern, or introduce a new class-variable protocol.
- A new op family inheriting `Op` directly: first check whether an existing family's `forward()` flow already fits before creating a new base class. Record the decision in the PR.
- Per-op workarounds MUST NOT be promoted to a base-class shared mechanism (mixin, class attribute, shared method, opt-out flag) within the same op-family migration PR — even when multiple ops share the workaround. Promote only via a separate design PR that shows the mechanism is a genuine family invariant (would belong in the base even if no op had taken a shortcut), not a shared shortcut.
- PyTorch fallback at forward time is permitted only when TileLang cannot express the operation at the required shape AND no closed-form replacement exists in tensor primitives; document the call site with the blocking limitation and a tracking issue. Helper conveniences (`x.float().mean(...)` for clarity) are out of scope — the rule targets full-operator delegation.
- Inline roofline state contract: for every `signature.inputs` / `signature.params` name **referenced** by the op's manifest `roofline` expressions, the op exposes it on `self`. Inputs: `self.<input>` with `.shape` and `.ndim`, OR `self.<input>_shape` as a shape tuple/list. Params: `self.<param>`. Unreferenced names need not be exposed. See [docs/design/roofline.md §4.4.3](../../docs/design/roofline.md).
- Dynamo-traced `forward` MUST NOT construct a `Kernel` or enter a TileLang builder; call-time kernel resolution goes through the compile dispatch boundary. See [ops-design.md](../../docs/design/ops-design.md#compile-dispatch-boundary).

16
.claude/domain-rules/testing-budget.md
View File

@ -1,16 +0,0 @@
## Boundary
- **OWNS**: `tests/`, `workloads/` (test stage creates workload definitions first)
- **MUST NOT WRITE**: `tileops/ops/`, `tileops/kernels/`, `benchmarks/`, `tileops/manifest/`
→ [trust-model.md §Test](../../docs/design/trust-model.md#test) | [testing.md §Tests](../../docs/design/testing.md#tests)
______________________________________________________________________
- Every test case traces to a specific code path, dtype dispatch, or regression. No cases for combinatorial confidence.
- Test all supported dtypes. Don't cross dtype and shape coverage unless the combination triggers a distinct code path.
- Don't generate fixtures from `tileops/manifest/` workloads. Test parameters are a curated correctness subset.
- Before committing: drop scaffolding tests that guarded intermediate implementation steps and don't guard any final code path.
- Run `scripts/test_node_delta.py` on PRs touching test files. Growth on existing files → include the script output + a one-line justification in the PR body. New test files only → no delta report.
- Binary-op tests cover broadcast semantics: bias-add `(B,S,D)+(1,1,D)`, row `(B,S,D)+(B,S,1)`, scalar `(M,N)+(1,1)`. Applies to arithmetic, comparison, logical, bitwise.
- Skill development tests stay local — never commit anything under `.claude/skills/`.

35
.claude/review-checklists/approval-gate.md
View File

@ -1,35 +0,0 @@
Run before approving any PR. Apply each item if its scope matches the diff. If any applicable check fails, REQUEST_CHANGES until the developer pushes a triage commit.
## Tests
- [ ] **Per-case verdict.** Triage every added/modified test case as `keep` / `shrink` / `delete`. Inline only on blockers; clean test PRs stay clean (`criteria.md §3`).
| Verdict | When | Inline? |
| ------------------------------ | ---------------------------------------------------------------------------- | ----------- |
| `keep — guards <path/dtype>` | distinct code path or dtype (per `docs/design/testing.md §Test case policy`) | no |
| `shrink — fold to <axis>` | Cartesian expansion; fold to "boundary + one representative interior point" | **blocker** |
| `delete — duplicate of <node>` | same-failure-mode duplicate of a kept case | **blocker** |
Cases the reviewer cannot classify with confidence count as untriaged → **blocker inline** asking the developer for rationale. Every blocker must be resolved (shrink / delete, or downgrade to `keep` with rationale) before APPROVE.
- [ ] **Numerical floor.** Run `python scripts/test_node_delta.py --base upstream/main` (prereq: `upstream` points to tile-ai/TileOPs and is fresh — `git fetch upstream` first). REQUEST_CHANGES with the full node-ID list if existing-file growth > 25% AND any case carries an unresolved `shrink`/`delete` blocker or is untriaged. Absence of an inline is not itself a blocker — silent `keep` is the default.
- [ ] **Critical-path floor.** Never remove the last test on an output-distinguishing input: tile boundary, vectorization alignment, degenerate dimension (size = 1), or a dispatch branch carrying observable behavior. Tests with no output-distinguishing input are removable.
- [ ] **No AC defense.** Reject "AC-N required this matrix" — AC text does not bind the merged suite.
## Authoring discipline
- [ ] **PR body.** Conforms to `.foundry/mold/pr-body-template.md`; records final state only — what the PR does (Summary, scoped to the merged diff) + verification facts (test plan, pre-commit, structural readiness, test node delta). Strip dev-process narration: per-round fix history, tally IDs (`T001–T0NN`), "Driven by review iteration", reviewer-by-reviewer changelogs, abandoned approaches. Those belong in commit history / review threads. REQUEST_CHANGES if found.
- [ ] **Replies.** Outcome only — `Done in <sha>.`, `Won't-fix: <one-line reason>.` No commit-by-commit narration, "what I tried", thread/tally IDs, root-cause essays, or design restatements. Process detail ages out and clutters the thread. REQUEST_CHANGES asking the developer to edit the comment to a one-liner if found.
## Review process
- [ ] **Batch-once.** If only cleanup-class issues (keep / shrink / delete / rename / dedupe) remain with no correctness blockers, surface every such item from the full diff in this single pass. Don't defer — either include now or demote to advisory (no longer gates APPROVE).
- [ ] **Re-run on triage.** Re-run every applicable check above on the developer's triage commit before approving.
## Scope-specific
- [ ] **Skill edits (`.claude/skills/**`).** Require a tightening pass before APPROVE — condense wording without changing what the skill instructs. Verify: semantics preserved, every step has exactly one valid execution path, no example included unless load-bearing, retained examples reference durable concepts rather than implementation details that age out.

17
.claude/review-checklists/benchmark.md
View File

@ -1,17 +0,0 @@
For `[Bench]` PRs. Benchmarks live downstream of every other layer and must never silently substitute for tests or for the manifest.
Load `.claude/domain-rules/benchmark.md` before reviewing.
Two non-negotiable principles cut across every event:
- **Independent baselines.** Every benchmark must record at least one non-`"tileops"` baseline (PyTorch, vendor, third-party kernel). A benchmark that only times TileOps against TileOps is a self-comparison, not a benchmark — reject it.
- **No correctness assertions.** Benchmarks measure performance; they do not gate behavior. `assert torch.allclose(...)` and equivalents belong in `tests/`. A correctness check inside `benchmarks/` is a trust-layer violation regardless of how convenient it is.
#### Checklist
- [ ] **Boundary respected.** Diff stays inside `benchmarks/`. No edits to `tileops/ops/`, `tileops/kernels/`, `tests/`, `workloads/`, or `tileops/manifest/` (`.claude/domain-rules/benchmark.md §Boundary`).
- [ ] **Independent baseline present.** Each new/edited benchmark records ≥1 non-`"tileops"` baseline. If the external baseline is conditional, a local torch fallback is registered.
- [ ] **No correctness gating.** No `assert`, `torch.allclose`, or equivalent inside `benchmarks/`. Numeric mismatches surface through report columns, not exceptions.
- [ ] **Realistic shapes.** Shape constants reflect real DNN workloads (LLaMA-family or equivalent), annotated with the model/scenario they represent. Arbitrary flat numbers (e.g., 262K, 1M, 4M) are rejected.
- [ ] **Tag and FLOPs/memory hygiene.** Tags are lowercase hyphen-separated; `"tileops"`-prefixed tags are TileOps entries, all others are baselines. `calculate_flops()` / `calculate_memory()` return numeric or `None` consistently.
- [ ] **`record()` arg style consistent.** Within one file, `BenchmarkReport.record()` uses Op object (preferred) or string name uniformly — no mid-file flips.

19
.claude/review-checklists/doc.md
View File

@ -1,19 +0,0 @@
For `[Doc]` and `[Design]`.
A flag in this domain names one of: a contradiction (cite file:line on both sides), a missing follow-up reference, a scope violation (quote the offending line), or a re-introduced removal (cite the prior commit).
#### Design docs (`docs/design/*.md`)
TileOps is a design-first project. Design docs guide agent development; tight scope keeps agents on top-level decisions instead of mechanics. Also load `.claude/domain-rules/design-docs.md`.
- [ ] **Top-level only.** Content is a target convention, module boundary, or contract. Reject added codebase mechanics, file enumerations, or implementation snapshots.
- [ ] **No contradictions.** Cross-check against neighboring design docs and `tileops/manifest/`. Flag conflicting MUST / SHOULD with file:line on both sides.
- [ ] **Concise.** Flag content that does not constrain the decision — history narration, illustrative examples, rationale that doesn't explain a choice.
- [ ] **Drift-free.** If the doc states a target the code or manifest doesn't satisfy, the PR includes the change or links a follow-up issue.
#### Other docs
Covers READMEs, `CLAUDE.md` family, agent-facing skill docs, and source comments. Lighter bar — only consistency and drift matter.
- [ ] **No contradictions** with current code, manifest, or design docs. Cite file:line on both sides.
- [ ] **Drift-free.** Implied code or manifest change is included in the PR or linked as a follow-up issue.

13
.claude/review-checklists/feature.md
View File

@ -1,13 +0,0 @@
For `[Feat]` and `[Enhancement]`. For op/kernel PRs the structural axis is governed by `docs/design/ops-design.md` and `docs/design/ops-design-reference.md`; this checklist covers cross-cutting axes only.
#### Checklist
- [ ] Op/kernel public interface matches its `tileops/manifest/` entry. No entry → wrong PR order
- [ ] Manifest edits, if any, stay within the carve-out whitelist (`.claude/rules/manifest-trust-model.md`)
- [ ] If feature is an op, spot-check the diff against at least one of the structural checks below; flag any divergence (full rules in `docs/design/ops-design-reference.md`):
- Class name ≡ manifest entry key
- `__init__` kwargs derivable from manifest (`static_dims` / `dtype` / `signature.params`)
- `default_kernel_map` matches `source.kernel_map` verbatim
- `forward` validation order: `_validate_dtypes` → shape rules → `static_dims` commitment → kernel call
- `_validate_dtypes` matches manifest `dtype` / `dtype_combos`
- `eval_roofline` is a plain-Python body (no class-level expression strings, no `ast.parse`)

40
.claude/review-checklists/manifest.md
View File

@ -1,40 +0,0 @@
For `[Maintain]`, `[Refactor][Manifest]`, and any PR that flips a manifest entry's `status`. The manifest is the authoritative spec for op interfaces — every check below exists to keep that authority real.
Load `.claude/domain-rules/manifest-spec.md` before reviewing.
Two non-negotiable principles cut across every event:
- **Reference semantic alignment.** Any change to an op's spec (signature, shape rules, dtype combos, roofline vars) must map back to an authoritative reference — PyTorch public API for `torch.nn.*` / `torch.nn.functional.*` names; the paper or vendor docs otherwise. Reverse-engineering from current TileOps code is forbidden — spec is upstream of code.
- **`status` field truthfulness.** `status: implemented` is a hard claim that code conforms to the entry. The reviewer's job is to *disprove* it, not to take it on faith. Status flips that don't reflect actual conformance corrupt the trust model.
#### Add-manifest (new entry)
PRs from the `add-manifest` skill, or any PR that adds a previously-absent op entry.
- [ ] **Reference cited and authoritative.** Entry's `source` field points to PyTorch / paper / vendor docs. Reviewer can derive shape and dtype rules from that link alone.
- [ ] **Signature matches reference exactly.** For `torch.nn.*` / `torch.nn.functional.*` names: parameter names, order, and defaults all match PyTorch's public API. No invented parameters.
- [ ] **Required fields present.** `signature`, `shape_rules`, `dtype_combos`, `roofline`; `kernel_map` and `static_dims` where the family requires them.
- [ ] **Lands as `spec-only`.** New entries never land as `implemented`, regardless of any existing code claiming to be ready.
- [ ] **Validator green.** `scripts/validate_manifest.py` passes with no checks disabled.
- [ ] **No code change.** Diff does not modify `tileops/ops/` or `tileops/kernels/`.
#### Fix-manifest (patch existing entry)
PRs from the `fix-manifest` skill — patches one missing structural field (`kernel_map`, `static_dims`) on an existing entry.
- [ ] **Scope is one structural field.** Diff modifies exactly one missing field. Edits to `signature`, `shape_rules`, `dtype_combos`, or `roofline` belong to `add-manifest`, not `fix-manifest` — reject and split.
- [ ] **Reference still aligns.** Other reference-derivable fields (`signature`, `shape_rules`, `dtype_combos`) on the same entry have not silently drifted from the source URL. Spot-check at least one.
- [ ] **Validator green.**
- [ ] **No code change.** Diff does not modify `tileops/ops/` or `tileops/kernels/`.
#### Status flip (`spec-only` ↔ `implemented`)
Often bundled with a `[Refactor][Ops]` op-migration PR.
- [ ] **Conformance verified, not asserted.** Reviewer diffs the op's `__init__` and `forward` against the manifest entry field-by-field — names, types, defaults, shapes.
- [ ] **Spec tests actually run.** No `pytest.skip`, `xfail`, or weakened assertion left from the `spec-only` era. Grep the test file before approving.
- [ ] **`FIXME(staged-rollout)` markers tied to this op removed.**
- [ ] **Flip back to `spec-only`** is legitimate only when implementation is removed or known-broken. Challenge any other rationale.
- [ ] **Pure flip.** This PR changes `status` only — no rewrite of `signature` / `shape_rules` / `roofline`. If the entry needs spec edits to match implementation, that is reverse-engineering from code; reject and require a separate `add-manifest`- or `fix-manifest`-style PR.
- [ ] **`source.kernel_map` present.** `status: implemented` requires the Op→Kernel dispatch map. The validator currently emits a *warning* for missing `kernel_map`; the human reviewer escalates that to a blocker on a flipped entry.
- [ ] **`workloads` non-empty and tensor-input shapes complete.** `status: implemented` requires at least 2 workloads (`test_every_op_has_at_least_two_workloads`). Each row must declare a shape for *every* tensor input in `signature.inputs` — `input_shape` alone is not enough when the op has additional tensor inputs (e.g. `mask_shape`, `value_shape: []`, `min_shape`, `max_shape`). Empty `workloads: []` or a row missing any required tensor-input shape on a flipped entry is a blocker — copy from the sibling variant or derive from the op's typical usage shapes.

32
.claude/review-checklists/pre-review.md
View File

@ -1,32 +0,0 @@
Common rules for every checklist in this folder. Load before any title-specific checklist.
## Load order
1. `docs/design/trust-model.md` — stage contracts (`manifest → test → implementation → benchmark`).
1. The title-specific checklist.
## Review lens
Apply trust-model rules as a review lens: surface cross-layer diffs as comments with a concrete citation; the author replies with rationale; the reviewer judges on content. Provenance labels (`automated` / `needs-review` / `nightshift`) record origin; review criteria apply uniformly.
## Cross-layer review criteria
| Criterion | Pass | Fail |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| Oracle origin | `ref_program` resolves to PyTorch / NumPy / closed-form / IEEE-754 | Agent-fabricated literal as expected value |
| Coverage set | Unchanged or strictly expanded; deletions target code paths the same diff removes | dtype / shape / sign cells removed with the code path still present |
| New-path coverage | A test reaches each diff-added code path on an input that makes its output differ from the alternative. Aliases (paths with no output-distinguishing input) are exempt. | A diff-added path's output-distinguishing input has no test |
Cite the failing criterion by name.
## Comment quality
| Rule | Required form |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| Concrete pointer | `file:line`, entry path, field name, parametrize axis, ref URL, test name, or the offending diff line |
| Decidable claim | A concrete claim about that pointer. Hedged forms ("looks reasonable", "may want to verify", "consider revising", "could be clearer") fail this rule |
| Suggestion scope | Suggestions stay within fixes for flagged items |
## Scope
Checklists are the floor. Add PR-specific checks when the diff warrants them.

12
.claude/review-checklists/refactor.md
View File

@ -1,12 +0,0 @@
For `[Refactor]`.
#### Checklist
- [ ] Any `xfail` / `skip` / loosened assertion has a `FIXME(staged-rollout)` block per `.claude/rules/code-style.md`
- [ ] If the op has a manifest entry, signature still matches it
- [ ] **Trust-layer scope.** Refactor stays within one trust layer per `docs/design/trust-model.md` (manifest / test (incl. `workloads/`) / implementation / benchmark). Cross-layer touches are judged by `pre-review.md` criteria (oracle origin, coverage set, new-path coverage).
#### Sub-types
- `[Refactor][Manifest]` → load `manifest.md` instead. `tileops/ops/` and `tileops/kernels/` must NOT change in the same PR.
- `[Refactor][Ops]` / `[Refactor][<Family>]` that flips `status` → also load `manifest.md`.

12
.claude/review-checklists/testing.md
View File

@ -1,12 +0,0 @@
For PRs that add or modify files under `tests/`. Single focus: justify the test node growth declared in the PR body's `## Test node delta` section. Without this guard, parametrize stacks expand into performance sweeps and dilute every test run.
Load `.claude/domain-rules/testing-budget.md` and `docs/design/testing.md §Test case policy` before reviewing.
**Burden of proof flips above threshold.** If growth crosses the threshold below, the author justifies; reviewer silence is not approval.
#### Checklist
- [ ] **Trigger.** Compute `delta_pct = (HEAD − Base) / Base × 100` from the PR body's `## Test node delta`. If `delta_pct > 10%`, every check below is required; otherwise this file is informational.
- [ ] **Per-case purpose stated.** Each new case (or each new parametrize cell) serves exactly one of: dtype correctness / kernel-branch shape coverage / feature coverage / regression — per `docs/design/testing.md §Test case policy`. The PR body justification names which, with file:line.
- [ ] **No Cartesian-product expansion.** Reject parametrize stacks whose growth is the product of two or more axes' cardinalities without a per-cell rationale. Crossing axes is allowed only when each cell maps to a distinct code path the author can name; otherwise the stack is a performance sweep, not a UT.
- [ ] **Test layer hygiene.** Tests must not import from `benchmarks/` per `docs/design/trust-model.md`. Cross-layer touches are judged by `pre-review.md` criteria (oracle origin, coverage set, new-path coverage).

39
.claude/rules/code-style.md
View File

@ -1,39 +0,0 @@
- Every `tileops/kernels/*` subpackage MUST have an `__init__.py` with explicit `__all__` and `from .module import Symbol` re-exports.
- Intra-package imports: relative (`from .op import Op`). Cross-package: absolute (`tileops.foo.bar`).
- No file-level lint suppressions (`# ruff: noqa`, `# flake8: noqa`). Use targeted inline `# noqa: XXXX` only.
- TIR parameter type: `T.Tensor(shape, dtype)`, never the deprecated `T.Buffer`.
- Reinterpret cast: `T.reinterpret(value, dtype)` (value first), never the deprecated dtype-first form.
- Each TileLang kernel is one `@T.prim_func` whose body opens `with T.Kernel(...)`; sub-routines use `@T.macro`, never nested `prim_func`.
- No narrow-type literal casts (`T.cast(1.0, "float16")`). Reference `x.dtype`, or compute in a wider intermediate and cast at the boundary.
- Promote overflow-prone fp16/bf16 math (cubic, division, `exp`, softmax accumulators) to fp32; cast back to storage dtype at the boundary.
- Decorate each `_<op>_kernel` builder (the `@tilelang.jit`-wrapping `Callable`) with `@functools.lru_cache(maxsize=<N>)`; every parameter must be hashable. Default `maxsize=32`; use `64` only when the distinct-config working set demands it; `maxsize=None` only for intrinsically bounded config spaces. Document any non-default choice at the call site.
- Tag tests degraded by a process constraint (e.g. trust model splitting manifest and code PRs) with `FIXME(staged-rollout)`. Cleanup line names the invariant to restore — never a PR number. Scan: `grep -rn 'FIXME(staged-rollout)'`.
```python
# FIXME(staged-rollout): <one-line summary of what's degraded>
#
# Broken invariant: <what contract is currently violated>
# Why: <which process constraint requires this temporary state>
# Cleanup: <concrete condition that triggers removal of this marker>
```
- PascalCase abbreviations stay fully uppercase: `RMSNormKernel`, `SSDDecodeOp`, `FusedAddRMSNormFwdOp`.
- Filenames: all-lowercase with underscores. Multi-letter abbreviations stay lowercase (`rms_norm.py`, `ssd_decode.py`); never capitalize a single letter. Never contract norm names (`rms_norm`, not `rmsnorm`).
- Docstrings: Google style. One-line summary, blank line, then optional `Args:` / `Returns:` / `Raises:` / `Example:`. Internal helpers may use a single-line summary. Never mix Sphinx (`:param:`) or NumPy headers in one file.
- Expand domain abbreviations on first use in a docstring: `State Space Model (SSM)`, `State-Space Dual (SSD)`. Later uses may abbreviate.
- Shipped source (code, docstrings, manifest YAML) must not reference issue/PR numbers, AC labels, round numbers, reviewer names, or `Follow-up: #N`. See [domain-rules/manifest-spec.md](../domain-rules/manifest-spec.md).
Discovery scan: `grep -rnE '(^|[^[:alnum:]])#[0-9]{3,}|AC-[0-9]+|round-[0-9]+ review|[Ff]ollow-up:[[:space:]]*#' --exclude-dir=manifest tileops/ tests/ benchmarks/ scripts/`

18
.claude/rules/manifest-trust-model.md
View File

@ -1,18 +0,0 @@
- For the OWNS / boundary chart of the manifest stage, see [docs/design/trust-model.md](../../docs/design/trust-model.md) §Manifest.
- When the op name matches a PyTorch op (`torch.nn.*`, `torch.nn.functional.*`), the manifest signature must match PyTorch's public API. Do not invent parameters.
- Implementation does not conform to spec → set `status: spec-only`, fix code in follow-up PR. Never modify manifest to match code.
- Do not remove `roofline.vars`, `shape_rules`, or `params` to silence validator errors.
## Status flip carve-out
An implementation PR may edit the aligned op's manifest entry only at:
- `status` (any direction).
- `source.kernel_map` entries.
- `source.test` and `source.bench` path values. Discoverability pointers, not contractual fields — may be retargeted at the per-op test/bench file authored by the same PR.
- `workloads` — **only** when the same PR flips `status: spec-only → implemented` on that op (promotion forces non-empty workloads to satisfy `test_every_op_has_at_least_two_workloads`).
- `torch_compile_fullgraph` — **only** together with its registered compile-test evidence.
Every other field — `family`, `ref_api`, `signature`, `roofline.*`, `params`, `output-dtype`, `shape_rules`, `source.kernel`, `source.op`, `source.bench_manifest_driven`, and any other `workloads` edit — needs a separate manifest-only PR with human review.
The carve-out narrows the prohibition; it does not relax the trust boundary.

6
.claude/rules/security.md
View File

@ -1,6 +0,0 @@
- **Never** hardcode secrets, tokens, passwords, or API keys in any file.
- **Never** print or echo secrets in GitHub workflow steps (e.g. `echo ${{ secrets.XXX }}`). Use environment variables and let GitHub mask them automatically.
- **Always** review GitHub workflow changes for secret exposure before committing: check `env:` blocks, `run:` scripts, and `with:` parameters.
- **Never** commit `.env`, `.pem`, `.key`, `credentials.json`, or similar sensitive files. These are in `.gitignore`.
- When creating or modifying workflows, use `${{ secrets.GITHUB_TOKEN }}` via `env:` — never inline in shell commands.
- Gitleaks runs as a pre-commit hook to catch leaked secrets. Do not bypass it (`SKIP=gitleaks` is not allowed without explicit user approval).

5
.claude/settings.json
View File

@ -1,5 +0,0 @@
{
"enabledPlugins": {
"superpowers@claude-plugins-official": true
}
}

136
.claude/skills/add-manifest/SKILL.md
View File

@ -1,136 +0,0 @@
---
name: add-manifest
description: Generate or re-align one `tileops/manifest/` entry from a reference-API docs URL. Caller provides the manifest key (`op_name`); skill writes that one entry. Idempotent.
---
## Arguments
| Argument | Required | Description |
| --------- | -------- | ----------------------------------------------------------------------------------------------------------------------- |
| `op_name` | Yes | Manifest key (e.g., `RMSNormFwdOp`). Caller-supplied, never derived. For variants, caller invokes once per emitted key. |
| `ref_url` | Yes | HTTPS docs URL for the Tensor op. Must match `^https://[A-Za-z0-9./_-]+\.html$`. |
## Contract
**One entry per invocation.** No splitting, no variant orchestration. For primary + variant, caller invokes the skill twice with different `op_name`s.
**Idempotent.** Auto-derivable fields are rewritten from the reference; human-curated fields are preserved if the entry exists, defaulted otherwise.
| Auto-derivable (always rewritten from reference) | Human-curated (preserved if entry exists, else default) |
| ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
| `signature.{inputs,outputs,params}` | `family` (default: from sibling-entry copy or BLOCKED) |
| `signature.shape_rules` | `ref_api` (default: derived from `ref_url`'s last path segment) |
| `signature.dtype_combos` | `workloads` (default: `[]`) |
| `roofline.{flops,bytes,vars}` (well-known op) | `source.{kernel,op,test,bench,kernel_map,bench_manifest_driven}` (default: from RESOLVE_SOURCES + `bench_manifest_driven: false`) |
| | `status` (default: `spec-only`) |
| | Adjacent comments (best-effort) |
**Termination**: draft PR created → success. Invalid URL / un-derivable roofline / source-path or family resolution failure → BLOCKED.
**Constraints**: never edit op / kernel / test / bench code. Never invent params outside the reference. Never set `status: implemented` (that is `align-op@FLIP_STATUS`).
**File to edit**: write the entry into `tileops/manifest/<family>.yaml`, where `<family>` is the entry's `family` field. The manifest is split one file per family; do not create new files or move entries between files. Use `ruamel.yaml` for round-trip preservation of comments and key order.
**Caller responsibility**: `op_name` and `ref_url` must point at the same op. The skill does not enforce alignment between them — TileOPs identity may legitimately differ from any reference's naming (e.g., `MultiHeadAttentionFwdOp` ↔ `torch.nn.functional.scaled_dot_product_attention`). Wrong pairing produces a broken manifest entry silently.
## Workflow
```mermaid
stateDiagram-v2
[*] --> VALIDATE_INPUT
VALIDATE_INPUT --> [*]: invalid
VALIDATE_INPUT --> READ_EXISTING
READ_EXISTING --> READ_REFERENCE: snapshot saved (entry present)
READ_EXISTING --> RESOLVE_SOURCES: entry absent
RESOLVE_SOURCES --> READ_REFERENCE
RESOLVE_SOURCES --> [*]: source / family resolution failed → BLOCKED
READ_REFERENCE --> DRAFT_ENTRY
DRAFT_ENTRY --> VALIDATE
VALIDATE --> DRAFT_ENTRY: L0 fail
VALIDATE --> RUN_AUDIT: L0 pass
RUN_AUDIT --> CREATE_ISSUE
CREATE_ISSUE --> CREATE_PR
CREATE_PR --> [*]
```
## Steps
### 1. VALIDATE_INPUT
Reject `ref_url` not matching the regex. Reject `op_name` not matching `^[A-Z][A-Za-z0-9]+(Fwd|Bwd)Op$`.
### 2. READ_EXISTING
Look up `op_name` in `tileops/manifest/`.
- **Present** → snapshot the human-curated fields per the Contract table. Source paths come from the existing `source.*`. Proceed to READ_REFERENCE.
- **Absent** → greenfield. Proceed to RESOLVE_SOURCES.
### 3. RESOLVE_SOURCES (greenfield only)
Lookup is **class-based**, not filename-based — many TileOPs ops share a file (e.g., `SumFwdOp` and `MeanFwdOp` both in `tileops/ops/reduction/reduce.py`).
1. **`source.op`**: scan `tileops/ops/**/*.py` for `class <op_name>(...)` (AST or `grep -rlE "^class <op_name>\(" tileops/ops/`).
- Exactly one match → that file path.
- Zero matches → true greenfield. Default to `tileops/ops/<snake_name>.py` (use a family subdirectory if a sibling-family entry suggests one). `<snake_name>` = `op_name` minus trailing `FwdOp` / `BwdOp`, snake_cased (`RMSNormFwdOp` → `rms_norm`). File may not exist yet; caller scaffolds afterward.
- Multiple → BLOCKED disambiguation.
1. **`source.kernel`** (required by L0; `fix-manifest` cannot fill it later):
- If `source.op` was found by class lookup: read its imports for a `Kernel` subclass; apply class-lookup under `tileops/kernels/**/*.py`. One match → that file. Multiple → BLOCKED disambiguation.
- No kernel import (kernel-less op) → `source.kernel = source.op`.
- Otherwise → BLOCKED `evidence_needed: source.kernel for <op_name>`.
1. `source.test = tests/ops/test_<snake_name>.py`; `source.bench = benchmarks/ops/bench_<snake_name>.py`. Missing files: record absent.
1. **`family`** (required by L0; cannot be empty):
- Copy from a sibling manifest entry whose `source.op` parent-dir or basename overlaps.
- No matching sibling → BLOCKED `evidence_needed: family for <op_name>`. Never invent.
### 4. READ_REFERENCE
`WebFetch(ref_url)`. Sole source of truth.
| Reference param kind | Goes to |
| -------------------- | -------------------------------------- |
| Tensor | `signature.inputs` (positional order) |
| non-Tensor | `signature.params` (`type`, `default`) |
| return | `signature.outputs` |
Names match the reference verbatim. Include every reference param even if the kernel ignores it. Exclude `float64` and `complex32/64/128` (TileOPs is GPU-only).
For references with `Optional[Tensor]` inputs, the caller has decided which slice corresponds to `op_name` (primary = required only; variant = primary + chosen optional). The skill emits inputs accordingly.
### 5. DRAFT_ENTRY
Snapshot present (re-align) → preserve human-curated fields verbatim. Snapshot absent (greenfield) → use Contract defaults. Auto-derivable fields:
- `signature.inputs`: ordered dict in the reference's positional order. Per input: `dtype` = supported set joined with `|` (reference dtypes minus `float64` and complex types); `shape` only if fixed rank; `layout` only if non-default; `constraints` if applicable.
- `signature.outputs`: same shape as inputs. Use `same_as(<ref>)` where applicable.
- `signature.params`: ordered dict, each `{type, default}`.
- `signature.shape_rules`: Python expressions for derived dims and inter-tensor constraints.
- `signature.dtype_combos`: only if supported set ⊂ Cartesian product; else omit.
- `roofline`: required by L0. Well-known op (conv / pool / matmul / norm / reduction): standard formula. Fixed-rank: shape names auto-bind, use `elem_bytes`. Arbitrary-rank: `vars` mapping. Not derivable → BLOCKED `evidence_needed: roofline.flops|bytes for <op_name>`.
### 6. VALIDATE
```bash
python scripts/validate_manifest.py --check-op <op_name>
```
L0 must pass. On fail: edit entry, rerun. L1–L4 failures go to the follow-up issue, not blocking.
### 7. RUN_AUDIT
Invoke `audit-family` for the op's family → `.foundry/migrations/<family>.json`.
### 8. CREATE_ISSUE
Invoke `foundry:creating-issue`. Per `semantic_gap` op the body MUST contain: kernel feasibility (cite kernel code; classify each missing param `trivial` / `kernel-change` / `blocked`); class-structure impact; effort per gap item; family dependencies. MUST also list outstanding human decisions (`workloads`, `roofline`) and resolution path. MUST NOT duplicate validator-reported facts. Record the issue URL.
### 9. CREATE_PR
Invoke `foundry:creating-pull-request` (draft):
| Snapshot at READ_EXISTING | Title | Branch |
| ------------------------- | ----------------------------------------------------------- | ---------------------------------------- |
| absent | `[Maintain][Manifest] Add <op_name>` | `maintain/manifest/<op-slug>` |
| present | `[Refactor][Manifest] Re-align <op_name> spec to <ref_api>` | `refactor/manifest/regenerate-<op-slug>` |
Body: which fields were rewritten vs. preserved, validator results, `Related: #<issue from step 8>`. Title and branch must match `.claude/conventions/types.sh`.

156
.claude/skills/align-family/SKILL.md
View File

@ -1,156 +0,0 @@
---
name: align-family
description: Drive the full migration for an op family — audit, delegate per-op alignment to align-op, run cross-op cleanup. Two terminal outcomes: SUCCESS opens a PR; CLEANUP_REGRESSION exits blocked without a PR when post-cleanup tests fail.
---
## Arguments
Family name from `tileops/manifest/` (e.g., `reduction`, `norm`, `attention`).
## Contract
- **Input**: `family` name
- **Output** (two terminal outcomes):
- **SUCCESS**: PR URL + final report — all ops processed via `align-op` (promoted or blocked), cleanup succeeds, PR opens.
- **BLOCKED**: blocked report (no PR) — reached when CLEANUP detects a regression in a promoted op's tests after dual-path removal; see `CLEANUP_REGRESSION` terminal in the state diagram. Distinct from the non-terminal per-op `REPORT_BLOCKED` state.
- **Termination**: all ops processed (promoted or blocked via `align-op`) and either (a) CLEANUP + CREATE_PR succeed, or (b) a promoted op's tests fail after CLEANUP's dual-path removal, causing the run to exit via `CLEANUP_REGRESSION` with the regression recorded. (`REPORT_BLOCKED` is the non-terminal per-op blocked state and never terminates the run.)
## Trust Model
- `align-family` delegates every per-op stage to `align-op`, invoked as a **separate sub-agent** per op. The family orchestrator never runs any atomic per-op skill directly — those live inside `align-op`'s contract.
- `align-family` does **not** write `tileops/manifest/`. After the refactor, `align-op` is the sole manifest writer (at its own FLIP_STATUS step); `align-family` observes status transitions via `align-op`'s SUCCESS return. No `align-family` stage edits, modifies, or flips the manifest.
- Directly-invoked sub-skills of `align-family` are exactly two: `audit-family` (in AUDIT) and `align-op` (per op).
| Stage | Sub-skill |
| -------- | -------------- |
| AUDIT | `audit-family` |
| ALIGN_OP | `align-op` |
## Workflow
```mermaid
stateDiagram-v2
[*] --> AUDIT
AUDIT --> GROUP_BY_BASE: gap report generated
GROUP_BY_BASE --> ROUTE: ops grouped by base class
ROUTE --> ALIGN_OP: ready (mode=minor) or semantic_gap (mode=redesign)
ROUTE --> REPORT_BLOCKED: classification=blocked (audit-family)
ALIGN_OP --> CLEANUP_GATE: align-op SUCCESS
ALIGN_OP --> REPORT_BLOCKED: align-op BLOCKED
REPORT_BLOCKED --> CLEANUP_GATE: record blocked reason
CLEANUP_GATE --> ROUTE: group incomplete, next op
CLEANUP_GATE --> CLEANUP: all siblings promoted or blocked
CLEANUP --> ROUTE: legacy path removed, next group
ROUTE --> CREATE_PR: all ops processed
CLEANUP --> CLEANUP_REGRESSION: promoted op test fails after cleanup
CLEANUP_REGRESSION --> [*]: terminal blocked, no PR
CREATE_PR --> [*]
```
## Orchestrator Discipline
### Clean worktree between sub-agents
After each sub-agent returns and before dispatching the next, verify:
```bash
test -z "$(git status --porcelain)"
```
This catches tracked changes, staged changes, AND untracked files. If not clean: the sub-agent's commit failed (pre-commit hook, staging issue) or left new files uncommitted. Orchestrator commits on behalf, then proceeds. Every agent must start with a clean worktree.
### Dual-path is acceptable during migration
When `align-op` rewrites a base class during its per-op pipeline, it may create a dual-path `__init__` (legacy + spec) to keep unmigrated sibling tests passing. This is correct temporary debt — the cleanup gate removes it.
**Dual-path definition**: a class `__init__` with runtime branching to support two incompatible construction interfaces, and `forward` dispatching to two execution paths. Not polymorphism — same semantics, temporary interface coexistence.
## Steps
### 1. AUDIT
```
/audit-family <family>
```
Gap report written to `.foundry/migrations/<family>.json`.
### 2. GROUP_BY_BASE
Group ops by `base_class` from the gap report. Each group is a set of sibling ops sharing a base class. Process groups in order; within each group, process ops in order (first op likely fixes the base class, subsequent ops validate).
`base_class` is a required field in the gap report. audit-family must populate it for every op entry. If an op inherits `Op` directly (no intermediate base class), its `base_class` is `"Op"` — these ops form a single group but are independent (no shared base class to rewrite, so cleanup gate is a no-op for this group).
Track group completion: a group is complete when all its ops are `promoted` or `blocked`.
### 3. ROUTE
Read the gap report from AUDIT. For each op in the current group, route by `classification` (the field `audit-family` populates):
| Gap-report `classification` | Action | Why |
| --------------------------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ready` | → ALIGN_OP with `--mode=minor` | Existing code already conforms; align-op's minor path runs the spec tests (DONE_SKIP since they pass), then the shared downstream + flip. |
| `semantic_gap` | → ALIGN_OP with `--mode=redesign` | Family-scoped historical migration treats every `semantic_gap` op as a structural redesign by default — legacy code is being replaced wholesale. For per-op `semantic_gap` cases that are actually minor manifest deltas, use single-op `align-op <op> --mode=minor` directly instead of `align-family`. A future `recommended_mode` gap-report field could refine the routing automatically. |
| `blocked` | → REPORT_BLOCKED with audit's `reason`. Skip align-op. | Audit determined the op cannot be migrated autonomously (no `pytorch_equivalent`, kernel-layer change required, etc.). No per-op work to do. |
The mapping is deterministic — `align-family` MUST pass `--mode=` explicitly so `align-op` never falls into its interactive prompt branch in a batch family migration.
### 4. ALIGN_OP (per op)
For each op routed here, invoke `align-op` as a **separate sub-agent** with the mode determined in Step 3:
```
align-op <op_name> --mode=<minor|redesign>
```
`align-op` owns the entire per-op pipeline internally — its internal stages (classify, dispatch on case, test / implement / bench, revalidate, flip status, cleanup, report) are `align-op`'s contract, not `align-family`'s. See [`align-op/SKILL.md`](../align-op/SKILL.md) for the authoritative stage list and the conditional-IMPLEMENT rule. `align-family` does not manage or observe `align-op`'s internal stages; the only interface between them is `align-op`'s SUCCESS / BLOCKED return.
Per-op outcome:
- `align-op` returns SUCCESS → op is `promoted` (manifest status already flipped by `align-op`'s FLIP_STATUS). Record the returned report. Proceed to CLEANUP_GATE.
- `align-op` returns BLOCKED → op is `blocked`. Capture `align-op`'s BLOCKED reason verbatim as the per-op result. Proceed to CLEANUP_GATE.
`align-family` MUST NOT re-flip manifest status or re-run per-op validation on its own; `align-op`'s SUCCESS return is the single source of truth that the manifest was flipped.
The flip performed by `align-op` MUST stay within the [Status flip carve-out](../../rules/manifest-trust-model.md#status-flip-carve-out); `align-family` MUST NOT batch contractual-field edits across ops in the same PR.
### 5. CLEANUP_GATE
After each per-op outcome — whether `align-op` returned SUCCESS / BLOCKED or ROUTE recorded an audit-classified `blocked` op directly to `REPORT_BLOCKED` — check group completion:
- All siblings in the current base-class group are `promoted` or `blocked`? → trigger CLEANUP
- Otherwise → continue to next op (ROUTE → ALIGN_OP)
### 6. CLEANUP
Remove dual-path legacy code from the base class. This step fires once per base-class group, after all siblings have gone through `align-op` (SUCCESS or BLOCKED).
Actions:
1. Remove legacy `__init__` branch (`if M is not None and N is not None` path)
1. Remove `_legacy` flag and `_forward_legacy` method
1. Remove `M`, `N` keyword-only parameters from `__init__`
1. Run tests and `--check-op` for **promoted ops only** (blocked ops' tests may legitimately fail)
1. Commit cleanup changes
If any promoted op's test fails after cleanup → transition to `CLEANUP_REGRESSION` (terminal): record the regression, skip `CREATE_PR`, exit with `blocked` status. Do not proceed with a broken state.
**Two distinct blocked states**, split so the diagram has one meaning per state:
- **`REPORT_BLOCKED`** — per-op `align-op` returned BLOCKED. Records the reason, returns to `CLEANUP_GATE`, continues with sibling ops. Non-terminal.
- **`CLEANUP_REGRESSION`** — a promoted op's tests fail after CLEANUP's dual-path removal. Terminal: the family migration exits without opening a PR. The blocked report becomes the run's final artefact.
**Timeout policy for blocked ops**: if a group has blocked ops that prevent the cleanup gate from firing for an extended period, the orchestrator may force cleanup — remove legacy path and mark blocked ops' tests as `xfail`. This is a human decision, not automatic.
### 7. CREATE_PR
After all ops processed:
- Collect all per-op reports returned by `align-op`
- Create PR with:
- Migration summary (promoted / blocked counts)
- Per-op change table (derived from `align-op` reports)
- Observations surfaced by `align-op` (e.g., `needs_kernel_work`, `needs_human_decision`) for human doc review
- Blocked ops with `align-op`'s BLOCKED reasons

327
.claude/skills/align-op/SKILL.md
View File

@ -1,327 +0,0 @@
---
name: align-op
description: Per-op orchestrator that brings a single op into alignment with its manifest entry. Classifies the op into one of three cases (green field / interface redesign / minor delta), dispatches to the right path (scaffold-op for new, archive+rescaffold+port for redesign, implement-op for minor), then runs the shared downstream (test → bench → validate → flip status → report). Complements the family-scoped `align-family`; per-op entry when you know the op you want to touch.
---
## Arguments
- `op_name` (positional) — manifest key, e.g. `CumsumFwdOp`.
- `--mode=green|redesign|minor` (optional) — override the automatic classification. When omitted, `CLASSIFY` decides (auto if unambiguous, otherwise prompt).
- `--classify-only` (optional) — stop after `CLASSIFY`; write `mode.json` and return without executing any case path. Use to ask "which case is this op in?" without side effects.
## Contract
- **Input**: `op_name` must be present in [`tileops/manifest/`](../../../tileops/manifest/) with `status: spec-only` and a non-empty `source.kernel_map` (same preconditions as scaffold-op; see [PRE_CHECK](#pre_check)).
- **Path and data bindings used throughout this skill** (resolved by the orchestrator once at `PRE_CHECK` when the manifest entry is first loaded, then passed into every sub-skill invocation):
- `<source_op>` — manifest `source.op` path (e.g., `tileops/ops/reduction/cumsum.py`).
- `<source_test>` — manifest `source.test` path (e.g., `tests/ops/test_cumulative.py`).
- `<source_bench>` — manifest `source.bench` path.
- `<source_kernel>` — manifest `source.kernel` path (the primary kernel implementation file).
- `<manifest_signature>` — the `signature` sub-tree from the op's manifest entry, passed verbatim to test-op / implement-op.
- `<pytorch_equivalent>` — manifest `ref_api` value (e.g., `"torch.cumsum"`) or `null` if the op has no PyTorch reference. Required by test-op.
- **Output** (SUCCESS path): op file at `source.op` aligned with the manifest; test file `source.test` aligned; `__init__.py` registrations consistent; `status` flipped `spec-only → implemented` (single commit). Side-artefacts in `.foundry/plan/<op_name>/`: `mode.json` (classification), `plan.json` (scaffold-op's §1/§2/§3 when that skill ran), `kernel-check.json` (redesign case only), `pre-rewrite/source.py` (redesign case, removed at CLEANUP on SUCCESS).
- **Termination (success)**: `python scripts/validate_manifest.py --check-op <op_name>` reports no errors + `python -m pytest <source_test> -v` passes + benchmark produces numbers + manifest status flipped.
- **Termination (blocked)**: any sub-skill (scaffold-op / test-op / implement-op / bench-op) returns blocked; or scaffold-op §1 drift; or REVALIDATE fails. Kernel-layer mismatches surfaced by `KERNEL_CHECK` are **informational only** and never cause BLOCKED by themselves — BLOCKED is reached only if a kernel drift propagates into a downstream sub-skill failure (e.g., bench-op runtime error, REVALIDATE regression). Archives are kept for post-mortem.
- **Constraints**:
- Only align-op (and only at FLIP_STATUS) may modify `tileops/manifest/`. Sub-skills never touch the manifest.
- MUST NOT modify kernel code. Kernel-layer work, if needed, is surfaced via `kernel-check.json` as a separate follow-up.
- MUST NOT expand to multi-op scope; that is `align-family`'s role.
## Trust model
- `CLASSIFY`, `DISPATCH`, `FLIP_STATUS`, `CLEANUP`, `REPORT` are orchestrator stages (align-op itself). Every other stage delegates to an atomic skill as a **separate sub-agent invocation**:
| Stage | Sub-skill |
| ------------- | ------------------------------------------------------------ |
| GREEN path | `scaffold-op` |
| REDESIGN path | `scaffold-op` (after ARCHIVE + CLEAR) |
| MINOR path | `implement-op` |
| TEST | `test-op` |
| IMPLEMENT | `implement-op` (green / redesign only; minor already did it) |
| BENCH | `bench-op` |
Separate invocations preserve the per-skill contracts (e.g., scaffold-op's §1 fact-freeze; implement-op's no-test-modification rule).
- After each sub-skill returns, align-op verifies `git status --porcelain` is empty before dispatching the next. If a sub-skill left an uncommitted change, align-op commits on its behalf with `Sub-skill [name]: [summary]` before proceeding.
## Workflow
```mermaid
stateDiagram-v2
[*] --> PRE_CHECK
PRE_CHECK --> CLASSIFY: manifest preconditions pass
PRE_CHECK --> BLOCKED: prereq missing
CLASSIFY --> CLASSIFY_ONLY_EXIT: --classify-only flag
CLASSIFY --> DISPATCH: mode decided (auto or --mode or user-declared)
DISPATCH --> GREEN_PATH: case = green
DISPATCH --> REDESIGN_PATH: case = redesign
DISPATCH --> MINOR_PATH: case = minor
GREEN_PATH --> TEST: scaffold-op succeeded
REDESIGN_PATH --> KERNEL_CHECK: rescaffold + port done
KERNEL_CHECK --> TEST: kernel-check.json written
MINOR_PATH --> TEST: implement-op succeeded (minor-case main stage ran here)
TEST --> IMPLEMENT: tests fail on current code (expected gap to close)
TEST --> BENCH: tests already pass (DONE_SKIP), usually minor path
IMPLEMENT --> BENCH: implementation closes the gap, tests pass
IMPLEMENT --> BLOCKED: gap beyond op-layer (e.g. kernel rewrite required)
BENCH --> REVALIDATE: benchmark produces numbers
REVALIDATE --> FLIP_STATUS: check-op and pytest both pass
REVALIDATE --> BLOCKED: regression
FLIP_STATUS --> CLEANUP: manifest status flipped
CLEANUP --> REPORT: pre-rewrite dropped on redesign, plan artefacts kept
REPORT --> [*]
CLASSIFY_ONLY_EXIT --> [*]
GREEN_PATH --> BLOCKED: scaffold failed (§1 drift or validator error)
REDESIGN_PATH --> BLOCKED: scaffold or port failed
MINOR_PATH --> BLOCKED: implement-op failed
BLOCKED --> [*]: return to caller with reason
```
## Steps
### <a id="pre_check"></a>1. PRE_CHECK
Preconditions identical to `scaffold-op`'s — orchestrator enforces them up front so sub-skills never see ill-formed input:
- `op_name` in `tileops/manifest/` → proceed; otherwise BLOCKED ("op not in manifest").
- `status: spec-only` → proceed; `implemented` → BLOCKED ("already aligned; flip status to spec-only in a manifest PR first if you intend to re-align"); missing/other → BLOCKED.
- `source.kernel_map` declared and non-empty → proceed; missing → BLOCKED with the same guidance scaffold-op uses (add in a prerequisite manifest PR).
- Every value in `source.kernel_map` resolves to an importable symbol → proceed; otherwise BLOCKED ("kernel class not found at expected path" — kernel must exist for op layer to align, regardless of case).
### 2. CLASSIFY
Decide which case applies. Machine-decidable input: does `source.op` exist?
| Input | Auto case | User prompt? |
| -------------------------- | --------- | ----------------------------------------------------------- |
| `source.op` does not exist | `green` | no |
| `source.op` exists | (unknown) | yes — "redesign (rewrite + port) or minor (in-place edit)?" |
`--mode=<case>` overrides prompting only when consistent with file presence. The orchestrator validates this during CLASSIFY and BLOCKs immediately on invalid combinations so sub-skills never see contradictory input:
- `source.op` **missing** → only `green` is valid. `--mode=minor` → BLOCKED ("`source.op` is missing; cannot edit a non-existent op file; use `--mode=green` or omit `--mode`"). `--mode=redesign` → BLOCKED ("`source.op` is missing; no archive source to rewrite; use `--mode=green` or omit `--mode`").
- `source.op` **exists** → `--mode=green` → BLOCKED ("`source.op` already exists; green-field scaffold would silently overwrite; use `--mode=redesign` for rewrite+port or `--mode=minor` for in-place edit").
**First-op bias.** If no op in the same `family` has `status: implemented` and follows the canonical pattern (`docs/design/ops-design.md` § Step 3), set `auto-case = redesign` (skip the prompt). mode.json: `decided_by: "auto"`, `reason: "no canonical-pattern precedent in family <name>"`. User may override with `--mode=minor`.
Write `.foundry/plan/<op_name>/mode.json`:
```json
{
"op_name": "CumsumFwdOp",
"case": "redesign",
"file_present": true,
"kernel_class_importable": true,
"decided_by": "user_prompt",
"reason": "User declared: manifest _static_axes shape rewritten; structural redesign.",
"classified_at": "YYYY-MM-DDTHH:MM:SSZ"
}
```
`decided_by` is one of `auto` / `user_prompt` / `flag_override`. `reason` is free-form text.
If `--classify-only` was passed, terminate here and print the mode.json content. No other side effects.
### 3. DISPATCH — case-specific main stage
Each path produces the aligned op file under `source.op` plus whatever artefacts the sub-skill creates under `.foundry/plan/<op_name>/`.
#### 3a. GREEN path (`case = green`)
```
scaffold-op <op_name>
```
Sub-skill does PRE_CHECK → DRY_RUN (plan.json) → EMIT → REGISTER → VALIDATE → REPORT. align-op waits for SUCCESS or BLOCKED; on BLOCKED, surface the row and terminate.
#### 3b. REDESIGN path (`case = redesign`)
Sequence:
1. **ARCHIVE** — `mkdir -p .foundry/plan/<op_name>/pre-rewrite/`, copy `source.op` there as `source.py` (rename: strip family path, keep basename). The archive is the source of truth for manual porting. It persists until CLEANUP.
1. **CLEAR** — remove `source.op` from the tree; remove the op's `from .<module> import <ClassName>` line and its `__all__` entry from the package `__init__.py`. Commit as `[Chore] align-op: archive <op_name> before rescaffold`.
1. **SCAFFOLD** — `scaffold-op <op_name>`. Target now absent, PRE_CHECK passes, emits the 17 mechanical slots.
1. **PORT** — read `pre-rewrite/source.py` and port op-specific content that the scaffold cannot produce:
- Optional hooks (`_pad_value`, `_validate_dim`, `_pre_kernel`, `_post_kernel`, `_cache_key` override).
- Family-specific protocol variables (`_op_kind`, `_kernel_key`, `_kernel_cls`, etc.) if the op was a T1 thin wrapper.
- Any `forward` body specifics beyond the universal pattern (kernel-specific reshape/movedim choreography).
- Any class-level non-slot attributes the old file had that still make sense under the new spec.
Commit as `[Feat] align-op: port business logic for <op_name> from pre-rewrite`. If the agent is uncertain whether a specific override should be ported, record an `open_questions` item in plan.json §3 (`needs_human_decision`) and port conservatively.
1. **KERNEL_CHECK** — see §5 below.
#### 3c. MINOR path (`case = minor`)
```
implement-op(
op_name=<op_name>,
manifest_signature=<manifest_signature>,
source_op=<source_op>,
source_test=<source_test>
)
```
Sub-skill does ANALYZE → DIAGNOSE → IMPLEMENT → VALIDATE → MARK_DONE → COMMIT. align-op waits for SUCCESS or BLOCKED.
### 4. Skipped anchor (reserved)
### <a id="kernel_check"></a>5. KERNEL_CHECK (redesign path only)
Determine whether the kernel layer also needs work. align-op does **not** modify kernel code; it surfaces the question.
For each Kernel class referenced in `source.kernel_map`:
1. Inspect the kernel's `__init__` / `forward` / `_build_program` signatures (wherever applicable) in its source file.
1. Compare against the new op's kernel-build call emitted by scaffold-op (`self.kernel_map[<key>](<args>)`). Specifically check:
- Argument names and positional order.
- Argument types.
- Any layout / dtype expectations the kernel documents.
1. Classify per kernel:
- `aligned` — new op's kernel invocation matches the kernel's ctor; no kernel work.
- `signature_drift` — arg names or order differ; kernel ctor must be adjusted.
- `semantic_drift` — the kernel expects a different data layout / dtype than the new op provides (e.g. op now passes `(M, N)` where kernel expects `(N, M)`).
- `unknown` — cannot determine from static inspection.
Write `.foundry/plan/<op_name>/kernel-check.json`:
```json
{
"op_name": "CumsumFwdOp",
"checked_at": "YYYY-MM-DDTHH:MM:SSZ",
"kernels": [
{
"dispatch_key": "cumulative_fwd",
"kernel_class": "CumulativeKernel",
"kernel_source": "tileops/kernels/reduction/cumulative.py",
"classification": "aligned",
"op_call": "self.kernel_map['cumulative_fwd'](M, N, 'sum', self.dtype, tune=self.tune)",
"kernel_ctor": "__init__(self, M, N, op_kind, dtype, *, tune=False)",
"notes": "Positional and named args match; no kernel work required."
}
]
}
```
Non-`aligned` entries surface in REPORT as `needs_kernel_work` follow-ups. align-op itself continues to TEST — the downstream path may still pass if the kernel drift only affects performance (not correctness), or fail fast if the kernel mismatch causes runtime errors, which REVALIDATE will catch.
### 6. TEST
```
test-op(
op_name=<op_name>,
manifest_signature=<manifest_signature>,
pytorch_equivalent=<pytorch_equivalent>,
source_test=<source_test>
)
```
Sub-skill writes tests against the new spec. Termination:
- **tests fail on current code** (expected TDD seed) → proceed to IMPLEMENT.
- **DONE_SKIP** (tests already pass, e.g. a sibling migration fixed the base class, or minor-path `implement-op` already closed the gap in Step 3c) → skip IMPLEMENT, proceed to BENCH.
### 7. IMPLEMENT
```
implement-op(
op_name=<op_name>,
manifest_signature=<manifest_signature>,
source_op=<source_op>,
source_test=<source_test>
)
```
Closes the gap between the emitted op file and the tests from Step 6. Applies to:
- **Green path**: `scaffold-op` produced the 17 mechanical slots, but not optional hooks or family protocol vars; `implement-op` fills any that are required for the tests to pass.
- **Redesign path**: the `PORT` sub-step in Step 3b did a first pass; `implement-op` closes residual gaps surfaced by the tests.
- **Minor path**: **skipped** — `implement-op` already ran as the minor-case main stage in Step 3c. If TEST didn't DONE_SKIP here, that signals spec-drift beyond the minor-case scope and becomes BLOCKED.
BLOCKED if the gap requires kernel-layer changes (align-op is op-layer only; kernel work surfaces via `kernel-check.json` from KERNEL_CHECK or as a `blocked` return from `implement-op`).
### 8. BENCH
```
bench-op(
op_name=<op_name>,
source_bench=<source_bench>,
source_op=<source_op>
)
```
Produces numbers. Sub-skill unchanged. If BLOCKED and reason is not kernel-related, propagate blocked.
### 9. REVALIDATE
```bash
python scripts/validate_manifest.py --check-op <op_name>
python -m pytest <source_test> -v
```
Both must pass. Regression after benchmark changes → BLOCKED.
### 10. FLIP_STATUS
Orchestrator (not a sub-skill) edits the manifest:
- `ops.<op_name>.status: spec-only` → `status: implemented`
- Commit as `[Refactor][Manifest] promote <op_name> to implemented`.
This is the only manifest write in the entire workflow, and it MUST stay within the [Status flip carve-out](../../rules/manifest-trust-model.md#status-flip-carve-out); any contractual-field change requires a separate manifest-only PR.
### 11. CLEANUP
On SUCCESS path:
- Delete `.foundry/plan/<op_name>/pre-rewrite/` (redesign case only; archive purpose is served).
- Keep `mode.json`, `plan.json`, `kernel-check.json` as audit trail — they are under `.foundry/plan/` which is gitignored but persists in the local worktree.
On BLOCKED path: keep all artefacts for post-mortem.
### 12. REPORT
Single-page summary printed to stdout. Always includes:
```
Status: SUCCESS | BLOCKED
Op: <op_name>
Case: green | redesign | minor
Mode decided by: auto | user_prompt | flag_override
File: <source.op> (<lines>)
Sub-skills run:
- scaffold-op: <SUCCESS|BLOCKED|skipped>
- implement-op: <...>
- test-op: <...>
- bench-op: <...>
Plan artefacts (.foundry/plan/<op_name>/):
- mode.json
- plan.json (if scaffold-op ran)
- kernel-check.json (if redesign path)
- pre-rewrite/ (redesign path, cleaned on SUCCESS)
Status flipped: spec-only → implemented (commit <sha>)
Follow-ups:
- <needs_kernel_work for kernel X> (from kernel-check.json non-aligned entries)
- <needs_doc_fix for slot S21> (from plan.json §3)
- <needs_human_decision about port of _pad_value> (from port observations)
```
On BLOCKED, replace "Status flipped" line with the blocking error and list remaining follow-ups.
## Interaction with `align-family`
`align-family` is the family-scoped orchestrator and delegates every per-op stage to `align-op`. Its workflow is `AUDIT → GROUP_BY_BASE → ROUTE → (per op: ALIGN_OP) → CLEANUP_GATE → CLEANUP → CREATE_PR`; the family orchestrator never invokes the atomic per-op skills (`scaffold-op` / `test-op` / `implement-op` / `bench-op`) directly — every per-op stage runs inside `align-op`'s contract.
- Use `align-op <op>` for per-op work (green field, redesign, or minor delta).
- Use `align-family <family>` for family-scoped historical migration of many ops at once.
They do not conflict. `align-op` never manages cross-op cleanup gates; that remains `align-family`'s. `align-op`'s `FLIP_STATUS` is the sole manifest-write site, observed by `align-family` via `align-op`'s SUCCESS return.
## Non-goals
- **Kernel scaffolding / kernel-layer edits.** align-op surfaces kernel work as a follow-up via `kernel-check.json`; a separate (future) `kernel-scaffold` / `kernel-align` skill will own that layer.
- **Family-level cleanup.** Cross-op dual-path removal lives in `align-family` and is not a concern of per-op alignment.
- **General auto-detection of "redesign vs minor."** The distinction is a design judgement; align-op prompts or accepts `--mode`. The one exception is the **first-op bias** in CLASSIFY (no canonical-pattern precedent in the family → auto `redesign`). Beyond that one case, no auto-detection.
- **Manifest changes (other than FLIP_STATUS).** Per the trust model, manifest changes live in separate manifest PRs.

102
.claude/skills/audit-family/SKILL.md
View File

@ -1,102 +0,0 @@
---
name: audit-family
description: Compare each op's code signature against its manifest spec, classify gaps, produce a structured report.
---
## Arguments
Family name from `tileops/manifest/` (e.g., `reduction`, `norm`, `attention`).
## Contract
- **Input**: `family` name
- **Output**: `.foundry/migrations/<family>.json`
- **Termination**: all ops classified. `ready` ops verified via `--check-op`.
## Workflow
```mermaid
stateDiagram-v2
[*] --> LOAD_MANIFEST
LOAD_MANIFEST --> FILTER: manifest loaded
FILTER --> SELECT_OP: spec-only ops filtered by family
SELECT_OP --> READ_CODE: op selected
READ_CODE --> COMPARE_SIGNATURE: code inspected
COMPARE_SIGNATURE --> CHECK_OP_VALIDATE: no signature difference
COMPARE_SIGNATURE --> CHECK_PYTORCH_REF: difference found
CHECK_OP_VALIDATE --> mark_ready: --check-op passes all levels
CHECK_OP_VALIDATE --> CHECK_PYTORCH_REF: --check-op fails (deeper gap)
CHECK_PYTORCH_REF --> mark_semantic_gap: pytorch_equivalent exists
CHECK_PYTORCH_REF --> mark_blocked: no PyTorch reference
mark_ready --> SELECT_OP: next op
mark_semantic_gap --> SELECT_OP: next op
mark_blocked --> SELECT_OP: next op
SELECT_OP --> WRITE_REPORT: all ops classified
WRITE_REPORT --> [*]
```
Key gate: `pytorch_equivalent` determines autonomous vs human-required migration. `--check-op` confirms `ready` classification.
## Classification
| Classification | Condition | Downstream |
| -------------- | ------------------------------------------------------------------ | ----------------------------------- |
| `ready` | `--check-op` passes, no signature difference | Orchestrator flips status directly |
| `semantic_gap` | Manifest-code difference + `pytorch_equivalent` exists | test-op → implement-op |
| `blocked` | Difference but no PyTorch reference; or kernel-level change needed | Terminate. `reason` field explains. |
## Gap Report Format
Location: `.foundry/migrations/<family>.json`
Top-level:
```json
{
"family": "reduction",
"audited_at": "2026-04-03T...",
"total": 21,
"summary": {"ready": 0, "semantic_gap": 21, "blocked": 0},
"ops": { "<op_name>": { ... } }
}
```
Per-op entry. **Example — field set is not exhaustive. Add fields that aid the next step, omit those that don't.**
```json
{
"status": "spec-only",
"source_op": "tileops/ops/reduction/softmax.py",
"base_class": "_SoftmaxBaseOp",
"classification": "semantic_gap",
"missing_params": ["dim"],
"manifest_signature": {
"inputs": {"x": {"dtype": "float16 | bfloat16"}},
"outputs": {"y": {"dtype": "same_as(x)"}},
"params": {"dim": {"type": "int", "default": -1}},
"shape_rules": ["y.shape == x.shape"]
},
"manifest_param_order": ["x", "dim"],
"pytorch_equivalent": "torch.nn.functional.softmax",
"notes": "dim hardcoded to -1"
}
```
Required fields: `classification`, `source_op`, `base_class`, `manifest_signature`. `base_class` is the immediate parent class name (e.g., `_SoftmaxBaseOp`); if the op inherits `Op` directly, use `"Op"`. The orchestrator uses `base_class` for GROUP_BY_BASE grouping. Gap report is a starting point — agent reads live code and manifest during downstream skills.
`pytorch_equivalent`: corresponding PyTorch function, or `null`. Not every op has one.
## Steps
1. Load the merged manifest via `from tileops.manifest import load_manifest`, or read `tileops/manifest/<family>.yaml` directly when scoping to one family.
1. Filter ops where `family == <arg>` and `status == spec-only`
1. For each op:
a. Read source file (`source_op`), find Op class, extract `__init__` and `forward` explicit named params
b. Compare against `manifest_signature` (inputs, params)
c. If no difference → run `python scripts/validate_manifest.py --check-op <name>` to confirm → `ready` or deeper gap
d. If difference → determine `pytorch_equivalent`:
- Strip `_fwd` suffix, match against `torch.nn.functional`, `torch`, `torch.special`, `torch.linalg` → `semantic_gap`
- `_bwd` ops → always `blocked` (PyTorch backward is autograd-internal, no public reference function for autonomous testing)
- No match → `blocked`
1. Write gap report to `.foundry/migrations/<family>.json`
1. Print summary table

55
.claude/skills/bench-op/SKILL.md
View File

@ -1,55 +0,0 @@
---
name: bench-op
description: Fix benchmark file to work with the new Op interface. Run benchmark, fix errors, repeat until it produces numbers.
---
## Arguments
`op_name`, `source_bench`, `source_op` — passed by align-family orchestrator.
## Contract
- **Input**: `op_name`, `source_bench`, `source_op`
- **Output**: updated benchmark file + commit
- **Termination (success)**: benchmark runs to completion, produces numeric output (latency/TFLOPS/bandwidth)
- **Termination (blocked)**: failure is not interface-related (e.g., kernel bug). Return `blocked` with reason.
- **Constraint**: must NOT modify Op implementation or tests. Benchmark-only changes.
- **Environment**: local GPU required.
## Workflow
```mermaid
stateDiagram-v2
[*] --> RUN_BENCH
RUN_BENCH --> DONE: benchmark produces numbers
RUN_BENCH --> FIX_ERROR: runtime error
FIX_ERROR --> RUN_BENCH: benchmark code fixed
FIX_ERROR --> BLOCKED: error is not interface-related
BLOCKED --> [*]: return blocked to orchestrator
DONE --> COMMIT
COMMIT --> [*]
```
## Steps
### 1. RUN_BENCH
Execute the benchmark file:
```bash
python -m pytest <source_bench> -v
```
If it runs and produces numbers → DONE.
### 2. FIX_ERROR
Read the traceback. The error is the signal — typically the benchmark constructs the Op with the old interface (e.g., `Op(M, N, dtype)` instead of `Op(dim, dtype)`).
Fix the benchmark construction code to use the new Op interface. This is interactive — fix what breaks, don't predict errors in advance.
If the error is NOT about interface mismatch (e.g., kernel crash, CUDA error) → BLOCKED.
### 3. COMMIT
Commit benchmark changes only. Follow `docs/design/testing.md` benchmark requirements.

131
.claude/skills/fix-manifest/SKILL.md
View File

@ -1,131 +0,0 @@
---
name: fix-manifest
description: Patch one missing structural field (kernel_map, static_dims) on an existing tileops/manifest/ entry. Auto-detects the field via the validator or takes `--field=<name>`. Reference-derivable fields (signature.*, shape_rules, dtype_combos, roofline) belong to add-manifest, not here.
---
## Arguments
| Argument | Required | Description |
| ---------------- | -------- | -------------------------------------------------------------------------------------------------- |
| `op_name` | Yes | One manifest key, or comma-separated list (e.g., `RMSNormFwdOp` or `SumFwdOp,MeanFwdOp,VarFwdOp`). |
| `--field=<name>` | No | One of `kernel_map`, `static_dims`. Omit to auto-detect. |
| `--dry-run` | No | Print diff and exit; no write, no PR. |
Multi-op: same `--field` applied to every op in the list. Multi-field is not supported — run again.
## Contract
- **MAY write** in `tileops/manifest/<family>.yaml` (the family file owning the entry): `source.kernel_map`, `signature.static_dims`. These two fields are derived from on-disk op / kernel evidence, not from the reference API. Use `ruamel.yaml` for round-trip preservation.
- **MUST NOT write** anything else. Reference-derivable fields (`signature.{inputs,outputs,params,shape_rules,dtype_combos}`, `roofline.*`) belong to `add-manifest` — re-aligning those fields requires re-fetching the reference URL, which is `add-manifest`'s job. Other fields (`status`, `family`, `ref_api`, `workloads`, `source.{kernel,op,test,bench,bench_manifest_driven}`) are human-curated and not touched by either skill.
- **MUST NOT** create new entries — use `add-manifest`.
- **MUST NOT** flip `status` (that is `align-op@FLIP_STATUS`).
- **MUST NOT** edit op / kernel / test / bench code.
- **One field per invocation.**
- **Termination**: every patched op's validator output is **monotonic** (no new error category vs. before the patch), or BLOCKED.
## Workflow
```mermaid
stateDiagram-v2
[*] --> PRE_CHECK
PRE_CHECK --> [*]: entry missing → BLOCKED
PRE_CHECK --> DIAGNOSE
DIAGNOSE --> INFER: target in allowed list
DIAGNOSE --> [*]: forbidden field → BLOCKED
DIAGNOSE --> [*]: nothing to fix
INFER --> PATCH
INFER --> [*]: cannot infer → BLOCKED
PATCH --> VALIDATE
VALIDATE --> [*]: regression → revert and BLOCK
VALIDATE --> CREATE_PR
CREATE_PR --> [*]
```
## Steps
### 1. PRE_CHECK
Resolve `op_name` in `tileops/manifest/`. Missing → BLOCKED: `op not in manifest; use add-manifest`.
### 2. DIAGNOSE
When `--field=` is provided: must be `kernel_map` or `static_dims`; else BLOCKED. Skip both checks below.
When `--field=` is omitted, run two checks in strict order:
**Check A — `kernel_map` presence.** If `source.kernel_map` is missing or empty → target = `kernel_map`, jump to INFER.
The validator only warns on missing `kernel_map` when `status == implemented`, so spec-only entries need this explicit check. Do NOT extend it to `static_dims` — `docs/design/manifest.md` (R7, R20) explicitly allows `static_dims` to be absent on fixed-rank ops; absence-only patching would manufacture changes for valid entries.
**Check B — validator output.** Run `python scripts/validate_manifest.py --check-op <op_name>`. Parse the first error:
- Field is `static_dims` → target = `static_dims`, jump to INFER.
- Field is reference-derivable (`signature.{inputs,outputs,params,shape_rules,dtype_combos}`, `roofline.*`) → BLOCKED with redirect: `"<field> belongs to add-manifest; re-align this entry with /add-manifest <op_name> <ref_url>"`.
- Other forbidden fields (`status`, `family`, etc.) → BLOCKED. Name the field, why it is out of scope, and what owns it.
- No errors and Check A also empty → no-op; print `nothing to fix` and exit 0.
Write `.foundry/plan/<op_name>/fix-diagnosis.json`: `{op_name, target_field, validator_excerpt, action}`.
### 3. INFER
Build the patch payload from on-disk evidence. **Never guess** — if inference is impossible, BLOCKED with `evidence_needed: <what>`.
**`kernel_map`** — read the op file:
- T2 (L1-direct): copy `default_kernel_map()`'s return dict verbatim.
- T1 (thin wrapper, see `docs/design/ops-design.md` § "Family-specific protocol variables"): family bases expose `default_kernel_map()` returning `{self._kernel_key: self._kernel_cls}`. Read it; substitute the subclass's `_kernel_key` / `_kernel_cls`.
- Output format per `docs/design/manifest.md` § kernel_map: `{<dispatch_key>: <BareKernelClassName>}` — bare class name, NOT fully-qualified.
**`static_dims`** — `signature.inputs` shape names that the op binds at construction time (each entry in the op's `__init__` kwarg block, excluding `dtype` / `kernel_map` / `tune` / `signature.params` entries — see `docs/design/ops-design.md` § "Step 3"). Cross-check with `roofline.vars` if present.
### 4. PATCH
First capture the validator baseline — **before any file mutation**:
```bash
python scripts/validate_manifest.py --check-op <op_name> > /tmp/fix-manifest-<op>-before.txt
```
Do NOT use `git stash` for this — unsafe on a dirty tree (pulls in unrelated user changes).
Then insert each new key as a **sibling** of existing keys in its parent block, at this exact position (verifiable from any sibling entry):
| Field | YAML path | Position |
| ------------- | ----------------------- | ----------------------------------------------------------------------------------------------------- |
| `kernel_map` | `source.kernel_map` | between `source.kernel` and `source.op` |
| `static_dims` | `signature.static_dims` | between `signature.params` and `signature.shape_rules`; if `params` absent, after `signature.outputs` |
Preserve adjacent comments. Do not reorder unrelated keys. If the existing entry deviates from the canonical layout, fall back to the order in `docs/design/manifest.md`.
### 5. VALIDATE
Capture after-baseline and diff:
```bash
python scripts/validate_manifest.py --check-op <op_name> > /tmp/fix-manifest-<op>-after.txt
diff /tmp/fix-manifest-<op>-before.txt /tmp/fix-manifest-<op>-after.txt
```
Acceptable iff after's errors are a subset of before's (monotonic). Any new error → revert that op's patch and BLOCKED.
For multi-op runs: per-op independent. One op's regression reverts only that op's patch; siblings proceed.
Spec-only entries usually carry pre-existing errors (the reason they are spec-only — typically `[signature]` mismatches). Those are out of scope here — `align-op` closes them later.
### 6. CREATE_PR
If `--dry-run`, print diff and exit 0. Otherwise invoke `foundry:creating-pull-request`:
| Element | Single-op | Multi-op |
| ------- | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- |
| title | `[Maintain][Manifest] fix <field> for <op_name>` (use `[Fix][Manifest]` if validator was actively rejecting) | `[Maintain][Manifest] add <field> for <family> spec-only ops` |
| branch | `maintain/manifest/fix-<op-slug>-<field>` | `maintain/manifest/fix-<family>-<field>` |
| body | which field, evidence, validator before/after, scope guard | per-op evidence table; per-op monotonic-check result; scope guard |
## Guardrails
- One field per invocation.
- Never widen scope to a forbidden field — emit BLOCKED.
- Never invent values; payload must trace to a file or to `ref_api`.
- Never flip `status`.
- Validator output ambiguous → STOP, ask user.

166
.claude/skills/follow-up/SKILL.md
View File

@ -1,166 +0,0 @@
---
name: follow-up
description: Introspect a development session and generate follow-up issues for deferred work, discovered problems, and coverage gaps. Max 3 issues per invocation.
---
## Args
| Argument | Description |
| -------------- | ----------------------------------------------------------------------------------------- |
| `<PR_NUMBER>` | Required. TileOPs PR number. |
| `--nightshift` | Skip the interactive presentation; auto-accept all candidates; inject `nightshift` label. |
## Contract
Input: PR ref + conversation (if available). Output: ≤3 follow-up issues, in-scope fixes committed, out-of-scope suggestions printed to stdout. **Never edit the PR body** — the review skill owns it.
## Modes
- **Session-rich**: introspection is primary signal; PR supplements.
- **Session-poor**: PR diff + human reviewer comments only.
## Steps
### 1. Resolve PR
```bash
NIGHTSHIFT=0
for arg in "$@"; do
case "$arg" in
--nightshift) NIGHTSHIFT=1 ;;
-*) echo "Unknown flag: $arg" >&2; exit 1 ;;
*) PR_NUMBER="${PR_NUMBER:-$arg}" ;;
esac
done
[[ -z "${PR_NUMBER:-}" ]] && { echo "Usage: /follow-up <PR_NUMBER>" >&2; exit 1; }
gh pr view "$PR_NUMBER" --json number,title,url,body
OWNER_REPO=$(gh repo view --json nameWithOwner -q '.nameWithOwner')
```
PR not found → terminate.
### 2. Collect
| Source | How |
| ----------------- | ----------------------------------------------------- |
| Diff | `gh pr diff "$PR_NUMBER"` |
| Session | Scan for deferrals, workarounds, blocked items |
| In-code markers | Grep changed files for `TODO`, `FIXME`, `HACK`, `XXX` |
| Reviewer comments | Both endpoints below, filtered to non-author non-bot |
```bash
export PR_AUTHOR=$(gh pr view "$PR_NUMBER" --json author -q '.author.login')
FILTER='[.[] | select(.user.login != env.PR_AUTHOR and .user.type != "Bot"
and (.user.login | test("copilot|gemini|github-actions"; "i") | not))]'
gh api "repos/$OWNER_REPO/pulls/$PR_NUMBER/comments" --paginate --jq "$FILTER"
gh api "repos/$OWNER_REPO/issues/$PR_NUMBER/comments" --paginate --jq "$FILTER"
```
### 3. Classify
**Issue-worthy** (→ follow-up issue):
| Category | Signal |
| ---------------- | ------------------------------------------------------ |
| Scope deferral | "not in this PR", explicit defer |
| Fragile coupling | Workarounds, monkey-patches |
| Coverage gap | Untested cases, missing edge cases, skipped benchmarks |
| Consistency gap | Doc drift; same problem in other modules |
**Suggestion** (no issue):
| Tier | Signal | Action |
| ------------ | ----------------------------------------------- | ------ |
| In-scope | Touches only this PR's files; small, mechanical | Step 7 |
| Out-of-scope | Outside diff, expands behavior, or subjective | Step 8 |
Uncertain → out-of-scope. Nothing in either bucket → step 8 (empty report). Do not manufacture follow-ups.
### 4. Merge → max 3 issues
Same module / root cause → merge. Different + independent → keep separate.
### 5. Present
`--nightshift`: skip; auto-accept; → step 6.
Default — render in dependency order, wait on `Actions:`:
```
Follow-up candidates from PR #<N>: <title>
1. [TYPE][SCOPE] <title> Category: <…> | <1–2 sentences>
2. [TYPE][SCOPE] <title> ← parallel with #1
3. [TYPE][SCOPE] <title> ← depends on #1
Execution order: {#1, #2} → #3
In-scope fixes:
- <file:line> — <nit>
Out-of-scope suggestions:
- <nit>
Actions: confirm all / drop by N / edit / move <item> to out-of-scope
```
### 6. Create — delegate to `creating-issue`
For each confirmed item: write a draft to a tmpfile, then invoke `foundry:creating-issue --from-draft <tmpfile>`. **Never `gh issue create` directly** — that bypasses the 5-section HARD GATE which `foundry:pipeline` Phase A re-validates downstream.
Draft body must conform to the canonical template at `foundry/skills/creating-issue/SKILL.md` Step 4 — `creating-issue` is the single owner; do not duplicate or paraphrase its section names here.
Frontmatter:
```yaml
---
type: <FEAT|BUG|PERF|REFACTOR|DOCS|TEST>
component: <affected module>
labels: [follow-up] # add `nightshift` only when --nightshift was passed
target_repo: <OWNER_REPO>
---
```
Labels (the `nightshift` label is human-curated — fail fast, do not auto-create):
```bash
gh label list --search follow-up --json name --jq '.[].name' | grep -qx follow-up \
|| gh label create follow-up --color c5def5 --description "Generated from dev session introspection"
if [[ "$NIGHTSHIFT" == "1" ]]; then
gh label list --search nightshift --json name --jq '.[].name' | grep -qx nightshift \
|| { echo "nightshift label missing in $OWNER_REPO" >&2; exit 1; }
fi
```
### 7. Apply in-scope fixes
For each: verify file is in `gh pr diff --name-only "$PR_NUMBER"` (else demote); Edit; run a fast check (`pre-commit run --files <paths>` or module-scoped unit tests). Commit the batch:
```
[Chore] apply in-scope follow-up suggestions from PR #$PR_NUMBER review
- <file:line> — <what changed>
```
Fix fails its check or adds unrelated diff → `git restore`, demote. Never force-push or amend.
### 8. Report
Stdout only. Omit empty sections.
```
PR #<PR_NUMBER> follow-up complete.
Issues created:
- #<N> — <summary>
Applied (commit <sha>):
- <file:line> — <summary>
Out-of-scope suggestions:
- <nit>
Execution order: {#A, #B} → #C
```
Nothing in any bucket: `PR #<PR_NUMBER>: no follow-up issues or suggestions.`

99
.claude/skills/implement-op/SKILL.md
View File

@ -1,99 +0,0 @@
---
name: implement-op
description: Modify op code to match the manifest-declared interface, making spec tests pass.
---
## Arguments
`op_name`, `manifest_signature`, `source_op`, `source_test` — passed by align-family orchestrator.
## Contract
- **Input**: `op_name`, `manifest_signature`, `source_op`, `source_test`
- **Output**: modified op code + commit + `observations` list (returned to orchestrator)
- **Termination (success)**: `python scripts/validate_manifest.py --check-op <name>` all levels pass + new tests pass.
- **Termination (blocked)**: fix requires changes beyond Op layer. Return `blocked` with reason.
- **Constraint**: must NOT modify `tileops/manifest/`. Must NOT modify tests written by `test-op` in this align-op run (spec contract). MAY update pre-existing tests in `<source_test>` whose call-sites use the legacy API. The parent orchestrator (`align-op`) handles the manifest flip at FLIP_STATUS, bound by the [Status flip carve-out](../../rules/manifest-trust-model.md#status-flip-carve-out). If your implementation needs a contractual-field change to make tests pass, return BLOCKED — do not edit the manifest yourself.
- **Behavioral compatibility**: default param values (from manifest) must produce identical results to the old implementation. The old API shape (e.g., `__init__(M, N)`) is NOT preserved — the manifest defines the target interface.
## Workflow
```mermaid
stateDiagram-v2
[*] --> ANALYZE
ANALYZE --> DIAGNOSE: semantic knowledge extracted
DIAGNOSE --> VALIDATE: gap already resolved (base class fixed by previous op)
DIAGNOSE --> IMPLEMENT: gap exists
IMPLEMENT --> VALIDATE: sub-step completed
VALIDATE --> IMPLEMENT: sub-step failed
VALIDATE --> MARK_DONE: all pass
VALIDATE --> BLOCKED: fix exceeds Op layer scope
MARK_DONE --> COMMIT
COMMIT --> [*]
BLOCKED --> [*]: return blocked to orchestrator
```
## Dual-path policy
Before refactoring a base class, count its subclasses:
```bash
grep -rlE "class\s+[A-Z][A-Za-z0-9]*\s*\(\s*<BaseName>\s*\)" tileops/ops/
```
- **One subclass** (the op being migrated): refactor the base in place. No dual-path.
- **Multiple subclasses**: keep the legacy `__init__` path alongside the new one so unmigrated siblings still pass. Cleanup gate removes the legacy path after all siblings migrate.
Do NOT preemptively migrate siblings to avoid dual-path — that violates per-op scope.
## Steps
### 1. ANALYZE
Read existing code (`source_op`) to extract semantic knowledge:
- What the computation does (the algorithm)
- Where constraints are hardcoded (e.g., `dim=-1`, `reshape(-1, N)`)
- What the generalization path is
Manifest = WHAT (target interface). Existing code = HOW (computation logic). The delta is the work.
This analysis is internal — not persisted.
### 2. DIAGNOSE
Check if the gap still exists. A previous op's migration may have fixed the shared base class.
- Gap resolved → skip to VALIDATE
- Gap exists → proceed to IMPLEMENT
### 3. IMPLEMENT
Decompose into independently verifiable sub-steps. Each sub-step either succeeds or fails with precise location (→ BLOCKED). No retry loops — if a sub-step fails with the same error twice, the task is beyond current scope.
Agent determines sub-steps based on the specific gap. Do not follow a fixed recipe.
### 4. VALIDATE
Run all checks:
```bash
python scripts/validate_manifest.py --check-op <op_name>
python -m pytest <source_test> -v
```
All must pass. If not, return to IMPLEMENT.
### 5. MARK_DONE
Record `observations` — design knowledge discovered during migration:
- Patterns (e.g., "scan ops need transpose, reduction ops reshape")
- Edge cases
- Abstraction opportunities
Do NOT modify manifest or design docs. Observations are returned to orchestrator and surfaced in PR for human review.
### 6. COMMIT
Commit code changes only. Do not commit manifest changes.

98
.claude/skills/resolve-tileops/SKILL.md
View File

@ -1,98 +0,0 @@
---
name: resolve-tileops
description: Per-round driver of stateful agent-driven review-resolution on a tile-ai/TileOPs PR (developer side). Designed for /loop dynamic mode — re-fires until a terminal action. Outer caller must run preflight.sh once before starting the loop. State persists in `.foundry/runs/{issue-<N> | pr-<PR>}/resolve/`.
---
## Arguments
| Argument | Required | Description |
| ------------- | -------- | -------------------------------- |
| `<PR_NUMBER>` | Yes | TileOPs PR number (e.g. `1133`). |
Resolution work runs in **this agent session** — no external subagent.
## Step 1: Pre-round
```bash
PR=$ARGUMENTS
PRE=$(bash .claude/skills/resolve-tileops/round-pre.sh "$PR") || exit 1
ACTION=$(echo "$PRE" | jq -r .action)
ROUND=$(echo "$PRE" | jq -r .round)
RUN_DIR=$(echo "$PRE" | jq -r .run_dir)
SNAP=$(echo "$PRE" | jq -r .snap_prefix)
MESSAGE=$(echo "$PRE" | jq -r .message)
```
Branch on `ACTION`:
- `terminate-success` / `terminate-diverged` / `terminate-external` /
`terminate-stalled` — write retrospective (see below), print
`MESSAGE`, **return without ScheduleWakeup**.
- `idle` — print `MESSAGE`, ScheduleWakeup with `delaySeconds=180`,
`prompt=/resolve-tileops <PR>`, `reason="PR #<PR> idle — polling"`. Return.
- `continue` — proceed to Step 2.
### Retrospective (terminal actions only)
Write a terse retrospective directly to `$RUN_DIR/retrospective.md` based
on this session's understanding of what happened across the loop's rounds.
Required:
- **Problem** (1–2 lines on core reviewer concerns)
- **Resolution** (`all-addressed` | `partial` | `unresolved`, one line)
Optional (only if substantive):
- **Approach** (1–2 lines on techniques applied)
- **Follow-up** (concrete deferred items, one per line)
Action-oriented; no long prose.
## Step 2: Load context
**Round 1 only** — read into your working context:
- `.claude/skills/resolve-tileops/procedure.md` — triage / fix / reply / resolve threads.
- `.claude/skills/resolve-tileops/criteria.md` — reply formats and hard rules.
Round 2+ rely on session memory.
**Every round** — read this round's inputs:
- `$RUN_DIR/inbox-history/round-NN.md` — this round's inbox guidance,
if any (only present if `inbox.md` was non-empty when round-pre archived it).
- `$SNAP.new-reviews.json`
- `$SNAP.new-inline-comments.json`
- `$SNAP.unresolved-threads.json`
- `$SNAP.ci.json`
- `$SNAP.auto-resolve.json` — JSON action plan from the stale-bot auto-resolver. `resolve[]` lists threads the classifier *planned* to auto-reply + resolve (known bot, anchored to a stale commit, no non-whitelisted participant anywhere in the thread); per-thread outcomes live under `executed.{resolved, reply_failed, resolve_failed}` so consumers can distinguish a planned-and-completed resolve from a planned-but-failed one. `unknown_bot_like[]` and `$RUN_DIR/round-NN.unknown-bot-like.json` flag bot-like logins missing from `known-bots.json` for human triage. `skip[]` lists threads the resolver intentionally left untouched (humans, mixed-author threads, or bots already at HEAD).
## Step 3: Resolve
Apply `procedure.md`. Inbox guidance for this round (if any) overrides
default behavior where they conflict.
Do NOT post a structured trailer in any PR comment — `round-post.sh`
detects round effects from GitHub state.
## Step 4: Post-round
```bash
POST=$(bash .claude/skills/resolve-tileops/round-post.sh "$PR") || exit 1
PUSHED_SHA=$(echo "$POST" | jq -r .pushed_sha)
THREADS_RESOLVED=$(echo "$POST" | jq -r .threads_resolved)
OPEN_AFTER=$(echo "$POST" | jq -r .open_after)
```
Print: `Round $ROUND done — pushed=$PUSHED_SHA, threads_resolved=$THREADS_RESOLVED, open_after=$OPEN_AFTER.`
## Step 5: Self-schedule (only under /loop)
After a real work round, ScheduleWakeup:
- `prompt`: `/resolve-tileops <PR>`
- `delaySeconds`: `180`
- `reason`: `"PR #<PR> resolve round <ROUND> — pushed=<PUSHED_SHA>"`
Outside `/loop`: skip; just return.

227
.claude/skills/resolve-tileops/auto-resolve-stale.sh
View File

@ -1,227 +0,0 @@
#!/usr/bin/env bash
# auto-resolve-stale.sh — classify unresolved review threads for the
# stale-bot auto-resolution policy and (in normal mode) execute the
# GraphQL mutations that auto-reply + resolve qualifying threads.
#
# Inputs:
# --threads <file> JSON {head_sha, threads:[{id, comments:{nodes:[
# {id, databaseId, author:{login}, commit:{oid},
# body, path, line}]}}]}
# --bots <file> known-bots.json
# --run-dir <dir> where to drop unknown-bot-like artifact
# --round <NN> zero-padded round number for artifact filename
# --dry-run do not call the GraphQL API; just emit the action plan
#
# Stdout: JSON {resolve:[{thread_id,comment_id,login,reply}],
# unknown_bot_like:[{thread_id,login}],
# skip:[{thread_id,reason}]}
#
# Side effect: $RUN_DIR/round-<NN>.unknown-bot-like.json (only when
# at least one unknown bot-like thread was seen this call).
set -euo pipefail
REPLY_TEXT="Not assessed on latest HEAD"
THREADS_FILE=""; BOTS_FILE=""; RUN_DIR=""; ROUND=""; DRY_RUN=0
while (( $# )); do
case "$1" in
--threads) THREADS_FILE="$2"; shift 2 ;;
--bots) BOTS_FILE="$2"; shift 2 ;;
--run-dir) RUN_DIR="$2"; shift 2 ;;
--round) ROUND="$2"; shift 2 ;;
--dry-run) DRY_RUN=1; shift ;;
*) echo "auto-resolve-stale: unknown arg $1" >&2; exit 2 ;;
esac
done
[[ -f "$THREADS_FILE" ]] || { echo "auto-resolve-stale: missing --threads" >&2; exit 2; }
[[ -f "$BOTS_FILE" ]] || { echo "auto-resolve-stale: missing --bots" >&2; exit 2; }
[[ -n "$RUN_DIR" ]] || { echo "auto-resolve-stale: missing --run-dir" >&2; exit 2; }
[[ "$ROUND" =~ ^[0-9]{2}$ ]] || { echo "auto-resolve-stale: --round must be zero-padded 2-digit" >&2; exit 2; }
command -v jq >/dev/null 2>&1 || { echo "auto-resolve-stale: missing jq" >&2; exit 2; }
# Build the action plan in pure jq — single pass, no shell-side per-thread state.
# REPLY_TEXT is passed in as --arg so the action plan and the executed
# mutation body below share a single source of truth.
# Validate the unknown-bot-like policy up front so the config field is
# load-bearing rather than decorative. Only "log_for_manual_triage" is
# implemented today — any other value fails fast.
POLICY=$(jq -r '.policy.unknown_bot_like_login // ""' "$BOTS_FILE")
if [[ "$POLICY" != "log_for_manual_triage" ]]; then
echo "auto-resolve-stale: unsupported policy.unknown_bot_like_login='$POLICY' (expected 'log_for_manual_triage')" >&2
exit 2
fi
PLAN=$(jq --slurpfile bots "$BOTS_FILE" --arg reply "$REPLY_TEXT" '
# Normalise both sides: strip a trailing "[bot]" suffix before comparing.
# GitHub returns either "copilot-pull-request-reviewer" or
# "copilot-pull-request-reviewer[bot]" depending on the API; treat them
# as the same identity for the whitelist check.
def strip_bot: sub("\\[bot\\]$"; "");
def is_known($known; $login):
($known | map(strip_bot) | index($login | strip_bot)) != null;
# Bot-like pattern: unknown identities that look like a GitHub App.
# Only the literal "[bot]" suffix counts — that suffix is reserved by
# GitHub for GitHub Apps and cannot appear in a human username. Any
# login ending in "[bot]" that is not on the whitelist is bucketed as
# an unknown bot-like identity for human triage, regardless of the
# prefix. A bare "-reviewer" / "-bot" without the suffix is a regular
# user account (e.g. "alice-reviewer") and stays in the human bucket.
def is_bot_like($login):
$login | test("\\[bot\\]$");
. as $in
| ($bots[0].review_bot_logins // []) as $known
| ($in.head_sha) as $head
| [ $in.threads[]
| . as $t
| ($t.comments.nodes[0]) as $first
| ($first.author.login // "") as $login
| ($first.commit.oid // "") as $oid
# Whole-thread author check. Collect every non-whitelisted login
# found anywhere in the thread, then split it into:
# - human_repliers: login that is not bot-like (no [bot] suffix)
# - unknown_bot_repliers: login that is bot-like but not on the
# whitelist (e.g. a previously-unseen GitHub App)
# ANY non-whitelisted participant — human OR unknown bot — must
# disqualify the thread from auto-resolve. Recording them in
# unknown_bot_like surfaces them for human triage as the PR
# contract requires; resolving the thread anyway would silently
# swallow that signal.
| [ $t.comments.nodes[]
| (.author.login // "")
| select(. != "" and (is_known($known; .) | not))
] as $unknown_logins
| ($unknown_logins | map(select(is_bot_like(.) | not)) | length) as $human_repliers
| ($unknown_logins | map(select(is_bot_like(.)))) as $unknown_bot_logins
| {
thread_id: $t.id,
comment_id: ($first.id // ""),
login: $login,
oid: $oid,
known_bot: is_known($known; $login),
bot_like: is_bot_like($login),
# Distinct buckets:
# missing_oid: comment has no commit anchor → cannot judge stale
# stale: oid present and != head
# at_head: oid present and == head
missing_oid: ($oid == ""),
stale: ($oid != $head and $oid != ""),
mixed: (($human_repliers > 0) or (($unknown_bot_logins | length) > 0)),
unknown_bot_logins: ($unknown_bot_logins | unique)
}
] as $rows
| {
resolve: [
$rows[] | select(.known_bot and .stale and (.mixed | not))
| { thread_id, comment_id, login, reply: $reply }
],
# A thread enters unknown_bot_like if its root is a bot-like
# unknown identity OR if any non-root comment introduces an
# unknown bot-like login. Each (thread, login) pair is emitted
# once so the human-triage artifact captures every unrecognized
# GitHub App that touched the thread.
unknown_bot_like: [
$rows[]
| . as $r
| (
([ if ($r.bot_like and ($r.known_bot|not)) then $r.login else empty end ]
+ $r.unknown_bot_logins)
| unique
| .[]
| { thread_id: $r.thread_id, login: . }
)
],
skip: [
$rows[]
| select(
.mixed # any non-whitelisted participant in thread
or ((.known_bot|not) and (.bot_like|not)) # human-rooted (no other comments)
or (.known_bot and (.stale|not)) # bot not stale (at HEAD or missing oid)
)
| {
thread_id,
reason: (
if .mixed and .known_bot then "mixed_thread_known_bot_root"
elif .mixed and .bot_like then "mixed_thread_unknown_bot_like_root"
elif .mixed then "mixed_thread_human_root"
elif .known_bot and .missing_oid then "known_bot_missing_commit_oid"
elif .known_bot and (.stale|not) then "known_bot_at_head"
elif (.bot_like|not) then "human_reviewer"
else "other"
end
)
}
]
}
' "$THREADS_FILE")
if (( DRY_RUN )); then
# Dry-run is a pure read: no GraphQL mutations AND no on-disk side
# effects. The artifact write below belongs only to the live path so
# tests / future tooling can inspect the action plan without leaving
# a stale unknown-bot-like artifact behind.
printf '%s\n' "$PLAN"
exit 0
fi
# Drop the artifact for human triage when there is anything to record.
ARTIFACT="$RUN_DIR/round-${ROUND}.unknown-bot-like.json"
UNKNOWN_COUNT=$(printf '%s' "$PLAN" | jq '.unknown_bot_like | length')
if (( UNKNOWN_COUNT > 0 )); then
mkdir -p "$RUN_DIR"
printf '%s' "$PLAN" | jq '.unknown_bot_like' > "$ARTIFACT"
fi
# Execute mutations for each resolve entry: post a reply on the first
# comment, then — only if the reply succeeded — mark the thread resolved.
# Skipping the resolve when the reply fails preserves the contract that
# every auto-resolved thread carries the neutral reply, so a thread we
# could not reply to is left unresolved for the next round / human
# triage. Per-thread failures are logged but do not abort the loop, so
# one transient API hiccup doesn't strand the rest of the batch.
#
# GH_BIN allows the test harness to inject a mock gh binary that
# simulates reply / resolve failures without touching the network.
GH_BIN="${GH_BIN:-gh}"
command -v "$GH_BIN" >/dev/null 2>&1 || { echo "auto-resolve-stale: missing $GH_BIN" >&2; exit 2; }
RESOLVE_COUNT=$(printf '%s' "$PLAN" | jq '.resolve | length')
RESOLVED_IDS=()
REPLY_FAILED_IDS=()
RESOLVE_FAILED_IDS=()
if (( RESOLVE_COUNT > 0 )); then
while IFS= read -r tid; do
[[ -n "$tid" ]] || continue
if "$GH_BIN" api graphql -f query='
mutation($tid:ID!,$body:String!){
addPullRequestReviewThreadReply(input:{
pullRequestReviewThreadId:$tid, body:$body
}){ comment{ id } }
}' -F tid="$tid" -F body="$REPLY_TEXT" >/dev/null; then
if "$GH_BIN" api graphql -f query='
mutation($tid:ID!){
resolveReviewThread(input:{threadId:$tid}){ thread{ id isResolved } }
}' -F tid="$tid" >/dev/null; then
RESOLVED_IDS+=("$tid")
else
echo "auto-resolve-stale: resolve failed for $tid" >&2
RESOLVE_FAILED_IDS+=("$tid")
fi
else
echo "auto-resolve-stale: reply failed for $tid; leaving thread unresolved" >&2
REPLY_FAILED_IDS+=("$tid")
fi
done < <(printf '%s' "$PLAN" | jq -r '.resolve[].thread_id')
fi
# Annotate the plan with execution outcomes so the caller (and the test
# harness) can distinguish "resolved" from "reply failed, left open".
# Use `jq -n '$ARGS.positional' --args …` to lift each shell array into
# a JSON string array in one shot — the `${arr[@]+"${arr[@]}"}` guard is
# needed under `set -u` so an empty array does not trip "unbound
# variable".
printf '%s' "$PLAN" | jq \
--argjson resolved "$(jq -n '$ARGS.positional' --args ${RESOLVED_IDS[@]+"${RESOLVED_IDS[@]}"})" \
--argjson reply_failed "$(jq -n '$ARGS.positional' --args ${REPLY_FAILED_IDS[@]+"${REPLY_FAILED_IDS[@]}"})" \
--argjson resolve_failed "$(jq -n '$ARGS.positional' --args ${RESOLVE_FAILED_IDS[@]+"${RESOLVE_FAILED_IDS[@]}"})" \
'. + {executed: {resolved: $resolved, reply_failed: $reply_failed, resolve_failed: $resolve_failed}}'

27
.claude/skills/resolve-tileops/criteria.md
View File

@ -1,27 +0,0 @@
### 1. Inline reply format
- **Accept**: `Adopted. <what was fixed>. See <short_sha>.`
- **Reject**: `<conclusion>. <evidence or reasoning>.`
- **Defer**: `Valid point. Out of scope for this PR — <reason>. Will address in follow-up.`
One reply per thread. Resolve every thread that was replied to, regardless of verdict.
### 2. Top-level note (optional, only when something cross-cutting can't fit inline)
```
Addressed in <short_sha>. See thread replies for specifics.
### Cross-cutting (optional)
- <one line per pattern handled jointly across threads>
### Deferred (optional)
- <one line per item out of scope, with reason>
```
### 3. Hard rules
- Do NOT restate inline replies. If a thread already has the reply, the top-level note must not repeat it.
- No per-file / per-line bullets. Those live inline.
- No GitHub review IDs (`#4181055953`) — meaningless to humans.
- One short paragraph + at most two markdown sections in any top-level note. If you can't fit it, rethink whether it belongs inline.
- All replies in English; concise — conclusion, action, reasoning. No filler.

11
.claude/skills/resolve-tileops/known-bots.json
View File

@ -1,11 +0,0 @@
{
"review_bot_logins": [
"copilot-pull-request-reviewer",
"Copilot",
"gemini-code-assist",
"github-actions"
],
"policy": {
"unknown_bot_like_login": "log_for_manual_triage"
}
}

88
.claude/skills/resolve-tileops/preflight.sh
View File

@ -1,88 +0,0 @@
#!/usr/bin/env bash
# preflight.sh <PR_NUMBER>
#
# Validates env and initializes per-PR resolve state. Idempotent: round 1
# does cold init (env checks, TASK_ROOT resolution from PR body, mkdir,
# meta.json), round 2+ scans existing state and returns it without rework.
# The skill body assumes preflight has succeeded.
#
# Stdout: absolute path to the run dir's meta.json (single line).
# Stderr: human-readable status / errors.
# Exit 0: state ready. Non-zero: env or arg error.
set -euo pipefail
PR="${1:?usage: preflight.sh <PR_NUMBER>}"
[[ "$PR" =~ ^[0-9]+$ ]] || { echo "preflight: PR must be a positive integer" >&2; exit 1; }
REPO="tile-ai/TileOPs"
# Dependency checks first — both branches below use jq/gh.
command -v gh >/dev/null 2>&1 || { echo "preflight: missing gh" >&2; exit 1; }
command -v jq >/dev/null 2>&1 || { echo "preflight: missing jq" >&2; exit 1; }
SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Anchor state to the main checkout's `.foundry/runs/`, not cwd. When
# invoked from a linked worktree (e.g. via foundry pipeline HANDOFF),
# `--show-toplevel` returns the worktree path; using it would scatter
# resolve state across worktrees. `git rev-parse --git-common-dir`
# returns the main worktree's `.git` directory regardless of which
# worktree we run from; its parent is the main checkout. Single
# command, no pipe — avoids SIGPIPE under `set -o pipefail`.
GIT_COMMON_DIR="$(git -C "$SKILL_DIR" rev-parse --git-common-dir 2>/dev/null)" \
|| { echo "preflight: cannot resolve repo root from \$SKILL_DIR=$SKILL_DIR" >&2; exit 1; }
# --git-common-dir is relative to the invoking worktree when that
# worktree is the main one, absolute when invoked from a linked
# worktree. Normalize to absolute, then take parent.
[[ "$GIT_COMMON_DIR" != /* ]] && GIT_COMMON_DIR="$SKILL_DIR/$GIT_COMMON_DIR"
REPO_PATH="$(cd "$GIT_COMMON_DIR/.." && pwd)"
# Round 2+ fast path: an existing run dir's meta.json already pins this PR.
META=""
for m in "$REPO_PATH/.foundry/runs"/*/resolve/meta.json; do
[[ -f "$m" ]] || continue
if [[ "$(jq -r '.pr_number' "$m" 2>/dev/null)" = "$PR" ]]; then
META="$m"
break
fi
done
if [[ -n "$META" ]]; then
echo "preflight: state exists for PR #$PR at $META" >&2
echo "$META"
exit 0
fi
# Round 1 cold start: validate repo remote, resolve TASK_ROOT, create state.
git -C "$REPO_PATH" remote -v \
| awk '/tile-ai\/TileOPs(\.git)?[[:space:]]+\(fetch\)/ {found=1; exit} END{exit !found}' \
|| { echo "preflight: no git remote in $REPO_PATH points to tile-ai/TileOPs" >&2; exit 1; }
PR_BODY=$(gh pr view "$PR" --repo "$REPO" --json body --jq .body) \
|| { echo "preflight: gh pr view failed for PR #$PR (auth? missing?)" >&2; exit 1; }
ISSUE=$(printf '%s' "$PR_BODY" \
| grep -oiE '(Closes|Fixes|Resolves)[[:space:]]+#[0-9]+' \
| head -1 \
| grep -oE '[0-9]+' \
|| true)
if [[ -n "$ISSUE" ]]; then
TASK_ROOT="$REPO_PATH/.foundry/runs/issue-$ISSUE"
else
TASK_ROOT="$REPO_PATH/.foundry/runs/pr-$PR"
fi
RUN_DIR="$TASK_ROOT/resolve"
META="$RUN_DIR/meta.json"
mkdir -p "$RUN_DIR/rounds" "$RUN_DIR/inbox-history"
: > "$RUN_DIR/inbox.md"
jq -n --arg pr "$PR" --arg repo "$REPO" '{
pr_number:($pr|tonumber), repo:$repo,
status:"active",
round:0, max_rounds:15,
last_processed_review_id:0,
last_processed_review_comment_id:0,
last_pushed_sha:null,
consecutive_idle:0
}' > "$META"
echo "preflight: state initialized for PR #$PR at $META" >&2
echo "$META"

53
.claude/skills/resolve-tileops/procedure.md
View File

@ -1,53 +0,0 @@
## Triage each unresolved thread and summary item
For each inline thread's root comment **and** each actionable item from review summaries, classify:
| Verdict | Criteria | Action |
| ---------- | ----------------------------------------------------------------- | ------------------------------------- |
| **Accept** | Feedback is correct and fix belongs in this PR's scope | Fix the code |
| **Reject** | Feedback is incorrect, irrelevant, or based on a misunderstanding | Reply with reasoning |
| **Defer** | Feedback is valid but fix would significantly expand PR scope | Acknowledge, explain why out of scope |
**Bias toward Accept.** Only defer when the fix would:
- Touch files/modules unrelated to the PR's stated purpose.
- Require a design decision not yet made.
- Be large enough to warrant its own review cycle.
Evaluate all feedback on merit, regardless of whether the reviewer is human or bot.
## Apply fixes
For each accepted comment, read the relevant file, understand the context, and make the **minimal** fix. Commit message: `[Chore][<scope>] address review feedback` where `<scope>` comes from the PR title's `[Type][Scope]` pattern (omit `[<scope>]` if the title has none).
## Reply and resolve threads; optional top-level note
### Inline threads (always)
For each thread, reply **inline** then resolve:
```bash
# Reply
gh api "repos/$REPO/pulls/$PR/comments/<root_comment_id>/replies" \
-f body="<reply>"
# Resolve thread
gh api graphql -f query='
mutation($id:ID!) {
resolveReviewThread(input:{threadId:$id}) {
thread { isResolved }
}
}' -f id="<thread_node_id>"
```
### Top-level PR comment (optional)
Skip if every item is covered by an inline reply. Only post when there is something inline can't carry: cross-cutting context, a defer rationale that spans the PR, or a one-line pointer to the fix commit.
```bash
gh api "repos/$REPO/issues/$PR/comments" -f body="<top-level note>"
```
**Deduplicate**: if multiple reviewers (or summary + inline) raise the same point, fix once, reference the same commit in each reply.
Resolve every thread that was replied to, regardless of verdict.

138
.claude/skills/resolve-tileops/round-post.sh
View File

@ -1,138 +0,0 @@
#!/usr/bin/env bash
# round-post.sh <PR_NUMBER>
#
# Post-round work for resolve-tileops: detect what changed during the
# round (push, threads resolved), write the round summary, advance meta.
# Reads baseline from $RUN_DIR/.round-pre.json (left by round-pre.sh).
#
# Stdout: single JSON line summary {round, pushed_sha, threads_resolved,
# open_after}.
# Stderr: human-readable status / errors.
# Exit 0: round finalized. Non-zero: missing state.
set -euo pipefail
PR="${1:?usage: round-post.sh <PR_NUMBER>}"
[[ "$PR" =~ ^[0-9]+$ ]] || { echo "round-post: PR must be a positive integer" >&2; exit 1; }
command -v gh >/dev/null 2>&1 || { echo "round-post: missing gh" >&2; exit 1; }
command -v jq >/dev/null 2>&1 || { echo "round-post: missing jq" >&2; exit 1; }
REPO="tile-ai/TileOPs"
SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Anchor state lookup to the main checkout (see preflight.sh).
GIT_COMMON_DIR="$(git -C "$SKILL_DIR" rev-parse --git-common-dir 2>/dev/null)" \
|| { echo "round-post: cannot resolve repo root from \$SKILL_DIR=$SKILL_DIR" >&2; exit 1; }
[[ "$GIT_COMMON_DIR" != /* ]] && GIT_COMMON_DIR="$SKILL_DIR/$GIT_COMMON_DIR"
REPO_PATH="$(cd "$GIT_COMMON_DIR/.." && pwd)"
META=""
for m in "$REPO_PATH/.foundry/runs"/*/resolve/meta.json; do
[[ -f "$m" ]] || continue
if [[ "$(jq -r '.pr_number' "$m" 2>/dev/null)" = "$PR" ]]; then
META="$m"
break
fi
done
[[ -n "$META" ]] || { echo "round-post: no state for PR #$PR" >&2; exit 1; }
RUN_DIR=$(dirname "$META")
PRE_JSON="$RUN_DIR/.round-pre.json"
[[ -f "$PRE_JSON" ]] \
|| { echo "round-post: missing $PRE_JSON — was round-pre.sh run with action=continue?" >&2; exit 1; }
HEAD_SHA_BEFORE=$(jq -r '.head_sha' "$PRE_JSON")
UNRESOLVED_BEFORE=$(jq -r '.unresolved_before' "$PRE_JSON")
REVIEWER_STATE_BEFORE=$(jq -r '.reviewer_state_before' "$PRE_JSON")
# Use the PRE-round watermark when advancing meta. If the reviewer added
# new comments mid-round (after round-pre snapshotted but before
# round-post), they'd be lost if we used the post-round max. Reading from
# the baseline guarantees mid-round feedback is processed next round.
PRE_LATEST_REVIEW_ID=$(jq -r '.latest_review_id' "$PRE_JSON")
PRE_LATEST_REVIEW_COMMENT_ID=$(jq -r '.latest_review_comment_id' "$PRE_JSON")
NEW_HEAD_SHA=$(gh pr view "$PR" --repo "$REPO" --json headRefOid --jq .headRefOid)
PUSHED_SHA="none"
[[ "$NEW_HEAD_SHA" != "$HEAD_SHA_BEFORE" ]] && PUSHED_SHA="$NEW_HEAD_SHA"
# Paginate reviewThreads — same rationale as round-pre.sh.
NEW_UNRESOLVED=0
cursor=''
while :; do
page=$(gh api graphql -f query='
query($owner:String!,$repo:String!,$pr:Int!,$after:String){
repository(owner:$owner,name:$repo){
pullRequest(number:$pr){
reviewThreads(first:100, after:$after){
nodes{ isResolved }
pageInfo{ hasNextPage endCursor }
}
}
}
}' -F owner=tile-ai -F repo=TileOPs -F pr="$PR" \
${cursor:+-f after="$cursor"})
page_unresolved=$(printf '%s' "$page" \
| jq '[.data.repository.pullRequest.reviewThreads.nodes[]|select(.isResolved==false)]|length')
NEW_UNRESOLVED=$((NEW_UNRESOLVED + page_unresolved))
has_next=$(printf '%s' "$page" \
| jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.hasNextPage')
[[ "$has_next" == "true" ]] || break
cursor=$(printf '%s' "$page" \
| jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.endCursor')
done
# Clamp at 0 — new threads opened during the round can make the raw
# delta negative, but reporting "-3 threads_resolved" is misleading.
THREADS_RESOLVED_RAW=$(( UNRESOLVED_BEFORE - NEW_UNRESOLVED ))
if (( THREADS_RESOLVED_RAW < 0 )); then
THREADS_RESOLVED=0
else
THREADS_RESOLVED=$THREADS_RESOLVED_RAW
fi
ROUND=$(jq -r '.round' "$META")
NEXT_ROUND=$((ROUND + 1))
N=$(printf '%02d' "$NEXT_ROUND")
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
# Round summary file
jq -n --argjson r "$NEXT_ROUND" --arg now "$NOW" \
--arg sha_b "$HEAD_SHA_BEFORE" --arg sha_a "$NEW_HEAD_SHA" \
--arg pushed "$PUSHED_SHA" \
--argjson resolved "$THREADS_RESOLVED" \
--argjson unresolved_after "$NEW_UNRESOLVED" \
--arg reviewer_state "$REVIEWER_STATE_BEFORE" \
'{round:$r, finished_at:$now, head_sha_before:$sha_b, head_sha_after:$sha_a,
pushed_sha:$pushed, threads_resolved:$resolved,
unresolved_after:$unresolved_after, reviewer_state_before:$reviewer_state}' \
> "$RUN_DIR/rounds/round-$N.json"
# Advance meta. last_pushed_sha is sticky: only touched when a push
# actually happened this round, preserving its prior value (null on a
# fresh state, or the last real sha) otherwise. Watermarks come from the
# PRE-round baseline so mid-round reviewer activity is picked up next
# round.
if [[ "$PUSHED_SHA" != "none" ]]; then
jq --argjson r "$NEXT_ROUND" \
--argjson rid "$PRE_LATEST_REVIEW_ID" \
--argjson cid "$PRE_LATEST_REVIEW_COMMENT_ID" \
--arg pushed "$PUSHED_SHA" \
'.round=$r | .last_processed_review_id=$rid
| .last_processed_review_comment_id=$cid
| .last_pushed_sha=$pushed' \
"$META" > "$META.tmp" && mv "$META.tmp" "$META"
else
jq --argjson r "$NEXT_ROUND" \
--argjson rid "$PRE_LATEST_REVIEW_ID" \
--argjson cid "$PRE_LATEST_REVIEW_COMMENT_ID" \
'.round=$r | .last_processed_review_id=$rid
| .last_processed_review_comment_id=$cid' \
"$META" > "$META.tmp" && mv "$META.tmp" "$META"
fi
rm -f "$PRE_JSON"
jq -n \
--argjson r "$NEXT_ROUND" \
--arg pushed "$PUSHED_SHA" \
--argjson resolved "$THREADS_RESOLVED" \
--argjson open_after "$NEW_UNRESOLVED" \
'{round:$r, pushed_sha:$pushed, threads_resolved:$resolved, open_after:$open_after}'

360
.claude/skills/resolve-tileops/round-pre.sh
View File

@ -1,360 +0,0 @@
#!/usr/bin/env bash
# round-pre.sh <PR_NUMBER>
#
# Pre-round work for resolve-tileops: locate state, snapshot the current
# PR view, decide action (continue/idle/terminate). On 'continue', gather
# this round's input snapshots and archive any inbox.
#
# Prerequisite: preflight.sh must have already initialized state for this
# PR. round-pre.sh does NOT init; it errors if state is missing.
#
# Stdout: single JSON object describing the action and (on continue) the
# snapshot prefix the skill body should read.
# Stderr: human-readable status / errors.
# Exit 0: action ready (skill body should branch on .action).
# Exit non-zero: missing state or upstream failure.
set -euo pipefail
PR="${1:?usage: round-pre.sh <PR_NUMBER>}"
[[ "$PR" =~ ^[0-9]+$ ]] || { echo "round-pre: PR must be a positive integer" >&2; exit 1; }
command -v gh >/dev/null 2>&1 || { echo "round-pre: missing gh" >&2; exit 1; }
command -v jq >/dev/null 2>&1 || { echo "round-pre: missing jq" >&2; exit 1; }
REVIEWER_LOGIN="${RESOLVE_REVIEWER_LOGIN:-Ibuki-wind}"
SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Anchor state lookup to the main checkout (see preflight.sh).
GIT_COMMON_DIR="$(git -C "$SKILL_DIR" rev-parse --git-common-dir 2>/dev/null)" \
|| { echo "round-pre: cannot resolve repo root from \$SKILL_DIR=$SKILL_DIR" >&2; exit 1; }
[[ "$GIT_COMMON_DIR" != /* ]] && GIT_COMMON_DIR="$SKILL_DIR/$GIT_COMMON_DIR"
REPO_PATH="$(cd "$GIT_COMMON_DIR/.." && pwd)"
# Locate state. preflight.sh must have created it.
META=""
for m in "$REPO_PATH/.foundry/runs"/*/resolve/meta.json; do
[[ -f "$m" ]] || continue
if [[ "$(jq -r '.pr_number' "$m" 2>/dev/null)" = "$PR" ]]; then
META="$m"
break
fi
done
[[ -n "$META" ]] \
|| { echo "round-pre: no state for PR #$PR — run preflight.sh first" >&2; exit 1; }
RUN_DIR=$(dirname "$META")
ROUND=$(jq -r '.round' "$META")
MAX_ROUNDS=$(jq -r '.max_rounds' "$META")
LAST_REVIEW_ID_PREV=$(jq -r '.last_processed_review_id' "$META")
LAST_REVIEW_COMMENT_ID_PREV=$(jq -r '.last_processed_review_comment_id' "$META")
# Stall safety net. Increment on idle (no progress this round), reset on
# continue. Hitting MAX_IDLE terminates the loop so a dead counterpart
# (e.g. review-loop crashed) doesn't leave us polling forever. Hardcoded
# rather than read from meta.json so a state file from an older skill
# version with a stricter threshold doesn't silently override the
# current floor.
CONSECUTIVE_IDLE=$(jq -r '.consecutive_idle // 0' "$META")
MAX_IDLE=20
# Pin `gh pr view` to the repo recorded in meta.json. Preflight already
# stamped this to the canonical base repo; without `--repo`, gh defaults
# to the worktree's origin remote, which in a fork checkout points at
# the contributor's fork and either fails or fetches the wrong PR.
META_REPO=$(jq -r '.repo // empty' "$META")
[[ -n "$META_REPO" ]] \
|| { echo "round-pre: meta.json missing .repo — re-run preflight.sh" >&2; exit 1; }
[[ "$META_REPO" =~ ^[^/]+/[^/]+$ ]] \
|| { echo "round-pre: meta.json .repo='$META_REPO' must be exactly 'owner/name' (single slash, both halves non-empty) — re-run preflight.sh" >&2; exit 1; }
PR_JSON=$(gh pr view "$PR" --repo "$META_REPO" --json state,headRefOid,isDraft 2>/dev/null) \
|| { echo "round-pre: gh pr view failed" >&2; exit 1; }
PR_STATE=$(echo "$PR_JSON" | jq -r .state)
HEAD_SHA=$(echo "$PR_JSON" | jq -r .headRefOid)
# Split META_REPO into owner/name for the downstream
# `gh api repos/<OWNER>/<NAME>/...` calls. Older versions of this
# script derived these from a `baseRepository` json field, which gh
# CLI 2.88.1 dropped.
REPO_OWNER="${META_REPO%%/*}"
REPO_NAME="${META_REPO##*/}"
REPO="$REPO_OWNER/$REPO_NAME"
# Reviews + inline comments — paginate so PRs with >1 page don't
# silently lose the latest IDs / state.
# --slurp + --jq are mutually exclusive in `gh api`; pipe through
# external jq instead. --paginate --slurp produces an array of pages
# (each page is itself an array of items), so jq flattens with [.[][]].
ALL_REVIEWS=$(gh api --paginate --slurp "repos/$REPO/pulls/$PR/reviews")
ALL_COMMENTS=$(gh api --paginate --slurp "repos/$REPO/pulls/$PR/comments")
LATEST_REVIEWER_STATE=$(printf '%s' "$ALL_REVIEWS" \
| jq -r "[.[][]|select(.user.login==\"$REVIEWER_LOGIN\")] | sort_by(.submitted_at) | last | .state // \"NONE\"")
LATEST_REVIEW_ID=$(printf '%s' "$ALL_REVIEWS" \
| jq -r "[.[][]|select(.user.login==\"$REVIEWER_LOGIN\")|.id]|max // 0")
LATEST_REVIEW_COMMENT_ID=$(printf '%s' "$ALL_COMMENTS" \
| jq -r "[.[][]|select(.user.login==\"$REVIEWER_LOGIN\")|.id]|max // 0")
# Unresolved review thread count — paginate via cursor so PRs with
# >100 threads don't undercount.
count_unresolved() {
local cursor='' total=0 page page_unresolved has_next
while :; do
page=$(gh api graphql -f query='
query($owner:String!,$repo:String!,$pr:Int!,$after:String){
repository(owner:$owner,name:$repo){
pullRequest(number:$pr){
reviewThreads(first:100, after:$after){
nodes{ isResolved }
pageInfo{ hasNextPage endCursor }
}
}
}
}' -F owner="$REPO_OWNER" -F repo="$REPO_NAME" -F pr="$PR" \
${cursor:+-f after="$cursor"})
page_unresolved=$(printf '%s' "$page" \
| jq '[.data.repository.pullRequest.reviewThreads.nodes[]|select(.isResolved==false)]|length')
total=$((total + page_unresolved))
has_next=$(printf '%s' "$page" \
| jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.hasNextPage')
[[ "$has_next" == "true" ]] || break
cursor=$(printf '%s' "$page" \
| jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.endCursor')
done
echo "$total"
}
UNRESOLVED=$(count_unresolved)
# Decide action — first match wins. PR_STATE==DRAFT does not stop.
ACTION=""; MESSAGE=""
case "$PR_STATE" in
MERGED|CLOSED) ACTION="terminate-external"; MESSAGE="PR #$PR is $PR_STATE — stopping." ;;
esac
if [[ -z "$ACTION" && "$ROUND" -ge "$MAX_ROUNDS" ]]; then
ACTION="terminate-diverged"
MESSAGE="Reached max rounds ($MAX_ROUNDS) — human attention needed."
fi
if [[ -z "$ACTION" \
&& "$UNRESOLVED" -eq 0 \
&& "$LATEST_REVIEWER_STATE" == "APPROVED" \
&& "$LATEST_REVIEW_ID" == "$LAST_REVIEW_ID_PREV" \
&& "$LATEST_REVIEW_COMMENT_ID" == "$LAST_REVIEW_COMMENT_ID_PREV" ]]; then
# Approve + everything processed → exit. Watermark equality on both
# review-id and comment-id ensures we don't skip a not-yet-processed
# APPROVE review body that has no inline comments.
ACTION="terminate-success"
MESSAGE="PR #$PR converged — all threads resolved, reviewer approved."
fi
# Idle gate: only sleep when there's nothing to do. Unresolved threads
# (from any source) override idle — the dev should still process them
# even if the canonical reviewer hasn't posted new activity.
if [[ -z "$ACTION" \
&& "$UNRESOLVED" -eq 0 \
&& "$LATEST_REVIEW_ID" == "$LAST_REVIEW_ID_PREV" \
&& "$LATEST_REVIEW_COMMENT_ID" == "$LAST_REVIEW_COMMENT_ID_PREV" ]]; then
ACTION="idle"
MESSAGE="No new review feedback for PR #$PR — sleeping."
fi
[[ -z "$ACTION" ]] && ACTION="continue"
# Stall counter: increment on idle, reset on continue. If idle persists
# beyond max_idle, escalate to terminate-stalled — protects against the
# review-loop dying silently while we poll forever.
if [[ "$ACTION" == "idle" ]]; then
NEW_IDLE=$((CONSECUTIVE_IDLE + 1))
if (( NEW_IDLE >= MAX_IDLE )); then
ACTION="terminate-stalled"
MESSAGE="No reviewer activity for $NEW_IDLE consecutive rounds (max_idle=$MAX_IDLE) — terminating."
fi
jq --argjson n "$NEW_IDLE" '.consecutive_idle=$n' "$META" \
> "$META.tmp" && mv "$META.tmp" "$META"
elif [[ "$ACTION" == "continue" ]]; then
if (( CONSECUTIVE_IDLE != 0 )); then
jq '.consecutive_idle=0' "$META" \
> "$META.tmp" && mv "$META.tmp" "$META"
fi
fi
NEXT_ROUND=$((ROUND + 1))
SNAP_PREFIX=""
if [[ "$ACTION" == "continue" ]]; then
N=$(printf '%02d' "$NEXT_ROUND")
SNAP_PREFIX="$RUN_DIR/rounds/round-$N"
mkdir -p "$RUN_DIR/rounds"
printf '%s' "$ALL_REVIEWS" \
| jq "[.[][]|select(.user.login==\"$REVIEWER_LOGIN\" and .id>$LAST_REVIEW_ID_PREV)|{id,state,body,submitted_at}]" \
> "$SNAP_PREFIX.new-reviews.json"
printf '%s' "$ALL_COMMENTS" \
| jq "[.[][]|select(.user.login==\"$REVIEWER_LOGIN\" and .id>$LAST_REVIEW_COMMENT_ID_PREV)|{id,path,line,body,in_reply_to_id,created_at}]" \
> "$SNAP_PREFIX.new-inline-comments.json"
# Snapshot ALL unresolved threads (paginated). Inner comments() is
# also paginated below: the auto-resolver's whole-thread author check
# depends on seeing every comment, so a thread that overflows the
# first 100 comments would otherwise look bot-only and get
# mis-resolved.
: > "$SNAP_PREFIX.unresolved-threads.json"
collected=()
cursor=''
while :; do
page=$(gh api graphql -f query='
query($owner:String!,$repo:String!,$pr:Int!,$after:String){
repository(owner:$owner,name:$repo){
pullRequest(number:$pr){
reviewThreads(first:100, after:$after){
nodes{
id isResolved
comments(first:100){
pageInfo{ hasNextPage endCursor }
nodes{
id databaseId author{login} body path line
commit{ oid }
}
}
}
pageInfo{ hasNextPage endCursor }
}
}
}
}' -F owner="$REPO_OWNER" -F repo="$REPO_NAME" -F pr="$PR" \
${cursor:+-f after="$cursor"})
items=$(printf '%s' "$page" \
| jq -c '.data.repository.pullRequest.reviewThreads.nodes|map(select(.isResolved==false))[]')
if [[ -n "$items" ]]; then
while IFS= read -r line; do
collected+=("$line")
done <<< "$items"
fi
has_next=$(printf '%s' "$page" \
| jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.hasNextPage')
[[ "$has_next" == "true" ]] || break
cursor=$(printf '%s' "$page" \
| jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.endCursor')
done
# Per-thread comments completion: any thread whose first comments page
# was truncated gets follow-up node(id) queries until exhausted, then
# the new nodes are merged into the thread's comments.nodes array.
# Without this, the whole-thread author check in auto-resolve-stale.sh
# could miss a human reply that landed past comment #100.
for i in "${!collected[@]}"; do
thread="${collected[$i]}"
has_more=$(printf '%s' "$thread" | jq -r '.comments.pageInfo.hasNextPage // false')
[[ "$has_more" != "true" ]] && continue
cursor=$(printf '%s' "$thread" | jq -r '.comments.pageInfo.endCursor')
thread_id=$(printf '%s' "$thread" | jq -r '.id')
extra_nodes='[]'
while :; do
page=$(gh api graphql -f query='
query($id:ID!,$after:String){
node(id:$id){
... on PullRequestReviewThread{
comments(first:100, after:$after){
pageInfo{ hasNextPage endCursor }
nodes{
id databaseId author{login} body path line
commit{ oid }
}
}
}
}
}' -F id="$thread_id" -f after="$cursor")
page_nodes=$(printf '%s' "$page" | jq -c '.data.node.comments.nodes')
extra_nodes=$(jq -nc --argjson a "$extra_nodes" --argjson b "$page_nodes" '$a + $b')
has_next=$(printf '%s' "$page" | jq -r '.data.node.comments.pageInfo.hasNextPage')
[[ "$has_next" == "true" ]] || break
cursor=$(printf '%s' "$page" | jq -r '.data.node.comments.pageInfo.endCursor')
done
collected[$i]=$(printf '%s' "$thread" \
| jq -c --argjson extra "$extra_nodes" \
'.comments.nodes = (.comments.nodes + $extra) | .comments.pageInfo.hasNextPage = false')
done
# Emit the final array.
if (( ${#collected[@]} == 0 )); then
echo '[]' > "$SNAP_PREFIX.unresolved-threads.json"
else
{
echo '['
first_page=1
for thread in "${collected[@]}"; do
[[ "$first_page" -eq 1 ]] && first_page=0 || echo ','
printf '%s' "$thread"
done
echo ']'
} > "$SNAP_PREFIX.unresolved-threads.json"
fi
# Stale-bot auto-resolve: scoped to known bot identities anchored to a
# commit older than current HEAD. Humans and bots-at-HEAD are skipped;
# unknown bot-like logins are recorded for human triage. The classifier
# consumes the unresolved-threads snapshot directly so the pagination
# contract above is the single source of truth.
# Contract: $SNAP_PREFIX.auto-resolve.json is ALWAYS present after
# round-pre completes. Downstream consumers may rely on the file
# existing with the standard shape `{resolve, unknown_bot_like, skip}`.
# On classifier-missing or classifier-crash we still emit a valid empty
# plan plus an `error` field so the failure is visible without breaking
# the consumer's jq pipeline.
AR_OUT="$SNAP_PREFIX.auto-resolve.json"
if [[ -x "$SKILL_DIR/auto-resolve-stale.sh" && -f "$SKILL_DIR/known-bots.json" ]]; then
AR_INPUT="$SNAP_PREFIX.auto-resolve-input.json"
jq -n --arg sha "$HEAD_SHA" \
--slurpfile threads "$SNAP_PREFIX.unresolved-threads.json" \
'{head_sha:$sha, threads:$threads[0]}' > "$AR_INPUT"
# Write to a tmpfile and only mv into place on success — guarantees
# downstream readers never consume a half-written / empty file when
# the classifier crashes mid-emit.
AR_TMP="$AR_OUT.tmp"
if "$SKILL_DIR/auto-resolve-stale.sh" \
--threads "$AR_INPUT" \
--bots "$SKILL_DIR/known-bots.json" \
--run-dir "$RUN_DIR" \
--round "$N" \
> "$AR_TMP"; then
mv "$AR_TMP" "$AR_OUT"
else
echo "round-pre: auto-resolve-stale exited non-zero" >&2
rm -f "$AR_TMP"
jq -n --arg err "auto-resolve-stale exited non-zero" \
'{resolve:[], unknown_bot_like:[], skip:[], error:$err}' > "$AR_OUT"
fi
else
jq -n --arg err "auto-resolve-stale.sh or known-bots.json missing" \
'{resolve:[], unknown_bot_like:[], skip:[], error:$err}' > "$AR_OUT"
fi
gh pr checks "$PR" --repo "$REPO" --json name,state,conclusion \
> "$SNAP_PREFIX.ci.json" 2>/dev/null || echo '[]' > "$SNAP_PREFIX.ci.json"
# Archive inbox for this round, then clear it. Skill body reads the
# archived copy if it wants this round's guidance. Ensure inbox-history
# exists in case the state dir was partially deleted/corrupted.
if [[ -s "$RUN_DIR/inbox.md" ]]; then
mkdir -p "$RUN_DIR/inbox-history"
mv "$RUN_DIR/inbox.md" "$RUN_DIR/inbox-history/round-$N.md"
: > "$RUN_DIR/inbox.md"
fi
# Persist baseline so round-post.sh can compute deltas without
# re-querying. Critically, persist LATEST_REVIEW_ID and
# LATEST_REVIEW_COMMENT_ID so round-post advances the watermark to the
# PRE-round max — items that arrive mid-round get picked up next round.
jq -n --arg sha "$HEAD_SHA" \
--argjson unresolved "$UNRESOLVED" \
--arg state "$LATEST_REVIEWER_STATE" \
--argjson rid "$LATEST_REVIEW_ID" \
--argjson cid "$LATEST_REVIEW_COMMENT_ID" \
'{head_sha:$sha, unresolved_before:$unresolved, reviewer_state_before:$state,
latest_review_id:$rid, latest_review_comment_id:$cid}' \
> "$RUN_DIR/.round-pre.json"
fi
jq -n \
--arg action "$ACTION" \
--arg run_dir "$RUN_DIR" \
--arg snap_prefix "$SNAP_PREFIX" \
--argjson round "$NEXT_ROUND" \
--arg message "$MESSAGE" \
'{action:$action, round:$round, run_dir:$run_dir, snap_prefix:$snap_prefix, message:$message}'

33
.claude/skills/review-tileops/README.md
View File

@ -1,33 +0,0 @@
# review-tileops setup
This skill reviews PRs as a GitHub identity distinct from the PR author. The identity is per-developer machine config — it is not in the repo.
## One-time setup
1. Have a second GitHub account (separate from the one you author PRs with).
1. Pick any directory path you like, then authenticate `gh` against it as that account:
```bash
GH_CONFIG_DIR=/path/to/your/reviewer-config gh auth login --hostname github.com
```
1. Export the same path in your shell rc:
```bash
export TILEOPS_REVIEW_GH_CONFIG_DIR=/path/to/your/reviewer-config
```
## Before reviewing each PR
Run once per PR (the loop driver does this automatically; for a manual single-shot review, run it yourself):
```bash
bash .claude/skills/review-tileops/preflight.sh <PR_NUMBER>
```
It verifies the env var, the `hosts.yml`, and that the reviewer login is distinct from the PR author. The skill itself does no preflight — it assumes this has passed.
## If preflight errors
Each error message names the next command to run. Follow it.

35
.claude/skills/review-tileops/SKILL.md
View File

@ -1,35 +0,0 @@
---
name: review-tileops
description: Single-shot review of a tile-ai/TileOPs PR as a separate GitHub identity from the PR author. Manual / interactive — for autonomous multi-round review until APPROVE, run `bash .claude/skills/review-tileops/loop.sh <PR>` instead.
---
## Input
`$ARGUMENTS`: integer PR number in `tile-ai/TileOPs`. E.g. `1122`.
## Step 0: Preflight
```bash
bash .claude/skills/review-tileops/preflight.sh <PR> || exit 1
export GH_CONFIG_DIR="$TILEOPS_REVIEW_GH_CONFIG_DIR"
```
## Step 1: Gather inputs
```bash
gh pr view <PR> --repo tile-ai/TileOPs \
--json number,title,body,files,headRefOid,state
gh pr diff <PR> --repo tile-ai/TileOPs
```
Parse the title's first two bracket tokens — `[type][scope] description`. Look both up in `loading.yaml`:
- Always load entries under `always:`.
- For each token, if it appears as a key under `match:`, load the listed checklist files.
- Multi-match across the two tokens → union. Both unmatched → no domain checklist; rely on general judgment.
Read each loaded checklist file under `.claude/review-checklists/`, plus `criteria.md`. Skim unresolved review threads (`gh api graphql … reviewThreads`), recent non-reviewer comments (`gh api repos/.../issues/<N>/comments` and `…/pulls/<N>/comments`), and CI status (`gh pr checks <N>`) if relevant to the diff.
## Step 2: Execute the review
Follow `procedure.md` end to end. Submit one atomic review at the end.

44
.claude/skills/review-tileops/criteria.md
View File

@ -1,44 +0,0 @@
### 1. Submit
```bash
gh api repos/tile-ai/TileOPs/pulls/<N>/reviews \
-f event="<EVENT>" \
-f body="<SUMMARY>" \
-f 'comments=[{"path":"<file>","line":<line>,"body":"<comment>"}, ...]'
```
`<EVENT>`: `REQUEST_CHANGES` if any blocking issue, `APPROVE` if clean, `COMMENT` for non-blocking questions only.
### 2. Inline format
```
<what is wrong and why> → <what to change>
```
One comment per issue. Name the function, variable, or pattern. The reader is an agent that executes fixes literally.
### 3. Summary format (markdown)
The summary is for what does **not** fit in an inline comment. Per-file issues belong inline (§2), not in a summary list. Omit empty sections.
```
### Overall
<one line: top risk + next step. No event echo, no item list.>
### Cross-cutting concerns
- <pattern spanning multiple files — only what an inline comment can't carry>
```
Hard rules:
- Do NOT restate inline comments. If a finding is already inline, it must not appear in the summary.
- Per-file/per-line items go in the `comments=[...]` array of §1, never in the summary body.
- Clean PR: one line, `Clean — no issues.`
### 4. Hard rules
- Do not comment outside the PR diff.
- Do not invent issues on a clean PR.
- No eager design-doc reads — only when a guard names a doc AND the diff makes that item ambiguous, or the divergence trigger fires.
- All review text in English.
- All `gh` calls run with `GH_CONFIG_DIR` already exported by the caller. Repo is fixed at `tile-ai/TileOPs`.

30
.claude/skills/review-tileops/loading.yaml
View File

@ -1,30 +0,0 @@
# PR title → review-checklists mapping.
#
# A TileOPs PR title always has the form `[type][scope] description` (the
# scope bracket is optional but conventional). The loop / single-shot
# review checks BOTH bracket tokens against `match` below; any token that
# matches contributes its checklists. Multi-match across the two tokens
# unions; no-match in either contributes nothing. If both miss, the
# review runs without any domain-specific checklist (the reviewer
# applies general judgment).
#
# Add a new entry here when a new type or scope earns a dedicated
# checklist; do NOT add path-based triggers — keep title-driven matching
# as the single rule. If a PR's title misclassifies its content,
# iterate the title.
# Always loaded, regardless of title match.
always:
- pre-review.md
# Token → list of checklist files under .claude/review-checklists/.
match:
Feat: [feature.md]
Enhancement: [feature.md]
Refactor: [refactor.md]
Maintain: [manifest.md]
Manifest: [manifest.md]
Doc: [doc.md]
Design: [doc.md]
Test: [testing.md]
Bench: [benchmark.md]

1046
.claude/skills/review-tileops/loop.sh
View File

File diff suppressed because it is too large Load Diff

37
.claude/skills/review-tileops/preflight.sh
View File

@ -1,37 +0,0 @@
#!/usr/bin/env bash
# Run once per PR, before the first review round, to verify reviewer-identity
# wiring. Subsequent rounds (whether single-shot or loop-driven) skip these
# checks — the skill itself assumes preflight has passed.
set -euo pipefail
PR="${1:?usage: preflight.sh <PR_NUMBER>}"
REPO="tile-ai/TileOPs"
if ! command -v codex >/dev/null 2>&1; then
echo "error: codex CLI not found in PATH." >&2
echo " the review loop runs each round under Codex; install it from https://github.com/openai/codex" >&2
exit 1
fi
if [ -z "${TILEOPS_REVIEW_GH_CONFIG_DIR:-}" ]; then
echo "error: TILEOPS_REVIEW_GH_CONFIG_DIR is not set." >&2
echo " export it in your shell rc; see .claude/skills/review-tileops/README.md" >&2
exit 1
fi
if [ ! -f "$TILEOPS_REVIEW_GH_CONFIG_DIR/hosts.yml" ]; then
echo "error: $TILEOPS_REVIEW_GH_CONFIG_DIR/hosts.yml not found." >&2
echo " run: GH_CONFIG_DIR=$TILEOPS_REVIEW_GH_CONFIG_DIR gh auth login --hostname github.com" >&2
exit 1
fi
REVIEWER=$(GH_CONFIG_DIR="$TILEOPS_REVIEW_GH_CONFIG_DIR" gh api user --jq .login)
AUTHOR=$(GH_CONFIG_DIR="$TILEOPS_REVIEW_GH_CONFIG_DIR" gh pr view "$PR" --repo "$REPO" --json author --jq .author.login)
if [ "$REVIEWER" = "$AUTHOR" ]; then
echo "error: reviewer ($REVIEWER) equals PR #$PR author." >&2
echo " TILEOPS_REVIEW_GH_CONFIG_DIR points at the author's gh config; reviewer must be distinct." >&2
exit 1
fi
echo "OK: reviewer=$REVIEWER PR #$PR author=$AUTHOR (distinct)"

10
.claude/skills/review-tileops/procedure.md
View File

@ -1,10 +0,0 @@
**Free-form review (step 2) is the primary review step.** Project-specific guards (step 4) are a regression net applied last — they exist because the free-form pass is fallible at known classes of issue, NOT because they replace it. Skipping or shortening step 2 to lean on guards is a failure mode.
1. **Read every changed source file in full.** The diff alone lacks surrounding context.
1. **Free-form review (primary).** Flag: logic errors, edge cases, API misuse, races, resource leaks, broken invariants, error-handling gaps, perf regressions, dead code, unclear names, missing tests. This is where the bulk of your reasoning belongs.
1. **Triage unresolved review threads.** For each thread, judge whether the developer's latest change resolves the concern. If unresolved, surface as an inline comment.
1. **Apply each loaded project-specific guard (last, supplement).** Walk its bullets against the diff. Add anything new to the step-2 findings.
1. **Approval-gate decision.** If your draft event is `APPROVE`: read `.claude/review-checklists/approval-gate.md` and run every applicable check (each item self-scopes to the diff per the file's preamble). Downgrade to `REQUEST_CHANGES` if any fail.
1. **Compose inline comments** per `criteria.md` §2 — one per issue, format `<what is wrong and why> → <what to change>`. Name the function, variable, or pattern.
1. **Compose the summary** per `criteria.md` §3. Omit empty sections. A clean PR gets a single line: `Clean — no issues.`
1. **Submit one atomic review** per `criteria.md` §1, with the inline comments in the `comments=[…]` array. Follow `criteria.md` §4 hard rules.

191
.claude/skills/review-tileops/round-post.sh
View File

@ -1,191 +0,0 @@
#!/usr/bin/env bash
# round-post.sh — post-codex hook for the review loop.
#
# Implements Rule 2 (same-path 5-strike monitor): track per-path
# consecutive-blocker streaks across rounds. When any path's streak
# reaches >= 5 consecutive rounds with at least one blocker on it,
# ensure the PR carries the GitHub label "agent-stuck" so a human
# can take a look.
#
# Monitoring-only: this hook MUST NOT modify the round's blockers,
# the unresolved-threads file, the new-review-comments.json file,
# the PR title, or post any comment. Its only side-effects on the
# PR are GitHub label operations (idempotent).
#
# Inputs (env or positional, env wins):
# RUN_DIR — review/ root for this loop run.
# ROUND — integer round number that just finished.
# COMMENTS_JSON — path to round-NN.codex-blockers.json (the post-
# codex snapshot of review comments authored by
# REVIEWER_LOGIN this round; used as the source of
# blocker paths). The pre-codex
# round-NN.new-review-comments.json is the WRONG
# artifact — it explicitly filters out the reviewer
# and contains human comments instead.
# REPO — "owner/repo" string for `gh` operations.
# Default: tile-ai/TileOPs.
# PR — PR number for `gh pr edit`.
#
# Positional fallback: round-post.sh <RUN_DIR> <ROUND> <COMMENTS_JSON> <REPO> <PR>
#
# Optional env (for testing):
# GH_BIN — gh executable to use (default: "gh"). Tests can
# point this at a stub. If empty or "none", label
# operations are skipped entirely (monitor still
# updates region-history.json).
#
# Output: exit 0 on success. Logs to stderr; nothing on stdout.
set -euo pipefail
RUN_DIR="${RUN_DIR:-${1:-}}"
ROUND="${ROUND:-${2:-}}"
COMMENTS_JSON="${COMMENTS_JSON:-${3:-}}"
REPO="${REPO:-${4:-tile-ai/TileOPs}}"
PR="${PR:-${5:-}}"
GH_BIN="${GH_BIN:-gh}"
if [[ -z "$RUN_DIR" || -z "$ROUND" || -z "$COMMENTS_JSON" ]]; then
echo "round-post.sh: missing args; usage: RUN_DIR=... ROUND=... COMMENTS_JSON=... [REPO=... PR=...] round-post.sh" >&2
exit 2
fi
HISTORY="$RUN_DIR/region-history.json"
THRESHOLD=5
LABEL="agent-stuck"
LABEL_DESC="Autonomous loop appears stuck and needs human attention"
LABEL_COLOR="FBCA04"
# --- 1. Extract blocker paths from this round's review comments ---
#
# Source-of-truth contract: this round's codex-blockers.json is the
# post-codex snapshot of review comments authored by the reviewer login
# (i.e. comments codex itself just posted as part of this review).
# Entries with severity=blocker (or category=blocker) carry the .path
# the reviewer flagged. We treat any entry whose .severity or .category
# is "blocker" as a blocker. If neither field exists (older format or
# free-form review), every entry counts. Round-post is monitoring-only:
# the only side-effect is the `agent-stuck` label, which is itself
# idempotent and safe to apply early. Over-counting accelerates the
# threshold (i.e. the label may land sooner than a strict-blocker count
# would warrant); it never produces an unsafe action.
PATHS_THIS_ROUND=()
if [[ -f "$COMMENTS_JSON" ]]; then
# tolerate empty / malformed file by falling back to []
if ! jq empty "$COMMENTS_JSON" >/dev/null 2>&1; then
echo "round-post.sh: $COMMENTS_JSON is not valid JSON; treating as empty" >&2
elif ! jq -e 'type == "array"' "$COMMENTS_JSON" >/dev/null 2>&1; then
# `jq empty` accepts ANY valid JSON, including objects. If the
# artifact is e.g. `{path:"x"}` (a single comment object, or an
# error-wrapper produced by a failed gh call), `.[]` would extract
# its values and miscount them as blocker paths. Require an array
# before iterating; non-array input is treated as no blockers this
# round (counters will reset, label stays unchanged).
echo "round-post.sh: $COMMENTS_JSON is not a JSON array; treating as empty" >&2
else
while IFS= read -r p; do
[[ -n "$p" && "$p" != "null" ]] && PATHS_THIS_ROUND+=("$p")
done < <(jq -r '
[
.[]
| select(
(.severity // "" | ascii_downcase) == "blocker"
or (.category // "" | ascii_downcase) == "blocker"
or ((.severity // "") == "" and (.category // "") == "")
)
| .path // empty
]
| unique
| .[]' "$COMMENTS_JSON")
fi
fi
# --- 2. Update region-history.json: increment present, reset absent ---
#
# Schema (stable across rounds; loaded by future tools):
# {
# "counters": { "<path>": <int>, ... },
# "events": [ {"round": int, "path": str, "comment_ids": [int]}, ... ]
# }
#
# Backward compat: missing file → start from empty state.
# Malformed file → also reset to empty state (don't let bad JSON kill
# the monitor; counters are best-effort).
if [[ ! -f "$HISTORY" ]] || ! jq empty "$HISTORY" 2>/dev/null; then
echo '{"counters":{},"events":[]}' > "$HISTORY"
fi
# Build a JSON array of paths-this-round for jq.
PATHS_JSON=$(printf '%s\n' "${PATHS_THIS_ROUND[@]:-}" \
| jq -R . \
| jq -s 'map(select(. != ""))')
NEW_HISTORY=$(jq \
--argjson paths "$PATHS_JSON" \
'
. as $h
| (.counters // {}) as $cur
# increment counters for paths present this round; reset absent.
| (
($paths | map({key:., value: (($cur[.] // 0) + 1)}) | from_entries)
) as $next
| .counters = $next
| .events = (.events // [])
' "$HISTORY")
printf '%s\n' "$NEW_HISTORY" > "$HISTORY.tmp" && mv "$HISTORY.tmp" "$HISTORY"
# --- 3. Determine which paths just hit the threshold this round ---
TRIGGERED_PATHS=()
while IFS= read -r p; do
[[ -n "$p" ]] && TRIGGERED_PATHS+=("$p")
done < <(jq -r --argjson t "$THRESHOLD" '
.counters
| to_entries
| map(select(.value == $t))
| .[].key' "$HISTORY")
# --- 4. Append events for triggered paths (postmortem trail) ---
if [[ "${#TRIGGERED_PATHS[@]}" -gt 0 ]]; then
# collect comment ids (best-effort) so the event log carries enough
# info to triangulate which findings drove the strike.
for path in "${TRIGGERED_PATHS[@]}"; do
if [[ -f "$COMMENTS_JSON" ]] \
&& jq empty "$COMMENTS_JSON" >/dev/null 2>&1 \
&& jq -e 'type == "array"' "$COMMENTS_JSON" >/dev/null 2>&1; then
IDS_JSON=$(jq --arg p "$path" '[.[]|select(.path==$p)|.id // empty]' "$COMMENTS_JSON" 2>/dev/null || echo '[]')
else
IDS_JSON='[]'
fi
UPDATED=$(jq \
--argjson r "$ROUND" \
--arg p "$path" \
--argjson ids "$IDS_JSON" \
'.events += [{round:$r, path:$p, comment_ids:$ids}]' \
"$HISTORY")
printf '%s\n' "$UPDATED" > "$HISTORY.tmp" && mv "$HISTORY.tmp" "$HISTORY"
done
fi
# --- 5. Apply the agent-stuck label if any path triggered ---
#
# Idempotent: gh label create is no-op when the label exists; gh pr
# edit --add-label is no-op when the PR already has it. We swallow
# benign "already exists" errors so a transient gh failure doesn't
# crash the loop (Rule 2 is monitor-only by design).
if [[ "${#TRIGGERED_PATHS[@]}" -gt 0 && -n "$PR" && "$GH_BIN" != "none" && -n "$GH_BIN" ]]; then
if ! command -v "$GH_BIN" >/dev/null 2>&1 && [[ ! -x "$GH_BIN" ]]; then
echo "round-post.sh: $GH_BIN not found; skipping label application" >&2
else
"$GH_BIN" label create "$LABEL" \
--repo "$REPO" \
--description "$LABEL_DESC" \
--color "$LABEL_COLOR" \
>/dev/null 2>&1 || true
"$GH_BIN" pr edit "$PR" \
--repo "$REPO" \
--add-label "$LABEL" \
>/dev/null 2>&1 || true
fi
fi
exit 0

264
.claude/skills/review-tileops/round-pre.sh
View File

@ -1,264 +0,0 @@
#!/usr/bin/env bash
# round-pre.sh — pre-codex hook for the review loop.
#
# Implements Rule 1 (same-SHA APPROVE skip): if any prior round in
# this loop run already produced an APPROVE on the *current* HEAD
# sha, skip the codex invocation entirely. Reuse the prior APPROVE
# verbatim by writing a marker round-NN.json that carries the same
# codex_event/blockers as the prior round and points back at it
# via the "approve_reused_from" field.
#
# Inputs (env or positional, env wins):
# RUN_DIR — review/ root for this loop run; must contain rounds/.
# NEXT_ROUND — integer round number about to run (e.g. 5).
# HEAD_SHA — full git sha of the PR HEAD this round would review.
# LATEST_ISSUE_ID — current max id of any non-reviewer top-level PR
# (issue) comment.
# LATEST_REVIEW_ID — current max id of any non-reviewer review/inline
# comment (incl. thread replies).
# Each is independent. When supplied, the reuse
# decision requires the prior approving round's
# counterpart watermark to match. A caller may
# supply only one — the unsupplied dimension is
# not constrained. If a NEW human comment of
# either kind landed on the same SHA after the
# prior APPROVE, the matched watermark mismatches
# and this hook returns "proceed" so codex re-
# reviews and ingests the comment. When BOTH are
# absent or empty, the watermark guard is skipped
# entirely (legacy behavior, used by tests that
# pre-date the watermark fields).
#
# Positional fallback:
# round-pre.sh <RUN_DIR> <NEXT_ROUND> <HEAD_SHA> [LATEST_ISSUE_ID] [LATEST_REVIEW_ID]
#
# Output:
# - exit 0 + stdout "skip" → loop should skip codex; round-NN.json is
# already written by this script.
# - exit 0 + stdout "proceed" → no prior APPROVE on this SHA, OR a
# recoverable condition was hit (missing
# args, missing rounds dir, malformed
# prior round file, etc.). The loop's
# ``|| echo "proceed"`` fallback expects
# this contract: round-pre.sh is a
# monitor and MUST NOT exit non-zero on
# recoverable conditions, otherwise
# corrupt state would be silently
# re-mapped to "proceed" and could
# re-enable a same-SHA re-review.
# - non-zero exit → reserved for truly unexpected errors
# (e.g. jq missing). Currently unused.
#
# Determinism: pure file scan. No gh / git / network. No LLM.
set -uo pipefail
RUN_DIR="${RUN_DIR:-${1:-}}"
NEXT_ROUND="${NEXT_ROUND:-${2:-}}"
HEAD_SHA="${HEAD_SHA:-${3:-}}"
LATEST_ISSUE_ID="${LATEST_ISSUE_ID:-${4:-}}"
LATEST_REVIEW_ID="${LATEST_REVIEW_ID:-${5:-}}"
if [[ -z "$RUN_DIR" || -z "$NEXT_ROUND" || -z "$HEAD_SHA" ]]; then
# Missing args is a recoverable misconfiguration: emit "proceed" so the
# loop continues with codex rather than failing fast. (See contract
# note above.)
echo "round-pre.sh: missing args; usage: RUN_DIR=... NEXT_ROUND=... HEAD_SHA=... round-pre.sh" >&2
echo "proceed"
exit 0
fi
ROUNDS_DIR="$RUN_DIR/rounds"
if [[ ! -d "$ROUNDS_DIR" ]]; then
# No prior rounds → nothing to reuse; loop proceeds normally.
echo "proceed"
exit 0
fi
# Scan prior round-*.json files for an APPROVE on the same head_sha_after.
# Pure jq filter; oldest-first sort iterates rounds chronologically.
#
# Selection contract: when LATEST_ISSUE_ID and/or LATEST_REVIEW_ID are
# provided, we MUST iterate ALL same-SHA APPROVE candidates and reuse
# the one whose recorded watermarks match every supplied LATEST_*_ID
# (per-dimension; unsupplied dimensions are unconstrained). Breaking
# on the first SHA match would cause this failure mode:
#
# round 5: APPROVE on sha=X, issue_wm=100, review_wm=50
# <human comments land; watermarks → 200, 60>
# round 6: re-APPROVE on sha=X, issue_wm=200, review_wm=60
# round 7: must skip, but old code found round 5 first, saw stale
# watermark=100 != 200, returned "proceed" — re-running codex
# even though round 6 is a perfectly reusable APPROVE.
#
# So: keep scanning. Each watermark dimension is independent — only the
# LATEST_*_ID values the caller actually supplied are required to match;
# unsupplied dimensions are unconstrained. If no watermarks are supplied
# (or no prior round recorded them) fall back to legacy SHA-only reuse
# using the earliest APPROVE.
PRIOR_FILE=""
PRIOR_ROUND=""
PRIOR_BLOCKERS=0
PRIOR_ISSUE_ID=""
PRIOR_REVIEW_ID=""
# Track the earliest same-SHA APPROVE as a fallback for the legacy
# (no-watermark) path so pre-watermark round files keep skipping.
FALLBACK_FILE=""
FALLBACK_ROUND=""
FALLBACK_BLOCKERS=0
FALLBACK_ISSUE_ID=""
FALLBACK_REVIEW_ID=""
# Use a sorted glob so iteration order is stable across filesystems.
# Iterate via while-read against a NUL-safe stream so paths with spaces
# or other shell-meta characters survive intact (paths come from the
# RUN_DIR caller-supplied root). Malformed JSON in any candidate file is
# treated as "not an approval" and skipped silently — corrupt round
# files must never crash the monitor; downstream consumers are
# responsible for surfacing them.
shopt -s nullglob
while IFS= read -r -d '' f; do
[[ -f "$f" ]] || continue
ev=$(jq -r '.codex_event // empty' "$f" 2>/dev/null) || ev=""
sha=$(jq -r '.head_sha_after // empty' "$f" 2>/dev/null) || sha=""
[[ "$ev" == "APPROVE" && "$sha" == "$HEAD_SHA" ]] || continue
cand_round=$(jq -r '.round // empty' "$f" 2>/dev/null) || cand_round=""
cand_blockers=$(jq -r '.blockers_after // 0' "$f" 2>/dev/null) || cand_blockers=0
# Backward-compat: pre-namespace-split round files only carry
# last_human_comment_id; treat it as the issue-comment seed (the legacy
# max() across both endpoints was dominated by issue ids in practice).
cand_issue_id=$(jq -r '.last_issue_comment_id // .last_human_comment_id // empty' "$f" 2>/dev/null) || cand_issue_id=""
cand_review_id=$(jq -r '.last_review_comment_id // empty' "$f" 2>/dev/null) || cand_review_id=""
# Defensive numeric validation: --argjson rejects non-JSON-numeric
# values, which would crash marker generation below. A corrupt prior
# round file (e.g. blockers_after: "many", round: null) must not
# break the monitor — coerce to safe defaults so the marker still
# writes and reuse can proceed. Empty strings from `// empty` above
# are normalized here too.
[[ "$cand_round" =~ ^[0-9]+$ ]] || cand_round=0
[[ "$cand_blockers" =~ ^[0-9]+$ ]] || cand_blockers=0
[[ "$cand_issue_id" =~ ^[0-9]+$ ]] || cand_issue_id=""
[[ "$cand_review_id" =~ ^[0-9]+$ ]] || cand_review_id=""
# Remember the earliest same-SHA APPROVE for the legacy fallback.
if [[ -z "$FALLBACK_FILE" ]]; then
FALLBACK_FILE="$f"
FALLBACK_ROUND="$cand_round"
FALLBACK_BLOCKERS="$cand_blockers"
FALLBACK_ISSUE_ID="$cand_issue_id"
FALLBACK_REVIEW_ID="$cand_review_id"
fi
# Watermark-aware reuse: pick this candidate only if every watermark
# the caller actually supplied matches its recorded counterpart. A
# caller that provides only LATEST_ISSUE_ID does not constrain the
# review-id dimension, so we must NOT silently require the recorded
# review id to be 0. Continue scanning otherwise so a later same-SHA
# APPROVE with fresher watermarks can still win.
ok=true
if [[ -n "$LATEST_ISSUE_ID" \
&& "${cand_issue_id:-0}" != "$LATEST_ISSUE_ID" ]]; then
ok=false
fi
if [[ -n "$LATEST_REVIEW_ID" \
&& "${cand_review_id:-0}" != "$LATEST_REVIEW_ID" ]]; then
ok=false
fi
if [[ ( -n "$LATEST_ISSUE_ID" || -n "$LATEST_REVIEW_ID" ) \
&& "$ok" == "true" ]]; then
PRIOR_FILE="$f"
PRIOR_ROUND="$cand_round"
PRIOR_BLOCKERS="$cand_blockers"
PRIOR_ISSUE_ID="$cand_issue_id"
PRIOR_REVIEW_ID="$cand_review_id"
break
fi
done < <(find "$ROUNDS_DIR" -maxdepth 1 -type f -name 'round-*.json' -print0 2>/dev/null | sort -z)
# Resolve which candidate (if any) wins.
if [[ -z "$PRIOR_FILE" ]]; then
if [[ -z "$FALLBACK_FILE" ]]; then
# No same-SHA APPROVE at all → nothing to reuse.
echo "proceed"
exit 0
fi
if [[ -n "$LATEST_ISSUE_ID" || -n "$LATEST_REVIEW_ID" ]]; then
# Caller supplied watermarks and NO same-SHA APPROVE matched them.
# That means a fresh human comment (issue or review) has landed
# since every prior APPROVE on this SHA — codex must re-review.
echo "proceed"
exit 0
fi
# Legacy path: no watermarks from caller → reuse the earliest match.
PRIOR_FILE="$FALLBACK_FILE"
PRIOR_ROUND="$FALLBACK_ROUND"
PRIOR_BLOCKERS="$FALLBACK_BLOCKERS"
PRIOR_ISSUE_ID="$FALLBACK_ISSUE_ID"
PRIOR_REVIEW_ID="$FALLBACK_REVIEW_ID"
fi
# Found a prior APPROVE on this SHA. Emit a marker round-NN.json that
# reuses the prior outcome and clearly notes the reuse so postmortem
# tools can distinguish skipped rounds from genuinely-reviewed rounds.
N=$(printf '%02d' "$NEXT_ROUND")
MARKER="$ROUNDS_DIR/round-$N.json"
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
# Carry the watermarks forward so a downstream reuse decision (or
# postmortem) can tell whether new human comments have landed since the
# *original* approving round. Use the caller-provided LATEST_*_ID when
# present; otherwise inherit the prior round's recorded watermark.
MARKER_ISSUE_ID="${LATEST_ISSUE_ID:-${PRIOR_ISSUE_ID:-0}}"
MARKER_REVIEW_ID="${LATEST_REVIEW_ID:-${PRIOR_REVIEW_ID:-0}}"
[[ -z "$MARKER_ISSUE_ID" ]] && MARKER_ISSUE_ID=0
[[ -z "$MARKER_REVIEW_ID" ]] && MARKER_REVIEW_ID=0
# Final numeric validation before --argjson. Any of the values below
# could still be non-numeric here (e.g. caller-supplied NEXT_ROUND or
# LATEST_*_ID came in malformed). Coerce or bail to "proceed" so the
# loop falls back to a real codex run rather than silently emitting
# "skip" without writing a marker.
[[ "$NEXT_ROUND" =~ ^[0-9]+$ ]] || { echo "round-pre.sh: NEXT_ROUND not numeric ($NEXT_ROUND); proceeding" >&2; echo "proceed"; exit 0; }
[[ "$PRIOR_BLOCKERS" =~ ^[0-9]+$ ]] || PRIOR_BLOCKERS=0
[[ "$PRIOR_ROUND" =~ ^[0-9]+$ ]] || PRIOR_ROUND=0
[[ "$MARKER_ISSUE_ID" =~ ^[0-9]+$ ]] || MARKER_ISSUE_ID=0
[[ "$MARKER_REVIEW_ID" =~ ^[0-9]+$ ]] || MARKER_REVIEW_ID=0
if ! jq -n \
--argjson r "$NEXT_ROUND" \
--arg now "$NOW" \
--arg sha "$HEAD_SHA" \
--arg ev "APPROVE" \
--argjson bl "$PRIOR_BLOCKERS" \
--argjson prior_round "$PRIOR_ROUND" \
--argjson iid "$MARKER_ISSUE_ID" \
--argjson rid "$MARKER_REVIEW_ID" \
'{round:$r, finished_at:$now,
head_sha_before:$sha, head_sha_after:$sha,
codex_event:$ev, blockers_after:$bl,
last_issue_comment_id:$iid,
last_review_comment_id:$rid,
approve_reused_from: $prior_round,
skipped_codex: true}' \
> "$MARKER" 2>/dev/null; then
# Marker generation failed (jq error, disk full, etc.). Without a
# marker we MUST NOT emit "skip" — the loop would skip codex *and*
# have no recorded round file, leaving the postmortem trail broken.
# Force a real codex re-run instead.
rm -f "$MARKER" 2>/dev/null || true
echo "round-pre.sh: marker generation failed; forcing codex re-run" >&2
echo "proceed"
exit 0
fi
# Defensive: if jq exited 0 but the marker is missing or empty, also
# fall back to "proceed" (e.g. redirection failure on a read-only FS).
if [[ ! -s "$MARKER" ]]; then
rm -f "$MARKER" 2>/dev/null || true
echo "round-pre.sh: marker missing/empty after jq; forcing codex re-run" >&2
echo "proceed"
exit 0
fi
echo "skip"
exit 0

120
.claude/skills/review-tileops/signals.sh
View File

@ -1,120 +0,0 @@
# signals.sh — pure helpers that compute the per-poll "fresh round" signal.
# Sourced, never executed; no shebang on purpose.
#
# The review loop fires a fresh round whenever the PR's externally
# observable state has materially changed. Sourced by `loop.sh`.
# Pure: no gh / git / network calls; callers pass the raw GitHub
# fields in.
#
# Public functions:
#
# sha256_text <text>
# Print the lowercase sha256 of <text> (truncated to 16 hex chars to
# keep meta.json compact). Single-line output, no newline-terminated
# trailing bytes via printf '%s'.
#
# pr_body_hash <body>
# Stable hash of a PR body. Normalizes CRLF to LF so a body edited on
# GitHub web (CRLF) and locally (LF) compares equal when textually
# identical. Empty body → fixed sentinel hash of empty string.
#
# pr_labels_hash <labels_json_array>
# Stable hash of a labels set. <labels_json_array> is the raw
# `.labels` field from `gh pr view --json labels` — a JSON array of
# `{name,...}` objects. Hash is order-independent (labels are a set,
# not a list).
#
# signature_diff_reason \
# <head_now> <head_prev> \
# <body_now> <body_prev> \
# <labels_now> <labels_prev> \
# <issue_id_now> <issue_id_prev> \
# <review_id_now> <review_id_prev> \
# <inbox_present>
# Print a single short reason string naming which signal fired, or
# the empty string if no signal changed. Reasons are stable tokens
# (used in log lines and asserted by tests):
# - "head changed"
# - "body changed"
# - "labels changed"
# - "issue comment"
# - "review comment"
# - "inbox prompt"
# When multiple signals change in one tick, the first one in the
# priority order above wins — every signal still triggers exactly
# one round, so picking one stable token avoids double-firing.
set -uo pipefail
sha256_text() {
# printf '%s' avoids the implicit trailing newline of echo which would
# otherwise make `sha256_text ""` differ from a true-empty hash.
# Prefer GNU `sha256sum`; fall back to BSD/macOS `shasum -a 256`.
local out
if command -v sha256sum >/dev/null 2>&1; then
out=$(printf '%s' "$1" | sha256sum)
else
out=$(printf '%s' "$1" | shasum -a 256)
fi
printf '%s' "$out" | cut -c1-16
}
pr_body_hash() {
local body="${1-}"
# Normalize CRLF → LF so web-edit and CLI-edit bodies compare equal.
body="${body//$'\r'/}"
sha256_text "$body"
}
pr_labels_hash() {
local labels_json="${1:-[]}"
# Sort label names so the hash is order-independent. Serialize as
# compact JSON (not join(",")) so a label name containing a comma
# cannot collide with two labels split on commas. `jq -e` would exit
# non-zero on empty input; default to "[]" so the empty-labels case
# yields a stable, deterministic hash.
#
# ``agent-stuck`` is loop-owned (round-post.sh applies it); excluding
# it keeps the loop from observing its own write as a fresh label
# signal on the next poll.
local sorted
sorted=$(printf '%s' "$labels_json" \
| jq -c 'if type=="array" then [.[].name | select(. != "agent-stuck")] | sort else [] end' \
2>/dev/null) || sorted="[]"
sha256_text "$sorted"
}
signature_diff_reason() {
local head_now="$1" head_prev="$2"
local body_now="$3" body_prev="$4"
local labels_now="$5" labels_prev="$6"
local issue_now="$7" issue_prev="$8"
local review_now="$9" review_prev="${10}"
local inbox_present="${11:-0}"
if [[ "$head_now" != "$head_prev" ]]; then
printf '%s' "head changed"
return 0
fi
if [[ "$body_now" != "$body_prev" && -n "$body_prev" ]]; then
printf '%s' "body changed"
return 0
fi
if [[ "$labels_now" != "$labels_prev" && -n "$labels_prev" ]]; then
printf '%s' "labels changed"
return 0
fi
if [[ "$issue_now" != "$issue_prev" ]]; then
printf '%s' "issue comment"
return 0
fi
if [[ "$review_now" != "$review_prev" ]]; then
printf '%s' "review comment"
return 0
fi
if [[ "$inbox_present" -eq 1 ]]; then
printf '%s' "inbox prompt"
return 0
fi
printf '%s' ""
}

207
.claude/skills/scaffold-op/SKILL.md
View File

@ -1,207 +0,0 @@
---
name: scaffold-op
description: Scaffold a new T2 (L1-direct) Op file from a single `tileops/manifest/` entry by following the 7-step playbook in docs/design/ops-design.md. Emits the 17 scaffold slots (S1-S7, S12-S21); leaves family-specific protocol variables, optional hooks, and kernel implementations to downstream skills.
---
## Arguments
`op_name` (positional) — manifest key for the op to scaffold, equal to the target `cls.__name__` (e.g. `CumsumFwdOp`).
## Contract
- **Input**: `op_name` must be present in [`tileops/manifest/`](../../../tileops/manifest/) with `status: spec-only` and a non-empty `source.kernel_map`. `source.kernel_map` is manifest-level source of truth for Op→Kernel dispatch and cannot be derived by the scaffold (dispatch keys are kernel-internal conventions); adding it for a spec-only entry is a prerequisite manifest PR.
- **Output**: new file at the exact path declared by manifest `source.op` (e.g., `tileops/ops/reduction/cumsum.py`), containing the 17 scaffold slots; one-line `from .<module> import <ClassName>` added to the package `__init__.py` at that path's parent directory (e.g., `tileops/ops/reduction/__init__.py`) with a matching `__all__` entry. Note: the filesystem package directory (parent of `source.op`) is not always the same as the manifest `family` field — for example, `CumsumFwdOp` has `family: scan` but lives under `tileops/ops/reduction/`. Always key paths off `source.op`, never off `family`. Plus a side-artefact at `.foundry/plan/<op_name>/plan.json` carrying the DRY_RUN self-audit (not tracked in git).
- **Termination (success)**: `python scripts/validate_manifest.py --check-op <op_name>` reports **no errors** for this op. Warnings are allowed and passed through to the final summary.
- **Termination (blocked)**: any validator error for `op_name` that the scaffold cannot fix by re-reading the playbook's slot rules. Do NOT commit; report with the failing rows from the validator.
- **Constraints**:
- MUST NOT emit family-specific protocol variables (`_op_kind`, `_kernel_key`, `_kernel_cls`, `_kernel_handles_padding`, `_op_name`, `kernel_cls`).
- MUST NOT emit optional hooks (`_pad_value`, `_validate_dim`, `_pre_kernel`, `_post_kernel`, `_cache_key` override).
- MUST NOT implement the kernel itself.
- MUST NOT modify `tileops/manifest/`, tests, benchmarks, or any existing op file.
- MUST NOT extend scope to a T1 (family-base) subclass — the scaffold is T2 only.
## Workflow
```mermaid
stateDiagram-v2
[*] --> READ
READ --> PRE_CHECK
PRE_CHECK --> DRY_RUN
PRE_CHECK --> BLOCKED: precondition failed
DRY_RUN --> EMIT: plan.json written
EMIT --> REGISTER
REGISTER --> VALIDATE
VALIDATE --> REPORT
VALIDATE --> BLOCKED: validator error or §1 drift
REPORT --> [*]
BLOCKED --> [*]
```
`DRY_RUN` writes `plan.json` to freeze manifest-sourced facts before codegen. `VALIDATE` diffs the emitted file against `plan.json` §1 — any drift is a skill bug, not a manifest issue.
## Slot scope
Emit exactly the 17 slots in [`ops-design-reference.md § Slot Rules`](../../../docs/design/ops-design-reference.md#slot-rules): S1–S7, S12–S21. S8–S11 are reserved for T1 thin-wrapper subclasses and skipped.
Out of scope — leave empty:
| Item | Reason |
| ----------------------------------------------------------------------------- | -------------------------------------------------------- |
| Family protocol vars (`_op_kind`, `_kernel_key`, `_op_name`, …) | Kernel-dispatch convention; not in manifest |
| Optional hooks (`_pad_value`, `_validate_dim`, `_pre_kernel`, `_post_kernel`) | Op-specific business logic |
| `_cache_key` override | Recommended under dynamic shapes; depends on kernel math |
| Kernel implementations | Owned by kernel skill |
| Tests / benchmarks | Owned by `test-op` / `bench-op` |
These gaps surface as `NotImplementedError` or validator warnings; downstream skills fill them.
## Steps
### 1. READ
Load the manifest entry for `op_name`:
Before running the snippet, substitute `<op_name>` with the requested manifest key (the skill's positional argument — agent literal substitution, not shell interpolation):
```bash
python - "<op_name>" <<'PY'
import sys
from tileops.manifest import load_manifest
op_name = sys.argv[1]
entry = load_manifest()[op_name]
print(entry)
PY
```
Extract: `family`, `status`, `signature.inputs`, `signature.outputs`, `signature.params`, `signature.static_dims`, `signature.shape_rules`, `source.kernel_map`, `source.op`, `source.kernel`, `roofline.vars`, `roofline.flops`, `roofline.bytes`.
Derive the target file path from `source.op` (e.g. `tileops/ops/reduction/cumsum.py`). The **filesystem package directory** is `source.op`'s parent (e.g. `tileops/ops/reduction/`). Do not use the manifest `family` field to compute paths — it is a semantic label, and some ops have `family` distinct from their filesystem parent (e.g., `CumsumFwdOp` has `family: scan` but lives under `reduction/`). Module filename is `source.op`'s basename without `.py`.
### 2. PRE_CHECK
- `op_name` present in `tileops/manifest/` → proceed; otherwise BLOCKED ("op not in manifest").
- `status` field explicitly set to `spec-only` → proceed; `status: implemented` → BLOCKED ("op already implemented; use implement-op to migrate"); missing `status` or any other value → BLOCKED ("manifest entry must declare a valid top-level `status`; the validator treats `status` as required").
- `source.kernel_map` declared and non-empty → proceed; missing or empty → BLOCKED ("manifest entry needs `source.kernel_map` before scaffolding — add the dispatch map in a separate manifest PR per the trust model; the scaffold cannot invent dispatch keys because they are kernel-internal conventions"). Note: per `docs/design/manifest.md`, `source.kernel_map` is only required when `status: implemented`, so many existing `spec-only` entries lack it — these are the cases that need the manifest-PR prerequisite before scaffolding can run.
- Every value in `source.kernel_map` resolves to an importable symbol → proceed; otherwise BLOCKED ("kernel class not found at expected path").
- Target file `source.op` does NOT exist → proceed; exists → BLOCKED ("target file already present; scaffold would overwrite").
BLOCKED terminations return without writing any file.
### 3. DRY_RUN
Write `.foundry/plan/<op_name>/plan.json` with three sections:
| Section | Diffed at VALIDATE? | Content |
| --------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `locked_facts` (§1) | yes (hard error on drift) | Verbatim manifest extraction (op_name, class_name, family, module_path, kernel_imports, kernel_map, init_kwargs, forward_inputs/outputs, dtype_unions, dtype_combos, shape_rules, static_dims, roofline) |
| `agent_notes` (§2) | no | Judgement calls (docstring, kernel ctor signature observed, helper state, forward reshape strategy, codebase refs consulted) |
| `open_questions` (§3) | no | Ambiguities tagged `needs_doc_fix` / `needs_manifest_fix` / `needs_human_decision`; surfaced in REPORT, never block |
Always proceed to EMIT. Empty `open_questions` is fine.
Skeleton:
```json
{
"locked_facts": {
"op_name": "CumsumFwdOp",
"module_path": "tileops/ops/reduction/cumsum.py",
"kernel_map": {"cumulative_fwd": "CumulativeKernel"},
"init_kwargs": [{"name": "dim", "source": "signature.params.dim", "type": "int", "default": -1}],
"forward_inputs": ["x"],
"static_dims": {"N": "x.shape[dim]"},
"roofline": {"flops": "M * N", "bytes": "2 * M * N * elem_bytes"}
},
"agent_notes": {"docstring_summary": "...", "kernel_ctor_signature_observed": "..."},
"open_questions": [{"tag": "needs_human_decision", "topic": "...", "detail": "..."}]
}
```
### 4. EMIT
Follow [`docs/design/ops-design.md` § Scaffolding an Op from a Manifest Entry](../../../docs/design/ops-design.md#scaffolding-an-op-from-a-manifest-entry) Steps 1-7 in order. For each scaffold slot, read the authoritative rule at `docs/design/ops-design-reference.md#slot-sN` before emitting.
Key slot pointers (follow the reference, do not re-derive):
| Playbook step | Slots | Reference anchor |
| ------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Step 1 | S1, S2, S3, S4 | [S1](../../../docs/design/ops-design-reference.md#slot-s1)-[S4](../../../docs/design/ops-design-reference.md#slot-s4) |
| Step 2 | S5, S6, S7 | [S5](../../../docs/design/ops-design-reference.md#slot-s5)-[S7](../../../docs/design/ops-design-reference.md#slot-s7) |
| Step 3 | S21, S12, S13 | [S21](../../../docs/design/ops-design-reference.md#slot-s21), [S12](../../../docs/design/ops-design-reference.md#slot-s12), [S13](../../../docs/design/ops-design-reference.md#slot-s13) |
| Step 4 | S14, S15, S16 | [S14](../../../docs/design/ops-design-reference.md#slot-s14)-[S16](../../../docs/design/ops-design-reference.md#slot-s16) |
| Step 5 | S17, S18 | [S17](../../../docs/design/ops-design-reference.md#slot-s17), [S18](../../../docs/design/ops-design-reference.md#slot-s18) |
| Step 6 | S19 | [S19](../../../docs/design/ops-design-reference.md#slot-s19) |
| Step 7 | S20 | [S20](../../../docs/design/ops-design-reference.md#slot-s20) |
If a slot's rule is ambiguous for the given manifest entry (e.g. multi-kernel `kernel_map`, multiple independent dtype axes, fixed-rank vs arbitrary-rank branching), STOP and surface the ambiguity in the final report instead of guessing. Do not expand scope.
### 5. REGISTER
Append to the `__init__.py` at `dirname(source.op)`:
| File style | Action |
| ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| Grouping comments `# --- <KernelClassName> ops ---` (e.g. `tileops/ops/reduction/__init__.py`) | Place import under matching kernel block; create a new block (alphabetical) when no existing block references the kernel |
| Flat imports (e.g. `tileops/ops/norm/__init__.py`) | Append alongside existing imports following file's ordering convention; do NOT introduce grouping comments |
Add matching `<ClassName>` to `__all__`, preserving any existing sectioning.
### 6. VALIDATE
**(a) §1 post-check** — diff `plan.json.locked_facts` against the emitted artefacts:
- Parse `source.op` with `ast`: extract class name, base, imports, `__all__`, `__init__` kwargs (names / defaults / types in order), `forward` params, `default_kernel_map` dict, `_static_axes`.
- Parse `dirname(source.op)/__init__.py` with both `ast` (presence + `__all__` membership) and raw text (block placement under `# --- <KernelClassName> ops ---` when the file uses grouping comments; skip placement check on flat-style files).
- Any `locked_facts` field mismatch → BLOCKED `§1 drift: <field>`. Skill deviated from its own contract; fix the emitted file (or revert and restart DRY_RUN if the plan was wrong). Do NOT edit `plan.json` to match.
**(b) Manifest validator**:
```bash
python scripts/validate_manifest.py --check-op <op_name>
```
`--check-op` runs L0–L4 even on `status: spec-only`. Classify output:
- **ERROR** → BLOCKED. Copy the row to REPORT. For L2 / L3 parity failures, re-check the emitted `_infer_output_shapes` / `_validate_dtypes` against manifest `shape_rules` / `dtype_combos` first — likely a mis-emit.
- **WARNING** → pass through.
Never edit the manifest to silence an error. No `parity_opt_out` escape hatch. Demote to `status: spec-only` only when the implementation genuinely cannot conform.
### 7. REPORT
```
Status: SUCCESS | BLOCKED
Op: <op_name>
File: <path> (<lines>)
Package registration: <parent-of-source.op>/__init__.py (+1 import, +1 __all__)
Plan: .foundry/plan/<op_name>/plan.json
§1 drift: <none | <field>: <plan> vs <emitted>>
Validator: <N errors, M warnings>
<if BLOCKED, list each blocking error row verbatim>
Warnings:
- <warning>
Open questions (from plan.json §3):
needs_doc_fix: - <topic>: <detail>
needs_manifest_fix: - <topic>: <detail>
needs_human_decision: - <topic>: <detail>
Not filled (downstream hand-off):
- Kernel implementation
- Optional hooks (family-specific)
- Family protocol variables (family-specific)
- _cache_key override (recommended under dynamic shapes)
```
SUCCESS → commit `[Feat][OPS] scaffold <op_name>`. BLOCKED → leave tree dirty, return. `plan.json` persists either way.
## Non-goals
- Not a codegen engine — agent procedure. The input/output contract is codegen-compatible.
- Not a migration driver — `status: implemented` ops use `implement-op` / `align-family`.
- Not a kernel scaffold.
If a slot rule is internally inconsistent or cannot be followed mechanically, file a doc issue and BLOCK; do not invent slot behaviour.

74
.claude/skills/test-op/SKILL.md
View File

@ -1,74 +0,0 @@
---
name: test-op
description: Write tests for the target spec using PyTorch as ground truth, verify they fail on current code.
---
## Arguments
`op_name`, `manifest_signature`, `pytorch_equivalent`, `source_test` — passed by align-family orchestrator.
## Contract
- **Input**: `op_name`, `manifest_signature`, `pytorch_equivalent`, `source_test`
- **Output**: updated test file + commit
- **Constraint**: must NOT modify op implementation. Test-only.
- **Trust model**: this agent must be a different invocation from implement-op.
## Workflow
```mermaid
stateDiagram-v2
[*] --> READ_SPEC
READ_SPEC --> ASSESS_EXISTING: manifest signature + pytorch_equivalent loaded
ASSESS_EXISTING --> WRITE_TESTS_EXTEND: semantic extension — old assertions still valid
ASSESS_EXISTING --> WRITE_TESTS_REPLACE: semantic update — old assertions incompatible
WRITE_TESTS_EXTEND --> VERIFY_FAILS: new tests added, existing kept
WRITE_TESTS_REPLACE --> VERIFY_FAILS: outdated tests deleted, new tests written
VERIFY_FAILS --> DONE: confirmed failing on current code
VERIFY_FAILS --> DONE_SKIP: tests already pass (base class fixed by previous op)
```
## Steps
### 1. READ_SPEC
Read `manifest_signature` to determine target interface:
- `signature.inputs` → `forward()` params (tensor inputs)
- `signature.params` → `__init__()` params (configuration)
- This follows Op design convention in `docs/design/ops-design.md`. The manifest is the source of truth.
### 2. ASSESS_EXISTING
Read current test file (`source_test`). Compare existing test construction and assertions against the new spec:
- Old assertions still valid under new spec → **semantic extension** (keep existing tests, add new ones)
- Old assertions incompatible (e.g., construction API changes from `Op(M, N)` to `Op(dim)`) → **semantic update** (delete outdated tests, write replacements)
### 3. WRITE_TESTS
Write tests using PyTorch reference as ground truth:
```python
# Example — derive from manifest, don't copy this literally
expected = torch.nn.functional.softmax(x, dim=dim)
actual = op(x)
torch.testing.assert_close(actual, expected, rtol=rtol, atol=atol)
```
- Use `TestBase` pattern (`gen_inputs()` + `ref_program()` + `check()`). Follow `docs/design/testing.md`.
- Write tests in `source_test` file. No new files.
- For integer outputs (manifest `outputs.*.dtype` is int type), use `torch.equal` for exact comparison.
- Parameterize: supported dtypes (FP16, BF16), representative dim values, keepdim True/False where applicable.
### 4. VERIFY_FAILS
Run the new tests against current code:
```bash
python -m pytest <source_test> -v
```
New tests must **fail** on current code. Construction-time error counts (e.g., current `__init__` doesn't accept `dim`).
**DONE_SKIP**: if tests already pass (base class fixed by a previous op's migration), this is valid. Proceed to implement-op.

33
.foundry/config.json
View File

@ -1,33 +0,0 @@
{
"scripts": {
"lint": "pre-commit run --all-files"
},
"pr": {
"ai_reviews": [
{"name": "gemini", "kind": "comment", "body": "/gemini review"},
{"name": "copilot", "kind": "copilot"}
]
},
"knowledge": {
"creating_issue": ["docs/design/trust-model.md"],
"planning_issue": ["docs/design/trust-model.md"],
"developer": null,
"reviewer": [
".claude/rules/manifest-trust-model.md",
".claude/rules/code-style.md",
".claude/rules/security.md",
".claude/domain-rules/manifest-spec.md",
".claude/domain-rules/manifest-validator.md",
".claude/domain-rules/ops-design.md",
".claude/domain-rules/benchmark.md",
".claude/domain-rules/testing-budget.md",
".claude/domain-rules/design-docs.md",
"docs/design/trust-model.md"
],
"gatekeeper": [
".claude/rules/manifest-trust-model.md",
".claude/rules/security.md",
"docs/design/trust-model.md"
]
}
}

68
.foundry/mold/benchmark-template.md
View File

@ -1,68 +0,0 @@
<!--
INSTRUCTIONS FOR THE AGENT (do not copy into the PR body).
Layout principle: one section per op, one table per section, TileOPs and
baseline side-by-side on the same row so readers can compare without
mentally joining two tables.
Filling in the template below:
- One row per measurement (shape × dtype) within an op's table.
- Baseline column header names the baseline (`torch (ms)`, `FA3 (ms)`,
`triton (ms)`). Don't write a generic "Baseline". Multiple baselines →
add more columns and more Speedup columns (`vs torch`, `vs FA3`).
- Speedup is always present — it's the first number readers look for.
Format `4.96×`, two decimals, computed as baseline_ms / tileops_ms.
- Throughput column: show ONE — TFLOPS for compute-bound ops (matmul,
attention), BW (TB/s) for memory-bound ops (reductions, elementwise,
norms). Pure data movement → BW only. Don't list both.
- Show throughput for TileOPs only; baseline's absolute throughput is
noise once Speedup is given.
- Drop the Shape column if the op only varies dtype; in that case put
the fixed shape in the section header (e.g. `### {OpName} (4096, 4096)`)
so the benchmark's scale stays visible. Never put autotune config
(`block_m`, `threads`) in the table — implementation detail.
- **Preserve the original shape tuple — never flatten to a single
element count.** Write `(4096, 4096)` for a 2D input, `(2, 4096, 128)`
for a 3D one, and `(4194304,)` (or `(4M,)`) for a genuinely 1D op.
A bare `4M` loses dimensionality: readers can't tell `(4M,)` from
`(2K, 2K)` from `(64, 64K)`, yet those have very different access
patterns. Use `K`/`M` only inside a tuple to keep large numbers
readable, never to replace it.
- Environment block goes once at the top, not per op.
- Takeaways = conclusions, not data repetition. Wins, losses with a brief
reason (not blocking), dtype/shape patterns.
Bench-file authoring rules (apply when writing benchmarks/ops/*.py, NOT
copied into the PR body):
- All dtypes in `SUPPORTED_DTYPES`; ≥3 shapes per op, include non-pow2
if supported.
- Shapes must map to real model geometry. Default LLaMA sizes:
hidden ∈ {4096, 5120, 8192}, intermediate ∈ {10240, 11008, 14336,
20480, 28672}, seq_len ∈ {2048, 4096}.
- Every benchmark must record at least one baseline. Tags: `tileops`
(TileOPs implementation; required exactly once per config; variants
like `tileops-lut` also count) and `torch` / `FA3` / `fla` / `triton`
(baselines for comparison). External baselines may be conditional,
but the else branch must fall back to torch — never silently skip.
- `BenchmarkReport.record()` first arg must be the Op object, never a
string literal.
-->
## Benchmark
**Environment**: \{GPU}, CUDA \{ver}, PyTorch \{ver}, TileLang \{ver}
### \{OpName}
| Shape | dtype | TileOPs (ms) | \{baseline} (ms) | Speedup | BW (TB/s) |
| ----- | ----- | ------------ | ---------------- | ------- | --------- |
<!-- Repeat one ### section per op. Drop the Shape column if the op only
varies dtype. Swap `BW (TB/s)` for `TFLOPS` on compute-bound ops.
Shape values are tuples: `(4096, 4096)`, `(2, 4096, 128)`, `(4M,)`. -->
**Takeaways:** {wins · losses with brief reason · dtype/shape patterns}
**Command:** `PYTHONPATH="$PWD" python -m pytest benchmarks/ops/bench_{op}.py -v`

49
.foundry/mold/body-sections.md
View File

@ -1,49 +0,0 @@
# Issue Body Sections
Structural contract for the issue body — the template and per-section rules below are what the pipeline parses. Constraints authoring policy (work-shape declarations, defaults) lives in [docs/design/trust-model.md §Issue-authoring](../../docs/design/trust-model.md#issue-authoring-declaring-scope).
## 1. Template
Copy verbatim. Replace each `{...}`. Keep all five top-level sections.
```markdown
## Description
### Symptom / Motivation
{what is observed or motivates the change}
### Root Cause Analysis
{file paths, logic errors, missing features — "N/A" for feature requests}
### Related Files
{key files, functions, or configs}
## Goal
{concrete objective}
## Plan
<!-- type: {proposal | fixed} -->
1. {at least one step}
## Constraints
- {one bullet per constraint or scope declaration — see trust-model.md §Issue-authoring for the three work-shape forms}
## Acceptance Criteria
- [ ] Modified files pass unit tests
- [ ] {additional criteria as needed}
```
## 2. Per-section rules
| Section | Required form |
| ------------------- | -------------------------------------------------------------------------------------------------- |
| Description | Three subsections (`Symptom / Motivation`, `Root Cause Analysis`, `Related Files`), each non-empty |
| Goal | Non-empty single-paragraph objective |
| Plan | `<!-- type: -->` comment (value `proposal` or `fixed`) plus at least one step (`- ` or `1.`) |
| Constraints | At least one list item (`- `) |
| Acceptance Criteria | At least one checkbox, including `- [ ] Modified files pass unit tests` |
## 3. Cross-references
- Constraints authoring policy: [docs/design/trust-model.md §Issue-authoring](../../docs/design/trust-model.md#issue-authoring-declaring-scope)
- Reviewer-side criteria: [.claude/review-checklists/pre-review.md](../../.claude/review-checklists/pre-review.md)

31
.foundry/mold/pr-body-template.md
View File

@ -1,31 +0,0 @@
<!--
KEEP IT SHORT. Reviewers read PRs to learn WHAT changed and HOW to verify it.
- Summary: 3–5 bullets, one line each. No prose paragraphs.
- Do NOT recount the development process (reviewer rounds, follow-up issues filed,
prior commits reverted, design rationale that already lives in the linked issue).
- Do NOT restate the linked issue's motivation; "Closes #N" already links it.
- One short paragraph of cross-cutting context is fine when bullets can't carry it.
- If a section doesn't apply, delete the header — never leave empty sections.
-->
Closes #\{issue-number}
## Summary
- {what was added/fixed/changed}
- {what was removed/replaced}
## Test plan
- [x] pre-commit passed
- [x] pytest passed
<!-- Delete inapplicable optional sections entirely. Never leave empty headers. -->
## Benchmark
<!-- Required for kernel/op changes. Format: benchmark-template.md -->
## Regression
<!-- Recommended for bugfix/refactor -->

41
.foundry/mold/pre-create-checks.md
View File

@ -1,41 +0,0 @@
# Pre-Create Checks
## 1. PR title pre-flight validation
**HARD GATE.** Validate the PR title locally against the same source of truth that CI uses, **before** calling `gh pr create`. This guarantees the `validate-pr-title` required check cannot fail.
```bash
source .claude/conventions/types.sh
TITLE="[{{Type}}] {{Description}}" # substitute your actual title
if [[ ! "$TITLE" =~ $COMMIT_MSG_PATTERN ]]; then
echo "BLOCKED: title does not match CI pattern: $TITLE"
echo "Pattern: $COMMIT_MSG_PATTERN"
fi
```
**If validation fails:** fix the title and re-validate. Do NOT proceed to `gh pr create`.
## 2. Test node delta
**Skip entirely** if PR does not modify files under `tests/`.
If PR adds or modifies test files:
```bash
git fetch upstream main --quiet
python scripts/test_node_delta.py --base upstream/main
```
The script auto-detects changed test files via `git diff`. If auto-detect fails (e.g. in a worktree), pass files explicitly:
```bash
python scripts/test_node_delta.py --base upstream/main tests/ops/test_<name>.py
```
**Interpreting output:**
- **No growth on existing files** → nothing to report.
- **Growth on existing files** → include script output and one-line justification in PR body under `## Test node delta`.
- **New test files only** (delta shows "all N nodes from new files") → no delta report needed.
**SOFT GATE:** Does not block PR creation, but missing justification will be flagged during review (per [`.claude/domain-rules/testing-budget.md`](../../.claude/domain-rules/testing-budget.md)).

10
.foundry/mold/title-format.md
View File

@ -1,10 +0,0 @@
# Issue Title Format
**Format:** `[TYPE][COMPONENT] short description in lowercase`
- **TYPE**: FEAT | BUG | PERF | REFACTOR | DOCS | TEST | META | BENCHMARK — canonical list in `.claude/conventions/types.sh`
- **COMPONENT**: mandatory — kernel name or subsystem (GEMV, GEMM, FLASH_ATTN, CI, TOOLING, …)
- Max 80 characters total, description in lowercase
- Do NOT use Conventional Commits style (`feat(scope): …`)
Examples: `[FEAT][GEMV] add batched forward pass` · `[BUG][FLASH_ATTN] fix bf16 overflow in softmax` · `[PERF][GEMM] optimize shared memory layout for H100`

211
.foundry/scripts/run-affected-tests.sh
View File

@ -1,211 +0,0 @@
#!/bin/bash
# Run tests and benchmarks specified in a context.json file
# Usage: run-affected-tests.sh <context.json>
#
# Reads test_targets and bench_targets from the JSON file,
# executes each via pytest, and outputs structured JSON results.
#
# Designed to be called by lifecycle-issue-fixer and lifecycle-pull-request skills.
set -euo pipefail
# --- Argument validation ---
# Check jq availability first (before any jq usage)
if ! command -v jq &>/dev/null; then
echo '{"status":"error","message":"jq not found. Install jq or ensure it is on PATH."}' >&1
exit 1
fi
if [[ $# -lt 1 ]]; then
jq -n '{status: "error", message: "Usage: run-affected-tests.sh <context.json>"}' >&1
exit 1
fi
CONTEXT_FILE="$1"
if [[ ! -f "$CONTEXT_FILE" ]]; then
jq -n --arg file "$CONTEXT_FILE" '{status: "error", message: "Context file not found: \($file)"}' >&1
exit 1
fi
# --- Parse context.json ---
# Validate JSON structure
if ! jq -e '.' "$CONTEXT_FILE" &>/dev/null; then
echo '{"status":"error","message":"Invalid JSON in context file"}' >&1
exit 1
fi
# Extract test_targets (required, must be JSON array)
TEST_TARGETS=$(jq -c '.test_targets // empty' "$CONTEXT_FILE")
if [[ -z "$TEST_TARGETS" ]]; then
jq -n '{status: "error", message: "context.json missing required field: test_targets"}' >&1
exit 1
fi
if ! jq -e 'type == "array"' <<<"$TEST_TARGETS" >/dev/null 2>&1; then
jq -n '{status: "error", message: "context.json field test_targets must be a JSON array"}' >&1
exit 1
fi
if ! jq -e 'length > 0' <<<"$TEST_TARGETS" >/dev/null 2>&1; then
jq -n '{status: "error", message: "context.json field test_targets must contain at least one target"}' >&1
exit 1
fi
# Extract bench_targets (optional, defaults to empty array, must be JSON array)
BENCH_TARGETS=$(jq -c '.bench_targets // []' "$CONTEXT_FILE")
if ! jq -e 'type == "array"' <<<"$BENCH_TARGETS" >/dev/null 2>&1; then
jq -n '{status: "error", message: "context.json field bench_targets must be a JSON array"}' >&1
exit 1
fi
# Logging (stderr only)
log_info() {
echo "[run-affected-tests] $1" >&2
}
# --- Run tests ---
TEST_RESULTS="[]"
BENCH_RESULTS="[]"
WARNINGS="[]"
TOTAL_TESTS=0
TOTAL_PASSED=0
TOTAL_FAILED=0
TOTAL_BENCH=0
BENCH_PASSED=0
BENCH_FAILED=0
run_pytest_file() {
local file="$1"
local category="$2" # "test" or "bench"
# Reject absolute paths and path traversal attempts
if [[ "$file" == /* ]] || [[ "$file" == *../* ]]; then
WARNINGS=$(echo "$WARNINGS" | jq --arg w "${file} rejected: absolute or traversal path" '. + [$w]')
log_info "WARNING: $file rejected (path security check), skipping"
return
fi
if [[ ! -f "$file" ]]; then
WARNINGS=$(echo "$WARNINGS" | jq --arg w "${file} not found, skipped" '. + [$w]')
log_info "WARNING: $file not found, skipping"
return
fi
log_info "Running: $file"
# Run pytest and capture output
local pytest_output
local exit_code=0
pytest_output=$(PYTHONPATH="$PWD" python -m pytest -v --tb=short -q -- "$file" 2>&1) || exit_code=$?
# Parse pytest summary counts (e.g., "3 passed, 1 failed, 2 errors")
local passed failed errors
passed=$(echo "$pytest_output" | grep -oE '[0-9]+ passed' | tail -1 | grep -oE '[0-9]+' || echo 0)
failed=$(echo "$pytest_output" | grep -oE '[0-9]+ failed' | tail -1 | grep -oE '[0-9]+' || echo 0)
errors=$(echo "$pytest_output" | grep -oE '[0-9]+ errors?' | tail -1 | grep -oE '[0-9]+' || echo 0)
# Extract failure/error names if any
local failures="[]"
if [[ $failed -gt 0 ]] || [[ $errors -gt 0 ]]; then
failures=$(echo "$pytest_output" | { grep -E '^(FAILED |ERROR )' || true; } | sed -E 's/^(FAILED |ERROR )//; s/ - .*$//' | jq -R -s 'split("\n") | map(select(. != ""))')
fi
# Determine status: non-zero exit code OR any failed/error count means failure
# This catches collection errors (e.g., ModuleNotFoundError) where pytest exits
# non-zero but reports no "failed" count in the summary line.
local status="pass"
if [[ $exit_code -ne 0 ]] || [[ $failed -gt 0 ]] || [[ $errors -gt 0 ]]; then
status="fail"
# If pytest exited non-zero but no failed/errors were parsed,
# this is a collection/import error — count it as 1 error
if [[ $failed -eq 0 ]] && [[ $errors -eq 0 ]]; then
errors=1
failures=$(echo "$pytest_output" | tail -5 | jq -R -s 'split("\n") | map(select(. != ""))')
fi
fi
# Include errors in failed count for aggregation
local total_failed=$((failed + errors))
local result
result=$(jq -n \
--arg file "$file" \
--arg status "$status" \
--argjson passed "$passed" \
--argjson failed "$total_failed" \
--argjson failures "$failures" \
'{file: $file, status: $status, passed: $passed, failed: $failed, failures: $failures}')
if [[ "$category" == "test" ]]; then
TEST_RESULTS=$(echo "$TEST_RESULTS" | jq --argjson r "$result" '. + [$r]')
TOTAL_TESTS=$((TOTAL_TESTS + passed + total_failed))
TOTAL_PASSED=$((TOTAL_PASSED + passed))
TOTAL_FAILED=$((TOTAL_FAILED + total_failed))
else
BENCH_RESULTS=$(echo "$BENCH_RESULTS" | jq --argjson r "$result" '. + [$r]')
TOTAL_BENCH=$((TOTAL_BENCH + passed + total_failed))
BENCH_PASSED=$((BENCH_PASSED + passed))
BENCH_FAILED=$((BENCH_FAILED + total_failed))
fi
}
# Run test targets (process substitution avoids subshell — variable updates preserved)
log_info "=== Running test targets ==="
while IFS= read -r file; do
run_pytest_file "$file" "test"
done < <(jq -r '.[]' <<<"$TEST_TARGETS")
# Run bench targets
log_info "=== Running benchmark targets ==="
while IFS= read -r file; do
run_pytest_file "$file" "bench"
done < <(jq -r '.[]' <<<"$BENCH_TARGETS")
# --- Determine overall status ---
OVERALL_STATUS="pass"
if [[ $TOTAL_FAILED -gt 0 ]] || [[ $BENCH_FAILED -gt 0 ]]; then
OVERALL_STATUS="fail"
fi
# Check for partial: some passed, some failed
if [[ "$OVERALL_STATUS" == "fail" ]] && [[ $TOTAL_PASSED -gt 0 || $BENCH_PASSED -gt 0 ]]; then
OVERALL_STATUS="partial"
fi
# --- Output structured JSON ---
jq -n \
--arg status "$OVERALL_STATUS" \
--argjson test_total "$TOTAL_TESTS" \
--argjson test_passed "$TOTAL_PASSED" \
--argjson test_failed "$TOTAL_FAILED" \
--argjson test_results "$TEST_RESULTS" \
--argjson bench_total "$TOTAL_BENCH" \
--argjson bench_passed "$BENCH_PASSED" \
--argjson bench_failed "$BENCH_FAILED" \
--argjson bench_results "$BENCH_RESULTS" \
--argjson warnings "$WARNINGS" \
'{
status: $status,
tests: {
total: $test_total,
passed: $test_passed,
failed: $test_failed,
results: $test_results
},
benchmarks: {
total: $bench_total,
passed: $bench_passed,
failed: $bench_failed,
results: $bench_results
},
warnings: $warnings
}'
# Exit non-zero when tests/benchmarks failed so callers can use exit code as HARD GATE
if [[ "$OVERALL_STATUS" != "pass" ]]; then
exit 1
fi

95
.foundry/scripts/validate.sh
View File

@ -1,95 +0,0 @@
#!/bin/bash
# Validate PR metadata for TileOPs
# Usage: validate.sh <owner/repo> <pr_number>
#
# Checks: title format, body sections, labels, MCP pitfall
# Exit 0 = all checks pass
# Exit 1 = at least one check failed
set -euo pipefail
# Source canonical type definitions
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../../../.." && pwd)"
source "$REPO_ROOT/.claude/conventions/types.sh"
OWNER_REPO="${1:?Usage: validate.sh <owner/repo> <pr_number>}"
PR_NUMBER="${2:?Usage: validate.sh <owner/repo> <pr_number>}"
ERRORS=0
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
pass() { echo -e "${GREEN}✓${NC} $1"; }
fail() { echo -e "${RED}✗${NC} $1" >&2; ERRORS=$((ERRORS + 1)); }
warn() { echo -e "${YELLOW}⚠${NC} $1" >&2; }
# --- Fetch PR data ---
PR_JSON=$(gh pr view "$PR_NUMBER" --repo "$OWNER_REPO" --json title,body,labels 2>&1) || {
echo -e "${RED}✗ Failed to fetch PR #${PR_NUMBER}: ${PR_JSON}${NC}" >&2
exit 1
}
TITLE=$(echo "$PR_JSON" | jq -r '.title')
BODY=$(echo "$PR_JSON" | jq -r '.body')
LABEL_COUNT=$(echo "$PR_JSON" | jq '.labels | length')
echo "=== PR #${PR_NUMBER} validation ==="
echo "Title: ${TITLE}"
echo ""
# --- Checks ---
# 1. PR title format
if [[ "$TITLE" =~ $COMMIT_MSG_PATTERN ]]; then
pass "PR title follows [Type] Description format"
else
fail "PR title must match [Type] Description (e.g. [Feat] Add forward op)"
fi
# 2. ## Summary section
if echo "$BODY" | grep -q '## Summary'; then
pass "PR body contains ## Summary section"
else
fail "PR body must contain ## Summary section"
fi
# 3. ## Test plan section
if echo "$BODY" | grep -q '## Test plan'; then
pass "PR body contains ## Test plan section"
else
fail "PR body must contain ## Test plan section"
fi
# 4. At least one label
if [[ "$LABEL_COUNT" -gt 0 ]]; then
LABELS=$(echo "$PR_JSON" | jq -r '[.labels[].name] | join(", ")')
pass "PR has ${LABEL_COUNT} label(s): ${LABELS}"
else
fail "PR must have at least one label"
fi
# 5. No literal \n in body (MCP pitfall)
if echo "$BODY" | grep -q '\\n'; then
fail "PR body contains literal \\\\n — use actual newlines instead"
else
pass "PR body uses actual newlines (no MCP pitfall)"
fi
# 6. BugFix should have Regression section (warning only)
if [[ "$TITLE" =~ ^\[BugFix\] ]] && ! echo "$BODY" | grep -q '## Regression'; then
warn "[BugFix] PR should include ## Regression section"
fi
echo ""
if [[ $ERRORS -gt 0 ]]; then
echo -e "${RED}FAILED: ${ERRORS} check(s) failed${NC}" >&2
exit 1
else
echo -e "${GREEN}ALL CHECKS PASSED${NC}"
exit 0
fi

35
.foundry/scripts/wait-for-gpu.sh
View File

@ -1,35 +0,0 @@
#!/bin/bash
# Wait for GPU 0 to become idle before running benchmarks.
#
# Exit codes:
# 0 — GPU 0 is idle (no compute processes)
# 1 — nvidia-smi not found (no GPU)
# 2 — GPU 0 still busy after all retries
#
# Usage: wait-for-gpu.sh [max_retries] [interval_seconds]
set -euo pipefail
MAX_RETRIES="${1:-5}"
INTERVAL="${2:-30}"
if ! command -v nvidia-smi &>/dev/null; then
echo "nvidia-smi not found — no GPU available"
exit 1
fi
for ((i = 1; i <= MAX_RETRIES; i++)); do
PROCS=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader -i 0 2>/dev/null || true)
if [[ -z "$PROCS" ]]; then
echo "GPU 0 is idle"
exit 0
fi
if [[ "$i" -lt "$MAX_RETRIES" ]]; then
echo "GPU 0 busy (attempt $i/$MAX_RETRIES), waiting ${INTERVAL}s..."
sleep "$INTERVAL"
fi
done
echo "GPU 0 still busy after $MAX_RETRIES attempts"
exit 2

47
.github/ISSUE_TEMPLATE/new_operator_subtask.md vendored
View File

@ -1,47 +0,0 @@
---
name: New Operator Sub-task
about: A specific sub-task for implementing a part of a new operator
title: '[<Feat/Perf/Fix>][Ops] <Operator Name>: <manifest/test/impl/bench scope>'
labels: sub-task, operator
assignees: ''
---
## Parent Issue
<!-- Link to the main tracking issue for this operator using #IssueID -->
Part of #
## Task Type
<!-- Please check the relevant component for this sub-issue -->
- [ ] **Manifest / Spec** (required for any public Op)
- [ ] **Kernel / Op Implementation**
- [ ] **Correctness Tests / Workloads**
- [ ] **Benchmark / Performance**
## Description
<!-- Detailed description of what needs to be implemented in this step -->
## Checklist
<!--
Refer to:
- docs/design/manifest.md for manifest fields and validation levels
- docs/design/ops-design.md for Op/Kernel implementation rules
- docs/design/testing.md for tests, workloads, and benchmark structure
- docs/design/roofline.md for roofline authoring and benchmark consumers
-->
- [ ] Public Op has a `tileops/manifest/` entry (or updates an existing entry).
- [ ] Manifest `signature` declares inputs, outputs, params, shape rules, and dtype coverage.
- [ ] Manifest `workloads` declare benchmark shapes/dtypes; unit-test edge cases are not generated from manifest workloads.
- [ ] Manifest `roofline` is present and consumable by `op.eval_roofline()`.
- [ ] Manifest `source` declares kernel, op, test, bench, and `source.kernel_map` when dispatching kernels.
- [ ] Op constructor, `forward()`, and `default_kernel_map` match the manifest entry.
- [ ] Tests use an independent reference implementation and cover relevant FP16/BF16 and edge cases.
- [ ] Benchmarks consume manifest workloads and record at least one non-`tileops` baseline unless explicitly justified.
- [ ] PR title and commits follow the current `[Feat][Scope]` / `[Perf][Scope]` / `[Fix][Scope]` convention.
- [ ] Implementation follows **Google Python Style** for code and docstrings.

5
.github/actionlint.yaml vendored
View File

@ -1,5 +0,0 @@
self-hosted-runner:
labels:
- tile-ops
- venv
- nightly

267
.github/actions/reclaim-runner-disk/action.yml vendored
View File

@ -1,267 +0,0 @@
name: Reclaim runner disk
description: >
Trim the persistent bind-mounted /ci-cache (compiled-kernel / triton / pip
caches and the tilelang autotuner cache) on self-hosted CI runners. Fails
fast if the cache volume is still near full after reclaim, so downstream
steps report disk pressure clearly instead of a cryptic mkdir error.
inputs:
cache-age-days:
description: Age threshold (days) for trimming cache files.
required: false
default: "7"
min-avail-gib:
description: >
Hard floor on free space (GiB) in RUNNER_TEMP after reclaim. Below this
the step fails with an explicit message.
required: false
default: "5"
reclaim-below-gib:
description: >
Only attempt reclaim work when RUNNER_TEMP has less than this many GiB
free, unless force-reclaim is enabled.
required: false
default: "30"
cache-trim-cooldown-minutes:
description: >
Minimum interval between the expensive recursive cache-trim passes on the
same runner. Cheap disk checks still run on every invocation.
required: false
default: "0"
cache-dirs:
description: >
Newline-separated list of cache directory roots whose files older than
cache-age-days should be trimmed at file granularity. Defaults target
the bind-mounted /ci-cache runner cache layout.
Do NOT include atomic cache roots here (see atomic-cache-dirs); file-
level trim on an atomic root can produce a half-dead "directory exists
but sentinel missing" state that crashes downstream consumers.
required: false
default: |
/ci-cache/triton
/ci-cache/pip
/ci-cache/tilelang/tmp
atomic-cache-dirs:
description: >
Newline-separated list of cache directory roots whose first-level
subdirectories are *atomic units*: consumers assume "directory exists
=> contents complete" and the subdir is only valid when the sentinel
file (best_config.json, hardcoded) is present. These roots get two
passes instead of file-level trim:
1. Unconditional sentinel-repair: remove any first-level subdir
missing the sentinel (self-heals half-dead state from prior runs).
2. Age-based atomic-trim: if the newest file in a first-level subdir
is older than cache-age-days, delete the whole subdir — never
individual files inside it.
Default targets the tilelang autotuner cache, which stores per-shape
tuning artefacts alongside a best_config.json sentinel.
required: false
default: |
/ci-cache/tilelang/autotuner
df-paths:
description: Space-separated paths to pass to `df -h` for before/after logging.
required: false
default: ""
force-reclaim:
description: >
When "true", bypass reclaim-below-gib and cache-trim cooldown checks and
always run the full reclaim pass.
required: false
default: "false"
skip-atomic-age-trim:
description: >
When "true", skip the age-based atomic-trim pass on atomic cache roots.
Sentinel-repair on those roots still runs unconditionally (it self-heals
half-dead state and is cheap). Callers that run on every PR should opt
out of the destructive whole-subdir trim so it only runs via the daily
runner-maintenance.yml job; co-locating it on the per-PR path evicts
autotuner entries the next PR would otherwise reuse.
required: false
default: "false"
runs:
using: composite
steps:
- name: Reclaim runner disk
shell: bash
env:
CACHE_AGE_DAYS: ${{ inputs.cache-age-days }}
MIN_AVAIL_GIB: ${{ inputs.min-avail-gib }}
RECLAIM_BELOW_GIB: ${{ inputs.reclaim-below-gib }}
CACHE_TRIM_COOLDOWN_MINUTES: ${{ inputs.cache-trim-cooldown-minutes }}
CACHE_DIRS: ${{ inputs.cache-dirs }}
ATOMIC_CACHE_DIRS: ${{ inputs.atomic-cache-dirs }}
DF_PATHS: ${{ inputs.df-paths }}
FORCE_RECLAIM: ${{ inputs.force-reclaim }}
SKIP_ATOMIC_AGE_TRIM: ${{ inputs.skip-atomic-age-trim }}
run: |
set -uo pipefail
# The reclaim primitives (sentinel-repair, atomic-trim, trim-files)
# live in reclaim_cache.sh (colocated with this action.yml) so
# they can be exercised under pytest without a self-hosted
# runner, *and* so the gpu-smoke trusted-action sparse-checkout
# (which only pulls `.github/actions`) still picks them up.
# This composite action is a thin policy wrapper — gating (when
# to run), not mechanism (how to trim).
#
# ${GITHUB_ACTION_PATH} resolves to
# .../.github/actions/reclaim-runner-disk
# and the script sits alongside action.yml so the trusted-action
# sparse-checkout (which only pulls .github/actions) still picks
# it up on the gpu-smoke job.
RECLAIM_SCRIPT="${GITHUB_ACTION_PATH}/reclaim_cache.sh"
if [[ ! -f "${RECLAIM_SCRIPT}" || ! -r "${RECLAIM_SCRIPT}" ]]; then
echo "::error::reclaim-runner-disk: expected ${RECLAIM_SCRIPT} to exist and be readable"
exit 1
fi
# Parse a newline-separated input (as received from YAML `|` block
# scalars) into a bash array, stripping CR and skipping blank
# lines. Usage:
# parse_list_into_array TARGET_ARRAY <<< "${INPUT_VAR}"
# The named array is declared via `declare -n`; callers must have
# declared the target array with `local -a`/`declare -a` first, or
# let this helper create it implicitly (we do the latter — the
# helper resets the target to a fresh empty array on each call).
parse_list_into_array() {
local -n _target_array="$1"
_target_array=()
local _line
while IFS= read -r _line; do
_line="${_line//$'\r'/}"
[[ -z "$_line" ]] && continue
_target_array+=( "$_line" )
done
}
# Validate numeric inputs up-front so typos fail with a clear error
# instead of a cryptic bash arithmetic/find error deeper in the script.
for var in CACHE_AGE_DAYS MIN_AVAIL_GIB RECLAIM_BELOW_GIB CACHE_TRIM_COOLDOWN_MINUTES; do
val="${!var}"
if [[ ! "${val}" =~ ^[0-9]+$ ]]; then
echo "::error::reclaim-runner-disk: input ${var}='${val}' is not a non-negative integer"
exit 1
fi
done
if [[ -n "${DF_PATHS}" ]]; then
read -r -a DF_TARGETS <<< "${DF_PATHS}"
else
DF_TARGETS=()
for path in "${RUNNER_TEMP:-}" /ci-cache; do
[[ -n "${path}" ]] && DF_TARGETS+=( "${path}" )
done
fi
echo "Disk before reclaim:"
df -h "${DF_TARGETS[@]}" 2>/dev/null || true
if [[ -z "${RUNNER_TEMP:-}" ]]; then
echo "::error::reclaim-runner-disk: RUNNER_TEMP is unset; cannot enforce disk floor"
exit 1
fi
AVAIL_KB=$(df -Pk "${RUNNER_TEMP}" | awk 'NR==2 {print $4}')
MIN_KB=$(( MIN_AVAIL_GIB * 1024 * 1024 ))
RECLAIM_BELOW_KB=$(( RECLAIM_BELOW_GIB * 1024 * 1024 ))
# Stamp lives on the persistent bind-mounted cache (not the ephemeral container fs under
# /home/ci-runner), so the cache-trim cooldown survives across ephemeral runner jobs.
CACHE_TRIM_STAMP="/ci-cache/.ci-maintenance/reclaim-runner-disk-cache-trim.stamp"
DID_RECLAIM="false"
SHOULD_RECLAIM="false"
# 0) Unconditional sentinel-repair on atomic cache roots.
# Runs regardless of SHOULD_RECLAIM so that half-dead state
# left by an earlier file-level trim (directory exists but
# best_config.json missing) is healed on the *next* invocation,
# not only when disk pressure crosses the reclaim threshold.
# The per-subdir cost is a single readdir + stat, so this is
# safe to run every time.
if [[ -n "${ATOMIC_CACHE_DIRS}" ]]; then
parse_list_into_array ATOMIC_ARGS <<< "${ATOMIC_CACHE_DIRS}"
if (( ${#ATOMIC_ARGS[@]} > 0 )); then
bash "${RECLAIM_SCRIPT}" sentinel-repair "${ATOMIC_ARGS[@]}" || true
fi
fi
if [[ "${FORCE_RECLAIM}" == "true" ]]; then
SHOULD_RECLAIM="true"
echo "force-reclaim enabled: running full reclaim pass"
elif [[ -n "${AVAIL_KB:-}" && "${AVAIL_KB}" -lt "${RECLAIM_BELOW_KB}" ]]; then
SHOULD_RECLAIM="true"
echo "Free space below ${RECLAIM_BELOW_GIB}GiB: running reclaim pass"
else
echo "Free space is healthy; skipping reclaim pass"
fi
# 1) Age-based trim of the file-granularity cache roots, plus
# atomic-granularity trim of atomic cache roots.
SHOULD_TRIM_CACHE="false"
if [[ "${SHOULD_RECLAIM}" == "true" && ( -n "${CACHE_DIRS}" || -n "${ATOMIC_CACHE_DIRS}" ) ]]; then
if [[ "${FORCE_RECLAIM}" == "true" || "${CACHE_TRIM_COOLDOWN_MINUTES}" == "0" ]]; then
SHOULD_TRIM_CACHE="true"
elif [[ ! -e "${CACHE_TRIM_STAMP}" ]]; then
SHOULD_TRIM_CACHE="true"
else
now_epoch=$(date +%s)
stamp_epoch=$(stat -c %Y "${CACHE_TRIM_STAMP}" 2>/dev/null || echo 0)
cooldown_seconds=$(( CACHE_TRIM_COOLDOWN_MINUTES * 60 ))
if (( now_epoch - stamp_epoch >= cooldown_seconds )); then
SHOULD_TRIM_CACHE="true"
else
remaining_minutes=$(( (cooldown_seconds - (now_epoch - stamp_epoch) + 59) / 60 ))
echo "Skipping cache trim: ${remaining_minutes} minute(s) left in cooldown"
fi
fi
fi
if [[ "${SHOULD_TRIM_CACHE}" == "true" ]]; then
DID_RECLAIM="true"
# File-level trim for non-atomic roots (triton cache, pip, wheel).
if [[ -n "${CACHE_DIRS}" ]]; then
parse_list_into_array FILE_TRIM_ARGS <<< "${CACHE_DIRS}"
if (( ${#FILE_TRIM_ARGS[@]} > 0 )); then
bash "${RECLAIM_SCRIPT}" trim-files "${CACHE_AGE_DAYS}" "${FILE_TRIM_ARGS[@]}" || true
fi
fi
# Whole-subdir trim for atomic roots (tilelang autotuner).
# Atomic roots are NEVER file-trimmed: doing so can strand a
# subdir without its best_config.json sentinel and crash the
# next consumer.
#
# Callers that run on every PR (gpu-smoke.yml) opt out via
# skip-atomic-age-trim so the destructive whole-subdir eviction
# only happens via the daily runner-maintenance.yml job;
# sentinel-repair above still ran unconditionally, so half-dead
# state is healed regardless of this opt-out.
if [[ "${SKIP_ATOMIC_AGE_TRIM}" == "true" ]]; then
echo "Skipping atomic age-trim (opted out)"
elif [[ -n "${ATOMIC_CACHE_DIRS}" ]]; then
parse_list_into_array ATOMIC_TRIM_ARGS <<< "${ATOMIC_CACHE_DIRS}"
if (( ${#ATOMIC_TRIM_ARGS[@]} > 0 )); then
bash "${RECLAIM_SCRIPT}" atomic-trim "${CACHE_AGE_DAYS}" "${ATOMIC_TRIM_ARGS[@]}" || true
fi
fi
mkdir -p "$(dirname "${CACHE_TRIM_STAMP}")"
touch "${CACHE_TRIM_STAMP}"
fi
if [[ "${SHOULD_RECLAIM}" != "true" ]]; then
echo "Reclaim work skipped because free space is above the proactive threshold"
elif [[ "${DID_RECLAIM}" != "true" ]]; then
echo "Reclaim pass ran, but nothing was eligible for deletion"
fi
echo "Disk after reclaim:"
df -h "${DF_TARGETS[@]}" 2>/dev/null || true
# Hard guard: abort early with a clear message if still near-full.
AVAIL_KB=$(df -Pk "${RUNNER_TEMP}" | awk 'NR==2 {print $4}')
if [[ -n "${AVAIL_KB:-}" && "${AVAIL_KB}" -lt "${MIN_KB}" ]]; then
echo "::error::Runner ${RUNNER_NAME:-unknown} has <${MIN_AVAIL_GIB}GiB free on ${RUNNER_TEMP} after reclaim (${AVAIL_KB} KiB). Manual cleanup required."
exit 1
fi

185
.github/actions/reclaim-runner-disk/reclaim_cache.sh vendored
View File

@ -1,185 +0,0 @@
#!/usr/bin/env bash
# reclaim_cache.sh — cache-reclaim primitives used by the
# .github/actions/reclaim-runner-disk composite action.
#
# The action.yml composite used to embed all reclaim logic inline, which
# meant the cache-trim paths could only be exercised on a self-hosted
# runner. That made it impossible to catch regressions at PR time; a bad
# trim rule only surfaced after merge when it had already corrupted the
# live cache. This script exposes the trim primitives as
# pytest-driveable subcommands so tests/test_reclaim_action.py can
# validate behaviour against a tmp_path fixture.
#
# The core safety invariant this script enforces is that some cache
# roots (notably /ci-cache/tilelang/autotuner) store
# *atomic* first-level subdirectories: consumers assume "directory
# exists => contents complete" and the directory is only valid if the
# sentinel file (best_config.json) is present. File-level `-mtime`
# trimming on these roots can delete the sentinel while leaving the
# rest of the directory in place, producing a half-dead state that
# crashes the next consumer. The atomic-trim and sentinel-repair
# subcommands here operate at whole-subdir granularity and never touch
# individual files inside an atomic root.
#
# Subcommands:
#
# sentinel-repair <root> [<root>...]
# For each atomic root, delete any first-level subdirectory that
# does not contain ${SENTINEL_FILENAME:-best_config.json}. This
# self-heals half-dead state left by older reclaim passes.
#
# atomic-trim <age-days> <root> [<root>...]
# For each atomic root, compute the newest file mtime anywhere
# inside each first-level subdirectory. If that mtime is older
# than <age-days> days, delete the whole subdirectory. Never
# trims individual files inside atomic roots.
#
# trim-files <age-days> <root> [<root>...]
# File-level `-mtime +N -delete` trim for non-atomic cache
# roots, followed by empty-directory cleanup. This is the legacy
# behaviour used for /ci-cache/triton, /ci-cache/pip, etc.
#
# All subcommands are idempotent, tolerate missing roots (skipped
# silently), and never exit non-zero for per-directory errors so a
# transient `rm` failure cannot abort the whole reclaim step.
set -uo pipefail
SENTINEL_FILENAME="${SENTINEL_FILENAME:-best_config.json}"
_log() {
echo "$@"
}
# sentinel_repair <root> [<root>...]
#
# Delete any first-level subdirectory of each root that is missing the
# sentinel file. Roots that do not exist are skipped. The sentinel
# filename is read from ${SENTINEL_FILENAME:-best_config.json}.
sentinel_repair() {
local root subdir
for root in "$@"; do
[[ -z "$root" ]] && continue
[[ -d "$root" ]] || continue
while IFS= read -r -d '' subdir; do
if [[ ! -e "${subdir}/${SENTINEL_FILENAME}" ]]; then
_log "sentinel-repair: removing ${subdir} (missing ${SENTINEL_FILENAME})"
rm -rf "$subdir" 2>/dev/null || true
fi
done < <(find "$root" -mindepth 1 -maxdepth 1 -type d -print0 2>/dev/null)
done
}
# atomic_trim <age-days> <root> [<root>...]
#
# For each first-level subdirectory of each root, compute the newest
# file mtime in the whole subtree. If that mtime is older than
# <age-days> days, delete the whole subdirectory. Never deletes
# individual files; the unit of trim is the subdirectory.
atomic_trim() {
local age_days="$1"
shift
local now_epoch cutoff root subdir newest_mtime
# Validate age_days is a plain non-negative integer before feeding it
# into arithmetic. Bash defaults to base-8 for literals with a leading
# zero, so a caller-supplied value like "08"/"09" (e.g. from a
# zero-padded workflow input) would abort the whole trim with
# "value too great for base". We fail *open* — log and return 0 —
# so a bad input can't crash the wider reclaim pass, and we force
# base-10 parsing via `10#...` for defence in depth.
if [[ ! "$age_days" =~ ^[0-9]+$ ]]; then
_log "atomic-trim: ignoring invalid age_days='${age_days}' (expected non-negative integer)"
return 0
fi
now_epoch=$(date +%s)
cutoff=$(( now_epoch - 10#${age_days} * 86400 ))
for root in "$@"; do
[[ -z "$root" ]] && continue
[[ -d "$root" ]] || continue
while IFS= read -r -d '' subdir; do
# Use the newest FILE mtime anywhere in the subtree — directory
# mtimes must not participate. Rationale: a cache restore/extract
# can bump the subdir's own mtime to "now" while every regular
# file inside is still at its original (old) timestamp; counting
# the dir mtime would wrongly mark the subdir as fresh and defeat
# age-based reclaim. Only fall back to the subdir mtime when the
# subtree has no regular files at all.
newest_mtime=$(find "$subdir" -type f -printf '%T@\n' 2>/dev/null \
| awk 'NR == 1 || $1 > max { max = $1 } END { if (NR > 0) printf "%d\n", max }')
if [[ -z "$newest_mtime" ]]; then
newest_mtime=$(stat -c %Y "$subdir" 2>/dev/null || echo 0)
fi
if (( newest_mtime < cutoff )); then
_log "atomic-trim: removing ${subdir} (newest mtime ${newest_mtime} < cutoff ${cutoff})"
rm -rf "$subdir" 2>/dev/null || true
fi
done < <(find "$root" -mindepth 1 -maxdepth 1 -type d -print0 2>/dev/null)
done
}
# trim_files <age-days> <root> [<root>...]
#
# File-level age-based trim for non-atomic caches. Deletes files older
# than <age-days>, then deletes any now-empty directories. Empty-dir
# cleanup does NOT gate on -mtime because file deletion bumps the
# parent directory's mtime to "now" and a mtime filter would leave the
# newly-empty dirs behind.
trim_files() {
local age_days="$1"
shift
local root
# Same validation as atomic_trim: refuse non-integer age_days and fail
# open. `find -mtime` itself is tolerant of leading-zero strings, but
# we keep the contract identical across subcommands so a caller that
# passes a bogus value gets the same behaviour everywhere.
if [[ ! "$age_days" =~ ^[0-9]+$ ]]; then
_log "trim-files: ignoring invalid age_days='${age_days}' (expected non-negative integer)"
return 0
fi
age_days=$(( 10#${age_days} ))
for root in "$@"; do
[[ -z "$root" ]] && continue
[[ -d "$root" ]] || continue
find "$root" -type f -mtime "+${age_days}" -delete 2>/dev/null || true
find "$root" -depth -mindepth 1 -type d -empty -delete 2>/dev/null || true
done
}
_usage() {
cat >&2 <<'EOF'
usage: reclaim_cache.sh <subcommand> [args...]
subcommands:
sentinel-repair <root> [<root>...]
atomic-trim <age-days> <root> [<root>...]
trim-files <age-days> <root> [<root>...]
EOF
exit 2
}
main() {
[[ $# -ge 1 ]] || _usage
local cmd="$1"
shift
case "$cmd" in
sentinel-repair)
sentinel_repair "$@"
;;
atomic-trim)
[[ $# -ge 2 ]] || _usage
atomic_trim "$@"
;;
trim-files)
[[ $# -ge 2 ]] || _usage
trim_files "$@"
;;
*)
_usage
;;
esac
}
# Allow sourcing the file in tests without executing main.
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi

222
.github/runner/Dockerfile vendored
View File

@ -1,222 +0,0 @@
# syntax=docker/dockerfile:1.7
# Multi-stage CI runner image for the self-hosted GPU runner.
ARG BASE_IMAGE=nvidia/cuda:12.9.1-devel-ubuntu22.04
# ── runtime ──
# devel (not runtime) base: nvcc is needed both at build time (FA3 / tilelang source builds)
# and at runtime (tilelang JIT).
FROM ${BASE_IMAGE} AS runtime
ENV DEBIAN_FRONTEND=noninteractive PIP_NO_CACHE_DIR=1 PIP_BREAK_SYSTEM_PACKAGES=1
# 22.04 ships python3.10; install 3.12 from deadsnakes and make it the default python.
# software-properties-common is purged after adding the PPA to keep the image lean.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
software-properties-common ca-certificates curl gnupg \
&& add-apt-repository -y ppa:deadsnakes/ppa \
&& apt-get update \
&& apt-get install -y --no-install-recommends \
python3.12 python3.12-dev python3.12-venv \
git build-essential cmake ninja-build unzip xz-utils sudo \
&& apt-get purge -y --auto-remove software-properties-common \
&& apt-get clean && rm -rf /var/lib/apt/lists/* \
&& python3.12 -m ensurepip --upgrade \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12 1 \
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.12 1 \
&& python -m pip install --upgrade pip
COPY constraints.txt /tmp/constraints.txt
# torch / torchvision / torchaudio: cu129 wheels from the pytorch index (the PyPI default is a
# different CUDA variant, so pin +cu129). Installing torchvision/torchaudio as cu129 here means
# vllm later finds its exact-pinned versions already satisfied and never swaps them off cu129.
RUN python -m pip install \
--index-url https://download.pytorch.org/whl/cu129 \
--extra-index-url https://pypi.org/simple \
-c /tmp/constraints.txt \
"torch==2.10.0+cu129" "torchvision==0.25.0+cu129" "torchaudio==2.10.0+cu129"
# tilelang's build backend (scikit-build-core + patchelf, its [build-system].requires) and its
# runtime deps, installed here so the tilelang stage can build with --no-build-isolation. cmake
# from pip provides >=3.26.1 (tilelang's [tool.scikit-build] floor); jammy apt cmake is 3.22.
RUN python -m pip install -c /tmp/constraints.txt \
setuptools wheel ninja scikit-build-core patchelf cmake \
triton apache-tvm-ffi cloudpickle ml_dtypes numpy psutil tqdm \
typing_extensions Cython z3-solver torch_c_dlpack_ext einops "PyYAML>=6.0"
ARG MAX_JOBS=64
ARG NVCC_THREADS=4
# TORCH_CUDA_ARCH_LIST=9.0: the runner pool is Hopper (H200) only. Drives the FA3 and tilelang
# source builds in later stages, and runtime tilelang JIT.
ENV MAX_JOBS=${MAX_JOBS} NVCC_THREADS=${NVCC_THREADS} TORCH_CUDA_ARCH_LIST=9.0
# ── post-fa3 ──
# Test tooling + FlashAttention-3 (a bench baseline). tilelang is NOT built here — it is built
# last (the `tilelang` stage) so a SHA bump rebuilds only that layer and so it links against
# the final post-bench stack.
FROM runtime AS post-fa3
RUN python -m pip install --no-cache-dir -c /tmp/constraints.txt \
"pytest==9.0.2" "pytest-xdist>=3.0" "ruff==0.14.13"
# FlashAttention-3 (Hopper) has no PyPI wheel — build it from the repo's hopper/ dir.
# Fetch ONLY the csrc/cutlass submodule the build needs: a full --recursive also pulls the AMD
# composable_kernel submodule, which the sm_90 build never uses and which is large and
# slow/timeout-prone to clone. Best-effort: a bench baseline must not break the image. The
# source tree is removed whether or not the build succeeds, so it never lands in the image.
RUN git clone --depth 1 --branch v2.8.3 \
https://github.com/Dao-AILab/flash-attention.git /tmp/flash-attention \
&& cd /tmp/flash-attention \
&& git submodule update --init --depth 1 csrc/cutlass \
&& cd hopper && python setup.py install; \
status=$?; rm -rf /tmp/flash-attention; \
[ "$status" -eq 0 ] || echo "WARNING: FlashAttention-3 (hopper) build failed (non-fatal)"
# ── fa2 ──
# FlashAttention-2 in its OWN stage/layer: no prebuilt wheel for this torch/cu129/py, so it
# builds from source (~minutes). Isolating it means changes to the bench-install loop
# (fullstack) never invalidate this expensive compile. flash-attn does not re-resolve torch.
# Best-effort: a bench baseline must not fail the image.
FROM post-fa3 AS fa2
RUN python -m pip install --no-cache-dir --no-build-isolation \
--index-url https://download.pytorch.org/whl/cu129 \
--extra-index-url https://pypi.org/simple \
-c /tmp/constraints.txt "flash-attn==2.8.3" \
|| echo "WARNING: flash-attn (FA2) install failed (non-fatal)"
# ── fullstack ──
# Remaining bench baselines (comparison targets). Best-effort. Freeze the cu129 torch trio
# that `runtime` installed into a constraint first so a bench dep's unversioned `torch`
# requirement cannot swap torch off the cu129 stack; constraints.txt itself stays bare so it
# also fits the CPU preflight. vllm pulls flashinfer 0.6.6 (upgraded below) and has no
# apache-tvm-ffi pin, so it does not conflict with tilelang. sgl-kernel is not installed.
FROM fa2 AS fullstack
RUN python -m pip freeze | grep -iE '^(torch|torchvision|torchaudio)==' > /tmp/torch-cu129.txt \
&& for pkg in \
"flash-linear-attention==0.4.2" \
"vllm==0.19.1"; do \
python -m pip install --no-cache-dir --no-build-isolation \
--index-url https://download.pytorch.org/whl/cu129 \
--extra-index-url https://pypi.org/simple \
-c /tmp/constraints.txt -c /tmp/torch-cu129.txt "$pkg" \
|| echo "WARNING: bench baseline '$pkg' install failed (non-fatal)"; \
done
# The Mamba benchmarks compare against the official mamba_ssm Triton kernels. Keep this
# best-effort and dependency-free so it cannot re-resolve the CUDA-specific torch/triton stack
# or pull the newer mamba_ssm tilelang/apache-tvm-ffi pins into the runner image. mamba_ssm's
# package import loads selective_scan_cuda, so the CUDA extension must be built even when the
# benchmark uses the Triton modules.
RUN MAMBA_FORCE_BUILD=TRUE \
python -m pip install --no-cache-dir --no-build-isolation --no-deps \
-c /tmp/constraints.txt -c /tmp/torch-cu129.txt \
"mamba-ssm==2.3.1" \
|| echo "WARNING: mamba-ssm 2.3.1 install failed (non-fatal)"
# The grouped-GEMM benchmark can compare against DeepGEMM. Install from the official tag
# instead of PyPI's stale 1.0.0 sdist, and fetch only the submodules needed for its headers.
ARG DEEPGEMM_GIT_SHA=c9f8b34dcdacc20aa746b786f983492c51072870
RUN { mkdir -p /tmp/DeepGEMM \
&& cd /tmp/DeepGEMM \
&& git init -q \
&& git remote add origin https://github.com/deepseek-ai/DeepGEMM.git \
&& for attempt in 1 2 3 4 5; do \
git -c http.version=HTTP/1.1 fetch --depth 1 origin "${DEEPGEMM_GIT_SHA}" && break; \
status="$?"; \
echo "WARNING: DeepGEMM fetch failed (attempt ${attempt}/5, status ${status})"; \
if [ "${attempt}" = 5 ]; then exit "${status}"; fi; \
sleep "$((attempt * 10))"; \
done \
&& git checkout -q FETCH_HEAD \
&& for attempt in 1 2 3 4 5; do \
git -c http.version=HTTP/1.1 submodule update --init --depth 1 --jobs 1 \
third-party/cutlass third-party/fmt && break; \
status="$?"; \
echo "WARNING: DeepGEMM submodule fetch failed (attempt ${attempt}/5, status ${status})"; \
rm -rf third-party/cutlass third-party/fmt .git/modules/third-party/cutlass .git/modules/third-party/fmt; \
if [ "${attempt}" = 5 ]; then exit "${status}"; fi; \
sleep "$((attempt * 10))"; \
done \
&& DG_FORCE_BUILD=1 python -m pip install --no-cache-dir --no-build-isolation --no-deps \
-c /tmp/constraints.txt -c /tmp/torch-cu129.txt .; } \
|| echo "WARNING: DeepGEMM v2.1.1.post3 install failed (non-fatal)"; \
rm -rf /tmp/DeepGEMM
# The attention benchmarks (benchmarks/ops/attention/) need a flashinfer newer than the 0.6.6
# vllm pulls. Upgrade flashinfer-python/-cubin with --no-deps so torch and vllm's other deps are
# untouched; add plain cuda-tile (flashinfer>=0.6.7 needs it — the [tileiras] extra would pull a
# CUDA-13 toolchain). vllm's flashinfer pin is left unsatisfied, which is safe: vllm's only
# TileOPs-used path (fused_moe) does not import flashinfer.
RUN { python -m pip install --no-cache-dir --no-deps -c /tmp/constraints.txt -c /tmp/torch-cu129.txt \
"flashinfer-python==0.6.11.post2" "flashinfer-cubin==0.6.11.post2" \
&& python -m pip install --no-cache-dir -c /tmp/constraints.txt -c /tmp/torch-cu129.txt \
cuda-tile nvidia-cudnn-frontend "nvidia-cutlass-dsl>=4.5.0" nvidia-ml-py; } \
|| echo "WARNING: flashinfer 0.6.11.post2 upgrade failed (non-fatal)"
# ── tilelang ──
# Built LAST (after all bench): a TILELANG_GIT_SHA bump then rebuilds only this layer (bench
# stays cached), and tilelang compiles/links against the exact final stack so its ABI always
# matches what ships. ARG is declared here — not earlier — so changing it never invalidates
# the bench layers above.
FROM fullstack AS tilelang
ARG TILELANG_GIT_SHA
ARG TILELANG_VERSION
# main mode: --build-arg TILELANG_GIT_SHA=<commit> → clone + compile that commit
# release mode: --build-arg TILELANG_VERSION=<version> → pip install the PyPI release
# Either way --no-deps, so pip never re-resolves the cu129 stack.
RUN if [ -n "${TILELANG_GIT_SHA}" ]; then \
mkdir -p /tmp/tilelang && cd /tmp/tilelang \
&& git init -q \
&& git remote add origin https://github.com/tile-ai/tilelang.git \
&& git fetch --depth 1 origin "${TILELANG_GIT_SHA}" \
&& git checkout -q FETCH_HEAD \
&& git submodule update --init --depth 1 --recursive \
3rdparty/composable_kernel 3rdparty/cutlass 3rdparty/tvm \
&& mkdir -p /opt/tilelang-wheels \
&& CMAKE_BUILD_PARALLEL_LEVEL="${MAX_JOBS}" \
python -m pip wheel . --no-deps --no-build-isolation -w /opt/tilelang-wheels \
&& python -m pip install --no-deps /opt/tilelang-wheels/tilelang-*.whl \
&& rm -rf /tmp/tilelang; \
elif [ -n "${TILELANG_VERSION}" ]; then \
python -m pip install --no-deps "tilelang==${TILELANG_VERSION}"; \
else \
echo "ERROR: set --build-arg TILELANG_GIT_SHA=<commit> (main) or TILELANG_VERSION=<ver> (release)"; \
exit 1; \
fi
# Guard the final stack (GPU-free, runs at build): tilelang must import, the installed
# apache-tvm-ffi must sit in the tilelang wheel's declared ABI range, and torch must still be
# the cu129 build.
COPY scripts/ci/verify_runtime_stack.py /tmp/verify_runtime_stack.py
RUN python /tmp/verify_runtime_stack.py
# ── final ──
FROM tilelang AS final
# Cache dir defaults (host bind-mounts /ci-cache at runtime; pre-created below so they stay
# writable when run unmounted). PIP_NO_CACHE_DIR=0 re-enables caching (build stages disable it).
ENV AGENT_TOOLSDIRECTORY=/home/ci-runner/runner/_work/_tool \
TILELANG_CACHE_DIR=/ci-cache/tilelang \
TILELANG_TMP_DIR=/ci-cache/tilelang/tmp \
TRITON_CACHE_DIR=/ci-cache/triton \
PIP_CACHE_DIR=/ci-cache/pip \
PIP_NO_CACHE_DIR=0
ARG RUNNER_VERSION=2.334.0
# The /ci-cache chown below serves only the unmounted image-verification path
# (`docker run <image> python ...`, see README): ci-runner writes the root-created cache dirs
# directly. In the mounted CI path the host bind-mount shadows these dirs, so the chown is a
# no-op and host ownership wins.
# useradd before WORKDIR: home stays ci-runner-owned, and WORKDIR precedes the relative-path RUN (hadolint).
# Pin UID/GID 1000 so the account deterministically owns the shared /ci-cache bind-mount across
# hosts (it matches the majority of existing warm-cache entries; host normalization preserves them).
# FlashInfer's flashinfer-cubin package keeps its packaged cubin tree under site-packages.
# Some baselines materialize TRTLLM GEMM entries there at runtime, so the unprivileged runner
# must own the writable directories instead of inheriting pip's root-owned directories.
RUN groupadd -g 1000 ci-runner && useradd -u 1000 -g 1000 -m -s /bin/bash ci-runner \
&& FLASHINFER_CUBIN_DIR="$(python -c 'import pathlib, flashinfer_cubin; print(pathlib.Path(flashinfer_cubin.get_cubin_dir()))' 2>/dev/null || true)" \
&& if [ -n "${FLASHINFER_CUBIN_DIR}" ]; then \
mkdir -p "${FLASHINFER_CUBIN_DIR}/flashinfer/trtllm/gemm"; \
find "${FLASHINFER_CUBIN_DIR}" -type d -exec chown ci-runner:ci-runner {} +; \
fi
WORKDIR /home/ci-runner/runner
RUN mkdir -p /home/ci-runner/runner/_work/_tool \
&& curl -fsSL -o runner.tar.gz \
"https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" \
&& tar xzf runner.tar.gz && rm runner.tar.gz \
&& ./bin/installdependencies.sh \
&& apt-get clean && rm -rf /var/lib/apt/lists/* \
&& mkdir -p /ci-cache/pip /ci-cache/tilelang/tmp /ci-cache/triton \
&& chown -R ci-runner:ci-runner /home/ci-runner /ci-cache
# chmod as root before the USER drop.
COPY --chown=ci-runner:ci-runner .github/runner/entrypoint.sh ./entrypoint.sh
RUN chmod +x ./entrypoint.sh
USER ci-runner
ENTRYPOINT ["./entrypoint.sh"]

157
.github/runner/README.md vendored
View File

@ -1,157 +0,0 @@
# CI runner image
Multi-stage image for the self-hosted GPU runner. It bakes a tilelang wheel (compiled once
from a pinned commit, or installed from a PyPI release) plus the test/benchmark stack onto a
public CUDA base, so CI never recompiles tilelang per PR.
Built **manually on a GPU host** (needs `nvcc`), then pushed to `ghcr.io`. It is **not**
built in CI.
## Prerequisites
- An NVIDIA GPU host with a CUDA 12.9-capable driver and `nvcc`.
- Docker with BuildKit enabled (`DOCKER_BUILDKIT=1`).
- Run from the **repository root** — the build context must contain `constraints.txt`,
`scripts/ci/verify_runtime_stack.py`, and `.github/runner/entrypoint.sh` (the Dockerfile
copies all three).
## Build
Provide tilelang one of two ways — pass **exactly one** of these build-args:
- **main commit**: `--build-arg TILELANG_GIT_SHA=<commit>` — shallow-fetches and compiles that
commit. The Dockerfile carries no commit literal; the commit you pass is the single source
of truth, and the image tag records it.
- **release**: `--build-arg TILELANG_VERSION=<version>` — `pip install tilelang==<version>`.
```bash
# from the repository root (main-commit mode)
DOCKER_BUILDKIT=1 docker build \
-f .github/runner/Dockerfile \
--target final \
--build-arg TILELANG_GIT_SHA=65dbc9837beedf6882a40a08e18ea571d92fd6a5 \
-t ghcr.io/tile-ai/tileops-runner:65dbc98 \
.
```
Tag with the tilelang commit's **short SHA** (`:65dbc98`). If you rebuild the same commit,
add a numeric suffix (`:65dbc98-2`).
## Roll out an updated runner image
Changes to this Dockerfile are not picked up by CI automatically. After a Dockerfile change
lands, rebuild and tag the image from a GPU host using the build command above, then push it:
```bash
docker push ghcr.io/tile-ai/tileops-runner:<new-tag>
```
Then redeploy the self-hosted runner launcher to use the new tag (maintainer task, done
outside this repository). Merging the TileOPs PR only changes the image recipe; the live
self-hosted runners keep using their existing image until that manual rollout is done.
### Build args
| Arg | Default | Purpose |
| ------------------ | ------------------------------------------ | ------------------------------------------------------------------------- |
| `TILELANG_GIT_SHA` | *(none)* | tilelang commit to shallow-clone and compile (main mode). |
| `TILELANG_VERSION` | *(none)* | tilelang PyPI version to `pip install` (release mode). |
| `BASE_IMAGE` | `nvidia/cuda:12.9.1-devel-ubuntu22.04` | Public CUDA `devel` base (Python 3.12 via deadsnakes). |
| `MAX_JOBS` | `64` | Parallelism for the tilelang / FA2 / FA3 source builds. |
| `NVCC_THREADS` | `4` | Per-`nvcc` threads. |
| `DEEPGEMM_GIT_SHA` | `c9f8b34dcdacc20aa746b786f983492c51072870` | DeepGEMM commit for the grouped-GEMM benchmark baseline (`v2.1.1.post3`). |
| `RUNNER_VERSION` | `2.334.0` | GitHub Actions runner version baked into `final`. |
Set exactly one of `TILELANG_GIT_SHA` / `TILELANG_VERSION`; the build fails fast if neither is set.
### Stages (`--target`)
| Stage | Contents |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `runtime` | Python 3.12 + torch / torchvision / torchaudio `2.10.0 / 0.25.0 / 2.10.0 +cu129` + triton `3.6.0` + tilelang build/runtime deps (incl. `apache-tvm-ffi 0.1.11`). **No tilelang itself.** |
| `post-fa3` | `runtime` + pytest / pytest-xdist / ruff + FlashAttention-3 (built from the `hopper/` source). |
| `fa2` | `post-fa3` + FlashAttention-2 (`flash-attn 2.8.3`, source-built in its own layer so changes to the bench loop never recompile it). |
| `fullstack` | `fa2` + flash-linear-attention `0.4.2` + vLLM `0.19.1` + mamba-ssm `2.3.1` + DeepGEMM `2.1.1.post3`, then flashinfer-python/-cubin upgraded to `0.6.11.post2` (`--no-deps`, so torch stays +cu129). sgl-kernel is not installed. |
| `tilelang` | `fullstack` + the tilelang wheel (`--no-deps`), then the build-time guard. Built **last** so a SHA bump rebuilds only this layer. |
| `final` | `tilelang` + the GitHub Actions runner (no TileOPs source baked). |
Build an earlier stage for debugging with `--target runtime` (etc.).
The `tilelang` stage ends by running `scripts/ci/verify_runtime_stack.py` (GPU-free): it fails
the build unless tilelang imports, the installed `apache-tvm-ffi` sits inside the tilelang
wheel's declared range, and torch is still the cu129 build.
## Verify the built image
```bash
docker run --rm --gpus all ghcr.io/tile-ai/tileops-runner:65dbc98 python - <<'PY'
import torch
print("torch", torch.__version__, "cuda", torch.version.cuda) # expect 2.10.0+cu129, cuda 12.9
import tilelang; print("tilelang", tilelang.__version__)
import flashinfer, flashinfer_cubin
print("flashinfer", flashinfer.__version__)
from pathlib import Path
cubin_dir = Path(flashinfer_cubin.get_cubin_dir())
probe_dir = cubin_dir / "flashinfer" / "trtllm" / "gemm" / "_tileops_write_probe"
probe_dir.mkdir(parents=True, exist_ok=True)
(probe_dir / "probe.txt").write_text("ok")
print("flashinfer cubin write OK")
import selective_scan_cuda
import mamba_ssm
from mamba_ssm.ops.triton.ssd_combined import mamba_chunk_scan_combined
assert mamba_chunk_scan_combined is not None
print("mamba", mamba_ssm.__version__)
import deep_gemm
print("deep_gemm", deep_gemm.__version__)
# cuBLAS probe: matmul / bmm / einsum on the GPU
a = torch.randn(512, 512, device="cuda", dtype=torch.float16)
b = torch.randn(512, 512, device="cuda", dtype=torch.float16)
assert torch.matmul(a, b).isfinite().all()
ab = torch.randn(8, 128, 128, device="cuda", dtype=torch.float16)
assert torch.bmm(ab, ab).isfinite().all()
assert torch.einsum("bik,bkj->bij", ab, ab).isfinite().all()
print("cuBLAS probe OK")
PY
```
Then run the smoke tests against a checkout of this repo:
```bash
docker run --rm --gpus all -v "$PWD:/src" -w /src \
ghcr.io/tile-ai/tileops-runner:65dbc98 \
bash -c 'scripts/ci/install_tileops.sh && pytest -m smoke'
```
`install_tileops.sh` installs TileOPs `--no-deps` against the baked stack; it fails fast if
tilelang is missing (the image provides it).
## Run as a self-hosted runner
`entrypoint.sh` registers an ephemeral runner (one job per container), then deregisters on
exit. Provide a registration token and the target URL; bind-mount the host cache. The
entrypoint removes `RUNNER_TOKEN` from the environment before the runner starts, so jobs
cannot read the registration token.
```bash
docker run -d --gpus all \
-e RUNNER_URL=https://github.com/tile-ai/TileOPs \
-e RUNNER_TOKEN=<registration-token> \
-e RUNNER_LABELS=self-hosted,tile-ops,venv \
-v <host-cache-dir>:/ci-cache \
ghcr.io/tile-ai/tileops-runner:65dbc98
```
The image sets cache env vars (`TILELANG_CACHE_DIR`, `TRITON_CACHE_DIR`, `PIP_CACHE_DIR`, …)
under `/ci-cache`; the directories are pre-created so the container also works unmounted.
## Bumping the tilelang commit
A commit (or release) bump always rebuilds, but **never edits the Dockerfile**: rebuild with a
new `--build-arg TILELANG_GIT_SHA=<commit>` (or `TILELANG_VERSION=<version>`) and a new
`:<short-sha>` tag, push to `ghcr.io`, then point the runner at the new tag. Because tilelang
is the last stage, only its layer recompiles — the bench layers (FA2 / FA3 / vLLM / …) stay
cached. Switching between a release and a main commit is the same — only the build-arg and tag
change.

61
.github/runner/entrypoint.sh vendored
View File

@ -1,61 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
# Group-writable (0775 dirs / 0664 files) so compiled kernels in the shared /ci-cache are
# reusable across runner UIDs that share the ci-runner group.
umask 002
# Ad-hoc command passthrough: `docker run <image> python ...` / `bash -c ...` runs the
# given command directly (image verification, smoke tests; see README). With no args the
# container registers an ephemeral self-hosted runner (below).
if [ "$#" -gt 0 ]; then
exec "$@"
fi
# ── Validate required environment variables ──────────────────────────────────
: "${RUNNER_TOKEN:?Environment variable RUNNER_TOKEN is required}"
: "${RUNNER_URL:?Environment variable RUNNER_URL is required}"
# Jobs run as children of the listener and inherit its environment; a job that reads
# RUNNER_TOKEN can register a rogue runner with trusted labels while the token is valid.
# Hold the token in an unexported shell variable and drop it from the environment before
# anything else runs. unset first: if reg_token arrived exported from the container
# environment, plain assignment would keep the export attribute and leak the token anyway.
unset reg_token
reg_token="${RUNNER_TOKEN}"
unset RUNNER_TOKEN
RUNNER_NAME="${RUNNER_NAME:-$(hostname)}"
RUNNER_LABELS="${RUNNER_LABELS:-self-hosted,tile-ops,venv}"
RUNNER_WORKDIR="${RUNNER_WORKDIR:-_work}"
# ── Cleanup function — deregister runner on exit ─────────────────────────────
cleanup() {
echo "Removing runner registration..."
./config.sh remove --token "${reg_token}" 2>/dev/null || true
}
trap cleanup EXIT INT TERM
# ── Configure the runner (ephemeral: one job per lifecycle) ──────────────────
./config.sh \
--url "${RUNNER_URL}" \
--token "${reg_token}" \
--name "${RUNNER_NAME}" \
--labels "${RUNNER_LABELS}" \
--work "${RUNNER_WORKDIR}" \
--ephemeral \
--replace \
--unattended \
--disableupdate
# ── Run one job, then exit ───────────────────────────────────────────────────
# Make container stop signals reach the runner so an idle ephemeral runner shuts down promptly
# instead of waiting out docker stop's grace:
# - RUNNER_MANUALLY_TRAP_SIG=1 makes run.sh trap SIGTERM/SIGINT and forward SIGINT to the
# listener process group (its default mode does not trap, so the signal is otherwise lost).
# - exec makes run.sh PID 1 so docker stop's SIGTERM is delivered to it; a bash wrapper as
# PID 1 would ignore an untrapped SIGTERM and stall until SIGKILL.
# Trade-off: a stop signal cancels an in-progress job rather than draining it.
export RUNNER_MANUALLY_TRAP_SIG=1
exec ./run.sh

110
.github/workflows/auto-label.yml vendored
View File

@ -1,110 +0,0 @@
name: auto-label
permissions:
contents: read
issues: write
pull-requests: write
on:
issues:
types: [opened, edited]
pull_request_target:
types: [opened, edited]
jobs:
apply-label:
runs-on: ubuntu-latest
steps:
- name: Skip non-title edits
id: guard
env:
EVENT_NAME: ${{ github.event_name }}
EVENT_ACTION: ${{ github.event.action }}
EVENT_PATH: ${{ github.event_path }}
run: |
set -euo pipefail
if [[ "$EVENT_ACTION" != "edited" ]]; then
echo "should_run=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if ! jq -e '.changes | has("title")' "$EVENT_PATH" >/dev/null; then
echo "Body-only edit detected; skipping label sync."
echo "should_run=false" >> "$GITHUB_OUTPUT"
exit 0
fi
echo "should_run=true" >> "$GITHUB_OUTPUT"
- uses: actions/checkout@v4
if: ${{ steps.guard.outputs.should_run == 'true' }}
with:
ref: ${{ github.event.pull_request.base.sha || github.sha }}
sparse-checkout: .claude/conventions
sparse-checkout-cone-mode: false
- name: Extract type from title and apply label
if: ${{ steps.guard.outputs.should_run == 'true' }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
EVENT_NAME: ${{ github.event_name }}
ISSUE_TITLE: ${{ github.event.issue.title }}
ISSUE_NUMBER: ${{ github.event.issue.number }}
PR_TITLE: ${{ github.event.pull_request.title }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: |
source .claude/conventions/types.sh
LABEL=""
if [ "$EVENT_NAME" = "issues" ]; then
TITLE="$ISSUE_TITLE"
NUMBER="$ISSUE_NUMBER"
GH_CMD="issue"
else
TITLE="$PR_TITLE"
NUMBER="$PR_NUMBER"
GH_CMD="pr"
fi
# Try commit/PR-style types first (e.g. [BugFix])
TYPE=$(echo "$TITLE" | grep -oP "^\[\K(${COMMIT_PR_TYPES})(?=\])" || true)
if [ -n "$TYPE" ]; then
LABEL="${TYPE_TO_LABEL[$TYPE]:-}"
elif [ "$EVENT_NAME" = "issues" ]; then
# Fall back to ALL-CAPS issue types (e.g. [BUG])
TYPE=$(echo "$TITLE" | grep -oP "^\[\K(${ISSUE_TYPES})(?=\])" || true)
if [ -n "$TYPE" ]; then
LABEL="${ISSUE_TYPE_TO_LABEL[$TYPE]:-}"
fi
fi
if [ -z "$TYPE" ] || [ -z "$LABEL" ]; then
echo "No recognized type tag in title, skipping label"
exit 0
fi
echo "Detected type: $TYPE -> label: $LABEL"
# Check if correct label already applied
CURRENT_LABELS=$(gh "$GH_CMD" view "$NUMBER" --repo "$GITHUB_REPOSITORY" --json labels --jq '.labels[].name')
if echo "$CURRENT_LABELS" | grep -qx "$LABEL"; then
echo "Label '$LABEL' already applied, skipping"
exit 0
fi
# Remove any stale type labels
for l in $ALL_TYPE_LABELS; do
if echo "$CURRENT_LABELS" | grep -qx "$l"; then
gh "$GH_CMD" edit "$NUMBER" --repo "$GITHUB_REPOSITORY" --remove-label "$l"
fi
done
# Ensure label exists (create if missing)
if ! gh label list --repo "$GITHUB_REPOSITORY" --search "$LABEL" --json name --jq '.[].name' | grep -qx "$LABEL"; then
gh label create "$LABEL" --repo "$GITHUB_REPOSITORY" --description "Auto-created by labeler" --force
echo "Created label: $LABEL"
fi
# Apply the matching label
gh "$GH_CMD" edit "$NUMBER" --repo "$GITHUB_REPOSITORY" --add-label "$LABEL"
echo "Applied label: $LABEL"

650
.github/workflows/gpu-smoke.yml vendored
View File

@ -1,650 +0,0 @@
name: GPU Smoke
permissions:
contents: read
checks: read
pull-requests: read
on:
push:
branches: [main, testbed]
tags: ["v*"]
pull_request:
branches: [main, testbed]
types: [opened, synchronize, reopened, ready_for_review]
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
ci-prereq:
runs-on: ubuntu-latest
outputs:
skip: ${{ steps.check.outputs.skip }}
steps:
- name: Wait for CI prerequisites
id: check
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
EVENT_ACTION: ${{ github.event.action || '' }}
IS_DRAFT: ${{ github.event.pull_request.draft || false }}
HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
run: |
set -euo pipefail
if [[ "$EVENT_NAME" != "pull_request" ]]; then
echo "skip=false" >> "$GITHUB_OUTPUT"
exit 0
fi
if [[ "$IS_DRAFT" == "true" ]]; then
echo "Draft PR; skipping GPU smoke."
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
required_checks=(pre-commit gitleaks actionlint)
deadline=$((SECONDS + 900))
while true; do
checks_json=$(gh api "repos/${{ github.repository }}/commits/$HEAD_SHA/check-runs")
latest_checks=$(echo "$checks_json" | jq '
.check_runs
| group_by(.name)
| map(max_by(.id))
')
pending="false"
failed=""
for check_name in "${required_checks[@]}"; do
status=$(echo "$latest_checks" | jq -r --arg name "$check_name" '
map(select(.name == $name)) | first | .status // ""
')
conclusion=$(echo "$latest_checks" | jq -r --arg name "$check_name" '
map(select(.name == $name)) | first | .conclusion // ""
')
if [[ -z "$status" || "$status" != "completed" ]]; then
pending="true"
continue
fi
if [[ "$conclusion" == "skipped" ]]; then
pending="true"
continue
fi
if [[ "$conclusion" != "success" ]]; then
failed="$check_name"
break
fi
done
title_status=$(echo "$latest_checks" | jq -r '
map(select(.name == "validate-pr-title")) | first | .status // ""
')
title_conclusion=$(echo "$latest_checks" | jq -r '
map(select(.name == "validate-pr-title")) | first | .conclusion // ""
')
if [[ -n "$title_status" && "$title_status" == "completed" && "$title_conclusion" != "success" && "$title_conclusion" != "skipped" ]]; then
failed="validate-pr-title"
elif [[ -n "$title_status" && "$title_status" != "completed" ]]; then
pending="true"
fi
if [[ -n "$failed" ]]; then
echo "CI prerequisite failed: $failed"
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if [[ "$pending" == "false" ]]; then
break
fi
if (( SECONDS >= deadline )); then
echo "Timed out waiting for CI prerequisites."
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
sleep 10
done
if [[ "$EVENT_ACTION" == "ready_for_review" ]]; then
CHECKS=$(echo "$latest_checks" | jq '
[
.[]
| select(.name == "gpu-smoke")
| .conclusion
]
')
ALL_SUCCESS=$(echo "$CHECKS" | jq 'all(. == "success")')
COUNT=$(echo "$CHECKS" | jq 'length')
if [[ "$ALL_SUCCESS" == "true" && "$COUNT" -ge 1 ]]; then
echo "GPU smoke already passed for SHA $HEAD_SHA; skipping"
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
fi
echo "skip=false" >> "$GITHUB_OUTPUT"
security-policy:
needs: ci-prereq
if: ${{ github.repository == 'tile-ai/TileOPs' && needs.ci-prereq.outputs.skip != 'true' }}
runs-on: ubuntu-latest
outputs:
is_fork: ${{ steps.policy.outputs.is_fork }}
allow_fast_path: ${{ steps.policy.outputs.allow_fast_path }}
skip_gpu_smoke: ${{ steps.policy.outputs.skip_gpu_smoke }}
scope: ${{ steps.policy.outputs.scope }}
pytest_targets: ${{ steps.policy.outputs.pytest_targets }}
full_tier_targets: ${{ steps.policy.outputs.full_tier_targets }}
reason: ${{ steps.policy.outputs.reason }}
steps:
- name: Checkout test files
uses: actions/checkout@v4
with:
repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
ref: ${{ github.event.pull_request.head.sha || github.sha }}
persist-credentials: false
fetch-depth: 1
# Whole tests/ tree: the policy step existence-checks changed
# test files before selecting them.
sparse-checkout: |
tests
- name: Determine GPU policy and pytest scope
id: policy
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
PR_NUMBER: ${{ github.event.pull_request.number || '' }}
HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
BASE_REPO: ${{ github.repository }}
PR_AUTHOR: ${{ github.event.pull_request.user.login || '' }}
run: |
set -euo pipefail
add_target() {
local target="$1"
[[ -e "$target" ]] || return 0
if [[ -z "${seen_targets[$target]:-}" ]]; then
selected_targets+=("$target")
seen_targets["$target"]=1
fi
}
is_fork="false"
allow_fast_path="true"
skip_gpu_smoke="false"
scope="full-smoke"
pytest_targets="tests"
full_tier_targets=""
reason="GPU smoke will run."
# `is_fork` is the trust boundary that routes runs-on (resident shared-cache pool vs
# overlay-isolated fork pool) and the trusted-action ref below. Derive it from the PR
# author's collaborator permission on the base repo, NOT author_association: private org
# membership can report a non-member association (would wrongly isolate trusted authors),
# and COLLABORATOR is not proof of write access (would wrongly trust read-only authors).
# Only write/maintain/admin are trusted; everyone else — and any lookup failure — fails
# closed to the fork pool.
if [[ "$EVENT_NAME" != "pull_request" ]]; then
is_fork="false" # push / schedule / workflow_dispatch are trusted
elif [[ "$HEAD_REPO" == "$BASE_REPO" ]]; then
is_fork="false" # same-repo branch PR: author already has base-repo write access
else
role=$(gh api "repos/${BASE_REPO}/collaborators/${PR_AUTHOR}/permission" \
--jq '.role_name // .permission' 2>/dev/null || echo "")
case "$role" in
admin|maintain|write)
is_fork="false"
echo "Author ${PR_AUTHOR} has '${role}' on ${BASE_REPO}; routing to resident pool."
;;
*)
is_fork="true"
echo "Author ${PR_AUTHOR} permission='${role:-unknown}'; routing to fork pool (fail-closed)."
;;
esac
fi
if [[ "$EVENT_NAME" == "pull_request" ]]; then
mapfile -t changed_files < <(
gh api "repos/${BASE_REPO}/pulls/${PR_NUMBER}/files" --paginate --jq '.[].filename'
)
echo "Changed files (${#changed_files[@]}):"
printf ' - %s\n' "${changed_files[@]}"
# Changed test files (basename test_*.py, existing at head);
# the PR gate runs these under -m "smoke or full".
declare -A seen_full_targets=()
full_targets_list=()
for path in "${changed_files[@]}"; do
case "$path" in
tests/*/test_*.py|tests/test_*.py)
if [[ -e "$path" && -z "${seen_full_targets[$path]:-}" ]]; then
full_targets_list+=("$path")
seen_full_targets["$path"]=1
fi
;;
esac
done
full_tier_targets="${full_targets_list[*]}"
# Fast path: if no Python files changed, GPU smoke is unnecessary.
# For fork PRs, also check for high-risk files (workflow, build
# config, etc.) that must still trigger GPU smoke even without
# Python changes to avoid security bypass.
has_python="false"
has_high_risk="false"
for path in "${changed_files[@]}"; do
if [[ "$path" == *.py ]]; then
has_python="true"
fi
if [[ "$is_fork" == "true" ]]; then
case "$path" in
.github/workflows/*|pyproject.toml|requirements*.txt|requirements*.in|setup.py|setup.cfg|Dockerfile*|scripts/bootstrap*|scripts/install*)
has_high_risk="true"
;;
esac
fi
done
if [[ "$has_python" == "false" && "$has_high_risk" == "false" ]]; then
skip_gpu_smoke="true"
reason="No Python files changed; GPU smoke skipped."
{
echo "skip_gpu_smoke=$skip_gpu_smoke"
echo "is_fork=$is_fork"
echo "allow_fast_path=$allow_fast_path"
echo "scope=skip"
echo "pytest_targets="
echo "full_tier_targets="
echo "reason=$reason"
} >> "$GITHUB_OUTPUT"
exit 0
fi
if [[ "$has_python" == "false" && "$has_high_risk" == "true" ]]; then
# Fork PR changed high-risk build/infra files but no Python.
# Run full GPU smoke to validate the environment is not broken.
skip_gpu_smoke="false"
allow_fast_path="false"
scope="full-smoke"
pytest_targets="tests"
reason="Fork PR changed high-risk file(s) without Python changes; running full GPU smoke."
{
echo "skip_gpu_smoke=$skip_gpu_smoke"
echo "is_fork=$is_fork"
echo "allow_fast_path=$allow_fast_path"
echo "scope=$scope"
echo "pytest_targets=$pytest_targets"
echo "full_tier_targets=$full_tier_targets"
echo "reason=$reason"
} >> "$GITHUB_OUTPUT"
exit 0
fi
scope="skip"
declare -A seen_targets=()
selected_targets=()
for path in "${changed_files[@]}"; do
if [[ "$is_fork" == "true" ]]; then
case "$path" in
.github/ISSUE_TEMPLATE/*|.github/pull_request_template.md|tileops/ops/*.py|tests/ops/test_*.py|*.md)
;;
.github/workflows/*|pyproject.toml|requirements*.txt|requirements*.in|setup.py|setup.cfg|Dockerfile*|scripts/bootstrap*|scripts/install*)
allow_fast_path="false"
reason="Fork PR changed a high-risk file: $path"
;;
*)
allow_fast_path="false"
reason="Fork PR changed a file outside the future fast-path policy: $path"
;;
esac
fi
case "$path" in
.github/ISSUE_TEMPLATE/*|.github/pull_request_template.md|.github/workflows/auto-label.yml|.github/workflows/stale-issues.yml|*.md)
;;
tests/ops/test_*.py)
add_target "$path"
if [[ "$scope" == "skip" ]]; then
scope="targeted"
fi
;;
*)
scope="full-smoke"
reason="Shared or unsupported path $path; falling back to full smoke."
break
;;
esac
done
if [[ "$scope" == "skip" ]]; then
if [[ "$allow_fast_path" == "true" ]]; then
# All changed files matched docs/metadata patterns; skip GPU smoke
# for both trusted and fork PRs.
skip_gpu_smoke="true"
pytest_targets=""
reason="Docs-only or metadata-only change; GPU smoke skipped."
elif [[ "$is_fork" == "true" ]]; then
scope="full-smoke"
pytest_targets="tests"
reason="${reason}; current policy still runs GPU smoke."
else
skip_gpu_smoke="true"
pytest_targets=""
reason="Docs-only or metadata-only change; GPU smoke skipped."
fi
elif [[ "$scope" == "targeted" ]]; then
if [[ "${#selected_targets[@]}" -eq 0 ]]; then
scope="full-smoke"
pytest_targets="tests"
reason="No mapped tests were collected; falling back to full smoke."
else
pytest_targets="${selected_targets[*]}"
if [[ "$is_fork" == "true" && "$allow_fast_path" != "true" ]]; then
reason="${reason}; current policy still runs targeted GPU smoke."
else
reason="Running targeted GPU smoke tests."
fi
fi
fi
fi
{
echo "is_fork=$is_fork"
echo "allow_fast_path=$allow_fast_path"
echo "skip_gpu_smoke=$skip_gpu_smoke"
echo "scope=$scope"
echo "pytest_targets=$pytest_targets"
echo "full_tier_targets=$full_tier_targets"
echo "reason=$reason"
} >> "$GITHUB_OUTPUT"
gpu-smoke:
needs: [ci-prereq, security-policy]
if: ${{ github.repository == 'tile-ai/TileOPs' && needs.ci-prereq.outputs.skip != 'true' && needs.security-policy.outputs.skip_gpu_smoke != 'true' }}
# Trust-level routing: untrusted (external) PRs go to the on-demand `fork` pool whose
# runner mounts an overlay cache (read-only shared lower + throwaway upper) so their cache
# writes never reach the shared cache. Trusted runs use the resident shared-cache pool.
runs-on: ${{ needs.security-policy.outputs.is_fork == 'true' && fromJSON('["self-hosted", "tile-ops", "fork"]') || fromJSON('["self-hosted", "tile-ops", "nightly"]') }}
# Hard backstop so a wedged kernel can never hold the single runner indefinitely; the
# per-test --timeout below is the first line of defense, this is the ceiling.
timeout-minutes: 90
steps:
- name: Checkout code for fork PR
if: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository }}
uses: actions/checkout@v4
with:
repository: ${{ github.event.pull_request.head.repo.full_name }}
ref: ${{ github.event.pull_request.head.sha }}
persist-credentials: false
fetch-depth: 1
- name: Checkout code for trusted branch
if: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }}
uses: actions/checkout@v4
with:
repository: ${{ github.repository }}
ref: ${{ github.event.pull_request.head.sha || github.sha }}
persist-credentials: false
fetch-depth: 1
# Trusted actions checkout: must run AFTER the workspace checkouts so
# their `git clean -ffdx` pass does not wipe `.trusted/`. Anything
# loaded via `uses: ./...` from here on MUST resolve through
# `.trusted/` so that fork-controlled PR code cannot substitute the
# composite action.
#
# Ref selection reuses the single `is_fork` trust result computed in security-policy
# (collaborator-permission based), so it stays consistent with runs-on:
#
# - untrusted (is_fork == true: external author or lookup failure)
# → `base.sha` (authoritative; the untrusted author cannot alter it)
# - trusted (same-repo PR, write+ member-fork PR, or push)
# → head sha (author already has write access to the base repo)
- name: Checkout trusted actions (base repo)
uses: actions/checkout@v4
with:
repository: ${{ github.repository }}
ref: ${{ needs.security-policy.outputs.is_fork == 'true' && github.event.pull_request.base.sha || github.event.pull_request.head.sha || github.sha }}
path: .trusted
persist-credentials: false
fetch-depth: 1
sparse-checkout: |
.github/actions
- name: Reclaim runner disk
uses: ./.trusted/.github/actions/reclaim-runner-disk
with:
reclaim-below-gib: "10"
cache-trim-cooldown-minutes: "120"
cache-dirs: |
/ci-cache/triton
/ci-cache/pip
/ci-cache/tilelang/tmp
atomic-cache-dirs: |
/ci-cache/tilelang/autotuner
df-paths: "/ci-cache"
# Daily runner-maintenance.yml owns the destructive autotuner
# age-trim; per-PR runs only need sentinel-repair (still
# unconditional) so they don't evict tuning artefacts the next
# PR would otherwise reuse.
skip-atomic-age-trim: "true"
- name: Validate GPU frequency
run: |
set -euo pipefail
TARGET_CLOCK_MHZ=1500
RETRIES=5
SLEEP_SECONDS=2
if ! command -v nvidia-smi >/dev/null 2>&1; then
echo "::error::nvidia-smi is not available on this runner."
exit 1
fi
echo "Detected GPUs:"
nvidia-smi -L
attempt=1
while [ "$attempt" -le "$RETRIES" ]; do
echo "Validation attempt ${attempt}/${RETRIES}"
mismatch=0
while IFS=',' read -r gpu_index gpu_name gpu_clock; do
gpu_index="$(echo "$gpu_index" | xargs)"
gpu_name="$(echo "$gpu_name" | xargs)"
gpu_clock="$(echo "$gpu_clock" | xargs)"
echo "GPU ${gpu_index} (${gpu_name}) clock: ${gpu_clock} MHz"
if [[ "$gpu_clock" != "$TARGET_CLOCK_MHZ" ]]; then
mismatch=1
fi
done < <(nvidia-smi --query-gpu=index,name,clocks.current.graphics --format=csv,noheader,nounits)
if [[ "$mismatch" -eq 0 ]]; then
echo "All GPUs locked at ${TARGET_CLOCK_MHZ} MHz."
break
fi
if [[ "$attempt" -eq "$RETRIES" ]]; then
echo "::error::GPU frequency validation failed. Expected ${TARGET_CLOCK_MHZ} MHz on all GPUs."
exit 1
fi
sleep "$SLEEP_SECONDS"
attempt=$((attempt + 1))
done
- name: Install TileOPs (image-baked stack, --no-deps)
# tilelang + the full runtime/dev stack are baked into the runner image; this installs
# only tileops itself (editable, --no-deps) against constraints.txt. Fork write
# isolation is provided by the `fork` runner pool's overlay cache, not the workflow.
run: bash scripts/ci/install_tileops.sh
- name: Run tests
env:
EVENT_NAME: ${{ github.event_name }}
TEST_SCOPE: ${{ needs.security-policy.outputs.scope }}
PYTEST_TARGETS: ${{ needs.security-policy.outputs.pytest_targets }}
FULL_TIER_TARGETS: ${{ needs.security-policy.outputs.full_tier_targets }}
run: |
set -euo pipefail
# Per-test hang guard: a stuck kernel (e.g. a cold-compile loop or a spinning CUDA
# call) fails as a timeout with a thread dump instead of wedging the job. --no-deps
# tileops install doesn't bring this; pin via constraints.txt so the install is
# deterministic (the runner image should bake it and drop this install — follow-up).
python3 -m pip install -q -c constraints.txt pytest-timeout
current_dir="$(pwd)"
export PYTHONPATH="${current_dir}${PYTHONPATH:+:$PYTHONPATH}"
echo "PYTHONPATH=$PYTHONPATH"
python3 - <<'PY'
import os
print("Runtime cache env:")
for key in ("TILELANG_CACHE_DIR", "TILELANG_TMP_DIR", "TRITON_CACHE_DIR"):
print(f"{key}={os.environ.get(key)}")
PY
TEST_MARK_EXPR="smoke or full"
if [[ "$EVENT_NAME" == "pull_request" ]]; then
TEST_MARK_EXPR="smoke"
fi
echo "Resolved test scope: ${TEST_SCOPE}"
echo "Resolved pytest targets: ${PYTEST_TARGETS}"
echo "Diff-scoped full-tier targets: ${FULL_TIER_TARGETS:-<none>}"
read -r -a TARGETS <<< "${PYTEST_TARGETS}"
read -r -a FULL_TARGETS <<< "${FULL_TIER_TARGETS:-}"
# Diff-scoped full-tier PR gate: changed test files run
# -m "smoke or full"; everything else stays smoke-only for latency.
# Residual gap (accepted): a PR breaking full-tier cases in files it
# does not touch is only caught by the push-to-main run — closing it
# would cost a full suite per PR. Push-tier failure triage: bisect
# PRs merged since the last green push run.
if [[ "$EVENT_NAME" == "pull_request" && "${#FULL_TARGETS[@]}" -gt 0 ]]; then
if [[ "$TEST_SCOPE" == "targeted" ]]; then
# Targeted scope already selects exactly these files; promote
# the single pass instead of running twice.
TEST_MARK_EXPR="smoke or full"
FULL_TARGETS=()
else
echo "Running diff-scoped pytest -m \"smoke or full\" on ${#FULL_TARGETS[@]} changed test file(s)"
diff_status=0
python3 -m pytest -q "${FULL_TARGETS[@]}" -m "smoke or full" \
--timeout=600 --timeout-method=thread \
--junit-xml=gpu_smoke_full_results.xml | tee gpu_smoke_full.log \
|| diff_status=$?
# Exit 5 = no tests collected; not a failure.
if [[ "$diff_status" -ne 0 && "$diff_status" -ne 5 ]]; then
exit "$diff_status"
fi
# Already ran above; exclude from the smoke pass.
for full_target in "${FULL_TARGETS[@]}"; do
TARGETS+=("--ignore=${full_target}")
done
fi
fi
# Marker the baseline pass actually executes; the report step
# labels the tier from it.
echo "EXECUTED_MARK_EXPR=${TEST_MARK_EXPR}" >> "$GITHUB_ENV"
echo "Running pytest -m \"${TEST_MARK_EXPR}\" on ${#TARGETS[@]} target(s)"
python3 -m pytest -q "${TARGETS[@]}" -m "${TEST_MARK_EXPR}" \
--timeout=600 --timeout-method=thread \
--junit-xml=gpu_smoke_results.xml | tee gpu_smoke.log
- name: Generate gpu-smoke report
if: ${{ always() && (hashFiles('gpu_smoke_results.xml') != '' || hashFiles('gpu_smoke_full_results.xml') != '') }}
env:
EVENT_NAME: ${{ github.event_name }}
run: |
set -euo pipefail
# Set by Run tests once the tier is resolved; fall back to the
# event default only if Run tests died before resolving it.
if [[ -n "${EXECUTED_MARK_EXPR:-}" ]]; then
REPORT_TARGET="${EXECUTED_MARK_EXPR// or /, }"
else
REPORT_TARGET="smoke, full"
if [[ "$EVENT_NAME" == "pull_request" ]]; then
REPORT_TARGET="smoke"
fi
fi
if [[ -f gpu_smoke_results.xml ]]; then
python3 scripts/gpu_smoke_report.py \
--test-xml gpu_smoke_results.xml \
--target "${REPORT_TARGET}" \
--output gpu_smoke_main_report.md \
|| echo "::warning::gpu_smoke_report.py failed"
fi
if [[ -f gpu_smoke_full_results.xml ]]; then
python3 scripts/gpu_smoke_report.py \
--test-xml gpu_smoke_full_results.xml \
--target "smoke, full (diff-scoped changed test files)" \
--output gpu_smoke_diff_report.md \
|| echo "::warning::gpu_smoke_report.py failed"
fi
for part in gpu_smoke_main_report.md gpu_smoke_diff_report.md; do
if [[ -f "$part" ]]; then
cat "$part" >> gpu_smoke_report.md
fi
done
- name: Post gpu-smoke report to step summary
if: ${{ always() && hashFiles('gpu_smoke_report.md') != '' }}
run: cat gpu_smoke_report.md >> "$GITHUB_STEP_SUMMARY"
- name: Cache stats
if: ${{ always() }}
env:
IS_FORK: ${{ needs.security-policy.outputs.is_fork }}
run: |
# Cache stats is informational only. Do NOT use `set -e`: a transient
# failure in find/du/wc (e.g. a cache file vanishing mid-scan due to a
# concurrent cleanup) must not fail the gpu-smoke job.
set -u
echo "=== Cache stats (is_fork=${IS_FORK}) ==="
for cache_dir in "${TILELANG_CACHE_DIR:-}" "${TRITON_CACHE_DIR:-}" "${PIP_CACHE_DIR:-}"; do
if [[ -z "${cache_dir}" ]]; then
continue
fi
if [[ -d "${cache_dir}" ]]; then
# Tolerate errors from find/du/wc; emit "unknown" when a scan
# fails rather than aborting the step. Enable pipefail inside
# the subshell so a find/du failure upstream of wc/cut
# propagates to the || fallback instead of being masked.
file_count=$(set -o pipefail; find "${cache_dir}" -type f 2>/dev/null | wc -l 2>/dev/null) || file_count="unknown"
cache_size=$(set -o pipefail; du -sh "${cache_dir}" 2>/dev/null | cut -f1 2>/dev/null) || cache_size="unknown"
if [[ -z "${file_count}" ]]; then file_count="unknown"; fi
if [[ -z "${cache_size}" ]]; then cache_size="unknown"; fi
echo "${cache_dir}: ${file_count} files, ${cache_size}"
else
echo "${cache_dir}: does not exist"
fi
done
- name: Upload artifacts
uses: actions/upload-artifact@v4
if: always()
with:
name: gpu_smoke.log
path: |
gpu_smoke.log
gpu_smoke_results.xml
gpu_smoke_full.log
gpu_smoke_full_results.xml
gpu_smoke_report.md
retention-days: 7

127
.github/workflows/manifest-stats.yml vendored
View File

@ -1,127 +0,0 @@
name: Manifest Stats
permissions:
contents: write # publish-stats job pushes to the `stats` branch
on:
push:
branches: [main]
pull_request:
branches: [main]
types: [opened, synchronize, reopened, ready_for_review]
schedule:
# Daily fallback in case a push event was missed.
- cron: "17 6 * * *"
workflow_dispatch:
# All publish-capable events (push, schedule, workflow_dispatch) share a
# single mutex so a slower scheduled run can never overwrite a newer push.
# PRs are isolated per number.
concurrency:
group: >-
${{ github.event_name == 'pull_request'
&& format('manifest-stats-pr-{0}', github.event.pull_request.number)
|| 'manifest-stats-publish' }}
cancel-in-progress: false
jobs:
report:
name: Manifest stats summary
runs-on: ubuntu-latest
steps:
- name: Checkout PR head
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Install minimal dependencies
run: pip install pyyaml
- name: Generate current stats
env:
PYTHONPATH: ${{ github.workspace }}
run: |
set -euo pipefail
mkdir -p _stats
python scripts/manifest_stats.py --format json --output _stats/current.json
python scripts/manifest_stats.py --format md --output _stats/current.md
- name: Compute diff vs base branch (PR only)
id: diff
if: github.event_name == 'pull_request'
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
set -euo pipefail
git fetch --depth=1 origin "$BASE_SHA"
git worktree add /tmp/base "$BASE_SHA"
# Use the PR-branch script (it may not exist on base) but load
# tileops.manifest from the base worktree so the snapshot reflects
# the base ref's manifest contents.
PYTHONPATH=/tmp/base python scripts/manifest_stats.py \
--format json --output /tmp/base-stats.json
git worktree remove --force /tmp/base
PYTHONPATH=${{ github.workspace }} python scripts/manifest_stats.py \
--format md --diff /tmp/base-stats.json --output _stats/comment.md
- name: Write Job Summary
run: |
if [[ -f _stats/comment.md ]]; then
cat _stats/comment.md >> "$GITHUB_STEP_SUMMARY"
else
cat _stats/current.md >> "$GITHUB_STEP_SUMMARY"
fi
- name: Upload stats artifact
uses: actions/upload-artifact@v4
with:
name: manifest-stats
path: _stats/
retention-days: 14
publish-stats:
name: Publish stats to orphan branch
needs: report
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Install minimal dependencies
run: pip install pyyaml
- name: Generate publish payload
env:
PYTHONPATH: ${{ github.workspace }}
run: |
set -euo pipefail
mkdir -p _publish
python scripts/manifest_stats.py --format json --output _publish/manifest-stats.json
python scripts/manifest_stats.py --format md --output _publish/manifest-stats.md
python scripts/manifest_stats.py --badge-output _publish
- name: Push to stats branch
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_branch: stats
publish_dir: _publish
force_orphan: true
commit_message: "chore(stats): refresh manifest stats from ${{ github.sha }}"
user_name: "github-actions[bot]"
user_email: "41898282+github-actions[bot]@users.noreply.github.com"

438
.github/workflows/nightly.yml vendored
View File

@ -1,438 +0,0 @@
name: Nightly
permissions: read-all
on:
workflow_dispatch:
schedule:
- cron: "0 18 * * *"
concurrency:
group: nightly-${{ github.ref }}
cancel-in-progress: false
# ---------------------------------------------------------------------------
# All nightly jobs run on a containerized self-hosted runner (label: nightly).
# The runner container is provisioned by the host with GPU access, persistent
# cache mounts, and the runtime environment variables below.
# ---------------------------------------------------------------------------
env:
TILELANG_CACHE_DIR: /ci-cache/tilelang
TILELANG_TMP_DIR: /ci-cache/tilelang/tmp
TRITON_CACHE_DIR: /ci-cache/triton
PIP_CACHE_DIR: /ci-cache/pip
MAX_JOBS: "64"
jobs:
# =========================================================================
# Phase 1 — Benchmark (exclusive GPU access for accurate profiling)
#
# The persistent /ci-cache (TILELANG_CACHE_DIR) carries compiled kernels and
# autotuner results across runs; benchmark setup repopulates it on a cold key
# (e.g. after a tilelang bump). No separate warmup phase. For a one-off cold
# cache, run scripts/warmup_kernel_cache.py manually.
# =========================================================================
benchmark:
if: ${{ github.repository == 'tile-ai/TileOPs' && (github.event_name == 'schedule' || github.ref == 'refs/heads/main') }}
timeout-minutes: 120
runs-on: [self-hosted, tile-ops, nightly]
env:
# Keep the long benchmark suite from fragmenting CUDA allocator segments
# before late large MoE input tensors allocate 14-21 GiB weight blocks.
PYTORCH_ALLOC_CONF: expandable_segments:True
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Verify nightly runner environment
run: scripts/ci/verify_nightly_runner.sh
shell: bash
- name: Install TileOPs (image-baked stack, --no-deps)
run: bash scripts/ci/install_tileops.sh
shell: bash
- name: Run benchmark ops
id: benchmark_tests
continue-on-error: true
run: |
set -euo pipefail
# Validate GPU frequency
TARGET_CLOCK_MHZ=1500
RETRIES=5
for i in $(seq 1 "${RETRIES}"); do
gpu_clock=$(nvidia-smi --query-gpu=clocks.current.graphics --format=csv,noheader,nounits | xargs)
echo "GPU clock: ${gpu_clock} MHz (attempt ${i}/${RETRIES})"
if [ "${gpu_clock}" = "${TARGET_CLOCK_MHZ}" ]; then break; fi
if [ "${i}" -eq "${RETRIES}" ]; then
echo "::error::GPU frequency validation failed. Expected ${TARGET_CLOCK_MHZ} MHz."
exit 1
fi
sleep 2
done
export PYTHONPATH="${GITHUB_WORKSPACE}${PYTHONPATH:+:$PYTHONPATH}"
echo "Runtime cache env:"
echo "TILELANG_CACHE_DIR=${TILELANG_CACHE_DIR:-}"
echo "TILELANG_TMP_DIR=${TILELANG_TMP_DIR:-}"
echo "TRITON_CACHE_DIR=${TRITON_CACHE_DIR:-}"
set -o pipefail
python3 -m pip install -q -c constraints.txt pytest-timeout
python3 -m pytest -q benchmarks/ops --timeout=900 --timeout-method=thread --junit-xml=bench_results.xml | tee tileops_benchmarks.log
if [ -f profile_run.log ]; then
{ echo; echo "===== profile_run.log summary ====="; cat profile_run.log; } >> tileops_benchmarks.log
else
echo "::warning::profile_run.log not found; benchmark may have failed partially" >> tileops_benchmarks.log
fi
shell: bash
- name: Upload benchmark artifacts
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: tileops_benchmark_${{ github.run_id }}
path: |
tileops_benchmarks.log
bench_results.xml
if-no-files-found: warn
retention-days: 14
- name: Fail benchmark job if pytest failed
if: ${{ always() && steps.benchmark_tests.outcome == 'failure' }}
run: exit 1
shell: bash
# =========================================================================
# Phase 2 — Op correctness tests (serial after benchmark for GPU exclusivity)
#
# Runs the `full` and `nightly` tiers. The `smoke` tier is the PR critical
# path and is already validated by every PR and every push-to-main gpu-smoke
# (-m "smoke or full"), so re-running it here is pure duplication. `nightly`
# is the heavy tier that runs nowhere else; `full` stays as a scheduled
# regression / environment-drift net.
# =========================================================================
op_test:
if: ${{ always() && github.repository == 'tile-ai/TileOPs' && (github.event_name == 'schedule' || github.ref == 'refs/heads/main') }}
needs: [benchmark]
timeout-minutes: 180
runs-on: [self-hosted, tile-ops, nightly]
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Verify nightly runner environment
run: scripts/ci/verify_nightly_runner.sh
shell: bash
- name: Install TileOPs (image-baked stack, --no-deps)
run: bash scripts/ci/install_tileops.sh
shell: bash
- name: Run full op tests
id: op_tests
continue-on-error: true
run: |
set -euo pipefail
export PYTHONPATH="${GITHUB_WORKSPACE}${PYTHONPATH:+:$PYTHONPATH}"
echo "Runtime cache env:"
echo "TILELANG_CACHE_DIR=${TILELANG_CACHE_DIR:-}"
echo "TILELANG_TMP_DIR=${TILELANG_TMP_DIR:-}"
echo "TRITON_CACHE_DIR=${TRITON_CACHE_DIR:-}"
set -o pipefail
python3 -m pip install -q -c constraints.txt pytest-timeout
python3 -m pytest tests/ -m "full or nightly" -v --tb=short --timeout=900 --timeout-method=thread --junit-xml=test_results.xml | tee tileops_op_test.log
shell: bash
- name: Download benchmark artifacts
uses: actions/download-artifact@v4
if: ${{ always() }}
with:
name: tileops_benchmark_${{ github.run_id }}
continue-on-error: true
- name: Resolve previous perf history artifact
id: perf_history_source
if: ${{ always() }}
uses: actions/github-script@v7
with:
script: |
const { owner, repo } = context.repo;
const workflowId = "nightly.yml";
const currentRunId = context.runId;
const runs = await github.paginate(
github.rest.actions.listWorkflowRuns,
{
owner,
repo,
workflow_id: workflowId,
branch: "main",
status: "completed",
per_page: 100,
},
);
for (const run of runs) {
if (run.id === currentRunId || run.conclusion !== "success") {
continue;
}
const artifacts = await github.paginate(
github.rest.actions.listWorkflowRunArtifacts,
{
owner,
repo,
run_id: run.id,
per_page: 100,
},
);
const artifact = artifacts.find(
(candidate) =>
!candidate.expired && candidate.name === "tileops_perf_history",
);
if (!artifact) {
continue;
}
core.info(
`Using perf history artifact from run ${run.id} (${run.html_url})`,
);
core.setOutput("run_id", String(run.id));
return;
}
core.info("No previous perf history artifact found; nightly report will start a new history baseline.");
- name: Download previous perf history
uses: actions/download-artifact@v4
if: ${{ always() && steps.perf_history_source.outputs.run_id != '' }}
with:
name: tileops_perf_history
path: .perf_history
github-token: ${{ github.token }}
repository: ${{ github.repository }}
run-id: ${{ steps.perf_history_source.outputs.run_id }}
- name: Generate nightly report
if: ${{ always() }}
run: |
set -euo pipefail
export PYTHONPATH="${GITHUB_WORKSPACE}${PYTHONPATH:+:$PYTHONPATH}"
HISTORY_ARGS=()
if [ -f .perf_history/perf_history.json ]; then
HISTORY_ARGS=(--history .perf_history/perf_history.json)
fi
python3 scripts/nightly_report.py \
--test-xml test_results.xml \
--bench-xml bench_results.xml \
"${HISTORY_ARGS[@]}" \
--output nightly_report.md \
--history-out perf_history.json \
|| echo "::warning::nightly_report.py failed"
shell: bash
- name: Post nightly report to step summary
if: ${{ always() && hashFiles('nightly_report.md') != '' }}
run: cat nightly_report.md >> "$GITHUB_STEP_SUMMARY"
shell: bash
- name: Upload op test artifacts
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: tileops_op_test_${{ github.run_id }}
path: |
test_results.xml
tileops_op_test.log
nightly_report.md
if-no-files-found: warn
retention-days: 14
- name: Upload perf history
uses: actions/upload-artifact@v4
if: ${{ always() && hashFiles('perf_history.json') != '' }}
with:
name: tileops_perf_history
path: perf_history.json
overwrite: true
retention-days: 14
- name: Fail op test job if pytest failed
if: ${{ always() && steps.op_tests.outcome == 'failure' }}
run: exit 1
shell: bash
# =========================================================================
# Phase 2b — Publish benchmark data to the nightly-bench orphan branch
# Consumed by tile-ai/TileOPs.github.io to render the docs Benchmarks page.
# Runs on a github-hosted runner (no GPU needed) and uses the default token
# to force-push the day's XML snapshot, mirroring the manifest-stats pattern.
# =========================================================================
publish-bench-data:
if: ${{ always() && github.repository == 'tile-ai/TileOPs' && (github.event_name == 'schedule' || github.ref == 'refs/heads/main') }}
needs: [op_test]
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Download benchmark artifact
uses: actions/download-artifact@v4
with:
name: tileops_benchmark_${{ github.run_id }}
path: _bench
continue-on-error: true
- name: Download op test artifact
uses: actions/download-artifact@v4
with:
name: tileops_op_test_${{ github.run_id }}
path: _optest
continue-on-error: true
- name: Assemble publish payload
id: assemble
run: |
set -euo pipefail
mkdir -p _publish
ok=1
if [ -f _bench/bench_results.xml ]; then
cp _bench/bench_results.xml _publish/bench_results.xml
else
echo "::warning::bench_results.xml missing; skipping nightly-bench publish"
ok=0
fi
if [ -f _optest/test_results.xml ]; then
cp _optest/test_results.xml _publish/test_results.xml
else
echo "::warning::test_results.xml missing; page will render without test status"
fi
if [ "$ok" -eq 1 ]; then
run_date="$(date -u +%Y-%m-%d)"
printf '{\n "commit": "%s",\n "date": "%s",\n "gpu": "NVIDIA H200",\n "run_id": "%s"\n}\n' \
"${GITHUB_SHA}" "${run_date}" "${GITHUB_RUN_ID}" > _publish/meta.json
fi
echo "ok=${ok}" >> "$GITHUB_OUTPUT"
shell: bash
- name: Publish to nightly-bench branch
if: ${{ steps.assemble.outputs.ok == '1' }}
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_branch: nightly-bench
publish_dir: _publish
force_orphan: true
commit_message: "chore(bench): nightly benchmark data from ${{ github.sha }}"
user_name: "github-actions[bot]"
user_email: "41898282+github-actions[bot]@users.noreply.github.com"
# =========================================================================
# Phase 3 — Packaging and smoke test
# =========================================================================
packaging:
if: ${{ always() && github.repository == 'tile-ai/TileOPs' && (github.event_name == 'schedule' || github.ref == 'refs/heads/main') }}
needs: [op_test]
permissions:
contents: write
runs-on: [self-hosted, tile-ops, nightly]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Verify nightly runner environment
run: scripts/ci/verify_nightly_runner.sh
shell: bash
- name: Install TileOPs (image-baked stack, --no-deps)
run: bash scripts/ci/install_tileops.sh
shell: bash
- name: Build and test wheel package
run: |
set -euo pipefail
exec > >(tee packaging.log) 2>&1
# Validate GPU frequency
TARGET_CLOCK_MHZ=1500
RETRIES=5
for i in $(seq 1 "${RETRIES}"); do
gpu_clock=$(nvidia-smi --query-gpu=clocks.current.graphics --format=csv,noheader,nounits | xargs)
echo "GPU clock: ${gpu_clock} MHz (attempt ${i}/${RETRIES})"
if [ "${gpu_clock}" = "${TARGET_CLOCK_MHZ}" ]; then break; fi
if [ "${i}" -eq "${RETRIES}" ]; then
echo "::error::GPU frequency validation failed. Expected ${TARGET_CLOCK_MHZ} MHz."
exit 1
fi
sleep 2
done
python3 -m pip install -c constraints.txt build pytest pytest-timeout
echo "=== Build wheel package ==="
python3 -m build --wheel
echo "=== Build artifacts ==="
ls -lh dist/*.whl
echo "=== Install built wheel ==="
python3 -m pip install dist/*.whl
TMP_TEST_DIR="$(mktemp -d "${RUNNER_TEMP}/packaging-test.XXXXXX")"
cp -r tests "${TMP_TEST_DIR}/tests"
cp pyproject.toml "${TMP_TEST_DIR}/"
cd "${TMP_TEST_DIR}"
echo "=== Runtime cache env ==="
echo "TILELANG_CACHE_DIR=${TILELANG_CACHE_DIR:-}"
echo "TILELANG_TMP_DIR=${TILELANG_TMP_DIR:-}"
echo "TRITON_CACHE_DIR=${TRITON_CACHE_DIR:-}"
echo "=== Run pytest packaging ==="
python3 -m pytest -q tests/ops -m "packaging" \
--timeout=900 --timeout-method=thread \
--junit-xml="${GITHUB_WORKSPACE}/packaging_smoke.xml"
shell: bash
- name: Upload wheel artifact
if: ${{ success() }}
uses: actions/upload-artifact@v4
with:
name: tileops-wheel-${{ github.sha }}
path: dist/*.whl
if-no-files-found: error
retention-days: 14
- name: Upload packaging logs
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: packaging-logs-${{ github.sha }}
path: |
packaging.log
packaging_smoke.xml
if-no-files-found: warn
retention-days: 14
- name: Publish wheel to GitHub Release (tag only)
if: ${{ startsWith(github.ref, 'refs/tags/v') }}
uses: softprops/action-gh-release@v2
with:
files: dist/*.whl
fail_on_unmatched_files: true

328
.github/workflows/preflight.yml vendored
View File

@ -1,328 +0,0 @@
name: Preflight Checks
permissions:
contents: read
checks: read
pull-requests: read
on:
push:
branches: [main, testbed]
tags: ["v*"]
pull_request:
branches: [main, testbed]
types: [opened, synchronize, reopened, ready_for_review]
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
validate-pr-title:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: .claude/conventions
sparse-checkout-cone-mode: false
- name: Validate PR title format
env:
EVENT_NAME: ${{ github.event_name }}
PR_TITLE: ${{ github.event.pull_request.title }}
run: |
set -euo pipefail
if [[ "$EVENT_NAME" != "pull_request" ]]; then
echo "Not a pull request event; skipping PR title validation."
exit 0
fi
source .claude/conventions/types.sh
if [[ ! "$PR_TITLE" =~ $COMMIT_MSG_PATTERN ]]; then
echo "::error::PR title does not follow TileOPs format."
echo "Expected: [Type] description or [Type][Scope] description"
echo "Types: $COMMIT_PR_TYPES"
echo "Got: $PR_TITLE"
exit 1
fi
echo "PR title is valid: $PR_TITLE"
detect-changes:
runs-on: ubuntu-latest
outputs:
manifest: ${{ steps.filter.outputs.manifest }}
benchmark: ${{ steps.filter.outputs.benchmark }}
steps:
- name: Detect relevant changed paths
id: filter
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
PR_NUMBER: ${{ github.event.pull_request.number || '' }}
REPO: ${{ github.repository }}
WORKFLOW_PATH: .github/workflows/preflight.yml
run: |
set -euo pipefail
# Non-PR events (push to main, tags, manual dispatch) always run
# both gated jobs — no base ref to diff against.
if [[ "$EVENT_NAME" != "pull_request" ]]; then
echo "manifest=true" >> "$GITHUB_OUTPUT"
echo "benchmark=true" >> "$GITHUB_OUTPUT"
exit 0
fi
# Fail-open: if the API call errors out or returns nothing, run
# everything. The optimization must never reduce CI reliability.
if ! FILES=$(gh api --paginate "repos/$REPO/pulls/$PR_NUMBER/files" --jq '.[].filename' 2>&1); then
echo "::warning::gh api failed; defaulting to run all gated jobs"
echo "$FILES"
echo "manifest=true" >> "$GITHUB_OUTPUT"
echo "benchmark=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if [[ -z "$FILES" ]]; then
echo "::warning::no files returned; defaulting to run all gated jobs"
echo "manifest=true" >> "$GITHUB_OUTPUT"
echo "benchmark=true" >> "$GITHUB_OUTPUT"
exit 0
fi
echo "Changed files in PR #$PR_NUMBER:"
echo "$FILES"
manifest=false
benchmark=false
while IFS= read -r f; do
[[ -z "$f" ]] && continue
case "$f" in
tileops/manifest/*.yaml|tileops/manifest/__init__.py|scripts/validate_manifest.py|"$WORKFLOW_PATH")
manifest=true
;;
esac
# `benchmark-contract-tests` runs `pytest benchmarks/tests`, which
# imports `workloads.workload_base`, `tileops.manifest`, and reads
# the per-family files under `tileops/manifest/*.yaml` via
# `load_workloads()` / `workloads_to_params()`. Any of those layers
# can break the contract tests, so include them all.
case "$f" in
benchmarks/*|workloads/*|tileops/manifest/*.yaml|tileops/manifest/__init__.py|"$WORKFLOW_PATH")
benchmark=true
;;
esac
done <<< "$FILES"
echo "manifest=$manifest"
echo "benchmark=$benchmark"
echo "manifest=$manifest" >> "$GITHUB_OUTPUT"
echo "benchmark=$benchmark" >> "$GITHUB_OUTPUT"
ci-gate:
runs-on: ubuntu-latest
outputs:
skip: ${{ steps.check.outputs.skip }}
steps:
- name: Check existing CI results for this SHA
id: check
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
EVENT_ACTION: ${{ github.event.action || '' }}
TARGET_BRANCH: ${{ github.base_ref || '' }}
IS_DRAFT: ${{ github.event.pull_request.draft || false }}
HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
run: |
set -euo pipefail
if [[ "$EVENT_NAME" != "pull_request" ]]; then
echo "skip=false" >> "$GITHUB_OUTPUT"
exit 0
fi
if [[ "$IS_DRAFT" == "true" ]]; then
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
case "$TARGET_BRANCH" in
main|testbed) ;;
*) echo "skip=true" >> "$GITHUB_OUTPUT"; exit 0 ;;
esac
if [[ "$EVENT_ACTION" != "ready_for_review" ]]; then
echo "skip=false" >> "$GITHUB_OUTPUT"
exit 0
fi
# `benchmark-contract-tests` is intentionally skipped on PRs that
# don't touch its inputs (see detect-changes). Keep that job
# skippable, but still require the core preflight checks to be
# successful so `ready_for_review` re-runs them and unblocks
# gpu-smoke.
CHECKS=$(gh api "repos/${{ github.repository }}/commits/$HEAD_SHA/check-runs" --jq '
[
.check_runs
| group_by(.name)
| map(max_by(.id))[]
| select(.name == "pre-commit" or .name == "gitleaks" or .name == "actionlint" or .name == "benchmark-contract-tests")
| {name: .name, conclusion: .conclusion}
]
')
ALL_OK=$(echo "$CHECKS" | jq '
all(
if .name == "benchmark-contract-tests"
then (.conclusion == "success" or .conclusion == "skipped")
else (.conclusion == "success")
end
)
')
COUNT=$(echo "$CHECKS" | jq 'length')
if [[ "$ALL_OK" == "true" && "$COUNT" -ge 4 ]]; then
echo "CI already passed for SHA $HEAD_SHA; skipping"
echo "skip=true" >> "$GITHUB_OUTPUT"
else
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
pre-commit:
needs: [validate-pr-title, ci-gate]
if: ${{ github.repository == 'tile-ai/TileOPs' && needs.ci-gate.outputs.skip != 'true' }}
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Run pre-commit
uses: pre-commit/action@v3.0.1
with:
extra_args: --all-files --show-diff-on-failure
gitleaks:
needs: [validate-pr-title, ci-gate]
if: ${{ github.repository == 'tile-ai/TileOPs' && needs.ci-gate.outputs.skip != 'true' }}
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Install gitleaks
env:
GITLEAKS_VERSION: "8.30.0"
run: |
set -euo pipefail
install_dir="${RUNNER_TEMP}/bin"
archive="gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz"
mkdir -p "$install_dir"
curl -sSfL "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/${archive}" \
-o "${RUNNER_TEMP}/${archive}"
tar -xzf "${RUNNER_TEMP}/${archive}" -C "$install_dir" gitleaks
echo "$install_dir" >> "$GITHUB_PATH"
- name: Run gitleaks
run: gitleaks dir . --redact --no-banner
validate-manifest:
needs: [validate-pr-title, ci-gate, detect-changes]
if: ${{ github.repository == 'tile-ai/TileOPs' && needs.ci-gate.outputs.skip != 'true' && needs.detect-changes.outputs.manifest == 'true' }}
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Install TileOPs and validator dependencies
run: pip install -e . -c constraints.txt
- name: Validate ops manifest
run: python scripts/validate_manifest.py --levels schema,signature,shape,dtype,bench --strict
actionlint:
needs: [validate-pr-title, ci-gate]
if: ${{ github.repository == 'tile-ai/TileOPs' && needs.ci-gate.outputs.skip != 'true' }}
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Install actionlint
env:
ACTIONLINT_VERSION: "1.7.11"
run: |
set -euo pipefail
install_dir="${RUNNER_TEMP}/bin"
mkdir -p "$install_dir"
tarball="actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz"
curl -sSfL \
"https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/${tarball}" \
-o "${RUNNER_TEMP}/${tarball}"
tar -xzf "${RUNNER_TEMP}/${tarball}" -C "$install_dir" actionlint
echo "$install_dir" >> "$GITHUB_PATH"
- name: Run actionlint
run: actionlint -color
compile-contract-gate:
# Always-on by design: no detect-changes path condition, so manifest-only
# and test-only PRs both run the declaration/evidence equality test.
needs: [validate-pr-title, ci-gate]
if: ${{ github.repository == 'tile-ai/TileOPs' && needs.ci-gate.outputs.skip != 'true' }}
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Install TileOPs (with dev extras)
run: pip install -e ".[dev]" -c constraints.txt
- name: Run manifest and compile-contract structural tests
run: python -m pytest -q tests/test_validate_manifest.py
benchmark-contract-tests:
# CPU-only contract tests for benchmark workload protocols.
# Scoped to `benchmarks/tests/` so the GPU-bound `benchmarks/ops/`
# suites (nightly-only) are NOT collected on PR CI.
needs: [validate-pr-title, ci-gate, detect-changes]
if: ${{ github.repository == 'tile-ai/TileOPs' && needs.ci-gate.outputs.skip != 'true' && needs.detect-changes.outputs.benchmark == 'true' }}
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Install TileOPs (with dev extras)
run: pip install -e ".[dev]" -c constraints.txt
- name: Run benchmark contract tests
run: python -m pytest -q benchmarks/tests

42
.github/workflows/runner-maintenance.yml vendored
View File

@ -1,42 +0,0 @@
name: Runner Maintenance
# Heavy disk reclamation for the self-hosted runner. Runs once per day at
# 18:00 UTC (02:00 Asia/Shanghai) so it lands in an off-peak window. Per-PR
# `gpu-smoke` jobs only attempt the heavier reclaim path when disk is under
# pressure, and still rate-limit the expensive cache-trim pass via a
# cooldown; this scheduled job forces the full cleanup pass on days with
# little PR traffic.
#
# GPU smoke and nightly share the persistent cache tree bind-mounted at
# `/ci-cache/*` inside the runner container.
on:
schedule:
- cron: "0 18 * * *"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: runner-maintenance
cancel-in-progress: false
jobs:
reclaim-disk:
if: ${{ github.repository == 'tile-ai/TileOPs' }}
# `nightly` (not `fork`) keeps maintenance on the resident shared-cache runners; the
# destructive autotuner age-trim must never run against a fork pool's overlay cache.
runs-on: [self-hosted, tile-ops, nightly]
timeout-minutes: 60
steps:
# Invoke the action directly via owner/repo/path@ref so this workflow
# does not depend on `actions/checkout` succeeding first — in the
# disk-full scenarios this job is meant to remediate, a prior checkout
# could itself fail for lack of space and prevent reclaim from ever
# running. `@main` is trusted because this workflow only triggers on
# schedule / workflow_dispatch; no fork-PR code path reaches it.
- name: Reclaim runner disk
uses: tile-ai/TileOPs/.github/actions/reclaim-runner-disk@main
with:
force-reclaim: "true"

97
.github/workflows/stale-issues.yml vendored
View File

@ -1,97 +0,0 @@
name: stale-issues
permissions:
issues: write
on:
schedule:
- cron: "0 9 * * 1" # Every Monday at 09:00 UTC
workflow_dispatch:
jobs:
close-stale-issues:
runs-on: ubuntu-latest
steps:
- name: Close issues with no comments for 4 weeks
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
set -euo pipefail
CUTOFF=$(date -u -d "28 days ago" +%Y-%m-%dT%H:%M:%SZ)
echo "Closing issues with no comment activity since: $CUTOFF"
OWNER="${GITHUB_REPOSITORY%/*}"
REPO="${GITHUB_REPOSITORY#*/}"
# Batch-fetch open issues with last comment time via GraphQL
ISSUES_TO_CLOSE=()
CURSOR=null
while :; do
if [ "$CURSOR" = "null" ]; then
# shellcheck disable=SC2016
PAGE=$(gh api graphql -f owner="$OWNER" -f name="$REPO" -f query='
query($owner: String!, $name: String!) {
repository(owner: $owner, name: $name) {
issues(first: 100, states: OPEN, orderBy: {field: UPDATED_AT, direction: ASC}) {
pageInfo { hasNextPage endCursor }
nodes {
number
createdAt
labels(first: 100) { nodes { name } }
comments(last: 1) { nodes { createdAt } }
}
}
}
}')
else
# shellcheck disable=SC2016
PAGE=$(gh api graphql -f owner="$OWNER" -f name="$REPO" -f cursor="$CURSOR" -f query='
query($owner: String!, $name: String!, $cursor: String!) {
repository(owner: $owner, name: $name) {
issues(first: 100, states: OPEN, orderBy: {field: UPDATED_AT, direction: ASC}, after: $cursor) {
pageInfo { hasNextPage endCursor }
nodes {
number
createdAt
labels(first: 100) { nodes { name } }
comments(last: 1) { nodes { createdAt } }
}
}
}
}')
fi
HAS_NEXT=$(echo "$PAGE" | jq -r '.data.repository.issues.pageInfo.hasNextPage')
CURSOR=$(echo "$PAGE" | jq -r '.data.repository.issues.pageInfo.endCursor // "null"')
while IFS=" " read -r NUMBER LAST_ACTIVITY; do
[ -z "$NUMBER" ] && continue
ISSUES_TO_CLOSE+=("$NUMBER:$LAST_ACTIVITY")
done < <(
echo "$PAGE" | jq -r --arg cutoff "$CUTOFF" '
.data.repository.issues.nodes[]
| select([.labels.nodes[].name] | index("never-stale") | not)
| .number as $n
| (.comments.nodes[0].createdAt // .createdAt) as $last
| select($last < $cutoff)
| "\($n) \($last)"
'
)
[ "$HAS_NEXT" = "true" ] || break
done
if [ "${#ISSUES_TO_CLOSE[@]}" -eq 0 ]; then
echo "No stale issues found."
exit 0
fi
for ENTRY in "${ISSUES_TO_CLOSE[@]}"; do
NUMBER="${ENTRY%%:*}"
LAST_ACTIVITY="${ENTRY#*:}"
echo "Closing #$NUMBER (last activity: $LAST_ACTIVITY)"
gh issue close "$NUMBER" \
--repo "$GITHUB_REPOSITORY" \
--comment "Closed automatically: no comment activity for 4 weeks. Reopen if still relevant."
done

79
.gitignore vendored
View File

@ -1,79 +0,0 @@
# Compiled Object files
*.o
*.obj
*.pyc
# Editor files
*~
*.swp
*.swo
.vscode/
.vs/
.idea/*
# Build artifacts
build/
dist/
wheelhouse/
build_sdist/
__pycache__
*.egg-info
# Virtual environments
venv/
.venv/
.tox/
# Git merge artifacts
*.orig
\#*
\.#*
# macOS
**/.DS_Store
# Logs and data
*.log
*.pkl_*
# Test / lint caches
.pytest_cache
.hypothesis
.ruff_cache
.cache/
# Output / profiling
output/
output*/
profile_out/
tmp/
.tmp/
debug/
# Local lock / scheduler state
/.lock
.claude/*.lock
# Exception: keep the debug folder for tests
!testing/python/debug
# Secrets and credentials
.env
.env.*
*.pem
*.key
credentials.json
secrets.yaml
secrets.yml
docs/plans/
.humanize*
.foundry/runs/
.foundry/migrations/
.foundry/plan/
.foundry/config.local.json
CLAUDE.local.md
.claude/worktrees/
docs/superpowers/
*.ncu-rep

0
benchmarks/kernels/__init__.py → .nojekyll
View File

64
.pre-commit-config.yaml
View File

@ -1,64 +0,0 @@
ci:
autofix_prs: false
autofix_commit_msg: "[Chore][Lint] pre-commit.ci auto fixes"
autoupdate_commit_msg: "[CI] [pre-commit.ci] autoupdate"
autoupdate_schedule: monthly
default_stages: [pre-commit, pre-push, manual]
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
- id: check-symlinks
- id: destroyed-symlinks
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-added-large-files
- id: check-merge-conflict
fail_fast: true
- id: check-executables-have-shebangs
- id: check-shebang-scripts-are-executable
- id: detect-private-key
- id: check-yaml
- id: check-toml
- id: check-ast
fail_fast: true
- id: debug-statements
- id: file-contents-sorter
args: [--ignore-case]
files: ^docs/spelling_wordlist\.txt$
- repo: https://github.com/gitleaks/gitleaks
rev: v8.30.0
hooks:
- id: gitleaks
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.14.13
hooks:
- id: ruff-check
args: [--fix, --exit-non-zero-on-fix, --config, pyproject.toml]
- repo: https://github.com/codespell-project/codespell
rev: v2.4.1
hooks:
- id: codespell
additional_dependencies: [".[toml]"]
exclude: |
(?x)(
^.+\.(cpp|hpp|cxx|cc|c|h|cu|cuh)$|
^.+\.svg$|
^.*\brequirements\b.*\.txt$
)
- repo: https://github.com/executablebooks/mdformat
rev: 0.7.22
hooks:
- id: mdformat
additional_dependencies:
- mdformat-gfm==0.4.1
- mdformat-black==0.1.1
- mdformat-myst==0.2.1
- mdformat-tables==1.0.0
- mdformat-toc==0.3.0

50
CLAUDE.md
View File

@ -1,50 +0,0 @@
# CLAUDE.md
## Project Overview
TileOPs is a high-performance LLM operator library built on TileLang. The goal is to provide efficient, modular, and maintainable AI workload implementations.
This project follows **design-first, spec-driven** development: design docs and `tileops/manifest/` are the authoritative spec; code conforms to the spec, not the other way around.
## Development Environment
Activate a virtual environment, then `make install` (deps + pre-commit hooks).
## Key References
### Design
- [architecture.md](docs/design/architecture.md) — system modules (M1-M8), data flow, agent production loop, directory structure
- [ops-design.md](docs/design/ops-design.md) — Op interface execution guide (how to add a new op)
- [ops-design-reference.md](docs/design/ops-design-reference.md) — Op interface detail reference (interface tables, codegen, naming, protocol)
- [manifest.md](docs/design/manifest.md) — `tileops/manifest/` spec format (signature, workloads, roofline, source)
- [roofline.md](docs/design/roofline.md) — `tileops/manifest/` `roofline` field spec: performance model, authoring, and per-consumer contracts (validator / benchmark / M5 / codegen)
### Process
- [trust-model.md](docs/design/trust-model.md) — trust boundaries (manifest → test → implementation → benchmark), workloads layer contract
- [testing.md](docs/design/testing.md) — test/benchmark framework, core abstractions, tolerances, reporting rules
- [tileops-skills.md](docs/tileops-skills.md) — developer decision guide: which repo-provided skill to use for which task
## Reading the ops manifest
The manifest lives at `tileops/manifest/`, one or more YAML files per op family — most families use a single file; large families may be sharded across multiple files. The `tileops.manifest` package merges them into a single `ops` dict at runtime.
- **Programmatic reads**: prefer `from tileops.manifest import load_manifest, load_workloads`. Never re-implement the merge.
- **Structural inspection**: parse the relevant family file with `yaml.safe_load` and index `ops` by op name. Pick the file from the op's family field rather than scanning all of them.
- **Edits**: edit the single family file that owns the op. Use a round-trip parser (`ruamel.yaml`) to preserve comments and key order. Op names must remain unique across files — duplicates raise at load time.
- Reserve `Read`/`grep` for targeted line lookups inside one family file, not structural reading.
## Domain Rules (load on demand)
Read the relevant context file **before** modifying files in that domain. Do not load them if your task does not touch that domain.
| When you modify | Read first |
| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `tests/` | [.claude/domain-rules/testing-budget.md](.claude/domain-rules/testing-budget.md) |
| `tileops/manifest/` | [.claude/domain-rules/manifest-spec.md](.claude/domain-rules/manifest-spec.md) |
| `scripts/validate_manifest.py`, `tests/test_validate_manifest.py` | [.claude/domain-rules/manifest-validator.md](.claude/domain-rules/manifest-validator.md) |
| `tileops/ops/`, `tileops/kernels/` | [.claude/domain-rules/ops-design.md](.claude/domain-rules/ops-design.md) |
| `benchmarks/` | [.claude/domain-rules/benchmark.md](.claude/domain-rules/benchmark.md) |
| `workloads/` | [docs/design/trust-model.md](docs/design/trust-model.md) |
| `docs/design/` | [.claude/domain-rules/design-docs.md](.claude/domain-rules/design-docs.md) |

21
LICENSE
View File

@ -1,21 +0,0 @@
MIT License
Copyright (c) Tile-AI.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE

43
Makefile
View File

@ -1,43 +0,0 @@
.PHONY: install install-bench lint test test-smoke test-full test-nightly bench clean help
install:
pip install -e '.[dev]' -v
pre-commit install
install-bench:
PIP_NO_BUILD_ISOLATION=1 pip install -e '.[dev,bench]' 'git+https://github.com/fla-org/native-sparse-attention.git@bd67af59b90afa34b25f61d2922e612d10dba3bd' -v
pre-commit install
lint:
pre-commit run --all-files
test:
python -m pytest -q tests
test-smoke:
python -m pytest -q tests -m smoke
test-full:
python -m pytest -q tests -m "smoke or full"
test-nightly:
python -m pytest -q tests -m "smoke or full or nightly"
bench:
python -m pytest benchmarks/
clean:
rm -rf build/ dist/ tileops.egg-info
help:
@echo "Available targets:"
@echo " install Install dependencies and pre-commit hooks"
@echo " install-bench Install with benchmark baseline libraries"
@echo " lint Run linters on all files"
@echo " test Run the test suite"
@echo " test-smoke Run smoke-tier tests"
@echo " test-full Run smoke + full-tier tests"
@echo " test-nightly Run smoke + full + nightly-tier tests"
@echo " bench Run benchmarks"
@echo " clean Remove build artifacts"
@echo " help Show this help message"

99
README.md
View File

@ -1,99 +0,0 @@
<div align="center">
<img src="https://raw.githubusercontent.com/tile-ai/TileOPs/main/assets/logo.png" width="350"/>
<h1>TileOPs</h1>
<p><strong>Spec-driven GPU operator library for LLMs — designed for AI agents to build, evaluate, and optimize</strong></p>
<p>Built on <a href="https://github.com/tile-ai/tilelang">TileLang</a></p>
<!-- <p>
<a href="https://pypi.org/project/tileops/"><img src="https://img.shields.io/badge/PyPI-tileops-1E90FF" alt="PyPI version" height="20"></a>
</p> -->
<p>
<a href="https://github.com/tile-ai/TileOPs/tree/main/tileops/manifest"><img src="https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftile-ai%2FTileOPs%2Fstats%2Fmanifest-implemented.json" alt="Spec coverage"></a>
<a href="https://github.com/tile-ai/TileOPs/tree/main/benchmarks"><img src="https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftile-ai%2FTileOPs%2Fstats%2Fmanifest-benchmark.json" alt="Bench coverage"></a>
</p>
<p>
<a href="#installation"><b>Installation</b></a> |
<a href="#quick-start"><b>Quick Start</b></a> |
<a href="#documentation"><b>Docs</b></a>
</p>
</div>
> **Status**: TileOPs is under active development. APIs may change.
## Overview
TileOPs is a GPU operator library for LLM training and inference, built on [TileLang](https://github.com/tile-ai/tilelang). Beyond providing a growing collection of production-quality operators, TileOPs explores a **spec-driven development model** where AI agents can read declarative operator specifications, generate kernel implementations, and evaluate them against hardware-theoretical performance bounds — with minimal human scaffolding.
### Architecture
Every operator is split into two layers with a strict boundary:
- **Op** (L2) — stateless Python entry point. Handles validation, dtype casting, and memory layout. CUDA-Graph compatible; `torch.compile(fullgraph=True)` support is declared per op in the manifest.
- **Kernel** (L1) — TileLang GPU implementation with hardware-specific optimizations (Hopper).
This separation keeps user-facing behavior independent of GPU strategy, allowing agents and developers to modify either layer without side effects on the other.
### Key Properties
- **Spec-driven** — each operator is declared in a machine-readable manifest (`tileops/manifest/`) that specifies signatures, workloads, and roofline formulas, serving as the entry point for both agent code generation and automated validation
- **Roofline-evaluated** — kernel performance is measured against Speed-of-Light hardware bounds, not relative baselines
- **Auto-tuning** — built-in search over tile sizes, pipelines, and scheduling parameters
- **Lightweight** — depends only on TileLang, PyTorch, and einops
## Installation
TileOPs is under active development and is installed from source; PyPI releases will begin with the first stable release. A CUDA-capable GPU is required.
### Prerequisites
- Python >= 3.10
- PyTorch >= 2.1, < 2.11 (CI validates 2.10)
- CUDA Toolkit 12.x
- NVIDIA GPU: **Hopper** (SM_90)
- [TileLang](https://github.com/tile-ai/tilelang) >= 0.1.9, < 0.2.0 (CI validates 0.1.11)
### From source
```bash
git clone https://github.com/tile-ai/TileOPs
cd TileOPs
make install # dev dependencies + pre-commit hooks
```
> [!NOTE]
> If CUDA and TileLang are already installed system-wide and you encounter build issues:
> `PIP_NO_BUILD_ISOLATION=1 pip install -e '.[dev]' -v && pre-commit install`
Verify:
```bash
python -m pytest tests/ -q # requires a CUDA GPU
```
## Quick Start
```python
import torch
from tileops.ops import GemmOp
M, N, K = 1024, 1024, 512
dtype = torch.float16
gemm = GemmOp() # shapes and dtype are inferred at call time
a = torch.randn(M, K, device="cuda", dtype=dtype)
b = torch.randn(N, K, device="cuda", dtype=dtype) # trans_b=True by default
d = gemm(a, b) # equals a @ b.T
```
## Documentation
Design docs and development guides are in [`docs/`](docs/). The full API reference and performance tables are published at [TileOPs.github.io](https://github.com/tile-ai/TileOPs.github.io).
## Contributing
See [docs/](docs/) for design docs. Branch and commit conventions are in [`.claude/conventions/types.sh`](.claude/conventions/types.sh).
## License
TileOPs is released under the [MIT License](LICENSE).

33
THIRD_PARTY_NOTICES.md
View File

@ -1,33 +0,0 @@
# Third-Party Notices
This repository includes or adapts code from the following MIT-licensed
projects. The top-level `LICENSE` contains the MIT license text used for this
repository distribution; the original copyright notices below are retained in
the relevant source files.
## Qwen FlashQLA
Files under `tileops/kernels/gated_deltanet/gdn_prefill/` implement a TileOps
version of the CP-split Gated DeltaNet prefill schedule. The schedule-level
reference for the h-state / corrected-segment-start part of this implementation
comes from Qwen FlashQLA:
- `__init__.py`
- `cp_fwd.py`
- `fused_fwd.py`
- `prepare_h.py`
- `tilelang_compat.py`
- `utils.py`
Original notice:
```text
Copyright (c) 2026 The Qwen team, Alibaba Group.
Licensed under the MIT License.
```
The TileOps versions are not direct wrappers around the FlashQLA kernels. They
adapt the CP-split scheduling idea into the TileOps operator API, BTHD dispatch,
TileLang compatibility layer, benchmarking, tests, and local replay/output
implementation. The utility file keeps the upstream copyright notice present in
the FlashQLA utility source.

BIN
assets/logo.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 169 KiB

265
bench_results.xml Normal file
View File

File diff suppressed because one or more lines are too long

6
benchmarks/__init__.py
View File

@ -1,6 +0,0 @@
from .benchmark_base import BenchmarkBase, BenchmarkReport
__all__ = [
"BenchmarkBase",
"BenchmarkReport",
]

738
benchmarks/benchmark_base.py
View File

@ -1,738 +0,0 @@
import contextlib
import logging
import subprocess
import sys
import threading
from abc import ABC, abstractmethod
from datetime import datetime
from typing import (
Any,
Callable,
Generic,
Optional,
Protocol,
TypeVar,
runtime_checkable,
)
import pytest
import torch
from torch.autograd.profiler import DeviceType
from tileops.manifest import (
WORKLOAD_RESERVED_KEYS,
load_manifest,
load_workloads,
single_input_workload_contract,
)
def _workload_contract(op_name: str) -> tuple[str, frozenset[str]]:
"""Resolve the shared workload contract for an op known to exist."""
sig = load_manifest()[op_name].get("signature") or {}
contract = single_input_workload_contract(sig)
if contract is None:
raise KeyError(
f"workloads_to_params({op_name!r}) needs exactly one manifest "
"tensor input; multi-input ops use their own bench files."
)
return contract
# Benchmark capability protocols
@runtime_checkable
class ShapeDtypeWorkload(Protocol):
"""Structural type for workloads that carry shape and dtype metadata.
Any object with ``shape`` and ``dtype`` satisfies this protocol.
Used by helpers that only need tensor metadata, not input generation
capability.
"""
shape: tuple[int, ...]
dtype: torch.dtype
@runtime_checkable
class InputGeneratingWorkload(Protocol):
"""Structural type for workloads that can generate benchmark inputs."""
def gen_inputs(self) -> tuple[Any, ...]: ...
@runtime_checkable
class BenchmarkWorkload(ShapeDtypeWorkload, InputGeneratingWorkload, Protocol):
"""Full benchmark workload: shape/dtype metadata + input generation.
This is the standard contract for benchmark workloads that need both
roofline metadata extraction and input tensor generation.
Workloads satisfy this protocol when they define ``shape`` and ``dtype``
metadata in addition to implementing ``gen_inputs()``.
"""
...
W = TypeVar("W")
_logger = logging.getLogger("tileops.bench")
# Thread-local storage for conftest hook to pick up per-test bench results.
# A single test function may call record() multiple times (tileops + baseline).
_bench_results = threading.local()
# Latest bench_kernel measurement metadata; deviations from the default
# protocol are surfaced in results by BenchmarkBase._build_result.
_bench_meta = threading.local()
class _CuptiProjectionError(Exception):
"""CUPTI trace lacked a projected annotation window for every repeat."""
# Name of the ``record_function`` annotation wrapping the timed call. Kineto
# projects this scope onto the device timeline. The L2-flush ``cache.zero_()``
# is synchronized to completion before the window opens (see ``bench_kernel``),
# so its device event cannot fall inside a window regardless of how the
# projection behaves; kernels the timed call launches do.
_KERNEL_REGION = "tileops_bench_kernel"
def _sum_kernel_time_us(kineto_results):
"""Sum device time of the kernels the timed call launched.
Sums only kernels inside a :data:`_KERNEL_REGION` annotation window, so the
L2-flush fill is excluded and the kernel under test is counted regardless of
its name. A call launching several kernels contributes all of them.
Iterates the C++ Kineto events directly to bypass ``key_averages()``, which
is ~16x slower (~130ms of Python parsing/tree-building) for large traces.
Returns:
``(total_us, n_regions)``: summed kernel time in microseconds and the
number of annotation windows. The caller checks ``n_regions ==
n_repeat`` to confirm the scope projected on every iteration.
"""
import bisect
windows: list[tuple[int, int]] = []
kernels: list[tuple[int, int]] = [] # (start_ns, duration_ns)
for evt in kineto_results.events():
if evt.device_type() != DeviceType.CUDA:
continue
if evt.is_user_annotation():
if evt.name() == _KERNEL_REGION:
windows.append((evt.start_ns(), evt.end_ns()))
continue
kernels.append((evt.start_ns(), evt.duration_ns()))
windows.sort()
starts = [w[0] for w in windows]
ends = [w[1] for w in windows]
total_us = 0.0
for start_ns, dur_ns in kernels:
# Count only kernels that fall inside a timed-call window; everything
# outside (notably the L2-flush fill) is excluded.
idx = bisect.bisect_right(starts, start_ns) - 1
if idx >= 0 and start_ns < ends[idx]:
total_us += dur_ns / 1000.0
return total_us, len(windows)
# L2 cache flush buffer (sized to actual L2, allocated lazily)
_l2_flush_cache: Optional[torch.Tensor] = None
def _get_l2_flush_cache() -> torch.Tensor:
global _l2_flush_cache
if _l2_flush_cache is None:
l2_bytes = torch.cuda.get_device_properties(0).L2_cache_size
if l2_bytes <= 0:
_logger.warning(
"L2 cache size query returned %d; flushing a 256 MB buffer "
"instead", l2_bytes,
)
l2_bytes = int(256e6)
_l2_flush_cache = torch.empty(l2_bytes // 4, dtype=torch.int, device="cuda")
return _l2_flush_cache
def _native_output_suppressor():
"""Return an fd-level output suppressor that is safe under pytest capture.
tilelang's ``suppress_stdout_stderr`` dup2's ``/dev/null`` over
``sys.stdout.fileno()``; under pytest fd capture that fileno is the
capture tmpfile and the redirect corrupts it (``EBADF`` on later reads).
Suppress only when stdout/stderr are the process fds 1/2.
"""
try:
native = sys.stdout.fileno() == 1 and sys.stderr.fileno() == 2
except (AttributeError, OSError, ValueError):
# Streams without a real descriptor (io.StringIO, capsys) or with
# fileno() unsupported: fd-level suppression is impossible.
native = False
if not native:
return contextlib.nullcontext()
from tilelang.profiler.bench import suppress_stdout_stderr
return suppress_stdout_stderr()
# NVIDIA SOL-ExecBench–style benchmark
def bench_kernel(
fn: Callable,
args: tuple[Any, ...] = (),
n_warmup: int = 10,
n_repeat: int = 50,
n_trials: int = 3,
) -> float:
"""Benchmark a GPU kernel with pure kernel timing via CUPTI.
Protocol (adapted from NVIDIA SOL-ExecBench, arxiv.org/abs/2603.19173):
1. Lock GPU clocks externally (nvidia-smi).
2. Run *n_warmup* un-timed iterations with L2 flush.
3. For each of *n_trials* trials, profile *n_repeat* iterations
under CUPTI to get pure kernel execution time (no launch overhead).
L2 is flushed before every iteration. Input tensors are cloned
each iteration so the kernel always sees fresh addresses.
4. Report the median trial mean (robust to outlier trials).
Uses CUPTI via torch.profiler for accurate kernel-only timing, with
direct Kineto C++ event iteration to avoid Python parsing overhead.
Falls back to CUDA events if CUPTI is unavailable.
Args:
fn: Callable to benchmark. If *args* is provided, called as
``fn(*cloned_args)``; otherwise called as ``fn()``.
args: Tensor arguments to clone each iteration. Non-tensor
values are passed through unchanged.
n_warmup: Warmup iterations (default 10).
n_repeat: Timed iterations per trial (default 50).
n_trials: Independent trials (default 3).
Returns:
Kernel latency in **milliseconds**.
"""
if not isinstance(args, tuple):
raise TypeError(
f"bench_kernel expects a tuple of args, got {type(args).__name__}. "
"Check that gen_inputs() returns a tuple."
)
cache = _get_l2_flush_cache()
has_args = len(args) > 0
# Pre-clone a small pool of input tensors so the kernel sees different
# addresses across iterations. Skip cloning if total tensor memory
# exceeds 1 GB to avoid OOM on large workloads.
_N_CLONES = 3
_MAX_CLONE_BYTES = 1 << 30 # 1 GB
if has_args:
tensor_mask = tuple(isinstance(a, torch.Tensor) for a in args)
total_bytes = sum(a.nelement() * a.element_size()
for a, m in zip(args, tensor_mask, strict=True) if m)
if total_bytes * _N_CLONES <= _MAX_CLONE_BYTES:
arg_pool = [
tuple(a.clone() if m else a for a, m in zip(args, tensor_mask, strict=True))
for _ in range(_N_CLONES)
]
def _run(i):
return fn(*arg_pool[i % _N_CLONES])
else:
_logger.warning(
"bench_kernel: inputs total %.2f GiB; skipping per-iteration "
"cloning (kernel sees identical addresses)",
total_bytes / (1 << 30),
)
arg_pool = None
def _run(i):
return fn(*args)
else:
arg_pool = None
def _run(i):
return fn()
_bench_meta.inputs_cloned = arg_pool is not None or not has_args
# Warmup (no profiling)
for i in range(n_warmup):
cache.zero_()
_run(i)
torch.cuda.synchronize()
# One plain profiler context per trial; torch.profiler.schedule is avoided
# because queued launches leak across its warmup/active boundary.
# Kineto's window projection may include a flush merely enqueued before
# the window, so the flush is drained (sync) before the timed call and
# the call is drained before the next flush; the syncs add host-side
# latency only.
trial_means: list[float] = []
try:
with _native_output_suppressor():
for _ in range(n_trials):
with torch.profiler.profile(
# CPU activity is required for Kineto to project the
# annotation window; it never adds device time.
activities=[
torch.profiler.ProfilerActivity.CPU,
torch.profiler.ProfilerActivity.CUDA,
],
) as profiler:
for i in range(n_repeat):
cache.zero_()
torch.cuda.synchronize()
with torch.profiler.record_function(_KERNEL_REGION):
_run(i)
torch.cuda.synchronize()
total_us, n_regions = _sum_kernel_time_us(profiler.profiler.kineto_results)
# Untrustworthy trace → CUDA-events fallback; genuine CUDA
# errors and OOM propagate.
if n_regions != n_repeat:
raise _CuptiProjectionError(
f"{n_regions}/{n_repeat} annotation windows projected"
)
trial_means.append((total_us / n_repeat) * 1e-3)
_bench_meta.timing = "cupti"
except _CuptiProjectionError as exc:
_logger.warning(
"CUPTI projection failed (%s); falling back to CUDA-events "
"timing, which includes launch overhead", exc,
)
trial_means = []
# Fallback to CUDA events if CUPTI failed
if not trial_means:
_bench_meta.timing = "cuda-events"
for _ in range(n_trials):
start_events = [torch.cuda.Event(enable_timing=True) for _ in range(n_repeat)]
end_events = [torch.cuda.Event(enable_timing=True) for _ in range(n_repeat)]
for i in range(n_repeat):
cache.zero_()
start_events[i].record()
_run(i)
end_events[i].record()
torch.cuda.synchronize()
times = [s.elapsed_time(e) for s, e in zip(start_events, end_events, strict=True)]
trial_means.append(sum(times) / len(times))
# Free the arg pool and release cached GPU memory to prevent
# accumulation across hundreds of benchmark calls.
if arg_pool is not None:
del arg_pool
torch.cuda.empty_cache()
trial_means.sort()
return trial_means[len(trial_means) // 2]
def _get_env_metadata() -> list[str]:
"""Collect GPU model, driver version, CUDA version, and torch version."""
lines = []
lines.append(f"- **Torch version**: {torch.__version__}")
lines.append(f"- **CUDA version (torch)**: {torch.version.cuda or 'N/A'}")
if torch.cuda.is_available():
gpu_name = torch.cuda.get_device_name(0)
lines.append(f"- **GPU model**: {gpu_name}")
else:
lines.append("- **GPU model**: N/A (no CUDA device)")
# Try to get NVIDIA driver version and clocks from nvidia-smi.
gpu_query_fields = [
"driver_version",
"clocks.current.sm",
"clocks.current.memory",
"clocks.applications.graphics",
"clocks.applications.memory",
]
gpu_query_values = []
try:
result = subprocess.run(
[
"nvidia-smi",
f"--query-gpu={','.join(gpu_query_fields)}",
"--format=csv,noheader,nounits",
],
capture_output=True, text=True, timeout=5,
)
if result.returncode == 0:
gpu_query_values = [
part.strip() for part in result.stdout.splitlines()[0].split(",")
]
except (FileNotFoundError, subprocess.TimeoutExpired):
pass
driver = gpu_query_values[0] if len(gpu_query_values) == len(gpu_query_fields) else "N/A"
lines.append(f"- **Driver version**: {driver}")
if len(gpu_query_values) == len(gpu_query_fields):
sm_clock, mem_clock, app_sm_clock, app_mem_clock = gpu_query_values[1:]
lines.append(
"- **GPU clocks**: "
f"SM current {sm_clock} MHz, memory current {mem_clock} MHz, "
f"application SM {app_sm_clock} MHz, "
f"application memory {app_mem_clock} MHz"
)
return lines
class BenchmarkBase(Generic[W], ABC):
"""Abstract base class for op benchmarking.
Generic over workload type so subclasses can declare the exact
capability they need. ``WorkloadBase`` remains the typical in-repo
implementation, but the public contract is the type parameter.
Subclass must implement calculate_flops() and calculate_memory().
"""
def __init__(self, workload: W):
self.workload = workload
@abstractmethod
def calculate_flops(self) -> Optional[float]:
raise NotImplementedError
@abstractmethod
def calculate_memory(self) -> Optional[float]:
raise NotImplementedError
def profile(self,
functor: Any,
*inputs: Any) -> dict:
"""Profile a callable and return structured results.
Uses the NVIDIA SOL-ExecBench protocol: CUPTI kernel timing,
10 warmup, 50 repeats × 3 trials, L2 flush sized to actual
cache, input tensors cloned each iteration.
"""
with torch.no_grad():
latency = bench_kernel(functor, args=inputs)
return self._build_result(latency)
def profile_autograd(self, functor: Any) -> dict:
"""Profile a callable that requires autograd (e.g. fwd+bwd).
Same as profile() but without torch.no_grad(), so the callable
can build autograd graphs and call .backward() internally.
The functor must be a zero-arg closure that captures its inputs.
"""
latency = bench_kernel(functor)
return self._build_result(latency)
def _build_result(self, latency: float) -> dict:
result = {"latency_ms": latency}
# Deviations from the default protocol must be visible in reports.
timing = getattr(_bench_meta, "timing", None)
if timing is not None and timing != "cupti":
result["timing"] = timing
if getattr(_bench_meta, "inputs_cloned", True) is False:
result["inputs_cloned"] = False
flops = self.calculate_flops()
if flops is not None:
result["tflops"] = flops / latency * 1e-9
memory = self.calculate_memory()
if memory is not None:
result["bandwidth_tbs"] = memory / latency * 1e-9
return result
# Manifest-driven benchmark helpers
def _workload_extra_params(w: dict, shape_key: str) -> dict[str, Any]:
"""Return op-call params on a workload entry, stripping reserved keys."""
reserved = WORKLOAD_RESERVED_KEYS | {shape_key}
return {
k: v
for k, v in w.items()
if isinstance(k, str) and k not in reserved and not k.startswith("__")
}
def workloads_to_params(op_name: str, include_extra: bool = False) -> list:
"""Convert manifest workload dicts for *op_name* to pytest params.
Each entry becomes ``pytest.param(shape, dtype, id=...)``; with
``include_extra=True`` a third element carries the op-call params
declared on the workload entry (e.g. ``{"dim": 0}``).
"""
workloads = load_workloads(op_name) # canonical not-found error
shape_key, allowed = _workload_contract(op_name)
params = []
for w in workloads:
if shape_key not in w:
raise KeyError(
f"workload {w.get('label', w)!r} of {op_name!r} is missing "
f"{shape_key!r} (derived from the signature's input name)."
)
unknown = sorted(
repr(k) for k in w
if not isinstance(k, str) or (k not in allowed and not k.startswith("__"))
)
if unknown:
raise KeyError(
f"workload {w.get('label', w)!r} of {op_name!r} has unknown "
f"keys {unknown}; allowed: {sorted(allowed)}."
)
shape = tuple(w[shape_key])
label = w.get("label", "x".join(str(s) for s in shape))
extra = _workload_extra_params(w, shape_key) if include_extra else {}
for dtype_str in w["dtypes"]:
dtype = getattr(torch, dtype_str)
# Copy ``extra`` per parametrization so mutation in one test case
# cannot leak into later cases sharing the workload entry.
param_args = (
(shape, dtype, dict(extra))
if include_extra
else (shape, dtype)
)
params.append(pytest.param(*param_args, id=f"{label}-{dtype_str}"))
return params
def workload_field_params(workloads: list, keys: tuple) -> list:
"""Turn manifest workload dicts into pytest params.
First workload is marked ``smoke``, the rest ``full``. Keys ending in
``dtype`` are resolved to ``torch.dtype`` values.
"""
params = []
for i, w in enumerate(workloads):
args = [getattr(torch, w[k]) if k.endswith("dtype") else w[k] for k in keys]
params.append(
pytest.param(
*args,
marks=pytest.mark.smoke if i == 0 else pytest.mark.full,
id=w["label"],
)
)
return params
class ManifestBenchmark(BenchmarkBase[ShapeDtypeWorkload]):
"""Generic benchmark that reads FLOP/memory counts from an Op instance.
Accepts an op name, an instantiated Op, and any workload satisfying
:class:`ShapeDtypeWorkload`. The op must implement ``eval_roofline()``.
Dynamic-shape ops may bind roofline variables during ``forward()``, so
this helper calls ``op.eval_roofline()`` only while building a result
after profiling has executed the op.
Usage::
op = SumFwdOp(dtype=dtype, dim=0)
bm = ManifestBenchmark("SumFwdOp", op, workload)
result = bm.profile(op, *inputs)
"""
def __init__(
self,
op_name: str,
op: Any,
workload: ShapeDtypeWorkload,
):
super().__init__(workload)
self._op_name = op_name
self._op = op
self._roofline_cache: Optional[tuple[float, float]] = None
def _get_roofline(self) -> tuple[float, float]:
if self._roofline_cache is None:
flops, mem_bytes = self._op.eval_roofline()
self._roofline_cache = (float(flops), float(mem_bytes))
return self._roofline_cache
def calculate_flops(self) -> Optional[float]:
return self._get_roofline()[0]
def calculate_memory(self) -> Optional[float]:
return self._get_roofline()[1]
def _extract_op_config(op: object) -> Optional[dict]:
"""Return the kernel config for an Op instance, or None if unavailable.
Handles the three Op patterns currently used in tileops:
1. **Eager-init** (e.g. ``GemmOp``): ``op.kernel`` is a Kernel
instance set in ``__init__``.
2. **Lazy with dummy kernel** (e.g. ``FFTC2COp``): ``op.kernel`` is a
default Kernel and ``op._kernel_cache`` may hold others.
3. **Pure lazy cache** (e.g. ``_SoftmaxBaseOp`` and the spec-conformant
reduction ops): ``op._kernel_cache`` is the only source; ``op.kernel``
is unset.
A direct ``op.config`` attribute (legacy / explicit override) takes
precedence over kernel introspection.
"""
op_config = getattr(op, "config", None)
if op_config:
return op_config
kernel = getattr(op, "kernel", None)
op_config = getattr(kernel, "config", None) if kernel is not None else None
if op_config:
return op_config
# Pure lazy-cache pattern: pick any cached kernel's config. All cached
# kernels for a given op share dtype/op_kind, so taking the first is
# sufficient for the benchmark report (which records one entry per call).
cache = getattr(op, "_kernel_cache", None)
if cache:
try:
first_kernel = next(iter(cache.values()))
except StopIteration:
first_kernel = None
if first_kernel is not None:
op_config = getattr(first_kernel, "config", None)
if op_config:
return op_config
return None
class BenchmarkReport:
"""Collects benchmark results and dumps a markdown report.
All methods are static — use as BenchmarkReport.record(...).
Call clear() at session start, dump() at session end.
"""
_records: dict = {}
@staticmethod
def record(op_or_name, params: dict, result: dict, tag: str = "tileops") -> None:
"""Record a benchmark result.
Args:
op_or_name: Op instance or benchmark group name string.
If an Op instance, class name and module are extracted automatically.
params: Parameter dict (typically from locals())
result: Dict with latency_ms, tflops, bandwidth_tbs
tag: Label to distinguish implementations (e.g. "tileops", "FA3", "fla")
"""
if isinstance(op_or_name, str):
name = op_or_name
op_module = None
op_config = None
else:
name = op_or_name.__class__.__name__
op_module = op_or_name.__class__.__module__
op_config = _extract_op_config(op_or_name)
# Filter params to only include serializable benchmark parameters.
# Tuples of primitives (e.g. ``shape=(4096, 4096)``) are preserved
# verbatim so the profile log carries the original input geometry
# rather than a flattened element count.
def _is_serializable(v: Any) -> bool:
if isinstance(v, (int, float, bool, str, torch.dtype)):
return True
if isinstance(v, tuple):
return all(_is_serializable(x) for x in v)
return False
filtered_params = {
k: v for k, v in params.items()
if k not in ("test", "bm", "op", "inputs", "result", "result_bl",
"baseline_fn", "tune")
and not k.startswith("_")
and _is_serializable(v)
}
record_entry = {
"params": filtered_params,
"result": result,
"tag": tag,
}
if op_config:
record_entry["config"] = op_config
BenchmarkReport._records.setdefault(name, []).append(record_entry)
# Accumulate in thread-local for conftest hook.
if not hasattr(_bench_results, "entries"):
_bench_results.entries = []
entry = {"tag": tag, "op": name, **result}
if op_module:
entry["op_module"] = op_module
_bench_results.entries.append(entry)
_logger.info("op=%s module=%s tag=%s latency_ms=%.4f tflops=%.2f",
name, op_module or "N/A", tag,
result.get("latency_ms", 0),
result.get("tflops", 0))
@staticmethod
def dump(path: str) -> None:
"""Write all collected results to a markdown-formatted log file."""
if not BenchmarkReport._records:
return
lines = [
"# TileOPs Benchmark Report",
f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
"",
"## Environment",
"",
]
lines.extend(_get_env_metadata())
lines.append("")
default_result_keys = ["latency_ms", "tflops", "bandwidth_tbs"]
for name, entries in BenchmarkReport._records.items():
if not entries:
continue
lines.append(f"## {name}")
lines.append("")
# Group by tag
tag_entries = {}
for entry in entries:
tag_entries.setdefault(entry["tag"], []).append(entry)
result_keys = list(default_result_keys)
for entry in entries:
for key in entry["result"]:
if key not in result_keys:
result_keys.append(key)
for tag, tag_group in tag_entries.items():
lines.append(f"### {tag}")
lines.append("")
param_keys = list(tag_group[0]["params"].keys())
has_config = any("config" in e for e in tag_group)
header_parts = param_keys + result_keys
if has_config:
header_parts.append("config")
lines.append("| " + " | ".join(header_parts) + " |")
lines.append("| " + " | ".join(["---"] * len(header_parts)) + " |")
for entry in tag_group:
row = [str(entry["params"].get(k, "")) for k in param_keys]
for rk in result_keys:
val = entry["result"].get(rk)
if val is None:
row.append("N/A")
elif isinstance(val, (int, float)) and not isinstance(val, bool):
row.append(f"{val:.4f}")
else:
row.append(str(val))
if has_config:
cfg = entry.get("config")
row.append(str(cfg) if cfg else "")
lines.append("| " + " | ".join(row) + " |")
lines.append("")
with open(path, "w") as f:
f.write("\n".join(lines))
print(f"Benchmark report saved to {path}")
@staticmethod
def clear() -> None:
"""Clear all collected records."""
BenchmarkReport._records.clear()

98
benchmarks/conftest.py
View File

@ -1,98 +0,0 @@
import gc
import pytest
import torch
from benchmarks.benchmark_base import BenchmarkReport, _bench_results
def _release_cuda_cache_after_case() -> None:
"""Drop per-case Python references and cached CUDA blocks between benchmarks."""
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
@pytest.fixture(autouse=True)
def setup() -> None:
torch.manual_seed(1235)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(1235)
def pytest_sessionstart(session):
BenchmarkReport.clear()
def pytest_sessionfinish(session, exitstatus):
BenchmarkReport.dump("profile_run.log")
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
"""After bench test execution, attach perf data to the item as properties."""
_bench_results.entries = []
try:
yield
entries = getattr(_bench_results, "entries", [])
if not entries:
return
# Separate tileops entry (tag starts with "tileops") from baselines.
tileops_entry = None
baseline_entries = []
for e in entries:
if e["tag"].startswith("tileops"):
if tileops_entry is None:
tileops_entry = e
else:
baseline_entries.append(e)
if tileops_entry:
item.user_properties.append(("op", tileops_entry["op"]))
if "op_module" in tileops_entry:
item.user_properties.append(("op_module", tileops_entry["op_module"]))
tag = tileops_entry["tag"]
if tag != "tileops" and tag.startswith("tileops_"):
item.user_properties.append(("tileops_variant", tag[len("tileops_"):]))
item.user_properties.append(("tileops_latency_ms",
f"{tileops_entry.get('latency_ms', 0):.4f}"))
tflops = tileops_entry.get("tflops")
if tflops is not None:
item.user_properties.append(("tileops_tflops", f"{tflops:.2f}"))
bw = tileops_entry.get("bandwidth_tbs")
if bw is not None:
item.user_properties.append(("tileops_bandwidth_tbs", f"{bw:.2f}"))
# Write all baselines into JUnit XML properties.
# The first baseline uses the legacy unprefixed names (baseline_tag, etc.)
# for backward compatibility. Additional baselines use "{tag}_latency_ms",
# "{tag}_tflops", "{tag}_ratio" so the report can display multiple columns.
for idx, be in enumerate(baseline_entries):
tag = be["tag"]
bl_latency = be.get("latency_ms", 0)
bl_tflops = be.get("tflops")
if idx == 0:
# Legacy unprefixed keys — consumed by existing nightly_report.py
item.user_properties.append(("baseline_tag", tag))
item.user_properties.append(("baseline_latency_ms", f"{bl_latency:.4f}"))
if bl_tflops is not None:
item.user_properties.append(("baseline_tflops", f"{bl_tflops:.2f}"))
if tileops_entry:
tl = tileops_entry.get("latency_ms", 0)
if tl > 0 and bl_latency > 0:
item.user_properties.append(("baseline_ratio",
f"{bl_latency / tl:.4f}"))
# Tag-prefixed keys — always written for every baseline
item.user_properties.append((f"{tag}_latency_ms", f"{bl_latency:.4f}"))
if bl_tflops is not None:
item.user_properties.append((f"{tag}_tflops", f"{bl_tflops:.2f}"))
if tileops_entry:
tl = tileops_entry.get("latency_ms", 0)
if tl > 0 and bl_latency > 0:
item.user_properties.append((f"{tag}_ratio", f"{bl_latency / tl:.4f}"))
finally:
_bench_results.entries = []
_release_cuda_cache_after_case()

77
benchmarks/hardware/README.md
View File

@ -1,77 +0,0 @@
# Hardware Microbenchmarks
GPU hardware characterization benchmarks that produce calibration factors for `tileops/perf/profiles/`.
## Prerequisites
- NVIDIA GPU with CUDA toolkit (`nvcc` in PATH)
- TileOPs installed (`pip install -e .` from project root)
- Root/sudo access for clock locking (recommended)
## HBM Bandwidth
Measures peak HBM bandwidth using vectorized CUDA kernels (float4 load/store) with cudaEvent timing. The calibration factor is derived from the **STREAM Triad** kernel (`a[i] = b[i] + s*c[i]`, 2 reads + 1 write), the industry-standard pattern for roofline bandwidth calibration.
Triad's 2:1 read:write ratio is closer to real compute kernels than pure copy (1:1), which suffers worst-case HBM bus turnaround overhead. Copy, read-only, and write-only results are included as reference measurements.
### Lock GPU clocks (recommended)
GPU boost clocks fluctuate during benchmarks. Lock memory and SM clocks to their maximum for stable, reproducible results:
```bash
# Lock clocks (requires root/sudo)
sudo nvidia-smi -lgc $(nvidia-smi --query-gpu=clocks.max.sm --format=csv,noheader,nounits)
sudo nvidia-smi -lmc $(nvidia-smi --query-gpu=clocks.max.mem --format=csv,noheader,nounits)
```
After benchmarking, reset to default:
```bash
sudo nvidia-smi -rgc
sudo nvidia-smi -rmc
```
### Run
Run from project root:
```bash
python benchmarks/hardware/memory/hbm_bandwidth.py --profile h200 --arch sm_90
```
Options:
| Flag | Default | Description |
| ----------- | ------- | ----------------------------------------------------------------------- |
| `--profile` | `h200` | GPU profile name (reads theoretical peak from `tileops/perf/profiles/`) |
| `--arch` | `sm_90` | CUDA compute capability for nvcc |
| `--size-mb` | `2048` | Working set size in MB |
### Output
```
Measured peak (triad vec4): 4070.44 GB/s
Theoretical: 4800.0 GB/s
Calibration: 0.8480
Update tileops/perf/profiles/h200.yaml:
hbm.calibration: 0.8480
```
### Methodology
- **Calibration kernel:** STREAM Triad `a[i] = b[i] + s*c[i]` (2 reads + 1 write, `float4` vectorized)
- **Reference kernels:** Copy (1:1 read:write), Read-only, Write-only
- **Timing:** `cudaEvent` (GPU-side, no host overhead)
- **Warmup:** 100 iterations per config (ensures boost clocks stabilize)
- **Measurement:** 200 iterations × 5 runs, report best and median
- **Working set:** 2 GB default (>> L2 cache, ensures HBM is measured)
- **Calibration source:** best Triad bandwidth across block size sweep (128/256/512)
## Adding a new GPU profile
1. Create `tileops/perf/profiles/<gpu>.yaml` with theoretical specs from the datasheet
1. Lock GPU clocks (see above)
1. Run `python benchmarks/hardware/memory/hbm_bandwidth.py --profile <gpu> --arch <sm_XX>`
1. Update `<gpu>.yaml` with the measured calibration factor
1. Reset GPU clocks

5
benchmarks/hardware/__init__.py
View File

@ -1,5 +0,0 @@
"""Hardware characterization microbenchmark suite.
Measures GPU memory bandwidth, compute throughput, and system overhead
to produce calibration factors for GPU profiles (tileops/perf/profiles/).
"""

1
benchmarks/hardware/memory/__init__.py
View File

@ -1 +0,0 @@
"""Memory subsystem microbenchmarks."""

118
benchmarks/hardware/memory/hbm_bandwidth.py
View File

@ -1,118 +0,0 @@
"""HBM Bandwidth Benchmark — Python wrapper for hbm_saturation.cu.
Compiles and runs the CUDA microbenchmark, parses output, and prints
the calibration factor for tileops/perf/profiles/.
Calibration is derived from the STREAM Triad kernel (a = b + s*c, 2 reads +
1 write). Triad is the industry standard for roofline bandwidth calibration:
McCalpin, J.D., 1995. "Memory Bandwidth and Machine Balance in Current
High Performance Computers." IEEE TCCA Newsletter.
https://www.cs.virginia.edu/stream/
Williams, S., Waterman, A. & Patterson, D., 2009. "Roofline: An Insightful
Visual Performance Model for Multicore Architectures." CACM 52(4).
Usage:
python benchmarks/hardware/memory/hbm_bandwidth.py [--profile h200] [--size-mb 2048]
"""
import argparse
import subprocess
import sys
import tempfile
from pathlib import Path
from tileops.perf import load_profile
_CU_SRC = Path(__file__).parent / "hbm_saturation.cu"
def _compile(cu_path, binary_path, arch="sm_90"):
"""Compile the CUDA source. Raises on failure."""
cmd = [
"nvcc", "-O3", f"-arch={arch}",
"-Wno-deprecated-gpu-targets",
"-o", str(binary_path), str(cu_path),
]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
print(f"nvcc compilation failed:\n{result.stderr}", file=sys.stderr)
sys.exit(1)
def _run(binary_path, size_mb, theo_peak_gbs):
"""Run the benchmark binary and return stdout lines."""
cmd = [str(binary_path), str(size_mb), str(theo_peak_gbs)]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
if result.returncode != 0:
print(f"Benchmark failed:\n{result.stderr}", file=sys.stderr)
sys.exit(1)
return result.stdout.strip().splitlines()
def _parse_triad_peak(lines):
"""Extract the best Triad bandwidth (GB/s) from CSV output.
Only considers lines starting with 'triad,' — STREAM Triad (2 reads +
1 write) is the standard calibration kernel for roofline analysis.
Its 2:1 read:write ratio is closer to real compute kernels than pure
copy (1:1), which suffers worst-case HBM bus turnaround overhead.
"""
best_gbs = 0.0
for line in lines:
if not line.startswith("triad,"):
continue
parts = line.split(",")
if len(parts) >= 6:
try:
gbs = float(parts[5]) # best_gbs column
best_gbs = max(best_gbs, gbs)
except ValueError:
continue
return best_gbs
def main():
parser = argparse.ArgumentParser(description="HBM bandwidth microbenchmark")
parser.add_argument("--profile", default="h200", help="GPU profile name")
parser.add_argument("--size-mb", type=int, default=2048, help="Working set size in MB")
parser.add_argument("--arch", default="sm_90", help="CUDA architecture")
args = parser.parse_args()
profile = load_profile(args.profile)
theo_peak_gbs = profile["hbm"]["theoretical"] / 1e9
print(f"Profile: {args.profile}")
print(f"Theoretical HBM BW: {theo_peak_gbs:.1f} GB/s")
print(f"Working set: {args.size_mb} MB")
print()
with tempfile.TemporaryDirectory() as tmpdir:
binary = Path(tmpdir) / "hbm_saturation"
print("Compiling hbm_saturation.cu ...")
_compile(_CU_SRC, binary, arch=args.arch)
print("Running benchmark (5 runs x 200 reps, this may take a few minutes) ...\n")
lines = _run(binary, args.size_mb, theo_peak_gbs)
# Print raw output
for line in lines:
print(line)
# Extract calibration from STREAM Triad results
measured_peak = _parse_triad_peak(lines)
if measured_peak > 0 and theo_peak_gbs > 0:
calibration = measured_peak / theo_peak_gbs
print(f"\n{'='*60}")
print(f"Measured peak (triad vec4): {measured_peak:.2f} GB/s")
print(f"Theoretical: {theo_peak_gbs:.1f} GB/s")
print(f"Calibration: {calibration:.4f}")
print(f"\nUpdate tileops/perf/profiles/{args.profile}.yaml:")
print(f" hbm.calibration: {calibration:.4f}")
print(f"{'='*60}")
if __name__ == "__main__":
main()

263
benchmarks/hardware/memory/hbm_saturation.cu
View File

@ -1,263 +0,0 @@
// HBM Saturation Benchmark
// Measures peak HBM bandwidth with vectorized CUDA kernels.
// Uses STREAM Triad [1] as the primary calibration pattern for roofline
// analysis [2], with Copy/Read/Write as reference measurements.
//
// Triad (a[i] = b[i] + s*c[i]) has a 2:1 read:write ratio — closer to real
// compute kernels than pure Copy (1:1), which suffers worst-case HBM bus
// turnaround overhead. BabelStream [3] establishes this methodology for GPUs.
//
// References:
// [1] McCalpin, J.D., 1995. "Memory Bandwidth and Machine Balance in Current
// High Performance Computers." IEEE TCCA Newsletter, pp.19-25.
// https://www.cs.virginia.edu/stream/
// [2] Williams, S., Waterman, A. & Patterson, D., 2009. "Roofline: An
// Insightful Visual Performance Model for Multicore Architectures."
// Communications of the ACM, 52(4), pp.65-76.
// [3] Deakin, T. et al., 2018. "Evaluating attainable memory bandwidth of
// parallel programming models via BabelStream." International Journal of
// Computational Science and Engineering, 17(3), pp.247-262.
// https://github.com/UoB-HPC/BabelStream
//
// Compile: nvcc -O3 -arch=sm_90 -Wno-deprecated-gpu-targets -o hbm_saturation hbm_saturation.cu
// Usage: ./hbm_saturation [size_mb] [theo_peak_gbs] (defaults: 2048, 4800)
#include <stdio.h>
#include <stdlib.h>
#include <cuda_runtime.h>
#include <algorithm>
#include <vector>
#include <functional>
#define CHECK_CUDA(call) do { \
cudaError_t err = (call); \
if (err != cudaSuccess) { \
fprintf(stderr, "CUDA error at %s:%d: %s\n", __FILE__, __LINE__, cudaGetErrorString(err)); \
exit(1); \
} \
} while(0)
#define CHECK_LAST() CHECK_CUDA(cudaGetLastError())
// ============================================================
// Kernels
// ============================================================
// Read kernels: use volatile to prevent compiler from optimizing away loads.
// Accumulate into a register and write once to prevent the reduction overhead
// from dominating measurement at small sizes.
__global__ void k_read_vec4(const float4* __restrict__ data, volatile float* __restrict__ out, long long n) {
float sum = 0.0f;
long long idx = (long long)blockIdx.x * blockDim.x + threadIdx.x;
long long stride = (long long)gridDim.x * blockDim.x;
for (long long i = idx; i < n; i += stride) {
float4 v = data[i];
sum += v.x + v.y + v.z + v.w;
}
// Single atomic per warp — negligible overhead vs memory-bound loop above
for (int o = 16; o > 0; o >>= 1) sum += __shfl_down_sync(0xffffffff, sum, o);
if (threadIdx.x % 32 == 0) atomicAdd((float*)out, sum);
}
// Write kernels
__global__ void k_write_vec4(float4* __restrict__ data, long long n, float val) {
long long idx = (long long)blockIdx.x * blockDim.x + threadIdx.x;
long long stride = (long long)gridDim.x * blockDim.x;
float4 v = make_float4(val, val, val, val);
for (long long i = idx; i < n; i += stride) data[i] = v;
}
// Copy kernels (read + write, 1:1 ratio)
__global__ void k_copy_vec4(const float4* __restrict__ src, float4* __restrict__ dst, long long n) {
long long idx = (long long)blockIdx.x * blockDim.x + threadIdx.x;
long long stride = (long long)gridDim.x * blockDim.x;
for (long long i = idx; i < n; i += stride) dst[i] = src[i];
}
// STREAM Triad: a[i] = b[i] + scalar * c[i] (2 reads + 1 write, 2:1 ratio)
// Industry-standard pattern for roofline calibration (closer to real kernel
// memory access patterns than pure copy).
__global__ void k_triad_vec4(const float4* __restrict__ b, const float4* __restrict__ c,
float4* __restrict__ a, long long n, float scalar) {
long long idx = (long long)blockIdx.x * blockDim.x + threadIdx.x;
long long stride = (long long)gridDim.x * blockDim.x;
for (long long i = idx; i < n; i += stride) {
float4 vb = b[i];
float4 vc = c[i];
a[i] = make_float4(vb.x + scalar * vc.x, vb.y + scalar * vc.y,
vb.z + scalar * vc.z, vb.w + scalar * vc.w);
}
}
// ============================================================
// Benchmark helper
// ============================================================
struct BenchResult {
float best_ms;
float median_ms;
float best_gbs;
float median_gbs;
};
BenchResult run_bench(std::function<void()> launch, long long total_bytes,
int warmup = 100, int reps = 200) {
std::vector<float> latencies;
for (int run = 0; run < 5; run++) {
// Warmup: enough iterations for GPU clocks to stabilize
for (int i = 0; i < warmup; i++) launch();
CHECK_CUDA(cudaDeviceSynchronize());
cudaEvent_t t0, t1;
CHECK_CUDA(cudaEventCreate(&t0));
CHECK_CUDA(cudaEventCreate(&t1));
CHECK_CUDA(cudaEventRecord(t0));
for (int i = 0; i < reps; i++) launch();
CHECK_CUDA(cudaEventRecord(t1));
CHECK_CUDA(cudaEventSynchronize(t1));
float ms;
CHECK_CUDA(cudaEventElapsedTime(&ms, t0, t1));
latencies.push_back(ms / reps);
CHECK_CUDA(cudaEventDestroy(t0));
CHECK_CUDA(cudaEventDestroy(t1));
}
std::sort(latencies.begin(), latencies.end());
float best = latencies[0];
float median = latencies[2]; // middle of 5
auto to_gbs = [&](float ms) -> float {
return (ms > 0) ? (float)total_bytes / (ms * 1e6f) : 0.0f;
};
return {best, median, to_gbs(best), to_gbs(median)};
}
// ============================================================
// Main
// ============================================================
int main(int argc, char* argv[]) {
long long size_mb = 2048;
float theo_peak_gbs = 4800.0f;
if (argc >= 2) size_mb = atoll(argv[1]);
if (argc >= 3) theo_peak_gbs = atof(argv[2]);
long long n_floats = size_mb * 1024LL * 1024LL / sizeof(float);
long long nbytes = n_floats * sizeof(float);
cudaDeviceProp prop;
cudaGetDeviceProperties(&prop, 0);
int sm_count = prop.multiProcessorCount;
printf("GPU: %s | SMs: %d | L2: %d MB\n", prop.name, sm_count,
prop.l2CacheSize / (1024*1024));
printf("Working set: %lld MB (%lld float32 elements)\n", size_mb, n_floats);
printf("Each config: 5 runs x 200 reps, warmup 100, reporting best and median\n");
printf("theo_peak_gbs=%.1f\n\n", theo_peak_gbs);
// Allocate with cudaMalloc (guaranteed 256-byte aligned)
// d_src, d_dst used by copy/read/write; d_src2 is the second input for triad
float *d_src, *d_src2, *d_dst;
volatile float *d_out;
CHECK_CUDA(cudaMalloc(&d_src, nbytes));
CHECK_CUDA(cudaMalloc(&d_src2, nbytes));
CHECK_CUDA(cudaMalloc(&d_dst, nbytes));
CHECK_CUDA(cudaMalloc((void**)&d_out, sizeof(float)));
// Initialize with a valid float pattern (not byte-fill).
// Non-zero values prevent hardware memory compression from inflating results.
{
std::vector<float> host_data(n_floats, 1.0f);
CHECK_CUDA(cudaMemcpy(d_src, host_data.data(), nbytes, cudaMemcpyHostToDevice));
std::fill(host_data.begin(), host_data.end(), 2.0f);
CHECK_CUDA(cudaMemcpy(d_src2, host_data.data(), nbytes, cudaMemcpyHostToDevice));
CHECK_CUDA(cudaMemset(d_dst, 0, nbytes));
CHECK_CUDA(cudaMemset((void*)d_out, 0, sizeof(float)));
}
// CSV header
printf("op,vec_width,block_size,best_ms,median_ms,best_gbs,median_gbs,pct_of_theo\n");
auto print_row = [theo_peak_gbs](const char* op, const char* vec, int bs, BenchResult r) {
printf("%s,%s,%d,%.4f,%.4f,%.2f,%.2f,%.1f%%\n",
op, vec, bs,
r.best_ms, r.median_ms, r.best_gbs, r.median_gbs,
r.best_gbs / theo_peak_gbs * 100.0f);
};
// Block sizes to sweep
int bss[] = {128, 256, 512};
// ============================================================
// 1. Copy (read + write, 1:1 ratio) — reference
// Bytes reported = 2 * nbytes (read src + write dst)
// Note: 1:1 read:write creates worst-case bus turnaround pressure.
// Real kernels are typically read-heavy and achieve higher bandwidth.
// ============================================================
printf("# Copy (read+write, 1:1 ratio) — reference\n");
for (int bi = 0; bi < 3; bi++) {
int bs = bss[bi];
int nblocks = sm_count * (2048 / bs);
long long n = n_floats / 4;
auto launch = [&]() { k_copy_vec4<<<nblocks, bs>>>((float4*)d_src, (float4*)d_dst, n); CHECK_LAST(); };
auto r = run_bench(launch, 2 * nbytes);
print_row("copy", "vec4", bs, r);
}
// ============================================================
// 2. STREAM Triad: a[i] = b[i] + s*c[i] (2 reads + 1 write)
// Bytes reported = 3 * nbytes (read b + read c + write a)
// 2:1 read:write ratio — closer to real compute kernels than copy.
// Used as the primary calibration measurement.
// ============================================================
printf("# Triad (2 reads + 1 write) — primary calibration\n");
for (int bi = 0; bi < 3; bi++) {
int bs = bss[bi];
int nblocks = sm_count * (2048 / bs);
long long n = n_floats / 4;
auto launch = [&]() {
k_triad_vec4<<<nblocks, bs>>>((float4*)d_src, (float4*)d_src2,
(float4*)d_dst, n, 3.14f);
CHECK_LAST();
};
auto r = run_bench(launch, 3 * nbytes);
print_row("triad", "vec4", bs, r);
}
// ============================================================
// 3. Read-only (for reference, not used for calibration)
// ============================================================
printf("# Read-only — reference\n");
for (int bi = 0; bi < 3; bi++) {
int bs = bss[bi];
int nblocks = sm_count * (2048 / bs);
long long n = n_floats / 4;
auto launch = [&]() { k_read_vec4<<<nblocks, bs>>>((float4*)d_src, d_out, n); CHECK_LAST(); };
auto r = run_bench(launch, nbytes);
print_row("read", "vec4", bs, r);
}
// ============================================================
// 4. Write-only (for reference, not used for calibration)
// ============================================================
printf("# Write-only — reference\n");
for (int bi = 0; bi < 3; bi++) {
int bs = bss[bi];
int nblocks = sm_count * (2048 / bs);
long long n = n_floats / 4;
auto launch = [&]() { k_write_vec4<<<nblocks, bs>>>((float4*)d_dst, n, 1.0f); CHECK_LAST(); };
auto r = run_bench(launch, nbytes);
print_row("write", "vec4", bs, r);
}
CHECK_CUDA(cudaFree(d_src));
CHECK_CUDA(cudaFree(d_src2));
CHECK_CUDA(cudaFree(d_dst));
CHECK_CUDA(cudaFree((void*)d_out));
return 0;
}

80
benchmarks/kernels/bench_moe_shared_expert_mlp.py
View File

@ -1,80 +0,0 @@
"""Benchmark for SharedExpertMLPKernel vs PyTorch MLP."""
import pytest
import torch
from benchmarks.benchmark_base import BenchmarkBase, BenchmarkReport
from tileops.kernels.moe import SharedExpertMLPKernel
from workloads.workload_base import FixtureBase, WorkloadBase
class SharedMLPBenchTest(WorkloadBase):
def __init__(self, num_tokens, hidden_size, ffn_size, dtype):
self.num_tokens = num_tokens
self.hidden_size = hidden_size
self.ffn_size = ffn_size
self.dtype = dtype
def gen_inputs(self):
device = torch.device("cuda")
hidden = torch.randn(self.num_tokens, self.hidden_size, dtype=self.dtype, device=device)
w_gate_up = torch.randn(self.ffn_size * 2, self.hidden_size, dtype=self.dtype, device=device)
w_down = torch.randn(self.hidden_size, self.ffn_size, dtype=self.dtype, device=device)
return hidden, w_gate_up, w_down
class SharedMLPBenchFixture(FixtureBase):
PARAMS = [
(
"num_tokens, hidden_size, ffn_size, dtype",
[
pytest.param(512, 2048, 8192, torch.bfloat16, marks=pytest.mark.smoke),
pytest.param(2048, 2048, 8192, torch.bfloat16, marks=pytest.mark.full),
pytest.param(4096, 2048, 8192, torch.bfloat16, marks=pytest.mark.full),
],
)
]
class SharedMLPBenchmark(BenchmarkBase):
def calculate_flops(self):
t = self.workload
return 2 * t.num_tokens * t.hidden_size * t.ffn_size * 3 # gate + up + down
def calculate_memory(self):
t = self.workload
elem = 2 # bf16
return elem * (t.num_tokens * t.hidden_size + 3 * t.ffn_size * t.hidden_size + 3 * t.num_tokens * t.ffn_size)
@SharedMLPBenchFixture
def test_shared_mlp_bench(num_tokens, hidden_size, ffn_size, dtype):
test = SharedMLPBenchTest(num_tokens, hidden_size, ffn_size, dtype)
bm = SharedMLPBenchmark(test)
hidden, w_gate_up, w_down = test.gen_inputs()
# TileLang kernel
kernel = SharedExpertMLPKernel(num_tokens=num_tokens, hidden_size=hidden_size,
ffn_size=ffn_size, dtype=dtype)
kernel(hidden, w_gate_up, w_down) # warmup
torch.cuda.synchronize()
result = bm.profile(kernel, hidden, w_gate_up, w_down)
BenchmarkReport.record(kernel, locals(), result, tag="tileops")
# PyTorch baseline
def pytorch_fn(hidden, w_gate_up, w_down):
gate = torch.nn.functional.linear(hidden, w_gate_up[:ffn_size])
up = torch.nn.functional.linear(hidden, w_gate_up[ffn_size:])
gate_up = torch.nn.functional.silu(gate) * up
return torch.nn.functional.linear(gate_up, w_down)
pytorch_fn(hidden, w_gate_up, w_down) # warmup
torch.cuda.synchronize()
result_torch = bm.profile(pytorch_fn, hidden, w_gate_up, w_down)
BenchmarkReport.record(kernel, locals(), result_torch, tag="torch")
if __name__ == "__main__":
pytest.main([__file__, "-v", "-m", "smoke"])

0
benchmarks/ops/__init__.py
View File

0
benchmarks/ops/attention/__init__.py
View File

94
benchmarks/ops/attention/bench_deepseek_dsa_decode.py
View File

@ -1,94 +0,0 @@
import pytest
import torch
from benchmarks.benchmark_base import BenchmarkReport, ManifestBenchmark
from benchmarks.ops.attention.manifest_params import dsa_decode_args, manifest_params
from tileops.manifest import load_workloads
from tileops.ops import DeepSeekSparseAttentionDecodeWithKVCacheFwdOp
from workloads.attention.deepseek import DsaDecodeTest
_OP_NAME = "DeepSeekSparseAttentionDecodeWithKVCacheFwdOp"
class _DsaDecodeTestBaseline(DsaDecodeTest):
"""Adds baseline ref_program for benchmark profiling."""
def ref_program(self, q: torch.Tensor, kv: torch.Tensor,
indices: torch.Tensor) -> torch.Tensor:
q = q.float()
kv = kv.float()
indices = indices.transpose(1, 2)
b, sq, h, dim_q = q.shape
b, sk, g, _ = kv.shape
q_start_index_s = self.q_start_index_s
if self.q_start_index_s is None:
q_start_index_s = sk * self.stride_kv - sq
assert kv.shape[-1] == self.dim + self.dim_tail, 'you should assign dim otherwise'
dim = self.dim
k = kv
v = kv[..., :dim]
b, _, _, dim_v = v.shape
g_index = g
h_index = h // g
compressed_causal_mask = torch.arange(
q_start_index_s, sq + q_start_index_s, dtype=torch.int32,
device="cuda").view(-1, 1) >= torch.arange(
self.stride_kv - 1,
sk * self.stride_kv,
self.stride_kv,
dtype=torch.int32,
device="cuda").view(1, -1)
mask = q.new_zeros(b, g_index, sq, sk + 1, dtype=torch.bool).scatter(3, indices.long(), 1)
mask = mask[..., :-1]
mask = mask & compressed_causal_mask.view(1, 1, sq, sk)
mask[:, :, :self.stride_kv - 1, 0] = True
mask = mask.view(b, g_index, 1, sq, sk)
q = q.view(b, sq, g, -1, dim_q)
score = torch.einsum("bmghd,bngd->bghmn", q, k)
sm_scale = dim_q**-0.5 if self.sm_scale is None else self.sm_scale
score = score.masked_fill(~mask, float("-inf")).mul(sm_scale)
p = score.softmax(dim=-1)
p = p.view(b, g_index, h_index, -1, sq, sk)
p = p.view(b, g, -1, sq, sk)
o = torch.einsum("bghmn,bngd->bmghd", p.type(v.dtype), v)
o = o.reshape(b, sq, h, dim_v)
return o.to(torch.float16)
_DSA_DECODE_BENCH_PARAMS = manifest_params(
load_workloads(_OP_NAME),
dsa_decode_args,
tune=False,
)
@pytest.mark.parametrize(
"batch, heads, seq_len_q, seq_len_kv, dim, dim_tail, topk, stride_kv, heads_kv, q_start_index_s, sm_scale, dtype, tune",
_DSA_DECODE_BENCH_PARAMS,
)
def test_dsa_decode_bench(batch: int, heads: int, seq_len_q: int, seq_len_kv: int, dim: int,
dim_tail: int, topk: int, stride_kv: int, heads_kv: int,
q_start_index_s: int, sm_scale: float, dtype: torch.dtype,
tune: bool) -> None:
test = _DsaDecodeTestBaseline(
batch, heads, seq_len_q, seq_len_kv, dim, dim_tail, topk, stride_kv, heads_kv,
q_start_index_s, sm_scale=sm_scale, dtype=dtype)
inputs = test.gen_inputs()
op = DeepSeekSparseAttentionDecodeWithKVCacheFwdOp(
batch, heads, seq_len_q, seq_len_kv, dim, dim_tail, topk, stride_kv, heads_kv,
q_start_index_s, sm_scale=sm_scale, dtype=dtype, tune=tune)
bm = ManifestBenchmark(_OP_NAME, op, test)
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag="tileops")
result_bl = bm.profile(test.ref_program, *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="torch-ref")
if __name__ == "__main__":
pytest.main([__file__, "-vvs"])

85
benchmarks/ops/attention/bench_deepseek_mla_decode.py
View File

@ -1,85 +0,0 @@
import pytest
import torch
import torch.nn.functional as F
from einops import einsum, rearrange
from benchmarks.benchmark_base import BenchmarkReport, ManifestBenchmark
from benchmarks.ops.attention.manifest_params import manifest_params, mla_decode_args
from tileops.manifest import load_workloads
from tileops.ops import MultiHeadLatentAttentionDecodeWithKVCacheFwdOp
from workloads.attention.deepseek import MlaDecodeTest
_OP_NAME = "MultiHeadLatentAttentionDecodeWithKVCacheFwdOp"
class _MlaDecodeTestBaseline(MlaDecodeTest):
"""Adds baseline ref_program for benchmark profiling."""
def ref_program(self, q: torch.Tensor, q_pe: torch.Tensor, kv: torch.Tensor,
k_pe: torch.Tensor) -> torch.Tensor:
"""
Inputs:
- q (Tensor): [batch, heads, dim]
- q_pe (Tensor): [batch, heads, dim_pe]
- kv (Tensor): [batch, seqlen_kv, heads_kv, dim]
- k_pe (Tensor): [batch, seqlen_kv, heads_kv, dim_pe]
Outputs:
- output (Tensor): [batch, heads, dim]
"""
dim = q.shape[-1]
dim_pe = q_pe.shape[-1]
num_head_groups = q.shape[1] // kv.shape[2]
scale = (dim + dim_pe)**0.5
Q = rearrange(
q, 'b (h g) d -> b g h d',
g=num_head_groups) # [batch_size, num_head_groups, groups, dim]
Q_pe = rearrange(
q_pe, 'b (h g) d -> b g h d',
g=num_head_groups) # [batch_size, num_head_groups, groups, dim_pe]
KV = rearrange(kv, 'b n h d -> b h n d') # [batch_size, groups, seqlen_kv, dim]
K_pe = rearrange(k_pe,
'b n h d -> b h n d') # [batch_size, num_head_groups, groups, dim_pe]
query = torch.concat([Q, Q_pe], dim=-1)
key = torch.concat([KV, K_pe], dim=-1)
scores = einsum(
query, key,
'b g h d, b h s d -> b g h s') # [batch_size, num_head_groups, groups, seqlen_kv]
attention = F.softmax(
scores / scale, dim=-1) # [batch_size, num_head_groups, groups, seqlen_kv]
out = einsum(attention, KV,
'b g h s, b h s d -> b g h d') # [batch_size, num_head_groups, groups, dim]
out = rearrange(out, 'b g h d -> b (h g) d') # [batch_size, heads, dim]
return out
_MLA_DECODE_BENCH_PARAMS = manifest_params(load_workloads(_OP_NAME), mla_decode_args)
@pytest.mark.parametrize(
"batch, heads, heads_kv, seq_len_kv, dim, dim_pe, dtype, tune",
_MLA_DECODE_BENCH_PARAMS,
)
def test_mla_decode_bench(batch: int, heads: int, heads_kv: int, seq_len_kv: int, dim: int,
dim_pe: int, dtype: torch.dtype, tune: bool) -> None:
test = _MlaDecodeTestBaseline(batch, heads, heads_kv, seq_len_kv, dim, dim_pe, dtype)
inputs = test.gen_inputs()
op = MultiHeadLatentAttentionDecodeWithKVCacheFwdOp(
batch, heads, heads_kv, seq_len_kv, dim, dim_pe, dtype, tune=tune)
bm = ManifestBenchmark(_OP_NAME, op, test)
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag="tileops")
result_bl = bm.profile(test.ref_program, *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="torch-ref")
if __name__ == "__main__":
pytest.main([__file__, "-vvs"])

617
benchmarks/ops/attention/bench_gqa.py
View File

@ -1,617 +0,0 @@
from typing import Optional
import pytest
import torch
from torch.nn import functional as F
from benchmarks.benchmark_base import BenchmarkBase, BenchmarkReport, ManifestBenchmark
from benchmarks.ops.attention.manifest_params import (
gqa_prefill_args,
gqa_prefill_paged_args,
gqa_qkv_args,
manifest_params,
)
from tileops.kernels.attention import (
GQAFwdKernel,
GQAFwdWgmmaPipelinedKernel,
GQAFwdWsPersistentCausalKernel,
GQAFwdWsPersistentKernel,
GQAPrefillFwdKernel,
GQAPrefillFwdWsPersistentCausalKernel,
)
from tileops.manifest import load_workloads
from tileops.ops import (
GroupedQueryAttentionBwdOp,
GroupedQueryAttentionFwdOp,
GroupedQueryAttentionPrefillFwdOp,
GroupedQueryAttentionPrefillPagedWithKVCacheFwdOp,
GroupedQueryAttentionPrefillVarlenFwdOp,
)
from workloads.attention.gqa import (
GQAPrefillFwdTest,
GQAPrefillPagedWithKVCacheFwdTest,
GQAPrefillVarlenFwdTest,
GroupedQueryAttentionBwdTest,
GroupedQueryAttentionFwdTest,
)
_GQA_FWD_OP = "GroupedQueryAttentionFwdOp"
_GQA_BWD_OP = "GroupedQueryAttentionBwdOp"
_GQA_PREFILL_FWD_OP = "GroupedQueryAttentionPrefillFwdOp"
_GQA_PREFILL_PAGED_WITH_KV_CACHE_FWD_OP = "GroupedQueryAttentionPrefillPagedWithKVCacheFwdOp"
class GQAPrefillVarlenFwdBenchmark(BenchmarkBase[GQAPrefillVarlenFwdTest]):
def calculate_flops(self) -> Optional[float]:
t = self.workload
visible = 0
for q_len, kv_len in zip(t.q_lens, t.kv_lens, strict=True):
visible += q_len * kv_len - q_len * (q_len - 1) / 2 if t.is_causal else q_len * kv_len
return 4.0 * t.heads * visible * t.dim
def calculate_memory(self) -> Optional[float]:
t = self.workload
query_size = sum(t.q_lens) * t.heads * t.dim
kv_size = sum(t.kv_lens) * t.heads_kv * t.dim
output_size = query_size
cu_seqlens_size = 2 * (t.batch + 1)
return (
query_size + 2 * kv_size + output_size
) * t.dtype.itemsize + cu_seqlens_size * torch.int32.itemsize
def _fa3_gqa_fwd(test: GroupedQueryAttentionFwdTest):
"""Return FA3 forward baseline callable, or None if not installed."""
try:
from flash_attn_interface import flash_attn_func
except ImportError:
return None
def baseline_fn(q, k, v):
out = flash_attn_func(q, k, v, causal=test.is_causal)
return out[0] if isinstance(out, tuple) else out
return baseline_fn
def _fa3_gqa_bwd(test: GroupedQueryAttentionBwdTest):
"""Return FA3 backward baseline callable, or None if not installed."""
try:
from flash_attn_interface import flash_attn_func
except ImportError:
return None
@torch.enable_grad()
def baseline_fn(q, k, v, o, grad_output, lse):
q = q.detach().requires_grad_(True)
k = k.detach().requires_grad_(True)
v = v.detach().requires_grad_(True)
out = flash_attn_func(q, k, v, causal=test.is_causal)
out = out[0] if isinstance(out, tuple) else out
out.backward(grad_output)
return q.grad, k.grad, v.grad
return baseline_fn
def _uniform_packed_prefill_inputs(
q: torch.Tensor,
k: torch.Tensor,
v: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor,
torch.Tensor, torch.Tensor]:
batch, seq_len_q, _, _ = q.shape
_, seq_len_kv, heads_kv, _ = k.shape
cu_q = torch.arange(batch + 1, device=q.device, dtype=torch.int32) * seq_len_q
cu_kv = torch.arange(batch + 1, device=q.device, dtype=torch.int32) * seq_len_kv
q_scale = torch.ones((batch, heads_kv), device=q.device, dtype=torch.float32)
return (
q.reshape(batch * seq_len_q, q.shape[2], q.shape[3]).contiguous(),
k.reshape(batch * seq_len_kv, heads_kv, k.shape[3]).contiguous(),
v.reshape(batch * seq_len_kv, heads_kv, v.shape[3]).contiguous(),
cu_q,
cu_kv,
q_scale,
torch.ones_like(q_scale),
torch.ones_like(q_scale),
)
def _flashinfer_gqa_fwd(test, q, k, v):
"""FlashInfer ragged-prefill baseline. Handles seq_len_q != seq_len_kv (square is
the seq_len_q == seq_len_kv case). Returns callable or None."""
try:
from flashinfer.prefill import BatchPrefillWithRaggedKVCacheWrapper
except ImportError:
return None
B, Sq, H, D = q.shape
Skv = k.shape[1]
Hkv = k.shape[2]
qo_indptr = torch.arange(0, B + 1, dtype=torch.int32, device=q.device) * Sq
kv_indptr = torch.arange(0, B + 1, dtype=torch.int32, device=q.device) * Skv
workspace = torch.empty(128 * 1024 * 1024, dtype=torch.uint8, device=q.device)
wrapper = BatchPrefillWithRaggedKVCacheWrapper(workspace, kv_layout="NHD")
wrapper.plan(
qo_indptr=qo_indptr,
kv_indptr=kv_indptr,
num_qo_heads=H,
num_kv_heads=Hkv,
head_dim_qk=D,
causal=test.is_causal,
logits_soft_cap=getattr(test, "softcap", None) or 0.0,
sm_scale=getattr(test, "sm_scale", None),
q_data_type=q.dtype,
)
def run_fn(q, k, v):
return wrapper.run(
q.reshape(-1, H, D),
k.reshape(-1, Hkv, D),
v.reshape(-1, Hkv, D),
).reshape(B, Sq, H, D)
return run_fn
def _torch_gqa_fwd(test):
"""Torch SDPA forward baseline."""
def fn(q, k, v):
out = F.scaled_dot_product_attention(
q.transpose(1, 2),
k.transpose(1, 2),
v.transpose(1, 2),
is_causal=test.is_causal,
enable_gqa=True,
)
return out.transpose(1, 2)
return fn
def _torch_gqa_bwd(test):
"""Torch SDPA backward baseline (includes forward recompute)."""
@torch.enable_grad()
def fn(q, k, v, o, grad_output, lse):
q = q.detach().requires_grad_(True)
k = k.detach().requires_grad_(True)
v = v.detach().requires_grad_(True)
out = F.scaled_dot_product_attention(
q.transpose(1, 2),
k.transpose(1, 2),
v.transpose(1, 2),
is_causal=test.is_causal,
enable_gqa=True,
)
out.transpose(1, 2).contiguous().backward(grad_output)
return q.grad, k.grad, v.grad
return fn
def _torch_gqa_prefill_ref(test: GQAPrefillFwdTest):
"""Materialized torch reference for dense prefill with bottom-right causal mask."""
def fn(q, k, v):
groups = test.heads // test.heads_kv
q_bhsd = q.transpose(1, 2).float()
k_bhsd = k.repeat_interleave(groups, dim=2).transpose(1, 2).float()
v_bhsd = v.repeat_interleave(groups, dim=2).transpose(1, 2).float()
sm_scale = getattr(test, "sm_scale", None)
softcap = getattr(test, "softcap", None)
scores = torch.matmul(q_bhsd, k_bhsd.transpose(-2, -1)) * (
test.dim**-0.5 if sm_scale is None else sm_scale
)
if softcap is not None and softcap > 0:
scores = softcap * torch.tanh(scores / softcap)
if test.is_causal:
offset = test.seq_len_kv - test.seq_len_q
q_pos = torch.arange(test.seq_len_q, device=q.device)[:, None] + offset
k_pos = torch.arange(test.seq_len_kv, device=q.device)[None, :]
mask = k_pos <= q_pos
scores = scores.masked_fill(
~mask.view(1, 1, test.seq_len_q, test.seq_len_kv), float("-inf")
)
probs = torch.softmax(scores, dim=-1)
return torch.matmul(probs, v_bhsd).transpose(1, 2).to(q.dtype).contiguous()
return fn
def _torch_gqa_prefill_varlen_ref(test: GQAPrefillVarlenFwdTest):
"""Materialized torch reference for packed-varlen prefill."""
def fn(q, k, v, cu_seqlens_q, cu_seqlens_kv):
groups = test.heads // test.heads_kv
outputs = []
for b in range(test.batch):
q_start = int(cu_seqlens_q[b].item())
q_end = int(cu_seqlens_q[b + 1].item())
kv_start = int(cu_seqlens_kv[b].item())
kv_end = int(cu_seqlens_kv[b + 1].item())
q_i = q[q_start:q_end].transpose(0, 1).float()
k_i = k[kv_start:kv_end].repeat_interleave(groups, dim=1).permute(1, 0, 2).float()
v_i = v[kv_start:kv_end].repeat_interleave(groups, dim=1).permute(1, 0, 2).float()
q_len = q_end - q_start
kv_len = kv_end - kv_start
scores = torch.matmul(q_i, k_i.transpose(-2, -1)) * (test.dim**-0.5)
if test.is_causal:
offset = kv_len - q_len
q_pos = torch.arange(q_len, device=q.device)[:, None] + offset
kv_pos = torch.arange(kv_len, device=q.device)[None, :]
mask = kv_pos <= q_pos
scores = scores.masked_fill(~mask.view(1, q_len, kv_len), float("-inf"))
probs = torch.softmax(scores, dim=-1)
outputs.append(torch.matmul(probs, v_i).transpose(0, 1).to(q.dtype).contiguous())
return torch.cat(outputs, dim=0)
return fn
def _tileops_gqa_variant(op: GroupedQueryAttentionFwdOp) -> str:
kernel = op.kernel
if isinstance(kernel, GQAPrefillFwdWsPersistentCausalKernel):
return "prefill_ws_causal"
if isinstance(kernel, GQAPrefillFwdKernel):
return "prefill"
if isinstance(kernel, GQAFwdWsPersistentCausalKernel):
return "ws_causal"
if isinstance(kernel, GQAFwdWsPersistentKernel):
return "ws_noncausal"
if isinstance(kernel, GQAFwdWgmmaPipelinedKernel):
return "wgmma_pipelined"
if isinstance(kernel, GQAFwdKernel):
return "legacy"
return kernel.__class__.__name__
# GQA forward benchmark parameters.
#
# Three head profiles cover the mainstream LLM GQA configurations:
# small (32:8:128) — Llama-3.1-8B, Qwen3-8B, Mistral-24B
# medium (64:8:128) — Llama-3.1-70B, Qwen3-32B, Qwen2.5-72B
# large (128:8:128) — Llama-3.1-405B
# head_dim=128 and kv_heads=8 are near-universal across Llama, Qwen3, and Mistral.
#
# Inference prefill (fp16): seq_len from 1K to 128K covers short chat to
# full-context workloads. B=1 because prefill is single-request in practice.
#
# Training (bf16): seq_len 2K-8K covers SFT (2K) and pretraining (4K-8K).
# B=1-2 reflects typical micro-batch sizes. No long-context training configs
# since >90% of pretraining compute is at 4K-8K.
_GQA_FWD_BENCH_PARAMS = manifest_params(load_workloads(_GQA_FWD_OP), gqa_qkv_args)
@pytest.mark.parametrize(
"batch, seq_len, heads, heads_kv, dim, causal, dtype, tune",
_GQA_FWD_BENCH_PARAMS,
)
def test_gqa_fwd_bench(
batch: int,
seq_len: int,
heads: int,
heads_kv: int,
dim: int,
causal: bool,
dtype: torch.dtype,
tune: bool,
) -> None:
test = GroupedQueryAttentionFwdTest(batch, heads, heads_kv, seq_len, dim, causal, dtype)
inputs = test.gen_inputs()
op = GroupedQueryAttentionFwdOp(batch, heads, heads_kv, seq_len, dim, causal, dtype, tune=tune)
bm = ManifestBenchmark(_GQA_FWD_OP, op, test)
tileops_variant = _tileops_gqa_variant(op)
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag=f"tileops_{tileops_variant}")
fa3_fn = _fa3_gqa_fwd(test)
if fa3_fn is not None:
result_bl = bm.profile(fa3_fn, *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="fa3")
fi_fn = _flashinfer_gqa_fwd(test, *inputs)
if fi_fn is not None:
result_fi = bm.profile(fi_fn, *inputs)
BenchmarkReport.record(op, locals(), result_fi, tag="flashinfer")
if fa3_fn is None and fi_fn is None:
result_bl = bm.profile(_torch_gqa_fwd(test), *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="torch-sdpa")
# GQA backward benchmark parameters (training only).
# Backward is only used during training — extract the training subset from
# _GQA_FWD_BENCH_PARAMS by ID prefix to avoid manual duplication.
_GQA_BWD_BENCH_PARAMS = manifest_params(load_workloads(_GQA_BWD_OP), gqa_qkv_args)
@pytest.mark.parametrize(
"batch, seq_len, heads, heads_kv, dim, causal, dtype, tune",
_GQA_BWD_BENCH_PARAMS,
)
def test_gqa_bwd_bench(
batch: int,
seq_len: int,
heads: int,
heads_kv: int,
dim: int,
causal: bool,
dtype: torch.dtype,
tune: bool,
) -> None:
test = GroupedQueryAttentionBwdTest(batch, heads, heads_kv, seq_len, dim, causal, dtype)
inputs = test.gen_inputs()
op = GroupedQueryAttentionBwdOp(batch, heads, heads_kv, seq_len, dim, causal, dtype, tune=tune)
bm = ManifestBenchmark(_GQA_BWD_OP, op, test)
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag="tileops")
fa3_fn = _fa3_gqa_bwd(test)
if fa3_fn is not None:
result_bl = bm.profile(fa3_fn, *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="fa3")
else:
result_bl = bm.profile(_torch_gqa_bwd(test), *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="torch-sdpa")
# No FlashInfer baseline for bwd (FlashInfer has no backward API)
_GQA_PREFILL_FWD_BENCH_PARAMS = manifest_params(
[workload for workload in load_workloads(_GQA_PREFILL_FWD_OP) if workload.get("backend") != "fp8"],
gqa_prefill_args,
tune=False,
)
@pytest.mark.parametrize(
"batch, seq_len_q, seq_len_kv, heads, heads_kv, dim, causal, backend, "
"validate_uniform_cu_seqlens, sm_scale, softcap, dtype, tune",
_GQA_PREFILL_FWD_BENCH_PARAMS,
)
def test_gqa_prefill_fwd_bench(
batch: int,
seq_len_q: int,
seq_len_kv: int,
heads: int,
heads_kv: int,
dim: int,
causal: bool,
backend: str,
validate_uniform_cu_seqlens: bool,
sm_scale: Optional[float],
softcap: Optional[float],
dtype: torch.dtype,
tune: bool,
) -> None:
test = GQAPrefillFwdTest(batch, heads, heads_kv, seq_len_q, seq_len_kv, dim, causal, dtype)
test.sm_scale = sm_scale
test.softcap = softcap
inputs = test.gen_inputs()
packed_inputs = _uniform_packed_prefill_inputs(*inputs)
op = GroupedQueryAttentionPrefillFwdOp(
batch=batch,
heads=heads,
heads_kv=heads_kv,
dim=dim,
max_seqlen_q=seq_len_q,
max_seqlen_kv=seq_len_kv,
is_causal=causal,
dtype=dtype,
tune=tune,
backend=backend,
validate_uniform_cu_seqlens=validate_uniform_cu_seqlens,
sm_scale=sm_scale,
softcap=softcap,
)
bm = ManifestBenchmark(_GQA_PREFILL_FWD_OP, op, test)
result = bm.profile(op, *packed_inputs)
BenchmarkReport.record(op, locals(), result, tag="tileops")
result_bl = bm.profile(_torch_gqa_prefill_ref(test), *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="torch-ref")
fi_fn = _flashinfer_gqa_fwd(test, *inputs)
if fi_fn is not None:
result_fi = bm.profile(fi_fn, *inputs)
BenchmarkReport.record(op, locals(), result_fi, tag="flashinfer")
_GQA_PREFILL_VARLEN_FWD_BENCH_PARAMS = [
pytest.param(
4,
[512, 512, 512, 512],
[1024, 1024, 1024, 1024],
32,
8,
128,
True,
torch.float16,
False,
id="llama-3.1-8b-prefill-varlen-uniform-fp16",
),
pytest.param(
4,
[128, 256, 640, 512],
[512, 768, 1280, 1024],
32,
8,
128,
True,
torch.float16,
False,
id="llama-3.1-8b-prefill-varlen-mixed-fp16",
),
pytest.param(
2,
[512, 512],
[1024, 2048],
64,
8,
128,
True,
torch.bfloat16,
False,
id="llama-3.1-70b-prefill-varlen-q-lt-kv-bf16",
),
]
@pytest.mark.parametrize(
"batch, q_lens, kv_lens, heads, heads_kv, dim, causal, dtype, tune",
_GQA_PREFILL_VARLEN_FWD_BENCH_PARAMS,
)
def test_gqa_prefill_varlen_fwd_bench(
batch: int,
q_lens: list[int],
kv_lens: list[int],
heads: int,
heads_kv: int,
dim: int,
causal: bool,
dtype: torch.dtype,
tune: bool,
) -> None:
test = GQAPrefillVarlenFwdTest(batch, heads, heads_kv, q_lens, kv_lens, dim, causal, dtype)
inputs = test.gen_inputs()
op = GroupedQueryAttentionPrefillVarlenFwdOp(
batch, heads, heads_kv, dim, test.max_seqlen_q, test.max_seqlen_kv, causal, dtype, tune=tune
)
bm = GQAPrefillVarlenFwdBenchmark(test)
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag="tileops")
result_bl = bm.profile(_torch_gqa_prefill_varlen_ref(test), *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="torch-ref")
def _fp8_paged_cache_inputs(
test: GQAPrefillPagedWithKVCacheFwdTest,
) -> tuple[torch.Tensor, ...]:
q, k_new, v_new, k_pages, v_pages, cu_seqlens_q, cache_seqlens, block_table, max_seqlen_q = (
test.gen_inputs()
)
k_scale = torch.full((1,), 0.01, dtype=torch.float32, device=q.device)
v_scale = torch.full((1,), 0.01, dtype=torch.float32, device=q.device)
fp8_max = torch.finfo(torch.float8_e4m3fn).max
k_pages = (k_pages / k_scale).clamp(-fp8_max, fp8_max).to(torch.float8_e4m3fn).contiguous()
v_pages = (v_pages / v_scale).clamp(-fp8_max, fp8_max).to(torch.float8_e4m3fn).contiguous()
return (
q,
k_new,
v_new,
k_pages,
v_pages,
k_scale,
v_scale,
cu_seqlens_q,
cache_seqlens,
block_table,
max_seqlen_q,
)
_GQA_PREFILL_PAGED_WITH_KV_CACHE_FWD_BENCH_PARAMS = manifest_params(
load_workloads(_GQA_PREFILL_PAGED_WITH_KV_CACHE_FWD_OP),
gqa_prefill_paged_args,
tune=False,
)
@pytest.mark.parametrize(
"batch, q_lens, cache_lens, heads, heads_kv, page_size, dim, causal, fuse_rope, "
"rotary_dim, softcap, cache_dtype, dtype, tune",
_GQA_PREFILL_PAGED_WITH_KV_CACHE_FWD_BENCH_PARAMS,
)
def test_gqa_prefill_paged_with_kv_cache_fwd_bench(
batch: int,
q_lens: list[int],
cache_lens: list[int],
heads: int,
heads_kv: int,
page_size: int,
dim: int,
causal: bool,
fuse_rope: bool,
rotary_dim: Optional[int],
softcap: Optional[float],
cache_dtype: Optional[torch.dtype],
dtype: torch.dtype,
tune: bool,
) -> None:
fp8_dtype = getattr(torch, "float8_e4m3fn", None)
if cache_dtype == fp8_dtype and fp8_dtype is not None:
if fuse_rope or rotary_dim is not None:
pytest.skip("FP8 paged KV cache benchmark does not support fused RoPE")
elif cache_dtype is not None and fp8_dtype is None:
pytest.skip("torch fp8 is unavailable")
test = GQAPrefillPagedWithKVCacheFwdTest(
batch,
heads,
heads_kv,
q_lens,
cache_lens,
page_size,
dim,
causal,
dtype,
fuse_rope=fuse_rope,
rotary_dim=rotary_dim,
softcap=softcap,
)
if cache_dtype == fp8_dtype and fp8_dtype is not None:
inputs = _fp8_paged_cache_inputs(test)
else:
q, k_new, v_new, k_pages, v_pages, cu_seqlens_q, cache_seqlens, block_table, max_seqlen_q = (
test.gen_inputs()
)
k_scale = torch.ones((1,), dtype=torch.float32, device=q.device)
v_scale = torch.ones((1,), dtype=torch.float32, device=q.device)
inputs = (
q,
k_new,
v_new,
k_pages,
v_pages,
k_scale,
v_scale,
cu_seqlens_q,
cache_seqlens,
block_table,
max_seqlen_q,
)
op = GroupedQueryAttentionPrefillPagedWithKVCacheFwdOp(
batch=batch,
heads=heads,
heads_kv=heads_kv,
max_pages_per_req=test.max_pages_per_req,
page_size=page_size,
dim=dim,
is_causal=causal,
dtype=dtype,
cache_dtype=cache_dtype,
softcap=softcap,
tune=tune,
fuse_rope=fuse_rope,
max_position=test.max_total_len if fuse_rope else None,
rotary_dim=rotary_dim,
)
op.total_q = test.total_q
op.q_lens = q_lens
op.cache_lens = cache_lens
op.max_seqlen_q = test.max_seqlen_q
bm = ManifestBenchmark(_GQA_PREFILL_PAGED_WITH_KV_CACHE_FWD_OP, op, test)
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag="tileops")
if __name__ == "__main__":
pytest.main([__file__, "-vvs"])

173
benchmarks/ops/attention/bench_gqa_decode.py
View File

@ -1,173 +0,0 @@
import pytest
import torch
import torch.nn.functional as F
from benchmarks.benchmark_base import BenchmarkReport, ManifestBenchmark
from benchmarks.ops.attention.manifest_params import gqa_decode_args, manifest_params
from tileops.manifest import load_workloads
from tileops.ops import GroupedQueryAttentionDecodeWithKVCacheFwdOp
from workloads.attention.gqa import GroupedQueryAttentionDecodeTest
_OP_NAME = "GroupedQueryAttentionDecodeWithKVCacheFwdOp"
class _GroupedQueryAttentionDecodeTestBaseline(GroupedQueryAttentionDecodeTest):
"""Adds baseline ref_program for benchmark profiling."""
def ref_program(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:
q_bhsd = q.unsqueeze(1).transpose(1, 2) # [B, H, 1, D]
groups = self.heads // self.heads_kv
k_bhsd = k.repeat_interleave(groups, dim=2).transpose(1, 2).float()
v_bhsd = v.repeat_interleave(groups, dim=2).transpose(1, 2).float()
scores = torch.matmul(q_bhsd.float(), k_bhsd.transpose(-2, -1)) * self.sm_scale
if self.softcap > 0:
scores = self.softcap * torch.tanh(scores / self.softcap)
probs = torch.softmax(scores, dim=-1)
output_bhsd = torch.matmul(probs, v_bhsd)
return output_bhsd.transpose(1, 2).squeeze(1).to(q.dtype).contiguous()
def _fa3_gqa_decode_fwd(test):
"""Return FA3 KV-cache decode baseline callable, or None if not installed."""
if test.sm_scale != test.dim**-0.5 or test.softcap != 0.0:
return None
try:
from flash_attn_interface import flash_attn_with_kvcache
except ImportError:
return None
cache_seqlens = torch.full(
(test.batch,), test.seq_len_kv, dtype=torch.int32, device="cuda")
def baseline_fn(q, k, v):
# Q is (B, H, D); FA3 KV-cache decode expects (B, S_q, H, D).
out = flash_attn_with_kvcache(q.unsqueeze(1), k, v, cache_seqlens=cache_seqlens)
out = out[0] if isinstance(out, tuple) else out
return out.squeeze(1)
return baseline_fn
def _flashinfer_gqa_decode_fwd(test, q, k, v):
"""Set up FlashInfer decode for a non-paged KV cache.
For a single request (B == 1) the KV cache is contiguous, so the
``single_decode_with_kv_cache`` kernel is the apples-to-apples baseline
(no paging overhead). For B > 1 the contiguous (B, S_kv, H_kv, D) KV is
reshaped into paged format with page_size=256 and the batched paged
decode kernel is used.
FlashInfer decode kernels support group_size (Q/KV head ratio) up to 8.
"""
if test.sm_scale != test.dim**-0.5 or test.softcap != 0.0:
return None
# Q is (B, H, D) — single token per request
# K/V is (B, S_kv, H_kv, D)
B, H, D = q.shape
Hkv = k.shape[2]
if H // Hkv > 8:
return None # FlashInfer decode kernel does not support group_size > 8
if B == 1:
try:
from flashinfer.decode import single_decode_with_kv_cache
except ImportError:
return None
# single_decode expects q (H, D) and k/v (S_kv, H_kv, D) for one request.
q_s = q.reshape(H, D)
k_s = k.reshape(k.shape[1], Hkv, D)
v_s = v.reshape(v.shape[1], Hkv, D)
def run_fn(q, k, v):
return single_decode_with_kv_cache(
q_s, k_s, v_s, kv_layout="NHD", use_tensor_cores=True,
)
return run_fn
try:
from flashinfer.decode import BatchDecodeWithPagedKVCacheWrapper
except ImportError:
return None
Skv = k.shape[1]
page_size = 256
pages_per_seq = (Skv + page_size - 1) // page_size
# Reshape (B, S_kv, H_kv, D) → (B * pages_per_seq, page_size, H_kv, D)
# Pad S_kv to a multiple of page_size if needed
if Skv % page_size != 0:
pad = page_size * pages_per_seq - Skv
k = torch.nn.functional.pad(k, (0, 0, 0, 0, 0, pad))
v = torch.nn.functional.pad(v, (0, 0, 0, 0, 0, pad))
k_paged = k.reshape(B * pages_per_seq, page_size, Hkv, D)
v_paged = v.reshape(B * pages_per_seq, page_size, Hkv, D)
kv_data = (k_paged, v_paged)
total_pages = B * pages_per_seq
indptr = torch.arange(0, B + 1, dtype=torch.int32, device=q.device) * pages_per_seq
indices = torch.arange(0, total_pages, dtype=torch.int32, device=q.device)
last_page_len_val = Skv - (pages_per_seq - 1) * page_size
last_page_len = torch.full((B,), last_page_len_val, dtype=torch.int32, device=q.device)
workspace = torch.empty(128 * 1024 * 1024, dtype=torch.uint8, device=q.device)
wrapper = BatchDecodeWithPagedKVCacheWrapper(workspace, kv_layout="NHD")
wrapper.plan(
indptr=indptr, indices=indices, last_page_len=last_page_len,
num_qo_heads=H, num_kv_heads=Hkv, head_dim=D,
page_size=page_size,
q_data_type=q.dtype,
)
def run_fn(q, k, v):
return wrapper.run(q, kv_data)
return run_fn
_GQA_DECODE_BENCH_PARAMS = manifest_params(load_workloads(_OP_NAME), gqa_decode_args)
@pytest.mark.parametrize(
"batch, heads, heads_kv, seq_len_kv, dim, sm_scale, softcap, dtype, tune",
_GQA_DECODE_BENCH_PARAMS,
)
def test_gqa_decode_bench(batch: int, heads: int, heads_kv: int, seq_len_kv: int, dim: int,
sm_scale: float | None, softcap: float | None, dtype: torch.dtype,
tune: bool) -> None:
test = _GroupedQueryAttentionDecodeTestBaseline(
batch, heads, heads_kv, seq_len_kv, dim, dtype, sm_scale=sm_scale, softcap=softcap)
inputs = test.gen_inputs()
op = GroupedQueryAttentionDecodeWithKVCacheFwdOp(
batch,
heads,
heads_kv,
seq_len_kv,
dim,
dtype,
sm_scale=sm_scale,
softcap=softcap,
tune=tune,
)
bm = ManifestBenchmark(_OP_NAME, op, test)
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag="tileops")
fa3_fn = _fa3_gqa_decode_fwd(test)
if fa3_fn is not None:
result_bl = bm.profile(fa3_fn, *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="fa3")
fi_fn = _flashinfer_gqa_decode_fwd(test, *inputs)
if fi_fn is not None:
result_fi = bm.profile(fi_fn, *inputs)
BenchmarkReport.record(op, locals(), result_fi, tag="flashinfer")
if fa3_fn is None and fi_fn is None:
result_bl = bm.profile(test.ref_program, *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="torch-sdpa")
if __name__ == "__main__":
pytest.main([__file__, "-vvs"])

174
benchmarks/ops/attention/bench_gqa_decode_paged.py
View File

@ -1,174 +0,0 @@
import math
import pytest
import torch
import torch.nn.functional as F
from torch.nn.attention import SDPBackend, sdpa_kernel
from benchmarks.benchmark_base import BenchmarkReport, ManifestBenchmark
from benchmarks.ops.attention.manifest_params import gqa_decode_paged_args, manifest_params
from tileops.manifest import load_workloads
from tileops.ops import GroupedQueryAttentionDecodePagedWithKVCacheFwdOp
from workloads.attention.gqa import GroupedQueryAttentionDecodePagedTest
_OP_NAME = "GroupedQueryAttentionDecodePagedWithKVCacheFwdOp"
class _GroupedQueryAttentionDecodePagedTestBaseline(GroupedQueryAttentionDecodePagedTest):
"""Adds baseline ref_program for benchmark profiling."""
def ref_program(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor,
real_seqlen_kv: torch.Tensor, block_table: torch.Tensor) -> torch.Tensor:
"""Reassemble paged K/V to logical layout per batch, then GQA (expand to heads) + SDPA."""
batch, _, dim = q.shape
seqlen_kv, _, _ = k.shape
kv_group_num = self.heads // self.heads_kv
out_list = []
for i_b in range(batch):
q_b = q[i_b:i_b + 1, :, :]
k_logical = torch.zeros(seqlen_kv, self.heads_kv, dim, dtype=q.dtype, device=q.device)
v_logical = torch.zeros(seqlen_kv, self.heads_kv, dim, dtype=q.dtype, device=q.device)
num_pages = math.ceil(real_seqlen_kv[i_b].item() / self.page_size)
for i_paged in range(num_pages):
start_pos = block_table[i_b, i_paged].item() * self.page_size
end_pos = min(start_pos + self.page_size, seqlen_kv)
page_len = end_pos - start_pos
k_logical[i_paged * self.page_size:i_paged * self.page_size +
page_len, :, :] = k[start_pos:end_pos, :, :]
v_logical[i_paged * self.page_size:i_paged * self.page_size +
page_len, :, :] = v[start_pos:end_pos, :, :]
k_logical = k_logical[:real_seqlen_kv[i_b].item(), :, :]
v_logical = v_logical[:real_seqlen_kv[i_b].item(), :, :]
group_id = torch.arange(self.heads, dtype=torch.long, device=q.device) // kv_group_num
k_bhsd = k_logical[:, group_id, :].unsqueeze(0).transpose(1, 2)
v_bhsd = v_logical[:, group_id, :].unsqueeze(0).transpose(1, 2)
q_bhsd = q_b.unsqueeze(2)
with sdpa_kernel(backends=[SDPBackend.MATH]):
out_b = F.scaled_dot_product_attention(q_bhsd, k_bhsd, v_bhsd)
out_b = out_b.squeeze(2)
out_list.append(out_b)
return torch.cat(out_list, dim=0)
def _fa3_gqa_decode_paged(test, k, v):
"""Set up FA3 paged decode. Returns callable or None.
FA3 requires page_block_size to be a multiple of 256.
"""
if test.page_size % 256 != 0:
return None
try:
from flash_attn_interface import flash_attn_with_kvcache
except ImportError:
return None
num_pages = k.shape[0] // test.page_size
k_paged = k.view(num_pages, test.page_size, test.heads_kv, test.dim)
v_paged = v.view(num_pages, test.page_size, test.heads_kv, test.dim)
def baseline_fn(q, k, v, real_seqlen_kv, block_table):
# Q is (batch, heads, dim) — add seq dim for flash_attn
out = flash_attn_with_kvcache(
q.unsqueeze(1), k_paged, v_paged,
cache_seqlens=real_seqlen_kv.int(),
page_table=block_table.int())
out = out[0] if isinstance(out, tuple) else out
return out.squeeze(1)
return baseline_fn
def _flashinfer_gqa_decode_paged(test, q, k, v, real_seqlen_kv, block_table):
"""Set up FlashInfer paged decode wrapper. Returns callable or None.
FlashInfer decode kernel supports group_size (Q/KV head ratio) up to 8.
"""
try:
from flashinfer.decode import BatchDecodeWithPagedKVCacheWrapper
except ImportError:
return None
if test.heads // test.heads_kv > 8:
return None # FlashInfer decode kernel does not support group_size > 8
batch = q.shape[0]
num_pages = k.shape[0] // test.page_size
k_paged = k.view(num_pages, test.page_size, test.heads_kv, test.dim)
v_paged = v.view(num_pages, test.page_size, test.heads_kv, test.dim)
kv_data = (k_paged, v_paged)
pages_per_batch = (real_seqlen_kv.int() + test.page_size - 1) // test.page_size
indptr = torch.zeros(batch + 1, dtype=torch.int32, device=q.device)
indptr[1:] = torch.cumsum(pages_per_batch, dim=0)
indices_list = []
for b in range(batch):
n = pages_per_batch[b].item()
indices_list.append(block_table[b, :n])
indices = torch.cat(indices_list)
last_page_len = (real_seqlen_kv.int() - 1) % test.page_size + 1
workspace = torch.empty(128 * 1024 * 1024, dtype=torch.uint8, device=q.device)
wrapper = BatchDecodeWithPagedKVCacheWrapper(workspace, kv_layout="NHD")
wrapper.plan(
indptr=indptr,
indices=indices,
last_page_len=last_page_len,
num_qo_heads=test.heads,
num_kv_heads=test.heads_kv,
head_dim=test.dim,
page_size=test.page_size,
q_data_type=test.dtype,
)
def run_fn(q, k, v, real_seqlen_kv, block_table):
# Q is (batch, heads, dim)
return wrapper.run(q, kv_data)
return run_fn
_GQA_DECODE_PAGED_BENCH_PARAMS = manifest_params(
load_workloads(_OP_NAME),
gqa_decode_paged_args,
)
@pytest.mark.parametrize(
"batch, heads, heads_kv, seqlen_kv, dim, page_size, sm_scale, softcap, dtype, tune",
_GQA_DECODE_PAGED_BENCH_PARAMS,
)
def test_gqa_decode_paged_bench(batch: int, heads: int, heads_kv: int, seqlen_kv: int, dim: int,
page_size: int, sm_scale: float | None,
softcap: float | None, dtype: torch.dtype, tune: bool) -> None:
test = _GroupedQueryAttentionDecodePagedTestBaseline(
batch, heads, heads_kv, seqlen_kv, dim, page_size, dtype,
sm_scale=sm_scale, softcap=softcap)
inputs = test.gen_inputs()
q, k, v, real_seqlen_kv, block_table = inputs
op = GroupedQueryAttentionDecodePagedWithKVCacheFwdOp(
batch, heads, heads_kv, seqlen_kv, dim, page_size, dtype,
sm_scale=sm_scale, softcap=softcap, tune=tune)
bm = ManifestBenchmark(_OP_NAME, op, test)
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag="tileops")
fa3_fn = _fa3_gqa_decode_paged(test, k, v)
if fa3_fn is not None:
result_fa3 = bm.profile(fa3_fn, *inputs)
BenchmarkReport.record(op, locals(), result_fa3, tag="fa3")
fi_fn = _flashinfer_gqa_decode_paged(test, *inputs)
if fi_fn is not None:
result_fi = bm.profile(fi_fn, *inputs)
BenchmarkReport.record(op, locals(), result_fi, tag="flashinfer")
if fa3_fn is None and fi_fn is None:
result_bl = bm.profile(test.ref_program, *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="torch-ref")
if __name__ == "__main__":
pytest.main([__file__, "-vvs"])

158
benchmarks/ops/attention/bench_gqa_fp8.py
View File

@ -1,158 +0,0 @@
from dataclasses import dataclass
import pytest
import torch
from benchmarks.benchmark_base import BenchmarkReport, bench_kernel
from tileops.manifest import load_workloads
from tileops.ops import GroupedQueryAttentionPrefillFwdOp
from tileops.testing.gqa_fp8_utils import (
quantize_kv_fa3_descale,
quantize_q_fa3_gqa_descale,
)
_OP_NAME = "GroupedQueryAttentionPrefillFwdOp"
@dataclass(frozen=True)
class GQAFp8TensorCoreBenchCase:
batch: int
seq_len: int
heads: int
heads_kv: int
dim: int
validate_uniform_cu_seqlens: bool
out_dtype: torch.dtype
label: str
def _manifest_cases() -> list[GQAFp8TensorCoreBenchCase]:
cases: list[GQAFp8TensorCoreBenchCase] = []
for workload in load_workloads(_OP_NAME):
if workload.get("backend") != "fp8":
continue
batch = workload["batch"]
seq_len = workload["max_seqlen_q"]
heads = workload["heads"]
heads_kv = workload["heads_kv"]
dim = workload["dim"]
for dtype_name in workload["dtypes"]:
out_dtype = getattr(torch, dtype_name)
cases.append(
GQAFp8TensorCoreBenchCase(
batch=batch,
seq_len=seq_len,
heads=heads,
heads_kv=heads_kv,
dim=dim,
validate_uniform_cu_seqlens=workload.get(
"validate_uniform_cu_seqlens", True
),
out_dtype=out_dtype,
label=f"{workload['label']}-{dtype_name}",
)
)
return cases
def _make_inputs(case: GQAFp8TensorCoreBenchCase) -> tuple[torch.Tensor, ...]:
torch.manual_seed(0)
q = (
torch.randn(
case.batch, case.seq_len, case.heads, case.dim, device="cuda", dtype=torch.float16
)
* 0.25
)
k = (
torch.randn(
case.batch, case.seq_len, case.heads_kv, case.dim, device="cuda", dtype=torch.float16
)
* 0.25
)
v = (
torch.randn(
case.batch, case.seq_len, case.heads_kv, case.dim, device="cuda", dtype=torch.float16
)
* 0.25
)
q_fp8, q_descale = quantize_q_fa3_gqa_descale(q, case.heads_kv)
k_fp8, k_descale = quantize_kv_fa3_descale(k)
v_fp8, v_descale = quantize_kv_fa3_descale(v)
cu = torch.tensor([0, case.seq_len], device="cuda", dtype=torch.int32)
return (
q_fp8.reshape(case.batch * case.seq_len, case.heads, case.dim).contiguous(),
k_fp8.reshape(case.batch * case.seq_len, case.heads_kv, case.dim).contiguous(),
v_fp8.reshape(case.batch * case.seq_len, case.heads_kv, case.dim).contiguous(),
cu,
cu,
q_descale,
k_descale,
v_descale,
)
def _fa3_gqa_fp8_fwd(case: GQAFp8TensorCoreBenchCase):
try:
from flash_attn_interface import flash_attn_func
except Exception:
return None
def _run(q, k, v, cu_q, cu_kv, q_descale, k_descale, v_descale):
del cu_q, cu_kv
return flash_attn_func(
q.reshape(case.batch, case.seq_len, case.heads, case.dim),
k.reshape(case.batch, case.seq_len, case.heads_kv, case.dim),
v.reshape(case.batch, case.seq_len, case.heads_kv, case.dim),
causal=False,
q_descale=q_descale,
k_descale=k_descale,
v_descale=v_descale,
)
return _run
@pytest.mark.parametrize("case", [pytest.param(c, id=c.label) for c in _manifest_cases()])
def test_gqa_prefill_fp8_tensor_core_bench(case: GQAFp8TensorCoreBenchCase) -> None:
if not hasattr(torch, "float8_e4m3fn"):
pytest.skip("torch fp8 is unavailable")
if not torch.cuda.is_available() or torch.cuda.get_device_capability()[0] < 9:
pytest.skip("requires Hopper FP8 WGMMA")
op = GroupedQueryAttentionPrefillFwdOp(
batch=case.batch,
heads=case.heads,
heads_kv=case.heads_kv,
dim=case.dim,
max_seqlen_q=case.seq_len,
max_seqlen_kv=case.seq_len,
is_causal=False,
dtype=case.out_dtype,
backend="fp8",
validate_uniform_cu_seqlens=case.validate_uniform_cu_seqlens,
)
inputs = _make_inputs(case)
op(*inputs)
torch.cuda.synchronize()
latency_ms = bench_kernel(op, args=inputs, n_warmup=1, n_repeat=3, n_trials=1)
flops, bytes_moved = op.eval_roofline()
result = {
"latency_ms": latency_ms,
"tflops": flops / latency_ms * 1e-9 if latency_ms > 0 else 0.0,
"gbps": bytes_moved / latency_ms * 1e-6 if latency_ms > 0 else 0.0,
"flops": flops,
"bytes": bytes_moved,
}
BenchmarkReport.record(op, {"case": case.label}, result, tag="tileops")
fa3_fn = _fa3_gqa_fp8_fwd(case)
if fa3_fn is not None:
fa3_latency_ms = bench_kernel(fa3_fn, args=inputs, n_warmup=1, n_repeat=3, n_trials=1)
fa3_result = {
"latency_ms": fa3_latency_ms,
"tflops": flops / fa3_latency_ms * 1e-9 if fa3_latency_ms > 0 else 0.0,
"gbps": bytes_moved / fa3_latency_ms * 1e-6 if fa3_latency_ms > 0 else 0.0,
"flops": flops,
"bytes": bytes_moved,
}
BenchmarkReport.record(op, {"case": case.label}, fa3_result, tag="fa3")

166
benchmarks/ops/attention/bench_gqa_sliding_window.py
View File

@ -1,166 +0,0 @@
"""Benchmark for GroupedQueryAttentionSlidingWindowFwdOp vs FA3 baseline."""
import pytest
import torch
from torch.nn import functional as F
from benchmarks.benchmark_base import BenchmarkReport, ManifestBenchmark
from benchmarks.ops.attention.manifest_params import (
gqa_sliding_window_args,
manifest_params,
)
from tileops.manifest import load_workloads
from tileops.ops import GroupedQueryAttentionSlidingWindowFwdOp
from workloads.attention.gqa import GroupedQueryAttentionSlidingWindowFwdTest
_OP_NAME = "GroupedQueryAttentionSlidingWindowFwdOp"
def _torch_sliding_window_fwd(test):
"""Torch SDPA forward baseline with explicit sliding window mask."""
def fn(q, k, v):
S = test.seq
q_idx = torch.arange(S, device=q.device).unsqueeze(1)
k_idx = torch.arange(S, device=q.device).unsqueeze(0)
mask = torch.zeros(S, S, dtype=torch.bool, device=q.device)
if test.is_causal:
mask |= k_idx > q_idx
if test.wl >= 0:
mask |= k_idx < q_idx - test.wl
if test.wr >= 0:
mask |= k_idx > q_idx + test.wr
attn_mask = torch.zeros(S, S, dtype=q.dtype, device=q.device)
attn_mask.masked_fill_(mask, float("-inf"))
out = F.scaled_dot_product_attention(
q.transpose(1, 2),
k.transpose(1, 2),
v.transpose(1, 2),
attn_mask=attn_mask,
enable_gqa=True,
)
return out.transpose(1, 2)
return fn
def _fa3_baseline(is_causal, wl, wr):
"""Return FA3 sliding-window baseline callable, or None if not installed."""
try:
from flash_attn_interface import flash_attn_func
except ImportError:
return None
def baseline_fn(q, k, v):
out = flash_attn_func(q, k, v, causal=is_causal, window_size=(wl, wr))
return out[0] if isinstance(out, tuple) else out
return baseline_fn
def _flashinfer_sliding_window_fwd(test, q, k, v):
"""Set up FlashInfer batched prefill with sliding window. Returns callable or None.
FlashInfer only supports window_left; skip when window_right >= 0.
"""
if test.wr >= 0:
return None
try:
from flashinfer.prefill import BatchPrefillWithRaggedKVCacheWrapper
except ImportError:
return None
B, S, H, D = q.shape
Hkv = k.shape[2]
cu_seqlens = torch.arange(0, B + 1, dtype=torch.int32, device=q.device) * S
workspace = torch.empty(128 * 1024 * 1024, dtype=torch.uint8, device=q.device)
wrapper = BatchPrefillWithRaggedKVCacheWrapper(workspace, kv_layout="NHD")
wrapper.plan(
qo_indptr=cu_seqlens,
kv_indptr=cu_seqlens,
num_qo_heads=H,
num_kv_heads=Hkv,
head_dim_qk=D,
causal=test.is_causal,
window_left=test.wl,
q_data_type=q.dtype,
)
def run_fn(q, k, v):
return wrapper.run(
q.reshape(-1, H, D),
k.reshape(-1, Hkv, D),
v.reshape(-1, Hkv, D),
).reshape(B, S, H, D)
return run_fn
_GQA_SLIDING_WINDOW_FWD_BENCH_PARAMS = manifest_params(
load_workloads(_OP_NAME),
gqa_sliding_window_args,
)
@pytest.mark.parametrize(
"batch, seq, heads, heads_kv, dim, is_causal, wl, wr, dtype, tune",
_GQA_SLIDING_WINDOW_FWD_BENCH_PARAMS,
)
def test_gqa_sliding_window_fwd_bench(
batch: int,
seq: int,
heads: int,
heads_kv: int,
dim: int,
is_causal: bool,
wl: int,
wr: int,
dtype: torch.dtype,
tune: bool,
) -> None:
test = GroupedQueryAttentionSlidingWindowFwdTest(
batch, seq, heads, heads_kv, dim, is_causal, wl, wr, dtype
)
inputs = test.gen_inputs()
op = GroupedQueryAttentionSlidingWindowFwdOp(
batch=batch,
heads=heads,
heads_kv=heads_kv,
seq_len=seq,
dim=dim,
is_causal=is_causal,
window_size_left=wl,
window_size_right=wr,
dtype=dtype,
tune=tune,
)
bm = ManifestBenchmark(_OP_NAME, op, test)
# Warmup: trigger JIT compilation before timed profiling
op(*inputs)
torch.cuda.synchronize()
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag="tileops")
# FA3 baseline
fa3_fn = _fa3_baseline(is_causal, wl, wr)
if fa3_fn is not None:
result_bl = bm.profile(fa3_fn, *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="fa3")
# FlashInfer baseline
fi_fn = _flashinfer_sliding_window_fwd(test, *inputs)
if fi_fn is not None:
result_fi = bm.profile(fi_fn, *inputs)
BenchmarkReport.record(op, locals(), result_fi, tag="flashinfer")
if fa3_fn is None and fi_fn is None:
result_bl = bm.profile(_torch_sliding_window_fwd(test), *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="torch")
if __name__ == "__main__":
pytest.main([__file__, "-vvs"])

209
benchmarks/ops/attention/bench_gqa_sliding_window_varlen.py
View File

@ -1,209 +0,0 @@
"""Benchmark for GroupedQueryAttentionSlidingWindowVarlenFwdOp vs FA3 baseline."""
import pytest
import torch
from torch.nn import functional as F
from benchmarks.benchmark_base import BenchmarkReport, ManifestBenchmark
from benchmarks.ops.attention.manifest_params import (
gqa_sliding_window_varlen_args,
manifest_params,
)
from tileops.manifest import load_workloads
from tileops.ops import GroupedQueryAttentionSlidingWindowVarlenFwdOp
from workloads.attention.gqa import (
GroupedQueryAttentionSlidingWindowVarlenFwdTest,
)
_OP_NAME = "GroupedQueryAttentionSlidingWindowVarlenFwdOp"
_GQA_SLIDING_WINDOW_VARLEN_FWD_BENCH_PARAMS = manifest_params(
load_workloads(_OP_NAME),
gqa_sliding_window_varlen_args,
tune=False,
)
def _torch_sliding_window_varlen_fwd(test):
"""Torch SDPA forward baseline: unpack varlen to padded batch, single SDPA call."""
def fn(q, k, v, cu_seqlens_q, cu_seqlens_k, max_seqlen_q):
B = test.batch
seqlens_q = test.seqlens_q
seqlens_k = test.seqlens_k
max_sq = max(seqlens_q)
max_sk = max(seqlens_k)
H, Hkv, D = test.heads, test.heads_kv, test.dim
# Unpack packed varlen tensors to padded [B, max_s, H, D]
q_pad = q.new_zeros(B, max_sq, H, D)
k_pad = k.new_zeros(B, max_sk, Hkv, D)
v_pad = v.new_zeros(B, max_sk, Hkv, D)
for i in range(B):
qs, qe = cu_seqlens_q[i].item(), cu_seqlens_q[i + 1].item()
ks, ke = cu_seqlens_k[i].item(), cu_seqlens_k[i + 1].item()
q_pad[i, : qe - qs] = q[qs:qe]
k_pad[i, : ke - ks] = k[ks:ke]
v_pad[i, : ke - ks] = v[ks:ke]
# Build combined mask [B, max_sq, max_sk]
q_idx = torch.arange(max_sq, device=q.device).view(1, -1, 1)
k_idx = torch.arange(max_sk, device=q.device).view(1, 1, -1)
sq_t = torch.tensor(seqlens_q, device=q.device, dtype=torch.long).view(B, 1, 1)
sk_t = torch.tensor(seqlens_k, device=q.device, dtype=torch.long).view(B, 1, 1)
offsets = sk_t - sq_t # per-sample offset [B, 1, 1]
# Padding positions
mask = (q_idx >= sq_t) | (k_idx >= sk_t)
# Sliding window + causal constraints
if test.is_causal:
mask = mask | (k_idx > q_idx + offsets)
if test.wl >= 0:
mask = mask | (k_idx < q_idx + offsets - test.wl)
if test.wr >= 0:
mask = mask | (k_idx > q_idx + offsets + test.wr)
attn_mask = torch.zeros(B, max_sq, max_sk, dtype=q.dtype, device=q.device)
attn_mask.masked_fill_(mask, float("-inf"))
# SDPA call: transpose to [B, H, S, D], mask broadcasts as [B, 1, max_sq, max_sk]
out = F.scaled_dot_product_attention(
q_pad.transpose(1, 2),
k_pad.transpose(1, 2),
v_pad.transpose(1, 2),
attn_mask=attn_mask.unsqueeze(1),
enable_gqa=True,
)
out = out.transpose(1, 2) # [B, max_sq, H, D]
# Repack valid positions to [total_q, H, D]
parts = [out[i, : seqlens_q[i]] for i in range(B)]
return torch.cat(parts, dim=0)
return fn
def _fa3_varlen_baseline(max_seqlen_k, is_causal, wl, wr):
"""Return FA3 varlen baseline callable, or None if not installed."""
try:
from flash_attn_interface import flash_attn_varlen_func
except ImportError:
return None
def baseline_fn(q, k, v, cu_seqlens_q, cu_seqlens_k, max_seqlen_q):
out = flash_attn_varlen_func(
q,
k,
v,
cu_seqlens_q=cu_seqlens_q,
cu_seqlens_k=cu_seqlens_k,
max_seqlen_q=max_seqlen_q,
max_seqlen_k=max_seqlen_k,
causal=is_causal,
window_size=(wl, wr),
)
return out[0] if isinstance(out, tuple) else out
return baseline_fn
def _flashinfer_varlen_sliding_window_fwd(test, q, k, v, cu_seqlens_q, cu_seqlens_k, max_seqlen_q):
"""Set up FlashInfer ragged prefill wrapper. Returns callable or None.
FlashInfer only supports window_left; skip when window_right >= 0.
"""
if test.wr >= 0:
return None
try:
from flashinfer.prefill import BatchPrefillWithRaggedKVCacheWrapper
except ImportError:
return None
workspace = torch.empty(128 * 1024 * 1024, dtype=torch.uint8, device=q.device)
wrapper = BatchPrefillWithRaggedKVCacheWrapper(workspace, kv_layout="NHD")
wrapper.plan(
qo_indptr=cu_seqlens_q,
kv_indptr=cu_seqlens_k,
num_qo_heads=test.heads,
num_kv_heads=test.heads_kv,
head_dim_qk=test.dim,
causal=test.is_causal,
window_left=test.wl,
q_data_type=q.dtype,
)
def run_fn(q, k, v, cu_seqlens_q, cu_seqlens_k, max_seqlen_q):
return wrapper.run(q, k, v)
return run_fn
@pytest.mark.parametrize(
"batch, seqlens_q, seqlens_k, heads, heads_kv, dim, is_causal, wl, wr, dtype, tune",
_GQA_SLIDING_WINDOW_VARLEN_FWD_BENCH_PARAMS,
)
def test_gqa_sliding_window_varlen_fwd_bench(
batch: int,
seqlens_q,
seqlens_k,
heads: int,
heads_kv: int,
dim: int,
is_causal: bool,
wl: int,
wr: int,
dtype: torch.dtype,
tune: bool,
) -> None:
test = GroupedQueryAttentionSlidingWindowVarlenFwdTest(
batch, seqlens_q, seqlens_k, heads, heads_kv, dim, is_causal, wl, wr, dtype
)
inputs = test.gen_inputs()
q, k, v, cu_seqlens_q, cu_seqlens_k, max_seqlen_q = inputs
op = GroupedQueryAttentionSlidingWindowVarlenFwdOp(
batch=batch,
heads=heads,
heads_kv=heads_kv,
dim=dim,
is_causal=is_causal,
window_size_left=wl,
window_size_right=wr,
dtype=dtype,
tune=tune,
)
op.total_q = sum(seqlens_q)
op.total_k = sum(seqlens_k)
op.q_lens = seqlens_q
op.k_lens = seqlens_k
op.max_seqlen_q = max(seqlens_q)
op.max_seqlen_k = max(seqlens_k)
bm = ManifestBenchmark(_OP_NAME, op, test)
# Warmup: trigger JIT compilation before timed profiling
op(*inputs)
torch.cuda.synchronize()
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag="tileops")
# FA3 baseline
max_seqlen_k = max(seqlens_k)
fa3_fn = _fa3_varlen_baseline(max_seqlen_k, is_causal, wl, wr)
if fa3_fn is not None:
result_bl = bm.profile(fa3_fn, *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="fa3")
# FlashInfer baseline
fi_fn = _flashinfer_varlen_sliding_window_fwd(test, *inputs)
if fi_fn is not None:
result_fi = bm.profile(fi_fn, *inputs)
BenchmarkReport.record(op, locals(), result_fi, tag="flashinfer")
if fa3_fn is None and fi_fn is None:
result_bl = bm.profile(_torch_sliding_window_varlen_fwd(test), *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="torch")
if __name__ == "__main__":
pytest.main([__file__, "-vvs"])

137
benchmarks/ops/attention/bench_mean_pooling.py
View File

@ -1,137 +0,0 @@
from typing import Optional
import pytest
import torch
from benchmarks.benchmark_base import BenchmarkBase, BenchmarkReport
from tileops.ops import MeanPoolingForwardOp
from workloads.attention.mean_pooling import MeanPoolingTest
from workloads.nsa_utils import prepare_chunk_indices
class _MeanPoolingTestBaseline(MeanPoolingTest):
"""Adds baseline ref_program for benchmark profiling."""
def ref_program(self, x: torch.Tensor, offsets: torch.Tensor,
indices: torch.Tensor) -> torch.Tensor:
_ = indices
batch_size, seq_len, heads, dim = x.shape
if self.use_offsets == 0:
output = torch.empty(
batch_size, self.chunks_per_bacth, heads, dim, dtype=x.dtype, device=x.device)
for chunk_id in range(self.chunks_per_bacth):
start_token = chunk_id * self.chunk_size
end_token = min(start_token + self.chunk_size, seq_len)
output[:, chunk_id] = x[:, start_token:end_token].mean(dim=1)
else:
offsets = offsets.to(x.device)
lengths = offsets[1:] - offsets[:-1]
chunk_counts = ((lengths + self.chunk_size - 1) // self.chunk_size).tolist()
total_chunks = sum(chunk_counts)
output = torch.empty(
batch_size, total_chunks, heads, dim, dtype=x.dtype, device=x.device)
chunk_idx = 0
for b in range(batch_size):
for seq_id, chunks_i in enumerate(chunk_counts):
seq_start = offsets[seq_id].item()
seq_end = offsets[seq_id + 1].item()
for local_chunk_id in range(chunks_i):
chunk_start = seq_start + local_chunk_id * self.chunk_size
chunk_end = min(chunk_start + self.chunk_size, seq_end)
output[b, chunk_idx] = x[b, chunk_start:chunk_end].mean(dim=0)
chunk_idx += 1
return output
class MeanPoolingBenchmark(BenchmarkBase[MeanPoolingTest]):
def calculate_flops(self) -> Optional[float]:
t = self.workload
# Mean pooling: sum chunk_size elements + divide, per output element
return t.batch_size * t.chunks_per_bacth * t.heads * t.dim * t.chunk_size
def calculate_memory(self) -> Optional[float]:
t = self.workload
# Read input + write output
input_bytes = t.batch_size * t.seq_len * t.heads * t.dim * t.dtype.itemsize
output_bytes = t.batch_size * t.chunks_per_bacth * t.heads * t.dim * t.dtype.itemsize
return input_bytes + output_bytes
_MEAN_POOLING_BENCH_PARAMS = [
pytest.param(1, 8192, 64, 128, 64, torch.float16, torch.float32, True, None, id="dense-mainstream"),
pytest.param(2, 2048, 64, 128, 64, torch.float16, torch.float32, True, None, id="dense-batched"),
pytest.param(
1, 8192, 64, 128, 64, torch.float16, torch.float32, True,
torch.tensor([0, 2048, 4096, 6144, 8192], dtype=torch.int32, device="cuda"),
id="varlen-long",
),
pytest.param(
1, 1000, 64, 128, 32, torch.float16, torch.float32, True,
torch.tensor([0, 100, 300, 600, 1000], dtype=torch.int32, device="cuda"),
id="varlen-tail",
),
]
@pytest.mark.parametrize(
"batch_size, seq_len, heads, dim, chunk_size, dtype, accum_dtype, tune, offsets",
_MEAN_POOLING_BENCH_PARAMS,
)
def test_mean_pooling_bench(batch_size: int, seq_len: int, heads: int, dim: int, chunk_size: int,
dtype: torch.dtype, accum_dtype: torch.dtype, tune: bool,
offsets: Optional[torch.Tensor]) -> None:
if offsets is not None:
assert batch_size == 1
assert offsets[-1] == seq_len
indices = prepare_chunk_indices(offsets, chunk_size)
chunks_per_bacth = indices.shape[0]
seq_num = offsets.shape[0] - 1
use_offsets = 1
else:
offsets = torch.arange(
0, (batch_size + 1) * seq_len,
seq_len,
dtype=torch.int32,
device='cuda',
requires_grad=False)
chunks_per_bacth = (seq_len + chunk_size - 1) // chunk_size
indices = torch.empty((chunks_per_bacth, 2), dtype=torch.int32, device='cuda')
seq_num = batch_size
use_offsets = 0
params = {
"batch_size": batch_size,
"seq_len": seq_len,
"heads": heads,
"dim": dim,
"chunk_size": chunk_size,
"chunks_per_bacth": chunks_per_bacth,
"seq_num": seq_num,
"use_offsets": use_offsets,
"dtype": dtype,
"accum_dtype": accum_dtype,
"tune": tune,
}
test = _MeanPoolingTestBaseline(
batch_size=batch_size, seq_len=seq_len, heads=heads, dim=dim,
chunk_size=chunk_size, chunks_per_bacth=chunks_per_bacth,
seq_num=seq_num, use_offsets=use_offsets,
dtype=dtype, accum_dtype=accum_dtype,
offsets=offsets, indices=indices)
bm = MeanPoolingBenchmark(test)
inputs = test.gen_inputs()
op = MeanPoolingForwardOp(**params)
result = bm.profile(op, *inputs)
BenchmarkReport.record(op, locals(), result, tag="tileops")
result_bl = bm.profile(test.ref_program, *inputs)
BenchmarkReport.record(op, locals(), result_bl, tag="torch-ref")
if __name__ == "__main__":
pytest.main([__file__, "-vvs"])

Some files were not shown because too many files have changed in this diff Show More