b3003ff157
## Problem Several legacy binary benchmarks pass a short string such as `maximum`, `minimum`, or `cmp_eq` to `BenchmarkReport.record()` instead of passing the Op instance that was actually profiled. String records contain neither the canonical class identity nor `op_module`. The nightly report therefore treats a legacy alias and its manifest-driven benchmark as different operators. For example, the same `MaximumFwdOp` implementation can appear once as `MaximumFwdOp` and again as `maximum`. Downstream documentation then has to classify the alias from its name alone, which can also place `maximum` under Reduction because it contains `max`. This affects the standard binary arithmetic, comparison, logical, and bitwise benchmark groups. The measurements themselves are valid; the recorded operator identity is not canonical. ## Solution Pass the real Op instance to `BenchmarkReport.record()` for both the TileOPs measurement and its baseline: ```python op = op_cls(...) result = bm.profile(op, *inputs) BenchmarkReport.record(op, locals(), result, tag='tileops') result_bl = bm.profile(baseline_fn, *inputs) BenchmarkReport.record(op, locals(), result_bl, tag='torch') ``` `BenchmarkReport` then derives the identity consistently from: ```python op.__class__.__name__ op.__class__.__module__ ``` This consolidates legacy aliases under canonical identities such as `MaximumFwdOp`, `MinimumFwdOp`, and `EqFwdOp`, while preserving the benchmark parameters, measurements, and baseline tags. The change is intentionally limited to identity recording. It does not add display-category metadata or move taxonomy into TileOPs; benchmark page classification remains owned by TileOPs.github.io. ## Changes - record standard binary arithmetic benchmarks with their real Op instances - record comparison benchmarks with their real Op instances instead of `cmp_*` aliases - record logical and bitwise benchmarks with their real Op instances - preserve canonical class names and `op_module` metadata in nightly JUnit output ## Validation - `python -m py_compile benchmarks/ops/bench_binary_elementwise.py` - `git diff --check` GPU benchmark execution was not available in the local worktree. Related documentation taxonomy fix: tile-ai/TileOPs.github.io#13 |
||
|---|---|---|
| .claude | ||
| .foundry | ||
| .github | ||
| assets | ||
| benchmarks | ||
| docs | ||
| scripts | ||
| tests | ||
| tileops | ||
| workloads | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| CLAUDE.md | ||
| LICENSE | ||
| Makefile | ||
| README.md | ||
| THIRD_PARTY_NOTICES.md | ||
| constraints.txt | ||
| pyproject.toml | ||
README.md
TileOPs
Spec-driven GPU operator library for LLMs — designed for AI agents to build, evaluate, and optimize
Built on TileLang
Status: TileOPs is under active development. APIs may change.
Overview
TileOPs is a GPU operator library for LLM training and inference, built on TileLang. Beyond providing a growing collection of production-quality operators, TileOPs explores a spec-driven development model where AI agents can read declarative operator specifications, generate kernel implementations, and evaluate them against hardware-theoretical performance bounds — with minimal human scaffolding.
Architecture
Every operator is split into two layers with a strict boundary:
- Op (L2) — stateless Python entry point. Handles validation, dtype casting, and memory layout. CUDA-Graph compatible;
torch.compile(fullgraph=True)support is declared per op in the manifest. - Kernel (L1) — TileLang GPU implementation with hardware-specific optimizations (Hopper).
This separation keeps user-facing behavior independent of GPU strategy, allowing agents and developers to modify either layer without side effects on the other.
Key Properties
- Spec-driven — each operator is declared in a machine-readable manifest (
tileops/manifest/) that specifies signatures, workloads, and roofline formulas, serving as the entry point for both agent code generation and automated validation - Roofline-evaluated — kernel performance is measured against Speed-of-Light hardware bounds, not relative baselines
- Auto-tuning — built-in search over tile sizes, pipelines, and scheduling parameters
- Lightweight — depends only on TileLang, PyTorch, and einops
Installation
TileOPs is under active development and is installed from source; PyPI releases will begin with the first stable release. A CUDA-capable GPU is required.
Prerequisites
- Python >= 3.10
- PyTorch >= 2.1, < 2.11 (CI validates 2.10)
- CUDA Toolkit 12.x
- NVIDIA GPU: Hopper (SM_90)
- TileLang >= 0.1.9, < 0.2.0 (CI validates 0.1.11)
From source
git clone https://github.com/tile-ai/TileOPs
cd TileOPs
make install # dev dependencies + pre-commit hooks
[!NOTE] If CUDA and TileLang are already installed system-wide and you encounter build issues:
PIP_NO_BUILD_ISOLATION=1 pip install -e '.[dev]' -v && pre-commit install
Verify:
python -m pytest tests/ -q # requires a CUDA GPU
Quick Start
import torch
from tileops.ops import GemmOp
M, N, K = 1024, 1024, 512
dtype = torch.float16
gemm = GemmOp() # shapes and dtype are inferred at call time
a = torch.randn(M, K, device="cuda", dtype=dtype)
b = torch.randn(N, K, device="cuda", dtype=dtype) # trans_b=True by default
d = gemm(a, b) # equals a @ b.T
Documentation
Design docs and development guides are in docs/. The full API reference and performance tables are published at TileOPs.github.io.
Contributing
See docs/ for design docs. Branch and commit conventions are in .claude/conventions/types.sh.
License
TileOPs is released under the MIT License.