Compare commits
2 Commits
master
...
summary-sc
| Author | SHA1 | Date |
|---|---|---|
MaseChen
|
2710546a66 | |
|
MaseChen
|
2e3611651b |
329
FAQ.md
329
FAQ.md
|
|
@ -1,329 +0,0 @@
|
|||
# 常见问题 FAQ
|
||||
|
||||
> 最后整理：2026-07-13
|
||||
|
||||
本文档汇总沐曦“揭榜挂帅”两项赛题的常见问题。使用 `Ctrl+F`（Windows/Linux）或 `⌘F`（macOS）搜索关键词。
|
||||
|
||||
环境版本、评测配置、时间安排可能调整。请以仓库 README、XPU-OJ 公告、比赛群通知和容器中的实际版本为准。发现内容过期或未覆盖的问题，请[提交 Issue](https://gitlink.org.cn/metax-maca/op_optimization/issues)。
|
||||
|
||||
## 快速导航
|
||||
|
||||
- [重要入口与咨询方式](#entry)：XPU-OJ 开放情况、问题反馈渠道和赛事联系人。
|
||||
- [XPU-OJ、提交与排行榜](#xpuoj)：账号申请、提交环境、测试样例、评测硬件和排名指标。
|
||||
- [赛题一：TileLang 与 Fused MoE](#track-one)：提交限制、算子实现、Baseline、性能优化和决赛加分规则。
|
||||
- [赛题二：AI Agent 与推理算子库](#track-two)：任务范围、Agent 参与证明、Baseline、测试参数和上游代码引用。
|
||||
- [评测规则与通用技术问题](#evaluation)：技术资料发布、正确性与稳定性要求，以及 FAQ 内容纠错。
|
||||
- [环境、镜像与算力资源](#environment)：比赛镜像、MACA 与 PyTorch 版本、开发工具、算力券和资源申请。
|
||||
- [报名、组队与资格审核](#registration)：参赛资格、跨校组队、指导教师、材料盖章和审核流程。
|
||||
|
||||
<a id="entry"></a>
|
||||
|
||||
## 重要入口与咨询方式
|
||||
|
||||
<a id="q-xpuoj-open"></a>
|
||||
|
||||
### ❓ 问题 1：XPU-OJ 平台是否已经上线？
|
||||
|
||||
**回答：** XPU-OJ 已开放。账号申领流程和使用指南见[赛事 XPU-OJ 账号申领说明](赛事XPUOJ账号申领说明.md)。
|
||||
|
||||
<a id="q-contact"></a>
|
||||
|
||||
### ❓ 问题 2：两个赛题的联系人不同，遇到问题应该联系谁？
|
||||
|
||||
**回答：** 请先通过 [GitLink Issue](https://gitlink.org.cn/metax-maca/op_optimization/issues) 提问，维护者会把可复用的答案更新到本文档。比赛群用于接收赛事通知和临时信息。需要单独沟通时，请按对应比赛方案联系章老师或杨老师。
|
||||
|
||||
## XPU-OJ、提交与排行榜
|
||||
|
||||
<a id="q-moe-score-decrease"></a>
|
||||
|
||||
### ❓ 问题 1：XPUOJ测评 MoE 耗时减少了但是分数反而降低了
|
||||
|
||||
**回答：**
|
||||
|
||||
针对近期部分同学反馈的“XPU.OJ”第三方评测系统中基线（baseline）不稳定的问题，我们高度重视，并已第一时间组织排查与测试。在此，我们对因此给大家带来的困扰深表歉意，也衷心感谢各位同学提出的宝贵意见。
|
||||
目前，相关问题已修复完毕。为确保评测的公平性与准确性，我们将对现有榜单进行清空处理。历史提交记录仍可查看，但后续排名将统一以基线修复后重新提交的算子成绩为准。
|
||||
|
||||
比赛期间，我们将持续关注系统运行状态，也欢迎大家继续向我们反馈建议。
|
||||
祝大家比赛顺利，取得理想成绩！
|
||||
|
||||
<a id="q-xpuoj-environment"></a>
|
||||
|
||||
### ❓ 问题 2：XPU-OJ 与模力方舟的运行环境一致吗？
|
||||
|
||||
**回答：** 排行榜评测环境与模力方舟开发环境保持一致。版本调整时以 XPU-OJ 公告为准。
|
||||
|
||||
<a id="q-xpuoj-account"></a>
|
||||
|
||||
### ❓ 问题 3：如何申请 XPU-OJ 账号？
|
||||
|
||||
**回答：** 请按照[赛事 XPU-OJ 账号申领说明](赛事XPUOJ账号申领说明.md)提交申请。账号发放进度以赛事通知和回复邮件为准。
|
||||
|
||||
<a id="q-official-ranking"></a>
|
||||
|
||||
### ❓ 问题 4：赛题一 MoE 初赛排名以哪个入口为准？
|
||||
|
||||
**回答：** 正式排名和初筛结果以 XPU-OJ 的评测结果为准。Sample benchmark 用于本地功能验证、调试和性能对比，不作为正式榜单依据。
|
||||
|
||||
<a id="q-test-cases"></a>
|
||||
|
||||
### ❓ 问题 5：MACA C++、Triton 和 TileLang 是否分别设榜？
|
||||
|
||||
**回答：** 不按语言分别设榜。每个任务支持 MACA C++、Triton 和 TileLang，团队可以使用一种或多种语言提交。排行榜采用通过正确性和稳定性测试后的最高成绩。
|
||||
|
||||
<a id="q-ranking-metric"></a>
|
||||
|
||||
### ❓ 问题 6：排行榜使用 latency、speedup 还是综合 score？
|
||||
|
||||
**回答：** 当前 XPU-OJ 以 speedup 作为核心排名指标。赛事方调整计算方式时，以 XPU-OJ 公告为准。
|
||||
|
||||
<a id="track-one"></a>
|
||||
|
||||
## 赛题一：TileLang 与 Fused MoE
|
||||
|
||||
<a id="q-track-one-submission"></a>
|
||||
|
||||
### ❓ 问题 1：赛题一的提交要求有哪些调整？
|
||||
|
||||
**回答：** 正式提交禁止使用 `MACA Maca running` 方式，也不能使用 PyTorch 实现算子。参赛者需使用 TileLang 实现并提交。
|
||||
|
||||
<a id="q-ops-reference"></a>
|
||||
|
||||
### ❓ 问题 2：OPS 目录中的 TileLang、CUDA、CUTLASS 和 MACA 代码有什么用途？
|
||||
|
||||
**回答：** 这些代码用于解释算子的实现原理和设计思路，可作为 TileLang 实现的参考。
|
||||
|
||||
<a id="q-baseline-modification"></a>
|
||||
|
||||
### ❓ 问题 3：官方 Baseline 可以修改到什么范围？
|
||||
|
||||
**回答：** 参赛者可以重新设计和优化算子实现，但需保持与统一 Workload 测试框架的接口兼容。
|
||||
|
||||
<a id="q-gemm-optimization"></a>
|
||||
|
||||
### ❓ 问题 4：GEMM 计算中可以引入其他优化策略吗？
|
||||
|
||||
**回答：** 可以，前提是实现符合赛题规则和评测要求。
|
||||
|
||||
<a id="q-benchmark-modification"></a>
|
||||
|
||||
### ❓ 问题 5：可以优化 `fusedmoe_benchmark` 吗？
|
||||
|
||||
**回答：** 本地修改 benchmark 不会提高正式成绩，XPU-OJ 使用赛事方的评测框架。参赛者应把优化工作放在规定的算子实现和允许修改的接口上。
|
||||
|
||||
<a id="q-forward-modification"></a>
|
||||
|
||||
### ❓ 问题 6：可以修改 `fusedmoe_benchmark.py` 中的 MoE forward 吗？
|
||||
|
||||
**回答：** 正式成绩以 XPU-OJ 的独立评测为准。本地修改 forward 不能替代对提交算子的优化，也不会改变赛事方的评测代码。
|
||||
|
||||
<a id="q-async-copy"></a>
|
||||
|
||||
### ❓ 问题 7：赛题一允许使用异步拷贝吗？
|
||||
|
||||
**回答：** 不允许。当前比赛规则禁用异步拷贝。
|
||||
|
||||
<a id="q-baseline-performance"></a>
|
||||
|
||||
### ❓ 问题 8：Fused MoE 初赛成绩如何影响决赛？
|
||||
|
||||
**回答：** 当前规则只给初赛前 10 名决赛加分，第 11 名及之后不获得额外初赛加分。
|
||||
|
||||
<a id="track-two"></a>
|
||||
|
||||
## 赛题二：AI Agent 与推理算子库
|
||||
|
||||
<a id="q-track-two-update"></a>
|
||||
|
||||
### ❓ 问题 1：赛题二的内容有什么调整？
|
||||
|
||||
**回答：** “Agent 推理算子库优化 - FlashAttention KV Cache Decode”新增 `mctlass/cute` 要求。参赛者需基于 `mctlass/cute` 实现或优化对应任务。
|
||||
|
||||
<a id="q-track-two-selection"></a>
|
||||
|
||||
### ❓ 问题 2：赛题二可以选择几个任务？
|
||||
|
||||
**回答：** 参赛团队可以从 FlashInfer、FlashAttention、MCTLASS/Fused MoE 等方向选择一项或多项。提交多个任务时，每个任务按赛事规则取有效最高成绩。支持语言包括 Triton、MXMACA C++ 和 TileLang。
|
||||
|
||||
<a id="q-agent-proof"></a>
|
||||
|
||||
### ❓ 问题 3：如何证明 AI Agent 参与了优化过程？
|
||||
|
||||
**回答：** 参赛团队应保留 Agent 配置、Skill 文件、关键提示词、操作日志、代码变更记录、测试结果和复现实验步骤，等待赛事方发布核验细则。
|
||||
|
||||
<a id="q-agent-baseline"></a>
|
||||
|
||||
### ❓ 问题 4：Agent 赛题的性能 baseline 使用哪个版本？
|
||||
|
||||
**回答：** 当前评测使用赛事方提供的 baseline，标准环境为 `PyTorch-Agent / 2.8.0 / Python 3.12 / MACA 3.7.1.5`。赛事方变更 baseline 或评测方式时会发布通知。
|
||||
|
||||
<a id="q-mla-dimensions"></a>
|
||||
|
||||
### ❓ 问题 5：MLA 的 `QK dim = 576, VO dim = 512` 与 `race_tests` 参数冲突吗？
|
||||
|
||||
**回答：** 不冲突。`race_tests` 中的 `dim=512, pe_dim=64` 对应 `QK dim = 576, V dim = 512`。
|
||||
|
||||
<a id="q-nsa-ranking"></a>
|
||||
|
||||
### ❓ 问题 6：NSA 的 109 个测试 case 如何计算榜单成绩？
|
||||
|
||||
**回答：** XPU-OJ 通过统一接口统计测试总运行时间，再根据 baseline 计算整体 speedup。
|
||||
|
||||
<a id="q-upstream-code"></a>
|
||||
|
||||
### ❓ 问题 7：可以引用或修改 FlashInfer、FlashAttention 等上游代码吗？
|
||||
|
||||
**回答：** 可以。参赛团队需遵守上游项目许可证，保留版权和许可证声明，并在提交材料中说明引用范围、迁移工作和自主优化内容。
|
||||
|
||||
<a id="evaluation"></a>
|
||||
|
||||
## 环境、镜像与算力资源
|
||||
|
||||
<a id="q-environment-image"></a>
|
||||
|
||||
### ❓ 问题 1：比赛使用哪个在线算力环境？
|
||||
|
||||
**回答：** 比赛使用模力方舟沐曦算力专区。仓库 README 当前标注的统一镜像为：
|
||||
|
||||
```text
|
||||
PyTorch-Agent / 2.8.0 / Python 3.12 / MACA 3.7.1.5
|
||||
```
|
||||
|
||||
创建实例和连接环境的步骤见[模力方舟快速使用 SOP](模力方舟快速使用SOP.md)。
|
||||
|
||||
<a id="q-download-maca"></a>
|
||||
|
||||
### ❓ 问题 2：如何获取 MACA 镜像或安装包？
|
||||
|
||||
**回答：** 参赛者可以在[模力方舟沐曦算力专区](https://ai.gitee.com/compute/metax)选择 `PyTorch-Agent` 镜像。需要单独获取软件包时，请前往[沐曦开发者社区软件中心](https://developer.metax-tech.com/softnova/docker?chip_name=%E6%9B%A6%E4%BA%91C500%E7%B3%BB%E5%88%97&package_kind=AI&dimension=docker)，并选择与比赛标准环境一致的 MACA `3.7.1.5` 版本。
|
||||
|
||||
<a id="q-download-pytorch"></a>
|
||||
|
||||
### ❓ 问题 3：比赛使用的 PyTorch 镜像可以下载吗？
|
||||
|
||||
**回答：** 可以。请在[沐曦开发者社区](https://developer.metax-tech.com/)或其 [PyTorch 镜像列表](https://developer.metax-tech.com/softnova/docker?chip_name=%E6%9B%A6%E4%BA%91C500%E7%B3%BB%E5%88%97&package_kind=AI&dimension=docker&deliver_type=%E5%88%86%E5%B1%82%E5%8C%85&ai_frame=pytorch)中查询。下载前请核对比赛镜像的 Python、PyTorch 和 MACA 版本。
|
||||
|
||||
<a id="q-version-mismatch"></a>
|
||||
|
||||
### ❓ 问题 4：页面标注的 MACA 版本与容器内版本不一致怎么办？
|
||||
|
||||
**回答：** 比赛标准版本为 MACA `3.7.1.5`。
|
||||
|
||||
<a id="q-pytorch-source"></a>
|
||||
|
||||
### ❓ 问题 5：Linux 版本的 mcprofiler 是否可用？
|
||||
|
||||
**回答：** mcprofiler 的 Linux 版本已经打包进模力方舟上的 pytorch-agent 比赛镜像。
|
||||
|
||||
<a id="q-compute-coupons"></a>
|
||||
|
||||
### ❓ 问题 6：算力券按团队还是按个人领取？
|
||||
|
||||
**回答：** 新人礼和启悟社区算力券按学生个人发放，符合条件的团队成员均可领取。团队主申请人的额度用完后，其他成员可以继续申请资源。
|
||||
|
||||
- [沐曦开发者社区新人礼](https://developer.metax-tech.com/activities/6)
|
||||
- [启悟社区学生算力券](https://developer.metax-tech.com/activities/11)
|
||||
- [赛事算力券活动](https://developer.metax-tech.com/activities/17)
|
||||
|
||||
<a id="q-more-compute"></a>
|
||||
|
||||
### ❓ 问题 7：算力额度不足时可以追加申请吗？
|
||||
|
||||
**回答：** 可以先领取上述活动中的算力券。仍需额外资源时，请发送需求邮件至 `opensource@metax-tech.com`。
|
||||
|
||||
<a id="q-commercial-agent-cost"></a>
|
||||
|
||||
<a id="registration"></a>
|
||||
|
||||
## 报名、组队与资格审核
|
||||
|
||||
<a id="q-register-both"></a>
|
||||
|
||||
### ❓ 问题 1：同一团队或个人可以同时参加两个赛题吗？
|
||||
|
||||
**回答：** 可以。同一团队或个人可以同时报名两个赛题。
|
||||
|
||||
<a id="q-register-multiple-tracks"></a>
|
||||
|
||||
### ❓ 问题 2：同一名学生可以报名不同赛道的不同赛题吗？
|
||||
|
||||
**回答：** 可以。赛事不统一限制学生报名不同赛道或不同赛题，但同一作品不得用相同核心技术内容重复申报不同赛题。
|
||||
|
||||
<a id="q-new-graduate"></a>
|
||||
|
||||
### ❓ 问题 3：本科应届毕业、尚未正式入学的研一新生可以报名吗？
|
||||
|
||||
**回答：** 可以。参赛者可联系原本科学校完成认证手续，并以本科生身份报名。
|
||||
|
||||
<a id="q-advisor-required"></a>
|
||||
|
||||
### ❓ 问题 4：参赛必须配备指导教师吗？
|
||||
|
||||
**回答：** 不强制。填写指导教师时，每支队伍可以设置 1 至 3 名指导教师。
|
||||
|
||||
<a id="q-advisor-team-limit"></a>
|
||||
|
||||
### ❓ 问题 5：一名指导教师最多可以指导几支队伍？
|
||||
|
||||
**回答：** 赛事暂未设置统一的硬性上限。指导教师应根据可投入的时间控制队伍数量。
|
||||
|
||||
<a id="q-cross-school-stamp"></a>
|
||||
|
||||
### ❓ 问题 6：跨校组队时，报名表应该由哪所学校盖章？
|
||||
|
||||
**回答：** 资格审批阶段，每名参赛者需到本人学校的校团委或院团委完成盖章确认。后续材料由团队牵头学生统一整理和提交。
|
||||
|
||||
<a id="q-upload-stamped-form"></a>
|
||||
|
||||
### ❓ 问题 7：提交报名后还可以补充已盖章的报名表扫描件吗？
|
||||
|
||||
**回答：** 审核人员发现材料缺少盖章时，会退回申请。团队补齐材料后可以重新提交。尚未完成盖章的团队应先与学院、校团委或学校相关部门确认办理方式。
|
||||
|
||||
<a id="q-stamp-department"></a>
|
||||
|
||||
### ❓ 问题 8：资格审查材料应该加盖哪个部门的公章？
|
||||
|
||||
**回答：** 各高校的管理口径不同。教务处、学生处等学籍或学生管理部门通常可以办理，参赛团队应以本校校团委或相关管理部门的要求为准。
|
||||
|
||||
<a id="q-no-youth-league"></a>
|
||||
|
||||
### ❓ 问题 9：学校未设校团委，可以用院系公章替代吗？
|
||||
|
||||
**回答：** 赛事原则上要求校级部门公章。学校未设校团委时，可以联系校级学工、双创或教务部门盖章，并提交情况说明。院系公章不能直接替代校级部门公章。
|
||||
|
||||
<a id="q-student-status-proof"></a>
|
||||
|
||||
### ❓ 问题 10：学校无法配合盖章，可以用学籍证明替代吗？
|
||||
|
||||
**回答：** 不可以。参赛团队应使用报名系统导出的报名表，并按要求完成学校盖章。
|
||||
|
||||
<a id="q-public-notice"></a>
|
||||
|
||||
### ❓ 问题 11：公示材料需要包含哪些内容？
|
||||
|
||||
**回答：** 请参考赛事工作群发布的参考文本，并按学校要求调整。跨校团队涉及的学校应分别在学校官网公示，公示渠道原则上使用学校官网。
|
||||
|
||||
<a id="q-review-deadline"></a>
|
||||
|
||||
### ❓ 问题 12：报名审核需要在报名截止日前完成吗？
|
||||
|
||||
**回答：** 原则上需要。往届出现过系统延后关闭的情况，但本届参赛团队不应据此推迟材料提交或审核。
|
||||
|
||||
<a id="q-review-flow"></a>
|
||||
|
||||
### ❓ 问题 13：报名材料的审核顺序是什么？
|
||||
|
||||
**回答：** 学生提交材料后，学校校团委先审核；学校审核通过后，企业再审核。企业审核通过即视为报名成功。
|
||||
|
||||
<a id="q-school-review-account"></a>
|
||||
|
||||
### ❓ 问题 14：后台显示“校团委审核”，但学校不了解审核事项，怎么办？
|
||||
|
||||
**回答：** 校团委需在报名系统内完成审核。省级团委通常会向各高校团委发放账号和密码。学校未收到或不了解安排时，请学校联系省级团委确认。
|
||||
|
||||
## FAQ 维护约定
|
||||
|
||||
1. 参赛者通过 Issue 提交问题。
|
||||
2. 维护者确认答案后更新本文档。
|
||||
3. 每个答案保留稳定锚点；需要时附上来源 Issue 或公告。
|
||||
4. 维护者在原 Issue 中回复 FAQ 锚点链接，并关闭已经解决的问题。
|
||||
5. 涉及版本、日期、评测参数的答案应标注确认日期。
|
||||
19
README.md
19
README.md
|
|
@ -1,22 +1,5 @@
|
|||
# 降低Token 成本，攻坚国产推理生态｜沐曦两大赛题登陆 2026 揭榜挂帅擂台赛，邀青年共破局！
|
||||
|
||||
## 常用入口
|
||||
|
||||
- [常见问题 FAQ](FAQ.md)
|
||||
- [沐曦通用GPU MXMACA编译器内建函数编程指南](https://developer.metax-tech.com/api/client/document/preview/1395/index.html)
|
||||
|
||||
## XPU-OJ 基线修复及榜单调整通知
|
||||
|
||||
针对近期部分同学反馈的“XPU.OJ”第三方评测系统中基线（baseline）不稳定的问题，我们高度重视，并已第一时间组织排查与测试。在此，我们对因此给大家带来的困扰深表歉意，也衷心感谢各位同学提出的宝贵意见。
|
||||
目前，相关问题已修复完毕。为确保评测的公平性与准确性，我们将对现有榜单进行清空处理。历史提交记录仍可查看，但后续排名将统一以基线修复后重新提交的算子成绩为准。
|
||||
|
||||
比赛期间，我们将持续关注系统运行状态，也欢迎大家继续向我们反馈建议。
|
||||
祝大家比赛顺利，取得理想成绩！
|
||||
|
||||
- XPU-OJ 地址：[https://xpuoj.com/](https://xpuoj.com/)
|
||||
- 账号申领说明：[赛事 XPU-OJ 账号申领说明](赛事XPUOJ账号申领说明.md)
|
||||
- 账号申领邮箱：`opensource@metax-tech.com`
|
||||
|
||||
2026 年度中国青年科技创新「揭榜挂帅」擂台赛正式启幕。沐曦股份重磅发布两大 AI 算力硬核榜题，聚焦国产 GPU 大模型推理算子优化，以硬核赛事搭建科研攻关平台，邀全国青年学子、科研人才揭榜攻坚，用技术重构推理效率，用创新拉低每 Token 算力成本！
|
||||
|
||||
## 两大重磅赛题 直击推理成本核心痛点
|
||||
|
|
@ -46,7 +29,7 @@
|
|||
|
||||
### 赛题二：基于 AI Agent 开发范式的国产 GPU 大模型推理算子库优化
|
||||
|
||||
大模型推理具有高并发、长序列、高<EFBFBD><EFBFBD><EFBFBD>用频次等特点，FlashInfer、FlashAttention、Fused MoE 等核心算子直接决定模型服务的吞吐、延迟与显存开销，影响单 Token 综合推理成本。
|
||||
大模型推理具有高并发、长序列、高调用频次等特点，FlashInfer、FlashAttention、Fused MoE 等核心算子直接决定模型服务的吞吐、延迟与显存开销，影响单 Token 综合推理成本。
|
||||
|
||||
本赛题面向沐曦国产 GPU 及 MXMACA 软件栈，鼓励参赛团队构建或使用 AI Agent / Skill 工作流，围绕推理算子库开展代码理解、算子迁移、性能分析、Kernel 优化、自动调优、Benchmark 验证和多轮迭代，探索“Agent 驱动算子优化”的新型开发范式。
|
||||
|
||||
|
|
|
|||
|
|
@ -366,7 +366,7 @@ pip install pandas
|
|||
|
||||
```bash
|
||||
# 检查脚本文件
|
||||
python -c "import os; scripts = ['bench_common.py', 'bench_batch_decode.py', 'bench_batch_prefill_paged.py', 'bench_batch_prefill_ragged.py', 'bench_batch_mla.py']; [print(f'✓ {s}') if os.path.exists(s) else print(f'✗ {s} missing') for s in scripts]"
|
||||
python -c "import os; scripts = ['bench_common.py', 'benchmark_result.py', 'summarize_results.py', 'bench_batch_decode.py', 'bench_batch_prefill_paged.py', 'bench_batch_prefill_ragged.py', 'bench_batch_mla.py']; [print(f'✓ {s}') if os.path.exists(s) else print(f'✗ {s} missing') for s in scripts]"
|
||||
|
||||
# 测试脚本导入
|
||||
python -c "from bench_common import setup_workspace, get_csv_path; print('脚本导入正常')"
|
||||
|
|
@ -376,6 +376,8 @@ python -c "from bench_common import setup_workspace, get_csv_path; print('脚本
|
|||
|
||||
```plaintext
|
||||
✓ bench_common.py
|
||||
✓ benchmark_result.py
|
||||
✓ summarize_results.py
|
||||
✓ bench_batch_decode.py
|
||||
✓ bench_batch_prefill_paged.py
|
||||
✓ bench_batch_prefill_ragged.py
|
||||
|
|
@ -414,6 +416,57 @@ Results saved to BatchPrefillWithRaggedKVCacheWrapper_20260626_xxxxxx.csv
|
|||
| `out of memory` | 减小 `batch_size` 或 `seq_len` 参数 |
|
||||
| 运行时间过长 | 脚本会自动调整重复次数，耐心等待 |
|
||||
|
||||
Benchmark 会逐个执行 case。单个 case 出错时，脚本会把 `status=failed`、异常类型和错误信息写入 CSV 后继续执行；只要存在失败 case，CSV 仍会保存，但进程最终返回退出码 `1`。
|
||||
|
||||
***
|
||||
|
||||
**结构化 CSV 与汇总报告：**
|
||||
|
||||
新版 CSV 使用 `schema_version=1`，通用字段如下：
|
||||
|
||||
| 字段 | 含义 |
|
||||
| --- | --- |
|
||||
| `schema_version` | 结果格式版本，当前为 `1` |
|
||||
| `api` | FlashInfer wrapper 名称 |
|
||||
| case 参数列 | 如 `batch_size`、`seq_len`、`head_dim` 等 |
|
||||
| `status` | `ok` 或 `failed` |
|
||||
| `time_ms` / `bandwidth_GB_s` / `tflops` | 成功 case 的性能指标 |
|
||||
| `error_type` / `error` | 失败 case 的异常类型和错误信息 |
|
||||
|
||||
批量汇总一个或多个结果文件：
|
||||
|
||||
```bash
|
||||
python summarize_results.py results/*.csv \
|
||||
--output flashinfer_benchmark_summary.md
|
||||
```
|
||||
|
||||
显式比较优化前后的同一组 case：
|
||||
|
||||
```bash
|
||||
python summarize_results.py \
|
||||
--baseline baseline/*.csv \
|
||||
--candidate candidate/*.csv \
|
||||
--output flashinfer_benchmark_regression.md
|
||||
```
|
||||
|
||||
默认比较模式只报告变化。CI 中可增加门禁，例如任一匹配 case 的耗时上升或带宽、TFLOPS 下降超过 5% 时返回退出码 `1`：
|
||||
|
||||
```bash
|
||||
python summarize_results.py \
|
||||
--baseline baseline/*.csv \
|
||||
--candidate candidate/*.csv \
|
||||
--fail-on-regression 5 \
|
||||
--output flashinfer_benchmark_regression.md
|
||||
```
|
||||
|
||||
| 退出码 | 含义 |
|
||||
| --- | --- |
|
||||
| `0` | 输入有效，且没有 benchmark/candidate 失败或触发回归门禁 |
|
||||
| `1` | 存在失败 case、候选缺失基线 case，或触发回归门禁 |
|
||||
| `2` | 参数、CSV 文件或 schema 无效 |
|
||||
|
||||
旧版 CSV 没有 `schema_version/status/error` 字段时，只有在表头符合已知 FlashInfer benchmark 格式且性能指标有效的情况下才会被兼容读取；报告会将其标记为 `legacy_inferred_ok`，表示成功状态来自格式推断而非原始记录。
|
||||
|
||||
***
|
||||
|
||||
**查看结果命令示例：**
|
||||
|
|
@ -984,7 +1037,7 @@ FlashInfer 方向包含 **4 个可选算子题目**，均属于同一比赛通
|
|||
| --- | --- | --- |
|
||||
| Benchmark 运行时间过长 | 参数组合过多， workload 较大 | 耐心等待，脚本会自动调整重复次数 |
|
||||
| `KeyError: 'BatchPrefillWithPagedKVCacheKernel'` | profiler 未捕获目标 kernel | 检查 `target_kernels` 配置是否正确 |
|
||||
| CSV 文件为空 | 测试未正常完成 | 检查 GPU 显存是否充足，重新运行 |
|
||||
| CSV 文件为空或汇总器返回退出码 `2` | 测试未正常完成或 CSV schema 无效 | 查看汇总报告的 `Input Errors`，检查文件完整性后重新运行 |
|
||||
|
||||
### 8.4 代码问题
|
||||
|
||||
|
|
@ -1450,4 +1503,4 @@ extern "C" void run_kernel(
|
|||
}
|
||||
```
|
||||
|
||||
[*回退到 Step 8*](#step%208提交%20oj%20冒烟代码)
|
||||
[*回退到 Step 8*](#step%208提交%20oj%20冒烟代码)
|
||||
|
|
|
|||
|
|
@ -404,7 +404,7 @@ bash scripts/run_fused_moe_i8_tn_benchmark.sh --backend all --warmup 5 --iters 2
|
|||
|
||||
#### Step 7：登录 XPU-OJ 平台并进入题目页面
|
||||
|
||||
**目标：**访问 XPU-OJ 评测平台，登录并进入 Fused MoE 算子对应任务页面，熟悉页面布局。
|
||||
**目标：**访问 XPU-OJ 评测平台，找到 `fused_moe_i8_tn`算子题目，熟悉题目页面布局。
|
||||
|
||||
**操作：**
|
||||
|
||||
|
|
@ -412,13 +412,13 @@ bash scripts/run_fused_moe_i8_tn_benchmark.sh --backend all --warmup 5 --iters 2
|
|||
|
||||
[](https://www.picgo.net/image/image1.4tmFfp)
|
||||
|
||||
2. 进入 Fused MoE 任务页面：点击顶部导航栏【比赛】，选择【进行中】，找到 Fused MoE 任务点击右侧【进入】按钮。
|
||||
2. 进入比赛页面：点击顶部导航栏【比赛】，选择【进行中】，找到对应比赛进入。
|
||||
|
||||
[](https://www.picgo.net/image/image2.4i0rxb)
|
||||
[](https://www.picgo.net/image/image2.4tmnl6)
|
||||
|
||||
Fused MoE 任务包含 `Fused MoE i8 tn` 一个题目，直接点击即可进入题目页面：
|
||||
3. 进入题目页面：本算子对应比赛题目6：`Fused MoE i8 tn`，点击进入题目页面。
|
||||
|
||||
[](https://www.picgo.net/image/image3.4i0cXl)
|
||||
[](https://www.picgo.net/image/image3.4tmZJu)
|
||||
|
||||
完成上述步骤可进入如下题目页面：
|
||||
|
||||
|
|
@ -427,7 +427,7 @@ bash scripts/run_fused_moe_i8_tn_benchmark.sh --backend all --warmup 5 --iters 2
|
|||
* 右侧：提交区域，输入编写的`run_kernel(...)`后在下方选择对应的语言即可提交。提交后可通过上方导航栏【我的提交】查看历史提交。
|
||||
|
||||
|
||||
[](https://www.picgo.net/image/image4.4i0eCw)
|
||||
[](https://www.picgo.net/image/image4.4tmM4N)
|
||||
|
||||
#### Step 8：理解 XPU-OJ 评测接口和精度要求
|
||||
|
||||
|
|
@ -457,7 +457,7 @@ bash scripts/run_fused_moe_i8_tn_benchmark.sh --backend all --warmup 5 --iters 2
|
|||
|
||||
* 容差：`rtol=2e-2, atol=5e-3`；
|
||||
|
||||
* 通过率：`matched_ratio ≫ 0.99`（至少 99% 的元素在容差范围内）。
|
||||
* 通过率：`matched_ratio >= 0.99`（至少 99% 的元素在容差范围内）。
|
||||
|
||||
|
||||
#### Step 9：提交 OJ 冒烟代码
|
||||
|
|
@ -475,7 +475,7 @@ bash scripts/run_fused_moe_i8_tn_benchmark.sh --backend all --warmup 5 --iters 2
|
|||
4. 等待结果：评测时间与题目测试点数量、队列状态和平台负载有关，通常需要等待数十秒到数分钟。以平台实际返回为准。
|
||||
|
||||
|
||||
[](https://www.picgo.net/image/image5.4i15ww)
|
||||
[](https://www.picgo.net/image/image5.4tmuBi)
|
||||
|
||||
**预期结果：**
|
||||
|
||||
|
|
@ -497,7 +497,7 @@ bash scripts/run_fused_moe_i8_tn_benchmark.sh --backend all --warmup 5 --iters 2
|
|||
* 在提交记录中可查看每次提交的状态、总得分、耗时、内存：
|
||||
|
||||
|
||||
[](https://www.picgo.net/image/image6.4irF5q)
|
||||
[](https://www.picgo.net/image/image6.4tm6n2)
|
||||
|
||||
* 此页面下滑还可查看单测试点检查器信息（SPJ Report）：
|
||||
|
||||
|
|
@ -553,18 +553,18 @@ bash scripts/run_fused_moe_i8_tn_benchmark.sh --backend all --warmup 5 --iters 2
|
|||
|
||||
1. 查看榜单
|
||||
|
||||
[](https://www.picgo.net/image/image8.4i1Bzd)
|
||||
[](https://www.picgo.net/image/image8.4tm3rO)
|
||||
|
||||
点击【排行榜】进入榜单页面，可查看：
|
||||
|
||||
* 总得分：各题目得分总和，本任务只包含一个题目，因此该题得分即为总分。
|
||||
* 总得分：各题目得分总和，决定最终排名。
|
||||
|
||||
* 个人排名及其它选手的排名：根据总分进行排名，方便参赛者纵向对比。
|
||||
* 个人排名：页面顶部显示【我的排名】与【我的总分】。
|
||||
|
||||
* 每题得分：表格中每列对应一个题目，方便参赛者横向对比。
|
||||
* 每题得分：表格中每列对应一个题目，便于横向对比。
|
||||
|
||||
|
||||
[](https://www.picgo.net/image/image9.4i4SOw)
|
||||
[](https://www.picgo.net/image/image9.4t3S4J)
|
||||
|
||||
2. 制定优化方向
|
||||
|
||||
|
|
|
|||
|
|
@ -8,7 +8,16 @@ import pandas as pd
|
|||
import torch
|
||||
|
||||
import flashinfer
|
||||
from bench_common import dtype, page_block_size, setup_workspace, setup_paged_kv_indptr, run_with_profiler, get_csv_path, compute_reps
|
||||
from benchmark_result import execute_benchmark_case, has_failures, STATUS_OK
|
||||
from bench_common import (
|
||||
compute_reps,
|
||||
dtype,
|
||||
get_csv_path,
|
||||
page_block_size,
|
||||
run_with_profiler,
|
||||
setup_paged_kv_indptr,
|
||||
setup_workspace,
|
||||
)
|
||||
|
||||
target_kernels = ["BatchPrefillWithPagedKVCacheKernel"]
|
||||
|
||||
|
|
@ -69,22 +78,39 @@ def run_benchmark():
|
|||
for idx, (bs, sl_kv, hd) in enumerate(test_cases, 1):
|
||||
num_qo_heads = 32
|
||||
num_kv_heads = 8 if hd == 64 else 4
|
||||
ms, io, flops = bench_batch_decode(bs, sl_kv, num_qo_heads, num_kv_heads, hd, page_block_size)
|
||||
bw = io / ms / 1e6
|
||||
tflops = flops / ms / 1e9
|
||||
records.append({
|
||||
"api": api_name,
|
||||
case = {
|
||||
"batch_size": bs,
|
||||
"seq_len_q": 1,
|
||||
"seq_len_kv": sl_kv,
|
||||
"num_qo_heads": num_qo_heads,
|
||||
"num_kv_heads": num_kv_heads,
|
||||
"head_dim": hd,
|
||||
"time_ms": ms,
|
||||
"bandwidth_GB_s": bw,
|
||||
"tflops": tflops,
|
||||
})
|
||||
print(f" [{idx}/{total_cases}] bs={bs}, kv_len={sl_kv}, hd={hd}: {ms:.3f}ms, {bw:.2f} GB/s, {tflops:.2f} TFLOPs")
|
||||
}
|
||||
record = execute_benchmark_case(
|
||||
api_name,
|
||||
case,
|
||||
lambda: bench_batch_decode(
|
||||
bs,
|
||||
sl_kv,
|
||||
num_qo_heads,
|
||||
num_kv_heads,
|
||||
hd,
|
||||
page_block_size,
|
||||
),
|
||||
)
|
||||
records.append(record)
|
||||
if record["status"] == STATUS_OK:
|
||||
print(
|
||||
f" [{idx}/{total_cases}] bs={bs}, kv_len={sl_kv}, hd={hd}: "
|
||||
f"{record['time_ms']:.3f}ms, "
|
||||
f"{record['bandwidth_GB_s']:.2f} GB/s, "
|
||||
f"{record['tflops']:.2f} TFLOPs"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
f" [{idx}/{total_cases}] bs={bs}, kv_len={sl_kv}, hd={hd}: "
|
||||
f"FAILED: {record['error_type']}: {record['error']}"
|
||||
)
|
||||
|
||||
return records
|
||||
|
||||
|
|
@ -98,4 +124,6 @@ if __name__ == "__main__":
|
|||
df = pd.DataFrame(records)
|
||||
csv_path = get_csv_path("BatchDecodeWithPagedKVCacheWrapper")
|
||||
df.to_csv(csv_path, index=False)
|
||||
print(f"\nResults saved to {csv_path}")
|
||||
print(f"\nResults saved to {csv_path}")
|
||||
if has_failures(records):
|
||||
raise SystemExit(1)
|
||||
|
|
|
|||
|
|
@ -8,7 +8,14 @@ import pandas as pd
|
|||
import torch
|
||||
|
||||
import flashinfer
|
||||
from bench_common import dtype, page_block_size, setup_workspace, run_with_profiler, get_csv_path, compute_reps
|
||||
from benchmark_result import execute_benchmark_case, has_failures, STATUS_OK
|
||||
from bench_common import (
|
||||
compute_reps,
|
||||
dtype,
|
||||
get_csv_path,
|
||||
run_with_profiler,
|
||||
setup_workspace,
|
||||
)
|
||||
|
||||
target_kernels = ["BatchMLAPagedAttentionKernel"]
|
||||
|
||||
|
|
@ -79,21 +86,34 @@ def run_benchmark():
|
|||
|
||||
print(f"[{api_name}] Starting benchmark, total cases: {total_cases}")
|
||||
for idx, (num_heads, bs, sl) in enumerate(test_cases, 1):
|
||||
ms, io, flops = bench_batch_mla_paged_attention(bs, sl, num_heads, head_dim_ckv, head_dim_kpe)
|
||||
bw = io / ms / 1e6
|
||||
tflops = flops / ms / 1e9
|
||||
records.append({
|
||||
"api": api_name,
|
||||
case = {
|
||||
"batch_size": bs,
|
||||
"seq_len": sl,
|
||||
"num_heads": num_heads,
|
||||
"head_dim_ckv": head_dim_ckv,
|
||||
"head_dim_kpe": head_dim_kpe,
|
||||
"time_ms": ms,
|
||||
"bandwidth_GB_s": bw,
|
||||
"tflops": tflops,
|
||||
})
|
||||
print(f" [{idx}/{total_cases}] bs={bs}, sl={sl}, num_heads={num_heads}: {ms:.3f}ms, {bw:.2f} GB/s, {tflops:.2f} TFLOPs")
|
||||
}
|
||||
record = execute_benchmark_case(
|
||||
api_name,
|
||||
case,
|
||||
lambda: bench_batch_mla_paged_attention(
|
||||
bs, sl, num_heads, head_dim_ckv, head_dim_kpe
|
||||
),
|
||||
)
|
||||
records.append(record)
|
||||
if record["status"] == STATUS_OK:
|
||||
print(
|
||||
f" [{idx}/{total_cases}] bs={bs}, sl={sl}, "
|
||||
f"num_heads={num_heads}: {record['time_ms']:.3f}ms, "
|
||||
f"{record['bandwidth_GB_s']:.2f} GB/s, "
|
||||
f"{record['tflops']:.2f} TFLOPs"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
f" [{idx}/{total_cases}] bs={bs}, sl={sl}, "
|
||||
f"num_heads={num_heads}: FAILED: "
|
||||
f"{record['error_type']}: {record['error']}"
|
||||
)
|
||||
|
||||
return records
|
||||
|
||||
|
|
@ -107,4 +127,6 @@ if __name__ == "__main__":
|
|||
df = pd.DataFrame(records)
|
||||
csv_path = get_csv_path("BatchMLAPagedAttentionWrapper")
|
||||
df.to_csv(csv_path, index=False)
|
||||
print(f"\nResults saved to {csv_path}")
|
||||
print(f"\nResults saved to {csv_path}")
|
||||
if has_failures(records):
|
||||
raise SystemExit(1)
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ from bench_common import (
|
|||
get_csv_path,
|
||||
compute_reps,
|
||||
)
|
||||
from benchmark_result import execute_benchmark_case, has_failures, STATUS_OK
|
||||
|
||||
target_kernels = ["BatchPrefillWithPagedKVCacheKernel"]
|
||||
|
||||
|
|
@ -95,27 +96,33 @@ def run_benchmark():
|
|||
for idx, (head_dim, bs, sl) in enumerate(test_cases, 1):
|
||||
num_qo_heads = 32
|
||||
num_kv_heads = 8 if head_dim == 64 else 4
|
||||
ms, io, flops = bench_batch_prefill_with_paged_kv_cache(
|
||||
bs, sl, num_qo_heads, num_kv_heads, head_dim
|
||||
)
|
||||
bw = io / ms / 1e6
|
||||
tflops = flops / ms / 1e9
|
||||
records.append(
|
||||
{
|
||||
"api": api_name,
|
||||
"batch_size": bs,
|
||||
"seq_len": sl,
|
||||
"num_qo_heads": num_qo_heads,
|
||||
"num_kv_heads": num_kv_heads,
|
||||
"head_dim": head_dim,
|
||||
"time_ms": ms,
|
||||
"bandwidth_GB_s": bw,
|
||||
"tflops": tflops,
|
||||
}
|
||||
)
|
||||
print(
|
||||
f" [{idx}/{total_cases}] bs={bs}, sl={sl}, hd={head_dim}: {ms:.3f}ms, {bw:.2f} GB/s, {tflops:.2f} TFLOPs"
|
||||
case = {
|
||||
"batch_size": bs,
|
||||
"seq_len": sl,
|
||||
"num_qo_heads": num_qo_heads,
|
||||
"num_kv_heads": num_kv_heads,
|
||||
"head_dim": head_dim,
|
||||
}
|
||||
record = execute_benchmark_case(
|
||||
api_name,
|
||||
case,
|
||||
lambda: bench_batch_prefill_with_paged_kv_cache(
|
||||
bs, sl, num_qo_heads, num_kv_heads, head_dim
|
||||
),
|
||||
)
|
||||
records.append(record)
|
||||
if record["status"] == STATUS_OK:
|
||||
print(
|
||||
f" [{idx}/{total_cases}] bs={bs}, sl={sl}, hd={head_dim}: "
|
||||
f"{record['time_ms']:.3f}ms, "
|
||||
f"{record['bandwidth_GB_s']:.2f} GB/s, "
|
||||
f"{record['tflops']:.2f} TFLOPs"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
f" [{idx}/{total_cases}] bs={bs}, sl={sl}, hd={head_dim}: "
|
||||
f"FAILED: {record['error_type']}: {record['error']}"
|
||||
)
|
||||
|
||||
return records
|
||||
|
||||
|
|
@ -131,3 +138,5 @@ if __name__ == "__main__":
|
|||
csv_path = get_csv_path("BatchPrefillWithPagedKVCacheWrapper")
|
||||
df.to_csv(csv_path, index=False)
|
||||
print(f"\nResults saved to {csv_path}")
|
||||
if has_failures(records):
|
||||
raise SystemExit(1)
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ from bench_common import (
|
|||
get_csv_path,
|
||||
compute_reps,
|
||||
)
|
||||
from benchmark_result import execute_benchmark_case, has_failures, STATUS_OK
|
||||
|
||||
target_kernels = [
|
||||
"BatchPrefillWithRaggedKVCacheKernel",
|
||||
|
|
@ -97,28 +98,41 @@ def run_benchmark():
|
|||
for idx, ((head_dim_qk, head_dim_vo), bs, sl) in enumerate(test_cases, 1):
|
||||
num_qo_heads = 32
|
||||
num_kv_heads = 4
|
||||
ms, io, flops = bench_batch_prefill_with_ragged_kv_cache(
|
||||
bs, sl, num_qo_heads, num_kv_heads, head_dim_qk, head_dim_vo
|
||||
)
|
||||
bw = io / ms / 1e6
|
||||
tflops = flops / ms / 1e9
|
||||
records.append(
|
||||
{
|
||||
"api": api_name,
|
||||
"batch_size": bs,
|
||||
"seq_len": sl,
|
||||
"num_qo_heads": num_qo_heads,
|
||||
"num_kv_heads": num_kv_heads,
|
||||
"head_dim_qk": head_dim_qk,
|
||||
"head_dim_vo": head_dim_vo,
|
||||
"time_ms": ms,
|
||||
"bandwidth_GB_s": bw,
|
||||
"tflops": tflops,
|
||||
}
|
||||
)
|
||||
print(
|
||||
f" [{idx}/{total_cases}] bs={bs}, sl={sl}, hd=[{head_dim_qk},{head_dim_vo}]: {ms:.3f}ms, {bw:.2f} GB/s, {tflops:.2f} TFLOPs"
|
||||
case = {
|
||||
"batch_size": bs,
|
||||
"seq_len": sl,
|
||||
"num_qo_heads": num_qo_heads,
|
||||
"num_kv_heads": num_kv_heads,
|
||||
"head_dim_qk": head_dim_qk,
|
||||
"head_dim_vo": head_dim_vo,
|
||||
}
|
||||
record = execute_benchmark_case(
|
||||
api_name,
|
||||
case,
|
||||
lambda: bench_batch_prefill_with_ragged_kv_cache(
|
||||
bs,
|
||||
sl,
|
||||
num_qo_heads,
|
||||
num_kv_heads,
|
||||
head_dim_qk,
|
||||
head_dim_vo,
|
||||
),
|
||||
)
|
||||
records.append(record)
|
||||
if record["status"] == STATUS_OK:
|
||||
print(
|
||||
f" [{idx}/{total_cases}] bs={bs}, sl={sl}, "
|
||||
f"hd=[{head_dim_qk},{head_dim_vo}]: "
|
||||
f"{record['time_ms']:.3f}ms, "
|
||||
f"{record['bandwidth_GB_s']:.2f} GB/s, "
|
||||
f"{record['tflops']:.2f} TFLOPs"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
f" [{idx}/{total_cases}] bs={bs}, sl={sl}, "
|
||||
f"hd=[{head_dim_qk},{head_dim_vo}]: FAILED: "
|
||||
f"{record['error_type']}: {record['error']}"
|
||||
)
|
||||
|
||||
return records
|
||||
|
||||
|
|
@ -134,3 +148,5 @@ if __name__ == "__main__":
|
|||
csv_path = get_csv_path("BatchPrefillWithRaggedKVCacheWrapper")
|
||||
df.to_csv(csv_path, index=False)
|
||||
print(f"\nResults saved to {csv_path}")
|
||||
if has_failures(records):
|
||||
raise SystemExit(1)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,80 @@
|
|||
"""Shared result schema and helpers for FlashInfer benchmarks."""
|
||||
|
||||
import math
|
||||
|
||||
|
||||
SCHEMA_VERSION = "1"
|
||||
STATUS_OK = "ok"
|
||||
STATUS_FAILED = "failed"
|
||||
STATUS_LEGACY_OK = "legacy_inferred_ok"
|
||||
|
||||
NUMERIC_COLUMNS = ("time_ms", "bandwidth_GB_s", "tflops")
|
||||
RESULT_COLUMNS = (
|
||||
"schema_version",
|
||||
"api",
|
||||
"status",
|
||||
*NUMERIC_COLUMNS,
|
||||
"error_type",
|
||||
"error",
|
||||
)
|
||||
RESERVED_COLUMNS = frozenset((*RESULT_COLUMNS, "_source"))
|
||||
|
||||
|
||||
def _validate_metric(name, value, *, positive=False):
|
||||
number = float(value)
|
||||
if not math.isfinite(number):
|
||||
raise ValueError(f"{name} must be finite, got {value!r}")
|
||||
if positive and number <= 0:
|
||||
raise ValueError(f"{name} must be greater than zero, got {value!r}")
|
||||
if not positive and number < 0:
|
||||
raise ValueError(f"{name} must not be negative, got {value!r}")
|
||||
return number
|
||||
|
||||
|
||||
def execute_benchmark_case(api, case, benchmark_fn):
|
||||
"""Execute one benchmark case and return a schema-v1 result record."""
|
||||
conflicting = RESERVED_COLUMNS.intersection(case)
|
||||
if conflicting:
|
||||
names = ", ".join(sorted(conflicting))
|
||||
raise ValueError(f"case fields use reserved result columns: {names}")
|
||||
|
||||
record = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"api": api,
|
||||
**case,
|
||||
}
|
||||
|
||||
try:
|
||||
time_ms, io_bytes, flops = benchmark_fn()
|
||||
time_ms = _validate_metric("time_ms", time_ms, positive=True)
|
||||
io_bytes = _validate_metric("io_bytes", io_bytes)
|
||||
flops = _validate_metric("flops", flops)
|
||||
bandwidth = _validate_metric(
|
||||
"bandwidth_GB_s", io_bytes / time_ms / 1e6
|
||||
)
|
||||
tflops = _validate_metric("tflops", flops / time_ms / 1e9)
|
||||
except Exception as exc:
|
||||
return {
|
||||
**record,
|
||||
"status": STATUS_FAILED,
|
||||
"time_ms": "",
|
||||
"bandwidth_GB_s": "",
|
||||
"tflops": "",
|
||||
"error_type": type(exc).__name__,
|
||||
"error": str(exc),
|
||||
}
|
||||
|
||||
return {
|
||||
**record,
|
||||
"status": STATUS_OK,
|
||||
"time_ms": time_ms,
|
||||
"bandwidth_GB_s": bandwidth,
|
||||
"tflops": tflops,
|
||||
"error_type": "",
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def has_failures(records):
|
||||
"""Return whether a collection contains at least one failed case."""
|
||||
return any(record.get("status") == STATUS_FAILED for record in records)
|
||||
|
|
@ -0,0 +1,742 @@
|
|||
"""Summarize and compare FlashInfer benchmark CSV files."""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import math
|
||||
from collections import Counter, defaultdict
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
from benchmark_result import (
|
||||
NUMERIC_COLUMNS,
|
||||
RESERVED_COLUMNS,
|
||||
SCHEMA_VERSION,
|
||||
STATUS_FAILED,
|
||||
STATUS_LEGACY_OK,
|
||||
STATUS_OK,
|
||||
)
|
||||
|
||||
|
||||
EXIT_OK = 0
|
||||
EXIT_BENCHMARK_FAILURE = 1
|
||||
EXIT_INPUT_ERROR = 2
|
||||
SUCCESS_STATUSES = frozenset((STATUS_OK, STATUS_LEGACY_OK))
|
||||
|
||||
CASE_COLUMN_ORDER = (
|
||||
"batch_size",
|
||||
"seq_len",
|
||||
"seq_len_q",
|
||||
"seq_len_kv",
|
||||
"num_heads",
|
||||
"num_qo_heads",
|
||||
"num_kv_heads",
|
||||
"head_dim",
|
||||
"head_dim_qk",
|
||||
"head_dim_vo",
|
||||
"head_dim_ckv",
|
||||
"head_dim_kpe",
|
||||
)
|
||||
|
||||
COMMON_LEGACY_COLUMNS = frozenset(("api", *NUMERIC_COLUMNS))
|
||||
LEGACY_SCHEMAS = (
|
||||
COMMON_LEGACY_COLUMNS
|
||||
| frozenset(
|
||||
(
|
||||
"batch_size",
|
||||
"seq_len_q",
|
||||
"seq_len_kv",
|
||||
"num_qo_heads",
|
||||
"num_kv_heads",
|
||||
"head_dim",
|
||||
)
|
||||
),
|
||||
COMMON_LEGACY_COLUMNS
|
||||
| frozenset(
|
||||
(
|
||||
"batch_size",
|
||||
"seq_len",
|
||||
"num_heads",
|
||||
"head_dim_ckv",
|
||||
"head_dim_kpe",
|
||||
)
|
||||
),
|
||||
COMMON_LEGACY_COLUMNS
|
||||
| frozenset(
|
||||
(
|
||||
"batch_size",
|
||||
"seq_len",
|
||||
"num_qo_heads",
|
||||
"num_kv_heads",
|
||||
"head_dim",
|
||||
)
|
||||
),
|
||||
COMMON_LEGACY_COLUMNS
|
||||
| frozenset(
|
||||
(
|
||||
"batch_size",
|
||||
"seq_len",
|
||||
"num_qo_heads",
|
||||
"num_kv_heads",
|
||||
"head_dim_qk",
|
||||
"head_dim_vo",
|
||||
)
|
||||
),
|
||||
)
|
||||
V1_REQUIRED_COLUMNS = frozenset(
|
||||
(
|
||||
"schema_version",
|
||||
"api",
|
||||
"status",
|
||||
*NUMERIC_COLUMNS,
|
||||
"error_type",
|
||||
"error",
|
||||
)
|
||||
)
|
||||
INTERNAL_COLUMNS = frozenset(("_source", "_line"))
|
||||
|
||||
|
||||
@dataclass
|
||||
class InputIssue:
|
||||
source: str
|
||||
message: str
|
||||
group: str = "input"
|
||||
|
||||
|
||||
@dataclass
|
||||
class LoadResult:
|
||||
group: str
|
||||
paths: list
|
||||
records: list = field(default_factory=list)
|
||||
issues: list = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def valid_sources(self):
|
||||
return {record["_source"] for record in self.records}
|
||||
|
||||
|
||||
def parse_float(value):
|
||||
try:
|
||||
number = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return number if math.isfinite(number) else None
|
||||
|
||||
|
||||
def _metric_error(record, column):
|
||||
value = parse_float(record.get(column))
|
||||
if value is None:
|
||||
return f"{column} must be a finite number"
|
||||
if column == "time_ms" and value <= 0:
|
||||
return "time_ms must be greater than zero"
|
||||
if column != "time_ms" and value < 0:
|
||||
return f"{column} must not be negative"
|
||||
record[column] = value
|
||||
return None
|
||||
|
||||
|
||||
def _matches_known_schema(fieldnames):
|
||||
return any(schema.issubset(fieldnames) for schema in LEGACY_SCHEMAS)
|
||||
|
||||
|
||||
def _validate_headers(fieldnames):
|
||||
if not fieldnames:
|
||||
return None, "CSV header is missing"
|
||||
if any(name is None or not name for name in fieldnames):
|
||||
return None, "CSV contains an empty header name"
|
||||
if any(name != name.strip() for name in fieldnames):
|
||||
return None, "CSV header names must not contain surrounding whitespace"
|
||||
if len(fieldnames) != len(set(fieldnames)):
|
||||
return None, "CSV contains duplicate header names"
|
||||
|
||||
names = frozenset(fieldnames)
|
||||
has_v1_marker = "schema_version" in names or "status" in names
|
||||
if has_v1_marker:
|
||||
missing = sorted(V1_REQUIRED_COLUMNS - names)
|
||||
if missing:
|
||||
return None, f"schema v1 is missing columns: {', '.join(missing)}"
|
||||
if not _matches_known_schema(names):
|
||||
return None, "CSV does not match a known FlashInfer benchmark schema"
|
||||
return "v1", None
|
||||
|
||||
if not _matches_known_schema(names):
|
||||
return None, "CSV does not match a known legacy FlashInfer schema"
|
||||
return "legacy", None
|
||||
|
||||
|
||||
def _validate_row(raw_row, schema, source, line_number):
|
||||
if None in raw_row:
|
||||
return None, "row has more values than the CSV header"
|
||||
if any(value is None for value in raw_row.values()):
|
||||
return None, "row has fewer values than the CSV header"
|
||||
|
||||
record = {key: value.strip() for key, value in raw_row.items()}
|
||||
if not record.get("api"):
|
||||
return None, "api must not be empty"
|
||||
|
||||
if schema == "v1":
|
||||
if record.get("schema_version") != SCHEMA_VERSION:
|
||||
return None, (
|
||||
"unsupported schema_version "
|
||||
f"{record.get('schema_version')!r}; expected {SCHEMA_VERSION!r}"
|
||||
)
|
||||
status = record.get("status")
|
||||
if status not in (STATUS_OK, STATUS_FAILED):
|
||||
return None, f"unsupported status {status!r}"
|
||||
if status == STATUS_OK and (
|
||||
record.get("error_type") or record.get("error")
|
||||
):
|
||||
return None, "successful rows must not contain error details"
|
||||
if status == STATUS_FAILED and not (
|
||||
record.get("error_type") or record.get("error")
|
||||
):
|
||||
return None, "failed rows must contain error_type or error"
|
||||
else:
|
||||
record["schema_version"] = ""
|
||||
record["status"] = STATUS_LEGACY_OK
|
||||
record["error_type"] = ""
|
||||
record["error"] = ""
|
||||
|
||||
if record["status"] in SUCCESS_STATUSES:
|
||||
for column in NUMERIC_COLUMNS:
|
||||
error = _metric_error(record, column)
|
||||
if error:
|
||||
return None, error
|
||||
|
||||
record["_source"] = source
|
||||
record["_line"] = line_number
|
||||
return record, None
|
||||
|
||||
|
||||
def load_csv(path, group="input"):
|
||||
source = str(path)
|
||||
records = []
|
||||
issues = []
|
||||
try:
|
||||
with path.open("r", encoding="utf-8-sig", newline="") as csv_file:
|
||||
reader = csv.DictReader(csv_file)
|
||||
schema, error = _validate_headers(reader.fieldnames)
|
||||
if error:
|
||||
return [], [InputIssue(source, error, group)]
|
||||
|
||||
for line_number, row in enumerate(reader, 2):
|
||||
record, row_error = _validate_row(
|
||||
row, schema, source, line_number
|
||||
)
|
||||
if row_error:
|
||||
issues.append(
|
||||
InputIssue(
|
||||
source,
|
||||
f"line {line_number}: {row_error}",
|
||||
group,
|
||||
)
|
||||
)
|
||||
else:
|
||||
records.append(record)
|
||||
except (OSError, UnicodeError, csv.Error) as exc:
|
||||
return [], [InputIssue(source, f"unable to read CSV: {exc}", group)]
|
||||
|
||||
if not records and not issues:
|
||||
issues.append(InputIssue(source, "CSV contains no benchmark rows", group))
|
||||
if issues:
|
||||
return [], issues
|
||||
return records, []
|
||||
|
||||
|
||||
def load_inputs(paths, group="input"):
|
||||
result = LoadResult(group=group, paths=list(paths))
|
||||
for path in paths:
|
||||
records, issues = load_csv(path, group)
|
||||
result.records.extend(records)
|
||||
result.issues.extend(issues)
|
||||
return result
|
||||
|
||||
|
||||
def _case_columns(record):
|
||||
columns = [
|
||||
key
|
||||
for key in record
|
||||
if key not in RESERVED_COLUMNS
|
||||
and key not in INTERNAL_COLUMNS
|
||||
and not key.startswith("_")
|
||||
]
|
||||
order = {name: index for index, name in enumerate(CASE_COLUMN_ORDER)}
|
||||
return sorted(columns, key=lambda name: (order.get(name, len(order)), name))
|
||||
|
||||
|
||||
def describe_case(record):
|
||||
parts = [
|
||||
f"{key}={record[key]}"
|
||||
for key in _case_columns(record)
|
||||
if str(record.get(key, "")).strip()
|
||||
]
|
||||
return ", ".join(parts) if parts else "n/a"
|
||||
|
||||
|
||||
def case_identity(record):
|
||||
values = tuple(
|
||||
(key, str(record.get(key, "")).strip())
|
||||
for key in _case_columns(record)
|
||||
)
|
||||
return record["api"], values
|
||||
|
||||
|
||||
def group_by_api(records):
|
||||
grouped = defaultdict(list)
|
||||
for record in records:
|
||||
grouped[record["api"]].append(record)
|
||||
return dict(grouped)
|
||||
|
||||
|
||||
def best_record(records, column):
|
||||
candidates = [
|
||||
record for record in records if record["status"] in SUCCESS_STATUSES
|
||||
]
|
||||
if not candidates:
|
||||
return None
|
||||
key = lambda record: record[column]
|
||||
return min(candidates, key=key) if column == "time_ms" else max(
|
||||
candidates, key=key
|
||||
)
|
||||
|
||||
|
||||
def _escape_markdown_fragment(value):
|
||||
text = str(value)
|
||||
for old, new in (
|
||||
("\\", "\\\\"),
|
||||
("|", "\\|"),
|
||||
("`", "\\`"),
|
||||
("*", "\\*"),
|
||||
("_", "\\_"),
|
||||
("#", "\\#"),
|
||||
("[", "\\["),
|
||||
("]", "\\]"),
|
||||
("<", "<"),
|
||||
(">", ">"),
|
||||
):
|
||||
text = text.replace(old, new)
|
||||
return text
|
||||
|
||||
|
||||
def escape_markdown(value):
|
||||
normalized = str(value).replace("\r\n", "\n").replace("\r", "\n")
|
||||
return "<br>".join(
|
||||
_escape_markdown_fragment(part) for part in normalized.split("\n")
|
||||
)
|
||||
|
||||
|
||||
def format_metric(value):
|
||||
return f"{value:.6g}"
|
||||
|
||||
|
||||
def format_delta(value):
|
||||
if math.isinf(value):
|
||||
return "+∞%" if value > 0 else "-∞%"
|
||||
return f"{value:+.2f}%"
|
||||
|
||||
|
||||
def _status_text(records):
|
||||
statuses = Counter(record["status"] for record in records)
|
||||
return ", ".join(
|
||||
f"{key}={value}" for key, value in sorted(statuses.items())
|
||||
)
|
||||
|
||||
|
||||
def _input_file_lines(result):
|
||||
records_by_source = defaultdict(list)
|
||||
issues_by_source = defaultdict(list)
|
||||
for record in result.records:
|
||||
records_by_source[record["_source"]].append(record)
|
||||
for issue in result.issues:
|
||||
issues_by_source[issue.source].append(issue)
|
||||
|
||||
lines = ["## Input Files", "", "| File | Rows | Status |", "|---|---:|---|"]
|
||||
for path in result.paths:
|
||||
source = str(path)
|
||||
file_records = records_by_source[source]
|
||||
if issues_by_source[source]:
|
||||
status = "invalid"
|
||||
else:
|
||||
status = _status_text(file_records)
|
||||
lines.append(
|
||||
f"| {escape_markdown(source)} | {len(file_records)} | "
|
||||
f"{escape_markdown(status)} |"
|
||||
)
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def _input_error_lines(issues):
|
||||
if not issues:
|
||||
return []
|
||||
lines = ["## Input Errors", ""]
|
||||
for issue in issues:
|
||||
lines.append(
|
||||
f"- **{escape_markdown(issue.group)}** — "
|
||||
f"{escape_markdown(issue.source)}: "
|
||||
f"{escape_markdown(issue.message)}"
|
||||
)
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def _failure_lines(records):
|
||||
failures = [
|
||||
record for record in records if record["status"] == STATUS_FAILED
|
||||
]
|
||||
if not failures:
|
||||
return []
|
||||
lines = [
|
||||
"## Failure Details",
|
||||
"",
|
||||
"| API | Case | Source | Error |",
|
||||
"|---|---|---|---|",
|
||||
]
|
||||
for record in failures:
|
||||
error = f"{record.get('error_type', '')}: {record.get('error', '')}"
|
||||
lines.append(
|
||||
f"| {escape_markdown(record['api'])} | "
|
||||
f"{escape_markdown(describe_case(record))} | "
|
||||
f"{escape_markdown(record['_source'])} | "
|
||||
f"{escape_markdown(error.strip(': '))} |"
|
||||
)
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def _grouped_failure_lines(grouped_records):
|
||||
failures = [
|
||||
(group, record)
|
||||
for group, record in grouped_records
|
||||
if record["status"] == STATUS_FAILED
|
||||
]
|
||||
if not failures:
|
||||
return []
|
||||
lines = [
|
||||
"## Failure Details",
|
||||
"",
|
||||
"| Run | API | Case | Source | Error |",
|
||||
"|---|---|---|---|---|",
|
||||
]
|
||||
for group, record in failures:
|
||||
error = f"{record.get('error_type', '')}: {record.get('error', '')}"
|
||||
lines.append(
|
||||
f"| {escape_markdown(group)} | "
|
||||
f"{escape_markdown(record['api'])} | "
|
||||
f"{escape_markdown(describe_case(record))} | "
|
||||
f"{escape_markdown(record['_source'])} | "
|
||||
f"{escape_markdown(error.strip(': '))} |"
|
||||
)
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def build_summary_markdown(result):
|
||||
lines = [
|
||||
"# FlashInfer Benchmark Summary",
|
||||
"",
|
||||
f"- Input files: {len(result.paths)}",
|
||||
f"- Valid files: {len(result.valid_sources)}",
|
||||
f"- Total rows: {len(result.records)}",
|
||||
"",
|
||||
]
|
||||
lines.extend(_input_file_lines(result))
|
||||
|
||||
for api, api_records in sorted(group_by_api(result.records).items()):
|
||||
lines.extend(
|
||||
[
|
||||
f"## API: {escape_markdown(api)}",
|
||||
"",
|
||||
f"- Rows: {len(api_records)}",
|
||||
f"- Status: {escape_markdown(_status_text(api_records))}",
|
||||
]
|
||||
)
|
||||
for column in NUMERIC_COLUMNS:
|
||||
record = best_record(api_records, column)
|
||||
if record is None:
|
||||
continue
|
||||
label = "Minimum" if column == "time_ms" else "Maximum"
|
||||
lines.append(
|
||||
f"- {label} {column}: {format_metric(record[column])} "
|
||||
f"({escape_markdown(describe_case(record))}; "
|
||||
f"source={escape_markdown(record['_source'])})"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
lines.extend(_failure_lines(result.records))
|
||||
lines.extend(_input_error_lines(result.issues))
|
||||
if any(
|
||||
record["status"] == STATUS_LEGACY_OK for record in result.records
|
||||
):
|
||||
lines.extend(
|
||||
[
|
||||
"## Compatibility Note",
|
||||
"",
|
||||
"- `legacy_inferred_ok` means the legacy CSV had no explicit "
|
||||
"status column; success was inferred only after validating its "
|
||||
"known FlashInfer schema and performance metrics.",
|
||||
"",
|
||||
]
|
||||
)
|
||||
return "\n".join(lines).rstrip() + "\n"
|
||||
|
||||
|
||||
def build_case_index(records, group):
|
||||
index = {}
|
||||
duplicate_keys = set()
|
||||
issues = []
|
||||
for record in records:
|
||||
key = case_identity(record)
|
||||
if key in index:
|
||||
first = index[key]
|
||||
issues.append(
|
||||
InputIssue(
|
||||
record["_source"],
|
||||
"duplicate case also found at "
|
||||
f"{first['_source']}:{first['_line']}: "
|
||||
f"{record['api']} ({describe_case(record)})",
|
||||
group,
|
||||
)
|
||||
)
|
||||
duplicate_keys.add(key)
|
||||
else:
|
||||
index[key] = record
|
||||
for key in duplicate_keys:
|
||||
index.pop(key, None)
|
||||
return index, issues
|
||||
|
||||
|
||||
def _percent_delta(baseline, candidate):
|
||||
if baseline == 0:
|
||||
return 0.0 if candidate == 0 else math.inf
|
||||
return (candidate / baseline - 1.0) * 100.0
|
||||
|
||||
|
||||
def _is_threshold_regression(deltas, threshold):
|
||||
if threshold is None:
|
||||
return False
|
||||
return (
|
||||
deltas["time_ms"] > threshold
|
||||
or deltas["bandwidth_GB_s"] < -threshold
|
||||
or deltas["tflops"] < -threshold
|
||||
)
|
||||
|
||||
|
||||
def build_regression_markdown(baseline, candidate, threshold=None):
|
||||
baseline_index, baseline_duplicates = build_case_index(
|
||||
baseline.records, "baseline"
|
||||
)
|
||||
candidate_index, candidate_duplicates = build_case_index(
|
||||
candidate.records, "candidate"
|
||||
)
|
||||
comparison_issues = baseline_duplicates + candidate_duplicates
|
||||
|
||||
baseline_keys = set(baseline_index)
|
||||
candidate_keys = set(candidate_index)
|
||||
matched_keys = sorted(baseline_keys & candidate_keys)
|
||||
missing_keys = sorted(baseline_keys - candidate_keys)
|
||||
new_keys = sorted(candidate_keys - baseline_keys)
|
||||
|
||||
lines = [
|
||||
"# FlashInfer Benchmark Regression",
|
||||
"",
|
||||
f"- Baseline files: {len(baseline.paths)}",
|
||||
f"- Candidate files: {len(candidate.paths)}",
|
||||
f"- Matched cases: {len(matched_keys)}",
|
||||
f"- Missing candidate cases: {len(missing_keys)}",
|
||||
f"- New candidate cases: {len(new_keys)}",
|
||||
"- Regression gate: "
|
||||
+ ("report only" if threshold is None else f"{threshold:g}%"),
|
||||
"",
|
||||
"## Matched Cases",
|
||||
"",
|
||||
"| API | Case | State | Time ms (base → cand, Δ) | "
|
||||
"Bandwidth GB/s (base → cand, Δ) | TFLOPS (base → cand, Δ) | Sources |",
|
||||
"|---|---|---|---|---|---|---|",
|
||||
]
|
||||
|
||||
threshold_failed = False
|
||||
candidate_failed = False
|
||||
for key in matched_keys:
|
||||
base_record = baseline_index[key]
|
||||
candidate_record = candidate_index[key]
|
||||
base_success = base_record["status"] in SUCCESS_STATUSES
|
||||
candidate_success = candidate_record["status"] in SUCCESS_STATUSES
|
||||
deltas = None
|
||||
|
||||
if base_success and candidate_success:
|
||||
deltas = {
|
||||
column: _percent_delta(
|
||||
base_record[column], candidate_record[column]
|
||||
)
|
||||
for column in NUMERIC_COLUMNS
|
||||
}
|
||||
regressed = _is_threshold_regression(deltas, threshold)
|
||||
state = "regression" if regressed else "compared"
|
||||
threshold_failed = threshold_failed or regressed
|
||||
elif base_success and not candidate_success:
|
||||
state = "candidate failed"
|
||||
candidate_failed = True
|
||||
elif not base_success and candidate_success:
|
||||
state = "recovered"
|
||||
else:
|
||||
state = "both failed"
|
||||
candidate_failed = True
|
||||
|
||||
if deltas:
|
||||
metric_cells = [
|
||||
f"{format_metric(base_record[column])} → "
|
||||
f"{format_metric(candidate_record[column])}, "
|
||||
f"{format_delta(deltas[column])}"
|
||||
for column in NUMERIC_COLUMNS
|
||||
]
|
||||
else:
|
||||
metric_cells = ["n/a", "n/a", "n/a"]
|
||||
sources = f"{base_record['_source']} → {candidate_record['_source']}"
|
||||
lines.append(
|
||||
f"| {escape_markdown(key[0])} | "
|
||||
f"{escape_markdown(describe_case(base_record))} | "
|
||||
f"{escape_markdown(state)} | "
|
||||
f"{escape_markdown(metric_cells[0])} | "
|
||||
f"{escape_markdown(metric_cells[1])} | "
|
||||
f"{escape_markdown(metric_cells[2])} | "
|
||||
f"{escape_markdown(sources)} |"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
if missing_keys:
|
||||
lines.extend(["## Missing Candidate Cases", ""])
|
||||
for key in missing_keys:
|
||||
record = baseline_index[key]
|
||||
lines.append(
|
||||
f"- {escape_markdown(record['api'])}: "
|
||||
f"{escape_markdown(describe_case(record))} "
|
||||
f"(baseline source={escape_markdown(record['_source'])})"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
if new_keys:
|
||||
lines.extend(["## New Candidate Cases", ""])
|
||||
for key in new_keys:
|
||||
record = candidate_index[key]
|
||||
lines.append(
|
||||
f"- {escape_markdown(record['api'])}: "
|
||||
f"{escape_markdown(describe_case(record))} "
|
||||
f"(candidate source={escape_markdown(record['_source'])})"
|
||||
)
|
||||
if record["status"] == STATUS_FAILED:
|
||||
candidate_failed = True
|
||||
lines.append("")
|
||||
|
||||
grouped_failures = [
|
||||
("baseline", record)
|
||||
for record in baseline.records
|
||||
if record["status"] == STATUS_FAILED
|
||||
] + [
|
||||
("candidate", record)
|
||||
for record in candidate.records
|
||||
if record["status"] == STATUS_FAILED
|
||||
]
|
||||
if grouped_failures:
|
||||
lines.extend(_grouped_failure_lines(grouped_failures))
|
||||
|
||||
all_issues = baseline.issues + candidate.issues + comparison_issues
|
||||
lines.extend(_input_error_lines(all_issues))
|
||||
|
||||
comparison_failed = (
|
||||
candidate_failed or bool(missing_keys) or threshold_failed
|
||||
)
|
||||
return (
|
||||
"\n".join(lines).rstrip() + "\n",
|
||||
comparison_failed,
|
||||
comparison_issues,
|
||||
)
|
||||
|
||||
|
||||
def parse_args(argv=None):
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="summarize_results.py", description=__doc__
|
||||
)
|
||||
parser.add_argument(
|
||||
"csv",
|
||||
nargs="*",
|
||||
type=Path,
|
||||
help="Benchmark CSV files for summary mode.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--baseline",
|
||||
nargs="+",
|
||||
type=Path,
|
||||
help="Baseline CSV files for regression mode.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--candidate",
|
||||
nargs="+",
|
||||
type=Path,
|
||||
help="Candidate CSV files for regression mode.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--fail-on-regression",
|
||||
type=float,
|
||||
metavar="PERCENT",
|
||||
help="Return 1 when a metric regresses by more than this percentage.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=Path,
|
||||
default=Path("flashinfer_benchmark_summary.md"),
|
||||
help="Markdown report output path.",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
regression_mode = args.baseline is not None or args.candidate is not None
|
||||
if regression_mode:
|
||||
if args.csv:
|
||||
parser.error("positional CSV files cannot be used with regression mode")
|
||||
if args.baseline is None or args.candidate is None:
|
||||
parser.error("regression mode requires both --baseline and --candidate")
|
||||
elif not args.csv:
|
||||
parser.error("provide CSV files or use --baseline with --candidate")
|
||||
|
||||
if args.fail_on_regression is not None:
|
||||
if not regression_mode:
|
||||
parser.error("--fail-on-regression requires regression mode")
|
||||
if not math.isfinite(args.fail_on_regression):
|
||||
parser.error("--fail-on-regression must be finite")
|
||||
if args.fail_on_regression < 0:
|
||||
parser.error("--fail-on-regression must not be negative")
|
||||
return args
|
||||
|
||||
|
||||
def _write_report(path, markdown):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(markdown, encoding="utf-8")
|
||||
print(f"Summary saved to {path}")
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
args = parse_args(argv)
|
||||
if args.baseline is not None:
|
||||
baseline = load_inputs(args.baseline, "baseline")
|
||||
candidate = load_inputs(args.candidate, "candidate")
|
||||
markdown, comparison_failed, comparison_issues = (
|
||||
build_regression_markdown(
|
||||
baseline, candidate, args.fail_on_regression
|
||||
)
|
||||
)
|
||||
_write_report(args.output, markdown)
|
||||
if baseline.issues or candidate.issues or comparison_issues:
|
||||
return EXIT_INPUT_ERROR
|
||||
return EXIT_BENCHMARK_FAILURE if comparison_failed else EXIT_OK
|
||||
|
||||
result = load_inputs(args.csv)
|
||||
markdown = build_summary_markdown(result)
|
||||
_write_report(args.output, markdown)
|
||||
if result.issues:
|
||||
return EXIT_INPUT_ERROR
|
||||
if any(record["status"] == STATUS_FAILED for record in result.records):
|
||||
return EXIT_BENCHMARK_FAILURE
|
||||
return EXIT_OK
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -0,0 +1,372 @@
|
|||
import csv
|
||||
import io
|
||||
import math
|
||||
import tempfile
|
||||
import unittest
|
||||
from contextlib import redirect_stderr, redirect_stdout
|
||||
from pathlib import Path
|
||||
|
||||
from benchmark_result import (
|
||||
SCHEMA_VERSION,
|
||||
STATUS_FAILED,
|
||||
STATUS_OK,
|
||||
execute_benchmark_case,
|
||||
)
|
||||
import summarize_results
|
||||
|
||||
|
||||
V1_FIELDS = [
|
||||
"schema_version",
|
||||
"api",
|
||||
"batch_size",
|
||||
"seq_len",
|
||||
"num_qo_heads",
|
||||
"num_kv_heads",
|
||||
"head_dim",
|
||||
"status",
|
||||
"time_ms",
|
||||
"bandwidth_GB_s",
|
||||
"tflops",
|
||||
"error_type",
|
||||
"error",
|
||||
]
|
||||
LEGACY_FIELDS = [
|
||||
"api",
|
||||
"batch_size",
|
||||
"seq_len",
|
||||
"num_qo_heads",
|
||||
"num_kv_heads",
|
||||
"head_dim",
|
||||
"time_ms",
|
||||
"bandwidth_GB_s",
|
||||
"tflops",
|
||||
]
|
||||
|
||||
|
||||
def success_row(**overrides):
|
||||
row = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"api": "PagedWrapper",
|
||||
"batch_size": "1",
|
||||
"seq_len": "1024",
|
||||
"num_qo_heads": "32",
|
||||
"num_kv_heads": "4",
|
||||
"head_dim": "128",
|
||||
"status": STATUS_OK,
|
||||
"time_ms": "2",
|
||||
"bandwidth_GB_s": "10",
|
||||
"tflops": "20",
|
||||
"error_type": "",
|
||||
"error": "",
|
||||
}
|
||||
row.update({key: str(value) for key, value in overrides.items()})
|
||||
return row
|
||||
|
||||
|
||||
def failure_row(**overrides):
|
||||
row = success_row(
|
||||
status=STATUS_FAILED,
|
||||
time_ms="",
|
||||
bandwidth_GB_s="",
|
||||
tflops="",
|
||||
error_type="RuntimeError",
|
||||
error="benchmark failed",
|
||||
)
|
||||
row.update({key: str(value) for key, value in overrides.items()})
|
||||
return row
|
||||
|
||||
|
||||
class BenchmarkResultTests(unittest.TestCase):
|
||||
def test_successful_case_computes_metrics(self):
|
||||
record = execute_benchmark_case(
|
||||
"Demo", {"batch_size": 1}, lambda: (2.0, 20_000_000, 4_000_000_000)
|
||||
)
|
||||
|
||||
self.assertEqual(record["status"], STATUS_OK)
|
||||
self.assertEqual(record["time_ms"], 2.0)
|
||||
self.assertEqual(record["bandwidth_GB_s"], 10.0)
|
||||
self.assertEqual(record["tflops"], 2.0)
|
||||
|
||||
def test_exception_and_invalid_metrics_become_failures(self):
|
||||
def raise_error():
|
||||
raise RuntimeError("broken")
|
||||
|
||||
for callback, expected_error in (
|
||||
(raise_error, "RuntimeError"),
|
||||
(lambda: (0.0, 1, 1), "ValueError"),
|
||||
(lambda: (math.nan, 1, 1), "ValueError"),
|
||||
(lambda: (1.0, math.inf, 1), "ValueError"),
|
||||
):
|
||||
with self.subTest(expected_error=expected_error):
|
||||
record = execute_benchmark_case(
|
||||
"Demo", {"batch_size": 1}, callback
|
||||
)
|
||||
self.assertEqual(record["status"], STATUS_FAILED)
|
||||
self.assertEqual(record["error_type"], expected_error)
|
||||
|
||||
def test_keyboard_interrupt_is_not_swallowed(self):
|
||||
def interrupt():
|
||||
raise KeyboardInterrupt()
|
||||
|
||||
with self.assertRaises(KeyboardInterrupt):
|
||||
execute_benchmark_case("Demo", {"batch_size": 1}, interrupt)
|
||||
|
||||
|
||||
class SummarizeResultsTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temporary_directory = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.temporary_directory.name)
|
||||
|
||||
def tearDown(self):
|
||||
self.temporary_directory.cleanup()
|
||||
|
||||
def write_csv(self, name, fieldnames, rows, encoding="utf-8"):
|
||||
path = self.root / name
|
||||
with path.open("w", encoding=encoding, newline="") as csv_file:
|
||||
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
return path
|
||||
|
||||
def run_summary(self, paths, output_name="summary.md"):
|
||||
output = self.root / output_name
|
||||
argv = [*(str(path) for path in paths), "--output", str(output)]
|
||||
with redirect_stdout(io.StringIO()):
|
||||
exit_code = summarize_results.main(argv)
|
||||
return exit_code, output.read_text(encoding="utf-8")
|
||||
|
||||
def run_regression(
|
||||
self, baseline, candidate, threshold=None, output_name="regression.md"
|
||||
):
|
||||
output = self.root / output_name
|
||||
argv = [
|
||||
"--baseline",
|
||||
*(str(path) for path in baseline),
|
||||
"--candidate",
|
||||
*(str(path) for path in candidate),
|
||||
]
|
||||
if threshold is not None:
|
||||
argv.extend(("--fail-on-regression", str(threshold)))
|
||||
argv.extend(("--output", str(output)))
|
||||
with redirect_stdout(io.StringIO()):
|
||||
exit_code = summarize_results.main(argv)
|
||||
return exit_code, output.read_text(encoding="utf-8")
|
||||
|
||||
def test_failed_metrics_are_not_selected_and_markdown_is_escaped(self):
|
||||
path = self.write_csv(
|
||||
"mixed.csv",
|
||||
V1_FIELDS,
|
||||
[
|
||||
success_row(),
|
||||
failure_row(
|
||||
batch_size=2,
|
||||
time_ms=0,
|
||||
bandwidth_GB_s=9999,
|
||||
tflops=9999,
|
||||
error="bad | value\n## forged section",
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
exit_code, report = self.run_summary([path])
|
||||
|
||||
self.assertEqual(exit_code, summarize_results.EXIT_BENCHMARK_FAILURE)
|
||||
self.assertIn("Minimum time_ms: 2", report)
|
||||
self.assertNotIn("Minimum time_ms: 0", report)
|
||||
self.assertIn("bad \\| value<br>\\#\\# forged section", report)
|
||||
self.assertIn(summarize_results.escape_markdown(path), report)
|
||||
|
||||
def test_all_failures_are_included(self):
|
||||
rows = [failure_row(batch_size=index, error=f"error-{index}") for index in range(1, 7)]
|
||||
path = self.write_csv("failures.csv", V1_FIELDS, rows)
|
||||
|
||||
exit_code, report = self.run_summary([path])
|
||||
|
||||
self.assertEqual(exit_code, summarize_results.EXIT_BENCHMARK_FAILURE)
|
||||
for index in range(1, 7):
|
||||
self.assertIn(f"error-{index}", report)
|
||||
self.assertNotIn("omitted", report)
|
||||
|
||||
def test_legacy_bom_csv_is_accepted_and_labeled(self):
|
||||
legacy_row = {
|
||||
key: value
|
||||
for key, value in success_row().items()
|
||||
if key in LEGACY_FIELDS
|
||||
}
|
||||
path = self.write_csv(
|
||||
"legacy.csv", LEGACY_FIELDS, [legacy_row], encoding="utf-8-sig"
|
||||
)
|
||||
|
||||
exit_code, report = self.run_summary([path])
|
||||
|
||||
self.assertEqual(exit_code, summarize_results.EXIT_OK)
|
||||
self.assertIn("legacy\\_inferred\\_ok=1", report)
|
||||
self.assertIn("Compatibility Note", report)
|
||||
|
||||
def test_invalid_numeric_empty_and_unrelated_csvs_return_input_error(self):
|
||||
invalid_numeric = self.write_csv(
|
||||
"nan.csv", V1_FIELDS, [success_row(time_ms="NaN")]
|
||||
)
|
||||
empty = self.write_csv("empty.csv", V1_FIELDS, [])
|
||||
unrelated = self.write_csv(
|
||||
"unrelated.csv", ["owner", "note"], [{"owner": "alice", "note": "x"}]
|
||||
)
|
||||
|
||||
exit_code, report = self.run_summary(
|
||||
[invalid_numeric, empty, unrelated]
|
||||
)
|
||||
|
||||
self.assertEqual(exit_code, summarize_results.EXIT_INPUT_ERROR)
|
||||
self.assertIn("Input files: 3", report)
|
||||
self.assertIn("Valid files: 0", report)
|
||||
self.assertIn("must be a finite number", report)
|
||||
self.assertIn("contains no benchmark rows", report)
|
||||
self.assertIn("known legacy FlashInfer schema", report)
|
||||
|
||||
def test_missing_duplicate_headers_and_unknown_schema_are_rejected(self):
|
||||
missing = self.write_csv(
|
||||
"missing.csv",
|
||||
[field for field in V1_FIELDS if field != "error"],
|
||||
[],
|
||||
)
|
||||
duplicate = self.root / "duplicate.csv"
|
||||
duplicate.write_text(
|
||||
"api,api,time_ms,bandwidth_GB_s,tflops\nA,A,1,2,3\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
unknown = self.write_csv(
|
||||
"unknown.csv", V1_FIELDS, [success_row(schema_version=2)]
|
||||
)
|
||||
|
||||
exit_code, report = self.run_summary([missing, duplicate, unknown])
|
||||
|
||||
self.assertEqual(exit_code, summarize_results.EXIT_INPUT_ERROR)
|
||||
self.assertIn("missing columns: error", report)
|
||||
self.assertIn("duplicate header names", report)
|
||||
self.assertIn("unsupported schema\\_version", report)
|
||||
|
||||
def test_regression_is_report_only_by_default_and_gate_is_optional(self):
|
||||
baseline = self.write_csv(
|
||||
"baseline.csv", V1_FIELDS, [success_row(time_ms=10, bandwidth_GB_s=100, tflops=50)]
|
||||
)
|
||||
candidate = self.write_csv(
|
||||
"candidate.csv", V1_FIELDS, [success_row(time_ms=11, bandwidth_GB_s=95, tflops=45)]
|
||||
)
|
||||
|
||||
exit_code, report = self.run_regression([baseline], [candidate])
|
||||
gated_code, gated_report = self.run_regression(
|
||||
[baseline], [candidate], threshold=5, output_name="gated.md"
|
||||
)
|
||||
|
||||
self.assertEqual(exit_code, summarize_results.EXIT_OK)
|
||||
self.assertIn("report only", report)
|
||||
self.assertIn("+10.00%", report)
|
||||
self.assertEqual(gated_code, summarize_results.EXIT_BENCHMARK_FAILURE)
|
||||
self.assertIn("regression", gated_report)
|
||||
|
||||
def test_zero_baseline_throughput_is_reported_without_division_error(self):
|
||||
baseline = self.write_csv(
|
||||
"baseline.csv",
|
||||
V1_FIELDS,
|
||||
[success_row(bandwidth_GB_s=0, tflops=0)],
|
||||
)
|
||||
candidate = self.write_csv(
|
||||
"candidate.csv",
|
||||
V1_FIELDS,
|
||||
[success_row(bandwidth_GB_s=10, tflops=20)],
|
||||
)
|
||||
|
||||
exit_code, report = self.run_regression(
|
||||
[baseline], [candidate], threshold=5
|
||||
)
|
||||
|
||||
self.assertEqual(exit_code, summarize_results.EXIT_OK)
|
||||
self.assertIn("+∞%", report)
|
||||
|
||||
def test_regression_status_transitions_and_case_sets(self):
|
||||
baseline = self.write_csv(
|
||||
"baseline.csv",
|
||||
V1_FIELDS,
|
||||
[
|
||||
success_row(batch_size=1),
|
||||
failure_row(batch_size=2),
|
||||
success_row(batch_size=3),
|
||||
],
|
||||
)
|
||||
candidate = self.write_csv(
|
||||
"candidate.csv",
|
||||
V1_FIELDS,
|
||||
[
|
||||
failure_row(batch_size=1),
|
||||
success_row(batch_size=2),
|
||||
success_row(batch_size=4),
|
||||
],
|
||||
)
|
||||
|
||||
exit_code, report = self.run_regression([baseline], [candidate])
|
||||
|
||||
self.assertEqual(exit_code, summarize_results.EXIT_BENCHMARK_FAILURE)
|
||||
self.assertIn("candidate failed", report)
|
||||
self.assertIn("recovered", report)
|
||||
self.assertIn("Missing Candidate Cases", report)
|
||||
self.assertIn("New Candidate Cases", report)
|
||||
|
||||
def test_both_failed_is_reported_as_candidate_failure(self):
|
||||
baseline = self.write_csv(
|
||||
"baseline.csv", V1_FIELDS, [failure_row(error="old failure")]
|
||||
)
|
||||
candidate = self.write_csv(
|
||||
"candidate.csv", V1_FIELDS, [failure_row(error="new failure")]
|
||||
)
|
||||
|
||||
exit_code, report = self.run_regression([baseline], [candidate])
|
||||
|
||||
self.assertEqual(exit_code, summarize_results.EXIT_BENCHMARK_FAILURE)
|
||||
self.assertIn("both failed", report)
|
||||
self.assertIn("old failure", report)
|
||||
self.assertIn("new failure", report)
|
||||
|
||||
def test_recovery_and_new_successful_case_do_not_fail_comparison(self):
|
||||
baseline = self.write_csv(
|
||||
"baseline.csv", V1_FIELDS, [failure_row(batch_size=1)]
|
||||
)
|
||||
candidate = self.write_csv(
|
||||
"candidate.csv",
|
||||
V1_FIELDS,
|
||||
[success_row(batch_size=1), success_row(batch_size=2)],
|
||||
)
|
||||
|
||||
exit_code, report = self.run_regression([baseline], [candidate])
|
||||
|
||||
self.assertEqual(exit_code, summarize_results.EXIT_OK)
|
||||
self.assertIn("recovered", report)
|
||||
self.assertIn("New Candidate Cases", report)
|
||||
|
||||
def test_duplicate_case_in_a_run_is_an_input_error(self):
|
||||
baseline = self.write_csv(
|
||||
"baseline.csv", V1_FIELDS, [success_row(), success_row()]
|
||||
)
|
||||
candidate = self.write_csv(
|
||||
"candidate.csv", V1_FIELDS, [success_row()]
|
||||
)
|
||||
|
||||
exit_code, report = self.run_regression([baseline], [candidate])
|
||||
|
||||
self.assertEqual(exit_code, summarize_results.EXIT_INPUT_ERROR)
|
||||
self.assertIn("duplicate case", report)
|
||||
|
||||
def test_invalid_cli_combinations_return_exit_two(self):
|
||||
with redirect_stderr(io.StringIO()):
|
||||
with self.assertRaises(SystemExit) as missing_candidate:
|
||||
summarize_results.parse_args(["--baseline", "baseline.csv"])
|
||||
with self.assertRaises(SystemExit) as summary_gate:
|
||||
summarize_results.parse_args(
|
||||
["input.csv", "--fail-on-regression", "5"]
|
||||
)
|
||||
|
||||
self.assertEqual(missing_candidate.exception.code, 2)
|
||||
self.assertEqual(summary_gate.exception.code, 2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -26,7 +26,7 @@
|
|||
基础镜像：
|
||||
|
||||
```plaintext
|
||||
PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.1.5
|
||||
PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.2.1
|
||||
```
|
||||
|
||||
## 二、学习目标
|
||||
|
|
@ -202,7 +202,7 @@ fusedmoe_v2.1
|
|||
本教程使用的镜像是：
|
||||
|
||||
```plaintext
|
||||
PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.1.5
|
||||
PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.2.1
|
||||
```
|
||||
|
||||
这意味着你不需要从零安装 PyTorch、MACA、mxcc 等底层组件。你需要做的是进入镜像、确认环境、放入源码并运行 baseline。
|
||||
|
|
@ -256,7 +256,7 @@ MiniMax-M2.7
|
|||
3. 选择镜像：
|
||||
|
||||
```plaintext
|
||||
PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.1.5
|
||||
PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.2.1
|
||||
```
|
||||
|
||||
4. 点击创建实例。
|
||||
|
|
@ -308,7 +308,7 @@ print("cuda available:", torch.cuda.is_available())
|
|||
PY
|
||||
```
|
||||
|
||||
期望输出：torch: 2.8.0+metax3.7.1.5
|
||||
期望输出：torch: 2.8.0+metax3.7.1.3
|
||||
|
||||
检查 MACA / mxcc：
|
||||
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@
|
|||
|
||||
## 推进流程
|
||||
|
||||
**第一步：获取算力。** 赛事提供曦云 C500 在线算力，无需自备硬件。在沐曦开发者社区领取算力券，于模力方舟平台租用实例，选择镜像 `PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.1.5`。详见 [模力方舟快速使用 SOP](../模力方舟快速使用SOP.md)。
|
||||
**第一步：获取算力。** 赛事提供曦云 C500 在线算力，无需自备硬件。在沐曦开发者社区领取算力券，于模力方舟平台租用实例，选择镜像 `PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.2.1`。详见 [模力方舟快速使用 SOP](../模力方舟快速使用SOP.md)。
|
||||
|
||||
**第二步：部署 Agent。** 推荐安装 OpenCode，通过模力方舟 API 接入 MiniMax-M2.7 模型。部署后执行 `mx-smi` 确认 Agent 可操作当前环境。详见 [模力方舟 Agent 部署准备教程](../基于AI%20Agent开发范式的国产GPU大模型推理算子库优化/模力方舟Agent部署准备教程.md)。
|
||||
|
||||
|
|
@ -78,7 +78,7 @@
|
|||
- 报名：2026 年 5 月 30 日 – 6 月 30 日，[挑战杯官网](https://2026.tiaozhanbei.net/)
|
||||
- 作品提交截止：2026 年 9 月 5 日
|
||||
- 团队上限 10 人，指导教师上限 3 人
|
||||
- 统一开发与评测镜像：`PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.1.5`
|
||||
- 统一开发与评测镜像：`PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.2.1`
|
||||
- Benchmark 须使用整张单卡（64 GB），日常开发建议 16–32 GB
|
||||
- 正确性测试为硬性门槛，未通过的作品不参与性能排名
|
||||
|
||||
|
|
|
|||
|
|
@ -20,9 +20,9 @@
|
|||
|
||||
## 4.创建实例
|
||||
|
||||
使用本赛事专属镜像：PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.1.5。
|
||||
使用本赛事专属镜像：PyTorch-Agent / 2.8.0 / Python 3.12 / maca 3.7.2.1。
|
||||
|
||||
|
||||
|
||||
|
||||
## 5.项目创作
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue