forked from vllm-ascend/ccf-vllm-ascend
docs: record strict accuracy verification
This commit is contained in:
Col-lin
parent
e2cd297c5e
commit
26802027a0
|
|
@ -10,6 +10,8 @@
|
|||
|
||||
**优化提交：** `f0ab80524e2088763ab1e9859534f05912850194`
|
||||
|
||||
**精度协议与验收加固提交：** `e2cd297c5e5756e0908707a647251c278027dbbf`
|
||||
|
||||
**结项日期：** 2026 年 7 月
|
||||
|
||||
---
|
||||
|
|
@ -310,7 +312,7 @@ export NPU_CHIP_INDEX=0
|
|||
| `scripts/bench_one.py` | 执行单次推理并采集吞吐、延迟和 HBM 证据 |
|
||||
| `scripts/run_accuracy_gsm8k_gpqa.sh` | 运行 GSM8K 与 GPQA 三组精度矩阵 |
|
||||
| `scripts/accuracy_gsm8k_gpqa.py` | 数据加载、推理、评分和 example JSONL 输出 |
|
||||
| `scripts/verify_accuracy_matrix.py` | 校验样本数、参数、数据集哈希和精度下降 |
|
||||
| `scripts/verify_accuracy_matrix.py` | 强制正式协议、样本数、数据集哈希、模型/seed 一致性和零精度下降 |
|
||||
| `scripts/run_reproduce_all.sh` | 性能后精度的一键复现入口 |
|
||||
| `scripts/check_npu_health.sh` | 正式测试前 NPU 健康门禁 |
|
||||
| `scripts/detect_npu_index.py` | 获取物理 NPU index |
|
||||
|
|
@ -344,14 +346,14 @@ export NPU_CHIP_INDEX=0
|
|||
| triton-ascend | `3.2.0.dev20260322` |
|
||||
| 模型 | `Qwen2.5-1.5B-Instruct` |
|
||||
|
||||
上表对应 7 月 21 日正式性能环境。2026 年 7 月 30 日修复精度协议后，在迁移服务器上使用 CANN 9.0.0、Python 3.11.6、vLLM 0.18.0、torch 2.9.0+cpu、torch-npu 2.9.0.post2 和 Ascend 910B2C 重新执行三组完整精度矩阵。三组 case 使用同一环境，因此精度差值不受跨环境比较影响；本报告不据此推导跨 CANN 版本的性能结论。
|
||||
上表对应 7 月 21 日正式性能环境。2026 年 7 月 30 日修复精度协议后，在迁移服务器上使用 CANN 9.0.0、Python 3.11.6、vLLM 0.18.0+empty、vLLM Ascend `0.1.dev1+g72dc68973.d20260729`、torch 2.9.0+cpu、torch-npu 2.9.0.post2、triton-ascend 3.2.1 和 Ascend 910B2C 重新执行三组完整精度矩阵。三组 case 使用同一环境，因此精度差值不受跨环境比较影响；本报告不据此推导跨 CANN 版本的性能结论。
|
||||
|
||||
## 5.2 功能测试
|
||||
|
||||
精度协议和脚本验收相关 unittest（2026 年 7 月 30 日）：
|
||||
|
||||
```text
|
||||
test_accuracy_repro.py: 24/24 passed
|
||||
test_accuracy_repro.py: 26/26 passed
|
||||
test_*scripts.py: 18/18 passed
|
||||
compileall: passed
|
||||
```
|
||||
|
|
@ -359,10 +361,10 @@ compileall: passed
|
|||
实际 Ascend 环境中的完整 offloader suite：
|
||||
|
||||
```text
|
||||
76 passed in 11.94s
|
||||
78 passed in 11.77s
|
||||
```
|
||||
|
||||
两组 `unittest` 属于不同测试范围，不相加。`test_accuracy_repro.py` 也在服务器单独复跑并通过 24/24。
|
||||
两组 `unittest` 属于不同测试范围，不相加。`test_accuracy_repro.py` 也在服务器单独复跑并通过 26/26。
|
||||
|
||||
## 5.3 性能测试
|
||||
|
||||
|
|
@ -411,9 +413,12 @@ MAX_ACCURACY_DROP=0
|
|||
验证汇总返回：
|
||||
|
||||
```text
|
||||
verification_mode: formal
|
||||
all_passed: true
|
||||
```
|
||||
|
||||
正式模式不再把 baseline 的任意元数据直接视为权威，而是强制 1319/198 样本数、固定数据集 SHA-256、`qwen-chat-v1`、temperature 0、seed 0、512/8-token 预算，并检查三组 case 的 model、seed 和 dataset source 完全一致。`ACCURACY_LIMIT>0` 时脚本才显式启用 `--allow-limited`，且该模式只放宽样本数，不得作为提交成绩。
|
||||
|
||||
材料记录的 example JSONL SHA-256：
|
||||
|
||||
```text
|
||||
|
|
@ -424,7 +429,7 @@ GPQA:
|
|||
0f403ae99640dde7c6c27bd993da9acc62752b87a1f81a655ede98de058914aa
|
||||
```
|
||||
|
||||
三组 case 在每个数据集上记录了相同哈希，说明保存的预测输出逐字节一致。当前 `verify_accuracy_matrix.py` 不自动复算 example JSONL 哈希，因此正式复现时仍应保留原始 JSONL，并通过 `sha256sum` 或等价工具再次核验。
|
||||
三组 case 在每个数据集上记录了相同哈希，说明保存的预测输出逐字节一致。当前 `verify_accuracy_matrix.py` 不自动复算 example JSONL 哈希，因此正式复现时仍应保留原始 JSONL，并通过 `sha256sum` 或等价工具再次核验。本次最终归档的 `formal-run-manifest.json` 已记录两组 example 哈希和其余关键产物哈希。
|
||||
|
||||
生成诊断同样在三组 case 间一致：
|
||||
|
||||
|
|
@ -433,21 +438,23 @@ GPQA:
|
|||
| GSM8K | 1319/1319 | 12/1319 | 266776 |
|
||||
| GPQA Diamond | 198/198 | 0/198 | 396 |
|
||||
|
||||
旧脚本直接向 instruct 模型输入原始文本，GSM8K 默认 128 tokens 导致 1304/1319 个输出被截断；GPQA 又要求整段输出只能是单个字母，带解释的明确答案也会被判为无法解析。修复后统一应用 Qwen chat template，明确答案格式，并仅从 GPQA 输出开头解析选项，避免从正文任意位置搜索字母造成虚高。23.7374% 是该 1.5B 模型在本次固定协议下的 baseline 表现，不能泛化为其他模型或提示协议；矩阵能够证明的是两个 offload case 没有引入额外精度下降。
|
||||
旧脚本直接向 instruct 模型输入原始文本，GSM8K 默认 128 tokens 导致 1304/1319 个输出被截断；GPQA 又要求整段输出只能是单个字母，带解释的明确答案也会被判为无法解析。修复后统一应用 Qwen chat template，明确答案格式，并仅解析独立、带标签或完整括号包围的开头选项；`A careful...`、`C or D` 和未闭合括号等含糊文本均拒绝评分，避免虚高。归档中的三组 GPQA 输出各有 198/198 个纯单字母预测，收紧解析后仍各为 47/198。23.7374% 是该 1.5B 模型在本次固定协议下的 baseline 表现，不能泛化为其他模型或提示协议；矩阵能够证明的是两个 offload case 没有引入额外精度下降。
|
||||
|
||||
## 5.5 精度验收标准
|
||||
|
||||
两个参数 case 通过精度验证必须同时满足：
|
||||
|
||||
1. 数据集名称、行数和 SHA-256 与 baseline 一致。
|
||||
1. 数据集名称、正式行数和 SHA-256 与内置正式合同一致。
|
||||
2. 记录的 `offload_group_size` 与 `offload_num_in_group` 与 case 定义一致。
|
||||
3. 结果中的 `correct/total` 与 accuracy 计算一致。
|
||||
4. 相对 baseline 的 accuracy drop 不大于 0。
|
||||
5. 保存的 example JSONL 哈希与 baseline 一致。
|
||||
3. prompt protocol、temperature、生成预算和 seed 与正式合同一致。
|
||||
4. 三组 case 的 model、seed 和 dataset source 完全一致。
|
||||
5. 结果中的 `correct/total` 与 accuracy 计算一致。
|
||||
6. 相对 baseline 的 accuracy drop 不大于 0。
|
||||
7. 保存的 example JSONL 哈希与 baseline 一致。
|
||||
|
||||
## 5.6 证据限制
|
||||
|
||||
当前 Git 仓库仍未包含体积较大的正式原始归档。7 月 21 日九次性能原始记录仍需从远端归档取得；修复后的六组精度 JSONL、结果 JSON、汇总表和完整日志已下载到 Git 工作区外的 `D:\PDSL\Ascend\reproduction-results-20260730`，归档 SHA-256 为 `90391cb26a48f6a3a80b3b96288ba774c46cb0020196a71a07b09f802b1e296e`。
|
||||
当前 Git 仓库仍未包含体积较大的正式原始归档。7 月 21 日九次性能原始记录仍需从远端归档取得；修复后的六组精度 JSONL、结果 JSON、严格汇总、完整日志、运行清单和测试 XML 已下载到 Git 工作区外的 `D:\PDSL\Ascend\reproduction-results-20260730`。最终归档名为 `accuracy-fixed-bc33c33-formal-20260730-e2cd297.tar.gz`，SHA-256 为 `b95b3c0ccc5b30ab166c058ba5e3fb7ca14e91197f35f514486b177512ca571f`。
|
||||
|
||||
正式评审复现应以一键脚本重新生成结果，或将以下远端归档制作成不可变提交附件：
|
||||
|
||||
|
|
@ -576,7 +583,7 @@ python -m pytest tests/model_executor/offloader -q
|
|||
正式记录：
|
||||
|
||||
```text
|
||||
76 passed in 11.94s
|
||||
78 passed in 11.77s
|
||||
```
|
||||
|
||||
## A.4 性能复现
|
||||
|
|
@ -609,6 +616,8 @@ performance-run.log
|
|||
|
||||
```bash
|
||||
cd /data/vllm-ascend-v0.18.0
|
||||
CANN_ENV=/usr/local/Ascend/cann-9.0.0/set_env.sh \
|
||||
VENV_ACTIVATE=/data/vllm-ascend-v0.18.0/.venv/bin/activate \
|
||||
MODEL_PATH=/data/models/Qwen2.5-1.5B-Instruct \
|
||||
DATA_DIR=/data/accuracy-gsm8k-gpqa \
|
||||
RESULTS_DIR=/data/v018-accuracy-results \
|
||||
|
|
|
|||
|
|
@ -31,9 +31,11 @@ The v0.18.0 base scan did not find an existing implementation of these parameter
|
|||
|
||||
The performance evidence below was measured in the CANN 8.5.1 environment.
|
||||
After server migration, the corrected `qwen-chat-v1` accuracy matrix was rerun
|
||||
on CANN 9.0.0 with Python 3.11.6, vLLM 0.18.0, torch 2.9.0+cpu, and torch-npu
|
||||
2.9.0.post2. All three accuracy cases in that matrix used the same migrated
|
||||
runtime; no cross-CANN performance conclusion is inferred from those results.
|
||||
on CANN 9.0.0 with Python 3.11.6, vLLM 0.18.0+empty,
|
||||
vLLM Ascend `0.1.dev1+g72dc68973.d20260729`, torch 2.9.0+cpu,
|
||||
torch-npu 2.9.0.post2, and triton-ascend 3.2.1. All three accuracy cases in
|
||||
that matrix used the same migrated runtime; no cross-CANN performance
|
||||
conclusion is inferred from those results.
|
||||
|
||||
Remote paths:
|
||||
|
||||
|
|
@ -127,6 +129,8 @@ Full GSM8K and GPQA Diamond:
|
|||
|
||||
```bash
|
||||
cd /data/vllm-ascend-v0.18.0
|
||||
CANN_ENV=/usr/local/Ascend/cann-9.0.0/set_env.sh \
|
||||
VENV_ACTIVATE=/data/vllm-ascend-v0.18.0/.venv/bin/activate \
|
||||
MODEL_PATH=/data/models/Qwen2.5-1.5B-Instruct \
|
||||
DATA_DIR=/data/accuracy-gsm8k-gpqa \
|
||||
RESULTS_DIR=/data/v018-accuracy-results \
|
||||
|
|
@ -134,7 +138,9 @@ ACCURACY_LIMIT=0 \
|
|||
bash scripts/run_accuracy_gsm8k_gpqa.sh
|
||||
```
|
||||
|
||||
For a plumbing smoke test, set `ACCURACY_LIMIT=2`; limited results are not submission accuracy evidence.
|
||||
For a plumbing smoke test, set `ACCURACY_LIMIT=2`; the runner then passes
|
||||
`--allow-limited` to the verifier. Limited mode relaxes only the formal row
|
||||
count and is not submission accuracy evidence.
|
||||
|
||||
The formal entry point uses the `qwen-chat-v1` protocol: both datasets are
|
||||
rendered with the model tokenizer's Qwen chat template, GSM8K requests a final
|
||||
|
|
@ -144,8 +150,11 @@ line in the form `The answer is <number>.`, and GPQA requests one leading
|
|||
|
||||
After all six dataset/case runs, `verify_accuracy_matrix.py` checks dataset
|
||||
hashes, sample counts, claimed parameter settings, prompt protocol, sampling
|
||||
settings, and accuracy against the baseline. The default
|
||||
`MAX_ACCURACY_DROP=0` requires zero accuracy loss. It writes:
|
||||
settings, seed, model, data source, and accuracy against the baseline. Its
|
||||
default formal mode additionally requires the fixed 1319/198 row counts, known
|
||||
dataset hashes, exact `qwen-chat-v1` metadata, temperature 0, seed 0, and
|
||||
512/8-token budgets. `MAX_ACCURACY_DROP=0` requires zero accuracy loss. It
|
||||
writes:
|
||||
|
||||
```text
|
||||
accuracy-summary.json
|
||||
|
|
@ -164,8 +173,12 @@ temperature 0 and seed 0. No sample limit was applied.
|
|||
| GSM8K test | 1319 | 953/1319 (72.2517%) | 953/1319 (72.2517%) | 953/1319 (72.2517%) | 0.0000 |
|
||||
| GPQA Diamond | 198 | 47/198 (23.7374%) | 47/198 (23.7374%) | 47/198 (23.7374%) | 0.0000 |
|
||||
|
||||
`accuracy-summary.json` reports `all_passed: true`. In addition, the three
|
||||
GSM8K example JSONL files have the same SHA-256
|
||||
`accuracy-summary.json` reports `verification_mode: formal` and
|
||||
`all_passed: true`. The generation used commit `bc33c33`; commit `e2cd297`
|
||||
tightened the parser and formal verifier, then revalidated the saved outputs.
|
||||
All 198 GPQA predictions per case are literal single letters, so the parser
|
||||
hardening does not change the measured score. In addition, the three GSM8K
|
||||
example JSONL files have the same SHA-256
|
||||
`79f223c71ae7265bd37d7fc79a582f3b896823ba8a0256b52cbe370df9267c20`,
|
||||
and the three GPQA example files have the same SHA-256
|
||||
`0f403ae99640dde7c6c27bd993da9acc62752b87a1f81a655ede98de058914aa`.
|
||||
|
|
@ -178,9 +191,13 @@ obsolete raw-text protocol's near-total truncation; no GPQA output reached its
|
|||
are not a general claim about model quality.
|
||||
|
||||
The downloaded archive
|
||||
`reproduction-results-20260730/accuracy-fixed-bc33c33-formal-20260730.tar.gz`
|
||||
`reproduction-results-20260730/accuracy-fixed-bc33c33-formal-20260730-e2cd297.tar.gz`
|
||||
is kept outside the Git worktree. Its SHA-256 is
|
||||
`90391cb26a48f6a3a80b3b96288ba774c46cb0020196a71a07b09f802b1e296e`.
|
||||
`b95b3c0ccc5b30ab166c058ba5e3fb7ca14e91197f35f514486b177512ca571f`.
|
||||
The archive includes `formal-run-manifest.json` with generation and
|
||||
verification commits, commands, runtime versions, formal settings, dataset
|
||||
hashes, complete model/tokenizer hashes, artifact hashes, and device state. It
|
||||
also includes the strict summary and the 78-test JUnit XML report.
|
||||
|
||||
## Unified Reproduction
|
||||
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ After the accuracy protocol correction on 2026-07-30, the focused local
|
|||
regression results were:
|
||||
|
||||
```text
|
||||
test_accuracy_repro.py: 24/24 passed
|
||||
test_accuracy_repro.py: 26/26 passed
|
||||
test_*scripts.py: 18/18 passed
|
||||
compileall: passed
|
||||
```
|
||||
|
|
@ -34,12 +34,13 @@ python -m pytest tests/model_executor/offloader -q
|
|||
```
|
||||
|
||||
```text
|
||||
76 passed in 11.94s
|
||||
78 passed in 11.77s
|
||||
```
|
||||
|
||||
## Remote Runtime
|
||||
|
||||
Verified under `/data/vllm-ascend-v0.18.0/.venv`:
|
||||
Historical July 21 performance runtime under
|
||||
`/data/vllm-ascend-v0.18.0/.venv`:
|
||||
|
||||
```text
|
||||
vllm 0.18.0+empty
|
||||
|
|
@ -51,6 +52,18 @@ npu_available True
|
|||
npu_device_count 1
|
||||
```
|
||||
|
||||
The corrected July 30 accuracy runtime was:
|
||||
|
||||
```text
|
||||
Python 3.11.6
|
||||
vllm 0.18.0+empty
|
||||
vllm-ascend 0.1.dev1+g72dc68973.d20260729
|
||||
torch 2.9.0+cpu
|
||||
torch-npu 2.9.0.post2
|
||||
triton-ascend 3.2.1
|
||||
CANN 9.0.0
|
||||
```
|
||||
|
||||
The rebuilt `AddRmsNormBias` custom operator loaded and produced the expected values:
|
||||
|
||||
```text
|
||||
|
|
@ -85,9 +98,12 @@ of 512 tokens for GSM8K and 8 tokens for GPQA.
|
|||
| GSM8K | 1319 | 953 (72.2517%) | 953 (72.2517%) | 953 (72.2517%) |
|
||||
| GPQA Diamond | 198 | 47 (23.7374%) | 47 (23.7374%) | 47 (23.7374%) |
|
||||
|
||||
The verifier returned `all_passed: true` with `MAX_ACCURACY_DROP=0`. The
|
||||
example JSONL hashes are identical across all three cases for each dataset, so
|
||||
the claimed parameter cases reproduce every baseline prediction byte-for-byte.
|
||||
The strict verifier returned `verification_mode: formal` and
|
||||
`all_passed: true` with `MAX_ACCURACY_DROP=0`. It now fixes the formal dataset
|
||||
hashes and row counts, exact protocol and sampling settings, seed 0, and
|
||||
cross-case model/data-source identity. The example JSONL hashes are identical
|
||||
across all three cases for each dataset, so the claimed parameter cases
|
||||
reproduce every baseline prediction byte-for-byte.
|
||||
|
||||
| Dataset | Parsed answers | At token limit | Example JSONL SHA-256 |
|
||||
| --- | ---: | ---: | --- |
|
||||
|
|
@ -97,15 +113,17 @@ the claimed parameter cases reproduce every baseline prediction byte-for-byte.
|
|||
Remote artifacts:
|
||||
`/data/reproduction-results/accuracy-fixed-bc33c33-formal-20260730`.
|
||||
The immutable downloaded archive is
|
||||
`reproduction-results-20260730/accuracy-fixed-bc33c33-formal-20260730.tar.gz`
|
||||
`reproduction-results-20260730/accuracy-fixed-bc33c33-formal-20260730-e2cd297.tar.gz`
|
||||
outside the Git worktree, with SHA-256
|
||||
`90391cb26a48f6a3a80b3b96288ba774c46cb0020196a71a07b09f802b1e296e`.
|
||||
`b95b3c0ccc5b30ab166c058ba5e3fb7ca14e91197f35f514486b177512ca571f`.
|
||||
It includes `formal-run-manifest.json`, the complete model/tokenizer file
|
||||
hashes, the strict summaries, and `offloader-tests-e2cd297.xml`.
|
||||
|
||||
## Final Device Health
|
||||
|
||||
For the corrected accuracy run, logical runtime device 0 mapped to physical NPU
|
||||
6/chip 0. The server used CANN 9.0.0, Python 3.11.6, vLLM 0.18.0,
|
||||
torch 2.9.0+cpu, and torch-npu 2.9.0.post2. After the formal matrix:
|
||||
6/chip 0. The server used CANN 9.0.0 and the corrected runtime listed above.
|
||||
After strict reverification:
|
||||
|
||||
```text
|
||||
NPU 6 910B2C
|
||||
|
|
|
|||
Loading…
Reference in New Issue