ccf-vllm-ascend/docs/v018-deployment-and-reproduction.md

# vLLM Ascend v0.18.0 Deployment and Reproduction

## Scope

This submission adapts and tests these two parameters on vLLM Ascend v0.18.0:

- `--offload-group-size`
- `--offload-num-in-group`

The comparison matrix is fixed to three cases:

| Case | group size | num in group |
| --- | ---: | ---: |
| `baseline_0_1` | 0 | 1 |
| `group_8_num_1` | 8 | 1 |
| `group_8_num_2` | 8 | 2 |

The v0.18.0 base scan did not find an existing implementation of these parameters, so the additional requirement to outperform an upstream implementation is not applicable. Performance scripts still quantify throughput, latency, process RSS, and NPU HBM trade-offs against the baseline.

## Verified Software Matrix

| Component | Version |
| --- | --- |
| vLLM | `0.18.0+empty` |
| vLLM Ascend source base | `72dc68973bd7e6ceef9a122de316a7ee9c9a84aa` |
| CANN | `8.5.1` |
| torch | `2.9.0+cpu` |
| torch-npu | `2.9.0` |
| triton-ascend | `3.2.0.dev20260322` |
| Python | `3.11` |

The performance evidence below was measured in the CANN 8.5.1 environment.
After server migration, the corrected `qwen-chat-v1` accuracy matrix was rerun
on CANN 9.0.0 with Python 3.11.6, vLLM 0.18.0+empty,
vLLM Ascend `0.1.dev1+g72dc68973.d20260729`, torch 2.9.0+cpu,
torch-npu 2.9.0.post2, and triton-ascend 3.2.1. All three accuracy cases in
that matrix used the same migrated runtime; no cross-CANN performance
conclusion is inferred from those results.

Remote paths:

```text
/data/vllm-ascend-v0.18.0
/data/vllm-source-v0.18.0
/data/vllm-ascend-v0.18.0/.venv
/usr/local/Ascend/cann-8.5.1/cann-8.5.1/set_env.sh
/usr/local/Ascend/cann-9.0.0/set_env.sh
/data/models/Qwen2.5-1.5B-Instruct
```

## Runtime Installation

Install the CANN 8.5.1 toolkit, 910B operator package, and NNAL package first. Then rebuild every CANN-bound plugin artifact:

```bash
cd /data/vllm-ascend-v0.18.0
bash scripts/install_v018_runtime_compat.sh
```

The installer pins the Python packages, downloads the exact Triton wheel, clears only generated build artifacts under the guarded repository path, and rebuilds the editable vLLM Ascend package.

## Device Mapping and Health

The runtime uses logical device 0. The reproduction scripts derive the physical
NPU index from `npu-smi info -m` before running the health query. After the
cloud instance restart, logical device 0 maps to physical NPU 0/chip 0 and the
device reports `Health Status : OK`:

```bash
export ASCEND_RT_VISIBLE_DEVICES=0
bash scripts/check_npu_health.sh
```

Set `NPU_INDEX` and `NPU_CHIP_INDEX` only when an explicit override is needed.
Do not bypass the health check for formal runs. `SKIP_NPU_HEALTH_CHECK=1`
exists only for environments whose driver does not implement the health query.

## Dataset Evidence

Materialized datasets:

| Dataset | Rows | SHA-256 |
| --- | ---: | --- |
| GSM8K test | 1319 | `32c548f08195e19e33408b844dd7be6aa4bcae457d957bc05909ff4bd4a00595` |
| GPQA Diamond | 198 | `0c16aaa98af019ca691f98eeec9f99ca4cabfb9c920c9e19e40ebedf8d6a0174` |

Files are under `/data/accuracy-gsm8k-gpqa`. Each accuracy result records the dataset source, revision, row count, and hash.

## Performance Reproduction

```bash
cd /data/vllm-ascend-v0.18.0
MODEL_PATH=/data/models/Qwen2.5-1.5B-Instruct \
RESULTS_DIR=/data/v018-performance-results \
REPEATS=3 WARMUP_ROUNDS=1 MEASURED_ROUNDS=3 \
PROMPT_COUNT=8 INPUT_TOKENS=128 MAX_TOKENS=64 MAX_MODEL_LEN=512 \
bash scripts/run_benchmark_all.sh
```

Artifacts:

```text
performance-runs.jsonl
performance-summary.json
performance-summary.csv
performance-summary.md
performance-run.log
```

### Formal performance result

The formal run at `/data/v018-performance-formal-20260721` contains nine raw
runs: three independent model loads for each case. The table reports medians.

| Case | Selected layers | Weight storage saved | KV cache available | Output tok/s | Change | P50 round latency |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| `baseline_0_1` | none | 0.0000 GB | 27.27 GiB | 742.77 | 0.00% | 0.6927 s |
| `group_8_num_1` | 7, 15, 23 | 0.2615 GB | 27.53 GiB | 678.85 | -8.61% | 0.7501 s |
| `group_8_num_2` | 6, 7, 14, 15, 22, 23 | 0.5230 GB | 27.79 GiB | 383.49 | -48.37% | 1.3351 s |

The two parameters provide a quantifiable memory/throughput trade-off. Do not
interpret the post-generation `npu-smi` delta as weight storage: vLLM consumes
newly available HBM for a larger KV cache. The model-loader logs provide the
weight-storage and KV-capacity evidence shown above.

## Accuracy Reproduction

Full GSM8K and GPQA Diamond:

```bash
cd /data/vllm-ascend-v0.18.0
CANN_ENV=/usr/local/Ascend/cann-9.0.0/set_env.sh \
VENV_ACTIVATE=/data/vllm-ascend-v0.18.0/.venv/bin/activate \
MODEL_PATH=/data/models/Qwen2.5-1.5B-Instruct \
DATA_DIR=/data/accuracy-gsm8k-gpqa \
RESULTS_DIR=/data/v018-accuracy-results \
ACCURACY_LIMIT=0 \
bash scripts/run_accuracy_gsm8k_gpqa.sh
```

For a plumbing smoke test, set `ACCURACY_LIMIT=2`; the runner then passes
`--allow-limited` to the verifier. Limited mode relaxes only the formal row
count and is not submission accuracy evidence.

The formal entry point uses the `qwen-chat-v1` protocol: both datasets are
rendered with the model tokenizer's Qwen chat template, GSM8K requests a final
line in the form `The answer is <number>.`, and GPQA requests one leading
`A/B/C/D` choice. The default generation budgets are 512 tokens for GSM8K and
8 tokens for GPQA.

After all six dataset/case runs, `verify_accuracy_matrix.py` checks dataset
hashes, sample counts, claimed parameter settings, prompt protocol, sampling
settings, seed, model, data source, and accuracy against the baseline. Its
default formal mode additionally requires the fixed 1319/198 row counts, known
dataset hashes, exact `qwen-chat-v1` metadata, temperature 0, seed 0, and
512/8-token budgets. `MAX_ACCURACY_DROP=0` requires zero accuracy loss. It
writes:

```text
accuracy-summary.json
accuracy-summary.csv
accuracy-summary.md
```

### Formal accuracy result

The corrected full run at
`/data/reproduction-results/accuracy-fixed-bc33c33-formal-20260730` used
temperature 0 and seed 0. No sample limit was applied.

| Dataset | Samples | Baseline | `group_8_num_1` | `group_8_num_2` | Accuracy change |
| --- | ---: | ---: | ---: | ---: | ---: |
| GSM8K test | 1319 | 953/1319 (72.2517%) | 953/1319 (72.2517%) | 953/1319 (72.2517%) | 0.0000 |
| GPQA Diamond | 198 | 47/198 (23.7374%) | 47/198 (23.7374%) | 47/198 (23.7374%) | 0.0000 |

`accuracy-summary.json` reports `verification_mode: formal` and
`all_passed: true`. The generation used commit `bc33c33`; commit `e2cd297`
tightened the parser and formal verifier, then revalidated the saved outputs.
All 198 GPQA predictions per case are literal single letters, so the parser
hardening does not change the measured score. In addition, the three GSM8K
example JSONL files have the same SHA-256
`79f223c71ae7265bd37d7fc79a582f3b896823ba8a0256b52cbe370df9267c20`,
and the three GPQA example files have the same SHA-256
`0f403ae99640dde7c6c27bd993da9acc62752b87a1f81a655ede98de058914aa`.
Thus both claimed parameter cases reproduce the baseline output byte-for-byte.

All 1319 GSM8K outputs and all 198 GPQA outputs were parseable in every case.
Only 12 GSM8K outputs per case reached the 512-token limit, compared with the
obsolete raw-text protocol's near-total truncation; no GPQA output reached its
8-token limit. The scores characterize this fixed 1.5B model and protocol and
are not a general claim about model quality.

The downloaded archive
`reproduction-results-20260730/accuracy-fixed-bc33c33-formal-20260730-e2cd297.tar.gz`
is kept outside the Git worktree. Its SHA-256 is
`b95b3c0ccc5b30ab166c058ba5e3fb7ca14e91197f35f514486b177512ca571f`.
The archive includes `formal-run-manifest.json` with generation and
verification commits, commands, runtime versions, formal settings, dataset
hashes, complete model/tokenizer hashes, artifact hashes, and device state. It
also includes the strict summary and the 78-test JUnit XML report.

## Unified Reproduction

```bash
cd /data/vllm-ascend-v0.18.0
MODEL_PATH=/data/models/Qwen2.5-1.5B-Instruct \
DATA_DIR=/data/accuracy-gsm8k-gpqa \
REPRO_RESULTS_DIR=/data/v018-reproduction-results \
bash scripts/run_reproduce_all.sh
```

The unified script runs performance before accuracy and writes `reproduction-manifest.json` with commands, package versions, source commit, model path, and artifact locations.

## Verified Server State

The July 21 performance environment mapped logical device 0 to physical NPU 0.
On the migrated server used for the corrected July 30 accuracy run, logical
device 0 mapped to physical NPU 6/chip 0. The device reported `Health: OK`, and
`npu-smi info` reported no running NPU process after the matrix completed.

The environment contains an unrelated optional `msserviceprofiler` entry point
whose module is absent. vLLM logs that plugin-load error and continues; Ascend
platform activation, model execution, result generation, and shutdown all
complete successfully.