Col_lin
/
ccf-vllm-ascend
forked from vllm-ascend/ccf-vllm-ascend
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
ccf-vllm-ascend/docs/v018-deployment-and-reprodu...

8.8 KiB
Raw Blame History

vLLM Ascend v0.18.0 Deployment and Reproduction

Scope

This submission adapts and tests these two parameters on vLLM Ascend v0.18.0:

  • --offload-group-size
  • --offload-num-in-group

The comparison matrix is fixed to three cases:

Case group size num in group
baseline_0_1 0 1
group_8_num_1 8 1
group_8_num_2 8 2

The v0.18.0 base scan did not find an existing implementation of these parameters, so the additional requirement to outperform an upstream implementation is not applicable. Performance scripts still quantify throughput, latency, process RSS, and NPU HBM trade-offs against the baseline.

Verified Software Matrix

Component Version
vLLM 0.18.0+empty
vLLM Ascend source base 72dc68973bd7e6ceef9a122de316a7ee9c9a84aa
CANN 8.5.1
torch 2.9.0+cpu
torch-npu 2.9.0
triton-ascend 3.2.0.dev20260322
Python 3.11

The performance evidence below was measured in the CANN 8.5.1 environment. After server migration, the corrected qwen-chat-v1 accuracy matrix was rerun on CANN 9.0.0 with Python 3.11.6, vLLM 0.18.0+empty, vLLM Ascend 0.1.dev1+g72dc68973.d20260729, torch 2.9.0+cpu, torch-npu 2.9.0.post2, and triton-ascend 3.2.1. All three accuracy cases in that matrix used the same migrated runtime; no cross-CANN performance conclusion is inferred from those results.

Remote paths:

/data/vllm-ascend-v0.18.0
/data/vllm-source-v0.18.0
/data/vllm-ascend-v0.18.0/.venv
/usr/local/Ascend/cann-8.5.1/cann-8.5.1/set_env.sh
/usr/local/Ascend/cann-9.0.0/set_env.sh
/data/models/Qwen2.5-1.5B-Instruct

Runtime Installation

Install the CANN 8.5.1 toolkit, 910B operator package, and NNAL package first. Then rebuild every CANN-bound plugin artifact:

cd /data/vllm-ascend-v0.18.0
bash scripts/install_v018_runtime_compat.sh

The installer pins the Python packages, downloads the exact Triton wheel, clears only generated build artifacts under the guarded repository path, and rebuilds the editable vLLM Ascend package.

Device Mapping and Health

The runtime uses logical device 0. The reproduction scripts derive the physical NPU index from npu-smi info -m before running the health query. After the cloud instance restart, logical device 0 maps to physical NPU 0/chip 0 and the device reports Health Status : OK:

export ASCEND_RT_VISIBLE_DEVICES=0
bash scripts/check_npu_health.sh

Set NPU_INDEX and NPU_CHIP_INDEX only when an explicit override is needed. Do not bypass the health check for formal runs. SKIP_NPU_HEALTH_CHECK=1 exists only for environments whose driver does not implement the health query.

Dataset Evidence

Materialized datasets:

Dataset Rows SHA-256
GSM8K test 1319 32c548f08195e19e33408b844dd7be6aa4bcae457d957bc05909ff4bd4a00595
GPQA Diamond 198 0c16aaa98af019ca691f98eeec9f99ca4cabfb9c920c9e19e40ebedf8d6a0174

Files are under /data/accuracy-gsm8k-gpqa. Each accuracy result records the dataset source, revision, row count, and hash.

Performance Reproduction

cd /data/vllm-ascend-v0.18.0
MODEL_PATH=/data/models/Qwen2.5-1.5B-Instruct \
RESULTS_DIR=/data/v018-performance-results \
REPEATS=3 WARMUP_ROUNDS=1 MEASURED_ROUNDS=3 \
PROMPT_COUNT=8 INPUT_TOKENS=128 MAX_TOKENS=64 MAX_MODEL_LEN=512 \
bash scripts/run_benchmark_all.sh

Artifacts:

performance-runs.jsonl
performance-summary.json
performance-summary.csv
performance-summary.md
performance-run.log

Formal performance result

The formal run at /data/v018-performance-formal-20260721 contains nine raw runs: three independent model loads for each case. The table reports medians.

Case Selected layers Weight storage saved KV cache available Output tok/s Change P50 round latency
baseline_0_1 none 0.0000 GB 27.27 GiB 742.77 0.00% 0.6927 s
group_8_num_1 7, 15, 23 0.2615 GB 27.53 GiB 678.85 -8.61% 0.7501 s
group_8_num_2 6, 7, 14, 15, 22, 23 0.5230 GB 27.79 GiB 383.49 -48.37% 1.3351 s

The two parameters provide a quantifiable memory/throughput trade-off. Do not interpret the post-generation npu-smi delta as weight storage: vLLM consumes newly available HBM for a larger KV cache. The model-loader logs provide the weight-storage and KV-capacity evidence shown above.

Accuracy Reproduction

Full GSM8K and GPQA Diamond:

cd /data/vllm-ascend-v0.18.0
CANN_ENV=/usr/local/Ascend/cann-9.0.0/set_env.sh \
VENV_ACTIVATE=/data/vllm-ascend-v0.18.0/.venv/bin/activate \
MODEL_PATH=/data/models/Qwen2.5-1.5B-Instruct \
DATA_DIR=/data/accuracy-gsm8k-gpqa \
RESULTS_DIR=/data/v018-accuracy-results \
ACCURACY_LIMIT=0 \
bash scripts/run_accuracy_gsm8k_gpqa.sh

For a plumbing smoke test, set ACCURACY_LIMIT=2; the runner then passes --allow-limited to the verifier. Limited mode relaxes only the formal row count and is not submission accuracy evidence.

The formal entry point uses the qwen-chat-v1 protocol: both datasets are rendered with the model tokenizer's Qwen chat template, GSM8K requests a final line in the form The answer is <number>., and GPQA requests one leading A/B/C/D choice. The default generation budgets are 512 tokens for GSM8K and 8 tokens for GPQA.

After all six dataset/case runs, verify_accuracy_matrix.py checks dataset hashes, sample counts, claimed parameter settings, prompt protocol, sampling settings, seed, model, data source, and accuracy against the baseline. Its default formal mode additionally requires the fixed 1319/198 row counts, known dataset hashes, exact qwen-chat-v1 metadata, temperature 0, seed 0, and 512/8-token budgets. MAX_ACCURACY_DROP=0 requires zero accuracy loss. It writes:

accuracy-summary.json
accuracy-summary.csv
accuracy-summary.md

Formal accuracy result

The corrected full run at /data/reproduction-results/accuracy-fixed-bc33c33-formal-20260730 used temperature 0 and seed 0. No sample limit was applied.

Dataset Samples Baseline group_8_num_1 group_8_num_2 Accuracy change
GSM8K test 1319 953/1319 (72.2517%) 953/1319 (72.2517%) 953/1319 (72.2517%) 0.0000
GPQA Diamond 198 47/198 (23.7374%) 47/198 (23.7374%) 47/198 (23.7374%) 0.0000

accuracy-summary.json reports verification_mode: formal and all_passed: true. The generation used commit bc33c33; commit e2cd297 tightened the parser and formal verifier, then revalidated the saved outputs. All 198 GPQA predictions per case are literal single letters, so the parser hardening does not change the measured score. In addition, the three GSM8K example JSONL files have the same SHA-256 79f223c71ae7265bd37d7fc79a582f3b896823ba8a0256b52cbe370df9267c20, and the three GPQA example files have the same SHA-256 0f403ae99640dde7c6c27bd993da9acc62752b87a1f81a655ede98de058914aa. Thus both claimed parameter cases reproduce the baseline output byte-for-byte.

All 1319 GSM8K outputs and all 198 GPQA outputs were parseable in every case. Only 12 GSM8K outputs per case reached the 512-token limit, compared with the obsolete raw-text protocol's near-total truncation; no GPQA output reached its 8-token limit. The scores characterize this fixed 1.5B model and protocol and are not a general claim about model quality.

The downloaded archive reproduction-results-20260730/accuracy-fixed-bc33c33-formal-20260730-e2cd297.tar.gz is kept outside the Git worktree. Its SHA-256 is b95b3c0ccc5b30ab166c058ba5e3fb7ca14e91197f35f514486b177512ca571f. The archive includes formal-run-manifest.json with generation and verification commits, commands, runtime versions, formal settings, dataset hashes, complete model/tokenizer hashes, artifact hashes, and device state. It also includes the strict summary and the 78-test JUnit XML report.

Unified Reproduction

cd /data/vllm-ascend-v0.18.0
MODEL_PATH=/data/models/Qwen2.5-1.5B-Instruct \
DATA_DIR=/data/accuracy-gsm8k-gpqa \
REPRO_RESULTS_DIR=/data/v018-reproduction-results \
bash scripts/run_reproduce_all.sh

The unified script runs performance before accuracy and writes reproduction-manifest.json with commands, package versions, source commit, model path, and artifact locations.

Verified Server State

The July 21 performance environment mapped logical device 0 to physical NPU 0. On the migrated server used for the corrected July 30 accuracy run, logical device 0 mapped to physical NPU 6/chip 0. The device reported Health: OK, and npu-smi info reported no running NPU process after the matrix completed.

The environment contains an unrelated optional msserviceprofiler entry point whose module is absent. vLLM logs that plugin-load error and continues; Ascend platform activation, model execution, result generation, and shutdown all complete successfully.