forked from vllm-ascend/ccf-vllm-ascend
docs: plan corrected accuracy protocol
This commit is contained in:
Col-lin
parent
460c56bafa
commit
8e1f6d900e
|
|
@ -0,0 +1,445 @@
|
|||
# Accuracy Protocol Fix Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace the truncating raw-text accuracy runner with one reproducible Qwen chat protocol that produces meaningful GSM8K and GPQA scores.
|
||||
|
||||
**Architecture:** Keep dataset loading and offload matrix orchestration unchanged. Add dataset-specific prompt construction, render every prompt through the model tokenizer's chat template, return structured generation records, and make the matrix verifier reject protocol mismatches between cases.
|
||||
|
||||
**Tech Stack:** Python 3, `unittest`, vLLM offline `LLM.generate`, Qwen tokenizer chat template, Bash, JSONL.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Produce one formal score set only, named `qwen-chat-v1`.
|
||||
- Use `temperature=0` for every case.
|
||||
- Use 512 generated tokens for GSM8K and 8 generated tokens for GPQA.
|
||||
- Preserve the existing GSM8K and GPQA dataset hashes and seeded GPQA option order.
|
||||
- Apply exactly the same prompt and scoring protocol to baseline, `group_8_num_1`, and `group_8_num_2`.
|
||||
- Permit zero accuracy drop relative to baseline.
|
||||
- Add no new runtime dependency.
|
||||
|
||||
## File Structure
|
||||
|
||||
- Modify `scripts/accuracy_gsm8k_gpqa.py`: prompt protocol, chat rendering, structured generation records, parsing, and result metadata.
|
||||
- Modify `scripts/run_accuracy_gsm8k_gpqa.sh`: formal per-dataset generation budgets.
|
||||
- Modify `scripts/verify_accuracy_matrix.py`: cross-case protocol and sampling consistency checks.
|
||||
- Modify `tests/model_executor/offloader/test_accuracy_repro.py`: all regression and matrix tests.
|
||||
- Modify `docs/v018-deployment-and-reproduction.md`: corrected protocol and final measured scores after the server run.
|
||||
- Modify `docs/v018-test-evidence.md`: corrected accuracy evidence after the server run.
|
||||
- Modify `docs/finals/决赛项目结项书.md`: replace obsolete 128/16-token score claims with the measured corrected matrix.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Dataset Prompts And GPQA Parsing
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/model_executor/offloader/test_accuracy_repro.py`
|
||||
- Modify: `scripts/accuracy_gsm8k_gpqa.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `PROMPT_PROTOCOL = "qwen-chat-v1"`.
|
||||
- Produces: `chat_messages_for_prompt(dataset: str, prompt: str) -> list[dict[str, str]]`.
|
||||
- Changes: `prompt_for_example(dataset, row, seed) -> tuple[str, str]` to include a dataset-specific answer instruction.
|
||||
- Changes: `parse_gpqa_answer(text) -> str | None` to parse only an explicit leading answer.
|
||||
|
||||
- [ ] **Step 1: Write failing prompt and parser tests**
|
||||
|
||||
Add tests with hand-derived expectations:
|
||||
|
||||
```python
|
||||
def test_formal_prompts_define_unambiguous_answer_formats(self):
|
||||
accuracy = load_accuracy_module()
|
||||
gsm_prompt, _ = accuracy.prompt_for_example(
|
||||
"gsm8k", {"question": "What is 2 + 3?", "answer": "#### 5"}, "0"
|
||||
)
|
||||
self.assertIn("The answer is <number>.", gsm_prompt)
|
||||
self.assertIn("What is 2 + 3?", gsm_prompt)
|
||||
|
||||
gpqa_row = {
|
||||
"Question": "Which value is correct?",
|
||||
"Correct Answer": "correct",
|
||||
"Incorrect Answer 1": "wrong one",
|
||||
"Incorrect Answer 2": "wrong two",
|
||||
"Incorrect Answer 3": "wrong three",
|
||||
}
|
||||
gpqa_prompt, _ = accuracy.prompt_for_example("gpqa", gpqa_row, "0")
|
||||
self.assertIn("Return only one capital letter", gpqa_prompt)
|
||||
self.assertTrue(gpqa_prompt.rstrip().endswith("Answer:"))
|
||||
|
||||
def test_chat_messages_use_dataset_specific_system_roles(self):
|
||||
accuracy = load_accuracy_module()
|
||||
messages = accuracy.chat_messages_for_prompt("gsm8k", "user prompt")
|
||||
self.assertEqual([message["role"] for message in messages], ["system", "user"])
|
||||
self.assertIn("math", messages[0]["content"].lower())
|
||||
self.assertEqual(messages[1]["content"], "user prompt")
|
||||
|
||||
def test_gpqa_parser_accepts_only_an_explicit_leading_choice(self):
|
||||
accuracy = load_accuracy_module()
|
||||
for text in ("C", "C. explanation", "Answer: C\nExplanation", "(c)"):
|
||||
self.assertEqual(accuracy.parse_gpqa_answer(text), "C")
|
||||
self.assertIsNone(accuracy.parse_gpqa_answer("I considered C, but chose B."))
|
||||
self.assertIsNone(accuracy.parse_gpqa_answer("Because C is plausible."))
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused tests and verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m unittest discover \
|
||||
-s tests/model_executor/offloader \
|
||||
-p test_accuracy_repro.py
|
||||
```
|
||||
|
||||
Expected: failures because the prompts lack the new instructions,
|
||||
`chat_messages_for_prompt` does not exist, and explanatory GPQA answers are rejected.
|
||||
|
||||
- [ ] **Step 3: Implement the prompt protocol and anchored parser**
|
||||
|
||||
Add exact protocol constants and message construction:
|
||||
|
||||
```python
|
||||
PROMPT_PROTOCOL = "qwen-chat-v1"
|
||||
SYSTEM_MESSAGES = {
|
||||
"gsm8k": "You are a precise math problem solver. Follow the requested answer format.",
|
||||
"gpqa": "You are a precise expert multiple-choice question solver.",
|
||||
}
|
||||
|
||||
def chat_messages_for_prompt(dataset, prompt):
|
||||
return [
|
||||
{"role": "system", "content": SYSTEM_MESSAGES[dataset]},
|
||||
{"role": "user", "content": prompt},
|
||||
]
|
||||
```
|
||||
|
||||
Build GSM8K prompts with a final-line answer instruction and GPQA prompts with
|
||||
an answer-only instruction. Replace the full-string GPQA regex with an anchored
|
||||
prefix regex using `match`, so later prose cannot supply the answer.
|
||||
|
||||
- [ ] **Step 4: Run the focused tests and verify GREEN**
|
||||
|
||||
Run the command from Step 2. Expected: all three tests pass.
|
||||
|
||||
- [ ] **Step 5: Run the complete accuracy unit test module**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m unittest discover \
|
||||
-s tests/model_executor/offloader \
|
||||
-p test_accuracy_repro.py
|
||||
```
|
||||
|
||||
Expected: all tests pass; any old standalone-only parser assertion is updated to
|
||||
the new explicit-leading-answer contract.
|
||||
|
||||
- [ ] **Step 6: Commit Task 1**
|
||||
|
||||
```bash
|
||||
git add scripts/accuracy_gsm8k_gpqa.py tests/model_executor/offloader/test_accuracy_repro.py
|
||||
git commit -m "fix: define formal accuracy prompts and parsing"
|
||||
```
|
||||
|
||||
### Task 2: Chat Rendering And Structured Generation Records
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/model_executor/offloader/test_accuracy_repro.py`
|
||||
- Modify: `scripts/accuracy_gsm8k_gpqa.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Changes: `generate_prompts(args, dataset, prompts) -> list[dict[str, object]]`.
|
||||
- Each generation record contains `text`, `output_token_count`, and `finish_reason`.
|
||||
- Produces: `render_chat_prompts(tokenizer, dataset, prompts) -> list[str]`.
|
||||
|
||||
- [ ] **Step 1: Write failing chat rendering and generation record tests**
|
||||
|
||||
Use a fake tokenizer only at the external tokenizer boundary:
|
||||
|
||||
```python
|
||||
class FakeTokenizer:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def apply_chat_template(
|
||||
self, messages, *, tokenize, add_generation_prompt
|
||||
):
|
||||
self.calls.append((messages, tokenize, add_generation_prompt))
|
||||
return "rendered:" + messages[-1]["content"]
|
||||
```
|
||||
|
||||
Make `FakeLLM.get_tokenizer()` return it and make `FakeLLM.generate()` return a
|
||||
complete vLLM-shaped candidate:
|
||||
|
||||
```python
|
||||
SimpleNamespace(
|
||||
outputs=[
|
||||
SimpleNamespace(
|
||||
text="The answer is 5.",
|
||||
token_ids=[10, 11, 12, 13],
|
||||
finish_reason="stop",
|
||||
)
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
Assert that the real function returns:
|
||||
|
||||
```python
|
||||
[{
|
||||
"text": "The answer is 5.",
|
||||
"output_token_count": 4,
|
||||
"finish_reason": "stop",
|
||||
}]
|
||||
```
|
||||
|
||||
and that `LLM.generate` receives `["rendered:prompt"]`.
|
||||
|
||||
- [ ] **Step 2: Run the focused generation test and verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m unittest discover \
|
||||
-s tests/model_executor/offloader \
|
||||
-p test_accuracy_repro.py
|
||||
```
|
||||
|
||||
Expected: failure because `generate_prompts` has no dataset argument, does not
|
||||
render chat messages, and returns plain text.
|
||||
|
||||
- [ ] **Step 3: Implement chat rendering and generation records**
|
||||
|
||||
Instantiate `LLM` exactly once, call `llm.get_tokenizer()`, render every prompt
|
||||
with:
|
||||
|
||||
```python
|
||||
tokenizer.apply_chat_template(
|
||||
chat_messages_for_prompt(dataset, prompt),
|
||||
tokenize=False,
|
||||
add_generation_prompt=True,
|
||||
)
|
||||
```
|
||||
|
||||
Raise `RuntimeError("unable to apply chat template for <dataset>")` if rendering
|
||||
fails or does not return a non-empty string. Convert each vLLM request output
|
||||
into the three-field record. Use an empty record with finish reason `missing`
|
||||
when no candidate is returned.
|
||||
|
||||
- [ ] **Step 4: Run the focused test and verify GREEN**
|
||||
|
||||
Run the command from Step 2. Expected: pass.
|
||||
|
||||
- [ ] **Step 5: Run the complete accuracy unit test module**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m unittest discover \
|
||||
-s tests/model_executor/offloader \
|
||||
-p test_accuracy_repro.py
|
||||
```
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
- [ ] **Step 6: Commit Task 2**
|
||||
|
||||
```bash
|
||||
git add scripts/accuracy_gsm8k_gpqa.py tests/model_executor/offloader/test_accuracy_repro.py
|
||||
git commit -m "fix: apply model chat template during accuracy runs"
|
||||
```
|
||||
|
||||
### Task 3: Result Diagnostics, Formal Budgets, And Matrix Contracts
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/model_executor/offloader/test_accuracy_repro.py`
|
||||
- Modify: `scripts/accuracy_gsm8k_gpqa.py`
|
||||
- Modify: `scripts/run_accuracy_gsm8k_gpqa.sh`
|
||||
- Modify: `scripts/verify_accuracy_matrix.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `summarize_generation(records, max_tokens) -> dict[str, object]`.
|
||||
- Changes: `build_result_metadata(..., generation_summary)` to include
|
||||
`prompt_protocol` and `generation_summary`.
|
||||
- Requires every matrix result to share `prompt_protocol` and
|
||||
`sampling_settings` with its dataset baseline.
|
||||
|
||||
- [ ] **Step 1: Write failing result and verifier tests**
|
||||
|
||||
Add a generation summary test using literal records:
|
||||
|
||||
```python
|
||||
records = [
|
||||
{"text": "x", "output_token_count": 512, "finish_reason": "length"},
|
||||
{"text": "y", "output_token_count": 17, "finish_reason": "stop"},
|
||||
]
|
||||
self.assertEqual(
|
||||
accuracy.summarize_generation(records, 512),
|
||||
{
|
||||
"at_token_limit": 1,
|
||||
"finish_reasons": {"length": 1, "stop": 1},
|
||||
"output_tokens": 529,
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
Update matrix fixtures to contain:
|
||||
|
||||
```python
|
||||
"prompt_protocol": {"name": "qwen-chat-v1", "chat_template": True},
|
||||
"sampling_settings": {"temperature": 0, "max_tokens": 512},
|
||||
```
|
||||
|
||||
Add a verifier test that changes one candidate protocol name to `raw-v0` and
|
||||
expects `RuntimeError` containing `prompt protocol`. Add another that changes
|
||||
`max_tokens` and expects `RuntimeError` containing `sampling settings`.
|
||||
|
||||
- [ ] **Step 2: Run the focused tests and verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m unittest discover \
|
||||
-s tests/model_executor/offloader \
|
||||
-p test_accuracy_repro.py
|
||||
```
|
||||
|
||||
Expected: failures because summary and cross-case protocol checks do not exist.
|
||||
|
||||
- [ ] **Step 3: Persist diagnostics and enforce matrix consistency**
|
||||
|
||||
Write `parsed_answer`, `output_token_count`, and `finish_reason` to each example
|
||||
JSONL row. Add protocol and summary objects to each result JSON. In
|
||||
`verify_matrix`, capture baseline protocol and sampling settings, then reject a
|
||||
candidate when either differs.
|
||||
|
||||
Change defaults:
|
||||
|
||||
```bash
|
||||
GSM8K_MAX_TOKENS=${GSM8K_MAX_TOKENS:-512}
|
||||
GPQA_MAX_TOKENS=${GPQA_MAX_TOKENS:-8}
|
||||
```
|
||||
|
||||
and change the Python CLI `--max-tokens` default to `512`.
|
||||
|
||||
- [ ] **Step 4: Run the focused tests and verify GREEN**
|
||||
|
||||
Run the command from Step 2. Expected: pass.
|
||||
|
||||
- [ ] **Step 5: Run all relevant local verification**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m unittest discover \
|
||||
-s tests/model_executor/offloader \
|
||||
-p test_accuracy_repro.py
|
||||
python -m compileall -q scripts/accuracy_gsm8k_gpqa.py scripts/verify_accuracy_matrix.py
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: all unit tests pass, compilation exits 0, and `git diff --check`
|
||||
produces no output.
|
||||
|
||||
- [ ] **Step 6: Commit Task 3**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
scripts/accuracy_gsm8k_gpqa.py \
|
||||
scripts/run_accuracy_gsm8k_gpqa.sh \
|
||||
scripts/verify_accuracy_matrix.py \
|
||||
tests/model_executor/offloader/test_accuracy_repro.py
|
||||
git commit -m "test: record and enforce formal accuracy protocol"
|
||||
```
|
||||
|
||||
### Task 4: Ascend Reproduction And Evidence Refresh
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/v018-deployment-and-reproduction.md`
|
||||
- Modify: `docs/v018-test-evidence.md`
|
||||
- Modify: `docs/finals/决赛项目结项书.md`
|
||||
- Produce outside Git: formal server logs and JSON results under a new dated
|
||||
reproduction directory.
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the `qwen-chat-v1` scripts and the existing Qwen2.5-1.5B-Instruct
|
||||
model on the healthy Ascend server.
|
||||
- Produces: one full three-case GSM8K/GPQA matrix and updated evidence documents.
|
||||
|
||||
- [ ] **Step 1: Deploy the committed branch to the server**
|
||||
|
||||
Fetch the branch on the server and verify that its HEAD equals the local
|
||||
implementation commit. Do not place credentials in shell history or repository
|
||||
files.
|
||||
|
||||
- [ ] **Step 2: Run server-side unit tests**
|
||||
|
||||
Activate the v0.18 environment and run:
|
||||
|
||||
```bash
|
||||
python -m unittest discover \
|
||||
-s tests/model_executor/offloader \
|
||||
-p test_accuracy_repro.py
|
||||
```
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
- [ ] **Step 3: Run a two-example baseline smoke test**
|
||||
|
||||
Invoke the formal runner with `ACCURACY_LIMIT=2` into a dedicated smoke result
|
||||
directory. Inspect both result JSON files and confirm:
|
||||
|
||||
- `prompt_protocol.name` is `qwen-chat-v1`.
|
||||
- GSM8K uses 512 tokens; GPQA uses 8.
|
||||
- Example JSONL rows contain parsed answer and generation diagnostics.
|
||||
|
||||
- [ ] **Step 4: Run the full formal matrix**
|
||||
|
||||
Use the existing full-dataset entry point with `ACCURACY_LIMIT=0` and a new
|
||||
dated result directory. Keep `MAX_ACCURACY_DROP=0`.
|
||||
|
||||
Expected:
|
||||
|
||||
- 1319 GSM8K rows and 198 GPQA Diamond rows per case.
|
||||
- Matrix verification reports `all_passed: true`.
|
||||
- Candidate accuracies equal their baseline for both datasets.
|
||||
- GSM8K no longer has near-total length truncation.
|
||||
- GPQA parsed count is nonzero and reflects explicit leading choices.
|
||||
|
||||
- [ ] **Step 5: Sync and independently inspect artifacts**
|
||||
|
||||
Download the new result directory locally. Recompute hashes for all six example
|
||||
JSONL files, inspect result metadata, and calculate exact correct/total and
|
||||
length-limit counts from the downloaded artifacts.
|
||||
|
||||
- [ ] **Step 6: Refresh evidence documents with measured values**
|
||||
|
||||
Replace obsolete `51/1319` and `0/198` claims in the three listed Markdown
|
||||
documents with values read from the new formal JSON results. State the exact
|
||||
protocol, token budgets, parsed counts, truncation counts, and the relative
|
||||
offload accuracy change. Do not modify the PPTX or DOCX in this task.
|
||||
|
||||
- [ ] **Step 7: Verify docs and repository state**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
rg -n "qwen-chat-v1|GSM8K_MAX_TOKENS|GPQA_MAX_TOKENS|accuracy" \
|
||||
docs/v018-deployment-and-reproduction.md \
|
||||
docs/v018-test-evidence.md \
|
||||
docs/finals/决赛项目结项书.md
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
Expected: all three documents cite the corrected protocol and measured results;
|
||||
the only uncommitted changes are the intended evidence updates.
|
||||
|
||||
- [ ] **Step 8: Commit the refreshed evidence**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
docs/v018-deployment-and-reproduction.md \
|
||||
docs/v018-test-evidence.md \
|
||||
docs/finals/决赛项目结项书.md
|
||||
git commit -m "docs: refresh corrected accuracy evidence"
|
||||
```
|
||||
Loading…
Reference in New Issue