From 8e1f6d900e52d2a45f43da0d32cf652bf59c01c2 Mon Sep 17 00:00:00 2001
From: Col-lin <3078046011@qq.com>
Date: Thu, 30 Jul 2026 14:46:02 +0800
Subject: [PATCH] docs: plan corrected accuracy protocol

---
 .../plans/2026-07-30-accuracy-protocol-fix.md | 445 ++++++++++++++++++
 1 file changed, 445 insertions(+)
 create mode 100644 docs/superpowers/plans/2026-07-30-accuracy-protocol-fix.md

diff --git a/docs/superpowers/plans/2026-07-30-accuracy-protocol-fix.md b/docs/superpowers/plans/2026-07-30-accuracy-protocol-fix.md
new file mode 100644
index 00000000..a6cf1cab
--- /dev/null
+++ b/docs/superpowers/plans/2026-07-30-accuracy-protocol-fix.md
@@ -0,0 +1,445 @@
+# Accuracy Protocol Fix Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Replace the truncating raw-text accuracy runner with one reproducible Qwen chat protocol that produces meaningful GSM8K and GPQA scores.
+
+**Architecture:** Keep dataset loading and offload matrix orchestration unchanged. Add dataset-specific prompt construction, render every prompt through the model tokenizer's chat template, return structured generation records, and make the matrix verifier reject protocol mismatches between cases.
+
+**Tech Stack:** Python 3, `unittest`, vLLM offline `LLM.generate`, Qwen tokenizer chat template, Bash, JSONL.
+
+## Global Constraints
+
+- Produce one formal score set only, named `qwen-chat-v1`.
+- Use `temperature=0` for every case.
+- Use 512 generated tokens for GSM8K and 8 generated tokens for GPQA.
+- Preserve the existing GSM8K and GPQA dataset hashes and seeded GPQA option order.
+- Apply exactly the same prompt and scoring protocol to baseline, `group_8_num_1`, and `group_8_num_2`.
+- Permit zero accuracy drop relative to baseline.
+- Add no new runtime dependency.
+
+## File Structure
+
+- Modify `scripts/accuracy_gsm8k_gpqa.py`: prompt protocol, chat rendering, structured generation records, parsing, and result metadata.
+- Modify `scripts/run_accuracy_gsm8k_gpqa.sh`: formal per-dataset generation budgets.
+- Modify `scripts/verify_accuracy_matrix.py`: cross-case protocol and sampling consistency checks.
+- Modify `tests/model_executor/offloader/test_accuracy_repro.py`: all regression and matrix tests.
+- Modify `docs/v018-deployment-and-reproduction.md`: corrected protocol and final measured scores after the server run.
+- Modify `docs/v018-test-evidence.md`: corrected accuracy evidence after the server run.
+- Modify `docs/finals/决赛项目结项书.md`: replace obsolete 128/16-token score claims with the measured corrected matrix.
+
+---
+
+### Task 1: Dataset Prompts And GPQA Parsing
+
+**Files:**
+- Modify: `tests/model_executor/offloader/test_accuracy_repro.py`
+- Modify: `scripts/accuracy_gsm8k_gpqa.py`
+
+**Interfaces:**
+- Produces: `PROMPT_PROTOCOL = "qwen-chat-v1"`.
+- Produces: `chat_messages_for_prompt(dataset: str, prompt: str) -> list[dict[str, str]]`.
+- Changes: `prompt_for_example(dataset, row, seed) -> tuple[str, str]` to include a dataset-specific answer instruction.
+- Changes: `parse_gpqa_answer(text) -> str | None` to parse only an explicit leading answer.
+
+- [ ] **Step 1: Write failing prompt and parser tests**
+
+Add tests with hand-derived expectations:
+
+```python
+def test_formal_prompts_define_unambiguous_answer_formats(self):
+    accuracy = load_accuracy_module()
+    gsm_prompt, _ = accuracy.prompt_for_example(
+        "gsm8k", {"question": "What is 2 + 3?", "answer": "#### 5"}, "0"
+    )
+    self.assertIn("The answer is <number>.", gsm_prompt)
+    self.assertIn("What is 2 + 3?", gsm_prompt)
+
+    gpqa_row = {
+        "Question": "Which value is correct?",
+        "Correct Answer": "correct",
+        "Incorrect Answer 1": "wrong one",
+        "Incorrect Answer 2": "wrong two",
+        "Incorrect Answer 3": "wrong three",
+    }
+    gpqa_prompt, _ = accuracy.prompt_for_example("gpqa", gpqa_row, "0")
+    self.assertIn("Return only one capital letter", gpqa_prompt)
+    self.assertTrue(gpqa_prompt.rstrip().endswith("Answer:"))
+
+def test_chat_messages_use_dataset_specific_system_roles(self):
+    accuracy = load_accuracy_module()
+    messages = accuracy.chat_messages_for_prompt("gsm8k", "user prompt")
+    self.assertEqual([message["role"] for message in messages], ["system", "user"])
+    self.assertIn("math", messages[0]["content"].lower())
+    self.assertEqual(messages[1]["content"], "user prompt")
+
+def test_gpqa_parser_accepts_only_an_explicit_leading_choice(self):
+    accuracy = load_accuracy_module()
+    for text in ("C", "C. explanation", "Answer: C\nExplanation", "(c)"):
+        self.assertEqual(accuracy.parse_gpqa_answer(text), "C")
+    self.assertIsNone(accuracy.parse_gpqa_answer("I considered C, but chose B."))
+    self.assertIsNone(accuracy.parse_gpqa_answer("Because C is plausible."))
+```
+
+- [ ] **Step 2: Run the focused tests and verify RED**
+
+Run:
+
+```bash
+python -m unittest discover \
+  -s tests/model_executor/offloader \
+  -p test_accuracy_repro.py
+```
+
+Expected: failures because the prompts lack the new instructions,
+`chat_messages_for_prompt` does not exist, and explanatory GPQA answers are rejected.
+
+- [ ] **Step 3: Implement the prompt protocol and anchored parser**
+
+Add exact protocol constants and message construction:
+
+```python
+PROMPT_PROTOCOL = "qwen-chat-v1"
+SYSTEM_MESSAGES = {
+    "gsm8k": "You are a precise math problem solver. Follow the requested answer format.",
+    "gpqa": "You are a precise expert multiple-choice question solver.",
+}
+
+def chat_messages_for_prompt(dataset, prompt):
+    return [
+        {"role": "system", "content": SYSTEM_MESSAGES[dataset]},
+        {"role": "user", "content": prompt},
+    ]
+```
+
+Build GSM8K prompts with a final-line answer instruction and GPQA prompts with
+an answer-only instruction. Replace the full-string GPQA regex with an anchored
+prefix regex using `match`, so later prose cannot supply the answer.
+
+- [ ] **Step 4: Run the focused tests and verify GREEN**
+
+Run the command from Step 2. Expected: all three tests pass.
+
+- [ ] **Step 5: Run the complete accuracy unit test module**
+
+Run:
+
+```bash
+python -m unittest discover \
+  -s tests/model_executor/offloader \
+  -p test_accuracy_repro.py
+```
+
+Expected: all tests pass; any old standalone-only parser assertion is updated to
+the new explicit-leading-answer contract.
+
+- [ ] **Step 6: Commit Task 1**
+
+```bash
+git add scripts/accuracy_gsm8k_gpqa.py tests/model_executor/offloader/test_accuracy_repro.py
+git commit -m "fix: define formal accuracy prompts and parsing"
+```
+
+### Task 2: Chat Rendering And Structured Generation Records
+
+**Files:**
+- Modify: `tests/model_executor/offloader/test_accuracy_repro.py`
+- Modify: `scripts/accuracy_gsm8k_gpqa.py`
+
+**Interfaces:**
+- Changes: `generate_prompts(args, dataset, prompts) -> list[dict[str, object]]`.
+- Each generation record contains `text`, `output_token_count`, and `finish_reason`.
+- Produces: `render_chat_prompts(tokenizer, dataset, prompts) -> list[str]`.
+
+- [ ] **Step 1: Write failing chat rendering and generation record tests**
+
+Use a fake tokenizer only at the external tokenizer boundary:
+
+```python
+class FakeTokenizer:
+    def __init__(self):
+        self.calls = []
+
+    def apply_chat_template(
+        self, messages, *, tokenize, add_generation_prompt
+    ):
+        self.calls.append((messages, tokenize, add_generation_prompt))
+        return "rendered:" + messages[-1]["content"]
+```
+
+Make `FakeLLM.get_tokenizer()` return it and make `FakeLLM.generate()` return a
+complete vLLM-shaped candidate:
+
+```python
+SimpleNamespace(
+    outputs=[
+        SimpleNamespace(
+            text="The answer is 5.",
+            token_ids=[10, 11, 12, 13],
+            finish_reason="stop",
+        )
+    ]
+)
+```
+
+Assert that the real function returns:
+
+```python
+[{
+    "text": "The answer is 5.",
+    "output_token_count": 4,
+    "finish_reason": "stop",
+}]
+```
+
+and that `LLM.generate` receives `["rendered:prompt"]`.
+
+- [ ] **Step 2: Run the focused generation test and verify RED**
+
+Run:
+
+```bash
+python -m unittest discover \
+  -s tests/model_executor/offloader \
+  -p test_accuracy_repro.py
+```
+
+Expected: failure because `generate_prompts` has no dataset argument, does not
+render chat messages, and returns plain text.
+
+- [ ] **Step 3: Implement chat rendering and generation records**
+
+Instantiate `LLM` exactly once, call `llm.get_tokenizer()`, render every prompt
+with:
+
+```python
+tokenizer.apply_chat_template(
+    chat_messages_for_prompt(dataset, prompt),
+    tokenize=False,
+    add_generation_prompt=True,
+)
+```
+
+Raise `RuntimeError("unable to apply chat template for <dataset>")` if rendering
+fails or does not return a non-empty string. Convert each vLLM request output
+into the three-field record. Use an empty record with finish reason `missing`
+when no candidate is returned.
+
+- [ ] **Step 4: Run the focused test and verify GREEN**
+
+Run the command from Step 2. Expected: pass.
+
+- [ ] **Step 5: Run the complete accuracy unit test module**
+
+Run:
+
+```bash
+python -m unittest discover \
+  -s tests/model_executor/offloader \
+  -p test_accuracy_repro.py
+```
+
+Expected: all tests pass.
+
+- [ ] **Step 6: Commit Task 2**
+
+```bash
+git add scripts/accuracy_gsm8k_gpqa.py tests/model_executor/offloader/test_accuracy_repro.py
+git commit -m "fix: apply model chat template during accuracy runs"
+```
+
+### Task 3: Result Diagnostics, Formal Budgets, And Matrix Contracts
+
+**Files:**
+- Modify: `tests/model_executor/offloader/test_accuracy_repro.py`
+- Modify: `scripts/accuracy_gsm8k_gpqa.py`
+- Modify: `scripts/run_accuracy_gsm8k_gpqa.sh`
+- Modify: `scripts/verify_accuracy_matrix.py`
+
+**Interfaces:**
+- Produces: `summarize_generation(records, max_tokens) -> dict[str, object]`.
+- Changes: `build_result_metadata(..., generation_summary)` to include
+  `prompt_protocol` and `generation_summary`.
+- Requires every matrix result to share `prompt_protocol` and
+  `sampling_settings` with its dataset baseline.
+
+- [ ] **Step 1: Write failing result and verifier tests**
+
+Add a generation summary test using literal records:
+
+```python
+records = [
+    {"text": "x", "output_token_count": 512, "finish_reason": "length"},
+    {"text": "y", "output_token_count": 17, "finish_reason": "stop"},
+]
+self.assertEqual(
+    accuracy.summarize_generation(records, 512),
+    {
+        "at_token_limit": 1,
+        "finish_reasons": {"length": 1, "stop": 1},
+        "output_tokens": 529,
+    },
+)
+```
+
+Update matrix fixtures to contain:
+
+```python
+"prompt_protocol": {"name": "qwen-chat-v1", "chat_template": True},
+"sampling_settings": {"temperature": 0, "max_tokens": 512},
+```
+
+Add a verifier test that changes one candidate protocol name to `raw-v0` and
+expects `RuntimeError` containing `prompt protocol`. Add another that changes
+`max_tokens` and expects `RuntimeError` containing `sampling settings`.
+
+- [ ] **Step 2: Run the focused tests and verify RED**
+
+Run:
+
+```bash
+python -m unittest discover \
+  -s tests/model_executor/offloader \
+  -p test_accuracy_repro.py
+```
+
+Expected: failures because summary and cross-case protocol checks do not exist.
+
+- [ ] **Step 3: Persist diagnostics and enforce matrix consistency**
+
+Write `parsed_answer`, `output_token_count`, and `finish_reason` to each example
+JSONL row. Add protocol and summary objects to each result JSON. In
+`verify_matrix`, capture baseline protocol and sampling settings, then reject a
+candidate when either differs.
+
+Change defaults:
+
+```bash
+GSM8K_MAX_TOKENS=${GSM8K_MAX_TOKENS:-512}
+GPQA_MAX_TOKENS=${GPQA_MAX_TOKENS:-8}
+```
+
+and change the Python CLI `--max-tokens` default to `512`.
+
+- [ ] **Step 4: Run the focused tests and verify GREEN**
+
+Run the command from Step 2. Expected: pass.
+
+- [ ] **Step 5: Run all relevant local verification**
+
+Run:
+
+```bash
+python -m unittest discover \
+  -s tests/model_executor/offloader \
+  -p test_accuracy_repro.py
+python -m compileall -q scripts/accuracy_gsm8k_gpqa.py scripts/verify_accuracy_matrix.py
+git diff --check
+```
+
+Expected: all unit tests pass, compilation exits 0, and `git diff --check`
+produces no output.
+
+- [ ] **Step 6: Commit Task 3**
+
+```bash
+git add \
+  scripts/accuracy_gsm8k_gpqa.py \
+  scripts/run_accuracy_gsm8k_gpqa.sh \
+  scripts/verify_accuracy_matrix.py \
+  tests/model_executor/offloader/test_accuracy_repro.py
+git commit -m "test: record and enforce formal accuracy protocol"
+```
+
+### Task 4: Ascend Reproduction And Evidence Refresh
+
+**Files:**
+- Modify: `docs/v018-deployment-and-reproduction.md`
+- Modify: `docs/v018-test-evidence.md`
+- Modify: `docs/finals/决赛项目结项书.md`
+- Produce outside Git: formal server logs and JSON results under a new dated
+  reproduction directory.
+
+**Interfaces:**
+- Consumes: the `qwen-chat-v1` scripts and the existing Qwen2.5-1.5B-Instruct
+  model on the healthy Ascend server.
+- Produces: one full three-case GSM8K/GPQA matrix and updated evidence documents.
+
+- [ ] **Step 1: Deploy the committed branch to the server**
+
+Fetch the branch on the server and verify that its HEAD equals the local
+implementation commit. Do not place credentials in shell history or repository
+files.
+
+- [ ] **Step 2: Run server-side unit tests**
+
+Activate the v0.18 environment and run:
+
+```bash
+python -m unittest discover \
+  -s tests/model_executor/offloader \
+  -p test_accuracy_repro.py
+```
+
+Expected: all tests pass.
+
+- [ ] **Step 3: Run a two-example baseline smoke test**
+
+Invoke the formal runner with `ACCURACY_LIMIT=2` into a dedicated smoke result
+directory. Inspect both result JSON files and confirm:
+
+- `prompt_protocol.name` is `qwen-chat-v1`.
+- GSM8K uses 512 tokens; GPQA uses 8.
+- Example JSONL rows contain parsed answer and generation diagnostics.
+
+- [ ] **Step 4: Run the full formal matrix**
+
+Use the existing full-dataset entry point with `ACCURACY_LIMIT=0` and a new
+dated result directory. Keep `MAX_ACCURACY_DROP=0`.
+
+Expected:
+
+- 1319 GSM8K rows and 198 GPQA Diamond rows per case.
+- Matrix verification reports `all_passed: true`.
+- Candidate accuracies equal their baseline for both datasets.
+- GSM8K no longer has near-total length truncation.
+- GPQA parsed count is nonzero and reflects explicit leading choices.
+
+- [ ] **Step 5: Sync and independently inspect artifacts**
+
+Download the new result directory locally. Recompute hashes for all six example
+JSONL files, inspect result metadata, and calculate exact correct/total and
+length-limit counts from the downloaded artifacts.
+
+- [ ] **Step 6: Refresh evidence documents with measured values**
+
+Replace obsolete `51/1319` and `0/198` claims in the three listed Markdown
+documents with values read from the new formal JSON results. State the exact
+protocol, token budgets, parsed counts, truncation counts, and the relative
+offload accuracy change. Do not modify the PPTX or DOCX in this task.
+
+- [ ] **Step 7: Verify docs and repository state**
+
+Run:
+
+```bash
+rg -n "qwen-chat-v1|GSM8K_MAX_TOKENS|GPQA_MAX_TOKENS|accuracy" \
+  docs/v018-deployment-and-reproduction.md \
+  docs/v018-test-evidence.md \
+  docs/finals/决赛项目结项书.md
+git diff --check
+git status --short
+```
+
+Expected: all three documents cite the corrected protocol and measured results;
+the only uncommitted changes are the intended evidence updates.
+
+- [ ] **Step 8: Commit the refreshed evidence**
+
+```bash
+git add \
+  docs/v018-deployment-and-reproduction.md \
+  docs/v018-test-evidence.md \
+  docs/finals/决赛项目结项书.md
+git commit -m "docs: refresh corrected accuracy evidence"
+```