# Accuracy Protocol Fix Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Replace the truncating raw-text accuracy runner with one reproducible Qwen chat protocol that produces meaningful GSM8K and GPQA scores.

**Architecture:** Keep dataset loading and offload matrix orchestration unchanged. Add dataset-specific prompt construction, render every prompt through the model tokenizer's chat template, return structured generation records, and make the matrix verifier reject protocol mismatches between cases.

**Tech Stack:** Python 3, `unittest`, vLLM offline `LLM.generate`, Qwen tokenizer chat template, Bash, JSONL.

## Global Constraints

- Produce one formal score set only, named `qwen-chat-v1`.
- Use `temperature=0` for every case.
- Use 512 generated tokens for GSM8K and 8 generated tokens for GPQA.
- Preserve the existing GSM8K and GPQA dataset hashes and seeded GPQA option order.
- Apply exactly the same prompt and scoring protocol to baseline, `group_8_num_1`, and `group_8_num_2`.
- Permit zero accuracy drop relative to baseline.
- Add no new runtime dependency.

## File Structure

- Modify `scripts/accuracy_gsm8k_gpqa.py`: prompt protocol, chat rendering, structured generation records, parsing, and result metadata.
- Modify `scripts/run_accuracy_gsm8k_gpqa.sh`: formal per-dataset generation budgets.
- Modify `scripts/verify_accuracy_matrix.py`: cross-case protocol and sampling consistency checks.
- Modify `tests/model_executor/offloader/test_accuracy_repro.py`: all regression and matrix tests.
- Modify `docs/v018-deployment-and-reproduction.md`: corrected protocol and final measured scores after the server run.
- Modify `docs/v018-test-evidence.md`: corrected accuracy evidence after the server run.
- Modify `docs/finals/决赛项目结项书.md`: replace obsolete 128/16-token score claims with the measured corrected matrix.

---

### Task 1: Dataset Prompts And GPQA Parsing

**Files:**
- Modify: `tests/model_executor/offloader/test_accuracy_repro.py`
- Modify: `scripts/accuracy_gsm8k_gpqa.py`

**Interfaces:**
- Produces: `PROMPT_PROTOCOL = "qwen-chat-v1"`.
- Produces: `chat_messages_for_prompt(dataset: str, prompt: str) -> list[dict[str, str]]`.
- Changes: `prompt_for_example(dataset, row, seed) -> tuple[str, str]` to include a dataset-specific answer instruction.
- Changes: `parse_gpqa_answer(text) -> str | None` to parse only an explicit leading answer.

- [ ] **Step 1: Write failing prompt and parser tests**

Add tests with hand-derived expectations:

```python
def test_formal_prompts_define_unambiguous_answer_formats(self):
    accuracy = load_accuracy_module()
    gsm_prompt, _ = accuracy.prompt_for_example(
        "gsm8k", {"question": "What is 2 + 3?", "answer": "#### 5"}, "0"
    )
    self.assertIn("The answer is <number>.", gsm_prompt)
    self.assertIn("What is 2 + 3?", gsm_prompt)

    gpqa_row = {
        "Question": "Which value is correct?",
        "Correct Answer": "correct",
        "Incorrect Answer 1": "wrong one",
        "Incorrect Answer 2": "wrong two",
        "Incorrect Answer 3": "wrong three",
    }
    gpqa_prompt, _ = accuracy.prompt_for_example("gpqa", gpqa_row, "0")
    self.assertIn("Return only one capital letter", gpqa_prompt)
    self.assertTrue(gpqa_prompt.rstrip().endswith("Answer:"))

def test_chat_messages_use_dataset_specific_system_roles(self):
    accuracy = load_accuracy_module()
    messages = accuracy.chat_messages_for_prompt("gsm8k", "user prompt")
    self.assertEqual([message["role"] for message in messages], ["system", "user"])
    self.assertIn("math", messages[0]["content"].lower())
    self.assertEqual(messages[1]["content"], "user prompt")

def test_gpqa_parser_accepts_only_an_explicit_leading_choice(self):
    accuracy = load_accuracy_module()
    for text in ("C", "C. explanation", "Answer: C\nExplanation", "(c)"):
        self.assertEqual(accuracy.parse_gpqa_answer(text), "C")
    self.assertIsNone(accuracy.parse_gpqa_answer("I considered C, but chose B."))
    self.assertIsNone(accuracy.parse_gpqa_answer("Because C is plausible."))
```

- [ ] **Step 2: Run the focused tests and verify RED**

Run:

```bash
python -m unittest discover \
  -s tests/model_executor/offloader \
  -p test_accuracy_repro.py
```

Expected: failures because the prompts lack the new instructions,
`chat_messages_for_prompt` does not exist, and explanatory GPQA answers are rejected.

- [ ] **Step 3: Implement the prompt protocol and anchored parser**

Add exact protocol constants and message construction:

```python
PROMPT_PROTOCOL = "qwen-chat-v1"
SYSTEM_MESSAGES = {
    "gsm8k": "You are a precise math problem solver. Follow the requested answer format.",
    "gpqa": "You are a precise expert multiple-choice question solver.",
}

def chat_messages_for_prompt(dataset, prompt):
    return [
        {"role": "system", "content": SYSTEM_MESSAGES[dataset]},
        {"role": "user", "content": prompt},
    ]
```

Build GSM8K prompts with a final-line answer instruction and GPQA prompts with
an answer-only instruction. Replace the full-string GPQA regex with an anchored
prefix regex using `match`, so later prose cannot supply the answer.

- [ ] **Step 4: Run the focused tests and verify GREEN**

Run the command from Step 2. Expected: all three tests pass.

- [ ] **Step 5: Run the complete accuracy unit test module**

Run:

```bash
python -m unittest discover \
  -s tests/model_executor/offloader \
  -p test_accuracy_repro.py
```

Expected: all tests pass; any old standalone-only parser assertion is updated to
the new explicit-leading-answer contract.

- [ ] **Step 6: Commit Task 1**

```bash
git add scripts/accuracy_gsm8k_gpqa.py tests/model_executor/offloader/test_accuracy_repro.py
git commit -m "fix: define formal accuracy prompts and parsing"
```

### Task 2: Chat Rendering And Structured Generation Records

**Files:**
- Modify: `tests/model_executor/offloader/test_accuracy_repro.py`
- Modify: `scripts/accuracy_gsm8k_gpqa.py`

**Interfaces:**
- Changes: `generate_prompts(args, dataset, prompts) -> list[dict[str, object]]`.
- Each generation record contains `text`, `output_token_count`, and `finish_reason`.
- Produces: `render_chat_prompts(tokenizer, dataset, prompts) -> list[str]`.

- [ ] **Step 1: Write failing chat rendering and generation record tests**

Use a fake tokenizer only at the external tokenizer boundary:

```python
class FakeTokenizer:
    def __init__(self):
        self.calls = []

    def apply_chat_template(
        self, messages, *, tokenize, add_generation_prompt
    ):
        self.calls.append((messages, tokenize, add_generation_prompt))
        return "rendered:" + messages[-1]["content"]
```

Make `FakeLLM.get_tokenizer()` return it and make `FakeLLM.generate()` return a
complete vLLM-shaped candidate:

```python
SimpleNamespace(
    outputs=[
        SimpleNamespace(
            text="The answer is 5.",
            token_ids=[10, 11, 12, 13],
            finish_reason="stop",
        )
    ]
)
```

Assert that the real function returns:

```python
[{
    "text": "The answer is 5.",
    "output_token_count": 4,
    "finish_reason": "stop",
}]
```

and that `LLM.generate` receives `["rendered:prompt"]`.

- [ ] **Step 2: Run the focused generation test and verify RED**

Run:

```bash
python -m unittest discover \
  -s tests/model_executor/offloader \
  -p test_accuracy_repro.py
```

Expected: failure because `generate_prompts` has no dataset argument, does not
render chat messages, and returns plain text.

- [ ] **Step 3: Implement chat rendering and generation records**

Instantiate `LLM` exactly once, call `llm.get_tokenizer()`, render every prompt
with:

```python
tokenizer.apply_chat_template(
    chat_messages_for_prompt(dataset, prompt),
    tokenize=False,
    add_generation_prompt=True,
)
```

Raise `RuntimeError("unable to apply chat template for <dataset>")` if rendering
fails or does not return a non-empty string. Convert each vLLM request output
into the three-field record. Use an empty record with finish reason `missing`
when no candidate is returned.

- [ ] **Step 4: Run the focused test and verify GREEN**

Run the command from Step 2. Expected: pass.

- [ ] **Step 5: Run the complete accuracy unit test module**

Run:

```bash
python -m unittest discover \
  -s tests/model_executor/offloader \
  -p test_accuracy_repro.py
```

Expected: all tests pass.

- [ ] **Step 6: Commit Task 2**

```bash
git add scripts/accuracy_gsm8k_gpqa.py tests/model_executor/offloader/test_accuracy_repro.py
git commit -m "fix: apply model chat template during accuracy runs"
```

### Task 3: Result Diagnostics, Formal Budgets, And Matrix Contracts

**Files:**
- Modify: `tests/model_executor/offloader/test_accuracy_repro.py`
- Modify: `scripts/accuracy_gsm8k_gpqa.py`
- Modify: `scripts/run_accuracy_gsm8k_gpqa.sh`
- Modify: `scripts/verify_accuracy_matrix.py`

**Interfaces:**
- Produces: `summarize_generation(records, max_tokens) -> dict[str, object]`.
- Changes: `build_result_metadata(..., generation_summary)` to include
  `prompt_protocol` and `generation_summary`.
- Requires every matrix result to share `prompt_protocol` and
  `sampling_settings` with its dataset baseline.

- [ ] **Step 1: Write failing result and verifier tests**

Add a generation summary test using literal records:

```python
records = [
    {"text": "x", "output_token_count": 512, "finish_reason": "length"},
    {"text": "y", "output_token_count": 17, "finish_reason": "stop"},
]
self.assertEqual(
    accuracy.summarize_generation(records, 512),
    {
        "at_token_limit": 1,
        "finish_reasons": {"length": 1, "stop": 1},
        "output_tokens": 529,
    },
)
```

Update matrix fixtures to contain:

```python
"prompt_protocol": {"name": "qwen-chat-v1", "chat_template": True},
"sampling_settings": {"temperature": 0, "max_tokens": 512},
```

Add a verifier test that changes one candidate protocol name to `raw-v0` and
expects `RuntimeError` containing `prompt protocol`. Add another that changes
`max_tokens` and expects `RuntimeError` containing `sampling settings`.

- [ ] **Step 2: Run the focused tests and verify RED**

Run:

```bash
python -m unittest discover \
  -s tests/model_executor/offloader \
  -p test_accuracy_repro.py
```

Expected: failures because summary and cross-case protocol checks do not exist.

- [ ] **Step 3: Persist diagnostics and enforce matrix consistency**

Write `parsed_answer`, `output_token_count`, and `finish_reason` to each example
JSONL row. Add protocol and summary objects to each result JSON. In
`verify_matrix`, capture baseline protocol and sampling settings, then reject a
candidate when either differs.

Change defaults:

```bash
GSM8K_MAX_TOKENS=${GSM8K_MAX_TOKENS:-512}
GPQA_MAX_TOKENS=${GPQA_MAX_TOKENS:-8}
```

and change the Python CLI `--max-tokens` default to `512`.

- [ ] **Step 4: Run the focused tests and verify GREEN**

Run the command from Step 2. Expected: pass.

- [ ] **Step 5: Run all relevant local verification**

Run:

```bash
python -m unittest discover \
  -s tests/model_executor/offloader \
  -p test_accuracy_repro.py
python -m compileall -q scripts/accuracy_gsm8k_gpqa.py scripts/verify_accuracy_matrix.py
git diff --check
```

Expected: all unit tests pass, compilation exits 0, and `git diff --check`
produces no output.

- [ ] **Step 6: Commit Task 3**

```bash
git add \
  scripts/accuracy_gsm8k_gpqa.py \
  scripts/run_accuracy_gsm8k_gpqa.sh \
  scripts/verify_accuracy_matrix.py \
  tests/model_executor/offloader/test_accuracy_repro.py
git commit -m "test: record and enforce formal accuracy protocol"
```

### Task 4: Ascend Reproduction And Evidence Refresh

**Files:**
- Modify: `docs/v018-deployment-and-reproduction.md`
- Modify: `docs/v018-test-evidence.md`
- Modify: `docs/finals/决赛项目结项书.md`
- Produce outside Git: formal server logs and JSON results under a new dated
  reproduction directory.

**Interfaces:**
- Consumes: the `qwen-chat-v1` scripts and the existing Qwen2.5-1.5B-Instruct
  model on the healthy Ascend server.
- Produces: one full three-case GSM8K/GPQA matrix and updated evidence documents.

- [ ] **Step 1: Deploy the committed branch to the server**

Fetch the branch on the server and verify that its HEAD equals the local
implementation commit. Do not place credentials in shell history or repository
files.

- [ ] **Step 2: Run server-side unit tests**

Activate the v0.18 environment and run:

```bash
python -m unittest discover \
  -s tests/model_executor/offloader \
  -p test_accuracy_repro.py
```

Expected: all tests pass.

- [ ] **Step 3: Run a two-example baseline smoke test**

Invoke the formal runner with `ACCURACY_LIMIT=2` into a dedicated smoke result
directory. Inspect both result JSON files and confirm:

- `prompt_protocol.name` is `qwen-chat-v1`.
- GSM8K uses 512 tokens; GPQA uses 8.
- Example JSONL rows contain parsed answer and generation diagnostics.

- [ ] **Step 4: Run the full formal matrix**

Use the existing full-dataset entry point with `ACCURACY_LIMIT=0` and a new
dated result directory. Keep `MAX_ACCURACY_DROP=0`.

Expected:

- 1319 GSM8K rows and 198 GPQA Diamond rows per case.
- Matrix verification reports `all_passed: true`.
- Candidate accuracies equal their baseline for both datasets.
- GSM8K no longer has near-total length truncation.
- GPQA parsed count is nonzero and reflects explicit leading choices.

- [ ] **Step 5: Sync and independently inspect artifacts**

Download the new result directory locally. Recompute hashes for all six example
JSONL files, inspect result metadata, and calculate exact correct/total and
length-limit counts from the downloaded artifacts.

- [ ] **Step 6: Refresh evidence documents with measured values**

Replace obsolete `51/1319` and `0/198` claims in the three listed Markdown
documents with values read from the new formal JSON results. State the exact
protocol, token budgets, parsed counts, truncation counts, and the relative
offload accuracy change. Do not modify the PPTX or DOCX in this task.

- [ ] **Step 7: Verify docs and repository state**

Run:

```bash
rg -n "qwen-chat-v1|GSM8K_MAX_TOKENS|GPQA_MAX_TOKENS|accuracy" \
  docs/v018-deployment-and-reproduction.md \
  docs/v018-test-evidence.md \
  docs/finals/决赛项目结项书.md
git diff --check
git status --short
```

Expected: all three documents cite the corrected protocol and measured results;
the only uncommitted changes are the intended evidence updates.

- [ ] **Step 8: Commit the refreshed evidence**

```bash
git add \
  docs/v018-deployment-and-reproduction.md \
  docs/v018-test-evidence.md \
  docs/finals/决赛项目结项书.md
git commit -m "docs: refresh corrected accuracy evidence"
```
