## Why
The E2E Council blocked **#12857** (no test PR opened) because the
post-heal **Sentinel re-audit found 2 critical quality issues** — even
though **9/10 tests passed**. The tests were **not worthless**; 2 just
*under-claimed* (the body asserted something weaker than the test name
promised):
1. `"should not persist FTS default across page reload"` — re-searches
after reload and asserts the column visible, which passes whether it
*persisted* or *re-resolved*. Never verifies non-persistence.
2. `"should clear FTS marker and re-resolve for new stream on stream
change"` — only checks "table rendered", never that FTS actually
re-resolved.
Blocking was correct (green ≠ meaningful), but a **dead-end block throws
away the other 90%**. The pipeline should **fix the problem and make it
right**, not give up. A no-PR outcome should be a *true last resort*.
## What — The Refiner (Phase 5.5)
A remediation loop **inside `verify_heal`**, after the Sentinel re-audit
and before the quality gate (**reuses the already-booted binary** — no
rebuild):
- **Only fires when needed.** If the re-audit is clean (PASS / 0
critical), the step reads the verdict, sets `ran=false`, and exits —
**no DeepSeek call, no test re-run**. Cheap on the happy path.
- **Strengthen-first.** Feeds the exact Sentinel findings to the
Refiner, which **adds/tightens assertions** so each test verifies what
its name promises → re-runs on the live binary (**green must hold**) →
re-checks assertion-integrity → re-audits. Capped at **2 rounds**.
- **Anti-laundering (the whole game).** Assertions may only be
**ADDED/strengthened — never dropped, inverted in place, or
renamed-to-dodge** a missing assertion. If strengthening goes red
because the **feature isn't wired**, it takes the honest `test.fixme` +
evidence exit (no green-wash).
- **`assert_integrity.py --strengthen`**: permits *added* negative
matchers (assert-absence is legit strengthening) while **still blocking
any dropped assertion** (removed/skipped guards + all static checks
stay). Unit-proven in the commit.
- **Quality gate** now judges the **post-refine** state; blocks only if
the Refiner couldn't keep the spec green honestly.
- **No-PR report** now phrases a block as a true last resort ("the
Refiner attempted to strengthen … but could not resolve every finding
honestly").
### Reuse path (clobber off)
`verify_heal` now also runs on **reuse runs**, and the "already covered"
short-circuit is suppressed on reuse — so **re-running `/e2e` on a
previously-BLOCKED spec re-validates + refines the existing tests into a
PR** instead of stopping at "already covered". No new tests are added on
reuse; the Refiner only strengthens what's there. (Cost: a reuse run
rebuilds even when tests were already perfect — acceptable since re-runs
are explicit; can add a "skip if a green test PR already exists" guard
later.)
## Validation
- `assert_integrity.py` unit-checked: added negative → FAIL without
`--strengthen`, PASS with it; dropped assertion → still FAIL with
`--strengthen`.
- Workflow YAML validated; step order confirmed **Sentinel re-audit →
Refiner → Quality gate**.
- Live end-to-end best validated by re-running `/e2e` on **#12857 with
clobber off** — the Refiner should strengthen the 2 flagged tests and
open the PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Post-PR "enrichment" follow-ups for the E2E test bot, plus a CLA fix.
(Was stacked on #12682; rebased onto `main` after that merged.)
## What this adds
**1. Test-case summary table**
When the test PR opens, the bot posts a **table of the test cases it
added/changed** — in the test PR body *and* the "E2E Council" comment on
the source PR:
| Test case | Type | What it verifies |
|---|---|---|
| `should …` | new / append / extend | one-line intent |
(Engineer emits `test-summary.json`; `pr_back` renders it. No table when
there's nothing to add.)
**2. Auto-address the AI review (OFF by default)**
New final job `review_address`, gated by repo variable
**`E2E_AUTO_REVIEW`** (default off). After the test PR opens it waits ~6
min for the repo's AI review, then a **conservative "Responder" agent**:
- **validates** each finding (the bot is noisy — default is to *not*
change code),
- fixes **only clearly-valid** ones in `tests/ui-testing` (never
`.github` / `.opencode` / `playwright.yml` / source),
- pushes tests-only fixes (the repo's Playwright CI re-validates the
PR),
- posts a rationale on the test PR and minimizes the bot comment.
One bounded pass. Scoped `GITHUB_TOKEN` only. **To enable:** create repo
variable `E2E_AUTO_REVIEW=true`.
**3. CLA fix — commit as `github-actions[bot]`**
The pipeline previously committed generated tests as `openobserve-bot
<bot@openobserve.ai>`, which isn't a GitHub account → CLA assistant
couldn't match/sign it and blocked the test PR. Now it commits as the
canonical **`github-actions[bot]`** identity (a real bot account GitHub
resolves via its noreply email), across generate + verify_heal +
review_address.
> **Admin action:** allowlist `github-actions[bot]` in CLA assistant,
then `recheck`.
---
<details>
<summary>Design notes</summary>
- Responder is deliberately conservative: a wrong "fix" that breaks a
passing test is worse than leaving a nitpick. It dismisses
stale/incorrect/style findings.
- Both auto-features are flag-gated OFF (same posture as
`E2E_AUTO_ON_MERGE`) — auto-acting warrants opt-in.
- No heavy re-build in `review_address`; the repo's existing Playwright
CI re-validates the pushed fix.
- CodeQL untrusted-checkout on the new `review_address` checkout (#416)
dismissed — same allowlist false positive as prior PRs.
</details>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Follow-up to #12652 that makes the **E2E Council** workflow correct and
pleasant for real use, after the first live dry-run triages exposed
gaps. v1 scope is unchanged: **OSS-only, manual-first, dry-run** (only
Job 1 triage runs; Jobs 2–4 are gated scaffolds).
## Changes
**Trigger is strictly PR-bound**
- Removed the free-form `oss_branch` input (it let you generate tests
detached from a PR, and dispatching from `main` diffed
main-against-itself = nothing).
- `pr_number` is now **required** for manual dispatch and accepts a
**number, `#123`, or a full PR URL** (normalized to an integer).
- A test is only ever generated for a PR. No PR ⇒ no run (empty/missing
diff fails fast with a clear message).
**`/e2e` is an exact match**
- Triggers only when the comment body is exactly `/e2e`. `Generate
/e2e`, `/e2e please`, etc. do **not** fire anything (the job's `if` is
false → no reaction, no comment, no run).
**Agent discovery fixed (the important one)**
- The first live run logged `agent "e2e-ci-triage" not found. Falling
back to default agent` — OpenCode does not auto-discover agents nested
under `.opencode/agents/<subdir>/`. It only worked by luck (the default
agent read the spec file).
- Fixed by registering all six agents in `opencode.jsonc` under the
`agent` key with `prompt: "{file:...}"`. **Confirmed working** on the
next run (`> e2e-ci-triage · deepseek-v4-pro`, no fallback).
**Immediate user feedback on a run**
- Posts a 👀 reaction on the `/e2e` comment and an "⏳ triage in
progress…" comment within seconds, then **edits that same comment in
place** with the decision (no second comment). Runs with `always()` so a
failed/empty run still updates it (never stuck on "in progress").
**Form/docs polish**
- Decluttered the Run-workflow form (short input labels; full guidance
lives in the workflow header).
## Verified live (dry-run triages on this PR)
- OpenCode installs (v1.17.7); DeepSeek model `deepseek/deepseek-v4-pro`
works.
- Secure path confirmed: PR read as a **diff via API**, no untrusted
PR-head checkout.
- Triage decisions correct + conservative (CI-only change → skip).
- Agent discovery fix confirmed (named agent loads as the system
prompt).
## Not in scope (still scaffolds)
Jobs 2–4 (generate → heal → PR-back). Build those once the triage gate
is trusted over a dry-run period.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What this adds
A CI-side **E2E test-generation pipeline** (Council of Agents) that runs
in GitHub Actions via **OpenCode on DeepSeek-V4-Pro**, fronting the
existing Playwright + TestDino CI.
**v1 scope: OSS-only, manual-first, dry-run.** On this first cut the
workflow does **only triage** — it classifies a change and posts the
decision as a PR comment. It does **not** generate tests, edit
`playwright.yml`, run anything, or open PRs.
## Files
- `.opencode/agents/e2e_ci_council_of_agents/` — 6 CI-adjusted agents
(triage, analyst, architect, engineer, sentinel, healer) + README.
`mode: primary`, file-based handoff, non-interactive. Engineer also
gains `playwright.yml` registration. Scribe excluded; ENT skipped at
triage.
- `opencode.jsonc` — DeepSeek provider.
- `.github/workflows/e2e-council.yml` — Job 1 (triage + dry-run comment)
is **live**; Jobs 2–4 (generate / heal / PR-back) are **gated
scaffolds** for when we flip the switch.
- `.gitignore` — track only the CI agent folder under `.opencode`.
## Triggers
- `workflow_dispatch` (pick branch) and `/e2e` comment on a PR.
Auto-on-merge is **off**.
- Both read the workflow from the **default branch**, so they only
activate after this merges.
## Before first run
- [ ] Add repo secret **`DEEPSEEK_API_KEY`**.
- [ ] Merge to main.
- [ ] Trigger via Actions → "E2E Council (Test Generation)" → Run, or
`/e2e` on a PR.
## Verify on first run (deferred by design)
- OpenCode install command/version.
- Whether `--agent e2e-ci-*` resolves agents nested under
`.opencode/agents/<subdir>/`.
- DeepSeek model id `deepseek/deepseek-v4-pro`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Harsh Mahajan