jianmu
/
openobserve
mirror of https://github.com/openobserve/openobserve
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

ci(e2e-council): test-quality gate — stop certifying green-but-worthless specs (#12722) 

## The problem (from the #12719 review)

The pipeline's only hard gate was **"the Healer's authoritative run is
green."** That objective *rewards* making a spec pass by gutting the
assertions that test the feature. In #12719 the result was a suite that:
- dropped the fix-query-card assertion,
- asserted `count >= 0` (a tautology),
- buried real assertions inside `if`s whose `else` just logs, and
- **asserted the feature is absent** (`expectFixQueryCardNotVisible`) —
certifying a half-built feature and set to **turn red when the feature
is finished**.

> A passing test and a valuable test are different things, and we only
measured the first.

This PR makes **"done" mean "the tests would fail if the feature were
broken."**

## Deterministic backbone (no LLM) —
`.github/scripts/assert_integrity.py`

The most important guards don't depend on the model behaving:
- **Diff-aware heal guard** — snapshot the spec's assertions *before*
healing; **reject** any heal that **deleted**, **inverted**
(`toBeVisible` → `not.toBeVisible`), or **skipped** assertions to reach
green.
- **Static anti-patterns** — tautologies (`count >= 0`, `expect(true)`)
and **conditional-only** tests (every `expect` inside an `if` → the test
can pass asserting nothing).
- Conservative by design (only high-confidence patterns) — a gate with
false positives gets disabled by humans. Restored from a **trusted ref**
in verify_heal (never the PR tree's copy).

## Workflow

- New **Assertion-integrity gate** after the authoritative run + a
combined **Quality gate decision** (integrity **+** Sentinel verdict).
`pr_back` now requires `quality_status == pass` → a green-but-worthless
spec **never becomes a PR**.
- The post-heal **Sentinel re-audit now feeds the blocking gate** (was
advisory-only). LLM flakiness fails **open**; the deterministic gate is
the hard teeth.
- **`feature-incomplete`** honest exit: when a test can't pass because
the feature isn't wired, the Healer parks it `test.fixme` + writes
**evidence**; the workflow opens **no PR** and posts the gap (file:line)
on the source PR — the single highest-value output.
- `pr_back` **reconciles `test-summary.json` against the FINAL spec**
(drops phantom rows) so the PR body can't claim tests that don't exist.

## Prompts

- **Healer:** hard anti-weakening rules + the `feature-incomplete`
honest exit.
- **Sentinel:** tautology / conditional-only / negative-only-on-feature
/ name-mismatch as **CRITICAL** checks.
- **Engineer:** no dead page-object scaffolding; ban `waitForTimeout` as
a sync primitive; assertions must be real and match the test name.
- **Architect:** one spec = one area (no Frankenstein specs);
de-duplicate scenarios.

## Considered and deferred: mutation-sanity (extra feature-absent build)

Theoretically the cleanest definition of a valuable test, but **dropped
for now**: it overlaps the cheap static checks, has **no clean
"feature-absent" baseline** under merged/multi-PR input, risks false
positives that erode trust, and doubles the build. The static guards
catch the same #12719 failures at near-zero cost; prevention (Healer
can't invert + honest `feature-incomplete`) beats the expensive
detector.

## Testing

- `assert_integrity.py` unit-tested against tautology / conditional-only
(incl. single-line) / heal-removed / heal-inverted fixtures, with no
false positive on a legit `if` + unconditional `expect`.
- Workflow YAML validated; analyzer compiles.
- **First live run still needed** (per the project's "run the path
before done" rule) — to see the gate block a real weak spec and the
`feature-incomplete` comment fire.

Stacked on #12716 (ENT sister flow); the diff collapses to just the
quality program once #12716 lands.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Shrinath Rao 2026-06-21 15:00:03 +05:30 committed by GitHub
parent f702288553
commit 900ebe67c6
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
8 changed files with 687 additions and 48 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

241
.github/scripts/assert_integrity.py vendored Normal file
View File

@ -0,0 +1,241 @@
#!/usr/bin/env python3
"""
Assertion-integrity analyzer for the E2E Council pipeline (deterministic — no LLM).
Two modes:
fingerprint <spec.js> -> print a JSON fingerprint of the spec's assertions
check <spec.js> [--baseline <fingerprint.json>] [--allow-weakening]
-> static anti-pattern scan + (optional) diff-aware
heal-weakening check; writes nothing, prints a JSON
verdict, exits 2 if any CRITICAL finding.
Conservative by design: only HIGH-CONFIDENCE patterns are flagged as critical, because a gate with
false positives gets disabled by humans (the same trap that made mutation-testing a poor fit).
"""
import sys, re, json
NEG_MATCHERS = re.compile(r'\.not\.|\.toBeHidden\(|\.toBeFalsy\(')
TAUTOLOGIES = [
(re.compile(r'toBeGreaterThanOrEqual\(\s*0\s*\)'), 'count >= 0 is always true'),
(re.compile(r'toBeGreaterThan\(\s*-1\s*\)'), 'count > -1 is always true'),
(re.compile(r'toBeLessThan\(\s*Infinity\s*\)'), '< Infinity is always true'),
(re.compile(r'expect\(\s*true\s*\)\.toBe(Truthy)?\('), 'expect(true) asserts nothing'),
(re.compile(r'expect\(\s*1\s*\)\.toBe\(\s*1\s*\)'), 'expect(1).toBe(1) asserts nothing'),
(re.compile(r'expect\(\s*false\s*\)\.toBe(Falsy)?\('), 'expect(false) asserts nothing'),
]
EXPECT = re.compile(r'\bexpect\(')
TEST_START = re.compile(r'\b(test|it)(\.skip|\.fixme|\.only)?\s*\(')
SKIP_FIXME = re.compile(r'\b(test|it)\.(skip|fixme)\s*\(|\btest\.skip\(\s*\)|\.fixme\(')
def strip_noise(line):
"""Blank out comments + string literals so matchers don't fire inside them (best-effort,
line-local). Handles `//` line comments and single-line `/* */` blocks; multi-line `/* */`
blocks are rare in generated specs and not tracked across lines."""
line = re.sub(r'/\*.*?\*/', '', line) # single-line block comment
line = re.sub(r'//.*$', '', line) # line comment
line = re.sub(r'`[^`]*`', '``', line)
line = re.sub(r"'[^']*'", "''", line)
line = re.sub(r'"[^"]*"', '""', line)
return line
def fingerprint(src):
"""Return counts that summarize a spec's assertion shape: total expect()s, negative matchers,
skip/fixme, and test blocks. Compared before vs. after heal to detect weakening."""
code = '\n'.join(strip_noise(l) for l in src.split('\n'))
return {
'expects': len(EXPECT.findall(code)),
'negatives': len(NEG_MATCHERS.findall(code)),
'skips': len(re.findall(r'\.(skip|fixme)\s*\(', code)),
'tests': len(TEST_START.findall(code)),
}
def find_test_blocks(lines):
"""Yield (name_line_idx, start_idx, end_idx) for each test()/it() block by matching the block's
braces. Ends exactly when the brace depth returns to 0 (== 0, not <= 0, so a stray nested
structure can't terminate the block early)."""
blocks = []
i, n = 0, len(lines)
while i < n:
if TEST_START.search(strip_noise(lines[i])):
depth, j, started = 0, i, False
while j < n:
for ch in strip_noise(lines[j]):
if ch == '{':
depth += 1; started = True
elif ch == '}':
depth -= 1
if started and depth == 0:
break
j += 1
blocks.append((i, i, j))
i = j + 1
else:
i += 1
return blocks
# `if (...)` / `else` that directly precedes a statement on the SAME line with NO opening brace —
# i.e. a braceless single-statement conditional like `if (x) await expect(y).toBeVisible();`.
BRACELESS_GUARD = re.compile(r'(?:\bif\s*\([^{}]*\)|\belse)\s*(?:await\s+)?$')
def conditional_only_tests(lines):
"""A test where EVERY expect() sits inside an if/else block (so the test can end asserting nothing).
Scans `{`, `}` and `expect(` events in POSITIONAL order so a single-line `if(){expect}else{}` is
judged with the correct brace stack (not after all braces on the line are processed)."""
flagged = []
for name_i, s, e in find_test_blocks(lines):
total, conditional = 0, 0
stack = [] # frame value = True if introduced by if/else
for raw in lines[s:e + 1]:
line = strip_noise(raw)
events = sorted(
[(m.start(), m.group()) for m in re.finditer(r'[{}]', line)] +
[(m.start(), 'E') for m in EXPECT.finditer(line)]
)
for pos, kind in events:
if kind == '{':
stack.append(bool(re.search(r'\b(if|else)\b', line[:pos])))
elif kind == '}':
if stack:
stack.pop()
else: # an expect(
total += 1
# conditional if inside a braced if/else frame OR guarded by a braceless
# single-statement `if (...)` / `else` immediately before it on this line
if any(stack) or BRACELESS_GUARD.search(line[:pos]):
conditional += 1
if total > 0 and conditional == total:
flagged.append((name_i + 1, total))
return flagged
# Assertion-ish calls whose failure should surface — if these sit in a try{} whose catch swallows
# (no throw), the test passes even when the assertion fails.
ASSERT_TOKEN = re.compile(r'\bexpect\s*\(|\.waitFor\s*\(|\.toBe(Visible|Hidden|Truthy|Falsy)\b|\.toContainText\b|\.toHaveText\b')
def _match_brace(text, open_idx):
"""Index of the '}' matching the '{' at open_idx (-1 if unbalanced)."""
depth = 0
for i in range(open_idx, len(text)):
if text[i] == '{':
depth += 1
elif text[i] == '}':
depth -= 1
if depth == 0:
return i
return -1
def swallowed_assertion_tests(lines):
"""A test where an assertion sits inside a `try {}` whose matching `catch {}` does NOT re-throw —
the catch swallows the failure (logs/returns), so the test passes even when the assertion fails.
A `catch` that `throw`s, or a try/catch with no assertion inside the try (e.g. cleanup), is fine."""
flagged = []
for name_i, s, e in find_test_blocks(lines):
block = '\n'.join(strip_noise(l) for l in lines[s:e + 1])
for m in re.finditer(r'\bcatch\b\s*(\([^)]*\))?\s*\{', block):
co = block.index('{', m.start())
cc = _match_brace(block, co)
if cc < 0:
continue
if re.search(r'\bthrow\b', block[co + 1:cc]):
continue # catch re-throws → the failure surfaces → fine
pre = block[:m.start()]
tclose = pre.rstrip().rfind('}')
if tclose < 0:
continue
depth, topen = 0, -1
for i in range(tclose, -1, -1):
if pre[i] == '}':
depth += 1
elif pre[i] == '{':
depth -= 1
if depth == 0:
topen = i
break
if topen < 0 or not re.search(r'\btry\s*$', pre[:topen]):
continue
if ASSERT_TOKEN.search(pre[topen + 1:tclose]):
flagged.append(name_i + 1)
break
return flagged
def check(spec_path, baseline=None, allow_weakening=False):
"""Run the static anti-pattern scan (tautologies, conditional-only assertions) and, when a
baseline is given, the diff-aware heal-weakening guard. Returns a verdict dict; never raises on
a malformed/partial baseline — missing baseline keys fall back to the current value (skip that
comparison) rather than crashing the gate."""
src = open(spec_path, encoding='utf-8').read()
lines = src.split('\n')
findings = []
# --- static anti-patterns ---
for idx, raw in enumerate(lines, 1):
line = strip_noise(raw)
for pat, why in TAUTOLOGIES:
if pat.search(line):
findings.append({'severity': 'critical', 'rule': 'tautology', 'line': idx, 'detail': why})
for line_no, total in conditional_only_tests(lines):
findings.append({'severity': 'critical', 'rule': 'conditional-only-assertions', 'line': line_no,
'detail': f'all {total} expect() in this test are inside if/else — the test can pass asserting nothing'})
for line_no in swallowed_assertion_tests(lines):
findings.append({'severity': 'critical', 'rule': 'swallowed-assertion', 'line': line_no,
'detail': 'an assertion sits in a try{} whose catch swallows the failure (no throw) — the test passes even when the assertion fails'})
# No real coverage: a spec with tests where EVERY one is skip/fixme is a green run that ran
# nothing (e.g. "fixme everything" to dodge a gap). A genuinely feature-incomplete spec should
# be blocked anyway — so flag it here too. (Applies even with --allow-weakening: this is static.)
fp0 = fingerprint(src)
if fp0['tests'] > 0 and fp0['skips'] >= fp0['tests']:
findings.append({'severity': 'critical', 'rule': 'no-runnable-tests',
'detail': f"every test is skip/fixme ({fp0['skips']} skip/fixme vs {fp0['tests']} tests) — the spec asserts nothing at runtime"})
# --- diff-aware heal-weakening guard ---
fp = fingerprint(src)
if isinstance(baseline, dict) and not allow_weakening:
base_expects = baseline.get('expects', fp['expects'])
base_negatives = baseline.get('negatives', fp['negatives'])
base_skips = baseline.get('skips', fp['skips'])
if fp['expects'] < base_expects:
findings.append({'severity': 'critical', 'rule': 'heal-removed-assertions',
'detail': f"expect() count dropped {base_expects} -> {fp['expects']} during heal"})
if fp['negatives'] > base_negatives:
findings.append({'severity': 'critical', 'rule': 'heal-inverted-assertions',
'detail': f"negative matchers rose {base_negatives} -> {fp['negatives']} during heal (positive flipped to negative?)"})
if fp['skips'] > base_skips:
findings.append({'severity': 'critical', 'rule': 'heal-skipped-tests',
'detail': f"skip/fixme rose {base_skips} -> {fp['skips']} during heal (use the feature-incomplete path, not silent skipping)"})
critical = [f for f in findings if f['severity'] == 'critical']
return {'verdict': 'FAIL' if critical else 'PASS', 'critical_count': len(critical),
'fingerprint': fp, 'findings': findings}
def main():
if len(sys.argv) < 3 or sys.argv[1] not in ('fingerprint', 'check'):
sys.stderr.write(__doc__); sys.exit(64)
mode, spec = sys.argv[1], sys.argv[2]
if mode == 'fingerprint':
print(json.dumps(fingerprint(open(spec, encoding='utf-8').read())))
return
baseline, allow = None, False
args = sys.argv[3:]
if '--baseline' in args:
baseline = json.load(open(args[args.index('--baseline') + 1]))
if '--allow-weakening' in args:
allow = True
result = check(spec, baseline, allow)
print(json.dumps(result, indent=2))
sys.exit(2 if result['verdict'] == 'FAIL' else 0)
if __name__ == '__main__':
main()

282
.github/workflows/e2e-council.yml vendored
View File

@ -79,7 +79,11 @@ concurrency:
# One run at a time per source PR (covers comment, dispatch, and auto-on-merge events) so two
# triggers can't race on the same autoe2e/pr-<n> branch.
group: e2e-council-${{ github.event.issue.number || github.event.pull_request.number || github.event.inputs.pr_number || github.ref }}
cancel-in-progress: true
# cancel-in-progress is FALSE for issue_comment events: the pipeline posts its own comments on the
# source PR (pr_back's link, the no-PR report), which fire issue_comment events in this SAME group.
# With cancel-in-progress=true those would self-cancel the run mid-pr_back. A comment now QUEUES
# behind the running pipeline instead. Dispatch / pull_request still supersede (latest wins).
cancel-in-progress: ${{ github.event_name != 'issue_comment' }}
env:
# Model every agent runs on (provider `deepseek` configured in opencode.jsonc, keyed by the
@ -401,7 +405,22 @@ jobs:
body += `- **Skip:** ${clean(t.skip, 10)}${t.skip_reason ? ` — _${clean(t.skip_reason)}_` : ''}\n`;
if (t.feature_title) body += `- **Feature:** ${clean(t.feature_title, 120)} (area: ${clean(t.area, 40) || '?'})\n`;
if (t.spec_path) body += `- **Would write:** \`${clean(t.spec_path, 200)}\`\n`;
body += `\n**Rationale:** ${clean(t.rationale, 4000)}\n`;
// Rationale is rendered DETERMINISTICALLY from a structured object so the format is
// identical every run (the LLM only supplies the prose per field, never the layout).
// Falls back to a legacy free-form string if `rationale` isn't an object.
const r = t.rationale;
if (r && typeof r === 'object') {
const yn = (v) => (v === true || v === 'true') ? 'Yes' : 'No';
const sec = (label, val, reason) => reason ? `\n**${label}: ${val}** — ${clean(reason, 1500)}\n` : '';
body += sec('EDITION', (String(t.edition || '').toUpperCase() || '?'), r.edition);
body += sec('SKIP', yn(t.skip), r.skip || t.skip_reason);
body += sec('E2E WARRANTED', yn(t.needs_e2e), r.e2e);
body += sec('AREA', (clean(t.area, 40) || '?'), r.area);
body += sec('GROUP', (clean(t.playwright_group, 40) || '?'), r.group);
body += sec('EXISTING TESTS', yn(t.existing_tests_in_diff), r.existing_tests);
} else {
body += `\n**Rationale:** ${clean(t.rationale, 4000)}\n`;
}
if ('${{ steps.mode.outputs.dry_run }}' === 'true') {
body += `\n_Dry-run (v1 manual-first): no tests generated._`;
}
@ -752,6 +771,7 @@ jobs:
pull-requests: write # post a failure comment on the source PR (scoped GITHUB_TOKEN; PR comments need this)
outputs:
heal_status: ${{ steps.finalrun.outputs.status }}
quality_status: ${{ steps.quality.outputs.status }} # pass only if integrity + Sentinel both clear
env:
ZO_ROOT_USER_EMAIL: root@example.com
ZO_ROOT_USER_PASSWORD: Complexpass#123
@ -798,6 +818,11 @@ jobs:
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then RESTORE_REF="$REF_NAME"; fi
git fetch origin "$RESTORE_REF" --depth=1
git checkout "origin/$RESTORE_REF" -- .opencode opencode.jsonc
# Also restore the deterministic assertion-integrity analyzer from the SAME trusted ref — it
# must never be the PR tree's (untrusted) copy, since it gates whether the PR is opened.
git checkout "origin/$RESTORE_REF" -- .github/scripts/assert_integrity.py 2>/dev/null \
&& echo "Restored assert_integrity.py from origin/$RESTORE_REF" \
|| echo "::warning::assert_integrity.py not on origin/$RESTORE_REF — integrity gate will no-op this run"
echo "Restored pipeline agents/config from origin/$RESTORE_REF"
# No artifact download here on purpose: the hand-off (docs/test_generator/ci/) is committed on
@ -880,6 +905,28 @@ jobs:
echo "spec=$SPEC" >> "$GITHUB_OUTPUT"
echo "rel=${SPEC#tests/ui-testing/}" >> "$GITHUB_OUTPUT"
- name: Fingerprint assertions (pre-heal baseline)
# Snapshot the GENERATED spec's assertion shape BEFORE healing, so the integrity gate (after
# the final run) can detect heal-by-weakening — assertions deleted, positives flipped to
# negatives, or tests skipped — by comparing against this baseline.
env:
SPEC: ${{ steps.spec.outputs.spec }}
run: |
if [ -f .github/scripts/assert_integrity.py ]; then
python3 .github/scripts/assert_integrity.py fingerprint "$SPEC" > docs/test_generator/ci/assert-baseline.json \
&& echo "pre-heal baseline: $(cat docs/test_generator/ci/assert-baseline.json)"
else
echo "::warning::analyzer missing — no assertion baseline (diff-weakening guard disabled this run)"
fi
# Snapshot files ALREADY dirty outside tests/ BEFORE heal — the build step seds product
# files (e.g. src/handler/http/router/ui.rs → /tmp/dist). The scope-lock subtracts this so it
# only flags files the HEALER changed, never the build's own pre-heal modifications.
git status --porcelain -- . \
':(exclude)tests/ui-testing/**' ':(exclude)docs/test_generator/**' \
':(exclude).opencode/**' ':(exclude)opencode.jsonc' ':(exclude).github/**' \
| sed 's/^...//' | sort > docs/test_generator/ci/pre-heal-dirty.txt
echo "pre-heal out-of-scope dirty (build artifacts, baselined out): $(wc -l < docs/test_generator/ci/pre-heal-dirty.txt) file(s)"
- name: Heal — fresh-context loop until the spec passes
id: heal
# Up to 3 FRESH healer invocations (clean context each). Each does a few internal iterations;
@ -900,16 +947,49 @@ jobs:
for attempt in 1 2 3; do
echo "::group::Heal attempt ${attempt}/3 (fresh context)"
opencode run --agent e2e-ci-healer --model "$PIPELINE_MODEL" \
"Heal attempt ${attempt} of 3 — you have a FRESH context. Heal ONLY this spec (do NOT run the whole suite): ${SPEC}. A live OpenObserve is booted at ${ZO_BASE_URL}. FIRST, if docs/test_generator/ci/heal-notes.md exists, read it — it lists what earlier attempts already tried; do NOT repeat those approaches. Run 'cd tests/ui-testing && npx playwright test ${REL} --workers=4 --reporter=line', diagnose, fix the spec/page objects, re-run — up to 3 internal iterations. NEVER touch /tmp or any path outside the repo (the sandbox blocks it and the command fails); read stdout directly. If you get it passing, write docs/test_generator/ci/heal-result.json with status=passing. If still failing after your iterations, APPEND a concise note (test name, the failure, what you tried) to docs/test_generator/ci/heal-notes.md, write heal-result.json status=failing, and stop. Keep reasoning concise."
"Heal attempt ${attempt} of 3 — you have a FRESH context. Heal ONLY this spec (do NOT run the whole suite): ${SPEC}. A live OpenObserve is booted at ${ZO_BASE_URL}. FIRST, if docs/test_generator/ci/heal-notes.md exists, read it — it lists what earlier attempts already tried; do NOT repeat those approaches. Run 'cd tests/ui-testing && npx playwright test ${REL} --workers=4 --reporter=line', diagnose, fix the spec/page objects, re-run — up to 3 internal iterations. NEVER touch /tmp or any path outside the repo (the sandbox blocks it and the command fails); read stdout directly. SCOPE LOCK: change ONLY files under tests/ui-testing/. NEVER edit product code (src/**, web/src/**, *.rs, *.vue), config, or playwright.yml, and NEVER build/compile/restart the product (no cargo build/check, no npm run build) — the binary is already booted; a gate reverts+rejects any out-of-scope change. If you get it passing HONESTLY (no deleted/inverted/conditionalized assertions, no product edits), write docs/test_generator/ci/heal-result.json with status=passing. If a test cannot pass because the FEATURE is not wired in this build (component never renders / code path commented out), do NOT weaken it — mark it test.fixme('reason, see #issue'), keep the assertion intact, and write status=feature-incomplete with an 'evidence' field naming the file:line that proves the gap. If still failing for an ordinary reason after your iterations, APPEND a concise note (test name, failure, what you tried) to docs/test_generator/ci/heal-notes.md, write heal-result.json status=failing, and stop. NEVER weaken an assertion to force green — a deterministic gate compares assertions before/after heal and will reject it. Keep reasoning concise."
echo "::endgroup::"
STATUS=$(jq -r '.status // "failing"' docs/test_generator/ci/heal-result.json 2>/dev/null || echo failing)
if [ "$STATUS" = "passing" ]; then
echo "::notice::Healer reports passing on attempt ${attempt}."; break
fi
if [ "$STATUS" = "feature-incomplete" ]; then
echo "::notice::Healer reports feature-incomplete — the feature isn't wired in this build; retrying won't help. Stopping the loop."; break
fi
if [ "$attempt" -lt 3 ]; then echo "::notice::Attempt ${attempt} did not pass → retrying with a fresh context.";
else echo "::notice::3 fresh-context attempts exhausted — leaving it to the authoritative run."; fi
done
- name: Healer scope-lock (deterministic — only tests/ui-testing may change)
id: scope
# HARD boundary: the Healer fixes TESTS, not the product. If it edited anything outside
# tests/ui-testing/ (most often it tried to patch src/** or web/src/** and rebuild the binary
# to force a green — a fake pass that never reproduces, since those edits aren't committed), we
# REVERT those changes and BLOCK the run. docs/test_generator/ (handoff) + the restored pipeline
# files (.opencode/opencode.jsonc/.github) are exempt — they're stripped/restored anyway.
run: |
# Current out-of-scope dirty files…
CUR=$(git status --porcelain -- . \
':(exclude)tests/ui-testing/**' ':(exclude)docs/test_generator/**' \
':(exclude).opencode/**' ':(exclude)opencode.jsonc' ':(exclude).github/**' \
| sed 's/^...//' | sort)
# …minus the files the BUILD already dirtied before heal (e.g. ui.rs sed). Whatever remains
# was changed by the HEALER — a genuine scope violation.
PRE=docs/test_generator/ci/pre-heal-dirty.txt
[ -f "$PRE" ] || : > "$PRE" # ensure it exists (empty) so comm has a left/right operand
VIOL=$(comm -23 <(printf '%s\n' "$CUR" | sed '/^$/d') <(sort "$PRE"))
if [ -n "$VIOL" ]; then
echo "status=violation" >> "$GITHUB_OUTPUT"
echo "::error::Healer SCOPE VIOLATION — it changed files OUTSIDE tests/ui-testing/ (likely patched the product to force a green). Reverting + blocking:"
printf '%s\n' "$VIOL"
# Revert tracked edits and remove any new untracked files so nothing leaks to the branch.
printf '%s\n' "$VIOL" | while IFS= read -r f; do [ -n "$f" ] && git checkout -- "$f" 2>/dev/null || true; done
git clean -fd -- $(printf '%s ' $VIOL) 2>/dev/null || true
else
echo "status=clean" >> "$GITHUB_OUTPUT"
echo "Scope OK — the Healer touched only tests/ui-testing/ (+ handoff)."
fi
- name: Authoritative final run (the workflow decides pass/fail, not the LLM)
id: finalrun
timeout-minutes: 15
@ -925,10 +1005,40 @@ jobs:
echo "::warning::Generated spec still failing after healing."
fi
- name: Assertion-integrity gate (deterministic — blocks weak / gamed-green specs)
id: integrity
# A spec can exit 0 yet test NOTHING (tautologies like `count >= 0`, assertions only inside
# `if` blocks) OR have been healed to green by deleting/inverting the assertions that catch the
# bug (#12719). This deterministic check (no LLM) is the teeth: green is necessary, not
# sufficient. Runs only when the authoritative run passed — a failing run is handled below.
if: steps.finalrun.outputs.status == 'passing'
env:
SPEC: ${{ steps.spec.outputs.spec }}
run: |
if [ ! -f .github/scripts/assert_integrity.py ]; then
echo "::warning::analyzer missing — integrity gate skipped (treating as pass)"
echo "status=pass" >> "$GITHUB_OUTPUT"; exit 0
fi
# The Healer's HONEST 'feature-incomplete' path is allowed to park tests as fixme — only then
# relax the diff-weakening guard. Static tautology/conditional-only checks ALWAYS apply.
HSTATUS=$(jq -r '.status // ""' docs/test_generator/ci/heal-result.json 2>/dev/null || echo "")
ALLOW=""; [ "$HSTATUS" = "feature-incomplete" ] && ALLOW="--allow-weakening"
BASE=""; [ -f docs/test_generator/ci/assert-baseline.json ] && BASE="--baseline docs/test_generator/ci/assert-baseline.json"
if python3 .github/scripts/assert_integrity.py check "$SPEC" $BASE $ALLOW > docs/test_generator/ci/assert-verdict.json; then
echo "status=pass" >> "$GITHUB_OUTPUT"
echo "::notice::Assertion-integrity gate PASSED — assertions are real and were not weakened by healing."
else
echo "status=fail" >> "$GITHUB_OUTPUT"
echo "::error::Assertion-integrity gate FAILED — spec is green but does not meaningfully test the feature:"
jq -r '.findings[] | " - [\(.rule)] line \(.line // "-"): \(.detail)"' docs/test_generator/ci/assert-verdict.json 2>/dev/null \
|| cat docs/test_generator/ci/assert-verdict.json
fi
- name: Sentinel re-audit (post-heal)
# ADVISORY ONLY: the authoritative final run already decided pass/fail. This re-audit is a
# quality annotation, so it must NEVER abort the job — continue-on-error keeps a green heal
# from being flipped to "failed" just because the audit (or opencode) hiccuped.
# The Sentinel's verdict now FEEDS the blocking quality gate below (it is no longer purely
# advisory). The STEP itself stays continue-on-error so an opencode hiccup never crashes the
# job — if no fresh verdict is written we fall back to the (PASS) generate-time verdict, i.e.
# LLM flakiness fails OPEN. The deterministic integrity gate above is the hard, no-flake teeth.
continue-on-error: true
if: steps.finalrun.outputs.status == 'passing'
env:
@ -936,7 +1046,40 @@ jobs:
SPEC: ${{ steps.spec.outputs.spec }}
run: |
opencode run --agent e2e-ci-sentinel --model "$PIPELINE_MODEL" \
"Re-audit the healed spec/page objects for ${SPEC}; update docs/test_generator/ci/sentinel-verdict.json."
"Re-audit the healed spec/page objects for ${SPEC}. Apply ALL critical checks, especially the assertion-quality ones (tautologies, conditional-only assertions, negative-only assertions on the feature under test, test-name/assertion mismatch). Update docs/test_generator/ci/sentinel-verdict.json with an honest verdict."
- name: Quality gate decision (deterministic integrity + Sentinel verdict)
id: quality
# The single source of truth for "is this PR allowed to open." Blocks if EITHER the
# deterministic integrity gate failed OR the Sentinel returned a FAIL/critical verdict. pr_back
# gates on this output, so a green-but-worthless spec never becomes a PR.
if: steps.finalrun.outputs.status == 'passing'
env:
INTEGRITY: ${{ steps.integrity.outputs.status }}
SCOPE: ${{ steps.scope.outputs.status }}
run: |
# Healer scope violation (edited product code) → never trust the green; block. (A passing
# final run here may be against a Healer-patched binary that won't reproduce.)
if [ "$SCOPE" = "violation" ]; then
echo "status=fail" >> "$GITHUB_OUTPUT"
echo "::error::Quality gate FAILED — Healer scope violation (edited files outside tests/). No PR."
exit 0
fi
# feature-incomplete does NOT block on its own (the balance): as long as SOME real tests
# pass, open the PR with the green tests + fixme placeholders. The "every test is fixme →
# zero real coverage" case is caught by the integrity gate's `no-runnable-tests` check
# below, which fails. Evidence for the gaps rides in the fixme test titles.
HSTATUS=$(jq -r '.status // ""' docs/test_generator/ci/heal-result.json 2>/dev/null || echo "")
[ "$HSTATUS" = "feature-incomplete" ] && echo "::notice::Healer parked some tests as fixme (feature-incomplete) — opening a PR with the passing tests + fixme placeholders, provided real coverage exists."
SENT=$(jq -r '.verdict // "PASS"' docs/test_generator/ci/sentinel-verdict.json 2>/dev/null || echo PASS)
SENT_CRIT=$(jq -r '.critical_count // 0' docs/test_generator/ci/sentinel-verdict.json 2>/dev/null || echo 0)
if [ "$INTEGRITY" = "fail" ] || [ "$SENT" = "FAIL" ] || { [ "$SENT_CRIT" -gt 0 ] 2>/dev/null; }; then
echo "status=fail" >> "$GITHUB_OUTPUT"
echo "::error::Quality gate FAILED (integrity=$INTEGRITY, sentinel=$SENT, critical=$SENT_CRIT) — no test PR will be opened."
else
echo "status=pass" >> "$GITHUB_OUTPUT"
echo "::notice::Quality gate PASSED (integrity=$INTEGRITY, sentinel=$SENT)."
fi
- name: Commit + push healer fixes to autoe2e branch
env:
@ -971,13 +1114,15 @@ jobs:
path: |
o2.log
docs/test_generator/ci/
docs/test_generator/audit-reports/
docs/test_generator/execution-reports/
tests/ui-testing/test-results/
- name: Report heal failure on the source PR (root cause + run link)
# Runs whenever the spec did NOT end up passing (build error, boot error, healer timeout,
# or still-failing). Posts the ROOT CAUSE + run link on the dev's PR, then logs the
# comment URL as the final step (so the dev can go verify). No PR is opened in this case.
if: always() && steps.finalrun.outputs.status != 'passing'
# Runs when no PR will open: the spec didn't pass (build/boot/timeout/still-failing) OR it
# passed but the QUALITY GATE blocked it (gamed-green / healed-by-weakening / feature-incomplete).
# Posts the ROOT CAUSE + run link on the dev's PR. No PR is opened in any of these cases.
if: always() && (steps.finalrun.outputs.status != 'passing' || steps.quality.outputs.status == 'fail')
env:
# GITHUB_TOKEN (NOT ORG_ADMIN_TOKEN): this job BUILDS + RUNS the PR's code, so an
# org-admin token must never be present here (a malicious build could intercept it).
@ -990,39 +1135,73 @@ jobs:
SPEC_OUTCOME: ${{ steps.spec.outcome }}
HEAL_OUTCOME: ${{ steps.heal.outcome }}
FINAL_STATUS: ${{ steps.finalrun.outputs.status }}
QUALITY_STATUS: ${{ steps.quality.outputs.status }}
SCOPE_STATUS: ${{ steps.scope.outputs.status }}
run: |
RUN_URL="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
# Root cause = the most-upstream failing stage (build -> boot -> pw-setup -> spec-resolve -> heal -> final).
EXTRA="" # optional appended detail (integrity findings / feature-incomplete evidence)
# OUTCOME = short label; CAUSE = one-line why. Root cause = most-upstream failing stage
# (build -> boot -> pw-setup -> spec-resolve -> heal -> final -> quality).
if [ "$BUILD_OUTCOME" = "failure" ]; then
CAUSE="Building OpenObserve from the feature branch failed (compile / frontend build error)."
OUTCOME="Build failed"; CAUSE="Building OpenObserve from the feature branch failed (compile / frontend build error)."
elif [ "$BOOT_OUTCOME" = "failure" ]; then
CAUSE="OpenObserve built but failed to boot/serve on :5080 within the timeout."
OUTCOME="Boot failed"; CAUSE="OpenObserve built but failed to boot/serve on :5080 within the timeout."
elif [ "$PWINSTALL_OUTCOME" = "failure" ]; then
CAUSE="Setting up the test environment failed (npm ci / Playwright browser install) — a CI/infra problem, not the generated test."
OUTCOME="Test-env setup failed (CI/infra)"; CAUSE="npm ci / Playwright browser install failed — a CI/infra problem, not the generated test."
elif [ "$SPEC_OUTCOME" = "failure" ]; then
CAUSE="Could not resolve the generated spec path from run metadata (generation/hand-off problem — run-context.json missing or spec_path invalid)."
OUTCOME="Spec resolution failed"; CAUSE="Could not resolve the generated spec path from run metadata (generation / hand-off problem)."
elif [ "$HEAL_OUTCOME" = "failure" ] || [ "$HEAL_OUTCOME" = "cancelled" ]; then
CAUSE="The healer hit its 30-minute cap (hung or too-slow heal loop) and was stopped before the tests passed."
OUTCOME="Heal timed out"; CAUSE="The healer hit its time cap (hung or too-slow heal loop) before the tests passed."
elif [ "$FINAL_STATUS" = "failing" ]; then
CAUSE="The generated spec still failed after healing (<=3 iterations) — likely a genuinely broken/flaky test or a real product issue."
OUTCOME="Tests still failing after heal"; CAUSE="The generated spec still failed after healing — likely a genuinely broken/flaky test or a real product issue."
elif [ "$SCOPE_STATUS" = "violation" ]; then
OUTCOME="Blocked — Healer edited product code"; CAUSE="The Healer tried to edit product code (outside \`tests/ui-testing/\`) to force a green — reverted + blocked. This usually means the feature itself is broken: fix the feature, then the tests should pass."
elif [ "$QUALITY_STATUS" = "fail" ]; then
# Report the ACTUAL blocker (priority: zero-coverage -> integrity gate -> Sentinel), not just
# "feature-incomplete" — a feature-incomplete heal can still have passing tests and be blocked
# by the Sentinel instead. Read the verdict files written by the gate steps.
IVERDICT=$(jq -r '.verdict // "PASS"' docs/test_generator/ci/assert-verdict.json 2>/dev/null || echo PASS)
HAS_NORUN=$(jq -e '.findings[]? | select(.rule=="no-runnable-tests")' docs/test_generator/ci/assert-verdict.json >/dev/null 2>&1 && echo yes || echo no)
SENTV=$(jq -r '.verdict // "PASS"' docs/test_generator/ci/sentinel-verdict.json 2>/dev/null || echo PASS)
SENTC=$(jq -r '.critical_count // 0' docs/test_generator/ci/sentinel-verdict.json 2>/dev/null || echo 0)
if [ "$HAS_NORUN" = "yes" ]; then
OUTCOME="Feature incomplete — every test parked (no green-wash)"; CAUSE="No test could pass honestly because the feature isn't wired in this build — nothing was certified. The assertions are kept as \`fixme\` and will go green once the feature is finished."
EV=$(jq -r '.evidence // empty' docs/test_generator/ci/heal-result.json 2>/dev/null || echo "")
[ -n "$EV" ] && EXTRA="- **Evidence:** ${EV}"
elif [ "$IVERDICT" = "FAIL" ]; then
OUTCOME="Blocked — weak / gamed tests (integrity gate)"; CAUSE="The deterministic assertion-integrity gate found anti-patterns — tautologies, conditional-only or try/catch-swallowed assertions, or assertions weakened/inverted during healing."
elif [ "$SENTV" = "FAIL" ] || { [ "$SENTC" -gt 0 ] 2>/dev/null; }; then
OUTCOME="Blocked — Sentinel found ${SENTC} critical quality issue(s)"; CAUSE="The Sentinel (LLM quality auditor) flagged ${SENTC} critical issue(s) the deterministic gate doesn't cover — e.g. a skippable / try-catch-swallowed assertion, a test-name vs assertion mismatch, or a raw selector in the spec. See the run's audit in the logs."
else
OUTCOME="Blocked — quality gate"; CAUSE="The quality gate did not pass (see the run)."
fi
if [ -f docs/test_generator/ci/assert-verdict.json ]; then
F=$(jq -r '.findings[]? | " - `\(.rule)`\(if .line then " (line \(.line))" else "" end): \(.detail)"' docs/test_generator/ci/assert-verdict.json 2>/dev/null || echo "")
[ -n "$F" ] && EXTRA=$(printf '%s\n- **Findings:**\n%s' "$EXTRA" "$F")
fi
else
CAUSE="Verify/heal did not complete (see the run)."
OUTCOME="Did not complete"; CAUSE="Verify/heal did not complete (see the run)."
fi
MARKER="<!-- e2e-council-testpr -->"
# Summary facts (best-effort, from the hand-off).
FEATURE=$(jq -r '.feature_title // empty' docs/test_generator/ci/run-context.json 2>/dev/null || echo "")
AREA=$(jq -r '.area // empty' docs/test_generator/ci/run-context.json 2>/dev/null || echo "")
PASSED=$(jq -r '.passed // empty' docs/test_generator/ci/heal-result.json 2>/dev/null || echo "")
SKIPPED=$(jq -r '.skipped // empty' docs/test_generator/ci/heal-result.json 2>/dev/null || echo "")
FAILED=$(jq -r '.failed // empty' docs/test_generator/ci/heal-result.json 2>/dev/null || echo "")
# Distinct marker from the pr_back PR-link sticky, and we POST a NEW comment EVERY run
# (no patch/overwrite) so each attempt's outcome is preserved as its own entry.
{
printf '%s\n' "$MARKER"
printf '🤖 **E2E Council** — I generated E2E tests for #%s but could **not** get them passing, so **no test PR was opened** (working code only).\n\n' "$SRC_PR"
printf '**Root cause:** %s\n\n' "$CAUSE"
printf '[View run + logs](%s)\n' "$RUN_URL"
printf '%s\n' "<!-- e2e-council-report -->"
printf '### 🤖 E2E Council — no test PR opened\n\n'
printf -- '- **Outcome:** %s\n' "$OUTCOME"
[ -n "$FEATURE" ] && { printf -- '- **Feature:** %s' "$FEATURE"; [ -n "$AREA" ] && printf ' (area: %s)' "$AREA"; printf '\n'; }
[ -n "${PASSED}${SKIPPED}${FAILED}" ] && printf -- '- **Tests:** %s passed · %s parked (fixme) · %s failed\n' "${PASSED:-0}" "${SKIPPED:-0}" "${FAILED:-0}"
printf -- '- **Why:** %s\n' "$CAUSE"
[ -n "$EXTRA" ] && printf '%s\n' "$EXTRA"
printf -- '- **Run:** [view logs](%s)\n' "$RUN_URL"
} > /tmp/healfail.md
CID=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${SRC_PR}/comments" \
--jq "[.[] | select(.body | startswith(\"${MARKER}\"))][0].id // empty")
if [ -n "$CID" ]; then
URL=$(gh api -X PATCH "repos/${GITHUB_REPOSITORY}/issues/comments/${CID}" -F body=@/tmp/healfail.md --jq '.html_url')
else
URL=$(gh api -X POST "repos/${GITHUB_REPOSITORY}/issues/${SRC_PR}/comments" -F body=@/tmp/healfail.md --jq '.html_url')
fi
echo "::notice::Posted heal-failure comment with root cause: $URL"
URL=$(gh api -X POST "repos/${GITHUB_REPOSITORY}/issues/${SRC_PR}/comments" -F body=@/tmp/healfail.md --jq '.html_url')
echo "::notice::Posted no-PR report comment: $URL"
# ---------------------------------------------------------------------------
# JOB 4 — Open the test PR — ONLY if Job 3 healed the tests to passing. The branch +
@ -1041,7 +1220,8 @@ jobs:
needs.triage.outputs.skip == 'false' &&
needs.triage.outputs.needs_e2e == 'true' &&
needs.triage.outputs.author_allowed == 'true' &&
needs.verify_heal.outputs.heal_status == 'passing'
needs.verify_heal.outputs.heal_status == 'passing' &&
needs.verify_heal.outputs.quality_status == 'pass'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
@ -1114,7 +1294,35 @@ jobs:
[ -n "$SLUG" ] || SLUG="feature" # sanitize LLM-written slug before it hits the PR title
SPEC=$(jq -r '.spec_path // "(see branch)"' meta/run-context.json 2>/dev/null || echo "(see branch)")
# Reviewer-facing summary table of the test cases added/changed (Engineer's test-summary.json).
# --- Reconcile the summary against the FINAL healed spec (truthful PR body). ---
# test-summary.json is written from the Engineer's INTENT; the Healer may have changed the
# spec afterward. Fetch the committed spec and DROP any row whose test title isn't actually
# in it — so the PR body can never claim tests/assertions that don't exist (#12719 §1.3).
SUMMARY=meta/test-summary.json
if [ -f "$SUMMARY" ] && [ "$(jq 'length' "$SUMMARY" 2>/dev/null || echo 0)" -gt 0 ]; then
SPEC_PATH="$SPEC"
if [ -f meta/coverage-decision.json ]; then
CDS=$(jq -r '.target_spec // empty' meta/coverage-decision.json 2>/dev/null || echo "")
[ -n "$CDS" ] && SPEC_PATH="$CDS"
fi
if echo "$SPEC_PATH" | grep -Eq '^tests/ui-testing/playwright-tests/[A-Za-z0-9._/-]+\.spec\.js$' \
&& gh api "repos/${GITHUB_REPOSITORY}/contents/${SPEC_PATH}?ref=${BRANCH}" --jq '.content' 2>/dev/null | tr -d '\n' | base64 -d > /tmp/final_spec.js; then
KEPT='[]'
while IFS= read -r row; do
T=$(printf '%s' "$row" | jq -r '.title // empty')
if [ -n "$T" ] && grep -Fq "$T" /tmp/final_spec.js; then
KEPT=$(printf '%s' "$KEPT" | jq --argjson r "$row" '. + [$r]')
else
echo "::warning::Dropping phantom summary row (no matching test in the final spec): ${T:-<empty>}"
fi
done < <(jq -c '.[]' "$SUMMARY")
printf '%s' "$KEPT" > "$SUMMARY"
else
echo "::notice::Could not fetch the final spec to reconcile the summary — leaving it as-is."
fi
fi
# Reviewer-facing summary table of the test cases added/changed (reconciled to the final spec).
TABLE=""
if [ -f meta/test-summary.json ] && [ "$(jq 'length' meta/test-summary.json 2>/dev/null || echo 0)" -gt 0 ]; then
TABLE=$(jq -r '"\n\n**Test cases added / changed:**\n\n| Test case | Type | What it verifies |\n|---|---|---|\n" + ([.[] | "| `\(.title)` | \(.action) | \(.verifies) |"] | join("\n"))' meta/test-summary.json 2>/dev/null || echo "")
@ -1179,7 +1387,8 @@ jobs:
needs.triage.outputs.needs_e2e == 'true' &&
needs.triage.outputs.author_allowed == 'true' &&
needs.generate.outputs.has_changes == 'true' &&
needs.verify_heal.outputs.heal_status == 'passing'
needs.verify_heal.outputs.heal_status == 'passing' &&
needs.verify_heal.outputs.quality_status == 'pass'
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
@ -1304,7 +1513,8 @@ jobs:
needs.triage.outputs.needs_e2e == 'true' &&
needs.triage.outputs.author_allowed == 'true' &&
needs.generate.outputs.has_changes == 'true' &&
needs.verify_heal.outputs.heal_status == 'passing'
needs.verify_heal.outputs.heal_status == 'passing' &&
needs.verify_heal.outputs.quality_status == 'pass'
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:

45
.opencode/agents/e2e_ci_council_of_agents/e2e-ci-analyst.md
View File

@ -47,6 +47,41 @@ disabled.
---
## PHASE 1.5: Wiring trace — is each behavior actually REACHABLE? (do this EVERY time)
A `v-if`/`v-else-if` only matters if its condition can ever be **true**. The biggest cause of a
generated test that can't pass is a behavior that's **gated on state nothing ever sets** — the
component exists but the feature isn't wired. **No feature is isolated** — its pieces span
components, composables, stores, and sometimes other (already-merged) PRs. So for **every key
user-facing behavior** (especially the headline one), trace the gating state **backward across the
whole codebase from `main`**, not just the diff:
1. Find the condition that renders it, e.g. `v-else-if="isQueryError"`.
2. Find what that resolves to, e.g. `isQueryError = QUERY_ERROR_CODES.has(errorCode)`, and where
`errorCode` comes from (a prop? store? `searchObj.data.errorCode`?).
3. **Grep the WHOLE app for every place that state is assigned** — does ANY active code path set it
to a triggering value?
```bash
grep -rn "searchObj.data.errorCode\s*=" web/src/ # is it set, or only reset to 0?
```
Watch for the traps: the assignment is **commented out**, **hardcoded to a non-triggering value**
(`:error-code="0"`), set **only on a different path** (streaming vs classic, histogram vs main),
or set **only in a sibling/follow-up PR**. Check the consumer bindings too (what prop value the
parent passes).
For each behavior, classify it in the design doc as one of:
- **WIRED** — at least one real path sets the gating state. **Name that exact path** (file:line) so
the Engineer writes the test to exercise *that* path → it goes green.
- **UNWIRED (feature-incomplete)** — NO active path sets it (commented out / hardcoded / absent
everywhere). Record the precise file:line evidence. The Architect will plan this as a parked
`test.fixme`, not a test that will fail.
This front-loads the "is it wired" decision into planning — the Engineer then writes green tests for
WIRED behaviors and honest `fixme`s for UNWIRED ones, so the Healer never has to discover this after
a wasted iteration.
---
## PHASE 2: Write the Feature Design Document
Write to: `docs/test_generator/features/<feature_slug>-feature.md`
@ -81,6 +116,12 @@ Use this structure:
#### Actions
| Action | Trigger | Result |
## Behavior Reachability (wiring trace — Phase 1.5)
| Behavior | Gating condition | State source (file:line) | Status |
|----------|------------------|--------------------------|--------|
| Fix-query card on SQL error | `v-else-if="isQueryError"` | `searchObj.data.errorCode` — set at `useX.ts:NNN` | **WIRED** (test this path) |
| <behavior> | <condition> | <commented out / hardcoded 0 / never set> | **UNWIRED** (feature-incomplete — fixme) |
## User Workflows
### Workflow 1: <primary use case>
**Steps:** 1. <action> → <response> ...
@ -112,6 +153,10 @@ DO:
- Extract **real** `data-test` attributes. If one doesn't exist for a needed element, mark it
`NEEDS SELECTOR` so the Engineer adds a robust fallback rather than fabricating one.
- Map flows by following the code; document all visibility conditions and states.
- **Trace wiring (Phase 1.5) for every key behavior** — grep the whole app for what sets the gating
state, and mark each behavior WIRED (name the path) or UNWIRED (feature-incomplete, with evidence).
This is the single highest-value thing you do: it's what lets the Engineer write green tests for
the paths that work and honest `fixme`s for the ones that don't.
DO NOT:
- Guess selectors, assume functionality, skip edge cases, or invent test cases.

28
.opencode/agents/e2e_ci_council_of_agents/e2e-ci-architect.md
View File

@ -69,8 +69,35 @@ already (partly) covered and choose the **least-duplicative** action.
- `needs_registration`: **true ONLY for `action: new`** (a brand-new spec must be added to
playwright.yml). For `append`/`extend` the file is already registered → **false**.
### One spec = one feature area (no Frankenstein specs)
If the diff spans **multiple test areas** (e.g. a Logs change AND an Alerts/incidents change),
do **not** cram both into one spec. Pick the **dominant feature** for this run, put its tests in
the correct area folder with that area's tags (a Logs feature → `Logs/…` + `@logs`; an Alerts
feature → `Alerts/…` + `@alerts`), and **note the un-covered secondary area** in your test plan so
a human knows it still needs tests. A spec whose folder/tags don't match its tests (Alerts tests
inside a `Logs/` spec) is a defect.
### De-duplicate scenarios
Do not emit several P-cases that perform the **same action and assert the same thing** (e.g. four
tests that all fire the same error query and check "error visible"). Each scenario must add
distinct coverage; merge near-duplicates into one.
---
## Honour the Analyst's wiring map (WIRED vs UNWIRED)
Read the **Behavior Reachability** table in the feature design doc. Plan each behavior by its status —
this is how we produce green tests AND avoid both green-washing and over-blocking:
- **WIRED** → plan a normal test that exercises **the exact path the Analyst named** (the one that
actually sets the gating state). This test should go green.
- **UNWIRED (feature-incomplete)** → do **NOT** plan a test that will fail or a weakened/negative one.
Plan it as a **`fixme` placeholder** with the Analyst's evidence (file:line of the missing/commented
wiring) so the Engineer writes `test.fixme('<behavior> — not wired: <evidence>')`. Mark the scenario
`Wiring: UNWIRED`. The pipeline surfaces these as a feature-gap report, not a failed run.
If **every** headline behavior is UNWIRED, the feature is genuinely incomplete — say so plainly in the
plan; do not pad with tautological/always-green filler just to have "tests."
## Write the Test Plan
Write to: `docs/test_generator/test-plans/<feature_slug>-test-plan.md`
@ -96,6 +123,7 @@ Structure:
### P0 — <critical path scenario>
#### <Test Case Name>
**Objective:** <what it verifies>
**Wiring:** WIRED (path: `file:line`) | UNWIRED (fixme — evidence: `file:line`)
**Pre-conditions:** <setup>
**Steps:** 1. <action> 2. <action>
**Expected Results:** - <outcome>

29
.opencode/agents/e2e_ci_council_of_agents/e2e-ci-engineer.md
View File

@ -50,6 +50,19 @@ selectors in the spec), exactly as for new specs. Preserve `mode: 'parallel'`
file to serial. If the existing file is somehow `serial`, leave its mode as-is but keep your
added test independent.
### Honour each scenario's `Wiring:` marker (from the test plan)
The Architect tags every scenario WIRED or UNWIRED (from the Analyst's reachability trace):
- **WIRED** → write a normal, real test that exercises the **named working path** (the one that sets
the gating state). It should pass.
- **UNWIRED (feature-incomplete)** → write the test as **`test.fixme('<name> — not wired: <evidence file:line>')`**
with the **real assertion body kept intact** (so it goes green when the feature is finished). Do NOT
weaken it, invert it, or turn it into a tautology, and do NOT write a passing test that asserts the
feature is absent. A `fixme` with evidence is the honest representation of a gap.
This is the balance: green tests for what works + honest `fixme`s for what doesn't, in ONE spec — so a
PR opens with real coverage instead of being blocked, and the Healer never has to discover gaps later.
Only when **every** scenario is UNWIRED is the feature genuinely incomplete (no PR; the plan says so).
> **Minimal-diff rule for append/extend:** the goal is the smallest possible change to the
> existing file — a reviewer should see only the added/changed test, nothing else.
@ -102,6 +115,18 @@ test.describe("<feature_title> testcases", () => {
- **Reuse** existing page objects in `tests/ui-testing/pages/`. Only add new methods when none
fit, and add them to the **existing** page file for that area. Define new locators at the
**top** of the page file.
- **No dead scaffolding** — add ONLY page-object methods the spec actually calls. Do not emit
speculative `expectXVisible()` helpers "for completeness" if no test uses them; they're noise
for human reviewers and rot. Every method you add must be called by at least one test.
- **Assertions must be real and match the test name.** No tautologies (`toBeGreaterThanOrEqual(0)`
on a count, `toBeTruthy()` on always-true). Don't bury the only assertion inside an `if` whose
`else` just logs — the test must assert unconditionally. If the test name promises a behaviour
(e.g. "fix-query card appears"), assert exactly that.
- **Never wrap an assertion in a `try/catch` that swallows the failure.** A `catch` that logs and
`return`s (or just continues) makes the test **pass even when the assertion failed** — a silent
escape hatch. Let assertions throw. If you must `try/catch` for genuine optional/cleanup steps,
keep the real `expect(...)` **outside** the `try`, or `throw` in the `catch`. (A deterministic gate
rejects assertion-in-try with a swallowing catch.)
```javascript
this.newButton = '[data-test="feature-new-btn"]';
async clickNewButton() { await this.page.locator(this.newButton).click(); }
@ -191,7 +216,9 @@ Sentinel's bar so the audit passes on attempt 1.
**WARNINGS — avoid these too (Sentinel reports them):**
- Locators must be declared at the **top** of each page-object file (as properties), not inline.
- No brittle selectors (xpath, `nth-child`, framework-generated classes).
- ≤3 `waitForTimeout` per test — prefer `waitForLoadState`/`toBeVisible`.
- **Do not use `waitForTimeout(<n>)` to synchronize** — it's flaky by construction. Wait on the
thing you care about with auto-retrying `await expect(...).toBeVisible()` / `waitForLoadState`.
A fixed sleep is acceptable only for a rare deliberate settle, never as the primary wait.
- Use `PageManager` (`pm.…`) for all interactions; reuse existing page objects.
- If a test creates data, add cleanup in `tests/ui-testing/playwright-tests/cleanup.spec.js`.

67
.opencode/agents/e2e_ci_council_of_agents/e2e-ci-healer.md
View File

@ -77,9 +77,61 @@ For iteration `i` in 1..3:
4. **Apply the fix** to the page object or spec (keep selectors out of the spec — the Sentinel
re-audits afterward). Continue to iteration `i+1`.
Keep fixes minimal and framework-compliant. Never weaken assertions or wrap tests in
always-true conditionals to force a pass — that fails the Sentinel re-audit and defeats the
purpose.
Keep fixes minimal and framework-compliant.
---
## ⛔ HARD RULES — you may fix the TEST, never weaken what it CHECKS
Your job is "green **for the right reason**." A deterministic gate now compares the spec's
assertions before vs. after you heal, and a green run that tests nothing is **rejected** — so
weakening is not a shortcut, it's a guaranteed block.
### 🚫 SCOPE LOCK — you may ONLY touch files under `tests/ui-testing/`
You fix **tests**, not the product. This is an absolute boundary, enforced deterministically (a
gate fails the run + reverts your changes if you cross it — so crossing it is pointless):
- **NEVER edit, create, or delete any file outside `tests/ui-testing/`** — in particular **never**
touch product code (`src/**`, `web/src/**`, `*.rs`, `*.vue`, `*.ts` outside the test dir),
config, `Cargo.*`, `package.json`, `.github/**`, `.opencode/**`, or `playwright.yml`.
- **NEVER build, compile, or restart the product** — no `cargo build`/`cargo check`/`cargo run`,
no `npm run build`, no rebuilding/relaunching OpenObserve. The binary is already booted; you run
Playwright against it, nothing more.
- If a test fails because the **product/feature is broken or incomplete**, that is **NOT yours to
fix** — report `status: "feature-incomplete"` with evidence (below). Patching `src/` or `web/src/`
to make a test pass produces a fake green that never reproduces (those edits are not committed)
and **will be rejected**.
**You MAY** fix (ONLY within `tests/ui-testing/`): selectors, locators (in page objects),
timing/waits, navigation, setup, data freshness, test independence. These make a test *run* correctly.
**You may NOT**, to force a pass:
- **Delete an assertion** or remove a test. (Assertion count must not drop.)
- **Invert an assertion** — e.g. change `expect(x).toBeVisible()` → `not.toBeVisible()` /
`toBeHidden()`. If the thing that *should* appear doesn't, that is a **finding**, not a heal.
- **Make an assertion conditional** — wrapping the real `expect(...)` in an `if (...)` whose
`else` just logs, so the test can pass having checked nothing.
- **Weaken to a tautology** — `toBeGreaterThanOrEqual(0)` on a count, `toBeTruthy()` on an
always-true value, `expect(true)…`.
A failing assertion that is *correct about what the feature should do* means the **feature**,
not the test, is the problem. Do not heal that away.
### When the feature is genuinely not wired → status `feature-incomplete` (the honest exit)
If, after diagnosing, a test fails because the **product behaviour it asserts does not exist in
this build** (the component never renders, the code path is commented out / stubbed, the API
returns nothing on the active path) — **do NOT weaken the test to pass.** Instead:
1. Mark only the genuinely-blocked test(s) `test.fixme('<one-line reason>, see #<issue>')` — keep
the assertion body intact so it goes green when the feature is finished. Leave all other
tests asserting normally.
2. Write `heal-result.json` with `status: "feature-incomplete"` and a concrete `evidence` string
naming the exact file:line that proves the gap (e.g. `errorCode is never propagated —
useSearchBar.ts:1068 is commented out, so the fix-query card never renders`).
This `feature-incomplete` report is the single most valuable thing you can produce — it catches a
real product gap instead of hiding it. The workflow will NOT open a normal PR; it surfaces your
evidence to the developer.
**Preserve full parallelism.** The Engineer writes every describe with `mode: 'parallel'`. Do
**not** switch it to `mode: 'serial'` (or remove parallelism) as a quick fix — a flaky parallel
@ -111,7 +163,14 @@ ever do it, state the concrete reason in the execution report.
```json
{ "status": "passing", "iterations": 2, "total": 3, "passed": 3, "failed": 0 }
```
`status` is `"passing"` only if every test passed; otherwise `"failing"`.
`status`:
- `"passing"` — every test passed **honestly** (no assertions deleted/inverted/conditionalized).
- `"feature-incomplete"` — a test can't pass because the **feature isn't wired in this build**;
you parked it `test.fixme(...)` and MUST include an `evidence` field (the file:line proving
the gap). Example: `{ "status": "feature-incomplete", "evidence": "useSearchBar.ts:1068 commented out — fix-query card never renders", "total": 5, "passed": 3, "skipped": 2 }`.
- `"failing"` — still red for an ordinary reason (flake, selector you couldn't resolve, etc.).
Never report `"passing"` if you weakened an assertion to get there — the deterministic gate will
catch it and reject the run anyway.
After healing, the workflow runs the **Sentinel re-audit** on any files you changed. The PR is
opened normally if `status == "passing"` and the re-audit is PASS; otherwise as **draft** with

25
.opencode/agents/e2e_ci_council_of_agents/e2e-ci-sentinel.md
View File

@ -45,13 +45,28 @@ If `run-context.json` is missing or `skip: true`, stop.
exceptions. (Page object files are exempt — see scope above.)
2. **Missing assertions** — every test must have ≥1 real assertion.
3. **Vacuous / always-pass assertions** — `expect(true).toBe(true)`, `expect(1).toBe(1)`,
`if (visible) {...} else { expect(true).toBe(true) }`. Check **what** is asserted, not just
that an `expect` exists.
4. **`console.log`** present (use `testLogger`).
5. **Missing `await`** on async operations.
6. **Hardcoded credentials** — `password` / `secret` / `apiKey` / `token` literals (only
**`toBeGreaterThanOrEqual(0)` / `toBeGreaterThan(-1)` on a count** (always true),
`.toBeTruthy()` on a guaranteed-truthy value. Check **what** is asserted, not just that an
`expect` exists.
4. **Conditional-only assertions** — a test where **every** `expect(...)` sits inside an `if (...)`
whose `else` does not also assert (so the test can reach its end having checked nothing). The
real assertion must not be skippable.
5. **Negative-only on the feature under test** — if the spec's feature is about a component that
should APPEAR (e.g. a "fix-query card" for a query-error feature) and **every** assertion about
that component is `not.toBeVisible()` / `toBeHidden()`, BLOCK: "test asserts the feature is
absent — it likely certifies an incomplete feature and will break when it's finished."
6. **Test-name / assertion mismatch** — the title claims something the body never checks (title
says "fix-query card" but the card is never asserted visible; title says "while loading" but no
assertion runs during a loading state). The test must actually verify what its name promises.
7. **`console.log`** present (use `testLogger`).
8. **Missing `await`** on async operations.
9. **Hardcoded credentials** — `password` / `secret` / `apiKey` / `token` literals (only
`process.env.*` is allowed).
> Checks 3–6 are the **assertion-quality** rules. A deterministic gate in the workflow also enforces
> 3 and 4 (and rejects any heal that deleted/inverted/skipped assertions), so do not rely on it —
> apply 3–6 yourself: a green spec that tests nothing is the failure mode we most need you to catch.
## AUTO-FIX (apply silently, then continue — these do NOT fail the run)
- Missing `testLogger` import → add it.

18
.opencode/agents/e2e_ci_council_of_agents/e2e-ci-triage.md
View File

@ -146,8 +146,22 @@ For an OSS change that needs E2E, derive:
}
```
`docs/test_generator/ci/triage.json` — the same fields plus a human-readable `rationale`
string explaining the decision (this is what the workflow posts as the dry-run PR comment).
`docs/test_generator/ci/triage.json` — the same fields **plus a structured `rationale` OBJECT**
(the workflow renders it into the PR comment in a FIXED format — you supply only the prose per
field, never the layout). Emit exactly these keys, each a concise 1–3 sentence explanation:
```json
"rationale": {
"edition": "why OSS vs ENT — the file PATHS you checked + any enterprise keywords (or their absence)",
"skip": "why skip / why not — is the change user-facing? any skip marker/label/comment?",
"e2e": "why E2E is or isn't warranted — what user-facing behavior needs verifying",
"area": "why this area — where the bulk of the UI changes live + where sibling specs are",
"group": "why this playwright_group (matrix shard)",
"existing_tests": "what existing test changes are in the diff and whether they cover the NEW feature (maintenance vs real coverage)"
}
```
Write **every** key (use a short note like "n/a — skipped" if a field doesn't apply). Keep each
value plain prose — do NOT add your own bold labels or headers; the workflow adds those. The
`rationale` object only needs to be in `triage.json` (not `run-context.json`).
## Decision discipline