ci(test): e2e-council enrichment — summary table, auto-review, + CLA bot-identity fix (#12684)
Post-PR "enrichment" follow-ups for the E2E test bot, plus a CLA fix. (Was stacked on #12682; rebased onto `main` after that merged.) ## What this adds **1. Test-case summary table** When the test PR opens, the bot posts a **table of the test cases it added/changed** — in the test PR body *and* the "E2E Council" comment on the source PR: | Test case | Type | What it verifies | |---|---|---| | `should …` | new / append / extend | one-line intent | (Engineer emits `test-summary.json`; `pr_back` renders it. No table when there's nothing to add.) **2. Auto-address the AI review (OFF by default)** New final job `review_address`, gated by repo variable **`E2E_AUTO_REVIEW`** (default off). After the test PR opens it waits ~6 min for the repo's AI review, then a **conservative "Responder" agent**: - **validates** each finding (the bot is noisy — default is to *not* change code), - fixes **only clearly-valid** ones in `tests/ui-testing` (never `.github` / `.opencode` / `playwright.yml` / source), - pushes tests-only fixes (the repo's Playwright CI re-validates the PR), - posts a rationale on the test PR and minimizes the bot comment. One bounded pass. Scoped `GITHUB_TOKEN` only. **To enable:** create repo variable `E2E_AUTO_REVIEW=true`. **3. CLA fix — commit as `github-actions[bot]`** The pipeline previously committed generated tests as `openobserve-bot <bot@openobserve.ai>`, which isn't a GitHub account → CLA assistant couldn't match/sign it and blocked the test PR. Now it commits as the canonical **`github-actions[bot]`** identity (a real bot account GitHub resolves via its noreply email), across generate + verify_heal + review_address. > **Admin action:** allowlist `github-actions[bot]` in CLA assistant, then `recheck`. --- <details> <summary>Design notes</summary> - Responder is deliberately conservative: a wrong "fix" that breaks a passing test is worse than leaving a nitpick. It dismisses stale/incorrect/style findings. - Both auto-features are flag-gated OFF (same posture as `E2E_AUTO_ON_MERGE`) — auto-acting warrants opt-in. - No heavy re-build in `review_address`; the repo's existing Playwright CI re-validates the pushed fix. - CodeQL untrusted-checkout on the new `review_address` checkout (#416) dismissed — same allowlist false positive as prior PRs. </details> 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Shrinath Rao
GitHub
parent
5c286261e2
commit
93d5e5b90d
|
|
@ -590,8 +590,8 @@ jobs:
|
|||
SLUG=$(jq -r '.feature_slug // "feature"' docs/test_generator/ci/run-context.json | tr -cd 'a-zA-Z0-9_-')
|
||||
[ -n "$SLUG" ] || SLUG="feature" # LLM-written; strip anything but [a-zA-Z0-9_-] before it hits the commit msg
|
||||
BRANCH="${WORK_BRANCH}"
|
||||
git config user.name "openobserve-bot"
|
||||
git config user.email "bot@openobserve.ai"
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
if [ "$REUSE" = "true" ]; then
|
||||
echo "Reuse mode: extending the existing $BRANCH (already checked out)."
|
||||
else
|
||||
|
|
@ -909,8 +909,8 @@ jobs:
|
|||
WORK_BRANCH: ${{ needs.triage.outputs.work_branch }}
|
||||
run: |
|
||||
BRANCH="${WORK_BRANCH}"
|
||||
git config user.name "openobserve-bot"
|
||||
git config user.email "bot@openobserve.ai"
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
# The anti-hijack restore staged .opencode/opencode.jsonc — unstage so healer commits carry
|
||||
# ONLY test changes (pipeline files must never reach the test PR).
|
||||
git restore --staged .opencode opencode.jsonc 2>/dev/null || true
|
||||
|
|
@ -1079,8 +1079,15 @@ jobs:
|
|||
[ -n "$SLUG" ] || SLUG="feature" # sanitize LLM-written slug before it hits the PR title
|
||||
SPEC=$(jq -r '.spec_path // "(see branch)"' meta/run-context.json 2>/dev/null || echo "(see branch)")
|
||||
|
||||
# Reviewer-facing summary table of the test cases added/changed (Engineer's test-summary.json).
|
||||
TABLE=""
|
||||
if [ -f meta/test-summary.json ] && [ "$(jq 'length' meta/test-summary.json 2>/dev/null || echo 0)" -gt 0 ]; then
|
||||
TABLE=$(jq -r '"\n\n**Test cases added / changed:**\n\n| Test case | Type | What it verifies |\n|---|---|---|\n" + ([.[] | "| `\(.title)` | \(.action) | \(.verifies) |"] | join("\n"))' meta/test-summary.json 2>/dev/null || echo "")
|
||||
fi
|
||||
|
||||
printf '🤖 Auto-generated E2E tests for #%s — **healed to passing** against a live OpenObserve before this PR was opened.\n\n- Branch `%s` (stacked on `%s`), so the tests run against the feature code and pass before it merges.\n- Spec: `%s`\n\nThe repo Playwright CI re-validates here. Review is for **quality + coverage** — the tests already pass.\n\n_Generated by the E2E Council pipeline._\n' \
|
||||
"$SRC_PR" "$BRANCH" "$BASE_REF" "$SPEC" > /tmp/pr-body.md
|
||||
printf '%s\n' "$TABLE" >> /tmp/pr-body.md
|
||||
|
||||
# Idempotent: re-runs force-update the branch, so an existing PR auto-reflects it.
|
||||
EXISTING=$(gh pr list --repo "$GITHUB_REPOSITORY" --head "$BRANCH" --state open --json number --jq '.[0].number // empty')
|
||||
|
|
@ -1096,8 +1103,9 @@ jobs:
|
|||
|
||||
[ -n "$TEST_PR" ] || { echo "::warning::could not resolve test PR number"; exit 0; }
|
||||
MARKER="<!-- e2e-council-testpr -->"
|
||||
printf '%s\n🤖 **E2E Council** → healed, passing E2E tests are in #%s (branch `%s`).' \
|
||||
printf '%s\n🤖 **E2E Council** → healed, passing E2E tests are in #%s (branch `%s`).\n' \
|
||||
"$MARKER" "$TEST_PR" "$BRANCH" > /tmp/sticky.md
|
||||
printf '%s\n' "$TABLE" >> /tmp/sticky.md
|
||||
CID=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${SRC_PR}/comments" \
|
||||
--jq "[.[] | select(.body | startswith(\"${MARKER}\"))][0].id // empty")
|
||||
if [ -n "$CID" ]; then
|
||||
|
|
@ -1118,3 +1126,116 @@ jobs:
|
|||
if gh pr edit "$TEST_PR" --repo "$GITHUB_REPOSITORY" --add-reviewer "$U" >/dev/null 2>&1; then
|
||||
echo "requested review from @$U"; else echo "::notice::could not request review from @$U (self/perm)"; fi
|
||||
done
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# JOB 5 — Auto-address the AI review on the test PR (OPTIONAL, OFF by default).
|
||||
# After the test PR is opened, wait for the repo's `ai-code-review`, then a CONSERVATIVE Responder
|
||||
# validates the findings, fixes ONLY clearly-valid ones in the test files, pushes (tests-only), and
|
||||
# posts a rationale + minimizes the bot comment. ONE bounded pass. Gated by `vars.E2E_AUTO_REVIEW`.
|
||||
# The repo's Playwright CI re-validates the pushed fix (the spec is registered on the PR). Uses the
|
||||
# scoped GITHUB_TOKEN only (no ORG_ADMIN_TOKEN — this job runs an agent over test/feature code).
|
||||
# ---------------------------------------------------------------------------
|
||||
review_address:
|
||||
needs: [triage, generate, verify_heal, pr_back]
|
||||
if: >
|
||||
vars.E2E_AUTO_REVIEW == 'true' &&
|
||||
needs.triage.outputs.dry_run == 'false' &&
|
||||
needs.triage.outputs.skip == 'false' &&
|
||||
needs.triage.outputs.needs_e2e == 'true' &&
|
||||
needs.triage.outputs.author_allowed == 'true' &&
|
||||
needs.generate.outputs.has_changes == 'true' &&
|
||||
needs.verify_heal.outputs.heal_status == 'passing'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: write # push tests-only review fixes to autoe2e/pr-<n>
|
||||
pull-requests: write
|
||||
issues: write
|
||||
steps:
|
||||
- name: Checkout the autoe2e branch (test files to fix)
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ needs.triage.outputs.work_branch }}
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
|
||||
- name: Restore agent files from a trusted ref (anti-hijack)
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
REF_NAME: ${{ github.ref_name }}
|
||||
run: |
|
||||
RESTORE_REF=main
|
||||
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then RESTORE_REF="$REF_NAME"; fi
|
||||
git fetch origin "$RESTORE_REF" --depth=1
|
||||
git checkout "origin/$RESTORE_REF" -- .opencode opencode.jsonc
|
||||
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 22
|
||||
- name: Install OpenCode
|
||||
run: |
|
||||
npm install -g opencode-ai@1.17.7 || { echo "::error::OpenCode install failed"; exit 1; }
|
||||
|
||||
- name: Wait for the AI review + capture it
|
||||
id: review
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
WORK_BRANCH: ${{ needs.triage.outputs.work_branch }}
|
||||
run: |
|
||||
TEST_PR=$(gh pr list --repo "$GITHUB_REPOSITORY" --head "$WORK_BRANCH" --state open --json number --jq '.[0].number // empty')
|
||||
[ -n "$TEST_PR" ] || { echo "::notice::no open test PR for $WORK_BRANCH — nothing to address."; echo "found=false" >> "$GITHUB_OUTPUT"; exit 0; }
|
||||
echo "test_pr=$TEST_PR" >> "$GITHUB_OUTPUT"
|
||||
MARKER="<!-- ai-code-review -->"
|
||||
mkdir -p docs/test_generator/ci
|
||||
# Poll up to ~6 min for the review bot's comment (it lands a couple minutes after the PR opens).
|
||||
for i in $(seq 1 12); do
|
||||
BODY=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${TEST_PR}/comments" --jq "[.[] | select(.body | startswith(\"${MARKER}\"))][-1].body // empty")
|
||||
if [ -n "$BODY" ]; then
|
||||
printf '%s' "$BODY" > docs/test_generator/ci/ai-review.md
|
||||
echo "found=true" >> "$GITHUB_OUTPUT"
|
||||
echo "::notice::Captured AI review on test PR #$TEST_PR (after ${i} polls)."
|
||||
exit 0
|
||||
fi
|
||||
sleep 30
|
||||
done
|
||||
echo "::notice::No AI review appeared within the poll window — skipping addressal."
|
||||
echo "found=false" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Responder — validate + fix clearly-valid findings
|
||||
if: steps.review.outputs.found == 'true'
|
||||
env:
|
||||
DEEPSEEK_API_KEY_E2E: ${{ secrets.DEEPSEEK_API_KEY_E2E }}
|
||||
run: |
|
||||
opencode run --agent e2e-ci-responder --model "$PIPELINE_MODEL" \
|
||||
"Read docs/test_generator/ci/ai-review.md (the repo's AI review of this test PR). Validate each finding conservatively; fix ONLY clearly-valid ones in tests/ui-testing (NEVER touch .github, .opencode, opencode.jsonc, playwright.yml, or source). Write docs/test_generator/ci/review-response.md with what you fixed / dismissed / noted."
|
||||
|
||||
- name: Push fixes (tests only) + post response + minimize the bot review
|
||||
if: steps.review.outputs.found == 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
WORK_BRANCH: ${{ needs.triage.outputs.work_branch }}
|
||||
TEST_PR: ${{ steps.review.outputs.test_pr }}
|
||||
run: |
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
# Restore staged pipeline files; commit ONLY test changes (never .opencode/workflow).
|
||||
git restore --staged .opencode opencode.jsonc 2>/dev/null || true
|
||||
git add tests/ui-testing
|
||||
if ! git diff --cached --quiet; then
|
||||
git commit -m "test(autoe2e): address AI review on #${TEST_PR} (auto)"
|
||||
git push "https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" "HEAD:refs/heads/${WORK_BRANCH}"
|
||||
echo "Pushed review fixes to ${WORK_BRANCH} (repo Playwright CI re-validates the PR)."
|
||||
else
|
||||
echo "No code changes from the Responder (all findings dismissed/noted)."
|
||||
fi
|
||||
# Post the rationale reply on the test PR.
|
||||
if [ -f docs/test_generator/ci/review-response.md ]; then
|
||||
gh api -X POST "repos/${GITHUB_REPOSITORY}/issues/${TEST_PR}/comments" -F body=@docs/test_generator/ci/review-response.md >/dev/null \
|
||||
&& echo "Posted review response on #${TEST_PR}."
|
||||
fi
|
||||
# Minimize the AI review comment (same github-actions[bot] author → permitted). Best-effort.
|
||||
NODE=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${TEST_PR}/comments" --jq "[.[] | select(.body | startswith(\"<!-- ai-code-review -->\"))][-1].node_id // empty")
|
||||
if [ -n "$NODE" ]; then
|
||||
gh api graphql -f query='mutation($id:ID!){minimizeComment(input:{subjectId:$id,classifier:RESOLVED}){minimizedComment{isMinimized}}}' -f id="$NODE" >/dev/null 2>&1 \
|
||||
&& echo "Minimized the AI review comment." || echo "::notice::could not minimize the AI review (best-effort)."
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -209,6 +209,20 @@ Treat this as a gate on *yourself*: a spec that trips any CRITICAL above is not
|
|||
(`mkdir -p` first) listing: spec path, page objects touched (new methods/locators), the
|
||||
`playwright.yml` group + filename to register, and any open risks (e.g. `NEEDS SELECTOR`
|
||||
items the Analyst flagged and how you handled them).
|
||||
5. A machine-readable **test summary** → `docs/test_generator/ci/test-summary.json`: one entry per
|
||||
test case you **added or changed this run** (skip unchanged sibling tests), so the PR-back job
|
||||
can render a reviewer-facing table. Shape:
|
||||
```json
|
||||
[
|
||||
{ "title": "should open the demo page", "action": "new", "verifies": "demo page loads and the header is visible" },
|
||||
{ "title": "should filter rows", "action": "append", "verifies": "the filter control updates the results table" }
|
||||
]
|
||||
```
|
||||
`action` = this case's coverage action (`new` for a new spec's tests, `append`/`extend` for ones
|
||||
added/modified on an existing spec). `verifies` = one concise human sentence. For `action: none`
|
||||
write `[]`. Keep `title` byte-identical to the `test("…")` name in the spec.
|
||||
> It must be valid JSON. `pr_back` renders it as a table but **skips it gracefully** (no table) if
|
||||
> the file is missing, empty `[]`, or unparseable — so never block generation on it.
|
||||
|
||||
> **Where outputs go:** you run in an ephemeral CI runner with no commit access. Your files are
|
||||
> uploaded as a build artifact and the **PR-back job (Job 4) commits them** to a `test/<slug>`
|
||||
|
|
|
|||
|
|
@ -0,0 +1,78 @@
|
|||
---
|
||||
description: "CI Responder (Phase 6, optional). Reads the repo's AI code-review on the generated test PR, VALIDATES each finding, fixes only the clearly-valid ones in the test files, and writes a rationale. Conservative — the review bot is noisy. Non-interactive."
|
||||
mode: primary
|
||||
---
|
||||
|
||||
# The Responder — AI-Review Addresser (CI, Phase 6)
|
||||
|
||||
You are **The Responder** for OpenObserve's automated E2E pipeline. After the test PR is opened, the
|
||||
repo's AI code-review bot posts findings. Your job: **read those findings, decide which are real, fix
|
||||
the real ones in the test files, and explain the rest** — then hand off a rationale the workflow
|
||||
posts back. You run non-interactively and hand off via files.
|
||||
|
||||
> **CORE PRINCIPLE — be conservative. The review bot is noisy and frequently wrong** (stale
|
||||
> re-emissions, misunderstandings of the framework/CI, style nitpicks). Your default is to **NOT
|
||||
> change code**. Only change a test when a finding is *clearly, concretely correct* and the fix is
|
||||
> obviously safe. A wrong "fix" that breaks a passing test is far worse than leaving a nitpick.
|
||||
|
||||
## Input (read first)
|
||||
|
||||
```bash
|
||||
cat docs/test_generator/ci/ai-review.md # the review bot's comment body (the findings)
|
||||
cat docs/test_generator/ci/run-context.json # feature context (spec_path, area, slug)
|
||||
cat docs/test_generator/ci/coverage-decision.json 2>/dev/null # what was generated (action, target_spec)
|
||||
```
|
||||
If `ai-review.md` is missing or empty, write an empty response (below) and stop.
|
||||
|
||||
The spec + page objects are checked out under `tests/ui-testing/`. Read the spec being reviewed and
|
||||
its page objects before judging any finding.
|
||||
|
||||
## Validate each finding (this is the real work)
|
||||
|
||||
For every finding, classify it:
|
||||
|
||||
- **VALID → fix** — a concrete, correct problem in the generated test that you can fix safely and
|
||||
minimally: e.g. a genuinely wrong/brittle selector, a missing/weak assertion, an un-awaited async
|
||||
call, a real logic error, a hardcoded credential. Fix it in the **page object / spec** (keep
|
||||
selectors out of the spec; preserve `mode: 'parallel'`; keep the change minimal).
|
||||
- **INVALID → dismiss** — wrong, stale, or not-applicable. Common cases to **reject**: claims that
|
||||
contradict the framework (`global-setup` handles auth; `pm.*` page objects; env-injected creds),
|
||||
misreadings of shell/`jq`/CI behavior, "issues" that don't exist in the current code, or findings
|
||||
about files you didn't write (`playwright.yml`, `.opencode/**`, the workflow).
|
||||
- **NITPICK → note, don't change** — style/preference/perf micro-suggestions that don't affect
|
||||
correctness. Acknowledge briefly; do not touch code for these.
|
||||
|
||||
When unsure → treat as INVALID/NITPICK (do not change code). Bias hard toward not editing.
|
||||
|
||||
## HARD limits (safety)
|
||||
|
||||
- **Edit ONLY files under `tests/ui-testing/`.** NEVER edit `.github/**`, `.opencode/**`,
|
||||
`opencode.jsonc`, `playwright.yml`, or any product/source file.
|
||||
- **Do NOT run, register, or re-generate** — you only adjust existing test files. (The repo's
|
||||
Playwright CI re-validates the PR after your push.)
|
||||
- **Do NOT weaken or delete tests** to satisfy a finding. Never replace a real assertion with a
|
||||
trivial one. If a finding would require weakening coverage, dismiss it.
|
||||
- Keep every fix **minimal and surgical** — a reviewer should see only the change the finding warranted.
|
||||
|
||||
## OUTPUT (always write this file)
|
||||
|
||||
Write `docs/test_generator/ci/review-response.md` — the comment the workflow posts on the test PR:
|
||||
|
||||
```markdown
|
||||
🤖 **E2E Council — review addressed**
|
||||
|
||||
**Fixed:**
|
||||
- <finding> → <what you changed> (`file`)
|
||||
|
||||
**Dismissed (not applicable / incorrect):**
|
||||
- <finding> → <one-line why> (e.g. "auth is handled by global-setup", "stale — not in current code")
|
||||
|
||||
**Noted (style/nitpick, no change):**
|
||||
- <finding> → <brief acknowledgement>
|
||||
```
|
||||
|
||||
Omit any section that's empty. If you changed **nothing**, say so plainly (e.g. "Reviewed the N
|
||||
findings; none required a code change — rationale below."). Keep it concise and factual.
|
||||
|
||||
Print a one-line summary (counts: fixed / dismissed / noted) and finish. Non-interactive — do not
|
||||
wait for approval, do not commit or push (the workflow commits your test-file edits).
|
||||
|
|
@ -52,6 +52,10 @@
|
|||
"e2e-ci-healer": {
|
||||
"mode": "primary",
|
||||
"prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-healer.md}"
|
||||
},
|
||||
"e2e-ci-responder": {
|
||||
"mode": "primary",
|
||||
"prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-responder.md}"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue