From 93d5e5b90d408a8b1cdb7c10bef19c96210c535b Mon Sep 17 00:00:00 2001 From: Shrinath Rao Date: Tue, 16 Jun 2026 11:16:08 +0530 Subject: [PATCH] =?UTF-8?q?ci(test):=20e2e-council=20enrichment=20?= =?UTF-8?q?=E2=80=94=20summary=20table,=20auto-review,=20+=20CLA=20bot-ide?= =?UTF-8?q?ntity=20fix=20(#12684)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-PR "enrichment" follow-ups for the E2E test bot, plus a CLA fix. (Was stacked on #12682; rebased onto `main` after that merged.) ## What this adds **1. Test-case summary table** When the test PR opens, the bot posts a **table of the test cases it added/changed** — in the test PR body *and* the "E2E Council" comment on the source PR: | Test case | Type | What it verifies | |---|---|---| | `should …` | new / append / extend | one-line intent | (Engineer emits `test-summary.json`; `pr_back` renders it. No table when there's nothing to add.) **2. Auto-address the AI review (OFF by default)** New final job `review_address`, gated by repo variable **`E2E_AUTO_REVIEW`** (default off). After the test PR opens it waits ~6 min for the repo's AI review, then a **conservative "Responder" agent**: - **validates** each finding (the bot is noisy — default is to *not* change code), - fixes **only clearly-valid** ones in `tests/ui-testing` (never `.github` / `.opencode` / `playwright.yml` / source), - pushes tests-only fixes (the repo's Playwright CI re-validates the PR), - posts a rationale on the test PR and minimizes the bot comment. One bounded pass. Scoped `GITHUB_TOKEN` only. **To enable:** create repo variable `E2E_AUTO_REVIEW=true`. **3. CLA fix — commit as `github-actions[bot]`** The pipeline previously committed generated tests as `openobserve-bot `, which isn't a GitHub account → CLA assistant couldn't match/sign it and blocked the test PR. Now it commits as the canonical **`github-actions[bot]`** identity (a real bot account GitHub resolves via its noreply email), across generate + verify_heal + review_address. > **Admin action:** allowlist `github-actions[bot]` in CLA assistant, then `recheck`. ---
Design notes - Responder is deliberately conservative: a wrong "fix" that breaks a passing test is worse than leaving a nitpick. It dismisses stale/incorrect/style findings. - Both auto-features are flag-gated OFF (same posture as `E2E_AUTO_ON_MERGE`) — auto-acting warrants opt-in. - No heavy re-build in `review_address`; the repo's existing Playwright CI re-validates the pushed fix. - CodeQL untrusted-checkout on the new `review_address` checkout (#416) dismissed — same allowlist false positive as prior PRs.
🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .github/workflows/e2e-council.yml | 131 +++++++++++++++++- .../e2e-ci-engineer.md | 14 ++ .../e2e-ci-responder.md | 78 +++++++++++ opencode.jsonc | 4 + 4 files changed, 222 insertions(+), 5 deletions(-) create mode 100644 .opencode/agents/e2e_ci_council_of_agents/e2e-ci-responder.md diff --git a/.github/workflows/e2e-council.yml b/.github/workflows/e2e-council.yml index b989a9ed85..2db741dbc3 100644 --- a/.github/workflows/e2e-council.yml +++ b/.github/workflows/e2e-council.yml @@ -590,8 +590,8 @@ jobs: SLUG=$(jq -r '.feature_slug // "feature"' docs/test_generator/ci/run-context.json | tr -cd 'a-zA-Z0-9_-') [ -n "$SLUG" ] || SLUG="feature" # LLM-written; strip anything but [a-zA-Z0-9_-] before it hits the commit msg BRANCH="${WORK_BRANCH}" - git config user.name "openobserve-bot" - git config user.email "bot@openobserve.ai" + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" if [ "$REUSE" = "true" ]; then echo "Reuse mode: extending the existing $BRANCH (already checked out)." else @@ -909,8 +909,8 @@ jobs: WORK_BRANCH: ${{ needs.triage.outputs.work_branch }} run: | BRANCH="${WORK_BRANCH}" - git config user.name "openobserve-bot" - git config user.email "bot@openobserve.ai" + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" # The anti-hijack restore staged .opencode/opencode.jsonc — unstage so healer commits carry # ONLY test changes (pipeline files must never reach the test PR). git restore --staged .opencode opencode.jsonc 2>/dev/null || true @@ -1079,8 +1079,15 @@ jobs: [ -n "$SLUG" ] || SLUG="feature" # sanitize LLM-written slug before it hits the PR title SPEC=$(jq -r '.spec_path // "(see branch)"' meta/run-context.json 2>/dev/null || echo "(see branch)") + # Reviewer-facing summary table of the test cases added/changed (Engineer's test-summary.json). + TABLE="" + if [ -f meta/test-summary.json ] && [ "$(jq 'length' meta/test-summary.json 2>/dev/null || echo 0)" -gt 0 ]; then + TABLE=$(jq -r '"\n\n**Test cases added / changed:**\n\n| Test case | Type | What it verifies |\n|---|---|---|\n" + ([.[] | "| `\(.title)` | \(.action) | \(.verifies) |"] | join("\n"))' meta/test-summary.json 2>/dev/null || echo "") + fi + printf '🤖 Auto-generated E2E tests for #%s — **healed to passing** against a live OpenObserve before this PR was opened.\n\n- Branch `%s` (stacked on `%s`), so the tests run against the feature code and pass before it merges.\n- Spec: `%s`\n\nThe repo Playwright CI re-validates here. Review is for **quality + coverage** — the tests already pass.\n\n_Generated by the E2E Council pipeline._\n' \ "$SRC_PR" "$BRANCH" "$BASE_REF" "$SPEC" > /tmp/pr-body.md + printf '%s\n' "$TABLE" >> /tmp/pr-body.md # Idempotent: re-runs force-update the branch, so an existing PR auto-reflects it. EXISTING=$(gh pr list --repo "$GITHUB_REPOSITORY" --head "$BRANCH" --state open --json number --jq '.[0].number // empty') @@ -1096,8 +1103,9 @@ jobs: [ -n "$TEST_PR" ] || { echo "::warning::could not resolve test PR number"; exit 0; } MARKER="" - printf '%s\n🤖 **E2E Council** → healed, passing E2E tests are in #%s (branch `%s`).' \ + printf '%s\n🤖 **E2E Council** → healed, passing E2E tests are in #%s (branch `%s`).\n' \ "$MARKER" "$TEST_PR" "$BRANCH" > /tmp/sticky.md + printf '%s\n' "$TABLE" >> /tmp/sticky.md CID=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${SRC_PR}/comments" \ --jq "[.[] | select(.body | startswith(\"${MARKER}\"))][0].id // empty") if [ -n "$CID" ]; then @@ -1118,3 +1126,116 @@ jobs: if gh pr edit "$TEST_PR" --repo "$GITHUB_REPOSITORY" --add-reviewer "$U" >/dev/null 2>&1; then echo "requested review from @$U"; else echo "::notice::could not request review from @$U (self/perm)"; fi done + + # --------------------------------------------------------------------------- + # JOB 5 — Auto-address the AI review on the test PR (OPTIONAL, OFF by default). + # After the test PR is opened, wait for the repo's `ai-code-review`, then a CONSERVATIVE Responder + # validates the findings, fixes ONLY clearly-valid ones in the test files, pushes (tests-only), and + # posts a rationale + minimizes the bot comment. ONE bounded pass. Gated by `vars.E2E_AUTO_REVIEW`. + # The repo's Playwright CI re-validates the pushed fix (the spec is registered on the PR). Uses the + # scoped GITHUB_TOKEN only (no ORG_ADMIN_TOKEN — this job runs an agent over test/feature code). + # --------------------------------------------------------------------------- + review_address: + needs: [triage, generate, verify_heal, pr_back] + if: > + vars.E2E_AUTO_REVIEW == 'true' && + needs.triage.outputs.dry_run == 'false' && + needs.triage.outputs.skip == 'false' && + needs.triage.outputs.needs_e2e == 'true' && + needs.triage.outputs.author_allowed == 'true' && + needs.generate.outputs.has_changes == 'true' && + needs.verify_heal.outputs.heal_status == 'passing' + runs-on: ubuntu-latest + timeout-minutes: 20 + permissions: + contents: write # push tests-only review fixes to autoe2e/pr- + pull-requests: write + issues: write + steps: + - name: Checkout the autoe2e branch (test files to fix) + uses: actions/checkout@v4 + with: + ref: ${{ needs.triage.outputs.work_branch }} + fetch-depth: 0 + persist-credentials: false + + - name: Restore agent files from a trusted ref (anti-hijack) + env: + EVENT_NAME: ${{ github.event_name }} + REF_NAME: ${{ github.ref_name }} + run: | + RESTORE_REF=main + if [ "$EVENT_NAME" = "workflow_dispatch" ]; then RESTORE_REF="$REF_NAME"; fi + git fetch origin "$RESTORE_REF" --depth=1 + git checkout "origin/$RESTORE_REF" -- .opencode opencode.jsonc + + - uses: actions/setup-node@v4 + with: + node-version: 22 + - name: Install OpenCode + run: | + npm install -g opencode-ai@1.17.7 || { echo "::error::OpenCode install failed"; exit 1; } + + - name: Wait for the AI review + capture it + id: review + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + WORK_BRANCH: ${{ needs.triage.outputs.work_branch }} + run: | + TEST_PR=$(gh pr list --repo "$GITHUB_REPOSITORY" --head "$WORK_BRANCH" --state open --json number --jq '.[0].number // empty') + [ -n "$TEST_PR" ] || { echo "::notice::no open test PR for $WORK_BRANCH — nothing to address."; echo "found=false" >> "$GITHUB_OUTPUT"; exit 0; } + echo "test_pr=$TEST_PR" >> "$GITHUB_OUTPUT" + MARKER="" + mkdir -p docs/test_generator/ci + # Poll up to ~6 min for the review bot's comment (it lands a couple minutes after the PR opens). + for i in $(seq 1 12); do + BODY=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${TEST_PR}/comments" --jq "[.[] | select(.body | startswith(\"${MARKER}\"))][-1].body // empty") + if [ -n "$BODY" ]; then + printf '%s' "$BODY" > docs/test_generator/ci/ai-review.md + echo "found=true" >> "$GITHUB_OUTPUT" + echo "::notice::Captured AI review on test PR #$TEST_PR (after ${i} polls)." + exit 0 + fi + sleep 30 + done + echo "::notice::No AI review appeared within the poll window — skipping addressal." + echo "found=false" >> "$GITHUB_OUTPUT" + + - name: Responder — validate + fix clearly-valid findings + if: steps.review.outputs.found == 'true' + env: + DEEPSEEK_API_KEY_E2E: ${{ secrets.DEEPSEEK_API_KEY_E2E }} + run: | + opencode run --agent e2e-ci-responder --model "$PIPELINE_MODEL" \ + "Read docs/test_generator/ci/ai-review.md (the repo's AI review of this test PR). Validate each finding conservatively; fix ONLY clearly-valid ones in tests/ui-testing (NEVER touch .github, .opencode, opencode.jsonc, playwright.yml, or source). Write docs/test_generator/ci/review-response.md with what you fixed / dismissed / noted." + + - name: Push fixes (tests only) + post response + minimize the bot review + if: steps.review.outputs.found == 'true' + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + WORK_BRANCH: ${{ needs.triage.outputs.work_branch }} + TEST_PR: ${{ steps.review.outputs.test_pr }} + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + # Restore staged pipeline files; commit ONLY test changes (never .opencode/workflow). + git restore --staged .opencode opencode.jsonc 2>/dev/null || true + git add tests/ui-testing + if ! git diff --cached --quiet; then + git commit -m "test(autoe2e): address AI review on #${TEST_PR} (auto)" + git push "https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" "HEAD:refs/heads/${WORK_BRANCH}" + echo "Pushed review fixes to ${WORK_BRANCH} (repo Playwright CI re-validates the PR)." + else + echo "No code changes from the Responder (all findings dismissed/noted)." + fi + # Post the rationale reply on the test PR. + if [ -f docs/test_generator/ci/review-response.md ]; then + gh api -X POST "repos/${GITHUB_REPOSITORY}/issues/${TEST_PR}/comments" -F body=@docs/test_generator/ci/review-response.md >/dev/null \ + && echo "Posted review response on #${TEST_PR}." + fi + # Minimize the AI review comment (same github-actions[bot] author → permitted). Best-effort. + NODE=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${TEST_PR}/comments" --jq "[.[] | select(.body | startswith(\"\"))][-1].node_id // empty") + if [ -n "$NODE" ]; then + gh api graphql -f query='mutation($id:ID!){minimizeComment(input:{subjectId:$id,classifier:RESOLVED}){minimizedComment{isMinimized}}}' -f id="$NODE" >/dev/null 2>&1 \ + && echo "Minimized the AI review comment." || echo "::notice::could not minimize the AI review (best-effort)." + fi diff --git a/.opencode/agents/e2e_ci_council_of_agents/e2e-ci-engineer.md b/.opencode/agents/e2e_ci_council_of_agents/e2e-ci-engineer.md index 9f2cac9bfd..8631188343 100644 --- a/.opencode/agents/e2e_ci_council_of_agents/e2e-ci-engineer.md +++ b/.opencode/agents/e2e_ci_council_of_agents/e2e-ci-engineer.md @@ -209,6 +209,20 @@ Treat this as a gate on *yourself*: a spec that trips any CRITICAL above is not (`mkdir -p` first) listing: spec path, page objects touched (new methods/locators), the `playwright.yml` group + filename to register, and any open risks (e.g. `NEEDS SELECTOR` items the Analyst flagged and how you handled them). +5. A machine-readable **test summary** → `docs/test_generator/ci/test-summary.json`: one entry per + test case you **added or changed this run** (skip unchanged sibling tests), so the PR-back job + can render a reviewer-facing table. Shape: + ```json + [ + { "title": "should open the demo page", "action": "new", "verifies": "demo page loads and the header is visible" }, + { "title": "should filter rows", "action": "append", "verifies": "the filter control updates the results table" } + ] + ``` + `action` = this case's coverage action (`new` for a new spec's tests, `append`/`extend` for ones + added/modified on an existing spec). `verifies` = one concise human sentence. For `action: none` + write `[]`. Keep `title` byte-identical to the `test("…")` name in the spec. + > It must be valid JSON. `pr_back` renders it as a table but **skips it gracefully** (no table) if + > the file is missing, empty `[]`, or unparseable — so never block generation on it. > **Where outputs go:** you run in an ephemeral CI runner with no commit access. Your files are > uploaded as a build artifact and the **PR-back job (Job 4) commits them** to a `test/` diff --git a/.opencode/agents/e2e_ci_council_of_agents/e2e-ci-responder.md b/.opencode/agents/e2e_ci_council_of_agents/e2e-ci-responder.md new file mode 100644 index 0000000000..5180413107 --- /dev/null +++ b/.opencode/agents/e2e_ci_council_of_agents/e2e-ci-responder.md @@ -0,0 +1,78 @@ +--- +description: "CI Responder (Phase 6, optional). Reads the repo's AI code-review on the generated test PR, VALIDATES each finding, fixes only the clearly-valid ones in the test files, and writes a rationale. Conservative — the review bot is noisy. Non-interactive." +mode: primary +--- + +# The Responder — AI-Review Addresser (CI, Phase 6) + +You are **The Responder** for OpenObserve's automated E2E pipeline. After the test PR is opened, the +repo's AI code-review bot posts findings. Your job: **read those findings, decide which are real, fix +the real ones in the test files, and explain the rest** — then hand off a rationale the workflow +posts back. You run non-interactively and hand off via files. + +> **CORE PRINCIPLE — be conservative. The review bot is noisy and frequently wrong** (stale +> re-emissions, misunderstandings of the framework/CI, style nitpicks). Your default is to **NOT +> change code**. Only change a test when a finding is *clearly, concretely correct* and the fix is +> obviously safe. A wrong "fix" that breaks a passing test is far worse than leaving a nitpick. + +## Input (read first) + +```bash +cat docs/test_generator/ci/ai-review.md # the review bot's comment body (the findings) +cat docs/test_generator/ci/run-context.json # feature context (spec_path, area, slug) +cat docs/test_generator/ci/coverage-decision.json 2>/dev/null # what was generated (action, target_spec) +``` +If `ai-review.md` is missing or empty, write an empty response (below) and stop. + +The spec + page objects are checked out under `tests/ui-testing/`. Read the spec being reviewed and +its page objects before judging any finding. + +## Validate each finding (this is the real work) + +For every finding, classify it: + +- **VALID → fix** — a concrete, correct problem in the generated test that you can fix safely and + minimally: e.g. a genuinely wrong/brittle selector, a missing/weak assertion, an un-awaited async + call, a real logic error, a hardcoded credential. Fix it in the **page object / spec** (keep + selectors out of the spec; preserve `mode: 'parallel'`; keep the change minimal). +- **INVALID → dismiss** — wrong, stale, or not-applicable. Common cases to **reject**: claims that + contradict the framework (`global-setup` handles auth; `pm.*` page objects; env-injected creds), + misreadings of shell/`jq`/CI behavior, "issues" that don't exist in the current code, or findings + about files you didn't write (`playwright.yml`, `.opencode/**`, the workflow). +- **NITPICK → note, don't change** — style/preference/perf micro-suggestions that don't affect + correctness. Acknowledge briefly; do not touch code for these. + +When unsure → treat as INVALID/NITPICK (do not change code). Bias hard toward not editing. + +## HARD limits (safety) + +- **Edit ONLY files under `tests/ui-testing/`.** NEVER edit `.github/**`, `.opencode/**`, + `opencode.jsonc`, `playwright.yml`, or any product/source file. +- **Do NOT run, register, or re-generate** — you only adjust existing test files. (The repo's + Playwright CI re-validates the PR after your push.) +- **Do NOT weaken or delete tests** to satisfy a finding. Never replace a real assertion with a + trivial one. If a finding would require weakening coverage, dismiss it. +- Keep every fix **minimal and surgical** — a reviewer should see only the change the finding warranted. + +## OUTPUT (always write this file) + +Write `docs/test_generator/ci/review-response.md` — the comment the workflow posts on the test PR: + +```markdown +🤖 **E2E Council — review addressed** + +**Fixed:** +- → (`file`) + +**Dismissed (not applicable / incorrect):** +- → (e.g. "auth is handled by global-setup", "stale — not in current code") + +**Noted (style/nitpick, no change):** +- → +``` + +Omit any section that's empty. If you changed **nothing**, say so plainly (e.g. "Reviewed the N +findings; none required a code change — rationale below."). Keep it concise and factual. + +Print a one-line summary (counts: fixed / dismissed / noted) and finish. Non-interactive — do not +wait for approval, do not commit or push (the workflow commits your test-file edits). diff --git a/opencode.jsonc b/opencode.jsonc index b72b060ee7..1f7e6049d0 100644 --- a/opencode.jsonc +++ b/opencode.jsonc @@ -52,6 +52,10 @@ "e2e-ci-healer": { "mode": "primary", "prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-healer.md}" + }, + "e2e-ci-responder": { + "mode": "primary", + "prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-responder.md}" } } }