feat(gatekeeper): add optional advisory flags (soft layer, default-off, deterministic)
Add a complementary soft layer to the hard gates: advisory_flags surface governance hints (e.g. oversized-but-not-blocked PRs -> suggest splitting) in the scorecard WITHOUT affecting score or verdict. Hard gates stay the real teeth; advisory leaves the call to humans. - additive & backward-compatible: default off; policies lacking the section behave exactly as before (verified: default policy on real PR #275 yields zero advisory section, PASS 87/100 unchanged) - deterministic: consumes only already-collected PR data (no wall-clock input), preserving same-input -> same-score -> same-verdict - first flag large_pr_files (changed-files >= threshold) answers the 166-open-PR backlog reality where big PRs should be flagged, not blocked - evaluate_advisory_flags() is separate from score_dimensions/decide_verdict; the 5-dim scoring and verdict logic are untouched - 7 new unit tests (15 total) incl. an invariant that enabling advisory does NOT change total/verdict; verified live on real open PR #275 - REFERENCE.md 1.9 + gatekeeper.yaml + verification.md E + demo scorecard Extends the merged gatekeeper line (#90 skill -> #219 workflow -> #220 issueops). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
何开元
parent
71ca2bb683
commit
00f2eab716
|
|
@ -55,13 +55,15 @@ python3 scripts/gatekeeper_sweep.py \
|
|||
| 注入真实审查发现 | 同一 PR + `findings.example.json` | ❌ REQUEST_CHANGES 55/100（裁决翻转，确定性可复算） |
|
||||
| `--apply` 真实回写 | 自有 fork 的演练 PR | 评分卡评论 + tracking issue + 裁决标签全部由 API 回执确认 |
|
||||
| **全仓批量体检** | 本仓库**全部 113 个 open PR** | 113/113 成功：PASS 105 / COMMENT 6 / REQUEST_CHANGES 2，均分 88.5；96% 未关联 issue |
|
||||
| 单测 | `tests/test_scoring.py` | 全绿（锁定四个权威裁决案例的分值与裁决） |
|
||||
| **咨询标记（advisory）** | 真实 open PR #275（10 文件） | 触发「建议拆分」提示，但**裁决不变** PASS 87/100（软建议不改判） |
|
||||
| 单测 | `tests/test_scoring.py` | 15 全绿（8 评分案例 + 7 advisory，锁定确定性） |
|
||||
|
||||
## 设计要点
|
||||
|
||||
- **确定性评分**：AI 只负责产出「发现列表」（可选注入），扣分与裁决由纯函数完成——同策略 + 同 PR → 同裁决，可逐位手算复现、可审计。
|
||||
- **安全默认**：默认 dry-run 什么都不写；即便策略开了 `auto_merge`，也必须 `verdict == PASS` 且显式 `--apply` 才会合并；强语义的 approve/reject 始终留给人，自动裁决只以建议性 `common` 评论 + 标签呈现。
|
||||
- **原生适配 GitLink**：PR 标题/描述取自 `pr +view` 的 `issue.subject/description`；标签挂载走「`label +list` 查 id → Raw API `POST /:owner/:repo/issues/<issue_id>`」；尊重 `common/approved/rejected` 三态 review。
|
||||
- **硬门禁 + 软建议两层**：硬门禁命中即拦截（真牙齿）；可选的 `advisory_flags`（默认关闭、向后兼容、确定性）只在评分卡里提示、不改裁决——把「超体量 PR 该提示拆分但不该阻断」这类治理建议留给人工，正回应活跃仓库的 PR 积压实况。
|
||||
- **零依赖、零常驻**：纯标准库脚本 + `gitlink-cli`，无需部署 webhook 服务或数据库，CI 一条 step 即可接入（见 `ci-example/`）；确定性意味着**大规模治理零 AI 成本**。
|
||||
|
||||
## 许可证
|
||||
|
|
|
|||
|
|
@ -40,14 +40,23 @@
|
|||
- 完整报告（含全量明细表）：[`../examples/demo-outputs/sweep-report-2026-06-10.md`](../examples/demo-outputs/sweep-report-2026-06-10.md)
|
||||
- 诚实口径：批扫不注入审查发现（review_findings 维未评、按满分计），CI 统一 `--skip-ci`（unknown 半分）——总分代表「除人工/AI 审查外的工程卫生分」，偏乐观
|
||||
|
||||
## E. 单元测试（确定性回归护栏）
|
||||
## E. 咨询标记（advisory）→ 真实 PR 触发但不改裁决
|
||||
|
||||
开启 `advisory_flags`（`enabled: true, large_pr_files: 10`）对本仓库真实 open PR `#275`（feat(shortcut): add repo upload，10 个变更文件）跑 dry-run：
|
||||
|
||||
- `large_pr_files` 标记**触发**（10 ≥ 阈值 10），评分卡新增 `### 💡 Advisory` 节提示「建议拆分」
|
||||
- **裁决不变**：仍为 **PASS 87/100**——咨询标记不参与评分、不改变裁决（与 `test_scoring.py` 的 `test_advisory_does_not_change_verdict` 不变量一致）
|
||||
- 这是与硬门禁互补的「软」一层：硬门禁命中即拦截（真牙齿），咨询标记只提示、把判断留给人——正回应 D 节里 166-PR 积压仓库「超体量 PR 该提示拆分但不该阻断」的真实场景
|
||||
- 产物：[`../examples/demo-outputs/scorecard-advisory.md`](../examples/demo-outputs/scorecard-advisory.md)
|
||||
|
||||
## F. 单元测试（确定性回归护栏）
|
||||
|
||||
```bash
|
||||
$ python3 tests/test_scoring.py
|
||||
OK
|
||||
OK # 15 tests（8 评分案例 + 7 advisory）
|
||||
```
|
||||
|
||||
锁定四个权威裁决案例（PASS / REQUEST_CHANGES / COMMENT / 硬门禁直拒）的**总分与裁决**与 Skill 文档逐位一致；任何改动若破坏「同输入 → 同分 → 同裁决」，测试立即变红。
|
||||
锁定四个权威裁决案例（PASS / REQUEST_CHANGES / COMMENT / 硬门禁直拒）的**总分与裁决**与 Skill 文档逐位一致；advisory 测试另锁「默认关闭向后兼容」与「开启触发也不改总分/裁决」两条不变量。任何改动若破坏「同输入 → 同分 → 同裁决」，测试立即变红。
|
||||
|
||||
## 真实运行当场暴露过的问题（透明记录）
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,26 @@
|
|||
## 🛡️ Gatekeeper Report — PR #275 feat(shortcut): add repo upload
|
||||
|
||||
**Verdict: ✅ PASS** · Score: 87/100 · policy: adv-demo-policy.yaml@v1
|
||||
|
||||
| Dimension | Weight | Score | Notes |
|
||||
|-----------|:------:|:-----:|-------|
|
||||
| Review findings | 40 | 40/40 | 0 blocker / 0 major / 0 minor / 0 nit |
|
||||
| Test coverage | 20 | 17/20 | 3 src / 2 test files |
|
||||
| PR hygiene | 15 | 10/15 | desc ✓ / linked issue ✗ / size ✓ |
|
||||
| Commit quality | 15 | 15/15 | 0/0 conventional |
|
||||
| CI status | 10 | 5/10 | unknown |
|
||||
|
||||
### 👥 Suggested reviewers (4)
|
||||
- @doc-maintainer — 3 file(s): README.md, README.zh-CN.md, doc/changes/repo-upload-shortcut.md
|
||||
- @go-reviewer — 5 file(s): internal/client/client.go, internal/client/client_test.go, shortcuts/common/types.go …
|
||||
- @qa-reviewer — 2 file(s): internal/client/client_test.go, shortcuts/repo/repo_test.go
|
||||
- @maintainer — 2 file(s): internal/i18n/locales/en-US.json, internal/i18n/locales/zh-CN.json
|
||||
|
||||
### 💡 Advisory (1)
|
||||
> 建议性提示，不参与评分、不改变裁决。
|
||||
- `large_pr_files`: 变更 10 个文件，达到建议拆分阈值 10（未触发硬门禁，建议拆分为更小的 PR 以便审查）
|
||||
|
||||
### Next steps
|
||||
1. 满足合并门禁；如策略开启 auto_merge 且操作者带 --apply，可执行合并
|
||||
---
|
||||
*Generated by gitlink-gatekeeper · policy-as-code PR gate · re-run after changes*
|
||||
|
|
@ -84,6 +84,12 @@ DEFAULT_POLICY: dict[str, Any] = {
|
|||
"auto_merge": False,
|
||||
"merge_method": "squash",
|
||||
},
|
||||
# 咨询标记（advisory）：默认全关；开启后仅在评分卡中给出建议性提示，
|
||||
# 不参与评分、不改变裁决。向后兼容——旧策略无此段时等同关闭。
|
||||
"advisory_flags": {
|
||||
"enabled": False,
|
||||
"large_pr_files": 0, # 变更文件数 ≥ 此值则提示拆分（0=不启用；建议 < hard_gates.max_changed_files）
|
||||
},
|
||||
}
|
||||
|
||||
VERDICT_EMOJI = {"PASS": "✅", "REQUEST_CHANGES": "❌", "COMMENT": "💬"}
|
||||
|
|
@ -641,6 +647,35 @@ def evaluate_hard_gates(inp: ScoreInput, policy: dict[str, Any]) -> list[dict[st
|
|||
return failures
|
||||
|
||||
|
||||
def evaluate_advisory_flags(inp: ScoreInput, policy: dict[str, Any]) -> list[dict[str, str]]:
|
||||
"""咨询标记（advisory）——建议性、不参与评分、不改变裁决。
|
||||
|
||||
与硬门禁互补的「软」一层：硬门禁是真牙齿（命中即 REQUEST_CHANGES），
|
||||
咨询标记只在评分卡里给出提示，把治理建议（如积压仓库里的超体量 PR
|
||||
应拆分但不应阻断）留给人工判断。
|
||||
|
||||
设计约束：
|
||||
- 默认全关、向后兼容（策略无 advisory_flags 段时返回空，行为与旧版完全一致）；
|
||||
- 确定性——仅消费已采集的 PR 数据，不依赖当前时间等非确定性输入，
|
||||
保持「同输入 → 同输出」，与评分主逻辑一致可复算。
|
||||
"""
|
||||
cfg = policy.get("advisory_flags") or {}
|
||||
if not cfg.get("enabled"):
|
||||
return []
|
||||
flags: list[dict[str, str]] = []
|
||||
threshold = int(cfg.get("large_pr_files", 0) or 0)
|
||||
n_files = len(inp.changed_files)
|
||||
if threshold > 0 and n_files >= threshold:
|
||||
flags.append(
|
||||
{
|
||||
"flag": "large_pr_files",
|
||||
"detail": f"变更 {n_files} 个文件，达到建议拆分阈值 {threshold}"
|
||||
f"（未触发硬门禁，建议拆分为更小的 PR 以便审查）",
|
||||
}
|
||||
)
|
||||
return flags
|
||||
|
||||
|
||||
def decide_verdict(
|
||||
total: int, hard_gate_failed: bool, policy: dict[str, Any]
|
||||
) -> str:
|
||||
|
|
@ -667,6 +702,7 @@ def render_scorecard(
|
|||
policy_label: str,
|
||||
routing: dict[str, Any] | None,
|
||||
tracking_issue: Any = None,
|
||||
advisory: list[dict[str, str]] | None = None,
|
||||
) -> str:
|
||||
emoji = VERDICT_EMOJI[verdict]
|
||||
total = dims["total"]
|
||||
|
|
@ -719,6 +755,13 @@ def render_scorecard(
|
|||
lines.append(f"- `{f['gate']}`: {f['detail']}")
|
||||
lines.append("")
|
||||
|
||||
if advisory:
|
||||
lines.append(f"### 💡 Advisory ({len(advisory)})")
|
||||
lines.append("> 建议性提示，不参与评分、不改变裁决。")
|
||||
for a in advisory:
|
||||
lines.append(f"- `{a['flag']}`: {a['detail']}")
|
||||
lines.append("")
|
||||
|
||||
must = [f for f in inp.findings if f.severity in ("blocker", "major")]
|
||||
should = [f for f in inp.findings if f.severity == "minor"]
|
||||
nits = [f for f in inp.findings if f.severity == "nit"]
|
||||
|
|
@ -1184,9 +1227,13 @@ def main(argv: list[str] | None = None) -> int:
|
|||
)
|
||||
dims = score_dimensions(inp, policy)
|
||||
failures = evaluate_hard_gates(inp, policy)
|
||||
advisory = evaluate_advisory_flags(inp, policy)
|
||||
verdict = decide_verdict(dims["total"], bool(failures), policy)
|
||||
scorecard = render_scorecard(inp, dims, failures, verdict, policy_label, routing)
|
||||
print(f" Score: {dims['total']}/100 · 硬门禁失败 {len(failures)} 项 · 裁决: {verdict}")
|
||||
scorecard = render_scorecard(
|
||||
inp, dims, failures, verdict, policy_label, routing, advisory=advisory
|
||||
)
|
||||
adv_note = f" · 咨询标记 {len(advisory)} 条" if advisory else ""
|
||||
print(f" Score: {dims['total']}/100 · 硬门禁失败 {len(failures)} 项 · 裁决: {verdict}{adv_note}")
|
||||
print()
|
||||
|
||||
# 落盘本地产物（无论 dry-run 与否都生成，便于复核 / 验证记录）
|
||||
|
|
@ -1229,7 +1276,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
# 把编号补进评分卡 Next steps 后重新落盘（评论将带上回链）
|
||||
scorecard = render_scorecard(
|
||||
inp, dims, failures, verdict, policy_label, routing,
|
||||
tracking_issue=tracking_issue_no,
|
||||
tracking_issue=tracking_issue_no, advisory=advisory,
|
||||
)
|
||||
scorecard_path.write_text(scorecard + "\n", encoding="utf-8")
|
||||
|
||||
|
|
@ -1289,6 +1336,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
"scores": {k: v for k, v in dims.items() if k != "total"},
|
||||
"total": dims["total"],
|
||||
"hard_gate_failures": failures,
|
||||
"advisory_flags": advisory,
|
||||
"verdict": verdict,
|
||||
"verdict_label": verdict_label,
|
||||
"tracking_issue": tracking_issue_no,
|
||||
|
|
|
|||
|
|
@ -244,5 +244,72 @@ class TestVerdictBoundaries(unittest.TestCase):
|
|||
self.assertEqual(decide_verdict(100, True, DEFAULT_POLICY), "REQUEST_CHANGES")
|
||||
|
||||
|
||||
evaluate_advisory_flags = gw.evaluate_advisory_flags
|
||||
|
||||
|
||||
def _copy_policy(**advisory):
|
||||
"""深拷一份默认策略并覆盖 advisory_flags，避免污染共享 dict。"""
|
||||
p = _json.loads(_json.dumps(DEFAULT_POLICY))
|
||||
p["advisory_flags"] = advisory
|
||||
return p
|
||||
|
||||
|
||||
class TestAdvisoryFlags(unittest.TestCase):
|
||||
"""咨询标记（advisory）：加性、默认关闭、不参与评分/裁决、确定性。"""
|
||||
|
||||
def _input(self, n_files: int) -> ScoreInput:
|
||||
# 一个干净的 PASS 输入（无 findings、有测试、CI 过、描述充分、关联 issue）
|
||||
return _build(
|
||||
pr_id="1", title="feat: x", desc_len=60, linked_issue=True,
|
||||
n_files=n_files, src=1, tests=1, commits=(1, 1), ci="passing",
|
||||
findings_counts={},
|
||||
)
|
||||
|
||||
def test_default_off_is_backward_compatible(self):
|
||||
# 默认策略未显式开启 → 返回空，行为与旧版完全一致
|
||||
self.assertEqual(evaluate_advisory_flags(self._input(200), DEFAULT_POLICY), [])
|
||||
|
||||
def test_missing_section_returns_empty(self):
|
||||
# 策略完全没有 advisory_flags 段（旧策略）→ 不报错、返回空
|
||||
p = _json.loads(_json.dumps(DEFAULT_POLICY))
|
||||
p.pop("advisory_flags", None)
|
||||
self.assertEqual(evaluate_advisory_flags(self._input(200), p), [])
|
||||
|
||||
def test_enabled_below_threshold_no_flag(self):
|
||||
p = _copy_policy(enabled=True, large_pr_files=30)
|
||||
self.assertEqual(evaluate_advisory_flags(self._input(29), p), [])
|
||||
|
||||
def test_enabled_at_threshold_flags(self):
|
||||
p = _copy_policy(enabled=True, large_pr_files=30)
|
||||
flags = evaluate_advisory_flags(self._input(30), p)
|
||||
self.assertEqual(len(flags), 1)
|
||||
self.assertEqual(flags[0]["flag"], "large_pr_files")
|
||||
|
||||
def test_zero_threshold_disabled_even_if_enabled(self):
|
||||
# large_pr_files=0 表示该标记不启用，即便 enabled=True
|
||||
p = _copy_policy(enabled=True, large_pr_files=0)
|
||||
self.assertEqual(evaluate_advisory_flags(self._input(500), p), [])
|
||||
|
||||
def test_advisory_does_not_change_verdict(self):
|
||||
# 关键不变量：开 advisory 且触发标记，total 与 verdict 必须与不开时逐位一致
|
||||
inp = self._input(50)
|
||||
base_dims, base_fail, base_verdict = _run(inp)
|
||||
p = _copy_policy(enabled=True, large_pr_files=30)
|
||||
adv = evaluate_advisory_flags(inp, p)
|
||||
self.assertEqual(len(adv), 1) # 确实触发了
|
||||
dims = score_dimensions(inp, p)
|
||||
verdict = decide_verdict(dims["total"], bool(evaluate_hard_gates(inp, p)), p)
|
||||
self.assertEqual(dims["total"], base_dims["total"]) # 分数不变
|
||||
self.assertEqual(verdict, base_verdict) # 裁决不变
|
||||
|
||||
def test_deterministic(self):
|
||||
# 同输入两次调用结果完全相同
|
||||
p = _copy_policy(enabled=True, large_pr_files=10)
|
||||
inp = self._input(20)
|
||||
self.assertEqual(
|
||||
evaluate_advisory_flags(inp, p), evaluate_advisory_flags(inp, p)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
|
|
|||
|
|
@ -95,6 +95,17 @@
|
|||
| `auto_merge` | bool | `false` | **仅当 `true` 且裁决=PASS 且显式 `--apply` 时才合并** |
|
||||
| `merge_method` | enum | `squash` | `merge` \| `rebase` \| `squash` |
|
||||
|
||||
### 1.9 `advisory_flags`（可选 · 咨询标记 · 默认全关）
|
||||
|
||||
与硬门禁互补的「软」一层。硬门禁是真牙齿（命中即 `REQUEST_CHANGES`）；咨询标记**只在评分卡中给出建议性提示，不参与评分、不改变裁决**，把治理建议留给人工判断。默认关闭、**向后兼容**（策略无此段时等同关闭，旧行为不变）；**确定性**——仅消费已采集的 PR 数据，不依赖当前时间等非确定性输入，与评分主逻辑同样可复算。
|
||||
|
||||
| 字段 | 类型 | 默认值 | 含义 |
|
||||
|------|------|--------|------|
|
||||
| `enabled` | bool | `false` | 总开关，默认关闭 |
|
||||
| `large_pr_files` | int | `0` | 变更文件数 ≥ 此值则提示拆分（`0`=不启用；建议 `< hard_gates.max_changed_files`，使其落在「偏大但未触硬门禁」区间） |
|
||||
|
||||
> 典型用途：在 PR 积压的活跃仓库里，对「超体量但未触硬门禁」的 PR 提示拆分而**不阻断**——既给出治理信号，又不把判断权从人手里夺走。命中的标记渲染在评分卡的 `### 💡 Advisory` 小节，并写入 `summary.json` 的 `advisory_flags` 字段。
|
||||
|
||||
---
|
||||
|
||||
## 2. 评分算法规格（确定性，可复现）
|
||||
|
|
|
|||
|
|
@ -48,3 +48,12 @@ behavior:
|
|||
apply_label: true # 按裁决打标签
|
||||
auto_merge: false # 仅当 true 且裁决=PASS 且显式 --apply 时才合并
|
||||
merge_method: squash # merge | rebase | squash
|
||||
|
||||
# —— 咨询标记（advisory，可选）——
|
||||
# 与硬门禁互补的「软」一层：硬门禁命中即 REQUEST_CHANGES（真牙齿）；
|
||||
# 咨询标记只在评分卡里提示、不参与评分、不改变裁决（建议留给人工判断）。
|
||||
# 默认关闭、向后兼容；确定性——仅用已采集数据，不依赖当前时间等非确定性输入。
|
||||
# 典型用途：PR 积压的活跃仓库里，对「超体量但未触硬门禁」的 PR 提示拆分而不阻断。
|
||||
advisory_flags:
|
||||
enabled: false # 总开关，默认关闭
|
||||
large_pr_files: 0 # 变更文件数 ≥ 此值则提示拆分（0=不启用；建议 < hard_gates.max_changed_files）
|
||||
|
|
|
|||
Loading…
Reference in New Issue