diff --git a/examples/research/repro-audit/README.md b/examples/research/repro-audit/README.md new file mode 100644 index 0000000..48e3c16 --- /dev/null +++ b/examples/research/repro-audit/README.md @@ -0,0 +1,56 @@ +# 科研项目复现性审计(repro-audit) + +把 [`gitlink-repro-audit` Skill](../../../skills/gitlink-repro-audit/SKILL.md) 包成**可直接运行的科研辅助工具**(子赛题四:应用 GitLink 辅助科研): + +> **采集 → 评估 → 报告 → 回写**:用 `repo +tree` / `file +view --raw` / `release +list` 采集仓库结构、README 与发布信息,做八维**确定性复现性检查**(文档 / 许可证 / 引用 / 依赖固化 / 运行入口 / 数据说明 / 测试 / 版本固化),输出 0-100 评分卡 + A/B/C/D 等级 + 逐项修复建议,并(仅在 `--apply` 时)用 `issue +create` 把报告作为改进 tracking issue 回写。 + +## 科研赋能价值 + +- **课题组自查**:论文投稿/开源发布前对代码仓库做复现性体检,逐项补齐 +- **学术规范检查**:许可证、引用信息(CITATION.cff/BibTeX)、数据可得性一次性核验 +- **可进 CI**:退出码 `2` 表示存在明显复现缺口,可作为科研仓库的发布门禁 + +## 交付物 + +- `scripts/repro_audit.py`:完整审计闭环(纯标准库,Python ≥3.9,零第三方依赖) +- `tests/test_audit.py`:确定性回归护栏(7/7 全绿,同输入同分同等级) +- `docs/verification.md`:真实科研仓库验证证据 +- `examples/demo-outputs/`:两个真实仓库的审计报告 + +## 快速运行(默认 dry-run,不写远端) + +```bash +npm install -g @gitlink-ai/cli +gitlink-cli auth login + +python3 scripts/repro_audit.py --owner --repo --output-dir outputs +# 回写改进 tracking issue(请先确认报告内容):加 --apply +``` + +## 已在真实科研仓库验证 + +见 [`docs/verification.md`](docs/verification.md): + +| 仓库 | 类型 | 得分 | 等级 | +|------|------|------|------| +| GengzhaoWang/UAV-Paper | 论文笔记仓库 | 8/100 | D(复现困难,符合预期) | +| momomym/paper-summarizer | 科研工具代码仓库 | 58/100 | C(缺引用/数据说明/release,建议明确) | + +两个仓库得分区分度显著,逐项证据均可在仓库页面人工复核。 + +## 评分标准 + +| 检查项 | 分值 | 判据 | +|--------|------|------| +| README 与运行说明 | 15 | README 存在且含运行/复现章节(仅 README 得 8) | +| 开源许可证 | 15 | LICENSE / COPYING | +| 引用信息 | 10 | CITATION.cff 或 README BibTeX | +| 依赖清单 | 15 | requirements.txt / environment.yml / go.mod 等 | +| 运行入口 | 10 | Makefile / run.sh / main.py / Dockerfile / scripts | +| 数据可得性说明 | 10 | data 目录或 README 数据说明 | +| 测试/验证代码 | 10 | tests 目录 | +| 版本发布 | 15 | ≥1 个 release | + +等级:≥85 A / ≥70 B / ≥50 C / <50 D。 + +> 依赖 `file` 快捷命令组(PR #330);在其合并前可用 `--cli` 指向包含该命令的本地构建。 diff --git a/examples/research/repro-audit/docs/verification.md b/examples/research/repro-audit/docs/verification.md new file mode 100644 index 0000000..925c435 --- /dev/null +++ b/examples/research/repro-audit/docs/verification.md @@ -0,0 +1,35 @@ +# 真实平台验证记录 + +验证日期:2026-07-05;平台:生产环境 gitlink.org.cn;CLI:包含 `file` 命令组(PR #330)的本地构建。 + +## 1. 论文笔记仓库(预期低分) + +```bash +python3 scripts/repro_audit.py --owner GengzhaoWang --repo UAV-Paper +``` + +- 结果:**8/100 — D(复现困难)**,退出码 `2` +- 证据:仅有 README(无运行说明章节),无 LICENSE / 引用 / 依赖清单 / 入口 / 数据说明 / 测试 / release +- 完整报告:[`../examples/demo-outputs/repro-audit-GengzhaoWang-UAV-Paper.md`](../examples/demo-outputs/repro-audit-GengzhaoWang-UAV-Paper.md) + +## 2. 科研工具代码仓库(中等分,建议明确) + +```bash +python3 scripts/repro_audit.py --owner momomym --repo paper-summarizer +``` + +- 结果:**58/100 — C(存在明显缺口)**,退出码 `2` +- 检出:LICENSE ✅、requirements.txt ✅、scripts 入口 ✅、tests ✅ +- 缺口:README 无运行章节(部分分 8/15)、无引用信息、无数据说明、无 release +- 完整报告:[`../examples/demo-outputs/repro-audit-momomym-paper-summarizer.md`](../examples/demo-outputs/repro-audit-momomym-paper-summarizer.md) + +两个真实仓库得分区分度显著(8 vs 58),每项证据均可在仓库网页人工复核。 + +## 3. 确定性回归护栏 + +```bash +python3 tests/test_audit.py +# Ran 7 tests ... OK +``` + +覆盖:满分样例(100/A)、空仓库(0/D)、README 无运行说明的部分分、README 内引用识别、scripts 目录作为入口、同输入同输出确定性、失败项必附修复建议。 diff --git a/examples/research/repro-audit/examples/demo-outputs/repro-audit-GengzhaoWang-UAV-Paper.md b/examples/research/repro-audit/examples/demo-outputs/repro-audit-GengzhaoWang-UAV-Paper.md new file mode 100644 index 0000000..682b4e1 --- /dev/null +++ b/examples/research/repro-audit/examples/demo-outputs/repro-audit-GengzhaoWang-UAV-Paper.md @@ -0,0 +1,17 @@ +# 科研复现性审计报告:GengzhaoWang/UAV-Paper(ref: 默认分支) + +**总分:8/100 — 等级 D(复现困难)** + +| 检查项 | 得分 | 证据 | 建议 | +|--------|------|------|------| +| README 与运行说明 | 8/15 | README=README.md;运行/复现章节=未检出 | 补充 README 并加入 How to run / 复现步骤章节 | +| 开源许可证 | 0/15 | 许可证文件=缺失 | 添加 LICENSE 文件(科研代码推荐 MIT/Apache-2.0/BSD) | +| 引用信息 | 0/10 | CITATION 文件=无;README 引用章节=无 | 添加 CITATION.cff 或在 README 中给出 BibTeX | +| 依赖清单(环境固化) | 0/15 | 依赖文件=缺失 | 添加 requirements.txt / environment.yml 等依赖清单并固定版本 | +| 运行入口 | 0/10 | 入口=未检出 | 提供 Makefile / run.sh / main.py 等一键运行入口 | +| 数据可得性说明 | 0/10 | 数据目录=无;README 数据说明=无 | 在 README 中说明数据集来源、获取方式与预处理步骤 | +| 测试/验证代码 | 0/10 | 测试目录=无 | 添加 tests/ 验证关键结果可复算 | +| 版本发布(成果固化) | 0/15 | release 数=0 | 为论文对应的代码状态打 tag 并创建 release | + +--- +*由 repro-audit 生成(确定性检查,同输入同输出)。* \ No newline at end of file diff --git a/examples/research/repro-audit/examples/demo-outputs/repro-audit-momomym-paper-summarizer.md b/examples/research/repro-audit/examples/demo-outputs/repro-audit-momomym-paper-summarizer.md new file mode 100644 index 0000000..6a895ac --- /dev/null +++ b/examples/research/repro-audit/examples/demo-outputs/repro-audit-momomym-paper-summarizer.md @@ -0,0 +1,17 @@ +# 科研复现性审计报告:momomym/paper-summarizer(ref: 默认分支) + +**总分:58/100 — 等级 C(存在明显缺口)** + +| 检查项 | 得分 | 证据 | 建议 | +|--------|------|------|------| +| README 与运行说明 | 8/15 | README=README.md;运行/复现章节=未检出 | 补充 README 并加入 How to run / 复现步骤章节 | +| 开源许可证 | 15/15 | 许可证文件=LICENSE | — | +| 引用信息 | 0/10 | CITATION 文件=无;README 引用章节=无 | 添加 CITATION.cff 或在 README 中给出 BibTeX | +| 依赖清单(环境固化) | 15/15 | 依赖文件=requirements.txt | — | +| 运行入口 | 10/10 | 入口=scripts | — | +| 数据可得性说明 | 0/10 | 数据目录=无;README 数据说明=无 | 在 README 中说明数据集来源、获取方式与预处理步骤 | +| 测试/验证代码 | 10/10 | 测试目录=tests | — | +| 版本发布(成果固化) | 0/15 | release 数=0 | 为论文对应的代码状态打 tag 并创建 release | + +--- +*由 repro-audit 生成(确定性检查,同输入同输出)。* \ No newline at end of file diff --git a/examples/research/repro-audit/scripts/__pycache__/repro_audit.cpython-312.pyc b/examples/research/repro-audit/scripts/__pycache__/repro_audit.cpython-312.pyc new file mode 100644 index 0000000..2380313 Binary files /dev/null and b/examples/research/repro-audit/scripts/__pycache__/repro_audit.cpython-312.pyc differ diff --git a/examples/research/repro-audit/scripts/repro_audit.py b/examples/research/repro-audit/scripts/repro_audit.py new file mode 100644 index 0000000..8f92024 --- /dev/null +++ b/examples/research/repro-audit/scripts/repro_audit.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +"""科研项目复现性审计(repro-audit)。 + +面向科研代码仓库的确定性复现性评估: +1. 采集:`repo +tree` 获取仓库结构,`file +view --raw` 拉取 README,`release +list` 查发布 +2. 评估:八个维度的复现性检查项(文档/许可证/引用/依赖/入口/数据说明/测试/版本固化) +3. 报告:0-100 评分卡 + A/B/C/D 等级 + 逐项修复建议 +4. 回写:--apply 时用 `issue +create` 把报告作为改进 tracking issue 提交 + +纯标准库实现(Python >= 3.9),gitlink-cli 为唯一外部依赖。默认 dry-run。 +""" + +import argparse +import json +import re +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path + + +@dataclass +class CheckResult: + name: str + weight: int + score: int # 0..weight + evidence: str + advice: str + + +def run_cli(args, cli="gitlink-cli"): + result = subprocess.run([cli, *args], capture_output=True, text=True, check=False) + if result.returncode != 0: + raise RuntimeError( + f"gitlink-cli {' '.join(args)} 失败: {result.stderr.strip() or result.stdout.strip()}" + ) + return result.stdout + + +def get_json(args, cli="gitlink-cli"): + payload = json.loads(run_cli([*args, "--format", "json"], cli=cli)) + return payload.get("data", payload) + + +def list_entries(owner, repo, ref, path="", cli="gitlink-cli"): + args = ["repo", "+tree", "--owner", owner, "--repo", repo] + if ref: + args += ["--ref", ref] + if path: + args += ["--path", path] + data = get_json(args, cli=cli) + entries = data.get("entries", data) if isinstance(data, dict) else data + out = [] + if isinstance(entries, list): + for e in entries: + if isinstance(e, dict) and e.get("name"): + out.append((e["name"], e.get("type", ""))) + return out + + +def fetch_readme(owner, repo, ref, names, cli="gitlink-cli"): + for name in names: + if re.match(r"(?i)^readme(\.|$)", name): + args = ["file", "+view", "--owner", owner, "--repo", repo, "--path", name, "--raw"] + if ref: + args += ["--ref", ref] + try: + return name, run_cli(args, cli=cli) + except RuntimeError: + continue + return None, "" + + +def count_releases(owner, repo, cli="gitlink-cli"): + try: + data = get_json(["release", "+list", "--owner", owner, "--repo", repo], cli=cli) + except RuntimeError: + return 0 + releases = data.get("releases", data) if isinstance(data, dict) else data + return len(releases) if isinstance(releases, list) else 0 + + +DEP_FILES = [ + "requirements.txt", "environment.yml", "environment.yaml", "Pipfile", + "pyproject.toml", "setup.py", "go.mod", "package.json", "Cargo.toml", + "pom.xml", "build.gradle", "CMakeLists.txt", "DESCRIPTION", "renv.lock", +] +ENTRY_FILES = ["Makefile", "makefile", "run.sh", "train.sh", "main.py", "run.py", "train.py", "Dockerfile", "docker-compose.yml"] +CITATION_FILES = ["CITATION.cff", "CITATION", "CITATION.bib", "citation.bib"] +LICENSE_RE = re.compile(r"(?i)^(license|licence|copying)(\.|$)") +DATA_HINT_RE = re.compile(r"(?i)\b(dataset|data set|数据集|数据来源|download.*data|data/)\b") +REPRO_HINT_RE = re.compile(r"(?i)(reproduc|复现|实验设置|experiment setup|how to run|usage|快速开始|quick\s?start)") + + +def audit(names, dirs, readme_text, release_count): + """确定性打分:names=顶层文件名,dirs=顶层目录名。返回 CheckResult 列表。""" + nameset = set(names) + results = [] + + def add(name, weight, ok, evidence, advice, partial=None): + score = weight if ok else (partial if partial else 0) + results.append(CheckResult(name, weight, score, evidence, "" if ok else advice)) + + readme_name = next((n for n in names if re.match(r"(?i)^readme(\.|$)", n)), None) + has_usage = bool(readme_text and REPRO_HINT_RE.search(readme_text)) + add("README 与运行说明", 15, bool(readme_name and has_usage), + f"README={readme_name or '缺失'};运行/复现章节={'有' if has_usage else '未检出'}", + "补充 README 并加入 How to run / 复现步骤章节", + partial=8 if readme_name else 0) + + lic = next((n for n in names if LICENSE_RE.match(n)), None) + add("开源许可证", 15, bool(lic), f"许可证文件={lic or '缺失'}", + "添加 LICENSE 文件(科研代码推荐 MIT/Apache-2.0/BSD)") + + cit = next((n for n in names if n in CITATION_FILES), None) + cite_in_readme = bool(readme_text and re.search(r"(?i)(citation|引用|bibtex|@(article|inproceedings))", readme_text)) + add("引用信息", 10, bool(cit or cite_in_readme), + f"CITATION 文件={cit or '无'};README 引用章节={'有' if cite_in_readme else '无'}", + "添加 CITATION.cff 或在 README 中给出 BibTeX") + + dep = next((n for n in names if n in DEP_FILES), None) + add("依赖清单(环境固化)", 15, bool(dep), f"依赖文件={dep or '缺失'}", + "添加 requirements.txt / environment.yml 等依赖清单并固定版本") + + entry = next((n for n in names if n in ENTRY_FILES), None) + script_dir = next((d for d in dirs if d in ("scripts", "bin")), None) + add("运行入口", 10, bool(entry or script_dir), + f"入口={entry or script_dir or '未检出'}", + "提供 Makefile / run.sh / main.py 等一键运行入口") + + data_dir = next((d for d in dirs if re.match(r"(?i)^(data|datasets?)$", d)), None) + data_in_readme = bool(readme_text and DATA_HINT_RE.search(readme_text)) + add("数据可得性说明", 10, bool(data_dir or data_in_readme), + f"数据目录={data_dir or '无'};README 数据说明={'有' if data_in_readme else '无'}", + "在 README 中说明数据集来源、获取方式与预处理步骤") + + test_dir = next((d for d in dirs if re.match(r"(?i)^tests?$", d)), None) + add("测试/验证代码", 10, bool(test_dir), f"测试目录={test_dir or '无'}", + "添加 tests/ 验证关键结果可复算") + + add("版本发布(成果固化)", 15, release_count > 0, f"release 数={release_count}", + "为论文对应的代码状态打 tag 并创建 release") + + return results + + +GRADE = [(85, "A(可复现性良好)"), (70, "B(基本可复现)"), (50, "C(存在明显缺口)"), (0, "D(复现困难)")] + + +def render_report(owner, repo, ref, results): + total = sum(r.score for r in results) + grade = next(g for t, g in GRADE if total >= t) + lines = [ + f"# 科研复现性审计报告:{owner}/{repo}(ref: {ref or '默认分支'})", + "", + f"**总分:{total}/100 — 等级 {grade}**", + "", + "| 检查项 | 得分 | 证据 | 建议 |", + "|--------|------|------|------|", + ] + for r in results: + lines.append(f"| {r.name} | {r.score}/{r.weight} | {r.evidence} | {r.advice or '—'} |") + lines += ["", "---", "*由 repro-audit 生成(确定性检查,同输入同输出)。*"] + return "\n".join(lines), total + + +def main(): + parser = argparse.ArgumentParser(description="科研项目复现性审计") + parser.add_argument("--owner", required=True) + parser.add_argument("--repo", required=True) + parser.add_argument("--ref", default="") + parser.add_argument("--apply", action="store_true", help="把报告作为 tracking issue 回写(默认 dry-run)") + parser.add_argument("--output-dir", default="outputs") + parser.add_argument("--cli", default="gitlink-cli") + args = parser.parse_args() + + entries = list_entries(args.owner, args.repo, args.ref, cli=args.cli) + names = [n for n, t in entries if t != "dir"] + dirs = [n for n, t in entries if t == "dir"] + _, readme_text = fetch_readme(args.owner, args.repo, args.ref, names, cli=args.cli) + releases = count_releases(args.owner, args.repo, cli=args.cli) + + results = audit(names, dirs, readme_text, releases) + report, total = render_report(args.owner, args.repo, args.ref, results) + + out_dir = Path(args.output_dir) + out_dir.mkdir(parents=True, exist_ok=True) + path = out_dir / f"repro-audit-{args.owner}-{args.repo}.md" + path.write_text(report, encoding="utf-8") + print(report) + print(f"\n报告已保存:{path}", file=sys.stderr) + + if args.apply: + run_cli([ + "issue", "+create", "--owner", args.owner, "--repo", args.repo, + "--title", f"[repro-audit] 复现性审计报告({total}/100)", + "--body", report, "--format", "json", + ], cli=args.cli) + print("已创建 tracking issue。", file=sys.stderr) + + return 0 if total >= 70 else 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/research/repro-audit/tests/test_audit.py b/examples/research/repro-audit/tests/test_audit.py new file mode 100644 index 0000000..2143e23 --- /dev/null +++ b/examples/research/repro-audit/tests/test_audit.py @@ -0,0 +1,73 @@ +"""确定性回归护栏:同输入 → 同分 → 同等级。""" + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "scripts")) + +from repro_audit import audit, render_report # noqa: E402 + +GOOD_README = """# Project + +## How to run + +```bash +make train +``` + +## Dataset + +Download the dataset from ... + +## Citation + +```bibtex +@article{x2026} +``` +""" + + +class AuditTest(unittest.TestCase): + def test_full_marks(self): + names = ["README.md", "LICENSE", "CITATION.cff", "requirements.txt", "Makefile"] + dirs = ["data", "tests", "src"] + results = audit(names, dirs, GOOD_README, release_count=2) + report, total = render_report("o", "r", "", results) + self.assertEqual(total, 100) + self.assertIn("等级 A", report) + + def test_empty_repo(self): + results = audit([], [], "", release_count=0) + report, total = render_report("o", "r", "", results) + self.assertEqual(total, 0) + self.assertIn("等级 D", report) + + def test_readme_without_usage_partial(self): + results = audit(["README.md"], [], "# hi", release_count=0) + readme = next(r for r in results if r.name == "README 与运行说明") + self.assertEqual(readme.score, 8) # 有 README 无运行说明 → 部分分 + + def test_citation_in_readme_counts(self): + results = audit(["README.md"], [], "## Citation\n@article{x}", release_count=0) + cit = next(r for r in results if r.name == "引用信息") + self.assertEqual(cit.score, cit.weight) + + def test_scripts_dir_as_entry(self): + results = audit([], ["scripts"], "", release_count=0) + entry = next(r for r in results if r.name == "运行入口") + self.assertEqual(entry.score, entry.weight) + + def test_deterministic(self): + a = audit(["README.md", "go.mod"], ["tests"], GOOD_README, 1) + b = audit(["README.md", "go.mod"], ["tests"], GOOD_README, 1) + self.assertEqual([vars(x) for x in a], [vars(y) for y in b]) + + def test_advice_present_for_failures(self): + results = audit([], [], "", 0) + for r in results: + self.assertTrue(r.advice, f"{r.name} 应给出修复建议") + + +if __name__ == "__main__": + unittest.main() diff --git a/skills/gitlink-repro-audit/SKILL.md b/skills/gitlink-repro-audit/SKILL.md new file mode 100644 index 0000000..3e2f845 --- /dev/null +++ b/skills/gitlink-repro-audit/SKILL.md @@ -0,0 +1,69 @@ +--- +name: gitlink-repro-audit +version: 1.0.0 +description: "科研项目复现性审计:对 GitLink 上的科研代码仓库做八维复现性检查(文档/许可证/引用/依赖/入口/数据说明/测试/版本固化),生成 0-100 评分卡与修复建议,并可回写改进 tracking issue。当用户需要评估科研仓库可复现性、准备论文代码发布、或做学术规范检查时触发。" +metadata: + requires: + bins: ["gitlink-cli"] + cliHelp: "gitlink-cli repo --help" +--- + +# gitlink-repro-audit(科研复现性审计) + +**CRITICAL — 开始前必须先阅读 [`../gitlink-shared/SKILL.md`](../gitlink-shared/SKILL.md),其中包含认证、权限处理和 API 注意事项。** +**CRITICAL — 审计本身只读;`--apply` 回写 tracking issue 前必须先向用户展示报告并确认。** +**CRITICAL — GitLink 操作只能用 `gitlink-cli`。** + +> **前置条件:** 先阅读 [`../gitlink-shared/SKILL.md`](../gitlink-shared/SKILL.md) + +--- + +## 功能概述 + +论文代码「跑不起来」是科研复现危机的核心。本技能对科研代码仓库做系统化复现性评估: + +| 检查项 | 分值 | 判据 | +|--------|------|------| +| README 与运行说明 | 15 | 存在 README 且含 How to run / 复现步骤章节(仅有 README 得 8 分) | +| 开源许可证 | 15 | LICENSE / COPYING 文件 | +| 引用信息 | 10 | CITATION.cff 或 README 中的 BibTeX/引用章节 | +| 依赖清单(环境固化) | 15 | requirements.txt / environment.yml / go.mod / package.json 等 | +| 运行入口 | 10 | Makefile / run.sh / main.py / Dockerfile / scripts 目录 | +| 数据可得性说明 | 10 | data 目录或 README 中的数据集来源说明 | +| 测试/验证代码 | 10 | tests 目录 | +| 版本发布(成果固化) | 15 | 至少一个 release | + +总分映射等级:≥85 A(可复现性良好)/ ≥70 B / ≥50 C / <50 D(复现困难)。 + +## 使用方式 + +### 方式一:确定性脚本(推荐,可进 CI) + +```bash +python3 examples/research/repro-audit/scripts/repro_audit.py \ + --owner --repo [--ref ] [--apply] +``` + +- 默认 dry-run 只输出报告;`--apply` 用 `issue +create` 回写改进 tracking issue +- 退出码:`0` 总分 ≥70(基本可复现),`2` 存在明显缺口 + +### 方式二:AI Agent 手工执行命令链 + +```bash +# 1. 结构采集 +gitlink-cli repo +tree --owner --repo --format json +# 2. README 内容 +gitlink-cli file +view --owner --repo --path README.md --raw +# 3. 版本发布 +gitlink-cli release +list --owner --repo --format json +# 4.(确认后)回写改进 issue +gitlink-cli issue +create --owner --repo -t "[repro-audit] 复现性审计报告" -b "<报告>" +``` + +AI 按上表逐项打分,报告须给出每项证据与修复建议,不允许笼统结论。 + +## 注意事项 + +- 判据基于仓库顶层结构与 README 文本,属于必要条件检查:高分不代表结果一定可复现,低分则一定存在工程缺口。 +- 对论文笔记类仓库(纯 Markdown)等级普遍偏低,属预期行为;报告建议仅在其确为「实验代码仓库」时回写。 +- 与 [`../gitlink-doc-sync/SKILL.md`](../gitlink-doc-sync/SKILL.md)(文档一致性)、[`../gitlink-license-compliance/SKILL.md`](../gitlink-license-compliance/SKILL.md)(许可证合规)互补:本技能聚焦复现性维度的整体评估。