mirror of https://github.com/apache/cassandra
Add suite of skills for AI
patch by Alex Petrov; reviewed by Stefan Miklosovic for CASSANDRA-21373
This commit is contained in:
Alex Petrov
Stefan Miklosovic
parent
a434c91c56
commit
3831d8265d
|
|
@ -43,6 +43,7 @@
|
||||||
<fileset dir="." includesfile="${build.dir}/.ratinclude">
|
<fileset dir="." includesfile="${build.dir}/.ratinclude">
|
||||||
<!-- Config files with not much creativity -->
|
<!-- Config files with not much creativity -->
|
||||||
<exclude name=".asf.yaml"/>
|
<exclude name=".asf.yaml"/>
|
||||||
|
<exclude name=".claude/**/*"/>
|
||||||
<exclude name=".git-blame-ignore-revs"/>
|
<exclude name=".git-blame-ignore-revs"/>
|
||||||
<exclude name=".idea/codeStyles/Project.xml"/>
|
<exclude name=".idea/codeStyles/Project.xml"/>
|
||||||
<exclude name=".idea/codeStyles/codeStyleConfig.xml"/>
|
<exclude name=".idea/codeStyles/codeStyleConfig.xml"/>
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,176 @@
|
||||||
|
# Correctness Skills
|
||||||
|
|
||||||
|
A collection of skills for finding, understanding, reproducing, and preventing bugs.
|
||||||
|
|
||||||
|
General approach and here is to try and codify whatever an engineer that cares about correctness would do to validate the code. Sometimes we do not have
|
||||||
|
a sufficient amount of time to write a spec, or do one more pass of the review, or review a patch we have not been involved in. This is a collection of
|
||||||
|
things I have been using to check if my own work is up to a standard, but also to pursue some of the ideas I would otherwise not have time to pursue:
|
||||||
|
explore a speculative idea, write a TLA+ specification for a subsystem, dig into the files with high commit density, etc.
|
||||||
|
|
||||||
|
This started with a small pull request review prompt, and grew as different "what if" ideas were popping up. While listening to a podcast, I heard a person
|
||||||
|
on it mentioned that they kept a log of issues popping up again and again and they were encoding them, either in form of code review checklist, or in a form
|
||||||
|
of tooling, which is where [bug-archaeology](#bug-archaeology) came from. I have indexed 3000 bugs from Apache Cassandra codebase, and made a library/checklist of
|
||||||
|
things that can be used to verify the code in the future.
|
||||||
|
|
||||||
|
Even though the tool was yielding good results and finding issues in Apache Cassandra codebase, I was not able to quantitatively confirm its quality. At which
|
||||||
|
point I have introduced evals, and started iterating. Compare the tool's output either to its own previous run, or to a simpler prompt or some popular skill
|
||||||
|
that does the same thing. Iterate, improve, and refine until you can score better. Without evals, results are purely anecdotal. Evals give you a way to quantify
|
||||||
|
and iterate.
|
||||||
|
|
||||||
|
Recent eval that I have taken a note of was; finding _human confirmed_ issues in the codebase the tool _was not indexed on_:
|
||||||
|
|
||||||
|
```
|
||||||
|
Volume & signal-to-noise
|
||||||
|
|
||||||
|
┌─────────────────────────────┬──────┬────────┐
|
||||||
|
│ Metric │<this>│<other> │
|
||||||
|
├─────────────────────────────┼──────┼────────┤
|
||||||
|
│ Total findings (20 commits) │ 151 │ ~193 │
|
||||||
|
├─────────────────────────────┼──────┼────────┤
|
||||||
|
│ Avg findings/commit │ 7.6 │ 9.7 │
|
||||||
|
├─────────────────────────────┼──────┼────────┤
|
||||||
|
│ High+Critical share │ 37% │ 20% │
|
||||||
|
├─────────────────────────────┼──────┼────────┤
|
||||||
|
│ Low findings share │ ~30% │ 53% │
|
||||||
|
└─────────────────────────────┴──────┴────────┘
|
||||||
|
|
||||||
|
<this> finds fewer things but a far higher fraction are High/Critical. <popular open source skill> finds more but is noisier (over half Low severity).
|
||||||
|
|
||||||
|
---
|
||||||
|
Coverage asymmetry
|
||||||
|
|
||||||
|
- Both found: 3 bugs (all resource leaks with visible lifecycle asymmetry)
|
||||||
|
- <this> only: 1 bug (TOCTOU double-call in SHA 19 — Concurrency specialist)
|
||||||
|
- <popular open source skill> only: 2 bugs
|
||||||
|
```
|
||||||
|
|
||||||
|
Original "archaeology" later got split into shallow- and deep- review skills, which helped to get consistently high score on larger patches, too.
|
||||||
|
New(er) targeted-review is an experiment that came after trying (and failing) to generalize existing bugs/issues into semgrep scripts: essentially,
|
||||||
|
patterns were at first too noisy, and later were catching only very specific permutations of issues, so this approach allows to do "semantic" checks
|
||||||
|
by allowing the model to make a checklist of most probable patterns and then going through the patch once again. Evals for this one are somewhat
|
||||||
|
more difficult to quantify, as they are designed to trigger only for specific issues (which they do).
|
||||||
|
|
||||||
|
A `mega-review` patch is a final iteration that chunks up larger patches and performs multiple steps/iterations over each one of them based on a
|
||||||
|
set of rules.
|
||||||
|
|
||||||
|
For writing repros, you will often have to guide the model and suppress its attempts to go deep into internals to conjure up a repro that does exactly
|
||||||
|
what it wants but does not reveal the real issue. Introduce guidelines and close the loop by setting strict and clear exit criteria.
|
||||||
|
|
||||||
|
**Example prompt**
|
||||||
|
|
||||||
|
Once the skills are installed, try this multi-pass review workflow on a subsystem or commit SHA:
|
||||||
|
|
||||||
|
> Review all the code related to \<subsystem\> (or just give it an SHA). Do the first pass
|
||||||
|
> using `/shallow-review`. Then analyze the code using `/patch-explainer`, identify core
|
||||||
|
> components that might be most prone to critical mistakes, and use `/deep-review` skill on
|
||||||
|
> the files related to these components. Do a third `/deep-review` pass from the files
|
||||||
|
> identified as "hot" using `/heatmap` skill, and a final pass of `/targeted-review` for the
|
||||||
|
> files that contain the most tricky logic.
|
||||||
|
|
||||||
|
This chains four skills — broad scan, structural understanding, targeted deep review, and
|
||||||
|
churn-guided deep review — to progressively narrow focus onto the code most likely to
|
||||||
|
contain correctness bugs.
|
||||||
|
|
||||||
|
## Where Do I Start?
|
||||||
|
|
||||||
|
Pick your entry point based on what you're looking at right now:
|
||||||
|
|
||||||
|
| You have… | Start with |
|
||||||
|
|----------------------------------|-----------------------------------------|
|
||||||
|
| A small patch to review | [shallow-review](#shallow-review), then [deep-review](#deep-review) for flagged areas |
|
||||||
|
| A medium-to-large patch (50-1000 LOC) | [targeted-review](#targeted-review) |
|
||||||
|
| A large patch or feature branch (1000+ LOC) | [mega-review](#mega-review) |
|
||||||
|
| A bug report to reproduce | [write-reproducer](#write-reproducer) |
|
||||||
|
| Code you don't understand yet | [patch-explainer](#patch-explainer) |
|
||||||
|
| A protocol or algorithm to verify| [tla-plus](#tla-plus) |
|
||||||
|
| A repo and no idea where to look | [heatmap](#heatmap) |
|
||||||
|
| A repo's bug history to learn from | [bug-archaeology](#bug-archaeology) |
|
||||||
|
| A Cassandra cluster test to write| [cassandra-injvm-dtest](#cassandra-injvm-dtest) |
|
||||||
|
|
||||||
|
## Skills
|
||||||
|
|
||||||
|
### shallow-review
|
||||||
|
|
||||||
|
Quick, broad bug scan. Six specialist agents review the same patch in parallel, each through a different lens: logic & types, boundaries & I/O, concurrency & state, resources & serialization, absence analysis, and API completeness. Findings are merged and deduplicated. Good as a first pass — it's fast and catches surface-level issues across a wide area.
|
||||||
|
|
||||||
|
→ `shallow-review/`
|
||||||
|
|
||||||
|
### deep-review
|
||||||
|
|
||||||
|
Focused, thorough review of specific files using the full 444-pattern catalog. Starts with a heatmap pass to identify the highest-churn files and lines, then concentrates review effort there — reading source (not just diffs), searching the codebase for context, and cross-referencing against the complete pattern database. Use it when shallow-review flags something worth digging into, or when reviewing critical-path code changes.
|
||||||
|
|
||||||
|
→ `deep-review/`
|
||||||
|
|
||||||
|
### targeted-review
|
||||||
|
|
||||||
|
Findings-driven review for medium-to-large patches (50–1000 LOC). Runs patch-explainer and codebase-analysis in parallel to ground the review, then makes 3–5 independent passes over the bug-pattern catalog to pick only the categories whose diff signals match. Items selected across multiple passes are promoted; the result is grouped by review focus and dispatched to parallel subagents — each with a tight, evidence-selected checklist rather than a fixed set of lenses. Higher signal-to-noise than shallow-review on complex changes, less exhaustive than deep-review on large ones.
|
||||||
|
|
||||||
|
→ `targeted-review/`
|
||||||
|
|
||||||
|
### mega-review
|
||||||
|
|
||||||
|
Multi-pass deep review for large patches (1000+ LOC) or feature branches. Decomposes the patch into HIGH/MEDIUM/LOW-risk files and commits, then runs deep-review per file, targeted-review across the high-risk scope, and shallow-review per commit — all in parallel. A cross-cut phase follows to amplify patterns found across files, verify fix correctness, and check cross-subsystem consistency. A coverage gate ensures every file was reviewed before findings are merged. Use when a single-pass review would spread attention too thin.
|
||||||
|
|
||||||
|
→ `mega-review/`
|
||||||
|
|
||||||
|
### write-reproducer
|
||||||
|
|
||||||
|
Turns a bug description into a minimal, self-contained, runnable reproduction. Covers the full workflow: failure characterization, scope selection, writing the repro, verifying it fails for the right reason, and minimizing to the smallest possible trigger.
|
||||||
|
|
||||||
|
→ `write-reproducer/`
|
||||||
|
|
||||||
|
### patch-explainer
|
||||||
|
|
||||||
|
Deep code analysis with ASCII visualizations. Produces diagrams showing structure, data/control flow, state transitions, before/after comparisons, concurrency interactions, assumptions, and failure modes. Use it to build understanding before reviewing, or to explain a change to someone else.
|
||||||
|
|
||||||
|
→ `patch-explainer/`
|
||||||
|
|
||||||
|
### tla-plus
|
||||||
|
|
||||||
|
Create, run, and verify TLA+ and PlusCal formal specifications. Model distributed systems, protocols, concurrent algorithms, and state machines. Can compose specs from code, find divergences between spec and implementation, and surface race conditions and invariant violations through exhaustive model checking.
|
||||||
|
|
||||||
|
> **Note:** This skill requires `tla2tools.jar` to run the TLC model checker. Place it in `tla-plus/lib/` before use.
|
||||||
|
|
||||||
|
→ `tla-plus/`
|
||||||
|
|
||||||
|
### heatmap
|
||||||
|
|
||||||
|
Git heatmap analysis that identifies high-churn files and lines — the places where bugs statistically concentrate. Use it to decide where to focus review effort in a large codebase, during bug hunts, security audits, or when onboarding onto unfamiliar code.
|
||||||
|
|
||||||
|
→ `heatmap/`
|
||||||
|
|
||||||
|
### bug-archaeology
|
||||||
|
|
||||||
|
Mines bug patterns from a repository's commit history. Discovers bug-fix commits via git log heuristics, analyzes each with subagents, and synthesizes a generalized `PATTERNS.md` with repo-specific details stripped. Use it to learn what kinds of bugs a codebase tends to produce — then feed those patterns into reviews.
|
||||||
|
|
||||||
|
→ `bug-archaeology/`
|
||||||
|
|
||||||
|
### cassandra-injvm-dtest
|
||||||
|
|
||||||
|
Guide for writing Apache Cassandra in-JVM distributed tests. Covers cluster creation, configuration, instance lifecycle, query execution, message filtering for fault injection, and debugging classloader isolation issues. Domain-specific, but included here since correctness testing in distributed databases is its own discipline.
|
||||||
|
|
||||||
|
→ `cassandra-injvm-dtest/`
|
||||||
|
|
||||||
|
## Typical Workflows
|
||||||
|
|
||||||
|
**Reviewing a patch for correctness:**
|
||||||
|
heatmap → patch-explainer → shallow-review → deep-review (on hot files)
|
||||||
|
|
||||||
|
**Reviewing a medium-to-large patch:**
|
||||||
|
targeted-review → deep-review (on files flagged HIGH)
|
||||||
|
|
||||||
|
**Reviewing a large patch or feature branch:**
|
||||||
|
mega-review (orchestrates targeted-review, shallow-review, and deep-review automatically)
|
||||||
|
|
||||||
|
**Investigating a bug report:**
|
||||||
|
patch-explainer (understand the area) → write-reproducer → shallow-review (on the fix)
|
||||||
|
|
||||||
|
**Verifying a protocol change:**
|
||||||
|
patch-explainer → tla-plus (model the protocol) → deep-review (on the implementation)
|
||||||
|
|
||||||
|
**Learning a new codebase's failure modes:**
|
||||||
|
bug-archaeology → heatmap → deep-review (on the overlap between hot code and historical bug patterns)
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0. See [LICENSE.txt](../../LICENSE.txt) for details.
|
||||||
|
|
@ -0,0 +1,126 @@
|
||||||
|
---
|
||||||
|
name: bug-archaeology
|
||||||
|
version: "1.0.0"
|
||||||
|
description: >
|
||||||
|
Mine bug patterns from any git repository. Discovers bug-fix commits via git log
|
||||||
|
heuristics, analyzes each in parallel with subagents, writes individual analysis files,
|
||||||
|
and synthesizes a generalized PATTERNS.md with repo-specific details stripped.
|
||||||
|
Invoke explicitly with /bug-archaeology.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Bug Archaeology
|
||||||
|
|
||||||
|
Systematic extraction of reusable bug-detection heuristics from a repository's commit history.
|
||||||
|
|
||||||
|
## Arguments
|
||||||
|
|
||||||
|
All optional. Parse from the user's invocation string.
|
||||||
|
|
||||||
|
| Arg | Default | Description |
|
||||||
|
|-----|---------|-------------|
|
||||||
|
| `--range` | `HEAD~200..HEAD` | Git revision range |
|
||||||
|
| `--path` | (all files) | Restrict to commits touching this path glob |
|
||||||
|
| `--since` / `--until` | (none) | Date filter for `git log` |
|
||||||
|
| `--limit` | 200 | Max bug-fix commits to analyze |
|
||||||
|
| `--batch-size` | 10 | Commits per subagent |
|
||||||
|
| `--output` | `./bug-archaeology/` | Output directory |
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
Three sequential phases. Each phase completes fully before the next starts.
|
||||||
|
|
||||||
|
### Phase 1: Discovery
|
||||||
|
|
||||||
|
Find bug-fix commits via git log heuristics.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git log --no-merges --pretty=format:"%H %s" [--since=X] [--until=X] <range> [-- <path>] \
|
||||||
|
| grep -iE '(fix|bug|patch|resolve|repair|correct|workaround|hotfix|regression|#[0-9]+|[A-Z]+-[0-9]+)'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Filter out** commits whose message matches ONLY cosmetic keywords (`typo`, `whitespace`, `import`, `format`, `style`, `cleanup`, `rename`) with no bug keywords.
|
||||||
|
|
||||||
|
**Truncate** to `--limit`.
|
||||||
|
|
||||||
|
**Write** `QUEUE.txt` to the output directory:
|
||||||
|
```
|
||||||
|
0001 abc123def First line of commit message
|
||||||
|
0002 def456abc Another commit message
|
||||||
|
```
|
||||||
|
|
||||||
|
**Resumption**: If `PROGRESS.md` exists, read it and exclude commits already marked `done` or `skipped`. Re-include `pending` and `error` entries.
|
||||||
|
|
||||||
|
### Phase 2: Analysis (parallelized)
|
||||||
|
|
||||||
|
Spawn subagents to analyze commits in batches. Each commit is independent.
|
||||||
|
|
||||||
|
**Subagent prompt template** — adapt as needed:
|
||||||
|
```
|
||||||
|
Read the file at references/per-bug-format.md for the output format specification.
|
||||||
|
|
||||||
|
Analyze these commits from the repository at <REPO_PATH>:
|
||||||
|
<list of "NNNN hash message" lines>
|
||||||
|
|
||||||
|
For each commit:
|
||||||
|
1. Run: git show <hash>
|
||||||
|
2. If the commit is cosmetic, test-only, refactor-only, or a merge with no
|
||||||
|
original changes — mark it as skipped with a one-word reason.
|
||||||
|
3. Otherwise, write the analysis to <OUTPUT>/bug-<NNNN>-<slug>.md
|
||||||
|
following the format in per-bug-format.md exactly.
|
||||||
|
|
||||||
|
Return a status line per commit: NNNN <hash> done|skipped [reason-or-filename]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Parallelism**: Launch up to 5 subagent batches concurrently. Wait for all to complete before the next wave.
|
||||||
|
|
||||||
|
**Progress tracking**: After each wave, append results to `PROGRESS.md`:
|
||||||
|
```markdown
|
||||||
|
| # | Commit | Status | File / Reason |
|
||||||
|
|---|--------|--------|---------------|
|
||||||
|
| 0001 | abc123d | done | bug-0001-null-after-lookup.md |
|
||||||
|
| 0002 | def456a | skipped | cosmetic |
|
||||||
|
```
|
||||||
|
|
||||||
|
**Errors**: Mark as `error` with a one-line reason. Don't retry — the user re-invokes with `--resume`.
|
||||||
|
|
||||||
|
### Phase 3: Synthesis
|
||||||
|
|
||||||
|
After all analysis is complete, produce `PATTERNS.md`. Read `references/synthesis-format.md` for the full specification.
|
||||||
|
|
||||||
|
**Process**:
|
||||||
|
1. Read all `bug-*.md` files in the output directory
|
||||||
|
2. Parse the `Tags` section from each — extract `category` and `code-pattern`
|
||||||
|
3. Group bugs by `code-pattern` tag
|
||||||
|
4. For each group of 2+ bugs with the same code-pattern:
|
||||||
|
- Extract the shared Detection Rule (generalize if they differ across instances)
|
||||||
|
- Pick the most representative Example
|
||||||
|
5. **Generalize** — strip repo-specific details:
|
||||||
|
- Replace specific class/method names with structural descriptions
|
||||||
|
- Remove ticket IDs, author names, project-specific subsystem labels
|
||||||
|
- Keep language-specific API names only if scope is that language or broader
|
||||||
|
6. Group patterns by `category` and write `PATTERNS.md`
|
||||||
|
7. Patterns with only 1 instance go in a "Singletons" section at the end
|
||||||
|
|
||||||
|
## Reference Files
|
||||||
|
|
||||||
|
| File | Loaded by | Purpose |
|
||||||
|
|------|-----------|---------|
|
||||||
|
| `references/per-bug-format.md` | Subagents (Phase 2) | Per-bug output template and field guidelines |
|
||||||
|
| `references/synthesis-format.md` | Orchestrator (Phase 3) | PATTERNS.md structure and generalization rules |
|
||||||
|
|
||||||
|
## Evaluating the Skill
|
||||||
|
|
||||||
|
Full instructions are in `references/EVAL-PROMPT.md`. Summary:
|
||||||
|
|
||||||
|
1. **Pick a bug** from the archaeology output and trace its introducing commit via `git blame`.
|
||||||
|
2. **Extract the patch** with `git diff ${INTRO}~1 $INTRO`.
|
||||||
|
3. **Choose a review skill** — the user must specify one:
|
||||||
|
- `/shallow-review` — quick 6-specialist parallel scan, any patch size
|
||||||
|
- `/deep-review` — focused on specific files, complex logic
|
||||||
|
- `/targeted-review` — findings-driven, medium-to-large patches
|
||||||
|
- `/mega-review` — large feature branches or 1000+ LOC diffs
|
||||||
|
4. **Run the chosen skill** against the extracted patch.
|
||||||
|
5. **Score** the output: Exact / Partial / Different bug / Miss.
|
||||||
|
6. **Prod misses** — tell the skill what it missed and ask which checklist item would have caught it; use the answer to improve that skill's reference files.
|
||||||
|
|
||||||
|
Target: >90% hit rate (exact + partial), <5% miss rate across a batch of N bugs.
|
||||||
|
|
@ -0,0 +1,126 @@
|
||||||
|
# Bug-Archaeology -- Eval Prompt
|
||||||
|
|
||||||
|
Run this to test a review skill against a known bug-introducing patch.
|
||||||
|
|
||||||
|
## Setup: Find an Introducing Patch
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Pick a random bug from archaeology, trace to the introducing commit
|
||||||
|
BUG_DIR=/path/to/bug-archaeology
|
||||||
|
REPO=/path/to/cassandra
|
||||||
|
|
||||||
|
# 1. Pick a bug file
|
||||||
|
BUGFILE=$(ls $BUG_DIR/bug-*.md | shuf -n1)
|
||||||
|
FIX_HASH=$(grep -oP 'Commit\*\*: \K\w+' "$BUGFILE" | head -1)
|
||||||
|
|
||||||
|
# 2. Find the primary changed Java file
|
||||||
|
PRIMARY=$(git -C $REPO diff --name-only "${FIX_HASH}~1" $FIX_HASH | grep '\.java$' | grep -v 'test/' | head -1)
|
||||||
|
|
||||||
|
# 3. Find the first removed (buggy) line number
|
||||||
|
LINE=$(git -C $REPO diff "${FIX_HASH}~1" $FIX_HASH -- "$PRIMARY" | \
|
||||||
|
awk '/^@@ -([0-9]+)/{line=$0; gsub(/.*-/,"",line); gsub(/,.*/,"",line)} /^-[^-]/{if(length($0)>6) {print line; exit}}' )
|
||||||
|
|
||||||
|
# 4. Blame to find introducing commit
|
||||||
|
INTRO=$(git -C $REPO blame -L "$LINE,$LINE" "${FIX_HASH}~1" --porcelain -- "$PRIMARY" | head -1 | cut -c1-40)
|
||||||
|
|
||||||
|
# 5. Extract introducing diff
|
||||||
|
git -C $REPO diff "${INTRO}~1" $INTRO -- "$PRIMARY" > /tmp/eval_patch.diff
|
||||||
|
echo "Bug: $(head -1 "$BUGFILE")"
|
||||||
|
echo "Introducing commit: $INTRO"
|
||||||
|
echo "Diff: $(wc -l < /tmp/eval_patch.diff) lines"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Choosing a Review Skill
|
||||||
|
|
||||||
|
The user must specify which skill to use. Pick based on patch characteristics:
|
||||||
|
|
||||||
|
| Skill | When to use | LOC |
|
||||||
|
|-------|-------------|-----|
|
||||||
|
| `/shallow-review` | Quick first pass, broad surface scan | Any |
|
||||||
|
| `/deep-review` | Focused on specific files, complex logic | Any |
|
||||||
|
| `/targeted-review` | Findings-driven, medium-to-large changes | Medium+ |
|
||||||
|
| `/mega-review` | Large feature branches, multi-commit ranges | 1000+ |
|
||||||
|
|
||||||
|
## Eval: Running the Review
|
||||||
|
|
||||||
|
Invoke the chosen skill against the extracted patch. Provide the patch path as context.
|
||||||
|
|
||||||
|
### shallow-review
|
||||||
|
```
|
||||||
|
/shallow-review
|
||||||
|
|
||||||
|
Patch: /tmp/eval_patch.diff
|
||||||
|
```
|
||||||
|
|
||||||
|
### deep-review
|
||||||
|
```
|
||||||
|
/deep-review
|
||||||
|
|
||||||
|
Patch: /tmp/eval_patch.diff
|
||||||
|
Focus file: <PRIMARY file from setup>
|
||||||
|
```
|
||||||
|
|
||||||
|
### targeted-review
|
||||||
|
```
|
||||||
|
/targeted-review
|
||||||
|
|
||||||
|
Patch: /tmp/eval_patch.diff
|
||||||
|
```
|
||||||
|
|
||||||
|
### mega-review
|
||||||
|
```
|
||||||
|
/mega-review
|
||||||
|
|
||||||
|
Patch: /tmp/eval_patch.diff
|
||||||
|
```
|
||||||
|
|
||||||
|
## Scoring
|
||||||
|
|
||||||
|
After the skill reports, compare findings against the actual bug:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
grep -A5 '## Root Cause' "$BUGFILE"
|
||||||
|
```
|
||||||
|
|
||||||
|
Score the review output as:
|
||||||
|
- **Exact**: The skill identified the specific bug that was later fixed
|
||||||
|
- **Partial**: The skill flagged the right area/pattern but described a different specific issue
|
||||||
|
- **Different bug**: The skill found real bugs, but not the target
|
||||||
|
- **Miss**: The skill found nothing relevant
|
||||||
|
|
||||||
|
For `shallow-review`, also track **per-specialist** hits (Logic, Boundary, Concurrency, Resources, Absence, Completeness) to identify which domains need improvement.
|
||||||
|
|
||||||
|
## Comparison Eval
|
||||||
|
|
||||||
|
To compare skill effectiveness, run the same patch through multiple skills and record scores side by side:
|
||||||
|
|
||||||
|
| Patch | shallow | deep | targeted | mega |
|
||||||
|
|-------|---------|------|----------|------|
|
||||||
|
| bug-0001 | Exact | Partial | Exact | Exact |
|
||||||
|
| bug-0042 | Miss | Exact | Partial | Exact |
|
||||||
|
|
||||||
|
This identifies which skill is most reliable for which bug categories.
|
||||||
|
|
||||||
|
## Prodding (for Misses)
|
||||||
|
|
||||||
|
If a skill missed, tell it what the bug actually was and ask:
|
||||||
|
|
||||||
|
```
|
||||||
|
You missed this bug: [description of actual bug].
|
||||||
|
Which checklist or pattern in the skill would have helped you find it?
|
||||||
|
If none exists, propose a new item to add.
|
||||||
|
```
|
||||||
|
|
||||||
|
Use the feedback to improve the relevant skill's reference files.
|
||||||
|
|
||||||
|
## Batch Eval
|
||||||
|
|
||||||
|
For batch evaluation across N bugs:
|
||||||
|
1. Extract N patches using the setup script
|
||||||
|
2. For each patch, invoke the chosen skill (one invocation per patch)
|
||||||
|
3. Wait for results, score each
|
||||||
|
4. Tabulate: exact/partial/different/miss rates
|
||||||
|
5. For shallow-review: also tabulate per-specialist hit rates
|
||||||
|
6. Optionally compare two skills on the same patches
|
||||||
|
|
||||||
|
Target: >90% hit rate (exact + partial), <5% miss rate.
|
||||||
|
|
@ -0,0 +1,108 @@
|
||||||
|
# Per-Bug Analysis Format
|
||||||
|
|
||||||
|
Every analysis file MUST follow this structure. No extra sections. No prose padding.
|
||||||
|
|
||||||
|
## Output Template
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# Bug NNNN: [title — what broke, not how it was fixed]
|
||||||
|
|
||||||
|
**Commit**: <hash> · **Ticket**: <ID or n/a>
|
||||||
|
|
||||||
|
## Detection Rule
|
||||||
|
|
||||||
|
[One imperative sentence. Starts with "When" or "If". A reusable heuristic
|
||||||
|
that would catch this CLASS of bug during review. NOT a description of this
|
||||||
|
specific bug. Two bugs with identical detection rules are the same pattern.]
|
||||||
|
|
||||||
|
## Code Smell
|
||||||
|
|
||||||
|
[What the buggy code LOOKS LIKE to a reviewer. The observable surface pattern.
|
||||||
|
1-2 sentences max. Describe what you'd grep for or see during review.]
|
||||||
|
|
||||||
|
## Tags
|
||||||
|
|
||||||
|
- scope: [universal | <language> | distributed-systems | project-specific]
|
||||||
|
- category: [see category list below]
|
||||||
|
- code-pattern: [2-4 word hyphenated tag, e.g. "null-after-lookup"]
|
||||||
|
- severity: [silent-wrong-result | crash | hang | data-loss | performance | cosmetic]
|
||||||
|
- subsystem: [free-form — infer from file paths and module structure]
|
||||||
|
|
||||||
|
## Root Cause
|
||||||
|
|
||||||
|
[2-3 sentences. What was actually wrong and why. Include the key insight.]
|
||||||
|
|
||||||
|
## Example
|
||||||
|
|
||||||
|
```<language>
|
||||||
|
// BEFORE
|
||||||
|
[minimal buggy code — essential lines only, ≤15 lines]
|
||||||
|
```
|
||||||
|
|
||||||
|
```<language>
|
||||||
|
// AFTER
|
||||||
|
[minimal fixed code — same scope, ≤15 lines]
|
||||||
|
```
|
||||||
|
```
|
||||||
|
|
||||||
|
## Field Guidelines
|
||||||
|
|
||||||
|
### Detection Rule — THE MOST IMPORTANT FIELD
|
||||||
|
|
||||||
|
Write it as a checklist item a reviewer walks through on EVERY review:
|
||||||
|
|
||||||
|
- **General enough** to catch the same bug in a different file/context
|
||||||
|
- **Specific enough** that a "yes" answer actually indicates a likely bug
|
||||||
|
- **Imperative** — starts with "When" or "If", reads as a yes/no check
|
||||||
|
|
||||||
|
Good: "When a callback is registered on a future AND the caller also blocks on future.get(), verify that all state mutations happen after get(), not in the callback."
|
||||||
|
|
||||||
|
Bad: "Check that bootstrapFinished() is called in the right place." (too specific)
|
||||||
|
Bad: "Be careful with race conditions." (too vague)
|
||||||
|
|
||||||
|
### Code Smell — WHAT IT LOOKS LIKE
|
||||||
|
|
||||||
|
Describe the visual/structural pattern a reviewer or grep would find:
|
||||||
|
|
||||||
|
Good: "A Futures.addCallback() on the same future that is later awaited with .get() in the same method, where the callback mutates shared state."
|
||||||
|
|
||||||
|
Bad: "Race condition in bootstrap code." (not visual, not greppable)
|
||||||
|
|
||||||
|
### Tags
|
||||||
|
|
||||||
|
**scope**:
|
||||||
|
- `universal`: Any language, any codebase (off-by-one, resource not closed on error path)
|
||||||
|
- `<language>` (e.g. `java`, `python`, `go`): Language-specific, not domain-specific
|
||||||
|
- `distributed-systems`: Applies to any distributed system (epoch boundaries, topology changes)
|
||||||
|
- `project-specific`: Only meaningful in this project's context
|
||||||
|
|
||||||
|
**category** (recommended list — extend if none fit):
|
||||||
|
`logic-error` | `race-condition` | `missing-null-check` | `off-by-one` | `resource-leak` | `wrong-constant` | `wrong-serialization` | `state-not-cleaned` | `deadlock` | `silent-skip` | `missing-override` | `non-atomic-write` | `wrong-filter-result`
|
||||||
|
|
||||||
|
**code-pattern** — the clustering key. Short, stable tag for grouping related bugs. Reuse existing tags when the pattern matches; invent new ones when genuinely novel.
|
||||||
|
|
||||||
|
Examples: `cast-truncates-size`, `null-after-map-lookup`, `close-not-on-all-paths`, `retry-no-escape`, `serialize-size-mismatch`, `stale-state-after-lifecycle`, `mutable-shared-buffer`, `else-if-masks-second`, `write-not-atomic`, `sentinel-not-handled`
|
||||||
|
|
||||||
|
### Root Cause — BRIEF
|
||||||
|
|
||||||
|
2-3 sentences. Don't repeat the detection rule. Focus on the mechanism.
|
||||||
|
|
||||||
|
### Example — MINIMAL
|
||||||
|
|
||||||
|
Minimum lines to show the bug and fix. Strip imports, logging, comments. 5-15 lines per side.
|
||||||
|
|
||||||
|
## What to SKIP
|
||||||
|
|
||||||
|
- Purely cosmetic commits (typo, import reorder, whitespace, formatting)
|
||||||
|
- Commits that only add tests without fixing a bug
|
||||||
|
- Pure refactors with no behavioral change
|
||||||
|
- Merge commits with no original changes
|
||||||
|
|
||||||
|
Mark skipped commits with a one-word reason.
|
||||||
|
|
||||||
|
## What NOT to Include
|
||||||
|
|
||||||
|
- No "Test" section
|
||||||
|
- No author attribution
|
||||||
|
- No long prose explanations
|
||||||
|
- No issue tracker descriptions or external context
|
||||||
|
|
@ -0,0 +1,92 @@
|
||||||
|
# PATTERNS.md Synthesis Format
|
||||||
|
|
||||||
|
Instructions for producing the generalized pattern catalog from individual bug analysis files.
|
||||||
|
|
||||||
|
## Process
|
||||||
|
|
||||||
|
1. Read all `bug-*.md` files in the output directory
|
||||||
|
2. Parse each file's `## Tags` section — extract `category` and `code-pattern`
|
||||||
|
3. Group bugs sharing the same `code-pattern` tag
|
||||||
|
4. For groups with 2+ bugs: synthesize a generalized pattern entry
|
||||||
|
5. For singletons (1 bug): collect in a separate section
|
||||||
|
|
||||||
|
## Generalization Rules
|
||||||
|
|
||||||
|
The goal: a reader with no knowledge of the source repository can use PATTERNS.md as a bug-detection checklist.
|
||||||
|
|
||||||
|
**Strip:**
|
||||||
|
- All ticket IDs (PROJ-1234, #567, etc.)
|
||||||
|
- Author names, reviewer names
|
||||||
|
- Project-specific subsystem labels (replace with generic structural descriptions or omit)
|
||||||
|
- Specific class/method names — replace with structural descriptions:
|
||||||
|
- `ColumnFamilyStore.memtable` → "a shared mutable field"
|
||||||
|
- `AccordJournal.replay()` → "the journal replay path"
|
||||||
|
- `SSTableReader.compareTo()` → "a compareTo override"
|
||||||
|
|
||||||
|
**Keep:**
|
||||||
|
- Language-specific API names when scope is that language or broader
|
||||||
|
- `ByteBuffer.array()` without `arrayOffset()` — keep (Java-scoped, reusable)
|
||||||
|
- `Futures.addCallback()` racing `future.get()` — keep (Java-scoped)
|
||||||
|
- Generic distributed-systems concepts (epoch, topology change, quorum, leader election)
|
||||||
|
- The `code-pattern` tag itself (it's already designed to be generic)
|
||||||
|
|
||||||
|
**Merge Detection Rules:** If multiple bugs share a code-pattern but have slightly different Detection Rules, write one generalized rule that covers all instances. Prefer the most broadly applicable wording.
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# Bug Patterns
|
||||||
|
|
||||||
|
N patterns extracted from M analyzed commits.
|
||||||
|
|
||||||
|
## [Category Name] (X patterns, Y bugs)
|
||||||
|
|
||||||
|
### [code-pattern-tag] (Z bugs)
|
||||||
|
|
||||||
|
**Detection Rule**: [generalized imperative sentence]
|
||||||
|
|
||||||
|
**Code Smell**: [what to look for — greppable or visually scannable]
|
||||||
|
|
||||||
|
**Scope**: universal | <language> | distributed-systems
|
||||||
|
|
||||||
|
**Severity**: [most common across instances]
|
||||||
|
|
||||||
|
**Example**:
|
||||||
|
```
|
||||||
|
// BEFORE
|
||||||
|
[most illustrative before snippet from any instance, ≤10 lines]
|
||||||
|
```
|
||||||
|
```
|
||||||
|
// AFTER
|
||||||
|
[matching after snippet]
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### [next-pattern] (Z bugs)
|
||||||
|
...
|
||||||
|
|
||||||
|
## Singletons (observed once — may not generalize)
|
||||||
|
|
||||||
|
### [code-pattern-tag]
|
||||||
|
|
||||||
|
**Detection Rule**: ...
|
||||||
|
**Code Smell**: ...
|
||||||
|
**Scope**: ...
|
||||||
|
**From**: bug-NNNN (commit <short-hash>)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Ordering
|
||||||
|
|
||||||
|
1. Categories sorted by total bug count (most bugs first)
|
||||||
|
2. Patterns within each category sorted by instance count (most instances first)
|
||||||
|
3. Singletons section last, sorted alphabetically by code-pattern tag
|
||||||
|
|
||||||
|
## Quality Check
|
||||||
|
|
||||||
|
Before finalizing PATTERNS.md, verify:
|
||||||
|
- No ticket IDs appear anywhere in the file
|
||||||
|
- No project-specific class/method names appear (except in code examples where the structural pattern is clear from context)
|
||||||
|
- Every Detection Rule starts with "When" or "If"
|
||||||
|
- Every pattern with 2+ bugs has a representative Example
|
||||||
|
- Singleton patterns are clearly labeled as possibly non-generalizable
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,674 @@
|
||||||
|
# Advanced Testing Patterns
|
||||||
|
|
||||||
|
Advanced patterns for complex testing scenarios in the Cassandra in-JVM dtest framework.
|
||||||
|
|
||||||
|
## CMS (Cluster Metadata Service) Testing
|
||||||
|
|
||||||
|
### Testing CMS Convergence
|
||||||
|
|
||||||
|
Wait for all nodes to reach the same cluster metadata state:
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testCMSConvergence() throws IOException {
|
||||||
|
try (Cluster cluster = Cluster.build(3).start()) {
|
||||||
|
// Perform schema change
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 2}");
|
||||||
|
|
||||||
|
// Wait for CMS to quiesce
|
||||||
|
ClusterUtils.waitForCMSToQuiesce(cluster);
|
||||||
|
|
||||||
|
// Verify all nodes at same epoch
|
||||||
|
Epoch epoch1 = ClusterUtils.getCurrentEpoch(cluster.get(1));
|
||||||
|
Epoch epoch2 = ClusterUtils.getCurrentEpoch(cluster.get(2));
|
||||||
|
Epoch epoch3 = ClusterUtils.getCurrentEpoch(cluster.get(3));
|
||||||
|
|
||||||
|
Assert.assertEquals(epoch1, epoch2);
|
||||||
|
Assert.assertEquals(epoch2, epoch3);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pausing CMS Operations
|
||||||
|
|
||||||
|
Test scenarios by pausing CMS at specific points:
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testPausedCMS() throws Exception {
|
||||||
|
try (Cluster cluster = Cluster.build(3).start()) {
|
||||||
|
IInvokableInstance cms = cluster.get(1);
|
||||||
|
|
||||||
|
// Pause before adding node
|
||||||
|
Callable<Epoch> pauseHandle = ClusterUtils.pauseBeforeCommit(
|
||||||
|
cms,
|
||||||
|
transformation -> transformation instanceof AddNode
|
||||||
|
);
|
||||||
|
|
||||||
|
// Start add node operation in background
|
||||||
|
Future<?> addNodeFuture = CompletableFuture.runAsync(() -> {
|
||||||
|
IInvokableInstance newNode = ClusterUtils.addInstance(cluster);
|
||||||
|
newNode.startup();
|
||||||
|
});
|
||||||
|
|
||||||
|
// Wait for pause
|
||||||
|
Epoch pausedEpoch = pauseHandle.call();
|
||||||
|
System.out.println("Paused at epoch: " + pausedEpoch);
|
||||||
|
|
||||||
|
// Verify transformation hasn't completed
|
||||||
|
Epoch currentEpoch = ClusterUtils.getCurrentEpoch(cluster.get(2));
|
||||||
|
Assert.assertTrue(currentEpoch.getEpoch() < pausedEpoch.getEpoch());
|
||||||
|
|
||||||
|
// Resume
|
||||||
|
ClusterUtils.unpauseCommits(cms);
|
||||||
|
|
||||||
|
// Wait for completion
|
||||||
|
addNodeFuture.get(30, TimeUnit.SECONDS);
|
||||||
|
|
||||||
|
// Verify convergence
|
||||||
|
ClusterUtils.waitForCMSToQuiesce(cluster);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Testing CMS Snapshot
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testCMSSnapshot() throws IOException {
|
||||||
|
try (Cluster cluster = Cluster.build(2).start()) {
|
||||||
|
// Make some changes
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 1}");
|
||||||
|
|
||||||
|
// Trigger snapshot
|
||||||
|
Epoch snapshotEpoch = ClusterUtils.snapshotClusterMetadata(cluster.get(1));
|
||||||
|
|
||||||
|
// Verify snapshot created
|
||||||
|
Assert.assertNotNull(snapshotEpoch);
|
||||||
|
Assert.assertTrue(snapshotEpoch.getEpoch() > 0);
|
||||||
|
|
||||||
|
// Verify both nodes see snapshot
|
||||||
|
Epoch epoch1 = ClusterUtils.getCurrentEpoch(cluster.get(1));
|
||||||
|
Assert.assertEquals(snapshotEpoch, epoch1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Fetch Log from CMS
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testFetchFromCMS() throws IOException {
|
||||||
|
try (Cluster cluster = Cluster.build(3).start()) {
|
||||||
|
// Node 3 is behind
|
||||||
|
ClusterUtils.dropAllEntriesBeginningAt(cluster.get(3), Epoch.create(5));
|
||||||
|
|
||||||
|
// Perform operations
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 2}");
|
||||||
|
|
||||||
|
// Get target epoch from leader
|
||||||
|
Epoch targetEpoch = ClusterUtils.getCurrentEpoch(cluster.get(1));
|
||||||
|
|
||||||
|
// Fetch log on node 3
|
||||||
|
Epoch fetchedEpoch = ClusterUtils.fetchLogFromCMS(cluster.get(3), targetEpoch);
|
||||||
|
|
||||||
|
// Verify node 3 caught up
|
||||||
|
Assert.assertEquals(targetEpoch, fetchedEpoch);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Accord (CEP-15) Testing
|
||||||
|
|
||||||
|
### Testing Accord Transactions
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testAccordTransaction() throws IOException {
|
||||||
|
try (Cluster cluster = Cluster.build()
|
||||||
|
.withNodes(3)
|
||||||
|
.withConfig(config -> config
|
||||||
|
.set("accord.enabled", "true")
|
||||||
|
.set("accord.journal_directory", "/tmp/accord"))
|
||||||
|
.start()) {
|
||||||
|
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 2}");
|
||||||
|
cluster.schemaChange("CREATE TABLE test.accounts (" +
|
||||||
|
"id int PRIMARY KEY, " +
|
||||||
|
"balance int) " +
|
||||||
|
"WITH accord.enabled = true");
|
||||||
|
|
||||||
|
// Execute Accord transaction
|
||||||
|
cluster.coordinator(1).execute(
|
||||||
|
"BEGIN TRANSACTION " +
|
||||||
|
"INSERT INTO test.accounts (id, balance) VALUES (1, 100); " +
|
||||||
|
"INSERT INTO test.accounts (id, balance) VALUES (2, 200); " +
|
||||||
|
"COMMIT TRANSACTION",
|
||||||
|
ConsistencyLevel.QUORUM
|
||||||
|
);
|
||||||
|
|
||||||
|
// Verify transaction completed
|
||||||
|
Object[][] rows = cluster.coordinator(2).execute(
|
||||||
|
"SELECT * FROM test.accounts",
|
||||||
|
ConsistencyLevel.QUORUM
|
||||||
|
);
|
||||||
|
|
||||||
|
Assert.assertEquals(2, rows.length);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Query Accord Transaction State
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testAccordTxnState() throws IOException {
|
||||||
|
try (Cluster cluster = buildAccordCluster(3)) {
|
||||||
|
// Get TxnId from transaction
|
||||||
|
TxnId txnId = ...; // Obtain from transaction execution
|
||||||
|
|
||||||
|
// Query state across cluster
|
||||||
|
String state = ClusterUtils.queryTxnStateAsString(cluster, txnId);
|
||||||
|
System.out.println("Transaction state:\n" + state);
|
||||||
|
|
||||||
|
// Query specific nodes
|
||||||
|
LinkedHashMap<String, SimpleQueryResult> results =
|
||||||
|
ClusterUtils.queryTxnState(cluster, txnId, 1, 2);
|
||||||
|
|
||||||
|
for (Map.Entry<String, SimpleQueryResult> entry : results.entrySet()) {
|
||||||
|
System.out.println("Node: " + entry.getKey());
|
||||||
|
SimpleQueryResult result = entry.getValue();
|
||||||
|
while (result.hasNext()) {
|
||||||
|
Row row = result.next();
|
||||||
|
System.out.println(" " + Arrays.toString(row.toObjectArray()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Wait for Accord Epoch Ready
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testAccordEpochReady() throws IOException {
|
||||||
|
try (Cluster cluster = buildAccordCluster(3)) {
|
||||||
|
// Get current epoch
|
||||||
|
long currentEpoch = cluster.get(1).callOnInstance(() -> {
|
||||||
|
return ClusterMetadata.current().epoch.getEpoch();
|
||||||
|
});
|
||||||
|
|
||||||
|
// Wait for epoch to be ready for reads
|
||||||
|
ClusterUtils.awaitAccordEpochReady(cluster, currentEpoch);
|
||||||
|
|
||||||
|
// Now safe to perform Accord operations
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing Topology Changes
|
||||||
|
|
||||||
|
### Bootstrap with Streaming
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testBootstrapStreaming() throws IOException {
|
||||||
|
try (Cluster cluster = Cluster.build(2)
|
||||||
|
.withConfig(c -> c.with(NETWORK, GOSSIP))
|
||||||
|
.start()) {
|
||||||
|
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 2}");
|
||||||
|
cluster.schemaChange("CREATE TABLE test.data (k int PRIMARY KEY, v int)");
|
||||||
|
|
||||||
|
// Insert data
|
||||||
|
for (int i = 0; i < 1000; i++) {
|
||||||
|
cluster.coordinator(1).execute(
|
||||||
|
"INSERT INTO test.data (k, v) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.QUORUM, i, i * 10);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Force flush to create SSTables
|
||||||
|
cluster.forEach(instance -> instance.runOnInstance(() -> {
|
||||||
|
StorageService.instance.forceKeyspaceFlush("test");
|
||||||
|
}));
|
||||||
|
|
||||||
|
// Bootstrap new node
|
||||||
|
IInvokableInstance newNode = ClusterUtils.addInstance(cluster, config -> {
|
||||||
|
config.set("auto_bootstrap", true)
|
||||||
|
.with(NETWORK, GOSSIP);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Monitor log for streaming
|
||||||
|
long mark = newNode.logs().mark();
|
||||||
|
newNode.startup();
|
||||||
|
|
||||||
|
// Wait for bootstrap
|
||||||
|
newNode.logs().watchFor(mark, "Bootstrap completed");
|
||||||
|
ClusterUtils.awaitRingJoin(cluster.get(1), newNode);
|
||||||
|
|
||||||
|
// Verify data streamed correctly
|
||||||
|
int count = newNode.callOnInstance(() -> {
|
||||||
|
ColumnFamilyStore cfs = Keyspace.open("test").getColumnFamilyStore("data");
|
||||||
|
return cfs.getLiveSSTables().size();
|
||||||
|
});
|
||||||
|
|
||||||
|
Assert.assertTrue("Expected SSTables on new node", count > 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Testing Decommission with Pending Writes
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testDecommissionWithWrites() throws Exception {
|
||||||
|
try (Cluster cluster = Cluster.build(3).start()) {
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 2}");
|
||||||
|
cluster.schemaChange("CREATE TABLE test.data (k int PRIMARY KEY, v int)");
|
||||||
|
|
||||||
|
IInvokableInstance leaving = cluster.get(3);
|
||||||
|
|
||||||
|
// Start continuous writes
|
||||||
|
AtomicBoolean stopWrites = new AtomicBoolean(false);
|
||||||
|
AtomicInteger writeCount = new AtomicInteger(0);
|
||||||
|
|
||||||
|
Future<?> writeFuture = CompletableFuture.runAsync(() -> {
|
||||||
|
int i = 0;
|
||||||
|
while (!stopWrites.get()) {
|
||||||
|
try {
|
||||||
|
cluster.coordinator(1).execute(
|
||||||
|
"INSERT INTO test.data (k, v) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.QUORUM, i++, i * 10);
|
||||||
|
writeCount.incrementAndGet();
|
||||||
|
Thread.sleep(10);
|
||||||
|
} catch (Exception e) {
|
||||||
|
// Expected during decommission
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Start decommission
|
||||||
|
Future<Boolean> decommissionFuture = CompletableFuture.supplyAsync(() -> {
|
||||||
|
return ClusterUtils.decommission(leaving);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Wait for decommission
|
||||||
|
boolean success = decommissionFuture.get(60, TimeUnit.SECONDS);
|
||||||
|
Assert.assertTrue("Decommission failed", success);
|
||||||
|
|
||||||
|
// Stop writes
|
||||||
|
stopWrites.set(true);
|
||||||
|
writeFuture.get(5, TimeUnit.SECONDS);
|
||||||
|
|
||||||
|
// Verify node left
|
||||||
|
ClusterUtils.assertNotInRing(cluster.get(1), leaving);
|
||||||
|
|
||||||
|
System.out.println("Completed writes during decommission: " + writeCount.get());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Testing Replace with Data
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testReplaceWithData() throws IOException {
|
||||||
|
try (Cluster cluster = Cluster.build(3).start()) {
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 3}");
|
||||||
|
cluster.schemaChange("CREATE TABLE test.data (k int PRIMARY KEY, v int)");
|
||||||
|
|
||||||
|
// Insert data
|
||||||
|
for (int i = 0; i < 100; i++) {
|
||||||
|
cluster.coordinator(1).execute(
|
||||||
|
"INSERT INTO test.data (k, v) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.ALL, i, i * 10);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stop node 2 abruptly
|
||||||
|
IInvokableInstance failed = cluster.get(2);
|
||||||
|
ClusterUtils.stopAbrupt(cluster, failed);
|
||||||
|
|
||||||
|
// Verify data still readable on quorum
|
||||||
|
Object[][] rows = cluster.coordinator(1).execute(
|
||||||
|
"SELECT COUNT(*) FROM test.data",
|
||||||
|
ConsistencyLevel.QUORUM
|
||||||
|
);
|
||||||
|
Assert.assertEquals(100L, rows[0][0]);
|
||||||
|
|
||||||
|
// Replace failed node
|
||||||
|
IInvokableInstance replacement = ClusterUtils.replaceHostAndStart(
|
||||||
|
cluster,
|
||||||
|
failed
|
||||||
|
);
|
||||||
|
|
||||||
|
// Wait for replacement to join
|
||||||
|
ClusterUtils.awaitRingJoin(cluster.get(1), replacement);
|
||||||
|
|
||||||
|
// Verify data on replacement
|
||||||
|
int count = replacement.callOnInstance(() -> {
|
||||||
|
ColumnFamilyStore cfs = Keyspace.open("test").getColumnFamilyStore("data");
|
||||||
|
return (int) cfs.getMeanRowCount();
|
||||||
|
});
|
||||||
|
|
||||||
|
Assert.assertTrue("Expected data on replacement", count > 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing Failure Scenarios
|
||||||
|
|
||||||
|
### Network Partition Testing
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testNetworkPartition() throws IOException {
|
||||||
|
try (Cluster cluster = Cluster.build(3).start()) {
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 3}");
|
||||||
|
cluster.schemaChange("CREATE TABLE test.data (k int PRIMARY KEY, v int)");
|
||||||
|
|
||||||
|
// Partition: node 1,2 | node 3
|
||||||
|
cluster.filters().allVerbs()
|
||||||
|
.from(1).to(3).drop().on();
|
||||||
|
cluster.filters().allVerbs()
|
||||||
|
.from(2).to(3).drop().on();
|
||||||
|
cluster.filters().allVerbs()
|
||||||
|
.from(3).to(1, 2).drop().on();
|
||||||
|
|
||||||
|
// Write to majority partition (should succeed)
|
||||||
|
cluster.coordinator(1).execute(
|
||||||
|
"INSERT INTO test.data (k, v) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.QUORUM, 1, 10);
|
||||||
|
|
||||||
|
// Try write from minority partition (should timeout)
|
||||||
|
try {
|
||||||
|
cluster.coordinator(3).execute(
|
||||||
|
"INSERT INTO test.data (k, v) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.QUORUM, 2, 20);
|
||||||
|
Assert.fail("Expected timeout");
|
||||||
|
} catch (Exception e) {
|
||||||
|
Assert.assertTrue(e.getCause() instanceof UnavailableException ||
|
||||||
|
e.getCause() instanceof WriteTimeoutException);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Heal partition
|
||||||
|
cluster.filters().reset();
|
||||||
|
|
||||||
|
// Wait for gossip to converge
|
||||||
|
ClusterUtils.awaitGossipSchemaMatch(cluster);
|
||||||
|
|
||||||
|
// Verify read repair
|
||||||
|
Object[][] rows = cluster.coordinator(3).execute(
|
||||||
|
"SELECT * FROM test.data WHERE k = 1",
|
||||||
|
ConsistencyLevel.ALL
|
||||||
|
);
|
||||||
|
|
||||||
|
Assert.assertEquals(1, rows.length);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Testing Read Repair
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testReadRepair() throws IOException {
|
||||||
|
try (Cluster cluster = Cluster.build(3).start()) {
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 3}");
|
||||||
|
cluster.schemaChange("CREATE TABLE test.data (k int PRIMARY KEY, v int) " +
|
||||||
|
"WITH read_repair = 'BLOCKING'");
|
||||||
|
|
||||||
|
// Write to only 2 replicas
|
||||||
|
cluster.filters().inbound()
|
||||||
|
.to(3)
|
||||||
|
.verbs(Verb.MUTATION_REQ.id)
|
||||||
|
.drop()
|
||||||
|
.on();
|
||||||
|
|
||||||
|
cluster.coordinator(1).execute(
|
||||||
|
"INSERT INTO test.data (k, v) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.QUORUM, 1, 10);
|
||||||
|
|
||||||
|
cluster.filters().reset();
|
||||||
|
|
||||||
|
// Read at ALL should trigger read repair
|
||||||
|
Object[][] rows = cluster.coordinator(1).execute(
|
||||||
|
"SELECT * FROM test.data WHERE k = 1",
|
||||||
|
ConsistencyLevel.ALL
|
||||||
|
);
|
||||||
|
|
||||||
|
Assert.assertEquals(1, rows.length);
|
||||||
|
|
||||||
|
// Verify node 3 now has data
|
||||||
|
SimpleQueryResult result = cluster.get(3).executeInternalWithResult(
|
||||||
|
"SELECT v FROM test.data WHERE k = 1"
|
||||||
|
);
|
||||||
|
|
||||||
|
Assert.assertTrue(result.hasNext());
|
||||||
|
Assert.assertEquals(10, result.next().getInteger("v"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Testing Hinted Handoff
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testHintedHandoff() throws Exception {
|
||||||
|
try (Cluster cluster = Cluster.build(3)
|
||||||
|
.withConfig(c -> c
|
||||||
|
.set("max_hint_window", "10000")
|
||||||
|
.set("hinted_handoff_enabled", "true"))
|
||||||
|
.start()) {
|
||||||
|
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 3}");
|
||||||
|
cluster.schemaChange("CREATE TABLE test.data (k int PRIMARY KEY, v int)");
|
||||||
|
|
||||||
|
// Stop node 3
|
||||||
|
ClusterUtils.stopUnchecked(cluster.get(3));
|
||||||
|
|
||||||
|
// Write (will create hints for node 3)
|
||||||
|
cluster.coordinator(1).execute(
|
||||||
|
"INSERT INTO test.data (k, v) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.ONE, 1, 10);
|
||||||
|
|
||||||
|
// Verify hint created
|
||||||
|
int hintCount = cluster.get(1).callOnInstance(() -> {
|
||||||
|
HintsCatalog catalog = HintsService.instance.getCatalog();
|
||||||
|
return catalog.allHints().size();
|
||||||
|
});
|
||||||
|
|
||||||
|
Assert.assertTrue("Expected hints to be created", hintCount > 0);
|
||||||
|
|
||||||
|
// Restart node 3
|
||||||
|
cluster.get(3).startup();
|
||||||
|
ClusterUtils.awaitRingJoin(cluster.get(1), cluster.get(3));
|
||||||
|
|
||||||
|
// Wait for hints to be delivered
|
||||||
|
Thread.sleep(5000);
|
||||||
|
|
||||||
|
// Verify data on node 3
|
||||||
|
SimpleQueryResult result = cluster.get(3).executeInternalWithResult(
|
||||||
|
"SELECT v FROM test.data WHERE k = 1"
|
||||||
|
);
|
||||||
|
|
||||||
|
Assert.assertTrue(result.hasNext());
|
||||||
|
Assert.assertEquals(10, result.next().getInteger("v"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing Repair
|
||||||
|
|
||||||
|
### Full Repair Test
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testFullRepair() throws Exception {
|
||||||
|
try (Cluster cluster = Cluster.build(3).start()) {
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 3}");
|
||||||
|
cluster.schemaChange("CREATE TABLE test.data (k int PRIMARY KEY, v int)");
|
||||||
|
|
||||||
|
// Create inconsistency
|
||||||
|
cluster.filters().inbound().to(3).verbs(Verb.MUTATION_REQ.id).drop().on();
|
||||||
|
|
||||||
|
for (int i = 0; i < 100; i++) {
|
||||||
|
cluster.coordinator(1).execute(
|
||||||
|
"INSERT INTO test.data (k, v) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.QUORUM, i, i * 10);
|
||||||
|
}
|
||||||
|
|
||||||
|
cluster.filters().reset();
|
||||||
|
|
||||||
|
// Verify node 3 is missing data
|
||||||
|
int countBefore = cluster.get(3).callOnInstance(() -> {
|
||||||
|
ColumnFamilyStore cfs = Keyspace.open("test").getColumnFamilyStore("data");
|
||||||
|
return cfs.getLiveSSTables().size();
|
||||||
|
});
|
||||||
|
|
||||||
|
Assert.assertEquals(0, countBefore);
|
||||||
|
|
||||||
|
// Run repair
|
||||||
|
NodeToolResult result = cluster.get(1).nodetoolResult("repair", "test", "data");
|
||||||
|
result.asserts().success();
|
||||||
|
|
||||||
|
// Wait for repair to complete
|
||||||
|
Thread.sleep(10000);
|
||||||
|
|
||||||
|
// Verify node 3 now has data
|
||||||
|
int countAfter = cluster.get(3).callOnInstance(() -> {
|
||||||
|
ColumnFamilyStore cfs = Keyspace.open("test").getColumnFamilyStore("data");
|
||||||
|
return cfs.getLiveSSTables().size();
|
||||||
|
});
|
||||||
|
|
||||||
|
Assert.assertTrue("Expected data after repair", countAfter > 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Multi-Datacenter Patterns
|
||||||
|
|
||||||
|
### Testing LOCAL_QUORUM
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testLocalQuorum() throws IOException {
|
||||||
|
try (Cluster cluster = builder()
|
||||||
|
.withRacks(2, 1, 3) // 2 DCs, 1 rack each, 3 nodes
|
||||||
|
.start()) {
|
||||||
|
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'NetworkTopologyStrategy', " +
|
||||||
|
"'datacenter1': 3, 'datacenter2': 3}");
|
||||||
|
cluster.schemaChange("CREATE TABLE test.data (k int PRIMARY KEY, v int)");
|
||||||
|
|
||||||
|
// Partition DC2 from DC1
|
||||||
|
for (int dc1Node = 1; dc1Node <= 3; dc1Node++) {
|
||||||
|
for (int dc2Node = 4; dc2Node <= 6; dc2Node++) {
|
||||||
|
cluster.filters().allVerbs()
|
||||||
|
.from(dc1Node).to(dc2Node).drop().on();
|
||||||
|
cluster.filters().allVerbs()
|
||||||
|
.from(dc2Node).to(dc1Node).drop().on();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// LOCAL_QUORUM write in DC1 should succeed
|
||||||
|
cluster.coordinator(1).execute(
|
||||||
|
"INSERT INTO test.data (k, v) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.LOCAL_QUORUM, 1, 10);
|
||||||
|
|
||||||
|
// LOCAL_QUORUM read in DC1 should succeed
|
||||||
|
Object[][] rows = cluster.coordinator(2).execute(
|
||||||
|
"SELECT * FROM test.data WHERE k = 1",
|
||||||
|
ConsistencyLevel.LOCAL_QUORUM
|
||||||
|
);
|
||||||
|
|
||||||
|
Assert.assertEquals(1, rows.length);
|
||||||
|
|
||||||
|
// EACH_QUORUM should fail (can't reach DC2)
|
||||||
|
try {
|
||||||
|
cluster.coordinator(1).execute(
|
||||||
|
"INSERT INTO test.data (k, v) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.EACH_QUORUM, 2, 20);
|
||||||
|
Assert.fail("Expected unavailable");
|
||||||
|
} catch (Exception e) {
|
||||||
|
Assert.assertTrue(e.getCause() instanceof UnavailableException);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance Testing Patterns
|
||||||
|
|
||||||
|
### Testing Under Load
|
||||||
|
|
||||||
|
```java
|
||||||
|
@Test
|
||||||
|
public void testUnderLoad() throws Exception {
|
||||||
|
try (Cluster cluster = Cluster.build(3)
|
||||||
|
.withConfig(c -> c
|
||||||
|
.set("concurrent_reads", "128")
|
||||||
|
.set("concurrent_writes", "128"))
|
||||||
|
.start()) {
|
||||||
|
|
||||||
|
cluster.schemaChange("CREATE KEYSPACE test WITH replication = " +
|
||||||
|
"{'class': 'SimpleStrategy', 'replication_factor': 2}");
|
||||||
|
cluster.schemaChange("CREATE TABLE test.data (k int PRIMARY KEY, v int)");
|
||||||
|
|
||||||
|
int numThreads = 10;
|
||||||
|
int numOpsPerThread = 1000;
|
||||||
|
ExecutorService executor = Executors.newFixedThreadPool(numThreads);
|
||||||
|
AtomicInteger successCount = new AtomicInteger(0);
|
||||||
|
AtomicInteger failureCount = new AtomicInteger(0);
|
||||||
|
|
||||||
|
List<Future<?>> futures = new ArrayList<>();
|
||||||
|
for (int t = 0; t < numThreads; t++) {
|
||||||
|
final int threadId = t;
|
||||||
|
futures.add(executor.submit(() -> {
|
||||||
|
for (int i = 0; i < numOpsPerThread; i++) {
|
||||||
|
try {
|
||||||
|
int key = threadId * numOpsPerThread + i;
|
||||||
|
cluster.coordinator((key % 3) + 1).execute(
|
||||||
|
"INSERT INTO test.data (k, v) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.QUORUM, key, key * 10);
|
||||||
|
successCount.incrementAndGet();
|
||||||
|
} catch (Exception e) {
|
||||||
|
failureCount.incrementAndGet();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wait for all threads
|
||||||
|
for (Future<?> future : futures) {
|
||||||
|
future.get();
|
||||||
|
}
|
||||||
|
|
||||||
|
executor.shutdown();
|
||||||
|
|
||||||
|
System.out.println("Successes: " + successCount.get());
|
||||||
|
System.out.println("Failures: " + failureCount.get());
|
||||||
|
|
||||||
|
Assert.assertTrue("Too many failures", failureCount.get() < successCount.get() * 0.01);
|
||||||
|
|
||||||
|
// Verify data
|
||||||
|
Object[][] rows = cluster.coordinator(1).execute(
|
||||||
|
"SELECT COUNT(*) FROM test.data",
|
||||||
|
ConsistencyLevel.QUORUM
|
||||||
|
);
|
||||||
|
|
||||||
|
long count = (long) rows[0][0];
|
||||||
|
Assert.assertTrue("Expected data written", count > 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,641 @@
|
||||||
|
# Classloader Deep Dive
|
||||||
|
|
||||||
|
Comprehensive guide to understanding and debugging classloader issues in the Cassandra in-JVM dtest framework.
|
||||||
|
|
||||||
|
## Classloader Architecture
|
||||||
|
|
||||||
|
### The Three ClassLoader Layers
|
||||||
|
|
||||||
|
The in-JVM dtest framework uses a three-tier classloader hierarchy:
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────┐
|
||||||
|
│ Test ClassLoader (Your Tests) │
|
||||||
|
│ - JUnit test code │
|
||||||
|
│ - Test utilities │
|
||||||
|
│ - Direct Cluster/Instance access │
|
||||||
|
└─────────────────────────────────────────┘
|
||||||
|
↓
|
||||||
|
┌─────────────────────────────────────────┐
|
||||||
|
│ Shared ClassLoader │
|
||||||
|
│ - Primitives (String, Integer, etc.) │
|
||||||
|
│ - Config objects (@Shared) │
|
||||||
|
│ - API interfaces (ICoordinator, etc.) │
|
||||||
|
│ - Serializable contracts │
|
||||||
|
└─────────────────────────────────────────┘
|
||||||
|
↓
|
||||||
|
┌─────────────────────────────────────────┐
|
||||||
|
│ Instance ClassLoaders (Per Node) │
|
||||||
|
│ - DatabaseDescriptor │
|
||||||
|
│ - StorageService │
|
||||||
|
│ - All Cassandra internal classes │
|
||||||
|
│ - Logging configuration │
|
||||||
|
│ - Each instance gets own copy │
|
||||||
|
└─────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### What Gets Loaded Where
|
||||||
|
|
||||||
|
**Shared ClassLoader** (available to all):
|
||||||
|
- Java primitives: String, Integer, Long, etc.
|
||||||
|
- Config objects: InstanceConfig, IInstanceConfig
|
||||||
|
- API interfaces: ICoordinator, IInstance, IInvokableInstance
|
||||||
|
- ByteBuffer and related classes
|
||||||
|
- Classes marked with `@Shared` annotation
|
||||||
|
- Serialization infrastructure
|
||||||
|
|
||||||
|
**Instance ClassLoader** (per-instance isolation):
|
||||||
|
- DatabaseDescriptor (each instance has own static state)
|
||||||
|
- StorageService (separate singleton per instance)
|
||||||
|
- All Cassandra core classes (schema, storage, etc.)
|
||||||
|
- Logging configuration (logback per instance)
|
||||||
|
- Classes marked with `@Isolated` annotation
|
||||||
|
- Everything else in org.apache.cassandra.*
|
||||||
|
|
||||||
|
### Why This Matters
|
||||||
|
|
||||||
|
Each instance needs its own isolated copy of Cassandra's static state:
|
||||||
|
- DatabaseDescriptor holds configuration → each node needs different config
|
||||||
|
- StorageService is a singleton → each node needs separate instance
|
||||||
|
- Schema metadata is static → each node may have different schema versions
|
||||||
|
|
||||||
|
## Code Transfer Mechanism
|
||||||
|
|
||||||
|
### How Lambdas Cross Boundaries
|
||||||
|
|
||||||
|
When you call `runOnInstance(() -> { ... })`:
|
||||||
|
|
||||||
|
1. **Serialization** (Test ClassLoader):
|
||||||
|
```
|
||||||
|
Lambda → Java Serialization → byte[]
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Transfer** (Across Boundary):
|
||||||
|
```
|
||||||
|
byte[] passed to IsolatedExecutor
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Deserialization** (Instance ClassLoader):
|
||||||
|
```
|
||||||
|
byte[] → Java Deserialization → Lambda (in instance classloader)
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Execution** (Instance Context):
|
||||||
|
```
|
||||||
|
Lambda runs with access to instance's Cassandra classes
|
||||||
|
```
|
||||||
|
|
||||||
|
### What Can Be Captured
|
||||||
|
|
||||||
|
**Safe to Capture** (automatically serialized):
|
||||||
|
```java
|
||||||
|
// Primitives
|
||||||
|
final int value = 42;
|
||||||
|
final String name = "test";
|
||||||
|
final long timestamp = System.currentTimeMillis();
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
System.out.println(value); // OK
|
||||||
|
System.out.println(name); // OK
|
||||||
|
System.out.println(timestamp); // OK
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Unsafe to Capture** (serialization will fail):
|
||||||
|
```java
|
||||||
|
// Non-serializable objects from test
|
||||||
|
FileOutputStream fos = new FileOutputStream("test.txt");
|
||||||
|
AtomicInteger counter = new AtomicInteger(0);
|
||||||
|
DatabaseDescriptor descriptor = ...; // From test classloader
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
fos.write(...); // FAIL: NotSerializableException
|
||||||
|
counter.incrementAndGet(); // FAIL: NotSerializableException
|
||||||
|
descriptor.get...(); // FAIL: Even if serializable, wrong classloader
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common Issues and Solutions
|
||||||
|
|
||||||
|
### Issue 1: ClassCastException
|
||||||
|
|
||||||
|
**Symptom:**
|
||||||
|
```
|
||||||
|
java.lang.ClassCastException: org.apache.cassandra.dht.Murmur3Partitioner$LongToken cannot be cast to org.apache.cassandra.dht.Token
|
||||||
|
```
|
||||||
|
|
||||||
|
**Cause:**
|
||||||
|
Object created in test classloader, trying to use in instance classloader. Even though the classes have the same name, they're loaded by different classloaders and are considered different types.
|
||||||
|
|
||||||
|
**Wrong:**
|
||||||
|
```java
|
||||||
|
// Create token in test classloader
|
||||||
|
Token token = new Murmur3Partitioner.LongToken(0);
|
||||||
|
|
||||||
|
// Try to use in instance
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
DatabaseDescriptor.setPartitionerUnsafe(token); // FAIL: ClassCastException
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Right:**
|
||||||
|
```java
|
||||||
|
// Create token inside instance classloader
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
Token token = new Murmur3Partitioner.LongToken(0);
|
||||||
|
DatabaseDescriptor.setPartitionerUnsafe(token); // OK
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why This Happens:**
|
||||||
|
```
|
||||||
|
Test CL: Token.class → Object ID 0x1234
|
||||||
|
Instance CL: Token.class → Object ID 0x5678
|
||||||
|
|
||||||
|
0x1234 cannot be cast to 0x5678 even if identical bytecode
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue 2: NotSerializableException
|
||||||
|
|
||||||
|
**Symptom:**
|
||||||
|
```
|
||||||
|
java.io.NotSerializableException: MyCustomClass
|
||||||
|
```
|
||||||
|
|
||||||
|
**Cause:**
|
||||||
|
Lambda captured non-serializable object from test scope.
|
||||||
|
|
||||||
|
**Wrong:**
|
||||||
|
```java
|
||||||
|
List<String> names = new ArrayList<>(); // ArrayList is Serializable
|
||||||
|
Thread thread = new Thread(() -> {}); // Thread is NOT Serializable
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
names.add("test"); // OK - ArrayList is Serializable
|
||||||
|
thread.start(); // FAIL - Thread is not Serializable
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Right:**
|
||||||
|
```java
|
||||||
|
// Only capture serializable primitives
|
||||||
|
final String threadName = "test-thread";
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
Thread thread = new Thread(() -> {}, threadName); // Create inside
|
||||||
|
thread.start(); // OK
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue 3: Static State Not Shared
|
||||||
|
|
||||||
|
**Symptom:**
|
||||||
|
Static fields modified in one instance don't affect others.
|
||||||
|
|
||||||
|
**Cause:**
|
||||||
|
Each instance has separate copy of static state.
|
||||||
|
|
||||||
|
**Wrong Assumption:**
|
||||||
|
```java
|
||||||
|
// Set on instance 1
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
DatabaseDescriptor.setPartitioner(...);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Expect to see on instance 2
|
||||||
|
cluster.get(2).runOnInstance(() -> {
|
||||||
|
// DatabaseDescriptor is DIFFERENT instance here
|
||||||
|
Partitioner p = DatabaseDescriptor.getPartitioner(); // Won't see change from node 1
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Right Approach:**
|
||||||
|
```java
|
||||||
|
// Configure each instance separately
|
||||||
|
cluster.stream().forEach(instance -> {
|
||||||
|
instance.runOnInstance(() -> {
|
||||||
|
DatabaseDescriptor.setPartitioner(...);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue 4: ClassNotFoundException
|
||||||
|
|
||||||
|
**Symptom:**
|
||||||
|
```
|
||||||
|
java.lang.ClassNotFoundException: com.mycompany.TestUtils
|
||||||
|
```
|
||||||
|
|
||||||
|
**Cause:**
|
||||||
|
Class from test classpath not available in instance classloader.
|
||||||
|
|
||||||
|
**Wrong:**
|
||||||
|
```java
|
||||||
|
// TestUtils only in test classpath
|
||||||
|
import com.mycompany.TestUtils;
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
TestUtils.doSomething(); // FAIL: ClassNotFoundException
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Right:**
|
||||||
|
```java
|
||||||
|
// Either:
|
||||||
|
// 1. Move TestUtils to Cassandra dependencies
|
||||||
|
// 2. Mark TestUtils as @Shared
|
||||||
|
// 3. Use only Cassandra classes inside runOnInstance
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
// Use only Cassandra classes
|
||||||
|
StorageService.instance.method(); // OK
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue 5: Lambda Capture Confusion
|
||||||
|
|
||||||
|
**Symptom:**
|
||||||
|
Unexpected behavior or null values inside lambda.
|
||||||
|
|
||||||
|
**Cause:**
|
||||||
|
Captured variables must be effectively final.
|
||||||
|
|
||||||
|
**Wrong:**
|
||||||
|
```java
|
||||||
|
int counter = 0;
|
||||||
|
|
||||||
|
for (int i = 0; i < 3; i++) {
|
||||||
|
cluster.get(i + 1).runOnInstance(() -> {
|
||||||
|
System.out.println(counter); // FAIL: counter must be final
|
||||||
|
});
|
||||||
|
counter++;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Right:**
|
||||||
|
```java
|
||||||
|
for (int i = 0; i < 3; i++) {
|
||||||
|
final int nodeId = i + 1; // Effectively final
|
||||||
|
cluster.get(nodeId).runOnInstance(() -> {
|
||||||
|
System.out.println("Node: " + nodeId); // OK
|
||||||
|
});
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Debugging Techniques
|
||||||
|
|
||||||
|
### Technique 1: Identify ClassLoader
|
||||||
|
|
||||||
|
Print which classloader loaded a class:
|
||||||
|
|
||||||
|
```java
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
System.out.println("Current thread CL: " +
|
||||||
|
Thread.currentThread().getContextClassLoader());
|
||||||
|
|
||||||
|
System.out.println("Token CL: " +
|
||||||
|
Token.class.getClassLoader());
|
||||||
|
|
||||||
|
System.out.println("DatabaseDescriptor CL: " +
|
||||||
|
DatabaseDescriptor.class.getClassLoader());
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected output:
|
||||||
|
```
|
||||||
|
Current thread CL: IsolatedClassLoader@abc123
|
||||||
|
Token CL: IsolatedClassLoader@abc123
|
||||||
|
DatabaseDescriptor CL: IsolatedClassLoader@abc123
|
||||||
|
```
|
||||||
|
|
||||||
|
### Technique 2: Verify Serialization
|
||||||
|
|
||||||
|
Test if object can be serialized before using in lambda:
|
||||||
|
|
||||||
|
```java
|
||||||
|
public static boolean isSerializable(Object obj) {
|
||||||
|
try {
|
||||||
|
ByteArrayOutputStream baos = new ByteArrayOutputStream();
|
||||||
|
ObjectOutputStream oos = new ObjectOutputStream(baos);
|
||||||
|
oos.writeObject(obj);
|
||||||
|
oos.close();
|
||||||
|
return true;
|
||||||
|
} catch (NotSerializableException e) {
|
||||||
|
System.out.println("NOT serializable: " + e.getMessage());
|
||||||
|
return false;
|
||||||
|
} catch (IOException e) {
|
||||||
|
throw new RuntimeException(e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Usage
|
||||||
|
MyObject obj = new MyObject();
|
||||||
|
if (isSerializable(obj)) {
|
||||||
|
// Safe to capture in lambda
|
||||||
|
final MyObject capturedObj = obj;
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
// Use capturedObj
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
// Must create inside runOnInstance
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Technique 3: Explicit Transfer
|
||||||
|
|
||||||
|
Use IsolatedExecutor.transfer() to explicitly move objects:
|
||||||
|
|
||||||
|
```java
|
||||||
|
import org.apache.cassandra.distributed.impl.IsolatedExecutor;
|
||||||
|
|
||||||
|
// Create in test classloader
|
||||||
|
MySerializableObject obj = new MySerializableObject();
|
||||||
|
|
||||||
|
// Explicitly transfer to instance classloader
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
MySerializableObject transferred = (MySerializableObject)
|
||||||
|
IsolatedExecutor.transferAdhoc(obj, Thread.currentThread().getContextClassLoader());
|
||||||
|
|
||||||
|
// Now 'transferred' is in correct classloader
|
||||||
|
useObject(transferred);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Technique 4: Inspect Lambda Captures
|
||||||
|
|
||||||
|
Use reflection to see what a lambda captured:
|
||||||
|
|
||||||
|
```java
|
||||||
|
import java.lang.reflect.Field;
|
||||||
|
|
||||||
|
Consumer<Object> lambda = (obj) -> {
|
||||||
|
System.out.println(obj);
|
||||||
|
};
|
||||||
|
|
||||||
|
// Inspect captured fields
|
||||||
|
for (Field field : lambda.getClass().getDeclaredFields()) {
|
||||||
|
field.setAccessible(true);
|
||||||
|
System.out.println("Captured: " + field.getName() +
|
||||||
|
" = " + field.get(lambda));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Technique 5: Compare Class Identity
|
||||||
|
|
||||||
|
Check if two class references are actually the same:
|
||||||
|
|
||||||
|
```java
|
||||||
|
Class<?> testToken = Token.class; // From test classloader
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
Class<?> instanceToken = Token.class; // From instance classloader
|
||||||
|
|
||||||
|
// These will be DIFFERENT
|
||||||
|
System.out.println("Test CL Token: " + System.identityHashCode(testToken));
|
||||||
|
System.out.println("Instance CL Token: " + System.identityHashCode(instanceToken));
|
||||||
|
System.out.println("Same class? " + (testToken == instanceToken)); // false!
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## Annotations: @Shared and @Isolated
|
||||||
|
|
||||||
|
### @Shared Annotation
|
||||||
|
|
||||||
|
Mark classes to be loaded in shared classloader:
|
||||||
|
|
||||||
|
```java
|
||||||
|
import org.apache.cassandra.utils.Shared;
|
||||||
|
|
||||||
|
@Shared
|
||||||
|
public class MySharedData implements Serializable {
|
||||||
|
private String value;
|
||||||
|
|
||||||
|
// Can be used across classloader boundaries
|
||||||
|
public MySharedData(String value) {
|
||||||
|
this.value = value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Use @Shared for:**
|
||||||
|
- Config/DTO classes
|
||||||
|
- Data transfer objects
|
||||||
|
- Immutable value classes
|
||||||
|
- Things that need to cross boundaries
|
||||||
|
|
||||||
|
### @Isolated Annotation
|
||||||
|
|
||||||
|
Mark classes to be loaded per-instance:
|
||||||
|
|
||||||
|
```java
|
||||||
|
import org.apache.cassandra.utils.Isolated;
|
||||||
|
|
||||||
|
@Isolated
|
||||||
|
public class MyInstanceState {
|
||||||
|
private static MyInstanceState instance;
|
||||||
|
|
||||||
|
// Each Cassandra instance gets own copy
|
||||||
|
public static MyInstanceState getInstance() {
|
||||||
|
if (instance == null)
|
||||||
|
instance = new MyInstanceState();
|
||||||
|
return instance;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Use @Isolated for:**
|
||||||
|
- Stateful singletons
|
||||||
|
- Classes with static state
|
||||||
|
- Instance-specific configuration
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
### 1. Minimize Lambda Captures
|
||||||
|
|
||||||
|
**Bad:**
|
||||||
|
```java
|
||||||
|
List<String> results = new ArrayList<>();
|
||||||
|
Map<String, Integer> counts = new HashMap<>();
|
||||||
|
AtomicInteger total = new AtomicInteger();
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
// Captures 3 objects
|
||||||
|
results.add("test");
|
||||||
|
counts.put("key", 1);
|
||||||
|
total.incrementAndGet();
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Good:**
|
||||||
|
```java
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
// Create everything inside
|
||||||
|
List<String> results = new ArrayList<>();
|
||||||
|
results.add("test");
|
||||||
|
// Process locally
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Use Return Values
|
||||||
|
|
||||||
|
**Bad:**
|
||||||
|
```java
|
||||||
|
AtomicInteger result = new AtomicInteger();
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
int count = countSomething();
|
||||||
|
result.set(count); // Won't work - different classloader
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Good:**
|
||||||
|
```java
|
||||||
|
int result = cluster.get(1).callOnInstance(() -> {
|
||||||
|
return countSomething(); // Return primitive
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Create Objects in Target ClassLoader
|
||||||
|
|
||||||
|
**Bad:**
|
||||||
|
```java
|
||||||
|
Token token = new Murmur3Partitioner.LongToken(0);
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
useToken(token); // ClassCastException
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Good:**
|
||||||
|
```java
|
||||||
|
final long tokenValue = 0;
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
Token token = new Murmur3Partitioner.LongToken(tokenValue);
|
||||||
|
useToken(token); // OK
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Use Primitives for Communication
|
||||||
|
|
||||||
|
**Bad:**
|
||||||
|
```java
|
||||||
|
MyComplexObject obj = new MyComplexObject(...);
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
process(obj); // May fail serialization
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Good:**
|
||||||
|
```java
|
||||||
|
final int id = obj.getId();
|
||||||
|
final String name = obj.getName();
|
||||||
|
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
MyComplexObject obj = new MyComplexObject(id, name);
|
||||||
|
process(obj); // OK - created in instance classloader
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Coordinate via CQL
|
||||||
|
|
||||||
|
**Bad:**
|
||||||
|
```java
|
||||||
|
// Try to share objects between instances
|
||||||
|
Object sharedState = cluster.get(1).callOnInstance(() -> {
|
||||||
|
return getState();
|
||||||
|
});
|
||||||
|
|
||||||
|
cluster.get(2).runOnInstance(() -> {
|
||||||
|
setState(sharedState); // ClassCastException
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Good:**
|
||||||
|
```java
|
||||||
|
// Share via CQL
|
||||||
|
cluster.coordinator(1).execute(
|
||||||
|
"INSERT INTO shared.state (id, value) VALUES (?, ?)",
|
||||||
|
ConsistencyLevel.QUORUM, 1, "state-value");
|
||||||
|
|
||||||
|
// Read on other instance
|
||||||
|
Object[][] rows = cluster.coordinator(2).execute(
|
||||||
|
"SELECT value FROM shared.state WHERE id = ?",
|
||||||
|
ConsistencyLevel.QUORUM, 1);
|
||||||
|
```
|
||||||
|
|
||||||
|
## Advanced: Understanding IsolatedExecutor
|
||||||
|
|
||||||
|
### Serialization Process
|
||||||
|
|
||||||
|
```java
|
||||||
|
// In test classloader
|
||||||
|
SerializableCallable<String> lambda = () -> "Hello";
|
||||||
|
|
||||||
|
// IsolatedExecutor.transfer():
|
||||||
|
byte[] serialized = serializeOneObject(lambda); // Test CL → bytes
|
||||||
|
|
||||||
|
// Transfer to instance classloader
|
||||||
|
Object deserialized = deserializeOneObject(serialized); // bytes → Instance CL
|
||||||
|
|
||||||
|
// Now deserialized is in instance classloader
|
||||||
|
```
|
||||||
|
|
||||||
|
### Custom Transfer
|
||||||
|
|
||||||
|
For complex scenarios:
|
||||||
|
|
||||||
|
```java
|
||||||
|
// Serialize in test classloader
|
||||||
|
ByteArrayOutputStream baos = new ByteArrayOutputStream();
|
||||||
|
ObjectOutputStream oos = new ObjectOutputStream(baos);
|
||||||
|
oos.writeObject(myObject);
|
||||||
|
byte[] bytes = baos.toByteArray();
|
||||||
|
|
||||||
|
// Transfer and deserialize in instance classloader
|
||||||
|
cluster.get(1).runOnInstance(() -> {
|
||||||
|
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
|
||||||
|
ObjectInputStream ois = new ObjectInputStream(bais) {
|
||||||
|
@Override
|
||||||
|
protected Class<?> resolveClass(ObjectStreamClass desc)
|
||||||
|
throws IOException, ClassNotFoundException {
|
||||||
|
// Force resolution in instance classloader
|
||||||
|
return Class.forName(desc.getName(), false,
|
||||||
|
Thread.currentThread().getContextClassLoader());
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
Object obj = ois.readObject();
|
||||||
|
// obj is now in instance classloader
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting Checklist
|
||||||
|
|
||||||
|
When encountering classloader issues:
|
||||||
|
|
||||||
|
1. **Check what you're capturing**
|
||||||
|
- Is it a primitive? → OK
|
||||||
|
- Is it Serializable? → Maybe OK
|
||||||
|
- Is it a Cassandra class? → Create inside runOnInstance
|
||||||
|
|
||||||
|
2. **Verify serialization**
|
||||||
|
- Use isSerializable() helper
|
||||||
|
- Check for non-serializable fields
|
||||||
|
|
||||||
|
3. **Identify the classloader**
|
||||||
|
- Print classloader info
|
||||||
|
- Compare class identity
|
||||||
|
|
||||||
|
4. **Simplify the lambda**
|
||||||
|
- Remove captures one by one
|
||||||
|
- Create objects inside
|
||||||
|
|
||||||
|
5. **Use primitives**
|
||||||
|
- Extract primitive values
|
||||||
|
- Pass only primitives
|
||||||
|
|
||||||
|
6. **Return, don't modify**
|
||||||
|
- Return results from callOnInstance
|
||||||
|
- Don't try to modify captured objects
|
||||||
|
|
@ -0,0 +1,672 @@
|
||||||
|
# ClusterUtils Comprehensive Reference
|
||||||
|
|
||||||
|
Complete reference for `org.apache.cassandra.distributed.shared.ClusterUtils`.
|
||||||
|
|
||||||
|
## Lifecycle Management
|
||||||
|
|
||||||
|
### start()
|
||||||
|
```java
|
||||||
|
<I extends IInstance> I start(I inst, Consumer<WithProperties> fn)
|
||||||
|
<I extends IInstance> I start(I inst, BiConsumer<I, WithProperties> fn)
|
||||||
|
```
|
||||||
|
Start instance with system properties that are cleared after startup.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```java
|
||||||
|
ClusterUtils.start(instance, properties -> {
|
||||||
|
properties.set(RING_DELAY, "5000");
|
||||||
|
properties.set(BROADCAST_INTERVAL_MS, "30000");
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### stopUnchecked()
|
||||||
|
```java
|
||||||
|
void stopUnchecked(IInstance i)
|
||||||
|
```
|
||||||
|
Stop instance in blocking manner, throwing runtime exceptions on failure.
|
||||||
|
|
||||||
|
### stopAbrupt()
|
||||||
|
```java
|
||||||
|
<I extends IInstance> void stopAbrupt(ICluster<I> cluster, I inst)
|
||||||
|
```
|
||||||
|
Simulate kill -9 by blocking all messages to/from node before shutdown.
|
||||||
|
|
||||||
|
### stopAll()
|
||||||
|
```java
|
||||||
|
<I extends IInstance> void stopAll(ICluster<I> cluster)
|
||||||
|
```
|
||||||
|
Stop all instances without cleaning cluster state.
|
||||||
|
|
||||||
|
### restartUnchecked()
|
||||||
|
```java
|
||||||
|
void restartUnchecked(IInstance instance)
|
||||||
|
```
|
||||||
|
Stop and restart instance in blocking manner.
|
||||||
|
|
||||||
|
## Instance Management
|
||||||
|
|
||||||
|
### addInstance()
|
||||||
|
```java
|
||||||
|
<I extends IInstance> I addInstance(AbstractCluster<I> cluster, Consumer<IInstanceConfig> fn)
|
||||||
|
<I extends IInstance> I addInstance(AbstractCluster<I> cluster)
|
||||||
|
<I extends IInstance> I addInstance(AbstractCluster<I> cluster, IInstanceConfig other, Consumer<IInstanceConfig> fn)
|
||||||
|
<I extends IInstance> I addInstance(AbstractCluster<I> cluster, String dc, String rack)
|
||||||
|
<I extends IInstance> I addInstance(AbstractCluster<I> cluster, String dc, String rack, Consumer<IInstanceConfig> fn)
|
||||||
|
```
|
||||||
|
Create and add new instance to cluster without starting it.
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
```java
|
||||||
|
// Add with same config as existing nodes
|
||||||
|
IInstance inst = ClusterUtils.addInstance(cluster);
|
||||||
|
|
||||||
|
// Add with custom config
|
||||||
|
IInstance inst = ClusterUtils.addInstance(cluster, config -> {
|
||||||
|
config.set("auto_bootstrap", true);
|
||||||
|
config.set("num_tokens", "256");
|
||||||
|
});
|
||||||
|
|
||||||
|
// Add to specific DC/rack
|
||||||
|
IInstance inst = ClusterUtils.addInstance(cluster, "dc1", "rack1", config -> {
|
||||||
|
config.set("concurrent_reads", "64");
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### replaceHostAndStart()
|
||||||
|
```java
|
||||||
|
<I extends IInstance> I replaceHostAndStart(AbstractCluster<I> cluster, I toReplace)
|
||||||
|
<I extends IInstance> I replaceHostAndStart(AbstractCluster<I> cluster, I toReplace, Consumer<WithProperties> fn)
|
||||||
|
<I extends IInstance> I replaceHostAndStart(AbstractCluster<I> cluster, I toReplace, BiConsumer<I, WithProperties> fn)
|
||||||
|
<I extends IInstance> I replaceHostAndStart(AbstractCluster<I> cluster, I toReplace, BiConsumer<I, WithProperties> fn, Consumer<IInstanceConfig> configFn)
|
||||||
|
```
|
||||||
|
Create and start new instance that replaces an existing instance.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```java
|
||||||
|
IInvokableInstance replacement = ClusterUtils.replaceHostAndStart(
|
||||||
|
cluster,
|
||||||
|
failedNode,
|
||||||
|
(inst, properties) -> {
|
||||||
|
properties.set(RING_DELAY, "5000");
|
||||||
|
properties.set(BROADCAST_INTERVAL_MS, "30000");
|
||||||
|
},
|
||||||
|
config -> {
|
||||||
|
config.set("concurrent_compactors", "8");
|
||||||
|
}
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
### startHostReplacement()
|
||||||
|
```java
|
||||||
|
<I extends IInstance> I startHostReplacement(I toReplace, I inst)
|
||||||
|
<I extends IInstance> I startHostReplacement(I toReplace, I inst, BiConsumer<I, WithProperties> fn)
|
||||||
|
```
|
||||||
|
Start instance with properties needed for host replacement.
|
||||||
|
|
||||||
|
## Token & Metadata
|
||||||
|
|
||||||
|
### getTokenMetadataTokens()
|
||||||
|
```java
|
||||||
|
List<String> getTokenMetadataTokens(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Get all tokens from TokenMap as list of strings.
|
||||||
|
|
||||||
|
### getLocalTokens()
|
||||||
|
```java
|
||||||
|
Collection<String> getLocalTokens(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Get tokens assigned to this instance.
|
||||||
|
|
||||||
|
### getTokens()
|
||||||
|
```java
|
||||||
|
List<String> getTokens(IInstance instance)
|
||||||
|
```
|
||||||
|
Get tokens from config (only works if configured, not learned/generated).
|
||||||
|
|
||||||
|
### getTokenCount()
|
||||||
|
```java
|
||||||
|
int getTokenCount(IInvokableInstance instance)
|
||||||
|
```
|
||||||
|
Get num_tokens from config.
|
||||||
|
|
||||||
|
### getPartitionerName()
|
||||||
|
```java
|
||||||
|
String getPartitionerName(IInstance instance)
|
||||||
|
```
|
||||||
|
Get configured partitioner name.
|
||||||
|
|
||||||
|
### getPrimaryRanges()
|
||||||
|
```java
|
||||||
|
List<Range> getPrimaryRanges(IInvokableInstance instance, String keyspace)
|
||||||
|
```
|
||||||
|
Get primary token ranges for instance for given keyspace.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```java
|
||||||
|
List<ClusterUtils.Range> ranges = ClusterUtils.getPrimaryRanges(
|
||||||
|
cluster.get(1),
|
||||||
|
"my_keyspace"
|
||||||
|
);
|
||||||
|
for (ClusterUtils.Range range : ranges) {
|
||||||
|
System.out.println(range.left() + " to " + range.right());
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cluster Metadata & CMS
|
||||||
|
|
||||||
|
### getClusterMetadataVersion()
|
||||||
|
```java
|
||||||
|
Epoch getClusterMetadataVersion(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Get current cluster metadata epoch.
|
||||||
|
|
||||||
|
### getCurrentEpoch()
|
||||||
|
```java
|
||||||
|
Epoch getCurrentEpoch(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Get current epoch from ClusterMetadata.
|
||||||
|
|
||||||
|
### getNextEpoch()
|
||||||
|
```java
|
||||||
|
Epoch getNextEpoch(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Get next epoch from ClusterMetadata.
|
||||||
|
|
||||||
|
### maxEpoch()
|
||||||
|
```java
|
||||||
|
Epoch maxEpoch(ICluster<IInvokableInstance> cluster)
|
||||||
|
Epoch maxEpoch(ICluster<IInvokableInstance> cluster, int... nodes)
|
||||||
|
```
|
||||||
|
Find largest epoch across cluster or specified nodes.
|
||||||
|
|
||||||
|
### waitForCMSToQuiesce()
|
||||||
|
```java
|
||||||
|
void waitForCMSToQuiesce(ICluster<IInvokableInstance> cluster, int... cmsNodes)
|
||||||
|
void waitForCMSToQuiesce(ICluster<IInvokableInstance> cluster, Epoch awaitedEpoch, int... ignored)
|
||||||
|
void waitForCMSToQuiesce(ICluster<IInvokableInstance> cluster, Epoch awaitedEpoch, boolean fetchLogWhenBehind, int... ignored)
|
||||||
|
```
|
||||||
|
Wait for all nodes to reach same cluster metadata state.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```java
|
||||||
|
// Wait for all nodes to converge
|
||||||
|
ClusterUtils.waitForCMSToQuiesce(cluster);
|
||||||
|
|
||||||
|
// Wait for specific epoch
|
||||||
|
Epoch targetEpoch = ClusterUtils.maxEpoch(cluster);
|
||||||
|
ClusterUtils.waitForCMSToQuiesce(cluster, targetEpoch);
|
||||||
|
```
|
||||||
|
|
||||||
|
### fetchLogFromCMS()
|
||||||
|
```java
|
||||||
|
Epoch fetchLogFromCMS(IInvokableInstance inst, Epoch awaitedEpoch)
|
||||||
|
Epoch fetchLogFromCMS(IInvokableInstance inst, long awaitedEpoch)
|
||||||
|
```
|
||||||
|
Fetch cluster metadata log from CMS up to specified epoch.
|
||||||
|
|
||||||
|
### snapshotClusterMetadata()
|
||||||
|
```java
|
||||||
|
Epoch snapshotClusterMetadata(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Trigger cluster metadata snapshot and return resulting epoch.
|
||||||
|
|
||||||
|
### getPeerEpochs()
|
||||||
|
```java
|
||||||
|
Map<String, Epoch> getPeerEpochs(IInvokableInstance requester)
|
||||||
|
```
|
||||||
|
Request current epochs from all peers in cluster.
|
||||||
|
|
||||||
|
### getCMSMembers()
|
||||||
|
```java
|
||||||
|
Set<String> getCMSMembers(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Get set of CMS member addresses.
|
||||||
|
|
||||||
|
### getPeerDirectoryDebugStrings()
|
||||||
|
```java
|
||||||
|
List<String> getPeerDirectoryDebugStrings(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Get debug strings for peer directory (one line per node).
|
||||||
|
|
||||||
|
### getTokenMapDebugStrings()
|
||||||
|
```java
|
||||||
|
List<String> getTokenMapDebugStrings(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Get debug strings for token map.
|
||||||
|
|
||||||
|
### logTokenMapDebugString()
|
||||||
|
```java
|
||||||
|
void logTokenMapDebugString(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Log token map debug info on instance.
|
||||||
|
|
||||||
|
### getDataPlacementDebugInfo()
|
||||||
|
```java
|
||||||
|
Map<String, List[]> getDataPlacementDebugInfo(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Get placement debug info: for each keyspace, returns 2-element array with read and write replica lists.
|
||||||
|
|
||||||
|
### logDataPlacementDebugString()
|
||||||
|
```java
|
||||||
|
void logDataPlacementDebugString(IInvokableInstance inst, boolean byEndpoint)
|
||||||
|
```
|
||||||
|
Log data placement debug info.
|
||||||
|
|
||||||
|
## CMS Pausing & Control
|
||||||
|
|
||||||
|
### pauseBeforeCommit()
|
||||||
|
```java
|
||||||
|
Callable<Epoch> pauseBeforeCommit(IInvokableInstance cmsInstance, SerializablePredicate<Transformation> predicate)
|
||||||
|
```
|
||||||
|
Pause CMS before committing transformation matching predicate. Returns callable that waits for pause.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```java
|
||||||
|
Callable<Epoch> pauseHandle = ClusterUtils.pauseBeforeCommit(
|
||||||
|
cmsNode,
|
||||||
|
transformation -> transformation instanceof AddNode
|
||||||
|
);
|
||||||
|
|
||||||
|
// Wait for pause (blocks until predicate matches)
|
||||||
|
Epoch pausedAt = pauseHandle.call();
|
||||||
|
|
||||||
|
// Do something while paused
|
||||||
|
// ...
|
||||||
|
|
||||||
|
// Resume
|
||||||
|
ClusterUtils.unpauseCommits(cmsNode);
|
||||||
|
```
|
||||||
|
|
||||||
|
### getSequenceAfterCommit()
|
||||||
|
```java
|
||||||
|
Callable<Epoch> getSequenceAfterCommit(IInvokableInstance cmsInstance,
|
||||||
|
SerializableBiPredicate<Transformation, Commit.Result> predicate)
|
||||||
|
```
|
||||||
|
Get epoch after transformation commits matching predicate.
|
||||||
|
|
||||||
|
### pauseBeforeEnacting()
|
||||||
|
```java
|
||||||
|
Callable<Void> pauseBeforeEnacting(IInvokableInstance instance, long epoch)
|
||||||
|
Callable<Void> pauseBeforeEnacting(IInvokableInstance instance, Epoch epoch)
|
||||||
|
```
|
||||||
|
Pause before enacting specific epoch. Returns callable that waits for pause.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```java
|
||||||
|
Callable<Void> pauseHandle = ClusterUtils.pauseBeforeEnacting(instance, targetEpoch);
|
||||||
|
|
||||||
|
// Wait for pause
|
||||||
|
pauseHandle.call();
|
||||||
|
|
||||||
|
// Perform operations while paused
|
||||||
|
// ...
|
||||||
|
|
||||||
|
// Resume
|
||||||
|
ClusterUtils.unpauseEnactment(instance);
|
||||||
|
```
|
||||||
|
|
||||||
|
### pauseAfterEnacting()
|
||||||
|
```java
|
||||||
|
Callable<Void> pauseAfterEnacting(IInvokableInstance instance, Epoch epoch)
|
||||||
|
```
|
||||||
|
Pause after enacting specific epoch.
|
||||||
|
|
||||||
|
### unpauseCommits()
|
||||||
|
```java
|
||||||
|
void unpauseCommits(IInvokableInstance instance)
|
||||||
|
```
|
||||||
|
Resume CMS commit processing.
|
||||||
|
|
||||||
|
### unpauseEnactment()
|
||||||
|
```java
|
||||||
|
void unpauseEnactment(IInvokableInstance instance)
|
||||||
|
```
|
||||||
|
Resume epoch enactment.
|
||||||
|
|
||||||
|
### clearAndUnpause()
|
||||||
|
```java
|
||||||
|
void clearAndUnpause(IInvokableInstance instance)
|
||||||
|
```
|
||||||
|
Clear all pause conditions and resume.
|
||||||
|
|
||||||
|
### dropAllEntriesBeginningAt()
|
||||||
|
```java
|
||||||
|
void dropAllEntriesBeginningAt(IInvokableInstance instance, Epoch epoch)
|
||||||
|
```
|
||||||
|
Add filter to drop all log entries at or after specified epoch.
|
||||||
|
|
||||||
|
### clearEntryFilters()
|
||||||
|
```java
|
||||||
|
void clearEntryFilters(IInvokableInstance instance)
|
||||||
|
```
|
||||||
|
Clear all entry filters.
|
||||||
|
|
||||||
|
## Ring & Gossip Monitoring
|
||||||
|
|
||||||
|
### ring()
|
||||||
|
```java
|
||||||
|
List<RingInstanceDetails> ring(IInstance inst)
|
||||||
|
```
|
||||||
|
Get ring from nodetool ring output.
|
||||||
|
|
||||||
|
**RingInstanceDetails fields:**
|
||||||
|
- `address`: Node address
|
||||||
|
- `rack`: Rack name
|
||||||
|
- `status`: Up/Down
|
||||||
|
- `state`: Normal/Leaving/Joining/Moving
|
||||||
|
- `token`: Token value
|
||||||
|
|
||||||
|
### assertInRing()
|
||||||
|
```java
|
||||||
|
List<RingInstanceDetails> assertInRing(IInstance instance, IInstance expectedInRing)
|
||||||
|
```
|
||||||
|
Assert target instance is in ring.
|
||||||
|
|
||||||
|
### assertNotInRing()
|
||||||
|
```java
|
||||||
|
List<RingInstanceDetails> assertNotInRing(IInstance instance, IInstance expectedInRing)
|
||||||
|
```
|
||||||
|
Assert target instance is NOT in ring.
|
||||||
|
|
||||||
|
### assertRingState()
|
||||||
|
```java
|
||||||
|
List<RingInstanceDetails> assertRingState(IInstance instance, IInstance expectedInRing, String state)
|
||||||
|
```
|
||||||
|
Assert target has specific state (Normal, Leaving, Joining, etc.).
|
||||||
|
|
||||||
|
### assertRingIs()
|
||||||
|
```java
|
||||||
|
List<RingInstanceDetails> assertRingIs(IInstance instance, IInstance... expectedInRing)
|
||||||
|
List<RingInstanceDetails> assertRingIs(IInstance instance, Collection<? extends IInstance> expectedInRing)
|
||||||
|
List<RingInstanceDetails> assertRingIs(IInstance instance, Set<String> expectedRingAddresses)
|
||||||
|
```
|
||||||
|
Assert ring contains exactly the expected instances.
|
||||||
|
|
||||||
|
### awaitRingJoin()
|
||||||
|
```java
|
||||||
|
List<RingInstanceDetails> awaitRingJoin(IInstance instance, IInstance expectedInRing)
|
||||||
|
List<RingInstanceDetails> awaitRingJoin(IInstance instance, String expectedInRing)
|
||||||
|
void awaitRingJoin(Cluster cluster, int[] nodes, IInvokableInstance expectedInRing)
|
||||||
|
```
|
||||||
|
Wait for target to join ring (status=Up, state=Normal).
|
||||||
|
|
||||||
|
### awaitRingHealthy()
|
||||||
|
```java
|
||||||
|
List<RingInstanceDetails> awaitRingHealthy(IInstance src)
|
||||||
|
```
|
||||||
|
Wait for all instances in ring to be Up and Normal.
|
||||||
|
|
||||||
|
### awaitRingStatus()
|
||||||
|
```java
|
||||||
|
List<RingInstanceDetails> awaitRingStatus(IInstance instance, IInstance expectedInRing, String status)
|
||||||
|
```
|
||||||
|
Wait for target to have specific status (Up/Down).
|
||||||
|
|
||||||
|
### awaitRingState()
|
||||||
|
```java
|
||||||
|
List<RingInstanceDetails> awaitRingState(IInstance instance, IInstance expectedInRing, String state)
|
||||||
|
```
|
||||||
|
Wait for target to have specific state.
|
||||||
|
|
||||||
|
### gossipInfo()
|
||||||
|
```java
|
||||||
|
Map<String, Map<String, String>> gossipInfo(IInstance inst)
|
||||||
|
```
|
||||||
|
Get gossip information from nodetool gossipinfo.
|
||||||
|
|
||||||
|
**Returns:** Map from address to state map (generation, heartbeat, STATUS, etc.)
|
||||||
|
|
||||||
|
### assertGossipInfo()
|
||||||
|
```java
|
||||||
|
void assertGossipInfo(IInstance instance, InetSocketAddress expectedInGossip,
|
||||||
|
int expectedGeneration, int expectedHeartbeat)
|
||||||
|
```
|
||||||
|
Assert specific gossip generation and heartbeat values.
|
||||||
|
|
||||||
|
### awaitGossipStatus()
|
||||||
|
```java
|
||||||
|
Map<String, Map<String, String>> awaitGossipStatus(IInstance instance,
|
||||||
|
IInstance expectedInGossip,
|
||||||
|
String targetStatus)
|
||||||
|
```
|
||||||
|
Wait for target to have specific gossip status (e.g., "NORMAL").
|
||||||
|
|
||||||
|
### awaitGossipSchemaMatch()
|
||||||
|
```java
|
||||||
|
void awaitGossipSchemaMatch(ICluster<? extends IInstance> cluster)
|
||||||
|
void awaitGossipSchemaMatch(IInstance instance)
|
||||||
|
```
|
||||||
|
Wait for schema IDs to match in gossip across cluster.
|
||||||
|
|
||||||
|
### awaitGossipStateMatch()
|
||||||
|
```java
|
||||||
|
void awaitGossipStateMatch(ICluster<? extends IInstance> cluster,
|
||||||
|
IInstance expectedInGossip,
|
||||||
|
ApplicationState key)
|
||||||
|
```
|
||||||
|
Wait for specific ApplicationState value to converge across cluster.
|
||||||
|
|
||||||
|
## System Tables
|
||||||
|
|
||||||
|
### awaitInPeers()
|
||||||
|
```java
|
||||||
|
void awaitInPeers(Cluster cluster, int[] nodes, IInstance expectedInPeers)
|
||||||
|
void awaitInPeers(IInstance instance, IInstance expectedInPeers)
|
||||||
|
```
|
||||||
|
Wait for target to appear in system.peers with full info (tokens, dc, rack).
|
||||||
|
|
||||||
|
### isInPeers()
|
||||||
|
```java
|
||||||
|
boolean isInPeers(IInstance instance, IInstance expectedInPeers)
|
||||||
|
```
|
||||||
|
Check if target is in system.peers with complete information.
|
||||||
|
|
||||||
|
## Directories & Files
|
||||||
|
|
||||||
|
### getDataDirectories()
|
||||||
|
```java
|
||||||
|
List<File> getDataDirectories(IInstance instance)
|
||||||
|
```
|
||||||
|
Get all data directories for instance.
|
||||||
|
|
||||||
|
### getCommitLogDirectory()
|
||||||
|
```java
|
||||||
|
File getCommitLogDirectory(IInstance instance)
|
||||||
|
```
|
||||||
|
Get commitlog directory.
|
||||||
|
|
||||||
|
### getHintsDirectory()
|
||||||
|
```java
|
||||||
|
File getHintsDirectory(IInstance instance)
|
||||||
|
```
|
||||||
|
Get hints directory.
|
||||||
|
|
||||||
|
### getSavedCachesDirectory()
|
||||||
|
```java
|
||||||
|
File getSavedCachesDirectory(IInstance instance)
|
||||||
|
```
|
||||||
|
Get saved caches directory.
|
||||||
|
|
||||||
|
### getJournalDirectory()
|
||||||
|
```java
|
||||||
|
File getJournalDirectory(IInstance instance)
|
||||||
|
```
|
||||||
|
Get Accord journal directory.
|
||||||
|
|
||||||
|
### getCdcRawDirectory()
|
||||||
|
```java
|
||||||
|
File getCdcRawDirectory(IInstance instance)
|
||||||
|
```
|
||||||
|
Get CDC raw directory.
|
||||||
|
|
||||||
|
### getDirectories()
|
||||||
|
```java
|
||||||
|
List<File> getDirectories(IInstance instance)
|
||||||
|
```
|
||||||
|
Get all writable directories (data, commitlog, hints, saved caches, journal, cdc).
|
||||||
|
|
||||||
|
### cleanup()
|
||||||
|
```java
|
||||||
|
void cleanup(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Stop instance, delete all directories, restart with clean state.
|
||||||
|
|
||||||
|
## Node Operations
|
||||||
|
|
||||||
|
### decommission()
|
||||||
|
```java
|
||||||
|
boolean decommission(IInvokableInstance leaving)
|
||||||
|
```
|
||||||
|
Decommission node. Returns true if successful.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```java
|
||||||
|
boolean success = ClusterUtils.decommission(cluster.get(3));
|
||||||
|
Assert.assertTrue("Decommission failed", success);
|
||||||
|
|
||||||
|
ClusterUtils.assertNotInRing(cluster.get(1), cluster.get(3));
|
||||||
|
```
|
||||||
|
|
||||||
|
### getNodeId()
|
||||||
|
```java
|
||||||
|
NodeId getNodeId(IInvokableInstance target)
|
||||||
|
int getNodeId(IInvokableInstance target, IInvokableInstance executor)
|
||||||
|
```
|
||||||
|
Get NodeId for target instance.
|
||||||
|
|
||||||
|
### cancelInProgressSequences()
|
||||||
|
```java
|
||||||
|
boolean cancelInProgressSequences(IInvokableInstance executor)
|
||||||
|
boolean cancelInProgressSequences(NodeId nodeId, IInvokableInstance executor)
|
||||||
|
```
|
||||||
|
Cancel in-progress sequences for node. Returns true if successful.
|
||||||
|
|
||||||
|
### mode()
|
||||||
|
```java
|
||||||
|
StorageService.Mode mode(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Get current StorageService operation mode.
|
||||||
|
|
||||||
|
**Modes:** STARTING, NORMAL, JOINING, LEAVING, DECOMMISSIONED, MOVING, DRAINING, DRAINED
|
||||||
|
|
||||||
|
### assertModeJoined()
|
||||||
|
```java
|
||||||
|
void assertModeJoined(IInvokableInstance inst)
|
||||||
|
```
|
||||||
|
Assert instance is in NORMAL mode.
|
||||||
|
|
||||||
|
### isMigrating()
|
||||||
|
```java
|
||||||
|
boolean isMigrating(IInvokableInstance instance)
|
||||||
|
```
|
||||||
|
Check if CMS is currently migrating.
|
||||||
|
|
||||||
|
## Log Monitoring
|
||||||
|
|
||||||
|
### runAndWaitForLogs()
|
||||||
|
```java
|
||||||
|
void runAndWaitForLogs(Runnable r, String waitString, AbstractCluster<I> cluster)
|
||||||
|
void runAndWaitForLogs(Runnable r, String waitString, IInstance... instances)
|
||||||
|
```
|
||||||
|
Mark log positions, run operation, wait for log message to appear on all instances.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```java
|
||||||
|
ClusterUtils.runAndWaitForLogs(
|
||||||
|
() -> cluster.get(2).nodetool("bootstrap"),
|
||||||
|
"Bootstrap completed",
|
||||||
|
cluster.get(1), cluster.get(2), cluster.get(3)
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
## Address Management
|
||||||
|
|
||||||
|
### updateAddress()
|
||||||
|
```java
|
||||||
|
void updateAddress(IInstance instance, String address)
|
||||||
|
```
|
||||||
|
Change instance's address. Only call when instance is down.
|
||||||
|
|
||||||
|
**Warning:** Modifies broadcast_address, listen_address, broadcast_rpc_address, rpc_address.
|
||||||
|
|
||||||
|
### getBroadcastAddressHostWithPortString()
|
||||||
|
```java
|
||||||
|
String getBroadcastAddressHostWithPortString(IInstance target)
|
||||||
|
```
|
||||||
|
Get broadcast address in "host:port" format (e.g., "127.0.0.1:7190").
|
||||||
|
|
||||||
|
### getNativeInetSocketAddress()
|
||||||
|
```java
|
||||||
|
InetSocketAddress getNativeInetSocketAddress(IInstance target)
|
||||||
|
```
|
||||||
|
Get native protocol address (CQL port) as InetSocketAddress.
|
||||||
|
|
||||||
|
### getIntConfig()
|
||||||
|
```java
|
||||||
|
int getIntConfig(IInstanceConfig config, String configName, int defaultValue)
|
||||||
|
```
|
||||||
|
Safely get integer config value with fallback.
|
||||||
|
|
||||||
|
## Accord Testing
|
||||||
|
|
||||||
|
### queryTxnState()
|
||||||
|
```java
|
||||||
|
<T extends IInstance> LinkedHashMap<String, SimpleQueryResult> queryTxnState(
|
||||||
|
AbstractCluster<T> cluster,
|
||||||
|
TxnId txnId,
|
||||||
|
int... nodes)
|
||||||
|
```
|
||||||
|
Query Accord transaction state from virtual table across nodes.
|
||||||
|
|
||||||
|
### queryTxnStateAsString()
|
||||||
|
```java
|
||||||
|
<T extends IInstance> String queryTxnStateAsString(AbstractCluster<T> cluster, TxnId txnId, int... nodes)
|
||||||
|
<T extends IInstance> void queryTxnStateAsString(StringBuilder sb, AbstractCluster<T> cluster, TxnId txnId, int... nodes)
|
||||||
|
```
|
||||||
|
Get Accord transaction state as formatted string.
|
||||||
|
|
||||||
|
### tableId()
|
||||||
|
```java
|
||||||
|
TableId tableId(Cluster cluster, String ks, String table)
|
||||||
|
```
|
||||||
|
Get TableId for keyspace and table.
|
||||||
|
|
||||||
|
### awaitAccordEpochReady()
|
||||||
|
```java
|
||||||
|
void awaitAccordEpochReady(Cluster cluster, long epoch)
|
||||||
|
```
|
||||||
|
Wait for Accord epoch to be ready for reads on all instances.
|
||||||
|
|
||||||
|
## Utility Classes
|
||||||
|
|
||||||
|
### Range
|
||||||
|
```java
|
||||||
|
public static class Range implements Serializable {
|
||||||
|
public final String left, right;
|
||||||
|
|
||||||
|
public Range(String left, String right)
|
||||||
|
public Range(long left, long right)
|
||||||
|
|
||||||
|
public long left()
|
||||||
|
public long right()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Represents a token range.
|
||||||
|
|
||||||
|
### RingInstanceDetails
|
||||||
|
```java
|
||||||
|
public static final class RingInstanceDetails {
|
||||||
|
private final String address;
|
||||||
|
private final String rack;
|
||||||
|
private final String status; // Up/Down
|
||||||
|
private final String state; // Normal/Leaving/Joining/Moving
|
||||||
|
private final String token;
|
||||||
|
|
||||||
|
public String getAddress()
|
||||||
|
public String getRack()
|
||||||
|
public String getStatus()
|
||||||
|
public String getState()
|
||||||
|
public String getToken()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Parsed nodetool ring output for single instance.
|
||||||
|
|
@ -0,0 +1,284 @@
|
||||||
|
---
|
||||||
|
name: deep-review
|
||||||
|
version: "1.0.0"
|
||||||
|
description: >
|
||||||
|
Deep file-focused code review for correctness bugs. Unlike shallow-review which runs 6
|
||||||
|
specialists in parallel across the entire patch, deep-review focuses on user-specified
|
||||||
|
files with full pattern catalogs (500+ patterns), codebase investigation, and source-level
|
||||||
|
context gathering. Use when: the user specifies particular files for focused review, a
|
||||||
|
shallow review flagged areas that need deeper investigation, reviewing critical-path code
|
||||||
|
changes, examining complex serialization/lifecycle/state-machine changes. The user
|
||||||
|
instructs which files to focus on (typically a subset of files in the patch).
|
||||||
|
---
|
||||||
|
|
||||||
|
# Deep Code Review
|
||||||
|
|
||||||
|
Focused, thorough review of user-specified files using the full 500+ pattern catalog.
|
||||||
|
Each file is reviewed with context gathering (reading source, not just diff), codebase
|
||||||
|
searches, and cross-referencing against the complete pattern database.
|
||||||
|
|
||||||
|
```
|
||||||
|
+------------------+
|
||||||
|
| Input Patch |
|
||||||
|
+--------+---------+
|
||||||
|
|
|
||||||
|
+--------v---------+
|
||||||
|
| User selects |
|
||||||
|
| focus files |
|
||||||
|
+--------+---------+
|
||||||
|
|
|
||||||
|
+--------------+--------------+
|
||||||
|
| |
|
||||||
|
+---------v---------+ +-----------v-----------+
|
||||||
|
| Phase 0: Context | | Phase 0: Context |
|
||||||
|
| (read full files, | | (read full files, |
|
||||||
|
| callers, types) | | callers, types) |
|
||||||
|
+---------+---------+ +-----------+-----------+
|
||||||
|
| |
|
||||||
|
+---------v---------+ +-----------v-----------+
|
||||||
|
| Phase 1: Deep | | Phase 1: Deep |
|
||||||
|
| specialist review | | specialist review |
|
||||||
|
| (full checklists) | | (full checklists) |
|
||||||
|
+---------+---------+ +-----------+-----------+
|
||||||
|
| |
|
||||||
|
+---------v---------+ +-----------v-----------+
|
||||||
|
| Phase 2: Codebase | | Phase 2: Codebase |
|
||||||
|
| investigation | | investigation |
|
||||||
|
+---------+---------+ +-----------+-----------+
|
||||||
|
| |
|
||||||
|
+--------------+--------------+
|
||||||
|
|
|
||||||
|
+--------v---------+
|
||||||
|
| Phase 3: Cross- |
|
||||||
|
| file symmetry & |
|
||||||
|
| merge findings |
|
||||||
|
+------------------+
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
1. **Receive patch** and user's file focus list
|
||||||
|
2. **For each focus file**: run Phase 0 (context), Phase 1 (deep checklists), Phase 2 (search)
|
||||||
|
3. **Cross-file symmetry** pass across all focus files
|
||||||
|
4. **Merge & report** with confidence ranking
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 0: Context Gathering (per focus file)
|
||||||
|
|
||||||
|
Before touching any checklist, deeply understand each focus file. This is the key difference
|
||||||
|
from shallow review — you read the actual source, not just the diff.
|
||||||
|
|
||||||
|
**Prompt template**:
|
||||||
|
```
|
||||||
|
Read the full source of the focus file: `{FILE_PATH}`
|
||||||
|
Read the patch for context: `{PATCH_PATH}`
|
||||||
|
|
||||||
|
For this file, gather:
|
||||||
|
1. CLASS HIERARCHY: What does it extend? What interfaces? Read the parent class/interface.
|
||||||
|
2. SIBLING CLASSES: Are there parallel implementations? Read at least one sibling.
|
||||||
|
3. CALLERS: Who calls the modified methods? How do they use return values? (grep + read)
|
||||||
|
4. LIFECYCLE: When is this object created, used, destroyed?
|
||||||
|
5. STATE MACHINE (if applicable): Map every state and every transition. What events are handled
|
||||||
|
in each state? What events are *not* handled (missing transitions)? What states are never
|
||||||
|
reached? Where is this state enum pattern-matched — are all states covered at every match site?
|
||||||
|
6. INVARIANTS: Read Javadoc, assertions, and any test files for this class.
|
||||||
|
7. SERIALIZATION: If it serializes — read serialize, deserialize, serializedSize together.
|
||||||
|
|
||||||
|
Output a structured context summary for use by the deep review phases.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1: Deep Specialist Review (per focus file)
|
||||||
|
|
||||||
|
Each focus file is reviewed against the FULL domain checklists (not the trimmed shallow
|
||||||
|
versions). The reviewer selects which domains apply based on Phase 0 context.
|
||||||
|
|
||||||
|
### Domain Selection
|
||||||
|
|
||||||
|
Based on Phase 0 context, load the relevant deep checklists:
|
||||||
|
|
||||||
|
| File contains | Load checklist |
|
||||||
|
|---|---|
|
||||||
|
| Conditions, comparisons, control flow, types | `references/deep/deep-logic.md` |
|
||||||
|
| Arithmetic, indices, ranges, ByteBuffer, I/O | `references/deep/deep-boundary.md` |
|
||||||
|
| Shared state, locks, lifecycle, state machines | `references/deep/deep-concurrency.md` |
|
||||||
|
| Serialization, resources, metrics, config | `references/deep/deep-resources.md` |
|
||||||
|
| New classes, new fields, registrations | `references/deep/deep-absence-completeness.md` |
|
||||||
|
| Refactoring, merges, shell scripts, parsing | `references/deep/deep-cross-cutting.md` |
|
||||||
|
|
||||||
|
Typically 2-4 domains are relevant per file. Load all relevant ones.
|
||||||
|
|
||||||
|
**Prompt template**:
|
||||||
|
```
|
||||||
|
You are performing a DEEP REVIEW of: `{FILE_PATH}`
|
||||||
|
|
||||||
|
CONTEXT (from Phase 0):
|
||||||
|
{CONTEXT_SUMMARY}
|
||||||
|
|
||||||
|
PATCH (the changes being reviewed):
|
||||||
|
Read: `{PATCH_PATH}`
|
||||||
|
|
||||||
|
CHECKLISTS (load each relevant one):
|
||||||
|
Read: `{SKILL_DIR}/references/deep/deep-logic.md`
|
||||||
|
Read: `{SKILL_DIR}/references/deep/deep-boundary.md`
|
||||||
|
[... load relevant domains ...]
|
||||||
|
|
||||||
|
For EVERY checklist item that could apply to this file:
|
||||||
|
1. Check whether the pattern exists in the code
|
||||||
|
2. If the pattern matches: search the codebase for confirming/refuting evidence
|
||||||
|
3. Rate confidence: High (confirmed by evidence), Medium (pattern matches but
|
||||||
|
uncertain), Low (possible but speculative)
|
||||||
|
|
||||||
|
Report ALL findings. For each:
|
||||||
|
- Pattern ID and name
|
||||||
|
- Location (file:line)
|
||||||
|
- Confidence (High/Medium/Low)
|
||||||
|
- What's wrong (2-3 sentences with reasoning)
|
||||||
|
- Evidence gathered (what you searched for, what you found/didn't find)
|
||||||
|
- Suggested fix
|
||||||
|
|
||||||
|
Or "No finding in this domain for this file" if nothing applies.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2: Codebase Investigation (per focus file)
|
||||||
|
|
||||||
|
For each finding from Phase 1, and for absence/completeness patterns, execute targeted
|
||||||
|
codebase searches.
|
||||||
|
|
||||||
|
**Search patterns to execute:**
|
||||||
|
|
||||||
|
### For new fields:
|
||||||
|
```bash
|
||||||
|
grep -rn "serialize\|deserialize\|serializedSize\|equals\|hashCode\|toString" {FILE_DIR}/
|
||||||
|
```
|
||||||
|
|
||||||
|
### For new registrations:
|
||||||
|
```bash
|
||||||
|
grep -rn "addListener\|register\|subscribe\|addMetric\|removeSensor\|deregister" {FILE_DIR}/
|
||||||
|
```
|
||||||
|
|
||||||
|
### For new enum constants:
|
||||||
|
```bash
|
||||||
|
grep -rn "switch.*{ENUM_TYPE}\|case.*{CONSTANT}" --include="*.java" {PROJECT_ROOT}/
|
||||||
|
```
|
||||||
|
|
||||||
|
### For interface implementations:
|
||||||
|
```bash
|
||||||
|
grep -rn "implements {INTERFACE}" --include="*.java" {PROJECT_ROOT}/
|
||||||
|
```
|
||||||
|
|
||||||
|
### For method callers:
|
||||||
|
```bash
|
||||||
|
grep -rn "{METHOD_NAME}" --include="*.java" {PROJECT_ROOT}/
|
||||||
|
```
|
||||||
|
|
||||||
|
### For parallel paths:
|
||||||
|
```bash
|
||||||
|
grep -rn "class.*extends.*{PARENT_CLASS}" --include="*.java" {PROJECT_ROOT}/
|
||||||
|
```
|
||||||
|
|
||||||
|
**Prompt template**:
|
||||||
|
```
|
||||||
|
You are investigating findings from the deep review of: `{FILE_PATH}`
|
||||||
|
|
||||||
|
FINDINGS TO INVESTIGATE:
|
||||||
|
{PHASE_1_FINDINGS}
|
||||||
|
|
||||||
|
For each finding:
|
||||||
|
1. Execute the relevant codebase searches (grep, read files)
|
||||||
|
2. Confirm or refute the finding based on evidence
|
||||||
|
3. If confirmed: search for additional instances of the same pattern
|
||||||
|
4. If refuted: explain why with evidence
|
||||||
|
|
||||||
|
For absence patterns:
|
||||||
|
1. Build the search list from the diff events
|
||||||
|
2. Execute each search
|
||||||
|
3. If no match found: report as confirmed finding
|
||||||
|
4. If match found: report as refuted (the symmetric code exists)
|
||||||
|
|
||||||
|
Report updated findings with evidence.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3: Cross-File Symmetry & Merge
|
||||||
|
|
||||||
|
After all focus files are reviewed:
|
||||||
|
|
||||||
|
**Prompt template**:
|
||||||
|
```
|
||||||
|
Review findings across all focus files for cross-file patterns:
|
||||||
|
|
||||||
|
1. CROSS-FILE SYMMETRY: For each change in file A, is there a structurally parallel
|
||||||
|
change needed in file B? Check:
|
||||||
|
- Serialization families (if serialize changed, did deserialize change?)
|
||||||
|
- Interface implementations (if one implementation changed, do siblings need it?)
|
||||||
|
- Version-gated paths (if one version branch changed, do others?)
|
||||||
|
- Test coverage (does the test file cover the changed behavior?)
|
||||||
|
|
||||||
|
2. FINDING REINFORCEMENT: Do findings across files reinforce each other?
|
||||||
|
- Logic + Absence at same conceptual location → boost to High
|
||||||
|
- Concurrency + Resources on same lifecycle → boost to High
|
||||||
|
- Pattern flagged in multiple files → likely systematic issue
|
||||||
|
|
||||||
|
3. MERGE & DEDUP: Combine all findings, deduplicate, rank.
|
||||||
|
|
||||||
|
4. THREE-POINT TEST each finding:
|
||||||
|
- The code construct actually exists (not inferred)
|
||||||
|
- The bug is possible given visible context (not speculative)
|
||||||
|
- The finding is actionable (what specifically should change)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Report Format
|
||||||
|
|
||||||
|
```
|
||||||
|
## Deep Review: [target]
|
||||||
|
|
||||||
|
### Focus Files
|
||||||
|
- `path/to/File1.java` — [brief description of changes]
|
||||||
|
- `path/to/File2.java` — [brief description of changes]
|
||||||
|
|
||||||
|
### Context Summary
|
||||||
|
[Key facts from Phase 0: class hierarchy, lifecycle, shared state, invariants]
|
||||||
|
|
||||||
|
### Findings (ranked by confidence)
|
||||||
|
|
||||||
|
#### Finding 1: [title]
|
||||||
|
- **Location**: [file:line]
|
||||||
|
- **Confidence**: High / Medium / Low
|
||||||
|
- **Domain**: [Logic / Boundary / Concurrency / Resources / Absence / Completeness]
|
||||||
|
- **Pattern**: [pattern name from checklist]
|
||||||
|
- **What's wrong**: [2-3 sentences with reasoning]
|
||||||
|
- **Evidence**: [what was searched, what was found]
|
||||||
|
- **Suggested fix**: [specific change]
|
||||||
|
|
||||||
|
#### Finding 2: ...
|
||||||
|
|
||||||
|
### Domain Coverage (per file)
|
||||||
|
| File | Logic | Boundary | Concurrency | Resources | Absence | Completeness |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| File1.java | 2 findings | - | 1 finding | - | checked | checked |
|
||||||
|
| File2.java | - | 1 finding | - | 3 findings | checked | checked |
|
||||||
|
|
||||||
|
### Cross-File Symmetry
|
||||||
|
- [Any asymmetries found]
|
||||||
|
- [Any reinforcing patterns]
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Reference Files
|
||||||
|
|
||||||
|
### Deep specialist checklists
|
||||||
|
- `references/deep/deep-logic.md` — Full logic, types, constants, filtering patterns
|
||||||
|
- `references/deep/deep-boundary.md` — Full boundary, overflow, null, I/O patterns
|
||||||
|
- `references/deep/deep-concurrency.md` — Full concurrency, state, lifecycle patterns
|
||||||
|
- `references/deep/deep-resources.md` — Full serialization, resource, crash-safety patterns
|
||||||
|
- `references/deep/deep-absence-completeness.md` — Full absence + completeness with search methodology
|
||||||
|
- `references/deep/deep-cross-cutting.md` — Refactoring, parsing, platform, dispatch patterns
|
||||||
|
|
@ -0,0 +1,266 @@
|
||||||
|
# Deep Absence & Completeness — Extended Checklist
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context Gathering (REQUIRED)
|
||||||
|
|
||||||
|
For each target file, gather:
|
||||||
|
|
||||||
|
1. **All interfaces/superclasses**: What contracts must this class fulfill?
|
||||||
|
2. **All sibling implementations**: What parallel classes exist for the same interface?
|
||||||
|
3. **All callers**: Who creates and uses this object?
|
||||||
|
4. **Serialization family**: Does this class participate in serialize/deserialize/equals/toString?
|
||||||
|
5. **Registration set**: What metrics, listeners, handlers does this class register?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Absence: Registration & Lifecycle Depth
|
||||||
|
|
||||||
|
### Listener/metric leak search (systematic)
|
||||||
|
For EVERY `addListener`, `register`, `subscribe`, `addMetric`, `createSensor`, `put` into registry:
|
||||||
|
- Search the same class for matching `remove`/`deregister`/`removeSensor`
|
||||||
|
- Check BOTH `close()` and `stop()` and `destroy()` paths
|
||||||
|
- Check the failure/exception path separately from success
|
||||||
|
- If multiple registrations happen in sequence: does cleanup handle partial failure?
|
||||||
|
|
||||||
|
### Self-registering class loading (specific)
|
||||||
|
- For `static final INSTANCE = new Foo()`: is the class actually loaded from the startup path?
|
||||||
|
- Grep for references to the class name. If only optional code paths reference it, static init never runs.
|
||||||
|
|
||||||
|
### Background task lifecycle (specific)
|
||||||
|
- For every `scheduleWithFixedDelay`, `scheduleAtFixedRate`, `Timer`, `ScheduledExecutor`:
|
||||||
|
- Is there a `shutdown()`/`cancel()` mechanism?
|
||||||
|
- Is it called from test teardown?
|
||||||
|
- Is it gated on a feature flag?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Absence: Event & Handler Coverage Depth
|
||||||
|
|
||||||
|
### Enum constant dispatch search (systematic)
|
||||||
|
For EVERY new enum constant or event type:
|
||||||
|
- `grep -rn "switch.*EnumType\|case.*ENUM_CONSTANT"` across the entire codebase
|
||||||
|
- List ALL switch/dispatch/if-else chains. Is the new constant handled in EACH?
|
||||||
|
- Check `default:` clauses — is `break` (silent skip) or `throw` (fail-fast) correct?
|
||||||
|
|
||||||
|
### State machine handler completeness (specific)
|
||||||
|
For each `onTimeout()`, `onError()`, `onFailure()`, `onRetry()` override:
|
||||||
|
- Read sibling classes implementing the same interface
|
||||||
|
- What follow-up action do they perform?
|
||||||
|
- Does THIS implementation perform the equivalent action or just log/no-op?
|
||||||
|
- Will a no-op leave the state machine stuck?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Absence: Field Completeness Depth
|
||||||
|
|
||||||
|
### New field propagation search (systematic)
|
||||||
|
For EVERY new field added to a class:
|
||||||
|
- Is the field written in the serialization method (serialize/write/encode/marshal)?
|
||||||
|
- Is the field read in the deserialization method (deserialize/read/decode/unmarshal)?
|
||||||
|
- Is the field measured in the size method (serializedSize/encodedSize/size), if one exists?
|
||||||
|
- Symmetry: do all three handle the field in the same order and under the same conditional guards?
|
||||||
|
- Search for `equals` method — is the field compared?
|
||||||
|
- Search for `hashCode` method — is the field hashed?
|
||||||
|
- Search for `toString` method — is the field printed?
|
||||||
|
- Search for copy constructor, `clone()`, `sharedCopy()`, `toBuilder()` — is the field copied?
|
||||||
|
- Search for `describe()`, `toMap()`, snapshot methods — is the field included?
|
||||||
|
|
||||||
|
### Parallel method consistency (specific)
|
||||||
|
- Compare field lists across `equals`, `hashCode`, `toString`, `serialize`. Do they agree?
|
||||||
|
- If one method has N fields and another has N-1, which field is missing and why?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Absence: Parallel Path Coverage Depth
|
||||||
|
|
||||||
|
### Structural symmetry search (systematic)
|
||||||
|
For each modification to path A:
|
||||||
|
- Identify structurally parallel path B (same class, same interface, same event family)
|
||||||
|
- Does path B have the corresponding modification?
|
||||||
|
- Specifically check:
|
||||||
|
- If a guard `if (X) return` was added to path A: does path B have it?
|
||||||
|
- If a method call was added to one if-branch: does the else-branch need it?
|
||||||
|
- If a field was added to one describe/serialize: do sibling methods have it?
|
||||||
|
- If one state callback was modified: are all sibling callbacks updated?
|
||||||
|
|
||||||
|
### Fix-patch parallel path search
|
||||||
|
When reviewing a bug-fix:
|
||||||
|
- What was missing from the original code?
|
||||||
|
- Did the fix add it EVERYWHERE?
|
||||||
|
- Same lookup on parallel path (fast/slow, NIO/fallback)?
|
||||||
|
- Other catch blocks or early-return paths?
|
||||||
|
- Other version guards?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Absence: Scope & Visibility Depth
|
||||||
|
|
||||||
|
### Private/package-private access search
|
||||||
|
For every new `private` or package-private member:
|
||||||
|
- Is `@VisibleForTesting` used on a singleton accessed cross-package?
|
||||||
|
|
||||||
|
### Feature flag / version guard coverage search
|
||||||
|
For every feature flag check and every `version >= X` guard:
|
||||||
|
- Does it guard the main operation?
|
||||||
|
- Does it guard setup?
|
||||||
|
- Does it guard teardown?
|
||||||
|
- Does it guard secondary/dependent operations?
|
||||||
|
- For version guards: is the same version condition used consistently, or do different paths use different thresholds?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Completeness: Interface Implementation Depth
|
||||||
|
|
||||||
|
### Predicate method override search
|
||||||
|
For every new class implementing an interface:
|
||||||
|
- List ALL abstract/default methods in the interface
|
||||||
|
- For each predicate (`isX()`, `hasX()`, `canX()`):
|
||||||
|
- What is the default return value?
|
||||||
|
- Is that default CORRECT for this implementation?
|
||||||
|
- If the class has state that makes the predicate true, the override is MANDATORY
|
||||||
|
|
||||||
|
### Lifecycle method override search
|
||||||
|
- Does the class hold resources the parent doesn't know about?
|
||||||
|
- Does it need to override `close()`, `release()`, `abort()`?
|
||||||
|
- Does it need to override `sharedCopy()`, `clone()`, copy constructor?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Completeness: Factory & Dispatch Depth
|
||||||
|
|
||||||
|
### Factory method type routing
|
||||||
|
For each factory dispatching on a type parameter:
|
||||||
|
- List ALL possible values of the discriminator
|
||||||
|
- For each value: verify the returned concrete class is CORRECT
|
||||||
|
- If a generic `else`/`default` handles multiple values: verify each is correct
|
||||||
|
- After consolidation: does each value produce the same type as before?
|
||||||
|
|
||||||
|
### Overload binding after signature change
|
||||||
|
- When a method adds a new parameter with default-providing overload:
|
||||||
|
- List ALL callers
|
||||||
|
- Do callers that need the new behavior call the new signature?
|
||||||
|
- Old callers silently bind to old overload
|
||||||
|
|
||||||
|
### Extract-method refactoring
|
||||||
|
- After extracting code into a helper:
|
||||||
|
- Does the call site retain an operation the helper also performs?
|
||||||
|
- Double-write? Double-close? Double-init?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Completeness: Accumulation & Constants Depth
|
||||||
|
|
||||||
|
### Accumulation operator (= vs +=)
|
||||||
|
For every `field = expression` in a loop:
|
||||||
|
- Is the intent accumulation (`+=`) or replacement (`=`)?
|
||||||
|
- Is this per-partition/per-item where total is needed?
|
||||||
|
- For compound stats (min, max, sum, count): is each updated with the correct operator?
|
||||||
|
|
||||||
|
### Constant vs accessor
|
||||||
|
- Is `Foo.bar()` called where `Foo.BAR` (constant) is correct?
|
||||||
|
- Is `Foo.BAR` referenced where `Foo.bar()` (dynamic accessor) is needed?
|
||||||
|
|
||||||
|
### Constructor parameter not stored
|
||||||
|
- Does a constructor accept a parameter never assigned to a field?
|
||||||
|
- Does `this.field = null` appear instead of `this.field = parameter`?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Additional Absence Patterns from Multi-Project Corpus
|
||||||
|
|
||||||
|
### Registered-object retention (weight: medium)
|
||||||
|
- For `registry.put(...)` / `manager.add(...)` inside factory or builder methods: is the registered object stored in a field of the enclosing owner?
|
||||||
|
- If the object is only local-scoped: trace who calls `deregister` at shutdown — is the lookup key still reachable?
|
||||||
|
- Does the owner of the registry outlive the registered object's natural lifetime?
|
||||||
|
|
||||||
|
### Idempotency on re-entrant lifecycle (weight: medium)
|
||||||
|
- For every `init`, `close`, `open`, `start`, `stop`: can it be called twice? (schema reload, listener callback, reset path)
|
||||||
|
- Is there a has-initialized or has-closed flag?
|
||||||
|
- If cleanup fires side effects (metrics, callbacks, observer notifications): does a second call re-fire them?
|
||||||
|
- Does a CAS-style initialize treat "already initialized by another thread" as success, not error?
|
||||||
|
|
||||||
|
### Rollback / counter-decrement on failure path (weight: medium)
|
||||||
|
- For every new increment, add, or `put` on the success path:
|
||||||
|
- Does the cancellation path decrement / remove?
|
||||||
|
- Does the task-rejection path decrement / remove?
|
||||||
|
- Does the exception-caught path decrement / remove?
|
||||||
|
- Does the timeout path decrement / remove?
|
||||||
|
- For state-session maps cleared on failure: is there also a clear on success?
|
||||||
|
- For metric counters incremented before async submit: is the counter reverted if the submit throws `RejectedExecutionException`?
|
||||||
|
|
||||||
|
### Missing metric / notification on sibling branch (weight: medium)
|
||||||
|
- Compare the new code against every sibling if/else, switch case, and cache-hit fast path
|
||||||
|
- Does each branch record the metric that one branch records?
|
||||||
|
- For state-transition listeners: does every state branch fire the same side-effect propagation calls?
|
||||||
|
- For close-path metric recording: is the recording inside `try` or inside `finally`?
|
||||||
|
|
||||||
|
### Configuration / parameter propagation depth (weight: high)
|
||||||
|
For every setting parsed from config, DDL, or builder input:
|
||||||
|
- Trace the value: parse → validate → store → copy into builder → constructor → factory call → downstream consumer
|
||||||
|
- At each hop, does the next layer actually read and apply the value, or does it fall back to a default?
|
||||||
|
- For wrapper / decorator classes: are child objects constructed with the parent's per-operation state, or with globals?
|
||||||
|
- For third-party builder chains (Netty, Jackson, Thrift): is every caller-relevant setter called, or does the library default silently win?
|
||||||
|
- For refactored constructors: does every existing call site still pass the now-required parameter?
|
||||||
|
|
||||||
|
### Version / runtime fallback depth (weight: medium)
|
||||||
|
- For new wire-format fields: is there a `version >= X` guard on both write and read?
|
||||||
|
- For new protocol features: is the cluster-wide minimum checked, not just the local capability?
|
||||||
|
- For reflective access to JVM internals: is there a fallback for the next Java version, or is the path guarded by a try/catch that includes the renamed-field exception?
|
||||||
|
- For native-library loads: does the catch handler include every exception the loader can throw (`LinkageError`, `UnsatisfiedLinkError`, `NoClassDefFoundError`)?
|
||||||
|
|
||||||
|
### Drain of trailing bytes on early-exit (weight: medium)
|
||||||
|
- For every `break` / `return` / `continue` added inside a framed-stream loop (checksummed read, length-prefixed records):
|
||||||
|
- Does the exit path advance past the checksum / footer?
|
||||||
|
- Does the exit path update the stream's logical position tracker?
|
||||||
|
- Does the exit path match what the normal-completion path does?
|
||||||
|
- For deserializers with partial-consume semantics: is there a drain step before returning?
|
||||||
|
|
||||||
|
### Directory / cross-file sync depth (weight: medium)
|
||||||
|
- Before a file is listed or deleted: has the directory been fsync'd since the last creation / removal?
|
||||||
|
- Before closing a buffered writer: is flush + fsync performed, or only close?
|
||||||
|
- When flushing a set of interdependent files (parent + child, index + data): is the order deterministic and is each flush complete before the next?
|
||||||
|
- Does the atomic-rename path rely on the directory fsync that may be missing?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Additional Completeness Patterns from Multi-Project Corpus
|
||||||
|
|
||||||
|
### Subclass missing override of measurement / copy / abort / interrupt / factory method (weight: medium)
|
||||||
|
- When a subclass adds instance fields or resources: does it override every method whose inherited parent-layout implementation would silently under-measure, under-copy, or no-op?
|
||||||
|
- Specifically look for: `unsharedHeapSize` / `estimateSize`, `abort()`, `interrupt()`, `sharedCopy()` / `clone()`, factory-style methods returning the same type.
|
||||||
|
- A parent default `return false` / `return 0` / no-op inherited by a stateful subclass silently corrupts the contract.
|
||||||
|
|
||||||
|
### Decorator / wrapper missing override of default-throwing method (weight: low)
|
||||||
|
- For every interface method whose default implementation throws `UnsupportedOperationException` or `NotImplementedError`: does the decorator override it?
|
||||||
|
- If the decorator only forwards the "core" methods and silently inherits default-throwing stubs for auxiliary ones, the first caller hitting the missed method crashes.
|
||||||
|
|
||||||
|
### New subtype / variant unhandled in type-dispatch (weight: medium)
|
||||||
|
- When a new subtype is introduced, grep ALL `instanceof` / `case` / visitor `visitXxx` chains in the codebase.
|
||||||
|
- Is the new variant handled in each? Silent fallthrough to default usually returns null or the raw unconverted value.
|
||||||
|
- Check Scala `sealed` trait usage from Java callers — exhaustiveness guarantees are lost across languages.
|
||||||
|
|
||||||
|
### Parameter forwarding dropped through wrapper / delegation (weight: medium)
|
||||||
|
- When a wrapper or factory forwards a call to an inner implementation: are ALL parameters propagated?
|
||||||
|
- Do tracing context, credentials, or consistency level get silently dropped through overload delegation to a narrower signature?
|
||||||
|
- When a field value is forwarded through N layers, does each layer carry it? A single layer that hardcodes the default / null silently drops caller intent.
|
||||||
|
|
||||||
|
### Constructor path bypasses registration / setup (weight: medium)
|
||||||
|
- If a class has multiple constructors: does each perform all required registration / validation / init steps?
|
||||||
|
- Secondary constructors that delegate with omitted arguments may skip `addToRegistry`, `validate`, `start`, or `postConstruct` calls.
|
||||||
|
- Factory method with two branches (e.g. from-bytes vs from-memory): does each branch apply the same post-creation setup?
|
||||||
|
|
||||||
|
### Field silently overwritten by hardcoded initializer / setter never called (weight: medium)
|
||||||
|
- Does a field's inline initializer assign a constant that is never overwritten in the constructor body — so the caller's constructor argument is discarded?
|
||||||
|
- Does a builder declare an option but the terminal method never read it (parallel execution, custom handler, etc.)?
|
||||||
|
- Does a result builder compute a field but forget to call `setXxx` before `build()`?
|
||||||
|
|
||||||
|
### equals / hashCode asymmetry with cache / identity fields (weight: medium)
|
||||||
|
- Does `equals` omit a field that determines the cached value, so distinct configurations collide?
|
||||||
|
- Does `equals` identify objects by display name only, omitting the stable unique identifier?
|
||||||
|
- Does `equals` delegate to a type-specific comparator without a type guard, throwing on mismatched operand types?
|
||||||
|
- Does `equals()` call `Objects.equal(this, x)` or a reference-equality utility, creating identity comparison?
|
||||||
|
|
||||||
|
### New field default-value / first-read visibility (weight: low)
|
||||||
|
- Does the newly added field have a non-null default?
|
||||||
|
- Can monitoring, `toString`, or a debug accessor read the field before startup assigns it?
|
||||||
|
- Is the field accessed on every construction path, or only on one?
|
||||||
|
|
@ -0,0 +1,253 @@
|
||||||
|
# Deep Boundary & I/O — Extended Checklist
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context Gathering (REQUIRED)
|
||||||
|
|
||||||
|
Before applying the checklist, read the TARGET FILES (not just the diff) to understand:
|
||||||
|
|
||||||
|
1. **Data ranges**: What are the valid ranges for indices, sizes, positions in this code?
|
||||||
|
2. **Sentinel values**: What does -1, 0, null, MAX_VALUE mean in this context?
|
||||||
|
3. **Buffer lifecycle**: Who allocates, positions, slices, and releases buffers?
|
||||||
|
4. **I/O contracts**: What does the underlying read/write API guarantee? Partial reads?
|
||||||
|
5. **Arithmetic units**: What units are sizes, timestamps, and positions in?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Off-by-One & Range Depth
|
||||||
|
|
||||||
|
### Time unit at API boundary (weight: high)
|
||||||
|
- Does a timestamp cross an API boundary in the wrong unit?
|
||||||
|
- Is a time value displayed or returned in a unit that differs from what the caller expects?
|
||||||
|
- Is the wrong `TimeUnit` enum argument passed?
|
||||||
|
|
||||||
|
### Boundary comparison (weight: high)
|
||||||
|
- State the boundary condition in English. Does the operator (`<` vs `<=`) match?
|
||||||
|
- Is an inclusive bound treated as exclusive (or vice versa)?
|
||||||
|
- Is the equality case at the boundary missed?
|
||||||
|
- Does a chunk-boundary alignment assertion fire for legitimate partial chunks?
|
||||||
|
|
||||||
|
### Integer overflow (weight: high)
|
||||||
|
- Does multiplying `int` config by 1024 or 1024*1024 overflow without long promotion?
|
||||||
|
- Does a `Comparator` use integer subtraction (overflow risk)?
|
||||||
|
- Does a narrowing cast on large values (file positions, byte counts) silently truncate?
|
||||||
|
|
||||||
|
### Comparator bugs (weight: medium)
|
||||||
|
- Does the comparator use `*.compare()` methods (not subtraction)?
|
||||||
|
- Does it handle equal elements correctly?
|
||||||
|
- Does it satisfy the transitivity contract?
|
||||||
|
- Are operands consistent in ordering across comparisons?
|
||||||
|
|
||||||
|
### Recursive iterator overflow (weight: medium)
|
||||||
|
- Does `computeNext()`, `hasNext()`, or a helper call itself recursively?
|
||||||
|
- Will it overflow the stack on large inputs?
|
||||||
|
|
||||||
|
### Recursive wrapping chain causes stack overflow (weight: low)
|
||||||
|
A defensive wrapper (e.g., `unmodifiableCollection`) re-applied on every rebuild without checking if the input is already wrapped. Chain depth grows with invocation count and overflows the stack during traversal.
|
||||||
|
|
||||||
|
### MB/GB to bytes overflow (weight: medium)
|
||||||
|
- When converting MB/GB to bytes: is at least one operand `long` before multiplication?
|
||||||
|
- Does the intermediate product overflow at 2 GB?
|
||||||
|
|
||||||
|
### Loop stop condition (weight: medium)
|
||||||
|
- Does the loop use `> limit` instead of `>= limit` (or vice versa)?
|
||||||
|
- Does it process one too many or one too few items?
|
||||||
|
|
||||||
|
### Buffer offset adjustment (weight: medium)
|
||||||
|
- Does `buffer.array()` account for `position()` and `arrayOffset()`?
|
||||||
|
- After `slice()`, is the non-zero offset accounted for?
|
||||||
|
|
||||||
|
### Binary search post-adjustment off-by-one (weight: low)
|
||||||
|
After binary search in a sorted range structure, adjusting the returned index by +1 for an exact match is wrong when the invariant requires the element to stay at the matched position.
|
||||||
|
|
||||||
|
### Binary search return value not adjusted for non-zero fromIndex base (weight: low)
|
||||||
|
When `binarySearch` accepts a non-zero `fromIndex`, a custom implementation may return a relative index while the caller expects absolute.
|
||||||
|
|
||||||
|
### Binary search with transposed forward / reverse sublist ranges (weight: low)
|
||||||
|
When a binary search is split into forward and reversed halves, are the correct sublist ranges passed to each direction? Transposed ranges silently skip valid matches.
|
||||||
|
|
||||||
|
### Int intermediate in file-position arithmetic overflows (weight: low)
|
||||||
|
When an array or buffer offset is computed from a multi-gigabyte file position, an early `(int)` cast silently truncates.
|
||||||
|
|
||||||
|
### For-each loop with manual counter incremented at body start is off-by-one (weight: low)
|
||||||
|
A manual counter incremented at the top of a for-each loop body is always one ahead of the current element when used as an array index.
|
||||||
|
|
||||||
|
### Compressed-section boundary at exact chunk multiple includes extra chunk (weight: low)
|
||||||
|
An endpoint that falls exactly on a chunk boundary does not require the next chunk; failing to check this sends one extra chunk.
|
||||||
|
|
||||||
|
### Accumulated counter with constant increment exceeds valid input domain (weight: low)
|
||||||
|
The accumulated value can drift past the boundary of the valid input range.
|
||||||
|
|
||||||
|
### Loop intended to check one beyond interval uses wrong comparator (weight: low)
|
||||||
|
A scan loop meant to "check one extra entry beyond the interval" silently skips that entry when the boundary uses `<` instead of `<=`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Null & Bounds Check Depth
|
||||||
|
|
||||||
|
### Map / registry lookup null (weight: high)
|
||||||
|
- Is the result of a map or registry lookup null-checked before use?
|
||||||
|
- Can a background task race with a concurrent removal of the entry it just looked up?
|
||||||
|
- Do paths that accept user-supplied keys null-guard the lookup result?
|
||||||
|
|
||||||
|
### Collection/array bounds (weight: high)
|
||||||
|
- `list.get(0)` without `isEmpty()`?
|
||||||
|
- `array[idx]` without length check?
|
||||||
|
- `Optional.get()` without `isPresent()`?
|
||||||
|
- Parallel-list length mismatch?
|
||||||
|
|
||||||
|
### indexOf/substring without -1 check (weight: high)
|
||||||
|
- Is the return value of `indexOf` / `lastIndexOf` used as an offset into `substring(...)` without a `!= -1` guard?
|
||||||
|
- Does `split(...)` return a single element when the delimiter is absent, and does the caller assume a multi-element result?
|
||||||
|
|
||||||
|
### Nullable lifecycle fields (weight: high)
|
||||||
|
- Is a field initialized in `start()`/`setup()` rather than constructor?
|
||||||
|
- Is it dereferenced during shutdown or startup race?
|
||||||
|
- Is it accessed from a wrong construction path?
|
||||||
|
|
||||||
|
### Infinite loops (weight: high)
|
||||||
|
- Does a `while(true)` or `while(!done)` have an explicit escape path?
|
||||||
|
- Is there a timeout, maximum-retry count, or interrupt check?
|
||||||
|
- Can the condition EVER become true given possible upstream stalls?
|
||||||
|
|
||||||
|
### Missing instanceof before cast (weight: high)
|
||||||
|
- Is every cast preceded by a matching `instanceof` guard?
|
||||||
|
- When a new implementation is added, does it pass existing type checks?
|
||||||
|
|
||||||
|
### Empty collection edge cases (weight: high)
|
||||||
|
- What happens with zero items? Does the code guard against empty inputs?
|
||||||
|
- Does empty credentials, config, or options cause a different (broken) code path?
|
||||||
|
|
||||||
|
### Empty or impossible range not detected before execution (weight: low)
|
||||||
|
A logically empty range (e.g., start > end, or bounds that can never overlap) is not detected before execution, causing wasted work or incorrect results.
|
||||||
|
|
||||||
|
### Nullable subtype-specific field accessed without subtype check (weight: low)
|
||||||
|
A field populated only for certain subtypes is accessed without checking the subtype, causing NPE.
|
||||||
|
|
||||||
|
### Null-accumulator first-encounter guard skips initialization (weight: low)
|
||||||
|
Guards of the form `if (field != null && ...)` incorrectly skip the first-encounter case when the field starts as null.
|
||||||
|
|
||||||
|
### Claim / lock method returns null (weight: low)
|
||||||
|
A "mark for exclusive use" or "claim" method returns null meaning nothing available, but callers dereference directly.
|
||||||
|
|
||||||
|
### Constructor parameter assigned as null instead of passed value (weight: low)
|
||||||
|
`this.field = null` instead of `this.field = parameter` silently discards the configuration.
|
||||||
|
|
||||||
|
### Update validation rejects no-op changes due to missing backfill (weight: medium)
|
||||||
|
When an update enforces immutability, it must copy the existing value into the incoming object so a no-change update does not trigger a spurious "field was changed" error.
|
||||||
|
|
||||||
|
### Helper silently returns empty on invalid input (weight: low)
|
||||||
|
A helper logs a warning and returns an empty collection on invalid input; callers block waiting for work that was never submitted.
|
||||||
|
|
||||||
|
### File.listFiles() null (weight: medium)
|
||||||
|
- Is `File.listFiles()` result checked for null before iteration?
|
||||||
|
- Is it called on a non-directory path?
|
||||||
|
|
||||||
|
### File.length() returns 0 for directories (weight: low)
|
||||||
|
When computing disk usage by summing `file.length()`, directories return 0 instead of recursive content size.
|
||||||
|
|
||||||
|
### poll() before processing prevents retry on exception (weight: low)
|
||||||
|
In a cleanup loop, `poll()` removes an item before processing finishes; any exception leaves it unprocessed.
|
||||||
|
|
||||||
|
### Division by zero (weight: low)
|
||||||
|
- Is a runtime-derived divisor (e.g., `collection.size()`) checked for zero?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ByteBuffer & I/O Depth
|
||||||
|
|
||||||
|
### Position-advancing get (specific pattern)
|
||||||
|
- Does no-arg `ByteBuffer.get()` advance position inside a comparator or shared view?
|
||||||
|
- Is the buffer owned by this code, or is it a shared/read-only view?
|
||||||
|
|
||||||
|
### Shared buffer passed to helper without snapshot (weight: medium)
|
||||||
|
- Is a shared buffer handed to a helper whose relative reads advance position as a side effect?
|
||||||
|
- Is there a subsequent reader of the same buffer that will read from the wrong offset?
|
||||||
|
- Does a rewind-to-restore pattern race with concurrent readers of the same buffer?
|
||||||
|
|
||||||
|
### Write-side buffer capacity not checked before put (weight: medium)
|
||||||
|
- Does a write of a fixed-width trailer / marker / end-of-segment token precede a `remaining() >= N` check?
|
||||||
|
- Does an encode loop exit on input-consumed instead of output-flushed — so when output exceeds input, the tail is silently truncated?
|
||||||
|
- Does a decompressor variant that does not validate output bounds run on untrusted network data?
|
||||||
|
|
||||||
|
### Zero-remaining buffer not guarded before length-prefix read (weight: low)
|
||||||
|
Before reading a length prefix (or fixed-width header) from a buffer, is `remaining() >= headerSize` checked? A legitimately-empty component otherwise throws an underflow exception instead of being treated as "no content".
|
||||||
|
|
||||||
|
### rewind() instead of flip() after write (weight: low)
|
||||||
|
After populating a buffer for handoff to a reader, `flip()` sets limit = position then resets position to 0. Using `rewind()` instead resets position only, exposing uninitialized bytes beyond the written region.
|
||||||
|
|
||||||
|
### NIO read completeness (weight: low)
|
||||||
|
- Does `Channel.read()` assume a single call fills the entire buffer?
|
||||||
|
- Is there a loop until `remaining() == 0` or EOF?
|
||||||
|
|
||||||
|
### EOF handling (specific pattern)
|
||||||
|
- Does NIO `read()` distinguish -1 (EOF) from 0 (empty buffer)?
|
||||||
|
- Does `InputStream.read()` return 0 at end-of-stream instead of -1?
|
||||||
|
|
||||||
|
### Partial file left on exception during write (weight: low)
|
||||||
|
- Does an exception in the write path leave a half-written file without an abort / delete step? A subsequent read treats it as complete.
|
||||||
|
- Does writing to an existing file fail to truncate first, leaving stale bytes beyond the new content?
|
||||||
|
|
||||||
|
### Retry/backoff unbounded (weight: medium)
|
||||||
|
- Does exponential backoff have a ceiling?
|
||||||
|
- Can the lower bound grow past the upper bound?
|
||||||
|
|
||||||
|
### Sets.union lazy view (weight: low)
|
||||||
|
- Is `Sets.union` accumulated in a loop, creating chain depth = iteration count?
|
||||||
|
- Should the view be materialized into a concrete set?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Interval & Range Depth
|
||||||
|
|
||||||
|
### Interval endpoint operator (weight: low)
|
||||||
|
- At shared boundaries: is `< 0` vs `<= 0` consistent with open/closed convention?
|
||||||
|
- Does the documented convention match the implementation?
|
||||||
|
|
||||||
|
### Half-open to closed conversion (weight: low)
|
||||||
|
- When wrapping `[a, b)` API for `[a, b]` range: is upper bound shifted by +1?
|
||||||
|
|
||||||
|
### Range bound partial enforcement (weight: low)
|
||||||
|
- Is start boundary enforced but finish boundary ignored (or vice versa)?
|
||||||
|
|
||||||
|
### Wrap-around range causes infinite linear scan (weight: low)
|
||||||
|
In a backward or reverse iteration, does the loop guard use strict `<` (or `>`) against an endpoint that may lie on the wrong side of the start in a wrapping range?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Arithmetic Pitfalls
|
||||||
|
|
||||||
|
### Missing parentheses (weight: low)
|
||||||
|
- Does `n + 1 / 2` lack parentheses, evaluating as `n + 0`?
|
||||||
|
|
||||||
|
### Floating-point loop drift (weight: low)
|
||||||
|
- Does a float/double accumulator loop from 0.0 to 1.0 miss the final value?
|
||||||
|
|
||||||
|
### Ceiling division (specific pattern)
|
||||||
|
- Is `(n + d - 1) / d` used correctly for ceiling division?
|
||||||
|
- Is `round-down-to-multiple` producing zero when value < N?
|
||||||
|
|
||||||
|
### Capacity vs read bounds conflated (weight: low)
|
||||||
|
- Is buffer capacity treated as exclusive for both reads AND seeks?
|
||||||
|
- Should seek-to-end (`<= capacity`) be allowed while read-at-end is not?
|
||||||
|
|
||||||
|
### Compressed vs uncompressed size (weight: medium)
|
||||||
|
- Is the compressed or uncompressed size used for file-split decisions?
|
||||||
|
- Are progress numerator and denominator using the same size metric?
|
||||||
|
|
||||||
|
### nanoTime deadline addition overflows (weight: low)
|
||||||
|
Computing a deadline as `nanoTime() + duration` overflows near `Long.MAX_VALUE`; the safe pattern is `nanoTime() - start < duration`.
|
||||||
|
|
||||||
|
### Arithmetic on Integer.MAX_VALUE sentinel overflows without guard (weight: low)
|
||||||
|
Any `limit + 1` pattern where `limit` may be `Integer.MAX_VALUE` silently wraps to negative.
|
||||||
|
|
||||||
|
### Probabilistic gate does not short-circuit on boundary values (weight: low)
|
||||||
|
A probability check always calls the RNG even when configured to 0.0 or 1.0.
|
||||||
|
|
||||||
|
### Expiry / deadline computed from raw sentinel without substitution (weight: low)
|
||||||
|
- Is a configuration value like `retention = -1` (meaning "infinite") passed as-is to `now + duration`, producing an already-expired instant?
|
||||||
|
- Is an epoch-offset constant defined as negative and then subtracted (double-negation) instead of added?
|
||||||
|
|
||||||
|
### Serialized size miscounted against actual write (weight: medium)
|
||||||
|
- Does the size calculation add a field's length prefix but forget the payload bytes?
|
||||||
|
- Does a conditional field get counted unconditionally AND again inside its branch — declared size > actual bytes written?
|
||||||
|
- Does the field order used by the size calculation diverge from the write path so a refactor changes one but not the other?
|
||||||
|
|
@ -0,0 +1,334 @@
|
||||||
|
# Deep Concurrency & State — Extended Checklist
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context Gathering (REQUIRED)
|
||||||
|
|
||||||
|
Before applying the checklist, read the TARGET FILES (not just the diff) to understand:
|
||||||
|
|
||||||
|
1. **Thread model**: Which threads access this object? Is it confined to one thread?
|
||||||
|
2. **Lock inventory**: What locks protect this state? What is the lock ordering?
|
||||||
|
3. **Lifecycle**: Who creates, starts, stops, and destroys this object?
|
||||||
|
4. **Shared state**: What fields are read/written from multiple threads?
|
||||||
|
5. **State machine**: What states can this object be in? What transitions are valid?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TOCTOU & Atomicity Depth
|
||||||
|
|
||||||
|
### TOCTOU on shared mutable state (weight: high)
|
||||||
|
- Is a field read in one step and acted upon in a separate unsynchronized step?
|
||||||
|
- Is a boolean flag checked then used without holding a lock across both operations?
|
||||||
|
- Is a state-machine check separate from the state transition?
|
||||||
|
- Is an existence check separate from the create/insert operation?
|
||||||
|
- Is a volatile field or concurrent collection read twice without local capture? Concurrent nullification between check and use.
|
||||||
|
|
||||||
|
### Non-atomic compound operations (weight: high)
|
||||||
|
- Are get-then-put, check-then-act operations on ConcurrentHashMap done without external lock?
|
||||||
|
- Are operations across multiple concurrent maps done without a shared lock?
|
||||||
|
- Is `putIfAbsent` return value ignored, with caller using its own argument?
|
||||||
|
|
||||||
|
### Producer-consumer aliasing (weight: medium)
|
||||||
|
- Does the producer retain a reference to an object after handing it off via a queue?
|
||||||
|
- After `queue.put(obj)`, does the producer continue modifying `obj`?
|
||||||
|
|
||||||
|
### Atomic swap with concurrent writers still holding old instance (weight: low)
|
||||||
|
- After `AtomicReference.getAndSet(newCollector)`, is the old collector drained on a path that can race with writers who already resolved to the old reference?
|
||||||
|
- Does a "replace landmark" / "replace accumulator" pattern leave concurrent updaters writing to the retired instance, dropping or duplicating entries?
|
||||||
|
|
||||||
|
### Snapshot-under-lock defeated by downstream unsynchronized re-read (weight: low)
|
||||||
|
- Is a shared field captured into a local under lock, but a subsequent call in the same critical section re-reads the original unsynchronized reference (e.g., `this.field.foo()` instead of `local.foo()`)?
|
||||||
|
- Does a helper called with the local variable internally dereference the owning object's field directly?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Collection Concurrency Depth
|
||||||
|
|
||||||
|
### ConcurrentModificationException (weight: high)
|
||||||
|
- Is a for-each loop iterating a shared mutable collection while another thread modifies it?
|
||||||
|
- Is `collection.remove()` called inside a for-each over the same collection?
|
||||||
|
- Is a method inside the loop body modifying the same collection being iterated?
|
||||||
|
|
||||||
|
### Unsynchronized reads of shared collections (weight: high)
|
||||||
|
- Is a shared `HashMap`/`ArrayList` written under synchronization but read without?
|
||||||
|
- Is `.keySet()`, `.entrySet()`, or `.values()` returned as a live view without copying?
|
||||||
|
- Is the collection iterated outside the lock that protects it?
|
||||||
|
|
||||||
|
### Live view iteration (specific pattern)
|
||||||
|
- Does this code iterate `map.entrySet()` or `list.subList()` without copying first?
|
||||||
|
- Does a getter return internal state (Map, Set, List field) that's iterated outside the lock?
|
||||||
|
|
||||||
|
### Shared ByteBuffer position mutated as read side-effect (weight: high)
|
||||||
|
- Is a shared or pooled `ByteBuffer` read via relative `get()` / relative-get helpers without calling `duplicate()` or `slice()` first?
|
||||||
|
- Does a deserialization helper return a reference to a static/singleton buffer rather than a fresh copy, letting callers permanently exhaust it?
|
||||||
|
- Does a comparator or serializer advance position on a caller-supplied buffer and then try to rewind/flip, leaving concurrent observers exposed?
|
||||||
|
- Is a heap ByteBuffer sliced but `arrayOffset()` omitted when indexing into the backing array?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Visibility & Memory Model Depth
|
||||||
|
|
||||||
|
### Missing volatile (weight: medium)
|
||||||
|
- Is a non-final, non-volatile shared field written on one thread and read on another?
|
||||||
|
- Is a lazily-initialized cached field shared across threads without volatile?
|
||||||
|
- Is a mutable singleton reference shared without visibility guarantee?
|
||||||
|
|
||||||
|
### Signal before publish (weight: medium)
|
||||||
|
- Does a latch countDown or future complete fire BEFORE the data structures woken threads will read are fully updated?
|
||||||
|
- Is the data published with a release fence (volatile write, synchronized exit)?
|
||||||
|
|
||||||
|
### Comparator reads live state (weight: medium)
|
||||||
|
- Does a Comparator read from a volatile, shared, or continuously-updating field during sort?
|
||||||
|
- Could concurrent writes change values mid-sort, violating transitivity?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Lifecycle & Initialization Depth
|
||||||
|
|
||||||
|
### Constructor publishes this (weight: medium)
|
||||||
|
- Does the constructor spawn a thread capturing `this`?
|
||||||
|
- Does the constructor register a listener, management callback, or observer with `this`?
|
||||||
|
- Does the constructor start an executor that references `this`?
|
||||||
|
|
||||||
|
### Double-checked locking (weight: medium)
|
||||||
|
- Is the field re-read after leaving the synchronized block?
|
||||||
|
- Are null-check and value-read separate field accesses?
|
||||||
|
- Could another thread observe an intermediate state?
|
||||||
|
|
||||||
|
### Listener registration timing (weight: medium)
|
||||||
|
- Is an event listener registered AFTER an initial state snapshot is taken?
|
||||||
|
- Is a listener registered AFTER the async operation is initiated?
|
||||||
|
- Is there a window where events between snapshot and registration are missed?
|
||||||
|
|
||||||
|
### Shutdown ordering (weight: high)
|
||||||
|
- Does shutdown mirror startup in reverse order?
|
||||||
|
- Is a flag set before the guarded operation completes?
|
||||||
|
- Do static initializers fire too early?
|
||||||
|
- Do offline tools assume a live cluster?
|
||||||
|
|
||||||
|
### Non-blocking stop signal followed by destructive state change (weight: medium)
|
||||||
|
- Does shutdown send a "stop" flag/signal to a background task and then proceed to clear, truncate, or delete the state the task reads/writes without joining, awaiting, or confirming the task exited?
|
||||||
|
- Does a feature-disable path unregister or null-out collaborators while periodic tasks are still mid-iteration?
|
||||||
|
|
||||||
|
### Reference count not acquired before concurrent read (weight: high)
|
||||||
|
- Does a reader access a shared reference-counted resource without first acquiring a reference?
|
||||||
|
- Can a concurrent evictor / deleter / compactor free the resource between the null / existence check and the read, causing use-after-free?
|
||||||
|
- Is the refcount increment combined with a CAS such that a losing participant leaves the refcount inflated?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Counter & Accounting Depth
|
||||||
|
|
||||||
|
### Increment without matching decrement (weight: medium)
|
||||||
|
- Is a counter incremented before an operation that can fail?
|
||||||
|
- Is the rollback path present on every failure branch?
|
||||||
|
- Is the counter decremented before associated cleanup finishes?
|
||||||
|
|
||||||
|
### Metric at wrong point (weight: high)
|
||||||
|
- Is a metric updated before the operation it measures (early-exit paths inflate)?
|
||||||
|
- Is a metric updated after a different operation (measures wrong thing)?
|
||||||
|
- Is a metric outside the guard that controls the measured operation?
|
||||||
|
- Is the metric in an estimation loop that diverges from the actual selection loop?
|
||||||
|
|
||||||
|
### Size/count accumulator gaps (pattern)
|
||||||
|
- Is a size accumulator updated in SOME write paths but not ALL?
|
||||||
|
- Do tombstones, metadata, and index blocks all contribute to the accumulator?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## State Machine Depth
|
||||||
|
|
||||||
|
### Missing side-effect in branch (weight: high)
|
||||||
|
- List ALL branches of the state-machine handler. Does every branch perform the required side-effect?
|
||||||
|
- Specifically check: gossip sync, metric update, status propagation, transport stop.
|
||||||
|
|
||||||
|
### State cleanup (weight: high)
|
||||||
|
- Does reset/clear/truncate update EVERY companion structure?
|
||||||
|
- Does lazy-init `if (x == null) x = init()` conflate "not yet initialized" with "already closed"?
|
||||||
|
- Does a success path miss cleanup that the error path has?
|
||||||
|
- Does persistent state get written when operation starts rather than commits?
|
||||||
|
- Does a periodic task short-circuit on `!enabled` without retracting published state?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deadlock Depth
|
||||||
|
|
||||||
|
### Blocking get in handler (weight: low)
|
||||||
|
- Does `future.get()` inside an RPC verb handler starve the handler thread pool?
|
||||||
|
- Does the completion of the awaited future depend on the same pool?
|
||||||
|
|
||||||
|
### Lock ordering (specific patterns)
|
||||||
|
- Does a locked section delegate to a public API that independently acquires the same lock?
|
||||||
|
- Does a `synchronized` method call into Netty/IO callback that re-enters the same lock?
|
||||||
|
- Does `static synchronized` call code that loads another class with its own static init?
|
||||||
|
|
||||||
|
### Thread pool starvation (specific patterns)
|
||||||
|
- Does `SynchronousQueue` with `CallerRunsPolicy` block the submitter?
|
||||||
|
- Does a bounded blocking queue in Netty pipeline block I/O threads needed to drain it?
|
||||||
|
|
||||||
|
### Missing lock on shared metadata mutation (weight: low)
|
||||||
|
- Does a metadata-change path skip a flush or compaction lock held by concurrent background tasks?
|
||||||
|
- Can the background task write files, index entries, or offsets against a now-invalidated metadata object?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Scope & Guard Mismatch Depth
|
||||||
|
|
||||||
|
### Scope mismatch (weight: high)
|
||||||
|
- Is a teardown step outside the `if` block of the operation it relates to?
|
||||||
|
- Is a notification unconditional when it should be inside a conditional?
|
||||||
|
- Is an invariant guard inside an optional branch instead of enclosing scope?
|
||||||
|
- Is validation logic in one path (local-apply) but not another (remote-announce)?
|
||||||
|
|
||||||
|
### Conditional decrement chain (pattern)
|
||||||
|
- Does a conditional decrement chain across multiple if/else branches miscount?
|
||||||
|
- Is a dirty flag set after a write loop instead of inside it?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Thread Pool & Dispatch Depth
|
||||||
|
|
||||||
|
### Wrong executor stage (weight: medium)
|
||||||
|
- Is a message handler, task, or state mutation dispatched to the wrong executor stage?
|
||||||
|
- Does it run outside its thread-confinement contract?
|
||||||
|
|
||||||
|
### Counter scope (weight: medium)
|
||||||
|
- Is a counter increment outside the conditional brace, firing unconditionally?
|
||||||
|
- Is it supposed to count only matching items but counts every iteration?
|
||||||
|
|
||||||
|
### Self-dispatch on confined / single-threaded executor (weight: medium)
|
||||||
|
- Does code already running on a single-threaded executor (or thread-confined actor) submit new work to that same executor and block on the result?
|
||||||
|
- Does a coordinator-and-replica handler that runs on one executor dispatch back to the same executor for the local replica?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Reentrancy & Async Hazards
|
||||||
|
|
||||||
|
### Reentrancy (weight: low)
|
||||||
|
- Does a method holding a lock call back into code that modifies the same shared state?
|
||||||
|
- Is this common in cancellation and eviction paths?
|
||||||
|
|
||||||
|
### ThreadLocal across async boundary (weight: low)
|
||||||
|
- Is thread-local state (tracing, request context) set on submitter but read in async task?
|
||||||
|
- Could the submitter clear the thread-local before the task reads it?
|
||||||
|
|
||||||
|
### CAS retry safety (weight: low)
|
||||||
|
- Is an unbounded CAS retry loop used where contention is not brief?
|
||||||
|
- Does a boolean CAS elect a single writer while others continue with stale state?
|
||||||
|
|
||||||
|
### Interrupt mid-execution corrupts shared I/O channel (weight: low)
|
||||||
|
- Can a task be interrupted while holding or using a shared network channel?
|
||||||
|
- Does a subsequent blocking call on the shared channel throw `ClosedByInterruptException`, propagating the interrupt beyond the interrupted task's scope?
|
||||||
|
- Should the channel be dedicated per-task or the interrupt absorbed locally?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Mutability and Immutability Violations
|
||||||
|
|
||||||
|
### Immutable collection returned where caller expects mutable (weight: low)
|
||||||
|
A method returns `Arrays.asList()`, `Collections.singletonList()`, or an unmodifiable list but the caller calls `add()` or `remove()` on it, throwing `UnsupportedOperationException`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Specific Concurrency Patterns
|
||||||
|
|
||||||
|
### Version-gated conditional field read or written in only one branch (weight: high)
|
||||||
|
A conditional field guarded by message variant/type/status appears in only one branch of serialize or deserialize, serializedSize early-return bypasses shared trailing field, protocol version not threaded through to collection element decoder, flag consumed before conditional early-return, or version-gated field written redundantly without matching deserialize logic.
|
||||||
|
|
||||||
|
### Configuration option silently dropped: null passed where config should flow (weight: high)
|
||||||
|
A config knob is parsed but not threaded to the builder: cipher suites passed as null, streaming encryption settings dropped, max frame size not set.
|
||||||
|
|
||||||
|
### Throwable.getMessage(), getCause(), or empty stack trace used without null check (weight: medium)
|
||||||
|
`getMessage()` returns null for no-arg constructors; `getCause()` is null with no chained cause; stack trace array has zero elements with `writableStackTrace=false`.
|
||||||
|
|
||||||
|
### Copy-paste error: duplicate column name, duplicate code block (weight: medium)
|
||||||
|
A copy-pasted block references the same variable, column name, or literal twice instead of the intended distinct values.
|
||||||
|
|
||||||
|
### Atomic Counter and Collection Out of Sync (weight: medium)
|
||||||
|
A liveness or completion check uses an atomic counter while the actual data lives in a separate collection; the counter advances ahead due to CPU reordering.
|
||||||
|
|
||||||
|
### Race Between Registration and Resource Readiness (weight: medium)
|
||||||
|
A resource tracking set registration or availability marker is set before or after the resource is actually ready.
|
||||||
|
|
||||||
|
### Exception Handler Catch Type Mismatch Bypasses Cleanup (weight: medium)
|
||||||
|
A try-block allocates a pooled resource and the catch block only handles one exception type; exceptions thrown as a different type escape the catch entirely.
|
||||||
|
|
||||||
|
### Getter method mutates state (weight: low)
|
||||||
|
A method named `getCompletedTasks()` calls `incrementAndGet()` on an atomic counter instead of `get()`.
|
||||||
|
|
||||||
|
### Thread-Local Used for Instance-Scoped State (weight: low)
|
||||||
|
A ThreadLocal caches a value that logically belongs to a specific object instance; when multiple instances share threads, the cached value cross-contaminates.
|
||||||
|
|
||||||
|
### Shared Cursor / Traversal State at Instance Level (weight: low)
|
||||||
|
A class holds a mutable traversal cursor as an instance field; concurrent callers corrupt each other's state.
|
||||||
|
|
||||||
|
### Decrement Before Cleanup Completes (weight: low)
|
||||||
|
A resource-admission counter is decremented before associated cleanup (socket close, thread exit) finishes.
|
||||||
|
|
||||||
|
### putIfAbsent Return Value Ignored (weight: low)
|
||||||
|
`putIfAbsent` called without checking its return value; subsequent code operates on newly created object instead of the winner.
|
||||||
|
|
||||||
|
### Idempotency Not Handled in Retry / Re-Entry Path (weight: low)
|
||||||
|
An operation that can be retried does not distinguish where idempotency is required from where a duplicate indicates a bug.
|
||||||
|
|
||||||
|
### Global / Static Flag Toggled in Test Without finally Reset (weight: low)
|
||||||
|
A global flag toggled before assertions is reset at end of try body instead of `finally`; assertion failures leave it dirty.
|
||||||
|
|
||||||
|
### Singleton Response Object Reused Across Concurrent Requests (weight: low)
|
||||||
|
A protocol message class carrying per-request mutable fields is exposed as a static singleton.
|
||||||
|
|
||||||
|
### Concurrent Directory Creation Race (weight: low)
|
||||||
|
`mkdirs()` result drives an error path without considering a concurrent creator could race.
|
||||||
|
|
||||||
|
### Lock Acquired But Critical Work Done Between lock() and try (weight: low)
|
||||||
|
Mutations are placed between `lock.lock()` and the paired `try` block, running outside the critical section.
|
||||||
|
|
||||||
|
### finally Block Throws, Suppressing Cleanup of Subsequent Resources (weight: low)
|
||||||
|
A call that can throw inside `finally` suppresses the original exception and prevents subsequent cleanup.
|
||||||
|
|
||||||
|
### Losing Race Participant Does Not Dispose Resource (weight: low)
|
||||||
|
An object holding a resource is constructed before putIfAbsent; the loser does not dispose.
|
||||||
|
|
||||||
|
### Cache-line contention in array of independently-updated counters (weight: low)
|
||||||
|
Adjacent atomic counters suffer false sharing. Check for padding or striped design.
|
||||||
|
|
||||||
|
### Fire-and-forget async operation with no in-flight deduplication (weight: low)
|
||||||
|
An async fetch submitted to an executor is not tracked; a second caller triggers a parallel operation.
|
||||||
|
|
||||||
|
### Broadcast invalidation causes thundering-herd recomputation (weight: low)
|
||||||
|
A cache invalidation broadcast causes all holders to race to repopulate concurrently.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## State Cleanup — Concurrency-Domain Patterns
|
||||||
|
|
||||||
|
### Coupled Data Structures Only Partially Reset / Cleared (weight: high)
|
||||||
|
When "reset", "clear", or "truncate" touches one of several coupled structures, some are missed — derived counters, companion maps, version trackers, or cached unions.
|
||||||
|
|
||||||
|
### Merge or library-upgrade breaks interface method signature (weight: high)
|
||||||
|
After a merge, one or more implementing classes retain the old signature; without `@Override`, the mismatch is silent.
|
||||||
|
|
||||||
|
### Success Path Missing Cleanup That Error Path Has (weight: medium)
|
||||||
|
Cleanup logic exists in the error handler but is missing from the normal completion path.
|
||||||
|
|
||||||
|
### Stale Snapshot Used for Deletion / Cleanup Logic (weight: medium)
|
||||||
|
Code that computes "what changed" by diffing snapshots misses changes when a snapshot is skipped.
|
||||||
|
|
||||||
|
### Virtual method called from base-class constructor returns stale default (weight: low)
|
||||||
|
An overridable method called from the base-class constructor returns a default because the subclass hasn't finished initialization.
|
||||||
|
|
||||||
|
### Open Marker / Cursor Cleared Between Segments (weight: low)
|
||||||
|
An iterator state machine resets "currently open" state as side-effect of reading it, losing information spanning iteration boundaries.
|
||||||
|
|
||||||
|
### Wrapper class enable/disable lifecycle not propagated to inner delegates (weight: low)
|
||||||
|
State changes only affect the wrapper's flag, not the delegates.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Guard removal unsafe for mixed-mode rolling upgrade (weight: medium)
|
||||||
|
A guard is removed, but older nodes in a rolling upgrade still depend on the guarded behavior.
|
||||||
|
|
||||||
|
### Clear-and-refill on shared collection creates race window (weight: low)
|
||||||
|
A shared collection is cleared then refilled; concurrent readers between clear and refill see empty state.
|
||||||
|
|
||||||
|
### Constructor parameter accepted but never stored or used (weight: low)
|
||||||
|
A constructor accepts a parameter that is never assigned to a field.
|
||||||
|
|
@ -0,0 +1,170 @@
|
||||||
|
# Deep Cross-Cutting Patterns — Extended Checklist
|
||||||
|
---
|
||||||
|
|
||||||
|
## Refactoring & Merge Artifacts
|
||||||
|
|
||||||
|
### Refactored signature not updated (weight: high)
|
||||||
|
- After method parameter/name/type change: do ALL callers (across all modules) use the new form?
|
||||||
|
- Are callers outside the default build target checked?
|
||||||
|
|
||||||
|
### JDK version incompatibility (weight: high)
|
||||||
|
- Are `--add-opens` declarations present for module access?
|
||||||
|
- Are reflective field names stable across JDK versions?
|
||||||
|
- Is a newer JDK API used in code targeting an older JDK?
|
||||||
|
|
||||||
|
### Merge result drops original input from aggregation (weight: low)
|
||||||
|
A method that collects "extra" entries from triggers or plugins and merges them omits the original input from the result, silently discarding it.
|
||||||
|
|
||||||
|
### Wrapper type introduced but raw comparisons remain (weight: low)
|
||||||
|
After wrapping a primitive type in a richer wrapper, leftover code still uses the raw type for comparisons.
|
||||||
|
|
||||||
|
### Map.put in aggregation loop silently drops prior values (weight: low)
|
||||||
|
Aggregation code uses `put` instead of merge/compute, dropping all but the last contribution from each key.
|
||||||
|
|
||||||
|
### Refactored guard condition moved to wrong scope (weight: low)
|
||||||
|
Moving a guard during refactoring places it in a utility method that fires for all callers, not just the intended one.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Shell Scripts & Platform
|
||||||
|
|
||||||
|
### Version comparison (weight: high)
|
||||||
|
- Is version comparison lexicographic instead of numeric?
|
||||||
|
- Does it handle SNAPSHOT/pre-release labels?
|
||||||
|
|
||||||
|
### Locale/platform (weight: high)
|
||||||
|
- Does code assume English locale output?
|
||||||
|
- Does it assume POSIX file-sharing semantics?
|
||||||
|
|
||||||
|
### Library contract change (weight: medium)
|
||||||
|
- Has a dependency version change silently changed API behavior?
|
||||||
|
- Does a JAR version mismatch exist between classpath and packaging?
|
||||||
|
|
||||||
|
### Regex metacharacter (weight: low)
|
||||||
|
- Is `.` unescaped in `String.split()` or `grep`?
|
||||||
|
- Is `|`, `*`, `+`, `^` literal but used as regex?
|
||||||
|
|
||||||
|
### Mutable cached / derived value diverges from authoritative source (weight: high)
|
||||||
|
A shadow static field or cached derived value is updated independently from the authoritative source, causing the two to silently diverge after any write to either side.
|
||||||
|
|
||||||
|
### Shell argument forwarding splits on spaces (weight: low)
|
||||||
|
Bare `$@` without double-quoting splits arguments containing spaces.
|
||||||
|
|
||||||
|
### Regex pre-processor does not respect quoting/escaping context (weight: low)
|
||||||
|
A regex-based pre-processor that operates without awareness of string-literal quoting silently corrupts valid user data.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Missing Method Overrides
|
||||||
|
|
||||||
|
### Subclass overwrites parent field (weight: low)
|
||||||
|
- Does a subclass constructor re-initialize a parent-set field back to default?
|
||||||
|
- Does the parent's computed value get lost?
|
||||||
|
|
||||||
|
### Copy method missing (weight: low)
|
||||||
|
- When a subclass adds fields: does it override `sharedCopy()`/`clone()`?
|
||||||
|
- Does the copy silently lose subclass state?
|
||||||
|
|
||||||
|
### Self-referential equals (weight: low)
|
||||||
|
- Does `equals()` delegate to `Objects.equal(this, x)` creating identity comparison?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Crash Safety & Atomic Writes
|
||||||
|
|
||||||
|
### Files.copy idempotency (weight: low)
|
||||||
|
- Does `Files.copy()` use `REPLACE_EXISTING` if it can run twice?
|
||||||
|
|
||||||
|
### Write without truncation (weight: low)
|
||||||
|
- Does opening a file for write without `TRUNCATE_EXISTING` leave stale bytes beyond the new content?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Output Formatting & CLI
|
||||||
|
|
||||||
|
### CLI fixed-width format strings break on long field values (weight: medium)
|
||||||
|
Hardcoded fixed-width format strings in tools truncate or misalign output for long values.
|
||||||
|
|
||||||
|
### Column access by position instead of by name in result set iteration (weight: medium)
|
||||||
|
Code accesses result columns by position or `values()` iteration. Server-side column reordering silently breaks access.
|
||||||
|
|
||||||
|
### Boolean parser treats absence as false without validating value (weight: low)
|
||||||
|
- Does the parser only check the positive branch, treating everything else as false?
|
||||||
|
- Is garbage input silently accepted?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Versioning & Compatibility
|
||||||
|
|
||||||
|
### Protocol-version field sent unconditionally (weight: medium)
|
||||||
|
- Does a serializer write a field unconditionally even when the negotiated protocol version does not contain it? Recipients misparse subsequent fields.
|
||||||
|
- Does a method assert the messaging version equals the local current version, throwing in any mixed-version cluster?
|
||||||
|
|
||||||
|
### Versioned deserialization missing defaults for absent optional fields (weight: medium)
|
||||||
|
When deserializing from an older node that omits optional fields, nullable fields are not replaced with safe defaults before passing to non-null constructors.
|
||||||
|
|
||||||
|
### Validation rule correct for user input but wrong for internal reconstruction (weight: low)
|
||||||
|
A validation rule rejects valid internally-stored data during reconstruction.
|
||||||
|
|
||||||
|
### Reflection invoke() passes null instead of empty array for no-arg method (weight: low)
|
||||||
|
`method.invoke(null, (Object) null)` passes null as a single argument instead of an empty array.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Filesystem Path & Identity
|
||||||
|
|
||||||
|
### Shell script argument quoting / word-splitting vulnerability (weight: medium)
|
||||||
|
A shell script stores arguments in a single unquoted variable, uses an unfiltered glob, or embeds multi-line command substitution into a path variable.
|
||||||
|
|
||||||
|
### Filesystem path or string identity mismatch (weight: medium)
|
||||||
|
`String.startsWith()` used for path containment causing prefix false positives, or a set lookup misses entries with suffixes (UUIDs, extensions) appended to the stored key.
|
||||||
|
|
||||||
|
### Off-heap class missing override of size/length query (weight: low)
|
||||||
|
When a class stores data off-heap and overrides data accessors but not size-query methods, the base class falls back to heap-materializing data on the hot path.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Additional Patterns from Multi-Project Corpus
|
||||||
|
|
||||||
|
### Serializable class field / transient without custom readObject (weight: low)
|
||||||
|
- Does a `Serializable` class hold a field typed as a non-serializable third-party container?
|
||||||
|
- Is a field marked `transient` without a `readObject` method to reconstruct it? After deserialization the field is null and crashes on first access.
|
||||||
|
|
||||||
|
### Wrapper / delegate drops auxiliary argument (weight: medium)
|
||||||
|
- When a wrapper serializer / deserializer / client forwards to an inner instance, does it propagate every argument — tracing context, consistency level, auth credentials, client options?
|
||||||
|
- Does the wrapper fall back to a subset overload, silently dropping the auxiliary parameter?
|
||||||
|
- For sibling clients / serializers: is the propagation step applied to each, or only the primary?
|
||||||
|
|
||||||
|
### Duplicate-code versioned directories drift out of sync (weight: low)
|
||||||
|
- When a bug-fix is applied to one versioned copy of a subsystem (format-variant, protocol-version-gated implementation, legacy vs current): is it propagated to every sibling copy?
|
||||||
|
- Grep for the fixed function name across peer directories (legacy/, current/, format variants) to find stale copies.
|
||||||
|
|
||||||
|
### Lookup key transformed asymmetrically between insert and get (weight: medium)
|
||||||
|
- Is the map populated with a post-transform (normalized / lowercased / resolved) key while the lookup uses the pre-transform key (or vice versa)?
|
||||||
|
- For case-insensitive lookups: do both `put` and `get` normalize case? For address translation: is the same mapping applied before insert and before lookup?
|
||||||
|
|
||||||
|
### Dependency-upgrade API incompatibility or silent deprecation (weight: medium)
|
||||||
|
- After a dependency version bump: do fluent / chained API calls exist on the host's runtime version?
|
||||||
|
- Do configuration methods called on the upgraded client still apply, or have they silently become no-ops returning stub values?
|
||||||
|
- Does a plugin-version regression change published artifact classifiers?
|
||||||
|
|
||||||
|
### serializedSize counts length prefix only, omitting field bytes (weight: low)
|
||||||
|
- Does `serializedSize` for a variable-length field count only the length prefix without adding the actual byte count?
|
||||||
|
- Compare serializedSize field-by-field against serialize: does each field contribute X in serialize but Y in size?
|
||||||
|
|
||||||
|
### Guard / validation present at high-level entry but bypassed by lower-level factory path (weight: medium)
|
||||||
|
- If validation exists on the public API entry point: is every lower-level factory / internal constructor also gated, or can callers reach the state-mutation via a back-door that skips validation?
|
||||||
|
- Does one alternative creation path (from-bytes vs from-memory; primary vs copy-of) apply a required post-setup step the other omits?
|
||||||
|
|
||||||
|
### New error code / exception class unmapped in error-handling table (weight: medium)
|
||||||
|
- When a new error code or exception type is introduced: is the error-handling mapping table updated (separately from the dispatch switch)?
|
||||||
|
- Does an exception class incorrectly inherit a retriable parent, causing infinite retry loops on non-recoverable cases?
|
||||||
|
- Does treating a transient-state error code as success silently skip retry instead of triggering state rediscovery?
|
||||||
|
|
||||||
|
### Fat-artifact missing class / optional dependency NoClassDefFoundError (weight: medium)
|
||||||
|
- When a feature depends on a class that is compile-time-present but may be omitted from the deployed fat artifact: is there a runtime fallback or explicit classpath declaration?
|
||||||
|
- Is an optional module / codec excluded from the packaged assembly but referenced from the startup path?
|
||||||
|
|
||||||
|
### Stub returning null instead of delegating to backing field (weight: medium)
|
||||||
|
- Does a DTO / interface-impl stub return `null` / `0` / `false` from a method that should delegate to a field populated by the builder?
|
||||||
|
- Does a predicate in an optimizing variant of a paired evaluator unconditionally return the conservative result, silently skipping the optimization for every input?
|
||||||
|
|
@ -0,0 +1,373 @@
|
||||||
|
# Deep Logic & Types — Extended Checklist
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context Gathering (REQUIRED)
|
||||||
|
|
||||||
|
Before applying the checklist, read the TARGET FILES (not just the diff) to understand:
|
||||||
|
|
||||||
|
1. **Class hierarchy**: What does this class extend? What interfaces does it implement?
|
||||||
|
2. **Sibling classes**: Are there parallel implementations that should mirror this code?
|
||||||
|
3. **Callers**: Who calls the methods being changed? How do they use the return values?
|
||||||
|
4. **Lifecycle**: When is this object created, used, and destroyed?
|
||||||
|
5. **Invariants**: What contracts does this class maintain? Read Javadoc, assertions, tests.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Condition & Comparison
|
||||||
|
|
||||||
|
### Type-mismatch in equals/contains (weight: high)
|
||||||
|
- For every `equals()` or `contains()` call: do BOTH sides have the same runtime type? Check for subtype vs supertype, wrapper vs unwrapped form, domain object vs its key.
|
||||||
|
- For map lookups: does the key type match the map's key type exactly? A type mismatch causes a silent lookup miss.
|
||||||
|
- Does `Arrays.asList()` wrap an already-List, creating `List<List<T>>`? `contains()` then compares wrong element type.
|
||||||
|
|
||||||
|
### else-if mutual exclusion (weight: medium)
|
||||||
|
- For every `else if`: can BOTH conditions be true simultaneously? If yes, the second is silently skipped.
|
||||||
|
- Does `continue` in a multi-concern loop body suppress unrelated work?
|
||||||
|
|
||||||
|
### Map lookup direction (weight: medium)
|
||||||
|
- Is the map queried with value where key is expected (or vice versa)?
|
||||||
|
|
||||||
|
### Catch clause scope (weight: medium)
|
||||||
|
- Is the catch clause too broad? `catch (Exception)` that swallows errors making retry loops infinite.
|
||||||
|
- Is it too narrow, missing a related exception subtype?
|
||||||
|
|
||||||
|
### Conditional counting (weight: low)
|
||||||
|
- Is a counter incremented per item in a collection that may contain duplicates?
|
||||||
|
- Is a "remaining" counter passed as "completed" to a progress API expecting monotonically increasing values?
|
||||||
|
|
||||||
|
### Dead code / unused accumulation (weight: low)
|
||||||
|
- Does a collector accumulate state but no code reads the result?
|
||||||
|
- Does a predicate always return a hard-coded constant instead of computing from fields?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sentinel & Default Values
|
||||||
|
|
||||||
|
### Sentinel flows into arithmetic without guard (weight: high)
|
||||||
|
- Does a sentinel value (`Long.MIN_VALUE`, `Integer.MAX_VALUE`, `-1`, `0`) flow into arithmetic or comparison without a guard?
|
||||||
|
- Does `Map.getOrDefault` return a zero/empty default that is then treated as real data?
|
||||||
|
- Does a "always-present" filter break callers who rely on genuine negative results?
|
||||||
|
|
||||||
|
### Hardcoded value where caller context should flow (weight: medium)
|
||||||
|
- Is a hardcoded constant (timeout, mode, level) used where the caller's context should be forwarded?
|
||||||
|
- Is a derived default computed before the field it depends on is set?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Variable & Operand
|
||||||
|
|
||||||
|
### Wrong variable, field, or argument (weight: high)
|
||||||
|
- Does comparison or operation use a field from outer scope instead of the loop variable?
|
||||||
|
- Does code use source buffer instead of destination?
|
||||||
|
- Does getter read derived state from child instead of authoritative field?
|
||||||
|
- Does a constructor or factory argument end up in the wrong slot? Read the call as a sentence to verify each argument matches its role.
|
||||||
|
|
||||||
|
### Error messages reference wrong object (weight: high)
|
||||||
|
- Format string references wrong variable or swaps argument order?
|
||||||
|
- Error message names wrong flag/field or uses full object instead of `.name`?
|
||||||
|
|
||||||
|
### Logger class copy-pasted (weight: high)
|
||||||
|
- Is `LoggerFactory.getLogger()` passing the correct enclosing class, not copy-pasted from another file?
|
||||||
|
|
||||||
|
### Constructor overload resolution (weight: high)
|
||||||
|
- Does subclass delegate to wrong parent overload (parameter meanings swapped)?
|
||||||
|
- Is a required parameter not forwarded after signature change?
|
||||||
|
- Does overload resolution ambiguity from class hierarchy cause wrong binding?
|
||||||
|
|
||||||
|
### Copy-paste in parallel registrations (weight: medium)
|
||||||
|
- When registering similar items in sequence: does one entry duplicate a neighbor's argument?
|
||||||
|
- In adjacent metric registrations: are name/supplier pairs swapped?
|
||||||
|
|
||||||
|
### add() vs addAll() after collection type change (weight: low)
|
||||||
|
When a map value type changes from a single element to a `Collection`, callers still using `add(entry.getValue())` add the entire collection as one element.
|
||||||
|
|
||||||
|
### Boolean parameter conflates two orthogonal concerns (weight: low)
|
||||||
|
A boolean parameter is repurposed as a proxy for a different concern, so a caller needing different values for each has no valid representation.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Iterator & Loop
|
||||||
|
|
||||||
|
### Sub-iterator advancement (weight: medium)
|
||||||
|
- Does chained sub-iterator use `if` instead of `while` to skip empties?
|
||||||
|
- Is a shared iterator consumed across multiple input items?
|
||||||
|
|
||||||
|
### Separator logic (weight: low)
|
||||||
|
- Is "skip separator on first element" pattern applied when separator is actually a terminator?
|
||||||
|
- Is the wrong iterator checked for `hasNext()` when two parallel iterators run?
|
||||||
|
|
||||||
|
### Mutable iterator in lambda (weight: low)
|
||||||
|
- Is `.next()` called inside a lambda passed to `forEach`, advancing past unintended elements?
|
||||||
|
|
||||||
|
### Parallel data structure loop bounded by wrong size (weight: low)
|
||||||
|
When iterating two parallel arrays, using only one's length as the bound silently over- or under-processes the other.
|
||||||
|
|
||||||
|
### Loop guard evaluates outer-container field instead of loop variable (weight: low)
|
||||||
|
Inside a loop, a null-check or sentinel-guard references a field of the outer container, evaluating the same way every iteration. Common after rename where outer variable was renamed but inner references were missed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Refactoring Artifacts
|
||||||
|
|
||||||
|
### Stale identifier after rename (weight: high)
|
||||||
|
After a rename: have ALL references been updated — string literals, error messages, scripts, `Class.forName()` string-typed names?
|
||||||
|
|
||||||
|
### Merge artifacts (weight: high)
|
||||||
|
After merge/rebase: is the resolution semantically correct (not just syntactically)? Does merge leave wrong variable name, wrong import, wrong constant, duplicate declaration, inverted condition, or dead code from the other branch?
|
||||||
|
|
||||||
|
### Refactored guard moved to wrong scope (weight: medium)
|
||||||
|
Moving a guard during refactoring places it in a utility method that fires for all callers, not just the intended one.
|
||||||
|
|
||||||
|
### Helper extracted to utility loses caller-specific empty-input guard (weight: low)
|
||||||
|
When a utility is extracted from one caller and reused by others: does it still depend on a guard that only the original caller applied at its call site?
|
||||||
|
|
||||||
|
### Deprecated API drops parameters when delegating to new form (weight: low)
|
||||||
|
When a deprecated API path forwards to a shared builder, any option the old path accepted but does not forward causes a silent no-op.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Inverted / Negated Conditions
|
||||||
|
|
||||||
|
### Inverted boolean guard (weight: high)
|
||||||
|
- For EVERY boolean guard: read aloud as a sentence. Does the polarity match the intent?
|
||||||
|
- Is `!` where there should be none? `<` instead of `>=`? `==` instead of `!=`?
|
||||||
|
|
||||||
|
### Short-circuit loop semantics inverted (weight: low)
|
||||||
|
Early-return on negative match with default true flips semantics, silently producing wrong results.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Filtering & Selection
|
||||||
|
|
||||||
|
### Missing guard for empty input (weight: high)
|
||||||
|
- What happens when the collection/iterator has zero elements?
|
||||||
|
- Does empty credentials, config, or options cause a different (broken) code path?
|
||||||
|
|
||||||
|
### Result ordering driven by wrong source (weight: high)
|
||||||
|
- Is ordering driven by an authoritative ordered list, not `Map.values()` or `Map.keySet()`?
|
||||||
|
|
||||||
|
### Missing or overly broad filter conflates enablement with fast-path (weight: medium)
|
||||||
|
A single boolean guard used for both "should this subsystem activate" and "should the inner fast path short-circuit".
|
||||||
|
|
||||||
|
### Scope or namespace not included in cache key or identifier resolution (weight: medium)
|
||||||
|
A cache is keyed on a local name without its owning namespace, allowing collisions across namespaces.
|
||||||
|
|
||||||
|
### Wrong collection queried for lookup (weight: medium)
|
||||||
|
Code looks up a value in the wrong map (e.g., primary-key set instead of regular-attribute map).
|
||||||
|
|
||||||
|
### Assertion used where defensive check needed for legitimate runtime state (weight: medium)
|
||||||
|
An `assert` guards against a state that legitimately arises at runtime, causing crashes in production.
|
||||||
|
|
||||||
|
### Mutable shared object passed to multiple consumers without copy (weight: medium)
|
||||||
|
A stateful object with mutable counters/cursors/flags is constructed once and passed to multiple independent consumers in a loop.
|
||||||
|
|
||||||
|
### Transformation / pipeline stage ordering error (weight: medium)
|
||||||
|
In a chained iterator pipeline, stages are applied in the wrong order so a dependent stage misses events from an extension it requires.
|
||||||
|
|
||||||
|
### Aggregate returns wrong result on empty input (weight: low)
|
||||||
|
An aggregate function fails to return a row on empty input, violating expected semantics.
|
||||||
|
|
||||||
|
### Filter predicate applied to one branch of union but not both (weight: low)
|
||||||
|
A canonical view is built by merging two collections with a filtering step on only one.
|
||||||
|
|
||||||
|
### Boolean predicate named for one context misused in another (weight: low)
|
||||||
|
A predicate named for context A is used in context B where its boolean sense inverts.
|
||||||
|
|
||||||
|
### Mutable object from cache modified in-place, affecting all holders (weight: low)
|
||||||
|
A mutable object retrieved from a cache or factory is modified in-place by one consumer, affecting all other consumers sharing the same reference.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Type Dispatch & Enum Coverage
|
||||||
|
|
||||||
|
### Incomplete switch / enum dispatch — missing case (weight: medium)
|
||||||
|
A switch statement or if-chain over an enum does not cover a newly added or uncommon value, causing fallthrough or skipping the needed action.
|
||||||
|
|
||||||
|
### Wrong type returned by fallback/default in type-dispatch (weight: medium)
|
||||||
|
A switch on type ends with a hardcoded fallback type that is wrong for the uncommon case.
|
||||||
|
|
||||||
|
### Non-exhaustive switch lacks meaningful default (weight: low)
|
||||||
|
A `switch` on an enum without exhaustive cases or a meaningful `default` crashes or falls through silently.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Arithmetic & Units
|
||||||
|
|
||||||
|
### Unit mismatch in time / size arithmetic (weight: high)
|
||||||
|
A duration, timestamp, or size is computed by mixing incompatible units (nanoseconds added to milliseconds, int packed into long without widening) without explicit conversion.
|
||||||
|
|
||||||
|
### Unit-conversion factor applied in wrong direction (weight: medium)
|
||||||
|
- Does the multiply/divide direction match the source→target units? `bytes * 1024` instead of `bytes / 1024` produces a result 2^10 too high.
|
||||||
|
|
||||||
|
### Progress / ratio uses mismatched numerator and denominator (weight: medium)
|
||||||
|
- Does progress report values in different units for numerator vs denominator?
|
||||||
|
- Are `bytesRead` / `totalBytes` swapped, producing inverted completion percentage?
|
||||||
|
|
||||||
|
### Bits-vs-bytes unit label or conversion factor wrong (weight: medium)
|
||||||
|
Throughput display or throttle confuses bits and bytes, or the unit label says "MB/s" when the value is in "Mb/s".
|
||||||
|
|
||||||
|
### Comparator / ordering logic incorrect (weight: high)
|
||||||
|
A custom comparator is asymmetric, uses subtraction instead of `Integer.compare()`, handles sentinel values inconsistently, or does not cover all three cases.
|
||||||
|
|
||||||
|
### Lambda comparator argument order reversed (weight: low)
|
||||||
|
Comparators intended for descending order that use `(o1, o2)` instead of `(o2, o1)` silently sort in the wrong direction.
|
||||||
|
|
||||||
|
### Capacity check uses element count instead of byte count (weight: low)
|
||||||
|
An upper-bound check expressed in count units rather than byte units allows the structure to exceed its byte-capacity limit.
|
||||||
|
|
||||||
|
### Conditional decrement chain miscounts across branches (weight: low)
|
||||||
|
Computing an index by starting from a base and applying conditional decrements across branching logic applies the wrong number of adjustments.
|
||||||
|
|
||||||
|
### Elapsed time computed as `now - (now - timestamp)` = timestamp (weight: low)
|
||||||
|
The expression simplifies to `timestamp` itself (not elapsed duration), causing duration comparisons to produce nonsensical results.
|
||||||
|
|
||||||
|
### Two different size metrics conflated (weight: low)
|
||||||
|
A value returned as "bytes read from file" is used as "bytes sent over network" — the two differ for compressed or encoded content.
|
||||||
|
|
||||||
|
### float/double loses precision before BigDecimal construction (weight: low)
|
||||||
|
Widening a `float` to `double` or passing through floating-point as an intermediate step introduces spurious digits or truncates large values.
|
||||||
|
|
||||||
|
### Timezone offset derived from current time instead of timestamp being converted (weight: low)
|
||||||
|
A helper derives a timezone offset from `now` instead of the timestamp being converted.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Control Flow & Dispatch
|
||||||
|
|
||||||
|
### Two code paths for same operation diverge (weight: low)
|
||||||
|
When parallel code paths exist (NIO vs non-NIO, streaming vs local), one path omits a required step that the other performs.
|
||||||
|
|
||||||
|
### Scheduling loop misses re-arm on one branch (weight: low)
|
||||||
|
A periodic task's scheduling loop misses re-arming the next event on the "work done" branch, causing the task to stop running.
|
||||||
|
|
||||||
|
### Inner worker void return hides failure from dispatch loop (weight: low)
|
||||||
|
A `void` helper inside a dispatch loop cannot communicate failure back to the loop controller, so the loop continues past failures.
|
||||||
|
|
||||||
|
### Unconditional resource acquisition before mode/config check (weight: low)
|
||||||
|
A connection or resource is acquired unconditionally before a branch that checks whether it is needed, failing on environments where the unused mode is disabled.
|
||||||
|
|
||||||
|
### Mutable state overwritten between evaluation sites (weight: low)
|
||||||
|
An object carries mutable evaluation state and is called from multiple sites; a later call's state overwrites the earlier one.
|
||||||
|
|
||||||
|
### Feature removal leaves leftover conditional references (weight: low)
|
||||||
|
After removing a feature, leftover references to the removed concept in conditional guards invert or short-circuit important logic.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Logging & Diagnostics
|
||||||
|
|
||||||
|
### Eager evaluation in disabled log/debug path (weight: medium)
|
||||||
|
A log statement's arguments include method calls (collection formatting, type serialization, `String.format`) that are evaluated even when the log level is disabled.
|
||||||
|
|
||||||
|
### SLF4J `{}` vs printf `%s`/`%d` mismatch (weight: medium)
|
||||||
|
A logger call uses printf placeholders instead of SLF4J `{}`, or has more tokens than arguments.
|
||||||
|
|
||||||
|
### Tool exits 0 on failure / error swallowed silently (weight: low)
|
||||||
|
A CLI tool logs an error but returns exit code 0, or I/O errors are logged and swallowed instead of propagated.
|
||||||
|
|
||||||
|
### Typo or misspelling in user-facing string (weight: low)
|
||||||
|
An error message, log line, or CLI output contains a misspelled word, duplicate word, or wrong technical term.
|
||||||
|
|
||||||
|
### Documentation or help text not updated for new/renamed feature (weight: medium)
|
||||||
|
When a feature is added, renamed, or removed, its documentation — help strings, CLI usage text, example commands — is not updated.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Performance (detectable in review)
|
||||||
|
|
||||||
|
### Hot-path re-derivation (weight: low)
|
||||||
|
A method called in an inner loop delegates to an expensive operation on every invocation instead of caching.
|
||||||
|
|
||||||
|
### Unnecessary auto-boxing (weight: low)
|
||||||
|
A generic container parameterized with boxed types auto-boxes on every comparison in a tight loop.
|
||||||
|
|
||||||
|
### O(n) in hot path (weight: low)
|
||||||
|
A method called in a hot path iterates a collection where a reverse lookup map would give O(1).
|
||||||
|
|
||||||
|
### Scheduling at wrong granularity (weight: low)
|
||||||
|
A recurring task that should run once globally is scheduled per-item, creating N copies. Or a throttle fires on logical units instead of at the byte-transfer layer.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Structural Patterns from Multi-Project Corpus
|
||||||
|
|
||||||
|
### Assertion guards only one variant of several that legitimately reach the site (weight: medium)
|
||||||
|
- For every `assert` on state / subtype: enumerate every valid value that legitimately reaches this call site.
|
||||||
|
- Alternative modes, concurrent-not-found, zero-output paths, and edge-mode subclasses all fire spurious `AssertionError` when only one variant was considered.
|
||||||
|
|
||||||
|
### ByteBuffer position-advancing read without duplicate (weight: medium)
|
||||||
|
- For every relative `get()` / `put()` / `read()` on a shared or caller-supplied buffer: is it preceded by `.duplicate()` / `.slice()`?
|
||||||
|
- Position advances silently corrupt subsequent reads; singleton/cached buffers are permanently exhausted.
|
||||||
|
- Is `rewind()` used where `flip()` was intended? `rewind()` leaves limit at capacity, exposing uninitialized tail bytes.
|
||||||
|
|
||||||
|
### Reversed-iteration bound inclusion flipped (weight: medium)
|
||||||
|
- For every bound-inclusion check on a reversed-iteration path: is the LOGICAL bound reversed to match iteration direction? Exclusivity applied to the wrong endpoint silently returns wrong rows.
|
||||||
|
- For direction-sensitive skip guards in a reverse context: is the comparator's sense flipped? Guards fire when they shouldn't, re-emitting already-seen items on subsequent pages.
|
||||||
|
|
||||||
|
### CAS / retry loop reuses already-mutated source (weight: medium)
|
||||||
|
- Does a CAS retry pass the SOURCE object to a mutating merge? First iteration mutates it; subsequent retries operate on altered data.
|
||||||
|
- Does a distribution loop iterate a shared collection without removing consumed entries, so every consumer receives the full set instead of a disjoint subset?
|
||||||
|
|
||||||
|
### Unit-conversion factor in wrong direction (weight: medium)
|
||||||
|
- Does the multiply/divide direction match the source→target units? `bytes * 1024` instead of `bytes / 1024` produces a result 2^10 too high.
|
||||||
|
|
||||||
|
### Progress ratio uses mismatched numerator and denominator (weight: medium)
|
||||||
|
- Does a ratio sample numerator and denominator at different moments, allowing negative or >100% aggregate values?
|
||||||
|
|
||||||
|
### In-flight counter incremented without matching decrement on error path (weight: medium)
|
||||||
|
- Is the counter incremented before work is submitted, and does the rejection / cancel / exception path decrement?
|
||||||
|
- Does a metric counter stay permanently divergent from actual state after a failed async write?
|
||||||
|
|
||||||
|
### Equal-bounds degenerate case skipped in range predicate (weight: medium)
|
||||||
|
- When `min == max == value`, can a range predicate prove exclusion or match? Many "might match" defaults never prune the provably-disjoint single-value case.
|
||||||
|
|
||||||
|
### Derived cache not refreshed after configuration reload (weight: low)
|
||||||
|
- Is a derived cache populated at startup but never invalidated when its source configuration is reloaded?
|
||||||
|
- Is a setup-time configuration resolved before the per-target entity is known, freezing settings at global defaults?
|
||||||
|
|
||||||
|
### Inline field initializer survives constructor / setter (weight: low)
|
||||||
|
- When a field has a hardcoded inline initializer and the constructor is supposed to overwrite it, does the assignment actually fire on all code paths?
|
||||||
|
- Does a copy constructor propagate a time-sensitive field (expiry, timestamp) unchanged when the new object represents a distinct lifetime?
|
||||||
|
|
||||||
|
### Unconditional defensive-wrap chain grows on each rebuild (weight: low)
|
||||||
|
- Does a constructor/rebuild path wrap its input in an unmodifiable / defensive wrapper without first checking whether the input is already wrapped?
|
||||||
|
- Repeated rebuilds build a linearly-growing delegation chain that overflows the stack on iteration.
|
||||||
|
|
||||||
|
### Version / threshold string compared lexicographically (weight: low)
|
||||||
|
- For every version or threshold comparison: is it numeric-aware? String compare silently admits `"9" > "11"`.
|
||||||
|
|
||||||
|
### Mutable data structure returns construction-time constant instead of current state (weight: low)
|
||||||
|
A method that should reflect current state instead returns a construction-time constant.
|
||||||
|
|
||||||
|
### Subclass identity not tested in type-check method (weight: low)
|
||||||
|
When a subclass represents a semantically distinct variant, type-check methods on the parent that ignore the subclass identity misclassify instances.
|
||||||
|
|
||||||
|
### Base set used after augmented set was constructed (weight: low)
|
||||||
|
Code builds an "all" set by augmenting a base set, then subsequent operations use the base set instead of the augmented one.
|
||||||
|
|
||||||
|
### Reference equality used instead of value equality (weight: medium)
|
||||||
|
Code uses `==` to compare Strings or other value objects that may not be the same instance. After any refactoring that changes object lifecycle, identity checks break silently.
|
||||||
|
|
||||||
|
### Two-pass algorithm uses different reference time per pass (weight: low)
|
||||||
|
When a two-pass algorithm calls `currentTimeMillis()` independently in each pass, time-sensitive decisions can differ between passes.
|
||||||
|
|
||||||
|
### Cross-constraint satisfiability not checked at definition time (weight: low)
|
||||||
|
Compound declarative predicates are validated individually but not cross-checked for contradictions or redundancies.
|
||||||
|
|
||||||
|
### Method has multiple paths but only some fulfill a shared postcondition (weight: low)
|
||||||
|
When all branches of a method must meter or track a resource but only some branches contain the tracking call, the others silently skip it.
|
||||||
|
|
||||||
|
### Paired min/max config fields but only one forwarded (weight: low)
|
||||||
|
When a class has parallel min/max config fields, forwarding only one silently drops the other setting.
|
||||||
|
|
||||||
|
### Method return value ignored by caller assuming in-place mutation (weight: low)
|
||||||
|
When a method both mutates an argument and returns a value, callers that ignore the return value silently lose data.
|
||||||
|
|
||||||
|
### Direct field access in compareTo/equals breaks subclasses (weight: low)
|
||||||
|
A comparison implementation that reads final fields directly instead of through accessor methods cannot be overridden.
|
||||||
|
|
||||||
|
### Async future wrapping conflates cancellation with execution failure (weight: low)
|
||||||
|
An async future wrapper converts both `CancellationException` and execution failures into the same exception type.
|
||||||
|
|
||||||
|
### Deduplication / visited-set populated from wrong source (weight: low)
|
||||||
|
When merging two data sources with deduplication, the "already-seen" set is populated from the wrong source.
|
||||||
|
|
@ -0,0 +1,353 @@
|
||||||
|
# Deep Resources & Serialization — Extended Checklist
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context Gathering (REQUIRED)
|
||||||
|
|
||||||
|
Before applying the checklist, read the TARGET FILES (not just the diff) to understand:
|
||||||
|
|
||||||
|
1. **Serialization contract**: Read both `serialize()` and `deserialize()` end-to-end. List every field.
|
||||||
|
2. **Version gates**: What protocol/format versions exist? Which fields are gated?
|
||||||
|
3. **Resource ownership**: Who creates, who closes? Is ownership transferred?
|
||||||
|
4. **Error paths**: Trace every exception exit. What resources are held at each exit?
|
||||||
|
5. **Lifecycle**: When are resources acquired and released? What cleanup runs on shutdown?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Serialization Depth
|
||||||
|
|
||||||
|
### Field count/order mismatch (weight: high)
|
||||||
|
- Compare `serialize()`, `deserialize()`, and `serializedSize()` line by line. Do they write, read, and measure the SAME fields in the SAME order under the SAME conditional guards?
|
||||||
|
- Is a field present in `serialize()` but missing from `serializedSize()` (or vice versa)?
|
||||||
|
- Is a `sizeof()` result computed but never accumulated (dead code)?
|
||||||
|
- Is a length field missing from the symmetric read/write contract?
|
||||||
|
|
||||||
|
### Wire format mismatch (weight: high)
|
||||||
|
- Does custom payload use `byte[]` where protocol specifies `ByteBuffer`?
|
||||||
|
- Are column names encoded as ASCII instead of UTF-8?
|
||||||
|
- Is Thrift wire format assumed when native protocol is in use (or vice versa)?
|
||||||
|
- Does nodetool use `String.getBytes` instead of the key validator?
|
||||||
|
|
||||||
|
### Empty/null buffers (weight: high)
|
||||||
|
- Can an empty buffer reach a serializer validate method?
|
||||||
|
- Can a zero-length value enter code assuming non-empty content?
|
||||||
|
- Does the code handle UNSET sentinel values alongside null?
|
||||||
|
|
||||||
|
### Wrong stream/variable (weight: high)
|
||||||
|
- After creating a wrapper stream (tee, counting, checked): does ALL subsequent I/O use the WRAPPER?
|
||||||
|
- Is the original field used instead of a locally transformed variable?
|
||||||
|
- Is `writableBytes()` called where `readableBytes()` is needed?
|
||||||
|
|
||||||
|
### Version compatibility (weight: high)
|
||||||
|
- Does the serializer restrict to current version only, breaking rolling upgrades?
|
||||||
|
- Has an enum ordinal been renumbered by removing middle constants?
|
||||||
|
- Do new schema columns break older peers?
|
||||||
|
- Is the digest computation version-aware?
|
||||||
|
|
||||||
|
### Nullable dereferencing (weight: high)
|
||||||
|
- Can individual UDT fields be absent?
|
||||||
|
- Can frozen collection elements be null?
|
||||||
|
- Can bind values be null or UNSET?
|
||||||
|
- Is a method return value discarded where it should replace the parameter?
|
||||||
|
|
||||||
|
### Checksum/digest range (weight: medium)
|
||||||
|
- Does the CRC cover ALL serialized fields (including newly added ones)?
|
||||||
|
- Is `ByteBuffer.duplicate()` taken before write (not after)?
|
||||||
|
- Is the digest computation version-aware across upgrade boundaries?
|
||||||
|
|
||||||
|
### Collection serialization order (weight: medium)
|
||||||
|
- Are frozen maps serialized via `SortedMap` with correct type comparator?
|
||||||
|
- Are JSON map keys wrapped in strings?
|
||||||
|
|
||||||
|
### VInt signed/unsigned (weight: low)
|
||||||
|
- Is `computeVIntSize` used where `computeUnsignedVIntSize` is needed (or vice versa)?
|
||||||
|
|
||||||
|
### Length prefix width (weight: low)
|
||||||
|
- Is a 4-byte length prefix written but 2-byte prefix expected on deserialization?
|
||||||
|
|
||||||
|
### Enum deserialization (weight: low)
|
||||||
|
- Does enum-switched deserialization preserve a previously assigned field for a case where it should be absent?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Resource Leak Depth
|
||||||
|
|
||||||
|
### Exception mid-sequence (weight: high)
|
||||||
|
- When multiple AutoCloseable resources are acquired in sequence: if the second allocation fails, is the first released?
|
||||||
|
- Is there a try/catch or try-with-resources chain guarding stepwise allocation?
|
||||||
|
|
||||||
|
### Loop-built closeables (weight: medium)
|
||||||
|
- When Closeable resources are built in a loop: if an exception occurs partway, are already-created resources closed?
|
||||||
|
- Is each individual `release()` in a cleanup loop independently try/caught?
|
||||||
|
|
||||||
|
### Double-counting/release (weight: low)
|
||||||
|
- Is a resource counter incremented/decremented twice for the same event?
|
||||||
|
- Is a "free" operation using a pre-mutation delta?
|
||||||
|
|
||||||
|
### Close side effects (weight: low)
|
||||||
|
- Does `close()` trigger side-effecting callbacks without an idempotency guard?
|
||||||
|
- Could close fire twice (once from try-with-resources, once from caller)?
|
||||||
|
|
||||||
|
### Short-circuit cleanup (weight: low)
|
||||||
|
- Does a cleanup loop stop at the first exception instead of continuing to release all?
|
||||||
|
- Is each cleanup in the loop independently guarded?
|
||||||
|
|
||||||
|
### Ref-counted asymmetry (weight: low)
|
||||||
|
- Is a reference count + lifecycle transaction released as a pair on error?
|
||||||
|
- Does one half leak on error paths?
|
||||||
|
|
||||||
|
### View returned to pool (pattern)
|
||||||
|
- When a `ByteBuffer` is obtained via `nioBuffer()`, `slice()`, `duplicate()`: is the ORIGINAL returned to the pool?
|
||||||
|
- Does the pool receive a view it can't recycle?
|
||||||
|
|
||||||
|
### Filtered-out items (weight: low)
|
||||||
|
- When an iterator of closeable objects is filtered: are filtered-out items closed?
|
||||||
|
|
||||||
|
### Metric registered without deregistration (weight: low)
|
||||||
|
- Does every `addMetric` have a matching `removeMetric` in `close()`?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Lifecycle & Configuration Depth
|
||||||
|
|
||||||
|
### Config not threaded (weight: high)
|
||||||
|
- Is a config knob parsed but never threaded to the builder?
|
||||||
|
- Is `null` passed where config should flow?
|
||||||
|
- Are streaming encryption settings dropped?
|
||||||
|
|
||||||
|
### Throwable gotchas (weight: medium)
|
||||||
|
- Can `getMessage()` return null (no-arg exception constructor)?
|
||||||
|
- Can `getCause()` be null (no chained cause)?
|
||||||
|
- Can `getStackTrace()` return zero elements (`writableStackTrace=false`)?
|
||||||
|
|
||||||
|
### Success path missing cleanup (weight: medium)
|
||||||
|
- Does the success path miss cleanup that the error path has?
|
||||||
|
- Is session removal, counter decrement, or map eviction only on the error branch?
|
||||||
|
|
||||||
|
### State persisted too early (weight: medium)
|
||||||
|
- Is persistent state written when an operation starts rather than when it commits?
|
||||||
|
- Could a crash leave stale state that prevents retry?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File I/O & Crash Safety Depth
|
||||||
|
|
||||||
|
### File write modes (specific patterns)
|
||||||
|
- `WRITE | CREATE` without `TRUNCATE_EXISTING` on pre-existing file: old content past new EOF remains.
|
||||||
|
- `Files.copy()` without `REPLACE_EXISTING` in code that can run twice.
|
||||||
|
- Jackson `writeValue(OutputStream)` auto-closes: is write-to-temp, fsync, rename, fsync-parent used instead?
|
||||||
|
|
||||||
|
### Multi-file atomicity (pattern)
|
||||||
|
- When multiple related files are persisted: are ALL flushed and synced before the directory sync?
|
||||||
|
- Is there a missing flush on any one file in the set?
|
||||||
|
|
||||||
|
### Correlated state changes (pattern)
|
||||||
|
- Are multiple correlated state changes announced as separate operations instead of bundled atomically?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Metrics & Observability Depth
|
||||||
|
|
||||||
|
### Metric timing (weight: high)
|
||||||
|
- Is a metric updated before the operation it measures?
|
||||||
|
- Is it after a different operation?
|
||||||
|
- Is it outside the guard that controls the measured operation?
|
||||||
|
|
||||||
|
### Progress counter (weight: low)
|
||||||
|
- Does progress increment per inner-loop page instead of per outer logical unit?
|
||||||
|
- Is the count off by orders of magnitude for wide rows?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Platform & Encoding Depth
|
||||||
|
|
||||||
|
### Locale/charset (weight: medium)
|
||||||
|
- Is `DecimalFormat` used without `Locale.ROOT`?
|
||||||
|
- Is `String.getBytes()` called without explicit charset?
|
||||||
|
- Does `Boolean.valueOf()` need to be `parseBoolean()`?
|
||||||
|
|
||||||
|
### I/O stream (specific patterns)
|
||||||
|
- Does `InputStream.read()` return raw signed byte without `& 0xFF`?
|
||||||
|
- Is `EINTR` handled in blocking system calls?
|
||||||
|
- Is `MessageDigest` shared across threads without synchronization?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Filesystem Path Depth
|
||||||
|
|
||||||
|
### Path comparison (weight: medium)
|
||||||
|
- Is `String.startsWith()` used for path containment instead of `Path.startsWith()`?
|
||||||
|
- Do symlinks break path comparisons?
|
||||||
|
|
||||||
|
### Wrong directory level (weight: medium)
|
||||||
|
- Does `directory.getName()` return the wrong component after hierarchy changes?
|
||||||
|
- Is a directory level omitted in path construction?
|
||||||
|
|
||||||
|
### Layout migration (weight: low)
|
||||||
|
- Does a directory layout change include migration of existing data from old location?
|
||||||
|
- Are files unreachable through the new path abstraction?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Missing from §18: Serialization — Specific Patterns
|
||||||
|
|
||||||
|
### Transient or non-serializable field in Serializable class (weight: medium)
|
||||||
|
Transient field set only in constructor silently null after deserialization, or third-party library types fail remote deserialization.
|
||||||
|
|
||||||
|
### Validation delegated to wrong serializer, wrong validator package, or bypassed entirely (weight: medium)
|
||||||
|
Nested tuple/UDT validation routed to BytesSerializer instead of type-aware serializer, collection validate() called from legacy CQL2 package.
|
||||||
|
|
||||||
|
### Endianness or byte-order not accounted for in bit-shift arithmetic (weight: low)
|
||||||
|
ByteBuffer.getLong feeds into bit-shift arithmetic assuming big-endian on little-endian buffers, or unsafe put/get does not account for BIG_ENDIAN on s390x.
|
||||||
|
|
||||||
|
### Protocol field order violation (encode vs decode sequence mismatch) (weight: low)
|
||||||
|
Encode writes flags before consistency level while decode reads them in opposite order, or paging state buffer not flipped before being read after write.
|
||||||
|
|
||||||
|
### Deserialization-order sensitivity not enforced (drain instead of throw) (weight: low)
|
||||||
|
A streaming deserializer throws `IllegalStateException` when a caller short-circuits iteration, instead of draining unconsumed items.
|
||||||
|
|
||||||
|
### ByteBuffer heap vs direct path divergence (weight: low)
|
||||||
|
VectorCodec deserialization using slice() and rewind() on heap buffers produces different behavior than on direct buffers.
|
||||||
|
|
||||||
|
### Variadic or positional argument swap in serialization call (weight: low)
|
||||||
|
Two same-typed arguments swapped in format call, silently producing reversed serialized fields.
|
||||||
|
|
||||||
|
### Negative-index sentinel encoding mishandled at position zero (weight: low)
|
||||||
|
Using `(-1 - x)` as a not-found sentinel conflates the not-found-at-zero case with valid positions.
|
||||||
|
|
||||||
|
### Role or identity derived from heuristic instead of explicit field (weight: low)
|
||||||
|
Encoding role as a derived heuristic ("if X == 0, I must be the receiver") is fragile.
|
||||||
|
|
||||||
|
### Multi-branch size calculation omits a component in one path (weight: low)
|
||||||
|
When `serializedSize()` has fast-path and slow-path branches, a component written in all paths may be counted in only one branch.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Missing from §20: Resource Leaks — Specific Patterns
|
||||||
|
|
||||||
|
### close() / release called recursively or at wrong lifecycle point (weight: low)
|
||||||
|
A `close()` method that performs both bookkeeping and native-resource release is called mid-operation, causing double-free or use-after-free.
|
||||||
|
|
||||||
|
### Fallback code path missing stateful transitions present in primary path (weight: low)
|
||||||
|
A fallback code path mirrors a primary path but omits seek, flush, or close calls.
|
||||||
|
|
||||||
|
### abort() Not Overridden in Subclass That Adds Resources (weight: low)
|
||||||
|
A subclass adds new Closeable resources but inherits the parent's `abort()`, which does not know about the new resources.
|
||||||
|
|
||||||
|
### Superclass Method Signature Change Silently Orphans Override (weight: low)
|
||||||
|
A superclass changes a method signature and the old override silently becomes unreachable dead code without `@Override`.
|
||||||
|
|
||||||
|
### Iterator next() Result Not Closed on All Paths (weight: low)
|
||||||
|
An iterator yields AutoCloseable items but `next()` results are passed directly to a consumer without try-with-resources.
|
||||||
|
|
||||||
|
### Half-open to closed range conversion missing +1 shift on upper bound (weight: low)
|
||||||
|
Wrapping a `[a, b)` distribution API for a `[a, b]` range without shifting the upper bound silently excludes the maximum value.
|
||||||
|
|
||||||
|
### Resource ownership split between task body and lifecycle callback (weight: low)
|
||||||
|
A resource owner submitted to an executor has close() in task body try/finally, but if the task is cancelled, cleanup never runs.
|
||||||
|
|
||||||
|
### Resource opened outside normal lifecycle tracking system (weight: low)
|
||||||
|
A ref-counted resource opened from a non-standard path (snapshot, offline tool) bypasses tracking and is never closed.
|
||||||
|
|
||||||
|
### Double-close of shared AutoCloseable fires side-effects twice (weight: low)
|
||||||
|
When both try-with-resources and caller close, side-effects (metric increments, counter decrements) fire twice.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Missing from §12: Filesystem Paths — Specific Patterns
|
||||||
|
|
||||||
|
### Missing validation for schema feature combination or filesystem constraint (weight: medium)
|
||||||
|
DDL does not check prerequisites: static columns require clustering columns, COMPACT STORAGE incompatible with collections, table name not validated against filesystem length.
|
||||||
|
|
||||||
|
### Wrong path, directory, or base resolution (weight: medium)
|
||||||
|
SSTable written to wrong data directory (bare disk path, not keyspace/table), trigger directory resolved from JAR location.
|
||||||
|
|
||||||
|
### Name stripping / directory parsing fails on unexpected suffix (weight: low)
|
||||||
|
Stripping a file extension does not remove additional well-known suffixes (`-tmp`, `-old`), or a path-construction step omits a directory level.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## State Cleanup — Resources-Domain Patterns (§19)
|
||||||
|
|
||||||
|
### Metric invisible to reporters, double-counted, stale at read time (weight: medium)
|
||||||
|
Metric implemented as bare interface invisible to monitoring exporters, chunk cache double-counted, disk-space metric computed at wrong time, or metrics library upgrade silently changes unit contract.
|
||||||
|
|
||||||
|
### Hinting code path not audited for non-idempotent mutation types (weight: low)
|
||||||
|
A new mutation type (counter write) is added without auditing the hinting path, which attempts to hint a non-idempotent operation.
|
||||||
|
|
||||||
|
### Early-Return Path Skips Cursor/Tracking State Update (weight: low)
|
||||||
|
An early-return path inside an accumulation loop skips the normal "advance the cursor / update tracking state" step.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Additional Patterns from Multi-Project Corpus
|
||||||
|
|
||||||
|
### Shared ByteBuffer position side-effect on read (weight: high)
|
||||||
|
- Is a ByteBuffer obtained from a static singleton, cached entry, registry map, method parameter, or cross-thread queue `duplicate()`d before ANY relative `get()`, `getInt`, `slice()`, `get(byte[])`, comparator read, or checksum update?
|
||||||
|
- Is a no-argument relative `get()` used in a comparator that can be called repeatedly on the same shared key?
|
||||||
|
- Does a deserialization helper return a reference to a shared singleton buffer whose position the caller will advance?
|
||||||
|
- Does a digest / array copy from a heap ByteBuffer use `array()` + `position()` without ALSO adding `arrayOffset()`? Sliced heap buffers silently produce wrong digests.
|
||||||
|
|
||||||
|
### Buffer flip vs rewind / channel read completeness (weight: medium)
|
||||||
|
- After writing into a ByteBuffer, is `flip()` called (not `rewind()`)? `rewind()` leaves limit = capacity and exposes uninitialized bytes past the write point.
|
||||||
|
- Is a channel `read()` return value checked (>=0) before `flip()` + `get()`? Closed-channel reads return 0 and cause `BufferUnderflowException`.
|
||||||
|
- Is a write buffer flipped before being returned from a `serialize(...)` helper (otherwise position == end and no bytes are readable)?
|
||||||
|
|
||||||
|
### serializedSize double-count / variant confusion (weight: medium)
|
||||||
|
- Does `serializedSize()` delegate to the on-disk / format-specific `serializedSize` variant, not the in-memory variant, when the output is on-disk or wire format?
|
||||||
|
- For variable-length fields: does the size include BOTH the length prefix AND the encoded bytes?
|
||||||
|
- Is `serializedSize()` updated after every format change, or does it still describe the prior format?
|
||||||
|
- Is a field's byte count added unconditionally AND re-added inside a conditional branch?
|
||||||
|
- Is the same `size()` result reused for both the total-record size and an embedded sub-record length prefix when the two require different values?
|
||||||
|
|
||||||
|
### Reference count not acquired before access (weight: medium)
|
||||||
|
- Is a ref-counted resource's counter incremented BEFORE it is read (not after the null check, not after `containsKey`, not after `get`)?
|
||||||
|
- Does a CAS-deserialize path acquire the ref while holding the map slot, not after the atomic read?
|
||||||
|
- Does a read-only scan (cleanup scan, cardinality estimate, diagnostic dump) acquire a ref before traversing backing files?
|
||||||
|
- Is an "observe-then-acquire" retry loop aware that the resource can be freed between the two steps?
|
||||||
|
|
||||||
|
### Close-on-shared-dependency / composite close (weight: medium)
|
||||||
|
- Does `close()` release EVERY Closeable field it holds, not just the primary? Audit for omitted delegates when fields were added.
|
||||||
|
- If `close()` shuts down a shared static executor / connection pool / registry, is this instance the owner? Non-owning close corrupts sibling instances.
|
||||||
|
- Does a composite appender / container define its own `stop` / `close` that orders children explicitly rather than relying on child-cascade shutdown?
|
||||||
|
|
||||||
|
### Cancellation / early-return / partial-construction resource loss (weight: medium)
|
||||||
|
- If the constructor fails mid-sequence, does `close()` tolerate null fields (no NPE during best-effort cleanup)?
|
||||||
|
- Is a field opened in the constructor assigned to the owner field BEFORE any throw / early return?
|
||||||
|
- Does the submit-after-shutdown-check race release resources held by the partially-constructed task when submit throws `RejectedExecutionException`?
|
||||||
|
- Are resources held in a cancellation / exception-handler callback released even when the task body's try / finally never runs?
|
||||||
|
- Is an iterator returned to a caller still owning its Closeables, or was it wrapped in try-with-resources whose `close` fires before the caller reads?
|
||||||
|
|
||||||
|
### File writer durability / partial-file-as-complete (weight: low)
|
||||||
|
- Does a `BufferedWriter` or serializing writer call `flush()` + `fsync()` before `close()` — not just `close()`?
|
||||||
|
- Is an error path guaranteed to call `abort()` on the writer so a partial file is distinguishable from a complete one on restart?
|
||||||
|
|
||||||
|
### Background task in constructor / static-init / short-lived entrypoint (weight: medium)
|
||||||
|
- Does a constructor start a background thread that might observe partially-constructed state (non-final fields read on the thread before init completes)?
|
||||||
|
- Does a CLI entry point or short-lived tool call a process-exit helper to reap non-daemon threads before returning?
|
||||||
|
- Is a periodic task submitted from a `static` initializer that may fire before its target executor is started?
|
||||||
|
- Is there a strong reference cycle (long-lived owner ↔ scheduled task / leak-detector callback) that prevents garbage collection?
|
||||||
|
|
||||||
|
### Re-arming / reschedule condition wrong (weight: medium)
|
||||||
|
- Is the re-arm decision driven by "did work occur" rather than by a transient collection being non-empty or by a side-effecting method's return value?
|
||||||
|
- For a one-shot scheduled refresh: does it reschedule itself after each successful execution?
|
||||||
|
|
||||||
|
### Config resolved at construction / cached with no fallback (weight: medium)
|
||||||
|
- Is a config value resolved lazily per target (supplier / factory), not frozen at construction from a global context where per-target overrides were not yet applied?
|
||||||
|
- When the underlying config source (file, secret manager) is temporarily unavailable, does the resolver fall back to the last-known-good cached value rather than throw?
|
||||||
|
- Is a `static final` captured at class-load time from a subsystem that the caller must configure first?
|
||||||
|
|
||||||
|
### Close semantics: null means both closed and not-yet-initialized (weight: low)
|
||||||
|
- Does a lazy wrapper that clears its delegate to null on close also cause post-close calls to silently re-initialize rather than fail?
|
||||||
|
- Distinguish "closed" from "not yet initialized" via an explicit state flag.
|
||||||
|
|
||||||
|
### Management endpoint rename without alias (weight: low)
|
||||||
|
- Does a rename of a registered metric or management endpoint also register the prior name as a deprecated alias?
|
||||||
|
- Existing monitoring clients otherwise silently break.
|
||||||
|
|
||||||
|
### Transient or non-serializable field in Serializable class (weight: low)
|
||||||
|
- Does a field declared `transient` or backed by a non-serializable library type (ByteBuffer, Guava collection, native handle) survive Java serialization, or is it silently null on the far side?
|
||||||
|
- Are dependencies captured in a holder class dropped during serialization, deferring the failure to first use post-deserialization?
|
||||||
|
|
||||||
|
### Metric present on one branch, missing on sibling (weight: low)
|
||||||
|
- When a metric is recorded on the success (or timeout) path, is an equivalent recording also present on the paired failure (or success) path?
|
||||||
|
- Is a metric recorded at the moment the measured condition first holds, rather than inside a timeout callback that fires later?
|
||||||
|
|
@ -0,0 +1,123 @@
|
||||||
|
---
|
||||||
|
name: heatmap
|
||||||
|
version: "1.0.0"
|
||||||
|
description: Use git heatmap analysis to identify high-churn files and lines as candidates for thorough review or bug hunting. Works for PR reviews, security audits, bug hunts, or any code analysis task.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Heatmap-Guided Review
|
||||||
|
|
||||||
|
Use the bundled `heatmap.py` script to identify the hottest (most frequently and recently changed) code in a repository, then focus review effort on those areas. Hot code is statistically more likely to contain bugs: it changes often, accumulates complexity, and is where active development risk concentrates.
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Invoke this skill whenever you want to prioritize where to look in a codebase — during PR reviews, bug hunts, security audits, onboarding exploration, or any task where you need to decide which files and lines deserve the closest attention.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- Python 3 must be available.
|
||||||
|
- The target must be a git repository.
|
||||||
|
- The heatmap script is bundled with this skill at `~/.claude/skills/heatmap/heatmap.py`. All commands below use `HEATMAP` as a placeholder:
|
||||||
|
```bash
|
||||||
|
HEATMAP="$HOME/.claude/skills/heatmap/heatmap.py"
|
||||||
|
```
|
||||||
|
If `heatmap` is available on PATH, you may use that instead.
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
### Step 1: Determine the repository path and scope
|
||||||
|
|
||||||
|
Figure out the git repository root. If the user provided a repo path, use that. If working in a repo already, use the current directory. If reviewing a PR, determine which files were changed.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
HEATMAP="$HOME/.claude/skills/heatmap/heatmap.py"
|
||||||
|
# Find the repo root
|
||||||
|
git -C <path> rev-parse --show-toplevel
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Run repo-level heatmap
|
||||||
|
|
||||||
|
Get the hottest files in the repository. Use `--json` for machine-readable output. Adjust `--top` based on context (20-50 for a broad scan, more for large repos).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 "$HEATMAP" repo --json --top 50 <repo_path>
|
||||||
|
```
|
||||||
|
|
||||||
|
This returns JSON array of objects with: `path`, `heat`, `commits`, `last_modified_days`.
|
||||||
|
|
||||||
|
### Step 3: Cross-reference with the task context
|
||||||
|
|
||||||
|
**For PR reviews:** Intersect the heatmap results with the files changed in the PR. Files that are both changed in the PR AND high on the heatmap are the highest-priority review targets — they are already complex/volatile and the PR is adding more changes to them.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Get PR changed files
|
||||||
|
git diff --name-only <base_branch>...HEAD
|
||||||
|
```
|
||||||
|
|
||||||
|
**For bug hunts:** The hottest files ARE the candidates — frequent recent changes correlate with bugs. Focus on the top 10-20.
|
||||||
|
|
||||||
|
**For security audits:** Filter heatmap results to security-sensitive paths (auth, crypto, input parsing, network, serialization).
|
||||||
|
|
||||||
|
**For general exploration:** Use the heatmap as a map of where active development is happening.
|
||||||
|
|
||||||
|
### Step 4: Run line-level heatmap on top candidates
|
||||||
|
|
||||||
|
For each high-priority file (typically 3-8 files), run the line-level heatmap to find the hottest regions within those files.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 "$HEATMAP" file --json <file_path_relative_to_repo> <repo_path>
|
||||||
|
```
|
||||||
|
|
||||||
|
This returns JSON array of objects with: `line`, `heat`, `content`.
|
||||||
|
|
||||||
|
### Step 5: Identify hot zones
|
||||||
|
|
||||||
|
From the line-level results, identify contiguous regions of high heat. These "hot zones" are where to focus review. Look for:
|
||||||
|
|
||||||
|
- **Clusters of hot lines** — contiguous blocks of high-heat code indicate areas under active rework. These are prime bug candidates because:
|
||||||
|
- Multiple recent changes suggest the logic is not yet settled
|
||||||
|
- Each change is an opportunity for regression
|
||||||
|
- Complex interactions between recent changes may not be fully tested
|
||||||
|
|
||||||
|
- **Hot lines surrounded by cold code** — surgical edits in otherwise stable code may indicate bug fixes, workarounds, or special-case handling that deserves scrutiny.
|
||||||
|
|
||||||
|
- **Hot function/method boundaries** — if a method signature or its first few lines are hot, the contract may have changed recently, affecting all callers.
|
||||||
|
|
||||||
|
### Step 6: Produce the review focus list
|
||||||
|
|
||||||
|
Output a ranked list of review targets, structured as:
|
||||||
|
|
||||||
|
```
|
||||||
|
## Heatmap Review Targets
|
||||||
|
|
||||||
|
### Priority 1: <file_path> (heat: X, commits: Y)
|
||||||
|
- **Hot zones**: lines A-B (description of what this code does)
|
||||||
|
- **Why it matters**: <context — e.g., "most changed file in repo AND modified in this PR">
|
||||||
|
- **What to look for**: <specific guidance based on the code — race conditions, edge cases, etc.>
|
||||||
|
|
||||||
|
### Priority 2: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 7: Deep review
|
||||||
|
|
||||||
|
For each priority target, read the hot zones and perform the actual review. The heatmap tells you WHERE to look; your expertise tells you WHAT to look for. Common patterns in hot code:
|
||||||
|
|
||||||
|
- **State management bugs**: Hot code often manages complex state. Check for inconsistent updates, missing synchronization, or partial failures.
|
||||||
|
- **Edge cases**: Frequent changes often mean edge cases keep being discovered. Look for more.
|
||||||
|
- **Regression risk**: If code was recently fixed, check whether the fix is complete and doesn't break other paths.
|
||||||
|
- **Missing tests**: Hot code that lacks test coverage is the highest-risk combination.
|
||||||
|
|
||||||
|
## Output format
|
||||||
|
|
||||||
|
Always present findings as a prioritized list with:
|
||||||
|
1. File path and heat metrics
|
||||||
|
2. Specific line ranges to focus on
|
||||||
|
3. What the hot code does (brief)
|
||||||
|
4. What risks to look for (specific to the code, not generic)
|
||||||
|
|
||||||
|
## Tips
|
||||||
|
|
||||||
|
- Heat is relative — compare files against each other, not against an absolute threshold.
|
||||||
|
- Files with high heat but few commits have large individual changes (risky). Files with high heat and many commits are under constant churn (also risky, differently).
|
||||||
|
- `last_modified_days` close to 0 means very recent changes — highest chance of unfound bugs.
|
||||||
|
- Use `--since` to adjust the time window. Default is 2 years. For recent bug hunts, try `--since "6 months ago"`.
|
||||||
|
- The `--no-color` flag is useful when piping output, but prefer `--json` for programmatic use.
|
||||||
|
|
@ -0,0 +1,603 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
# or more contributor license agreements. See the NOTICE file
|
||||||
|
# distributed with this work for additional information
|
||||||
|
# regarding copyright ownership. The ASF licenses this file
|
||||||
|
# to you under the Apache License, Version 2.0 (the
|
||||||
|
# "License"); you may not use this file except in compliance
|
||||||
|
# with the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
"""
|
||||||
|
Git Heatmap — visualize code change frequency and recency.
|
||||||
|
|
||||||
|
Heat formula:
|
||||||
|
decay(days) = 2^(-days / HALF_LIFE_DAYS) # half-life = 60 days
|
||||||
|
size_bonus(lines) = 1 + 1/(1 + lines/SMALL_CHANGE_SCALE) # small edits → bonus up to 2x
|
||||||
|
commit_heat(d, l) = decay(d) * size_bonus(l)
|
||||||
|
|
||||||
|
Repo-level: file_heat = Σ commit_heat(days_ago, added+deleted) per commit
|
||||||
|
Line-level: line_heat = Σ commit_heat(days_ago, churn) per commit that touched that line,
|
||||||
|
with line positions tracked through diffs via VersionMap.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import colorsys
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import List, Tuple, Optional, Dict
|
||||||
|
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Constants
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
HALF_LIFE_DAYS = 60
|
||||||
|
SMALL_CHANGE_SCALE = 10
|
||||||
|
|
||||||
|
# Extensions excluded from repo-level scans by default (non-code files).
|
||||||
|
DEFAULT_EXCLUDED_EXTENSIONS = frozenset({
|
||||||
|
'.txt', '.md', '.rst', '.adoc',
|
||||||
|
'.json', '.yaml', '.yml', '.toml', '.ini', '.cfg', '.conf', '.properties',
|
||||||
|
'.xml', '.csv', '.tsv',
|
||||||
|
'.lock', '.sum',
|
||||||
|
'.png', '.jpg', '.jpeg', '.gif', '.ico', '.svg', '.webp',
|
||||||
|
'.pdf', '.doc', '.docx',
|
||||||
|
'.zip', '.tar', '.gz', '.bz2', '.xz',
|
||||||
|
'.woff', '.woff2', '.ttf', '.eot',
|
||||||
|
'.min.js', '.min.css',
|
||||||
|
'.map',
|
||||||
|
'.license', '.licence',
|
||||||
|
})
|
||||||
|
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Core heat functions
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def decay(days_ago: float) -> float:
|
||||||
|
"""Exponential decay: 1.0 at day 0, 0.5 at HALF_LIFE_DAYS."""
|
||||||
|
return 2.0 ** (-days_ago / HALF_LIFE_DAYS)
|
||||||
|
|
||||||
|
|
||||||
|
def size_bonus(lines_changed: int) -> float:
|
||||||
|
"""Bonus multiplier for small changes.
|
||||||
|
|
||||||
|
Returns value in [1.0, 2.0):
|
||||||
|
1 line → ~1.91
|
||||||
|
10 lines → 1.50
|
||||||
|
100 lines→ ~1.09
|
||||||
|
"""
|
||||||
|
if lines_changed <= 0:
|
||||||
|
return 1.0
|
||||||
|
return 1.0 + 1.0 / (1.0 + lines_changed / SMALL_CHANGE_SCALE)
|
||||||
|
|
||||||
|
|
||||||
|
def commit_heat(days_ago: float, lines_changed: int) -> float:
|
||||||
|
"""Heat contributed by one commit = decay × size_bonus."""
|
||||||
|
return decay(days_ago) * size_bonus(lines_changed)
|
||||||
|
|
||||||
|
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# VersionMap — maps line numbers from old file version to new file version
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
class VersionMap:
|
||||||
|
"""Maps line numbers from parent (old) to child (new) file version.
|
||||||
|
|
||||||
|
Built from unified-diff hunk headers. For each hunk @@ -a,b +c,d @@:
|
||||||
|
• min(b,d) old lines map 1:1 to new lines (modification)
|
||||||
|
• extra old lines (b>d) are marked deleted → map_line returns -1
|
||||||
|
• extra new lines (d>b) are pure insertions → no old line maps to them
|
||||||
|
• lines outside hunks shift by the cumulative delta
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, hunks: List[Tuple[int, int, int, int]]):
|
||||||
|
self.regions: List[Tuple[str, int, float, int]] = []
|
||||||
|
cum_delta = 0
|
||||||
|
prev_old_end = 0
|
||||||
|
|
||||||
|
for os_, oc, ns, nc in hunks:
|
||||||
|
if oc > 0:
|
||||||
|
# Unchanged lines before this hunk
|
||||||
|
if os_ > prev_old_end + 1:
|
||||||
|
self.regions.append(('shift', prev_old_end + 1, os_ - 1, cum_delta))
|
||||||
|
# Modified lines (1:1 mapping)
|
||||||
|
min_c = min(oc, nc)
|
||||||
|
if min_c > 0:
|
||||||
|
self.regions.append(('modify', os_, os_ + min_c - 1, cum_delta))
|
||||||
|
# Deleted old lines
|
||||||
|
if oc > nc:
|
||||||
|
self.regions.append(('deleted', os_ + nc, os_ + oc - 1, 0))
|
||||||
|
prev_old_end = os_ + oc - 1
|
||||||
|
else:
|
||||||
|
# Pure insertion: os_ is the line *before* the insertion point
|
||||||
|
if os_ >= prev_old_end + 1:
|
||||||
|
self.regions.append(('shift', prev_old_end + 1, os_, cum_delta))
|
||||||
|
prev_old_end = os_
|
||||||
|
cum_delta += nc - oc
|
||||||
|
|
||||||
|
# Trailing region
|
||||||
|
self.regions.append(('shift', prev_old_end + 1, float('inf'), cum_delta))
|
||||||
|
|
||||||
|
def map_line(self, old_line: int) -> int:
|
||||||
|
"""Map old line → new line. Returns -1 if deleted."""
|
||||||
|
for kind, start, end, delta in self.regions:
|
||||||
|
if start <= old_line <= end:
|
||||||
|
return -1 if kind == 'deleted' else old_line + delta
|
||||||
|
return old_line
|
||||||
|
|
||||||
|
|
||||||
|
def map_through_versions(line: int, version_maps: List[VersionMap]) -> int:
|
||||||
|
"""Chain multiple VersionMaps. Returns -1 if deleted at any step."""
|
||||||
|
for vm in version_maps:
|
||||||
|
line = vm.map_line(line)
|
||||||
|
if line == -1:
|
||||||
|
return -1
|
||||||
|
return line
|
||||||
|
|
||||||
|
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Git helpers
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def git(repo_path: str, *args) -> str:
|
||||||
|
result = subprocess.run(
|
||||||
|
['git', '-C', repo_path] + list(args),
|
||||||
|
capture_output=True, text=True,
|
||||||
|
)
|
||||||
|
if result.returncode != 0:
|
||||||
|
raise RuntimeError(f"git failed: {' '.join(args)}\n{result.stderr[:500]}")
|
||||||
|
return result.stdout
|
||||||
|
|
||||||
|
|
||||||
|
HUNK_RE = re.compile(r'^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@')
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_rename(filename: str) -> str:
|
||||||
|
"""Resolve git numstat rename notation {old => new}."""
|
||||||
|
m = re.search(r'\{([^}]*) => ([^}]*)\}', filename)
|
||||||
|
if m:
|
||||||
|
return filename[:m.start()] + m.group(2) + filename[m.end():]
|
||||||
|
return filename
|
||||||
|
|
||||||
|
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Repo-level parsing
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class FileCommit:
|
||||||
|
hash: str
|
||||||
|
timestamp: int
|
||||||
|
added: int
|
||||||
|
deleted: int
|
||||||
|
|
||||||
|
|
||||||
|
def parse_repo_log(repo_path: str, since: Optional[str] = None) -> Dict[str, List[FileCommit]]:
|
||||||
|
"""Return {filename: [FileCommit, ...]} from git log --numstat."""
|
||||||
|
args = ['log', '--format=commit %H %at', '--numstat', '-m', '--first-parent']
|
||||||
|
if since:
|
||||||
|
args.append(f'--since={since}')
|
||||||
|
output = git(repo_path, *args)
|
||||||
|
|
||||||
|
files: Dict[str, List[FileCommit]] = {}
|
||||||
|
cur_hash = cur_ts = None
|
||||||
|
|
||||||
|
for line in output.splitlines():
|
||||||
|
if line.startswith('commit '):
|
||||||
|
parts = line.split()
|
||||||
|
cur_hash, cur_ts = parts[1], int(parts[2])
|
||||||
|
elif line and cur_hash and '\t' in line:
|
||||||
|
parts = line.split('\t')
|
||||||
|
if len(parts) < 3 or parts[0] == '-':
|
||||||
|
continue
|
||||||
|
added, deleted = int(parts[0]), int(parts[1])
|
||||||
|
fname = resolve_rename(parts[2])
|
||||||
|
files.setdefault(fname, []).append(
|
||||||
|
FileCommit(cur_hash, cur_ts, added, deleted))
|
||||||
|
return files
|
||||||
|
|
||||||
|
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# File-level (line) parsing
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class FileDiffCommit:
|
||||||
|
hash: str
|
||||||
|
timestamp: int
|
||||||
|
hunks: List[Tuple[int, int, int, int]]
|
||||||
|
added_lines: List[int] # new-side line numbers with '+' prefix
|
||||||
|
total_churn: int # added + deleted line count
|
||||||
|
|
||||||
|
|
||||||
|
def parse_file_log(repo_path: str, file_path: str,
|
||||||
|
since: Optional[str] = None) -> List[FileDiffCommit]:
|
||||||
|
"""Parse git log -p -U0 for one file. Returns list newest-first."""
|
||||||
|
args = ['log', '--format=commit %H %at', '-p', '-U0',
|
||||||
|
'--follow', '-m', '--first-parent']
|
||||||
|
if since:
|
||||||
|
args.append(f'--since={since}')
|
||||||
|
args += ['--', file_path]
|
||||||
|
output = git(repo_path, *args)
|
||||||
|
|
||||||
|
commits: List[FileDiffCommit] = []
|
||||||
|
cur_hash = cur_ts = None
|
||||||
|
hunks: List[Tuple[int,int,int,int]] = []
|
||||||
|
added_lines: List[int] = []
|
||||||
|
add_cnt = del_cnt = 0
|
||||||
|
new_counter = 0
|
||||||
|
in_hunk = False
|
||||||
|
|
||||||
|
def flush():
|
||||||
|
nonlocal hunks, added_lines, add_cnt, del_cnt, in_hunk
|
||||||
|
if cur_hash and (hunks or added_lines):
|
||||||
|
commits.append(FileDiffCommit(
|
||||||
|
cur_hash, cur_ts, list(hunks), list(added_lines),
|
||||||
|
add_cnt + del_cnt))
|
||||||
|
hunks, added_lines = [], []
|
||||||
|
add_cnt = del_cnt = 0
|
||||||
|
in_hunk = False
|
||||||
|
|
||||||
|
for raw_line in output.splitlines():
|
||||||
|
if raw_line.startswith('commit '):
|
||||||
|
flush()
|
||||||
|
parts = raw_line.split()
|
||||||
|
if len(parts) >= 3:
|
||||||
|
cur_hash, cur_ts = parts[1], int(parts[2])
|
||||||
|
elif raw_line.startswith('@@'):
|
||||||
|
m = HUNK_RE.match(raw_line)
|
||||||
|
if m:
|
||||||
|
os_ = int(m.group(1))
|
||||||
|
oc = int(m.group(2)) if m.group(2) else 1
|
||||||
|
ns = int(m.group(3))
|
||||||
|
nc = int(m.group(4)) if m.group(4) else 1
|
||||||
|
hunks.append((os_, oc, ns, nc))
|
||||||
|
new_counter = ns
|
||||||
|
in_hunk = True
|
||||||
|
elif in_hunk:
|
||||||
|
if raw_line.startswith('+'):
|
||||||
|
added_lines.append(new_counter)
|
||||||
|
new_counter += 1
|
||||||
|
add_cnt += 1
|
||||||
|
elif raw_line.startswith('-'):
|
||||||
|
del_cnt += 1
|
||||||
|
elif raw_line.startswith('\\'):
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
new_counter += 1 # context (shouldn't appear with -U0)
|
||||||
|
flush()
|
||||||
|
return commits
|
||||||
|
|
||||||
|
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Heat computation
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class FileHeat:
|
||||||
|
path: str
|
||||||
|
heat: float
|
||||||
|
last_modified_days: float
|
||||||
|
num_commits: int
|
||||||
|
|
||||||
|
|
||||||
|
def _normalise_dir(d: str) -> str:
|
||||||
|
"""Normalise a directory path for prefix matching: strip ./ and ensure trailing /."""
|
||||||
|
d = d.replace('\\', '/')
|
||||||
|
d = d.lstrip('./')
|
||||||
|
if d and not d.endswith('/'):
|
||||||
|
d += '/'
|
||||||
|
return d
|
||||||
|
|
||||||
|
|
||||||
|
def _is_dir_pattern(pattern: str) -> bool:
|
||||||
|
"""Heuristic: patterns containing '/' are directory prefixes, not extensions."""
|
||||||
|
return '/' in pattern
|
||||||
|
|
||||||
|
|
||||||
|
def _should_exclude(fpath: str, excluded_exts: frozenset,
|
||||||
|
excluded_dirs: Tuple[str, ...] = ()) -> bool:
|
||||||
|
"""Check whether a file path matches any excluded extension or directory."""
|
||||||
|
# Directory prefixes
|
||||||
|
for dp in excluded_dirs:
|
||||||
|
if fpath.startswith(dp):
|
||||||
|
return True
|
||||||
|
# Multi-part extensions (e.g. .min.js)
|
||||||
|
lower = fpath.lower()
|
||||||
|
for ext in excluded_exts:
|
||||||
|
if ext.count('.') > 1 and lower.endswith(ext):
|
||||||
|
return True
|
||||||
|
# Single-part extension
|
||||||
|
_, ext = os.path.splitext(lower)
|
||||||
|
# Extensionless basename match (e.g. LICENSE → .license)
|
||||||
|
basename = os.path.basename(lower)
|
||||||
|
return ext in excluded_exts or ('.' + basename) in excluded_exts
|
||||||
|
|
||||||
|
|
||||||
|
def compute_repo_heat(repo_path: str, *, now: Optional[float] = None,
|
||||||
|
since: Optional[str] = None, top: Optional[int] = None,
|
||||||
|
existing_only: bool = True,
|
||||||
|
directory: Optional[str] = None,
|
||||||
|
exclude: Optional[List[str]] = None,
|
||||||
|
no_default_exclude: bool = False) -> List[FileHeat]:
|
||||||
|
if now is None:
|
||||||
|
now = time.time()
|
||||||
|
file_commits = parse_repo_log(repo_path, since=since)
|
||||||
|
|
||||||
|
current_files = None
|
||||||
|
if existing_only:
|
||||||
|
current_files = set(git(repo_path, 'ls-files').splitlines())
|
||||||
|
|
||||||
|
# Build the set of excluded extensions
|
||||||
|
if no_default_exclude:
|
||||||
|
excluded_exts: frozenset = frozenset()
|
||||||
|
else:
|
||||||
|
excluded_exts = DEFAULT_EXCLUDED_EXTENSIONS
|
||||||
|
|
||||||
|
# Split --exclude items into directory prefixes vs. extensions
|
||||||
|
excluded_dirs: List[str] = []
|
||||||
|
if exclude:
|
||||||
|
extra_exts: List[str] = []
|
||||||
|
for item in exclude:
|
||||||
|
if _is_dir_pattern(item):
|
||||||
|
excluded_dirs.append(_normalise_dir(item))
|
||||||
|
else:
|
||||||
|
extra_exts.append(item if item.startswith('.') else f'.{item}')
|
||||||
|
if extra_exts:
|
||||||
|
excluded_exts = excluded_exts | frozenset(extra_exts)
|
||||||
|
excluded_dirs_t = tuple(excluded_dirs)
|
||||||
|
|
||||||
|
# Normalise --dir prefix
|
||||||
|
dir_prefix: Optional[str] = None
|
||||||
|
if directory:
|
||||||
|
dp = _normalise_dir(directory)
|
||||||
|
dir_prefix = dp if dp else None
|
||||||
|
|
||||||
|
results = []
|
||||||
|
for fpath, commits in file_commits.items():
|
||||||
|
if current_files is not None and fpath not in current_files:
|
||||||
|
continue
|
||||||
|
if dir_prefix is not None and not fpath.startswith(dir_prefix):
|
||||||
|
continue
|
||||||
|
if _should_exclude(fpath, excluded_exts, excluded_dirs_t):
|
||||||
|
continue
|
||||||
|
heat = 0.0
|
||||||
|
last_ts = 0
|
||||||
|
for fc in commits:
|
||||||
|
days = max(0.0, (now - fc.timestamp) / 86400.0)
|
||||||
|
heat += commit_heat(days, fc.added + fc.deleted)
|
||||||
|
last_ts = max(last_ts, fc.timestamp)
|
||||||
|
results.append(FileHeat(
|
||||||
|
fpath, heat,
|
||||||
|
(now - last_ts) / 86400.0 if last_ts else float('inf'),
|
||||||
|
len(commits)))
|
||||||
|
|
||||||
|
results.sort(key=lambda r: r.heat, reverse=True)
|
||||||
|
return results[:top] if top else results
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class LineHeat:
|
||||||
|
line_number: int
|
||||||
|
heat: float
|
||||||
|
content: str
|
||||||
|
|
||||||
|
|
||||||
|
def compute_line_heat(repo_path: str, file_path: str, *,
|
||||||
|
now: Optional[float] = None,
|
||||||
|
since: Optional[str] = None,
|
||||||
|
max_commits: int = 500) -> List[LineHeat]:
|
||||||
|
if now is None:
|
||||||
|
now = time.time()
|
||||||
|
|
||||||
|
full = os.path.join(repo_path, file_path)
|
||||||
|
with open(full, 'r', errors='replace') as f:
|
||||||
|
lines = f.readlines()
|
||||||
|
n = len(lines)
|
||||||
|
if n == 0:
|
||||||
|
return []
|
||||||
|
|
||||||
|
diffs = parse_file_log(repo_path, file_path, since=since)
|
||||||
|
if max_commits:
|
||||||
|
diffs = diffs[:max_commits]
|
||||||
|
|
||||||
|
heat = [0.0] * (n + 1) # 1-indexed
|
||||||
|
|
||||||
|
# version_maps[j] = VersionMap for diffs[j] (maps V_{j+1} → V_j)
|
||||||
|
# V_0 = HEAD
|
||||||
|
version_maps: List[VersionMap] = []
|
||||||
|
|
||||||
|
for i, cd in enumerate(diffs):
|
||||||
|
days = max(0.0, (now - cd.timestamp) / 86400.0)
|
||||||
|
h = commit_heat(days, cd.total_churn)
|
||||||
|
|
||||||
|
for new_line in cd.added_lines:
|
||||||
|
head_line = new_line
|
||||||
|
# Map V_i → V_0 through version_maps[i-1] … version_maps[0]
|
||||||
|
for j in range(i - 1, -1, -1):
|
||||||
|
head_line = version_maps[j].map_line(head_line)
|
||||||
|
if head_line == -1:
|
||||||
|
break
|
||||||
|
if head_line != -1 and 1 <= head_line <= n:
|
||||||
|
heat[head_line] += h
|
||||||
|
|
||||||
|
version_maps.append(VersionMap(cd.hunks))
|
||||||
|
|
||||||
|
return [LineHeat(i + 1, heat[i + 1], lines[i].rstrip('\r\n'))
|
||||||
|
for i in range(n)]
|
||||||
|
|
||||||
|
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Output formatting
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def heat_to_rgb(normalised: float) -> Tuple[int, int, int]:
|
||||||
|
hue = (1.0 - normalised) * 0.667 # blue 240° → red 0°
|
||||||
|
r, g, b = colorsys.hsv_to_rgb(hue, 0.8, 0.95)
|
||||||
|
return int(r * 255), int(g * 255), int(b * 255)
|
||||||
|
|
||||||
|
|
||||||
|
def _bg(r, g, b):
|
||||||
|
return f"\033[48;2;{r};{g};{b}m"
|
||||||
|
|
||||||
|
_RST = "\033[0m"
|
||||||
|
|
||||||
|
|
||||||
|
def format_repo_heat(results: List[FileHeat], color: bool = True) -> str:
|
||||||
|
if not results:
|
||||||
|
return "No files found."
|
||||||
|
mx = max(r.heat for r in results)
|
||||||
|
out = []
|
||||||
|
out.append(f"{'#':>5} {'Heat':>10} {'Cmts':>6} {'Last modified':>14} File")
|
||||||
|
out.append("─" * 90)
|
||||||
|
for i, r in enumerate(results):
|
||||||
|
norm = r.heat / mx if mx else 0
|
||||||
|
hs = f"{r.heat:10.2f}"
|
||||||
|
ds = (f"{r.last_modified_days:.0f}d ago" if r.last_modified_days < 365
|
||||||
|
else f"{r.last_modified_days/365:.1f}y ago")
|
||||||
|
if color:
|
||||||
|
cr, cg, cb = heat_to_rgb(norm)
|
||||||
|
hs = f"{_bg(cr,cg,cb)} {hs} {_RST}"
|
||||||
|
out.append(f"{i+1:>5} {hs} {r.num_commits:>6} {ds:>14} {r.path}")
|
||||||
|
return "\n".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def format_line_heat(results: List[LineHeat], color: bool = True,
|
||||||
|
show_all: bool = False, context: int = 2) -> str:
|
||||||
|
"""Format line-level heat.
|
||||||
|
|
||||||
|
By default only lines with heat > 0 are shown, grouped into snippets
|
||||||
|
with ``context`` surrounding lines and ``...`` separators.
|
||||||
|
Pass ``show_all=True`` to display every line.
|
||||||
|
"""
|
||||||
|
if not results:
|
||||||
|
return "Empty file."
|
||||||
|
mx = max(r.heat for r in results) or 1.0
|
||||||
|
w = len(str(len(results)))
|
||||||
|
|
||||||
|
def fmt(r: LineHeat) -> str:
|
||||||
|
norm = r.heat / mx
|
||||||
|
pct = norm * 100
|
||||||
|
if color:
|
||||||
|
cr, cg, cb = heat_to_rgb(norm)
|
||||||
|
bar = f"{_bg(cr,cg,cb)} {pct:5.1f}% {_RST}"
|
||||||
|
else:
|
||||||
|
bar = f"{pct:6.1f}%"
|
||||||
|
return f"{r.line_number:>{w}} {bar} {r.content}"
|
||||||
|
|
||||||
|
if show_all:
|
||||||
|
return "\n".join(fmt(r) for r in results)
|
||||||
|
|
||||||
|
# Build set of line indices (0-based) to display: hot lines + context
|
||||||
|
hot_indices = {i for i, r in enumerate(results) if r.heat > 0}
|
||||||
|
if not hot_indices:
|
||||||
|
return "No lines with heat > 0."
|
||||||
|
|
||||||
|
visible: set = set()
|
||||||
|
for i in hot_indices:
|
||||||
|
for j in range(max(0, i - context), min(len(results), i + context + 1)):
|
||||||
|
visible.add(j)
|
||||||
|
|
||||||
|
out: list = []
|
||||||
|
prev_i = -2 # sentinel
|
||||||
|
for i in sorted(visible):
|
||||||
|
if i > prev_i + 1:
|
||||||
|
out.append(f"{'':>{w}} {'...'}")
|
||||||
|
out.append(fmt(results[i]))
|
||||||
|
prev_i = i
|
||||||
|
# trailing ellipsis if we cut off before end
|
||||||
|
if max(visible) < len(results) - 1:
|
||||||
|
out.append(f"{'':>{w}} {'...'}")
|
||||||
|
return "\n".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# CLI
|
||||||
|
# ═══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(
|
||||||
|
description="Git Heatmap — code change frequency × recency",
|
||||||
|
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||||
|
epilog="""\
|
||||||
|
examples:
|
||||||
|
python heatmap.py repo --top 20 ./kafka
|
||||||
|
python heatmap.py repo --dir src/main --top 20 ./kafka
|
||||||
|
python heatmap.py repo --exclude xml,proto,src/generated/ ./kafka
|
||||||
|
python heatmap.py repo --no-default-exclude ./kafka
|
||||||
|
python heatmap.py file clients/src/main/java/org/apache/kafka/common/utils/Utils.java ./kafka
|
||||||
|
python heatmap.py file --all --context 5 some/file.py .
|
||||||
|
""")
|
||||||
|
sub = ap.add_subparsers(dest='cmd')
|
||||||
|
|
||||||
|
rp = sub.add_parser('repo', help='file-level heatmap')
|
||||||
|
rp.add_argument('repo_path', nargs='?', default='.')
|
||||||
|
rp.add_argument('--top', type=int, default=30)
|
||||||
|
rp.add_argument('--since', default='2 years ago')
|
||||||
|
rp.add_argument('--dir', default=None,
|
||||||
|
help='limit scan to files under this directory (e.g. src/main)')
|
||||||
|
rp.add_argument('--exclude', default=None,
|
||||||
|
help='comma-separated extensions or directory paths to exclude '
|
||||||
|
'(e.g. xml,proto,src/java/org/apache/cassandra/config/)')
|
||||||
|
rp.add_argument('--no-default-exclude', action='store_true',
|
||||||
|
help='disable default exclusion of non-code files (txt, md, json, …)')
|
||||||
|
rp.add_argument('--no-color', action='store_true')
|
||||||
|
rp.add_argument('--json', action='store_true')
|
||||||
|
|
||||||
|
fp = sub.add_parser('file', help='line-level heatmap')
|
||||||
|
fp.add_argument('file_path')
|
||||||
|
fp.add_argument('repo_path', nargs='?', default='.')
|
||||||
|
fp.add_argument('--since', default='2 years ago')
|
||||||
|
fp.add_argument('--max-commits', type=int, default=500)
|
||||||
|
fp.add_argument('--all', action='store_true', dest='show_all',
|
||||||
|
help='show all lines including 0%% heat (default: snippets only)')
|
||||||
|
fp.add_argument('--context', type=int, default=2,
|
||||||
|
help='lines of context around hot lines in snippet mode (default: 2)')
|
||||||
|
fp.add_argument('--no-color', action='store_true')
|
||||||
|
fp.add_argument('--json', action='store_true')
|
||||||
|
|
||||||
|
args = ap.parse_args()
|
||||||
|
if not args.cmd:
|
||||||
|
ap.print_help()
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
if args.cmd == 'repo':
|
||||||
|
exclude_ext = args.exclude.split(',') if args.exclude else None
|
||||||
|
res = compute_repo_heat(args.repo_path, since=args.since, top=args.top,
|
||||||
|
directory=args.dir,
|
||||||
|
exclude=exclude_ext,
|
||||||
|
no_default_exclude=args.no_default_exclude)
|
||||||
|
if args.json:
|
||||||
|
import json
|
||||||
|
print(json.dumps([{'path': r.path, 'heat': r.heat,
|
||||||
|
'commits': r.num_commits,
|
||||||
|
'last_modified_days': round(r.last_modified_days, 1)}
|
||||||
|
for r in res], indent=2))
|
||||||
|
else:
|
||||||
|
print(format_repo_heat(res, color=not args.no_color))
|
||||||
|
|
||||||
|
elif args.cmd == 'file':
|
||||||
|
res = compute_line_heat(args.repo_path, args.file_path,
|
||||||
|
since=args.since, max_commits=args.max_commits)
|
||||||
|
if args.json:
|
||||||
|
import json
|
||||||
|
print(json.dumps([{'line': r.line_number, 'heat': round(r.heat, 4),
|
||||||
|
'content': r.content} for r in res], indent=2))
|
||||||
|
else:
|
||||||
|
print(format_line_heat(res, color=not args.no_color,
|
||||||
|
show_all=args.show_all,
|
||||||
|
context=args.context))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
|
|
@ -0,0 +1,109 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
TARGET_DIR="${SKILLS_DIR:-$HOME/.claude/skills}"
|
||||||
|
DRY_RUN=false
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<EOF
|
||||||
|
Usage: $(basename "$0") [--target DIR] [--dry-run] [skill ...]
|
||||||
|
|
||||||
|
Installs skills into TARGET_DIR (default: ~/.claude/skills).
|
||||||
|
Overwrites existing skills from this repo; does not touch other skills.
|
||||||
|
|
||||||
|
Options:
|
||||||
|
--target DIR Install to DIR instead of ~/.claude/skills
|
||||||
|
--dry-run Show what would happen without copying anything
|
||||||
|
--help Show this help
|
||||||
|
|
||||||
|
Arguments:
|
||||||
|
skill ... Install only the named skill(s) (default: all)
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
SELECTED_SKILLS=()
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--target)
|
||||||
|
TARGET_DIR="$2"
|
||||||
|
shift 2
|
||||||
|
;;
|
||||||
|
--dry-run)
|
||||||
|
DRY_RUN=true
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--help|-h)
|
||||||
|
usage
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
-*)
|
||||||
|
echo "Unknown option: $1" >&2
|
||||||
|
usage >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
SELECTED_SKILLS+=("$1")
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# Discover all skills in this repo (any directory containing SKILL.md)
|
||||||
|
ALL_SKILLS=()
|
||||||
|
while IFS= read -r skill_file; do
|
||||||
|
skill_dir="$(dirname "$skill_file")"
|
||||||
|
# Only include direct children of SCRIPT_DIR, not nested
|
||||||
|
if [[ "$(dirname "$skill_dir")" == "$SCRIPT_DIR" ]]; then
|
||||||
|
ALL_SKILLS+=("$(basename "$skill_dir")")
|
||||||
|
fi
|
||||||
|
done < <(find "$SCRIPT_DIR" -name "SKILL.md" | sort)
|
||||||
|
|
||||||
|
if [[ ${#SELECTED_SKILLS[@]} -gt 0 ]]; then
|
||||||
|
SKILLS_TO_INSTALL=("${SELECTED_SKILLS[@]}")
|
||||||
|
else
|
||||||
|
SKILLS_TO_INSTALL=("${ALL_SKILLS[@]}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == false ]]; then
|
||||||
|
mkdir -p "$TARGET_DIR"
|
||||||
|
fi
|
||||||
|
|
||||||
|
installed=0
|
||||||
|
skipped=0
|
||||||
|
for skill in "${SKILLS_TO_INSTALL[@]}"; do
|
||||||
|
src="$SCRIPT_DIR/$skill"
|
||||||
|
dst="$TARGET_DIR/$skill"
|
||||||
|
|
||||||
|
if [[ ! -d "$src" ]] || [[ ! -f "$src/SKILL.md" ]]; then
|
||||||
|
echo "warning: '$skill' not found in $SCRIPT_DIR, skipping" >&2
|
||||||
|
((skipped++)) || true
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
if [[ -d "$dst" ]]; then
|
||||||
|
echo "would overwrite $dst"
|
||||||
|
else
|
||||||
|
echo "would install $dst"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
if [[ -d "$dst" ]]; then
|
||||||
|
rm -rf "$dst"
|
||||||
|
echo "overwriting $skill"
|
||||||
|
else
|
||||||
|
echo "installing $skill"
|
||||||
|
fi
|
||||||
|
cp -r "$src" "$dst"
|
||||||
|
fi
|
||||||
|
((installed++)) || true
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
echo "${installed} skill(s) would be installed to $TARGET_DIR"
|
||||||
|
if [[ $skipped -gt 0 ]]; then echo "${skipped} skill(s) skipped (not found)"; fi
|
||||||
|
else
|
||||||
|
echo "${installed} skill(s) installed to $TARGET_DIR"
|
||||||
|
if [[ $skipped -gt 0 ]]; then echo "${skipped} skill(s) skipped (not found)"; fi
|
||||||
|
fi
|
||||||
|
|
@ -0,0 +1,405 @@
|
||||||
|
---
|
||||||
|
name: mega-review
|
||||||
|
version: "1.0.0"
|
||||||
|
description: >
|
||||||
|
Multi-pass deep code review for large patches (1000+ LOC) that maximizes real bug detection.
|
||||||
|
Orchestrates targeted-review, shallow-review, and deep-review in parallel across all HIGH and
|
||||||
|
MEDIUM risk files and commits: understands the feature holistically, splits by file and commit,
|
||||||
|
runs deep review on every HIGH/MEDIUM file, targeted review across the same scope, and
|
||||||
|
per-commit shallow review, followed by cross-cut consistency checks. Use when: reviewing a
|
||||||
|
large patch (feature branch, multi-commit, or single large diff, 1000+ LOC), doing a thorough
|
||||||
|
pre-merge review, or when shallower reviews miss bugs due to patch size. Triggers on: "review
|
||||||
|
this branch", "review these commits", "review this feature", "mega review", "thorough review",
|
||||||
|
"full review of X commits", or when the user specifies a commit range or feature for review.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Mega-Review: Multi-Pass Large Patch Review
|
||||||
|
|
||||||
|
This skill orchestrates `/targeted-review`, `/shallow-review`, and `/deep-review` with
|
||||||
|
cross-cut consistency checks to maximize bug detection on large patches (1000+ LOC). The input
|
||||||
|
can be a commit range, a feature branch, or any large diff — it does not have to span multiple
|
||||||
|
commits.
|
||||||
|
|
||||||
|
**You MUST invoke `/targeted-review`, `/shallow-review`, and `/deep-review` as part of this
|
||||||
|
workflow.** Those skills have specialist checklists, pattern catalogs, and structured review
|
||||||
|
processes that this skill does not duplicate. This skill's job is to understand the change
|
||||||
|
holistically, decompose it into reviewable chunks, and delegate the actual code review to the
|
||||||
|
appropriate sub-skills at the right granularity.
|
||||||
|
|
||||||
|
## Why This Exists
|
||||||
|
|
||||||
|
A single pass of `/shallow-review` over 7000 LOC spreads attention too thin. Real bugs hide in:
|
||||||
|
- **Offset arithmetic** that works for the simple case but breaks when offset params are added
|
||||||
|
- **Variable lifecycle issues** where a loop modifies a variable but the final operation needs the original
|
||||||
|
- **Cross-cut inconsistencies** where fixes or refactors in one subsystem introduce bugs in another
|
||||||
|
- **Algorithmic boundary conditions** in new data structure code
|
||||||
|
|
||||||
|
The key insight: some bugs are findable by mechanical checking (every System.arraycopy, every
|
||||||
|
variable swap), while others require deep understanding. This skill does BOTH.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
+---------------------+
|
||||||
|
| PHASE 1: Decompose |
|
||||||
|
| & Classify | Produce decomposition map + coverage checklist.
|
||||||
|
+----------+----------+
|
||||||
|
|
|
||||||
|
+-------+--------+----------+
|
||||||
|
| | |
|
||||||
|
+---v----+ +-------v------+ +-v-----------+
|
||||||
|
|PHASE 2 | | PHASE 3 | | PHASE 4 |
|
||||||
|
|Deep | | Targeted | | Shallow |
|
||||||
|
|Review | | Review | | Review |
|
||||||
|
|(HIGH + | | (HIGH + | | (HIGH + |
|
||||||
|
| MEDIUM | | MEDIUM | | MEDIUM |
|
||||||
|
| files) | | files) | | commits) |
|
||||||
|
+---+----+ +-------+------+ +-+-----------+
|
||||||
|
| | |
|
||||||
|
+----------------+------------+
|
||||||
|
|
|
||||||
|
+----------v----------+
|
||||||
|
| PHASE 5: Cross-Cut | Cross-commit/cross-subsystem consistency, pattern
|
||||||
|
| Passes | amplification, and fix verification checks.
|
||||||
|
+----------+----------+ One subagent per pass type (5A / 5B / 5C).
|
||||||
|
|
|
||||||
|
+----------v----------+
|
||||||
|
| PHASE 6: Merge & | Deduplicate, verify plausibility, rank.
|
||||||
|
| Validate | Combine findings from ALL phases.
|
||||||
|
+---------------------+
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Chunking Strategy
|
||||||
|
|
||||||
|
This skill's core principle: **the orchestrator holds global state; subagents hold only their
|
||||||
|
slice.** A large patch reviewed by a single agent suffers attention dilution — its context fills
|
||||||
|
with earlier files, earlier findings, and earlier grep output, and it starts missing things. The
|
||||||
|
fix is to reset context at every subagent boundary.
|
||||||
|
|
||||||
|
### Orchestrator responsibilities
|
||||||
|
- Holds the full patch, decomposition map, and accumulated findings from all phases
|
||||||
|
- Before spawning each subagent, assembles a **briefing packet**: the chunk + a tight 100–200
|
||||||
|
word context summary written by the orchestrator
|
||||||
|
- After each subagent returns, integrates findings into the global picture
|
||||||
|
|
||||||
|
### What each subagent receives
|
||||||
|
- **Its chunk**: the diff for its commit / grep hits for its audit type / file diff for its deep pass
|
||||||
|
- **Orchestrator context**: subsystem name, risk level, what the code is trying to do (one paragraph)
|
||||||
|
- **Focused task**: exactly what to look for and why — not a generic "find bugs"
|
||||||
|
- **Relevant prior findings only**: findings from earlier phases that touch THIS chunk's files,
|
||||||
|
not the full findings list from all files
|
||||||
|
|
||||||
|
### What each subagent does
|
||||||
|
Each subagent **invokes a review skill** on its chunk — it is not a generic agent. The skill
|
||||||
|
determines the review depth and structure. The chunk is the scope; the briefing packet is the
|
||||||
|
context. Subagents may read additional source files, grep for callers, or check sibling classes
|
||||||
|
to inform the review, but must not receive the full patch as input — that defeats the purpose.
|
||||||
|
|
||||||
|
### Chunk boundaries by phase
|
||||||
|
|
||||||
|
| Phase | Chunk unit | Skill to invoke | What the subagent receives |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 2 (Deep) | One HIGH or MEDIUM file | `/deep-review` | That file's diff + full current source |
|
||||||
|
| 3 (Targeted) | All HIGH + MEDIUM files as a group | `/targeted-review` | Diff for those files + Phase 1 risk classification |
|
||||||
|
| 4 (Shallow) | One HIGH or MEDIUM commit | `/shallow-review` | That commit's diff only |
|
||||||
|
| 5 (Cross-cut) | One pass type (5A / 5B / 5C) | inline only — no sub-skill | Finding list relevant to that pass + grep targets |
|
||||||
|
|
||||||
|
**Phases 2, 3, and 4 run in parallel** — launch all their subagents in a single message after Phase 1 completes. Do not wait for Phase 2 before starting Phase 3 or 4.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 1: Decompose & Classify
|
||||||
|
|
||||||
|
**This phase is done inline — no sub-skill invocation.** Its job is to produce a decomposition
|
||||||
|
map and coverage checklist fast, then immediately hand off to the parallel Phases 2/3/4.
|
||||||
|
Every minute spent in Phase 1 is a minute not spent finding bugs.
|
||||||
|
|
||||||
|
**Determining the input scope** — two modes depending on what the user provided:
|
||||||
|
|
||||||
|
*Mode A — git boundary given* (commit range, branch, or single SHA): proceed directly to
|
||||||
|
Step 1a.
|
||||||
|
|
||||||
|
*Mode B — no git boundary* (user describes a feature, subsystem, or set of files without
|
||||||
|
specifying a range):
|
||||||
|
1. Discover candidate files via grep or directory inspection:
|
||||||
|
```bash
|
||||||
|
grep -rl "<keyword>" src/ | grep '\.java$' | sort > /tmp/file_list.txt
|
||||||
|
# or: find src/java/org/apache/cassandra/<package> -name '*.java' | sort
|
||||||
|
```
|
||||||
|
2. **Show the list to the user and ask for confirmation** before going further — file count,
|
||||||
|
file names, anything obviously missing or out of scope. Do not skip this step.
|
||||||
|
3. Produce a working diff against a sensible base (usually `main`):
|
||||||
|
```bash
|
||||||
|
git diff main -- $(cat /tmp/file_list.txt | tr '\n' ' ') > /tmp/mega_review.diff
|
||||||
|
# For files with no git history (new/untracked), read them directly — there is no diff.
|
||||||
|
```
|
||||||
|
4. Use this confirmed file list as the direct seed for the coverage checklist in Step 1e —
|
||||||
|
paste the files into the `File` column verbatim.
|
||||||
|
5. In Phase 4, there are no commit boundaries: chunk by **file** instead of by commit —
|
||||||
|
one `/shallow-review` invocation per file. Skip Step 1a and go straight to Step 1b.
|
||||||
|
|
||||||
|
**Step 1a — Get commit shape:**
|
||||||
|
```bash
|
||||||
|
git log --oneline --stat <base>..<head>
|
||||||
|
# Or for a single commit: git show --stat <SHA>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 1b — Skim diff stats (do NOT read file bodies):**
|
||||||
|
```bash
|
||||||
|
git diff <base>..<head> --stat
|
||||||
|
# What to note: file names, extension types, rough LOC changes, new vs modified files
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 1c — Classify each commit/subsystem** by risk:
|
||||||
|
- **HIGH**: New algorithms, data structure rewrites, complex feature additions, new orchestration layers, state machine changes, bootstrap/recovery logic
|
||||||
|
- **MEDIUM**: Feature additions with new control flow, non-trivial interface changes with multiple callers
|
||||||
|
- **LOW**: Renames, moves, mechanical refactors, test-only or documentation changes
|
||||||
|
|
||||||
|
**Step 1d — Classify each source file** by type:
|
||||||
|
- **Algorithmic**: Array operations, tree traversals, binary search, merges, set operations
|
||||||
|
- **State machine**: Bootstrap, recovery, catchup, topology, lifecycle management
|
||||||
|
- **Wiring**: Delegation, API changes, parameter threading
|
||||||
|
- **Concurrency**: Shared state, synchronization
|
||||||
|
- **Documentation/Config**: Markdown, YAML, config files, skill definitions
|
||||||
|
|
||||||
|
**Step 1e — Write `/tmp/mega_coverage.md`** (the coverage checklist). Every changed non-test
|
||||||
|
file gets one row; pre-assign reviewers now based on risk:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
| File | LOC changed | Risk | Type | Assigned reviewers | Min required | Covered? |
|
||||||
|
|------|-------------|------|------|--------------------|--------------|----------|
|
||||||
|
| src/Foo.java | 320 | HIGH | Algorithmic | Phase 2 (deep), Phase 3 (targeted) | 2 | — |
|
||||||
|
| src/Bar.java | 45 | MEDIUM | Wiring | Phase 2 (deep), Phase 3 (targeted) | 2 | — |
|
||||||
|
| docs/SKILL.md | 18 | LOW | Documentation | Phase 4 (sha: def456) | 1 | — |
|
||||||
|
```
|
||||||
|
|
||||||
|
Assignment rules:
|
||||||
|
- **HIGH-risk file** → Phase 2 (deep) + Phase 3 (targeted) — min 2; also covered by Phase 4
|
||||||
|
- **MEDIUM-risk file** → Phase 2 (deep) + Phase 3 (targeted) — min 2; also covered by Phase 4
|
||||||
|
- **LOW-risk file** → Phase 4 (shallow) — min 1
|
||||||
|
|
||||||
|
**Step 1f — IMMEDIATELY start Phases 2, 3, and 4.** Do not pause to write analysis. Do not
|
||||||
|
invoke patch-explainer or any other sub-skill. The coverage checklist IS the Phase 1 output.
|
||||||
|
In the same response, spawn all Phase 2/3/4 subagents — see "Launching Phases 2/3/4" below.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 2: Deep Review — invoke `/deep-review`
|
||||||
|
|
||||||
|
For every HIGH- and MEDIUM-risk file, invoke `/deep-review` scoped to that file. Each file
|
||||||
|
gets a **separate, fresh subagent** — they run in parallel and share no context with each other.
|
||||||
|
This phase runs **in parallel with Phases 3 and 4** — start it as soon as Phase 1 completes.
|
||||||
|
|
||||||
|
**Per-file input packet** (what the orchestrator provides to each `/deep-review` invocation):
|
||||||
|
- The diff for that file only: `git diff <base>..<head> -- <file>`
|
||||||
|
- The file's full current source (deep-review reads this itself, but point it at the right file)
|
||||||
|
- A 100–150 word orchestrator briefing:
|
||||||
|
- File classification (algorithmic / state machine / wiring / concurrency)
|
||||||
|
- What the file does in the context of the overall change (1–2 sentences from Phase 1)
|
||||||
|
- Risk level and why it was classified HIGH or MEDIUM
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 3: Targeted Review — invoke `/targeted-review`
|
||||||
|
|
||||||
|
Invoke the `/targeted-review` skill scoped to **all HIGH- and MEDIUM-risk files** identified
|
||||||
|
in Phase 1 — not the full patch. Targeted-review is designed for focused review: it picks only
|
||||||
|
the bug-pattern categories whose diff signals match, then spawns tight subagents per focus area.
|
||||||
|
Feeding it the full patch dilutes that focus; feeding it the files that matter most lets it go
|
||||||
|
deep where it counts.
|
||||||
|
|
||||||
|
**Selecting the scope**:
|
||||||
|
- Include all files classified HIGH- or MEDIUM-risk in Phase 1
|
||||||
|
- Include files touched in multiple commits (cross-commit modification from the decomposition map)
|
||||||
|
- Exclude LOW-risk files, test-only files, and pure documentation/config changes
|
||||||
|
|
||||||
|
**When invoking `/targeted-review`**:
|
||||||
|
- Pass only the diff for the selected files (not the full patch)
|
||||||
|
- Include the Phase 1 risk classification and file-type notes as context so targeted-review
|
||||||
|
subagents understand what to focus on
|
||||||
|
|
||||||
|
This phase runs **in parallel with Phases 2 and 4** — start it as soon as Phase 1 completes.
|
||||||
|
Collect all findings and tag each with `Phase 3 (targeted-review)` and its subagent focus.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 4: Per-Commit Shallow Reviews — invoke `/shallow-review`
|
||||||
|
|
||||||
|
For each HIGH- and MEDIUM-risk commit, invoke `/shallow-review` scoped to that commit's diff.
|
||||||
|
Each invocation is a **fresh subagent** — it receives only that commit's diff, not the full
|
||||||
|
patch or accumulated findings from other commits.
|
||||||
|
|
||||||
|
This phase runs **in parallel with Phases 2 and 3** — start it as soon as Phase 1 completes.
|
||||||
|
|
||||||
|
**Step 4a**: Extract per-commit diff to its own file:
|
||||||
|
```bash
|
||||||
|
git diff <SHA>~1..<SHA> > /tmp/patch_<SHA>.diff
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 4b**: Assemble a briefing packet for `/shallow-review` containing ONLY:
|
||||||
|
- The commit's diff (`/tmp/patch_<SHA>.diff`)
|
||||||
|
- A 100–150 word orchestrator briefing covering:
|
||||||
|
- What category the commit is (algorithmic, state machine, wiring, concurrency)
|
||||||
|
- What risk level (HIGH/MEDIUM) and why
|
||||||
|
- For "Fix" commits: "This commit says 'Fix X' — verify the fix is correct, complete,
|
||||||
|
and does not introduce new bugs. Fix commits have a higher-than-average bug rate."
|
||||||
|
|
||||||
|
**Step 4c**: Run all HIGH- and MEDIUM-risk commit reviews in a single parallel batch. Collect
|
||||||
|
findings and tag each with the commit SHA.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 5: Cross-Cut Passes
|
||||||
|
|
||||||
|
**Dispatch 5A, 5B, and 5C as parallel subagents.** Each receives a focused slice of findings
|
||||||
|
and a targeted task — not the full findings dump from all prior phases. The orchestrator
|
||||||
|
pre-filters: each subagent receives only the findings relevant to its pass.
|
||||||
|
|
||||||
|
### 5A: Pattern Amplification
|
||||||
|
|
||||||
|
Patterns come entirely from **bugs actually found in Phases 2–4** — there is no pre-loaded catalog.
|
||||||
|
If Phases 2–4 found zero bugs, 5A is skipped.
|
||||||
|
|
||||||
|
**Orchestrator's job** (done before spawning anything):
|
||||||
|
1. Collect all findings from Phases 2–4
|
||||||
|
2. For each distinct bug, characterize the structural pattern — not the specific instance, but
|
||||||
|
the general shape: e.g. "method accepts a `start` offset parameter, but an internal
|
||||||
|
`arraycopy` call ignores it and uses hardcoded `0`"
|
||||||
|
3. Decide the search scope based on the pattern type:
|
||||||
|
- **Patch-only** — pattern is structural and specific to new code introduced in this PR
|
||||||
|
(new algorithm, new data structure, new method overload family). Unlikely to exist elsewhere.
|
||||||
|
- **Codebase-wide** — pattern is a convention violation or contract bug that could affect
|
||||||
|
any code following the same convention (serialization encoding, API return value contracts,
|
||||||
|
lifecycle protocols, error code handling). The same mistake recurs wherever the convention
|
||||||
|
is followed, including in code not touched by this PR.
|
||||||
|
4. Grep at the chosen scope for locations that could match the pattern shape
|
||||||
|
5. Spawn one subagent per pattern, passing it: the pattern description + the grep hits + the
|
||||||
|
scope used (so it knows whether to look beyond the patch if it finds new leads)
|
||||||
|
|
||||||
|
**Each subagent's job**: verify whether each grep hit is actually the same bug — read the
|
||||||
|
surrounding source as needed. Report confirmed additional instances only.
|
||||||
|
|
||||||
|
One confirmed bug → one pattern → potentially many additional instances found. This is the
|
||||||
|
highest-yield step when Phases 2–4 produce findings.
|
||||||
|
|
||||||
|
### 5B: Cross-Commit/Cross-Subsystem Consistency
|
||||||
|
|
||||||
|
**Subagent input**: The decomposition map from Phase 1 (file list, method signatures changed,
|
||||||
|
rename list) + the list of caller sites extracted by the orchestrator. Not the full patch.
|
||||||
|
|
||||||
|
1. **Interface contracts**: For each changed method signature, verify ALL callers updated
|
||||||
|
2. **Rename completeness**: `grep` for old names that should no longer appear
|
||||||
|
3. **Assumption conflicts**: When one part of the patch assumes X and another assumes Y about the same state
|
||||||
|
|
||||||
|
### 5C: Fix Verification
|
||||||
|
|
||||||
|
**Subagent input**: One subagent per "Fix" commit. Each receives: that commit's diff only +
|
||||||
|
a one-paragraph description of the bug it claims to fix.
|
||||||
|
|
||||||
|
For each fix commit:
|
||||||
|
1. Read the fix carefully
|
||||||
|
2. Try to construct an input that would still trigger the original bug
|
||||||
|
3. Try to construct an input that triggers a new bug introduced by the fix
|
||||||
|
|
||||||
|
### 5D: Optional Additional Review Passes
|
||||||
|
|
||||||
|
If Phases 2–5C revealed areas not yet covered, run additional targeted passes:
|
||||||
|
|
||||||
|
- If 5A found a pattern class in files not yet reviewed → `/deep-review` on those files with
|
||||||
|
the specific pattern as focus guidance
|
||||||
|
- If 5B found cross-cut inconsistencies in files not yet reviewed → `/shallow-review` on the
|
||||||
|
combined diff of those files
|
||||||
|
- If a fix commit (5C) looks suspicious → `/deep-review` on that file with guidance:
|
||||||
|
"Verify this fix is correct and does not introduce new bugs."
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Coverage Gate — Verify Before Phase 6
|
||||||
|
|
||||||
|
Before merging findings, verify `/tmp/mega_coverage.md` is fully green.
|
||||||
|
|
||||||
|
**Step 1 — Mark completions**: For each file row, confirm every assigned reviewer phase ran
|
||||||
|
and reported (a "no findings" report counts). Update **Covered?**:
|
||||||
|
- `✓` — all assigned reviewers completed
|
||||||
|
- `GAP` — one or more assigned reviewers did not run
|
||||||
|
|
||||||
|
**Step 2 — Fill gaps**: For every `GAP` row, launch additional review passes now:
|
||||||
|
- File with 0 reviewers completed → `/shallow-review` on that file's diff
|
||||||
|
- HIGH- or MEDIUM-risk file with only 1 of 2 reviewers completed → `/shallow-review` on that file's diff
|
||||||
|
|
||||||
|
Run all gap-fill passes in a single parallel batch, then update the checklist.
|
||||||
|
|
||||||
|
**Step 3 — Gate**: Do not proceed to Phase 6 until every row in `/tmp/mega_coverage.md`
|
||||||
|
shows `✓`. If gaps remain after one gap-fill pass, repeat Step 2.
|
||||||
|
|
||||||
|
**Step 4 — Include the final checklist** in the Phase 6 report under a "Coverage" section.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 6: Merge & Validate Findings
|
||||||
|
|
||||||
|
### Deduplication
|
||||||
|
Same location + same root cause = one finding. Boost confidence when found by multiple phases.
|
||||||
|
|
||||||
|
### Plausibility Validation
|
||||||
|
|
||||||
|
Every finding MUST pass ALL of:
|
||||||
|
1. **The code exists**: Point to exact file:line in the current code
|
||||||
|
2. **The bug is achievable**: Describe a concrete input that triggers it
|
||||||
|
3. **The result is wrong**: Explain what the code produces vs what it should produce
|
||||||
|
4. **Tests don't catch it**: Explain why (e.g., "tests only call the simple overload without offsets")
|
||||||
|
|
||||||
|
Discard findings that fail any of these.
|
||||||
|
|
||||||
|
### Confidence Ranking
|
||||||
|
- **High**: Found by Phase 2 (deep review), or found by 2+ phases independently
|
||||||
|
- **Medium**: Single phase found it, plausible trigger exists but complex to verify
|
||||||
|
- **Low**: Pattern match suggests possible bug but trigger is speculative
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Report Format
|
||||||
|
|
||||||
|
```
|
||||||
|
## Mega-Review: [commit range / feature name]
|
||||||
|
|
||||||
|
### Summary
|
||||||
|
- [N] commits/subsystems, [M] files, [L] LOC reviewed
|
||||||
|
- [X] bugs found: [H] high-confidence, [M] medium, [L] low
|
||||||
|
|
||||||
|
### Findings (ranked by confidence)
|
||||||
|
|
||||||
|
#### [HIGH] Finding 1: [precise title]
|
||||||
|
- **Location**: file:line
|
||||||
|
- **Found by**: Phase [N] ([audit/review type])
|
||||||
|
- **The bug**: [what the code does wrong, precisely]
|
||||||
|
- **Trigger**: [specific input that causes it]
|
||||||
|
- **Expected**: [what should happen]
|
||||||
|
- **Actual**: [what happens instead]
|
||||||
|
- **Why tests miss it**: [explanation]
|
||||||
|
- **Fix**: [specific code change]
|
||||||
|
|
||||||
|
[... more findings ...]
|
||||||
|
|
||||||
|
### Phase Coverage
|
||||||
|
| Phase | Scope | Findings |
|
||||||
|
|---|---|---|
|
||||||
|
| 1: Decompose & classify | git log + stat | (coverage map) |
|
||||||
|
| 2: Deep review | [N] HIGH+MEDIUM files | [M] bugs |
|
||||||
|
| 3: Targeted review | HIGH+MEDIUM files, [N] foci | [M] bugs |
|
||||||
|
| 4: Shallow review | [N] HIGH+MEDIUM commits | [M] bugs |
|
||||||
|
| 5A: Pattern amplification | [N] patterns | [M] additional bugs |
|
||||||
|
| 5B: Cross-cut | [checks] | [M] bugs |
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Anti-Patterns (What NOT to Report)
|
||||||
|
|
||||||
|
- **Style/naming issues** — Not bugs.
|
||||||
|
- **Performance concerns** — Not bugs (unless they cause incorrect results).
|
||||||
|
- **"Might be wrong" without a trigger** — If you can't describe a concrete input that causes
|
||||||
|
incorrect output, it's not a finding.
|
||||||
|
- **Documentation mismatches** — Only report if the mismatch causes callers to use the API wrong.
|
||||||
|
|
@ -0,0 +1,271 @@
|
||||||
|
---
|
||||||
|
name: patch-explainer
|
||||||
|
version: "1.0.0"
|
||||||
|
description: Deep code analysis with ASCII visualizations showing structure, flow, and state transitions. Use when analyzing patches/diffs, explaining classes or subsystems, understanding code architecture, reviewing changes for inconsistencies, or when asked to visualize how code works. Provides before/after diagrams, data/control flow, concurrency analysis, assumptions, and failure modes. Triggers on explain this patch/code/class, how does X work, show me the flow, visualize this change, code review requests, or proactive analysis during PR reviews.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Patch Explainer
|
||||||
|
|
||||||
|
Deeply analyze and visualize code using ASCII diagrams to reveal structure, behavior, and interactions.
|
||||||
|
|
||||||
|
## What This Skill Does
|
||||||
|
|
||||||
|
Provides comprehensive code analysis with rich ASCII visualizations for:
|
||||||
|
- **Patches/diffs**: Before/after states, what changed and why
|
||||||
|
- **Classes**: Purpose, structure, state transitions, workflows
|
||||||
|
- **Subsystems**: Component interactions, data flow, architecture
|
||||||
|
- **Repositories**: Overall structure, key abstractions, major patterns
|
||||||
|
- **Code reviews**: Finding inconsistencies, analyzing assumptions and failure modes
|
||||||
|
|
||||||
|
## Analysis Approach
|
||||||
|
|
||||||
|
### For Any Code (Patches, Classes, Subsystems, Repositories)
|
||||||
|
|
||||||
|
1. **Identify the core purpose** - What does this code do and why does it exist?
|
||||||
|
|
||||||
|
2. **Create ASCII visualizations** showing:
|
||||||
|
- Structure (components, layers, dependencies)
|
||||||
|
- Behavior (data flow, control flow, state transitions)
|
||||||
|
- Interactions (sequence diagrams, message passing, concurrency)
|
||||||
|
- For patches: before/after comparisons
|
||||||
|
|
||||||
|
3. **Deep reasoning** about:
|
||||||
|
- **Assumptions**: What must be true for this to work correctly?
|
||||||
|
- **Failure modes**: What could go wrong at each step?
|
||||||
|
- **Concurrency**: If multiple threads/processes, show their interactions
|
||||||
|
- **Invariants**: What must always hold true?
|
||||||
|
|
||||||
|
4. **Focus on the "why"**:
|
||||||
|
- Why does this code exist?
|
||||||
|
- Why was this approach chosen?
|
||||||
|
- For changes: Why this modification? What problem does it solve?
|
||||||
|
|
||||||
|
### Visualization Strategy
|
||||||
|
|
||||||
|
Choose diagram types based on what you're explaining:
|
||||||
|
|
||||||
|
**State machines** - When code manages states/status:
|
||||||
|
```
|
||||||
|
[Initial] --event--> [Processing] --success--> [Complete]
|
||||||
|
|
|
||||||
|
failure
|
||||||
|
|
|
||||||
|
v
|
||||||
|
[Failed]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Data flow** - When showing information movement:
|
||||||
|
```
|
||||||
|
Input → [Transform A] → [Transform B] → Output
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sequence diagrams** - When showing component interactions:
|
||||||
|
```
|
||||||
|
Client Server Database
|
||||||
|
| | |
|
||||||
|
|--request----->| |
|
||||||
|
| |----query------->|
|
||||||
|
| |<---result-------|
|
||||||
|
|<--response----| |
|
||||||
|
```
|
||||||
|
|
||||||
|
**Component structure** - When showing architecture:
|
||||||
|
```
|
||||||
|
┌─────────────────────┐
|
||||||
|
│ Application │
|
||||||
|
└──────────┬──────────┘
|
||||||
|
|
|
||||||
|
┌──────────┴──────────┐
|
||||||
|
│ Service Layer │
|
||||||
|
└──────────┬──────────┘
|
||||||
|
|
|
||||||
|
┌──────────┴──────────┐
|
||||||
|
│ Data Layer │
|
||||||
|
└─────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**Before/after** - When explaining changes:
|
||||||
|
```
|
||||||
|
BEFORE AFTER
|
||||||
|
─────────────────────────────────
|
||||||
|
[A] → [B] → [C] [A] → [Cache?] ─Yes→ [C]
|
||||||
|
|
|
||||||
|
No
|
||||||
|
v
|
||||||
|
[B] → [C]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Reference Materials
|
||||||
|
|
||||||
|
**ASCII diagram templates**: See [ascii_patterns.md](references/ascii_patterns.md) for:
|
||||||
|
- State machine patterns
|
||||||
|
- Data flow diagrams
|
||||||
|
- Sequence diagrams
|
||||||
|
- Component/architecture diagrams
|
||||||
|
- Before/after comparisons
|
||||||
|
- Concurrency patterns
|
||||||
|
- Control flow patterns
|
||||||
|
- Dependency graphs
|
||||||
|
|
||||||
|
**Analysis methodology**: See [analysis_framework.md](references/analysis_framework.md) for:
|
||||||
|
- Structured analysis process for patches vs existing code
|
||||||
|
- Concurrency analysis checklist
|
||||||
|
- Assumption identification patterns
|
||||||
|
- Failure mode identification
|
||||||
|
- Analysis depth guidelines by scope (function → class → subsystem → repository)
|
||||||
|
|
||||||
|
## Analysis Workflow
|
||||||
|
|
||||||
|
### For Patches/Diffs
|
||||||
|
|
||||||
|
1. Read the patch to understand what changed
|
||||||
|
2. Identify the change type (bug fix, feature, refactor, optimization)
|
||||||
|
3. Create before/after ASCII diagrams showing:
|
||||||
|
- Old behavior vs new behavior
|
||||||
|
- State transitions that changed
|
||||||
|
- Data flow modifications
|
||||||
|
4. Explain what fundamentally changed:
|
||||||
|
- What problem did the old code have?
|
||||||
|
- How does the new code solve it?
|
||||||
|
- What assumptions changed?
|
||||||
|
5. Analyze failure modes:
|
||||||
|
- What could go wrong with this change?
|
||||||
|
- Are there edge cases not handled?
|
||||||
|
- Concurrency implications?
|
||||||
|
6. Focus on the "why": What problem does each modification solve?
|
||||||
|
|
||||||
|
### For Classes
|
||||||
|
|
||||||
|
1. Read the class to understand its purpose
|
||||||
|
2. Create ASCII diagram showing:
|
||||||
|
- Class structure (key fields/methods)
|
||||||
|
- State machine if it manages state
|
||||||
|
- How it fits with related classes
|
||||||
|
3. Explain key workflows (2-3 most important methods)
|
||||||
|
4. Identify assumptions and invariants
|
||||||
|
5. Analyze failure modes and edge cases
|
||||||
|
6. Focus on the "why": Why does this class exist? What problem does it solve?
|
||||||
|
|
||||||
|
### For Subsystems
|
||||||
|
|
||||||
|
1. Identify the components in the subsystem
|
||||||
|
2. Create ASCII diagram showing:
|
||||||
|
- Component relationships
|
||||||
|
- Key interfaces/boundaries
|
||||||
|
- Main data flows
|
||||||
|
3. Trace 2-3 key workflows through the subsystem
|
||||||
|
4. Explain how components interact
|
||||||
|
5. Identify assumptions across component boundaries
|
||||||
|
6. Focus on the "why": What is this subsystem's role in the larger system?
|
||||||
|
|
||||||
|
### For Repositories
|
||||||
|
|
||||||
|
1. Understand the high-level architecture
|
||||||
|
2. Create ASCII diagram showing:
|
||||||
|
- Major modules/packages
|
||||||
|
- Layered architecture
|
||||||
|
- Key abstractions
|
||||||
|
3. Identify core workflows
|
||||||
|
4. Explain main patterns used throughout
|
||||||
|
5. Highlight critical subsystems
|
||||||
|
6. Focus on the "why": What problem does this codebase solve?
|
||||||
|
|
||||||
|
## Concurrency Analysis
|
||||||
|
|
||||||
|
When analyzing concurrent code, always show:
|
||||||
|
|
||||||
|
**Thread interactions**:
|
||||||
|
```
|
||||||
|
Thread A State Thread B
|
||||||
|
| | |
|
||||||
|
|--lock()------------>| |
|
||||||
|
| [LOCKED] |
|
||||||
|
|--modify()---------->| |
|
||||||
|
|--unlock()---------->| |
|
||||||
|
| [UNLOCKED] |
|
||||||
|
| |<---------lock()-----|
|
||||||
|
| [LOCKED] |
|
||||||
|
```
|
||||||
|
|
||||||
|
**Race conditions**:
|
||||||
|
```
|
||||||
|
Thread A Thread B
|
||||||
|
| |
|
||||||
|
|--read(x=10) |
|
||||||
|
| |--read(x=10)
|
||||||
|
|--x=x+1 |
|
||||||
|
| |--x=x+1
|
||||||
|
|--write(x=11) |
|
||||||
|
| |--write(x=11) ⚠ Lost update!
|
||||||
|
|
||||||
|
Result: x=11 (expected: x=12)
|
||||||
|
```
|
||||||
|
|
||||||
|
Check for:
|
||||||
|
- Shared mutable state without synchronization
|
||||||
|
- Lock ordering violations (potential deadlocks)
|
||||||
|
- Race conditions in read-modify-write operations
|
||||||
|
- Missing memory barriers
|
||||||
|
|
||||||
|
## Output Structure
|
||||||
|
|
||||||
|
Structure your analysis as:
|
||||||
|
|
||||||
|
### 1. Executive Summary
|
||||||
|
Brief overview (2-3 sentences) of what this code does and why it matters.
|
||||||
|
|
||||||
|
### 2. Visual Overview
|
||||||
|
High-level ASCII diagram showing the main structure or flow.
|
||||||
|
|
||||||
|
### 3. Detailed Analysis
|
||||||
|
Organized by aspect:
|
||||||
|
- **Purpose**: What it does and why it exists
|
||||||
|
- **Structure**: Components and their relationships
|
||||||
|
- **Key Workflows**: Step-by-step execution of main scenarios
|
||||||
|
- **State Management**: States and transitions (if applicable)
|
||||||
|
- **Assumptions**: What must be true for correctness
|
||||||
|
- **Failure Modes**: What could go wrong
|
||||||
|
- **Concurrency**: Thread safety analysis (if applicable)
|
||||||
|
|
||||||
|
### 4. For Patches: Before/After
|
||||||
|
- Before state (with diagram)
|
||||||
|
- After state (with diagram)
|
||||||
|
- Why this change was made
|
||||||
|
- Critical modifications explained
|
||||||
|
|
||||||
|
### 5. Key Insights
|
||||||
|
Bullet points highlighting:
|
||||||
|
- Most important findings
|
||||||
|
- Potential issues or risks
|
||||||
|
- Recommendations (if applicable)
|
||||||
|
|
||||||
|
## Prioritization
|
||||||
|
|
||||||
|
Focus on high-impact elements:
|
||||||
|
- **Critical paths**: Main workflows that matter most
|
||||||
|
- **High-risk code**: Concurrency, error handling, resource management
|
||||||
|
- **Complex logic**: Non-obvious algorithms or subtle interactions
|
||||||
|
- **Key abstractions**: Core interfaces and contracts
|
||||||
|
|
||||||
|
Minimize or skip:
|
||||||
|
- Boilerplate code
|
||||||
|
- Simple getters/setters
|
||||||
|
- Standard patterns done correctly
|
||||||
|
- Low-risk trivial changes
|
||||||
|
|
||||||
|
## Example Scenarios
|
||||||
|
|
||||||
|
**Scenario 1**: "Explain this patch"
|
||||||
|
→ Show before/after diagrams, explain what changed and why, analyze assumptions and failure modes
|
||||||
|
|
||||||
|
**Scenario 2**: "How does SafeCommandStore work?"
|
||||||
|
→ Show class structure, explain the exclusive access pattern, trace key methods, analyze thread safety
|
||||||
|
|
||||||
|
**Scenario 3**: "Explain the coordination subsystem"
|
||||||
|
→ Show component diagram, trace PreAccept→Accept→Commit flow, explain message passing, show quorum tracking
|
||||||
|
|
||||||
|
**Scenario 4**: "What's the architecture of this codebase?"
|
||||||
|
→ Show high-level module structure, identify key abstractions, explain main patterns, highlight critical subsystems
|
||||||
|
|
||||||
|
**Scenario 5**: During code review (proactive)
|
||||||
|
→ Analyze changes for inconsistencies, check assumptions, identify potential race conditions, verify error handling
|
||||||
|
|
@ -0,0 +1,395 @@
|
||||||
|
# Code Analysis Framework
|
||||||
|
|
||||||
|
This reference provides a structured approach for deeply analyzing code, whether examining changes (patches/diffs) or understanding existing code (classes, subsystems, repositories).
|
||||||
|
|
||||||
|
## Analysis Dimensions
|
||||||
|
|
||||||
|
Every code analysis should address these dimensions:
|
||||||
|
|
||||||
|
### 1. Purpose & Context
|
||||||
|
- **What**: What does this code do?
|
||||||
|
- **Why it exists**: What problem does it solve?
|
||||||
|
- **Where it fits**: How does it relate to surrounding components?
|
||||||
|
|
||||||
|
### 2. Structure & Organization
|
||||||
|
- **Components**: What are the key pieces?
|
||||||
|
- **Relationships**: How do components interact?
|
||||||
|
- **Hierarchy**: What are the layers/abstractions?
|
||||||
|
|
||||||
|
### 3. Behavior & Flow
|
||||||
|
- **Data flow**: How does data move through the system?
|
||||||
|
- **Control flow**: What is the execution path?
|
||||||
|
- **State transitions**: What states exist and how do they change?
|
||||||
|
|
||||||
|
### 4. Assumptions & Invariants
|
||||||
|
- **Preconditions**: What must be true before execution?
|
||||||
|
- **Postconditions**: What must be true after execution?
|
||||||
|
- **Invariants**: What must always be true?
|
||||||
|
|
||||||
|
### 5. Failure Modes & Edge Cases
|
||||||
|
- **What could go wrong**: Identify failure points
|
||||||
|
- **Error handling**: How are failures managed?
|
||||||
|
- **Edge cases**: Boundary conditions and corner cases
|
||||||
|
|
||||||
|
### 6. Concurrency & Timing
|
||||||
|
- **Threading model**: Single-threaded, multi-threaded, async?
|
||||||
|
- **Synchronization**: Locks, atomics, message passing?
|
||||||
|
- **Race conditions**: Potential data races or ordering issues?
|
||||||
|
- **Deadlocks**: Circular dependencies in locking?
|
||||||
|
|
||||||
|
## Analysis Process for Patches/Diffs
|
||||||
|
|
||||||
|
When analyzing changes (git diff, patch file, PR):
|
||||||
|
|
||||||
|
### Step 1: Identify the Change Type
|
||||||
|
Categorize the change:
|
||||||
|
- **Bug fix**: Correcting incorrect behavior
|
||||||
|
- **Feature addition**: New functionality
|
||||||
|
- **Refactoring**: Restructuring without behavior change
|
||||||
|
- **Performance optimization**: Improving efficiency
|
||||||
|
- **Technical debt**: Cleanup, documentation, tests
|
||||||
|
|
||||||
|
### Step 2: Understand the Before State
|
||||||
|
- What was the old behavior?
|
||||||
|
- What was the problem with the old code?
|
||||||
|
- What assumptions did the old code make?
|
||||||
|
|
||||||
|
### Step 3: Understand the After State
|
||||||
|
- What is the new behavior?
|
||||||
|
- How does it differ from the old behavior?
|
||||||
|
- What new assumptions are introduced?
|
||||||
|
|
||||||
|
### Step 4: Analyze the Transition
|
||||||
|
- **Why this change**: What problem does it solve?
|
||||||
|
- **How it works**: Mechanism of the change
|
||||||
|
- **Critical modifications**: Which changes are most important?
|
||||||
|
- **Side effects**: What else is affected?
|
||||||
|
|
||||||
|
### Step 5: Visualize Before/After
|
||||||
|
Create ASCII diagrams showing:
|
||||||
|
- State transitions (before vs after)
|
||||||
|
- Data flow changes
|
||||||
|
- Control flow modifications
|
||||||
|
- Component relationship changes
|
||||||
|
|
||||||
|
### Step 6: Deep Reasoning
|
||||||
|
For each significant change, ask:
|
||||||
|
- **Correctness**: Is this change correct?
|
||||||
|
- **Completeness**: Does it fully solve the problem?
|
||||||
|
- **Safety**: Does it introduce new issues?
|
||||||
|
- **Assumptions**: What must be true for this to work?
|
||||||
|
- **Failure modes**: What could go wrong?
|
||||||
|
|
||||||
|
## Analysis Process for Existing Code
|
||||||
|
|
||||||
|
When analyzing existing code (class, subsystem, repository):
|
||||||
|
|
||||||
|
### Step 1: Identify Purpose
|
||||||
|
- What is this code responsible for?
|
||||||
|
- Why does it exist?
|
||||||
|
- What problem domain does it address?
|
||||||
|
|
||||||
|
### Step 2: Map the Structure
|
||||||
|
Create hierarchy diagram showing:
|
||||||
|
- Main components/classes
|
||||||
|
- Package/module organization
|
||||||
|
- Key abstractions
|
||||||
|
- Dependency relationships
|
||||||
|
|
||||||
|
### Step 3: Trace Key Workflows
|
||||||
|
Identify and trace 2-3 most important workflows:
|
||||||
|
- Entry points
|
||||||
|
- Step-by-step execution
|
||||||
|
- Data transformations
|
||||||
|
- Exit points/results
|
||||||
|
|
||||||
|
### Step 4: Identify State & Transitions
|
||||||
|
- What state does this code manage?
|
||||||
|
- How does state change over time?
|
||||||
|
- What are the valid state transitions?
|
||||||
|
- What triggers transitions?
|
||||||
|
|
||||||
|
### Step 5: Analyze Interactions
|
||||||
|
- How do components communicate?
|
||||||
|
- What are the interfaces/APIs?
|
||||||
|
- Message passing patterns
|
||||||
|
- Synchronous vs asynchronous calls
|
||||||
|
|
||||||
|
### Step 6: Deep Reasoning
|
||||||
|
For critical paths:
|
||||||
|
- **Invariants**: What must always be true?
|
||||||
|
- **Assumptions**: What does the code assume?
|
||||||
|
- **Concurrency**: How does it handle concurrent access?
|
||||||
|
- **Failure handling**: How are errors managed?
|
||||||
|
- **Edge cases**: What are the boundary conditions?
|
||||||
|
|
||||||
|
## Levels of Granularity
|
||||||
|
|
||||||
|
Adjust depth based on scope:
|
||||||
|
|
||||||
|
### Single Function/Method
|
||||||
|
- Purpose and return value
|
||||||
|
- Parameter requirements
|
||||||
|
- Side effects
|
||||||
|
- Key algorithm steps
|
||||||
|
- Error conditions
|
||||||
|
|
||||||
|
### Single Class
|
||||||
|
- Responsibility (what it does)
|
||||||
|
- State (fields/properties)
|
||||||
|
- Key methods and their relationships
|
||||||
|
- Lifecycle (creation → use → cleanup)
|
||||||
|
- Invariants maintained
|
||||||
|
|
||||||
|
### Subsystem (Multiple Classes)
|
||||||
|
- High-level purpose
|
||||||
|
- Component breakdown
|
||||||
|
- Interface boundaries
|
||||||
|
- Main workflows (2-3 key scenarios)
|
||||||
|
- Inter-component communication
|
||||||
|
|
||||||
|
### Repository (Full Codebase)
|
||||||
|
- Architecture overview (layers/modules)
|
||||||
|
- Core abstractions
|
||||||
|
- Key subsystems and their roles
|
||||||
|
- Major data flows
|
||||||
|
- Critical patterns used throughout
|
||||||
|
|
||||||
|
## Concurrency Analysis Checklist
|
||||||
|
|
||||||
|
When analyzing concurrent code:
|
||||||
|
|
||||||
|
### Data Race Detection
|
||||||
|
- Identify all shared mutable state
|
||||||
|
- Check for unsynchronized access
|
||||||
|
- Verify read/write ordering
|
||||||
|
- Look for missing memory barriers
|
||||||
|
|
||||||
|
### Deadlock Detection
|
||||||
|
- Identify all lock acquisitions
|
||||||
|
- Check for lock ordering violations
|
||||||
|
- Look for circular dependencies
|
||||||
|
- Verify timeout mechanisms
|
||||||
|
|
||||||
|
### Thread Safety Patterns
|
||||||
|
- Immutability: Are objects immutable?
|
||||||
|
- Synchronization: Are critical sections protected?
|
||||||
|
- Lock-free: Are atomic operations used correctly?
|
||||||
|
- Thread-local: Is state properly isolated?
|
||||||
|
- Message passing: Are channels/queues used safely?
|
||||||
|
|
||||||
|
### Async/Await Analysis
|
||||||
|
- Understand execution context switches
|
||||||
|
- Identify blocking operations in async code
|
||||||
|
- Check for proper error propagation
|
||||||
|
- Verify cancellation handling
|
||||||
|
|
||||||
|
## Assumption Identification Patterns
|
||||||
|
|
||||||
|
Look for implicit assumptions in:
|
||||||
|
|
||||||
|
### Ordering Assumptions
|
||||||
|
```
|
||||||
|
// Assumes operation A completes before B
|
||||||
|
operationA();
|
||||||
|
operationB(); // Depends on A's results
|
||||||
|
```
|
||||||
|
|
||||||
|
### Null/Validity Assumptions
|
||||||
|
```
|
||||||
|
// Assumes value is never null
|
||||||
|
value.method(); // No null check
|
||||||
|
```
|
||||||
|
|
||||||
|
### State Assumptions
|
||||||
|
```
|
||||||
|
// Assumes object is in specific state
|
||||||
|
process(); // Requires initialization first
|
||||||
|
```
|
||||||
|
|
||||||
|
### Concurrency Assumptions
|
||||||
|
```
|
||||||
|
// Assumes single-threaded access
|
||||||
|
counter++; // Not atomic
|
||||||
|
```
|
||||||
|
|
||||||
|
### Resource Assumptions
|
||||||
|
```
|
||||||
|
// Assumes resource is available
|
||||||
|
file.write(data); // Doesn't check for full disk
|
||||||
|
```
|
||||||
|
|
||||||
|
## Failure Mode Identification
|
||||||
|
|
||||||
|
Common failure patterns:
|
||||||
|
|
||||||
|
### Resource Exhaustion
|
||||||
|
- Memory leaks
|
||||||
|
- File descriptor leaks
|
||||||
|
- Thread pool exhaustion
|
||||||
|
- Connection pool exhaustion
|
||||||
|
|
||||||
|
### Timing Issues
|
||||||
|
- Race conditions
|
||||||
|
- Deadlocks
|
||||||
|
- Timeouts not handled
|
||||||
|
- Retry storms
|
||||||
|
|
||||||
|
### Data Issues
|
||||||
|
- Null pointer exceptions
|
||||||
|
- Array/buffer overflows
|
||||||
|
- Type mismatches
|
||||||
|
- Encoding issues
|
||||||
|
|
||||||
|
### Error Propagation
|
||||||
|
- Exceptions swallowed
|
||||||
|
- Errors not logged
|
||||||
|
- Inconsistent error states
|
||||||
|
- Missing rollback logic
|
||||||
|
|
||||||
|
### External Dependencies
|
||||||
|
- Network failures
|
||||||
|
- Database unavailability
|
||||||
|
- Third-party API changes
|
||||||
|
- Configuration errors
|
||||||
|
|
||||||
|
## Visualization Guidelines
|
||||||
|
|
||||||
|
### When to Use Each Diagram Type
|
||||||
|
|
||||||
|
**State Machine**: When analyzing:
|
||||||
|
- Status/state fields that change
|
||||||
|
- Lifecycle of objects
|
||||||
|
- Protocol state machines
|
||||||
|
- Transaction states
|
||||||
|
|
||||||
|
**Data Flow**: When analyzing:
|
||||||
|
- Information transformation
|
||||||
|
- Pipeline processing
|
||||||
|
- Request/response patterns
|
||||||
|
- Data dependencies
|
||||||
|
|
||||||
|
**Sequence Diagram**: When analyzing:
|
||||||
|
- Component interactions
|
||||||
|
- Message protocols
|
||||||
|
- API call sequences
|
||||||
|
- Distributed system operations
|
||||||
|
|
||||||
|
**Component Diagram**: When analyzing:
|
||||||
|
- System architecture
|
||||||
|
- Module relationships
|
||||||
|
- Layer separation
|
||||||
|
- Dependency structure
|
||||||
|
|
||||||
|
**Before/After**: When analyzing:
|
||||||
|
- Code changes/patches
|
||||||
|
- Refactoring
|
||||||
|
- Bug fixes
|
||||||
|
- Architecture evolution
|
||||||
|
|
||||||
|
### Diagram Complexity Guidelines
|
||||||
|
|
||||||
|
**Simple (1-5 elements)**: Show core concept clearly
|
||||||
|
**Medium (6-15 elements)**: Show main flow with key branches
|
||||||
|
**Complex (16+ elements)**: Break into multiple focused diagrams
|
||||||
|
|
||||||
|
Prefer multiple simple diagrams over one complex diagram.
|
||||||
|
|
||||||
|
## Communication Strategy
|
||||||
|
|
||||||
|
### Structure Your Explanation
|
||||||
|
|
||||||
|
1. **Executive Summary** (1-2 paragraphs)
|
||||||
|
- What this code does
|
||||||
|
- Why it exists
|
||||||
|
- Key insight in one sentence
|
||||||
|
|
||||||
|
2. **Visual Overview** (ASCII diagram)
|
||||||
|
- High-level structure or flow
|
||||||
|
- Main components/states
|
||||||
|
- Critical paths highlighted
|
||||||
|
|
||||||
|
3. **Detailed Analysis** (organized by aspect)
|
||||||
|
- Purpose & behavior
|
||||||
|
- Structure & components
|
||||||
|
- Assumptions & invariants
|
||||||
|
- Failure modes
|
||||||
|
|
||||||
|
4. **Critical Insights** (bullet points)
|
||||||
|
- Most important findings
|
||||||
|
- Potential issues found
|
||||||
|
- Recommendations if applicable
|
||||||
|
|
||||||
|
### Prioritization
|
||||||
|
|
||||||
|
Focus on:
|
||||||
|
- **High impact**: Changes to critical paths
|
||||||
|
- **High risk**: Concurrency, error handling, resource management
|
||||||
|
- **High complexity**: Tricky algorithms, subtle interactions
|
||||||
|
- **High value**: Key abstractions, main workflows
|
||||||
|
|
||||||
|
Skip or minimize:
|
||||||
|
- Boilerplate code
|
||||||
|
- Obvious getters/setters
|
||||||
|
- Standard patterns done correctly
|
||||||
|
- Low-risk refactoring
|
||||||
|
|
||||||
|
## Example Analysis Template
|
||||||
|
|
||||||
|
```
|
||||||
|
# Analysis of [Component/Patch Name]
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
[2-3 sentences: what it is, what it does, why it matters]
|
||||||
|
|
||||||
|
## High-Level Structure
|
||||||
|
[ASCII diagram showing main components/flow]
|
||||||
|
|
||||||
|
## Key Behaviors
|
||||||
|
### Workflow 1: [Name]
|
||||||
|
[Sequence diagram or flow diagram]
|
||||||
|
- Step 1: ...
|
||||||
|
- Step 2: ...
|
||||||
|
|
||||||
|
### Workflow 2: [Name]
|
||||||
|
[Sequence diagram or flow diagram]
|
||||||
|
|
||||||
|
## State Management
|
||||||
|
[State machine diagram if applicable]
|
||||||
|
|
||||||
|
## Critical Assumptions
|
||||||
|
1. **[Assumption]**: [Explanation]
|
||||||
|
- Why: [Why this assumption exists]
|
||||||
|
- Risk: [What breaks if invalid]
|
||||||
|
|
||||||
|
2. **[Assumption]**: ...
|
||||||
|
|
||||||
|
## Failure Modes
|
||||||
|
1. **[Failure scenario]**: [Description]
|
||||||
|
- Trigger: [What causes it]
|
||||||
|
- Impact: [What goes wrong]
|
||||||
|
- Mitigation: [How it's handled or should be]
|
||||||
|
|
||||||
|
2. **[Failure scenario]**: ...
|
||||||
|
|
||||||
|
## Concurrency Analysis (if applicable)
|
||||||
|
[Thread interaction diagram]
|
||||||
|
- Shared state: [What's shared]
|
||||||
|
- Synchronization: [How it's protected]
|
||||||
|
- Risks: [Potential races/deadlocks]
|
||||||
|
|
||||||
|
## Before/After (for patches only)
|
||||||
|
### Before
|
||||||
|
[Diagram of old behavior]
|
||||||
|
|
||||||
|
### After
|
||||||
|
[Diagram of new behavior]
|
||||||
|
|
||||||
|
### Why This Change
|
||||||
|
[Explanation of the improvement]
|
||||||
|
|
||||||
|
## Key Insights
|
||||||
|
- [Most important finding 1]
|
||||||
|
- [Most important finding 2]
|
||||||
|
- [Most important finding 3]
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,400 @@
|
||||||
|
# ASCII Diagram Patterns
|
||||||
|
|
||||||
|
This reference provides templates and examples for creating clear ASCII diagrams when visualizing code.
|
||||||
|
|
||||||
|
## State Machine Diagrams
|
||||||
|
|
||||||
|
### Simple State Transition
|
||||||
|
|
||||||
|
```
|
||||||
|
[State A] --event--> [State B] --event--> [State C]
|
||||||
|
```
|
||||||
|
|
||||||
|
### State Machine with Multiple Paths
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────┐
|
||||||
|
│ Initial │
|
||||||
|
└────┬────┘
|
||||||
|
│
|
||||||
|
┌───────┴───────┐
|
||||||
|
│ │
|
||||||
|
event_a event_b
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
┌────────┐ ┌────────┐
|
||||||
|
│ State1 │ │ State2 │
|
||||||
|
└───┬────┘ └───┬────┘
|
||||||
|
│ │
|
||||||
|
success failure
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
┌────────┐ ┌────────┐
|
||||||
|
│ Done │ │ Error │
|
||||||
|
└────────┘ └────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### State Machine with Self-Loops
|
||||||
|
|
||||||
|
```
|
||||||
|
retry
|
||||||
|
↻
|
||||||
|
┌──────┐
|
||||||
|
│ ▼
|
||||||
|
┌──┴────────┐ success ┌──────────┐
|
||||||
|
│ Pending │──────────→ │ Complete │
|
||||||
|
└───────────┘ └──────────┘
|
||||||
|
│
|
||||||
|
│ timeout
|
||||||
|
▼
|
||||||
|
┌──────────┐
|
||||||
|
│ Failed │
|
||||||
|
└──────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Flow Diagrams
|
||||||
|
|
||||||
|
### Linear Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
Input → [Process A] → [Process B] → [Process C] → Output
|
||||||
|
```
|
||||||
|
|
||||||
|
### Branching Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
┌→ [Handler A] → Result A
|
||||||
|
│
|
||||||
|
Input → [Router] ───┼→ [Handler B] → Result B
|
||||||
|
│
|
||||||
|
└→ [Handler C] → Result C
|
||||||
|
```
|
||||||
|
|
||||||
|
### Convergent Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
[Source A] ─┐
|
||||||
|
├→ [Combiner] → Output
|
||||||
|
[Source B] ─┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Bidirectional Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────┐ request ┌─────────┐
|
||||||
|
│ Client │ ─────────→ │ Server │
|
||||||
|
└─────────┘ ←───────── └─────────┘
|
||||||
|
response
|
||||||
|
```
|
||||||
|
|
||||||
|
## Sequence Diagrams
|
||||||
|
|
||||||
|
### Simple Interaction
|
||||||
|
|
||||||
|
```
|
||||||
|
Component A Component B Component C
|
||||||
|
│ │ │
|
||||||
|
│──call()────────────→│ │
|
||||||
|
│ │──query()───────────→│
|
||||||
|
│ │←──result───────────│
|
||||||
|
│←──response──────────│ │
|
||||||
|
│ │ │
|
||||||
|
```
|
||||||
|
|
||||||
|
### With Timing/Phases
|
||||||
|
|
||||||
|
```
|
||||||
|
Thread 1 Thread 2 Shared State
|
||||||
|
│ │ │
|
||||||
|
│──lock()───────┼────────────────→│
|
||||||
|
│ │ │ [LOCKED]
|
||||||
|
│──read()───────┼────────────────→│
|
||||||
|
│←──value───────┼─────────────────│
|
||||||
|
│ │ │
|
||||||
|
│──unlock()─────┼────────────────→│
|
||||||
|
│ │ │ [UNLOCKED]
|
||||||
|
│ │──lock()────────→│
|
||||||
|
│ │ │ [LOCKED]
|
||||||
|
│ │──write()───────→│
|
||||||
|
│ │ │
|
||||||
|
```
|
||||||
|
|
||||||
|
## Component/Architecture Diagrams
|
||||||
|
|
||||||
|
### Layered Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────┐
|
||||||
|
│ Application Layer │
|
||||||
|
│ (Business Logic, Workflows) │
|
||||||
|
└──────────────┬──────────────────────┘
|
||||||
|
│
|
||||||
|
┌──────────────┴──────────────────────┐
|
||||||
|
│ Service Layer │
|
||||||
|
│ (API, Coordination, Validation) │
|
||||||
|
└──────────────┬──────────────────────┘
|
||||||
|
│
|
||||||
|
┌──────────────┴──────────────────────┐
|
||||||
|
│ Data Layer │
|
||||||
|
│ (Storage, Persistence, Cache) │
|
||||||
|
└─────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Component Interaction
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────┐
|
||||||
|
│ Client │
|
||||||
|
└──────┬───────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌──────────────┐ ┌──────────────┐
|
||||||
|
│ Controller │─────→│ Service │
|
||||||
|
└──────┬───────┘ └──────┬───────┘
|
||||||
|
│ │
|
||||||
|
│ ┌──────┴───────┐
|
||||||
|
│ │ │
|
||||||
|
▼ ▼ ▼
|
||||||
|
┌──────────────┐ ┌────────┐ ┌─────────┐
|
||||||
|
│ Cache │ │ DB │ │ Queue │
|
||||||
|
└──────────────┘ └────────┘ └─────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Nested Components
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────┐
|
||||||
|
│ Node │
|
||||||
|
│ ┌─────────────────────────────────┐ │
|
||||||
|
│ │ CommandStore │ │
|
||||||
|
│ │ ┌──────────┐ ┌──────────┐ │ │
|
||||||
|
│ │ │ Commands │ │ Listeners│ │ │
|
||||||
|
│ │ └──────────┘ └──────────┘ │ │
|
||||||
|
│ └─────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌─────────────────────────────────┐ │
|
||||||
|
│ │ MessageRouter │ │
|
||||||
|
│ └─────────────────────────────────┘ │
|
||||||
|
└─────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Before/After Comparisons
|
||||||
|
|
||||||
|
### Side-by-Side
|
||||||
|
|
||||||
|
```
|
||||||
|
BEFORE AFTER
|
||||||
|
─────────────────────────────────────────────────
|
||||||
|
┌──────────┐ ┌──────────┐
|
||||||
|
│ Single │ │ Step 1 │
|
||||||
|
│ Step │ └────┬─────┘
|
||||||
|
└────┬─────┘ │
|
||||||
|
│ ▼
|
||||||
|
▼ ┌──────────┐
|
||||||
|
┌──────────┐ │ Step 2 │
|
||||||
|
│ Result │ └────┬─────┘
|
||||||
|
└──────────┘ │
|
||||||
|
▼
|
||||||
|
┌──────────┐
|
||||||
|
│ Result │
|
||||||
|
└──────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### With Annotations
|
||||||
|
|
||||||
|
```
|
||||||
|
OLD: [Start] → [Process] → [End]
|
||||||
|
↑
|
||||||
|
│
|
||||||
|
(bottleneck!)
|
||||||
|
|
||||||
|
NEW: [Start] → [Cache?] ─Yes→ [End]
|
||||||
|
│
|
||||||
|
No
|
||||||
|
↓
|
||||||
|
[Process] → [Cache] → [End]
|
||||||
|
(store)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Concurrency Patterns
|
||||||
|
|
||||||
|
### Race Condition
|
||||||
|
|
||||||
|
```
|
||||||
|
Thread A Thread B
|
||||||
|
│ │
|
||||||
|
│─read(x=10) │
|
||||||
|
│ │─read(x=10)
|
||||||
|
│─compute(x+1) │
|
||||||
|
│ │─compute(x+1)
|
||||||
|
│─write(x=11) │
|
||||||
|
│ │─write(x=11) ⚠ Lost update!
|
||||||
|
│ │
|
||||||
|
Result: x=11 (expected: x=12)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Proper Synchronization
|
||||||
|
|
||||||
|
```
|
||||||
|
Thread A Shared State Thread B
|
||||||
|
│ │ │
|
||||||
|
│──lock()───────────────→ │ │
|
||||||
|
│ │ [LOCKED by A] │
|
||||||
|
│──read(x=10)─────────── │ │
|
||||||
|
│──write(x=11)────────→ │ │
|
||||||
|
│──unlock()───────────→ │ │
|
||||||
|
│ │ [UNLOCKED] │
|
||||||
|
│ │ ←──────────lock()───────│
|
||||||
|
│ │ [LOCKED by B] │
|
||||||
|
│ │ ──────────read(x=11)───→│
|
||||||
|
│ │ ←─────────write(x=12)───│
|
||||||
|
│ │ ←──────────unlock()─────│
|
||||||
|
│ │ [UNLOCKED] │
|
||||||
|
```
|
||||||
|
|
||||||
|
### Message Passing
|
||||||
|
|
||||||
|
```
|
||||||
|
Producer Queue Consumer
|
||||||
|
│ │ │
|
||||||
|
│──produce(msg1)──→ [msg1] │
|
||||||
|
│──produce(msg2)──→ [msg1,msg2] │
|
||||||
|
│ │ ←──consume()──────│
|
||||||
|
│ [msg2] (got msg1) │
|
||||||
|
│──produce(msg3)──→ [msg2,msg3] │
|
||||||
|
│ │ ←──consume()──────│
|
||||||
|
│ [msg3] (got msg2) │
|
||||||
|
```
|
||||||
|
|
||||||
|
## Control Flow Patterns
|
||||||
|
|
||||||
|
### Conditional Logic
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────┐
|
||||||
|
│ Check │
|
||||||
|
└────┬─────┘
|
||||||
|
│
|
||||||
|
┌──────┴──────┐
|
||||||
|
│ │
|
||||||
|
condition condition
|
||||||
|
true? false?
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
┌────────┐ ┌────────┐
|
||||||
|
│ Path A │ │ Path B │
|
||||||
|
└───┬────┘ └───┬────┘
|
||||||
|
│ │
|
||||||
|
└──────┬──────┘
|
||||||
|
▼
|
||||||
|
┌─────────┐
|
||||||
|
│ Merge │
|
||||||
|
└─────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Loop with Exit Condition
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────┐
|
||||||
|
│ Initialize │
|
||||||
|
└──────┬───────┘
|
||||||
|
│
|
||||||
|
┌──────▼───────┐
|
||||||
|
│ Loop Body │←──┐
|
||||||
|
└──────┬───────┘ │
|
||||||
|
│ │
|
||||||
|
┌──────▼───────┐ │
|
||||||
|
│ Continue? │ │
|
||||||
|
└──────┬───────┘ │
|
||||||
|
│ │
|
||||||
|
┌────┴────┐ │
|
||||||
|
│ │ │
|
||||||
|
Yes No │
|
||||||
|
│ │ │
|
||||||
|
└─────────┘ │
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌──────────────┐
|
||||||
|
│ Exit │
|
||||||
|
└──────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Exception Handling
|
||||||
|
|
||||||
|
```
|
||||||
|
┌────────────┐
|
||||||
|
│ Start │
|
||||||
|
└─────┬──────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌────────────┐
|
||||||
|
│ Try Block │
|
||||||
|
└─────┬──────┘
|
||||||
|
│
|
||||||
|
├──success──────→ [Continue]
|
||||||
|
│
|
||||||
|
└──exception────→ ┌─────────────┐
|
||||||
|
│ Catch Block │
|
||||||
|
└──────┬──────┘
|
||||||
|
│
|
||||||
|
┌──────┴──────┐
|
||||||
|
│ │
|
||||||
|
recoverable fatal
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
[Retry] [Fail]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Dependency Graphs
|
||||||
|
|
||||||
|
### Simple Dependencies
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────┐
|
||||||
|
│ A │
|
||||||
|
└──┬──┘
|
||||||
|
│
|
||||||
|
┌────┴────┐
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
┌───┐ ┌───┐
|
||||||
|
│ B │ │ C │
|
||||||
|
└─┬─┘ └─┬─┘
|
||||||
|
│ │
|
||||||
|
└────┬────┘
|
||||||
|
▼
|
||||||
|
┌───┐
|
||||||
|
│ D │
|
||||||
|
└───┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Circular Dependencies (Problem)
|
||||||
|
|
||||||
|
```
|
||||||
|
┌───┐
|
||||||
|
│ A │
|
||||||
|
└─┬─┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌───┐ ⚠ CYCLE!
|
||||||
|
│ B │
|
||||||
|
└─┬─┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌───┐
|
||||||
|
│ C │
|
||||||
|
└─┬─┘
|
||||||
|
│
|
||||||
|
└──→ [back to A]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tips for Effective Diagrams
|
||||||
|
|
||||||
|
1. **Use consistent spacing** - Align elements for readability
|
||||||
|
2. **Add legends** - Explain symbols when not obvious
|
||||||
|
3. **Show critical paths** - Highlight the main flow
|
||||||
|
4. **Mark problems** - Use ⚠, ✗, ❌ for issues
|
||||||
|
5. **Mark fixes** - Use ✓, ✔, ✅ for solutions
|
||||||
|
6. **Annotate** - Add brief explanations with (parentheses)
|
||||||
|
7. **Keep it simple** - Don't try to show everything at once
|
||||||
|
8. **Focus on one aspect** - Separate diagrams for different concerns (data flow vs. control flow)
|
||||||
|
|
@ -0,0 +1,125 @@
|
||||||
|
# Edge Case Explorer -- Eval Prompt
|
||||||
|
|
||||||
|
Run this to test the skill against a known bug-introducing patch.
|
||||||
|
|
||||||
|
## Setup: Find an introducing patch
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Pick a random bug from archaeology, trace to the introducing commit
|
||||||
|
BUG_DIR=/path/to/bug-archaeology
|
||||||
|
REPO=/path/to/cassandra
|
||||||
|
|
||||||
|
# 1. Pick a bug file
|
||||||
|
BUGFILE=$(ls $BUG_DIR/bug-*.md | shuf -n1)
|
||||||
|
FIX_HASH=$(grep -oP 'Commit\*\*: \K\w+' "$BUGFILE" | head -1)
|
||||||
|
|
||||||
|
# 2. Find the primary changed Java file
|
||||||
|
PRIMARY=$(git -C $REPO diff --name-only "${FIX_HASH}~1" $FIX_HASH | grep '\.java$' | grep -v 'test/' | head -1)
|
||||||
|
|
||||||
|
# 3. Find the first removed (buggy) line number
|
||||||
|
LINE=$(git -C $REPO diff "${FIX_HASH}~1" $FIX_HASH -- "$PRIMARY" | \
|
||||||
|
awk '/^@@ -([0-9]+)/{line=$0; gsub(/.*-/,"",line); gsub(/,.*/,"",line)} /^-[^-]/{if(length($0)>6) {print line; exit}}' )
|
||||||
|
|
||||||
|
# 4. Blame to find introducing commit
|
||||||
|
INTRO=$(git -C $REPO blame -L "$LINE,$LINE" "${FIX_HASH}~1" --porcelain -- "$PRIMARY" | head -1 | cut -c1-40)
|
||||||
|
|
||||||
|
# 5. Extract introducing diff
|
||||||
|
git -C $REPO diff "${INTRO}~1" $INTRO -- "$PRIMARY" > /tmp/eval_patch.diff
|
||||||
|
echo "Bug: $(head -1 "$BUGFILE")"
|
||||||
|
echo "Introducing commit: $INTRO"
|
||||||
|
echo "Diff: $(wc -l < /tmp/eval_patch.diff) lines"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Eval: Ensemble Mode (default)
|
||||||
|
|
||||||
|
Launch 5 specialist agents in parallel, each reading only their checklist + the patch.
|
||||||
|
Use sonnet model for speed.
|
||||||
|
|
||||||
|
```
|
||||||
|
# For each specialist, launch an agent with:
|
||||||
|
# 1. Their specialist checklist file
|
||||||
|
# 2. The patch to review
|
||||||
|
# 3. A focused prompt (see SKILL.md for templates)
|
||||||
|
|
||||||
|
Agent(model=sonnet, prompt="""
|
||||||
|
You are a {SPECIALIST} specialist. Review ONLY for {DOMAIN_KEYWORDS}.
|
||||||
|
Read your checklist: `{SKILL_DIR}/references/general/pass2-specialists/pass2-{name}.md`
|
||||||
|
Then read the patch: `/tmp/eval_patch.diff`
|
||||||
|
Report your single best finding in <=50 words: Pattern ID, Location, What's wrong.
|
||||||
|
Or "No finding in my domain".
|
||||||
|
""")
|
||||||
|
|
||||||
|
# Specialists to launch (all 5 in parallel):
|
||||||
|
# 1. LOGIC -- pass2-logic.md
|
||||||
|
# 2. BOUNDARY -- pass2-boundary.md
|
||||||
|
# 3. CONCURRENCY -- pass2-concurrency.md
|
||||||
|
# 4. RESOURCES -- pass2-resources.md
|
||||||
|
# 5. ABSENCE -- detection-signatures.md
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key constraints for eval agents
|
||||||
|
|
||||||
|
- Each agent reads exactly 2 files: their checklist + the patch
|
||||||
|
- Use sonnet model for faster turnaround
|
||||||
|
- Ask for concise output (<=50 words per finding)
|
||||||
|
- Run all 5 in parallel (single message with 5 Agent tool calls)
|
||||||
|
|
||||||
|
## Scoring
|
||||||
|
|
||||||
|
After all specialists report, merge findings and compare against the actual bug:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# The actual bug description is in the bug file
|
||||||
|
grep -A5 '## Root Cause' "$BUGFILE"
|
||||||
|
```
|
||||||
|
|
||||||
|
Score the **ensemble** (merged set of all specialist findings) as:
|
||||||
|
- **Exact**: At least one specialist identified the specific bug that was later fixed
|
||||||
|
- **Partial**: At least one specialist flagged the right area/pattern but described a different specific issue
|
||||||
|
- **Different bug**: Specialists found real bugs, but not the target
|
||||||
|
- **Miss**: No specialist found anything relevant
|
||||||
|
|
||||||
|
Also track **per-specialist** hits to identify which domains need improvement.
|
||||||
|
|
||||||
|
## Comparison Eval: Single vs Ensemble
|
||||||
|
|
||||||
|
To compare, also run the same patch through a single generalist agent:
|
||||||
|
|
||||||
|
```
|
||||||
|
You are reviewing a code patch for potential bugs. You have NO access to the broader
|
||||||
|
codebase -- only the patch and the reference files below.
|
||||||
|
|
||||||
|
Read (in order):
|
||||||
|
1. `{SKILL_DIR}/references/general/pass1-strategic-patterns.md`
|
||||||
|
2. `{SKILL_DIR}/references/general/pass2-tactical-checklist.md`
|
||||||
|
3. `{SKILL_DIR}/references/general/detection-signatures.md`
|
||||||
|
|
||||||
|
Then read: `/tmp/eval_patch.diff`
|
||||||
|
|
||||||
|
Report top 2-3 bugs found. Keep response under 400 words.
|
||||||
|
```
|
||||||
|
|
||||||
|
Score both and compare hit rates.
|
||||||
|
|
||||||
|
## Prodding (for misses)
|
||||||
|
|
||||||
|
If the ensemble missed, tell each specialist what the bug actually was and ask:
|
||||||
|
|
||||||
|
```
|
||||||
|
You missed this bug: [description of actual bug].
|
||||||
|
Which checklist question in YOUR specialist file would have helped you find it?
|
||||||
|
If none exists, propose a new question to add to your checklist.
|
||||||
|
```
|
||||||
|
|
||||||
|
Use the feedback to improve the specialist checklist files.
|
||||||
|
|
||||||
|
## Batch Eval
|
||||||
|
|
||||||
|
For batch evaluation across N bugs:
|
||||||
|
1. Extract N patches using the setup script
|
||||||
|
2. For each patch, launch 5 specialists (one message per patch)
|
||||||
|
3. Wait for results, merge, score
|
||||||
|
4. Tabulate: exact/partial/different/miss rates, per-specialist hit rates
|
||||||
|
5. Compare against single-agent baseline on the same patches
|
||||||
|
|
||||||
|
Target: >90% hit rate (exact + partial), <5% miss rate.
|
||||||
|
|
@ -0,0 +1,315 @@
|
||||||
|
---
|
||||||
|
name: shallow-review
|
||||||
|
version: "1.0.0"
|
||||||
|
description: >
|
||||||
|
Shallow (quick) ensemble bug-finding review using 6 specialist agents in parallel. Each
|
||||||
|
specialist reviews the same patch through a different lens: Logic & Types, Boundaries &
|
||||||
|
I/O, Concurrency & State, Resources & Serialization, Absence Analysis, and API
|
||||||
|
Completeness. Findings are merged and deduplicated. Best for: quick first-pass review of
|
||||||
|
patches, triage of diffs, broad surface-level bug scan. For deeper file-focused review
|
||||||
|
with full pattern catalogs and codebase investigation, use the deep-review skill instead.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Ensemble Code Review
|
||||||
|
|
||||||
|
Six specialist agents review the same patch in parallel. Each applies Phase 0 (understand
|
||||||
|
the change, form hypotheses) then a focused checklist. Their findings are merged and
|
||||||
|
deduplicated.
|
||||||
|
|
||||||
|
```
|
||||||
|
+------------------+
|
||||||
|
| Input Patch |
|
||||||
|
+--------+---------+
|
||||||
|
|
|
||||||
|
+-------+------+---------+---------+-----------+
|
||||||
|
| | | | | |
|
||||||
|
+---v--+ +--v--+ +-v-----+ +-v------+ +v-----+ +---v-------+
|
||||||
|
|Logic | |Bound| |Concurr| |Resource| |Absenc| | Complete. |
|
||||||
|
|&Type | | &I/O| |&State | |& Serde | |(2-ph)| | & Contr. |
|
||||||
|
+--+---+ +--+--+ +---+---+ +---+----+ +--+---+ +---+-------+
|
||||||
|
| | | | | |
|
||||||
|
+--------+--------+---------+----------+--------+
|
||||||
|
|
|
||||||
|
+----------v----------+
|
||||||
|
| Pass 3: Symmetry |
|
||||||
|
| (serial, optional) |
|
||||||
|
+----------+----------+
|
||||||
|
|
|
||||||
|
+----------v----------+
|
||||||
|
| Merge & Dedup |
|
||||||
|
| (main agent) |
|
||||||
|
+---------------------+
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
1. **Identify target** — patch, diff, file, or subsystem
|
||||||
|
2. **Launch 6 specialists** in parallel (see prompts below)
|
||||||
|
3. **Run Pass 3 Symmetry** check (serial, after specialists complete)
|
||||||
|
4. **Merge** findings: deduplicate, rank by confidence
|
||||||
|
5. **Report** in unified format
|
||||||
|
|
||||||
|
## Phase 0: Understand Before Checking
|
||||||
|
|
||||||
|
Every specialist runs Phase 0 BEFORE touching the checklist:
|
||||||
|
|
||||||
|
1. **Summarize** the patch in 2–3 sentences (feature, fix, refactor, move?)
|
||||||
|
2. **Hypothesize** — list 3–5 things that could go wrong with this type of change
|
||||||
|
3. **Note** code shapes that warrant deeper investigation
|
||||||
|
|
||||||
|
This switches the review from bottom-up (checklist→code) to top-down
|
||||||
|
(code→hypotheses→verification). The checklist then verifies and extends the hypotheses.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The Six Specialists
|
||||||
|
|
||||||
|
### 1. Logic & Types (52% of bugs)
|
||||||
|
|
||||||
|
**Priority**: wrong comparisons, missing returns, wrong constants, boolean polarity, sentinel ambiguity.
|
||||||
|
|
||||||
|
**Reference**: `references/general/specialists/logic.md`
|
||||||
|
|
||||||
|
**Prompt**:
|
||||||
|
```
|
||||||
|
You are a LOGIC specialist reviewing a patch for correctness bugs.
|
||||||
|
|
||||||
|
PHASE 0 — Understand the patch first (before reading checklist):
|
||||||
|
Read the patch: `{PATCH_PATH}`
|
||||||
|
1. Summarize what this patch does in 2-3 sentences.
|
||||||
|
2. List 3-5 things that could go wrong with this type of change.
|
||||||
|
3. Note any suspicious code shapes (control flow, comparisons, constants, sentinels).
|
||||||
|
|
||||||
|
PHASE 1 — Apply checklist:
|
||||||
|
Read your checklist: `{SKILL_DIR}/references/general/specialists/logic.md`
|
||||||
|
Apply questions that match your hypotheses and code shapes. Check all 30 items
|
||||||
|
against the patch but spend the most effort where your hypotheses pointed.
|
||||||
|
|
||||||
|
Report ALL findings above a noise floor. For each: Location, Confidence
|
||||||
|
(High/Medium/Low), What's wrong (1-2 sentences).
|
||||||
|
Or "No finding in my domain" if nothing applies.
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Boundaries & I/O (17% of bugs)
|
||||||
|
|
||||||
|
**Priority**: off-by-one, integer overflow, null/bounds checks, buffer sizing, I/O completeness.
|
||||||
|
|
||||||
|
**Reference**: `references/general/specialists/boundary.md`
|
||||||
|
|
||||||
|
**Prompt**:
|
||||||
|
```
|
||||||
|
You are a BOUNDARY specialist reviewing a patch for correctness bugs.
|
||||||
|
|
||||||
|
PHASE 0 — Understand the patch first (before reading checklist):
|
||||||
|
Read the patch: `{PATCH_PATH}`
|
||||||
|
1. Summarize what this patch does in 2-3 sentences.
|
||||||
|
2. List 3-5 things that could go wrong with this type of change.
|
||||||
|
3. Note any arithmetic, indexing, range, or I/O operations.
|
||||||
|
|
||||||
|
PHASE 1 — Apply checklist:
|
||||||
|
Read your checklist: `{SKILL_DIR}/references/general/specialists/boundary.md`
|
||||||
|
Apply questions that match your hypotheses and code shapes. Check all 22 items
|
||||||
|
against the patch but spend the most effort where your hypotheses pointed.
|
||||||
|
|
||||||
|
Report ALL findings above a noise floor. For each: Location, Confidence
|
||||||
|
(High/Medium/Low), What's wrong (1-2 sentences).
|
||||||
|
Or "No finding in my domain" if nothing applies.
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Concurrency & State (16% of bugs)
|
||||||
|
|
||||||
|
**Priority**: races, TOCTOU, live-view iteration, lock ordering, state cleanup.
|
||||||
|
|
||||||
|
**Reference**: `references/general/specialists/concurrency.md`
|
||||||
|
|
||||||
|
**Prompt**:
|
||||||
|
```
|
||||||
|
You are a CONCURRENCY specialist reviewing a patch for correctness bugs.
|
||||||
|
|
||||||
|
PHASE 0 — Understand the patch first (before reading checklist):
|
||||||
|
Read the patch: `{PATCH_PATH}`
|
||||||
|
1. Summarize what this patch does in 2-3 sentences.
|
||||||
|
2. List 3-5 things that could go wrong with this type of change.
|
||||||
|
3. Note any shared state, locks, collections, lifecycle operations.
|
||||||
|
|
||||||
|
PHASE 1 — Apply checklist:
|
||||||
|
Read your checklist: `{SKILL_DIR}/references/general/specialists/concurrency.md`
|
||||||
|
Apply questions that match your hypotheses and code shapes. Check all 24 items
|
||||||
|
against the patch but spend the most effort where your hypotheses pointed.
|
||||||
|
|
||||||
|
Report ALL findings above a noise floor. For each: Location, Confidence
|
||||||
|
(High/Medium/Low), What's wrong (1-2 sentences).
|
||||||
|
Or "No finding in my domain" if nothing applies.
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Resources & Serialization (15% of bugs)
|
||||||
|
|
||||||
|
**Priority**: resource leaks, serialization mismatches, wrong metric types, background tasks, wrapper bypass.
|
||||||
|
|
||||||
|
**Reference**: `references/general/specialists/resources.md`
|
||||||
|
|
||||||
|
**Prompt**:
|
||||||
|
```
|
||||||
|
You are a RESOURCES specialist reviewing a patch for correctness bugs.
|
||||||
|
|
||||||
|
PHASE 0 — Understand the patch first (before reading checklist):
|
||||||
|
Read the patch: `{PATCH_PATH}`
|
||||||
|
1. Summarize what this patch does in 2-3 sentences.
|
||||||
|
2. List 3-5 things that could go wrong with this type of change.
|
||||||
|
3. Note any resource allocation, serialization methods, metrics, streams.
|
||||||
|
|
||||||
|
PHASE 1 — Apply checklist:
|
||||||
|
Read your checklist: `{SKILL_DIR}/references/general/specialists/resources.md`
|
||||||
|
Apply questions that match your hypotheses and code shapes. Check all 26 items
|
||||||
|
against the patch but spend the most effort where your hypotheses pointed.
|
||||||
|
|
||||||
|
Report ALL findings above a noise floor. For each: Location, Confidence
|
||||||
|
(High/Medium/Low), What's wrong (1-2 sentences).
|
||||||
|
Or "No finding in my domain" if nothing applies.
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Absence Analysis (cross-cutting)
|
||||||
|
|
||||||
|
**Priority**: find what SHOULD be present but ISN'T — missing guards, cleanup, handlers, registrations.
|
||||||
|
|
||||||
|
**Reference**: `references/general/specialists/absence.md`
|
||||||
|
|
||||||
|
**Prompt**:
|
||||||
|
```
|
||||||
|
You are an ABSENCE specialist. Your job: find bugs that are purely absent — missing
|
||||||
|
guards, cleanup, handlers, or conditions that leave no signal in the diff.
|
||||||
|
|
||||||
|
PHASE 0 — Understand the patch first:
|
||||||
|
Read the patch: `{PATCH_PATH}`
|
||||||
|
1. Summarize what this patch does in 2-3 sentences.
|
||||||
|
2. List 3-5 things that SHOULD be present but might be missing.
|
||||||
|
|
||||||
|
PHASE 1 — Build the search list:
|
||||||
|
Read your checklist: `{SKILL_DIR}/references/general/specialists/absence.md`
|
||||||
|
For each event in the diff matching items (a)-(q), add to your search list.
|
||||||
|
|
||||||
|
PHASE 2 — Execute searches and report:
|
||||||
|
For each item in your search list:
|
||||||
|
- Use Grep and Read tools to gather evidence from the codebase
|
||||||
|
- If match found (remove/deregister exists): discard
|
||||||
|
- If no match found: report as finding
|
||||||
|
|
||||||
|
Report ALL findings above a noise floor. For each: what's missing, where it should
|
||||||
|
be, what evidence you searched for, Confidence (High/Medium/Low).
|
||||||
|
Or "No absence finding" if nothing applies.
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. API Completeness & Contracts
|
||||||
|
|
||||||
|
**Priority**: missing fields, missing overrides, registration symmetry, visibility, event coverage.
|
||||||
|
|
||||||
|
**Reference**: `references/general/specialists/completeness.md`
|
||||||
|
|
||||||
|
**Prompt**:
|
||||||
|
```
|
||||||
|
You are an API COMPLETENESS specialist reviewing a patch for correctness bugs.
|
||||||
|
|
||||||
|
PHASE 0 — Understand the patch first (before reading checklist):
|
||||||
|
Read the patch: `{PATCH_PATH}`
|
||||||
|
1. Summarize what this patch does in 2-3 sentences.
|
||||||
|
2. List 3-5 things that could be incomplete about this change.
|
||||||
|
3. Note any new classes, new fields, new registrations, visibility changes.
|
||||||
|
|
||||||
|
PHASE 1 — Apply checklist:
|
||||||
|
Read your checklist: `{SKILL_DIR}/references/general/specialists/completeness.md`
|
||||||
|
Apply questions that match your hypotheses and code shapes. Check all 22 items
|
||||||
|
against the patch but spend the most effort where your hypotheses pointed.
|
||||||
|
|
||||||
|
Report ALL findings above a noise floor. For each: Location, Confidence
|
||||||
|
(High/Medium/Low), What's wrong (1-2 sentences).
|
||||||
|
Or "No finding in my domain" if nothing applies.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Pass 3: Cross-Path Symmetry (serial, after specialists complete)
|
||||||
|
|
||||||
|
**Prompt**:
|
||||||
|
```
|
||||||
|
You are a SYMMETRY specialist. For each code path modified in this patch, identify
|
||||||
|
whether a structurally parallel path (same class, interface, event family, version range)
|
||||||
|
was NOT modified. For each asymmetry:
|
||||||
|
- Where path A was changed
|
||||||
|
- Where parallel path B exists
|
||||||
|
- What was changed on A that is absent on B
|
||||||
|
- Whether the asymmetry is intentional or suspicious
|
||||||
|
|
||||||
|
Read the patch: `{PATCH_PATH}`
|
||||||
|
Search the codebase as needed using Grep and Read.
|
||||||
|
Report all asymmetries, Confidence High/Medium/Low.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Merge & Dedup
|
||||||
|
|
||||||
|
After all specialists report:
|
||||||
|
|
||||||
|
1. **Collect** all findings from 6 specialists + symmetry
|
||||||
|
2. **Deduplicate** — same code location + related reasons → combine
|
||||||
|
3. **Rank** by confidence — findings confirmed by multiple specialists rank highest
|
||||||
|
4. **Cross-check** reinforcement:
|
||||||
|
- Logic + Absence at same location → boost to High
|
||||||
|
- Concurrency + Resources on same lifecycle → boost to High
|
||||||
|
- Any finding flagged by 3+ specialists → treat as confirmed
|
||||||
|
5. **3-point test** each finding:
|
||||||
|
- The code construct actually exists in the diff (not inferred)
|
||||||
|
- The bug is possible given the visible context (not speculative)
|
||||||
|
- The finding is actionable (what specifically should change)
|
||||||
|
6. **Specialist Silence Rule**: If Absence or Completeness reports nothing on a >100-line
|
||||||
|
diff, note: "Consider verifying Phase 2 searches executed for registration symmetry,
|
||||||
|
handler coverage, and field completeness."
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Report Format
|
||||||
|
|
||||||
|
```
|
||||||
|
## Review: [target]
|
||||||
|
|
||||||
|
### Findings (ranked by confidence)
|
||||||
|
|
||||||
|
#### Finding 1: [title]
|
||||||
|
- **Location**: [file:line]
|
||||||
|
- **Confidence**: High / Medium / Low
|
||||||
|
- **Flagged by**: [specialists]
|
||||||
|
- **What's wrong**: [1-2 sentences]
|
||||||
|
|
||||||
|
### Specialist Coverage
|
||||||
|
- Logic: [N findings / no finding]
|
||||||
|
- Boundary: [N findings / no finding]
|
||||||
|
- Concurrency: [N findings / no finding]
|
||||||
|
- Resources: [N findings / no finding]
|
||||||
|
- Absence: [N findings / no finding]
|
||||||
|
- Completeness: [N findings / no finding]
|
||||||
|
- Symmetry: [N asymmetries / none]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Statistical Priors
|
||||||
|
|
||||||
|
| Category | Share | Specialist |
|
||||||
|
|---|---|---|
|
||||||
|
| Logic error in condition | 26% | Logic |
|
||||||
|
| Wrong constant / default | 15% | Logic |
|
||||||
|
| Missing null / bounds check | 13% | Boundary |
|
||||||
|
| Incorrect filtering / result | 11% | Logic |
|
||||||
|
| Race condition | 9% | Concurrency |
|
||||||
|
| Wrong serialization | 8% | Resources |
|
||||||
|
| State not cleaned up | 7% | Concurrency |
|
||||||
|
| Off-by-one | 4% | Boundary |
|
||||||
|
| Resource leak | 3% | Resources |
|
||||||
|
|
||||||
|
## Reference Files
|
||||||
|
|
||||||
|
### Specialist checklists (trimmed, ensemble mode)
|
||||||
|
- `references/general/specialists/logic.md` — 30 items, Logic specialist
|
||||||
|
- `references/general/specialists/boundary.md` — 22 items, Boundary specialist
|
||||||
|
- `references/general/specialists/concurrency.md` — 24 items, Concurrency specialist
|
||||||
|
- `references/general/specialists/resources.md` — 26 items, Resources specialist
|
||||||
|
- `references/general/specialists/absence.md` — Phase 1/2 search patterns, Absence specialist
|
||||||
|
- `references/general/specialists/completeness.md` — 22 items, Completeness specialist
|
||||||
|
|
@ -0,0 +1,98 @@
|
||||||
|
# Absence Analysis — Specialist Checklist
|
||||||
|
|
||||||
|
Core domain: purely absent code — missing guards, cleanup, handlers, conditions that leave
|
||||||
|
no visible signal in the diff. Two-phase agent: build a search list, then execute searches.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1: Build Search List
|
||||||
|
|
||||||
|
For each event in the diff, add an item to your search list:
|
||||||
|
|
||||||
|
### Registration & Lifecycle
|
||||||
|
- (a) New `addListener` / `register` / `subscribe` / `addMetric` call → Search for matching remove/deregister
|
||||||
|
- (b) New field added to a class → Verify presence in serialize/deserialize/equals/toString
|
||||||
|
|
||||||
|
### Event & Handler Coverage
|
||||||
|
- (c) New enum constant or event type → Search for ALL switch/dispatch/if-else chains; verify new constant handled
|
||||||
|
- (d) New `onTimeout()` / `onError()` override with short body → Search sibling classes for expected follow-up actions
|
||||||
|
|
||||||
|
### Resource Safety
|
||||||
|
- (e) New `AutoCloseable` created without try-with-resources → Identify exception exits before close
|
||||||
|
- (f) New object created in loop body (Event/Task/Request) → Check if enclosing class tracks in-flight instances
|
||||||
|
|
||||||
|
### Parallel Path Coverage
|
||||||
|
- (g) Method or code block added to path A with clear parallel path B → Verify path B has corresponding addition
|
||||||
|
- (h) New private/package-private field or method → Search for subclasses or cross-package classes needing access
|
||||||
|
|
||||||
|
## Phase 1b: High-Signal Absence Patterns
|
||||||
|
|
||||||
|
- (i) **Missing `return`**: non-void method called without return on its own line
|
||||||
|
- (j) **Missing try-with-resources**: AutoCloseable not wrapped, especially when parallel path in same file DOES wrap
|
||||||
|
- (k) **Missing interface override**: new class implements interface but doesn't override predicate whose default is wrong
|
||||||
|
- (l) **Missing class loading**: self-registering class not referenced from startup → registrations never execute
|
||||||
|
- (m) **Missing shutdown mechanism**: background scheduled task with no disable/stop path
|
||||||
|
- (n) **Missing empty return**: disabled path returns non-empty collection instead of empty
|
||||||
|
- (o) **Missing buffer unwrap**: pool buffer returned as slice/view instead of original allocation
|
||||||
|
- (p) **Missing wrapper usage**: reads/writes bypass tee/counting/checked wrapper stream
|
||||||
|
- (q) **Missing scope check**: code moved to new scope where variables no longer guaranteed non-null
|
||||||
|
- (r) **Missing config propagation**: value parsed or accepted by an outer layer but never threaded into the builder / constructor / factory that actually applies it — the setting silently falls back to its default
|
||||||
|
- (s) **Missing rollback on failure**: counter/collection populated on success path with no symmetric decrement/clear in the cancel, reject, throw, or timeout paths
|
||||||
|
- (t) **Missing retained reference**: object passed into a registry inside a factory/builder without being stored on `this` → the shutdown path has nothing to deregister
|
||||||
|
- (u) **Missing idempotency guard**: init/close/register can fire twice (schema reload, listener callback, reset path) without a has-run / already-closed flag
|
||||||
|
- (v) **Missing version/runtime fallback**: new wire field, protocol feature, or reflective JVM access has no branch for older peers / older JDKs / missing native library
|
||||||
|
- (w) **Missing drain on early exit**: checksum suffix, record footer, or remaining-items not consumed when an error / limit / break returns early from a framed stream
|
||||||
|
- (x) **Missing metric on sibling branch**: metric-record or state-propagation call present on one if/switch branch but absent from the symmetric branch or the cache-hit fast path
|
||||||
|
- (y) **Missing directory/fsync before dependent step**: deletion, listing, or cross-file rename proceeds without first fsyncing the directory or flushing sibling buffered writers
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2: Execute Searches
|
||||||
|
|
||||||
|
For each item in your search list:
|
||||||
|
- Use Grep and Read tools to gather evidence from the codebase
|
||||||
|
- If match found (remove/deregister exists): discard
|
||||||
|
- If no match found: report as finding with confidence level
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Fix-Patch Mode
|
||||||
|
|
||||||
|
When reviewing a **bug-fix** (not a feature), flip the question:
|
||||||
|
|
||||||
|
> "What was the original code missing — and did the fix add it EVERYWHERE?"
|
||||||
|
|
||||||
|
| What the fix added | Where else is it missing? |
|
||||||
|
|---|---|
|
||||||
|
| Null/bounds check on one path | Same lookup on parallel path (fast/slow, NIO/fallback) |
|
||||||
|
| `future.completeExceptionally(e)` | Other catch blocks or early-return paths |
|
||||||
|
| Feature-flag guard before operation | Setup, teardown, secondary operations |
|
||||||
|
| Version guard on new field | Matching `serializedSize()` and `deserialize()` branches |
|
||||||
|
| `close()`/`release()` on error path | Success path (also check for double-close) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What's NOT in the Patch
|
||||||
|
|
||||||
|
### Missing Symmetric Updates
|
||||||
|
- Every `serialize()` change needs matching `deserialize()` and `serializedSize()`
|
||||||
|
- New enum constant must appear in every switch/dispatch
|
||||||
|
- New field must appear in equals/hashCode/toString/clone/compareTo
|
||||||
|
- Renamed method must have zero references to old name
|
||||||
|
|
||||||
|
### Missing Callers
|
||||||
|
- Changed method signature: do all callers pass new parameter?
|
||||||
|
- Parameter parsed or validated but never forwarded: trace every hop from parse → store → builder → constructor → downstream factory
|
||||||
|
- New required field: do all constructors include it?
|
||||||
|
- New abstract method: do all subclasses implement it?
|
||||||
|
|
||||||
|
### Missing Guards
|
||||||
|
- Feature-flag check at every site touching guarded subsystem
|
||||||
|
- Mode guard handles all operating modes, not just default
|
||||||
|
- New entity property consulted by every mutation/deletion path
|
||||||
|
|
||||||
|
### Missing Error Handling
|
||||||
|
- Early return in checksummed read: still consume trailing checksum bytes?
|
||||||
|
- Every `catch` block mirrors the full exception wrapping chain?
|
||||||
|
- Error path that skips content: still advance position-tracking state?
|
||||||
|
- Rollback / counter decrement / cache eviction: symmetric on every failure path (cancel, reject, throw, timeout)?
|
||||||
|
|
@ -0,0 +1,72 @@
|
||||||
|
# Boundary & I/O — Specialist Checklist
|
||||||
|
|
||||||
|
22 highest-signal questions. 17% of all bugs fall in this domain.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Off-by-One & Range
|
||||||
|
|
||||||
|
1. [ ] Does this code iterate from the wrong start or stop one element too early/late? Check `<` vs `<=`, `begin` vs `begin + 1`, missing last chunk on exact boundary.
|
||||||
|
→ Also: round-robin wrap-around should use `>=` not `==`; pre-filled array loops must offset start index.
|
||||||
|
|
||||||
|
2. [ ] Does a range fail to handle `start == end` (degenerate/empty)? Does a binary search adjust by +1/-1 without handling the exact-match case?
|
||||||
|
|
||||||
|
3. [ ] Does an exponential backoff counter start at 1 when the formula assumes 0-based? `pow(base, attempts)` over-scales the first retry if attempts isn't decremented.
|
||||||
|
→ Also: cumulative-division loops accumulate floating-point residual, drifting into an extra iteration.
|
||||||
|
|
||||||
|
4. [ ] In a backward / reverse scan across a wrapping range (begin > end in token space), does the termination condition still hold? Strict `<` sentinels on wrapping ranges never terminate.
|
||||||
|
|
||||||
|
## Integer Overflow
|
||||||
|
|
||||||
|
5. [ ] Does integer-to-bytes conversion (`mebibytes * 1024 * 1024`) perform all multiplication in `int`? At least one operand needs `L` suffix. Does `(int) longValue` on a file size/offset truncate above 2GB?
|
||||||
|
|
||||||
|
6. [ ] Does integer division truncate to zero when divisor exceeds dividend? Does `(value / N) * N` produce zero when `value < N` — causing infinite loops or zero-sized allocations?
|
||||||
|
|
||||||
|
7. [ ] Does arithmetic on `Integer.MAX_VALUE` sentinel (e.g., `limit + 1`) overflow without a guard? Does a helper returning `long` compute intermediates in `int` before widening?
|
||||||
|
|
||||||
|
## Null & Bounds
|
||||||
|
|
||||||
|
8. [ ] Does this code dereference a lookup result (map, schema, metadata) without null check? Does `File.listFiles()` result get used without null check (returns null on I/O error)?
|
||||||
|
→ Also: shared field read twice without local capture → concurrent nullification between check and use.
|
||||||
|
|
||||||
|
9. [ ] Does this code index into array/list without checking emptiness? Does `subList(0, N)` or `list.get(0)` execute without a size guard?
|
||||||
|
|
||||||
|
10. [ ] Does a division use a runtime-derived divisor (collection size, mean) that can be zero?
|
||||||
|
|
||||||
|
11. [ ] Does `String.indexOf(...)` / `lastIndexOf(...)` feed directly into `substring(...)` or arithmetic without a `!= -1` guard? Does `split(...)` return a one-element array when the delimiter is absent, while the caller assumes multiple elements?
|
||||||
|
|
||||||
|
## Sentinel Values
|
||||||
|
|
||||||
|
12. [ ] When an API returns -1 (EOF), null (absent), or 0 (not found), does EVERY path check the sentinel BEFORE using the return value in arithmetic or narrowing cast?
|
||||||
|
|
||||||
|
13. [ ] Does a negative sentinel (`-1`/`-2`/large-negative) meaning "unset" or "infinite" get substituted for its effective value before arithmetic? `now + rawDuration` otherwise jumps into the past; sentinel-as-count causes infinite loops.
|
||||||
|
|
||||||
|
## ByteBuffer & I/O
|
||||||
|
|
||||||
|
14. [ ] Does `ByteBuffer.array()` call account for `arrayOffset()`? After `slice()`, offset is nonzero. Does no-arg `ByteBuffer.get()` advance position inside a comparator or shared view?
|
||||||
|
|
||||||
|
15. [ ] Does a `read()` call return fewer bytes than requested without the caller looping? Use `readFully()` for exact-count reads. NIO `Channel.read()` is never guaranteed to fill the buffer in one call.
|
||||||
|
|
||||||
|
16. [ ] Does `ByteBuffer.array()` get called on a potentially direct (off-heap) buffer without a `hasArray()` guard?
|
||||||
|
|
||||||
|
17. [ ] Does a write into a fixed-size buffer skip the `remaining() >= N` check before `put`? End-of-segment markers, trailers, or compressed output can overflow when the buffer is exactly full. Does a compression / encode loop exit on input-consumed rather than output-flushed, silently truncating the tail?
|
||||||
|
|
||||||
|
18. [ ] After writing into a ByteBuffer, is `flip()` used (sets limit = position) instead of `rewind()` (leaves limit at capacity) before handing it to a reader? `rewind()` exposes uninitialized bytes past the written region.
|
||||||
|
|
||||||
|
## Arithmetic
|
||||||
|
|
||||||
|
19. [ ] Does `n + 1 / 2` lack parentheses, evaluating as `n + 0`? Does a scaled index (`offset = i * stride`) get boundary-checked against the raw counter `i` instead of the scaled value?
|
||||||
|
|
||||||
|
20. [ ] Does a `subList`/`slice` end-index use a different origin (absolute vs relative) than the start index?
|
||||||
|
|
||||||
|
21. [ ] Does a `serializedSize()` helper double-count a conditional field (adding it unconditionally and again inside a branch), or count only the length prefix without the payload bytes? Declared size and actual written bytes then diverge, producing framing errors.
|
||||||
|
|
||||||
|
22. [ ] Does a write path leave a partial file on exception (no abort/delete, or no `TRUNCATE_EXISTING` when re-opening)? A subsequent read treats truncated or stale-tail content as complete.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## False Positives — Do NOT Flag
|
||||||
|
|
||||||
|
- Single-element collections passed to `size()` — only flag when size can be zero at runtime
|
||||||
|
- `ByteBuffer.array()` on buffers known to be heap-allocated (`ByteBuffer.allocate()`)
|
||||||
|
- Null checks after builders/constructors that guarantee non-null returns
|
||||||
|
|
@ -0,0 +1,71 @@
|
||||||
|
# API Completeness & Contracts — Specialist Checklist
|
||||||
|
|
||||||
|
22 highest-signal questions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Interface & Override Completeness
|
||||||
|
|
||||||
|
1. [ ] When a new class implements an interface or extends an abstract class, does it override every behavioral predicate method (`isX()`, `hasX()`, `canX()`) whose default (false/no-op) is wrong for this implementation?
|
||||||
|
→ Also: list ALL abstract/default methods; a class with TTL must override `isExpired()`.
|
||||||
|
|
||||||
|
2. [ ] Does a new class override all lifecycle methods (`close()`, `release()`, `abort()`) when it holds resources the parent doesn't know about?
|
||||||
|
|
||||||
|
3. [ ] When a new subtype / variant is added to a type family, is it handled in every type-dispatch site — not only enum switches, but also `instanceof` chains, visitor `visitXxx` methods, type-converter maps, and pattern-match cases? Silent fallthrough to default returns null or the wrong value.
|
||||||
|
|
||||||
|
4. [ ] Does a subclass that adds instance fields override the memory-measurement method (`unsharedHeapSize`, `estimateSize`, size query)? Does a decorator/wrapper override every interface method whose default implementation throws `UnsupportedOperationException`?
|
||||||
|
|
||||||
|
## Field Completeness
|
||||||
|
|
||||||
|
5. [ ] When a new field is added to a class, is it present in: serialize/deserialize/serializedSize, equals/hashCode, toString, copy constructors, builder `build()`, describe/toMap?
|
||||||
|
|
||||||
|
6. [ ] Does a descriptor/serializer method include ALL fields that sibling methods include? Compare field-by-field against `equals`, `hashCode`, `toString`, `serialize`.
|
||||||
|
|
||||||
|
7. [ ] Do `equals` / `hashCode` include EVERY field that determines identity (not only the display name or primary key), and does `equals` guard against mismatched operand types (or delegate to a type-specific comparator with a type guard)? Is a `Serializable` class's declared field type itself serializable — or is a `transient` field read without a custom `readObject` reconstructor?
|
||||||
|
|
||||||
|
## Registration Symmetry
|
||||||
|
|
||||||
|
8. [ ] For every registration (`addListener`, `register`, `subscribe`, `put` into registry, `addMetric`): is there a matching removal on every shutdown/close/destroy path — both success AND failure?
|
||||||
|
→ Also: if multiple registrations happen in sequence, does cleanup remove all even if one throws?
|
||||||
|
|
||||||
|
9. [ ] Is a self-registering class (`static final INSTANCE = new Foo()`) actually LOADED from the startup path? If nothing references the class, static init never runs, registrations are silently absent.
|
||||||
|
|
||||||
|
## Visibility
|
||||||
|
|
||||||
|
10. [ ] When a field/method is declared `private` or package-private: does any class outside the package or any subclass need access? `@VisibleForTesting` only widens to package-private, not beyond.
|
||||||
|
→ Also: was the member previously `protected` and narrowed by this patch? Check all subclasses.
|
||||||
|
|
||||||
|
## Accumulation
|
||||||
|
|
||||||
|
11. [ ] For `field = expression` where `field` tracks a running total across iterations: should it be `field += expression` instead? Check metric accumulators, counters, and aggregated stats in loops.
|
||||||
|
|
||||||
|
## Event & Dispatch Completeness
|
||||||
|
|
||||||
|
12. [ ] When a new enum constant or event type is added, is it handled in ALL switch statements, dispatch maps, and handler registrations?
|
||||||
|
→ Also: silent `default: break` may swallow events that should be errors.
|
||||||
|
|
||||||
|
13. [ ] Does a state machine handler (`onTimeout`, `onError`, `onRetry`) enqueue the appropriate follow-up action, or does it only log and return, leaving the state machine stuck?
|
||||||
|
|
||||||
|
14. [ ] When a new error code / exception class / response status is introduced, is it added to the error-MAPPING table (separate from the dispatch switch)? An unmapped code typically converts retriable to fatal — or fatal to silently-swallowed success.
|
||||||
|
|
||||||
|
## Method Binding & Refactoring
|
||||||
|
|
||||||
|
15. [ ] When a method adds a new parameter with a default-providing overload, do ALL callers that need the new behavior call the new signature? Old callers silently bind to the old overload.
|
||||||
|
|
||||||
|
16. [ ] After extracting code into a helper, does the call site retain an operation the helper also performs? (Double-write, double-close from extract-method refactoring.)
|
||||||
|
|
||||||
|
17. [ ] When a wrapper / delegate forwards a call to an inner implementation, are ALL arguments propagated — including "auxiliary" ones like tracing context, consistency level, auth credentials, client options — not just the primary value? A single layer that falls through to a no-arg overload silently drops caller intent.
|
||||||
|
|
||||||
|
18. [ ] When a code block is duplicated across versioned subdirectories (format variants, protocol-version-gated copies, legacy vs current implementations), has a bug-fix applied to one copy been propagated to every sibling? Grep for the fixed symbol across peer directories to find stale copies.
|
||||||
|
|
||||||
|
## Factory Routing
|
||||||
|
|
||||||
|
19. [ ] When a factory dispatches on a type parameter to create a concrete object: for each discriminator value, is the returned class CORRECT? After consolidating multiple factories, verify each value produces the same type as before.
|
||||||
|
|
||||||
|
20. [ ] If a class has multiple constructors or factory paths (primary vs secondary, from-bytes vs from-memory, copy-of vs fresh), does each path perform every required registration / validation / post-construct setup step, or is a required step skipped in the secondary path?
|
||||||
|
|
||||||
|
## Constants
|
||||||
|
|
||||||
|
21. [ ] Does this code call `Foo.bar()` where `Foo.BAR` (constant) is correct, or vice versa? Does it call the wrong overload when a more specific one enforces a required value?
|
||||||
|
|
||||||
|
22. [ ] Does a constructor accept a parameter that is never assigned to a field, silently discarding it? Does a field's inline initializer assign a constant that the constructor never overwrites, so caller-supplied values vanish?
|
||||||
|
|
@ -0,0 +1,84 @@
|
||||||
|
# Concurrency & State — Specialist Checklist
|
||||||
|
|
||||||
|
24 highest-signal questions. 16% of all bugs fall in this domain.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Races & Atomicity
|
||||||
|
|
||||||
|
1. [ ] Does this code read a shared field then conditionally write without holding a lock across both? (TOCTOU — check-then-act without atomicity.)
|
||||||
|
→ Also: compound operations on ConcurrentHashMap (get-then-put, check-then-act) across multiple maps need external synchronization.
|
||||||
|
|
||||||
|
2. [ ] Does this code iterate a shared mutable collection returned from a getter (`map.entrySet()`, `transaction.originals()`, `list.subList()`) without copying it first?
|
||||||
|
→ Also: `collection.remove()` inside for-each over same collection → ConcurrentModificationException.
|
||||||
|
|
||||||
|
3. [ ] Does `putIfAbsent()` return value get ignored, with caller using its own argument instead of the race winner?
|
||||||
|
|
||||||
|
4. [ ] Does a signal/notify (latch countDown, future complete) fire BEFORE the data structures that woken threads will read have been fully updated? Does a cache invalidation / tombstone broadcast fire BEFORE the underlying write has committed, letting a racing reader repopulate the cache with the pre-change value?
|
||||||
|
|
||||||
|
5. [ ] Does this code reuse a shared / pooled / singleton `ByteBuffer` via relative `get()` / `read()` without first calling `duplicate()` or `slice()`? Position advances as a side effect and corrupts subsequent readers on the same thread or concurrent observers.
|
||||||
|
|
||||||
|
6. [ ] Does a reader access a shared reference-counted resource (off-heap buffer, SSTable, Chunk, cache entry) without acquiring a refcount BEFORE the read? A concurrent evictor can free the resource between existence check and use.
|
||||||
|
|
||||||
|
7. [ ] After an atomic swap replaces a shared mutable collector / accumulator / landmark, can producers that already resolved to the OLD reference continue writing to the retired instance before it is drained?
|
||||||
|
|
||||||
|
8. [ ] Does a commit / compare-and-swap path re-read a version token or precondition value instead of reusing the one captured at the start of the transaction? A concurrent modification between the two reads bypasses conflict detection.
|
||||||
|
|
||||||
|
## Visibility & Fields
|
||||||
|
|
||||||
|
9. [ ] Does a non-final, non-volatile shared field get written on one thread and read on another without a lock?
|
||||||
|
→ Also: shared field read twice without local capture → concurrent nullification between check and use.
|
||||||
|
|
||||||
|
10. [ ] Does a `Comparator` read from a live mutable field during sort? Concurrent writes violate transitivity.
|
||||||
|
|
||||||
|
11. [ ] Is a shared field captured into a local under lock, but the downstream call inside the same critical section re-reads the original unsynchronized reference (e.g., `this.field.foo()` instead of `local.foo()`)? The snapshot is defeated.
|
||||||
|
|
||||||
|
## Locking & Deadlocks
|
||||||
|
|
||||||
|
12. [ ] Does a `synchronized` method call into another class that also has `synchronized`? Trace every path from within a `synchronized` block — if any blocks on I/O or acquires another lock, flag it.
|
||||||
|
→ Also: `static synchronized` calling code that loads another class → static initializer deadlock.
|
||||||
|
|
||||||
|
13. [ ] Does a message handler or task block on `future.get()` whose completion depends on the same thread pool?
|
||||||
|
→ Also: code running on a single-threaded / confined executor submitting new work to that SAME executor and blocking on the result.
|
||||||
|
|
||||||
|
14. [ ] Does a schema / metadata mutation path skip the per-store flush or compaction lock held by background tasks? Concurrent flushes can otherwise write files/index entries for a now-invalidated schema.
|
||||||
|
|
||||||
|
## Lifecycle & Ordering
|
||||||
|
|
||||||
|
15. [ ] Does a constructor spawn a thread, register a management callback, or start an executor capturing `this` before the constructor finishes?
|
||||||
|
|
||||||
|
16. [ ] Does this code register a listener AFTER the event-producing operation has started? Register first, then read state.
|
||||||
|
|
||||||
|
17. [ ] Does a shutdown step send a non-blocking "stop" flag/signal to a background task and then proceed to clear, truncate, or delete state the task depends on — without joining or awaiting the task? Does a coordinator process replies before the dispatch loop has finished registering all expected recipients?
|
||||||
|
|
||||||
|
## State Management
|
||||||
|
|
||||||
|
18. [ ] Does this code unconditionally `map.put(key, newValue)` where the map may already contain state that should be preserved or merged?
|
||||||
|
→ Also: creating a new object to represent an entity without checking whether a previous instance exists.
|
||||||
|
|
||||||
|
19. [ ] Does a reset/truncate/clear path update EVERY companion field atomically? (map + counter, union cache + source collections, dirty flag + buffer offset)
|
||||||
|
|
||||||
|
20. [ ] Does a teardown step, accounting update, or notification sit OUTSIDE the same `if` block as the operation it relates to? (Scope mismatch.)
|
||||||
|
|
||||||
|
21. [ ] Does a query / read consult only an immutable snapshot collection while a parallel "live current" pointer exists that has not yet been registered into the snapshot? A registration window returns a stale or empty answer.
|
||||||
|
|
||||||
|
## Counters & Accounting
|
||||||
|
|
||||||
|
22. [ ] Does this code increment a counter before an operation that may fail, without rollback on failure?
|
||||||
|
→ Also: success path missing cleanup that error path performs (session removal, counter decrement, map eviction).
|
||||||
|
|
||||||
|
## State Machines
|
||||||
|
|
||||||
|
23. [ ] Does a state-machine handler have branches that omit a required side-effect that all other branches perform?
|
||||||
|
→ Also: periodic task short-circuits on `!enabled` without retracting previously published state.
|
||||||
|
|
||||||
|
24. [ ] Does `condition.await()` or `Object.wait()` lack a `while(predicate)` loop guard against spurious wakeups?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## False Positives — Do NOT Flag
|
||||||
|
|
||||||
|
- CopyOnWriteArrayList/ConcurrentHashMap iteration (designed for safe concurrent access)
|
||||||
|
- Immutable `final` field set in constructor then read from other threads
|
||||||
|
- Volatile boolean flag for simple shutdown signal
|
||||||
|
- ThreadLocal in request-scoped code without async handoff
|
||||||
|
|
@ -0,0 +1,103 @@
|
||||||
|
# Logic & Types — Specialist Checklist
|
||||||
|
|
||||||
|
30 highest-signal questions. 52% of all bugs fall in this domain.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Control Flow & Return Values
|
||||||
|
|
||||||
|
1. [ ] Does a non-void method call appear on its own statement line WITHOUT `return`? `foo();` where `return foo();` was intended discards the result and falls through.
|
||||||
|
→ Also: check every `if/else` branch in non-void methods — each must return, throw, or break.
|
||||||
|
|
||||||
|
2. [ ] Does this code use `else if` where independent `if` statements are needed? If two conditions can both be true, `else if` silently skips the second.
|
||||||
|
→ Also: `continue` in multi-concern loop bodies suppresses unrelated work the same way.
|
||||||
|
|
||||||
|
3. [ ] Does a `switch` have `throw`/`return` after the closing brace instead of inside a `default:` case? It executes unconditionally after any matched branch.
|
||||||
|
|
||||||
|
4. [ ] Does `orElse(sideEffect())` appear where `orElseGet(() -> sideEffect())` is needed? `orElse` evaluates eagerly regardless of Optional presence.
|
||||||
|
|
||||||
|
5. [ ] Does a helper or initializer throw a generic `RuntimeException` / fire `assert` for a state that legitimately arises (snapshot files, counter verbs, static rows, concurrent not-found, zero-output compactions, alternative mode subclass)? Callers treat spurious assertion failures as hard errors.
|
||||||
|
|
||||||
|
## Comparisons & Equality
|
||||||
|
|
||||||
|
6. [ ] Does this code use `==` to compare objects with value semantics — strings from config, metadata after copy, boxed Long/Integer outside -128..127?
|
||||||
|
→ Also: `Objects.hash(array)` uses identity hash; need `Arrays.hashCode`.
|
||||||
|
|
||||||
|
7. [ ] Does this code compare raw bytes for typed data instead of using the type's own `compare()` method? Reversed types, composites, and floats all have special comparison semantics.
|
||||||
|
|
||||||
|
8. [ ] Does a single field or variable represent two semantically distinct events? E.g., `lastTime` set at operation START but tested as if it records FINISH. Each event needs its own field.
|
||||||
|
→ Also: initial value (0 or `now`) causes wrong elapsed-time on first run.
|
||||||
|
|
||||||
|
9. [ ] Does a range/overlap predicate skip the equal-bounds degenerate case (`min == max`) or treat a bound-inclusion as absolute when iteration is reverse-ordered? `left-inclusive` semantics flip under `ReversedType` / descending scans; slice exclusivity then applies to the wrong endpoint.
|
||||||
|
→ Also: version/threshold strings compared lexicographically report `"9" > "11"`.
|
||||||
|
|
||||||
|
## Constants & Defaults
|
||||||
|
|
||||||
|
10. [ ] Does a constant's unit match the expected unit? Does a `static final` capture a config value at class-load time, defeating runtime changes? Does a hardcoded literal after config parsing silently override file-based configuration?
|
||||||
|
|
||||||
|
11. [ ] When adding a new constant to a numbered enum or registry, does the numeric ID collide with an existing one? Scan ALL existing entries for duplicate IDs. Numbering gaps suggest deleted entries whose IDs must not be reused.
|
||||||
|
|
||||||
|
12. [ ] Does a disabled-feature path return a non-empty collection (`Arrays.asList(noOp)`, `singletonList(noOp)`) instead of `Collections.emptyList()`? Callers iterating the result execute spurious elements.
|
||||||
|
|
||||||
|
13. [ ] Does a unit-conversion expression multiply/divide in the WRONG direction (`bytes * 1024` for bytes→KB, `fraction * 100` where the caller expects a fraction)? Does a constructor silently ignore its own argument because the field is hardcoded at inline initializer and never reassigned?
|
||||||
|
|
||||||
|
## Parameters & Variables
|
||||||
|
|
||||||
|
14. [ ] Does this method accept multiple parameters of the same type where swapping them compiles but produces wrong results? (`min`/`max`, `start`/`end`, `expected`/`actual`, `coordinator`/`replica`)
|
||||||
|
|
||||||
|
15. [ ] When code is moved within a diff, do all variable references still hold at the new location? A variable non-null inside `if (x != null)` becomes potentially null if code moves outside that block.
|
||||||
|
→ Also: imports or local variables visible at old location but shadowed or absent at new.
|
||||||
|
|
||||||
|
16. [ ] After extracting code into a helper method, does the call site retain an operation the helper also performs? (Double-write, double-close from extract-method refactoring.)
|
||||||
|
→ Also: does the new helper miss an empty-input / not-null guard that only the original caller applied locally?
|
||||||
|
|
||||||
|
17. [ ] Does a retry / distribution / CAS loop pass a SOURCE object directly to a mutating helper, so later iterations operate on already-mutated data? Does a two-phase constructor compute a derived value (serialized size, routing target) BEFORE the second phase populates its inputs?
|
||||||
|
|
||||||
|
## Boolean & Predicate Errors
|
||||||
|
|
||||||
|
18. [ ] Does a boolean flag have inverted semantics relative to its name — `enabled` set when feature is off, `preload` true when preloading is skipped?
|
||||||
|
|
||||||
|
19. [ ] For a boolean guard whose name implies a compound reason (`bumpEpoch`, `hasChanged`, `needsRebalance`): does the assignment have fewer OR-conditions than the gated operation requires?
|
||||||
|
→ Also: compare against sibling guards in same method; count distinct actions the operation performs.
|
||||||
|
|
||||||
|
20. [ ] Does a predicate in a recursive type / wrapper / decorator hierarchy return `false` at the base class and get overridden only on leaf types? Container and decorator types then inherit the `false` default and are never recognized as containing the relevant sub-type.
|
||||||
|
|
||||||
|
## Time & Units
|
||||||
|
|
||||||
|
21. [ ] Does this code mix time units — microseconds added to nanosecond clock, nanoseconds passed to API expecting milliseconds? Does a nanoTime deadline use addition that overflows near `Long.MAX_VALUE`?
|
||||||
|
|
||||||
|
22. [ ] Does a progress / ratio display swap numerator and denominator, sample them at different moments, or mix compressed-bytes against uncompressed-total? Reported percentages become inverted, impossible, or wildly skewed.
|
||||||
|
|
||||||
|
## Iterators & Loops
|
||||||
|
|
||||||
|
23. [ ] Does this code advance an iterator with a single `if` where a `while` is needed — when chaining sub-iterators or skipping empty segments? Does a `pop()`/consume call process only one element when all should be consumed?
|
||||||
|
→ Also: recursive `computeNext()`/`hasNext()` overflows the stack on large inputs.
|
||||||
|
|
||||||
|
24. [ ] Inside a loop body, does a null-check / sentinel-guard expression reference a field of the OUTER container (or enclosing scope) instead of the freshly-fetched LOOP variable? The outer field evaluates the same every iteration; the per-element check never fires. Common after rename refactors.
|
||||||
|
|
||||||
|
25. [ ] Does a constructor / rebuild path wrap its input in an unmodifiable / defensive / lazy-union wrapper without first checking whether it is already wrapped? Repeated rebuilds build a linearly-growing delegation chain that overflows the stack on traversal.
|
||||||
|
|
||||||
|
## Sentinel Values
|
||||||
|
|
||||||
|
26. [ ] Does a sentinel value for "absent/unknown" overlap with a legitimate data value? (Zero-length meaning both "not set" and "empty but valid"; `-1` used for both EOF and valid index.)
|
||||||
|
→ Also: every code path must check sentinels (-1, null, 0) BEFORE using the return in arithmetic.
|
||||||
|
|
||||||
|
27. [ ] Is a counter/refcount/in-flight metric incremented on entry path with NO matching decrement on the cancel / reject / error / RejectedExecutionException path? In-flight counters silently grow unbounded when submissions fail.
|
||||||
|
|
||||||
|
## Wrapper & Stream Usage
|
||||||
|
|
||||||
|
28. [ ] After creating a wrapper stream (tee, counting, checked), does ALL subsequent I/O use the WRAPPER? A read/write bypassing the wrapper loses tracking, caching, or checksum guarantees.
|
||||||
|
|
||||||
|
29. [ ] Does a `serializedSize()` computation use a flag value or enum ordinal where `TypeSizes.sizeof(...)` is needed? Numeric coincidence breaks if the flag value ever changes.
|
||||||
|
|
||||||
|
30. [ ] Is a derived cache / topology / endpoint map populated at registration or construction time but never invalidated / refreshed when its source configuration is reloaded or the schema is altered?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## False Positives — Do NOT Flag
|
||||||
|
|
||||||
|
- Intentional no-op in documented disabled path returning a sentinel
|
||||||
|
- SLF4J `{}` with trivial arguments (`.toString()`, `.name()`)
|
||||||
|
- Package-private visibility on non-public API implementation classes
|
||||||
|
- Static final for JVM-lifetime settings (system properties, server ports)
|
||||||
|
- Lazy init without null guard in single-threaded context (per-partition, per-request)
|
||||||
|
|
@ -0,0 +1,91 @@
|
||||||
|
# Resources & Serialization — Specialist Checklist
|
||||||
|
|
||||||
|
26 highest-signal questions. 15% of all bugs fall in this domain.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Serialization
|
||||||
|
|
||||||
|
1. [ ] Do `serialize`, `deserialize`, and `serializedSize` write, read, and measure fields in exactly the same order under the same conditional guards?
|
||||||
|
→ Also: CRC/checksum must cover the same fields; version-gated branches must have matching size arithmetic.
|
||||||
|
|
||||||
|
2. [ ] Does `serializedSize()` return a hardcoded byte count? Verify it matches the actual type: 4 for int/float/LocalDate, 8 for long/double. If the represented type changed during refactoring, was the size constant updated?
|
||||||
|
|
||||||
|
3. [ ] Does a `serializedSize()` computation use a flag value or semantic constant where `TypeSizes.sizeof(...)` is needed? Numeric coincidence breaks if the flag value changes.
|
||||||
|
|
||||||
|
4. [ ] For a variable-length field: does `serializedSize()` count both the length prefix AND the payload bytes? Does it delegate to the on-disk / format-specific size helper, not the in-memory variant?
|
||||||
|
|
||||||
|
5. [ ] After wrapping a stream for monitoring/caching/tee (e.g., `TeeDataInputPlus`, `CountingOutputStream`), does ALL subsequent I/O use the WRAPPER, not the original? Bypass loses tracking/checksum.
|
||||||
|
|
||||||
|
6. [ ] When a `ByteBuffer` is obtained from a static singleton / cache / registry / cross-thread queue, is `duplicate()` called before any relative `get()` / comparator read? Does digest/array access from a heap buffer add `arrayOffset()` to `position()`?
|
||||||
|
|
||||||
|
## Resource Leaks
|
||||||
|
|
||||||
|
7. [ ] Does this code allocate an `AutoCloseable` resource without try-with-resources? Trace each exception exit point — is the resource closed on every path?
|
||||||
|
→ Also: multiple resources acquired in sequence need chaining; first leaks if second allocation fails.
|
||||||
|
|
||||||
|
8. [ ] When a `ByteBuffer` is obtained via `nioBuffer()`, `slice()`, `duplicate()` — is the ORIGINAL allocation returned to the pool, not the derived view? Pools cannot recycle views.
|
||||||
|
|
||||||
|
9. [ ] Does a shared `AutoCloseable` resource (stored in cache, map, or registry) use reference counting? Without it, the first consumer to `close()` invalidates all others.
|
||||||
|
→ Also: `close()` triggered from cache eviction while other threads hold references.
|
||||||
|
|
||||||
|
10. [ ] Does `close()` have an idempotency guard? Side-effecting callbacks (metrics, counters, listeners) fire twice on double-close.
|
||||||
|
|
||||||
|
11. [ ] Does a composite `close()` release EVERY Closeable it holds (not just the primary)? Do cancellation / rejected-submit / early-return paths release resources the caller already acquired? Is a field assigned BEFORE any constructor `throw`, so the owner retains a reference with which to close it?
|
||||||
|
|
||||||
|
12. [ ] Does a buffered / serializing writer call `flush()` + `fsync()` before `close()`, and `abort()` on exception? Plain `close()` on a buffered writer silently loses the last buffer; no-abort leaves a partial file treated as complete on restart.
|
||||||
|
|
||||||
|
## Metric Types
|
||||||
|
|
||||||
|
13. [ ] Does a latency metric use `Histogram` where `Timer` is correct? `Histogram` records dimensionless values; `Timer` records durations with time-unit support. Check consistency within the same class.
|
||||||
|
|
||||||
|
14. [ ] When a metric is recorded on one branch (success, or cache-miss), is the equivalent recording also present on the paired branch (failure, or cache-hit)? Asymmetry produces permanently divergent dashboards.
|
||||||
|
|
||||||
|
## Background Tasks & Lifecycle
|
||||||
|
|
||||||
|
15. [ ] Does a `static` initializer or `static final` field start a background scheduled task unconditionally? It fires during unit tests causing flakiness. Check for a disable/shutdown mechanism.
|
||||||
|
|
||||||
|
16. [ ] Does a `static final` capture a config value at class-load time that should support runtime reconfiguration? Check if the config has a setter, listener, or reload path.
|
||||||
|
|
||||||
|
17. [ ] Does a re-arming periodic task's reschedule decision depend on "did we do work" (correct) vs a side-effecting return value or transient-collection size (spins or stops)? Is a CLI tool / short-lived entrypoint reaping non-daemon threads before return?
|
||||||
|
|
||||||
|
## Test & Simulation Determinism
|
||||||
|
|
||||||
|
18. [ ] Does this code call a factory/parser without optional determinism parameters (randomizers, clocks, RNG seeds) needed for fuzz/simulation reproducibility?
|
||||||
|
→ Also: `Math.random()`, `new Random()`, `ThreadLocalRandom` in simulation-reachable paths break replay.
|
||||||
|
|
||||||
|
## Retry & Idempotency
|
||||||
|
|
||||||
|
19. [ ] Does this code path handle retry or replay? If so, does it exclude non-idempotent operations (counters, CAS) that produce wrong results when applied more than once?
|
||||||
|
|
||||||
|
## Overrides
|
||||||
|
|
||||||
|
20. [ ] Does a subclass adding new fields override every copy-producing method (`sharedCopy()`, `clone()`, copy constructor)? Does every method intended to override carry `@Override`?
|
||||||
|
→ Also: after superclass signature change, the old override silently becomes dead code without `@Override`.
|
||||||
|
|
||||||
|
## File I/O
|
||||||
|
|
||||||
|
21. [ ] Does this code write with `WRITE | CREATE` without `TRUNCATE_EXISTING` on a pre-existing file? Old content past new EOF remains as garbage.
|
||||||
|
|
||||||
|
## Version Compatibility
|
||||||
|
|
||||||
|
22. [ ] When a class is moved between modules, do the new serialize/deserialize methods handle ALL version-gated fields from the original? Compare field-by-field under each version guard.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
23. [ ] Does a constructor accept a parameter that is never assigned to a field, silently discarding config? Is the setting resolved lazily per-target (supplier) rather than frozen at construction time from a global default?
|
||||||
|
|
||||||
|
24. [ ] Does a feature flag guard the main operation but not setup, teardown, or secondary paths?
|
||||||
|
|
||||||
|
25. [ ] Is configuration propagated through EVERY layer (parse → validate → store → builder → constructor → downstream factory)? A single layer that hardcodes the default silently drops caller intent (tracing context, consistency level, auth credentials, client options are frequent victims).
|
||||||
|
|
||||||
|
26. [ ] Does the code use `ObjectMapper.findAndRegisterModules()`? Auto-discovery pulls in unintended modules from the classpath and silently changes serialization behavior — prefer explicit registration.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## False Positives — Do NOT Flag
|
||||||
|
|
||||||
|
- Single-use try-with-resources for one AutoCloseable (sufficient for simple cases)
|
||||||
|
- Static final for JVM-lifetime settings (system properties, JVM flags)
|
||||||
|
- Histogram for count-based distributions (partition sizes, key counts) — only flag for TIME
|
||||||
|
- Missing randomizer in non-simulation production code paths
|
||||||
|
|
@ -0,0 +1,272 @@
|
||||||
|
---
|
||||||
|
name: targeted-review
|
||||||
|
version: "1.0.0"
|
||||||
|
description: >
|
||||||
|
Findings-driven targeted code review. Understands the patch via patch-explainer and
|
||||||
|
surrounding code via codebase-analysis, then consults a catalog of bug patterns
|
||||||
|
(~11 categories under references/categories/). Picks only categories whose diff-signals
|
||||||
|
match the patch and items whose "look for" hints match actual code shapes, groups them
|
||||||
|
by review focus (function/file/feature), and spawns parallel subagents — each with a
|
||||||
|
tight scope and selective checklist. Use for medium-to-large patches (50-1000 LOC) as
|
||||||
|
a focused alternative to whole-patch ensemble or whole-file deep review. Triggers on:
|
||||||
|
"review this patch/diff/change", "find bugs in this change", "scoped review", "what
|
||||||
|
could go wrong here", "review using findings", or proactive PR review.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Targeted Review
|
||||||
|
|
||||||
|
A meta-review skill. It does not encode bug patterns directly — it orchestrates other
|
||||||
|
skills and a categorized findings catalog into a tight, evidence-driven review.
|
||||||
|
|
||||||
|
## What it does
|
||||||
|
|
||||||
|
```
|
||||||
|
+------------------+
|
||||||
|
| Input patch |
|
||||||
|
+--------+---------+
|
||||||
|
|
|
||||||
|
+-----------+-----------+
|
||||||
|
| |
|
||||||
|
+---------v---------+ +---------v-----------+
|
||||||
|
| patch-explainer | | codebase-analysis |
|
||||||
|
| (sub-skill) | | (sub-skill, scoped) |
|
||||||
|
| what changed, | | invariants, callers,|
|
||||||
|
| flow, assumptions | | parallel paths |
|
||||||
|
+---------+---------+ +---------+-----------+
|
||||||
|
| |
|
||||||
|
+-----------+-----------+
|
||||||
|
|
|
||||||
|
+---------------v----------------+
|
||||||
|
| Pick categories + items |
|
||||||
|
| × 3-5 independent iterations |
|
||||||
|
| (each draw surfaces different |
|
||||||
|
| items from the same catalog) |
|
||||||
|
+---------------+----------------+
|
||||||
|
|
|
||||||
|
+--------v---------+
|
||||||
|
| Pool & prioritize|
|
||||||
|
| (items from 2+ |
|
||||||
|
| iterations → |
|
||||||
|
| higher priority)|
|
||||||
|
+--------+---------+
|
||||||
|
|
|
||||||
|
+--------v---------+
|
||||||
|
| Group items by |
|
||||||
|
| review focus |
|
||||||
|
| (file/func/feat) |
|
||||||
|
+--------+---------+
|
||||||
|
|
|
||||||
|
+------------+------------+
|
||||||
|
| | |
|
||||||
|
+-------v---+ +-----v-----+ +---v-------+
|
||||||
|
| review | | review | | review |
|
||||||
|
| subagent | | subagent | | subagent |
|
||||||
|
| scope A | | scope B | | scope C |
|
||||||
|
+-------+---+ +-----+-----+ +---+-------+
|
||||||
|
| | |
|
||||||
|
+------------+------------+
|
||||||
|
|
|
||||||
|
+--------v---------+
|
||||||
|
| Merge & report |
|
||||||
|
+------------------+
|
||||||
|
```
|
||||||
|
|
||||||
|
## When to use this skill
|
||||||
|
|
||||||
|
- A patch or diff is on the table for review (single commit, multi-commit branch, or staged changes)
|
||||||
|
- The user wants more focus than `shallow-review` provides (six fixed lenses on the whole patch) but doesn't want to commit to `deep-review`'s file-by-file deep pass
|
||||||
|
- A `*-findings/` corpus exists or the categorized catalog under `references/categories/` is sufficient — the categories alone are usable; project corpora improve coverage
|
||||||
|
- Medium-to-large patches where dispatching focused subagents avoids overwhelming a single reviewer
|
||||||
|
|
||||||
|
## When NOT to use this skill
|
||||||
|
|
||||||
|
- Trivial 1-3 line changes — direct review is faster
|
||||||
|
- Pure refactors with no behavior change — `shallow-review` symmetry pass is enough
|
||||||
|
- The user wants a single comprehensive write-up of one file — use `deep-review` instead
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
You will go through seven phases. Phases 1a and 1b run in parallel. Phases 2-6 are sequential.
|
||||||
|
|
||||||
|
### Phase 1a: Understand the patch — call the patch-explainer skill
|
||||||
|
|
||||||
|
Invoke the `patch-explainer` skill on the patch. Use the Skill tool. You want patch-explainer
|
||||||
|
to produce its standard analysis (purpose, before/after diagrams, flow, assumptions,
|
||||||
|
failure modes).
|
||||||
|
|
||||||
|
If the patch is already provided as a file path, point patch-explainer at it. If you only
|
||||||
|
have a commit ref, ask for `git show <ref>` or read it from `git diff`. For multi-commit
|
||||||
|
branches, it is fine to feed the cumulative diff.
|
||||||
|
|
||||||
|
Capture patch-explainer's full output. You will refer to it in Phases 3-5.
|
||||||
|
|
||||||
|
### Phase 1b: Understand the surroundings — call the codebase-analysis skill
|
||||||
|
|
||||||
|
In parallel with 1a, invoke the `codebase-analysis` skill in **targeted mode** on the
|
||||||
|
affected subsystem only — not the whole repo. Pass it:
|
||||||
|
|
||||||
|
- The list of files touched by the patch
|
||||||
|
- The subsystem name if obvious (e.g., "the journal replay subsystem")
|
||||||
|
- A directive to focus on: invariants, lifecycle, parallel implementations, and recent
|
||||||
|
bug history in the affected area
|
||||||
|
|
||||||
|
You want codebase-analysis to produce a scoped `analysis/system-analysis.md` that gives
|
||||||
|
you the context the patch sits inside — what the surrounding code expects, where parallel
|
||||||
|
paths exist, what historical bugs touched this area.
|
||||||
|
|
||||||
|
If invoking the full skill is too heavy for a small patch, instead spawn a research
|
||||||
|
subagent with this prompt: "Read these files: {touched files}. Summarize: (a) the
|
||||||
|
invariants the surrounding code maintains, (b) any parallel implementations or sibling
|
||||||
|
classes, (c) callers of the modified methods, (d) any recent bug-fix commits in this
|
||||||
|
area. Under 400 words." This is a lighter alternative that retains the spirit of
|
||||||
|
codebase-analysis. Prefer the full skill when the area is unfamiliar or large.
|
||||||
|
|
||||||
|
### Phase 2: Pick categories — 3-5 independent iterations
|
||||||
|
|
||||||
|
Category selection and item picking are speculative: you're making probabilistic judgments
|
||||||
|
about which patterns might apply, and a single pass will reliably miss some. Run **3-5
|
||||||
|
independent iterations** of Phases 2+3, each starting from scratch with the same patch
|
||||||
|
and context but without looking back at what prior iterations chose. Treat each iteration
|
||||||
|
like a fresh read — you are more likely to notice different things each time.
|
||||||
|
|
||||||
|
For each iteration, read `references/categories/INDEX.md` — it lists all 11 categories
|
||||||
|
with their descriptions and diff signals in one file (~200 lines). For each category, ask:
|
||||||
|
does the patch contain ANY of the diff signals listed?
|
||||||
|
- If yes → mark as **load**
|
||||||
|
- If no → skip
|
||||||
|
|
||||||
|
Typically 3-7 categories load per iteration. If you would load more than 8, re-examine.
|
||||||
|
If fewer than 2, the patch is probably too small for this skill.
|
||||||
|
|
||||||
|
### Phase 3: Pick items within each loaded category (one pass per iteration)
|
||||||
|
|
||||||
|
Within each iteration, read the **full** content of each loaded category file. For each
|
||||||
|
finding (`### F-NN: title` + body + `**Look for:** {hint}`), ask:
|
||||||
|
|
||||||
|
1. Does the "Look for" hint match a code shape that **actually appears in this patch**?
|
||||||
|
2. Does the patch context (from patch-explainer + codebase-analysis) make this finding
|
||||||
|
**plausibly applicable** — not just a keyword match?
|
||||||
|
|
||||||
|
If yes to both → keep the item.
|
||||||
|
If no → drop.
|
||||||
|
|
||||||
|
Be selective: a category typically has 25-40 findings; keep 3-12 per category per
|
||||||
|
iteration. Repeat for all iterations before moving to Phase 3b.
|
||||||
|
|
||||||
|
### Phase 3b: Pool and prioritize across iterations
|
||||||
|
|
||||||
|
After all iterations are complete, merge the item lists:
|
||||||
|
|
||||||
|
- An item selected in **2 or more iterations** is a strong candidate — mark it **priority**.
|
||||||
|
- An item selected in only one iteration is still in scope — mark it **speculative**.
|
||||||
|
- Discard nothing yet; let the subagents decide.
|
||||||
|
|
||||||
|
The iteration count next to an item is a confidence signal for the subagents, not a
|
||||||
|
gate. A speculative item can still turn into a real bug; a priority item can still be
|
||||||
|
a false alarm.
|
||||||
|
|
||||||
|
### Phase 4: Group items by review focus
|
||||||
|
|
||||||
|
Now you have a pooled list of selected items across categories and iterations. Group them
|
||||||
|
by **review focus** — what part of the change a single reviewer should examine. A focus
|
||||||
|
is one of:
|
||||||
|
|
||||||
|
- A specific function or method (`MyClass.replay()`)
|
||||||
|
- A specific file or small group (`FooSerializer.java` and its sibling `FooDeserializer.java`)
|
||||||
|
- A feature, behavior, or concept ("the new fsync coordination", "the version-gated dispatch path")
|
||||||
|
- A specific cross-cutting concern in the patch ("any place we read the new `dirty` flag")
|
||||||
|
|
||||||
|
Aim for 2-5 foci per patch. Each focus should have **5-15 checklist items** drawn from
|
||||||
|
across the categories — not too few (subagent has nothing to do) and not too many
|
||||||
|
(subagent gets overwhelmed). Include each item's priority annotation (priority /
|
||||||
|
speculative) so the subagent knows where to focus first.
|
||||||
|
|
||||||
|
If multiple foci would receive the same checklist item, that's fine — the same pattern
|
||||||
|
can apply at multiple sites. Tag each item with the category it came from so the
|
||||||
|
subagent knows the rationale.
|
||||||
|
|
||||||
|
### Phase 5: Spawn focused review subagents
|
||||||
|
|
||||||
|
For each focus, spawn one review subagent in parallel using the Agent tool. Use the
|
||||||
|
template in `references/subagent-prompt-template.md`. The subagent prompt must contain:
|
||||||
|
|
||||||
|
- The patch (path, or pasted, or ref)
|
||||||
|
- A 100-word summary of patch-explainer's findings (relevant slice only — not the full output)
|
||||||
|
- A 100-word summary of codebase-analysis's findings (relevant slice only)
|
||||||
|
- The focus statement (what part of the change to review)
|
||||||
|
- The checklist items, each with: category, priority annotation, finding text, and
|
||||||
|
look-for hint
|
||||||
|
- Instructions on how to gather evidence (read source, grep, check parallel paths) and
|
||||||
|
what confidence to assign
|
||||||
|
|
||||||
|
Each subagent should produce: a list of findings, with location, confidence, and what
|
||||||
|
evidence was checked.
|
||||||
|
|
||||||
|
Run all subagents in a single tool call (parallel). They are independent.
|
||||||
|
|
||||||
|
### Phase 6: Merge and report
|
||||||
|
|
||||||
|
When all subagents return, merge findings:
|
||||||
|
|
||||||
|
1. **Deduplicate** — same code location + similar reasoning → one finding, both subagents
|
||||||
|
credited
|
||||||
|
2. **Rank** by confidence — High first, then Medium, then Low
|
||||||
|
3. **Cross-reinforce** — if a finding was raised by multiple subagents, or if multiple
|
||||||
|
findings cluster on the same lifecycle/state, boost confidence
|
||||||
|
4. **Three-point check** each surviving finding:
|
||||||
|
- The code construct actually exists in the diff (not inferred from absence alone)
|
||||||
|
- The bug is plausible given the visible context (not speculative)
|
||||||
|
- The finding is actionable (the reader can tell what to change)
|
||||||
|
|
||||||
|
Use the format in `references/report-format.md`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Reference files
|
||||||
|
|
||||||
|
| Reference | When to read |
|
||||||
|
|---|---|
|
||||||
|
| `references/subagent-prompt-template.md` | Phase 5, when spawning each review subagent |
|
||||||
|
| `references/report-format.md` | Phase 6, when merging and presenting findings |
|
||||||
|
| `references/categories/INDEX.md` | Phase 2 — single-file scan of all category descriptions + diff signals |
|
||||||
|
| `references/categories/<name>.md` | Phase 3 — full body of each loaded category, picked from INDEX |
|
||||||
|
|
||||||
|
The categories in `references/categories/` are project-agnostic patterns mined from real
|
||||||
|
bug fixes across distributed-systems projects (Cassandra, Kafka, Iceberg). They describe
|
||||||
|
generalized code-level shapes, not project-specific issues.
|
||||||
|
|
||||||
|
## How this differs from related skills
|
||||||
|
|
||||||
|
| Skill | Scope | Pattern source | Subagent dispatch |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `shallow-review` (cassandra-edge-case-explorer) | Whole patch | 6 fixed lenses (~100 items total) | One subagent per lens, fixed |
|
||||||
|
| `deep-review` | User-named files | 444-pattern catalog | One subagent per file, full catalog |
|
||||||
|
| `targeted-review` (this) | Whole patch, focus-decomposed | ~11 categories (300+ findings) — selectively loaded | One subagent per focus, selectively populated checklist |
|
||||||
|
| `mega-review` | Large branches | Orchestrates the above | Multi-pass |
|
||||||
|
|
||||||
|
Use `targeted-review` when:
|
||||||
|
- The patch spans 50-1000 LOC
|
||||||
|
- You want to spend tokens on **what matters for THIS patch**, not on a fixed catalog
|
||||||
|
- You have a `*-findings/` corpus or are reviewing systems-level code that benefits from
|
||||||
|
the curated categories
|
||||||
|
|
||||||
|
## Pitfalls and guardrails
|
||||||
|
|
||||||
|
- **Don't skip Phase 1.** Calling patch-explainer and codebase-analysis is what lets you
|
||||||
|
pick categories and items intelligently. Without that grounding, you'll loose-match
|
||||||
|
on keywords and overwhelm subagents.
|
||||||
|
- **Don't look at prior iterations while picking.** The whole value of multiple iterations
|
||||||
|
is independent draws. If you anchor on the first iteration's selections, you get one
|
||||||
|
draw repeated instead of several independent ones.
|
||||||
|
- **Don't load every category "just in case".** The whole value is selection. If you
|
||||||
|
match more than 8 categories in an iteration, you've probably been too generous.
|
||||||
|
- **Don't pass the full catalog to subagents.** Each subagent gets only the items
|
||||||
|
selected for its focus. Passing all items defeats the purpose.
|
||||||
|
- **Don't substitute findings for understanding.** A finding that "matches a code shape"
|
||||||
|
is a hypothesis to verify — the subagent confirms or refutes by reading the code, not
|
||||||
|
by trusting the pattern. Mark uncertain findings as Low confidence.
|
||||||
|
- **The checklist is the menu, not the order.** A subagent should also report bugs it
|
||||||
|
notices that aren't on the checklist — the checklist primes attention, it doesn't cap
|
||||||
|
it.
|
||||||
|
|
@ -0,0 +1,209 @@
|
||||||
|
# Categories index
|
||||||
|
|
||||||
|
Quick-scan view of all categories with their diff signals. Use this in Phase 2 of the
|
||||||
|
workflow to decide which categories to load. After picking categories, read the full
|
||||||
|
file for each (e.g., `serialization-and-versioning.md`) in Phase 3.
|
||||||
|
|
||||||
|
Each category file follows the same structure:
|
||||||
|
1. `# Category: Name`
|
||||||
|
2. One-paragraph description
|
||||||
|
3. `## Diff signals (when to load this category)` — bullet list of patch shapes
|
||||||
|
4. `## Findings` — numbered findings, each with body and `**Look for:**` hint
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## api-contracts-and-completeness
|
||||||
|
|
||||||
|
Bugs where adding a new field, type, event, or capability requires symmetrical updates across multiple sites — overrides, equality/hashCode, builders, dispatch tables, switch cases, register/deregister pairs — and one or more required updates are silently omitted, leaving the system structurally inconsistent.
|
||||||
|
|
||||||
|
**Diff signals:**
|
||||||
|
- A new field added to a class that has `equals`, `hashCode`, `toString`, `compareTo`, copy, clone, or builder methods
|
||||||
|
- A new enum constant or new subclass added (look for missing cases in `switch` statements or `instanceof` chains elsewhere)
|
||||||
|
- A new method added to an interface or abstract class (look for missing overrides in implementations)
|
||||||
|
- A new event/message/verb type registered (look for unhandled cases in dispatchers)
|
||||||
|
- New `register`, `addListener`, `subscribe`, `addCloseable`, `acquire`, or symmetric pair operations
|
||||||
|
- A new serializer/deserializer overload, signature change, or wire-format field
|
||||||
|
- New constructor parameter, builder method, or option that must be propagated to factory call sites
|
||||||
|
- Changes to a serialization size method, write method, or read method (the three must stay in sync)
|
||||||
|
- A `default` method in an interface that returns a hard-coded constant (subclasses may need to override)
|
||||||
|
- A new metric, management endpoint, sensor, or gauge registration (verify deregistration paths)
|
||||||
|
- A new field in a request/response/copy builder (verify it's copied in `copy`/`from`/`toBuilder` paths)
|
||||||
|
- New configuration property added to a class (verify it's forwarded through every constructor and factory)
|
||||||
|
- A subclass adds fields, resources, or dependencies (verify base-class lifecycle methods are overridden)
|
||||||
|
|
||||||
|
## boundaries-and-numbers
|
||||||
|
|
||||||
|
Bugs involving numeric arithmetic, integer overflow, off-by-one errors, unit/dimensional mismatches, buffer-bound violations, sentinel/special-value confusion, and floating-point pitfalls — wherever a number, index, or quantity is computed, compared, or interpreted incorrectly.
|
||||||
|
|
||||||
|
**Diff signals:**
|
||||||
|
- Arithmetic operators (`+`, `-`, `*`, `/`, `%`, `<<`, `>>`) on `int`/`long`/`short` fields, especially before a widening cast
|
||||||
|
- Index expressions (`a[i]`, `list.get(i)`), `subList`, `slice`, or pre/post-increment in a loop body
|
||||||
|
- Comparison operators (`<`, `<=`, `>`, `>=`, `==`) used as loop bounds, threshold checks, or boundary conditions
|
||||||
|
- Casts between numeric types (`(int)`, `(short)`, `(long)`, `toInt`, `toLong`)
|
||||||
|
- Unit-bearing names or constants (`ms`, `Ms`, `Nanos`, `Seconds`, `MB`, `KB`, `Bytes`, `MiB`, `MILLIS_PER_*`, `BITS_PER_*`)
|
||||||
|
- `TimeUnit`, `Duration`, `Instant`, `currentTimeMillis()`, `nanoTime()`
|
||||||
|
- ByteBuffer operations: `position()`, `limit()`, `flip()`, `rewind()`, `slice()`, `duplicate()`, `arrayOffset()`, `array()`, `getInt()`, `putShort()`
|
||||||
|
- Buffer/array allocation sized from an external length or count (`new byte[len]`, `ByteBuffer.allocate(...)`)
|
||||||
|
- Sentinel constants like `-1`, `Long.MAX_VALUE`, `Integer.MIN_VALUE`, `MAX_VALUE`, `Short.MAX_VALUE`
|
||||||
|
- Division operations that may produce zero from integer truncation or divide by an unguarded denominator
|
||||||
|
- Floating-point accumulators, percentage/ratio calculations, `Math.round`, `setScale`, `BigDecimal.toPlainString`
|
||||||
|
- Histogram/percentile/bucket/level/tier code with bucket array indexing
|
||||||
|
- TTL, expiry, deadline, timeout, backoff, throttle, or rate-limit calculations
|
||||||
|
- Length-prefixed encoding/decoding with explicit width (2-byte, 4-byte, varint) or `serializedSize()` methods
|
||||||
|
- New configuration fields with numeric defaults or unit suffixes
|
||||||
|
|
||||||
|
## concurrency-and-locking
|
||||||
|
|
||||||
|
Bugs caused by races, ordering hazards, lock-discipline violations, publication-safety gaps, and circular waits between cooperating threads or async tasks.
|
||||||
|
|
||||||
|
**Diff signals:**
|
||||||
|
- New or changed `synchronized`, `Lock`, `ReadWriteLock`, `ReentrantLock`, `mutex`, semaphore, latch, or condition-variable usage
|
||||||
|
- `volatile`, `Atomic*`, `AtomicReference`, `compareAndSet`, `getAndSet`, or memory-fence-related changes
|
||||||
|
- `Future`, `CompletableFuture`, `await`, `get()`, `join()`, callbacks chained on async results
|
||||||
|
- A new background thread, executor submission, scheduler, or task pool — especially with bounded queues or single-threaded executors
|
||||||
|
- Changes to publish-then-signal sequences (set flag / count-down / notify / complete a future after writing shared state)
|
||||||
|
- Iteration over a shared/concurrent collection without explicit lock acquisition, or methods returning live views of shared collections
|
||||||
|
- Lazy or double-checked initialization of fields read from multiple threads
|
||||||
|
- Reference-counted resource lifecycle calls (`acquire`, `release`, `retain`, refcount inc/dec)
|
||||||
|
- Cancellation, abort, shutdown, or close paths that interact with in-flight work
|
||||||
|
- Reading "current state" / membership / counters as separate steps from acting on them (check-then-act, observe-then-acquire)
|
||||||
|
- Comparators or sort keys that read mutable fields, metrics, or time
|
||||||
|
- Atomic swap-then-drain, swap-then-clear, or replace-then-process patterns
|
||||||
|
|
||||||
|
## conditions-and-predicates
|
||||||
|
|
||||||
|
Bugs where a comparison, guard, predicate, or branching condition is logically wrong: wrong operator polarity, wrong constant or field, wrong direction flag, asymmetric vs symmetric comparator confusion, missing replay-time filter, conflated conditions, or a sentinel guard that tests the wrong variable.
|
||||||
|
|
||||||
|
**Diff signals:**
|
||||||
|
- New or modified comparison operators (`<`, `<=`, `>`, `>=`, `==`, `!=`) in a guard, loop condition, or boundary test
|
||||||
|
- A boolean expression that gates a side effect, branch, early return, or filter step
|
||||||
|
- An `equals()`, `hashCode()`, `compareTo()`, or `Comparator` implementation (added, modified, or its field set changed)
|
||||||
|
- A predicate method (`isXxx`, `hasXxx`, `canXxx`, `matches`, `contains`) added or modified
|
||||||
|
- A `switch` over an enum or type discriminator (especially with `default` branches or fall-through)
|
||||||
|
- A direction/order flag (ascending/descending, forward/reverse, oldest/newest) passed to a helper
|
||||||
|
- An error message or log statement that quotes a constant, threshold, or field name
|
||||||
|
- A composite key access where indexed positions or named components are read
|
||||||
|
- A `filter`, `where`, `predicate`, or `Predicate<T>` lambda added, removed, or re-targeted
|
||||||
|
- A type check via `instanceof`, `getClass() ==`, or a custom type-discriminator predicate
|
||||||
|
- Changes to membership tests (`contains`, `containsKey`, `Set.of`, allowlist/denylist)
|
||||||
|
- Sentinel value handling (`-1`, `null`, `Long.MAX_VALUE`, `EMPTY_BUFFER`) in a comparison or guard
|
||||||
|
|
||||||
|
## io-and-crash-safety
|
||||||
|
|
||||||
|
Bugs in durable persistence, flush/sync ordering, atomic file replacement, partial reads/writes, checksum handling, and journal/log replay where crashes, errors, or restarts can corrupt data, lose writes, or resurrect deleted state.
|
||||||
|
|
||||||
|
**Diff signals:**
|
||||||
|
- New or changed file writes, especially overwrite-in-place, rename/move, or `FileChannel.write` / `OutputStream.write` paths
|
||||||
|
- Calls to `fsync`, `force()`, `flush()`, `sync()`, `close()` on file/channel/writer objects, or removal/addition of any of these
|
||||||
|
- Use of `FileChannel.write`, `read`, `read(ByteBuffer)`, `InputStream.read(byte[])` without explicit "read fully" / "write fully" loops
|
||||||
|
- Checksum / CRC / digest computation tied to a file or record (`CRC32`, `Adler32`, `MessageDigest`, `Hashing`, custom digest update over a buffer)
|
||||||
|
- Commit-log, journal, write-ahead-log, replay, recovery, or checkpoint code paths
|
||||||
|
- Truncation, snapshot, or restore operations on files or persistent state
|
||||||
|
- Auto-closing serializer / try-with-resources around a writer that owns the file lifecycle
|
||||||
|
- Atomic-rename helpers (`Files.move`, `ATOMIC_MOVE`, `renameTo`, `replace`) or marker-file presence checks for operation completion
|
||||||
|
- Buffer pool / off-heap memory release coordinated with disk I/O completion
|
||||||
|
- Multiple related files written together (data + index, data + summary, manifest + segments)
|
||||||
|
- File-open flags such as `TRUNCATE_EXISTING`, `CREATE_NEW`, `APPEND`, `O_DSYNC`, or their absence
|
||||||
|
|
||||||
|
## lifecycle-and-ordering
|
||||||
|
|
||||||
|
Bugs caused by performing operations in the wrong sequence relative to a component's lifecycle: registering listeners after events fire, exposing services before initialization completes, tearing down dependencies in the wrong order, polling without deadlines, or assuming async work is done when it isn't.
|
||||||
|
|
||||||
|
**Diff signals:**
|
||||||
|
- New `register*`/`addListener`/`subscribe`/`addObserver` calls or changes to where they are placed
|
||||||
|
- Changes to constructor or `init()`/`start()` ordering, or new initialization phases
|
||||||
|
- Changes to `close()`/`shutdown()`/`stop()` ordering, drain loops, or teardown sequencing
|
||||||
|
- New management endpoint/metrics registration, or unregistration paths
|
||||||
|
- Polling loops (`while (!ready)`, `await*Until*`, retry-on-condition) without explicit deadlines
|
||||||
|
- New asynchronous tasks (`submit`, `schedule`, `Future`, `CompletableFuture`) that produce state observable elsewhere
|
||||||
|
- Conditional readiness flags, "is started/initialized/ready" guards, or status-broadcast methods
|
||||||
|
- Changes to bootstrap/join/handshake/handoff/quarantine sequences
|
||||||
|
- Background thread spawning (especially inside constructors or factory methods)
|
||||||
|
- Static initializers / class-load-time evaluation of configuration or singletons
|
||||||
|
|
||||||
|
## null-and-type-safety
|
||||||
|
|
||||||
|
Bugs where null-valued fields, return values, or unset optional fields are dereferenced without a guard, and bugs where a runtime type assumption (cast, instanceof, generic narrowing) does not hold for all values that actually flow through the code path.
|
||||||
|
|
||||||
|
**Diff signals:**
|
||||||
|
- A new `cast`, `(SomeType) x`, `instanceof`, or generics narrowing on a value that crosses an API boundary or comes from a polymorphic source.
|
||||||
|
- A new lookup result (`map.get`, `registry.lookup`, `find`, schema/metadata lookup, `socket.getChannel`, `File.list`) chained or dereferenced without a null guard.
|
||||||
|
- A new optional / nullable field added to a config, schema, protocol record, or wire format (especially "introduce nullable union", "make field optional", "schema evolution").
|
||||||
|
- New `Optional` usage (`.get()`, `.orElse(null)`, `orElseThrow`, eager-evaluated `orElse(...)` arguments).
|
||||||
|
- Changes to constructors / lifecycle (`close()`, `cleanup()`, `shutdown()`) where a field may be null on early-failure or never-initialized paths.
|
||||||
|
- Changes that move a null check, change `&&`/`||` connecting null-checks, or add/remove `containsKey` before `get`.
|
||||||
|
- Refactor that widens a return type to a supertype or moves a method onto a more abstract type while existing callers/fields keep the concrete type.
|
||||||
|
- New `toArray()`, `Collectors.toMap`, auto-unboxing of `Map.get`, or `Long`/`Integer` reference-equality (`==`) comparisons.
|
||||||
|
- A new factory or builder whose return type is wider/narrower than callers assume.
|
||||||
|
|
||||||
|
## refactor-aftermath
|
||||||
|
|
||||||
|
Bugs whose root cause is incomplete or inconsistent refactoring — leftover guards, stale references after a rename, narrowed/broadened types not propagated to all sites, dead branches preserved past their owning feature, deprecated overloads with stale defaults, mixed-version mismatches between the old and new shape, and parsers/regexes that handle only the new format.
|
||||||
|
|
||||||
|
**Diff signals:**
|
||||||
|
- A symbol rename, method rename, or package move where some call sites are still updated by hand.
|
||||||
|
- A new overload added next to an existing one (especially if the old one is `@Deprecated` or kept as a shim).
|
||||||
|
- A type narrowing or broadening (e.g., concrete → interface, primitive → wrapper, scalar → collection) on a public field/parameter/return type.
|
||||||
|
- A new parser, regex, or format reader that replaces a hand-rolled one (especially for paths, keys, identifiers, or wire formats).
|
||||||
|
- Removal of a feature flag, config option, or guard, with surrounding setup/teardown left intact.
|
||||||
|
- A serialization/wire-format version bump, new field added to one side only, or a digest/identity computation that changed shape.
|
||||||
|
- Lambdas added inside hot loops where the surrounding class already exposes a context object.
|
||||||
|
- A call site where the new code is structurally similar to a sibling/copy in another module that was the original target of the fix.
|
||||||
|
- An `equals`/`hashCode`/comparator touched after fields are added or renamed.
|
||||||
|
- Tests, callers, or override hooks that still reference an old method name or old field set.
|
||||||
|
|
||||||
|
## serialization-and-versioning
|
||||||
|
|
||||||
|
Bugs in encoding/decoding symmetry, wire and on-disk formats, version-gated paths, schema evolution, mixed-version compatibility, and switch- or enum-based decode dispatch.
|
||||||
|
|
||||||
|
**Diff signals:**
|
||||||
|
- A `serialize`, `deserialize`, `serializedSize`, `read`, or `write` method, or any pair of these together
|
||||||
|
- A `switch` or `if/else if` ladder that dispatches on a type tag, enum, or kind discriminator
|
||||||
|
- A version comparison: `if (version >= ...)`, `protocolVersion`, `apiVersion`, `formatVersion`, `MessagingService.VERSION_*`, or any version-gated branch
|
||||||
|
- An enum used as a wire-protocol or on-disk discriminator (especially with explicit ordinal/code values, or `values()[i]` decoding)
|
||||||
|
- A `ByteBuffer` operation: `array()`, `arrayOffset()`, `position()`, `slice()`, `duplicate()`, `flip()`, `rewind()`, or any relative read/write
|
||||||
|
- A length prefix (`writeInt`, `writeShort`, `writeUnsignedVInt`) followed/preceded by a payload write/read
|
||||||
|
- A digest or checksum computation over fields, buffers, or sub-records (`MessageDigest.update`, `XXHash`, `CRC32`)
|
||||||
|
- Schema-evolution markers: nullable→non-nullable changes, new fields added to records, `taggedFields`, `nullableVersions`, `flexibleVersions`, `ignorable`
|
||||||
|
- Auto-generated message class changes (Avro, Thrift, Kafka JSON message protocol, Protobuf)
|
||||||
|
- A "compatibility", "legacy", "pre-X.Y", or "old format" code path
|
||||||
|
- Header / context / metadata propagation through wrapper serializers (`Headers headers` arg added or removed)
|
||||||
|
- `getBytes()` / `String.getBytes()` / charset-related calls in a serialization context
|
||||||
|
- Charset, locale, or byte-order assumptions (`UTF_8`, `Locale.getDefault`, `ByteOrder.nativeOrder`)
|
||||||
|
- A deserializer constructor that takes raw bytes plus a type tag, kind flag, or context object
|
||||||
|
|
||||||
|
## state-and-resource-cleanup
|
||||||
|
|
||||||
|
Stale state, leaked references, and orphaned resources that arise when teardown, cancellation, removal, or replay paths fail to fully restore the system to a clean baseline.
|
||||||
|
|
||||||
|
**Diff signals:**
|
||||||
|
- `close()`, `release()`, `unref()`, `decrementReferenceCount()`, `dispose()`, `shutdown()` calls
|
||||||
|
- `try`/`finally`, `try-with-resources`, or `AutoCloseable` introduction or removal
|
||||||
|
- Cancellation, abort, timeout, rejection, or error-path branches that release/return resources
|
||||||
|
- `clear()`, `remove()`, `deregister()`, `unregister()`, `retainAll()`, `evict()` on long-lived collections, registries, caches, or maps
|
||||||
|
- Tombstone, purge, GC, expiry, eviction, retention, or compaction predicates
|
||||||
|
- Counter increment/decrement pairs, latches, semaphores, in-flight counters, pending queues
|
||||||
|
- Replay, restore-from-log, snapshot loading, or restart bootstrap of in-memory state
|
||||||
|
- New background threads, periodic tasks, scheduled futures, or executors with lifecycle management
|
||||||
|
- Reference counting, off-heap buffer management, native handle ownership
|
||||||
|
- Listener/callback registration without matching deregistration in lifecycle hooks
|
||||||
|
- New or renamed lifecycle methods (`onLeave`, `onRemoved`, `onDelete`, `dispose`, `cleanup`)
|
||||||
|
- Iterator/cursor/stream code that may be returned past its enclosing scope
|
||||||
|
|
||||||
|
## validation-and-input-handling
|
||||||
|
|
||||||
|
Bugs where input validators, parsers, format detectors, and pre-conversion guards either miss cases, anchor on the wrong delimiter, fail to account for downstream length/charset/OS limits, or skip pre-use checks (null, -1, empty buffer, length, type) before consuming a value.
|
||||||
|
|
||||||
|
**Diff signals:**
|
||||||
|
- New or modified `validate*`, `check*`, `verify*`, `assert*`, `isValid*`, `parse*`, `tokenize*`, `decode*` methods
|
||||||
|
- Calls to `indexOf`, `lastIndexOf`, `String.split`, `Pattern.compile`, `Matcher`, `String.replace*`, `substring`, `startsWith`/`endsWith`, `getBytes`/`new String`
|
||||||
|
- Construction or modification of regex character classes, anchors, escapes, or quantifiers
|
||||||
|
- File path / URI / address parsing: `getFileName`, `Paths.get`, `URI`, `InetAddress`, `InetSocketAddress`, host:port splitting, IPv4/IPv6 literals
|
||||||
|
- Functions appending suffixes/prefixes (`-`, `_`, UUID, extension, port) before passing names to filesystem, network, or registry calls
|
||||||
|
- Code that consumes an `Optional`, nullable lookup, or map `get()` result without `.isPresent()`/`!= null`/`getOrDefault`
|
||||||
|
- Conditions calling `indexOf(...)` then `substring(...)` without a `-1` guard
|
||||||
|
- Pre-conversion / pre-deserialization filters that drop or pass through unsupported attribute types, sentinels, or empty values
|
||||||
|
- New constraint, length, charset, locale, or boundary checks; or removal of an existing one
|
||||||
|
- Conversion of user-supplied identifiers / option strings / config values to typed objects (enum, UUID, BigDecimal, Date)
|
||||||
|
- Length, capacity, or quantity bounds that interact with appended fixed-size data (suffixes, length prefixes, framing headers)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,218 @@
|
||||||
|
# Category: API Contracts and Completeness
|
||||||
|
|
||||||
|
Bugs where adding a new field, type, event, or capability requires symmetrical updates across multiple sites — overrides, equality/hashCode, builders, dispatch tables, switch cases, register/deregister pairs — and one or more required updates are silently omitted, leaving the system structurally inconsistent.
|
||||||
|
|
||||||
|
## Diff signals (when to load this category)
|
||||||
|
|
||||||
|
Load this category if the patch contains ANY of:
|
||||||
|
- A new field added to a class that has `equals`, `hashCode`, `toString`, `compareTo`, copy, clone, or builder methods
|
||||||
|
- A new enum constant or new subclass added (look for missing cases in `switch` statements or `instanceof` chains elsewhere)
|
||||||
|
- A new method added to an interface or abstract class (look for missing overrides in implementations)
|
||||||
|
- A new event/message/verb type registered (look for unhandled cases in dispatchers)
|
||||||
|
- New `register`, `addListener`, `subscribe`, `addCloseable`, `acquire`, or symmetric pair operations
|
||||||
|
- A new serializer/deserializer overload, signature change, or wire-format field
|
||||||
|
- New constructor parameter, builder method, or option that must be propagated to factory call sites
|
||||||
|
- Changes to a serialization size method, write method, or read method (the three must stay in sync)
|
||||||
|
- A `default` method in an interface that returns a hard-coded constant (subclasses may need to override)
|
||||||
|
- A new metric, management endpoint, sensor, or gauge registration (verify deregistration paths)
|
||||||
|
- A new field in a request/response/copy builder (verify it's copied in `copy`/`from`/`toBuilder` paths)
|
||||||
|
- New configuration property added to a class (verify it's forwarded through every constructor and factory)
|
||||||
|
- A subclass adds fields, resources, or dependencies (verify base-class lifecycle methods are overridden)
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### F-01: New field omitted from equals/hashCode
|
||||||
|
A class adds a new field with semantic significance, but its `equals` and/or `hashCode` are not updated, so two instances differing only in that field compare equal and collide in hash-based collections. Schema-change notifications, cache lookups, and de-dup operations silently fire on the wrong identity.
|
||||||
|
**Look for:** Classes with `equals`/`hashCode` overrides plus a new field added in the diff — verify the field appears in both methods.
|
||||||
|
|
||||||
|
### F-02: equals overridden without hashCode (or vice versa)
|
||||||
|
A class overrides one of `equals`/`hashCode` but not the other (or includes a new field in one but not the other), violating the language contract and causing instances to behave inconsistently in hash maps and sets.
|
||||||
|
**Look for:** Modified `equals` without a matching `hashCode` change (or vice versa); also look for `hashCode` calling `Objects.hash()` on byte arrays where `Arrays.hashCode()` is required.
|
||||||
|
|
||||||
|
### F-03: hashCode includes the same field twice
|
||||||
|
A `hashCode` repeats one field in place of another, so objects differing only on the omitted field collide and are treated equal in hash collections.
|
||||||
|
**Look for:** `Objects.hash(...)` calls in the diff where two arguments look like they were copied; check parity with the `equals` field list.
|
||||||
|
|
||||||
|
### F-04: equals delegates to typed comparator without instanceof guard
|
||||||
|
An `equals(Object)` implementation casts or delegates without first checking the runtime type of the operand, throwing a `ClassCastException` or `RuntimeException` for cross-type comparisons instead of returning `false`.
|
||||||
|
**Look for:** `equals` methods that call `compareTo` or a typed helper before any `instanceof` test on the argument.
|
||||||
|
|
||||||
|
### F-05: equals compares this to itself or to wrong field
|
||||||
|
An `equals` bridge method casts the argument but mistakenly passes `this` to a helper, or compares one operand's field to itself rather than the other operand's corresponding field. Comparison is constant (always true or always false) regardless of inputs.
|
||||||
|
**Look for:** `equals` bodies where every field comparison reads from the same operand, or methods that pass `this` rather than the cast parameter.
|
||||||
|
|
||||||
|
### F-06: New enum constant without matching switch case
|
||||||
|
A new enum constant or message-type constant is added to the type but not to one or more switch statements that map enum values to behavior. The unhandled case falls through to `default`, throws `UnsupportedOperationException`, or silently no-ops.
|
||||||
|
**Look for:** New enum entry in the diff — grep for `switch` statements over that enum type elsewhere in the codebase and verify each handles the new case.
|
||||||
|
|
||||||
|
### F-07: Builder copy/from method omits a new field
|
||||||
|
A request, mutation, or configuration builder gains a new field, but the `copy`, `toBuilder`, `from`, or `clone` constructor that propagates source-to-destination fields is not updated, so caller-specified values are silently discarded.
|
||||||
|
**Look for:** Diff adds a new field to a class with a copy-constructor or `toBuilder` method — check that the new field is copied in every such method.
|
||||||
|
|
||||||
|
### F-08: Builder/factory accepts parameter but never applies it
|
||||||
|
A constructor or builder method accepts an option (encryption settings, throttle rate, headers, callbacks, frame size) but never assigns it to the constructed object or forwards it to the underlying library, silently discarding the caller's value.
|
||||||
|
**Look for:** Constructor parameters with no corresponding field assignment, or setters that never store their input — search for the parameter name on the right-hand side of an assignment.
|
||||||
|
|
||||||
|
### F-09: Setter writes wrong field; getter never wired
|
||||||
|
A field is exposed via a setter but the assignment targets the wrong field, or a getter is hardcoded to return `null` / a constant rather than reading the backing field, silently dropping the stored data.
|
||||||
|
**Look for:** Setters whose assignment target name does not match the parameter name, and accessors with no `return field;` body.
|
||||||
|
|
||||||
|
### F-10: New copy/clone-introduced field shares mutable reference
|
||||||
|
A copy method copies a field by reference (rather than deep-copying a mutable sub-collection), so mutations through the copy propagate back to the original and corrupt subsequent uses.
|
||||||
|
**Look for:** New `copy`/`clone`/copy-constructor methods — verify mutable collections, maps, and arrays are defensively copied.
|
||||||
|
|
||||||
|
### F-11: Subclass overrides one method but not its sibling
|
||||||
|
A subclass overrides a data-access method (e.g., `iterator`) but inherits a paired method (`size`, `count`, `isEmpty`, `toString`) from the superclass, returning answers derived from a different (often empty or pre-mutation) state.
|
||||||
|
**Look for:** Subclasses that override one of a logically paired set of methods — check that all related methods are overridden consistently.
|
||||||
|
|
||||||
|
### F-12: Subclass adds resources/fields but no override of lifecycle method
|
||||||
|
A subclass adds new closeable resources, fields, or dependencies, but does not override `close`, `abort`, `interrupt`, `stop`, `enable/disable`, or memory-measurement methods inherited from the parent. The base implementation runs and silently misses the subclass's state, leaking resources or reporting wrong sizes.
|
||||||
|
**Look for:** Subclasses with new `Closeable`/`AutoCloseable` fields or background threads — verify they override all relevant lifecycle methods.
|
||||||
|
|
||||||
|
### F-13: Override silently dead due to renamed/missing @Override
|
||||||
|
A subclass override loses its annotation, gets misspelled, or has a drifted signature relative to the parent; without `@Override` the compiler accepts it, but the inherited base method runs at runtime and the override is dead code.
|
||||||
|
**Look for:** New or modified methods in subclasses without `@Override` annotation — verify the signature exactly matches the intended parent method.
|
||||||
|
|
||||||
|
### F-14: Default interface method returns hard-coded constant
|
||||||
|
A `default` method on an interface returns a hard-coded sentinel ("never matches", `Optional.empty()`, `false`, `null`) instead of inspecting state. Mutable implementations that legitimately need a real answer inherit the constant unless they remember to override.
|
||||||
|
**Look for:** Interface `default` methods returning literals — every concrete implementer must override unless the constant is truly correct.
|
||||||
|
|
||||||
|
### F-15: Decorator/wrapper omits override for new interface method
|
||||||
|
When an interface gains a new method, decorator and forwarding subclasses that wrap a delegate must override the new method to forward to the delegate; absent overrides reach a `default` that throws or returns wrong values.
|
||||||
|
**Look for:** Interface gains a new method — find every wrapper/decorator class and verify it overrides the new method to delegate.
|
||||||
|
|
||||||
|
### F-16: Subclass missing factory-style return type override
|
||||||
|
A factory-style method like `create`, `with`, or `copy` declared on a base class returns a base-class instance; subclasses that should preserve their identity must override to return the subclass type, otherwise the base method silently returns the wrong concrete type.
|
||||||
|
**Look for:** Subclasses inheriting `Self`-returning methods from a base — check they override to preserve their type.
|
||||||
|
|
||||||
|
### F-17: Register without deregister — leak / stale subscription
|
||||||
|
A constructor registers an instance with a metric registry, event bus, schema listener, management endpoint, or observer, but the corresponding `close`/`stop` method is missing the matching deregister call. Entries accumulate across lifecycle cycles.
|
||||||
|
**Look for:** New `register*`, `addListener`, `subscribe`, `recordSensor`, `addCloseable` calls — verify the matching `deregister*`/`removeListener`/`unsubscribe` exists in `close`/`stop`.
|
||||||
|
|
||||||
|
### F-18: Register/deregister name mismatch
|
||||||
|
A metric, sensor, or management endpoint is registered under one name but the deregister call uses a different name (typo, stale rename, missing namespace component); the deregister silently does nothing and the registration leaks indefinitely.
|
||||||
|
**Look for:** Compare the string/key used in `register*` vs `deregister*`/`unregister*` calls — they must match exactly.
|
||||||
|
|
||||||
|
### F-19: New event/verb/handler not added to dispatch table
|
||||||
|
A new RPC verb, message type, or event family is defined but the dispatcher's routing map, switch, or interceptor chain is not updated, so the message is silently dropped, asserts at runtime, or maps to the wrong handler.
|
||||||
|
**Look for:** New verb/event constants in diff — grep for the central dispatcher and verify the new constant is added.
|
||||||
|
|
||||||
|
### F-20: New serializer registered with no deserializer counterpart
|
||||||
|
A new serializer or wire-format writer is added but the corresponding read path or version handler is missing, so the new field round-trips correctly only on the sender; the receiver mis-decodes.
|
||||||
|
**Look for:** Diff modifies a `write*`/`serialize` method without touching the matching `read*`/`deserialize` method — the two must change together.
|
||||||
|
|
||||||
|
### F-21: Serializer write/size/read methods drift
|
||||||
|
A field is added or reordered in the `serialize` method but the parallel `serializedSize` or `deserialize` method is not updated. Buffer allocation is wrong, or fields are read from the wrong byte offsets, corrupting all subsequent fields.
|
||||||
|
**Look for:** Any change to a serialization method — locate the matching size and deserialization methods and verify field order, presence, and width all match.
|
||||||
|
|
||||||
|
### F-22: Metric / management endpoint registered only under new name after rename
|
||||||
|
A metric or management endpoint is renamed but only the new name is registered. Existing monitoring tools, dashboards, and alerting that reference the old name silently break.
|
||||||
|
**Look for:** Metric/endpoint rename — preserve a deprecated alias registration unless the diff explicitly removes the old name.
|
||||||
|
|
||||||
|
### F-23: Hardcoded enumeration not updated when new entries added
|
||||||
|
An iteration over a hard-coded list (function registrations, schema objects, system tables, well-known verbs) misses dynamically-added or newly-introduced entries, so any operation derived from that enumeration silently omits the new entries.
|
||||||
|
**Look for:** Static arrays/lists of enumeration entries — verify the new entry is appended.
|
||||||
|
|
||||||
|
### F-24: Type-dispatch method missing branch for new variant
|
||||||
|
An `if/else if` chain or `instanceof` cascade that classifies polymorphic inputs is not extended for a newly added subtype; the new variant falls through to a wrong default branch and is decoded, serialized, or routed incorrectly.
|
||||||
|
**Look for:** `instanceof` cascades or type-tag dispatchers in the diff or referenced from changed types — verify all subtypes are enumerated.
|
||||||
|
|
||||||
|
### F-25: Switch statement falls through to throw on new enum value
|
||||||
|
A `switch` over an enum lacks a case for a newer constant; the `default` branch throws `UnsupportedOperationException` or `AssertionError` rather than returning a meaningful result, crashing on the first invocation that exercises the new value.
|
||||||
|
**Look for:** `switch` statements over enums where the `default` throws — verify exhaustiveness against the current enum definition.
|
||||||
|
|
||||||
|
### F-26: Validation present in one path, missing in parallel path
|
||||||
|
A guard, validation, or normalization step lives in one entry point (e.g., the local-apply path) but is omitted from a parallel path (the remote-announce path), allowing invalid or non-canonical inputs to bypass the check.
|
||||||
|
**Look for:** Two parallel call sites that should apply the same check — search for the validation function and verify both invoke it.
|
||||||
|
|
||||||
|
### F-27: New field omitted from human/wire serialization symmetry
|
||||||
|
A field exists in the human-readable / textual output path but not in the structured/wire output (or vice versa); machine consumers see no field while operators do, or vice versa.
|
||||||
|
**Look for:** Both `toString`/`toJson`/`toDisplayString` and binary serialize methods on the same class — verify all fields appear in both.
|
||||||
|
|
||||||
|
### F-28: Constructor accepts but ignores parameter (silent default)
|
||||||
|
A class's constructor accepts a parameter (credentials, callback, configuration object), but never stores or forwards it; the object silently runs with the default value of that parameter.
|
||||||
|
**Look for:** Constructor parameters that don't appear on the right-hand side of any field assignment in the constructor body.
|
||||||
|
|
||||||
|
### F-29: Method overload with wider type masks intended narrower call
|
||||||
|
A method gains a new overload (e.g., generic vs. concrete, Object vs. typed); existing call sites silently dispatch to the new (or old) overload due to overload resolution rules, applying wrong semantics without any compile error.
|
||||||
|
**Look for:** New overloads of an existing method name — audit all call sites to confirm they bind to the intended overload.
|
||||||
|
|
||||||
|
### F-30: Async method's contract mismatch — exception thrown vs failed future
|
||||||
|
A method declared to return a future or `CompletableFuture` throws synchronously instead of returning a failed future, or documents one error mode but emits another. Callers using future-chained error handling miss the failure entirely.
|
||||||
|
**Look for:** `throw` statements inside methods returning a future-like type — failures should generally be wrapped in a failed future.
|
||||||
|
|
||||||
|
### F-31: Setter mutates intended-immutable field, bypassing rebuild
|
||||||
|
A class designed for immutability gets a setter that mutates a field in place, bypassing the version-bump or copy-on-write mechanism that would propagate the change to derived state.
|
||||||
|
**Look for:** New setters on classes that have a `withFoo`/`copy` builder pattern — the setter likely shouldn't exist.
|
||||||
|
|
||||||
|
### F-32: Snapshot-friendly accessor silently uses live-read
|
||||||
|
A method documented or contractually expected to return a snapshot (immutable view) reads from a live mutable reference, so a concurrent mutation between the read and downstream use produces inconsistent behavior.
|
||||||
|
**Look for:** Methods named `snapshot`, `view`, or `current` that don't actually defensively copy or wrap their result.
|
||||||
|
|
||||||
|
### F-33: Live mutable view returned where defensive copy expected
|
||||||
|
A getter returns a live internal collection or mutable map (rather than an unmodifiable view or copy), allowing external callers to silently corrupt internal state.
|
||||||
|
**Look for:** Getters that `return this.someCollection` rather than `Collections.unmodifiableSet(...)` or a copy.
|
||||||
|
|
||||||
|
### F-34: Visitor/dispatch missing case for new type variant
|
||||||
|
A concrete visitor subclass omits an override for a newly added type variant; the inherited base implementation returns null or a wrong default, producing a `NullPointerException` at the caller.
|
||||||
|
**Look for:** New visitor types added — check every subclass of the visitor base for a corresponding new override.
|
||||||
|
|
||||||
|
### F-35: Hand-rolled cache key omits a value-determining field
|
||||||
|
A cache key's equality and hash methods omit one or more fields that influence the cached value, so distinct logical configurations collide on a single entry and reuse the wrong value.
|
||||||
|
**Look for:** Cache key classes with new fields added — verify they participate in the key's `equals` and `hashCode`.
|
||||||
|
|
||||||
|
### F-36: New configuration property accepted at one layer, dropped before use
|
||||||
|
A config option is parsed and validated at the public entry point but never threaded to the constructor or builder that actually applies it; the setting is silently ignored and the default takes effect.
|
||||||
|
**Look for:** New config keys appearing in parsers — trace through the constructor chain to the place that actually configures the underlying behavior.
|
||||||
|
|
||||||
|
### F-37: Validation loosened in entry-point but enforced in inner factory
|
||||||
|
Validation exists at the high-level public entry point but lower-level factories or alternate construction paths bypass it, allowing invalid inputs to be persisted.
|
||||||
|
**Look for:** Multiple constructor or factory methods on the same class — verify validation runs uniformly across all of them.
|
||||||
|
|
||||||
|
### F-38: Constraints assumed by callsite contract violated by new caller
|
||||||
|
A new call site uses a method in a way that violates an undocumented or weakly-documented contract (e.g., passing an unmodifiable collection where mutation is required, calling close from outside the lifecycle owner, calling outside the synchronized context).
|
||||||
|
**Look for:** New call sites of existing helpers that pass `Collections.emptyList()`, `Collections.unmodifiableMap()`, or unusual sentinel arguments.
|
||||||
|
|
||||||
|
### F-39: Field added but omitted from serialization helper / persistence query
|
||||||
|
A new field is added to a record, but the serialization-to-persistence helper, DDL emitter, or schema-mutation query is not updated, so the field is silently never persisted and reverts to its default after restart.
|
||||||
|
**Look for:** New fields on records mapped to schema rows — verify both the persist path and the load path read/write the new column.
|
||||||
|
|
||||||
|
### F-40: Subclass field-shadowing instead of method override
|
||||||
|
A subclass redeclares a field with the same name as the parent rather than overriding methods that read/write that field; queries on the parent's field always return stale values.
|
||||||
|
**Look for:** Subclasses that re-declare a field with a name already present on the parent — almost always a bug.
|
||||||
|
|
||||||
|
### F-41: Call-site forgotten after rename / signature change
|
||||||
|
A method is renamed or its return type narrowed; some call sites are updated but stale ones still reference the old name (resolved against an unrelated overload that compiles) or wrap the new return type in an obsolete conversion. Wrong logic runs silently.
|
||||||
|
**Look for:** Any rename or signature change — search for every call site of both the old and new names in the entire codebase.
|
||||||
|
|
||||||
|
### F-42: Dispatch / subtype check enumerates only first variant
|
||||||
|
A predicate, switch, or `instanceof` chain checks one subtype but does not handle a newly-introduced sibling subtype; the missing case falls through to a generic default that has wrong semantics for the unhandled type.
|
||||||
|
**Look for:** `instanceof X` checks where the diff added a sibling class `Y` — verify `Y` is handled.
|
||||||
|
|
||||||
|
### F-43: Bridge/no-op stub introduced as a default still allows old API path
|
||||||
|
A `default` method is introduced as a bridge to a renamed API; existing implementers can keep using the deprecated method silently. Removing the default without making the new method abstract leaves wrong runtime behavior unguarded.
|
||||||
|
**Look for:** New `default` methods on interfaces that delegate to deprecated alternates — these silently mask incomplete migrations.
|
||||||
|
|
||||||
|
### F-44: Wrapper/serde missing forwarding of newly added parameter
|
||||||
|
Wrapper serializers, deserializers, or interceptors that delegate to inner instances are not updated to forward a newly-added parameter (e.g., `Headers`, `topic`, `context`); the inner call receives a default and silently drops the data.
|
||||||
|
**Look for:** Wrapper classes with delegate calls — when the inner interface gains a parameter, every wrapper must forward it.
|
||||||
|
|
||||||
|
### F-45: Schema gains field but parallel descriptor arrays not extended
|
||||||
|
A structured record adds a new field but parallel type-descriptor or name-descriptor arrays used in a generic deserializer are not extended; reads pick up the wrong array slot and cast to the wrong type.
|
||||||
|
**Look for:** Parallel arrays indexed by field position — verify they're updated together.
|
||||||
|
|
||||||
|
### F-46: New version constant or compatibility check incomplete
|
||||||
|
A new protocol version, file format, or schema version is added but compatibility checks (gates, migrations, fall-back paths) are only partially updated; nodes mixing old and new versions misbehave silently.
|
||||||
|
**Look for:** New version constants — search for all `if (version >= ...)` checks and verify completeness.
|
||||||
|
|
||||||
|
### F-47: serializedSize accumulator loses an added field's contribution
|
||||||
|
A field is added (or its byte width changes) in the write path, but the `serializedSize` calculator drops or ignores the new contribution; output framing is off, allocation is wrong, and downstream readers misalign.
|
||||||
|
**Look for:** Any modification to a write method — verify the accompanying `serializedSize` totals the same fields with the same widths.
|
||||||
|
|
||||||
|
### F-48: Copy-paste callsite carries argument from sibling method
|
||||||
|
A new method copy-pasted from a sibling retains an argument, name, or constant from the original that no longer applies in the new scope; the wrong context value is forwarded silently.
|
||||||
|
**Look for:** Newly added methods that look like copies of nearby siblings — verify every reference inside has been updated for the new context.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
These patterns share a structural shape: a system has multiple sites that must change in lockstep when a new field, type, event, or capability is introduced. A diff that touches one site without touching the symmetric site introduces a silent incorrectness — the code compiles, runs, and even passes most tests, but produces wrong results, leaks resources, or skips work because the contract between the sites is no longer maintained. The diff signals above are the strongest indicator that this category applies; once it does, the review checklist is dominated by "find every other site that should also change."
|
||||||
|
|
@ -0,0 +1,208 @@
|
||||||
|
# Category: Boundaries and Numbers
|
||||||
|
|
||||||
|
Bugs involving numeric arithmetic, integer overflow, off-by-one errors, unit/dimensional mismatches, buffer-bound violations, sentinel/special-value confusion, and floating-point pitfalls — wherever a number, index, or quantity is computed, compared, or interpreted incorrectly.
|
||||||
|
|
||||||
|
## Diff signals (when to load this category)
|
||||||
|
|
||||||
|
Load this category if the patch contains ANY of:
|
||||||
|
- Arithmetic operators (`+`, `-`, `*`, `/`, `%`, `<<`, `>>`) on `int`/`long`/`short` fields, especially before a widening cast
|
||||||
|
- Index expressions (`a[i]`, `list.get(i)`), `subList`, `slice`, or pre/post-increment in a loop body
|
||||||
|
- Comparison operators (`<`, `<=`, `>`, `>=`, `==`) used as loop bounds, threshold checks, or boundary conditions
|
||||||
|
- Casts between numeric types (`(int)`, `(short)`, `(long)`, `toInt`, `toLong`)
|
||||||
|
- Unit-bearing names or constants (`ms`, `Ms`, `Nanos`, `Seconds`, `MB`, `KB`, `Bytes`, `MiB`, `MILLIS_PER_*`, `BITS_PER_*`)
|
||||||
|
- `TimeUnit`, `Duration`, `Instant`, `currentTimeMillis()`, `nanoTime()`
|
||||||
|
- ByteBuffer operations: `position()`, `limit()`, `flip()`, `rewind()`, `slice()`, `duplicate()`, `arrayOffset()`, `array()`, `getInt()`, `putShort()`
|
||||||
|
- Buffer/array allocation sized from an external length or count (`new byte[len]`, `ByteBuffer.allocate(...)`)
|
||||||
|
- Sentinel constants like `-1`, `Long.MAX_VALUE`, `Integer.MIN_VALUE`, `MAX_VALUE`, `Short.MAX_VALUE`
|
||||||
|
- Division operations that may produce zero from integer truncation or divide by an unguarded denominator
|
||||||
|
- Floating-point accumulators, percentage/ratio calculations, `Math.round`, `setScale`, `BigDecimal.toPlainString`
|
||||||
|
- Histogram/percentile/bucket/level/tier code with bucket array indexing
|
||||||
|
- TTL, expiry, deadline, timeout, backoff, throttle, or rate-limit calculations
|
||||||
|
- Length-prefixed encoding/decoding with explicit width (2-byte, 4-byte, varint) or `serializedSize()` methods
|
||||||
|
- New configuration fields with numeric defaults or unit suffixes
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### F-01: Int overflow on multiply-then-widen
|
||||||
|
A computation multiplies two `int` operands (or shifts an int) and only widens the product to `long` after the multiplication, so values above ~2 GB silently wrap to a negative number that is then stored as a size, offset, or threshold.
|
||||||
|
**Look for:** `long x = a * b` where `a`/`b` are `int`; `MB << 20`, `kb * 1024`, `count * sizeOf` patterns missing an explicit `(long)` cast on either operand.
|
||||||
|
|
||||||
|
### F-02: Per-level/per-bucket accumulator wraps int
|
||||||
|
Per-shard, per-level, or per-bucket counts are summed into a 32-bit accumulator that exceeds `Integer.MAX_VALUE` for large datasets, wrapping to a negative value reported as pending tasks, byte position, or row count.
|
||||||
|
**Look for:** `int total +=` inside a loop iterating shards/levels/segments; a status/aggregation field that accumulates across many partitions.
|
||||||
|
|
||||||
|
### F-03: Off-by-one loop counter as index
|
||||||
|
A loop counter is pre-incremented at the start of the body and then used as an array index in the same body, so every access is one position ahead of the intended element.
|
||||||
|
**Look for:** `i++` near the top of a loop body followed by `arr[i]` later in the same iteration; or `++i` in the middle of an expression that also indexes an array.
|
||||||
|
|
||||||
|
### F-04: Strict vs non-strict boundary comparator
|
||||||
|
A boundary or threshold check uses `<` where `<=` is required (or vice versa), so the element exactly at the boundary is included/excluded incorrectly: duplicate emissions on page boundaries, infinite loops at empty quotients, premature EOF, last-bucket dropped, retained-when-equal segments not recycled.
|
||||||
|
**Look for:** Boundary helpers that compare `pos > limit`, `count >= max`, `current < end` near pagination, slice-bound, segment recycle, or merge eligibility code.
|
||||||
|
|
||||||
|
### F-05: BufferOverflow on exact-full buffer write
|
||||||
|
A fixed-size record (end-of-segment marker, header, footer) is written without first checking that enough space remains, so when the buffer is exactly full at the start of the write the operation throws `BufferOverflowException`.
|
||||||
|
**Look for:** A series of `put*` calls into a `ByteBuffer` without a preceding `remaining() >= N` check, especially at segment rotation/finalization boundaries.
|
||||||
|
|
||||||
|
### F-06: Floating accumulator drift past domain
|
||||||
|
A floating-point accumulator is stepped by a fixed delta inside a count-bounded sampling loop; accumulated rounding error causes the loop to reach a value past the valid domain on the final iteration, passing an out-of-domain argument (e.g., `log(0)`, `asin(1.0+eps)`) to a sampled function.
|
||||||
|
**Look for:** `for (int i=0; i<n; i++) { x += step; sample(x); }` patterns where `n*step` should equal a domain boundary.
|
||||||
|
|
||||||
|
### F-07: Loop runs one iteration too many
|
||||||
|
A numerical sampling or chunking loop terminates one iteration past intent, feeding the boundary value to a function that may be undefined, infinite, or duplicate-producing at that boundary.
|
||||||
|
**Look for:** `for (int i=0; i<=n; i++)` where the loop should be `<n`; cumulative-division loops emitting one extra partition.
|
||||||
|
|
||||||
|
### F-08: Rate metric sampling-window vs reporting-unit mismatch
|
||||||
|
A rate or percentile metric is sampled over a window (e.g., 60 s) but reported in a different unit (e.g., 1 hour), so the displayed value is a fixed multiple too high or low.
|
||||||
|
**Look for:** Sensor/timer registration where the configured window duration and the reported "per X" unit are independent constants; metrics named `*_per_hour` derived from a `60s` window.
|
||||||
|
|
||||||
|
### F-09: Dimensionless parameter receives unit-bearing value
|
||||||
|
A parameter declared as a dimensionless fraction or count receives a unit-bearing value (e.g., a millisecond integer like 300000 used as a backoff multiplier), producing wildly wrong results.
|
||||||
|
**Look for:** Method calls passing a positional numeric argument where the parameter name suggests a fraction (`*Factor`, `*Ratio`, `*Probability`) but the call site has a long ms/ns value.
|
||||||
|
|
||||||
|
### F-10: Time-unit mismatch in compare or convert
|
||||||
|
A timestamp expressed in milliseconds is compared against, added to, or subtracted from one expressed in microseconds (or seconds, or nanoseconds), producing comparisons off by 1000× or 1,000,000×.
|
||||||
|
**Look for:** Two timestamp variables compared without unit normalization; constants like `1000`, `1_000_000`, `1_000_000_000` near `System.currentTimeMillis` or `System.nanoTime`; `TimeUnit.NANOSECONDS.toX(...)` paired with a value already in another unit.
|
||||||
|
|
||||||
|
### F-11: Wrong unit factor in conversion
|
||||||
|
A unit-conversion expression multiplies by a constant that converts in the opposite direction (e.g., bits-per-byte instead of bytes-per-bit), producing values 8×, 1024×, or 60× off.
|
||||||
|
**Look for:** Hardcoded conversion constants (`8`, `1024`, `1024*1024`, `60`, `3600`) where the symbol/name suggests one direction but the operation goes the other way.
|
||||||
|
|
||||||
|
### F-12: Integer division silently yields zero
|
||||||
|
An expression divides a small numerator by a larger denominator using integer arithmetic, truncating to zero and then either looping forever, sizing a buffer to zero, treating the quotient as "unlimited", or producing zero progress.
|
||||||
|
**Look for:** `(small int) / (large int)` patterns at the head of loops, before `>>`, before `Math.log`, or before being applied as a stride or per-element budget; `bytesBudget / count` patterns.
|
||||||
|
|
||||||
|
### F-13: Divisor not zero-checked
|
||||||
|
A counter or denominator is used in a divide/modulo without a zero-check, throwing `ArithmeticException` on first invocation, on empty input, or after all rows are tombstones.
|
||||||
|
**Look for:** `x / y` or `x % y` where `y` is read from a collection size, a counter, or a method that can return 0; especially in retry-formula code, hash-bucketing, or page-size computation.
|
||||||
|
|
||||||
|
### F-14: Double-applied conversion or formula
|
||||||
|
A helper already applies a formula (quorum, scaling, conversion) and the caller applies the same formula again to the helper's result, producing values that are too small, too large, or doubled.
|
||||||
|
**Look for:** Two consecutive applications of the same constant (`*1024`, `(n/2)+1`); chained calls through wrapper methods that each scale the same value.
|
||||||
|
|
||||||
|
### F-15: Premature truncation before scaling
|
||||||
|
An integer divide is evaluated before multiplication by a scale factor, truncating the result to zero or losing precision.
|
||||||
|
**Look for:** `a/b * c` patterns where `c` is the scale; missing parentheses around `(a*c)/b`.
|
||||||
|
|
||||||
|
### F-16: Cast to narrower type loses high bits
|
||||||
|
A `long` value is cast to `int` (or `int` to `short`) before further use as an index, length, or position, silently producing wrong or negative values for large inputs.
|
||||||
|
**Look for:** `(int)(long)`, `(int) channel.size()`, `(int) (pos / chunkSize)`, especially in code that handles files, offsets, or capacities larger than 2 GB.
|
||||||
|
|
||||||
|
### F-17: Type narrower than actual domain
|
||||||
|
A field, parameter, or counter is typed narrower (`short`, `int`) than the domain of values it can carry; under sustained use a counter overflows the type, producing negative values or wrap-around behavior.
|
||||||
|
**Look for:** `short` fields incremented per timeout/event; `int` byte-position accumulators in code that handles multi-GB data; numeric overload selection that picks a narrower variant.
|
||||||
|
|
||||||
|
### F-18: Byte-budget vs item-count mismatch
|
||||||
|
A check enforces an item-count limit on data that should be measured in bytes (or vice versa), so large items exhaust memory before the count limit fires, or the reservoir contains far fewer than expected.
|
||||||
|
**Look for:** `count > bytesLimit` or `bytes > countLimit` style guards; histogram/buffer pool sizing where the limit unit mismatches the loop variable's unit.
|
||||||
|
|
||||||
|
### F-19: Fixed-width length prefix overflow
|
||||||
|
A length-prefixed encoding uses a 2-byte (or fixed-width) field but the actual payload exceeds the prefix's maximum (e.g., > 65535 bytes), causing silent truncation, framing corruption, or runtime exception.
|
||||||
|
**Look for:** `writeShort(payload.length)` style calls; `Short.MAX_VALUE` or `0xFFFF` next to length validation; user-controlled string concatenation that feeds into a length-prefixed serializer.
|
||||||
|
|
||||||
|
### F-20: Asymmetric size-read / size-write width
|
||||||
|
A serializer writes a length prefix at one width (e.g., 4 bytes) and a deserializer reads it at a different width (e.g., 2 bytes) — or signed vs unsigned varint variants — corrupting all subsequent bytes in the stream.
|
||||||
|
**Look for:** Paired `writeInt`/`readShort` (or `varInt`/`varSignedInt`); mismatched constants in `serializedSize()` and the actual write loop.
|
||||||
|
|
||||||
|
### F-21: Histogram bucket-overflow / bucket-too-small
|
||||||
|
A percentile or histogram computation does not handle observations beyond the largest bucket, throwing `ArrayIndexOutOfBoundsException` when all observations exceed the ceiling, or returning the lower bound of the highest bucket as the maximum.
|
||||||
|
**Look for:** Histogram percentile/max methods without an "overflow bucket" branch; merging two histograms whose lengths differ; fixed `BUCKET_COUNT` constants.
|
||||||
|
|
||||||
|
### F-22: Sentinel sign/value collides with valid domain
|
||||||
|
A negative sentinel like `-1` (meaning "uninitialized" or "no timestamp") compares as a normal integer; valid values can equal the sentinel, or the sentinel's negative value participates in arithmetic and breaks comparisons / arithmetic / array indexing.
|
||||||
|
**Look for:** `Long.MIN_VALUE`/`-1` used as an "absent" marker compared with `<` or `>`; sentinel passed to `nextInt(n)` or used as an exponent / array index.
|
||||||
|
|
||||||
|
### F-23: NaN guard missing in min/max shortcut
|
||||||
|
A min/max bounds shortcut reads precomputed column bounds without a NaN guard and returns a wrong answer because NaN values are unordered and excluded from stored bounds.
|
||||||
|
**Look for:** Range-pruning helpers reading `min`/`max` columns of floating-point types; `Math.min`/`Math.max` over `double` inputs without `isNaN`/`isFinite` checks.
|
||||||
|
|
||||||
|
### F-24: Modulo can produce negative result
|
||||||
|
A modulo expression on a hash code or on a value that may be negative produces a negative result that is then used directly as an array index, causing `ArrayIndexOutOfBoundsException`.
|
||||||
|
**Look for:** `hash % capacity` or `value % bucketCount` used as an index without `Math.floorMod` or `& (cap-1)` (when capacity is power-of-two).
|
||||||
|
|
||||||
|
### F-25: Sign-extension on byte read
|
||||||
|
Reading a signed `byte` and treating it as an unsigned value (without `& 0xFF`) flips the high-bit-set bytes to negative, causing length prefixes, version codes, or flags to be misinterpreted.
|
||||||
|
**Look for:** `int b = stream.read()` followed by arithmetic without masking; `getLong()`/`getInt()` that should be byte-swap-aware; `ByteBuffer.getLong()` without setting byte order explicitly.
|
||||||
|
|
||||||
|
### F-26: Comparator subtraction overflow
|
||||||
|
A comparator returns `(long)a - (long)b` cast to `int`, or returns `a - b` for `int`s where the difference may exceed `Integer.MAX_VALUE`, causing wrong ordering for far-apart values.
|
||||||
|
**Look for:** `compare()` methods that return `(int) (longA - longB)` or `valA - valB`; replace with `Long.compare`/`Integer.compare`.
|
||||||
|
|
||||||
|
### F-27: Range with equal endpoints treated as full range
|
||||||
|
A range constructor accepts equal left and right endpoints and treats the result as the entire ring/space rather than a degenerate single-point range, causing `intersects` to always succeed and queries to fan out cluster-wide.
|
||||||
|
**Look for:** `new Range(min, max)` from a dynamically computed min/max with no `min.equals(max)` short-circuit; wrap-around range logic.
|
||||||
|
|
||||||
|
### F-28: Inverted operands or direction flag
|
||||||
|
A subtraction (or unit conversion) is written with operands in the wrong order — `current - deadline` instead of `deadline - current`, or `epoch - now` instead of `now - epoch` — producing negative durations, immediate timeouts, infinite loops, or absurd elapsed-time logs.
|
||||||
|
**Look for:** Elapsed-time, retry-count, and TTL formulas with subtraction; sleep/timeout computations whose result is logged as negative.
|
||||||
|
|
||||||
|
### F-29: Deadline overflow when adding duration to clock
|
||||||
|
A deadline is computed by adding a duration to a monotonic clock value near `Long.MAX_VALUE`, causing the addition to overflow and produce a deadline in the past so the loop exits immediately.
|
||||||
|
**Look for:** `nanoTime() + timeoutNanos` or `currentTimeMillis() + duration` without saturating-add; `Math.addExact` rare; `if (deadline < now)` paths.
|
||||||
|
|
||||||
|
### F-30: Empty/zero passed to API that rejects empty/zero
|
||||||
|
A library API rejects a zero count, empty collection, or zero-length argument with `IllegalArgumentException` (e.g., `RateLimiter.acquire(0)`, `Random.nextInt(0)`, `BigDecimal` scale ops), and the caller does not guard against the degenerate case.
|
||||||
|
**Look for:** Pre-existing collection size or counter passed unchecked into `nextInt`, `acquire`, `setScale`, or other domain-restricted APIs.
|
||||||
|
|
||||||
|
### F-31: BigDecimal toPlainString blows up memory on large exponent
|
||||||
|
Calling `toPlainString()` on a `BigDecimal` with a large negative scale expands the number to a string of billions of characters, causing OOM.
|
||||||
|
**Look for:** `BigDecimal.toPlainString()` on user-supplied input; large negative exponent on serialization; `setScale(0, ...)` on values exceeding `long`.
|
||||||
|
|
||||||
|
### F-32: Array-offset / position not added in slice arithmetic
|
||||||
|
A buffer accessor computes an absolute position from a relative offset without adding the buffer's `position()` (or `arrayOffset()`), returning garbage or out-of-bounds values for any sliced or advanced buffer.
|
||||||
|
**Look for:** `buffer.array()` without adding `arrayOffset`; `getInt(i)` or `get(i)` with hardcoded `0` on a buffer whose position may be non-zero; `digest.update(buf.array(), 0, buf.remaining())`.
|
||||||
|
|
||||||
|
### F-33: Position not duplicated before relative read
|
||||||
|
A relative `getInt`/`get` is called on a shared buffer without first calling `duplicate()` (or `slice()`), advancing the original's position as a side effect and corrupting all subsequent reads from the same buffer.
|
||||||
|
**Look for:** `buf.getX()` without explicit `buf.duplicate().getX()` on shared/cached buffers; comparator methods that call relative-get on instance fields.
|
||||||
|
|
||||||
|
### F-34: rewind/flip confusion on output buffer
|
||||||
|
A buffer is `rewind()`-ed instead of `flip()`-ped (or neither) before being returned for reading, exposing uninitialized bytes beyond the written data or producing zero readable bytes.
|
||||||
|
**Look for:** Methods returning a `ByteBuffer` filled by writes without a `flip()` immediately before return; mark/reset paths that restore some fields but not the dirty flag.
|
||||||
|
|
||||||
|
### F-35: Backoff/retry has no ceiling or no floor
|
||||||
|
A backoff formula's growth term has no upper bound (or no lower bound), so at high attempt counts it can exceed sane limits, hit a `Math.min` of zero, or invert a clamp; alternatively the first delay is inflated by using `base^attempts` with a 1-based counter so attempt 1 already multiplies by `base`.
|
||||||
|
**Look for:** Exponential-backoff helpers without `Math.min(cap, ...)`; `Math.pow(base, attempt)` with `attempt` not zeroed; backoff multiplier bounded by an unrelated session timeout.
|
||||||
|
|
||||||
|
### F-36: Configuration value truncates before write to wider field
|
||||||
|
A configuration value held as `int` is multiplied by a unit factor in `int` arithmetic and then assigned to a `long` field, truncating before the widening assignment.
|
||||||
|
**Look for:** `long fieldBytes = configMb * 1024 * 1024` where `configMb` is `int`; replace with `(long) configMb * ...`.
|
||||||
|
|
||||||
|
### F-38: Deep dereference / recursive accumulator stack overflow
|
||||||
|
Recursive descent over a long flat input (CRTP-style or tail-recursion via `Iterables.concat`) consumes one stack frame per element, overflowing the JVM stack on large inputs.
|
||||||
|
**Look for:** Recursive `computeNext()`, `Iterables.concat` accumulated in a loop, recursive split helpers without iterative replacement.
|
||||||
|
|
||||||
|
### F-39: Off-by-one in length-vs-index check
|
||||||
|
A bounds guard uses strict greater-than against array length (`i > arr.length`) when greater-or-equal is required, allowing `i == arr.length` to pass into an indexing operation.
|
||||||
|
**Look for:** `if (i > arr.length)` then `arr[i]`; `if (size > limit)` checks placed after insertion instead of before.
|
||||||
|
|
||||||
|
### F-41: Unit-mismatched comparison via shadow constant
|
||||||
|
A timestamp stored in seconds is compared against another in milliseconds (or microseconds), or a millisecond TimeUnit is paired with a value computed in nanoseconds, causing unit-induced wrong comparisons or premature expiry by a 1000× factor.
|
||||||
|
**Look for:** Seconds-resolution `localDeletionTime` compared against microsecond-resolution `writeTime`; `TimeUnit.MILLISECONDS.sleep(nanos)` patterns.
|
||||||
|
|
||||||
|
### F-42: Empty-default treated as "smallest" in min-finder
|
||||||
|
A min-finding loop initializes the accumulator to an empty collection (size 0) and only updates when a strictly smaller is found; no collection is smaller than empty, so no candidate ever wins.
|
||||||
|
**Look for:** `min = Collections.emptyList()` or `min = empty`; subsequent loop comparisons with strict `<`.
|
||||||
|
|
||||||
|
### F-43: Over-1.0 / over-100% percentage
|
||||||
|
A progress ratio uses decompressed-byte counts in the numerator while the denominator is compressed bytes (or live count vs total estimate), making the percentage exceed 100% or go negative.
|
||||||
|
**Look for:** Progress display computing `done/total` where `done` and `total` come from different layers; ratio comparisons sensitive to dimension.
|
||||||
|
|
||||||
|
### F-44: Splitting at fixed size never reduces below threshold
|
||||||
|
A retry-on-overflow loop always splits at the same size and never reduces below the rejection threshold, so the same item is rejected and re-split forever, producing an infinite loop or stack overflow.
|
||||||
|
**Look for:** Split-and-retry helpers using a constant chunk size; recursion with no halving on the retry path.
|
||||||
|
|
||||||
|
### F-45: Read full unsigned value, then narrow without validation
|
||||||
|
A wide unsigned (or large signed) value is read from the wire into the correct type, then immediately cast to a narrower type without first validating that the value fits, silently truncating large values to wrong/negative numbers.
|
||||||
|
**Look for:** `int x = (int) wireLong` patterns near deserialization; numeric configuration fields reduced to a smaller type after reading.
|
||||||
|
|
||||||
|
### F-46: Misaligned index from stored vs actual array length
|
||||||
|
Code uses an index stored at write time but the corresponding array length at read time is shorter (e.g., schema evolved, fewer parallel-array entries), causing `IndexOutOfBoundsException`.
|
||||||
|
**Look for:** Loops bounded by one collection's `size()` while indexing another; comparator/parallel-list iterations.
|
||||||
|
|
||||||
|
### F-47: Half-open / open-closed interval mishandled
|
||||||
|
Interval intersection, containment, or merge code uses an open-closed interval where closed-closed is required (or vice versa), so points exactly at the boundary fall on the wrong side; alternatively a comparator treats wraparound as full-ring.
|
||||||
|
**Look for:** `interval.contains` helpers comparing `x > start && x < end` vs `>=`/`<=`; range-merge helpers around intersection of intervals.
|
||||||
|
|
||||||
|
### F-49: Counter overflow for short types
|
||||||
|
A counter typed as `short` is incremented per event (timeout, retry); after enough events it overflows to a negative value, wedging the producer with `IllegalArgumentException` or `NegativeArraySizeException`.
|
||||||
|
**Look for:** `short` fields incremented in retry/timeout paths; cumulative counters without periodic reset.
|
||||||
|
|
@ -0,0 +1,182 @@
|
||||||
|
# Category: Concurrency and Locking
|
||||||
|
|
||||||
|
Bugs caused by races, ordering hazards, lock-discipline violations, publication-safety gaps, and circular waits between cooperating threads or async tasks.
|
||||||
|
|
||||||
|
## Diff signals (when to load this category)
|
||||||
|
|
||||||
|
Load this category if the patch contains ANY of:
|
||||||
|
- New or changed `synchronized`, `Lock`, `ReadWriteLock`, `ReentrantLock`, `mutex`, semaphore, latch, or condition-variable usage
|
||||||
|
- `volatile`, `Atomic*`, `AtomicReference`, `compareAndSet`, `getAndSet`, or memory-fence-related changes
|
||||||
|
- `Future`, `CompletableFuture`, `await`, `get()`, `join()`, callbacks chained on async results
|
||||||
|
- A new background thread, executor submission, scheduler, or task pool — especially with bounded queues or single-threaded executors
|
||||||
|
- Changes to publish-then-signal sequences (set flag / count-down / notify / complete a future after writing shared state)
|
||||||
|
- Iteration over a shared/concurrent collection without explicit lock acquisition, or methods returning live views of shared collections
|
||||||
|
- Lazy or double-checked initialization of fields read from multiple threads
|
||||||
|
- Reference-counted resource lifecycle calls (`acquire`, `release`, `retain`, refcount inc/dec)
|
||||||
|
- Cancellation, abort, shutdown, or close paths that interact with in-flight work
|
||||||
|
- Reading "current state" / membership / counters as separate steps from acting on them (check-then-act, observe-then-acquire)
|
||||||
|
- Comparators or sort keys that read mutable fields, metrics, or time
|
||||||
|
- Atomic swap-then-drain, swap-then-clear, or replace-then-process patterns
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### F-01: Atomic swap then drain window
|
||||||
|
A shared mutable collector is drained after atomically swapping in a replacement, but writers in flight when the swap occurred append to the old instance after the drain completes, dropping or duplicating entries.
|
||||||
|
**Look for:** A `getAndSet`/CAS replacing a mutable container followed by a separate read of the previous instance, with no fence or coordination ensuring writers have observed the swap before draining.
|
||||||
|
|
||||||
|
### F-02: Inconsistent lock acquisition order between components
|
||||||
|
Two cooperating components acquire each other's locks in opposite orders under concurrent execution, producing a deadlock.
|
||||||
|
**Look for:** Two methods on different classes that each call into the other while holding their own monitor; synchronized methods that invoke a peer's synchronized method.
|
||||||
|
|
||||||
|
### F-03: Cancel/abort path bypasses serialization queue under same lock
|
||||||
|
A cancellation path bypasses the normal serialized work queue and directly mutates shared state while already holding its governing lock, causing reentrant corruption.
|
||||||
|
**Look for:** A `cancel`/`abort`/`close` method that acquires the same lock used by the regular task pipeline and calls handlers inline rather than enqueuing.
|
||||||
|
|
||||||
|
### F-04: Coordinator awaits future whose precondition needs its own notification
|
||||||
|
A coordinator blocks on a future whose completion depends on a notification only the coordinator can send — classic self-wait deadlock.
|
||||||
|
**Look for:** `future.get()`/`await()` followed by code that would have triggered the resolving event; single-threaded executors blocked on tasks that need to run on the same executor.
|
||||||
|
|
||||||
|
### F-05: Self-deadlock by re-acquiring same monitor
|
||||||
|
Calling a public synchronized method from within a section already holding that monitor deadlocks if the lock is non-reentrant or recurses unsafely if it is.
|
||||||
|
**Look for:** Synchronized method that calls another synchronized method on the same singleton or class object.
|
||||||
|
|
||||||
|
### F-06: Lock held across blocking I/O whose callback needs the same lock
|
||||||
|
A coarse-grained lock is held while blocking on an I/O response whose completion handler also tries to acquire the same lock, deadlocking the calling thread and the I/O-completion thread.
|
||||||
|
**Look for:** `synchronized`/`lock.lock()` enclosing `future.get()`, `channel.read()`, RPC `await()`, or blocking queue `put()` whose handler acquires the same monitor.
|
||||||
|
|
||||||
|
### F-07: Mutable shared field neither final nor volatile
|
||||||
|
A field shared across threads is neither `final` nor `volatile`, so the JMM gives no visibility guarantee — readers may observe stale values indefinitely.
|
||||||
|
**Look for:** Plain non-final field assigned by one thread and read by others without synchronization or atomic wrapper.
|
||||||
|
|
||||||
|
### F-08: Publication ordering — signal set before guarded state ready
|
||||||
|
A "publishing" flag/counter signals readers that dependent state is ready, but the signal is assigned before the dependent fields, so a concurrent reader sees the signal while the data is still null. Equivalently: a condition variable is signalled before shared state is updated; an atomic counter reaches its threshold before the backing collection is fully published, so a woken thread observes fewer responses than expected.
|
||||||
|
**Look for:** A volatile/atomic write (counter `incrementAndGet() == N`, flag set, future complete, `signal()`, `countDown()`) executed before assignments to fields the consumer will read after observing the signal.
|
||||||
|
|
||||||
|
### F-09: TOCTOU between flag and guarded resource
|
||||||
|
A check of a flag and the subsequent retrieval of the resource it guards happen under no common lock; concurrent removal can observe the flag as true but find the resource null or destroyed.
|
||||||
|
**Look for:** `if (enabled) { resource.use(); }` or `if (map.containsKey(k)) map.get(k).foo()` with no lock spanning both calls.
|
||||||
|
|
||||||
|
### F-10: Comparator reads live mutable state, breaking total order
|
||||||
|
A comparator reads a field that mutates concurrently (a metric, rate, or shared map), so it returns inconsistent results across calls and violates the comparator contract — `IllegalArgumentException` from the sort.
|
||||||
|
**Look for:** Comparator implementations that dereference instance fields or shared maps mutated by other threads during the sort.
|
||||||
|
|
||||||
|
### F-11: Snapshot read inside lock, but downstream re-reads original
|
||||||
|
Inside a critical section, code reads a shared mutable reference into a local snapshot but then passes the original (un-snapshotted) reference downstream, defeating the snapshot.
|
||||||
|
**Look for:** `Foo local = this.foo; doStuff(this.foo);` instead of `doStuff(local);` inside a synchronized block.
|
||||||
|
|
||||||
|
### F-12: Double-checked field accessed twice with race between
|
||||||
|
Lazy-init / DCL pattern reads a nullable field twice with the null check on the first read and dereference on the second; a concurrent write to null between the reads NPEs despite the guard.
|
||||||
|
**Look for:** `if (this.field != null) this.field.method();` instead of capturing the field into a local first.
|
||||||
|
|
||||||
|
### F-13: Unsynchronized iteration of a shared collection
|
||||||
|
Iterating a non-thread-safe collection (`HashMap`, `ArrayList`) — or returning a live view of one — while another thread mutates it throws `ConcurrentModificationException` or yields torn reads. Even `Collections.synchronizedX` requires explicit external synchronization for iteration.
|
||||||
|
**Look for:** Fields of type `HashMap`/`ArrayList` accessed across threads; methods returning `map.values()`, `keySet()`, or `entrySet()` directly; iteration of synchronized wrappers without surrounding `synchronized(map)`.
|
||||||
|
|
||||||
|
### F-14: Cache invalidation fires before commit
|
||||||
|
A cache-invalidation notification is sent before the underlying state change is committed; a thread re-reading between invalidation and commit caches the pre-change value permanently.
|
||||||
|
**Look for:** `invalidate(k); store.put(k, v);` patterns instead of `store.put(k, v); invalidate(k);`.
|
||||||
|
|
||||||
|
### F-15: Compound check-then-act on concurrent collection
|
||||||
|
`isEmpty()` / `containsKey()` / `size()` on a concurrent collection followed by an action assuming the result still holds; concurrent modification between calls produces a stale view, NPE, or `NoSuchElementException`.
|
||||||
|
**Look for:** `if (!set.isEmpty()) set.first();`, `if (map.containsKey(k)) doX(map.get(k));`, `if (counter.get() > 0) decrement()`.
|
||||||
|
|
||||||
|
### F-16: Counter decremented before associated cleanup completes
|
||||||
|
A capacity / reference / in-flight counter is decremented before the resources tied to the slot are fully released; an admission loop sees the new capacity and submits work that races the still-running cleanup.
|
||||||
|
**Look for:** `counter.decrementAndGet()` followed by additional cleanup; release callbacks that update accounting before tearing down underlying resources.
|
||||||
|
|
||||||
|
### F-17: Eviction frees resource while consumers hold raw pointers
|
||||||
|
A cache eviction or release callback unconditionally frees native/off-heap memory while consumers hold raw references obtained without ref-counting, producing use-after-free.
|
||||||
|
**Look for:** Eviction listeners that `free`/`close` resources accessed via plain references rather than ref-count handles; reads of off-heap memory not preceded by `acquire()`/`retain()`.
|
||||||
|
|
||||||
|
### F-18: Observe-then-acquire window for shared resource
|
||||||
|
A resource view is observed and references are acquired in a separate step; a concurrent free between observation and acquisition causes use-after-free.
|
||||||
|
**Look for:** `Resource r = lookup(); r.acquire();` (instead of an atomic `tryAcquire`); or two-step retain patterns across a lock boundary.
|
||||||
|
|
||||||
|
### F-19: Cancellation signal is non-blocking; clearing proceeds before task finishes
|
||||||
|
A background task is signalled to stop but the signal is non-blocking; a destructive state-clearing operation proceeds before the task observes the signal, allowing the task to write into already-deleted state.
|
||||||
|
**Look for:** `task.cancel()` immediately followed by state mutation, with no `awaitTermination`/`join` between.
|
||||||
|
|
||||||
|
### F-20: Background thread started inside constructor — escape of `this`
|
||||||
|
A background thread is started inside a constructor; the thread observes a partially-constructed owner object before all fields are assigned.
|
||||||
|
**Look for:** `new Thread(...).start()` or executor submission inside a constructor, especially in non-final classes; passing `this` to external registries before the constructor completes.
|
||||||
|
|
||||||
|
### F-21: Shared mutable scratch buffer treated as instance-scoped
|
||||||
|
A scratch buffer, builder, or thread-local "hint" stored as an instance field is reset and overwritten by concurrent callers, corrupting each other's results.
|
||||||
|
**Look for:** Mutable byte buffers, `StringBuilder`s, or thread-locals declared as instance fields, used by multiple threads.
|
||||||
|
|
||||||
|
### F-22: Latch never decrements on exception path
|
||||||
|
A latch's `countDown()` is positioned after a setup block that can throw; on exception the latch is never decremented and the waiter hangs forever.
|
||||||
|
**Look for:** `countDown()` outside a `finally` block; especially after I/O, allocation, or constructor calls that may throw.
|
||||||
|
|
||||||
|
### F-23: Submission counter incremented before submit; rejection forgets decrement
|
||||||
|
An in-flight task counter is incremented before submission; the rejection-handling path omits the matching decrement, leaving the counter permanently inflated.
|
||||||
|
**Look for:** `pending.incrementAndGet(); executor.submit(task);` without a corresponding decrement in the catch/reject path.
|
||||||
|
|
||||||
|
### F-24: Bounded executor blocks producer; consumer needs producer thread
|
||||||
|
All threads in a bounded pool block on futures whose continuations are queued behind the blocking threads — pool deadlock. Equivalently: a synchronous-handoff queue blocks the submitter when the pool saturates.
|
||||||
|
**Look for:** Code submitting tasks that block on `future.get()` of other tasks dispatched to the same executor / single-threaded stage; `SynchronousQueue` in `ThreadPoolExecutor` constructors.
|
||||||
|
|
||||||
|
### F-25: CAS retry mutates source argument or rolls back via plain set
|
||||||
|
A CAS retry passes the source object directly to a mutating merge that modifies it in place; subsequent iterations operate on already-mutated data. Equivalently: a shared atomic counter is rolled back by a plain `set()` instead of a CAS.
|
||||||
|
**Look for:** `while (!ref.compareAndSet(old, merge(old, src)))` where `merge` mutates inputs; `counter.set(local + delta)` instead of `addAndGet`.
|
||||||
|
|
||||||
|
### F-26: CAS spin loop without progress-detection or backoff
|
||||||
|
An unbounded CAS retry has no fallback to blocking under contention; threads spin without progress, consuming CPU in a livelock.
|
||||||
|
**Look for:** `while (!field.compareAndSet(...))` with no iteration cap, no backoff, and no escape if the condition can never be met.
|
||||||
|
|
||||||
|
### F-27: Two correlated values updated non-atomically
|
||||||
|
Two paired counters / fields representing one logical pair are updated and read in separate unsynchronized steps; a torn read produces inconsistent state (e.g., zero denominator in a ratio, or one map updated and the sibling stale).
|
||||||
|
**Look for:** Two `Atomic*` fields or two related maps mutated in separate statements rather than under a common lock or atomic record.
|
||||||
|
|
||||||
|
### F-28: Two-step state-machine read-then-write across threads
|
||||||
|
Reading the current state and writing the new state execute as separate unsynchronized operations; a concurrent thread can change state between them, producing a stale write or invalid transition.
|
||||||
|
**Look for:** `if (state == A) state = B;` patterns; non-atomic state machines outside any lock or `compareAndSet`.
|
||||||
|
|
||||||
|
### F-29: Shutdown stops downstream before draining in-flight work
|
||||||
|
Shutting down a consumer subsystem before its message source races with in-flight messages; or shutdown waits only for the work queue, not the executor producing into it. In-flight messages arrive at partially-torn-down state and corrupt or lose data.
|
||||||
|
**Look for:** `consumer.stop(); source.stop();` ordering; `queue.drain(); resource.close();` without first stopping the producer-side executor.
|
||||||
|
|
||||||
|
### F-30: Shutdown guard check and submit are not atomic
|
||||||
|
A shutdown flag is checked and a task submitted in separate steps; concurrent shutdown between them causes the submit to throw and the partially-constructed task to leak its acquired resources.
|
||||||
|
**Look for:** `if (!shutdown) executor.submit(...)` outside any synchronized region, especially when the task holds resources.
|
||||||
|
|
||||||
|
### F-31: Lock released before snapshot of shared collection is iterated
|
||||||
|
A lock is released before iterating a snapshot of a shared collection; a concurrent writer deletes underlying resources during iteration, producing missed data or use-after-free.
|
||||||
|
**Look for:** Pattern: `lock.lock(); copy = new ArrayList(shared); lock.unlock(); for (x : copy) x.use();` where iterated objects may be freed.
|
||||||
|
|
||||||
|
### F-32: Check-then-put on concurrent map is not atomic
|
||||||
|
Two threads call `containsKey` then `put` on a concurrent map; both insert, and one silently overwrites the other.
|
||||||
|
**Look for:** `if (!map.containsKey(k)) map.put(k, v);` instead of `putIfAbsent` / `computeIfAbsent`.
|
||||||
|
|
||||||
|
### F-33: New object published before its internal resources are ready
|
||||||
|
An object is exposed via shared state before its constructor / initializer has finished assigning internal resources; threads observing the published reference between init steps find an incomplete object and throw.
|
||||||
|
**Look for:** Fields published to a shared map / volatile field before all the constructor work has run; factory methods that register `this` before final assignments.
|
||||||
|
|
||||||
|
### F-34: Thread-local context cleared while async work still references it
|
||||||
|
An async task captures contextual state cleared in a `finally` when the coordinating operation completes; a response arriving after completion finds the context null and NPEs.
|
||||||
|
**Look for:** `ThreadLocal` clears in `finally` blocks while async tasks remain in flight referencing them.
|
||||||
|
|
||||||
|
### F-35: Spurious wakeup not handled in condition wait
|
||||||
|
A condition-wait loop uses a single `await` without re-checking the predicate on wakeup; spurious wakeup causes the caller to return null before the actual response arrives.
|
||||||
|
**Look for:** `cond.await()` followed by use of guarded state without a surrounding `while (!predicate)` loop.
|
||||||
|
|
||||||
|
### F-36: Single shared mutable message reused across concurrent sends
|
||||||
|
A request message object is created once and reused for all endpoints in a fan-out loop; all requests share the same ID/headers and concurrent responses collide.
|
||||||
|
**Look for:** Messages constructed outside a per-recipient loop, then mutated and dispatched repeatedly.
|
||||||
|
|
||||||
|
### F-37: Pooled buffer returned while in-flight write still uses it
|
||||||
|
A pooled buffer is returned to the pool before the in-flight network write completes, allowing it to be overwritten and corrupting the original payload.
|
||||||
|
**Look for:** `pool.release(buf)` placed in a synchronous path while I/O write is still pending asynchronously.
|
||||||
|
|
||||||
|
### F-38: Shared buffer's position advanced as side effect of a read
|
||||||
|
A read method on a shared buffer (`get`, `getInt`, `getLong`) advances the position as a side effect; subsequent readers from the same buffer see wrong bytes.
|
||||||
|
**Look for:** `buf.getInt()` / `buf.get()` / `buf.array()` on a shared `ByteBuffer` without a prior `duplicate()` or `slice()`.
|
||||||
|
|
||||||
|
### F-39: Heartbeat / liveness reply sent after shutdown started
|
||||||
|
A liveness-probe response handler sends a reply unconditionally without checking whether the local node is shutting down, causing peers to re-mark it as alive after a shutdown notification.
|
||||||
|
**Look for:** Heartbeat / probe handlers that omit a `shuttingDown` guard; reply paths whose state should depend on lifecycle.
|
||||||
|
|
||||||
|
### F-40: Cleanup loop fails first item and abandons rest
|
||||||
|
A cleanup loop does not wrap each iteration in its own exception handler; the first failed release terminates the loop, leaving remaining resources permanently unreleased.
|
||||||
|
**Look for:** `for (X x : resources) x.close();` without per-iteration `try/catch` or `addSuppressed` handling.
|
||||||
|
|
||||||
|
|
@ -0,0 +1,173 @@
|
||||||
|
# Category: Conditions and Predicates
|
||||||
|
|
||||||
|
Bugs where a comparison, guard, predicate, or branching condition is logically wrong: wrong operator polarity, wrong constant or field, wrong direction flag, asymmetric vs symmetric comparator confusion, missing replay-time filter, conflated conditions, or a sentinel guard that tests the wrong variable.
|
||||||
|
|
||||||
|
## Diff signals (when to load this category)
|
||||||
|
|
||||||
|
Load this category if the patch contains ANY of:
|
||||||
|
- New or modified comparison operators (`<`, `<=`, `>`, `>=`, `==`, `!=`) in a guard, loop condition, or boundary test
|
||||||
|
- A boolean expression that gates a side effect, branch, early return, or filter step
|
||||||
|
- An `equals()`, `hashCode()`, `compareTo()`, or `Comparator` implementation (added, modified, or its field set changed)
|
||||||
|
- A predicate method (`isXxx`, `hasXxx`, `canXxx`, `matches`, `contains`) added or modified
|
||||||
|
- A `switch` over an enum or type discriminator (especially with `default` branches or fall-through)
|
||||||
|
- A direction/order flag (ascending/descending, forward/reverse, oldest/newest) passed to a helper
|
||||||
|
- An error message or log statement that quotes a constant, threshold, or field name
|
||||||
|
- A composite key access where indexed positions or named components are read
|
||||||
|
- A `filter`, `where`, `predicate`, or `Predicate<T>` lambda added, removed, or re-targeted
|
||||||
|
- A type check via `instanceof`, `getClass() ==`, or a custom type-discriminator predicate
|
||||||
|
- Changes to membership tests (`contains`, `containsKey`, `Set.of`, allowlist/denylist)
|
||||||
|
- Sentinel value handling (`-1`, `null`, `Long.MAX_VALUE`, `EMPTY_BUFFER`) in a comparison or guard
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### F-01: Wrong polarity comparison in availability/threshold check
|
||||||
|
A comparison uses `!=` or strict-vs-non-strict in the wrong direction (e.g. `!=` instead of `<`, `>` instead of `>=`), so a value satisfying the intended predicate triggers the failure branch and vice versa. Common in replica-count, quorum, capacity, and threshold checks.
|
||||||
|
**Look for:** `if (count != required)` where intent is "fewer than required"; `if (size > limit)` after append where intent is "would exceed"; tombstone/cleanup `<=` vs `<` at boundary.
|
||||||
|
|
||||||
|
### F-02: Asymmetric vs symmetric comparator confusion in binary search
|
||||||
|
A symmetric comparator is used to binary-search a collection of intervals (or vice versa) with a point key, leaving equality cases ambiguous so the search lands on the wrong side of a boundary.
|
||||||
|
**Look for:** `Collections.binarySearch` or `Arrays.binarySearch` over interval/range structures; comparator used both for sort order and for skip/seek operations.
|
||||||
|
|
||||||
|
### F-03: Comparison reads wrong field of composite key/object
|
||||||
|
A containment, equality, or eligibility check accesses the wrong indexed component (or wrong named field) of a composite key or struct, producing incorrect routing decisions with no runtime error.
|
||||||
|
**Look for:** `key.get(i)` / `tuple._2` / `components[N]` inside an `equals`/`compareTo`/`contains` body; index constant (`0`, `1`) used in field-position lookups.
|
||||||
|
|
||||||
|
### F-04: Wrong direction/ordering flag passed to helper
|
||||||
|
A boolean direction flag (forward/reverse, ascending/descending, oldest-first/newest-first) is passed with the wrong literal to an ordered-read or scan helper, causing diffs to be applied in reverse order or pages to scan the wrong end.
|
||||||
|
**Look for:** `boolean reversed`, `boolean ascending`, `Direction.FORWARD/REVERSE` arguments at call sites; reversed-scan code paths that don't swap start/end bounds.
|
||||||
|
|
||||||
|
### F-05: Error or log message quotes a similarly-named but wrong constant
|
||||||
|
Diagnostic output references one constant while the preceding condition enforces a different (similarly-named) one, misleading operators about what threshold actually fired.
|
||||||
|
**Look for:** `String.format(... %s ..., CONSTANT_A)` immediately following `if (x > CONSTANT_B)`; error messages that hardcode enum names copied from sibling branches.
|
||||||
|
|
||||||
|
### F-06: Inverted boolean condition (`!flag` vs `flag`)
|
||||||
|
A negation is added or removed by mistake, causing the guarded branch to fire under the opposite circumstances. Often hidden in `isEmpty()`/`!isEmpty()`, `isEnabled()`/`!isEnabled()`, error-vs-success classifiers, and `var x = a ? b : c` ternaries with branches swapped.
|
||||||
|
**Look for:** `if (!isEmpty(...))` flips, ternary with single-letter variable in test, recently changed boolean predicate methods that return `!expr`.
|
||||||
|
|
||||||
|
### F-07: Conflated AND/OR in compound guard
|
||||||
|
A skip-guard combines two independent safety conditions with `AND` instead of `OR` (or vice versa), permitting inputs that satisfy only one condition to bypass; or two orthogonal conditions are conflated into a single boolean.
|
||||||
|
**Look for:** `if (a && b)` short-circuiting in skip/abort logic; recently modified compound conditions where one side reads as defense-in-depth.
|
||||||
|
|
||||||
|
### F-08: Missing replay-time / state-restore filter
|
||||||
|
A filter that removes superseded or already-applied entries is skipped on the startup/replay path while applied on the live write path, leaving stale entries visible as current state after restart.
|
||||||
|
**Look for:** Replay/recovery loops in `start()`, `loadFromDisk()`, `replay()` paths; filters present in writer but absent in reader.
|
||||||
|
|
||||||
|
### F-09: Sentinel guard tests the wrong variable
|
||||||
|
A null/sentinel/disabled-state check is wired to an outer-scope or unrelated variable, so the guard never fires for the variable it is meant to protect, allowing the sentinel value to flow downstream.
|
||||||
|
**Look for:** Guard like `if (other != null)` immediately followed by `if (... self ...)` field access; loop-sentinel check on outer-iteration var while body uses inner-loop var.
|
||||||
|
|
||||||
|
### F-10: equals/hashCode omits a semantically-significant field
|
||||||
|
After a new field is added to a class, `equals()` and/or `hashCode()` are not updated, so logically distinct instances collide in caches/sets, change-detection misses updates, and notifications fail to fire.
|
||||||
|
**Look for:** Added field that does not appear in nearby `equals`/`hashCode` methods; record-style classes with manual `equals` overrides.
|
||||||
|
|
||||||
|
### F-11: Predicate too broad — accepts inputs it should reject
|
||||||
|
A type or eligibility check uses a broader supertype/wildcard than necessary, accepting inputs that share a class hierarchy but not the contract the caller assumes; or a switch's `default` swallows cases that should be explicit.
|
||||||
|
**Look for:** `instanceof BaseType` where a specific subtype was intended; `default:` branch that should be unreachable; check on count instead of membership.
|
||||||
|
|
||||||
|
### F-12: Predicate too narrow — rejects valid inputs
|
||||||
|
A predicate or filter excludes a category it should accept (a new enum variant, a wrapper type, a previously-unhandled subtype), silently dropping or misrouting valid input.
|
||||||
|
**Look for:** Hand-maintained type/enum dispatches recently extended in one place but not another; allowlist/denylist sets missing newly-added entries.
|
||||||
|
|
||||||
|
### F-13: Type-identity check ignores decorator/wrapper
|
||||||
|
`instanceof X` or `getClass() == X.class` is applied to a wrapped value without first unwrapping the decorator (e.g. reversed-order, frozen-collection wrappers), so the predicate returns false for wrapped instances of X.
|
||||||
|
**Look for:** `instanceof` on a column type, expression type, or AST node where decorator wrappers exist; missing `.unwrap()`/`getBaseType()` call before the test.
|
||||||
|
|
||||||
|
### F-14: Recursive/inherited predicate's base case is wrong
|
||||||
|
A type-hierarchy predicate returns the wrong default at the base class and is overridden only on leaf types, so container/decorator/wrapper types inherit the base-class default and are never recognized.
|
||||||
|
**Look for:** Boolean methods named `isComplex`, `containsX`, `referencesY` defined on base type with `return false` and only sometimes overridden.
|
||||||
|
|
||||||
|
### F-15: Symmetric ordering applied to asymmetric / circular type
|
||||||
|
A range or comparator operation assumes total ordering, but the underlying type's order is partial, circular, or NaN-bearing; sorting "for convenience" or using min/max on such values produces wrong/empty results.
|
||||||
|
**Look for:** `Math.min`/`Math.max` on raw token / wrap-around values; comparators on collections containing `NaN`, `null`, or sentinel boundary subtypes; sort applied before a query against circular-token ranges.
|
||||||
|
|
||||||
|
### F-16: Comparator ignores per-column reversed sort direction
|
||||||
|
A range-bounds check or paging cursor uses an operator without flipping it for `DESC`-ordered columns, producing inverted results for queries against reversed-order data.
|
||||||
|
**Look for:** Hardcoded `>`/`<` in range filters that touch column metadata; missing `reverseIfNeeded()`/operator flip when iterating over `ClusteringComparator` with `isReversed()` columns.
|
||||||
|
|
||||||
|
### F-17: equals() compares against `this` or wrong object reference
|
||||||
|
An `equals()` bridge or delegated comparator passes `this` instead of the cast argument, or compares a field to itself, so the method always returns true for distinct instances or always returns false for equal ones.
|
||||||
|
**Look for:** `bridge.equals(this, other)` patterns where one argument should be the casted parameter; `field == field` rather than `field == other.field`.
|
||||||
|
|
||||||
|
### F-18: Comparator reads buffer with side-effecting (positional) form
|
||||||
|
A comparator or equality method uses the relative `getInt()`/`get()` form on a shared `ByteBuffer` without first duplicating it, advancing the buffer position as a side effect and corrupting subsequent reads.
|
||||||
|
**Look for:** `buffer.getInt()` / `buffer.get()` inside a `Comparator` or `equals` method without a preceding `.duplicate()` or `.slice()`.
|
||||||
|
|
||||||
|
### F-19: Boundary check at exact-equal / degenerate-range case
|
||||||
|
A range non-overlap check, intersection skip-guard, or "might-match" predicate uses strict `<` where `<=` is needed (or vice versa), missing the exact-equal endpoint case and either issuing the same value twice or failing to make forward progress.
|
||||||
|
**Look for:** Iterator skip guards `current >= target` instead of `current > target`; range non-overlap tests that never trigger when ranges share an endpoint.
|
||||||
|
|
||||||
|
### F-20: Empty-buffer / empty-collection treated as null or special value
|
||||||
|
A check uses `buffer.equals(EMPTY)` or `field == null` interchangeably with emptiness, but the type uses zero-length encoding to mean a real zero value (e.g. integer zero), so legitimate values are silently dropped from the index/result set.
|
||||||
|
**Look for:** `buffer.remaining() == 0`, `value == EMPTY_BYTE_BUFFER` checks in indexing/serialization paths; dual interpretation of null vs empty.
|
||||||
|
|
||||||
|
### F-21: Membership/lookup uses different representation than insertion
|
||||||
|
A `Map`/`Set` is built using one string/key representation (e.g. address without port, lowercased name) and queried with another, so every lookup misses and the guarded counter, status, or filter is silently bypassed.
|
||||||
|
**Look for:** Two distinct address-formatting/normalization functions used at insert vs lookup; case-insensitive comparison on one side only; `String` keys built with different formatters.
|
||||||
|
|
||||||
|
### F-22: Filter/exclusion never matches due to hidden suffix or normalization
|
||||||
|
An exact-name filter checks bare logical names, but on-disk or runtime names carry a suffix (UUID, version, generation, separator), so the check never matches and the filter is silently bypassed.
|
||||||
|
**Look for:** `set.contains(rawName)` / `name.equals(constant)` against directory or table names; missing `startsWith` + delimiter combination for prefix checks.
|
||||||
|
|
||||||
|
### F-23: Switch over enum is missing a case (silent fall-through)
|
||||||
|
A switch statement omits a case for a newly-added enum constant, falling through to a `default:` that throws, returns null, or silently does nothing — the missing case becomes a runtime failure or silent drop.
|
||||||
|
**Look for:** `switch (someEnum) { ... default: ... }` without an exhaustive set of `case` labels; recently-added enum constants near switch dispatches.
|
||||||
|
|
||||||
|
### F-24: Branches swapped in if/else or ternary
|
||||||
|
The if-branch and else-branch are exchanged, so live data is routed through the deletion handler (or vice versa), or a null-handling ternary dereferences the null operand.
|
||||||
|
**Look for:** `cond ? maybeNull.field : default` (NPE risk), `if (isDeleted) writeLive() else writeTombstone()` ordering inversions during merges.
|
||||||
|
|
||||||
|
### F-25: Membership-by-iteration in a hot path with O(n) cost
|
||||||
|
A membership test iterates an entire live collection instead of using an indexed lookup, producing quadratic complexity in startup or hot loops; or a count comparison is used where membership semantics are required.
|
||||||
|
**Look for:** `for (entry : map) if (entry.equals(x))` patterns inside frequent loops; comparison of `list.size()` against expected count rather than `list.containsAll(...)`.
|
||||||
|
|
||||||
|
### F-26: Unconditional "always true / always false" validator
|
||||||
|
A validation method returns true for all inputs (stub left in) or false at base class with sparse overrides, so invalid inputs pass silently or valid inputs are uniformly rejected.
|
||||||
|
**Look for:** Methods that just `return true;` / `return false;` whose name implies a real check; predicate returning constant under one configuration.
|
||||||
|
|
||||||
|
### F-27: Compatibility-direction check inverted
|
||||||
|
A "is X compatible with Y" check passes the arguments in the wrong order (asks if old can read new instead of new can read old, or whether the candidate can reach the existing instead of the reverse).
|
||||||
|
**Look for:** `isCompatible(a, b)` / `canRead(a, b)` calls on schema, type, or version objects where direction matters; type-compatibility checks during migration paths.
|
||||||
|
|
||||||
|
### F-28: Lexicographic vs numeric comparison
|
||||||
|
A version, count, or numeric value stored as a string is compared lexicographically (so `"10" < "9"`), causing version dispatch to misclassify two-digit versions as older or newer than intended.
|
||||||
|
**Look for:** Shell `[ "$x" \> "$y" ]` or `String.compareTo` against version strings or numeric tokens; sort over stringified IDs.
|
||||||
|
|
||||||
|
### F-29: Sentinel value participates in arithmetic without unwrapping
|
||||||
|
A "disabled" or "uninitialized" sentinel (e.g. `-1`, `Long.MAX_VALUE`) flows into comparisons or arithmetic that don't recognize it, producing nonsensical thresholds, immediate timeouts, or infinite loops.
|
||||||
|
**Look for:** Subtractions involving fields documented as sentinel-bearing; `Math.min(timeout, ...)` without filtering out the disabled marker; counter initialized to `-1` compared with `<` against a positive count.
|
||||||
|
|
||||||
|
### F-30: Identity-equality on freshly-constructed singleton
|
||||||
|
An identity check (`==`) is used on values from a factory that does not return a canonical singleton for the special case (e.g. empty buffer), so logically-identical instances compare unequal and the special path is missed.
|
||||||
|
**Look for:** `value == EMPTY_INSTANCE` / `instance == SENTINEL` checks where the sentinel is constructed via a factory that may not intern; `value.equals(EMPTY)` would be safer.
|
||||||
|
|
||||||
|
### F-31: Local-DC vs remote-DC filter polarity flipped
|
||||||
|
A "from local DC" / "from outside local DC" filter is implemented with the wrong sign, so in a single-DC cluster every response is discarded (or the wrong subset is counted) and operations time out or reach quorum on wrong votes.
|
||||||
|
**Look for:** `if (replica.getDatacenter().equals(local))` skip vs include conditions; consistency-level filters that exclude based on locality.
|
||||||
|
|
||||||
|
### F-32: Guard fires after the side effect it was meant to gate
|
||||||
|
An irreversible side effect (file creation, increment, allocation) is performed before the guard that decides whether to do it, so revoking the guard later cannot undo the effect and may permanently break subsequent operations.
|
||||||
|
**Look for:** `createHardLink(); if (!shouldCreate) ...` ordering inversions; metric increments before the guard check; mutations followed by validation that "fixes" by throwing.
|
||||||
|
|
||||||
|
### F-33: Compound predicate validated in isolation; cross-predicate satisfiability not checked
|
||||||
|
Multiple sibling predicates on the same field (e.g. `x > 5 AND x < 3`) are accepted at definition time because each is individually well-formed, allowing contradictory combinations to be persisted and later produce unexpected empty results or runtime errors.
|
||||||
|
**Look for:** Constraint/policy registration that validates entries one-at-a-time; missing `Range.intersect(...)`/satisfiability checks between sibling predicates.
|
||||||
|
|
||||||
|
### F-34: Stale or coincidental field used as control flag
|
||||||
|
A control flag is recomputed from a value comparison that coincidentally matches the default, or an enum field is reused with two distinct meanings, so callers in the second meaning silently take the first meaning's path.
|
||||||
|
**Look for:** Boolean fields whose initialization comment says "for X" but read site uses for Y; double-purpose timestamp/flag fields; flags that conflate "started" with "in progress".
|
||||||
|
|
||||||
|
### F-35: Exclusion set lookup uses path-equality but exclusions differ in prefix
|
||||||
|
An exclusion list contains objects with path-based equality, but the items being checked differ only by directory prefix (or vice versa), so the lookup always misses despite representing the same logical entity.
|
||||||
|
**Look for:** `excluded.contains(file)` where `file` was constructed from a different root than the entries; canonical-vs-relative path mixing.
|
||||||
|
|
||||||
|
### F-36: Inclusion-predicate polarity mirrored incorrectly between paired methods
|
||||||
|
Two complementary methods (include/exclude, accept/reject, isAlive/isDead) have inverted polarities on one path, so one method accepts what the other would reject — producing inconsistent results across read/write or upgrade boundaries.
|
||||||
|
**Look for:** Pairs of methods like `acceptsX`/`rejectsX`, `shouldKeep`/`shouldDrop`, recently changed in only one location.
|
||||||
|
|
||||||
|
### F-37: Equality / membership uses `Objects.hash()` on byte arrays
|
||||||
|
`Objects.hash(byteArray)` invokes identity-based array hashing while `equals` does a content comparison (or vice versa), violating the equals/hashCode contract and producing wrong results in hash-based collections.
|
||||||
|
**Look for:** `Objects.hash(byteArrayField)` rather than `Arrays.hashCode(byteArrayField)`; arrays passed to varargs hash methods.
|
||||||
|
|
||||||
|
### F-38: `equals()` only checks for non-null instead of correct type
|
||||||
|
An `equals` short-circuits on `other != null` rather than `other instanceof MyClass`, returning true for any non-null object of any type, producing false-positive equality across unrelated classes.
|
||||||
|
**Look for:** `if (other == null) return false; ... return field.equals(other.field);` without an `instanceof` check.
|
||||||
|
|
@ -0,0 +1,124 @@
|
||||||
|
# Category: I/O and Crash Safety
|
||||||
|
|
||||||
|
Bugs in durable persistence, flush/sync ordering, atomic file replacement, partial reads/writes, checksum handling, and journal/log replay where crashes, errors, or restarts can corrupt data, lose writes, or resurrect deleted state.
|
||||||
|
|
||||||
|
## Diff signals (when to load this category)
|
||||||
|
|
||||||
|
Load this category if the patch contains ANY of:
|
||||||
|
- New or changed file writes, especially overwrite-in-place, rename/move, or `FileChannel.write` / `OutputStream.write` paths
|
||||||
|
- Calls to `fsync`, `force()`, `flush()`, `sync()`, `close()` on file/channel/writer objects, or removal/addition of any of these
|
||||||
|
- Use of `FileChannel.read`, `read(ByteBuffer)`, `InputStream.read(byte[])` without explicit "read fully" / "write fully" loops
|
||||||
|
- Checksum / CRC / digest computation tied to a file or record (`CRC32`, `Adler32`, `MessageDigest`, custom digest update over a buffer)
|
||||||
|
- Commit-log, journal, write-ahead-log, replay, recovery, or checkpoint code paths
|
||||||
|
- Truncation, snapshot, or restore operations on files or persistent state
|
||||||
|
- Auto-closing serializer / try-with-resources around a writer that owns the file lifecycle
|
||||||
|
- Atomic-rename helpers (`Files.move`, `ATOMIC_MOVE`, `renameTo`) or marker-file presence checks for operation completion
|
||||||
|
- Buffer pool / off-heap memory release coordinated with disk I/O completion
|
||||||
|
- Multiple related files written together (data + index, segments + manifest)
|
||||||
|
- File-open flags such as `TRUNCATE_EXISTING`, `CREATE_NEW`, `APPEND`, `O_DSYNC`, or their absence
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### F-01: State file overwritten in-place with auto-closing serializer
|
||||||
|
A safety-critical state file is overwritten in-place using a serializer that auto-closes the underlying output stream on its own close, making a post-write `fsync` and atomic rename impossible. A crash mid-write corrupts the file and converts a recoverable error into an unrecoverable startup failure.
|
||||||
|
**Look for:** `serializer.serialize(out)` where `out` is a fresh `FileOutputStream`/`Writer` opened directly on the destination path, with no temp-file + rename pattern and no explicit `getFD().sync()` before close.
|
||||||
|
|
||||||
|
### F-02: `read()` not retried on short read
|
||||||
|
A stream `read()` fills the buffer with fewer bytes than requested without retrying. The remainder is uninitialized, and downstream processing of the partial data produces silently corrupt output.
|
||||||
|
**Look for:** `in.read(buf)`, `channel.read(buf)`, `is.read(arr, off, len)` without a wrapping loop that advances `off` until `len` bytes are consumed; call sites that ignore the returned length.
|
||||||
|
|
||||||
|
### F-03: `write()` not retried on partial write
|
||||||
|
A `FileChannel.write()` (or equivalent) may write fewer bytes than requested; the call site assumes the entire buffer was written. Result is silent partial writes to storage files.
|
||||||
|
**Look for:** `channel.write(buf)` or `out.write(arr, off, len)` without a loop checking `buf.hasRemaining()`; absence of a `writeFully` helper at the boundary to durable storage.
|
||||||
|
|
||||||
|
### F-04: Buffered writer closed without explicit flush/sync
|
||||||
|
A buffered writer is closed without first flushing the buffer or syncing the file descriptor, so a crash after close can silently lose buffered data even though `close()` returned cleanly.
|
||||||
|
**Look for:** `bufferedWriter.close()` paths that lack a preceding `flush()` + `getFD().sync()` / `force(true)` for files where durability is required.
|
||||||
|
|
||||||
|
### F-05: Checksum computed over wrong byte range or pre-write buffer
|
||||||
|
A checksum is computed over a buffer's pre-write region (covering zeros) or over only a subset of fields actually written. Stored checksums never validate, or validate against partial coverage.
|
||||||
|
**Look for:** `ByteBuffer.duplicate()` called after `serialize()` instead of before; CRC accumulator updated for some serialized fields but not others; digest computed before `flip()`.
|
||||||
|
|
||||||
|
### F-06: Checksum recorded before final close mutates the file
|
||||||
|
A file checksum is computed and written before a `close()` that itself performs truncation or buffer flushing. The on-disk content changes after the checksum is recorded, invalidating it.
|
||||||
|
**Look for:** `writeChecksum(...)` immediately followed by `out.close()` / `channel.close()` that may flush bytes; checksum on an open writer that still has pending data.
|
||||||
|
|
||||||
|
### F-07: Checksum suffix not consumed on error-skip path
|
||||||
|
An error-handling skip path returns early without consuming the trailing checksum bytes that follow each record. The stream is left positioned at checksum bytes that get misread as the next record's header.
|
||||||
|
**Look for:** `continue` / `return` inside a record-deserialization loop with no preceding `readFully(crcSize)` or position adjustment; layouts where each record is followed by a fixed-size CRC suffix.
|
||||||
|
|
||||||
|
### F-08: Integrity check fails when write batch or flush limit is exceeded
|
||||||
|
A configured batch-size or flush-period limit on a log/store is exceeded; downstream readers see torn writes that fail CRC or checksum validation because the limit is enforced asymmetrically between writer and reader.
|
||||||
|
**Look for:** Configurable flush thresholds paired with a CRC/checksum validator that assumes whole-record durability; writer paths that accumulate beyond the configured limit before issuing `force()` / `flush()`.
|
||||||
|
|
||||||
|
### F-09: Reader buffer not reset after checksum-mismatch exception
|
||||||
|
A shared mutable reader buffer is not reset to an empty view after a checksum-mismatch exception. The next read attempt observes stale data from the previously failed chunk.
|
||||||
|
**Look for:** Catch block for `CorruptDataException` / `IOException` in a chunk reader that does not call `buffer.clear()` / `position(0).limit(0)` before propagating or retrying.
|
||||||
|
|
||||||
|
### F-10: Omitting flush for one of several related files
|
||||||
|
When transitioning a mutable structure to a persisted read-only form, the flush for one of several related files (e.g., index but not data, or manifest but not children) is omitted. After recovery one file looks current while the others are stale.
|
||||||
|
**Look for:** A method that writes/closes multiple files but only calls `force()` / `fsync` on a subset; sibling files (`.db` + `.idx` + `.summary`, segments + manifest) where one path lacks a sync.
|
||||||
|
|
||||||
|
### F-11: File opened without `TRUNCATE_EXISTING`
|
||||||
|
A file-open utility omits the truncate-existing flag, so when a shorter payload is written to a pre-existing file the tail of the old content remains and the file contains a mix of new and stale bytes.
|
||||||
|
**Look for:** `Files.newByteChannel(path, CREATE, WRITE)` or `new FileOutputStream(file)` without `TRUNCATE_EXISTING`; rewrite-in-place paths that assume the new payload is at least as long as the old.
|
||||||
|
|
||||||
|
### F-12: Recovery start position advanced past truncation without comparing target
|
||||||
|
A recovery start position is advanced by a truncation record without checking whether that record post-dates the restore target. Records within the recovery window are silently skipped on restart.
|
||||||
|
**Look for:** Replay loops that consult a "last truncation" marker and unconditionally skip earlier offsets; absence of comparison between truncation timestamp and the requested recovery target.
|
||||||
|
|
||||||
|
### F-13: Replay path does not consult truncation records
|
||||||
|
A replay path does not consult truncation records at all, so data written before a truncation is resurrected when the log is replayed after restart.
|
||||||
|
**Look for:** Commit-log / journal replay loops with no filter step for truncation entries; replay code that bypasses the same truncation guard used in the live write path.
|
||||||
|
|
||||||
|
### F-18: New writer allocated immediately after closing previous
|
||||||
|
A new writer is allocated immediately after closing the previous one without checking whether there is more data to write. An empty writer is created on the final iteration and left in a dangling state on disk.
|
||||||
|
**Look for:** Compaction / split / segment-rotation loops that call `writer.close(); writer = new Writer(nextPath())` unconditionally inside the loop instead of guarding the next allocation on `iterator.hasNext()`.
|
||||||
|
|
||||||
|
### F-19: Empty file created on every idle flush cycle
|
||||||
|
A flush loop unconditionally opens and writes an output file before checking whether the source iterator has any data. Empty files are created on every idle flush cycle and accumulate.
|
||||||
|
**Look for:** `try (Writer w = openOutput(...)) { while (it.hasNext()) ... }` patterns where the writer is opened before the `hasNext()` check; periodic flush schedulers that materialize the destination eagerly.
|
||||||
|
|
||||||
|
### F-20: Partially-written file left on disk after exception
|
||||||
|
A file writer opened in an error-prone path is never aborted on exception; a partially-written output file remains on disk and is treated as a complete valid file by later readers.
|
||||||
|
**Look for:** `Writer w = open(...)` without try/finally that calls `w.abort()` / deletes the partial file on exception; commit semantics relying only on close success while the failure path leaves the destination intact.
|
||||||
|
|
||||||
|
### F-23: Directory not fsynced after delete/rename
|
||||||
|
A deletion or rename step does not sync the directory entry. A prior change may not yet be visible after a crash, causing the next pass to re-discover stale entries or fail to find committed names.
|
||||||
|
**Look for:** `Files.delete(...)` / `Files.move(..., ATOMIC_MOVE)` followed immediately by directory iteration without `directoryChannel.force(true)` or platform-equivalent.
|
||||||
|
|
||||||
|
### F-24: Stream `read()` returning 0 conflated with EOF (or vice versa)
|
||||||
|
A `read()` return value conflates zero bytes (try again) with end-of-stream (-1, closed). Genuine EOF is silently treated as a retry condition (loop never exits), or a transient zero-byte read closes a still-live stream.
|
||||||
|
**Look for:** `if (n == 0) return EOF;` on a non-blocking channel; `while (n != -1)` loops that treat a transient zero-byte read as "more to come"; missing `if (n < 0) closed=true;` guards.
|
||||||
|
|
||||||
|
### F-28: Dirty flag set after write loop, skipped on early exception
|
||||||
|
A dirty flag is set unconditionally after a write loop exits; if the loop terminates early due to an exception, the flag is never set and the buffer is not flushed, silently losing data.
|
||||||
|
**Look for:** `for (...) { write(...) } dirty = true;` with no try/finally; flush schedulers that gate work on `if (dirty)` while the write path can throw before reaching the flag assignment.
|
||||||
|
|
||||||
|
### F-29: Pooled buffer returned before in-flight write completes
|
||||||
|
A pooled buffer is returned to its pool before the in-flight network/disk write using it completes, allowing it to be overwritten and corrupting the original payload.
|
||||||
|
**Look for:** `pool.release(buf)` in the same scope as `channel.write(buf)` without awaiting completion; reference-counted buffers handed to async APIs without an extra `acquire()`.
|
||||||
|
|
||||||
|
### F-30: Use-after-free of off-heap buffer racing eviction
|
||||||
|
A cache get reads off-heap native memory without first incrementing the reference count; a concurrent eviction frees the memory between the map lookup and the read, causing a use-after-free crash or returning torn bytes.
|
||||||
|
**Look for:** `Buffer b = cache.get(key); read(b);` where the cache may evict and free `b`'s backing memory; missing `tryAcquire()` / `incrementRef()` before the read.
|
||||||
|
|
||||||
|
### F-31: Removing fsync as an "optimization" breaks crash safety
|
||||||
|
Removing `fsync` from checkpoint, truncation, or commit paths causes file corruption on ungraceful shutdown, leading to unrecoverable data loss on restart. The dropped barrier was the precondition for the higher-level atomicity claim (e.g., atomic-rename only guarantees ordering after the source is durable).
|
||||||
|
**Look for:** Diffs that delete `force(true)` / `fsync` / `getFD().sync()` calls in checkpoint, truncation, or commit paths; comments claiming the call was "redundant"; removal of a flush immediately preceding `Files.move(..., ATOMIC_MOVE)`.
|
||||||
|
|
||||||
|
### F-32: Snapshot scheduled at non-aligned offset
|
||||||
|
Scheduling a checkpoint/snapshot at the current position without verifying batch-boundary alignment produces an unreplayable snapshot. Recovery loads a snapshot pointing into the middle of a batch and cannot resume.
|
||||||
|
**Look for:** `snapshotAt(currentOffset)` without a preceding alignment check; snapshot writers that record an offset chosen by a clock or external signal rather than from a known-quiesced batch boundary.
|
||||||
|
|
||||||
|
### F-33: IOException swallowed on file close skips recovery
|
||||||
|
A swallowed `IOException` during file closure prevents error propagation; the runtime skips recovery on next startup and may leave index files in an inconsistent state with respect to the data files they describe. Filesystem errors that should trigger the disk-failure policy are silently logged instead.
|
||||||
|
**Look for:** `try { file.close(); } catch (IOException ignored) {}`; I/O catch blocks that only call `logger.warn(...)` without invoking the project's failure-policy dispatcher.
|
||||||
|
|
||||||
|
### F-34: Truncation resets file size but not companion offset
|
||||||
|
A truncation operation resets the file's physical size but does not update a companion offset-tracking field, leaving the field pointing past the new end of file and causing subsequent reads to access non-existent data.
|
||||||
|
**Look for:** `channel.truncate(newSize)` paired with no update to a sibling `lastValidPosition` / `endOffset` / `tailIndex` field; classes that maintain redundant size state.
|
||||||
|
|
||||||
|
### F-35: Channel fill assumes single read returns full buffer
|
||||||
|
A channel fill reads once and treats the result as the full buffer; short reads silently truncate data for all subsequent accesses from that buffer. Common in helpers misleadingly named `fill()` or `loadFully()`.
|
||||||
|
**Look for:** `channel.read(buf); buf.flip(); deserialize(buf);` with no loop ensuring `buf.remaining() == 0` after the fill; helpers that issue exactly one underlying read.
|
||||||
|
|
@ -0,0 +1,195 @@
|
||||||
|
# Category: Lifecycle and Ordering
|
||||||
|
|
||||||
|
Bugs caused by performing operations in the wrong sequence relative to a component's lifecycle: registering listeners after events fire, exposing services before initialization completes, tearing down dependencies in the wrong order, polling without deadlines, or assuming async work is done when it isn't.
|
||||||
|
|
||||||
|
## Diff signals (when to load this category)
|
||||||
|
|
||||||
|
Load this category if the patch contains ANY of:
|
||||||
|
- New `register*`/`addListener`/`subscribe`/`addObserver` calls or changes to where they are placed
|
||||||
|
- Changes to constructor or `init()`/`start()` ordering, or new initialization phases
|
||||||
|
- Changes to `close()`/`shutdown()`/`stop()` ordering, drain loops, or teardown sequencing
|
||||||
|
- New management endpoint/metrics registration, or unregistration paths
|
||||||
|
- Polling loops (`while (!ready)`, `await*Until*`, retry-on-condition) without explicit deadlines
|
||||||
|
- New asynchronous tasks (`submit`, `schedule`, `Future`, `CompletableFuture`) that produce state observable elsewhere
|
||||||
|
- Conditional readiness flags, "is started/initialized/ready" guards, or status-broadcast methods
|
||||||
|
- Changes to bootstrap/join/handshake/handoff/quarantine sequences
|
||||||
|
- Background thread spawning (especially inside constructors or factory methods)
|
||||||
|
- Static initializers / class-load-time evaluation of configuration or singletons
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### F-01: State reported before listener registered (lost-update window)
|
||||||
|
Component reports its current state to a caller and then registers an event listener; updates that arrive between the snapshot and the registration are silently lost.
|
||||||
|
**Look for:** sequence of `getState()` / `getCurrentX()` followed by `addListener(...)` / `subscribe(...)` on the same source — should be reversed (register first, then snapshot).
|
||||||
|
|
||||||
|
### F-02: Listener registered after async operation already started
|
||||||
|
A handler/listener is wired up after the producer has begun emitting events; events fired in the gap between start and registration are permanently missed.
|
||||||
|
**Look for:** `start()` / `submit()` / `connect()` / `kickOff()` calls that precede the matching `addEventListener(...)` / `onComplete(...)` / `register(...)` for the same component.
|
||||||
|
|
||||||
|
### F-03: Management interface registered at end of long initialization
|
||||||
|
Metrics, REST, or similar management surface is registered in the final step of multi-phase init — operators cannot observe or control the system during the preceding phases.
|
||||||
|
**Look for:** Management endpoint or metrics registry registration calls, or HTTP endpoint binding placed after multiple heavyweight init steps in `start()` / `init()`.
|
||||||
|
|
||||||
|
### F-04: Async-rebuild visibility window (queryable before ready)
|
||||||
|
Component is marked queryable / "available" when the request arrives, but the underlying data structure is still being asynchronously rebuilt or populated, so queries succeed at the routing layer but fail or return empty.
|
||||||
|
**Look for:** `setReady(true)` / status flag set immediately after submitting an async rebuild task, or readiness derived from registration rather than completion.
|
||||||
|
|
||||||
|
### F-05: Startup polling loop without deadline
|
||||||
|
A polling loop waits for an external readiness condition (peer, file, network, downstream service) without a timeout or maximum-attempt guard, hanging forever if upstream never converges.
|
||||||
|
**Look for:** `while (!ready) { sleep(...); }` / `await(...)` / `pollUntil(...)` with no deadline parameter, no maximum iteration count, and no way for the caller to abort.
|
||||||
|
|
||||||
|
### F-06: Consumer torn down before its message source
|
||||||
|
Shutdown sequence stops the message handler/consumer before stopping the producer or transport that feeds it, so in-flight messages arrive at partially-disposed state and produce errors or silent loss.
|
||||||
|
**Look for:** `close()` / `shutdown()` blocks where consumer/handler is stopped first and the inbound channel/queue/socket/listener is stopped after; correct order is usually source-first.
|
||||||
|
|
||||||
|
### F-07: Background thread init failure not signalled to spawning thread
|
||||||
|
A worker thread fails during initialization but the spawning thread is not notified; subsequent shutdown waits indefinitely on the worker that never came up.
|
||||||
|
**Look for:** new `Thread(...)`/`executor.submit(...)` followed by `thread.join()` or latch wait in shutdown without an exception channel back to the spawner.
|
||||||
|
|
||||||
|
### F-08: Background thread started before owner is fully constructed
|
||||||
|
A constructor spawns a thread that immediately reads `this.someField`; if construction has not finished, the thread observes an uninitialized field and crashes or sees stale state.
|
||||||
|
**Look for:** `new Thread(this::run).start()` or `executor.submit(this::work)` inside a constructor body before all field assignments complete.
|
||||||
|
|
||||||
|
### F-09: Liveness/identity announced before component is ready to serve
|
||||||
|
A node or component registers itself as healthy / available before the underlying service (transport, data, dependencies) is ready; clients route requests to it and they fail.
|
||||||
|
**Look for:** Health-check registration, heartbeat publication, or service-discovery enrollment placed earlier in startup than the corresponding bind / data-load / dependency-init step.
|
||||||
|
|
||||||
|
### F-10: Liveness probe replies during shutdown
|
||||||
|
A liveness response handler responds "alive" without checking whether the local node has begun shutting down, causing peers to re-mark the node as alive after a shutdown notification was already broadcast.
|
||||||
|
**Look for:** ping/heartbeat handlers that send a positive reply unconditionally, with no check of the local `isShuttingDown` / `state == STOPPING` flag.
|
||||||
|
|
||||||
|
### F-11: Service-stop ordering leaves a window where node looks down but still accepts work
|
||||||
|
Shutdown stops the liveness/advertising channel before closing client-facing transport, leaving an interval where peers think the node is gone but it still accepts (and then drops) client connections.
|
||||||
|
**Look for:** `gossip.stop()` / `failureDetector.stop()` placed before `clientTransport.close()` in shutdown sequences.
|
||||||
|
|
||||||
|
### F-12: Async cleanup not awaited before next operation proceeds
|
||||||
|
Code triggers an async cleanup/close and immediately starts the next phase, racing against the cleanup that has not yet released resources or state.
|
||||||
|
**Look for:** `closeAsync()` / `removeAsync()` / fire-and-forget delete followed by re-creation or restart of the same identifier in the same method.
|
||||||
|
|
||||||
|
### F-13: Capacity counter decremented before resources fully released
|
||||||
|
A slot/permit counter is decremented to "free" the slot before the associated resources are actually released; a concurrent admitter sees the free slot and submits new work that races the cleanup.
|
||||||
|
**Look for:** `slots.release()` / `count.decrementAndGet()` placed before `resource.close()` / `cleanup()` calls in finally blocks or completion callbacks.
|
||||||
|
|
||||||
|
### F-14: Shutdown drain waits only on queue, not on producers
|
||||||
|
Shutdown waits for a work queue to drain but does not wait for the executor or upstream stage that produces into the queue, racing close against in-flight writes.
|
||||||
|
**Look for:** `queue.awaitEmpty()` / latch-on-queue without a corresponding `producerExecutor.awaitTermination()` or producer-side join in shutdown paths.
|
||||||
|
|
||||||
|
### F-15: Static initializer evaluates config before subsystem configures it
|
||||||
|
A `static final` field is initialized at class-load time by reading from a global singleton or configuration that the caller has not yet populated, freezing the constant at a stale (often default) value for the JVM lifetime.
|
||||||
|
**Look for:** `static final X = SomeService.get*()` / `Config.read(...)` in field initializers; class-load triggers run before the configurer.
|
||||||
|
|
||||||
|
### F-16: Background task scheduled in static initializer
|
||||||
|
A periodic/scheduled task is submitted from a static initializer that runs whenever the class is first loaded; the target executor or runtime may not yet be initialized, causing a startup race or NPE.
|
||||||
|
**Look for:** `executor.scheduleAtFixedRate(...)` / `Timer.schedule(...)` calls in `static {}` blocks or static field initializers.
|
||||||
|
|
||||||
|
### F-17: Lifecycle method runs before initialization completes
|
||||||
|
An `init()`, factory `create()`, or registration step is interleaved so that a `stop()`, callback, or query method can be called between construction and full setup, finding partially-built state and crashing or silently no-opping.
|
||||||
|
**Look for:** an `init()`-set readiness flag inside `run()` (rather than the constructor) — an early `stop()` is silently overwritten when `run()` begins; or a query method that hits a not-yet-initialized field.
|
||||||
|
|
||||||
|
### F-18: Two-stage initialization where the readiness flag is observable too early
|
||||||
|
A flag like `isReady` / `started` is set right after starting an async init, not after init completes; readers see "ready" but the underlying object is still being built.
|
||||||
|
**Look for:** `this.ready = true;` placed after `executor.submit(initTask)` / `service.startAsync()` rather than inside the task's completion callback.
|
||||||
|
|
||||||
|
### F-19: One-shot scheduled task never reschedules itself
|
||||||
|
A periodic refresh job uses a one-shot schedule and updates state once, but never reschedules; the periodic cycle silently stops after the first execution.
|
||||||
|
**Look for:** `scheduler.schedule(task, delay)` (one-shot) where `scheduleAtFixedRate` / `scheduleWithFixedDelay` was intended; or a task body that omits its own re-arm call.
|
||||||
|
|
||||||
|
### F-20: Periodic task gates re-arm on flush/return value rather than time
|
||||||
|
A periodic loop decides whether to re-arm based on the side-effect return of the work it just did instead of always re-arming on a clock; an unusual return value silently halts the schedule.
|
||||||
|
**Look for:** `if (worked) scheduleNext()` patterns at the bottom of scheduled handlers — re-arm should be unconditional or tied to lifecycle state, not work outcome.
|
||||||
|
|
||||||
|
### F-21: Cleanup callback registered before resource is committed
|
||||||
|
A cleanup/close callback is wired up in a constructor before the underlying resource is committed to long-lived storage; if the partially-built object is discarded, the callback is orphaned and cleanup never runs.
|
||||||
|
**Look for:** `addCloseListener(...)` / `onClose(...)` in a constructor that runs before `commit()` / `register(this)` succeeds.
|
||||||
|
|
||||||
|
### F-22: Resource registration after submission to executor (race window)
|
||||||
|
A resource is submitted to an executor and only afterwards registered into a tracking set; a concurrent completion can observe an absent entry, drop cleanup, or allow a duplicate submission.
|
||||||
|
**Look for:** `executor.submit(task)` followed by `tracker.put(task.id, task)` — registration must precede submission or be performed by the task itself before yielding.
|
||||||
|
|
||||||
|
### F-23: Static initializer triggers cyclic class load that deadlocks
|
||||||
|
A static initializer calls a startup method that throws; subsequent class access re-triggers the initializer, which tries to acquire locks already held by the interrupted initialization, deadlocking.
|
||||||
|
**Look for:** `static { ... Service.initialize(); ... }` blocks where `initialize()` can throw and the class is referenced from any thread that may resume before recovery.
|
||||||
|
|
||||||
|
### F-24: Constructor passes `this` to external observers before construction completes
|
||||||
|
A constructor registers `this` with an external listener, registry, or another thread before all fields are assigned, so observers see a partially-constructed object.
|
||||||
|
**Look for:** `registry.register(this)` / `executor.submit(this::work)` / `bus.subscribe(this)` calls inside a constructor body, especially before final field assignments.
|
||||||
|
|
||||||
|
### F-25: Migration / setup path adds new entry before removing old one
|
||||||
|
A migration that replaces one entry with another adds the new entry before deleting the old; a reader observing the intermediate state sees both, causing double-processing or ambiguous lookups.
|
||||||
|
**Look for:** `add(new); remove(old);` ordering in migration helpers — should usually be `remove(old); add(new);` or use atomic replace.
|
||||||
|
|
||||||
|
### F-26: Bootstrap / data transfer skipped when guard inverted
|
||||||
|
A guard around bootstrap (`if (!schemaPresent)`) silently skips the data-transfer phase under conditions the operator did not anticipate, leaving the node joined-but-empty.
|
||||||
|
**Look for:** any `bootstrap()` / `joinRing()` / `loadInitialState()` call gated by a single boolean derived from local-only state — combine with explicit operator intent.
|
||||||
|
|
||||||
|
### F-27: Async write/initialization not awaited before continuation
|
||||||
|
Caller invokes an async write / `submit` / `executeAsync()` and continues without awaiting; the next step assumes the operation has completed, producing wrong results or losing writes.
|
||||||
|
**Look for:** discarded `Future` / `CompletableFuture` return values, especially in initialization, schema-update, or migration code paths.
|
||||||
|
|
||||||
|
### F-28: Initialization step depends on virtual method called before subclass init
|
||||||
|
A base-class constructor calls a virtual / overridable method whose subclass override depends on subclass fields not yet assigned; the call returns stale defaults and a critical step is skipped.
|
||||||
|
**Look for:** virtual method invocations in superclass constructors / `init()` that subclasses override.
|
||||||
|
|
||||||
|
### F-29: Field initialization order — dependent field declared before its dependency
|
||||||
|
A `static` field references another `static` field declared later in the source; initialization order causes the dependent field to read null/zero, often triggering NPE in subsequent calls (e.g., logger usage).
|
||||||
|
**Look for:** static field declarations where one initializer calls a method on another static field declared further down in the file.
|
||||||
|
|
||||||
|
### F-30: Setter triggers side effect that reads not-yet-set field
|
||||||
|
A setter invoked from within a constructor triggers a side effect (notification, recompute) that reads a field assigned later in the same constructor, producing NPE or a wrong derived value.
|
||||||
|
**Look for:** field setters in constructors that fire change-notifications / recomputes; reorder to assign all fields first, then notify.
|
||||||
|
|
||||||
|
### F-31: Termination notifications fire from a state machine that can re-enter
|
||||||
|
A "became active / available" notification fires from a state-machine transition that can occur multiple times (e.g., during a transient resigned state), causing downstream consumers to receive false positives.
|
||||||
|
**Look for:** notify/event-fire calls in state-transition handlers — move to the actual single-shot lifecycle event that fires once on first reachability.
|
||||||
|
|
||||||
|
### F-32: Component dependency initialized after dependent component
|
||||||
|
A dependency is initialized after a dependent component that has already started to use it, so the dependent silently skips checks (e.g., authorization) or operates with default behavior.
|
||||||
|
**Look for:** ordering of `start*()` calls in a top-level bootstrap method where component A depends on component B but B is started later.
|
||||||
|
|
||||||
|
### F-33: Shutdown sequence references field that was never initialized
|
||||||
|
A `close()` / `shutdown()` method dereferences a field that is only assigned if initialization completed successfully; an early-failure shutdown throws NPE that masks the original startup error.
|
||||||
|
**Look for:** unguarded field accesses in `close()` / `cleanup()` paths — guard with `if (field != null)` or initialize fields to safe defaults.
|
||||||
|
|
||||||
|
### F-34: Startup decision made after fixed-duration sleep instead of explicit convergence wait
|
||||||
|
A startup or initialization decision is made after `Thread.sleep(constant)` waiting for external state to converge; if convergence is slower than the sleep, the node acts on stale/incomplete state.
|
||||||
|
**Look for:** `Thread.sleep(N)` in startup paths followed by reading peer/membership/config state — replace with an explicit readiness signal, condition variable, or polling loop with a timeout.
|
||||||
|
|
||||||
|
### F-35: Snapshot taken before write barrier issued (lost durability window)
|
||||||
|
A durability/flush boundary position is sampled before the write barrier or atomic position publish, so concurrent writers can land past the sample yet still be in the flushing batch and lost on truncation.
|
||||||
|
**Look for:** `lastWritePos = currentPos()` / `snapshot()` calls placed before the corresponding `barrier()` / `fence()` / atomic publish.
|
||||||
|
|
||||||
|
### F-36: Liveness arrival not reported on first observation
|
||||||
|
A failure-detector or arrival-reporter is bypassed on an endpoint's very first observation, so the detector starts with no samples and permanently considers the endpoint unreachable.
|
||||||
|
**Look for:** First-observation paths that skip the `reportArrival()` / `recordSample()` call that the steady-state path performs.
|
||||||
|
|
||||||
|
### F-37: Listener removed on first response when concurrent in-flight requests share it
|
||||||
|
A listener / session is unregistered on the first response, but concurrent in-flight requests share the same listener and lose their reference before completion, silently dropping later events.
|
||||||
|
**Look for:** `tracker.remove(id)` inside a single-response callback when more than one outstanding request can map to the same `id`.
|
||||||
|
|
||||||
|
### F-38: Termination flag and queue cleared in non-atomic steps
|
||||||
|
A termination handler sets a "stopped" flag and clears a work queue in separate steps; a message received between the two causes an assertion failure or work submission to a half-shut-down state.
|
||||||
|
**Look for:** `this.stopped = true;` followed (or preceded) by `queue.clear()` in shutdown paths without a single critical section guarding both.
|
||||||
|
|
||||||
|
### F-39: Worker pool deadlocks because submit blocks waiting on its own continuation
|
||||||
|
A bounded executor's tasks submit further work back to the same executor; under contention, all worker threads are blocked waiting for queue space while the only thing that can free slots is themselves.
|
||||||
|
**Look for:** `executor.submit(...)` calls made from inside tasks already running on the same executor — refactor to a separate executor or use a non-blocking handoff.
|
||||||
|
|
||||||
|
### F-40: Blocking call inside a callback that runs on the only thread that can complete it
|
||||||
|
A callback synchronously waits on a future whose completion requires the same single-threaded stage that's running the callback, deadlocking the stage.
|
||||||
|
**Look for:** `future.get()` / `await()` calls inside completion handlers, RPC handlers, or single-threaded stage callbacks.
|
||||||
|
|
||||||
|
### F-41: Subscription state mutated by unintended thread
|
||||||
|
Shared subscription / membership state is mutated on the calling (application) thread while a background thread expects to own it, producing torn reads and inconsistent assignments between calls.
|
||||||
|
**Look for:** direct mutation of subscription / assignment / state-machine fields from public API methods rather than enqueueing the change to the owning thread.
|
||||||
|
|
||||||
|
### F-42: Init readiness check uses stale proxy rather than authoritative signal
|
||||||
|
A "ready" condition is derived from a proxy (e.g., presence of pending entries) rather than directly comparing committed-vs-applied counters, releasing dependent operations before the system is truly ready.
|
||||||
|
**Look for:** readiness checks that test indirect side-effects (queue size, presence flag) instead of the canonical "X applied through Y" condition.
|
||||||
|
|
||||||
|
### F-43: Background scheduling fires before required state is established
|
||||||
|
An initialization task is gated by a fixed-duration timer rather than an explicit lifecycle event, so it can fire before required state (e.g., topology, schema) is established.
|
||||||
|
**Look for:** `scheduleOnce(initTask, delay)` patterns where the delay is hoping a separate prerequisite will have completed; use explicit dependency / ready-event instead.
|
||||||
|
|
||||||
|
### F-44: Application reports ready/running before async setup completes
|
||||||
|
A service publishes its `running` / `ready` status before the asynchronous setup it spawned completes, giving callers a false signal of initialization completion.
|
||||||
|
**Look for:** `ready.complete()` / `setStatus(RUNNING)` calls placed immediately after `submitAsync(setup)` rather than inside the setup's completion callback.
|
||||||
|
|
@ -0,0 +1,166 @@
|
||||||
|
# Category: Null and Type Safety
|
||||||
|
|
||||||
|
Bugs where null-valued fields, return values, or unset optional fields are dereferenced without a guard, and bugs where a runtime type assumption (cast, instanceof, generic narrowing) does not hold for all values that actually flow through the code path.
|
||||||
|
|
||||||
|
## Diff signals (when to load this category)
|
||||||
|
|
||||||
|
Load this category if the patch contains ANY of:
|
||||||
|
- A new `cast`, `(SomeType) x`, `instanceof`, or generics narrowing on a value that crosses an API boundary or comes from a polymorphic source.
|
||||||
|
- A new lookup result (`map.get`, `registry.lookup`, `find`, schema/metadata lookup, `socket.getChannel`, `File.list`) chained or dereferenced without a null guard.
|
||||||
|
- A new optional / nullable field added to a config, schema, protocol record, or wire format (especially "introduce nullable union", "make field optional", "schema evolution").
|
||||||
|
- New `Optional` usage (`.get()`, `.orElse(null)`, `orElseThrow`, eager-evaluated `orElse(...)` arguments).
|
||||||
|
- Changes to constructors / lifecycle (`close()`, `cleanup()`, `shutdown()`) where a field may be null on early-failure or never-initialized paths.
|
||||||
|
- Changes that move a null check, change `&&`/`||` connecting null-checks, or add/remove `containsKey` before `get`.
|
||||||
|
- Refactor that widens a return type to a supertype or moves a method onto a more abstract type while existing callers/fields keep the concrete type.
|
||||||
|
- New `toArray()`, `Collectors.toMap`, auto-unboxing of `Map.get`, or `Long`/`Integer` reference-equality (`==`) comparisons.
|
||||||
|
- A new factory or builder whose return type is wider/narrower than callers assume.
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### F-01: Method on field that can be absent from config file
|
||||||
|
A method (`length`, `toString`, `iterator`) is called directly on a configuration field that is null when the user omits it from the config file. Some call sites null-check the field but others do not.
|
||||||
|
**Look for:** `config.someField.xxx()` where `someField` has no default initializer in the config class — search the class for any sibling site that does `if (config.someField != null)`.
|
||||||
|
|
||||||
|
### F-02: Map.get / registry lookup result dereferenced without null check
|
||||||
|
Result of a `map.get(...)` or registry/metadata/schema lookup is chained or passed to a downstream method that immediately dereferences it; the lookup can legitimately return null for unknown, removed, or not-yet-initialized keys.
|
||||||
|
**Look for:** `something.lookup(x).field` or `map.get(k).method(...)` with no `if (... != null)` between them; especially in startup, recovery, streaming, and topology paths.
|
||||||
|
|
||||||
|
### F-03: Field nulled between check and use (DCL / TOCTOU)
|
||||||
|
A nullable field is read into a local with a null check, then re-read directly (instead of the local) for the dereference, allowing a concurrent writer to clear it between reads. Variant: `volatile Optional` checked with `isPresent()` then `get()`.
|
||||||
|
**Look for:** `if (this.x != null) this.x.foo()` (rather than `var x = this.x; if (x != null) x.foo()`); also `opt.isPresent() && ... opt.get()` on volatile / shared fields.
|
||||||
|
|
||||||
|
### F-04: Null guard's closing brace misplaced — dependent code falls outside
|
||||||
|
A method call requiring the guarded value to be non-null sits one line below the closing brace of the `if (x != null) { ... }` block, executing unconditionally.
|
||||||
|
**Look for:** Statement immediately after a null-guarded block that still references the guarded variable; review brace alignment in any patch that touches a null guard.
|
||||||
|
|
||||||
|
### F-05: Factory return value cast to unsupported subtype
|
||||||
|
A factory method's return is unconditionally cast to a specific subtype, but at runtime the factory may return a different implementation (e.g., a different replication strategy, partitioner, or visitor-variant subclass).
|
||||||
|
**Look for:** `(ConcreteSubtype) factory.create(...)` or `(ConcreteSubtype) field` where `field`/return type is the abstract supertype; check whether other implementations exist in the same hierarchy.
|
||||||
|
|
||||||
|
### F-06: Range / value constructed with concrete key type where abstract position type expected
|
||||||
|
A constructor for a range or container is invoked with a concrete key type, and a downstream cast expects the abstract position/bound type. The runtime cast throws on the concrete value.
|
||||||
|
**Look for:** `new Range<ConcreteKey>(...)` flowing into code that does `(Position) range.left` or similar; verify generic parameter narrowing is consistent end-to-end.
|
||||||
|
|
||||||
|
### F-07: Array field declared with concrete type post-refactor expects abstraction
|
||||||
|
An array or collection field declared with a concrete element type was valid before a refactor; afterwards code expects a broader abstract element type and casts fail on the legacy concrete values.
|
||||||
|
**Look for:** `ConcreteImpl[] field` with usage code that does `(AbstractType) field[i]` or stores values via an abstract-typed setter.
|
||||||
|
|
||||||
|
### F-08: Nullable union/optional field deserialized without null check
|
||||||
|
A schema-evolved record's nullable union (or thrift/protobuf optional) field is dereferenced (often `.toString()` or method call) when older senders omit it, throwing NPE on receipt.
|
||||||
|
**Look for:** Newly-introduced `nullableVersions`, optional thrift field, or Avro union with null; any read site that calls a method on the field without `isSet`/null check.
|
||||||
|
|
||||||
|
### F-09: Avro union field: null default requires null as first union member
|
||||||
|
An Avro union field declared with a `null` default has the non-null type listed first; the Avro spec requires `null` to be the first member when it is the default, so deserialization fails.
|
||||||
|
**Look for:** Avro schema definitions where the union is `["string", "null"]` with `"default": null`; the correct form is `["null", "string"]`.
|
||||||
|
|
||||||
|
### F-10: Conditionally-initialized field guarded only by the original flag
|
||||||
|
Fields initialized only when a feature flag is true are dereferenced later guarded only by the same flag. Toggling the flag at runtime (or hot reload) leaves fields null while the flag is false.
|
||||||
|
**Look for:** `if (feature) field.method()` where `field = ...` only inside a parallel `if (feature)` constructor branch; check if the flag is dynamically toggleable.
|
||||||
|
|
||||||
|
### F-11: Lazy-initialized field accessed concurrently without synchronization
|
||||||
|
Lazy-initialized fields are accessed on multiple threads without happens-before; one thread observes a null intermediate state during another's initialization.
|
||||||
|
**Look for:** Non-volatile field with single-checked lazy init; or `compareAndSet` initialization where readers do not retry.
|
||||||
|
|
||||||
|
### F-12: Close() / cleanup() dereferences uninitialized fields after partial construction
|
||||||
|
A constructor throws midway, leaving some fields null; the `close()` / `cleanup()` / `shutdown()` path dereferences them unconditionally, throwing NPE that masks the original error.
|
||||||
|
**Look for:** Field assignments late in the constructor; matching `close()` that calls `field.close()` without `if (field != null)`.
|
||||||
|
|
||||||
|
### F-13: Lazy resource initialized only when data arrives, close called unconditionally
|
||||||
|
A resource is allocated only when at least one record is processed; `close()` always calls a method on it. Empty input → NPE in close.
|
||||||
|
**Look for:** Field assignment guarded by data presence (`if (firstBatch) { field = ... }`); paired close that does not guard.
|
||||||
|
|
||||||
|
### F-14: Auto-unboxing Map.get returning null
|
||||||
|
A primitive variable receives `map.get(key)` for a `Map<K, Long>`/`<K, Double>`; absent key returns null and unboxing throws NPE. Often paired with arithmetic (`/=`, `+`).
|
||||||
|
**Look for:** `long x = someMap.get(...)`, `int y = otherMap.get(...)`; switch to `getOrDefault` or `containsKey` guard.
|
||||||
|
|
||||||
|
### F-15: Collectors.toMap with null values
|
||||||
|
`Collectors.toMap` rejects null values with NPE; legitimate "no value yet" entries crash the collection.
|
||||||
|
**Look for:** `.collect(Collectors.toMap(...))` where the value mapper can return null; replace with `HashMap` + `put`.
|
||||||
|
|
||||||
|
### F-16: Optional.orElse with side-effect-bearing argument
|
||||||
|
`Optional.orElse(expensiveCall())` evaluates the argument eagerly even when the Optional is present, causing double processing or unintended side effects.
|
||||||
|
**Look for:** `.orElse(...)` where the argument calls a constructor, fetcher, or any non-trivial expression; prefer `orElseGet`.
|
||||||
|
|
||||||
|
### F-17: Optional.get() without isPresent
|
||||||
|
`.get()` called on an Optional without `isPresent()` (or via `orElse(null)` followed by an unguarded dereference). Common with `Optional`-wrapped timeouts.
|
||||||
|
**Look for:** `.get()` on Optional or `orElse(null)` and immediate `.method()`.
|
||||||
|
|
||||||
|
### F-18: Polymorphic loop with unconditional downcast
|
||||||
|
A loop iterates a heterogeneous collection and casts each element to one specific subtype; introducing a second registered subtype throws ClassCastException at runtime.
|
||||||
|
**Look for:** `for (Item it : items) { ((Concrete) it).foo() }` where `items` element type is a sealed/sub-typed interface; verify all subtype instances are handled.
|
||||||
|
|
||||||
|
### F-19: Downcast before instanceof guard
|
||||||
|
A type-narrowing downcast is performed before the type-guard condition that would justify it; a non-matching object throws CCE before the guard runs.
|
||||||
|
**Look for:** `Concrete c = (Concrete) x; if (x instanceof Concrete) { ... }` — reorder to test first.
|
||||||
|
|
||||||
|
### F-20: instanceof tests parent type, cast targets subtype
|
||||||
|
`instanceof Parent` guards a cast `(Subtype) x`; passing a Parent that is not the Subtype satisfies the guard but throws CCE on cast.
|
||||||
|
**Look for:** Mismatch between the type in `instanceof` and the type in the subsequent cast.
|
||||||
|
|
||||||
|
### F-21: Collection.toArray() returns Object[] cast to typed array
|
||||||
|
`stream.toArray()` (no-arg) or `Collection.toArray()` returns `Object[]`; casting to `T[]` throws CCE at runtime.
|
||||||
|
**Look for:** `(T[]) coll.toArray()` or `(T[]) stream.toArray()` — should use `toArray(T[]::new)` or pass a typed array.
|
||||||
|
|
||||||
|
### F-22: Reference equality on boxed numeric types
|
||||||
|
`==` comparison between two `Long`/`Integer` boxes (e.g., from autoboxing or `Map.get`) holds only inside the JVM integer cache range; outside that range every comparison is false.
|
||||||
|
**Look for:** `==` between two non-primitive numeric typed expressions; replace with `.equals()`.
|
||||||
|
|
||||||
|
### F-23: TreeSet / sorted collection over non-Comparable elements
|
||||||
|
Constructing a `TreeSet` / `TreeMap` (or any sorted structure) without a comparator over elements that don't implement `Comparable` compiles but throws CCE on the second insertion. Variant: passing a comparator from one type hierarchy to a container of another.
|
||||||
|
**Look for:** `new TreeSet()` / `new TreeMap()` without a comparator argument and elements lacking a known Comparable contract.
|
||||||
|
|
||||||
|
### F-24: Decorator wrapper not unwrapped before instanceof / cast
|
||||||
|
`instanceof` or cast applied directly to a wrapped/decorated type (e.g., reversed-order column type, frozen-collection wrapper) without first unwrapping; check returns false or cast throws.
|
||||||
|
**Look for:** `x instanceof Concrete` or `(Concrete) x` on a value known to be wrapped by a `ReversedType`/`FrozenType`/decorator pattern.
|
||||||
|
|
||||||
|
### F-25: Filesystem / Socket API returning null treated as collection
|
||||||
|
`File.listFiles`, `File.list`, `Path.toFile().list`, or `Socket.getChannel()` returns null on error or non-NIO socket; result is iterated with for-each or passed to a constructor that does not accept null.
|
||||||
|
**Look for:** `for (File f : dir.listFiles())` or `new X(socket.getChannel())` with no null guard.
|
||||||
|
|
||||||
|
### F-26: Lookup that may return null fed into addSuppressed / collection that rejects null
|
||||||
|
A nullable lookup result is added to a futures list, suppressed-exception chain, or null-rejecting collection; iteration or aggregation later NPEs.
|
||||||
|
**Look for:** `futures.add(maybeNull)` followed by `futures.forEach(f -> f.get())`; `addSuppressed(null)`.
|
||||||
|
|
||||||
|
### F-27: Field marked transient with no readObject reconstruction
|
||||||
|
A `Serializable` class marks a field `transient` but has no custom `readObject` — the field is silently null after deserialization, causing NPEs at runtime.
|
||||||
|
**Look for:** `transient` modifier on a field used unconditionally after deserialization; check for missing `readObject`.
|
||||||
|
|
||||||
|
### F-28: Static field used in earlier static initializer
|
||||||
|
A static field referenced inside another static field's initializer is declared after it; JVM static initialization order leaves it null when the dependent initializer runs.
|
||||||
|
**Look for:** `private static final X X1 = X2.something()` where `X2` is declared lower in the same class — reorder, or initialize in `<clinit>`.
|
||||||
|
|
||||||
|
### F-29: Singleton accessed during early startup before populated
|
||||||
|
A static `current()` singleton is invoked during early-startup code that runs before the singleton is set up; `current()` returns null and dereferences NPE.
|
||||||
|
**Look for:** `Foo.current().something()` invoked from constructors, class loading, or pre-initialization paths.
|
||||||
|
|
||||||
|
### F-30: Empty buffer / zero-length array deserialized without guard
|
||||||
|
A deserializer constructs a value type from a raw byte array without checking emptiness; the constructor throws on an empty array. Variant: `toString()` not null-guarded and NPEs when deserialize returns null.
|
||||||
|
**Look for:** `new ValueType(bytes)` or `deserialize(buf)` followed by unguarded method call; verify empty/zero-length input is handled.
|
||||||
|
|
||||||
|
### F-31: Missing override after method signature change drifts subclass to dead code
|
||||||
|
A subclass method becomes unreachable when the superclass signature changes because no `@Override` annotation flagged the drift; runtime calls dispatch to the wrong implementation, often producing null returns or wrong types downstream.
|
||||||
|
**Look for:** New parameter or different return type on a base method; subclass implementations missing `@Override`; verify subclass signature matches.
|
||||||
|
|
||||||
|
### F-32: Visitor / dispatcher missing case for new variant
|
||||||
|
A visitor or switch over a polymorphic/enum hierarchy adds a new variant in one place but a sibling visitor/handler returns null or falls through to a default that crashes the caller.
|
||||||
|
**Look for:** New enum constant or new `Type` subclass; search for switches/visitors handling its siblings and check the new value is covered.
|
||||||
|
|
||||||
|
### F-33: Map keyed by wrapper type, queried with byte[] / different type
|
||||||
|
A map keyed on a wrapper type is queried with a raw byte[] / different wrapper; lookup uses reference equality and always returns null. Variant: `Map.get(Object)` accepts wrong-type silently.
|
||||||
|
**Look for:** `map.get(rawBytes)` where map's key type is `ByteBuffer`/`Wrapper`; `Map.get` of a value not of K's type.
|
||||||
|
|
||||||
|
### F-34: Generic type parameter narrowed by inference, breaking metric / mock layer
|
||||||
|
A lambda registered without explicit generic witness is inferred as a wider wrapper type; downstream consumer boxes it incorrectly and throws CCE. Variant: mock serializer typed `byte[]` versus interceptor expecting `String`.
|
||||||
|
**Look for:** `register(x -> someValue)` where `x` is a generic and the inferred type is wider than callers expect; consider explicit type witness `register(this.<Long>foo(...))`.
|
||||||
|
|
||||||
|
### F-35: Empty-collection vs null-collection conflated
|
||||||
|
Some callers treat empty and null as equivalent absence; others distinguish them. A null sneaks into code that checks emptiness via `.isEmpty()` and NPEs, or vice versa. Variant: cache returns null for absent vs empty for "no data", and downstream cache treats them inconsistently.
|
||||||
|
**Look for:** `isEmpty()` calls on values that other call sites null-check; conflicting documentation about null vs empty.
|
||||||
|
|
||||||
|
### F-36: Cast inside a sort comparator on sparse / null cell values
|
||||||
|
A sort comparator dereferences cell values without null guard; sparse rows trigger NPE.
|
||||||
|
**Look for:** `comparator.compare(a, b)` body that calls `.field()` on `a` or `b` without checking null; especially in `Collections.sort` of objects with optional fields.
|
||||||
|
|
||||||
|
### F-37: equals() / hashCode() omits new field after refactor
|
||||||
|
Equality comparison ignores a recently-added field, so two objects differing only in that field compare equal. Variant: equals checks only non-null instead of correct type.
|
||||||
|
**Look for:** `equals` / `hashCode` methods that have not been updated after a field added; compare against the constructor field list.
|
||||||
|
|
@ -0,0 +1,119 @@
|
||||||
|
# Category: Refactor Aftermath
|
||||||
|
|
||||||
|
Bugs whose root cause is incomplete or inconsistent refactoring — leftover guards, stale references after a rename, narrowed/broadened types not propagated to all sites, dead branches preserved past their owning feature, deprecated overloads with stale defaults, mixed-version mismatches between the old and new shape, and parsers/regexes that handle only the new format.
|
||||||
|
|
||||||
|
## Diff signals (when to load this category)
|
||||||
|
|
||||||
|
Load this category if the patch contains ANY of:
|
||||||
|
- A symbol rename, method rename, or package move where some call sites are still updated by hand.
|
||||||
|
- A new overload added next to an existing one (especially if the old one is `@Deprecated` or kept as a shim).
|
||||||
|
- A type narrowing or broadening (e.g., concrete → interface, primitive → wrapper, scalar → collection) on a public field/parameter/return type.
|
||||||
|
- A new parser, regex, or format reader that replaces a hand-rolled one (especially for paths, keys, identifiers, or wire formats).
|
||||||
|
- Removal of a feature flag, config option, or guard, with surrounding setup/teardown left intact.
|
||||||
|
- A serialization/wire-format version bump, new field added to one side only, or a digest/identity computation that changed shape.
|
||||||
|
- Lambdas added inside hot loops where the surrounding class already exposes a context object.
|
||||||
|
- A call site where the new code is structurally similar to a sibling/copy in another module that was the original target of the fix.
|
||||||
|
- An `equals`/`hashCode`/comparator touched after fields are added or renamed.
|
||||||
|
- Tests, callers, or override hooks that still reference an old method name or old field set.
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### F-01: Guard removed but teardown left unconditional
|
||||||
|
A refactor removes an option-flag or precondition guard from a costly operation, but the matching teardown/cleanup step remains unconditional and now runs even when the guarded operation never executed.
|
||||||
|
**Look for:** A removed `if (flag)` or `if (enabled)` around a setup block whose paired `close`/`release`/`reset`/`undo` is still outside any guard.
|
||||||
|
|
||||||
|
### F-02: New parser/regex requires field only present in modern format
|
||||||
|
A re-implemented path/key/identifier parser uses a regex or grammar that treats a field present only in the new format as required, silently skipping or rejecting all legacy inputs that omit that field.
|
||||||
|
**Look for:** New `Pattern.compile(...)`, split-on-delimiter loops, or grammar rules introduced alongside an existing format; check whether the old format had fewer/optional components.
|
||||||
|
|
||||||
|
### F-03: Stale call sites still reference old API path/name
|
||||||
|
After a method, accessor, or predicate is moved, renamed, or replaced, some call sites continue to compile against the old symbol (often via leftover imports or static helpers), silently invoking the obsolete behavior.
|
||||||
|
**Look for:** Two coexisting symbols with overlapping names (e.g., `getX`/`getXValue`, `Foo.bar`/`FooHelper.bar`); grep for the old name and check that every hit has been updated.
|
||||||
|
|
||||||
|
### F-04: Deprecated overload never populates the parameter the new one carries
|
||||||
|
A deprecated overload kept for binary compatibility hardcodes a sentinel/null/default for a parameter that the new overload requires callers to supply, so the operation completes against an empty/missing input and silently no-ops.
|
||||||
|
**Look for:** `@Deprecated` overloads that delegate to the new overload while passing `null`, `0`, `EMPTY`, or `Range.empty()`.
|
||||||
|
|
||||||
|
### F-05: `equals`/`hashCode` not updated for new fields
|
||||||
|
After fields are added or renamed, the equality/hash methods are only partially updated (or not updated at all), so logically distinct objects compare equal, hash collisions appear, or change-notification logic stops firing.
|
||||||
|
**Look for:** New fields in a class whose `equals`/`hashCode`/comparator method body does not mention them; also fields that were removed but still referenced inside `equals`.
|
||||||
|
|
||||||
|
### F-06: Mixed-version digest computed over structurally different fields
|
||||||
|
Two software versions independently compute an identity digest/hash/checksum; after one side adds, removes, or reorders the fields, hard equality between the two digests permanently fails during rolling deployments.
|
||||||
|
**Look for:** Digest functions over a field list that differs from the on-the-wire representation, or that includes fields one side omits (e.g., tombstones, optional metadata).
|
||||||
|
|
||||||
|
### F-07: Lambda in hot loop captures locals when a context object exists
|
||||||
|
A lambda introduced in a refactor captures multiple locals and is passed into a tight loop, forcing per-iteration heap allocation, when the surrounding class already exposes a context object that could be threaded through instead.
|
||||||
|
**Look for:** New lambdas inside `for`/`while` bodies that close over more than one local variable; check whether a `this`-style context object is available.
|
||||||
|
|
||||||
|
### F-08: Concrete-typed field/array now expects broadened abstraction
|
||||||
|
A field, array, or parameter declared with the original concrete type was valid pre-refactor; after the refactor it is expected to hold a broader abstraction, so still-extant concrete values fail the runtime cast or `ArrayStoreException`.
|
||||||
|
**Look for:** `T[]` arrays filled by methods returning `Supertype`; downcasts immediately after lookups whose return type was widened.
|
||||||
|
|
||||||
|
### F-09: Subclass override silently disconnected after parent signature drift
|
||||||
|
A parent class changes a method signature/name during refactor; subclasses retain the old name/signature without `@Override`, so the intended override becomes unreachable dead code and the inherited base method runs instead.
|
||||||
|
**Look for:** Override-style methods missing `@Override`; matching method names in a hierarchy where the parent was recently touched.
|
||||||
|
|
||||||
|
### F-10: Old field still set in some constructors but not all after split
|
||||||
|
A monolithic constructor split into multiple paths leaves one path setting a flag/field that another no longer touches; later code unconditionally consults that field and observes a stale or default value.
|
||||||
|
**Look for:** Two constructors with divergent field-assignment lists; init flags toggled in only one branch after a method-extraction refactor.
|
||||||
|
|
||||||
|
### F-11: Static config cached at class load freezes pre-refactor value
|
||||||
|
A static-final cache of a configuration value or singleton handle is computed at class load before later refactored code repopulates the source; the cache freezes a stale/uninitialized value that ignores all later updates.
|
||||||
|
**Look for:** `static final` fields initialized from a global registry/config; check whether the registry is now populated lazily or by a startup phase that may run later.
|
||||||
|
|
||||||
|
### F-12: Cache or fallback never populated after producer was removed
|
||||||
|
A refactor removes the code that populated a fallback cache, derived view, or secondary lookup; remaining consumers still query it and silently receive default/empty/null answers.
|
||||||
|
**Look for:** Code that reads from a structure whose only known producer was deleted in the same change; `containsKey` checks that always return false.
|
||||||
|
|
||||||
|
### F-13: Old-format end-marker / sentinel skipped by new reader
|
||||||
|
A rewritten deserializer or scanner does not recognize an end-of-section marker, sentinel, or framing byte that was valid in the legacy format, so valid records following the marker are silently skipped or misframed.
|
||||||
|
**Look for:** New deserializers that only handle the current version's framing; legacy formats with EOF/end-of-record markers not enumerated in the new switch.
|
||||||
|
|
||||||
|
### F-14: New variant added to enum/type but old switch/dispatch not updated
|
||||||
|
A new constant or subtype is added but a sibling switch, dispatch table, or instanceof chain is not extended, so the new case falls through to the default (often throwing or silently producing wrong output).
|
||||||
|
**Look for:** Recently-added enum constants or subclasses; grep for switches over the enum/base type and confirm every case is handled, especially in serialization and routing.
|
||||||
|
|
||||||
|
### F-15: Wire-format field added on one side only
|
||||||
|
A protocol field is read by the new version but never written by the old, or written by the new and not handled by the old; mixed-version peers misframe everything after the missing/extra field.
|
||||||
|
**Look for:** Symmetric serialize/deserialize pairs where one side gained a field guarded only by a single-sided version check; missing `nullableVersions`/`ignorable` annotations.
|
||||||
|
|
||||||
|
### F-16: Hard assertion / `UnsupportedOperationException` for legacy data
|
||||||
|
After a format upgrade, a deserializer or comparator throws unconditionally on legacy inputs that the prior implementation handled, crashing reads of pre-upgrade data instead of falling back.
|
||||||
|
**Look for:** Newly-added `assert version == CURRENT`, `throw new UnsupportedOperationException`, or strict `instanceof` checks at deserialization entry points.
|
||||||
|
|
||||||
|
### F-17: Removed feature's flag still consulted by surviving code
|
||||||
|
A feature removed in a refactor leaves its enable/disable flag, mode constant, or branch condition behind; remaining code paths consult the flag and route into now-unreachable or wrong behavior.
|
||||||
|
**Look for:** Boolean fields, enum constants, or string mode names whose only writers were deleted; conditions that can never become true (or never false).
|
||||||
|
|
||||||
|
### F-18: New parameter hardcoded at internal call site
|
||||||
|
A method gains a new parameter; an internal call site (or refactored shim) passes a fixed literal instead of forwarding the caller's value, silently overriding per-call configuration with a global default.
|
||||||
|
**Look for:** Recently-added parameters where some call sites pass `null`, `false`, `0`, or `Version.LATEST`; bridge/adapter methods that omit forwarding the new field.
|
||||||
|
|
||||||
|
### F-19: Bridge `default` method silently keeps old implementations alive
|
||||||
|
An interface method gets a `default` implementation as part of a bridging refactor; subclasses that should have been forced to update silently inherit the bridge and never adopt the new behavior.
|
||||||
|
**Look for:** New `default` methods on interfaces with many implementers; combined with a "deprecated" sibling whose behavior the default delegates to.
|
||||||
|
|
||||||
|
### F-20: Type wrapped in richer wrapper but call sites still compare raw
|
||||||
|
A primitive or simple type is wrapped in a richer container during refactor; call sites that previously compared with `==` or `.equals()` against the raw form silently never match, causing map lookups to miss and conditions to never fire.
|
||||||
|
**Look for:** Recently introduced wrapper types (e.g., `Token`, `Identifier`, `Version`); grep for raw-type comparisons against the wrapper.
|
||||||
|
|
||||||
|
### F-21: Field omitted from copy/clone after being added to source
|
||||||
|
A new field added to a class is not added to the matching copy/clone/builder/`with`-style helper, so derived objects silently lose the field even though the source retains it.
|
||||||
|
**Look for:** `copy()`, `clone()`, `toBuilder()`, `merge()` whose body lists fields explicitly; new fields not yet enumerated there.
|
||||||
|
|
||||||
|
### F-22: Existing serializer's size method stale after format change
|
||||||
|
A serializer's body is updated to write a new field or shape, but the matching `serializedSize`/length-prefix method is not, so framing/length-prefix values diverge from actual bytes written.
|
||||||
|
**Look for:** `serialize`/`serializedSize` pairs touched in the same diff where one method gained a field the other did not.
|
||||||
|
|
||||||
|
### F-23: Refactor accidentally removes a single property-assignment line
|
||||||
|
A reorganization accidentally drops a single `this.x = arg.x` (or equivalent setter call), so a user-configured value is silently replaced with the field's default initializer for the lifetime of the object.
|
||||||
|
**Look for:** Constructors and copy methods touched in the same diff where one previously-assigned field is no longer mentioned; check builder chains for missing forwards.
|
||||||
|
|
||||||
|
### F-24: Side-effect setter no longer invoked from new factory path
|
||||||
|
After splitting a single creation path into multiple paths (factory + constructor + builder), one path skips a required side-effect call (registration, listener attach, default validator install), so objects produced via that path are missing initialization others rely on.
|
||||||
|
**Look for:** Multiple creation paths producing the same type; check that each calls every step the original monolithic code did.
|
||||||
|
|
||||||
|
### F-25: Stale comparator/sort key after underlying fields changed shape
|
||||||
|
A sort comparator continues reading a field via the pre-refactor accessor (e.g., raw byte form, ordinal, or stale name), so ordering is invariant or wrong even though the data has moved to a new representation.
|
||||||
|
**Look for:** Comparators that read fields directly rather than via accessors that were updated; comparators bound to a no-longer-implemented interface.
|
||||||
|
|
@ -0,0 +1,195 @@
|
||||||
|
# Category: Serialization and Versioning
|
||||||
|
|
||||||
|
Bugs in encoding/decoding symmetry, wire and on-disk formats, version-gated paths, schema evolution, mixed-version compatibility, and switch- or enum-based decode dispatch.
|
||||||
|
|
||||||
|
## Diff signals (when to load this category)
|
||||||
|
|
||||||
|
Load this category if the patch contains ANY of:
|
||||||
|
- A `serialize`, `deserialize`, `serializedSize`, `read`, or `write` method, or any pair of these together
|
||||||
|
- A `switch` or `if/else if` ladder that dispatches on a type tag, enum, or kind discriminator
|
||||||
|
- A version comparison: `if (version >= ...)`, `protocolVersion`, `apiVersion`, `formatVersion`, `MessagingService.VERSION_*`, or any version-gated branch
|
||||||
|
- An enum used as a wire-protocol or on-disk discriminator (especially with explicit ordinal/code values, or `values()[i]` decoding)
|
||||||
|
- A `ByteBuffer` operation: `array()`, `arrayOffset()`, `position()`, `slice()`, `duplicate()`, `flip()`, `rewind()`, or any relative read/write
|
||||||
|
- A length prefix (`writeInt`, `writeShort`, `writeUnsignedVInt`) followed/preceded by a payload write/read
|
||||||
|
- A digest or checksum computation over fields, buffers, or sub-records (`MessageDigest.update`, `XXHash`, `CRC32`)
|
||||||
|
- Schema-evolution markers: nullable→non-nullable changes, new fields added to records, `taggedFields`, `nullableVersions`, `flexibleVersions`, `ignorable`
|
||||||
|
- Auto-generated message class changes (Avro, Thrift, Kafka JSON message protocol, Protobuf)
|
||||||
|
- A "compatibility", "legacy", "pre-X.Y", or "old format" code path
|
||||||
|
- Header / context / metadata propagation through wrapper serializers (`Headers headers` arg added or removed)
|
||||||
|
- `getBytes()` / `String.getBytes()` / charset-related calls in a serialization context
|
||||||
|
- Charset, locale, or byte-order assumptions (`UTF_8`, `Locale.getDefault`, `ByteOrder.nativeOrder`)
|
||||||
|
- A deserializer constructor that takes raw bytes plus a type tag, kind flag, or context object
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### F-01: Conditional write paired with unconditional read
|
||||||
|
A serializer writes a field only when present (e.g. behind an isSet flag) while the deserializer reads the field unconditionally, so when the field is absent the reader consumes bytes belonging to the next field.
|
||||||
|
**Look for:** `if (cond) out.writeXxx(...)` in serialize without a matching `if (cond)` guard in deserialize, or vice versa.
|
||||||
|
|
||||||
|
### F-02: Switch-case field bleeds across decoded variants
|
||||||
|
In a switch- or kind-based deserialization path, a field from a prior decoded value bleeds into the current object because it is not explicitly cleared when the variant indicates it should be absent.
|
||||||
|
**Look for:** A `switch(kind)` in deserialize where some branches assign a field and others do not, with no `obj.field = null` reset before dispatch.
|
||||||
|
|
||||||
|
### F-03: serializedSize disagrees with serialize
|
||||||
|
The `serializedSize` method omits a field, double-counts a field, uses the wrong width, or hits a different conditional branch than `serialize`, causing buffer underflow/overrun or framing errors.
|
||||||
|
**Look for:** Pair `serializedSize` and `serialize` side-by-side; check field count, conditional branches, and variable-length sizing helpers (`sizeof`, `sizeOfUnsignedVInt`).
|
||||||
|
|
||||||
|
### F-04: Round-trip not closed (write/read/size asymmetry)
|
||||||
|
A field is written but never read, read but never written, or read in a different order than it is written; bytes after the first asymmetry decode at wrong offsets and the round-trip silently drops or corrupts data.
|
||||||
|
**Look for:** A diff that touches one of `serialize`/`deserialize`/`serializedSize` but not all three; sequential `writeX(a); writeY(b)` not mirrored verbatim by `readX(); readY()` on the other side.
|
||||||
|
|
||||||
|
### F-05: Length-prefix width mismatch
|
||||||
|
A field is written with one length-prefix width (e.g. 4-byte int) but read with another (e.g. 2-byte short), or the prefix encoding (signed vs unsigned varint, fixed vs variable) differs between sides.
|
||||||
|
**Look for:** `writeInt`/`readShort` or `writeVInt`/`readUnsignedVInt` mismatches around the same field; or `vintSize(x)` paired with `writeUnsignedVInt(x)`.
|
||||||
|
|
||||||
|
### F-06: Mixed-version digest mismatch from divergent fields
|
||||||
|
Two software versions compute an identity digest over structurally different fields, so a hard equality check between digests always fails during a mixed-version deployment, triggering continuous re-synchronization or repair loops.
|
||||||
|
**Look for:** A `digest.update` or `hash` call inside a serializer where the field set or order has changed across versions without a version gate.
|
||||||
|
|
||||||
|
### F-07: Non-canonical encoding fed directly to a digest
|
||||||
|
Raw bytes of a value type with multiple equivalent encodings (e.g. variable-length numbers, sorted vs insertion-order maps) are fed directly to a hash without first normalizing; replicas with identical logical state produce different digests.
|
||||||
|
**Look for:** `hash.update(buffer)` where `buffer` came from a type that has a non-canonical wire form (collections, decimals, big ints).
|
||||||
|
|
||||||
|
### F-08: Wire-discriminator enum reordering shifts ordinals
|
||||||
|
An enum used as a wire-protocol discriminator has constants inserted, removed, or reordered, silently shifting the ordinals/codes of all subsequent constants and breaking peers on the older version.
|
||||||
|
**Look for:** Reorder of an enum where `ordinal()` or `values()[i]` is used in serialization, or removal of a middle constant; check for explicit `code` overrides.
|
||||||
|
|
||||||
|
### F-09: Wire-discriminator enum has unhandled cases
|
||||||
|
A type-dispatch encoder/decoder handles only some variants of a discriminator and falls through to the wrong branch (or a default that picks the wrong serializer) for newly added or rare variants.
|
||||||
|
**Look for:** A `switch` on an enum or kind with `default` that assumes an existing variant; missing case for newly added kinds.
|
||||||
|
|
||||||
|
### F-10: Buffer position/duplicate omitted before relative read
|
||||||
|
A `ByteBuffer` returned from a shared structure is read with relative `getInt`/`getLong` before being `duplicate()`d, so the side-effecting position advance corrupts subsequent reads from the same buffer.
|
||||||
|
**Look for:** `buffer.getInt()` / `buffer.get(...)` without a prior `buffer.duplicate()` when the buffer is a shared resource.
|
||||||
|
|
||||||
|
### F-11: arrayOffset ignored on heap ByteBuffer
|
||||||
|
A digest, byte copy, or array constructor uses `buffer.array()` and `buffer.position()` but omits `buffer.arrayOffset()`, so sliced or duplicated buffers read the wrong byte range.
|
||||||
|
**Look for:** `buffer.array()` next to `buffer.position()` without `buffer.arrayOffset()`; especially in checksum, digest, or `new String(bytes, ...)` paths.
|
||||||
|
|
||||||
|
### F-12: Direct buffer with array() throws UnsupportedOperationException
|
||||||
|
Calling `buffer.array()` without first calling `buffer.hasArray()` crashes for off-heap (direct) buffers.
|
||||||
|
**Look for:** `buffer.array()` without a `hasArray()` guard.
|
||||||
|
|
||||||
|
### F-13: Buffer not flipped (or wrongly rewound) before downstream consumption
|
||||||
|
After writing, code calls `rewind()` instead of `flip()` (so the limit stays at capacity and stale bytes leak), or omits `flip()` entirely so the position is at the end and zero bytes are readable.
|
||||||
|
**Look for:** `buffer.rewind()` after a write; missing `buffer.flip()` before returning a buffer for reading.
|
||||||
|
|
||||||
|
### F-14: Buffer position tracking corrupted by raw/wrapper mismatch
|
||||||
|
Position-aware reads go wrong because: a `slice()`/relative `get` is called on a shared buffer each loop without advancing position, the buffer is rewound on an input the deserializer doesn't own, or writes go through the raw stream while a counting/tracking decorator's position diverges.
|
||||||
|
**Look for:** Loops that re-`slice` the same source buffer; `position(0)`/`rewind()` on borrowed input; writes through `rawStream` while sizes accumulate via a `TrackingInputStream`/`CountingOutputStream` wrapper.
|
||||||
|
|
||||||
|
### F-15: Schema-evolved record sender omits a field; receiver NPEs
|
||||||
|
A field declared nullable (or just newly added) is omitted by the sender, but the receiver dereferences it without a null/presence guard.
|
||||||
|
**Look for:** A protocol-generated optional or nullable field touched without `isSet(field)` / `field != null` checks; auto-generated `getX()` calls in deserialize paths.
|
||||||
|
|
||||||
|
### F-16: New field added but version-gated paths don't read it
|
||||||
|
A new field is added to the format without a corresponding version guard on read/write, so older nodes encounter unexpected bytes during a rolling upgrade.
|
||||||
|
**Look for:** A new `out.writeXxx`/`in.readXxx` line not wrapped in `if (version >= NEW_VERSION)`; check both serialize and serializedSize.
|
||||||
|
|
||||||
|
### F-17: New schema field not back-stubbed in older version
|
||||||
|
A new field is added to the current schema version but the older version's definition isn't given a forward-compatible stub or default, so older nodes fail when they receive records containing the new field.
|
||||||
|
**Look for:** A schema definition change without a corresponding "legacy" or "pre-X" stub update.
|
||||||
|
|
||||||
|
### F-18: Version-equality assertion in shared serializer
|
||||||
|
A serializer asserts `version == CURRENT_VERSION` (or hard-codes a fixed version), throwing in any mixed-version cluster or when reading historical on-disk data.
|
||||||
|
**Look for:** `assert version == ...` or `Preconditions.checkArgument(version == ...)` inside serialize/deserialize.
|
||||||
|
|
||||||
|
### F-19: Mixed-version exception treated as fatal
|
||||||
|
A handler throws or escalates on receiving a message from an older protocol version instead of skipping/logging gracefully, breaking rolling upgrades.
|
||||||
|
**Look for:** An `IllegalArgumentException`/`UnsupportedOperationException` thrown from a deserialize switch when version is less than current.
|
||||||
|
|
||||||
|
### F-20: Wrong serializer chosen for sub-variant
|
||||||
|
A polymorphic dispatch uses a catch-all/default serializer for a type with multiple sub-variants, so one sub-variant is encoded with the wrong wire format and corrupts cross-version interop.
|
||||||
|
**Look for:** A `getSerializer(type)` or `serializerFor(kind)` that returns a generic serializer when a specific subtype needs its own.
|
||||||
|
|
||||||
|
### F-21: Type-info read from data instead of authoritative schema
|
||||||
|
A serialization path reads type info from the data object rather than the schema/header context; if the schema has changed since the data was written, the stale type produces corrupt or unreadable bytes.
|
||||||
|
**Look for:** `value.getType()` (instead of `column.type`) inside serialize; `column.type` (instead of the dropped-column header type) inside deserialize.
|
||||||
|
|
||||||
|
### F-22: Headers/metadata silently dropped by wrapper serde
|
||||||
|
A wrapper serializer/deserializer calls a no-headers overload of the inner serde or hardcodes empty headers, dropping caller-supplied context.
|
||||||
|
**Look for:** A wrapper that takes `(topic, headers, value)` but calls `inner.serialize(topic, value)` or `inner.serialize(topic, EMPTY_HEADERS, value)`.
|
||||||
|
|
||||||
|
### F-23: Charset/locale not pinned in serialization
|
||||||
|
`String.getBytes()` is called without a charset, or a `NumberFormat` inherits the JVM default locale; serialized bytes vary across nodes/locales and round-trip parsing fails.
|
||||||
|
**Look for:** `getBytes()` with no charset arg; `new DecimalFormat(pattern)` (no locale); `String.toLowerCase()` (no `Locale.ROOT`).
|
||||||
|
|
||||||
|
### F-24: Byte order / native alignment assumption
|
||||||
|
Native memory reads/writes use an "unaligned access OK" fast path without accounting for big-endian, or `ByteBuffer.getLong` is paired with bit shifts assuming big-endian; values are byte-swapped on the other architecture.
|
||||||
|
**Look for:** `Unsafe.getLong/putLong` used with explicit shifts; `buffer.getLong()` followed by `>> 56` style decoding without `order(BIG_ENDIAN)`.
|
||||||
|
|
||||||
|
### F-25: Hand-rolled encoding diverges from canonical serializer
|
||||||
|
A type's bytes are produced by ad-hoc `getBytes`/`put` calls instead of the canonical serializer (or vice versa), producing a different binary representation that fails round-trip.
|
||||||
|
**Look for:** Inline `buffer.putLong(uuid.getMostSignificantBits())` etc. instead of using a shared `UUIDSerializer`.
|
||||||
|
|
||||||
|
### F-26: Field width or position changed without protocol gate
|
||||||
|
A length prefix's width changed (1→2→4 bytes) or a field's relative position moved across versions; readers using the older fixed-width helper produce garbled values.
|
||||||
|
**Look for:** A constant like `LENGTH_BYTES = 2` that diverges from a parallel `writeShort`/`writeInt`; or a "skip 4 bytes" placed before a moved field.
|
||||||
|
|
||||||
|
### F-27: Sentinel rendered as data
|
||||||
|
A sentinel value (e.g. UUID-zero, `-1`, `null`) is serialized as a literal value (string `"null"`, integer `-1`) rather than being omitted or carrying the absence semantics, leaving consumers unable to distinguish absence from a real value.
|
||||||
|
**Look for:** `writeString(uuid.toString())` without an absence check; serializing `Optional` via `%d` rather than presence guard.
|
||||||
|
|
||||||
|
### F-28: Protocol version tag hardcoded or stale
|
||||||
|
A factory or message-builder hardcodes a stale or sentinel protocol version (`LATEST_PRODUCTION`, `0`, current), bypassing version-aware dispatch and selecting the wrong wire format. Auto-negotiation is also suppressed when an explicit version is supplied to a library that treats it as opt-out.
|
||||||
|
**Look for:** `new XxxRequest(... , VERSION_X, ...)` with a static version constant; `client.builder().protocolVersion(VERSION_X)` against peers with mixed support.
|
||||||
|
|
||||||
|
### F-29: New error code missing from error-handling table
|
||||||
|
A protocol upgrade introduces a new error code that the client's error-handling map does not contain, so the default branch treats a retriable condition as a fatal error.
|
||||||
|
**Look for:** A `Map<Errors, Handler>` or `switch (errorCode)` with `default → throw`; check whether new codes added in a sibling commit are missing entries.
|
||||||
|
|
||||||
|
### F-30: Compatibility shim swallows or rejects legacy data
|
||||||
|
A compatibility reader either throws on encountering an unknown/legacy field, or silently drops a field whose presence was semantically significant; records that depended on that field disappear or upgrades stall.
|
||||||
|
**Look for:** `throw new UnsupportedOperationException("legacy")` in a read path; or a quiet `// skip legacy field` that was the carrier of state.
|
||||||
|
|
||||||
|
### F-31: Skip path leaves unconsumed checksum/trailer in stream
|
||||||
|
An error-handling skip returns early without consuming a checksum suffix or sub-record trailer; the stream is left positioned on bytes that are misread as the next record's header.
|
||||||
|
**Look for:** `if (failure) return;` inside a deserialize loop where the format has a fixed-size suffix per record.
|
||||||
|
|
||||||
|
### F-32: Validation passes structure but not semantics
|
||||||
|
A validator checks the byte count or structural form but does not decode the fields and verify their inter-field constraints, so semantically invalid (but well-framed) values are accepted.
|
||||||
|
**Look for:** A `validate(buffer)` whose body only inspects buffer length / signature, never decoding content.
|
||||||
|
|
||||||
|
### F-33: Variable-length field prefix consumed conditionally
|
||||||
|
A variable-length-field length prefix must always be read regardless of any presence-flag, but the read sits behind an early-return; subsequent entries decode from misaligned bytes.
|
||||||
|
**Look for:** A read of `length = in.readUnsignedVInt()` after an `if (!present) return;` early exit.
|
||||||
|
|
||||||
|
### F-34: Deserializer crashes on legitimate empty input
|
||||||
|
An assertion `length > 0` or a length-zero short-circuit converts a valid empty value into an exception, or emits a `null` literal indistinguishable from true absence.
|
||||||
|
**Look for:** `assert length > 0;` or `if (length == 0) return null;` in a deserialize helper called for legitimately-empty buffers.
|
||||||
|
|
||||||
|
### F-35: Delimiter not escaped inside serialized field
|
||||||
|
A delimiter-based serialization format does not escape the delimiter when it appears in a field value; round-tripping ambiguous data corrupts the parse.
|
||||||
|
**Look for:** `String.join(",", parts)` or hand-rolled delimited writers that don't escape; `split(...)` on the read side.
|
||||||
|
|
||||||
|
### F-36: Field default flips compatibility
|
||||||
|
A schema-field default changes (e.g. nullable → non-nullable, default `0` → default `-1`, or enum default added later); old clients/servers send or expect the old default and the new code crashes or computes wrong state.
|
||||||
|
**Look for:** Schema-definition diffs touching `default`, `nullable`, `nullableVersions`, `ignorable`, `taggedVersions`; correlate with read paths that don't accept the old default.
|
||||||
|
|
||||||
|
### F-37: Same field written twice (outer + inner) after format move
|
||||||
|
After a version refactor that moves a field from outer to inner serializer, both serializers write it; the field appears twice on the wire and corrupts the stream.
|
||||||
|
**Look for:** Two `out.writeX(field)` calls along a serialize chain; check both call sites in the new format.
|
||||||
|
|
||||||
|
### F-38: API version stability flag stuck at "unstable"
|
||||||
|
A protocol version's stability flag is left as preview/unstable after the feature ships, so clients permanently negotiate a lower version.
|
||||||
|
**Look for:** `latestVersionUnstable: true` or `STABLE = false` constants associated with shipped protocol versions.
|
||||||
|
|
||||||
|
### F-39: Hand-maintained switch duplicated across implementations diverges
|
||||||
|
Multiple sibling implementations (read/write of nullable vs non-nullable types, two coordinator versions, refactored sub-classes) hand-maintain parallel switches; a fix to one is missed in the other and the formats drift.
|
||||||
|
**Look for:** Two `switch (type)` blocks with similar shape in sibling files; check whether a fix touches only one.
|
||||||
|
|
||||||
|
### F-40: Serialized form diverges from logical equality
|
||||||
|
A type's `serialize` output differs for two values that compare equal (or two equal serializations decode to unequal objects); comparisons over the wire produce false mismatches. Special case: non-deterministic map iteration order leaks into the wire format.
|
||||||
|
**Look for:** Custom `equals` that ignores a field that `serialize` writes; ordering-sensitive serialization (`for (Entry e : hashMap.entrySet()) out.writeX(e)`) for an unordered logical type.
|
||||||
|
|
||||||
|
### F-41: New record/handler not registered with serialization registry
|
||||||
|
A new schema entry, message verb, or serializer subtype is registered with the wrong serializer, no handler, or omitted from a hand-maintained registry; messages of that type are dropped, mis-decoded, or omitted from gossip/propagation.
|
||||||
|
**Look for:** A new sealed-type/enum constant introduced without a parallel entry in the `Serializers` map / `MessagingService.registerVerb` / serialization registry.
|
||||||
|
|
||||||
|
## Footnotes
|
||||||
|
|
||||||
|
These signals overlap with other categories — when in doubt, also load:
|
||||||
|
- **boundaries-and-arithmetic** for length-prefix arithmetic overflow, position math, vint boundaries
|
||||||
|
- **api-contracts-and-completeness** for builder-omits-field, copy-omits-field, equals/hashCode incompleteness
|
||||||
|
- **lifecycle-and-state** for buffer-recycled-before-write, registration-order bugs
|
||||||
|
- **concurrency** for non-volatile field read twice across a deserialize boundary
|
||||||
|
|
@ -0,0 +1,177 @@
|
||||||
|
# Category: State and Resource Cleanup
|
||||||
|
|
||||||
|
Stale state, leaked references, and orphaned resources that arise when teardown, cancellation, removal, or replay paths fail to fully restore the system to a clean baseline.
|
||||||
|
|
||||||
|
## Diff signals (when to load this category)
|
||||||
|
|
||||||
|
Load this category if the patch contains ANY of:
|
||||||
|
- `close()`, `release()`, `unref()`, `decrementReferenceCount()`, `dispose()`, `shutdown()` calls
|
||||||
|
- `try`/`finally`, `try-with-resources`, or `AutoCloseable` introduction or removal
|
||||||
|
- Cancellation, abort, timeout, rejection, or error-path branches that release/return resources
|
||||||
|
- `clear()`, `remove()`, `deregister()`, `unregister()`, `retainAll()`, `evict()` on long-lived collections, registries, caches, or maps
|
||||||
|
- Tombstone, purge, GC, expiry, eviction, retention, or compaction predicates
|
||||||
|
- Counter increment/decrement pairs, latches, semaphores, in-flight counters, pending queues
|
||||||
|
- Replay, restore-from-log, snapshot loading, or restart bootstrap of in-memory state
|
||||||
|
- New background threads, periodic tasks, scheduled futures, or executors with lifecycle management
|
||||||
|
- Reference counting, off-heap buffer management, native handle ownership
|
||||||
|
- Listener/callback registration without matching deregistration in lifecycle hooks
|
||||||
|
- New or renamed lifecycle methods (`onLeave`, `onRemoved`, `onDelete`, `dispose`, `cleanup`)
|
||||||
|
- Iterator/cursor/stream code that may be returned past its enclosing scope
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### F-01: Counter incremented before submit, no decrement on rejection
|
||||||
|
A pending/in-flight counter is incremented before submitting work to an executor, but the cancellation or rejection path that fires when the submit throws omits the matching decrement, so the counter grows unboundedly.
|
||||||
|
**Look for:** `counter.increment()` / `pending++` followed by `executor.submit(...)` with no decrement in the catch/onReject branch.
|
||||||
|
|
||||||
|
### F-02: Reference released only on success path
|
||||||
|
A reference-counted resource, lock, latch, or pending acknowledgement is released only in the success branch; failure, timeout, or no-ack paths skip the release and leak the handle indefinitely.
|
||||||
|
**Look for:** `ref.release()` or `lock.unlock()` reachable only after a happy-path `if (ok)` rather than from `finally`.
|
||||||
|
|
||||||
|
### F-03: Resource opened then leaked when next call throws
|
||||||
|
A pooled resource, file handle, or buffer is allocated and immediately used by a call that can throw, with no try/finally to release on the exception path; the resource is permanently leaked.
|
||||||
|
**Look for:** allocation followed by a fallible call without surrounding `try { ... } finally { resource.release(); }`.
|
||||||
|
|
||||||
|
### F-04: Multi-step init missing partial cleanup on mid-way failure
|
||||||
|
A constructor or initialization sequence acquires multiple resources serially; when a later step throws, already-acquired earlier resources are not released, so each failed startup leaks them.
|
||||||
|
**Look for:** sequential `open(...)` / `new ...()` calls in `init`/`start`/constructors with no progressive try/catch unwind.
|
||||||
|
|
||||||
|
### F-05: Subclass adds resources but does not override abort/close
|
||||||
|
A subclass adds new closeable fields but does not override the parent's abort or close method; the parent implementation never sees the new fields and they leak on every shutdown or failure.
|
||||||
|
**Look for:** new closeable field in a subclass with no matching `@Override close()` / `abort()`.
|
||||||
|
|
||||||
|
### F-06: Cleanup override silently dead after rename
|
||||||
|
A subclass override targets the old name (or signature) of a parent cleanup method that has since been renamed/changed; without `@Override` the override is no longer reachable and resources leak silently. Same shape applies to listener registrations using stale interface method names.
|
||||||
|
**Look for:** override-style methods without `@Override`; recently renamed lifecycle methods on the parent; anonymous listener implementations whose method names don't match the interface.
|
||||||
|
|
||||||
|
### F-07: try-with-resources missing on inline-opened streams
|
||||||
|
A stream/reader/connection is opened inline as a method argument (or assigned to a local that is never used in a finally block); descriptor leaks on every exception thrown before close.
|
||||||
|
**Look for:** `new FileInputStream(...)` / `Files.newInputStream(...)` / `getConnection()` not wrapped in try-with-resources.
|
||||||
|
|
||||||
|
### F-08: Resource not registered with owning container's close tracker
|
||||||
|
A resource created inside a lifecycle-managed object is wrapped and returned (or stored) without being registered with the container's close-tracker, so the container's own cleanup path skips it.
|
||||||
|
**Look for:** new resource returned from a factory in a "transaction" / "txn" / "tracker" / "scope" without an `addCloseable` / `register` call.
|
||||||
|
|
||||||
|
### F-09: Cleanup loop terminates on first failure, leaks rest
|
||||||
|
A loop that releases multiple resources does not wrap each iteration in its own try/catch; the first throwing release aborts the loop and leaves all subsequent resources permanently held.
|
||||||
|
**Look for:** `for (var r : resources) r.close();` without per-iteration try/catch.
|
||||||
|
|
||||||
|
### F-10: Background thread spawned without owner reference
|
||||||
|
A long-lived background thread (or scheduled future) is started without storing a reference; on shutdown or early-close there is no handle to interrupt or join it, so it leaks indefinitely.
|
||||||
|
**Look for:** `new Thread(...).start()` or `executor.scheduleAtFixedRate(...)` whose return value is ignored.
|
||||||
|
|
||||||
|
### F-11: Listener / metric / sensor registered with no matching deregistration
|
||||||
|
A listener, sensor, or metric is registered when an entity is created but the corresponding remove path does not deregister it, so registrations accumulate unboundedly across churn or restarts.
|
||||||
|
**Look for:** `registry.register(...)` / `addListener(...)` / `metrics.add(...)` in constructor with no symmetric `unregister`/`removeListener`/`metrics.remove(...)` in close/onRemove/onLeave.
|
||||||
|
|
||||||
|
### F-12: Per-entity caches never released on entity removal
|
||||||
|
In-memory per-entity state (map/cache keyed on entity ID) is populated on add/join but never cleared on remove/leave, causing unbounded memory growth under churn.
|
||||||
|
**Look for:** `map.put(entityId, ...)` in onJoin/onAdd with no `map.remove(entityId)` in onLeave/onRemove/onExpire.
|
||||||
|
|
||||||
|
### F-13: Terminal-state entries never purged because deadline never converges
|
||||||
|
Each replica/node independently derives a cleanup deadline for a terminal-state entry, and propagation of the terminal state is suppressed; the deadline never converges and the entry is never garbage-collected.
|
||||||
|
**Look for:** local-only timestamps used as purge deadlines on entries that should reach a globally-terminal state.
|
||||||
|
|
||||||
|
### F-14: Replay skips superseded-entry filter; stale state visible after restart
|
||||||
|
When restoring state from a persistent log, the filter that removes entries already superseded by an advanced progress marker is skipped; old entries appear current after restart.
|
||||||
|
**Look for:** restore/replay paths that call `add(...)` on each record without first checking it against the highest-applied progress/epoch.
|
||||||
|
|
||||||
|
### F-16: Cleanup callback fires before guarded resource is constructed
|
||||||
|
A completion callback shared between paths is invoked unconditionally even on the path that did not yet create the guarded resource; cleanup runs against partially-built or absent state. Same shape: a cleanup listener is registered before the resource is published, so it orphans if the object is discarded before use.
|
||||||
|
**Look for:** completion handler scheduled before the resource's construction completes; `addOnClose(...)` in a constructor before the protected resource is fully published.
|
||||||
|
|
||||||
|
### F-17: Reset clears one of two coupled structures, leaves the other stale
|
||||||
|
A reset/clear operation zeroes out one structure but leaves a sibling collection, derived counter, parallel chain, or wait-list intact; subsequent operations see a half-reset state. Includes failing to reset derived counters after collection clear, and walking only one chain in a builder reset.
|
||||||
|
**Look for:** `reset()` / `clear()` methods that touch one field of a pair without explicitly clearing the related field; `collection.clear()` without `count = 0`.
|
||||||
|
|
||||||
|
### F-18: Stop flag set after queue is cleared; loop never re-checks
|
||||||
|
A shutdown flag is set after a shared queue has been drained, but the worker loop never re-checks the flag at the outer iteration level; the worker continues processing newly-arriving items past close.
|
||||||
|
**Look for:** `stopped = true` placed after `queue.clear()` with no `while (!stopped)` guard at the outer loop.
|
||||||
|
|
||||||
|
### F-19: Reset clears state but does not unblock waiters on invalidated state
|
||||||
|
A reset clears coupled data structures but does not signal/notify threads waiting on the now-invalidated condition; waiters block indefinitely on stale mappings.
|
||||||
|
**Look for:** `reset()` / `invalidate()` methods that clear shared state without `notifyAll()` / `signalAll()` on associated condition variables.
|
||||||
|
|
||||||
|
### F-20: Periodic task short-circuits on disabled flag, never retracts published state
|
||||||
|
A periodic task that exits early when a feature flag is off never retracts state it previously published; downstream components remain permanently blocked by stale entries.
|
||||||
|
**Look for:** early-return on feature-flag-off in periodic tasks that previously called publish/announce/register paths.
|
||||||
|
|
||||||
|
### F-21: Dead-node pointers retained after unlink; unbounded growth
|
||||||
|
A linked structure retains internal dead-node pointers after unlink operations; sustained add/remove workload causes monotonic memory growth.
|
||||||
|
**Look for:** custom linked-list/tree implementations whose unlink path nulls only adjacent fields without clearing back-pointers / forward-pointers on the removed node.
|
||||||
|
|
||||||
|
### F-22: Use-after-free: ref count not incremented before access
|
||||||
|
An off-heap buffer is read or compare-and-swapped without first incrementing its reference count; a concurrent eviction frees the underlying memory between lookup and access, producing a use-after-free.
|
||||||
|
**Look for:** `cache.get(...)` or `map.get(...)` returning a buffer/handle followed by access without `acquire()` / `incrementReferenceCount()`.
|
||||||
|
|
||||||
|
### F-23: Eviction frees resource while consumers hold raw pointers
|
||||||
|
A cache eviction callback unconditionally frees a resource while consumers hold raw (non-counted) pointers; eviction races with active reads cause use-after-free.
|
||||||
|
**Look for:** eviction listeners that call `free`/`close` without first checking ref count or quiescence; callers that hold raw pointers without counted handles.
|
||||||
|
|
||||||
|
### F-24: Drained-but-leaked: drain after atomic swap
|
||||||
|
A shared collector is drained after atomically swapping in a replacement; the window between swap and drain allows concurrent writers to append to the old instance, silently dropping or duplicating entries.
|
||||||
|
**Look for:** `collector.set(newInstance)` followed by `oldInstance.drain()` without quiescence or ordering barrier.
|
||||||
|
|
||||||
|
### F-25: Cleanup loop reads from one source to delete in another
|
||||||
|
A selective-delete cleanup reads from one source to decide what to delete in another, missing orphaned entries when the two sources are out of sync; truncate-and-repopulate is more correct.
|
||||||
|
**Look for:** `for (k in source1.keys()) target.remove(k)` patterns; consider whether `target` may contain keys not in `source1`.
|
||||||
|
|
||||||
|
### F-26: Index/snapshot rebuild without dropping stale entries
|
||||||
|
An index rebuild adds new entries but does not first drop stale on-disk or in-memory entries for the items being rebuilt; old entries survive and corrupt subsequent queries.
|
||||||
|
**Look for:** rebuild paths that `add` / `merge` without a preceding `delete` / `truncate` of the rebuilt subset.
|
||||||
|
|
||||||
|
### F-27: Cancellation indistinguishable from successful completion
|
||||||
|
A background job writes "completed" status unconditionally at the end regardless of cancellation; downstream observers cannot tell the two apart and skip needed cleanup.
|
||||||
|
**Look for:** unconditional `status = COMPLETED` at end of run paths; no branch on `cancelled`/`interrupted` flag.
|
||||||
|
|
||||||
|
### F-28: In-flight dedup flag set on dispatch, cleared only on success
|
||||||
|
An in-flight deduplication flag is set when a request is dispatched but cleared only on the success path; failures leave the flag set forever, permanently blocking retries. Same shape: sticky boolean "active" flag set on add but never cleared on remove.
|
||||||
|
**Look for:** `inFlight.add(key)` followed by `inFlight.remove(key)` only inside an `if (success)` callback; `hasActive = true` with no path back to `false`.
|
||||||
|
|
||||||
|
### F-29: Map entries never removed when value becomes empty
|
||||||
|
Map entries whose values become empty (empty collection, drained queue) are never removed, so stale keys persist and corrupt membership/size queries.
|
||||||
|
**Look for:** `map.get(k).remove(v)` patterns that don't follow with `if (map.get(k).isEmpty()) map.remove(k);`.
|
||||||
|
|
||||||
|
### F-30: Static cache keyed on path never invalidated on file replacement
|
||||||
|
A static cache keyed on file path is never evicted after the underlying file is replaced; subsequent operations on the new file receive stale metadata.
|
||||||
|
**Look for:** static `Map<Path, Metadata>` populated on first access with no invalidation hook tied to file overwrite/replace.
|
||||||
|
|
||||||
|
### F-31: Iterator returned past close
|
||||||
|
An iterator is returned to the caller after being closed in a finally block, or after its underlying resource has been released; the caller's first `next()` reads from a closed iterator.
|
||||||
|
**Look for:** `try { ... return iter; } finally { iter.close(); }` patterns; iterator escaping the scope of its underlying resource.
|
||||||
|
|
||||||
|
### F-32: Filtered-out closeable items never closed
|
||||||
|
An iterator-style filter silently skips items without closing them; filtered-out objects hold open file/buffer handles that are never released. Same shape: `CloseableIterable` assigned to a plain `Iterable`, erasing the close contract.
|
||||||
|
**Look for:** `.filter(...)` over `Closeable`/`AutoCloseable` streams without explicit `close` on rejected items; downcast/upcast that drops `Closeable`.
|
||||||
|
|
||||||
|
### F-33: Resource opened in constructor not assigned to field before throwing
|
||||||
|
A handle opened inside a constructor is not assigned to its field before an early return or thrown exception; the close method cannot reach it and the descriptor leaks.
|
||||||
|
**Look for:** `Resource r = open()` followed by potentially-throwing work before `this.field = r`.
|
||||||
|
|
||||||
|
### F-34: Background factory creates new instance without checking previous still running
|
||||||
|
A constructor used for both transient and durable instances unconditionally starts a periodic background task; frequent transient creation accumulates unbounded background work.
|
||||||
|
**Look for:** `start()` / `schedule(...)` calls in constructors with no idempotence/active check; transient lifetimes for the same class.
|
||||||
|
|
||||||
|
### F-35: Lazy cache ignores changing input parameters
|
||||||
|
A lazily-initialized cache stores the result of the first computation and reuses it regardless of changed inputs, returning stale results.
|
||||||
|
**Look for:** `if (cache == null) cache = compute(input);` patterns inside methods whose `input` differs across calls.
|
||||||
|
|
||||||
|
### F-36: Cleanup gated on liveness check; skipped after entity already gone
|
||||||
|
A cleanup or unsubscribe operation is gated behind a membership/liveness check; when membership has already advanced to EMPTY/DEAD the cleanup is silently skipped, leaving stale state to outlive the entity it was supposed to follow.
|
||||||
|
**Look for:** `if (group.exists()) cleanup()` patterns where `group` may already be empty/expired but its associated state still needs cleanup.
|
||||||
|
|
||||||
|
### F-37: Half-released paired resources on rejection
|
||||||
|
When a task submission is rejected, only half of a paired resource (e.g., reference count released, transaction left open) is released; the other half leaks.
|
||||||
|
**Look for:** rejection handlers that release one resource handle while leaving a related handle (txn, lock, ref) untouched.
|
||||||
|
|
||||||
|
### F-38: Two-phase cleanup counter never decremented on fast-path
|
||||||
|
A reference counter initialized to total task count expects each completion to decrement it; a fast-path that bypasses the response handler never decrements, leaving the pending record permanently un-removed.
|
||||||
|
**Look for:** `counter = N` followed by handlers that decrement on slow-path; verify all bypass / short-circuit paths also decrement.
|
||||||
|
|
||||||
|
### F-39: Failed remote cleanup silently dropped, no retry
|
||||||
|
A remote cleanup call (e.g., delete on another node) that fails is silently swallowed without retry or alerting; orphaned remote resources accumulate forever.
|
||||||
|
**Look for:** `try { remote.delete(...); } catch (Exception e) { log.warn(...); }` with no retry queue or escalation.
|
||||||
|
|
||||||
|
### F-40: Session removed from cache before its resources are released
|
||||||
|
A session is removed from a cache before releasing its associated resources; the release code then operates on already-removed state and silently skips cleanup. Same shape: pending-init queues whose entries are never closed when the parent collection is cleared on shutdown.
|
||||||
|
**Look for:** `cache.remove(sessionId)` followed by `session.release()`; verify pending/bootstrapping queues are drained-and-closed in `shutdown()`.
|
||||||
|
|
@ -0,0 +1,180 @@
|
||||||
|
# Category: Validation and Input Handling
|
||||||
|
|
||||||
|
Bugs where input validators, parsers, format detectors, and pre-conversion guards either miss cases, anchor on the wrong delimiter, fail to account for downstream length/charset/OS limits, or skip pre-use checks (null, -1, empty buffer, length, type) before consuming a value.
|
||||||
|
|
||||||
|
## Diff signals (when to load this category)
|
||||||
|
|
||||||
|
Load this category if the patch contains ANY of:
|
||||||
|
- New or modified `validate*`, `check*`, `verify*`, `assert*`, `isValid*`, `parse*`, `tokenize*`, `decode*` methods
|
||||||
|
- Calls to `indexOf`, `lastIndexOf`, `String.split`, `Pattern.compile`, `Matcher`, `String.replace*`, `substring`, `startsWith`/`endsWith`, `getBytes`/`new String`
|
||||||
|
- Construction or modification of regex character classes, anchors, escapes, or quantifiers
|
||||||
|
- File path / URI / address parsing: `getFileName`, `Paths.get`, `URI`, `InetAddress`, `InetSocketAddress`, host:port splitting, IPv4/IPv6 literals
|
||||||
|
- Functions appending suffixes/prefixes (`-`, `_`, UUID, extension, port) before passing names to filesystem, network, or registry calls
|
||||||
|
- Code that consumes an `Optional`, nullable lookup, or map `get()` result without `.isPresent()`/`!= null`/`getOrDefault`
|
||||||
|
- Conditions calling `indexOf(...)` then `substring(...)` without a `-1` guard
|
||||||
|
- Pre-conversion / pre-deserialization filters that drop or pass through unsupported attribute types, sentinels, or empty values
|
||||||
|
- New constraint, length, charset, locale, or boundary checks; or removal of an existing one
|
||||||
|
- Conversion of user-supplied identifiers / option strings / config values to typed objects (enum, UUID, BigDecimal, Date)
|
||||||
|
- Length, capacity, or quantity bounds that interact with appended fixed-size data (suffixes, length prefixes, framing headers)
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### F-01: Validator passes but downstream limit (OS/filesystem/protocol) fails after suffix append
|
||||||
|
A name passes format validation but a fixed-length suffix is appended later (UUID, generation marker, extension), and the combined length exceeds an OS or protocol cap.
|
||||||
|
**Look for:** Name-validation methods checking length/format against a constant `MAX_NAME_LEN` while the same name is later concatenated with `"-" + uuid`, `.tmp`, or a numeric generation; verify the validator subtracts the suffix budget.
|
||||||
|
|
||||||
|
### F-02: indexOf followed by substring with no -1 guard
|
||||||
|
A delimiter is located with `indexOf`/`lastIndexOf` and the index is passed straight into `substring` without checking for `-1`, throwing `StringIndexOutOfBoundsException` on inputs that lack the delimiter.
|
||||||
|
**Look for:** `s.substring(s.indexOf(c) + 1)` or `s.substring(0, s.indexOf(c))` without a preceding `if (idx < 0)` branch.
|
||||||
|
|
||||||
|
### F-03: Pre-conversion validator omits unsupported-attribute check
|
||||||
|
A converter accepts an input with attributes the target format cannot represent (TTL, timestamp resolution, headers, multi-cell collection) and silently discards them rather than rejecting the conversion.
|
||||||
|
**Look for:** Conversion / migration / serialization functions whose validation step touches only the value's primary type and not its qualifying attributes; expect a missing branch like `if (src.hasFoo()) throw ...`.
|
||||||
|
|
||||||
|
### F-04: Address parsed by splitting on the wrong delimiter
|
||||||
|
Multi-segment addresses (IPv6, host:port with brackets, qualified namespace.table) are split on a delimiter that also appears inside one of the segments, retaining only the first part.
|
||||||
|
**Look for:** `addr.split(":")` applied to strings that may contain IPv6 literals, or `name.split(".")` applied without limit/escape; check whether IPv6 brackets, percent-encoding, or quoted segments are stripped first.
|
||||||
|
|
||||||
|
### F-05: Stripping a literal prefix with regex misinterprets metacharacters
|
||||||
|
A "strip prefix" or "strip suffix" routine uses `String.replaceFirst`/`replaceAll`/`split` directly on a literal prefix that may contain regex metacharacters (`.`, `+`, `?`, `[`, `(`, `\`).
|
||||||
|
**Look for:** `s.replaceAll(prefix, "")`, `s.split(sep)` where `prefix`/`sep` is a path, version, hostname, or platform separator; expect missing `Pattern.quote(...)` or `replace` (literal) usage.
|
||||||
|
|
||||||
|
### F-06: Filename prefix filter does not append a separator after the name
|
||||||
|
A prefix filter (`name.startsWith(prefix)`) matches longer names that share the same prefix but are different entities, opening the wrong files.
|
||||||
|
**Look for:** `name.startsWith(targetName)` or "list files for table X" filters that don't append `"-"`, `"."`, or another separator to the matched name.
|
||||||
|
|
||||||
|
### F-07: Path-prefix containment check uses raw string startsWith
|
||||||
|
A directory containment / ancestry check compares paths with `startsWith` without ensuring a path-separator boundary, so `/foo/barbaz` matches `/foo/bar`.
|
||||||
|
**Look for:** `path.startsWith(parent)` or `parent.isPrefixOf(child)` style logic without a final `/` (or platform separator) appended; also watch for `Path.startsWith` confusion with `String.startsWith`.
|
||||||
|
|
||||||
|
### F-08: Regex character class with mid-class hyphen creates an unintended range
|
||||||
|
A character class uses an unescaped `-` between two literal characters, silently producing an ASCII range that includes characters the author did not intend.
|
||||||
|
**Look for:** `[a-z\-_.+]` style classes where `-` appears between non-adjacent literals; verify hyphen is at the start, end, or escaped.
|
||||||
|
|
||||||
|
### F-09: split() on a regex metacharacter passed as literal
|
||||||
|
`String.split` is called with a single character (`.`, `|`, `+`, `*`, `(`) that the regex engine interprets specially, producing all-empty strings or no split at all.
|
||||||
|
**Look for:** `path.split(File.separator)` (where separator is `\` on Windows), `s.split(".")`, `s.split("|")` without `Pattern.quote`.
|
||||||
|
|
||||||
|
### F-10: Anchored regex matches only first occurrence, missing duplicates
|
||||||
|
A pattern uses `^` or `\A` (or implicit `matches()`) when `find()` was needed, so a config option only triggers a check on the first occurrence and a duplicate elsewhere is silently appended.
|
||||||
|
**Look for:** Detection of an existing JVM/CLI flag with `^-Dfoo` or `pattern.matcher(s).matches()` that should be `find()`.
|
||||||
|
|
||||||
|
### F-11: Format-time validation differs from runtime validation
|
||||||
|
A schema/DDL accepts a combination at definition time that runtime then rejects, or vice versa, because the two validators are written by different code paths and only one knows the new constraint.
|
||||||
|
**Look for:** Two validation entry points (`validateDdl` and `validateExecution`, or `validate` vs `validateForBuild`); one was updated for a new feature and the other was not.
|
||||||
|
|
||||||
|
### F-12: Boolean parser only matches positive form, silently returns false on typos
|
||||||
|
A property/boolean parser tests for "true"/"yes"/"1" with regex but returns `false` (instead of throwing) for any non-matching input, silently swallowing typos.
|
||||||
|
**Look for:** Parsing methods that pattern-match a positive set and have an unconditional `return false` fallthrough; expect operator typos like "treu" to pass validation.
|
||||||
|
|
||||||
|
### F-13: Compound declarative predicates accepted without satisfiability check
|
||||||
|
Predicates on the same field (constraints, range bounds, multiple `restrictions`) are validated individually but never cross-checked for satisfiability, allowing contradictory combinations to be stored and queried later with confusing results.
|
||||||
|
**Look for:** Per-restriction `validate()` calls in a loop with no follow-up "do these together make sense" pass; e.g. `x > 5 AND x < 3` accepted.
|
||||||
|
|
||||||
|
### F-14: User-supplied identifier passed to case-sensitive lookup without quoting
|
||||||
|
An identifier that contains uppercase, special characters, or reserved words is forwarded to a case-sensitive backend without being quoted, so writes go to the wrong key or lookups silently miss.
|
||||||
|
**Look for:** Calls to driver/JDBC/CQL/SQL APIs that accept an identifier; check whether identifiers needing quoting are wrapped with the API's quoting helper. Also watch for identifiers double-quoted by mistake.
|
||||||
|
|
||||||
|
### F-15: Empty buffer or empty array bypasses null guard but fails on first access
|
||||||
|
A guard checks for `null` but not for empty (`length == 0`, `remaining() == 0`, `isEmpty()`); the empty value passes the guard and the first index/array access throws or returns garbage.
|
||||||
|
**Look for:** `if (arr != null) ... arr[0]`, `if (buf != null) buf.getInt()`, `if (s != null) s.charAt(0)` without `!arr.isEmpty()`/`buf.hasRemaining()`/`!s.isEmpty()`.
|
||||||
|
|
||||||
|
### F-16: User-supplied collection passed to method that asserts non-empty
|
||||||
|
A public method documents an empty-collection-allowed contract but starts with an assertion or first-element access that rejects valid empty input.
|
||||||
|
**Look for:** Methods whose entry has `assert !c.isEmpty()` or `c.iterator().next()` while their callers may legitimately pass empty inputs (secondary index writes, no-op queries, single-replica scenarios).
|
||||||
|
|
||||||
|
### F-17: Validation rejects single-element grammar where one-or-more was intended
|
||||||
|
A grammar quantifier requires two-or-more comma-separated items, or a list rule rejects an empty list literal that should be valid.
|
||||||
|
**Look for:** `+` quantifier where `*` was intended (or `(item ',' item)+` patterns); empty-collection literals that fail to parse.
|
||||||
|
|
||||||
|
### F-18: Reserved word / reserved-character validation only on one entry path
|
||||||
|
Validation rejects reserved class/keyspace names on the user-facing CRUD path but a sibling internal/replay/migration path constructs the same entity without the check, allowing reserved-name rows to land in storage.
|
||||||
|
**Look for:** Validation guards in user-facing controllers absent from internal write, recovery, replay, migration, or factory call paths.
|
||||||
|
|
||||||
|
### F-19: Length validation uses one width while serializer uses another
|
||||||
|
A field is length-checked against one width (e.g., `Short.MAX_VALUE`) but encoded with a different width (4-byte length prefix vs 2-byte), so values that pass validation later fail at serialization time with an opaque error.
|
||||||
|
**Look for:** Length-check constants (`MAX_NAME_BYTES`, `MAX_LEN`) decoupled from the actual serializer's width; mismatched `writeShort(len)` / `readInt(len)` pairs.
|
||||||
|
|
||||||
|
### F-20: Number / decimal parser accepts adversarial exponent without bound check
|
||||||
|
`BigDecimal.toPlainString`, `setScale`, or similar methods are called on user input without bounding the exponent, allowing a small input to expand into gigabytes of digits or to exhaust memory.
|
||||||
|
**Look for:** `new BigDecimal(s).toPlainString()`, `setScale(0, ...)`, parsing of arbitrary-precision numbers from network/CLI; expect missing exponent / precision caps.
|
||||||
|
|
||||||
|
### F-21: Charset-less getBytes / new String produces platform-dependent bytes
|
||||||
|
`String.getBytes()` or `new String(bytes)` is called without an explicit `Charset`, so checksum, hash, or wire-protocol bytes vary by JVM default locale.
|
||||||
|
**Look for:** Bare `getBytes()` / `new String(b)` in code that computes digests, builds wire frames, or persists identifiers; require explicit `StandardCharsets.UTF_8`.
|
||||||
|
|
||||||
|
### F-22: Locale-sensitive case folding applied inconsistently
|
||||||
|
Identifier normalization uses locale-independent case folding for one component but the default locale for another (`toUpperCase()` vs `toUpperCase(Locale.ROOT)`), producing mismatched keys on JVMs with locale-sensitive case mappings (Turkish "I").
|
||||||
|
**Look for:** Mixed `toLowerCase()` / `toLowerCase(Locale.ROOT)` calls inside the same identifier-handling pipeline.
|
||||||
|
|
||||||
|
### F-23: Locale-sensitive number/date format breaks fixed parser
|
||||||
|
A `NumberFormat`, `DateFormat`, or `String.format` call inherits the JVM default locale and emits a comma decimal separator that a fixed regex parser then fails to round-trip.
|
||||||
|
**Look for:** Format objects constructed without `Locale.ROOT`/`Locale.US`; fixed regexes like `\d+\.\d+` that reject locale-formatted output.
|
||||||
|
|
||||||
|
### F-24: Date / timestamp parser omits a valid format variant
|
||||||
|
A timestamp-format enumeration covers most variants but omits one (space-delimited offset, ISO week-year, fractional-second precision), silently failing to parse legitimate timestamps from peers or files.
|
||||||
|
**Look for:** Hand-rolled `DateFormat`/`SimpleDateFormat` lists; check that all documented input variants are covered, including space-vs-`T` separators and `Z`-vs-offset.
|
||||||
|
|
||||||
|
### F-25: Lenient date parser silently rolls over out-of-range fields
|
||||||
|
A date parser left in lenient mode rolls "Feb 30" into March without rejecting it, accepting and silently shifting invalid input.
|
||||||
|
**Look for:** `DateFormat`/`SimpleDateFormat` constructed without `setLenient(false)`; calls accepting user-supplied date strings.
|
||||||
|
|
||||||
|
### F-26: Negative or sentinel value bypasses range / unit conversion check
|
||||||
|
A configuration accepts `-1` (or another sentinel meaning "disabled") that the type converter cannot represent, throwing on first use; or a sentinel used as a real value (e.g. timestamp `-1`) participates in arithmetic where a real comparison is expected.
|
||||||
|
**Look for:** Config setters where the legacy sentinel must be mapped to `null`/`Optional.empty()` before converting; sentinel constants like `-1`, `Long.MIN_VALUE`, `Long.MAX_VALUE` used in comparisons without an "is sentinel" guard.
|
||||||
|
|
||||||
|
### F-27: Unit-suffixed property parsed without applying unit
|
||||||
|
A property whose name advertises units (`...Mb`, `...Kb`, `...Ms`, `...Bytes`) is read as a raw integer and never multiplied by the corresponding factor, so the stored value is 1024x or 1000x off.
|
||||||
|
**Look for:** `getInt("foo.size.mb")` immediately stored to a `long bytes` field; or values written to a "kb" field while the value is in bytes.
|
||||||
|
|
||||||
|
### F-28: Mutually-exclusive options accepted together with no validation
|
||||||
|
Two configuration keys that should be mutually exclusive (auto+manual, min+max, encrypt+plaintext) have no cross-check at parse time, surfacing as confusing runtime errors later.
|
||||||
|
**Look for:** `if (configA.set() && configB.set()) throw ...` missing in the validation pass; expect new options added without updating the cross-validation matrix.
|
||||||
|
|
||||||
|
### F-29: User-supplied option flag silently overridden by hardcoded default
|
||||||
|
A constructor accepts a parameter (encryption options, throttle, headers) but never wires it through, so the default is silently used regardless of caller input.
|
||||||
|
**Look for:** Constructor parameter with no field assignment, or a setter call that re-applies the hardcoded default after the user value was assigned.
|
||||||
|
|
||||||
|
### F-30: One namespace's identifier passed to a lookup keyed by another namespace
|
||||||
|
A lookup populated using one identifier representation (post-quote, with port, with prefix) is queried using a different representation (raw, without port, stripped), so every valid entry silently misses.
|
||||||
|
**Look for:** Pairs of put/get on the same map where the key is built differently on each side; address strings with/without port, identifiers with/without quoting, paths absolute-vs-relative.
|
||||||
|
|
||||||
|
### F-31: Reserved character in user input corrupts downstream framing
|
||||||
|
A delimiter character that the format reserves (gossip separator, CSV separator, log4j pattern) is accepted in user input without escaping, so the field corrupts the encoded record.
|
||||||
|
**Look for:** Constructors / setters of structured identifiers that take an arbitrary string; check that the reserved character is rejected or escaped on entry.
|
||||||
|
|
||||||
|
### F-32: Validation that throws bypasses caller's "not found" / no-op contract
|
||||||
|
A throwing helper (`getOrThrow`) is used on user-supplied input where the contract is "missing == empty"; the raw exception escapes to the user instead of being normalized.
|
||||||
|
**Look for:** `Map.get` lookups on user input followed by `.orElseThrow()` or assertions, in code paths where absence should yield an empty result; or "not found" rethrown as a generic runtime exception.
|
||||||
|
|
||||||
|
### F-33: Empty input array / config reaches code that derives an internal size from it
|
||||||
|
External empty input is passed to a constructor that derives internal-array size from the input length, so the zero-size array later throws `ArrayIndexOutOfBoundsException` on first access.
|
||||||
|
**Look for:** Constructors that allocate `new T[input.length]` from caller-supplied data without an early `if (input.length == 0)` check.
|
||||||
|
|
||||||
|
### F-34: Boolean type guard accepts only one variant, missing a sibling
|
||||||
|
A guard checks `instanceof FrozenCollection` but not the matching `MultiCellCollection`, or vice versa; one valid variant takes the unsafe default path or is rejected.
|
||||||
|
**Look for:** `instanceof` chains that enumerate one subtype; verify a structural sibling isn't silently routed elsewhere. Common in collection / wrapped-type / decorator scenarios.
|
||||||
|
|
||||||
|
### F-35: Tokenization halts at a delimiter without consuming remaining tokens
|
||||||
|
A startup script / option parser halts at a delimiter (`--`, `;`) without forwarding remaining arguments, silently dropping user-provided options.
|
||||||
|
**Look for:** `for arg in $@` loops with `break` on a delimiter; argparse / getopt loops that don't accumulate the tail.
|
||||||
|
|
||||||
|
### F-36: Sub-name / nested name parsing uses left-to-right with fixed token count
|
||||||
|
A compound identifier with a variable-length leading component is parsed left-to-right by counting fixed tokens, misidentifying fields when the leading component itself contains the delimiter.
|
||||||
|
**Look for:** `split("/", 2)` on URLs/paths where the host or scheme can contain `/`; right-anchored parsing (`lastIndexOf`) is usually correct.
|
||||||
|
|
||||||
|
### F-37: Whitespace not trimmed from tokens before lookup
|
||||||
|
Tokens from `split(...)` are passed straight to a lookup without `trim()`, causing every entry with surrounding whitespace to fail validation.
|
||||||
|
**Look for:** CSV / config parsing pipelines that immediately do `Map.get(token)` after `split(",")`.
|
||||||
|
|
||||||
|
### F-38: Membership check uses wrong key type, silently misses every entry
|
||||||
|
`Map.get` accepts `Object` and silently returns `null` when the wrong key type (boxed-vs-primitive, wrapper-vs-raw, byte[]-vs-String) is passed.
|
||||||
|
**Look for:** Lookups where the value type doesn't match the map's declared key type; `set.contains(name)` where `name` is a wrapper but elements are raw.
|
||||||
|
|
||||||
|
### F-39: Trusting stored boolean / metadata flag without cross-validation
|
||||||
|
A migration trusts a stored boolean flag that an earlier version may have written incorrectly, carrying the stale value into a new schema.
|
||||||
|
**Look for:** Migration code paths reading a single flag from an old format without verifying it against an independent heuristic; expect "trust but verify" to be missing.
|
||||||
|
|
||||||
|
### F-40: Buffer length / framing prefix mismatch between writer and reader
|
||||||
|
A writer uses one length-prefix width (4-byte) but the reader uses another (2-byte), or one path writes a field unconditionally while the other reads it conditionally, corrupting all subsequent bytes.
|
||||||
|
**Look for:** Asymmetric `writeInt(len)` / `readShort(len)` pairs across versions; conditional vs unconditional field reads gated on differing flags.
|
||||||
|
|
@ -0,0 +1,98 @@
|
||||||
|
# Report format
|
||||||
|
|
||||||
|
Use this format in Phase 6, after all review subagents have returned. The aim is a
|
||||||
|
report that a human reviewer can act on without re-doing the work — every finding
|
||||||
|
should answer "where, what, why-I-believe-it, what-to-do".
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Format
|
||||||
|
|
||||||
|
```
|
||||||
|
# Targeted Review: {patch identifier — commit, branch, or short description}
|
||||||
|
|
||||||
|
## Patch summary
|
||||||
|
{2-3 sentences: what the patch does, drawn from patch-explainer's executive summary}
|
||||||
|
|
||||||
|
## Categories considered
|
||||||
|
- Loaded: {list — e.g., serialization-and-versioning, concurrency-and-locking, ...}
|
||||||
|
- Skipped: {list with one-word reason — e.g., "lifecycle-and-ordering (no init/register
|
||||||
|
changes)", "io-and-crash-safety (no persistence touched)"}
|
||||||
|
|
||||||
|
## Foci dispatched
|
||||||
|
- Focus A: {one-line statement} — {N items, K findings}
|
||||||
|
- Focus B: {one-line statement} — {N items, K findings}
|
||||||
|
- Focus C: ...
|
||||||
|
|
||||||
|
## Findings (ranked by confidence)
|
||||||
|
|
||||||
|
### Finding 1: {short descriptive title}
|
||||||
|
- **Location**: {file:line}
|
||||||
|
- **Confidence**: High / Medium / Low
|
||||||
|
- **Found by**: {focus name(s)}
|
||||||
|
- **Category**: {category-id, or "off-checklist"}
|
||||||
|
- **What's wrong**: {2-3 sentences with reasoning}
|
||||||
|
- **Evidence**: {what was searched / read / found}
|
||||||
|
- **Suggested fix**: {specific change}
|
||||||
|
|
||||||
|
### Finding 2: ...
|
||||||
|
|
||||||
|
## Cross-cutting observations
|
||||||
|
{Any patterns that emerged from multiple foci — e.g., "two foci independently flagged
|
||||||
|
missing equals/hashCode updates for a new field", or "the new fsync coordination is
|
||||||
|
tested at one call site but not at the parallel one in BackupWriter"}
|
||||||
|
|
||||||
|
## What was NOT reviewed
|
||||||
|
{Brief list of categories or aspects skipped, so the reader knows the boundary of this
|
||||||
|
review. Keep terse — it's a closure note, not a full audit.}
|
||||||
|
|
||||||
|
## Recommended follow-ups
|
||||||
|
{Optional — if findings cluster around a particular concern that would benefit from
|
||||||
|
deeper investigation, name it. E.g., "The serialization symmetry findings warrant a
|
||||||
|
dedicated deep-review pass on FooSerializer.java and its sibling Bar deserializer."}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Ranking and dedup
|
||||||
|
|
||||||
|
Within "Findings (ranked by confidence)":
|
||||||
|
|
||||||
|
1. **Sort** by confidence (High → Medium → Low), then by impact (correctness > efficiency
|
||||||
|
> style)
|
||||||
|
|
||||||
|
2. **Deduplicate** — if two foci flagged the same root cause at the same location,
|
||||||
|
merge into one finding and credit both. If they reached different conclusions, keep
|
||||||
|
both with a note on the disagreement.
|
||||||
|
|
||||||
|
3. **Boost** confidence when:
|
||||||
|
- Multiple foci independently flagged the same site (mark as "Found by: Focus A, B")
|
||||||
|
- A finding ties to a category that has historically high incidence in this corpus
|
||||||
|
(e.g., logic errors are the largest category — a logic finding deserves attention)
|
||||||
|
- Codebase-analysis surfaced the same concern as a known invariant or past bug
|
||||||
|
|
||||||
|
4. **Three-point test** each finding before including:
|
||||||
|
- The code construct exists in the diff (not inferred from absence alone)
|
||||||
|
- The bug is plausible given visible context (not pure speculation)
|
||||||
|
- The finding is actionable (the reader can tell what to change)
|
||||||
|
|
||||||
|
If a finding fails the test, drop it or downgrade to a "uncertain — investigation
|
||||||
|
note" at the bottom.
|
||||||
|
|
||||||
|
## Tone and concision
|
||||||
|
|
||||||
|
- Lead each finding with location and confidence — the reader should be able to scan and
|
||||||
|
decide priority in one pass.
|
||||||
|
- 2-3 sentence reasoning, not paragraphs. The diff is on screen; don't restate it.
|
||||||
|
- Evidence should be concrete: "Grepped for `equals(` in {Class}.java — no override
|
||||||
|
found, and parent class compares only by id" is useful. "I checked and it seems
|
||||||
|
wrong" is not.
|
||||||
|
- Suggested fix should be specific. "Add a null check" is too vague. "Wrap line 142 in
|
||||||
|
`if (config.getFoo() != null)`" is right.
|
||||||
|
|
||||||
|
## Length
|
||||||
|
|
||||||
|
Aim for the whole report to fit on one screen of scrolling for small patches and 2-3
|
||||||
|
screens for larger ones. If the report is longer, group findings under sub-headings by
|
||||||
|
focus or by category. The reader should be able to print it out and walk through
|
||||||
|
findings during review.
|
||||||
|
|
@ -0,0 +1,121 @@
|
||||||
|
# Review subagent prompt template
|
||||||
|
|
||||||
|
Use this template when spawning a review subagent in Phase 5. Each subagent reviews ONE
|
||||||
|
focus with a tailored checklist drawn from the categorized findings.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Template
|
||||||
|
|
||||||
|
```
|
||||||
|
You are reviewing a focused part of a patch for correctness bugs. You will gather
|
||||||
|
evidence from the actual source — not just pattern-match against a checklist.
|
||||||
|
|
||||||
|
PATCH
|
||||||
|
-----
|
||||||
|
{patch path or ref, or pasted diff}
|
||||||
|
|
||||||
|
PATCH SUMMARY (from patch-explainer; trimmed)
|
||||||
|
---------------------------------------------
|
||||||
|
{2-5 bullet points on what changed and why, only the parts relevant to your focus}
|
||||||
|
|
||||||
|
SURROUNDING CONTEXT (from codebase-analysis; trimmed)
|
||||||
|
-----------------------------------------------------
|
||||||
|
{2-5 bullet points: invariants, lifecycle, parallel implementations, callers — only what
|
||||||
|
is relevant to your focus}
|
||||||
|
|
||||||
|
YOUR FOCUS
|
||||||
|
----------
|
||||||
|
{One sentence: what part of the change you are reviewing.
|
||||||
|
Examples:
|
||||||
|
- "Review the new replay() method in JournalReplay.java and its interaction with fsync"
|
||||||
|
- "Review the version-gated decoding in MessageDeserializer.decode()"
|
||||||
|
- "Review the new field 'dirtyVersion' wherever it is read or written"
|
||||||
|
- "Review the registration symmetry of the new MetricsListener" }
|
||||||
|
|
||||||
|
CHECKLIST
|
||||||
|
---------
|
||||||
|
For each item below: check whether the pattern actually applies to your focus area.
|
||||||
|
Gather evidence (read the source, grep for callers, check parallel paths) before
|
||||||
|
reporting.
|
||||||
|
|
||||||
|
{For each selected item:}
|
||||||
|
- [{category-id}] {Finding text}
|
||||||
|
Look for: {hint}
|
||||||
|
|
||||||
|
{Repeat for each item in this focus's checklist — typically 5-15 items.}
|
||||||
|
|
||||||
|
INSTRUCTIONS
|
||||||
|
------------
|
||||||
|
1. Read the patch and the relevant source files. Do not rely on summaries alone.
|
||||||
|
|
||||||
|
2. For EACH checklist item, decide one of:
|
||||||
|
- APPLIES — the pattern matches a real construct in the focus area, and the bug is
|
||||||
|
plausible given context. Report it.
|
||||||
|
- DOES NOT APPLY — the pattern keyword matched but the code does not actually have
|
||||||
|
the bug shape, OR the surrounding code prevents the failure. Skip silently.
|
||||||
|
- UNCERTAIN — the pattern might apply but you cannot confirm without more digging.
|
||||||
|
Report at Low confidence with the open question.
|
||||||
|
|
||||||
|
3. Also report bugs you notice in the focus area that are NOT on the checklist. The
|
||||||
|
checklist is a starting point, not a fence.
|
||||||
|
|
||||||
|
4. For evidence: prefer reading the actual source over inferring. Use Grep to find
|
||||||
|
callers, parallel paths, or symmetric code. Use Read to inspect files referenced in
|
||||||
|
the diff.
|
||||||
|
|
||||||
|
5. Confidence rubric:
|
||||||
|
- HIGH: code construct exists in diff, you have evidence the bug occurs, you can
|
||||||
|
point to specific failure inputs or scenarios.
|
||||||
|
- MEDIUM: code construct exists, the bug is plausible, but you couldn't fully verify
|
||||||
|
the failure path or you're inferring some context.
|
||||||
|
- LOW: pattern match without strong evidence; flag for the human to investigate.
|
||||||
|
|
||||||
|
OUTPUT
|
||||||
|
------
|
||||||
|
For each finding:
|
||||||
|
|
||||||
|
## Finding: {short descriptive title}
|
||||||
|
- Location: {file:line}
|
||||||
|
- Category: {category-id from checklist, or "off-checklist"}
|
||||||
|
- Confidence: High / Medium / Low
|
||||||
|
- What's wrong: {1-3 sentences explaining the bug}
|
||||||
|
- Evidence: {what you searched / read / found, briefly}
|
||||||
|
- Suggested fix: {specific change, if obvious}
|
||||||
|
|
||||||
|
If no findings in this focus: say so explicitly with one sentence on what you checked.
|
||||||
|
|
||||||
|
End with a coverage note:
|
||||||
|
Checklist coverage: {N/M items applied}, {K off-checklist findings}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How to populate the template
|
||||||
|
|
||||||
|
For each focus you defined in Phase 4:
|
||||||
|
|
||||||
|
1. **Patch summary**: extract from patch-explainer's output, the parts relevant to this
|
||||||
|
focus — typically the before/after for the relevant function/file plus any noted
|
||||||
|
assumptions or failure modes that touch the focus area.
|
||||||
|
|
||||||
|
2. **Surrounding context**: extract from codebase-analysis output, the parts about
|
||||||
|
invariants, callers, parallel paths that touch the focus area.
|
||||||
|
|
||||||
|
3. **Focus statement**: be specific and narrow. "Review the journal replay subsystem"
|
||||||
|
is too broad. "Review JournalReplay.replay() and its handling of fsync ordering with
|
||||||
|
the new dirty flag" is right.
|
||||||
|
|
||||||
|
4. **Checklist items**: each item has three parts — the category id (e.g.,
|
||||||
|
`concurrency-and-locking`), the finding text (one sentence describing the bug
|
||||||
|
pattern), and the look-for hint (a concrete code-shape search). Tag each so the
|
||||||
|
reviewer knows the rationale.
|
||||||
|
|
||||||
|
## Sizing guidance
|
||||||
|
|
||||||
|
- **Too few items (<3)**: the focus is too narrow. Either merge with another focus, or
|
||||||
|
drop the focus and review inline.
|
||||||
|
- **Too many items (>15)**: the focus is too broad. Split it (e.g., one focus per
|
||||||
|
function, not one per file).
|
||||||
|
- **Sweet spot**: 5-12 items per focus. Each subagent should be able to apply each item
|
||||||
|
thoughtfully, not skim.
|
||||||
|
|
@ -0,0 +1,220 @@
|
||||||
|
---
|
||||||
|
name: tla-plus
|
||||||
|
version: "1.0.0"
|
||||||
|
description: Create, run, and verify TLA+ and PlusCal formal specifications. Use for modeling distributed systems, protocols, concurrent algorithms, state machines. Can compose specs from code, find divergences between spec and implementation, spot concurrency bugs and invariant violations. Use when asked to "write a TLA+ spec", "model check", "verify protocol", "find race conditions", or "formal verification".
|
||||||
|
---
|
||||||
|
|
||||||
|
# TLA+ Formal Specification Skill
|
||||||
|
|
||||||
|
Write non-trivial TLA+ specifications, run the TLC model checker, and bridge the gap between formal specs and implementation code.
|
||||||
|
|
||||||
|
## Setup (Run Once)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash SKILL_DIR/scripts/setup.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
This downloads `tla2tools.jar` (v1.8.0) and verifies Java 11+. The JAR is stored at `SKILL_DIR/lib/tla2tools.jar`.
|
||||||
|
|
||||||
|
## Quick Reference
|
||||||
|
|
||||||
|
Load the right reference for your task:
|
||||||
|
|
||||||
|
| Task | Reference |
|
||||||
|
|------|-----------|
|
||||||
|
| TLA+ syntax, operators, types | [references/language.md](references/language.md) |
|
||||||
|
| PlusCal syntax, processes, labels | [references/pluscal.md](references/pluscal.md) |
|
||||||
|
| Distributed systems patterns | [references/patterns/distributed-systems.md](references/patterns/distributed-systems.md) |
|
||||||
|
| Protocol specifications | [references/patterns/protocols.md](references/patterns/protocols.md) |
|
||||||
|
| Invariants, safety, liveness | [references/patterns/invariants.md](references/patterns/invariants.md) |
|
||||||
|
| Code ↔ spec mapping, bug finding | [references/patterns/code-to-spec.md](references/patterns/code-to-spec.md) |
|
||||||
|
|
||||||
|
**Load references on-demand** — read the ones relevant to the current task before writing any spec.
|
||||||
|
|
||||||
|
## Templates
|
||||||
|
|
||||||
|
Start from a template when creating a new spec:
|
||||||
|
|
||||||
|
| Template | Use For |
|
||||||
|
|----------|---------|
|
||||||
|
| [templates/basic.tla](templates/basic.tla) | Pure TLA+ state machine |
|
||||||
|
| [templates/pluscal.tla](templates/pluscal.tla) | PlusCal algorithm with processes |
|
||||||
|
| [templates/distributed.tla](templates/distributed.tla) | Distributed system with messages |
|
||||||
|
|
||||||
|
## Running Specs
|
||||||
|
|
||||||
|
### Full Check (Recommended)
|
||||||
|
|
||||||
|
Translates PlusCal (if present), parses, and model-checks in one step:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash SKILL_DIR/scripts/check.sh spec.tla --config spec.cfg
|
||||||
|
```
|
||||||
|
|
||||||
|
### Individual Tools
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Parse only (syntax check)
|
||||||
|
bash SKILL_DIR/scripts/parse.sh spec.tla
|
||||||
|
|
||||||
|
# Translate PlusCal to TLA+
|
||||||
|
bash SKILL_DIR/scripts/translate.sh spec.tla -nocfg
|
||||||
|
|
||||||
|
# Model check with TLC
|
||||||
|
bash SKILL_DIR/scripts/tlc.sh spec.tla --config spec.cfg --workers auto
|
||||||
|
bash SKILL_DIR/scripts/tlc.sh spec.tla --no-deadlock # suppress deadlock check
|
||||||
|
```
|
||||||
|
|
||||||
|
### Direct Java Invocation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
JAR="SKILL_DIR/lib/tla2tools.jar"
|
||||||
|
|
||||||
|
# Parse
|
||||||
|
java -cp "$JAR" tla2sany.SANY spec.tla
|
||||||
|
|
||||||
|
# Translate PlusCal
|
||||||
|
java -cp "$JAR" pcal.trans -nocfg spec.tla
|
||||||
|
|
||||||
|
# Model check
|
||||||
|
java -cp "$JAR" tlc2.TLC -workers auto -config spec.cfg spec.tla
|
||||||
|
|
||||||
|
# REPL
|
||||||
|
java -cp "$JAR" tlc2.REPL
|
||||||
|
|
||||||
|
# Dump state graph (for visualization)
|
||||||
|
java -cp "$JAR" tlc2.TLC -dump dot,actionlabels,colorize states.dot spec.tla
|
||||||
|
```
|
||||||
|
|
||||||
|
## Workflow: Writing a Spec
|
||||||
|
|
||||||
|
### 1. Understand the System
|
||||||
|
|
||||||
|
Before writing any TLA+:
|
||||||
|
- Identify the **concurrent agents** (processes, nodes, threads)
|
||||||
|
- Identify **shared mutable state** (queues, databases, locks, counters)
|
||||||
|
- Identify **the key safety property** ("what must never happen?")
|
||||||
|
- Identify **the key liveness property** ("what must eventually happen?")
|
||||||
|
|
||||||
|
### 2. Choose PlusCal vs Pure TLA+
|
||||||
|
|
||||||
|
**Use PlusCal when:**
|
||||||
|
- The system is naturally sequential/imperative
|
||||||
|
- You're modeling code (threads, goroutines, async tasks)
|
||||||
|
- You need `await`, `while`, `goto` semantics
|
||||||
|
- The audience is programmers
|
||||||
|
|
||||||
|
**Use Pure TLA+ when:**
|
||||||
|
- You need fine-grained fairness control
|
||||||
|
- The system is naturally a state machine
|
||||||
|
- You need interruptible/restartable actions
|
||||||
|
- You need refinement mappings
|
||||||
|
- A single label would need to update a variable twice
|
||||||
|
|
||||||
|
### 3. Start Small, Then Expand
|
||||||
|
|
||||||
|
1. Write the **simplest possible spec** that captures the core behavior
|
||||||
|
2. Add a **type invariant** immediately
|
||||||
|
3. Run TLC with small constants (2-3 nodes, 2-3 messages)
|
||||||
|
4. Add **safety invariants** one at a time, running after each
|
||||||
|
5. Add **liveness properties** with fairness
|
||||||
|
6. Increase constants to expand coverage
|
||||||
|
|
||||||
|
### 4. Config File
|
||||||
|
|
||||||
|
Every spec needs a `.cfg` file. Create one alongside the `.tla`:
|
||||||
|
|
||||||
|
```
|
||||||
|
SPECIFICATION Spec
|
||||||
|
|
||||||
|
\* Properties to check
|
||||||
|
INVARIANT TypeInvariant
|
||||||
|
INVARIANT Safety
|
||||||
|
|
||||||
|
\* Uncomment for liveness (slower)
|
||||||
|
\* PROPERTY Liveness
|
||||||
|
|
||||||
|
\* Constants
|
||||||
|
CONSTANT
|
||||||
|
NumNodes = 3
|
||||||
|
MaxMsgs = 5
|
||||||
|
NULL = NULL
|
||||||
|
|
||||||
|
\* Uncomment to bound state space
|
||||||
|
\* CONSTRAINT StateConstraint
|
||||||
|
|
||||||
|
\* Uncomment to ignore deadlocks
|
||||||
|
\* CHECK_DEADLOCK FALSE
|
||||||
|
```
|
||||||
|
|
||||||
|
## Workflow: Spec From Code
|
||||||
|
|
||||||
|
When the user has existing code and wants a spec:
|
||||||
|
|
||||||
|
1. **Read the code** — understand concurrency structure, shared state, synchronization
|
||||||
|
2. **Read** [references/patterns/code-to-spec.md](references/patterns/code-to-spec.md)
|
||||||
|
3. **Map constructs** — threads→processes, locks→await, channels→queues
|
||||||
|
4. **Abstract** — don't model every line, only concurrency-relevant operations
|
||||||
|
5. **Write spec** — start with the concurrent skeleton, add detail iteratively
|
||||||
|
6. **Run TLC** — find deadlocks, invariant violations, liveness failures
|
||||||
|
7. **Report** — translate any TLC error traces back to code execution paths
|
||||||
|
|
||||||
|
## Workflow: Finding Bugs
|
||||||
|
|
||||||
|
When comparing a spec to code, or looking for bugs:
|
||||||
|
|
||||||
|
1. **Read** [references/patterns/code-to-spec.md](references/patterns/code-to-spec.md)
|
||||||
|
2. **Write a spec** modeling the suspected buggy area
|
||||||
|
3. **Run with small constants** — TLC is exhaustive, small state space is fine
|
||||||
|
4. **Interpret error traces** — map TLA+ state sequences to code execution paths
|
||||||
|
5. **Check atomicity** — does the code do atomically what the spec's label does?
|
||||||
|
6. **Check completeness** — does the code handle all `either/or` branches?
|
||||||
|
7. **Check ordering** — does the code enforce the same happens-before relations?
|
||||||
|
|
||||||
|
If TLC finds a violation:
|
||||||
|
- The error trace is a **concrete counterexample** — a specific sequence of events
|
||||||
|
- Map each state in the trace to concrete code state
|
||||||
|
- The transition that causes the violation is the bug location
|
||||||
|
|
||||||
|
## Interpreting TLC Output
|
||||||
|
|
||||||
|
### Success
|
||||||
|
```
|
||||||
|
Model checking completed. No error has been found.
|
||||||
|
Estimates of the probability that TLC did not check all reachable states
|
||||||
|
because two distinct states had the same fingerprint:
|
||||||
|
calculated (optimistic): val: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
### Invariant Violation
|
||||||
|
```
|
||||||
|
Error: Invariant SafetyInvariant is violated.
|
||||||
|
The following behavior constitutes a counter-example:
|
||||||
|
State 1: <Initial predicate>
|
||||||
|
/\ var1 = ...
|
||||||
|
State 2: <Action line X>
|
||||||
|
/\ var1 = ... (changed values in red)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deadlock
|
||||||
|
```
|
||||||
|
Error: Deadlock reached.
|
||||||
|
```
|
||||||
|
→ No process can make progress. Check `await` conditions and process completion.
|
||||||
|
|
||||||
|
### Liveness Violation
|
||||||
|
```
|
||||||
|
Error: Temporal properties were violated.
|
||||||
|
```
|
||||||
|
→ Check fairness settings. Add `WF_vars` or `SF_vars`. Trace may include a "stuttering" suffix.
|
||||||
|
|
||||||
|
## Key Principles
|
||||||
|
|
||||||
|
1. **Type invariants first** — catch spec bugs before checking real properties
|
||||||
|
2. **Small constants** — 3 nodes finds most bugs; 10 nodes takes 1000× longer
|
||||||
|
3. **Safety before liveness** — invariants are faster to check
|
||||||
|
4. **One property at a time** — add incrementally, run after each addition
|
||||||
|
5. **Fairness is opt-in** — TLA+ assumes everything can crash by default
|
||||||
|
6. **`=>` with `\A`, `/\` with `\E`** — never use `=>` with `\E`
|
||||||
|
7. **Sequences are 1-indexed** — always
|
||||||
|
8. **Every action must specify all variables** — use `UNCHANGED` for untouched ones
|
||||||
|
|
@ -0,0 +1,248 @@
|
||||||
|
# TLA+ Language Reference
|
||||||
|
|
||||||
|
## Module Structure
|
||||||
|
|
||||||
|
```tla
|
||||||
|
---- MODULE ModuleName ----
|
||||||
|
EXTENDS Integers, Sequences, FiniteSets, TLC
|
||||||
|
CONSTANTS Const1, Const2
|
||||||
|
VARIABLES var1, var2
|
||||||
|
|
||||||
|
\* Operator definitions
|
||||||
|
\* ...
|
||||||
|
|
||||||
|
====
|
||||||
|
```
|
||||||
|
|
||||||
|
The module name MUST match the filename (without `.tla`).
|
||||||
|
|
||||||
|
## Types and Values
|
||||||
|
|
||||||
|
### Primitives
|
||||||
|
- **Integers**: `0`, `1`, `-5`. Requires `EXTENDS Integers` for arithmetic.
|
||||||
|
- **Strings**: `"hello"`. Only `=` and `#` work on strings. Used as opaque identifiers.
|
||||||
|
- **Booleans**: `TRUE`, `FALSE`.
|
||||||
|
- **Model values**: Uninterpreted constants, defined in cfg file.
|
||||||
|
|
||||||
|
### Complex Types
|
||||||
|
- **Sets**: `{1, 2, 3}`, `{}`. Unordered, unique elements. Same-type elements only.
|
||||||
|
- **Sequences (Tuples)**: `<<1, 2, 3>>`, `<<>>`. Ordered, 1-indexed.
|
||||||
|
- **Structures (Records)**: `[name |-> "Alice", age |-> 30]`. Access: `s.name` or `s["name"]`.
|
||||||
|
- **Functions**: `[x \in 1..10 |-> x * x]`. Access: `f[3]`.
|
||||||
|
|
||||||
|
## Operators
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* No-argument operator (constant)
|
||||||
|
MaxSize == 10
|
||||||
|
|
||||||
|
\* Parameterized operator
|
||||||
|
Max(a, b) == IF a > b THEN a ELSE b
|
||||||
|
|
||||||
|
\* LET for local definitions
|
||||||
|
ThreeMax(a, b, c) ==
|
||||||
|
LET M(x, y) == IF x > y THEN x ELSE y
|
||||||
|
IN M(M(a, b), c)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Boolean Logic
|
||||||
|
|
||||||
|
| Logic | Symbol | Math |
|
||||||
|
|-------|--------|------|
|
||||||
|
| and | `/\` | ∧ |
|
||||||
|
| or | `\/` | ∨ |
|
||||||
|
| not | `~` | ¬ |
|
||||||
|
| implies | `=>` | ⇒ |
|
||||||
|
| iff | `<=>` | ⇔ |
|
||||||
|
|
||||||
|
**Bullet-point notation** — whitespace-sensitive vertical layout:
|
||||||
|
```tla
|
||||||
|
/\ A
|
||||||
|
/\ \/ B
|
||||||
|
\/ C
|
||||||
|
/\ D
|
||||||
|
\* Means: A /\ (B \/ C) /\ D
|
||||||
|
```
|
||||||
|
|
||||||
|
## Set Operations
|
||||||
|
|
||||||
|
| Operation | Syntax |
|
||||||
|
|-----------|--------|
|
||||||
|
| Membership | `x \in S` |
|
||||||
|
| Not member | `x \notin S` |
|
||||||
|
| Subset-or-equal | `S \subseteq T` |
|
||||||
|
| Union | `S \union T` |
|
||||||
|
| Intersection | `S \intersect T` |
|
||||||
|
| Set difference | `S \ T` |
|
||||||
|
| Cardinality | `Cardinality(S)` (needs `FiniteSets`) |
|
||||||
|
| Power set | `SUBSET S` |
|
||||||
|
| Cartesian product | `S \X T` |
|
||||||
|
| Integer range | `a..b` |
|
||||||
|
| All booleans | `BOOLEAN` |
|
||||||
|
| Map | `{f(x) : x \in S}` |
|
||||||
|
| Filter | `{x \in S : P(x)}` |
|
||||||
|
|
||||||
|
## Sequence Operations (EXTENDS Sequences)
|
||||||
|
|
||||||
|
| Operation | Syntax |
|
||||||
|
|-----------|--------|
|
||||||
|
| Append | `Append(s, e)` |
|
||||||
|
| Concat | `s1 \o s2` |
|
||||||
|
| Head | `Head(s)` |
|
||||||
|
| Tail | `Tail(s)` (all but first) |
|
||||||
|
| Length | `Len(s)` |
|
||||||
|
| Subsequence | `SubSeq(s, from, to)` |
|
||||||
|
| Index | `s[i]` (1-indexed!) |
|
||||||
|
|
||||||
|
Helper: `Range(s) == {s[i] : i \in 1..Len(s)}`
|
||||||
|
|
||||||
|
## Function Operations
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Function definition
|
||||||
|
f == [x \in 1..10 |-> x * x]
|
||||||
|
|
||||||
|
\* Function set (all functions from S to T)
|
||||||
|
[S -> T]
|
||||||
|
|
||||||
|
\* EXCEPT for updating functions
|
||||||
|
f' = [f EXCEPT ![key] = newval]
|
||||||
|
f' = [f EXCEPT ![k1] = v1, ![k2] = v2]
|
||||||
|
f' = [f EXCEPT ![k] = @ + 1] \* @ = old value
|
||||||
|
```
|
||||||
|
|
||||||
|
## Structures (Records)
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Create
|
||||||
|
msg == [type |-> "request", from |-> "A", data |-> 42]
|
||||||
|
|
||||||
|
\* Access
|
||||||
|
msg.type \* "request"
|
||||||
|
msg["type"] \* "request"
|
||||||
|
|
||||||
|
\* Set of all possible records
|
||||||
|
MsgType == [type: {"request", "response"}, from: Servers, data: Nat]
|
||||||
|
|
||||||
|
\* Update with EXCEPT
|
||||||
|
msg' = [msg EXCEPT !.data = 99]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quantifiers
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* For all
|
||||||
|
\A x \in S: P(x)
|
||||||
|
|
||||||
|
\* There exists
|
||||||
|
\E x \in S: P(x)
|
||||||
|
|
||||||
|
\* Multiple variables
|
||||||
|
\A x \in S, y \in T: P(x, y)
|
||||||
|
|
||||||
|
\* CHOOSE — deterministic selection
|
||||||
|
CHOOSE x \in S: P(x)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Critical rules**:
|
||||||
|
- `\A x \in {}: P(x)` is always TRUE
|
||||||
|
- `\E x \in {}: P(x)` is always FALSE
|
||||||
|
- Use `=>` with `\A`, use `/\` with `\E` (never `=>` with `\E`)
|
||||||
|
|
||||||
|
## IF-THEN-ELSE and CASE
|
||||||
|
|
||||||
|
```tla
|
||||||
|
IF cond THEN expr1 ELSE expr2
|
||||||
|
|
||||||
|
CASE x = 1 -> "one"
|
||||||
|
[] x = 2 -> "two"
|
||||||
|
[] OTHER -> "many"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Temporal Operators
|
||||||
|
|
||||||
|
| Operator | Name | Meaning |
|
||||||
|
|----------|------|---------|
|
||||||
|
| `[]P` | Always (box) | P is true in every state |
|
||||||
|
| `<>P` | Eventually (diamond) | P is true in at least one state |
|
||||||
|
| `<>[]P` | Eventually always | P becomes permanently true |
|
||||||
|
| `[]<>P` | Infinitely often | P is true infinitely often |
|
||||||
|
| `P ~> Q` | Leads-to | If P is true, Q is eventually true |
|
||||||
|
|
||||||
|
## Actions (Primed Variables)
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* x' is the value of x in the NEXT state
|
||||||
|
Next == x' = x + 1
|
||||||
|
|
||||||
|
\* UNCHANGED — variable stays the same
|
||||||
|
UNCHANGED x
|
||||||
|
UNCHANGED <<x, y, z>>
|
||||||
|
|
||||||
|
\* Box action formula
|
||||||
|
[][A]_v == [](A \/ UNCHANGED v)
|
||||||
|
|
||||||
|
\* Angle action formula
|
||||||
|
<<A>>_v == A /\ (v # v')
|
||||||
|
```
|
||||||
|
|
||||||
|
## Spec Structure (Pure TLA+)
|
||||||
|
|
||||||
|
```tla
|
||||||
|
vars == <<x, y, z>>
|
||||||
|
Init == x = 0 /\ y = 0 /\ z = 0
|
||||||
|
Next == Action1 \/ Action2 \/ Action3
|
||||||
|
Spec == Init /\ [][Next]_vars
|
||||||
|
|
||||||
|
\* With fairness
|
||||||
|
Spec == Init /\ [][Next]_vars /\ WF_vars(Next)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fairness
|
||||||
|
|
||||||
|
```tla
|
||||||
|
WF_v(A) \* Weak fairness: if A is eventually always enabled, it eventually happens
|
||||||
|
SF_v(A) \* Strong fairness: if A is always eventually enabled, it eventually happens
|
||||||
|
ENABLED A \* True if action A can happen in current state
|
||||||
|
```
|
||||||
|
|
||||||
|
## Standard Modules
|
||||||
|
|
||||||
|
| Module | Provides |
|
||||||
|
|--------|----------|
|
||||||
|
| `Integers` | `+`, `-`, `*`, `\div`, `%`, `..`, `Int`, `Nat` |
|
||||||
|
| `Naturals` | Same as Integers but only naturals |
|
||||||
|
| `Sequences` | `Append`, `Head`, `Tail`, `Len`, `SubSeq`, `\o`, `Seq(S)` |
|
||||||
|
| `FiniteSets` | `Cardinality`, `IsFiniteSet` |
|
||||||
|
| `TLC` | `Print`, `Assert`, `:>`, `@@` (function constructors) |
|
||||||
|
| `Bags` | Bag operations |
|
||||||
|
|
||||||
|
## TLC-specific Operators (EXTENDS TLC)
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Function constructor: key :> value
|
||||||
|
f == "a" :> 1 @@ "b" :> 2 \* Record-like function
|
||||||
|
|
||||||
|
\* Print (debugging)
|
||||||
|
Print(expr, val) \* prints expr, evaluates to val
|
||||||
|
|
||||||
|
\* Assert
|
||||||
|
Assert(cond, msg)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Config File (.cfg)
|
||||||
|
|
||||||
|
```
|
||||||
|
SPECIFICATION Spec
|
||||||
|
|
||||||
|
INVARIANT TypeInvariant
|
||||||
|
INVARIANT SafetyInvariant
|
||||||
|
|
||||||
|
PROPERTY Liveness
|
||||||
|
|
||||||
|
CONSTANT
|
||||||
|
NumServers = 3
|
||||||
|
NULL = NULL
|
||||||
|
|
||||||
|
CHECK_DEADLOCK FALSE
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,202 @@
|
||||||
|
# Code-to-Spec and Spec-to-Code Patterns
|
||||||
|
|
||||||
|
## When to Write a Spec From Code
|
||||||
|
|
||||||
|
1. **Concurrent code**: Locks, channels, async workflows, thread pools
|
||||||
|
2. **Distributed systems**: Consensus, replication, failover
|
||||||
|
3. **State machines**: Protocol handlers, workflow engines, parsers
|
||||||
|
4. **Complex invariants**: Financial calculations, resource allocation
|
||||||
|
5. **Bug reproduction**: Reproduce race conditions or deadlocks
|
||||||
|
|
||||||
|
## Code → Spec Translation Strategy
|
||||||
|
|
||||||
|
### Step 1: Identify the Abstraction Level
|
||||||
|
|
||||||
|
Don't model every line. Model the **concurrency-relevant** operations:
|
||||||
|
- Shared state mutations
|
||||||
|
- Message sends/receives
|
||||||
|
- Lock acquisitions/releases
|
||||||
|
- State transitions
|
||||||
|
- Decision points
|
||||||
|
|
||||||
|
### Step 2: Map Code Constructs to TLA+
|
||||||
|
|
||||||
|
| Code Construct | TLA+ Equivalent |
|
||||||
|
|----------------|-----------------|
|
||||||
|
| Thread / goroutine / async task | `process` |
|
||||||
|
| Mutex lock | `await lock = NULL; lock := self` |
|
||||||
|
| Channel send (buffered) | `await Len(chan) < cap; chan := Append(chan, msg)` |
|
||||||
|
| Channel send (unbuffered) | `procedure` with two-phase handshake |
|
||||||
|
| Condition variable / await | `await predicate` |
|
||||||
|
| Atomic CAS | Single label with conditional update |
|
||||||
|
| try-finally / defer | Explicit label for cleanup |
|
||||||
|
| Loop | `while` |
|
||||||
|
| Map/Dict | `function [key \in Keys |-> val]` |
|
||||||
|
| Enum / tagged union | Set of strings + discriminator field |
|
||||||
|
| Null / None / nil | `NULL` constant (model value) |
|
||||||
|
| Thread-local variable | Process-local `variables` |
|
||||||
|
| Global shared state | Module-level `variables` |
|
||||||
|
|
||||||
|
### Step 3: Write the Spec
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Example: Go worker pool with bounded channel
|
||||||
|
---- MODULE WorkerPool ----
|
||||||
|
EXTENDS Integers, Sequences, TLC
|
||||||
|
|
||||||
|
CONSTANTS NumWorkers, NumJobs, ChanSize, NULL
|
||||||
|
|
||||||
|
Workers == 1..NumWorkers
|
||||||
|
Jobs == 1..NumJobs
|
||||||
|
|
||||||
|
(*--algorithm worker_pool
|
||||||
|
variables
|
||||||
|
jobChan = <<>>;
|
||||||
|
results = [j \in Jobs |-> NULL];
|
||||||
|
submitted = 0;
|
||||||
|
completed = 0;
|
||||||
|
|
||||||
|
define
|
||||||
|
TypeInvariant ==
|
||||||
|
/\ Len(jobChan) <= ChanSize
|
||||||
|
/\ submitted <= NumJobs
|
||||||
|
/\ completed <= submitted
|
||||||
|
|
||||||
|
AllJobsDone == completed = NumJobs
|
||||||
|
Correct == AllJobsDone => \A j \in Jobs: results[j] # NULL
|
||||||
|
end define;
|
||||||
|
|
||||||
|
process producer = 0
|
||||||
|
variables nextJob = 1;
|
||||||
|
begin
|
||||||
|
Submit:
|
||||||
|
while nextJob <= NumJobs do
|
||||||
|
await Len(jobChan) < ChanSize;
|
||||||
|
jobChan := Append(jobChan, nextJob);
|
||||||
|
submitted := submitted + 1;
|
||||||
|
nextJob := nextJob + 1;
|
||||||
|
end while;
|
||||||
|
end process;
|
||||||
|
|
||||||
|
fair process worker \in Workers
|
||||||
|
variables job = NULL;
|
||||||
|
begin
|
||||||
|
Work:
|
||||||
|
while completed < NumJobs do
|
||||||
|
Dequeue:
|
||||||
|
await jobChan # <<>>;
|
||||||
|
job := Head(jobChan);
|
||||||
|
jobChan := Tail(jobChan);
|
||||||
|
Process:
|
||||||
|
results[job] := job * 10; \* abstract computation
|
||||||
|
completed := completed + 1;
|
||||||
|
end while;
|
||||||
|
end process;
|
||||||
|
end algorithm; *)
|
||||||
|
====
|
||||||
|
```
|
||||||
|
|
||||||
|
## Spec → Code Verification
|
||||||
|
|
||||||
|
### Finding Differences Between Spec and Code
|
||||||
|
|
||||||
|
1. **List all state variables** in the spec. Map each to code variables/fields.
|
||||||
|
2. **List all actions** in the spec. Map each to code functions/methods.
|
||||||
|
3. **Check atomicity**: Does the code perform what one TLA+ label does in a single atomic step?
|
||||||
|
4. **Check ordering**: Does the code enforce the same ordering constraints?
|
||||||
|
5. **Check completeness**: Does the code handle all `either/or` branches?
|
||||||
|
|
||||||
|
### Common Divergence Points
|
||||||
|
|
||||||
|
| Spec Says | Code Does | Bug? |
|
||||||
|
|-----------|-----------|------|
|
||||||
|
| Atomic label | Multiple statements without lock | Race condition |
|
||||||
|
| `await condition` | Polling without check | Missed wakeup |
|
||||||
|
| `either A or B` | Only implements A | Missing error handling |
|
||||||
|
| `\E x \in S` | Picks first/hardcoded | Correctness depends on choice |
|
||||||
|
| Set of messages | FIFO queue | May hide ordering bugs |
|
||||||
|
| Bounded variable | Unbounded counter | Overflow |
|
||||||
|
| `UNCHANGED rest` | Modifies extra state | Side effect bug |
|
||||||
|
| Fair process | Thread can starve | Liveness violation |
|
||||||
|
|
||||||
|
### Systematic Code Review Against Spec
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## Verification Checklist
|
||||||
|
|
||||||
|
### State Mapping
|
||||||
|
- Variable X in spec → field Y in code
|
||||||
|
- Type constraints match (spec range ⊆ code type range)
|
||||||
|
|
||||||
|
### Action Mapping
|
||||||
|
- Action A in spec → method M in code
|
||||||
|
- Precondition (await/guard) correctly checked
|
||||||
|
- State update matches
|
||||||
|
- Atomicity level matches
|
||||||
|
- All UNCHANGED variables truly unchanged
|
||||||
|
|
||||||
|
### Invariant Mapping
|
||||||
|
- TypeInvariant → type system + assertions
|
||||||
|
- SafetyInvariant → defensive checks / tests
|
||||||
|
- Liveness → timeout / retry / monitoring
|
||||||
|
|
||||||
|
### Edge Cases
|
||||||
|
- Empty collections handled (spec: `\A x \in {}: TRUE`)
|
||||||
|
- Concurrent access protected
|
||||||
|
- Error/exception paths modeled
|
||||||
|
```
|
||||||
|
|
||||||
|
## Spotting Bugs in Code Using a Spec
|
||||||
|
|
||||||
|
### Step 1: Write a Minimal Spec
|
||||||
|
Focus on the suspected buggy area. You don't need to model the entire system.
|
||||||
|
|
||||||
|
### Step 2: Run with Small Constants
|
||||||
|
```
|
||||||
|
CONSTANT
|
||||||
|
NumNodes = 3
|
||||||
|
MaxMsgs = 5
|
||||||
|
NULL = NULL
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Look for These Symptoms
|
||||||
|
|
||||||
|
| TLC Says | Likely Code Bug |
|
||||||
|
|----------|----------------|
|
||||||
|
| Deadlock | Missing wakeup, lock ordering issue |
|
||||||
|
| Invariant violated | Logic error, missing check |
|
||||||
|
| Liveness violated | Starvation, missing fairness, live-lock |
|
||||||
|
| State space explosion | Model too detailed — abstract more |
|
||||||
|
| Assert failed | Impossible state reached — logic error |
|
||||||
|
|
||||||
|
### Step 4: Translate Error Trace to Code Path
|
||||||
|
|
||||||
|
TLC gives you a step-by-step state sequence. Walk through your code following the same sequence of events to reproduce the bug.
|
||||||
|
|
||||||
|
## Modeling Existing Protocols
|
||||||
|
|
||||||
|
When verifying an implementation of a known protocol (Raft, Paxos, 2PC, etc.):
|
||||||
|
|
||||||
|
1. Start from the paper's pseudocode or description
|
||||||
|
2. Model one step at a time, checking after each addition
|
||||||
|
3. Use the **same variable names** as the paper when possible
|
||||||
|
4. Write invariants from the paper's correctness claims
|
||||||
|
5. Run with small constants first, increase gradually
|
||||||
|
|
||||||
|
### Bounding the Model
|
||||||
|
|
||||||
|
Real systems have unbounded state. TLA+ models must be finite.
|
||||||
|
|
||||||
|
```tla
|
||||||
|
CONSTANTS
|
||||||
|
MaxTerm = 3 \* bound election terms
|
||||||
|
MaxLogLen = 4 \* bound log length
|
||||||
|
MaxMsgs = 10 \* bound messages in flight
|
||||||
|
|
||||||
|
\* Use as state constraint in .cfg:
|
||||||
|
\* CONSTRAINT StateConstraint
|
||||||
|
StateConstraint ==
|
||||||
|
/\ \A n \in Nodes: term[n] <= MaxTerm
|
||||||
|
/\ \A n \in Nodes: Len(log[n]) <= MaxLogLen
|
||||||
|
/\ Cardinality(msgs) <= MaxMsgs
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,265 @@
|
||||||
|
# Distributed Systems Patterns
|
||||||
|
|
||||||
|
## Message Passing with Queues
|
||||||
|
|
||||||
|
The most common pattern for distributed systems. Model each node as a process, communication as queue operations.
|
||||||
|
|
||||||
|
### FIFO Queue (Mutable)
|
||||||
|
|
||||||
|
```tla
|
||||||
|
variables
|
||||||
|
queue = <<>>; \* sequence of messages
|
||||||
|
|
||||||
|
\* Send
|
||||||
|
queue := Append(queue, msg);
|
||||||
|
|
||||||
|
\* Receive (destructive)
|
||||||
|
await queue # <<>>;
|
||||||
|
msg := Head(queue);
|
||||||
|
queue := Tail(queue);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Per-Process Queues
|
||||||
|
|
||||||
|
```tla
|
||||||
|
variables
|
||||||
|
queues = [p \in Processes |-> <<>>]; \* each process has inbox
|
||||||
|
|
||||||
|
\* Send to one process
|
||||||
|
queues[dest] := Append(queues[dest], msg);
|
||||||
|
|
||||||
|
\* Broadcast to all
|
||||||
|
queues := [p \in Processes |-> Append(queues[p], msg)];
|
||||||
|
|
||||||
|
\* Send to subset (models unreliable delivery)
|
||||||
|
with recipients \in SUBSET Processes do
|
||||||
|
queues := [p \in recipients |-> Append(queues[p], msg)] @@ queues;
|
||||||
|
end with;
|
||||||
|
|
||||||
|
\* Receive from own queue
|
||||||
|
await queues[self] # <<>>;
|
||||||
|
msg := Head(queues[self]);
|
||||||
|
queues[self] := Tail(queues[self]);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Message Types
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Typed messages with discriminator
|
||||||
|
RequestMsg == [type: {"request"}, from: Nodes, data: DataType, id: 1..MaxId]
|
||||||
|
ResponseMsg == [type: {"response"}, from: Nodes, data: DataType, id: 1..MaxId]
|
||||||
|
MessageType == RequestMsg \union ResponseMsg
|
||||||
|
```
|
||||||
|
|
||||||
|
## Network Failures
|
||||||
|
|
||||||
|
### Message Loss (At-Most-Once Delivery)
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Instead of deterministic send, nondeterministically drop
|
||||||
|
either
|
||||||
|
queues[dest] := Append(queues[dest], msg); \* delivered
|
||||||
|
or
|
||||||
|
skip; \* lost
|
||||||
|
end either;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Message Reordering
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Use a SET instead of a sequence for the channel
|
||||||
|
variables network = [p \in Processes |-> {}];
|
||||||
|
|
||||||
|
\* Send
|
||||||
|
network[dest] := network[dest] \union {msg};
|
||||||
|
|
||||||
|
\* Receive any message (unordered)
|
||||||
|
with m \in network[self] do
|
||||||
|
msg := m;
|
||||||
|
network[self] := network[self] \ {m};
|
||||||
|
end with;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Node Crashes
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Model with nondeterministic crash action
|
||||||
|
process node \in Nodes
|
||||||
|
variables alive = TRUE;
|
||||||
|
begin
|
||||||
|
Run:
|
||||||
|
while alive do
|
||||||
|
either
|
||||||
|
\* normal operation
|
||||||
|
or
|
||||||
|
alive := FALSE; \* crash
|
||||||
|
end either;
|
||||||
|
end while;
|
||||||
|
end process;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Consensus Patterns
|
||||||
|
|
||||||
|
### Two-Phase Commit
|
||||||
|
|
||||||
|
```tla
|
||||||
|
---- MODULE TwoPhaseCommit ----
|
||||||
|
EXTENDS Integers, Sequences, FiniteSets, TLC
|
||||||
|
|
||||||
|
CONSTANTS Coordinators, Participants, NULL
|
||||||
|
ASSUME Cardinality(Coordinators) = 1
|
||||||
|
|
||||||
|
(*--algorithm two_phase_commit
|
||||||
|
variables
|
||||||
|
coord_state = [c \in Coordinators |-> "init"];
|
||||||
|
part_state = [p \in Participants |-> "working"];
|
||||||
|
prepared = [c \in Coordinators |-> {}];
|
||||||
|
msgs = {};
|
||||||
|
|
||||||
|
define
|
||||||
|
TypeInvariant ==
|
||||||
|
/\ coord_state \in [Coordinators -> {"init", "waiting", "committed", "aborted"}]
|
||||||
|
/\ part_state \in [Participants -> {"working", "prepared", "committed", "aborted"}]
|
||||||
|
|
||||||
|
\* Safety: no participant commits while another aborts
|
||||||
|
Consistency ==
|
||||||
|
\A p1, p2 \in Participants:
|
||||||
|
~(part_state[p1] = "committed" /\ part_state[p2] = "aborted")
|
||||||
|
end define;
|
||||||
|
|
||||||
|
process coordinator \in Coordinators
|
||||||
|
begin
|
||||||
|
C_Prepare:
|
||||||
|
coord_state[self] := "waiting";
|
||||||
|
msgs := msgs \union {[type |-> "prepare", from |-> self]};
|
||||||
|
|
||||||
|
C_Decide:
|
||||||
|
await prepared[self] = Participants \/ \E p \in Participants: part_state[p] = "aborted";
|
||||||
|
if prepared[self] = Participants then
|
||||||
|
coord_state[self] := "committed";
|
||||||
|
msgs := msgs \union {[type |-> "commit", from |-> self]};
|
||||||
|
else
|
||||||
|
coord_state[self] := "aborted";
|
||||||
|
msgs := msgs \union {[type |-> "abort", from |-> self]};
|
||||||
|
end if;
|
||||||
|
end process;
|
||||||
|
|
||||||
|
process participant \in Participants
|
||||||
|
begin
|
||||||
|
P_Prepare:
|
||||||
|
await \E m \in msgs: m.type = "prepare";
|
||||||
|
either
|
||||||
|
part_state[self] := "prepared";
|
||||||
|
with c \in Coordinators do
|
||||||
|
prepared[c] := prepared[c] \union {self};
|
||||||
|
end with;
|
||||||
|
or
|
||||||
|
part_state[self] := "aborted";
|
||||||
|
end either;
|
||||||
|
|
||||||
|
P_Decide:
|
||||||
|
either
|
||||||
|
await \E m \in msgs: m.type = "commit";
|
||||||
|
part_state[self] := "committed";
|
||||||
|
or
|
||||||
|
await \E m \in msgs: m.type = "abort";
|
||||||
|
part_state[self] := "aborted";
|
||||||
|
end either;
|
||||||
|
end process;
|
||||||
|
end algorithm; *)
|
||||||
|
====
|
||||||
|
```
|
||||||
|
|
||||||
|
### Leader Election (Bully Algorithm Sketch)
|
||||||
|
|
||||||
|
```tla
|
||||||
|
variables
|
||||||
|
leader = NULL;
|
||||||
|
election_in_progress = FALSE;
|
||||||
|
candidates = {};
|
||||||
|
|
||||||
|
define
|
||||||
|
\* At most one leader at any time
|
||||||
|
AtMostOneLeader ==
|
||||||
|
\A n1, n2 \in Nodes:
|
||||||
|
(leader = n1 /\ leader = n2) => n1 = n2
|
||||||
|
|
||||||
|
\* Eventually a leader is elected (liveness)
|
||||||
|
EventuallyLeader == <>(leader # NULL)
|
||||||
|
end define;
|
||||||
|
```
|
||||||
|
|
||||||
|
## State Machine Pattern
|
||||||
|
|
||||||
|
Model systems as explicit state machines with transition functions.
|
||||||
|
|
||||||
|
```tla
|
||||||
|
VARIABLE state
|
||||||
|
|
||||||
|
Trans(from, to) ==
|
||||||
|
/\ state = from
|
||||||
|
/\ state' = to
|
||||||
|
|
||||||
|
Init == state = "idle"
|
||||||
|
|
||||||
|
Next ==
|
||||||
|
\/ Trans("idle", "connecting")
|
||||||
|
\/ Trans("connecting", "connected")
|
||||||
|
\/ Trans("connecting", "failed")
|
||||||
|
\/ Trans("connected", "disconnecting")
|
||||||
|
\/ Trans("disconnecting", "idle")
|
||||||
|
\/ Trans("failed", "idle")
|
||||||
|
|
||||||
|
Spec == Init /\ [][Next]_state
|
||||||
|
```
|
||||||
|
|
||||||
|
## Clocks and Logical Time
|
||||||
|
|
||||||
|
### Lamport Clocks
|
||||||
|
|
||||||
|
```tla
|
||||||
|
variables
|
||||||
|
clock = [p \in Processes |-> 0];
|
||||||
|
|
||||||
|
\* Local event
|
||||||
|
clock[self] := clock[self] + 1;
|
||||||
|
|
||||||
|
\* Send: increment, attach clock
|
||||||
|
clock[self] := clock[self] + 1;
|
||||||
|
msg := [data |-> payload, ts |-> clock[self]];
|
||||||
|
|
||||||
|
\* Receive: merge clocks
|
||||||
|
clock[self] := IF msg.ts > clock[self]
|
||||||
|
THEN msg.ts + 1
|
||||||
|
ELSE clock[self] + 1;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quorum Systems
|
||||||
|
|
||||||
|
```tla
|
||||||
|
CONSTANT Nodes
|
||||||
|
|
||||||
|
Quorum == {Q \in SUBSET Nodes : Cardinality(Q) * 2 > Cardinality(Nodes)}
|
||||||
|
|
||||||
|
\* Any two quorums overlap
|
||||||
|
QuorumOverlap == \A Q1, Q2 \in Quorum: Q1 \intersect Q2 # {}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Replication
|
||||||
|
|
||||||
|
```tla
|
||||||
|
variables
|
||||||
|
replicas = [n \in Nodes |-> [key \in Keys |-> NULL]];
|
||||||
|
|
||||||
|
define
|
||||||
|
\* Eventually consistent: if no more writes, all replicas converge
|
||||||
|
EventualConsistency ==
|
||||||
|
<>[](\A n1, n2 \in Nodes, k \in Keys:
|
||||||
|
replicas[n1][k] = replicas[n2][k])
|
||||||
|
|
||||||
|
\* Strong consistency: all replicas always agree
|
||||||
|
StrongConsistency ==
|
||||||
|
\A n1, n2 \in Nodes, k \in Keys:
|
||||||
|
replicas[n1][k] = replicas[n2][k]
|
||||||
|
end define;
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,211 @@
|
||||||
|
# Invariants and Properties Guide
|
||||||
|
|
||||||
|
## Types of Properties
|
||||||
|
|
||||||
|
| Type | Checks | Example |
|
||||||
|
|------|--------|---------|
|
||||||
|
| **Invariant** | Every state individually | `counter >= 0` |
|
||||||
|
| **Action property** | Every transition | `counter' >= counter` |
|
||||||
|
| **Safety (temporal)** | No bad behavior sequence | `\E s \in S: [](s \in online)` |
|
||||||
|
| **Liveness** | Good things eventually happen | `<>(status = "done")` |
|
||||||
|
|
||||||
|
## Type Invariants
|
||||||
|
|
||||||
|
The first invariant to write. Constrains each variable to its valid domain.
|
||||||
|
|
||||||
|
```tla
|
||||||
|
TypeInvariant ==
|
||||||
|
/\ counter \in 0..MaxVal
|
||||||
|
/\ lock \in Processes \union {NULL}
|
||||||
|
/\ queue \in Seq(MessageType)
|
||||||
|
/\ state \in [Nodes -> {"idle", "active", "done"}]
|
||||||
|
/\ seen \subseteq AllItems
|
||||||
|
/\ pc \in [Processes -> {"Init", "Work", "Done"}]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rules**:
|
||||||
|
- Be as tight as possible. `counter \in 0..MaxVal` beats `counter \in Int`.
|
||||||
|
- Include `pc` if you need label-dependent reasoning.
|
||||||
|
- Type invariants catch spec bugs early — always write them first.
|
||||||
|
|
||||||
|
## Safety Invariants
|
||||||
|
|
||||||
|
Assert that dangerous states never occur.
|
||||||
|
|
||||||
|
### Mutual Exclusion
|
||||||
|
```tla
|
||||||
|
MutualExclusion ==
|
||||||
|
\A p1, p2 \in Processes:
|
||||||
|
(in_cs[p1] /\ in_cs[p2]) => p1 = p2
|
||||||
|
|
||||||
|
\* Equivalent with cardinality:
|
||||||
|
MutualExclusion ==
|
||||||
|
Cardinality({p \in Processes : in_cs[p]}) <= 1
|
||||||
|
```
|
||||||
|
|
||||||
|
### No Overdraft
|
||||||
|
```tla
|
||||||
|
NoOverdraft ==
|
||||||
|
\A a \in Accounts: balance[a] >= 0
|
||||||
|
```
|
||||||
|
|
||||||
|
### Conservation (Closed System)
|
||||||
|
```tla
|
||||||
|
\* Total money in system never changes
|
||||||
|
Conservation ==
|
||||||
|
LET total == SumAll(balance)
|
||||||
|
IN total = InitialTotal
|
||||||
|
```
|
||||||
|
|
||||||
|
### Consistency Across Replicas
|
||||||
|
```tla
|
||||||
|
\* If a value is committed, all replicas agree
|
||||||
|
Consistency ==
|
||||||
|
\A k \in Keys:
|
||||||
|
committed[k] =>
|
||||||
|
\A r1, r2 \in Replicas:
|
||||||
|
data[r1][k] = data[r2][k]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Conditional Invariants with `=>`
|
||||||
|
|
||||||
|
Use implication to restrict when an invariant applies.
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Only check after algorithm completes
|
||||||
|
Correct == pc = "Done" => result = ExpectedResult
|
||||||
|
|
||||||
|
\* Only check under certain conditions
|
||||||
|
QueueBound == is_running => Len(queue) <= MaxQueueSize
|
||||||
|
|
||||||
|
\* Process-specific
|
||||||
|
WorkerCorrect ==
|
||||||
|
\A w \in Workers:
|
||||||
|
pc[w] = "Done" => output[w] = f(input[w])
|
||||||
|
```
|
||||||
|
|
||||||
|
## Action Properties
|
||||||
|
|
||||||
|
Check transitions, not individual states. Use `[][...]_vars` syntax.
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Monotonically increasing counter
|
||||||
|
CounterMonotonic == [][counter' >= counter]_counter
|
||||||
|
|
||||||
|
\* Lock ownership can't jump between processes
|
||||||
|
LockNotStolen ==
|
||||||
|
[][lock \in Processes => (lock' = lock \/ lock' = NULL)]_lock
|
||||||
|
|
||||||
|
\* Version numbers only increase
|
||||||
|
VersionMonotonic ==
|
||||||
|
[][\A n \in Nodes: version[n]' >= version[n]]_version
|
||||||
|
```
|
||||||
|
|
||||||
|
**Remember**: `[][A]_v` means `[](A \/ UNCHANGED v)` — stuttering is always allowed.
|
||||||
|
|
||||||
|
## Liveness Properties
|
||||||
|
|
||||||
|
Assert that the system makes progress.
|
||||||
|
|
||||||
|
### Eventually Done
|
||||||
|
```tla
|
||||||
|
\* All processes eventually finish
|
||||||
|
Termination == <>(\A p \in Processes: pc[p] = "Done")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Eventually Consistent
|
||||||
|
```tla
|
||||||
|
EventualConsistency ==
|
||||||
|
<>(\A r1, r2 \in Replicas: data[r1] = data[r2])
|
||||||
|
```
|
||||||
|
|
||||||
|
### Leads-to
|
||||||
|
```tla
|
||||||
|
\* Every request eventually gets a response
|
||||||
|
RequestHandled ==
|
||||||
|
\A r \in Requests:
|
||||||
|
r \in pending ~> r \in completed
|
||||||
|
|
||||||
|
\* Every enqueued item eventually dequeued
|
||||||
|
QueueDrained ==
|
||||||
|
\A item \in Items:
|
||||||
|
item \in Range(queue) ~> item \notin Range(queue)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Starvation Freedom
|
||||||
|
```tla
|
||||||
|
\* Every process eventually enters critical section
|
||||||
|
NoStarvation ==
|
||||||
|
\A p \in Processes:
|
||||||
|
pc[p] = "Waiting" ~> pc[p] = "InCS"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composing Temporal Operators
|
||||||
|
|
||||||
|
| Pattern | Meaning | Use Case |
|
||||||
|
|---------|---------|----------|
|
||||||
|
| `<>[]P` | P becomes permanently true | Convergence, termination |
|
||||||
|
| `[]<>P` | P keeps happening | Repeated service, heartbeats |
|
||||||
|
| `[]<>P /\ []<>Q` | Both keep happening | Fair scheduling |
|
||||||
|
| `P ~> Q` | P always leads to Q | Request-response guarantee |
|
||||||
|
| `P ~> <>[]Q` | P leads to permanent Q | Recovery guarantee |
|
||||||
|
|
||||||
|
## Fairness Requirements
|
||||||
|
|
||||||
|
Liveness properties almost always need fairness to avoid trivial stuttering counterexamples.
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Weak fairness: if action is continuously enabled, it eventually executes
|
||||||
|
Spec == Init /\ [][Next]_vars /\ WF_vars(Next)
|
||||||
|
|
||||||
|
\* Per-action fairness
|
||||||
|
Spec == Init /\ [][Next]_vars
|
||||||
|
/\ \A p \in Processes: WF_vars(Process(p))
|
||||||
|
|
||||||
|
\* Strong fairness for lock-waiting actions
|
||||||
|
Spec == Init /\ [][Next]_vars
|
||||||
|
/\ \A p \in Processes: SF_vars(AcquireLock(p))
|
||||||
|
/\ \A p \in Processes: WF_vars(ReleaseLock(p))
|
||||||
|
```
|
||||||
|
|
||||||
|
**When to use which**:
|
||||||
|
- **WF**: action is continuously enabled (unconditional progress)
|
||||||
|
- **SF**: action is repeatedly enabled/disabled (e.g., waiting for a lock)
|
||||||
|
|
||||||
|
## Debugging Properties
|
||||||
|
|
||||||
|
### Print and Assert
|
||||||
|
```tla
|
||||||
|
\* In PlusCal (needs EXTENDS TLC)
|
||||||
|
assert x > 0;
|
||||||
|
print <<"x =", x, "y =", y>>;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Canary Invariants
|
||||||
|
```tla
|
||||||
|
\* Force a trace to see what states exist
|
||||||
|
DEBUG_SeenAllStates == Cardinality(explored) < MaxStates
|
||||||
|
|
||||||
|
\* Force a trace that reaches a specific state
|
||||||
|
DEBUG_NeverReaches == ~(state = "target_state")
|
||||||
|
```
|
||||||
|
|
||||||
|
### ALIAS in Config
|
||||||
|
```tla
|
||||||
|
\* In .cfg file: ALIAS DebugAlias
|
||||||
|
\* In .tla file:
|
||||||
|
DebugAlias == [
|
||||||
|
state |-> state,
|
||||||
|
state_next |-> state',
|
||||||
|
queue_len |-> Len(queue),
|
||||||
|
msg_count |-> Cardinality(msgs)
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common Mistakes
|
||||||
|
|
||||||
|
1. **Using `=>` with `\E`**: `\E x: P(x) => Q(x)` is almost certainly wrong. Use `\E x: P(x) /\ Q(x)`.
|
||||||
|
2. **Forgetting fairness**: Liveness properties fail trivially without fair processes.
|
||||||
|
3. **Invariant too strong**: Checking `result = expected` globally when it only holds at end — use `pc = "Done" => ...`.
|
||||||
|
4. **Missing `UNCHANGED`**: In pure TLA+, every action must specify all variables.
|
||||||
|
5. **Quantifier over indices**: `\A i, j \in 1..Len(s): s[i] # s[j]` fails when `i = j`. Use `i # j =>`.
|
||||||
|
|
@ -0,0 +1,201 @@
|
||||||
|
# Protocol Specification Patterns
|
||||||
|
|
||||||
|
## Protocol Structure Template
|
||||||
|
|
||||||
|
A protocol spec typically has:
|
||||||
|
1. **State variables** for each participant
|
||||||
|
2. **Message types** as sets of records
|
||||||
|
3. **Actions** for each protocol step (one per label/action)
|
||||||
|
4. **Invariants** for safety properties
|
||||||
|
5. **Liveness** for progress guarantees
|
||||||
|
|
||||||
|
```tla
|
||||||
|
---- MODULE ProtocolName ----
|
||||||
|
EXTENDS Integers, Sequences, FiniteSets, TLC
|
||||||
|
|
||||||
|
CONSTANTS Nodes, NULL, MaxMsgs
|
||||||
|
|
||||||
|
VARIABLES
|
||||||
|
node_state, \* [Nodes -> StateSet]
|
||||||
|
msgs, \* set or sequence of messages
|
||||||
|
history \* auxiliary variable for properties
|
||||||
|
|
||||||
|
vars == <<node_state, msgs, history>>
|
||||||
|
|
||||||
|
\* --- Message Types ---
|
||||||
|
MsgType == [type: {"req", "ack", "nack"}, from: Nodes, to: Nodes, data: DataType]
|
||||||
|
|
||||||
|
\* --- Helpers ---
|
||||||
|
Send(m) == msgs' = msgs \union {m}
|
||||||
|
Discard(m) == msgs' = msgs \ {m}
|
||||||
|
|
||||||
|
\* --- Actions ---
|
||||||
|
Init ==
|
||||||
|
/\ node_state = [n \in Nodes |-> "idle"]
|
||||||
|
/\ msgs = {}
|
||||||
|
/\ history = <<>>
|
||||||
|
|
||||||
|
SendRequest(n) ==
|
||||||
|
/\ node_state[n] = "idle"
|
||||||
|
/\ node_state' = [node_state EXCEPT ![n] = "waiting"]
|
||||||
|
/\ \E dest \in Nodes \ {n}:
|
||||||
|
Send([type |-> "req", from |-> n, to |-> dest, data |-> "payload"])
|
||||||
|
/\ UNCHANGED history
|
||||||
|
|
||||||
|
HandleRequest(n) ==
|
||||||
|
/\ \E m \in msgs:
|
||||||
|
/\ m.type = "req" /\ m.to = n
|
||||||
|
/\ node_state' = [node_state EXCEPT ![n] = "processing"]
|
||||||
|
/\ Send([type |-> "ack", from |-> n, to |-> m.from, data |-> m.data])
|
||||||
|
/\ Discard(m) \* careful: updating msgs twice — may need to combine
|
||||||
|
/\ UNCHANGED history
|
||||||
|
|
||||||
|
Next == \E n \in Nodes: SendRequest(n) \/ HandleRequest(n)
|
||||||
|
|
||||||
|
\* --- Properties ---
|
||||||
|
TypeInvariant ==
|
||||||
|
/\ node_state \in [Nodes -> {"idle", "waiting", "processing", "done"}]
|
||||||
|
/\ msgs \subseteq MsgType
|
||||||
|
|
||||||
|
Safety == \* domain-specific
|
||||||
|
|
||||||
|
Liveness == \A n \in Nodes: node_state[n] = "waiting" ~> node_state[n] = "done"
|
||||||
|
|
||||||
|
Spec == Init /\ [][Next]_vars /\ \A n \in Nodes: WF_vars(Next)
|
||||||
|
====
|
||||||
|
```
|
||||||
|
|
||||||
|
## Request-Response Protocol
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Client sends request, server responds
|
||||||
|
\* Invariant: every response corresponds to a prior request
|
||||||
|
|
||||||
|
VARIABLES
|
||||||
|
client_state, \* [Clients -> {"idle", "waiting", "done"}]
|
||||||
|
server_state, \* [Servers -> {"idle", "processing"}]
|
||||||
|
requests, \* set of request messages in flight
|
||||||
|
responses, \* set of response messages in flight
|
||||||
|
request_log \* auxiliary: tracks all requests ever sent
|
||||||
|
|
||||||
|
SendRequest(c) ==
|
||||||
|
/\ client_state[c] = "idle"
|
||||||
|
/\ client_state' = [client_state EXCEPT ![c] = "waiting"]
|
||||||
|
/\ \E s \in Servers:
|
||||||
|
requests' = requests \union {[from |-> c, to |-> s, id |-> next_id]}
|
||||||
|
/\ request_log' = request_log \union {[from |-> c, id |-> next_id]}
|
||||||
|
/\ UNCHANGED <<server_state, responses>>
|
||||||
|
|
||||||
|
HandleRequest(s) ==
|
||||||
|
/\ \E req \in requests:
|
||||||
|
/\ req.to = s
|
||||||
|
/\ server_state' = [server_state EXCEPT ![s] = "processing"]
|
||||||
|
/\ responses' = responses \union {[from |-> s, to |-> req.from, id |-> req.id]}
|
||||||
|
/\ requests' = requests \ {req}
|
||||||
|
/\ UNCHANGED <<client_state, request_log>>
|
||||||
|
|
||||||
|
\* Safety: every response has a matching request
|
||||||
|
ResponseMatchesRequest ==
|
||||||
|
\A resp \in responses:
|
||||||
|
\E req \in request_log: req.id = resp.id /\ req.from = resp.to
|
||||||
|
```
|
||||||
|
|
||||||
|
## Handshake Protocols
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Three-phase handshake (like TCP SYN/SYN-ACK/ACK)
|
||||||
|
VARIABLE
|
||||||
|
conn_state \* [Nodes \X Nodes -> {"closed", "syn_sent", "syn_rcvd", "established"}]
|
||||||
|
|
||||||
|
Init == conn_state = [p \in Nodes \X Nodes |-> "closed"]
|
||||||
|
|
||||||
|
SynSend(a, b) ==
|
||||||
|
/\ conn_state[<<a,b>>] = "closed"
|
||||||
|
/\ conn_state' = [conn_state EXCEPT ![<<a,b>>] = "syn_sent"]
|
||||||
|
/\ msgs' = msgs \union {[type |-> "SYN", from |-> a, to |-> b]}
|
||||||
|
|
||||||
|
SynAck(b, a) ==
|
||||||
|
/\ \E m \in msgs: m.type = "SYN" /\ m.from = a /\ m.to = b
|
||||||
|
/\ conn_state[<<b,a>>] = "closed"
|
||||||
|
/\ conn_state' = [conn_state EXCEPT ![<<b,a>>] = "syn_rcvd"]
|
||||||
|
/\ msgs' = msgs \union {[type |-> "SYN_ACK", from |-> b, to |-> a]}
|
||||||
|
|
||||||
|
Ack(a, b) ==
|
||||||
|
/\ \E m \in msgs: m.type = "SYN_ACK" /\ m.from = b /\ m.to = a
|
||||||
|
/\ conn_state[<<a,b>>] = "syn_sent"
|
||||||
|
/\ conn_state' = [conn_state EXCEPT ![<<a,b>>] = "established"]
|
||||||
|
/\ msgs' = msgs \union {[type |-> "ACK", from |-> a, to |-> b]}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Token-Based Protocols
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Mutual exclusion via token passing
|
||||||
|
VARIABLES
|
||||||
|
token_holder, \* Nodes \union {NULL}
|
||||||
|
in_cs \* [Nodes -> BOOLEAN]
|
||||||
|
|
||||||
|
TokenPass(from, to) ==
|
||||||
|
/\ token_holder = from
|
||||||
|
/\ ~in_cs[from]
|
||||||
|
/\ token_holder' = to
|
||||||
|
/\ UNCHANGED in_cs
|
||||||
|
|
||||||
|
EnterCS(n) ==
|
||||||
|
/\ token_holder = n
|
||||||
|
/\ ~in_cs[n]
|
||||||
|
/\ in_cs' = [in_cs EXCEPT ![n] = TRUE]
|
||||||
|
/\ UNCHANGED token_holder
|
||||||
|
|
||||||
|
ExitCS(n) ==
|
||||||
|
/\ in_cs[n]
|
||||||
|
/\ in_cs' = [in_cs EXCEPT ![n] = FALSE]
|
||||||
|
/\ UNCHANGED token_holder
|
||||||
|
|
||||||
|
\* Safety: at most one node in CS
|
||||||
|
MutualExclusion ==
|
||||||
|
\A n1, n2 \in Nodes:
|
||||||
|
(in_cs[n1] /\ in_cs[n2]) => n1 = n2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gossip Protocol
|
||||||
|
|
||||||
|
```tla
|
||||||
|
VARIABLES
|
||||||
|
knowledge \* [Nodes -> SUBSET Information]
|
||||||
|
|
||||||
|
Init == knowledge = [n \in Nodes |-> {info_n}] \* each knows its own info
|
||||||
|
|
||||||
|
Gossip(a, b) ==
|
||||||
|
/\ knowledge' = [knowledge EXCEPT
|
||||||
|
![a] = @ \union knowledge[b],
|
||||||
|
![b] = @ \union knowledge[a]]
|
||||||
|
|
||||||
|
\* Convergence: eventually everyone knows everything
|
||||||
|
AllInfo == UNION {knowledge[n] : n \in Nodes}
|
||||||
|
EventualConvergence == <>(\A n \in Nodes: knowledge[n] = AllInfo)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Retry with Backoff (Abstract)
|
||||||
|
|
||||||
|
```tla
|
||||||
|
VARIABLES
|
||||||
|
attempt, \* [Agents -> Nat]
|
||||||
|
status \* [Agents -> {"pending", "success", "failed"}]
|
||||||
|
|
||||||
|
Retry(a) ==
|
||||||
|
/\ status[a] = "failed"
|
||||||
|
/\ attempt[a] < MaxRetries
|
||||||
|
/\ attempt' = [attempt EXCEPT ![a] = @ + 1]
|
||||||
|
/\ status' = [status EXCEPT ![a] = "pending"]
|
||||||
|
|
||||||
|
Succeed(a) ==
|
||||||
|
/\ status[a] = "pending"
|
||||||
|
/\ status' = [status EXCEPT ![a] = "success"]
|
||||||
|
/\ UNCHANGED attempt
|
||||||
|
|
||||||
|
Fail(a) ==
|
||||||
|
/\ status[a] = "pending"
|
||||||
|
/\ status' = [status EXCEPT ![a] = "failed"]
|
||||||
|
/\ UNCHANGED attempt
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,251 @@
|
||||||
|
# PlusCal Reference
|
||||||
|
|
||||||
|
PlusCal is a higher-level language that compiles to TLA+. It provides imperative-style syntax while retaining full TLA+ expressiveness for operators and invariants.
|
||||||
|
|
||||||
|
## Algorithm Structure
|
||||||
|
|
||||||
|
```tla
|
||||||
|
---- MODULE MySpec ----
|
||||||
|
EXTENDS Integers, Sequences, TLC
|
||||||
|
|
||||||
|
\* Constants and operators can go here
|
||||||
|
|
||||||
|
(* --algorithm algorithm_name
|
||||||
|
variables
|
||||||
|
x = 0;
|
||||||
|
y \in 1..5; \* nondeterministic: TLC checks ALL starting values
|
||||||
|
|
||||||
|
define
|
||||||
|
\* Operators that can reference PlusCal variables
|
||||||
|
TypeInvariant == x \in Nat /\ y \in 1..5
|
||||||
|
end define;
|
||||||
|
|
||||||
|
begin
|
||||||
|
Label1:
|
||||||
|
x := x + 1;
|
||||||
|
Label2:
|
||||||
|
y := y - 1;
|
||||||
|
end algorithm; *)
|
||||||
|
|
||||||
|
\* BEGIN TRANSLATION (auto-generated TLA+)
|
||||||
|
\* END TRANSLATION
|
||||||
|
|
||||||
|
\* Post-translation operators (can reference pc, stack, etc.)
|
||||||
|
====
|
||||||
|
```
|
||||||
|
|
||||||
|
## Variables
|
||||||
|
|
||||||
|
```tla
|
||||||
|
variables
|
||||||
|
counter = 0; \* fixed initial value
|
||||||
|
seq = <<1, 2, 3>>; \* sequence
|
||||||
|
record = [a |-> 1, b |-> 2]; \* structure
|
||||||
|
x \in 1..10; \* nondeterministic: all values tested
|
||||||
|
items \in SUBSET {"a", "b", "c"}; \* all subsets tested
|
||||||
|
```
|
||||||
|
|
||||||
|
Using `\in` creates **multiple starting states**. TLC checks every combination.
|
||||||
|
|
||||||
|
## Labels
|
||||||
|
|
||||||
|
Labels define the **grain of atomicity**. Everything in one label happens in a single step.
|
||||||
|
|
||||||
|
### Rules
|
||||||
|
1. Algorithm body must start with a label
|
||||||
|
2. `while` loops must start with a label
|
||||||
|
3. Each variable can only be updated **once per label** (use `||` for simultaneous assignment)
|
||||||
|
4. `goto` must be followed by a label
|
||||||
|
5. `macro` and `with` blocks cannot contain labels
|
||||||
|
6. If any branch in `if`/`either` has a label, the end of the block must be followed by a label
|
||||||
|
|
||||||
|
### Simultaneous Assignment
|
||||||
|
|
||||||
|
```tla
|
||||||
|
Label:
|
||||||
|
seq[1] := seq[1] + 1 ||
|
||||||
|
seq[2] := seq[2] - 1;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Control Flow
|
||||||
|
|
||||||
|
### if-elsif-else
|
||||||
|
|
||||||
|
```tla
|
||||||
|
if condition then
|
||||||
|
x := 1;
|
||||||
|
elsif other_condition then
|
||||||
|
x := 2;
|
||||||
|
else
|
||||||
|
x := 3;
|
||||||
|
end if;
|
||||||
|
```
|
||||||
|
|
||||||
|
### while
|
||||||
|
|
||||||
|
```tla
|
||||||
|
Loop: \* REQUIRED label before while
|
||||||
|
while condition do
|
||||||
|
\* body (nonatomic — each iteration is a separate step)
|
||||||
|
end while;
|
||||||
|
```
|
||||||
|
|
||||||
|
### either-or (nondeterministic choice)
|
||||||
|
|
||||||
|
```tla
|
||||||
|
either
|
||||||
|
x := 1;
|
||||||
|
or
|
||||||
|
x := 2;
|
||||||
|
or
|
||||||
|
x := 3;
|
||||||
|
end either;
|
||||||
|
```
|
||||||
|
|
||||||
|
### with (temporary bindings and nondeterminism)
|
||||||
|
|
||||||
|
```tla
|
||||||
|
\* Deterministic
|
||||||
|
with tmp = x do
|
||||||
|
y := tmp + 1;
|
||||||
|
end with;
|
||||||
|
|
||||||
|
\* Nondeterministic — picks any element from set
|
||||||
|
with val \in {1, 2, 3} do
|
||||||
|
x := val;
|
||||||
|
end with;
|
||||||
|
```
|
||||||
|
|
||||||
|
### goto, skip, assert, print
|
||||||
|
|
||||||
|
```tla
|
||||||
|
A:
|
||||||
|
goto C; \* must be followed by a label
|
||||||
|
B:
|
||||||
|
skip; \* noop
|
||||||
|
C:
|
||||||
|
assert x > 0; \* fails model check if false (needs EXTENDS TLC)
|
||||||
|
print x; \* debug output
|
||||||
|
```
|
||||||
|
|
||||||
|
### await (blocking)
|
||||||
|
|
||||||
|
```tla
|
||||||
|
WaitForLock:
|
||||||
|
await lock = NULL; \* label only enabled when condition is TRUE
|
||||||
|
lock := self;
|
||||||
|
```
|
||||||
|
|
||||||
|
If no process can proceed → **deadlock** (reported as error by TLC).
|
||||||
|
|
||||||
|
## Macros
|
||||||
|
|
||||||
|
Simple textual substitution. Must be defined before `begin`. Cannot contain labels.
|
||||||
|
|
||||||
|
```tla
|
||||||
|
macro send(queue, msg) begin
|
||||||
|
queue := Append(queue, msg);
|
||||||
|
end macro;
|
||||||
|
|
||||||
|
macro recv(queue, var) begin
|
||||||
|
await queue # <<>>;
|
||||||
|
var := Head(queue);
|
||||||
|
queue := Tail(queue);
|
||||||
|
end macro;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Procedures
|
||||||
|
|
||||||
|
Like macros but can contain labels. Use `call` to invoke, `return` to exit.
|
||||||
|
|
||||||
|
```tla
|
||||||
|
procedure enqueue(item)
|
||||||
|
begin
|
||||||
|
Enqueue:
|
||||||
|
queue := Append(queue, item);
|
||||||
|
return;
|
||||||
|
end procedure;
|
||||||
|
|
||||||
|
\* In a process:
|
||||||
|
call enqueue("hello");
|
||||||
|
```
|
||||||
|
|
||||||
|
## Processes
|
||||||
|
|
||||||
|
### Single Process
|
||||||
|
|
||||||
|
```tla
|
||||||
|
process worker = 1
|
||||||
|
begin
|
||||||
|
Work:
|
||||||
|
\* ...
|
||||||
|
end process;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Process Set
|
||||||
|
|
||||||
|
```tla
|
||||||
|
process worker \in 1..NumWorkers
|
||||||
|
variables local_var = 0; \* each process gets its own copy
|
||||||
|
begin
|
||||||
|
Work:
|
||||||
|
local_var := self; \* self = process value (1, 2, ... NumWorkers)
|
||||||
|
end process;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rules
|
||||||
|
- All processes must have **comparable types** (all ints, all strings, or all model values)
|
||||||
|
- Different processes **cannot share label names**
|
||||||
|
- `self` keyword returns the process's identifier value
|
||||||
|
|
||||||
|
### Fairness
|
||||||
|
|
||||||
|
```tla
|
||||||
|
fair process worker \in Workers \* weak fairness — guaranteed to not stop forever
|
||||||
|
fair+ process worker \in Workers \* strong fairness — runs if repeatedly enabled
|
||||||
|
|
||||||
|
\* Per-label strong fairness:
|
||||||
|
CriticalStep:+
|
||||||
|
x := x + 1;
|
||||||
|
```
|
||||||
|
|
||||||
|
## The `pc` Variable
|
||||||
|
|
||||||
|
The translator creates `pc` to track the current label:
|
||||||
|
- Single process: `pc` is a string (`"LabelName"` or `"Done"`)
|
||||||
|
- Multiple processes: `pc` is a function (`pc[self]`)
|
||||||
|
|
||||||
|
Common pattern for post-completion invariants:
|
||||||
|
```tla
|
||||||
|
IsCorrect == pc = "Done" => result = ExpectedValue
|
||||||
|
|
||||||
|
\* With multiple processes:
|
||||||
|
AllDone == \A p \in Procs: pc[p] = "Done"
|
||||||
|
Correct == AllDone => result = Expected
|
||||||
|
```
|
||||||
|
|
||||||
|
## Translation
|
||||||
|
|
||||||
|
PlusCal algorithms must be translated to TLA+ before model checking:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
java -cp tla2tools.jar pcal.trans -nocfg spec.tla
|
||||||
|
```
|
||||||
|
|
||||||
|
The translation appears between `\* BEGIN TRANSLATION` and `\* END TRANSLATION` markers in the same file. Do not edit translation output by hand.
|
||||||
|
|
||||||
|
## Define Block
|
||||||
|
|
||||||
|
Operators in `define` can reference PlusCal variables. They appear after `variables` and before `begin`:
|
||||||
|
|
||||||
|
```tla
|
||||||
|
define
|
||||||
|
TypeInvariant ==
|
||||||
|
/\ counter \in 0..MaxVal
|
||||||
|
/\ lock \in Processes \union {NULL}
|
||||||
|
|
||||||
|
IsUnique(s) ==
|
||||||
|
\A i, j \in 1..Len(s):
|
||||||
|
i # j => s[i] # s[j]
|
||||||
|
end define;
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,32 @@
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Run a complete TLA+ check: parse, translate PlusCal (if present), then model-check
|
||||||
|
# Usage: check.sh <spec.tla> [--config spec.cfg] [--workers auto]
|
||||||
|
|
||||||
|
SKILL_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
JAR_PATH="$SKILL_DIR/lib/tla2tools.jar"
|
||||||
|
|
||||||
|
if [ ! -f "$JAR_PATH" ]; then
|
||||||
|
echo "ERROR: tla2tools.jar not found. Run: bash $SKILL_DIR/scripts/setup.sh"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
SPEC="${1:?Usage: check.sh <spec.tla> [--config spec.cfg]}"
|
||||||
|
shift
|
||||||
|
|
||||||
|
# Step 1: Translate PlusCal if the spec contains an algorithm
|
||||||
|
if grep -q '\-\-algorithm\|--fair algorithm' "$SPEC" 2>/dev/null; then
|
||||||
|
echo "=== Translating PlusCal ==="
|
||||||
|
java -cp "$JAR_PATH" pcal.trans -nocfg "$SPEC" 2>&1
|
||||||
|
echo ""
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Step 2: Parse with SANY
|
||||||
|
echo "=== Parsing with SANY ==="
|
||||||
|
java -cp "$JAR_PATH" tla2sany.SANY "$SPEC" 2>&1
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 3: Run TLC
|
||||||
|
echo "=== Model Checking with TLC ==="
|
||||||
|
exec bash "$SKILL_DIR/scripts/tlc.sh" "$SPEC" "$@"
|
||||||
|
|
@ -0,0 +1,20 @@
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# SANY Parser - Parse and validate TLA+ specs
|
||||||
|
# Usage: parse.sh <spec.tla>
|
||||||
|
|
||||||
|
SKILL_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
JAR_PATH="$SKILL_DIR/lib/tla2tools.jar"
|
||||||
|
|
||||||
|
if [ ! -f "$JAR_PATH" ]; then
|
||||||
|
echo "ERROR: tla2tools.jar not found. Run setup.sh first."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -z "${1:-}" ]; then
|
||||||
|
echo "Usage: parse.sh <spec.tla>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
exec java -cp "$JAR_PATH" tla2sany.SANY "$1"
|
||||||
|
|
@ -0,0 +1,89 @@
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# TLA+ Tools Setup Script
|
||||||
|
# Downloads tla2tools.jar and verifies Java 11+ is available
|
||||||
|
|
||||||
|
SKILL_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
JAR_PATH="$SKILL_DIR/lib/tla2tools.jar"
|
||||||
|
TLA2TOOLS_VERSION="1.8.0"
|
||||||
|
TLA2TOOLS_URL="https://github.com/tlaplus/tlaplus/releases/download/v${TLA2TOOLS_VERSION}/tla2tools.jar"
|
||||||
|
MAVEN_URL="https://repo1.maven.org/maven2/org/lamport/tla2tools/${TLA2TOOLS_VERSION}/tla2tools-${TLA2TOOLS_VERSION}.jar"
|
||||||
|
|
||||||
|
# Check Java version
|
||||||
|
if ! command -v java &>/dev/null; then
|
||||||
|
echo "ERROR: Java not found. TLA+ tools require Java 11+."
|
||||||
|
echo "Install with: brew install openjdk@11"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
JAVA_VER=$(java -version 2>&1 | head -1 | sed 's/.*"\([0-9]*\)\..*/\1/')
|
||||||
|
if [ "$JAVA_VER" -lt 11 ]; then
|
||||||
|
echo "ERROR: Java $JAVA_VER found, but TLA+ tools require Java 11+."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "OK: Java $JAVA_VER found."
|
||||||
|
|
||||||
|
# Download tla2tools.jar if missing
|
||||||
|
mkdir -p "$SKILL_DIR/lib"
|
||||||
|
if [ -f "$JAR_PATH" ]; then
|
||||||
|
echo "OK: tla2tools.jar already present at $JAR_PATH"
|
||||||
|
else
|
||||||
|
echo "Downloading tla2tools.jar v${TLA2TOOLS_VERSION}..."
|
||||||
|
# Try GitHub releases first, then Maven Central
|
||||||
|
if curl -fsSL -L -o "$JAR_PATH" "$TLA2TOOLS_URL" 2>/dev/null; then
|
||||||
|
echo "OK: Downloaded from GitHub releases."
|
||||||
|
elif curl -fsSL -L -o "$JAR_PATH" "$MAVEN_URL" 2>/dev/null; then
|
||||||
|
echo "OK: Downloaded from Maven Central."
|
||||||
|
else
|
||||||
|
echo "Download failed. Trying to build from local tlaplus source..."
|
||||||
|
# Try building from the tlaplus repo if present nearby
|
||||||
|
TLAPLUS_SRC=""
|
||||||
|
for candidate in \
|
||||||
|
"$SKILL_DIR/../../tlaplus/tlaplus" \
|
||||||
|
"$SKILL_DIR/../../../tlaplus" \
|
||||||
|
"$HOME/p/ai/tlaplus/tlaplus" \
|
||||||
|
"./tlaplus"; do
|
||||||
|
if [ -f "$candidate/tlatools/org.lamport.tlatools/customBuild.xml" ]; then
|
||||||
|
TLAPLUS_SRC="$(cd "$candidate" && pwd)"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -n "$TLAPLUS_SRC" ] && command -v ant &>/dev/null; then
|
||||||
|
echo "Found tlaplus source at: $TLAPLUS_SRC"
|
||||||
|
cd "$TLAPLUS_SRC/tlatools/org.lamport.tlatools"
|
||||||
|
mkdir -p test-class
|
||||||
|
ant -f customBuild.xml compile dist 2>&1 | tail -5
|
||||||
|
if [ -f "dist/tla2tools.jar" ]; then
|
||||||
|
cp dist/tla2tools.jar "$JAR_PATH"
|
||||||
|
echo "OK: Built tla2tools.jar from source."
|
||||||
|
else
|
||||||
|
echo "ERROR: Build failed."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo ""
|
||||||
|
echo "ERROR: Could not download or build tla2tools.jar."
|
||||||
|
echo ""
|
||||||
|
echo "Please download it manually and place at:"
|
||||||
|
echo " $JAR_PATH"
|
||||||
|
echo ""
|
||||||
|
echo "Download from:"
|
||||||
|
echo " https://github.com/tlaplus/tlaplus/releases/download/v${TLA2TOOLS_VERSION}/tla2tools.jar"
|
||||||
|
echo ""
|
||||||
|
echo "Or clone the tlaplus repo and build with: ant -f customBuild.xml dist"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Verify the jar works
|
||||||
|
if java -cp "$JAR_PATH" tlc2.TLC -h 2>&1 | head -3; then
|
||||||
|
echo ""
|
||||||
|
echo "Setup complete. JAR at: $JAR_PATH"
|
||||||
|
else
|
||||||
|
echo "ERROR: tla2tools.jar appears corrupted. Delete it and re-run setup."
|
||||||
|
rm -f "$JAR_PATH"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
@ -0,0 +1,68 @@
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# TLC Model Checker Runner
|
||||||
|
# Usage: tlc.sh <spec.tla> [--config spec.cfg] [--workers auto] [extra TLC flags...]
|
||||||
|
|
||||||
|
SKILL_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
JAR_PATH="$SKILL_DIR/lib/tla2tools.jar"
|
||||||
|
|
||||||
|
if [ ! -f "$JAR_PATH" ]; then
|
||||||
|
echo "ERROR: tla2tools.jar not found. Run setup.sh first."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
SPEC=""
|
||||||
|
CONFIG=""
|
||||||
|
WORKERS="auto"
|
||||||
|
EXTRA_ARGS=()
|
||||||
|
NO_DEADLOCK=""
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--config|-config)
|
||||||
|
CONFIG="$2"; shift 2;;
|
||||||
|
--workers|-workers)
|
||||||
|
WORKERS="$2"; shift 2;;
|
||||||
|
--no-deadlock)
|
||||||
|
NO_DEADLOCK="1"; shift;;
|
||||||
|
*.tla)
|
||||||
|
SPEC="$1"; shift;;
|
||||||
|
*)
|
||||||
|
EXTRA_ARGS+=("$1"); shift;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "$SPEC" ]; then
|
||||||
|
echo "Usage: tlc.sh <spec.tla> [--config spec.cfg] [--workers auto] [--no-deadlock]"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Derive config from spec name if not provided
|
||||||
|
if [ -z "$CONFIG" ]; then
|
||||||
|
CFG_CANDIDATE="${SPEC%.tla}.cfg"
|
||||||
|
if [ -f "$CFG_CANDIDATE" ]; then
|
||||||
|
CONFIG="$CFG_CANDIDATE"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
CMD=(java -XX:+UseParallelGC -cp "$JAR_PATH" tlc2.TLC)
|
||||||
|
CMD+=(-workers "$WORKERS")
|
||||||
|
|
||||||
|
if [ -n "$CONFIG" ]; then
|
||||||
|
CMD+=(-config "$CONFIG")
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -n "$NO_DEADLOCK" ]; then
|
||||||
|
CMD+=(-deadlock)
|
||||||
|
fi
|
||||||
|
|
||||||
|
CMD+=(-noGenerateSpecTE)
|
||||||
|
if [ ${#EXTRA_ARGS[@]} -gt 0 ]; then
|
||||||
|
CMD+=("${EXTRA_ARGS[@]}")
|
||||||
|
fi
|
||||||
|
CMD+=("$SPEC")
|
||||||
|
|
||||||
|
echo "Running: ${CMD[*]}"
|
||||||
|
echo "---"
|
||||||
|
exec "${CMD[@]}"
|
||||||
|
|
@ -0,0 +1,23 @@
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# PlusCal Translator
|
||||||
|
# Usage: translate.sh <spec.tla> [-nocfg]
|
||||||
|
|
||||||
|
SKILL_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
JAR_PATH="$SKILL_DIR/lib/tla2tools.jar"
|
||||||
|
|
||||||
|
if [ ! -f "$JAR_PATH" ]; then
|
||||||
|
echo "ERROR: tla2tools.jar not found. Run setup.sh first."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -z "${1:-}" ]; then
|
||||||
|
echo "Usage: translate.sh <spec.tla> [-nocfg]"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
SPEC="$1"
|
||||||
|
shift
|
||||||
|
|
||||||
|
exec java -cp "$JAR_PATH" pcal.trans "$@" "$SPEC"
|
||||||
|
|
@ -0,0 +1,54 @@
|
||||||
|
---- MODULE BasicSpec ----
|
||||||
|
\* Basic TLA+ specification template (pure TLA+, no PlusCal)
|
||||||
|
EXTENDS Integers, FiniteSets, TLC
|
||||||
|
|
||||||
|
CONSTANTS NULL
|
||||||
|
|
||||||
|
VARIABLES state, data
|
||||||
|
|
||||||
|
vars == <<state, data>>
|
||||||
|
|
||||||
|
\* --- Type definitions ---
|
||||||
|
StateSet == {"init", "running", "done"}
|
||||||
|
DataSet == 0..10
|
||||||
|
|
||||||
|
\* --- Initial state ---
|
||||||
|
Init ==
|
||||||
|
/\ state = "init"
|
||||||
|
/\ data = 0
|
||||||
|
|
||||||
|
\* --- Actions ---
|
||||||
|
Start ==
|
||||||
|
/\ state = "init"
|
||||||
|
/\ state' = "running"
|
||||||
|
/\ UNCHANGED data
|
||||||
|
|
||||||
|
Step ==
|
||||||
|
/\ state = "running"
|
||||||
|
/\ data < 10
|
||||||
|
/\ data' = data + 1
|
||||||
|
/\ UNCHANGED state
|
||||||
|
|
||||||
|
Finish ==
|
||||||
|
/\ state = "running"
|
||||||
|
/\ data = 10
|
||||||
|
/\ state' = "done"
|
||||||
|
/\ UNCHANGED data
|
||||||
|
|
||||||
|
\* --- Next state relation ---
|
||||||
|
Next == Start \/ Step \/ Finish
|
||||||
|
|
||||||
|
\* --- Specification ---
|
||||||
|
Spec == Init /\ [][Next]_vars
|
||||||
|
|
||||||
|
\* --- Properties ---
|
||||||
|
TypeInvariant ==
|
||||||
|
/\ state \in StateSet
|
||||||
|
/\ data \in DataSet
|
||||||
|
|
||||||
|
Safety == state = "done" => data = 10
|
||||||
|
|
||||||
|
Liveness == <>(state = "done")
|
||||||
|
|
||||||
|
FairSpec == Spec /\ WF_vars(Next)
|
||||||
|
====
|
||||||
|
|
@ -0,0 +1,98 @@
|
||||||
|
---- MODULE DistributedSpec ----
|
||||||
|
\* Distributed system specification template
|
||||||
|
EXTENDS Integers, Sequences, FiniteSets, TLC
|
||||||
|
|
||||||
|
CONSTANTS Nodes, NULL, MaxMsgs
|
||||||
|
|
||||||
|
VARIABLES
|
||||||
|
node_state, \* [Nodes -> NodeStateSet]
|
||||||
|
msgs, \* Messages in flight
|
||||||
|
history \* Auxiliary for properties
|
||||||
|
|
||||||
|
vars == <<node_state, msgs, history>>
|
||||||
|
|
||||||
|
\* ---- Message Types ----
|
||||||
|
RequestMsg == [type: {"request"}, from: Nodes, to: Nodes, data: Nat]
|
||||||
|
ResponseMsg == [type: {"response"}, from: Nodes, to: Nodes, data: Nat]
|
||||||
|
AllMsgTypes == RequestMsg \union ResponseMsg
|
||||||
|
|
||||||
|
\* ---- Helpers ----
|
||||||
|
Send(m) == msgs' = msgs \union {m}
|
||||||
|
Discard(m) == msgs' = msgs \ {m}
|
||||||
|
NoMsgChange == UNCHANGED msgs
|
||||||
|
|
||||||
|
\* ---- Initial State ----
|
||||||
|
Init ==
|
||||||
|
/\ node_state = [n \in Nodes |-> "idle"]
|
||||||
|
/\ msgs = {}
|
||||||
|
/\ history = <<>>
|
||||||
|
|
||||||
|
\* ---- Actions ----
|
||||||
|
SendRequest(n) ==
|
||||||
|
/\ node_state[n] = "idle"
|
||||||
|
/\ \E dest \in Nodes \ {n}:
|
||||||
|
/\ node_state' = [node_state EXCEPT ![n] = "waiting"]
|
||||||
|
/\ Send([type |-> "request", from |-> n, to |-> dest, data |-> 0])
|
||||||
|
/\ UNCHANGED history
|
||||||
|
|
||||||
|
HandleRequest(n) ==
|
||||||
|
/\ \E m \in msgs:
|
||||||
|
/\ m.type = "request"
|
||||||
|
/\ m.to = n
|
||||||
|
/\ node_state[n] = "idle"
|
||||||
|
/\ node_state' = [node_state EXCEPT ![n] = "idle"]
|
||||||
|
/\ msgs' = (msgs \ {m}) \union
|
||||||
|
{[type |-> "response", from |-> n, to |-> m.from, data |-> m.data]}
|
||||||
|
/\ history' = Append(history, m)
|
||||||
|
|
||||||
|
HandleResponse(n) ==
|
||||||
|
/\ \E m \in msgs:
|
||||||
|
/\ m.type = "response"
|
||||||
|
/\ m.to = n
|
||||||
|
/\ node_state[n] = "waiting"
|
||||||
|
/\ node_state' = [node_state EXCEPT ![n] = "done"]
|
||||||
|
/\ Discard(m)
|
||||||
|
/\ history' = Append(history, m)
|
||||||
|
|
||||||
|
\* ---- Network Failures (optional) ----
|
||||||
|
DropMessage ==
|
||||||
|
/\ msgs # {}
|
||||||
|
/\ \E m \in msgs: msgs' = msgs \ {m}
|
||||||
|
/\ UNCHANGED <<node_state, history>>
|
||||||
|
|
||||||
|
\* ---- Next State ----
|
||||||
|
Next ==
|
||||||
|
\/ \E n \in Nodes:
|
||||||
|
\/ SendRequest(n)
|
||||||
|
\/ HandleRequest(n)
|
||||||
|
\/ HandleResponse(n)
|
||||||
|
\* \/ DropMessage \* uncomment to model lossy network
|
||||||
|
|
||||||
|
\* ---- Properties ----
|
||||||
|
TypeInvariant ==
|
||||||
|
/\ node_state \in [Nodes -> {"idle", "waiting", "done"}]
|
||||||
|
/\ msgs \subseteq AllMsgTypes
|
||||||
|
|
||||||
|
\* No response without a request
|
||||||
|
Safety ==
|
||||||
|
\A m \in msgs:
|
||||||
|
m.type = "response" =>
|
||||||
|
\E h \in Range(history): h.type = "request" /\ h.from = m.to
|
||||||
|
|
||||||
|
\* Every request eventually gets a response
|
||||||
|
Liveness ==
|
||||||
|
\A n \in Nodes:
|
||||||
|
node_state[n] = "waiting" ~> node_state[n] = "done"
|
||||||
|
|
||||||
|
\* ---- Bounding ----
|
||||||
|
StateConstraint ==
|
||||||
|
/\ Cardinality(msgs) <= MaxMsgs
|
||||||
|
/\ Len(history) <= MaxMsgs * 2
|
||||||
|
|
||||||
|
\* ---- Specification ----
|
||||||
|
Spec == Init /\ [][Next]_vars
|
||||||
|
FairSpec == Spec /\ \A n \in Nodes: WF_vars(Next)
|
||||||
|
|
||||||
|
\* ---- Helpers ----
|
||||||
|
Range(s) == {s[i] : i \in 1..Len(s)}
|
||||||
|
====
|
||||||
|
|
@ -0,0 +1,54 @@
|
||||||
|
---- MODULE PlusCalSpec ----
|
||||||
|
\* PlusCal specification template
|
||||||
|
EXTENDS Integers, Sequences, FiniteSets, TLC
|
||||||
|
|
||||||
|
CONSTANTS NumWorkers, NULL
|
||||||
|
|
||||||
|
Workers == 1..NumWorkers
|
||||||
|
|
||||||
|
(*--algorithm spec_name
|
||||||
|
variables
|
||||||
|
shared_state = 0;
|
||||||
|
queue = <<>>;
|
||||||
|
|
||||||
|
define
|
||||||
|
\* ---- Type Invariant ----
|
||||||
|
TypeInvariant ==
|
||||||
|
/\ shared_state \in 0..100
|
||||||
|
/\ queue \in Seq(1..NumWorkers)
|
||||||
|
|
||||||
|
\* ---- Safety ----
|
||||||
|
AllDone == \A w \in Workers: pc[w] = "Done"
|
||||||
|
Correct == AllDone => shared_state = NumWorkers
|
||||||
|
|
||||||
|
\* ---- Helpers ----
|
||||||
|
QueueNotFull == Len(queue) < 10
|
||||||
|
end define;
|
||||||
|
|
||||||
|
\* ---- Macros ----
|
||||||
|
macro enqueue(q, val) begin
|
||||||
|
q := Append(q, val);
|
||||||
|
end macro;
|
||||||
|
|
||||||
|
macro dequeue(q, var) begin
|
||||||
|
await q # <<>>;
|
||||||
|
var := Head(q);
|
||||||
|
q := Tail(q);
|
||||||
|
end macro;
|
||||||
|
|
||||||
|
\* ---- Processes ----
|
||||||
|
fair process worker \in Workers
|
||||||
|
variables local = 0;
|
||||||
|
begin
|
||||||
|
Start:
|
||||||
|
local := shared_state;
|
||||||
|
Work:
|
||||||
|
shared_state := local + 1;
|
||||||
|
end process;
|
||||||
|
|
||||||
|
end algorithm; *)
|
||||||
|
|
||||||
|
\* BEGIN TRANSLATION (chksum(pcal) = "..." /\ chksum(tla) = "...")
|
||||||
|
\* END TRANSLATION
|
||||||
|
|
||||||
|
====
|
||||||
|
|
@ -0,0 +1,216 @@
|
||||||
|
---
|
||||||
|
name: write-reproducer
|
||||||
|
version: "1.0.0"
|
||||||
|
description: "Write minimal, reliable bug reproducers (repros) for any project. Use when: (1) a bug report or issue needs a test that reliably demonstrates the failure, (2) converting a described failure scenario into runnable code, (3) minimizing an existing complex test to its essential trigger conditions. Covers the full repro workflow: failure characterization, scope selection, writing the repro, verifying it fails for the right reason, and minimizing to the smallest possible trigger."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Write Bug Reproducer
|
||||||
|
|
||||||
|
Your goal is to turn a bug description into a minimal, self-contained, runnable reproduction that triggers a specific, machine-checkable failure criterion.
|
||||||
|
|
||||||
|
A reproducer has three parts — keep them distinct in your head and in your code:
|
||||||
|
|
||||||
|
- **Trigger**: the minimum input, state, schedule, or environment that causes the bug
|
||||||
|
- **Harness**: how the trigger is executed (test runner, script, container, cluster test)
|
||||||
|
- **Oracle**: the machine-checkable failure criterion (assertion, exit code, log pattern, invariant violation)
|
||||||
|
|
||||||
|
Follow the five phases below in order. Do not skip ahead. Write the failure criterion before writing any repro code.
|
||||||
|
|
||||||
|
## Phase 1: Understand and classify
|
||||||
|
|
||||||
|
Restate the bug in one sentence: "The bug is..."
|
||||||
|
|
||||||
|
Classify along three axes:
|
||||||
|
|
||||||
|
- **Bug class**: crash, incorrect output, hang/deadlock, race condition, assertion/invariant violation, performance regression, resource leak, serialization bug, consistency anomaly. See `references/bug-taxonomy.md` for the full taxonomy when classification is unclear.
|
||||||
|
- **Scope**: pure function, module, single process, multiple processes, cluster.
|
||||||
|
- **Determinism**: deterministic, flaky (state what makes it flaky), requires rare schedule or timing.
|
||||||
|
|
||||||
|
If the user gave only symptoms with no code, list the 2-4 most likely causes before proceeding. Do not pick one yet — write hypotheses, not conclusions.
|
||||||
|
|
||||||
|
If the user included code or pointed at a codebase, read the suspect region and identify the likely code path before moving on.
|
||||||
|
|
||||||
|
## Phase 2: Define the failure criterion
|
||||||
|
|
||||||
|
This is the most important phase. Before writing any repro code, write out the oracle in one of these machine-checkable forms:
|
||||||
|
|
||||||
|
- **Assertion with concrete values**: `assert expected == actual` with both spelled out
|
||||||
|
- **Exit code**: `expected: 0, actual: 139 (SIGSEGV)`
|
||||||
|
- **Regex against stdout/stderr**: exact string fragment to match
|
||||||
|
- **Log grep**: file and pattern
|
||||||
|
- **Exception type and message fragment**: the specific error and location
|
||||||
|
- **Linearizability/isolation check result**: checker verdict (e.g., linearizability violation, cycle detected)
|
||||||
|
- **Invariant violation**: state the invariant in one line; state what breaks it
|
||||||
|
|
||||||
|
Record the failure criterion verbatim. It will be used in Phase 4 (to write the assertion) and Phase 5 (to prevent shrinking into a different bug).
|
||||||
|
|
||||||
|
Without an explicit oracle, shrinking can produce something that "fails" but not in the described way, and reviewers cannot tell whether the repro is successful.
|
||||||
|
|
||||||
|
## Phase 3: Detect context and choose repro form
|
||||||
|
|
||||||
|
Inspect the working directory for build-system markers and prefer the project's existing test framework and conventions. Explore existing tests to understand helpers, fixtures, naming patterns, and how integration tests are structured.
|
||||||
|
|
||||||
|
Pick the lowest tier on the minimality lattice that can reliably exhibit the failure. See `references/minimality-lattice.md` for tier definitions and decision rules.
|
||||||
|
|
||||||
|
- Tier 1: expression / REPL snippet
|
||||||
|
- Tier 2: single-file snippet with `main`
|
||||||
|
- Tier 3: single test in the project's existing framework
|
||||||
|
- Tier 4: multi-file project fixture
|
||||||
|
- Tier 5: containerized environment + test script
|
||||||
|
- Tier 6: ephemeral cluster
|
||||||
|
- Tier 7: deterministic simulation
|
||||||
|
|
||||||
|
Write one sentence naming the tier and why, based on what you found in the project.
|
||||||
|
|
||||||
|
For concurrency bugs: `references/concurrency.md`
|
||||||
|
For distributed-systems bugs: `references/distributed-systems.md`
|
||||||
|
For invariant bugs where a property test is the natural form: `references/property-based-testing.md`
|
||||||
|
For bugs found by fuzzing or where a fuzz harness is the natural repro: `references/fuzzing.md`
|
||||||
|
|
||||||
|
## Phase 4: Draft the repro
|
||||||
|
|
||||||
|
### Structure
|
||||||
|
|
||||||
|
Label the three parts in comments:
|
||||||
|
|
||||||
|
```
|
||||||
|
// TRIGGER: <what causes the bug>
|
||||||
|
// HARNESS: <how we run it>
|
||||||
|
// ORACLE: <what we check>
|
||||||
|
```
|
||||||
|
|
||||||
|
Use the project's test framework and conventions. Match existing test style. Do not introduce a new framework unless the existing one cannot express the bug.
|
||||||
|
|
||||||
|
### The pass-then-invert strategy
|
||||||
|
|
||||||
|
For subtle assertion bugs where the correct behavior is easy to state but the failing test is hard to get right, use **pass-then-invert**: first write a test that *passes* on the current buggy behavior (mirroring the observed wrong output in its assertions), then invert the assertions so they encode the *correct* expectation and therefore fail on the bug. This reduces hallucinated APIs and setup errors compared to writing a failing test directly.
|
||||||
|
|
||||||
|
Use pass-then-invert when:
|
||||||
|
- The bug produces a wrong value (not a crash or exception)
|
||||||
|
- The correct expected value is known
|
||||||
|
- Direct "write a failing test" attempts keep failing for setup reasons
|
||||||
|
|
||||||
|
Do not use pass-then-invert when:
|
||||||
|
- The bug is a crash, exception, or hang — just trigger it directly
|
||||||
|
- The oracle is an absence check (no exception, no crash) — inversion doesn't apply
|
||||||
|
|
||||||
|
### Verification before presenting
|
||||||
|
|
||||||
|
The repro must compile or parse cleanly. Resolve imports against the project's actual dependencies. Do not hallucinate APIs — if unsure of a method signature, read the library source or ask.
|
||||||
|
|
||||||
|
If execution is available, run the repro and confirm the failure criterion triggers. If not, trace through the code mentally and state your confidence level.
|
||||||
|
|
||||||
|
### Generating multiple candidates
|
||||||
|
|
||||||
|
When the bug is unclear or the first attempt fails, generate 2-5 candidate repro strategies:
|
||||||
|
- Smallest direct unit test
|
||||||
|
- Integration test if state/config matters
|
||||||
|
- Standalone script if no test harness is obvious
|
||||||
|
- Property-based test if the bug is about an invariant
|
||||||
|
- Fuzz seed / deterministic scenario for correctness bugs
|
||||||
|
|
||||||
|
Execute each candidate with a timeout. Classify each result using `references/failure-classification.md`. Refine until one produces an issue-relevant failure.
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
|
||||||
|
The repro must include a comment or docstring stating:
|
||||||
|
- Issue reference and one-line bug title
|
||||||
|
- Expected (correct) behavior
|
||||||
|
- Actual (buggy) behavior
|
||||||
|
- Failure criterion (verbatim from Phase 2)
|
||||||
|
|
||||||
|
## Phase 5: Shrink and validate
|
||||||
|
|
||||||
|
### Validate first
|
||||||
|
|
||||||
|
Run the repro (or trace through it mentally) and confirm:
|
||||||
|
1. The failure criterion from Phase 2 triggers
|
||||||
|
2. The failure is the *described* bug, not a test setup error
|
||||||
|
3. The failure message matches what Phase 2 specified
|
||||||
|
|
||||||
|
See `references/failure-classification.md` for how to classify test results and distinguish issue-relevant failures from setup errors.
|
||||||
|
|
||||||
|
### Then shrink
|
||||||
|
|
||||||
|
Use the subtraction method from `references/minimization.md`. Work through dimensions in order (cheapest first):
|
||||||
|
1. Reduce infrastructure scale (instances, partitions, replication)
|
||||||
|
2. Reduce data volume (items, size, batches)
|
||||||
|
3. Remove non-default configuration
|
||||||
|
4. Remove setup steps
|
||||||
|
5. Replace real sleeps with controlled time or polling
|
||||||
|
6. Simplify trigger sequence
|
||||||
|
|
||||||
|
See `references/shrinking-checklist.md` for the guardrail checklist.
|
||||||
|
|
||||||
|
### Over-minimization guardrail
|
||||||
|
|
||||||
|
After each shrinking step, re-check that the failure is still *the same failure* — same assertion, same exception type, same message fragment from Phase 2. If the failure changed, revert that step. This prevents shrinking into a different bug.
|
||||||
|
|
||||||
|
### For nondeterministic bugs
|
||||||
|
|
||||||
|
Wrap the trigger in an N-iteration harness with a retry budget. Report the observed failure rate (e.g., "fails 23/1000 iterations"). Do not use sleep-based synchronization — it makes tests flaky on different hardware. Prefer event-based scheduling, explicit synchronization primitives, or deterministic simulation.
|
||||||
|
|
||||||
|
Always print and record seeds for randomized tests so failing scenarios can be replayed.
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
Always produce:
|
||||||
|
|
||||||
|
1. **Repro file(s)** with `// TRIGGER`, `// HARNESS`, `// ORACLE` comments. Correct file placement matching the project's test layout.
|
||||||
|
2. **Status**: one of `REPRODUCED` / `CANDIDATE` / `BLOCKED`. See `references/output-contract.md` for what each status means and requires.
|
||||||
|
3. **Run command**: the exact command to execute the repro.
|
||||||
|
4. **Failure summary**: "This test should fail on the current code with: `<expected failure message>`"
|
||||||
|
5. **For flaky bugs**: failure rate and seed.
|
||||||
|
6. **For blocked repros**: what's missing, what was attempted, next best direction.
|
||||||
|
|
||||||
|
## Hard rules
|
||||||
|
|
||||||
|
- **Write the failure criterion before the repro code.** Without an explicit oracle, you cannot distinguish a successful repro from a coincidental failure. This is the single most common failure mode.
|
||||||
|
|
||||||
|
- **Do not hallucinate APIs.** If unsure whether a method, class, or flag exists, read the source or ask. Hallucinated APIs are the most frequent reason LLM-generated repros fail to compile.
|
||||||
|
|
||||||
|
- **Assert expected behavior, not current broken behavior.** The oracle encodes what *should* happen. The test fails because reality doesn't match.
|
||||||
|
|
||||||
|
- **Do not shrink past the bug.** After every shrinking step, re-verify the failure matches Phase 2. See `references/shrinking-checklist.md`.
|
||||||
|
|
||||||
|
- **Flaky repros must declare their flakiness.** A repro that reproduces 23/1000 times must say so and wrap itself in a retry loop.
|
||||||
|
|
||||||
|
- **Match the project's conventions.** Use the test framework, style, and helpers that already exist. Do not introduce a new framework unless the existing one cannot express the bug.
|
||||||
|
|
||||||
|
- **Do not fix the bug.** Create tests/scripts/harnesses only. Do not edit production code unless explicitly asked.
|
||||||
|
|
||||||
|
- **Do not use `:latest` image tags or unpinned dependency versions.** A repro is not reproducible if its dependencies drift.
|
||||||
|
|
||||||
|
- **Do not confuse setup errors with reproduced bugs.** An error in test setup is not the same as the error caused by the bug. See `references/failure-classification.md`.
|
||||||
|
|
||||||
|
## When to escalate to higher tiers
|
||||||
|
|
||||||
|
Escalate to Tier 5+ (containerized, cluster, deterministic simulation) when:
|
||||||
|
|
||||||
|
- The bug involves two or more processes (client + server, replica + leader)
|
||||||
|
- The bug requires a specific network condition (partition, delay, drop)
|
||||||
|
- The bug requires clock skew or node failure scenarios
|
||||||
|
- The bug is about isolation, linearizability, or consensus
|
||||||
|
- The user mentions Cassandra, Kafka, or any distributed system
|
||||||
|
|
||||||
|
For these cases read `references/distributed-systems.md` before drafting.
|
||||||
|
|
||||||
|
## When to use a property-based test
|
||||||
|
|
||||||
|
Use a property-based test when:
|
||||||
|
|
||||||
|
- The bug is about an invariant (round-trip, idempotence, commutativity, ordering, bounded sums)
|
||||||
|
- Shrinking comes for free in the property framework
|
||||||
|
- You want the test to serve as a regression test with seed persistence
|
||||||
|
|
||||||
|
See `references/property-based-testing.md`.
|
||||||
|
|
||||||
|
## When to use a fuzz harness
|
||||||
|
|
||||||
|
Use a fuzz harness when:
|
||||||
|
|
||||||
|
- The bug was originally found by fuzzing and a crashing input exists
|
||||||
|
- The input is a byte stream or structured binary format
|
||||||
|
- A single example fails but many nearby inputs are interesting for regression
|
||||||
|
|
||||||
|
See `references/fuzzing.md`.
|
||||||
|
|
@ -0,0 +1,103 @@
|
||||||
|
# Bug Taxonomy
|
||||||
|
|
||||||
|
Use this reference when Phase 1 classification is unclear. Each class includes defining characteristics, common signals, and the typical oracle form.
|
||||||
|
|
||||||
|
## Crash
|
||||||
|
|
||||||
|
The process terminates abnormally.
|
||||||
|
|
||||||
|
**Signals**: segfault, SIGABRT, unhandled exception propagating to main, panic, core dump.
|
||||||
|
**Oracle**: exit code != 0, signal number, or specific panic/exception message.
|
||||||
|
**Typical tier**: Tier 2-3 (single file or single test).
|
||||||
|
|
||||||
|
## Incorrect output (silent wrong result)
|
||||||
|
|
||||||
|
The program completes normally but produces the wrong answer.
|
||||||
|
|
||||||
|
**Signals**: wrong return value, missing rows, extra rows, wrong aggregation, wrong serialization output.
|
||||||
|
**Oracle**: `assert expected == actual` with concrete values.
|
||||||
|
**Typical tier**: Tier 2-3. For database bugs, often Tier 5-6 with a SQL script or metamorphic oracle.
|
||||||
|
|
||||||
|
This is the hardest class to repro because there is no crash to trigger — you must know the correct answer. Prefer pass-then-invert when the correct value is known. For database bugs, consider metamorphic testing (NoREC, TLP, PQS) when the correct answer is hard to compute.
|
||||||
|
|
||||||
|
## Hang / deadlock
|
||||||
|
|
||||||
|
The process stops making progress.
|
||||||
|
|
||||||
|
**Signals**: no output, CPU at 0% (deadlock) or 100% (livelock/infinite loop), timeout.
|
||||||
|
**Oracle**: test timeout + thread dump showing blocked threads, or absence of expected output within time bound.
|
||||||
|
**Typical tier**: Tier 3 (unit test with timeout) or Tier 6 (cluster with timeout).
|
||||||
|
|
||||||
|
Distinguish deadlock (threads waiting on each other) from livelock (threads running but not progressing) from infinite loop (single thread never terminating). Thread dumps are diagnostic for the first two.
|
||||||
|
|
||||||
|
## Race condition
|
||||||
|
|
||||||
|
The outcome depends on thread/process scheduling.
|
||||||
|
|
||||||
|
**Signals**: intermittent failures, "works on my machine", different results on different runs, Heisenbugs that disappear under debugging.
|
||||||
|
**Oracle**: assertion that fails intermittently; often needs N-iteration harness.
|
||||||
|
**Typical tier**: Tier 3 (jcstress for JMM races), Tier 6 (cluster test for distributed races), Tier 7 (deterministic simulation).
|
||||||
|
|
||||||
|
See `concurrency.md` for detailed guidance.
|
||||||
|
|
||||||
|
## Assertion / invariant violation
|
||||||
|
|
||||||
|
A documented or implicit invariant is violated.
|
||||||
|
|
||||||
|
**Signals**: AssertionError in production code, negative values where only positive expected, duplicates where uniqueness required, sum doesn't balance, ordering violated.
|
||||||
|
**Oracle**: state the invariant and the violation. Property-based tests are the natural repro form.
|
||||||
|
**Typical tier**: Tier 3 (property test) or Tier 6 (cluster invariant check).
|
||||||
|
|
||||||
|
## Performance regression
|
||||||
|
|
||||||
|
An operation became measurably slower or uses more resources than expected.
|
||||||
|
|
||||||
|
**Signals**: increased latency, higher CPU/memory usage, benchmark regression.
|
||||||
|
**Oracle**: `assert duration < threshold` or `assert memory < threshold` with concrete values from a known-good baseline.
|
||||||
|
**Typical tier**: Tier 3 (microbenchmark) or Tier 4 (benchmark project).
|
||||||
|
|
||||||
|
Performance repros are inherently noisy. Pin the hardware, use statistical tests (e.g., compare distributions, not single runs), and document the baseline.
|
||||||
|
|
||||||
|
## Resource leak
|
||||||
|
|
||||||
|
Resources are not released after use.
|
||||||
|
|
||||||
|
**Signals**: growing memory, file descriptor exhaustion, connection pool depletion, thread count growth.
|
||||||
|
**Oracle**: resource count after N iterations exceeds baseline by more than a threshold, or OOM/EMFILE error.
|
||||||
|
**Typical tier**: Tier 3 (loop test that measures resources).
|
||||||
|
|
||||||
|
## Serialization / encoding bug
|
||||||
|
|
||||||
|
Data is incorrectly encoded, decoded, or transformed between representations.
|
||||||
|
|
||||||
|
**Signals**: corrupt bytes, wrong field values after deserialization, version incompatibility, schema mismatch.
|
||||||
|
**Oracle**: round-trip property (`deserialize(serialize(x)) == x`) or specific field assertion.
|
||||||
|
**Typical tier**: Tier 3 (property test with round-trip check).
|
||||||
|
|
||||||
|
## Consistency anomaly (distributed systems)
|
||||||
|
|
||||||
|
A distributed system violates its stated consistency guarantee.
|
||||||
|
|
||||||
|
**Subtypes** (using Adya/Kingsbury terminology):
|
||||||
|
- **G0 (dirty write)**: two transactions both modify the same object, neither sees the other
|
||||||
|
- **G1a (aborted read)**: read data from an aborted transaction
|
||||||
|
- **G1b (intermediate read)**: read intermediate state of another transaction
|
||||||
|
- **G1c (circular information flow)**: dependency cycle between committed transactions
|
||||||
|
- **G2 (anti-dependency cycle)**: serialization anomaly involving anti-dependencies
|
||||||
|
- **Lost update**: concurrent writes lose one update
|
||||||
|
- **Stale read**: read returns a value older than a committed write that should be visible
|
||||||
|
- **Dirty read**: read uncommitted data from another transaction
|
||||||
|
|
||||||
|
**Signals**: Jepsen/Elle/Knossos/Porcupine checker failures, data loss after recovery, divergent replicas.
|
||||||
|
**Oracle**: linearizability checker returns false, Elle reports specific anomaly type, Porcupine finds illegal history.
|
||||||
|
**Typical tier**: Tier 6 (Jepsen, cluster dtest) or Tier 7 (deterministic simulation).
|
||||||
|
|
||||||
|
See `distributed-systems.md` for detailed guidance.
|
||||||
|
|
||||||
|
## Metadata / schema bug
|
||||||
|
|
||||||
|
The system's metadata (schema, catalog, configuration state) becomes inconsistent.
|
||||||
|
|
||||||
|
**Signals**: schema says X but data says Y, dropped column still queryable, phantom table, inconsistent metadata across nodes.
|
||||||
|
**Oracle**: metadata query returns unexpected result, or operation succeeds that should fail (or vice versa).
|
||||||
|
**Typical tier**: Tier 3 (single-instance) or Tier 6 (cluster for distributed schema).
|
||||||
|
|
@ -0,0 +1,149 @@
|
||||||
|
# Concurrency Bug Repros
|
||||||
|
|
||||||
|
Concurrency bugs depend on scheduling — the same code produces different outcomes depending on thread interleaving. This makes them hard to trigger reliably and easy to shrink into a different bug.
|
||||||
|
|
||||||
|
## Principles
|
||||||
|
|
||||||
|
1. **Do not use `Thread.sleep()` as synchronization.** Sleep-based tests are flaky: they fail on slow hardware and pass on fast hardware, or vice versa. They also make tests slow.
|
||||||
|
|
||||||
|
2. **Make the schedule explicit.** Use one of these approaches:
|
||||||
|
- Barriers, latches, and condition variables to force a specific interleaving
|
||||||
|
- Stress harnesses that run many iterations and report failure rate
|
||||||
|
- Controlled concurrency testing (CCT) tools that systematically explore interleavings
|
||||||
|
- Deterministic simulation that controls the scheduler
|
||||||
|
|
||||||
|
3. **Report failure rate.** If the bug is probabilistic, run N iterations and report how many failed.
|
||||||
|
|
||||||
|
4. **Print and record seeds.** Any randomized element (thread scheduling, input generation) must print its seed so failures can be replayed.
|
||||||
|
|
||||||
|
## Approach 1: Event-based schedule control
|
||||||
|
|
||||||
|
Force a specific interleaving using barriers or latches:
|
||||||
|
|
||||||
|
```
|
||||||
|
// Pseudocode — the general pattern
|
||||||
|
barrier = new CyclicBarrier(2)
|
||||||
|
|
||||||
|
thread1 {
|
||||||
|
step1() // T1 does step 1
|
||||||
|
barrier.await() // sync point: both threads reach here
|
||||||
|
step3() // T1 does step 3 (after T2 did step 2)
|
||||||
|
}
|
||||||
|
|
||||||
|
thread2 {
|
||||||
|
barrier.await() // wait for T1 to finish step 1
|
||||||
|
step2() // T2 does step 2
|
||||||
|
}
|
||||||
|
|
||||||
|
// ORACLE: check final state
|
||||||
|
```
|
||||||
|
|
||||||
|
This makes the test deterministic — it always produces the same interleaving. Use this when you know exactly which schedule triggers the bug.
|
||||||
|
|
||||||
|
## Approach 2: Stress harness (N-iteration)
|
||||||
|
|
||||||
|
Run the trigger many times and check the invariant after each:
|
||||||
|
|
||||||
|
```
|
||||||
|
// Pseudocode
|
||||||
|
failures = 0
|
||||||
|
for i in 1..N:
|
||||||
|
result = run_concurrent_scenario()
|
||||||
|
if not invariant(result):
|
||||||
|
failures++
|
||||||
|
record_failure(i, result)
|
||||||
|
break // or continue to measure rate
|
||||||
|
|
||||||
|
assert failures > 0, "bug did not reproduce in N iterations"
|
||||||
|
// Or for showing the bug is fixed:
|
||||||
|
assert failures == 0, f"failed {failures}/{N} times"
|
||||||
|
```
|
||||||
|
|
||||||
|
Guidelines:
|
||||||
|
- N=1000 is a reasonable starting point
|
||||||
|
- Report the failure rate (e.g., "fails 23/1000")
|
||||||
|
- Record the first failure details for diagnosis
|
||||||
|
- Use a timeout per iteration to catch hangs
|
||||||
|
|
||||||
|
## Approach 3: jcstress (Java)
|
||||||
|
|
||||||
|
jcstress is the gold standard for Java Memory Model bugs. It runs millions of iterations automatically and reports observed outcome frequencies.
|
||||||
|
|
||||||
|
```java
|
||||||
|
@JCStressTest
|
||||||
|
@Outcome(id = "1, 1", expect = Expect.ACCEPTABLE, desc = "Sequentially consistent")
|
||||||
|
@Outcome(id = "0, 0", expect = Expect.ACCEPTABLE, desc = "Both early reads")
|
||||||
|
@Outcome(id = "0, 1", expect = Expect.FORBIDDEN, desc = "Broken invariant")
|
||||||
|
@State
|
||||||
|
public class ReproRace {
|
||||||
|
int x; int flag;
|
||||||
|
|
||||||
|
@Actor
|
||||||
|
public void writer() {
|
||||||
|
x = 42;
|
||||||
|
flag = 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Actor
|
||||||
|
public void reader(II_Result r) {
|
||||||
|
r.r1 = flag;
|
||||||
|
r.r2 = x;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Each `@Actor` method runs in its own thread. jcstress explores many interleavings and reports which `@Outcome` patterns were observed.
|
||||||
|
|
||||||
|
## Approach 4: Controlled concurrency testing (CCT)
|
||||||
|
|
||||||
|
Systematic tools that explore thread interleavings:
|
||||||
|
|
||||||
|
- **CHESS**: iterative preemption bounding — explores all interleavings with up to K preemptions
|
||||||
|
- **PCT/PPCT**: probabilistic concurrency testing — assigns random priorities to threads, mathematically guaranteed coverage
|
||||||
|
- **QL**: Q-learning-based exploration — learns which interleavings are interesting
|
||||||
|
- **Period**: periodical scheduling — imposes periodic preemption patterns
|
||||||
|
|
||||||
|
These are research tools; in practice, most teams use stress harnesses or jcstress.
|
||||||
|
|
||||||
|
## Approach 5: Linearizability checking
|
||||||
|
|
||||||
|
For concurrent data structures or distributed systems, check that the observed history is linearizable:
|
||||||
|
|
||||||
|
1. Record a history of operations with start/end timestamps and return values
|
||||||
|
2. Feed it to a linearizability checker
|
||||||
|
3. The checker returns true (history is linearizable) or false (violation found)
|
||||||
|
|
||||||
|
Tools:
|
||||||
|
- **Porcupine** (Go): fast, handles concurrent maps/registers/queues. 1000x-10,000x faster than Knossos.
|
||||||
|
- **Knossos** (Clojure): Jepsen's checker, handles arbitrary models
|
||||||
|
- **Elle** (Clojure): specifically for database transaction isolation — detects G0, G1a/b/c, G2 anomalies
|
||||||
|
|
||||||
|
History format (Porcupine):
|
||||||
|
```go
|
||||||
|
history := []porcupine.Event{
|
||||||
|
{Kind: porcupine.CallEvent, Value: WriteInput{Key: "k", Value: 1}, Id: 0},
|
||||||
|
{Kind: porcupine.ReturnEvent, Value: WriteOutput{}, Id: 0},
|
||||||
|
{Kind: porcupine.CallEvent, Value: ReadInput{Key: "k"}, Id: 1},
|
||||||
|
{Kind: porcupine.ReturnEvent, Value: ReadOutput{Value: 0}, Id: 1}, // stale read!
|
||||||
|
}
|
||||||
|
ok := porcupine.CheckEvents(model, history)
|
||||||
|
// ORACLE: ok should be true; if false, linearizability violated
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common interleaving patterns that cause bugs
|
||||||
|
|
||||||
|
1. **Check-then-act**: `if (x != null) x.method()` — x can become null between check and use
|
||||||
|
2. **Read-modify-write**: `counter = counter + 1` — lost update without atomicity
|
||||||
|
3. **Publication without barrier**: object constructed but reference published without happens-before
|
||||||
|
4. **Double-checked locking**: broken without volatile/atomic in many memory models
|
||||||
|
5. **Iterator invalidation**: modifying a collection while iterating over it concurrently
|
||||||
|
6. **Close-during-use**: resource closed by one thread while another is using it
|
||||||
|
|
||||||
|
## What to include in a concurrency repro
|
||||||
|
|
||||||
|
1. The exact threads and their operations
|
||||||
|
2. The specific interleaving (or schedule constraint) that triggers the bug
|
||||||
|
3. Whether the bug requires specific hardware (# cores, NUMA, weak memory model)
|
||||||
|
4. Failure rate if the test is probabilistic
|
||||||
|
5. Seeds if any element is randomized
|
||||||
|
6. Thread dump or outcome log showing the violation
|
||||||
|
|
@ -0,0 +1,165 @@
|
||||||
|
# Distributed Systems Bug Repros
|
||||||
|
|
||||||
|
Distributed bugs involve multiple nodes, network conditions, clock behavior, or distributed coordination. They require higher tiers on the minimality lattice (Tier 5-7).
|
||||||
|
|
||||||
|
## Key principle: always try a lower tier first
|
||||||
|
|
||||||
|
Many "distributed" bugs can actually be triggered on a single node. Try single-instance first. If the bug disappears, that's valuable: it means the bug genuinely requires distributed coordination, and you now know the minimum instance count.
|
||||||
|
|
||||||
|
## Jepsen test structure
|
||||||
|
|
||||||
|
Jepsen is the canonical framework for distributed correctness testing. A Jepsen test has:
|
||||||
|
|
||||||
|
- **Control node**: orchestrates the test
|
||||||
|
- **DB nodes**: run the system under test (typically 5 nodes)
|
||||||
|
- **Generator**: produces a stream of operations (reads, writes, CAS)
|
||||||
|
- **Client**: translates operations into system-specific API calls
|
||||||
|
- **Nemesis**: injects faults (partitions, kills, clock skew, disk corruption)
|
||||||
|
- **Checker**: validates the history (linearizability, sequential consistency, etc.)
|
||||||
|
|
||||||
|
```clojure
|
||||||
|
;; Jepsen test skeleton (Clojure)
|
||||||
|
(deftest register-test
|
||||||
|
(let [test (jepsen/run!
|
||||||
|
(merge tests/noop-test
|
||||||
|
{:name "register"
|
||||||
|
:nodes ["n1" "n2" "n3" "n4" "n5"]
|
||||||
|
:client (register-client)
|
||||||
|
:nemesis (nemesis/partition-random-halves)
|
||||||
|
:generator (gen/phases
|
||||||
|
(->> (gen/mix [r w cas])
|
||||||
|
(gen/stagger 1/10)
|
||||||
|
(gen/nemesis
|
||||||
|
(gen/seq (cycle [(gen/sleep 5)
|
||||||
|
{:type :info :f :start}
|
||||||
|
(gen/sleep 5)
|
||||||
|
{:type :info :f :stop}])))
|
||||||
|
(gen/time-limit 60)))
|
||||||
|
:checker (checker/linearizable
|
||||||
|
{:model (model/cas-register)
|
||||||
|
:algorithm :linear})}))]
|
||||||
|
(is (:valid? (:results test)))))
|
||||||
|
```
|
||||||
|
|
||||||
|
Jepsen saves histories to `store/<test-name>/<date>/` for offline replay.
|
||||||
|
|
||||||
|
## Cassandra in-JVM dtest patterns
|
||||||
|
|
||||||
|
For Cassandra-specific distributed bugs, in-JVM dtests are faster than Jepsen because all nodes run in one JVM.
|
||||||
|
|
||||||
|
### Basic cluster test
|
||||||
|
```java
|
||||||
|
try (Cluster cluster = Cluster.build().withNodes(3)
|
||||||
|
.withConfig(c -> c.with(Feature.GOSSIP, Feature.NETWORK))
|
||||||
|
.start()) {
|
||||||
|
// setup, trigger, oracle
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Node failure during operation
|
||||||
|
```java
|
||||||
|
// TRIGGER: write data, kill a node, verify reads
|
||||||
|
cluster.coordinator(1).execute("INSERT INTO ks.t (pk, v) VALUES (1, 1)", ConsistencyLevel.ALL);
|
||||||
|
cluster.get(2).shutdown().get();
|
||||||
|
Object[][] result = cluster.coordinator(1).execute("SELECT v FROM ks.t WHERE pk = 1", ConsistencyLevel.QUORUM);
|
||||||
|
assertThat(result[0][0]).isEqualTo(1);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Network partition simulation
|
||||||
|
```java
|
||||||
|
// Message filtering to simulate partition
|
||||||
|
cluster.filters().verbs(Verb.MUTATION_REQ.id).from(1).to(2).drop();
|
||||||
|
// Now mutations from node 1 won't reach node 2
|
||||||
|
```
|
||||||
|
|
||||||
|
### Bootstrap / decommission / topology change
|
||||||
|
```java
|
||||||
|
// Start with 3 nodes, bootstrap a 4th
|
||||||
|
IInstanceConfig config = cluster.newInstanceConfig();
|
||||||
|
config.set("auto_bootstrap", true);
|
||||||
|
cluster.bootstrap(config).startup();
|
||||||
|
// Run workload during bootstrap, check for data loss
|
||||||
|
```
|
||||||
|
|
||||||
|
### Repair
|
||||||
|
```java
|
||||||
|
cluster.get(1).nodetoolResult("repair", "ks").asserts().success();
|
||||||
|
// Verify data consistency after repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cassandra simulator (CEP-10)
|
||||||
|
|
||||||
|
The Cassandra simulator provides deterministic execution for Cassandra cluster scenarios. It controls the thread scheduler and network, making races reproducible with a seed.
|
||||||
|
|
||||||
|
Key namespace: `org.apache.cassandra.simulator.*`
|
||||||
|
|
||||||
|
Seeds are printed on failure. To replay: pass the same seed to get the exact same execution.
|
||||||
|
|
||||||
|
## testcontainers patterns
|
||||||
|
|
||||||
|
When you need real external services but not a full cluster framework:
|
||||||
|
|
||||||
|
```java
|
||||||
|
// Java
|
||||||
|
@Testcontainers
|
||||||
|
class ReproTest {
|
||||||
|
@Container
|
||||||
|
static KafkaContainer kafka = new KafkaContainer(
|
||||||
|
DockerImageName.parse("confluentinc/cp-kafka:7.6.0"));
|
||||||
|
|
||||||
|
@Container
|
||||||
|
static PostgreSQLContainer<?> pg = new PostgreSQLContainer<>("postgres:16.2-alpine");
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void issue_xxx() {
|
||||||
|
// Use kafka.getBootstrapServers() and pg.getJdbcUrl()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Pin images by version (not `:latest`). Use `@Container` for lifecycle management.
|
||||||
|
|
||||||
|
## Fault injection patterns
|
||||||
|
|
||||||
|
### Network partitions
|
||||||
|
- **Symmetric**: both sides cannot communicate
|
||||||
|
- **Asymmetric**: A can send to B but not receive from B
|
||||||
|
- **Partial**: only specific message types are dropped
|
||||||
|
|
||||||
|
### Node failures
|
||||||
|
- **Clean shutdown**: graceful stop
|
||||||
|
- **Kill -9**: immediate termination, no cleanup
|
||||||
|
- **Pause/resume**: simulate GC pause or freeze (SIGSTOP/SIGCONT)
|
||||||
|
|
||||||
|
### Clock issues
|
||||||
|
- **Skew**: offset one node's clock by minutes/hours
|
||||||
|
- **Drift**: gradually increase clock rate
|
||||||
|
- **Jump**: sudden clock change (forward or backward)
|
||||||
|
|
||||||
|
### Disk issues
|
||||||
|
- **Full disk**: fill the data directory
|
||||||
|
- **Slow disk**: inject latency on I/O
|
||||||
|
- **Corrupt data file**: flip bits in specific files
|
||||||
|
|
||||||
|
## What to include in a distributed repro
|
||||||
|
|
||||||
|
1. **Topology**: number of nodes, datacenter layout, replication factor
|
||||||
|
2. **Workload**: operations, rate, consistency levels
|
||||||
|
3. **Fault scenario**: what fault, when injected, duration
|
||||||
|
4. **Oracle**: what invariant is violated (linearizability, durability, liveness)
|
||||||
|
5. **Timing**: when the fault is injected relative to the workload
|
||||||
|
6. **Version**: exact commit or release version
|
||||||
|
7. **Configuration**: any non-default settings
|
||||||
|
|
||||||
|
## Minimization for distributed repros
|
||||||
|
|
||||||
|
Reduce in this order:
|
||||||
|
1. **Node count**: 5 → 3 → 2 → 1 (if it reproduces at 1, it's not a distributed bug)
|
||||||
|
2. **Replication factor**: 3 → 2 → 1
|
||||||
|
3. **Workload size**: 1000 ops → 100 → 10 → 1
|
||||||
|
4. **Fault complexity**: partition + kill + clock skew → just partition → just kill
|
||||||
|
5. **Concurrency**: many clients → 2 → 1
|
||||||
|
6. **Configuration**: remove non-default settings one at a time
|
||||||
|
7. **Schema**: remove unused columns, tables, indexes
|
||||||
|
|
||||||
|
After each reduction, verify the *same* invariant violation occurs.
|
||||||
|
|
@ -0,0 +1,86 @@
|
||||||
|
# Failure Classification
|
||||||
|
|
||||||
|
Use this reference to classify test results and distinguish issue-relevant failures from noise.
|
||||||
|
|
||||||
|
## Why classification matters
|
||||||
|
|
||||||
|
The most common failure mode in LLM-generated repros is declaring success on a non-relevant failure: a test fails, but for the wrong reason (setup error, missing dependency, hallucinated API). Classification prevents this.
|
||||||
|
|
||||||
|
## Classification scheme
|
||||||
|
|
||||||
|
### REPRODUCED
|
||||||
|
The test fails with a failure that directly corresponds to the described bug.
|
||||||
|
|
||||||
|
Requirements:
|
||||||
|
- The failure matches the Phase 2 failure criterion
|
||||||
|
- The failure occurs in the code under test, not in test setup
|
||||||
|
- The failure message/exception can be explained in terms of the bug report
|
||||||
|
- Removing the buggy behavior would make the test pass
|
||||||
|
|
||||||
|
### CANDIDATE
|
||||||
|
The test fails with behavior *consistent* with the report, but certainty is incomplete.
|
||||||
|
|
||||||
|
Common reasons for CANDIDATE instead of REPRODUCED:
|
||||||
|
- Expected behavior was inferred (not stated by the user)
|
||||||
|
- The failure matches the description but the assertion tests a proxy, not the exact symptom
|
||||||
|
- The bug is nondeterministic and the test only triggered once
|
||||||
|
|
||||||
|
### BLOCKED
|
||||||
|
The repro cannot be executed or the oracle is too ambiguous.
|
||||||
|
|
||||||
|
Common reasons:
|
||||||
|
- Missing external service that can't be emulated
|
||||||
|
- Missing data/schema/configuration
|
||||||
|
- The bug report is too vague to construct an oracle
|
||||||
|
- The repro requires infrastructure not available locally
|
||||||
|
|
||||||
|
### Classification of individual test runs
|
||||||
|
|
||||||
|
When executing a candidate repro, classify the result as one of:
|
||||||
|
|
||||||
|
| Classification | Meaning | Next action |
|
||||||
|
|---------------|---------|-------------|
|
||||||
|
| PASS | Test passes, does not reproduce | Strengthen the oracle or scenario |
|
||||||
|
| COMPILE_ERROR | Code doesn't compile | Fix imports, types, dependencies |
|
||||||
|
| TEST_DISCOVERY_ERROR | Test framework can't find/load the test | Fix test placement, naming, annotations |
|
||||||
|
| SETUP_ERROR | Test infrastructure fails (DB connection, missing fixture) | Fix setup, add missing dependencies |
|
||||||
|
| UNRELATED_RUNTIME_ERROR | Runtime error not matching the bug | Fix the test; the error is a test problem |
|
||||||
|
| UNRELATED_ASSERTION | Assertion fails but for wrong reason | Rewrite the assertion to match the real bug |
|
||||||
|
| RELATED_RUNTIME_ERROR | Runtime error matching the bug description | This may be the repro — verify it matches Phase 2 |
|
||||||
|
| RELATED_ASSERTION | Assertion fails for the right reason | This is likely the repro — verify against Phase 2 |
|
||||||
|
| FLAKY_OR_NONDETERMINISTIC | Inconsistent results across runs | Wrap in N-iteration harness, report rate |
|
||||||
|
| TIMEOUT | Test did not complete | May indicate hang/deadlock (if that's the bug) or may indicate broken test setup |
|
||||||
|
| NEEDS_EXTERNAL_SYSTEM | Requires a service not available | Escalate tier or use testcontainers/docker-compose |
|
||||||
|
| ORACLE_AMBIGUOUS | Multiple interpretations of correct behavior | Ask the user to clarify expected behavior |
|
||||||
|
|
||||||
|
## How to distinguish issue-relevant failures from setup errors
|
||||||
|
|
||||||
|
A failure is issue-relevant if:
|
||||||
|
|
||||||
|
1. The exception/assertion occurs in the **code under test**, not in test infrastructure
|
||||||
|
2. The stack trace passes through the **suspected code path** described in the bug report
|
||||||
|
3. The failure **message** contains the same keywords/values as the bug description
|
||||||
|
4. The failure **persists** after fixing test scaffolding problems
|
||||||
|
5. The failure **disappears** when the described bug is fixed (if a fix is available)
|
||||||
|
|
||||||
|
A failure is a setup error if:
|
||||||
|
|
||||||
|
1. The exception occurs before the trigger code runs (in `@Before`, fixture setup, import resolution)
|
||||||
|
2. The exception is about **missing classes, methods, or dependencies** (not about the bug)
|
||||||
|
3. The exception is about **connection refused, port not available, service unavailable**
|
||||||
|
4. The test fails the same way regardless of whether the bug is present
|
||||||
|
|
||||||
|
## Recording results
|
||||||
|
|
||||||
|
For each candidate run, record:
|
||||||
|
|
||||||
|
```
|
||||||
|
Command: <exact command>
|
||||||
|
Exit code: <number>
|
||||||
|
Classification: <one of the above>
|
||||||
|
Relevance: <why this matches or doesn't match the bug>
|
||||||
|
Stdout excerpt: <first relevant lines>
|
||||||
|
Stderr excerpt: <first relevant lines>
|
||||||
|
Stack trace: <top 5 frames if applicable>
|
||||||
|
Next action: <what to do next>
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,233 @@
|
||||||
|
# Fuzz Testing for Repros
|
||||||
|
|
||||||
|
Use a fuzz harness as the repro when the bug involves parsing, deserialization, or processing of untrusted input, or when a crashing input already exists from a fuzzer.
|
||||||
|
|
||||||
|
## When to use a fuzz harness as the repro
|
||||||
|
|
||||||
|
- A crashing input from a fuzzer already exists — just package it with the harness
|
||||||
|
- The bug is in a parser, deserializer, codec, or input processor
|
||||||
|
- The input space is large and a single example misses the point — the harness itself is the repro
|
||||||
|
- You want the repro to also serve as a regression fuzzer
|
||||||
|
|
||||||
|
## When NOT to use a fuzz harness
|
||||||
|
|
||||||
|
- The bug requires specific multi-step state setup (use a regular test instead)
|
||||||
|
- The bug is about business logic, not input processing
|
||||||
|
- The input structure is too complex for byte-level fuzzing and no structure-aware fuzzer exists
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Choose a technique
|
||||||
|
|
||||||
|
Answer these questions in order to pick the right harness type:
|
||||||
|
|
||||||
|
```
|
||||||
|
1. Does the input have a well-defined structure (JSON, XML, SQL, a protocol)?
|
||||||
|
YES → Grammar-based / structured fuzzing (use Arbitrary, autofuzz, or a grammar mutator)
|
||||||
|
NO → continue ↓
|
||||||
|
|
||||||
|
2. Do you have a corpus of valid sample inputs?
|
||||||
|
YES → Mutation + coverage-guided fuzzing (AFL++, libFuzzer, Jazzer)
|
||||||
|
NO → continue ↓
|
||||||
|
|
||||||
|
3. Is the target a small function with complex conditionals (math, parsers)?
|
||||||
|
YES → Concolic / symbolic execution
|
||||||
|
NO → continue ↓
|
||||||
|
|
||||||
|
4. Quick regression baseline only?
|
||||||
|
→ Random / byte-mutation fuzzing (simplest harness, weakest coverage)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Most practical choice:** mutation fuzzing on a seed corpus, with structure-aware input generation if the format is known.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Harness anatomy
|
||||||
|
|
||||||
|
Every fuzz harness has three responsibilities:
|
||||||
|
|
||||||
|
```
|
||||||
|
function fuzz_target(data: bytes):
|
||||||
|
try:
|
||||||
|
parsed = parse(data) // TRIGGER: bytes → your type
|
||||||
|
result = function_under_test(parsed)
|
||||||
|
assert_invariants(result) // ORACLE: check your property
|
||||||
|
catch ExpectedError:
|
||||||
|
return // normal; fuzzer continues
|
||||||
|
catch UnexpectedError:
|
||||||
|
CRASH / re-raise // fuzzer records this input
|
||||||
|
```
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
- Accept raw bytes (or a structured type from a framework like `Arbitrary`)
|
||||||
|
- Never exit on expected / graceful errors — only crash on unexpected ones
|
||||||
|
- Keep it fast: no I/O, no network, no sleeps
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Oracle types
|
||||||
|
|
||||||
|
Choose the oracle that matches the bug class:
|
||||||
|
|
||||||
|
| Oracle | Use when |
|
||||||
|
|--------|----------|
|
||||||
|
| **Crash** | Any panic / exception / segfault is the bug |
|
||||||
|
| **Assertion** | An `assert` or invariant inside the code should fire |
|
||||||
|
| **Differential** | Compare output against a reference implementation |
|
||||||
|
| **Invariant** | `decode(encode(x)) == x`, monotonicity, commutativity, etc. |
|
||||||
|
|
||||||
|
The crash oracle is implicit in most frameworks (any panic = finding). For correctness bugs, you must write the assertion explicitly.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Frameworks by language
|
||||||
|
|
||||||
|
| Language | Framework | Coverage-guided | Structure-aware |
|
||||||
|
|----------|-----------|----------------|-----------------|
|
||||||
|
| Rust | cargo-fuzz (libFuzzer) | Yes | Yes (via Arbitrary) |
|
||||||
|
| Rust | honggfuzz-rs | Yes | No |
|
||||||
|
| C/C++ | libFuzzer | Yes | No |
|
||||||
|
| C/C++ | AFL++ | Yes | Via grammar mutators |
|
||||||
|
| Java | Jazzer | Yes | Yes (via autofuzz + @FuzzTest) |
|
||||||
|
| Python | Atheris | Yes | No |
|
||||||
|
| Go | go-fuzz / native `testing.F` | Yes | No |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Packaging a fuzz repro
|
||||||
|
|
||||||
|
A fuzz repro consists of:
|
||||||
|
|
||||||
|
1. **The fuzz target**: a function that takes bytes (or a structured input) and feeds them to the code under test
|
||||||
|
2. **The crashing input**: the specific bytes/file that trigger the bug (stored in a corpus or artifacts directory)
|
||||||
|
3. **The replay command**: how to run the target with the specific input
|
||||||
|
|
||||||
|
### Rust (cargo-fuzz)
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// fuzz/fuzz_targets/parse_input.rs
|
||||||
|
#![no_main]
|
||||||
|
use libfuzzer_sys::fuzz_target;
|
||||||
|
|
||||||
|
fuzz_target!(|data: &[u8]| {
|
||||||
|
// TRIGGER: feed arbitrary bytes to the parser
|
||||||
|
// ORACLE: implicit — any panic, crash, or sanitizer violation is a finding
|
||||||
|
let _ = my_crate::parse(data);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
Replay:
|
||||||
|
```bash
|
||||||
|
cargo +nightly fuzz run parse_input fuzz/artifacts/parse_input/<crash-hash>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rust with Arbitrary (structured fuzzing)
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#![no_main]
|
||||||
|
use libfuzzer_sys::fuzz_target;
|
||||||
|
use arbitrary::Arbitrary;
|
||||||
|
|
||||||
|
#[derive(Arbitrary, Debug)]
|
||||||
|
struct Input {
|
||||||
|
header: [u8; 4],
|
||||||
|
payload: Vec<u8>,
|
||||||
|
flags: u8,
|
||||||
|
}
|
||||||
|
|
||||||
|
fuzz_target!(|input: Input| {
|
||||||
|
let _ = my_crate::process(&input.header, &input.payload, input.flags);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Java (Jazzer)
|
||||||
|
|
||||||
|
```java
|
||||||
|
import com.code_intelligence.jazzer.api.FuzzedDataProvider;
|
||||||
|
import com.code_intelligence.jazzer.junit.FuzzTest;
|
||||||
|
|
||||||
|
class ParserFuzzTest {
|
||||||
|
@FuzzTest
|
||||||
|
void fuzzParser(FuzzedDataProvider data) {
|
||||||
|
byte[] input = data.consumeRemainingAsBytes();
|
||||||
|
// TRIGGER
|
||||||
|
MyParser.parse(input);
|
||||||
|
// ORACLE: implicit — Jazzer detects crashes, sanitizer violations,
|
||||||
|
// and specific bug detectors (SSRF, path traversal, command injection)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Go (native fuzz)
|
||||||
|
|
||||||
|
```go
|
||||||
|
func FuzzParse(f *testing.F) {
|
||||||
|
// Seed corpus
|
||||||
|
f.Add([]byte("valid input"))
|
||||||
|
f.Add([]byte(""))
|
||||||
|
|
||||||
|
f.Fuzz(func(t *testing.T, data []byte) {
|
||||||
|
// TRIGGER
|
||||||
|
_, _ = Parse(data)
|
||||||
|
// ORACLE: implicit — panics are findings
|
||||||
|
})
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Replay:
|
||||||
|
```bash
|
||||||
|
go test -run FuzzParse/testdata/fuzz/FuzzParse/<crash-file>
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Seeds
|
||||||
|
|
||||||
|
Better seeds → faster coverage and more realistic mutations.
|
||||||
|
|
||||||
|
- Use real inputs from production logs, test fixtures, or existing unit tests
|
||||||
|
- Include both valid and near-invalid examples
|
||||||
|
- Keep seeds small (< 1 KB each when possible)
|
||||||
|
- Store seeds in a corpus directory that persists across runs for regression coverage
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Distilling a fuzz finding into a plain test
|
||||||
|
|
||||||
|
Sometimes a fuzz finding should be converted into a plain unit test for clarity:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#[test]
|
||||||
|
fn issue_xxx_parser_panics_on_truncated_header() {
|
||||||
|
// This input was found by cargo-fuzz (seed: <hash>)
|
||||||
|
// It triggers a panic in the header length check
|
||||||
|
let input = b"\x00\x01"; // truncated header (needs 4 bytes)
|
||||||
|
|
||||||
|
// ORACLE: should return Err, not panic
|
||||||
|
let result = std::panic::catch_unwind(|| my_crate::parse(input));
|
||||||
|
assert!(result.is_ok(), "parser panicked on truncated input");
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Do this when:
|
||||||
|
- The crashing input is small and the bug is clear
|
||||||
|
- You want the test in the regular test suite, not in the fuzz directory
|
||||||
|
- The fuzz infrastructure isn't available in CI
|
||||||
|
|
||||||
|
Keep the fuzz harness when:
|
||||||
|
- The input is large or opaque
|
||||||
|
- You want continued fuzzing to find related bugs
|
||||||
|
- The crashing input is one of many interesting nearby inputs
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Test-case reduction for fuzz findings
|
||||||
|
|
||||||
|
When the crashing input is large, reduce it:
|
||||||
|
|
||||||
|
- **cargo-fuzz**: `cargo +nightly fuzz tmin <target> <crash-file>`
|
||||||
|
- **libFuzzer**: built-in minimizer via `-minimize_crash=1`
|
||||||
|
- **AFL++**: `afl-tmin`
|
||||||
|
- **For structured inputs**: use Perses (grammar-aware) or C-Reduce (C/C++ specific)
|
||||||
|
|
||||||
|
Always verify the reduced input triggers the *same* crash (same stack trace, same sanitizer report).
|
||||||
|
|
@ -0,0 +1,99 @@
|
||||||
|
# Minimality Lattice
|
||||||
|
|
||||||
|
A repro sits on a tier. Pick the lowest tier that reliably exhibits the failure. A 300-line Jepsen scenario is more minimal than 2000 lines of ad-hoc bash if the bug requires a 5-node cluster with partitions.
|
||||||
|
|
||||||
|
## Tier 1: Expression / REPL snippet
|
||||||
|
|
||||||
|
**What**: a single expression or a few lines in a REPL.
|
||||||
|
**When**: pure-function semantics bugs, e.g., `assert (a+b)+c == a+(b+c)` for floating-point.
|
||||||
|
**When not**: anything requiring state, I/O, or multiple operations.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```python
|
||||||
|
>>> round(2.675, 2)
|
||||||
|
2.67 # expected 2.68
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tier 2: Single-file snippet with `main`
|
||||||
|
|
||||||
|
**What**: one file with a main function, no test framework, no build system.
|
||||||
|
**When**: logic bugs in a library that can be triggered by calling a function from main. Compiler/optimizer crashes triggered by a single source file.
|
||||||
|
**When not**: when the bug needs the project's test infrastructure or external services.
|
||||||
|
|
||||||
|
Example: a single `.java` file with `public static void main` that calls the buggy method and prints/asserts.
|
||||||
|
|
||||||
|
## Tier 3: Single test in the project's framework
|
||||||
|
|
||||||
|
**What**: one test method using the project's existing test framework (JUnit, pytest, cargo test, go test).
|
||||||
|
**When**: most bugs. This is the default tier. The bug can be triggered by calling code within the project's dependency graph.
|
||||||
|
**When not**: when the bug requires external services, multiple processes, or cluster topology.
|
||||||
|
|
||||||
|
This is the sweet spot for most repros. It runs fast, integrates into CI, and uses familiar conventions.
|
||||||
|
|
||||||
|
## Tier 4: Multi-file project fixture
|
||||||
|
|
||||||
|
**What**: a small project (pom.xml / Cargo.toml + a few source files) that demonstrates the bug.
|
||||||
|
**When**: build/tool/classpath bugs, or bugs that require specific project configuration. Bugs in build plugins, code generators, or IDE tooling.
|
||||||
|
**When not**: when a single test file within the existing project suffices.
|
||||||
|
|
||||||
|
Example: a minimal Maven project with one pom.xml and one test class that triggers a Surefire plugin bug.
|
||||||
|
|
||||||
|
## Tier 5: Docker Compose + test script
|
||||||
|
|
||||||
|
**What**: docker-compose.yml defining services + a test script that runs against them.
|
||||||
|
**When**: integration bugs requiring multiple processes (app + database, app + message queue), but where the processes don't need to form a cluster or have distributed coordination.
|
||||||
|
**When not**: when testcontainers within a Tier 3 test can provide the same services, or when the bug is about distributed coordination (use Tier 6).
|
||||||
|
|
||||||
|
Pin image digests, not `:latest`. Include a `reproduce.sh` that does `docker-compose up -d && sleep-or-poll && run-test && docker-compose down`.
|
||||||
|
|
||||||
|
## Tier 6: Ephemeral cluster
|
||||||
|
|
||||||
|
**What**: a multi-node cluster test using project-specific or general-purpose distributed test infrastructure.
|
||||||
|
**When**: bugs involving replication, leader election, failover, consensus, network partitions, distributed coordination, consistency anomalies.
|
||||||
|
**When not**: when the bug can be triggered on a single instance (always try Tier 3 first).
|
||||||
|
|
||||||
|
Frameworks by ecosystem:
|
||||||
|
- **Cassandra**: in-JVM dtests (`Cluster.build().withNodes(N)...`), CQLTester for single-node
|
||||||
|
- **General distributed**: Jepsen (Clojure, controls real/Docker nodes, built-in checkers)
|
||||||
|
- **Any language**: testcontainers (starts real services in Docker, used from test code)
|
||||||
|
- **Kafka**: `EmbeddedKafkaCluster` or testcontainers-kafka
|
||||||
|
|
||||||
|
## Tier 7: Deterministic simulation
|
||||||
|
|
||||||
|
**What**: a simulation environment that provides full determinism (same seed = same execution).
|
||||||
|
**When**: rare races, liveness bugs, fault-tolerance properties that are hard to trigger reliably even with cluster tests. When you need to explore millions of schedules.
|
||||||
|
**When not**: when the bug reproduces reliably at Tier 6, or when no simulation framework exists for the system.
|
||||||
|
|
||||||
|
Frameworks:
|
||||||
|
- **Cassandra**: CEP-10 cluster and code simulator (`org.apache.cassandra.simulator.*`)
|
||||||
|
- **Rust**: turmoil (used by S2), madsim (used by RisingWave)
|
||||||
|
- **General**: Antithesis (commercial, language-agnostic via Docker)
|
||||||
|
- **FoundationDB**: Flow (C++ actor model with deterministic simulation)
|
||||||
|
- **TigerBeetle**: VOPR (Viewstamped Operation Replication simulator)
|
||||||
|
|
||||||
|
## Decision flowchart
|
||||||
|
|
||||||
|
```
|
||||||
|
Can the bug be triggered by calling one function with known inputs?
|
||||||
|
YES → Tier 1-2 (expression or single file)
|
||||||
|
NO ↓
|
||||||
|
|
||||||
|
Can the bug be triggered within the project's test framework?
|
||||||
|
YES → Tier 3 (single test)
|
||||||
|
NO ↓
|
||||||
|
|
||||||
|
Does the bug require specific project/build configuration?
|
||||||
|
YES → Tier 4 (project fixture)
|
||||||
|
NO ↓
|
||||||
|
|
||||||
|
Does the bug require multiple processes but NOT distributed coordination?
|
||||||
|
YES → Tier 5 (docker-compose)
|
||||||
|
NO ↓
|
||||||
|
|
||||||
|
Does the bug involve distributed coordination, replication, or consensus?
|
||||||
|
YES → Is the bug reliably reproducible?
|
||||||
|
YES → Tier 6 (ephemeral cluster)
|
||||||
|
NO → Tier 7 (deterministic simulation)
|
||||||
|
```
|
||||||
|
|
||||||
|
Always try one tier lower first. If the bug disappears, that's valuable information — it means the bug requires the complexity you removed.
|
||||||
|
|
@ -0,0 +1,140 @@
|
||||||
|
# Minimization Guide
|
||||||
|
|
||||||
|
A minimal reproducer has exactly the code needed to trigger the bug — no more. Minimization makes the root cause obvious, removes false leads, and makes the test fast to run.
|
||||||
|
|
||||||
|
## The subtraction method
|
||||||
|
|
||||||
|
Minimization is a **binary search over code removal**:
|
||||||
|
|
||||||
|
1. Start with the first version that triggers the bug
|
||||||
|
2. Identify a candidate to remove (a config, a step, a dependency, a data item)
|
||||||
|
3. Remove it
|
||||||
|
4. Run the test
|
||||||
|
5. **Bug still fails** → keep the removal, continue
|
||||||
|
6. **Bug no longer fails** → undo the removal (that element is essential)
|
||||||
|
7. Repeat until nothing more can be removed
|
||||||
|
|
||||||
|
Work through these dimensions in order (cheapest reductions first):
|
||||||
|
|
||||||
|
### Dimension 1: Infrastructure scale
|
||||||
|
|
||||||
|
| What to try | How |
|
||||||
|
|---|---|
|
||||||
|
| Reduce service/instance count | Run with 1 instance instead of N |
|
||||||
|
| Reduce data partitions/shards | Use 1 partition/shard instead of many |
|
||||||
|
| Reduce replication/redundancy | Use minimal replication factor |
|
||||||
|
| Remove extra resources | Delete all resources (tables, queues, topics, buckets) except the one involved in the trigger |
|
||||||
|
|
||||||
|
If the bug disappears when you reduce instance count, that's an important signal — the bug involves distributed coordination. Keep the minimum count that reproduces it.
|
||||||
|
|
||||||
|
### Dimension 2: Data volume
|
||||||
|
|
||||||
|
| What to try | How |
|
||||||
|
|---|---|
|
||||||
|
| Reduce item count | Process 1 item instead of 100/1000 |
|
||||||
|
| Reduce item size | Use minimal payloads instead of large ones |
|
||||||
|
| Reduce batch size | Remove batching tuning, let defaults apply |
|
||||||
|
|
||||||
|
### Dimension 3: Configuration
|
||||||
|
|
||||||
|
Remove each non-default config one at a time. Start with configs that seem unrelated to the bug:
|
||||||
|
|
||||||
|
```
|
||||||
|
# Before: many custom settings
|
||||||
|
config.set("timeout", 5000)
|
||||||
|
config.set("retries", 3)
|
||||||
|
config.set("batch_size", 16384)
|
||||||
|
config.set("compression", "snappy")
|
||||||
|
config.set("buffer_size", 33554432)
|
||||||
|
config.set("max_connections", 10)
|
||||||
|
config.set("cache_ttl", 60)
|
||||||
|
config.set("log_level", "debug")
|
||||||
|
|
||||||
|
# Try removing each — if bug persists without it, delete it
|
||||||
|
```
|
||||||
|
|
||||||
|
### Dimension 4: Setup steps
|
||||||
|
|
||||||
|
Remove preparatory operations one at a time:
|
||||||
|
- Initial "warmup" operations before the trigger
|
||||||
|
- Extra resource creation steps
|
||||||
|
- Pre-population of data that may not be needed
|
||||||
|
- Initialization sequences that precede the trigger
|
||||||
|
|
||||||
|
Ask: "Does the bug appear on a fresh state, or only after some prior state is established?" If prior state is needed, keep only the minimum state setup.
|
||||||
|
|
||||||
|
### Dimension 5: Timing
|
||||||
|
|
||||||
|
Replace real sleeps with deterministic time control:
|
||||||
|
|
||||||
|
```
|
||||||
|
# BEFORE (fragile, slow)
|
||||||
|
service.send(request)
|
||||||
|
sleep(5000) # wait for async processing
|
||||||
|
assert_expected()
|
||||||
|
|
||||||
|
# AFTER (fast, deterministic)
|
||||||
|
# Option A: Use a fake/mock clock and advance it exactly
|
||||||
|
fake_clock.advance(exact_threshold + 1)
|
||||||
|
assert_expected()
|
||||||
|
|
||||||
|
# Option B: Poll for the condition with a timeout
|
||||||
|
wait_until(lambda: condition_is_met(), timeout=30_000, message="condition not reached")
|
||||||
|
```
|
||||||
|
|
||||||
|
If `sleep()` is in the test because the code needs time to do something asynchronous, use a polling/retry helper with a timeout instead.
|
||||||
|
|
||||||
|
### Dimension 6: Trigger sequence
|
||||||
|
|
||||||
|
Sometimes the trigger sequence can be simplified:
|
||||||
|
- Can the bug be triggered with a single API call instead of a sequence?
|
||||||
|
- Can the trigger be a direct call to an internal function instead of going through the public API?
|
||||||
|
- Can the ordering be simplified (A then B, instead of A, B, C, B, A, C)?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What "minimal" looks like in practice
|
||||||
|
|
||||||
|
**Before minimization** (real-world repro attempt):
|
||||||
|
```
|
||||||
|
3 service instances, 6 partitions, replication=3, 100 data items,
|
||||||
|
5 custom configs, warmup operations, 10s sleep, then trigger
|
||||||
|
```
|
||||||
|
|
||||||
|
**After minimization**:
|
||||||
|
```
|
||||||
|
1 instance, 1 partition, no replication, 1 data item, default configs,
|
||||||
|
direct trigger call
|
||||||
|
```
|
||||||
|
|
||||||
|
If the bug requires 2 instances (distributed coordination), keep 2. If it requires 3 partitions (hash-based routing), keep 3. The goal is **necessary minimum**, not arbitrary minimum.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verification checklist
|
||||||
|
|
||||||
|
Before declaring the repro final:
|
||||||
|
|
||||||
|
- Test fails with the expected exception or wrong value (not a test setup error)
|
||||||
|
- Test passes after the bug fix is applied (confirms it's a true repro, not a pre-existing failure)
|
||||||
|
- Removing any single setup step makes the test pass (confirms minimality)
|
||||||
|
- Test name/description references the issue tracker ticket
|
||||||
|
- Documentation states: expected behavior, actual (buggy) behavior
|
||||||
|
- No `sleep()` unless truly necessary (prefer fake clocks or polling helpers)
|
||||||
|
- No hardcoded ports, paths, or machine-specific values
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Common traps
|
||||||
|
|
||||||
|
**Trap: Test passes on first run, fails intermittently**
|
||||||
|
The bug is a race condition. Use a fake clock to control ordering, or use synchronization primitives (barriers, latches, controlled scheduling) to inject deterministic interleavings. A flaky repro is not a repro.
|
||||||
|
|
||||||
|
**Trap: The bug only shows up after many iterations**
|
||||||
|
Reduce the iteration count to the minimum. If the bug requires N iterations, find the minimum N and document why. Consider using a loop with an early-exit assertion.
|
||||||
|
|
||||||
|
**Trap: The test fails but for the wrong reason**
|
||||||
|
The setup may be broken (wrong resource, wrong config, wrong state). Always read the failure message. A `NullPointerException` in the setup is not the same as the `NullPointerException` caused by the bug.
|
||||||
|
|
||||||
|
**Trap: Minimization accidentally tests the wrong thing**
|
||||||
|
After each removal, re-read the assertion. If you stripped away too much, the test might still fail but now for a different reason. The failure message should stay the same throughout minimization.
|
||||||
|
|
@ -0,0 +1,123 @@
|
||||||
|
# Output Contract
|
||||||
|
|
||||||
|
Every repro run must produce one of three statuses: REPRODUCED, CANDIDATE, or BLOCKED.
|
||||||
|
|
||||||
|
## REPRODUCED
|
||||||
|
|
||||||
|
Use only when there is an executable failure that matches the described bug.
|
||||||
|
|
||||||
|
Required elements:
|
||||||
|
- **Status**: REPRODUCED
|
||||||
|
- **Files**: list of created/modified files with their paths
|
||||||
|
- **Run command**: exact command to execute the repro
|
||||||
|
- **Failure**: the observed failure message, exception, or assertion
|
||||||
|
- **Why this matches**: one sentence connecting the failure to the bug description
|
||||||
|
- **Failure criterion**: verbatim from Phase 2
|
||||||
|
- **Minimization notes**: what was removed during shrinking, what was kept and why
|
||||||
|
|
||||||
|
Optional:
|
||||||
|
- Seed (if randomized)
|
||||||
|
- Failure rate (if nondeterministic)
|
||||||
|
- Environment notes
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```
|
||||||
|
Status: REPRODUCED
|
||||||
|
|
||||||
|
Files:
|
||||||
|
- src/test/java/com/example/ReproIssue42Test.java
|
||||||
|
|
||||||
|
Run command:
|
||||||
|
mvn test -pl core -Dtest=ReproIssue42Test
|
||||||
|
|
||||||
|
Failure:
|
||||||
|
AssertionError: expected [50] but was [40]
|
||||||
|
at ReproIssue42Test.testSumExcludesDeletedRows(ReproIssue42Test.java:38)
|
||||||
|
|
||||||
|
Why this matches:
|
||||||
|
The bug report says SUM should include rows 2 and 3 (20+30=50) but returns 40,
|
||||||
|
matching the described symptom of deleted row 1's value being subtracted twice.
|
||||||
|
|
||||||
|
Failure criterion (Phase 2):
|
||||||
|
assert sum == 50, actual sum is 40
|
||||||
|
|
||||||
|
Minimization:
|
||||||
|
Reduced from 10 rows to 3. Removed unrelated columns (name, email).
|
||||||
|
Removed non-default compaction config (not needed to trigger).
|
||||||
|
|
||||||
|
This test should fail on the current code with:
|
||||||
|
"AssertionError: expected [50] but was [40]"
|
||||||
|
```
|
||||||
|
|
||||||
|
## CANDIDATE
|
||||||
|
|
||||||
|
Use when the repro is plausible but not fully proven.
|
||||||
|
|
||||||
|
Required elements:
|
||||||
|
- **Status**: CANDIDATE
|
||||||
|
- **Reason**: why this is CANDIDATE rather than REPRODUCED
|
||||||
|
- **Files**: list of created files
|
||||||
|
- **Run command**: exact command
|
||||||
|
- **Observed failure**: what happened
|
||||||
|
- **What needs confirmation**: specific questions for the user
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```
|
||||||
|
Status: CANDIDATE
|
||||||
|
|
||||||
|
Reason:
|
||||||
|
The expected value was inferred from existing tests, not from the bug report.
|
||||||
|
The test fails with behavior consistent with the report, but the oracle may
|
||||||
|
be testing a proxy rather than the exact symptom.
|
||||||
|
|
||||||
|
Files:
|
||||||
|
- tests/test_repro_issue_42.py
|
||||||
|
|
||||||
|
Run command:
|
||||||
|
pytest tests/test_repro_issue_42.py -v
|
||||||
|
|
||||||
|
Observed failure:
|
||||||
|
AssertionError: assert result == 50, got 40
|
||||||
|
|
||||||
|
What needs confirmation:
|
||||||
|
- Is the expected sum 50? The bug report says "wrong result" but doesn't state the correct value.
|
||||||
|
- Should deleted rows be excluded from SUM? The test assumes yes.
|
||||||
|
```
|
||||||
|
|
||||||
|
## BLOCKED
|
||||||
|
|
||||||
|
Use when execution is impossible or the oracle is too ambiguous to construct a meaningful repro.
|
||||||
|
|
||||||
|
Required elements:
|
||||||
|
- **Status**: BLOCKED
|
||||||
|
- **Reason**: what prevented successful reproduction
|
||||||
|
- **What was attempted**: list of approaches tried
|
||||||
|
- **What was produced**: any partial artifacts (standalone harness, mocked scenario)
|
||||||
|
- **What's needed**: specific information or infrastructure that would unblock
|
||||||
|
- **Next best path**: recommendation for the user
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```
|
||||||
|
Status: BLOCKED
|
||||||
|
|
||||||
|
Reason:
|
||||||
|
The bug requires a running Kafka cluster with specific topic configuration,
|
||||||
|
and no Kafka infrastructure is available locally or via testcontainers.
|
||||||
|
|
||||||
|
What was attempted:
|
||||||
|
1. Tried to reproduce with an in-memory mock — bug requires real partition rebalancing
|
||||||
|
2. Tried testcontainers-kafka — image pull failed (no Docker daemon)
|
||||||
|
|
||||||
|
What was produced:
|
||||||
|
- tests/test_repro_issue_42.py (test logic is correct, needs Kafka)
|
||||||
|
- docker-compose-kafka.yml (untested, would need Docker)
|
||||||
|
|
||||||
|
What's needed:
|
||||||
|
- Docker daemon running, or
|
||||||
|
- Access to a Kafka cluster with admin permissions, or
|
||||||
|
- Kafka broker logs from the original failure
|
||||||
|
|
||||||
|
Next best path:
|
||||||
|
Run docker-compose-kafka.yml in an environment with Docker,
|
||||||
|
then execute: pytest tests/test_repro_issue_42.py -v
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,131 @@
|
||||||
|
# Property-Based Testing for Repros
|
||||||
|
|
||||||
|
Property-based testing (PBT) is the preferred repro form when the bug is about an invariant. The framework generates random inputs, checks the property, and automatically shrinks counterexamples to minimal failures.
|
||||||
|
|
||||||
|
## When to use PBT as the repro form
|
||||||
|
|
||||||
|
- **Round-trip bugs**: `deserialize(serialize(x)) == x` fails for some inputs
|
||||||
|
- **Idempotence bugs**: `f(f(x)) != f(x)` for some operation
|
||||||
|
- **Commutativity bugs**: `f(g(x)) != g(f(x))` when they should commute
|
||||||
|
- **Ordering bugs**: sort, merge, or priority violations
|
||||||
|
- **Bounded sum bugs**: values exceed expected bounds
|
||||||
|
- **Monotonicity bugs**: a metric that should only increase decreases
|
||||||
|
- **Equivalence bugs**: two implementations of the same spec disagree
|
||||||
|
- **Model-based bugs**: a simple reference model disagrees with the real implementation
|
||||||
|
|
||||||
|
## Key advantage: automatic shrinking
|
||||||
|
|
||||||
|
When PBT finds a counterexample, it automatically shrinks it to a minimal failing case. This replaces Phase 5 manual minimization for the input/trigger dimension.
|
||||||
|
|
||||||
|
### Integrated shrinking (Hypothesis, proptest) vs type-based shrinking (QuickCheck, jqwik)
|
||||||
|
|
||||||
|
- **Integrated shrinking** (Hypothesis for Python, proptest for Rust): shrinking happens at the generator/strategy level, inside the generation process. Invariants from `filter()` and `assume()` are respected during shrinking. This produces smaller, valid counterexamples.
|
||||||
|
- **Type-based shrinking** (QuickCheck for Haskell, jqwik for Java, quickcheck for Rust): shrinking happens at the type level, after generation. May produce invalid inputs that violate generator constraints, requiring extra `assume()` guards.
|
||||||
|
|
||||||
|
Prefer integrated shrinking (Hypothesis, proptest) when available. When using type-based shrinking, add explicit `assume()` checks to filter invalid shrunk inputs.
|
||||||
|
|
||||||
|
## Framework reference
|
||||||
|
|
||||||
|
| Language | Framework | Shrinking | Seed persistence |
|
||||||
|
|----------|-----------|-----------|-----------------|
|
||||||
|
| Python | Hypothesis | Integrated | `.hypothesis/` directory |
|
||||||
|
| Rust | proptest | Integrated | `proptest-regressions/` files |
|
||||||
|
| Rust | quickcheck | Type-based | Manual seed printing |
|
||||||
|
| Java | jqwik | Type-based | `.jqwik-database` file |
|
||||||
|
| Java | QuickTheories | Type-based | Seed in failure message |
|
||||||
|
| Haskell | QuickCheck | Type-based | `--seed` flag |
|
||||||
|
| Go | gopter | Type-based | Seed in failure message |
|
||||||
|
| Scala | ScalaCheck | Type-based | Seed in failure message |
|
||||||
|
| Clojure | test.check | Type-based | `:seed` in result map |
|
||||||
|
|
||||||
|
## Common property patterns
|
||||||
|
|
||||||
|
### Round-trip (bijection)
|
||||||
|
```
|
||||||
|
for all x:
|
||||||
|
decode(encode(x)) == x
|
||||||
|
```
|
||||||
|
|
||||||
|
### Idempotence
|
||||||
|
```
|
||||||
|
for all x:
|
||||||
|
f(f(x)) == f(x)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Commutativity
|
||||||
|
```
|
||||||
|
for all x, y:
|
||||||
|
f(x, y) == f(y, x)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Associativity
|
||||||
|
```
|
||||||
|
for all x, y, z:
|
||||||
|
f(f(x, y), z) == f(x, f(y, z))
|
||||||
|
```
|
||||||
|
|
||||||
|
### Monotonicity
|
||||||
|
```
|
||||||
|
for all x, y where x <= y:
|
||||||
|
f(x) <= f(y)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Invariant preservation
|
||||||
|
```
|
||||||
|
for all state, operation:
|
||||||
|
if invariant(state):
|
||||||
|
invariant(apply(operation, state))
|
||||||
|
```
|
||||||
|
|
||||||
|
### Oracle / model comparison
|
||||||
|
```
|
||||||
|
for all inputs:
|
||||||
|
real_implementation(inputs) == simple_model(inputs)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Metamorphic relation
|
||||||
|
```
|
||||||
|
for all x, transform:
|
||||||
|
f(transform(x)) == expected_transform(f(x))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stateful (model-based) testing
|
||||||
|
|
||||||
|
For bugs that require a sequence of operations, use stateful PBT. The framework generates random operation sequences, applies them to both a real system and a simple model, and checks that they agree after each step.
|
||||||
|
|
||||||
|
```
|
||||||
|
model = SimpleModel()
|
||||||
|
real = RealSystem()
|
||||||
|
|
||||||
|
for operation in generated_operation_sequence:
|
||||||
|
model_result = model.apply(operation)
|
||||||
|
real_result = real.apply(operation)
|
||||||
|
assert model_result == real_result, f"diverged at {operation}"
|
||||||
|
```
|
||||||
|
|
||||||
|
Frameworks with stateful testing support:
|
||||||
|
- Hypothesis: `RuleBasedStateMachine`
|
||||||
|
- jqwik: `@Property` with `ActionSequence`
|
||||||
|
- proptest: manual state machine via `proptest!` macro
|
||||||
|
- QuickTheories: manual, using `withExamples()`
|
||||||
|
|
||||||
|
## Seed management
|
||||||
|
|
||||||
|
Every PBT failure has a seed. Seeds are essential for:
|
||||||
|
1. **Reproducibility**: re-run with the same seed to get the same counterexample
|
||||||
|
2. **Regression**: store the seed so the counterexample is tested in CI
|
||||||
|
3. **Debugging**: step through the exact failing scenario
|
||||||
|
|
||||||
|
Best practices:
|
||||||
|
- Always print the seed on failure (most frameworks do this automatically)
|
||||||
|
- Store regression seeds in source control (Hypothesis and proptest do this automatically)
|
||||||
|
- Include the seed in the repro README when sharing
|
||||||
|
|
||||||
|
## Danger: shrinking into a different bug
|
||||||
|
|
||||||
|
When PBT shrinks a counterexample, it may shrink into a *different* property violation than the original. This is the "shrinkers are fuzzers" problem.
|
||||||
|
|
||||||
|
Mitigation:
|
||||||
|
- Use tight, specific properties (not just "doesn't crash")
|
||||||
|
- After shrinking, verify the failure message matches the original
|
||||||
|
- If using integrated shrinking, this is less likely because the shrinker respects generator constraints
|
||||||
|
|
@ -0,0 +1,59 @@
|
||||||
|
# Shrinking Checklist
|
||||||
|
|
||||||
|
Use this checklist after every shrinking step to prevent over-minimization — shrinking into a different bug than the one described.
|
||||||
|
|
||||||
|
## After each removal, verify ALL of the following
|
||||||
|
|
||||||
|
- **Same failure criterion**: the failure matches the Phase 2 failure criterion verbatim
|
||||||
|
- **Same exception type**: if the oracle is an exception, it's the same class (not a different exception from broken test setup)
|
||||||
|
- **Same assertion message fragment**: the assertion failure message contains the same key words
|
||||||
|
- **Same exit code**: if the oracle is an exit code, it hasn't changed
|
||||||
|
- **Same log line**: if the oracle is a log grep, the same line still appears
|
||||||
|
- **Same stack trace root**: the deepest application frame in the stack trace is the same (test framework frames above it may differ)
|
||||||
|
|
||||||
|
If ANY answer is "no", **revert the last shrinking step**. That element was load-bearing.
|
||||||
|
|
||||||
|
## Trace features
|
||||||
|
|
||||||
|
For ambiguous cases, define a small set of "trace features" before starting shrinking:
|
||||||
|
|
||||||
|
1. The exception class name
|
||||||
|
2. The first application stack frame (file:line or class.method)
|
||||||
|
3. A key phrase from the error message
|
||||||
|
4. The exit code or signal
|
||||||
|
|
||||||
|
All trace features must match after every shrinking step. If any changes, the shrink went too far.
|
||||||
|
|
||||||
|
## Common over-minimization traps
|
||||||
|
|
||||||
|
### Trap: Removing setup causes a different crash
|
||||||
|
You removed a setup step, the test still fails, but now it's a `NullPointerException` in initialization instead of the `IndexOutOfBoundsException` described in the bug. **Revert** — the setup step was needed to reach the real bug.
|
||||||
|
|
||||||
|
### Trap: Reducing concurrency removes the race
|
||||||
|
You reduced from 10 threads to 1, and the test still fails... but now it fails for a sequential logic bug, not the race condition. The concurrency was essential. **Revert** and try reducing to 2 threads instead.
|
||||||
|
|
||||||
|
### Trap: Shrinking the input changes the code path
|
||||||
|
You reduced the input from 100 items to 1 item, and the test still fails... but the code now takes a different branch (e.g., a "small input" fast path instead of the "large input" path where the bug lives). **Revert** and find the minimum input size that takes the same code path.
|
||||||
|
|
||||||
|
### Trap: Removing configuration hides the bug
|
||||||
|
You reverted a non-default config to its default, and the test passes. That config was a trigger condition. **Keep it** and document why it's needed.
|
||||||
|
|
||||||
|
## When to stop shrinking
|
||||||
|
|
||||||
|
Stop when:
|
||||||
|
1. Removing any single element causes the test to pass (or fail differently)
|
||||||
|
2. All the shrinking checklist items pass
|
||||||
|
3. Every remaining line is load-bearing — it either sets up the trigger, executes the trigger, or checks the oracle
|
||||||
|
|
||||||
|
A good minimal repro has three clear sections (TRIGGER, HARNESS, ORACLE) and nothing else.
|
||||||
|
|
||||||
|
## When to use automated reducers instead
|
||||||
|
|
||||||
|
If the failing input is large (>100 lines of text, >1KB of binary), consider:
|
||||||
|
- **C-Reduce / creduce**: for C/C++ source files
|
||||||
|
- **Perses**: for any language with an ANTLR grammar (language-agnostic)
|
||||||
|
- **cargo-fuzz tmin**: for Rust fuzz corpus entries
|
||||||
|
- **libFuzzer -minimize_crash=1**: for libFuzzer corpus entries
|
||||||
|
- **`scripts/shrink_text.py`**: for line-level ddmin on text files
|
||||||
|
|
||||||
|
These tools automate the "remove and re-test" loop and are much faster than manual reduction for large inputs.
|
||||||
|
|
@ -0,0 +1,208 @@
|
||||||
|
# Test Patterns for Reproducers
|
||||||
|
|
||||||
|
Common patterns for writing bug reproducers across different test levels. Adapt to the project's language, test framework, and conventions.
|
||||||
|
|
||||||
|
## Contents
|
||||||
|
1. [Unit tests with fake time](#1-unit-tests-with-fake-time)
|
||||||
|
2. [Mock/stub tests](#2-mockstub-tests)
|
||||||
|
3. [Single-instance integration tests](#3-single-instance-integration-tests)
|
||||||
|
4. [Multi-instance integration tests](#4-multi-instance-integration-tests)
|
||||||
|
5. [Async waiting helpers](#5-async-waiting-helpers)
|
||||||
|
6. [Internal/white-box tests](#6-internalwhite-box-tests)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Unit tests with fake time
|
||||||
|
|
||||||
|
Use when the bug involves timeouts, expiry, TTLs, retry backoff, or any time the code reads the clock.
|
||||||
|
|
||||||
|
**Pattern:** Replace the real clock with a controllable fake. Advance time explicitly to trigger the condition.
|
||||||
|
|
||||||
|
```
|
||||||
|
# Pseudocode — adapt to your language/framework
|
||||||
|
fake_clock = FakeClock(start_time=0)
|
||||||
|
component = MyComponent(clock=fake_clock)
|
||||||
|
|
||||||
|
component.start()
|
||||||
|
fake_clock.advance(SESSION_TIMEOUT + 1)
|
||||||
|
|
||||||
|
assert component.is_expired() # Deterministic, no real waiting
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key principle:** If the production code accepts a `Clock`/`Time`/`TimeProvider` interface, inject a fake. If it doesn't, that's a useful finding — the code may need refactoring for testability.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Mock/stub tests
|
||||||
|
|
||||||
|
Use when the bug is in component logic that depends on external services, but the bug itself doesn't require a real service.
|
||||||
|
|
||||||
|
**When to use mocks:**
|
||||||
|
- Client-side logic (retry logic, error handling, response parsing)
|
||||||
|
- Callback/handler behavior
|
||||||
|
- State machine transitions
|
||||||
|
- Serialization/deserialization bugs
|
||||||
|
|
||||||
|
**Pattern:**
|
||||||
|
```
|
||||||
|
# Pseudocode
|
||||||
|
mock_service = MockService()
|
||||||
|
client = MyClient(service=mock_service)
|
||||||
|
|
||||||
|
# Inject specific responses or errors
|
||||||
|
mock_service.next_response = ErrorResponse(code=503)
|
||||||
|
|
||||||
|
result = client.send(request)
|
||||||
|
|
||||||
|
# Assert client-side handling
|
||||||
|
assert result.was_retried
|
||||||
|
assert mock_service.call_count == 3 # retried twice
|
||||||
|
```
|
||||||
|
|
||||||
|
**Error injection:** Configure mocks to return specific errors, delays, or malformed responses to trigger the bug's conditions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Single-instance integration tests
|
||||||
|
|
||||||
|
Use when the bug requires a real service/server but not multiple instances.
|
||||||
|
|
||||||
|
**Pattern:** Start a single embedded/test instance, run the reproducer against it.
|
||||||
|
|
||||||
|
```
|
||||||
|
# Pseudocode — adapt to your project's test infra
|
||||||
|
server = start_test_server(config=default_config)
|
||||||
|
|
||||||
|
client = create_client(server.address)
|
||||||
|
client.create_resource("test-resource")
|
||||||
|
result = client.perform_operation("test-resource", data)
|
||||||
|
|
||||||
|
assert result == expected_value
|
||||||
|
|
||||||
|
server.stop()
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key considerations:**
|
||||||
|
- Use the project's existing test fixtures/helpers for starting services
|
||||||
|
- Prefer embedded/in-process servers over external process management
|
||||||
|
- Use random/ephemeral ports to avoid conflicts
|
||||||
|
- Clean up resources in teardown
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Multi-instance integration tests
|
||||||
|
|
||||||
|
Use when the bug requires replication, leader election, failover, consensus, or cross-instance coordination.
|
||||||
|
|
||||||
|
**Pattern:** Start N instances, create resources that span them, then trigger the distributed condition.
|
||||||
|
|
||||||
|
```
|
||||||
|
# Pseudocode
|
||||||
|
cluster = start_test_cluster(instances=3)
|
||||||
|
|
||||||
|
cluster.create_resource("test-resource", replication=3)
|
||||||
|
|
||||||
|
# Trigger a distributed event (failover, partition, etc.)
|
||||||
|
cluster.stop_instance(leader_id)
|
||||||
|
|
||||||
|
# Assert correct behavior after the event
|
||||||
|
result = cluster.read("test-resource")
|
||||||
|
assert result == expected_value
|
||||||
|
|
||||||
|
cluster.stop_all()
|
||||||
|
```
|
||||||
|
|
||||||
|
**Minimize first:** Always try to reproduce with 1 instance before using N. Only increase if the bug requires cross-instance interaction.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Async waiting helpers
|
||||||
|
|
||||||
|
Use instead of `sleep()` to wait for asynchronous conditions. Most test frameworks provide these, or they're easy to write.
|
||||||
|
|
||||||
|
**Polling pattern:**
|
||||||
|
```
|
||||||
|
# Pseudocode
|
||||||
|
def wait_until(condition, timeout_ms=30000, poll_interval_ms=100, message=""):
|
||||||
|
deadline = now() + timeout_ms
|
||||||
|
while now() < deadline:
|
||||||
|
if condition():
|
||||||
|
return
|
||||||
|
sleep(poll_interval_ms)
|
||||||
|
raise TimeoutError(message)
|
||||||
|
|
||||||
|
# Usage
|
||||||
|
wait_until(
|
||||||
|
lambda: client.get_status() == "ready",
|
||||||
|
timeout_ms=30000,
|
||||||
|
message="Service did not become ready"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Retry-on-exception pattern:**
|
||||||
|
```
|
||||||
|
# Pseudocode
|
||||||
|
def retry_until_no_exception(timeout_ms, fn):
|
||||||
|
deadline = now() + timeout_ms
|
||||||
|
last_error = None
|
||||||
|
while now() < deadline:
|
||||||
|
try:
|
||||||
|
fn()
|
||||||
|
return
|
||||||
|
except Exception as e:
|
||||||
|
last_error = e
|
||||||
|
sleep(100)
|
||||||
|
raise last_error
|
||||||
|
```
|
||||||
|
|
||||||
|
**Common framework equivalents:**
|
||||||
|
| Language/Framework | Helper |
|
||||||
|
|---|---|
|
||||||
|
| Java/JUnit | `Awaitility.await().atMost(...)`, `TestUtils.waitForCondition()` |
|
||||||
|
| Python/pytest | `tenacity.retry()`, custom polling loop |
|
||||||
|
| JavaScript/Jest | `waitFor()`, `jest.advanceTimersByTime()` |
|
||||||
|
| Go | `require.Eventually()` (testify), `time.After` + select |
|
||||||
|
| Rust | `tokio::time::timeout()`, custom polling |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Internal/white-box tests
|
||||||
|
|
||||||
|
For bugs deep in a component's internals, test the class/function directly without the full service stack.
|
||||||
|
|
||||||
|
**When to use:**
|
||||||
|
- The bug is in a specific algorithm or data structure
|
||||||
|
- Starting the full service is slow or complex
|
||||||
|
- You want to test exact internal state transitions
|
||||||
|
|
||||||
|
**Pattern:**
|
||||||
|
```
|
||||||
|
# Pseudocode — construct the internal component directly
|
||||||
|
component = InternalComponent(
|
||||||
|
config=minimal_config,
|
||||||
|
clock=fake_clock,
|
||||||
|
dependency=mock_dependency,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Call internal methods directly
|
||||||
|
component.process(input_data)
|
||||||
|
|
||||||
|
# Assert internal state
|
||||||
|
assert component.internal_state == expected_state
|
||||||
|
```
|
||||||
|
|
||||||
|
**Trade-off:** White-box tests are faster and more precise but couple to internal APIs. If the internal API changes, the test breaks even if the bug is still present. Prefer this only when integration-level testing is too coarse to isolate the trigger.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Discovering project test conventions
|
||||||
|
|
||||||
|
Before writing a reproducer, explore the project to understand:
|
||||||
|
|
||||||
|
1. **Test framework:** What framework is used? (JUnit, pytest, Jest, Go testing, etc.)
|
||||||
|
2. **Test layout:** Where do tests live? (`src/test/`, `tests/`, `*_test.go`, `*.spec.ts`, etc.)
|
||||||
|
3. **Test utilities:** What helpers exist? Look for `testutils/`, `test_helpers/`, `fixtures/`, `conftest.py`, etc.
|
||||||
|
4. **Test infrastructure:** How are integration tests run? Embedded servers, Docker, test containers, etc.
|
||||||
|
5. **Naming conventions:** How are tests named? What style do existing tests follow?
|
||||||
|
|
||||||
|
Match the project's existing patterns — a reproducer that follows project conventions is easier to review and integrate.
|
||||||
|
|
@ -0,0 +1,126 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Detect project context: language, build system, test framework, test command.
|
||||||
|
# Prints key=value pairs to stdout.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
DIR="${1:-.}"
|
||||||
|
|
||||||
|
detect_language() {
|
||||||
|
if [ -f "$DIR/Cargo.toml" ]; then echo "rust"
|
||||||
|
elif [ -f "$DIR/pom.xml" ] || [ -f "$DIR/build.gradle" ] || [ -f "$DIR/build.gradle.kts" ]; then echo "java"
|
||||||
|
elif [ -f "$DIR/pyproject.toml" ] || [ -f "$DIR/setup.py" ] || [ -f "$DIR/setup.cfg" ]; then echo "python"
|
||||||
|
elif [ -f "$DIR/go.mod" ]; then echo "go"
|
||||||
|
elif [ -f "$DIR/package.json" ]; then echo "javascript"
|
||||||
|
elif [ -f "$DIR/CMakeLists.txt" ] || [ -f "$DIR/Makefile" ]; then echo "c_cpp"
|
||||||
|
elif [ -f "$DIR/mix.exs" ]; then echo "elixir"
|
||||||
|
elif [ -f "$DIR/build.sbt" ]; then echo "scala"
|
||||||
|
elif [ -f "$DIR/project.clj" ] || [ -f "$DIR/deps.edn" ]; then echo "clojure"
|
||||||
|
else echo "unknown"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
detect_build_system() {
|
||||||
|
if [ -f "$DIR/Cargo.toml" ]; then echo "cargo"
|
||||||
|
elif [ -f "$DIR/pom.xml" ]; then echo "maven"
|
||||||
|
elif [ -f "$DIR/build.gradle" ] || [ -f "$DIR/build.gradle.kts" ]; then echo "gradle"
|
||||||
|
elif [ -f "$DIR/pyproject.toml" ]; then echo "pyproject"
|
||||||
|
elif [ -f "$DIR/setup.py" ]; then echo "setuptools"
|
||||||
|
elif [ -f "$DIR/go.mod" ]; then echo "go_modules"
|
||||||
|
elif [ -f "$DIR/package.json" ]; then
|
||||||
|
if [ -f "$DIR/pnpm-lock.yaml" ]; then echo "pnpm"
|
||||||
|
elif [ -f "$DIR/yarn.lock" ]; then echo "yarn"
|
||||||
|
else echo "npm"
|
||||||
|
fi
|
||||||
|
elif [ -f "$DIR/CMakeLists.txt" ]; then echo "cmake"
|
||||||
|
elif [ -f "$DIR/Makefile" ]; then echo "make"
|
||||||
|
else echo "unknown"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
detect_test_framework() {
|
||||||
|
local lang
|
||||||
|
lang=$(detect_language)
|
||||||
|
case "$lang" in
|
||||||
|
rust) echo "cargo_test" ;;
|
||||||
|
java)
|
||||||
|
if grep -rql "junit-jupiter\|org.junit.jupiter" "$DIR/pom.xml" "$DIR/build.gradle" "$DIR/build.gradle.kts" 2>/dev/null; then
|
||||||
|
echo "junit5"
|
||||||
|
elif grep -rql "junit\|org.junit" "$DIR/pom.xml" "$DIR/build.gradle" "$DIR/build.gradle.kts" 2>/dev/null; then
|
||||||
|
echo "junit4"
|
||||||
|
else
|
||||||
|
echo "junit5" # default for java
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
python)
|
||||||
|
if grep -rql "pytest" "$DIR/pyproject.toml" "$DIR/setup.cfg" "$DIR/tox.ini" 2>/dev/null; then
|
||||||
|
echo "pytest"
|
||||||
|
elif [ -f "$DIR/tox.ini" ]; then
|
||||||
|
echo "tox"
|
||||||
|
else
|
||||||
|
echo "pytest" # default for python
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
go) echo "go_test" ;;
|
||||||
|
javascript)
|
||||||
|
if grep -q '"vitest"' "$DIR/package.json" 2>/dev/null; then echo "vitest"
|
||||||
|
elif grep -q '"jest"' "$DIR/package.json" 2>/dev/null; then echo "jest"
|
||||||
|
elif grep -q '"mocha"' "$DIR/package.json" 2>/dev/null; then echo "mocha"
|
||||||
|
else echo "jest"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
*) echo "unknown" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
detect_test_command() {
|
||||||
|
local framework
|
||||||
|
framework=$(detect_test_framework)
|
||||||
|
case "$framework" in
|
||||||
|
cargo_test) echo "cargo test" ;;
|
||||||
|
junit5|junit4)
|
||||||
|
if [ -f "$DIR/pom.xml" ]; then echo "mvn test"
|
||||||
|
else echo "gradle test"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
pytest) echo "pytest" ;;
|
||||||
|
tox) echo "tox" ;;
|
||||||
|
go_test) echo "go test ./..." ;;
|
||||||
|
vitest) echo "npx vitest run" ;;
|
||||||
|
jest) echo "npx jest" ;;
|
||||||
|
mocha) echo "npx mocha" ;;
|
||||||
|
*) echo "unknown" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
detect_test_dirs() {
|
||||||
|
local lang
|
||||||
|
lang=$(detect_language)
|
||||||
|
case "$lang" in
|
||||||
|
java)
|
||||||
|
if [ -d "$DIR/src/test" ]; then echo "src/test/java"
|
||||||
|
elif [ -d "$DIR/test" ]; then echo "test"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
python)
|
||||||
|
if [ -d "$DIR/tests" ]; then echo "tests"
|
||||||
|
elif [ -d "$DIR/test" ]; then echo "test"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
rust) echo "src (inline) or tests/" ;;
|
||||||
|
go) echo "alongside source (*_test.go)" ;;
|
||||||
|
javascript)
|
||||||
|
if [ -d "$DIR/__tests__" ]; then echo "__tests__"
|
||||||
|
elif [ -d "$DIR/tests" ]; then echo "tests"
|
||||||
|
elif [ -d "$DIR/test" ]; then echo "test"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
*) echo "unknown" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "language=$(detect_language)"
|
||||||
|
echo "build_system=$(detect_build_system)"
|
||||||
|
echo "test_framework=$(detect_test_framework)"
|
||||||
|
echo "test_command=$(detect_test_command)"
|
||||||
|
echo "test_dirs=$(detect_test_dirs)"
|
||||||
|
|
@ -0,0 +1,125 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Line-level delta debugging minimizer.
|
||||||
|
|
||||||
|
Given an input file and a test command, reduces the file to a minimal
|
||||||
|
set of lines that still causes the test command to exit with a non-zero
|
||||||
|
status (or match a specific pattern).
|
||||||
|
|
||||||
|
Based on Zeller's ddmin algorithm.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python shrink_text.py INPUT_FILE TEST_COMMAND [--pattern REGEX]
|
||||||
|
|
||||||
|
INPUT_FILE: the file to minimize
|
||||||
|
TEST_COMMAND: command that returns 0 when the input "still fails"
|
||||||
|
(i.e., the bug is still present). The command receives
|
||||||
|
the candidate file path as its last argument.
|
||||||
|
--pattern: optional regex that must appear in the command's
|
||||||
|
stdout/stderr for the failure to count as "same bug"
|
||||||
|
|
||||||
|
Example:
|
||||||
|
python shrink_text.py crash_input.sql "./check_bug.sh" --pattern "IndexOutOfBounds"
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
|
||||||
|
|
||||||
|
def read_lines(path):
|
||||||
|
with open(path, 'r') as f:
|
||||||
|
return f.readlines()
|
||||||
|
|
||||||
|
|
||||||
|
def write_lines(path, lines):
|
||||||
|
with open(path, 'w') as f:
|
||||||
|
f.writelines(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def is_interesting(lines, test_cmd, pattern, tmpdir):
|
||||||
|
"""Returns True if the reduced input still triggers the bug."""
|
||||||
|
candidate = os.path.join(tmpdir, 'candidate')
|
||||||
|
write_lines(candidate, lines)
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
test_cmd + [candidate],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=120,
|
||||||
|
)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
return False
|
||||||
|
|
||||||
|
if result.returncode == 0:
|
||||||
|
return False
|
||||||
|
|
||||||
|
if pattern:
|
||||||
|
combined = result.stdout + result.stderr
|
||||||
|
if not re.search(pattern, combined):
|
||||||
|
return False
|
||||||
|
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def ddmin(lines, test_fn):
|
||||||
|
"""Delta debugging minimization at line granularity."""
|
||||||
|
n = 2
|
||||||
|
while len(lines) >= 2:
|
||||||
|
chunk_size = max(len(lines) // n, 1)
|
||||||
|
some_progress = False
|
||||||
|
|
||||||
|
for i in range(0, len(lines), chunk_size):
|
||||||
|
complement = lines[:i] + lines[i + chunk_size:]
|
||||||
|
if not complement:
|
||||||
|
continue
|
||||||
|
if test_fn(complement):
|
||||||
|
lines = complement
|
||||||
|
n = max(n - 1, 2)
|
||||||
|
some_progress = True
|
||||||
|
print(f" reduced to {len(lines)} lines", file=sys.stderr)
|
||||||
|
break
|
||||||
|
|
||||||
|
if not some_progress:
|
||||||
|
if n >= len(lines):
|
||||||
|
break
|
||||||
|
n = min(n * 2, len(lines))
|
||||||
|
|
||||||
|
return lines
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description='Line-level delta debugging minimizer')
|
||||||
|
parser.add_argument('input_file', help='File to minimize')
|
||||||
|
parser.add_argument('test_command', nargs='+', help='Command that returns 0 when bug is present')
|
||||||
|
parser.add_argument('--pattern', help='Regex that must appear in output for same-bug check')
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
original_lines = read_lines(args.input_file)
|
||||||
|
print(f"Original: {len(original_lines)} lines", file=sys.stderr)
|
||||||
|
|
||||||
|
pattern = re.compile(args.pattern) if args.pattern else None
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory() as tmpdir:
|
||||||
|
def test_fn(lines):
|
||||||
|
return is_interesting(lines, args.test_command, pattern, tmpdir)
|
||||||
|
|
||||||
|
if not test_fn(original_lines):
|
||||||
|
print("ERROR: original input does not trigger the bug with the given command",
|
||||||
|
file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
reduced = ddmin(original_lines, test_fn)
|
||||||
|
|
||||||
|
output_path = args.input_file + '.reduced'
|
||||||
|
write_lines(output_path, reduced)
|
||||||
|
print(f"Reduced: {len(reduced)} lines (from {len(original_lines)})", file=sys.stderr)
|
||||||
|
print(f"Written to: {output_path}", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
|
|
@ -0,0 +1,119 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Validate a reproducer: run it and check the oracle.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# validate_repro.sh COMMAND [--pattern REGEX] [--exit-code N] [--runs N]
|
||||||
|
#
|
||||||
|
# Examples:
|
||||||
|
# validate_repro.sh "pytest tests/test_repro.py -v" --pattern "AssertionError"
|
||||||
|
# validate_repro.sh "mvn test -pl core -Dtest=ReproTest" --exit-code 1
|
||||||
|
# validate_repro.sh "cargo test repro_issue" --pattern "panicked" --runs 10
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
COMMAND=""
|
||||||
|
PATTERN=""
|
||||||
|
EXPECTED_EXIT=""
|
||||||
|
RUNS=1
|
||||||
|
TIMEOUT=120
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--pattern) PATTERN="$2"; shift 2 ;;
|
||||||
|
--exit-code) EXPECTED_EXIT="$2"; shift 2 ;;
|
||||||
|
--runs) RUNS="$2"; shift 2 ;;
|
||||||
|
--timeout) TIMEOUT="$2"; shift 2 ;;
|
||||||
|
*)
|
||||||
|
if [ -z "$COMMAND" ]; then
|
||||||
|
COMMAND="$1"
|
||||||
|
else
|
||||||
|
COMMAND="$COMMAND $1"
|
||||||
|
fi
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "$COMMAND" ]; then
|
||||||
|
echo "Usage: validate_repro.sh COMMAND [--pattern REGEX] [--exit-code N] [--runs N]" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
FAILURES=0
|
||||||
|
PASSES=0
|
||||||
|
ERRORS=0
|
||||||
|
|
||||||
|
for i in $(seq 1 "$RUNS"); do
|
||||||
|
OUTPUT_FILE=$(mktemp)
|
||||||
|
EXIT_CODE=0
|
||||||
|
|
||||||
|
if timeout "$TIMEOUT" bash -c "$COMMAND" > "$OUTPUT_FILE" 2>&1; then
|
||||||
|
EXIT_CODE=0
|
||||||
|
else
|
||||||
|
EXIT_CODE=$?
|
||||||
|
fi
|
||||||
|
|
||||||
|
COMBINED=$(cat "$OUTPUT_FILE")
|
||||||
|
rm -f "$OUTPUT_FILE"
|
||||||
|
|
||||||
|
# Check exit code
|
||||||
|
if [ -n "$EXPECTED_EXIT" ] && [ "$EXIT_CODE" != "$EXPECTED_EXIT" ]; then
|
||||||
|
ERRORS=$((ERRORS + 1))
|
||||||
|
if [ "$RUNS" -eq 1 ]; then
|
||||||
|
echo "UNEXPECTED EXIT CODE: expected $EXPECTED_EXIT, got $EXIT_CODE"
|
||||||
|
fi
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if the test failed (non-zero exit)
|
||||||
|
if [ "$EXIT_CODE" -eq 0 ]; then
|
||||||
|
PASSES=$((PASSES + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check pattern if specified
|
||||||
|
if [ -n "$PATTERN" ]; then
|
||||||
|
if echo "$COMBINED" | grep -qE "$PATTERN"; then
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
else
|
||||||
|
ERRORS=$((ERRORS + 1))
|
||||||
|
if [ "$RUNS" -eq 1 ]; then
|
||||||
|
echo "WRONG FAILURE: test failed but pattern '$PATTERN' not found in output"
|
||||||
|
echo "--- output ---"
|
||||||
|
echo "$COMBINED" | head -20
|
||||||
|
echo "--- end ---"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "=== Validation Report ==="
|
||||||
|
echo "Command: $COMMAND"
|
||||||
|
echo "Runs: $RUNS"
|
||||||
|
echo "Failures: $FAILURES (issue-relevant)"
|
||||||
|
echo "Passes: $PASSES"
|
||||||
|
echo "Errors: $ERRORS (wrong failure or unexpected exit)"
|
||||||
|
|
||||||
|
if [ "$FAILURES" -gt 0 ]; then
|
||||||
|
if [ "$RUNS" -eq 1 ]; then
|
||||||
|
echo ""
|
||||||
|
echo "RESULT: REPRODUCED"
|
||||||
|
else
|
||||||
|
RATE=$(echo "scale=1; $FAILURES * 100 / $RUNS" | bc)
|
||||||
|
echo "Rate: ${RATE}% ($FAILURES/$RUNS)"
|
||||||
|
echo ""
|
||||||
|
echo "RESULT: REPRODUCED (${RATE}% failure rate)"
|
||||||
|
fi
|
||||||
|
exit 0
|
||||||
|
elif [ "$PASSES" -eq "$RUNS" ]; then
|
||||||
|
echo ""
|
||||||
|
echo "RESULT: NOT REPRODUCED (all runs passed)"
|
||||||
|
exit 1
|
||||||
|
else
|
||||||
|
echo ""
|
||||||
|
echo "RESULT: INCONCLUSIVE (failures were not issue-relevant)"
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
@ -1118,6 +1118,7 @@
|
||||||
<!-- AI stuff -->
|
<!-- AI stuff -->
|
||||||
<exclude name="AGENTS.md" />
|
<exclude name="AGENTS.md" />
|
||||||
<exclude name="CLAUDE.md" />
|
<exclude name="CLAUDE.md" />
|
||||||
|
<exclude name=".claude/**" />
|
||||||
</tarfileset>
|
</tarfileset>
|
||||||
|
|
||||||
<!-- python driver -->
|
<!-- python driver -->
|
||||||
|
|
@ -1170,6 +1171,7 @@
|
||||||
<!-- AI stuff -->
|
<!-- AI stuff -->
|
||||||
<exclude name="AGENTS.md" />
|
<exclude name="AGENTS.md" />
|
||||||
<exclude name="CLAUDE.md" />
|
<exclude name="CLAUDE.md" />
|
||||||
|
<exclude name=".claude/**" />
|
||||||
</tarfileset>
|
</tarfileset>
|
||||||
<!-- Shell includes in bin/ (default mode) -->
|
<!-- Shell includes in bin/ (default mode) -->
|
||||||
<tarfileset dir="${dist.dir}" prefix="${final.name}">
|
<tarfileset dir="${dist.dir}" prefix="${final.name}">
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue