Merge 32ec311792 into 021d2135b5

## The bug

Selecting checks in Synthetic Monitoring and pressing **Move** opened a
dialog reading:

```
Current Folder:  Foo
Select Folder:   Foo
```

The same folder name twice — which reads as "move this to where it
already is".

Nothing could actually go wrong: `primary-button-disabled` already
blocked a same-folder submit. The dialog simply gave no clue *why* Move
was greyed out, because the destination it had chosen for you was the
one destination it would not accept.

## Cause

`MoveAcrossFolders` seeded `selectedFolder` from `activeFolderId`, and
`SelectFolderDropDown` listed every folder including that one.

## Fix

- `SelectFolderDropDown` gains an optional `excludeFolderId` prop: it
filters that folder out of the options and refuses to seed the selection
with it.
- `MoveAcrossFolders` passes `:excludeFolderId="activeFolderId"`, opens
with no destination chosen, and additionally disables Move while the
destination is empty — *"not the active folder"* is not the same as
*"somewhere to move to"*.
- The folder-list watcher now keeps a still-offerable choice. Creating a
folder from the **+** button arrives here as a store list change, and
the re-seed would otherwise clear the folder just created and selected.

## Why the prop is opt-in

`SelectFolderDropDown` has six other consumers — ImportAlert,
AlertList's clone dialog, CreateReport, AddSlo, SloList, and the
synthetics *duplicate* dialog. Every one of those picks a folder to file
something **into**, where opening on the active folder is correct.

Unset, the filter matches every folder (`!== undefined`) and the seeding
is untouched, so those call sites are byte-for-byte unchanged.

## Scope

This dialog is shared, so the change reaches **alerts, pipelines,
reports, SLOs and synthetics**. Behaviour is strictly narrowed: the
option removed was already a guaranteed no-op behind the disabled-submit
guard.

`MoveDashboardToAnotherFolder` uses a separate `SelectFolderDropdown`
under `components/dashboards/` and has the same behaviour, but it is a
different component tree and is deliberately left alone.

## Tests

5 new cases on the dropdown (default offers everything, excluded id
dropped, blank start, normal seeding still works, choice survives a list
change) and 1 on the dialog.

Three existing tests asserted the old seeding and are updated. **One
deserves review:** `should compute primaryButtonDisabled=false when
selectedFolder value is null` asserted that a *null* destination
**enables** Move — executing that submits `dst_folder_id: null`. I read
it as wrong rather than intentional and inverted it. If that null path
is load-bearing somewhere I have not found, this is the thing to
challenge.

## Verification

Run on this branch, based on `main`:

- `src/components/common/sidebar/` — 291 passed (6 files)
- Consumers: SyntheticMonitoring, AlertList, ImportAlert,
MoveDashboardToAnotherFolder — 216 passed, 10 pre-existing skips
- `eslint` on all four changed files — clean
- `npm run lint:design:strict` — OK, no regressions
- `npm run type-check` — exit 0

.claude/skills/ui-architect/SKILL.md

@@ -333,6 +333,16 @@ considering the UI done:
       from `<div>` + utility classes. Classes are for layout only.
 - [ ] Tabular data uses `OTable` with `OTableColumnDef[]` columns; server mode
       only for backend-paginated data.
+- [ ] **Server mode was checked against the backend**: every `sortable: true`
+      column has a real sort key in the handler (an unknown key falls back
+      silently and orders by something else), and any page-relative device
+      (`ODataBarCell` bars, a `#subheader` count strip) is on a **client**-paginated
+      table only. No hand-rolled `#` index column (`show-index`), no positional
+      `columns.splice`, and column `size` fits the header + sort chevron.
+      See [core-controls-table](references/core-controls-table.md).
+- [ ] A figure/label that already exists on another screen reuses **that screen's
+      formatter and i18n key** (promote a component-local formatter into
+      `utils/formatters.ts` rather than copying it).
 - [ ] Listing page uses the **full-height flush skeleton** (root
       `flex flex-col h-full p-0`, header `shrink-0 border-b` — OPageHeader bakes
       in its own `px-page-edge`, never add a `px-*`, table wrapper

.claude/skills/ui-architect/references/calm-signal.md

@@ -52,10 +52,36 @@ All token-backed and dark-mode-safe. Reuse these before inventing anything.
   `badgeGroups.ts` (`alertStatus`, `alertType`, `severity`, `streamType`,
   `userRole`, `serviceStatus`, …). One registry → the same value is the same
   colour everywhere. Need a new family? Add a group there, don't hand-roll a pill.
+  Check the group covers **every** value the API can return (a missing key falls
+  back to a generic chip) — and don't set `size` on the group: a group-level size
+  silently overrides the call sites, so its chips end up a different size from the
+  Status chip beside them. Pass `size` at the call site, matching its siblings.
 - **Row state signal** — an extreme-left colour **rail** via `OTable`'s
   per-row `getRowStyle` (inset box-shadow, rem width, token colour) + a **light
-  exception highlight** via `row-class` (tint only the rows that need action —
-  never the normal ones).
+  exception highlight** via `row-class`. The rail and the wash are two different
+  strengths of the same signal, and the rail is the default:
+
+  | | Rail (`getRowStyle`) | Wash (`row-class`) |
+  | --- | --- | --- |
+  | Cost | a few px at the row edge | ~the whole row |
+  | Use for | **every** state, always | only the two cases below |
+
+  A full-row wash is the loudest thing on a list, so it earns its place only when
+  **all three** hold: the state means **act now** (not merely "not green"), it is
+  **rare in a healthy system**, and **the row is the unit of action**. In practice
+  that leaves exactly two washes:
+  - `!bg-status-error-bg` — failing/errored/offline. *Alerts* failed, *Pipelines*
+    errored, *Nodes* offline, *Synthetics* failed.
+  - `!bg-surface-panel` — paused/disabled. This one is **de-emphasis, not alarm**:
+    the row is deliberately inert, so it recedes rather than shouts.
+
+  Everything else keeps a clean row and reads from the rail — including states
+  that are "bad but not urgent": **degraded/warning** (worth noticing, not worth
+  acting on this second), **stale**, **unknown**, **never-ran**. And whatever the
+  state means, if it is **common** the wash is wrong regardless: *Streams* drops it
+  entirely, because "never ingested" and "quiet for a day" describe a large share
+  of rows in a normal org and a table where most rows are tinted signals nothing.
+  Pages with no failure state at all — Users, Roles, Dashboards — never wash.
 - **Recency** — `OTimeCell` `mode="relative"` (`"3 min ago"`) with a hot/warm/
   cold dot, instead of a raw timestamp column.
 - **People** — `OUserCell` for owner/author columns.
@@ -77,15 +103,43 @@ Once a strip is `selectable`, four rules keep every strip behaving identically
   active → paused`, `P1 → P2 → P3 → P4`, `degraded → paused → active → draft →
   archived`. This mirrors the row rail's "critical colour on the left edge": what
   needs attention lives on the left, everywhere.
-- **The "All" / "Total" tile is LAST and never highlighted.** It only clears the
-  facet; it is never itself the active tile. Wire `:selected-key` to the *raw
-  filter value* (`null` / `"all"` when unfiltered) so nothing shows the ring while
-  viewing everything — never fall back to selecting "All" as the default.
+- **The "All" / "Total" tile is LAST, stays CLICKABLE, and is never highlighted.**
+  It only clears the facet; it is never itself the active tile. Wire `:selected-key`
+  to the *raw filter value* (`null` / `"all"` when unfiltered) so nothing shows the
+  ring while viewing everything — never fall back to selecting "All" as the default.
+  **Never put `selectable: false` on it** — that is the one wrong way to express
+  "never highlighted": it turns the tile into a plain `<div>`, so clicking it does
+  nothing and the strip has no way back to unfiltered (the bug users report as
+  "All isn't clickable"). "Never highlighted" is achieved by the `selectedKey`
+  wiring above, not by disabling the tile. `selectable: false` is only for a tile
+  that is genuinely not a facet at all (a pure read-out with no matching filter).
 - **Selection toggles off.** Re-clicking the already-active tile clears the filter
   (back to unfiltered), matching every other strip:
   `onSelect(key) → filter = key === "all" || filter === key ? cleared : key`.
 - **Selected state is an accent border, not a fill** — the `OStatCard` default;
   see Step 3.
+- **A strip in `#subheader` is a CLAIM ABOUT THE ROWS — scope decides position.**
+  Inside the table frame, a strip promises "these numbers describe the list below",
+  so it must be computed from the same filtered set *and* be facet-clickable. Before
+  writing one, check the fetch: **if the list is server-paginated you only hold one
+  page**, so page-local sums are not totals — never sum the visible page and label
+  it "Total". When the numbers genuinely can't meet the promise (server pagination,
+  or totals that come from a different, wider endpoint such as an org-summary API),
+  do **not** put them in `#subheader` — lift them to the page level as a read-only
+  strip in `OPageLayout`'s **`#subnav`**, where they read as page context instead of
+  a row summary, and label them for their real scope ("Total Streams", not
+  "Streams"). What stays honest under pagination is anything derived from the single
+  row in front of you: relative recency, a state rail, a per-row ratio. Reference:
+  Alerts (client-side list → filterable `#subheader` strip) vs Streams
+  (server-paginated list → org footprint in `#subnav`).
+- **`ODataBarCell` needs the whole set — client-paginated tables only.** Its bar is
+  a share of the `max` the caller computes over the rendered rows, so on a
+  **server-paginated** table it silently means "biggest on this page": the scale
+  changes as you page, and the same stream draws a different bar on page 1 and page
+  4. On such a table drop the bars and let the (sortable) numbers rank the rows —
+  right-aligned + `tabular-nums` already scans fine, and the stray part-width
+  underlines read as artefacts rather than data. Same test as the strip: can this
+  mark be computed from data you actually hold?
 
 **Tile → section → drawer linkage.** When a tile drills into a table or a detail
 drawer, reuse the **same glyph + tone** on the section header and the drawer header
@@ -97,12 +151,55 @@ strip feeds a drill-down.
 
 ---
 
+### Before you colour a state, check how it CLEARS
+
+A State column is a claim about *now*. Read the write path of whatever field backs
+it and confirm something resets it — an expiry/retention job, a success write, a
+status transition. A sticky error field (written on failure, never cleared) pins a
+row to "Errored" forever and the column becomes noise within a week. Pipelines'
+`last_error` is safe because the backend expires it on a retention interval;
+verify the equivalent before promoting any field to a chip. If nothing clears it,
+label it for what it is ("Last error", a timestamp) instead of a live state.
+
+### Grey vs amber — the two "not green" states
+
+They are not interchangeable, and the wrong pick cries wolf:
+
+- **Grey = no data / unknown / not in use.** Never ingested, never run, not yet
+  configured. Usually benign — a stream created five minutes ago, a schema-only
+  stream, a job that has not had its first run. Pair with a muted "Never".
+- **Amber = it WAS working and went quiet, or is degrading.** Silence from a thing
+  that used to report is the case worth a second look.
+
+Amber on the "never" case fires on every freshly created object, which trains
+people to ignore the colour. When in doubt, grey.
+
 ## Step 3 — Keep everything else calm
 
 Colour only earns attention if most of the screen stays quiet:
 
+- **Earn every tile — a strip is not a page decoration.** Before adding one, put
+  each tile to three questions: does the number **vary**, does someone **act** on
+  it, and is it **not already on screen**? Tiles that fail are noise dressed as
+  signal:
+  - *structurally constant* — "System accounts" is 1 in almost every org, so the
+    tile is a label with a number stuck to it;
+  - *almost always zero* — "New this week" on a list that gains an item a quarter;
+  - *derivable from its neighbours* — "In use" beside "Unused" and "Total";
+  - *already in the footer* — `footerTitle` renders "N Dashboards" under every
+    table, so a Total tile alone is not a reason to have a strip.
+  A page whose only candidates fail these gets **no strip** — keep the per-row
+  signals (relative recency, a state rail, a count column) and stop. Dashboards,
+  Service Accounts and Roles all ended up here: pages where a strip added pixels
+  and no information. Roles is the clearest case — "Unused" is just the member
+  column sorted ascending, so two tiles restated what the rows already said. **A
+  count column plus sorting usually beats a strip**; reach for a strip only when
+  the page has a real distribution to summarise (Users across roles, monitors
+  across health) *and* the tiles double as the facet.
 - **Highlight exceptions, not the norm.** Tint the failed/paused rows; leave the
   healthy majority clean. A table where every row is coloured signals nothing.
+  And if a page has no true failure state (a catalog list), the calm answer is a
+  rail with no row wash at all — not a wash invented for symmetry with Alerts.
 - **Muted zero.** A `0` renders muted, not in the loud tone colour, and "no
   data" is a `—`, not a wall of zeros. (`OStatCard` does this.)
 - **State is border/colour, not fills.** Selected/hover on interactive tiles use

.claude/skills/ui-architect/references/conventions.md

@@ -284,6 +284,31 @@ view (src/views)
 test mock a service, and keeps the store from silting up with ephemeral state. A
 component that calls axios directly can't be reused or tested without a network.
 
+### The same figure must render identically everywhere
+
+**What.** When a number/label you're adding already appears on another screen —
+the same API field, the same meaning — reuse that screen's **formatter and i18n
+key**; don't re-derive either. If the existing formatter is a private function
+inside a component, promote it to `utils/formatters.ts` and have both import it.
+
+**Why.** Two call sites formatting one field always drift: Home → Usage printed
+`2.9B` while a new Streams tile printed `2,900,000,000` from the identical
+`/summary` payload, and a duplicated `formatEventCount` was the cause. Same for
+labels — parallel `page.summaryStreams` / `home.streams` keys with the same
+English value will diverge the first time someone edits one. Sharing the key is
+what makes the two surfaces stay in sync.
+
+**How.**
+- Before adding a formatter, grep `utils/formatters.ts` (and the sibling screen)
+  for one that already exists — `formatEventCount`, `formatSizeFromMB`,
+  `addCommasToNumber`, `formatLargeNumber`, `formatTimeWithSuffix`.
+- Reuse the other screen's i18n key outright when the label is the same thing
+  (`t("home.totalDataIngested")` from the Streams page is correct, not a
+  duplicate). A new key is for a genuinely new label.
+- Match the icon and tone family too when the tiles sit on comparable surfaces —
+  but never copy a *semantic* colour (green/amber/red) onto a non-health number
+  just to match; that's the one thing worth diverging on.
+
 ### Registering a new page in navigation
 
 **What.** A new page isn't reachable until it's (1) registered as a **route** and

.claude/skills/ui-architect/references/core-controls-table.md

@@ -304,6 +304,53 @@ interface OTableColumnDef<TData = any> {
 }
 ```
 
+**Server mode — check the backend BEFORE you write the column.** In
+`sorting="server"` the table does not sort anything: it emits the column **id** as
+the `sort` field and the backend does the work. OpenObserve's list endpoints match
+that field against a fixed list and **fall through to a default on anything else**
+(`stream_comparator`'s `_ => a.name.cmp(&b.name)` is typical) — no 400, no console
+error, just rows quietly ordered by something else. Symptom: the header shows its
+arrow, the rows re-order, and the order is wrong.
+
+- Open the handler and read its sort match/allow-list. A column gets
+  `sortable: true` **only if a key exists for it**; otherwise `sortable: false`
+  plus a one-line comment naming the missing key. Never leave a control that
+  silently lies.
+- Name the column `id` after the backend key (`doc_time_max`, not `lastIngested`)
+  so the mapping is obvious, or map it explicitly via `sortFieldMap`.
+- **Derived columns can't be rescued client-side.** Under server pagination you
+  hold one page of N rows, so sorting a computed ratio locally would just reorder
+  that page. Either add the sort key to the backend or ship the column unsorted.
+- Same reasoning for `filterMode="server"` and any `keyword`/facet param: if the
+  endpoint ignores it, the UI shows an unfiltered list as if it were filtered.
+- **The dev UI usually talks to a REMOTE backend**, not your working tree —
+  `web/.env`'s `VITE_OPENOBSERVE_ENDPOINT` points at a shared dev deployment. A
+  Rust change in your branch is not live there, so "the API ignores my new
+  param" is expected until it deploys. Confirm which is which before debugging the
+  frontend: check the request in DevTools (the param IS being sent), then check
+  whether the endpoint you're hitting is local.
+
+**Column hygiene** — mistakes that keep recurring:
+
+- **Never hand-roll a `#` index column.** A data column with `accessorKey: "id"`
+  is draggable, resizable, hideable, and its numbers stop matching the visible
+  rows the moment the table is sorted or filtered. Pass `showIndex` and let OTable
+  render the fixed gutter.
+- **Hide conditional columns by id, not by position** — `columns.splice(2, 1)`
+  silently removes the wrong column as soon as someone inserts one above it. Use
+  `columns.filter((c) => c.id !== "region")`.
+- **Size a column for its header plus the sort chevron.** The shared `COL.*`
+  presets fit typical *values*, not long labels — `COL.sizeBytes` truncates
+  "Compressed Size" to "Compressed Si…". Set an explicit `size` (+ `minSize` floor)
+  when the header is longer than the numbers under it.
+- **`persistColumns` persists widths as well as visibility**, keyed by `tableId`
+  in localStorage. Once a user drags a column, later changes to that column's
+  default `size` never reach them — test width changes in a fresh profile, and
+  point users at the column menu's reset action.
+- **One number per numeric cell.** Two values in one right-aligned cell means
+  neither owns the column's edge and both look indented at random; give the second
+  value its own column.
+
 **Main emits:**
 - Pagination: `update:currentPage`, `update:pageSize`, `pagination-change`
 - Sorting: `update:sortBy`, `update:sortOrder`, `sort-change`
@@ -368,7 +415,7 @@ Prebuilt cell components — pass as a column's `cell`. Import individually or f
 - **OUserCell** (`@/lib/core/Table/cells/OUserCell.vue`) — person/owner/created-by column; renders email or explicit `name` as truncated plain text, dash when empty.
 - **ONumberCell** (`@/lib/core/Table/cells/ONumberCell.vue`) — consistent numeric rendering (tabular-nums); `format` = `number`/`compact`/`bytesFromMB`/`durationSec`/`durationMs`/`durationUs`/`durationNs`/`percent`. Pair the column with `meta.align: "right"`.
 - **OCodeCell** (`@/lib/core/Table/cells/OCodeCell.vue`) — monospace identifiers / SQL / tokens, truncated with title tooltip and optional hover copy button (`copy`, default true).
-- **ODataBarCell** (`@/lib/core/Table/cells/ODataBarCell.vue`) — value with a proportional background bar (width = value/`max`, caller supplies the column max and pre-formatted `display`); `variant` `default`/`warning`/`danger` for threshold columns.
+- **ODataBarCell** (`@/lib/core/Table/cells/ODataBarCell.vue`) — value with a proportional background bar (width = value/`max`, caller supplies the column max and pre-formatted `display`); `variant` `default`/`warning`/`danger` for threshold columns. ⚠️ **Client-paginated tables only:** the caller can only compute `max` over the rows it holds, so on a server-paginated table the bar means "biggest on this page" — the scale shifts as you page and the same row draws a different bar. There, drop the bars and let sortable numbers rank the rows.
 
 Also exported from the barrel: `statusVariant`, `humanizeStatus` helpers (+ `StatusTone`, `StatusVariantResult` types) for status badge styling.
 

.github/workflows/playwright.yml

@@ -285,6 +285,13 @@ jobs:
                 "alerts-template-prebuilt-guard.spec.js",
                 "alerts-form-validation.spec.js",
                 "anomaly-detection-form-validation.spec.js",
+                # Alerts 4.0 (multi-alerts) e2e — PR #13547
+                "alerts-priority-tags.spec.js",
+                "alerts-multialert-api.spec.js",
+                "alerts-multialert-ui.spec.js",
+                "alerts-multialert-regression.spec.js",
+                "alerts-multialert-firing.spec.js",
+                "alerts-multialert-transitions.spec.js",
               ]
           - testfolder: "Dashboards-Core"
             browser: "chrome"
@@ -321,6 +328,12 @@ jobs:
                 "dashboard-legends-copy.spec.js",
                 "dashboard-raw-query.spec.js",
               ]
+          # Dashboard panel tests are grouped by the feature they exercise so
+          # each shard stays well under the ephemeral-runner endurance window.
+          # A single combined 16-spec shard ran ~40 min on CI and got reclaimed
+          # mid-run ("runner lost communication"). Add new specs to the shard of
+          # the feature they cover; the Tables shard also owns the pagination and
+          # pivot-table specs (previously their own single-spec shards).
           - testfolder: "Dashboards-Charts"
             browser: "chrome"
             run_files:
@@ -330,17 +343,29 @@ jobs:
                 "dashboard-maps.spec.js",
                 "dashboard-multi-y-axis.spec.js",
                 "dashboard-html-chart.spec.js",
-                "visualize.spec.js",
-                "visualize-vrl.spec.js",
-                "dashboard-create-alert.spec.js",
                 "dashboard-sankey.spec.js",
                 "dashboard-pie-donut.spec.js",
+                "dashboard-metric-camelcase.spec.js",
+                "dashboard-create-alert.spec.js",
+              ]
+          - testfolder: "Dashboards-Visualize"
+            browser: "chrome"
+            run_files:
+              [
+                "visualize.spec.js",
+                "visualize-vrl.spec.js",
+              ]
+          - testfolder: "Dashboards-Tables"
+            browser: "chrome"
+            run_files:
+              [
                 "dashboard-table-chart.spec.js",
                 "dashboard-table-copy-cell.spec.js",
                 "dashboard-table-csv-download.spec.js",
                 "dashboard-table-column-formatting.spec.js",
                 "dashboard-table-column-filtering.spec.js",
-                "dashboard-metric-camelcase.spec.js",
+                "dashboard-table-pagination.spec.js",
+                "dashboard-pivot-table.spec.js",
               ]
           - testfolder: "Dashboards-Streaming"
             browser: "chrome"
@@ -460,12 +485,6 @@ jobs:
                 "dashboard-variables-stream-field.spec.js",
                 "dashboard-mustache-variables.spec.js",
               ]
-          - testfolder: "Dashboard-Table-Pagination"
-            browser: "chrome"
-            run_files: ["dashboard-table-pagination.spec.js"]
-          - testfolder: "Dashboard-Pivot-Table"
-            browser: "chrome"
-            run_files: ["dashboard-pivot-table.spec.js"]
           - testfolder: "Dashboards-Panel-Level-DateTime-Config"
             browser: "chrome"
             run_files:
@@ -615,7 +634,7 @@ jobs:
               "Logs-Builder-Basic"|"Logs-Builder-Advanced"|"Logs-Core"|"Logs-Features")
                 ACTUAL_FOLDER="Logs"
                 ;;
-              "Dashboards-Core"|"Dashboards-Settings"|"Dashboards-Charts"|"Dashboards-Streaming"|"Dashboards-Variables"|"Dashboard-Table-Pagination"|"Dashboard-Pivot-Table"|"Dashboards-Panel-Level-DateTime-Config"|"Dashboard-Config-Settings"|"Dashboards-Isolated")
+              "Dashboards-Core"|"Dashboards-Settings"|"Dashboards-Charts"|"Dashboards-Visualize"|"Dashboards-Tables"|"Dashboards-Streaming"|"Dashboards-Variables"|"Dashboards-Panel-Level-DateTime-Config"|"Dashboard-Config-Settings"|"Dashboards-Isolated")
                 ACTUAL_FOLDER="Dashboards"
                 ;;
               "RUM-Token")

.gitignore

@@ -5,6 +5,11 @@ report.json
 flamegraph.svg
 flamegraph.html
 
+# node
+node_modules/
+.vite/
+.vitest/
+
 # Generated by Cargo
 # will have compiled files and executables
 /examples/
@@ -92,3 +97,11 @@ tests/ui-testing/.env.old
 /.claude/skills/*
 !/.claude/skills/ui-architect/
 !/.claude/skills/eslint-error-handling/
+
+alert_mock
+node_modules/
+# Local verification artifacts (browser screenshots, per-crate scratch DBs)
+/alerts-list-f2.png
+/final-list.png
+/form-advanced.png
+src/*/data/

Cargo.lock

@@ -2480,12 +2480,14 @@ dependencies = [
  "itertools 0.14.0",
  "log",
  "o2_enterprise",
+ "object_store",
  "parking_lot 0.12.5",
  "parquet",
  "rand 0.10.1",
  "schema",
  "search",
  "search_service",
+ "tantivy",
  "tantivy_utils",
  "tokio",
 ]
@@ -2551,6 +2553,7 @@ dependencies = [
  "dotenv_config",
  "dotenvy",
  "expect-test",
+ "fastdivide",
  "faststr",
  "float-cmp",
  "futures",
@@ -7258,6 +7261,7 @@ dependencies = [
  "openobserve-api-grpc",
  "openobserve-api-http",
  "openobserve-api-management",
+ "openobserve-api-pipelines",
  "openobserve-core",
  "openobserve-jobs",
  "openobserve-mcp",
@@ -10529,6 +10533,7 @@ dependencies = [
  "futures",
  "futures-util",
  "hashbrown 0.16.1",
+ "hashlink 0.11.0",
  "infra",
  "itertools 0.14.0",
  "log",

Cargo.toml

@@ -223,6 +223,7 @@ sourcemap.workspace = true
 audit.workspace = true
 openobserve-api-http.workspace = true
 openobserve-api-grpc.workspace = true
+openobserve-api-management.workspace = true
 web.workspace = true
 openobserve-api-common.workspace = true
 openobserve-core.workspace = true
@@ -243,7 +244,7 @@ openobserve-mcp.workspace = true
 arrow.workspace = true
 bytes.workspace = true
 enrichment-data.workspace = true
-openobserve-api-management.workspace = true
+openobserve-api-pipelines.workspace = true
 expect-test.workspace = true
 base64.workspace = true
 float-cmp.workspace = true
@@ -455,6 +456,7 @@ sqlparser = { version = "0.62", features = ["serde", "visitor"] }
 dotenv_config = "0.2"
 dotenvy = "0.15"
 env_logger = "0.11"
+fastdivide = "0.4"
 faststr = { version = "0.2", features = ["serde"] }
 flate2 = { version = "1.0", features = ["zlib"] }
 futures = "0.3"

I18N_ENFORCEMENT_GUIDE.md

@@ -3,13 +3,28 @@
 How the OpenObserve web app guarantees that user-facing text is translatable, and
 what to do when a check fires.
 
-**Status: enforcement complete and green.** `lint:ci` 0 errors · `type-check:app`
-0 errors · spec-inclusive `type-check` 0 errors · `format:check` clean.
+**Status: enforcement complete.** `lint:ci` 0 errors · `type-check:app` **exit 0**
+(with `--composite false`, which the npm script passes, no TS6307 surfaces) ·
+`format:check` clean.
+
+> **Known measurement gap:** the "spec-inclusive" `type-check` (tsconfig.vitest.json)
+> currently checks **no spec files at all** — its include is `src/**/*.spec.{ts,js}`,
+> and TypeScript globs do **not** brace-expand, so the pattern matches nothing.
+> Any past "spec-inclusive type-check 0 errors" claim was vacuous. Fixing it means
+> splitting the include into `src/**/*.spec.ts` + `src/**/*.spec.js` AND then fixing
+> the spec-only type errors that surface (e.g. `metricGrouping.spec.ts` assigns keys
+> that don't exist in en-US to `I18nKey` fixtures). Tracked in §9.
 
 The mechanisms are done and so is the message cleanup — **0 fake plurals** remain in
-en-US. What is left is narrower: **non-English plural rules**, the last **6 `gt` sites**,
-and **one CI gap**. See [§9 Outstanding work](#9-outstanding-work) — that section is the
-to-do list, kept in sync with measurements.
+en-US, the 10 surviving `gt()` calls are each individually justified (§8), `TEXT_ATTRS`
+has been retired in favour of prop types (§8), and the dead-key sweep is complete (§9).
+What is left is narrower: **non-English plural rules** and **spec type-checking**
+(a broken tsconfig glob plus a missing PR gate, §9.3). See
+[§9 Outstanding work](#9-outstanding-work) — that section is the to-do list, kept in
+sync with measurements.
+
+Converting the 71 dynamic key sites so they can be type-checked was **considered and
+declined** — it is recorded under §4 as an accepted limit, not pending work.
 
 ---
 
@@ -89,11 +104,11 @@ only two remaining `(s)` values are the seconds unit. Keep it that way.
 `{{ t('common.save') }}` · `:label="t('x')"` · `:label="row.name"` (a variable
 contributes no literal) · `:label="'—'"` (no letters).
 
-### Two curated lists
+### One curated list
+
+`TEXT_ATTRS` is **gone** — prop types now decide what counts as a text prop, so there
+is no list to maintain (§8). Declaring a prop `I18nText` is what makes it enforced.
 
-- **`TEXT_ATTRS`** (47 names) — props that carry user-facing text. Feeds both the
-  static and bound rules, so "what counts as a text prop" is defined once. **Add a
-  name here when a component takes UI text through a new prop.**
 - **`NON_TRANSLATABLE`** (39 entries) — tokens that must read identically in every
   language: units (`px`, `ms`, `ns`, `min`), symbols (`×`, `→`, `~`), protocol and
   spec identifiers (`GET`, `UTC`, `SQL`, `PromQL`, the OpenTelemetry statuses
@@ -121,7 +136,7 @@ follows a pattern the library already used for icons (`iconLeft?: IconName`).
 | **`I18nKey`**        | a field holding an **i18n key as data** (`titleKey`, `labelKey`)            |
 | **`useI18nTyped()`** | replaces `useI18n()` in components — `t` returns `I18nText`                 |
 | **`TranslateFn`**    | the type of `t` when a composable/util takes it as a **parameter**          |
-| **`gt()`**           | last resort for genuinely context-free code — **1 file**, see below         |
+| **`gt()`**           | last resort for genuinely context-free code — **4 files**, see below        |
 | **`raw()`**          | the explicit opt-out for text that must not be translated                   |
 
 ```ts
@@ -155,7 +170,7 @@ presets.ts(76,5): error TS2820: Type '"emptyState.noLogs.titel"' is not assignab
   to type 'I18nKey'. Did you mean '"emptyState.noLogs.title"'?
 ```
 
-Cost: **~31 s** for the full app type-check with the 10,665-key union. Not a
+Cost: **~31 s** for the full app type-check with the ~9,800-key union. Not a
 perf concern.
 
 ### Getting a `t`: three cases, in order of preference
@@ -179,7 +194,7 @@ perf concern.
    }
    ```
 
-   This is the standard for non-component code (**27 files**). It is preferred over
+   This is the standard for non-component code (**35 files**). It is preferred over
    `gt()` because these functions are also called directly by specs, outside any
    component, where `useI18n()` would throw. Specs pass the real translator:
 
@@ -192,13 +207,27 @@ perf concern.
    composable translates, `t` belongs on that function, not on the composable.
 
 3. **Genuinely context-free code** → `gt("some.key")`. Only where there is neither a
-   setup context nor a caller to thread `t` through. Exactly **one** file qualifies:
-   `useUnauthorizedErrorGrouper.ts`, reached from an axios 403 interceptor registered
-   once at module load (interceptor → 300 ms `setTimeout` → toast → click handler).
+   setup context nor a caller to thread `t` through. Exactly **four** files qualify
+   (10 call sites), each for a distinct structural reason:
+
+   - `useUnauthorizedErrorGrouper.ts` (6) — reached from an axios 403 interceptor
+     registered once at module load (interceptor → 300 ms `setTimeout` → toast →
+     click handler). The founding case.
+   - `ingestion/setupCard/content/kubernetes.ts` (2) — the setup-card registry pins
+     every builder to `(subs: CardSubstitutions) => RichCardContent`; threading `t`
+     would change that shared contract for ~20 cards. Static prose in cards uses
+     `descriptionKey`/`helpKey` (renderer-resolved); `gt` covers only the one
+     description that interpolates a URL, translated at card-build time.
+   - `utils/query/searchError.ts` (1) — the translated **default** of an optional
+     `fallback: I18nText` parameter, evaluated per call; callers that hold `t`
+     override it (e.g. `parseSearchError(err, gt("search.unknownError"))`).
+   - `usePanelPromQLExecutor.ts` (1) — reached through composable chains that are
+     not guaranteed to run in a setup context, where `useI18n()` would throw.
 
    `gt` is deliberately a narrow escape hatch, not an alternative to case 2. Before
-   reaching for it, check whether a caller can pass `t` instead — 43 of the original
-   49 `gt` sites turned out to be case 2.
+   reaching for it, check whether a caller can pass `t` instead — the overwhelming
+   majority of historical `gt` sites (including the import validators and
+   `useQualityDetailCharts`, converted since) turned out to be case 2.
 
 > **Keys resolved at display time, not module load.** When storing keys as data, keep
 > the `I18nKey` map at module scope but resolve it inside the function that renders.
@@ -250,11 +279,18 @@ annotation is made by hand at the declaration rather than by a pattern match.
 | `t()` keys — literal                             | enforced (ESLint, `.vue` **and** `.ts`)           |
 | i18n keys stored as data                         | enforced (`I18nKey`, 19 files)                    |
 | Toast / notification text                        | enforced (`I18nText`, incl. the store + wrappers) |
-| Text reached via `useI18nTyped()`                | branded across **663** files                      |
+| Text reached via `useI18nTyped()`                | branded across **668** files                      |
 | Composables / utils taking `t: TranslateFn`      | **27** files                                      |
 
-**Numbers:** en-US grew 10,216 → **10,665 keys**; **311** sit under `toastMessages.*`
-(28 module groups). 62 `raw()` opt-outs. **6** `gt()` call sites, all in one file.
+**Numbers** (re-measured 2026-07-31, after the post-review debt cleanup): en-US holds
+**9,837 keys** across **72 namespaces** (10,216 → 11,002 as text was migrated in,
+→ 9,648 after the dead-key sweep §9, → 9,837 as the remaining hardcoded prose —
+import validators, setup-card descriptions, status labels, dialog buttons — was
+keyed); **309** sit under `toastMessages.*`. **977** `raw()` opt-outs — the count
+rose steeply when props became `I18nText`, since every genuinely non-translatable
+value (units, tokens, glyphs, API data) has to say so explicitly, then fell as
+`raw()`-wrapped prose was converted to keys. **10** `gt()` call sites in **4**
+files (see §3 case 3 for the per-file justification).
 
 **Where the two checkers divide.** `no-missing-keys` validates `t('x.y')` **calls**;
 it cannot see a key assigned to a field. Keys stored as **data** are validated instead
@@ -264,27 +300,54 @@ be green to claim key coverage; neither alone is sufficient.
 
 ### Not covered
 
-- **Dynamic keys** — ``t(`about.feature_${id}`)`` (**312** sites). No lint or type
-  check can resolve these. The `t` key parameter is deliberately permissive
-  (`I18nKey | (string & {})`) so they keep compiling.
+- **Dynamic keys — 71 sites. Considered and DECLINED; do not re-open as pending work.**
+
+  ``t(`about.feature_${id}`)`` and similar. A renamed or deleted key reached this way is
+  caught by nothing: ESLint sees no literal, `t`'s permissive
+  `I18nKey | (string & {})` parameter accepts any string, vue-i18n returns the key
+  instead of throwing, and the production build strips its dev warning. It renders the
+  raw dotted path on screen.
+
+  Nothing is currently broken — every key these sites can request was expanded and
+  verified present. The exposure is a future rename, concentrated in `onlineEvals`
+  (40 of the 71 sites).
+
+  Closing it would mean removing the `(string & {})` arm, which surfaces **117 errors**:
+  ~80 are unrelated consumers declaring their translator loosely as
+  `(key: string) => string` (a contravariance failure, not a key problem), 15 need a
+  variable widened to a literal union, 9 need a `Record<string, I18nKey>` map, and a
+  handful are genuine key/code mismatches. Two-thirds of the work is therefore not about
+  dynamic keys at all, which is why this was judged not worth it as a standalone project.
+
+  If the guarantee is ever wanted, the cheap route is a second strict export
+  (`(key: I18nKey)`) used by new code only — no flag day, no 98-file PR.
+
+  > Count these with a word boundary: `grep 't(\`'` also matches `` fetch(`…`) `` and any
+other call ending in `t`, which inflates the figure roughly fourfold. A further 21
+  > sites use backticks but interpolate nothing — those are ordinary static keys.
+
 - **Interfaces not yet annotated.** `I18nText` guards what it is applied to. That is
   _incomplete_, never _wrong_ — coverage grows one declaration at a time, at the
   definition site, with no central registry to keep in sync.
 - **Other locales.** The 14 non-en locales are generated from en-US and lag behind;
   `localeDir` points at en-US only, on purpose. Never hand-edit them.
-- **Unused keys.** `@intlify/vue-i18n/no-unused-keys` is available but **not
-  enabled**. Measured on this repo it reports 2,421 keys of which only ~1,302 are
-  genuinely dead — 681 are reached via a key-string, 437 via a dynamic prefix. If
-  you enable it, use `warn`, never `enableFix`, and populate `ignores` with the
-  dynamic prefixes; its autofix would delete live translations.
+- **Unused keys.** `@intlify/vue-i18n/no-unused-keys` is available but **not enabled**.
+  It reports 2,421 keys, but it only sees `t('literal')` calls — so keys reached
+  through a variable or a dynamic prefix look unused to it, and its autofix would
+  delete them. A direct measurement (literal tokens + dynamic prefixes + the one
+  concatenation site) put the genuinely dead set at **1,363**, since deleted (§9).
+  If you ever enable the rule, use `warn`, never `enableFix`, and populate `ignores`
+  with the dynamic prefixes. Note that even the direct measurement missed 9 live keys
+  (§9) — no automated sweep sees keys held in JSON data files.
 
 ---
 
 ## 5. Deliberately deferred: `strictTemplates`
 
-Typing a component prop `label: I18nText` only gates `<OButton label="Save" />` if
-Vue's `strictTemplates` is on. It is **not** enabled, which is why `TEXT_ATTRS`
-still exists.
+A **declared** prop typed `label: I18nText` gates `<OButton label="Save" />` with or
+without `strictTemplates` — declared props are always checked. `strictTemplates`
+governs **undeclared** attributes only. This was verified by probe, and it is what
+allowed `TEXT_ATTRS` to be retired (§8) while `strictTemplates` stays off.
 
 Measured twice on this branch:
 
@@ -297,10 +360,9 @@ Of those, ~303 are genuine `TS2322` type mismatches (real latent bugs); the rest
 undeclared pass-through attributes.
 
 **It was left out on purpose.** It is a large _type-safety_ migration, not i18n work,
-and its i18n payoff is only retiring the `TEXT_ATTRS` list — whose real gap was
-measured at **five sites** (all fixed here: `reveal-tooltip`, `hide-tooltip`,
-`unstable-dimension-tooltip`, `date-disabled-tooltip`). Worth doing as its own PR;
-not worth burying this one under 2,655 unrelated errors.
+and it turned out to carry **no i18n payoff at all** — retiring `TEXT_ATTRS` did not
+require it. Worth doing as its own PR for the ~303 genuine `TS2322` mismatches; not
+worth burying this one under 2,655 unrelated errors.
 
 ---
 
@@ -320,8 +382,17 @@ not worth burying this one under 2,655 unrelated errors.
 | A recurring unit/symbol/code token                   | add to `NON_TRANSLATABLE` with a one-line reason                                                      |
 | A whole file that is genuinely code                  | add to the SyntaxGuide exemption block                                                                |
 
-New text-carrying **component prop** → add its name to `TEXT_ATTRS`.
+New text-carrying **component prop** → declare it `I18nText`. In `<script setup>`,
+`defineProps<{ label: I18nText }>()`. In the Options API the double cast is required,
+because `StringConstructor` resolves to an unbranded `string`:
+
+```ts
+title: { type: String as unknown as PropType<I18nText>, default: raw("") }
+```
+
 New **interface field** carrying text or a key → declare it `I18nText` / `I18nKey`.
+New **i18n key stored in a `.json` data file** → add it to the `localeKeys.spec.ts`
+guard; no type or lint rule can see it (§9).
 
 ---
 
@@ -397,10 +468,25 @@ Recorded so they are not re-reported as open:
 - **Verb injection.** 5 messages interpolated an English verb into a sentence
   (`"Failed to {action} the alert"`), which is unlocalisable word order; each is now a
   complete message per case.
-- **`gt` proliferation.** 49 sites / 22 files → **6 sites / 1 file**, by passing
-  `t: TranslateFn` (§3, case 2).
+- **`gt` proliferation.** 49 sites / 22 files → **10 sites / 4 files**, by passing
+  `t: TranslateFn` (§3, case 2). The audit (2026-07-31) converted every site whose
+  caller could supply `t` — the onlineEvals import validators now take `t` via their
+  `ctx`, `useQualityDetailCharts` and `computePrefixAssignment` take it as a
+  parameter. The 10 that remain are each structurally unable to receive a `t`
+  (axios interceptor, fixed registry contract, optional-param default, non-setup
+  composable chain — the per-file list is in §3 case 3) and are **intentional and
+  final**. Two alternatives were considered and rejected: deferring keys to render
+  time, and injecting a translator at the composition root. `gt` stays exported as
+  a deliberate, narrow escape hatch; **do not re-raise this as cleanup.**
 - **The `useI18n` import ban** is clean tree-wide. The only remaining `vue-i18n` import
   outside the exempt paths is `createI18n` in a test fixture, which the ban permits.
+- **`TEXT_ATTRS` is gone.** The 47-name attribute allowlist and the bound-prop half of
+  `local/no-bare-bound-text-props` were deleted; prop types now do that job (§3). A
+  declared `I18nText` prop is checked in templates **without** `strictTemplates` —
+  `strictTemplates` governs _undeclared_ attributes only, so §5 is not a blocker for
+  this. Type checking is also _stricter_ than the old rule: it rejects a plain `string`
+  variable, which the rule allowed. `vue/no-bare-strings-in-template` stays enabled,
+  because a bare text node has no prop to annotate; `NON_TRANSLATABLE` still feeds it.
 - **Fake plurals everywhere else.** The remaining 104 `(s)` messages are gone — pipe
   plurals went 53 → **139**. Breakdown of how they were fixed:
   - 69 already interpolated `{count}`/`{n}`, so the call sites were already correct and
@@ -424,9 +510,50 @@ Recorded so they are not re-reported as open:
 
 ## 9. Outstanding work
 
-Ordered by value. Item 3 is the cheapest; item 1 is the only user-visible one.
+### ~~1. Delete dead keys from `en-US.json`~~ — done
 
-### 1. Non-English plural rules (`pluralRules`)
+`en-US.json` went from **11,002 → 9,648 keys** (75 → 71 namespaces, ~90 KB smaller).
+1,363 keys were deleted and **9 were restored** after audit; see below. (The count
+has since grown again to **9,837 / 72 namespaces** as the post-review debt cleanup
+keyed the remaining hardcoded prose — that growth is live keys, not resurrected
+dead ones.)
+
+**What the scan excluded up front.** Keys with a literal reference in prod code or
+specs, keys reachable through a dynamic prefix (a conservative exclusion — spared
+because a dynamic key _might_ reach them, not because they are proven live), and the
+5 keys reachable through the one concatenation site (`utils/common.ts`, `"message." +
+code`).
+
+**The audit that followed, and why it mattered.** Six independent read-only agents
+re-checked all 1,363 deletions, each told to hunt specifically for what a
+quoted-literal scan structurally cannot see. They found **9 genuine false deletions**
+across two distinct blind spots — both now closed by guards that were verified to
+fail when the key is removed:
+
+| Blind spot                                                        | Keys | Guard added                                                   |
+| ----------------------------------------------------------------- | ---- | ------------------------------------------------------------- |
+| Keys living in a **JSON data file** (`constants/features.json`)   | 5    | `localeKeys.spec.ts` — resolves every key features.json cites |
+| Template literal **assigned to a variable** before reaching `t()` | 4    | `useGreeting.ts` — `const key: I18nKey = …`                   |
+
+The first is the more important lesson: TypeScript widens imported JSON string values
+to `string`, so keys stored in a data file cannot be typed as `I18nKey` and are
+invisible to _every_ static check. Annotating `FeatureAvailability` documents intent
+but **does not enforce** — `FEATURE_REGISTRY` casts with `as`, and the widening defeats
+it regardless. Only the spec guard actually catches this.
+
+**If you delete keys again.** Regenerate the list (the set moves as code changes) and
+scan **`.json`/`.md`/`.yaml` as well as `.ts`/`.vue`** — the original sweep read only
+JS/TS and that is exactly how the features.json keys were lost. Never use
+`no-unused-keys --fix`: it shares the blind spot and would delete the dynamic-reachable
+keys. `no-missing-keys` and `I18nKey` together catch an over-delete of any _literal_
+reference, but neither sees the two cases above.
+
+**Other locales.** The 14 non-English files still carry ~1,317 of the deleted keys.
+This is expected and self-healing: `scripts/translations/README.md` documents the
+pipeline as pruning keys removed from `en-US.json`. Harmless meanwhile — nothing
+requests them and en-US is the fallback.
+
+### 2. Non-English plural rules (`pluralRules`)
 
 The en-US messages are now real pipe plurals, but **branch selection still uses
 vue-i18n's built-in default for every locale** because this repo configures no
@@ -444,26 +571,47 @@ generated and must not be hand-edited, so they keep their `(s)` until regenerate
 This degrades safely: if en-US has a pipe and fr-FR does not, vue-i18n returns the
 French message unchanged rather than mis-selecting a branch.
 
-### 2. The last 6 `gt()` sites
+### 3. Spec type-checking does not exist — anywhere
 
-`useUnauthorizedErrorGrouper.ts`. Blocked on architecture, not effort — see §3 case 3
-for why there is no caller to thread `t` through. Two designs were considered and
-rejected (deferring keys to render time; injecting a translator at the composition
-root). Closing this is what would allow `gt` to be deleted from `types/i18n.ts`
-entirely.
+Two separate problems compound here.
 
-### 3. Spec-inclusive `type-check` does not gate PRs
+**First, the config is broken.** `tsconfig.vitest.json`'s include is
+`src/**/*.spec.{ts,js}` — but TypeScript globs do **not** support brace expansion,
+so the pattern matches **zero files**. `npm run type-check` "passes" in seconds
+because it checks nothing but `vitest.config.ts`/`vite.config.ts` (verify with
+`npx tsc -p tsconfig.vitest.json --listFilesOnly`). Any past claim that the tree
+is "green under both configs" was vacuous. Additionally, the npm script's
+`NODE_OPTIONS=…` prefix is POSIX-only, so `npm run type-check` errors outright on
+Windows shells.
 
-`unit-tests.yml` runs `type-check:app`, which **excludes specs**. The spec-inclusive
-`type-check` runs only in `npm-update.yml`, a scheduled dependency job — so a spec that
-fails to compile can reach `main`. Adding one step to `unit-tests.yml` closes it, and
-the tree is currently green under both, so it would go in clean.
+**Second, even a working spec type-check would not gate PRs.** `unit-tests.yml`
+(the PR gate) runs only `type-check:app`, which excludes specs. Vitest transpiles
+through esbuild, which strips types without checking them, so a type-broken spec
+still runs and can pass while testing the wrong thing. This branch hit exactly
+that — when `useDashboardPanelData()` gained its `t` parameter, **55 specs** kept
+the old signature while `type-check:app` reported 0 errors.
+
+Fix is two steps, in order:
+
+1. Split the include in `tsconfig.vitest.json` into `src/**/*.spec.ts` and
+   `src/**/*.spec.js`, run it, and fix what surfaces — spec-only type errors exist
+   today (e.g. `metricGrouping.spec.ts` assigns non-existent keys to `I18nKey`
+   fixtures), so expect this to start **red**, not clean.
+2. Then add the step next to the existing one in `unit-tests.yml`:
+
+```yaml
+- name: Type-check specs (any error fails)
+  working-directory: web
+  run: npm run type-check
+```
 
 ### 4. Standing limits (accepted, not scheduled)
 
-- **312 dynamic keys** — unresolvable by design (§4).
-- **`no-unused-keys` not enabled** — ~1,302 genuinely dead keys of the 2,421 it
-  reports; its autofix would delete live translations (§4).
+- **71 dynamic keys** — conversion considered and **declined**; see §4 for the full
+  reasoning and what closing it would cost.
+- **`no-unused-keys` not enabled** — it reports 2,421 keys but shares the blind spot
+  that makes ~1,000 of those false positives, and its autofix deletes what it flags.
+  The dead-key cleanup in §9.1 uses a directly-measured list instead (§4).
 - **`strictTemplates` deferred** — 2,655 errors, a type-safety migration rather than
   i18n work (§5).
 - **`eslint.config.js` duplicate rule keys** — `prettier/prettier` appears 3×,

scripts/ai-review/agents/code-quality.md

@@ -43,7 +43,7 @@ they hold the authoritative, ESLint-encoded conventions. Flag changed code that
 - **No `any`, no `!` non-null assertions, no use-site `as` casts** (except `as const` and
   `Array/Object as PropType<T>`) — type at the declaration site instead
 - **Mutating a prop** directly (`vue/no-mutating-props`) — must go through a computed alias / emit
-- New Quasar (`<q-*>`) elements or `Notify` where an O2 component / `toast()` exists (ESLint
-  `vue/no-restricted-html-elements` / `no-restricted-imports` gives the exact replacement)
+- Bare HTML controls or third-party UI primitives where an O2 component / `toast()` exists
+  (ESLint `vue/no-restricted-html-elements` / `no-restricted-imports` gives the exact replacement)
 - Hardcoded px / hex colors / user-facing strings instead of design tokens + i18n
 - New code that is not type-clean or lint-clean (the gates are a hard 0)

scripts/ai-review/shared-rules.md

@@ -21,7 +21,7 @@ This is **OpenObserve**, an open-source observability platform written in **Rust
 - **Frontend**: Vue 3 SPA with Vite, using Vuex for state management. When reviewing changes under
   `web/`, **read the tracked skills `.claude/skills/ui-architect/SKILL.md` (UI house rules) and
   `.claude/skills/eslint-error-handling/SKILL.md` (lint/type-check playbook)** first — they are the
-  authoritative source for frontend conventions (O2 component library over Quasar, no
+  authoritative source for frontend conventions (build from the O2 component library, no
   `any`/`!`/use-site `as`, type-clean + lint-clean, design tokens + i18n over hardcoded strings).
   Flag deviations from them.
 - **Testing**: Rust unit tests with `cargo test`, E2E with Playwright

src/api/common/src/auth/validator.rs

@@ -784,7 +784,8 @@ async fn check_and_create_org(user_id: &str, method: &Method, path: &str) -> Res
         && (path_columns[2].eq("alerts")
             || path_columns[2].eq("folders")
             || path_columns[2].eq("reports")
-            || path_columns[2].eq("synthetics"))
+            || path_columns[2].eq("synthetics")
+            || path_columns[2].eq("incidents"))
     {
         path_columns[1]
     } else {

src/api/http/src/handler/http/mod.rs

@@ -14,6 +14,5 @@
 // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 use openobserve_api_common::auth;
-pub mod models;
 pub mod request;
 pub mod router;

src/api/http/src/handler/http/request/mod.rs

@@ -15,25 +15,3 @@
 
 pub mod mcp;
 pub mod ratelimit;
-
-pub use openobserve_api_ingest::request::{
-    CONTENT_TYPE_JSON, CONTENT_TYPE_PROTO, clusters, logs, metrics, rum,
-};
-#[cfg(feature = "cloud")]
-pub use openobserve_api_management::request::cloud;
-#[cfg(feature = "profiling")]
-pub use openobserve_api_management::request::profiling;
-#[cfg(feature = "enterprise")]
-pub use openobserve_api_management::request::{
-    action_server, domain_management, eval_jobs, license, providers, score_configs, scorers,
-};
-pub use openobserve_api_management::request::{
-    actions, gen_ai, keys, kv, model_pricing, service_accounts, service_streams, short_url,
-    sourcemaps, status, stream, synthetics,
-};
-#[cfg(feature = "enterprise")]
-pub use openobserve_api_management::request::{ai, anomaly_detection, workflows};
-#[cfg(feature = "enterprise")]
-pub use openobserve_api_pipelines::request::re_pattern;
-pub use openobserve_api_pipelines::request::{enrichment_table, functions, pipeline, pipelines};
-pub use openobserve_api_search::search::patterns;

src/api/http/src/handler/http/router/mod.rs

@@ -25,9 +25,16 @@ use axum::{
 };
 use config::get_config;
 use openobserve_api_common::X_O2_ASSISTANT_SESSION_ID;
+use openobserve_api_ingest::request::{clusters, logs, metrics, rum};
+#[cfg(feature = "cloud")]
+use openobserve_api_management::request::cloud;
+#[cfg(feature = "profiling")]
+use openobserve_api_management::request::profiling;
 use openobserve_api_management::request::{
-    alerts, authz, dashboards, folders, organization, users,
+    alerts, authz, dashboards, folders, kv, model_pricing, organization, service_accounts,
+    short_url, slos, sourcemaps, status, stream, users,
 };
+use openobserve_api_pipelines::request::{enrichment_table, functions, pipeline, pipelines};
 use openobserve_api_search::{promql, search, traces};
 use openobserve_core::auth::AuthExtractor;
 use tower_http::{
@@ -46,9 +53,17 @@ use {
         auditor::{AuditMessage, Protocol, ResponseMeta},
         config::get_config as get_o2_config,
     },
+    openobserve_api_management::request::{
+        actions, ai, anomaly_detection, domain_management, eval_jobs, gen_ai, keys, license,
+        providers, score_configs, scorers, service_streams, synthetics, workflows,
+    },
+    openobserve_api_pipelines::request::re_pattern,
+    openobserve_api_search::search::patterns,
 };
 
-use super::request::*;
+use super::request::mcp;
+#[cfg(feature = "enterprise")]
+use super::request::ratelimit;
 use crate::{
     common::meta::{middleware_data::RumExtraData, proxy::PathParamProxyURL},
     handler::http::{
@@ -64,7 +79,7 @@ pub mod decompression;
 pub mod middlewares;
 pub mod openapi;
 
-pub use common::meta::http::ERROR_HEADER;
+use common::meta::http::ERROR_HEADER;
 
 /// Create CORS layer for axum
 pub fn cors_layer() -> CorsLayer {
@@ -633,6 +648,23 @@ pub fn basic_routes() -> Router {
         router = router.route("/docs", get(|| async { Redirect::permanent("/swagger/") }));
     }
 
+    // External alert source webhooks — token-authenticated inside the handler itself
+    // (never via auth_middleware), so these must stay in basic_routes rather than
+    // service_routes. See GHSA-wffq-g8qf-ccmv: do not widen the shared token
+    // classifier in validator.rs to cover this token type.
+    router = router
+        .route(
+            "/api/v2/{org_id}/incidents/events",
+            post(alerts::external_events::ingest_events),
+        )
+        .route(
+            "/api/v2/{org_id}/incidents/events/{token}",
+            post(alerts::external_events::ingest_events_url_token),
+        )
+        .route_layer(DefaultBodyLimit::max(
+            alerts::external_events::MAX_BODY_BYTES,
+        ));
+
     router
 }
 
@@ -829,6 +861,26 @@ pub fn service_routes() -> Router {
         .route("/v2/{org_id}/reports/{report_id}/enable", patch(dashboards::reports::enable_report_v2))
         .route("/v2/{org_id}/reports/{report_id}/trigger", put(dashboards::reports::trigger_report_v2))
 
+        // SLOs. Deliberately NOT enterprise-gated: nothing about SLO
+        // measurement is an enterprise capability, and the handlers already
+        // return 501 when ZO_SLO_ENABLED is false. Literal segments are
+        // registered before the {slo_id} catch-all, per the router's ordering
+        // rule.
+        .route(
+            "/{org_id}/slos",
+            get(slos::list_slos).post(slos::create_slo),
+        )
+        // Before the {slo_id} catch-all, or "move" is parsed as an SLO id.
+        .route("/{org_id}/slos/move", post(slos::move_slos))
+        .route("/{org_id}/slos/{slo_id}/enable", put(slos::enable_slo))
+        .route("/{org_id}/slos/{slo_id}/groups", get(slos::get_slo_groups))
+        .route(
+            "/{org_id}/slos/{slo_id}",
+            get(slos::get_slo)
+                .put(slos::update_slo)
+                .delete(slos::delete_slo),
+        )
+
         // Folders (v2)
         .route("/v2/{org_id}/folders/{folder_type}", get(folders::list_folders).post(folders::create_folder))
         .route("/v2/{org_id}/folders/{folder_type}/{folder_id}", get(folders::get_folder).put(folders::update_folder).delete(folders::delete_folder))
@@ -837,6 +889,8 @@ pub fn service_routes() -> Router {
         // Alerts (v2)
         .route("/v2/{org_id}/alerts", get(alerts::list_alerts).post(alerts::create_alert))
         .route("/v2/{org_id}/alerts/{alert_id}", get(alerts::get_alert).put(alerts::update_alert).delete(alerts::delete_alert))
+        .route("/v2/{org_id}/alerts/{alert_id}/groups", get(alerts::list_alert_groups))
+        .route("/v2/{org_id}/alerts/{alert_id}/groups/transitions", get(alerts::list_alert_group_transitions))
         .route("/v2/{org_id}/alerts/{alert_id}/export", post(alerts::export_alert))
         .route("/v2/{org_id}/alerts/bulk", delete(alerts::delete_alert_bulk))
         .route("/v2/{org_id}/alerts/{alert_id}/enable", patch(alerts::enable_alert))
@@ -846,18 +900,25 @@ pub fn service_routes() -> Router {
         .route("/v2/{org_id}/alerts/{alert_id}/clone", post(alerts::clone_alert))
         .route("/v2/{org_id}/alerts/generate_sql", post(alerts::generate_sql))
         .route("/v2/{org_id}/alerts/move", patch(alerts::move_alerts))
+        .route("/v2/{org_id}/alerts/tags", get(alerts::list_alert_tags))
         .route("/v2/{org_id}/alerts/history", get(alerts::history::get_alert_history))
         .route("/v2/{org_id}/alerts/dedup/summary", get(alerts::dedup_stats::get_dedup_summary))
 
         // Alerts - incidents must be before alerts to avoid route conflicts
         .route("/v2/{org_id}/alerts/incidents", get(alerts::incidents::list_incidents))
         .route("/v2/{org_id}/alerts/incidents/stats", get(alerts::incidents::get_incident_stats))
+        .route("/v2/{org_id}/alerts/incidents/external-alerts/{external_alert_id}/payload", get(alerts::incidents::get_external_alert_payload))
         .route("/v2/{org_id}/alerts/incidents/{incident_id}", get(alerts::incidents::get_incident))
         .route("/v2/{org_id}/alerts/incidents/{incident_id}/rca", post(alerts::incidents::trigger_incident_rca).delete(alerts::incidents::cancel_incident_rca))
         .route("/v2/{org_id}/alerts/incidents/{incident_id}/rca/history", get(alerts::incidents::get_incident_rca_history))
         .route("/v2/{org_id}/alerts/incidents/{incident_id}/update", patch(alerts::incidents::update_incident))
         .route("/v2/{org_id}/alerts/incidents/{incident_id}/events", get(alerts::incidents::get_incident_events))
         .route("/v2/{org_id}/alerts/incidents/{incident_id}/events/comment", post(alerts::incidents::post_incident_comment))
+        .route("/v2/{org_id}/incidents/integrations", get(alerts::incident_integrations::list_integrations).post(alerts::incident_integrations::create_integration))
+        .route("/v2/{org_id}/incidents/integrations/{integration_id}", delete(alerts::incident_integrations::delete_integration))
+        .route("/v2/{org_id}/incidents/integrations/{integration_id}/enable", patch(alerts::incident_integrations::set_integration_enabled))
+        .route("/v2/{org_id}/incidents/integrations/{integration_id}/rotate", post(alerts::incident_integrations::rotate_integration_token))
+        .route("/v2/{org_id}/incidents/integrations/{integration_id}/senders", get(alerts::incident_integrations::list_integration_senders))
 
         // Alert templates
         .route("/{org_id}/alerts/templates", get(alerts::templates::list_templates).post(alerts::templates::save_template))
@@ -958,7 +1019,7 @@ pub fn service_routes() -> Router {
             .route("/{org_id}/settings/gen_ai/agent_registry", delete(gen_ai::clear_agent_registry))
             .route("/{org_id}/gen_ai/agents", get(gen_ai::list_scored_agents));
 
-        if get_o2_config().common.online_evals_enabled {
+        if get_o2_config().llm_eval_config.enabled {
             router = router
                 // LLM Providers (Online Eval Phase 2)
                 .route("/{org_id}/providers", get(providers::list_providers).post(providers::create_provider))

src/api/http/src/handler/http/router/openapi.rs

@@ -16,17 +16,27 @@
 use config::{get_config, meta::stream::StreamType};
 #[cfg(feature = "enterprise")]
 use o2_ratelimit::dataresource::default_rules::OpenapiInfo;
+use openobserve_api_ingest::request::{clusters, logs, metrics, rum};
+use openobserve_api_management::request::{
+    actions, gen_ai, keys, kv, service_accounts, service_streams, short_url, status, stream,
+    synthetics,
+};
+use openobserve_api_pipelines::request::{enrichment_table, functions, pipeline, pipelines};
+use openobserve_api_search::search::patterns;
 use utoipa::{
     Modify, OpenApi,
     openapi::security::{Http, HttpAuthScheme, SecurityScheme},
 };
 
-use crate::{common::meta, handler::http::request};
+use crate::{
+    common::meta,
+    handler::http::request::{mcp, ratelimit},
+};
 
 #[derive(OpenApi)]
 #[openapi(
     paths(
-        request::status::healthz,
+        status::healthz,
         openobserve_api_management::request::users::list,
         openobserve_api_management::request::users::save,
         openobserve_api_management::request::users::update,
@@ -50,23 +60,23 @@ use crate::{common::meta, handler::http::request};
         openobserve_api_management::request::organization::system_settings::set_user_setting,
         openobserve_api_management::request::organization::system_settings::delete_org_setting,
         openobserve_api_management::request::organization::system_settings::delete_user_setting,
-        request::stream::list,
-        request::stream::schema,
-        request::stream::create,
-        request::stream::update_settings,
-        request::stream::delete_fields,
-        request::stream::delete,
-        request::logs::ingest::bulk,
-        request::logs::ingest::multi,
-        request::logs::ingest::json,
-        request::logs::loki::loki_push,
+        stream::list,
+        stream::schema,
+        stream::create,
+        stream::update_settings,
+        stream::delete_fields,
+        stream::delete,
+        logs::ingest::bulk,
+        logs::ingest::multi,
+        logs::ingest::json,
+        logs::loki::loki_push,
         openobserve_api_search::traces::traces_write,
         openobserve_api_search::traces::get_latest_traces,
         openobserve_api_search::traces::session::get_latest_sessions,
         openobserve_api_search::traces::session::get_session_details,
         openobserve_api_search::traces::user::get_latest_users,
         openobserve_api_search::traces::dag::get_trace_dag,
-        request::metrics::ingest::json,
+        metrics::ingest::json,
         openobserve_api_search::promql::remote_write,
         openobserve_api_search::promql::query_get,
         openobserve_api_search::promql::query_range_get,
@@ -75,11 +85,11 @@ use crate::{common::meta, handler::http::request};
         openobserve_api_search::promql::labels_get,
         openobserve_api_search::promql::label_values,
         openobserve_api_search::promql::format_query_get,
-        request::enrichment_table::save_enrichment_table,
-        request::enrichment_table::save_enrichment_table_from_url,
-        request::rum::ingest::log,
-        request::rum::ingest::data,
-        request::rum::ingest::sessionreplay,
+        enrichment_table::save_enrichment_table,
+        enrichment_table::save_enrichment_table_from_url,
+        rum::ingest::log,
+        rum::ingest::data,
+        rum::ingest::sessionreplay,
         openobserve_api_search::search::search,
         openobserve_api_search::search::search_partition,
         openobserve_api_search::search::around_v1,
@@ -103,12 +113,12 @@ use crate::{common::meta, handler::http::request};
         openobserve_api_management::request::folders::deprecated::get_folder,
         openobserve_api_management::request::folders::deprecated::get_folder_by_name,
         openobserve_api_management::request::folders::deprecated::update_folder,
-        request::functions::list_functions,
-        request::functions::update_function,
-        request::functions::save_function,
-        request::functions::delete_function,
-        request::functions::list_pipeline_dependencies,
-        request::functions::test_function,
+        functions::list_functions,
+        functions::update_function,
+        functions::save_function,
+        functions::delete_function,
+        functions::list_pipeline_dependencies,
+        functions::test_function,
         openobserve_api_management::request::dashboards::create_dashboard,
         openobserve_api_management::request::dashboards::update_dashboard,
         openobserve_api_management::request::dashboards::list_dashboards,
@@ -137,6 +147,7 @@ use crate::{common::meta, handler::http::request};
         openobserve_api_management::request::alerts::clone_alert,
         openobserve_api_management::request::alerts::generate_sql,
         openobserve_api_management::request::alerts::move_alerts,
+        openobserve_api_management::request::alerts::list_alert_tags,
         openobserve_api_management::request::alerts::history::get_alert_history,
         openobserve_api_management::request::alerts::incidents::list_incidents,
         openobserve_api_management::request::alerts::incidents::get_incident,
@@ -156,38 +167,38 @@ use crate::{common::meta, handler::http::request};
         openobserve_api_management::request::alerts::destinations::save_destination,
         openobserve_api_management::request::alerts::destinations::update_destination,
         openobserve_api_management::request::alerts::destinations::delete_destination,
-        request::kv::get,
-        request::kv::set,
-        request::kv::delete,
-        request::kv::list,
-        request::clusters::list_clusters,
-        request::short_url::shorten,
-        request::short_url::retrieve,
-        request::ratelimit::list_module_ratelimit,
-        request::ratelimit::list_role_ratelimit,
-        request::ratelimit::update_ratelimit,
-        request::service_accounts::list,
-        request::service_accounts::save,
-        request::service_accounts::update,
-        request::service_accounts::delete,
-        request::mcp::handle_mcp_post,
-        request::mcp::handle_mcp_get,
-        request::mcp::oauth_authorization_server_metadata,
-        request::pipeline::save_pipeline,
-        request::pipeline::list_pipelines,
-        request::pipeline::get_pipeline,
-        request::pipeline::list_streams_with_pipeline,
-        request::pipeline::delete_pipeline,
-        request::pipeline::update_pipeline,
-        request::pipeline::enable_pipeline,
-        request::pipeline::enable_pipeline_bulk,
-        request::pipelines::history::get_pipeline_history,
-        request::pipelines::backfill::create_backfill,
-        request::pipelines::backfill::list_backfills,
-        request::pipelines::backfill::get_backfill,
-        request::pipelines::backfill::enable_backfill,
-        request::pipelines::backfill::update_backfill,
-        request::pipelines::backfill::delete_backfill,
+        kv::get,
+        kv::set,
+        kv::delete,
+        kv::list,
+        clusters::list_clusters,
+        short_url::shorten,
+        short_url::retrieve,
+        ratelimit::list_module_ratelimit,
+        ratelimit::list_role_ratelimit,
+        ratelimit::update_ratelimit,
+        service_accounts::list,
+        service_accounts::save,
+        service_accounts::update,
+        service_accounts::delete,
+        mcp::handle_mcp_post,
+        mcp::handle_mcp_get,
+        mcp::oauth_authorization_server_metadata,
+        pipeline::save_pipeline,
+        pipeline::list_pipelines,
+        pipeline::get_pipeline,
+        pipeline::list_streams_with_pipeline,
+        pipeline::delete_pipeline,
+        pipeline::update_pipeline,
+        pipeline::enable_pipeline,
+        pipeline::enable_pipeline_bulk,
+        pipelines::history::get_pipeline_history,
+        pipelines::backfill::create_backfill,
+        pipelines::backfill::list_backfills,
+        pipelines::backfill::get_backfill,
+        pipelines::backfill::enable_backfill,
+        pipelines::backfill::update_backfill,
+        pipelines::backfill::delete_backfill,
         openobserve_api_management::request::dashboards::reports::create_report,
         openobserve_api_management::request::dashboards::reports::update_report,
         openobserve_api_management::request::dashboards::reports::list_reports,
@@ -204,12 +215,12 @@ use crate::{common::meta, handler::http::request};
         openobserve_api_management::request::dashboards::reports::enable_report_v2,
         openobserve_api_management::request::dashboards::reports::trigger_report_v2,
         openobserve_api_management::request::dashboards::reports::move_reports,
-        request::actions::action::upload_zipped_action,
-        request::actions::action::delete_action,
-        request::actions::action::serve_action_zip,
-        request::actions::action::update_action_details,
-        request::actions::action::list_actions,
-        request::actions::action::get_action_from_id,
+        actions::action::upload_zipped_action,
+        actions::action::delete_action,
+        actions::action::serve_action_zip,
+        actions::action::update_action_details,
+        actions::action::list_actions,
+        actions::action::get_action_from_id,
         openobserve_api_management::request::authz::fga::create_role,
         openobserve_api_management::request::authz::fga::delete_role,
         openobserve_api_management::request::authz::fga::get_roles,
@@ -223,11 +234,11 @@ use crate::{common::meta, handler::http::request};
         openobserve_api_management::request::authz::fga::get_groups,
         openobserve_api_management::request::authz::fga::get_group_details,
         openobserve_api_management::request::authz::fga::delete_group,
-        request::keys::save,
-        request::keys::get,
-        request::keys::list,
-        request::keys::delete,
-        request::keys::update,
+        keys::save,
+        keys::get,
+        keys::list,
+        keys::delete,
+        keys::update,
         openobserve_api_search::search::search_job::submit_job,
         openobserve_api_search::search::search_job::list_status,
         openobserve_api_search::search::search_job::get_status,
@@ -237,17 +248,17 @@ use crate::{common::meta, handler::http::request};
         openobserve_api_search::search::search_job::retry_job,
         openobserve_api_search::search::search_stream::search_http2_stream,
         openobserve_api_search::search::search_stream::values_http2_stream,
-        request::patterns::extract_patterns,
+        patterns::extract_patterns,
         openobserve_core::traces::service_graph::api::get_current_topology,
-        request::service_streams::list_services,
-        request::service_streams::get_dimension_analytics,
-        request::service_streams::correlate_streams,
-        request::service_streams::get_identity_config,
-        request::service_streams::save_identity_config,
-        request::gen_ai::clear_agent_registry,
-        request::gen_ai::get_agent_mapping,
-        request::gen_ai::list_scored_agents,
-        request::gen_ai::save_agent_mapping,
+        service_streams::list_services,
+        service_streams::get_dimension_analytics,
+        service_streams::correlate_streams,
+        service_streams::get_identity_config,
+        service_streams::save_identity_config,
+        gen_ai::clear_agent_registry,
+        gen_ai::get_agent_mapping,
+        gen_ai::list_scored_agents,
+        gen_ai::save_agent_mapping,
         openobserve_api_management::request::alerts::deduplication::get_config,
         openobserve_api_management::request::alerts::deduplication::set_config,
         openobserve_api_management::request::alerts::deduplication::delete_config,
@@ -255,19 +266,27 @@ use crate::{common::meta, handler::http::request};
         openobserve_api_management::request::alerts::deduplication::preview_semantic_groups_diff,
         openobserve_api_management::request::alerts::deduplication::save_semantic_groups,
         openobserve_api_management::request::alerts::dedup_stats::get_dedup_summary,
-        request::synthetics::list_synthetics,
-        request::synthetics::create_synthetic,
-        request::synthetics::get_synthetic,
-        request::synthetics::update_synthetic,
-        request::synthetics::delete_synthetic,
-        request::synthetics::set_synthetic_enabled,
-        request::synthetics::run_synthetic_now,
-        request::synthetics::list_locations,
-        request::synthetics::list_runs,
-        request::synthetics::get_run_detail,
-        request::synthetics::job_resolve,
-        request::synthetics::job_lease,
-        request::synthetics::job_ack,
+        openobserve_api_management::request::slos::list_slos,
+        openobserve_api_management::request::slos::get_slo,
+        openobserve_api_management::request::slos::create_slo,
+        openobserve_api_management::request::slos::update_slo,
+        openobserve_api_management::request::slos::delete_slo,
+        openobserve_api_management::request::slos::enable_slo,
+        openobserve_api_management::request::slos::get_slo_groups,
+        openobserve_api_management::request::slos::move_slos,
+        synthetics::list_synthetics,
+        synthetics::create_synthetic,
+        synthetics::get_synthetic,
+        synthetics::update_synthetic,
+        synthetics::delete_synthetic,
+        synthetics::set_synthetic_enabled,
+        synthetics::run_synthetic_now,
+        synthetics::list_locations,
+        synthetics::list_runs,
+        synthetics::get_run_detail,
+        synthetics::job_resolve,
+        synthetics::job_lease,
+        synthetics::job_ack,
     ),
     components(
         schemas(
@@ -284,8 +303,8 @@ use crate::{common::meta, handler::http::request};
             config::meta::stream::StreamStats,
             config::meta::stream::UpdateStreamSettings,
             config::meta::gen_ai::GenAiAgentMappingConfig,
-            request::gen_ai::GenAiAgentListItem,
-            request::gen_ai::GenAiAgentListResponse,
+            gen_ai::GenAiAgentListItem,
+            gen_ai::GenAiAgentListResponse,
             config::meta::dashboards::Dashboard,
             config::meta::dashboards::v1::AxisItem,
             config::meta::dashboards::v1::Dashboard,
@@ -317,7 +336,7 @@ use crate::{common::meta, handler::http::request};
             // Enrichment Tables
             config::meta::enrichment_table::EnrichmentTableStatus,
             config::meta::enrichment_table::EnrichmentTableUrlJob,
-            crate::handler::http::request::enrichment_table::EnrichmentTableUrlRequest,
+            enrichment_table::EnrichmentTableUrlRequest,
             // Dashboards
             openobserve_api_management::models::dashboards::DashboardRequestBody,
             openobserve_api_management::models::dashboards::DashboardResponseBody,
@@ -336,6 +355,7 @@ use crate::{common::meta, handler::http::request};
             openobserve_api_management::models::alerts::responses::GetAlertResponseBody,
             openobserve_api_management::models::alerts::responses::ListAlertsResponseBody,
             openobserve_api_management::models::alerts::responses::ListAlertsResponseBodyItem,
+            openobserve_api_management::request::alerts::AlertTagCount,
             openobserve_api_management::models::alerts::responses::EnableAlertResponseBody,
             openobserve_api_management::models::alerts::Alert,
             openobserve_api_management::models::alerts::TriggerCondition,
@@ -429,7 +449,7 @@ use crate::{common::meta, handler::http::request};
             meta::organization::RumIngestionToken,
             openobserve_api_management::request::organization::assume_service_account::AssumeServiceAccountRequest,
             openobserve_api_management::request::organization::assume_service_account::AssumeServiceAccountResponse,
-            request::status::HealthzResponse,
+            status::HealthzResponse,
             ingestion_common::BulkResponse,
             ingestion_common::BulkResponseItem,
             ingestion_common::ShardResponse,
@@ -437,7 +457,7 @@ use crate::{common::meta, handler::http::request};
             config::meta::promql::Metadata,
             config::meta::promql::MetricType,
             // Service Streams (enterprise)
-            request::service_streams::CorrelationRequest,
+            service_streams::CorrelationRequest,
             config::meta::service_streams::CorrelationResponse,
             config::meta::service_streams::DimensionAnalytics,
             config::meta::service_streams::DimensionAnalyticsSummary,
@@ -454,8 +474,8 @@ use crate::{common::meta, handler::http::request};
             config::meta::alerts::deduplication::SendStrategy,
             openobserve_api_management::request::alerts::dedup_stats::DedupSummaryResponse,
             // Backfill
-            request::pipelines::backfill::BackfillRequest,
-            request::pipelines::backfill::BackfillResponse,
+            pipelines::backfill::BackfillRequest,
+            pipelines::backfill::BackfillResponse,
             openobserve_core::alerts::backfill::BackfillJobStatus,
             // Synthetics
             config::meta::synthetics::Synthetic,

src/api/ingest/src/request/logs/ingest.rs

@@ -38,8 +38,7 @@ use opentelemetry_proto::tonic::collector::logs::v1::ExportLogsServiceRequest;
 use prost::Message;
 
 use crate::{
-    common::meta::http::HttpResponse as MetaHttpResponse,
-    request::{CONTENT_TYPE_JSON, CONTENT_TYPE_PROTO},
+    common::meta::http::{CONTENT_TYPE_JSON, CONTENT_TYPE_PROTO, HttpResponse as MetaHttpResponse},
     service::{
         ingestion::get_thread_id,
         logs::{self, otlp::handle_request},

src/api/ingest/src/request/logs/loki.rs

@@ -27,8 +27,10 @@ use prost::Message;
 use proto::loki_rpc;
 
 use crate::{
-    common::meta::loki::{LokiError, LokiPushRequest},
-    request::{CONTENT_TYPE_JSON, CONTENT_TYPE_PROTO},
+    common::meta::{
+        http::{CONTENT_TYPE_JSON, CONTENT_TYPE_PROTO},
+        loki::{LokiError, LokiPushRequest},
+    },
     service::{ingestion::get_thread_id, logs},
 };
 

src/api/ingest/src/request/metrics/ingest.rs

@@ -31,8 +31,7 @@ use openobserve_core::auth::UserEmail;
 use openobserve_core::ingestion::check_ingestion_allowed;
 
 use crate::{
-    common::meta::http::HttpResponse as MetaHttpResponse,
-    request::{CONTENT_TYPE_JSON, CONTENT_TYPE_PROTO},
+    common::meta::http::{CONTENT_TYPE_JSON, CONTENT_TYPE_PROTO, HttpResponse as MetaHttpResponse},
     service::metrics,
 };
 

src/api/ingest/src/request/mod.rs

@@ -17,5 +17,3 @@ pub mod clusters;
 pub mod logs;
 pub mod metrics;
 pub mod rum;
-
-pub use common::meta::http::{CONTENT_TYPE_JSON, CONTENT_TYPE_PROTO};

src/api/management/src/models/alerts/mod.rs

@@ -27,6 +27,33 @@ use serde_json::Value as JsonValue;
 use svix_ksuid::Ksuid;
 use utoipa::ToSchema;
 
+/// Deserialize an optional float that has to survive a `#[serde(flatten)]`.
+///
+/// `serde_json` is built workspace-wide with `arbitrary_precision` (root
+/// `Cargo.toml`), which represents a non-integer number as a magic one-key
+/// map rather than visiting `f64`. `CreateAlertRequestBody`/`UpdateAlert...`
+/// flatten the whole `Alert`, and `flatten` makes serde buffer the body
+/// through its `Content` type first — at which point that map can no longer
+/// be visited as an `f64`. A plain `Option<f64>` field therefore rejects
+/// `99.5` with "invalid type: map, expected f64" while quietly accepting
+/// `99`, so a fractional warning threshold is unreachable over the API.
+///
+/// Routing through `serde_json::Number` reads both the buffered map and a
+/// direct number, and still rejects strings and non-numeric values.
+fn de_opt_f64<'de, D>(deserializer: D) -> Result<Option<f64>, D::Error>
+where
+    D: serde::Deserializer<'de>,
+{
+    let parsed = Option::<serde_json::Number>::deserialize(deserializer)?;
+    parsed
+        .map(|n| {
+            n.as_f64().ok_or_else(|| {
+                serde::de::Error::custom(format!("`{n}` is not representable as a number"))
+            })
+        })
+        .transpose()
+}
+
 /// Alert configuration for monitoring streams and triggering notifications.
 ///
 /// An alert watches a stream (logs, metrics, or traces) using SQL or PromQL queries,
@@ -145,6 +172,24 @@ pub struct Alert {
     #[serde(default)]
     #[schema(example = json!(["abcde12345"]))]
     pub workflows: Vec<String>,
+
+    /// Priority 1..=5 (P1 = most urgent), Feature 2 / PT-1.
+    ///
+    /// Mutable configuration; display + propagation only — it does not affect
+    /// when the alert fires. Omitted = unset.
+    ///
+    /// `value_type` is required because the enum serializes as an integer via
+    /// serde `try_from`/`into`; without it the OpenAPI schema would advertise a
+    /// string enum and lie about the payload.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[schema(value_type = Option<u8>, example = 3)]
+    pub priority: Option<config::meta::alerts::priority::AlertPriority>,
+
+    /// Selection tags (PT-6): `prod`, `service:checkout`. Normalized and
+    /// validated on save; omitted when empty.
+    #[serde(default, skip_serializing_if = "Vec::is_empty")]
+    #[schema(example = json!(["prod", "service:checkout"]))]
+    pub tags: Vec<String>,
 }
 
 /// Configuration for when and how an alert should be triggered.
@@ -177,6 +222,23 @@ pub struct TriggerCondition {
     #[schema(example = 100)]
     pub threshold_count: i64,
 
+    /// Optional WARNING threshold, sharing `operator` with `threshold` — one
+    /// operator for both levels, no mixed directions. Omitted = single-level
+    /// alert, i.e. exactly the current behaviour. Must be strictly less severe than `threshold` —
+    /// "less severe" is operator-dependent, so `>` requires a smaller value and
+    /// `<` a larger one; `=`/`!=`/`contains` reject it outright.
+    #[serde(rename = "warning_threshold", skip_serializing_if = "Option::is_none")]
+    #[serde(default)]
+    #[schema(example = 50)]
+    pub warning_threshold_count: Option<i64>,
+
+    /// Whether a Warning-level match sends a notification. Defaults to true —
+    /// opting out is explicit. Set false for "page me only on critical" —
+    /// warnings still update state, history and the UI.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[schema(example = true)]
+    pub notify_on_warning: Option<bool>,
+
     /// How often (in minutes) to run the alert query. Used with frequency_type="minutes".
     #[serde(rename = "frequency")]
     #[serde(default)]
@@ -293,6 +355,28 @@ pub struct QueryCondition {
 
     /// Condition to apply to PromQL results. Required with type="promql".
     pub promql_condition: Option<Condition>,
+    /// Optional WARNING value for the PromQL condition, sharing
+    /// `promql_condition.operator` with critical. Omitted = single-level.
+    // Lenient: reached through CreateAlertRequestBody's `#[serde(flatten)]`,
+    // which buffers via `Value`, where `arbitrary_precision` makes a number a
+    // map. Without this a FRACTIONAL warning is rejected while an integer one
+    // is accepted. Both branches found this independently; `de_opt_f64` is the
+    // local helper and `config::meta::slo::lenient_f64` the shared one used
+    // where this helper is not reachable.
+    #[serde(
+        default,
+        skip_serializing_if = "Option::is_none",
+        deserialize_with = "de_opt_f64"
+    )]
+    #[schema(example = 300.0)]
+    pub promql_warning_value: Option<f64>,
+
+    /// Evaluate and page per SERIES rather than collapsing the query to one
+    /// verdict (M-9). PromQL's counterpart to `aggregation.multi_alert`; the
+    /// group key is the series' full label set, chosen by the expression's own
+    /// `by (…)` clause.
+    #[serde(default, skip_serializing_if = "std::ops::Not::not")]
+    pub promql_multi_alert: bool,
 
     /// Aggregation configuration for "custom" query type.
     pub aggregation: Option<Aggregation>,
@@ -308,13 +392,40 @@ pub struct QueryCondition {
     /// Historical comparison periods for anomaly detection.
     #[serde(default)]
     pub multi_time_range: Option<Vec<CompareHistoricData>>,
+
+    /// SLO condition. Required with type="slo" (Feature 5, D42).
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[schema(value_type = Option<Object>)]
+    pub slo_condition: Option<config::meta::slo::condition::SloCondition>,
 }
 
 #[derive(Clone, Debug, Serialize, Deserialize, ToSchema, PartialEq)]
 pub struct Aggregation {
     pub group_by: Option<Vec<String>>,
     pub function: AggFunction,
+    /// CRITICAL threshold for the aggregate value.
     pub having: Condition,
+    /// Optional WARNING threshold, sharing `having.operator` and
+    /// `having.column` with critical. Omitted = single-level aggregation
+    /// alert. Must be strictly less severe than `having.value` — direction
+    /// depends on the operator.
+    // Same lenient deserialization as `promql_warning_value`, same reason.
+    #[serde(
+        default,
+        skip_serializing_if = "Option::is_none",
+        deserialize_with = "de_opt_f64"
+    )]
+    #[schema(example = 50.0)]
+    pub warning_value: Option<f64>,
+    /// Opt in to per-group evaluation (multi-alerts): each group gets its own
+    /// level, state row and notifications. Omitted = `false` = the alert
+    /// evaluates as a single collapsed result, exactly as before.
+    ///
+    /// Requires a non-empty `group_by`, an orderable `having.operator`, and
+    /// "any group" count thresholds — see `alerts_2.md` M-9/M-10.
+    #[serde(default, skip_serializing_if = "std::ops::Not::not")]
+    #[schema(example = false)]
+    pub multi_alert: bool,
 }
 
 #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ToSchema)]
@@ -352,6 +463,10 @@ pub enum QueryType {
     SQL,
     #[serde(rename = "promql")]
     PromQL,
+    /// Feature 5 (D28). An SLO alert reads precomputed SLO status rather than
+    /// running a query.
+    #[serde(rename = "slo")]
+    Slo,
 }
 
 #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)]
@@ -449,6 +564,8 @@ impl From<(meta_alerts::alert::Alert, Option<Trigger>)> for Alert {
             deduplication: alert.deduplication,
             creates_incident: alert.creates_incident,
             workflows: alert.workflows,
+            priority: alert.priority,
+            tags: alert.tags,
         }
     }
 }
@@ -459,6 +576,8 @@ impl From<meta_alerts::TriggerCondition> for TriggerCondition {
             period_minutes: value.period,
             operator: value.operator.into(),
             threshold_count: value.threshold,
+            warning_threshold_count: value.warning_threshold,
+            notify_on_warning: value.notify_on_warning,
             frequency_minutes: value.frequency / 60,
             cron: value.cron,
             frequency_type: value.frequency_type.into(),
@@ -496,12 +615,15 @@ impl From<meta_alerts::QueryCondition> for QueryCondition {
             sql: value.sql,
             promql: value.promql,
             promql_condition: value.promql_condition.map(|pc| pc.into()),
+            promql_warning_value: value.promql_warning_value,
+            promql_multi_alert: value.promql_multi_alert,
             aggregation: value.aggregation.map(|a| a.into()),
             vrl_function: value.vrl_function,
             search_event_type: value.search_event_type.map(|t| t.into()),
             multi_time_range: value
                 .multi_time_range
                 .map(|cs| cs.into_iter().map(|c| c.into()).collect()),
+            slo_condition: value.slo_condition,
         }
     }
 }
@@ -512,6 +634,8 @@ impl From<meta_alerts::Aggregation> for Aggregation {
             group_by: value.group_by,
             function: value.function.into(),
             having: value.having.into(),
+            warning_value: value.warning_value,
+            multi_alert: value.multi_alert,
         }
     }
 }
@@ -540,6 +664,7 @@ impl From<meta_alerts::QueryType> for QueryType {
             meta_alerts::QueryType::Custom => Self::Custom,
             meta_alerts::QueryType::SQL => Self::SQL,
             meta_alerts::QueryType::PromQL => Self::PromQL,
+            meta_alerts::QueryType::Slo => Self::Slo,
         }
     }
 }
@@ -631,6 +756,8 @@ impl From<Alert> for meta_alerts::alert::Alert {
         alert.deduplication = value.deduplication;
         alert.creates_incident = value.creates_incident;
         alert.workflows = value.workflows;
+        alert.priority = value.priority;
+        alert.tags = value.tags;
 
         alert
     }
@@ -643,6 +770,8 @@ impl From<TriggerCondition> for meta_alerts::TriggerCondition {
             period: value.period_minutes,
             operator: value.operator.into(),
             threshold: value.threshold_count,
+            warning_threshold: value.warning_threshold_count,
+            notify_on_warning: value.notify_on_warning,
             frequency: value.frequency_minutes * 60,
             cron: value.cron,
             frequency_type: value.frequency_type.into(),
@@ -679,12 +808,15 @@ impl From<QueryCondition> for meta_alerts::QueryCondition {
             sql: value.sql,
             promql: value.promql,
             promql_condition: value.promql_condition.map(|pc| pc.into()),
+            promql_warning_value: value.promql_warning_value,
+            promql_multi_alert: value.promql_multi_alert,
             aggregation: value.aggregation.map(|a| a.into()),
             vrl_function: value.vrl_function,
             search_event_type: value.search_event_type.map(|t| t.into()),
             multi_time_range: value
                 .multi_time_range
                 .map(|cs| cs.into_iter().map(|c| c.into()).collect()),
+            slo_condition: value.slo_condition,
         }
     }
 }
@@ -695,6 +827,8 @@ impl From<Aggregation> for meta_alerts::Aggregation {
             group_by: value.group_by,
             function: value.function.into(),
             having: value.having.into(),
+            warning_value: value.warning_value,
+            multi_alert: value.multi_alert,
         }
     }
 }
@@ -723,6 +857,7 @@ impl From<QueryType> for meta_alerts::QueryType {
             QueryType::Custom => Self::Custom,
             QueryType::SQL => Self::SQL,
             QueryType::PromQL => Self::PromQL,
+            QueryType::Slo => Self::Slo,
         }
     }
 }
@@ -1186,6 +1321,8 @@ mod tests {
     #[test]
     fn test_trigger_condition_from_meta_converts_frequency_to_minutes() {
         let meta = meta_alerts::TriggerCondition {
+            warning_threshold: None,
+            notify_on_warning: None,
             period: 15,
             operator: meta_alerts::Operator::GreaterThan,
             threshold: 5,
@@ -1211,6 +1348,8 @@ mod tests {
     #[test]
     fn test_trigger_condition_to_meta_converts_frequency_to_seconds() {
         let tc = TriggerCondition {
+            warning_threshold_count: None,
+            notify_on_warning: None,
             period_minutes: 10,
             operator: Operator::LessThan,
             threshold_count: 3,
@@ -1236,6 +1375,8 @@ mod tests {
     #[test]
     fn test_aggregation_from_meta() {
         let meta = meta_alerts::Aggregation {
+            warning_value: None,
+            multi_alert: false,
             group_by: Some(vec!["service".to_string(), "region".to_string()]),
             function: meta_alerts::AggFunction::Count,
             having: meta_alerts::Condition {
@@ -1257,6 +1398,8 @@ mod tests {
     #[test]
     fn test_aggregation_to_meta() {
         let agg = Aggregation {
+            warning_value: None,
+            multi_alert: false,
             group_by: None,
             function: AggFunction::Avg,
             having: Condition {
@@ -1280,10 +1423,13 @@ mod tests {
             sql: Some("SELECT count(*) FROM logs".to_string()),
             promql: None,
             promql_condition: None,
+            promql_warning_value: None,
+            promql_multi_alert: false,
             aggregation: None,
             vrl_function: None,
             search_event_type: None,
             multi_time_range: None,
+            slo_condition: None,
         };
         let qc = QueryCondition::from(meta);
         assert!(matches!(qc.query_type, QueryType::SQL));
@@ -1292,6 +1438,65 @@ mod tests {
         assert!(qc.aggregation.is_none());
     }
 
+    /// The API model is the only path a client can set this flag through, so
+    /// a conversion that silently dropped it would make the whole feature
+    /// unreachable over HTTP while every layer below it still worked.
+    #[test]
+    fn test_promql_multi_alert_survives_the_round_trip_through_the_api_model() {
+        let qc = QueryCondition {
+            query_type: QueryType::PromQL,
+            conditions: None,
+            sql: None,
+            promql: Some("sum by (pod) (rate(errors[5m]))".to_string()),
+            promql_condition: None,
+            promql_warning_value: None,
+            promql_multi_alert: true,
+            aggregation: None,
+            vrl_function: None,
+            search_event_type: None,
+            multi_time_range: None,
+            slo_condition: None,
+        };
+        let meta = meta_alerts::QueryCondition::from(qc);
+        assert!(meta.promql_multi_alert);
+        assert!(meta.multi_alert_enabled());
+
+        // ...and back out again, for the GET that renders the edit form.
+        let back = QueryCondition::from(meta);
+        assert!(back.promql_multi_alert);
+    }
+
+    #[test]
+    fn test_promql_multi_alert_defaults_off_through_the_api_model() {
+        let meta = meta_alerts::QueryCondition {
+            query_type: meta_alerts::QueryType::PromQL,
+            conditions: None,
+            sql: None,
+            promql: Some("up == 0".to_string()),
+            promql_condition: None,
+            promql_warning_value: None,
+            promql_multi_alert: false,
+            aggregation: None,
+            vrl_function: None,
+            search_event_type: None,
+            multi_time_range: None,
+            slo_condition: None,
+        };
+        assert!(!QueryCondition::from(meta).promql_multi_alert);
+    }
+
+    /// A request body that never mentions the field must parse, and parse as
+    /// off — that is every PromQL alert any existing client sends.
+    #[test]
+    fn test_an_api_payload_without_the_field_parses_as_off() {
+        let body = serde_json::json!({
+            "type": "promql",
+            "promql": "up == 0",
+        });
+        let qc: QueryCondition = serde_json::from_value(body).expect("deserializable");
+        assert!(!qc.promql_multi_alert);
+    }
+
     #[test]
     fn test_query_condition_to_meta_sql() {
         let qc = QueryCondition {
@@ -1300,10 +1505,13 @@ mod tests {
             sql: Some("SELECT count(*) FROM logs".to_string()),
             promql: None,
             promql_condition: None,
+            promql_warning_value: None,
+            promql_multi_alert: false,
             aggregation: None,
             vrl_function: Some("fn".to_string()),
             search_event_type: None,
             multi_time_range: None,
+            slo_condition: None,
         };
         let meta = meta_alerts::QueryCondition::from(qc);
         assert!(matches!(meta.query_type, meta_alerts::QueryType::SQL));

src/api/management/src/models/alerts/requests.rs

@@ -108,7 +108,12 @@ pub struct AnomalyAlertFields {
     pub retrain_interval_days: Option<i32>,
     /// Percentile threshold (50.0–99.9). Default: 97.0
     /// Also accepts the name `threshold` (integer, e.g. 97) for API convenience.
-    #[serde(alias = "threshold")]
+    #[serde(
+        default,
+        alias = "threshold",
+        skip_serializing_if = "Option::is_none",
+        deserialize_with = "config::meta::slo::lenient_f64::deserialize_opt"
+    )]
     pub percentile: Option<f64>,
     pub rcf_num_trees: Option<i32>,
     pub rcf_tree_size: Option<i32>,
@@ -173,6 +178,13 @@ pub struct UpdateAnomalyAlertFields {
     pub detection_window_seconds: Option<i64>,
     pub training_window_days: Option<i32>,
     pub retrain_interval_days: Option<i32>,
+    /// Also accepts the name `threshold`, matching the create body.
+    #[serde(
+        default,
+        alias = "threshold",
+        skip_serializing_if = "Option::is_none",
+        deserialize_with = "config::meta::slo::lenient_f64::deserialize_opt"
+    )]
     pub percentile: Option<f64>,
     pub alert_enabled: Option<bool>,
     pub enabled: Option<bool>,
@@ -235,6 +247,28 @@ pub struct ListAlertsQuery {
     /// Optional alert type filter: `all` (default), `scheduled`, `realtime`,
     /// or `anomaly_detection`.
     pub alert_type: Option<meta_alerts::AlertTypeFilter>,
+
+    /// Optional priority filter (PT-3), OR semantics.
+    ///
+    /// Accepts BOTH shapes, because both are natural to type and to generate:
+    ///   * repeated — `?priority=1&priority=2`
+    ///   * comma-separated — `?priority=1,2`
+    ///
+    /// Each value may be `1` or `P1`. Declared as a `Vec` so repeated keys
+    /// deserialize; absent leaves it empty, which means *no filter*.
+    #[serde(default)]
+    pub priority: Vec<String>,
+
+    /// Optional tag filter (PT-8). Comma-separated, **AND** semantics — an
+    /// alert must carry every listed tag.
+    pub tags: Option<String>,
+
+    /// Optional sort column: `priority` or `name`. Anything else keeps the
+    /// historical ordering.
+    pub sort_by: Option<String>,
+
+    /// Sort direction: `desc` for descending, anything else ascending.
+    pub sort_order: Option<String>,
 }
 
 /// HTTP URL query component that contains parameters for enabling alerts.
@@ -275,8 +309,58 @@ impl ListAlertsQuery {
                 .page_size
                 .map(|page_size| (page_size, self.page_idx.unwrap_or(0))),
             alert_type: self.alert_type.unwrap_or_default(),
+            priority: parse_priority_filter(&self.priority),
+            // Resolved by the service layer, which owns the alert cache the
+            // infra layer cannot reach (PT-8).
+            tag_alert_ids: None,
+            sort_by: match self.sort_by.as_deref() {
+                Some("priority") => Some(meta_alerts::AlertSortField::Priority),
+                Some("name") => Some(meta_alerts::AlertSortField::Name),
+                _ => None,
+            },
+            sort_desc: matches!(self.sort_order.as_deref(), Some("desc")),
         }
     }
+
+    /// The requested tag filter, normalized leniently (PT-8/D22).
+    ///
+    /// Invalid tokens are KEPT: dropping them would collapse `?tags=!!!` into
+    /// an empty filter, and an empty filter matches every alert — a request
+    /// that must return nothing would return everything.
+    pub fn requested_tags(&self) -> Vec<String> {
+        let raw: Vec<String> = self
+            .tags
+            .as_deref()
+            .unwrap_or_default()
+            .split(',')
+            .map(|t| t.to_string())
+            .collect();
+        config::meta::alerts::tags::normalize_filter_tags(&raw)
+    }
+}
+
+/// Parse a priority filter, accepting repeated keys and comma-separated
+/// values, `1` and `P1` alike.
+///
+/// Returns `None` only when NOTHING was requested. When the caller did ask for
+/// priorities but none parsed, this returns `Some(empty)` — which the query
+/// layer treats as "match nothing". Returning `None` there would turn
+/// `?priority=P9` into an unfiltered list and hand back every alert, the
+/// match-all bug the tag filter already guards against.
+fn parse_priority_filter(
+    raw: &[String],
+) -> Option<Vec<config::meta::alerts::priority::AlertPriority>> {
+    // Split each entry on commas so both shapes collapse to one token list.
+    let tokens: Vec<&str> = raw
+        .iter()
+        .flat_map(|v| v.split(','))
+        .map(|t| t.trim())
+        .filter(|t| !t.is_empty())
+        .collect();
+    if tokens.is_empty() {
+        return None; // genuinely no filter
+    }
+    Some(tokens.iter().filter_map(|t| t.parse().ok()).collect())
 }
 
 #[derive(Deserialize, ToSchema)]
@@ -324,6 +408,101 @@ pub fn combine_detection_function(
 mod tests {
     use super::*;
 
+    /// Minimal create body carrying an aggregation warning threshold.
+    fn body_with_warning_value(warning: &str) -> String {
+        format!(
+            r#"{{
+                "name": "agg",
+                "stream_type": "logs",
+                "stream_name": "s",
+                "is_real_time": false,
+                "destinations": ["d"],
+                "query_condition": {{
+                    "type": "custom",
+                    "conditions": [],
+                    "aggregation": {{
+                        "group_by": ["host"],
+                        "function": "avg",
+                        "having": {{"column": "latency", "operator": ">=", "value": 500}},
+                        "warning_value": {warning}
+                    }}
+                }},
+                "trigger_condition": {{
+                    "period": 1, "operator": ">=", "threshold": 1,
+                    "frequency": 1, "frequency_type": "minutes", "silence": 0
+                }}
+            }}"#
+        )
+    }
+
+    fn parse_warning_value(warning: &str) -> Result<Option<f64>, serde_json::Error> {
+        let body: CreateAlertRequestBody = serde_json::from_str(&body_with_warning_value(warning))?;
+        Ok(body
+            .alert
+            .query_condition
+            .aggregation
+            .and_then(|a| a.warning_value))
+    }
+
+    /// The regression: `serde_json`'s `arbitrary_precision` renders a float as
+    /// a magic map, and `CreateAlertRequestBody`'s `#[serde(flatten)]` buffers
+    /// the body before the field is read — so a plain `Option<f64>` used to
+    /// reject every fractional threshold with "invalid type: map, expected
+    /// f64" while accepting the integer next to it. An average-latency
+    /// warning band is fractional by nature, so this made the field
+    /// unreachable over the API.
+    #[test]
+    fn test_fractional_warning_value_survives_the_flatten_buffer() {
+        assert_eq!(parse_warning_value("199.5").unwrap(), Some(199.5));
+        assert_eq!(parse_warning_value("0.25").unwrap(), Some(0.25));
+        assert_eq!(parse_warning_value("-2.5").unwrap(), Some(-2.5));
+    }
+
+    #[test]
+    fn test_integer_and_absent_warning_values_are_unchanged() {
+        assert_eq!(parse_warning_value("200").unwrap(), Some(200.0));
+        assert_eq!(parse_warning_value("null").unwrap(), None);
+    }
+
+    #[test]
+    fn test_non_numeric_warning_value_is_still_rejected() {
+        // Leniency is only about representation, not about accepting junk.
+        assert!(parse_warning_value("\"200\"").is_err());
+        assert!(parse_warning_value("true").is_err());
+    }
+
+    /// Pins the defect the `de_opt_f64` workaround exists for, so the two
+    /// tests above cannot silently become tautologies.
+    ///
+    /// A plain `Option<f64>` behind a `#[serde(flatten)]` still cannot read a
+    /// fractional number while `arbitrary_precision` is on. **If this test
+    /// ever fails, the bug is fixed upstream (or the feature was dropped) and
+    /// `de_opt_f64` can be deleted** — it is not a test of our code so much as
+    /// of the constraint our code works around.
+    #[test]
+    fn test_plain_option_f64_behind_flatten_still_cannot_read_a_float() {
+        #[derive(serde::Deserialize, Debug)]
+        struct Inner {
+            plain: Option<f64>,
+        }
+        #[derive(serde::Deserialize, Debug)]
+        struct Outer {
+            #[serde(flatten)]
+            inner: Inner,
+        }
+
+        // The integer form works, which is exactly why the bug hid so well.
+        let ok: Outer = serde_json::from_str(r#"{"plain": 200}"#).unwrap();
+        assert_eq!(ok.inner.plain, Some(200.0));
+
+        let err = serde_json::from_str::<Outer>(r#"{"plain": 199.5}"#)
+            .expect_err("arbitrary_precision + flatten should still reject a bare f64 float");
+        assert!(
+            err.to_string().contains("invalid type: map"),
+            "unexpected failure mode, workaround may need revisiting: {err}"
+        );
+    }
+
     #[test]
     fn test_combine_none_function_returns_none() {
         assert_eq!(combine_detection_function(None, None), None);
@@ -457,6 +636,10 @@ mod tests {
             page_size: Some(20),
             page_idx: Some(1),
             alert_type: None,
+            priority: vec![],
+            tags: None,
+            sort_by: None,
+            sort_order: None,
         };
         let params = q.into("my_org");
         assert_eq!(params.org_id, "my_org");
@@ -478,6 +661,10 @@ mod tests {
             page_size: None,
             page_idx: None,
             alert_type: None,
+            priority: vec![],
+            tags: None,
+            sort_by: None,
+            sort_order: None,
         };
         let params = q.into("org2");
         assert_eq!(params.org_id, "org2");
@@ -497,6 +684,10 @@ mod tests {
             page_size: Some(5),
             page_idx: None,
             alert_type: None,
+            priority: vec![],
+            tags: None,
+            sort_by: None,
+            sort_order: None,
         };
         let params = q.into("org3");
         assert_eq!(params.page_size_and_idx, Some((5, 0)));
@@ -517,3 +708,124 @@ pub struct GenerateSqlRequestBody {
     /// The conditions field within QueryCondition supports both V1 and V2 formats
     pub query_condition: QueryCondition,
 }
+
+#[cfg(test)]
+mod priority_tag_query_tests {
+    use config::meta::alerts::priority::AlertPriority;
+
+    use super::*;
+
+    fn q(
+        priority: Option<&str>,
+        tags: Option<&str>,
+        sort_by: Option<&str>,
+        order: Option<&str>,
+    ) -> ListAlertsQuery {
+        // `priority` is a Vec so repeated query keys deserialize; a single
+        // comma-separated string is still one entry.
+        let priority = priority.map(|p| vec![p.to_string()]).unwrap_or_default();
+        ListAlertsQuery {
+            folder: None,
+            alert_name_substring: None,
+            stream_type: None,
+            stream_name: None,
+            enabled: None,
+            owner: None,
+            page_size: None,
+            page_idx: None,
+            alert_type: None,
+            priority,
+            tags: tags.map(|s| s.to_string()),
+            sort_by: sort_by.map(|s| s.to_string()),
+            sort_order: order.map(|s| s.to_string()),
+        }
+    }
+
+    /// Query strings are typed by humans, so both `1` and `P1` parse (PT-3).
+    #[test]
+    fn test_priority_filter_accepts_both_forms_and_ors_them() {
+        let p = q(Some("1,P2, p3"), None, None, None).into("org");
+        assert_eq!(
+            p.priority,
+            Some(vec![
+                AlertPriority::P1,
+                AlertPriority::P2,
+                AlertPriority::P3
+            ])
+        );
+    }
+
+    /// A bad filter value narrows nothing rather than failing the page — a
+    /// filter is a view, not a write.
+    #[test]
+    fn test_unparseable_priority_values_are_skipped_not_fatal() {
+        let p = q(Some("banana,2,P9"), None, None, None).into("org");
+        assert_eq!(p.priority, Some(vec![AlertPriority::P2]));
+    }
+
+    #[test]
+    fn test_absent_priority_means_no_filter() {
+        // None, NOT Some(empty): nothing was requested, so nothing is filtered.
+        assert_eq!(q(None, None, None, None).into("org").priority, None);
+    }
+
+    /// REGRESSION GUARD: an all-invalid priority filter must narrow to NOTHING.
+    ///
+    /// If parsing returned `None` here, the query layer would read it as "no
+    /// filter" and `?priority=P9` would hand back every alert — the same
+    /// match-all bug the tag filter guards against.
+    #[test]
+    fn test_all_invalid_priority_filter_matches_nothing_not_everything() {
+        let p = q(Some("P9,banana"), None, None, None).into("org");
+        assert_eq!(
+            p.priority,
+            Some(vec![]),
+            "must be an empty filter set, never None"
+        );
+    }
+
+    /// PT-3 requires the repeated form as well as the comma-separated one.
+    #[test]
+    fn test_repeated_priority_keys_are_supported() {
+        let mut query = q(None, None, None, None);
+        query.priority = vec!["1".to_string(), "P2".to_string()];
+        assert_eq!(
+            query.into("org").priority,
+            Some(vec![AlertPriority::P1, AlertPriority::P2])
+        );
+    }
+
+    #[test]
+    fn test_sort_by_and_order_parse() {
+        let p = q(None, None, Some("priority"), Some("desc")).into("org");
+        assert_eq!(p.sort_by, Some(meta_alerts::AlertSortField::Priority));
+        assert!(p.sort_desc);
+
+        let n = q(None, None, Some("name"), None).into("org");
+        assert_eq!(n.sort_by, Some(meta_alerts::AlertSortField::Name));
+        assert!(!n.sort_desc);
+
+        // Unknown column keeps the historical ordering rather than erroring.
+        let u = q(None, None, Some("nonsense"), None).into("org");
+        assert_eq!(u.sort_by, None);
+    }
+
+    /// REGRESSION GUARD: an all-invalid tag filter must NOT collapse to empty,
+    /// because an empty filter matches every alert.
+    #[test]
+    fn test_invalid_tags_are_retained_so_the_filter_still_narrows() {
+        let tags = q(None, Some("!!!"), None, None).requested_tags();
+        assert_eq!(tags, vec!["!!!"], "must not collapse to a match-all filter");
+    }
+
+    #[test]
+    fn test_tag_filter_is_case_normalized_and_drops_separator_blanks() {
+        let tags = q(None, Some("Prod,,SERVICE:Checkout"), None, None).requested_tags();
+        assert_eq!(tags, vec!["prod", "service:checkout"]);
+    }
+
+    #[test]
+    fn test_absent_tags_means_no_filter() {
+        assert!(q(None, None, None, None).requested_tags().is_empty());
+    }
+}

src/api/management/src/models/alerts/responses.rs

@@ -70,6 +70,158 @@ pub struct ListAlertsResponseBodyItem {
     /// Last error message from training or detection. Only present for `anomaly_detection` items.
     #[serde(skip_serializing_if = "Option::is_none")]
     pub last_error: Option<String>,
+    /// Outcome of the most recent evaluation: "firing" | "normal" | "error" |
+    /// "notify_failed" | "succeeded". Absent when the alert has never run.
+    ///
+    /// This is the LAST RUN OUTCOME, not a live "currently firing" flag —
+    /// always render it alongside `last_outcome_at`. See Part IV of alerts.md.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub last_outcome: Option<String>,
+    /// When `last_outcome` was recorded (microseconds).
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub last_outcome_at: Option<i64>,
+    /// When `last_outcome` last CHANGED (microseconds) — i.e. how long the
+    /// alert has been in its current state.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub last_outcome_since: Option<i64>,
+    /// Severity of the last classification: "ok" | "warning" | "critical".
+    ///
+    /// A SEPARATE axis from `last_outcome`: that says whether the evaluation
+    /// fired, this says how bad. An alert can be `firing` at `warning`.
+    /// Absent for single-level alerts that never classified.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub level: Option<String>,
+    /// When `level` last CHANGED — powers "critical for 20 minutes".
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub level_since: Option<i64>,
+    /// Configured priority as the integer storage id 1..=5 (Feature 2, PT-3).
+    ///
+    /// A THIRD axis: `enabled` is is-it-running, `last_outcome` is
+    /// did-it-fire, `level` is how-bad-now, and this is how much humans care.
+    /// Absent when unset, which is every pre-Feature-2 alert.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    #[schema(value_type = Option<u8>, example = 3)]
+    pub priority: Option<u8>,
+    /// Normalized selection tags (PT-6). Omitted when empty.
+    #[serde(skip_serializing_if = "Vec::is_empty")]
+    pub tags: Vec<String>,
+    /// Multi-alerts only (§5.4): how many groups the last evaluation observed,
+    /// counted **before** the M-6 cap truncated them. Absent for every alert
+    /// that has not opted in.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub groups_observed: Option<i32>,
+    /// Multi-alerts only: how many of those groups were warning-or-worse,
+    /// also counted pre-cap. With `groups_observed` this is the "N of M groups
+    /// firing" chip. Counting retained state rows instead would silently
+    /// under-report whenever more than the cap's worth of groups fire, which
+    /// is the silent truncation M-6 forbids.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub groups_firing: Option<i32>,
+    /// Whether `groups_observed` is a `>=` lower bound — the bounded fetch
+    /// page came back full, so groups beyond it were never seen. Render the
+    /// count with a `≥`.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub groups_observed_is_lower_bound: Option<bool>,
+    /// Whether `groups_firing` is a `>=` lower bound. Tracked separately
+    /// because the two diverge: a full page that still reached healthy groups
+    /// has seen every firing group, so this stays exact while
+    /// `groups_observed` does not.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub groups_firing_is_lower_bound: Option<bool>,
+}
+
+/// One tracked group of a multi-alert (§5.4's group table).
+#[derive(Clone, Debug, Serialize, ToSchema)]
+pub struct AlertGroupResponseItem {
+    /// Deterministic hash of the group's label set — the state identity, and
+    /// the key the per-group history filter takes.
+    pub group_key: String,
+    /// Rendered `k=v,k=v` labels, for display only. `group_key` stays the
+    /// identity because a readable rendering is ambiguous once a label value
+    /// contains the separators.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub group_labels: Option<String>,
+    /// Parsed form of `group_labels`, so the UI does not re-parse the string.
+    pub labels: Vec<AlertGroupLabel>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub level: Option<String>,
+    /// When `level` last changed — "critical for 20 minutes".
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub level_since: Option<i64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub last_outcome: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub last_outcome_at: Option<i64>,
+    /// Last evaluation that actually included this group (M-7). A group whose
+    /// `last_seen` is falling behind is on its way to being resolved.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub last_seen: Option<i64>,
+    /// Per-group silence window (MN-2): suppress re-delivery until this
+    /// instant. Absent when the group is not silenced.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub silenced_until: Option<i64>,
+    /// Level of this group's last *successful* delivery — what escalation is
+    /// measured against. Absent when it has never paged.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub last_notified_level: Option<String>,
+}
+
+/// One `key=value` pair of a group's label set.
+#[derive(Clone, Debug, Serialize, ToSchema)]
+pub struct AlertGroupLabel {
+    pub name: String,
+    pub value: String,
+}
+
+/// HTTP response body for `ListAlertGroups`.
+#[derive(Clone, Debug, Serialize, ToSchema)]
+pub struct ListAlertGroupsResponseBody {
+    /// Tracked groups, most severe first.
+    pub list: Vec<AlertGroupResponseItem>,
+    /// Pre-cap totals from the rollup row, so the caller can render
+    /// "N of M groups firing" without counting the (post-cap) list above.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub groups_observed: Option<i32>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub groups_firing: Option<i32>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub groups_observed_is_lower_bound: Option<bool>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub groups_firing_is_lower_bound: Option<bool>,
+    /// True when the last evaluation observed more groups than the M-6 cap
+    /// tracks, so `list` is a truncated view. Drives the cap banner — M-6
+    /// forbids truncating silently.
+    pub capped: bool,
+    /// The cap in force, for the banner's wording.
+    pub group_cap: usize,
+}
+
+/// One per-group state transition (M-8's history source).
+#[derive(Clone, Debug, Serialize, ToSchema)]
+pub struct AlertGroupTransitionItem {
+    pub group_key: String,
+    /// Carried on the transition itself, so history outlives the state row the
+    /// reaper deletes.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub group_labels: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub from_level: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub to_level: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub from_outcome: Option<String>,
+    pub to_outcome: String,
+    pub at: i64,
+    /// Observed value at transition time. `None` where nothing was observed —
+    /// a group that vanished has no reading, and rendering 0 would be a lie.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub value: Option<f64>,
+}
+
+/// HTTP response body for `ListAlertGroupTransitions`.
+#[derive(Clone, Debug, Serialize, ToSchema)]
+pub struct ListAlertGroupTransitionsResponseBody {
+    pub list: Vec<AlertGroupTransitionItem>,
 }
 
 /// HTTP response body for `EnableAlert` endpoint.
@@ -137,6 +289,19 @@ impl TryFrom<(meta_folders::Folder, meta_alerts::Alert, Option<Trigger>)>
             last_trained_at: None,
             status: None,
             last_error: None,
+            last_outcome: None,
+            last_outcome_at: None,
+            last_outcome_since: None,
+            level: None,
+            level_since: None,
+            priority: alert.priority.map(|p| p.to_i32() as u8),
+            tags: alert.tags,
+            // Filled from the rollup state row by `enrich_with_run_state`,
+            // alongside the other run-state fields above.
+            groups_observed: None,
+            groups_firing: None,
+            groups_observed_is_lower_bound: None,
+            groups_firing_is_lower_bound: None,
         })
     }
 }
@@ -206,11 +371,35 @@ pub fn anomaly_config_to_list_item(v: &serde_json::Value) -> Option<ListAlertsRe
         is_real_time: false,
         last_trained_at: v.get("training_completed_at").and_then(|t| t.as_i64()),
         status,
+        // Anomaly configs do not flow through the alert scheduler's state
+        // write path; leave run state unset rather than implying "never fired".
+        last_outcome: None,
+        last_outcome_at: None,
+        last_outcome_since: None,
+        level: None,
+        level_since: None,
         last_error: v
             .get("last_error")
             .and_then(|e| e.as_str())
             .filter(|s| !s.is_empty())
             .map(String::from),
+        // Feature 2: anomaly configs now carry the same triage metadata as
+        // alerts, so they filter and sort alongside them instead of being
+        // excluded wholesale.
+        priority: v
+            .get("priority")
+            .and_then(|p| p.as_u64())
+            .and_then(|p| u8::try_from(p).ok())
+            .filter(|p| (1..=5).contains(p)),
+        tags: v
+            .get("tags")
+            .and_then(|t| serde_json::from_value::<Vec<String>>(t.clone()).ok())
+            .unwrap_or_default(),
+        // Anomaly configs have no grouping, so there is nothing to count.
+        groups_observed: None,
+        groups_firing: None,
+        groups_observed_is_lower_bound: None,
+        groups_firing_is_lower_bound: None,
     })
 }
 
@@ -284,6 +473,17 @@ mod tests {
             last_trained_at: None,
             status: None,
             last_error: None,
+            last_outcome: None,
+            last_outcome_at: None,
+            last_outcome_since: None,
+            level: None,
+            level_since: None,
+            priority: None,
+            tags: vec![],
+            groups_observed: None,
+            groups_firing: None,
+            groups_observed_is_lower_bound: None,
+            groups_firing_is_lower_bound: None,
         };
         let json = serde_json::to_value(&item).unwrap();
         let obj = json.as_object().unwrap();
@@ -292,6 +492,13 @@ mod tests {
         assert!(!obj.contains_key("last_trained_at"));
         assert!(!obj.contains_key("status"));
         assert!(!obj.contains_key("last_error"));
+        // §5.4: a non-multi alert must not advertise a group summary at all —
+        // an absent field reads as "not a multi-alert", a zero would read as
+        // "observed no groups".
+        assert!(!obj.contains_key("groups_observed"));
+        assert!(!obj.contains_key("groups_firing"));
+        assert!(!obj.contains_key("groups_observed_is_lower_bound"));
+        assert!(!obj.contains_key("groups_firing_is_lower_bound"));
     }
 
     #[test]
@@ -467,3 +674,78 @@ pub struct GenerateSqlMetadata {
     /// Whether GROUP BY is present
     pub has_group_by: bool,
 }
+#[cfg(test)]
+mod anomaly_priority_tag_tests {
+    use super::*;
+
+    fn cfg(extra: serde_json::Value) -> serde_json::Value {
+        let mut base = serde_json::json!({
+            "anomaly_id": <Ksuid as svix_ksuid::KsuidLike>::new(None, None).to_string(),
+            "name": "anom",
+            "folder_id": "default",
+            "stream_name": "s",
+            "stream_type": "logs",
+            "enabled": true,
+        });
+        if let (Some(b), Some(e)) = (base.as_object_mut(), extra.as_object()) {
+            for (k, v) in e {
+                b.insert(k.clone(), v.clone());
+            }
+        }
+        base
+    }
+
+    /// Feature 2: anomaly configs surface priority/tags on the list, so they
+    /// render and filter alongside alerts instead of always showing "—".
+    #[test]
+    fn test_anomaly_list_item_carries_priority_and_tags() {
+        let item = anomaly_config_to_list_item(&cfg(serde_json::json!({
+            "priority": 2,
+            "tags": ["prod", "service:checkout"],
+        })))
+        .expect("should map");
+        assert_eq!(item.priority, Some(2));
+        assert_eq!(item.tags, vec!["prod", "service:checkout"]);
+    }
+
+    /// Pre-Feature-2 configs have neither key; they must map to unset rather
+    /// than failing the whole list.
+    #[test]
+    fn test_anomaly_list_item_without_the_fields_is_unset() {
+        let item = anomaly_config_to_list_item(&cfg(serde_json::json!({}))).expect("should map");
+        assert_eq!(item.priority, None);
+        assert!(item.tags.is_empty());
+    }
+
+    /// A corrupt row must degrade to unset, never take the alert list down:
+    /// an out-of-range id is not a valid priority, and a non-array tags blob
+    /// is not a tag list.
+    #[test]
+    fn test_anomaly_list_item_degrades_on_corrupt_values() {
+        for bad in [
+            serde_json::json!(0),
+            serde_json::json!(6),
+            serde_json::json!(99),
+        ] {
+            let item =
+                anomaly_config_to_list_item(&cfg(serde_json::json!({ "priority": bad }))).unwrap();
+            assert_eq!(item.priority, None, "id {bad} must not decode");
+        }
+        let item = anomaly_config_to_list_item(&cfg(serde_json::json!({
+            "tags": {"not": "an array"}
+        })))
+        .unwrap();
+        assert!(item.tags.is_empty());
+    }
+
+    /// The list response omits both when unset, so a pre-Feature-2 anomaly
+    /// config serializes exactly as it did before.
+    #[test]
+    fn test_unset_fields_are_omitted_from_the_response() {
+        let item = anomaly_config_to_list_item(&cfg(serde_json::json!({}))).unwrap();
+        let json = serde_json::to_value(&item).unwrap();
+        let obj = json.as_object().unwrap();
+        assert!(!obj.contains_key("priority"));
+        assert!(!obj.contains_key("tags"));
+    }
+}

src/api/management/src/models/eval_jobs/mod.rs

@@ -518,7 +518,10 @@ mod tests {
         let job = infra::table::online_eval_jobs::OnlineEvalJob::try_from(body).unwrap();
 
         assert_eq!(job.target_scope, TargetScope::Trace);
-        assert_eq!(job.trace_config.unwrap().idle_window_secs, 120);
+        assert_eq!(
+            job.trace_config.unwrap().idle_window_secs,
+            infra::table::online_eval_jobs::DEFAULT_TRACE_IDLE_WINDOW_SECS
+        );
         assert!(job.session_config.is_none());
     }
 
@@ -539,7 +542,10 @@ mod tests {
         let job = infra::table::online_eval_jobs::OnlineEvalJob::try_from(body).unwrap();
 
         assert_eq!(job.target_scope, TargetScope::Session);
-        assert_eq!(job.session_config.unwrap().idle_window_secs, 120);
+        assert_eq!(
+            job.session_config.unwrap().idle_window_secs,
+            infra::table::online_eval_jobs::DEFAULT_SESSION_IDLE_WINDOW_SECS
+        );
         assert!(job.trace_config.is_none());
     }
 
@@ -712,13 +718,13 @@ mod tests {
     }
 
     #[test]
-    fn test_eval_job_request_body_rejects_idle_window_below_scheduler_poll_interval() {
+    fn test_eval_job_request_body_rejects_a_non_positive_idle_window() {
         let json = r#"{
             "name": "j",
             "stream": "traces",
             "streamType": "traces",
             "targetScope": "trace",
-            "traceConfig": { "idleWindowSecs": 44, "maxAgeSecs": 1800 },
+            "traceConfig": { "idleWindowSecs": 0, "maxAgeSecs": 1800 },
             "filterCondition": {},
             "scorers": ["scorer-1"],
             "samplingMode": "all",
@@ -728,7 +734,7 @@ mod tests {
         let body: EvalJobRequestBody = serde_json::from_str(json).unwrap();
         let err = infra::table::online_eval_jobs::OnlineEvalJob::try_from(body).unwrap_err();
 
-        assert_eq!(err, "Completion idle window must be at least 45 seconds");
+        assert_eq!(err, "Completion idle window must be at least 1 second");
     }
 
     #[test]

src/api/management/src/request/alerts/external_events.rs

@@ -0,0 +1,252 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Token-authenticated inbound webhook endpoint for external alert sources
+//! (Grafana, Alertmanager, generic webhooks). Deliberately registered in
+//! `basic_routes()` (not the authenticated `service_routes` scope) — token
+//! validation happens only inside this handler, never in
+//! `src/api/common/src/auth/validator.rs`. See GHSA-wffq-g8qf-ccmv: widening
+//! the shared token classifier there caused an auth bypass in the past; this
+//! new `o2iat_` token type must not repeat that mistake.
+
+use axum::{Json, extract::Path, http::HeaderMap, response::Response};
+
+use crate::common::meta::http::HttpResponse as MetaHttpResponse;
+
+/// Maximum number of individual alerts accepted in a single webhook request.
+pub const MAX_ALERTS_PER_REQUEST: usize = 200;
+
+/// Maximum request body size accepted for these routes (bytes).
+pub const MAX_BODY_BYTES: usize = 1_048_576;
+
+/// Extracts the integration token from either the `Authorization: Bearer` header
+/// or a path-embedded token, preferring the header when both are present.
+#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
+pub(crate) fn extract_token(auth_header: Option<&str>, path_token: Option<&str>) -> Option<String> {
+    if let Some(h) = auth_header
+        && let Some(t) = h.strip_prefix("Bearer ")
+        && t.starts_with(infra::table::incident_integrations::INCIDENT_INTEGRATION_TOKEN_PREFIX)
+    {
+        return Some(t.to_string());
+    }
+    path_token
+        .filter(|t| {
+            t.starts_with(infra::table::incident_integrations::INCIDENT_INTEGRATION_TOKEN_PREFIX)
+        })
+        .map(|t| t.to_string())
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn ingest_events(
+    Path(org_id): Path<String>,
+    headers: HeaderMap,
+    Json(body): Json<serde_json::Value>,
+) -> Response {
+    let auth = headers.get("authorization").and_then(|v| v.to_str().ok());
+    handle_events(org_id, extract_token(auth, None), headers, body).await
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn ingest_events_url_token(
+    Path((org_id, token)): Path<(String, String)>,
+    headers: HeaderMap,
+    Json(body): Json<serde_json::Value>,
+) -> Response {
+    handle_events(org_id, extract_token(None, Some(&token)), headers, body).await
+}
+
+#[cfg(feature = "enterprise")]
+async fn handle_events(
+    org_id: String,
+    token: Option<String>,
+    headers: HeaderMap,
+    body: serde_json::Value,
+) -> Response {
+    use o2_enterprise::enterprise::common::config::get_config as o2_config;
+    if !o2_config().incidents.enabled {
+        return MetaHttpResponse::forbidden("External alert sources not enabled");
+    }
+    let Some(token) = token else {
+        return MetaHttpResponse::not_found("not found"); // never confirm token semantics
+    };
+    let integration = match infra::table::incident_integrations::find_by_token(&token).await {
+        Ok(Some(i)) if i.org_id == org_id => i,
+        Ok(_) => return MetaHttpResponse::not_found("not found"),
+        Err(e) => return MetaHttpResponse::internal_error(e),
+    };
+
+    let now = chrono::Utc::now().timestamp_micros();
+    let ua = headers.get("user-agent").and_then(|v| v.to_str().ok());
+    let detected = if integration.source_type == "auto" {
+        openobserve_core::alerts::external_alerts::detect_source(ua, &body)
+    } else {
+        // explicit source_type pins the parser
+        match integration.source_type.as_str() {
+            "grafana" => openobserve_core::alerts::external_alerts::DetectedSource::Grafana,
+            "alertmanager" => {
+                openobserve_core::alerts::external_alerts::DetectedSource::Alertmanager
+            }
+            _ => openobserve_core::alerts::external_alerts::DetectedSource::Generic,
+        }
+    };
+
+    let events = match openobserve_core::alerts::external_alerts::normalize(detected, &body, now) {
+        Ok(evs) => evs,
+        Err(reason) => {
+            let _ = infra::table::incident_integrations::touch_sender(
+                infra::table::incident_integrations::TouchSenderParams {
+                    integration_id: &integration.id,
+                    detected_source: detected.as_str(),
+                    sender_label: None,
+                    now,
+                    accepted: 0,
+                    rejected: 1,
+                    saw_resolved: false,
+                },
+            )
+            .await;
+            return MetaHttpResponse::bad_request(reason);
+        }
+    };
+    if events.len() > MAX_ALERTS_PER_REQUEST {
+        return MetaHttpResponse::bad_request(format!(
+            "too many alerts in one request: {} > {MAX_ALERTS_PER_REQUEST}",
+            events.len()
+        )); // 400; sources should configure max_alerts/grouping
+    }
+
+    let base_destinations: Vec<String> = integration.config["destinations"]
+        .as_array()
+        .map(|a| {
+            a.iter()
+                .filter_map(|v| v.as_str().map(String::from))
+                .collect()
+        })
+        .unwrap_or_default();
+
+    let (mut accepted, mut rejected, mut saw_resolved) = (0u32, 0u32, false);
+    let mut errors: Vec<serde_json::Value> = Vec::new();
+    for (i, ev) in events.iter().enumerate() {
+        if ev.status == config::meta::alerts::incidents::ExternalAlertStatus::Resolved {
+            saw_resolved = true;
+        }
+        match infra::table::external_alerts::upsert_event(
+            &org_id,
+            &integration.id,
+            detected.as_str(),
+            ev,
+        )
+        .await
+        {
+            Ok((record, outcome)) => {
+                accepted += 1;
+                use infra::table::external_alerts::UpsertOutcome::*;
+                if matches!(outcome, Inserted | Refreshed | Reopened)
+                    && let Err(e) = openobserve_core::alerts::incidents::correlate_external_event(
+                        &org_id,
+                        &record,
+                        base_destinations.clone(),
+                    )
+                    .await
+                {
+                    log::warn!(
+                        "[external_alerts] correlation failed for {}: {e}",
+                        record.id
+                    );
+                } else if matches!(outcome, ResolvedApplied)
+                    && let Err(e) =
+                        openobserve_core::alerts::incidents::try_auto_resolve_incident_for_external_alert(
+                            &org_id, &record.id,
+                        )
+                        .await
+                {
+                    log::warn!(
+                        "[external_alerts] auto-resolve check failed for {}: {e}",
+                        record.id
+                    );
+                }
+            }
+            Err(e) => {
+                rejected += 1;
+                errors.push(serde_json::json!({"index": i, "reason": e.to_string()}));
+            }
+        }
+    }
+    let sender_label = openobserve_core::alerts::external_alerts::derive_sender_label(&events);
+    let _ = infra::table::incident_integrations::touch_sender(
+        infra::table::incident_integrations::TouchSenderParams {
+            integration_id: &integration.id,
+            detected_source: detected.as_str(),
+            sender_label: sender_label.as_deref(),
+            now,
+            accepted,
+            rejected,
+            saw_resolved,
+        },
+    )
+    .await;
+
+    Response::builder()
+        .status(axum::http::StatusCode::ACCEPTED)
+        .header(axum::http::header::CONTENT_TYPE, "application/json")
+        .body(
+            serde_json::json!({"accepted": accepted, "rejected": rejected, "errors": errors})
+                .to_string()
+                .into(),
+        )
+        .unwrap()
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn ingest_events(
+    _path: Path<String>,
+    _headers: HeaderMap,
+    _body: Json<serde_json::Value>,
+) -> Response {
+    MetaHttpResponse::forbidden("Not Supported")
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn ingest_events_url_token(
+    _path: Path<(String, String)>,
+    _headers: HeaderMap,
+    _body: Json<serde_json::Value>,
+) -> Response {
+    MetaHttpResponse::forbidden("Not Supported")
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_extract_token_prefers_bearer_then_path() {
+        assert_eq!(
+            extract_token(Some("Bearer o2iat_abc"), None),
+            Some("o2iat_abc".to_string())
+        );
+        assert_eq!(
+            extract_token(None, Some("o2iat_path")),
+            Some("o2iat_path".to_string())
+        );
+        assert_eq!(
+            extract_token(Some("Bearer o2iat_abc"), Some("o2iat_path")),
+            Some("o2iat_abc".to_string())
+        );
+        assert_eq!(extract_token(Some("Basic xyz"), None), None); // wrong scheme
+        assert_eq!(extract_token(Some("Bearer nope_prefix"), None), None); // wrong prefix
+        assert_eq!(extract_token(None, None), None);
+    }
+}

src/api/management/src/request/alerts/history.rs

@@ -22,7 +22,7 @@ use common::utils::sql::escape_like;
 use config::{
     meta::{
         search::{Query as SearchQuery, Request as SearchRequest},
-        self_reporting::usage::TRIGGERS_STREAM,
+        self_reporting::usage::{TRIGGERS_STREAM, TriggerDataType, normalize_outcome},
         stream::StreamType,
     },
     utils::time::now_micros,
@@ -95,6 +95,29 @@ pub struct AlertHistoryEntry {
     /// Number of anomalies found in this evaluation run (anomaly detection only).
     #[serde(skip_serializing_if = "Option::is_none")]
     pub anomaly_count: Option<i32>,
+    // ── Value context (T-9/T-10, alerts_2.md §7.5) ──────────────────────────
+    // Absent on rows written before these fields existed; the UI renders "—".
+    /// Severity of the matched threshold: "ok" | "warning" | "critical".
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub level: Option<String>,
+    /// The value that was compared (row count, or the aggregate).
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub actual_value: Option<f64>,
+    /// The threshold that matched. Absent on `normal` rows — T-10 renders
+    /// those as actual value + Ok, with no threshold.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub threshold_value: Option<f64>,
+    /// Operator, so the row reads standalone ("112 >= 100").
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub threshold_operator: Option<String>,
+    /// Which group/series produced `actual_value` ("host=b,region=eu");
+    /// absent for count alerts and pre-change rows.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub group_label: Option<String>,
+    /// True when `actual_value` is a LOWER BOUND (legacy capped count fetch,
+    /// §7.5) — the UI renders "≥ N". Absent = exact.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub value_is_lower_bound: Option<bool>,
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
@@ -333,28 +356,26 @@ pub async fn get_alert_history(
             .hits
             .into_iter()
             .map(|hit| {
+                // Post-cutover rows already carry `firing`/`normal` directly;
+                // legacy rows stored `completed` regardless of the anomaly
+                // count, so the normalizer re-reads `success_response` for
+                // those. See Part III of alerts.md.
                 let raw_status = hit
                     .get("status")
                     .and_then(|v| v.as_str())
                     .unwrap_or("unknown");
-                // Derive result: parse anomalies_found from success_response JSON.
-                let anomaly_count = hit
-                    .get("success_response")
-                    .and_then(|v| v.as_str())
+                let success_response = hit.get("success_response").and_then(|v| v.as_str());
+                let anomaly_count = success_response
                     .and_then(|s| serde_json::from_str::<serde_json::Value>(s).ok())
                     .and_then(|v| v["anomalies_found"].as_i64())
                     .unwrap_or(0);
-                let result = match raw_status {
-                    "failed" => "failed".to_string(),
-                    "skipped" => "skipped".to_string(),
-                    _ => {
-                        if anomaly_count > 0 {
-                            "anomaly".to_string()
-                        } else {
-                            "normal".to_string()
-                        }
-                    }
-                };
+                let result = normalize_outcome(
+                    raw_status,
+                    &TriggerDataType::AnomalyDetection,
+                    success_response,
+                )
+                .map(|o| o.to_string())
+                .unwrap_or_else(|| "unknown".to_string());
                 AlertHistoryEntry {
                     timestamp: hit.get("_timestamp").and_then(|v| v.as_i64()).unwrap_or(0),
                     alert_name: String::new(),
@@ -380,6 +401,14 @@ pub async fn get_alert_history(
                     grouped: None,
                     group_size: None,
                     anomaly_count: Some(anomaly_count as i32),
+                    // Anomaly rows: the count IS the observed value; the level
+                    // comes from the normalized outcome upstream.
+                    level: None,
+                    actual_value: Some(anomaly_count as f64),
+                    threshold_value: None,
+                    threshold_operator: None,
+                    group_label: None,
+                    value_is_lower_bound: None,
                 }
             })
             .collect();
@@ -620,7 +649,8 @@ pub async fn get_alert_history(
     // Step 2: Get the actual paginated results
     // Build data query with LIMIT/OFFSET for pagination
     let data_sql = format!(
-        "SELECT _timestamp, org, key, status, is_realtime, is_silenced, \
+        "SELECT _timestamp, org, key, status, level, actual_value, threshold_value, \
+         threshold_operator, group_label, value_is_lower_bound, is_realtime, is_silenced, \
          start_time, end_time, retries, \
          delay_in_secs, evaluation_took_in_secs, \
          source_node, query_took, error \
@@ -687,11 +717,20 @@ pub async fn get_alert_history(
                 .and_then(|v| v.as_str())
                 .unwrap_or(&org_id)
                 .to_string(),
+            // Normalize legacy values so the API speaks one vocabulary across
+            // the retention window (Part III of alerts.md).
             status: hit
                 .get("status")
                 .and_then(|v| v.as_str())
-                .unwrap_or("unknown")
-                .to_string(),
+                .and_then(|raw| {
+                    normalize_outcome(
+                        raw,
+                        &TriggerDataType::Alert,
+                        hit.get("success_response").and_then(|v| v.as_str()),
+                    )
+                })
+                .map(|o| o.to_string())
+                .unwrap_or_else(|| "unknown".to_string()),
             is_realtime: hit
                 .get("is_realtime")
                 .and_then(|v| v.as_bool())
@@ -729,6 +768,25 @@ pub async fn get_alert_history(
                 .and_then(|v| v.as_i64())
                 .map(|v| v as i32),
             anomaly_count: None,
+            // Value context (T-9). `level` is stored as an int; map it back to
+            // the string vocabulary the UI already understands.
+            level: hit
+                .get("level")
+                .and_then(|v| v.as_i64())
+                .and_then(|v| config::meta::alerts::level::AlertLevel::from_i32(v as i32))
+                .map(|l| l.to_string()),
+            actual_value: hit.get("actual_value").and_then(|v| v.as_f64()),
+            threshold_value: hit.get("threshold_value").and_then(|v| v.as_f64()),
+            threshold_operator: hit
+                .get("threshold_operator")
+                .and_then(|v| v.as_str())
+                .map(String::from),
+            group_label: hit
+                .get("group_label")
+                .and_then(|v| v.as_str())
+                .filter(|s| !s.is_empty())
+                .map(String::from),
+            value_is_lower_bound: hit.get("value_is_lower_bound").and_then(|v| v.as_bool()),
         });
     }
 
@@ -921,24 +979,19 @@ pub async fn get_all_anomaly_history(
             .get("status")
             .and_then(|v| v.as_str())
             .unwrap_or("unknown");
-        let anomaly_count = hit
-            .get("success_response")
-            .and_then(|v| v.as_str())
+        let success_response = hit.get("success_response").and_then(|v| v.as_str());
+        let anomaly_count = success_response
             .and_then(|s| serde_json::from_str::<serde_json::Value>(s).ok())
             .and_then(|v| v["anomalies_found"].as_i64())
             .unwrap_or(0);
 
-        let status = match raw_status {
-            "failed" => "failed",
-            "skipped" => "skipped",
-            _ => {
-                if anomaly_count > 0 {
-                    "anomaly"
-                } else {
-                    "normal"
-                }
-            }
-        };
+        let status = normalize_outcome(
+            raw_status,
+            &TriggerDataType::AnomalyDetection,
+            success_response,
+        )
+        .map(|o| o.to_string())
+        .unwrap_or_else(|| "unknown".to_string());
 
         bucket.push(serde_json::json!({
             "timestamp": hit.get("_timestamp").and_then(|v| v.as_i64()).unwrap_or(0),
@@ -1016,6 +1069,12 @@ mod tests {
             grouped: None,
             group_size: None,
             anomaly_count: None,
+            level: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
 
         assert_eq!(entry.alert_name, "test_alert");
@@ -1067,6 +1126,12 @@ mod tests {
             grouped: None,
             group_size: None,
             anomaly_count: None,
+            level: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
 
         let response = AlertHistoryResponse {
@@ -1107,6 +1172,12 @@ mod tests {
             grouped: None,
             group_size: None,
             anomaly_count: None,
+            level: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
 
         assert_eq!(entry.status, "error");
@@ -1321,6 +1392,12 @@ mod tests {
             grouped: Some(false),
             group_size: None,
             anomaly_count: None,
+            level: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
 
         let json = serde_json::to_string(&entry).unwrap();
@@ -1358,6 +1435,12 @@ mod tests {
             grouped: None,
             group_size: None,
             anomaly_count: None,
+            level: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
 
         let response = AlertHistoryResponse {
@@ -1401,6 +1484,12 @@ mod tests {
             grouped: None,
             group_size: None,
             anomaly_count: None,
+            level: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
         let json = serde_json::to_value(&entry).unwrap();
         let obj = json.as_object().unwrap();
@@ -1437,6 +1526,12 @@ mod tests {
             grouped: Some(true),
             group_size: Some(3),
             anomaly_count: Some(2),
+            level: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
         let json = serde_json::to_value(&entry).unwrap();
         let obj = json.as_object().unwrap();

src/api/management/src/request/alerts/incident_integrations.rs

@@ -0,0 +1,500 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Admin-facing CRUD for external alert source integrations. Unlike
+//! `external_events`, these routes are authenticated and registered in
+//! `service_routes()` — auth is enforced by the router's auth middleware,
+//! not inside these handlers.
+
+use axum::{Json, extract::Path, response::Response};
+#[cfg(feature = "enterprise")]
+use openobserve_api_common::extractors::Headers;
+#[cfg(feature = "enterprise")]
+use openobserve_core::auth::UserEmail;
+use serde::{Deserialize, Serialize};
+
+use crate::common::meta::http::HttpResponse as MetaHttpResponse;
+
+/// Allowed values for `source_type` on create.
+const ALLOWED_SOURCE_TYPES: &[&str] = &["auto", "grafana", "alertmanager", "generic"];
+
+#[derive(Debug, Deserialize)]
+#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
+pub struct CreateIntegrationPayload {
+    pub name: String,
+    pub source_type: Option<String>,
+    pub config: Option<serde_json::Value>,
+}
+
+#[derive(Debug, Deserialize)]
+#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
+pub struct SetEnabledPayload {
+    pub enabled: bool,
+}
+
+#[derive(Debug, Serialize)]
+#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
+pub struct IntegrationResponse {
+    pub id: String,
+    pub org_id: String,
+    pub name: String,
+    pub source_type: String,
+    pub token: String,
+    pub enabled: bool,
+    pub config: serde_json::Value,
+    pub created_by: String,
+    pub created_at: i64,
+    pub updated_at: i64,
+    pub url: String,
+}
+
+#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
+impl IntegrationResponse {
+    fn from_record(
+        org_id: &str,
+        record: infra::table::incident_integrations::IncidentIntegrationRecord,
+    ) -> Self {
+        let url = format!("/api/v2/{org_id}/incidents/events/{}", record.token);
+        Self {
+            id: record.id,
+            org_id: record.org_id,
+            name: record.name,
+            source_type: record.source_type,
+            token: record.token,
+            enabled: record.enabled,
+            config: record.config,
+            created_by: record.created_by,
+            created_at: record.created_at,
+            updated_at: record.updated_at,
+            url,
+        }
+    }
+}
+
+#[derive(Debug, Serialize)]
+#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
+pub struct ListIntegrationsResponse {
+    pub integrations: Vec<IntegrationResponse>,
+}
+
+#[derive(Debug, Serialize)]
+#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
+pub struct SenderResponse {
+    pub integration_id: String,
+    pub detected_source: String,
+    pub display_name: String,
+    pub first_received_at: i64,
+    pub last_received_at: i64,
+    pub accepted_count: i64,
+    pub rejected_count: i64,
+    pub resolved_seen: bool,
+    pub resolve_wiring_hint: bool,
+}
+
+#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
+impl From<infra::table::incident_integrations::SenderRecord> for SenderResponse {
+    fn from(r: infra::table::incident_integrations::SenderRecord) -> Self {
+        let resolve_wiring_hint = r.accepted_count > 0 && !r.resolved_seen;
+        let display_name = openobserve_core::alerts::external_alerts::resolve_display_name(
+            &r.detected_source,
+            r.sender_label.as_deref(),
+        );
+        Self {
+            integration_id: r.integration_id,
+            detected_source: r.detected_source,
+            display_name,
+            first_received_at: r.first_received_at,
+            last_received_at: r.last_received_at,
+            accepted_count: r.accepted_count,
+            rejected_count: r.rejected_count,
+            resolved_seen: r.resolved_seen,
+            resolve_wiring_hint,
+        }
+    }
+}
+
+/// Validates a create-integration payload: non-empty name (<=100 chars) and
+/// an allowed `source_type` (defaulting to "auto" when absent).
+#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
+pub(crate) fn validate_create(payload: &CreateIntegrationPayload) -> Result<(), String> {
+    if payload.name.trim().is_empty() {
+        return Err("name cannot be empty".to_string());
+    }
+    if payload.name.len() > 100 {
+        return Err("name cannot exceed 100 characters".to_string());
+    }
+    if let Some(st) = payload.source_type.as_deref()
+        && !ALLOWED_SOURCE_TYPES.contains(&st)
+    {
+        return Err(format!(
+            "source_type must be one of: {}",
+            ALLOWED_SOURCE_TYPES.join(", ")
+        ));
+    }
+    Ok(())
+}
+
+#[cfg(feature = "enterprise")]
+fn gate_enabled() -> Option<Response> {
+    use o2_enterprise::enterprise::common::config::get_config as o2_config;
+    if !o2_config().incidents.enabled {
+        return Some(MetaHttpResponse::forbidden(
+            "External alert sources not enabled",
+        ));
+    }
+    None
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn list_integrations(
+    Path(org_id): Path<String>,
+    Headers(user_email): Headers<UserEmail>,
+) -> Response {
+    if let Some(resp) = gate_enabled() {
+        return resp;
+    }
+
+    if let Err(e) =
+        infra::table::incident_integrations::ensure_default_for_org(&org_id, &user_email.user_id)
+            .await
+    {
+        return MetaHttpResponse::internal_error(e);
+    }
+
+    match infra::table::incident_integrations::list_by_org(&org_id).await {
+        Ok(records) => MetaHttpResponse::json(ListIntegrationsResponse {
+            integrations: records
+                .into_iter()
+                .map(|r| IntegrationResponse::from_record(&org_id, r))
+                .collect(),
+        }),
+        Err(e) => MetaHttpResponse::internal_error(e),
+    }
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn create_integration(
+    Path(org_id): Path<String>,
+    Headers(user_email): Headers<UserEmail>,
+    Json(payload): Json<CreateIntegrationPayload>,
+) -> Response {
+    if let Some(resp) = gate_enabled() {
+        return resp;
+    }
+    if let Err(e) = validate_create(&payload) {
+        return MetaHttpResponse::bad_request(e);
+    }
+
+    let now = chrono::Utc::now().timestamp_micros();
+    let record = infra::table::incident_integrations::IncidentIntegrationRecord {
+        id: config::ider::uuid(),
+        org_id: org_id.clone(),
+        name: payload.name,
+        source_type: payload.source_type.unwrap_or_else(|| "auto".to_string()),
+        token: infra::table::incident_integrations::generate_token(),
+        enabled: true,
+        config: payload.config.unwrap_or_else(|| serde_json::json!({})),
+        created_by: user_email.user_id,
+        created_at: now,
+        updated_at: now,
+    };
+
+    match infra::table::incident_integrations::add(&record).await {
+        Ok(()) => MetaHttpResponse::json(IntegrationResponse::from_record(&org_id, record)),
+        Err(e) => MetaHttpResponse::bad_request(e),
+    }
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn set_integration_enabled(
+    Path((org_id, integration_id)): Path<(String, String)>,
+    Json(payload): Json<SetEnabledPayload>,
+) -> Response {
+    if let Some(resp) = gate_enabled() {
+        return resp;
+    }
+
+    match infra::table::incident_integrations::set_enabled(
+        &org_id,
+        &integration_id,
+        payload.enabled,
+    )
+    .await
+    {
+        Ok(()) => MetaHttpResponse::ok("updated"),
+        Err(e) => MetaHttpResponse::internal_error(e),
+    }
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn rotate_integration_token(
+    Path((org_id, integration_id)): Path<(String, String)>,
+) -> Response {
+    if let Some(resp) = gate_enabled() {
+        return resp;
+    }
+
+    match infra::table::incident_integrations::rotate_token(&org_id, &integration_id).await {
+        Ok(token) => MetaHttpResponse::json(serde_json::json!({"token": token})),
+        Err(e) => MetaHttpResponse::internal_error(e),
+    }
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn delete_integration(
+    Path((org_id, integration_id)): Path<(String, String)>,
+) -> Response {
+    if let Some(resp) = gate_enabled() {
+        return resp;
+    }
+
+    let integration = match infra::table::incident_integrations::list_by_org(&org_id).await {
+        Ok(records) => records.into_iter().find(|r| r.id == integration_id),
+        Err(e) => return MetaHttpResponse::internal_error(e),
+    };
+    let Some(integration) = integration else {
+        return MetaHttpResponse::not_found("Integration not found");
+    };
+    if integration.name == infra::table::incident_integrations::DEFAULT_INTEGRATION_NAME {
+        return MetaHttpResponse::bad_request(
+            "The default alert source cannot be deleted — disable it instead",
+        );
+    }
+
+    match infra::table::incident_integrations::delete(&org_id, &integration_id).await {
+        Ok(true) => MetaHttpResponse::ok("deleted"),
+        Ok(false) => MetaHttpResponse::not_found("Integration not found"),
+        Err(e) => MetaHttpResponse::internal_error(e),
+    }
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn list_integration_senders(
+    Path((org_id, integration_id)): Path<(String, String)>,
+) -> Response {
+    if let Some(resp) = gate_enabled() {
+        return resp;
+    }
+
+    // Verify the integration belongs to this org before returning sender data.
+    let belongs = match infra::table::incident_integrations::list_by_org(&org_id).await {
+        Ok(records) => records.iter().any(|r| r.id == integration_id),
+        Err(e) => return MetaHttpResponse::internal_error(e),
+    };
+    if !belongs {
+        return MetaHttpResponse::not_found("Integration not found");
+    }
+
+    match infra::table::incident_integrations::list_senders(&integration_id).await {
+        Ok(senders) => {
+            let senders: Vec<SenderResponse> =
+                senders.into_iter().map(SenderResponse::from).collect();
+            MetaHttpResponse::json(serde_json::json!({"senders": senders}))
+        }
+        Err(e) => MetaHttpResponse::internal_error(e),
+    }
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn list_integrations(_path: Path<String>) -> Response {
+    MetaHttpResponse::forbidden("Not Supported")
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn create_integration(
+    _path: Path<String>,
+    _body: Json<CreateIntegrationPayload>,
+) -> Response {
+    MetaHttpResponse::forbidden("Not Supported")
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn set_integration_enabled(
+    _path: Path<(String, String)>,
+    _body: Json<SetEnabledPayload>,
+) -> Response {
+    MetaHttpResponse::forbidden("Not Supported")
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn rotate_integration_token(_path: Path<(String, String)>) -> Response {
+    MetaHttpResponse::forbidden("Not Supported")
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn list_integration_senders(_path: Path<(String, String)>) -> Response {
+    MetaHttpResponse::forbidden("Not Supported")
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn delete_integration(_path: Path<(String, String)>) -> Response {
+    MetaHttpResponse::forbidden("Not Supported")
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn payload(name: &str, source_type: Option<&str>) -> CreateIntegrationPayload {
+        CreateIntegrationPayload {
+            name: name.to_string(),
+            source_type: source_type.map(String::from),
+            config: None,
+        }
+    }
+
+    #[test]
+    fn test_validate_create_rejects_empty_name() {
+        assert_eq!(
+            validate_create(&payload("", None)),
+            Err("name cannot be empty".to_string())
+        );
+        assert_eq!(
+            validate_create(&payload("   ", None)),
+            Err("name cannot be empty".to_string())
+        );
+    }
+
+    #[test]
+    fn test_validate_create_rejects_long_name() {
+        let long_name = "a".repeat(101);
+        assert!(validate_create(&payload(&long_name, None)).is_err());
+        let ok_name = "a".repeat(100);
+        assert!(validate_create(&payload(&ok_name, None)).is_ok());
+    }
+
+    #[test]
+    fn test_validate_create_rejects_bad_source_type() {
+        assert!(validate_create(&payload("n", Some("bogus"))).is_err());
+    }
+
+    #[test]
+    fn test_validate_create_accepts_allowed_source_types() {
+        for st in ALLOWED_SOURCE_TYPES {
+            assert!(validate_create(&payload("n", Some(st))).is_ok());
+        }
+    }
+
+    #[test]
+    fn test_validate_create_defaults_source_type_none_ok() {
+        assert!(validate_create(&payload("n", None)).is_ok());
+    }
+
+    #[test]
+    fn test_create_payload_deserializes() {
+        let json = r#"{"name":"grafana-prod","source_type":"grafana","config":{"destinations":["slack"]}}"#;
+        let p: CreateIntegrationPayload = serde_json::from_str(json).unwrap();
+        assert_eq!(p.name, "grafana-prod");
+        assert_eq!(p.source_type.as_deref(), Some("grafana"));
+        assert_eq!(p.config.unwrap()["destinations"][0], "slack");
+    }
+
+    #[test]
+    fn test_create_payload_optional_fields_absent() {
+        let json = r#"{"name":"minimal"}"#;
+        let p: CreateIntegrationPayload = serde_json::from_str(json).unwrap();
+        assert_eq!(p.name, "minimal");
+        assert!(p.source_type.is_none());
+        assert!(p.config.is_none());
+    }
+
+    #[test]
+    fn test_set_enabled_payload_round_trip() {
+        let json = r#"{"enabled":false}"#;
+        let p: SetEnabledPayload = serde_json::from_str(json).unwrap();
+        assert!(!p.enabled);
+    }
+
+    #[test]
+    fn test_sender_response_resolve_wiring_hint_true_when_accepted_but_unresolved() {
+        let r = infra::table::incident_integrations::SenderRecord {
+            integration_id: "i1".into(),
+            detected_source: "grafana".into(),
+            first_received_at: 1,
+            last_received_at: 2,
+            accepted_count: 5,
+            rejected_count: 0,
+            resolved_seen: false,
+            sender_label: None,
+        };
+        let resp: SenderResponse = r.into();
+        assert!(resp.resolve_wiring_hint);
+    }
+
+    #[test]
+    fn test_sender_response_resolve_wiring_hint_false_when_resolved_seen() {
+        let r = infra::table::incident_integrations::SenderRecord {
+            integration_id: "i1".into(),
+            detected_source: "grafana".into(),
+            first_received_at: 1,
+            last_received_at: 2,
+            accepted_count: 5,
+            rejected_count: 0,
+            resolved_seen: true,
+            sender_label: None,
+        };
+        let resp: SenderResponse = r.into();
+        assert!(!resp.resolve_wiring_hint);
+    }
+
+    #[test]
+    fn test_sender_response_resolve_wiring_hint_false_when_no_accepted() {
+        let r = infra::table::incident_integrations::SenderRecord {
+            integration_id: "i1".into(),
+            detected_source: "grafana".into(),
+            first_received_at: 1,
+            last_received_at: 2,
+            accepted_count: 0,
+            rejected_count: 3,
+            resolved_seen: false,
+            sender_label: None,
+        };
+        let resp: SenderResponse = r.into();
+        assert!(!resp.resolve_wiring_hint);
+    }
+
+    #[test]
+    fn test_sender_response_display_name_uses_label_when_present() {
+        let record = infra::table::incident_integrations::SenderRecord {
+            integration_id: "int-1".to_string(),
+            detected_source: "generic".to_string(),
+            sender_label: Some("solarwinds".to_string()),
+            first_received_at: 1,
+            last_received_at: 2,
+            accepted_count: 3,
+            rejected_count: 0,
+            resolved_seen: false,
+        };
+        let response = SenderResponse::from(record);
+        assert_eq!(response.display_name, "solarwinds");
+        assert_eq!(response.detected_source, "generic");
+    }
+
+    #[test]
+    fn test_sender_response_display_name_falls_back_to_detected_source() {
+        let record = infra::table::incident_integrations::SenderRecord {
+            integration_id: "int-1".to_string(),
+            detected_source: "grafana".to_string(),
+            sender_label: None,
+            first_received_at: 1,
+            last_received_at: 2,
+            accepted_count: 3,
+            rejected_count: 0,
+            resolved_seen: false,
+        };
+        let response = SenderResponse::from(record);
+        assert_eq!(response.display_name, "grafana");
+    }
+}

src/api/management/src/request/alerts/incidents.rs

@@ -294,6 +294,47 @@ pub async fn update_incident(
     }
 }
 
+#[cfg(feature = "enterprise")]
+/// GetExternalAlertPayload
+#[utoipa::path(
+    get,
+    path = "/v2/{org_id}/alerts/incidents/external-alerts/{external_alert_id}/payload",
+    context_path = "/api",
+    tag = "Incidents",
+    operation_id = "GetExternalAlertPayload",
+    summary = "Get the raw payload of an external alert",
+    description = "Retrieves the original, unmodified webhook payload received for an external alert event. Used to inspect exactly what a source system (Grafana, Alertmanager, etc.) sent.",
+    security(("Authorization" = [])),
+    params(
+        ("org_id" = String, Path, description = "Organization name"),
+        ("external_alert_id" = String, Path, description = "External alert ID"),
+    ),
+    responses(
+        (status = 200, description = "Success", content_type = "application/json", body = ()),
+        (status = 404, description = "Not found", content_type = "application/json", body = ()),
+    ),
+    extensions(
+        ("x-o2-ratelimit" = json!({"module": "Alerts", "operation": "get"})),
+        ("x-o2-mcp" = json!({"description": "Get the raw payload of an external alert", "category": "alerts"}))
+    )
+)]
+pub async fn get_external_alert_payload(
+    Path((org_id, external_alert_id)): Path<(String, String)>,
+) -> Response {
+    match infra::table::external_alerts::get_by_id(&org_id, &external_alert_id).await {
+        Ok(Some(record)) => MetaHttpResponse::json(serde_json::json!({
+            "id": record.id,
+            "detected_source": record.detected_source,
+            "source_url": record.source_url,
+            "first_seen_at": record.first_seen_at,
+            "last_seen_at": record.last_seen_at,
+            "last_payload": record.last_payload,
+        })),
+        Ok(None) => MetaHttpResponse::not_found("External alert not found"),
+        Err(e) => MetaHttpResponse::internal_error(e),
+    }
+}
+
 #[cfg(feature = "enterprise")]
 /// GetIncidentStats
 #[utoipa::path(
@@ -967,6 +1008,33 @@ pub async fn get_incident(_path: Path<(String, String)>) -> Response {
     MetaHttpResponse::forbidden("Not Supported")
 }
 
+#[cfg(not(feature = "enterprise"))]
+#[utoipa::path(
+    get,
+    path = "/v2/{org_id}/alerts/incidents/external-alerts/{external_alert_id}/payload",
+    context_path = "/api",
+    tag = "Incidents",
+    operation_id = "GetExternalAlertPayload",
+    summary = "Get the raw payload of an external alert",
+    description = "Retrieves the original webhook payload for an external alert event. This endpoint is only available with enterprise features enabled.",
+    security(("Authorization" = [])),
+    params(
+        ("org_id" = String, Path, description = "Organization name"),
+        ("external_alert_id" = String, Path, description = "External alert ID"),
+    ),
+    responses(
+        (status = 200, description = "Success", content_type = "application/json", body = ()),
+        (status = 403, description = "Enterprise feature", content_type = "application/json", body = ()),
+    ),
+    extensions(
+        ("x-o2-ratelimit" = json!({"module": "Alerts", "operation": "get"})),
+        ("x-o2-mcp" = json!({"description": "Get the raw payload of an external alert", "category": "alerts"}))
+    )
+)]
+pub async fn get_external_alert_payload(_path: Path<(String, String)>) -> Response {
+    MetaHttpResponse::forbidden("Not Supported")
+}
+
 #[cfg(not(feature = "enterprise"))]
 #[utoipa::path(
     get,

src/api/management/src/request/alerts/mod.rs

@@ -21,6 +21,7 @@ use axum::{
     http::StatusCode,
     response::Response,
 };
+use axum_extra::extract::Query as ExtraQuery;
 use config::meta::{
     alerts::alert::{Alert as MetaAlert, AlertTypeFilter},
     triggers::{Trigger, TriggerModule},
@@ -57,9 +58,10 @@ use crate::{
             UpdateAlertRequestBody,
         },
         responses::{
-            AlertBulkEnableResponse, EnableAlertResponseBody, GenerateSqlMetadata,
-            GenerateSqlResponseBody, GetAlertResponseBody, ListAlertsResponseBody,
-            ListAlertsResponseBodyItem,
+            AlertBulkEnableResponse, AlertGroupLabel, AlertGroupResponseItem,
+            AlertGroupTransitionItem, EnableAlertResponseBody, GenerateSqlMetadata,
+            GenerateSqlResponseBody, GetAlertResponseBody, ListAlertGroupTransitionsResponseBody,
+            ListAlertGroupsResponseBody, ListAlertsResponseBody, ListAlertsResponseBodyItem,
         },
     },
     request::{
@@ -71,7 +73,9 @@ use crate::{
 pub mod dedup_stats;
 pub mod deduplication;
 pub mod destinations;
+pub mod external_events;
 pub mod history;
+pub mod incident_integrations;
 pub mod incidents;
 pub mod templates;
 
@@ -184,10 +188,213 @@ async fn create_anomaly_alert(
             .filter(|f| !f.is_empty())
             .or_else(|| Some(query_folder_id.to_string()).filter(|f| !f.is_empty())),
         owner,
+        // Feature 2: anomaly configs take the same triage metadata as
+        // alerts, threaded from the shared request body.
+        priority: req_body.alert.priority,
+        tags: req_body.alert.tags,
     };
 
     match openobserve_core::anomaly_detection::create_config(org_id, req).await {
         Ok(v) => MetaHttpResponse::json(v),
+        // A bad tag is user input: the same 400 the alert save path gives.
+        Err(e)
+            if e.downcast_ref::<config::meta::alerts::tags::TagError>()
+                .is_some() =>
+        {
+            MetaHttpResponse::bad_request(e.to_string())
+        }
+        Err(e) => MetaHttpResponse::internal_error(e.to_string()),
+    }
+}
+
+/// Split a rendered `k=v,k=v` label string back into pairs.
+///
+/// Rendering is lossy where a value contains the separators, which is exactly
+/// why `group_key` — not this — is the identity. Splitting on the first `=`
+/// keeps `path=/a=b` intact, the common case; anything genuinely ambiguous
+/// still displays, it just may split oddly.
+fn parse_group_labels(rendered: Option<&str>) -> Vec<AlertGroupLabel> {
+    rendered
+        .unwrap_or("")
+        .split(',')
+        .filter(|s| !s.is_empty())
+        .filter_map(|pair| {
+            pair.split_once('=').map(|(name, value)| AlertGroupLabel {
+                name: name.to_string(),
+                value: value.to_string(),
+            })
+        })
+        .collect()
+}
+
+/// ListAlertGroups
+#[utoipa::path(
+    get,
+    path = "/v2/{org_id}/alerts/{alert_id}/groups",
+    context_path = "/api",
+    tag = "Alerts",
+    operation_id = "ListAlertGroups",
+    summary = "List a multi-alert's tracked groups",
+    description = "Returns the per-group state rows of a multi-alert, most severe first, with the pre-cap group counts needed to render 'N of M groups firing'. Empty for alerts that have not opted in to per-group evaluation.",
+    security(
+        ("Authorization"= [])
+    ),
+    params(
+        ("org_id" = String, Path, description = "Organization name"),
+        ("alert_id" = String, Path, description = "Alert ID"),
+      ),
+    responses(
+        (status = 200, description = "Success",  content_type = "application/json", body = inline(ListAlertGroupsResponseBody)),
+        (status = 404, description = "NotFound", content_type = "application/json", body = ()),
+    ),
+    extensions(
+        ("x-o2-ratelimit" = json!({"module": "Alerts", "operation": "get"})),
+        ("x-o2-mcp" = json!({"description": "List the per-group states of a multi-alert", "category": "alerts"}))
+    )
+)]
+pub async fn list_alert_groups(Path((org_id, alert_id)): Path<(String, String)>) -> Response {
+    let Ok(ksuid) = Ksuid::from_str(&alert_id) else {
+        return MetaHttpResponse::not_found(format!("invalid alert id {alert_id}"));
+    };
+    let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
+    // Resolve through the alert itself so this endpoint inherits the same
+    // org scoping and not-found behaviour as every other alert read — a raw
+    // state-table query would happily serve another org's group labels, which
+    // carry host and service names.
+    if let Err(e) = alert::get_by_id(client, &org_id, ksuid).await {
+        return e.into();
+    }
+
+    let mut groups = match infra::table::alert_states::list_groups(&alert_id).await {
+        Ok(g) => g,
+        Err(e) => return MetaHttpResponse::internal_error(e.to_string()),
+    };
+    // Most severe first (§5.4), then by key so equal levels keep a stable
+    // order across polls instead of shuffling on every refresh.
+    groups.sort_by(|a, b| {
+        let rank = |s: &config::meta::alerts::state::AlertState| {
+            s.level.map(|l| l.severity_rank()).unwrap_or(0)
+        };
+        rank(b)
+            .cmp(&rank(a))
+            .then_with(|| a.group_key.cmp(&b.group_key))
+    });
+
+    let rollup = infra::table::alert_states::get_rollups(std::slice::from_ref(&alert_id))
+        .await
+        .ok()
+        .and_then(|mut r| r.pop());
+    let groups_observed = rollup
+        .as_ref()
+        .and_then(|r| r.groups_observed)
+        .and_then(|n| i32::try_from(n).ok());
+    let group_cap = config::get_config().limit.alert_max_groups;
+
+    let list: Vec<AlertGroupResponseItem> = groups
+        .into_iter()
+        .map(|g| AlertGroupResponseItem {
+            labels: parse_group_labels(g.group_labels.as_deref()),
+            group_key: g.group_key,
+            group_labels: g.group_labels,
+            level: g.level.map(|l| l.to_string()),
+            level_since: g.level_since,
+            last_outcome: g.last_outcome.map(|o| o.to_string()),
+            last_outcome_at: g.last_outcome_at,
+            last_seen: g.last_seen,
+            silenced_until: g.silenced_until,
+            last_notified_level: g.last_notified_level.map(|l| l.to_string()),
+        })
+        .collect();
+
+    // Compare the PRE-cap observed total against the cap, not the length of
+    // `list`: the retained rows are post-cap, so they would report "cap of
+    // cap" and an overflowing alert would look identical to one that fit.
+    let capped =
+        group_cap > 0 && groups_observed.is_some_and(|observed| observed as usize > group_cap);
+
+    MetaHttpResponse::json(ListAlertGroupsResponseBody {
+        list,
+        groups_observed,
+        groups_firing: rollup
+            .as_ref()
+            .and_then(|r| r.groups_firing)
+            .and_then(|n| i32::try_from(n).ok()),
+        groups_observed_is_lower_bound: rollup
+            .as_ref()
+            .and_then(|r| r.groups_observed_is_lower_bound),
+        groups_firing_is_lower_bound: rollup.as_ref().and_then(|r| r.groups_firing_is_lower_bound),
+        capped,
+        group_cap,
+    })
+}
+
+/// ListAlertGroupTransitions
+#[utoipa::path(
+    get,
+    path = "/v2/{org_id}/alerts/{alert_id}/groups/transitions",
+    context_path = "/api",
+    tag = "Alerts",
+    operation_id = "ListAlertGroupTransitions",
+    summary = "Per-group state history for a multi-alert",
+    description = "Returns level/outcome transitions for a multi-alert, newest first, optionally scoped to one group (M-8). Reads the durable transitions table rather than the triggers stream, so history survives group reaping.",
+    security(
+        ("Authorization"= [])
+    ),
+    params(
+        ("org_id" = String, Path, description = "Organization name"),
+        ("alert_id" = String, Path, description = "Alert ID"),
+        ("group_key" = Option<String>, Query, description = "Restrict to one group. Omit for every group."),
+        ("limit" = Option<u64>, Query, description = "Max rows (default 100, max 1000)"),
+      ),
+    responses(
+        (status = 200, description = "Success",  content_type = "application/json", body = inline(ListAlertGroupTransitionsResponseBody)),
+        (status = 404, description = "NotFound", content_type = "application/json", body = ()),
+    ),
+    extensions(
+        ("x-o2-ratelimit" = json!({"module": "Alerts", "operation": "get"})),
+        ("x-o2-mcp" = json!({"description": "Per-group alert state history", "category": "alerts"}))
+    )
+)]
+pub async fn list_alert_group_transitions(
+    Path((org_id, alert_id)): Path<(String, String)>,
+    Query(query): Query<HashMap<String, String>>,
+) -> Response {
+    let Ok(ksuid) = Ksuid::from_str(&alert_id) else {
+        return MetaHttpResponse::not_found(format!("invalid alert id {alert_id}"));
+    };
+    let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
+    if let Err(e) = alert::get_by_id(client, &org_id, ksuid).await {
+        return e.into();
+    }
+
+    // `None` means every group, which is NOT `Some("")` — that is the rollup
+    // row's own key. An empty query value therefore has to mean "unset".
+    let group_key = query
+        .get("group_key")
+        .map(|s| s.trim())
+        .filter(|s| !s.is_empty());
+    let limit = query
+        .get("limit")
+        .and_then(|l| l.parse::<u64>().ok())
+        .unwrap_or(100)
+        .clamp(1, 1000);
+
+    match infra::table::alert_states::list_transitions_filtered(&alert_id, group_key, limit).await {
+        Ok(transitions) => MetaHttpResponse::json(ListAlertGroupTransitionsResponseBody {
+            list: transitions
+                .into_iter()
+                .map(|t| AlertGroupTransitionItem {
+                    group_key: t.group_key,
+                    group_labels: t.group_labels,
+                    from_level: t.from_level.map(|l| l.to_string()),
+                    to_level: t.to_level.map(|l| l.to_string()),
+                    from_outcome: t.from_outcome.map(|o| o.to_string()),
+                    to_outcome: t.to_outcome.to_string(),
+                    at: t.at,
+                    value: t.value,
+                })
+                .collect(),
+        }),
         Err(e) => MetaHttpResponse::internal_error(e.to_string()),
     }
 }
@@ -582,10 +789,24 @@ async fn build_and_run_anomaly_update(
         enabled: fields.enabled,
         folder_id: fields.folder_id,
         owner,
+        // The v2 PUT carries the FULL alert body, so this is replace
+        // semantics: wrapping in `Some` means an omitted priority clears it,
+        // matching how tags behave one line down.
+        priority: Some(alert.priority),
+        // `Some(vec![])` clears; the shared body always supplies a Vec,
+        // so an edit that removes every tag does clear them.
+        tags: Some(alert.tags),
     };
 
     match openobserve_core::anomaly_detection::update_config(org_id, anomaly_id, req).await {
         Ok(v) => MetaHttpResponse::json(v),
+        // Same 400 as create: an invalid tag is the caller's to fix.
+        Err(e)
+            if e.downcast_ref::<config::meta::alerts::tags::TagError>()
+                .is_some() =>
+        {
+            MetaHttpResponse::bad_request(e.to_string())
+        }
         Err(e) => MetaHttpResponse::internal_error(e.to_string()),
     }
 }
@@ -755,6 +976,102 @@ pub async fn delete_alert_bulk(
     })
 }
 
+/// Query parameters for the tag facet endpoint (PT-8b).
+#[derive(Debug, serde::Deserialize, utoipa::IntoParams)]
+#[into_params(style = Form, parameter_in = Query)]
+#[serde(rename_all = "snake_case")]
+pub struct ListAlertTagsQuery {
+    /// Optional case-insensitive prefix filter for autocomplete.
+    pub prefix: Option<String>,
+    /// Maximum tags to return. Defaults to 100, capped at 1000.
+    pub limit: Option<usize>,
+    /// Restrict to one folder, matching the list endpoint's scope.
+    pub folder: Option<String>,
+}
+
+/// One tag and how many visible alerts carry it.
+#[derive(Debug, serde::Serialize, utoipa::ToSchema)]
+pub struct AlertTagCount {
+    pub tag: String,
+    pub count: u64,
+}
+
+/// ListAlertTags
+///
+/// Distinct alert tags for autocomplete and facets (PT-8b).
+///
+/// **Authorization is load-bearing, not incidental (D23):** tag values leak
+/// service, environment, team and customer names, so this returns only tags
+/// carried by alerts the caller may actually list. It reuses the list
+/// endpoint's permission path rather than scanning the org-wide cache.
+#[utoipa::path(
+    get,
+    path = "/{org_id}/alerts/tags",
+    context_path = "/api/v2",
+    tag = "Alerts",
+    operation_id = "ListAlertTags",
+    summary = "List distinct alert tags",
+    description = "Returns distinct tags across the alerts the caller can see, with occurrence counts, for autocomplete and filter facets.",
+    security(("Authorization" = [])),
+    params(("org_id" = String, Path, description = "Organization name"), ListAlertTagsQuery),
+    responses(
+        (status = 200, description = "Success", content_type = "application/json", body = Vec<AlertTagCount>),
+        (status = 403, description = "Forbidden", content_type = "application/json"),
+    ),
+)]
+pub async fn list_alert_tags(
+    Path(org_id): Path<String>,
+    Query(query): Query<ListAlertTagsQuery>,
+    #[cfg(feature = "enterprise")] Headers(user_email): Headers<UserEmail>,
+) -> Response {
+    #[cfg(not(feature = "enterprise"))]
+    let user_id = None;
+    #[cfg(feature = "enterprise")]
+    let user_id = Some(user_email.user_id.as_str());
+
+    // Bounded (PT-8b): 1,000 alerts x 64 tags is 64,000 values, so "return
+    // everything" is not an option the response size can afford.
+    const DEFAULT_LIMIT: usize = 100;
+    const MAX_LIMIT: usize = 1000;
+    let limit = query.limit.unwrap_or(DEFAULT_LIMIT).clamp(1, MAX_LIMIT);
+
+    // Resolve the caller's VISIBLE alerts through the same permission path the
+    // list endpoint uses — this is what keeps the facet from leaking tags off
+    // alerts the caller cannot see.
+    let mut params = config::meta::alerts::alert::ListAlertsParams::new(&org_id);
+    if let Some(folder) = query.folder.clone() {
+        params = params.in_folder(&folder);
+    }
+    let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
+    let visible_ids: Vec<String> = match alert::list_v2(client, user_id, params).await {
+        Ok(list) => list
+            .into_iter()
+            .filter_map(|(_, a)| a.id.map(|id| id.to_string()))
+            .collect(),
+        Err(e) => return e.into(),
+    };
+
+    let mut counts = db::alerts::alert::tag_counts_for_alerts(&org_id, &visible_ids).await;
+
+    if let Some(prefix) = query.prefix.as_deref() {
+        let prefix = prefix.trim().to_lowercase();
+        if !prefix.is_empty() {
+            counts.retain(|(tag, _)| tag.starts_with(&prefix));
+        }
+    }
+
+    // Deterministic order: most-used first, then lexicographic so the tail is
+    // stable rather than hash-ordered.
+    counts.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0)));
+    counts.truncate(limit);
+
+    let body: Vec<AlertTagCount> = counts
+        .into_iter()
+        .map(|(tag, count)| AlertTagCount { tag, count })
+        .collect();
+    MetaHttpResponse::json(body)
+}
+
 /// ListAlerts
 #[utoipa::path(
     get,
@@ -785,7 +1102,11 @@ pub async fn delete_alert_bulk(
 )]
 pub async fn list_alerts(
     Path(org_id): Path<String>,
-    Query(query): Query<ListAlertsQuery>,
+    // `axum_extra`'s Query (serde_html_form) rather than axum's
+    // (serde_urlencoded): only the former deserializes REPEATED keys into a
+    // `Vec`, which PT-3 requires for `?priority=1&priority=2`. axum's Query
+    // errors with "invalid type: string, expected a sequence".
+    ExtraQuery(query): ExtraQuery<ListAlertsQuery>,
     #[cfg(feature = "enterprise")] Headers(user_email): Headers<UserEmail>,
 ) -> Response {
     #[cfg(not(feature = "enterprise"))]
@@ -800,12 +1121,38 @@ pub async fn list_alerts(
     #[cfg(feature = "enterprise")]
     let page_size_and_idx = query.page_size.map(|s| (s, query.page_idx.unwrap_or(0)));
 
+    // Resolve the tag filter to an alert-ID set BEFORE building the query
+    // (PT-8): the tags column is JSON, and the filter must enter the SQL as an
+    // ID predicate so pagination and sorting stay correct rather than
+    // post-filtering an already-fetched page.
+    let requested_tags = query.requested_tags();
+
     #[cfg(not(feature = "enterprise"))]
-    let params = query.into(&org_id);
+    let mut params = query.into(&org_id);
     #[cfg(feature = "enterprise")]
     let mut params = query.into(&org_id);
 
+    if !requested_tags.is_empty() {
+        // `Some(empty)` is meaningful: no alert carries these tags, so the
+        // result must be empty. Leaving it `None` would match everything.
+        params = params.with_tag_alert_ids(
+            db::alerts::alert::resolve_alert_ids_by_tags(&org_id, &requested_tags).await,
+        );
+    }
+
     let alert_type = params.alert_type;
+    // Anomaly configs are merged in AFTER the SQL query, so the priority/tag
+    // filters in the query never touch them — they must be applied in Rust to
+    // the merged rows instead (see below). Captured before `params` is moved.
+    // (enterprise-only consumers; OSS builds merge no anomaly configs)
+    #[cfg_attr(not(feature = "enterprise"), allow(unused_variables))]
+    let priority_filter = params.priority.clone();
+    #[cfg_attr(not(feature = "enterprise"), allow(unused_variables))]
+    let tag_filter = requested_tags.clone();
+    // Anomaly rows bypass the SQL ORDER BY too, so a merged list must be
+    // re-sorted in memory.
+    #[cfg_attr(not(feature = "enterprise"), allow(unused_variables))]
+    let requested_sort = (params.sort_by, params.sort_desc);
 
     // In enterprise builds, pagination is applied after merging regular alerts with
     // anomaly detection configs, so we fetch all matching results from the DB here.
@@ -863,7 +1210,33 @@ pub async fn list_alerts(
     )
     .await
     {
-        list.extend(configs.iter().filter_map(anomaly_config_to_list_item));
+        // Apply the Feature-2 filters here, because these rows bypassed the
+        // SQL WHERE clause entirely. Without this a priority or tag filter
+        // would return every anomaly config alongside the matching alerts.
+        let before = list.len();
+        list.extend(
+            configs
+                .iter()
+                .filter_map(anomaly_config_to_list_item)
+                .filter(|item| match &priority_filter {
+                    None => true,
+                    // An empty set means "asked for priorities, none valid" —
+                    // matches nothing, same as the SQL side.
+                    Some(wanted) => item
+                        .priority
+                        .is_some_and(|p| wanted.iter().any(|w| w.to_i32() as u8 == p)),
+                })
+                .filter(|item| {
+                    config::meta::alerts::tags::matches_all_tags(&item.tags, &tag_filter)
+                }),
+        );
+        // Without this the appended rows pin to the tail whatever order was
+        // requested, and the pagination below cuts the combined list in the
+        // wrong places. Only when something merged — an untouched list keeps
+        // the database's own collation.
+        if list.len() > before {
+            sort_merged_alert_list(&mut list, requested_sort.0, requested_sort.1);
+        }
     }
 
     // Apply pagination to the combined list (regular alerts + anomaly configs).
@@ -879,9 +1252,108 @@ pub async fn list_alerts(
         list
     };
 
+    // Enrich with durable run state (Part IV of alerts.md). One batched query
+    // over the page that is actually being returned — not per alert.
+    let mut list = list;
+    enrich_with_run_state(&mut list).await;
+
     MetaHttpResponse::json(ListAlertsResponseBody { list })
 }
 
+/// Attach `last_outcome` / `last_outcome_at` / `last_outcome_since` to a page of
+/// alerts from the `alert_states` rollup rows.
+///
+/// Best-effort: if the lookup fails the list is still returned, just without run
+/// state. A state table problem must not take down the alerts page.
+/// Re-sort a merged (regular + anomaly) list the way the SQL ORDER BY sorts
+/// the regular one (PT-3): unset priority LAST in both directions, ties broken
+/// on (name, folder name, id) so pagination stays a total order.
+#[cfg(feature = "enterprise")]
+fn sort_merged_alert_list(
+    list: &mut [ListAlertsResponseBodyItem],
+    sort_by: Option<config::meta::alerts::alert::AlertSortField>,
+    sort_desc: bool,
+) {
+    use std::cmp::Ordering;
+
+    use config::meta::alerts::alert::AlertSortField;
+
+    let tail = |a: &ListAlertsResponseBodyItem, b: &ListAlertsResponseBodyItem| {
+        a.name
+            .cmp(&b.name)
+            .then_with(|| a.folder_name.cmp(&b.folder_name))
+            .then_with(|| a.alert_id.to_string().cmp(&b.alert_id.to_string()))
+    };
+    match sort_by {
+        Some(AlertSortField::Priority) => list.sort_by(|a, b| {
+            // NULL priority sorts last regardless of direction, matching the
+            // SQL's explicit CASE.
+            let nulls = (a.priority.is_none() as u8).cmp(&(b.priority.is_none() as u8));
+            let pri = match (a.priority, b.priority) {
+                (Some(x), Some(y)) if sort_desc => y.cmp(&x),
+                (Some(x), Some(y)) => x.cmp(&y),
+                _ => Ordering::Equal,
+            };
+            nulls.then(pri).then_with(|| tail(a, b))
+        }),
+        Some(AlertSortField::Name) => list.sort_by(|a, b| {
+            let name = if sort_desc {
+                b.name.cmp(&a.name)
+            } else {
+                a.name.cmp(&b.name)
+            };
+            name.then_with(|| a.folder_name.cmp(&b.folder_name))
+                .then_with(|| a.alert_id.to_string().cmp(&b.alert_id.to_string()))
+        }),
+        // Historical default, matching the SQL arm.
+        None => list.sort_by(|a, b| {
+            a.name
+                .cmp(&b.name)
+                .then_with(|| a.folder_name.cmp(&b.folder_name))
+        }),
+    }
+}
+
+async fn enrich_with_run_state(list: &mut [ListAlertsResponseBodyItem]) {
+    if list.is_empty() {
+        return;
+    }
+    let ids: Vec<String> = list.iter().map(|i| i.alert_id.to_string()).collect();
+    let states = match infra::table::alert_states::get_rollups(&ids).await {
+        Ok(s) => s,
+        Err(e) => {
+            log::warn!("failed to load alert run state for list: {e}");
+            return;
+        }
+    };
+    if states.is_empty() {
+        return;
+    }
+    let by_id: std::collections::HashMap<_, _> = states
+        .into_iter()
+        .map(|s| (s.alert_id.clone(), s))
+        .collect();
+    for item in list.iter_mut() {
+        if let Some(state) = by_id.get(&item.alert_id.to_string()) {
+            item.last_outcome = state.last_outcome.as_ref().map(|o| o.to_string());
+            item.last_outcome_at = state.last_outcome_at;
+            item.last_outcome_since = state.since;
+            item.level = state.level.map(|l| l.to_string());
+            item.level_since = state.level_since;
+            // §5.4: the group counts live on the rollup row and are the only
+            // source for the "N of M groups firing" chip — they are computed
+            // pre-cap, so they cannot be reconstructed by counting the
+            // retained state rows. The exactness markers ride along because
+            // exactness likewise cannot be re-derived from the counts and a
+            // mutable cap setting.
+            item.groups_observed = state.groups_observed.and_then(|n| i32::try_from(n).ok());
+            item.groups_firing = state.groups_firing.and_then(|n| i32::try_from(n).ok());
+            item.groups_observed_is_lower_bound = state.groups_observed_is_lower_bound;
+            item.groups_firing_is_lower_bound = state.groups_firing_is_lower_bound;
+        }
+    }
+}
+
 /// EnableAlert
 #[utoipa::path(
     patch,

src/api/management/src/request/mod.rs

@@ -49,6 +49,7 @@ pub mod scorers;
 pub mod service_accounts;
 pub mod service_streams;
 pub mod short_url;
+pub mod slos;
 pub mod sourcemaps;
 pub mod status;
 pub mod stream;

src/api/management/src/request/slos/mod.rs

@@ -0,0 +1,434 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! SLO CRUD (`alerts_2.md` §6b).
+//!
+//! SLOs live in **alert folders** and are authorized as `alerts`, following
+//! the `anomaly_detection` precedent. An SLO is alerting configuration and the
+//! alerts built on it are ordinary alert rows (D28), so a separate permission
+//! surface would mean granting two things to accomplish one.
+
+use axum::{
+    Json,
+    extract::{Path, Query},
+    http::StatusCode,
+    response::{IntoResponse, Response},
+};
+use config::meta::slo::{Slo, SloStatusView};
+use openobserve_api_common::extractors::Headers;
+use openobserve_core::{auth::UserEmail, slo::service as slo_service};
+use serde::{Deserialize, Serialize};
+use utoipa::ToSchema;
+
+use crate::common::meta::http::HttpResponse as MetaHttpResponse;
+
+#[derive(Debug, Default, Deserialize, ToSchema)]
+pub struct ListQuery {
+    /// Restrict to one folder. Absent lists every folder in the org.
+    pub folder: Option<String>,
+}
+
+/// An SLO plus its current measurement, which is what a list view needs.
+#[derive(Debug, Serialize, ToSchema)]
+pub struct SloListItem {
+    #[serde(flatten)]
+    pub slo: Slo,
+    /// `None` until the first pass has measured anything. Deliberately not
+    /// zeroed: "not yet measured" and "measured as zero" are different, and a
+    /// UI that conflates them shows a brand-new SLO as 0% available.
+    pub status: Option<SloStatusView>,
+}
+
+#[derive(Debug, Serialize, ToSchema)]
+pub struct SloListResponse {
+    pub list: Vec<SloListItem>,
+}
+
+fn disabled() -> Option<Response> {
+    if config::get_config().slo.enabled {
+        return None;
+    }
+    Some(
+        MetaHttpResponse::error(
+            StatusCode::NOT_IMPLEMENTED.as_u16(),
+            "SLOs are disabled. Set ZO_SLO_ENABLED=true to enable them.".to_string(),
+        )
+        .into_response(),
+    )
+}
+
+/// List SLOs in an organization.
+#[utoipa::path(
+    get,
+    path = "/{org_id}/slos",
+    context_path = "/api",
+    tag = "SLOs",
+    operation_id = "ListSlos",
+    summary = "List SLOs",
+    security(("Authorization" = [])),
+    params(
+        ("org_id" = String, Path, description = "Organization identifier"),
+        ("folder" = Option<String>, Query, description = "Filter by folder ID"),
+    ),
+    responses(
+        (status = 200, description = "Success", content_type = "application/json", body = SloListResponse),
+        (status = 500, description = "Internal Server Error", content_type = "application/json", body = MetaHttpResponse),
+    ),
+)]
+#[tracing::instrument(skip_all, fields(org_id = %org_id))]
+pub async fn list_slos(Path(org_id): Path<String>, Query(q): Query<ListQuery>) -> Response {
+    if let Some(r) = disabled() {
+        return r;
+    }
+    match openobserve_core::slo::service::list_with_status(&org_id, q.folder.as_deref()).await {
+        Ok(list) => MetaHttpResponse::json(SloListResponse {
+            list: list
+                .into_iter()
+                .map(|(slo, status)| SloListItem { slo, status })
+                .collect(),
+        }),
+        Err(e) => internal(e),
+    }
+}
+
+/// Get one SLO with its current measurement.
+#[utoipa::path(
+    get,
+    path = "/{org_id}/slos/{slo_id}",
+    context_path = "/api",
+    tag = "SLOs",
+    operation_id = "GetSlo",
+    summary = "Get an SLO",
+    security(("Authorization" = [])),
+    params(
+        ("org_id" = String, Path, description = "Organization identifier"),
+        ("slo_id" = String, Path, description = "SLO identifier"),
+    ),
+    responses(
+        (status = 200, description = "Success", content_type = "application/json", body = SloListItem),
+        (status = 404, description = "Not Found", content_type = "application/json", body = MetaHttpResponse),
+    ),
+)]
+#[tracing::instrument(skip_all, fields(org_id = %org_id, slo_id = %slo_id))]
+pub async fn get_slo(Path((org_id, slo_id)): Path<(String, String)>) -> Response {
+    if let Some(r) = disabled() {
+        return r;
+    }
+    match openobserve_core::slo::service::get_with_status(&org_id, &slo_id).await {
+        Ok(Some((slo, status))) => MetaHttpResponse::json(SloListItem { slo, status }),
+        Ok(None) => not_found(),
+        Err(e) => internal(e),
+    }
+}
+
+/// Create an SLO.
+#[utoipa::path(
+    post,
+    path = "/{org_id}/slos",
+    context_path = "/api",
+    tag = "SLOs",
+    operation_id = "CreateSlo",
+    summary = "Create an SLO",
+    security(("Authorization" = [])),
+    params(("org_id" = String, Path, description = "Organization identifier")),
+    request_body(content = Slo, description = "SLO definition", content_type = "application/json"),
+    responses(
+        (status = 200, description = "Created", content_type = "application/json", body = MetaHttpResponse),
+        (status = 400, description = "Bad Request", content_type = "application/json", body = MetaHttpResponse),
+    ),
+)]
+#[tracing::instrument(skip_all, fields(org_id = %org_id))]
+pub async fn create_slo(
+    Path(org_id): Path<String>,
+    Headers(user_email): Headers<UserEmail>,
+    Json(mut slo): Json<Slo>,
+) -> Response {
+    if let Some(r) = disabled() {
+        return r;
+    }
+    // The path segment is authoritative: it is what the permission check ran
+    // against, so a body claiming a different org must not be honoured.
+    slo.org = org_id;
+    if slo.id.is_empty() {
+        slo.id = config::ider::generate();
+    }
+    if slo.folder_id.is_empty() {
+        slo.folder_id = config::meta::folder::DEFAULT_FOLDER.to_string();
+    }
+    if slo.owner.is_none() {
+        slo.owner = Some(user_email.user_id.clone());
+    }
+
+    match slo_service::create(&mut slo).await {
+        Ok(()) => MetaHttpResponse::json(
+            MetaHttpResponse::message(StatusCode::OK, "SLO saved")
+                .with_id(slo.id.clone())
+                .with_name(slo.name.clone()),
+        ),
+        Err(e) => save_error(e),
+    }
+}
+
+/// Update an SLO.
+#[utoipa::path(
+    put,
+    path = "/{org_id}/slos/{slo_id}",
+    context_path = "/api",
+    tag = "SLOs",
+    operation_id = "UpdateSlo",
+    summary = "Update an SLO",
+    security(("Authorization" = [])),
+    params(
+        ("org_id" = String, Path, description = "Organization identifier"),
+        ("slo_id" = String, Path, description = "SLO identifier"),
+    ),
+    request_body(content = Slo, description = "SLO definition", content_type = "application/json"),
+    responses(
+        (status = 200, description = "Updated", content_type = "application/json", body = MetaHttpResponse),
+        (status = 400, description = "Bad Request", content_type = "application/json", body = MetaHttpResponse),
+    ),
+)]
+#[tracing::instrument(skip_all, fields(org_id = %org_id, slo_id = %slo_id))]
+pub async fn update_slo(
+    Path((org_id, slo_id)): Path<(String, String)>,
+    Headers(user_email): Headers<UserEmail>,
+    Json(mut slo): Json<Slo>,
+) -> Response {
+    if let Some(r) = disabled() {
+        return r;
+    }
+    // Both taken from the path, for the same reason as create: they are what
+    // the permission check ran against.
+    slo.org = org_id;
+    slo.id = slo_id;
+    if slo.owner.is_none() {
+        slo.owner = Some(user_email.user_id.clone());
+    }
+
+    match slo_service::update(&mut slo).await {
+        Ok(()) => MetaHttpResponse::json(
+            MetaHttpResponse::message(StatusCode::OK, "SLO updated")
+                .with_id(slo.id.clone())
+                .with_name(slo.name.clone()),
+        ),
+        Err(openobserve_core::slo::service::SloError::NotFound) => not_found(),
+        Err(e) => save_error(e),
+    }
+}
+
+/// Delete an SLO.
+#[utoipa::path(
+    delete,
+    path = "/{org_id}/slos/{slo_id}",
+    context_path = "/api",
+    tag = "SLOs",
+    operation_id = "DeleteSlo",
+    summary = "Delete an SLO",
+    security(("Authorization" = [])),
+    params(
+        ("org_id" = String, Path, description = "Organization identifier"),
+        ("slo_id" = String, Path, description = "SLO identifier"),
+    ),
+    responses(
+        (status = 200, description = "Deleted", content_type = "application/json", body = MetaHttpResponse),
+        (status = 404, description = "Not Found", content_type = "application/json", body = MetaHttpResponse),
+    ),
+)]
+#[tracing::instrument(skip_all, fields(org_id = %org_id, slo_id = %slo_id))]
+pub async fn delete_slo(Path((org_id, slo_id)): Path<(String, String)>) -> Response {
+    if let Some(r) = disabled() {
+        return r;
+    }
+    match slo_service::delete(&org_id, &slo_id).await {
+        Ok(true) => {
+            MetaHttpResponse::json(MetaHttpResponse::message(StatusCode::OK, "SLO deleted"))
+        }
+        Ok(false) => not_found(),
+        Err(e) => internal(anyhow::anyhow!(e.to_string())),
+    }
+}
+
+#[derive(Debug, Deserialize, ToSchema)]
+pub struct MoveSlosRequestBody {
+    /// The SLOs to relocate.
+    pub slo_ids: Vec<String>,
+    /// Destination folder. An **alert** folder — SLOs share the alert folder
+    /// namespace rather than having a type of their own.
+    pub dst_folder_id: String,
+}
+
+/// Move SLOs between folders.
+#[utoipa::path(
+    post,
+    path = "/{org_id}/slos/move",
+    context_path = "/api",
+    tag = "SLOs",
+    operation_id = "MoveSlos",
+    summary = "Move SLOs between folders",
+    description = "Relocates one or more SLOs into another folder. SLOs share the alert folder namespace, so the destination is an alert folder. A move never changes an SLO's definition and never restarts its measurement.",
+    security(("Authorization" = [])),
+    params(("org_id" = String, Path, description = "Organization identifier")),
+    request_body(content = inline(MoveSlosRequestBody), description = "The SLOs and the destination folder", content_type = "application/json"),
+    responses(
+        (status = 200, description = "Moved", content_type = "application/json", body = MetaHttpResponse),
+        (status = 404, description = "Not Found", content_type = "application/json", body = MetaHttpResponse),
+        (status = 409, description = "Name already used in the destination", content_type = "application/json", body = MetaHttpResponse),
+    ),
+    extensions(
+        ("x-o2-ratelimit" = json!({"module": "SLOs", "operation": "update"})),
+    )
+)]
+#[tracing::instrument(skip_all, fields(org_id = %org_id))]
+pub async fn move_slos(
+    Path(org_id): Path<String>,
+    Headers(user_email): Headers<UserEmail>,
+    Json(req_body): Json<MoveSlosRequestBody>,
+) -> Response {
+    if let Some(r) = disabled() {
+        return r;
+    }
+    if req_body.slo_ids.is_empty() {
+        return MetaHttpResponse::error(
+            StatusCode::BAD_REQUEST.as_u16(),
+            "no SLOs given to move".to_string(),
+        )
+        .into_response();
+    }
+    match slo_service::move_to_folder(
+        &org_id,
+        &req_body.slo_ids,
+        &req_body.dst_folder_id,
+        Some(&user_email.user_id),
+    )
+    .await
+    {
+        // Nothing matched: every id was unknown or belonged to another org.
+        // Reported rather than passed off as success, which is what a bare
+        // "moved" would do for a typo'd id.
+        Ok(0) => not_found(),
+        Ok(n) => MetaHttpResponse::json(MetaHttpResponse::message(
+            StatusCode::OK,
+            if n == 1 { "SLO moved" } else { "SLOs moved" },
+        )),
+        Err(e) => save_error(e),
+    }
+}
+
+#[derive(Debug, Deserialize, ToSchema)]
+pub struct EnableQuery {
+    pub value: bool,
+}
+
+/// Enable or pause an SLO.
+///
+/// Separate from update because pausing must never be able to change the
+/// definition — and therefore can never bump the generation or discard
+/// measurement.
+#[utoipa::path(
+    put,
+    path = "/{org_id}/slos/{slo_id}/enable",
+    context_path = "/api",
+    tag = "SLOs",
+    operation_id = "EnableSlo",
+    summary = "Enable or pause an SLO",
+    security(("Authorization" = [])),
+    params(
+        ("org_id" = String, Path, description = "Organization identifier"),
+        ("slo_id" = String, Path, description = "SLO identifier"),
+        ("value" = bool, Query, description = "true to enable, false to pause"),
+    ),
+    responses(
+        (status = 200, description = "Success", content_type = "application/json", body = MetaHttpResponse),
+        (status = 404, description = "Not Found", content_type = "application/json", body = MetaHttpResponse),
+    ),
+)]
+#[tracing::instrument(skip_all, fields(org_id = %org_id, slo_id = %slo_id))]
+pub async fn enable_slo(
+    Path((org_id, slo_id)): Path<(String, String)>,
+    Query(q): Query<EnableQuery>,
+) -> Response {
+    if let Some(r) = disabled() {
+        return r;
+    }
+    match slo_service::set_enabled(&org_id, &slo_id, q.value).await {
+        Ok(true) => MetaHttpResponse::json(MetaHttpResponse::message(
+            StatusCode::OK,
+            if q.value { "SLO enabled" } else { "SLO paused" },
+        )),
+        Ok(false) => not_found(),
+        Err(e) => internal(anyhow::anyhow!(e.to_string())),
+    }
+}
+
+/// The per-group breakdown for one SLO.
+#[utoipa::path(
+    get,
+    path = "/{org_id}/slos/{slo_id}/groups",
+    context_path = "/api",
+    tag = "SLOs",
+    operation_id = "GetSloGroups",
+    summary = "Per-group SLO status",
+    security(("Authorization" = [])),
+    params(
+        ("org_id" = String, Path, description = "Organization identifier"),
+        ("slo_id" = String, Path, description = "SLO identifier"),
+    ),
+    responses(
+        (status = 200, description = "Success", content_type = "application/json", body = Object),
+    ),
+)]
+#[tracing::instrument(skip_all, fields(org_id = %org_id, slo_id = %slo_id))]
+pub async fn get_slo_groups(Path((org_id, slo_id)): Path<(String, String)>) -> Response {
+    if let Some(r) = disabled() {
+        return r;
+    }
+    match openobserve_core::slo::service::group_status(&org_id, &slo_id).await {
+        Ok(groups) => MetaHttpResponse::json(serde_json::json!({ "list": groups })),
+        Err(e) => internal(e),
+    }
+}
+
+fn not_found() -> Response {
+    MetaHttpResponse::error(StatusCode::NOT_FOUND.as_u16(), "SLO not found".to_string())
+        .into_response()
+}
+
+fn internal(e: anyhow::Error) -> Response {
+    tracing::error!("[slo] request failed: {e}");
+    MetaHttpResponse::error(StatusCode::INTERNAL_SERVER_ERROR.as_u16(), e.to_string())
+        .into_response()
+}
+
+/// Map a save failure to a status the caller can act on.
+///
+/// A budget rejection is a 4xx carrying the arithmetic, not a 500: the user
+/// can fix it by deleting an SLO or narrowing this one, and §6b.4d requires
+/// the rejection to show its working.
+fn save_error(e: openobserve_core::slo::service::SloError) -> Response {
+    use openobserve_core::slo::service::SloError;
+    let status = match &e {
+        SloError::Validation(_) => StatusCode::BAD_REQUEST,
+        SloError::Budget(_) => StatusCode::PAYLOAD_TOO_LARGE,
+        SloError::NotFound => StatusCode::NOT_FOUND,
+        // A name clash is the user's to fix, not a server fault.
+        SloError::DuplicateName(_) | SloError::MoveNameConflict => StatusCode::CONFLICT,
+        SloError::FolderNotFound(_) => StatusCode::NOT_FOUND,
+        SloError::Db(_) => StatusCode::INTERNAL_SERVER_ERROR,
+    };
+    if status == StatusCode::INTERNAL_SERVER_ERROR {
+        tracing::error!("[slo] save failed: {e}");
+    }
+    MetaHttpResponse::error(status.as_u16(), e.to_string()).into_response()
+}

src/api/management/src/request/status/mod.rs

@@ -206,6 +206,10 @@ struct ConfigResponse<'a> {
     online_evals_enabled: bool,
     anomaly_detection_enabled: bool,
     synthetics_enabled: bool,
+    /// SLO measurement (`ZO_SLO_ENABLED`). Not enterprise-gated — the SLO APIs
+    /// answer 501 while it is off, so the UI uses this to hide the menu entry
+    /// rather than offer a page that cannot work.
+    slo_enabled: bool,
     enable_cross_linking: bool,
     show_fts_field_values: bool,
     search_inspector_enabled: bool,
@@ -364,7 +368,7 @@ pub async fn zo_config() -> impl IntoResponse {
     // Anomaly detection is on when the enterprise feature is compiled in, unless turned off at
     // runtime via O2_ANOMALY_DETECTION_DISABLED. When disabled the UI hides the anomaly tab.
     let anomaly_detection_enabled = enterprise_value!(false, !o2cfg.anomaly_detection.disabled);
-    let online_evals_enabled = enterprise_value!(false, o2cfg.common.online_evals_enabled);
+    let online_evals_enabled = enterprise_value!(false, o2cfg.llm_eval_config.enabled);
     let synthetics_enabled = enterprise_value!(false, o2cfg.synthetics.enabled);
 
     #[cfg(all(feature = "cloud", not(feature = "enterprise")))]
@@ -479,6 +483,7 @@ pub async fn zo_config() -> impl IntoResponse {
         online_evals_enabled,
         anomaly_detection_enabled,
         synthetics_enabled,
+        slo_enabled: cfg.slo.enabled,
         enable_cross_linking: cfg.common.enable_cross_linking,
         show_fts_field_values: cfg.common.show_fts_field_values,
         search_inspector_enabled: cfg.common.search_inspector_enabled,

src/api/management/src/request/synthetics/mod.rs

@@ -34,7 +34,7 @@ use crate::service::auth::{UserEmail, check_permissions};
 pub struct ListSyntheticsQuery {
     pub folder: Option<String>,
     #[serde(rename = "type")]
-    pub monitor_type: Option<config::meta::synthetics::SyntheticType>,
+    pub check_type: Option<config::meta::synthetics::SyntheticType>,
     pub enabled: Option<bool>,
     pub location: Option<String>,
     pub tag: Option<String>,
@@ -46,7 +46,7 @@ impl From<ListSyntheticsQuery> for config::meta::synthetics::ListSyntheticsParam
     fn from(q: ListSyntheticsQuery) -> Self {
         Self {
             folder_id: q.folder,
-            monitor_type: q.monitor_type,
+            check_type: q.check_type,
             enabled: q.enabled,
             location: q.location,
             tag: q.tag,
@@ -88,11 +88,11 @@ pub struct ListRunsQuery {
     context_path = "/api",
     tag = "Synthetics",
     operation_id = "ListSyntheticsRuns",
-    summary = "List runs for a monitor",
+    summary = "List runs for a check",
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
         ("start_time" = Option<i64>, Query, description = "Filter runs with scheduled_ts >= start_time (microseconds)"),
         ("end_time" = Option<i64>, Query, description = "Filter runs with scheduled_ts <= end_time (microseconds)"),
         ("page" = Option<i64>, Query, description = "Page number (0-indexed, default 0)"),
@@ -146,7 +146,7 @@ pub async fn list_runs(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
         ("run_id" = String, Path, description = "Run ID (KSUID)"),
     ),
     responses(
@@ -332,7 +332,7 @@ pub async fn job_upload(
     }
 }
 
-// ── Monitors ──────────────────────────────────────────────────────────────────
+// ── Checks ──────────────────────────────────────────────────────────────────
 
 #[utoipa::path(
     get,
@@ -345,7 +345,7 @@ pub async fn job_upload(
     params(
         ("org_id" = String, Path, description = "Organization name"),
         ("folder" = Option<String>, Query, description = "Filter by folder ID (KSUID)"),
-        ("type" = Option<String>, Query, description = "Filter by monitor type (http|browser|tcp|tls|ssh)"),
+        ("type" = Option<String>, Query, description = "Filter by check type (http|browser|tcp|tls|ssh)"),
         ("enabled" = Option<bool>, Query, description = "Filter by enabled status"),
         ("location" = Option<String>, Query, description = "Filter by location"),
         ("tag" = Option<String>, Query, description = "Filter by tag"),
@@ -428,7 +428,7 @@ pub async fn create_synthetic(
         )
         .await
         {
-            Ok(monitor) => MetaHttpResponse::json(monitor),
+            Ok(check) => MetaHttpResponse::json(check),
             Err(e) => {
                 let msg = e.to_string();
                 if msg.starts_with("validation: ") {
@@ -457,7 +457,7 @@ pub async fn create_synthetic(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
     ),
     responses(
         (status = 200, description = "Success", content_type = "application/json", body = config::meta::synthetics::Synthetic),
@@ -488,8 +488,8 @@ pub async fn get_synthetic(
             return MetaHttpResponse::forbidden("Forbidden");
         }
         match o2_enterprise::enterprise::synthetics::service::get_synthetic(&org_id, &id).await {
-            Ok(Some(monitor)) => MetaHttpResponse::json(monitor),
-            Ok(None) => MetaHttpResponse::not_found("monitor not found"),
+            Ok(Some(check)) => MetaHttpResponse::json(check),
+            Ok(None) => MetaHttpResponse::not_found("check not found"),
             Err(e) => {
                 tracing::error!("[synthetics] get_synthetic: {e}");
                 MetaHttpResponse::error(StatusCode::INTERNAL_SERVER_ERROR.as_u16(), e.to_string())
@@ -514,7 +514,7 @@ pub async fn get_synthetic(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
         ("folder" = Option<String>, Query, description = "Current folder ID of the synthetic (for RBAC)"),
     ),
     request_body(content = config::meta::synthetics::Synthetic, description = "Updated synthetic definition", content_type = "application/json"),
@@ -550,7 +550,7 @@ pub async fn update_synthetic(
         match o2_enterprise::enterprise::synthetics::service::update_synthetic(&org_id, &id, body)
             .await
         {
-            Ok(monitor) => MetaHttpResponse::json(monitor),
+            Ok(check) => MetaHttpResponse::json(check),
             Err(e) => {
                 let msg = e.to_string();
                 if msg.starts_with("validation: ") {
@@ -582,7 +582,7 @@ pub async fn update_synthetic(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
         ("folder" = Option<String>, Query, description = "Current folder ID of the synthetic (for RBAC)"),
     ),
     responses(
@@ -614,8 +614,8 @@ pub async fn delete_synthetic(
             return MetaHttpResponse::forbidden("Forbidden");
         }
         match o2_enterprise::enterprise::synthetics::service::delete_synthetic(&org_id, &id).await {
-            Ok(true) => MetaHttpResponse::ok("monitor deleted"),
-            Ok(false) => MetaHttpResponse::not_found("monitor not found"),
+            Ok(true) => MetaHttpResponse::ok("check deleted"),
+            Ok(false) => MetaHttpResponse::not_found("check not found"),
             Err(e) => {
                 tracing::error!("[synthetics] delete_synthetic: {e}");
                 MetaHttpResponse::error(StatusCode::INTERNAL_SERVER_ERROR.as_u16(), e.to_string())
@@ -662,7 +662,7 @@ pub async fn delete_synthetics_bulk(
         )
         .await
         {
-            Ok(_) => MetaHttpResponse::ok("monitors deleted"),
+            Ok(_) => MetaHttpResponse::ok("checks deleted"),
             Err(e) => {
                 tracing::error!("[synthetics] delete_synthetics_bulk: {e}");
                 MetaHttpResponse::error(StatusCode::INTERNAL_SERVER_ERROR.as_u16(), e.to_string())
@@ -732,7 +732,7 @@ pub async fn move_synthetics(
         )
         .await
         {
-            Ok(_) => MetaHttpResponse::ok("monitors moved"),
+            Ok(_) => MetaHttpResponse::ok("checks moved"),
             Err(e) => {
                 tracing::error!("[synthetics] move_synthetics: {e}");
                 MetaHttpResponse::error(StatusCode::INTERNAL_SERVER_ERROR.as_u16(), e.to_string())
@@ -757,7 +757,7 @@ pub async fn move_synthetics(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
     ),
     request_body(content = Object, description = r#"{"enabled": true}"#, content_type = "application/json"),
     responses(
@@ -799,11 +799,11 @@ pub async fn set_synthetic_enabled(
         .await
         {
             Ok(true) => MetaHttpResponse::ok(if enabled {
-                "monitor enabled"
+                "check enabled"
             } else {
-                "monitor paused"
+                "check paused"
             }),
-            Ok(false) => MetaHttpResponse::not_found("monitor not found"),
+            Ok(false) => MetaHttpResponse::not_found("check not found"),
             Err(e) => {
                 tracing::error!("[synthetics] set_synthetic_enabled: {e}");
                 MetaHttpResponse::error(StatusCode::INTERNAL_SERVER_ERROR.as_u16(), e.to_string())
@@ -828,7 +828,7 @@ pub async fn set_synthetic_enabled(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
     ),
     responses(
         (status = 202, description = "Accepted — scheduler will fire within 5 seconds"),
@@ -886,7 +886,7 @@ pub async fn run_synthetic_now(
     context_path = "/api",
     tag = "Synthetics",
     operation_id = "SyntheticsJobResolve",
-    summary = "Resolve a job — probe fetches monitor config (authenticated via o2syn_ token)",
+    summary = "Resolve a job — probe fetches check config (authenticated via o2syn_ token)",
     params(
         ("org_id" = String, Path, description = "Organization name"),
     ),
@@ -1106,20 +1106,31 @@ async fn process_ack(
         key: format!("{}/{}", resp.synthetics_name, resp.synthetics_id),
         start_time: checked_at,
         end_time: checked_at,
-        status: config::meta::self_reporting::usage::TriggerDataStatus::Completed,
+        status: config::meta::self_reporting::usage::RunOutcome::Succeeded,
         success_response: Some(status.clone()),
         error: error.clone(),
         evaluation_took_in_secs: Some(response_time_ms / 1000.0),
         ..Default::default()
     });
 
-    // Notify once per run, not once per job ack.
-    if resp.run_complete && !resp.destinations.is_empty() {
+    // Notify once per run, not once per job ack — and only when the check's own
+    // `alert_if_fails` / `cooldown_mins` settings say so. This used to fire on
+    // every completed run that had a destination, which is why `alert_if_fails:
+    // 3` alerted on the first failure and a 30-minute cooldown sent thirty
+    // notifications.
+    use o2_enterprise::enterprise::synthetics::job_api::AlertDecision;
+    let recovery = matches!(resp.alert, AlertDecision::Recovered);
+    let flaky = matches!(resp.alert, AlertDecision::Flaky);
+    // A degrading target is `warning` on every run for as long as the condition
+    // lasts, so this one is throttled by transition upstream, not by cooldown.
+    let degraded = matches!(resp.alert, AlertDecision::Degraded);
+    let should_notify = !matches!(resp.alert, AlertDecision::Silent);
+    if should_notify && !resp.destinations.is_empty() {
         let notification = openobserve_core::synthetics::CheckNotification {
             org_id: resp.org_id.clone(),
-            monitor_name: resp.synthetics_name.clone(),
-            monitor_id: resp.synthetics_id.clone(),
-            monitor_type: resp.synthetic_type.clone(),
+            check_name: resp.synthetics_name.clone(),
+            check_id: resp.synthetics_id.clone(),
+            check_type: resp.synthetic_type.clone(),
             target: resp.target.clone(),
             destinations: resp.destinations.clone(),
             run_id: resp.run_id.clone(),
@@ -1127,6 +1138,13 @@ async fn process_ack(
             job_count: resp.job_count as i64,
             error: error.clone(),
             checked_at,
+            recovery,
+            consecutive_failures: resp.consecutive_failures,
+            flaky,
+            degraded,
+            status_reason: resp.status_reason.clone(),
+            failing_locations: resp.failing_locations.clone(),
+            passing_locations: resp.passing_locations.clone(),
         };
         tokio::spawn(async move {
             openobserve_core::synthetics::notify_check_result(notification).await;

src/api/search/src/promql/mod.rs

@@ -735,8 +735,8 @@ pub async fn metadata(
     params(
         ("org_id" = String, Path, description = "Organization name"),
         ("match[]" = String, Query, description = "<series_selector>: Series selector argument that selects the series to return"),
-        ("start" = Option<String>, Query, description = "<rfc3339 | unix_timestamp>: Start timestamp"),
-        ("end" = Option<String>, Query, description = "<rfc3339 | unix_timestamp>: End timestamp"),
+        ("start" = Option<String>, Query, description = "<rfc3339 | unix_timestamp>: Start timestamp, optional, defaults to 24 hours before end"),
+        ("end" = Option<String>, Query, description = "<rfc3339 | unix_timestamp>: End timestamp, optional, defaults to now"),
     ),
     responses(
         (status = 200, description = "Success", content_type = "application/json", body = Object, example = json!({
@@ -763,7 +763,7 @@ pub async fn metadata(
     ),
     extensions(
         ("x-o2-ratelimit" = json!({"module": "Metrics", "operation": "get"})),
-        ("x-o2-mcp" = json!({"description": "Get Prometheus series, must have start and end time", "category": "metrics"}))
+        ("x-o2-mcp" = json!({"description": "Get Prometheus series, start and end time are optional (default: last 24 hours)", "category": "metrics"}))
     )
 )]
 pub async fn series_get(

src/api/search/src/traces/mod.rs

@@ -56,26 +56,6 @@ pub(crate) mod schema_compat;
 pub mod session;
 pub mod user;
 
-#[derive(Default, Clone, Debug)]
-pub(crate) struct TraceDetail {
-    pub(crate) start_time: i64,
-    pub(crate) end_time: i64,
-    pub(crate) gen_ai_usage_input_tokens: i64,
-    pub(crate) gen_ai_usage_output_tokens: i64,
-    pub(crate) gen_ai_usage_total_tokens: i64,
-    pub(crate) gen_ai_usage_cost: f64,
-    pub(crate) gen_ai_usage_cache_read_input_tokens: i64,
-    pub(crate) gen_ai_usage_cache_creation_input_tokens: i64,
-    pub(crate) gen_ai_usage_cost_cache_read_input: f64,
-    pub(crate) gen_ai_usage_cost_cache_creation_input: f64,
-    pub(crate) gen_ai_usage_cost_estimated_without_cache: f64,
-    pub(crate) gen_ai_usage_cost_cache_read_savings: f64,
-    pub(crate) gen_ai_usage_cost_net_cache_impact: f64,
-    pub(crate) error_count: i64,
-    pub(crate) user_id: Option<String>,
-    pub(crate) first_user_message: Option<String>,
-}
-
 /// TracesIngest
 #[utoipa::path(
     post,

src/api/search/src/traces/user.rs

@@ -414,7 +414,6 @@ pub async fn get_latest_users(
                 gen_ai_usage_cost: json::get_float_value(
                     item.get("gen_ai_usage_cost_details").unwrap_or_default(),
                 ),
-                ..Default::default()
             },
         );
     }
@@ -458,7 +457,13 @@ pub async fn get_latest_users(
     })
 }
 
-use super::TraceDetail;
+#[derive(Default, Clone, Debug)]
+struct TraceDetail {
+    start_time: i64,
+    end_time: i64,
+    gen_ai_usage_total_tokens: i64,
+    gen_ai_usage_cost: f64,
+}
 
 #[derive(Debug, Serialize)]
 struct UserResponseItem {

src/common/src/meta/organization.rs

@@ -285,7 +285,7 @@ pub struct TriggerStatus {
 #[derive(Serialize, Deserialize, Debug)]
 pub struct TriggerStatusSearchResult {
     pub module: usage::TriggerDataType,
-    pub status: usage::TriggerDataStatus,
+    pub status: usage::RunOutcome,
 }
 
 impl TriggerStatus {
@@ -302,10 +302,15 @@ impl TriggerStatus {
         for result in results {
             if result.module == module {
                 match result.status {
-                    usage::TriggerDataStatus::Completed
-                    | usage::TriggerDataStatus::ConditionNotSatisfied => status.healthy += 1,
-                    usage::TriggerDataStatus::Failed => status.failed += 1,
-                    usage::TriggerDataStatus::Skipped => status.warning += 1,
+                    // Health = did the evaluation run cleanly. A firing alert
+                    // is a healthy trigger; a failed delivery is not.
+                    usage::RunOutcome::Firing
+                    | usage::RunOutcome::Normal
+                    | usage::RunOutcome::Succeeded => status.healthy += 1,
+                    usage::RunOutcome::Error | usage::RunOutcome::NotifyFailed => {
+                        status.failed += 1
+                    }
+                    usage::RunOutcome::Skipped => status.warning += 1,
                 }
             }
         }
@@ -1141,11 +1146,11 @@ mod tests {
         let results = vec![
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::DerivedStream,
-                status: usage::TriggerDataStatus::Completed,
+                status: usage::RunOutcome::Firing,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::DerivedStream,
-                status: usage::TriggerDataStatus::ConditionNotSatisfied,
+                status: usage::RunOutcome::Normal,
             },
         ];
 
@@ -1162,15 +1167,15 @@ mod tests {
         let results = vec![
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Completed,
+                status: usage::RunOutcome::Firing,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Failed,
+                status: usage::RunOutcome::Error,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Failed,
+                status: usage::RunOutcome::Error,
             },
         ];
 
@@ -1186,15 +1191,15 @@ mod tests {
         let results = vec![
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::DerivedStream,
-                status: usage::TriggerDataStatus::Completed,
+                status: usage::RunOutcome::Firing,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::DerivedStream,
-                status: usage::TriggerDataStatus::Skipped,
+                status: usage::RunOutcome::Skipped,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::DerivedStream,
-                status: usage::TriggerDataStatus::Skipped,
+                status: usage::RunOutcome::Skipped,
             },
         ];
 
@@ -1211,19 +1216,19 @@ mod tests {
         let results = vec![
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Completed,
+                status: usage::RunOutcome::Firing,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Failed,
+                status: usage::RunOutcome::Error,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Skipped,
+                status: usage::RunOutcome::Skipped,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::ConditionNotSatisfied,
+                status: usage::RunOutcome::Normal,
             },
         ];
 
@@ -1251,15 +1256,15 @@ mod tests {
         let results = vec![
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::DerivedStream,
-                status: usage::TriggerDataStatus::Completed,
+                status: usage::RunOutcome::Firing,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Failed,
+                status: usage::RunOutcome::Error,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::DerivedStream,
-                status: usage::TriggerDataStatus::Failed,
+                status: usage::RunOutcome::Error,
             },
         ];
 
@@ -1276,11 +1281,11 @@ mod tests {
         let results = vec![
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Completed,
+                status: usage::RunOutcome::Firing,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Failed,
+                status: usage::RunOutcome::Error,
             },
         ];
 
@@ -1298,15 +1303,15 @@ mod tests {
         let results = vec![
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Failed,
+                status: usage::RunOutcome::Error,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Failed,
+                status: usage::RunOutcome::Error,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::Alert,
-                status: usage::TriggerDataStatus::Failed,
+                status: usage::RunOutcome::Error,
             },
         ];
 
@@ -1322,11 +1327,11 @@ mod tests {
         let results = vec![
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::DerivedStream,
-                status: usage::TriggerDataStatus::Skipped,
+                status: usage::RunOutcome::Skipped,
             },
             TriggerStatusSearchResult {
                 module: usage::TriggerDataType::DerivedStream,
-                status: usage::TriggerDataStatus::Skipped,
+                status: usage::RunOutcome::Skipped,
             },
         ];
 

src/compaction/Cargo.toml

@@ -28,11 +28,13 @@ ingester.workspace = true
 itertools.workspace = true
 log.workspace = true
 o2_enterprise = { workspace = true, optional = true, default-features = false }
+object_store.workspace = true
 parking_lot.workspace = true
 parquet.workspace = true
 rand.workspace = true
 schema.workspace = true
 search.workspace = true
 search_service.workspace = true
+tantivy.workspace = true
 tantivy_utils.workspace = true
 tokio.workspace = true

src/compaction/src/bloom/builder.rs

@@ -0,0 +1,127 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Build SBBF blooms by iterating tantivy term dictionaries.
+//!
+//! For an indexed field, tantivy already stores a deduplicated term
+//! dictionary. Iterating it is much cheaper than re-scanning the parquet
+//! column — terms come back already unique, sorted, and as raw bytes.
+//!
+//! Used by the compactor merge hook to build per-(file, field)
+//! blooms after the .ttv file is written.
+
+use std::collections::HashSet;
+
+use anyhow::Context;
+use hashbrown::HashMap;
+use infra::bloom::{BloomBuilder, FieldBloom};
+use tantivy::Index;
+use tantivy_utils::puffin_directory::reader::warm_up_terms;
+
+/// Build per-field SBBFs for one file using a group-uniform `num_blocks`.
+///
+/// Every file in a (stream, hour, bloom_ver) group must pass the same
+/// `num_blocks` so the transposed `.bf` layout can read one block-row per
+/// group (see `infra::bloom` module docs). The caller derives it once from
+/// the configured expected cardinality.
+///
+/// Behavior:
+/// - Fields not present in the schema are silently skipped — the compactor passes the union of
+///   `index_fields ∩ bloom_filter_fields` over potentially many streams, and not every field exists
+///   everywhere.
+/// - Terms across all segments of the index are merged into one bloom per field. Today
+///   `create_tantivy_index` produces a single segment, but this is robust to that changing.
+pub(super) async fn build_blooms_from_index(
+    index: &Index,
+    file_id: u64,
+    fields: &[String],
+    num_blocks: u32,
+) -> Result<Vec<FieldBloom>, anyhow::Error> {
+    if fields.is_empty() {
+        return Ok(Vec::new());
+    }
+
+    let schema = index.schema();
+    let reader = index
+        .reader_builder()
+        .reload_policy(tantivy::ReloadPolicy::Manual)
+        .num_warming_threads(0)
+        .try_into()
+        .context("open tantivy reader")?;
+    let searcher = reader.searcher();
+
+    let warm_terms: HashMap<tantivy::schema::Field, HashMap<tantivy::Term, bool>> = HashMap::new();
+    let mut need_all_term_fields = HashSet::new();
+    for field in fields {
+        let Ok(field) = schema.get_field(field) else {
+            continue;
+        };
+        need_all_term_fields.insert(field);
+    }
+
+    // warm_up_terms operates on one SegmentReader at a time; warm each segment.
+    // need_all_term_fields / need_fast_field are consumed per call, so clone them.
+    for seg in searcher.segment_readers() {
+        warm_up_terms(
+            seg,
+            &warm_terms,
+            need_all_term_fields.clone(),
+            HashSet::new(),
+        )
+        .await?;
+    }
+
+    let mut builder = BloomBuilder::new();
+
+    for field_name in fields {
+        let Ok(field) = schema.get_field(field_name) else {
+            continue;
+        };
+
+        // Skip fields with no terms in this file so they don't become an
+        // empty column in the transposed matrix.
+        let mut has_terms = false;
+        for seg in searcher.segment_readers() {
+            if let Ok(inv) = seg.inverted_index(field)
+                && inv.terms().num_terms() > 0
+            {
+                has_terms = true;
+                break;
+            }
+        }
+        if !has_terms {
+            continue;
+        }
+
+        // Uniform block count across the whole group (caller-provided).
+        let idx = builder.begin_with_blocks(file_id, field_name, num_blocks);
+
+        for seg in searcher.segment_readers() {
+            let inv = match seg.inverted_index(field) {
+                Ok(i) => i,
+                Err(_) => continue,
+            };
+            let mut stream = inv
+                .terms()
+                .stream()
+                .with_context(|| format!("stream terms for {field_name}"))?;
+            while let Some((term_bytes, _info)) = stream.next() {
+                builder.insert(idx, term_bytes);
+            }
+        }
+    }
+
+    Ok(builder.finish())
+}

src/compaction/src/bloom/compact.rs

@@ -0,0 +1,349 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Bloom filter build + orphan cleanup for the compactor.
+//!
+//! One public entry point, [`build_for_stream`], called once per hour bucket
+//! at the end of a compaction merge round. It does two things in order:
+//!
+//! 1. **Orphan cleanup.** Given the `bloom_ver` values of the files this merge round just deleted,
+//!    [`cleanup_orphan_blooms`] retires any `.bf` no longer referenced by a live row in the bucket.
+//! 2. **Build.** It queries the bucket itself (`query_for_bloom` → rows with `index_size > 0` and
+//!    `bloom_ver = 0`) and builds `.bf` coverage for those files. The caller does not pass a file
+//!    list; this module owns both the "which files" and the "how to bloom" decisions.
+//!
+//! `bloom_ver` is not append-only: a merge produces new output files with a
+//! fresh `bloom_ver` and deletes the small inputs, so an old `.bf` whose every
+//! file got merged away becomes an orphan — step 1 catches it.
+//!
+//! Failure is **always non-fatal**. A failed build leaves `bloom_ver = 0`
+//! on the affected files and the search side falls back to opening every
+//! tantivy file in the bucket. A failed cleanup leaves an orphan `.bf` for
+//! data-retention to reap. Neither affects correctness, so neither
+//! propagates to the caller as a hard error.
+
+use std::{collections::HashSet, sync::Arc};
+
+use anyhow::Context;
+use bytes::Bytes;
+use config::{
+    get_config,
+    meta::stream::{FileKey, FileListDeleted, StreamType},
+    utils::{inverted_index::to_tantivy_name, time::now_micros},
+};
+use infra::{
+    bloom::{BloomWriter, FieldBloom, path::bloom_path},
+    errors::Result,
+    file_list as infra_file_list, storage,
+};
+use tantivy::Directory;
+use tantivy_utils::puffin_directory::{
+    caching_directory::CachingDirectory, footer_cache::FooterCache, reader::PuffinDirReader,
+};
+
+use super::builder::build_blooms_from_index;
+
+/// Clean orphan `.bf`s for `orphan_blooms`, then build `.bf` coverage for the
+/// bucket's `bloom_ver = 0` files (queried internally via `query_for_bloom`).
+///
+/// `date_key` is the `YYYY/MM/DD/HH` string used in `file_list.date`.
+/// `orphan_blooms` is the `bloom_ver` values of the files the merge round just
+/// deleted — used only for cleanup. Files already carrying a non-zero
+/// `bloom_ver` are never re-stamped, so no live file migrates off a `bloom_ver`
+/// that a dumped row may still point to.
+///
+/// Returns `Ok(false)` when bloom is disabled or there is nothing to build (no
+/// target fields, nothing at `bloom_ver = 0`); `Ok(true)` when a build ran.
+/// Recoverable per-file/per-chunk errors are logged, not propagated — never
+/// surfaces a build failure to user requests.
+pub(crate) async fn build_for_stream(
+    org_id: &str,
+    stream_type: StreamType,
+    stream_name: &str,
+    date_key: &str,
+    is_incremental: bool,
+    orphan_blooms: Vec<i64>,
+) -> Result<bool> {
+    let cfg = get_config();
+    if !cfg.common.bloom_filter_enabled {
+        return Ok(false);
+    }
+
+    // clean up orphan blooms
+    if let Err(e) =
+        cleanup_orphan_blooms(org_id, stream_type, stream_name, date_key, orphan_blooms).await
+    {
+        log::warn!("[BLOOM_BUILD] cleanup orphan blooms failed: {e}");
+    }
+
+    // don't build bloom for incremental round
+    if is_incremental {
+        return Ok(false);
+    }
+
+    // Resolve target fields from current stream settings: a field is bloomed
+    // only when the operator put it in BOTH `bloom_filter_fields` and
+    // `index_fields` (bloom is built on top of the tantivy index).
+    let stream_settings = infra::schema::get_settings(org_id, stream_name, stream_type).await;
+    let bloom_filter_fields =
+        infra::schema::get_stream_setting_bloom_filter_fields(&stream_settings);
+    if bloom_filter_fields.is_empty() {
+        return Ok(false);
+    }
+    let index_fields: HashSet<String> =
+        infra::schema::get_stream_setting_index_fields(&stream_settings)
+            .into_iter()
+            .collect();
+    let target_fields: Vec<String> = bloom_filter_fields
+        .into_iter()
+        .filter(|f| index_fields.contains(f))
+        .collect();
+    if target_fields.is_empty() {
+        return Ok(false);
+    }
+
+    // Pull every file currently in this hour bucket.
+    let mut files =
+        infra_file_list::query_for_bloom(org_id, stream_type, stream_name, date_key).await?;
+    if files.is_empty() {
+        return Ok(false);
+    }
+
+    // Build blooms only for files without a `.bf` yet (bloom_ver = 0).
+    //
+    // **Sub-grouping**: a single hour can hold thousands of files. Since the
+    // transposed `.bf` holds every file's SBBF in memory before serialize
+    // (M × B × 32 bytes), one giant `.bf` would OOM the compactor at high
+    // volume. So we split the new files into chunks of at most
+    // `max_files_per_bf` and write one `.bf` per chunk, each with its own
+    // `bloom_ver`. The search side groups by `(date, bloom_ver)`, so it
+    // naturally reads one block-row per chunk. Total read bytes stay
+    // `M × 32`; only the request count grows to `ceil(M / max_files_per_bf)`.
+    //
+    // Files are sorted by record count first so similar-sized files share a
+    // chunk — minimizes the uniform-`B` padding waste (B is sized to each
+    // chunk's max cardinality).
+    let fpp = cfg.common.bloom_filter_fpp;
+    let max_files_per_bf = cfg.common.bloom_filter_max_files_per_bf.max(1);
+    files.sort_by_key(|k| std::cmp::Reverse(k.meta.records));
+
+    let base_ver = now_micros();
+    let chunk_total = files.len().div_ceil(max_files_per_bf);
+    for (chunk_idx, chunk) in files.chunks(max_files_per_bf).enumerate() {
+        let bloom_ver = base_ver + chunk_idx as i64;
+        let bloom_path = bloom_path(org_id, stream_type, stream_name, date_key, bloom_ver);
+        match build_for_chunk(org_id, bloom_ver, &bloom_path, chunk, &target_fields, fpp).await {
+            Ok((took, num_blocks, contributing_ids)) => {
+                log::info!(
+                    "[BLOOM_BUILD] {bloom_path}: wrote chunk {}/{chunk_total}, num_blocks={num_blocks} covering {contributing_ids} files in {took} ms",
+                    chunk_idx + 1,
+                );
+            }
+            Err(e) => {
+                log::warn!("[BLOOM_BUILD] {bloom_path}: build chunk failed: {e}");
+            }
+        }
+    }
+
+    Ok(true)
+}
+
+// B for this chunk is sized from the chunk's MAX record count, used as
+// a safe NDV upper bound: a file can't hold more distinct values than
+// rows. This never under-sizes (no saturation) for the target
+// high-cardinality fields where distinct ≈ rows. It over-sizes for
+// fields that repeat heavily (distinct ≪ rows) — acceptable; exact
+// sizing would read each file's tantivy term count in a pre-pass.
+async fn build_for_chunk(
+    org_id: &str,
+    bloom_ver: i64,
+    bf_path: &str,
+    files: &[FileKey],
+    target_fields: &[String],
+    fpp: f64,
+) -> Result<(u64, u32, usize)> {
+    let start = std::time::Instant::now();
+
+    let max_records = files
+        .iter()
+        .map(|f| f.meta.records.max(0) as u64)
+        .max()
+        .unwrap_or(0)
+        .max(1);
+    let num_blocks = infra::bloom::num_blocks_for(max_records, fpp);
+
+    let mut all_blooms: Vec<FieldBloom> = Vec::new();
+    let mut contributing_ids: Vec<i64> = Vec::new();
+    for f in files {
+        match build_for_file(f, target_fields, num_blocks).await {
+            Ok(mut blooms) if !blooms.is_empty() => {
+                all_blooms.append(&mut blooms);
+                contributing_ids.push(f.id);
+            }
+            Ok(_) => {} // no blooms (no index / field absent) — leave at 0
+            Err(e) => {
+                log::warn!(
+                    "[BLOOM_BUILD] {bf_path}: skipping {} (bloom build failed): {e}",
+                    f.key
+                );
+            }
+        }
+    }
+    if all_blooms.is_empty() {
+        return Ok((0, 0, 0));
+    }
+
+    // Distinct bloom_ver per chunk (base + idx) → distinct `.bf` path.
+    let blob: Vec<u8> = BloomWriter::serialize(all_blooms).context("serialize blooms")?;
+    let bf_account = storage::get_account(org_id, bf_path).unwrap_or_default();
+    storage::put(&bf_account, bf_path, Bytes::from(blob))
+        .await
+        .context("upload blooms")?;
+
+    debug_assert!(
+        contributing_ids.iter().all(|id| *id > 0),
+        "bloom builder must only stamp file_list rows with assigned ids"
+    );
+    infra_file_list::update_bloom_ver(&contributing_ids, bloom_ver)
+        .await
+        .context("update bloom ver")?;
+
+    let took = start.elapsed().as_millis() as u64;
+
+    Ok((took, num_blocks, contributing_ids.len()))
+}
+
+async fn build_for_file(
+    file: &FileKey,
+    target_fields: &[String],
+    num_blocks: u32,
+) -> anyhow::Result<Vec<FieldBloom>> {
+    let Some(ttv_file_name) = to_tantivy_name(&file.key) else {
+        return Ok(Vec::new()); // not an indexable file
+    };
+    if file.meta.index_size == 0 {
+        return Ok(Vec::new()); // no .ttv was emitted
+    }
+    let file_account = file.account.clone();
+    let puffin_dir =
+        Arc::new(get_tantivy_directory(&file_account, &ttv_file_name, file.meta.index_size).await?);
+    let footer_cache = FooterCache::from_directory(puffin_dir.clone(), &ttv_file_name).await?;
+    let cache_dir = CachingDirectory::new_with_cacher(puffin_dir, Arc::new(footer_cache));
+    let reader_directory: Box<dyn Directory> = Box::new(cache_dir);
+    let index = tantivy::Index::open(reader_directory).context("open index")?;
+
+    // file.id is the file_list row id, assigned by the INSERT that
+    // happened in `write_file_list` before this build runs. Always > 0
+    // by the time we get here.
+    let file_id = file.id as u64;
+    build_blooms_from_index(&index, file_id, target_fields, num_blocks).await
+}
+
+/// Open a tantivy puffin directory for `file_name` in `file_account`.
+///
+/// Mirrors the search-side `get_tantivy_directory` helper: builds a synthetic
+/// `ObjectMeta` (the `.ttv` is immutable, so a fixed `last_modified` is fine)
+/// and opens a streaming puffin reader over it.
+async fn get_tantivy_directory(
+    file_account: &str,
+    file_name: &str,
+    file_size: i64,
+) -> anyhow::Result<PuffinDirReader> {
+    let file_account = file_account.to_string();
+    let source = object_store::ObjectMeta {
+        location: file_name.into(),
+        last_modified: *config::utils::time::BASE_TIME,
+        size: file_size as u64,
+        e_tag: None,
+        version: None,
+    };
+    Ok(PuffinDirReader::from_path(file_account, source).await?)
+}
+
+/// Retire any `.bf` whose `bloom_ver` is no longer referenced by a live file
+/// in this (stream, `date_key`) bucket.
+///
+/// `deleted_bloom_vers` is the set of `bloom_ver` values carried by files a
+/// merge round just deleted (the caller dedups / drops the 0 sentinel, but we
+/// guard again here). For each, we probe `file_list` for any surviving row at
+/// that version; if none, the `.bf` is enqueued for deletion via the existing
+/// `file_list_deleted` queue.
+///
+/// Orphan detection is a **cleanup optimization, not correctness**: if the
+/// EXISTS probe or the enqueue fails, we log and move on — the stale `.bf`
+/// stays until data-retention reaps the day's subtree. Errors never
+/// propagate.
+async fn cleanup_orphan_blooms(
+    org_id: &str,
+    stream_type: StreamType,
+    stream_name: &str,
+    date_key: &str,
+    orphan_blooms: Vec<i64>,
+) -> Result<()> {
+    if orphan_blooms.is_empty() {
+        return Ok(());
+    }
+    let candidates: HashSet<i64> = orphan_blooms.into_iter().collect();
+
+    let mut orphans: Vec<FileListDeleted> = Vec::new();
+    for v_old in candidates {
+        match infra_file_list::bloom_ver_referenced(
+            org_id,
+            stream_type,
+            stream_name,
+            date_key,
+            v_old,
+        )
+        .await
+        {
+            Ok(true) => {} // still referenced by a live file — keep the `.bf`
+            Ok(false) => {
+                let path = bloom_path(org_id, stream_type, stream_name, date_key, v_old);
+                let account = storage::get_account(org_id, &path).unwrap_or_default();
+                orphans.push(FileListDeleted {
+                    id: 0,
+                    account,
+                    file: path,
+                    index_file: false, // a `.bf` has no companion `.ttv`
+                    flattened: false,
+                });
+            }
+            Err(e) => {
+                log::warn!(
+                    "[BLOOM_CLEANUP] {org_id}/{stream_type}/{stream_name}/{date_key}: \
+                     bloom_ver_referenced({v_old}) failed: {e}; leaving `.bf` for retention"
+                );
+            }
+        }
+    }
+    if orphans.is_empty() {
+        return Ok(());
+    }
+
+    let created_at = now_micros();
+    let n = orphans.len();
+    if let Err(e) = infra_file_list::batch_add_deleted(org_id, created_at, &orphans).await {
+        log::warn!(
+            "[BLOOM_CLEANUP] {org_id}/{stream_type}/{stream_name}/{date_key}: \
+             enqueue {n} orphan `.bf` for deletion failed: {e}"
+        );
+    } else {
+        log::info!(
+            "[BLOOM_CLEANUP] {org_id}/{stream_type}/{stream_name}/{date_key}: \
+             enqueued {n} orphan `.bf` for deletion"
+        );
+    }
+    Ok(())
+}

src/compaction/src/bloom/mod.rs

@@ -13,14 +13,14 @@
 // You should have received a copy of the GNU Affero General Public License
 // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-//! This module contains models that can be serialized and deserialized as JSON
-//! for HTTP responses and requests.
+//! Bloom-filter build side for compaction.
+//!
+//! - [`builder`] extracts per-(file, field) SBBFs from a tantivy term dictionary.
+//! - [`compact`] is the compactor entry point that owns "which files to bloom" and writes the
+//!   transposed `.bf` for each hour bucket.
+//!
+//! The search-side bloom pruner lives in the search crate, while the underlying
+//! SBBF format and reader/writer live in `infra::bloom`.
 
-pub use openobserve_api_management::models::action;
-#[cfg(feature = "enterprise")]
-pub use openobserve_api_management::models::ai;
-#[cfg(feature = "cloud")]
-pub use openobserve_api_management::models::billings;
-#[cfg(feature = "enterprise")]
-pub use openobserve_api_management::models::{eval_jobs, providers, score_configs, scorers};
-pub use openobserve_api_pipelines::models::pipelines;
+mod builder;
+pub(crate) mod compact;

src/compaction/src/lib.rs

@@ -32,6 +32,7 @@ use infra::{
 use o2_enterprise::enterprise::common::downsampling::get_matching_downsampling_rules;
 use tokio::sync::mpsc;
 
+mod bloom;
 pub mod deleted;
 pub mod dump;
 pub mod flatten;

src/compaction/src/merge.rs

@@ -620,7 +620,32 @@ pub async fn merge_by_stream(
         orphan_blooms.extend(task.await??);
     }
 
-    let _ = (is_incremental, orphan_blooms);
+    // Build bloom filters for the current hour. Failures are non-fatal: files
+    // left at bloom_ver = 0 fall back to the regular Tantivy search path.
+    let build_start = std::time::Instant::now();
+    match crate::bloom::compact::build_for_stream(
+        org_id,
+        stream_type,
+        stream_name,
+        &date_start,
+        is_incremental,
+        orphan_blooms,
+    )
+    .await
+    {
+        Ok(false) => {}
+        Ok(true) => {
+            let build_time = build_start.elapsed().as_millis();
+            log::info!(
+                "[COMPACTOR] bloom build for {org_id}/{stream_type}/{stream_name}/{date_start} took: {build_time} ms"
+            );
+        }
+        Err(e) => {
+            log::warn!(
+                "[COMPACTOR] bloom build for {org_id}/{stream_type}/{stream_name}/{date_start} failed: {e}"
+            );
+        }
+    }
 
     // update job status
     if let Err(e) = infra_file_list::set_job_done(&[job_id]).await {

src/config/Cargo.toml

@@ -35,6 +35,7 @@ datafusion.workspace = true
 dashmap.workspace = true
 dotenv_config.workspace = true
 dotenvy.workspace = true
+fastdivide.workspace = true
 faststr.workspace = true
 futures.workspace = true
 local-ip-address.workspace = true

src/config/src/config.rs

@@ -52,7 +52,7 @@ pub type RwAHashSet<K> = tokio::sync::RwLock<HashSet<K>>;
 pub type RwBTreeMap<K, V> = tokio::sync::RwLock<BTreeMap<K, V>>;
 
 // for DDL commands and migrations
-pub const DB_SCHEMA_VERSION: u64 = 55;
+pub const DB_SCHEMA_VERSION: u64 = 63;
 pub const DB_SCHEMA_KEY: &str = "/db_schema_version/";
 
 // global version variables
@@ -632,6 +632,76 @@ pub struct Config {
     pub pipeline: Pipeline,
     pub health_check: HealthCheck,
     pub enrichment_table: EnrichmentTable,
+    pub slo: Slo,
+}
+
+/// Feature 5 — SLO measurement (`alerts_2.md` §6b).
+#[derive(Debug, Serialize, EnvConfig, Default)]
+pub struct Slo {
+    // Development is in progress; the default stays false until the feature
+    // ships. The UI follows this flag rather than duplicating the decision:
+    // it is published as `slo_enabled` on /config and MainLayout.vue hides the
+    // SLO menu entry while it is off.
+    #[env_config(
+        name = "ZO_SLO_ENABLED",
+        default = false,
+        help = "Enable SLO measurement and SLO-based alerts. Set false to switch the feature off entirely: no per-SLO ingest job is scheduled, nothing is written to the slo_slices stream, and the SLO APIs answer 501."
+    )]
+    pub enabled: bool,
+    #[env_config(
+        name = "ZO_SLO_INGEST_DELAY_SECS",
+        default = 60,
+        help = "How far behind now the ingest job reads, so late-arriving data is present before a slice is measured. A slice is never measured until it is this far in the past."
+    )]
+    pub ingest_delay_secs: i64,
+    #[env_config(
+        name = "ZO_SLO_RECOMPUTE_SLICES",
+        default = 3,
+        help = "How many trailing slices each pass recomputes, to pick up data that arrived after those slices were first measured. Re-emitted rows win on revision."
+    )]
+    pub recompute_slices: i64,
+    #[env_config(
+        name = "ZO_SLO_MIN_COVERAGE",
+        default = 0.9,
+        help = "Coverage floor, 0..1. Below this the SLO reads as no-data and its alerts FREEZE rather than resolving — unmeasured time must never read as uptime."
+    )]
+    pub min_coverage: f64,
+    #[env_config(
+        name = "ZO_SLO_MAX_GROUPS",
+        default = 500,
+        help = "Hard cap on status rows per SLO. Group cardinality past this trips GroupOverflow rather than silently truncating."
+    )]
+    pub max_groups: i64,
+    #[env_config(
+        name = "ZO_SLO_MAX_SLICE_ROWS_PER_ORG",
+        default = 250000000,
+        help = "Per-org budget over logical (group, slice) rows. Bounds the SLOs x GROUPS x window product, which is indefensible even where each factor is individually fine."
+    )]
+    pub max_slice_rows_per_org: i64,
+    #[env_config(
+        name = "ZO_SLO_REVISION_HEADROOM",
+        default = 1.2,
+        help = "Multiplier pricing physical excess (late-data re-emissions) over logical rows. Values below 1.0 are clamped; it is a multiplier, not a discount."
+    )]
+    pub revision_headroom: f64,
+    #[env_config(
+        name = "ZO_SLO_RECONCILE_INTERVAL_SECS",
+        default = 3600,
+        help = "How often the running aggregate is rebuilt from the slices. This is the bound on cache drift after a crash, and is load-bearing rather than hygiene."
+    )]
+    pub reconcile_interval_secs: i64,
+    #[env_config(
+        name = "ZO_SLO_BACKFILL_CHUNK_SECS",
+        default = 86400,
+        help = "How much history one backfill chunk covers. One aggregate query per chunk produces every slice in it."
+    )]
+    pub backfill_chunk_secs: i64,
+    #[env_config(
+        name = "ZO_SLO_MAX_BURN_WINDOW_PAIRS",
+        default = 8,
+        help = "Max distinct (long, short) burn-rate window pairs precomputed per SLO per pass. Alerts share these, so the cost is per SLO, not per alert."
+    )]
+    pub max_burn_window_pairs: i64,
 }
 
 #[derive(Serialize, EnvConfig, Default)]
@@ -1507,6 +1577,18 @@ pub struct Common {
     pub dashboard_placeholder: String,
     #[env_config(name = "ZO_AGGREGATION_TOPK_ENABLED", default = true)]
     pub aggregation_topk_enabled: bool,
+    #[env_config(
+        name = "ZO_DF_USE_AGG_TOPK_HEAP",
+        default = true,
+        help = "Use the heap implementation for eligible aggregate TopK plans"
+    )]
+    pub use_agg_topk_heap: bool,
+    #[env_config(
+        name = "ZO_DF_TOPK_HEAP_MAX_LIMIT",
+        default = 500,
+        help = "Maximum aggregate TopK limit that uses the heap implementation"
+    )]
+    pub agg_topk_heap_max_limit: u64,
     #[env_config(name = "ZO_SEARCH_INSPECTOR_ENABLED", default = false)]
     pub search_inspector_enabled: bool,
     #[env_config(name = "ZO_UTF8_VIEW_ENABLED", default = true)]
@@ -1753,8 +1835,44 @@ pub struct Limit {
     pub http_slow_log_threshold: u64,
     #[env_config(name = "ZO_ALERT_SCHEDULE_INTERVAL", default = 10)] // seconds
     pub alert_schedule_interval: i64,
+    #[env_config(
+        name = "ZO_ALERT_HYBRID_COUNT_THRESHOLD",
+        default = 100,
+        help = "Count-based alerts whose row sentinel exceeds this switch to a COUNT(*) decision query plus a 100-row payload sample (alerts_2.md 4.4c). Clamped up to the 100-row floor."
+    )]
+    pub alert_hybrid_count_threshold: i64,
     #[env_config(name = "ZO_ALERT_SCHEDULE_CONCURRENCY", default = 5)]
     pub alert_schedule_concurrency: i64,
+    #[env_config(
+        name = "ZO_ALERT_MAX_GROUPS",
+        default = 500,
+        help = "Cardinality cap for multi-alerts (alerts_2.md M-6): the most per-group state rows one alert may track. Overflow is evaluated and counted but not persisted beyond the cap, and the true count is surfaced as a warning. 0 = unlimited."
+    )]
+    pub alert_max_groups: usize,
+    #[env_config(
+        name = "ZO_ALERT_GROUP_DISAPPEARANCE_K",
+        default = 3,
+        help = "A multi-alert group unseen for K x the alert's frequency is resolved to Ok (alerts_2.md M-7). Must exceed 1, or a single slow evaluation resolves every group and re-fires it on the next pass."
+    )]
+    pub alert_group_disappearance_k: i64,
+    #[env_config(
+        name = "ZO_ALERT_GROUP_REAP_GRACE_SECS",
+        default = 3600,
+        help = "How long a resolved multi-alert group's state row is retained before deletion (alerts_2.md M-7). Its transition history is kept regardless."
+    )]
+    pub alert_group_reap_grace_secs: i64,
+    #[env_config(
+        name = "ZO_ALERT_MAX_GROUP_NOTIFICATIONS_PER_EVAL",
+        default = 0,
+        help = "Cap on per-group notifications sent by one multi-alert evaluation (alerts_2.md §5.5 MN-8/D48). 0 = unlimited, which is the default because paging per group is the feature's contract and the group cap already bounds the worst case. Dispatch is worst-first, so a cap always delivers the most severe groups; anything dropped is logged."
+    )]
+    pub alert_max_group_notifications_per_eval: usize,
+    #[env_config(
+        name = "ZO_ALERT_GROUP_SWEEP_INTERVAL",
+        default = 60,
+        help = "How often the multi-alert group lifecycle sweep runs, in seconds (alerts_2.md M-7). The sweep only decides fates on elapsed time, so it need not match any alert's frequency. 0 disables it, which stops vanished groups from ever resolving or being reaped."
+    )]
+    pub alert_group_sweep_interval: u64,
     #[env_config(name = "ZO_ALERT_SCHEDULE_TIMEOUT", default = 90)] // seconds
     pub alert_schedule_timeout: i64,
     #[env_config(
@@ -1810,6 +1928,18 @@ pub struct Limit {
         help = "Max backfill jobs pulled per cycle and the backfill worker-pool size. Only used when ZO_SCHEDULER_PER_MODULE_PULLERS=true. Defaults to 1 (smallest budget) so bulk backfills never crowd out latency-sensitive modules."
     )]
     pub scheduler_backfill_concurrency: i64,
+    #[env_config(
+        name = "ZO_SCHEDULER_SLO_CONCURRENCY",
+        default = 0,
+        help = "Max SLO SLI-ingest jobs pulled per cycle and the worker-pool size. Only used when ZO_SCHEDULER_PER_MODULE_PULLERS=true. 0 inherits ZO_ALERT_SCHEDULE_CONCURRENCY."
+    )]
+    pub scheduler_slo_concurrency: i64,
+    #[env_config(
+        name = "ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY",
+        default = 1,
+        help = "Max SLO backfill jobs pulled per cycle and the SLO backfill worker-pool size. Only used when ZO_SCHEDULER_PER_MODULE_PULLERS=true. Defaults to 1 so a bulk historical scan never crowds out latency-sensitive incremental SLI passes."
+    )]
+    pub scheduler_slo_backfill_concurrency: i64,
     #[env_config(
         name = "ZO_SCHEDULER_ANOMALY_CONCURRENCY",
         default = 0,
@@ -1844,6 +1974,18 @@ pub struct Limit {
         help = "Poll cadence in seconds for the backfill puller. Only used when ZO_SCHEDULER_PER_MODULE_PULLERS=true. 0 inherits ZO_ALERT_SCHEDULE_INTERVAL."
     )]
     pub scheduler_backfill_interval: i64,
+    #[env_config(
+        name = "ZO_SCHEDULER_SLO_INTERVAL",
+        default = 0, // seconds
+        help = "Poll cadence in seconds for the SLO SLI-ingest puller. Only used when ZO_SCHEDULER_PER_MODULE_PULLERS=true. 0 inherits ZO_ALERT_SCHEDULE_INTERVAL."
+    )]
+    pub scheduler_slo_interval: i64,
+    #[env_config(
+        name = "ZO_SCHEDULER_SLO_BACKFILL_INTERVAL",
+        default = 0, // seconds
+        help = "Poll cadence in seconds for the SLO backfill puller. Only used when ZO_SCHEDULER_PER_MODULE_PULLERS=true. 0 inherits ZO_ALERT_SCHEDULE_INTERVAL."
+    )]
+    pub scheduler_slo_backfill_interval: i64,
     #[env_config(
         name = "ZO_SCHEDULER_ANOMALY_INTERVAL",
         default = 0, // seconds
@@ -3796,6 +3938,18 @@ pub fn ensure_not_empty(s: &str, name: &str) -> Result<(), anyhow::Error> {
 
 #[cfg(test)]
 mod tests {
+    /// Every `#[env_config]` default must parse.
+    ///
+    /// The macro reads defaults from a **string literal**, so a Rust digit
+    /// separator (`86_400`) is not a number to it — it is a `ParseIntError`
+    /// raised inside `init()`, which panics the process at startup rather
+    /// than failing anything reviewable. This test is the only cheap guard:
+    /// it forces every default through the same parse the binary does.
+    #[test]
+    fn every_env_config_default_parses() {
+        let _ = super::Config::init().expect("a default failed to parse");
+    }
+
     use super::*;
 
     #[test]

src/config/src/meta/alerts/aggregation_level.rs

@@ -0,0 +1,925 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Multi-level thresholds for AGGREGATION alerts — `alerts_2.md` §4.4.
+//!
+//! Aggregation alerts do NOT use `TriggerCondition.threshold`. Their critical
+//! threshold is `Aggregation.having.value` — a `serde_json::Value`, so it may
+//! be an int, a float, or a JSON string holding a number. The warning
+//! counterpart is `Aggregation.warning_value` (already f64).
+//!
+//! Classification funnels through `level::evaluate_level_values`, the same core
+//! the count path uses, so the two cannot disagree about precedence or
+//! boundaries.
+
+use serde_json::Value;
+
+use super::{
+    Aggregation, Operator,
+    level::{AlertLevel, compare, evaluate_level_values},
+};
+
+/// Why an aggregation's threshold pair could not be used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum AggThresholdError {
+    /// `having.value` is not a number and not a numeric string.
+    ///
+    /// Deliberately an error rather than a default: coercing to `0.0` would
+    /// make every group critical under `>`.
+    NonNumericCritical,
+    /// The warning value is not strictly less severe than critical, given the
+    /// operator's direction.
+    WarningNotLessSevere,
+    /// `having.operator` has no severity ordering, so a warning value is
+    /// meaningless.
+    OperatorNotOrderable,
+}
+
+impl std::fmt::Display for AggThresholdError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::NonNumericCritical => {
+                f.write_str("aggregation threshold (having.value) is not numeric")
+            }
+            Self::WarningNotLessSevere => f.write_str(
+                "aggregation warning value must be less severe than the having threshold",
+            ),
+            Self::OperatorNotOrderable => f.write_str(
+                "the aggregation operator has no severity ordering; a warning value is not supported",
+            ),
+        }
+    }
+}
+
+impl std::error::Error for AggThresholdError {}
+
+/// Coerce an untyped condition value to f64.
+///
+/// Accepts JSON numbers and numeric strings — the UI has historically submitted
+/// condition values as strings. Everything else is rejected.
+fn as_f64(v: &Value) -> Option<f64> {
+    match v {
+        Value::Number(n) => n.as_f64(),
+        Value::String(s) => s.trim().parse::<f64>().ok(),
+        _ => None,
+    }
+}
+
+/// Extract `(critical, warning)` for an aggregation alert.
+pub fn aggregation_thresholds(agg: &Aggregation) -> Result<(f64, Option<f64>), AggThresholdError> {
+    let critical = as_f64(&agg.having.value).ok_or(AggThresholdError::NonNumericCritical)?;
+    Ok((critical, agg.warning_value))
+}
+
+/// Classify one aggregate value against an aggregation's thresholds.
+///
+/// Shares `evaluate_level_values` with the count path — see
+/// `test_paths_agree_on_identical_inputs`.
+pub fn evaluate_aggregation_level(
+    actual: f64,
+    agg: &Aggregation,
+) -> Result<Option<AlertLevel>, AggThresholdError> {
+    let (critical, warning) = aggregation_thresholds(agg)?;
+    Ok(evaluate_level_values(
+        actual,
+        agg.having.operator,
+        critical,
+        warning,
+    ))
+}
+
+/// Value the SQL `HAVING` clause should filter on (§4.4 option B).
+///
+/// The clause widens to the **less severe** threshold so every group that could
+/// be warning-or-worse is returned, and Rust then classifies each one. Filtering
+/// on the critical threshold instead would drop the entire warning band inside
+/// the database, where no amount of downstream logic could recover it.
+///
+/// One query per evaluation, and classification stays in the shared helper.
+pub fn having_filter_value(agg: &Aggregation) -> Result<f64, AggThresholdError> {
+    let (critical, warning) = aggregation_thresholds(agg)?;
+    Ok(widened_threshold(agg.having.operator, critical, warning))
+}
+
+/// Widened comparison value for the "one query, both bands" strategy.
+///
+/// Returns the LESS severe of the two thresholds, so a single query/filter
+/// admits every item that could be warning-or-worse. Shared by the SQL
+/// `HAVING` clause and the PromQL expression, which face the same problem.
+pub fn widened_threshold(op: Operator, critical: f64, warning: Option<f64>) -> f64 {
+    let Some(w) = warning else { return critical };
+    match op {
+        // Larger is more severe -> the smaller value is the wider net.
+        Operator::GreaterThan | Operator::GreaterThanEquals => critical.min(w),
+        // Smaller is more severe -> the larger value is the wider net.
+        Operator::LessThan | Operator::LessThanEquals => critical.max(w),
+        // Unorderable operators cannot carry a warning (validation rejects
+        // them); fall back to critical rather than guessing.
+        _ => critical,
+    }
+}
+
+/// The two-axis evaluator shared by every alert type whose threshold is
+/// applied per ITEM and then gated by an item COUNT.
+///
+/// Both aggregation alerts (per group, then group count) and PromQL alerts
+/// (per series, then series count) have this shape:
+///
+/// 1. each item's value is classified against critical / warning
+/// 2. the alert fires only if enough items matched, per `TriggerCondition`
+///
+/// Counts are tracked separately per level so an item in the warning band
+/// cannot inflate the critical count. With `warning = None` this reduces
+/// exactly to the legacy behaviour (G5).
+pub fn evaluate_level_over_items(
+    item_values: &[f64],
+    op: Operator,
+    critical: f64,
+    warning: Option<f64>,
+    tc: &crate::meta::alerts::TriggerCondition,
+) -> Option<AlertLevel> {
+    let mut critical_items = 0i64;
+    let mut firing_items = 0i64; // warning-or-worse
+
+    for v in item_values {
+        match evaluate_level_values(*v, op, critical, warning) {
+            Some(AlertLevel::Critical) => {
+                critical_items += 1;
+                firing_items += 1;
+            }
+            Some(AlertLevel::Warning) => firing_items += 1,
+            _ => {}
+        }
+    }
+
+    if compare(critical_items as f64, tc.operator, tc.threshold as f64) {
+        return Some(AlertLevel::Critical);
+    }
+    if warning.is_some() || tc.warning_threshold.is_some() {
+        let count_threshold = tc.warning_threshold.unwrap_or(tc.threshold);
+        if compare(firing_items as f64, tc.operator, count_threshold as f64) {
+            return Some(AlertLevel::Warning);
+        }
+    }
+    None
+}
+
+/// Aggregation wrapper over [`evaluate_level_over_items`]: per-group value
+/// classification plus the group-count threshold.
+pub fn evaluate_aggregation_alert(
+    group_values: &[f64],
+    agg: &Aggregation,
+    tc: &crate::meta::alerts::TriggerCondition,
+) -> Result<Option<AlertLevel>, AggThresholdError> {
+    let (critical, warning) = aggregation_thresholds(agg)?;
+    Ok(evaluate_level_over_items(
+        group_values,
+        agg.having.operator,
+        critical,
+        warning,
+        tc,
+    ))
+}
+
+/// `ORDER BY` fragment that sorts groups **worst-first** for a multi-alert's
+/// bounded fetch (`alerts_2.md` §5.3).
+///
+/// The multi path drops the `HAVING` filter so healthy groups come back too
+/// (otherwise a recovering group is indistinguishable from a vanished one), but
+/// it still reads a bounded page. Ordering is what makes that page usable: the
+/// worst groups are provably inside it, so the rollup level is always exact and
+/// the M-6 cap admits the true top of the distribution.
+///
+/// Buckets by **severity band**, not by raw value. Ordering by the aggregate
+/// itself would retain "most extreme within a band", so ordinary jitter between
+/// two equally-Critical groups would churn the retained row set every
+/// evaluation — the instability `classify_groups`' `(severity_rank desc,
+/// group_key asc)` admission contract exists to avoid. Callers append the
+/// `group_by` columns as a deterministic tiebreak.
+///
+/// Requires an orderable operator; `=`/`!=` have no worst-first direction,
+/// which is why M-10 refuses them for multi-alerts.
+pub fn severity_order_sql(
+    agg: &Aggregation,
+    value_alias: &str,
+) -> Result<String, AggThresholdError> {
+    let (critical, warning) = aggregation_thresholds(agg)?;
+    let op = match agg.having.operator {
+        Operator::GreaterThan => ">",
+        Operator::GreaterThanEquals => ">=",
+        Operator::LessThan => "<",
+        Operator::LessThanEquals => "<=",
+        _ => return Err(AggThresholdError::OperatorNotOrderable),
+    };
+
+    let mut sql = format!("CASE WHEN \"{value_alias}\" {op} {critical} THEN 2");
+    if let Some(w) = warning {
+        sql.push_str(&format!(" WHEN \"{value_alias}\" {op} {w} THEN 1"));
+    }
+    sql.push_str(" ELSE 0 END DESC");
+    Ok(sql)
+}
+
+/// Validate an aggregation's threshold pair (§4.5, applied to
+/// `having.operator`).
+pub fn validate_aggregation_thresholds(agg: &Aggregation) -> Result<(), AggThresholdError> {
+    let (critical, warning) = aggregation_thresholds(agg)?;
+    let Some(warning) = warning else {
+        // Single-level aggregations stay valid for every operator (G5).
+        return Ok(());
+    };
+    match agg.having.operator {
+        Operator::GreaterThan | Operator::GreaterThanEquals if warning < critical => Ok(()),
+        Operator::LessThan | Operator::LessThanEquals if warning > critical => Ok(()),
+        Operator::GreaterThan
+        | Operator::GreaterThanEquals
+        | Operator::LessThan
+        | Operator::LessThanEquals => Err(AggThresholdError::WarningNotLessSevere),
+        _ => Err(AggThresholdError::OperatorNotOrderable),
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use serde_json::json;
+
+    use crate::meta::alerts::{
+        AggFunction, Aggregation, Condition, Operator, TriggerCondition,
+        aggregation_level::{
+            AggThresholdError, aggregation_thresholds, evaluate_aggregation_alert,
+            evaluate_aggregation_level, evaluate_level_over_items, having_filter_value,
+            severity_order_sql, validate_aggregation_thresholds,
+        },
+        level::{AlertLevel, evaluate_level_values},
+    };
+
+    fn having(op: Operator, value: serde_json::Value) -> Condition {
+        Condition {
+            column: "alert_agg_value".to_string(),
+            operator: op,
+            value,
+            ignore_case: false,
+        }
+    }
+
+    fn agg(op: Operator, critical: serde_json::Value, warning: Option<f64>) -> Aggregation {
+        Aggregation {
+            group_by: None,
+            function: AggFunction::Avg,
+            having: having(op, critical),
+            warning_value: warning,
+            multi_alert: false,
+        }
+    }
+
+    // ── having.value extraction ─────────────────────────────────────────────
+    // `Value` is untyped, so every shape a user or an older payload can produce
+    // must be handled explicitly. Silent coercion is the dangerous failure:
+    // a non-numeric threshold read as 0.0 would make EVERY group critical
+    // under `>`.
+
+    #[test]
+    fn test_extracts_integer_threshold() {
+        let a = agg(Operator::GreaterThan, json!(100), None);
+        assert_eq!(aggregation_thresholds(&a).unwrap(), (100.0, None));
+    }
+
+    #[test]
+    fn test_extracts_float_threshold() {
+        let a = agg(Operator::GreaterThan, json!(99.5), None);
+        assert_eq!(aggregation_thresholds(&a).unwrap(), (99.5, None));
+    }
+
+    /// The UI has historically submitted numeric condition values as strings.
+    #[test]
+    fn test_extracts_numeric_string_threshold() {
+        let a = agg(Operator::GreaterThan, json!("100"), None);
+        assert_eq!(aggregation_thresholds(&a).unwrap(), (100.0, None));
+
+        let a = agg(Operator::GreaterThan, json!("99.5"), None);
+        assert_eq!(aggregation_thresholds(&a).unwrap(), (99.5, None));
+    }
+
+    #[test]
+    fn test_extracts_negative_threshold() {
+        let a = agg(Operator::LessThan, json!(-5.5), None);
+        assert_eq!(aggregation_thresholds(&a).unwrap(), (-5.5, None));
+    }
+
+    #[test]
+    fn test_non_numeric_threshold_is_an_error_not_zero() {
+        for bad in [
+            json!("abc"),
+            json!(null),
+            json!(true),
+            json!([1]),
+            json!({}),
+        ] {
+            let a = agg(Operator::GreaterThan, bad.clone(), None);
+            assert_eq!(
+                aggregation_thresholds(&a),
+                Err(AggThresholdError::NonNumericCritical),
+                "threshold {bad} must fail loudly; coercing to 0.0 would make \
+                 every group critical under `>`"
+            );
+        }
+    }
+
+    #[test]
+    fn test_warning_value_is_carried_through() {
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        assert_eq!(aggregation_thresholds(&a).unwrap(), (100.0, Some(50.0)));
+    }
+
+    #[test]
+    fn test_absent_warning_value_means_single_level() {
+        let a = agg(Operator::GreaterThan, json!(100), None);
+        assert_eq!(aggregation_thresholds(&a).unwrap().1, None);
+    }
+
+    /// Older stored aggregations have no `warning_value` key at all.
+    #[test]
+    fn test_aggregation_deserializes_without_warning_value() {
+        let raw = json!({
+            "group_by": ["host"],
+            "function": "avg",
+            "having": { "column": "alert_agg_value", "operator": ">", "value": 100 }
+        });
+        let a: Aggregation = serde_json::from_value(raw).unwrap();
+        assert_eq!(a.warning_value, None, "absent = single-level, not an error");
+    }
+
+    // ── Classification ──────────────────────────────────────────────────────
+
+    #[test]
+    fn test_critical_takes_precedence_for_aggregations() {
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        assert_eq!(
+            evaluate_aggregation_level(150.0, &a).unwrap(),
+            Some(AlertLevel::Critical)
+        );
+    }
+
+    #[test]
+    fn test_warning_band_for_aggregations() {
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        assert_eq!(
+            evaluate_aggregation_level(75.0, &a).unwrap(),
+            Some(AlertLevel::Warning)
+        );
+    }
+
+    #[test]
+    fn test_no_match_for_aggregations() {
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        assert_eq!(evaluate_aggregation_level(10.0, &a).unwrap(), None);
+    }
+
+    #[test]
+    fn test_fractional_aggregate_values() {
+        // The whole reason aggregation thresholds are f64: averages are not
+        // integers.
+        let a = agg(Operator::GreaterThanEquals, json!(99.5), Some(50.25));
+        assert_eq!(
+            evaluate_aggregation_level(99.5, &a).unwrap(),
+            Some(AlertLevel::Critical)
+        );
+        assert_eq!(
+            evaluate_aggregation_level(50.25, &a).unwrap(),
+            Some(AlertLevel::Warning)
+        );
+        assert_eq!(evaluate_aggregation_level(50.24, &a).unwrap(), None);
+    }
+
+    #[test]
+    fn test_less_than_direction_for_aggregations() {
+        // For `<`, critical is the SMALLER number.
+        let a = agg(Operator::LessThan, json!(10.0), Some(25.0));
+        assert_eq!(
+            evaluate_aggregation_level(5.0, &a).unwrap(),
+            Some(AlertLevel::Critical)
+        );
+        assert_eq!(
+            evaluate_aggregation_level(20.0, &a).unwrap(),
+            Some(AlertLevel::Warning)
+        );
+        assert_eq!(evaluate_aggregation_level(30.0, &a).unwrap(), None);
+    }
+
+    #[test]
+    fn test_classification_propagates_extraction_errors() {
+        let a = agg(Operator::GreaterThan, json!("nonsense"), None);
+        assert_eq!(
+            evaluate_aggregation_level(1.0, &a),
+            Err(AggThresholdError::NonNumericCritical)
+        );
+    }
+
+    // ── The property that keeps the two paths honest ────────────────────────
+
+    /// The count path and the aggregation path must classify identical inputs
+    /// identically. §4.4 warns that a divergence here is silent — both paths
+    /// keep working, they just disagree about severity.
+    #[test]
+    fn test_paths_agree_on_identical_inputs() {
+        let cases: &[(Operator, f64, Option<f64>)] = &[
+            (Operator::GreaterThan, 100.0, Some(50.0)),
+            (Operator::GreaterThanEquals, 100.0, Some(50.0)),
+            (Operator::LessThan, 10.0, Some(25.0)),
+            (Operator::LessThanEquals, 10.0, Some(25.0)),
+            (Operator::GreaterThan, 100.0, None),
+        ];
+        let actuals = [0.0, 9.9, 10.0, 25.0, 49.9, 50.0, 99.9, 100.0, 150.0];
+
+        for (op, critical, warning) in cases {
+            let a = agg(*op, json!(*critical), *warning);
+            for actual in actuals {
+                let via_agg = evaluate_aggregation_level(actual, &a).unwrap();
+                let via_values = evaluate_level_values(actual, *op, *critical, *warning);
+                assert_eq!(
+                    via_agg, via_values,
+                    "paths disagree: op={op:?} crit={critical} warn={warning:?} actual={actual}"
+                );
+            }
+        }
+    }
+
+    // ── Validation parity (§4.5 applied to having.operator) ─────────────────
+
+    #[test]
+    fn test_validation_accepts_correct_direction() {
+        assert!(
+            validate_aggregation_thresholds(&agg(Operator::GreaterThan, json!(100), Some(50.0)))
+                .is_ok()
+        );
+        assert!(
+            validate_aggregation_thresholds(&agg(Operator::LessThan, json!(10), Some(25.0)))
+                .is_ok()
+        );
+    }
+
+    #[test]
+    fn test_validation_rejects_wrong_direction() {
+        let a = agg(Operator::GreaterThan, json!(50), Some(100.0));
+        assert_eq!(
+            validate_aggregation_thresholds(&a),
+            Err(AggThresholdError::WarningNotLessSevere)
+        );
+    }
+
+    #[test]
+    fn test_validation_rejects_equal_thresholds() {
+        let a = agg(Operator::GreaterThan, json!(100), Some(100.0));
+        assert_eq!(
+            validate_aggregation_thresholds(&a),
+            Err(AggThresholdError::WarningNotLessSevere)
+        );
+    }
+
+    #[test]
+    fn test_validation_rejects_unorderable_operators() {
+        for op in [Operator::EqualTo, Operator::NotEqualTo] {
+            let a = agg(op, json!(100), Some(50.0));
+            assert_eq!(
+                validate_aggregation_thresholds(&a),
+                Err(AggThresholdError::OperatorNotOrderable),
+                "{op:?} has no severity ordering"
+            );
+        }
+    }
+
+    #[test]
+    fn test_validation_accepts_single_level_for_every_operator() {
+        // G5: aggregation alerts with no warning value stay valid regardless of
+        // operator — including the unorderable ones that are legal today.
+        for op in [
+            Operator::EqualTo,
+            Operator::NotEqualTo,
+            Operator::GreaterThan,
+            Operator::GreaterThanEquals,
+            Operator::LessThan,
+            Operator::LessThanEquals,
+        ] {
+            let a = agg(op, json!(100), None);
+            assert!(
+                validate_aggregation_thresholds(&a).is_ok(),
+                "{op:?} must stay valid without a warning value"
+            );
+        }
+    }
+
+    #[test]
+    fn test_validation_rejects_non_numeric_critical() {
+        let a = agg(Operator::GreaterThan, json!("abc"), Some(50.0));
+        assert_eq!(
+            validate_aggregation_thresholds(&a),
+            Err(AggThresholdError::NonNumericCritical)
+        );
+    }
+
+    // ── Per-group classification (feeds Feature 3, multi-alerts) ──────────────────────────
+
+    /// `HAVING` is evaluated per group in the database, so the classifier must
+    /// work over a set of per-group aggregate values.
+    #[test]
+    fn test_per_group_aggregate_values_classify_independently() {
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        let per_group = [("host-a", 150.0), ("host-b", 75.0), ("host-c", 10.0)];
+
+        let levels: Vec<_> = per_group
+            .iter()
+            .map(|(_, v)| evaluate_aggregation_level(*v, &a).unwrap())
+            .collect();
+
+        assert_eq!(levels[0], Some(AlertLevel::Critical));
+        assert_eq!(levels[1], Some(AlertLevel::Warning));
+        assert_eq!(levels[2], None);
+    }
+
+    // ── Group-count threshold (the OTHER axis) ──────────────────────────────
+    // An aggregation alert has TWO thresholds, and the UI shows both rows:
+    //   1. `having.value` / `warning_value` — the aggregate VALUE per group ("avg(latency) > 500")
+    //   2. `trigger_condition.threshold`    — how many GROUPS must match ("...for at least 3
+    //      groups")
+    // Classifying per group without re-applying (2) silently turns "3 groups"
+    // into "any group".
+
+    fn tc(op: Operator, threshold: i64, warning: Option<i64>) -> TriggerCondition {
+        TriggerCondition {
+            operator: op,
+            threshold,
+            warning_threshold: warning,
+            ..Default::default()
+        }
+    }
+
+    /// Regression: the group-count threshold must still gate firing.
+    #[test]
+    fn test_group_count_threshold_is_still_applied() {
+        // avg > 100 critical; needs >= 3 groups.
+        let a = agg(Operator::GreaterThan, json!(100), None);
+        let t = tc(Operator::GreaterThanEquals, 3, None);
+
+        // Only 2 groups over the value threshold -> must NOT fire.
+        assert_eq!(
+            evaluate_aggregation_alert(&[150.0, 200.0], &a, &t).unwrap(),
+            None,
+            "2 groups cannot satisfy a >= 3 group-count threshold"
+        );
+        // 3 groups -> fires.
+        assert_eq!(
+            evaluate_aggregation_alert(&[150.0, 200.0, 300.0], &a, &t).unwrap(),
+            Some(AlertLevel::Critical)
+        );
+    }
+
+    /// With no warning value the behaviour must be byte-identical to the
+    /// pre-multi-level implementation: count of HAVING-matching groups against
+    /// the trigger threshold (G5).
+    #[test]
+    fn test_single_level_aggregation_matches_legacy_semantics() {
+        let a = agg(Operator::GreaterThan, json!(100), None);
+        let t = tc(Operator::GreaterThanEquals, 2, None);
+
+        assert_eq!(
+            evaluate_aggregation_alert(&[10.0, 20.0], &a, &t).unwrap(),
+            None
+        );
+        assert_eq!(evaluate_aggregation_alert(&[150.0], &a, &t).unwrap(), None);
+        assert_eq!(
+            evaluate_aggregation_alert(&[150.0, 160.0], &a, &t).unwrap(),
+            Some(AlertLevel::Critical)
+        );
+    }
+
+    /// Warning counts groups that cross the WARNING value; critical counts only
+    /// those crossing the critical value. A group in the warning band must not
+    /// inflate the critical count.
+    #[test]
+    fn test_warning_and_critical_counts_are_independent() {
+        // crit avg > 100, warn avg > 50; needs >= 2 groups either way.
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        let t = tc(Operator::GreaterThanEquals, 2, None);
+
+        // Two groups in the warning band only -> Warning, not Critical.
+        assert_eq!(
+            evaluate_aggregation_alert(&[60.0, 70.0], &a, &t).unwrap(),
+            Some(AlertLevel::Warning)
+        );
+        // One critical + one warning: critical count is 1 (< 2) so it cannot be
+        // Critical; warning-or-worse count is 2 -> Warning.
+        assert_eq!(
+            evaluate_aggregation_alert(&[150.0, 60.0], &a, &t).unwrap(),
+            Some(AlertLevel::Warning)
+        );
+        // Two critical -> Critical.
+        assert_eq!(
+            evaluate_aggregation_alert(&[150.0, 160.0], &a, &t).unwrap(),
+            Some(AlertLevel::Critical)
+        );
+    }
+
+    /// A separate group-count warning threshold is honoured when present:
+    /// "critical at 5 groups, warning at 2".
+    #[test]
+    fn test_group_count_warning_threshold_is_used_when_set() {
+        let a = agg(Operator::GreaterThan, json!(100), None);
+        let t = tc(Operator::GreaterThanEquals, 5, Some(2));
+
+        assert_eq!(evaluate_aggregation_alert(&[150.0], &a, &t).unwrap(), None);
+        assert_eq!(
+            evaluate_aggregation_alert(&[150.0, 160.0], &a, &t).unwrap(),
+            Some(AlertLevel::Warning),
+            "2 groups meets the warning count but not the critical count"
+        );
+        let five = [150.0; 5];
+        assert_eq!(
+            evaluate_aggregation_alert(&five, &a, &t).unwrap(),
+            Some(AlertLevel::Critical)
+        );
+    }
+
+    #[test]
+    fn test_no_matching_groups_never_fires() {
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        let t = tc(Operator::GreaterThanEquals, 1, None);
+        assert_eq!(
+            evaluate_aggregation_alert(&[1.0, 2.0], &a, &t).unwrap(),
+            None
+        );
+        assert_eq!(evaluate_aggregation_alert(&[], &a, &t).unwrap(), None);
+    }
+
+    // ── PromQL shares the same two-axis shape ───────────────────────────────
+    // PromQL bakes the value threshold into the query itself
+    // (`(expr) > 500`), then counts matching SERIES against
+    // `trigger_condition.threshold` — structurally identical to aggregation's
+    // HAVING + group-count. It therefore uses the same generalized evaluator,
+    // so the three paths cannot drift apart.
+
+    #[test]
+    fn test_generalized_evaluator_matches_the_aggregation_wrapper() {
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        let t = tc(Operator::GreaterThanEquals, 2, None);
+        let values = [150.0, 60.0, 10.0];
+
+        assert_eq!(
+            evaluate_level_over_items(&values, Operator::GreaterThan, 100.0, Some(50.0), &t),
+            evaluate_aggregation_alert(&values, &a, &t).unwrap(),
+            "the aggregation wrapper must be a thin shim over the shared evaluator"
+        );
+    }
+
+    #[test]
+    fn test_promql_series_counting_with_two_levels() {
+        // crit value > 500, warn > 300; fire when >= 2 series match.
+        let t = tc(Operator::GreaterThanEquals, 2, None);
+
+        // Two series over the WARNING value only -> Warning.
+        assert_eq!(
+            evaluate_level_over_items(
+                &[350.0, 400.0],
+                Operator::GreaterThan,
+                500.0,
+                Some(300.0),
+                &t
+            ),
+            Some(AlertLevel::Warning)
+        );
+        // Two series over CRITICAL -> Critical.
+        assert_eq!(
+            evaluate_level_over_items(
+                &[600.0, 700.0],
+                Operator::GreaterThan,
+                500.0,
+                Some(300.0),
+                &t
+            ),
+            Some(AlertLevel::Critical)
+        );
+        // One critical + one warning: critical count is 1 (< 2) -> Warning.
+        assert_eq!(
+            evaluate_level_over_items(
+                &[600.0, 350.0],
+                Operator::GreaterThan,
+                500.0,
+                Some(300.0),
+                &t
+            ),
+            Some(AlertLevel::Warning)
+        );
+        // Below the series count -> nothing.
+        assert_eq!(
+            evaluate_level_over_items(&[600.0], Operator::GreaterThan, 500.0, Some(300.0), &t),
+            None
+        );
+    }
+
+    /// With no warning value, PromQL behaviour is unchanged: the query filters
+    /// at the critical value and the series count decides (G5).
+    #[test]
+    fn test_promql_single_level_is_unchanged() {
+        let t = tc(Operator::GreaterThanEquals, 2, None);
+        assert_eq!(
+            evaluate_level_over_items(&[600.0], Operator::GreaterThan, 500.0, None, &t),
+            None
+        );
+        assert_eq!(
+            evaluate_level_over_items(&[600.0, 700.0], Operator::GreaterThan, 500.0, None, &t),
+            Some(AlertLevel::Critical)
+        );
+    }
+
+    /// The widened filter value is the same concept for PromQL: query at the
+    /// less severe threshold so the warning band comes back.
+    #[test]
+    fn test_widened_value_for_promql_style_thresholds() {
+        use crate::meta::alerts::aggregation_level::widened_threshold;
+        // `>`: warning is smaller -> query at the warning value.
+        assert_eq!(
+            widened_threshold(Operator::GreaterThan, 500.0, Some(300.0)),
+            300.0
+        );
+        // `<`: warning is larger -> query at the warning value.
+        assert_eq!(
+            widened_threshold(Operator::LessThan, 10.0, Some(25.0)),
+            25.0
+        );
+        // Single-level: the critical value.
+        assert_eq!(widened_threshold(Operator::GreaterThan, 500.0, None), 500.0);
+    }
+
+    // ── The widening/classification contract ────────────────────────────────
+
+    /// P0 regression guard.
+    ///
+    /// Widening the HAVING clause is only safe if the caller RE-CLASSIFIES the
+    /// returned rows. The first implementation widened the SQL but still
+    /// classified `records.len()` against `TriggerCondition`, so aggregation
+    /// alerts saw a larger row set and fired MORE often than before the change
+    /// — a spurious-firing regression, not merely a missing feature.
+    ///
+    /// This pins the invariant: rows admitted by the widened filter span both
+    /// severity bands, so row *presence* cannot imply Critical.
+    #[test]
+    fn test_widened_filter_admits_rows_that_are_not_critical() {
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        let filter = having_filter_value(&a).unwrap();
+
+        // A group that passes the widened SQL filter...
+        let warning_band_value = 75.0;
+        assert!(
+            warning_band_value > filter,
+            "value must survive the widened HAVING"
+        );
+        // ...is NOT critical. Treating its presence as a firing at critical is
+        // exactly the bug.
+        assert_eq!(
+            evaluate_aggregation_level(warning_band_value, &a).unwrap(),
+            Some(AlertLevel::Warning)
+        );
+    }
+
+    /// A value below even the widened filter must classify as no-match, so a
+    /// caller that forgets to re-classify cannot accidentally look correct.
+    #[test]
+    fn test_values_below_the_widened_filter_do_not_match() {
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        let filter = having_filter_value(&a).unwrap();
+        assert_eq!(evaluate_aggregation_level(filter, &a).unwrap(), None);
+        assert_eq!(evaluate_aggregation_level(filter - 1.0, &a).unwrap(), None);
+    }
+
+    /// The rollup across per-group aggregates takes the most severe group —
+    /// and the reported value must come from THAT group, so history's
+    /// "fired at X against Y" refers to one coherent observation.
+    #[test]
+    fn test_rollup_reports_the_worst_groups_value() {
+        use crate::meta::alerts::level::AlertLevel as L;
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        let groups = [("a", 75.0), ("b", 500.0), ("c", 60.0)];
+
+        let classified: Vec<_> = groups
+            .iter()
+            .filter_map(|(g, v)| {
+                evaluate_aggregation_level(*v, &a)
+                    .unwrap()
+                    .map(|l| (*g, *v, l))
+            })
+            .collect();
+        let worst = classified
+            .iter()
+            .max_by_key(|(_, _, l)| l.severity_rank())
+            .unwrap();
+
+        assert_eq!(worst.0, "b");
+        assert_eq!(worst.1, 500.0, "the value must come from the worst group");
+        assert_eq!(worst.2, L::Critical);
+    }
+
+    /// §4.4 option B: the `HAVING` clause widens to the LESS severe threshold
+    /// so every group that could be warning-or-worse comes back, then Rust
+    /// classifies. Sizing off the critical threshold would silently drop the
+    /// whole warning band in the database.
+    #[test]
+    fn test_having_widens_to_the_less_severe_threshold() {
+        // `>`: warning is the smaller number, so filter on it.
+        let a = agg(Operator::GreaterThan, json!(100), Some(50.0));
+        assert_eq!(having_filter_value(&a).unwrap(), 50.0);
+
+        // `<`: warning is the larger number, so filter on it.
+        let a = agg(Operator::LessThan, json!(10), Some(25.0));
+        assert_eq!(having_filter_value(&a).unwrap(), 25.0);
+
+        // Single-level: the critical threshold is the filter.
+        let a = agg(Operator::GreaterThan, json!(100), None);
+        assert_eq!(having_filter_value(&a).unwrap(), 100.0);
+    }
+
+    // ── §5.3: worst-first ordering for the multi-alert fetch ────────────────
+
+    #[test]
+    fn test_severity_order_ranks_critical_above_warning_above_healthy() {
+        let a = agg(Operator::GreaterThan, json!(90), Some(80.0));
+        assert_eq!(
+            severity_order_sql(&a, "alert_agg_value").unwrap(),
+            "CASE WHEN \"alert_agg_value\" > 90 THEN 2 \
+             WHEN \"alert_agg_value\" > 80 THEN 1 ELSE 0 END DESC"
+        );
+    }
+
+    #[test]
+    fn test_severity_order_without_a_warning_band_has_two_buckets() {
+        let a = agg(Operator::GreaterThan, json!(90), None);
+        assert_eq!(
+            severity_order_sql(&a, "alert_agg_value").unwrap(),
+            "CASE WHEN \"alert_agg_value\" > 90 THEN 2 ELSE 0 END DESC"
+        );
+    }
+
+    #[test]
+    fn test_severity_order_follows_the_operator_direction() {
+        // For `<` the WORST group is the smallest, so the comparison — not the
+        // sort direction — is what flips. Emitting `ASC` on the raw value
+        // instead would put the healthiest groups first and the cap would
+        // retain exactly the wrong ones.
+        let a = agg(Operator::LessThan, json!(10), Some(20.0));
+        let sql = severity_order_sql(&a, "alert_agg_value").unwrap();
+        assert_eq!(
+            sql,
+            "CASE WHEN \"alert_agg_value\" < 10 THEN 2 \
+             WHEN \"alert_agg_value\" < 20 THEN 1 ELSE 0 END DESC"
+        );
+        assert!(
+            sql.ends_with("DESC"),
+            "severity rank always sorts descending; the operator carries the direction"
+        );
+    }
+
+    #[test]
+    fn test_severity_order_buckets_rather_than_ranking_raw_values() {
+        // Two equally-Critical groups must be interchangeable to the sort, so
+        // ordinary jitter between them cannot churn the retained row set. The
+        // proof is that the aggregate appears only inside comparisons, never as
+        // a bare sort key.
+        let a = agg(Operator::GreaterThan, json!(90), Some(80.0));
+        let sql = severity_order_sql(&a, "alert_agg_value").unwrap();
+        assert!(sql.starts_with("CASE WHEN"));
+        assert!(
+            !sql.contains("END DESC, \"alert_agg_value\""),
+            "the raw aggregate must not be a secondary sort key"
+        );
+    }
+
+    #[test]
+    fn test_severity_order_rejects_an_unorderable_operator() {
+        // `=` has no worst-first direction. M-10 refuses these for multi-alerts
+        // precisely so this is unreachable in practice — but the SQL builder
+        // must not invent an ordering if it ever is reached.
+        for op in [Operator::EqualTo, Operator::NotEqualTo] {
+            let a = agg(op, json!(90), None);
+            assert!(matches!(
+                severity_order_sql(&a, "alert_agg_value"),
+                Err(AggThresholdError::OperatorNotOrderable)
+            ));
+        }
+    }
+
+    #[test]
+    fn test_severity_order_rejects_a_non_numeric_threshold() {
+        let a = agg(Operator::GreaterThan, json!("not a number"), None);
+        assert!(severity_order_sql(&a, "alert_agg_value").is_err());
+    }
+}

src/config/src/meta/alerts/alert.rs

@@ -21,7 +21,10 @@ use utoipa::ToSchema;
 
 use crate::{
     meta::{
-        alerts::{QueryCondition, TriggerCondition, deduplication::DeduplicationConfig},
+        alerts::{
+            QueryCondition, TriggerCondition, deduplication::DeduplicationConfig,
+            priority::AlertPriority,
+        },
         stream::StreamType,
         triggers::{ScheduledTriggerData, Trigger},
     },
@@ -101,6 +104,31 @@ pub struct Alert {
     pub creates_incident: bool,
     #[serde(default)]
     pub workflows: Vec<String>,
+    /// How much humans care about this alert (PT-1). `None` = unset, which is
+    /// every pre-Feature-2 alert.
+    ///
+    /// **Mutable** configuration — editable on any update, like `name`.
+    /// Display + propagation only: it must never influence evaluation,
+    /// silence, delivery or incident severity (PT-5 / D19).
+    ///
+    /// `value_type` is required here because the enum serializes as an
+    /// integer via serde `try_from`/`into`; without it the generated OpenAPI
+    /// would advertise a string enum and lie about the payload.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[schema(value_type = Option<u8>, example = 3)]
+    pub priority: Option<AlertPriority>,
+    /// Selection tags (PT-6): bare (`prod`) or `key:value`
+    /// (`service:checkout`), normalized and validated at save by
+    /// `tags::normalize_tags`.
+    ///
+    /// NOT `context_attributes` — that field is free-form KV shipped into
+    /// notification payloads with no validation. These are the filtering /
+    /// scoping primitive.
+    ///
+    /// Skipped when empty so alerts that set no tags serialize exactly as
+    /// they did before Feature 2 (G5).
+    #[serde(default, skip_serializing_if = "Vec::is_empty")]
+    pub tags: Vec<String>,
 }
 
 impl MemorySize for Alert {
@@ -122,6 +150,8 @@ impl MemorySize for Alert {
             + self.last_edited_by.mem_size()
             + self.deduplication.mem_size()
             + self.workflows.mem_size()
+            + self.tags.mem_size()
+            + std::mem::size_of::<Option<AlertPriority>>()
     }
 }
 
@@ -160,6 +190,8 @@ impl Default for Alert {
             deduplication: None,
             creates_incident: false,
             workflows: vec![],
+            priority: None,
+            tags: vec![],
         }
     }
 }
@@ -245,6 +277,10 @@ pub enum AlertTypeFilter {
     Scheduled,
     Realtime,
     AnomalyDetection,
+    /// Feature 5 (SA-16). Filters to alerts whose `slo_id` is set — the
+    /// column, not the JSON payload, which is why it can be a SQL predicate
+    /// rather than an app-side scan (D60).
+    Slo,
 }
 
 /// Parameters for listing alerts.
@@ -277,6 +313,43 @@ pub struct ListAlertsParams {
 
     /// The optional alert type filter. Defaults to `All`.
     pub alert_type: AlertTypeFilter,
+
+    /// Optional priority filter (PT-3). Multiple values are OR-ed, so
+    /// `?priority=1&priority=2` returns P1 **or** P2.
+    ///
+    /// `None` = no filter. `Some(empty)` = the caller asked for priorities but
+    /// none were valid, which MUST match nothing — collapsing that back to
+    /// "no filter" would make `?priority=P9` return every alert, the same
+    /// match-all bug the tag filter guards against.
+    ///
+    /// Alerts with no priority are excluded whenever a filter is present:
+    /// "show me the P1s" must not surface unprioritized alerts.
+    pub priority: Option<Vec<AlertPriority>>,
+
+    /// Tag filter (PT-8), **already resolved to alert IDs** by the service
+    /// layer, which owns the in-memory alert cache the infra layer cannot
+    /// reach. `None` = no tag filter.
+    ///
+    /// `Some(empty)` means "no alert carries these tags" and MUST match
+    /// nothing — collapsing it back to `None` would turn a zero-result filter
+    /// into a match-all, the same class of bug the filter parser guards
+    /// against.
+    pub tag_alert_ids: Option<Vec<String>>,
+
+    /// Optional sort column (PT-3). `None` keeps the historical ordering
+    /// (name, then folder name).
+    pub sort_by: Option<AlertSortField>,
+
+    /// Sort direction; ignored when `sort_by` is `None`.
+    pub sort_desc: bool,
+}
+
+/// Columns the alert list can be sorted by (PT-3).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum AlertSortField {
+    /// Ascending = most urgent first, because P1 stores as 1.
+    Priority,
+    Name,
 }
 
 impl ListAlertsParams {
@@ -292,9 +365,33 @@ impl ListAlertsParams {
             owner: None,
             page_size_and_idx: None,
             alert_type: AlertTypeFilter::All,
+            priority: None,
+            tag_alert_ids: None,
+            sort_by: None,
+            sort_desc: false,
         }
     }
 
+    /// Filter by one or more priorities (OR). An empty vec means "matched
+    /// nothing", NOT "no filter" — see the field docs.
+    pub fn with_priorities(mut self, priorities: Vec<AlertPriority>) -> Self {
+        self.priority = Some(priorities);
+        self
+    }
+
+    /// Filter by a tag-resolved alert-ID set (see `tag_alert_ids`).
+    pub fn with_tag_alert_ids(mut self, ids: Vec<String>) -> Self {
+        self.tag_alert_ids = Some(ids);
+        self
+    }
+
+    /// Sort by a column. Ascending priority = most urgent first (PT-3).
+    pub fn sorted_by(mut self, field: AlertSortField, desc: bool) -> Self {
+        self.sort_by = Some(field);
+        self.sort_desc = desc;
+        self
+    }
+
     /// Filter alerts by the given folder ID surrogate key.
     pub fn in_folder(mut self, folder_id: &str) -> Self {
         self.folder_id = Some(folder_id.to_string());
@@ -741,4 +838,161 @@ mod tests {
         assert!(obj.contains_key("updated_at"));
         assert!(obj.contains_key("deduplication"));
     }
+
+    // ── Feature 2: list params (PT-3, PT-8) ─────────────────────────────────
+
+    #[test]
+    fn test_list_params_default_to_no_priority_tag_or_sort_filters() {
+        let p = ListAlertsParams::new("org");
+        assert_eq!(p.priority, None);
+        assert_eq!(p.tag_alert_ids, None, "None = no tag filter at all");
+        assert_eq!(p.sort_by, None, "None keeps the historical ordering");
+        assert!(!p.sort_desc);
+    }
+
+    #[test]
+    fn test_priority_filter_accepts_multiple_values_for_or_semantics() {
+        let p = ListAlertsParams::new("org")
+            .with_priorities(vec![AlertPriority::P1, AlertPriority::P2]);
+        assert_eq!(p.priority, Some(vec![AlertPriority::P1, AlertPriority::P2]));
+    }
+
+    /// Same distinction the tag filter needs: "no filter" and "a filter that
+    /// matched nothing" must not collapse together, or `?priority=P9` returns
+    /// every alert instead of none.
+    #[test]
+    fn test_empty_priority_set_is_distinct_from_no_priority_filter() {
+        let no_filter = ListAlertsParams::new("org");
+        assert_eq!(no_filter.priority, None);
+
+        let matched_nothing = ListAlertsParams::new("org").with_priorities(vec![]);
+        assert_eq!(matched_nothing.priority, Some(vec![]));
+        assert_ne!(no_filter.priority, matched_nothing.priority);
+    }
+
+    /// The distinction that prevents a match-all bug: "no tag filter" (`None`)
+    /// and "a tag filter that matched nothing" (`Some(vec![])`) must stay
+    /// different, or a zero-result filter silently returns every alert.
+    #[test]
+    fn test_empty_resolved_tag_set_is_distinct_from_no_tag_filter() {
+        let no_filter = ListAlertsParams::new("org");
+        assert_eq!(no_filter.tag_alert_ids, None);
+
+        let matched_nothing = ListAlertsParams::new("org").with_tag_alert_ids(vec![]);
+        assert_eq!(matched_nothing.tag_alert_ids, Some(vec![]));
+        assert_ne!(no_filter.tag_alert_ids, matched_nothing.tag_alert_ids);
+    }
+
+    #[test]
+    fn test_sort_builder_records_field_and_direction() {
+        let asc = ListAlertsParams::new("org").sorted_by(AlertSortField::Priority, false);
+        assert_eq!(asc.sort_by, Some(AlertSortField::Priority));
+        assert!(!asc.sort_desc);
+
+        let desc = ListAlertsParams::new("org").sorted_by(AlertSortField::Name, true);
+        assert_eq!(desc.sort_by, Some(AlertSortField::Name));
+        assert!(desc.sort_desc);
+    }
+
+    // ── Feature 2: priority & tags (PT-1, PT-6) ─────────────────────────────
+    // These test the PRODUCTION `Alert`, unlike the stand-in pattern test in
+    // `priority.rs` which proves only serde-attribute behaviour.
+
+    #[test]
+    fn test_alert_defaults_have_no_priority_and_no_tags() {
+        let alert = Alert::default();
+        assert_eq!(alert.priority, None, "unset is the default, never P1");
+        assert!(alert.tags.is_empty());
+    }
+
+    /// G5: an alert that configures neither field must serialize EXACTLY as it
+    /// did before Feature 2 — no new keys, so stored JSON and API payloads are
+    /// byte-identical for every existing alert.
+    #[test]
+    fn test_unset_priority_and_empty_tags_are_omitted_entirely() {
+        let alert = Alert::default();
+        let json = serde_json::to_value(&alert).unwrap();
+        let obj = json.as_object().unwrap();
+        assert!(
+            !obj.contains_key("priority"),
+            "unset priority must not appear"
+        );
+        assert!(!obj.contains_key("tags"), "empty tags must not appear");
+    }
+
+    #[test]
+    fn test_priority_and_tags_round_trip_through_serde() {
+        let alert = Alert {
+            priority: Some(AlertPriority::P2),
+            tags: vec!["prod".to_string(), "service:checkout".to_string()],
+            ..Default::default()
+        };
+        let json = serde_json::to_value(&alert).unwrap();
+        // Integer wire form (D17) — matches the storage column exactly.
+        assert_eq!(json["priority"], serde_json::json!(2));
+        assert_eq!(
+            json["tags"],
+            serde_json::json!(["prod", "service:checkout"])
+        );
+
+        let back: Alert = serde_json::from_value(json).unwrap();
+        assert_eq!(back.priority, Some(AlertPriority::P2));
+        assert_eq!(back.tags, alert.tags);
+    }
+
+    /// PT-1: priority is MUTABLE static configuration. "Static" contrasts it
+    /// with evaluated state; it does not mean write-once. An edit must be able
+    /// to raise it, lower it, and clear it back to unset.
+    #[test]
+    fn test_priority_is_mutable_including_back_to_unset() {
+        let mut alert = Alert::default();
+        alert.priority = Some(AlertPriority::P4);
+        assert_eq!(alert.priority, Some(AlertPriority::P4));
+
+        alert.priority = Some(AlertPriority::P1); // raised
+        assert_eq!(alert.priority, Some(AlertPriority::P1));
+
+        alert.priority = None; // cleared
+        let json = serde_json::to_value(&alert).unwrap();
+        assert!(
+            !json.as_object().unwrap().contains_key("priority"),
+            "clearing must return to absent, not leave a stale value"
+        );
+    }
+
+    #[test]
+    fn test_tags_are_mutable_including_back_to_empty() {
+        let mut alert = Alert::default();
+        alert.tags = vec!["prod".to_string()];
+        alert.tags.clear();
+        let json = serde_json::to_value(&alert).unwrap();
+        assert!(!json.as_object().unwrap().contains_key("tags"));
+    }
+
+    /// PT-1/PT-6: unlike the warning family (rejected on realtime by D12),
+    /// priority and tags are inert metadata and ARE allowed on realtime
+    /// alerts — excluding them would punch holes in list filtering.
+    #[test]
+    fn test_realtime_alerts_may_carry_priority_and_tags() {
+        let alert = Alert {
+            is_real_time: true,
+            priority: Some(AlertPriority::P3),
+            tags: vec!["prod".to_string()],
+            ..Default::default()
+        };
+        let json = serde_json::to_value(&alert).unwrap();
+        assert_eq!(json["priority"], serde_json::json!(3));
+        assert_eq!(json["tags"], serde_json::json!(["prod"]));
+    }
+
+    /// Old payloads (no such keys) must still deserialize — the fields are
+    /// additive.
+    #[test]
+    fn test_pre_feature2_payload_still_deserializes() {
+        let legacy = serde_json::json!({ "name": "old", "org_id": "o" });
+        let alert: Alert = serde_json::from_value(legacy).unwrap();
+        assert_eq!(alert.name, "old");
+        assert_eq!(alert.priority, None);
+        assert!(alert.tags.is_empty());
+    }
 }

src/config/src/meta/alerts/composite.rs

@@ -0,0 +1,779 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Composite alerts — Feature 4 of `alerts_2.md`.
+//!
+//! TODO(composite): the feature is deferred — this module is pure logic with
+//! tests and is deliberately not wired into the scheduler, API, or UI yet.
+//!
+//! Pure logic only: expression parsing/evaluation over child *states*, the
+//! stale-child policy (§6.4), and the write-time guards (child counts, cycles).
+//! Composites never re-run child queries — that is the whole point of building
+//! them on the durable state layer.
+
+use std::collections::HashMap;
+
+use serde::{Deserialize, Serialize};
+
+use super::level::AlertLevel;
+
+/// Maximum children per composite (C-1). Bounded so one composite cannot
+/// fan out an unbounded state read on every evaluation.
+pub const MAX_CHILDREN: usize = 10;
+/// Minimum children — a "composite" of one is just the child.
+pub const MIN_CHILDREN: usize = 2;
+/// Maximum composite nesting depth (D6).
+pub const MAX_DEPTH: usize = 2;
+/// Staleness multiplier: a child is stale after K x its own frequency (§6.4).
+pub const STALE_FREQUENCY_MULTIPLIER: i64 = 3;
+
+/// Boolean expression over child alerts.
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub enum CompositeExpr {
+    Child(String),
+    And(Box<CompositeExpr>, Box<CompositeExpr>),
+    Or(Box<CompositeExpr>, Box<CompositeExpr>),
+    Not(Box<CompositeExpr>),
+}
+
+/// The slice of a child's state a composite needs.
+#[derive(Clone, Debug, PartialEq)]
+pub struct ChildState {
+    pub level: Option<AlertLevel>,
+    /// When the level was last *computed* (`alert_states.level_at`).
+    ///
+    /// Deliberately not `last_outcome_at`: a child erroring every minute
+    /// refreshes its outcome timestamp while its level goes stale, which would
+    /// keep a long-broken child looking "fresh" to composites.
+    pub level_at: Option<i64>,
+    pub frequency_secs: i64,
+}
+
+impl ChildState {
+    fn is_stale(&self, now: i64) -> bool {
+        match self.level_at {
+            // Never classified — no basis for a truth value.
+            None => true,
+            Some(at) => {
+                let window = STALE_FREQUENCY_MULTIPLIER
+                    .saturating_mul(self.frequency_secs)
+                    .saturating_mul(1_000_000);
+                now.saturating_sub(at) > window
+            }
+        }
+    }
+}
+
+/// What a composite does with a child whose state has gone stale (§6.4).
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub enum StaleChildPolicy {
+    /// Trust the frozen state (default).
+    #[serde(rename = "use_last_state")]
+    UseLastState,
+    /// A stale child never satisfies the expression.
+    #[serde(rename = "treat_as_false")]
+    TreatAsFalse,
+    /// Fail-safe for absence-of-heartbeat patterns.
+    #[serde(rename = "treat_as_true")]
+    TreatAsTrue,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum CompositeError {
+    /// The expression references a child absent from the supplied state map.
+    /// Loud failure beats a silent `false` that would never fire.
+    UnknownChild(String),
+    TooFewChildren {
+        got: usize,
+        min: usize,
+    },
+    TooManyChildren {
+        got: usize,
+        max: usize,
+    },
+    DuplicateChild(String),
+    /// The reference chain loops; carries the offending path.
+    Cycle(Vec<String>),
+    TooDeep {
+        max: usize,
+    },
+    Parse(String),
+}
+
+impl std::fmt::Display for CompositeError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::UnknownChild(c) => write!(f, "unknown child alert `{c}`"),
+            Self::TooFewChildren { got, min } => {
+                write!(f, "composite needs at least {min} children, got {got}")
+            }
+            Self::TooManyChildren { got, max } => {
+                write!(f, "composite allows at most {max} children, got {got}")
+            }
+            Self::DuplicateChild(c) => write!(f, "child `{c}` referenced more than once"),
+            Self::Cycle(path) => write!(f, "composite reference cycle: {}", path.join(" -> ")),
+            Self::TooDeep { max } => write!(f, "composite nesting exceeds depth {max}"),
+            Self::Parse(m) => write!(f, "invalid expression: {m}"),
+        }
+    }
+}
+
+impl std::error::Error for CompositeError {}
+
+// ── Parsing ─────────────────────────────────────────────────────────────────
+
+#[derive(Clone, Debug, PartialEq)]
+enum Token {
+    Ident(String),
+    And,
+    Or,
+    Not,
+    LParen,
+    RParen,
+}
+
+fn tokenize(input: &str) -> Result<Vec<Token>, CompositeError> {
+    let chars: Vec<char> = input.chars().collect();
+    let mut out = Vec::new();
+    let mut i = 0;
+    while i < chars.len() {
+        let c = chars[i];
+        match c {
+            c if c.is_whitespace() => i += 1,
+            '(' => {
+                out.push(Token::LParen);
+                i += 1;
+            }
+            ')' => {
+                out.push(Token::RParen);
+                i += 1;
+            }
+            '!' => {
+                out.push(Token::Not);
+                i += 1;
+            }
+            '&' | '|' => {
+                if i + 1 < chars.len() && chars[i + 1] == c {
+                    out.push(if c == '&' { Token::And } else { Token::Or });
+                    i += 2;
+                } else {
+                    return Err(CompositeError::Parse(format!("expected `{c}{c}`")));
+                }
+            }
+            // Operand tokens: alert ksuids in stored expressions, readable
+            // names in tests and the UI. Braces are accepted so `{ksuid}` is
+            // valid without changing the grammar.
+            c if c.is_alphanumeric() || c == '_' || c == '-' || c == '{' || c == '}' => {
+                let start = i;
+                while i < chars.len()
+                    && (chars[i].is_alphanumeric()
+                        || chars[i] == '_'
+                        || chars[i] == '-'
+                        || chars[i] == '{'
+                        || chars[i] == '}')
+                {
+                    i += 1;
+                }
+                let raw: String = chars[start..i].iter().collect();
+                out.push(Token::Ident(
+                    raw.trim_matches(|c| c == '{' || c == '}').to_string(),
+                ));
+            }
+            other => {
+                return Err(CompositeError::Parse(format!(
+                    "unexpected character `{other}`"
+                )));
+            }
+        }
+    }
+    Ok(out)
+}
+
+struct Parser {
+    tokens: Vec<Token>,
+    pos: usize,
+}
+
+impl Parser {
+    fn peek(&self) -> Option<&Token> {
+        self.tokens.get(self.pos)
+    }
+
+    /// or := and ( "||" and )*
+    fn parse_or(&mut self) -> Result<CompositeExpr, CompositeError> {
+        let mut lhs = self.parse_and()?;
+        while matches!(self.peek(), Some(Token::Or)) {
+            self.pos += 1;
+            let rhs = self.parse_and()?;
+            lhs = CompositeExpr::Or(Box::new(lhs), Box::new(rhs));
+        }
+        Ok(lhs)
+    }
+
+    /// and := unary ( "&&" unary )*  — binds tighter than `||`.
+    fn parse_and(&mut self) -> Result<CompositeExpr, CompositeError> {
+        let mut lhs = self.parse_unary()?;
+        while matches!(self.peek(), Some(Token::And)) {
+            self.pos += 1;
+            let rhs = self.parse_unary()?;
+            lhs = CompositeExpr::And(Box::new(lhs), Box::new(rhs));
+        }
+        Ok(lhs)
+    }
+
+    /// unary := "!" unary | atom  — binds tighter than `&&`.
+    fn parse_unary(&mut self) -> Result<CompositeExpr, CompositeError> {
+        if matches!(self.peek(), Some(Token::Not)) {
+            self.pos += 1;
+            return Ok(CompositeExpr::Not(Box::new(self.parse_unary()?)));
+        }
+        self.parse_atom()
+    }
+
+    fn parse_atom(&mut self) -> Result<CompositeExpr, CompositeError> {
+        match self.peek().cloned() {
+            Some(Token::Ident(name)) => {
+                self.pos += 1;
+                Ok(CompositeExpr::Child(name))
+            }
+            Some(Token::LParen) => {
+                self.pos += 1;
+                let inner = self.parse_or()?;
+                if !matches!(self.peek(), Some(Token::RParen)) {
+                    return Err(CompositeError::Parse("unclosed `(`".to_string()));
+                }
+                self.pos += 1;
+                Ok(inner)
+            }
+            Some(t) => Err(CompositeError::Parse(format!("unexpected token {t:?}"))),
+            None => Err(CompositeError::Parse(
+                "unexpected end of expression".to_string(),
+            )),
+        }
+    }
+}
+
+/// Parse a composite expression. `&&` binds tighter than `||`, `!` tighter than
+/// both — matching every mainstream language, because getting this wrong
+/// silently changes the meaning of every composite with no error anywhere.
+pub fn parse_expr(input: &str) -> Result<CompositeExpr, CompositeError> {
+    let tokens = tokenize(input)?;
+    if tokens.is_empty() {
+        return Err(CompositeError::Parse("empty expression".to_string()));
+    }
+    let mut p = Parser { tokens, pos: 0 };
+    let expr = p.parse_or()?;
+    if p.pos != p.tokens.len() {
+        return Err(CompositeError::Parse(
+            "trailing tokens after expression".to_string(),
+        ));
+    }
+    Ok(expr)
+}
+
+// ── Evaluation ──────────────────────────────────────────────────────────────
+
+fn level_truth(level: Option<AlertLevel>, warning_counts_as_firing: bool) -> bool {
+    match level {
+        Some(AlertLevel::Critical) => true,
+        Some(AlertLevel::Warning) => warning_counts_as_firing,
+        _ => false,
+    }
+}
+
+fn child_truth(
+    name: &str,
+    states: &HashMap<String, ChildState>,
+    warning_counts_as_firing: bool,
+    stale_policy: StaleChildPolicy,
+    now: i64,
+) -> Result<bool, CompositeError> {
+    let state = states
+        .get(name)
+        .ok_or_else(|| CompositeError::UnknownChild(name.to_string()))?;
+
+    if state.is_stale(now) {
+        return Ok(match stale_policy {
+            StaleChildPolicy::TreatAsFalse => false,
+            StaleChildPolicy::TreatAsTrue => true,
+            // Fall back to the frozen level; a never-classified child yields
+            // false rather than panicking.
+            StaleChildPolicy::UseLastState => level_truth(state.level, warning_counts_as_firing),
+        });
+    }
+    Ok(level_truth(state.level, warning_counts_as_firing))
+}
+
+/// Evaluate a composite over child states. Never touches child queries.
+pub fn evaluate_expr(
+    expr: &CompositeExpr,
+    states: &HashMap<String, ChildState>,
+    warning_counts_as_firing: bool,
+    stale_policy: StaleChildPolicy,
+    now: i64,
+) -> Result<bool, CompositeError> {
+    Ok(match expr {
+        CompositeExpr::Child(name) => {
+            child_truth(name, states, warning_counts_as_firing, stale_policy, now)?
+        }
+        CompositeExpr::Not(inner) => {
+            !evaluate_expr(inner, states, warning_counts_as_firing, stale_policy, now)?
+        }
+        CompositeExpr::And(a, b) => {
+            evaluate_expr(a, states, warning_counts_as_firing, stale_policy, now)?
+                && evaluate_expr(b, states, warning_counts_as_firing, stale_policy, now)?
+        }
+        CompositeExpr::Or(a, b) => {
+            evaluate_expr(a, states, warning_counts_as_firing, stale_policy, now)?
+                || evaluate_expr(b, states, warning_counts_as_firing, stale_policy, now)?
+        }
+    })
+}
+
+/// Level a composite reports.
+///
+/// OPEN DECISION (D9): a firing composite is always `Critical` — composites
+/// carry no thresholds of their own, so there is no basis for an intermediate
+/// level.
+pub fn result_level(fired: bool) -> AlertLevel {
+    if fired {
+        AlertLevel::Critical
+    } else {
+        AlertLevel::Ok
+    }
+}
+
+// ── Write-time guards ───────────────────────────────────────────────────────
+
+/// Child count and uniqueness (C-1).
+pub fn validate_children(children: &[String]) -> Result<(), CompositeError> {
+    if children.len() < MIN_CHILDREN {
+        return Err(CompositeError::TooFewChildren {
+            got: children.len(),
+            min: MIN_CHILDREN,
+        });
+    }
+    if children.len() > MAX_CHILDREN {
+        return Err(CompositeError::TooManyChildren {
+            got: children.len(),
+            max: MAX_CHILDREN,
+        });
+    }
+    let mut seen = std::collections::HashSet::new();
+    for c in children {
+        if !seen.insert(c) {
+            return Err(CompositeError::DuplicateChild(c.clone()));
+        }
+    }
+    Ok(())
+}
+
+/// Depth of a composite reference subtree. Plain alerts (absent from
+/// `existing`) are depth 0; a composite is 1 + its deepest composite child.
+fn depth_of(id: &str, existing: &HashMap<String, Vec<String>>) -> usize {
+    match existing.get(id) {
+        None => 0,
+        Some(children) => {
+            1 + children
+                .iter()
+                .map(|c| depth_of(c, existing))
+                .max()
+                .unwrap_or(0)
+        }
+    }
+}
+
+/// Reject reference cycles and over-deep nesting at write time (C-4, D6).
+///
+/// Detected on write, never at evaluation: a cycle discovered mid-evaluation
+/// would be an infinite loop in the scheduler.
+pub fn validate_no_cycle(
+    composite_id: &str,
+    children: &[String],
+    existing: &HashMap<String, Vec<String>>,
+) -> Result<(), CompositeError> {
+    // Depth-first search back toward `composite_id`.
+    fn walk(
+        current: &str,
+        target: &str,
+        existing: &HashMap<String, Vec<String>>,
+        path: &mut Vec<String>,
+    ) -> Option<Vec<String>> {
+        if current == target {
+            path.push(current.to_string());
+            return Some(path.clone());
+        }
+        if path.iter().any(|p| p == current) {
+            // A pre-existing loop between other composites; stop descending.
+            return None;
+        }
+        path.push(current.to_string());
+        if let Some(kids) = existing.get(current) {
+            for k in kids {
+                if let Some(found) = walk(k, target, existing, path) {
+                    return Some(found);
+                }
+            }
+        }
+        path.pop();
+        None
+    }
+
+    for child in children {
+        let mut path = Vec::new();
+        if let Some(cycle) = walk(child, composite_id, existing, &mut path) {
+            return Err(CompositeError::Cycle(cycle));
+        }
+    }
+
+    // Depth of the composite as it would exist after this write.
+    let deepest = children
+        .iter()
+        .map(|c| depth_of(c, existing))
+        .max()
+        .unwrap_or(0);
+    if 1 + deepest > MAX_DEPTH {
+        return Err(CompositeError::TooDeep { max: MAX_DEPTH });
+    }
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use std::collections::HashMap;
+
+    use crate::meta::alerts::{
+        composite::{
+            ChildState, CompositeError, CompositeExpr, StaleChildPolicy, evaluate_expr, parse_expr,
+            validate_children, validate_no_cycle,
+        },
+        level::AlertLevel,
+    };
+
+    /// A child whose level was computed recently.
+    /// NOTE: staleness runs on `level_at` — when the level was last COMPUTED
+    /// from a successful evaluation — not on any "last evaluation" time. An
+    /// alert erroring every minute stays fresh on last_outcome_at while its
+    /// level rots; `level_at` is immune to that (alerts_2.md §6.4/§7.6).
+    fn fresh(level: AlertLevel) -> ChildState {
+        ChildState {
+            level: Some(level),
+            // 60s ago, inside the K x 60s = 180s staleness window.
+            level_at: Some(NOW - 60_000_000),
+            frequency_secs: 60,
+        }
+    }
+
+    /// A child whose level is far older than K× its frequency.
+    fn stale(level: AlertLevel) -> ChildState {
+        ChildState {
+            level: Some(level),
+            // 600s ago, well beyond the 180s window.
+            //
+            // NOTE: the original fixtures used level_at 1_000 vs 1 against
+            // NOW = 1_000_000 — a 1ms difference against a 180s threshold, so
+            // BOTH read as fresh and every staleness test passed vacuously.
+            level_at: Some(NOW - 600_000_000),
+            frequency_secs: 60,
+        }
+    }
+
+    fn states(pairs: &[(&str, ChildState)]) -> HashMap<String, ChildState> {
+        pairs
+            .iter()
+            .map(|(k, v)| (k.to_string(), v.clone()))
+            .collect()
+    }
+
+    /// Microseconds. Large enough that the fixtures below stay positive.
+    const NOW: i64 = 1_000_000_000;
+
+    // ── Expression parsing ──────────────────────────────────────────────────
+
+    #[test]
+    fn test_parse_single_child() {
+        let e = parse_expr("a").unwrap();
+        assert_eq!(e, CompositeExpr::Child("a".to_string()));
+    }
+
+    #[test]
+    fn test_parse_and_or_not() {
+        assert!(parse_expr("a && b").is_ok());
+        assert!(parse_expr("a || b").is_ok());
+        assert!(parse_expr("!a").is_ok());
+        assert!(parse_expr("(a && b) || !c").is_ok());
+    }
+
+    #[test]
+    fn test_parse_rejects_malformed_expressions() {
+        for bad in ["a &&", "&& b", "(a && b", "a b", "", "a && && b", "!"] {
+            assert!(
+                parse_expr(bad).is_err(),
+                "expression {bad:?} must be rejected"
+            );
+        }
+    }
+
+    #[test]
+    fn test_and_binds_tighter_than_or() {
+        // `a || b && c` must parse as `a || (b && c)`, matching every
+        // mainstream language. Getting this wrong silently changes semantics.
+        //
+        // The operand values must DISCRIMINATE between the two parses:
+        //   a=true, b=false, c=false
+        //     a || (b && c)  =  true  || false  =  TRUE   <- correct
+        //     (a || b) && c  =  true  && false  =  FALSE  <- wrong precedence
+        // An earlier version used a=false,b=true,c=false, where both parses
+        // yield false — it passed regardless of precedence.
+        let states = states(&[
+            ("a", fresh(AlertLevel::Critical)),
+            ("b", fresh(AlertLevel::Ok)),
+            ("c", fresh(AlertLevel::Ok)),
+        ]);
+        let e = parse_expr("a || b && c").unwrap();
+        assert!(
+            evaluate_expr(&e, &states, true, StaleChildPolicy::UseLastState, NOW).unwrap(),
+            "must be true; a false result means OR bound tighter than AND"
+        );
+    }
+
+    #[test]
+    fn test_not_binds_tighter_than_and() {
+        // `!a && b` is `(!a) && b`, not `!(a && b)`.
+        //   a=true, b=true  ->  (!a) && b = false ;  !(a && b) = false   (same)
+        //   a=false, b=true ->  (!a) && b = TRUE  ;  !(a && b) = TRUE    (same)
+        //   a=true, b=false ->  (!a) && b = false ;  !(a && b) = TRUE    <- differs
+        let s = states(&[
+            ("a", fresh(AlertLevel::Critical)),
+            ("b", fresh(AlertLevel::Ok)),
+        ]);
+        let e = parse_expr("!a && b").unwrap();
+        assert!(
+            !evaluate_expr(&e, &s, true, StaleChildPolicy::UseLastState, NOW).unwrap(),
+            "must be false; a true result means NOT was applied to the whole AND"
+        );
+    }
+
+    #[test]
+    fn test_parentheses_override_precedence() {
+        let states = states(&[
+            ("a", fresh(AlertLevel::Critical)),
+            ("b", fresh(AlertLevel::Ok)),
+            ("c", fresh(AlertLevel::Ok)),
+        ]);
+        // (a || b) && c  =>  true && false => false
+        let e = parse_expr("(a || b) && c").unwrap();
+        assert!(!evaluate_expr(&e, &states, true, StaleChildPolicy::UseLastState, NOW).unwrap());
+    }
+
+    // ── C-3: child truth mapping ────────────────────────────────────────────
+
+    #[test]
+    fn test_critical_child_is_true() {
+        let s = states(&[("a", fresh(AlertLevel::Critical))]);
+        let e = parse_expr("a").unwrap();
+        assert!(evaluate_expr(&e, &s, true, StaleChildPolicy::UseLastState, NOW).unwrap());
+    }
+
+    #[test]
+    fn test_ok_child_is_false() {
+        let s = states(&[("a", fresh(AlertLevel::Ok))]);
+        let e = parse_expr("a").unwrap();
+        assert!(!evaluate_expr(&e, &s, true, StaleChildPolicy::UseLastState, NOW).unwrap());
+    }
+
+    /// D5: default is that Warning counts as firing.
+    #[test]
+    fn test_warning_child_truth_follows_the_flag() {
+        let s = states(&[("a", fresh(AlertLevel::Warning))]);
+        let e = parse_expr("a").unwrap();
+
+        assert!(
+            evaluate_expr(&e, &s, true, StaleChildPolicy::UseLastState, NOW).unwrap(),
+            "warning_counts_as_firing = true (default)"
+        );
+        assert!(
+            !evaluate_expr(&e, &s, false, StaleChildPolicy::UseLastState, NOW).unwrap(),
+            "warning_counts_as_firing = false"
+        );
+    }
+
+    #[test]
+    fn test_negation_inverts_child_truth() {
+        let s = states(&[("a", fresh(AlertLevel::Ok))]);
+        let e = parse_expr("!a").unwrap();
+        assert!(evaluate_expr(&e, &s, true, StaleChildPolicy::UseLastState, NOW).unwrap());
+    }
+
+    #[test]
+    fn test_realistic_composite_expression() {
+        // (disk_full && high_latency) || db_down
+        let s = states(&[
+            ("disk_full", fresh(AlertLevel::Critical)),
+            ("high_latency", fresh(AlertLevel::Ok)),
+            ("db_down", fresh(AlertLevel::Critical)),
+        ]);
+        let e = parse_expr("(disk_full && high_latency) || db_down").unwrap();
+        assert!(evaluate_expr(&e, &s, true, StaleChildPolicy::UseLastState, NOW).unwrap());
+    }
+
+    // ── §6.4: stale-child policy ────────────────────────────────────────────
+
+    #[test]
+    fn test_stale_child_use_last_state_keeps_its_truth() {
+        let s = states(&[("a", stale(AlertLevel::Critical))]);
+        let e = parse_expr("a").unwrap();
+        assert!(
+            evaluate_expr(&e, &s, true, StaleChildPolicy::UseLastState, NOW).unwrap(),
+            "default policy trusts the frozen state"
+        );
+    }
+
+    #[test]
+    fn test_stale_child_treat_as_false() {
+        let s = states(&[("a", stale(AlertLevel::Critical))]);
+        let e = parse_expr("a").unwrap();
+        assert!(!evaluate_expr(&e, &s, true, StaleChildPolicy::TreatAsFalse, NOW).unwrap());
+    }
+
+    #[test]
+    fn test_stale_child_treat_as_true_supports_heartbeat_patterns() {
+        let s = states(&[("a", stale(AlertLevel::Ok))]);
+        let e = parse_expr("a").unwrap();
+        assert!(
+            evaluate_expr(&e, &s, true, StaleChildPolicy::TreatAsTrue, NOW).unwrap(),
+            "fail-safe: absence of a heartbeat should be able to fire"
+        );
+    }
+
+    #[test]
+    fn test_never_evaluated_child_uses_the_stale_policy() {
+        // A child that has never run has no level at all — distinct from Ok.
+        let never = ChildState {
+            level: None,
+            level_at: None,
+            frequency_secs: 60,
+        };
+        let s = states(&[("a", never)]);
+        let e = parse_expr("a").unwrap();
+
+        assert!(!evaluate_expr(&e, &s, true, StaleChildPolicy::TreatAsFalse, NOW).unwrap());
+        assert!(evaluate_expr(&e, &s, true, StaleChildPolicy::TreatAsTrue, NOW).unwrap());
+        assert!(
+            !evaluate_expr(&e, &s, true, StaleChildPolicy::UseLastState, NOW).unwrap(),
+            "no last state to use → falls back to false rather than panicking"
+        );
+    }
+
+    #[test]
+    fn test_missing_child_state_is_an_error_not_a_silent_false() {
+        // A referenced child with no entry at all means the caller assembled
+        // the state map wrongly. Failing loudly beats a silent false.
+        let s = states(&[("a", fresh(AlertLevel::Critical))]);
+        let e = parse_expr("a && missing").unwrap();
+        let err = evaluate_expr(&e, &s, true, StaleChildPolicy::UseLastState, NOW).unwrap_err();
+        assert_eq!(err, CompositeError::UnknownChild("missing".to_string()));
+    }
+
+    // ── C-1: child-count limits ─────────────────────────────────────────────
+
+    #[test]
+    fn test_requires_at_least_two_children() {
+        let err = validate_children(&["a".to_string()]).unwrap_err();
+        assert_eq!(err, CompositeError::TooFewChildren { got: 1, min: 2 });
+    }
+
+    #[test]
+    fn test_accepts_between_two_and_ten_children() {
+        for n in 2..=10 {
+            let kids: Vec<String> = (0..n).map(|i| format!("a{i}")).collect();
+            assert!(
+                validate_children(&kids).is_ok(),
+                "{n} children must be valid"
+            );
+        }
+    }
+
+    #[test]
+    fn test_rejects_more_than_ten_children() {
+        let kids: Vec<String> = (0..11).map(|i| format!("a{i}")).collect();
+        let err = validate_children(&kids).unwrap_err();
+        assert_eq!(err, CompositeError::TooManyChildren { got: 11, max: 10 });
+    }
+
+    #[test]
+    fn test_rejects_duplicate_children() {
+        let kids = vec!["a".to_string(), "a".to_string()];
+        assert_eq!(
+            validate_children(&kids).unwrap_err(),
+            CompositeError::DuplicateChild("a".to_string())
+        );
+    }
+
+    // ── C-4: cycle rejection ────────────────────────────────────────────────
+
+    #[test]
+    fn test_rejects_direct_self_reference() {
+        let err = validate_no_cycle("c1", &["c1".to_string()], &HashMap::new()).unwrap_err();
+        assert_eq!(err, CompositeError::Cycle(vec!["c1".to_string()]));
+    }
+
+    #[test]
+    fn test_rejects_transitive_cycle() {
+        // c1 -> c2 -> c1
+        let mut existing = HashMap::new();
+        existing.insert("c2".to_string(), vec!["c1".to_string()]);
+
+        let err = validate_no_cycle("c1", &["c2".to_string()], &existing).unwrap_err();
+        assert!(
+            matches!(err, CompositeError::Cycle(_)),
+            "a transitive cycle must be rejected at write time, not discovered at eval"
+        );
+    }
+
+    #[test]
+    fn test_accepts_acyclic_composite_of_composite() {
+        // c1 -> c2 -> (plain alerts). Depth 2, no cycle (D6).
+        let mut existing = HashMap::new();
+        existing.insert("c2".to_string(), vec![]);
+        assert!(validate_no_cycle("c1", &["c2".to_string()], &existing).is_ok());
+    }
+
+    /// D6: depth beyond 2 is rejected in v1.
+    #[test]
+    fn test_rejects_depth_greater_than_two() {
+        // c1 -> c2 -> c3 is depth 3.
+        let mut existing = HashMap::new();
+        existing.insert("c2".to_string(), vec!["c3".to_string()]);
+        existing.insert("c3".to_string(), vec![]);
+
+        let err = validate_no_cycle("c1", &["c2".to_string()], &existing).unwrap_err();
+        assert_eq!(err, CompositeError::TooDeep { max: 2 });
+    }
+
+    // ── Composite result maps to a level ────────────────────────────────────
+
+    /// OPEN DECISION (not covered by alerts_2.md): a firing composite is
+    /// reported as Critical, never Warning — composites have no thresholds of
+    /// their own, so there is no basis for an intermediate level. If composites
+    /// should instead inherit the worst child level (so a warning-only
+    /// composite reports Warning), this is the test to change.
+    #[test]
+    fn test_composite_result_maps_to_a_level() {
+        use crate::meta::alerts::composite::result_level;
+        assert_eq!(result_level(true), AlertLevel::Critical);
+        assert_eq!(result_level(false), AlertLevel::Ok);
+    }
+}

src/config/src/meta/alerts/deduplication.rs

@@ -283,6 +283,22 @@ impl DeduplicationConfig {
     }
 }
 
+/// Whether an existing dedup reservation should suppress this notification
+/// (`alerts_2.md` §5.5 MN-6).
+///
+/// Deduplication currently records a fingerprint as *seen* the moment a row
+/// passes — before the notification is sent — so a send that then fails is
+/// suppressed as a duplicate on the retry, and the page is lost for the whole
+/// dedup window. The `notification_sent` column already exists for exactly
+/// this and has never been read.
+///
+/// The rule: a reservation suppresses only once it has been **confirmed** by a
+/// successful delivery. An unconfirmed reservation inside the window is a
+/// delivery that never landed, so the next evaluation must be allowed through.
+pub fn reservation_suppresses(notification_sent: bool, within_window: bool) -> bool {
+    within_window && notification_sent
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -670,4 +686,45 @@ mod tests {
     fn test_default_group_wait() {
         assert_eq!(default_group_wait(), 30);
     }
+
+    // ── §5.5 MN-6: reserve, then confirm on successful delivery ─────────────
+
+    #[test]
+    fn test_an_unconfirmed_reservation_does_not_suppress() {
+        // THE bug this rule exists for. The previous evaluation reserved the
+        // fingerprint and its send then failed, so nothing was delivered.
+        // Suppressing here would convert one transient webhook error into a
+        // page silently lost for the whole dedup window.
+        assert!(!reservation_suppresses(false, true));
+    }
+
+    #[test]
+    fn test_a_confirmed_reservation_inside_the_window_suppresses() {
+        // The ordinary case dedup exists for: it really was delivered.
+        assert!(reservation_suppresses(true, true));
+    }
+
+    #[test]
+    fn test_an_expired_reservation_never_suppresses() {
+        // Outside the window nothing suppresses, confirmed or not — otherwise
+        // a delivered alert could never fire again.
+        assert!(!reservation_suppresses(true, false));
+        assert!(!reservation_suppresses(false, false));
+    }
+
+    #[test]
+    fn test_confirmation_is_required_not_merely_preferred() {
+        // Stated as the invariant rather than a case list: within the window,
+        // suppression must track confirmation exactly. An implementation that
+        // ignored `notification_sent` (today's behaviour) fails this.
+        for within in [true, false] {
+            for sent in [true, false] {
+                assert_eq!(
+                    reservation_suppresses(sent, within),
+                    within && sent,
+                    "within_window={within}, notification_sent={sent}"
+                );
+            }
+        }
+    }
 }

src/config/src/meta/alerts/incidents.rs

@@ -388,9 +388,17 @@ pub struct IncidentAlert {
     pub incident_id: String,
     pub alert_id: String,
     pub alert_name: String,
+    #[serde(default)]
+    pub alert_kind: AlertKind,
     pub alert_fired_at: i64,
     pub correlation_reason: CorrelationReason,
     pub created_at: i64,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub source_url: Option<String>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub labels: Option<std::collections::HashMap<String, String>>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub detected_source: Option<String>,
 }
 
 /// Incident with its alerts (for detail view)
@@ -497,6 +505,13 @@ pub enum IncidentCorrelationOutcome {
         incident_id: String,
         service_name: String,
     },
+    /// The alert re-fired at a HIGHER severity than the incident currently
+    /// carries (Warning→Critical, T-8). The incident's severity was upgraded
+    /// and a notification must be sent — an escalation is never a repeat.
+    SeverityEscalated {
+        incident_id: String,
+        service_name: String,
+    },
 }
 
 impl IncidentCorrelationOutcome {
@@ -504,7 +519,8 @@ impl IncidentCorrelationOutcome {
         match self {
             Self::NewIncidentCreated { incident_id, .. }
             | Self::NewAlertTypeJoined { incident_id, .. }
-            | Self::ExistingAlertRepeated { incident_id, .. } => incident_id,
+            | Self::ExistingAlertRepeated { incident_id, .. }
+            | Self::SeverityEscalated { incident_id, .. } => incident_id,
         }
     }
 
@@ -512,7 +528,8 @@ impl IncidentCorrelationOutcome {
         match self {
             Self::NewIncidentCreated { service_name, .. }
             | Self::NewAlertTypeJoined { service_name, .. }
-            | Self::ExistingAlertRepeated { service_name, .. } => service_name,
+            | Self::ExistingAlertRepeated { service_name, .. }
+            | Self::SeverityEscalated { service_name, .. } => service_name,
         }
     }
 }
@@ -533,6 +550,72 @@ fn default_true() -> bool {
     true
 }
 
+/// Whether an incident's member alert is an OpenObserve alert or an
+/// externally-ingested one (External Alert Sources feature).
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize, ToSchema)]
+#[serde(rename_all = "snake_case")]
+pub enum AlertKind {
+    #[default]
+    Internal,
+    External,
+}
+
+impl AlertKind {
+    pub fn as_str(&self) -> &'static str {
+        match self {
+            AlertKind::Internal => "internal",
+            AlertKind::External => "external",
+        }
+    }
+
+    pub fn from_stored(s: &str) -> Self {
+        match s {
+            "external" => AlertKind::External,
+            _ => AlertKind::Internal,
+        }
+    }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize, ToSchema)]
+#[serde(rename_all = "lowercase")]
+pub enum ExternalAlertStatus {
+    #[default]
+    Firing,
+    Resolved,
+}
+
+/// One normalized alert event from an external source (Grafana, Alertmanager,
+/// generic JSON). Produced by the normalizers, consumed by persistence +
+/// correlation. Self-contained: `raw` keeps the per-alert slice of the
+/// original payload.
+#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
+pub struct ExternalAlertEvent {
+    pub status: ExternalAlertStatus,
+    pub dedup_key: String,
+    pub title: String,
+    pub severity: IncidentSeverity,
+    pub labels: HashMap<String, String>,
+    /// Source event time (startsAt/endsAt) in epoch micros — NOT receipt time.
+    pub event_ts: i64,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub source_url: Option<String>,
+    #[serde(default)]
+    pub raw: serde_json::Value,
+}
+
+/// Built-in default severity mapping (spec §4.9: "no JSON for humans").
+/// critical→P1, error/major→P2, warning/minor→P3, info→P4, P1..P4 passthrough,
+/// anything else → P3.
+pub fn map_external_severity(s: &str) -> IncidentSeverity {
+    match s.to_ascii_lowercase().as_str() {
+        "critical" | "crit" | "fatal" | "p1" => IncidentSeverity::P1,
+        "error" | "major" | "high" | "p2" => IncidentSeverity::P2,
+        "warning" | "warn" | "minor" | "p3" => IncidentSeverity::P3,
+        "info" | "low" | "ok" | "p4" => IncidentSeverity::P4,
+        _ => IncidentSeverity::P3,
+    }
+}
+
 /// Statistics for incidents dashboard
 #[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
 pub struct IncidentStats {
@@ -2123,4 +2206,111 @@ mod tests {
             IncidentEventType::AIAnalysisComplete
         ));
     }
+
+    #[test]
+    fn test_alert_kind_serde_and_stored() {
+        assert_eq!(
+            serde_json::to_string(&AlertKind::External).unwrap(),
+            "\"external\""
+        );
+        assert_eq!(AlertKind::from_stored("external"), AlertKind::External);
+        assert_eq!(AlertKind::from_stored("internal"), AlertKind::Internal);
+        assert_eq!(AlertKind::from_stored("garbage"), AlertKind::Internal); // safe default
+        assert_eq!(AlertKind::External.as_str(), "external");
+    }
+
+    #[test]
+    fn test_incident_alert_external_fields_serde() {
+        let alert = IncidentAlert {
+            incident_id: "inc1".to_string(),
+            alert_id: "ext1".to_string(),
+            alert_name: "External Alert".to_string(),
+            alert_kind: AlertKind::External,
+            alert_fired_at: 1000,
+            correlation_reason: CorrelationReason::AlertId,
+            created_at: 1000,
+            source_url: Some("https://example.com/alert/1".to_string()),
+            labels: Some(std::collections::HashMap::from([(
+                "service".to_string(),
+                "checkout".to_string(),
+            )])),
+            detected_source: Some("pagerduty".to_string()),
+        };
+
+        let json = serde_json::to_value(&alert).unwrap();
+        assert_eq!(json["alert_kind"], "external");
+        assert_eq!(json["source_url"], "https://example.com/alert/1");
+        assert_eq!(json["detected_source"], "pagerduty");
+
+        let round_tripped: IncidentAlert = serde_json::from_value(json).unwrap();
+        assert_eq!(round_tripped.alert_kind, AlertKind::External);
+        assert_eq!(
+            round_tripped.source_url,
+            Some("https://example.com/alert/1".to_string())
+        );
+    }
+
+    #[test]
+    fn test_incident_alert_legacy_json_defaults_to_internal() {
+        // Legacy stored/older payloads have no alert_kind or external fields at all.
+        let legacy_json = serde_json::json!({
+            "incident_id": "inc1",
+            "alert_id": "alert1",
+            "alert_name": "My Alert",
+            "alert_fired_at": 1000,
+            "correlation_reason": "alert_id",
+            "created_at": 1000,
+        });
+
+        let alert: IncidentAlert = serde_json::from_value(legacy_json).unwrap();
+        assert_eq!(alert.alert_kind, AlertKind::Internal);
+        assert_eq!(alert.source_url, None);
+        assert_eq!(alert.labels, None);
+        assert_eq!(alert.detected_source, None);
+    }
+
+    #[test]
+    fn test_external_alert_status_serde() {
+        assert_eq!(
+            serde_json::to_string(&ExternalAlertStatus::Firing).unwrap(),
+            "\"firing\""
+        );
+        let s: ExternalAlertStatus = serde_json::from_str("\"resolved\"").unwrap();
+        assert_eq!(s, ExternalAlertStatus::Resolved);
+    }
+
+    #[test]
+    fn test_map_external_severity() {
+        assert_eq!(map_external_severity("critical"), IncidentSeverity::P1);
+        assert_eq!(map_external_severity("CRITICAL"), IncidentSeverity::P1);
+        assert_eq!(map_external_severity("error"), IncidentSeverity::P2);
+        assert_eq!(map_external_severity("warning"), IncidentSeverity::P3);
+        assert_eq!(map_external_severity("info"), IncidentSeverity::P4);
+        assert_eq!(map_external_severity("p2"), IncidentSeverity::P2);
+        assert_eq!(
+            map_external_severity("unknown-things"),
+            IncidentSeverity::P3
+        ); // default
+    }
+
+    #[test]
+    fn test_external_alert_event_serde_roundtrip() {
+        let ev = ExternalAlertEvent {
+            status: ExternalAlertStatus::Firing,
+            dedup_key: "abc123".into(),
+            title: "High CPU".into(),
+            severity: IncidentSeverity::P1,
+            labels: std::collections::HashMap::from([(
+                "namespace".to_string(),
+                "prod".to_string(),
+            )]),
+            event_ts: 1_722_300_000_000_000,
+            source_url: Some("https://grafana.example/alerting/x".into()),
+            raw: serde_json::json!({"k": "v"}),
+        };
+        let json = serde_json::to_string(&ev).unwrap();
+        let back: ExternalAlertEvent = serde_json::from_str(&json).unwrap();
+        assert_eq!(back.dedup_key, "abc123");
+        assert_eq!(back.severity, IncidentSeverity::P1);
+    }
 }

src/config/src/meta/alerts/mod.rs

@@ -30,9 +30,18 @@ use crate::{
     },
 };
 
+pub mod aggregation_level;
 pub mod alert;
+pub mod composite;
 pub mod deduplication;
+pub mod dispatch;
+pub mod grouping;
 pub mod incidents;
+pub mod level;
+pub mod priority;
+pub mod state;
+pub mod state_level;
+pub mod tags;
 
 #[derive(Clone, Debug, Serialize, Deserialize, ToSchema, PartialEq, Default)]
 #[serde(default)]
@@ -42,7 +51,19 @@ pub struct TriggerCondition {
     #[serde(default)]
     pub operator: Operator, // >=
     #[serde(default)]
-    pub threshold: i64, // 3 times
+    pub threshold: i64, // 3 times = CRITICAL level
+    /// Warning threshold, sharing `operator` with `threshold` — one operator
+    /// for both levels, no mixed directions (T-2). `None` = single-level
+    /// alert, i.e. exactly the legacy behaviour.
+    /// Validated by `level::validate_thresholds` — "less severe" is
+    /// direction-dependent, so this is NOT simply `< threshold`.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub warning_threshold: Option<i64>,
+    /// Whether a Warning-level match delivers a notification (D11).
+    /// `None` = true — warnings notify unless explicitly opted out.
+    /// Persisted in `trigger_thresholds`.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub notify_on_warning: Option<bool>,
     /// (seconds)
     #[serde(default)]
     pub frequency: i64, // 1 minute
@@ -321,11 +342,89 @@ impl TriggerCondition {
     }
 }
 
+impl TriggerCondition {
+    /// How long a group of this alert may go unobserved before M-7 resolves it,
+    /// in microseconds, measured from `last_seen`.
+    ///
+    /// Two schedule shapes, and they cannot share an implementation. A fixed
+    /// frequency is a constant, so `K × frequency` is the whole answer. A cron
+    /// alert's numeric `frequency` is **not** the cadence it runs at, and the
+    /// gap between consecutive fires is not even constant (weekday-only,
+    /// monthly, DST), so its deadline has to be read off the schedule itself —
+    /// anchored to this row's `last_seen` so it cannot drift between sweeps.
+    pub fn group_resolve_threshold_micros(&self, last_seen: i64, k: i64) -> i64 {
+        use crate::meta::alerts::grouping::{
+            cron_resolve_threshold_micros, resolve_threshold_micros,
+        };
+
+        if self.frequency_type != FrequencyType::Cron {
+            return resolve_threshold_micros(self.frequency, k);
+        }
+
+        // An unparseable expression or an out-of-range timestamp must never
+        // resolve a live group: saturating high leaves the row alone until the
+        // schedule can be read, while any finite fallback would resolve groups
+        // on a cadence nobody configured.
+        let (Ok(schedule), Some(anchor)) = (
+            Schedule::from_str(&self.cron),
+            chrono::DateTime::from_timestamp_micros(last_seen),
+        ) else {
+            return i64::MAX;
+        };
+
+        // Same DST-aware offset resolution as `get_next_trigger_time_*`, so the
+        // sweep and the scheduler agree on when this alert actually fires.
+        let offset_minutes = match get_timezone_from_string(self.timezone.as_deref(), 0) {
+            Ok(tz) => get_offset_minutes_from_tz(&tz, anchor),
+            Err(_) => 0,
+        };
+        let Some(tz_offset) = FixedOffset::east_opt(offset_minutes * 60) else {
+            return i64::MAX;
+        };
+
+        let anchored = anchor.with_timezone(&tz_offset);
+        let occurrences = schedule
+            .after(&anchored)
+            .take(k.max(0) as usize)
+            .map(|d| d.timestamp_micros());
+        cron_resolve_threshold_micros(last_seen, occurrences, k)
+    }
+}
+
 #[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct TriggerEvalResults {
     pub data: Option<Vec<Map<String, Value>>>,
     pub end_time: i64,
     pub query_took: Option<i64>,
+    /// Severity of the matched threshold (`alerts_2.md` Feature 1).
+    /// `None` when nothing matched, or for evaluations with no level axis.
+    /// Always `Some` when `data` is `Some` for a condition-bearing module.
+    pub level: Option<level::AlertLevel>,
+    /// The value that was compared — row count, or the aggregate for
+    /// aggregation alerts. Recorded on the trigger record (T-9) so history can
+    /// show "112 vs 100". For count alerts this is a LOWER BOUND once the
+    /// search cap is reached (`alerts_2.md` §7.5).
+    pub actual_value: Option<f64>,
+    /// Which group/series produced `actual_value` ("host=b,region=eu"), for
+    /// grouped aggregation and PromQL alerts (T-9). `None` for count alerts —
+    /// a row count has no group identity.
+    pub group_label: Option<String>,
+    /// True when `actual_value` is a LOWER BOUND, not exact: the legacy
+    /// SingleQuery count path fetched exactly its cap, so the true count may
+    /// be higher (§7.5). History renders a `≥` prefix. Hybrid evaluations are
+    /// always exact.
+    pub value_is_lower_bound: bool,
+    /// Per-group view of this evaluation — `Some` only for an alert that opted
+    /// in to multi-alerts (M-9). `None` puts state persistence on exactly the
+    /// path it took before this feature existed.
+    ///
+    /// Classification happens during evaluation rather than at persist time
+    /// because that is where the aggregation's own thresholds are in scope:
+    /// `TriggerCondition.threshold` is the group-COUNT gate for an aggregation
+    /// alert, so classifying against it there would compare aggregates to a
+    /// count.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub group_classification: Option<grouping::GroupClassification>,
 }
 
 #[derive(Clone, Default, Debug, Serialize, Deserialize, ToSchema, PartialEq)]
@@ -359,6 +458,44 @@ pub struct QueryCondition {
     pub sql: Option<String>,
     pub promql: Option<String>,              // (cpu usage / cpu total)
     pub promql_condition: Option<Condition>, // value >= 80
+    /// WARNING value for the PromQL condition (alerts_2.md Feature 1).
+    ///
+    /// A sibling field rather than a member of `Condition`, which is shared by
+    /// every filter in the product and must not grow alert-specific knobs.
+    /// Shares `promql_condition.operator` with critical. `None` = single-level.
+    ///
+    /// Lenient deserialization: this arrives through
+    /// `CreateAlertRequestBody`'s `#[serde(flatten)]`, which buffers via
+    /// `Value`, where `arbitrary_precision` makes a number a map (D61).
+    /// Without it a FRACTIONAL warning is rejected while an integer one works
+    /// — found by end-to-end testing.
+    #[serde(
+        default,
+        skip_serializing_if = "Option::is_none",
+        deserialize_with = "crate::meta::slo::lenient_f64::deserialize_opt"
+    )]
+    pub promql_warning_value: Option<f64>,
+    /// Per-group alerting for a PromQL alert (M-9), where a "group" is one
+    /// returned SERIES.
+    ///
+    /// A sibling of `promql_condition` rather than a member of
+    /// [`Aggregation::multi_alert`], because a PromQL alert has no
+    /// `aggregation` at all — its grouping is expressed in the PromQL itself
+    /// (`sum by (pod) (…)`), not in a `group_by` column list.
+    ///
+    /// The group key is the series' FULL label set. That is what a series'
+    /// identity already is in Prometheus, and it means the expression stays
+    /// the single place grouping is decided — a separate label picker could
+    /// only ever disagree with the `by (…)` clause beside it.
+    ///
+    /// `#[serde(default)]` carries the same backward-compatibility guarantee
+    /// as its aggregation counterpart: an alert stored before this field
+    /// existed deserializes to `false` and keeps its collapsed evaluation
+    /// byte-for-byte. Nothing is inferred from the query returning several
+    /// series — that would change paging cadence for every existing PromQL
+    /// alert that happens to be unaggregated.
+    #[serde(default, skip_serializing_if = "std::ops::Not::not")]
+    pub promql_multi_alert: bool,
     pub aggregation: Option<Aggregation>,
     #[serde(default)]
     pub vrl_function: Option<String>,
@@ -366,6 +503,46 @@ pub struct QueryCondition {
     pub search_event_type: Option<SearchEventType>,
     #[serde(default)]
     pub multi_time_range: Option<Vec<CompareHistoricData>>,
+    /// Feature 5 (D42): the SLO condition, when `query_type` is `Slo`.
+    ///
+    /// Persisted in its own `alerts.query_slo_condition` column following the
+    /// `query_aggregation` precedent — deliberately not `trigger_thresholds`,
+    /// whose documented scope is threshold and level configuration only (D1).
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub slo_condition: Option<crate::meta::slo::condition::SloCondition>,
+}
+
+impl QueryCondition {
+    /// Whether this alert evaluates and pages per group.
+    ///
+    /// The aggregation and PromQL families store the opt-in in different places
+    /// because they define a group differently — an aggregation alert by its
+    /// `group_by` columns, a PromQL alert by each returned series' labels.
+    /// Every caller downstream of evaluation cares only about the answer, so
+    /// asking through one accessor is what keeps a family from being silently
+    /// honoured by only half the dispatch path.
+    ///
+    /// **SLO alerts answer `false`, and that is correct — do not "fix" it.**
+    /// `SloCondition.multi_alert` exists (SA-13) and `slo::evaluate` does read
+    /// every group when it is set, but only to pick the WORST one:
+    /// `evaluate_slo_alert` collapses the results into a single `level` +
+    /// `group_label` and never populates `group_classification`. Per-group
+    /// state rows are written only when a classification exists
+    /// (`persist_alert_run_state`), so an SLO alert has none, and SA-13's
+    /// "reuses Feature 3 verbatim" is not yet implemented.
+    ///
+    /// Returning `true` here would therefore switch OFF the alert-level
+    /// delivery decision — silence windows, the escalation baseline,
+    /// `notify_on_warning` — with nothing per-group to replace it, because no
+    /// per-group dispatch runs. That is a regression, not a fix. When SA-13
+    /// lands, this arm changes at the same time as the dispatch that justifies
+    /// it. Pinned by `test_slo_alerts_deliberately_answer_false`.
+    pub fn multi_alert_enabled(&self) -> bool {
+        match self.query_type {
+            QueryType::PromQL => self.promql_multi_alert,
+            _ => self.aggregation.as_ref().is_some_and(|a| a.multi_alert),
+        }
+    }
 }
 
 impl MemorySize for QueryCondition {
@@ -528,7 +705,36 @@ impl IntoIterator for ConditionList {
 pub struct Aggregation {
     pub group_by: Option<Vec<String>>,
     pub function: AggFunction,
+    /// CRITICAL threshold. `having.value` is untyped (`serde_json::Value`) and
+    /// may be an int, a float, or a numeric string.
     pub having: Condition,
+    /// WARNING threshold, sharing `having.operator` and `having.column` with
+    /// critical (alerts_2.md §4.4). `None` = single-level aggregation alert,
+    /// i.e. exactly the legacy behaviour. Stored as f64 because aggregate
+    /// values (averages, percentiles) are not integers.
+    /// Same lenient deserialization as `promql_warning_value`, for the same
+    /// reason (D61) — aggregate warnings are the field most likely to be
+    /// fractional.
+    #[serde(
+        default,
+        skip_serializing_if = "Option::is_none",
+        deserialize_with = "crate::meta::slo::lenient_f64::deserialize_opt"
+    )]
+    pub warning_value: Option<f64>,
+    /// Opt-in to per-group evaluation — multi-alerts, `alerts_2.md` M-9/D26.
+    ///
+    /// `#[serde(default)]` is the whole backward-compatibility guarantee: an
+    /// aggregation stored before this field existed cannot contain it, so it
+    /// deserializes to `false` and the alert keeps its legacy collapsed
+    /// evaluation byte-for-byte. Nothing is inferred from `group_by` being
+    /// present — that would silently change paging cadence and reset silence
+    /// fingerprints for every existing grouped alert.
+    ///
+    /// Validated by [`grouping::validate_multi_alert`] (M-10): requires a
+    /// non-empty `group_by`, an orderable `having.operator`, and "any group"
+    /// count gates.
+    #[serde(default, skip_serializing_if = "std::ops::Not::not")]
+    pub multi_alert: bool,
 }
 
 impl MemorySize for Aggregation {
@@ -613,6 +819,12 @@ pub enum QueryType {
     SQL,
     #[serde(rename = "promql")]
     PromQL,
+    /// Feature 5 (D28). An SLO alert is an ordinary `alerts` row whose
+    /// condition reads precomputed SLO status rather than running a query —
+    /// which is what lets five alerts on one SLO cost five cheap status reads
+    /// and ZERO extra raw-data scans.
+    #[serde(rename = "slo")]
+    Slo,
 }
 
 impl std::fmt::Display for QueryType {
@@ -621,6 +833,7 @@ impl std::fmt::Display for QueryType {
             QueryType::Custom => write!(f, "custom"),
             QueryType::SQL => write!(f, "sql"),
             QueryType::PromQL => write!(f, "promql"),
+            QueryType::Slo => write!(f, "slo"),
         }
     }
 }
@@ -631,6 +844,7 @@ impl From<&str> for QueryType {
             "custom" => QueryType::Custom,
             "sql" => QueryType::SQL,
             "promql" => QueryType::PromQL,
+            "slo" => QueryType::Slo,
             _ => QueryType::Custom,
         }
     }
@@ -2182,6 +2396,155 @@ mod test {
         assert!(AggFunction::try_from("unknown").is_err());
     }
 
+    // ── multi_alert_enabled: one question, two storage locations ────────────
+
+    fn agg_with_multi(multi: bool) -> Aggregation {
+        Aggregation {
+            group_by: Some(vec!["host".to_string()]),
+            function: AggFunction::Avg,
+            having: Condition {
+                column: "alert_agg_value".to_string(),
+                operator: Operator::GreaterThan,
+                value: serde_json::json!(90),
+                ignore_case: false,
+            },
+            warning_value: None,
+            multi_alert: multi,
+        }
+    }
+
+    #[test]
+    fn test_multi_alert_enabled_reads_the_aggregation_for_sql_alerts() {
+        let q = QueryCondition {
+            query_type: QueryType::SQL,
+            aggregation: Some(agg_with_multi(true)),
+            ..Default::default()
+        };
+        assert!(q.multi_alert_enabled());
+    }
+
+    #[test]
+    fn test_multi_alert_enabled_reads_the_promql_flag_for_promql_alerts() {
+        let q = QueryCondition {
+            query_type: QueryType::PromQL,
+            promql_multi_alert: true,
+            ..Default::default()
+        };
+        assert!(q.multi_alert_enabled());
+    }
+
+    /// The cross-family trap. A PromQL alert carrying a stray aggregation must
+    /// NOT be treated as per-group: its rows have no `group_by` columns, so the
+    /// aggregation extractor would hand every series the same empty label set
+    /// and collapse them into one group.
+    #[test]
+    fn test_a_promql_alert_ignores_an_aggregation_multi_alert_flag() {
+        let q = QueryCondition {
+            query_type: QueryType::PromQL,
+            promql_multi_alert: false,
+            aggregation: Some(agg_with_multi(true)),
+            ..Default::default()
+        };
+        assert!(!q.multi_alert_enabled());
+    }
+
+    /// And the mirror: a SQL alert must not be switched on by the PromQL flag,
+    /// which nothing in its evaluation path would honour.
+    #[test]
+    fn test_a_sql_alert_ignores_the_promql_multi_alert_flag() {
+        let q = QueryCondition {
+            query_type: QueryType::SQL,
+            promql_multi_alert: true,
+            aggregation: Some(agg_with_multi(false)),
+            ..Default::default()
+        };
+        assert!(!q.multi_alert_enabled());
+    }
+
+    /// SLO alerts must answer `false` until SA-13 actually lands.
+    ///
+    /// `slo_condition.multi_alert` reads every group, but only so
+    /// `evaluate_slo_alert` can pick the worst; it never builds a
+    /// `GroupClassification`, and group state rows are written only when one
+    /// exists. So there is no per-group dispatch for an SLO alert — and
+    /// answering `true` would switch off the ALERT-level delivery decision
+    /// (silence, escalation baseline, notify_on_warning) with nothing to
+    /// replace it. This test is the tripwire for anyone who reads
+    /// `SloCondition.multi_alert` and assumes the accessor is missing a case.
+    #[test]
+    fn test_slo_alerts_deliberately_answer_false() {
+        let q = QueryCondition {
+            query_type: QueryType::Slo,
+            aggregation: None,
+            ..Default::default()
+        };
+        assert!(
+            !q.multi_alert_enabled(),
+            "turning this true silences SLO alerts: it disables alert-level \
+             delivery gating, and no per-group dispatch runs for QueryType::Slo. \
+             Land SA-13's dispatch first, in the same change."
+        );
+    }
+
+    #[test]
+    fn test_multi_alert_enabled_is_false_by_default_for_every_query_type() {
+        for qt in [
+            QueryType::Custom,
+            QueryType::SQL,
+            QueryType::PromQL,
+            QueryType::Slo,
+        ] {
+            let q = QueryCondition {
+                query_type: qt.clone(),
+                ..Default::default()
+            };
+            assert!(!q.multi_alert_enabled(), "{qt} defaulted to enabled");
+        }
+    }
+
+    // ── Upgrade safety for the new flag ─────────────────────────────────────
+
+    /// THE guarantee (M-9/D26): every PromQL alert already stored was
+    /// serialized without this field. If it read back as anything but `false`,
+    /// all of them would switch to per-series evaluation on deploy — different
+    /// paging cadence, and every in-flight silence fingerprint invalidated.
+    #[test]
+    fn test_promql_multi_alert_defaults_to_false_when_absent_from_stored_json() {
+        let stored = serde_json::json!({
+            "type": "promql",
+            "promql": "up == 0",
+        });
+        let parsed: QueryCondition = serde_json::from_value(stored).expect("deserializable");
+        assert!(!parsed.promql_multi_alert);
+        assert!(!parsed.multi_alert_enabled());
+    }
+
+    /// `skip_serializing_if` keeps the field out of the payload when off, so
+    /// turning the feature on never rewrites rows that are not using it.
+    #[test]
+    fn test_promql_multi_alert_is_omitted_from_json_when_off() {
+        let q = QueryCondition {
+            query_type: QueryType::PromQL,
+            promql_multi_alert: false,
+            ..Default::default()
+        };
+        let v = serde_json::to_value(&q).expect("serializable");
+        assert!(v.get("promql_multi_alert").is_none(), "{v}");
+    }
+
+    #[test]
+    fn test_promql_multi_alert_round_trips_when_on() {
+        let q = QueryCondition {
+            query_type: QueryType::PromQL,
+            promql_multi_alert: true,
+            ..Default::default()
+        };
+        let v = serde_json::to_value(&q).expect("serializable");
+        assert_eq!(v.get("promql_multi_alert"), Some(&serde_json::json!(true)));
+        let back: QueryCondition = serde_json::from_value(v).expect("deserializable");
+        assert!(back.promql_multi_alert);
+    }
+
     #[test]
     fn test_query_type_display() {
         assert_eq!(QueryType::Custom.to_string(), "custom");

src/config/src/meta/alerts/priority.rs

@@ -0,0 +1,362 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Alert priority — Feature 2 (PT-1, PT-2).
+//!
+//! A **third** axis, distinct from the two Feature 1 established:
+//!
+//! * `RunOutcome`  — did the evaluation fire?
+//! * `AlertLevel`  — how bad is it *right now*? (evaluated state)
+//! * `AlertPriority` — how much do humans care about this alert? (**mutable** static configuration
+//!   — "static" contrasts it with *evaluated* state, it does NOT mean write-once: priority is
+//!   editable on any update, like name or description. PT-1.)
+//!
+//! Priority is **display + propagation only**: it filters/sorts the alert
+//! list and is exposed to notification templates so receivers can route on
+//! it. It must never influence evaluation, silence, delivery, or incident
+//! severity — that is a separate, explicitly opted-into extension.
+//!
+//! Deliberately NOT `IncidentSeverity` (P1–P4): different scale, different
+//! concept, different lifecycle.
+
+use serde::{Deserialize, Serialize};
+use utoipa::ToSchema;
+
+/// How much humans care about this alert. P1 is the most urgent.
+///
+/// Serialized as an **integer** (1–5), which is both the storage
+/// representation and the API shape; `Display` renders the familiar `"P3"`
+/// form for UI and template substitution.
+// `Ord` is deliberately NOT derived. Declaration order would make `P1 < P5`
+// true, so a bare `a.priority > b.priority` at a call site reads exactly
+// backwards ("greater" = less urgent). Sorting happens in SQL (PT-3), so
+// nothing here needs it; compare via `is_more_urgent_than` or `to_i32`.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, ToSchema)]
+#[serde(try_from = "u8", into = "u8")]
+// NOTE: the wire form is an integer (see the serde attribute above). utoipa
+// only accepts `value_type` at the FIELD level, so the `Alert.priority` field
+// carries `#[schema(value_type = u8)]` — without it the generated OpenAPI
+// would advertise a string enum and lie about the payload.
+pub enum AlertPriority {
+    P1,
+    P2,
+    P3,
+    P4,
+    P5,
+}
+
+/// Why a priority value was rejected.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum PriorityError {
+    /// Not one of the five defined priorities.
+    OutOfRange(String),
+}
+
+impl AlertPriority {
+    /// Durable storage id for the `alerts.priority` column.
+    ///
+    /// These are persisted — **never reorder or reuse**. P1 = 1 so that a
+    /// plain `ORDER BY priority ASC` surfaces the most urgent alerts first.
+    pub fn to_i32(&self) -> i32 {
+        match self {
+            Self::P1 => 1,
+            Self::P2 => 2,
+            Self::P3 => 3,
+            Self::P4 => 4,
+            Self::P5 => 5,
+        }
+    }
+
+    /// Inverse of [`Self::to_i32`]; `None` for any value outside 1..=5.
+    pub fn from_i32(v: i32) -> Option<Self> {
+        match v {
+            1 => Some(Self::P1),
+            2 => Some(Self::P2),
+            3 => Some(Self::P3),
+            4 => Some(Self::P4),
+            5 => Some(Self::P5),
+            _ => None,
+        }
+    }
+
+    /// `"P1"`..`"P5"` — the form shown in the UI and substituted into
+    /// notification templates.
+    pub fn as_str(&self) -> &'static str {
+        match self {
+            Self::P1 => "P1",
+            Self::P2 => "P2",
+            Self::P3 => "P3",
+            Self::P4 => "P4",
+            Self::P5 => "P5",
+        }
+    }
+
+    /// True when `self` is more urgent than `other` (P1 is most urgent).
+    ///
+    /// Named rather than relying on `Ord`, because the natural integer
+    /// ordering is *inverted* with respect to urgency and a bare `<` at a
+    /// call site would read exactly backwards.
+    pub fn is_more_urgent_than(&self, other: Self) -> bool {
+        // Smaller storage id = more urgent (P1 = 1). This is the one place
+        // that inversion is allowed to be written out.
+        self.to_i32() < other.to_i32()
+    }
+}
+
+impl std::fmt::Display for AlertPriority {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.write_str(self.as_str())
+    }
+}
+
+impl std::fmt::Display for PriorityError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::OutOfRange(v) => {
+                write!(f, "invalid alert priority `{v}`: expected P1 through P5")
+            }
+        }
+    }
+}
+
+impl std::error::Error for PriorityError {}
+
+impl std::str::FromStr for AlertPriority {
+    type Err = PriorityError;
+
+    /// Accepts `"P1"`/`"p1"` and the bare `"1"`, so query parameters and
+    /// template inputs both parse without the caller guessing which form.
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        let token = s.trim();
+        // Accept an optional `P`/`p` prefix, then require exactly one digit
+        // in range. `strip_prefix` on a char slice keeps "PP1" and "P" from
+        // sneaking through, since what remains must still match exactly.
+        let digits = token.strip_prefix(['P', 'p']).unwrap_or(token);
+        match digits {
+            "1" => Ok(Self::P1),
+            "2" => Ok(Self::P2),
+            "3" => Ok(Self::P3),
+            "4" => Ok(Self::P4),
+            "5" => Ok(Self::P5),
+            // Report the TRIMMED token: the user sees what was parsed, not
+            // their incidental whitespace.
+            _ => Err(PriorityError::OutOfRange(token.to_string())),
+        }
+    }
+}
+
+impl TryFrom<u8> for AlertPriority {
+    type Error = PriorityError;
+
+    fn try_from(v: u8) -> Result<Self, Self::Error> {
+        Self::from_i32(v as i32).ok_or_else(|| PriorityError::OutOfRange(v.to_string()))
+    }
+}
+
+impl From<AlertPriority> for u8 {
+    fn from(p: AlertPriority) -> u8 {
+        p.to_i32() as u8
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::str::FromStr;
+
+    use super::*;
+
+    const ALL: [AlertPriority; 5] = [
+        AlertPriority::P1,
+        AlertPriority::P2,
+        AlertPriority::P3,
+        AlertPriority::P4,
+        AlertPriority::P5,
+    ];
+
+    /// PT-2: storage ids are durable. This test exists to FAIL loudly if
+    /// anyone renumbers the enum — every persisted row would silently change
+    /// meaning.
+    #[test]
+    fn test_storage_ids_are_pinned_and_never_reordered() {
+        assert_eq!(AlertPriority::P1.to_i32(), 1);
+        assert_eq!(AlertPriority::P2.to_i32(), 2);
+        assert_eq!(AlertPriority::P3.to_i32(), 3);
+        assert_eq!(AlertPriority::P4.to_i32(), 4);
+        assert_eq!(AlertPriority::P5.to_i32(), 5);
+    }
+
+    #[test]
+    fn test_storage_id_round_trips() {
+        for p in ALL {
+            assert_eq!(AlertPriority::from_i32(p.to_i32()), Some(p));
+        }
+    }
+
+    /// Unset priority is represented by a NULL column / `None`, never by a
+    /// sentinel id — so 0 and 6 must not decode.
+    #[test]
+    fn test_out_of_range_ids_do_not_decode() {
+        for v in [-1, 0, 6, 99] {
+            assert_eq!(AlertPriority::from_i32(v), None, "id {v} must not decode");
+        }
+    }
+
+    /// P1 = 1 is deliberate: `ORDER BY priority ASC` must put the most urgent
+    /// alerts at the top of the list without a CASE expression (PT-3).
+    #[test]
+    fn test_ascending_storage_id_means_descending_urgency() {
+        let mut ids: Vec<i32> = ALL.iter().map(|p| p.to_i32()).collect();
+        ids.sort();
+        let sorted: Vec<AlertPriority> = ids
+            .into_iter()
+            .map(|i| AlertPriority::from_i32(i).unwrap())
+            .collect();
+        assert_eq!(sorted[0], AlertPriority::P1, "most urgent must sort first");
+        assert_eq!(sorted[4], AlertPriority::P5, "least urgent must sort last");
+    }
+
+    #[test]
+    fn test_urgency_comparison_is_not_the_integer_order() {
+        assert!(AlertPriority::P1.is_more_urgent_than(AlertPriority::P3));
+        assert!(!AlertPriority::P3.is_more_urgent_than(AlertPriority::P1));
+        // Equal priorities are not "more urgent" than each other.
+        assert!(!AlertPriority::P2.is_more_urgent_than(AlertPriority::P2));
+        // Guard against a naive `self < other` implementation: P1 has the
+        // SMALLEST id but the HIGHEST urgency.
+        assert!(AlertPriority::P1.to_i32() < AlertPriority::P5.to_i32());
+        assert!(AlertPriority::P1.is_more_urgent_than(AlertPriority::P5));
+    }
+
+    /// PT-4: the `"P3"` form is what reaches templates and the UI.
+    #[test]
+    fn test_display_uses_the_p_form() {
+        assert_eq!(AlertPriority::P1.as_str(), "P1");
+        assert_eq!(AlertPriority::P5.to_string(), "P5");
+        for p in ALL {
+            assert_eq!(p.to_string(), p.as_str());
+        }
+    }
+
+    #[test]
+    fn test_parses_both_p_form_and_bare_integer() {
+        assert_eq!(AlertPriority::from_str("P1").unwrap(), AlertPriority::P1);
+        assert_eq!(AlertPriority::from_str("p2").unwrap(), AlertPriority::P2);
+        assert_eq!(AlertPriority::from_str("3").unwrap(), AlertPriority::P3);
+        assert_eq!(AlertPriority::from_str(" P4 ").unwrap(), AlertPriority::P4);
+    }
+
+    #[test]
+    fn test_parse_rejects_junk_and_names_the_offender() {
+        for bad in ["", "P0", "P6", "0", "6", "banana", "P", "-1", "PP1", "1.0"] {
+            let err = AlertPriority::from_str(bad).unwrap_err();
+            // Match the BACKTICK-QUOTED form. A bare `contains(bad)` is
+            // satisfiable by the message's own "expected P1 through P5" text
+            // for inputs like "P", which made this assertion vacuous.
+            assert!(
+                err.to_string().contains(&format!("`{bad}`")),
+                "error for `{bad}` must name the offending value, got: {err}"
+            );
+        }
+    }
+
+    /// Defect #4: the reported value is the TRIMMED token, so the user sees
+    /// what was actually parsed rather than their incidental whitespace.
+    #[test]
+    fn test_parse_error_reports_the_trimmed_token() {
+        let err = AlertPriority::from_str("  P9  ").unwrap_err();
+        assert_eq!(err, PriorityError::OutOfRange("P9".to_string()));
+    }
+
+    /// Pins the equivalence that makes SQL `ORDER BY priority ASC` correct.
+    ///
+    /// NOTE: this does NOT — and cannot — enforce that `Ord` stays underived.
+    /// Stable Rust has no negative trait bound, so re-adding the derive would
+    /// fail nothing here. The prohibition lives as a comment on the type;
+    /// this test only guards the id/urgency relationship it rests on.
+    #[test]
+    fn test_smaller_storage_id_means_more_urgent() {
+        // If `Ord` were derived, this would be the tempting (and wrong) way to
+        // ask "is a more urgent than b" — it must be written explicitly.
+        assert!(AlertPriority::P1.is_more_urgent_than(AlertPriority::P2));
+        assert_eq!(
+            AlertPriority::P1.to_i32() < AlertPriority::P2.to_i32(),
+            AlertPriority::P1.is_more_urgent_than(AlertPriority::P2),
+            "smaller id == more urgent; this equivalence is what makes SQL ORDER BY ASC correct"
+        );
+    }
+
+    /// Serde is the INTEGER form: it matches the storage column, so the API
+    /// and the DB cannot drift apart.
+    ///
+    /// Every value is exercised because serde goes through `TryFrom<u8>` /
+    /// `From<AlertPriority> for u8` — implementations SEPARATE from
+    /// `to_i32`/`from_i32`. Testing one value would leave a wrong mapping for
+    /// P1, P4 or P5 undetected.
+    #[test]
+    fn test_serializes_as_an_integer_for_every_value() {
+        for (p, n) in ALL.iter().zip(1u8..=5) {
+            let json = serde_json::to_string(p).unwrap();
+            assert_eq!(json, n.to_string(), "{p:?} must serialize as {n}");
+
+            let back: AlertPriority = serde_json::from_str(&n.to_string()).unwrap();
+            assert_eq!(back, *p, "{n} must deserialize to {p:?}");
+        }
+    }
+
+    /// The serde path and the storage path must agree — they are different
+    /// code, and a divergence would silently corrupt rows on write.
+    #[test]
+    fn test_serde_and_storage_conversions_agree() {
+        for p in ALL {
+            let via_serde: u8 = p.into();
+            assert_eq!(
+                i32::from(via_serde),
+                p.to_i32(),
+                "serde and storage ids disagree for {p:?}"
+            );
+        }
+    }
+
+    #[test]
+    fn test_serde_rejects_out_of_range() {
+        assert!(serde_json::from_str::<AlertPriority>("0").is_err());
+        assert!(serde_json::from_str::<AlertPriority>("6").is_err());
+    }
+
+    /// Pins the serde PATTERN that `Alert.priority` must adopt: `None`
+    /// round-trips as *absent*, never as `0` or `null`.
+    ///
+    /// SCOPE WARNING: this uses a stand-in struct, so it proves nothing about
+    /// the real `Alert` — it passes today precisely BECAUSE `Alert` has no
+    /// `priority` field yet. The production contract (field present, correct
+    /// default, survives an update round-trip) belongs to the storage/CRUD
+    /// layer and is tracked separately. Do not read this as covering it.
+    #[test]
+    fn test_option_pattern_omits_unset_priority() {
+        #[derive(Serialize, Deserialize, PartialEq, Debug)]
+        struct Holder {
+            #[serde(default, skip_serializing_if = "Option::is_none")]
+            priority: Option<AlertPriority>,
+        }
+        let none = Holder { priority: None };
+        assert_eq!(serde_json::to_string(&none).unwrap(), "{}");
+        let parsed: Holder = serde_json::from_str("{}").unwrap();
+        assert_eq!(parsed, none);
+        let some = Holder {
+            priority: Some(AlertPriority::P2),
+        };
+        assert_eq!(serde_json::to_string(&some).unwrap(), r#"{"priority":2}"#);
+    }
+}

src/config/src/meta/alerts/state.rs

@@ -0,0 +1,607 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Durable per-alert run state — Part IV of `alerts.md`.
+//!
+//! This module holds the *pure* decision logic that governs the `alert_states`
+//! and `alert_state_transitions` tables. The tables themselves live in
+//! `infra::table::alert_states`; everything that decides **whether** and **what**
+//! to write lives here so it is unit-testable without a database.
+
+use serde::{Deserialize, Serialize};
+
+use crate::meta::{alerts::level::AlertLevel, self_reporting::usage::RunOutcome};
+
+/// `group_key` of the per-alert rollup row. Grouped monitors additionally get
+/// one row per label set; the rollup row is what list views read.
+pub const ROLLUP_GROUP_KEY: &str = "";
+
+/// Current run state for one `(alert_id, group_key)` pair.
+#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
+pub struct AlertState {
+    pub alert_id: String,
+    pub group_key: String,
+    /// `None` = never evaluated. Distinct from any real outcome.
+    pub last_outcome: Option<RunOutcome>,
+    pub last_outcome_at: Option<i64>,
+    /// When `last_outcome` last *changed*. Stable across repeated same-outcome runs.
+    pub since: Option<i64>,
+    // ── Level axis (alerts_2.md §7.2) ───────────────────────────────────────
+    // Independent of the outcome axis above: `firing -> notify_failed` moves
+    // `since` while the level (and `level_since`) stay put.
+    /// Severity of the last successful classification. `None` = no level
+    /// (single-level legacy alert, or never classified).
+    pub level: Option<AlertLevel>,
+    /// When `level` last *changed* — powers "critical for 20 minutes".
+    pub level_since: Option<i64>,
+    /// When `level` was last *computed* from a successful evaluation.
+    /// Freshness, not change-time: composite staleness (§6.4) runs on this, so
+    /// an alert erroring every minute cannot look fresh while its level rots.
+    pub level_at: Option<i64>,
+    /// Last evaluation that actually *included* this group (M-7).
+    ///
+    /// A separate clock from `last_outcome_at` on purpose. Resolving a vanished
+    /// group records a real outcome at the resolution time, so `last_outcome_at`
+    /// must advance — but the group was not seen then, so `last_seen` must not.
+    /// Overloading one field for both would either make the recovery row claim
+    /// a timestamp at which the group was still firing, or reset the
+    /// disappearance clock so the row could never be reaped.
+    pub last_seen: Option<i64>,
+    /// Rendered labels for UI and templates (M-4). `None` on the rollup row.
+    pub group_labels: Option<String>,
+    /// **Rollup row only**: the true number of groups the last evaluation
+    /// observed, before the M-6 cap truncated them. `None` on group rows.
+    ///
+    /// Persisted rather than recomputed because the cap-overflow warning has to
+    /// render from the stored row on list and detail views, long after the
+    /// evaluation that produced it. The retained row count cannot substitute —
+    /// it is post-cap, so an overflowing alert would report "500 of 500" and be
+    /// indistinguishable from one that never overflowed.
+    pub groups_observed: Option<usize>,
+    /// **Rollup row only**: how many of `groups_observed` were firing
+    /// (warning-or-worse), before the M-6 cap. `None` on group rows.
+    ///
+    /// Counted pre-cap for the same reason as `groups_observed`, and stored
+    /// separately from it because the "N of M groups firing" chip cannot be
+    /// derived from the retained rows: past the cap those are truncated, so
+    /// counting them under-reports exactly when the number matters most.
+    pub groups_firing: Option<usize>,
+    /// Whether `groups_observed` is a `≥` lower bound rather than exact — the
+    /// bounded fetch page came back full, so more groups may exist below it
+    /// (§5.3). `None` = written before this was tracked.
+    ///
+    /// Persisted rather than recomputed: exactness cannot be recovered later
+    /// from the count and a cap that is mutable config.
+    pub groups_observed_is_lower_bound: Option<bool>,
+    /// Whether `groups_firing` is a `≥` lower bound.
+    ///
+    /// Tracked separately from `groups_observed_is_lower_bound` because the two
+    /// genuinely diverge: a full page that reached healthy groups has seen
+    /// *every* firing group (the fetch is severity-ordered), so the firing
+    /// count is exact while the observed count is not.
+    pub groups_firing_is_lower_bound: Option<bool>,
+    // ── Per-group delivery state (alerts_2.md §5.5 MN-2) ────────────────────
+    // §7.1's `delivery_decision` fed per group. Deliberately NOT named
+    // `delivery_silenced_until`: that is the ScheduledTriggerData field
+    // non-multi alerts keep using — the state row is the per-group home.
+    //
+    // Written ONLY by the delivery callbacks (`dispatch::delivery_success_update`),
+    // never by evaluation: `apply_outcome` carries both forward untouched. That
+    // one-writer rule is what makes MN-6 hold — a failed send leaves them
+    // unadvanced, so the group re-qualifies on the next evaluation.
+    /// Per-group silence window: suppress same-level re-delivery until this
+    /// instant (micros). `None` = not silenced.
+    pub silenced_until: Option<i64>,
+    /// The level of this group's last *successful* delivery — what
+    /// `delivery_decision` measures escalation against.
+    pub last_notified_level: Option<AlertLevel>,
+}
+
+impl AlertState {
+    /// A row that has never been evaluated.
+    pub fn empty(alert_id: &str, group_key: &str) -> Self {
+        Self {
+            alert_id: alert_id.to_string(),
+            group_key: group_key.to_string(),
+            last_outcome: None,
+            last_outcome_at: None,
+            since: None,
+            level: None,
+            level_since: None,
+            level_at: None,
+            last_seen: None,
+            group_labels: None,
+            groups_observed: None,
+            groups_firing: None,
+            groups_observed_is_lower_bound: None,
+            groups_firing_is_lower_bound: None,
+            silenced_until: None,
+            last_notified_level: None,
+        }
+    }
+
+    /// True when the last recorded outcome was a firing one. `None` (never
+    /// evaluated) is not firing.
+    pub fn is_firing(&self) -> bool {
+        self.last_outcome.as_ref().is_some_and(|o| o.is_firing())
+    }
+}
+
+/// An append-only state change, written to `alert_state_transitions`.
+#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
+pub struct StateTransition {
+    pub alert_id: String,
+    pub group_key: String,
+    /// `None` on the first evaluation of an alert.
+    pub from_outcome: Option<RunOutcome>,
+    pub to_outcome: RunOutcome,
+    /// Level before/after. `None` when the alert has no level axis, or when
+    /// this transition was driven purely by an outcome change.
+    pub from_level: Option<AlertLevel>,
+    pub to_level: Option<AlertLevel>,
+    pub at: i64,
+    /// Observed value at transition time — the source for per-group history
+    /// (M-8, §7.2). `None` where no value was observed, which includes the
+    /// disappearance transition: a group that stopped being returned has no
+    /// value, and recording 0 would render as a real measurement.
+    pub value: Option<f64>,
+    /// Rendered labels, duplicated from the state row on purpose: the state row
+    /// is reaped after the grace period (M-7) while transitions are retained,
+    /// and `group_key` is a hash. Without this, history outlives the only thing
+    /// that could say which host it was about.
+    pub group_labels: Option<String>,
+}
+
+/// What `apply_outcome` decided to do about an observed evaluation result.
+#[derive(Clone, Debug, PartialEq)]
+pub struct StateUpdate {
+    /// The row to persist. `None` means "write nothing" — the observation is
+    /// not allowed to overwrite existing state.
+    pub state: Option<AlertState>,
+    /// Emitted only when the outcome actually changed.
+    pub transition: Option<StateTransition>,
+}
+
+impl StateUpdate {
+    /// A decision to persist nothing at all.
+    pub fn noop() -> Self {
+        Self {
+            state: None,
+            transition: None,
+        }
+    }
+
+    pub fn is_noop(&self) -> bool {
+        self.state.is_none() && self.transition.is_none()
+    }
+}
+
+/// Whether an observed outcome is allowed to overwrite stored state.
+///
+/// `Skipped` means the alert was never evaluated (silenced, paused, org
+/// deleting). Letting it overwrite would erase a real firing state, so skipped
+/// runs are dropped entirely.
+pub fn should_persist(outcome: &RunOutcome) -> bool {
+    !matches!(outcome, RunOutcome::Skipped)
+}
+
+/// Fold an observed outcome into the previous state.
+///
+/// `alert_id` and `group_key` are always supplied by the caller — they must not
+/// be recovered from `prev`, which is `None` on an alert's first ever
+/// evaluation. Deriving them would write identity-less rows that can never be
+/// joined back to their alert.
+///
+/// - `Skipped` observations are dropped (see [`should_persist`]).
+/// - A changed outcome moves `since` and emits a transition.
+/// - A repeated outcome refreshes `last_outcome_at` but leaves `since` alone and emits no
+///   transition — this is what keeps writes transition-bounded.
+pub fn apply_outcome(
+    alert_id: &str,
+    group_key: &str,
+    prev: Option<&AlertState>,
+    outcome: RunOutcome,
+    level: Option<AlertLevel>,
+    at: i64,
+) -> StateUpdate {
+    if !should_persist(&outcome) {
+        return StateUpdate::noop();
+    }
+
+    debug_assert!(
+        !alert_id.is_empty(),
+        "apply_outcome requires a non-empty alert_id"
+    );
+
+    let alert_id = alert_id.to_string();
+    let group_key = group_key.to_string();
+
+    // ── Outcome axis ────────────────────────────────────────────────────────
+    let previous_outcome = prev.and_then(|p| p.last_outcome.clone());
+    let outcome_changed = previous_outcome.as_ref() != Some(&outcome);
+    let since = if outcome_changed {
+        Some(at)
+    } else {
+        prev.and_then(|p| p.since).or(Some(at))
+    };
+
+    // ── Level axis (independent of the above) ───────────────────────────────
+    let previous_level = prev.and_then(|p| p.level);
+    let (level, level_since, level_at, level_changed) = match level {
+        // No level computed this run — e.g. a query error. Carry the whole
+        // level axis forward untouched, including freshness: an evaluation
+        // that observed nothing must not make the level look fresh.
+        None => (
+            previous_level,
+            prev.and_then(|p| p.level_since),
+            prev.and_then(|p| p.level_at),
+            false,
+        ),
+        Some(new_level) => {
+            let changed = previous_level != Some(new_level);
+            let level_since = if changed {
+                Some(at)
+            } else {
+                prev.and_then(|p| p.level_since).or(Some(at))
+            };
+            // Freshness always advances on a successful classification.
+            (Some(new_level), level_since, Some(at), changed)
+        }
+    };
+
+    let state = AlertState {
+        alert_id: alert_id.clone(),
+        group_key: group_key.clone(),
+        last_outcome: Some(outcome.clone()),
+        last_outcome_at: Some(at),
+        since,
+        level,
+        level_since,
+        level_at,
+        // An observation, by definition, saw the group.
+        last_seen: Some(at),
+        group_labels: prev.and_then(|p| p.group_labels.clone()),
+        // Set by the per-group planner on the rollup row only.
+        groups_observed: prev.and_then(|p| p.groups_observed),
+        groups_firing: prev.and_then(|p| p.groups_firing),
+        groups_observed_is_lower_bound: prev.and_then(|p| p.groups_observed_is_lower_bound),
+        groups_firing_is_lower_bound: prev.and_then(|p| p.groups_firing_is_lower_bound),
+        // Delivery state is the callbacks' to write; evaluation only carries it.
+        silenced_until: prev.and_then(|p| p.silenced_until),
+        last_notified_level: prev.and_then(|p| p.last_notified_level),
+    };
+
+    // A change on EITHER axis is a transition — an escalation while still
+    // `firing` must be recorded, as must a delivery failure at a steady level.
+    let transition = (outcome_changed || level_changed).then_some(StateTransition {
+        alert_id,
+        group_key,
+        from_outcome: previous_outcome,
+        to_outcome: outcome,
+        from_level: previous_level,
+        to_level: level,
+        at,
+        // Filled by the per-group planner, which is the only caller that has
+        // an observed value and a label set.
+        value: None,
+        group_labels: None,
+    });
+
+    StateUpdate {
+        state: Some(state),
+        transition,
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn prev(outcome: RunOutcome, at: i64, since: i64) -> AlertState {
+        AlertState {
+            alert_id: "alert-1".to_string(),
+            group_key: ROLLUP_GROUP_KEY.to_string(),
+            last_outcome: Some(outcome),
+            last_outcome_at: Some(at),
+            since: Some(since),
+            level: None,
+            level_since: None,
+            level_at: None,
+            last_seen: Some(at),
+            group_labels: None,
+            groups_observed: None,
+            groups_firing: None,
+            groups_observed_is_lower_bound: None,
+            groups_firing_is_lower_bound: None,
+            silenced_until: None,
+            last_notified_level: None,
+        }
+    }
+
+    #[test]
+    fn test_rollup_group_key_is_empty_string() {
+        assert_eq!(ROLLUP_GROUP_KEY, "");
+    }
+
+    #[test]
+    fn test_empty_state_is_never_evaluated() {
+        let s = AlertState::empty("alert-1", ROLLUP_GROUP_KEY);
+        assert_eq!(s.last_outcome, None);
+        assert_eq!(s.last_outcome_at, None);
+        assert_eq!(s.since, None);
+        assert!(!s.is_firing());
+    }
+
+    #[test]
+    fn test_is_firing_tracks_run_outcome() {
+        assert!(prev(RunOutcome::Firing, 10, 10).is_firing());
+        // notify_failed still fired — see RunOutcome::is_firing.
+        assert!(prev(RunOutcome::NotifyFailed, 10, 10).is_firing());
+        assert!(!prev(RunOutcome::Normal, 10, 10).is_firing());
+        assert!(!prev(RunOutcome::Error, 10, 10).is_firing());
+    }
+
+    // ── should_persist ──────────────────────────────────────────────────────
+
+    #[test]
+    fn test_should_persist_rejects_only_skipped() {
+        assert!(!should_persist(&RunOutcome::Skipped));
+
+        assert!(should_persist(&RunOutcome::Firing));
+        assert!(should_persist(&RunOutcome::Normal));
+        assert!(should_persist(&RunOutcome::Succeeded));
+        assert!(should_persist(&RunOutcome::Error));
+        assert!(should_persist(&RunOutcome::NotifyFailed));
+    }
+
+    // ── apply_outcome ───────────────────────────────────────────────────────
+
+    #[test]
+    fn test_skipped_never_overwrites_existing_state() {
+        let existing = prev(RunOutcome::Firing, 100, 100);
+        let update = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&existing),
+            RunOutcome::Skipped,
+            None,
+            200,
+        );
+
+        assert!(
+            update.is_noop(),
+            "a silenced run must not erase a firing state"
+        );
+    }
+
+    #[test]
+    fn test_skipped_on_fresh_alert_writes_nothing() {
+        let update = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            None,
+            RunOutcome::Skipped,
+            None,
+            100,
+        );
+        assert!(update.is_noop());
+    }
+
+    #[test]
+    fn test_first_evaluation_creates_state_and_transition() {
+        let update = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            None,
+            RunOutcome::Firing,
+            None,
+            100,
+        );
+
+        let state = update.state.expect("first evaluation must persist state");
+        assert_eq!(state.last_outcome, Some(RunOutcome::Firing));
+        assert_eq!(state.last_outcome_at, Some(100));
+        assert_eq!(state.since, Some(100));
+        assert_eq!(state.group_key, ROLLUP_GROUP_KEY);
+        // Identity must come from the caller, not from `prev` (which is None
+        // here). An empty alert_id would orphan the row permanently.
+        assert_eq!(
+            state.alert_id, "alert-1",
+            "first-evaluation rows must carry their alert_id"
+        );
+
+        let t = update
+            .transition
+            .expect("first evaluation is a transition from nothing");
+        assert_eq!(t.from_outcome, None);
+        assert_eq!(t.to_outcome, RunOutcome::Firing);
+        assert_eq!(t.at, 100);
+        assert_eq!(t.alert_id, "alert-1");
+    }
+
+    #[test]
+    fn test_repeated_same_outcome_emits_no_transition() {
+        let existing = prev(RunOutcome::Normal, 100, 100);
+        let update = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&existing),
+            RunOutcome::Normal,
+            None,
+            200,
+        );
+
+        let state = update.state.expect("state should still refresh");
+        assert_eq!(
+            state.last_outcome_at,
+            Some(200),
+            "last_outcome_at tracks the latest run"
+        );
+        assert_eq!(
+            state.since,
+            Some(100),
+            "since must NOT move when the outcome is unchanged"
+        );
+        assert!(
+            update.transition.is_none(),
+            "unchanged outcome must not write a transition row"
+        );
+    }
+
+    #[test]
+    fn test_outcome_change_moves_since_and_emits_transition() {
+        let existing = prev(RunOutcome::Normal, 100, 50);
+        let update = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&existing),
+            RunOutcome::Firing,
+            None,
+            200,
+        );
+
+        let state = update.state.unwrap();
+        assert_eq!(state.last_outcome, Some(RunOutcome::Firing));
+        assert_eq!(state.since, Some(200), "since moves on a real change");
+
+        let t = update.transition.expect("changed outcome must transition");
+        assert_eq!(t.from_outcome, Some(RunOutcome::Normal));
+        assert_eq!(t.to_outcome, RunOutcome::Firing);
+        assert_eq!(t.at, 200);
+    }
+
+    #[test]
+    fn test_recovery_transition_is_recorded() {
+        let existing = prev(RunOutcome::Firing, 100, 100);
+        let update = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&existing),
+            RunOutcome::Normal,
+            None,
+            300,
+        );
+
+        let t = update
+            .transition
+            .expect("firing -> normal is the recovery event");
+        assert_eq!(t.from_outcome, Some(RunOutcome::Firing));
+        assert_eq!(t.to_outcome, RunOutcome::Normal);
+        assert!(!update.state.unwrap().is_firing());
+    }
+
+    /// `firing` -> `notify_failed` is a change in outcome even though both are
+    /// firing states, so it transitions.
+    #[test]
+    fn test_firing_to_notify_failed_transitions() {
+        let existing = prev(RunOutcome::Firing, 100, 100);
+        let update = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&existing),
+            RunOutcome::NotifyFailed,
+            None,
+            200,
+        );
+
+        let state = update.state.unwrap();
+        assert!(state.is_firing(), "notify_failed is still a firing state");
+        assert_eq!(state.since, Some(200));
+        assert!(update.transition.is_some());
+    }
+
+    #[test]
+    fn test_state_preserves_identity_from_previous_row() {
+        let existing = prev(RunOutcome::Normal, 100, 100);
+        let update = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&existing),
+            RunOutcome::Firing,
+            None,
+            200,
+        );
+
+        let state = update.state.unwrap();
+        assert_eq!(state.alert_id, "alert-1");
+        assert_eq!(state.group_key, ROLLUP_GROUP_KEY);
+
+        let t = update.transition.unwrap();
+        assert_eq!(t.alert_id, "alert-1");
+        assert_eq!(t.group_key, ROLLUP_GROUP_KEY);
+    }
+
+    #[test]
+    fn test_evaluation_carries_delivery_state_forward_untouched() {
+        // §5.5 MN-2/MN-6: delivery state has ONE writer — the delivery
+        // callbacks. If `apply_outcome` reset these on every evaluation, each
+        // run would erase the silence window and the alert would page every
+        // cycle regardless of silence; if it defaulted them on a fresh clone,
+        // same result.
+        let mut previous = prev(RunOutcome::Firing, 100, 100);
+        previous.silenced_until = Some(9_999);
+        previous.last_notified_level = Some(AlertLevel::Critical);
+
+        let update = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&previous),
+            RunOutcome::Firing,
+            Some(AlertLevel::Critical),
+            200,
+        );
+        let state = update.state.expect("an observed run writes state");
+
+        assert_eq!(state.silenced_until, Some(9_999));
+        assert_eq!(state.last_notified_level, Some(AlertLevel::Critical));
+    }
+
+    #[test]
+    fn test_per_group_state_is_keyed_independently() {
+        let pod_a = AlertState {
+            alert_id: "alert-1".to_string(),
+            group_key: "pod=a".to_string(),
+            last_outcome: Some(RunOutcome::Firing),
+            last_outcome_at: Some(100),
+            since: Some(100),
+            level: None,
+            level_since: None,
+            level_at: None,
+            last_seen: Some(100),
+            group_labels: None,
+            groups_observed: None,
+            groups_firing: None,
+            groups_observed_is_lower_bound: None,
+            groups_firing_is_lower_bound: None,
+            silenced_until: None,
+            last_notified_level: None,
+        };
+        let update = apply_outcome(
+            "alert-1",
+            "pod=a",
+            Some(&pod_a),
+            RunOutcome::Normal,
+            None,
+            200,
+        );
+        let state = update.state.unwrap();
+
+        assert_eq!(
+            state.group_key, "pod=a",
+            "per-group rows must keep their own key"
+        );
+        assert_eq!(update.transition.unwrap().group_key, "pod=a");
+    }
+}

src/config/src/meta/alerts/state_level.rs

@@ -0,0 +1,298 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Level tracking on alert state — §7.2 of `alerts_2.md`.
+//!
+//! TDD: tests only. Extends the shipped `alerts::state` module (Part IV of
+//! `alerts.md`) with the level axis. `apply_outcome` gains a level parameter
+//! and `AlertState` gains `level` + `level_since`.
+
+#[cfg(test)]
+mod tests {
+    use crate::meta::{
+        alerts::{
+            level::AlertLevel,
+            state::{AlertState, ROLLUP_GROUP_KEY, apply_outcome},
+        },
+        self_reporting::usage::RunOutcome,
+    };
+
+    /// Existing state at a given outcome+level, with independent `since` values.
+    fn existing(
+        outcome: RunOutcome,
+        outcome_since: i64,
+        level: AlertLevel,
+        level_since: i64,
+    ) -> AlertState {
+        AlertState {
+            alert_id: "alert-1".to_string(),
+            group_key: ROLLUP_GROUP_KEY.to_string(),
+            last_outcome: Some(outcome),
+            last_outcome_at: Some(outcome_since),
+            since: Some(outcome_since),
+            level: Some(level),
+            level_since: Some(level_since),
+            // Freshness: when the level was last COMPUTED (§7.6). Starts equal
+            // to level_since; diverges as same-level runs refresh it.
+            level_at: Some(level_since),
+            last_seen: Some(outcome_since),
+            group_labels: None,
+            groups_observed: None,
+            groups_firing: None,
+            groups_observed_is_lower_bound: None,
+            groups_firing_is_lower_bound: None,
+            silenced_until: None,
+            last_notified_level: None,
+        }
+    }
+
+    // ── The two axes must move independently ────────────────────────────────
+    // This is the defect caught reviewing alerts_2.md: reusing one `since`
+    // column for both outcome and level silently resets "critical for 20
+    // minutes" whenever the OUTCOME changes but the level does not.
+
+    #[test]
+    fn test_outcome_change_at_same_level_does_not_move_level_since() {
+        // firing -> notify_failed: the delivery broke, the severity did not.
+        let prev = existing(RunOutcome::Firing, 100, AlertLevel::Critical, 100);
+        let u = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&prev),
+            RunOutcome::NotifyFailed,
+            Some(AlertLevel::Critical),
+            500,
+        );
+
+        let s = u.state.expect("outcome change must persist");
+        assert_eq!(
+            s.since,
+            Some(500),
+            "outcome `since` moves on outcome change"
+        );
+        assert_eq!(
+            s.level_since,
+            Some(100),
+            "level_since must NOT move — it has been Critical since 100"
+        );
+    }
+
+    #[test]
+    fn test_level_change_at_same_outcome_moves_only_level_since() {
+        // Warning -> Critical while still `firing`: escalation.
+        let prev = existing(RunOutcome::Firing, 100, AlertLevel::Warning, 200);
+        let u = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&prev),
+            RunOutcome::Firing,
+            Some(AlertLevel::Critical),
+            600,
+        );
+
+        let s = u.state.expect("level change must persist");
+        assert_eq!(
+            s.since,
+            Some(100),
+            "outcome is unchanged (still firing), so its `since` holds"
+        );
+        assert_eq!(s.level_since, Some(600), "level_since moves on escalation");
+        assert_eq!(s.level, Some(AlertLevel::Critical));
+    }
+
+    #[test]
+    fn test_first_evaluation_sets_both_since_values() {
+        let u = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            None,
+            RunOutcome::Firing,
+            Some(AlertLevel::Warning),
+            100,
+        );
+        let s = u.state.unwrap();
+        assert_eq!(s.since, Some(100));
+        assert_eq!(s.level_since, Some(100));
+        assert_eq!(s.level_at, Some(100), "first computation sets freshness");
+        assert_eq!(s.level, Some(AlertLevel::Warning));
+        assert_eq!(s.alert_id, "alert-1", "identity comes from the caller");
+    }
+
+    // ── Transitions carry the level change ──────────────────────────────────
+
+    #[test]
+    fn test_escalation_emits_a_transition_with_both_levels() {
+        let prev = existing(RunOutcome::Firing, 100, AlertLevel::Warning, 100);
+        let u = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&prev),
+            RunOutcome::Firing,
+            Some(AlertLevel::Critical),
+            300,
+        );
+
+        let t = u
+            .transition
+            .expect("a level change is a transition even when the outcome is unchanged");
+        assert_eq!(t.from_level, Some(AlertLevel::Warning));
+        assert_eq!(t.to_level, Some(AlertLevel::Critical));
+        assert_eq!(t.at, 300);
+    }
+
+    #[test]
+    fn test_recovery_to_ok_emits_a_transition() {
+        let prev = existing(RunOutcome::Firing, 100, AlertLevel::Critical, 100);
+        let u = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&prev),
+            RunOutcome::Normal,
+            Some(AlertLevel::Ok),
+            900,
+        );
+
+        let t = u.transition.expect("recovery is a transition");
+        assert_eq!(t.from_level, Some(AlertLevel::Critical));
+        assert_eq!(t.to_level, Some(AlertLevel::Ok));
+        assert_eq!(u.state.unwrap().level_since, Some(900));
+    }
+
+    #[test]
+    fn test_repeated_same_level_and_outcome_emits_no_transition() {
+        // The transition-bounded write property from Part IV must survive the
+        // level axis: a steady Critical writes no new transition rows.
+        let prev = existing(RunOutcome::Firing, 100, AlertLevel::Critical, 100);
+        let u = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&prev),
+            RunOutcome::Firing,
+            Some(AlertLevel::Critical),
+            700,
+        );
+
+        assert!(
+            u.transition.is_none(),
+            "unchanged outcome AND level must not write a transition"
+        );
+        let s = u.state.expect("state still refreshes");
+        assert_eq!(s.last_outcome_at, Some(700), "freshness advances");
+        assert_eq!(s.since, Some(100));
+        assert_eq!(s.level_since, Some(100));
+    }
+
+    // ── skipped still never overwrites (Part IV rule, now with a level) ─────
+
+    #[test]
+    fn test_skipped_does_not_erase_level() {
+        let prev = existing(RunOutcome::Firing, 100, AlertLevel::Critical, 100);
+        let u = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&prev),
+            RunOutcome::Skipped,
+            None,
+            900,
+        );
+        assert!(
+            u.is_noop(),
+            "a silenced run must not erase a Critical level any more than it erases a firing outcome"
+        );
+    }
+
+    // ── Level is optional: single-level alerts pre-Feature-1 ────────────────
+
+    #[test]
+    fn test_none_level_is_accepted_for_alerts_without_levels() {
+        // Alerts written before Feature 1, or non-condition modules, have no
+        // level. That must not be conflated with Ok.
+        let u = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            None,
+            RunOutcome::Firing,
+            None,
+            100,
+        );
+        let s = u.state.unwrap();
+        assert_eq!(s.level, None, "no level is distinct from AlertLevel::Ok");
+        assert_eq!(s.level_since, None);
+    }
+
+    // ── level_at: the freshness clock composites depend on (§7.6) ───────────
+
+    #[test]
+    fn test_error_outcome_preserves_level_and_does_not_refresh_level_at() {
+        // A query error made no valid observation: the level axis must be
+        // completely untouched — value, since, AND freshness. If level_at were
+        // refreshed here, a child erroring every minute would look "fresh" to
+        // composites while its level rots (the exact bug §6.4 guards against).
+        let prev = existing(RunOutcome::Firing, 100, AlertLevel::Critical, 100);
+        let u = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&prev),
+            RunOutcome::Error,
+            None, // the caller has no level to offer — the query failed
+            900,
+        );
+
+        let s = u.state.expect("the outcome axis still records the error");
+        assert_eq!(s.last_outcome, Some(RunOutcome::Error));
+        assert_eq!(s.since, Some(900), "outcome changed firing -> error");
+        assert_eq!(
+            s.level,
+            Some(AlertLevel::Critical),
+            "error must not erase the last known level"
+        );
+        assert_eq!(s.level_since, Some(100));
+        assert_eq!(
+            s.level_at,
+            Some(100),
+            "freshness must NOT advance on an evaluation that computed nothing"
+        );
+    }
+
+    #[test]
+    fn test_successful_same_level_run_refreshes_level_at_only() {
+        // Steady Critical: freshness advances, change-time does not.
+        let prev = existing(RunOutcome::Firing, 100, AlertLevel::Critical, 100);
+        let u = apply_outcome(
+            "alert-1",
+            ROLLUP_GROUP_KEY,
+            Some(&prev),
+            RunOutcome::Firing,
+            Some(AlertLevel::Critical),
+            700,
+        );
+
+        let s = u.state.unwrap();
+        assert_eq!(s.level_at, Some(700), "recomputed this run -> fresh");
+        assert_eq!(s.level_since, Some(100), "unchanged level -> since holds");
+        assert!(u.transition.is_none());
+    }
+
+    #[test]
+    fn test_is_firing_reads_outcome_not_level() {
+        // `AlertState::is_firing` predates levels and must keep answering the
+        // OUTCOME question, so existing callers do not silently change meaning.
+        let s = existing(RunOutcome::Firing, 100, AlertLevel::Warning, 100);
+        assert!(s.is_firing());
+
+        let s = existing(RunOutcome::Normal, 100, AlertLevel::Ok, 100);
+        assert!(!s.is_firing());
+    }
+}

src/config/src/meta/alerts/tags.rs

@@ -0,0 +1,634 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Alert tags — Feature 2 (PT-6, PT-7, PT-8).
+//!
+//! Tags are the **selection primitive**: normalized, validated, and used to
+//! filter the alert list and (later) to scope composite membership and
+//! muting. They are NOT `context_attributes` — that field is free-form KV
+//! shipped into notification payloads, with no validation and a different
+//! purpose. The two stay separate.
+//!
+//! Format: bare (`prod`) or `key:value` (`service:checkout`).
+//!
+//! **Unicode-aware (D22).** We have international users, so `café`,
+//! `münchen` and `日本` are valid tags: a tag starts with a Unicode *letter*
+//! and its body is Unicode alphanumerics plus `_ - . / :`.
+//!
+//! Normalization **repairs** rather than rejects wherever the intent is
+//! unambiguous — case and surrounding whitespace are fixed silently, because
+//! rejecting `Prod` would be a papercut with no upside. Anything that changes
+//! meaning (illegal characters, a leading digit, over-length) is an error
+//! that names the offending tag.
+//!
+//! **Deliberate divergence from Datadog (D22):** they silently rewrite
+//! unsupported characters to `_`. We reject, because a silent rewrite changes
+//! the tag's *selector identity* — a filter that used to match quietly stops
+//! matching. Import-time normalization belongs in the importer, where a
+//! transformation is expected and can be reported to the user.
+
+use std::collections::HashSet;
+
+/// Longest single tag, in **characters** (`chars().count()`), not bytes.
+///
+/// With Unicode tags (D22) the two differ — `日本語` is 3 characters but 9
+/// bytes — so byte-length would silently impose a much shorter limit on
+/// non-Latin scripts.
+pub const MAX_TAG_LEN: usize = 200;
+
+/// Most tags on one alert, measured on what would be STORED (i.e. after
+/// de-duplication). Our own operational cap — unbounded tag lists are a
+/// payload and UI hazard, and nothing legitimate needs more.
+pub const MAX_TAGS: usize = 64;
+
+/// Hard bound on the RAW input length, checked before any normalization.
+///
+/// [`MAX_TAGS`] is measured after de-duplication, which means a caller could
+/// otherwise force unbounded lowercasing/validating work with a giant list of
+/// duplicates before being rejected. Generous enough that no legitimate
+/// request trips it.
+pub const MAX_INPUT_TAGS: usize = MAX_TAGS * 10;
+
+/// Why a tag list was rejected. Every variant carries the offending tag so
+/// the API can tell the user exactly which one to fix (PT-7).
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum TagError {
+    /// Does not begin with a Unicode letter.
+    MustStartWithLetter(String),
+    /// Contains a character that is neither a Unicode alphanumeric nor one of
+    /// `_ - . / :`.
+    IllegalCharacter { tag: String, ch: char },
+    /// Longer than [`MAX_TAG_LEN`] CHARACTERS (not bytes).
+    TooLong { tag: String, len: usize },
+    /// More than [`MAX_TAGS`] tags would be stored (counted AFTER
+    /// de-duplication, so the number is what the user would end up with).
+    TooMany(usize),
+    /// The raw input exceeded [`MAX_INPUT_TAGS`] before normalization.
+    TooManyRaw(usize),
+}
+
+impl std::fmt::Display for TagError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::MustStartWithLetter(t) => {
+                write!(f, "tag `{t}` must start with a letter")
+            }
+            Self::IllegalCharacter { tag, ch } => {
+                write!(
+                    f,
+                    "tag `{tag}` contains an illegal character `{ch}`; allowed: letters, digits, and _ - . / :"
+                )
+            }
+            Self::TooLong { tag, len } => {
+                write!(
+                    f,
+                    "tag `{tag}` is {len} characters; the maximum is {MAX_TAG_LEN}"
+                )
+            }
+            Self::TooMany(n) => {
+                write!(f, "{n} distinct tags; the maximum is {MAX_TAGS} per alert")
+            }
+            Self::TooManyRaw(n) => {
+                write!(
+                    f,
+                    "{n} tags supplied; refusing to process more than {MAX_INPUT_TAGS} entries"
+                )
+            }
+        }
+    }
+}
+
+impl std::error::Error for TagError {}
+
+/// Normalize and validate a tag list for storage (PT-7).
+///
+/// Repairs silently: lowercases, trims surrounding whitespace, drops entries
+/// that are empty once trimmed (a trailing comma in the UI is not an error),
+/// and de-duplicates **after** normalization, keeping first-seen order so the
+/// stored list is stable and predictable.
+///
+/// Errors on anything that would change meaning, naming the offending tag.
+///
+/// **Validation order is part of the contract** so error messages are stable:
+///
+/// 1. raw input bound ([`MAX_INPUT_TAGS`]) — cheapest, before any work;
+/// 2. per tag, in order: trim → drop-if-empty → Unicode-lowercase → must-start-with-a-letter →
+///    charset → character length. Structural problems are reported before length, because "you used
+///    a `!`" is more actionable than "your 300-character string is too long";
+/// 3. de-duplicate;
+/// 4. stored-count cap ([`MAX_TAGS`]).
+pub fn normalize_tags(tags: &[String]) -> Result<Vec<String>, TagError> {
+    // 1. Raw bound first — before any per-entry work, so a duplicate-stuffed payload cannot force
+    //    unbounded normalization (the MAX_TAGS cap is measured post-dedup and would be reached too
+    //    late).
+    if tags.len() > MAX_INPUT_TAGS {
+        return Err(TagError::TooManyRaw(tags.len()));
+    }
+
+    let mut out: Vec<String> = Vec::new();
+    let mut seen: HashSet<String> = HashSet::new();
+
+    for raw in tags {
+        // 2. Repairs that cannot change meaning.
+        let trimmed = raw.trim();
+        if trimmed.is_empty() {
+            continue; // separator artifact, not a user error
+        }
+        // Unicode lowercase, NOT full case folding: folding maps `ß` -> `ss`,
+        // silently rewriting the user's tag and changing its selector
+        // identity (D22).
+        let tag = trimmed.to_lowercase();
+
+        validate_tag(&tag)?;
+
+        // 3. De-duplicate, first-seen order wins so storage is stable.
+        if seen.insert(tag.clone()) {
+            out.push(tag);
+        }
+    }
+
+    // 4. Cap what would be STORED, so a list that only exceeds it before de-duplication still
+    //    saves.
+    if out.len() > MAX_TAGS {
+        return Err(TagError::TooMany(out.len()));
+    }
+    Ok(out)
+}
+
+/// Per-tag rules, applied in the documented order: structural problems are
+/// reported before length, because "you used a `!`" points at the real fix.
+fn validate_tag(tag: &str) -> Result<(), TagError> {
+    match tag.chars().next() {
+        // Unicode letter (D22) — `café`, `münchen`, `日本` all qualify.
+        Some(c) if c.is_alphabetic() => {}
+        _ => return Err(TagError::MustStartWithLetter(tag.to_string())),
+    }
+
+    for ch in tag.chars() {
+        // "Unicode" means letters and digits, not "any codepoint": symbols
+        // and emoji stay illegal.
+        if !(ch.is_alphanumeric() || matches!(ch, '_' | '-' | '.' | '/' | ':')) {
+            return Err(TagError::IllegalCharacter {
+                tag: tag.to_string(),
+                ch,
+            });
+        }
+    }
+
+    // CHARACTERS, not bytes — byte length would give non-Latin scripts a
+    // fraction of the allowance.
+    let len = tag.chars().count();
+    if len > MAX_TAG_LEN {
+        return Err(TagError::TooLong {
+            tag: tag.to_string(),
+            len,
+        });
+    }
+    Ok(())
+}
+
+/// Normalize tags for a FILTER (`?tags=a,b`) — lenient counterpart of
+/// [`normalize_tags`].
+///
+/// A filter must never 400. An unparseable or unknown tag simply matches
+/// nothing, which is the honest answer to "show me alerts tagged `!!!`".
+/// This applies the same case/whitespace repair so a user typing `Prod` in
+/// the URL still matches the stored `prod` — the bug this exists to prevent
+/// is a filter that silently returns zero rows because the caller forgot to
+/// normalize.
+///
+/// **Invalid tokens are KEPT, never dropped.** Dropping them would be a
+/// privilege-escalating bug rather than a nicety: `?tags=!!!` would normalize
+/// to an EMPTY filter, and an empty filter matches *every* alert — so a
+/// request that should return nothing would return everything. A preserved
+/// `!!!` cannot match any stored tag (storage rejects that character), which
+/// yields the correct empty result. Only blank entries — separator artifacts
+/// from `?tags=a,,b` — are removed.
+pub fn normalize_filter_tags(tags: &[String]) -> Vec<String> {
+    let mut out: Vec<String> = Vec::new();
+    let mut seen: HashSet<String> = HashSet::new();
+    for raw in tags {
+        let trimmed = raw.trim();
+        if trimmed.is_empty() {
+            continue; // `?tags=a,,b` — separator artifact
+        }
+        // NOTE: deliberately no `validate_tag` call. Keeping an invalid token
+        // is what makes `?tags=!!!` match nothing instead of everything.
+        let tag = trimmed.to_lowercase();
+        if seen.insert(tag.clone()) {
+            out.push(tag);
+        }
+    }
+    out
+}
+
+/// AND semantics for the list filter (PT-8): every requested tag must be
+/// present on the alert. An empty filter matches everything.
+///
+/// Both sides are expected to be normalized already — stored tags by
+/// [`normalize_tags`] at save, the filter by [`normalize_filter_tags`] at
+/// parse. This is a pure containment check.
+pub fn matches_all_tags(alert_tags: &[String], filter: &[String]) -> bool {
+    // AND: every requested tag must be present. Whole-token equality, never
+    // substring — `service` must not match `service:checkout` (D20).
+    filter
+        .iter()
+        .all(|wanted| alert_tags.iter().any(|have| have == wanted))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn v(items: &[&str]) -> Vec<String> {
+        items.iter().map(|s| s.to_string()).collect()
+    }
+
+    // ── Accepted shapes ─────────────────────────────────────────────────────
+
+    #[test]
+    fn test_bare_and_key_value_tags_are_both_valid() {
+        let out = normalize_tags(&v(&["prod", "service:checkout"])).unwrap();
+        assert_eq!(out, v(&["prod", "service:checkout"]));
+    }
+
+    #[test]
+    fn test_full_legal_punctuation_set_is_accepted() {
+        // _ - . / : — all of it, in one tag.
+        let tag = "web.server-01/prod:v2_x";
+        let out = normalize_tags(&v(&[tag])).unwrap();
+        assert_eq!(out, v(&[tag]));
+    }
+
+    // ── Unicode (D22) ───────────────────────────────────────────────────────
+
+    /// We have international users: accented Latin, Cyrillic and CJK tags are
+    /// first-class, not an edge case.
+    #[test]
+    fn test_unicode_letters_are_valid_tags() {
+        for tag in ["café", "münchen", "日本", "team:москва", "ключ:значение"]
+        {
+            assert!(
+                normalize_tags(&v(&[tag])).is_ok(),
+                "`{tag}` must be a valid tag under D22"
+            );
+        }
+    }
+
+    #[test]
+    fn test_unicode_tag_may_start_with_a_non_ascii_letter() {
+        assert_eq!(
+            normalize_tags(&v(&["日本:tokyo"])).unwrap(),
+            v(&["日本:tokyo"])
+        );
+    }
+
+    /// Case handling is Unicode-aware, so dedup works across scripts.
+    #[test]
+    fn test_unicode_case_is_lowercased_and_deduplicated() {
+        let out = normalize_tags(&v(&["CAFÉ", "café", "Café"])).unwrap();
+        assert_eq!(out, v(&["café"]), "one distinct tag after lowercasing");
+    }
+
+    /// Unicode **lowercasing**, deliberately NOT full case folding (D22).
+    ///
+    /// Full folding maps `ß` → `ss`, so `Straße` would be stored as
+    /// `strasse` — silently rewriting what the user typed, which is exactly
+    /// the selector-identity change D22 rejects. `CAFÉ`/`café` alone cannot
+    /// tell the two policies apart; this case can.
+    #[test]
+    fn test_lowercasing_is_not_full_case_folding() {
+        let sharp_s = normalize_tags(&v(&["Straße"])).unwrap();
+        assert_eq!(sharp_s, v(&["straße"]), "ß must be preserved, not expanded");
+
+        let expanded = normalize_tags(&v(&["STRASSE"])).unwrap();
+        assert_eq!(expanded, v(&["strasse"]));
+
+        assert_ne!(
+            sharp_s, expanded,
+            "these are DISTINCT tags; folding them together would rewrite the user's tag"
+        );
+    }
+
+    /// Symbols and emoji are still illegal — "Unicode" means letters and
+    /// digits, not "anything".
+    #[test]
+    fn test_unicode_does_not_mean_symbols_are_allowed() {
+        for bad in ["emoji🔥", "a→b", "a✓"] {
+            assert!(
+                matches!(
+                    normalize_tags(&v(&[bad])).unwrap_err(),
+                    TagError::IllegalCharacter { .. }
+                ),
+                "`{bad}` must still be rejected"
+            );
+        }
+    }
+
+    /// The length limit counts CHARACTERS. Measured in bytes, a 200-character
+    /// CJK tag would be ~600 and get rejected — silently giving non-Latin
+    /// scripts a third of the allowance.
+    #[test]
+    fn test_length_limit_counts_characters_not_bytes() {
+        let cjk = format!("日{}", "本".repeat(MAX_TAG_LEN - 1));
+        assert_eq!(cjk.chars().count(), MAX_TAG_LEN);
+        assert!(cjk.len() > MAX_TAG_LEN, "precondition: bytes exceed chars");
+        assert!(
+            normalize_tags(&v(&[&cjk])).is_ok(),
+            "200 characters must pass regardless of byte length"
+        );
+
+        let over = format!("日{}", "本".repeat(MAX_TAG_LEN));
+        match normalize_tags(&v(&[&over])).unwrap_err() {
+            TagError::TooLong { len, .. } => {
+                assert_eq!(len, MAX_TAG_LEN + 1, "len must be a CHARACTER count")
+            }
+            other => panic!("expected TooLong, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn test_empty_input_is_valid_and_yields_no_tags() {
+        assert_eq!(normalize_tags(&[]).unwrap(), Vec::<String>::new());
+    }
+
+    // ── Silent repairs ──────────────────────────────────────────────────────
+
+    #[test]
+    fn test_case_is_normalized_rather_than_rejected() {
+        let out = normalize_tags(&v(&["Prod", "SERVICE:Checkout"])).unwrap();
+        assert_eq!(out, v(&["prod", "service:checkout"]));
+    }
+
+    #[test]
+    fn test_surrounding_whitespace_is_trimmed() {
+        let out = normalize_tags(&v(&["  prod  ", "\tenv:dev\n"])).unwrap();
+        assert_eq!(out, v(&["prod", "env:dev"]));
+    }
+
+    /// A trailing comma in the tag input yields an empty entry; that is a UI
+    /// artifact, not a user error.
+    #[test]
+    fn test_blank_entries_are_dropped_not_rejected() {
+        let out = normalize_tags(&v(&["prod", "", "   ", "dev"])).unwrap();
+        assert_eq!(out, v(&["prod", "dev"]));
+    }
+
+    #[test]
+    fn test_duplicates_collapse_after_normalization_keeping_first_order() {
+        let out = normalize_tags(&v(&["prod", "Prod", " PROD ", "dev", "prod"])).unwrap();
+        assert_eq!(out, v(&["prod", "dev"]), "first-seen order must be stable");
+    }
+
+    // ── Rejections, each naming the offending tag ───────────────────────────
+
+    #[test]
+    fn test_tag_must_start_with_a_letter() {
+        for bad in [
+            "1prod", "_prod", ":prod", "-prod", ".prod", "/prod", "1日本", "٣test",
+        ] {
+            let err = normalize_tags(&v(&[bad])).unwrap_err();
+            assert_eq!(err, TagError::MustStartWithLetter(bad.to_string()));
+            assert!(
+                err.to_string().contains(bad),
+                "message must name the tag, got: {err}"
+            );
+        }
+    }
+
+    #[test]
+    fn test_illegal_characters_are_rejected() {
+        for bad in ["prod!", "a b", "emoji🔥", "a,b", "a=b", "a#b"] {
+            let err = normalize_tags(&v(&[bad])).unwrap_err();
+            match &err {
+                TagError::IllegalCharacter { tag, .. } => assert_eq!(tag, bad),
+                other => panic!("expected IllegalCharacter for `{bad}`, got {other:?}"),
+            }
+            assert!(err.to_string().contains(bad));
+        }
+    }
+
+    /// A comma is the list separator in `?tags=a,b`; allowing it inside a tag
+    /// would make the filter ambiguous.
+    #[test]
+    fn test_comma_is_illegal_because_it_separates_the_filter_list() {
+        assert!(matches!(
+            normalize_tags(&v(&["a,b"])).unwrap_err(),
+            TagError::IllegalCharacter { .. }
+        ));
+    }
+
+    #[test]
+    fn test_length_boundary_is_inclusive() {
+        let at_limit = format!("a{}", "b".repeat(MAX_TAG_LEN - 1));
+        assert_eq!(at_limit.len(), MAX_TAG_LEN);
+        assert!(normalize_tags(&v(&[&at_limit])).is_ok(), "200 chars is OK");
+
+        let over = format!("a{}", "b".repeat(MAX_TAG_LEN));
+        match normalize_tags(&v(&[&over])).unwrap_err() {
+            TagError::TooLong { len, .. } => assert_eq!(len, MAX_TAG_LEN + 1),
+            other => panic!("expected TooLong, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn test_tag_count_cap_is_inclusive() {
+        let ok: Vec<String> = (0..MAX_TAGS).map(|i| format!("tag{i}")).collect();
+        assert_eq!(normalize_tags(&ok).unwrap().len(), MAX_TAGS);
+
+        // 70 raw entries that collapse to 65 distinct tags. Using 65 UNIQUE
+        // inputs would leave the count ambiguous — input-count and stored-count
+        // would both be 65, so the test would pass under either reading.
+        let mut too_many: Vec<String> = (0..MAX_TAGS + 1).map(|i| format!("tag{i}")).collect();
+        for i in 0..5 {
+            too_many.push(format!("TAG{i}")); // duplicates after lowercasing
+        }
+        assert_eq!(too_many.len(), MAX_TAGS + 6);
+        assert_eq!(
+            normalize_tags(&too_many).unwrap_err(),
+            TagError::TooMany(MAX_TAGS + 1),
+            "the count must be what would be STORED (65), not what was sent (70)"
+        );
+    }
+
+    /// The cap applies to what is STORED, so a list that only exceeds it
+    /// before de-duplication must pass.
+    #[test]
+    fn test_cap_is_measured_after_deduplication() {
+        let mut dupes: Vec<String> = (0..MAX_TAGS).map(|i| format!("tag{i}")).collect();
+        dupes.push("tag0".to_string());
+        dupes.push("TAG1".to_string());
+        let out = normalize_tags(&dupes).expect("duplicates must not trip the cap");
+        assert_eq!(out.len(), MAX_TAGS);
+    }
+
+    // ── Validation ORDER is part of the contract (defect #4) ────────────────
+
+    /// A tag that is BOTH over-length and illegal must report the illegal
+    /// character: "you used a `!`" is actionable, "your 300-char string is
+    /// too long" sends the user to fix the wrong thing.
+    #[test]
+    fn test_structural_errors_are_reported_before_length() {
+        let bad = format!("a{}!", "b".repeat(MAX_TAG_LEN));
+        match normalize_tags(&v(&[&bad])).unwrap_err() {
+            TagError::IllegalCharacter { ch, .. } => assert_eq!(ch, '!'),
+            other => panic!("expected IllegalCharacter to win over TooLong, got {other:?}"),
+        }
+    }
+
+    /// Leading-character is checked before charset, so `1a!` names the more
+    /// fundamental problem rather than the incidental `!`.
+    #[test]
+    fn test_start_check_precedes_charset_check() {
+        assert_eq!(
+            normalize_tags(&v(&["1a!"])).unwrap_err(),
+            TagError::MustStartWithLetter("1a!".to_string())
+        );
+    }
+
+    /// An over-cap list containing an invalid tag reports the INVALID TAG:
+    /// trimming the list would not make the bad tag legal.
+    #[test]
+    fn test_invalid_tag_is_reported_before_the_stored_count_cap() {
+        let mut many: Vec<String> = (0..MAX_TAGS + 1).map(|i| format!("tag{i}")).collect();
+        many.push("bad!".to_string());
+        assert!(matches!(
+            normalize_tags(&many).unwrap_err(),
+            TagError::IllegalCharacter { .. }
+        ));
+    }
+
+    /// ...but an absurd RAW payload is refused before any of that work.
+    #[test]
+    fn test_raw_input_bound_precedes_all_per_tag_work() {
+        let huge: Vec<String> = (0..MAX_INPUT_TAGS + 1)
+            .map(|_| "bad!".to_string())
+            .collect();
+        assert_eq!(
+            normalize_tags(&huge).unwrap_err(),
+            TagError::TooManyRaw(MAX_INPUT_TAGS + 1),
+            "a giant payload must be refused before per-tag validation runs"
+        );
+    }
+
+    #[test]
+    fn test_raw_bound_is_inclusive_and_dedup_still_applies_under_it() {
+        // MAX_INPUT_TAGS identical entries: passes the raw bound, dedupes to 1.
+        let at_bound: Vec<String> = (0..MAX_INPUT_TAGS).map(|_| "prod".to_string()).collect();
+        assert_eq!(normalize_tags(&at_bound).unwrap(), v(&["prod"]));
+    }
+
+    // ── Lenient filter parsing (defect #5) ──────────────────────────────────
+
+    /// The filter path must never error — an unknown or malformed tag simply
+    /// matches nothing.
+    #[test]
+    fn test_filter_parsing_is_lenient_where_saving_is_strict() {
+        let out = normalize_filter_tags(&v(&["Prod", "  env:DEV  ", "", "!!!"]));
+        // Case/whitespace repaired identically to the save path...
+        assert!(out.contains(&"prod".to_string()));
+        assert!(out.contains(&"env:dev".to_string()));
+        // ...blanks dropped, and the malformed entry is NOT an error...
+        assert!(!out.contains(&"".to_string()));
+        // ...but the malformed entry is RETAINED as a token (see below).
+        assert!(
+            out.contains(&"!!!".to_string()),
+            "invalid tokens must survive normalization, got {out:?}"
+        );
+    }
+
+    /// REGRESSION GUARD for a privilege-escalating bug: if the filter parser
+    /// DROPPED invalid tokens, `?tags=!!!` would become the empty filter —
+    /// and the empty filter matches every alert. A request that must return
+    /// nothing would instead return everything.
+    #[test]
+    fn test_invalid_only_filter_matches_nothing_not_everything() {
+        let stored = normalize_tags(&v(&["prod", "service:checkout"])).unwrap();
+        let filter = normalize_filter_tags(&v(&["!!!"]));
+        assert!(
+            !filter.is_empty(),
+            "must not collapse to the match-all filter"
+        );
+        assert!(
+            !matches_all_tags(&stored, &filter),
+            "an invalid-only filter must match nothing"
+        );
+    }
+
+    /// The AND semantics must also hold when one term is valid and one is not.
+    #[test]
+    fn test_mixed_valid_and_invalid_filter_matches_nothing() {
+        let stored = normalize_tags(&v(&["prod"])).unwrap();
+        let filter = normalize_filter_tags(&v(&["prod", "!!!"]));
+        assert!(
+            !matches_all_tags(&stored, &filter),
+            "one unsatisfiable term must fail the whole AND filter"
+        );
+    }
+
+    /// Blanks ARE dropped — `?tags=a,,b` is a separator artifact, not a
+    /// request for a tag named "". An all-blank filter is "no filter".
+    #[test]
+    fn test_blank_only_filter_is_treated_as_no_filter() {
+        let filter = normalize_filter_tags(&v(&["", "   ", ""]));
+        assert!(filter.is_empty());
+        assert!(matches_all_tags(&v(&["prod"]), &filter));
+    }
+
+    /// The bug this pairing exists to prevent: a user types `Prod` in the URL
+    /// and gets zero rows because the stored form is `prod`.
+    #[test]
+    fn test_filter_normalization_makes_mixed_case_urls_match_stored_tags() {
+        let stored = normalize_tags(&v(&["Prod", "Service:Checkout"])).unwrap();
+        let filter = normalize_filter_tags(&v(&["PROD", "service:CHECKOUT"]));
+        assert!(
+            matches_all_tags(&stored, &filter),
+            "stored {stored:?} should match filter {filter:?}"
+        );
+    }
+
+    // ── Filtering (PT-8) ────────────────────────────────────────────────────
+
+    #[test]
+    fn test_filter_requires_every_tag_and_semantics() {
+        let alert = v(&["prod", "service:checkout", "team:payments"]);
+        assert!(matches_all_tags(&alert, &v(&["prod"])));
+        assert!(matches_all_tags(&alert, &v(&["prod", "team:payments"])));
+        // AND, not OR: one missing tag fails the whole filter.
+        assert!(!matches_all_tags(&alert, &v(&["prod", "env:staging"])));
+        assert!(!matches_all_tags(&alert, &v(&["nope"])));
+    }
+
+    #[test]
+    fn test_empty_filter_matches_everything() {
+        assert!(matches_all_tags(&v(&["prod"]), &[]));
+        assert!(matches_all_tags(&[], &[]));
+    }
+
+    #[test]
+    fn test_untagged_alert_matches_no_non_empty_filter() {
+        assert!(!matches_all_tags(&[], &v(&["prod"])));
+    }
+
+    /// Tags are exact tokens, not substrings: `service:checkout` must not be
+    /// matched by `service` or by `checkout`.
+    #[test]
+    fn test_filter_matches_whole_tags_not_substrings() {
+        let alert = v(&["service:checkout"]);
+        assert!(!matches_all_tags(&alert, &v(&["service"])));
+        assert!(!matches_all_tags(&alert, &v(&["checkout"])));
+        assert!(matches_all_tags(&alert, &v(&["service:checkout"])));
+    }
+}

src/config/src/meta/mod.rs

@@ -43,6 +43,7 @@ pub mod service_graph;
 pub mod service_streams;
 pub mod session;
 pub mod short_url;
+pub mod slo;
 pub mod sql;
 pub mod stream;
 pub mod synthetics;

src/config/src/meta/self_reporting/mod.rs

@@ -277,7 +277,7 @@ impl ReportingRunner {
 #[cfg(test)]
 mod tests {
     use tokio::time::Duration;
-    use usage::{TriggerData, TriggerDataStatus, TriggerDataType, UsageData, UsageEvent};
+    use usage::{RunOutcome, TriggerData, TriggerDataType, UsageData, UsageEvent};
 
     use super::*;
 
@@ -361,7 +361,7 @@ mod tests {
             next_run_at: 1234567890,
             is_realtime: true,
             is_silenced: false,
-            status: TriggerDataStatus::Completed,
+            status: RunOutcome::Succeeded,
             start_time: 1234567890,
             end_time: 1234567890,
             retries: 0,
@@ -380,6 +380,12 @@ mod tests {
             dedup_count: None,
             grouped: None,
             group_size: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            level: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
 
         let result = queue
@@ -395,7 +401,7 @@ mod tests {
                 assert_eq!(data.org, "test_org");
                 assert_eq!(data.module, TriggerDataType::Alert);
                 assert_eq!(data.key, "test_key");
-                assert_eq!(data.status, TriggerDataStatus::Completed);
+                assert_eq!(data.status, RunOutcome::Succeeded);
             }
             _ => panic!("Expected Trigger data"),
         }
@@ -895,7 +901,7 @@ mod tests {
             next_run_at: 1234567890,
             is_realtime: true,
             is_silenced: false,
-            status: TriggerDataStatus::Completed,
+            status: RunOutcome::Succeeded,
             start_time: 1234567890,
             end_time: 1234567890,
             retries: 0,
@@ -914,6 +920,12 @@ mod tests {
             dedup_count: None,
             grouped: None,
             group_size: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            level: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
 
         let error_data = error::ErrorData {
@@ -963,7 +975,7 @@ mod tests {
             next_run_at: 1234567890,
             is_realtime: true,
             is_silenced: false,
-            status: TriggerDataStatus::Completed,
+            status: RunOutcome::Succeeded,
             start_time: 1234567890,
             end_time: 1234567890,
             retries: 0,
@@ -982,6 +994,12 @@ mod tests {
             dedup_count: None,
             grouped: None,
             group_size: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            level: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
 
         let trigger_data2 = TriggerData {
@@ -1021,7 +1039,7 @@ mod tests {
             next_run_at: 1234567890,
             is_realtime: true,
             is_silenced: false,
-            status: TriggerDataStatus::Completed,
+            status: RunOutcome::Succeeded,
             start_time: 1234567890,
             end_time: 1234567890,
             retries: 0,
@@ -1040,6 +1058,12 @@ mod tests {
             dedup_count: None,
             grouped: None,
             group_size: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            level: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
 
         // Should succeed when queue has space

src/config/src/meta/self_reporting/usage.rs

@@ -49,16 +49,173 @@ pub fn is_reserved_self_reporting_stream(stream_name: &str) -> bool {
     RESERVED_SELF_REPORTING_STREAMS.contains(&stream_name)
 }
 
+/// Every reserved internal stream, self-reporting or otherwise.
+///
+/// `slo_slices` is not self-reporting — it is measurement data for Feature 5 —
+/// but it needs the identical protection, and for the identical reason: a user
+/// write into it would corrupt the numbers an SLO reports. It is listed
+/// separately rather than folded into [`RESERVED_SELF_REPORTING_STREAMS`] so
+/// that array keeps meaning what its name says.
+pub const RESERVED_INTERNAL_STREAMS: [&str; 6] = [
+    USAGE_STREAM,
+    STATS_STREAM,
+    TRIGGERS_STREAM,
+    ERROR_STREAM,
+    DATA_RETENTION_USAGE_STREAM,
+    crate::meta::slo::stream::SLO_SLICES_STREAM,
+];
+
+/// Returns true if `stream_name` is reserved for internal writes of any kind.
+///
+/// This is the predicate the create/delete/ingest guards should use. Internal
+/// writers bypass it the same way self-reporting does — via the
+/// `IngestionRequest::Usage` channel, for which `should_report_usage()` is
+/// false.
+pub fn is_reserved_internal_stream(stream_name: &str) -> bool {
+    RESERVED_INTERNAL_STREAMS.contains(&stream_name)
+}
+
+/// Outcome of a single scheduled evaluation — "did it fire?".
+///
+/// Part III of `alerts.md`. Replaces the former `TriggerDataStatus`, whose
+/// `Completed` variant collided by name with `TriggerStatus::Completed` (the
+/// job-queue state) while meaning something entirely different.
+///
+/// Condition-bearing modules (alerts, derived streams, anomaly detection) use
+/// [`RunOutcome::Firing`] / [`RunOutcome::Normal`]; modules with no condition
+/// (reports, workflows, synthetics) use [`RunOutcome::Succeeded`].
+///
+/// The legacy vocabulary is accepted on **read** via serde aliases so that
+/// `TriggerData` records written by an older build — which are read back as a
+/// typed struct from the on-disk self-reporting queue — do not panic the ingest
+/// task after an upgrade. Legacy values are never written. Because `completed`
+/// was module-dependent it aliases to the neutral `Succeeded` and is corrected
+/// by [`TriggerData::normalize_legacy_outcome`].
 #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
-pub enum TriggerDataStatus {
-    #[serde(rename = "completed")]
-    Completed,
-    #[serde(rename = "failed")]
-    Failed,
-    #[serde(rename = "condition_not_satisfied")]
-    ConditionNotSatisfied,
+pub enum RunOutcome {
+    /// Condition matched — the alert triggered and the notification was sent.
+    #[serde(rename = "firing")]
+    Firing,
+    /// Evaluated cleanly; nothing to alert on.
+    #[serde(rename = "normal", alias = "condition_not_satisfied")]
+    Normal,
+    /// Non-condition modules: report sent, derived stream written.
+    #[serde(rename = "succeeded", alias = "completed")]
+    Succeeded,
+    /// The evaluation itself failed (query error, timeout).
+    #[serde(rename = "error", alias = "failed")]
+    Error,
+    /// Never evaluated — silenced, paused, org deleting.
     #[serde(rename = "skipped")]
     Skipped,
+    /// Condition matched but delivery failed. Still a firing state: without
+    /// this, a webhook outage silently undercounts firings.
+    #[serde(rename = "notify_failed")]
+    NotifyFailed,
+}
+
+impl RunOutcome {
+    /// True when the alert actually triggered. `NotifyFailed` counts — the
+    /// condition matched, only delivery failed.
+    pub fn is_firing(&self) -> bool {
+        matches!(self, Self::Firing | Self::NotifyFailed)
+    }
+
+    /// Durable integer form, stored in `alert_states.last_outcome` (Part IV).
+    ///
+    /// These values are persisted — never reorder or reuse them.
+    pub fn to_i32(&self) -> i32 {
+        match self {
+            Self::Firing => 0,
+            Self::Normal => 1,
+            Self::Succeeded => 2,
+            Self::Error => 3,
+            Self::Skipped => 4,
+            Self::NotifyFailed => 5,
+        }
+    }
+
+    pub fn from_i32(v: i32) -> Option<Self> {
+        match v {
+            0 => Some(Self::Firing),
+            1 => Some(Self::Normal),
+            2 => Some(Self::Succeeded),
+            3 => Some(Self::Error),
+            4 => Some(Self::Skipped),
+            5 => Some(Self::NotifyFailed),
+            _ => None,
+        }
+    }
+
+    pub fn as_str(&self) -> &'static str {
+        match self {
+            Self::Firing => "firing",
+            Self::Normal => "normal",
+            Self::Succeeded => "succeeded",
+            Self::Error => "error",
+            Self::Skipped => "skipped",
+            Self::NotifyFailed => "notify_failed",
+        }
+    }
+}
+
+impl std::fmt::Display for RunOutcome {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.write_str(self.as_str())
+    }
+}
+
+/// Parse `anomalies_found` out of an anomaly run's `success_response` JSON.
+/// A missing or unparseable payload counts as zero — never as firing.
+fn anomalies_found(success_response: Option<&str>) -> i64 {
+    success_response
+        .and_then(|s| serde_json::from_str::<serde_json::Value>(s).ok())
+        .and_then(|v| v["anomalies_found"].as_i64())
+        .unwrap_or(0)
+}
+
+/// Map a raw `status` string from the `triggers` stream onto a [`RunOutcome`].
+///
+/// Handles both vocabularies, so it works across the retention window in which
+/// pre- and post-cutover rows coexist. Returns `None` for values that belong to
+/// neither.
+///
+/// This is the read-side migration described in Part III of `alerts.md`. It is
+/// temporary: once legacy rows age out of the triggers stream's retention
+/// window, only the passthrough branch is reachable and the rest can be deleted.
+pub fn normalize_outcome(
+    raw: &str,
+    module: &TriggerDataType,
+    success_response: Option<&str>,
+) -> Option<RunOutcome> {
+    match raw.to_lowercase().as_str() {
+        // ── current vocabulary ──
+        "firing" => Some(RunOutcome::Firing),
+        "normal" => Some(RunOutcome::Normal),
+        "succeeded" => Some(RunOutcome::Succeeded),
+        "error" => Some(RunOutcome::Error),
+        "skipped" => Some(RunOutcome::Skipped),
+        "notify_failed" => Some(RunOutcome::NotifyFailed),
+
+        // ── legacy vocabulary ──
+        "condition_not_satisfied" => Some(RunOutcome::Normal),
+        "failed" => Some(RunOutcome::Error),
+        "completed" => Some(match module {
+            // Anomaly rows stored `completed` whenever detection RAN, even with
+            // zero anomalies — the count lives in `success_response`.
+            TriggerDataType::AnomalyDetection => {
+                if anomalies_found(success_response) > 0 {
+                    RunOutcome::Firing
+                } else {
+                    RunOutcome::Normal
+                }
+            }
+            m if m.is_condition_bearing() => RunOutcome::Firing,
+            _ => RunOutcome::Succeeded,
+        }),
+
+        _ => None,
+    }
 }
 
 #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
@@ -81,6 +238,21 @@ pub enum TriggerDataType {
     Workflow,
     #[serde(rename = "synthetics")]
     Synthetics,
+    #[serde(rename = "slo")]
+    Slo,
+    #[serde(rename = "slo_backfill")]
+    SloBackfill,
+}
+
+impl TriggerDataType {
+    /// Whether this module evaluates a condition, and so can meaningfully be
+    /// "firing". Modules without one report [`RunOutcome::Succeeded`] instead.
+    pub fn is_condition_bearing(&self) -> bool {
+        matches!(
+            self,
+            Self::Alert | Self::DerivedStream | Self::AnomalyDetection
+        )
+    }
 }
 
 #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
@@ -93,7 +265,7 @@ pub struct TriggerData {
     pub next_run_at: i64,
     pub is_realtime: bool,
     pub is_silenced: bool,
-    pub status: TriggerDataStatus,
+    pub status: RunOutcome,
     pub start_time: i64,
     pub end_time: i64,
     pub retries: i32,
@@ -119,6 +291,33 @@ pub struct TriggerData {
     pub grouped: Option<bool>,
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub group_size: Option<i32>,
+    // ── Value context (T-9, alerts_2.md §7.5) ───────────────────────────────
+    // Lets history answer "fired at 112 against threshold 100" from the stream
+    // alone. `#[serde(default)]` is load-bearing: records written before these
+    // fields existed must still deserialize.
+    /// Observed value: row count for count alerts, `alert_agg_value` for
+    /// aggregation alerts. For count alerts this is a LOWER BOUND once the
+    /// search cap is reached.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub actual_value: Option<f64>,
+    /// The threshold that matched. `None` on `normal` rows.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub threshold_value: Option<f64>,
+    /// Operator, so the record reads standalone ("112 >= 100").
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub threshold_operator: Option<String>,
+    /// Computed `AlertLevel` as i32 — `Ok` on normal rows, never absent for a
+    /// level-bearing evaluation. `None` for non-condition modules and
+    /// error/skipped runs.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub level: Option<i32>,
+    /// Which group produced `actual_value` (worst group; D8).
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub group_label: Option<String>,
+    /// True when `actual_value` is a LOWER BOUND (legacy SingleQuery count
+    /// fetch hit its cap, §7.5) — history renders "≥ N". Absent = exact.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub value_is_lower_bound: Option<bool>,
 }
 
 impl Default for TriggerData {
@@ -131,7 +330,7 @@ impl Default for TriggerData {
             next_run_at: 0,
             is_realtime: false,
             is_silenced: false,
-            status: TriggerDataStatus::Completed,
+            status: RunOutcome::Succeeded,
             start_time: 0,
             end_time: 0,
             retries: 0,
@@ -150,6 +349,12 @@ impl Default for TriggerData {
             dedup_count: None,
             grouped: None,
             group_size: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            level: None,
+            group_label: None,
+            value_is_lower_bound: None,
         }
     }
 }
@@ -172,7 +377,7 @@ impl TriggerData {
             next_run_at: 0,
             is_realtime: false,
             is_silenced: false,
-            status: TriggerDataStatus::Completed,
+            status: RunOutcome::Succeeded,
             start_time: 0,
             end_time: 0,
             retries: 0,
@@ -192,9 +397,38 @@ impl TriggerData {
             dedup_count: Some(0),
             grouped: Some(false),
             group_size: Some(0),
+            actual_value: Some(0.0),
+            threshold_value: Some(0.0),
+            threshold_operator: Some(String::new()),
+            level: Some(0),
+            group_label: Some(String::new()),
+            value_is_lower_bound: Some(false),
         }
     }
 
+    /// Correct a legacy `completed` outcome that was read in through the
+    /// [`RunOutcome::Succeeded`] serde alias.
+    ///
+    /// Call this after deserializing a `TriggerData` that may have been written
+    /// by an older build (see `self_reporting::persistence`). Safe and
+    /// idempotent: current code never writes `Succeeded` for a condition-bearing
+    /// module, so a `Succeeded` on one of those can only be a legacy record.
+    pub fn normalize_legacy_outcome(&mut self) {
+        if self.status != RunOutcome::Succeeded || !self.module.is_condition_bearing() {
+            return;
+        }
+        self.status = match self.module {
+            TriggerDataType::AnomalyDetection => {
+                if anomalies_found(self.success_response.as_deref()) > 0 {
+                    RunOutcome::Firing
+                } else {
+                    RunOutcome::Normal
+                }
+            }
+            _ => RunOutcome::Firing,
+        };
+    }
+
     /// Returns all field names for TriggerData struct by introspecting a sample instance.
     ///
     /// This is primarily used for testing/validation. For schema creation, use
@@ -652,6 +886,549 @@ impl From<FileMeta> for RequestStats {
     }
 }
 
+#[cfg(test)]
+mod run_outcome_tests {
+    use super::*;
+
+    // ── Serialization: the wire vocabulary ──────────────────────────────────
+
+    #[test]
+    fn test_run_outcome_serialization() {
+        assert_eq!(
+            serde_json::to_string(&RunOutcome::Firing).unwrap(),
+            "\"firing\""
+        );
+        assert_eq!(
+            serde_json::to_string(&RunOutcome::Normal).unwrap(),
+            "\"normal\""
+        );
+        assert_eq!(
+            serde_json::to_string(&RunOutcome::Succeeded).unwrap(),
+            "\"succeeded\""
+        );
+        assert_eq!(
+            serde_json::to_string(&RunOutcome::Error).unwrap(),
+            "\"error\""
+        );
+        assert_eq!(
+            serde_json::to_string(&RunOutcome::Skipped).unwrap(),
+            "\"skipped\""
+        );
+        assert_eq!(
+            serde_json::to_string(&RunOutcome::NotifyFailed).unwrap(),
+            "\"notify_failed\""
+        );
+    }
+
+    #[test]
+    fn test_run_outcome_deserialization_roundtrip() {
+        for outcome in [
+            RunOutcome::Firing,
+            RunOutcome::Normal,
+            RunOutcome::Succeeded,
+            RunOutcome::Error,
+            RunOutcome::Skipped,
+            RunOutcome::NotifyFailed,
+        ] {
+            let s = serde_json::to_string(&outcome).unwrap();
+            let back: RunOutcome = serde_json::from_str(&s).unwrap();
+            assert_eq!(outcome, back, "roundtrip failed for {outcome:?}");
+        }
+    }
+
+    /// The old vocabulary MUST still deserialize. `TriggerData` is read back as a
+    /// typed struct from the on-disk self-reporting queue
+    /// (`persistence.rs:121`, an `.unwrap()`), so records written by a previous
+    /// build must not panic the ingest task after an upgrade.
+    #[test]
+    fn test_run_outcome_accepts_legacy_values_leniently() {
+        assert_eq!(
+            serde_json::from_str::<RunOutcome>("\"condition_not_satisfied\"").unwrap(),
+            RunOutcome::Normal
+        );
+        assert_eq!(
+            serde_json::from_str::<RunOutcome>("\"failed\"").unwrap(),
+            RunOutcome::Error
+        );
+        // `completed` is module-dependent, so it lands on the neutral variant and
+        // is corrected by `TriggerData::normalize_legacy_outcome`.
+        assert_eq!(
+            serde_json::from_str::<RunOutcome>("\"completed\"").unwrap(),
+            RunOutcome::Succeeded
+        );
+    }
+
+    /// Legacy values must never be *written* — aliases are read-only.
+    #[test]
+    fn test_run_outcome_never_serializes_legacy_values() {
+        for outcome in [RunOutcome::Normal, RunOutcome::Error, RunOutcome::Succeeded] {
+            let s = serde_json::to_string(&outcome).unwrap();
+            assert!(
+                !["\"completed\"", "\"failed\"", "\"condition_not_satisfied\""]
+                    .contains(&s.as_str()),
+                "{outcome:?} must not serialize to a legacy value, got {s}"
+            );
+        }
+    }
+
+    /// A full legacy `TriggerData` payload must survive the round trip that
+    /// `persistence.rs` performs, and land on the right outcome per module.
+    #[test]
+    fn test_trigger_data_normalize_legacy_outcome() {
+        // Condition-bearing module: legacy `completed` really meant "fired".
+        let mut td = TriggerData {
+            module: TriggerDataType::Alert,
+            status: serde_json::from_str("\"completed\"").unwrap(),
+            ..Default::default()
+        };
+        td.normalize_legacy_outcome();
+        assert_eq!(td.status, RunOutcome::Firing);
+
+        // Anomaly with no anomalies found is NOT firing.
+        let mut td = TriggerData {
+            module: TriggerDataType::AnomalyDetection,
+            status: serde_json::from_str("\"completed\"").unwrap(),
+            success_response: Some(r#"{"anomalies_found":0}"#.to_string()),
+            ..Default::default()
+        };
+        td.normalize_legacy_outcome();
+        assert_eq!(td.status, RunOutcome::Normal);
+
+        let mut td = TriggerData {
+            module: TriggerDataType::AnomalyDetection,
+            status: serde_json::from_str("\"completed\"").unwrap(),
+            success_response: Some(r#"{"anomalies_found":7}"#.to_string()),
+            ..Default::default()
+        };
+        td.normalize_legacy_outcome();
+        assert_eq!(td.status, RunOutcome::Firing);
+
+        // Non-condition module: `succeeded` is correct and must be left alone.
+        let mut td = TriggerData {
+            module: TriggerDataType::Report,
+            status: serde_json::from_str("\"completed\"").unwrap(),
+            ..Default::default()
+        };
+        td.normalize_legacy_outcome();
+        assert_eq!(td.status, RunOutcome::Succeeded);
+    }
+
+    /// The fixup must be idempotent and must not corrupt new-vocabulary records.
+    #[test]
+    fn test_normalize_legacy_outcome_is_idempotent_and_safe() {
+        for outcome in [
+            RunOutcome::Firing,
+            RunOutcome::Normal,
+            RunOutcome::Error,
+            RunOutcome::Skipped,
+            RunOutcome::NotifyFailed,
+        ] {
+            let mut td = TriggerData {
+                module: TriggerDataType::Alert,
+                status: outcome.clone(),
+                ..Default::default()
+            };
+            td.normalize_legacy_outcome();
+            assert_eq!(td.status, outcome, "fixup must not alter {outcome:?}");
+            td.normalize_legacy_outcome();
+            assert_eq!(td.status, outcome, "fixup must be idempotent");
+        }
+    }
+
+    // ── is_firing: the defect fix from Part III ─────────────────────────────
+
+    #[test]
+    fn test_is_firing() {
+        assert!(RunOutcome::Firing.is_firing());
+        // The whole point of `notify_failed`: the alert DID fire, delivery did
+        // not. It must still count toward firing totals.
+        assert!(RunOutcome::NotifyFailed.is_firing());
+
+        assert!(!RunOutcome::Normal.is_firing());
+        assert!(!RunOutcome::Succeeded.is_firing());
+        assert!(!RunOutcome::Error.is_firing());
+        assert!(!RunOutcome::Skipped.is_firing());
+    }
+
+    // ── Integer mapping: needed for the Part IV `last_outcome INT` column ────
+
+    /// These integers are DURABLE — they are what lands in the Part IV
+    /// `last_outcome INT` column. Pin the literal values: a roundtrip-only test
+    /// still passes if the variants are reordered, at which point every stored
+    /// row silently changes meaning.
+    #[test]
+    fn test_run_outcome_i32_values_are_pinned() {
+        assert_eq!(RunOutcome::Firing.to_i32(), 0);
+        assert_eq!(RunOutcome::Normal.to_i32(), 1);
+        assert_eq!(RunOutcome::Succeeded.to_i32(), 2);
+        assert_eq!(RunOutcome::Error.to_i32(), 3);
+        assert_eq!(RunOutcome::Skipped.to_i32(), 4);
+        assert_eq!(RunOutcome::NotifyFailed.to_i32(), 5);
+
+        assert_eq!(RunOutcome::from_i32(0), Some(RunOutcome::Firing));
+        assert_eq!(RunOutcome::from_i32(1), Some(RunOutcome::Normal));
+        assert_eq!(RunOutcome::from_i32(2), Some(RunOutcome::Succeeded));
+        assert_eq!(RunOutcome::from_i32(3), Some(RunOutcome::Error));
+        assert_eq!(RunOutcome::from_i32(4), Some(RunOutcome::Skipped));
+        assert_eq!(RunOutcome::from_i32(5), Some(RunOutcome::NotifyFailed));
+    }
+
+    #[test]
+    fn test_run_outcome_i32_roundtrip() {
+        for outcome in [
+            RunOutcome::Firing,
+            RunOutcome::Normal,
+            RunOutcome::Succeeded,
+            RunOutcome::Error,
+            RunOutcome::Skipped,
+            RunOutcome::NotifyFailed,
+        ] {
+            let n = outcome.to_i32();
+            assert_eq!(
+                RunOutcome::from_i32(n),
+                Some(outcome.clone()),
+                "i32 roundtrip failed for {outcome:?} (got {n})"
+            );
+        }
+    }
+
+    #[test]
+    fn test_run_outcome_from_i32_rejects_unknown() {
+        assert_eq!(RunOutcome::from_i32(-1), None);
+        assert_eq!(RunOutcome::from_i32(99), None);
+    }
+
+    // ── Condition-bearing modules ───────────────────────────────────────────
+
+    #[test]
+    fn test_is_condition_bearing() {
+        assert!(TriggerDataType::Alert.is_condition_bearing());
+        assert!(TriggerDataType::DerivedStream.is_condition_bearing());
+        assert!(TriggerDataType::AnomalyDetection.is_condition_bearing());
+
+        assert!(!TriggerDataType::Report.is_condition_bearing());
+        assert!(!TriggerDataType::CachedReport.is_condition_bearing());
+        assert!(!TriggerDataType::Workflow.is_condition_bearing());
+        assert!(!TriggerDataType::Synthetics.is_condition_bearing());
+        assert!(!TriggerDataType::Backfill.is_condition_bearing());
+        assert!(!TriggerDataType::AnomalyDetectionTraining.is_condition_bearing());
+    }
+
+    // ── normalize_outcome: the read-side migration (Part III) ───────────────
+
+    #[test]
+    fn test_normalize_legacy_completed_alert_is_firing() {
+        assert_eq!(
+            normalize_outcome("completed", &TriggerDataType::Alert, None),
+            Some(RunOutcome::Firing)
+        );
+    }
+
+    #[test]
+    fn test_normalize_legacy_completed_derived_stream_is_firing() {
+        assert_eq!(
+            normalize_outcome("completed", &TriggerDataType::DerivedStream, None),
+            Some(RunOutcome::Firing)
+        );
+    }
+
+    /// Anomaly rows store `completed` whenever detection RAN, even with zero
+    /// anomalies (`handlers.rs:198`). The count lives in `success_response`, so
+    /// the normalizer must parse it — mirroring `history.rs:362`.
+    #[test]
+    fn test_normalize_legacy_completed_anomaly_with_anomalies_is_firing() {
+        assert_eq!(
+            normalize_outcome(
+                "completed",
+                &TriggerDataType::AnomalyDetection,
+                Some(r#"{"anomalies_found":3}"#),
+            ),
+            Some(RunOutcome::Firing)
+        );
+    }
+
+    #[test]
+    fn test_normalize_legacy_completed_anomaly_without_anomalies_is_normal() {
+        assert_eq!(
+            normalize_outcome(
+                "completed",
+                &TriggerDataType::AnomalyDetection,
+                Some(r#"{"anomalies_found":0}"#),
+            ),
+            Some(RunOutcome::Normal)
+        );
+    }
+
+    /// A missing or unparseable `success_response` must not be read as firing.
+    #[test]
+    fn test_normalize_legacy_completed_anomaly_missing_response_is_normal() {
+        assert_eq!(
+            normalize_outcome("completed", &TriggerDataType::AnomalyDetection, None),
+            Some(RunOutcome::Normal)
+        );
+        assert_eq!(
+            normalize_outcome(
+                "completed",
+                &TriggerDataType::AnomalyDetection,
+                Some("not json"),
+            ),
+            Some(RunOutcome::Normal)
+        );
+    }
+
+    #[test]
+    fn test_normalize_legacy_completed_non_condition_module_is_succeeded() {
+        assert_eq!(
+            normalize_outcome("completed", &TriggerDataType::Report, None),
+            Some(RunOutcome::Succeeded)
+        );
+        assert_eq!(
+            normalize_outcome("completed", &TriggerDataType::Workflow, None),
+            Some(RunOutcome::Succeeded)
+        );
+    }
+
+    #[test]
+    fn test_normalize_legacy_condition_not_satisfied_is_normal() {
+        assert_eq!(
+            normalize_outcome("condition_not_satisfied", &TriggerDataType::Alert, None),
+            Some(RunOutcome::Normal)
+        );
+    }
+
+    #[test]
+    fn test_normalize_legacy_failed_is_error() {
+        assert_eq!(
+            normalize_outcome("failed", &TriggerDataType::Alert, None),
+            Some(RunOutcome::Error)
+        );
+    }
+
+    #[test]
+    fn test_normalize_legacy_skipped_unchanged() {
+        assert_eq!(
+            normalize_outcome("skipped", &TriggerDataType::Alert, None),
+            Some(RunOutcome::Skipped)
+        );
+    }
+
+    /// Post-cutover rows already carry the new vocabulary and must pass through.
+    #[test]
+    fn test_normalize_new_vocabulary_passthrough() {
+        for (raw, expected) in [
+            ("firing", RunOutcome::Firing),
+            ("normal", RunOutcome::Normal),
+            ("succeeded", RunOutcome::Succeeded),
+            ("error", RunOutcome::Error),
+            ("skipped", RunOutcome::Skipped),
+            ("notify_failed", RunOutcome::NotifyFailed),
+        ] {
+            assert_eq!(
+                normalize_outcome(raw, &TriggerDataType::Alert, None),
+                Some(expected),
+                "passthrough failed for {raw}"
+            );
+        }
+    }
+
+    #[test]
+    fn test_normalize_is_case_insensitive() {
+        assert_eq!(
+            normalize_outcome("COMPLETED", &TriggerDataType::Alert, None),
+            Some(RunOutcome::Firing)
+        );
+        assert_eq!(
+            normalize_outcome("Firing", &TriggerDataType::Alert, None),
+            Some(RunOutcome::Firing)
+        );
+    }
+
+    #[test]
+    fn test_normalize_unknown_returns_none() {
+        assert_eq!(normalize_outcome("", &TriggerDataType::Alert, None), None);
+        assert_eq!(
+            normalize_outcome("banana", &TriggerDataType::Alert, None),
+            None
+        );
+    }
+
+    // ── TriggerData still serializes its outcome under `status` ─────────────
+
+    /// Part III deliberately keeps the stream FIELD name `status` and changes
+    /// only the values — no schema change. Guard that.
+    #[test]
+    fn test_trigger_data_field_is_still_named_status() {
+        let td = TriggerData {
+            status: RunOutcome::Firing,
+            ..Default::default()
+        };
+        let v = serde_json::to_value(&td).unwrap();
+        assert_eq!(v.get("status").and_then(|s| s.as_str()), Some("firing"));
+        assert!(
+            v.get("outcome").is_none(),
+            "must NOT introduce an `outcome` field on the triggers stream"
+        );
+    }
+
+    #[test]
+    fn test_reflection_sample_still_exposes_status_field() {
+        let names = TriggerData::get_field_names();
+        assert!(names.contains(&"status".to_string()));
+        assert!(!names.contains(&"outcome".to_string()));
+    }
+
+    // ── T-9: value context on the trigger record (alerts_2.md §7.5) ─────────
+    // "Fired at 112 against threshold 100" must be reconstructable from the
+    // triggers stream alone. Today both values exist only as notification
+    // template variables and never reach the stream.
+
+    #[test]
+    fn test_trigger_data_records_actual_and_threshold() {
+        let td = TriggerData {
+            module: TriggerDataType::Alert,
+            status: RunOutcome::Firing,
+            actual_value: Some(112.0),
+            threshold_value: Some(100.0),
+            threshold_operator: Some(">=".to_string()),
+            level: Some(2), // AlertLevel::Critical
+            ..Default::default()
+        };
+        let v = serde_json::to_value(&td).unwrap();
+
+        assert_eq!(v.get("actual_value").and_then(|x| x.as_f64()), Some(112.0));
+        assert_eq!(
+            v.get("threshold_value").and_then(|x| x.as_f64()),
+            Some(100.0)
+        );
+        assert_eq!(
+            v.get("threshold_operator").and_then(|x| x.as_str()),
+            Some(">=")
+        );
+        assert_eq!(v.get("level").and_then(|x| x.as_i64()), Some(2));
+    }
+
+    /// A healthy run must still record what it observed — that is what makes
+    /// "how close did we get?" answerable from history. Per T-10 (as revised):
+    /// normal rows display actual value + Ok; only firing/warning rows display
+    /// a threshold.
+    #[test]
+    fn test_normal_runs_record_actual_value_and_level_ok() {
+        let td = TriggerData {
+            module: TriggerDataType::Alert,
+            status: RunOutcome::Normal,
+            actual_value: Some(12.0),
+            threshold_value: None,
+            // Level is the COMPUTED level: Ok (0) for a level-bearing normal
+            // run — never absent. `None` is reserved for non-condition modules
+            // and error/skipped runs (alerts_2.md §7.5).
+            level: Some(0),
+            ..Default::default()
+        };
+        assert_eq!(td.actual_value, Some(12.0));
+        assert_eq!(
+            td.threshold_value, None,
+            "no threshold matched, so none is recorded (T-10: rendered without one)"
+        );
+        assert_eq!(td.level, Some(0), "normal rows carry level = Ok, not None");
+    }
+
+    #[test]
+    fn test_value_fields_are_optional_and_omitted_when_unset() {
+        // Non-condition modules (reports, workflows) have no values to record;
+        // the fields must not bloat every record.
+        let td = TriggerData {
+            module: TriggerDataType::Report,
+            status: RunOutcome::Succeeded,
+            ..Default::default()
+        };
+        let v = serde_json::to_value(&td).unwrap();
+        assert!(v.get("actual_value").is_none());
+        assert!(v.get("threshold_value").is_none());
+        assert!(v.get("level").is_none());
+    }
+
+    /// The stream schema is generated by reflection, so new fields must appear
+    /// in the reflection sample or fresh orgs get a schema without them.
+    #[test]
+    fn test_reflection_sample_includes_the_value_context_fields() {
+        let names = TriggerData::get_field_names();
+        for f in [
+            "actual_value",
+            "threshold_value",
+            "threshold_operator",
+            "level",
+            "group_label",
+        ] {
+            assert!(
+                names.contains(&f.to_string()),
+                "reflection sample must include `{f}` or new orgs get a schema without it"
+            );
+        }
+    }
+
+    /// D8: one record per evaluation, carrying the worst group's context.
+    #[test]
+    fn test_group_label_identifies_which_group_produced_the_value() {
+        let td = TriggerData {
+            module: TriggerDataType::Alert,
+            status: RunOutcome::Firing,
+            actual_value: Some(500.0),
+            threshold_value: Some(100.0),
+            group_label: Some("host=b".to_string()),
+            ..Default::default()
+        };
+        assert_eq!(td.group_label.as_deref(), Some("host=b"));
+    }
+
+    #[test]
+    fn test_legacy_records_without_value_context_still_deserialize() {
+        // Rows written before T-9 have none of these fields.
+        let legacy = r#"{
+            "_timestamp": 0, "org": "o", "module": "alert", "key": "k",
+            "next_run_at": 0, "is_realtime": false, "is_silenced": false,
+            "status": "firing", "start_time": 0, "end_time": 0, "retries": 0,
+            "error": null, "success_response": null, "is_partial": null,
+            "delay_in_secs": null, "evaluation_took_in_secs": null,
+            "source_node": null, "query_took": null, "scheduler_trace_id": null,
+            "time_in_queue_ms": null
+        }"#;
+        let td: TriggerData = serde_json::from_str(legacy).unwrap();
+        assert_eq!(td.actual_value, None);
+        assert_eq!(td.level, None);
+    }
+}
+
+#[cfg(test)]
+mod reserved_stream_tests {
+    use super::*;
+
+    #[test]
+    fn slo_slices_is_reserved() {
+        // Not self-reporting, but it needs the identical protection: a user
+        // write into it would corrupt the numbers an SLO reports.
+        assert!(is_reserved_internal_stream(
+            crate::meta::slo::stream::SLO_SLICES_STREAM
+        ));
+        assert!(!is_reserved_self_reporting_stream(
+            crate::meta::slo::stream::SLO_SLICES_STREAM
+        ));
+    }
+
+    #[test]
+    fn every_self_reporting_stream_is_also_an_internal_stream() {
+        for s in RESERVED_SELF_REPORTING_STREAMS {
+            assert!(is_reserved_internal_stream(s), "{s} lost its protection");
+        }
+    }
+
+    #[test]
+    fn an_ordinary_stream_name_is_not_reserved() {
+        for s in ["logs", "default", "slo", "slices", "slo_slice"] {
+            assert!(!is_reserved_internal_stream(s), "{s} wrongly reserved");
+        }
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -793,45 +1570,9 @@ mod tests {
         assert!(!UsageType::Retention.is_function());
     }
 
-    #[test]
-    fn test_trigger_data_status_serialization() {
-        assert_eq!(
-            serde_json::to_string(&TriggerDataStatus::Completed).unwrap(),
-            "\"completed\""
-        );
-        assert_eq!(
-            serde_json::to_string(&TriggerDataStatus::Failed).unwrap(),
-            "\"failed\""
-        );
-        assert_eq!(
-            serde_json::to_string(&TriggerDataStatus::ConditionNotSatisfied).unwrap(),
-            "\"condition_not_satisfied\""
-        );
-        assert_eq!(
-            serde_json::to_string(&TriggerDataStatus::Skipped).unwrap(),
-            "\"skipped\""
-        );
-    }
-
-    #[test]
-    fn test_trigger_data_status_deserialization() {
-        assert_eq!(
-            serde_json::from_str::<TriggerDataStatus>("\"completed\"").unwrap(),
-            TriggerDataStatus::Completed
-        );
-        assert_eq!(
-            serde_json::from_str::<TriggerDataStatus>("\"failed\"").unwrap(),
-            TriggerDataStatus::Failed
-        );
-        assert_eq!(
-            serde_json::from_str::<TriggerDataStatus>("\"condition_not_satisfied\"").unwrap(),
-            TriggerDataStatus::ConditionNotSatisfied
-        );
-        assert_eq!(
-            serde_json::from_str::<TriggerDataStatus>("\"skipped\"").unwrap(),
-            TriggerDataStatus::Skipped
-        );
-    }
+    // NOTE: `TriggerDataStatus` serialization/deserialization tests were replaced
+    // by the `run_outcome_tests` module above when the enum became `RunOutcome`
+    // (Part III of alerts.md).
 
     #[test]
     fn test_trigger_data_type_serialization() {
@@ -883,7 +1624,7 @@ mod tests {
             next_run_at: 1234567890,
             is_realtime: true,
             is_silenced: false,
-            status: TriggerDataStatus::Completed,
+            status: RunOutcome::Succeeded,
             start_time: 1234567890,
             end_time: 1234567890,
             retries: 0,
@@ -902,6 +1643,12 @@ mod tests {
             dedup_count: None,
             grouped: None,
             group_size: None,
+            actual_value: None,
+            threshold_value: None,
+            threshold_operator: None,
+            level: None,
+            group_label: None,
+            value_is_lower_bound: None,
         };
 
         let json = serde_json::to_string(&trigger_data).unwrap();

src/config/src/meta/slo/budget.rs

@@ -0,0 +1,374 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! The per-org slice budget — S-14, D57.
+//!
+//! `ZO_SLO_MAX_PER_ORG` (200) and `ZO_SLO_MAX_GROUPS` (500) are each
+//! defensible alone; their **product is not** — 200 × 500 groups over the
+//! 97-day horizon is billions of rows. So the binding limit is on the product,
+//! and it is charged three ways that a naive design gets wrong:
+//!
+//! * **Reservations, not estimates.** A save-time check is bypassable by organic group growth: a
+//!   1-group SLO can grow to thousands with no further save to check it at.
+//! * **Residuals, not instant release.** A superseded generation's rows and a deleted SLO's rows
+//!   stay in the stream until the horizon, so releasing the charge at delete time lets
+//!   create/backfill/delete cycles exceed the budget arbitrarily.
+//! * **Logical rows with priced headroom.** The stream also holds late-data re-emissions, so the
+//!   budget is over *logical* `(group, slice)` rows with a revision multiplier — not a physical-row
+//!   invariant the formula cannot honestly deliver.
+
+/// Stream retention horizon: max window (90d) + 7d margin (D57).
+pub const RETENTION_HORIZON_SECS: i64 = 97 * 86_400;
+
+/// Reserved groups for an SLO (S-14a).
+///
+/// An ungrouped SLO reserves exactly **1** — it can never grow a second
+/// series, and an unconditional floor would charge it ~1.79M rows, letting 200
+/// ungrouped SLOs exceed the whole default budget.
+pub fn groups_reserved(is_grouped: bool, groups_estimate: i64, hard_cap: i64) -> i64 {
+    if !is_grouped {
+        // An ungrouped SLO can never grow a second series. An unconditional
+        // floor here charged it ~1.79M rows, which let 200 ungrouped SLOs
+        // exceed the whole default budget.
+        return 1;
+    }
+    (2 * groups_estimate).clamp(64, hard_cap)
+}
+
+/// Logical slice rows a reservation costs at the retention horizon, including
+/// the revision headroom multiplier.
+pub fn rows_charged(groups_reserved: i64, slice_interval_secs: i64, revision_headroom: f64) -> i64 {
+    if slice_interval_secs <= 0 {
+        return 0;
+    }
+    // Priced at the retention HORIZON, not the SLO's own window: retention is
+    // a stream property, so a 7-day SLO's slices still live 97 days (D57).
+    let slices = RETENTION_HORIZON_SECS / slice_interval_secs;
+    ((groups_reserved * slices) as f64 * revision_headroom) as i64
+}
+
+/// A charge's lifecycle state (S-14c).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ChargeState {
+    /// The SLO/generation is live and writing.
+    Active,
+    /// Superseded or deleted; the rows persist until they age out.
+    Residual { expires_at: i64 },
+}
+
+/// One row of `slo_budget_charges`.
+#[derive(Debug, Clone, PartialEq)]
+pub struct BudgetCharge {
+    pub slo_id: String,
+    pub generation: i32,
+    pub rows_charged: i64,
+    pub state: ChargeState,
+}
+
+/// Convert a charge to a residual expiring one horizon after its last write.
+pub fn to_residual(charge: BudgetCharge, last_write_secs: i64) -> BudgetCharge {
+    // Releasing at delete time would let create/backfill/delete cycles exceed
+    // the budget arbitrarily: the rows persist to the horizon regardless.
+    BudgetCharge {
+        state: ChargeState::Residual {
+            expires_at: last_write_secs + RETENTION_HORIZON_SECS,
+        },
+        ..charge
+    }
+}
+
+/// Total rows an org currently owes: active reservations plus unexpired
+/// residuals.
+pub fn org_usage(charges: &[BudgetCharge], now_secs: i64) -> i64 {
+    charges
+        .iter()
+        .filter(|c| match c.state {
+            ChargeState::Active => true,
+            ChargeState::Residual { expires_at } => now_secs < expires_at,
+        })
+        .map(|c| c.rows_charged)
+        .sum()
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum BudgetError {
+    /// The org would exceed its slice budget. Carries the arithmetic so the
+    /// 400 can show it.
+    OrgBudgetExceeded {
+        requested: i64,
+        in_use: i64,
+        limit: i64,
+    },
+    /// The preflight estimate is above the hard cardinality cap.
+    HardCapExceeded { estimate: i64, hard_cap: i64 },
+}
+
+/// Whether a new charge fits.
+pub fn can_admit(
+    charges: &[BudgetCharge],
+    requested_rows: i64,
+    limit: i64,
+    now_secs: i64,
+) -> Result<(), BudgetError> {
+    let in_use = org_usage(charges, now_secs);
+    if in_use + requested_rows > limit {
+        return Err(BudgetError::OrgBudgetExceeded {
+            requested: requested_rows,
+            in_use,
+            limit,
+        });
+    }
+    Ok(())
+}
+
+/// Whether the ingest job may raise a reservation in place, or must trip
+/// `GroupOverflow` (S-14b).
+pub fn can_raise_reservation(
+    charges: &[BudgetCharge],
+    slo_id: &str,
+    generation: i32,
+    new_rows: i64,
+    limit: i64,
+    now_secs: i64,
+) -> bool {
+    // The raise REPLACES this charge rather than adding to it — summing would
+    // double-count the SLO's own reservation and trip overflow early.
+    let others: Vec<BudgetCharge> = charges
+        .iter()
+        .filter(|c| !(c.slo_id == slo_id && c.generation == generation))
+        .cloned()
+        .collect();
+    can_admit(&others, new_rows, limit, now_secs).is_ok()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    const HEADROOM: f64 = 1.5;
+    const HARD_CAP: i64 = 10_000;
+
+    fn active(id: &str, generation: i32, rows: i64) -> BudgetCharge {
+        BudgetCharge {
+            slo_id: id.into(),
+            generation,
+            rows_charged: rows,
+            state: ChargeState::Active,
+        }
+    }
+
+    // ---- reservations ------------------------------------------------------
+
+    /// S-14a: the fix for the unconditional 64-floor, which charged every
+    /// ungrouped SLO ~1.79M rows.
+    #[test]
+    fn an_ungrouped_slo_reserves_exactly_one_group() {
+        assert_eq!(groups_reserved(false, 1, HARD_CAP), 1);
+        assert_eq!(
+            groups_reserved(false, 500, HARD_CAP),
+            1,
+            "an ungrouped SLO can never grow a second series"
+        );
+    }
+
+    #[test]
+    fn a_grouped_slo_reserves_double_the_estimate() {
+        assert_eq!(groups_reserved(true, 200, HARD_CAP), 400);
+    }
+
+    #[test]
+    fn a_grouped_reservation_has_a_floor_of_sixty_four() {
+        assert_eq!(groups_reserved(true, 5, HARD_CAP), 64);
+        assert_eq!(groups_reserved(true, 1, HARD_CAP), 64);
+    }
+
+    #[test]
+    fn a_grouped_reservation_is_clamped_at_the_hard_cap() {
+        assert_eq!(groups_reserved(true, 9_000, HARD_CAP), HARD_CAP);
+    }
+
+    #[test]
+    fn two_hundred_ungrouped_slos_fit_comfortably_in_the_default_budget() {
+        let per_slo = rows_charged(groups_reserved(false, 1, HARD_CAP), 300, HEADROOM);
+        let total = per_slo * 200;
+        assert!(
+            total < 250_000_000,
+            "200 ungrouped SLOs charged {total}, over the 250M default"
+        );
+    }
+
+    // ---- row arithmetic ----------------------------------------------------
+
+    #[test]
+    fn rows_are_charged_at_the_horizon_not_the_slo_window() {
+        // D57: retention is global, so a 7-day SLO's slices still live 97 days.
+        let rows = rows_charged(1, 300, 1.0);
+        assert_eq!(rows, RETENTION_HORIZON_SECS / 300);
+        assert_eq!(rows, 27_936);
+    }
+
+    #[test]
+    fn a_finer_slice_interval_charges_proportionally_more() {
+        assert_eq!(rows_charged(1, 60, 1.0), rows_charged(1, 300, 1.0) * 5);
+    }
+
+    #[test]
+    fn the_headroom_multiplier_prices_late_data_re_emissions() {
+        let bare = rows_charged(10, 300, 1.0);
+        let with_headroom = rows_charged(10, 300, 1.5);
+        assert_eq!(with_headroom, (bare as f64 * 1.5) as i64);
+    }
+
+    #[test]
+    fn a_five_hundred_group_slo_charges_the_documented_magnitude() {
+        // §6b.4d: ~14M logical rows steady state at the horizon.
+        let rows = rows_charged(500, 300, 1.0);
+        assert!(
+            (13_000_000..15_000_000).contains(&rows),
+            "expected ~14M, got {rows}"
+        );
+    }
+
+    // ---- admission ---------------------------------------------------------
+
+    #[test]
+    fn a_charge_that_fits_is_admitted() {
+        assert!(can_admit(&[active("a", 1, 100)], 50, 1_000, 0).is_ok());
+    }
+
+    #[test]
+    fn a_charge_that_would_exceed_the_limit_is_rejected_with_the_arithmetic() {
+        let err = can_admit(&[active("a", 1, 900)], 200, 1_000, 0).unwrap_err();
+        assert_eq!(
+            err,
+            BudgetError::OrgBudgetExceeded {
+                requested: 200,
+                in_use: 900,
+                limit: 1_000
+            }
+        );
+    }
+
+    #[test]
+    fn admission_is_exact_at_the_limit() {
+        assert!(can_admit(&[active("a", 1, 900)], 100, 1_000, 0).is_ok());
+        assert!(can_admit(&[active("a", 1, 900)], 101, 1_000, 0).is_err());
+    }
+
+    // ---- residuals: the create/delete cycling bypass ------------------------
+
+    #[test]
+    fn a_deleted_slo_becomes_a_residual_rather_than_releasing_immediately() {
+        let charge = to_residual(active("a", 1, 100), 1_000);
+        assert_eq!(
+            charge.state,
+            ChargeState::Residual {
+                expires_at: 1_000 + RETENTION_HORIZON_SECS
+            }
+        );
+    }
+
+    #[test]
+    fn an_unexpired_residual_still_counts_against_the_org() {
+        let charges = vec![to_residual(active("a", 1, 500), 1_000)];
+        assert_eq!(org_usage(&charges, 1_000 + 86_400), 500);
+    }
+
+    #[test]
+    fn an_expired_residual_stops_counting() {
+        let charges = vec![to_residual(active("a", 1, 500), 1_000)];
+        assert_eq!(org_usage(&charges, 1_000 + RETENTION_HORIZON_SECS + 1), 0);
+    }
+
+    /// The bypass residuals exist to close: without them, create → backfill →
+    /// delete in a loop consumes unbounded storage while the budget reads
+    /// zero.
+    #[test]
+    fn create_delete_cycling_cannot_exceed_the_budget() {
+        let mut charges = Vec::new();
+        let mut now = 0;
+        for i in 0..10 {
+            let c = active(&format!("slo{i}"), 1, 200);
+            // Admission must see the residuals from every prior cycle.
+            let admitted = can_admit(&charges, 200, 1_000, now).is_ok();
+            if admitted {
+                charges.push(to_residual(c, now));
+            }
+            now += 3_600;
+        }
+        let live = org_usage(&charges, now);
+        assert!(
+            live <= 1_000,
+            "cycling accumulated {live} rows against a 1,000 limit"
+        );
+        assert!(
+            charges.len() < 10,
+            "some creates must have been rejected; {} were admitted",
+            charges.len()
+        );
+    }
+
+    /// The other bypass: repeated computation edits, each leaving a full
+    /// superseded generation behind.
+    #[test]
+    fn repeated_generation_bumps_accumulate_residual_charges() {
+        let mut charges = vec![];
+        for generation in 1..=3 {
+            charges.push(to_residual(
+                active("a", generation, 300),
+                (generation as i64) * 60,
+            ));
+        }
+        assert_eq!(
+            org_usage(&charges, 1_000),
+            900,
+            "all three generations' rows are still in the stream"
+        );
+    }
+
+    #[test]
+    fn a_superseded_generation_and_its_successor_are_both_charged() {
+        let charges = vec![to_residual(active("a", 1, 300), 100), active("a", 2, 300)];
+        assert_eq!(org_usage(&charges, 200), 600);
+    }
+
+    // ---- runtime reservation raises ----------------------------------------
+
+    #[test]
+    fn a_reservation_can_be_raised_when_the_org_has_headroom() {
+        let charges = vec![active("a", 1, 100)];
+        assert!(can_raise_reservation(&charges, "a", 1, 300, 1_000, 0));
+    }
+
+    #[test]
+    fn a_reservation_cannot_be_raised_past_the_org_limit() {
+        let charges = vec![active("a", 1, 100), active("b", 1, 800)];
+        assert!(
+            !can_raise_reservation(&charges, "a", 1, 500, 1_000, 0),
+            "raising a to 500 would total 1300 against a 1000 limit"
+        );
+    }
+
+    /// The raise must replace the SLO's own charge, not add to it — otherwise
+    /// growth double-counts and trips overflow early.
+    #[test]
+    fn raising_replaces_the_slos_existing_charge() {
+        let charges = vec![active("a", 1, 900)];
+        assert!(
+            can_raise_reservation(&charges, "a", 1, 950, 1_000, 0),
+            "950 replaces 900; it does not sum to 1850"
+        );
+    }
+}

src/config/src/meta/slo/budget_rows.rs

@@ -0,0 +1,321 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Row-budget arithmetic (`alerts_2.md` §6b.4d, S-14).
+//!
+//! The limit exists because `SLOs × GROUPS × window` is indefensible even
+//! where each factor is individually fine: 100 SLOs × 500 groups × 90 days at
+//! 5-minute slices is 1.3 **billion** rows, and every factor in that product
+//! passes its own limit. So the budget is on the product.
+//!
+//! Two rules that look like details and are not:
+//!
+//! * **Priced at the horizon, not the window.** Slices live until retention regardless of how long
+//!   the SLO's window is, so a 7-day SLO occupies the same storage as a 90-day one. Pricing by
+//!   window would let an org buy unlimited storage by declaring short windows.
+//! * **Physical, not logical.** Late data and recomputes re-emit rows; the dedupe happens at read
+//!   time, not in storage. `ZO_SLO_REVISION_HEADROOM` prices that.
+
+/// The retention horizon slices are priced against: the longest window plus
+/// the 7-day grace that keeps a just-expired window readable.
+pub const SLICE_HORIZON_SECS: i64 = 97 * 86_400;
+
+/// Why a reservation could not be granted.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum BudgetError {
+    /// The org's budget is already committed. Carries the arithmetic, because
+    /// "quota exceeded" without numbers is unactionable — §6b.4d requires the
+    /// rejection to show its working.
+    OrgBudgetExceeded {
+        requested: i64,
+        active: i64,
+        residual: i64,
+        cap: i64,
+    },
+    /// The reservation itself is larger than any org could hold.
+    RequestExceedsCap { requested: i64, cap: i64 },
+}
+
+impl std::fmt::Display for BudgetError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::OrgBudgetExceeded {
+                requested,
+                active,
+                residual,
+                cap,
+            } => write!(
+                f,
+                "SLO would reserve {requested} slice rows; the org already holds \
+                 {active} active + {residual} residual of a {cap} row budget \
+                 ({} free)",
+                (cap - active - residual).max(0)
+            ),
+            Self::RequestExceedsCap { requested, cap } => write!(
+                f,
+                "SLO would reserve {requested} slice rows, more than the entire \
+                 {cap} row org budget"
+            ),
+        }
+    }
+}
+
+impl std::error::Error for BudgetError {}
+
+/// How many groups an SLO reserves.
+///
+/// `1` when ungrouped. Otherwise **twice** the preflight estimate, floored at
+/// 64 and capped: the doubling is headroom for organic group growth, because
+/// a reservation that exactly matched today's cardinality would trip the
+/// moment one new region appeared.
+pub fn groups_reserved(is_grouped: bool, estimate: Option<i64>, hard_cap: i64) -> i64 {
+    if !is_grouped {
+        return 1;
+    }
+    let doubled = estimate.unwrap_or(0).saturating_mul(2);
+    doubled.clamp(64, hard_cap.max(64))
+}
+
+/// Logical rows an SLO reserves: groups × slices-to-the-horizon × revision
+/// headroom.
+///
+/// Priced at the **horizon**, not the SLO's window — see the module note.
+pub fn rows_for_reservation(
+    groups_reserved: i64,
+    slice_interval_secs: i64,
+    revision_headroom: f64,
+) -> i64 {
+    if slice_interval_secs <= 0 {
+        return 0;
+    }
+    let slices = SLICE_HORIZON_SECS / slice_interval_secs;
+    let logical = groups_reserved.saturating_mul(slices);
+    // Headroom below 1.0 would under-price; it is a multiplier for
+    // re-emissions, not a discount.
+    let headroom = revision_headroom.max(1.0);
+    ((logical as f64) * headroom).ceil() as i64
+}
+
+/// Whether `requested` fits in the org's remaining budget.
+///
+/// Residual rows count against the cap. They are real storage — a superseded
+/// generation's slices persist to the horizon whether or not anything reads
+/// them — and not charging them would make delete-and-recreate an unlimited
+/// storage loophole (S-14c).
+pub fn check_headroom(
+    requested: i64,
+    active: i64,
+    residual: i64,
+    cap: i64,
+) -> Result<(), BudgetError> {
+    if requested > cap {
+        return Err(BudgetError::RequestExceedsCap { requested, cap });
+    }
+    if active.saturating_add(residual).saturating_add(requested) > cap {
+        return Err(BudgetError::OrgBudgetExceeded {
+            requested,
+            active,
+            residual,
+            cap,
+        });
+    }
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    const CAP: i64 = 250_000_000;
+
+    // ===================== reservation sizing =============================
+
+    #[test]
+    fn an_ungrouped_slo_reserves_one_group() {
+        assert_eq!(groups_reserved(false, None, 500), 1);
+        // Even if an estimate somehow came back — ungrouped is one series.
+        assert_eq!(groups_reserved(false, Some(400), 500), 1);
+    }
+
+    /// The doubling is headroom for organic growth: a reservation that exactly
+    /// matched today's cardinality would trip the first time a new region
+    /// appeared.
+    #[test]
+    fn a_grouped_slo_reserves_twice_its_estimate() {
+        assert_eq!(groups_reserved(true, Some(100), 500), 200);
+    }
+
+    #[test]
+    fn a_small_estimate_is_floored_at_64() {
+        assert_eq!(groups_reserved(true, Some(3), 500), 64);
+        assert_eq!(groups_reserved(true, Some(0), 500), 64);
+        // No preflight result is not a licence to reserve nothing.
+        assert_eq!(groups_reserved(true, None, 500), 64);
+    }
+
+    #[test]
+    fn a_large_estimate_is_capped() {
+        assert_eq!(groups_reserved(true, Some(10_000), 500), 500);
+    }
+
+    /// The floor must survive a hard cap set below it, or a misconfigured
+    /// deployment would reserve less than the minimum.
+    #[test]
+    fn the_floor_wins_over_an_absurdly_low_cap() {
+        assert_eq!(groups_reserved(true, Some(100), 10), 64);
+    }
+
+    #[test]
+    fn a_huge_estimate_does_not_overflow() {
+        assert_eq!(groups_reserved(true, Some(i64::MAX), 500), 500);
+    }
+
+    // ===================== row pricing ====================================
+
+    /// The §6b.4d figure: an ungrouped 5-minute SLO is negligible.
+    #[test]
+    fn an_ungrouped_five_minute_slo_is_cheap() {
+        // 97d / 5min = 27,936 slices.
+        assert_eq!(rows_for_reservation(1, 300, 1.0), 27_936);
+    }
+
+    #[test]
+    fn a_one_minute_slo_costs_five_times_a_five_minute_one() {
+        assert_eq!(
+            rows_for_reservation(1, 60, 1.0),
+            rows_for_reservation(1, 300, 1.0) * 5
+        );
+    }
+
+    /// Pricing by window rather than horizon would let an org buy unlimited
+    /// storage by declaring short windows — the slices live to the horizon
+    /// either way.
+    #[test]
+    fn the_price_does_not_depend_on_the_slo_window() {
+        // There is no window parameter, and that is the point. Pinned as a
+        // test so adding one is a deliberate act.
+        let seven_day_slo = rows_for_reservation(64, 300, 1.0);
+        let ninety_day_slo = rows_for_reservation(64, 300, 1.0);
+        assert_eq!(seven_day_slo, ninety_day_slo);
+    }
+
+    #[test]
+    fn revision_headroom_multiplies_and_rounds_up() {
+        let base = rows_for_reservation(1, 300, 1.0);
+        assert_eq!(
+            rows_for_reservation(1, 300, 1.5),
+            (base as f64 * 1.5).ceil() as i64
+        );
+    }
+
+    /// Headroom prices re-emissions. A value below 1.0 would under-price them,
+    /// so it is clamped rather than honoured.
+    #[test]
+    fn headroom_below_one_does_not_discount() {
+        assert_eq!(
+            rows_for_reservation(10, 300, 0.5),
+            rows_for_reservation(10, 300, 1.0)
+        );
+    }
+
+    #[test]
+    fn a_nonsensical_slice_interval_prices_at_zero_rather_than_dividing_by_it() {
+        assert_eq!(rows_for_reservation(10, 0, 1.0), 0);
+        assert_eq!(rows_for_reservation(10, -300, 1.0), 0);
+    }
+
+    /// The product this whole budget exists to bound (§6b.4d): each factor
+    /// passes its own limit, and together they are 1.3 billion rows.
+    #[test]
+    fn the_indefensible_product_is_rejected() {
+        let per_slo = rows_for_reservation(500, 300, 1.0);
+        let hundred_slos = per_slo * 100;
+        assert!(
+            hundred_slos > CAP,
+            "100 x 500 groups x 97d @5m = {hundred_slos}, which must not fit \
+             in a {CAP} row budget"
+        );
+    }
+
+    // ===================== headroom check =================================
+
+    #[test]
+    fn a_reservation_that_fits_is_granted() {
+        assert_eq!(check_headroom(1_000, 500, 500, CAP), Ok(()));
+    }
+
+    #[test]
+    fn a_reservation_that_exactly_fills_the_budget_is_granted() {
+        assert_eq!(check_headroom(10, 80, 10, 100), Ok(()));
+    }
+
+    #[test]
+    fn one_row_past_the_budget_is_rejected() {
+        assert_eq!(
+            check_headroom(11, 80, 10, 100),
+            Err(BudgetError::OrgBudgetExceeded {
+                requested: 11,
+                active: 80,
+                residual: 10,
+                cap: 100
+            })
+        );
+    }
+
+    /// Residual rows are real storage. Not charging them would make
+    /// delete-and-recreate an unlimited storage loophole (S-14c).
+    #[test]
+    fn residual_rows_count_against_the_budget() {
+        assert!(check_headroom(50, 0, 0, 100).is_ok());
+        assert!(
+            check_headroom(50, 0, 60, 100).is_err(),
+            "a residual charge was ignored"
+        );
+    }
+
+    #[test]
+    fn a_request_larger_than_the_whole_budget_says_so_specifically() {
+        assert_eq!(
+            check_headroom(500, 0, 0, 100),
+            Err(BudgetError::RequestExceedsCap {
+                requested: 500,
+                cap: 100
+            })
+        );
+    }
+
+    /// "Quota exceeded" without numbers is unactionable — §6b.4d requires the
+    /// rejection to show its arithmetic.
+    #[test]
+    fn the_rejection_shows_its_arithmetic() {
+        let msg = check_headroom(11, 80, 10, 100).unwrap_err().to_string();
+        for part in ["11", "80", "10", "100"] {
+            assert!(msg.contains(part), "{msg} omits {part}");
+        }
+    }
+
+    #[test]
+    fn the_free_figure_never_goes_negative_in_the_message() {
+        // An org already over budget (a cap lowered under it) must still get a
+        // sensible message rather than "-40 free".
+        let msg = check_headroom(10, 90, 50, 100).unwrap_err().to_string();
+        assert!(msg.contains("0 free"), "{msg}");
+    }
+
+    #[test]
+    fn saturating_arithmetic_survives_absurd_inputs() {
+        assert!(check_headroom(i64::MAX, i64::MAX, i64::MAX, CAP).is_err());
+    }
+}

src/config/src/meta/slo/coverage.rs

@@ -0,0 +1,427 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Coverage, and the refusal to fabricate uptime — §6b.4b, S-8, SA-17, SA-18.
+//!
+//! This is the module that decides whether an SLO alert may change state at
+//! all. Its single job is to distinguish **"we measured, and it was fine"**
+//! from **"we did not measure"**, because collapsing those two is the worst
+//! failure this feature can have: a search outage would read as "no errors
+//! observed" and recover every burn-rate alert in the org.
+//!
+//! Deliberate divergence from Datadog, which counts missing data in a Time
+//! Slice SLO as uptime (D34).
+//!
+//! For an `alert`-type SLI the same question is asked of a *source alert's*
+//! evaluations rather than of a query — see [`evaluation_is_measured`].
+
+use crate::meta::self_reporting::usage::RunOutcome;
+
+/// Coverage as a fraction in `[0, 1]`: observed slices over *expected* slices,
+/// where expected comes from the aligned grid — never from what a query
+/// happened to return.
+pub fn coverage(observed_slices: i64, expected_slices: i64) -> f64 {
+    if expected_slices <= 0 {
+        // Nothing was expected, so nothing was covered. Reporting 1.0 here
+        // would make an empty window look fully measured.
+        return 0.0;
+    }
+    // Clamped: duplicate rows must never report more than fully covered.
+    (observed_slices as f64 / expected_slices as f64).clamp(0.0, 1.0)
+}
+
+/// One window's worth of aggregated slices, as read from the status row.
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub struct WindowRead {
+    pub good: f64,
+    pub total: f64,
+    pub observed_slices: i64,
+    pub expected_slices: i64,
+}
+
+/// Why an evaluation could not observe anything. Each maps to "freeze the
+/// level" — never to a recovery.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum UnobservedReason {
+    /// Coverage for this window is under the floor (S-8).
+    BelowCoverageFloor,
+    /// The window is covered but carries no events, so the SLI is undefined
+    /// (SA-18). Usually itself an incident.
+    ZeroTotal,
+    /// The SLO's watermark has not advanced recently enough to trust (SA-14).
+    StaleWatermark,
+}
+
+/// The result of trying to observe a window.
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum Observation {
+    /// A real measurement — the SLI over the window.
+    Observed { sli: f64 },
+    /// Nothing was measured. The caller must leave `level`, `level_since` and
+    /// `level_at` untouched (§7.6).
+    Unobserved(UnobservedReason),
+}
+
+impl Observation {
+    pub fn is_observed(&self) -> bool {
+        matches!(self, Self::Observed { .. })
+    }
+
+    pub fn sli(&self) -> Option<f64> {
+        match self {
+            Self::Observed { sli } => Some(*sli),
+            Self::Unobserved(_) => None,
+        }
+    }
+}
+
+/// Decide whether a window read is a measurement.
+///
+/// `watermark_stale` is computed by [`super::window::watermark_is_stale`] and
+/// passed in so this stays a pure decision over already-gathered facts.
+pub fn observe(read: WindowRead, coverage_floor: f64, watermark_stale: bool) -> Observation {
+    // Precedence is fixed and deliberate. Staleness first: if the data is not
+    // current, nothing about it is a measurement, whatever its coverage says.
+    // Then coverage: we cannot claim a window was empty if we did not measure
+    // it. Only then emptiness.
+    if watermark_stale {
+        return Observation::Unobserved(UnobservedReason::StaleWatermark);
+    }
+    if coverage(read.observed_slices, read.expected_slices) < coverage_floor {
+        return Observation::Unobserved(UnobservedReason::BelowCoverageFloor);
+    }
+    match super::math::sli(read.good, read.total) {
+        Some(sli) => Observation::Observed { sli },
+        None => Observation::Unobserved(UnobservedReason::ZeroTotal),
+    }
+}
+
+/// Whether one evaluation of a **source alert** counts as a measurement, for
+/// an `alert`-type SLI (S-16, D65).
+///
+/// This is not a new classification — it is exactly §7.6's rule for which
+/// outcomes refresh `level_at`, i.e. which evaluations actually computed a
+/// level. An `alert` SLI's coverage is the fraction of slices containing at
+/// least one such record in the `triggers` stream.
+///
+/// The cases that are *absent* rather than negative matter just as much: a
+/// paused or disabled alert publishes **no record at all** (the scheduler
+/// returns before publishing), so its slices have no evidence of measurement
+/// and fall through gap-fill as uncovered.
+pub fn evaluation_is_measured(outcome: &RunOutcome) -> bool {
+    match outcome {
+        // Computed a level — the same set §7.6 refreshes `level_at` for.
+        RunOutcome::Firing | RunOutcome::Normal | RunOutcome::NotifyFailed => true,
+        // Observed nothing: the query failed, the run was skipped, or the
+        // outcome belongs to a non-condition module.
+        RunOutcome::Error | RunOutcome::Skipped | RunOutcome::Succeeded => false,
+    }
+}
+
+/// Whether an SLO's overall status should read as `NoData` (S-8).
+pub fn is_no_data(read: WindowRead, coverage_floor: f64) -> bool {
+    coverage(read.observed_slices, read.expected_slices) < coverage_floor
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::meta::self_reporting::usage::RunOutcome;
+
+    const FLOOR: f64 = 0.8;
+
+    fn full(good: f64, total: f64) -> WindowRead {
+        WindowRead {
+            good,
+            total,
+            observed_slices: 100,
+            expected_slices: 100,
+        }
+    }
+
+    // ---- coverage arithmetic ----------------------------------------------
+
+    #[test]
+    fn full_coverage_is_one() {
+        assert_eq!(coverage(100, 100), 1.0);
+    }
+
+    #[test]
+    fn no_observations_is_zero_coverage() {
+        assert_eq!(coverage(0, 100), 0.0);
+    }
+
+    #[test]
+    fn partial_coverage_is_the_fraction() {
+        assert!((coverage(71, 100) - 0.71).abs() < 1e-9);
+    }
+
+    #[test]
+    fn coverage_of_an_empty_window_is_zero_not_a_divide_by_zero() {
+        assert_eq!(coverage(0, 0), 0.0);
+    }
+
+    #[test]
+    fn coverage_is_clamped_at_one() {
+        // Defensive: duplicate rows must never report >100% covered.
+        assert_eq!(coverage(150, 100), 1.0);
+    }
+
+    // ---- the floor ---------------------------------------------------------
+
+    #[test]
+    fn a_well_covered_window_is_observed() {
+        let obs = observe(full(999.0, 1000.0), FLOOR, false);
+        assert!(obs.is_observed());
+        assert!((obs.sli().unwrap() - 99.9).abs() < 1e-9);
+    }
+
+    #[test]
+    fn coverage_exactly_at_the_floor_is_observed() {
+        let read = WindowRead {
+            observed_slices: 80,
+            expected_slices: 100,
+            ..full(999.0, 1000.0)
+        };
+        assert!(observe(read, FLOOR, false).is_observed());
+    }
+
+    #[test]
+    fn coverage_below_the_floor_is_unobserved() {
+        let read = WindowRead {
+            observed_slices: 71,
+            expected_slices: 100,
+            ..full(999.0, 1000.0)
+        };
+        assert_eq!(
+            observe(read, FLOOR, false),
+            Observation::Unobserved(UnobservedReason::BelowCoverageFloor)
+        );
+    }
+
+    /// The headline failure mode. A search outage leaves a window that looks
+    /// perfect — every slice it *did* see was good — and must NOT be read as a
+    /// recovery.
+    #[test]
+    fn a_search_outage_never_reads_as_a_healthy_window() {
+        let outage = WindowRead {
+            good: 50.0,
+            total: 50.0, // everything observed was perfect
+            observed_slices: 5,
+            expected_slices: 100,
+        };
+        let obs = observe(outage, FLOOR, false);
+        assert!(
+            !obs.is_observed(),
+            "a 5%-covered window must never report a 100% SLI"
+        );
+    }
+
+    // ---- zero total (SA-18) ------------------------------------------------
+
+    #[test]
+    fn a_covered_window_with_no_events_is_unobserved_not_healthy() {
+        let idle = WindowRead {
+            good: 0.0,
+            total: 0.0,
+            observed_slices: 100,
+            expected_slices: 100,
+        };
+        assert_eq!(
+            observe(idle, FLOOR, false),
+            Observation::Unobserved(UnobservedReason::ZeroTotal),
+            "traffic stopping is usually the incident, not a recovery"
+        );
+    }
+
+    #[test]
+    fn zero_total_is_never_reported_as_a_zero_burn_rate() {
+        let idle = WindowRead {
+            good: 0.0,
+            total: 0.0,
+            observed_slices: 100,
+            expected_slices: 100,
+        };
+        assert_eq!(observe(idle, FLOOR, false).sli(), None);
+    }
+
+    // ---- stale watermark (SA-14) -------------------------------------------
+
+    #[test]
+    fn a_stale_watermark_makes_the_window_unobserved() {
+        assert_eq!(
+            observe(full(999.0, 1000.0), FLOOR, true),
+            Observation::Unobserved(UnobservedReason::StaleWatermark)
+        );
+    }
+
+    #[test]
+    fn staleness_wins_over_good_coverage() {
+        // Perfect coverage but a frozen watermark: the data is stale, so
+        // nothing about it is a current measurement.
+        let obs = observe(full(1000.0, 1000.0), FLOOR, true);
+        assert!(!obs.is_observed());
+    }
+
+    // ---- precedence --------------------------------------------------------
+
+    /// When several reasons apply the caller only needs "unobserved", but the
+    /// reason drives the UI copy, so the precedence must be stable.
+    #[test]
+    fn unobserved_reasons_have_a_deterministic_precedence() {
+        let bad = WindowRead {
+            good: 0.0,
+            total: 0.0,
+            observed_slices: 1,
+            expected_slices: 100,
+        };
+        // Stale watermark is the most fundamental: the data is not current at
+        // all, so it outranks coverage and emptiness.
+        assert_eq!(
+            observe(bad, FLOOR, true),
+            Observation::Unobserved(UnobservedReason::StaleWatermark)
+        );
+        // Without staleness, coverage outranks zero-total: we cannot even say
+        // the window was empty if we did not measure it.
+        assert_eq!(
+            observe(bad, FLOOR, false),
+            Observation::Unobserved(UnobservedReason::BelowCoverageFloor)
+        );
+    }
+
+    // ---- NoData status -----------------------------------------------------
+
+    #[test]
+    fn no_data_tracks_the_coverage_floor() {
+        assert!(!is_no_data(full(1.0, 1.0), FLOOR));
+        assert!(is_no_data(
+            WindowRead {
+                observed_slices: 10,
+                expected_slices: 100,
+                ..full(1.0, 1.0)
+            },
+            FLOOR
+        ));
+    }
+
+    // ---- alert-SLI measurement availability (S-16, D65) --------------------
+
+    /// The outcomes that computed a level are exactly the ones that count as
+    /// measurement — the same partition §7.6 uses for `level_at`.
+    #[test]
+    fn evaluations_that_computed_a_level_count_as_measured() {
+        for outcome in [
+            RunOutcome::Firing,
+            RunOutcome::Normal,
+            RunOutcome::NotifyFailed,
+        ] {
+            assert!(
+                evaluation_is_measured(&outcome),
+                "{outcome:?} computed a level and must count as measured"
+            );
+        }
+    }
+
+    /// A query failure observed nothing, so it cannot contribute uptime — the
+    /// same reason §7.6 refuses to refresh `level_at` on an error.
+    #[test]
+    fn an_errored_evaluation_is_not_a_measurement() {
+        assert!(!evaluation_is_measured(&RunOutcome::Error));
+    }
+
+    #[test]
+    fn a_skipped_evaluation_is_not_a_measurement() {
+        assert!(!evaluation_is_measured(&RunOutcome::Skipped));
+    }
+
+    /// An interval in which the source alert was **paused** has no records at
+    /// all, so it must read as reduced coverage rather than as uptime. This is
+    /// the case S-16 exists for: pausing an alert is a routine operator
+    /// action, and counting that time as good would silently inflate the SLO
+    /// for as long as the pause lasted.
+    #[test]
+    fn a_paused_interval_reduces_coverage_rather_than_counting_as_uptime() {
+        // 100 slices expected; the source alert was paused for 40 of them, so
+        // those produced no trigger records at all.
+        let read = WindowRead {
+            good: 60.0,
+            total: 60.0, // every measured slice was healthy
+            observed_slices: 60,
+            expected_slices: 100,
+        };
+        assert!((coverage(60, 100) - 0.6).abs() < 1e-9);
+        assert_eq!(
+            observe(read, FLOOR, false),
+            Observation::Unobserved(UnobservedReason::BelowCoverageFloor),
+            "a 60%-covered window must not report the 100% SLI of the slices \
+             that happened to be measured"
+        );
+    }
+
+    /// The same, for intervals the alert *ran* but failed to evaluate. These
+    /// DO produce records, so they must be excluded by status rather than by
+    /// absence — a coverage rule keyed only on "is there a record" would count
+    /// them.
+    #[test]
+    fn an_errored_interval_reduces_coverage_rather_than_counting_as_uptime() {
+        let measured = [
+            RunOutcome::Normal,
+            RunOutcome::Error,
+            RunOutcome::Error,
+            RunOutcome::Normal,
+        ]
+        .iter()
+        .filter(|o| evaluation_is_measured(o))
+        .count();
+        assert_eq!(measured, 2, "the errored slices must not be counted");
+
+        let read = WindowRead {
+            good: 2.0,
+            total: 2.0,
+            observed_slices: measured as i64,
+            expected_slices: 4,
+        };
+        assert!(
+            !observe(read, FLOOR, false).is_observed(),
+            "half the window unmeasured must freeze, not report 100%"
+        );
+    }
+
+    // ---- per-window independence (SA-17) -----------------------------------
+
+    /// A 30-day window can be well covered while the last hour is a hole — and
+    /// the last hour is exactly what a burn-rate alert is about. Each window is
+    /// therefore gated on its OWN coverage.
+    #[test]
+    fn a_well_covered_long_window_does_not_vouch_for_a_broken_short_one() {
+        let long = WindowRead {
+            good: 43_000.0,
+            total: 43_200.0,
+            observed_slices: 43_100,
+            expected_slices: 43_200,
+        };
+        let short = WindowRead {
+            good: 4.0,
+            total: 4.0,
+            observed_slices: 1,
+            expected_slices: 5,
+        };
+        assert!(observe(long, FLOOR, false).is_observed());
+        assert!(
+            !observe(short, FLOOR, false).is_observed(),
+            "the short window must be judged on its own coverage"
+        );
+    }
+}

src/config/src/meta/slo/generation.rs

@@ -0,0 +1,391 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Definition generations — §6b.4a, S-13, D43, D59.
+//!
+//! Slice identity is `(slo_id, group_key, slice_start)`, which says nothing
+//! about *what was being measured*. Change the `good_expr`, the stream, the
+//! comparator, the slice interval or the grouping, and the next pass starts
+//! writing slices that mean something different from the ones already in the
+//! window — which would then be summed together for up to 90 days, producing a
+//! number that describes no definition that ever existed.
+//!
+//! **Generation is also the writing epoch (D59).** Two drafts tried to make
+//! A → B → A cheap by reusing a matching prior generation; each spawned a
+//! crop of correctness machinery (sealed per-epoch commit marks, status
+//! re-seeding, cross-epoch revision ordering) serving exactly one feature —
+//! fast revert. Cut. A computation-affecting edit, *including a revert*,
+//! always mints a fresh generation and rebuilds. That leaves one `reset_time`,
+//! one pair of committed marks, one `rev` space, and only the current
+//! generation readable.
+//!
+//! `target` is deliberately **not** computation-affecting: it is applied at
+//! read time (D56), so editing it never invalidates a slice.
+
+use super::SloDefinition;
+
+/// A stable hash over exactly the computation-affecting fields.
+///
+/// Canonical form — sorted keys, expressions re-rendered from their AST — so
+/// cosmetic re-edits (whitespace in a predicate) hash equal.
+pub fn definition_hash(definition: &SloDefinition) -> String {
+    use std::hash::{Hash, Hasher};
+
+    // Canonical form first: serialize, then re-parse into a BTreeMap-backed
+    // value so key order cannot affect the hash, and normalize the free-text
+    // expressions so a whitespace-only edit does not read as a semantic one.
+    let mut value = crate::utils::json::to_value(definition).unwrap_or(serde_json::Value::Null);
+    canonicalize(&mut value);
+    let canonical = serde_json::to_string(&value).unwrap_or_default();
+
+    let mut hasher = std::collections::hash_map::DefaultHasher::new();
+    canonical.hash(&mut hasher);
+    format!("{:016x}", hasher.finish())
+}
+
+/// Collapse insignificant whitespace inside every string, recursively.
+///
+/// A full AST re-render belongs at the query-safety boundary
+/// (`parse_predicate`), which is where a fragment is validated. This is the
+/// cheap half of the same idea: it stops reformatting from triggering a
+/// 90-day rebuild without pretending to understand SQL.
+fn canonicalize(value: &mut serde_json::Value) {
+    match value {
+        serde_json::Value::String(s) => {
+            *s = s.split_whitespace().collect::<Vec<_>>().join(" ");
+        }
+        serde_json::Value::Array(items) => items.iter_mut().for_each(canonicalize),
+        serde_json::Value::Object(map) => map.iter_mut().for_each(|(_, v)| canonicalize(v)),
+        _ => {}
+    }
+}
+
+/// Whether an edit requires a fresh generation and a rebuild.
+pub fn requires_new_generation(old: &SloDefinition, new: &SloDefinition) -> bool {
+    // Every field of SloDefinition is computation-affecting by construction —
+    // `target` is deliberately not a member (D56). So the hash IS the test.
+    definition_hash(old) != definition_hash(new)
+}
+
+/// Whether a writer whose pass began at `writer_generation` may still commit
+/// against the SLO's current generation (§6b.4b CAS fence).
+///
+/// Delivery of a columnar batch can outlive the generation that ordered it; a
+/// late commit must fail rather than advance the new generation's marks with
+/// the old generation's arithmetic.
+pub fn writer_may_commit(writer_generation: i32, current_generation: i32) -> bool {
+    // Equality, not `>=`: a mismatch in either direction means the writer's
+    // arithmetic does not describe the current definition.
+    writer_generation == current_generation
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::meta::{
+        alerts::Operator,
+        slo::{CountSource, QueryLanguage, SliConfig, SloDefinition},
+    };
+
+    fn count_def() -> SloDefinition {
+        SloDefinition {
+            sli_config: SliConfig::Count {
+                source: CountSource::SingleQuery {
+                    stream: "requests".into(),
+                    stream_type: "logs".into(),
+                    scope: Some("service = 'checkout'".into()),
+                    good_expr: "status_code < 500".into(),
+                },
+            },
+            group_by: None,
+            window_secs: 30 * 86_400,
+            slice_interval_secs: 60,
+        }
+    }
+
+    fn time_slice_def() -> SloDefinition {
+        SloDefinition {
+            sli_config: SliConfig::TimeSlice {
+                stream: "http_metrics".into(),
+                stream_type: "metrics".into(),
+                query_language: QueryLanguage::Sql,
+                query: "SELECT p95(duration_ms) AS zo_slo_value".into(),
+                scope: None,
+                comparator: Operator::LessThan,
+                threshold: 500.0,
+                absent_is_bad: false,
+            },
+            group_by: Some(vec!["region".into()]),
+            window_secs: 7 * 86_400,
+            slice_interval_secs: 300,
+        }
+    }
+
+    // ---- hashing -----------------------------------------------------------
+
+    #[test]
+    fn the_same_definition_hashes_the_same() {
+        assert_eq!(definition_hash(&count_def()), definition_hash(&count_def()));
+    }
+
+    #[test]
+    fn different_definitions_hash_differently() {
+        assert_ne!(
+            definition_hash(&count_def()),
+            definition_hash(&time_slice_def())
+        );
+    }
+
+    #[test]
+    fn the_hash_is_stable_across_calls_and_nonempty() {
+        let h = definition_hash(&count_def());
+        assert!(!h.is_empty());
+        for _ in 0..5 {
+            assert_eq!(definition_hash(&count_def()), h);
+        }
+    }
+
+    // ---- what forces a rebuild ---------------------------------------------
+
+    #[test]
+    fn an_identical_definition_needs_no_new_generation() {
+        assert!(!requires_new_generation(&count_def(), &count_def()));
+    }
+
+    #[test]
+    fn changing_the_good_predicate_forces_a_rebuild() {
+        let mut new = count_def();
+        new.sli_config = SliConfig::Count {
+            source: CountSource::SingleQuery {
+                stream: "requests".into(),
+                stream_type: "logs".into(),
+                scope: Some("service = 'checkout'".into()),
+                good_expr: "status_code < 400".into(),
+            },
+        };
+        assert!(requires_new_generation(&count_def(), &new));
+    }
+
+    #[test]
+    fn changing_the_scope_forces_a_rebuild() {
+        let mut new = count_def();
+        new.sli_config = SliConfig::Count {
+            source: CountSource::SingleQuery {
+                stream: "requests".into(),
+                stream_type: "logs".into(),
+                scope: Some("service = 'cart'".into()),
+                good_expr: "status_code < 500".into(),
+            },
+        };
+        assert!(requires_new_generation(&count_def(), &new));
+    }
+
+    #[test]
+    fn changing_the_stream_forces_a_rebuild() {
+        let mut new = count_def();
+        new.sli_config = SliConfig::Count {
+            source: CountSource::SingleQuery {
+                stream: "other_requests".into(),
+                stream_type: "logs".into(),
+                scope: Some("service = 'checkout'".into()),
+                good_expr: "status_code < 500".into(),
+            },
+        };
+        assert!(requires_new_generation(&count_def(), &new));
+    }
+
+    #[test]
+    fn changing_the_slice_interval_forces_a_rebuild() {
+        let mut new = count_def();
+        new.slice_interval_secs = 300;
+        assert!(requires_new_generation(&count_def(), &new));
+    }
+
+    #[test]
+    fn changing_the_window_forces_a_rebuild() {
+        let mut new = count_def();
+        new.window_secs = 90 * 86_400;
+        assert!(requires_new_generation(&count_def(), &new));
+    }
+
+    #[test]
+    fn changing_the_grouping_forces_a_rebuild() {
+        let mut new = count_def();
+        new.group_by = Some(vec!["region".into()]);
+        assert!(requires_new_generation(&count_def(), &new));
+    }
+
+    #[test]
+    fn changing_the_sli_type_forces_a_rebuild() {
+        assert!(requires_new_generation(&count_def(), &time_slice_def()));
+    }
+
+    #[test]
+    fn changing_a_time_slice_comparator_or_threshold_forces_a_rebuild() {
+        let base = time_slice_def();
+
+        let mut cmp_changed = base.clone();
+        cmp_changed.sli_config = SliConfig::TimeSlice {
+            stream: "http_metrics".into(),
+            stream_type: "metrics".into(),
+            query_language: QueryLanguage::Sql,
+            query: "SELECT p95(duration_ms) AS zo_slo_value".into(),
+            scope: None,
+            comparator: Operator::LessThanEquals,
+            threshold: 500.0,
+            absent_is_bad: false,
+        };
+        assert!(requires_new_generation(&base, &cmp_changed));
+
+        let mut threshold_changed = base.clone();
+        threshold_changed.sli_config = SliConfig::TimeSlice {
+            stream: "http_metrics".into(),
+            stream_type: "metrics".into(),
+            query_language: QueryLanguage::Sql,
+            query: "SELECT p95(duration_ms) AS zo_slo_value".into(),
+            scope: None,
+            comparator: Operator::LessThan,
+            threshold: 250.0,
+            absent_is_bad: false,
+        };
+        assert!(requires_new_generation(&base, &threshold_changed));
+    }
+
+    /// D56/S-13: the target is applied at read time, so it is deliberately
+    /// absent from `SloDefinition` and cannot force a rebuild. This test
+    /// documents that the type system enforces it.
+    #[test]
+    fn the_target_is_not_part_of_the_computation_affecting_definition() {
+        // If `target` were ever added to SloDefinition this would stop
+        // compiling, which is the point.
+        let def = count_def();
+        let json = serde_json::to_value(&def).unwrap();
+        assert!(
+            json.get("target").is_none(),
+            "target must not be computation-affecting (D56)"
+        );
+    }
+
+    /// D59: reverts rebuild. A → B → A produces a *third* generation, not a
+    /// reuse of the first — that is the whole simplification.
+    #[test]
+    fn a_revert_still_requires_a_new_generation() {
+        let a = count_def();
+        let mut b = count_def();
+        b.slice_interval_secs = 300;
+        assert!(requires_new_generation(&a, &b), "A -> B");
+        assert!(
+            requires_new_generation(&b, &a),
+            "B -> A must ALSO rebuild; generation reuse was cut (D59)"
+        );
+    }
+
+    /// The hash remains useful for diagnostics — a revert IS recognisable as
+    /// returning to a previous definition — it simply no longer drives
+    /// generation reuse (D59). This asserts the round trip, which the
+    /// duplicate-of-an-earlier-test it replaced did not.
+    #[test]
+    fn a_revert_is_recognisable_by_hash_even_though_it_still_rebuilds() {
+        let a = count_def();
+        let mut b = count_def();
+        b.slice_interval_secs = 300;
+
+        let (ha, hb) = (definition_hash(&a), definition_hash(&b));
+        assert_ne!(ha, hb, "A and B are different definitions");
+
+        // Revert to A: the hash comes back, but the rebuild still happens.
+        let a_again = count_def();
+        assert_eq!(definition_hash(&a_again), ha, "the hash round-trips");
+        assert!(
+            requires_new_generation(&b, &a_again),
+            "recognising the revert must NOT short-circuit the rebuild"
+        );
+    }
+
+    // ---- AST canonicalization ----------------------------------------------
+
+    fn count_def_with_scope(scope: &str) -> SloDefinition {
+        let mut d = count_def();
+        d.sli_config = SliConfig::Count {
+            source: CountSource::SingleQuery {
+                stream: "requests".into(),
+                stream_type: "logs".into(),
+                scope: Some(scope.into()),
+                good_expr: "status_code < 500".into(),
+            },
+        };
+        d
+    }
+
+    /// §6b.4a promises the hash is over the **canonical** form — expressions
+    /// re-rendered from their AST — so that reformatting a predicate does not
+    /// masquerade as a semantic change and trigger a 90-day rebuild.
+    #[test]
+    fn whitespace_only_predicate_edits_hash_identically() {
+        let a = count_def_with_scope("service = 'checkout'");
+        let b = count_def_with_scope("service   =    'checkout'");
+        assert_eq!(
+            definition_hash(&a),
+            definition_hash(&b),
+            "reformatting must not look like a semantic change"
+        );
+    }
+
+    #[test]
+    fn whitespace_only_predicate_edits_do_not_force_a_rebuild() {
+        let a = count_def_with_scope("service = 'checkout'");
+        let b = count_def_with_scope("  service = 'checkout'  ");
+        assert!(
+            !requires_new_generation(&a, &b),
+            "a cosmetic edit must not rebuild 90 days of history"
+        );
+    }
+
+    #[test]
+    fn a_semantic_predicate_edit_still_forces_a_rebuild() {
+        let a = count_def_with_scope("service = 'checkout'");
+        let b = count_def_with_scope("service = 'cart'");
+        assert!(requires_new_generation(&a, &b));
+    }
+
+    /// Canonicalization must not go so far as to erase meaning: these differ.
+    #[test]
+    fn logically_distinct_predicates_do_not_collide() {
+        let a = count_def_with_scope("a = 1 AND b = 2");
+        let b = count_def_with_scope("a = 1 OR b = 2");
+        assert_ne!(definition_hash(&a), definition_hash(&b));
+    }
+
+    // ---- the CAS fence -----------------------------------------------------
+
+    #[test]
+    fn a_writer_on_the_current_generation_may_commit() {
+        assert!(writer_may_commit(4, 4));
+    }
+
+    /// The stale-writer case: an ingest or backfill pass that finishes after a
+    /// computation edit landed must NOT advance the new generation's marks.
+    #[test]
+    fn a_writer_from_a_superseded_generation_may_not_commit() {
+        assert!(!writer_may_commit(3, 4));
+    }
+
+    #[test]
+    fn a_writer_from_the_future_may_not_commit_either() {
+        // Defensive: should be impossible, but a mismatch in either direction
+        // means the writer's arithmetic does not match the current definition.
+        assert!(!writer_may_commit(5, 4));
+    }
+}

src/config/src/meta/slo/group.rs

@@ -0,0 +1,413 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Grouped SLOs — the exact overall row and the two-tier cap (S-9, S-10, D46).
+//!
+//! The contradiction this module resolves: S-9 requires the overall row to
+//! aggregate **every** group, while S-10 keeps at most a few hundred per-group
+//! rows. Both are satisfied because the overall is computed **in the engine
+//! over the full group set** and the cap truncates only which *per-group* rows
+//! are persisted. Getting that wrong resolves in the direction that inflates
+//! the SLO of the worst service.
+//!
+//! The other subtlety is the time-slice overall. It is not a plain
+//! `MIN(good_flag)` over returned groups — that is blind to groups that
+//! returned nothing. It is a **three-valued** MIN over the *expected* set,
+//! ordered `Bad < Unknown < Good`, so a proven violation beats an unmeasured
+//! group and an all-good-but-incomplete slice is uncovered rather than good.
+
+/// A group's verdict for one slice, ordered by "how much it constrains the
+/// overall". `Bad` wins because a proven violation is a violation regardless
+/// of what else went unmeasured.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
+pub enum SliceVerdict {
+    /// The group was measured and violated the condition.
+    Bad,
+    /// The group was expected but produced no value — unmeasured.
+    Unknown,
+    /// The group was measured and satisfied the condition.
+    Good,
+}
+
+/// The overall verdict for a time-slice SLO's slice: the three-valued MIN over
+/// the **expected group set**.
+///
+/// Takes keyed observations and expected keys rather than a slice plus a
+/// count. A count cannot tell expected `{a, b}` from observed `{a, a}` or
+/// `{a, c}` — both have length 2 and would report `Good` while `b` went
+/// entirely unmeasured, inflating the exact overall row that S-9 depends on.
+///
+/// * every expected key missing from `observed` contributes `Unknown`
+/// * duplicate observations of one key collapse to their **worst** verdict
+/// * keys observed but not expected (a group appearing mid-window, before the active set catches
+///   up) do not fabricate absences and do not vote
+pub fn overall_time_slice(
+    observed: &[(String, SliceVerdict)],
+    expected: &[String],
+) -> SliceVerdict {
+    use std::collections::HashMap;
+
+    // Duplicates of one key collapse to their WORST verdict — a disagreeing
+    // duplicate must not be resolved optimistically.
+    let mut by_key: HashMap<&str, SliceVerdict> = HashMap::new();
+    for (key, verdict) in observed {
+        by_key
+            .entry(key.as_str())
+            .and_modify(|v| *v = (*v).min(*verdict))
+            .or_insert(*verdict);
+    }
+
+    if expected.is_empty() {
+        // Nothing was expected, so "every group was good" is unprovable rather
+        // than vacuously true.
+        return SliceVerdict::Unknown;
+    }
+
+    // Three-valued MIN over the EXPECTED set. A key observed but not expected
+    // neither votes nor fabricates an absence: the active set can lag a
+    // newly-appeared group by one pass.
+    expected
+        .iter()
+        .map(|key| {
+            by_key
+                .get(key.as_str())
+                .copied()
+                .unwrap_or(SliceVerdict::Unknown)
+        })
+        .min()
+        .unwrap_or(SliceVerdict::Unknown)
+}
+
+/// The overall good/total for a **count** SLO: a straight sum across every
+/// group, which is exactly the ungrouped query (S-9).
+pub fn overall_count(groups: &[(f64, f64)]) -> (f64, f64) {
+    // Sum, not a mean of ratios: the overall must be traffic-weighted, or a
+    // tiny perfect group would drag a huge broken one up.
+    groups
+        .iter()
+        .fold((0.0, 0.0), |(g, t), (good, total)| (g + good, t + total))
+}
+
+/// A candidate for the per-group status roster.
+#[derive(Debug, Clone, PartialEq)]
+pub struct RosterCandidate {
+    pub group_key: String,
+    /// Window SLI, aggregated from the persisted slices by the election pass.
+    /// Lower is worse.
+    pub sli: f64,
+}
+
+/// Elect the per-group status roster, worst-SLI-first with a deterministic
+/// tie-break (S-10 tier 2).
+///
+/// This is possible only because tier 1 persists slices for *every* observed
+/// group: ranking a group needs its window SLI, which needs slices that a
+/// naive "cap the slices" design would already have discarded (D55).
+pub fn elect_roster(mut candidates: Vec<RosterCandidate>, cap: usize) -> Vec<String> {
+    // Worst SLI first; ties broken on the group key so the roster is stable
+    // between elections and cannot churn which groups may page.
+    candidates.sort_by(|a, b| {
+        a.sli
+            .partial_cmp(&b.sli)
+            .unwrap_or(std::cmp::Ordering::Equal)
+            .then_with(|| a.group_key.cmp(&b.group_key))
+    });
+    candidates
+        .into_iter()
+        .take(cap)
+        .map(|c| c.group_key)
+        .collect()
+}
+
+/// Whether the observed group count has crossed the hard cap, which freezes
+/// per-group tracking while leaving the overall exact (S-10).
+pub fn is_group_overflow(observed_groups: i64, hard_cap: i64) -> bool {
+    observed_groups > hard_cap
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn cand(key: &str, sli: f64) -> RosterCandidate {
+        RosterCandidate {
+            group_key: key.into(),
+            sli,
+        }
+    }
+
+    // ---- verdict ordering --------------------------------------------------
+
+    #[test]
+    fn verdicts_are_ordered_bad_then_unknown_then_good() {
+        assert!(SliceVerdict::Bad < SliceVerdict::Unknown);
+        assert!(SliceVerdict::Unknown < SliceVerdict::Good);
+    }
+
+    // ---- the three-valued overall -----------------------------------------
+
+    fn obs(pairs: &[(&str, SliceVerdict)]) -> Vec<(String, SliceVerdict)> {
+        pairs.iter().map(|(k, v)| ((*k).to_string(), *v)).collect()
+    }
+
+    fn keys(ks: &[&str]) -> Vec<String> {
+        ks.iter().map(|k| (*k).to_string()).collect()
+    }
+
+    #[test]
+    fn all_groups_good_makes_the_slice_good() {
+        let o = obs(&[
+            ("a", SliceVerdict::Good),
+            ("b", SliceVerdict::Good),
+            ("c", SliceVerdict::Good),
+        ]);
+        assert_eq!(
+            overall_time_slice(&o, &keys(&["a", "b", "c"])),
+            SliceVerdict::Good
+        );
+    }
+
+    #[test]
+    fn any_bad_group_makes_the_slice_bad() {
+        let o = obs(&[
+            ("a", SliceVerdict::Good),
+            ("b", SliceVerdict::Bad),
+            ("c", SliceVerdict::Good),
+        ]);
+        assert_eq!(
+            overall_time_slice(&o, &keys(&["a", "b", "c"])),
+            SliceVerdict::Bad
+        );
+    }
+
+    #[test]
+    fn an_absent_group_makes_an_otherwise_good_slice_uncovered() {
+        let o = obs(&[("a", SliceVerdict::Good), ("b", SliceVerdict::Good)]);
+        assert_eq!(
+            overall_time_slice(&o, &keys(&["a", "b", "c"])),
+            SliceVerdict::Unknown,
+            "c was expected and never reported — `all groups good` is unprovable"
+        );
+    }
+
+    /// A count-based signature could not catch this: two observations of `a`
+    /// and none of `b` has the same length as one each.
+    #[test]
+    fn a_duplicated_group_does_not_stand_in_for_a_missing_one() {
+        let o = obs(&[("a", SliceVerdict::Good), ("a", SliceVerdict::Good)]);
+        assert_eq!(
+            overall_time_slice(&o, &keys(&["a", "b"])),
+            SliceVerdict::Unknown,
+            "b is unmeasured however many times a reported"
+        );
+    }
+
+    /// Likewise an unexpected key must not fill an expected one's slot.
+    #[test]
+    fn an_unexpected_group_does_not_stand_in_for_a_missing_one() {
+        let o = obs(&[("a", SliceVerdict::Good), ("zz", SliceVerdict::Good)]);
+        assert_eq!(
+            overall_time_slice(&o, &keys(&["a", "b"])),
+            SliceVerdict::Unknown
+        );
+    }
+
+    #[test]
+    fn duplicate_observations_of_one_key_collapse_to_the_worst() {
+        let o = obs(&[("a", SliceVerdict::Good), ("a", SliceVerdict::Bad)]);
+        assert_eq!(
+            overall_time_slice(&o, &keys(&["a"])),
+            SliceVerdict::Bad,
+            "a disagreeing duplicate must not be resolved optimistically"
+        );
+    }
+
+    /// The invariant break a previous PRD draft had: if absence dominated
+    /// badness, a bad group plus an absent sibling would make the slice
+    /// *uncovered*, excluding it from the overall's denominator and letting
+    /// the overall read HIGHER than the group.
+    #[test]
+    fn a_proven_violation_beats_an_unmeasured_sibling() {
+        let o = obs(&[("a", SliceVerdict::Bad)]);
+        assert_eq!(
+            overall_time_slice(&o, &keys(&["a", "b"])),
+            SliceVerdict::Bad,
+            "absence must not mask a violation"
+        );
+    }
+
+    #[test]
+    fn the_overall_never_exceeds_the_worst_group() {
+        // Ten slices; group A is bad in slice 3; group B is absent in slice 3.
+        let expected = keys(&["a", "b"]);
+        let mut a_good = 0;
+        let mut overall_good = 0;
+        let mut overall_measured = 0;
+        for i in 0..10 {
+            let o = if i == 3 {
+                obs(&[("a", SliceVerdict::Bad)])
+            } else {
+                obs(&[("a", SliceVerdict::Good), ("b", SliceVerdict::Good)])
+            };
+            if o.iter().any(|(k, v)| k == "a" && *v == SliceVerdict::Good) {
+                a_good += 1;
+            }
+            match overall_time_slice(&o, &expected) {
+                SliceVerdict::Good => {
+                    overall_good += 1;
+                    overall_measured += 1;
+                }
+                SliceVerdict::Bad => overall_measured += 1,
+                SliceVerdict::Unknown => {}
+            }
+        }
+        let a_sli = a_good as f64 / 10.0;
+        let overall_sli = overall_good as f64 / overall_measured as f64;
+        assert!(
+            overall_sli <= a_sli,
+            "overall {overall_sli} exceeded group A {a_sli} — S-9 invariant broken"
+        );
+    }
+
+    #[test]
+    fn an_empty_expected_set_is_unknown_not_good() {
+        assert_eq!(overall_time_slice(&[], &[]), SliceVerdict::Unknown);
+    }
+
+    #[test]
+    fn every_group_absent_is_unknown() {
+        assert_eq!(
+            overall_time_slice(&[], &keys(&["a", "b", "c"])),
+            SliceVerdict::Unknown
+        );
+    }
+
+    /// A group that reported before the active set caught up must not make the
+    /// slice uncovered.
+    #[test]
+    fn extra_observations_do_not_fabricate_absences() {
+        let o = obs(&[
+            ("a", SliceVerdict::Good),
+            ("b", SliceVerdict::Good),
+            ("c", SliceVerdict::Good),
+        ]);
+        assert_eq!(
+            overall_time_slice(&o, &keys(&["a", "b"])),
+            SliceVerdict::Good
+        );
+    }
+
+    #[test]
+    fn an_unexpected_group_does_not_vote_bad() {
+        // `zz` is not in the active set; its verdict must not drag the overall
+        // down before the roster admits it.
+        let o = obs(&[("a", SliceVerdict::Good), ("zz", SliceVerdict::Bad)]);
+        assert_eq!(overall_time_slice(&o, &keys(&["a"])), SliceVerdict::Good);
+    }
+
+    // ---- count overall -----------------------------------------------------
+
+    #[test]
+    fn count_overall_sums_every_group() {
+        let (good, total) = overall_count(&[(90.0, 100.0), (8.0, 10.0), (1.0, 1.0)]);
+        assert_eq!((good, total), (99.0, 111.0));
+    }
+
+    /// S-9: the count overall is `Σgood / Σtotal`, which weights by traffic —
+    /// NOT the mean of the per-group ratios.
+    #[test]
+    fn count_overall_is_traffic_weighted_not_an_average_of_ratios() {
+        // A tiny perfect group must not drag a huge broken one up.
+        let (good, total) = overall_count(&[(0.0, 1_000_000.0), (1.0, 1.0)]);
+        let overall = 100.0 * good / total;
+        let mean_of_ratios: f64 = (0.0 + 100.0) / 2.0;
+        assert!(overall < 0.01);
+        assert!(
+            (mean_of_ratios - 50.0).abs() < 1e-9 && overall < mean_of_ratios,
+            "averaging ratios would report 50%"
+        );
+    }
+
+    #[test]
+    fn count_overall_of_nothing_is_zero_zero() {
+        assert_eq!(overall_count(&[]), (0.0, 0.0));
+    }
+
+    // ---- roster election ---------------------------------------------------
+
+    #[test]
+    fn the_roster_keeps_the_worst_groups() {
+        let out = elect_roster(
+            vec![
+                cand("a", 99.9),
+                cand("b", 98.0),
+                cand("c", 99.99),
+                cand("d", 95.0),
+            ],
+            2,
+        );
+        assert_eq!(out, vec!["d".to_string(), "b".to_string()]);
+    }
+
+    #[test]
+    fn the_roster_is_capped() {
+        let cands: Vec<_> = (0..100)
+            .map(|i| cand(&format!("g{i:03}"), 99.0 - i as f64 * 0.01))
+            .collect();
+        assert_eq!(elect_roster(cands, 10).len(), 10);
+    }
+
+    #[test]
+    fn a_roster_smaller_than_the_cap_keeps_everything() {
+        let out = elect_roster(vec![cand("a", 99.0), cand("b", 98.0)], 500);
+        assert_eq!(out.len(), 2);
+    }
+
+    /// Determinism matters: an unstable roster would churn which groups can
+    /// page between elections.
+    #[test]
+    fn ties_break_deterministically_on_the_group_key() {
+        let out = elect_roster(
+            vec![cand("zebra", 99.0), cand("alpha", 99.0), cand("mike", 99.0)],
+            2,
+        );
+        assert_eq!(out, vec!["alpha".to_string(), "mike".to_string()]);
+    }
+
+    #[test]
+    fn election_is_order_independent() {
+        let forward = elect_roster(vec![cand("a", 99.0), cand("b", 98.0), cand("c", 97.0)], 2);
+        let reverse = elect_roster(vec![cand("c", 97.0), cand("b", 98.0), cand("a", 99.0)], 2);
+        assert_eq!(forward, reverse);
+    }
+
+    #[test]
+    fn a_zero_cap_elects_nobody() {
+        assert!(elect_roster(vec![cand("a", 1.0)], 0).is_empty());
+    }
+
+    // ---- overflow ----------------------------------------------------------
+
+    #[test]
+    fn observed_groups_under_the_hard_cap_do_not_overflow() {
+        assert!(!is_group_overflow(9_999, 10_000));
+        assert!(!is_group_overflow(10_000, 10_000));
+    }
+
+    #[test]
+    fn observed_groups_past_the_hard_cap_overflow() {
+        assert!(is_group_overflow(10_001, 10_000));
+    }
+}

src/config/src/meta/slo/lenient_f64.rs

@@ -0,0 +1,164 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Deserializing `f64` under `serde_json/arbitrary_precision` (D61).
+//!
+//! The workspace enables `serde_json`'s `arbitrary_precision` feature
+//! (`Cargo.toml`). That feature changes how a number is represented **inside a
+//! buffered `Value`**: instead of an f64 it becomes a one-key map,
+//! `{"$serde_json::private::Number": "14.4"}`. Nothing buffers in the common
+//! path, so this is invisible until something does — and then a plain
+//! `f64` field fails with `invalid type: map, expected f64`.
+//!
+//! Two things buffer:
+//!
+//! * an **internally-tagged enum**, which is why `SliConfig` and `CountSource` use adjacent tagging
+//!   instead (D61);
+//! * **`#[serde(flatten)]`**, which is how an `SloCondition` reaches this code — it arrives nested
+//!   inside `CreateAlertRequestBody`'s flattened `alert` field.
+//!
+//! The second was found by end-to-end testing, not by review: every unit test
+//! deserialized `SloCondition` directly, where no buffering happens and a
+//! plain `f64` works fine.
+
+use serde::{Deserialize, Deserializer, de::Error};
+
+/// Deserialize an `f64` that may have been buffered into an
+/// arbitrary-precision map.
+pub fn deserialize<'de, D>(d: D) -> Result<f64, D::Error>
+where
+    D: Deserializer<'de>,
+{
+    match Lenient::deserialize(d)? {
+        Lenient::Num(v) => Ok(v),
+        Lenient::Buffered(v) => from_value(&v).ok_or_else(|| D::Error::custom("expected a number")),
+    }
+}
+
+/// The `Option` form, for fields like `warning`.
+pub fn deserialize_opt<'de, D>(d: D) -> Result<Option<f64>, D::Error>
+where
+    D: Deserializer<'de>,
+{
+    match Option::<Lenient>::deserialize(d)? {
+        None => Ok(None),
+        Some(Lenient::Num(v)) => Ok(Some(v)),
+        Some(Lenient::Buffered(v)) => Ok(Some(
+            from_value(&v).ok_or_else(|| D::Error::custom("expected a number"))?,
+        )),
+    }
+}
+
+#[derive(Deserialize)]
+#[serde(untagged)]
+enum Lenient {
+    /// The ordinary path — no buffering happened.
+    Num(f64),
+    /// A buffered value, which under `arbitrary_precision` is a map.
+    Buffered(serde_json::Value),
+}
+
+/// Pull an f64 out of a `Value`, including the arbitrary-precision map form.
+fn from_value(v: &serde_json::Value) -> Option<f64> {
+    if let Some(n) = v.as_f64() {
+        return Some(n);
+    }
+    // `{"$serde_json::private::Number": "14.4"}` — the private key is not
+    // matched by name because it is private and could change; any one-key map
+    // whose value parses as a number is accepted instead.
+    let obj = v.as_object()?;
+    if obj.len() != 1 {
+        return None;
+    }
+    let raw = obj.values().next()?;
+    raw.as_str()
+        .and_then(|s| s.parse::<f64>().ok())
+        .or_else(|| raw.as_f64())
+}
+
+#[cfg(test)]
+mod tests {
+    use serde::Deserialize;
+
+    use super::*;
+
+    #[derive(Debug, Deserialize, PartialEq)]
+    struct Holder {
+        #[serde(deserialize_with = "deserialize")]
+        critical: f64,
+        #[serde(default, deserialize_with = "deserialize_opt")]
+        warning: Option<f64>,
+    }
+
+    /// The ordinary path must keep working exactly as before.
+    #[test]
+    fn a_plain_number_deserializes() {
+        let h: Holder = serde_json::from_str(r#"{"critical": 14.4, "warning": 6}"#).unwrap();
+        assert_eq!(h.critical, 14.4);
+        assert_eq!(h.warning, Some(6.0));
+    }
+
+    #[test]
+    fn an_absent_option_is_none() {
+        let h: Holder = serde_json::from_str(r#"{"critical": 1.0}"#).unwrap();
+        assert_eq!(h.warning, None);
+    }
+
+    #[test]
+    fn an_explicit_null_option_is_none() {
+        let h: Holder = serde_json::from_str(r#"{"critical": 1.0, "warning": null}"#).unwrap();
+        assert_eq!(h.warning, None);
+    }
+
+    /// The failure this module exists for. Reproduces the buffering that
+    /// `#[serde(flatten)]` performs: the JSON is parsed to a `Value` first,
+    /// which under `arbitrary_precision` turns every number into a map.
+    #[test]
+    fn a_number_buffered_through_value_deserializes() {
+        let v: serde_json::Value =
+            serde_json::from_str(r#"{"critical": 14.4, "warning": 6}"#).unwrap();
+        let h: Holder = serde_json::from_value(v).expect("must survive buffering");
+        assert_eq!(h.critical, 14.4);
+        assert_eq!(h.warning, Some(6.0));
+    }
+
+    /// The end-to-end shape: a struct reached through a flatten, which is how
+    /// an SloCondition arrives inside CreateAlertRequestBody.
+    #[test]
+    fn a_number_inside_a_flattened_struct_deserializes() {
+        #[derive(Debug, Deserialize)]
+        struct Outer {
+            name: String,
+            #[serde(flatten)]
+            inner: Holder,
+        }
+        let o: Outer =
+            serde_json::from_str(r#"{"name":"a","critical": 14.4, "warning": 6}"#).unwrap();
+        assert_eq!(o.name, "a");
+        assert_eq!(o.inner.critical, 14.4);
+        assert_eq!(o.inner.warning, Some(6.0));
+    }
+
+    #[test]
+    fn a_string_is_still_rejected() {
+        assert!(serde_json::from_str::<Holder>(r#"{"critical": "high"}"#).is_err());
+    }
+
+    #[test]
+    fn an_integer_widens() {
+        let h: Holder = serde_json::from_str(r#"{"critical": 14}"#).unwrap();
+        assert_eq!(h.critical, 14.0);
+    }
+}

src/config/src/meta/slo/math.rs

@@ -0,0 +1,369 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! The normative SLO math — `alerts_2.md` §6b.6a.
+//!
+//! Stated once, here, and derived everywhere else. Every number the UI shows
+//! and every threshold comparison an SLO alert makes comes through these
+//! functions.
+//!
+//! ```text
+//! sli(R)          = 100 × Σgood / Σtotal          undefined if Σtotal = 0 (SA-18)
+//! error_rate(R)   = 100 − sli(R)
+//! burn_rate(R)    = error_rate(R) / (100 − target)
+//! max_burn_rate   = 100 / (100 − target)          the SA-6 cap
+//!
+//! over the full window W:
+//!   consumed%  = 100 × burn_rate(W)
+//!   remaining% = 100 − consumed%
+//! ```
+//!
+//! The last identity matters: **error-budget consumption and burn rate are the
+//! same quantity at two scalings**, differing only in the window they are read
+//! over. That is why one evaluator serves both SLO-alert kinds.
+
+/// SLI as a percentage. `None` when there is nothing to divide by — a covered
+/// window with zero events is *not* 100% and *not* 0%, it is undefined, and
+/// SA-18 turns that into "unobserved" rather than a recovery.
+pub fn sli(good: f64, total: f64) -> Option<f64> {
+    // `<= 0` rather than `== 0`: a negative total is corruption, not a 0% SLI.
+    // NaN is checked explicitly — it must not reach the division either.
+    if total.is_nan() || total <= 0.0 {
+        return None;
+    }
+    Some(100.0 * good / total)
+}
+
+/// Error rate as a percentage: the complement of the SLI.
+pub fn error_rate(sli: f64) -> f64 {
+    100.0 - sli
+}
+
+/// Burn rate: observed error rate divided by the budgeted error rate.
+///
+/// 1.0 means the budget lands exactly at the window's end. 14.4 over a 30-day
+/// SLO exhausts it in about two days.
+pub fn burn_rate(sli: f64, target: f64) -> f64 {
+    error_rate(sli) / (100.0 - target)
+}
+
+/// The largest burn rate physically reachable for a target — `1/(1 − target)`.
+/// A threshold above this needs an error rate over 100% and can never fire
+/// (SA-6).
+pub fn max_burn_rate(target: f64) -> f64 {
+    // Identically `burn_rate(0.0, target)` — the burn rate of a totally failed
+    // window. Written out so the SA-6 cap reads as its own idea.
+    100.0 / (100.0 - target)
+}
+
+/// Percentage of the error budget consumed over the full window.
+/// Identically `100 × burn_rate(window)`.
+pub fn error_budget_consumed(sli_window: f64, target: f64) -> f64 {
+    100.0 * burn_rate(sli_window, target)
+}
+
+/// Percentage of the error budget remaining — **signed**. A blown budget reads
+/// negative and is rendered that way; it is never clamped to zero (S-6).
+pub fn error_budget_remaining(sli_window: f64, target: f64) -> f64 {
+    100.0 - error_budget_consumed(sli_window, target)
+}
+
+/// How long the window's budget lasts at a sustained burn rate:
+/// `window / burn`. `None` when the burn rate is zero or negative — the budget
+/// is not being consumed at all.
+pub fn time_to_exhaust_secs(window_secs: i64, burn: f64) -> Option<i64> {
+    // NaN is checked explicitly — it must not reach the division.
+    if burn.is_nan() || burn <= 0.0 {
+        return None;
+    }
+    Some((window_secs as f64 / burn) as i64)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// The §9 accuracy gate is 0.001 percentage points; these are closed-form
+    /// identities, so the tests hold themselves several orders tighter.
+    const EPS: f64 = 1e-9;
+
+    /// Absolute for small values, **relative** for large ones.
+    ///
+    /// A flat `1e-9` was tighter than f64 can represent for the larger
+    /// quantities here: `max_burn_rate(99.99)` is `9999.999999994885`, off by
+    /// 5.1e-9 purely from `100.0 - 99.99` not being exact. That is a
+    /// representation limit, not an arithmetic error — the relative error is
+    /// 5e-13 — and it is well inside the §9 gate. Asserting a flat epsilon
+    /// against a four-digit multiplier was the mistake.
+    fn close(a: f64, b: f64) -> bool {
+        (a - b).abs() <= EPS.max(b.abs() * 1e-12)
+    }
+
+    // ---- sli ---------------------------------------------------------------
+
+    #[test]
+    fn sli_is_the_good_fraction_as_a_percentage() {
+        assert!(close(sli(999.0, 1000.0).unwrap(), 99.9));
+        assert!(close(sli(1.0, 2.0).unwrap(), 50.0));
+    }
+
+    #[test]
+    fn sli_of_a_perfect_window_is_exactly_one_hundred() {
+        assert_eq!(sli(1000.0, 1000.0), Some(100.0));
+    }
+
+    #[test]
+    fn sli_of_a_totally_failed_window_is_exactly_zero() {
+        assert_eq!(sli(0.0, 1000.0), Some(0.0));
+    }
+
+    /// SA-18: a covered window with no events is *undefined*, not zero and not
+    /// a hundred. Returning a number here is how "traffic stopped entirely"
+    /// silently clears a page.
+    #[test]
+    fn sli_is_undefined_when_total_is_zero() {
+        assert_eq!(sli(0.0, 0.0), None);
+    }
+
+    #[test]
+    fn sli_is_undefined_for_a_negative_total() {
+        // Defensive: totals are counts or seconds and cannot be negative; if
+        // one ever is, that is corruption, not a 0% SLI.
+        assert_eq!(sli(0.0, -1.0), None);
+    }
+
+    // ---- error_rate --------------------------------------------------------
+
+    #[test]
+    fn error_rate_is_the_complement_of_the_sli() {
+        assert!(close(error_rate(99.9), 0.1));
+        assert!(close(error_rate(100.0), 0.0));
+        assert!(close(error_rate(0.0), 100.0));
+    }
+
+    // ---- burn_rate ---------------------------------------------------------
+
+    /// The §6b.6a sanity checks, encoded.
+    #[test]
+    fn burn_rate_is_zero_for_a_perfect_sli() {
+        assert!(close(burn_rate(100.0, 99.9), 0.0));
+    }
+
+    #[test]
+    fn burn_rate_is_one_when_the_sli_sits_exactly_on_target() {
+        assert!(close(burn_rate(99.9, 99.9), 1.0));
+        assert!(close(burn_rate(99.0, 99.0), 1.0));
+        assert!(close(burn_rate(95.0, 95.0), 1.0));
+    }
+
+    #[test]
+    fn burn_rate_of_a_totally_failed_window_equals_the_max() {
+        assert!(close(burn_rate(0.0, 99.9), max_burn_rate(99.9)));
+        assert!(close(burn_rate(0.0, 99.0), max_burn_rate(99.0)));
+    }
+
+    /// Datadog's headline example: 14.4 over a 30-day SLO.
+    #[test]
+    fn burn_rate_matches_the_datadog_worked_example() {
+        // A 99.9% target has a 0.1% budget. An observed error rate of 1.44%
+        // is 14.4× the budgeted rate.
+        assert!(close(burn_rate(100.0 - 1.44, 99.9), 14.4));
+    }
+
+    #[test]
+    fn burn_rate_scales_linearly_with_the_error_rate() {
+        let a = burn_rate(99.0, 99.9); // 1% errors
+        let b = burn_rate(98.0, 99.9); // 2% errors
+        assert!(close(b, a * 2.0));
+    }
+
+    // ---- max_burn_rate -----------------------------------------------------
+
+    #[test]
+    fn max_burn_rate_is_the_reciprocal_of_the_budget() {
+        assert!(close(max_burn_rate(99.0), 100.0));
+        assert!(close(max_burn_rate(99.9), 1000.0));
+        assert!(close(max_burn_rate(99.99), 10_000.0));
+        assert!(close(max_burn_rate(90.0), 10.0));
+    }
+
+    /// SA-6's direction, which an earlier PRD draft had inverted: TIGHTENING a
+    /// target RAISES the ceiling, so it can never strand an existing
+    /// threshold. Loosening is the dangerous direction.
+    #[test]
+    fn tightening_the_target_raises_the_max_burn_rate() {
+        assert!(max_burn_rate(99.9) > max_burn_rate(99.0));
+    }
+
+    #[test]
+    fn loosening_the_target_lowers_the_max_burn_rate() {
+        // 99.9 -> 99.0 drops the ceiling from 1000 to 100, which can strand a
+        // saved threshold of 500 as permanently unfireable.
+        let before = max_burn_rate(99.9);
+        let after = max_burn_rate(99.0);
+        assert!(after < before);
+        assert!(500.0 <= before && 500.0 > after);
+    }
+
+    #[test]
+    fn no_reachable_burn_rate_exceeds_the_max() {
+        for target in [90.0, 99.0, 99.9, 99.99] {
+            for sli_v in [0.0, 1.0, 50.0, 99.0, 100.0] {
+                assert!(
+                    burn_rate(sli_v, target) <= max_burn_rate(target) + EPS,
+                    "burn({sli_v}, {target}) exceeded the max"
+                );
+            }
+        }
+    }
+
+    // ---- error budget ------------------------------------------------------
+
+    #[test]
+    fn a_perfect_window_consumes_no_budget_and_leaves_all_of_it() {
+        assert!(close(error_budget_consumed(100.0, 99.9), 0.0));
+        assert!(close(error_budget_remaining(100.0, 99.9), 100.0));
+    }
+
+    #[test]
+    fn sitting_exactly_on_target_consumes_the_whole_budget() {
+        assert!(close(error_budget_consumed(99.9, 99.9), 100.0));
+        assert!(close(error_budget_remaining(99.9, 99.9), 0.0));
+    }
+
+    /// S-6: remaining is SIGNED. Clamping it to zero hides how deep the hole
+    /// is, which is the number an SRE actually needs.
+    #[test]
+    fn a_blown_budget_reports_negative_remaining() {
+        // 99.8% against a 99.9% target = twice the budget spent.
+        let remaining = error_budget_remaining(99.8, 99.9);
+        assert!(remaining < 0.0, "expected negative, got {remaining}");
+        assert!(close(remaining, -100.0));
+    }
+
+    #[test]
+    fn consumed_and_remaining_always_sum_to_one_hundred() {
+        for (sli_v, target) in [(100.0, 99.9), (99.95, 99.9), (99.9, 99.9), (99.5, 99.9)] {
+            let sum = error_budget_consumed(sli_v, target) + error_budget_remaining(sli_v, target);
+            assert!(close(sum, 100.0), "sli={sli_v} target={target} sum={sum}");
+        }
+    }
+
+    /// The identity that lets one evaluator serve both alert kinds: an
+    /// error-budget alert is a burn-rate read over the whole window, ×100.
+    #[test]
+    fn consumed_is_exactly_one_hundred_times_the_window_burn_rate() {
+        for (sli_v, target) in [(99.95, 99.9), (99.5, 99.9), (98.0, 99.0), (100.0, 95.0)] {
+            assert!(
+                close(
+                    error_budget_consumed(sli_v, target),
+                    100.0 * burn_rate(sli_v, target)
+                ),
+                "identity failed for sli={sli_v} target={target}"
+            );
+        }
+    }
+
+    // ---- time to exhaust ---------------------------------------------------
+
+    #[test]
+    fn budget_lasts_exactly_the_window_at_burn_rate_one() {
+        let window = 30 * 86_400;
+        assert_eq!(time_to_exhaust_secs(window, 1.0), Some(window));
+    }
+
+    /// Datadog's stated example: 14.4 exhausts a 30-day budget in ~2 days.
+    #[test]
+    fn burn_rate_fourteen_point_four_exhausts_thirty_days_in_about_two() {
+        let secs = time_to_exhaust_secs(30 * 86_400, 14.4).unwrap();
+        let days = secs as f64 / 86_400.0;
+        assert!((days - 2.083).abs() < 0.01, "got {days} days");
+    }
+
+    #[test]
+    fn a_higher_burn_rate_exhausts_the_budget_sooner() {
+        let w = 30 * 86_400;
+        assert!(time_to_exhaust_secs(w, 14.4).unwrap() < time_to_exhaust_secs(w, 3.0).unwrap());
+    }
+
+    #[test]
+    fn a_zero_burn_rate_never_exhausts_the_budget() {
+        assert_eq!(time_to_exhaust_secs(30 * 86_400, 0.0), None);
+    }
+
+    #[test]
+    fn a_negative_burn_rate_never_exhausts_the_budget() {
+        assert_eq!(time_to_exhaust_secs(30 * 86_400, -1.0), None);
+    }
+
+    // ---- cross-checks ------------------------------------------------------
+
+    /// Every suggested Datadog row should spend the documented fraction of the
+    /// budget over its long window.
+    ///
+    /// Derived through `burn_rate` and `error_budget_consumed` rather than from
+    /// literals: an earlier version of this test did the arithmetic inline and
+    /// would have passed with every function in this module returning zero.
+    #[test]
+    fn datadog_suggested_rows_consume_the_documented_budget_fraction() {
+        // (slo window days, burn, long window hours, documented budget %)
+        let rows: [(f64, f64, f64, f64); 9] = [
+            (30.0, 14.4, 1.0, 2.0),
+            (30.0, 6.0, 6.0, 5.0),
+            (30.0, 3.0, 24.0, 10.0),
+            (7.0, 16.8, 1.0, 10.0),
+            (7.0, 5.6, 6.0, 20.0),
+            (7.0, 2.8, 24.0, 40.0),
+            (90.0, 21.6, 1.0, 1.0),
+            (90.0, 10.8, 6.0, 3.0),
+            (90.0, 4.5, 24.0, 5.0),
+        ];
+        let target = 99.9;
+        for (window_days, burn, long_hours, budget_pct) in rows {
+            // Invert the burn rate to the SLI that would produce it, then push
+            // that SLI back through the real functions.
+            let sli_v = 100.0 - burn * (100.0 - target);
+            assert!(
+                close(burn_rate(sli_v, target), burn),
+                "burn_rate did not round-trip for {burn}"
+            );
+
+            // Budget spent over the long window = consumed% × (long / window).
+            let consumed_over_window = error_budget_consumed(sli_v, target);
+            let fraction = consumed_over_window * (long_hours / 24.0) / window_days;
+            assert!(
+                (fraction - budget_pct).abs() < 1e-6,
+                "burn {burn} over {long_hours}h of a {window_days}d SLO spends {fraction:.4}%, \
+                 Datadog documents {budget_pct}%"
+            );
+        }
+    }
+
+    /// The exhaustion helper must agree with the same rows: window ÷ burn.
+    #[test]
+    fn datadog_suggested_rows_exhaust_the_budget_over_their_long_window() {
+        // At burn B, the budget lasts window/B. Firing after `long` hours means
+        // the fraction spent is long / (window/B) — the check above, restated
+        // through time_to_exhaust_secs.
+        let window_secs = 30 * 86_400;
+        let secs = time_to_exhaust_secs(window_secs, 14.4).unwrap();
+        let spent_in_one_hour = 3600.0 / secs as f64;
+        assert!(
+            (spent_in_one_hour * 100.0 - 2.0).abs() < 0.01,
+            "expected ~2% of the budget in 1h, got {:.3}%",
+            spent_in_one_hour * 100.0
+        );
+    }
+}

src/config/src/meta/slo/slice.rs

@@ -0,0 +1,672 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Slice rows: dedupe, gap-fill, and the one clamp that matters — §6b.4a/§6b.4b.
+//!
+//! **Slices publish at-least-once, like every other stream in the product**
+//! (D64). There is no transactional publication protocol here, and that is a
+//! deliberate reversal of three earlier designs. The reasoning, in short:
+//!
+//! * A torn batch's rows are **not corrupt** — they were computed from real data by the query that
+//!   would have run anyway. The only thing that did not happen is their delta being folded into the
+//!   running aggregate.
+//! * The running aggregate is a **cache**, rebuilt from slices by reconciliation. So a torn batch
+//!   causes cache drift, which self-heals.
+//! * Alerts read the cache (`slo_status.burn_windows`, §6b.4c), never slices directly, so an
+//!   unaccounted slice cannot page anyone.
+//! * A partially-written batch shows up as **reduced coverage**, and coverage gating — which exists
+//!   for search outages anyway — already bounds the damage: above the floor it is bounded by
+//!   definition, below it the alert freezes.
+//! * The trailing-K recompute re-emits any slice near a crash on the next pass regardless.
+//!
+//! What survives from those designs, because each earns its place for a reason
+//! unrelated to torn batches:
+//!
+//! * **The watermark**, a forward clamp only. Not a commit barrier — it stops readers seeing the
+//!   *currently filling* slice, which for a time-slice SLI would classify against a partial bucket
+//!   and then flip.
+//! * **Latest-revision-wins dedupe**, needed for late data. `MAX(good)` is *wrong* here: a
+//!   recomputed time-slice can flip good → bad, and MAX would keep the stale 300 — a failure that
+//!   only ever over-reports uptime.
+//! * **Type-specific gap-fill** (D48): "no rows in the bucket" is an observation of zero traffic
+//!   for a count SLI and an absence of measurement for a time-slice one.
+//! * **Ingest-boundary validation**, so non-finite values never reach a slice.
+
+use super::SliType;
+
+/// One row of the `slo_slices` stream.
+#[derive(Debug, Clone, PartialEq)]
+pub struct SliceRow {
+    pub slo_id: String,
+    /// The definition this slice was measured under — and, since D59, also the
+    /// writing epoch. Reads filter to the current generation only.
+    pub definition_generation: i32,
+    /// `""` is the exact overall row (S-9).
+    pub group_key: String,
+    pub slice_start: i64,
+    /// `f64` **units**: events for a count SLI, seconds for time-slice and
+    /// alert-based. Unifying on units is what lets one evaluator serve all
+    /// three.
+    pub good: f64,
+    pub total: f64,
+    /// Monotonic within the generation; higher wins on the same key. Exists
+    /// for late-data re-emission, not for publication ordering.
+    pub rev: i64,
+}
+
+/// Whether a row is visible to readers.
+///
+/// One clamp: the row must belong to the current generation and start strictly
+/// before the watermark. Three earlier designs added a backward barrier here —
+/// per-writer committed marks, an abandoned-batch set, a write-ahead manifest
+/// — to hide rows from batches that never committed. D64 removed all of it:
+/// those rows are valid measurements whose delta was never folded into the
+/// cache, the cache is rebuilt by reconciliation, and alerts read the cache
+/// rather than slices.
+pub fn is_visible(row: &SliceRow, watermark_end: i64, generation: i32) -> bool {
+    row.definition_generation == generation && row.slice_start < watermark_end
+}
+
+/// Collapse duplicate revisions of the same key, keeping the **latest
+/// revision** (never the max value).
+///
+/// Input need not be sorted. Output is ascending by `(group_key, slice_start)`
+/// so downstream aggregation is deterministic.
+pub fn dedupe_latest_rev(rows: Vec<SliceRow>) -> Vec<SliceRow> {
+    use std::collections::BTreeMap;
+
+    // BTreeMap so the output order is deterministic by (group_key,
+    // slice_start) without a separate sort.
+    let mut best: BTreeMap<(String, i64), SliceRow> = BTreeMap::new();
+    for row in rows {
+        let key = (row.group_key.clone(), row.slice_start);
+        match best.get(&key) {
+            // Strictly greater: an equal revision must not flip the winner, or
+            // the result would depend on input order.
+            Some(existing) if existing.rev >= row.rev => {}
+            _ => {
+                best.insert(key, row);
+            }
+        }
+    }
+    best.into_values().collect()
+}
+
+/// Clamp and dedupe — the canonical read path.
+pub fn visible_slices(rows: Vec<SliceRow>, watermark_end: i64, generation: i32) -> Vec<SliceRow> {
+    // Clamp BEFORE dedupe: a row from another generation must not win a key
+    // just because its revision is higher.
+    let visible = rows
+        .into_iter()
+        .filter(|r| is_visible(r, watermark_end, generation))
+        .collect();
+    dedupe_latest_rev(visible)
+}
+
+/// What the ingest job emits for a bucket the query did not return.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum GapFill {
+    /// Emit `good = 0, total = 0` — a real observation of zero traffic, which
+    /// counts as covered.
+    CoveredZero,
+    /// Emit `good = 0, total = interval` — the slice was PROVEN empty and
+    /// absence is the failure (`absent_is_bad`). Covered, and fully bad.
+    CoveredBad,
+    /// Emit nothing — the slice is a gap and reduces coverage.
+    Nothing,
+}
+
+/// How a missing bucket is interpreted for a specific SLO definition.
+///
+/// Refines [`gap_fill_policy`]: the per-TYPE answer (D48) holds except where
+/// the definition itself says otherwise — a time-slice SLO with
+/// `absent_is_bad` treats a proven-empty bucket as downtime rather than as a
+/// gap. Callers holding a config should ask here; the type-level function
+/// remains for callers that have only the discriminant.
+pub fn gap_fill_policy_for(sli: &crate::meta::slo::SliConfig) -> GapFill {
+    if let crate::meta::slo::SliConfig::TimeSlice {
+        absent_is_bad: true,
+        ..
+    } = sli
+    {
+        return GapFill::CoveredBad;
+    }
+    gap_fill_policy(sli.sli_type())
+}
+
+/// How a missing bucket is interpreted, per SLI type (D48).
+pub fn gap_fill_policy(sli_type: SliType) -> GapFill {
+    match sli_type {
+        // "No rows" is a real observation of zero traffic.
+        SliType::Count => GapFill::CoveredZero,
+        // The aggregate had no input, so there is no value to compare.
+        SliType::TimeSlice => GapFill::Nothing,
+        // Coverage comes from the triggers stream instead (S-16, D65).
+        SliType::Alert => GapFill::Nothing,
+    }
+}
+
+/// Fill the buckets a successful query did not return, per the type's policy.
+///
+/// `observed` are the rows the query produced; `expected_starts` is the full
+/// aligned grid for the pass. Only applies to a **successful** query — a
+/// failed one emits nothing for any type, which is what coverage is for.
+pub fn fill_gaps(
+    observed: Vec<SliceRow>,
+    expected_starts: &[i64],
+    group_keys: &[String],
+    template: &SliceRow,
+    sli_type: SliType,
+) -> Vec<SliceRow> {
+    if gap_fill_policy(sli_type) == GapFill::Nothing {
+        return observed;
+    }
+
+    use std::collections::HashSet;
+    let present: HashSet<(String, i64)> = observed
+        .iter()
+        .map(|r| (r.group_key.clone(), r.slice_start))
+        .collect();
+
+    let mut out = observed;
+    for group_key in group_keys {
+        for &slice_start in expected_starts {
+            if present.contains(&(group_key.clone(), slice_start)) {
+                continue;
+            }
+            out.push(SliceRow {
+                group_key: group_key.clone(),
+                slice_start,
+                good: 0.0,
+                total: 0.0,
+                ..template.clone()
+            });
+        }
+    }
+    out
+}
+
+/// Whether a recomputed slice should be re-emitted (D55 write-on-change).
+///
+/// Re-emitting unconditionally makes the trailing-K recompute cost ~K physical
+/// rows per logical slice forever — at documented defaults that was ~8 billion
+/// rows over a 90-day window.
+pub fn should_emit(previous: Option<(f64, f64)>, recomputed: (f64, f64)) -> bool {
+    let Some((prev_good, prev_total)) = previous else {
+        return true;
+    };
+    // `same` rather than `!=` so an unchanged NaN compares as unchanged;
+    // `NaN != NaN` would make the slice churn on every pass forever.
+    let same = |a: f64, b: f64| a == b || (a.is_nan() && b.is_nan());
+    !(same(prev_good, recomputed.0) && same(prev_total, recomputed.1))
+}
+
+/// Which lane wrote a batch.
+///
+/// Still meaningful for **scheduling** — bulk backfill runs in its own
+/// concurrency lane so it cannot starve latency-sensitive incremental passes
+/// (D58) — but no longer for visibility. Both lanes write ordinary slices.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Writer {
+    Incremental,
+    Backfill,
+}
+
+/// Why an observation may not be persisted as a slice.
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum ObservationError {
+    /// `NaN` or `±inf`. Poisons every downstream aggregate, comparison and
+    /// serialized status value.
+    NotFinite { field: &'static str, value: f64 },
+    /// Counts and seconds cannot be negative.
+    Negative { field: &'static str, value: f64 },
+    /// More good units than total — the SLI would exceed 100%.
+    GoodExceedsTotal { good: f64, total: f64 },
+}
+
+/// Reject a non-finite or incoherent observation at the **ingest boundary**,
+/// before it can reach a slice row.
+///
+/// Deliberately not `should_emit`'s job: making write-on-change tolerate `NaN`
+/// stops the row churning, but still lets invalid data into the stream, where
+/// it silently corrupts every window aggregate that touches it.
+pub fn validate_observation(good: f64, total: f64) -> Result<(), ObservationError> {
+    for (field, value) in [("good", good), ("total", total)] {
+        if !value.is_finite() {
+            return Err(ObservationError::NotFinite { field, value });
+        }
+        if value < 0.0 {
+            return Err(ObservationError::Negative { field, value });
+        }
+    }
+    if good > total {
+        return Err(ObservationError::GoodExceedsTotal { good, total });
+    }
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn row(group: &str, start: i64, good: f64, total: f64, rev: i64) -> SliceRow {
+        SliceRow {
+            slo_id: "slo1".into(),
+            definition_generation: 1,
+            group_key: group.into(),
+            slice_start: start,
+            good,
+            total,
+            rev,
+        }
+    }
+
+    const WATERMARK: i64 = 10_000;
+
+    // ---- the forward clamp -------------------------------------------------
+
+    #[test]
+    fn rows_before_the_watermark_are_visible() {
+        assert!(is_visible(&row("", 9_700, 1.0, 1.0, 1), WATERMARK, 1));
+    }
+
+    /// The one thing the watermark is for: the currently-filling slice must
+    /// not be published. A bucket that is 10% full reads as 90% less traffic,
+    /// and for a time-slice SLI it can classify bad and then flip good.
+    #[test]
+    fn the_currently_filling_slice_is_not_visible() {
+        assert!(!is_visible(&row("", 10_000, 1.0, 1.0, 1), WATERMARK, 1));
+        assert!(!is_visible(&row("", 10_300, 1.0, 1.0, 1), WATERMARK, 1));
+    }
+
+    #[test]
+    fn rows_from_another_generation_are_invisible() {
+        let mut old = row("", 9_000, 1.0, 1.0, 99);
+        old.definition_generation = 1;
+        assert!(!is_visible(&old, WATERMARK, 2));
+    }
+
+    /// D64, stated as a test so the reversal is not silently undone: a row
+    /// whose batch never committed IS visible. It is a real measurement whose
+    /// delta was not folded into the cache; reconciliation repairs the cache,
+    /// and alerts read the cache rather than slices. Re-adding a backward
+    /// barrier would break this test, which is the intent.
+    #[test]
+    fn an_unaccounted_row_is_visible_because_publication_is_at_least_once() {
+        assert!(
+            is_visible(&row("", 9_000, 1.0, 1.0, 5), WATERMARK, 1),
+            "at-least-once publication: slices are not gated on a commit record"
+        );
+    }
+
+    // ---- dedupe: latest revision wins -------------------------------------
+
+    #[test]
+    fn dedupe_keeps_the_highest_revision() {
+        let out = dedupe_latest_rev(vec![
+            row("a", 100, 5.0, 10.0, 1),
+            row("a", 100, 7.0, 12.0, 2),
+        ]);
+        assert_eq!(out.len(), 1);
+        assert_eq!(out[0].good, 7.0);
+        assert_eq!(out[0].total, 12.0);
+    }
+
+    #[test]
+    fn dedupe_is_order_independent() {
+        let ascending = dedupe_latest_rev(vec![
+            row("a", 100, 5.0, 10.0, 1),
+            row("a", 100, 7.0, 12.0, 2),
+        ]);
+        let descending = dedupe_latest_rev(vec![
+            row("a", 100, 7.0, 12.0, 2),
+            row("a", 100, 5.0, 10.0, 1),
+        ]);
+        assert_eq!(ascending, descending);
+    }
+
+    /// D54, the one that matters: a recomputed time-slice can flip good→bad.
+    /// `MAX(good)` would keep the stale 300 and over-report uptime.
+    #[test]
+    fn dedupe_lets_a_recomputed_slice_flip_from_good_to_bad() {
+        let out = dedupe_latest_rev(vec![
+            row("a", 100, 300.0, 300.0, 1), // good slice
+            row("a", 100, 0.0, 300.0, 2),   // late data flipped it bad
+        ]);
+        assert_eq!(out.len(), 1);
+        assert_eq!(out[0].good, 0.0, "MAX would wrongly keep 300 here");
+    }
+
+    #[test]
+    fn dedupe_keys_on_group_and_slice_independently() {
+        let out = dedupe_latest_rev(vec![
+            row("a", 100, 1.0, 1.0, 1),
+            row("b", 100, 2.0, 2.0, 1),
+            row("a", 200, 3.0, 3.0, 1),
+        ]);
+        assert_eq!(out.len(), 3);
+    }
+
+    #[test]
+    fn dedupe_returns_deterministic_order() {
+        let out = dedupe_latest_rev(vec![
+            row("b", 200, 1.0, 1.0, 1),
+            row("a", 300, 1.0, 1.0, 1),
+            row("a", 100, 1.0, 1.0, 1),
+        ]);
+        let keys: Vec<_> = out
+            .iter()
+            .map(|r| (r.group_key.clone(), r.slice_start))
+            .collect();
+        let mut sorted = keys.clone();
+        sorted.sort();
+        assert_eq!(keys, sorted);
+    }
+
+    #[test]
+    fn dedupe_of_nothing_is_nothing() {
+        assert!(dedupe_latest_rev(vec![]).is_empty());
+    }
+
+    // ---- the composed read path -------------------------------------------
+
+    #[test]
+    fn visible_slices_clamps_then_dedupes() {
+        let rows = vec![
+            row("a", 9_000, 1.0, 2.0, 1),  // superseded
+            row("a", 9_000, 5.0, 9.0, 2),  // the later revision
+            row("a", 10_500, 9.0, 9.0, 1), // at/after the watermark
+        ];
+        let out = visible_slices(rows, WATERMARK, 1);
+        assert_eq!(out.len(), 1, "the unclosed slice is clamped away");
+        assert_eq!(
+            out[0].good, 5.0,
+            "the later revision wins — there is no commit record to gate on (D64)"
+        );
+    }
+
+    /// Order matters: clamping must happen BEFORE dedupe, or a row from
+    /// another generation could win a key on revision alone and then be
+    /// filtered out, leaving the key empty.
+    #[test]
+    fn clamping_happens_before_dedupe_not_after() {
+        let mut foreign = row("a", 9_000, 99.0, 99.0, 9);
+        foreign.definition_generation = 2;
+        let current = row("a", 9_000, 1.0, 1.0, 1);
+        let out = visible_slices(vec![foreign, current], WATERMARK, 1);
+        assert_eq!(out.len(), 1);
+        assert_eq!(
+            out[0].good, 1.0,
+            "a higher revision in the wrong generation must not consume the key"
+        );
+    }
+
+    #[test]
+    fn visible_slices_excludes_other_generations() {
+        let mut old = row("a", 9_000, 1.0, 1.0, 5);
+        old.definition_generation = 1;
+        let mut new = row("a", 9_000, 2.0, 2.0, 1);
+        new.definition_generation = 2;
+        let out = visible_slices(vec![old, new], WATERMARK, 2);
+        assert_eq!(out.len(), 1);
+        assert_eq!(out[0].definition_generation, 2);
+        assert_eq!(
+            out[0].good, 2.0,
+            "a higher rev in an older generation must never win"
+        );
+    }
+
+    // ---- gap-fill policy ---------------------------------------------------
+
+    /// D48: the two types disagree, and the disagreement is the point.
+    #[test]
+    fn count_treats_a_missing_bucket_as_measured_zero_traffic() {
+        assert_eq!(gap_fill_policy(SliType::Count), GapFill::CoveredZero);
+    }
+
+    #[test]
+    fn time_slice_treats_a_missing_bucket_as_unmeasured() {
+        assert_eq!(gap_fill_policy(SliType::TimeSlice), GapFill::Nothing);
+    }
+
+    #[test]
+    fn alert_sli_takes_coverage_from_the_ledger_not_gap_fill() {
+        assert_eq!(gap_fill_policy(SliType::Alert), GapFill::Nothing);
+    }
+
+    // ---- gap-fill behaviour ------------------------------------------------
+
+    #[test]
+    fn count_fills_missing_buckets_with_covered_zeros() {
+        let template = row("", 0, 0.0, 0.0, 1);
+        let out = fill_gaps(
+            vec![row("a", 300, 5.0, 10.0, 1)],
+            &[0, 300, 600],
+            &["a".to_string()],
+            &template,
+            SliType::Count,
+        );
+        assert_eq!(out.len(), 3, "every bucket in the grid is emitted");
+        let filled: Vec<_> = out.iter().filter(|r| r.slice_start != 300).collect();
+        assert!(filled.iter().all(|r| r.good == 0.0 && r.total == 0.0));
+    }
+
+    #[test]
+    fn time_slice_leaves_missing_buckets_absent() {
+        let template = row("", 0, 0.0, 0.0, 1);
+        let out = fill_gaps(
+            vec![row("a", 300, 300.0, 300.0, 1)],
+            &[0, 300, 600],
+            &["a".to_string()],
+            &template,
+            SliType::TimeSlice,
+        );
+        assert_eq!(out.len(), 1, "gaps stay gaps");
+        assert_eq!(out[0].slice_start, 300);
+    }
+
+    #[test]
+    fn gap_fill_covers_every_group_not_just_the_ones_that_reported() {
+        let template = row("", 0, 0.0, 0.0, 1);
+        let out = fill_gaps(
+            vec![row("a", 0, 1.0, 1.0, 1)],
+            &[0],
+            &["a".to_string(), "b".to_string()],
+            &template,
+            SliType::Count,
+        );
+        assert_eq!(out.len(), 2);
+        assert!(out.iter().any(|r| r.group_key == "b" && r.total == 0.0));
+    }
+
+    #[test]
+    fn gap_fill_preserves_observed_values() {
+        let template = row("", 0, 0.0, 0.0, 1);
+        let out = fill_gaps(
+            vec![row("a", 0, 7.0, 9.0, 3)],
+            &[0, 300],
+            &["a".to_string()],
+            &template,
+            SliType::Count,
+        );
+        let observed = out.iter().find(|r| r.slice_start == 0).unwrap();
+        assert_eq!((observed.good, observed.total), (7.0, 9.0));
+    }
+
+    // ---- write-on-change ---------------------------------------------------
+
+    #[test]
+    fn a_first_observation_is_always_emitted() {
+        assert!(should_emit(None, (1.0, 2.0)));
+    }
+
+    #[test]
+    fn an_unchanged_recompute_is_not_re_emitted() {
+        assert!(!should_emit(Some((1.0, 2.0)), (1.0, 2.0)));
+    }
+
+    #[test]
+    fn a_changed_recompute_is_re_emitted() {
+        assert!(should_emit(Some((1.0, 2.0)), (1.0, 3.0)));
+        assert!(should_emit(Some((1.0, 2.0)), (0.0, 2.0)));
+    }
+
+    /// The steady-state claim behind the §6b.4d volume numbers, exercised over
+    /// a realistic trailing-recompute window rather than by repetition: only
+    /// the slice whose value actually moved is re-emitted.
+    #[test]
+    fn a_recompute_pass_emits_only_the_slices_that_changed() {
+        let previous = [(0, (10.0, 10.0)), (300, (9.0, 10.0)), (600, (10.0, 10.0))];
+        let recomputed = [(0, (10.0, 10.0)), (300, (9.0, 11.0)), (600, (10.0, 10.0))];
+        let emitted: Vec<i64> = previous
+            .iter()
+            .zip(recomputed.iter())
+            .filter(|((_, prev), (_, now))| should_emit(Some(*prev), *now))
+            .map(|((start, _), _)| *start)
+            .collect();
+        assert_eq!(emitted, vec![300], "only the slice that gained late data");
+    }
+
+    /// Defence in depth only: `NaN` must be rejected at the ingest boundary
+    /// (see `validate_observation`), but if one ever reaches the trailing
+    /// buffer it must not make the slice churn forever — `NaN != NaN` under a
+    /// naive equality check would defeat write-on-change entirely.
+    #[test]
+    fn a_nan_value_does_not_re_emit_forever() {
+        assert!(
+            !should_emit(Some((f64::NAN, 10.0)), (f64::NAN, 10.0)),
+            "an unchanged NaN must compare as unchanged"
+        );
+    }
+
+    // ---- ingest-boundary validation ---------------------------------------
+
+    #[test]
+    fn a_coherent_observation_is_accepted() {
+        assert_eq!(validate_observation(5.0, 10.0), Ok(()));
+        assert_eq!(
+            validate_observation(0.0, 0.0),
+            Ok(()),
+            "zero traffic is valid"
+        );
+        assert_eq!(validate_observation(10.0, 10.0), Ok(()));
+    }
+
+    #[test]
+    fn non_finite_observations_are_rejected_before_persistence() {
+        for bad in [f64::NAN, f64::INFINITY, f64::NEG_INFINITY] {
+            assert!(
+                matches!(
+                    validate_observation(bad, 10.0),
+                    Err(ObservationError::NotFinite { .. })
+                ),
+                "good={bad} accepted"
+            );
+            assert!(
+                matches!(
+                    validate_observation(1.0, bad),
+                    Err(ObservationError::NotFinite { .. })
+                ),
+                "total={bad} accepted"
+            );
+        }
+    }
+
+    #[test]
+    fn negative_observations_are_rejected() {
+        assert!(matches!(
+            validate_observation(-1.0, 10.0),
+            Err(ObservationError::Negative { .. })
+        ));
+        assert!(matches!(
+            validate_observation(1.0, -10.0),
+            Err(ObservationError::Negative { .. })
+        ));
+    }
+
+    /// More good units than total would put the SLI above 100% and the burn
+    /// rate below zero.
+    #[test]
+    fn more_good_than_total_is_rejected() {
+        assert_eq!(
+            validate_observation(11.0, 10.0),
+            Err(ObservationError::GoodExceedsTotal {
+                good: 11.0,
+                total: 10.0
+            })
+        );
+    }
+}
+
+/// Tests for the config-aware gap-fill policy — the seam `absent_is_bad`
+/// turns on. Written before the function exists.
+#[cfg(test)]
+mod gap_fill_policy_for_tests {
+    use super::*;
+    use crate::meta::{
+        alerts::Operator,
+        slo::{CountSource, QueryLanguage, SliConfig},
+    };
+
+    fn ts(absent_is_bad: bool) -> SliConfig {
+        SliConfig::TimeSlice {
+            stream: "s".into(),
+            stream_type: "logs".into(),
+            query_language: QueryLanguage::Sql,
+            query: "count(*)".into(),
+            scope: None,
+            comparator: Operator::GreaterThanEquals,
+            threshold: 1.0,
+            absent_is_bad,
+        }
+    }
+
+    /// The freshness semantics: a slice the search proved empty is BAD, not a
+    /// gap. Only the POLICY changes — a failed search still writes nothing
+    /// for every type, because gap fill runs only after a successful query.
+    #[test]
+    fn an_absent_is_bad_time_slice_fills_covered_bad() {
+        assert_eq!(gap_fill_policy_for(&ts(true)), GapFill::CoveredBad);
+    }
+
+    /// Off keeps S-8 exactly: absence is a gap, coverage falls, the SLO
+    /// freezes rather than inventing downtime.
+    #[test]
+    fn a_plain_time_slice_still_fills_nothing() {
+        assert_eq!(gap_fill_policy_for(&ts(false)), GapFill::Nothing);
+    }
+
+    #[test]
+    fn count_and_alert_policies_are_unchanged() {
+        let count = SliConfig::Count {
+            source: CountSource::SingleQuery {
+                stream: "s".into(),
+                stream_type: "logs".into(),
+                scope: None,
+                good_expr: "ok".into(),
+            },
+        };
+        assert_eq!(gap_fill_policy_for(&count), GapFill::CoveredZero);
+        assert_eq!(
+            gap_fill_policy_for(&SliConfig::Alert {
+                alert_id: "a".into()
+            }),
+            GapFill::Nothing
+        );
+        // The type-level answer stays for callers that have no config.
+        assert_eq!(gap_fill_policy(SliType::Count), GapFill::CoveredZero);
+        assert_eq!(gap_fill_policy(SliType::TimeSlice), GapFill::Nothing);
+    }
+}

src/config/src/meta/slo/status_view.rs

@@ -0,0 +1,335 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! The read-time view of an SLO's measurement (`alerts_2.md` §6b.4c, D56).
+//!
+//! Nothing here is stored. `slo_status` keeps only **target-free** raw
+//! counts, and every number a user sees is derived from those plus the
+//! *current* target — which is what lets a target edit take effect instantly
+//! instead of invalidating 90 days of measurement.
+//!
+//! The other rule this type enforces is that **unmeasured time never reads as
+//! uptime** (D34). Below the coverage floor the view reports `no_data` and
+//! leaves the derived figures `None`, rather than reporting an SLI computed
+//! from a fraction of the window as though it described all of it.
+
+use serde::{Deserialize, Serialize};
+use utoipa::ToSchema;
+
+use super::math::{burn_rate, error_budget_remaining, sli, time_to_exhaust_secs};
+
+/// What the UI and the API render for one SLO or one group.
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, ToSchema)]
+pub struct SloStatusView {
+    pub group_key: String,
+    /// Fraction of the window actually measured, 0..1.
+    pub coverage: f64,
+    /// True when coverage is below the floor. Every derived figure is `None`
+    /// in that case — the SLO is frozen, not healthy and not breached.
+    pub no_data: bool,
+    /// Percentage, 0..100. `None` when frozen or not yet measured.
+    pub sli: Option<f64>,
+    /// **Percentage** of the error budget still unspent, not a fraction.
+    /// Negative once the budget is overspent, which is meaningful and
+    /// deliberately not clamped: "-80% remaining" is what a user needs to see
+    /// when they have burned 180% of the budget.
+    pub error_budget_remaining: Option<f64>,
+    /// Current burn rate — multiples of the budget-neutral rate.
+    pub burn_rate: Option<f64>,
+    /// Seconds until the budget is exhausted at the current burn. `None` when
+    /// the burn is at or below neutral, because nothing is being exhausted.
+    pub time_to_exhaust_secs: Option<i64>,
+    pub good: f64,
+    pub total: f64,
+    pub covered_slices: i64,
+    pub computed_at: Option<i64>,
+}
+
+impl SloStatusView {
+    /// Derive the view from raw counts and the current target.
+    #[allow(clippy::too_many_arguments)]
+    pub fn derive(
+        group_key: String,
+        good: Option<f64>,
+        total: Option<f64>,
+        covered_slices: Option<i64>,
+        expected_slices: i64,
+        target: f64,
+        window_secs: i64,
+        coverage_floor: f64,
+        computed_at: Option<i64>,
+    ) -> Self {
+        let good = good.unwrap_or(0.0);
+        let total = total.unwrap_or(0.0);
+        let covered = covered_slices.unwrap_or(0);
+        let coverage = if expected_slices > 0 {
+            (covered as f64 / expected_slices as f64).clamp(0.0, 1.0)
+        } else {
+            0.0
+        };
+
+        // Nothing measured yet is NOT the same as measured-and-empty. A brand
+        // new SLO must not render as 0% available.
+        let unmeasured = covered_slices.is_none() || covered == 0;
+        // A covered window with ZERO events has no SLI either — the ratio is
+        // undefined, not 0% (SA-18). `coverage::observe` already treats this
+        // as `Unobserved(ZeroTotal)`; this is the read path agreeing with it.
+        // Without this the view reports `no_data: false` alongside a null SLI,
+        // an inconsistent state the UI renders as "measured" with an em dash.
+        let sli_pct_raw = sli(good, total);
+        let no_data = unmeasured || coverage < coverage_floor || sli_pct_raw.is_none();
+
+        let sli_pct = if no_data { None } else { sli_pct_raw };
+        let (budget, burn, ttl) = match sli_pct {
+            Some(s) => {
+                let remaining = error_budget_remaining(s, target);
+                let b = burn_rate(s, target);
+                (
+                    Some(remaining),
+                    Some(b),
+                    time_to_exhaust_secs(window_secs, b),
+                )
+            }
+            None => (None, None, None),
+        };
+
+        Self {
+            group_key,
+            coverage,
+            no_data,
+            sli: sli_pct,
+            error_budget_remaining: budget,
+            burn_rate: burn,
+            time_to_exhaust_secs: ttl,
+            good,
+            total,
+            covered_slices: covered,
+            computed_at,
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn view(
+        good: Option<f64>,
+        total: Option<f64>,
+        covered: Option<i64>,
+        expected: i64,
+    ) -> SloStatusView {
+        SloStatusView::derive(
+            String::new(),
+            good,
+            total,
+            covered,
+            expected,
+            99.0,
+            30 * 86_400,
+            0.9,
+            Some(100),
+        )
+    }
+
+    #[test]
+    fn a_healthy_slo_derives_its_sli_and_budget() {
+        let v = view(Some(999.0), Some(1000.0), Some(100), 100);
+        assert!(!v.no_data);
+        assert_eq!(v.sli, Some(99.9));
+        // Target 99% allows 1% errors; 0.1% used leaves 90% of the budget.
+        // A PERCENTAGE, matching `math::error_budget_remaining` — not 0.9.
+        let remaining = v.error_budget_remaining.unwrap();
+        assert!((remaining - 90.0).abs() < 1e-9, "got {remaining}");
+    }
+
+    /// The property the whole coverage mechanism exists for (D34).
+    #[test]
+    fn unmeasured_time_never_reads_as_uptime() {
+        // Half the window measured, all of it good.
+        let v = view(Some(50.0), Some(50.0), Some(50), 100);
+        assert!(v.no_data, "50% coverage read as a real measurement");
+        assert_eq!(v.sli, None, "an SLI was reported from half a window");
+        assert_eq!(v.error_budget_remaining, None);
+        assert_eq!(v.burn_rate, None);
+    }
+
+    #[test]
+    fn coverage_exactly_at_the_floor_is_measured() {
+        let v = view(Some(90.0), Some(90.0), Some(90), 100);
+        assert!(!v.no_data, "the floor is inclusive");
+        assert_eq!(v.sli, Some(100.0));
+    }
+
+    #[test]
+    fn one_slice_below_the_floor_freezes() {
+        let v = view(Some(89.0), Some(89.0), Some(89), 100);
+        assert!(v.no_data);
+    }
+
+    /// "Not yet measured" and "measured as zero" are different, and a UI that
+    /// conflates them shows a brand-new SLO as 0% available.
+    #[test]
+    fn a_brand_new_slo_is_no_data_not_zero_percent() {
+        let v = view(None, None, None, 100);
+        assert!(v.no_data);
+        assert_eq!(v.sli, None);
+        assert_eq!(v.coverage, 0.0);
+    }
+
+    #[test]
+    fn a_fully_measured_but_empty_window_is_still_no_data() {
+        // covered = 0 means nothing was observed, even if expected > 0.
+        let v = view(Some(0.0), Some(0.0), Some(0), 100);
+        assert!(v.no_data);
+    }
+
+    /// A window with plenty of COVERAGE but no events at all. The SLI is
+    /// undefined, not 0% (SA-18), so the view must say no_data rather than
+    /// report "measured" with a null SLI — an inconsistent pair the UI would
+    /// render as a measured SLO showing an em dash.
+    ///
+    /// Found by end-to-end testing: gap-filled zero-traffic slices produce
+    /// exactly this shape, and every unit test had fed it non-zero totals.
+    #[test]
+    fn a_covered_window_with_zero_events_is_no_data() {
+        let v = view(Some(0.0), Some(0.0), Some(95), 100);
+        assert!(v.coverage > 0.9, "the window IS covered");
+        assert!(
+            v.no_data,
+            "covered but empty reported as measured — sli would be null"
+        );
+        assert_eq!(v.sli, None);
+        assert_eq!(v.error_budget_remaining, None);
+        assert_eq!(v.burn_rate, None);
+    }
+
+    /// The pair must never disagree: a null SLI and `no_data: false` together
+    /// is the inconsistency this guards.
+    #[test]
+    fn no_data_and_a_null_sli_always_agree() {
+        for (good, total, covered) in [
+            (Some(0.0), Some(0.0), Some(95)),
+            (Some(99.0), Some(100.0), Some(95)),
+            (None, None, None),
+            (Some(0.0), Some(100.0), Some(95)),
+            (Some(1.0), Some(1.0), Some(5)),
+        ] {
+            let v = view(good, total, covered, 100);
+            assert_eq!(
+                v.sli.is_none(),
+                v.no_data,
+                "sli={:?} disagrees with no_data={} for {good:?}/{total:?}/{covered:?}",
+                v.sli,
+                v.no_data
+            );
+        }
+    }
+
+    /// D56: the target is applied at READ time, so the same stored counts
+    /// yield different budgets under different targets — with no rebuild.
+    #[test]
+    fn the_target_is_applied_at_read_time() {
+        let counts = (Some(999.0), Some(1000.0), Some(100));
+        let lenient = SloStatusView::derive(
+            String::new(),
+            counts.0,
+            counts.1,
+            counts.2,
+            100,
+            99.0,
+            30 * 86_400,
+            0.9,
+            None,
+        );
+        let strict = SloStatusView::derive(
+            String::new(),
+            counts.0,
+            counts.1,
+            counts.2,
+            100,
+            99.95,
+            30 * 86_400,
+            0.9,
+            None,
+        );
+        assert_eq!(lenient.sli, strict.sli, "the SLI is target-free");
+        assert_ne!(
+            lenient.error_budget_remaining, strict.error_budget_remaining,
+            "the budget must move with the target"
+        );
+        assert!(
+            strict.error_budget_remaining.unwrap() < 0.0,
+            "99.9% against a 99.95% target has overspent its budget"
+        );
+    }
+
+    /// Negative remaining budget is meaningful and deliberately not clamped:
+    /// "180% consumed" is what a user needs to see.
+    #[test]
+    fn an_overspent_budget_reports_a_negative_remainder() {
+        let v = SloStatusView::derive(
+            String::new(),
+            Some(980.0),
+            Some(1000.0),
+            Some(100),
+            100,
+            99.0,
+            30 * 86_400,
+            0.9,
+            None,
+        );
+        let remaining = v.error_budget_remaining.unwrap();
+        assert!(remaining < 0.0, "got {remaining}");
+    }
+
+    #[test]
+    fn a_burn_at_or_below_neutral_never_exhausts() {
+        // Exactly on target: burn = 1.0, so the budget lasts exactly the
+        // window and nothing is being exhausted early.
+        let v = SloStatusView::derive(
+            String::new(),
+            Some(990.0),
+            Some(1000.0),
+            Some(100),
+            100,
+            99.0,
+            30 * 86_400,
+            0.9,
+            None,
+        );
+        assert_eq!(v.burn_rate, Some(1.0));
+
+        // No errors at all: burn 0, nothing to exhaust.
+        let perfect = view(Some(1000.0), Some(1000.0), Some(100), 100);
+        assert_eq!(perfect.burn_rate, Some(0.0));
+        assert_eq!(perfect.time_to_exhaust_secs, None);
+    }
+
+    #[test]
+    fn coverage_is_clamped_to_one() {
+        // More slices than expected (a late-data re-emission miscount) must
+        // not report 130% coverage.
+        let v = view(Some(1.0), Some(1.0), Some(130), 100);
+        assert_eq!(v.coverage, 1.0);
+    }
+
+    #[test]
+    fn a_zero_expected_window_does_not_divide_by_zero() {
+        let v = view(Some(1.0), Some(1.0), Some(0), 0);
+        assert_eq!(v.coverage, 0.0);
+        assert!(v.no_data);
+    }
+}

src/config/src/meta/slo/stream.rs

@@ -0,0 +1,160 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! The `slo_slices` reserved stream (`alerts_2.md` §6b.8, D32).
+//!
+//! Slices are a **stream**, not a meta-store table, for the reason the volume
+//! makes obvious: 90 days × 5-minute slices × groups × SLOs is timeseries
+//! data, and the meta store is SQLite in local deployments. `triggers` is the
+//! precedent — reserved name, schema by reflection over the Rust struct,
+//! written by the job that produces it.
+//!
+//! Every row carries `definition_generation`. Readers filter on it, which is
+//! what makes a generation bump a clean break rather than a migration: the old
+//! epoch's slices simply stop being visible and age out with retention.
+
+use serde::{Deserialize, Serialize};
+
+/// The reserved stream name. Registered in
+/// [`crate::meta::self_reporting::usage::RESERVED_INTERNAL_STREAMS`], which is
+/// what stops a user creating, ingesting into, or deleting it.
+pub const SLO_SLICES_STREAM: &str = "slo_slices";
+
+/// One measured slice.
+///
+/// `good`/`total` are stored raw rather than as a ratio: the running aggregate
+/// sums them, and summing pre-divided ratios would weight a slice with 3
+/// events the same as one with 30,000.
+#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub struct SloSliceRow {
+    /// Ingest timestamp — when the slice was *written*, not what it measures.
+    /// `slice_start` is the measurement time, and the two differ by the
+    /// evaluation delay and by however late the data arrived.
+    pub _timestamp: i64,
+    pub org: String,
+    pub slo_id: String,
+    /// The epoch this measurement belongs to. Readers filter on it (D59).
+    pub definition_generation: i32,
+    /// `""` is the ungrouped / rollup series (S-9).
+    pub group_key: String,
+    /// Display labels for the group, denormalized so a reader does not need
+    /// the definition to render a chart.
+    pub group_labels: String,
+    /// Aligned to the slice grid. THE measurement time.
+    pub slice_start: i64,
+    pub good: f64,
+    pub total: f64,
+    /// Monotonic per `(slo_id, generation, group_key, slice_start)`. Late data
+    /// and recomputes produce a higher revision, and readers keep the highest
+    /// (D54).
+    ///
+    /// This is for **dedupe**, not publication ordering — there is no
+    /// publication protocol to order (D64).
+    pub rev: i64,
+}
+
+impl SloSliceRow {
+    /// A fully-populated sample used to infer the Arrow schema, so every field
+    /// exists from the first write rather than appearing as data happens to
+    /// contain it. Mirrors `TriggerData::init_for_reflection`.
+    ///
+    /// Every field must be non-null and of its real type here: a field left
+    /// empty would be inferred as `Utf8` and then conflict with the first real
+    /// numeric write.
+    pub fn init_for_reflection() -> Self {
+        Self {
+            _timestamp: 1,
+            org: "org".to_string(),
+            slo_id: "slo".to_string(),
+            definition_generation: 1,
+            group_key: "group".to_string(),
+            group_labels: "label".to_string(),
+            slice_start: 1,
+            good: 1.0,
+            total: 1.0,
+            rev: 1,
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn the_reflection_sample_populates_every_field() {
+        // A field left at its default would be inferred with the wrong Arrow
+        // type and then conflict with the first real write.
+        let v = serde_json::to_value(SloSliceRow::init_for_reflection()).unwrap();
+        let obj = v.as_object().unwrap();
+        assert_eq!(obj.len(), 10, "a field was added without a sample value");
+        for (k, val) in obj {
+            assert!(!val.is_null(), "{k} is null in the reflection sample");
+            if let Some(s) = val.as_str() {
+                assert!(!s.is_empty(), "{k} is empty in the reflection sample");
+            }
+        }
+    }
+
+    /// `good` and `total` must infer as floats. If the sample used whole
+    /// numbers they could infer as integers, and the first fractional write
+    /// would then conflict with the stored schema.
+    #[test]
+    fn the_numeric_fields_are_floats_in_the_sample() {
+        let v = serde_json::to_value(SloSliceRow::init_for_reflection()).unwrap();
+        assert!(v["good"].is_f64(), "good must infer as a float");
+        assert!(v["total"].is_f64(), "total must infer as a float");
+    }
+
+    #[test]
+    fn a_slice_row_round_trips() {
+        let row = SloSliceRow {
+            _timestamp: 1_700_000_000_000_000,
+            org: "acme".into(),
+            slo_id: "slo1".into(),
+            definition_generation: 3,
+            group_key: "region=eu".into(),
+            group_labels: "region: eu".into(),
+            slice_start: 1_700_000_000,
+            good: 98.0,
+            total: 100.0,
+            rev: 2,
+        };
+        let json = serde_json::to_string(&row).unwrap();
+        assert_eq!(serde_json::from_str::<SloSliceRow>(&json).unwrap(), row);
+    }
+
+    /// The stored field names are a wire contract with the reader SQL. A
+    /// rename here silently breaks every query in §6b.4.
+    #[test]
+    fn the_field_names_are_pinned() {
+        let v = serde_json::to_value(SloSliceRow::init_for_reflection()).unwrap();
+        for field in [
+            "_timestamp",
+            "org",
+            "slo_id",
+            "definition_generation",
+            "group_key",
+            "group_labels",
+            "slice_start",
+            "good",
+            "total",
+            "rev",
+        ] {
+            assert!(v.get(field).is_some(), "field `{field}` was renamed");
+        }
+    }
+}

src/config/src/meta/slo/window.rs

@@ -0,0 +1,582 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Slice and window arithmetic — `alerts_2.md` §6b.4a.
+//!
+//! Two rules carry the whole ingest path:
+//!
+//! 1. **The range is `[start, end)` and `end` is the last *completed* slice.** A query with only
+//!    `_timestamp >= watermark` publishes the currently-open slice: a bucket that is 10% full reads
+//!    as 90% less traffic, and for a time-slice SLI a half-filled slice can classify bad and then
+//!    flip good on the next pass.
+//! 2. **Everything is aligned in UTC and computed in absolute seconds.** "30 days" is 30 × 86,400
+//!    s, not a calendar month — which is exactly why calendar windows are a v1 non-goal rather than
+//!    a rounding detail.
+
+/// Align a timestamp down to the start of its slice.
+pub fn align_down(ts_secs: i64, slice_interval_secs: i64) -> i64 {
+    if slice_interval_secs <= 0 {
+        return ts_secs;
+    }
+    // `div_euclid`, not `/`: integer division truncates toward zero, so a
+    // negative timestamp would align *up* and land inside the wrong bucket.
+    ts_secs.div_euclid(slice_interval_secs) * slice_interval_secs
+}
+
+/// Align a timestamp **up** to the next slice boundary, leaving an exact
+/// boundary alone.
+fn align_up(ts_secs: i64, slice_interval_secs: i64) -> i64 {
+    if slice_interval_secs <= 0 {
+        return ts_secs;
+    }
+    let down = align_down(ts_secs, slice_interval_secs);
+    if down == ts_secs {
+        down
+    } else {
+        down + slice_interval_secs
+    }
+}
+
+/// A closed ingest range: `[start, end)`, both aligned to the slice grid.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct IngestRange {
+    /// Inclusive lower bound, aligned.
+    pub start: i64,
+    /// **Exclusive** upper bound, aligned — the start of the first slice that
+    /// is *not* yet complete.
+    pub end: i64,
+}
+
+impl IngestRange {
+    /// Number of slices the range covers.
+    pub fn slice_count(&self, slice_interval_secs: i64) -> i64 {
+        if slice_interval_secs <= 0 {
+            return 0;
+        }
+        (self.end - self.start).max(0) / slice_interval_secs
+    }
+
+    /// Every aligned `slice_start` in the range, ascending.
+    pub fn slice_starts(&self, slice_interval_secs: i64) -> Vec<i64> {
+        if slice_interval_secs <= 0 {
+            return Vec::new();
+        }
+        let mut out = Vec::new();
+        let mut s = self.start;
+        while s < self.end {
+            out.push(s);
+            s += slice_interval_secs;
+        }
+        out
+    }
+}
+
+/// Inputs to the range computation, named so the call sites cannot transpose
+/// them.
+#[derive(Debug, Clone, Copy)]
+pub struct IngestRangeParams {
+    /// Wall clock at the start of the pass.
+    pub now_secs: i64,
+    /// The current published watermark; `None` on the very first pass of a
+    /// generation.
+    pub watermark_end: Option<i64>,
+    pub slice_interval_secs: i64,
+    /// Settle time for normal ingestion lag before a slice is closed
+    /// (`ZO_SLO_INGEST_DELAY_SECS`).
+    pub ingest_delay_secs: i64,
+    /// How many trailing slices to recompute for late data
+    /// (`ZO_SLO_RECOMPUTE_SLICES`).
+    pub recompute_slices: i64,
+    /// Floor for the first pass of a generation — never scan before this.
+    pub generation_reset_time: i64,
+}
+
+/// Compute the pass's `[start, end)`.
+///
+/// ```text
+/// end   = align_down(now - ingest_delay)          the last COMPLETE slice
+/// start = min(watermark_end, align_down(now) - K × slice)
+/// start = max(start, generation_reset_time)
+/// ```
+///
+/// `None` — the pass does nothing — **iff no new slice has closed**, i.e.
+/// `end <= watermark_end` (or the degenerate `end <= start`).
+///
+/// That condition is the whole reason this returns an `Option`, and it needs
+/// stating because the obvious alternative is wrong in both directions. With
+/// `K > 0` the range *always* reaches back K slices, so `start < end` holds
+/// unconditionally and a `end <= start` test alone could never fire — the job
+/// would re-query the same trailing slices on every scheduler tick, at up to
+/// `slice_interval / cadence` times the necessary query load. Gating on "did a
+/// slice close" costs nothing in late-data coverage: the trailing K slices are
+/// still recomputed at the next slice boundary, which is the soonest any of
+/// their values could matter.
+///
+/// A watermark ahead of `now` (clock skew) therefore also yields `None`, which
+/// is the safe response — never a backwards range.
+pub fn ingest_range(params: IngestRangeParams) -> Option<IngestRange> {
+    let slice = params.slice_interval_secs;
+    if slice <= 0 {
+        return None;
+    }
+
+    // The last COMPLETE slice: never the one still filling.
+    let end = align_down(params.now_secs - params.ingest_delay_secs, slice);
+
+    // Nothing new has closed — do not re-query the same trailing slices on
+    // every tick. A watermark ahead of `now` (clock skew) lands here too,
+    // which is the safe response.
+    if let Some(watermark) = params.watermark_end
+        && end <= watermark
+    {
+        return None;
+    }
+
+    // Reach back K slices for late data. On the first pass of a generation
+    // there is no watermark to reach back from.
+    let reach_back = align_down(params.now_secs, slice) - params.recompute_slices * slice;
+    let start = match params.watermark_end {
+        Some(watermark) => watermark.min(reach_back),
+        None => params.generation_reset_time,
+    };
+
+    // Never scan before the generation began, and keep `start` on the grid:
+    // an unaligned start would make the histogram emit a `slice_start` below
+    // it, i.e. a bucket that partly predates the generation.
+    let floor = align_up(params.generation_reset_time, slice);
+    let start = align_down(start, slice).max(floor);
+
+    if end <= start {
+        return None;
+    }
+    Some(IngestRange { start, end })
+}
+
+/// How many slices a read window *should* contain — the denominator of
+/// coverage. Derived from the aligned grid, never from what a query returned.
+pub fn expected_slices(from_secs: i64, to_secs: i64, slice_interval_secs: i64) -> i64 {
+    if slice_interval_secs <= 0 {
+        return 0;
+    }
+    (to_secs - from_secs).max(0) / slice_interval_secs
+}
+
+/// The `[from, to)` a read window covers, anchored at the watermark rather
+/// than the wall clock (SA-14).
+pub fn read_window(watermark_end: i64, window_secs: i64) -> (i64, i64) {
+    (watermark_end - window_secs, watermark_end)
+}
+
+/// Whether a watermark is too old to trust — `now > watermark + K × slice`
+/// (SA-14). A stale watermark means the evaluation is unobserved, not that the
+/// SLO recovered.
+pub fn watermark_is_stale(
+    now_secs: i64,
+    watermark_end: i64,
+    slice_interval_secs: i64,
+    stale_k: i64,
+) -> bool {
+    now_secs > watermark_end + stale_k * slice_interval_secs
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::meta::slo::{SLICE_60_SECS, SLICE_300_SECS};
+
+    const MIN: i64 = 60;
+    const FIVE_MIN: i64 = 300;
+
+    fn params(now: i64, watermark: Option<i64>, slice: i64) -> IngestRangeParams {
+        IngestRangeParams {
+            now_secs: now,
+            watermark_end: watermark,
+            slice_interval_secs: slice,
+            ingest_delay_secs: 60,
+            recompute_slices: 3,
+            generation_reset_time: 0,
+        }
+    }
+
+    // ---- alignment ---------------------------------------------------------
+
+    #[test]
+    fn align_down_snaps_to_the_slice_grid() {
+        assert_eq!(align_down(1_000_000, FIVE_MIN), 999_900);
+        assert_eq!(align_down(1_000_000, MIN), 1_000_000 - 40);
+    }
+
+    #[test]
+    fn align_down_is_idempotent() {
+        let aligned = align_down(1_234_567, FIVE_MIN);
+        assert_eq!(align_down(aligned, FIVE_MIN), aligned);
+    }
+
+    #[test]
+    fn align_down_leaves_an_exact_boundary_alone() {
+        assert_eq!(align_down(3600, FIVE_MIN), 3600);
+        assert_eq!(align_down(0, FIVE_MIN), 0);
+    }
+
+    // ---- the exclusive upper bound ----------------------------------------
+
+    /// The defect this rule exists to prevent: publishing the slice that is
+    /// still filling.
+    #[test]
+    fn range_never_includes_the_currently_open_slice() {
+        // now sits 100s into the slice that starts at 3600.
+        let now = 3600 + 100;
+        let r = ingest_range(IngestRangeParams {
+            ingest_delay_secs: 0,
+            ..params(now, Some(3000), FIVE_MIN)
+        })
+        .unwrap();
+        assert_eq!(r.end, 3600, "end must be the START of the open slice");
+        assert!(r.end <= now);
+    }
+
+    #[test]
+    fn range_end_is_aligned() {
+        let r = ingest_range(params(1_234_567, Some(1_200_000), FIVE_MIN)).unwrap();
+        assert_eq!(align_down(r.end, FIVE_MIN), r.end);
+    }
+
+    #[test]
+    fn ingest_delay_pushes_the_end_back_by_whole_slices() {
+        let now = 7200; // exactly on a slice boundary
+        let no_delay = ingest_range(IngestRangeParams {
+            ingest_delay_secs: 0,
+            ..params(now, Some(3600), FIVE_MIN)
+        })
+        .unwrap();
+        let delayed = ingest_range(IngestRangeParams {
+            ingest_delay_secs: 60,
+            ..params(now, Some(3600), FIVE_MIN)
+        })
+        .unwrap();
+        assert_eq!(no_delay.end, 7200);
+        assert_eq!(delayed.end, 6900, "a 60s delay drops the last slice");
+    }
+
+    /// The pass is a no-op while no new slice has closed — otherwise a job
+    /// running faster than its slice interval re-queries the same trailing
+    /// slices every tick.
+    #[test]
+    fn range_is_none_when_no_new_slice_has_closed() {
+        // end = align(7300 - 60) = 7200, which the watermark already covers.
+        assert_eq!(ingest_range(params(7300, Some(7200), FIVE_MIN)), None);
+    }
+
+    #[test]
+    fn range_is_some_again_as_soon_as_one_slice_closes() {
+        // Same watermark, one slice later: end = align(7600-60) = 7500.
+        let r = ingest_range(params(7600, Some(7200), FIVE_MIN)).unwrap();
+        assert_eq!(r.end, 7500);
+    }
+
+    #[test]
+    fn range_is_none_when_the_watermark_is_ahead_of_now() {
+        // Clock skew must never produce a backwards range.
+        assert_eq!(ingest_range(params(3600, Some(99_999), FIVE_MIN)), None);
+    }
+
+    // ---- the trailing recompute window ------------------------------------
+
+    #[test]
+    fn range_reaches_back_k_slices_for_late_data() {
+        let r = ingest_range(params(10_000, Some(9000), FIVE_MIN)).unwrap();
+        // now - delay = 9940 -> end = 9900; align(now) - 3 slices = 9000,
+        // and the watermark is also 9000, so start is 9000.
+        assert_eq!(r.end, 9900);
+        assert_eq!(r.start, 9000, "3 slices back from align(now)");
+    }
+
+    /// The recompute window must actually re-cover already-published slices,
+    /// otherwise late data is never picked up.
+    #[test]
+    fn the_recompute_window_reaches_behind_the_watermark() {
+        // A watermark well ahead of align(now) - K*slice: start is pulled back.
+        let r = ingest_range(params(10_000, Some(9_800), FIVE_MIN)).unwrap();
+        assert!(
+            r.start < 9_800,
+            "start {} must reach behind the watermark for late data",
+            r.start
+        );
+        assert_eq!(r.start, 9_000);
+    }
+
+    #[test]
+    fn recompute_never_reaches_past_the_generation_reset() {
+        let r = ingest_range(IngestRangeParams {
+            generation_reset_time: 9600,
+            ..params(10_000, Some(9000), FIVE_MIN)
+        })
+        .unwrap();
+        assert!(
+            r.start >= 9600,
+            "must not scan before the generation started: {}",
+            r.start
+        );
+    }
+
+    /// An unaligned reset would put `start` inside a bucket, so the histogram
+    /// emits a `slice_start` BELOW `start` — which then falls on the backfill
+    /// side of `reset_time` and is judged against the wrong committed mark.
+    #[test]
+    fn an_unaligned_generation_reset_still_yields_an_aligned_start() {
+        let r = ingest_range(IngestRangeParams {
+            generation_reset_time: 9601,
+            ..params(11_000, Some(9_000), FIVE_MIN)
+        })
+        .unwrap();
+        assert_eq!(
+            align_down(r.start, FIVE_MIN),
+            r.start,
+            "start {} is not on the slice grid",
+            r.start
+        );
+        assert!(
+            r.start >= 9601,
+            "start {} would scan data from before the generation began",
+            r.start
+        );
+    }
+
+    #[test]
+    fn an_unaligned_reset_never_emits_a_slice_owned_by_the_other_writer() {
+        let reset = 9601;
+        let r = ingest_range(IngestRangeParams {
+            generation_reset_time: reset,
+            ..params(11_000, Some(9_000), FIVE_MIN)
+        })
+        .unwrap();
+        for s in r.slice_starts(FIVE_MIN) {
+            assert!(
+                s >= reset,
+                "slice_start {s} is below reset_time {reset}, so the backfill \
+                 writer's mark would judge an incremental row"
+            );
+        }
+    }
+
+    #[test]
+    fn the_first_pass_of_a_generation_starts_at_the_reset_time() {
+        let r = ingest_range(IngestRangeParams {
+            watermark_end: None,
+            generation_reset_time: 9000,
+            ..params(10_000, None, FIVE_MIN)
+        })
+        .unwrap();
+        assert_eq!(r.start, 9000);
+    }
+
+    #[test]
+    fn zero_recompute_starts_exactly_at_the_watermark() {
+        let r = ingest_range(IngestRangeParams {
+            recompute_slices: 0,
+            ..params(10_000, Some(9000), FIVE_MIN)
+        })
+        .unwrap();
+        assert_eq!(r.start, 9000);
+    }
+
+    // ---- slice enumeration -------------------------------------------------
+
+    #[test]
+    fn slice_count_is_the_half_open_length() {
+        let r = IngestRange {
+            start: 0,
+            end: 1500,
+        };
+        assert_eq!(r.slice_count(FIVE_MIN), 5);
+    }
+
+    #[test]
+    fn slice_starts_are_ascending_aligned_and_exclude_the_end() {
+        let r = IngestRange {
+            start: 600,
+            end: 1800,
+        };
+        let starts = r.slice_starts(FIVE_MIN);
+        assert_eq!(starts, vec![600, 900, 1200, 1500]);
+        assert!(!starts.contains(&1800), "end is exclusive");
+    }
+
+    #[test]
+    fn an_empty_range_enumerates_nothing() {
+        let r = IngestRange {
+            start: 900,
+            end: 900,
+        };
+        assert_eq!(r.slice_count(FIVE_MIN), 0);
+        assert!(r.slice_starts(FIVE_MIN).is_empty());
+    }
+
+    // ---- agreement with the engine's bucketing -----------------------------
+
+    /// `histogram()` is rewritten to `date_bin(interval, source, origin)` with
+    /// **origin = 2001-01-01T00:00:00Z**, not the Unix epoch
+    /// (`rewrite_histogram.rs:250`). `align_down` aligns to the epoch. The two
+    /// therefore agree only when the origin is itself a whole number of
+    /// intervals from the epoch.
+    ///
+    /// It is — for the intervals we allow. 2001-01-01 is 978,307,200 s after
+    /// the epoch, which 60 and 300 both divide exactly. But that is a
+    /// coincidence of the constant, not a property of the design: a 7-minute
+    /// interval (420 s) leaves a remainder of 360 and the two bucketings drift
+    /// apart by a minute, which would silently misalign every slice against
+    /// the coverage grid.
+    ///
+    /// So this is pinned rather than assumed. If a new slice interval is ever
+    /// added, this test is what says whether the engine agrees with us.
+    #[test]
+    fn every_legal_slice_interval_agrees_with_the_histogram_origin() {
+        /// 2001-01-01T00:00:00Z, from `rewrite_histogram.rs`.
+        const DATE_BIN_ORIGIN_SECS: i64 = 978_307_200;
+
+        for slice in [SLICE_60_SECS, SLICE_300_SECS] {
+            assert_eq!(
+                DATE_BIN_ORIGIN_SECS % slice,
+                0,
+                "slice interval {slice}s does not divide the date_bin origin, so \
+                 histogram() buckets would not line up with align_down"
+            );
+        }
+    }
+
+    /// The same statement as a behavioural check: for a legal interval, our
+    /// bucket and the engine's are the same number.
+    #[test]
+    fn align_down_matches_date_bin_for_legal_intervals() {
+        const ORIGIN: i64 = 978_307_200;
+        let date_bin = |ts: i64, slice: i64| ORIGIN + (ts - ORIGIN).div_euclid(slice) * slice;
+
+        for slice in [SLICE_60_SECS, SLICE_300_SECS] {
+            for ts in [0, 1_000_000_000, 1_753_000_000, 2_000_000_123] {
+                assert_eq!(
+                    align_down(ts, slice),
+                    date_bin(ts, slice),
+                    "align_down and date_bin disagree at ts={ts}, slice={slice}"
+                );
+            }
+        }
+    }
+
+    /// The negative case, so the test above cannot pass vacuously: an interval
+    /// that does NOT divide the origin really does drift.
+    #[test]
+    fn an_interval_that_does_not_divide_the_origin_would_drift() {
+        const ORIGIN: i64 = 978_307_200;
+        let slice = 420; // 7 minutes — not a legal SLO slice, deliberately
+        let date_bin = ORIGIN + (1_000_000_000 - ORIGIN).div_euclid(slice) * slice;
+        assert_ne!(
+            align_down(1_000_000_000, slice),
+            date_bin,
+            "if these ever agree, the guard above has stopped meaning anything"
+        );
+    }
+
+    // ---- expected slices (the coverage denominator) ------------------------
+
+    #[test]
+    fn expected_slices_counts_the_grid_not_the_data() {
+        assert_eq!(expected_slices(0, 3600, FIVE_MIN), 12);
+        assert_eq!(expected_slices(0, 3600, MIN), 60);
+    }
+
+    #[test]
+    fn a_thirty_day_window_has_the_documented_slice_count() {
+        // The §6b.4d volume table: 8,640 slices per 30 days at 5-min.
+        assert_eq!(expected_slices(0, 30 * 86_400, FIVE_MIN), 8_640);
+        assert_eq!(expected_slices(0, 30 * 86_400, MIN), 43_200);
+    }
+
+    #[test]
+    fn a_ninety_day_window_at_five_minutes_is_the_documented_row_count() {
+        assert_eq!(expected_slices(0, 90 * 86_400, FIVE_MIN), 25_920);
+    }
+
+    #[test]
+    fn expected_slices_of_an_empty_window_is_zero() {
+        assert_eq!(expected_slices(1000, 1000, FIVE_MIN), 0);
+    }
+
+    /// These are `pub` entry points, so a zero interval must be defined rather
+    /// than a divide-by-zero panic — validate_slo rejects it upstream, but the
+    /// functions must not be a landmine for a caller that skips validation.
+    #[test]
+    fn a_zero_slice_interval_does_not_panic() {
+        assert_eq!(align_down(1234, 0), 1234);
+        assert_eq!(expected_slices(0, 3600, 0), 0);
+        let r = IngestRange { start: 0, end: 900 };
+        assert_eq!(r.slice_count(0), 0);
+        assert!(r.slice_starts(0).is_empty());
+    }
+
+    #[test]
+    fn a_negative_slice_interval_does_not_panic() {
+        assert_eq!(align_down(1234, -60), 1234);
+        assert_eq!(expected_slices(0, 3600, -60), 0);
+    }
+
+    #[test]
+    fn expected_slices_never_goes_negative() {
+        assert_eq!(expected_slices(2000, 1000, FIVE_MIN), 0);
+    }
+
+    // ---- read windows are anchored at the watermark ------------------------
+
+    /// SA-14: an alert that anchors at `now()` reads a window missing its most
+    /// recent slice — systematically optimistic, and worst during an incident.
+    #[test]
+    fn read_window_ends_at_the_watermark_not_the_clock() {
+        let (from, to) = read_window(9_000, 3600);
+        assert_eq!(to, 9_000);
+        assert_eq!(from, 9_000 - 3600);
+    }
+
+    #[test]
+    fn read_window_length_is_exactly_the_window() {
+        let (from, to) = read_window(1_000_000, 30 * 86_400);
+        assert_eq!(to - from, 30 * 86_400);
+    }
+
+    // ---- watermark staleness ----------------------------------------------
+
+    #[test]
+    fn a_fresh_watermark_is_not_stale() {
+        assert!(!watermark_is_stale(10_000, 9_900, FIVE_MIN, 3));
+    }
+
+    #[test]
+    fn a_watermark_older_than_k_slices_is_stale() {
+        // K=3 at 5-min slices = 900s of tolerance.
+        assert!(!watermark_is_stale(10_000, 9_150, FIVE_MIN, 3));
+        assert!(watermark_is_stale(10_000, 9_000, FIVE_MIN, 3));
+    }
+
+    #[test]
+    fn staleness_scales_with_the_slice_interval() {
+        // The same absolute gap is stale at 1-min slices but fine at 5-min.
+        let gap_secs = 500;
+        assert!(watermark_is_stale(10_000, 10_000 - gap_secs, MIN, 3));
+        assert!(!watermark_is_stale(10_000, 10_000 - gap_secs, FIVE_MIN, 3));
+    }
+
+    #[test]
+    fn a_future_watermark_is_never_stale() {
+        assert!(!watermark_is_stale(10_000, 20_000, FIVE_MIN, 3));
+    }
+}

src/config/src/meta/traces.rs

@@ -74,7 +74,12 @@ pub mod session {
             .collect()
     }
 
-    pub fn trace_ids_sql(stream: &str, session_columns: &[String], session_id: &str) -> String {
+    pub fn trace_ids_sql(
+        stream: &str,
+        session_columns: &[String],
+        session_id: &str,
+        ingest_cutoff_us: Option<i64>,
+    ) -> String {
         let escaped_session_id = escape_sql_string(session_id);
         let predicate = session_columns
             .iter()
@@ -84,14 +89,27 @@ pub mod session {
         format!(
             "SELECT trace_id, min({}) as zo_sql_timestamp \
              FROM {} \
-             WHERE ({predicate}) \
+             WHERE ({predicate}){} \
              GROUP BY trace_id \
              ORDER BY zo_sql_timestamp DESC, trace_id ASC",
             quote_identifier(TIMESTAMP_COL_NAME),
             quote_identifier(stream),
+            ingest_cutoff_predicate(ingest_cutoff_us),
         )
     }
 
+    /// An ingest-time upper bound appended to evidence queries: rows ingested
+    /// at or after the cutoff are excluded, so hydration cannot mix evidence
+    /// that arrived after an evaluation run's frozen boundary into that run.
+    fn ingest_cutoff_predicate(ingest_cutoff_us: Option<i64>) -> String {
+        ingest_cutoff_us.map_or_else(String::new, |cutoff_us| {
+            format!(
+                " AND {} < {cutoff_us}",
+                quote_identifier(crate::O2_INGEST_TS_COL_NAME)
+            )
+        })
+    }
+
     pub fn trace_ids_from_hits(hits: &[Value]) -> Vec<String> {
         let mut seen = HashSet::with_capacity(hits.len());
         let mut trace_ids = Vec::with_capacity(hits.len());
@@ -116,11 +134,16 @@ pub mod session {
         format!("{} IN ({values})", quote_identifier("trace_id"))
     }
 
-    pub fn span_rows_sql(stream: &str, trace_ids: &[String]) -> String {
+    pub fn span_rows_sql(
+        stream: &str,
+        trace_ids: &[String],
+        ingest_cutoff_us: Option<i64>,
+    ) -> String {
         format!(
-            "SELECT * FROM {} WHERE {} ORDER BY {} ASC, _o2_ingest_ts ASC, trace_id ASC, span_id ASC",
+            "SELECT * FROM {} WHERE {}{} ORDER BY {} ASC, _o2_ingest_ts ASC, trace_id ASC, span_id ASC",
             quote_identifier(stream),
             trace_id_predicate(trace_ids),
+            ingest_cutoff_predicate(ingest_cutoff_us),
             quote_identifier(TIMESTAMP_COL_NAME),
         )
     }
@@ -165,17 +188,37 @@ pub mod session {
                     "llm_session_id".to_string(),
                 ],
                 "session-'1",
+                None,
             );
 
             assert!(sql.contains("\"gen_ai_conversation_id\" = 'session-''1'"));
+            assert!(!sql.contains("_o2_ingest_ts"));
             assert!(sql.contains("OR \"llm_session_id\" = 'session-''1'"));
             assert!(sql.contains("GROUP BY trace_id"));
             assert!(sql.contains("ORDER BY zo_sql_timestamp DESC, trace_id ASC"));
         }
 
+        #[test]
+        fn evidence_queries_apply_the_frozen_ingest_cutoff() {
+            let sql = trace_ids_sql(
+                "traces",
+                &["llm_session_id".to_string()],
+                "session-1",
+                Some(1_000_000),
+            );
+            assert!(sql.contains("AND \"_o2_ingest_ts\" < 1000000 GROUP BY"));
+
+            let sql = span_rows_sql("traces", &["abc-123".to_string()], Some(1_000_000));
+            assert!(sql.contains("AND \"_o2_ingest_ts\" < 1000000 ORDER BY"));
+        }
+
         #[test]
         fn phase_two_selects_all_rows_for_discovered_traces() {
-            let sql = span_rows_sql("traces", &["abc-123".to_string(), "def-456".to_string()]);
+            let sql = span_rows_sql(
+                "traces",
+                &["abc-123".to_string(), "def-456".to_string()],
+                None,
+            );
 
             assert!(sql.contains("SELECT *"));
             assert!(sql.contains("\"trace_id\" IN ('abc-123', 'def-456')"));

src/config/src/meta/triggers.rs

@@ -38,6 +38,16 @@ pub enum TriggerModule {
     QueryRecommendations,
     Backfill,
     AnomalyDetection,
+    // APPEND ONLY. The implicit discriminant is the value stored in
+    // `scheduled_jobs.module` (this enum is persisted via `sqlx::Type` +
+    // `#[repr(i32)]`, not by name), so inserting a variant above this line
+    // silently remaps every existing row to a different module.
+    /// SLI ingest — one job per enabled SLO, cadence = its slice interval.
+    Slo,
+    /// Bulk historical fill. Its own lane, because a bulk scan sharing a
+    /// concurrency budget with latency-sensitive incremental passes would
+    /// starve them (§6b.9).
+    SloBackfill,
 }
 
 impl std::fmt::Display for TriggerModule {
@@ -49,6 +59,8 @@ impl std::fmt::Display for TriggerModule {
             Self::QueryRecommendations => write!(f, "query_recommendations"),
             Self::Backfill => write!(f, "backfill"),
             Self::AnomalyDetection => write!(f, "anomaly_detection"),
+            Self::Slo => write!(f, "slo"),
+            Self::SloBackfill => write!(f, "slo_backfill"),
         }
     }
 }
@@ -99,6 +111,20 @@ pub struct ScheduledTriggerData {
     pub last_satisfied_at: Option<i64>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub backfill_job: Option<BackfillJob>,
+    // ── Multi-level silence (alerts_2.md §7.1) ──────────────────────────────
+    // For alerts WITH a warning threshold, silence stops suppressing
+    // *evaluation* and suppresses only *delivery* — otherwise a
+    // Warning→Critical escalation during a silence window can never be
+    // observed. Single-level alerts keep the legacy behaviour (next_run_at is
+    // pushed forward) and never set these.
+    /// Deliver nothing until this timestamp, unless the level escalates.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub delivery_silenced_until: Option<i64>,
+    /// Severity of the last DELIVERED notification (`AlertLevel::to_i32`).
+    /// Escalation is measured against this, not against the previous
+    /// evaluation — otherwise a flap down and back up would re-notify.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub last_notified_level: Option<i32>,
 }
 
 #[derive(Clone, Debug, Serialize, Deserialize)]
@@ -165,6 +191,23 @@ mod tests {
             TriggerModule::AnomalyDetection.to_string(),
             "anomaly_detection"
         );
+        assert_eq!(TriggerModule::Slo.to_string(), "slo");
+        assert_eq!(TriggerModule::SloBackfill.to_string(), "slo_backfill");
+    }
+
+    /// The discriminant IS the stored value. A variant inserted above an
+    /// existing one would remap every `scheduled_jobs` row to a different
+    /// module — silently, with no migration to catch it.
+    #[test]
+    fn trigger_module_discriminants_are_pinned() {
+        assert_eq!(TriggerModule::Report as i32, 0);
+        assert_eq!(TriggerModule::Alert as i32, 1);
+        assert_eq!(TriggerModule::DerivedStream as i32, 2);
+        assert_eq!(TriggerModule::QueryRecommendations as i32, 3);
+        assert_eq!(TriggerModule::Backfill as i32, 4);
+        assert_eq!(TriggerModule::AnomalyDetection as i32, 5);
+        assert_eq!(TriggerModule::Slo as i32, 6);
+        assert_eq!(TriggerModule::SloBackfill as i32, 7);
     }
 
     #[test]
@@ -174,6 +217,8 @@ mod tests {
             tolerance: 42,
             last_satisfied_at: Some(999),
             backfill_job: None,
+            delivery_silenced_until: None,
+            last_notified_level: None,
         };
         data.reset();
         assert!(data.period_end_time.is_none());
@@ -197,6 +242,8 @@ mod tests {
             tolerance: 10,
             last_satisfied_at: Some(9_999_999),
             backfill_job: None,
+            delivery_silenced_until: None,
+            last_notified_level: None,
         };
         let json = data.to_json_string();
         let restored = ScheduledTriggerData::from_json_string(&json).unwrap();
@@ -357,6 +404,8 @@ mod tests {
             period_end_time: Some(500),
             tolerance: 5,
             last_satisfied_at: None,
+            delivery_silenced_until: None,
+            last_notified_level: None,
             backfill_job: Some(BackfillJob {
                 current_position: 42,
                 deletion_status: DeletionStatus::Pending,

src/config/src/metrics.rs

@@ -163,6 +163,46 @@ pub static INGEST_PARQUET_FILES: Lazy<IntGaugeVec> = Lazy::new(|| {
     )
     .expect("Metric created")
 });
+/// Checks waiting to be leased, per location and pool.
+///
+/// The other half of queue-lag visibility. A result record carries `scheduled_ts`
+/// and `started_ts`, so the delay of work that RAN is already derivable — but a
+/// check nobody leased produces no record at all, so the backlog it sits in is
+/// invisible from the results side by construction. Without this, "the queue is
+/// backed up" and "concurrency fixed it" are both unfalsifiable.
+pub static SYNTHETICS_PENDING_JOBS: Lazy<IntGaugeVec> = Lazy::new(|| {
+    IntGaugeVec::new(
+        Opts::new(
+            "synthetics_pending_jobs",
+            "Number of synthetics checks pending lease.".to_owned() + HELP_SUFFIX,
+        )
+        .namespace(NAMESPACE)
+        .const_labels(create_const_labels()),
+        &["location", "pool"],
+    )
+    .expect("Metric created")
+});
+
+/// Age of the oldest check still waiting, in seconds, per location and pool.
+///
+/// Reported alongside the count because they fail differently: a large count that
+/// drains every tick is throughput, while a small count whose oldest entry keeps
+/// ageing is a location that has stopped being served at all — one agent down, or
+/// no agent ever polling that pool. The count alone cannot tell those apart.
+pub static SYNTHETICS_OLDEST_PENDING_AGE_SECONDS: Lazy<IntGaugeVec> = Lazy::new(|| {
+    IntGaugeVec::new(
+        Opts::new(
+            "synthetics_oldest_pending_age_seconds",
+            "Age of the oldest synthetics check pending lease, in seconds.".to_owned()
+                + HELP_SUFFIX,
+        )
+        .namespace(NAMESPACE)
+        .const_labels(create_const_labels()),
+        &["location", "pool"],
+    )
+    .expect("Metric created")
+});
+
 pub static INGEST_PACK_FILES: Lazy<IntGaugeVec> = Lazy::new(|| {
     IntGaugeVec::new(
         Opts::new(
@@ -1952,6 +1992,71 @@ pub static QUEUE_OLDEST_MESSAGE_AGE_SECONDS: Lazy<IntGaugeVec> = Lazy::new(|| {
     .expect("Metric created")
 });
 
+pub static EVAL_SCHEDULER_PENDING_TARGETS: Lazy<IntGaugeVec> = Lazy::new(|| {
+    IntGaugeVec::new(
+        Opts::new(
+            "eval_scheduler_pending_targets",
+            "Current in-flight (pending) evaluation targets tracked by the eval scheduler, summed across the organization's streams",
+        )
+        .namespace(NAMESPACE)
+        .const_labels(create_const_labels()),
+        &["organization"],
+    )
+    .expect("Metric created")
+});
+
+pub static EVAL_SCHEDULER_PENDING_MEMORY_BYTES: Lazy<IntGaugeVec> = Lazy::new(|| {
+    IntGaugeVec::new(
+        Opts::new(
+            "eval_scheduler_pending_memory_bytes",
+            "Accounted bytes of pending evaluation targets, and the configured limit (state=used, limit)",
+        )
+        .namespace(NAMESPACE)
+        .const_labels(create_const_labels()),
+        &["state"],
+    )
+    .expect("Metric created")
+});
+
+pub static EVAL_SCHEDULER_FORCED_READY_TOTAL: Lazy<IntCounterVec> = Lazy::new(|| {
+    IntCounterVec::new(
+        Opts::new(
+            "eval_scheduler_forced_ready_total",
+            "Total pending evaluation targets force-evaluated early because the pending memory budget was exceeded",
+        )
+        .namespace(NAMESPACE)
+        .const_labels(create_const_labels()),
+        &["organization"],
+    )
+    .expect("Metric created")
+});
+
+pub static EVAL_SCHEDULER_EVICTED_EVIDENCE_TOTAL: Lazy<IntCounterVec> = Lazy::new(|| {
+    IntCounterVec::new(
+        Opts::new(
+            "eval_scheduler_evicted_evidence_total",
+            "Evaluation evidence dropped by pending-memory budget enforcement (kind=orphan: unbound session evidence whose session evaluation may be lost until a restart replays it; kind=binding: trace-to-session bindings). Sustained increases mean the budget does not match the workload",
+        )
+        .namespace(NAMESPACE)
+        .const_labels(create_const_labels()),
+        &["organization", "kind"],
+    )
+    .expect("Metric created")
+});
+
+pub static EVAL_SCHEDULER_WATERMARK_LAG_SECONDS: Lazy<IntGaugeVec> = Lazy::new(|| {
+    IntGaugeVec::new(
+        Opts::new(
+            "eval_scheduler_watermark_lag_seconds",
+            "Lag in seconds between the eval scheduler scan cursor and the committed (persisted) watermark, worst case across the organization's streams",
+        )
+        .namespace(NAMESPACE)
+        .const_labels(create_const_labels()),
+        &["organization"],
+    )
+    .expect("Metric created")
+});
+
 fn register_metrics(registry: &Registry) {
     // http latency
     registry
@@ -1985,6 +2090,12 @@ fn register_metrics(registry: &Registry) {
     registry
         .register(Box::new(INGEST_PARQUET_FILES.clone()))
         .expect("Metric registered");
+    registry
+        .register(Box::new(SYNTHETICS_PENDING_JOBS.clone()))
+        .expect("Metric registered");
+    registry
+        .register(Box::new(SYNTHETICS_OLDEST_PENDING_AGE_SECONDS.clone()))
+        .expect("Metric registered");
     registry
         .register(Box::new(INGEST_PACK_FILES.clone()))
         .expect("Metric registered");
@@ -2452,6 +2563,22 @@ fn register_metrics(registry: &Registry) {
     registry
         .register(Box::new(QUEUE_OLDEST_MESSAGE_AGE_SECONDS.clone()))
         .expect("Metric registered");
+    // eval scheduler pending-target metrics
+    registry
+        .register(Box::new(EVAL_SCHEDULER_PENDING_TARGETS.clone()))
+        .expect("Metric registered");
+    registry
+        .register(Box::new(EVAL_SCHEDULER_PENDING_MEMORY_BYTES.clone()))
+        .expect("Metric registered");
+    registry
+        .register(Box::new(EVAL_SCHEDULER_FORCED_READY_TOTAL.clone()))
+        .expect("Metric registered");
+    registry
+        .register(Box::new(EVAL_SCHEDULER_EVICTED_EVIDENCE_TOTAL.clone()))
+        .expect("Metric registered");
+    registry
+        .register(Box::new(EVAL_SCHEDULER_WATERMARK_LAG_SECONDS.clone()))
+        .expect("Metric registered");
 }
 
 pub fn create_const_labels() -> HashMap<String, String> {
@@ -2569,6 +2696,8 @@ mod tests {
         let _ = INGEST_ERRORS.clone();
         let _ = INGEST_WAL_USED_BYTES.clone();
         let _ = INGEST_PARQUET_FILES.clone();
+        let _ = SYNTHETICS_PENDING_JOBS.clone();
+        let _ = SYNTHETICS_OLDEST_PENDING_AGE_SECONDS.clone();
         let _ = INGEST_PACK_FILES.clone();
         let _ = INGEST_PACK_SEGMENTS.clone();
         let _ = INGEST_WAL_WRITE_BYTES.clone();

src/config/src/prebuilt_loader.rs

@@ -117,10 +117,10 @@ pub const PREBUILT_TEMPLATE_PREFIX: &str = "prebuilt_";
 /// `prebuilt_<type>` where `<type>` is a registered prebuilt destination type
 /// (slack, opsgenie, servicenow, ...).
 ///
-/// Lives next to `get_prebuilt_template` so the public API surface
-/// (`handler::http::models`), the service-layer guards
-/// (`service::alerts::templates`), and any future caller all derive the
-/// "is system template" answer from the same source of truth.
+/// Lives next to `get_prebuilt_template` so the API models
+/// (`openobserve_api_management::models`), the service-layer guards
+/// (`openobserve_core::alerts::templates`), and any future caller all derive
+/// the "is system template" answer from the same source of truth.
 ///
 /// User-created templates whose names happen to start with `prebuilt_` but
 /// don't match a registered type stay freely editable and deletable.

src/config/src/utils/parquet.rs

@@ -34,6 +34,7 @@ use parquet::{
     basic::{Compression, Encoding},
     file::{metadata::KeyValue, properties::WriterProperties},
 };
+use serde::{Deserialize, Serialize};
 use vortex::{
     VortexSessionDefault,
     array::{ArrayRef, VortexSessionExecute},
@@ -44,7 +45,31 @@ use vortex::{
     session::VortexSession,
 };
 
-use crate::{FileFormat, config::*, ider, meta::stream::FileMeta};
+use crate::{FileFormat, config::*, ider, meta::stream::FileMeta, utils::json};
+
+/// Key of the vortex metadata segment carrying the o2 [`FileMeta`].
+pub const VORTEX_FILE_META_KEY: &str = "o2_file_meta";
+
+/// Same four fields [`new_parquet_writer`] writes into the parquet footer.
+#[derive(Debug, Default, Serialize, Deserialize)]
+#[serde(default)]
+struct VortexFileMeta {
+    min_ts: i64,
+    max_ts: i64,
+    records: i64,
+    original_size: i64,
+}
+
+/// Encode `metadata` for the [`VORTEX_FILE_META_KEY`] segment.
+pub fn encode_vortex_file_meta(metadata: &FileMeta) -> Vec<u8> {
+    json::to_vec(&VortexFileMeta {
+        min_ts: metadata.min_ts,
+        max_ts: metadata.max_ts,
+        records: metadata.records,
+        original_size: metadata.original_size,
+    })
+    .expect("file meta is always serializable")
+}
 
 pub fn new_parquet_writer<'a>(
     buf: &'a mut Vec<u8>,
@@ -496,6 +521,31 @@ mod tests {
         assert_eq!(read_metadata.original_size, metadata.original_size);
     }
 
+    #[test]
+    fn test_encode_decode_vortex_file_meta() {
+        let metadata = FileMeta {
+            min_ts: -1,
+            max_ts: i64::MAX,
+            records: 7,
+            original_size: 8,
+            compressed_size: 9,
+            index_size: 10,
+            bloom_ver: 11,
+            flattened: true,
+        };
+        let decoded: VortexFileMeta =
+            json::from_slice(&encode_vortex_file_meta(&metadata)).unwrap();
+        assert_eq!(decoded.min_ts, metadata.min_ts);
+        assert_eq!(decoded.max_ts, metadata.max_ts);
+        assert_eq!(decoded.records, metadata.records);
+        assert_eq!(decoded.original_size, metadata.original_size);
+
+        // unknown and missing keys are tolerated
+        let decoded: VortexFileMeta = json::from_slice(br#"{"min_ts":5,"future":true}"#).unwrap();
+        assert_eq!(decoded.min_ts, 5);
+        assert_eq!(decoded.max_ts, 0);
+    }
+
     #[test]
     fn test_parse_file_key_columns() {
         let key = "files/default/logs/olympics/2022/10/03/10/6982652937134804993_1.parquet";

src/core/src/alerts/deduplication.rs

@@ -28,25 +28,150 @@ use config::{
 use infra::table::entity::alert_dedup_state;
 use sea_orm::{ActiveModelTrait, ColumnTrait, DatabaseConnection, EntityTrait, QueryFilter, Set};
 
+/// Append the evaluated level to a fingerprint for MULTI-LEVEL alerts.
+///
+/// A Warning batch and a Critical batch must never share a dedup identity —
+/// otherwise deduplication discards the Warning→Critical escalation that the
+/// scheduler explicitly allowed through silence (§7.1). Single-level alerts
+/// keep their legacy fingerprints byte-for-byte, so existing dedup state is
+/// not invalidated on upgrade. The `|level:` separator is deliberately not
+/// the `,dim=val` shape, so dimension parsing never mistakes it for a field.
+fn with_level_component(
+    base: String,
+    alert: &Alert,
+    level: Option<config::meta::alerts::level::AlertLevel>,
+) -> String {
+    let multi_level = alert.trigger_condition.warning_threshold.is_some()
+        || alert
+            .query_condition
+            .aggregation
+            .as_ref()
+            .is_some_and(|a| a.warning_value.is_some())
+        || alert.query_condition.promql_warning_value.is_some();
+    match (multi_level, level) {
+        (true, Some(l)) => format!("{base}|level:{l}"),
+        _ => base,
+    }
+}
+
 /// Calculate fingerprint for an alert result row
 ///
-/// Delegates to enterprise implementation.
+/// Delegates to enterprise implementation; the evaluated level is appended as
+/// an implicit component for multi-level alerts (see `with_level_component`).
 pub fn calculate_fingerprint(
     alert: &Alert,
     result_row: &Map<String, Value>,
     config: &DeduplicationConfig,
     org_config: Option<&GlobalDeduplicationConfig>,
     semantic_groups: &[config::meta::correlation::FieldAlias],
+    level: Option<config::meta::alerts::level::AlertLevel>,
 ) -> String {
-    o2_enterprise::enterprise::alerts::dedup::calculate_fingerprint(
+    let base = o2_enterprise::enterprise::alerts::dedup::calculate_fingerprint(
         alert,
         result_row,
         config,
         org_config,
         semantic_groups,
+    );
+
+    // Level and group are the two IMPLICIT fingerprint components (M-5). The
+    // group one is derived from the row itself rather than passed in, so every
+    // caller gets it — there is no way to compute a fingerprint for a
+    // multi-alert row and accidentally leave the group out, which would let
+    // one group's notification dedup away another's.
+    //
+    // `with_group_component` returns the base unchanged for an ungrouped alert
+    // and for the rollup key, so existing fingerprints stay byte-identical and
+    // no live silence window is invalidated by the upgrade.
+    let with_level = with_level_component(base, alert, level);
+    config::meta::alerts::grouping::with_group_component(
+        with_level,
+        row_group_key(alert, result_row).as_deref(),
     )
 }
 
+/// This row's group identity, for the M-5 fingerprint component.
+///
+/// `None` for anything that is not an opted-in multi-alert, which is what
+/// keeps every pre-existing alert's fingerprint unchanged.
+fn row_group_key(alert: &Alert, row: &Map<String, Value>) -> Option<String> {
+    if !alert.query_condition.multi_alert_enabled() {
+        return None;
+    }
+    // Same extractors the evaluation and dispatch use, so the fingerprint's
+    // notion of "which group" cannot drift from the state row's. Which one
+    // applies is decided by the family, exactly as it is in dispatch: reading
+    // `aggregation.group_by` for a PromQL alert yields no columns and hence no
+    // group component, which would give every SERIES the same fingerprint and
+    // let one series' notification dedup away another's — the precise failure
+    // the note above this function warns about.
+    let labels = match alert.query_condition.query_type {
+        config::meta::alerts::QueryType::PromQL => {
+            config::meta::alerts::dispatch::promql_series_labels(row)
+        }
+        _ => {
+            let group_by = alert
+                .query_condition
+                .aggregation
+                .as_ref()?
+                .group_by
+                .as_ref()?;
+            config::meta::alerts::dispatch::row_group_labels(row, group_by)
+        }
+    };
+    Some(config::meta::alerts::grouping::group_key(&labels))
+}
+
+/// One fingerprint reserved by a dedup pass, tagged with the group whose
+/// delivery must confirm it (§5.5 MN-4/MN-6). Group A's send can succeed
+/// while B's fails; confirming B on A's success would suppress B's retry as
+/// a duplicate for the rest of the window.
+pub struct ReservedFingerprint {
+    pub fingerprint: String,
+    /// `None` for anything that is not an opted-in multi-alert, whose single
+    /// send confirms every reservation.
+    pub group_key: Option<String>,
+}
+
+/// What one pass of [`apply_deduplication`] decided.
+pub struct DeduplicationOutcome {
+    /// The rows that survived deduplication and should be notified on.
+    pub rows: Vec<Map<String, Value>>,
+    /// Whether deduplication actually ran (it is opt-in per alert).
+    pub applied: bool,
+    /// Fingerprints RESERVED by this pass — recorded as seen but not yet
+    /// confirmed as delivered (§5.5 MN-6).
+    ///
+    /// The caller must pass these to [`confirm_notification_sent`] once the
+    /// notification actually goes out — per group for a multi-alert. Until it
+    /// does, they do not suppress: that is what lets a failed send be retried
+    /// instead of being swallowed as a duplicate for the rest of the window.
+    pub reserved: Vec<ReservedFingerprint>,
+}
+
+/// Confirm reservations whose notification was delivered (§5.5 MN-6).
+///
+/// Called only after a successful send. Anything left unconfirmed is treated
+/// as a delivery that never happened, so the next evaluation is allowed
+/// through — no retry bookkeeping required.
+pub async fn confirm_notification_sent(
+    db: &DatabaseConnection,
+    fingerprints: &[String],
+) -> Result<(), sea_orm::DbErr> {
+    if fingerprints.is_empty() {
+        return Ok(());
+    }
+    alert_dedup_state::Entity::update_many()
+        .col_expr(
+            alert_dedup_state::Column::NotificationSent,
+            sea_orm::sea_query::Expr::value(true),
+        )
+        .filter(alert_dedup_state::Column::Fingerprint.is_in(fingerprints.to_vec()))
+        .exec(db)
+        .await?;
+    Ok(())
+}
+
 /// Get or create deduplication state
 pub async fn get_dedup_state(
     db: &DatabaseConnection,
@@ -161,11 +286,19 @@ pub async fn apply_deduplication(
     db: &DatabaseConnection,
     alert: &Alert,
     result_rows: Vec<Map<String, Value>>,
-) -> Result<(Vec<Map<String, Value>>, bool), sea_orm::DbErr> {
+    level: Option<config::meta::alerts::level::AlertLevel>,
+) -> Result<DeduplicationOutcome, sea_orm::DbErr> {
     // Check if per-alert deduplication is enabled
     let dedup_config = match &alert.deduplication {
         Some(config) if config.enabled => config,
-        _ => return Ok((result_rows, false)), // Deduplication disabled, return all rows
+        // Deduplication disabled, return all rows
+        _ => {
+            return Ok(DeduplicationOutcome {
+                rows: result_rows,
+                applied: false,
+                reserved: Vec::new(),
+            });
+        }
     };
 
     // Get semantic groups from system_settings — the single source of truth
@@ -185,9 +318,14 @@ pub async fn apply_deduplication(
         dedup_config,
         org_config.as_ref(),
         &semantic_groups,
+        level,
     )
     .await
-    .map(|result| (result, true))
+    .map(|(rows, reserved)| DeduplicationOutcome {
+        rows,
+        applied: true,
+        reserved,
+    })
 }
 
 /// Enterprise implementation of apply_deduplication
@@ -198,7 +336,8 @@ async fn apply_deduplication_impl(
     dedup_config: &DeduplicationConfig,
     org_config: Option<&GlobalDeduplicationConfig>,
     semantic_groups: &[config::meta::correlation::FieldAlias],
-) -> Result<Vec<Map<String, Value>>, sea_orm::DbErr> {
+    level: Option<config::meta::alerts::level::AlertLevel>,
+) -> Result<(Vec<Map<String, Value>>, Vec<ReservedFingerprint>), sea_orm::DbErr> {
     let now = o2_enterprise::enterprise::alerts::dedup::current_timestamp_micros();
     let alert_id = alert.get_unique_key();
     let org_id = &alert.org_id;
@@ -210,14 +349,34 @@ async fn apply_deduplication_impl(
     );
 
     let mut deduplicated_rows = Vec::new();
+    // Reserved but unconfirmed until the notification actually lands.
+    let mut reserved = Vec::new();
 
     for row in result_rows {
-        let fingerprint =
-            calculate_fingerprint(alert, &row, dedup_config, org_config, semantic_groups);
+        let fingerprint = calculate_fingerprint(
+            alert,
+            &row,
+            dedup_config,
+            org_config,
+            semantic_groups,
+            level,
+        );
 
-        // Check if this fingerprint exists and is within time window
+        // A reservation suppresses only once a delivery has CONFIRMED it
+        // (§5.5 MN-6). Recording the fingerprint when a row merely *passes*
+        // dedup — which is what happens below — means a send that then fails
+        // is suppressed as a duplicate on every later evaluation while
+        // `last_seen_at` keeps extending the window: one transient webhook
+        // error swallows the page for the whole window. `notification_sent`
+        // has existed on this table since the feature shipped and was never
+        // read; it is the confirm flag.
         let should_send = match get_dedup_state(db, &fingerprint).await? {
-            Some(existing_state) if is_within_window(&existing_state, time_window_minutes) => {
+            Some(existing_state)
+                if config::meta::alerts::deduplication::reservation_suppresses(
+                    existing_state.notification_sent,
+                    is_within_window(&existing_state, time_window_minutes),
+                ) =>
+            {
                 // Within window - update occurrence count but don't send
                 if let Err(e) = save_dedup_state(
                     db,
@@ -300,9 +459,143 @@ async fn apply_deduplication_impl(
                 fingerprint
             );
 
+            // The reservation carries its group so the caller can confirm it
+            // against that group's OWN delivery outcome, not a sibling's.
+            reserved.push(ReservedFingerprint {
+                group_key: row_group_key(alert, &row),
+                fingerprint,
+            });
             deduplicated_rows.push(row);
         }
     }
 
-    Ok(deduplicated_rows)
+    Ok((deduplicated_rows, reserved))
+}
+
+#[cfg(test)]
+mod tests {
+    use config::meta::alerts::{
+        AggFunction, Aggregation, Condition, Operator, QueryType, alert::Alert,
+    };
+    use serde_json::{Map, Value, json};
+
+    use super::row_group_key;
+
+    fn row(pairs: &[(&str, Value)]) -> Map<String, Value> {
+        pairs
+            .iter()
+            .map(|(k, v)| (k.to_string(), v.clone()))
+            .collect()
+    }
+
+    fn promql_alert(multi: bool) -> Alert {
+        let mut a = Alert::default();
+        a.query_condition.query_type = QueryType::PromQL;
+        a.query_condition.promql = Some("sum by (pod) (rate(errors[5m]))".into());
+        a.query_condition.promql_multi_alert = multi;
+        a
+    }
+
+    fn agg_alert(multi: bool) -> Alert {
+        let mut a = Alert::default();
+        a.query_condition.query_type = QueryType::SQL;
+        a.query_condition.aggregation = Some(Aggregation {
+            group_by: Some(vec!["host".into()]),
+            function: AggFunction::Avg,
+            having: Condition {
+                column: "alert_agg_value".into(),
+                operator: Operator::GreaterThan,
+                value: json!(90),
+                ignore_case: false,
+            },
+            warning_value: None,
+            multi_alert: multi,
+        });
+        a
+    }
+
+    /// M-5, PromQL flavour. Without a per-series component in the fingerprint,
+    /// every series of one alert dedups against every other — so the first
+    /// series to page suppresses all the rest for the whole window, which is
+    /// the exact opposite of what per-series alerting is for.
+    #[test]
+    fn a_promql_multi_alert_gets_a_distinct_group_key_per_series() {
+        let alert = promql_alert(true);
+        let a = row_group_key(
+            &alert,
+            &row(&[("pod", json!("web-1")), ("value", json!(5.0))]),
+        );
+        let b = row_group_key(
+            &alert,
+            &row(&[("pod", json!("web-2")), ("value", json!(9.0))]),
+        );
+
+        assert!(
+            a.is_some(),
+            "a per-series alert must carry a group component"
+        );
+        assert_ne!(a, b, "two series must not share one fingerprint");
+    }
+
+    /// The value moves every evaluation; the identity must not.
+    #[test]
+    fn a_promql_series_keeps_its_key_as_its_value_changes() {
+        let alert = promql_alert(true);
+        let a = row_group_key(
+            &alert,
+            &row(&[("pod", json!("web-1")), ("value", json!(5.0))]),
+        );
+        let b = row_group_key(
+            &alert,
+            &row(&[("pod", json!("web-1")), ("value", json!(500.0))]),
+        );
+        // `is_some` first: without it this passes trivially when BOTH are None,
+        // which is exactly the broken state — `None == None`.
+        assert!(
+            a.is_some(),
+            "a per-series alert must carry a group component"
+        );
+        assert_eq!(a, b);
+    }
+
+    /// Upgrade safety: a PromQL alert that never opted in must keep the
+    /// fingerprint it has today, or every live silence window is invalidated.
+    #[test]
+    fn a_promql_alert_that_did_not_opt_in_has_no_group_component() {
+        let alert = promql_alert(false);
+        assert_eq!(
+            row_group_key(
+                &alert,
+                &row(&[("pod", json!("web-1")), ("value", json!(5.0))])
+            ),
+            None
+        );
+    }
+
+    #[test]
+    fn the_aggregation_family_is_unchanged() {
+        let alert = agg_alert(true);
+        let a = row_group_key(
+            &alert,
+            &row(&[("host", json!("a")), ("alert_agg_value", json!(1))]),
+        );
+        let b = row_group_key(
+            &alert,
+            &row(&[("host", json!("b")), ("alert_agg_value", json!(1))]),
+        );
+        assert!(a.is_some());
+        assert_ne!(a, b);
+
+        let simple = agg_alert(false);
+        assert_eq!(row_group_key(&simple, &row(&[("host", json!("a"))])), None);
+    }
+
+    /// An SLO alert has no per-group dispatch (see `multi_alert_enabled`), so
+    /// it must not grow a group component either.
+    #[test]
+    fn an_slo_alert_has_no_group_component() {
+        let mut alert = Alert::default();
+        alert.query_condition.query_type = QueryType::Slo;
+        assert_eq!(row_group_key(&alert, &row(&[("group", json!("g"))])), None);
+    }
 }

src/core/src/alerts/external_alerts/detect.rs

@@ -0,0 +1,90 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use serde_json::Value;
+
+/// Result of inspecting an inbound webhook payload to figure out which
+/// external alerting system produced it (spec §4.2).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum DetectedSource {
+    Grafana,
+    Alertmanager,
+    Generic,
+    Unknown,
+}
+
+impl DetectedSource {
+    pub fn as_str(&self) -> &'static str {
+        match self {
+            DetectedSource::Grafana => "grafana",
+            DetectedSource::Alertmanager => "alertmanager",
+            DetectedSource::Generic => "generic",
+            DetectedSource::Unknown => "unknown",
+        }
+    }
+}
+
+fn object_has_status_and_labels(obj: &serde_json::Map<String, Value>) -> bool {
+    matches!(obj.get("status"), Some(Value::String(_)))
+        && matches!(obj.get("labels"), Some(Value::Object(_)))
+}
+
+/// Detect which external alerting system produced `body`, using the
+/// `user_agent` header as a secondary signal for the Grafana/Alertmanager
+/// ambiguity (both share the same `alerts[]` wire format).
+pub fn detect_source(user_agent: Option<&str>, body: &Value) -> DetectedSource {
+    if let Some(obj) = body.as_object() {
+        let has_alerts_array = matches!(obj.get("alerts"), Some(Value::Array(_)));
+        let has_group_or_version = obj.contains_key("groupKey") || obj.contains_key("version");
+        if has_alerts_array && has_group_or_version {
+            let ua_is_grafana = user_agent.map(|ua| ua.contains("Grafana")).unwrap_or(false);
+            let has_grafana_keys =
+                obj.contains_key("orgId") || obj.contains_key("title") || obj.contains_key("state");
+            return if ua_is_grafana || has_grafana_keys {
+                DetectedSource::Grafana
+            } else {
+                DetectedSource::Alertmanager
+            };
+        }
+
+        if matches!(obj.get("source"), Some(Value::String(_))) {
+            return DetectedSource::Generic;
+        }
+
+        if object_has_status_and_labels(obj) {
+            return DetectedSource::Generic;
+        }
+
+        return DetectedSource::Unknown;
+    }
+
+    if let Some(arr) = body.as_array() {
+        if !arr.is_empty()
+            && arr.iter().all(|item| {
+                item.as_object()
+                    .map(|o| {
+                        matches!(o.get("source"), Some(Value::String(_)))
+                            || object_has_status_and_labels(o)
+                    })
+                    .unwrap_or(false)
+            })
+        {
+            return DetectedSource::Generic;
+        }
+        return DetectedSource::Unknown;
+    }
+
+    DetectedSource::Unknown
+}

src/core/src/alerts/external_alerts/generic.rs

@@ -0,0 +1,119 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::collections::HashMap;
+
+use config::meta::alerts::incidents::{
+    ExternalAlertEvent, ExternalAlertStatus, map_external_severity,
+};
+use serde_json::Value;
+
+use super::CLOCK_SKEW_TOLERANCE_MICROS;
+
+fn clamp_ts(ts: i64, now: i64) -> i64 {
+    let lo = now - CLOCK_SKEW_TOLERANCE_MICROS;
+    let hi = now + CLOCK_SKEW_TOLERANCE_MICROS;
+    ts.clamp(lo, hi)
+}
+
+fn dedup_key_hash(labels: &HashMap<String, String>) -> String {
+    let mut pairs: Vec<String> = labels.iter().map(|(k, v)| format!("{k}={v}")).collect();
+    pairs.sort();
+    let joined = pairs.join("\n");
+    format!(
+        "{:016x}",
+        config::utils::hash::sum64_bytes(joined.as_bytes())
+    )
+}
+
+fn normalize_one(item: &Value, now: i64) -> Result<ExternalAlertEvent, String> {
+    let obj = item
+        .as_object()
+        .ok_or_else(|| "generic alert item must be an object".to_string())?;
+
+    let status_str = obj
+        .get("status")
+        .and_then(|s| s.as_str())
+        .ok_or_else(|| "generic alert item missing required 'status' field".to_string())?;
+    let status = match status_str {
+        "firing" => ExternalAlertStatus::Firing,
+        "resolved" => ExternalAlertStatus::Resolved,
+        other => return Err(format!("unrecognized status '{other}'")),
+    };
+
+    let raw_labels = obj
+        .get("labels")
+        .and_then(|l| l.as_object())
+        .ok_or_else(|| "generic alert item missing required 'labels' field".to_string())?;
+
+    let mut labels: HashMap<String, String> = HashMap::new();
+    for (k, v) in raw_labels.iter() {
+        let s = match v {
+            Value::String(s) => s.clone(),
+            other => other.to_string(),
+        };
+        labels.insert(k.clone(), s);
+    }
+
+    let severity_raw = obj
+        .get("severity")
+        .and_then(|s| s.as_str())
+        .unwrap_or("warning");
+    let severity = map_external_severity(severity_raw);
+
+    let title = obj
+        .get("title")
+        .and_then(|t| t.as_str())
+        .map(|s| s.to_string())
+        .or_else(|| labels.values().next().cloned())
+        .unwrap_or_else(|| "external alert".to_string());
+
+    let dedup_key = obj
+        .get("dedup_key")
+        .and_then(|d| d.as_str())
+        .map(|s| s.to_string())
+        .unwrap_or_else(|| dedup_key_hash(&labels));
+
+    let event_ts = obj
+        .get("event_ts")
+        .and_then(|t| t.as_i64())
+        .map(|t| clamp_ts(t, now))
+        .unwrap_or(now);
+
+    let source_url = obj
+        .get("source_url")
+        .and_then(|s| s.as_str())
+        .map(|s| s.to_string());
+
+    Ok(ExternalAlertEvent {
+        status,
+        dedup_key,
+        title,
+        severity,
+        labels,
+        event_ts,
+        source_url,
+        raw: item.clone(),
+    })
+}
+
+/// Normalize a generic JSON payload — a single alert object or an array of
+/// alert objects, each requiring `status` and `labels`.
+pub fn normalize_generic(body: &Value, now: i64) -> Result<Vec<ExternalAlertEvent>, String> {
+    if let Some(arr) = body.as_array() {
+        return arr.iter().map(|item| normalize_one(item, now)).collect();
+    }
+    Ok(vec![normalize_one(body, now)?])
+}

src/core/src/alerts/external_alerts/grafana.rs

@@ -0,0 +1,151 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::collections::HashMap;
+
+use config::meta::alerts::incidents::{
+    ExternalAlertEvent, ExternalAlertStatus, map_external_severity,
+};
+use serde_json::Value;
+
+use super::CLOCK_SKEW_TOLERANCE_MICROS;
+
+/// Labels never surfaced on the normalized event (spec §4.2 deny-list).
+const LABEL_DENY_LIST: &[&str] = &["job"];
+
+fn labels_hash(labels: &serde_json::Map<String, Value>) -> String {
+    let mut pairs: Vec<String> = labels
+        .iter()
+        .map(|(k, v)| format!("{k}={}", value_to_label_string(v)))
+        .collect();
+    pairs.sort();
+    let joined = pairs.join("\n");
+    format!(
+        "{:016x}",
+        config::utils::hash::sum64_bytes(joined.as_bytes())
+    )
+}
+
+fn value_to_label_string(v: &Value) -> String {
+    match v {
+        Value::String(s) => s.clone(),
+        other => other.to_string(),
+    }
+}
+
+fn clamp_ts(ts: i64, now: i64) -> i64 {
+    let lo = now - CLOCK_SKEW_TOLERANCE_MICROS;
+    let hi = now + CLOCK_SKEW_TOLERANCE_MICROS;
+    ts.clamp(lo, hi)
+}
+
+fn parse_rfc3339_micros(s: &str) -> Option<i64> {
+    if s.is_empty() || s == "0001-01-01T00:00:00Z" {
+        return None;
+    }
+    chrono::DateTime::parse_from_rfc3339(s)
+        .ok()
+        .map(|dt| dt.timestamp_micros())
+}
+
+/// Normalize a Grafana or Alertmanager webhook payload — both share the same
+/// wire format (`alerts[]` array of firing/resolved entries).
+pub fn normalize_am_format(body: &Value, now: i64) -> Result<Vec<ExternalAlertEvent>, String> {
+    let alerts = body
+        .get("alerts")
+        .and_then(|a| a.as_array())
+        .ok_or_else(|| "missing alerts array".to_string())?;
+
+    let mut events = Vec::with_capacity(alerts.len());
+    for alert in alerts {
+        let status_str = alert
+            .get("status")
+            .and_then(|s| s.as_str())
+            .unwrap_or("firing");
+        let status = if status_str == "resolved" {
+            ExternalAlertStatus::Resolved
+        } else {
+            ExternalAlertStatus::Firing
+        };
+
+        let raw_labels = alert
+            .get("labels")
+            .and_then(|l| l.as_object())
+            .cloned()
+            .unwrap_or_default();
+
+        let dedup_key = alert
+            .get("fingerprint")
+            .and_then(|f| f.as_str())
+            .map(|s| s.to_string())
+            .unwrap_or_else(|| labels_hash(&raw_labels));
+
+        let severity_raw = raw_labels
+            .get("severity")
+            .and_then(|s| s.as_str())
+            .unwrap_or("warning");
+        let severity = map_external_severity(severity_raw);
+
+        let mut labels: HashMap<String, String> = HashMap::new();
+        for (k, v) in raw_labels.iter() {
+            if LABEL_DENY_LIST.contains(&k.as_str()) {
+                continue;
+            }
+            labels.insert(k.clone(), value_to_label_string(v));
+        }
+
+        let ts_field = if status == ExternalAlertStatus::Resolved {
+            "endsAt"
+        } else {
+            "startsAt"
+        };
+        let event_ts = alert
+            .get(ts_field)
+            .and_then(|v| v.as_str())
+            .and_then(parse_rfc3339_micros)
+            .unwrap_or(now);
+        let event_ts = clamp_ts(event_ts, now);
+
+        let title = labels
+            .get("alertname")
+            .cloned()
+            .or_else(|| {
+                alert
+                    .get("annotations")
+                    .and_then(|a| a.get("summary"))
+                    .and_then(|s| s.as_str())
+                    .map(|s| s.to_string())
+            })
+            .unwrap_or_else(|| "external alert".to_string());
+
+        let source_url = alert
+            .get("generatorURL")
+            .and_then(|s| s.as_str())
+            .map(|s| s.to_string());
+
+        events.push(ExternalAlertEvent {
+            status,
+            dedup_key,
+            title,
+            severity,
+            labels,
+            event_ts,
+            source_url,
+            raw: alert.clone(),
+        });
+    }
+
+    Ok(events)
+}

src/core/src/alerts/external_alerts/mod.rs

@@ -0,0 +1,290 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+pub mod detect;
+pub mod generic;
+pub mod grafana;
+
+use config::meta::alerts::incidents::ExternalAlertEvent;
+pub use detect::{DetectedSource, detect_source};
+
+/// Tolerance applied when clamping a source-reported event timestamp against
+/// our own receipt clock — guards against wildly skewed sender clocks
+/// (spec §4.2).
+pub const CLOCK_SKEW_TOLERANCE_MICROS: i64 = 3_600_000_000; // 1h
+
+/// Dispatch to the per-format normalizer for `detected`.
+pub fn normalize(
+    detected: DetectedSource,
+    body: &serde_json::Value,
+    now_micros: i64,
+) -> Result<Vec<ExternalAlertEvent>, String> {
+    match detected {
+        DetectedSource::Grafana | DetectedSource::Alertmanager => {
+            grafana::normalize_am_format(body, now_micros)
+        }
+        DetectedSource::Generic => generic::normalize_generic(body, now_micros),
+        DetectedSource::Unknown => Err("unrecognized payload format".to_string()),
+    }
+}
+
+/// Derives a display label for the sender of a batch of normalized events,
+/// from the first event's `source` label (spec §1: single request = single
+/// label). Returns `None` when absent, empty, or whitespace-only — callers
+/// fall back to the already-detected system name (spec §2).
+pub fn derive_sender_label(events: &[ExternalAlertEvent]) -> Option<String> {
+    events
+        .first()
+        .and_then(|e| e.labels.get("source"))
+        .map(|s| s.trim().to_string())
+        .filter(|s| !s.is_empty())
+}
+
+/// Resolves the display name shown in the status panel: the sender-provided
+/// label when present, otherwise the detected system name (grafana /
+/// alertmanager / generic) — exactly today's behavior.
+pub fn resolve_display_name(detected_source: &str, sender_label: Option<&str>) -> String {
+    sender_label
+        .filter(|s| !s.is_empty())
+        .unwrap_or(detected_source)
+        .to_string()
+}
+
+#[cfg(test)]
+mod tests {
+    use std::collections::HashMap;
+
+    use super::*;
+
+    fn grafana_payload() -> serde_json::Value {
+        serde_json::json!({
+            "receiver": "o2", "status": "firing", "orgId": 1,
+            "title": "[FIRING:1]", "state": "alerting", "groupKey": "{}:{}",
+            "version": "1", "externalURL": "https://grafana.example",
+            "alerts": [{
+                "status": "firing",
+                "labels": {"alertname": "HighCPU", "namespace": "prod", "deployment": "checkout", "severity": "critical", "job": "kube-state-metrics"},
+                "annotations": {"summary": "CPU high"},
+                "startsAt": "2026-07-30T10:02:00Z", "endsAt": "0001-01-01T00:00:00Z",
+                "generatorURL": "https://grafana.example/alerting/rule1",
+                "fingerprint": "abcdef0123456789"
+            }]
+        })
+    }
+
+    fn alertmanager_payload() -> serde_json::Value {
+        serde_json::json!({
+            "version": "4", "groupKey": "{}:{alertname=\"DiskLatency\"}",
+            "status": "firing", "receiver": "o2",
+            "groupLabels": {}, "commonLabels": {}, "commonAnnotations": {},
+            "externalURL": "http://am:9093", "truncatedAlerts": 0,
+            "alerts": [{
+                "status": "resolved",
+                "labels": {"alertname": "DiskLatency", "namespace": "prod", "severity": "warning"},
+                "annotations": {},
+                "startsAt": "2026-07-30T10:00:00Z", "endsAt": "2026-07-30T10:30:00Z",
+                "generatorURL": "http://prom/graph",
+                "fingerprint": "ffff000011112222"
+            }]
+        })
+    }
+
+    const NOW: i64 = 1_785_405_600_000_000; // fixed "now" for clamp determinism
+
+    #[test]
+    fn test_detect_grafana_vs_alertmanager_vs_generic() {
+        assert_eq!(
+            detect_source(Some("Grafana/12.0"), &grafana_payload()),
+            DetectedSource::Grafana
+        );
+        assert_eq!(
+            detect_source(None, &grafana_payload()),
+            DetectedSource::Grafana
+        ); // orgId/title/state keys
+        assert_eq!(
+            detect_source(Some("Alertmanager/0.27.0"), &alertmanager_payload()),
+            DetectedSource::Alertmanager
+        );
+        let generic =
+            serde_json::json!({"status": "firing", "labels": {"service": "x"}, "title": "t"});
+        assert_eq!(detect_source(None, &generic), DetectedSource::Generic);
+        assert_eq!(
+            detect_source(None, &serde_json::json!({"hello": 1})),
+            DetectedSource::Unknown
+        );
+    }
+
+    #[test]
+    fn test_normalize_grafana_extracts_event_and_denies_job_label() {
+        let evs = normalize(DetectedSource::Grafana, &grafana_payload(), NOW).unwrap();
+        assert_eq!(evs.len(), 1);
+        let ev = &evs[0];
+        assert_eq!(ev.dedup_key, "abcdef0123456789");
+        assert_eq!(
+            ev.status,
+            config::meta::alerts::incidents::ExternalAlertStatus::Firing
+        );
+        assert_eq!(
+            ev.severity,
+            config::meta::alerts::incidents::IncidentSeverity::P1
+        ); // critical
+        assert_eq!(ev.title, "HighCPU");
+        assert_eq!(ev.labels.get("namespace").unwrap(), "prod");
+        assert!(
+            !ev.labels.contains_key("job"),
+            "job label must be deny-listed"
+        );
+        assert_eq!(
+            ev.source_url.as_deref(),
+            Some("https://grafana.example/alerting/rule1")
+        );
+    }
+
+    #[test]
+    fn test_normalize_alertmanager_resolved_uses_endsat() {
+        let evs = normalize(DetectedSource::Alertmanager, &alertmanager_payload(), NOW).unwrap();
+        let ev = &evs[0];
+        assert_eq!(
+            ev.status,
+            config::meta::alerts::incidents::ExternalAlertStatus::Resolved
+        );
+        // 2026-07-30T10:30:00Z in micros
+        let expected = chrono::DateTime::parse_from_rfc3339("2026-07-30T10:30:00Z")
+            .unwrap()
+            .timestamp_micros();
+        assert_eq!(ev.event_ts, expected);
+    }
+
+    #[test]
+    fn test_event_ts_clamped_to_now_when_source_clock_is_wild() {
+        let mut p = grafana_payload();
+        p["alerts"][0]["startsAt"] = serde_json::json!("1999-01-01T00:00:00Z");
+        let evs = normalize(DetectedSource::Grafana, &p, NOW).unwrap();
+        assert_eq!(evs[0].event_ts, NOW - CLOCK_SKEW_TOLERANCE_MICROS);
+    }
+
+    #[test]
+    fn test_normalize_generic_object_and_array() {
+        let one = serde_json::json!({"status": "firing", "severity": "warning", "title": "db slow", "dedup_key": "k1", "labels": {"service": "db"}});
+        let evs = normalize(DetectedSource::Generic, &one, NOW).unwrap();
+        assert_eq!(evs.len(), 1);
+        assert_eq!(
+            evs[0].severity,
+            config::meta::alerts::incidents::IncidentSeverity::P3
+        );
+        assert_eq!(evs[0].event_ts, NOW); // no ts supplied → receipt time
+
+        let arr = serde_json::json!([one.clone(), {"status": "resolved", "title": "x", "labels": {"service": "db"}}]);
+        let evs = normalize(DetectedSource::Generic, &arr, NOW).unwrap();
+        assert_eq!(evs.len(), 2);
+        // no dedup_key on second → hash of sorted labels, deterministic
+        let evs2 = normalize(DetectedSource::Generic, &arr, NOW).unwrap();
+        assert_eq!(evs[1].dedup_key, evs2[1].dedup_key);
+        assert!(!evs[1].dedup_key.is_empty());
+    }
+
+    #[test]
+    fn test_unknown_format_errors() {
+        assert!(normalize(DetectedSource::Unknown, &serde_json::json!({"x": 1}), NOW).is_err());
+    }
+
+    fn event_with_label(source_label: Option<&str>) -> ExternalAlertEvent {
+        let mut labels = HashMap::new();
+        if let Some(s) = source_label {
+            labels.insert("source".to_string(), s.to_string());
+        }
+        labels.insert("alertname".to_string(), "test".to_string());
+        ExternalAlertEvent {
+            status: config::meta::alerts::incidents::ExternalAlertStatus::Firing,
+            dedup_key: "k".to_string(),
+            title: "t".to_string(),
+            severity: config::meta::alerts::incidents::IncidentSeverity::P3,
+            labels,
+            event_ts: 0,
+            source_url: None,
+            raw: serde_json::json!({}),
+        }
+    }
+
+    #[test]
+    fn test_derive_sender_label_present() {
+        let events = vec![event_with_label(Some("solarwinds"))];
+        assert_eq!(derive_sender_label(&events), Some("solarwinds".to_string()));
+    }
+
+    #[test]
+    fn test_derive_sender_label_absent() {
+        let events = vec![event_with_label(None)];
+        assert_eq!(derive_sender_label(&events), None);
+    }
+
+    #[test]
+    fn test_derive_sender_label_empty_string_treated_as_absent() {
+        let events = vec![event_with_label(Some(""))];
+        assert_eq!(derive_sender_label(&events), None);
+    }
+
+    #[test]
+    fn test_derive_sender_label_whitespace_only_treated_as_absent() {
+        let events = vec![event_with_label(Some("   "))];
+        assert_eq!(derive_sender_label(&events), None);
+    }
+
+    #[test]
+    fn test_derive_sender_label_trims_whitespace() {
+        let events = vec![event_with_label(Some("  solarwinds  "))];
+        assert_eq!(derive_sender_label(&events), Some("solarwinds".to_string()));
+    }
+
+    #[test]
+    fn test_derive_sender_label_empty_events_list() {
+        let events: Vec<ExternalAlertEvent> = vec![];
+        assert_eq!(derive_sender_label(&events), None);
+    }
+
+    #[test]
+    fn test_derive_sender_label_uses_first_event_only() {
+        let first = event_with_label(Some("first-sender"));
+        let mut second_labels = HashMap::new();
+        second_labels.insert("source".to_string(), "second-sender".to_string());
+        let second = ExternalAlertEvent {
+            labels: second_labels,
+            ..event_with_label(None)
+        };
+        let events = vec![first, second];
+        assert_eq!(
+            derive_sender_label(&events),
+            Some("first-sender".to_string())
+        );
+    }
+
+    #[test]
+    fn test_resolve_display_name_with_label() {
+        assert_eq!(
+            resolve_display_name("generic", Some("solarwinds")),
+            "solarwinds"
+        );
+    }
+
+    #[test]
+    fn test_resolve_display_name_without_label() {
+        assert_eq!(resolve_display_name("generic", None), "generic");
+    }
+
+    #[test]
+    fn test_resolve_display_name_empty_label_falls_back() {
+        assert_eq!(resolve_display_name("grafana", Some("")), "grafana");
+    }
+}

src/core/src/alerts/grouping.rs

@@ -33,11 +33,19 @@ static PENDING_BATCHES: Lazy<Arc<DashMap<String, PendingBatch>>> =
 #[derive(Clone, Debug)]
 pub struct PendingBatch {
     pub fingerprint: String,
+    /// Group identity for a per-group batch (§5.5). `None` for an ordinary
+    /// alert-level batch. Every entry shares the batch's fingerprint, and the
+    /// group is part of that fingerprint, so one batch is always one group.
+    pub group_labels: Option<std::collections::BTreeMap<String, String>>,
     pub org_id: String,
     pub alerts: Vec<BatchedAlert>,
     pub timer_started_at: i64,
     pub group_wait_seconds: i64,
     pub max_group_size: usize,
+    /// Evaluated level shared by every entry in this batch. Well-defined
+    /// because the fingerprint carries the level as an implicit component for
+    /// multi-level alerts — a Warning batch and a Critical batch are distinct.
+    pub level: Option<config::meta::alerts::level::AlertLevel>,
 }
 
 /// An alert waiting in a batch
@@ -57,10 +65,13 @@ impl PendingBatch {
         rows: Vec<json::Map<String, json::Value>>,
         group_wait_seconds: i64,
         max_group_size: usize,
+        level: Option<config::meta::alerts::level::AlertLevel>,
+        group_labels: Option<std::collections::BTreeMap<String, String>>,
     ) -> Self {
         let now = Utc::now().timestamp_micros();
         Self {
             fingerprint,
+            group_labels,
             org_id,
             alerts: vec![BatchedAlert {
                 alert,
@@ -70,6 +81,7 @@ impl PendingBatch {
             timer_started_at: now,
             group_wait_seconds,
             max_group_size,
+            level,
         }
     }
 
@@ -110,6 +122,11 @@ pub fn add_to_batch(
     rows: Vec<json::Map<String, json::Value>>,
     group_wait_seconds: i64,
     max_group_size: usize,
+    level: Option<config::meta::alerts::level::AlertLevel>,
+    // Group identity when this is a per-group batch (§5.5). Every entry
+    // sharing a fingerprint shares a group, because the group is part of the
+    // fingerprint — so this is set once, when the batch is created.
+    group_labels: Option<std::collections::BTreeMap<String, String>>,
 ) -> bool {
     let mut batch_ready = false;
     let mut is_new_batch = false;
@@ -161,6 +178,8 @@ pub fn add_to_batch(
                 rows,
                 group_wait_seconds,
                 max_group_size,
+                level,
+                group_labels,
             )
         });
 
@@ -378,6 +397,17 @@ pub async fn send_grouped_notification(
             rows_end_time,
             start_time,
             evaluation_timestamp,
+            // Well-defined for the whole batch: the fingerprint carries the
+            // level as an implicit component for multi-level alerts, so every
+            // entry classified the same. `{alert_level}` renders it.
+            batch.level,
+            // A batch aggregates several evaluations; no single exact count
+            // describes it — `{alert_count}` falls back to the row total.
+            None,
+            // Group identity for a batched per-group send (M-4). Every entry
+            // in a batch shares one fingerprint, and the group component is
+            // part of that fingerprint, so the whole batch is one group.
+            batch.group_labels.as_ref(),
         )
         .await
     {
@@ -450,6 +480,8 @@ mod tests {
             vec![],
             30,
             10,
+            None,
+            None,
         );
         assert_eq!(batch.alerts.len(), 1);
         assert!(!batch.is_full());
@@ -464,6 +496,8 @@ mod tests {
             vec![],
             30,
             2,
+            None,
+            None,
         );
         assert!(!batch.is_full());
         let added = batch.add_alert(make_alert(), vec![]);
@@ -480,6 +514,8 @@ mod tests {
             vec![],
             30,
             1,
+            None,
+            None,
         );
         assert!(batch.is_full());
         let added = batch.add_alert(make_alert(), vec![]);
@@ -496,6 +532,8 @@ mod tests {
             vec![],
             3600, // 1 hour wait
             10,
+            None,
+            None,
         );
         assert!(!batch.is_expired());
     }
@@ -512,6 +550,8 @@ mod tests {
             vec![],
             3600,
             10,
+            None,
+            None,
         );
         assert!(!ready);
         assert!(PENDING_BATCHES.contains_key(&fp));
@@ -531,6 +571,8 @@ mod tests {
             vec![],
             3600,
             2,
+            None,
+            None,
         );
         assert!(!ready1); // new batch, 1 alert, not full
 
@@ -541,6 +583,8 @@ mod tests {
             vec![],
             3600,
             2,
+            None,
+            None,
         );
         assert!(ready2); // 2nd alert fills batch, ready=true
 
@@ -559,6 +603,8 @@ mod tests {
             vec![],
             3600,
             10,
+            None,
+            None,
         );
 
         let batch = get_ready_batch(&fp);
@@ -575,8 +621,26 @@ mod tests {
         PENDING_BATCHES.remove(&fp1);
         PENDING_BATCHES.remove(&fp2);
 
-        add_to_batch(fp1.clone(), org.to_string(), make_alert(), vec![], 3600, 10);
-        add_to_batch(fp2.clone(), org.to_string(), make_alert(), vec![], 3600, 10);
+        add_to_batch(
+            fp1.clone(),
+            org.to_string(),
+            make_alert(),
+            vec![],
+            3600,
+            10,
+            None,
+            None,
+        );
+        add_to_batch(
+            fp2.clone(),
+            org.to_string(),
+            make_alert(),
+            vec![],
+            3600,
+            10,
+            None,
+            None,
+        );
 
         let count = get_pending_batch_count(org);
         assert!(count >= 2);

src/core/src/alerts/incidents.rs

@@ -23,7 +23,7 @@ use config::{
     meta::alerts::{
         alert::Alert,
         incidents::{
-            AlertEdge, AlertNode, CorrelationReason, EdgeType, Incident, IncidentAlert,
+            AlertEdge, AlertKind, AlertNode, CorrelationReason, EdgeType, Incident, IncidentAlert,
             IncidentCorrelationOutcome, IncidentEvent, IncidentTopology, IncidentWithAlerts,
         },
     },
@@ -43,6 +43,23 @@ struct FilteredSemanticResult {
     key_type: config::meta::alerts::incidents::KeyType,
 }
 
+/// Identity being correlated into an incident.
+///
+/// Decouples correlation/incident-creation from the concrete `Alert` type so that
+/// non-alert identities (e.g. external alert sources) can be correlated using the
+/// same machinery. For the internal alert path this is built from an `Alert`.
+#[derive(Debug, Clone)]
+pub struct CorrelationSubject {
+    /// internal: `alert.get_unique_key()`; external: `external_alerts.id`
+    pub id: String,
+    pub name: String,
+    pub org_id: String,
+    pub kind: config::meta::alerts::incidents::AlertKind,
+    pub base_destinations: Vec<String>,
+    /// Pre-mapped severity for external events; `None` → enterprise default (internal path)
+    pub severity: Option<config::meta::alerts::incidents::IncidentSeverity>,
+}
+
 /// Combined correlation result from both Service Discovery and semantic extraction
 struct ParallelCorrelationResult {
     service_discovery: Option<ServiceDiscoveryResult>,
@@ -64,10 +81,11 @@ async fn extract_filtered_semantic_dimensions(
     // Load ServiceIdentityConfig with auto-configuration applied
     let identity_config = crate::db::system_settings::get_service_identity_config(org_id).await;
 
-    // Validate config has at least one set
-    if identity_config.sets.is_empty() {
+    // No identity sets configured AND service is opted out — nothing to extract.
+    // Otherwise fall through: "service" alone is a viable dimension even with zero sets.
+    if identity_config.sets.is_empty() && identity_config.service_optional {
         log::debug!(
-            "[incidents] ServiceIdentityConfig for org {} has no identity sets, semantic extraction skipped",
+            "[incidents] ServiceIdentityConfig for org {} has no identity sets and service is optional, semantic extraction skipped",
             org_id
         );
         return None;
@@ -82,6 +100,12 @@ async fn extract_filtered_semantic_dimensions(
     for set in &identity_config.sets {
         all_distinguish_by.extend(set.distinguish_by.iter().cloned());
     }
+    // "service" is always a correlation dimension unless the org has explicitly
+    // opted out via service_optional (same toggle Service Discovery uses to match
+    // streams without requiring the service attribute).
+    if !identity_config.service_optional {
+        all_distinguish_by.push("service".to_string());
+    }
     // Remove duplicates and sort for deterministic processing
     all_distinguish_by.sort();
     all_distinguish_by.dedup();
@@ -299,13 +323,47 @@ async fn send_incident_notifications(
     event: &str,
     triggered_at: i64,
     dest_names: &[String],
+) {
+    // Preserve the alert/stream sub-object emitted for the internal alert path.
+    let alert_block = config::utils::json::json!({
+        "name": alert.name,
+        "stream": {
+            "name": alert.stream_name,
+            "type": alert.stream_type.to_string(),
+        }
+    });
+    send_incident_notifications_inner(
+        &alert.org_id,
+        &alert.name,
+        incident_id,
+        event,
+        triggered_at,
+        dest_names,
+        Some(alert_block),
+    )
+    .await;
+}
+
+/// Build an incident-specific notification payload and send to all given destinations.
+///
+/// Identity-agnostic core of [`send_incident_notifications`]. `alert_block`, when
+/// present, is emitted verbatim as the payload's `incident.alert` sub-object (used
+/// by the internal alert path to carry alert/stream metadata).
+#[cfg(feature = "enterprise")]
+#[allow(clippy::too_many_arguments)]
+async fn send_incident_notifications_inner(
+    org_id: &str,
+    subject_name: &str,
+    incident_id: &str,
+    event: &str,
+    triggered_at: i64,
+    dest_names: &[String],
+    alert_block: Option<Value>,
 ) {
     if dest_names.is_empty() {
         return;
     }
 
-    let org_id = alert.org_id.as_str();
-
     // Load incident to get severity, title and service_name.
     let (severity, title, service_name) =
         match infra::table::alert_incidents::get(org_id, incident_id).await {
@@ -332,6 +390,11 @@ async fn send_incident_notifications(
         .map(|dt: chrono::DateTime<chrono::Utc>| dt.format("%Y-%m-%dT%H:%M:%S%.3fZ").to_string())
         .unwrap_or_default();
 
+    // Internal path supplies a full alert/stream block; other subjects fall back to
+    // a name-only block derived from `subject_name`.
+    let alert_block =
+        alert_block.unwrap_or_else(|| config::utils::json::json!({ "name": subject_name }));
+
     let payload = config::utils::json::json!({
         "incident": {
             "id": incident_id,
@@ -339,13 +402,7 @@ async fn send_incident_notifications(
             "event": event,
             "service": service_name,
             "severity": severity,
-            "alert": {
-                "name": alert.name,
-                "stream": {
-                    "name": alert.stream_name,
-                    "type": alert.stream_type.to_string(),
-                }
-            },
+            "alert": alert_block,
             "time": time_str,
             "url": incident_url,
         }
@@ -482,6 +539,10 @@ pub async fn correlate_alert_to_incident(
     result_row: &Map<String, Value>,
     notify_rows: &[Map<String, Value>],
     triggered_at: i64,
+    // T-8: the evaluated level drives new-incident severity (Critical → P2,
+    // Warning → P3). None (manual triggers, single-level alerts) keeps the
+    // enterprise default.
+    eval_level: Option<config::meta::alerts::level::AlertLevel>,
 ) -> Result<Option<IncidentCorrelationOutcome>, anyhow::Error> {
     // Extract labels from result row as HashMap
     let mut labels: HashMap<String, String> = result_row
@@ -570,15 +631,26 @@ pub async fn correlate_alert_to_incident(
             sem_result.group_values
         );
     }
+    // Build the correlation subject for the internal alert path.
+    let subject = CorrelationSubject {
+        id: alert.get_unique_key(),
+        name: alert.name.clone(),
+        org_id: alert.org_id.to_string(),
+        kind: AlertKind::Internal,
+        base_destinations: alert.destinations.clone(),
+        severity: None,
+    };
+
     // Find or create incident
     let outcome = find_or_create_incident(
         &alert.org_id,
         &group_values,
         key_type,
-        alert,
+        &subject,
         triggered_at,
         &correlation_reason,
         &service_name,
+        eval_level,
     )
     .await?;
 
@@ -632,9 +704,11 @@ pub async fn correlate_alert_to_incident(
     if !notify_rows.is_empty() {
         match &outcome {
             IncidentCorrelationOutcome::NewIncidentCreated { incident_id, .. }
-            | IncidentCorrelationOutcome::NewAlertTypeJoined { incident_id, .. } => {
+            | IncidentCorrelationOutcome::NewAlertTypeJoined { incident_id, .. }
+            | IncidentCorrelationOutcome::SeverityEscalated { incident_id, .. } => {
                 let event = match &outcome {
                     IncidentCorrelationOutcome::NewIncidentCreated { .. } => "new_incident_created",
+                    IncidentCorrelationOutcome::SeverityEscalated { .. } => "severity_escalated",
                     _ => "new_alert_correlated",
                 };
                 let merged_destinations =
@@ -660,6 +734,175 @@ pub async fn correlate_alert_to_incident(
     Ok(Some(outcome))
 }
 
+/// External-event twin of [`correlate_alert_to_incident`].
+///
+/// Feeds an externally-ingested alert (from the External Alert Sources feature)
+/// through the same incident-correlation machinery as internal alerts. Labels are
+/// taken verbatim from the external record — there is no `query_condition` to
+/// enrich from, so the condition-dimension enrichment step is skipped.
+///
+/// `base_destinations` are the integration's configured destinations (for the org
+/// default, the default integration's config). Severity is parsed from the
+/// external record (falling back to `P3`) rather than resolved by the enterprise
+/// default used on the internal path.
+///
+/// Notification behaviour mirrors the internal path:
+/// - `NewIncidentCreated` / `NewAlertTypeJoined` → notify merged destinations.
+/// - `ExistingAlertRepeated` → notification suppressed (same subject already in incident).
+///
+/// The `#[cfg(feature = "cloud")]` AI-credit deduction from the internal path is
+/// intentionally NOT replicated here: it is structurally coupled to `&Alert` and
+/// P1 external events do not consume incident credits.
+pub async fn correlate_external_event(
+    org_id: &str,
+    external: &infra::table::external_alerts::ExternalAlertRecord,
+    base_destinations: Vec<String>,
+) -> Result<Option<IncidentCorrelationOutcome>, anyhow::Error> {
+    use config::meta::alerts::incidents::{IncidentSeverity, KeyType};
+
+    // Labels come straight from the external record (no query-condition enrichment).
+    let labels: HashMap<String, String> =
+        serde_json::from_value(external.labels.clone()).unwrap_or_default();
+
+    log::debug!(
+        "[incidents] External event '{}' labels: {:?}",
+        external.title,
+        labels
+    );
+
+    // Same parallel correlation the internal path uses.
+    let parallel_result = correlate_parallel(org_id, &labels).await;
+
+    let group_values = parallel_result.final_group_values;
+    let mut key_type = parallel_result.final_key_type;
+    let correlation_reason = parallel_result.correlation_reason;
+
+    // Extract service name using both results.
+    let service_name = extract_service_name_parallel(&labels, &parallel_result.service_discovery);
+
+    // If group_values is empty, isolate by subject id to prevent incorrect grouping.
+    if group_values.is_empty() {
+        key_type = KeyType::AlertId;
+        log::warn!(
+            "[incidents] External event '{}' has no group_values - isolated by alert_id",
+            external.title
+        );
+    }
+
+    log::info!(
+        "[incidents] External event '{}' correlation result: reason={}, key_type={:?}, dimensions={:?}, service_discovery_success={}, semantic_extraction_success={}",
+        external.title,
+        correlation_reason,
+        key_type,
+        group_values,
+        parallel_result.service_discovery.is_some(),
+        parallel_result.semantic_extraction.is_some()
+    );
+
+    // Build the correlation subject for the external path.
+    let subject = CorrelationSubject {
+        id: external.id.clone(),
+        name: external.title.clone(),
+        org_id: org_id.to_string(),
+        kind: AlertKind::External,
+        base_destinations,
+        severity: Some(external.severity.parse().unwrap_or(IncidentSeverity::P3)),
+    };
+
+    let triggered_at = external.last_seen_at;
+
+    // Find or create incident.
+    let outcome = find_or_create_incident(
+        org_id,
+        &group_values,
+        key_type,
+        &subject,
+        triggered_at,
+        &correlation_reason,
+        &service_name,
+        None, // external events carry no evaluated alert level
+    )
+    .await?;
+
+    // Send incident notification unless the outcome is a repeated alert (suppressed
+    // by design). External events always carry a single occurrence, so there is no
+    // empty-`notify_rows` manual-trigger case to guard against here.
+    match &outcome {
+        IncidentCorrelationOutcome::NewIncidentCreated { incident_id, .. }
+        | IncidentCorrelationOutcome::NewAlertTypeJoined { incident_id, .. }
+        | IncidentCorrelationOutcome::SeverityEscalated { incident_id, .. } => {
+            let event = match &outcome {
+                IncidentCorrelationOutcome::NewIncidentCreated { .. } => "new_incident_created",
+                IncidentCorrelationOutcome::SeverityEscalated { .. } => "severity_escalated",
+                _ => "new_alert_correlated",
+            };
+            let merged_destinations =
+                collect_incident_destinations(org_id, incident_id, &subject.base_destinations)
+                    .await;
+            send_incident_notifications_inner(
+                org_id,
+                &subject.name,
+                incident_id,
+                event,
+                triggered_at,
+                &merged_destinations,
+                None,
+            )
+            .await;
+        }
+        IncidentCorrelationOutcome::ExistingAlertRepeated { incident_id, .. } => {
+            log::debug!(
+                "[incidents] Suppressing notification for repeated external event in incident {incident_id}"
+            );
+        }
+    }
+
+    Ok(Some(outcome))
+}
+
+/// Auto-resolve the open incident containing `external.id`, but only once every
+/// other `External`-kind alert already linked to that incident is also resolved
+/// in `external_alerts` — a single source clearing shouldn't close an incident
+/// that other still-firing sources are correlated into.
+pub async fn try_auto_resolve_incident_for_external_alert(
+    org_id: &str,
+    external_alert_id: &str,
+) -> Result<(), anyhow::Error> {
+    let Some(incident) = infra::table::alert_incidents::find_open_incident_containing_alert(
+        org_id,
+        external_alert_id,
+    )
+    .await?
+    else {
+        return Ok(());
+    };
+
+    let links = infra::table::alert_incidents::get_incident_alerts(&incident.id).await?;
+    let external_alert_ids: Vec<String> = links
+        .into_iter()
+        .filter(|l| l.alert_kind == "external")
+        .map(|l| l.alert_id)
+        .collect();
+
+    if external_alert_ids.is_empty() {
+        return Ok(());
+    }
+
+    let records = infra::table::external_alerts::get_by_ids(org_id, &external_alert_ids).await?;
+    let all_resolved = records.iter().all(|r| r.state == "resolved");
+
+    if all_resolved {
+        update_status(org_id, &incident.id, "resolved", "system@openobserve.ai").await?;
+        log::info!(
+            "[incidents] Auto-resolved incident {} — all {} contributing external alert(s) resolved",
+            incident.id,
+            external_alert_ids.len()
+        );
+    }
+
+    Ok(())
+}
+
 /// Query Service Discovery for group_values using the correlation API
 ///
 /// Uses ServiceStorage::correlate() for proper dimension matching
@@ -780,19 +1023,31 @@ async fn create_new_incident(
     org_id: &str,
     group_values: &HashMap<String, String>,
     key_type: config::meta::alerts::incidents::KeyType,
-    alert: &Alert,
+    subject: &CorrelationSubject,
     triggered_at: i64,
     correlation_reason: &str,
     service_name: &str,
+    eval_level: Option<config::meta::alerts::level::AlertLevel>,
 ) -> Result<IncidentCorrelationOutcome, anyhow::Error> {
-    let severity = o2_enterprise::enterprise::alerts::incidents::determine_severity(None);
+    // Pre-mapped severity from the subject (external events) wins; otherwise
+    // T-8's evaluated level maps to incident severity — Critical opens a P2,
+    // Warning a P3. Without either (manual trigger, single-level alert) the
+    // enterprise default stands.
+    let severity = match (subject.severity, eval_level) {
+        (Some(s), _) => s.to_string(),
+        (None, Some(config::meta::alerts::level::AlertLevel::Critical)) => "P2".to_string(),
+        (None, Some(config::meta::alerts::level::AlertLevel::Warning)) => "P3".to_string(),
+        (None, _) => {
+            o2_enterprise::enterprise::alerts::incidents::determine_severity(None).to_string()
+        }
+    };
 
     let title =
-        o2_enterprise::enterprise::alerts::incidents::generate_title(&alert.name, group_values);
+        o2_enterprise::enterprise::alerts::incidents::generate_title(&subject.name, group_values);
 
     let incident = infra::table::alert_incidents::create(
         org_id,
-        severity,
+        &severity,
         serde_json::to_value(group_values)?,
         &key_type.to_string(),
         triggered_at,
@@ -825,8 +1080,9 @@ async fn create_new_incident(
     // Add the first alert to the incident
     infra::table::alert_incidents::add_alert_to_incident(
         &incident.id,
-        &alert.get_unique_key(),
-        &alert.name,
+        &subject.id,
+        &subject.name,
+        subject.kind.as_str(),
         triggered_at,
         correlation_reason,
     )
@@ -836,8 +1092,8 @@ async fn create_new_incident(
     if let Err(e) = infra::table::incident_events::record_alert(
         org_id,
         &incident.id,
-        &alert.get_unique_key(),
-        &alert.name,
+        &subject.id,
+        &subject.name,
         triggered_at,
     )
     .await
@@ -851,7 +1107,7 @@ async fn create_new_incident(
     log::info!(
         "[incidents] Created new incident {} for alert '{}' (key_type: {:?}, severity: {})",
         incident.id,
-        alert.name,
+        subject.name,
         key_type,
         severity
     );
@@ -880,7 +1136,7 @@ async fn create_new_incident(
         && let Err(e) = o2_enterprise::enterprise::super_cluster::queue::incidents_create(
             org_id,
             &key_type.to_string(),
-            severity,
+            &severity,
             serde_json::to_value(group_values)?,
             triggered_at,
             Some(title),
@@ -894,8 +1150,8 @@ async fn create_new_incident(
         org_id,
         &incident.id,
         service_name,
-        &alert.get_unique_key(),
-        &alert.name,
+        &subject.id,
+        &subject.name,
         triggered_at,
     );
 
@@ -967,16 +1223,17 @@ async fn find_or_create_incident(
     org_id: &str,
     group_values: &HashMap<String, String>,
     key_type: config::meta::alerts::incidents::KeyType,
-    alert: &Alert,
+    subject: &CorrelationSubject,
     triggered_at: i64,
     correlation_reason: &str,
     service_name: &str,
+    eval_level: Option<config::meta::alerts::level::AlertLevel>,
 ) -> Result<IncidentCorrelationOutcome, anyhow::Error> {
     use config::meta::alerts::incidents::{DimensionRelationship, KeyType};
 
     // STEP 1: AlertId exact match - check for existing incident with same alert_id
     if key_type == KeyType::AlertId {
-        let alert_id = alert.get_unique_key();
+        let alert_id = subject.id.clone();
 
         // Use the dedicated DB query: junction table lookup + incident fetch
         if let Some(incident) =
@@ -986,7 +1243,8 @@ async fn find_or_create_incident(
             let _ = infra::table::alert_incidents::add_alert_to_incident(
                 &incident.id,
                 &alert_id,
-                &alert.name,
+                &subject.name,
+                subject.kind.as_str(),
                 triggered_at,
                 correlation_reason,
             )
@@ -996,7 +1254,7 @@ async fn find_or_create_incident(
                 org_id,
                 &incident.id,
                 &alert_id,
-                &alert.name,
+                &subject.name,
                 triggered_at,
             )
             .await
@@ -1010,7 +1268,7 @@ async fn find_or_create_incident(
             if let Err(e) = crate::incidents::send_incident_event_trigger(
                 org_id,
                 &incident.id,
-                IncidentEvent::alert(&alert_id, &alert.name, triggered_at),
+                IncidentEvent::alert(&alert_id, &subject.name, triggered_at),
             )
             .await
             {
@@ -1025,10 +1283,66 @@ async fn find_or_create_incident(
                 &incident.id,
                 service_name,
                 &alert_id,
-                &alert.name,
+                &subject.name,
                 triggered_at,
             );
 
+            // T-8/§7.1: a repeat at HIGHER severity is an ESCALATION, not a
+            // repeat. A Warning-created P3 incident must upgrade to P2 and
+            // notify when the alert re-fires at Critical — the scheduler's
+            // silence layer explicitly let this delivery through, and
+            // suppressing it here would lose the only page for the
+            // escalation.
+            use config::meta::alerts::incidents::{IncidentEvent, IncidentSeverity};
+            let level_severity = eval_level.and_then(|l| match l {
+                config::meta::alerts::level::AlertLevel::Critical => Some(IncidentSeverity::P2),
+                config::meta::alerts::level::AlertLevel::Warning => Some(IncidentSeverity::P3),
+                _ => None,
+            });
+            // P1 is most urgent; higher urgency = escalation.
+            let urgency = |s: IncidentSeverity| match s {
+                IncidentSeverity::P1 => 4u8,
+                IncidentSeverity::P2 => 3,
+                IncidentSeverity::P3 => 2,
+                IncidentSeverity::P4 => 1,
+            };
+            if let Some(new_severity) = level_severity
+                && let Ok(current_severity) = incident.severity.parse::<IncidentSeverity>()
+                && urgency(new_severity) > urgency(current_severity)
+            {
+                infra::table::alert_incidents::update_severity(
+                    org_id,
+                    &incident.id,
+                    &new_severity.to_string(),
+                )
+                .await?;
+                if let Err(e) = infra::table::incident_events::append(
+                    org_id,
+                    &incident.id,
+                    IncidentEvent::severity_upgrade(
+                        current_severity,
+                        new_severity,
+                        format!("alert '{}' escalated to {}", subject.name, new_severity),
+                    ),
+                )
+                .await
+                {
+                    log::error!(
+                        "[Incidents] Failed to record severity-upgrade event for incident {}: {e}",
+                        incident.id
+                    );
+                }
+                log::info!(
+                    "[Incidents] Incident {} escalated {current_severity} -> {new_severity} by alert '{}'",
+                    incident.id,
+                    subject.name
+                );
+                return Ok(IncidentCorrelationOutcome::SeverityEscalated {
+                    incident_id: incident.id,
+                    service_name: service_name.to_string(),
+                });
+            }
+
             return Ok(IncidentCorrelationOutcome::ExistingAlertRepeated {
                 incident_id: incident.id,
                 service_name: service_name.to_string(),
@@ -1069,8 +1383,9 @@ async fn find_or_create_incident(
 
             let is_new_alert_type = infra::table::alert_incidents::add_alert_to_incident(
                 &existing.id,
-                &alert.get_unique_key(),
-                &alert.name,
+                &subject.id,
+                &subject.name,
+                subject.kind.as_str(),
                 triggered_at,
                 correlation_reason,
             )
@@ -1079,8 +1394,8 @@ async fn find_or_create_incident(
             if let Err(e) = infra::table::incident_events::record_alert(
                 org_id,
                 &existing.id,
-                &alert.get_unique_key(),
-                &alert.name,
+                &subject.id,
+                &subject.name,
                 triggered_at,
             )
             .await
@@ -1095,7 +1410,7 @@ async fn find_or_create_incident(
                 && let Err(e) = crate::incidents::send_incident_event_trigger(
                     org_id,
                     &existing.id,
-                    IncidentEvent::alert(alert.get_unique_key(), &alert.name, triggered_at),
+                    IncidentEvent::alert(&subject.id, &subject.name, triggered_at),
                 )
                 .await
             {
@@ -1158,8 +1473,8 @@ async fn find_or_create_incident(
                     o2_enterprise::enterprise::super_cluster::queue::incidents_add_alert(
                         org_id,
                         &existing.id,
-                        &alert.get_unique_key(),
-                        &alert.name,
+                        &subject.id,
+                        &subject.name,
                         triggered_at,
                         correlation_reason,
                     )
@@ -1172,8 +1487,8 @@ async fn find_or_create_incident(
                 org_id,
                 &existing.id,
                 service_name,
-                &alert.get_unique_key(),
-                &alert.name,
+                &subject.id,
+                &subject.name,
                 triggered_at,
             );
 
@@ -1246,10 +1561,11 @@ async fn find_or_create_incident(
         org_id,
         group_values,
         key_type,
-        alert,
+        subject,
         triggered_at,
         correlation_reason,
         service_name,
+        eval_level,
     )
     .await
 }
@@ -1286,12 +1602,13 @@ pub async fn get_incident_with_alerts(
     }
 
     // Convert incident_alerts to triggers
-    let triggers: Vec<IncidentAlert> = incident_alerts
+    let mut triggers: Vec<IncidentAlert> = incident_alerts
         .iter()
         .map(|a| IncidentAlert {
             incident_id: a.incident_id.clone(),
             alert_id: a.alert_id.clone(),
             alert_name: a.alert_name.clone(),
+            alert_kind: AlertKind::from_stored(&a.alert_kind),
             alert_fired_at: a.alert_fired_at,
             correlation_reason: a
                 .correlation_reason
@@ -1299,9 +1616,48 @@ pub async fn get_incident_with_alerts(
                 .and_then(|r| CorrelationReason::try_from(r.as_str()).ok())
                 .unwrap_or(CorrelationReason::AlertId),
             created_at: a.created_at,
+            source_url: None,
+            labels: None,
+            detected_source: None,
         })
         .collect();
 
+    // Enrich external-kind triggers with data from the external_alerts table.
+    // Internal alerts are left untouched (their details are hydrated below via
+    // the live-fetch path).
+    let external_ids: Vec<String> = triggers
+        .iter()
+        .filter(|t| t.alert_kind == AlertKind::External)
+        .map(|t| t.alert_id.clone())
+        .collect();
+
+    if !external_ids.is_empty() {
+        match infra::table::external_alerts::get_by_ids(&incident_org_id, &external_ids).await {
+            Ok(records) => {
+                let records_by_id: std::collections::HashMap<_, _> =
+                    records.into_iter().map(|r| (r.id.clone(), r)).collect();
+
+                for trigger in triggers
+                    .iter_mut()
+                    .filter(|t| t.alert_kind == AlertKind::External)
+                {
+                    if let Some(record) = records_by_id.get(&trigger.alert_id) {
+                        trigger.source_url = record.source_url.clone();
+                        trigger.labels = serde_json::from_value(record.labels.clone()).ok();
+                        trigger.detected_source = Some(record.detected_source.clone());
+                    }
+                }
+            }
+            Err(e) => {
+                log::warn!(
+                    "[incidents] Failed to fetch external alert details for incident {}: {}",
+                    incident_id,
+                    e
+                );
+            }
+        }
+    }
+
     // Get unique alert names from triggers
     let unique_alert_names: std::collections::HashSet<String> = incident_alerts
         .iter()

src/core/src/alerts/mod.rs

@@ -18,11 +18,11 @@ use arrow_schema::{DataType, Schema};
 use async_trait::async_trait;
 use chrono::{Duration, Utc};
 use config::{
-    TIMESTAMP_COL_NAME, ider,
+    TIMESTAMP_COL_NAME, get_config, ider,
     meta::{
         alerts::{
             AggFunction, AlertConditionParams, Condition, ConditionList, Operator, QueryCondition,
-            QueryType, TriggerCondition, TriggerEvalResults,
+            QueryType, TriggerCondition, TriggerEvalResults, grouping::GroupObservation,
         },
         cluster::RoleGroup,
         search::{SearchEventContext, SearchEventType, SqlQuery},
@@ -46,6 +46,7 @@ pub mod backfill;
 pub mod deduplication;
 pub mod derived_streams;
 pub mod destinations;
+pub mod external_alerts;
 #[cfg(feature = "enterprise")]
 pub mod grouping;
 #[cfg(feature = "enterprise")]
@@ -147,6 +148,16 @@ impl QueryConditionExt for QueryCondition {
                     v.to_string()
                 }
             }
+            QueryType::Slo => {
+                // An SLO alert runs no query. It reads the running aggregate
+                // the ingest pass already computed, which is why five alerts
+                // on one SLO cost five cheap status reads and ZERO extra
+                // raw-data scans (§6b.9). The caller branches before reaching
+                // here (`alert.rs`); this arm exists so the dispatch stays
+                // exhaustive and a mis-routed SLO alert degrades to "nothing
+                // matched" rather than running an empty SQL string.
+                return Ok(eval_results);
+            }
             QueryType::PromQL => {
                 let Some(v) = self.promql.as_ref() else {
                     return Ok(eval_results);
@@ -165,16 +176,39 @@ impl QueryConditionExt for QueryCondition {
                 };
                 let end = end_time;
                 let condition = self.promql_condition.as_ref().unwrap();
-                let req = promql_service::MetricsQueryRequest {
-                    query: format!(
+                // Multi-level PromQL (alerts_2.md §4.4, same strategy as the
+                // SQL HAVING): query at the LESS severe value so the warning
+                // band comes back too, then classify each series below.
+                // Single-level alerts widen to critical, i.e. the expression is
+                // byte-identical to before.
+                let promql_critical = to_float(&condition.value);
+                let promql_filter = config::meta::alerts::aggregation_level::widened_threshold(
+                    condition.operator,
+                    promql_critical,
+                    self.promql_warning_value,
+                );
+                // Observation completeness (M-11), the PromQL mirror of the
+                // SQL multi path dropping its HAVING: with the threshold in
+                // the expression, a recovered series just stops being
+                // returned and could only recover K evaluations late via the
+                // reaper, with a NULL value. Per-series alerts therefore run
+                // the raw expression and classify in Rust; single alerts keep
+                // the filtered query unchanged.
+                let query = if self.promql_multi_alert {
+                    format!("({v})")
+                } else {
+                    format!(
                         "({}) {} {}",
                         v,
                         match &condition.operator {
                             &Operator::EqualTo => "==".to_string(),
                             _ => condition.operator.to_string(),
                         },
-                        to_float(&condition.value)
-                    ),
+                        promql_filter
+                    )
+                };
+                let req = promql_service::MetricsQueryRequest {
+                    query,
                     start,
                     end,
                     step: std::cmp::max(
@@ -205,15 +239,18 @@ impl QueryConditionExt for QueryCondition {
                 .await
                 {
                     Ok(v) => v,
-                    Err(_) => {
-                        return Ok(eval_results);
+                    // A failed search is an ERROR, not an empty result. Returning
+                    // Ok here would record outcome=Normal/level=Ok and refresh
+                    // `level_at`, silently clearing a prior Critical (§7.6 —
+                    // errors must leave the level axis untouched).
+                    Err(e) => {
+                        return Err(anyhow::anyhow!("PromQL search error for alert query: {e}"));
                     }
                 };
                 let config::meta::promql::value::Value::Matrix(value) = resp else {
-                    log::warn!(
-                        "Alert evaluate: trace_id: {trace_id}, PromQL query {v} returned unexpected response: {resp:?}"
-                    );
-                    return Ok(eval_results);
+                    return Err(anyhow::anyhow!(
+                        "PromQL query returned unexpected (non-matrix) response: {resp:?}"
+                    ));
                 };
                 let values: Vec<_> =
                     value
@@ -231,16 +268,70 @@ impl QueryConditionExt for QueryCondition {
                         })
                         .collect();
 
-                let threshold = trigger_condition.threshold as usize;
-                eval_results.data = match trigger_condition.operator {
-                    Operator::EqualTo => (values.len() == threshold).then_some(values),
-                    Operator::NotEqualTo => (values.len() != threshold).then_some(values),
-                    Operator::GreaterThan => (values.len() > threshold).then_some(values),
-                    Operator::GreaterThanEquals => (values.len() >= threshold).then_some(values),
-                    Operator::LessThan => (values.len() < threshold).then_some(values),
-                    Operator::LessThanEquals => (values.len() <= threshold).then_some(values),
-                    _ => None,
-                };
+                // Two axes, exactly like aggregation: each SERIES is classified
+                // against the promql condition value, then the SERIES COUNT is
+                // gated by `trigger_condition`. Counting series alone would
+                // ignore severity and fire spuriously on the widened set.
+                let series_values: Vec<f64> = values
+                    .iter()
+                    .filter_map(|v| v.get("value").and_then(|x| x.as_f64()))
+                    .collect();
+                let level = config::meta::alerts::aggregation_level::evaluate_level_over_items(
+                    &series_values,
+                    condition.operator,
+                    promql_critical,
+                    self.promql_warning_value,
+                    trigger_condition,
+                );
+                // Worst series' value, so history reports one coherent
+                // observation rather than a bare series count. Direction is
+                // operator-aware: for `<`/`<=` the worst offender is the MIN.
+                //
+                // KNOWN LIMITATION (§7.5) — single alerts only: their filter
+                // is widened just to the warning level, so a healthy run
+                // returns no series and records actual_value=None — history
+                // shows "— → Ok". Per-series alerts run unfiltered (above)
+                // and record the real healthy reading.
+                eval_results.actual_value = config::meta::alerts::level::worst_observed_value(
+                    &series_values,
+                    condition.operator,
+                );
+                // T-9: label the worst SERIES by its PromQL labels, so history
+                // shows which series the value came from.
+                eval_results.group_label = eval_results.actual_value.and_then(|w| {
+                    values
+                        .iter()
+                        .find(|v| v.get("value").and_then(|x| x.as_f64()) == Some(w))
+                        .map(|v| {
+                            v.iter()
+                                .filter(|(k, _)| k != &"_timestamp" && k != &"value")
+                                .map(|(k, val)| match val.as_str() {
+                                    Some(s) => format!("{k}={s}"),
+                                    None => format!("{k}={val}"),
+                                })
+                                .collect::<Vec<_>>()
+                                .join(",")
+                        })
+                        .filter(|label| !label.is_empty())
+                });
+                // ── Per-series fan-out (M-1/M-2/M-3, gated by M-9) ──────────
+                // Additive, exactly like the aggregation path above: the
+                // worst-series collapse computed already is what the single
+                // per-evaluation trigger record needs (D8); this only ADDS the
+                // per-group view.
+                if self.promql_multi_alert {
+                    eval_results.group_classification =
+                        Some(config::meta::alerts::grouping::classify_promql_series(
+                            &values,
+                            condition.operator,
+                            promql_critical,
+                            self.promql_warning_value,
+                            get_config().limit.alert_max_groups,
+                        ));
+                }
+
+                eval_results.level = level;
+                eval_results.data = level.map(|_| values);
                 log::info!(
                     "Alert evaluate: trace_id: {trace_id}, PromQL query {v} returned response after filtering: {eval_results:?}"
                 );
@@ -283,10 +374,76 @@ impl QueryConditionExt for QueryCondition {
         } else {
             Some(end_time - time_diff)
         };
+        // Hybrid count evaluation (alerts_2.md §4.4c). Guards, in order:
+        //  - threshold bypass (search_event_type) — no threshold, no hybrid;
+        //  - aggregation — already exact, needs per-group rows;
+        //  - VRL — transforms rows post-query, a SQL count could disagree;
+        //  - multi-window — separate SQL list, out of scope v1.
+        let hybrid = self.search_event_type.is_none()
+            && self.aggregation.is_none()
+            && self.vrl_function.is_none()
+            && self
+                .multi_time_range
+                .as_ref()
+                .is_none_or(|mtr| mtr.is_empty())
+            && matches!(
+                config::meta::alerts::level::evaluation_strategy(
+                    trigger_condition,
+                    get_config().limit.alert_hybrid_count_threshold,
+                ),
+                config::meta::alerts::level::EvaluationStrategy::CountPlusSample { .. }
+            );
+
+        // Exact count from the COUNT(*) pre-query; also carries the decision.
+        let mut hybrid_exact_count: Option<f64> = None;
+        if hybrid {
+            let count_resp = run_alert_count_query(
+                &trace_id,
+                org_id,
+                stream_type,
+                &sql,
+                (start_time.unwrap_or(end_time - time_diff), end_time),
+                search_type,
+                search_event_context.clone(),
+            )
+            .await?;
+            let exact = count_resp.0;
+            eval_results.query_took = Some(count_resp.1);
+            eval_results.actual_value = Some(exact);
+            let level = config::meta::alerts::level::evaluate_level(exact, trigger_condition);
+            eval_results.level = level;
+            if level.is_none() {
+                // Not firing: the count alone decides. No payload query at all
+                // — a healthy hybrid evaluation is CHEAPER than the old
+                // 100-row floor fetch.
+                return Ok(eval_results);
+            }
+            hybrid_exact_count = Some(exact);
+        }
+
+        // Per-group evaluation reads a page sized to the M-6 cap, not to the
+        // threshold: for a multi-alert the count gate is always "any group"
+        // (M-10), so `required_search_size` would ask for a handful of rows and
+        // the fan-out would see a fraction of the groups.
+        let multi_group_cap = self
+            .aggregation
+            .as_ref()
+            .filter(|a| a.multi_alert && a.group_by.as_ref().is_some_and(|g| !g.is_empty()))
+            .map(|_| get_config().limit.alert_max_groups);
+
         let size = if self.search_event_type.is_some() {
             -1
+        } else if let Some(cap) = multi_group_cap {
+            // ONE row past the cap, so a full page is itself the overflow
+            // signal (M-6) and the persisted counts can be marked lower bounds
+            // honestly (§5.3). `cap == 0` means unlimited.
+            if cap == 0 { -1 } else { cap as i64 + 1 }
+        } else if hybrid {
+            // Decision already made from the exact count; this fetch is only
+            // the notification payload sample.
+            config::meta::alerts::level::PAYLOAD_SAMPLE_ROWS
         } else {
-            std::cmp::max(100, trigger_condition.threshold)
+            config::meta::alerts::level::required_search_size(trigger_condition)
         };
 
         let req_start = std::time::Instant::now();
@@ -521,17 +678,170 @@ impl QueryConditionExt for QueryCondition {
             Some(search_event_type) => search_event_type == SearchEventType::Alerts,
         };
         eval_results.data = if apply_threshold {
-            let threshold = trigger_condition.threshold as usize;
-            match trigger_condition.operator {
-                Operator::EqualTo => (records.len() == threshold).then_some(records),
-                Operator::NotEqualTo => (records.len() != threshold).then_some(records),
-                Operator::GreaterThan => (records.len() > threshold).then_some(records),
-                Operator::GreaterThanEquals => (records.len() >= threshold).then_some(records),
-                Operator::LessThan => (records.len() < threshold).then_some(records),
-                Operator::LessThanEquals => (records.len() <= threshold).then_some(records),
-                _ => None,
+            match self.aggregation.as_ref() {
+                // ── Aggregation alerts ──────────────────────────────────────
+                // The threshold is `having.value` / `warning_value` applied to
+                // each row's aggregate, NOT a row count. The SQL HAVING was
+                // widened to the less severe threshold (alerts_2.md §4.4), so
+                // the returned set deliberately includes the warning band and
+                // MUST be re-classified here — counting rows would both ignore
+                // severity and fire spuriously on the widened set.
+                Some(agg) => {
+                    // Classify each group's aggregate, then re-apply the
+                    // GROUP-COUNT threshold. Both axes must hold — dropping the
+                    // count silently rewrites "for at least 3 groups" as "for
+                    // any group".
+                    let classified: Vec<_> = records
+                        .iter()
+                        .filter_map(|r| r.get("alert_agg_value").and_then(|v| v.as_f64()))
+                        .collect();
+
+                    let level =
+                        config::meta::alerts::aggregation_level::evaluate_aggregation_alert(
+                            &classified,
+                            agg,
+                            trigger_condition,
+                        )
+                        .unwrap_or(None);
+
+                    // Report the worst group's value, so history's
+                    // "fired at X against Y" is one coherent observation.
+                    // Direction is operator-aware: for `<`/`<=` the worst
+                    // offender is the MIN, not the max.
+                    //
+                    // KNOWN LIMITATION (§7.5): the HAVING filter is widened
+                    // only to the warning level, so a healthy run returns no
+                    // rows and records actual_value=None — history shows
+                    // "— → Ok". Dropping the filter would cost a full
+                    // per-group fetch on every healthy evaluation;
+                    // deliberately deferred to the SLO work.
+                    let offenders: Vec<f64> = classified
+                        .iter()
+                        .filter(|v| {
+                            config::meta::alerts::aggregation_level::evaluate_aggregation_level(
+                                **v, agg,
+                            )
+                            .ok()
+                            .flatten()
+                            .is_some()
+                        })
+                        .cloned()
+                        .collect();
+                    let worst = config::meta::alerts::level::worst_observed_value(
+                        &offenders,
+                        agg.having.operator,
+                    );
+
+                    // T-9: identify WHICH group produced the worst value, so
+                    // history reads "avg(cpu)=97.2 for host=b" and not just a
+                    // number. Label = the group_by columns of that row.
+                    let group_label = worst.and_then(|w| {
+                        let group_by = agg.group_by.as_deref().unwrap_or(&[]);
+                        if group_by.is_empty() {
+                            return None;
+                        }
+                        records
+                            .iter()
+                            .find(|r| r.get("alert_agg_value").and_then(|v| v.as_f64()) == Some(w))
+                            .map(|r| {
+                                group_by
+                                    .iter()
+                                    .filter_map(|col| {
+                                        r.get(col).map(|v| match v.as_str() {
+                                            Some(s) => format!("{col}={s}"),
+                                            None => format!("{col}={v}"),
+                                        })
+                                    })
+                                    .collect::<Vec<_>>()
+                                    .join(",")
+                            })
+                            .filter(|label| !label.is_empty())
+                    });
+
+                    eval_results.level = level;
+                    eval_results.actual_value = worst;
+                    eval_results.group_label = group_label;
+
+                    // ── Per-group fan-out (M-1/M-2/M-3, gated by M-9) ───────
+                    // Purely additive: everything above still runs, because the
+                    // worst-group collapse is what the single per-evaluation
+                    // trigger record needs (D8) and what every non-multi alert
+                    // is evaluated by. This only *adds* the per-group view.
+                    if let Some(cap) = multi_group_cap
+                        && let Some(group_by) = agg.group_by.as_ref()
+                    {
+                        // Labels come from the SHARED extractor, not a local
+                        // copy: dispatch keys each group's notification
+                        // payload by `group_key(row_group_labels(row))`, so if
+                        // the two renderings ever diverged, every dispatch
+                        // item would fail to find its row and the feature
+                        // would break silently.
+                        let observations: Vec<GroupObservation> = records
+                            .iter()
+                            .filter_map(|r| {
+                                let value = r.get("alert_agg_value")?.as_f64()?;
+                                let labels =
+                                    config::meta::alerts::dispatch::row_group_labels(r, group_by);
+                                Some(GroupObservation::new(labels, value))
+                            })
+                            .collect();
+
+                        let classification = config::meta::alerts::grouping::classify_groups_by(
+                            observations,
+                            |v| {
+                                config::meta::alerts::aggregation_level::evaluate_aggregation_level(
+                                    v, agg,
+                                )
+                                .ok()
+                                .flatten()
+                            },
+                            cap,
+                        );
+
+                        // Both facts come free from the classification: the
+                        // page filled if we got everything we asked for, and it
+                        // reached healthy groups if not every observed group
+                        // was firing. Together they decide whether the counts
+                        // are exact and whether absence proves disappearance.
+                        let observed = classification.groups.len() + classification.dropped.len();
+                        let page = config::meta::alerts::grouping::FetchPage {
+                            filled: size > 0 && records.len() as i64 >= size,
+                            reached_healthy: classification.firing_observed < observed,
+                        };
+
+                        eval_results.group_classification = Some(classification.with_page(page));
+                    }
+
+                    level.map(|_| records)
+                }
+                // ── Count-based alerts ──────────────────────────────────────
+                None => {
+                    // Hybrid mode already decided from COUNT(*) — and it only
+                    // reaches here when firing, so `data` is Some
+                    // unconditionally. Re-deriving from the 100-row payload
+                    // sample would silently overwrite the exact count with a
+                    // clamped one.
+                    if let Some(exact) = hybrid_exact_count {
+                        eval_results.actual_value = Some(exact);
+                        // level already set from the exact count
+                        Some(records)
+                    } else {
+                        let actual = records.len() as f64;
+                        let level =
+                            config::meta::alerts::level::evaluate_level(actual, trigger_condition);
+                        eval_results.actual_value = Some(actual);
+                        // The fetch was capped at `size`; a full page means the
+                        // true count may be higher — record it as a lower
+                        // bound so history can render "≥ N" (§7.5).
+                        eval_results.value_is_lower_bound =
+                            size > 0 && records.len() as i64 >= size;
+                        eval_results.level = level;
+                        level.map(|_| records)
+                    }
+                }
             }
         } else {
+            // Threshold bypassed (non-alert search event types) — no level.
             Some(records)
         };
 
@@ -539,6 +849,72 @@ impl QueryConditionExt for QueryCondition {
     }
 }
 
+/// Run the §4.4c COUNT(*) decision query for a hybrid count-based alert.
+///
+/// Returns `(exact_count, query_took_ms)`. The user's SQL runs verbatim inside
+/// the wrapper, over the same time window the payload query would use, so the
+/// two cannot disagree about which rows exist.
+#[allow(clippy::too_many_arguments)]
+async fn run_alert_count_query(
+    trace_id: &str,
+    org_id: &str,
+    stream_type: StreamType,
+    sql: &str,
+    (start_time, end_time): (i64, i64),
+    search_type: Option<SearchEventType>,
+    search_event_context: Option<SearchEventContext>,
+) -> Result<(f64, i64), anyhow::Error> {
+    let req = config::meta::search::Request {
+        query: config::meta::search::Query {
+            sql: config::meta::alerts::level::count_query_sql(sql),
+            from: 0,
+            // COUNT(*) over a subquery yields exactly one row.
+            size: 1,
+            start_time,
+            end_time,
+            quick_mode: false,
+            query_type: "".to_string(),
+            track_total_hits: false,
+            action_id: None,
+            uses_zo_fn: false,
+            query_fn: None, // guard upstream: hybrid excludes VRL alerts
+            skip_wal: false,
+            sampling_config: None,
+            sampling_ratio: None,
+            streaming_output: false,
+            streaming_id: None,
+            histogram_interval: 0,
+            timezone: None,
+        },
+        encoding: config::meta::search::RequestEncoding::Empty,
+        regions: vec![],
+        clusters: vec![],
+        timeout: 0,
+        search_type,
+        search_event_context,
+        use_cache: false,
+        clear_cache: false,
+        local_mode: None,
+        agent_options: None,
+    };
+    let resp = SearchService::grpc_search::grpc_search(
+        trace_id,
+        org_id,
+        stream_type,
+        None,
+        &req,
+        Some(RoleGroup::Background),
+    )
+    .await?;
+    let count = resp
+        .hits
+        .first()
+        .and_then(|h| h.get("zo_alert_count"))
+        .and_then(|v| v.as_f64().or_else(|| v.as_i64().map(|i| i as f64)))
+        .ok_or_else(|| anyhow::anyhow!("alert count query returned no zo_alert_count column"))?;
+    Ok((count, resp.took as i64))
+}
+
 #[async_trait]
 pub trait ConditionListExt: Sync + Send + 'static {
     async fn len(&self) -> u32;
@@ -1077,7 +1453,22 @@ pub async fn build_sql(
                 ));
             }
         };
-        build_expr(&agg.having, "alert_agg_value", data_type)?
+        // Multi-level aggregations (alerts_2.md §4.4, option B): widen the
+        // HAVING clause to the LESS severe threshold so every group that could
+        // be warning-or-worse comes back, then classify each group in Rust via
+        // the shared helper. Filtering on the critical threshold would drop the
+        // entire warning band inside the database, where nothing downstream
+        // could recover it.
+        //
+        // Single-level aggregations widen to the critical value, i.e. the
+        // clause is byte-identical to before.
+        let filter_value = config::meta::alerts::aggregation_level::having_filter_value(agg)
+            .map_err(|e| anyhow::anyhow!("Invalid aggregation threshold: {e}"))?;
+        let widened = Condition {
+            value: serde_json::json!(filter_value),
+            ..agg.having.clone()
+        };
+        build_expr(&widened, "alert_agg_value", data_type)?
     };
 
     let func_expr = match agg.function {
@@ -1112,11 +1503,30 @@ pub async fn build_sql(
     if let Some(group) = agg.group_by.as_ref()
         && !group.is_empty()
     {
-        sql = format!(
-            "SELECT {}, {func_expr} AS alert_agg_value, MIN({TIMESTAMP_COL_NAME}) as zo_sql_min_time, MAX({TIMESTAMP_COL_NAME}) AS zo_sql_max_time FROM \"{stream_name}\"{where_sql} GROUP BY {} HAVING {having_expr}",
-            group.join(", "),
-            group.join(", "),
-        );
+        let cols = group.join(", ");
+        if agg.multi_alert {
+            // Multi-alerts (M-9) drop the HAVING filter. A group that falls
+            // back under the threshold must still be RETURNED: otherwise its
+            // recovery is indistinguishable from it vanishing, and it would
+            // only resolve via M-7's timeout — K evaluations late, and with a
+            // NULL value where the real reading should be.
+            //
+            // The page stays bounded, so it is ordered worst-first (§5.3).
+            // That ordering is what keeps the rollup level exact and lets the
+            // M-6 cap admit the true top of the distribution rather than an
+            // arbitrary slice. The group columns are the deterministic
+            // tiebreak within a severity band.
+            let severity_order =
+                config::meta::alerts::aggregation_level::severity_order_sql(agg, "alert_agg_value")
+                    .map_err(|e| anyhow::anyhow!("Invalid aggregation threshold: {e}"))?;
+            sql = format!(
+                "SELECT {cols}, {func_expr} AS alert_agg_value, MIN({TIMESTAMP_COL_NAME}) as zo_sql_min_time, MAX({TIMESTAMP_COL_NAME}) AS zo_sql_max_time FROM \"{stream_name}\"{where_sql} GROUP BY {cols} ORDER BY {severity_order}, {cols}"
+            );
+        } else {
+            sql = format!(
+                "SELECT {cols}, {func_expr} AS alert_agg_value, MIN({TIMESTAMP_COL_NAME}) as zo_sql_min_time, MAX({TIMESTAMP_COL_NAME}) AS zo_sql_max_time FROM \"{stream_name}\"{where_sql} GROUP BY {cols} HAVING {having_expr}"
+            );
+        }
     }
     if sql.is_empty() {
         sql = format!(

src/core/src/alerts/scheduler/worker.rs

@@ -609,6 +609,18 @@ fn resolve_module_configs(
             concurrency: pick_concurrency(cfg.limit.scheduler_anomaly_concurrency),
             poll_interval_secs: pick_interval(cfg.limit.scheduler_anomaly_interval),
         },
+        ModuleSchedulerConfig {
+            module: TriggerModule::Slo,
+            concurrency: pick_concurrency(cfg.limit.scheduler_slo_concurrency),
+            poll_interval_secs: pick_interval(cfg.limit.scheduler_slo_interval),
+        },
+        ModuleSchedulerConfig {
+            // Its own lane on purpose: a bulk historical scan sharing a
+            // budget with incremental SLI passes would starve them (§6b.9).
+            module: TriggerModule::SloBackfill,
+            concurrency: std::cmp::max(1, cfg.limit.scheduler_slo_backfill_concurrency),
+            poll_interval_secs: pick_interval(cfg.limit.scheduler_slo_backfill_interval),
+        },
         ModuleSchedulerConfig {
             module: TriggerModule::QueryRecommendations,
             concurrency: pick_concurrency(cfg.limit.scheduler_query_reco_concurrency),
@@ -749,10 +761,10 @@ mod tests {
         // never be pulled once per-module pullers are enabled.
         let cfg = config::Config::default();
         let configs = resolve_module_configs(&cfg, &make_config());
-        assert_eq!(configs.len(), 6);
+        assert_eq!(configs.len(), 8);
         let modules: std::collections::HashSet<_> =
             configs.iter().map(|c| c.module.clone()).collect();
-        assert_eq!(modules.len(), 6, "duplicate module in resolved configs");
+        assert_eq!(modules.len(), 8, "duplicate module in resolved configs");
         for m in [
             TriggerModule::Alert,
             TriggerModule::Report,
@@ -760,6 +772,8 @@ mod tests {
             TriggerModule::Backfill,
             TriggerModule::AnomalyDetection,
             TriggerModule::QueryRecommendations,
+            TriggerModule::Slo,
+            TriggerModule::SloBackfill,
         ] {
             assert!(modules.contains(&m), "missing module {m:?}");
         }
@@ -779,6 +793,7 @@ mod tests {
             TriggerModule::DerivedStream,
             TriggerModule::AnomalyDetection,
             TriggerModule::QueryRecommendations,
+            TriggerModule::Slo,
         ] {
             let c = find_module(&configs, m.clone());
             assert_eq!(c.concurrency, 3, "{m:?} should inherit base concurrency");
@@ -794,6 +809,13 @@ mod tests {
             backfill.poll_interval_secs, 10,
             "backfill inherits the alert pull frequency"
         );
+
+        // SLO backfill floors to 1 for the same reason the alert backfill
+        // does: a bulk historical scan must never crowd out the incremental
+        // passes that alerts read from.
+        let slo_backfill = find_module(&configs, TriggerModule::SloBackfill);
+        assert_eq!(slo_backfill.concurrency, 1, "slo backfill floors to 1");
+        assert_eq!(slo_backfill.poll_interval_secs, 10);
     }
 
     #[test]
@@ -805,6 +827,8 @@ mod tests {
         cfg.limit.scheduler_anomaly_concurrency = 4;
         cfg.limit.scheduler_query_reco_concurrency = 2;
         cfg.limit.scheduler_backfill_concurrency = 3;
+        cfg.limit.scheduler_slo_concurrency = 5;
+        cfg.limit.scheduler_slo_backfill_concurrency = 2;
         let configs = resolve_module_configs(&cfg, &base);
 
         // Alert has no dedicated var: it reuses the base (ZO_ALERT_SCHEDULE_CONCURRENCY).
@@ -826,6 +850,11 @@ mod tests {
             find_module(&configs, TriggerModule::Backfill).concurrency,
             3
         );
+        assert_eq!(find_module(&configs, TriggerModule::Slo).concurrency, 5);
+        assert_eq!(
+            find_module(&configs, TriggerModule::SloBackfill).concurrency,
+            2
+        );
     }
 
     #[test]
@@ -838,6 +867,8 @@ mod tests {
         cfg.limit.scheduler_derived_stream_interval = 120;
         cfg.limit.scheduler_backfill_interval = 60;
         cfg.limit.scheduler_anomaly_interval = 30;
+        cfg.limit.scheduler_slo_interval = 45;
+        cfg.limit.scheduler_slo_backfill_interval = 300;
         // query_reco left at 0 → inherits the alert pull frequency (10)
         let configs = resolve_module_configs(&cfg, &base);
 
@@ -862,6 +893,14 @@ mod tests {
             find_module(&configs, TriggerModule::AnomalyDetection).poll_interval_secs,
             30
         );
+        assert_eq!(
+            find_module(&configs, TriggerModule::Slo).poll_interval_secs,
+            45
+        );
+        assert_eq!(
+            find_module(&configs, TriggerModule::SloBackfill).poll_interval_secs,
+            300
+        );
         assert_eq!(
             find_module(&configs, TriggerModule::QueryRecommendations).poll_interval_secs,
             10,

src/core/src/anomaly_detection.rs

@@ -60,6 +60,32 @@ pub struct CreateAnomalyConfigRequest {
     pub enabled: Option<bool>,
     pub folder_id: Option<String>,
     pub owner: Option<String>,
+    /// Triage priority P1..P5 (Feature 2, PT-1). Anomaly configs appear in the
+    /// same alert list as scheduled/realtime alerts, so they carry the same
+    /// metadata. Absent = unset.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[schema(value_type = Option<u8>, example = 3)]
+    pub priority: Option<config::meta::alerts::priority::AlertPriority>,
+    /// Selection tags (PT-6), normalized and validated on save exactly as for
+    /// alerts — one `normalize_tags` serves both.
+    #[serde(default)]
+    pub tags: Vec<String>,
+}
+
+/// Deserializer that keeps "absent" and "explicit null" apart for a
+/// `Option<Option<T>>` field.
+///
+/// Serde's default collapses both to `None`: a plain `Option<Option<T>>`
+/// deserializes `null` to the OUTER `None`, so "clear this value" becomes
+/// indistinguishable from "field not supplied". That silently made priority
+/// unclearable through the direct anomaly endpoint; this restores the
+/// distinction (`null` -> `Some(None)`).
+fn double_option<'de, T, D>(deserializer: D) -> Result<Option<Option<T>>, D::Error>
+where
+    T: serde::Deserialize<'de>,
+    D: serde::Deserializer<'de>,
+{
+    serde::Deserialize::deserialize(deserializer).map(Some)
 }
 
 #[derive(Debug, Default, Serialize, Deserialize, ToSchema)]
@@ -82,6 +108,25 @@ pub struct UpdateAnomalyConfigRequest {
     pub enabled: Option<bool>,
     pub folder_id: Option<String>,
     pub owner: Option<String>,
+    /// Double-option so "absent" and "explicit null" stay distinguishable
+    /// (same shape as `TimedAnnotationUpdate::end_time`):
+    ///   * `None`             — field not supplied, leave the stored value
+    ///   * `Some(None)`       — clear the priority
+    ///   * `Some(Some(p))`    — set it
+    ///
+    /// A plain `Option` cannot express "clear", which made priority the only
+    /// field on an anomaly config that could be set but never unset — and
+    /// inconsistent with alerts, where clearing works.
+    #[serde(
+        default,
+        deserialize_with = "double_option",
+        skip_serializing_if = "Option::is_none"
+    )]
+    #[schema(value_type = Option<u8>, example = 3)]
+    pub priority: Option<Option<config::meta::alerts::priority::AlertPriority>>,
+    /// `None` leaves stored tags untouched; `Some(vec![])` clears them.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub tags: Option<Vec<String>>,
 }
 
 /// Resolve a folder name (e.g. "default") to the PK stored in `folders.id`.
@@ -316,6 +361,12 @@ pub async fn create_config(
     // Validate request
     validate_config_request(&req)?;
 
+    // Feature 2 (PT-7): same normalization the alerts path uses, so a tag
+    // means the same thing on both. Kept typed, not stringified, so the API
+    // layer can downcast it to a 400.
+    let normalized_tags =
+        config::meta::alerts::tags::normalize_tags(&req.tags).map_err(anyhow::Error::new)?;
+
     let db = ORM_CLIENT
         .get()
         .ok_or_else(|| anyhow::anyhow!("Database not initialized"))?;
@@ -386,6 +437,16 @@ pub async fn create_config(
         ),
         folder_id: folder_pk,
         owner: req.owner.clone(),
+        // Feature 2. Tags are NORMALIZED here, not merely validated, so the
+        // stored form matches what the alerts table stores and one filter
+        // compares like with like. NULL rather than 0/[] when unset, so an
+        // existing config's row is unchanged.
+        priority: req.priority.map(|p| p.to_i32()),
+        tags: if normalized_tags.is_empty() {
+            None
+        } else {
+            Some(serde_json::json!(normalized_tags))
+        },
         status: 0i32, // 0 = waiting
         retries: 0,
         last_updated: now_us,
@@ -610,6 +671,24 @@ pub async fn update_config(
     if let Some(owner) = req.owner {
         active_model.owner = Set(Some(owner));
     }
+    // Feature 2. `None` means "not supplied, leave as-is"; an explicit value
+    // replaces it. Tags go through the same normalization as the alerts path,
+    // so an edit cannot smuggle in a form the filter will never match.
+    // `Some(None)` clears, `Some(Some(_))` sets, `None` leaves alone — so a
+    // partial update (e.g. enable/disable) cannot wipe the priority.
+    if let Some(priority) = req.priority {
+        active_model.priority = Set(priority.map(|p| p.to_i32()));
+    }
+    if let Some(tags) = req.tags {
+        // Typed for the same reason as on create: the API downcasts to 400.
+        let normalized =
+            config::meta::alerts::tags::normalize_tags(&tags).map_err(anyhow::Error::new)?;
+        active_model.tags = Set(if normalized.is_empty() {
+            None // an explicit empty list clears the tags
+        } else {
+            Some(serde_json::json!(normalized))
+        });
+    }
 
     active_model.updated_at = Set(Utc::now().timestamp_micros());
 
@@ -895,6 +974,11 @@ pub async fn clone_config(
         alert_destinations: src.alert_destinations.clone(),
         folder_id: resolved_folder_id,
         owner: src.owner.clone(),
+        // Feature 2: a clone inherits the original's triage metadata —
+        // copying an alert that is P1/tagged and silently dropping both
+        // would hand back something that looks configured but is not.
+        priority: src.priority,
+        tags: src.tags.clone(),
         status: 0i32,
         retries: 0,
         last_updated: now_us,
@@ -1853,6 +1937,84 @@ pub async fn send_anomaly_alert(
 
 #[cfg(test)]
 mod tests {
+    // ── Feature 2: priority & tags on anomaly configs ───────────────────────
+
+    /// Tags round-trip through the create request, and an absent `tags` key
+    /// yields an empty list rather than failing — every pre-Feature-2 client
+    /// omits it.
+    #[test]
+    fn test_create_request_tags_default_to_empty() {
+        let without: CreateAnomalyConfigRequest = serde_json::from_str(
+            r#"{"name":"a","stream_name":"s","stream_type":"logs","query_mode":"filters",
+                "detection_function":"count","histogram_interval":"5m",
+                "schedule_interval":"15m","detection_window_seconds":3600}"#,
+        )
+        .unwrap();
+        assert!(without.tags.is_empty());
+        assert_eq!(without.priority, None);
+    }
+
+    /// The create path must store the NORMALIZED form, so a tag means the same
+    /// thing on an anomaly config as on an alert and one filter matches both.
+    #[test]
+    fn test_anomaly_tags_normalize_exactly_like_alert_tags() {
+        let raw = vec![
+            "  PROD  ".to_string(),
+            "Service:Checkout".to_string(),
+            "prod".to_string(),
+            "".to_string(),
+        ];
+        let normalized = config::meta::alerts::tags::normalize_tags(&raw).unwrap();
+        assert_eq!(normalized, vec!["prod", "service:checkout"]);
+    }
+
+    /// Invalid tags are rejected on anomaly configs too, naming the offender —
+    /// the same contract the alerts path has.
+    #[test]
+    fn test_anomaly_invalid_tag_is_rejected_and_named() {
+        let err = config::meta::alerts::tags::normalize_tags(&["1bad".to_string()]).unwrap_err();
+        assert!(err.to_string().contains("1bad"), "got: {err}");
+    }
+
+    /// Priority ids must be the SAME as the alerts table's, since one column
+    /// mapping and one enum serve both. If these ever diverge, a P2 anomaly
+    /// and a P2 alert would store different integers.
+    #[test]
+    fn test_anomaly_priority_ids_match_the_alert_scale() {
+        use config::meta::alerts::priority::AlertPriority;
+        for (p, id) in [
+            (AlertPriority::P1, 1),
+            (AlertPriority::P2, 2),
+            (AlertPriority::P3, 3),
+            (AlertPriority::P4, 4),
+            (AlertPriority::P5, 5),
+        ] {
+            assert_eq!(p.to_i32(), id);
+        }
+    }
+
+    /// PROBE: does serde distinguish "absent" from "explicit null" for the
+    /// double option? Written first because the answer decides whether the
+    /// clear-via-null path works, or whether only the Rust-constructed
+    /// `Some(None)` from the v2 handler does.
+    #[test]
+    fn test_update_request_priority_absent_vs_null() {
+        let absent: UpdateAnomalyConfigRequest = serde_json::from_str("{}").unwrap();
+        let null: UpdateAnomalyConfigRequest =
+            serde_json::from_str(r#"{"priority": null}"#).unwrap();
+        let set: UpdateAnomalyConfigRequest = serde_json::from_str(r#"{"priority": 3}"#).unwrap();
+        assert_eq!(absent.priority, None, "absent must leave the value alone");
+        assert_eq!(
+            null.priority,
+            Some(None),
+            "explicit null must mean CLEAR, distinct from absent"
+        );
+        assert_eq!(
+            set.priority,
+            Some(Some(config::meta::alerts::priority::AlertPriority::P3))
+        );
+    }
+
     use super::*;
     // ── combine_detection_fn ────────────────────────────────────────────────
 
@@ -1938,6 +2100,8 @@ mod tests {
             enabled: None,
             folder_id: None,
             owner: None,
+            priority: None,
+            tags: vec![],
         }
     }
 

src/core/src/auth.rs

@@ -1061,6 +1061,11 @@ mod tests {
         // If we reach here, the function completed successfully
     }
 
+    // Gated to match what it asserts. Without this the test runs under
+    // `--features enterprise` too, where `get_role` returns the mapped role
+    // rather than collapsing to Admin — so it failed the entire enterprise
+    // test build on a claim it never made about enterprise.
+    #[cfg(not(feature = "enterprise"))]
     #[test]
     fn test_get_role_non_enterprise() {
         let user_role = UserOrgRole {

src/core/src/http.rs

@@ -69,6 +69,15 @@ impl From<AlertError> for Response {
             | AlertError::AlertDestinationMissing
             | AlertError::TemplateNotConfigured { .. }
             | AlertError::RealtimeMissingCustomQuery
+            // Both are user input errors -> 400, same as the other
+            // trigger-condition validations.
+            | AlertError::InvalidWarningThreshold(_)
+            | AlertError::InvalidAggregationThreshold(_)
+            | AlertError::InvalidMultiAlert(_)
+            | AlertError::WarningThresholdOnRealtimeAlert
+            | AlertError::WarningOnCoverageGate { .. }
+            | AlertError::PromqlWarningWithoutCondition
+            | AlertError::InvalidTag(_)
             | AlertError::SqlMissingQuery
             | AlertError::SqlContainsSelectStar
             | AlertError::PromqlMissingQuery

src/core/src/ingestion/mod.rs

@@ -29,7 +29,7 @@ use config::{
     ider::SnowflakeIdGenerator,
     meta::{
         alerts::alert::Alert,
-        self_reporting::usage::{RequestStats, TriggerData, TriggerDataStatus, TriggerDataType},
+        self_reporting::usage::{RequestStats, RunOutcome, TriggerData, TriggerDataType},
         stream::{PartitionTimeLevel, StreamParams, StreamPartition, StreamType},
     },
     utils::{flatten, json::*, schema::format_partition_key},
@@ -150,7 +150,7 @@ pub async fn evaluate_trigger(triggers: TriggerAlertData) {
             next_run_at: now,
             is_realtime: true,
             is_silenced: false,
-            status: TriggerDataStatus::Completed,
+            status: RunOutcome::Firing,
             start_time: now,
             end_time: 0,
             retries: 0,
@@ -173,12 +173,12 @@ pub async fn evaluate_trigger(triggers: TriggerAlertData) {
             alert.name
         );
         match alert
-            .send_notification(&trace_id, val, now, None, now)
+            .send_notification(&trace_id, val, now, None, now, None, None, None)
             .await
         {
             Err(e) => {
                 log::error!("Failed to send notification: {e}");
-                trigger_data_stream.status = TriggerDataStatus::Failed;
+                trigger_data_stream.status = RunOutcome::NotifyFailed;
                 trigger_data_stream.error =
                     Some(format!("error sending notification for alert: {e}"));
             }

src/core/src/lib.rs

@@ -58,6 +58,7 @@ pub mod self_reporting;
 pub mod service;
 pub mod session;
 pub mod short_url;
+pub mod slo;
 pub mod stream;
 pub mod stream_utils;
 pub mod synthetics;

src/core/src/llm_evaluations/eval_jobs/mod.rs

@@ -361,6 +361,8 @@ pub async fn manual_evaluate(
                 o2_enterprise::enterprise::llm_evaluations::eval_jobs::tasks::EvaluationQueryWindow {
                     start_us: body.start_time,
                     end_us: body.end_time,
+                    // manual evaluations use the caller's window verbatim
+                    ingest_cutoff_us: None,
                 },
             reason: body
                 .reason

src/core/src/logs/bulk.rs

@@ -20,7 +20,7 @@ use std::{
 
 use axum::body::Bytes;
 #[cfg(feature = "cloud")]
-use config::meta::self_reporting::usage::is_reserved_self_reporting_stream;
+use config::meta::self_reporting::usage::is_reserved_internal_stream;
 use config::{
     BLOCKED_STREAMS, TIMESTAMP_COL_NAME, get_config,
     meta::stream::StreamType,
@@ -150,7 +150,7 @@ pub async fn ingest(
             // non-bulk `IngestionRequest::Usage` channel), so this is safe.
             // Cloud-only: OSS / self-hosted may legitimately use these names.
             #[cfg(feature = "cloud")]
-            if is_reserved_self_reporting_stream(&stream_name) {
+            if is_reserved_internal_stream(&stream_name) {
                 let err_msg =
                     format!("stream '{stream_name}' is reserved and cannot be ingested into");
                 log::warn!("[LOGS:BULK] {err_msg}");