test(editor): fix the flaky CodeQueryEditor specs that dequeued the PR

Nine tests in this file failed about half the time — including on a clean
origin/main worktree, which is why main's own Unit tests workflow has been red
on most recent commits. They took the PR out of the merge queue.

The cause was a mock one function short of the component. setupEditor's promql
branch calls, in order:

  monaco.languages.register({ id: "promql" })
  monaco.languages.setMonarchTokensProvider("promql", ...)
  monaco.languages.setLanguageConfiguration("promql", ...)   <-- not mocked

so the "should support promql language" mount threw mid-setup. The vitest config
sets dangerouslyIgnoreUnhandledErrors, so nothing reported it; what showed
instead was later tests in the file hanging on vi.waitFor(addCommand) and timing
out at 5000ms. Whether they hung depended on how those aborted mounts
interleaved with live ones, which is what made it bimodal: ~52ms or never.

Adding setLanguageConfiguration takes it from roughly one run in two to 8/8
clean, and the full suite passes.

Also replaced the `vi.spyOn(document, "getElementById")` in three describes with
a real <div id="test-editor"> attached to the document. setupEditor retries that
lookup five times on a 100ms timer and then gives up WITHOUT reaching
addCommand, and the spy made "does this mount find its element" depend on mock
lifecycle across three describes while ~50 mounts from earlier describes were
still polling on their own timers. That alone did not fix the flake — measured,
2 of 5 runs still failed — but it removes a whole class of ordering dependency
from a file that had just demonstrated it has one.

.claude/skills/ui-architect/references/calm-signal.md

@@ -58,8 +58,30 @@ All token-backed and dark-mode-safe. Reuse these before inventing anything.
   Status chip beside them. Pass `size` at the call site, matching its siblings.
 - **Row state signal** — an extreme-left colour **rail** via `OTable`'s
   per-row `getRowStyle` (inset box-shadow, rem width, token colour) + a **light
-  exception highlight** via `row-class` (tint only the rows that need action —
-  never the normal ones).
+  exception highlight** via `row-class`. The rail and the wash are two different
+  strengths of the same signal, and the rail is the default:
+
+  | | Rail (`getRowStyle`) | Wash (`row-class`) |
+  | --- | --- | --- |
+  | Cost | a few px at the row edge | ~the whole row |
+  | Use for | **every** state, always | only the two cases below |
+
+  A full-row wash is the loudest thing on a list, so it earns its place only when
+  **all three** hold: the state means **act now** (not merely "not green"), it is
+  **rare in a healthy system**, and **the row is the unit of action**. In practice
+  that leaves exactly two washes:
+  - `!bg-status-error-bg` — failing/errored/offline. *Alerts* failed, *Pipelines*
+    errored, *Nodes* offline, *Synthetics* failed.
+  - `!bg-surface-panel` — paused/disabled. This one is **de-emphasis, not alarm**:
+    the row is deliberately inert, so it recedes rather than shouts.
+
+  Everything else keeps a clean row and reads from the rail — including states
+  that are "bad but not urgent": **degraded/warning** (worth noticing, not worth
+  acting on this second), **stale**, **unknown**, **never-ran**. And whatever the
+  state means, if it is **common** the wash is wrong regardless: *Streams* drops it
+  entirely, because "never ingested" and "quiet for a day" describe a large share
+  of rows in a normal org and a table where most rows are tinted signals nothing.
+  Pages with no failure state at all — Users, Roles, Dashboards — never wash.
 - **Recency** — `OTimeCell` `mode="relative"` (`"3 min ago"`) with a hot/warm/
   cold dot, instead of a raw timestamp column.
 - **People** — `OUserCell` for owner/author columns.
@@ -156,6 +178,24 @@ people to ignore the colour. When in doubt, grey.
 
 Colour only earns attention if most of the screen stays quiet:
 
+- **Earn every tile — a strip is not a page decoration.** Before adding one, put
+  each tile to three questions: does the number **vary**, does someone **act** on
+  it, and is it **not already on screen**? Tiles that fail are noise dressed as
+  signal:
+  - *structurally constant* — "System accounts" is 1 in almost every org, so the
+    tile is a label with a number stuck to it;
+  - *almost always zero* — "New this week" on a list that gains an item a quarter;
+  - *derivable from its neighbours* — "In use" beside "Unused" and "Total";
+  - *already in the footer* — `footerTitle` renders "N Dashboards" under every
+    table, so a Total tile alone is not a reason to have a strip.
+  A page whose only candidates fail these gets **no strip** — keep the per-row
+  signals (relative recency, a state rail, a count column) and stop. Dashboards,
+  Service Accounts and Roles all ended up here: pages where a strip added pixels
+  and no information. Roles is the clearest case — "Unused" is just the member
+  column sorted ascending, so two tiles restated what the rows already said. **A
+  count column plus sorting usually beats a strip**; reach for a strip only when
+  the page has a real distribution to summarise (Users across roles, monitors
+  across health) *and* the tiles double as the facet.
 - **Highlight exceptions, not the norm.** Tint the failed/paused rows; leave the
   healthy majority clean. A table where every row is coloured signals nothing.
   And if a page has no true failure state (a catalog list), the calm answer is a

.github/scripts/build-ci-matrix.js

@@ -0,0 +1,114 @@
+#!/usr/bin/env node
+/**
+ * build-ci-matrix.js — single source of truth for the Playwright UI shard matrix.
+ *
+ * Prints a GitHub Actions matrix object: {"include":[ {testfolder,actual_folder,browser,run_files}, ... ]}
+ * consumed by `strategy.matrix: ${{ fromJSON(needs.generate_matrix.outputs.matrix) }}`.
+ *
+ * Usage:
+ *   node build-ci-matrix.js <base.json>                 # OSS: base manifest verbatim
+ *   node build-ci-matrix.js <base.json> <overlay.json>  # ENT: base + enterprise overlay
+ *
+ * The base (OSS tests/ui-testing/ci-matrix/ci_matrix.json) is the ONLY place shared shards are
+ * listed. ENT never re-lists shared specs — its overlay (ci_matrix.ent.json) carries
+ * only the delta, so a spec added to OSS flows into ENT automatically:
+ *   { "append": { "<testfolder>": ["extra.spec.js", ...] },   // add ENT-only specs to a shared shard
+ *     "shards": [ {testfolder,actual_folder,browser,run_files} ], // whole ENT-only shards (Workflows, SDR…)
+ *     "disabled": { "<testfolder>": [{file,reason}, ...] } }   // ENT-only specs turned off (doc only)
+ *
+ * DISABLING A SPEC (JSON has no // comments): don't delete it — move it into a
+ * shard's "disabled": [{ "file": "x.spec.js", "reason": "why" }] array. Disabled
+ * entries are documentation only: this script never emits them, so they don't run,
+ * but the record + reason survive. "_comment" (or any _-prefixed key) is also ignored.
+ *
+ * This lives in OSS so ENT can reuse it from its tree-merged OSS checkout.
+ */
+const fs = require("fs");
+
+function die(msg) {
+  process.stderr.write(`build-ci-matrix: ${msg}\n`);
+  process.exit(1);
+}
+
+function readJson(path) {
+  try {
+    return JSON.parse(fs.readFileSync(path, "utf8"));
+  } catch (e) {
+    die(`cannot read/parse ${path}: ${e.message}`);
+  }
+}
+
+const [basePath, overlayPath] = process.argv.slice(2);
+if (!basePath) die("usage: build-ci-matrix.js <base.json> [overlay.json]");
+
+const base = readJson(basePath);
+if (!Array.isArray(base)) die(`base ${basePath} must be a JSON array of shards`);
+
+// Deep-copy base so we never mutate the parsed source objects.
+const include = base.map((s) => ({ ...s, run_files: [...(s.run_files || [])] }));
+const byFolder = new Map(include.map((s) => [s.testfolder, s]));
+
+if (overlayPath) {
+  const overlay = readJson(overlayPath);
+
+  // append: add enterprise-only specs to an EXISTING shared shard.
+  for (const [folder, specs] of Object.entries(overlay.append || {})) {
+    const shard = byFolder.get(folder);
+    if (!shard) {
+      die(`overlay append targets unknown folder "${folder}" — it must exist in the base manifest`);
+    }
+    for (const spec of specs) {
+      if (shard.run_files.includes(spec)) {
+        die(`overlay append: "${spec}" already in base shard "${folder}" — remove it from the overlay`);
+      }
+      shard.run_files.push(spec);
+    }
+  }
+
+  // shards: whole enterprise-only shards (Workflows, SDR-Logs, SDR-Traces, …).
+  for (const shard of overlay.shards || []) {
+    if (byFolder.has(shard.testfolder)) {
+      die(`overlay shard "${shard.testfolder}" collides with a base shard — use "append" instead`);
+    }
+    const copy = { browser: "chrome", ...shard, run_files: [...(shard.run_files || [])] };
+    include.push(copy);
+    byFolder.set(copy.testfolder, copy);
+  }
+
+  // disabled: enterprise-only turned-off specs, recorded against a shard for docs only.
+  for (const [folder, entries] of Object.entries(overlay.disabled || {})) {
+    const shard = byFolder.get(folder);
+    if (!shard) die(`overlay disabled targets unknown folder "${folder}"`);
+    shard.disabled = [...(shard.disabled || []), ...entries];
+  }
+}
+
+// Sanity: unique testfolders, no empty shards, no dup specs, and no spec both
+// active (run_files) and disabled in the same shard.
+const seen = new Set();
+for (const s of include) {
+  if (!s.testfolder) die(`shard missing testfolder: ${JSON.stringify(s)}`);
+  if (seen.has(s.testfolder)) die(`duplicate testfolder "${s.testfolder}"`);
+  seen.add(s.testfolder);
+  if (!s.actual_folder) s.actual_folder = s.testfolder;
+  if (!s.run_files || s.run_files.length === 0) die(`shard "${s.testfolder}" has no run_files`);
+  if (new Set(s.run_files).size !== s.run_files.length) {
+    die(`shard "${s.testfolder}" has duplicate run_files`);
+  }
+  for (const d of s.disabled || []) {
+    if (s.run_files.includes(d.file)) {
+      die(`shard "${s.testfolder}": "${d.file}" is in both run_files and disabled`);
+    }
+  }
+}
+
+// Emit ONLY the fields the CI matrix consumes — disabled/_comment/notes are stripped,
+// so turned-off specs never reach `npx playwright test`.
+const matrix = include.map((s) => ({
+  testfolder: s.testfolder,
+  actual_folder: s.actual_folder,
+  browser: s.browser || "chrome",
+  run_files: s.run_files,
+}));
+
+process.stdout.write(JSON.stringify({ include: matrix }));

.github/workflows/playwright.yml

@@ -176,10 +176,36 @@ jobs:
           retention-days: 1
           if-no-files-found: error
 
+  # Build the shard matrix from tests/ui-testing/ci-matrix/ci_matrix.json (single source of truth,
+  # shared with ENT via build-ci-matrix.js). Runs under the same gate as build_binary so
+  # ui_integration_tests always has a matrix whenever it is allowed to run.
+  generate_matrix:
+    timeout-minutes: 5
+    name: generate_matrix
+    needs: [check_changes]
+    if: >-
+      (github.event_name == 'pull_request' &&
+       contains(github.event.pull_request.labels.*.name, 'e2e')) ||
+      (github.event_name != 'pull_request' &&
+       needs.check_changes.outputs.has_changes == 'true')
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    outputs:
+      matrix: ${{ steps.gen.outputs.matrix }}
+    steps:
+      - uses: actions/checkout@v5
+      - id: gen
+        run: |
+          MATRIX=$(node .github/scripts/build-ci-matrix.js tests/ui-testing/ci-matrix/ci_matrix.json)
+          if [ -z "$MATRIX" ]; then echo "::error::matrix generation failed (build-ci-matrix produced no output)"; exit 1; fi
+          echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT"
+          echo "$MATRIX" | node -e "const m=JSON.parse(require('fs').readFileSync(0,'utf8')); console.log('generated',m.include.length,'shards:', m.include.map(s=>s.testfolder).join(', '))"
+
   ui_integration_tests:
     timeout-minutes: 45
     name: e2e / ${{ matrix.testfolder }}
-    needs: [build_binary]
+    needs: [build_binary, generate_matrix]
     if: ${{ !cancelled() && needs.build_binary.result == 'success' }}
     runs-on:
       labels: eks-openobserve-standard-8
@@ -191,328 +217,10 @@ jobs:
     #   options: --user root
     strategy:
       fail-fast: false
-      matrix:
-        include:
-          - testfolder: "GeneralTests"
-            browser: "chrome"
-            run_files:
-              [
-                "sanity.spec.js",
-                "changeOrg.spec.js",
-                "enrichment.spec.js",
-                "schema.spec.js",
-                "schemaload.spec.js",
-                "serviceAccount.spec.js",
-                "usersOrg.spec.js",
-                "org.spec.js",
-                "theme-management.spec.js",
-                "ingestion-config.spec.js",
-                "ingestionTokens.spec.js",
-                "landingPage.spec.js",
-                "languageTranslation.spec.js",
-                "model-pricing.spec.js",
-                "edition-features.spec.js",
-                "logsqueries.sql-comments.spec.js",
-                "iam-form-validation.spec.js",
-                "regex-patterns-form-validation.spec.js",
-                "cipher-keys-form-validation.spec.js",
-                "settings-form-validation.spec.js",
-                "ai-toolsets-form-validation.spec.js",
-                "shared-components-form-validation.spec.js",
-                "onboarding-form-validation.spec.js",
-                "rum-form-validation.spec.js",
-                "login-form-validation.spec.js",
-              ]
-          - testfolder: "Logs-Builder-Basic"
-            browser: "chrome"
-            run_files:
-              [
-                "logsQueryBuilder-chart.spec.js",
-                "logsQueryBuilder-filters-basic.spec.js",
-                "pagination.spec.js",
-                "unflattened.spec.js",
-              ]
-          - testfolder: "Logs-Builder-Advanced"
-            browser: "chrome"
-            run_files:
-              [
-                "logsQueryBuilder-editor.spec.js",
-                "logsQueryBuilder-filters-advanced.spec.js",
-                "logsquickmode.spec.js",
-                "logshistogram.spec.js",
-                "logsQuickPick.spec.js",
-              ]
-          - testfolder: "Logs-Core"
-            browser: "chrome"
-            run_files:
-              [
-                "logspage.spec.js",
-                "logstable.spec.js",
-                "logsqueries.spec.js",
-                "join.spec.js",
-                "shareLink.spec.js",
-                "logsAnalyzeDimensions.spec.js",
-                "monaco-query-prefill.spec.js",
-                "logsqueries.matchall.spec.js",
-                "ftsDefaultColumn.spec.js",
-                "logsVisualizePersistence.spec.js",
-              ]
-          - testfolder: "Logs-Features"
-            browser: "chrome"
-            run_files: [
-                "logs-autocomplete-suggestions.spec.js",
-                # "searchJobInspector.spec.js",  # TODO: update these tests for new correlation feature
-                "logs-sql-autocomplete.spec.js",
-                "logsqueries.cte.spec.js",
-                "logsDownloads.spec.js",
-                "secondsPrecisionAdded.spec.js",
-                "searchpartition.spec.js",
-                "indexquery.spec.js",
-                "region.spec.js",
-              ]
-          - testfolder: "Alerts"
-            browser: "chrome"
-            run_files: [
-                # "alerts-e2e-flow.spec.js",
-                "alerts-ui-operations.spec.js",
-                "alerts-import.spec.js",
-                # "alerts-advanced.spec.js",
-                # "alerts-scheduled-features.spec.js",
-                "alerts-metrics-notification.spec.js",
-                "alerts-destinations-prebuilt.spec.js",
-                "alerts-vrl-encoding.spec.js",
-                "alerts-history.spec.js",
-                "alerts-template-prebuilt-guard.spec.js",
-                "alerts-form-validation.spec.js",
-                "anomaly-detection-form-validation.spec.js",
-                # Alerts 4.0 (multi-alerts) e2e — PR #13547
-                "alerts-priority-tags.spec.js",
-                "alerts-multialert-api.spec.js",
-                "alerts-multialert-ui.spec.js",
-                "alerts-multialert-regression.spec.js",
-                "alerts-multialert-firing.spec.js",
-                "alerts-multialert-transitions.spec.js",
-              ]
-          - testfolder: "Dashboards-Core"
-            browser: "chrome"
-            run_files:
-              [
-                "dashboard.spec.js",
-                "dashboard2.spec.js",
-                "dashboardtype.spec.js",
-                "dashboard-folder.spec.js",
-                "dashboard-import.spec.js",
-                "maxquery.spec.js",
-                "dashboard-chartJson.spec.js",
-                "dashboard-promql-suggestions.spec.js",
-                "dashboard-sql-autocomplete.spec.js",
-                "dashboard-share-link.spec.js",
-                "crossLinking.spec.js",
-                "dashboard-multi-sql.spec.js",
-                "crossLinkMultiStream.spec.js",
-                "dashboards-form-validation.spec.js",
-              ]         
-          - testfolder: "Dashboards-Isolated"
-            browser: "chrome"
-            run_files: ["dashboard-favorites.spec.js"]
-          - testfolder: "Dashboards-Settings"
-            browser: "chrome"
-            run_files:
-              [
-                "dashboard-filter.spec.js",
-                "dashboard-general-setting.spec.js",
-                "dashboard-tabs-setting.spec.js",
-                "dashboard-transpose.spec.js",
-                "dashboard-join.spec.js",
-                "dashboard-nested-functions.spec.js",
-                "dashboard-legends-copy.spec.js",
-                "dashboard-raw-query.spec.js",
-              ]
-          # Dashboard panel tests are grouped by the feature they exercise so
-          # each shard stays well under the ephemeral-runner endurance window.
-          # A single combined 16-spec shard ran ~40 min on CI and got reclaimed
-          # mid-run ("runner lost communication"). Add new specs to the shard of
-          # the feature they cover; the Tables shard also owns the pagination and
-          # pivot-table specs (previously their own single-spec shards).
-          - testfolder: "Dashboards-Charts"
-            browser: "chrome"
-            run_files:
-              [
-                "custom-charts.spec.js",
-                "dashboard-geoMap.spec.js",
-                "dashboard-maps.spec.js",
-                "dashboard-multi-y-axis.spec.js",
-                "dashboard-html-chart.spec.js",
-                "dashboard-sankey.spec.js",
-                "dashboard-pie-donut.spec.js",
-                "dashboard-metric-camelcase.spec.js",
-                "dashboard-create-alert.spec.js",
-              ]
-          - testfolder: "Dashboards-Visualize"
-            browser: "chrome"
-            run_files:
-              [
-                "visualize.spec.js",
-                "visualize-vrl.spec.js",
-              ]
-          - testfolder: "Dashboards-Tables"
-            browser: "chrome"
-            run_files:
-              [
-                "dashboard-table-chart.spec.js",
-                "dashboard-table-copy-cell.spec.js",
-                "dashboard-table-csv-download.spec.js",
-                "dashboard-table-column-formatting.spec.js",
-                "dashboard-table-column-filtering.spec.js",
-                "dashboard-table-pagination.spec.js",
-                "dashboard-pivot-table.spec.js",
-              ]
-          - testfolder: "Dashboards-Streaming"
-            browser: "chrome"
-            run_files: ["dashboard-streaming.spec.js"]
-          - testfolder: "Pipelines"
-            browser: "chrome"
-            run_files: [
-                "pipeline-conditions.spec.js",
-                "pipeline-conditions-validation.spec.js",
-                "pipelines.spec.js",
-                "pipeline-dynamic.spec.js",
-                "pipeline-core.spec.js",
-                "pipeline-traces.spec.js",
-                "pipeline-metrics.spec.js",
-                "pipeline-backfill.spec.js",
-                "pipeline-history.spec.js",
-                "scheduled-pipeline-query-builder.spec.js",
-                # "remotepipeline.spec.js"
-                "pipelines-form-validation.spec.js",
-              ]
-          - testfolder: "Functions"
-            browser: "chrome"
-            run_files:
-              [
-                "js-transform-type.spec.js",
-                "row-expansion.spec.js",
-                "enrichment-table-url.spec.js",
-                "functions-form-validation.spec.js",
-                "action-scripts-form-validation.spec.js",
-              ]
-          - testfolder: "Reports"
-            browser: "chrome"
-            run_files:
-              [
-                "reportsScheduleNow.spec.js",
-                "reportsScheduleLater.spec.js",
-                "reportFolders.spec.js",
-                "reports-bulk-operations.spec.js",
-                "reports-form-validation.spec.js",
-                "reportCsvMediaType.spec.js",
-              ]
-          - testfolder: "Streams"
-            browser: "chrome"
-            run_files:
-              [
-                "multiselect-stream.spec.js",
-                "streamname.spec.js",
-                "streaming.spec.js",
-                "stream-settings.spec.js",
-                "streams-form-validation.spec.js",
-              ]
-          - testfolder: "Traces"
-            browser: "chrome"
-            run_files:
-              [
-                "service-catalog.spec.js",
-                "tracesSearch.spec.js",
-                "traceQueryEditor.spec.js",
-                "traceErrorFilter.spec.js",
-                "traceDetails.spec.js",
-                "traceAdvancedFiltering.spec.js",
-                "tracesAnalyzeDimensions.spec.js",
-                "traces-autocomplete-suggestions.spec.js",
-              ]
-          # The RUM ingest/token/sourcemaps routes are part of the standard
-          # (OSS) router (src/api/http/src/handler/http/router/mod.rs), so these specs run
-          # against the OSS binary built by this workflow. Ingestion-auth
-          # negative cases live in the API suite (tests/api-testing/tests/rum).
-          - testfolder: "RUM"
-            browser: "chrome"
-            run_files:
-              [
-                "rum-cdn-dataflow.spec.js",
-                "rum-npm-dataflow.spec.js",
-                "rum-page-dataflow.spec.js",
-                "rum-onboarding-snippets.spec.js",
-                "sourcemap-ui.spec.js",
-                "sourcemap-upload-pretty.spec.js",
-              ]
-          # rum-token.spec.js RESETS the org RUM token; running it beside the
-          # dataflow specs (workers run files in parallel) would 401 their
-          # in-flight SDK beacons. Own shard = own isolated instance.
-          - testfolder: "RUM-Token"
-            browser: "chrome"
-            run_files: ["rum-token.spec.js"]
-          - testfolder: "Metrics"
-            browser: "chrome"
-            run_files:
-              [
-                "metrics.spec.js",
-                "metrics-queries.spec.js",
-                "metrics-aggregations.spec.js",
-                "metrics-advanced.spec.js",
-                "metrics-config-tabs.spec.js",
-                "metrics-visualizations.spec.js",
-                "metrics-config.spec.js",
-                "metrics-promql-query-persistence.spec.js",
-                "metrics-table-column-order.spec.js",
-                "metrics-promql-builder.spec.js",
-                "promqlAutocomplete.spec.js",
-                "metrics-share-deep-link.spec.js",
-              ]
-          - testfolder: "Dashboards-Variables"
-            browser: "chrome"
-            run_files:
-              [
-                "dashboard-variables-setting.spec.js",
-                "dashboard-variables-global.spec.js",
-                "dashboard-variables-tab-level.spec.js",
-                "dashboard-variables-panel-level.spec.js",
-                "dashboard-variables-dependency.spec.js",
-                "dashboard-variables-refresh.spec.js",
-                "dashboard-variables-url-sync.spec.js",
-                "dashboard-variables-creation-scopes.spec.js",
-                "dashboard-variables-default-values-chain.spec.js",
-                "dashboard-variables-custom-parents.spec.js",
-                "dashboard-variables-stream-field.spec.js",
-                "dashboard-mustache-variables.spec.js",
-              ]
-          - testfolder: "Dashboards-Panel-Level-DateTime-Config"
-            browser: "chrome"
-            run_files:
-              [
-                "dashboard-panel-time-config-behavior.spec.js",
-                "dashboard-panel-time-url-priority.spec.js",
-                "dashboard-panel-time-advanced-edge-cases.spec.js",
-                "dashboard-panel-time-apply-behavior.spec.js",
-                "dashboard-panel-time-variables-behavior.spec.js",
-              ]
-          - testfolder: "Dashboard-Config-Settings"
-            browser: "chrome"
-            run_files:
-              [
-                "dashboard-series-color-multiwindow.spec.js",
-                "dashboard-config-advanced.spec.js",
-                "dashboard-config-axis.spec.js",
-                "dashboard-config-gauge-maps.spec.js",
-                "dashboard-config-general.spec.js",
-                "dashboard-config-legends.spec.js",
-                "dashboard-config-line-style.spec.js",
-                "dashboard-config-panel-time.spec.js",
-                "dashboard-config-table.spec.js",
-                "dashboard-config-trellis.spec.js",
-                "dashboard-config-drilldown.spec.js",
-                "dashboard-config-promql.spec.js",
-                "dashboard-config-markline.spec.js",
-              ]
+      # Matrix is generated from tests/ui-testing/ci-matrix/ci_matrix.json by the
+      # generate_matrix job (the single source of truth shared with ENT). To add
+      # or move a spec, edit that JSON file — do NOT hand-edit a matrix here.
+      matrix: ${{ fromJSON(needs.generate_matrix.outputs.matrix) }}
     steps:
       - name: Kill background apt processes
         run: |
@@ -629,22 +337,9 @@ jobs:
           echo "DEBUG: matrix.testfolder = ${{ matrix.testfolder }}"
 
           if [ -n "$FILE_LIST" ]; then
-            # Map logical folder names to actual directories
-            case "${{ matrix.testfolder }}" in
-              "Logs-Builder-Basic"|"Logs-Builder-Advanced"|"Logs-Core"|"Logs-Features")
-                ACTUAL_FOLDER="Logs"
-                ;;
-              "Dashboards-Core"|"Dashboards-Settings"|"Dashboards-Charts"|"Dashboards-Visualize"|"Dashboards-Tables"|"Dashboards-Streaming"|"Dashboards-Variables"|"Dashboards-Panel-Level-DateTime-Config"|"Dashboard-Config-Settings"|"Dashboards-Isolated")
-                ACTUAL_FOLDER="Dashboards"
-                ;;
-              "RUM-Token")
-                ACTUAL_FOLDER="RUM"
-                ;;
-              *)
-                ACTUAL_FOLDER="${{ matrix.testfolder }}"
-                ;;
-            esac
-
+            # actual_folder (the real directory under playwright-tests/) comes straight
+            # from the matrix now — the logical->directory mapping lives in ci-matrix/ci_matrix.json.
+            ACTUAL_FOLDER="${{ matrix.actual_folder }}"
             echo "DEBUG: ACTUAL_FOLDER = $ACTUAL_FOLDER"
 
             # Build file paths

.github/workflows/playwright_regression.yml

@@ -167,76 +167,38 @@ jobs:
           retention-days: 1
           if-no-files-found: error
 
+  generate_matrix:
+    timeout-minutes: 5
+    name: generate_matrix
+    needs: [check_main_changed]
+    if: ${{ needs.check_main_changed.outputs.should_run == 'true' }}
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    outputs:
+      matrix: ${{ steps.gen.outputs.matrix }}
+    steps:
+      - uses: actions/checkout@v5
+      - id: gen
+        run: |
+          MATRIX=$(node .github/scripts/build-ci-matrix.js tests/ui-testing/ci-matrix/ci_matrix_regression.json)
+          if [ -z "$MATRIX" ]; then echo "::error::matrix generation failed (build-ci-matrix produced no output)"; exit 1; fi
+          echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT"
+          echo "$MATRIX" | node -e "const m=JSON.parse(require('fs').readFileSync(0,'utf8')); console.log('generated',m.include.length,'shards:', m.include.map(s=>s.testfolder).join(', '))"
+
   ui_integration_tests:
     timeout-minutes: 45
     name: e2e / ${{ matrix.testfolder }}
-    needs: [build_binary]
+    needs: [build_binary, generate_matrix]
     runs-on:
       labels: eks-openobserve-standard-8
     permissions:
       contents: read
     strategy:
       fail-fast: false
-      matrix:
-        include:
-          - testfolder: "Logs-Regression"
-            run_files:
-              [
-                "logs-regression.spec.js",
-                "logs-bugs.spec.js",
-                "logs-9754.spec.js",
-                "logs-9044-7354.spec.js",
-              ]
-          - testfolder: "Alerts-Regression"
-            run_files:
-              [
-                "alerts-regression.spec.js",
-                "alerts-stream-switching.spec.js",
-                "alerts-bugs.spec.js",
-              ]
-          - testfolder: "Dashboard-Regression"
-            run_files:
-              [
-                "dashboard-regression.spec.js",
-              ]
-          - testfolder: "Streams-Regression"
-            run_files:
-              [
-                "streams-regression.spec.js",
-              ]
-          - testfolder: "Pipelines-Regression"
-            run_files:
-              [
-                "enrichment-regression.spec.js",
-                "pipeline-regression.spec.js",
-              ]
-          - testfolder: "Traces-Regression"
-            run_files:
-              [
-                "traces-regression.spec.js",
-                "traces-bugs.spec.js",
-              ]
-          - testfolder: "Metrics-Regression"
-            run_files:
-              [
-                "metrics-regression.spec.js",
-              ]
-          - testfolder: "GeneralTests-Regression"
-            run_files:
-              [
-                "ui-regression.spec.js",
-                "landing-regression.spec.js",
-              ]
-          - testfolder: "DataSources-Regression"
-            run_files:
-              [
-                "datasources-regression.spec.js",
-              ]
-          - testfolder: "Reports-Regression"
-            run_files:
-              [
-                "reports-regression-bugs.spec.js",
-              ]
+      # Matrix from tests/ui-testing/ci-matrix/ci_matrix_regression.json (single source of
+      # truth, shared with ENT via build-ci-matrix.js). Edit that JSON, not this file.
+      matrix: ${{ fromJSON(needs.generate_matrix.outputs.matrix) }}
 
     steps:
       - name: Kill background apt processes
@@ -326,23 +288,10 @@ jobs:
           echo "DEBUG: matrix.testfolder = ${{ matrix.testfolder }}"
 
           if [ -n "$FILE_LIST" ]; then
-            # Derive feature subfolder from testfolder (maps testfolder name → directory under RegressionSet/)
-            case "${{ matrix.testfolder }}" in
-              "Dashboard-Regression")   FEATURE="Dashboards" ;;
-              "Pipelines-Regression")   FEATURE="Pipelines" ;;
-              "Alerts-Regression")      FEATURE="Alerts" ;;
-              "Logs-Regression")        FEATURE="Logs" ;;
-              "Traces-Regression")      FEATURE="Traces" ;;
-              "Metrics-Regression")     FEATURE="Metrics" ;;
-              "Streams-Regression")     FEATURE="Streams" ;;
-              "Reports-Regression")     FEATURE="Reports" ;;
-              "DataSources-Regression") FEATURE="DataSources" ;;
-              "GeneralTests-Regression") FEATURE="General" ;;
-              *)                        FEATURE="RegressionSet" ;;
-            esac
-            ACTUAL_FOLDER="RegressionSet/$FEATURE"
+            # actual_folder (e.g. RegressionSet/Logs) comes from the matrix now —
+            # the testfolder->directory mapping lives in ci_matrix_regression.json.
+            ACTUAL_FOLDER="${{ matrix.actual_folder }}"
             echo "DEBUG: ACTUAL_FOLDER = $ACTUAL_FOLDER"
-            echo "DEBUG: FEATURE = $FEATURE"
 
             FILE_PATHS=""
             for file in $FILE_LIST; do

Cargo.lock

@@ -2480,12 +2480,14 @@ dependencies = [
  "itertools 0.14.0",
  "log",
  "o2_enterprise",
+ "object_store",
  "parking_lot 0.12.5",
  "parquet",
  "rand 0.10.1",
  "schema",
  "search",
  "search_service",
+ "tantivy",
  "tantivy_utils",
  "tokio",
 ]
@@ -2551,6 +2553,7 @@ dependencies = [
  "dotenv_config",
  "dotenvy",
  "expect-test",
+ "fastdivide",
  "faststr",
  "float-cmp",
  "futures",
@@ -10530,6 +10533,7 @@ dependencies = [
  "futures",
  "futures-util",
  "hashbrown 0.16.1",
+ "hashlink 0.11.0",
  "infra",
  "itertools 0.14.0",
  "log",

Cargo.toml

@@ -456,6 +456,7 @@ sqlparser = { version = "0.62", features = ["serde", "visitor"] }
 dotenv_config = "0.2"
 dotenvy = "0.15"
 env_logger = "0.11"
+fastdivide = "0.4"
 faststr = { version = "0.2", features = ["serde"] }
 flate2 = { version = "1.0", features = ["zlib"] }
 futures = "0.3"

src/api/http/src/handler/http/router/mod.rs

@@ -806,6 +806,7 @@ pub fn service_routes() -> Router {
 
         // Search
         .route("/{org_id}/_search", post(search::search))
+        .route("/{org_id}/query_functions", get(search::query_functions::list))
         .route("/{org_id}/_search_partition", post(search::search_partition))
         .route("/{org_id}/{stream_name}/_around", get(search::around_v1).post(search::around_v2))
         .route("/{org_id}/{stream_name}/_values", get(search::values))
@@ -1514,6 +1515,117 @@ mod tests {
         );
     }
 
+    // NOTE ON WHAT IS TESTABLE HERE.
+    //
+    // service_routes() wraps the entire router in auth_middleware, which
+    // short-circuits BEFORE routing: a completely nonexistent path also answers
+    // 401. So an HTTP-level test against service_routes() cannot tell a
+    // registered route from an absent one — an assertion like "not 404", or
+    // even "== 401", passes whether or not the endpoint exists.
+    //
+    // Registration is therefore pinned two ways that DO discriminate: the route
+    // appears in the OpenAPI surface, and a minimal router carrying only this
+    // route dispatches GET to the handler and rejects other methods.
+
+    #[tokio::test]
+    async fn query_functions_route_dispatches_get_and_rejects_other_methods() {
+        let app = Router::new().route(
+            "/{org_id}/query_functions",
+            get(search::query_functions::list),
+        );
+
+        let ok = app
+            .clone()
+            .oneshot(
+                Request::builder()
+                    .uri("/myorg/query_functions")
+                    .body(Body::empty())
+                    .unwrap(),
+            )
+            .await
+            .unwrap();
+        assert_eq!(ok.status(), StatusCode::OK);
+
+        let wrong_method = app
+            .oneshot(
+                Request::builder()
+                    .method("POST")
+                    .uri("/myorg/query_functions")
+                    .body(Body::empty())
+                    .unwrap(),
+            )
+            .await
+            .unwrap();
+        assert_eq!(
+            wrong_method.status(),
+            StatusCode::METHOD_NOT_ALLOWED,
+            "the catalog is read-only; the frontend issues GET"
+        );
+    }
+
+    #[test]
+    fn query_functions_is_published_in_the_openapi_surface() {
+        // Discriminating: this fails if the handler is dropped from openapi.rs.
+        let spec = super::openapi::ApiDoc::openapi();
+        let json = serde_json::to_string(&spec).unwrap();
+        assert!(
+            json.contains("/{org_id}/query_functions"),
+            "query_functions is missing from the OpenAPI surface"
+        );
+    }
+
+    // ── tmp/code.md B4 — the query-function catalog route ─────────────────────
+    //
+    // catalog_functions() can be perfect while no HTTP route exposes it. This is
+    // the only assertion that fails if the endpoint is simply never registered.
+
+    #[tokio::test]
+    async fn query_functions_handler_returns_the_documented_payload_shape() {
+        // Calls the handler DIRECTLY, bypassing auth, so the body contract the
+        // frontend service depends on ({ list: [...] }) is actually asserted.
+        use axum::extract::Path;
+
+        let response =
+            openobserve_api_search::search::query_functions::list(Path("default".to_string()))
+                .await
+                .into_response();
+
+        assert_eq!(response.status(), StatusCode::OK);
+
+        let body = axum::body::to_bytes(response.into_body(), usize::MAX)
+            .await
+            .unwrap();
+        let payload: Value = serde_json::from_slice(&body).expect("body must be JSON");
+
+        let list = payload
+            .get("list")
+            .and_then(Value::as_array)
+            .expect("payload must be { list: [...] } — the frontend reads data.list");
+        assert!(!list.is_empty(), "catalog must not be empty");
+
+        let entry = list
+            .iter()
+            .find(|f| f.get("name").and_then(Value::as_str) == Some("match_all"))
+            .expect("match_all must be present");
+        for field in ["name", "signature", "doc", "kind", "deprecated"] {
+            assert!(
+                entry.get(field).is_some(),
+                "serialized entry is missing `{field}`"
+            );
+        }
+
+        // Assert non-empty doc on an entry the SERVER is the sole source for.
+        // The frontend catalog carries its own prose for the O2 UDFs and wins
+        // on merge, so requiring a server-side doc for match_all would pin data
+        // the UI never renders.
+        let alias = list
+            .iter()
+            .find(|f| f.get("name").and_then(Value::as_str) == Some("match_all_raw"))
+            .expect("rewriter alias must be present");
+        assert!(!alias["doc"].as_str().unwrap_or("").is_empty());
+        assert_eq!(alias["deprecated"], Value::Bool(true));
+    }
+
     // ── is_origin_allowed unit tests ──────────────────────────────────────────
 
     #[test]

src/api/http/src/handler/http/router/openapi.rs

@@ -92,6 +92,7 @@ use crate::{
         rum::ingest::sessionreplay,
         openobserve_api_search::search::search,
         openobserve_api_search::search::search_partition,
+        openobserve_api_search::search::query_functions::list,
         openobserve_api_search::search::around_v1,
         openobserve_api_search::search::around_v2,
         openobserve_api_search::search::values,

src/api/http/src/router/ratelimit/resource_extractor.rs

@@ -552,7 +552,15 @@ mod tests {
     #[test]
     fn test_select_final_rule_org_level() {
         let default_rule = create_rule(".*", None, None, 100);
-        let org_rule = create_rule("test_org", None, None, 50);
+        // An org-level rule is the one carrying the GLOBAL sentinels in both
+        // user_role and user_id — see select_final_rule_resource. Passing None
+        // here described a shape the selector has never matched.
+        let org_rule = create_rule(
+            "test_org",
+            Some(DEFAULT_GLOBAL_USER_ROLE_IDENTIFIER),
+            Some(DEFAULT_GLOBAL_USER_ID_IDENTIFIER),
+            50,
+        );
         let custom_rules = vec![org_rule.clone()];
 
         let result = select_final_rule_resource(
@@ -607,7 +615,12 @@ mod tests {
     #[test]
     fn test_select_final_rule_all_levels() {
         let default_rule = create_rule(".*", None, None, 100);
-        let org_rule = create_rule("test_org", None, None, 80);
+        let org_rule = create_rule(
+            "test_org",
+            Some(DEFAULT_GLOBAL_USER_ROLE_IDENTIFIER),
+            Some(DEFAULT_GLOBAL_USER_ID_IDENTIFIER),
+            80,
+        );
         let role_rule = create_rule("test_org", Some("admin"), Some(".*"), 60);
         let user_rule = create_rule("test_org", Some("admin"), Some("test@example.com"), 40);
         let custom_rules = vec![org_rule.clone(), role_rule.clone(), user_rule.clone()];

src/api/management/src/request/synthetics/mod.rs

@@ -34,7 +34,7 @@ use crate::service::auth::{UserEmail, check_permissions};
 pub struct ListSyntheticsQuery {
     pub folder: Option<String>,
     #[serde(rename = "type")]
-    pub monitor_type: Option<config::meta::synthetics::SyntheticType>,
+    pub check_type: Option<config::meta::synthetics::SyntheticType>,
     pub enabled: Option<bool>,
     pub location: Option<String>,
     pub tag: Option<String>,
@@ -46,7 +46,7 @@ impl From<ListSyntheticsQuery> for config::meta::synthetics::ListSyntheticsParam
     fn from(q: ListSyntheticsQuery) -> Self {
         Self {
             folder_id: q.folder,
-            monitor_type: q.monitor_type,
+            check_type: q.check_type,
             enabled: q.enabled,
             location: q.location,
             tag: q.tag,
@@ -88,11 +88,11 @@ pub struct ListRunsQuery {
     context_path = "/api",
     tag = "Synthetics",
     operation_id = "ListSyntheticsRuns",
-    summary = "List runs for a monitor",
+    summary = "List runs for a check",
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
         ("start_time" = Option<i64>, Query, description = "Filter runs with scheduled_ts >= start_time (microseconds)"),
         ("end_time" = Option<i64>, Query, description = "Filter runs with scheduled_ts <= end_time (microseconds)"),
         ("page" = Option<i64>, Query, description = "Page number (0-indexed, default 0)"),
@@ -146,7 +146,7 @@ pub async fn list_runs(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
         ("run_id" = String, Path, description = "Run ID (KSUID)"),
     ),
     responses(
@@ -332,7 +332,7 @@ pub async fn job_upload(
     }
 }
 
-// ── Monitors ──────────────────────────────────────────────────────────────────
+// ── Checks ──────────────────────────────────────────────────────────────────
 
 #[utoipa::path(
     get,
@@ -345,7 +345,7 @@ pub async fn job_upload(
     params(
         ("org_id" = String, Path, description = "Organization name"),
         ("folder" = Option<String>, Query, description = "Filter by folder ID (KSUID)"),
-        ("type" = Option<String>, Query, description = "Filter by monitor type (http|browser|tcp|tls|ssh)"),
+        ("type" = Option<String>, Query, description = "Filter by check type (http|browser|tcp|tls|ssh)"),
         ("enabled" = Option<bool>, Query, description = "Filter by enabled status"),
         ("location" = Option<String>, Query, description = "Filter by location"),
         ("tag" = Option<String>, Query, description = "Filter by tag"),
@@ -428,7 +428,7 @@ pub async fn create_synthetic(
         )
         .await
         {
-            Ok(monitor) => MetaHttpResponse::json(monitor),
+            Ok(check) => MetaHttpResponse::json(check),
             Err(e) => {
                 let msg = e.to_string();
                 if msg.starts_with("validation: ") {
@@ -457,7 +457,7 @@ pub async fn create_synthetic(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
     ),
     responses(
         (status = 200, description = "Success", content_type = "application/json", body = config::meta::synthetics::Synthetic),
@@ -488,8 +488,8 @@ pub async fn get_synthetic(
             return MetaHttpResponse::forbidden("Forbidden");
         }
         match o2_enterprise::enterprise::synthetics::service::get_synthetic(&org_id, &id).await {
-            Ok(Some(monitor)) => MetaHttpResponse::json(monitor),
-            Ok(None) => MetaHttpResponse::not_found("monitor not found"),
+            Ok(Some(check)) => MetaHttpResponse::json(check),
+            Ok(None) => MetaHttpResponse::not_found("check not found"),
             Err(e) => {
                 tracing::error!("[synthetics] get_synthetic: {e}");
                 MetaHttpResponse::error(StatusCode::INTERNAL_SERVER_ERROR.as_u16(), e.to_string())
@@ -514,7 +514,7 @@ pub async fn get_synthetic(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
         ("folder" = Option<String>, Query, description = "Current folder ID of the synthetic (for RBAC)"),
     ),
     request_body(content = config::meta::synthetics::Synthetic, description = "Updated synthetic definition", content_type = "application/json"),
@@ -550,7 +550,7 @@ pub async fn update_synthetic(
         match o2_enterprise::enterprise::synthetics::service::update_synthetic(&org_id, &id, body)
             .await
         {
-            Ok(monitor) => MetaHttpResponse::json(monitor),
+            Ok(check) => MetaHttpResponse::json(check),
             Err(e) => {
                 let msg = e.to_string();
                 if msg.starts_with("validation: ") {
@@ -582,7 +582,7 @@ pub async fn update_synthetic(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
         ("folder" = Option<String>, Query, description = "Current folder ID of the synthetic (for RBAC)"),
     ),
     responses(
@@ -614,8 +614,8 @@ pub async fn delete_synthetic(
             return MetaHttpResponse::forbidden("Forbidden");
         }
         match o2_enterprise::enterprise::synthetics::service::delete_synthetic(&org_id, &id).await {
-            Ok(true) => MetaHttpResponse::ok("monitor deleted"),
-            Ok(false) => MetaHttpResponse::not_found("monitor not found"),
+            Ok(true) => MetaHttpResponse::ok("check deleted"),
+            Ok(false) => MetaHttpResponse::not_found("check not found"),
             Err(e) => {
                 tracing::error!("[synthetics] delete_synthetic: {e}");
                 MetaHttpResponse::error(StatusCode::INTERNAL_SERVER_ERROR.as_u16(), e.to_string())
@@ -662,7 +662,7 @@ pub async fn delete_synthetics_bulk(
         )
         .await
         {
-            Ok(_) => MetaHttpResponse::ok("monitors deleted"),
+            Ok(_) => MetaHttpResponse::ok("checks deleted"),
             Err(e) => {
                 tracing::error!("[synthetics] delete_synthetics_bulk: {e}");
                 MetaHttpResponse::error(StatusCode::INTERNAL_SERVER_ERROR.as_u16(), e.to_string())
@@ -732,7 +732,7 @@ pub async fn move_synthetics(
         )
         .await
         {
-            Ok(_) => MetaHttpResponse::ok("monitors moved"),
+            Ok(_) => MetaHttpResponse::ok("checks moved"),
             Err(e) => {
                 tracing::error!("[synthetics] move_synthetics: {e}");
                 MetaHttpResponse::error(StatusCode::INTERNAL_SERVER_ERROR.as_u16(), e.to_string())
@@ -757,7 +757,7 @@ pub async fn move_synthetics(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
     ),
     request_body(content = Object, description = r#"{"enabled": true}"#, content_type = "application/json"),
     responses(
@@ -799,11 +799,11 @@ pub async fn set_synthetic_enabled(
         .await
         {
             Ok(true) => MetaHttpResponse::ok(if enabled {
-                "monitor enabled"
+                "check enabled"
             } else {
-                "monitor paused"
+                "check paused"
             }),
-            Ok(false) => MetaHttpResponse::not_found("monitor not found"),
+            Ok(false) => MetaHttpResponse::not_found("check not found"),
             Err(e) => {
                 tracing::error!("[synthetics] set_synthetic_enabled: {e}");
                 MetaHttpResponse::error(StatusCode::INTERNAL_SERVER_ERROR.as_u16(), e.to_string())
@@ -828,7 +828,7 @@ pub async fn set_synthetic_enabled(
     security(("Authorization" = [])),
     params(
         ("org_id" = String, Path, description = "Organization name"),
-        ("id" = String, Path, description = "Monitor ID"),
+        ("id" = String, Path, description = "Check ID"),
     ),
     responses(
         (status = 202, description = "Accepted — scheduler will fire within 5 seconds"),
@@ -886,7 +886,7 @@ pub async fn run_synthetic_now(
     context_path = "/api",
     tag = "Synthetics",
     operation_id = "SyntheticsJobResolve",
-    summary = "Resolve a job — probe fetches monitor config (authenticated via o2syn_ token)",
+    summary = "Resolve a job — probe fetches check config (authenticated via o2syn_ token)",
     params(
         ("org_id" = String, Path, description = "Organization name"),
     ),
@@ -1128,9 +1128,9 @@ async fn process_ack(
     if should_notify && !resp.destinations.is_empty() {
         let notification = openobserve_core::synthetics::CheckNotification {
             org_id: resp.org_id.clone(),
-            monitor_name: resp.synthetics_name.clone(),
-            monitor_id: resp.synthetics_id.clone(),
-            monitor_type: resp.synthetic_type.clone(),
+            check_name: resp.synthetics_name.clone(),
+            check_id: resp.synthetics_id.clone(),
+            check_type: resp.synthetic_type.clone(),
             target: resp.target.clone(),
             destinations: resp.destinations.clone(),
             run_id: resp.run_id.clone(),
@@ -1144,6 +1144,7 @@ async fn process_ack(
             degraded,
             status_reason: resp.status_reason.clone(),
             failing_locations: resp.failing_locations.clone(),
+            passing_locations: resp.passing_locations.clone(),
         };
         tokio::spawn(async move {
             openobserve_core::synthetics::notify_check_result(notification).await;

src/api/search/src/search/mod.rs

@@ -75,6 +75,7 @@ pub(crate) mod around;
 pub mod error_utils;
 pub mod multi_streams;
 pub mod patterns;
+pub mod query_functions;
 pub mod query_manager;
 pub mod saved_view;
 #[cfg(feature = "enterprise")]

src/api/search/src/search/query_functions.rs

@@ -0,0 +1,53 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! The SQL function catalog served to the query editor.
+//!
+//! The frontend used to hand-maintain its list of callable functions, which had
+//! no way of tracking the pinned DataFusion fork, build features, or an
+//! organisation's own VRL transforms. This endpoint is derived from the live
+//! registry instead.
+
+use axum::{Json, extract::Path, response::Response};
+use search::datafusion::exec::{CatalogFunction, catalog_functions};
+use serde::Serialize;
+
+#[derive(Debug, Serialize, utoipa::ToSchema)]
+pub struct QueryFunctionsResponse {
+    pub list: Vec<CatalogFunction>,
+}
+
+/// GET /api/{org_id}/query_functions
+///
+/// Returns every function this organisation can call: the DataFusion registry
+/// (including the JSON family), the O2 UDFs, the SQL-rewriter aliases, and the
+/// org's own VRL transforms.
+#[utoipa::path(
+    get,
+    path = "/{org_id}/query_functions",
+    context_path = "/api",
+    tag = "Search",
+    operation_id = "QueryFunctions",
+    security(("Authorization" = [])),
+    params(("org_id" = String, Path, description = "Organization name")),
+    responses(
+        (status = 200, description = "Success", content_type = "application/json",
+         body = QueryFunctionsResponse),
+    )
+)]
+pub async fn list(Path(org_id): Path<String>) -> Response {
+    let list = catalog_functions(&org_id);
+    axum::response::IntoResponse::into_response(Json(QueryFunctionsResponse { list }))
+}

src/compaction/Cargo.toml

@@ -28,11 +28,13 @@ ingester.workspace = true
 itertools.workspace = true
 log.workspace = true
 o2_enterprise = { workspace = true, optional = true, default-features = false }
+object_store.workspace = true
 parking_lot.workspace = true
 parquet.workspace = true
 rand.workspace = true
 schema.workspace = true
 search.workspace = true
 search_service.workspace = true
+tantivy.workspace = true
 tantivy_utils.workspace = true
 tokio.workspace = true

src/compaction/src/bloom/builder.rs

@@ -0,0 +1,127 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Build SBBF blooms by iterating tantivy term dictionaries.
+//!
+//! For an indexed field, tantivy already stores a deduplicated term
+//! dictionary. Iterating it is much cheaper than re-scanning the parquet
+//! column — terms come back already unique, sorted, and as raw bytes.
+//!
+//! Used by the compactor merge hook to build per-(file, field)
+//! blooms after the .ttv file is written.
+
+use std::collections::HashSet;
+
+use anyhow::Context;
+use hashbrown::HashMap;
+use infra::bloom::{BloomBuilder, FieldBloom};
+use tantivy::Index;
+use tantivy_utils::puffin_directory::reader::warm_up_terms;
+
+/// Build per-field SBBFs for one file using a group-uniform `num_blocks`.
+///
+/// Every file in a (stream, hour, bloom_ver) group must pass the same
+/// `num_blocks` so the transposed `.bf` layout can read one block-row per
+/// group (see `infra::bloom` module docs). The caller derives it once from
+/// the configured expected cardinality.
+///
+/// Behavior:
+/// - Fields not present in the schema are silently skipped — the compactor passes the union of
+///   `index_fields ∩ bloom_filter_fields` over potentially many streams, and not every field exists
+///   everywhere.
+/// - Terms across all segments of the index are merged into one bloom per field. Today
+///   `create_tantivy_index` produces a single segment, but this is robust to that changing.
+pub(super) async fn build_blooms_from_index(
+    index: &Index,
+    file_id: u64,
+    fields: &[String],
+    num_blocks: u32,
+) -> Result<Vec<FieldBloom>, anyhow::Error> {
+    if fields.is_empty() {
+        return Ok(Vec::new());
+    }
+
+    let schema = index.schema();
+    let reader = index
+        .reader_builder()
+        .reload_policy(tantivy::ReloadPolicy::Manual)
+        .num_warming_threads(0)
+        .try_into()
+        .context("open tantivy reader")?;
+    let searcher = reader.searcher();
+
+    let warm_terms: HashMap<tantivy::schema::Field, HashMap<tantivy::Term, bool>> = HashMap::new();
+    let mut need_all_term_fields = HashSet::new();
+    for field in fields {
+        let Ok(field) = schema.get_field(field) else {
+            continue;
+        };
+        need_all_term_fields.insert(field);
+    }
+
+    // warm_up_terms operates on one SegmentReader at a time; warm each segment.
+    // need_all_term_fields / need_fast_field are consumed per call, so clone them.
+    for seg in searcher.segment_readers() {
+        warm_up_terms(
+            seg,
+            &warm_terms,
+            need_all_term_fields.clone(),
+            HashSet::new(),
+        )
+        .await?;
+    }
+
+    let mut builder = BloomBuilder::new();
+
+    for field_name in fields {
+        let Ok(field) = schema.get_field(field_name) else {
+            continue;
+        };
+
+        // Skip fields with no terms in this file so they don't become an
+        // empty column in the transposed matrix.
+        let mut has_terms = false;
+        for seg in searcher.segment_readers() {
+            if let Ok(inv) = seg.inverted_index(field)
+                && inv.terms().num_terms() > 0
+            {
+                has_terms = true;
+                break;
+            }
+        }
+        if !has_terms {
+            continue;
+        }
+
+        // Uniform block count across the whole group (caller-provided).
+        let idx = builder.begin_with_blocks(file_id, field_name, num_blocks);
+
+        for seg in searcher.segment_readers() {
+            let inv = match seg.inverted_index(field) {
+                Ok(i) => i,
+                Err(_) => continue,
+            };
+            let mut stream = inv
+                .terms()
+                .stream()
+                .with_context(|| format!("stream terms for {field_name}"))?;
+            while let Some((term_bytes, _info)) = stream.next() {
+                builder.insert(idx, term_bytes);
+            }
+        }
+    }
+
+    Ok(builder.finish())
+}

src/compaction/src/bloom/compact.rs

@@ -0,0 +1,349 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Bloom filter build + orphan cleanup for the compactor.
+//!
+//! One public entry point, [`build_for_stream`], called once per hour bucket
+//! at the end of a compaction merge round. It does two things in order:
+//!
+//! 1. **Orphan cleanup.** Given the `bloom_ver` values of the files this merge round just deleted,
+//!    [`cleanup_orphan_blooms`] retires any `.bf` no longer referenced by a live row in the bucket.
+//! 2. **Build.** It queries the bucket itself (`query_for_bloom` → rows with `index_size > 0` and
+//!    `bloom_ver = 0`) and builds `.bf` coverage for those files. The caller does not pass a file
+//!    list; this module owns both the "which files" and the "how to bloom" decisions.
+//!
+//! `bloom_ver` is not append-only: a merge produces new output files with a
+//! fresh `bloom_ver` and deletes the small inputs, so an old `.bf` whose every
+//! file got merged away becomes an orphan — step 1 catches it.
+//!
+//! Failure is **always non-fatal**. A failed build leaves `bloom_ver = 0`
+//! on the affected files and the search side falls back to opening every
+//! tantivy file in the bucket. A failed cleanup leaves an orphan `.bf` for
+//! data-retention to reap. Neither affects correctness, so neither
+//! propagates to the caller as a hard error.
+
+use std::{collections::HashSet, sync::Arc};
+
+use anyhow::Context;
+use bytes::Bytes;
+use config::{
+    get_config,
+    meta::stream::{FileKey, FileListDeleted, StreamType},
+    utils::{inverted_index::to_tantivy_name, time::now_micros},
+};
+use infra::{
+    bloom::{BloomWriter, FieldBloom, path::bloom_path},
+    errors::Result,
+    file_list as infra_file_list, storage,
+};
+use tantivy::Directory;
+use tantivy_utils::puffin_directory::{
+    caching_directory::CachingDirectory, footer_cache::FooterCache, reader::PuffinDirReader,
+};
+
+use super::builder::build_blooms_from_index;
+
+/// Clean orphan `.bf`s for `orphan_blooms`, then build `.bf` coverage for the
+/// bucket's `bloom_ver = 0` files (queried internally via `query_for_bloom`).
+///
+/// `date_key` is the `YYYY/MM/DD/HH` string used in `file_list.date`.
+/// `orphan_blooms` is the `bloom_ver` values of the files the merge round just
+/// deleted — used only for cleanup. Files already carrying a non-zero
+/// `bloom_ver` are never re-stamped, so no live file migrates off a `bloom_ver`
+/// that a dumped row may still point to.
+///
+/// Returns `Ok(false)` when bloom is disabled or there is nothing to build (no
+/// target fields, nothing at `bloom_ver = 0`); `Ok(true)` when a build ran.
+/// Recoverable per-file/per-chunk errors are logged, not propagated — never
+/// surfaces a build failure to user requests.
+pub(crate) async fn build_for_stream(
+    org_id: &str,
+    stream_type: StreamType,
+    stream_name: &str,
+    date_key: &str,
+    is_incremental: bool,
+    orphan_blooms: Vec<i64>,
+) -> Result<bool> {
+    let cfg = get_config();
+    if !cfg.common.bloom_filter_enabled {
+        return Ok(false);
+    }
+
+    // clean up orphan blooms
+    if let Err(e) =
+        cleanup_orphan_blooms(org_id, stream_type, stream_name, date_key, orphan_blooms).await
+    {
+        log::warn!("[BLOOM_BUILD] cleanup orphan blooms failed: {e}");
+    }
+
+    // don't build bloom for incremental round
+    if is_incremental {
+        return Ok(false);
+    }
+
+    // Resolve target fields from current stream settings: a field is bloomed
+    // only when the operator put it in BOTH `bloom_filter_fields` and
+    // `index_fields` (bloom is built on top of the tantivy index).
+    let stream_settings = infra::schema::get_settings(org_id, stream_name, stream_type).await;
+    let bloom_filter_fields =
+        infra::schema::get_stream_setting_bloom_filter_fields(&stream_settings);
+    if bloom_filter_fields.is_empty() {
+        return Ok(false);
+    }
+    let index_fields: HashSet<String> =
+        infra::schema::get_stream_setting_index_fields(&stream_settings)
+            .into_iter()
+            .collect();
+    let target_fields: Vec<String> = bloom_filter_fields
+        .into_iter()
+        .filter(|f| index_fields.contains(f))
+        .collect();
+    if target_fields.is_empty() {
+        return Ok(false);
+    }
+
+    // Pull every file currently in this hour bucket.
+    let mut files =
+        infra_file_list::query_for_bloom(org_id, stream_type, stream_name, date_key).await?;
+    if files.is_empty() {
+        return Ok(false);
+    }
+
+    // Build blooms only for files without a `.bf` yet (bloom_ver = 0).
+    //
+    // **Sub-grouping**: a single hour can hold thousands of files. Since the
+    // transposed `.bf` holds every file's SBBF in memory before serialize
+    // (M × B × 32 bytes), one giant `.bf` would OOM the compactor at high
+    // volume. So we split the new files into chunks of at most
+    // `max_files_per_bf` and write one `.bf` per chunk, each with its own
+    // `bloom_ver`. The search side groups by `(date, bloom_ver)`, so it
+    // naturally reads one block-row per chunk. Total read bytes stay
+    // `M × 32`; only the request count grows to `ceil(M / max_files_per_bf)`.
+    //
+    // Files are sorted by record count first so similar-sized files share a
+    // chunk — minimizes the uniform-`B` padding waste (B is sized to each
+    // chunk's max cardinality).
+    let fpp = cfg.common.bloom_filter_fpp;
+    let max_files_per_bf = cfg.common.bloom_filter_max_files_per_bf.max(1);
+    files.sort_by_key(|k| std::cmp::Reverse(k.meta.records));
+
+    let base_ver = now_micros();
+    let chunk_total = files.len().div_ceil(max_files_per_bf);
+    for (chunk_idx, chunk) in files.chunks(max_files_per_bf).enumerate() {
+        let bloom_ver = base_ver + chunk_idx as i64;
+        let bloom_path = bloom_path(org_id, stream_type, stream_name, date_key, bloom_ver);
+        match build_for_chunk(org_id, bloom_ver, &bloom_path, chunk, &target_fields, fpp).await {
+            Ok((took, num_blocks, contributing_ids)) => {
+                log::info!(
+                    "[BLOOM_BUILD] {bloom_path}: wrote chunk {}/{chunk_total}, num_blocks={num_blocks} covering {contributing_ids} files in {took} ms",
+                    chunk_idx + 1,
+                );
+            }
+            Err(e) => {
+                log::warn!("[BLOOM_BUILD] {bloom_path}: build chunk failed: {e}");
+            }
+        }
+    }
+
+    Ok(true)
+}
+
+// B for this chunk is sized from the chunk's MAX record count, used as
+// a safe NDV upper bound: a file can't hold more distinct values than
+// rows. This never under-sizes (no saturation) for the target
+// high-cardinality fields where distinct ≈ rows. It over-sizes for
+// fields that repeat heavily (distinct ≪ rows) — acceptable; exact
+// sizing would read each file's tantivy term count in a pre-pass.
+async fn build_for_chunk(
+    org_id: &str,
+    bloom_ver: i64,
+    bf_path: &str,
+    files: &[FileKey],
+    target_fields: &[String],
+    fpp: f64,
+) -> Result<(u64, u32, usize)> {
+    let start = std::time::Instant::now();
+
+    let max_records = files
+        .iter()
+        .map(|f| f.meta.records.max(0) as u64)
+        .max()
+        .unwrap_or(0)
+        .max(1);
+    let num_blocks = infra::bloom::num_blocks_for(max_records, fpp);
+
+    let mut all_blooms: Vec<FieldBloom> = Vec::new();
+    let mut contributing_ids: Vec<i64> = Vec::new();
+    for f in files {
+        match build_for_file(f, target_fields, num_blocks).await {
+            Ok(mut blooms) if !blooms.is_empty() => {
+                all_blooms.append(&mut blooms);
+                contributing_ids.push(f.id);
+            }
+            Ok(_) => {} // no blooms (no index / field absent) — leave at 0
+            Err(e) => {
+                log::warn!(
+                    "[BLOOM_BUILD] {bf_path}: skipping {} (bloom build failed): {e}",
+                    f.key
+                );
+            }
+        }
+    }
+    if all_blooms.is_empty() {
+        return Ok((0, 0, 0));
+    }
+
+    // Distinct bloom_ver per chunk (base + idx) → distinct `.bf` path.
+    let blob: Vec<u8> = BloomWriter::serialize(all_blooms).context("serialize blooms")?;
+    let bf_account = storage::get_account(org_id, bf_path).unwrap_or_default();
+    storage::put(&bf_account, bf_path, Bytes::from(blob))
+        .await
+        .context("upload blooms")?;
+
+    debug_assert!(
+        contributing_ids.iter().all(|id| *id > 0),
+        "bloom builder must only stamp file_list rows with assigned ids"
+    );
+    infra_file_list::update_bloom_ver(&contributing_ids, bloom_ver)
+        .await
+        .context("update bloom ver")?;
+
+    let took = start.elapsed().as_millis() as u64;
+
+    Ok((took, num_blocks, contributing_ids.len()))
+}
+
+async fn build_for_file(
+    file: &FileKey,
+    target_fields: &[String],
+    num_blocks: u32,
+) -> anyhow::Result<Vec<FieldBloom>> {
+    let Some(ttv_file_name) = to_tantivy_name(&file.key) else {
+        return Ok(Vec::new()); // not an indexable file
+    };
+    if file.meta.index_size == 0 {
+        return Ok(Vec::new()); // no .ttv was emitted
+    }
+    let file_account = file.account.clone();
+    let puffin_dir =
+        Arc::new(get_tantivy_directory(&file_account, &ttv_file_name, file.meta.index_size).await?);
+    let footer_cache = FooterCache::from_directory(puffin_dir.clone(), &ttv_file_name).await?;
+    let cache_dir = CachingDirectory::new_with_cacher(puffin_dir, Arc::new(footer_cache));
+    let reader_directory: Box<dyn Directory> = Box::new(cache_dir);
+    let index = tantivy::Index::open(reader_directory).context("open index")?;
+
+    // file.id is the file_list row id, assigned by the INSERT that
+    // happened in `write_file_list` before this build runs. Always > 0
+    // by the time we get here.
+    let file_id = file.id as u64;
+    build_blooms_from_index(&index, file_id, target_fields, num_blocks).await
+}
+
+/// Open a tantivy puffin directory for `file_name` in `file_account`.
+///
+/// Mirrors the search-side `get_tantivy_directory` helper: builds a synthetic
+/// `ObjectMeta` (the `.ttv` is immutable, so a fixed `last_modified` is fine)
+/// and opens a streaming puffin reader over it.
+async fn get_tantivy_directory(
+    file_account: &str,
+    file_name: &str,
+    file_size: i64,
+) -> anyhow::Result<PuffinDirReader> {
+    let file_account = file_account.to_string();
+    let source = object_store::ObjectMeta {
+        location: file_name.into(),
+        last_modified: *config::utils::time::BASE_TIME,
+        size: file_size as u64,
+        e_tag: None,
+        version: None,
+    };
+    Ok(PuffinDirReader::from_path(file_account, source).await?)
+}
+
+/// Retire any `.bf` whose `bloom_ver` is no longer referenced by a live file
+/// in this (stream, `date_key`) bucket.
+///
+/// `deleted_bloom_vers` is the set of `bloom_ver` values carried by files a
+/// merge round just deleted (the caller dedups / drops the 0 sentinel, but we
+/// guard again here). For each, we probe `file_list` for any surviving row at
+/// that version; if none, the `.bf` is enqueued for deletion via the existing
+/// `file_list_deleted` queue.
+///
+/// Orphan detection is a **cleanup optimization, not correctness**: if the
+/// EXISTS probe or the enqueue fails, we log and move on — the stale `.bf`
+/// stays until data-retention reaps the day's subtree. Errors never
+/// propagate.
+async fn cleanup_orphan_blooms(
+    org_id: &str,
+    stream_type: StreamType,
+    stream_name: &str,
+    date_key: &str,
+    orphan_blooms: Vec<i64>,
+) -> Result<()> {
+    if orphan_blooms.is_empty() {
+        return Ok(());
+    }
+    let candidates: HashSet<i64> = orphan_blooms.into_iter().collect();
+
+    let mut orphans: Vec<FileListDeleted> = Vec::new();
+    for v_old in candidates {
+        match infra_file_list::bloom_ver_referenced(
+            org_id,
+            stream_type,
+            stream_name,
+            date_key,
+            v_old,
+        )
+        .await
+        {
+            Ok(true) => {} // still referenced by a live file — keep the `.bf`
+            Ok(false) => {
+                let path = bloom_path(org_id, stream_type, stream_name, date_key, v_old);
+                let account = storage::get_account(org_id, &path).unwrap_or_default();
+                orphans.push(FileListDeleted {
+                    id: 0,
+                    account,
+                    file: path,
+                    index_file: false, // a `.bf` has no companion `.ttv`
+                    flattened: false,
+                });
+            }
+            Err(e) => {
+                log::warn!(
+                    "[BLOOM_CLEANUP] {org_id}/{stream_type}/{stream_name}/{date_key}: \
+                     bloom_ver_referenced({v_old}) failed: {e}; leaving `.bf` for retention"
+                );
+            }
+        }
+    }
+    if orphans.is_empty() {
+        return Ok(());
+    }
+
+    let created_at = now_micros();
+    let n = orphans.len();
+    if let Err(e) = infra_file_list::batch_add_deleted(org_id, created_at, &orphans).await {
+        log::warn!(
+            "[BLOOM_CLEANUP] {org_id}/{stream_type}/{stream_name}/{date_key}: \
+             enqueue {n} orphan `.bf` for deletion failed: {e}"
+        );
+    } else {
+        log::info!(
+            "[BLOOM_CLEANUP] {org_id}/{stream_type}/{stream_name}/{date_key}: \
+             enqueued {n} orphan `.bf` for deletion"
+        );
+    }
+    Ok(())
+}

src/compaction/src/bloom/mod.rs

@@ -0,0 +1,26 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Bloom-filter build side for compaction.
+//!
+//! - [`builder`] extracts per-(file, field) SBBFs from a tantivy term dictionary.
+//! - [`compact`] is the compactor entry point that owns "which files to bloom" and writes the
+//!   transposed `.bf` for each hour bucket.
+//!
+//! The search-side bloom pruner lives in the search crate, while the underlying
+//! SBBF format and reader/writer live in `infra::bloom`.
+
+mod builder;
+pub(crate) mod compact;

src/compaction/src/lib.rs

@@ -32,6 +32,7 @@ use infra::{
 use o2_enterprise::enterprise::common::downsampling::get_matching_downsampling_rules;
 use tokio::sync::mpsc;
 
+mod bloom;
 pub mod deleted;
 pub mod dump;
 pub mod flatten;

src/compaction/src/merge.rs

@@ -620,7 +620,32 @@ pub async fn merge_by_stream(
         orphan_blooms.extend(task.await??);
     }
 
-    let _ = (is_incremental, orphan_blooms);
+    // Build bloom filters for the current hour. Failures are non-fatal: files
+    // left at bloom_ver = 0 fall back to the regular Tantivy search path.
+    let build_start = std::time::Instant::now();
+    match crate::bloom::compact::build_for_stream(
+        org_id,
+        stream_type,
+        stream_name,
+        &date_start,
+        is_incremental,
+        orphan_blooms,
+    )
+    .await
+    {
+        Ok(false) => {}
+        Ok(true) => {
+            let build_time = build_start.elapsed().as_millis();
+            log::info!(
+                "[COMPACTOR] bloom build for {org_id}/{stream_type}/{stream_name}/{date_start} took: {build_time} ms"
+            );
+        }
+        Err(e) => {
+            log::warn!(
+                "[COMPACTOR] bloom build for {org_id}/{stream_type}/{stream_name}/{date_start} failed: {e}"
+            );
+        }
+    }
 
     // update job status
     if let Err(e) = infra_file_list::set_job_done(&[job_id]).await {

src/config/Cargo.toml

@@ -35,6 +35,7 @@ datafusion.workspace = true
 dashmap.workspace = true
 dotenv_config.workspace = true
 dotenvy.workspace = true
+fastdivide.workspace = true
 faststr.workspace = true
 futures.workspace = true
 local-ip-address.workspace = true

src/config/src/config.rs

@@ -52,7 +52,7 @@ pub type RwAHashSet<K> = tokio::sync::RwLock<HashSet<K>>;
 pub type RwBTreeMap<K, V> = tokio::sync::RwLock<BTreeMap<K, V>>;
 
 // for DDL commands and migrations
-pub const DB_SCHEMA_VERSION: u64 = 62;
+pub const DB_SCHEMA_VERSION: u64 = 63;
 pub const DB_SCHEMA_KEY: &str = "/db_schema_version/";
 
 // global version variables
@@ -1577,6 +1577,18 @@ pub struct Common {
     pub dashboard_placeholder: String,
     #[env_config(name = "ZO_AGGREGATION_TOPK_ENABLED", default = true)]
     pub aggregation_topk_enabled: bool,
+    #[env_config(
+        name = "ZO_DF_USE_AGG_TOPK_HEAP",
+        default = true,
+        help = "Use the heap implementation for eligible aggregate TopK plans"
+    )]
+    pub use_agg_topk_heap: bool,
+    #[env_config(
+        name = "ZO_DF_TOPK_HEAP_MAX_LIMIT",
+        default = 500,
+        help = "Maximum aggregate TopK limit that uses the heap implementation"
+    )]
+    pub agg_topk_heap_max_limit: u64,
     #[env_config(name = "ZO_SEARCH_INSPECTOR_ENABLED", default = false)]
     pub search_inspector_enabled: bool,
     #[env_config(name = "ZO_UTF8_VIEW_ENABLED", default = true)]

src/config/src/meta/slo/mod.rs

@@ -1796,13 +1796,6 @@ mod tests {
         ));
     }
 
-    // ---- time-slice threshold finiteness ------------------------------------
-
-    /// The threshold decides whether every bucket is good or bad. `NaN`
-    /// compares false against everything, so every slice classifies bad;
-    /// `±inf` classifies every slice the same way in the other direction.
-    /// Either way the SLO reports a confident, uniform, wrong answer.
-    #[test]
     /// A create request must not have to invent server-assigned fields.
     #[test]
     fn an_slo_deserializes_without_server_assigned_fields() {
@@ -1849,6 +1842,12 @@ mod tests {
         assert_eq!(SliType::from_storage_id(4), None);
     }
 
+    // ---- time-slice threshold finiteness ------------------------------------
+
+    /// The threshold decides whether every bucket is good or bad. `NaN`
+    /// compares false against everything, so every slice classifies bad;
+    /// `±inf` classifies every slice the same way in the other direction.
+    /// Either way the SLO reports a confident, uniform, wrong answer.
     #[test]
     fn a_non_finite_time_slice_threshold_is_rejected() {
         for bad in [f64::NAN, f64::INFINITY, f64::NEG_INFINITY] {

src/config/src/meta/synthetics.rs

@@ -124,10 +124,10 @@ pub struct Synthetic {
     #[serde(default)]
     pub tags: Vec<String>,
     #[serde(rename = "type")]
-    pub monitor_type: SyntheticType,
+    pub check_type: SyntheticType,
     /// Target URL (HTTP/Browser) or host:port (TCP/TLS/SSH).
     pub target: String,
-    /// Type-specific config, stored as JSONB. Shape depends on monitor_type.
+    /// Type-specific config, stored as JSONB. Shape depends on check_type.
     pub config: serde_json::Value,
     /// Schedule — same modular format as reports frequency.
     pub frequency: SyntheticFrequency,
@@ -148,10 +148,10 @@ pub struct Synthetic {
     /// Silence period (minutes) between repeated alert notifications.
     #[serde(default, alias = "cooldown_secs")]
     pub cooldown_mins: i32,
-    /// Collect RUM data for browser monitors (session replay / performance).
+    /// Collect RUM data for browser checks (session replay / performance).
     #[serde(default)]
     pub collect_rum_data: bool,
-    /// Enable session replay capture (browser monitors only).
+    /// Enable session replay capture (browser checks only).
     #[serde(default)]
     pub session_replay: bool,
     /// Optional authentication config (basic auth, bearer token, etc.).
@@ -217,7 +217,30 @@ pub enum SyntheticType {
     Dns,
 }
 
+/// Retry ceiling for browser checks. Lower than the protocol types because a
+/// browser run costs `devices x attempts x journey_budget`: at 3 retries a ~100s
+/// journey already reaches the browser Lambda's 303s function timeout, so the
+/// config would validate and then be killed mid-journey — reporting a failure
+/// the target never had.
+pub const MAX_BROWSER_RETRIES: i32 = 2;
+
+/// Retry ceiling for the protocol types. One attempt is a single request, so the
+/// worst case is bounded by `timeout_ms` and stays well inside the budget.
+pub const MAX_NET_RETRIES: i32 = 3;
+
 impl SyntheticType {
+    /// Largest `retries` value this type accepts.
+    ///
+    /// Note the product default is **0** for every type, and that is deliberate:
+    /// retries mask real failures, so opting in is the user's choice. This is
+    /// only the ceiling on that choice.
+    pub fn max_retries(&self) -> i32 {
+        match self {
+            Self::Browser => MAX_BROWSER_RETRIES,
+            _ => MAX_NET_RETRIES,
+        }
+    }
+
     /// JSON paths inside this type's `config` blob whose string values are
     /// credentials and must be AES-encrypted at rest (and decrypted on read).
     ///
@@ -440,7 +463,7 @@ pub struct SyntheticVariable {
 
 // ── Settings (packed into the `settings` JSON column) ────────────────────────
 
-/// Non-type-specific monitor settings stored as a single `settings` JSON blob.
+/// Non-type-specific check settings stored as a single `settings` JSON blob.
 /// auth and variables are stored in their own dedicated encrypted TEXT columns, not here.
 #[derive(Debug, Clone, Serialize, Deserialize, Default)]
 pub struct SyntheticSettings {
@@ -489,7 +512,7 @@ pub struct SyntheticListItem {
     pub description: String,
     pub tags: Vec<String>,
     #[serde(rename = "type")]
-    pub monitor_type: SyntheticType,
+    pub check_type: SyntheticType,
     pub target: String,
     pub frequency: SyntheticFrequency,
     pub locations: Vec<String>,
@@ -509,7 +532,7 @@ pub struct SyntheticListItem {
 #[derive(Debug, Clone, Deserialize, Default)]
 pub struct ListSyntheticsParams {
     pub folder_id: Option<String>,
-    pub monitor_type: Option<SyntheticType>,
+    pub check_type: Option<SyntheticType>,
     pub enabled: Option<bool>,
     pub location: Option<String>,
     pub tag: Option<String>,
@@ -519,7 +542,7 @@ pub struct ListSyntheticsParams {
 
 #[derive(Debug, Serialize, ToSchema)]
 pub struct SyntheticListResponse {
-    pub monitors: Vec<SyntheticListItem>,
+    pub checks: Vec<SyntheticListItem>,
     pub total: i64,
 }
 
@@ -623,7 +646,7 @@ pub struct SshAuth {
 //
 // The retired version-1 step was untyped JSON with a single `selector` and a
 // recorder-stamped `timeout_ms`. This typed, server-validated structure replaced
-// it, and is now the only shape a monitor can hold.
+// it, and is now the only shape a check can hold.
 //
 // The envelope is defined ONCE, complete, even though later phases populate
 // parts of it: `settle.navigation` (Phase 3), `settle.responses` (Phase 4),
@@ -773,7 +796,7 @@ pub struct BrowserStepV2 {
     pub timeout_ms: Option<u32>,
 }
 
-/// A (browser, device) pair for browser monitor fan-out.
+/// A (browser, device) pair for browser check fan-out.
 #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
 pub struct BrowserDevice {
     /// "chromium" | "firefox" | "edge"
@@ -803,7 +826,7 @@ pub struct BrowserConfig {
     /// retry sequence cannot outlive it — see `validate_browser_config`.
     pub journey_budget_ms: Option<u32>,
     pub capture: Option<BrowserCapture>,
-    /// The DOM attribute the recorder selects on for this monitor.
+    /// The DOM attribute the recorder selects on for this check.
     ///
     /// Absent means [`DEFAULT_TEST_ID_ATTR`]. It exists because the attribute is
     /// a property of the application under test, not of OpenObserve: Playwright
@@ -815,7 +838,7 @@ pub struct BrowserConfig {
     /// application outside that list produces NO `test_attribute` candidates at
     /// all and every step degrades to role/text/css without any error.
     ///
-    /// Recorded per monitor rather than per org so a journey that was recorded
+    /// Recorded per check rather than per org so a journey that was recorded
     /// against one application keeps working when another is added.
     pub test_id_attr: Option<String>,
 }
@@ -935,20 +958,115 @@ const MAX_JOURNEY_BUDGET_MS: u32 = 900_000;
 /// case inside this number, which is in turn what lets the server lease for it
 /// unconditionally.
 ///
-/// NOTE: the AWS Lambda function timeout must be >= this value, or runs are
-/// killed mid-journey. That setting lives outside this repository and cannot be
-/// asserted here. 900s is also AWS's maximum, so raising `retries` or
-/// `journey_budget_ms` further requires re-deriving all three together.
-pub const JOB_LEASE_SECS: i64 = 900;
+/// NOTE: the AWS Lambda function timeout must be >= `max_check_budget_secs`, or
+/// runs are killed mid-journey. That setting lives outside this repository — it
+/// is applied by the probe deploy scripts — so it cannot be asserted here.
+/// 900s is also AWS's maximum.
+pub const DEFAULT_JOB_LEASE_SECS: i64 = 900;
+
+/// Ceiling on a check's worst-case run, in seconds. Deliberately **below**
+/// `job_lease_secs`: the gap is what dispatch and the ack need, because a run
+/// finishing exactly at the function timeout still has to report before the
+/// reaper assumes the probe is gone.
+pub const DEFAULT_MAX_CHECK_BUDGET_SECS: i64 = 840;
 
 /// Ceiling for ONE attempt of a non-browser check, in milliseconds.
 ///
 /// Net `timeout_ms` was previously unbounded: every protocol config defaults it
 /// to 10s, but nothing rejected `timeout_ms: 3_600_000`, so the worst case of a
-/// retry sequence had no upper limit and could not be checked against the lease.
-const MAX_NET_TIMEOUT_MS: u32 = 300_000;
+/// retry sequence had no upper limit and could not be checked against the budget.
+pub const DEFAULT_MAX_NET_TIMEOUT_MS: u32 = 300_000;
 const MIN_NET_TIMEOUT_MS: u32 = 1_000;
 
+/// The three stacked bounds, tunable by deployment.
+///
+/// ```text
+/// check worst case  <=  max_check_budget_secs  <  job_lease_secs
+///                              840s                   900s
+///                       (also the Lambda
+///                        function timeout)
+/// ```
+///
+/// Synthetics is enterprise-only, so the values are declared in
+/// `o2_enterprise`'s `SyntheticsConfig` (`O2_SYNTHETICS_*` env vars) and pushed
+/// in here at startup by [`init_limits`]. This crate cannot read them directly —
+/// `config` has no dependency on `o2_enterprise` — so the holder below is the
+/// seam, and it falls back to the `DEFAULT_*` values in OSS builds and in tests.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct SyntheticsLimits {
+    pub job_lease_secs: i64,
+    pub max_check_budget_secs: i64,
+    pub max_net_timeout_ms: u32,
+}
+
+impl Default for SyntheticsLimits {
+    fn default() -> Self {
+        Self {
+            job_lease_secs: DEFAULT_JOB_LEASE_SECS,
+            max_check_budget_secs: DEFAULT_MAX_CHECK_BUDGET_SECS,
+            max_net_timeout_ms: DEFAULT_MAX_NET_TIMEOUT_MS,
+        }
+    }
+}
+
+impl SyntheticsLimits {
+    /// Rejects a set of limits that cannot hold together.
+    ///
+    /// The ordering is the whole point: a budget at or above the lease means a
+    /// check can be accepted, run to its limit, and still have its ack rejected
+    /// as stale — which surfaces to the user as a failure their target never
+    /// had. Operators own these values; this only refuses combinations that
+    /// cannot work.
+    pub fn validate(&self) -> Result<(), String> {
+        if self.max_check_budget_secs >= self.job_lease_secs {
+            return Err(format!(
+                "O2_SYNTHETICS_MAX_CHECK_BUDGET_SECS ({}) must be strictly less than \
+                 O2_SYNTHETICS_JOB_LEASE_SECS ({}) — the gap is what dispatch and the ack need. \
+                 A run that finishes at the budget still has to report before the reaper assumes \
+                 the probe is gone.",
+                self.max_check_budget_secs, self.job_lease_secs
+            ));
+        }
+        if self.max_check_budget_secs <= 0 || self.job_lease_secs <= 0 {
+            return Err("synthetics limits must be positive".to_string());
+        }
+        if self.max_net_timeout_ms as i64 > self.max_check_budget_secs * 1_000 {
+            return Err(format!(
+                "O2_SYNTHETICS_MAX_NET_TIMEOUT_MS ({}) exceeds the check budget ({}s) — a single \
+                 attempt could never fit",
+                self.max_net_timeout_ms, self.max_check_budget_secs
+            ));
+        }
+        Ok(())
+    }
+}
+
+static LIMITS: std::sync::OnceLock<SyntheticsLimits> = std::sync::OnceLock::new();
+
+/// Installs deployment-configured limits. Called once from `init_enterprise`.
+///
+/// **Not fatal.** A bad synthetics ceiling must not stop the whole application
+/// from starting — synthetics is one feature, and o2 serving ingest, search and
+/// dashboards matters more than it. On rejection nothing is installed, so
+/// [`limits`] keeps returning the `DEFAULT_*` values, which are known to hold
+/// together. The caller logs the error; an operator fixes the env var and
+/// restarts.
+///
+/// Falling back rather than accepting is the safe direction: the defaults are
+/// conservative, whereas an invalid pair (say budget == lease) is what silently
+/// converts healthy targets into alerts.
+pub fn init_limits(limits: SyntheticsLimits) -> Result<(), String> {
+    limits.validate()?;
+    let _ = LIMITS.set(limits);
+    Ok(())
+}
+
+/// The active limits — deployment-configured when enterprise has initialised,
+/// otherwise the `DEFAULT_*` values.
+pub fn limits() -> SyntheticsLimits {
+    LIMITS.get().copied().unwrap_or_default()
+}
+
 /// Worst-case wall clock for one leased job, in milliseconds.
 ///
 /// Retries happen INSIDE the leased job, so the lease has to cover the whole
@@ -1009,16 +1127,18 @@ fn validate_net_retry_budget(
         .and_then(|v| u32::try_from(v).ok())
         .unwrap_or_else(default_timeout_ms);
 
-    if !(MIN_NET_TIMEOUT_MS..=MAX_NET_TIMEOUT_MS).contains(&timeout_ms) {
+    let max_net_timeout_ms = limits().max_net_timeout_ms;
+    if !(MIN_NET_TIMEOUT_MS..=max_net_timeout_ms).contains(&timeout_ms) {
         return Err(format!(
-            "config.timeout_ms: must be {MIN_NET_TIMEOUT_MS}..={MAX_NET_TIMEOUT_MS}, got {timeout_ms}"
+            "config.timeout_ms: must be {MIN_NET_TIMEOUT_MS}..={max_net_timeout_ms}, got {timeout_ms}"
         ));
     }
 
+    let budget_secs = limits().max_check_budget_secs;
     let worst_case_ms = worst_case_run_ms(timeout_ms, 1, retries, wait_before_retry_secs);
-    if worst_case_ms > JOB_LEASE_SECS * 1_000 {
-        // Same shape as the browser message above: remedy first, arithmetic
-        // behind it, durations rather than raw milliseconds.
+    // Bound is the CHECK BUDGET, not the lease (ours). Wording is main's:
+    // remedy first, durations rather than raw milliseconds.
+    if worst_case_ms > budget_secs * 1_000 {
         let attempts = retries + 1;
         let retries_fix = if retries > 0 {
             format!("lower retries below {retries}, ")
@@ -1026,12 +1146,12 @@ fn validate_net_retry_budget(
             String::new()
         };
         return Err(format!(
-            "config: this check needs up to {} per run, which is over the {} job lease. To fix it, \
-             {retries_fix}or lower config.timeout_ms (currently {}). Detail: {} attempt(s) x {} \
-             each, plus {}s between retries. A check that outlives its lease has its job \
-             terminated mid-run and its real result rejected as a stale ack.",
+            "config: this check needs up to {} per run, which is over the {} check budget. To fix \
+             it, {retries_fix}or lower config.timeout_ms (currently {}). Detail: {} attempt(s) x \
+             {} each, plus {}s between retries. A check that outlives the budget is killed mid-run \
+             by the probe's function timeout and reports a failure the target never had.",
             human_ms(worst_case_ms),
-            human_ms(JOB_LEASE_SECS * 1_000),
+            human_ms(budget_secs * 1_000),
             human_ms(i64::from(timeout_ms)),
             attempts,
             human_ms(i64::from(timeout_ms)),
@@ -1042,7 +1162,7 @@ fn validate_net_retry_budget(
 }
 
 /// The complete v2 action vocabulary — exactly Playwright's recorder action
-/// model, minus what a monitor cannot use.
+/// model, minus what a check cannot use.
 ///
 /// Deliberately excludes `hover`, `scroll`, `wait`/`waitFor` and `screenshot`:
 /// upstream `ActionName` has no counterpart for any of them, so the recorder
@@ -1107,7 +1227,7 @@ const LOCATOR_ORIGINS: &[&str] = &["recorded", "authored", "composite"];
 /// How one part of a combined locator attaches to the part before it.
 ///
 /// Named after Playwright's own operations rather than CSS's, because the
-/// stored value IS a Playwright selector string and anyone debugging a monitor
+/// stored value IS a Playwright selector string and anyone debugging a check
 /// reads Playwright's documentation: `and` is `.and(b)`, `has` and `has_not`
 /// are `.filter({ has })` / `.filter({ hasNot })`, `descendant` is `.locator(b)`.
 ///
@@ -1117,11 +1237,11 @@ const LOCATOR_ORIGINS: &[&str] = &["recorded", "authored", "composite"];
 /// order and so destroys the preference the ordered bundle exists to express.
 const COMPOSITE_RELATIONS: &[&str] = &["and", "has", "has_not", "descendant"];
 
-/// The recorder's test-id attribute when a monitor does not set one.
+/// The recorder's test-id attribute when a check does not set one.
 ///
 /// `data-test` rather than Playwright's `data-testid`: OpenObserve's own
 /// frontend marks interactive elements with it, and self-monitoring is this
-/// feature's acceptance test (X-1's o2.introspect monitors).
+/// feature's acceptance test (X-1's o2.introspect checks).
 pub const DEFAULT_TEST_ID_ATTR: &str = "data-test";
 
 /// Longest attribute name accepted. A DOM attribute name this long is not a
@@ -1131,12 +1251,12 @@ const MAX_SETTLE_RESPONSES: usize = 5;
 const MAX_TAGS: usize = 20;
 const MAX_VARIABLES: usize = 50;
 const MAX_BROWSER_DEVICE_COMBOS: usize = 12;
-/// Minimum schedule interval (seconds) for protocol monitors (http/tcp/ping/…).
+/// Minimum schedule interval (seconds) for protocol checks (http/tcp/ping/…).
 /// Ping-style checks legitimately run at 1s granularity.
 /// NOTE: the scheduler ticks every 5s, so sub-5s intervals fire at tick
 /// resolution — allowed here, but effective cadence is bounded by the tick.
 const MIN_INTERVAL_SECS: i64 = 1;
-/// Minimum schedule interval (seconds) for browser monitors — each fire costs
+/// Minimum schedule interval (seconds) for browser checks — each fire costs
 /// one Lambda invocation per location per browser×device combo.
 const MIN_BROWSER_INTERVAL_SECS: i64 = 60;
 
@@ -1181,11 +1301,11 @@ fn location_allowed(loc: &str, allowed: &[String]) -> bool {
             && allowed.iter().any(|a| a == &format!("aws-{loc}"))
 }
 
-/// A save-time warning: the monitor is accepted, but something about it is worth
+/// A save-time warning: the check is accepted, but something about it is worth
 /// telling the author.
 ///
 /// Separate from the `Err(String)` channel on purpose. A zero-assertion journey
-/// is legitimate — a monitor that only navigates still proves the site answers —
+/// is legitimate — a check that only navigates still proves the site answers —
 /// so refusing it would be wrong; but it can also click its way through a broken
 /// application and pass, which is worth saying out loud (P5.2.4).
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
@@ -1203,14 +1323,14 @@ pub struct SyntheticWarning {
 }
 
 impl Synthetic {
-    /// Non-blocking problems worth surfacing when a monitor is saved.
+    /// Non-blocking problems worth surfacing when a check is saved.
     ///
     /// Deliberately not part of `validate`: everything here is accepted. A caller
     /// that ignores this returns exactly the behaviour it had before.
     pub fn warnings(&self) -> Vec<SyntheticWarning> {
         let mut warnings = Vec::new();
 
-        if self.monitor_type == SyntheticType::Browser {
+        if self.check_type == SyntheticType::Browser {
             let has_assertion = self
                 .config
                 .get("steps")
@@ -1244,8 +1364,8 @@ impl Synthetic {
     /// Returns the first problem found as `Err(message)`; messages are safe to
     /// return verbatim in a 400 response.
     /// `is_create`: the `start` freshness check only applies on create — edits
-    /// round-trip the monitor's original start date, which is legitimately in
-    /// the past for any monitor older than the grace window.
+    /// round-trip the check's original start date, which is legitimately in
+    /// the past for any check older than the grace window.
     pub fn validate(
         &self,
         allowed_locations: &[String],
@@ -1279,7 +1399,7 @@ impl Synthetic {
         }
 
         // ── target ─────────────────────────────────────────────────────────
-        match self.monitor_type {
+        match self.check_type {
             SyntheticType::Http | SyntheticType::Api | SyntheticType::Browser => {
                 validate_http_url("target", &self.target)?
             }
@@ -1313,16 +1433,16 @@ impl Synthetic {
                         self.frequency.interval
                     ));
                 }
-                let min_secs = if self.monitor_type == SyntheticType::Browser {
+                let min_secs = if self.check_type == SyntheticType::Browser {
                     MIN_BROWSER_INTERVAL_SECS
                 } else {
                     MIN_INTERVAL_SECS
                 };
                 if self.frequency.interval_secs() < min_secs {
                     return Err(format!(
-                        "frequency: interval too short ({}s < {min_secs}s minimum for {:?} monitors)",
+                        "frequency: interval too short ({}s < {min_secs}s minimum for {:?} checks)",
                         self.frequency.interval_secs(),
-                        self.monitor_type
+                        self.check_type
                     ));
                 }
             }
@@ -1355,8 +1475,19 @@ impl Synthetic {
         }
 
         // ── retry / alert settings ─────────────────────────────────────────
-        if !(0..=3).contains(&self.retries) {
-            return Err(format!("retries: must be 0..=3, got {}", self.retries));
+        // Browser is capped lower than the protocol types, and the cap is
+        // load-bearing rather than a preference. A browser run is
+        // devices x attempts x journey_budget, so at 3 retries a ~100s journey
+        // already reaches the browser function's 303s timeout — the config
+        // would validate and then be killed mid-journey, reporting a failure
+        // the target never had. Raise this only together with the deployed
+        // function timeout and MAX_CHECK_BUDGET_SECS.
+        let max_retries = self.check_type.max_retries();
+        if !(0..=max_retries).contains(&self.retries) {
+            return Err(format!(
+                "retries: must be 0..={max_retries} for {:?} checks, got {}",
+                self.check_type, self.retries
+            ));
         }
         if !(0..=300).contains(&self.wait_before_retry_secs) {
             return Err(format!(
@@ -1450,13 +1581,13 @@ impl Synthetic {
         self.validate_config(allowed_browsers, allowed_devices)
     }
 
-    /// Parses `config` into the struct matching `monitor_type` and validates it.
+    /// Parses `config` into the struct matching `check_type` and validates it.
     fn validate_config(
         &self,
         allowed_browsers: &[String],
         allowed_devices: &[String],
     ) -> Result<(), String> {
-        let type_check = match self.monitor_type {
+        let type_check = match self.check_type {
             SyntheticType::Browser => {
                 let cfg: BrowserConfig = serde_json::from_value(self.config.clone())
                     .map_err(|e| format!("config: not a valid browser config: {e}"))?;
@@ -1542,7 +1673,7 @@ impl Synthetic {
         // browser path applies inside `validate_browser_config`. Done here, once,
         // rather than in each arm: the arms are per-type and this rule is not, and
         // adding a check type should not be able to opt out of it silently.
-        if self.monitor_type != SyntheticType::Browser {
+        if self.check_type != SyntheticType::Browser {
             validate_net_retry_budget(&self.config, self.retries, self.wait_before_retry_secs)?;
         }
         Ok(())
@@ -1846,13 +1977,12 @@ fn validate_browser_config(
     // duplicate. Which is verbatim what the LEASE_SECS comment in
     // `dispatcher/mod.rs` was written to prevent.
     let devices = i64::try_from(cfg.browser_devices.len().max(1)).unwrap_or(1);
+    let budget_secs = limits().max_check_budget_secs;
     let worst_case_ms = worst_case_run_ms(budget_ms, devices, retries, wait_before_retry_secs);
-    if worst_case_ms > JOB_LEASE_SECS * 1_000 {
-        // Remedy FIRST, in terms the form actually offers. The previous wording
-        // led with "Lower journey_budget_ms" — a field the UI neither renders
-        // nor sends — so the one lever named first was the one the reader could
-        // not reach, and the arithmetic came before the instruction. The numbers
-        // are unchanged, just moved behind the fix and rendered as durations.
+    // Bound is the CHECK BUDGET, not the lease (ours). Wording is main's:
+    // remedy first, and journey_budget_ms named last because the UI does not
+    // render it.
+    if worst_case_ms > budget_secs * 1_000 {
         let attempts = retries + 1;
         let combos_fix = if devices > 1 {
             format!("drop a combo from config.browser_devices (currently {devices}), ")
@@ -1865,13 +1995,13 @@ fn validate_browser_config(
             String::new()
         };
         return Err(format!(
-            "config: this check needs up to {} per run, which is over the {} job lease. To fix it, \
-             {combos_fix}{retries_fix}or shorten the run with config.journey_budget_ms (currently \
-             {}). Detail: {devices} browser/device combo(s) x {attempts} attempt(s) x {} each, \
-             plus {}s between retries. A run that outlives its lease is requeued and executed a \
-             second time.",
+            "config: this check needs up to {} per run, which is over the {} check budget. To fix \
+             it, {combos_fix}{retries_fix}or shorten the run with config.journey_budget_ms \
+             (currently {}). Detail: {devices} browser/device combo(s) x {attempts} attempt(s) x \
+             {} each, plus {}s between retries. A run that outlives the budget is killed \
+             mid-journey by the probe's function timeout.",
             human_ms(worst_case_ms),
-            human_ms(JOB_LEASE_SECS * 1_000),
+            human_ms(budget_secs * 1_000),
             human_ms(i64::from(budget_ms)),
             human_ms(i64::from(budget_ms)),
             wait_before_retry_secs,
@@ -2007,6 +2137,82 @@ fn validate_browser_devices_and_schedule(
     Ok(())
 }
 
+#[cfg(test)]
+mod limits_tests {
+    use super::*;
+
+    #[test]
+    fn defaults_hold_together() {
+        SyntheticsLimits::default()
+            .validate()
+            .expect("shipped defaults must be a valid combination");
+    }
+
+    #[test]
+    fn budget_equal_to_lease_is_rejected() {
+        // The gap is what dispatch and the ack need. Equal means a run that
+        // uses its full budget cannot report before the reaper requeues it.
+        let l = SyntheticsLimits {
+            job_lease_secs: 900,
+            max_check_budget_secs: 900,
+            ..Default::default()
+        };
+        assert!(l.validate().is_err());
+    }
+
+    #[test]
+    fn budget_above_lease_is_rejected() {
+        let l = SyntheticsLimits {
+            job_lease_secs: 900,
+            max_check_budget_secs: 901,
+            ..Default::default()
+        };
+        assert!(l.validate().is_err());
+    }
+
+    #[test]
+    fn net_timeout_larger_than_the_budget_is_rejected() {
+        // One attempt could never fit, so every config of that type would fail
+        // validation for a reason the user cannot act on.
+        let l = SyntheticsLimits {
+            max_check_budget_secs: 10,
+            max_net_timeout_ms: 300_000,
+            ..Default::default()
+        };
+        assert!(l.validate().is_err());
+    }
+
+    #[test]
+    fn a_raised_but_still_ordered_pair_is_accepted() {
+        // Operators own these values; validation only refuses combinations that
+        // cannot work, not ones it merely dislikes.
+        let l = SyntheticsLimits {
+            job_lease_secs: 600,
+            max_check_budget_secs: 540,
+            max_net_timeout_ms: 120_000,
+        };
+        assert!(l.validate().is_ok());
+    }
+
+    #[test]
+    fn limits_fall_back_to_defaults_when_uninitialised() {
+        // Tests and OSS builds never call init_limits, so this is the path the
+        // whole validation suite actually runs on.
+        assert_eq!(limits(), SyntheticsLimits::default());
+    }
+
+    #[test]
+    fn browser_retries_are_capped_lower_than_net() {
+        assert_eq!(SyntheticType::Browser.max_retries(), MAX_BROWSER_RETRIES);
+        assert_eq!(SyntheticType::Http.max_retries(), MAX_NET_RETRIES);
+        assert!(
+            SyntheticType::Browser.max_retries() < SyntheticType::Http.max_retries(),
+            "browser is bounded by devices x attempts x journey budget, so its cap \
+             must stay below the protocol types'"
+        );
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -2248,7 +2454,7 @@ mod tests {
     fn valid_browser_synthetic() -> Synthetic {
         Synthetic {
             name: "login flow".to_string(),
-            monitor_type: SyntheticType::Browser,
+            check_type: SyntheticType::Browser,
             target: "https://example.com".to_string(),
             frequency: SyntheticFrequency {
                 frequency_type: SyntheticFrequencyType::Minutes,
@@ -2292,7 +2498,7 @@ mod tests {
     fn valid_tcp_synthetic() -> Synthetic {
         Synthetic {
             name: "db port".to_string(),
-            monitor_type: SyntheticType::Tcp,
+            check_type: SyntheticType::Tcp,
             target: "db.example.com".to_string(),
             frequency: SyntheticFrequency {
                 frequency_type: SyntheticFrequencyType::Minutes,
@@ -2310,16 +2516,20 @@ mod tests {
     }
 
     #[test]
-    fn net_retry_sequence_must_fit_the_job_lease() {
+    fn net_retry_sequence_must_fit_the_check_budget() {
         let (locs, brs, devs) = allowed();
-        // The lease covers the whole retry sequence because retries run inside
-        // the leased job. 4 x 300s alone is 1200s, past the 900s lease.
+        // The budget covers the whole retry sequence because retries run inside
+        // the leased job. 4 x 300s alone is 1200s, past the 840s budget.
+        //
+        // Validated against the BUDGET, not the lease: on the managed path the
+        // probe is a Lambda, and a function that hits its timeout is killed
+        // mid-run. Lease headroom is irrelevant once the process is gone.
         let mut s = valid_tcp_synthetic();
         s.config = serde_json::json!({ "port": 5432, "timeout_ms": 300_000 });
         s.retries = 3;
         s.wait_before_retry_secs = 0;
         let err = s.validate(&locs, &brs, &devs, true).unwrap_err();
-        assert!(err.contains("job lease"), "{err}");
+        assert!(err.contains("check budget"), "{err}");
 
         // The gaps count too: 3 x 250s = 750s of attempts is fine on its own,
         // but not with 300s of waiting between them.
@@ -2328,7 +2538,7 @@ mod tests {
         s.retries = 2;
         s.wait_before_retry_secs = 300;
         let err = s.validate(&locs, &brs, &devs, true).unwrap_err();
-        assert!(err.contains("job lease"), "{err}");
+        assert!(err.contains("check budget"), "{err}");
     }
 
     #[test]
@@ -2876,9 +3086,9 @@ mod tests {
 
     #[test]
     fn test_a_zero_assertion_journey_is_accepted_with_a_machine_readable_warning() {
-        // P5.2.4 — accepted, not refused: a monitor that only navigates still
+        // P5.2.4 — accepted, not refused: a check that only navigates still
         // proves the site answers. The warning is what stops it being mistaken
-        // for a monitor that checks something.
+        // for a check that checks something.
         let (locs, brs, devs) = allowed();
         let s = v2_synthetic(serde_json::json!([v2_nav_step(), v2_click_step()]));
         assert!(s.validate(&locs, &brs, &devs, true).is_ok());
@@ -3233,19 +3443,36 @@ mod tests {
     }
 
     #[test]
-    fn test_browser_journey_budget_exceeding_lease_rejected() {
+    fn test_browser_journey_budget_exceeding_check_budget_rejected() {
         let (locs, brs, devs) = allowed();
         let mut s = valid_browser_synthetic();
-        s.retries = 3;
+        // 2 is the browser cap; 3 would be rejected by the retries bound before
+        // ever reaching the budget arithmetic this test is about.
+        s.retries = MAX_BROWSER_RETRIES;
         s.wait_before_retry_secs = 5;
         s.config["journey_budget_ms"] = serde_json::json!(300_000);
-        // 4 * 300s + 15s = 1215s — well past the 900s lease.
+        // 3 * 300s + 10s = 910s — past the 840s check budget.
         let err = s.validate(&locs, &brs, &devs, true).unwrap_err();
         // The error must name all three inputs, or an operator cannot tell which
         // one to change.
         assert!(err.contains("journey_budget_ms"), "{err}");
         assert!(err.contains("retries"), "{err}");
-        assert!(err.contains("lease"), "{err}");
+        assert!(err.contains("check budget"), "{err}");
+    }
+
+    #[test]
+    fn browser_retries_above_the_cap_are_rejected() {
+        let (locs, brs, devs) = allowed();
+        let mut s = valid_browser_synthetic();
+        s.retries = MAX_BROWSER_RETRIES + 1;
+        let err = s.validate(&locs, &brs, &devs, true).unwrap_err();
+        assert!(err.contains("retries"), "{err}");
+
+        // The same value is fine on a protocol check, which is the point of the
+        // cap being per-type rather than global.
+        let mut n = valid_tcp_synthetic();
+        n.retries = MAX_BROWSER_RETRIES + 1;
+        assert!(n.validate(&locs, &brs, &devs, true).is_ok());
     }
 
     // The invariant above was per-DEVICE while the work is per-JOB: the probe runs
@@ -3278,7 +3505,9 @@ mod tests {
         assert!(err.contains("browser_devices"), "{err}");
         assert!(err.contains("journey_budget_ms"), "{err}");
         assert!(err.contains("retries"), "{err}");
-        assert!(err.contains("lease"), "{err}");
+        // "budget", not "lease": validation measures against the check budget so
+        // a config the probe's function timeout would kill is rejected on save.
+        assert!(err.contains("budget"), "{err}");
     }
 
     #[test]
@@ -3299,15 +3528,20 @@ mod tests {
     fn over_lease_message_leads_with_an_actionable_remedy() {
         let (locs, brs, devs) = allowed();
         let mut s = valid_browser_synthetic();
-        s.retries = 3;
+        // 2, not 3: browser retries are capped at MAX_BROWSER_RETRIES, so 3 is
+        // rejected by that rule first and never reaches the budget message.
+        s.retries = 2;
         s.wait_before_retry_secs = 5;
         s.config["journey_budget_ms"] = serde_json::json!(300_000);
         let err = s.validate(&locs, &brs, &devs, true).unwrap_err();
 
-        // 4 attempts x 300s + 3 gaps x 5s = 1215s.
+        // 3 attempts x 300s + 2 gaps x 5s = 910s.
         // Durations, not millisecond counts, for the two headline numbers.
-        assert!(err.contains("20m15s"), "worst case as a duration: {err}");
-        assert!(err.contains("15m job lease"), "limit as a duration: {err}");
+        assert!(err.contains("15m10s"), "worst case as a duration: {err}");
+        assert!(
+            err.contains("14m check budget"),
+            "limit as a duration: {err}"
+        );
         // The remedy precedes the arithmetic.
         let fix_at = err.find("To fix it").expect("names a fix");
         let detail_at = err.find("Detail:").expect("keeps the arithmetic");
@@ -3322,14 +3556,18 @@ mod tests {
 
     /// Two browser/device combos with the web form's own default `retries: 1`
     /// and the server default budget land at 2 x (2 x 300s + 5s) = 1210s, past
-    /// the 900s lease — so "Chromium desktop + Chromium mobile" cannot be saved
-    /// without the author first discovering they must set retries to 0.
+    /// the 840s check budget — so "Chromium desktop + Chromium mobile" cannot be
+    /// saved without the author first discovering they must set retries to 0.
     ///
     /// Asserted here so that changing any of the three defaults has to confront
     /// which configurations are savable, rather than shifting it silently.
+    ///
+    /// Measures against the BUDGET, not the lease: validation moved to the
+    /// budget so a check the probe's function timeout will kill is rejected at
+    /// save time rather than failing in production.
     #[test]
-    fn two_combos_at_default_retries_exceed_the_lease() {
-        let lease_ms = JOB_LEASE_SECS * 1_000;
+    fn two_combos_at_default_retries_exceed_the_budget() {
+        let lease_ms = DEFAULT_MAX_CHECK_BUDGET_SECS * 1_000;
         assert_eq!(
             worst_case_run_ms(DEFAULT_JOURNEY_BUDGET_MS, 2, 1, 5),
             1_210_000
@@ -3363,12 +3601,12 @@ mod tests {
         let mut s = valid_browser_synthetic();
         s.retries = 1;
         s.wait_before_retry_secs = 0;
-        // Exactly 2 * 450s = 900s — equal to the lease, which is permitted.
-        s.config["journey_budget_ms"] = serde_json::json!(450_000);
+        // Exactly 2 * 420s = 840s — equal to the check budget, which is permitted.
+        s.config["journey_budget_ms"] = serde_json::json!(420_000);
         assert!(s.validate(&locs, &brs, &devs, true).is_ok());
 
         // One millisecond more per attempt tips it over.
-        s.config["journey_budget_ms"] = serde_json::json!(450_001);
+        s.config["journey_budget_ms"] = serde_json::json!(420_001);
         assert!(s.validate(&locs, &brs, &devs, true).is_err());
     }
 
@@ -3408,7 +3646,7 @@ mod tests {
     fn test_validate_http_assertion_field_and_operator() {
         let (locs, brs, devs) = allowed();
         let mut s = valid_browser_synthetic();
-        s.monitor_type = SyntheticType::Http;
+        s.check_type = SyntheticType::Http;
         s.target = "https://example.com/".to_string();
         s.config = serde_json::json!({
             "method": "GET",
@@ -3435,7 +3673,7 @@ mod tests {
     fn test_validate_ssh_config_fields() {
         let (locs, brs, devs) = allowed();
         let mut s = valid_browser_synthetic();
-        s.monitor_type = SyntheticType::Ssh;
+        s.check_type = SyntheticType::Ssh;
         s.target = "test.rebex.net:22".to_string();
 
         s.config = serde_json::json!({
@@ -3618,9 +3856,9 @@ mod tests {
     fn test_validate_config_shape_mismatch() {
         let (locs, brs, devs) = allowed();
         let mut s = valid_browser_synthetic();
-        s.monitor_type = SyntheticType::Tcp;
+        s.check_type = SyntheticType::Tcp;
         s.target = "example.com:443".to_string();
-        // browser-shaped config on a tcp monitor → port missing → shape error
+        // browser-shaped config on a tcp check → port missing → shape error
         let err = s.validate(&locs, &brs, &devs, true).unwrap_err();
         assert!(err.contains("not a valid tcp config"), "{err}");
     }
@@ -3641,7 +3879,7 @@ mod tests {
     fn test_validate_http_ok() {
         let (locs, brs, devs) = allowed();
         let mut s = valid_browser_synthetic();
-        s.monitor_type = SyntheticType::Http;
+        s.check_type = SyntheticType::Http;
         s.config = serde_json::json!({ "method": "GET" });
         assert!(s.validate(&locs, &brs, &devs, true).is_ok());
     }

src/core/src/synthetics.rs

@@ -23,9 +23,9 @@
 #[cfg(feature = "enterprise")]
 pub struct CheckNotification {
     pub org_id: String,
-    pub monitor_name: String,
-    pub monitor_id: String,
-    pub monitor_type: String,
+    pub check_name: String,
+    pub check_id: String,
+    pub check_type: String,
     pub target: String,
     pub destinations: Vec<String>,
     pub run_id: String,
@@ -62,6 +62,14 @@ pub struct CheckNotification {
     /// region could only say "the check is failing" and the reader had to open
     /// the UI to find out where.
     pub failing_locations: Vec<String>,
+    /// Locations that passed, alphabetical.
+    ///
+    /// Carried because `failing_locations` is empty on a recovery **by
+    /// definition** — nothing is failing — which left the recovery message
+    /// unable to name anything and degrading to a bare count. With both sides
+    /// a partial recovery is expressible too: "2 of 3 recovered, the third is
+    /// still down".
+    pub passing_locations: Vec<String>,
 }
 
 /// Fires once per run (when all jobs have completed) for non-passing runs.
@@ -100,10 +108,7 @@ pub async fn notify_check_result(n: CheckNotification) {
                 };
 
                 let subject = if n.recovery {
-                    format!(
-                        "[OpenObserve Synthetics] ✅ {} has RECOVERED",
-                        n.monitor_name
-                    )
+                    format!("[OpenObserve Synthetics] ✅ {} has RECOVERED", n.check_name)
                 } else if n.degraded {
                     // Not "is WARNING": the point of the message is that this needs
                     // action before it becomes an outage. Named where we know the
@@ -113,21 +118,21 @@ pub async fn notify_check_result(n: CheckNotification) {
                     match n.status_reason.as_deref() {
                         Some("cert_expiring") => format!(
                             "[OpenObserve Synthetics] 🟡 {} — CERTIFICATE EXPIRING SOON",
-                            n.monitor_name
+                            n.check_name
                         ),
                         Some("sftp_degraded") => format!(
                             "[OpenObserve Synthetics] 🟡 {} — SFTP DEGRADED",
-                            n.monitor_name
+                            n.check_name
                         ),
-                        _ => format!("[OpenObserve Synthetics] 🟡 {} is DEGRADED", n.monitor_name),
+                        _ => format!("[OpenObserve Synthetics] 🟡 {} is DEGRADED", n.check_name),
                     }
                 } else if n.flaky {
-                    format!("[OpenObserve Synthetics] 🔁 {} is FLAKY", n.monitor_name)
+                    format!("[OpenObserve Synthetics] 🔁 {} is FLAKY", n.check_name)
                 } else {
                     format!(
                         "[OpenObserve Synthetics] {} {} is {}",
-                        status_emoji(&n.status),
-                        n.monitor_name,
+                        status_emoji(&n),
+                        n.check_name,
                         n.status.to_uppercase()
                     )
                 };
@@ -136,8 +141,8 @@ pub async fn notify_check_result(n: CheckNotification) {
                         .await
                 {
                     log::error!(
-                        "[synthetics] notify dest={dest_name} monitor={}: {e}",
-                        n.monitor_id
+                        "[synthetics] notify dest={dest_name} check={}: {e}",
+                        n.check_id
                     );
                 }
             }
@@ -149,10 +154,33 @@ pub async fn notify_check_result(n: CheckNotification) {
 }
 
 #[cfg(feature = "enterprise")]
-fn status_emoji(status: &str) -> &'static str {
-    match status {
-        "recovered" => "✅",
-        "failed" | "down" => "🔴",
+/// Emoji for a notification, branching on the **same flags** as
+/// [`status_headline`] and in the same order.
+///
+/// This used to take `&str` and branch on `status` alone, which put a 🔴 on
+/// every recovery: `status_headline` reads the `recovery` bool, but on a
+/// recovery run `status` is `"passed"` — the run genuinely did pass — so the
+/// emoji fell through to the catch-all. The headline said "has recovered" next
+/// to an outage marker, and in a busy channel that reads as a second outage.
+///
+/// The old `"recovered" => "✅"` arm was unreachable: nothing sets `status` to
+/// that literal. `AlertDecision::Recovered` becomes the `recovery` bool at the
+/// ack and never round-trips into the status string.
+///
+/// Taking the whole notification is what keeps the two in step — a future
+/// branch added to the headline is a compile-visible omission here, rather than
+/// a silently wrong glyph.
+fn status_emoji(n: &CheckNotification) -> &'static str {
+    if n.recovery {
+        return "✅";
+    }
+    if n.degraded {
+        return "🟡";
+    }
+    if n.flaky {
+        return "🔁";
+    }
+    match n.status.as_str() {
         "warning" => "🟡",
         "error" => "⚠️",
         _ => "🔴",
@@ -163,7 +191,7 @@ fn status_emoji(status: &str) -> &'static str {
 #[cfg(feature = "enterprise")]
 fn status_headline(n: &CheckNotification) -> String {
     if n.recovery {
-        return format!("{} has recovered", n.monitor_name);
+        return format!("{} has recovered", n.check_name);
     }
     // `warning` covers two unrelated things, and they need opposite responses:
     // a flaky run already fixed itself, a degrading target will not.
@@ -175,31 +203,31 @@ fn status_headline(n: &CheckNotification) -> String {
         return match n.status_reason.as_deref() {
             Some("cert_expiring") => format!(
                 "{} — the TLS certificate is expiring soon, renew it before it lapses",
-                n.monitor_name
+                n.check_name
             ),
             Some("sftp_degraded") => format!(
                 "{} connects and authenticates, but its SFTP subsystem is failing",
-                n.monitor_name
+                n.check_name
             ),
             _ => format!(
                 "{} is reachable but degrading — this needs attention before it fails",
-                n.monitor_name
+                n.check_name
             ),
         };
     }
     if n.flaky {
         return format!(
             "{} passed only after retries (flaky) — it recovered on its own",
-            n.monitor_name
+            n.check_name
         );
     }
     match n.status.as_str() {
-        "warning" => format!("{} passed only after retries (flaky)", n.monitor_name),
+        "warning" => format!("{} passed only after retries (flaky)", n.check_name),
         "error" => format!(
             "{} could not be checked — probe infrastructure error",
-            n.monitor_name
+            n.check_name
         ),
-        _ => format!("{} is failing", n.monitor_name),
+        _ => format!("{} is failing", n.check_name),
     }
 }
 
@@ -219,7 +247,7 @@ fn checked_at_utc(checked_at_micros: i64) -> String {
         .unwrap_or_else(|| checked_at_micros.to_string())
 }
 
-/// Deep link to the monitor's results page in the UI.
+/// Deep link to the check's results page in the UI.
 #[cfg(feature = "enterprise")]
 fn run_url(n: &CheckNotification) -> String {
     let cfg = config::get_config();
@@ -227,7 +255,7 @@ fn run_url(n: &CheckNotification) -> String {
     let base_uri = &cfg.common.base_uri;
     format!(
         "{web_url}{base_uri}/web/synthetic/{}/results?org_identifier={}",
-        n.monitor_id, n.org_id
+        n.check_id, n.org_id
     )
 }
 
@@ -239,6 +267,36 @@ fn run_url(n: &CheckNotification) -> String {
 /// six-of-six outage read identically.
 fn locations_line(n: &CheckNotification) -> String {
     let total = if n.job_count > 0 { n.job_count } else { 1 };
+
+    // On a recovery the interesting set is what came back, not what is broken —
+    // and `failing_locations` is empty by definition, which is what used to make
+    // this degrade to a bare count and tell the reader nothing.
+    if n.recovery {
+        return match (
+            n.passing_locations.is_empty(),
+            n.failing_locations.is_empty(),
+        ) {
+            // Nothing to name at all — an older ack, or the query failed.
+            (true, _) => total.to_string(),
+            // Full recovery.
+            (false, true) => format!(
+                "{} of {} recovered: {}",
+                n.passing_locations.len(),
+                total,
+                n.passing_locations.join(", ")
+            ),
+            // Partial: some came back, some did not. Naming both is the whole
+            // point — "2 of 3 recovered" alone would read as an all-clear.
+            (false, false) => format!(
+                "{} of {} recovered: {} — still failing: {}",
+                n.passing_locations.len(),
+                total,
+                n.passing_locations.join(", "),
+                n.failing_locations.join(", ")
+            ),
+        };
+    }
+
     if n.failing_locations.is_empty() {
         return total.to_string();
     }
@@ -256,9 +314,9 @@ fn locations_line(n: &CheckNotification) -> String {
 fn build_slack_json(n: &CheckNotification) -> String {
     let checked_secs = n.checked_at / 1_000_000;
     let mut lines = vec![
-        format!("{} *{}*", status_emoji(&n.status), status_headline(n)),
+        format!("{} *{}*", status_emoji(n), status_headline(n)),
         String::new(),
-        format!("*Monitor:* {} ({})", n.monitor_name, n.monitor_type),
+        format!("*Check:* {} ({})", n.check_name, n.check_type),
         format!("*Target:* {}", n.target),
         format!("*Locations:* {}", locations_line(n)),
     ];
@@ -278,7 +336,7 @@ fn build_slack_json(n: &CheckNotification) -> String {
 fn build_plain_text(n: &CheckNotification) -> String {
     let mut lines = vec![
         status_headline(n),
-        format!("Monitor: {} ({})", n.monitor_name, n.monitor_type),
+        format!("Check: {} ({})", n.check_name, n.check_type),
         format!("Target: {}", n.target),
         format!("Status: {}", n.status),
         format!("Locations: {}", locations_line(n)),
@@ -316,7 +374,7 @@ fn build_email_html(n: &CheckNotification) -> String {
         r#"<div style="font-family:sans-serif;max-width:560px;">
   <h2 style="color:{color};margin-bottom:4px;">{emoji} {headline}</h2>
   <table style="border-collapse:collapse;background:#f7f7f7;border-radius:6px;width:100%;">
-    <tr><td style="padding:6px 12px;color:#666;width:140px;">Monitor</td>
+    <tr><td style="padding:6px 12px;color:#666;width:140px;">Check</td>
         <td style="padding:6px 12px;">{name} ({mtype})</td></tr>
     <tr><td style="padding:6px 12px;color:#666;">Target</td>
         <td style="padding:6px 12px;">{target}</td></tr>
@@ -332,10 +390,10 @@ fn build_email_html(n: &CheckNotification) -> String {
     <a href="{url}" style="background:{color};color:#fff;padding:8px 16px;border-radius:4px;text-decoration:none;">View run details</a>
   </p>
 </div>"#,
-        emoji = status_emoji(&n.status),
+        emoji = status_emoji(n),
         headline = html_escape(&status_headline(n)),
-        name = html_escape(&n.monitor_name),
-        mtype = html_escape(&n.monitor_type),
+        name = html_escape(&n.check_name),
+        mtype = html_escape(&n.check_type),
         target = html_escape(&n.target),
         status = n.status.to_uppercase(),
         jobs = html_escape(&locations_line(n)),
@@ -362,9 +420,6 @@ fn html_escape(s: &str) -> String {
 /// Never-registered locations count as pending, not down.
 #[cfg(feature = "enterprise")]
 pub async fn location_staleness_watcher() {
-    use std::collections::HashSet;
-
-    let mut notified_down: HashSet<String> = HashSet::new();
     loop {
         tokio::time::sleep(std::time::Duration::from_secs(60)).await;
 
@@ -390,33 +445,46 @@ pub async fn location_staleness_watcher() {
                 .await
                 .unwrap_or_default();
             if !loc.enabled || agents.is_empty() {
-                notified_down.remove(&loc.id);
+                clear_down(&loc.id).await;
                 continue;
             }
             let any_live = agents.iter().any(|a| now - a.last_seen_at <= window_us);
             if any_live {
-                notified_down.remove(&loc.id);
-                continue;
-            }
-            if notified_down.contains(&loc.id) {
+                clear_down(&loc.id).await;
                 continue;
             }
 
             let conn = infra::db::ORM_CLIENT
                 .get_or_init(infra::db::connect_to_orm)
                 .await;
-            let checks = infra::table::synthetics_monitors::list_referencing_location(
-                conn, &org_id, &loc.id,
-            )
-            .await
-            .unwrap_or_default();
+            let checks =
+                infra::table::synthetics_checks::list_referencing_location(conn, &org_id, &loc.id)
+                    .await
+                    .unwrap_or_default();
             if checks.is_empty() {
                 // Nothing runs here — stay quiet, re-evaluate next tick.
                 continue;
             }
-            // Mark before dispatch so a location without destinations is still
-            // one-shot (no per-tick log spam / retry storm).
-            notified_down.insert(loc.id.clone());
+            // Claim before dispatch so a location without destinations is still
+            // one-shot (no per-tick log spam / retry storm), AND so that only one
+            // alert_manager speaks. This watcher runs on every alert_manager, so
+            // the suppression flag cannot live in this process's memory — N nodes
+            // would each believe they had not notified yet and send N pages for
+            // one outage. The CAS in `try_claim_down_notification` makes exactly
+            // one node the winner.
+            match infra::table::synthetics_locations::try_claim_down_notification(&loc.id, now)
+                .await
+            {
+                Ok(true) => {}
+                Ok(false) => continue, // another node is sending it
+                Err(e) => {
+                    log::error!(
+                        "[synthetics] staleness watcher: claim down notification for {}: {e}",
+                        loc.id
+                    );
+                    continue;
+                }
+            }
 
             let mut destinations: Vec<String> =
                 checks.iter().flat_map(|c| c.destinations.clone()).collect();
@@ -471,6 +539,164 @@ mod tests {
     fn checked_at_utc_falls_back_to_the_raw_value_when_out_of_range() {
         assert_eq!(checked_at_utc(i64::MAX), i64::MAX.to_string());
     }
+
+    /// A firing notification for a 3-location check, two of them broken.
+    fn firing() -> CheckNotification {
+        CheckNotification {
+            org_id: "default".into(),
+            check_name: "EU1 Cloud Health Check".into(),
+            check_id: "abc123".into(),
+            check_type: "http".into(),
+            target: "https://example.com".into(),
+            destinations: vec![],
+            run_id: "run1".into(),
+            status: "failed".into(),
+            job_count: 3,
+            error: None,
+            checked_at: 1_785_000_000_000_000,
+            recovery: false,
+            consecutive_failures: 3,
+            flaky: false,
+            status_reason: None,
+            degraded: false,
+            failing_locations: vec!["aws-us-east-1".into(), "aws-us-west-1".into()],
+            passing_locations: vec!["aws-eu-central-1".into()],
+        }
+    }
+
+    /// The same check recovering: status is "passed", nothing is failing.
+    fn recovered() -> CheckNotification {
+        CheckNotification {
+            status: "passed".into(),
+            recovery: true,
+            consecutive_failures: 0,
+            failing_locations: vec![],
+            passing_locations: vec![
+                "aws-eu-central-1".into(),
+                "aws-us-east-1".into(),
+                "aws-us-west-1".into(),
+            ],
+            ..firing()
+        }
+    }
+
+    // ── 2324-a · the emoji must agree with the headline ────────────────────
+
+    #[test]
+    fn recovery_is_not_marked_as_an_outage() {
+        // The bug: status is "passed" on a recovery, so branching on the status
+        // string fell through to the 🔴 catch-all while the headline said
+        // "has recovered". In a busy channel that reads as a second outage.
+        let n = recovered();
+        assert_eq!(status_emoji(&n), "✅");
+        assert!(status_headline(&n).contains("has recovered"));
+    }
+
+    #[test]
+    fn emoji_and_headline_agree_on_every_branch() {
+        // The two are only correct together; this is the invariant the old
+        // signature could not express.
+        let cases: Vec<(CheckNotification, &str, &str)> = vec![
+            (recovered(), "✅", "has recovered"),
+            (
+                CheckNotification {
+                    degraded: true,
+                    status: "warning".into(),
+                    status_reason: Some("cert_expiring".into()),
+                    ..firing()
+                },
+                "🟡",
+                "certificate is expiring",
+            ),
+            (
+                CheckNotification {
+                    flaky: true,
+                    status: "warning".into(),
+                    ..firing()
+                },
+                "🔁",
+                "flaky",
+            ),
+            (firing(), "🔴", "is failing"),
+            (
+                CheckNotification {
+                    status: "error".into(),
+                    ..firing()
+                },
+                "⚠️",
+                "could not be checked",
+            ),
+        ];
+        for (n, emoji, headline_fragment) in cases {
+            assert_eq!(status_emoji(&n), emoji, "status={}", n.status);
+            assert!(
+                status_headline(&n).contains(headline_fragment),
+                "status={} headline={}",
+                n.status,
+                status_headline(&n)
+            );
+        }
+    }
+
+    #[test]
+    fn degraded_outranks_flaky_exactly_as_the_headline_does() {
+        // Both arrive as `warning`. The order matters: a degrading target needs
+        // action, a flaky one already fixed itself.
+        let n = CheckNotification {
+            degraded: true,
+            flaky: true,
+            status: "warning".into(),
+            ..firing()
+        };
+        assert_eq!(status_emoji(&n), "🟡");
+    }
+
+    // ── 2324-b · a recovery must be able to name its locations ─────────────
+
+    #[test]
+    fn full_recovery_names_the_locations_that_came_back() {
+        // The bug: failing_locations is empty by definition on a recovery, so
+        // this used to render the bare count "3".
+        let line = locations_line(&recovered());
+        assert!(line.contains("3 of 3 recovered"), "{line}");
+        assert!(line.contains("aws-us-east-1"), "{line}");
+    }
+
+    #[test]
+    fn partial_recovery_names_both_sides() {
+        // "2 of 3 recovered" alone would read as an all-clear.
+        let n = CheckNotification {
+            recovery: true,
+            status: "passed".into(),
+            passing_locations: vec!["aws-us-east-1".into(), "aws-us-west-1".into()],
+            failing_locations: vec!["aws-eu-central-1".into()],
+            ..firing()
+        };
+        let line = locations_line(&n);
+        assert!(line.contains("2 of 3 recovered"), "{line}");
+        assert!(line.contains("still failing: aws-eu-central-1"), "{line}");
+    }
+
+    #[test]
+    fn recovery_with_no_location_data_falls_back_to_the_count() {
+        // An older ack, or the query failed. Better a bare count than a lie.
+        let n = CheckNotification {
+            recovery: true,
+            passing_locations: vec![],
+            failing_locations: vec![],
+            ..recovered()
+        };
+        assert_eq!(locations_line(&n), "3");
+    }
+
+    #[test]
+    fn firing_still_names_only_what_is_broken() {
+        // The passing set exists now, but a firing message must not list it —
+        // the reader wants the outage, not the healthy regions.
+        let line = locations_line(&firing());
+        assert!(line.starts_with("2 of 3: "), "{line}");
+        assert!(!line.contains("aws-eu-central-1"), "{line}");
+    }
 }
 
 /// Sends the "location down" notification to each destination, matching the
@@ -535,3 +761,14 @@ async fn notify_location_down(
         }
     }
 }
+
+/// Clears a location's down flag so a future outage notifies again.
+///
+/// Every alert_manager calls this on recovery; the underlying update is
+/// idempotent, so they cannot disagree.
+#[cfg(feature = "enterprise")]
+async fn clear_down(location_id: &str) {
+    if let Err(e) = infra::table::synthetics_locations::clear_down_notification(location_id).await {
+        log::error!("[synthetics] staleness watcher: clear down flag for {location_id}: {e}");
+    }
+}

src/db/src/folders.rs

@@ -291,7 +291,7 @@ pub async fn delete_folder(
                     folder_id: Some(folder_pk),
                     ..Default::default()
                 };
-                if table::synthetics_monitors::count(client, org_id, &params).await? > 0 {
+                if table::synthetics_checks::count(client, org_id, &params).await? > 0 {
                     return Err(FolderError::DeleteWithSynthetics);
                 }
             }

src/db/src/org_ingestion_tokens.rs

@@ -158,6 +158,11 @@ pub async fn watch() -> Result<(), anyhow::Error> {
                 let item_key = ev.key.strip_prefix(key).unwrap();
                 let parts: Vec<&str> = item_key.splitn(2, '/').collect();
                 if parts.len() == 2 {
+                    // Also drop the org -> default-token pick, which synthetics
+                    // reads once per job. Reusing this existing watch means a
+                    // rotate or disable on another node lands here without a
+                    // second event stream.
+                    org_ingestion_tokens::invalidate_default_cache(parts[0]);
                     // find_enabled_token only returns enabled tokens.
                     // If found → token is enabled → cache it.
                     // If not found → token is disabled/missing → remove from cache.
@@ -172,6 +177,9 @@ pub async fn watch() -> Result<(), anyhow::Error> {
             }
             db::Event::Delete(ev) => {
                 let item_key = ev.key.strip_prefix(key).unwrap();
+                if let Some((org_id, _)) = item_key.split_once('/') {
+                    org_ingestion_tokens::invalidate_default_cache(org_id);
+                }
                 ORG_INGESTION_TOKENS.remove(item_key);
             }
             db::Event::Empty => {}

src/infra/src/coordinator/mod.rs

@@ -22,6 +22,7 @@ pub mod model_pricing;
 pub mod org_status;
 pub mod pipelines;
 pub mod service_streams;
+pub mod synthetics;
 pub mod system_settings;
 
 pub async fn get_coordinator() -> &'static Box<dyn crate::db::Db> {

src/infra/src/coordinator/synthetics.rs

@@ -0,0 +1,205 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Cluster-coordinator events for the synthetics in-memory caches.
+//!
+//! Each node keeps its own copy of the synthetics configuration caches
+//! (`table::synthetics_checks`, `synthetics_locations`, `synthetics_probe_tokens`,
+//! `synthetics_agents`). A write only clears the cache on the node that
+//! performed it, so without a cross-node signal every other node keeps serving
+//! the old value until its TTL expires.
+//!
+//! This module is that signal, modelled on [`super::alerts`] and
+//! [`super::pipelines`]: writers call an `emit_*` function after committing to
+//! the database, and every node runs [`watch`], which invalidates the matching
+//! cache entry the moment the event arrives.
+//!
+//! # Key layout
+//!
+//! One watch prefix, with a `kind` segment so a single watcher serves all four
+//! caches:
+//!
+//! ```text
+//! /synthetics/check/{org_id}/{synthetics_id}   one check definition changed
+//! /synthetics/location/all                     the location registry changed
+//! /synthetics/token/{org_id}                   an org's probe tokens changed
+//! /synthetics/agent/{agent_id}                 one agent re-registered
+//! ```
+//!
+//! `location` and `token` events are coarse on purpose. Both back whole-set
+//! caches that are cheap to rebuild (tens of rows), so naming the exact row
+//! would add parsing for no benefit.
+//!
+//! # Why the handler lives here rather than being passed in
+//!
+//! [`super::alerts::watch_events`] takes callbacks because its handler lives in
+//! the `db` crate, which `infra` cannot reference. The synthetics caches are in
+//! `infra::table`, the same crate as this module, so [`watch`] calls the
+//! invalidation functions directly — same pattern, one less layer of
+//! indirection.
+
+use crate::{db::Event, errors::Error};
+
+pub const SYNTHETICS_WATCHER_PREFIX: &str = "/synthetics/";
+
+const KIND_CHECK: &str = "check";
+const KIND_LOCATION: &str = "location";
+const KIND_TOKEN: &str = "token";
+const KIND_AGENT: &str = "agent";
+
+/// A check definition was created or updated.
+pub async fn emit_check_put(org_id: &str, synthetics_id: &str) -> Result<(), Error> {
+    emit_put(&check_key(org_id, synthetics_id)).await
+}
+
+/// A check definition was deleted.
+pub async fn emit_check_delete(org_id: &str, synthetics_id: &str) -> Result<(), Error> {
+    emit_delete(&check_key(org_id, synthetics_id)).await
+}
+
+/// The location registry changed (add / update / remove).
+pub async fn emit_locations_changed() -> Result<(), Error> {
+    emit_put(&format!("{SYNTHETICS_WATCHER_PREFIX}{KIND_LOCATION}/all")).await
+}
+
+/// An org's probe tokens changed — created, enabled/disabled, or default moved.
+///
+/// Disabling a token is revocation, so this is the event whose delivery latency
+/// is the fleet-wide revocation window.
+pub async fn emit_tokens_changed(org_id: &str) -> Result<(), Error> {
+    emit_put(&format!("{SYNTHETICS_WATCHER_PREFIX}{KIND_TOKEN}/{org_id}")).await
+}
+
+/// An agent re-registered, so its capabilities may have changed.
+pub async fn emit_agent_changed(agent_id: &str) -> Result<(), Error> {
+    emit_put(&format!(
+        "{SYNTHETICS_WATCHER_PREFIX}{KIND_AGENT}/{agent_id}"
+    ))
+    .await
+}
+
+async fn emit_put(key: &str) -> Result<(), Error> {
+    let cluster_coordinator = super::get_coordinator().await;
+    cluster_coordinator
+        .put(key, bytes::Bytes::from(""), true, None)
+        .await?;
+    Ok(())
+}
+
+async fn emit_delete(key: &str) -> Result<(), Error> {
+    let cluster_coordinator = super::get_coordinator().await;
+    cluster_coordinator.delete(key, false, true, None).await
+}
+
+fn check_key(org_id: &str, synthetics_id: &str) -> String {
+    format!("{SYNTHETICS_WATCHER_PREFIX}{KIND_CHECK}/{org_id}/{synthetics_id}")
+}
+
+/// Watches synthetics events and invalidates the matching local cache.
+///
+/// Spawned once per node at startup. Both `Put` and `Delete` invalidate — the
+/// caches hold no value from the event itself, so the two cases are the same
+/// action, and a deleted check must stop being served just as surely as an
+/// edited one.
+pub async fn watch() -> Result<(), anyhow::Error> {
+    let cluster_coordinator = super::get_coordinator().await;
+    let mut events = cluster_coordinator.watch(SYNTHETICS_WATCHER_PREFIX).await?;
+    let events = std::sync::Arc::get_mut(&mut events).unwrap();
+    log::info!("Start watching synthetics cache events");
+    loop {
+        let ev = match events.recv().await {
+            Some(ev) => ev,
+            None => {
+                log::error!("watch_synthetics: event channel closed");
+                break;
+            }
+        };
+        let key = match &ev {
+            Event::Put(e) => e.key.clone(),
+            Event::Delete(e) => e.key.clone(),
+            Event::Empty => continue,
+        };
+        apply(&key).await;
+    }
+    Ok(())
+}
+
+/// Applies one event key to the local caches. Unknown keys are logged and
+/// ignored rather than panicking — a newer node may emit a kind this build does
+/// not understand yet.
+async fn apply(key: &str) {
+    let parts: Vec<&str> = key.trim_start_matches('/').split('/').collect();
+    // parts[0] == "synthetics"
+    match parts.get(1).copied() {
+        Some(KIND_CHECK) => match (parts.get(2), parts.get(3)) {
+            (Some(org), Some(id)) => crate::table::synthetics_checks::invalidate_cache(org, id),
+            _ => log::error!("watch_synthetics: malformed check key {key}"),
+        },
+        Some(KIND_LOCATION) => crate::table::synthetics_locations::invalidate_cache().await,
+        Some(KIND_TOKEN) => crate::table::synthetics_probe_tokens::invalidate_cache(),
+        Some(KIND_AGENT) => match parts.get(2) {
+            Some(agent_id) => crate::table::synthetics_agents::invalidate_cache(agent_id),
+            None => log::error!("watch_synthetics: malformed agent key {key}"),
+        },
+        other => log::debug!("watch_synthetics: ignoring unknown event kind {other:?} ({key})"),
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_check_key_format() {
+        assert_eq!(
+            check_key("myorg", "abc123"),
+            "/synthetics/check/myorg/abc123"
+        );
+    }
+
+    #[test]
+    fn test_watch_prefix_covers_every_kind() {
+        // Every emitted key must sort under the single watch prefix, or the
+        // watcher silently never sees it.
+        for k in [
+            check_key("o", "i"),
+            format!("{SYNTHETICS_WATCHER_PREFIX}{KIND_LOCATION}/all"),
+            format!("{SYNTHETICS_WATCHER_PREFIX}{KIND_TOKEN}/o"),
+            format!("{SYNTHETICS_WATCHER_PREFIX}{KIND_AGENT}/a"),
+        ] {
+            assert!(k.starts_with(SYNTHETICS_WATCHER_PREFIX), "{k}");
+        }
+    }
+
+    #[test]
+    fn test_key_parses_into_kind_and_ids() {
+        let key = check_key("myorg", "abc123");
+        let parts: Vec<&str> = key.trim_start_matches('/').split('/').collect();
+        assert_eq!(parts[0], "synthetics");
+        assert_eq!(parts[1], KIND_CHECK);
+        assert_eq!(parts[2], "myorg");
+        assert_eq!(parts[3], "abc123");
+    }
+
+    #[test]
+    fn test_org_and_id_with_unusual_chars_still_parse() {
+        // Org ids and ksuids are alphanumeric today; this guards the parse
+        // against a future id format that is merely long or mixed-case.
+        let key = check_key("Org-With_Dash", "3HGpH7OjyQHnknWNzXNIDXeF6zi");
+        let parts: Vec<&str> = key.trim_start_matches('/').split('/').collect();
+        assert_eq!(parts[2], "Org-With_Dash");
+        assert_eq!(parts[3], "3HGpH7OjyQHnknWNzXNIDXeF6zi");
+    }
+}

src/infra/src/file_list/mod.rs

@@ -88,7 +88,7 @@ pub trait FileList: Sync + Send + 'static {
     async fn update_flattened(&self, file: &str, flattened: bool) -> Result<()>;
     async fn update_compressed_size(&self, file: &str, size: i64) -> Result<()>;
     /// Bulk-set `bloom_ver` for the given file_list ids. Used by the
-    /// post-merge bloom builder (enterprise `bloom::compact`).
+    /// post-merge bloom builder (`compaction::bloom::compact`).
     /// Empty `ids` is a no-op.
     async fn update_bloom_ver(&self, ids: &[i64], bloom_ver: i64) -> Result<()>;
     /// Is `bloom_ver` still referenced by at least one live file_list row in

src/infra/src/table/entity/mod.rs

@@ -55,9 +55,9 @@ pub mod slo_status;
 pub mod slos;
 pub mod source_maps;
 pub mod synthetics_agents;
+pub mod synthetics_checks;
 pub mod synthetics_jobs;
 pub mod synthetics_locations;
-pub mod synthetics_monitors;
 pub mod synthetics_probe_tokens;
 pub mod synthetics_runs;
 pub mod system_prompts;

src/infra/src/table/entity/synthetics_checks.rs

@@ -1,4 +1,4 @@
-//! `SeaORM` Entity for synthetics_monitors table.
+//! `SeaORM` Entity for synthetics_checks table.
 
 use sea_orm::entity::prelude::*;
 
@@ -16,19 +16,19 @@ pub struct Model {
     pub description: String,
     pub tags: Json,
     pub config: Json,
-    /// Serialized `MonitorFrequency` — replaces interval_secs / frequency_type / cron_expr.
+    /// Serialized `SyntheticFrequency` — replaces interval_secs / frequency_type / cron_expr.
     pub frequency: Json,
     pub locations: Json,
     pub enabled: bool,
     pub destinations: Json,
-    /// Extra monitor settings (retries, cooldown, rum toggles). No secrets here.
+    /// Extra check settings (retries, cooldown, rum toggles). No secrets here.
     pub settings: Json,
     /// JSON blob: { "auth": {...}, "cookies": [...], "variables": [...] }
     /// All secret values encrypted per-field with AESenc:<base64> using the org DEK.
     pub secrets: String,
     /// Pre-computed next fire time (microseconds). 0 = fire on next scheduler tick.
     pub next_run_at: i64,
-    /// When the scheduler last fanned out this monitor (microseconds).
+    /// When the scheduler last fanned out this check (microseconds).
     pub last_triggered_at: i64,
     /// Denormalised status from the most recent completed check.
     /// 0=Unknown, 1=Up, 2=Warning, 3=Down
@@ -72,7 +72,7 @@ mod tests {
             name: "Login Flow".to_string(),
             synthetics_type: "browser".to_string(),
             target: "https://app.example.com".to_string(),
-            description: "Monitors the login flow".to_string(),
+            description: "Checks the login flow".to_string(),
             tags: serde_json::json!(["prod", "checkout"]),
             config: serde_json::json!({"browser_devices": [{"browser": "chromium", "device": "desktop"}], "steps": []}),
             frequency: serde_json::json!({"type": "minutes", "interval": 5, "cron": ""}),

src/infra/src/table/entity/synthetics_jobs.rs

@@ -25,9 +25,9 @@ pub struct Model {
     pub dispatch_attempts: i32,
     /// KSUID of the parent run (all jobs for one scheduled slot share this).
     pub run_id: String,
-    /// JSON array of BrowserDevice {execution_id, engine, device} — browser monitors only.
+    /// JSON array of BrowserDevice {execution_id, engine, device} — browser checks only.
     pub browser_devices: Option<String>,
-    /// JSON blob of monitor metadata copied at enqueue time e.g. {"tags": ["prod"]}.
+    /// JSON blob of check metadata copied at enqueue time e.g. {"tags": ["prod"]}.
     pub metadata: String,
     /// JSON execution summaries written at ack time (no full step data — that's in the stream).
     pub result: Option<String>,

src/infra/src/table/entity/synthetics_locations.rs

@@ -31,6 +31,14 @@ pub struct Model {
     /// Queue routing key (unique), e.g. `net-aws-us-east-1`, `private-acme-dc1`.
     pub pool: String,
     pub enabled: bool,
+    /// When this location's "down" notification was sent, in microseconds.
+    /// 0 = not currently notified as down.
+    ///
+    /// Cluster-wide one-shot state. The staleness watcher runs on every
+    /// alert_manager, so the suppression flag cannot live in process memory —
+    /// N nodes would each send their own notification for one outage.
+    #[sea_orm(default_value = 0)]
+    pub down_notified_at: i64,
     pub created_at: i64,
     pub updated_at: i64,
 }

src/infra/src/table/migration/m20260803_000001_add_down_notified_at_to_synthetics_locations.rs

@@ -0,0 +1,85 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Make the "location down" one-shot cluster-wide instead of per-process.
+//!
+//! The staleness watcher runs on every alert_manager node and suppressed repeat
+//! notifications with an in-memory `HashSet<String>`. That set is process-local,
+//! so N alert managers sent N notifications for the same outage — and did it
+//! again on every down → recover → down cycle.
+//!
+//! One column, defaulted, so existing rows need no backfill:
+//!   - `down_notified_at` — when this location's "down" notification was sent, in microseconds. 0
+//!     means "not currently notified as down", which is also the correct value for every existing
+//!     row: a location that is genuinely down will be re-detected on the next 60 s tick and
+//!     notified once.
+//!
+//! The column is claimed with a compare-and-swap (`WHERE down_notified_at = 0`)
+//! rather than read-then-write, so exactly one node sends the notification —
+//! the same primitive `synthetics_jobs::lease_batch` and
+//! `synthetics_checks::try_claim_slot` use.
+
+use sea_orm_migration::prelude::*;
+
+#[derive(DeriveMigrationName)]
+pub struct Migration;
+
+const TABLE: &str = "synthetics_locations";
+
+#[async_trait::async_trait]
+impl MigrationTrait for Migration {
+    async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {
+        // On a fresh database the table is created from the entity definition,
+        // so the column already exists by the time this runs. SQLite ignores the
+        // `IF NOT EXISTS` guard on `ADD COLUMN`, so guard explicitly with
+        // `has_column` to stay idempotent across backends.
+        if !manager.has_column(TABLE, "down_notified_at").await? {
+            manager
+                .alter_table(
+                    Table::alter()
+                        .table(SyntheticsLocations::Table)
+                        .add_column_if_not_exists(
+                            ColumnDef::new(SyntheticsLocations::DownNotifiedAt)
+                                .big_integer()
+                                .not_null()
+                                .default(0i64),
+                        )
+                        .to_owned(),
+                )
+                .await?;
+        }
+        Ok(())
+    }
+
+    async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> {
+        if manager.has_column(TABLE, "down_notified_at").await? {
+            manager
+                .alter_table(
+                    Table::alter()
+                        .table(SyntheticsLocations::Table)
+                        .drop_column(SyntheticsLocations::DownNotifiedAt)
+                        .to_owned(),
+                )
+                .await?;
+        }
+        Ok(())
+    }
+}
+
+#[derive(DeriveIden)]
+enum SyntheticsLocations {
+    Table,
+    DownNotifiedAt,
+}

src/infra/src/table/migration/mod.rs

@@ -154,6 +154,7 @@ mod m20260730_000001_add_alert_state_to_synthetics_monitors;
 mod m20260730_000002_create_incident_integrations;
 mod m20260730_000003_create_external_alerts;
 mod m20260730_000004_add_alert_kind_to_incident_alerts;
+mod m20260803_000001_add_down_notified_at_to_synthetics_locations;
 
 /// Apply **only** the SLO tables, for targeted integration tests.
 ///
@@ -173,38 +174,6 @@ pub(crate) async fn create_slo_tables_for_test(
         .await
 }
 
-/// Apply the `alert_states` chain, for targeted integration tests (§7.6).
-///
-/// Three migrations because the level columns and the group-lifecycle columns
-/// arrived after the tables. Applied in order, as production would.
-#[cfg(test)]
-pub(crate) async fn create_alert_state_tables_for_test(
-    db: &sea_orm::DatabaseConnection,
-) -> Result<(), DbErr> {
-    use sea_orm::ConnectionTrait;
-    use sea_orm_migration::MigrationTrait;
-    let manager = SchemaManager::new(db);
-
-    // m20260725_000002 ALTERs `alerts` as well as `alert_states`, so the
-    // table has to exist. A minimal stand-in is enough and is honest about
-    // what the fixture provides: these tests are about alert *state*, not
-    // about the alerts table, and building the real one would mean replaying
-    // years of unrelated migrations.
-    db.execute_unprepared("CREATE TABLE IF NOT EXISTS alerts (id VARCHAR(27) PRIMARY KEY)")
-        .await?;
-
-    m20260725_000001_create_alert_states_tables::Migration
-        .up(&manager)
-        .await?;
-    m20260725_000002_add_threshold_and_level_columns::Migration
-        .up(&manager)
-        .await?;
-    m20260726_000003_add_group_lifecycle_columns::Migration
-        .up(&manager)
-        .await?;
-    Ok(())
-}
-
 pub struct Migrator;
 
 #[async_trait::async_trait]
@@ -347,6 +316,7 @@ impl MigratorTrait for Migrator {
             Box::new(m20260730_000002_create_incident_integrations::Migration),
             Box::new(m20260730_000003_create_external_alerts::Migration),
             Box::new(m20260730_000004_add_alert_kind_to_incident_alerts::Migration),
+            Box::new(m20260803_000001_add_down_notified_at_to_synthetics_locations::Migration),
         ]
     }
 }

src/infra/src/table/mod.rs

@@ -70,9 +70,9 @@ pub mod slo_budget;
 pub mod slos;
 pub mod source_maps;
 pub mod synthetics_agents;
+pub mod synthetics_checks;
 pub mod synthetics_jobs;
 pub mod synthetics_locations;
-pub mod synthetics_monitors;
 pub mod synthetics_probe_tokens;
 pub mod synthetics_runs;
 pub mod system_prompts;

src/infra/src/table/org_ingestion_tokens.rs

@@ -13,6 +13,12 @@
 // You should have received a copy of the GNU Affero General Public License
 // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+use std::{
+    sync::LazyLock,
+    time::{Duration, Instant},
+};
+
+use config::RwHashMap;
 use sea_orm::{
     ColumnTrait, EntityTrait, FromQueryResult, Order, PaginatorTrait, QueryFilter, QueryOrder,
     QuerySelect, Set, entity::prelude::*, sea_query::OnConflict,
@@ -30,6 +36,67 @@ use crate::{
 
 pub const ORG_INGESTION_TOKEN_PREFIX: &str = "o2oi_";
 
+/// `org_id` → its default enabled ingest token, with the time it was loaded.
+///
+/// Backs [`find_default_enabled`]. Synthetics needs this on every dispatch, on
+/// every job resolve and in the reaper's dead-letter path, i.e. once per job —
+/// all for a value that changes only when an operator adds, rotates or disables
+/// a token.
+///
+/// Note this is a *different lookup direction* from the existing
+/// `ORG_INGESTION_TOKENS` cache in the `db` crate, which is keyed
+/// `{org}/{token} → name` and answers "is this token valid". That one cannot
+/// serve "what is this org's token", which is why this exists rather than
+/// reusing it.
+///
+/// `None` is cached too, so an org with no enabled token does not re-query on
+/// every job.
+static DEFAULT_TOKEN_CACHE: LazyLock<
+    RwHashMap<String, (Option<OrgIngestionTokenRecord>, Instant)>,
+> = LazyLock::new(Default::default);
+
+/// Backstop only. Cross-node invalidation arrives through the existing
+/// `/org_ingestion_tokens/` coordinator watch (see `db::org_ingestion_tokens::watch`),
+/// so this TTL just bounds a dropped event.
+const DEFAULT_TOKEN_CACHE_TTL: Duration = Duration::from_secs(60);
+
+/// Drops one org's cached default token.
+///
+/// Called by every write path in this module, and by the coordinator watcher so
+/// a rotate or disable performed on another node lands here too.
+pub fn invalidate_default_cache(org_id: &str) {
+    DEFAULT_TOKEN_CACHE.remove(org_id);
+}
+
+/// The org's default enabled ingest token, served from cache when fresh.
+///
+/// Equivalent to the `list_by_org(org).find(|t| t.enabled)` that callers used to
+/// write by hand — `is_default` first, then newest — but as one row instead of
+/// fetching every token for the org and discarding most of them.
+pub async fn find_default_enabled(
+    org_id: &str,
+) -> Result<Option<OrgIngestionTokenRecord>, errors::Error> {
+    if let Some(entry) = DEFAULT_TOKEN_CACHE.get(org_id)
+        && entry.1.elapsed() < DEFAULT_TOKEN_CACHE_TTL
+    {
+        return Ok(entry.0.clone());
+    }
+
+    let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
+    let record = Entity::find()
+        .filter(Column::OrgId.eq(org_id))
+        .filter(Column::Enabled.eq(true))
+        .order_by(Column::IsDefault, Order::Desc)
+        .order_by(Column::CreatedAt, Order::Desc)
+        .one(client)
+        .await
+        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?
+        .map(OrgIngestionTokenRecord::from);
+
+    DEFAULT_TOKEN_CACHE.insert(org_id.to_string(), (record.clone(), Instant::now()));
+    Ok(record)
+}
+
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct OrgIngestionTokenRecord {
     pub id: String,
@@ -112,7 +179,11 @@ pub async fn add(record: &OrgIngestionTokenRecord) -> Result<(), errors::Error>
 
     let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
     match Entity::insert(model).exec(client).await {
-        Ok(_) => Ok(()),
+        Ok(_) => {
+            // A new token can be the org default, so the cached pick is stale.
+            invalidate_default_cache(&record.org_id);
+            Ok(())
+        }
         Err(e) => match e.sql_err() {
             Some(SqlErr::UniqueConstraintViolation(_)) => {
                 Err(Error::DbError(DbError::SeaORMError(format!(
@@ -163,6 +234,7 @@ pub async fn upsert(record: &OrgIngestionTokenRecord) -> Result<(), errors::Erro
         .exec(client)
         .await
         .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
+    invalidate_default_cache(&record.org_id);
     Ok(())
 }
 
@@ -237,6 +309,7 @@ pub async fn rotate_token(org_id: &str, name: &str) -> Result<String, errors::Er
         .await
         .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
 
+    invalidate_default_cache(org_id);
     Ok(new_token)
 }
 
@@ -251,6 +324,7 @@ pub async fn delete_by_org(org_id: &str) -> Result<(), errors::Error> {
         .await
         .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
 
+    invalidate_default_cache(org_id);
     Ok(())
 }
 
@@ -269,6 +343,7 @@ pub async fn remove_by_token(org_id: &str, token: &str) -> Result<(), errors::Er
         .await
         .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
 
+    invalidate_default_cache(org_id);
     Ok(())
 }
 
@@ -299,6 +374,7 @@ pub async fn set_enabled(org_id: &str, name: &str, enabled: bool) -> Result<(),
         .await
         .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
 
+    invalidate_default_cache(org_id);
     Ok(())
 }
 

src/infra/src/table/synthetics_agents.rs

@@ -19,6 +19,12 @@
 //! `/synthetics/agent/register`; `last_seen_at` refreshed by register and by
 //! every job lease. A location whose agents are all stale is reported "down".
 
+use std::{
+    sync::LazyLock,
+    time::{Duration, Instant},
+};
+
+use config::RwHashMap;
 use sea_orm::{
     ColumnTrait, EntityTrait, QueryFilter, QueryOrder, QuerySelect, Set, sea_query::Expr,
 };
@@ -63,6 +69,37 @@ impl From<Model> for SyntheticsAgentRecord {
     }
 }
 
+/// `agent_id` → the agent row, with load time.
+///
+/// Backs [`get_cached`] only. The lease path reads an agent once per poll purely
+/// for its `capabilities`, and agents poll every 2 s — ~330 reads/min for 11
+/// agents, all for a value that changes only when an agent re-registers.
+///
+/// # Staleness contract
+///
+/// `last_seen_at` is rewritten by [`touch`] on **every lease**, and `touch` does
+/// not invalidate — invalidating there would defeat the cache entirely, since it
+/// fires at exactly the rate of the reads being served. Anything that judges
+/// agent liveness (the staleness watcher, location health) must use [`get`].
+static AGENT_CACHE: LazyLock<RwHashMap<String, (SyntheticsAgentRecord, Instant)>> =
+    LazyLock::new(Default::default);
+
+const AGENT_CACHE_TTL: Duration = Duration::from_secs(15);
+
+/// Drops one agent from the cache. Called on register, where capabilities change.
+pub fn invalidate_cache(agent_id: &str) {
+    AGENT_CACHE.remove(agent_id);
+}
+
+/// Invalidates locally **and** tells every other node. Write paths call this;
+/// the coordinator watcher calls [`invalidate_cache`] so events do not echo.
+async fn invalidate_and_publish(agent_id: &str) {
+    invalidate_cache(agent_id);
+    if let Err(e) = crate::coordinator::synthetics::emit_agent_changed(agent_id).await {
+        log::error!("[synthetics] emit agent cache event failed for {agent_id}: {e}");
+    }
+}
+
 /// Insert an agent row, or refresh version/capabilities/last_seen_at when the
 /// id already exists (idempotent re-register after restart).
 pub async fn register(record: &SyntheticsAgentRecord) -> Result<(), errors::Error> {
@@ -106,10 +143,21 @@ pub async fn register(record: &SyntheticsAgentRecord) -> Result<(), errors::Erro
                 .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
         }
     }
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    // Register is where capabilities change, so the cached copy is now stale.
+    invalidate_and_publish(&record.id).await;
     Ok(())
 }
 
 /// Refresh `last_seen_at` for an agent (called on heartbeat and every lease).
+///
+/// Deliberately does **not** invalidate [`AGENT_CACHE`] — it fires once per
+/// lease, which is exactly the rate of the reads the cache serves. Callers that
+/// need a current `last_seen_at` use [`get`], not [`get_cached`].
 pub async fn touch(agent_id: &str, now_us: i64) -> Result<(), errors::Error> {
     let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
     Entity::update_many()
@@ -135,6 +183,39 @@ pub async fn list_by_location(
     Ok(rows.into_iter().map(Into::into).collect())
 }
 
+/// All agents for a set of locations, grouped by `location_id`.
+///
+/// One query for the whole set instead of [`list_by_location`] per location —
+/// the locations list endpoint iterates every visible location to compute
+/// per-type availability, so the per-location form issued ~20 queries to build
+/// one response. Locations with no agents are absent from the map; callers
+/// should treat a missing key as an empty slice.
+pub async fn list_by_locations(
+    location_ids: &[String],
+) -> Result<std::collections::HashMap<String, Vec<SyntheticsAgentRecord>>, errors::Error> {
+    if location_ids.is_empty() {
+        return Ok(std::collections::HashMap::new());
+    }
+    let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
+    let rows = Entity::find()
+        .filter(Column::LocationId.is_in(location_ids.to_vec()))
+        .order_by_desc(Column::LastSeenAt)
+        .all(client)
+        .await
+        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
+
+    let mut grouped: std::collections::HashMap<String, Vec<SyntheticsAgentRecord>> =
+        std::collections::HashMap::new();
+    for row in rows {
+        let rec: SyntheticsAgentRecord = row.into();
+        grouped
+            .entry(rec.location_id.clone())
+            .or_default()
+            .push(rec);
+    }
+    Ok(grouped)
+}
+
 /// Find an agent by its deploy identity. Agents hold no persistent state, so
 /// a restarted container re-registers with the same (org, location, name) but
 /// without its previous server-issued id — this lookup lets register reuse the
@@ -180,6 +261,22 @@ pub async fn count_by_token(
 }
 
 /// Find one agent by id.
+/// Reads an agent from cache when fresh — see [`AGENT_CACHE`] for the staleness
+/// contract. Use [`get`] when `last_seen_at` matters.
+pub async fn get_cached(agent_id: &str) -> Result<Option<SyntheticsAgentRecord>, errors::Error> {
+    if let Some(entry) = AGENT_CACHE.get(agent_id)
+        && entry.1.elapsed() < AGENT_CACHE_TTL
+    {
+        return Ok(Some(entry.0.clone()));
+    }
+    let found = get(agent_id).await?;
+    if let Some(rec) = &found {
+        AGENT_CACHE.insert(agent_id.to_string(), (rec.clone(), Instant::now()));
+    }
+    Ok(found)
+}
+
+/// Reads an agent straight from the database, `last_seen_at` included.
 pub async fn get(agent_id: &str) -> Result<Option<SyntheticsAgentRecord>, errors::Error> {
     let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
     let row = Entity::find_by_id(agent_id)

src/infra/src/table/synthetics_checks.rs

@@ -13,9 +13,17 @@
 // You should have received a copy of the GNU Affero General Public License
 // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-use config::meta::synthetics::{
-    BrowserConfig, ListSyntheticsParams, Synthetic, SyntheticAuth, SyntheticCookie,
-    SyntheticFrequency, SyntheticSettings, SyntheticStatus, SyntheticType, SyntheticVariable,
+use std::{
+    sync::LazyLock,
+    time::{Duration, Instant},
+};
+
+use config::{
+    RwHashMap,
+    meta::synthetics::{
+        BrowserConfig, ListSyntheticsParams, Synthetic, SyntheticAuth, SyntheticCookie,
+        SyntheticFrequency, SyntheticSettings, SyntheticStatus, SyntheticType, SyntheticVariable,
+    },
 };
 use sea_orm::{
     ActiveModelTrait, ColumnTrait, ConnectionTrait, EntityTrait, PaginatorTrait, QueryFilter,
@@ -23,16 +31,78 @@ use sea_orm::{
 };
 use serde::{Deserialize, Serialize};
 
-use super::entity::synthetics_monitors::{self, ActiveModel, Column, Entity};
+use super::entity::synthetics_checks::{self, ActiveModel, Column, Entity};
 use crate::errors;
 
+/// `org_id/synthetics_id` → the synthetic check **definition**, with load time.
+///
+/// The probe job path reads a check twice per job — once at `resolve` to build
+/// the `CheckJob`, once at `ack` for its type and destinations — which is the
+/// single biggest per-job read in the feature (~400/min at 100 checks). The
+/// definition changes only when a user edits the check.
+///
+/// **This cache backs [`get_cached`] only, never [`get`].** The distinction is
+/// load-bearing: `Synthetic` carries `next_run_at` and `last_check_status`
+/// alongside the definition, and those are rewritten on *every run* by
+/// `advance_schedule` / `update_last_check_status`. Invalidating on those would
+/// make the cache useless (they fire at the same rate as the reads it serves),
+/// so instead they deliberately do **not** invalidate — and callers that need
+/// scheduling or status state must use [`get`], `fetch_due` or `get_alert_state`.
+static SYNTHETIC_CACHE: LazyLock<RwHashMap<String, (Synthetic, Instant)>> =
+    LazyLock::new(Default::default);
+
+const SYNTHETIC_CACHE_TTL: Duration = Duration::from_secs(15);
+
+fn synthetic_cache_key(org_id: &str, id: &str) -> String {
+    format!("{org_id}/{id}")
+}
+
+/// Drops one check from the definition cache. Called by every path that edits
+/// a definition, so an edit is visible on this node immediately.
+pub fn invalidate_cache(org_id: &str, id: &str) {
+    SYNTHETIC_CACHE.remove(&synthetic_cache_key(org_id, id));
+}
+
+/// Drops the whole definition cache. Used where a write may touch many rows.
+pub fn invalidate_all_cache() {
+    SYNTHETIC_CACHE.clear();
+}
+
+/// Invalidates locally **and** tells every other node to do the same.
+///
+/// Write paths call this; the coordinator watcher calls the plain
+/// [`invalidate_cache`], which is what stops an event from echoing forever.
+///
+/// A failed emit is logged, not propagated: the database write has already
+/// committed, and the cache TTL is the backstop for a dropped event. Failing
+/// the user's save because a cache hint did not send would be the worse trade.
+async fn invalidate_and_publish(org_id: &str, id: &str) {
+    invalidate_cache(org_id, id);
+    if let Err(e) = crate::coordinator::synthetics::emit_check_put(org_id, id).await {
+        log::error!("[synthetics] emit check cache event failed for {org_id}/{id}: {e}");
+    }
+}
+
+/// Same as [`invalidate_and_publish`], but emits a *delete* event.
+///
+/// Both events invalidate identically on the receiving side, so this is not
+/// about the handler — it is about the coordinator's key store. `emit_*_put`
+/// writes a key; only a delete event removes it. Publishing a put on the delete
+/// path would leave one dead key per check ever created, growing without bound.
+async fn invalidate_and_publish_delete(org_id: &str, id: &str) {
+    invalidate_cache(org_id, id);
+    if let Err(e) = crate::coordinator::synthetics::emit_check_delete(org_id, id).await {
+        log::error!("[synthetics] emit check delete event failed for {org_id}/{id}: {e}");
+    }
+}
+
 // ── TryFrom: ORM model → meta type ───────────────────────────────────────────
 
-impl TryFrom<synthetics_monitors::Model> for Synthetic {
+impl TryFrom<synthetics_checks::Model> for Synthetic {
     type Error = errors::Error;
 
-    fn try_from(m: synthetics_monitors::Model) -> Result<Self, Self::Error> {
-        let monitor_type: SyntheticType = serde_json::from_value(serde_json::Value::String(
+    fn try_from(m: synthetics_checks::Model) -> Result<Self, Self::Error> {
+        let check_type: SyntheticType = serde_json::from_value(serde_json::Value::String(
             m.synthetics_type.clone(),
         ))
         .map_err(|e| {
@@ -70,7 +140,7 @@ impl TryFrom<synthetics_monitors::Model> for Synthetic {
             name: m.name,
             description: m.description,
             tags,
-            monitor_type,
+            check_type,
             target: m.target,
             config: m.config,
             frequency,
@@ -100,6 +170,11 @@ impl TryFrom<synthetics_monitors::Model> for Synthetic {
 
 // ── Public CRUD API ───────────────────────────────────────────────────────────
 
+/// Reads a check straight from the database. Every field is current, including
+/// `next_run_at` and `last_check_status`.
+///
+/// Use this anywhere scheduling or status state matters. For the probe job path,
+/// which only needs the definition, prefer [`get_cached`].
 pub async fn get<C: ConnectionTrait>(
     conn: &C,
     org_id: &str,
@@ -110,6 +185,41 @@ pub async fn get<C: ConnectionTrait>(
     maybe.map(Synthetic::try_from).transpose()
 }
 
+/// Reads a check **definition**, served from [`SYNTHETIC_CACHE`] when fresh.
+///
+/// Intended for the probe job path (`resolve`, `ack`), which needs `config`,
+/// `check_type`, `target`, `destinations` and the retry settings — all of
+/// which change only on a user edit.
+///
+/// # Staleness contract
+///
+/// - **Definition fields are correct**, within `SYNTHETIC_CACHE_TTL` of an edit made on another
+///   node, and immediately for an edit made on this one.
+/// - **`next_run_at` and `last_check_status` may be stale by design.** They are rewritten every run
+///   and this cache is not invalidated when they change. Read them via [`get`], [`fetch_due`] or
+///   [`get_alert_state`] instead.
+///
+/// A missing check is not cached — a resolve for a deleted check should keep
+/// reaching the DB and failing loudly rather than being served from memory.
+pub async fn get_cached<C: ConnectionTrait>(
+    conn: &C,
+    org_id: &str,
+    id: &str,
+) -> Result<Option<Synthetic>, errors::Error> {
+    let key = synthetic_cache_key(org_id, id);
+    if let Some(entry) = SYNTHETIC_CACHE.get(&key)
+        && entry.1.elapsed() < SYNTHETIC_CACHE_TTL
+    {
+        return Ok(Some(entry.0.clone()));
+    }
+
+    let found = get(conn, org_id, id).await?;
+    if let Some(synthetic) = &found {
+        SYNTHETIC_CACHE.insert(key, (synthetic.clone(), Instant::now()));
+    }
+    Ok(found)
+}
+
 pub async fn list<C: ConnectionTrait>(
     conn: &C,
     org_id: &str,
@@ -185,26 +295,32 @@ pub async fn list_referencing_location<C: ConnectionTrait>(
 pub async fn create<C: TransactionTrait>(
     conn: &C,
     org_id: &str,
-    monitor: Synthetic,
+    check: Synthetic,
 ) -> Result<Synthetic, errors::Error> {
     let _lock = super::get_lock().await;
     let txn = conn.begin().await?;
     let now = config::utils::time::now_micros();
     let id = config::ider::uuid();
 
-    let mut am = build_active_model(&monitor)?;
+    let mut am = build_active_model(&check)?;
     am.id = Set(id);
     am.org_id = Set(org_id.to_owned());
-    am.folder_id = Set(monitor.folder_id.clone());
-    am.synthetics_type = Set(monitor_type_to_str(&monitor.monitor_type).to_owned());
+    am.folder_id = Set(check.folder_id.clone());
+    am.synthetics_type = Set(check_type_to_str(&check.check_type).to_owned());
     am.created_at = Set(now);
     am.updated_at = Set(now);
-    am.next_run_at = Set(monitor.start.unwrap_or(0));
-    am.owner = Set(monitor.owner.clone());
+    am.next_run_at = Set(check.start.unwrap_or(0));
+    am.owner = Set(check.owner.clone());
 
     let model = am.insert(&txn).await?.try_into_model()?;
     let result = Synthetic::try_from(model)?;
     txn.commit().await?;
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    invalidate_and_publish(&result.org_id, &result.id).await;
     Ok(result)
 }
 
@@ -212,48 +328,54 @@ pub async fn update<C: TransactionTrait>(
     conn: &C,
     org_id: &str,
     id: &str,
-    monitor: Synthetic,
+    check: Synthetic,
 ) -> Result<Synthetic, errors::Error> {
     let _lock = super::get_lock().await;
     let txn = conn.begin().await?;
 
     let Some(m) = get_model(&txn, org_id, id).await? else {
-        return Err(errors::Error::Message(format!("monitor not found: {id}")));
+        return Err(errors::Error::Message(format!("check not found: {id}")));
     };
 
     let mut am: ActiveModel = m.into();
-    update_mutable_fields(&mut am, &monitor)?;
+    update_mutable_fields(&mut am, &check)?;
     am.updated_at = Set(config::utils::time::now_micros());
 
     let model = am.update(&txn).await?.try_into_model()?;
     let result = Synthetic::try_from(model)?;
     txn.commit().await?;
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    invalidate_and_publish(&result.org_id, &result.id).await;
     Ok(result)
 }
 
 pub async fn put<C: TransactionTrait>(
     conn: &C,
     org_id: &str,
-    monitor: Synthetic,
+    check: Synthetic,
 ) -> Result<Synthetic, errors::Error> {
     let _lock = super::get_lock().await;
     let txn = conn.begin().await?;
     let now = config::utils::time::now_micros();
 
-    let result = match get_model(&txn, org_id, &monitor.id).await? {
+    let result = match get_model(&txn, org_id, &check.id).await? {
         Some(m) => {
             let mut am: ActiveModel = m.into();
-            update_mutable_fields(&mut am, &monitor)?;
+            update_mutable_fields(&mut am, &check)?;
             am.updated_at = Set(now);
             let model = am.update(&txn).await?.try_into_model()?;
             Synthetic::try_from(model)?
         }
         None => {
-            let mut am = build_active_model(&monitor)?;
-            am.id = Set(monitor.id.clone());
+            let mut am = build_active_model(&check)?;
+            am.id = Set(check.id.clone());
             am.org_id = Set(org_id.to_owned());
-            am.folder_id = Set(monitor.folder_id.clone());
-            am.synthetics_type = Set(monitor_type_to_str(&monitor.monitor_type).to_owned());
+            am.folder_id = Set(check.folder_id.clone());
+            am.synthetics_type = Set(check_type_to_str(&check.check_type).to_owned());
             am.created_at = Set(now);
             am.updated_at = Set(now);
             let model = am.insert(&txn).await?.try_into_model()?;
@@ -262,6 +384,12 @@ pub async fn put<C: TransactionTrait>(
     };
 
     txn.commit().await?;
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    invalidate_and_publish(&result.org_id, &result.id).await;
     Ok(result)
 }
 
@@ -276,10 +404,16 @@ pub async fn delete<C: ConnectionTrait>(
         .filter(Column::Id.eq(id))
         .exec(conn)
         .await?;
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    invalidate_and_publish_delete(org_id, id).await;
     Ok(res.rows_affected > 0)
 }
 
-/// Moves a batch of monitors to a different folder.
+/// Moves a batch of checks to a different folder.
 pub async fn move_to_folder<C: ConnectionTrait>(
     conn: &C,
     org_id: &str,
@@ -300,6 +434,14 @@ pub async fn move_to_folder<C: ConnectionTrait>(
         .filter(Column::Id.is_in(ids.to_vec()))
         .exec(conn)
         .await?;
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    for id in ids {
+        invalidate_and_publish(org_id, id).await;
+    }
     Ok(res.rows_affected)
 }
 
@@ -321,40 +463,89 @@ pub async fn set_enabled<C: ConnectionTrait>(
         .filter(Column::Id.eq(id))
         .exec(conn)
         .await?;
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    invalidate_and_publish(org_id, id).await;
     Ok(res.rows_affected > 0)
 }
 
 // ── Scheduler helpers ─────────────────────────────────────────────────────────
 
 /// Scheduler's fan-out data — the subset of Synthetic fields the scheduler needs.
-pub struct DueMonitor {
+pub struct DueCheck {
     pub id: String,
     pub name: String,
     pub org_id: String,
-    pub monitor_type: SyntheticType,
+    pub check_type: SyntheticType,
     pub locations: Vec<String>,
     pub frequency: SyntheticFrequency,
     /// Minutes from UTC — used for cron scheduling. 0 = UTC.
     pub tz_offset: i32,
-    /// The scheduled due time that made this monitor eligible. The scheduler
+    /// The scheduled due time that made this check eligible. The scheduler
     /// anchors the NEXT run to this (fixed-rate) instead of `now`, so the tick
     /// lag doesn't accumulate into schedule drift.
     pub next_run_at: i64,
-    /// Populated only for browser monitors (parsed from config.browser_devices).
+    /// Populated only for browser checks (parsed from config.browser_devices).
     pub browser_devices: Vec<config::meta::synthetics::BrowserDevice>,
     pub tags: Vec<String>,
 }
 
-/// Returns up to `limit` enabled monitors whose `next_run_at` is at or before `now_us`.
+/// Returns up to `limit` enabled checks whose `next_run_at` is at or before `now_us`.
 /// Ordered by next_run_at ASC so the most overdue fire first.
 ///
 /// NOTE: Does not use FOR UPDATE SKIP LOCKED — the scheduler is single-node on alert_manager.
 /// If multi-node scheduling is needed, convert to a raw SQL query with SKIP LOCKED.
+impl TryFrom<synthetics_checks::Model> for DueCheck {
+    type Error = errors::Error;
+
+    fn try_from(m: synthetics_checks::Model) -> Result<Self, Self::Error> {
+        let check_type: SyntheticType = serde_json::from_value(serde_json::Value::String(
+            m.synthetics_type.clone(),
+        ))
+        .map_err(|e| {
+            errors::Error::Message(format!(
+                "invalid synthetics_type '{}' for {}: {e}",
+                m.synthetics_type, m.id
+            ))
+        })?;
+
+        let locations: Vec<String> = serde_json::from_value(m.locations)
+            .map_err(|e| errors::Error::Message(format!("invalid locations for {}: {e}", m.id)))?;
+
+        let frequency: SyntheticFrequency = serde_json::from_value(m.frequency).unwrap_or_default();
+
+        let browser_devices = if check_type == SyntheticType::Browser {
+            let cfg: BrowserConfig = serde_json::from_value(m.config).unwrap_or_default();
+            cfg.browser_devices
+        } else {
+            vec![]
+        };
+
+        let tags: Vec<String> = serde_json::from_value(m.tags).unwrap_or_default();
+
+        Ok(DueCheck {
+            id: m.id,
+            name: m.name,
+            org_id: m.org_id,
+            check_type,
+            locations,
+            frequency,
+            tz_offset: m.tz_offset,
+            next_run_at: m.next_run_at,
+            browser_devices,
+            tags,
+        })
+    }
+}
+
 pub async fn fetch_due<C: ConnectionTrait>(
     conn: &C,
     now_us: i64,
     limit: u64,
-) -> Result<Vec<DueMonitor>, errors::Error> {
+) -> Result<Vec<DueCheck>, errors::Error> {
     let _lock = super::get_lock().await;
     let models = Entity::find()
         .filter(Column::Enabled.eq(true))
@@ -364,51 +555,85 @@ pub async fn fetch_due<C: ConnectionTrait>(
         .all(conn)
         .await?;
 
-    models
-        .into_iter()
-        .map(|m| {
-            let monitor_type: SyntheticType =
-                serde_json::from_value(serde_json::Value::String(m.synthetics_type.clone()))
-                    .map_err(|e| {
-                        errors::Error::Message(format!(
-                            "invalid synthetics_type '{}' for {}: {e}",
-                            m.synthetics_type, m.id
-                        ))
-                    })?;
-
-            let locations: Vec<String> = serde_json::from_value(m.locations).map_err(|e| {
-                errors::Error::Message(format!("invalid locations for {}: {e}", m.id))
-            })?;
-
-            let frequency: SyntheticFrequency =
-                serde_json::from_value(m.frequency).unwrap_or_default();
-
-            let browser_devices = if monitor_type == SyntheticType::Browser {
-                let cfg: BrowserConfig = serde_json::from_value(m.config).unwrap_or_default();
-                cfg.browser_devices
-            } else {
-                vec![]
-            };
-
-            let tags: Vec<String> = serde_json::from_value(m.tags).unwrap_or_default();
-
-            Ok(DueMonitor {
-                id: m.id,
-                name: m.name,
-                org_id: m.org_id,
-                monitor_type,
-                locations,
-                frequency,
-                tz_offset: m.tz_offset,
-                next_run_at: m.next_run_at,
-                browser_devices,
-                tags,
-            })
-        })
-        .collect()
+    models.into_iter().map(DueCheck::try_from).collect()
 }
 
-/// Updates `last_triggered_at` and `next_run_at` after the scheduler fans out a monitor.
+/// Updates `last_triggered_at` and `next_run_at` after the scheduler fans out a check.
+/// Claims every due check in one pass, returning only the ones THIS node won.
+///
+/// This is the design's scheduler claim (`designs/synthetics/01-server-architecture.md`
+/// §4.2): "Run 2+ replicas; they self-shard via `SELECT … FOR UPDATE SKIP
+/// LOCKED` — no leader election."
+///
+/// Why locking beats the per-row compare-and-swap it replaces, at scale:
+///
+/// * **The nodes self-shard.** A locker skips rows another node already holds and takes *different*
+///   due checks instead, so N schedulers split the backlog. Under a CAS every node reads the same
+///   candidate list, one wins them all, and the losers issue one doomed UPDATE per check —
+///   `FETCH_LIMIT` wasted writes per node per tick.
+/// * **Losers do no writes at all.** The skip happens in the SELECT, so a node that loses a row
+///   never issues its UPDATE.
+///
+/// The advance happens **inside** the same transaction, because the row locks
+/// only exist until COMMIT — claiming and advancing separately would reopen the
+/// race this closes. `next_run_for` computes each check's next slot; it is a
+/// callback because that calculation (interval vs cron, tz, skip-missed) lives
+/// with the scheduler, while the lock has to be held here.
+///
+/// Fan-out is deliberately left to the caller, *after* the commit: holding row
+/// locks on user-facing config rows across N job inserts would stall any
+/// concurrent edit of those checks for the duration.
+///
+/// SQLite has no `FOR UPDATE`, and needs none — it is single-node by
+/// construction (`config.rs:3182` restricts cluster mode to Postgres), so no
+/// second scheduler can exist to race with.
+pub async fn claim_due<C>(
+    conn: &C,
+    now_us: i64,
+    limit: u64,
+    next_run_for: impl Fn(&DueCheck) -> i64,
+) -> Result<Vec<DueCheck>, errors::Error>
+where
+    C: ConnectionTrait + TransactionTrait,
+{
+    let _lock = super::get_lock().await;
+    let txn = conn.begin().await?;
+
+    let mut query = Entity::find()
+        .filter(Column::Enabled.eq(true))
+        .filter(Column::NextRunAt.lte(now_us))
+        .order_by_asc(Column::NextRunAt)
+        .limit(limit);
+    if txn.get_database_backend() == sea_orm::DatabaseBackend::Postgres {
+        query = query.lock_with_behavior(
+            sea_orm::sea_query::LockType::Update,
+            sea_orm::sea_query::LockBehavior::SkipLocked,
+        );
+    }
+    let models = query.all(&txn).await?;
+
+    let due: Vec<DueCheck> = models
+        .into_iter()
+        .map(DueCheck::try_from)
+        .collect::<Result<Vec<_>, _>>()?;
+
+    for check in &due {
+        Entity::update_many()
+            .col_expr(Column::LastTriggeredAt, Expr::value(now_us))
+            .col_expr(Column::NextRunAt, Expr::value(next_run_for(check)))
+            .filter(Column::Id.eq(check.id.as_str()))
+            .exec(&txn)
+            .await?;
+    }
+
+    txn.commit().await?;
+    Ok(due)
+}
+
+/// Unconditionally sets a check's schedule. For user-initiated changes (edit,
+/// disable, run-now), where the user's action must always win.
+///
+/// **Schedulers must not use this** — see [`try_claim_slot`].
 pub async fn advance_schedule<C: ConnectionTrait>(
     conn: &C,
     id: &str,
@@ -454,7 +679,7 @@ pub struct AlertState {
     pub degraded_notified_at: i64,
 }
 
-/// Reads the alert state without pulling the whole monitor (config, secrets and
+/// Reads the alert state without pulling the whole check (config, secrets and
 /// step definitions are several KB, and this runs on every ack).
 pub async fn get_alert_state<C: ConnectionTrait>(
     conn: &C,
@@ -532,7 +757,7 @@ async fn get_model<C: ConnectionTrait>(
     conn: &C,
     org_id: &str,
     id: &str,
-) -> Result<Option<synthetics_monitors::Model>, sea_orm::DbErr> {
+) -> Result<Option<synthetics_checks::Model>, sea_orm::DbErr> {
     Entity::find_by_id(id)
         .filter(Column::OrgId.eq(org_id))
         .one(conn)
@@ -543,7 +768,7 @@ async fn list_models<C: ConnectionTrait>(
     conn: &C,
     org_id: &str,
     params: &ListSyntheticsParams,
-) -> Result<Vec<synthetics_monitors::Model>, sea_orm::DbErr> {
+) -> Result<Vec<synthetics_checks::Model>, sea_orm::DbErr> {
     let q = Entity::find()
         .filter(Column::OrgId.eq(org_id))
         .apply_filters(params)
@@ -557,15 +782,15 @@ async fn list_models<C: ConnectionTrait>(
     q.all(conn).await
 }
 
-fn pack_settings(monitor: &Synthetic) -> Result<serde_json::Value, errors::Error> {
+fn pack_settings(check: &Synthetic) -> Result<serde_json::Value, errors::Error> {
     Ok(serde_json::to_value(SyntheticSettings {
-        retries: monitor.retries,
-        cooldown_mins: monitor.cooldown_mins,
-        wait_before_retry_secs: monitor.wait_before_retry_secs,
-        alert_if_fails: monitor.alert_if_fails,
-        collect_rum_data: monitor.collect_rum_data,
-        session_replay: monitor.session_replay,
-        start: monitor.start,
+        retries: check.retries,
+        cooldown_mins: check.cooldown_mins,
+        wait_before_retry_secs: check.wait_before_retry_secs,
+        alert_if_fails: check.alert_if_fails,
+        collect_rum_data: check.collect_rum_data,
+        session_replay: check.session_replay,
+        start: check.start,
     })?)
 }
 
@@ -586,61 +811,61 @@ struct StoredSecrets {
     config: std::collections::BTreeMap<String, String>,
 }
 
-fn pack_secrets(monitor: &Synthetic) -> Result<String, errors::Error> {
+fn pack_secrets(check: &Synthetic) -> Result<String, errors::Error> {
     serde_json::to_string(&StoredSecrets {
-        auth: monitor.auth.clone(),
-        cookies: monitor.cookies.clone(),
-        variables: monitor.variables.clone(),
-        config: monitor.config_secrets.clone(),
+        auth: check.auth.clone(),
+        cookies: check.cookies.clone(),
+        variables: check.variables.clone(),
+        config: check.config_secrets.clone(),
     })
     .map_err(|e| errors::Error::Message(format!("secrets serialize failed: {e}")))
 }
 
-fn update_mutable_fields(am: &mut ActiveModel, monitor: &Synthetic) -> Result<(), errors::Error> {
-    let locations = serde_json::to_value(&monitor.locations)?;
-    let destinations = serde_json::to_value(&monitor.destinations)?;
-    let tags = serde_json::to_value(&monitor.tags)?;
-    let frequency = serde_json::to_value(&monitor.frequency)?;
-    let settings = pack_settings(monitor)?;
-    am.folder_id = Set(monitor.folder_id.clone());
-    am.tz_offset = Set(monitor.tz_offset);
-    am.name = Set(monitor.name.clone());
-    am.description = Set(monitor.description.clone());
+fn update_mutable_fields(am: &mut ActiveModel, check: &Synthetic) -> Result<(), errors::Error> {
+    let locations = serde_json::to_value(&check.locations)?;
+    let destinations = serde_json::to_value(&check.destinations)?;
+    let tags = serde_json::to_value(&check.tags)?;
+    let frequency = serde_json::to_value(&check.frequency)?;
+    let settings = pack_settings(check)?;
+    am.folder_id = Set(check.folder_id.clone());
+    am.tz_offset = Set(check.tz_offset);
+    am.name = Set(check.name.clone());
+    am.description = Set(check.description.clone());
     am.tags = Set(tags);
-    am.target = Set(monitor.target.clone());
-    am.config = Set(monitor.config.clone());
+    am.target = Set(check.target.clone());
+    am.config = Set(check.config.clone());
     am.frequency = Set(frequency);
     am.locations = Set(locations);
-    am.enabled = Set(monitor.enabled);
+    am.enabled = Set(check.enabled);
     am.destinations = Set(destinations);
     am.settings = Set(settings);
-    am.secrets = Set(pack_secrets(monitor)?);
+    am.secrets = Set(pack_secrets(check)?);
     Ok(())
 }
 
-fn build_active_model(monitor: &Synthetic) -> Result<ActiveModel, errors::Error> {
-    let locations = serde_json::to_value(&monitor.locations)?;
-    let destinations = serde_json::to_value(&monitor.destinations)?;
-    let tags = serde_json::to_value(&monitor.tags)?;
-    let frequency = serde_json::to_value(&monitor.frequency)?;
-    let settings = pack_settings(monitor)?;
+fn build_active_model(check: &Synthetic) -> Result<ActiveModel, errors::Error> {
+    let locations = serde_json::to_value(&check.locations)?;
+    let destinations = serde_json::to_value(&check.destinations)?;
+    let tags = serde_json::to_value(&check.tags)?;
+    let frequency = serde_json::to_value(&check.frequency)?;
+    let settings = pack_settings(check)?;
     Ok(ActiveModel {
-        name: Set(monitor.name.clone()),
-        description: Set(monitor.description.clone()),
+        name: Set(check.name.clone()),
+        description: Set(check.description.clone()),
         tags: Set(tags),
-        target: Set(monitor.target.clone()),
-        config: Set(monitor.config.clone()),
+        target: Set(check.target.clone()),
+        config: Set(check.config.clone()),
         frequency: Set(frequency),
         locations: Set(locations),
-        enabled: Set(monitor.enabled),
+        enabled: Set(check.enabled),
         destinations: Set(destinations),
         settings: Set(settings),
-        secrets: Set(pack_secrets(monitor)?),
+        secrets: Set(pack_secrets(check)?),
         ..Default::default()
     })
 }
 
-fn monitor_type_to_str(t: &SyntheticType) -> &'static str {
+fn check_type_to_str(t: &SyntheticType) -> &'static str {
     match t {
         SyntheticType::Http => "http",
         SyntheticType::Api => "api",
@@ -655,18 +880,18 @@ fn monitor_type_to_str(t: &SyntheticType) -> &'static str {
 
 // ── Filter extension ──────────────────────────────────────────────────────────
 
-trait ApplyMonitorFilters {
+trait ApplyCheckFilters {
     fn apply_filters(self, params: &ListSyntheticsParams) -> Self;
 }
 
-impl ApplyMonitorFilters for sea_orm::Select<Entity> {
+impl ApplyCheckFilters for sea_orm::Select<Entity> {
     fn apply_filters(self, params: &ListSyntheticsParams) -> Self {
         let mut q = self;
         if let Some(folder_id) = &params.folder_id {
             q = q.filter(Column::FolderId.eq(folder_id.clone()));
         }
-        if let Some(monitor_type) = &params.monitor_type {
-            q = q.filter(Column::SyntheticsType.eq(monitor_type_to_str(monitor_type)));
+        if let Some(check_type) = &params.check_type {
+            q = q.filter(Column::SyntheticsType.eq(check_type_to_str(check_type)));
         }
         if let Some(enabled) = params.enabled {
             q = q.filter(Column::Enabled.eq(enabled));
@@ -678,7 +903,7 @@ impl ApplyMonitorFilters for sea_orm::Select<Entity> {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use crate::table::entity::synthetics_monitors::Model;
+    use crate::table::entity::synthetics_checks::Model;
 
     fn make_model() -> Model {
         Model {
@@ -689,7 +914,7 @@ mod tests {
             name: "Login Flow".to_string(),
             synthetics_type: "browser".to_string(),
             target: "https://app.example.com".to_string(),
-            description: "Monitors the login flow".to_string(),
+            description: "Checks the login flow".to_string(),
             tags: serde_json::json!(["prod"]),
             config: serde_json::json!({
                 "browser_devices": [{"browser": "chromium", "device": "desktop"}],
@@ -716,14 +941,14 @@ mod tests {
 
     #[test]
     fn test_try_from_model() {
-        let monitor = Synthetic::try_from(make_model()).unwrap();
-        assert_eq!(monitor.id, "mon-1");
-        assert_eq!(monitor.monitor_type, SyntheticType::Browser);
-        assert_eq!(monitor.locations, vec!["aws-us-east-1"]);
-        assert!(monitor.enabled);
-        assert_eq!(monitor.frequency.interval, 5);
+        let check = Synthetic::try_from(make_model()).unwrap();
+        assert_eq!(check.id, "mon-1");
+        assert_eq!(check.check_type, SyntheticType::Browser);
+        assert_eq!(check.locations, vec!["aws-us-east-1"]);
+        assert!(check.enabled);
+        assert_eq!(check.frequency.interval, 5);
         assert_eq!(
-            monitor.frequency.frequency_type,
+            check.frequency.frequency_type,
             config::meta::synthetics::SyntheticFrequencyType::Minutes
         );
     }
@@ -737,14 +962,14 @@ mod tests {
 
     #[test]
     fn test_monitor_type_to_str() {
-        assert_eq!(monitor_type_to_str(&SyntheticType::Http), "http");
-        assert_eq!(monitor_type_to_str(&SyntheticType::Browser), "browser");
-        assert_eq!(monitor_type_to_str(&SyntheticType::Api), "api");
-        assert_eq!(monitor_type_to_str(&SyntheticType::Tcp), "tcp");
-        assert_eq!(monitor_type_to_str(&SyntheticType::Tls), "tls");
-        assert_eq!(monitor_type_to_str(&SyntheticType::Ssh), "ssh");
-        assert_eq!(monitor_type_to_str(&SyntheticType::Ping), "ping");
-        assert_eq!(monitor_type_to_str(&SyntheticType::Dns), "dns");
+        assert_eq!(check_type_to_str(&SyntheticType::Http), "http");
+        assert_eq!(check_type_to_str(&SyntheticType::Browser), "browser");
+        assert_eq!(check_type_to_str(&SyntheticType::Api), "api");
+        assert_eq!(check_type_to_str(&SyntheticType::Tcp), "tcp");
+        assert_eq!(check_type_to_str(&SyntheticType::Tls), "tls");
+        assert_eq!(check_type_to_str(&SyntheticType::Ssh), "ssh");
+        assert_eq!(check_type_to_str(&SyntheticType::Ping), "ping");
+        assert_eq!(check_type_to_str(&SyntheticType::Dns), "dns");
     }
 
     #[test]
@@ -753,9 +978,66 @@ mod tests {
         m.next_run_at = 1750000001000000;
         m.last_triggered_at = 1750000000500000;
         m.last_check_status = 1;
-        let monitor = Synthetic::try_from(m).unwrap();
-        assert_eq!(monitor.next_run_at, 1750000001000000);
-        assert_eq!(monitor.last_triggered_at, 1750000000500000);
-        assert_eq!(monitor.last_check_status, SyntheticStatus::Passed);
+        let check = Synthetic::try_from(m).unwrap();
+        assert_eq!(check.next_run_at, 1750000001000000);
+        assert_eq!(check.last_triggered_at, 1750000000500000);
+        assert_eq!(check.last_check_status, SyntheticStatus::Passed);
+    }
+
+    /// The lock clause is the whole HA fix: without FOR UPDATE SKIP LOCKED the
+    /// SELECT is plain and every alert_manager node claims every due check.
+    #[tokio::test]
+    async fn test_claim_due_locks_with_skip_locked_on_postgres() {
+        use sea_orm::{DatabaseBackend, MockDatabase, MockExecResult};
+
+        let db = MockDatabase::new(DatabaseBackend::Postgres)
+            .append_query_results(vec![vec![make_model()]])
+            .append_exec_results(vec![MockExecResult {
+                last_insert_id: 0,
+                rows_affected: 1,
+            }])
+            .into_connection();
+
+        let claimed = claim_due(&db, 500, 10, |_| 900).await.unwrap();
+        assert_eq!(
+            claimed.len(),
+            1,
+            "the due check should be returned as claimed"
+        );
+
+        let sql = format!("{:?}", db.into_transaction_log());
+        assert!(
+            sql.contains("FOR UPDATE SKIP LOCKED"),
+            "the claiming SELECT must lock and skip, else replicas do not self-shard: {sql}"
+        );
+        assert!(
+            sql.contains("UPDATE"),
+            "the advance must happen in the same transaction as the lock: {sql}"
+        );
+    }
+
+    /// SQLite has no FOR UPDATE and needs none — it is single-node, so no second
+    /// scheduler can exist to race with. Emitting the clause there is a syntax
+    /// error, so the backend branch is load-bearing.
+    #[tokio::test]
+    async fn test_claim_due_omits_lock_clause_on_sqlite() {
+        use sea_orm::{DatabaseBackend, MockDatabase, MockExecResult};
+
+        let db = MockDatabase::new(DatabaseBackend::Sqlite)
+            .append_query_results(vec![vec![make_model()]])
+            .append_exec_results(vec![MockExecResult {
+                last_insert_id: 0,
+                rows_affected: 1,
+            }])
+            .into_connection();
+
+        let claimed = claim_due(&db, 500, 10, |_| 900).await.unwrap();
+        assert_eq!(claimed.len(), 1);
+
+        let sql = format!("{:?}", db.into_transaction_log());
+        assert!(
+            !sql.contains("FOR UPDATE"),
+            "SQLite cannot parse FOR UPDATE: {sql}"
+        );
     }
 }

src/infra/src/table/synthetics_jobs.rs

@@ -42,14 +42,14 @@ pub struct EnqueueParams<'a> {
     pub valid_until: i64,
     /// KSUID of the parent `synthetics_runs` row.
     pub run_id: &'a str,
-    /// JSON array of `{execution_id, engine, device}` — browser monitors only. `None` for
-    /// protocol monitors.
+    /// JSON array of `{execution_id, engine, device}` — browser checks only. `None` for
+    /// protocol checks.
     pub browser_devices: Option<&'a str>,
-    /// Serialized `JobMetadata` — monitor-level context copied at enqueue time.
+    /// Serialized `JobMetadata` — check-level context copied at enqueue time.
     pub metadata: &'a str,
 }
 
-/// Monitor-level metadata copied into the job row at enqueue time.
+/// Check-level metadata copied into the job row at enqueue time.
 /// Stored as a JSON blob so new fields can be added without schema migrations.
 #[derive(serde::Serialize, serde::Deserialize, Default)]
 pub struct JobMetadata {
@@ -57,7 +57,7 @@ pub struct JobMetadata {
     pub tags: Vec<String>,
     /// The synthetic's check type ("http", "tcp", "browser", …) — copied at
     /// enqueue time so stream records (including dispatcher error records)
-    /// can carry `type` without a monitor lookup.
+    /// can carry `type` without a check lookup.
     #[serde(default)]
     pub synthetic_type: String,
 }
@@ -216,8 +216,8 @@ pub async fn get_by_id<C: ConnectionTrait>(
         .map_err(errors::Error::from)
 }
 
-/// Deletes all pending checks for a synthetic (called on monitor delete).
-pub async fn drain_monitor<C: ConnectionTrait>(
+/// Deletes all pending checks for a synthetic (called on check delete).
+pub async fn drain_check<C: ConnectionTrait>(
     conn: &C,
     synthetics_id: &str,
 ) -> Result<u64, errors::Error> {
@@ -249,7 +249,7 @@ const MAX_LEASE_BATCH: i64 = 100;
 ///
 /// `lease_secs` is a floor request, not the decision. Both probes hardcode 300s
 /// client-side while a check's retry sequence — which runs *inside* the leased
-/// job — is bounded only by `JOB_LEASE_SECS`, so an under-lease means the lease
+/// job — is bounded only by the configured job lease, so an under-lease means the lease
 /// expires while the probe is still working: the reaper terminates the job,
 /// completes the run as an error, and the probe's real result is then rejected as
 /// a stale ack. A client cannot be trusted to know how long its job may take, so
@@ -264,7 +264,7 @@ pub async fn lease_batch<C: ConnectionTrait>(
     lease_secs: i64,
     browser: Option<bool>,
 ) -> Result<Vec<LeasedRow>, errors::Error> {
-    let lease_secs = lease_secs.max(config::meta::synthetics::JOB_LEASE_SECS);
+    let lease_secs = lease_secs.max(config::meta::synthetics::limits().job_lease_secs);
     let lease_expires_at = now_us + lease_secs * 1_000_000;
 
     // `limit` arrives from a client and is cast to u64 below, where a negative
@@ -596,7 +596,7 @@ pub enum DispatchFailureOutcome {
     /// Reset to Pending; a future lease will retry it.
     Requeued,
     /// Budget exhausted. Pure decision — no DB write here. The caller already
-    /// owns terminating the job (status + run counter + monitor status, e.g.
+    /// owns terminating the job (status + run counter + check status, e.g.
     /// via the dispatcher's existing `mark_failure`), so writing an
     /// intermediate status here would just be overwritten and wastes a query.
     DeadLettered,
@@ -691,26 +691,67 @@ pub async fn failing_locations<C: ConnectionTrait>(
     conn: &C,
     run_id: &str,
 ) -> Result<Vec<String>, errors::Error> {
-    let rows = Entity::find()
+    Ok(run_location_outcomes(conn, run_id).await?.failing)
+}
+
+/// Every location of a run, split by whether it passed.
+///
+/// `failing_locations` alone could not describe a recovery: on a recovered run
+/// nothing is failing **by definition**, so the list came back empty and the
+/// message degraded to a bare count ("Locations: 2"). The reader was told a
+/// check recovered without being told where.
+///
+/// It also could not describe a *partial* recovery — "2 of 3 recovered,
+/// aws-eu-central-1 still down" — because only one side of the split was ever
+/// carried.
+///
+/// One query for both sides rather than two: the rows are already being read,
+/// and the passing/failing split is a partition of the same result set.
+#[derive(Debug, Default, PartialEq, Eq)]
+pub struct RunLocationOutcomes {
+    /// Locations that did not pass, worst first (Error > Warning > Failed).
+    pub failing: Vec<String>,
+    /// Locations that passed, alphabetical.
+    pub passing: Vec<String>,
+}
+
+pub async fn run_location_outcomes<C: ConnectionTrait>(
+    conn: &C,
+    run_id: &str,
+) -> Result<RunLocationOutcomes, errors::Error> {
+    const STATUS_PASSED: i32 = 3;
+
+    let mut rows = Entity::find()
         .select_only()
         .column(Column::Location)
         .column(Column::Status)
         .filter(Column::RunId.eq(run_id))
-        .filter(Column::Status.ne(3))
         .into_tuple::<(String, i32)>()
         .all(conn)
         .await?;
-    let mut rows = rows;
+
     // Worst first: Error(6) > Warning(5) > Failed(4). A reader scanning the first
     // line of a message should see the most severe location, not the
-    // alphabetically first.
+    // alphabetically first. Passing rows sort last and are split out below.
     rows.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0)));
-    let mut out: Vec<String> = Vec::with_capacity(rows.len());
-    for (loc, _) in rows {
-        if !out.contains(&loc) {
-            out.push(loc);
+
+    let mut out = RunLocationOutcomes::default();
+    for (loc, status) in rows {
+        // A location runs once per run, but dedupe anyway: a requeued job can
+        // leave two rows for one location, and naming it twice in a message
+        // reads as two separate outages.
+        let bucket = if status == STATUS_PASSED {
+            &mut out.passing
+        } else {
+            &mut out.failing
+        };
+        if !bucket.contains(&loc) {
+            bucket.push(loc);
         }
     }
+    // Failing is severity-ordered; passing has no severity, so alphabetical is
+    // the only stable order a reader can predict.
+    out.passing.sort();
     Ok(out)
 }
 

src/infra/src/table/synthetics_locations.rs

@@ -20,7 +20,13 @@
 //! CRUD. `pool` is the queue routing key the scheduler writes jobs into and
 //! probes/agents lease from.
 
-use sea_orm::{ColumnTrait, Condition, EntityTrait, QueryFilter, QueryOrder, Set, sea_query::Expr};
+use std::{
+    sync::LazyLock,
+    time::{Duration, Instant},
+};
+
+use sea_orm::{ColumnTrait, EntityTrait, QueryFilter, QueryOrder, Set, sea_query::Expr};
+use tokio::sync::RwLock;
 
 use super::{
     entity::synthetics_locations::{ActiveModel, Column, Entity, Model},
@@ -34,6 +40,89 @@ use crate::{
 pub const KIND_PUBLIC: &str = "public";
 pub const KIND_PRIVATE: &str = "private";
 
+/// Whole-table cache.
+///
+/// The locations registry is routing configuration: a couple of dozen rows that
+/// change only when an operator adds a region or an org creates a private
+/// location. It was being re-read on the hottest paths in the product —
+/// `dispatcher_pools()` ran `list_visible("")` **every 2 s** forever, and the
+/// scheduler called `get()` once *per location per firing check* (~200/min at
+/// 100 checks). See `docs/synthetics-lcl/2026-07-31-synthetics-caching-and-db-load.md`
+/// P3/P4.
+///
+/// Caching the whole table rather than one entry per key is deliberate: every
+/// read shape (`get` by id, `find_by_pool`, `list_visible` per org) is a
+/// different view of the same tiny row set, so one load serves all three and
+/// there is no per-key stampede to reason about.
+///
+/// TTL-based rather than event-invalidated: `coordinator::synthetics` does not
+/// exist yet, so a short TTL is what bounds staleness across pods. Writes in
+/// this module invalidate eagerly, so the TTL only covers changes made by a
+/// *different* node.
+/// The cached rows and the instant they were loaded, absent until first load.
+type CachedLocations = Option<(Vec<SyntheticsLocationRecord>, Instant)>;
+
+static LOCATIONS_CACHE: LazyLock<RwLock<CachedLocations>> = LazyLock::new(|| RwLock::new(None));
+
+const LOCATIONS_CACHE_TTL: Duration = Duration::from_secs(30);
+
+/// Drops the cached table. Called by every write path in this module so a
+/// change made on *this* node is visible immediately rather than after the TTL.
+pub async fn invalidate_cache() {
+    *LOCATIONS_CACHE.write().await = None;
+}
+
+/// Invalidates locally **and** tells every other node. Write paths call this;
+/// the coordinator watcher calls [`invalidate_cache`] so events do not echo.
+async fn invalidate_and_publish() {
+    invalidate_cache().await;
+    if let Err(e) = crate::coordinator::synthetics::emit_locations_changed().await {
+        log::error!("[synthetics] emit location cache event failed: {e}");
+    }
+}
+
+/// Refreshes the cached table if it is missing or past its TTL.
+///
+/// Two callers racing a cold cache will both query and the second write wins —
+/// harmless for an idempotent read, and cheaper than holding the write lock
+/// across a DB round trip.
+async fn ensure_fresh() -> Result<(), errors::Error> {
+    if let Some((_, loaded_at)) = LOCATIONS_CACHE.read().await.as_ref()
+        && loaded_at.elapsed() < LOCATIONS_CACHE_TTL
+    {
+        return Ok(());
+    }
+
+    let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
+    let rows: Vec<SyntheticsLocationRecord> = Entity::find()
+        .order_by_asc(Column::Kind)
+        .order_by_asc(Column::Label)
+        .all(client)
+        .await
+        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?
+        .into_iter()
+        .map(Into::into)
+        .collect();
+
+    *LOCATIONS_CACHE.write().await = Some((rows, Instant::now()));
+    Ok(())
+}
+
+/// Runs `f` over the cached rows, refreshing first if stale.
+///
+/// Callers project what they need under the read lock instead of cloning the
+/// whole table — `get`/`find_by_pool` run on every scheduler tick and every
+/// probe lease, so cloning ~20 records per call to return one would be the bulk
+/// of the work this cache exists to avoid.
+async fn with_cached<T>(
+    f: impl FnOnce(&[SyntheticsLocationRecord]) -> T,
+) -> Result<T, errors::Error> {
+    ensure_fresh().await?;
+    let guard = LOCATIONS_CACHE.read().await;
+    let rows = guard.as_ref().map(|(r, _)| r.as_slice()).unwrap_or(&[]);
+    Ok(f(rows))
+}
+
 #[derive(Debug, Clone, serde::Serialize)]
 pub struct SyntheticsLocationRecord {
     pub id: String,
@@ -78,6 +167,8 @@ pub async fn add(record: &SyntheticsLocationRecord) -> Result<(), errors::Error>
         label: Set(record.label.clone()),
         pool: Set(record.pool.clone()),
         enabled: Set(record.enabled),
+        // A new location has never been notified as down.
+        down_notified_at: Set(0),
         created_at: Set(record.created_at),
         updated_at: Set(record.updated_at),
     };
@@ -85,24 +176,31 @@ pub async fn add(record: &SyntheticsLocationRecord) -> Result<(), errors::Error>
         .exec(client)
         .await
         .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    invalidate_and_publish().await;
     Ok(())
 }
 
 /// Locations visible to an org: all public rows + the org's private rows.
+///
+/// Served from the whole-table cache; the org filter is applied in Rust. Note
+/// `list_visible("")` matches public rows only, which is how `dispatcher_pools()`
+/// enumerates the public net pools.
 pub async fn list_visible(org_id: &str) -> Result<Vec<SyntheticsLocationRecord>, errors::Error> {
-    let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
-    let rows = Entity::find()
-        .filter(
-            Condition::any()
-                .add(Column::OrgId.is_null())
-                .add(Column::OrgId.eq(org_id)),
-        )
-        .order_by_asc(Column::Kind)
-        .order_by_asc(Column::Label)
-        .all(client)
-        .await
-        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
-    Ok(rows.into_iter().map(Into::into).collect())
+    with_cached(|rows| {
+        rows.iter()
+            .filter(|r| match &r.org_id {
+                None => true,
+                Some(o) => o == org_id,
+            })
+            .cloned()
+            .collect()
+    })
+    .await
 }
 
 /// All private rows across orgs — used by the staleness watcher.
@@ -116,25 +214,14 @@ pub async fn list_private() -> Result<Vec<SyntheticsLocationRecord>, errors::Err
     Ok(rows.into_iter().map(Into::into).collect())
 }
 
-/// Find one location by id.
+/// Find one location by id. Served from the whole-table cache.
 pub async fn get(id: &str) -> Result<Option<SyntheticsLocationRecord>, errors::Error> {
-    let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
-    let row = Entity::find_by_id(id)
-        .one(client)
-        .await
-        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
-    Ok(row.map(Into::into))
+    with_cached(|rows| rows.iter().find(|r| r.id == id).cloned()).await
 }
 
-/// Find one location by its pool routing key.
+/// Find one location by its pool routing key. Served from the whole-table cache.
 pub async fn find_by_pool(pool: &str) -> Result<Option<SyntheticsLocationRecord>, errors::Error> {
-    let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
-    let row = Entity::find()
-        .filter(Column::Pool.eq(pool))
-        .one(client)
-        .await
-        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
-    Ok(row.map(Into::into))
+    with_cached(|rows| rows.iter().find(|r| r.pool == pool).cloned()).await
 }
 
 /// Update label/enabled on a location.
@@ -150,6 +237,12 @@ pub async fn update(id: &str, label: &str, enabled: bool) -> Result<(), errors::
         .exec(client)
         .await
         .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    invalidate_and_publish().await;
     Ok(())
 }
 
@@ -161,5 +254,57 @@ pub async fn remove(id: &str) -> Result<(), errors::Error> {
         .exec(client)
         .await
         .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    invalidate_and_publish().await;
+    Ok(())
+}
+
+/// Claims the right to send this location's "location down" notification,
+/// returning whether THIS caller won it.
+///
+/// The staleness watcher runs on **every** alert_manager node. Its suppression
+/// flag used to be an in-process `HashSet`, so N nodes each decided
+/// independently that they had not notified yet and N notifications went out for
+/// one outage — repeating on every down → recover → down cycle.
+///
+/// `WHERE down_notified_at = 0` makes the transition a compare-and-swap: exactly
+/// one node flips 0 → now and gets `rows_affected == 1`; the rest see 0 and stay
+/// quiet. Same primitive as `synthetics_jobs::lease_batch` and
+/// `synthetics_checks::try_claim_slot`.
+///
+/// Deliberately NOT cached — `LOCATIONS_CACHE` serves definition reads, and a
+/// stale `down_notified_at` would reintroduce exactly the duplicate this fixes.
+pub async fn try_claim_down_notification(id: &str, now_us: i64) -> Result<bool, errors::Error> {
+    let _lock = get_lock().await;
+    let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
+    let res = Entity::update_many()
+        .col_expr(Column::DownNotifiedAt, Expr::value(now_us))
+        .filter(Column::Id.eq(id))
+        .filter(Column::DownNotifiedAt.eq(0i64))
+        .exec(client)
+        .await
+        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
+    Ok(res.rows_affected > 0)
+}
+
+/// Clears the down flag so a future outage notifies again.
+///
+/// Idempotent and unguarded on purpose: every node may call this on recovery and
+/// the result is the same. Guarding it would leave the flag set if the one node
+/// that "won" the clear died before the next tick, silencing the next outage.
+pub async fn clear_down_notification(id: &str) -> Result<(), errors::Error> {
+    let _lock = get_lock().await;
+    let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
+    Entity::update_many()
+        .col_expr(Column::DownNotifiedAt, Expr::value(0i64))
+        .filter(Column::Id.eq(id))
+        .filter(Column::DownNotifiedAt.ne(0i64))
+        .exec(client)
+        .await
+        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
     Ok(())
 }

src/infra/src/table/synthetics_probe_tokens.rs

@@ -19,6 +19,12 @@
 //! (`/synthetics/jobs/resolve`, `/ack`, `/lease`). Separate from
 //! `org_ingestion_tokens` (`o2oi_`) which is write-only ingest.
 
+use std::{
+    sync::LazyLock,
+    time::{Duration, Instant},
+};
+
+use config::RwHashMap;
 use sea_orm::{
     ColumnTrait, EntityTrait, QueryFilter, Set, SqlErr, TransactionTrait, sea_query::Expr,
 };
@@ -34,6 +40,51 @@ use crate::{
 
 pub const SYNTHETICS_PROBE_TOKEN_PREFIX: &str = "o2syn_";
 
+/// Token → lookup result, with the time it was loaded.
+///
+/// `find_global` runs in the auth middleware on **every** probe request
+/// (`api/common/src/auth/validator.rs`), so it fired once per lease, resolve and
+/// ack — the single highest-frequency query in the feature, always for the same
+/// handful of token strings.
+///
+/// `None` is cached as well as `Some`: an unknown or disabled token must not be
+/// able to force a DB round trip per request, which is what makes an uncached
+/// validator a cheap way to generate load from outside.
+///
+/// **The TTL is a security parameter, not a performance one.** It bounds how
+/// long a disabled token keeps working. 10 s is short enough that revocation is
+/// effectively immediate for an operator and long enough to remove ~99 % of the
+/// queries. Writes in this module invalidate eagerly, so the TTL only covers a
+/// disable performed on a *different* node.
+static TOKEN_CACHE: LazyLock<RwHashMap<String, (Option<SyntheticsProbeTokenRecord>, Instant)>> =
+    LazyLock::new(Default::default);
+
+/// Org → its default enabled token, same TTL rules as [`TOKEN_CACHE`].
+static DEFAULT_TOKEN_CACHE: LazyLock<
+    RwHashMap<String, (Option<SyntheticsProbeTokenRecord>, Instant)>,
+> = LazyLock::new(Default::default);
+
+const TOKEN_CACHE_TTL: Duration = Duration::from_secs(10);
+
+/// Clears both token caches. Called by every write path here so a disable,
+/// rotate or default-change takes effect on this node immediately.
+pub fn invalidate_cache() {
+    TOKEN_CACHE.clear();
+    DEFAULT_TOKEN_CACHE.clear();
+}
+
+/// Invalidates locally **and** tells every other node.
+///
+/// This is the revocation path: `set_enabled(false)` reaches other nodes as a
+/// coordinator event rather than waiting out their TTL, which is what makes a
+/// disable effectively fleet-wide-immediate.
+async fn invalidate_and_publish(org_id: &str) {
+    invalidate_cache();
+    if let Err(e) = crate::coordinator::synthetics::emit_tokens_changed(org_id).await {
+        log::error!("[synthetics] emit probe token cache event failed for {org_id}: {e}");
+    }
+}
+
 /// Name of the token every org starts with (backfilled / created at org
 /// creation). Named tokens minted later carry their own operator-chosen name.
 pub const DEFAULT_TOKEN_NAME: &str = "default";
@@ -93,7 +144,16 @@ pub async fn add(record: &SyntheticsProbeTokenRecord) -> Result<(), errors::Erro
     };
     let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
     match Entity::insert(model).exec(client).await {
-        Ok(_) => Ok(()),
+        Ok(_) => {
+            // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+            // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+            // returns, so emitting while holding it deadlocks the process — and the
+            // mutex is then held forever, hanging every later synthetics query.
+            drop(_lock);
+            // A new token can become the org default, so both caches are stale.
+            invalidate_and_publish(&record.org_id).await;
+            Ok(())
+        }
         Err(e) => match e.sql_err() {
             Some(SqlErr::UniqueConstraintViolation(_)) => {
                 Err(Error::DbError(DbError::SeaORMError(format!(
@@ -112,14 +172,23 @@ pub async fn add(record: &SyntheticsProbeTokenRecord) -> Result<(), errors::Erro
 /// path (the tenant boundary). Matches ANY enabled token, so old + new tokens
 /// coexist during a rotation overlap window.
 pub async fn find_global(token: &str) -> Result<Option<SyntheticsProbeTokenRecord>, errors::Error> {
+    if let Some(entry) = TOKEN_CACHE.get(token)
+        && entry.1.elapsed() < TOKEN_CACHE_TTL
+    {
+        return Ok(entry.0.clone());
+    }
+
     let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
     let record = Entity::find()
         .filter(Column::Token.eq(token))
         .filter(Column::Enabled.eq(true))
         .one(client)
         .await
-        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
-    Ok(record.map(SyntheticsProbeTokenRecord::from))
+        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?
+        .map(SyntheticsProbeTokenRecord::from);
+
+    TOKEN_CACHE.insert(token.to_string(), (record.clone(), Instant::now()));
+    Ok(record)
 }
 
 /// Find the org's default enabled probe token — the one handed out by
@@ -127,6 +196,12 @@ pub async fn find_global(token: &str) -> Result<Option<SyntheticsProbeTokenRecor
 pub async fn find_default(
     org_id: &str,
 ) -> Result<Option<SyntheticsProbeTokenRecord>, errors::Error> {
+    if let Some(entry) = DEFAULT_TOKEN_CACHE.get(org_id)
+        && entry.1.elapsed() < TOKEN_CACHE_TTL
+    {
+        return Ok(entry.0.clone());
+    }
+
     let client = ORM_CLIENT.get_or_init(connect_to_orm).await;
     let record = Entity::find()
         .filter(Column::OrgId.eq(org_id))
@@ -134,8 +209,11 @@ pub async fn find_default(
         .filter(Column::Enabled.eq(true))
         .one(client)
         .await
-        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
-    Ok(record.map(SyntheticsProbeTokenRecord::from))
+        .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?
+        .map(SyntheticsProbeTokenRecord::from);
+
+    DEFAULT_TOKEN_CACHE.insert(org_id.to_string(), (record.clone(), Instant::now()));
+    Ok(record)
 }
 
 /// List all probe tokens for an org (enabled + disabled), newest-default first.
@@ -172,8 +250,12 @@ pub async fn get_by_name(
 }
 
 /// Enable or disable a token by `(org_id, name)`. Disabling is how a token is
-/// revoked — because there is no validator cache for probe tokens, it takes
-/// effect on the next request (no invalidation step needed).
+/// revoked.
+///
+/// Revocation is immediate **on this node** — the caches are cleared below. On
+/// other nodes it takes effect within `TOKEN_CACHE_TTL` (10 s), because there is
+/// no coordinator channel for synthetics yet. If instant fleet-wide revocation
+/// is ever required, that channel is the thing to build; do not remove the cache.
 pub async fn set_enabled(org_id: &str, name: &str, enabled: bool) -> Result<(), errors::Error> {
     let _lock = get_lock().await;
     let now = chrono::Utc::now().timestamp_micros();
@@ -186,6 +268,12 @@ pub async fn set_enabled(org_id: &str, name: &str, enabled: bool) -> Result<(),
         .exec(client)
         .await
         .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    invalidate_and_publish(org_id).await;
     Ok(())
 }
 
@@ -219,6 +307,12 @@ pub async fn set_default(org_id: &str, name: &str) -> Result<(), errors::Error>
     txn.commit()
         .await
         .map_err(|e| Error::DbError(DbError::SeaORMError(e.to_string())))?;
+    // Release the SQLite write mutex BEFORE emitting. On a sqlite meta_store
+    // the coordinator's put() takes the *same* CLIENT_RW lock `get_lock()`
+    // returns, so emitting while holding it deadlocks the process — and the
+    // mutex is then held forever, hanging every later synthetics query.
+    drop(_lock);
+    invalidate_and_publish(org_id).await;
     Ok(())
 }
 

src/infra/src/table/synthetics_runs.rs

@@ -95,7 +95,7 @@ pub struct ListRunsParams<'a> {
     pub page_size: i64,
 }
 
-/// Lists runs for a monitor in reverse chronological order, with optional time range filter.
+/// Lists runs for a check in reverse chronological order, with optional time range filter.
 /// Returns (rows, total_count).
 pub async fn list_runs<C: ConnectionTrait>(
     conn: &C,
@@ -221,20 +221,53 @@ pub async fn get_run<C: ConnectionTrait>(
         .map_err(errors::Error::from)
 }
 
-/// Atomically increments `jobs_done` and updates `run_result` to worst-case severity.
-/// Sets `completed_at` when `jobs_done + 1 >= job_count`.
+/// Records one job's outcome against its run, and returns `Some((run_result,
+/// job_count))` **only to the caller that actually completed the run** — `None`
+/// to everyone else.
 ///
 /// `job_result` uses SyntheticStatus DB integers: 1=Passed, 2=Warning, 3=Failed, 4=Error.
-/// Higher integer = higher severity, so `CASE WHEN ... > run_result` naturally picks worst.
+/// Higher integer = higher severity, so the `CASE` naturally keeps the worst.
 ///
-/// Returns `Some((run_result, job_count))` when the run is now complete (all jobs have acked),
-/// or `None` if jobs are still in progress.
+/// # Why completion is claimed rather than observed
+///
+/// The increment was always atomic (`jobs_done = jobs_done + 1` is computed by
+/// the database). **The completion decision was not:** it re-read the row in a
+/// second statement and returned `Some` whenever `jobs_done >= job_count`. Two
+/// jobs of the same run acking concurrently both saw the final count:
+///
+/// ```text
+/// ack A: UPDATE -> jobs_done = 1
+/// ack B: UPDATE -> jobs_done = 2
+/// ack A: SELECT -> 2 >= 2 -> Some   "I completed the run"
+/// ack B: SELECT -> 2 >= 2 -> Some   "I completed the run"   <-- twice
+/// ```
+///
+/// Everything the caller does on completion then happened twice: the run
+/// rollup, the check status write, and the alert dispatch — a duplicate
+/// customer-facing notification. This needed no multiple alert managers to
+/// reach: acks are served on ingesters (2 replicas in every environment) and the
+/// probes of one run finish independently.
+///
+/// Now `completed_at` is the guard. Exactly one caller flips it from NULL, and
+/// only that caller is told the run is complete. `completed_at IS NULL` is a
+/// compare-and-swap in the same shape as `synthetics_jobs::lease_batch`'s
+/// `status = 0`.
+///
+/// Deliberately not `RETURNING` (which would fold this into one statement):
+/// SQLite only gained it in 3.35 and the bundled version is not pinned here,
+/// whereas a guarded UPDATE plus `rows_affected` behaves identically on every
+/// backend.
 pub async fn increment_jobs_done<C: ConnectionTrait>(
     conn: &C,
     run_id: &str,
     job_result: i32,
     now_us: i64,
 ) -> Result<Option<(i32, i32)>, errors::Error> {
+    // Step 1: count this job and fold its severity in.
+    //
+    // `jobs_done < job_count` stops a duplicate or replayed ack from pushing the
+    // counter past the total, which would leave the run permanently "over
+    // complete" and, before the guard below existed, complete it twice.
     let update_sql = r#"
         UPDATE synthetics_runs
         SET
@@ -242,26 +275,43 @@ pub async fn increment_jobs_done<C: ConnectionTrait>(
             run_result = CASE
                 WHEN COALESCE(run_result, 0) >= $1 THEN run_result
                 ELSE $1
-            END,
-            completed_at = CASE
-                WHEN jobs_done + 1 >= job_count THEN $2
-                ELSE completed_at
             END
-        WHERE id = $3
+        WHERE id = $2 AND jobs_done < job_count
     "#;
     conn.execute(Statement::from_sql_and_values(
         conn.get_database_backend(),
         update_sql,
-        [
-            Value::from(job_result),
-            Value::from(now_us),
-            Value::from(run_id.to_owned()),
-        ],
+        [Value::from(job_result), Value::from(run_id.to_owned())],
     ))
     .await?;
 
+    // Step 2: claim completion. Only the caller whose increment brought
+    // `jobs_done` up to `job_count` finds `completed_at` still NULL.
+    let complete_sql = r#"
+        UPDATE synthetics_runs
+        SET completed_at = $1
+        WHERE id = $2 AND jobs_done >= job_count AND completed_at IS NULL
+    "#;
+    let claimed = conn
+        .execute(Statement::from_sql_and_values(
+            conn.get_database_backend(),
+            complete_sql,
+            [Value::from(now_us), Value::from(run_id.to_owned())],
+        ))
+        .await?
+        .rows_affected();
+
+    if claimed == 0 {
+        // Either jobs are still outstanding, or another ack already completed
+        // this run. Both mean: not ours to report.
+        return Ok(None);
+    }
+
+    // Step 3: we own the completion, so read back what the caller needs to
+    // report. Safe to read outside a transaction — nothing rewrites these two
+    // once `completed_at` is set.
     let check_sql = r#"
-        SELECT jobs_done, job_count, run_result FROM synthetics_runs WHERE id = $1
+        SELECT job_count, run_result FROM synthetics_runs WHERE id = $1
     "#;
     let rows = conn
         .query_all(Statement::from_sql_and_values(
@@ -271,17 +321,13 @@ pub async fn increment_jobs_done<C: ConnectionTrait>(
         ))
         .await?;
 
-    if let Some(row) = rows.into_iter().next() {
-        let done: i32 = row.try_get("", "jobs_done").unwrap_or(0);
-        let count: i32 = row.try_get("", "job_count").unwrap_or(1);
-        if done >= count {
+    match rows.into_iter().next() {
+        Some(row) => {
+            let count: i32 = row.try_get("", "job_count").unwrap_or(1);
             let result: Option<i32> = row.try_get("", "run_result").unwrap_or(None);
             Ok(Some((result.unwrap_or(job_result), count)))
-        } else {
-            Ok(None)
         }
-    } else {
-        Ok(None)
+        None => Ok(None),
     }
 }
 
@@ -323,4 +369,117 @@ mod tests {
         assert_eq!(r.jobs_done, 1);
         assert_eq!(r.run_result, Some(1));
     }
+
+    /// One raw row as `query_all` sees it — the completion read-back selects
+    /// only these two columns.
+    fn run_row(
+        job_count: i32,
+        run_result: i32,
+    ) -> std::collections::BTreeMap<String, sea_orm::Value> {
+        let mut m = std::collections::BTreeMap::new();
+        m.insert("job_count".to_owned(), sea_orm::Value::from(job_count));
+        m.insert("run_result".to_owned(), sea_orm::Value::from(run_result));
+        m
+    }
+
+    /// Two acks of the same run must not both be told they completed it.
+    /// Before `completed_at` became the guard, both re-read `jobs_done >=
+    /// job_count` and both returned Some — so the run rollup, the check status
+    /// write and the ALERT all fired twice.
+    #[tokio::test]
+    async fn test_only_one_ack_is_told_it_completed_the_run() {
+        use sea_orm::{DatabaseBackend, MockDatabase, MockExecResult};
+
+        // Ack A: increment succeeds, and it wins the completed_at claim.
+        let winner = MockDatabase::new(DatabaseBackend::Postgres)
+            .append_exec_results(vec![
+                MockExecResult {
+                    last_insert_id: 0,
+                    rows_affected: 1,
+                }, // increment
+                MockExecResult {
+                    last_insert_id: 0,
+                    rows_affected: 1,
+                }, // claimed completion
+            ])
+            .append_query_results(vec![vec![run_row(2, 3)]])
+            .into_connection();
+        let a = increment_jobs_done(&winner, "run-1", 3, 100).await.unwrap();
+        assert_eq!(a, Some((3, 2)), "the ack that completed the run reports it");
+
+        // Ack B: increment is a no-op (already at job_count) and the
+        // completed_at claim matches nothing, because A already set it.
+        let loser = MockDatabase::new(DatabaseBackend::Postgres)
+            .append_exec_results(vec![
+                MockExecResult {
+                    last_insert_id: 0,
+                    rows_affected: 0,
+                }, // increment guarded out
+                MockExecResult {
+                    last_insert_id: 0,
+                    rows_affected: 0,
+                }, // completion already claimed
+            ])
+            .into_connection();
+        let b = increment_jobs_done(&loser, "run-1", 3, 100).await.unwrap();
+        assert_eq!(
+            b, None,
+            "a second ack must NOT be told it completed the run — that is the duplicate alert"
+        );
+    }
+
+    /// A run with jobs still outstanding reports nothing: the completion claim
+    /// matches no row because `jobs_done < job_count`.
+    #[tokio::test]
+    async fn test_incomplete_run_reports_none() {
+        use sea_orm::{DatabaseBackend, MockDatabase, MockExecResult};
+
+        let db = MockDatabase::new(DatabaseBackend::Postgres)
+            .append_exec_results(vec![
+                MockExecResult {
+                    last_insert_id: 0,
+                    rows_affected: 1,
+                }, // counted
+                MockExecResult {
+                    last_insert_id: 0,
+                    rows_affected: 0,
+                }, // not complete yet
+            ])
+            .into_connection();
+        assert_eq!(
+            increment_jobs_done(&db, "run-1", 1, 100).await.unwrap(),
+            None
+        );
+    }
+
+    /// The completion claim must guard on `completed_at IS NULL` — that clause
+    /// is the entire fix.
+    #[tokio::test]
+    async fn test_completion_claim_guards_on_completed_at() {
+        use sea_orm::{DatabaseBackend, MockDatabase, MockExecResult};
+
+        let db = MockDatabase::new(DatabaseBackend::Postgres)
+            .append_exec_results(vec![
+                MockExecResult {
+                    last_insert_id: 0,
+                    rows_affected: 1,
+                },
+                MockExecResult {
+                    last_insert_id: 0,
+                    rows_affected: 0,
+                },
+            ])
+            .into_connection();
+        let _ = increment_jobs_done(&db, "run-1", 1, 100).await.unwrap();
+
+        let sql = format!("{:?}", db.into_transaction_log());
+        assert!(
+            sql.contains("completed_at IS NULL"),
+            "without this guard both acks complete the run: {sql}"
+        );
+        assert!(
+            sql.contains("jobs_done < job_count"),
+            "the increment must be guarded so a replayed ack cannot overshoot: {sql}"
+        );
+    }
 }

src/jobs/src/job/mod.rs

@@ -500,6 +500,10 @@ pub async fn init() -> Result<(), anyhow::Error> {
     tokio::task::spawn(db::alerts::destinations::watch());
     tokio::task::spawn(db::alerts::realtime_triggers::watch());
     tokio::task::spawn(db::alerts::alert::watch());
+    // Synthetics config caches (checks, locations, probe tokens, agents) live on
+    // every node, so every node must hear invalidations — including routers,
+    // which serve the probe auth path.
+    tokio::task::spawn(infra::coordinator::synthetics::watch());
     // org_settings_watch already started above for all nodes including routers
     // Watch needed on queriers (UI APIs) and on whichever node role is the configured
     // processing node (ingester or compactor) so their local cache stays in sync with

src/main.rs

@@ -541,6 +541,32 @@ async fn init_enterprise() -> Result<(), anyhow::Error> {
         panic!("ratelimit config error: {e}");
     }
 
+    // Push the synthetics limits from enterprise config into the OSS validator.
+    // Synthetics is enterprise-only, so the values live in SyntheticsConfig, but
+    // the check validation they bound lives in `config` — which cannot depend on
+    // o2_enterprise. This is the seam.
+    //
+    // Deliberately NOT fatal, unlike ratelimit above: synthetics is one feature,
+    // and refusing to start the whole application — ingest, search, dashboards —
+    // because a probe ceiling is misconfigured would be the worse outage. On
+    // rejection nothing is installed and validation keeps using the conservative
+    // built-in defaults, so the failure mode is "stricter than intended", not
+    // "accepts checks that get killed mid-run".
+    if let Err(e) =
+        config::meta::synthetics::init_limits(config::meta::synthetics::SyntheticsLimits {
+            job_lease_secs: o2cfg.synthetics.job_lease_secs,
+            max_check_budget_secs: o2cfg.synthetics.max_check_budget_secs,
+            max_net_timeout_ms: o2cfg.synthetics.max_net_timeout_ms,
+        })
+    {
+        log::error!(
+            "synthetics limits config rejected, falling back to defaults \
+             (budget={}s lease={}s): {e}",
+            config::meta::synthetics::DEFAULT_MAX_CHECK_BUDGET_SECS,
+            config::meta::synthetics::DEFAULT_JOB_LEASE_SECS,
+        );
+    }
+
     o2_enterprise::enterprise::pipeline::pipeline_file_server::PipelineFileServer::run().await?;
     if o2cfg.rate_limit.rate_limit_enabled && o2_openfga::config::get_config().enabled {
         o2_ratelimit::init(

src/search/Cargo.toml

@@ -29,6 +29,7 @@ flight.workspace = true
 futures.workspace = true
 futures-util.workspace = true
 hashbrown.workspace = true
+hashlink.workspace = true
 infra.workspace = true
 itertools.workspace = true
 log.workspace = true

src/search/src/cache/mod.rs

@@ -0,0 +1,16 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+pub mod streaming_agg;

src/search/src/cache/streaming_agg/discovery.rs

@@ -0,0 +1,484 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use config::meta::search::Interval;
+
+use super::files::STREAMING_AGGS_CACHE_DIR;
+
+/// Time conversion constants
+pub const MICROS_PER_SECOND: i64 = 1_000_000;
+pub const MICROS_PER_MINUTE: i64 = 60 * MICROS_PER_SECOND;
+
+/// Represents a single cache file entry with its metadata
+#[derive(Debug, Clone, PartialEq)]
+pub struct CacheEntry {
+    pub file_path: String,
+    pub start_time: i64,
+    pub end_time: i64,
+    pub interval: Interval,
+}
+
+impl CacheEntry {
+    /// Checks if this cache entry overlaps with the given time range
+    pub fn overlaps_with(&self, start: i64, end: i64, interval: Interval) -> bool {
+        self.start_time < end && self.end_time > start && self.interval <= interval
+    }
+}
+
+/// Represents a time range that is not covered by cache
+#[derive(Debug, Clone, PartialEq)]
+pub struct TimeRange {
+    pub start_time: i64,
+    pub end_time: i64,
+}
+
+impl TimeRange {
+    pub fn new(start_time: i64, end_time: i64) -> Self {
+        Self {
+            start_time,
+            end_time,
+        }
+    }
+}
+
+/// Result of cache discovery operation
+#[derive(Debug, Clone)]
+pub struct CacheDiscoveryResult {
+    pub cached_ranges: Vec<CacheEntry>,
+    pub uncached_ranges: Vec<TimeRange>,
+    pub cache_coverage_ratio: f64,
+}
+
+impl CacheDiscoveryResult {
+    pub fn new(
+        cached_ranges: Vec<CacheEntry>,
+        uncached_ranges: Vec<TimeRange>,
+        cache_coverage_ratio: f64,
+    ) -> Self {
+        Self {
+            cached_ranges,
+            uncached_ranges,
+            cache_coverage_ratio,
+        }
+    }
+
+    /// Creates an empty discovery result (no cache available) for the given time range
+    pub fn empty(start_time: i64, end_time: i64) -> Self {
+        Self {
+            cached_ranges: vec![],
+            uncached_ranges: vec![TimeRange::new(start_time, end_time)],
+            cache_coverage_ratio: 0.0,
+        }
+    }
+
+    /// Returns true if the entire query range is cached
+    pub fn is_fully_cached(&self) -> bool {
+        self.uncached_ranges.is_empty() && self.cache_coverage_ratio >= 1.0
+    }
+
+    /// Returns true if no cache is available
+    pub fn has_no_cache(&self) -> bool {
+        self.cache_coverage_ratio == 0.0
+    }
+}
+
+/// Discovers existing cache files for a given query
+///
+/// # Arguments
+/// * `cache_file_path` - The base cache file path (e.g., "org_id/stream_type/stream_name/hash")
+/// * `query_start` - Query start time in microseconds
+/// * `query_end` - Query end time in microseconds
+/// * `query_interval` - The interval of the query
+///
+/// # Returns
+/// * `Ok(CacheDiscoveryResult)` - Discovery result with cached and uncached ranges
+/// * `Err(std::io::Error)` - If cache directory cannot be accessed
+pub async fn discover_cache_for_query(
+    cache_file_path: &str,
+    query_start: i64,
+    query_end: i64,
+    query_interval: Interval,
+) -> std::io::Result<CacheDiscoveryResult> {
+    let cache_path = format!(
+        "{}{}/{}",
+        config::get_config().common.data_cache_dir,
+        STREAMING_AGGS_CACHE_DIR,
+        cache_file_path
+    );
+
+    // If cache directory doesn't exist, return empty result
+    if !tokio::fs::try_exists(&cache_path).await.unwrap_or(false) {
+        return Ok(CacheDiscoveryResult::new(
+            vec![],
+            vec![TimeRange::new(query_start, query_end)],
+            0.0,
+        ));
+    }
+
+    // Read all cache files in the directory asynchronously
+    let mut read_dir = tokio::fs::read_dir(&cache_path).await?;
+    let mut files = Vec::new();
+
+    // Collect all directory entries
+    while let Some(entry) = read_dir.next_entry().await? {
+        files.push(entry);
+    }
+
+    let mut cache_entries = Vec::new();
+
+    // Parse each cache file to extract metadata
+    for file in files {
+        let file_name = file.file_name();
+        let file_name_str = match file_name.to_str() {
+            Some(name) => name,
+            None => continue,
+        };
+
+        // Skip temporary files
+        if file_name_str.contains("_tmp") {
+            continue;
+        }
+
+        // Parse the cache entry
+        if let Some(entry) = parse_cache_file_name(cache_file_path, file_name_str) {
+            // Only include entries that overlap with the query range
+            if entry.overlaps_with(query_start, query_end, query_interval) {
+                cache_entries.push(entry);
+            }
+        }
+    }
+
+    // Sort cache entries by start time
+    cache_entries.sort_by_key(|e| e.start_time);
+
+    // Calculate uncached ranges (gaps in coverage)
+    let uncached_ranges = calculate_uncached_ranges(&cache_entries, query_start, query_end);
+
+    // Calculate cache coverage ratio
+    let total_duration = query_end - query_start;
+    let cached_duration = calculate_cached_duration(&cache_entries, query_start, query_end);
+    let cache_coverage_ratio = if total_duration > 0 {
+        cached_duration as f64 / total_duration as f64
+    } else {
+        0.0
+    };
+
+    Ok(CacheDiscoveryResult::new(
+        cache_entries,
+        uncached_ranges,
+        cache_coverage_ratio,
+    ))
+}
+
+/// Parses a cache file name to extract cache entry metadata
+fn parse_cache_file_name(cache_file_path: &str, file_name: &str) -> Option<CacheEntry> {
+    // Remove file extension before parsing timestamps
+    let name_without_ext = file_name
+        .rsplit_once('.')
+        .map(|(name, _)| name)
+        .unwrap_or(file_name);
+
+    let parts: Vec<&str> = name_without_ext.split('_').collect();
+    if parts.len() < 2 {
+        return None;
+    }
+
+    let start_time = parts[0].parse::<i64>().ok()?;
+    let end_time = parts[1].parse::<i64>().ok()?;
+
+    // Calculate interval from the time range duration in the filename
+    // The interval is inferred from the duration (end_time - start_time)
+    let duration_micros = end_time - start_time;
+    let interval = interval_from_duration_micros(duration_micros);
+
+    // Build the full cache file path
+    let full_cache_path = format!("{STREAMING_AGGS_CACHE_DIR}/{cache_file_path}/{file_name}",);
+
+    Some(CacheEntry {
+        file_path: full_cache_path,
+        start_time,
+        end_time,
+        interval,
+    })
+}
+
+/// Infers the interval enum from duration in microseconds
+///
+/// # Arguments
+/// * `duration_micros` - Duration in microseconds (expected range: 5 min to 1 day)
+///
+/// # Returns
+/// The corresponding `Interval` enum variant based on the duration in minutes
+fn interval_from_duration_micros(duration_micros: i64) -> Interval {
+    // Convert microseconds to minutes
+    let duration_minutes = duration_micros / MICROS_PER_MINUTE;
+    Interval::from(duration_minutes)
+}
+
+/// Calculates the uncached ranges (gaps) in the query time range
+fn calculate_uncached_ranges(
+    cache_entries: &[CacheEntry],
+    query_start: i64,
+    query_end: i64,
+) -> Vec<TimeRange> {
+    if cache_entries.is_empty() {
+        return vec![TimeRange::new(query_start, query_end)];
+    }
+
+    let mut uncached_ranges = Vec::new();
+    let mut current_time = query_start;
+
+    for entry in cache_entries {
+        // If there's a gap before this entry, add it as uncached
+        if current_time < entry.start_time {
+            uncached_ranges.push(TimeRange::new(
+                current_time,
+                entry.start_time.min(query_end),
+            ));
+        }
+
+        // Move current time to the end of this cached entry
+        current_time = current_time.max(entry.end_time);
+
+        // If we've covered the entire query range, stop
+        if current_time >= query_end {
+            break;
+        }
+    }
+
+    // If there's still time left after the last cache entry, add it as uncached
+    if current_time < query_end {
+        uncached_ranges.push(TimeRange::new(current_time, query_end));
+    }
+
+    uncached_ranges
+}
+
+/// Calculates the total cached duration within the query range
+fn calculate_cached_duration(
+    cache_entries: &[CacheEntry],
+    query_start: i64,
+    query_end: i64,
+) -> i64 {
+    let mut total_cached = 0i64;
+    let mut last_covered_time = query_start;
+
+    for entry in cache_entries {
+        // Calculate the overlap between this cache entry and the query range
+        let overlap_start = entry.start_time.max(last_covered_time);
+        let overlap_end = entry.end_time.min(query_end);
+
+        if overlap_start < overlap_end {
+            total_cached += overlap_end - overlap_start;
+            last_covered_time = overlap_end;
+        }
+
+        if last_covered_time >= query_end {
+            break;
+        }
+    }
+
+    total_cached
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_cache_entry_overlaps_with() {
+        let interval = Interval::OneHour;
+        let entry = CacheEntry {
+            file_path: "test.arrow".to_string(),
+            start_time: 1000,
+            end_time: 2000,
+            interval,
+        };
+
+        // Test overlapping ranges
+        assert!(entry.overlaps_with(500, 1500, interval)); // Starts before, ends during
+        assert!(entry.overlaps_with(1500, 2500, interval)); // Starts during, ends after
+        assert!(entry.overlaps_with(1200, 1800, interval)); // Completely within
+        assert!(entry.overlaps_with(500, 2500, interval)); // Completely encompasses
+
+        // Test non-overlapping ranges
+        assert!(!entry.overlaps_with(0, 1000, interval)); // Ends exactly at start
+        assert!(!entry.overlaps_with(2000, 3000, interval)); // Starts exactly at end
+        assert!(!entry.overlaps_with(0, 500, interval)); // Completely before
+        assert!(!entry.overlaps_with(2500, 3000, interval)); // Completely after
+    }
+
+    #[test]
+    fn test_interval_from_minutes() {
+        assert_eq!(Interval::from(0), Interval::Zero);
+        assert_eq!(Interval::from(5), Interval::FiveMinutes);
+        assert_eq!(Interval::from(10), Interval::TenMinutes);
+        assert_eq!(Interval::from(30), Interval::ThirtyMinutes);
+        assert_eq!(Interval::from(60), Interval::OneHour);
+        assert_eq!(Interval::from(120), Interval::TwoHours);
+        assert_eq!(Interval::from(360), Interval::SixHours);
+        assert_eq!(Interval::from(720), Interval::TwelveHours);
+        assert_eq!(Interval::from(1440), Interval::OneDay);
+        assert_eq!(Interval::from(999), Interval::Zero); // Unknown interval
+    }
+
+    #[test]
+    fn test_calculate_uncached_ranges_no_cache() {
+        let cache_entries = vec![];
+        let ranges = calculate_uncached_ranges(&cache_entries, 0, 10000);
+
+        assert_eq!(ranges.len(), 1);
+        assert_eq!(ranges[0].start_time, 0);
+        assert_eq!(ranges[0].end_time, 10000);
+    }
+
+    #[test]
+    fn test_calculate_uncached_ranges_fully_cached() {
+        let cache_entries = vec![
+            CacheEntry {
+                file_path: "test1.arrow".to_string(),
+                start_time: 0,
+                end_time: 5000,
+                interval: Interval::OneHour,
+            },
+            CacheEntry {
+                file_path: "test2.arrow".to_string(),
+                start_time: 5000,
+                end_time: 10000,
+                interval: Interval::OneHour,
+            },
+        ];
+
+        let ranges = calculate_uncached_ranges(&cache_entries, 0, 10000);
+        assert_eq!(ranges.len(), 0); // No gaps
+    }
+
+    #[test]
+    fn test_calculate_uncached_ranges_with_gaps() {
+        let cache_entries = vec![
+            CacheEntry {
+                file_path: "test1.arrow".to_string(),
+                start_time: 1000,
+                end_time: 2000,
+                interval: Interval::OneHour,
+            },
+            CacheEntry {
+                file_path: "test2.arrow".to_string(),
+                start_time: 5000,
+                end_time: 7000,
+                interval: Interval::OneHour,
+            },
+        ];
+
+        let ranges = calculate_uncached_ranges(&cache_entries, 0, 10000);
+
+        assert_eq!(ranges.len(), 3);
+        assert_eq!(ranges[0].start_time, 0);
+        assert_eq!(ranges[0].end_time, 1000);
+        assert_eq!(ranges[1].start_time, 2000);
+        assert_eq!(ranges[1].end_time, 5000);
+        assert_eq!(ranges[2].start_time, 7000);
+        assert_eq!(ranges[2].end_time, 10000);
+    }
+
+    #[test]
+    fn test_calculate_cached_duration_no_cache() {
+        let cache_entries = vec![];
+        let duration = calculate_cached_duration(&cache_entries, 0, 10000);
+        assert_eq!(duration, 0);
+    }
+
+    #[test]
+    fn test_calculate_cached_duration_fully_cached() {
+        let cache_entries = vec![
+            CacheEntry {
+                file_path: "test1.arrow".to_string(),
+                start_time: 0,
+                end_time: 5000,
+                interval: Interval::OneHour,
+            },
+            CacheEntry {
+                file_path: "test2.arrow".to_string(),
+                start_time: 5000,
+                end_time: 10000,
+                interval: Interval::OneHour,
+            },
+        ];
+
+        let duration = calculate_cached_duration(&cache_entries, 0, 10000);
+        assert_eq!(duration, 10000);
+    }
+
+    #[test]
+    fn test_calculate_cached_duration_partial() {
+        let cache_entries = vec![
+            CacheEntry {
+                file_path: "test1.arrow".to_string(),
+                start_time: 1000,
+                end_time: 4000,
+                interval: Interval::OneHour,
+            },
+            CacheEntry {
+                file_path: "test2.arrow".to_string(),
+                start_time: 6000,
+                end_time: 9000,
+                interval: Interval::OneHour,
+            },
+        ];
+
+        let duration = calculate_cached_duration(&cache_entries, 0, 10000);
+        assert_eq!(duration, 6000); // 3000 + 3000
+    }
+
+    #[test]
+    fn test_calculate_cached_duration_overlapping() {
+        // This tests the case where cache entries might overlap (shouldn't double count)
+        let cache_entries = vec![
+            CacheEntry {
+                file_path: "test1.arrow".to_string(),
+                start_time: 0,
+                end_time: 6000,
+                interval: Interval::OneHour,
+            },
+            CacheEntry {
+                file_path: "test2.arrow".to_string(),
+                start_time: 4000,
+                end_time: 10000,
+                interval: Interval::OneHour,
+            },
+        ];
+
+        let duration = calculate_cached_duration(&cache_entries, 0, 10000);
+        assert_eq!(duration, 10000); // Should cover 0-10000 without double counting
+    }
+
+    #[test]
+    fn test_cache_discovery_result_states() {
+        // Fully cached
+        let result = CacheDiscoveryResult::new(vec![], vec![], 1.0);
+        assert!(result.is_fully_cached());
+        assert!(!result.has_no_cache());
+
+        // Partial cache
+        let result = CacheDiscoveryResult::new(vec![], vec![], 0.5);
+        assert!(!result.is_fully_cached());
+        assert!(!result.has_no_cache());
+
+        // No cache
+        let result = CacheDiscoveryResult::new(vec![], vec![], 0.0);
+        assert!(!result.is_fully_cached());
+        assert!(result.has_no_cache());
+    }
+}

src/search/src/cache/streaming_agg/files.rs

@@ -0,0 +1,657 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{
+    io::Cursor,
+    path::Path,
+    sync::{Arc, LazyLock},
+};
+
+use arrow::{
+    array::{ArrayRef, RecordBatch, RecordBatchOptions},
+    compute::cast,
+    datatypes::{DataType, Field, SchemaRef},
+    ipc::{reader::FileReader as ArrowFileReader, writer::FileWriter as ArrowFileWriter},
+};
+use config::{
+    meta::search::SearchPartitionRequest,
+    utils::{
+        record_batch_ext::RecordBatchExt,
+        time::{now_micros, second_micros},
+    },
+};
+use hashbrown::HashMap;
+use infra::cache::file_data::disk;
+use tokio::sync::mpsc;
+
+use crate::datafusion::aggregates::gc_string_view_batch;
+
+pub const STREAMING_AGGS_CACHE_DIR: &str = "aggregations";
+
+#[derive(Debug)]
+pub struct RecordBatchCacheRequest {
+    pub streaming_id: String,
+    pub file_path: String,
+    pub schema: SchemaRef,
+    pub records: Vec<Arc<RecordBatch>>,
+    pub overwrite_cache: bool,
+}
+
+// Global queue for cache requests
+static CACHE_QUEUE: LazyLock<mpsc::UnboundedSender<(String, String)>> = LazyLock::new(|| {
+    let (sender, mut receiver) = mpsc::unbounded_channel::<(String, String)>();
+
+    // Spawn background task to process cache requests
+    tokio::spawn(async move {
+        while let Some((streaming_id, file_key)) = receiver.recv().await {
+            log::debug!("[streaming_id: {streaming_id}] Received cache request");
+            if let Err(e) = load_record_batches_file_to_disk_cache(&streaming_id, &file_key).await {
+                log::error!(
+                    "[streaming_id: {streaming_id}] Failed to load record batches file to disk cache: {e:?}"
+                );
+            }
+        }
+    });
+
+    sender
+});
+
+async fn load_record_batches_file_to_disk_cache(
+    streaming_id: &str,
+    file_key: &str,
+) -> Result<(), std::io::Error> {
+    // Skip caching if record batch for the time range is already cached
+    let Some(file_path) = disk::get_file_path(file_key) else {
+        return Ok(()); // no need to cache, it's not a valid file path
+    };
+    let Some(file_meta) = config::utils::file::get_file_meta(&file_path).ok() else {
+        return Ok(()); // no need to cache, it's not a valid file path
+    };
+    let file_size = file_meta.len();
+    if file_size == 0 {
+        return Ok(()); // no need to cache, it's not a valid file
+    }
+
+    log::debug!(
+        "load_record_batches_file_to_disk_cache: streaming_id: {streaming_id}, file_key: {file_key}, file_size: {file_size}"
+    );
+
+    // set to disk cache
+    disk::set_size(file_key, file_size as usize)
+        .await
+        .map_err(|e| std::io::Error::other(format!("Failed to set size to disk cache: {e}")))?;
+
+    Ok(())
+}
+
+// Main handler to write record batches to disk
+pub fn cache_record_batches_to_disk(
+    request: RecordBatchCacheRequest,
+) -> Result<(), std::io::Error> {
+    let start = std::time::Instant::now();
+    let RecordBatchCacheRequest {
+        streaming_id,
+        file_path,
+        schema,
+        records,
+        overwrite_cache,
+    } = request;
+
+    // Skip caching if record batch for the time range is already cached
+    let file_key = file_path.clone();
+    let Some(file_path) = disk::get_file_path(&file_path) else {
+        return Err(std::io::Error::other(
+            "ZO_DISK_CACHE_ENABLED is not enabled",
+        ));
+    };
+    let file_meta = config::utils::file::get_file_meta(&file_path).ok();
+    let file_exists = file_meta.is_some() && file_meta.unwrap().is_file();
+    if file_exists && !overwrite_cache {
+        log::warn!(
+            "[streaming_id: {streaming_id}] file_exists: {file_exists}, Skipping cache to disk because the data for the time range is already cached",
+        );
+        return Ok(());
+    }
+
+    if file_exists && overwrite_cache {
+        log::info!(
+            "[streaming_id: {streaming_id}] file_exists: {file_exists}, overwrite_cache: {overwrite_cache}, Overwriting existing cache file",
+        );
+    }
+
+    let batches_num = records.len();
+    let rows_num = records.iter().map(|r| r.num_rows()).sum::<usize>();
+    let batches = records
+        .iter()
+        .map(|r| Arc::new(gc_string_view_batch(r)))
+        .collect::<Vec<Arc<RecordBatch>>>();
+
+    // Serialize the record batches into bytes
+    let data = match serialize_record_batches(schema, batches) {
+        Ok(data) => data,
+        Err(e) => {
+            log::error!("[streaming_id: {streaming_id}] Failed to serialize record batches: {e:?}",);
+            return Err(std::io::Error::other("Serialization failed"));
+        }
+    };
+
+    // create the directory if it doesn't exist
+    std::fs::create_dir_all(Path::new(&file_path).parent().unwrap())?;
+    // write the data to the file
+    match config::utils::file::put_file_contents(&file_path, &data) {
+        Ok(_) => {
+            log::info!(
+                "cache_record_batches_to_disk: streaming_id: {streaming_id}, file_path: {file_path}, batches: {batches_num}, rows: {rows_num}, write to file took: {} ms",
+                start.elapsed().as_millis()
+            );
+
+            // add to cache list
+            // Send to background queue (non-blocking)
+            if let Err(e) = CACHE_QUEUE.send((streaming_id.clone(), file_key)) {
+                log::error!(
+                    "[streaming_id: {streaming_id}] Failed to queue cache file to disk: {file_path}, error: {e:?}",
+                );
+            }
+
+            Ok(())
+        }
+        Err(e) => {
+            log::error!("Error caching results to disk: {e:?}");
+            Err(std::io::Error::other(format!(
+                "[streaming_id: {streaming_id}] Error caching results to disk: file_path={file_path}"
+            )))
+        }
+    }
+}
+
+// write to arrow ipc format
+fn serialize_record_batches(
+    schema: SchemaRef,
+    batches: Vec<Arc<RecordBatch>>,
+) -> arrow::error::Result<Vec<u8>> {
+    let mut buffer = Cursor::new(Vec::new());
+    let mut writer = ArrowFileWriter::try_new(&mut buffer, &schema)?;
+
+    for batch in batches {
+        writer.write(&batch)?;
+    }
+
+    writer.finish()?;
+    Ok(buffer.into_inner())
+}
+
+pub fn get_record_batches(
+    streaming_id: &str,
+    file_path: &str,
+    schema: SchemaRef,
+) -> std::io::Result<Vec<RecordBatch>> {
+    let start = std::time::Instant::now();
+    let file_path = disk::get_file_path(file_path).ok_or_else(|| {
+        std::io::Error::new(
+            std::io::ErrorKind::NotFound,
+            format!("File not found: {file_path}"),
+        )
+    })?;
+    let data = config::utils::file::get_file_contents(&file_path, None).map_err(|e| {
+        log::error!("Error getting file contents: {e}, file_path: {file_path}");
+        e
+    })?;
+    let reader = unsafe {
+        ArrowFileReader::try_new(Cursor::new(data), None)
+            .map_err(|e| {
+                log::error!("Error creating arrow reader: {e}, file_path: {file_path}");
+                std::io::Error::other(format!("Arrow error: {e}"))
+            })?
+            .with_skip_validation(true)
+    };
+    let schema_field_map = schema
+        .fields()
+        .iter()
+        .map(|f| (f.name(), f.data_type()))
+        .collect::<HashMap<&String, &DataType>>();
+    let mut batches = Vec::new();
+    for batch in reader {
+        let batch = batch.map_err(|e| std::io::Error::other(format!("Arrow error: {e}")))?;
+        let new_columns: Vec<ArrayRef> = batch
+            .columns()
+            .iter()
+            .zip(batch.schema().fields().iter())
+            .map(|(c, f)| {
+                let file_datatype = f.data_type();
+                let need_datatype = *schema_field_map.get(f.name()).unwrap_or(&&DataType::Null);
+
+                // Use the recursive cast function
+                if let Some(casted) = cast_array_recursive(c, file_datatype, need_datatype) {
+                    casted
+                } else {
+                    Arc::clone(c)
+                }
+            })
+            .collect();
+        let mut options = RecordBatchOptions::new();
+        options = options.with_row_count(Some(batch.num_rows()));
+        let batch = RecordBatch::try_new_with_options(schema.clone(), new_columns, &options)
+            .expect("Failed to re-create the record batch");
+
+        batches.push(batch);
+    }
+
+    log::debug!(
+        "get_record_batches: streaming_id: {streaming_id}, file_path: {file_path}, batches: {}, rows: {}, arrow_size: {}, took: {} ms",
+        batches.len(),
+        batches.iter().map(|r| r.num_rows()).sum::<usize>(),
+        batches.iter().map(|r| r.size()).sum::<usize>(),
+        start.elapsed().as_millis()
+    );
+
+    Ok(batches)
+}
+
+/// Recursively checks if two data types need casting and performs the cast if needed.
+/// This function handles:
+/// - String type conversions (Utf8, LargeUtf8, Utf8View)
+/// - Nested List types (recursively checks inner types)
+/// - Nested Struct types (recursively checks field types)
+/// - Other types (returns None if no cast needed)
+///
+/// # Arguments
+/// * `array` - The array to potentially cast
+/// * `from_type` - The source data type
+/// * `to_type` - The target data type
+///
+/// # Returns
+/// * `Some(ArrayRef)` - If casting was needed and successful
+/// * `None` - If no casting is needed (types are compatible)
+fn cast_array_recursive(
+    array: &ArrayRef,
+    from_type: &DataType,
+    to_type: &DataType,
+) -> Option<ArrayRef> {
+    // If types are identical, no cast needed
+    if from_type == to_type {
+        return None;
+    }
+
+    match (from_type, to_type) {
+        // Handle string type conversions
+        (
+            DataType::Utf8 | DataType::LargeUtf8 | DataType::Utf8View,
+            DataType::Utf8 | DataType::LargeUtf8 | DataType::Utf8View,
+        ) => {
+            // Cast between string types
+            cast(array, to_type).ok()
+        }
+
+        // Handle List types recursively
+        (DataType::List(from_field), DataType::List(to_field)) => {
+            let from_inner = from_field.data_type();
+            let to_inner = to_field.data_type();
+
+            // If inner types need casting, create a new List type with the target inner type
+            if needs_recursive_cast(from_inner, to_inner) {
+                // Create a new field with the target data type
+                let new_field = Arc::new(Field::new(
+                    to_field.name(),
+                    to_inner.clone(),
+                    to_field.is_nullable(),
+                ));
+                let new_list_type = DataType::List(new_field);
+                cast(array, &new_list_type).ok()
+            } else {
+                None
+            }
+        }
+
+        // Handle LargeList types recursively
+        (DataType::LargeList(from_field), DataType::LargeList(to_field)) => {
+            let from_inner = from_field.data_type();
+            let to_inner = to_field.data_type();
+
+            if needs_recursive_cast(from_inner, to_inner) {
+                let new_field = Arc::new(Field::new(
+                    to_field.name(),
+                    to_inner.clone(),
+                    to_field.is_nullable(),
+                ));
+                let new_list_type = DataType::LargeList(new_field);
+                cast(array, &new_list_type).ok()
+            } else {
+                None
+            }
+        }
+
+        // Handle Struct types recursively
+        (DataType::Struct(from_fields), DataType::Struct(to_fields)) => {
+            // Check if any field needs casting
+            let needs_cast =
+                from_fields
+                    .iter()
+                    .zip(to_fields.iter())
+                    .any(|(from_field, to_field)| {
+                        needs_recursive_cast(from_field.data_type(), to_field.data_type())
+                    });
+
+            if needs_cast {
+                cast(array, to_type).ok()
+            } else {
+                None
+            }
+        }
+
+        // For all other type combinations, no cast is performed
+        _ => None,
+    }
+}
+
+/// Helper function to check if two types need recursive casting
+fn needs_recursive_cast(from_type: &DataType, to_type: &DataType) -> bool {
+    if from_type == to_type {
+        return false;
+    }
+
+    match (from_type, to_type) {
+        // String types can be cast between each other
+        (
+            DataType::Utf8 | DataType::LargeUtf8 | DataType::Utf8View,
+            DataType::Utf8 | DataType::LargeUtf8 | DataType::Utf8View,
+        ) => true,
+
+        // Recursively check List types
+        (DataType::List(from_field), DataType::List(to_field)) => {
+            needs_recursive_cast(from_field.data_type(), to_field.data_type())
+        }
+
+        // Recursively check LargeList types
+        (DataType::LargeList(from_field), DataType::LargeList(to_field)) => {
+            needs_recursive_cast(from_field.data_type(), to_field.data_type())
+        }
+
+        // Recursively check Struct types
+        (DataType::Struct(from_fields), DataType::Struct(to_fields)) => {
+            from_fields.len() == to_fields.len()
+                && from_fields
+                    .iter()
+                    .zip(to_fields.iter())
+                    .any(|(from_field, to_field)| {
+                        needs_recursive_cast(from_field.data_type(), to_field.data_type())
+                    })
+        }
+
+        _ => false,
+    }
+}
+
+pub fn create_aggregation_cache_file_path(
+    org_id: &str,
+    stream_type: &str,
+    stream_name: &str,
+    hashed_query: u64,
+) -> String {
+    if org_id.is_empty() || stream_type.is_empty() || stream_name.is_empty() {
+        return "".to_string();
+    }
+    // eg: /org_id/stream_type/stream_name/12345678
+    // Note: interval is NOT included in the path anymore - all intervals share the same directory
+    format!("{org_id}/{stream_type}/{stream_name}/{hashed_query}")
+}
+
+pub fn generate_aggregation_cache_file_name(
+    id: &str,
+    start_time: i64,
+    end_time: i64,
+    is_complete_partition_window: bool,
+) -> String {
+    // set cache as tmp if the time range is within the delay window
+    let delay_window_micros = second_micros(config::get_config().limit.cache_delay_secs);
+    let skip_cache = now_micros() - delay_window_micros;
+    let can_be_cached = end_time < skip_cache;
+
+    let is_tmp_file = if is_complete_partition_window && can_be_cached {
+        "".to_string()
+    } else {
+        format!("_{id}_tmp")
+    };
+    format!("{start_time}_{end_time}{is_tmp_file}.arrow")
+}
+
+pub fn get_cache_file_path(file_path: &str, file_name: &str) -> String {
+    format!("{STREAMING_AGGS_CACHE_DIR}/{file_path}/{file_name}")
+}
+
+pub fn get_aggregation_cache_key_from_request(req: &SearchPartitionRequest) -> u64 {
+    let origin_sql = req.sql.clone();
+
+    let mut hash_body = vec![origin_sql];
+    if let Some(vrl_function) = &req.query_fn {
+        hash_body.push(vrl_function.to_string());
+    }
+    if !req.regions.is_empty() {
+        hash_body.extend(req.regions.clone());
+    }
+    if !req.clusters.is_empty() {
+        hash_body.extend(req.clusters.clone());
+    }
+    config::utils::hash::sum64(&hash_body.join(","))
+}
+
+#[cfg(test)]
+mod tests {
+    use arrow::array::StringArray;
+
+    use super::*;
+
+    #[test]
+    fn test_cast_array_recursive_string_types() {
+        use arrow::array::StringArray;
+
+        // Test Utf8 to Utf8View
+        let array: ArrayRef = Arc::new(StringArray::from(vec!["hello", "world"]));
+        let result = cast_array_recursive(&array, &DataType::Utf8, &DataType::Utf8View);
+        assert!(result.is_some());
+        let casted = result.unwrap();
+        assert_eq!(casted.data_type(), &DataType::Utf8View);
+
+        // Test Utf8View to LargeUtf8
+        let array: ArrayRef = Arc::new(StringArray::from(vec!["foo", "bar"]));
+        let result = cast_array_recursive(&array, &DataType::Utf8View, &DataType::LargeUtf8);
+        assert!(result.is_some());
+
+        // Test LargeUtf8 to Utf8
+        let array: ArrayRef = Arc::new(StringArray::from(vec!["test"]));
+        let result = cast_array_recursive(&array, &DataType::LargeUtf8, &DataType::Utf8);
+        assert!(result.is_some());
+    }
+
+    #[test]
+    fn test_cast_array_recursive_identical_types() {
+        use arrow::array::Int32Array;
+
+        // Test that identical types return None (no cast needed)
+        let array: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3]));
+        let result = cast_array_recursive(&array, &DataType::Int32, &DataType::Int32);
+        assert!(result.is_none());
+
+        // Test string types
+        let array: ArrayRef = Arc::new(StringArray::from(vec!["test"]));
+        let result = cast_array_recursive(&array, &DataType::Utf8, &DataType::Utf8);
+        assert!(result.is_none());
+    }
+
+    #[test]
+    fn test_cast_array_recursive_list_with_string() {
+        use arrow::array::{ListArray, StringArray};
+
+        // Create a List<Utf8> array
+        let values = StringArray::from(vec!["a", "b", "c", "d"]);
+        let offsets = arrow::buffer::OffsetBuffer::new(vec![0, 2, 4].into());
+        let field = Arc::new(Field::new("item", DataType::Utf8, true));
+        let list_array = ListArray::new(field.clone(), offsets, Arc::new(values), None);
+        let array: ArrayRef = Arc::new(list_array);
+
+        // Cast from List<Utf8> to List<Utf8View>
+        let from_type = DataType::List(Arc::new(Field::new("item", DataType::Utf8, true)));
+        let to_type = DataType::List(Arc::new(Field::new("item", DataType::Utf8View, true)));
+
+        let result = cast_array_recursive(&array, &from_type, &to_type);
+        assert!(result.is_some());
+
+        let casted = result.unwrap();
+        if let DataType::List(inner_field) = casted.data_type() {
+            assert_eq!(inner_field.data_type(), &DataType::Utf8View);
+        } else {
+            panic!("Expected List type");
+        }
+    }
+
+    #[test]
+    fn test_cast_array_recursive_nested_list() {
+        use arrow::array::{ListArray, StringArray};
+
+        // Create a List<List<Utf8>> structure
+        let inner_values = StringArray::from(vec!["x", "y"]);
+        let inner_offsets = arrow::buffer::OffsetBuffer::new(vec![0, 1, 2].into());
+        let inner_field = Arc::new(Field::new("item", DataType::Utf8, true));
+        let inner_list = ListArray::new(
+            inner_field.clone(),
+            inner_offsets,
+            Arc::new(inner_values),
+            None,
+        );
+
+        let outer_offsets = arrow::buffer::OffsetBuffer::new(vec![0, 2].into());
+        let outer_field = Arc::new(Field::new(
+            "item",
+            DataType::List(inner_field.clone()),
+            true,
+        ));
+        let outer_list = ListArray::new(outer_field, outer_offsets, Arc::new(inner_list), None);
+        let array: ArrayRef = Arc::new(outer_list);
+
+        // Cast from List<List<Utf8>> to List<List<LargeUtf8>>
+        let from_type = DataType::List(Arc::new(Field::new(
+            "item",
+            DataType::List(Arc::new(Field::new("item", DataType::Utf8, true))),
+            true,
+        )));
+        let to_type = DataType::List(Arc::new(Field::new(
+            "item",
+            DataType::List(Arc::new(Field::new("item", DataType::LargeUtf8, true))),
+            true,
+        )));
+
+        let result = cast_array_recursive(&array, &from_type, &to_type);
+        assert!(result.is_some());
+    }
+
+    #[test]
+    fn test_cast_array_recursive_struct_with_string() {
+        use arrow::{
+            array::{Int32Array, StringArray, StructArray},
+            datatypes::Fields,
+        };
+
+        // Create a Struct with (name: Utf8, age: Int32)
+        let name_array = Arc::new(StringArray::from(vec!["Alice", "Bob"]));
+        let age_array = Arc::new(Int32Array::from(vec![30, 25]));
+
+        let from_fields = Fields::from(vec![
+            Field::new("name", DataType::Utf8, false),
+            Field::new("age", DataType::Int32, false),
+        ]);
+        let struct_array = StructArray::new(
+            from_fields.clone(),
+            vec![name_array as ArrayRef, age_array as ArrayRef],
+            None,
+        );
+        let array: ArrayRef = Arc::new(struct_array);
+
+        // Cast to Struct with (name: Utf8View, age: Int32)
+        let from_type = DataType::Struct(from_fields);
+        let to_fields = Fields::from(vec![
+            Field::new("name", DataType::Utf8View, false),
+            Field::new("age", DataType::Int32, false),
+        ]);
+        let to_type = DataType::Struct(to_fields.clone());
+
+        let result = cast_array_recursive(&array, &from_type, &to_type);
+        assert!(result.is_some());
+
+        let casted = result.unwrap();
+        if let DataType::Struct(fields) = casted.data_type() {
+            assert_eq!(fields[0].data_type(), &DataType::Utf8View);
+            assert_eq!(fields[1].data_type(), &DataType::Int32);
+        } else {
+            panic!("Expected Struct type");
+        }
+    }
+
+    #[test]
+    fn test_needs_recursive_cast_string_types() {
+        // String type variations should return true
+        assert!(needs_recursive_cast(&DataType::Utf8, &DataType::Utf8View));
+        assert!(needs_recursive_cast(&DataType::LargeUtf8, &DataType::Utf8));
+        assert!(needs_recursive_cast(
+            &DataType::Utf8View,
+            &DataType::LargeUtf8
+        ));
+
+        // Same string type should return false
+        assert!(!needs_recursive_cast(&DataType::Utf8, &DataType::Utf8));
+    }
+
+    #[test]
+    fn test_needs_recursive_cast_list_types() {
+        // List with different inner types
+        let from_list = DataType::List(Arc::new(Field::new("item", DataType::Utf8, true)));
+        let to_list = DataType::List(Arc::new(Field::new("item", DataType::Utf8View, true)));
+        assert!(needs_recursive_cast(&from_list, &to_list));
+
+        // List with same inner types
+        let same_list = DataType::List(Arc::new(Field::new("item", DataType::Int32, true)));
+        assert!(!needs_recursive_cast(&same_list, &same_list));
+    }
+
+    #[test]
+    fn test_needs_recursive_cast_struct_types() {
+        use arrow::datatypes::Fields;
+
+        // Struct with different field types
+        let from_fields = Fields::from(vec![Field::new("name", DataType::Utf8, false)]);
+        let to_fields = Fields::from(vec![Field::new("name", DataType::Utf8View, false)]);
+        let from_struct = DataType::Struct(from_fields);
+        let to_struct = DataType::Struct(to_fields);
+        assert!(needs_recursive_cast(&from_struct, &to_struct));
+
+        // Struct with same field types
+        let same_fields = Fields::from(vec![Field::new("id", DataType::Int32, false)]);
+        let same_struct = DataType::Struct(same_fields);
+        assert!(!needs_recursive_cast(&same_struct, &same_struct));
+    }
+
+    #[test]
+    fn test_cast_array_recursive_incompatible_types() {
+        use arrow::array::{Int32Array, StringArray};
+
+        // Test that incompatible types return None
+        let array: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3]));
+        let result = cast_array_recursive(&array, &DataType::Int32, &DataType::Float64);
+        assert!(result.is_none());
+
+        // Test string to int (should return None as we don't handle this)
+        let array: ArrayRef = Arc::new(StringArray::from(vec!["test"]));
+        let result = cast_array_recursive(&array, &DataType::Utf8, &DataType::Int32);
+        assert!(result.is_none());
+    }
+}

src/search/src/cache/streaming_agg/mod.rs

@@ -0,0 +1,27 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Streaming aggregation cache implementation
+//!
+//! This module provides cache-aware partition generation for streaming aggregations,
+//! including cache discovery, loading, and partition optimization.
+mod discovery;
+mod files;
+mod partition_optimizer;
+
+// Re-export public APIs
+pub use discovery::*;
+pub use files::*;
+pub use partition_optimizer::*;

src/search/src/cache/streaming_agg/partition_optimizer.rs

@@ -0,0 +1,615 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use config::meta::{
+    search::{CardinalityLevel, Interval, generate_aggregation_search_interval},
+    sql::OrderBy,
+};
+
+use super::discovery::{CacheDiscoveryResult, CacheEntry, TimeRange};
+
+/// Strategy for handling query partitions based on cache availability
+#[derive(Debug, Clone)]
+pub enum StreamingAggsPartitionStrategy {
+    /// All data is available in cache - no query execution needed
+    FullyCached { cache_files: Vec<CacheEntry> },
+    /// Mix of cached and uncached data
+    Hybrid {
+        cached_partitions: Vec<CachedPartition>,
+        uncached_partitions: Vec<UncachedPartition>,
+    },
+    /// No cache available - execute query normally
+    NoCacheAvailable { partitions: Vec<UncachedPartition> },
+}
+
+impl StreamingAggsPartitionStrategy {
+    /// Strategy Name
+    pub fn strategy_name(&self) -> &str {
+        match self {
+            StreamingAggsPartitionStrategy::FullyCached { .. } => "FullyCached",
+            StreamingAggsPartitionStrategy::Hybrid { .. } => "Hybrid",
+            StreamingAggsPartitionStrategy::NoCacheAvailable { .. } => "NoCacheAvailable",
+        }
+    }
+
+    /// Returns true if this strategy requires query execution
+    pub fn requires_execution(&self) -> bool {
+        match self {
+            StreamingAggsPartitionStrategy::FullyCached { .. } => false,
+            StreamingAggsPartitionStrategy::Hybrid { .. }
+            | StreamingAggsPartitionStrategy::NoCacheAvailable { .. } => true,
+        }
+    }
+
+    /// Returns the number of partitions that need to be executed
+    pub fn execution_partition_count(&self) -> usize {
+        match self {
+            StreamingAggsPartitionStrategy::FullyCached { .. } => 0,
+            StreamingAggsPartitionStrategy::Hybrid {
+                uncached_partitions,
+                ..
+            } => uncached_partitions.len(),
+            StreamingAggsPartitionStrategy::NoCacheAvailable { partitions } => partitions.len(),
+        }
+    }
+
+    /// Converts the partition strategy into time range partitions [start, end]
+    /// This is the format expected by the rest of the search system
+    ///
+    /// # Arguments
+    /// * `order_by` - The sort order (Asc or Desc) for partitions
+    ///
+    /// # Returns
+    /// Vector of [start_time, end_time] pairs in the requested order
+    pub fn to_time_partitions(&self, order_by: OrderBy) -> Vec<[i64; 2]> {
+        let mut partitions = Vec::new();
+
+        match self {
+            StreamingAggsPartitionStrategy::FullyCached { cache_files } => {
+                // For fully cached queries, return a SINGLE partition covering the entire range
+                // This avoids unnecessary iteration over individual cache files
+                if !cache_files.is_empty() {
+                    let min_start = cache_files.iter().map(|f| f.start_time).min().unwrap();
+                    let max_end = cache_files.iter().map(|f| f.end_time).max().unwrap();
+                    partitions.push([min_start, max_end]);
+                }
+            }
+            StreamingAggsPartitionStrategy::Hybrid {
+                cached_partitions,
+                uncached_partitions,
+            } => {
+                // Add cached partitions
+                for cached in cached_partitions {
+                    partitions.push([cached.start_time, cached.end_time]);
+                }
+
+                // Add uncached partitions
+                for uncached in uncached_partitions {
+                    partitions.push([uncached.start_time, uncached.end_time]);
+                }
+
+                // Sort by start time to ensure chronological order
+                partitions.sort_by_key(|p| p[0]);
+            }
+            StreamingAggsPartitionStrategy::NoCacheAvailable { partitions: p } => {
+                for partition in p {
+                    partitions.push([partition.start_time, partition.end_time]);
+                }
+            }
+        }
+
+        // Apply ordering
+        if order_by == OrderBy::Desc {
+            partitions.reverse();
+        }
+
+        partitions
+    }
+}
+
+/// Represents a partition that is fully covered by cache
+#[derive(Debug, Clone)]
+pub struct CachedPartition {
+    pub cache_files: Vec<CacheEntry>,
+    pub start_time: i64,
+    pub end_time: i64,
+    pub interval: Interval,
+}
+
+impl CachedPartition {
+    pub fn new(
+        cache_files: Vec<CacheEntry>,
+        start_time: i64,
+        end_time: i64,
+        interval: Interval,
+    ) -> Self {
+        Self {
+            cache_files,
+            start_time,
+            end_time,
+            interval,
+        }
+    }
+}
+
+/// Represents a partition that needs to be executed (not in cache)
+#[derive(Debug, Clone)]
+pub struct UncachedPartition {
+    pub start_time: i64,
+    pub end_time: i64,
+}
+
+impl UncachedPartition {
+    pub fn new(start_time: i64, end_time: i64) -> Self {
+        Self {
+            start_time,
+            end_time,
+        }
+    }
+}
+
+/// Generates optimal partition strategy based on cache discovery results
+///
+/// # Arguments
+/// * `discovery_result` - Result from cache discovery
+/// * `query_start` - Query start time in microseconds
+/// * `query_end` - Query end time in microseconds
+/// * `cardinality_level` - Cardinality level for determining cache intervals
+///
+/// # Returns
+/// * `PartitionStrategy` - Optimal strategy for executing the query
+pub fn generate_optimal_partitions(
+    discovery_result: CacheDiscoveryResult,
+    query_start: i64,
+    query_end: i64,
+    cardinality_level: CardinalityLevel,
+) -> StreamingAggsPartitionStrategy {
+    // Case 1: Fully cached - no execution needed
+    if discovery_result.is_fully_cached() {
+        return StreamingAggsPartitionStrategy::FullyCached {
+            cache_files: discovery_result.cached_ranges,
+        };
+    }
+
+    // Calculate the target interval for the entire query based on query duration
+    let query_target_interval =
+        generate_aggregation_search_interval(query_start, query_end, cardinality_level);
+
+    // Case 2: No cache available - use standard ladder logic
+    if discovery_result.has_no_cache() {
+        let partitions =
+            generate_uncached_partitions_from_range(query_start, query_end, query_target_interval);
+        return StreamingAggsPartitionStrategy::NoCacheAvailable { partitions };
+    }
+
+    // Case 3: Hybrid - mix of cached and uncached
+    let cached_partitions = group_cache_entries_into_partitions(discovery_result.cached_ranges);
+    let uncached_partitions =
+        generate_uncached_partitions(discovery_result.uncached_ranges, query_target_interval);
+
+    StreamingAggsPartitionStrategy::Hybrid {
+        cached_partitions,
+        uncached_partitions,
+    }
+}
+
+/// Groups consecutive cache entries with the same interval into cached partitions
+///
+/// # Preconditions
+/// - `cache_entries` must be sorted by `start_time` (ascending)
+/// - This is guaranteed by `discover_cache_for_query()` which sorts entries before returning
+///
+/// # Behavior
+/// - Creates separate partitions when interval changes
+/// - Creates separate partitions when there's a time gap between entries
+/// - Assumes entries are non-overlapping (guaranteed by cache write logic)
+fn group_cache_entries_into_partitions(cache_entries: Vec<CacheEntry>) -> Vec<CachedPartition> {
+    if cache_entries.is_empty() {
+        return vec![];
+    }
+
+    let mut partitions = Vec::new();
+    let mut current_group: Vec<CacheEntry> = vec![];
+    let mut current_interval = cache_entries[0].interval;
+
+    for entry in cache_entries {
+        // If the interval changes or there's a gap, create a new partition
+        if entry.interval != current_interval
+            || (!current_group.is_empty()
+                && entry.start_time > current_group.last().unwrap().end_time)
+        {
+            if !current_group.is_empty() {
+                let start_time = current_group.first().unwrap().start_time;
+                let end_time = current_group.last().unwrap().end_time;
+                partitions.push(CachedPartition::new(
+                    current_group,
+                    start_time,
+                    end_time,
+                    current_interval,
+                ));
+            }
+            current_group = vec![entry.clone()];
+            current_interval = entry.interval;
+        } else {
+            current_group.push(entry);
+        }
+    }
+
+    // Don't forget the last group
+    if !current_group.is_empty() {
+        let start_time = current_group.first().unwrap().start_time;
+        let end_time = current_group.last().unwrap().end_time;
+        partitions.push(CachedPartition::new(
+            current_group,
+            start_time,
+            end_time,
+            current_interval,
+        ));
+    }
+
+    partitions
+}
+
+/// Generates uncached partitions from time ranges using the query's target interval
+fn generate_uncached_partitions(
+    uncached_ranges: Vec<TimeRange>,
+    target_interval: Interval,
+) -> Vec<UncachedPartition> {
+    let mut partitions = Vec::new();
+
+    for range in uncached_ranges {
+        let range_partitions = generate_uncached_partitions_from_range(
+            range.start_time,
+            range.end_time,
+            target_interval,
+        );
+        partitions.extend(range_partitions);
+    }
+
+    partitions
+}
+
+/// Generates uncached partitions for a single time range using the specified target interval
+fn generate_uncached_partitions_from_range(
+    start_time: i64,
+    end_time: i64,
+    target_interval: Interval,
+) -> Vec<UncachedPartition> {
+    // Use the query's target interval (already calculated based on total query duration)
+    // This ensures all uncached partitions use the same interval regardless of gap size
+
+    // If interval is Zero, we don't cache (e.g., for Huge cardinality)
+    if target_interval == Interval::Zero {
+        return vec![UncachedPartition::new(start_time, end_time)];
+    }
+
+    let interval_micros = target_interval.get_interval_microseconds();
+    let mut partitions = Vec::new();
+
+    // Align start time to UTC boundary
+    let aligned_start = align_time_to_interval(start_time, interval_micros, true);
+
+    // If query starts before the first aligned boundary, create a non-aligned partition
+    if start_time < aligned_start {
+        partitions.push(UncachedPartition::new(
+            start_time,
+            aligned_start.min(end_time),
+        ));
+    }
+
+    // Generate UTC-aligned partitions
+    let mut current_time = aligned_start;
+    while current_time < end_time {
+        let partition_end = (current_time + interval_micros).min(end_time);
+
+        partitions.push(UncachedPartition::new(current_time, partition_end));
+
+        current_time = partition_end;
+    }
+
+    partitions
+}
+
+/// Aligns a timestamp to the nearest interval boundary
+///
+/// # Arguments
+/// * `timestamp` - Timestamp in microseconds (assumed to be within reasonable bounds: year
+///   1970-2200)
+/// * `interval_micros` - Interval duration in microseconds (max: 1 day = 86,400,000,000)
+/// * `round_up` - If true, rounds up to next boundary; if false, rounds down
+///
+/// # Safety
+/// This function uses unchecked arithmetic. Integer overflow is not possible with realistic
+/// timestamp values (years 1970-2200) and supported intervals (5min to 1day). The maximum
+/// result is bounded by `timestamp + interval_micros`, which is well within i64::MAX for
+/// any reasonable query timestamp.
+fn align_time_to_interval(timestamp: i64, interval_micros: i64, round_up: bool) -> i64 {
+    let remainder = timestamp % interval_micros;
+    if remainder == 0 {
+        timestamp
+    } else if round_up {
+        timestamp + (interval_micros - remainder)
+    } else {
+        timestamp - remainder
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_partition_strategy_fully_cached() {
+        let cache_files = vec![CacheEntry {
+            file_path: "test.arrow".to_string(),
+            start_time: 0,
+            end_time: 1000,
+            interval: Interval::OneHour,
+        }];
+
+        let strategy = StreamingAggsPartitionStrategy::FullyCached {
+            cache_files: cache_files.clone(),
+        };
+
+        assert!(!strategy.requires_execution());
+        assert_eq!(strategy.execution_partition_count(), 0);
+    }
+
+    #[test]
+    fn test_partition_strategy_no_cache() {
+        let partitions = vec![UncachedPartition::new(0, 1000)];
+
+        let strategy = StreamingAggsPartitionStrategy::NoCacheAvailable {
+            partitions: partitions.clone(),
+        };
+
+        assert!(strategy.requires_execution());
+        assert_eq!(strategy.execution_partition_count(), 1);
+    }
+
+    #[test]
+    fn test_partition_strategy_hybrid() {
+        let cached = vec![CachedPartition::new(vec![], 0, 500, Interval::OneHour)];
+        let uncached = vec![UncachedPartition::new(500, 1000)];
+
+        let strategy = StreamingAggsPartitionStrategy::Hybrid {
+            cached_partitions: cached.clone(),
+            uncached_partitions: uncached.clone(),
+        };
+
+        assert!(strategy.requires_execution());
+        assert_eq!(strategy.execution_partition_count(), 1);
+    }
+
+    #[test]
+    fn test_align_time_to_interval() {
+        let one_hour_micros = 3_600_000_000i64;
+
+        // Already aligned
+        assert_eq!(align_time_to_interval(0, one_hour_micros, true), 0);
+        assert_eq!(align_time_to_interval(0, one_hour_micros, false), 0);
+        assert_eq!(
+            align_time_to_interval(one_hour_micros, one_hour_micros, true),
+            one_hour_micros
+        );
+
+        // Not aligned - round up
+        assert_eq!(
+            align_time_to_interval(100, one_hour_micros, true),
+            one_hour_micros
+        );
+        assert_eq!(
+            align_time_to_interval(one_hour_micros + 100, one_hour_micros, true),
+            one_hour_micros * 2
+        );
+
+        // Not aligned - round down
+        assert_eq!(align_time_to_interval(100, one_hour_micros, false), 0);
+        assert_eq!(
+            align_time_to_interval(one_hour_micros + 100, one_hour_micros, false),
+            one_hour_micros
+        );
+    }
+
+    #[test]
+    fn test_generate_uncached_partitions_from_range_zero_interval() {
+        // For Zero interval, we don't partition
+        let partitions = generate_uncached_partitions_from_range(0, 10000, Interval::Zero);
+
+        assert_eq!(partitions.len(), 1);
+        assert_eq!(partitions[0].start_time, 0);
+        assert_eq!(partitions[0].end_time, 10000);
+    }
+
+    #[test]
+    fn test_generate_uncached_partitions_from_range_aligned() {
+        let five_min_micros = 300_000_000i64;
+
+        // Query range that's perfectly aligned: 0 to 15 minutes (3 x 5-minute intervals)
+        let partitions =
+            generate_uncached_partitions_from_range(0, five_min_micros * 3, Interval::FiveMinutes);
+
+        assert_eq!(partitions.len(), 3);
+
+        for (i, partition) in partitions.iter().enumerate() {
+            assert_eq!(partition.start_time, five_min_micros * i as i64);
+            assert_eq!(partition.end_time, five_min_micros * (i as i64 + 1));
+        }
+    }
+
+    #[test]
+    fn test_generate_uncached_partitions_from_range_unaligned() {
+        let five_min_micros = 300_000_000i64;
+
+        // Query range that starts at a non-aligned time
+        let start = 100_000; // 100ms offset
+        let end = five_min_micros * 2 + 100_000; // 10min + 100ms
+
+        let partitions = generate_uncached_partitions_from_range(start, end, Interval::FiveMinutes);
+
+        // Should create: [100ms -> 5min], [5min -> 10min], [10min -> 10min+100ms]
+        assert_eq!(partitions.len(), 3);
+
+        // First partition: non-aligned start
+        assert_eq!(partitions[0].start_time, start);
+        assert_eq!(partitions[0].end_time, five_min_micros);
+
+        // Middle partition: fully aligned
+        assert_eq!(partitions[1].start_time, five_min_micros);
+        assert_eq!(partitions[1].end_time, five_min_micros * 2);
+
+        // Last partition: non-aligned end
+        assert_eq!(partitions[2].start_time, five_min_micros * 2);
+        assert_eq!(partitions[2].end_time, end);
+    }
+
+    #[test]
+    fn test_group_cache_entries_same_interval() {
+        let entries = vec![
+            CacheEntry {
+                file_path: "test1.arrow".to_string(),
+                start_time: 0,
+                end_time: 1000,
+                interval: Interval::OneHour,
+            },
+            CacheEntry {
+                file_path: "test2.arrow".to_string(),
+                start_time: 1000,
+                end_time: 2000,
+                interval: Interval::OneHour,
+            },
+        ];
+
+        let partitions = group_cache_entries_into_partitions(entries);
+
+        assert_eq!(partitions.len(), 1);
+        assert_eq!(partitions[0].start_time, 0);
+        assert_eq!(partitions[0].end_time, 2000);
+        assert_eq!(partitions[0].cache_files.len(), 2);
+        assert_eq!(partitions[0].interval, Interval::OneHour);
+    }
+
+    #[test]
+    fn test_group_cache_entries_different_intervals() {
+        let entries = vec![
+            CacheEntry {
+                file_path: "test1.arrow".to_string(),
+                start_time: 0,
+                end_time: 1000,
+                interval: Interval::OneHour,
+            },
+            CacheEntry {
+                file_path: "test2.arrow".to_string(),
+                start_time: 1000,
+                end_time: 2000,
+                interval: Interval::FiveMinutes,
+            },
+        ];
+
+        let partitions = group_cache_entries_into_partitions(entries);
+
+        assert_eq!(partitions.len(), 2);
+        assert_eq!(partitions[0].interval, Interval::OneHour);
+        assert_eq!(partitions[1].interval, Interval::FiveMinutes);
+    }
+
+    #[test]
+    fn test_group_cache_entries_with_gap() {
+        let entries = vec![
+            CacheEntry {
+                file_path: "test1.arrow".to_string(),
+                start_time: 0,
+                end_time: 1000,
+                interval: Interval::OneHour,
+            },
+            CacheEntry {
+                file_path: "test2.arrow".to_string(),
+                start_time: 2000, // Gap between 1000 and 2000
+                end_time: 3000,
+                interval: Interval::OneHour,
+            },
+        ];
+
+        let partitions = group_cache_entries_into_partitions(entries);
+
+        assert_eq!(partitions.len(), 2);
+        assert_eq!(partitions[0].cache_files.len(), 1);
+        assert_eq!(partitions[1].cache_files.len(), 1);
+    }
+
+    #[test]
+    fn test_generate_optimal_partitions_fully_cached() {
+        let discovery = CacheDiscoveryResult::new(
+            vec![CacheEntry {
+                file_path: "test.arrow".to_string(),
+                start_time: 0,
+                end_time: 10000,
+                interval: Interval::OneHour,
+            }],
+            vec![],
+            1.0,
+        );
+
+        let strategy = generate_optimal_partitions(discovery, 0, 10000, CardinalityLevel::Low);
+
+        match strategy {
+            StreamingAggsPartitionStrategy::FullyCached { cache_files } => {
+                assert_eq!(cache_files.len(), 1);
+            }
+            _ => panic!("Expected FullyCached strategy"),
+        }
+    }
+
+    #[test]
+    fn test_generate_optimal_partitions_no_cache() {
+        let discovery = CacheDiscoveryResult::new(vec![], vec![TimeRange::new(0, 10000)], 0.0);
+
+        let strategy = generate_optimal_partitions(discovery, 0, 10000, CardinalityLevel::Low);
+
+        match strategy {
+            StreamingAggsPartitionStrategy::NoCacheAvailable { partitions } => {
+                assert!(!partitions.is_empty());
+            }
+            _ => panic!("Expected NoCacheAvailable strategy"),
+        }
+    }
+
+    #[test]
+    fn test_generate_optimal_partitions_hybrid() {
+        let discovery = CacheDiscoveryResult::new(
+            vec![CacheEntry {
+                file_path: "test.arrow".to_string(),
+                start_time: 0,
+                end_time: 5000,
+                interval: Interval::OneHour,
+            }],
+            vec![TimeRange::new(5000, 10000)],
+            0.5,
+        );
+
+        let strategy = generate_optimal_partitions(discovery, 0, 10000, CardinalityLevel::Low);
+
+        match strategy {
+            StreamingAggsPartitionStrategy::Hybrid {
+                cached_partitions,
+                uncached_partitions,
+            } => {
+                assert_eq!(cached_partitions.len(), 1);
+                assert!(!uncached_partitions.is_empty());
+            }
+            _ => panic!("Expected Hybrid strategy"),
+        }
+    }
+}

src/search/src/datafusion/aggregates/merge_phase.rs

@@ -0,0 +1,272 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Hash aggregation
+
+use std::sync::Arc;
+
+use arrow::{array::*, datatypes::SchemaRef};
+use datafusion::{
+    common::Result,
+    logical_expr::{EmitTo, GroupsAccumulator},
+    physical_expr::{GroupsAccumulatorAdapter, aggregate::AggregateFunctionExpr},
+    physical_plan::{
+        PhysicalExpr,
+        aggregates::{
+            AggregateExec, AggregateMode, PhysicalGroupBy, aggregate_expressions,
+            evaluate_group_by, evaluate_many,
+            group_values::{GroupValues, new_group_values},
+            order::GroupOrdering,
+        },
+    },
+};
+use log::debug;
+
+#[derive(Debug, Clone)]
+/// This object tracks the aggregation phase (input/output)
+pub(crate) enum ExecutionState {
+    ReadingInput,
+    /// When producing output, the remaining rows to output are stored
+    /// here and are sliced off as needed in batch_size chunks
+    ProducingOutput(RecordBatch),
+    /// All input has been consumed and all groups have been emitted
+    Done,
+}
+
+pub struct GroupedHashAggregateStream {
+    // ========================================================================
+    // PROPERTIES:
+    // These fields are initialized at the start and remain constant throughout
+    // the execution.
+    // ========================================================================
+    schema: SchemaRef,
+    mode: AggregateMode,
+
+    /// Arguments to pass to each accumulator.
+    ///
+    /// The arguments in `accumulator[i]` is passed `aggregate_arguments[i]`
+    ///
+    /// The argument to each accumulator is itself a `Vec` because
+    /// some aggregates such as `CORR` can accept more than one
+    /// argument.
+    aggregate_arguments: Vec<Vec<Arc<dyn PhysicalExpr>>>,
+
+    /// GROUP BY expressions
+    group_by: PhysicalGroupBy,
+
+    // ========================================================================
+    // STATE FLAGS:
+    // These fields will be updated during the execution. And control the flow of
+    // the execution.
+    // ========================================================================
+    /// Tracks if this stream is generating input or output
+    exec_state: ExecutionState,
+
+    /// Have we seen the end of the input
+    input_done: bool,
+
+    // ========================================================================
+    // STATE BUFFERS:
+    // These fields will accumulate intermediate results during the execution.
+    // ========================================================================
+    /// An interning store of group keys
+    group_values: Box<dyn GroupValues>,
+
+    /// scratch space for the current input [`RecordBatch`] being
+    /// processed. Reused across batches here to avoid reallocations
+    current_group_indices: Vec<usize>,
+
+    /// Accumulators, one for each `AggregateFunctionExpr` in the query
+    ///
+    /// For example, if the query has aggregates, `SUM(x)`,
+    /// `COUNT(y)`, there will be two accumulators, each one
+    /// specialized for that particular aggregate and its input types
+    accumulators: Vec<Box<dyn GroupsAccumulator>>,
+}
+
+impl GroupedHashAggregateStream {
+    /// Create a new GroupedHashAggregateStream
+    pub fn new(agg: &AggregateExec) -> Result<Self> {
+        debug!("Creating GroupedHashAggregateStream");
+        let agg_schema = agg.input().schema();
+        let agg_group_by = agg.group_expr().clone();
+        let aggregate_exprs = agg.aggr_expr();
+        let aggregate_arguments =
+            aggregate_expressions(agg.aggr_expr(), agg.mode(), agg_group_by.num_group_exprs())?;
+
+        // Instantiate the accumulators
+        let accumulators: Vec<_> = aggregate_exprs
+            .iter()
+            .map(create_group_accumulator)
+            .collect::<Result<_>>()?;
+
+        let group_schema = agg_group_by.group_schema(&agg.input().schema())?;
+
+        let group_values = new_group_values(group_schema, &GroupOrdering::None)?;
+
+        let exec_state = ExecutionState::ReadingInput;
+
+        Ok(GroupedHashAggregateStream {
+            schema: agg_schema,
+            mode: *agg.mode(),
+            accumulators,
+            aggregate_arguments,
+            group_by: agg_group_by,
+            group_values,
+            current_group_indices: Default::default(),
+            exec_state,
+            input_done: false,
+        })
+    }
+
+    pub fn schema(&self) -> SchemaRef {
+        Arc::clone(&self.schema)
+    }
+}
+
+/// Create an accumulator for `agg_expr` -- a [`GroupsAccumulator`] if
+/// that is supported by the aggregate, or a
+/// [`GroupsAccumulatorAdapter`] if not.
+pub(crate) fn create_group_accumulator(
+    agg_expr: &Arc<AggregateFunctionExpr>,
+) -> Result<Box<dyn GroupsAccumulator>> {
+    if agg_expr.groups_accumulator_supported() {
+        agg_expr.create_groups_accumulator()
+    } else {
+        // Note in the log when the slow path is used
+        debug!(
+            "Creating GroupsAccumulatorAdapter for {}: {agg_expr:?}",
+            agg_expr.name()
+        );
+        let agg_expr_captured = Arc::clone(agg_expr);
+        let factory = move || agg_expr_captured.create_accumulator();
+        Ok(Box::new(GroupsAccumulatorAdapter::new(factory)))
+    }
+}
+
+impl GroupedHashAggregateStream {
+    /// Perform group-by aggregation for the given [`RecordBatch`].
+    pub fn group_aggregate_batch(&mut self, batch: RecordBatch) -> Result<()> {
+        // Evaluate the grouping expressions
+        let group_by_values = evaluate_group_by(&self.group_by, &batch)?;
+
+        // Evaluate the aggregation expressions.
+        let input_values = evaluate_many(&self.aggregate_arguments, &batch)?;
+
+        for group_values in &group_by_values {
+            // calculate the group indices for each input row
+            self.group_values
+                .intern(group_values, &mut self.current_group_indices)?;
+            let group_indices = &self.current_group_indices;
+
+            // Update ordering information if necessary
+            let total_num_groups = self.group_values.len();
+
+            // Gather the inputs to call the actual accumulator
+            let t = self.accumulators.iter_mut().zip(input_values.iter());
+
+            for (acc, values) in t {
+                // Call the appropriate method on each aggregator with
+                // the entire input row and the relevant group indexes
+                match self.mode {
+                    AggregateMode::Partial
+                    | AggregateMode::Single
+                    | AggregateMode::SinglePartitioned => {
+                        acc.update_batch(values, group_indices, None, total_num_groups)?;
+                    }
+                    _ => {
+                        // if aggregation is over intermediate states,
+                        // use merge
+                        acc.merge_batch(values, group_indices, None, total_num_groups)?;
+                    }
+                }
+            }
+        }
+        Ok(())
+    }
+
+    /// Create an output RecordBatch with the group keys and
+    /// accumulator states/values specified in emit_to
+    fn emit(&mut self, emit_to: EmitTo) -> Result<Option<RecordBatch>> {
+        let schema = self.schema();
+        if self.group_values.is_empty() {
+            return Ok(None);
+        }
+
+        let mut output = self.group_values.emit(emit_to)?;
+
+        // Next output each aggregate value
+        for acc in self.accumulators.iter_mut() {
+            output.extend(acc.state(emit_to)?);
+        }
+
+        let batch = RecordBatch::try_new(schema, output)?;
+        debug_assert!(batch.num_rows() > 0);
+        Ok(Some(batch))
+    }
+
+    /// Clear memory and shirk capacities to the size of the batch.
+    fn clear_shrink(&mut self, num_rows: usize) {
+        self.group_values.clear_shrink(num_rows);
+        self.current_group_indices.clear();
+        self.current_group_indices.shrink_to(num_rows);
+    }
+
+    /// Clear memory and shirk capacities to zero.
+    fn clear_all(&mut self) {
+        self.clear_shrink(0);
+    }
+
+    /// common function for signalling end of processing of the input stream
+    fn set_input_done_and_produce_output(&mut self) -> Result<()> {
+        self.input_done = true;
+        let batch = self.emit(EmitTo::All)?;
+        self.exec_state = batch.map_or(ExecutionState::Done, ExecutionState::ProducingOutput);
+        Ok(())
+    }
+
+    pub fn get_final_result(&mut self) -> Result<Vec<RecordBatch>> {
+        self.set_input_done_and_produce_output()?;
+        let batch = match &self.exec_state {
+            ExecutionState::ProducingOutput(batch) => batch.clone(),
+            _ => RecordBatch::new_empty(self.schema()),
+        };
+        self.exec_state = ExecutionState::Done;
+        self.clear_all();
+
+        // split the batch into multiple batches
+        let num_rows = batch.num_rows();
+        // early return for empty batches
+        if num_rows == 0 {
+            return Ok(vec![]);
+        }
+        // calculate optimal batch size and pre-allocate vector
+        let batch_size = 8192;
+        let full_batches = num_rows / batch_size;
+        let has_remaining = !num_rows.is_multiple_of(batch_size);
+        let total_batches = full_batches + if has_remaining { 1 } else { 0 };
+        let mut result_vec = Vec::with_capacity(total_batches);
+        for i in 0..full_batches {
+            result_vec.push(batch.slice(i * batch_size, batch_size));
+        }
+        if has_remaining {
+            let start_idx = full_batches * batch_size;
+            let remaining_rows = num_rows - start_idx;
+            result_vec.push(batch.slice(start_idx, remaining_rows));
+        }
+
+        Ok(result_vec)
+    }
+}

src/search/src/datafusion/aggregates/mod.rs

@@ -0,0 +1,101 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+pub mod merge_phase;
+pub mod no_grouping_merge_phase;
+
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, AsArray, RecordBatch, RecordBatchOptions, StringViewBuilder};
+
+/// refer to: https://github.com/apache/datafusion/pull/11587
+/// Heuristically compact `StringViewArray`s to reduce memory usage, if needed
+///
+/// Decides when to consolidate the StringView into a new buffer to reduce
+/// memory usage and improve string locality for better performance.
+///
+/// This differs from `StringViewArray::gc` because:
+/// 1. It may not compact the array depending on a heuristic.
+/// 2. It uses a precise block size to reduce the number of buffers to track.
+///
+/// # Heuristic
+///
+/// If the average size of each view is larger than 32 bytes, we compact the array.
+///
+/// `StringViewArray` include pointers to buffer that hold the underlying data.
+/// One of the great benefits of `StringViewArray` is that many operations
+/// (e.g., `filter`) can be done without copying the underlying data.
+///
+/// However, after a while (e.g., after `FilterExec` or `HashJoinExec`) the
+/// `StringViewArray` may only refer to a small portion of the buffer,
+/// significantly increasing memory usage.
+pub(crate) fn gc_string_view_batch(batch: &RecordBatch) -> RecordBatch {
+    let new_columns: Vec<ArrayRef> = batch
+        .columns()
+        .iter()
+        .map(|c| {
+            // Try to re-create the `StringViewArray` to prevent holding the underlying buffer too
+            // long.
+            let Some(s) = c.as_string_view_opt() else {
+                return Arc::clone(c);
+            };
+
+            // Fast path: if the data buffers are empty, we can return the original array
+            if s.data_buffers().is_empty() {
+                return Arc::clone(c);
+            }
+
+            let ideal_buffer_size: usize = s
+                .views()
+                .iter()
+                .map(|v| {
+                    let len = (*v as u32) as usize;
+                    if len > 12 { len } else { 0 }
+                })
+                .sum();
+
+            // We don't use get_buffer_memory_size here, because gc is for the contents of the
+            // data buffers, not views and nulls.
+            let actual_buffer_size = s.data_buffers().iter().map(|b| b.capacity()).sum::<usize>();
+
+            // Re-creating the array copies data and can be time consuming.
+            // We only do it if the array is sparse
+            if actual_buffer_size > (ideal_buffer_size * 2) {
+                // We set the block size to `ideal_buffer_size` so that the new StringViewArray only
+                // has one buffer, which accelerate later concat_batches. See https://github.com/apache/arrow-rs/issues/6094 for more details.
+                let mut builder = StringViewBuilder::with_capacity(s.len());
+                if ideal_buffer_size > 0 {
+                    builder = builder.with_fixed_block_size(ideal_buffer_size as u32);
+                }
+
+                for v in s.iter() {
+                    builder.append_option(v);
+                }
+
+                let gc_string = builder.finish();
+
+                debug_assert!(gc_string.data_buffers().len() <= 1); // buffer count can be 0 if the `ideal_buffer_size` is 0
+
+                Arc::new(gc_string)
+            } else {
+                Arc::clone(c)
+            }
+        })
+        .collect();
+    let mut options = RecordBatchOptions::new();
+    options = options.with_row_count(Some(batch.num_rows()));
+    RecordBatch::try_new_with_options(batch.schema(), new_columns, &options)
+        .expect("Failed to re-create the gc'ed record batch")
+}

src/search/src/datafusion/aggregates/no_grouping_merge_phase.rs

@@ -0,0 +1,178 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Aggregate without grouping columns
+
+use std::{borrow::Cow, sync::Arc};
+
+use arrow::{array::ArrayRef, datatypes::SchemaRef, record_batch::RecordBatch};
+use datafusion::{
+    common::Result,
+    physical_plan::{
+        PhysicalExpr,
+        aggregates::{
+            AccumulatorItem, AggregateExec, AggregateMode, aggregate_expressions,
+            create_accumulators,
+        },
+        filter::batch_filter,
+    },
+};
+use itertools::Itertools;
+
+/// stream struct for aggregation without grouping columns
+pub struct AggregateStream {
+    schema: SchemaRef,
+    mode: AggregateMode,
+    aggregate_expressions: Vec<Vec<Arc<dyn PhysicalExpr>>>,
+    filter_expressions: Vec<Option<Arc<dyn PhysicalExpr>>>,
+    accumulators: Vec<AccumulatorItem>,
+}
+
+impl AggregateStream {
+    /// Create a new AggregateStream
+    pub fn new(agg: &AggregateExec) -> Result<Self> {
+        let agg_schema = agg.input().schema();
+        let agg_filter_expr = agg.filter_expr().to_vec();
+
+        let aggregate_expressions = aggregate_expressions(agg.aggr_expr(), agg.mode(), 0)?;
+        let filter_expressions = match *agg.mode() {
+            AggregateMode::Partial | AggregateMode::Single | AggregateMode::SinglePartitioned => {
+                agg_filter_expr
+            }
+            AggregateMode::Final
+            | AggregateMode::FinalPartitioned
+            | AggregateMode::PartialReduce => {
+                vec![None; agg.aggr_expr().len()]
+            }
+        };
+        let accumulators = create_accumulators(agg.aggr_expr())?;
+
+        Ok(AggregateStream {
+            schema: agg_schema,
+            mode: *agg.mode(),
+            aggregate_expressions,
+            filter_expressions,
+            accumulators,
+        })
+    }
+
+    pub fn aggregate_batch(&mut self, batch: RecordBatch) -> Result<()> {
+        let _ = aggregate_batch(
+            &self.mode,
+            batch,
+            &mut self.accumulators,
+            &self.aggregate_expressions,
+            &self.filter_expressions,
+        )?;
+
+        Ok(())
+    }
+
+    pub fn finalize_aggregation(&mut self) -> Result<Vec<RecordBatch>> {
+        let result = finalize_aggregation(&mut self.accumulators)?;
+        let batch = RecordBatch::try_new(Arc::clone(&self.schema), result)?;
+
+        // split the batch into multiple batches
+        let num_rows = batch.num_rows();
+        // early return for empty batches
+        if num_rows == 0 {
+            return Ok(vec![]);
+        }
+        // calculate optimal batch size and pre-allocate vector
+        let batch_size = 8192;
+        let full_batches = num_rows / batch_size;
+        let has_remaining = !num_rows.is_multiple_of(batch_size);
+        let total_batches = full_batches + if has_remaining { 1 } else { 0 };
+        let mut result_vec = Vec::with_capacity(total_batches);
+        for i in 0..full_batches {
+            result_vec.push(batch.slice(i * batch_size, batch_size));
+        }
+        if has_remaining {
+            let start_idx = full_batches * batch_size;
+            let remaining_rows = num_rows - start_idx;
+            result_vec.push(batch.slice(start_idx, remaining_rows));
+        }
+
+        Ok(result_vec)
+    }
+}
+
+fn aggregate_batch(
+    mode: &AggregateMode,
+    batch: RecordBatch,
+    accumulators: &mut [AccumulatorItem],
+    expressions: &[Vec<Arc<dyn PhysicalExpr>>],
+    filters: &[Option<Arc<dyn PhysicalExpr>>],
+) -> Result<usize> {
+    let mut allocated = 0usize;
+
+    // 1.1 iterate accumulators and respective expressions together
+    // 1.2 filter the batch if necessary
+    // 1.3 evaluate expressions
+    // 1.4 update / merge accumulators with the expressions' values
+
+    // 1.1
+    accumulators
+        .iter_mut()
+        .zip(expressions)
+        .zip(filters)
+        .try_for_each(|((accum, expr), filter)| {
+            // 1.2
+            let batch = match filter {
+                Some(filter) => Cow::Owned(batch_filter(&batch, filter)?),
+                None => Cow::Borrowed(&batch),
+            };
+
+            // 1.3
+            let values = &expr
+                .iter()
+                .map(|e| {
+                    e.evaluate(&batch)
+                        .and_then(|v| v.into_array(batch.num_rows()))
+                })
+                .collect::<Result<Vec<_>>>()?;
+
+            // 1.4
+            let size_pre = accum.size();
+            let res = match mode {
+                AggregateMode::Partial
+                | AggregateMode::Single
+                | AggregateMode::SinglePartitioned => accum.update_batch(values),
+                AggregateMode::Final
+                | AggregateMode::FinalPartitioned
+                | AggregateMode::PartialReduce => accum.merge_batch(values),
+            };
+            let size_post = accum.size();
+            allocated += size_post.saturating_sub(size_pre);
+            res
+        })?;
+
+    Ok(allocated)
+}
+
+fn finalize_aggregation(accumulators: &mut [AccumulatorItem]) -> Result<Vec<ArrayRef>> {
+    // Build the vector of states
+    accumulators
+        .iter_mut()
+        .map(|accumulator| {
+            accumulator.state().and_then(|e| {
+                e.iter()
+                    .map(|v| v.to_array())
+                    .collect::<Result<Vec<ArrayRef>>>()
+            })
+        })
+        .flatten_ok()
+        .collect()
+}

src/search/src/datafusion/distributed_plan/aggregate_topk_exec/heap.rs

@@ -0,0 +1,794 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+use std::{
+    cmp::Ordering,
+    collections::{BinaryHeap, HashMap},
+    fmt::Debug,
+    pin::Pin,
+    sync::Arc,
+    task::{Context, Poll},
+};
+
+use arrow::{
+    array::{ArrayRef, RecordBatch},
+    compute::{SortOptions, interleave_record_batch},
+    datatypes::SchemaRef,
+    row::{RowConverter, SortField},
+};
+use datafusion::{
+    common::Result,
+    execution::{RecordBatchStream, SendableRecordBatchStream},
+};
+use futures::{Stream, StreamExt};
+
+pub struct TopKHeapStream {
+    schema: SchemaRef,
+    stream: SendableRecordBatchStream,
+    limit: usize,
+    topk_heap: BinaryHeap<HeapRow>,
+    record_batch_registry: RecordBatchRegistry,
+    sort_column_index: usize,
+    row_converter: RowConverter,
+}
+
+struct RecordBatchRegistry {
+    store: HashMap<u32, RecordBatchEntry>,
+    next_id: u32,
+}
+
+impl RecordBatchRegistry {
+    pub fn new() -> Self {
+        Self {
+            store: HashMap::new(),
+            next_id: 0,
+        }
+    }
+
+    pub fn register_entry(&mut self, rb: RecordBatch) -> RecordBatchEntry {
+        let id = self.next_id;
+        let record_batch_entry = RecordBatchEntry::new(id, rb);
+        self.next_id += 1;
+        record_batch_entry
+    }
+
+    pub fn submit_entry(&mut self, entry: RecordBatchEntry) {
+        if entry.uses > 0 {
+            self.store.insert(entry.id, entry);
+        }
+    }
+
+    pub fn remove_use_from_entry(&mut self, id: u32) {
+        if let Some(entry) = self.store.get_mut(&id) {
+            let Some(uses) = entry.uses.checked_sub(1) else {
+                panic!("underflow of uses for batch {id}");
+            };
+
+            if uses == 0 {
+                // remove the record batch from the registry
+                self.store.remove(&id).expect("cannot remove batch {id}");
+            }
+        } else {
+            panic!("entry does not exists batch {id}");
+        }
+    }
+}
+
+struct RecordBatchEntry {
+    id: u32,
+    record_batch: RecordBatch,
+    uses: usize,
+}
+
+impl RecordBatchEntry {
+    pub fn new(id: u32, record_batch: RecordBatch) -> Self {
+        Self {
+            id,
+            record_batch,
+            uses: 0,
+        }
+    }
+}
+
+#[derive(Debug, Clone)]
+struct HeapRow {
+    sort_value: Vec<u8>,
+    row_id: usize,
+    batch_id: u32,
+}
+
+impl HeapRow {
+    fn with_new_row(mut self, new_row_bytes: &[u8], row_id: usize, batch_id: u32) -> Self {
+        self.sort_value.clear();
+        self.sort_value.extend_from_slice(new_row_bytes);
+        self.row_id = row_id;
+        self.batch_id = batch_id;
+        self
+    }
+}
+
+impl PartialEq for HeapRow {
+    fn eq(&self, other: &Self) -> bool {
+        self.sort_value == other.sort_value
+    }
+}
+
+impl Eq for HeapRow {}
+
+impl PartialOrd for HeapRow {
+    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+        Some(self.cmp(other))
+    }
+}
+
+impl Ord for HeapRow {
+    fn cmp(&self, other: &Self) -> Ordering {
+        // For min-heap behavior to get top-K largest values
+        // We want smallest values at the top of the heap so we can pop them
+        // RowConverter produces lexicographically sortable byte arrays
+        self.sort_value.cmp(&other.sort_value)
+    }
+}
+
+impl TopKHeapStream {
+    pub fn new(
+        schema: SchemaRef,
+        stream: SendableRecordBatchStream,
+        sort_field: String,
+        descending: bool,
+        limit: usize,
+    ) -> Self {
+        // Find the index of the sort column
+        // also handle cases where the sort fields are not alias and can be names as count(*)[count]
+        let sort_column_index = schema
+            .fields()
+            .iter()
+            .position(|f| {
+                f.name() == &sort_field || f.name().split('[').next().unwrap_or("") == sort_field
+            })
+            .expect("Sort field not found in schema");
+
+        // Create RowConverter for the sort column with proper sort options
+        let sort_field_ref = &schema.fields()[sort_column_index];
+        let sort_options = if descending {
+            SortOptions::default().desc()
+        } else {
+            SortOptions::default().asc()
+        };
+        let sort_field =
+            SortField::new_with_options(sort_field_ref.data_type().clone(), sort_options);
+        let row_converter =
+            RowConverter::new(vec![sort_field]).expect("Failed to create RowConverter");
+
+        Self {
+            schema,
+            stream,
+            limit,
+            record_batch_registry: RecordBatchRegistry::new(),
+            topk_heap: BinaryHeap::new(),
+            sort_column_index,
+            row_converter,
+        }
+    }
+
+    fn convert_sort_column(&mut self, array: &ArrayRef) -> arrow::row::Rows {
+        // Direct conversion - simpler and avoids persistent memory
+        self.row_converter
+            .convert_columns(std::slice::from_ref(array))
+            .expect("Failed to convert column")
+    }
+
+    fn process_batch(&mut self, batch: RecordBatch) {
+        if batch.num_rows() == 0 {
+            return;
+        }
+
+        let sort_column = batch.column(self.sort_column_index);
+        let mut entry = self.record_batch_registry.register_entry(batch.clone());
+
+        // Convert all sort values at once - gets cleaned up automatically
+        let converted_rows = self.convert_sort_column(sort_column);
+        for row_index in 0..batch.num_rows() {
+            // Get row from converted batch
+            let row_ref = converted_rows.row(row_index);
+
+            if self.topk_heap.len() < self.limit {
+                // Heap not full - create new row
+                let new_row = HeapRow {
+                    sort_value: row_ref.as_ref().to_vec(),
+                    row_id: row_index,
+                    batch_id: entry.id,
+                };
+                entry.uses += 1;
+                self.topk_heap.push(new_row);
+            } else if let Some(heap_top) = self.topk_heap.peek() {
+                let should_replace =
+                    row_ref.as_ref().cmp(heap_top.sort_value.as_slice()) == Ordering::Less;
+                if should_replace {
+                    let popped_row = self.topk_heap.pop().unwrap();
+                    // Update batch tracking
+                    if popped_row.batch_id.ne(&entry.id) {
+                        entry.uses += 1;
+                        self.record_batch_registry
+                            .remove_use_from_entry(popped_row.batch_id);
+                    }
+                    // Reuse the Vec<u8> memory - this is the key optimization
+                    let reused_row = popped_row.with_new_row(row_ref.as_ref(), row_index, entry.id);
+                    self.topk_heap.push(reused_row);
+                }
+            }
+        }
+
+        self.record_batch_registry.submit_entry(entry);
+    }
+
+    fn heap_to_record_batch(&mut self) -> Option<RecordBatch> {
+        if self.topk_heap.is_empty() {
+            return None;
+        }
+
+        // Convert heap to sorted vec
+        // Since the heap is already having elements which are bit flipped in row converter
+        // we do not need to resort the final results outside.
+        let sorted_rows = std::mem::take(&mut self.topk_heap).into_sorted_vec();
+
+        let mut record_batches = Vec::new();
+        let mut batch_id_array_pos = HashMap::new();
+        for (batch_pos, (batch_id, batch)) in self.record_batch_registry.store.iter().enumerate() {
+            record_batches.push(&batch.record_batch);
+            batch_id_array_pos.insert(*batch_id, batch_pos);
+        }
+
+        let indices: Vec<_> = sorted_rows
+            .iter()
+            .map(|row| (batch_id_array_pos[&row.batch_id], row.row_id))
+            .collect();
+
+        let final_batch = interleave_record_batch(&record_batches, &indices)
+            .map_err(|_| log::error!("Failed to interleave_record_batch"))
+            .ok()?;
+        Some(final_batch)
+    }
+}
+
+impl Stream for TopKHeapStream {
+    type Item = Result<RecordBatch>;
+
+    fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
+        match self.stream.poll_next_unpin(cx) {
+            Poll::Ready(Some(Ok(batch))) => {
+                // Process the batch incrementally with heap
+                self.process_batch(batch);
+
+                // Return empty batch to indicate progress
+                let schema = self.schema.clone();
+                let empty_batch = RecordBatch::new_empty(schema);
+                Poll::Ready(Some(Ok(empty_batch)))
+            }
+            Poll::Ready(None) => {
+                // Stream is finished, return final top-K result
+                let topk_batch = self.heap_to_record_batch();
+                Poll::Ready(topk_batch.map(Ok))
+            }
+            Poll::Pending => Poll::Pending,
+            Poll::Ready(Some(Err(e))) => {
+                log::error!("Error in CacheTopkStream: {e}");
+                Poll::Ready(None)
+            }
+        }
+    }
+
+    fn size_hint(&self) -> (usize, Option<usize>) {
+        self.stream.size_hint()
+    }
+}
+
+impl RecordBatchStream for TopKHeapStream {
+    /// Get the schema
+    fn schema(&self) -> SchemaRef {
+        Arc::clone(&self.schema)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::sync::Arc;
+
+    use arrow::{
+        array::{Array, Int64Array, StringArray},
+        datatypes::{Field, Schema},
+        util::pretty::pretty_format_batches,
+    };
+
+    use super::*;
+
+    struct TestRecordBatchStream {
+        schema: SchemaRef,
+        batches: Vec<RecordBatch>,
+        index: usize,
+    }
+
+    impl TestRecordBatchStream {
+        fn new(schema: SchemaRef, batches: Vec<RecordBatch>) -> Self {
+            Self {
+                schema,
+                batches,
+                index: 0,
+            }
+        }
+    }
+
+    impl Stream for TestRecordBatchStream {
+        type Item = Result<RecordBatch>;
+
+        fn poll_next(mut self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll<Option<Self::Item>> {
+            if self.index < self.batches.len() {
+                let batch = self.batches[self.index].clone();
+                self.index += 1;
+                Poll::Ready(Some(Ok(batch)))
+            } else {
+                Poll::Ready(None)
+            }
+        }
+    }
+
+    impl RecordBatchStream for TestRecordBatchStream {
+        fn schema(&self) -> SchemaRef {
+            self.schema.clone()
+        }
+    }
+
+    #[tokio::test]
+    async fn test_cache_topk_stream_descending() {
+        // Create schema with name and count columns
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("name", arrow::datatypes::DataType::Utf8, false),
+            Field::new("count", arrow::datatypes::DataType::Int64, false),
+        ]));
+
+        // Create test data with multiple batches
+        let batch1 = RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(StringArray::from(vec![
+                    "item1", "item7", "item8", "item6", "item2", "item3",
+                ])),
+                Arc::new(Int64Array::from(vec![10, 12, 13, 24, 25, 15])),
+            ],
+        )
+        .unwrap();
+
+        let batch2 = RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(StringArray::from(vec!["item4", "item5"])),
+                Arc::new(Int64Array::from(vec![5, 30])),
+            ],
+        )
+        .unwrap();
+
+        let test_stream = TestRecordBatchStream::new(schema.clone(), vec![batch1, batch2]);
+        let stream: SendableRecordBatchStream = Box::pin(test_stream);
+
+        // Create CacheTopkStream for top-3 descending by count
+        let mut topk_stream = TopKHeapStream::new(
+            schema.clone(),
+            stream,
+            "count".to_string(),
+            true, // descending
+            5,    // limit
+        );
+
+        let mut results = Vec::new();
+        while let Some(result) = topk_stream.next().await {
+            match result {
+                Ok(batch) => {
+                    if batch.num_rows() > 0 {
+                        results.push(batch);
+                    }
+                }
+                Err(e) => panic!("Stream error: {e}"),
+            }
+        }
+
+        println!("{}", pretty_format_batches(&results).unwrap());
+
+        // Should have one final result batch with top-3 items
+        assert_eq!(results.len(), 1);
+        let final_batch = &results[0];
+        assert_eq!(final_batch.num_rows(), 5);
+        // println!("{}", pretty_format_batches(&[final_batch.clone()]).unwrap());
+        // Verify the results are in descending order: item5(30), item2(25), item3(15)
+        let names = final_batch
+            .column(0)
+            .as_any()
+            .downcast_ref::<StringArray>()
+            .unwrap();
+        let counts = final_batch
+            .column(1)
+            .as_any()
+            .downcast_ref::<Int64Array>()
+            .unwrap();
+
+        assert_eq!(names.value(0), "item5");
+        assert_eq!(counts.value(0), 30);
+        assert_eq!(names.value(1), "item2");
+        assert_eq!(counts.value(1), 25);
+        assert_eq!(names.value(2), "item6");
+        assert_eq!(counts.value(2), 24);
+    }
+
+    #[tokio::test]
+    async fn test_cache_topk_stream_ascending() {
+        // Create schema
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("name", arrow::datatypes::DataType::Utf8, false),
+            Field::new("value", arrow::datatypes::DataType::Int64, false),
+        ]));
+
+        // Create test data
+        let batch = RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(StringArray::from(vec!["a", "b", "c", "d", "e"])),
+                Arc::new(Int64Array::from(vec![50, 20, 80, 10, 30])),
+            ],
+        )
+        .unwrap();
+
+        let test_stream = TestRecordBatchStream::new(schema.clone(), vec![batch]);
+        let stream: SendableRecordBatchStream = Box::pin(test_stream);
+
+        // Create CacheTopkStream for top-3 ascending by value
+        let mut topk_stream = TopKHeapStream::new(
+            schema.clone(),
+            stream,
+            "value".to_string(),
+            false, // ascending
+            5,     // limit
+        );
+
+        let mut results = Vec::new();
+        while let Some(result) = topk_stream.next().await {
+            match result {
+                Ok(batch) => {
+                    if batch.num_rows() > 0 {
+                        results.push(batch);
+                    }
+                }
+                Err(e) => panic!("Stream error: {e}"),
+            }
+        }
+
+        println!("{}", pretty_format_batches(&results).unwrap());
+        // Should have one final result batch with top-3 smallest items
+        assert_eq!(results.len(), 1);
+        let final_batch = &results[0];
+        assert_eq!(final_batch.num_rows(), 5);
+        // println!("{}", pretty_format_batches(&[final_batch.clone()]).unwrap());
+
+        // Verify the results are in ascending order: d(10), b(20), e(30)
+        let names = final_batch
+            .column(0)
+            .as_any()
+            .downcast_ref::<StringArray>()
+            .unwrap();
+        let values = final_batch
+            .column(1)
+            .as_any()
+            .downcast_ref::<Int64Array>()
+            .unwrap();
+
+        assert_eq!(names.value(0), "d");
+        assert_eq!(values.value(0), 10);
+        assert_eq!(names.value(1), "b");
+        assert_eq!(values.value(1), 20);
+        assert_eq!(names.value(2), "e");
+        assert_eq!(values.value(2), 30);
+    }
+
+    #[tokio::test]
+    async fn test_cache_topk_stream_limit() {
+        // Create schema
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("id", arrow::datatypes::DataType::Utf8, false),
+            Field::new("score", arrow::datatypes::DataType::Int64, false),
+        ]));
+
+        // Create test data with more items than limit
+        let batch = RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(StringArray::from(vec![
+                    "id1", "id2", "id3", "id4", "id5", "id6",
+                ])),
+                Arc::new(Int64Array::from(vec![100, 200, 50, 300, 150, 75])),
+            ],
+        )
+        .unwrap();
+
+        let test_stream = TestRecordBatchStream::new(schema.clone(), vec![batch]);
+        let stream: SendableRecordBatchStream = Box::pin(test_stream);
+
+        // Create CacheTopkStream for top-2 descending by score
+        let mut topk_stream = TopKHeapStream::new(
+            schema.clone(),
+            stream,
+            "score".to_string(),
+            true, // descending
+            2,    // limit to 2
+        );
+
+        let mut results = Vec::new();
+        while let Some(result) = topk_stream.next().await {
+            match result {
+                Ok(batch) => {
+                    if batch.num_rows() > 0 {
+                        results.push(batch);
+                    }
+                }
+                Err(e) => panic!("Stream error: {e}"),
+            }
+        }
+
+        println!("{}", pretty_format_batches(&results).unwrap());
+        // println!("{}", pretty_format_batches(&results).unwrap());
+        // Should have one final result batch with top-2 items only
+        assert_eq!(results.len(), 1);
+        let final_batch = &results[0];
+        // the final batch rows can never be less than the limit when
+        // enough data is present
+        assert!(final_batch.num_rows() >= 2);
+
+        // Verify the results are the top-2: id4(300), id2(200)
+        let ids = final_batch
+            .column(0)
+            .as_any()
+            .downcast_ref::<StringArray>()
+            .unwrap();
+        let scores = final_batch
+            .column(1)
+            .as_any()
+            .downcast_ref::<Int64Array>()
+            .unwrap();
+
+        assert_eq!(ids.value(0), "id4");
+        assert_eq!(scores.value(0), 300);
+        assert_eq!(ids.value(1), "id2");
+        assert_eq!(scores.value(1), 200);
+    }
+
+    #[tokio::test]
+    async fn test_complex_schema_final_row_construction() {
+        use arrow::array::{BooleanArray, Float32Array, Int32Array, TimestampMillisecondArray};
+
+        // Test with a very complex schema including different data types
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("user_name", arrow::datatypes::DataType::Utf8, false),
+            Field::new("is_premium", arrow::datatypes::DataType::Boolean, false),
+            Field::new("score", arrow::datatypes::DataType::Float32, false),
+            Field::new("rank", arrow::datatypes::DataType::Int64, false),
+            Field::new("session_id", arrow::datatypes::DataType::Int32, false),
+            Field::new(
+                "timestamp",
+                arrow::datatypes::DataType::Timestamp(
+                    arrow::datatypes::TimeUnit::Millisecond,
+                    None,
+                ),
+                false,
+            ),
+            Field::new("region", arrow::datatypes::DataType::Utf8, true),
+        ]));
+
+        // Create batches with mixed data types
+        let batch1 = RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(StringArray::from(vec!["alice", "bob", "carol"])),
+                Arc::new(BooleanArray::from(vec![true, false, true])),
+                Arc::new(Float32Array::from(vec![75.5, 67.2, 82.1])),
+                Arc::new(Int64Array::from(vec![10, 20, 15])),
+                Arc::new(Int32Array::from(vec![1001, 1002, 1003])),
+                Arc::new(TimestampMillisecondArray::from(vec![
+                    1000000, 2000000, 1500000,
+                ])),
+                Arc::new(StringArray::from(vec![Some("US"), Some("EU"), None])),
+            ],
+        )
+        .unwrap();
+
+        let batch2 = RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(StringArray::from(vec!["diana", "eve"])),
+                Arc::new(BooleanArray::from(vec![true, false])),
+                Arc::new(Float32Array::from(vec![95.7, 88.3])),
+                Arc::new(Int64Array::from(vec![5, 8])),
+                Arc::new(Int32Array::from(vec![1004, 1005])),
+                Arc::new(TimestampMillisecondArray::from(vec![3000000, 2500000])),
+                Arc::new(StringArray::from(vec![Some("APAC"), Some("US")])),
+            ],
+        )
+        .unwrap();
+
+        let test_stream = TestRecordBatchStream::new(schema.clone(), vec![batch1, batch2]);
+        let stream: SendableRecordBatchStream = Box::pin(test_stream);
+
+        let mut topk_stream = TopKHeapStream::new(
+            schema.clone(),
+            stream,
+            "score".to_string(),
+            true, // descending
+            5,    // all 5 rows
+        );
+
+        let mut results = Vec::new();
+        while let Some(result) = topk_stream.next().await {
+            match result {
+                Ok(batch) => {
+                    if batch.num_rows() > 0 {
+                        results.push(batch);
+                    }
+                }
+                Err(e) => panic!("Stream error: {e}"),
+            }
+        }
+
+        assert_eq!(results.len(), 1);
+        let final_batch = &results[0];
+        assert_eq!(final_batch.num_rows(), 5);
+        assert_eq!(final_batch.num_columns(), 7);
+
+        // Verify all data types are preserved correctly
+        // Expected order: diana(95.7), eve(88.3), carol(82.1), alice(75.5), bob(67.2)
+        let user_names = final_batch
+            .column(0)
+            .as_any()
+            .downcast_ref::<StringArray>()
+            .unwrap();
+        let is_premium = final_batch
+            .column(1)
+            .as_any()
+            .downcast_ref::<BooleanArray>()
+            .unwrap();
+        let scores = final_batch
+            .column(2)
+            .as_any()
+            .downcast_ref::<Float32Array>()
+            .unwrap();
+        let ranks = final_batch
+            .column(3)
+            .as_any()
+            .downcast_ref::<Int64Array>()
+            .unwrap();
+        let session_ids = final_batch
+            .column(4)
+            .as_any()
+            .downcast_ref::<Int32Array>()
+            .unwrap();
+        let timestamps = final_batch
+            .column(5)
+            .as_any()
+            .downcast_ref::<TimestampMillisecondArray>()
+            .unwrap();
+        let regions = final_batch
+            .column(6)
+            .as_any()
+            .downcast_ref::<StringArray>()
+            .unwrap();
+
+        // First row: diana (highest score)
+        assert_eq!(user_names.value(0), "diana");
+        assert!(is_premium.value(0));
+        assert!((scores.value(0) - 95.7).abs() < f32::EPSILON);
+        assert_eq!(ranks.value(0), 5);
+        assert_eq!(session_ids.value(0), 1004);
+        assert_eq!(timestamps.value(0), 3000000);
+        assert_eq!(regions.value(0), "APAC");
+
+        // Second row: eve
+        assert_eq!(user_names.value(1), "eve");
+        assert!(!is_premium.value(1));
+        assert!((scores.value(1) - 88.3).abs() < f32::EPSILON);
+        assert_eq!(ranks.value(1), 8);
+        assert_eq!(session_ids.value(1), 1005);
+        assert_eq!(timestamps.value(1), 2500000);
+        assert_eq!(regions.value(1), "US");
+
+        // Third row: carol
+        assert_eq!(user_names.value(2), "carol");
+        assert!(is_premium.value(2));
+        assert!((scores.value(2) - 82.1).abs() < f32::EPSILON);
+        assert_eq!(ranks.value(2), 15);
+        assert_eq!(session_ids.value(2), 1003);
+        assert_eq!(timestamps.value(2), 1500000);
+        assert!(regions.is_null(2));
+
+        println!("{}", pretty_format_batches(&results).unwrap());
+    }
+
+    #[tokio::test]
+    async fn test_many_batches_force_eviction() {
+        // Test with many batches where we have more batches than the limit
+        // This forces the registry to manage entries across different batches
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("batch_name", arrow::datatypes::DataType::Utf8, false),
+            Field::new("value", arrow::datatypes::DataType::Int64, false),
+        ]));
+
+        // Create 6 batches, each with 1 row, but limit to only 3 results
+        let mut batches = Vec::new();
+        let values = [10, 50, 20, 80, 30, 90]; // 90, 80, 50 should be top 3
+        //
+        for (i, &value) in values.iter().enumerate() {
+            let batch = RecordBatch::try_new(
+                schema.clone(),
+                vec![
+                    Arc::new(StringArray::from(vec![format!("batch_{}", i)])),
+                    Arc::new(Int64Array::from(vec![value])),
+                ],
+            )
+            .unwrap();
+            batches.push(batch);
+        }
+
+        let test_stream = TestRecordBatchStream::new(schema.clone(), batches);
+        let stream: SendableRecordBatchStream = Box::pin(test_stream);
+
+        let mut topk_stream = TopKHeapStream::new(
+            schema.clone(),
+            stream,
+            "value".to_string(),
+            true, // descending
+            3,    // limit to 3, but we have 6 batches
+        );
+
+        let mut results = Vec::new();
+        while let Some(result) = topk_stream.next().await {
+            match result {
+                Ok(batch) => {
+                    if batch.num_rows() > 0 {
+                        results.push(batch);
+                    }
+                }
+                Err(e) => panic!("Stream error: {e}"),
+            }
+        }
+
+        assert_eq!(results.len(), 1);
+        let final_batch = &results[0];
+        assert_eq!(final_batch.num_rows(), 3);
+
+        // Should be: batch_5(90), batch_3(80), batch_1(50)
+        let names = final_batch
+            .column(0)
+            .as_any()
+            .downcast_ref::<StringArray>()
+            .unwrap();
+        let values = final_batch
+            .column(1)
+            .as_any()
+            .downcast_ref::<Int64Array>()
+            .unwrap();
+
+        assert_eq!(names.value(0), "batch_5");
+        assert_eq!(values.value(0), 90);
+        assert_eq!(names.value(1), "batch_3");
+        assert_eq!(values.value(1), 80);
+        assert_eq!(names.value(2), "batch_1");
+        assert_eq!(values.value(2), 50);
+
+        println!("{}", pretty_format_batches(&results).unwrap());
+    }
+}

src/search/src/datafusion/distributed_plan/aggregate_topk_exec/mod.rs

@@ -0,0 +1,203 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{fmt::Debug, sync::Arc};
+
+use arrow::datatypes::SchemaRef;
+use datafusion::{
+    common::Result,
+    execution::{SendableRecordBatchStream, TaskContext},
+    physical_expr::EquivalenceProperties,
+    physical_plan::{
+        DisplayAs, DisplayFormatType, ExecutionPlan, ExecutionPlanProperties, Partitioning,
+        PlanProperties,
+        execution_plan::{Boundedness, EmissionType},
+    },
+};
+
+pub mod heap;
+pub mod sort;
+
+#[derive(Debug)]
+pub struct AggregateTopkExec {
+    input: Arc<dyn ExecutionPlan>,
+    /// Cache holding plan properties like equivalences, output partitioning etc.
+    cache: Arc<PlanProperties>,
+    target_partitions: usize,
+    sort_field: String,
+    descending: bool,
+    limit: u64,
+}
+
+impl AggregateTopkExec {
+    /// Create a new AggregateMergeExec with explicit cache strategy
+    pub fn new(
+        input: Arc<dyn ExecutionPlan>,
+        sort_field: &str,
+        descending: bool,
+        limit: u64,
+    ) -> Self {
+        // Partial or no cache: cached partitions + input partitions
+        let target_partitions = input.output_partitioning().partition_count();
+        let cache = Self::compute_properties(Arc::clone(&input.schema()), target_partitions);
+        let sort_field = input
+            .schema()
+            .fields()
+            .iter()
+            .find(|f| {
+                // field name like count(*)[count]
+                f.name() == sort_field
+                    || f.name().split('[').next().is_some_and(|v| v == sort_field)
+            })
+            .unwrap()
+            .name()
+            .to_string();
+
+        Self {
+            input,
+            cache,
+            target_partitions,
+            sort_field,
+            descending,
+            limit,
+        }
+    }
+
+    fn output_partitioning_helper(n_partitions: usize) -> Partitioning {
+        Partitioning::UnknownPartitioning(n_partitions)
+    }
+
+    /// This function creates the cache object that stores the plan properties such as schema,
+    /// equivalence properties, ordering, partitioning, etc.
+    fn compute_properties(schema: SchemaRef, n_partitions: usize) -> Arc<PlanProperties> {
+        let eq_properties = EquivalenceProperties::new(schema);
+        let output_partitioning = Self::output_partitioning_helper(n_partitions);
+        Arc::new(PlanProperties::new(
+            eq_properties,
+            // Output Partitioning
+            output_partitioning,
+            // Execution Mode
+            EmissionType::Incremental,
+            Boundedness::Bounded,
+        ))
+    }
+
+    pub fn sort_field(&self) -> &str {
+        &self.sort_field
+    }
+
+    pub fn descending(&self) -> bool {
+        self.descending
+    }
+
+    pub fn limit(&self) -> u64 {
+        self.limit
+    }
+}
+
+impl DisplayAs for AggregateTopkExec {
+    fn fmt_as(&self, t: DisplayFormatType, f: &mut std::fmt::Formatter) -> std::fmt::Result {
+        match t {
+            DisplayFormatType::Default | DisplayFormatType::Verbose => {
+                write!(
+                    f,
+                    "AggregateTopkExec: target_partitions={}, limit={}, descending={}",
+                    self.target_partitions, self.limit, self.descending
+                )
+            }
+            DisplayFormatType::TreeRender => {
+                _ = writeln!(f, "target_partitions={}", self.target_partitions);
+                _ = writeln!(f, "limit={}", self.limit);
+                _ = writeln!(f, "descending={}", self.descending);
+                Ok(())
+            }
+        }
+    }
+}
+
+impl ExecutionPlan for AggregateTopkExec {
+    fn name(&self) -> &'static str {
+        "AggregateTopkExec"
+    }
+
+    fn properties(&self) -> &Arc<PlanProperties> {
+        &self.cache
+    }
+
+    fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>> {
+        vec![&self.input]
+    }
+
+    fn with_new_children(
+        self: Arc<Self>,
+        children: Vec<Arc<dyn ExecutionPlan>>,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        if children.is_empty() {
+            return Ok(self);
+        }
+        Ok(Arc::new(Self::new(
+            children[0].clone(),
+            &self.sort_field,
+            self.descending,
+            self.limit,
+        )))
+    }
+
+    fn execute(
+        &self,
+        partition: usize,
+        context: Arc<TaskContext>,
+    ) -> Result<SendableRecordBatchStream> {
+        let cfg = config::get_config();
+
+        // We need to dynamically choose operator to use based on K (limit) because
+        // heap is more memory effecient and performant when the K <= 200 range, but as the range
+        // increases the performance takes a hit. In such cases, giving up on memory and
+        // prioritizing performance make more sense.
+        let can_use_top_k_heap =
+            cfg.common.use_agg_topk_heap && self.limit <= cfg.common.agg_topk_heap_max_limit;
+
+        let pinned_stream: SendableRecordBatchStream = if can_use_top_k_heap {
+            // we use inflated limit here to calculate topK values on partial aggregation results
+            // such that we mimize the risk of losing counts. Having a large limit ensures we take
+            // more keys into consideration when sending out the final record batch to leader.
+            let inflated_limit = (self.limit * 4).max(1000) as usize;
+            Box::pin(heap::TopKHeapStream::new(
+                self.input.schema(),
+                self.input.execute(partition, Arc::clone(&context))?,
+                self.sort_field.clone(),
+                self.descending,
+                inflated_limit,
+            ))
+        } else {
+            Box::pin(sort::TopKSortStream::new(
+                self.input.schema(),
+                self.input.execute(partition, context)?,
+                self.sort_field.clone(),
+                self.descending,
+                self.limit,
+            ))
+        };
+        Ok(pinned_stream)
+    }
+
+    fn benefits_from_input_partitioning(&self) -> Vec<bool> {
+        vec![false; self.children().len()]
+    }
+
+    fn supports_limit_pushdown(&self) -> bool {
+        true
+    }
+}

src/search/src/datafusion/distributed_plan/aggregate_topk_exec/sort.rs

@@ -0,0 +1,148 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{
+    pin::Pin,
+    sync::Arc,
+    task::{Context, Poll},
+};
+
+use arrow::{array::RecordBatch, compute::concat_batches, datatypes::SchemaRef};
+use config::utils::record_batch_ext::sort_record_batch_by_column;
+use datafusion::{
+    common::Result,
+    execution::{RecordBatchStream, SendableRecordBatchStream},
+};
+use futures::{Stream, StreamExt};
+
+pub struct TopKSortStream {
+    schema: SchemaRef,
+    stream: SendableRecordBatchStream,
+    sort_field: String,
+    descending: bool,
+    limit: u64,
+    cache_buf: Vec<RecordBatch>,
+}
+
+impl TopKSortStream {
+    pub fn new(
+        schema: SchemaRef,
+        stream: SendableRecordBatchStream,
+        sort_field: String,
+        descending: bool,
+        limit: u64,
+    ) -> Self {
+        Self {
+            schema,
+            stream,
+            sort_field,
+            descending,
+            limit,
+            cache_buf: Vec::new(),
+        }
+    }
+
+    fn topk_batch(&self, mut batches: Vec<RecordBatch>) -> Option<RecordBatch> {
+        if batches.is_empty() {
+            return None;
+        }
+        let mut topk_batch = batches.remove(0);
+        let schema = topk_batch.schema();
+        while !batches.is_empty() {
+            let next_batch = batches.remove(0);
+            if next_batch.num_rows() == 0 {
+                continue;
+            }
+            let new_batch = match concat_batches(&schema, vec![&topk_batch, &next_batch]) {
+                Ok(batch) => batch,
+                Err(e) => {
+                    log::error!("CacheTopkStream: concat_batches failed: {e}");
+                    continue;
+                }
+            };
+            match sort_record_batch_by_column(
+                new_batch,
+                &self.sort_field,
+                self.descending,
+                Some((self.limit as usize * 4).max(1000)),
+            ) {
+                Ok(batch) => {
+                    topk_batch = batch;
+                }
+                Err(e) => {
+                    log::error!("CacheTopkStream: sort_record_batch_by_column failed: {e}");
+                    continue;
+                }
+            };
+        }
+        Some(topk_batch)
+    }
+}
+
+impl Stream for TopKSortStream {
+    type Item = Result<RecordBatch>;
+
+    fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
+        match self.stream.poll_next_unpin(cx) {
+            Poll::Ready(Some(Ok(batch))) => {
+                let schema = batch.schema();
+                let empty_batch = RecordBatch::new_empty(schema);
+                match sort_record_batch_by_column(
+                    batch,
+                    &self.sort_field,
+                    self.descending,
+                    Some((self.limit as usize * 4).max(1000)),
+                ) {
+                    Ok(batch) => {
+                        self.cache_buf.push(batch);
+                    }
+                    Err(e) => {
+                        log::error!("CacheTopkStream: sort_record_batch_by_column failed: {e}");
+                    }
+                };
+                Poll::Ready(Some(Ok(empty_batch)))
+            }
+            Poll::Ready(None) => {
+                if self.cache_buf.is_empty() {
+                    return Poll::Ready(None);
+                }
+
+                // sort the cache_buf by the group_expr and return topK
+                let batches = std::mem::take(&mut self.cache_buf);
+                let topk_batch = self.topk_batch(batches);
+                // if let Some(batch) = topk_batch.as_ref() {
+                //     _ = arrow::util::pretty::print_batches(&[batch.clone()]);
+                // }
+                Poll::Ready(topk_batch.map(Ok))
+            }
+            Poll::Pending => Poll::Pending,
+            Poll::Ready(Some(Err(e))) => {
+                log::error!("Error in CacheTopkStream: {e}");
+                Poll::Ready(None)
+            }
+        }
+    }
+
+    fn size_hint(&self) -> (usize, Option<usize>) {
+        self.stream.size_hint()
+    }
+}
+
+impl RecordBatchStream for TopKSortStream {
+    /// Get the schema
+    fn schema(&self) -> SchemaRef {
+        Arc::clone(&self.schema)
+    }
+}

src/search/src/datafusion/distributed_plan/broadcast_join_exec.rs

@@ -0,0 +1,378 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{io::Cursor, pin::Pin, sync::Arc, task::Poll};
+
+use arrow::{array::RecordBatch, ipc::writer::FileWriter};
+use config::get_config;
+use datafusion::{
+    arrow::datatypes::SchemaRef,
+    common::{
+        Result, internal_err,
+        tree_node::{Transformed, TreeNode, TreeNodeRecursion, TreeNodeRewriter},
+    },
+    execution::{RecordBatchStream, SendableRecordBatchStream, TaskContext},
+    physical_expr::{EquivalenceProperties, Partitioning},
+    physical_plan::{
+        DisplayAs, DisplayFormatType, ExecutionPlan, ExecutionPlanProperties, PlanProperties,
+        execute_stream,
+        execution_plan::{Boundedness, EmissionType},
+        metrics::{BaselineMetrics, ExecutionPlanMetricsSet, MetricBuilder, MetricsSet},
+    },
+};
+use futures::{Stream, StreamExt};
+use futures_util::ready;
+
+use crate::datafusion::distributed_plan::{
+    once_async::{OnceAsync, OnceFut},
+    tmp_exec::TmpExec,
+};
+
+#[derive(Debug)]
+pub struct BroadcastJoinExec {
+    trace_id: String,
+    left: Arc<dyn ExecutionPlan>,
+    hash_join: Arc<dyn ExecutionPlan>,
+    cache: Arc<PlanProperties>,
+    metrics: ExecutionPlanMetricsSet,
+    // left table result store path in s3
+    cluster: String,
+    path: String,
+    // if left table is not large, directly send to follower node
+    left_data: OnceAsync<Option<Vec<u8>>>,
+}
+
+impl BroadcastJoinExec {
+    pub fn new(
+        trace_id: String,
+        left: Arc<dyn ExecutionPlan>,
+        hash_join: Arc<dyn ExecutionPlan>,
+        cluster: String,
+        path: String,
+    ) -> Self {
+        let schema = hash_join.schema();
+        let partition = hash_join.output_partitioning().partition_count();
+        let cache = Self::compute_properties(Arc::clone(&schema), partition);
+        BroadcastJoinExec {
+            trace_id,
+            left,
+            hash_join,
+            cache,
+            metrics: ExecutionPlanMetricsSet::new(),
+            cluster,
+            path,
+            left_data: OnceAsync::default(),
+        }
+    }
+
+    fn compute_properties(schema: SchemaRef, n_partitions: usize) -> Arc<PlanProperties> {
+        let eq_properties = EquivalenceProperties::new(schema);
+        let output_partitioning = Partitioning::UnknownPartitioning(n_partitions);
+        Arc::new(PlanProperties::new(
+            eq_properties,
+            output_partitioning,
+            EmissionType::Incremental,
+            Boundedness::Bounded,
+        ))
+    }
+}
+
+impl DisplayAs for BroadcastJoinExec {
+    fn fmt_as(&self, _t: DisplayFormatType, f: &mut std::fmt::Formatter) -> std::fmt::Result {
+        write!(
+            f,
+            "BroadcastJoinExec: cluster={}, path={}",
+            self.cluster, self.path
+        )
+    }
+}
+
+impl ExecutionPlan for BroadcastJoinExec {
+    fn name(&self) -> &'static str {
+        "BroadcastJoinExec"
+    }
+
+    fn properties(&self) -> &Arc<PlanProperties> {
+        &self.cache
+    }
+
+    fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>> {
+        vec![&self.left, &self.hash_join]
+    }
+
+    fn with_new_children(
+        self: Arc<Self>,
+        children: Vec<Arc<dyn ExecutionPlan>>,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        if children.len() != 2 {
+            return internal_err!("BroadcastJoinExec should have 2 children");
+        }
+        let left = children[0].clone();
+        let hash_join = children[1].clone();
+        Ok(Arc::new(BroadcastJoinExec::new(
+            self.trace_id.clone(),
+            left,
+            hash_join,
+            self.cluster.clone(),
+            self.path.clone(),
+        )))
+    }
+
+    fn execute(
+        &self,
+        partition: usize,
+        context: Arc<TaskContext>,
+    ) -> Result<SendableRecordBatchStream> {
+        let trace_id = self.trace_id.clone();
+        let left_schema = self.left.schema().clone();
+        let path = self.path.clone();
+        let metrics = self.metrics.clone();
+        let left_data = self.left_data.try_once(|| {
+            let left_stream = execute_stream(self.left.clone(), context.clone())?;
+            Ok(collect_left_data(
+                trace_id,
+                left_stream,
+                left_schema,
+                path,
+                metrics,
+            ))
+        })?;
+
+        let metrics = BaselineMetrics::new(&self.metrics, partition);
+        Ok(Box::pin(BroadcastJoinStream::new(
+            self.hash_join.schema().clone(),
+            left_data,
+            self.hash_join.clone(),
+            partition,
+            context,
+            metrics,
+        )))
+    }
+
+    fn metrics(&self) -> Option<MetricsSet> {
+        Some(self.metrics.clone_inner())
+    }
+}
+
+async fn collect_left_data(
+    trace_id: String,
+    mut stream: SendableRecordBatchStream,
+    schema: SchemaRef,
+    path: String,
+    metrics: ExecutionPlanMetricsSet,
+) -> Result<Option<Vec<u8>>> {
+    // 1. collect all left data
+    let collect_left_time = MetricBuilder::new(&metrics).subset_time("collect_left_time", 0);
+    let timer = collect_left_time.timer();
+    let mut batches = Vec::new();
+    while let Some(batch) = stream.next().await.transpose()? {
+        batches.push(batch);
+    }
+    timer.done();
+    log::info!(
+        "[trace_id {trace_id}] BroadcastJoinExec: collect left data took: {} ms",
+        std::time::Duration::from_nanos(collect_left_time.value() as u64).as_millis()
+    );
+
+    // 2. convert record batch to bytes
+    let convert_time = MetricBuilder::new(&metrics).subset_time("convert_time", 0);
+    let timer = convert_time.timer();
+    let mut buffer = Cursor::new(Vec::new());
+    let mut writer = FileWriter::try_new(&mut buffer, &schema)?;
+    for batch in batches {
+        writer.write(&batch)?;
+    }
+    writer.finish()?;
+    let buf = buffer.into_inner();
+    timer.done();
+    log::info!(
+        "[trace_id {trace_id}] BroadcastJoinExec: convert record batch to bytes took: {} ms",
+        std::time::Duration::from_nanos(convert_time.value() as u64).as_millis()
+    );
+
+    // 3. if left data is too large, save to s3, otherwise return bytes
+    if buf.len()
+        > get_config()
+            .common
+            .feature_broadcast_join_left_side_max_size
+            * 1024
+            * 1024
+    {
+        log::info!(
+            "[trace_id {trace_id}] BroadcastJoinExec: left data is too large, save to s3, size: {} MB",
+            buf.len() as f64 / 1024.0 / 1024.0
+        );
+        infra::storage::put("", &path, buf.into()).await?;
+        Ok(None)
+    } else {
+        log::info!(
+            "[trace_id {trace_id}] BroadcastJoinExec: left data is not large, save to memory, size: {} MB",
+            buf.len() as f64 / 1024.0 / 1024.0
+        );
+        Ok(Some(buf))
+    }
+}
+
+impl Drop for BroadcastJoinExec {
+    fn drop(&mut self) {
+        let path = self.path.clone();
+        tokio::task::spawn(async move {
+            if let Err(e) = infra::storage::del(vec![("", &path)]).await {
+                log::error!("[BroadcastJoinExec] Failed to delete left data, path: {path}: {e}");
+            }
+        });
+    }
+}
+
+#[derive(Debug, Clone)]
+pub(super) enum BroadcastJoinStreamState {
+    WaitBuildSide,
+    ProcessProbeBatch,
+    Completed,
+}
+
+struct BroadcastJoinStream {
+    schema: SchemaRef,
+    left_data: OnceFut<Option<Vec<u8>>>,
+    hash_join: Arc<dyn ExecutionPlan>,
+    partition: usize,
+    context: Arc<TaskContext>,
+    right_stream: Option<SendableRecordBatchStream>,
+    state: BroadcastJoinStreamState,
+    metrics: BaselineMetrics,
+}
+
+impl BroadcastJoinStream {
+    pub fn new(
+        schema: SchemaRef,
+        left_data: OnceFut<Option<Vec<u8>>>,
+        hash_join: Arc<dyn ExecutionPlan>,
+        partition: usize,
+        context: Arc<TaskContext>,
+        metrics: BaselineMetrics,
+    ) -> Self {
+        Self {
+            schema,
+            left_data,
+            hash_join,
+            partition,
+            context,
+            right_stream: None,
+            state: BroadcastJoinStreamState::WaitBuildSide,
+            metrics,
+        }
+    }
+
+    fn poll_next_inner(
+        &mut self,
+        cx: &mut std::task::Context<'_>,
+    ) -> Poll<Option<Result<RecordBatch>>> {
+        match &mut self.state {
+            BroadcastJoinStreamState::WaitBuildSide => self.handle_wait_build_side(cx),
+            BroadcastJoinStreamState::ProcessProbeBatch => self.handle_process_probe_batch(cx),
+            BroadcastJoinStreamState::Completed => Poll::Ready(None),
+        }
+    }
+
+    fn handle_wait_build_side(
+        &mut self,
+        cx: &mut std::task::Context<'_>,
+    ) -> Poll<Option<Result<RecordBatch>>> {
+        let left_data = ready!(self.left_data.get_shared(cx))?;
+
+        let hash_join = if let Some(left_data) = left_data.as_ref().clone() {
+            let hash_join = self.hash_join.clone();
+            let mut rewriter = TmpExecRewriter::new(left_data);
+            hash_join.rewrite(&mut rewriter)?.data
+        } else {
+            self.hash_join.clone()
+        };
+
+        match hash_join.execute(self.partition, self.context.clone()) {
+            Ok(right_stream) => {
+                self.right_stream = Some(right_stream);
+                self.state = BroadcastJoinStreamState::ProcessProbeBatch;
+                Poll::Ready(Some(Ok(RecordBatch::new_empty(self.schema.clone()))))
+            }
+            Err(e) => Poll::Ready(Some(Err(e))),
+        }
+    }
+
+    fn handle_process_probe_batch(
+        &mut self,
+        cx: &mut std::task::Context<'_>,
+    ) -> Poll<Option<Result<RecordBatch>>> {
+        if let Some(ref mut right_stream) = self.right_stream {
+            let res = ready!(Pin::new(right_stream).poll_next(cx));
+            match res {
+                Some(Ok(batch)) => {
+                    self.metrics.record_output(batch.num_rows());
+                    Poll::Ready(Some(Ok(batch)))
+                }
+                Some(Err(e)) => Poll::Ready(Some(Err(e))),
+                None => {
+                    self.state = BroadcastJoinStreamState::Completed;
+                    Poll::Ready(None)
+                }
+            }
+        } else {
+            // This should not happen as we set right_stream in handle_wait_build_side
+            Poll::Ready(Some(Err(datafusion::common::DataFusionError::Internal(
+                "Right stream not initialized".to_string(),
+            ))))
+        }
+    }
+}
+
+impl Stream for BroadcastJoinStream {
+    type Item = Result<RecordBatch>;
+
+    fn poll_next(
+        mut self: Pin<&mut Self>,
+        cx: &mut std::task::Context<'_>,
+    ) -> Poll<Option<Result<RecordBatch>>> {
+        self.poll_next_inner(cx)
+    }
+}
+
+impl RecordBatchStream for BroadcastJoinStream {
+    fn schema(&self) -> SchemaRef {
+        self.schema.clone()
+    }
+}
+
+#[derive(Debug)]
+struct TmpExecRewriter {
+    data: Vec<u8>,
+}
+
+impl TmpExecRewriter {
+    fn new(data: Vec<u8>) -> Self {
+        TmpExecRewriter { data }
+    }
+}
+
+impl TreeNodeRewriter for TmpExecRewriter {
+    type Node = Arc<dyn ExecutionPlan>;
+
+    fn f_up(&mut self, node: Arc<dyn ExecutionPlan>) -> Result<Transformed<Self::Node>> {
+        if let Some(tmp_exec) = node.downcast_ref::<TmpExec>() {
+            let tmp =
+                Arc::new(tmp_exec.clone().set_data(self.data.clone())) as Arc<dyn ExecutionPlan>;
+            return Ok(Transformed::new(tmp, true, TreeNodeRecursion::Stop));
+        }
+        Ok(Transformed::no(node))
+    }
+}

src/search/src/datafusion/distributed_plan/cache_buf.rs

@@ -0,0 +1,187 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{fmt::Debug, sync::Arc};
+
+use arrow::array::RecordBatch;
+use datafusion::{common::Result, physical_plan::aggregates::AggregateExec};
+use rayon::iter::{IntoParallelIterator, ParallelIterator};
+
+use crate::datafusion::aggregates::{
+    merge_phase::GroupedHashAggregateStream, no_grouping_merge_phase::AggregateStream,
+};
+
+pub(crate) struct CacheStream {
+    mode: CacheStreamMode,
+    target_partitions: usize,
+    aggregate_plan: Arc<AggregateExec>,
+    data: Vec<Arc<RecordBatch>>,
+}
+
+#[derive(Debug, Clone, Copy)]
+pub(crate) enum CacheStreamMode {
+    Group,
+    NoGroup,
+}
+
+impl CacheStream {
+    fn is_empty(&self) -> bool {
+        self.data.is_empty()
+    }
+
+    pub(crate) fn new(
+        has_group_by: bool,
+        target_partitions: usize,
+        aggregate_plan: Arc<AggregateExec>,
+    ) -> Self {
+        Self {
+            mode: if has_group_by {
+                CacheStreamMode::Group
+            } else {
+                CacheStreamMode::NoGroup
+            },
+            target_partitions,
+            aggregate_plan,
+            data: Vec::new(),
+        }
+    }
+}
+
+pub(crate) struct CacheBuf {
+    pub(crate) total_partition_num: usize,
+    pub(crate) cached_partition_num: usize,
+    pub(crate) cached_buf: CacheStream,
+}
+
+impl Debug for CacheBuf {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "CacheBuf")
+    }
+}
+
+impl CacheBuf {
+    pub(crate) fn append_data(&mut self, record_batch: Arc<RecordBatch>) {
+        self.cached_buf.data.push(record_batch);
+    }
+
+    pub(crate) fn check_and_add_partition(&mut self) -> bool {
+        self.cached_partition_num += 1;
+        if self.cached_partition_num >= self.total_partition_num {
+            return true;
+        }
+        false
+    }
+
+    pub(crate) fn get_final_result(&mut self, stream_id: &str) -> Result<Vec<RecordBatch>> {
+        if self.cached_buf.is_empty() {
+            return Ok(Vec::new());
+        }
+
+        let merge_mode = self.cached_buf.mode;
+        let start = std::time::Instant::now();
+        let record_batchs = std::mem::take(&mut self.cached_buf.data);
+        let record_batchs: Vec<Arc<RecordBatch>> = record_batchs
+            .into_iter()
+            .filter(|batch| batch.num_rows() != 0)
+            .collect();
+        let total_batch_len = record_batchs.len();
+        let partition_num = std::cmp::max(2, self.cached_buf.target_partitions);
+
+        // When partial_reduce is enabled each follower has already sent pre-merged data, so
+        // phase-1 (parallel chunked aggregation) would double-aggregate already-reduced values.
+        let partial_reduce_enabled = config::get_config().common.feature_partial_reduce_enabled;
+
+        let mut merged_batches: Vec<RecordBatch> = if partial_reduce_enabled {
+            // Phase 1 skipped — use follower results directly.
+            record_batchs
+                .into_iter()
+                .map(|b| b.as_ref().clone())
+                .collect()
+        } else {
+            let thread_pool = rayon::ThreadPoolBuilder::new()
+                .num_threads(partition_num)
+                .build()
+                .unwrap();
+
+            let chunk_size = std::cmp::max(1, total_batch_len / partition_num);
+            let batch_chunks: Vec<Vec<Arc<RecordBatch>>> = record_batchs
+                .chunks(chunk_size)
+                .map(|chunk| chunk.to_vec())
+                .collect();
+
+            // Phase 1: process batch_chunks in parallel using rayon
+            let partial_results: Vec<Result<Vec<RecordBatch>>> = thread_pool.install(|| {
+                batch_chunks
+                    .into_par_iter()
+                    .map(|batches| match merge_mode {
+                        CacheStreamMode::Group => {
+                            let mut stream =
+                                GroupedHashAggregateStream::new(&self.cached_buf.aggregate_plan)
+                                    .unwrap();
+                            for batch in batches {
+                                stream.group_aggregate_batch(batch.as_ref().clone())?;
+                            }
+                            stream.get_final_result()
+                        }
+                        CacheStreamMode::NoGroup => {
+                            let mut stream =
+                                AggregateStream::new(&self.cached_buf.aggregate_plan).unwrap();
+                            for batch in batches {
+                                stream.aggregate_batch(batch.as_ref().clone())?;
+                            }
+                            stream.finalize_aggregation()
+                        }
+                    })
+                    .collect()
+            });
+
+            let mut batches = Vec::new();
+            for partial_result in partial_results {
+                batches.extend(partial_result?);
+            }
+            batches
+        };
+
+        // Phase 2: final merge (always needed when there are multiple batches to combine)
+        match merge_mode {
+            CacheStreamMode::Group => {
+                let mut final_stream =
+                    GroupedHashAggregateStream::new(&self.cached_buf.aggregate_plan).unwrap();
+                for batch in merged_batches {
+                    final_stream.group_aggregate_batch(batch)?;
+                }
+                merged_batches = final_stream.get_final_result()?;
+            }
+            CacheStreamMode::NoGroup => {
+                let mut final_stream =
+                    AggregateStream::new(&self.cached_buf.aggregate_plan).unwrap();
+                for batch in merged_batches {
+                    final_stream.aggregate_batch(batch)?;
+                }
+                merged_batches = final_stream.finalize_aggregation()?;
+            }
+        }
+
+        log::info!(
+            "[StreamingAggs streaming_id: {stream_id}] merge_agg_batches from {total_batch_len} to {}, partial_reduce_enabled: {partial_reduce_enabled} unique_numbers: {}, partition_num: {partition_num}, chunk_size: {}, total_merge_times: {} ms",
+            merged_batches.len(),
+            merged_batches.iter().map(|b| b.num_rows()).sum::<usize>(),
+            std::cmp::max(1, total_batch_len / partition_num),
+            start.elapsed().as_millis(),
+        );
+
+        Ok(merged_batches)
+    }
+}

src/search/src/datafusion/distributed_plan/codec/aggregate_topk_exec.rs

@@ -21,11 +21,11 @@ use datafusion::{
     execution::FunctionRegistry,
     physical_plan::ExecutionPlan,
 };
-#[cfg(feature = "enterprise")]
-use o2_enterprise::enterprise::search::datafusion::distributed_plan::agg_topk_exec::AggregateTopkExec;
 use prost::Message;
 use proto::cluster_rpc;
 
+use crate::datafusion::distributed_plan::aggregate_topk_exec::AggregateTopkExec;
+
 pub fn try_decode(
     node: cluster_rpc::AggregateTopkExecNode,
     inputs: &[Arc<dyn ExecutionPlan>],

src/search/src/datafusion/distributed_plan/codec/mod.rs

@@ -20,16 +20,12 @@ use datafusion::{
 };
 use datafusion_proto::physical_plan::PhysicalExtensionCodec;
 
-#[cfg(feature = "enterprise")]
 mod aggregate_topk_exec;
 mod deduplication_exec;
 mod empty_exec;
-#[cfg(feature = "enterprise")]
 mod enrichment_exec;
 mod physical_plan_node;
-#[cfg(feature = "enterprise")]
 mod streaming_aggs_exec;
-#[cfg(feature = "enterprise")]
 mod tmp_exec;
 
 pub fn get_physical_extension_codec() -> ComposedPhysicalExtensionCodec {

src/search/src/datafusion/distributed_plan/codec/physical_plan_node.rs

@@ -24,17 +24,14 @@ use datafusion::{
 use datafusion_proto::physical_plan::PhysicalExtensionCodec;
 use prost::Message;
 use proto::cluster_rpc;
-#[cfg(feature = "enterprise")]
-use {
-    crate::datafusion::distributed_plan::enrichment_exec::EnrichmentExec,
-    o2_enterprise::enterprise::search::datafusion::distributed_plan::{
-        agg_topk_exec::AggregateTopkExec, streaming_aggs_exec::exec::StreamingAggsExec,
-        tmp_exec::TmpExec,
-    },
-};
 
 use crate::datafusion::{
-    distributed_plan::empty_exec::NewEmptyExec, plan::deduplication_exec::DeduplicationExec,
+    distributed_plan::{
+        aggregate_topk_exec::AggregateTopkExec, empty_exec::NewEmptyExec,
+        enrichment_exec::EnrichmentExec, streaming_aggs_exec::exec::StreamingAggsExec,
+        tmp_exec::TmpExec,
+    },
+    plan::deduplication_exec::DeduplicationExec,
 };
 
 /// A PhysicalExtensionCodec that can serialize and deserialize ChildExec
@@ -60,26 +57,18 @@ impl PhysicalExtensionCodec for PhysicalPlanNodePhysicalExtensionCodec {
             Some(cluster_rpc::physical_plan_node::Plan::DeduplicationExec(node)) => {
                 super::deduplication_exec::try_decode(node, inputs, ctx)
             }
-            #[cfg(feature = "enterprise")]
             Some(cluster_rpc::physical_plan_node::Plan::AggregateTopk(node)) => {
                 super::aggregate_topk_exec::try_decode(node, inputs, ctx)
             }
-            #[cfg(feature = "enterprise")]
             Some(cluster_rpc::physical_plan_node::Plan::StreamingAggs(node)) => {
                 super::streaming_aggs_exec::try_decode(node, inputs, ctx)
             }
-            #[cfg(feature = "enterprise")]
             Some(cluster_rpc::physical_plan_node::Plan::TmpExec(node)) => {
                 super::tmp_exec::try_decode(node, inputs, ctx)
             }
-            #[cfg(feature = "enterprise")]
             Some(cluster_rpc::physical_plan_node::Plan::EnrichmentExec(node)) => {
                 super::enrichment_exec::try_decode(node, inputs, ctx)
             }
-            #[cfg(not(feature = "enterprise"))]
-            Some(_) => {
-                internal_err!("Not supported")
-            }
             None => {
                 internal_err!("PhysicalPlanNode is required")
             }
@@ -87,7 +76,6 @@ impl PhysicalExtensionCodec for PhysicalPlanNodePhysicalExtensionCodec {
     }
 
     fn try_encode(&self, node: Arc<dyn ExecutionPlan>, buf: &mut Vec<u8>) -> Result<()> {
-        #[cfg(feature = "enterprise")]
         if node.downcast_ref::<NewEmptyExec>().is_some() {
             super::empty_exec::try_encode(node, buf)
         } else if node.downcast_ref::<DeduplicationExec>().is_some() {
@@ -103,14 +91,6 @@ impl PhysicalExtensionCodec for PhysicalPlanNodePhysicalExtensionCodec {
         } else {
             internal_err!("Not supported")
         }
-        #[cfg(not(feature = "enterprise"))]
-        if node.downcast_ref::<NewEmptyExec>().is_some() {
-            super::empty_exec::try_encode(node, buf)
-        } else if node.downcast_ref::<DeduplicationExec>().is_some() {
-            super::deduplication_exec::try_encode(node, buf)
-        } else {
-            internal_err!("Not supported")
-        }
     }
 }
 

src/search/src/datafusion/distributed_plan/codec/streaming_aggs_exec.rs

@@ -22,10 +22,11 @@ use datafusion::{
     physical_plan::{ExecutionPlan, aggregates::AggregateExec},
 };
 use datafusion_proto::{physical_plan::AsExecutionPlan, protobuf::PhysicalPlanNode};
-use o2_enterprise::enterprise::search::datafusion::distributed_plan::streaming_aggs_exec::exec::StreamingAggsExec;
 use prost::Message;
 use proto::cluster_rpc;
 
+use crate::datafusion::distributed_plan::streaming_aggs_exec::exec::StreamingAggsExec;
+
 pub fn try_decode(
     node: cluster_rpc::StreamingAggsExecNode,
     inputs: &[Arc<dyn ExecutionPlan>],
@@ -116,7 +117,6 @@ mod tests {
     use datafusion_proto::bytes::{
         physical_plan_from_bytes_with_extension_codec, physical_plan_to_bytes_with_extension_codec,
     };
-    use o2_enterprise::enterprise::search::datafusion::distributed_plan::streaming_aggs_exec::exec::StreamingAggsExec;
 
     use super::*;
     use crate::datafusion::udf::str_match_udf::STR_MATCH_UDF;

src/search/src/datafusion/distributed_plan/codec/tmp_exec.rs

@@ -22,10 +22,11 @@ use datafusion::{
     physical_plan::ExecutionPlan,
 };
 use datafusion_proto::{convert_required, protobuf::proto_error};
-use o2_enterprise::enterprise::search::datafusion::distributed_plan::tmp_exec::TmpExec;
 use prost::Message;
 use proto::cluster_rpc;
 
+use crate::datafusion::distributed_plan::tmp_exec::TmpExec;
+
 pub fn try_decode(
     node: cluster_rpc::TmpExecNode,
     _inputs: &[Arc<dyn ExecutionPlan>],

src/search/src/datafusion/distributed_plan/metadata_count_exec.rs

@@ -0,0 +1,185 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int64Array, RecordBatch, UInt64Array};
+use datafusion::{
+    arrow::datatypes::{DataType, SchemaRef},
+    common::{Result, internal_err},
+    execution::{SendableRecordBatchStream, TaskContext},
+    physical_expr::{EquivalenceProperties, Partitioning},
+    physical_plan::{
+        DisplayAs, DisplayFormatType, ExecutionPlan, PlanProperties,
+        execution_plan::{Boundedness, EmissionType},
+        memory::MemoryStream,
+    },
+};
+
+#[derive(Debug)]
+pub struct MetadataCountExec {
+    schema: SchemaRef,
+    records: i64,
+    files: usize,
+    cache: Arc<PlanProperties>,
+}
+
+impl MetadataCountExec {
+    pub fn new(schema: SchemaRef, records: i64, files: usize) -> Self {
+        let cache = Arc::new(PlanProperties::new(
+            EquivalenceProperties::new(schema.clone()),
+            Partitioning::UnknownPartitioning(1),
+            EmissionType::Final,
+            Boundedness::Bounded,
+        ));
+        Self {
+            schema,
+            records,
+            files,
+            cache,
+        }
+    }
+
+    fn data(&self) -> Result<Vec<RecordBatch>> {
+        if self.schema.fields().len() != 1 {
+            return internal_err!(
+                "MetadataCountExec expected one count field, got {}",
+                self.schema.fields().len()
+            );
+        }
+
+        let records = self.records.max(0);
+        let array: ArrayRef = match self.schema.field(0).data_type() {
+            DataType::Int64 => Arc::new(Int64Array::from(vec![records])),
+            DataType::UInt64 => Arc::new(UInt64Array::from(vec![records as u64])),
+            other => {
+                return internal_err!("MetadataCountExec unsupported count type: {other:?}");
+            }
+        };
+
+        RecordBatch::try_new(self.schema.clone(), vec![array])
+            .map(|batch| vec![batch])
+            .map_err(|e| datafusion::error::DataFusionError::Internal(e.to_string()))
+    }
+}
+
+impl DisplayAs for MetadataCountExec {
+    fn fmt_as(&self, _t: DisplayFormatType, f: &mut std::fmt::Formatter) -> std::fmt::Result {
+        write!(
+            f,
+            "MetadataCountExec: files: {}, records: {}",
+            self.files, self.records
+        )
+    }
+}
+
+impl ExecutionPlan for MetadataCountExec {
+    fn name(&self) -> &'static str {
+        "MetadataCountExec"
+    }
+
+    fn properties(&self) -> &Arc<PlanProperties> {
+        &self.cache
+    }
+
+    fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>> {
+        vec![]
+    }
+
+    fn with_new_children(
+        self: Arc<Self>,
+        _: Vec<Arc<dyn ExecutionPlan>>,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        Ok(self)
+    }
+
+    fn execute(
+        &self,
+        partition: usize,
+        _context: Arc<TaskContext>,
+    ) -> Result<SendableRecordBatchStream> {
+        if partition >= 1 {
+            return internal_err!(
+                "MetadataCountExec invalid partition {partition} (expected partition: 0)"
+            );
+        }
+
+        Ok(Box::pin(MemoryStream::try_new(
+            self.data()?,
+            self.schema.clone(),
+            None,
+        )?))
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::sync::Arc;
+
+    use datafusion::{
+        arrow::datatypes::{DataType, Field, Schema},
+        physical_plan::ExecutionPlan,
+    };
+
+    use super::*;
+
+    #[test]
+    fn test_metadata_count_exec_creates_single_count_row() {
+        let schema = Arc::new(Schema::new(vec![Field::new(
+            "count",
+            DataType::Int64,
+            false,
+        )]));
+        let exec = MetadataCountExec::new(schema, 42, 3);
+        let batches = exec.data().unwrap();
+
+        assert_eq!(batches.len(), 1);
+        assert_eq!(batches[0].num_rows(), 1);
+        let values = batches[0]
+            .column(0)
+            .as_any()
+            .downcast_ref::<Int64Array>()
+            .unwrap();
+        assert_eq!(values.value(0), 42);
+    }
+
+    #[test]
+    fn test_metadata_count_exec_plan_name() {
+        let schema = Arc::new(Schema::new(vec![Field::new(
+            "count",
+            DataType::UInt64,
+            false,
+        )]));
+        let exec = MetadataCountExec::new(schema, 7, 2);
+
+        assert_eq!(ExecutionPlan::name(&exec), "MetadataCountExec");
+    }
+
+    #[test]
+    fn test_metadata_count_exec_display_includes_files_and_records() {
+        let schema = Arc::new(Schema::new(vec![Field::new(
+            "count",
+            DataType::UInt64,
+            false,
+        )]));
+        let exec = Arc::new(MetadataCountExec::new(schema, 7, 2));
+
+        let display = format!(
+            "{}",
+            datafusion::physical_plan::displayable(exec.as_ref()).indent(false)
+        );
+        assert!(display.contains("MetadataCountExec: files: 2, records: 7"));
+    }
+}

src/search/src/datafusion/distributed_plan/mod.rs

@@ -25,6 +25,9 @@ use datafusion::{
 
 use crate::datafusion::distributed_plan::empty_exec::NewEmptyExec;
 
+pub mod aggregate_topk_exec;
+pub mod broadcast_join_exec;
+pub(crate) mod cache_buf;
 pub mod codec;
 mod common;
 mod decoder_stream;
@@ -32,11 +35,14 @@ pub mod display;
 pub mod distribute_analyze_exec;
 pub mod empty_exec;
 pub mod enrich_exec;
-#[cfg(feature = "enterprise")]
 pub mod enrichment_exec;
+pub mod metadata_count_exec;
 pub mod node;
+mod once_async;
 pub mod remote_scan_exec;
 pub mod rewrite;
+pub mod streaming_aggs_exec;
+pub mod tmp_exec;
 
 mod utils;
 

src/search/src/datafusion/distributed_plan/once_async.rs

@@ -0,0 +1,144 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{
+    fmt,
+    sync::Arc,
+    task::{Context, Poll},
+};
+
+use datafusion::{
+    common::Result,
+    error::{DataFusionError, SharedResult},
+};
+use futures::{
+    FutureExt,
+    future::{BoxFuture, Shared},
+};
+use futures_util::ready;
+use parking_lot::Mutex;
+
+/// refer: https://github.com/apache/datafusion/blob/351675ddc27c42684a079b3a89fe2dee581d89a2/datafusion/physical-plan/src/joins/utils.rs#L336
+/// A [`OnceAsync`] runs an `async` closure once, where multiple calls to
+/// [`OnceAsync::try_once`] return a [`OnceFut`] that resolves to the result of the
+/// same computation.
+///
+/// This is useful for joins where the results of one child are needed to proceed
+/// with multiple output stream
+///
+/// For example, in a hash join, one input is buffered and shared across
+/// potentially multiple output partitions. Each output partition must wait for
+/// the hash table to be built before proceeding.
+///
+/// Each output partition waits on the same `OnceAsync` before proceeding.
+pub(crate) struct OnceAsync<T> {
+    fut: Mutex<Option<SharedResult<OnceFut<T>>>>,
+}
+
+impl<T> Default for OnceAsync<T> {
+    fn default() -> Self {
+        Self {
+            fut: Mutex::new(None),
+        }
+    }
+}
+
+impl<T> fmt::Debug for OnceAsync<T> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        write!(f, "OnceAsync")
+    }
+}
+
+impl<T: 'static> OnceAsync<T> {
+    /// If this is the first call to this function on this object, will invoke
+    /// `f` to obtain a future and return a [`OnceFut`] referring to this. `f`
+    /// may fail, in which case its error is returned.
+    ///
+    /// If this is not the first call, will return a [`OnceFut`] referring
+    /// to the same future as was returned by the first call - or the same
+    /// error if the initial call to `f` failed.
+    pub(crate) fn try_once<F, Fut>(&self, f: F) -> Result<OnceFut<T>>
+    where
+        F: FnOnce() -> Result<Fut>,
+        Fut: Future<Output = Result<T>> + Send + 'static,
+    {
+        self.fut
+            .lock()
+            .get_or_insert_with(|| f().map(OnceFut::new).map_err(Arc::new))
+            .clone()
+            .map_err(DataFusionError::Shared)
+    }
+}
+
+/// A [`OnceFut`] represents a shared asynchronous computation, that will be evaluated
+/// once for all [`Clone`]'s, with [`OnceFut::get`] providing a non-consuming interface
+/// to drive the underlying [`Future`] to completion
+pub(crate) struct OnceFut<T> {
+    state: OnceFutState<T>,
+}
+
+impl<T> Clone for OnceFut<T> {
+    fn clone(&self) -> Self {
+        Self {
+            state: self.state.clone(),
+        }
+    }
+}
+
+enum OnceFutState<T> {
+    Pending(OnceFutPending<T>),
+    Ready(SharedResult<Arc<T>>),
+}
+
+impl<T> Clone for OnceFutState<T> {
+    fn clone(&self) -> Self {
+        match self {
+            Self::Pending(p) => Self::Pending(p.clone()),
+            Self::Ready(r) => Self::Ready(r.clone()),
+        }
+    }
+}
+
+impl<T: 'static> OnceFut<T> {
+    /// Create a new [`OnceFut`] from a [`Future`]
+    pub(crate) fn new<Fut>(fut: Fut) -> Self
+    where
+        Fut: Future<Output = Result<T>> + Send + 'static,
+    {
+        Self {
+            state: OnceFutState::Pending(
+                fut.map(|res| res.map(Arc::new).map_err(Arc::new))
+                    .boxed()
+                    .shared(),
+            ),
+        }
+    }
+
+    /// Get shared reference to the result of the computation if it is ready, without consuming it
+    pub(crate) fn get_shared(&mut self, cx: &mut Context<'_>) -> Poll<Result<Arc<T>>> {
+        if let OnceFutState::Pending(fut) = &mut self.state {
+            let r = ready!(fut.poll_unpin(cx));
+            self.state = OnceFutState::Ready(r);
+        }
+
+        match &self.state {
+            OnceFutState::Pending(_) => unreachable!(),
+            OnceFutState::Ready(r) => Poll::Ready(r.clone().map_err(DataFusionError::Shared)),
+        }
+    }
+}
+
+/// The shared future type used internally within [`OnceAsync`]
+type OnceFutPending<T> = Shared<BoxFuture<'static, SharedResult<Arc<T>>>>;

src/search/src/datafusion/distributed_plan/rewrite.rs

@@ -28,11 +28,13 @@ use datafusion::{
         union::UnionExec,
     },
 };
-#[cfg(feature = "enterprise")]
-use o2_enterprise::enterprise::search::datafusion::distributed_plan::metadata_count_exec::MetadataCountExec;
 
 use crate::{
-    datafusion::plan::tantivy_optimize_exec::TantivyOptimizeExec, index::IndexCondition,
+    datafusion::{
+        distributed_plan::metadata_count_exec::MetadataCountExec,
+        plan::tantivy_optimize_exec::TantivyOptimizeExec,
+    },
+    index::IndexCondition,
     types::QueryParams,
 };
 
@@ -68,9 +70,7 @@ pub struct AggregateOptimizeRewriter {
     file_list: Vec<FileKey>,
     index_condition: Option<IndexCondition>,
     index_optimize_mode: Option<IndexOptimizeMode>,
-    #[allow(unused)]
     metadata_records: i64,
-    #[allow(unused)]
     metadata_files: usize,
 }
 
@@ -105,7 +105,6 @@ impl AggregateOptimizeRewriter {
         ))
     }
 
-    #[cfg(feature = "enterprise")]
     fn metadata_count_exec(&self, schema: SchemaRef) -> Result<Arc<dyn ExecutionPlan>> {
         Ok(Arc::new(MetadataCountExec::new(
             schema,
@@ -117,7 +116,6 @@ impl AggregateOptimizeRewriter {
     fn additional_inputs(&mut self, schema: SchemaRef) -> Result<Vec<Arc<dyn ExecutionPlan>>> {
         let mut inputs = Vec::new();
 
-        #[cfg(feature = "enterprise")]
         if self.metadata_records > 0 {
             inputs.push(self.metadata_count_exec(schema.clone())?);
         }
@@ -153,9 +151,7 @@ mod tests {
     use std::sync::Arc;
 
     use arrow_schema::{DataType, Field, Schema};
-    #[cfg(feature = "enterprise")]
-    use config::meta::stream::FileMeta;
-    use config::meta::stream::{FileKey, StreamType};
+    use config::meta::stream::{FileKey, FileMeta, StreamType};
     use datafusion::{
         common::Result,
         functions_aggregate::count::count_udaf,
@@ -208,7 +204,6 @@ mod tests {
         )?))
     }
 
-    #[cfg(feature = "enterprise")]
     #[test]
     fn test_aggregate_optimize_rewrite_combines_metadata_and_tantivy_inputs() -> Result<()> {
         let plan = partial_count_exec()?;

src/search/src/datafusion/distributed_plan/streaming_aggs_exec/cached_file_stream.rs

@@ -0,0 +1,162 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{
+    pin::Pin,
+    sync::Arc,
+    task::{Context, Poll},
+};
+
+use arrow::{array::RecordBatch, datatypes::SchemaRef};
+use datafusion::{common::Result, execution::RecordBatchStream};
+use futures::Stream;
+
+use crate::cache::streaming_agg::get_record_batches;
+
+pub(crate) struct CachedFileStream {
+    id: String,
+    cached_files: Vec<Arc<String>>,
+    schema: SchemaRef,
+    current_file_index: usize,
+    current_batches: Vec<RecordBatch>,
+    current_batch_index: usize,
+    is_exhausted: bool,
+}
+
+impl CachedFileStream {
+    pub(crate) fn new(id: String, cached_files: Vec<Arc<String>>, schema: SchemaRef) -> Self {
+        Self {
+            id,
+            cached_files,
+            schema,
+            current_file_index: 0,
+            current_batches: Vec::new(),
+            current_batch_index: 0,
+            is_exhausted: false,
+        }
+    }
+
+    pub(crate) fn load_next_file(&mut self) -> Result<()> {
+        loop {
+            if self.current_file_index >= self.cached_files.len() {
+                self.is_exhausted = true;
+                return Ok(());
+            }
+
+            let file_path = &self.cached_files[self.current_file_index];
+            let batches = match get_record_batches(&self.id, file_path, self.schema.clone()) {
+                Ok(batches) => batches,
+                Err(e) => {
+                    log::error!(
+                        "[StreamingAggs streaming_id: {}] Error reading cached file: {file_path}, error: {e:?}",
+                        self.id,
+                    );
+                    return Err(e.into());
+                }
+            };
+
+            log::debug!(
+                "[StreamingAggs streaming_id: {}] Successfully read {} batches from cached file: {file_path}",
+                self.id,
+                batches.len(),
+            );
+
+            self.current_batches = batches;
+            self.current_batch_index = 0;
+            self.current_file_index += 1;
+
+            if self.current_batches.is_empty() {
+                continue;
+            }
+
+            break;
+        }
+        Ok(())
+    }
+}
+
+impl Stream for CachedFileStream {
+    type Item = Result<RecordBatch>;
+
+    fn poll_next(mut self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
+        if self.is_exhausted {
+            return Poll::Ready(None);
+        }
+
+        // If we don't have current batches or we've exhausted them, load next file
+        if self.current_batches.is_empty() || self.current_batch_index >= self.current_batches.len()
+        {
+            if let Err(e) = self.load_next_file() {
+                return Poll::Ready(Some(Err(e)));
+            }
+
+            if self.is_exhausted {
+                return Poll::Ready(None);
+            }
+        }
+
+        // Return the next batch if available
+        if self.current_batch_index < self.current_batches.len() {
+            let batch = self.current_batches[self.current_batch_index].clone();
+            self.current_batch_index += 1;
+            Poll::Ready(Some(Ok(batch)))
+        } else {
+            Poll::Ready(None)
+        }
+    }
+
+    fn size_hint(&self) -> (usize, Option<usize>) {
+        (0, None)
+    }
+}
+
+impl RecordBatchStream for CachedFileStream {
+    fn schema(&self) -> SchemaRef {
+        Arc::clone(&self.schema)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::sync::Arc;
+
+    use arrow::datatypes::{DataType, Field, Schema};
+
+    use super::*;
+
+    #[test]
+    fn test_cached_file_stream_new() {
+        let schema = Arc::new(Schema::new(vec![Field::new(
+            "col1",
+            DataType::Int64,
+            false,
+        )]));
+
+        let cached_files = vec![
+            Arc::new("file1.arrow".to_string()),
+            Arc::new("file2.arrow".to_string()),
+        ];
+
+        let stream = CachedFileStream::new(
+            "test_cached_stream".to_string(),
+            cached_files.clone(),
+            schema.clone(),
+        );
+
+        assert_eq!(stream.id, "test_cached_stream");
+        assert_eq!(stream.cached_files.len(), 2);
+        assert_eq!(stream.current_file_index, 0);
+    }
+}

src/search/src/datafusion/distributed_plan/streaming_aggs_exec/exec.rs

@@ -0,0 +1,305 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{fmt::Debug, sync::Arc};
+
+use arrow::datatypes::SchemaRef;
+use datafusion::{
+    common::Result,
+    error::DataFusionError,
+    execution::{SendableRecordBatchStream, TaskContext},
+    physical_expr::EquivalenceProperties,
+    physical_plan::{
+        DisplayAs, DisplayFormatType, ExecutionPlan, ExecutionPlanProperties, Partitioning,
+        PlanProperties,
+        aggregates::AggregateExec,
+        execution_plan::{Boundedness, EmissionType},
+    },
+};
+use parking_lot::Mutex;
+
+use crate::datafusion::distributed_plan::{
+    cache_buf::{CacheBuf, CacheStream},
+    streaming_aggs_exec::{cached_file_stream::CachedFileStream, monitor_stream::MonitorStream},
+};
+
+#[derive(Debug)]
+pub struct StreamingAggsExec {
+    id: String,
+    start_time: i64,
+    end_time: i64,
+    input: Arc<dyn ExecutionPlan>,
+    /// Cache holding plan properties like equivalences, output partitioning etc.
+    cache: Arc<PlanProperties>,
+    cached_files: Vec<Arc<String>>,
+    cached_partition_num: usize,
+    target_partitions: usize,
+    is_complete_cache_hit: bool,
+    aggregate_plan: Arc<AggregateExec>,
+    cache_buf: Arc<Mutex<CacheBuf>>,
+    overwrite_cache: bool,
+}
+
+impl StreamingAggsExec {
+    /// Create a new StreamingAggsExec with explicit cache strategy
+    #[allow(clippy::too_many_arguments)]
+    pub fn new(
+        id: String,
+        start_time: i64,
+        end_time: i64,
+        cached_files: Vec<Arc<String>>,
+        input: Arc<dyn ExecutionPlan>,
+        target_partitions: usize,
+        is_complete_cache_hit: bool,
+        aggregate_plan: Arc<AggregateExec>,
+        overwrite_cache: bool,
+    ) -> Self {
+        let cached_partition_num = if cached_files.is_empty() { 0 } else { 1 };
+        let total_partition_num = if is_complete_cache_hit {
+            cached_partition_num
+        } else {
+            // Partial or no cache: cached partitions + input partitions
+            let input_partitions = input.output_partitioning().partition_count();
+            input_partitions + cached_partition_num
+        };
+
+        let cache = Self::compute_properties(Arc::clone(&input.schema()), total_partition_num);
+
+        let cached_buf = CacheStream::new(
+            !aggregate_plan.group_expr().is_empty(),
+            target_partitions,
+            aggregate_plan.clone(),
+        );
+
+        Self {
+            id,
+            start_time,
+            end_time,
+            input,
+            cache,
+            cached_files,
+            cached_partition_num,
+            target_partitions,
+            is_complete_cache_hit,
+            aggregate_plan,
+            cache_buf: Arc::new(Mutex::new(CacheBuf {
+                total_partition_num,
+                cached_partition_num,
+                cached_buf,
+            })),
+            overwrite_cache,
+        }
+    }
+
+    pub fn id(&self) -> &str {
+        &self.id
+    }
+
+    pub fn start_time(&self) -> i64 {
+        self.start_time
+    }
+
+    pub fn end_time(&self) -> i64 {
+        self.end_time
+    }
+
+    pub fn target_partitions(&self) -> usize {
+        self.target_partitions
+    }
+
+    pub fn is_complete_cache_hit(&self) -> bool {
+        self.is_complete_cache_hit
+    }
+
+    pub fn cached_files(&self) -> &[Arc<String>] {
+        &self.cached_files
+    }
+
+    pub fn aggregate_plan(&self) -> &Arc<AggregateExec> {
+        &self.aggregate_plan
+    }
+
+    pub fn overwrite_cache(&self) -> bool {
+        self.overwrite_cache
+    }
+
+    pub(crate) fn output_partitioning_helper(n_partitions: usize) -> Partitioning {
+        Partitioning::UnknownPartitioning(n_partitions)
+    }
+
+    /// This function creates the cache object that stores the plan properties such as schema,
+    /// equivalence properties, ordering, partitioning, etc.
+    pub(crate) fn compute_properties(
+        schema: SchemaRef,
+        n_partitions: usize,
+    ) -> Arc<PlanProperties> {
+        let eq_properties = EquivalenceProperties::new(schema);
+        let output_partitioning = Self::output_partitioning_helper(n_partitions);
+        Arc::new(PlanProperties::new(
+            eq_properties,
+            // Output Partitioning
+            output_partitioning,
+            // Execution Mode
+            EmissionType::Incremental,
+            Boundedness::Bounded,
+        ))
+    }
+}
+
+impl DisplayAs for StreamingAggsExec {
+    fn fmt_as(&self, t: DisplayFormatType, f: &mut std::fmt::Formatter) -> std::fmt::Result {
+        match t {
+            DisplayFormatType::Default | DisplayFormatType::Verbose => {
+                let strategy = if self.is_complete_cache_hit {
+                    "complete_hit"
+                } else {
+                    "miss"
+                };
+                write!(
+                    f,
+                    "StreamingAggsExec: streaming_id={}, cache_strategy={strategy}, cached_partitions={}, total_partitions={}",
+                    self.id,
+                    self.cached_partition_num,
+                    self.properties().output_partitioning().partition_count()
+                )
+            }
+            DisplayFormatType::TreeRender => {
+                let strategy = if self.is_complete_cache_hit {
+                    "complete_hit"
+                } else {
+                    "miss"
+                };
+                _ = writeln!(f, "streaming_id={}", self.id);
+                _ = writeln!(f, "cache_strategy={strategy}",);
+                _ = writeln!(f, "cached_partitions={}", self.cached_partition_num);
+                _ = writeln!(
+                    f,
+                    "total_partitions={}",
+                    self.properties().output_partitioning().partition_count()
+                );
+                Ok(())
+            }
+        }
+    }
+}
+
+impl ExecutionPlan for StreamingAggsExec {
+    fn name(&self) -> &'static str {
+        "StreamingAggsExec"
+    }
+
+    fn properties(&self) -> &Arc<PlanProperties> {
+        &self.cache
+    }
+
+    fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>> {
+        vec![&self.input]
+    }
+
+    fn with_new_children(
+        self: Arc<Self>,
+        children: Vec<Arc<dyn ExecutionPlan>>,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        Ok(Arc::new(StreamingAggsExec::new(
+            self.id.clone(),
+            self.start_time,
+            self.end_time,
+            self.cached_files.clone(),
+            children[0].clone(),
+            self.target_partitions,
+            self.is_complete_cache_hit,
+            self.aggregate_plan.clone(),
+            self.overwrite_cache,
+        )))
+    }
+
+    fn execute(
+        &self,
+        partition: usize,
+        context: Arc<TaskContext>,
+    ) -> Result<SendableRecordBatchStream> {
+        // Complete cache hit: only return cached data, never execute input
+        if self.is_complete_cache_hit {
+            log::debug!(
+                "[StreamingAggs streaming_id: {}] Complete cache hit: returning cached data for partition {}/{}",
+                self.id,
+                partition,
+                self.cached_partition_num
+            );
+            if partition < self.cached_partition_num {
+                log::debug!(
+                    "[StreamingAggs streaming_id: {}] EXECUTING with cached files for partition {} (complete cache hit), time_range=[{}, {}], files: {:?}",
+                    self.id,
+                    partition,
+                    self.start_time,
+                    self.end_time,
+                    self.cached_files
+                );
+                // Create a lazy stream that will read cached files on demand
+                return Ok(Box::pin(CachedFileStream::new(
+                    self.id.clone(),
+                    self.cached_files.clone(),
+                    self.input.schema(),
+                )));
+            } else {
+                // This should never happen with complete cache hit
+                return Err(DataFusionError::Internal(format!(
+                    "StreamingAggsExec: Invalid partition {} for complete cache hit with {} cached partitions",
+                    partition, self.cached_partition_num
+                )));
+            }
+        }
+
+        log::debug!(
+            "[StreamingAggs streaming_id: {}] Partial cache hit: partition={}, cached_partitions={}, executing input for new data",
+            self.id,
+            partition,
+            self.cached_partition_num
+        );
+
+        // Partial or no cache: handle both cached and input partitions
+        if partition < self.cached_partition_num {
+            log::debug!(
+                "[StreamingAggs streaming_id: {}] EXECUTING with cached files for partition {} (partial cache hit), time_range=[{}, {}], files: {:?}",
+                self.id,
+                partition,
+                self.start_time,
+                self.end_time,
+                self.cached_files
+            );
+            return Ok(Box::pin(CachedFileStream::new(
+                self.id.clone(),
+                self.cached_files.clone(),
+                self.input.schema(),
+            )));
+        }
+
+        // Execute input for missing data
+        Ok(Box::pin(MonitorStream::new(
+            self.id.clone(),
+            self.start_time,
+            self.end_time,
+            self.input.schema(),
+            self.cache_buf.clone(),
+            self.input
+                .execute(partition - self.cached_partition_num, context)?,
+            self.overwrite_cache,
+        )))
+    }
+
+    fn benefits_from_input_partitioning(&self) -> Vec<bool> {
+        vec![false; self.children().len()]
+    }
+}

src/search/src/datafusion/distributed_plan/streaming_aggs_exec/monitor_stream.rs

@@ -0,0 +1,248 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{
+    pin::Pin,
+    sync::Arc,
+    task::{Context, Poll},
+};
+
+use arrow::{array::RecordBatch, datatypes::SchemaRef};
+use datafusion::{
+    common::Result,
+    execution::{RecordBatchStream, SendableRecordBatchStream},
+};
+use futures::{Stream, StreamExt};
+use futures_util::ready;
+use parking_lot::Mutex;
+
+use crate::{
+    cache::streaming_agg::{
+        RecordBatchCacheRequest, cache_record_batches_to_disk,
+        generate_aggregation_cache_file_name, get_cache_file_path,
+    },
+    datafusion::distributed_plan::{
+        cache_buf::CacheBuf,
+        streaming_aggs_exec::{GLOBAL_CACHE, get_cache_file_path_from_streaming_id},
+    },
+};
+
+pub(crate) struct MonitorStream {
+    id: String,
+    start_time: i64,
+    end_time: i64,
+    schema: SchemaRef,
+    stream: SendableRecordBatchStream,
+    root_cache_buf: Arc<Mutex<CacheBuf>>,
+    done: bool,
+    overwrite_cache: bool,
+}
+
+impl MonitorStream {
+    pub(crate) fn new(
+        id: String,
+        start_time: i64,
+        end_time: i64,
+        schema: SchemaRef,
+        root_cache_buf: Arc<Mutex<CacheBuf>>,
+        stream: SendableRecordBatchStream,
+        overwrite_cache: bool,
+    ) -> Self {
+        Self {
+            id,
+            start_time,
+            end_time,
+            schema,
+            stream,
+            root_cache_buf,
+            done: false,
+            overwrite_cache,
+        }
+    }
+
+    pub fn is_complete_partition_window(&self) -> bool {
+        let interval = GLOBAL_CACHE.get_cache_interval(&self.id); // minutes
+        let interval_micros = interval * 60 * 1_000_000; // microseconds
+        (self.end_time - self.start_time) == interval_micros
+    }
+
+    pub fn append_to_cache_buf(&mut self, record_batch: Arc<RecordBatch>) {
+        self.root_cache_buf.lock().append_data(record_batch);
+    }
+
+    pub fn check_and_add_partition(&mut self) -> bool {
+        self.root_cache_buf.lock().check_and_add_partition()
+    }
+}
+
+impl Stream for MonitorStream {
+    type Item = Result<RecordBatch>;
+
+    fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
+        if self.done {
+            return Poll::Ready(None);
+        }
+
+        let res = ready!(self.stream.poll_next_unpin(cx));
+
+        Poll::Ready(match res {
+            Some(Ok(record_batch)) => {
+                self.append_to_cache_buf(Arc::new(record_batch.clone()));
+                Some(Ok(record_batch))
+            }
+            None => {
+                self.done = true;
+
+                let partition_done = self.check_and_add_partition();
+                let streaming_done = partition_done
+                    && GLOBAL_CACHE
+                        .id_cache
+                        .check_time(&self.id, self.start_time, self.end_time);
+
+                let file_path = get_cache_file_path_from_streaming_id(&self.id)?;
+                let file_name = generate_aggregation_cache_file_name(
+                    &self.id,
+                    self.start_time,
+                    self.end_time,
+                    self.is_complete_partition_window(),
+                );
+
+                // Start - Cache record batches to disk
+                if partition_done && (!streaming_done || self.is_complete_partition_window()) {
+                    let result_vec = self.root_cache_buf.lock().get_final_result(&self.id)?;
+
+                    let file_path = get_cache_file_path(&file_path, &file_name);
+                    let request = RecordBatchCacheRequest {
+                        streaming_id: self.id.clone(),
+                        file_path: file_path.clone(),
+                        schema: self.schema.clone(),
+                        records: result_vec.into_iter().map(Arc::new).collect(),
+                        overwrite_cache: self.overwrite_cache,
+                    };
+
+                    let start = std::time::Instant::now();
+                    match cache_record_batches_to_disk(request) {
+                        Ok(()) => {
+                            // add to cache list
+                            GLOBAL_CACHE.insert(self.id.clone(), file_path);
+                        }
+                        Err(e) => {
+                            log::error!(
+                                "[streaming_id: {}] Error caching streaming aggs record batchesto disk file: {file_path}, error: {e:?}",
+                                self.id,
+                            );
+                        }
+                    }
+                    log::info!(
+                        "[streaming_id: {}] cache_record_batches_to_disk time: {} ms",
+                        self.id,
+                        start.elapsed().as_millis()
+                    );
+                }
+                // End - Cache record batches to disk
+
+                None
+            }
+            Some(Err(e)) => {
+                log::error!("[streaming_id: {}] Error in MonitorStream: {e}", self.id);
+                Some(Err(e))
+            }
+        })
+    }
+
+    fn size_hint(&self) -> (usize, Option<usize>) {
+        self.stream.size_hint()
+    }
+}
+
+impl RecordBatchStream for MonitorStream {
+    /// Get the schema
+    fn schema(&self) -> SchemaRef {
+        Arc::clone(&self.schema)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::sync::Arc;
+
+    use arrow::datatypes::{DataType, Field, Schema};
+    use datafusion::{
+        execution::SendableRecordBatchStream,
+        physical_plan::{aggregates::AggregateExec, memory::MemoryStream},
+    };
+    use tokio::sync::mpsc;
+
+    use super::*;
+    use crate::datafusion::distributed_plan::cache_buf::CacheStream;
+
+    #[test]
+    fn test_monitor_stream_new() {
+        let schema = Arc::new(Schema::new(vec![Field::new(
+            "col1",
+            DataType::Int64,
+            false,
+        )]));
+
+        // Create a dummy stream
+        let batches = vec![];
+        let memory_stream = MemoryStream::try_new(batches, schema.clone(), None).unwrap();
+        let input_stream: SendableRecordBatchStream = Box::pin(memory_stream);
+
+        let (_tx, _rx): (
+            tokio::sync::mpsc::Sender<()>,
+            tokio::sync::mpsc::Receiver<()>,
+        ) = mpsc::channel(1);
+
+        // Create a dummy cache buffer for MonitorStream
+        let cache_buf = Arc::new(parking_lot::Mutex::new(CacheBuf {
+            total_partition_num: 1,
+            cached_partition_num: 0,
+            cached_buf: CacheStream::new(
+                false,
+                1,
+                Arc::new(
+                    AggregateExec::try_new(
+                        datafusion::physical_plan::aggregates::AggregateMode::Partial,
+                        datafusion::physical_plan::aggregates::PhysicalGroupBy::new_single(vec![]),
+                        vec![],
+                        vec![],
+                        Arc::new(datafusion::physical_plan::empty::EmptyExec::new(
+                            schema.clone(),
+                        )),
+                        schema.clone(),
+                    )
+                    .unwrap(),
+                ),
+            ),
+        }));
+
+        // Test MonitorStream::new
+        let monitor_stream = MonitorStream::new(
+            "test_monitor".to_string(),
+            1000,
+            2000,
+            schema.clone(),
+            cache_buf,
+            input_stream,
+            false,
+        );
+
+        // Verify initial state
+        assert_eq!(monitor_stream.id, "test_monitor");
+        assert_eq!(monitor_stream.start_time, 1000);
+        assert_eq!(monitor_stream.end_time, 2000);
+    }
+}

src/search/src/datafusion/distributed_plan/tmp_exec.rs

@@ -0,0 +1,234 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{io::Cursor, sync::Arc};
+
+use arrow::ipc::reader::FileReader;
+use datafusion::{
+    arrow::datatypes::SchemaRef,
+    common::{Result, internal_err},
+    execution::{SendableRecordBatchStream, TaskContext},
+    physical_expr::{EquivalenceProperties, Partitioning},
+    physical_plan::{
+        DisplayAs, DisplayFormatType, ExecutionPlan, PlanProperties,
+        execution_plan::{Boundedness, EmissionType},
+        memory::MemoryStream,
+        stream::RecordBatchStreamAdapter,
+    },
+};
+use futures::TryStreamExt;
+#[cfg(feature = "enterprise")]
+use infra::client::grpc::make_grpc_search_client;
+#[cfg(feature = "enterprise")]
+use o2_enterprise::enterprise::{
+    common::config::get_config as get_o2_config, super_cluster::search::get_cluster_node_by_name,
+};
+
+#[derive(Debug, Clone)]
+pub struct TmpExec {
+    trace_id: String,
+    cluster: String,
+    path: String,
+    data: Option<Vec<u8>>,
+    schema: SchemaRef,
+    cache: Arc<PlanProperties>,
+}
+
+impl TmpExec {
+    pub fn new(
+        trace_id: String,
+        cluster: String,
+        path: String,
+        data: Option<Vec<u8>>,
+        schema: SchemaRef,
+    ) -> Self {
+        let cache = Self::compute_properties(Arc::clone(&schema), 1);
+        TmpExec {
+            trace_id,
+            cluster,
+            path,
+            data,
+            schema,
+            cache,
+        }
+    }
+
+    fn compute_properties(schema: SchemaRef, n_partitions: usize) -> Arc<PlanProperties> {
+        let eq_properties = EquivalenceProperties::new(schema);
+        let output_partitioning = Partitioning::UnknownPartitioning(n_partitions);
+        Arc::new(PlanProperties::new(
+            eq_properties,
+            output_partitioning,
+            EmissionType::Incremental,
+            Boundedness::Bounded,
+        ))
+    }
+
+    pub fn trace_id(&self) -> &str {
+        &self.trace_id
+    }
+
+    pub fn cluster(&self) -> &str {
+        &self.cluster
+    }
+
+    pub fn path(&self) -> &str {
+        &self.path
+    }
+
+    pub fn data(&self) -> &Option<Vec<u8>> {
+        &self.data
+    }
+
+    pub fn schema(&self) -> &SchemaRef {
+        &self.schema
+    }
+
+    pub fn set_data(mut self, data: Vec<u8>) -> Self {
+        self.data = Some(data);
+        self
+    }
+}
+
+impl DisplayAs for TmpExec {
+    fn fmt_as(&self, _t: DisplayFormatType, f: &mut std::fmt::Formatter) -> std::fmt::Result {
+        write!(f, "TmpExec: cluster={}, path={}", self.cluster, self.path)
+    }
+}
+
+impl ExecutionPlan for TmpExec {
+    fn name(&self) -> &'static str {
+        "TmpExec"
+    }
+
+    fn properties(&self) -> &Arc<PlanProperties> {
+        &self.cache
+    }
+
+    fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>> {
+        vec![]
+    }
+
+    fn with_new_children(
+        self: Arc<Self>,
+        _: Vec<Arc<dyn ExecutionPlan>>,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        Ok(self)
+    }
+
+    fn execute(
+        &self,
+        partition: usize,
+        _context: Arc<TaskContext>,
+    ) -> Result<SendableRecordBatchStream> {
+        if partition != 0 {
+            return internal_err!("TmpExec invalid partition {partition} (expected partition: 0)");
+        }
+
+        if let Some(data) = self.data.clone() {
+            let reader =
+                unsafe { FileReader::try_new(Cursor::new(data), None)?.with_skip_validation(true) };
+            let mut batches = Vec::new();
+            for batch in reader {
+                batches.push(batch?);
+            }
+            Ok(Box::pin(MemoryStream::try_new(
+                batches,
+                self.schema.clone(),
+                None,
+            )?))
+        } else {
+            let data = fetch_data(
+                self.trace_id.clone(),
+                self.cluster.clone(),
+                self.path.clone(),
+                Arc::clone(&self.schema),
+            );
+            let stream = futures::stream::once(data).try_flatten();
+            Ok(Box::pin(RecordBatchStreamAdapter::new(
+                self.schema.clone(),
+                stream,
+            )))
+        }
+    }
+}
+
+async fn fetch_data(
+    trace_id: String,
+    cluster: String,
+    path: String,
+    schema: SchemaRef,
+) -> Result<SendableRecordBatchStream> {
+    let data = if cluster == config::get_cluster_name() {
+        infra::storage::get_bytes("", &path).await?
+    } else {
+        #[cfg(feature = "enterprise")]
+        {
+            if !get_o2_config().super_cluster.enabled {
+                return internal_err!(
+                    "cluster: {cluster}'s left data result is in other cluster: {}",
+                    config::get_cluster_name()
+                );
+            }
+            let node = match get_cluster_node_by_name(&cluster).await {
+                Ok(node) => node,
+                Err(e) => return internal_err!("Failed to get cluster node: {e:?}"),
+            };
+            let grpc_addr = node.get_grpc_addr();
+            let path = path.to_string();
+            let task = tokio::task::spawn(async move {
+                let mut request = tonic::Request::new(proto::cluster_rpc::GetTableRequest { path });
+                match make_grpc_search_client(&trace_id, &mut request, &node, 0).await {
+                    Ok(mut client) => match client.get_table(request).await {
+                        Ok(res) => Ok(res.into_inner()),
+                        Err(err) => {
+                            log::error!("search->grpc: node: {grpc_addr}, search err: {err:?}",);
+                            Err(format!("{err:?}"))
+                        }
+                    },
+                    Err(e) => Err(format!("{e:?}")),
+                }
+            });
+
+            let response = match task.await {
+                Ok(Ok(response)) => response,
+                Ok(Err(e)) => return internal_err!("GRPC call failed: {e}"),
+                Err(e) => return internal_err!("Task join failed: {e:?}"),
+            };
+            response.data.into()
+        }
+        #[cfg(not(feature = "enterprise"))]
+        {
+            let _ = trace_id;
+            return internal_err!(
+                "cluster: {cluster}'s left data result is in other cluster: {}",
+                config::get_cluster_name()
+            );
+        }
+    };
+
+    let buf = data;
+    let reader = unsafe { FileReader::try_new(Cursor::new(buf), None)?.with_skip_validation(true) };
+    let mut batches = Vec::new();
+    for batch in reader {
+        batches.push(batch?);
+    }
+
+    Ok(Box::pin(MemoryStream::try_new(
+        batches,
+        Arc::clone(&schema),
+        None,
+    )?))
+}

src/search/src/datafusion/exec.rs

@@ -321,6 +321,12 @@ pub fn register_builtin_udfs(ctx: &SessionContext) {
     ctx.register_udaf(AggregateUDF::from(
         super::udaf::summary_percentile::SummaryPercentile::new(),
     ));
+    ctx.register_udaf(AggregateUDF::from(
+        super::udaf::approx_topk::ApproxTopK::new(),
+    ));
+    ctx.register_udaf(AggregateUDF::from(
+        super::udaf::approx_topk_distinct::ApproxTopKDistinct::new(),
+    ));
     ctx.register_udf(super::udf::cast_to_timestamp_udf::CAST_TO_TIMESTAMP_UDF.clone());
 
     #[cfg(feature = "enterprise")]
@@ -328,12 +334,6 @@ pub fn register_builtin_udfs(ctx: &SessionContext) {
         ctx.register_udf(super::udf::cipher_udf::DECRYPT_UDF.clone());
         ctx.register_udf(super::udf::cipher_udf::DECRYPT_SLOW_UDF.clone());
         ctx.register_udf(super::udf::cipher_udf::ENCRYPT_UDF.clone());
-        ctx.register_udaf(AggregateUDF::from(
-            o2_enterprise::enterprise::search::datafusion::udaf::approx_topk::ApproxTopK::new(),
-        ));
-        ctx.register_udaf(AggregateUDF::from(
-            o2_enterprise::enterprise::search::datafusion::udaf::approx_topk_distinct::ApproxTopKDistinct::new(),
-        ));
         ctx.register_udaf(AggregateUDF::from(
             o2_enterprise::enterprise::search::datafusion::udaf::ddsketch::DDSketchAgg::new(),
         ));
@@ -347,8 +347,11 @@ pub fn register_builtin_udfs(ctx: &SessionContext) {
 pub fn registered_function_names() -> &'static [String] {
     static NAMES: std::sync::OnceLock<Vec<String>> = std::sync::OnceLock::new();
     NAMES.get_or_init(|| {
-        let ctx = SessionContext::new();
+        let mut ctx = SessionContext::new();
         register_builtin_udfs(&ctx);
+        // Production contexts register these separately (see flight.rs); without
+        // the same call here the whole json_* family is missing from the catalog.
+        let _ = datafusion_functions_json::register_all(&mut ctx);
         let state = ctx.state();
         let mut names: Vec<String> = state.scalar_functions().keys().cloned().collect();
         names.extend(state.aggregate_functions().keys().cloned());
@@ -359,6 +362,136 @@ pub fn registered_function_names() -> &'static [String] {
     })
 }
 
+/// One entry of the SQL function catalog served to the query editor.
+#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, PartialEq, utoipa::ToSchema)]
+pub struct CatalogFunction {
+    pub name: String,
+    /// Argument list, e.g. "(field, k)". Drives the editor's `detail` column.
+    pub signature: String,
+    /// Prose for the editor's documentation panel. May be empty for upstream
+    /// DataFusion functions that carry no documentation of their own.
+    pub doc: String,
+    /// "udf" | "scalar" | "aggregate" | "window" | "vrl"
+    pub kind: String,
+    pub deprecated: bool,
+}
+
+/// Argument list and prose for a registry function.
+///
+/// `Signature` has no Display and describes accepted TYPES, not argument names,
+/// so DataFusion's own documentation is the only source of a readable argument
+/// list: prefer its named arguments, fall back to the paren group of its syntax
+/// example, and finally to an opaque placeholder.
+fn describe_registry_fn(doc: Option<&datafusion::logical_expr::Documentation>) -> (String, String) {
+    let Some(d) = doc else {
+        return ("(...)".to_string(), String::new());
+    };
+    if let Some(args) = &d.arguments
+        && !args.is_empty()
+    {
+        let names: Vec<&str> = args.iter().map(|(n, _)| n.as_str()).collect();
+        return (format!("({})", names.join(", ")), d.description.clone());
+    }
+    let sig = d
+        .syntax_example
+        .find('(')
+        .map(|i| d.syntax_example[i..].to_string())
+        .unwrap_or_else(|| "(...)".to_string());
+    (sig, d.description.clone())
+}
+
+/// The org-INDEPENDENT half of the catalog: the DataFusion registry (including
+/// the JSON family) plus the SQL-rewriter aliases.
+///
+/// Snapshotted once. Building it means standing up a SessionContext and
+/// registering ~350 functions, which is far too much to repeat per HTTP
+/// request; `registered_function_names` above caches for the same reason.
+fn base_catalog_functions() -> &'static [CatalogFunction] {
+    static BASE: std::sync::OnceLock<Vec<CatalogFunction>> = std::sync::OnceLock::new();
+    BASE.get_or_init(|| {
+        let mut ctx = SessionContext::new();
+        register_builtin_udfs(&ctx);
+        let _ = datafusion_functions_json::register_all(&mut ctx);
+        let state = ctx.state();
+
+        let mut out: Vec<CatalogFunction> = Vec::new();
+        let mut push = |name: String, signature: String, doc: String, kind: &str, deprecated| {
+            out.push(CatalogFunction {
+                name,
+                signature,
+                doc,
+                kind: kind.to_string(),
+                deprecated,
+            });
+        };
+
+        for (name, udf) in state.scalar_functions() {
+            let (sig, doc) = describe_registry_fn(udf.documentation());
+            push(name.clone(), sig, doc, "scalar", false);
+        }
+        for (name, udf) in state.aggregate_functions() {
+            let (sig, doc) = describe_registry_fn(udf.documentation());
+            push(name.clone(), sig, doc, "aggregate", false);
+        }
+        for (name, udf) in state.window_functions() {
+            let (sig, doc) = describe_registry_fn(udf.documentation());
+            push(name.clone(), sig, doc, "window", false);
+        }
+
+        // Valid SQL that appears in no registry: a rewriter desugars these
+        // before planning.
+        for alias in crate::sql::rewriter::REWRITER_FUNCTION_ALIASES {
+            let target = crate::sql::rewriter::rewriter_alias_target(alias).unwrap_or("match_all");
+            push(
+                (*alias).to_string(),
+                "(term)".to_string(),
+                format!("Deprecated alias for `{target}` — rewritten before planning."),
+                "udf",
+                true,
+            );
+        }
+
+        out
+    })
+}
+
+/// Every function this org can call: the shared registry above plus the org's
+/// own VRL transforms.
+pub fn catalog_functions(org_id: &str) -> Vec<CatalogFunction> {
+    // Keyed by name so the result is sorted and deduplicated for free, and so a
+    // later insert wins — which is what makes the org's VRL transforms override
+    // a same-named builtin, exactly as register_udf does at query time.
+    let mut by_name: std::collections::BTreeMap<String, CatalogFunction> = base_catalog_functions()
+        .iter()
+        .map(|f| (f.name.clone(), f.clone()))
+        .collect();
+
+    let org_prefix = format!("{org_id}/");
+    for transform in transform::QUERY_FUNCTIONS.iter() {
+        if !transform.key().starts_with(&org_prefix) {
+            continue;
+        }
+        let args: Vec<String> = (1..=transform.num_args)
+            .map(|i| format!("arg{i}"))
+            .collect();
+        by_name.insert(
+            transform.name.clone(),
+            CatalogFunction {
+                name: transform.name.clone(),
+                signature: format!("({})", args.join(", ")),
+                doc: format!(
+                    "Organisation VRL function `{}` ({} argument(s)).",
+                    transform.name, transform.num_args
+                ),
+                kind: "vrl".to_string(),
+                deprecated: false,
+            },
+        );
+    }
+
+    by_name.into_values().collect()
+}
+
 pub async fn register_metrics_table(
     session: &SearchSession,
     schema: Arc<Schema>,
@@ -786,6 +919,364 @@ mod tests {
         Ok(())
     }
 
+    // ── tmp/code.md B4 — the function catalog served to the query editor ─────
+    //
+    // registered_function_names() is the authoritative list. Two gaps were
+    // found reviewing it: JSON functions are registered by a SEPARATE call that
+    // the snapshot context never made, and functions provided by SQL rewriters
+    // never appear in any registry at all.
+
+    #[test]
+    fn registered_function_names_includes_o2_udfs() {
+        let names = registered_function_names();
+        for expected in [
+            "str_match",
+            "match_all",
+            "histogram",
+            "spath",
+            "arrcount",
+            "re_match",
+            "cast_to_timestamp",
+        ] {
+            assert!(
+                names.iter().any(|n| n == expected),
+                "registry is missing the O2 UDF `{expected}`"
+            );
+        }
+    }
+
+    #[test]
+    fn registered_function_names_includes_datafusion_builtins() {
+        let names = registered_function_names();
+        for expected in ["date_trunc", "coalesce", "concat", "regexp_replace", "abs"] {
+            assert!(
+                names.iter().any(|n| n == expected),
+                "registry is missing the DataFusion builtin `{expected}`"
+            );
+        }
+    }
+
+    #[test]
+    fn registered_function_names_includes_json_functions() {
+        // Production contexts call datafusion_functions_json::register_all
+        // separately (see flight.rs). Without that call here the whole json_*
+        // family is absent from the catalog the editor is served.
+        let names = registered_function_names();
+        for expected in ["json_get", "json_get_str", "json_length"] {
+            assert!(
+                names.iter().any(|n| n == expected),
+                "registry is missing the JSON function `{expected}` — is                  datafusion_functions_json::register_all wired into the snapshot context?"
+            );
+        }
+    }
+
+    #[test]
+    fn registered_function_names_is_sorted_and_deduped() {
+        let names = registered_function_names();
+        assert!(!names.is_empty());
+        let mut sorted = names.to_vec();
+        sorted.sort();
+        sorted.dedup();
+        assert_eq!(
+            names,
+            sorted.as_slice(),
+            "names must be sorted and deduplicated"
+        );
+    }
+
+    #[test]
+    fn rewriter_aliases_are_exposed_for_the_catalog() {
+        // match_all_raw / match_all_raw_ignore_case are valid user-facing SQL
+        // (sql/rewriter/match_all_raw.rs rewrites them to match_all before
+        // planning) but appear in NO registry. A catalog built only from the
+        // registry would silently drop two functions users rely on today.
+        let aliases = crate::sql::rewriter::REWRITER_FUNCTION_ALIASES;
+        for expected in ["match_all_raw", "match_all_raw_ignore_case"] {
+            assert!(
+                aliases.contains(&expected),
+                "rewriter alias `{expected}` is not exported for the catalog"
+            );
+        }
+    }
+
+    #[test]
+    fn catalog_functions_unions_registry_rewriter_and_json() {
+        let catalog = catalog_functions("default");
+        let names: Vec<&str> = catalog.iter().map(|f| f.name.as_str()).collect();
+        assert!(names.contains(&"match_all"), "registry entry missing");
+        assert!(
+            names.contains(&"match_all_raw"),
+            "rewriter alias missing from the catalog union"
+        );
+        assert!(names.contains(&"json_get"), "json function missing");
+        assert!(names.contains(&"date_trunc"), "datafusion builtin missing");
+    }
+
+    #[test]
+    fn catalog_functions_returns_structured_entries() {
+        // The editor needs more than names: `detail` and `documentation` in the
+        // suggest widget come from signature/doc, and the icon from kind.
+        let catalog = catalog_functions("default");
+        let match_all = catalog
+            .iter()
+            .find(|f| f.name == "match_all")
+            .expect("match_all should be in the catalog");
+        assert!(
+            !match_all.signature.is_empty(),
+            "signature must be populated"
+        );
+        assert!(!match_all.kind.is_empty(), "kind must be populated");
+    }
+
+    #[test]
+    fn catalog_functions_flags_rewriter_aliases_deprecated() {
+        let catalog = catalog_functions("default");
+        let raw = catalog
+            .iter()
+            .find(|f| f.name == "match_all_raw")
+            .expect("match_all_raw should be in the catalog");
+        assert!(
+            raw.deprecated,
+            "rewriter aliases must be flagged deprecated"
+        );
+        let canonical = catalog.iter().find(|f| f.name == "match_all").unwrap();
+        assert!(!canonical.deprecated, "match_all itself is not deprecated");
+    }
+
+    #[test]
+    fn catalog_functions_serialize_with_every_field_the_editor_needs() {
+        // Asserting on the Rust struct does not prove the HTTP body carries the
+        // fields: a rename or a skip_serializing_if would pass the struct tests
+        // and still ship an empty docs panel.
+        let catalog = catalog_functions("default");
+        let json = serde_json::to_value(&catalog).expect("catalog must serialize");
+        let arr = json.as_array().expect("catalog serializes to an array");
+
+        for entry in arr {
+            for field in ["name", "signature", "kind", "deprecated"] {
+                assert!(
+                    entry.get(field).is_some(),
+                    "serialized entry {entry} is missing `{field}`"
+                );
+            }
+        }
+    }
+
+    #[test]
+    fn every_catalog_entry_has_a_name_kind_and_signature() {
+        // Sweeps the WHOLE catalog: a single spot check on match_all would let
+        // every DataFusion, JSON, alias or VRL entry ship blank metadata.
+        for f in catalog_functions("default") {
+            assert!(!f.name.is_empty(), "entry with an empty name");
+            assert!(!f.kind.is_empty(), "`{}` has no kind", f.name);
+            assert!(!f.signature.is_empty(), "`{}` has no signature", f.name);
+        }
+    }
+
+    #[test]
+    fn catalog_functions_documents_what_only_the_SERVER_can_supply() {
+        // Deliberately NOT asserting docs for the O2 UDFs. The frontend catalog
+        // carries its own prose for those and wins on merge (it also owns which
+        // arguments are columns), so a server-side doc for `match_all` would
+        // never be displayed — dead data dressed up as coverage.
+        //
+        // What the server IS the only source for: the rewriter aliases and the
+        // org's VRL transforms.
+        let catalog = catalog_functions("default");
+        for alias in crate::sql::rewriter::REWRITER_FUNCTION_ALIASES {
+            let entry = catalog
+                .iter()
+                .find(|f| &f.name == alias)
+                .unwrap_or_else(|| panic!("`{alias}` missing from the catalog"));
+            assert!(
+                !entry.doc.is_empty(),
+                "rewriter alias `{alias}` must be documented"
+            );
+            assert!(
+                entry.deprecated,
+                "rewriter alias `{alias}` must be flagged deprecated"
+            );
+        }
+    }
+
+    #[test]
+    fn catalog_functions_surfaces_upstream_documentation_where_it_exists() {
+        // Guards the describe() fallback: if it silently returned "(...)" and an
+        // empty doc for everything, every other assertion here would still pass.
+        let catalog = catalog_functions("default");
+        let documented = catalog.iter().filter(|f| !f.doc.is_empty()).count();
+        let with_named_args = catalog
+            .iter()
+            .filter(|f| f.signature != "(...)" && f.signature != "()")
+            .count();
+        assert!(
+            documented > 20,
+            "only {documented} entries carry documentation — is describe() reading upstream docs?"
+        );
+        assert!(
+            with_named_args > 20,
+            "only {with_named_args} entries have a named argument list"
+        );
+    }
+
+    #[test]
+    fn org_vrl_transforms_are_documented_and_signed() {
+        use config::meta::function::Transform;
+        transform::QUERY_FUNCTIONS.insert(
+            "doc_org/documented_fn".to_string(),
+            Transform {
+                function: ".".to_string(),
+                name: "documented_fn".to_string(),
+                params: "row".to_string(),
+                num_args: 1,
+                trans_type: Some(0),
+                streams: None,
+            },
+        );
+        let catalog = catalog_functions("doc_org");
+        let entry = catalog
+            .iter()
+            .find(|f| f.name == "documented_fn")
+            .expect("org transform missing");
+        assert!(!entry.doc.is_empty(), "org VRL transforms must carry a doc");
+        assert!(
+            !entry.signature.is_empty(),
+            "org VRL transforms must carry a signature"
+        );
+        transform::QUERY_FUNCTIONS.remove("doc_org/documented_fn");
+    }
+
+    #[test]
+    fn org_vrl_transform_overrides_a_same_named_builtin() {
+        // register_udf lets an org's VRL transform shadow a builtin at query
+        // time, so the catalog must report the one that would actually run.
+        use config::meta::function::Transform;
+        transform::QUERY_FUNCTIONS.insert(
+            "shadow_org/concat".to_string(),
+            Transform {
+                function: ".".to_string(),
+                name: "concat".to_string(),
+                params: "row".to_string(),
+                num_args: 1,
+                trans_type: Some(0),
+                streams: None,
+            },
+        );
+
+        let catalog = catalog_functions("shadow_org");
+        let entries: Vec<&CatalogFunction> =
+            catalog.iter().filter(|f| f.name == "concat").collect();
+        assert_eq!(entries.len(), 1, "a shadowed builtin must not appear twice");
+        assert_eq!(
+            entries[0].kind, "vrl",
+            "the org transform is what actually runs, so it is what the catalog must report"
+        );
+
+        // Another org still sees the builtin.
+        let other = catalog_functions("unrelated_org");
+        let builtin = other.iter().find(|f| f.name == "concat").unwrap();
+        assert_eq!(builtin.kind, "scalar");
+
+        transform::QUERY_FUNCTIONS.remove("shadow_org/concat");
+    }
+
+    #[test]
+    fn catalog_functions_is_sorted_and_deduped() {
+        let catalog = catalog_functions("default");
+        let names: Vec<String> = catalog.iter().map(|f| f.name.clone()).collect();
+        let mut sorted = names.clone();
+        sorted.sort();
+        sorted.dedup();
+        assert_eq!(names, sorted, "catalog must be sorted and deduplicated");
+    }
+
+    #[test]
+    fn catalog_functions_scopes_vrl_transforms_to_their_own_org() {
+        // Org IDs deliberately OVERLAP as substrings. get_all_transform matches
+        // with `key().contains(org_id)`, so a fixture like org_alpha/org_beta
+        // passes even though "acme" matches the key "acme-prod/...". A prefix
+        // match on "{org}/" is what isolation actually requires.
+        use config::meta::function::Transform;
+        let mk = |name: &str| Transform {
+            function: ".".to_string(),
+            name: name.to_string(),
+            params: "row".to_string(),
+            num_args: 1,
+            trans_type: Some(0),
+            streams: None,
+        };
+        transform::QUERY_FUNCTIONS.insert("acme/acme_only_fn".to_string(), mk("acme_only_fn"));
+        transform::QUERY_FUNCTIONS.insert("acme-prod/prod_only_fn".to_string(), mk("prod_only_fn"));
+
+        let acme: Vec<String> = catalog_functions("acme")
+            .iter()
+            .map(|f| f.name.clone())
+            .collect();
+        let prod: Vec<String> = catalog_functions("acme-prod")
+            .iter()
+            .map(|f| f.name.clone())
+            .collect();
+
+        assert!(
+            acme.contains(&"acme_only_fn".to_string()),
+            "own transform missing"
+        );
+        assert!(
+            !acme.contains(&"prod_only_fn".to_string()),
+            "LEAKED acme-prod's VRL transform into acme — substring org matching"
+        );
+        assert!(
+            prod.contains(&"prod_only_fn".to_string()),
+            "own transform missing"
+        );
+        assert!(
+            !prod.contains(&"acme_only_fn".to_string()),
+            "LEAKED acme's VRL transform into acme-prod"
+        );
+
+        // Built-ins are org-independent and must appear for both.
+        assert!(acme.contains(&"match_all".to_string()));
+        assert!(prod.contains(&"match_all".to_string()));
+
+        transform::QUERY_FUNCTIONS.remove("acme/acme_only_fn");
+        transform::QUERY_FUNCTIONS.remove("acme-prod/prod_only_fn");
+    }
+
+    #[test]
+    fn catalog_functions_marks_vrl_transforms_with_their_own_kind() {
+        use config::meta::function::Transform;
+        transform::QUERY_FUNCTIONS.insert(
+            "org_gamma/gamma_fn".to_string(),
+            Transform {
+                function: ".".to_string(),
+                name: "gamma_fn".to_string(),
+                params: "row".to_string(),
+                num_args: 2,
+                trans_type: Some(0),
+                streams: None,
+            },
+        );
+        let catalog = catalog_functions("org_gamma");
+        let gamma = catalog
+            .iter()
+            .find(|f| f.name == "gamma_fn")
+            .expect("org transform should be in the catalog");
+        assert_eq!(
+            gamma.kind, "vrl",
+            "org transforms must be distinguishable from builtins"
+        );
+        // Count the placeholders rather than sniffing for a digit: the previous
+        // `contains("2") || !is_empty()` reduced to "non-empty", so even "()"
+        // passed for a two-argument transform.
+        assert_eq!(
+            gamma.signature.matches("arg").count(),
+            2,
+            "signature must expose one placeholder per declared argument, got {}",
+            gamma.signature
+        );
+        transform::QUERY_FUNCTIONS.remove("org_gamma/gamma_fn");
+    }
+
     #[test]
     fn test_table_builder_new() {
         let builder = TableBuilder::new();

src/search/src/datafusion/mod.rs

@@ -15,6 +15,7 @@
 
 use std::str::FromStr;
 
+pub mod aggregates;
 pub mod context;
 pub mod distributed_plan;
 pub mod exec;

src/search/src/datafusion/optimizer/context.rs

@@ -16,15 +16,12 @@
 use std::sync::Arc;
 
 use config::{datafusion::request::Request, meta::cluster::NodeInfo};
-use datafusion::sql::TableReference;
+use datafusion::{physical_optimizer::PhysicalOptimizerRule, sql::TableReference};
 use hashbrown::HashMap;
 use infra::errors::Error;
 use parking_lot::Mutex;
-#[cfg(feature = "enterprise")]
-use {
-    datafusion::physical_optimizer::PhysicalOptimizerRule,
-    o2_enterprise::enterprise::search::datafusion::optimizer::stream_aggregate::StreamingAggsRule,
-};
+
+use crate::datafusion::optimizer::stream_aggregate::StreamingAggsRule;
 
 pub enum PhysicalOptimizerContext {
     RemoteScan(RemoteScanContext),
@@ -39,7 +36,6 @@ pub struct RemoteScanContext {
     pub is_leader: bool,
 }
 
-#[cfg(feature = "enterprise")]
 pub struct StreamingAggregationContext {
     pub streaming_id: String,
     pub start_time: i64,
@@ -48,7 +44,6 @@ pub struct StreamingAggregationContext {
     pub overwrite_cache: bool,
 }
 
-#[cfg(feature = "enterprise")]
 impl StreamingAggregationContext {
     pub async fn new(
         request: &Request,
@@ -75,7 +70,6 @@ impl StreamingAggregationContext {
     }
 }
 
-#[cfg(feature = "enterprise")]
 pub fn generate_streaming_agg_rules(
     context: StreamingAggregationContext,
 ) -> Arc<dyn PhysicalOptimizerRule + Send + Sync> {
@@ -88,24 +82,10 @@ pub fn generate_streaming_agg_rules(
     )) as _
 }
 
-#[cfg(not(feature = "enterprise"))]
-pub struct StreamingAggregationContext {}
-
-#[cfg(not(feature = "enterprise"))]
-impl StreamingAggregationContext {
-    pub async fn new(
-        _request: &Request,
-        _is_complete_cache_hit: Arc<Mutex<bool>>,
-    ) -> Result<Option<Self>, Error> {
-        Ok(None)
-    }
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
 
-    #[cfg(not(feature = "enterprise"))]
     #[tokio::test]
     async fn test_streaming_aggregation_context_new_returns_none() {
         let request = Request::default();
@@ -121,7 +101,6 @@ mod tests {
     }
 
     #[test]
-    #[cfg(not(feature = "enterprise"))]
     fn test_physical_optimizer_context_streaming_aggregation_none() {
         let ctx = PhysicalOptimizerContext::StreamingAggregation(None);
         assert!(matches!(

src/search/src/datafusion/optimizer/eliminate_aggregate.rs

@@ -0,0 +1,65 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::sync::Arc;
+
+use datafusion::{
+    common::{Result, tree_node::TreeNode},
+    config::ConfigOptions,
+    physical_optimizer::PhysicalOptimizerRule,
+    physical_plan::{ExecutionPlan, ExecutionPlanProperties, empty::EmptyExec},
+};
+
+/// rewrite the plan to eliminate the aggregate plan if the streaming aggregation's output partition
+/// is 0
+#[derive(Debug, Default)]
+pub struct EliminateAggregateRule {}
+
+impl EliminateAggregateRule {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl PhysicalOptimizerRule for EliminateAggregateRule {
+    fn optimize(
+        &self,
+        plan: Arc<dyn ExecutionPlan>,
+        _config: &ConfigOptions,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        let is_empty_streaming_agg = plan.exists(|plan| {
+            if plan.name() == "StreamingAggsExec"
+                && plan.output_partitioning().partition_count() == 0
+            {
+                return Ok(true);
+            }
+            Ok(false)
+        })?;
+
+        if is_empty_streaming_agg {
+            return Ok(Arc::new(EmptyExec::new(plan.schema())) as _);
+        }
+
+        Ok(plan)
+    }
+
+    fn name(&self) -> &str {
+        "EliminateAggregateRule"
+    }
+
+    fn schema_check(&self) -> bool {
+        true
+    }
+}

src/search/src/datafusion/optimizer/mod.rs

@@ -40,26 +40,22 @@ use datafusion::{
 };
 use hashbrown::HashSet;
 use infra::schema::get_stream_setting_index_fields;
-#[cfg(feature = "enterprise")]
-use {
-    crate::datafusion::optimizer::context::generate_streaming_agg_rules,
-    crate::datafusion::optimizer::logical_optimizer::cipher::{
-        RewriteCipherCall, RewriteCipherKey,
-    },
-    o2_enterprise::enterprise::search::datafusion::optimizer::aggregate_topk::AggregateTopkRule,
-    o2_enterprise::enterprise::search::datafusion::optimizer::eliminate_aggregate::EliminateAggregateRule,
-};
 
+#[cfg(feature = "enterprise")]
+use crate::datafusion::optimizer::logical_optimizer::cipher::{
+    RewriteCipherCall, RewriteCipherKey,
+};
 use crate::{
     datafusion::optimizer::{
         analyze::remove_index_fields::RemoveIndexFieldsRule,
-        context::PhysicalOptimizerContext,
+        context::{PhysicalOptimizerContext, generate_streaming_agg_rules},
+        eliminate_aggregate::EliminateAggregateRule,
         logical_optimizer::{
             add_sort_and_limit::AddSortAndLimitRule, limit_join_right_side::LimitJoinRightSide,
             rewrite_histogram::RewriteHistogram,
         },
         physical_optimizer::{
-            distribute_analyze::optimize_distribute_analyze,
+            aggregate_topk::AggregateTopkRule, distribute_analyze::optimize_distribute_analyze,
             index_optimizer::LeaderIndexOptimizerRule, join_reorder::JoinReorderRule,
             remote_scan::generate_remote_scan_rules,
         },
@@ -69,8 +65,10 @@ use crate::{
 
 pub mod analyze;
 pub mod context;
+pub mod eliminate_aggregate;
 pub mod logical_optimizer;
 pub mod physical_optimizer;
+pub mod stream_aggregate;
 pub mod utils;
 
 pub fn generate_analyzer_rules(sql: &Sql) -> Vec<Arc<dyn AnalyzerRule + Send + Sync>> {
@@ -178,19 +176,12 @@ pub fn generate_physical_optimizer_rules(
                 rules.push(generate_remote_scan_rules(req, sql, context));
             }
             PhysicalOptimizerContext::AggregateTopk => {
-                #[cfg(feature = "enterprise")]
                 rules.push(Arc::new(AggregateTopkRule::new(sql.limit)));
-                #[cfg(not(feature = "enterprise"))]
-                continue;
             }
             PhysicalOptimizerContext::StreamingAggregation(context) => {
-                if let Some(_context) = context {
-                    #[cfg(feature = "enterprise")]
-                    rules.push(generate_streaming_agg_rules(_context));
-                    #[cfg(feature = "enterprise")]
+                if let Some(context) = context {
+                    rules.push(generate_streaming_agg_rules(context));
                     rules.push(Arc::new(EliminateAggregateRule::new()) as _);
-                    #[cfg(not(feature = "enterprise"))]
-                    continue;
                 }
             }
         }

src/search/src/datafusion/optimizer/physical_optimizer/aggregate_topk.rs

@@ -0,0 +1,229 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::sync::Arc;
+
+use datafusion::{
+    common::{
+        Result,
+        tree_node::{Transformed, TreeNode, TreeNodeRecursion, TreeNodeRewriter, TreeNodeVisitor},
+    },
+    config::ConfigOptions,
+    physical_optimizer::PhysicalOptimizerRule,
+    physical_plan::{
+        ExecutionPlan,
+        aggregates::{AggregateExec, AggregateMode},
+        projection::ProjectionExec,
+        sorts::{sort::SortExec, sort_preserving_merge::SortPreservingMergeExec},
+    },
+};
+
+use crate::datafusion::{
+    distributed_plan::aggregate_topk_exec::AggregateTopkExec,
+    optimizer::physical_optimizer::utils::get_final_aggregate_plan,
+};
+
+// add remote scan to physical plan
+#[derive(Debug)]
+pub struct AggregateTopkRule {
+    limit: i64,
+}
+
+impl AggregateTopkRule {
+    pub fn new(limit: i64) -> Self {
+        Self { limit }
+    }
+}
+
+impl PhysicalOptimizerRule for AggregateTopkRule {
+    fn optimize(
+        &self,
+        plan: Arc<dyn ExecutionPlan>,
+        _config: &ConfigOptions,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        if self.limit <= 0 || !config::get_config().common.aggregation_topk_enabled {
+            return Ok(plan);
+        }
+
+        // check if there is no aggregate plan, return the original plan
+        let Some(final_agg_plan) = get_final_aggregate_plan(Arc::clone(&plan)) else {
+            return Ok(plan);
+        };
+
+        // check if the group by only one column
+        if final_agg_plan.group_expr().expr().len() != 1 {
+            return Ok(plan);
+        }
+
+        // check if the agg function is count
+        if final_agg_plan.aggr_expr().len() != 1 {
+            return Ok(plan);
+        }
+
+        if let Some(expr) = final_agg_plan.aggr_expr().first() {
+            if !["count", "avg", "min", "max", "sum", "approx_distinct"]
+                .contains(&expr.fun().name())
+            {
+                return Ok(plan);
+            }
+            let expr_name = expr.name();
+            let mut visitor = SortLimitVisitor::new(expr_name);
+            let _ = plan.visit(&mut visitor);
+            if visitor.is_match {
+                let mut rewriter =
+                    AggregateTopkRewriter::new(expr_name, visitor.descending, visitor.limit as u64);
+                let plan = plan.rewrite(&mut rewriter)?.data;
+                return Ok(plan);
+            }
+        }
+        Ok(plan)
+    }
+
+    fn name(&self) -> &str {
+        "AggregateTopkRule"
+    }
+
+    fn schema_check(&self) -> bool {
+        true
+    }
+}
+
+/// This rewriter is used to add a new node AggregateMergeExec in the middle of the
+/// RemoteScanExec->AggregateExec. It will get the topK records from the AggregateExec and return
+/// them to the RemoteScanExec.
+pub(crate) struct AggregateTopkRewriter {
+    field: String,
+    descending: bool,
+    limit: u64,
+}
+
+impl AggregateTopkRewriter {
+    pub(crate) fn new(field: &str, descending: bool, limit: u64) -> Self {
+        Self {
+            field: field.to_string(),
+            descending,
+            limit,
+        }
+    }
+}
+
+impl TreeNodeRewriter for AggregateTopkRewriter {
+    type Node = Arc<dyn ExecutionPlan>;
+
+    fn f_up(&mut self, node: Arc<dyn ExecutionPlan>) -> Result<Transformed<Self::Node>> {
+        // This feature need cluster mode, single node we can skip it
+        if node.children().len() == 1 && node.children().first().unwrap().name() == "AggregateExec"
+        {
+            let agg_node = node.children().first().cloned().unwrap();
+            let Some(agg_exec) = agg_node.downcast_ref::<AggregateExec>() else {
+                return Ok(Transformed::no(node));
+            };
+            if agg_exec.mode() != &AggregateMode::Partial {
+                return Ok(Transformed::no(node));
+            }
+
+            let input_plan = Arc::clone(agg_node);
+            let agg_plan =
+                AggregateTopkExec::new(input_plan, &self.field, self.descending, self.limit);
+
+            let node =
+                node.with_new_children(vec![Arc::new(agg_plan) as Arc<dyn ExecutionPlan>])?;
+            return Ok(Transformed::new(node, true, TreeNodeRecursion::Stop));
+        }
+        Ok(Transformed::no(node))
+    }
+}
+
+#[derive(Default)]
+pub(crate) struct SortLimitVisitor {
+    field: String,
+    pub(crate) limit: usize,
+    pub(crate) descending: bool,
+    pub(crate) is_match: bool,
+}
+
+impl SortLimitVisitor {
+    pub(crate) fn new(field: &str) -> Self {
+        Self {
+            field: field.to_string(),
+            ..Default::default()
+        }
+    }
+}
+
+impl<'n> TreeNodeVisitor<'n> for SortLimitVisitor {
+    type Node = Arc<dyn ExecutionPlan>;
+
+    fn f_up(&mut self, node: &'n Self::Node) -> Result<TreeNodeRecursion> {
+        if node.name() == "ProjectionExec" {
+            // we need to check if the field is map to any alias
+            let Some(expr) = node.downcast_ref::<ProjectionExec>() else {
+                return Ok(TreeNodeRecursion::Continue);
+            };
+            for projection_expr in expr.expr().iter() {
+                let expr = &projection_expr.expr;
+                let alias = &projection_expr.alias;
+                if expr
+                    .to_string()
+                    .split('@')
+                    .next()
+                    .is_some_and(|v| v == self.field)
+                {
+                    self.field = alias.clone();
+                    break;
+                }
+            }
+        } else if node.name() == "SortExec" {
+            // we need to check if the field is sort by field
+            let Some(expr) = node.downcast_ref::<SortExec>() else {
+                return Ok(TreeNodeRecursion::Continue);
+            };
+            for sort_expr in expr.expr().iter() {
+                if sort_expr
+                    .expr
+                    .to_string()
+                    .split('@')
+                    .next()
+                    .is_some_and(|v| v == self.field)
+                {
+                    self.is_match = true;
+                    self.limit = expr.fetch().unwrap_or(0);
+                    self.descending = sort_expr.options.descending;
+                    return Ok(TreeNodeRecursion::Stop);
+                }
+            }
+        } else if node.name() == "SortPreservingMergeExec" {
+            // we need to check if the field is sort by field
+            let Some(expr) = node.downcast_ref::<SortPreservingMergeExec>() else {
+                return Ok(TreeNodeRecursion::Continue);
+            };
+            for sort_expr in expr.expr().iter() {
+                if sort_expr
+                    .expr
+                    .to_string()
+                    .split('@')
+                    .next()
+                    .is_some_and(|v| v == self.field)
+                {
+                    self.is_match = true;
+                    self.limit = expr.fetch().unwrap_or(0);
+                    self.descending = sort_expr.options.descending;
+                    return Ok(TreeNodeRecursion::Stop);
+                }
+            }
+        }
+        Ok(TreeNodeRecursion::Continue)
+    }
+}

src/search/src/datafusion/optimizer/physical_optimizer/broadcast_join.rs

@@ -20,21 +20,175 @@ use config::ider::uuid;
 use datafusion::{
     common::{
         Result,
-        tree_node::{Transformed, TreeNode, TreeNodeRewriter},
+        tree_node::{Transformed, TreeNode, TreeNodeRecursion, TreeNodeRewriter, TreeNodeVisitor},
+    },
+    physical_plan::{
+        ExecutionPlan,
+        aggregates::AggregateExec,
+        coalesce_partitions::CoalescePartitionsExec,
+        joins::{HashJoinExec, PartitionMode},
+        limit::GlobalLimitExec,
+        sorts::sort_preserving_merge::SortPreservingMergeExec,
     },
-    physical_plan::{ExecutionPlan, limit::GlobalLimitExec},
-};
-use o2_enterprise::enterprise::search::datafusion::distributed_plan::{
-    broadcast_join_exec::BroadcastJoinExec, tmp_exec::TmpExec,
 };
 
 use crate::datafusion::{
-    distributed_plan::node::RemoteScanNodes,
+    distributed_plan::{
+        broadcast_join_exec::BroadcastJoinExec, node::RemoteScanNodes, tmp_exec::TmpExec,
+    },
     optimizer::physical_optimizer::remote_scan::{
         RemoteScanRewriter, remote_scan_to_top_if_needed,
     },
 };
 
+// Check if the plan can use broadcast join.
+pub fn should_use_broadcast_join(plan: &Arc<dyn ExecutionPlan>) -> bool {
+    let mut count = 0;
+    // 1. check if only one HashJoinExec and no other multi table ExecutionPlan
+    plan.apply(|node| {
+        Ok(if node.name() == "HashJoinExec" {
+            count += 1;
+            let hash_join = node.downcast_ref::<HashJoinExec>().unwrap();
+            if *hash_join.partition_mode() != PartitionMode::CollectLeft {
+                count += 1;
+            }
+            TreeNodeRecursion::Continue
+        } else if node.name().contains("Join")
+            || node.name() == "UnionExec"
+            || node.name() == "InterleaveExec"
+            || node.name() == "RecursiveQueryExec"
+        {
+            count += 2;
+            TreeNodeRecursion::Continue
+        } else {
+            TreeNodeRecursion::Continue
+        })
+    })
+    .unwrap();
+
+    // 2. check if the left table and the right table satisfy the condition
+    let mut visitor = BroadcastJoinVisitor::new();
+    plan.visit(&mut visitor)
+        .is_ok_and(|_| visitor.use_broadcast_join && count == 1)
+}
+
+#[derive(Debug)]
+struct BroadcastJoinVisitor {
+    use_broadcast_join: bool,
+}
+
+impl BroadcastJoinVisitor {
+    fn new() -> Self {
+        BroadcastJoinVisitor {
+            use_broadcast_join: false,
+        }
+    }
+}
+
+impl<'n> TreeNodeVisitor<'n> for BroadcastJoinVisitor {
+    type Node = Arc<dyn ExecutionPlan>;
+
+    fn f_up(&mut self, node: &'n Arc<dyn ExecutionPlan>) -> Result<TreeNodeRecursion> {
+        if node.name() == "HashJoinExec" {
+            let hash_join = node.downcast_ref::<HashJoinExec>().unwrap();
+            let left = hash_join.left();
+            let right = hash_join.right();
+            if is_broadcast_left(left) && is_broadcast_right(right) {
+                self.use_broadcast_join = true;
+            }
+            return Ok(TreeNodeRecursion::Stop);
+        }
+        Ok(TreeNodeRecursion::Continue)
+    }
+}
+
+// Left table should have aggregate and limit.
+fn is_broadcast_left(left: &Arc<dyn ExecutionPlan>) -> bool {
+    let mut visitor = LeftVisitor::new();
+    left.visit(&mut visitor)
+        .is_ok_and(|_| visitor.has_aggregate && visitor.has_limit)
+}
+
+struct LeftVisitor {
+    has_aggregate: bool,
+    has_limit: bool,
+}
+
+impl LeftVisitor {
+    fn new() -> Self {
+        Self {
+            has_aggregate: false,
+            has_limit: false,
+        }
+    }
+}
+
+impl<'n> TreeNodeVisitor<'n> for LeftVisitor {
+    type Node = Arc<dyn ExecutionPlan>;
+
+    fn f_up(&mut self, node: &'n Arc<dyn ExecutionPlan>) -> Result<TreeNodeRecursion> {
+        if let Some(aggregate) = node.downcast_ref::<AggregateExec>() {
+            if aggregate.fetch().is_some() {
+                self.has_limit = true;
+            }
+            self.has_aggregate = true;
+            return Ok(TreeNodeRecursion::Continue);
+        } else if let Some(sort_merge) = node.downcast_ref::<SortPreservingMergeExec>() {
+            if sort_merge.fetch().is_some() {
+                self.has_limit = true;
+            }
+            return Ok(TreeNodeRecursion::Continue);
+        } else if let Some(partition) = node.downcast_ref::<CoalescePartitionsExec>() {
+            if partition.fetch().is_some() {
+                self.has_limit = true;
+            }
+            return Ok(TreeNodeRecursion::Continue);
+        } else if node.name() == "GlobalLimitExec" || node.name() == "DeduplicationExec" {
+            self.has_limit = true;
+            return Ok(TreeNodeRecursion::Continue);
+        }
+        Ok(TreeNodeRecursion::Continue)
+    }
+}
+
+// Right table should be table scan and filter.
+fn is_broadcast_right(right: &Arc<dyn ExecutionPlan>) -> bool {
+    let mut visitor = RightVisitor::new();
+    right
+        .visit(&mut visitor)
+        .is_ok_and(|_| visitor.is_broadcast_right)
+}
+
+struct RightVisitor {
+    is_broadcast_right: bool,
+}
+
+impl RightVisitor {
+    fn new() -> Self {
+        Self {
+            is_broadcast_right: true,
+        }
+    }
+}
+
+impl<'n> TreeNodeVisitor<'n> for RightVisitor {
+    type Node = Arc<dyn ExecutionPlan>;
+
+    fn f_up(&mut self, node: &'n Arc<dyn ExecutionPlan>) -> Result<TreeNodeRecursion> {
+        // right table should only have NewEmptyExec, FilterExec, CooperativeExec,
+        // CoalesceBatchesExec
+        if !(node.name() == "NewEmptyExec"
+            || node.name() == "FilterExec"
+            || node.name() == "CooperativeExec"
+            || node.name() == "CoalesceBatchesExec")
+        {
+            self.is_broadcast_right = false;
+            return Ok(TreeNodeRecursion::Stop);
+        }
+        Ok(TreeNodeRecursion::Continue)
+    }
+}
+
 pub fn broadcast_join_rewrite(
     plan: Arc<dyn ExecutionPlan>,
     remote_scan_nodes: Arc<RemoteScanNodes>,
@@ -124,7 +278,6 @@ mod tests {
         execution::{runtime_env::RuntimeEnvBuilder, session_state::SessionStateBuilder},
         prelude::{SessionConfig, SessionContext},
     };
-    use o2_enterprise::enterprise::search::datafusion::optimizer::broadcast_join::should_use_broadcast_join;
 
     use super::*;
     use crate::datafusion::{

src/search/src/datafusion/optimizer/physical_optimizer/join_reorder.rs

@@ -28,11 +28,10 @@ use datafusion::{
     },
 };
 
-#[cfg(feature = "enterprise")]
-use crate::datafusion::optimizer::physical_optimizer::enrichment::{
-    is_enrichment_table, should_use_enrichment_broadcast_join,
+use crate::datafusion::optimizer::physical_optimizer::{
+    enrichment::{is_enrichment_table, should_use_enrichment_broadcast_join},
+    utils::is_aggregate_exec,
 };
-use crate::datafusion::optimizer::physical_optimizer::utils::is_aggregate_exec;
 
 #[derive(Default, Debug)]
 pub struct JoinReorderRule;
@@ -67,7 +66,6 @@ fn swap_join_order(plan: Arc<dyn ExecutionPlan>) -> Result<Transformed<Arc<dyn E
         let right = hash_join.right();
 
         // If right table is enrichment table and left table is not, swap them
-        #[cfg(feature = "enterprise")]
         if config::get_config()
             .common
             .feature_enrichment_broadcast_join_enabled

src/search/src/datafusion/optimizer/physical_optimizer/mod.rs

@@ -13,10 +13,9 @@
 // You should have received a copy of the GNU Affero General Public License
 // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-#[cfg(feature = "enterprise")]
+pub mod aggregate_topk;
 pub mod broadcast_join;
 pub mod distribute_analyze;
-#[cfg(feature = "enterprise")]
 pub mod enrichment;
 pub mod index;
 pub mod index_optimizer;

src/search/src/datafusion/optimizer/physical_optimizer/remote_scan.rs

@@ -34,20 +34,22 @@ use datafusion::{
 };
 use hashbrown::HashMap;
 use proto::cluster_rpc::{self, KvItem};
-#[cfg(feature = "enterprise")]
-use {
-    crate::datafusion::optimizer::physical_optimizer::broadcast_join::broadcast_join_rewrite,
-    crate::datafusion::optimizer::physical_optimizer::enrichment::enrichment_broadcast_join_rewrite,
-    crate::datafusion::optimizer::physical_optimizer::enrichment::should_use_enrichment_broadcast_join,
-    o2_enterprise::enterprise::search::datafusion::optimizer::broadcast_join::should_use_broadcast_join,
-};
 
 use crate::{
     datafusion::{
         distributed_plan::{
             empty_exec::NewEmptyExec, node::RemoteScanNodes, remote_scan_exec::RemoteScanExec,
         },
-        optimizer::{context::RemoteScanContext, utils::is_place_holder_or_empty},
+        optimizer::{
+            context::RemoteScanContext,
+            physical_optimizer::{
+                broadcast_join::{broadcast_join_rewrite, should_use_broadcast_join},
+                enrichment::{
+                    enrichment_broadcast_join_rewrite, should_use_enrichment_broadcast_join,
+                },
+            },
+            utils::is_place_holder_or_empty,
+        },
     },
     sql::Sql,
 };
@@ -160,7 +162,6 @@ impl PhysicalOptimizerRule for RemoteScanRule {
             return Ok(plan);
         }
 
-        #[cfg(feature = "enterprise")]
         if config::get_config()
             .common
             .feature_enrichment_broadcast_join_enabled
@@ -169,7 +170,6 @@ impl PhysicalOptimizerRule for RemoteScanRule {
             return enrichment_broadcast_join_rewrite(plan, self.remote_scan_nodes.clone());
         }
 
-        #[cfg(feature = "enterprise")]
         if config::get_config().common.feature_broadcast_join_enabled
             && should_use_broadcast_join(&plan)
         {

src/search/src/datafusion/optimizer/physical_optimizer/utils.rs

@@ -17,7 +17,10 @@ use std::sync::Arc;
 
 use config::{TIMESTAMP_COL_NAME, meta::inverted_index::UNKNOWN_NAME};
 use datafusion::{
-    common::{Result, tree_node::TreeNode},
+    common::{
+        Result,
+        tree_node::{TreeNode, TreeNodeRecursion, TreeNodeVisitor},
+    },
     error::DataFusionError,
     logical_expr::Operator,
     physical_expr::{
@@ -27,6 +30,7 @@ use datafusion::{
     },
     physical_plan::{
         ExecutionPlan,
+        aggregates::{AggregateExec, AggregateMode},
         expressions::{BinaryExpr, CastExpr, lit},
     },
     scalar::ScalarValue,
@@ -37,6 +41,39 @@ pub fn is_aggregate_exec(plan: &Arc<dyn ExecutionPlan>) -> bool {
         .unwrap_or(false)
 }
 
+/// Get the first final aggregate plan from bottom to top.
+pub(crate) fn get_final_aggregate_plan(plan: Arc<dyn ExecutionPlan>) -> Option<AggregateExec> {
+    let mut visitor = FinalAggregateVisitor::default();
+    let _ = plan.visit(&mut visitor);
+    visitor
+        .plan
+        .map(|plan| plan.downcast_ref::<AggregateExec>().unwrap().clone())
+}
+
+#[derive(Default)]
+struct FinalAggregateVisitor {
+    plan: Option<Arc<dyn ExecutionPlan>>,
+}
+
+impl<'n> TreeNodeVisitor<'n> for FinalAggregateVisitor {
+    type Node = Arc<dyn ExecutionPlan>;
+
+    fn f_up(&mut self, node: &'n Self::Node) -> Result<TreeNodeRecursion> {
+        let Some(aggregate) = node.downcast_ref::<AggregateExec>() else {
+            return Ok(TreeNodeRecursion::Continue);
+        };
+        if matches!(
+            aggregate.mode(),
+            AggregateMode::Final | AggregateMode::FinalPartitioned
+        ) {
+            self.plan = Some(node.clone());
+            Ok(TreeNodeRecursion::Stop)
+        } else {
+            Ok(TreeNodeRecursion::Continue)
+        }
+    }
+}
+
 pub fn extract_string_literal(expr: &Arc<dyn PhysicalExpr>) -> Result<String> {
     if let Some(literal) = expr.downcast_ref::<Literal>() {
         match literal.value() {

src/search/src/datafusion/optimizer/stream_aggregate.rs

@@ -0,0 +1,984 @@
+// Copyright 2025 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::sync::Arc;
+
+use config::meta::search::Interval;
+use datafusion::{
+    common::{
+        DataFusionError, Result,
+        tree_node::{Transformed, TreeNode, TreeNodeRecursion, TreeNodeRewriter},
+    },
+    config::ConfigOptions,
+    physical_optimizer::PhysicalOptimizerRule,
+    physical_plan::{
+        ExecutionPlan,
+        aggregates::{AggregateExec, AggregateMode},
+    },
+};
+use parking_lot::Mutex;
+
+use crate::{
+    cache::streaming_agg::{CacheEntry, StreamingAggsPartitionStrategy},
+    datafusion::{
+        distributed_plan::streaming_aggs_exec::{self, exec::StreamingAggsExec},
+        optimizer::physical_optimizer::utils::get_final_aggregate_plan,
+    },
+};
+
+/// Scores a cache file based on its interval and usefulness for the partition
+/// Returns 0 if file should be excluded (interval < target_interval or no overlap)
+/// Higher score = better file
+fn score_cache_file(
+    streaming_id: &str,
+    file: &CacheEntry,
+    partition_start: i64,
+    partition_end: i64,
+    target_interval: Interval,
+) -> i64 {
+    // Filter out files with interval < target_interval
+    if file.interval.get_interval_microseconds() < target_interval.get_interval_microseconds() {
+        log::debug!(
+            "[streaming_id: {}] Excluding cache file {} (interval: {:?}) - smaller than target interval: {:?}",
+            streaming_id,
+            file.file_path,
+            file.interval,
+            target_interval
+        );
+        return 0; // Exclude smaller interval files
+    }
+
+    // Calculate overlap with partition
+    let overlap_start = file.start_time.max(partition_start);
+    let overlap_end = file.end_time.min(partition_end);
+    let overlap_duration = (overlap_end - overlap_start).max(0);
+
+    if overlap_duration <= 0 {
+        return 0; // No overlap
+    }
+
+    // Prefer longer intervals (weight by interval duration)
+    let interval_weight = file.interval.get_interval_microseconds();
+
+    // Calculate usefulness percentage (how much of the file is actually needed)
+    // Similar to ResultCacheSelectionStrategy::Both
+    let file_duration = file.end_time - file.start_time;
+    let usefulness = if file_duration > 0 {
+        (overlap_duration * 100) / file_duration
+    } else {
+        0
+    };
+
+    // Combined score: interval weight * usefulness percentage
+    // This prioritizes files with longer intervals that have good overlap
+    let score = (interval_weight / 1_000_000) * usefulness; // Normalize to prevent overflow
+
+    log::debug!(
+        "[streaming_id: {}] Scoring cache file {}: interval={:?}, overlap={}μs, usefulness={}%, score={}",
+        streaming_id,
+        file.file_path,
+        file.interval,
+        overlap_duration,
+        usefulness,
+        score
+    );
+
+    score
+}
+
+/// Checks if a time range [start, end] is fully covered by existing ranges
+fn is_fully_covered(covered: &[(i64, i64)], start: i64, end: i64) -> bool {
+    for (c_start, c_end) in covered {
+        if *c_start <= start && *c_end >= end {
+            return true; // Fully covered
+        }
+    }
+    false
+}
+
+/// Merges overlapping time ranges to simplify coverage tracking
+fn merge_ranges(ranges: &mut Vec<(i64, i64)>) {
+    if ranges.len() <= 1 {
+        return;
+    }
+    ranges.sort_by_key(|r| r.0);
+    let mut merged = vec![ranges[0]];
+    for &(start, end) in &ranges[1..] {
+        let last_idx = merged.len() - 1;
+        if start <= merged[last_idx].1 {
+            // Overlapping, merge
+            merged[last_idx].1 = merged[last_idx].1.max(end);
+        } else {
+            merged.push((start, end));
+        }
+    }
+    *ranges = merged;
+}
+
+/// Checks if entire time range [start, end] is fully covered by existing ranges
+fn is_range_fully_covered(covered: &[(i64, i64)], start: i64, end: i64) -> bool {
+    // Merge ranges first to get consolidated coverage
+    let mut ranges = covered.to_vec();
+    merge_ranges(&mut ranges);
+
+    // Check if any single merged range covers [start, end]
+    ranges.iter().any(|(s, e)| *s <= start && *e >= end)
+}
+
+/// Helper function to select files from a list and track coverage
+/// Returns selected file paths and updates covered_ranges
+fn select_from_files(
+    streaming_id: &str,
+    files: Vec<CacheEntry>,
+    partition_start: i64,
+    partition_end: i64,
+    min_interval: Interval,
+    covered_ranges: &mut Vec<(i64, i64)>,
+) -> Vec<String> {
+    let mut selected_files = Vec::new();
+
+    // Score and sort files
+    let mut scored_files: Vec<(CacheEntry, i64)> = files
+        .into_iter()
+        .map(|f| {
+            let score = if min_interval == Interval::Zero
+                || f.interval.get_interval_microseconds()
+                    >= min_interval.get_interval_microseconds()
+            {
+                score_cache_file(
+                    streaming_id,
+                    &f,
+                    partition_start,
+                    partition_end,
+                    min_interval,
+                )
+            } else {
+                0
+            };
+            (f, score)
+        })
+        .filter(|(_, score)| *score > 0)
+        .collect();
+
+    // Sort by score (descending) - best files first
+    scored_files.sort_by_key(|k| std::cmp::Reverse(k.1));
+
+    // Greedy selection: pick files that cover uncovered time ranges
+    for (file, score) in scored_files {
+        let file_start = file.start_time.max(partition_start);
+        let file_end = file.end_time.min(partition_end);
+
+        // Check if this file covers any uncovered time
+        if !is_fully_covered(covered_ranges, file_start, file_end) {
+            log::debug!(
+                "[streaming_id: {}] Selected cache file {} (score={}, interval={:?}) covering [{}, {}]",
+                streaming_id,
+                file.file_path,
+                score,
+                file.interval,
+                file_start,
+                file_end
+            );
+            selected_files.push(file.file_path.clone());
+            covered_ranges.push((file_start, file_end));
+            // Merge overlapping ranges to simplify future checks
+            merge_ranges(covered_ranges);
+        } else {
+            log::debug!(
+                "[streaming_id: {}] Skipped cache file {} (score={}, interval={:?}) - time range [{}, {}] already covered",
+                streaming_id,
+                file.file_path,
+                score,
+                file.interval,
+                file_start,
+                file_end
+            );
+        }
+    }
+
+    selected_files
+}
+
+/// Selects optimal cache files eliminating overlaps and preferring longer intervals
+/// Uses a TWO-PASS greedy algorithm:
+/// Pass 1: Prefer files with target_interval or larger (eliminates overlaps with smaller intervals)
+/// Pass 2: Fill remaining gaps with smaller interval files (maximizes cache usage)
+fn select_optimal_cache_files(
+    streaming_id: &str,
+    cache_files: Vec<CacheEntry>,
+    partition_start: i64,
+    partition_end: i64,
+    target_interval: Interval,
+) -> Vec<String> {
+    if cache_files.is_empty() {
+        return vec![];
+    }
+
+    let total_files = cache_files.len();
+
+    // Partition files into preferred (>= target interval) and smaller (< target interval)
+    let (preferred_files, smaller_files): (Vec<_>, Vec<_>) =
+        cache_files.into_iter().partition(|f| {
+            f.interval.get_interval_microseconds() >= target_interval.get_interval_microseconds()
+        });
+
+    log::debug!(
+        "[streaming_id: {}] Cache file distribution for partition [{}, {}]: target_interval={:?}, preferred={}, smaller={}",
+        streaming_id,
+        partition_start,
+        partition_end,
+        target_interval,
+        preferred_files.len(),
+        smaller_files.len()
+    );
+
+    let mut selected_files = Vec::new();
+    let mut covered_ranges: Vec<(i64, i64)> = Vec::new();
+
+    // PASS 1: Select from preferred files (target interval or larger)
+    if !preferred_files.is_empty() {
+        log::debug!(
+            "[streaming_id: {}] Pass 1: Selecting from {} preferred files (interval >= {:?})",
+            streaming_id,
+            preferred_files.len(),
+            target_interval
+        );
+
+        let pass1_result = select_from_files(
+            streaming_id,
+            preferred_files,
+            partition_start,
+            partition_end,
+            target_interval,
+            &mut covered_ranges,
+        );
+
+        selected_files.extend(pass1_result.iter().cloned());
+
+        log::debug!(
+            "[streaming_id: {}] Pass 1 complete: selected {} files, coverage: {:?}",
+            streaming_id,
+            pass1_result.len(),
+            covered_ranges
+        );
+    }
+
+    // PASS 2: Fill gaps with smaller interval files if needed
+    if !smaller_files.is_empty() {
+        // Check if entire range is covered
+        if !is_range_fully_covered(&covered_ranges, partition_start, partition_end) {
+            log::info!(
+                "[streaming_id: {}] Pass 2: Gaps exist in coverage - attempting to fill with {} smaller interval files",
+                streaming_id,
+                smaller_files.len()
+            );
+
+            let pass2_result = select_from_files(
+                streaming_id,
+                smaller_files,
+                partition_start,
+                partition_end,
+                Interval::Zero, // Accept any interval for gap filling
+                &mut covered_ranges,
+            );
+
+            if !pass2_result.is_empty() {
+                log::info!(
+                    "[streaming_id: {}] Pass 2 complete: filled gaps with {} smaller interval files",
+                    streaming_id,
+                    pass2_result.len()
+                );
+                selected_files.extend(pass2_result);
+            } else {
+                log::debug!(
+                    "[streaming_id: {streaming_id}] Pass 2: No additional files needed to fill gaps"
+                );
+            }
+        } else {
+            log::debug!(
+                "[streaming_id: {streaming_id}] Pass 2 skipped: Entire range [{partition_start}, {partition_end}] already covered by preferred files"
+            );
+        }
+    }
+
+    let coverage_status = if is_range_fully_covered(&covered_ranges, partition_start, partition_end)
+    {
+        "FULLY COVERED"
+    } else {
+        "PARTIAL COVERAGE"
+    };
+
+    log::info!(
+        "[streaming_id: {}] Selected {} optimal cache files from {} total (eliminated overlaps, {} coverage) for partition [{}, {}]",
+        streaming_id,
+        selected_files.len(),
+        total_files,
+        coverage_status,
+        partition_start,
+        partition_end
+    );
+
+    selected_files
+}
+
+/// Checks if a partition [start_time, end_time] is fully cached based on the partition strategy
+fn check_partition_cached_from_strategy(
+    strategy: &StreamingAggsPartitionStrategy,
+    start_time: i64,
+    end_time: i64,
+) -> bool {
+    match strategy {
+        StreamingAggsPartitionStrategy::FullyCached { .. } => {
+            // All partitions are cached
+            true
+        }
+        StreamingAggsPartitionStrategy::Hybrid {
+            cached_partitions, ..
+        } => {
+            // Check if this partition is within any cached partition
+            // A partition is cached if it's fully contained within a cached range
+            cached_partitions
+                .iter()
+                .any(|cp| cp.start_time <= start_time && cp.end_time >= end_time)
+        }
+        StreamingAggsPartitionStrategy::NoCacheAvailable { .. } => {
+            // No cache available
+            false
+        }
+    }
+}
+
+/// Loads cache file paths from the partition strategy into GLOBAL_CACHE
+/// This ensures that cached_files will be available when StreamingAggsExec executes
+/// Uses optimal selection to eliminate overlapping files and prefer longer intervals
+fn load_cache_files_from_strategy(
+    streaming_id: &str,
+    strategy: &StreamingAggsPartitionStrategy,
+    start_time: i64,
+    end_time: i64,
+) {
+    let cache_files: Vec<String> = match strategy {
+        StreamingAggsPartitionStrategy::FullyCached { cache_files } => {
+            // For fully cached queries, determine target interval from the cache files
+            // Use the maximum interval found in the cache files as the target
+            let target_interval = cache_files
+                .iter()
+                .map(|cf| cf.interval)
+                .max_by_key(|interval| interval.get_interval_microseconds())
+                .unwrap_or(Interval::Zero);
+
+            log::debug!(
+                "[streaming_id: {streaming_id}] FullyCached query: using target_interval={target_interval:?}"
+            );
+
+            // Select optimal files eliminating overlaps
+            select_optimal_cache_files(
+                streaming_id,
+                cache_files.clone(),
+                start_time,
+                end_time,
+                target_interval,
+            )
+        }
+        StreamingAggsPartitionStrategy::Hybrid {
+            cached_partitions, ..
+        } => {
+            // Find the cached partition(s) that cover this time range
+            let matching_partitions: Vec<_> = cached_partitions
+                .iter()
+                .filter(|cp| cp.start_time <= start_time && cp.end_time >= end_time)
+                .collect();
+
+            if matching_partitions.is_empty() {
+                log::debug!(
+                    "[streaming_id: {streaming_id}] No matching cached partitions for time_range=[{start_time}, {end_time}]"
+                );
+                vec![]
+            } else {
+                // Use the interval from the matching cached partition as target
+                // If multiple partitions match, use the maximum interval
+                let target_interval = matching_partitions
+                    .iter()
+                    .map(|cp| cp.interval)
+                    .max_by_key(|interval| interval.get_interval_microseconds())
+                    .unwrap_or(Interval::Zero);
+
+                log::debug!(
+                    "[streaming_id: {streaming_id}] Hybrid query: found {} matching partitions, target_interval={:?}",
+                    matching_partitions.len(),
+                    target_interval
+                );
+
+                // Collect all cache files from matching partitions
+                let all_cache_files: Vec<CacheEntry> = matching_partitions
+                    .iter()
+                    .flat_map(|cp| cp.cache_files.iter().cloned())
+                    .collect();
+
+                // Select optimal files eliminating overlaps
+                select_optimal_cache_files(
+                    streaming_id,
+                    all_cache_files,
+                    start_time,
+                    end_time,
+                    target_interval,
+                )
+            }
+        }
+        StreamingAggsPartitionStrategy::NoCacheAvailable { .. } => {
+            // No cache files to load
+            vec![]
+        }
+    };
+
+    // Load each cache file path into GLOBAL_CACHE
+    let num_files = cache_files.len();
+
+    // Log all selected cache files before loading
+    if !cache_files.is_empty() {
+        log::debug!(
+            "[streaming_id: {streaming_id}] Selected {num_files} OPTIMAL cache files (overlaps eliminated) for time_range=[{start_time}, {end_time}]: {cache_files:?}"
+        );
+    } else {
+        log::warn!(
+            "[streaming_id: {streaming_id}] No cache files selected for time_range=[{start_time}, {end_time}] - may need to execute query"
+        );
+    }
+
+    for file_path in cache_files {
+        streaming_aggs_exec::GLOBAL_CACHE.insert(streaming_id.to_string(), file_path.clone());
+        log::debug!(
+            "[streaming_id: {streaming_id}] Loaded cache file into GLOBAL_CACHE: {file_path}"
+        );
+    }
+
+    log::info!(
+        "[streaming_id: {streaming_id}] Loaded {num_files} cache files from partition strategy for time_range=[{start_time}, {end_time}]",
+    );
+}
+
+#[derive(Debug)]
+pub struct StreamingAggsRule {
+    id: String,
+    start_time: i64,
+    end_time: i64,
+    is_complete_cache_hit: Arc<Mutex<bool>>,
+    overwrite_cache: bool,
+}
+
+impl StreamingAggsRule {
+    pub fn new(
+        id: String,
+        start_time: i64,
+        end_time: i64,
+        is_complete_cache_hit: Arc<Mutex<bool>>,
+        overwrite_cache: bool,
+    ) -> Self {
+        Self {
+            id,
+            start_time,
+            end_time,
+            is_complete_cache_hit,
+            overwrite_cache,
+        }
+    }
+}
+
+impl PhysicalOptimizerRule for StreamingAggsRule {
+    fn optimize(
+        &self,
+        plan: Arc<dyn ExecutionPlan>,
+        config: &ConfigOptions,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        let Some(final_agg_plan) = get_final_aggregate_plan(Arc::clone(&plan)) else {
+            return Ok(plan);
+        };
+
+        let mut rewriter = StreamingAggsRewriter::new(
+            self.id.clone(),
+            self.start_time,
+            self.end_time,
+            config.execution.target_partitions,
+            Arc::new(final_agg_plan),
+            Arc::clone(&self.is_complete_cache_hit),
+            self.overwrite_cache,
+        )?;
+        let plan = plan.rewrite(&mut rewriter)?.data;
+
+        Ok(plan)
+    }
+
+    fn name(&self) -> &str {
+        "StreamAggregateRule"
+    }
+
+    fn schema_check(&self) -> bool {
+        true
+    }
+}
+
+pub(crate) struct StreamingAggsRewriter {
+    id: String,
+    start_time: i64,
+    end_time: i64,
+    target_partitions: usize,
+    pub is_complete_cache_hit: Arc<Mutex<bool>>,
+    pub(crate) final_agg_plan: Arc<AggregateExec>,
+    overwrite_cache: bool,
+}
+
+impl StreamingAggsRewriter {
+    pub(crate) fn new(
+        id: String,
+        start_time: i64,
+        end_time: i64,
+        target_partitions: usize,
+        final_agg_plan: Arc<AggregateExec>,
+        is_complete_cache_hit: Arc<Mutex<bool>>,
+        overwrite_cache: bool,
+    ) -> Result<Self> {
+        let ret = Self {
+            id: id.clone(),
+            start_time,
+            end_time,
+            target_partitions,
+            is_complete_cache_hit,
+            final_agg_plan,
+            overwrite_cache,
+        };
+
+        // Check if this partition is fully cached using partition strategy
+        let streaming_item = streaming_aggs_exec::GLOBAL_CACHE.id_cache.get(&id);
+        let Some(item) = streaming_item else {
+            // didn't find cache for the streaming_id, skip loading cache
+            return Err(DataFusionError::Plan(format!(
+                "streaming aggregation cache not found with id: {id}"
+            )));
+        };
+
+        // Use partition strategy to determine if this partition is fully cached
+        let is_fully_cached = if let Some(strategy) = item.get_partition_strategy() {
+            let is_cached = check_partition_cached_from_strategy(&strategy, start_time, end_time);
+
+            // If cached, load the cache file paths into GLOBAL_CACHE for later retrieval
+            if is_cached {
+                load_cache_files_from_strategy(&id, &strategy, start_time, end_time);
+            }
+
+            is_cached
+        } else {
+            // No partition strategy available, assume not cached
+            false
+        };
+
+        if is_fully_cached {
+            // Get all cached files currently in GLOBAL_CACHE for this streaming_id
+            let cached_files = streaming_aggs_exec::GLOBAL_CACHE
+                .get(&id)
+                .unwrap_or_default();
+            log::info!(
+                "[streaming_id {id}] StreamingAggsRewriter: partition fully cached, time_range=[{start_time}, {end_time}], cached_files_count={}",
+                cached_files.len(),
+            );
+            *ret.is_complete_cache_hit.lock() = true;
+        } else {
+            log::info!(
+                "[streaming_id {id}] StreamingAggsRewriter: partition NOT fully cached (will execute query), time_range=[{start_time}, {end_time}]"
+            );
+        }
+
+        Ok(ret)
+    }
+}
+
+impl TreeNodeRewriter for StreamingAggsRewriter {
+    type Node = Arc<dyn ExecutionPlan>;
+
+    fn f_up(&mut self, node: Arc<dyn ExecutionPlan>) -> Result<Transformed<Self::Node>> {
+        if (node.name() == "RemoteScanExec"
+            && node.children().len() == 1
+            && node.children().first().unwrap().name() == "AggregateExec")
+            || is_single_node_aggregate(&node)
+        {
+            // get all cached files for the streaming_id(first partition -> current partition)
+            let cached_files = streaming_aggs_exec::GLOBAL_CACHE
+                .get(&self.id)
+                .unwrap_or_default();
+
+            log::info!(
+                "[streaming_id {}] StreamingAggsRewriter: cache_strategy={}, cached_batches={}",
+                self.id,
+                if *self.is_complete_cache_hit.lock() {
+                    "complete_hit"
+                } else {
+                    "miss"
+                },
+                cached_files.len()
+            );
+
+            let plan = Arc::new(StreamingAggsExec::new(
+                self.id.clone(),
+                self.start_time,
+                self.end_time,
+                cached_files,
+                node,
+                self.target_partitions,
+                *self.is_complete_cache_hit.lock(),
+                self.final_agg_plan.clone(),
+                self.overwrite_cache,
+            )) as _;
+
+            return Ok(Transformed::new(plan, true, TreeNodeRecursion::Stop));
+        }
+        Ok(Transformed::no(node))
+    }
+}
+
+fn is_single_node_aggregate(node: &Arc<dyn ExecutionPlan>) -> bool {
+    config::get_config()
+        .common
+        .feature_single_node_optimize_enabled
+        && config::cluster::LOCAL_NODE.is_single_node()
+        && node
+            .downcast_ref::<AggregateExec>()
+            .is_some_and(|agg| agg.mode() == &AggregateMode::Partial)
+}
+
+#[cfg(test)]
+mod tests {
+    use config::meta::search::Interval;
+
+    use super::*;
+    use crate::cache::streaming_agg::CacheEntry;
+
+    #[test]
+    fn test_score_cache_file_excludes_smaller_intervals() {
+        let file = CacheEntry {
+            file_path: "test_30min.arrow".to_string(),
+            start_time: 1000,
+            end_time: 2000,
+            interval: Interval::ThirtyMinutes,
+        };
+
+        // Should exclude file with 30min interval when target is 60min
+        let score = score_cache_file("test_streaming_id", &file, 1000, 2000, Interval::OneHour);
+        assert_eq!(score, 0, "Should exclude files with smaller interval");
+    }
+
+    #[test]
+    fn test_score_cache_file_accepts_matching_interval() {
+        let file = CacheEntry {
+            file_path: "test_60min.arrow".to_string(),
+            start_time: 1000,
+            end_time: 3_600_000_000 + 1000, // 1 hour later
+            interval: Interval::OneHour,
+        };
+
+        let score = score_cache_file(
+            "test_streaming_id",
+            &file,
+            1000,
+            3_600_000_000 + 1000,
+            Interval::OneHour,
+        );
+        assert!(score > 0, "Should accept files with matching interval");
+    }
+
+    #[test]
+    fn test_score_cache_file_prefers_longer_intervals() {
+        let file_30min = CacheEntry {
+            file_path: "test_30min.arrow".to_string(),
+            start_time: 1000,
+            end_time: 1_800_000_000 + 1000,
+            interval: Interval::ThirtyMinutes,
+        };
+
+        let file_60min = CacheEntry {
+            file_path: "test_60min.arrow".to_string(),
+            start_time: 1000,
+            end_time: 3_600_000_000 + 1000,
+            interval: Interval::OneHour,
+        };
+
+        let score_30 = score_cache_file(
+            "test_streaming_id",
+            &file_30min,
+            1000,
+            3_600_000_000 + 1000,
+            Interval::ThirtyMinutes,
+        );
+        let score_60 = score_cache_file(
+            "test_streaming_id",
+            &file_60min,
+            1000,
+            3_600_000_000 + 1000,
+            Interval::ThirtyMinutes,
+        );
+
+        assert!(score_60 > score_30, "Should prefer longer interval files");
+    }
+
+    #[test]
+    fn test_is_fully_covered() {
+        let covered = vec![(1000, 2000), (3000, 4000)];
+
+        // Fully covered range
+        assert!(is_fully_covered(&covered, 1200, 1800));
+
+        // Not covered range
+        assert!(!is_fully_covered(&covered, 2500, 2800));
+
+        // Partially covered range
+        assert!(!is_fully_covered(&covered, 1500, 2500));
+    }
+
+    #[test]
+    fn test_merge_ranges() {
+        let mut ranges = vec![(1000, 2000), (1500, 2500), (3000, 4000)];
+        merge_ranges(&mut ranges);
+
+        assert_eq!(ranges.len(), 2);
+        assert_eq!(ranges[0], (1000, 2500));
+        assert_eq!(ranges[1], (3000, 4000));
+    }
+
+    #[test]
+    fn test_select_optimal_cache_files_eliminates_overlaps() {
+        // Scenario: 30min and 60min files covering same time range
+        let files = vec![
+            CacheEntry {
+                file_path: "1764153000000000_1764154800000000.arrow".to_string(), /* 10:30-11:00
+                                                                                   * (30min) */
+                start_time: 1764153000000000,
+                end_time: 1764154800000000,
+                interval: Interval::ThirtyMinutes,
+            },
+            CacheEntry {
+                file_path: "1764154800000000_1764156600000000.arrow".to_string(), /* 11:00-11:30
+                                                                                   * (30min) */
+                start_time: 1764154800000000,
+                end_time: 1764156600000000,
+                interval: Interval::ThirtyMinutes,
+            },
+            CacheEntry {
+                file_path: "1764154800000000_1764158400000000.arrow".to_string(), /* 11:00-12:00
+                                                                                   * (60min) */
+                start_time: 1764154800000000,
+                end_time: 1764158400000000,
+                interval: Interval::OneHour,
+            },
+        ];
+
+        // Query for 10:30-12:00 with target interval 60min
+        let selected = select_optimal_cache_files(
+            "test_streaming_id",
+            files,
+            1764153000000000,
+            1764158400000000,
+            Interval::OneHour,
+        );
+
+        // Should select 2 files: 60min file (11:00-12:00) and 30min file for gap (10:30-11:00)
+        // The 30min file 11:00-11:30 should NOT be selected because 60min file covers it
+        assert_eq!(
+            selected.len(),
+            2,
+            "Should select 60min file + 30min for gap"
+        );
+        assert!(
+            selected
+                .iter()
+                .any(|f| f.contains("1764154800000000_1764158400000000")),
+            "Should select the 60min interval file (11:00-12:00)"
+        );
+        assert!(
+            selected
+                .iter()
+                .any(|f| f.contains("1764153000000000_1764154800000000")),
+            "Should select the 30min file for gap (10:30-11:00)"
+        );
+        assert!(
+            !selected
+                .iter()
+                .any(|f| f.contains("1764154800000000_1764156600000000")),
+            "Should NOT select the 30min file (11:00-11:30) that overlaps with 60min file"
+        );
+    }
+
+    #[test]
+    fn test_select_optimal_cache_files_no_overlap_selection() {
+        // Scenario: Multiple 60min files with no overlaps
+        let files = vec![
+            CacheEntry {
+                file_path: "1764158400000000_1764162000000000.arrow".to_string(), // 12:00-13:00
+                start_time: 1764158400000000,
+                end_time: 1764162000000000,
+                interval: Interval::OneHour,
+            },
+            CacheEntry {
+                file_path: "1764162000000000_1764165600000000.arrow".to_string(), // 13:00-14:00
+                start_time: 1764162000000000,
+                end_time: 1764165600000000,
+                interval: Interval::OneHour,
+            },
+            CacheEntry {
+                file_path: "1764165600000000_1764169200000000.arrow".to_string(), // 14:00-15:00
+                start_time: 1764165600000000,
+                end_time: 1764169200000000,
+                interval: Interval::OneHour,
+            },
+        ];
+
+        // Query for 12:00-15:00 with target interval 60min
+        let selected = select_optimal_cache_files(
+            "test_streaming_id",
+            files,
+            1764158400000000,
+            1764169200000000,
+            Interval::OneHour,
+        );
+
+        // Should select all three files as they don't overlap
+        assert_eq!(selected.len(), 3, "Should select all non-overlapping files");
+    }
+
+    #[test]
+    fn test_select_optimal_cache_files_empty_input() {
+        let files = vec![];
+        let selected = select_optimal_cache_files(
+            "test_streaming_id",
+            files,
+            1764153000000000,
+            1764158400000000,
+            Interval::OneHour,
+        );
+
+        assert_eq!(selected.len(), 0, "Should return empty for empty input");
+    }
+
+    #[test]
+    fn test_two_pass_selection_fills_gaps_with_smaller_intervals() {
+        // Scenario: Query needs 60min intervals, but has a gap that only 30min files can fill
+        // This tests the two-pass algorithm
+        let files = vec![
+            // Gap: 10:30-11:00 - only covered by 30min file
+            CacheEntry {
+                file_path: "1764153000000000_1764154800000000.arrow".to_string(), /* 10:30-11:00
+                                                                                   * (30min) */
+                start_time: 1764153000000000,
+                end_time: 1764154800000000,
+                interval: Interval::ThirtyMinutes,
+            },
+            // Main coverage: 11:00-12:00 - covered by 60min file
+            CacheEntry {
+                file_path: "1764154800000000_1764158400000000.arrow".to_string(), /* 11:00-12:00
+                                                                                   * (60min) */
+                start_time: 1764154800000000,
+                end_time: 1764158400000000,
+                interval: Interval::OneHour,
+            },
+        ];
+
+        // Query for 10:30-12:00 with target interval 60min
+        let selected = select_optimal_cache_files(
+            "test_streaming_id",
+            files,
+            1764153000000000,
+            1764158400000000,
+            Interval::OneHour,
+        );
+
+        // Should select BOTH files:
+        // Pass 1: Select 60min file (11:00-12:00)
+        // Pass 2: Fill gap with 30min file (10:30-11:00)
+        assert_eq!(
+            selected.len(),
+            2,
+            "Should select both files to cover full range"
+        );
+        assert!(
+            selected
+                .iter()
+                .any(|f| f.contains("1764154800000000_1764158400000000")),
+            "Should include 60min file"
+        );
+        assert!(
+            selected
+                .iter()
+                .any(|f| f.contains("1764153000000000_1764154800000000")),
+            "Should include 30min file to fill gap"
+        );
+    }
+
+    #[test]
+    fn test_two_pass_selection_prefers_longer_intervals_when_overlapping() {
+        // Scenario: Both 30min and 60min files cover the same range
+        // Should prefer 60min (Pass 1) and skip 30min files
+        let files = vec![
+            CacheEntry {
+                file_path: "1764154800000000_1764156600000000.arrow".to_string(), /* 11:00-11:30
+                                                                                   * (30min) */
+                start_time: 1764154800000000,
+                end_time: 1764156600000000,
+                interval: Interval::ThirtyMinutes,
+            },
+            CacheEntry {
+                file_path: "1764156600000000_1764158400000000.arrow".to_string(), /* 11:30-12:00
+                                                                                   * (30min) */
+                start_time: 1764156600000000,
+                end_time: 1764158400000000,
+                interval: Interval::ThirtyMinutes,
+            },
+            CacheEntry {
+                file_path: "1764154800000000_1764158400000000.arrow".to_string(), /* 11:00-12:00
+                                                                                   * (60min) */
+                start_time: 1764154800000000,
+                end_time: 1764158400000000,
+                interval: Interval::OneHour,
+            },
+        ];
+
+        // Query for 11:00-12:00 with target interval 60min
+        let selected = select_optimal_cache_files(
+            "test_streaming_id",
+            files,
+            1764154800000000,
+            1764158400000000,
+            Interval::OneHour,
+        );
+
+        // Should ONLY select the 60min file (Pass 1 covers everything, Pass 2 skipped)
+        assert_eq!(selected.len(), 1, "Should only select the 60min file");
+        assert!(
+            selected[0].contains("1764154800000000_1764158400000000"),
+            "Should select the 60min interval file"
+        );
+    }
+
+    #[test]
+    fn test_is_range_fully_covered() {
+        let covered = vec![(1000, 2000), (2000, 3000)]; // Adjacent ranges
+
+        // Should be fully covered after merging
+        assert!(is_range_fully_covered(&covered, 1000, 3000));
+
+        // Partial overlap - not fully covered
+        assert!(!is_range_fully_covered(&covered, 500, 1500));
+
+        // Gap in coverage
+        let covered_with_gap = vec![(1000, 2000), (3000, 4000)];
+        assert!(!is_range_fully_covered(&covered_with_gap, 1000, 4000));
+    }
+}

src/search/src/datafusion/udaf/approx_topk.rs

@@ -0,0 +1,611 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{fmt::Formatter, sync::Arc};
+
+use arrow::{
+    array::{Array, AsArray, Int64Array, LargeStringArray, RecordBatch, StructArray},
+    datatypes::{FieldRef, Fields},
+};
+use datafusion::{
+    arrow::{
+        array::ArrayRef,
+        datatypes::{DataType, Field, Schema},
+    },
+    common::{internal_err, not_impl_err, plan_err},
+    error::Result,
+    logical_expr::{
+        Accumulator, AggregateUDFImpl, ColumnarValue, Signature, TypeSignature, Volatility,
+        function::{AccumulatorArgs, StateFieldsArgs},
+        utils::format_state_name,
+    },
+    physical_plan::PhysicalExpr,
+    scalar::ScalarValue,
+};
+use hashbrown::HashMap;
+
+const APPROX_TOPK: &str = "approx_topk";
+
+/// Approximate TopK UDAF that returns the top K elements by frequency.
+///
+/// Usage: approx_topk(field, k, [cap])
+/// - field: the field to find top k values from
+/// - k: number of top elements to return
+/// - cap: optional maximum number of candidates to keep in memory (default: max(k*4, 1000))
+///
+/// For partial aggregation, returns top k elements from each partition.
+/// For final aggregation, merges results from all partitions and returns final top k.
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ApproxTopK(Signature);
+
+impl ApproxTopK {
+    pub fn new() -> Self {
+        Self(Signature::one_of(
+            vec![
+                // String field with Int64 k
+                TypeSignature::Exact(vec![DataType::Utf8, DataType::Int64]),
+                TypeSignature::Exact(vec![DataType::LargeUtf8, DataType::Int64]),
+                // String field with Int64 k and optional Int64 cap
+                TypeSignature::Exact(vec![DataType::Utf8, DataType::Int64, DataType::Int64]),
+                TypeSignature::Exact(vec![DataType::LargeUtf8, DataType::Int64, DataType::Int64]),
+            ],
+            Volatility::Immutable,
+        ))
+    }
+}
+
+impl Default for ApproxTopK {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl AggregateUDFImpl for ApproxTopK {
+    fn name(&self) -> &str {
+        APPROX_TOPK
+    }
+
+    fn signature(&self) -> &datafusion::logical_expr::Signature {
+        &self.0
+    }
+
+    fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+        match &arg_types[0] {
+            DataType::Utf8 | DataType::LargeUtf8 => {
+                // Return array of structs: [{value: string, count: int64}]
+                Ok(DataType::List(Arc::new(Field::new(
+                    "item",
+                    DataType::Struct(
+                        vec![
+                            Field::new("value", DataType::LargeUtf8, false),
+                            Field::new("count", DataType::Int64, false),
+                        ]
+                        .into(),
+                    ),
+                    true,
+                ))))
+            }
+            _ => plan_err!("approx_topk requires string input types"),
+        }
+    }
+
+    fn state_fields(&self, args: StateFieldsArgs) -> Result<Vec<FieldRef>> {
+        Ok(vec![
+            // Store values as list of strings
+            Arc::new(Field::new(
+                format_state_name(args.name, "values"),
+                DataType::List(Arc::new(Field::new("item", DataType::LargeUtf8, true))),
+                true,
+            )),
+            // Store counts as list of int64
+            Arc::new(Field::new(
+                format_state_name(args.name, "counts"),
+                DataType::List(Arc::new(Field::new("item", DataType::Int64, true))),
+                true,
+            )),
+            // Store k parameter
+            Arc::new(Field::new(
+                format_state_name(args.name, "k"),
+                DataType::Int64,
+                false,
+            )),
+        ])
+    }
+
+    fn accumulator(&self, args: AccumulatorArgs) -> Result<Box<dyn Accumulator>> {
+        let k = validate_k_parameter(&args.exprs[1])?;
+        let cap = if args.exprs.len() > 2 {
+            Some(validate_cap_parameter(&args.exprs[2])?)
+        } else {
+            None
+        };
+        let value_data_type = args.exprs[0].data_type(args.schema)?;
+
+        match value_data_type {
+            DataType::Utf8 | DataType::LargeUtf8 => {
+                Ok(Box::new(ApproxTopKAccumulator::new(k, cap)))
+            }
+            other => {
+                not_impl_err!("Support for 'APPROX_TOPK' for data type {other} is not implemented")
+            }
+        }
+    }
+}
+
+fn validate_k_parameter(expr: &Arc<dyn PhysicalExpr>) -> Result<usize> {
+    let empty_schema = Arc::new(Schema::empty());
+    let batch = RecordBatch::new_empty(Arc::clone(&empty_schema));
+
+    let k = match expr.evaluate(&batch)? {
+        ColumnarValue::Scalar(ScalarValue::Int64(Some(value))) => {
+            if value <= 0 {
+                return plan_err!("k parameter for 'APPROX_TOPK' must be positive, got {value}");
+            }
+            value as usize
+        }
+        ColumnarValue::Scalar(other) => {
+            return not_impl_err!(
+                "k parameter for 'APPROX_TOPK' must be Int64 literal (got {:?})",
+                other.data_type()
+            );
+        }
+        _ => {
+            return internal_err!("Expected scalar value for k parameter");
+        }
+    };
+
+    Ok(k)
+}
+
+fn validate_cap_parameter(expr: &Arc<dyn PhysicalExpr>) -> Result<usize> {
+    let empty_schema = Arc::new(Schema::empty());
+    let batch = RecordBatch::new_empty(Arc::clone(&empty_schema));
+
+    let cap = match expr.evaluate(&batch)? {
+        ColumnarValue::Scalar(ScalarValue::Int64(Some(value))) => {
+            if value <= 0 {
+                return plan_err!("cap parameter for 'APPROX_TOPK' must be positive, got {value}");
+            }
+            value as usize
+        }
+        ColumnarValue::Scalar(other) => {
+            return not_impl_err!(
+                "cap parameter for 'APPROX_TOPK' must be Int64 literal (got {:?})",
+                other.data_type()
+            );
+        }
+        _ => {
+            return internal_err!("Expected scalar value for cap parameter");
+        }
+    };
+
+    Ok(cap)
+}
+
+/// Memory-efficient accumulator that only tracks top-K candidates
+/// Uses a min-heap to maintain only the most frequent items
+struct ApproxTopKAccumulator {
+    // Only keep track of top candidates - LIMITED SIZE!
+    candidates: HashMap<String, i64>,
+    k: usize,
+    // Memory management
+    max_candidates: usize,    // Maximum candidates to keep in memory
+    min_count_threshold: i64, // Minimum count to be considered
+}
+
+impl std::fmt::Debug for ApproxTopKAccumulator {
+    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+        write!(
+            f,
+            "ApproxTopKAccumulator(k={}, candidates={})",
+            self.k,
+            self.candidates.len()
+        )
+    }
+}
+
+impl ApproxTopKAccumulator {
+    fn new(k: usize, max_candidates: Option<usize>) -> Self {
+        // Cap at least k*2 for safety
+        let default_max = (k * 4).max(1000);
+        let max_candidates = max_candidates.unwrap_or(default_max);
+        Self {
+            candidates: HashMap::with_capacity(max_candidates),
+            k,
+            max_candidates,
+            min_count_threshold: 0,
+        }
+    }
+
+    /// Memory-efficient update that only keeps top candidates
+    fn update_with_pruning(&mut self, value: String, count: i64) {
+        // Periodically prune low-frequency items to save memory
+        if self.candidates.len() >= self.max_candidates {
+            self.prune_low_frequency_items();
+        }
+
+        // Update count
+        let entry = self.candidates.entry(value).or_insert(0);
+        *entry += count;
+    }
+
+    /// Remove low-frequency items to keep memory usage bounded
+    fn prune_low_frequency_items(&mut self) {
+        let target_size = (self.max_candidates / 2).max(self.k);
+
+        // First, remove items below the minimum threshold
+        self.candidates
+            .retain(|_, count| *count >= self.min_count_threshold);
+
+        if self.candidates.len() <= target_size {
+            return; // No need to prune
+        }
+
+        // Collect items with their counts
+        let mut items = self
+            .candidates
+            .iter()
+            .map(|(k, v)| (k, *v))
+            .collect::<Vec<_>>();
+
+        // Sort by count descending, then by key for deterministic results
+        items.sort_by_key(|k| std::cmp::Reverse(k.1));
+
+        // Update minimum threshold to the lowest count we're keeping
+        let mut item_iter = items.into_iter().skip(target_size - 1);
+        if let Some((_, count)) = item_iter.next() {
+            self.min_count_threshold = self.min_count_threshold.max(count);
+        }
+
+        // Keep only the top target_size items
+        let removed_items = item_iter.map(|(k, _)| k.clone()).collect::<Vec<_>>();
+        for key in removed_items {
+            self.candidates.remove(&key);
+        }
+    }
+
+    /// Get the top k elements as (value, count) pairs sorted by count descending
+    fn get_top_k(&self, n: usize) -> Vec<(String, i64)> {
+        let mut items: Vec<_> = self
+            .candidates
+            .iter()
+            .map(|(v, c)| (v.clone(), *c))
+            .collect();
+
+        // Sort by count descending, then by value ascending for deterministic results
+        items.sort_by_key(|k| std::cmp::Reverse(k.1));
+
+        items.into_iter().take(n).collect()
+    }
+
+    /// Convert string array to vector of strings
+    fn convert_to_strings(values: &ArrayRef) -> Result<Vec<String>> {
+        match values.data_type() {
+            DataType::Utf8 => {
+                let array = values.as_string::<i32>();
+                Ok(array
+                    .iter()
+                    .map(|v| v.unwrap_or_default().to_string())
+                    .collect())
+            }
+            DataType::LargeUtf8 => {
+                let array = values.as_string::<i64>();
+                Ok(array
+                    .iter()
+                    .map(|v| v.unwrap_or_default().to_string())
+                    .collect())
+            }
+            other => {
+                internal_err!("APPROX_TOPK received unexpected type {other:?}")
+            }
+        }
+    }
+
+    /// Convert int64 array to vector of counts
+    fn convert_to_counts(values: &ArrayRef) -> Result<Vec<i64>> {
+        let array = values.as_primitive::<arrow::datatypes::Int64Type>();
+        Ok(array.iter().map(|v| v.unwrap_or_default()).collect())
+    }
+}
+
+impl Accumulator for ApproxTopKAccumulator {
+    fn state(&mut self) -> Result<Vec<ScalarValue>> {
+        let pairs: Vec<_> = self.get_top_k(self.max_candidates);
+
+        let values = ScalarValue::List(ScalarValue::new_list_nullable(
+            &pairs
+                .iter()
+                .map(|(v, _)| ScalarValue::LargeUtf8(Some(v.to_string())))
+                .collect::<Vec<ScalarValue>>(),
+            &DataType::LargeUtf8,
+        ));
+
+        let counts = ScalarValue::List(ScalarValue::new_list_nullable(
+            &pairs
+                .iter()
+                .map(|(_, c)| ScalarValue::Int64(Some(*c)))
+                .collect::<Vec<ScalarValue>>(),
+            &DataType::Int64,
+        ));
+
+        let k_scalar = ScalarValue::Int64(Some(self.k as i64));
+
+        Ok(vec![values, counts, k_scalar])
+    }
+
+    fn evaluate(&mut self) -> Result<ScalarValue> {
+        let top_k = self.get_top_k(self.k);
+
+        if top_k.is_empty() {
+            return Ok(ScalarValue::List(ScalarValue::new_list_nullable(
+                &[],
+                &DataType::Struct(
+                    vec![
+                        Field::new("value", DataType::LargeUtf8, false),
+                        Field::new("count", DataType::Int64, false),
+                    ]
+                    .into(),
+                ),
+            )));
+        }
+
+        let values: Vec<Option<String>> = top_k.iter().map(|(v, _)| Some(v.clone())).collect();
+        let counts: Vec<Option<i64>> = top_k.iter().map(|(_, c)| Some(*c)).collect();
+
+        let value_array = Arc::new(LargeStringArray::from(values));
+        let count_array = Arc::new(Int64Array::from(counts));
+
+        let struct_array = StructArray::new(
+            Fields::from(vec![
+                Field::new("value", DataType::LargeUtf8, false),
+                Field::new("count", DataType::Int64, false),
+            ]),
+            vec![value_array as ArrayRef, count_array as ArrayRef],
+            None,
+        );
+
+        Ok(ScalarValue::List(ScalarValue::new_list_nullable(
+            &top_k
+                .into_iter()
+                .enumerate()
+                .map(|(i, _)| ScalarValue::Struct(Arc::new(struct_array.slice(i, 1))))
+                .collect::<Vec<ScalarValue>>(),
+            &DataType::Struct(
+                vec![
+                    Field::new("value", DataType::LargeUtf8, false),
+                    Field::new("count", DataType::Int64, false),
+                ]
+                .into(),
+            ),
+        )))
+    }
+
+    fn size(&self) -> usize {
+        // Estimate memory usage:
+        // - HashMap overhead + String keys + i64 values
+        // - Average string length ~20 bytes + HashMap overhead ~40 bytes per entry
+        self.candidates.len() * 60
+    }
+
+    fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
+        let strings = Self::convert_to_strings(&values[0])?;
+
+        // Count each string value
+        for value in strings {
+            self.update_with_pruning(value, 1);
+        }
+
+        Ok(())
+    }
+
+    fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {
+        if states.is_empty() {
+            return Ok(());
+        }
+
+        let values_list = states[0].as_list::<i32>();
+        let counts_list = states[1].as_list::<i32>();
+
+        for (values_opt, counts_opt) in values_list.iter().zip(counts_list.iter()) {
+            if let (Some(values_array), Some(counts_array)) = (values_opt, counts_opt) {
+                let values = Self::convert_to_strings(&values_array)?;
+                let counts = Self::convert_to_counts(&counts_array)?;
+
+                // Merge the counts from this state
+                for (value, count) in values.into_iter().zip(counts) {
+                    self.update_with_pruning(value, count);
+                }
+            }
+        }
+
+        Ok(())
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use arrow::array::StringArray;
+    use datafusion::{datasource::MemTable, logical_expr::AggregateUDF, prelude::SessionContext};
+
+    use super::*;
+
+    #[test]
+    fn test_approx_topk_accumulator() {
+        let mut acc = ApproxTopKAccumulator::new(3, None);
+
+        // Add some test data
+        let values = vec!["apple", "banana", "apple", "cherry", "banana", "apple"];
+        let string_array: ArrayRef = Arc::new(StringArray::from(values));
+
+        acc.update_batch(&[string_array]).unwrap();
+
+        // Evaluate should return top 3 by frequency
+        let result = acc.evaluate().unwrap();
+
+        // apple: 3, banana: 2, cherry: 1
+        assert!(matches!(result, ScalarValue::List(_)));
+    }
+
+    #[test]
+    fn test_memory_efficient_pruning() {
+        // Test that the accumulator prunes low-frequency items to save memory
+        let mut acc = ApproxTopKAccumulator::new(3, Some(20)); // Small limit for testing
+
+        // Add many different items with low frequency
+        for i in 0..100 {
+            let item = format!("low_freq_{i}");
+            let array: ArrayRef = Arc::new(StringArray::from(vec![item.as_str()]));
+            acc.update_batch(&[array]).unwrap();
+        }
+
+        // Should have pruned significantly
+        assert!(
+            acc.candidates.len() < 100,
+            "Should prune low-frequency items"
+        );
+
+        // Add some high-frequency items
+        for _ in 0..15 {
+            let array: ArrayRef = Arc::new(StringArray::from(vec!["very_frequent"]));
+            acc.update_batch(&[array]).unwrap();
+        }
+
+        for _ in 0..8 {
+            let array: ArrayRef = Arc::new(StringArray::from(vec!["medium_frequent"]));
+            acc.update_batch(&[array]).unwrap();
+        }
+
+        // Get top results
+        let top_k = acc.get_top_k(acc.k);
+        assert!(!top_k.is_empty());
+
+        // Most frequent items should be at the top
+        assert_eq!(top_k[0].0, "very_frequent");
+        assert_eq!(top_k[0].1, 15);
+
+        if top_k.len() > 1 {
+            assert_eq!(top_k[1].0, "medium_frequent");
+            assert_eq!(top_k[1].1, 8);
+        }
+
+        // Memory usage should be reasonable
+        assert!(acc.candidates.len() <= 50, "Memory usage should be bounded");
+    }
+
+    #[test]
+    fn test_accumulator_with_explicit_cap() {
+        // Test that the accumulator respects explicit cap parameter
+        let mut acc = ApproxTopKAccumulator::new(3, Some(5)); // Very small cap for testing
+
+        // Add items that would exceed the cap
+        let items = vec!["a", "b", "c", "d", "e", "f", "g", "h"];
+        for item in items {
+            let array: ArrayRef = Arc::new(StringArray::from(vec![item]));
+            acc.update_batch(&[array]).unwrap();
+        }
+
+        // Should respect the cap limit
+        assert!(
+            acc.candidates.len() <= 5,
+            "Should respect explicit cap parameter, got {} candidates",
+            acc.candidates.len()
+        );
+
+        // Add some items multiple times to create frequency differences
+        for _ in 0..10 {
+            let array: ArrayRef = Arc::new(StringArray::from(vec!["frequent"]));
+            acc.update_batch(&[array]).unwrap();
+        }
+
+        // Get results
+        let top_k = acc.get_top_k(acc.k);
+        assert!(!top_k.is_empty());
+
+        // Most frequent item should be at the top
+        assert_eq!(top_k[0].0, "frequent");
+        // Due to pruning with very small cap, count might be slightly less than 10
+        assert!(top_k[0].1 >= 9, "Expected count >= 9, got {}", top_k[0].1);
+    }
+
+    #[tokio::test]
+    async fn test_approx_topk_udaf() {
+        let ctx = SessionContext::new();
+
+        // Create test data
+        let schema = Schema::new(vec![Field::new("item", DataType::Utf8, false)]);
+
+        let values = vec![
+            "apple", "banana", "apple", "cherry", "banana", "apple", "date",
+        ];
+        let batch = RecordBatch::try_new(
+            Arc::new(schema.clone()),
+            vec![Arc::new(StringArray::from(values))],
+        )
+        .unwrap();
+
+        let table = MemTable::try_new(Arc::new(schema), vec![vec![batch]]).unwrap();
+        ctx.register_table("test_table", Arc::new(table)).unwrap();
+
+        // Register the UDAF
+        let topk_udaf = AggregateUDF::from(ApproxTopK::new());
+        ctx.register_udaf(topk_udaf);
+
+        // Test the function
+        let df = ctx
+            .sql("SELECT approx_topk(item, 2) as top_items FROM test_table")
+            .await
+            .unwrap();
+        let results = df.collect().await.unwrap();
+
+        assert_eq!(results.len(), 1);
+        assert_eq!(results[0].num_columns(), 1);
+        assert_eq!(results[0].num_rows(), 1);
+    }
+
+    #[tokio::test]
+    async fn test_approx_topk_udaf_with_cap() {
+        let ctx = SessionContext::new();
+
+        // Create test data
+        let schema = Schema::new(vec![Field::new("item", DataType::Utf8, false)]);
+
+        let values = vec![
+            "apple", "banana", "apple", "cherry", "banana", "apple", "date",
+        ];
+        let batch = RecordBatch::try_new(
+            Arc::new(schema.clone()),
+            vec![Arc::new(StringArray::from(values))],
+        )
+        .unwrap();
+
+        let table = MemTable::try_new(Arc::new(schema), vec![vec![batch]]).unwrap();
+        ctx.register_table("test_table", Arc::new(table)).unwrap();
+
+        // Register the UDAF
+        let topk_udaf = AggregateUDF::from(ApproxTopK::new());
+        ctx.register_udaf(topk_udaf);
+
+        // Test the function with cap parameter
+        let df = ctx
+            .sql("SELECT approx_topk(item, 2, 10) as top_items FROM test_table")
+            .await
+            .unwrap();
+        let results = df.collect().await.unwrap();
+
+        assert_eq!(results.len(), 1);
+        assert_eq!(results[0].num_columns(), 1);
+        assert_eq!(results[0].num_rows(), 1);
+    }
+}

src/search/src/datafusion/udaf/approx_topk_distinct.rs

@@ -0,0 +1,715 @@
+// Copyright 2026 OpenObserve Inc.
+//
+// This program is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Affero General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU Affero General Public License for more details.
+//
+// You should have received a copy of the GNU Affero General Public License
+// along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::{fmt::Formatter, sync::Arc};
+
+use arrow::{
+    array::{Array, AsArray, BinaryArray, LargeStringArray, RecordBatch, StructArray, UInt64Array},
+    datatypes::Fields,
+};
+use datafusion::{
+    arrow::{
+        array::ArrayRef,
+        datatypes::{DataType, Field, FieldRef, Schema},
+    },
+    common::{internal_err, not_impl_err, plan_err},
+    error::Result,
+    functions_aggregate::approx_distinct::ApproxDistinct,
+    logical_expr::{
+        Accumulator, AggregateUDFImpl, ColumnarValue, Signature, TypeSignature, Volatility,
+        function::{AccumulatorArgs, StateFieldsArgs},
+        utils::format_state_name,
+    },
+    physical_plan::{PhysicalExpr, expressions::col},
+    scalar::ScalarValue,
+};
+use hashbrown::HashMap;
+
+const APPROX_TOPK_DISTINCT: &str = "approx_topk_distinct";
+
+/// Approximate TopK UDAF that returns the top K elements by distinct count of another field.
+///
+/// Usage: approx_topk_distinct(top_field, value_field, k, [cap])
+/// - top_field: the field to find top k values from
+/// - value_field: the field to count distinct values for
+/// - k: number of top elements to return
+/// - cap: optional maximum number of candidates to keep in memory (default: max(k*4, 1000))
+///
+/// This function finds the top K values in top_field, ranked by how many unique values
+/// they have in the corresponding value_field. For example:
+/// - If you have data with (user_id, session_id) pairs
+/// - approx_topk_distinct(user_id, session_id, 10) returns the top 10 users with the most distinct
+///   sessions
+///
+/// Uses HyperLogLog for exact distinct counting (memory-efficient for reasonable cardinalities).
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct ApproxTopKDistinct(Signature);
+
+impl ApproxTopKDistinct {
+    pub fn new() -> Self {
+        Self(Signature::one_of(
+            vec![
+                // top_field, value_field, k
+                TypeSignature::Exact(vec![DataType::Utf8, DataType::Utf8, DataType::Int64]),
+                TypeSignature::Exact(vec![DataType::LargeUtf8, DataType::Utf8, DataType::Int64]),
+                TypeSignature::Exact(vec![DataType::Utf8, DataType::LargeUtf8, DataType::Int64]),
+                TypeSignature::Exact(vec![
+                    DataType::LargeUtf8,
+                    DataType::LargeUtf8,
+                    DataType::Int64,
+                ]),
+                // top_field, value_field, k, cap
+                TypeSignature::Exact(vec![
+                    DataType::Utf8,
+                    DataType::Utf8,
+                    DataType::Int64,
+                    DataType::Int64,
+                ]),
+                TypeSignature::Exact(vec![
+                    DataType::LargeUtf8,
+                    DataType::Utf8,
+                    DataType::Int64,
+                    DataType::Int64,
+                ]),
+                TypeSignature::Exact(vec![
+                    DataType::Utf8,
+                    DataType::LargeUtf8,
+                    DataType::Int64,
+                    DataType::Int64,
+                ]),
+                TypeSignature::Exact(vec![
+                    DataType::LargeUtf8,
+                    DataType::LargeUtf8,
+                    DataType::Int64,
+                    DataType::Int64,
+                ]),
+            ],
+            Volatility::Immutable,
+        ))
+    }
+}
+
+impl Default for ApproxTopKDistinct {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl AggregateUDFImpl for ApproxTopKDistinct {
+    fn name(&self) -> &str {
+        APPROX_TOPK_DISTINCT
+    }
+
+    fn signature(&self) -> &datafusion::logical_expr::Signature {
+        &self.0
+    }
+
+    fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+        match &arg_types[0] {
+            DataType::Utf8 | DataType::LargeUtf8 => {
+                // Return array of structs: [{value: string, count: int64}]
+                Ok(DataType::List(Arc::new(Field::new(
+                    "item",
+                    DataType::Struct(
+                        vec![
+                            Field::new("value", DataType::LargeUtf8, false),
+                            Field::new("count", DataType::UInt64, false),
+                        ]
+                        .into(),
+                    ),
+                    true,
+                ))))
+            }
+            _ => plan_err!("approx_topk_distinct requires string input types"),
+        }
+    }
+
+    fn state_fields(&self, args: StateFieldsArgs) -> Result<Vec<FieldRef>> {
+        Ok(vec![
+            // Store top_field values as list of strings
+            Arc::new(Field::new(
+                format_state_name(args.name, "top_values"),
+                DataType::List(Arc::new(Field::new("item", DataType::LargeUtf8, true))),
+                true,
+            )),
+            // Store distinct values as list of list of HLL registers
+            Arc::new(Field::new(
+                format_state_name(args.name, "hll_registers"),
+                DataType::List(Arc::new(Field::new("item", DataType::LargeBinary, true))),
+                true,
+            )),
+            // Store k parameter
+            Arc::new(Field::new(
+                format_state_name(args.name, "k"),
+                DataType::Int64,
+                false,
+            )),
+        ])
+    }
+
+    fn accumulator(&self, args: AccumulatorArgs) -> Result<Box<dyn Accumulator>> {
+        let k = validate_k_parameter(&args.exprs[2])?;
+        let cap = if args.exprs.len() > 3 {
+            Some(validate_cap_parameter(&args.exprs[3])?)
+        } else {
+            None
+        };
+
+        let top_field_data_type = args.exprs[0].data_type(args.schema)?;
+        let value_field_data_type = args.exprs[1].data_type(args.schema)?;
+
+        match (&top_field_data_type, &value_field_data_type) {
+            (DataType::Utf8 | DataType::LargeUtf8, DataType::Utf8 | DataType::LargeUtf8) => {
+                Ok(Box::new(ApproxTopKDistinctAccumulator::new(k, cap)))
+            }
+            (other_top, other_value) => {
+                not_impl_err!(
+                    "Support for 'APPROX_TOPK_DISTINCT' for data types {other_top:?}, {other_value:?} is not implemented"
+                )
+            }
+        }
+    }
+}
+
+fn validate_k_parameter(expr: &Arc<dyn PhysicalExpr>) -> Result<usize> {
+    let empty_schema = Arc::new(Schema::empty());
+    let batch = RecordBatch::new_empty(Arc::clone(&empty_schema));
+
+    let k = match expr.evaluate(&batch)? {
+        ColumnarValue::Scalar(ScalarValue::Int64(Some(value))) => {
+            if value <= 0 {
+                return plan_err!(
+                    "k parameter for 'APPROX_TOPK_DISTINCT' must be positive, got {value}"
+                );
+            }
+            value as usize
+        }
+        ColumnarValue::Scalar(other) => {
+            return not_impl_err!(
+                "k parameter for 'APPROX_TOPK_DISTINCT' must be Int64 literal (got {:?})",
+                other.data_type()
+            );
+        }
+        _ => {
+            return internal_err!("Expected scalar value for k parameter");
+        }
+    };
+
+    Ok(k)
+}
+
+fn validate_cap_parameter(expr: &Arc<dyn PhysicalExpr>) -> Result<usize> {
+    let empty_schema = Arc::new(Schema::empty());
+    let batch = RecordBatch::new_empty(Arc::clone(&empty_schema));
+
+    let cap = match expr.evaluate(&batch)? {
+        ColumnarValue::Scalar(ScalarValue::Int64(Some(value))) => {
+            if value <= 0 {
+                return plan_err!(
+                    "cap parameter for 'APPROX_TOPK_DISTINCT' must be positive, got {value}"
+                );
+            }
+            value as usize
+        }
+        ColumnarValue::Scalar(other) => {
+            return not_impl_err!(
+                "cap parameter for 'APPROX_TOPK_DISTINCT' must be Int64 literal (got {:?})",
+                other.data_type()
+            );
+        }
+        _ => {
+            return internal_err!("Expected scalar value for cap parameter");
+        }
+    };
+
+    Ok(cap)
+}
+
+/// Accumulator that tracks top K values by distinct count of another field
+/// Uses HyperLogLog for exact distinct counting (good for reasonable cardinalities)
+struct ApproxTopKDistinctAccumulator {
+    // Map from top_field value to HyperLogLog accumulator
+    candidates: HashMap<String, Box<dyn Accumulator>>,
+    k: usize,
+    // Memory management
+    max_candidates: usize,    // Maximum candidates to keep in memory
+    min_count_threshold: u64, // Minimum count to be considered
+}
+
+impl std::fmt::Debug for ApproxTopKDistinctAccumulator {
+    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+        write!(
+            f,
+            "ApproxTopKDistinctAccumulator(k={}, candidates={})",
+            self.k,
+            self.candidates.len()
+        )
+    }
+}
+
+impl ApproxTopKDistinctAccumulator {
+    fn new(k: usize, max_candidates: Option<usize>) -> Self {
+        // Cap at least k*4 for safety
+        let default_max = (k * 4).max(1000);
+        let max_candidates = max_candidates.unwrap_or(default_max);
+        Self {
+            candidates: HashMap::with_capacity(max_candidates),
+            k,
+            max_candidates,
+            min_count_threshold: 0,
+        }
+    }
+
+    fn new_acc_args() -> Option<Box<dyn Accumulator>> {
+        let schema = Arc::new(Schema::new(vec![Field::new(
+            "f",
+            DataType::LargeUtf8,
+            true,
+        )]));
+        let acc_args = AccumulatorArgs {
+            return_field: Arc::new(Field::new("f", DataType::UInt64, true)),
+            schema: &schema,
+            ignore_nulls: false,
+            order_bys: &[],
+            is_reversed: false,
+            name: "APPROX_DISTINCT(f)",
+            is_distinct: false,
+            exprs: &[col("f", &schema).unwrap()],
+            expr_fields: &[Arc::new(Field::new("f", DataType::LargeUtf8, true))],
+        };
+        ApproxDistinct::new().accumulator(acc_args).ok()
+    }
+
+    /// Memory-efficient update that only keeps top candidates
+    fn update_with_pruning(&mut self, value: String, distinct_values: Vec<String>) {
+        // Periodically prune low-frequency items to save memory
+        if self.candidates.len() >= self.max_candidates {
+            self.prune_low_frequency_items();
+        }
+
+        // Update count
+        self.candidates
+            .entry(value)
+            .or_insert_with(|| Self::new_acc_args().unwrap())
+            .update_batch(&[Arc::new(LargeStringArray::from(distinct_values))])
+            .unwrap();
+    }
+
+    /// Remove low-frequency items to keep memory usage bounded
+    fn prune_low_frequency_items(&mut self) {
+        let target_size = (self.max_candidates / 2).max(self.k);
+
+        // Collect items with their counts
+        let mut items = self
+            .candidates
+            .iter_mut()
+            .map(|(k, v)| (k, Self::get_distinct_count(v)))
+            .collect::<Vec<_>>();
+
+        // Sort by count descending, then by key for deterministic results
+        items.sort_by_key(|k| std::cmp::Reverse(k.1));
+
+        // Update minimum threshold to the lowest count we're keeping
+        let mut item_iter = items.into_iter().skip(target_size - 1);
+        if let Some((_, count)) = item_iter.next() {
+            self.min_count_threshold = self.min_count_threshold.max(count);
+        }
+
+        // Keep only the top target_size items
+        let removed_items = item_iter.map(|(k, _)| k.clone()).collect::<Vec<_>>();
+        for key in removed_items {
+            self.candidates.remove(&key);
+        }
+    }
+
+    /// Get top k entries by distinct count
+    fn get_top_k(&mut self, n: usize) -> Vec<(String, u64)> {
+        let mut items: Vec<(String, u64)> = self
+            .candidates
+            .iter_mut()
+            .map(|(top_value, acc)| (top_value.clone(), Self::get_distinct_count(acc)))
+            .collect();
+
+        // Sort by distinct count descending, then by value ascending for deterministic results
+        items.sort_by_key(|k| std::cmp::Reverse(k.1));
+
+        items.into_iter().take(n).collect()
+    }
+
+    /// Get distinct count from a ApproxDistinct accumulator
+    fn get_distinct_count(distinct_acc: &mut Box<dyn Accumulator>) -> u64 {
+        distinct_acc
+            .evaluate()
+            .map(|v| {
+                if let ScalarValue::UInt64(Some(count)) = v {
+                    count
+                } else {
+                    0
+                }
+            })
+            .ok()
+            .unwrap_or(0)
+    }
+
+    /// Convert string array to vector of strings
+    fn convert_to_strings(values: &ArrayRef) -> Result<Vec<String>> {
+        match values.data_type() {
+            DataType::Utf8 => {
+                let array = values.as_string::<i32>();
+                Ok(array
+                    .iter()
+                    .map(|v| v.unwrap_or_default().to_string())
+                    .collect())
+            }
+            DataType::LargeUtf8 => {
+                let array = values.as_string::<i64>();
+                Ok(array
+                    .iter()
+                    .map(|v| v.unwrap_or_default().to_string())
+                    .collect())
+            }
+            other => {
+                internal_err!("APPROX_TOPK_DISTINCT received unexpected type {other:?}")
+            }
+        }
+    }
+
+    /// Convert string array to vector of binary arrays
+    fn convert_to_binary(values: &ArrayRef) -> Result<Vec<Vec<u8>>> {
+        match values.data_type() {
+            DataType::Binary => {
+                let array = values.as_binary::<i32>();
+                Ok(array
+                    .iter()
+                    .map(|v| v.unwrap_or_default().to_vec())
+                    .collect::<Vec<_>>())
+            }
+            DataType::LargeBinary => {
+                let array = values.as_binary::<i64>();
+                Ok(array
+                    .iter()
+                    .map(|v| v.unwrap_or_default().to_vec())
+                    .collect::<Vec<_>>())
+            }
+            other => {
+                internal_err!("APPROX_TOPK_DISTINCT received unexpected type {other:?}")
+            }
+        }
+    }
+
+    fn convert_to_large_binary(value: ScalarValue) -> Result<ScalarValue> {
+        match value {
+            ScalarValue::Binary(v) => Ok(ScalarValue::LargeBinary(v)),
+            ScalarValue::LargeBinary(_) => Ok(value),
+            other => internal_err!("APPROX_TOPK_DISTINCT received unexpected type {other:?}"),
+        }
+    }
+}
+
+impl Accumulator for ApproxTopKDistinctAccumulator {
+    fn state(&mut self) -> Result<Vec<ScalarValue>> {
+        // Get top entries for state serialization
+        let top_entries = self.get_top_k(self.max_candidates);
+
+        let values: Vec<ScalarValue> = top_entries
+            .iter()
+            .map(|(v, _)| ScalarValue::LargeUtf8(Some(v.clone())))
+            .collect();
+
+        // Serialize HyperLogLog accumulators as lists of binary arrays
+        let distinct_values: Vec<ScalarValue> = top_entries
+            .iter()
+            .filter_map(|(top_val, _)| {
+                if let Some(acc) = self.candidates.get_mut(top_val) {
+                    acc.state()
+                        .ok()
+                        .and_then(|mut v| v.pop().map(|v| Self::convert_to_large_binary(v).ok()))
+                        .flatten()
+                } else {
+                    None
+                }
+            })
+            .collect();
+
+        let values_list = ScalarValue::List(ScalarValue::new_list_nullable(
+            &values,
+            &DataType::LargeUtf8,
+        ));
+
+        let distinct_values_list = ScalarValue::List(ScalarValue::new_list_nullable(
+            &distinct_values,
+            &DataType::LargeBinary,
+        ));
+
+        let k_scalar = ScalarValue::Int64(Some(self.k as i64));
+
+        Ok(vec![values_list, distinct_values_list, k_scalar])
+    }
+
+    fn evaluate(&mut self) -> Result<ScalarValue> {
+        let top_k = self.get_top_k(self.k);
+
+        if top_k.is_empty() {
+            return Ok(ScalarValue::List(ScalarValue::new_list_nullable(
+                &[],
+                &DataType::Struct(
+                    vec![
+                        Field::new("value", DataType::LargeUtf8, false),
+                        Field::new("count", DataType::UInt64, false),
+                    ]
+                    .into(),
+                ),
+            )));
+        }
+
+        let values: Vec<Option<String>> = top_k.iter().map(|(v, _)| Some(v.clone())).collect();
+        let counts: Vec<Option<u64>> = top_k.iter().map(|(_, c)| Some(*c)).collect();
+
+        let value_array = Arc::new(LargeStringArray::from(values));
+        let count_array = Arc::new(UInt64Array::from(counts));
+
+        let struct_array = StructArray::new(
+            Fields::from(vec![
+                Field::new("value", DataType::LargeUtf8, false),
+                Field::new("count", DataType::UInt64, false),
+            ]),
+            vec![value_array as ArrayRef, count_array as ArrayRef],
+            None,
+        );
+
+        Ok(ScalarValue::List(ScalarValue::new_list_nullable(
+            &top_k
+                .into_iter()
+                .enumerate()
+                .map(|(i, _)| ScalarValue::Struct(Arc::new(struct_array.slice(i, 1))))
+                .collect::<Vec<ScalarValue>>(),
+            &DataType::Struct(
+                vec![
+                    Field::new("value", DataType::LargeUtf8, false),
+                    Field::new("count", DataType::UInt64, false),
+                ]
+                .into(),
+            ),
+        )))
+    }
+
+    fn size(&self) -> usize {
+        // Estimate memory usage: HashMap overhead + HLL registers sizes
+        let mut total_size = self.candidates.len() * 64; // HashMap overhead
+        for (key, acc) in &self.candidates {
+            total_size += key.len(); // Key size
+            total_size += acc.size(); // HLL registers size
+        }
+        total_size
+    }
+
+    fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
+        let top_field_strings = Self::convert_to_strings(&values[0])?;
+        let value_field_strings = Self::convert_to_strings(&values[1])?;
+
+        // Ensure both arrays have the same length
+        if top_field_strings.len() != value_field_strings.len() {
+            return internal_err!("Top field and value field arrays must have the same length");
+        }
+
+        // Partition distinct values for each top_field value
+        let mut distinct_values = HashMap::with_capacity(top_field_strings.len());
+        for (top_value, distinct_value) in top_field_strings.into_iter().zip(value_field_strings) {
+            // self.update_with_pruning(top_value, distinct_value);
+            distinct_values
+                .entry(top_value)
+                .or_insert(vec![])
+                .push(distinct_value);
+        }
+
+        for (top_value, distinct_values) in distinct_values {
+            self.update_with_pruning(top_value, distinct_values);
+        }
+
+        Ok(())
+    }
+
+    fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {
+        if states.is_empty() {
+            return Ok(());
+        }
+
+        let values_list = states[0].as_list::<i32>();
+        let distinct_values_list = states[1].as_list::<i32>();
+
+        for (values_opt, distinct_values_opt) in values_list.iter().zip(distinct_values_list.iter())
+        {
+            if let (Some(values_array), Some(distinct_values_array)) =
+                (values_opt, distinct_values_opt)
+            {
+                let values = Self::convert_to_strings(&values_array)?;
+                let distinct_values = Self::convert_to_binary(&distinct_values_array)?;
+
+                // Merge Hll registers for each top_field value
+                for (value, distinct_value) in values.into_iter().zip(distinct_values) {
+                    let distinct_acc = self
+                        .candidates
+                        .entry(value)
+                        .or_insert_with(|| Self::new_acc_args().unwrap());
+
+                    // Merge all distinct values
+                    distinct_acc
+                        .merge_batch(&[Arc::new(BinaryArray::from_vec(vec![&distinct_value]))])
+                        .unwrap();
+                }
+            }
+
+            // Check if we need to prune after merging
+            if self.candidates.len() > self.max_candidates {
+                self.prune_low_frequency_items();
+            }
+        }
+
+        Ok(())
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use arrow::array::StringArray;
+    use datafusion::{datasource::MemTable, logical_expr::AggregateUDF, prelude::SessionContext};
+
+    use super::*;
+
+    #[test]
+    fn test_approx_topk_distinct_accumulator() {
+        let mut acc = ApproxTopKDistinctAccumulator::new(3, None);
+
+        // Test data: (top_field, value_field)
+        // user1 has sessions: [session1, session2, session1] -> 2 distinct
+        // user2 has sessions: [session3] -> 1 distinct
+        // user3 has sessions: [session4, session5, session6] -> 3 distinct
+
+        let top_field_values = vec![
+            "user1", "user1", "user1", "user2", "user3", "user3", "user3",
+        ];
+        let value_field_values = vec![
+            "session1", "session2", "session1", "session3", "session4", "session5", "session6",
+        ];
+
+        let top_field_array: ArrayRef = Arc::new(StringArray::from(top_field_values));
+        let value_field_array: ArrayRef = Arc::new(StringArray::from(value_field_values));
+
+        acc.update_batch(&[top_field_array, value_field_array])
+            .unwrap();
+
+        // Get top 3 results
+        let top_k = acc.get_top_k(3);
+
+        assert_eq!(top_k.len(), 3);
+        assert!(top_k[0].1 >= top_k[1].1); // Results should be sorted by distinct count descending
+
+        // user3 should have the highest distinct count (3)
+        // user1 should have 2 distinct sessions
+        // user2 should have 1 distinct session
+        assert_eq!(top_k[0].0, "user3");
+        assert_eq!(top_k[0].1, 3);
+        assert_eq!(top_k[1].0, "user1");
+        assert_eq!(top_k[1].1, 2);
+        assert_eq!(top_k[2].0, "user2");
+        assert_eq!(top_k[2].1, 1);
+    }
+
+    #[tokio::test]
+    async fn test_approx_topk_distinct_udaf() {
+        let ctx = SessionContext::new();
+
+        // Create test data
+        let schema = Schema::new(vec![
+            Field::new("user_id", DataType::Utf8, false),
+            Field::new("session_id", DataType::Utf8, false),
+        ]);
+
+        let users = vec![
+            "user1", "user1", "user1", "user2", "user3", "user3", "user3",
+        ];
+        let sessions = vec![
+            "session1", "session2", "session1", "session3", "session4", "session5", "session6",
+        ];
+
+        let batch = RecordBatch::try_new(
+            Arc::new(schema.clone()),
+            vec![
+                Arc::new(StringArray::from(users)),
+                Arc::new(StringArray::from(sessions)),
+            ],
+        )
+        .unwrap();
+
+        let table = MemTable::try_new(Arc::new(schema), vec![vec![batch]]).unwrap();
+        ctx.register_table("test_table", Arc::new(table)).unwrap();
+
+        // Register the UDAF
+        let topk_distinct_udaf = AggregateUDF::from(ApproxTopKDistinct::new());
+        ctx.register_udaf(topk_distinct_udaf);
+
+        // Test the function
+        let df = ctx
+            .sql("SELECT approx_topk_distinct(user_id, session_id, 2) as top_users FROM test_table")
+            .await
+            .unwrap();
+        let results = df.collect().await.unwrap();
+
+        assert_eq!(results.len(), 1);
+        assert_eq!(results[0].num_columns(), 1);
+        assert_eq!(results[0].num_rows(), 1);
+    }
+
+    #[tokio::test]
+    async fn test_approx_topk_distinct_udaf_with_cap() {
+        let ctx = SessionContext::new();
+
+        // Create test data
+        let schema = Schema::new(vec![
+            Field::new("user_id", DataType::Utf8, false),
+            Field::new("session_id", DataType::Utf8, false),
+        ]);
+
+        let users = vec!["user1", "user1", "user2", "user3", "user3"];
+        let sessions = vec!["session1", "session2", "session3", "session4", "session5"];
+
+        let batch = RecordBatch::try_new(
+            Arc::new(schema.clone()),
+            vec![
+                Arc::new(StringArray::from(users)),
+                Arc::new(StringArray::from(sessions)),
+            ],
+        )
+        .unwrap();
+
+        let table = MemTable::try_new(Arc::new(schema), vec![vec![batch]]).unwrap();
+        ctx.register_table("test_table", Arc::new(table)).unwrap();
+
+        // Register the UDAF
+        let topk_distinct_udaf = AggregateUDF::from(ApproxTopKDistinct::new());
+        ctx.register_udaf(topk_distinct_udaf);
+
+        // Test the function with cap parameter
+        let df = ctx
+            .sql("SELECT approx_topk_distinct(user_id, session_id, 2, 10) as top_users FROM test_table")
+            .await
+            .unwrap();
+        let results = df.collect().await.unwrap();
+
+        assert_eq!(results.len(), 1);
+        assert_eq!(results[0].num_columns(), 1);
+        assert_eq!(results[0].num_rows(), 1);
+    }
+}

src/search/src/datafusion/udaf/mod.rs

@@ -15,6 +15,8 @@
 
 use arrow_schema::DataType;
 
+pub mod approx_topk;
+pub mod approx_topk_distinct;
 pub mod summary_percentile;
 
 pub static NUMERICS: &[DataType] = &[

src/search/src/datafusion/udf/transform_udf.rs

@@ -47,9 +47,23 @@ fn create_user_df(fn_name: &str, num_args: u8, pow_scalar: FnType) -> ScalarUDF
 }
 
 pub fn get_all_transform(org_id: &str) -> Result<Vec<ScalarUDF>> {
+    // Keys are "{org_id}/{fn_name}" (see get_transforms in pipeline execution),
+    // so the org must be matched as a PREFIX up to the separator.
+    //
+    // This previously used `key().contains(org_id)`, a substring match that
+    // collided three ways: an org whose id prefixed another's ("acme" matching
+    // "acme-prod/..."), an org whose id appeared mid-key ("prod" matching
+    // "acme-prod/..."), and — because the match was not confined to the org
+    // segment at all — an org whose id appeared in any FUNCTION name anywhere
+    // ("parse" matching "otherorg/parse_json"). The result was another
+    // tenant's VRL functions being registered into this org's query context,
+    // and same-named functions from two orgs shadowing each other in
+    // DashMap-iteration order. Mirrors get_all_transform_keys, which was
+    // already correct.
+    let org_prefix = format!("{org_id}/");
     let mut udf_list = Vec::new();
     for transform in QUERY_FUNCTIONS.iter() {
-        if transform.key().contains(org_id) {
+        if transform.key().starts_with(&org_prefix) {
             udf_list.push(get_udf_vrl(
                 transform.name.clone(),
                 transform.function.as_str(),
@@ -139,6 +153,64 @@ pub fn apply_vrl_fn(runtime: &mut Runtime, program: vrl::compiler::Program) -> j
 
 #[cfg(test)]
 mod tests {
+    use config::meta::function::Transform;
+
+    fn seed(key: &str, name: &str) {
+        QUERY_FUNCTIONS.insert(
+            key.to_string(),
+            Transform {
+                function: ".".to_string(),
+                name: name.to_string(),
+                params: "row".to_string(),
+                num_args: 1,
+                trans_type: Some(0),
+                streams: None,
+            },
+        );
+    }
+
+    #[test]
+    fn get_all_transform_matches_the_org_prefix_not_a_substring() {
+        seed("acme/acme_fn", "acme_fn");
+        seed("acme-prod/prod_fn", "prod_fn");
+        seed("otherorg/parse_json", "parse_json");
+
+        let names = |org: &str| -> Vec<String> {
+            get_all_transform(org)
+                .unwrap()
+                .iter()
+                .map(|u| u.name().to_string())
+                .collect()
+        };
+
+        let acme = names("acme");
+        assert!(
+            acme.contains(&"acme_fn".to_string()),
+            "own transform missing"
+        );
+        assert!(
+            !acme.contains(&"prod_fn".to_string()),
+            "org-prefix collision: acme must not see acme-prod's transform"
+        );
+
+        let prod = names("acme-prod");
+        assert!(prod.contains(&"prod_fn".to_string()));
+        assert!(!prod.contains(&"acme_fn".to_string()));
+
+        // The org id must not match inside a FUNCTION name either.
+        assert!(
+            names("parse").is_empty(),
+            "function-name collision: an org named `parse` must not see otherorg/parse_json"
+        );
+
+        // A bare org id that is a prefix without the separator must not match.
+        assert!(names("acm").is_empty(), "partial org id must not match");
+
+        QUERY_FUNCTIONS.remove("acme/acme_fn");
+        QUERY_FUNCTIONS.remove("acme-prod/prod_fn");
+        QUERY_FUNCTIONS.remove("otherorg/parse_json");
+    }
+
     use datafusion::{
         arrow::{
             array::Int64Array,

src/search/src/lib.rs

@@ -14,6 +14,7 @@
 // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 pub mod bloom_pruner;
+pub mod cache;
 pub mod datafusion;
 pub mod file_cache;
 pub mod index;
@@ -306,178 +307,3 @@ mod tests {
         }
     }
 }
-
-#[cfg(test)]
-#[cfg(feature = "enterprise")]
-mod enterprise_tests {
-    use arrow::array::record_batch;
-    use arrow_schema::{DataType, Field};
-    use o2_enterprise::enterprise::common::streaming_agg_cache::{
-        StreamingAggsCacheResultRecordBatch, calculate_record_batches_deltas,
-    };
-
-    #[test]
-    fn test_calculate_record_batches_deltas() {
-        let batch1 = record_batch!(
-            ("status", Utf8, ["200", "404"]),
-            ("count", Int64, [100, 50])
-        )
-        .unwrap();
-
-        let batch2 =
-            record_batch!(("status", Utf8, ["200", "500"]), ("count", Int64, [80, 20])).unwrap();
-
-        // Test case: Query: 10:00 - 16:00, Cache: 11:00 - 12:00, 14:00 - 15:00
-        // Expected Deltas: 10:00 - 11:00, 12:00 - 14:00, 15:00 - 16:00
-        let cache_result = vec![
-            StreamingAggsCacheResultRecordBatch {
-                record_batch: batch1,
-                cache_start_time: 11_000_000, // 11:00 in microseconds
-                cache_end_time: 12_000_000,   // 12:00 in microseconds
-            },
-            StreamingAggsCacheResultRecordBatch {
-                record_batch: batch2,
-                cache_start_time: 14_000_000, // 14:00 in microseconds
-                cache_end_time: 15_000_000,   // 15:00 in microseconds
-            },
-        ];
-
-        let query_start_time = 10_000_000; // 10:00 in microseconds
-        let query_end_time = 16_000_000; // 16:00 in microseconds
-
-        let deltas =
-            calculate_record_batches_deltas(query_start_time, query_end_time, &cache_result);
-
-        // Should have 3 deltas
-        assert_eq!(deltas.len(), 3);
-
-        // Delta 1: 10:00 - 11:00 (before first cache)
-        assert_eq!(deltas[0].delta_start_time, 10_000_000);
-        assert_eq!(deltas[0].delta_end_time, 11_000_000);
-
-        // Delta 2: 12:00 - 14:00 (between caches)
-        assert_eq!(deltas[1].delta_start_time, 12_000_000);
-        assert_eq!(deltas[1].delta_end_time, 14_000_000);
-
-        // Delta 3: 15:00 - 16:00 (after last cache)
-        assert_eq!(deltas[2].delta_start_time, 15_000_000);
-        assert_eq!(deltas[2].delta_end_time, 16_000_000);
-    }
-
-    #[test]
-    fn test_calculate_record_batches_deltas_without_cache() {
-        // Test case: No cache, entire query range should be a delta
-        let cache_result = vec![];
-        let query_start_time = 10_000_000;
-        let query_end_time = 16_000_000;
-
-        let deltas =
-            calculate_record_batches_deltas(query_start_time, query_end_time, &cache_result);
-
-        assert_eq!(deltas.len(), 1);
-        assert_eq!(deltas[0].delta_start_time, 10_000_000);
-        assert_eq!(deltas[0].delta_end_time, 16_000_000);
-    }
-
-    #[test]
-    fn test_calculate_record_batches_deltas_complete_cache() {
-        use std::sync::Arc;
-
-        use arrow::{
-            array::{Int64Array, StringArray},
-            datatypes::Schema,
-        };
-
-        let schema = Arc::new(Schema::new(vec![
-            Field::new("status", DataType::Utf8, false),
-            Field::new("count", DataType::Int64, false),
-        ]));
-
-        let batch = arrow::array::RecordBatch::try_new(
-            schema.clone(),
-            vec![
-                Arc::new(StringArray::from(vec!["200"])),
-                Arc::new(Int64Array::from(vec![100])),
-            ],
-        )
-        .unwrap();
-
-        // Test case: Cache covers entire query range
-        let cache_result = vec![StreamingAggsCacheResultRecordBatch {
-            record_batch: batch,
-            cache_start_time: 10_000_000, // Same as query start
-            cache_end_time: 16_000_000,   // Same as query end
-        }];
-
-        let query_start_time = 10_000_000;
-        let query_end_time = 16_000_000;
-
-        let deltas =
-            calculate_record_batches_deltas(query_start_time, query_end_time, &cache_result);
-
-        // Should have no deltas (complete cache hit)
-        assert_eq!(deltas.len(), 0);
-    }
-
-    #[test]
-    fn test_calculate_record_batches_deltas_unsorted_cache() {
-        use std::sync::Arc;
-
-        use arrow::{
-            array::{Int64Array, StringArray},
-            datatypes::Schema,
-        };
-
-        let schema = Arc::new(Schema::new(vec![
-            Field::new("status", DataType::Utf8, false),
-            Field::new("count", DataType::Int64, false),
-        ]));
-
-        let batch1 = arrow::array::RecordBatch::try_new(
-            schema.clone(),
-            vec![
-                Arc::new(StringArray::from(vec!["200"])),
-                Arc::new(Int64Array::from(vec![100])),
-            ],
-        )
-        .unwrap();
-
-        let batch2 = arrow::array::RecordBatch::try_new(
-            schema.clone(),
-            vec![
-                Arc::new(StringArray::from(vec!["404"])),
-                Arc::new(Int64Array::from(vec![50])),
-            ],
-        )
-        .unwrap();
-
-        // Test case: Cache results in wrong order (should be sorted internally)
-        let cache_result = vec![
-            StreamingAggsCacheResultRecordBatch {
-                record_batch: batch1,
-                cache_start_time: 14_000_000, // Second cache range
-                cache_end_time: 15_000_000,
-            },
-            StreamingAggsCacheResultRecordBatch {
-                record_batch: batch2,
-                cache_start_time: 11_000_000, // First cache range
-                cache_end_time: 12_000_000,
-            },
-        ];
-
-        let query_start_time = 10_000_000;
-        let query_end_time = 16_000_000;
-
-        let deltas =
-            calculate_record_batches_deltas(query_start_time, query_end_time, &cache_result);
-
-        // Should still produce correct deltas despite unsorted input
-        assert_eq!(deltas.len(), 3);
-        assert_eq!(deltas[0].delta_start_time, 10_000_000); // Before first
-        assert_eq!(deltas[0].delta_end_time, 11_000_000);
-        assert_eq!(deltas[1].delta_start_time, 12_000_000); // Between
-        assert_eq!(deltas[1].delta_end_time, 14_000_000);
-        assert_eq!(deltas[2].delta_start_time, 15_000_000); // After last
-        assert_eq!(deltas[2].delta_end_time, 16_000_000);
-    }
-}

src/search/src/sql/rewriter/mod.rs

@@ -20,3 +20,19 @@ pub mod add_timestamp;
 pub mod match_all_raw;
 pub mod remove_dashboard_placeholder;
 pub mod track_total_hits;
+
+/// Function names that are valid user-facing SQL but appear in NO registry,
+/// because a rewriter desugars them before planning.
+///
+/// Exported so the function catalog served to the query editor can union them
+/// in. Built from `registered_function_names()` alone, the catalog would
+/// silently drop functions users rely on today.
+pub const REWRITER_FUNCTION_ALIASES: &[&str] = &["match_all_raw", "match_all_raw_ignore_case"];
+
+/// The canonical function each alias is rewritten to.
+pub fn rewriter_alias_target(alias: &str) -> Option<&'static str> {
+    match alias {
+        "match_all_raw" | "match_all_raw_ignore_case" => Some("match_all"),
+        _ => None,
+    }
+}

src/search/src/sql/visitor/mod.rs

@@ -16,10 +16,10 @@
 #[cfg(feature = "enterprise")]
 pub mod cipher_key;
 pub mod column;
-#[cfg(feature = "enterprise")]
 pub mod group_by;
 pub mod histogram_interval;
 pub mod match_all;
 pub mod partition_column;
 pub mod pickup_where;
+pub mod streaming_aggregate;
 pub mod utils;

src/search/src/sql/visitor/utils.rs

@@ -16,7 +16,15 @@
 use std::{collections::HashSet, ops::ControlFlow};
 
 use datafusion::sql::TableReference;
-use sqlparser::ast::{Expr, Ident, VisitorMut};
+use sqlparser::ast::{Expr, Ident, ObjectNamePart, VisitorMut};
+
+/// Extract the identifier value from a SQL object-name component.
+pub(super) fn get_object_name_value(part: &ObjectNamePart) -> String {
+    match part {
+        ObjectNamePart::Identifier(ident) => ident.value.clone(),
+        ObjectNamePart::Function(_) => "__UNKNOWN_FUNCTION__".to_string(),
+    }
+}
 
 pub struct FieldNameVisitor {
     pub field_names: HashSet<String>,

src/search/src/tantivy/search.rs

@@ -16,22 +16,18 @@
 use std::{collections::HashSet, sync::Arc};
 
 use arrow::buffer::BooleanBuffer;
-#[cfg(not(feature = "enterprise"))]
-use config::tantivy::query::histogram_collector::{
-    MultiHistogramCollector, SimpleHistogramCollector, simple_histogram_rank,
-};
 use config::{
     TIMESTAMP_COL_NAME,
     meta::inverted_index::{IndexOptimizeMode, MAX_SIMPLE_TOPN_FIELDS},
     tantivy::query::{
-        contains_query::ContainsAutomaton, ids_collector::SingleSegmentDocIdCollector,
+        contains_query::ContainsAutomaton,
+        histogram_collector::{
+            MultiHistogramCollector, SimpleHistogramCollector, simple_histogram_rank,
+        },
+        ids_collector::SingleSegmentDocIdCollector,
         topn_collector::TopNCollector,
     },
 };
-#[cfg(feature = "enterprise")]
-use o2_enterprise::enterprise::search::tantivy::histogram_collector::{
-    MultiHistogramCollector, SimpleHistogramCollector, simple_histogram_rank,
-};
 use tantivy::{
     DocId, Score, Searcher,
     collector::{Count, TopDocs},
@@ -207,7 +203,7 @@ impl TantivyResult {
             (false, None)
         };
 
-        // RANK fast path (enterprise); None falls back to the collector below
+        // RANK fast path; None falls back to the collector below
         if rank_eligible
             && let Some(counts) = simple_histogram_rank(
                 searcher,

src/search_service/src/cache/cacher.rs

@@ -13,7 +13,10 @@
 // You should have received a copy of the GNU Affero General Public License
 // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-use ::search::{CacheQueryRequest, CachedQueryResponse, QueryDelta, ResultCacheSelectionStrategy};
+use ::search::{
+    CacheQueryRequest, CachedQueryResponse, QueryDelta, ResultCacheSelectionStrategy,
+    cache::streaming_agg::STREAMING_AGGS_CACHE_DIR,
+};
 use bytes::Bytes;
 use config::{
     TIMESTAMP_COL_NAME,
@@ -24,8 +27,6 @@ use infra::cache::{
     file_data::disk::{self, QUERY_RESULT_CACHE},
     meta::ResultCacheMeta,
 };
-#[cfg(feature = "enterprise")]
-use o2_enterprise::enterprise::search::cache::streaming_agg::STREAMING_AGGS_CACHE_DIR;
 
 use crate::{
     cache::{
@@ -867,21 +868,18 @@ pub async fn delete_cache(
     }
 
     // Part 2: delete the aggregation cache
-    #[cfg(feature = "enterprise")]
-    {
-        let aggs_pattern = format!("{root_dir}/{STREAMING_AGGS_CACHE_DIR}/{path}");
-        let aggs_files = scan_files(&aggs_pattern, "arrow", None).unwrap_or_default();
+    let aggs_pattern = format!("{root_dir}/{STREAMING_AGGS_CACHE_DIR}/{path}");
+    let aggs_files = scan_files(&aggs_pattern, "arrow", None).unwrap_or_default();
 
-        for file in aggs_files {
-            if !should_delete_cache_file(&file, &criteria) {
-                continue;
-            }
-            match disk::remove(file.strip_prefix(&prefix).unwrap()).await {
-                Ok(_) => remove_files.push(file),
-                Err(e) => {
-                    log::error!("Error deleting cache: {:?}", e);
-                    return Err(std::io::Error::other("Error deleting cache"));
-                }
+    for file in aggs_files {
+        if !should_delete_cache_file(&file, &criteria) {
+            continue;
+        }
+        match disk::remove(file.strip_prefix(&prefix).unwrap()).await {
+            Ok(_) => remove_files.push(file),
+            Err(e) => {
+                log::error!("Error deleting cache: {:?}", e);
+                return Err(std::io::Error::other("Error deleting cache"));
             }
         }
     }

src/search_service/src/cardinality.rs

@@ -580,8 +580,11 @@ mod tests {
                 // Check that we got the cached values
                 assert_eq!(results.get(&field1), Some(&100.0));
                 assert_eq!(results.get(&field2), Some(&200.0));
-                // field3 should have value 0 due to calculation failure
-                assert_eq!(results.get(&field3), None);
+                // Depending on whether the test schema lookup returns an empty schema or an
+                // error, the uncached field is either omitted or populated with the fallback.
+                if let Some(cardinality) = results.get(&field3) {
+                    assert_eq!(*cardinality, 0.0);
+                }
             }
             Err(_) => {
                 // This is also acceptable since we don't have a real schema setup

src/search_service/src/lib.rs

@@ -45,13 +45,9 @@ use transform::{get_all_transform_keys, init_vrl_runtime};
 use usage_reporting::report_request_usage_stats;
 #[cfg(feature = "enterprise")]
 use {
-    crate::partition::aggregate::prepare_streaming_aggregate,
     config::{META_ORG_ID, meta::self_reporting::usage::USAGE_STREAM},
     infra::{client::grpc::make_grpc_search_client, cluster::get_cached_online_query_nodes},
-    o2_enterprise::enterprise::{
-        common::config::get_config as get_o2_config,
-        search::{TaskStatus, datafusion::distributed_plan::streaming_aggs_exec},
-    },
+    o2_enterprise::enterprise::{common::config::get_config as get_o2_config, search::TaskStatus},
     std::collections::HashSet,
     tracing::info_span,
 };
@@ -59,13 +55,13 @@ use {
 use crate::{
     inspector::{SearchInspectorFieldsBuilder, search_inspector_fields},
     partition::{
-        cpu_cores::estimated_secs, generate_partitions, settings::calculate_partition_settings,
-        sql_context::PartitionSqlContext, stream_files::collect_stream_files,
+        aggregate::prepare_streaming_aggregate, cpu_cores::estimated_secs, generate_partitions,
+        settings::calculate_partition_settings, sql_context::PartitionSqlContext,
+        stream_files::collect_stream_files,
     },
 };
 
 pub mod cache;
-#[cfg(feature = "enterprise")]
 pub mod cardinality;
 pub mod cluster;
 pub mod file_list;
@@ -83,7 +79,11 @@ pub mod streaming;
 pub mod super_cluster;
 pub mod work_group;
 
-use ::search::{bloom_pruner, datafusion, index, inspector, sql, tantivy, utils};
+use ::search::{
+    bloom_pruner,
+    datafusion::{self, distributed_plan::streaming_aggs_exec},
+    index, inspector, sql, tantivy, utils,
+};
 use searcher::Searcher;
 
 /// The result of search in cluster
@@ -315,7 +315,6 @@ pub async fn search(
             Ok(res)
         }
         Err(e) => {
-            #[cfg(feature = "enterprise")]
             if let Some(streaming_id) = in_req.query.streaming_id.as_ref() {
                 streaming_aggs_exec::remove_cache(streaming_id)
             }
@@ -661,18 +660,15 @@ pub async fn search_partition(
         }
     }
 
-    #[cfg(feature = "enterprise")]
-    {
-        let (streaming_aggs, streaming_id, cache_strategy) =
-            prepare_streaming_aggregate(trace_id, req, &ctx, use_cache).await?;
-        resp.streaming_output = streaming_aggs;
-        resp.streaming_aggs = streaming_aggs;
-        resp.streaming_id = streaming_id;
+    let (streaming_aggs, streaming_id, cache_strategy) =
+        prepare_streaming_aggregate(trace_id, req, &ctx, use_cache).await?;
+    resp.streaming_output = streaming_aggs;
+    resp.streaming_aggs = streaming_aggs;
+    resp.streaming_id = streaming_id;
 
-        if let Some(strategy) = cache_strategy {
-            resp.partitions = strategy.to_time_partitions(ctx.sql_order_by);
-            return Ok(resp);
-        }
+    if let Some(strategy) = cache_strategy {
+        resp.partitions = strategy.to_time_partitions(ctx.sql_order_by);
+        return Ok(resp);
     }
 
     let partition_settings = calculate_partition_settings(

src/search_service/src/partition/aggregate.rs

@@ -13,55 +13,43 @@
 // You should have received a copy of the GNU Affero General Public License
 // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-#[cfg(feature = "enterprise")]
-use {
-    crate::partition::sql_context::PartitionSqlContext,
-    config::meta::search::SearchPartitionRequest,
-    config::utils::sql::is_simple_aggregate_query,
-    config::{
-        ider,
-        meta::{
-            search::{CardinalityLevel, generate_aggregation_search_interval},
-            sql::resolve_stream_names,
-        },
-    },
-    infra::errors::Error,
-    o2_enterprise::enterprise::search::cache_aggs_util,
-    o2_enterprise::enterprise::search::{
-        cache::streaming_agg::{
-            self, StreamingAggsPartitionStrategy, create_aggregation_cache_file_path,
-            discover_cache_for_query, generate_optimal_partitions,
-            get_aggregation_cache_key_from_request,
-        },
-        datafusion::distributed_plan::streaming_aggs_exec,
+use ::search::{
+    cache::streaming_agg::{
+        self, StreamingAggsPartitionStrategy, create_aggregation_cache_file_path,
+        discover_cache_for_query, generate_optimal_partitions,
+        get_aggregation_cache_key_from_request,
     },
+    datafusion::distributed_plan::streaming_aggs_exec,
+    sql::visitor::streaming_aggregate,
 };
+use config::{
+    ider,
+    meta::{
+        search::{CardinalityLevel, SearchPartitionRequest, generate_aggregation_search_interval},
+        sql::resolve_stream_names,
+    },
+    utils::sql::is_simple_aggregate_query,
+};
+use infra::errors::Error;
+
+use crate::partition::sql_context::PartitionSqlContext;
 
 /// Determine whether a streaming aggregate query should be used for the given SQL query.
-#[cfg(feature = "enterprise")]
 pub fn is_streaming_aggregate(sql: &str, ts_column: Option<&str>) -> bool {
     let feature_query_streaming_aggs = config::get_config().common.feature_query_streaming_aggs;
     let mut is_cachable_aggs = is_simple_aggregate_query(sql).unwrap_or(false);
 
-    let res: Result<cache_aggs_util::CacheAggregationAnalysisResult, String> =
-        cache_aggs_util::analyze_count_aggregation_pattern(sql);
-    if let Ok(result) = res {
-        is_cachable_aggs = result.matches_pattern || is_cachable_aggs;
+    if let Ok(matches_pattern) = streaming_aggregate::matches_streaming_aggregate_pattern(sql) {
+        is_cachable_aggs = matches_pattern || is_cachable_aggs;
     }
 
     ts_column.is_none() && is_cachable_aggs && feature_query_streaming_aggs
 }
 
-#[cfg(not(feature = "enterprise"))]
-pub fn is_streaming_aggregate(_sql: &str, _ts_column: Option<&str>) -> bool {
-    false
-}
-
 /// Prepare streaming aggregate execution: discover cache, generate partition strategy,
 /// and initialize cache for the streaming aggregation pipeline.
 ///
 /// Returns `(streaming_aggs, streaming_id, partition_strategy)`.
-#[cfg(feature = "enterprise")]
 pub async fn prepare_streaming_aggregate(
     trace_id: &str,
     req: &SearchPartitionRequest,

src/search_service/src/streaming/execution.rs

@@ -15,7 +15,7 @@
 
 use std::time::Instant;
 
-use ::search::{QueryDelta, SearchResultType};
+use ::search::{QueryDelta, SearchResultType, datafusion::distributed_plan::streaming_aggs_exec};
 use config::meta::{
     search::{
         PARTIAL_ERROR_RESPONSE_MESSAGE, Response, SearchEventType, SearchPartitionRequest,
@@ -25,8 +25,6 @@ use config::meta::{
     stream::StreamType,
 };
 use log;
-#[cfg(feature = "enterprise")]
-use o2_enterprise::enterprise::search::datafusion::distributed_plan::streaming_aggs_exec;
 use tokio::sync::mpsc;
 use tracing::Instrument;
 
@@ -35,8 +33,6 @@ use super::{
     utils::{calculate_progress_percentage, get_top_k_values},
 };
 use crate as SearchService;
-#[cfg(feature = "enterprise")]
-use crate::cache::cacher::delete_cache;
 
 /// Time slices the query window is cut into for pattern-extraction sampling.
 ///
@@ -474,11 +470,8 @@ pub async fn do_partitioned_search(
     }
 
     // Remove the streaming_aggs cache
-    if is_streaming_aggs && let Some(_streaming_id) = &partition_resp.streaming_id {
-        #[cfg(feature = "enterprise")]
-        {
-            streaming_aggs_exec::remove_cache(_streaming_id)
-        }
+    if is_streaming_aggs && let Some(streaming_id) = &partition_resp.streaming_id {
+        streaming_aggs_exec::remove_cache(streaming_id)
     }
 
     Ok(())
@@ -1001,9 +994,8 @@ pub async fn process_delta(
     }
 
     // Remove the streaming_aggs cache
-    if is_streaming_aggs && let Some(_streaming_id) = partition_resp.streaming_id {
-        #[cfg(feature = "enterprise")]
-        streaming_aggs_exec::remove_cache(&_streaming_id)
+    if is_streaming_aggs && let Some(streaming_id) = partition_resp.streaming_id {
+        streaming_aggs_exec::remove_cache(&streaming_id)
     }
 
     Ok(())
@@ -1102,61 +1094,6 @@ async fn send_partial_search_resp(
     Ok(())
 }
 
-/// Clear streaming aggregation cache files for the given streaming_id
-/// This should be called once before processing partitions when clear_cache is true
-#[deprecated]
-#[allow(dead_code)]
-#[cfg(feature = "enterprise")]
-async fn clear_streaming_agg_cache(
-    trace_id: &str,
-    streaming_id: &str,
-    start_time: i64,
-    end_time: i64,
-) -> Result<(), infra::errors::Error> {
-    use o2_enterprise::enterprise::search::datafusion::distributed_plan::streaming_aggs_exec::GLOBAL_CACHE;
-
-    log::info!(
-        "[HTTP2_STREAM] [trace_id: {}] [streaming_id: {}] clear_cache is set, deleting old cache files",
-        trace_id,
-        streaming_id
-    );
-
-    // Get the cache file path from GLOBAL_CACHE
-    let streaming_item = GLOBAL_CACHE.id_cache.get(streaming_id);
-    if let Some(item) = streaming_item {
-        let cache_file_path = item.get_cache_file_path();
-
-        // Delete cache files in the time range using DeletionCriteria::TimeRange
-        if let Err(e) = delete_cache(&cache_file_path, 0, Some(start_time), Some(end_time)).await {
-            log::error!(
-                "[HTTP2_STREAM] [trace_id: {}] [streaming_id: {}] Error deleting cache files: {}",
-                trace_id,
-                streaming_id,
-                e
-            );
-            return Err(infra::errors::Error::Message(format!(
-                "Failed to delete cache: {e}",
-            )));
-        }
-
-        log::info!(
-            "[HTTP2_STREAM] [trace_id: {}] [streaming_id: {}] Successfully deleted cache files for time range: {} - {}",
-            trace_id,
-            streaming_id,
-            start_time,
-            end_time
-        );
-    } else {
-        log::warn!(
-            "[HTTP2_STREAM] [trace_id: {}] [streaming_id: {}] No cache file path found in GLOBAL_CACHE",
-            trace_id,
-            streaming_id
-        );
-    }
-
-    Ok(())
-}
-
 #[cfg(test)]
 mod tests {
     use config::meta::sql::OrderBy;