call-site aware memory tracking (#13344)
* Initial memory tracking design doc draft, plus first round of review comments by me with // TODO annotations
* design/memory-tracker: address first-round review TODOs
Resolves all // TODO annotations from the initial draft:
- live-block table now optional via MEMORY_TRACKING_LIVE_TRACKING knob
- drop the mmap slab pool; std::malloc + in-tracker flag is sufficient
- add MEMORY_TRACKING_FORCE_SAMPLE_BYTES so large allocations are always
captured regardless of the count-rate sampler
- add live block / byte totals to MemoryTrackerSummary
- replace the manual coverage spot-check with a sentinel-function unit
test that introspects the aggregation table directly
- leave ALLOC_INSTRUMENTATION alone; new hooks sit next to (not
replacing) existing conditional ones
- drop SJLJ jargon, trim A4 alternative now that force-sample-large
collapses the byte-rate-vs-count-rate question
* flow: add sampled per-call-site memory tracker
Adds a sampled memory attribution layer (flow/MemoryTracker.{h,cpp})
hooked into the three primary allocation paths — global operator
new/delete, FastAllocator, and ArenaBlock::create — plus a periodic
TraceEvent dump driven from SystemMonitor. Knobs gate sample rate,
force-sample threshold, report cadence, top-N, and capture depth;
prod default is off. See design/memory-tracker.md.
Test fixes uncovered while bringing the unit tests up:
- memTrackerResetForTest now resets the per-thread sample counter and
force-sample threshold. Without this, a test that exercised the
off-switch path left gMemTrackerCounter at INT_MAX, which then
silently suppressed sampling for the remainder of the run.
- Slow-path reseed special-cases inverse==1 to keep the counter at 1.
The general formula 1 + r % (2*inverse) yields counter values 1 or
2 at inverse==1, sampling only ~67% of allocations rather than every
one, which broke a test that asserts exact alloc counts.
- Sentinel functions in MemoryTrackerTest.cpp now route the allocated
pointer through an asm-volatile escape() helper. Clang -O3 was
eliding the new/delete pair (P0593 heap fusion), so the test's
allocations never reached the operator-new override.
* design/memory-tracker: address second-round review
- threshold-based reporting (80 MB default, ~1% of 8 GB target RSS)
replaces fixed top-N
- prod report interval 60 s -> 10 min; sim stays 30 s
- single combined MemoryTrackerAddrCmd event with one addr2line
invocation per dump (positional mapping back to sites), keeping
frame 0 — old design's per-site format_backtrace dropped the
leaf alloc frame
- new R12 "Side-thread coverage" + "Side-thread safety" subsection
documenting the FP-elision crash mode found via joshua repros
(RandomUnitTests / IThreadPool seeds segfaulted in captureStackFP
when walking from FastAllocator<N>::~ThreadData into glibc's
FP-elided pthread shutdown machinery) and the stack-bounds
mitigation via pthread_getattr_np
- R7 wording: live bytes (not cumulative)
- CallSite struct in design overview aligned with implementation;
ForceSampledCount promoted from prose-only to struct + emitted
detail; exemplarFrames sized to MEMORY_TRACKER_MAX_FRAMES (=10)
matching the FRAMES knob's stated 1-10 range
- LIVE_TRACKING=false degraded-mode interaction documented
- stale "slab pool" refs removed (std::malloc was already in code)
and fdbserver.cpp self-contradiction resolved
- R3 / Rollout reconciled (table shows steady-state, step 1 lands
at 0)
- 4-6 frame count flagged as initial estimate, subject to refinement
- file:line citations stripped from path references (line numbers
drift; symbol names are stable)
* flow: threshold-based memory tracker reporting + side-thread safety
Implements the second-round design changes in flow/MemoryTracker.{cpp,h},
flow/Knobs.{cpp,h}, flow/SystemMonitor.cpp.
- MEMORY_TRACKING_TOP_N -> MEMORY_TRACKING_REPORT_BYTES_THRESHOLD
(int64_t, default 80,000,000). MEMORY_TRACKING_REPORT_INTERVAL prod
default 60.0 -> 600.0; sim still 30.0.
- memTrackerDump(int topN) -> memTrackerDump(int64_t bytesThreshold).
Filters by liveBytes (or cumulativeBytes when LIVE_TRACKING=false)
>= threshold; emits MemoryTrackerSite per qualifying site plus one
MemoryTrackerAddrCmd event with a single addr2line invocation
covering every qualifying site's frames in dump order. The Summary
event picks up SitesReported and ReportBytesThreshold details.
- AddrCmd is built directly here (not via platform::format_backtrace,
which deliberately drops index 0 for its single-site use case); the
leaf alloc frame is preserved.
- captureFramesFP gains a per-thread stack-bounds check via
pthread_getattr_np + pthread_attr_getstack, cached in TLS. Without
it, walking the FP chain from FastAllocator<N>::~ThreadData into
glibc's FP-elided pthread shutdown machinery follows an
uninitialized saved-FP slot and dereferences garbage. Fixes the
joshua-found segfaults on RandomUnitTests seeds 3288611985,
3731245491, and 2219741568 (all in the IThreadPool worker-exit
path). See design/memory-tracker.md "Side-thread safety".
* flow/MemoryTracker: stub the FP walker on non-Linux
pthread_getattr_np is glibc-specific and the macOS build broke on
it. Frame-pointer walking on macOS is also unreliable on its own
(system runtime has -fomit-frame-pointer in places we can't
control), so a "loose bounds" workaround would still risk crashes.
FDB is required to compile on macOS but is not run in production
there. Gate initStackBoundsForThread + the real captureFramesFP
on __linux__; provide a return-0 stub on non-Linux. The rest of
the tracker (sample counters, aggregation, dump) still compiles
and runs; per-call-site reports on macOS will just lack stack
attribution.
* flow: clang-format fixup for memory-tracker files
Whitespace-only. Catches up flow/Arena.cpp and flow/MemoryTrackerTest.cpp
with the project's clang-format style; the original implementation
commit (d587f82b) slipped these past the format pre-flight.
* edit for clarity, brevity, and uniform voice
* flow/Arena: fix double-tracking on the >256/huge ArenaBlock paths
ArenaBlock::create's >256 and huge paths go through
allocateAndMaybeKeepalive (`new uint8_t[]`), which fires the global
operator new[] hook in addition to the explicit memTrackerOnAlloc
that fires immediately after. Two sites tracked the same pointer;
on free only the explicit-Arena fingerprint was debited, so the
operator new[] fingerprint accumulated liveBytes monotonically and
LiveBytesTotal/LiveBlocksTotal skewed by +n/+1 per arena alloc/free
pair. Reported as B1 in the PR review.
Fix at the Arena layer (so non-arena allocateAndMaybeKeepalive
callers in serialize.h's PacketBuffer code remain attributed at the
operator-new layer): a MemTrackerSuppress RAII helper held across
the underlying new[]/delete[] in ArenaBlock::create and
ArenaBlock::destroyLeaf.
Adds accounting tests for FastAllocator<32>, Arena small, Arena
medium (the B1 path), and Arena huge. The load-bearing assertion is
"exactly one site has the sentinel's frames AND nonzero bytes" --
fails pre-fix for medium/huge with sites=2. Tests gate on __linux__
since captureFramesFP is a no-op on macOS.
* flow/memory-tracker: address PR review follow-ups
- Knobs.cpp: sim default for MEMORY_TRACKING_REPORT_BYTES_THRESHOLD
drops 80 MB -> 1 MB so sim dumps surface more sites for manual
sanity-checking. Prod unchanged.
- B2: memTrackerForEachSite holds MemTrackerSuppress across the
callback loop so callbacks that allocate (e.g. fprintf failure
dumps) don't re-enter tracking under SAMPLE_INVERSE=1.
- B8: drop the always-zero g_reentrantBailouts and its
SamplesDroppedReentry summary detail. Wiring it would require an
atomic in the inline hot path, which R1 forbids. Design doc
updated.
- B3/B6/B7/B9: explanatory comments only -- frame-strip-count
inlining assumption (B3), unstable sort acceptable under R6 (B6),
shared xorshift seed acceptable in practice (B7),
MEMORY_TRACKING_LIVE_TRACKING is startup-only (B9).
* flow/MemoryTracker: one addr2line per site, drop chunking
Move the addr2line command back onto MemoryTrackerSite as a per-site
AddrCmd detail and remove the MemoryTrackerAddrCmd event entirely.
Each AddrCmd carries exactly that site's stack -- short, well under
the trace-detail truncation cap, ready to paste. Replaces the
consolidated-then-chunked-into-byte-buckets approach, which split
stacks across chunk boundaries and made raw events hard to read.
* add standard Apache 2.0 license headers to new memory-tracker files
The three new files (flow/MemoryTracker.{cpp,h} and
flow/MemoryTrackerTest.cpp) shipped without the project-standard
copyright/license block. Adds the standard 19-line header to each;
file-purpose comments stay below it, switched to // line comments
to keep the license block visually distinct.
AGENTS.md gains a short "Source File Headers" section so the next
contributor doesn't repeat the omission.
* address review comments; reduce cost of enable check; maintain net estimates so users dont have to do it manually
* formatting
* unit test bug fix
* gglass review comments on memory-tracker.md design doc
* design doc updates, and fill in a plan for remaining tests/benchmarks
* delete useless simulation section
* flow/bench: add memory-tracker microbenchmarks; record measured overhead
Add flow/bench/BenchMemoryTracker.cpp (Google Benchmark) measuring the tracker's
per-op cost via three benchmarks -- raw malloc/free baseline, end-to-end
operator new/delete, and isolated memTrackerOnAlloc/OnFree -- each at sample
inverse 0 / 100 / 1. Auto-picked up by the flow_bench CONFIGURE_DEPENDS glob;
run with:
bin/flow_bench --benchmark_filter=memtracker
Replace the placeholder "< 5% delta" targets in the design doc's Microbenchmarks
section with the measured numbers and a projected per-second overhead at an
assumed 100K alloc/sec. Headline: ~1.9 ns/pair disabled, ~5.6 ns/pair at the
production 1% rate (~0.056% of a core at 100K/s, ~1/17th of R0's 1% ceiling);
the every-allocation rows are labeled a buggified worst case, not a default.
Testing: built flow_bench on the dev pod (clean, -Werror) and ran the memtracker
filter six times; results stable to within a few percent across runs.
* design/memory-tracker: describe the coverage test as implemented, not proposed
The "Coverage spot-check via sentinel functions" section described the
already-implemented `coverage` test in proposal tense ("Add an introspection
API:", "The unit test:"), which read as future work. Reword to present tense
referencing the actual `coverage`/`*Accounting` tests and the existing
`memTrackerForEachSite` API. No remaining references to unwritten test cases.
* remove extraneous detail from requirements section
* substantially revise microbenchmark results based on seeing 2M allocations/frees on a CPU-maxed storage server
* add script to drive A/B experiment for sampled memory allocation tracking
* flow/MemoryTracker: move global operator new/delete into a server-only TU
The global operator new/delete replacements that route allocations through
the memory tracker lived in flow/MemoryTracker.cpp, i.e. in the flow static
library. flow is linked into libfdb_c and every client binding, so the
interposition shipped into client artifacts and could interpose the whole
host process's allocator even with sampling off.
Move them into a new fdbserver/GlobalNewDelete.cpp, compiled directly into
the fdbserver executable (which clients never link), mirroring where the
legacy ALLOC_INSTRUMENTATION overrides already lived. This also makes the
replaceable-symbol interposition reliable (guaranteed in the final link)
rather than dependent on static-archive pull-in. The legacy
ALLOC_INSTRUMENTATION overrides move into the same file, selected by
#if defined(ALLOC_INSTRUMENTATION) / #else, so exactly one set of global
operators is ever defined. No CMake change is needed (fdbserver/*.cpp is
globbed).
Also reconcile the design doc with the code: override placement, the Files
section (drop the unnecessary flow/CMakeLists.txt edit), the sim knob-table
values (REPORT_BYTES_THRESHOLD, SAMPLE_INVERSE prod default), the
FORCE_SAMPLE_BYTES "-1 disables" sentinel wording, and drop the stale
"buggify inverse to 1" note -- the every-allocation and sampled/weighted
paths are already pinned deterministically by MemoryTrackerTest.cpp.
Testing: build green (run-ccmk5); fdbserver -r unittests -f /flow/MemoryTracker/
runs all 10 memory-tracker unit tests, 0 failed.
* contrib/mako_ab_memtracker: disable RocksDB direct I/O for tmpfs runs
RocksDB opens its DB with O_DIRECT by default; /mnt/ram (tmpfs), where the
harness puts its data dir, does not support direct I/O, so the storage
engine fails to Open and the cluster never configures (fdbcli "configure
new" hangs, mako never starts). Pass the existing
ROCKSDB_USE_DIRECT_READS / ROCKSDB_USE_DIRECT_IO_FLUSH_COMPACTION knobs (=0)
for the rocksdb arm only -- no new knobs added. redwood is unaffected.
* fdbserver/bench: move memory-tracker microbench to fdbserver_bench
The global operator new/delete override lives in fdbserver/GlobalNewDelete.cpp
(server-only, for client isolation), so flow_bench -- which links only flow --
could not exercise it: bench_memtracker_operator_new hit libc++'s operator new
and its Arg(100)/Arg(1) rows were identical to Arg(0).
Move BenchMemoryTracker.cpp to a new fdbserver/bench/ whose CMake compiles
GlobalNewDelete.cpp into the fdbserver_bench executable (via ADDL_SRCS), so the
real override is a strong definition in the bench link and the operator-new
benchmark measures the actual hooked path. It links only flow, not the fdbserver
dependency graph. Adds a BenchMain.cpp (BENCHMARK_MAIN equivalent) and wires the
subdirectory into fdbserver/CMakeLists.txt.
Testing: fdbserver_bench builds; bench_memtracker_operator_new now shows distinct
off / 1% / every-alloc costs (11.4 / 14.4 / 80.9 ns), confirming the override
fires.
* design/memory-tracker: reconcile with code and slim down
Reconcile the doc with the implementation and trim implementation detail that
duplicated the code and had begun to drift:
- Fix drift found in review: degraded-mode live/peak fields stay 0 (not
"tracking the cumulatives"); the dump thresholds on the estimated fields (fix
the memTrackerDump header comment too); the live-block table is allocated but
empty when live-tracking is off (not "never allocated"); list the test file
and its forceLinkMemoryTrackerTests() wiring.
- Slim ~220 lines: replace the CallSite struct, the full captureStackFP source,
both TraceEvent .detail() schemas, and the file-by-file inventory with prose
that defers exact structs/keys/constants to the code; soften the knob table
to intent (authoritative defaults live in Knobs.cpp).
- Refresh the microbench numbers from fdbserver_bench and annotate the host
(AMD EPYC 9R14, clang -O3); replace the A/B placeholder with the measured
-16.5% (redwood) / -9.9% (rocksdb); note it is a point-in-time snapshot.
- Frame R0 as the target the v1 single-lock design does not yet meet (ships off
pending lock sharding); add a one-line note on table teardown/fork behavior.
* fix clang-tidy error
* mako wrapper scripts: take care to clobber the ramdisk before runs to avoid low-space throttling
* design/memory-tracker: note the two mako A/B harnesses and the off-state result
Point at contrib/mako_ab_memtracker.py (sampling off vs on) and
contrib/mako_ab_binaries.py (vanilla main vs this PR built with tracking off),
and record the latter's measured off-state overhead of -0.53% (redwood) /
-0.10% (rocksdb) on a shared base commit — well within R0's 1% ceiling.
* Add the default code review guidance to AGENTS.md
* address big brother review comments; add some braces and trim some generated comments (hard to believe, but true)
* clang-tidy again
* Final read-through of this PR.
-- Add or enhance a few comments on important items (performance & reliability)
-- Delete some misc agent-written comments, typically exhibiting recency bias e.g. naming bugs identified in code review passes or describing mundane earlier bugs
-- Add braces (InsertBraces style)
* memory-tracker: address round-4 review
Correctness/robustness:
- operator new now runs the installed std::new_handler retry loop, so an
allocation failure (including the tracker's own map growth) reaches FDB's
platform::outOfMemory / FDB_EXIT_NO_MEM path instead of throwing past it.
- Reentrancy guard restored via MemTrackerSuppress RAII on every path (hot path,
dump, reset) so an exception can't permanently disable tracking on a thread.
- Sampling reseed draws from [1, 2N-1] (mean exactly N); the old [1, 2N] biased
the Est* estimate low by ~0.5/N.
Design:
- Drop runtime enable/disable (now an explicit Non-requirement): the sample knob
is read at startup only; park the counter when off. Removes the DISABLED_RESEED
re-park and the on/off reconciliation logic.
- Simulation samples 1-in-10 (was 1-in-2).
Portability (Windows is not build-tested here; lean on existing abstractions):
- Aligned operator new uses platform::aligned_alloc/aligned_free (overflow-guarded)
instead of posix_memalign; guard <pthread.h> under __linux__; add a
force_noinline macro (GNU-only, empty elsewhere); portable volatile-sink
escape() in the test.
Tooling/tests/docs:
- mako A/B scripts: validate the scratch mount (realpath + tmpfs + denylist) before
rm -rf, locate mako_storage_bench.sh via __file__, and black-format.
- Add operatorNewHonorsNewHandler and samplingRate tests; drop enableAfterOff.
- Reconcile the design doc with the code; prune low-value comments.
Testing:
- /flow/MemoryTracker/* unit tests: 11 pass, 0 fail (bin/fdbserver -r unittests).
- Joshua 100k (correctness-8.0.0): 99,995 pass / 1 fail. The single failure is
NativeCdcAssignmentPublication -- a NativeCdc test from main, not this PR
(CommitProxy failed_to_progress -> QuietDatabase DataDistributionActive ->
NativeCdcEndToEnd workload start timed_out). It reproduces bit-identically
(seed 2939264355, unseed 8776) with the tracker built off (sim
sample_inverse=0, confirmed via the MemoryTrackerSummary SampleInverse trace
field), so it is a pre-existing rare CDC/QuietDatabase flake unrelated to this
change.
* memory-tracker: fix GCC IPA-clone breaking frame-attribution tests
Under GCC -O3, IPA constant-propagation cloning (-fipa-cp-clone) specializes the
MemoryTrackerTest sentinels (each called with a constant N) into `.constprop`
clones emitted at a different address than the function symbol. The executed
code -- and thus the captured return addresses -- live in the clone, so the
tests' frameInside(frame, &sentinel) window missed them: fastAlloc32Accounting
aborted (sitesWithSentinelFrames == 0) in GCC CI while clang passed.
Add `noclone` to force_noinline on GCC so each sentinel stays a single body at
the address &fn yields. clang doesn't support noclone (and doesn't clone this
way), so it keeps noinline only; other compilers stay empty.
Testing: /flow/MemoryTracker/* passes 11/11 under both a gcc-toolset-13 build
and a clang build.
* memory-tracker: cheaper disabled alloc hot path (per-thread off flag)
When sampling is off, memTrackerOnAlloc previously still did three TLS accesses
every allocation -- a gInMemTracker load, a gMemTrackerCounter load-decrement-
STORE, and a gForceSampleBytes load. Add a per-thread gMemTrackerOff flag,
checked first, that a thread sets once its slow path observes sampling is off;
the disabled alloc path then short-circuits on a single TLS load + branch (no
counter store, no gForceSampleBytes load).
The flag is per-thread, not global, on purpose: the counter still bootstraps
sampling per thread (first alloc reaches the slow path and reads the knob), so a
single global gate set by an early main-thread allocation before FLOW_KNOBS is
ready would wrongly disable worker threads that bootstrap later. Free keeps the
global g_memTrackerEnabled gate (a free-only thread must see global state to
debit). memTrackerResetForTest clears the flag. Removes the now-dead INT_MAX
counter parking.
The saving is real but small (~2 loads + 1 store per alloc, well under the
mako A/B's run-to-run noise), so the redwood off-state A/B shows no resolvable
change; the win is on principle / at high allocation rates.
Testing: /flow/MemoryTracker/* passes 11/11 (bin/fdbserver -r unittests).
* memory-tracker: add FDB_MEMORY_TRACKER compile-time gate + per-path microbench
Add a compile-time switch, FDB_MEMORY_TRACKER (CMake option, default ON), that
removes the feature entirely when set to 0: the header hooks become no-op inlines,
flow/MemoryTracker.cpp and the MemoryTrackerTest TEST_CASEs are #if'd out (the
forceLink stub stays), and fdbserver/GlobalNewDelete.cpp defines no global
operator new/delete override (libc++'s allocator is used, which still honors the
installed new_handler). This gives operators an escape hatch to zero
always-compiled footprint, and lets the microbench measure present-but-disabled
vs absent cost per allocation path.
Extend fdbserver/bench/BenchMemoryTracker.cpp with plain per-path alloc/free loops
(operator new[] at several sizes, FastAllocator<64/96/256>, Arena medium/huge) so
the same binary built =1 (tracker present, sampling off by default) vs =0 (absent)
isolates each path's unweighted off-state overhead.
Measured (EPYC 9R14 @ 3.7 GHz, ns/op, =1 minus =0):
operator new[] small ~+1.1 ns (~11%); huge ~+2.5 ns
FastAllocator<N> ~0 (within the ~0.6 ns cross-build noise floor)
Arena block ~+3.3 ns (medium) / +4.8 ns (huge)
The per-op-costly paths (operator new, Arena) are the low-volume ones; the
high-volume path (FastAllocator, ~82% of redwood allocs) is ~free, so the weighted
off-state overhead is ~0.1-0.2% of a core -- under the R0 target, though R0 is now
framed as an unproven target with this gate as the escape hatch.
Testing: /flow/MemoryTracker/* passes 11/11 (default build); FDB_MEMORY_TRACKER=OFF
builds clean under -Werror.
* clang-tidy fix
* comment about why fdbserver/bench exists
* address Codex round 5 review comments (MSVC build, startup sequencing, memory allocation failures on tracker-internal bookkeeping, cross-compile, contrib script cleanup)
* add another disclaimer about MSVC/Windows being best effort
* memory-tracker: fail-open ordering + deflake huge-arena unit test
memTrackerSampleAlloc now performs both table insertions (aggregation-map and
live-map nodes) before mutating any per-site or global counter, so if the
tracker's own map growth throws std::bad_alloc the exception unwinds with all
totals in lockstep and the fail-open catch simply drops the sample. This
addresses an adversarial-review finding. Note such a failure does not occur in
practice: FDB "OOM" is an RSS threshold enforced by fdbmonitor (typically
~12-16 GB against an ~8 GB target), not a malloc/operator-new failure. This
feature is for finding leaks and untuned allocations that drive RSS growth,
well short of any allocation failure -- documented in flow/MemoryTracker.cpp and
design/memory-tracker.md so reviewers don't over-index on the bad_alloc path.
arenaHugeAccounting now identifies the huge blocks by their ~100 KB size
signature instead of the frame-pointer sentinel. The huge-Arena path is the
deepest tracked call chain; under a real (non-simulation) network the
best-effort frame-pointer walker is per-allocation nondeterministic there and
cannot reliably attribute the blocks to the test's frame, so the sentinel-frame
assertion flaked (~1-4% of runs). The recorded block size is correct regardless
of which frames were captured, no incidental or foreign-thread allocation comes
near 100 KB, and double-tracking still shows up as 2N. The shallower
fastAlloc32/arenaSmall/arenaMedium/operatorNew accounting tests keep exact
sentinel-frame attribution (reliable on their shorter paths) as the regression
guard.
Testing:
- /flow/MemoryTracker/* via `fdbserver -r unittests`: 500 runs across two seed
spaces (including the sequence that previously flaked), 14/14 test cases pass
every run, 0 failures.
- Builds clean with FDB_MEMORY_TRACKER on and off under -Werror; clang-format
and clang-tidy clean on the changed files.
- Joshua 100k on the parent commit: ensemble
20260728-160205-gglass-0b30fa74f2605807, ended=100000 pass=100000 fail=0.
These changes are a pure counter-update reorder (no simulation-reachable
behavior change -- bad_alloc is not injected in sim), a unit-test-only change,
and documentation, so that result remains representative.
* memory-tracker: note why the fdbserver-local bench binary exists
Add a short comment to BenchMain.cpp explaining that most microbenchmarks
belong in flow/bench and this fdbserver-local benchmark binary exists only for
benchmarks that must link fdbserver-only code, pointing at BenchMemoryTracker.cpp
for the detailed rationale rather than duplicating it.
This commit is contained in:
gxglass
GitHub
parent
5e7e0c348b
commit
aa705c8c02
15
AGENTS.md
15
AGENTS.md
|
|
@ -136,6 +136,21 @@ Before changing a serialized type that persists on disk, inspect its `serializer
|
|||
|
||||
Edit `.actor.cpp` and `.actor.h` sources, not actorcompiler-generated output under the build directory.
|
||||
|
||||
## Source File Headers
|
||||
|
||||
Every new `.cpp` / `.h` / `.actor.cpp` / `.actor.h` file starts with the standard Apache 2.0 license block, with the filename on line 2 and the current year on the copyright line. Copy from any existing file in the tree (e.g. `flow/Knobs.cpp`). Add file-purpose comments *after* the license block, not in place of it.
|
||||
|
||||
## Code Review
|
||||
|
||||
Unless you have specific instructions to the contrary, when asked to review code (named files or a diff), address all of these explicitly:
|
||||
|
||||
- What is it trying to accomplish?
|
||||
- Is it correct?
|
||||
- Are there bugs?
|
||||
- Are there omissions?
|
||||
- Are there things that could be done better?
|
||||
- Should it be LGTM'd? (clear yes / no / not-yet)
|
||||
|
||||
## Branching
|
||||
|
||||
PRs target `main`. Release branches receive cherry-picks rather than direct PRs — don't open backport PRs without confirming first.
|
||||
|
|
|
|||
|
|
@ -83,6 +83,12 @@ if(WITH_ACAC)
|
|||
add_compile_definitions(WITH_ACAC)
|
||||
endif()
|
||||
|
||||
option(FDB_MEMORY_TRACKER "Compile in the sampled per-call-site memory tracker (flow/MemoryTracker)" ON)
|
||||
if(NOT FDB_MEMORY_TRACKER)
|
||||
message(STATUS "Building FoundationDB with the memory tracker compiled out")
|
||||
add_compile_definitions(FDB_MEMORY_TRACKER=0)
|
||||
endif()
|
||||
|
||||
###############################################################################
|
||||
# Packages used for bindings
|
||||
###############################################################################
|
||||
|
|
|
|||
|
|
@ -0,0 +1,365 @@
|
|||
#!/usr/bin/env python3
|
||||
#
|
||||
# mako_ab_binaries.py
|
||||
#
|
||||
# This source file is part of the FoundationDB open source project
|
||||
#
|
||||
# Copyright 2013-2026 Apple Inc. and the FoundationDB project authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
"""
|
||||
A/B benchmark across two *different fdbserver binaries*:
|
||||
|
||||
A = vanilla main (a build with no memory-tracker code at all)
|
||||
B = this PR's build, with memory tracking turned OFF (sample_inverse=0)
|
||||
|
||||
The point is to isolate the cost of the always-compiled memory-tracker hooks
|
||||
when they are disabled — i.e. to check the design's claim that the off-state
|
||||
overhead is <=1%. Unlike mako_ab_memtracker.py (one binary, two knob settings),
|
||||
each arm here runs a different --build.
|
||||
|
||||
Only arm B is passed the memory-tracking knob; arm A (main) does not have it and
|
||||
would reject an unknown knob. Both arms get the RocksDB direct-I/O-off knobs for
|
||||
the rocksdb engine (main has those long-standing knobs too) since the data dir
|
||||
is on tmpfs.
|
||||
|
||||
Reuses contrib/mako_storage_bench.sh and the chart.js report from
|
||||
mako_ab_memtracker.py. Run on the dev pod, e.g.:
|
||||
|
||||
python3 contrib/mako_ab_binaries.py \
|
||||
--build-a /root/build_output4 --build-b /root/build_output5 \
|
||||
--engines redwood rocksdb --warmup 60 --seconds 240 \
|
||||
--outdir /mnt/ram/binab
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
from mako_ab_memtracker import parse_run, line_ds, PCTS, clobber_ramdisk
|
||||
|
||||
# Found next to this script rather than hard-coded to one workspace.
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
BENCH_DEFAULT = os.path.join(_HERE, "mako_storage_bench.sh")
|
||||
|
||||
# A = baseline blue, B = warm orange.
|
||||
COLOR = {"a": "#4477CC", "b": "#EE7733"}
|
||||
|
||||
|
||||
def rocksdb_tmpfs_knobs(engine):
|
||||
"""RocksDB opens its DB with O_DIRECT, which tmpfs does not support; both
|
||||
binaries need direct I/O off to run rocksdb on /mnt/ram. redwood needs
|
||||
nothing."""
|
||||
if engine == "rocksdb":
|
||||
return [
|
||||
"--knob_rocksdb_use_direct_reads=0",
|
||||
"--knob_rocksdb_use_direct_io_flush_compaction=0",
|
||||
]
|
||||
return []
|
||||
|
||||
|
||||
def run_arm(bench, build, engine, armkey, extra_knobs, warmup, seconds, rows, outbase):
|
||||
"""Run one (engine, arm) via mako_storage_bench.sh with the arm's build."""
|
||||
workdir = os.path.join(outbase, armkey)
|
||||
env = dict(os.environ)
|
||||
env["WORKDIR"] = workdir
|
||||
env["WARMUP_SECONDS"] = str(warmup)
|
||||
env["SECONDS_RUN"] = str(seconds)
|
||||
env["ROWS"] = str(rows)
|
||||
env["KNOBS"] = " ".join(extra_knobs + rocksdb_tmpfs_knobs(engine))
|
||||
print(f"\n=== {engine} / {armkey} build={build} ===", flush=True)
|
||||
print(f" WORKDIR={workdir} KNOBS={env['KNOBS']}", flush=True)
|
||||
subprocess.run(["bash", bench, build, engine], env=env, check=False)
|
||||
return os.path.join(workdir, engine)
|
||||
|
||||
|
||||
def source_version(build):
|
||||
"""The git source version baked into the binary (fdbserver --version). Used
|
||||
to prove A and B are genuinely different builds, and to record provenance."""
|
||||
fdbserver = os.path.join(build, "bin", "fdbserver")
|
||||
try:
|
||||
out = subprocess.run(
|
||||
[fdbserver, "--version"], capture_output=True, text=True, timeout=60
|
||||
).stdout
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None
|
||||
m = re.search(r"source version (\w+)", out)
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _harvest(rundir, dst):
|
||||
"""Copy the small result files off tmpfs to persistent storage before the
|
||||
bulky SS/cluster data is wiped."""
|
||||
os.makedirs(dst, exist_ok=True)
|
||||
for name in ("mako.json", "mako-run.txt"):
|
||||
src = os.path.join(rundir, name)
|
||||
if os.path.exists(src):
|
||||
shutil.copy2(src, os.path.join(dst, name))
|
||||
|
||||
|
||||
def collect(bench, build_a, build_b, engines, warmup, seconds, rows, ramdir, outdir):
|
||||
b_knobs = ["--knob_memory_tracking_sample_inverse=0"]
|
||||
data = {}
|
||||
for engine in engines:
|
||||
data[engine] = {}
|
||||
for armkey, build, knobs in (("a", build_a, []), ("b", build_b, b_knobs)):
|
||||
rundir = run_arm(
|
||||
bench, build, engine, armkey, knobs, warmup, seconds, rows, ramdir
|
||||
)
|
||||
dst = os.path.join(outdir, armkey, engine)
|
||||
_harvest(rundir, dst) # save results to /root for persistence + debugging
|
||||
metrics = parse_run(dst)
|
||||
print(f" -> overallTPS={metrics['overallTPS']}", flush=True)
|
||||
data[engine][armkey] = metrics
|
||||
return data
|
||||
|
||||
|
||||
def generate_html(data, outpath, meta, warmup, seconds, rows):
|
||||
engines = list(data.keys())
|
||||
la, lb = meta["label_a"], meta["label_b"]
|
||||
|
||||
rows_html = []
|
||||
for eng in engines:
|
||||
a = data[eng].get("a", {})
|
||||
b = data[eng].get("b", {})
|
||||
ta, tb = a.get("overallTPS"), b.get("overallTPS")
|
||||
# B relative to A: negative = B slower (overhead).
|
||||
d = (100.0 * (tb - ta) / ta) if (ta and tb) else None
|
||||
p99a = a.get("latency", {}).get("p99")
|
||||
p99b = b.get("latency", {}).get("p99")
|
||||
dp = (100.0 * (p99b - p99a) / p99a) if (p99a and p99b) else None
|
||||
|
||||
def fmt(v, s=""):
|
||||
return f"{v:,.0f}{s}" if isinstance(v, (int, float)) else "—"
|
||||
|
||||
def dfmt(v):
|
||||
if v is None:
|
||||
return "—"
|
||||
return f"{'+' if v >= 0 else ''}{v:.2f}%"
|
||||
|
||||
rows_html.append(
|
||||
f"<tr><td>{eng}</td><td>{fmt(ta)}</td><td>{fmt(tb)}</td>"
|
||||
f"<td class='delta'>{dfmt(d)}</td>"
|
||||
f"<td>{fmt(p99a,' µs')}</td><td>{fmt(p99b,' µs')}</td>"
|
||||
f"<td class='delta'>{dfmt(dp)}</td></tr>"
|
||||
)
|
||||
|
||||
tps_a = [data[e].get("a", {}).get("overallTPS") or 0 for e in engines]
|
||||
tps_b = [data[e].get("b", {}).get("overallTPS") or 0 for e in engines]
|
||||
tps_datasets = json.dumps(
|
||||
[
|
||||
{"label": la, "data": tps_a, "backgroundColor": COLOR["a"]},
|
||||
{"label": lb, "data": tps_b, "backgroundColor": COLOR["b"]},
|
||||
]
|
||||
)
|
||||
|
||||
blocks = []
|
||||
for i, eng in enumerate(engines):
|
||||
a = data[eng].get("a", {})
|
||||
b = data[eng].get("b", {})
|
||||
ps_datasets = json.dumps(
|
||||
[
|
||||
line_ds(f"{eng} {la}", a.get("persec", []), COLOR["a"]),
|
||||
line_ds(f"{eng} {lb}", b.get("persec", []), COLOR["b"], dashed=True),
|
||||
]
|
||||
)
|
||||
lat_labels = json.dumps([s for _, s in PCTS])
|
||||
lat_a = [a.get("latency", {}).get(s) for _, s in PCTS]
|
||||
lat_b = [b.get("latency", {}).get(s) for _, s in PCTS]
|
||||
lat_datasets = json.dumps(
|
||||
[
|
||||
{"label": la, "data": lat_a, "backgroundColor": COLOR["a"]},
|
||||
{"label": lb, "data": lat_b, "backgroundColor": COLOR["b"]},
|
||||
]
|
||||
)
|
||||
n = max(len(a.get("persec", [])), len(b.get("persec", [])), 1)
|
||||
blocks.append(
|
||||
f"""
|
||||
<h2>{eng}</h2>
|
||||
<div class="chart-box"><canvas id="ps{i}"></canvas></div>
|
||||
<div class="chart-box"><canvas id="lat{i}"></canvas></div>
|
||||
<script>
|
||||
new Chart(document.getElementById('ps{i}'), {{
|
||||
type: 'line',
|
||||
data: {{ labels: [...Array({n}).keys()], datasets: {ps_datasets} }},
|
||||
options: {{ animation:false, responsive:true,
|
||||
scales: {{ x: {{ title:{{display:true,text:'sample (~1 s each, incl. warmup)'}} }},
|
||||
y: {{ beginAtZero:true, title:{{display:true,text:'TPS'}} }} }},
|
||||
plugins: {{ title:{{display:true,text:'{eng}: per-second throughput'}} }} }}
|
||||
}});
|
||||
new Chart(document.getElementById('lat{i}'), {{
|
||||
type: 'bar',
|
||||
data: {{ labels: {lat_labels}, datasets: {lat_datasets} }},
|
||||
options: {{ animation:false, responsive:true,
|
||||
scales: {{ y: {{ beginAtZero:true, title:{{display:true,text:'transaction latency (µs)'}} }} }},
|
||||
plugins: {{ title:{{display:true,text:'{eng}: transaction latency percentiles'}} }} }}
|
||||
}});
|
||||
</script>"""
|
||||
)
|
||||
|
||||
same = meta["ver_a"] and meta["ver_a"] == meta["ver_b"]
|
||||
guard = ""
|
||||
if same:
|
||||
guard = (
|
||||
"<p class='warn'><b>WARNING:</b> both builds report the same source "
|
||||
"version — A and B may be the same binary; results are not meaningful.</p>"
|
||||
)
|
||||
|
||||
html = f"""<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>FDB Memory Tracker A/B: vanilla main vs PR (tracking off)</title>
|
||||
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.4/dist/chart.umd.min.js"></script>
|
||||
<style>
|
||||
body {{ font-family: sans-serif; max-width: 960px; margin: 40px auto; padding: 0 20px; background:#fafafa; }}
|
||||
h1 {{ font-size:1.5em; margin-bottom:.1em; }} h2 {{ margin-top:2em; }}
|
||||
p.sub {{ color:#555; }} p.warn {{ color:#b00; }}
|
||||
table {{ border-collapse:collapse; width:100%; margin:1em 0; background:#fff; }}
|
||||
th,td {{ border:1px solid #ddd; padding:6px 10px; text-align:right; }}
|
||||
th:first-child, td:first-child {{ text-align:left; }}
|
||||
td.delta {{ font-weight:bold; }}
|
||||
.chart-box {{ background:#fff; border:1px solid #ddd; border-radius:6px; padding:20px; margin-bottom:24px; }}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>FDB Memory Tracker A/B — vanilla main vs PR (tracking OFF)</h1>
|
||||
<p class="sub">Two different fdbserver binaries on the same host / workload, isolating the
|
||||
cost of the always-compiled memory-tracker code when it is <b>disabled</b>.
|
||||
<b>A</b> = {la} (build <code>{meta['build_a']}</code>, source <code>{meta['ver_a']}</code>);
|
||||
<b>B</b> = {lb} (build <code>{meta['build_b']}</code>, source <code>{meta['ver_b']}</code>,
|
||||
<code>memory_tracking_sample_inverse=0</code>).
|
||||
Single-host 1/1/1 loopback cluster on /mnt/ram (storage isolated, CPU-bound) via
|
||||
<code>contrib/mako_storage_bench.sh</code>. rows={rows:,}, warmup {warmup}s, run {seconds}s.
|
||||
Δ is B relative to A: a small negative Δ is the off-state overhead; the design
|
||||
target is ≤1%.</p>
|
||||
{guard}
|
||||
<p class="sub"><b>Caveat:</b> {meta['drift_note']}</p>
|
||||
|
||||
<h2>Summary</h2>
|
||||
<table>
|
||||
<tr><th>engine</th><th>TPS A ({la})</th><th>TPS B ({lb})</th><th>Δ TPS (B vs A)</th>
|
||||
<th>p99 lat A</th><th>p99 lat B</th><th>Δ p99</th></tr>
|
||||
{''.join(rows_html)}
|
||||
</table>
|
||||
|
||||
<div class="chart-box"><canvas id="tps"></canvas></div>
|
||||
<script>
|
||||
new Chart(document.getElementById('tps'), {{
|
||||
type: 'bar',
|
||||
data: {{ labels: {json.dumps(engines)}, datasets: {tps_datasets} }},
|
||||
options: {{ animation:false, responsive:true,
|
||||
scales: {{ y: {{ beginAtZero:true, title:{{display:true,text:'Overall TPS'}} }} }},
|
||||
plugins: {{ title:{{display:true,text:'Overall throughput (A vs B)'}} }} }}
|
||||
}});
|
||||
</script>
|
||||
{''.join(blocks)}
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
with open(outpath, "w") as f:
|
||||
f.write(html)
|
||||
return outpath
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(
|
||||
description="Vanilla-main vs PR-tracking-off A/B via mako_storage_bench.sh"
|
||||
)
|
||||
ap.add_argument(
|
||||
"--build-a", default="/root/build_output4", help="vanilla main build (arm A)"
|
||||
)
|
||||
ap.add_argument(
|
||||
"--build-b", default="/root/build_output5", help="this-PR build (arm B)"
|
||||
)
|
||||
ap.add_argument("--label-a", default="main (no tracker)")
|
||||
ap.add_argument("--label-b", default="PR, tracking off")
|
||||
ap.add_argument("--bench", default=BENCH_DEFAULT)
|
||||
ap.add_argument("--engines", nargs="+", default=["redwood", "rocksdb"])
|
||||
ap.add_argument("--warmup", type=int, default=60)
|
||||
ap.add_argument("--seconds", type=int, default=240)
|
||||
ap.add_argument("--rows", type=int, default=100000)
|
||||
ap.add_argument(
|
||||
"--ramdir",
|
||||
default="/mnt/ram/binab",
|
||||
help="tmpfs scratch for SS/cluster data (only SS data lives on /mnt/ram)",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--ram-mount", default="/mnt/ram", help="tmpfs mount clobbered clean at startup"
|
||||
)
|
||||
ap.add_argument(
|
||||
"--outdir",
|
||||
default="/root/binab_results",
|
||||
help="persistent results dir on /root (~1 TB); harvested off tmpfs per arm",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--report",
|
||||
default="/root/src/mako_binab.html",
|
||||
help="HTML output path (syncs to ~/src on the Mac)",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--drift-note",
|
||||
default="A and B may build from slightly different main "
|
||||
"revisions; the delta bundles the PR's off-state cost with any main drift.",
|
||||
)
|
||||
ap.add_argument("--report-only", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
meta = {
|
||||
"label_a": args.label_a,
|
||||
"label_b": args.label_b,
|
||||
"build_a": args.build_a,
|
||||
"build_b": args.build_b,
|
||||
"ver_a": source_version(args.build_a),
|
||||
"ver_b": source_version(args.build_b),
|
||||
"drift_note": args.drift_note,
|
||||
}
|
||||
print(f"A: {args.build_a} source={meta['ver_a']}")
|
||||
print(f"B: {args.build_b} source={meta['ver_b']}")
|
||||
|
||||
if args.report_only:
|
||||
data = {}
|
||||
for eng in args.engines:
|
||||
data[eng] = {
|
||||
"a": parse_run(os.path.join(args.outdir, "a", eng)),
|
||||
"b": parse_run(os.path.join(args.outdir, "b", eng)),
|
||||
}
|
||||
else:
|
||||
clobber_ramdisk(args.ramdir, args.ram_mount) # clean slate up front; no end-of-run cleanup
|
||||
os.makedirs(args.ramdir, exist_ok=True)
|
||||
os.makedirs(args.outdir, exist_ok=True)
|
||||
data = collect(
|
||||
args.bench,
|
||||
args.build_a,
|
||||
args.build_b,
|
||||
args.engines,
|
||||
args.warmup,
|
||||
args.seconds,
|
||||
args.rows,
|
||||
args.ramdir,
|
||||
args.outdir,
|
||||
)
|
||||
|
||||
generate_html(data, args.report, meta, args.warmup, args.seconds, args.rows)
|
||||
print(f"\nReport written: {args.report}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,570 @@
|
|||
#!/usr/bin/env python3
|
||||
#
|
||||
# mako_ab_memtracker.py
|
||||
#
|
||||
# This source file is part of the FoundationDB open source project
|
||||
#
|
||||
# Copyright 2013-2026 Apple Inc. and the FoundationDB project authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
"""
|
||||
A/B benchmark: FDB per-call-site memory tracker OFF vs ON (1:100 sampling).
|
||||
|
||||
Uses contrib/mako_storage_bench.sh (single-host 1/1/1 loopback cluster, storage
|
||||
role isolated on ~one core, CPU-bound on /mnt/ram) to run each engine under both
|
||||
arms back-to-back on the same host, then generates a self-contained chart.js
|
||||
report (mako_memtracker_ab.html) viewable in any browser.
|
||||
|
||||
Both arms are the SAME build; only runtime knobs differ:
|
||||
OFF: --knob_memory_tracking_sample_inverse=0
|
||||
ON : --knob_memory_tracking_sample_inverse=100
|
||||
Both also force --knob_memory_tracking_report_interval=30 so the tracker's
|
||||
periodic dump fires within the run (equal cadence in both arms -> fair) and so
|
||||
we can verify the knob took by reading MemoryTrackerSummary's SampleInverse out
|
||||
of the storage process trace.
|
||||
|
||||
Run on the dev pod, e.g.:
|
||||
python3 /root/src/fdb5/foundationdb/contrib/mako_ab_memtracker.py --build /root/build_output5 \
|
||||
--engines redwood rocksdb --warmup 60 --seconds 240
|
||||
|
||||
Only SS/cluster data goes on tmpfs (--ramdir under /mnt/ram, clobbered clean at
|
||||
startup); per-arm metrics are saved under --outdir on /root so the report
|
||||
survives the next run's clobber.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
# The /root and /mnt/ram defaults below are dev-pod defaults; override via flags
|
||||
# (or FDB_BUILD) elsewhere. BENCH is found next to this script rather than hard-coded.
|
||||
_HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
BENCH_DEFAULT = os.path.join(_HERE, "mako_storage_bench.sh")
|
||||
BUILD_DEFAULT = os.environ.get("FDB_BUILD", "/root/build_output5")
|
||||
|
||||
# OFF = baseline blue, ON = warm orange.
|
||||
COLOR = {"off": "#4477CC", "on": "#EE7733"}
|
||||
ARMS = [("off", 0), ("on", 100)] # (label, sample_inverse)
|
||||
PCTS = [
|
||||
("medianLatency", "p50"),
|
||||
("p95Latency", "p95"),
|
||||
("p99Latency", "p99"),
|
||||
("p99.9Latency", "p99.9"),
|
||||
]
|
||||
|
||||
|
||||
def _assert_safe_scratch_mount(mount):
|
||||
"""Refuse to recursively delete anything that isn't clearly a dedicated tmpfs
|
||||
scratch mount. Guards against a typo or a stray --ram-mount (e.g. /mnt, /, or
|
||||
$HOME) wiping unrelated data before the run even starts."""
|
||||
real = os.path.realpath(mount)
|
||||
denylist = {"/", "/mnt", "/tmp", "/root", "/home", os.path.expanduser("~")}
|
||||
if real in denylist or real.count("/") < 2:
|
||||
raise SystemExit(
|
||||
f"refusing to clobber unsafe scratch path: {mount!r} -> {real!r}"
|
||||
)
|
||||
if not os.path.ismount(real):
|
||||
raise SystemExit(
|
||||
f"refusing to clobber {real!r}: not a mountpoint (expected a tmpfs mount)"
|
||||
)
|
||||
fstype = subprocess.run(
|
||||
["stat", "-f", "-c", "%T", real], capture_output=True, text=True
|
||||
).stdout.strip()
|
||||
if fstype != "tmpfs":
|
||||
raise SystemExit(
|
||||
f"refusing to clobber {real!r}: filesystem is {fstype!r}, not tmpfs"
|
||||
)
|
||||
|
||||
|
||||
def clobber_ramdisk(ramdir, ram_mount):
|
||||
"""Clear only this benchmark's own scratch dir under the tmpfs so every run
|
||||
starts clean, without touching anything else sharing the mount (e.g. other
|
||||
processes' data under /dev/shm). The ramdisk (~24 GB) fills fast across runs.
|
||||
Done at startup rather than teardown: a killed run can't be trusted to have
|
||||
cleaned up, and leaving the last run's data in place until the next run keeps
|
||||
it available to inspect when something fails."""
|
||||
if not os.path.isdir(ramdir):
|
||||
return
|
||||
_assert_safe_scratch_mount(ram_mount)
|
||||
real_mount = os.path.realpath(ram_mount)
|
||||
real_ramdir = os.path.realpath(ramdir)
|
||||
# Only ever delete a directory strictly beneath the validated tmpfs mount, so a
|
||||
# stray --ramdir can't wipe the mount root or unrelated data elsewhere.
|
||||
if real_ramdir == real_mount or os.path.commonpath([real_mount, real_ramdir]) != real_mount:
|
||||
raise SystemExit(
|
||||
f"refusing to clobber {ramdir!r} -> {real_ramdir!r}: not strictly under mount {real_mount!r}"
|
||||
)
|
||||
subprocess.run(["rm", "-rf", real_ramdir], check=False)
|
||||
print(f"clobbered benchmark scratch dir {real_ramdir}", flush=True)
|
||||
|
||||
|
||||
def save_metrics(dst, metrics):
|
||||
"""Persist an arm's computed metrics to /root so the report survives the
|
||||
next run's ramdisk clobber (the traces alloc-rate/knob-verify are derived
|
||||
from live only while the run's tmpfs data exists)."""
|
||||
os.makedirs(dst, exist_ok=True)
|
||||
with open(os.path.join(dst, "metrics.json"), "w") as f:
|
||||
json.dump(metrics, f)
|
||||
|
||||
|
||||
def load_metrics(dst):
|
||||
try:
|
||||
with open(os.path.join(dst, "metrics.json")) as f:
|
||||
return json.load(f)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return {"overallTPS": None, "persec": [], "latency": {}, "ok": False}
|
||||
|
||||
|
||||
def run_arm(bench, build, engine, arm, inverse, warmup, seconds, rows, outbase):
|
||||
"""Run one (engine, arm) via mako_storage_bench.sh; return its output dir."""
|
||||
workdir = os.path.join(outbase, arm)
|
||||
env = dict(os.environ)
|
||||
env["WORKDIR"] = workdir
|
||||
env["WARMUP_SECONDS"] = str(warmup)
|
||||
env["SECONDS_RUN"] = str(seconds)
|
||||
env["ROWS"] = str(rows)
|
||||
knobs = [
|
||||
f"--knob_memory_tracking_sample_inverse={inverse}",
|
||||
"--knob_memory_tracking_report_interval=30",
|
||||
]
|
||||
if engine == "rocksdb":
|
||||
# RocksDB opens its DB with O_DIRECT by default; tmpfs (/mnt/ram, where
|
||||
# this harness runs its data dir) does not support direct I/O, so the
|
||||
# storage engine fails to Open and the cluster never configures. Turn
|
||||
# direct I/O off via RocksDB's existing knobs (no new knobs added).
|
||||
knobs += [
|
||||
"--knob_rocksdb_use_direct_reads=0",
|
||||
"--knob_rocksdb_use_direct_io_flush_compaction=0",
|
||||
]
|
||||
env["KNOBS"] = " ".join(knobs)
|
||||
print(f"\n=== {engine} / {arm} (sample_inverse={inverse}) ===", flush=True)
|
||||
print(f" WORKDIR={workdir} KNOBS={env['KNOBS']}", flush=True)
|
||||
subprocess.run(["bash", bench, build, engine], env=env, check=False)
|
||||
return os.path.join(workdir, engine)
|
||||
|
||||
|
||||
def verify_sample_inverse(rundir):
|
||||
"""Read MemoryTrackerSummary's SampleInverse from the storage trace.
|
||||
Returns the observed int, or None if no summary event was found."""
|
||||
lc = os.path.join(rundir, "loopback-cluster")
|
||||
observed = None
|
||||
if not os.path.isdir(lc):
|
||||
return None
|
||||
for root, _, files in os.walk(lc):
|
||||
for fn in files:
|
||||
if "trace" not in fn:
|
||||
continue
|
||||
try:
|
||||
with open(os.path.join(root, fn), errors="ignore") as fh:
|
||||
for line in fh:
|
||||
if "MemoryTrackerSummary" in line:
|
||||
m = re.search(r'SampleInverse["\s:=]+(-?\d+)', line)
|
||||
if m:
|
||||
observed = int(m.group(1))
|
||||
except OSError:
|
||||
pass
|
||||
return observed
|
||||
|
||||
|
||||
def alloc_rate_from_trace(rundir):
|
||||
"""Estimate the storage process's allocation rate from the ON arm's
|
||||
MemoryTrackerSummary events (emitted every report interval, 30s here):
|
||||
rate = delta(EstCumulativeAllocs)/delta(time). Only meaningful when
|
||||
sampling is on. Returns {alloc_per_sec, mb_per_sec, samples_per_sec} or
|
||||
None."""
|
||||
lc = os.path.join(rundir, "loopback-cluster")
|
||||
if not os.path.isdir(lc):
|
||||
return None
|
||||
|
||||
def field(line, key, cast=float):
|
||||
m = re.search(key + r'="?(-?[\d.]+)"?', line)
|
||||
return cast(m.group(1)) if m else None
|
||||
|
||||
rows = []
|
||||
for root, _, files in os.walk(lc):
|
||||
for fn in files:
|
||||
if "trace" not in fn:
|
||||
continue
|
||||
try:
|
||||
for line in open(os.path.join(root, fn), errors="ignore"):
|
||||
if "MemoryTrackerSummary" not in line:
|
||||
continue
|
||||
t = field(line, "Time")
|
||||
ea = field(line, "EstCumulativeAllocs", int)
|
||||
eb = field(line, "EstCumulativeBytes", int)
|
||||
se = field(line, "SamplesEmitted", int)
|
||||
mm = re.search(r'Machine="([^"]+)"', line)
|
||||
rm = re.search(r'Roles="([^"]*)"', line)
|
||||
if t is not None and ea is not None:
|
||||
rows.append(
|
||||
(
|
||||
mm.group(1) if mm else "?",
|
||||
rm.group(1) if rm else "",
|
||||
t,
|
||||
ea,
|
||||
eb or 0,
|
||||
se or 0,
|
||||
)
|
||||
)
|
||||
except OSError:
|
||||
pass
|
||||
if not rows:
|
||||
return None
|
||||
# Pick the storage process: the machine with the highest peak
|
||||
# EstCumulativeAllocs (storage allocates far more than stateless/log). Use
|
||||
# all of that machine's summaries (early role-less ones included) so even
|
||||
# short runs yield >=2 points to difference.
|
||||
by_mach = {}
|
||||
for r in rows:
|
||||
by_mach.setdefault(r[0], []).append(r)
|
||||
best = max(by_mach.values(), key=lambda rs: max(x[3] for x in rs))
|
||||
best.sort(key=lambda r: r[2])
|
||||
if len(best) < 2:
|
||||
return None
|
||||
ar, br, sr = [], [], []
|
||||
for a, b in zip(best, best[1:]):
|
||||
dt = b[2] - a[2]
|
||||
if dt <= 0:
|
||||
continue
|
||||
ar.append((b[3] - a[3]) / dt)
|
||||
br.append((b[4] - a[4]) / dt)
|
||||
sr.append((b[5] - a[5]) / dt)
|
||||
if len(ar) >= 3: # drop the first interval (startup ramp)
|
||||
ar, br, sr = ar[1:], br[1:], sr[1:]
|
||||
if not ar:
|
||||
return None
|
||||
return {
|
||||
"alloc_per_sec": statistics.median(ar),
|
||||
"mb_per_sec": statistics.median(br) / 1e6,
|
||||
"samples_per_sec": statistics.median(sr),
|
||||
}
|
||||
|
||||
|
||||
def parse_run(rundir):
|
||||
"""Extract metrics from a completed run dir."""
|
||||
out = {"overallTPS": None, "persec": [], "latency": {}, "ok": False}
|
||||
mj = os.path.join(rundir, "mako.json")
|
||||
if os.path.exists(mj):
|
||||
try:
|
||||
d = json.load(open(mj))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return out
|
||||
res = d.get("results", {})
|
||||
out["overallTPS"] = res.get("overallTPS")
|
||||
out["persec"] = [s["tps"] for s in d.get("samples", []) if "tps" in s]
|
||||
for key, short in PCTS:
|
||||
blk = res.get(key, {})
|
||||
if isinstance(blk, dict) and "TRANSACTION" in blk:
|
||||
out["latency"][short] = blk["TRANSACTION"] # microseconds
|
||||
out["ok"] = out["overallTPS"] is not None
|
||||
# Fallback: Overall TPS from the teed text report.
|
||||
if out["overallTPS"] is None:
|
||||
mt = os.path.join(rundir, "mako-run.txt")
|
||||
if os.path.exists(mt):
|
||||
for line in open(mt, errors="ignore"):
|
||||
m = re.search(r"Overall TPS:\s*([\d.]+)", line)
|
||||
if m:
|
||||
out["overallTPS"] = float(m.group(1))
|
||||
out["ok"] = True
|
||||
out["rate"] = alloc_rate_from_trace(rundir)
|
||||
return out
|
||||
|
||||
|
||||
def collect(bench, build, engines, warmup, seconds, rows, ramdir, outdir):
|
||||
data = {} # engine -> arm -> {metrics, verified_inverse}
|
||||
for engine in engines:
|
||||
data[engine] = {}
|
||||
for arm, inverse in ARMS:
|
||||
rundir = run_arm(
|
||||
bench, build, engine, arm, inverse, warmup, seconds, rows, ramdir
|
||||
)
|
||||
metrics = parse_run(rundir) # reads mako.json + alloc-rate off tmpfs
|
||||
observed = verify_sample_inverse(rundir) # reads the trace off tmpfs
|
||||
metrics["verified_inverse"] = observed
|
||||
metrics["expected_inverse"] = inverse
|
||||
ok = "OK" if observed == inverse else f"MISMATCH (saw {observed})"
|
||||
print(
|
||||
f" -> overallTPS={metrics['overallTPS']} "
|
||||
f"knob-verify SampleInverse={observed} expected={inverse} [{ok}]",
|
||||
flush=True,
|
||||
)
|
||||
save_metrics(os.path.join(outdir, arm, engine), metrics) # persist to /root
|
||||
data[engine][arm] = metrics
|
||||
return data
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# HTML / chart.js generation
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def line_ds(label, series, color, dashed=False):
|
||||
d = {
|
||||
"label": label,
|
||||
"data": series,
|
||||
"borderColor": color,
|
||||
"backgroundColor": color,
|
||||
"pointRadius": 0,
|
||||
"borderWidth": 2,
|
||||
"tension": 0.2,
|
||||
"fill": False,
|
||||
}
|
||||
if dashed:
|
||||
d["borderDash"] = [6, 3]
|
||||
return d
|
||||
|
||||
|
||||
def generate_html(data, outpath, warmup, seconds, rows):
|
||||
engines = list(data.keys())
|
||||
|
||||
# ---- Summary rows ----
|
||||
rows_html = []
|
||||
for eng in engines:
|
||||
off = data[eng].get("off", {})
|
||||
on = data[eng].get("on", {})
|
||||
t_off, t_on = off.get("overallTPS"), on.get("overallTPS")
|
||||
dtps = (100.0 * (t_on - t_off) / t_off) if (t_off and t_on) else None
|
||||
p99_off = off.get("latency", {}).get("p99")
|
||||
p99_on = on.get("latency", {}).get("p99")
|
||||
dp99 = (100.0 * (p99_on - p99_off) / p99_off) if (p99_off and p99_on) else None
|
||||
|
||||
def fmt(v, s=""):
|
||||
return f"{v:,.0f}{s}" if isinstance(v, (int, float)) else "—"
|
||||
|
||||
def dfmt(v):
|
||||
if v is None:
|
||||
return "—"
|
||||
sign = "+" if v >= 0 else ""
|
||||
return f"{sign}{v:.2f}%"
|
||||
|
||||
rows_html.append(
|
||||
f"<tr><td>{eng}</td><td>{fmt(t_off)}</td><td>{fmt(t_on)}</td>"
|
||||
f"<td class='delta'>{dfmt(dtps)}</td>"
|
||||
f"<td>{fmt(p99_off,' µs')}</td><td>{fmt(p99_on,' µs')}</td>"
|
||||
f"<td class='delta'>{dfmt(dp99)}</td>"
|
||||
f"<td>off={off.get('verified_inverse')} / on={on.get('verified_inverse')}</td></tr>"
|
||||
)
|
||||
|
||||
# ---- Per-engine charts ----
|
||||
tps_labels = engines
|
||||
tps_off = [data[e].get("off", {}).get("overallTPS") or 0 for e in engines]
|
||||
tps_on = [data[e].get("on", {}).get("overallTPS") or 0 for e in engines]
|
||||
tps_datasets = json.dumps(
|
||||
[
|
||||
{
|
||||
"label": "off (inverse=0)",
|
||||
"data": tps_off,
|
||||
"backgroundColor": COLOR["off"],
|
||||
},
|
||||
{
|
||||
"label": "on (inverse=100)",
|
||||
"data": tps_on,
|
||||
"backgroundColor": COLOR["on"],
|
||||
},
|
||||
]
|
||||
)
|
||||
|
||||
blocks = []
|
||||
for i, eng in enumerate(engines):
|
||||
off = data[eng].get("off", {})
|
||||
on = data[eng].get("on", {})
|
||||
# per-second time series
|
||||
ps_datasets = json.dumps(
|
||||
[
|
||||
line_ds(f"{eng} off", off.get("persec", []), COLOR["off"]),
|
||||
line_ds(f"{eng} on", on.get("persec", []), COLOR["on"], dashed=True),
|
||||
]
|
||||
)
|
||||
# latency percentiles
|
||||
lat_labels = json.dumps([s for _, s in PCTS])
|
||||
lat_off = [off.get("latency", {}).get(s) for _, s in PCTS]
|
||||
lat_on = [on.get("latency", {}).get(s) for _, s in PCTS]
|
||||
lat_datasets = json.dumps(
|
||||
[
|
||||
{"label": "off", "data": lat_off, "backgroundColor": COLOR["off"]},
|
||||
{"label": "on", "data": lat_on, "backgroundColor": COLOR["on"]},
|
||||
]
|
||||
)
|
||||
blocks.append(
|
||||
f"""
|
||||
<h2>{eng}</h2>
|
||||
<div class="chart-box"><canvas id="ps{i}"></canvas></div>
|
||||
<div class="chart-box"><canvas id="lat{i}"></canvas></div>
|
||||
<script>
|
||||
new Chart(document.getElementById('ps{i}'), {{
|
||||
type: 'line',
|
||||
data: {{ labels: [...Array({max(len(off.get('persec',[])), len(on.get('persec',[])), 1)}).keys()],
|
||||
datasets: {ps_datasets} }},
|
||||
options: {{ animation:false, responsive:true,
|
||||
scales: {{ x: {{ title:{{display:true,text:'sample (~1 s each, incl. warmup)'}} }},
|
||||
y: {{ beginAtZero:true, title:{{display:true,text:'TPS'}} }} }},
|
||||
plugins: {{ title:{{display:true,text:'{eng}: per-second throughput (off vs on)'}} }} }}
|
||||
}});
|
||||
new Chart(document.getElementById('lat{i}'), {{
|
||||
type: 'bar',
|
||||
data: {{ labels: {lat_labels}, datasets: {lat_datasets} }},
|
||||
options: {{ animation:false, responsive:true,
|
||||
scales: {{ y: {{ beginAtZero:true, title:{{display:true,text:'transaction latency (µs)'}} }} }},
|
||||
plugins: {{ title:{{display:true,text:'{eng}: transaction latency percentiles (off vs on)'}} }} }}
|
||||
}});
|
||||
</script>"""
|
||||
)
|
||||
|
||||
rate_bits = []
|
||||
for eng in engines:
|
||||
r = data[eng].get("on", {}).get("rate")
|
||||
if r:
|
||||
rate_bits.append(
|
||||
f"{eng} ≈ {r['alloc_per_sec'] / 1e6:.2f} M allocs/s "
|
||||
f"({r['mb_per_sec']:.0f} MB/s, {r['samples_per_sec'] / 1e3:.0f}K samples/s)"
|
||||
)
|
||||
rate_note = (
|
||||
(
|
||||
"<p class='sub'><b>Observed storage-process allocation rate (on arm)</b>, "
|
||||
"from the tracker's own <code>MemoryTrackerSummary</code> "
|
||||
"(ΔEstCumulativeAllocs / report interval): "
|
||||
+ "; ".join(rate_bits)
|
||||
+ ". This is the rate the per-free global-lock cost scales with — far above "
|
||||
"a single-threaded microbenchmark's assumed 100K/s, which (with cross-thread lock "
|
||||
"contention) is why the end-to-end overhead here exceeds the μbench estimate.</p>"
|
||||
)
|
||||
if rate_bits
|
||||
else ""
|
||||
)
|
||||
|
||||
html = f"""<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>FDB Memory Tracker A/B: off vs 1:100 sampling</title>
|
||||
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.4/dist/chart.umd.min.js"></script>
|
||||
<style>
|
||||
body {{ font-family: sans-serif; max-width: 960px; margin: 40px auto; padding: 0 20px; background:#fafafa; }}
|
||||
h1 {{ font-size:1.5em; margin-bottom:.1em; }} h2 {{ margin-top:2em; }}
|
||||
p.sub {{ color:#555; }}
|
||||
table {{ border-collapse:collapse; width:100%; margin:1em 0; background:#fff; }}
|
||||
th,td {{ border:1px solid #ddd; padding:6px 10px; text-align:right; }}
|
||||
th:first-child, td:first-child {{ text-align:left; }}
|
||||
td.delta {{ font-weight:bold; }}
|
||||
.chart-box {{ background:#fff; border:1px solid #ddd; border-radius:6px; padding:20px; margin-bottom:24px; }}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>FDB Memory Tracker A/B — off vs 1:100 sampling</h1>
|
||||
<p class="sub">Same build; only the runtime knob differs
|
||||
(<code>memory_tracking_sample_inverse</code> 0 vs 100, report interval forced to 30 s in both).
|
||||
Single-host 1/1/1 loopback cluster on /mnt/ram (storage isolated, CPU-bound) via
|
||||
<code>contrib/mako_storage_bench.sh</code>. Workload g18ui,
|
||||
rows={rows:,}, warmup {warmup}s, run {seconds}s. A TPS drop in the "on" arm means the
|
||||
tracker added CPU cost on the storage hot path; near-parity means no regression.</p>
|
||||
|
||||
<h2>Summary</h2>
|
||||
<table>
|
||||
<tr><th>engine</th><th>TPS off</th><th>TPS on</th><th>Δ TPS</th>
|
||||
<th>p99 lat off</th><th>p99 lat on</th><th>Δ p99</th>
|
||||
<th>knob verify (SampleInverse)</th></tr>
|
||||
{''.join(rows_html)}
|
||||
</table>
|
||||
{rate_note}
|
||||
|
||||
<div class="chart-box"><canvas id="tps"></canvas></div>
|
||||
<script>
|
||||
new Chart(document.getElementById('tps'), {{
|
||||
type: 'bar',
|
||||
data: {{ labels: {json.dumps(tps_labels)}, datasets: {tps_datasets} }},
|
||||
options: {{ animation:false, responsive:true,
|
||||
scales: {{ y: {{ beginAtZero:true, title:{{display:true,text:'Overall TPS'}} }} }},
|
||||
plugins: {{ title:{{display:true,text:'Overall throughput (off vs on)'}} }} }}
|
||||
}});
|
||||
</script>
|
||||
{''.join(blocks)}
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
with open(outpath, "w") as f:
|
||||
f.write(html)
|
||||
return outpath
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(
|
||||
description="Memory-tracker off/on A/B via mako_storage_bench.sh"
|
||||
)
|
||||
ap.add_argument("--build", default=BUILD_DEFAULT)
|
||||
ap.add_argument("--bench", default=BENCH_DEFAULT)
|
||||
ap.add_argument("--engines", nargs="+", default=["redwood", "rocksdb"])
|
||||
ap.add_argument("--warmup", type=int, default=60)
|
||||
ap.add_argument("--seconds", type=int, default=240)
|
||||
ap.add_argument("--rows", type=int, default=100000)
|
||||
ap.add_argument(
|
||||
"--ramdir",
|
||||
default="/mnt/ram/memtracker_ab",
|
||||
help="tmpfs scratch for SS/cluster data (only SS data lives on /mnt/ram)",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--ram-mount", default="/mnt/ram", help="tmpfs mount clobbered clean at startup"
|
||||
)
|
||||
ap.add_argument(
|
||||
"--outdir",
|
||||
default="/root/memtracker_ab_results",
|
||||
help="persistent results dir on /root (~1 TB); per-arm metrics saved here",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--report",
|
||||
default=None,
|
||||
help="HTML output path (default: /root/src/mako_memtracker_ab.html, "
|
||||
"which syncs to ~/src on the Mac)",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--report-only",
|
||||
action="store_true",
|
||||
help="Skip running; regenerate HTML from an existing --outdir",
|
||||
)
|
||||
args = ap.parse_args()
|
||||
|
||||
# Default the report into the okteto sync root (~/src <-> /root/src) so it
|
||||
# lands on the Mac for viewing without a separate copy step.
|
||||
report = args.report or "/root/src/mako_memtracker_ab.html"
|
||||
|
||||
if args.report_only:
|
||||
data = {}
|
||||
for eng in args.engines:
|
||||
data[eng] = {}
|
||||
for arm, inverse in ARMS:
|
||||
data[eng][arm] = load_metrics(os.path.join(args.outdir, arm, eng))
|
||||
else:
|
||||
clobber_ramdisk(args.ramdir, args.ram_mount) # clean slate up front; no end-of-run cleanup
|
||||
os.makedirs(args.ramdir, exist_ok=True)
|
||||
os.makedirs(args.outdir, exist_ok=True)
|
||||
data = collect(
|
||||
args.bench,
|
||||
args.build,
|
||||
args.engines,
|
||||
args.warmup,
|
||||
args.seconds,
|
||||
args.rows,
|
||||
args.ramdir,
|
||||
args.outdir,
|
||||
)
|
||||
|
||||
generate_html(data, report, args.warmup, args.seconds, args.rows)
|
||||
print(f"\nReport written: {report}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -229,3 +229,8 @@ if(NOT OPEN_FOR_IDE)
|
|||
endif()
|
||||
|
||||
target_link_libraries(fdbserver PUBLIC fdbctl)
|
||||
|
||||
if(NOT FOUNDATIONDB_CROSS_COMPILING) # FIXME(swift): make this work when
|
||||
# x-compiling.
|
||||
add_subdirectory(bench EXCLUDE_FROM_ALL)
|
||||
endif()
|
||||
|
|
|
|||
|
|
@ -0,0 +1,248 @@
|
|||
/*
|
||||
* GlobalNewDelete.cpp
|
||||
*
|
||||
* This source file is part of the FoundationDB open source project
|
||||
*
|
||||
* Copyright 2013-2026 Apple Inc. and the FoundationDB project authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
// Process-wide replacements for the global operator new / operator delete set,
|
||||
// owned by the fdbserver binary.
|
||||
//
|
||||
// These live here, in a translation unit compiled directly into the fdbserver
|
||||
// executable, rather than in the `flow` static library, for two reasons:
|
||||
//
|
||||
// 1. Correctness of interposition. operator new / operator delete are
|
||||
// replaceable functions; a definition sitting in a static archive is only
|
||||
// pulled into the link if the linker already needs some other symbol from
|
||||
// that same object file. Placing them in an executable TU guarantees the
|
||||
// replacements are part of the final link instead of relying on incidental
|
||||
// archive pull-in.
|
||||
//
|
||||
// 2. Client isolation. `flow` is linked into libfdb_c and every client
|
||||
// binding; a global-new override compiled into it would interpose the
|
||||
// entire host process's allocator in any application that loads the client.
|
||||
// fdbserver is a standalone executable that clients never link, so keeping
|
||||
// these here confines the interposition to the server.
|
||||
//
|
||||
// Exactly one implementation is compiled, chosen by the same ALLOC_INSTRUMENTATION
|
||||
// flags the legacy accounting framework uses (so the two never define the global
|
||||
// operators twice):
|
||||
//
|
||||
// - ALLOC_INSTRUMENTATION[_STDOUT] on -> legacy FastAlloc accounting hooks.
|
||||
// - otherwise -> the sampled per-call-site memory
|
||||
// tracker (flow/MemoryTracker.*).
|
||||
|
||||
#include <cstdlib>
|
||||
#include <new>
|
||||
|
||||
#include "flow/MemoryTracker.h" // for FDB_MEMORY_TRACKER (default on)
|
||||
|
||||
// TODO: the old ALLOC_INSTRUMENTATION doesn't seem to be usable at
|
||||
// scale. Consider deleting it.
|
||||
#if defined(ALLOC_INSTRUMENTATION) || defined(ALLOC_INSTRUMENTATION_STDOUT)
|
||||
|
||||
#include "flow/FastAlloc.h"
|
||||
|
||||
void* operator new(std::size_t size) {
|
||||
void* p = malloc(size);
|
||||
if (!p) {
|
||||
throw std::bad_alloc();
|
||||
}
|
||||
recordAllocation(p, size);
|
||||
return p;
|
||||
}
|
||||
void operator delete(void* ptr) throw() {
|
||||
recordDeallocation(ptr);
|
||||
free(ptr);
|
||||
}
|
||||
|
||||
void* operator new(std::size_t size, const std::nothrow_t&) throw() {
|
||||
void* p = malloc(size);
|
||||
recordAllocation(p, size);
|
||||
return p;
|
||||
}
|
||||
void operator delete(void* ptr, const std::nothrow_t&) throw() {
|
||||
recordDeallocation(ptr);
|
||||
free(ptr);
|
||||
}
|
||||
|
||||
void* operator new[](std::size_t size) {
|
||||
void* p = malloc(size);
|
||||
if (!p) {
|
||||
throw std::bad_alloc();
|
||||
}
|
||||
recordAllocation(p, size);
|
||||
return p;
|
||||
}
|
||||
void operator delete[](void* ptr) throw() {
|
||||
recordDeallocation(ptr);
|
||||
free(ptr);
|
||||
}
|
||||
|
||||
void* operator new[](std::size_t size, const std::nothrow_t&) throw() {
|
||||
void* p = malloc(size);
|
||||
recordAllocation(p, size);
|
||||
return p;
|
||||
}
|
||||
void operator delete[](void* ptr, const std::nothrow_t&) throw() {
|
||||
recordDeallocation(ptr);
|
||||
free(ptr);
|
||||
}
|
||||
|
||||
#else // sampled memory tracker, see design/memory-tracker.md
|
||||
|
||||
#include "flow/Platform.h" // aligned_alloc / aligned_free (portable across MSVC/POSIX)
|
||||
|
||||
#if FDB_MEMORY_TRACKER
|
||||
|
||||
// NOTE: We (Apple) do not maintain a local facility to build FDB with MSVC on
|
||||
// Windows, and CI only *configures* (not compiles) there — so the MSVC-specific
|
||||
// pieces below are best-effort and not compile-verified: the exact signatures of
|
||||
// the replaceable global operator new/delete set, and the
|
||||
// aligned_alloc/aligned_free ↔ _aligned_malloc/_aligned_free pairing routed
|
||||
// through flow/Platform.h. This code may have issues on MSVC; community help for
|
||||
// the Windows build would be welcome. (See flow/MemoryTracker.cpp for the parallel
|
||||
// note on the non-Linux frame walker.)
|
||||
|
||||
namespace {
|
||||
|
||||
// Retry through the installed std::new_handler on failure, as the default
|
||||
// operator new does. fdbserver installs platform::outOfMemory, so an allocation
|
||||
// failure (including the tracker's own map growth) reaches FDB's OOM diagnostics
|
||||
// and FDB_EXIT_NO_MEM rather than throwing straight past them.
|
||||
void* mallocWithNewHandler(std::size_t n) {
|
||||
void* p;
|
||||
while (!(p = std::malloc(n))) {
|
||||
std::new_handler h = std::get_new_handler();
|
||||
if (!h) {
|
||||
throw std::bad_alloc();
|
||||
}
|
||||
h();
|
||||
}
|
||||
return p;
|
||||
}
|
||||
|
||||
// Same handler loop for over-aligned allocations. C11 aligned_alloc requires the
|
||||
// size to be a multiple of the alignment, so round up (harmless over-allocation)
|
||||
// to accept arbitrary operator-new sizes.
|
||||
void* alignedAllocWithNewHandler(std::size_t alignment, std::size_t n) {
|
||||
std::size_t rounded = (n + alignment - 1) & ~(alignment - 1);
|
||||
if (rounded < n) {
|
||||
throw std::bad_alloc(); // round-up overflowed; the request can't be satisfied
|
||||
}
|
||||
void* p;
|
||||
while (!(p = aligned_alloc(alignment, rounded))) {
|
||||
std::new_handler h = std::get_new_handler();
|
||||
if (!h) {
|
||||
throw std::bad_alloc();
|
||||
}
|
||||
h();
|
||||
}
|
||||
return p;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void* operator new(std::size_t n) {
|
||||
void* p = mallocWithNewHandler(n);
|
||||
memTrackerOnAlloc(p, n);
|
||||
return p;
|
||||
}
|
||||
void operator delete(void* p) noexcept {
|
||||
memTrackerOnFree(p);
|
||||
std::free(p);
|
||||
}
|
||||
void operator delete(void* p, std::size_t) noexcept {
|
||||
memTrackerOnFree(p);
|
||||
std::free(p);
|
||||
}
|
||||
|
||||
void* operator new[](std::size_t n) {
|
||||
void* p = mallocWithNewHandler(n);
|
||||
memTrackerOnAlloc(p, n);
|
||||
return p;
|
||||
}
|
||||
void operator delete[](void* p) noexcept {
|
||||
memTrackerOnFree(p);
|
||||
std::free(p);
|
||||
}
|
||||
void operator delete[](void* p, std::size_t) noexcept {
|
||||
memTrackerOnFree(p);
|
||||
std::free(p);
|
||||
}
|
||||
|
||||
void* operator new(std::size_t n, const std::nothrow_t&) noexcept {
|
||||
try {
|
||||
void* p = mallocWithNewHandler(n);
|
||||
memTrackerOnAlloc(p, n);
|
||||
return p;
|
||||
} catch (...) {
|
||||
return nullptr;
|
||||
}
|
||||
}
|
||||
void operator delete(void* p, const std::nothrow_t&) noexcept {
|
||||
memTrackerOnFree(p);
|
||||
std::free(p);
|
||||
}
|
||||
|
||||
void* operator new[](std::size_t n, const std::nothrow_t&) noexcept {
|
||||
try {
|
||||
void* p = mallocWithNewHandler(n);
|
||||
memTrackerOnAlloc(p, n);
|
||||
return p;
|
||||
} catch (...) {
|
||||
return nullptr;
|
||||
}
|
||||
}
|
||||
void operator delete[](void* p, const std::nothrow_t&) noexcept {
|
||||
memTrackerOnFree(p);
|
||||
std::free(p);
|
||||
}
|
||||
|
||||
// C++17 over-aligned new/delete. aligned_alloc/aligned_free (flow/Platform.h)
|
||||
// keep the alloc and free sides paired on MSVC (_aligned_malloc/_aligned_free).
|
||||
void* operator new(std::size_t n, std::align_val_t a) {
|
||||
void* p = alignedAllocWithNewHandler(static_cast<std::size_t>(a), n);
|
||||
memTrackerOnAlloc(p, n);
|
||||
return p;
|
||||
}
|
||||
void operator delete(void* p, std::align_val_t) noexcept {
|
||||
memTrackerOnFree(p);
|
||||
aligned_free(p);
|
||||
}
|
||||
void operator delete(void* p, std::size_t, std::align_val_t) noexcept {
|
||||
memTrackerOnFree(p);
|
||||
aligned_free(p);
|
||||
}
|
||||
|
||||
void* operator new[](std::size_t n, std::align_val_t a) {
|
||||
void* p = alignedAllocWithNewHandler(static_cast<std::size_t>(a), n);
|
||||
memTrackerOnAlloc(p, n);
|
||||
return p;
|
||||
}
|
||||
void operator delete[](void* p, std::align_val_t) noexcept {
|
||||
memTrackerOnFree(p);
|
||||
aligned_free(p);
|
||||
}
|
||||
void operator delete[](void* p, std::size_t, std::align_val_t) noexcept {
|
||||
memTrackerOnFree(p);
|
||||
aligned_free(p);
|
||||
}
|
||||
|
||||
#else // !FDB_MEMORY_TRACKER — no global operator new/delete override; libc++'s is used.
|
||||
#endif // FDB_MEMORY_TRACKER
|
||||
|
||||
#endif // ALLOC_INSTRUMENTATION
|
||||
|
|
@ -0,0 +1,769 @@
|
|||
/*
|
||||
* MemoryTrackerTest.cpp
|
||||
*
|
||||
* This source file is part of the FoundationDB open source project
|
||||
*
|
||||
* Copyright 2013-2026 Apple Inc. and the FoundationDB project authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
// Unit tests for the per-call-site memory tracker.
|
||||
//
|
||||
// The "coverage" test uses sentinel functions: each sentinel triggers exactly
|
||||
// one allocation path (operator new, FastAllocator, Arena), and the test
|
||||
// confirms that some call site in the aggregation table contains a frame
|
||||
// inside that sentinel's body. We compare raw return-address values against
|
||||
// function-pointer values at runtime, so this works on stripped builds with
|
||||
// no symbolization.
|
||||
|
||||
#include "flow/Arena.h"
|
||||
#include "flow/FastAlloc.h"
|
||||
#include "flow/Knobs.h"
|
||||
#include "flow/MemoryTracker.h"
|
||||
#include "flow/Platform.h"
|
||||
#include "flow/UnitTest.h"
|
||||
|
||||
#include <climits>
|
||||
#include <cstdint>
|
||||
#include <cstdlib>
|
||||
#include <limits>
|
||||
#include <new>
|
||||
#include <vector>
|
||||
|
||||
// Force this TU to link. The TEST_CASE macro registers via a static
|
||||
// initializer; in a static library, a TU containing only static initializers
|
||||
// gets dropped by the linker because nothing references its symbols.
|
||||
// fdbserver/workloads/UnitTests.cpp calls this function to keep the TU.
|
||||
void forceLinkMemoryTrackerTests() {}
|
||||
|
||||
#if FDB_MEMORY_TRACKER
|
||||
|
||||
namespace {
|
||||
|
||||
// A sentinel is an out-of-line function that performs exactly one kind of
|
||||
// allocation, then returns its own address. We use the returned address to
|
||||
// recognize captured stack frames that fell inside the sentinel's body.
|
||||
constexpr uintptr_t SENTINEL_FUNC_SIZE = 4096;
|
||||
|
||||
// Defeat clang -O3 heap elision (P0593): if the allocated pointer doesn't
|
||||
// escape, the compiler is free to drop the new/delete pair entirely, which
|
||||
// then never reaches our operator-new override and the test sees zero samples.
|
||||
void* volatile gEscapeSink;
|
||||
inline void escape(void* p) {
|
||||
gEscapeSink = p;
|
||||
}
|
||||
|
||||
bool frameInside(void* frame, void* sentinel) {
|
||||
uintptr_t f = reinterpret_cast<uintptr_t>(frame);
|
||||
uintptr_t s = reinterpret_cast<uintptr_t>(sentinel);
|
||||
return f >= s && f < s + SENTINEL_FUNC_SIZE;
|
||||
}
|
||||
|
||||
force_noinline void* triggerOperatorNewSentinel(int n, int k) {
|
||||
for (int i = 0; i < n; i++) {
|
||||
auto* p = new int[k];
|
||||
p[0] = i;
|
||||
escape(p);
|
||||
delete[] p;
|
||||
}
|
||||
return reinterpret_cast<void*>(&triggerOperatorNewSentinel);
|
||||
}
|
||||
|
||||
force_noinline void* triggerFastAllocSentinel(int n) {
|
||||
for (int i = 0; i < n; i++) {
|
||||
void* p = FastAllocator<32>::allocate();
|
||||
escape(p);
|
||||
FastAllocator<32>::release(p);
|
||||
}
|
||||
return reinterpret_cast<void*>(&triggerFastAllocSentinel);
|
||||
}
|
||||
|
||||
force_noinline void* triggerArenaSentinel(int n) {
|
||||
// Force ArenaBlock::create by allocating large enough chunks to exceed
|
||||
// the small-block threshold.
|
||||
for (int i = 0; i < n; i++) {
|
||||
Arena a;
|
||||
// One ~512-byte allocation per arena -> goes through allocateAndMaybeKeepalive
|
||||
// path which has the explicit Arena hook.
|
||||
auto* p = new (a) uint8_t[600];
|
||||
escape(p);
|
||||
}
|
||||
return reinterpret_cast<void*>(&triggerArenaSentinel);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Accounting tests: verify byte/block counts come out right per allocation path
|
||||
// and there's no double-tracking.
|
||||
force_noinline void* allocateArenaMediumSentinel(int n, std::vector<Arena>& arenas) {
|
||||
for (int i = 0; i < n; i++) {
|
||||
arenas.emplace_back();
|
||||
auto* p = new (arenas.back()) uint8_t[600];
|
||||
escape(p);
|
||||
}
|
||||
return reinterpret_cast<void*>(&allocateArenaMediumSentinel);
|
||||
}
|
||||
|
||||
force_noinline void* allocateArenaHugeSentinel(int n, std::vector<Arena>& arenas) {
|
||||
for (int i = 0; i < n; i++) {
|
||||
arenas.emplace_back();
|
||||
auto* p = new (arenas.back()) uint8_t[100000];
|
||||
escape(p);
|
||||
}
|
||||
return reinterpret_cast<void*>(&allocateArenaHugeSentinel);
|
||||
}
|
||||
|
||||
force_noinline void* allocateArenaSmallSentinel(int n, std::vector<Arena>& arenas) {
|
||||
for (int i = 0; i < n; i++) {
|
||||
arenas.emplace_back();
|
||||
auto* p = new (arenas.back()) uint8_t[64];
|
||||
escape(p);
|
||||
}
|
||||
return reinterpret_cast<void*>(&allocateArenaSmallSentinel);
|
||||
}
|
||||
|
||||
force_noinline void* allocateOperatorNewSentinel(int n, int k, std::vector<int*>& ptrs) {
|
||||
for (int i = 0; i < n; i++) {
|
||||
auto* p = new int[k];
|
||||
escape(p);
|
||||
ptrs.push_back(p);
|
||||
}
|
||||
return reinterpret_cast<void*>(&allocateOperatorNewSentinel);
|
||||
}
|
||||
|
||||
force_noinline void* allocateFastAlloc32Sentinel(int n, std::vector<void*>& ptrs) {
|
||||
for (int i = 0; i < n; i++) {
|
||||
void* p = FastAllocator<32>::allocate();
|
||||
escape(p);
|
||||
ptrs.push_back(p);
|
||||
}
|
||||
return reinterpret_cast<void*>(&allocateFastAlloc32Sentinel);
|
||||
}
|
||||
|
||||
force_noinline void releaseFastAlloc32(std::vector<void*>& ptrs) {
|
||||
for (void* p : ptrs) {
|
||||
FastAllocator<32>::release(p);
|
||||
}
|
||||
ptrs.clear();
|
||||
}
|
||||
|
||||
struct AccountingSummary {
|
||||
int sitesWithSentinelFrames = 0;
|
||||
int64_t cumBytesSentinel = 0;
|
||||
int64_t cumAllocsSentinel = 0;
|
||||
int64_t liveBytesSentinel = 0;
|
||||
int64_t liveCountSentinel = 0;
|
||||
int totalSites = 0;
|
||||
};
|
||||
|
||||
AccountingSummary collectAccounting(void* sentinel) {
|
||||
AccountingSummary acc;
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite& s) {
|
||||
acc.totalSites++;
|
||||
bool touches = false;
|
||||
for (int i = 0; i < s.exemplarFrameCount; i++) {
|
||||
if (frameInside(s.exemplarFrames[i], sentinel)) {
|
||||
touches = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (touches && s.cumulativeBytes > 0) {
|
||||
acc.sitesWithSentinelFrames++;
|
||||
acc.cumBytesSentinel += s.cumulativeBytes;
|
||||
acc.cumAllocsSentinel += s.cumulativeAllocs;
|
||||
acc.liveBytesSentinel += s.liveBytes;
|
||||
acc.liveCountSentinel += s.liveCount;
|
||||
}
|
||||
});
|
||||
return acc;
|
||||
}
|
||||
|
||||
void dumpSitesForFailure(const char* tag) {
|
||||
fprintf(stderr, "[%s] dumping all tracker sites:\n", tag);
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite& s) {
|
||||
fprintf(stderr,
|
||||
" fp=%016llx liveBytes=%lld liveCount=%lld cumBytes=%lld cumAllocs=%lld frames=",
|
||||
(unsigned long long)s.fingerprint,
|
||||
(long long)s.liveBytes,
|
||||
(long long)s.liveCount,
|
||||
(long long)s.cumulativeBytes,
|
||||
(long long)s.cumulativeAllocs);
|
||||
for (int i = 0; i < s.exemplarFrameCount; i++) {
|
||||
fprintf(stderr, "%p ", s.exemplarFrames[i]);
|
||||
}
|
||||
fprintf(stderr, "\n");
|
||||
});
|
||||
}
|
||||
|
||||
class KnobOverride {
|
||||
public:
|
||||
explicit KnobOverride(int inverse = 1) : prevInverse(FLOW_KNOBS->MEMORY_TRACKING_SAMPLE_INVERSE) {
|
||||
auto* k = const_cast<FlowKnobs*>(FLOW_KNOBS);
|
||||
k->MEMORY_TRACKING_SAMPLE_INVERSE = inverse;
|
||||
}
|
||||
~KnobOverride() {
|
||||
auto* k = const_cast<FlowKnobs*>(FLOW_KNOBS);
|
||||
k->MEMORY_TRACKING_SAMPLE_INVERSE = prevInverse;
|
||||
}
|
||||
|
||||
private:
|
||||
int prevInverse;
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/coverage") {
|
||||
#ifndef __linux__
|
||||
// captureFramesFP is a no-op stub on non-Linux (FP walking through libc
|
||||
// can't be made reliable on macOS); tests that inspect captured frames
|
||||
// have nothing to inspect. Skip cleanly. The tracker still compiles
|
||||
// and the non-frame tests (offSwitch, freeOfUntrackedPtrIsNoop) still
|
||||
// run.
|
||||
return Void();
|
||||
#endif
|
||||
// Sample everything, reset, run sentinels, check.
|
||||
KnobOverride ko;
|
||||
memTrackerResetForTest();
|
||||
|
||||
void* opNew = triggerOperatorNewSentinel(50, 4);
|
||||
void* fastAlloc = triggerFastAllocSentinel(50);
|
||||
void* arena = triggerArenaSentinel(50);
|
||||
|
||||
bool foundOpNew = false;
|
||||
bool foundFastAlloc = false;
|
||||
bool foundArena = false;
|
||||
int siteCount = 0;
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite& s) {
|
||||
siteCount++;
|
||||
for (int i = 0; i < s.exemplarFrameCount; i++) {
|
||||
if (frameInside(s.exemplarFrames[i], opNew)) {
|
||||
foundOpNew = true;
|
||||
}
|
||||
if (frameInside(s.exemplarFrames[i], fastAlloc)) {
|
||||
foundFastAlloc = true;
|
||||
}
|
||||
if (frameInside(s.exemplarFrames[i], arena)) {
|
||||
foundArena = true;
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
if (!foundOpNew || !foundFastAlloc || !foundArena) {
|
||||
fprintf(stderr,
|
||||
"MemoryTracker/coverage: sites=%d opNewSentinel=%p fastAllocSentinel=%p arenaSentinel=%p\n",
|
||||
siteCount,
|
||||
opNew,
|
||||
fastAlloc,
|
||||
arena);
|
||||
fprintf(stderr,
|
||||
"MemoryTracker/coverage: foundOpNew=%d foundFastAlloc=%d foundArena=%d\n",
|
||||
foundOpNew,
|
||||
foundFastAlloc,
|
||||
foundArena);
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite& s) {
|
||||
fprintf(stderr,
|
||||
" site fp=%016llx liveBytes=%lld cumAllocs=%lld frames=",
|
||||
(unsigned long long)s.fingerprint,
|
||||
(long long)s.liveBytes,
|
||||
(long long)s.cumulativeAllocs);
|
||||
for (int i = 0; i < s.exemplarFrameCount; i++) {
|
||||
fprintf(stderr, "%p ", s.exemplarFrames[i]);
|
||||
}
|
||||
fprintf(stderr, "\n");
|
||||
});
|
||||
}
|
||||
|
||||
ASSERT(foundOpNew);
|
||||
ASSERT(foundFastAlloc);
|
||||
ASSERT(foundArena);
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/offSwitch") {
|
||||
// With sample inverse 0, no allocations are attributed. memTrackerResetForTest
|
||||
// arms this thread's off-latch straight from the knob (as memTrackerInit does
|
||||
// at startup), so even the first allocation short-circuits.
|
||||
auto* k = const_cast<FlowKnobs*>(FLOW_KNOBS);
|
||||
int prev = k->MEMORY_TRACKING_SAMPLE_INVERSE;
|
||||
k->MEMORY_TRACKING_SAMPLE_INVERSE = 0;
|
||||
|
||||
memTrackerResetForTest();
|
||||
|
||||
for (int i = 0; i < 100; i++) {
|
||||
auto* p = new int[4];
|
||||
p[0] = i;
|
||||
delete[] p;
|
||||
}
|
||||
|
||||
int siteCount = 0;
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite&) { siteCount++; });
|
||||
ASSERT_EQ(siteCount, 0);
|
||||
|
||||
// The enabled flag gates the free hot path: with sampling off it must be
|
||||
// false, so memTrackerOnFree short-circuits before taking g_mtLock.
|
||||
ASSERT(!g_memTrackerEnabled.value.load(std::memory_order_relaxed));
|
||||
|
||||
k->MEMORY_TRACKING_SAMPLE_INVERSE = prev;
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/operatorNewHonorsNewHandler") {
|
||||
// The global operator new override (fdbserver/GlobalNewDelete.cpp) must run
|
||||
// the std::new_handler retry loop so an allocation failure reaches FDB's OOM
|
||||
// path instead of throwing straight past it. (Where the override isn't linked,
|
||||
// the standard library's operator new provides the same contract, so this
|
||||
// still passes.)
|
||||
static bool handlerRan;
|
||||
handlerRan = false;
|
||||
std::new_handler prev = std::set_new_handler([]() {
|
||||
handlerRan = true;
|
||||
throw std::bad_alloc(); // break the retry loop
|
||||
});
|
||||
|
||||
bool caught = false;
|
||||
try {
|
||||
// volatile so the compiler can't fold the size and warn (-Walloc-size); malloc
|
||||
// reliably fails for SIZE_MAX, driving the handler loop.
|
||||
volatile std::size_t huge = std::numeric_limits<std::size_t>::max();
|
||||
void* p = ::operator new(huge);
|
||||
escape(p);
|
||||
} catch (const std::bad_alloc&) {
|
||||
caught = true;
|
||||
}
|
||||
std::set_new_handler(prev);
|
||||
|
||||
ASSERT(handlerRan);
|
||||
ASSERT(caught);
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/samplingRate") {
|
||||
// The reseed gap is uniform on [1, 2N-1] (mean N), so at inverse N the sampled
|
||||
// fraction should be ~1/N. Wide bounds keep it non-flaky across RNG state.
|
||||
constexpr int N = 10;
|
||||
constexpr int ALLOCS = 200000;
|
||||
KnobOverride ko(N);
|
||||
memTrackerResetForTest();
|
||||
|
||||
std::vector<int*> ptrs;
|
||||
ptrs.reserve(ALLOCS);
|
||||
for (int i = 0; i < ALLOCS; i++) {
|
||||
auto* p = new int[4];
|
||||
escape(p);
|
||||
ptrs.push_back(p);
|
||||
}
|
||||
|
||||
int64_t sampled = 0;
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite& s) {
|
||||
if (s.forceSampledCount == 0) {
|
||||
sampled += s.cumulativeAllocs;
|
||||
}
|
||||
});
|
||||
|
||||
for (auto* p : ptrs) {
|
||||
delete[] p;
|
||||
}
|
||||
ptrs.clear();
|
||||
|
||||
double frac = double(sampled) / ALLOCS;
|
||||
ASSERT(frac > 0.06 && frac < 0.15); // expect ~0.1
|
||||
memTrackerResetForTest();
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/freeOfUntrackedPtrIsNoop") {
|
||||
// memTrackerOnFree on a pointer the tracker never recorded must be a no-op.
|
||||
KnobOverride ko;
|
||||
memTrackerResetForTest();
|
||||
|
||||
int x = 0;
|
||||
memTrackerOnFree(&x); // not in any table
|
||||
memTrackerOnFree(nullptr);
|
||||
|
||||
int siteCount = 0;
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite&) { siteCount++; });
|
||||
ASSERT_EQ(siteCount, 0);
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/cumulativeIsMonotonic") {
|
||||
#ifndef __linux__
|
||||
return Void(); // see /coverage for rationale
|
||||
#endif
|
||||
// liveCount must return to ~0 after we free everything we allocated;
|
||||
// cumulativeAllocs must NOT decrement.
|
||||
KnobOverride ko;
|
||||
memTrackerResetForTest();
|
||||
|
||||
void* sentinel = triggerOperatorNewSentinel(100, 8);
|
||||
|
||||
int64_t maxCumulative = 0;
|
||||
int64_t finalLive = 0;
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite& s) {
|
||||
for (int i = 0; i < s.exemplarFrameCount; i++) {
|
||||
if (frameInside(s.exemplarFrames[i], sentinel)) {
|
||||
if (s.cumulativeAllocs > maxCumulative) {
|
||||
maxCumulative = s.cumulativeAllocs;
|
||||
}
|
||||
finalLive += s.liveCount;
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
ASSERT(maxCumulative >= 100);
|
||||
ASSERT_EQ(finalLive, 0); // every alloc was paired with delete
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/estimateScaling") {
|
||||
// End-to-end estimate check. With a fixed inverse N > 1 and no force-sampled
|
||||
// blocks, every sample at a site carries weight N, so the site's estimated
|
||||
// usage must be *exactly* N times its raw sampled counters. This verifies
|
||||
// the reported Est* numbers without depending on which specific allocations
|
||||
// happened to be sampled. Runs on all platforms (no frame inspection).
|
||||
constexpr int N = 8;
|
||||
KnobOverride ko(N);
|
||||
memTrackerResetForTest();
|
||||
|
||||
// Small allocations far below the force-sample threshold, so none
|
||||
// are force-sampled and every sampled block gets weight N.
|
||||
std::vector<int*> ptrs;
|
||||
ptrs.reserve(5000);
|
||||
for (int i = 0; i < 5000; i++) {
|
||||
auto* p = new int[4];
|
||||
escape(p);
|
||||
ptrs.push_back(p);
|
||||
}
|
||||
|
||||
int checked = 0;
|
||||
int64_t rawLiveBefore = 0;
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite& s) {
|
||||
if (s.forceSampledCount != 0) {
|
||||
return; // ignore any incidental force-sampled site (weight 1, not N)
|
||||
}
|
||||
ASSERT_EQ(s.estCumulativeBytes, s.cumulativeBytes * N);
|
||||
ASSERT_EQ(s.estCumulativeAllocs, s.cumulativeAllocs * N);
|
||||
ASSERT_EQ(s.estLiveBytes, s.liveBytes * N);
|
||||
ASSERT_EQ(s.estLiveCount, s.liveCount * N);
|
||||
ASSERT_EQ(s.estPeakBytes, s.peakBytes * N);
|
||||
rawLiveBefore += s.liveBytes;
|
||||
checked++;
|
||||
});
|
||||
ASSERT(checked > 0);
|
||||
ASSERT(rawLiveBefore > 0);
|
||||
|
||||
for (auto* p : ptrs) {
|
||||
delete[] p;
|
||||
}
|
||||
ptrs.clear();
|
||||
|
||||
// Symmetric debit: the per-site scaling invariant must still hold after the
|
||||
// frees (each free debits the estimate by exactly weight×size), and the live
|
||||
// total must have dropped. We check the invariant rather than "live == 0"
|
||||
// because incidental still-live allocations (e.g. the ptrs vector's own
|
||||
// backing buffer) legitimately remain tracked.
|
||||
int64_t rawLiveAfter = 0;
|
||||
int64_t estLiveAfter = 0;
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite& s) {
|
||||
if (s.forceSampledCount != 0) {
|
||||
return;
|
||||
}
|
||||
ASSERT_EQ(s.estLiveBytes, s.liveBytes * N);
|
||||
rawLiveAfter += s.liveBytes;
|
||||
estLiveAfter += s.estLiveBytes;
|
||||
});
|
||||
ASSERT_EQ(estLiveAfter, rawLiveAfter * N);
|
||||
ASSERT(rawLiveAfter < rawLiveBefore); // the freed blocks were debited
|
||||
|
||||
memTrackerResetForTest();
|
||||
return Void();
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Accounting tests. The "sites with the sentinel's frames" assertion is the
|
||||
// main check here.
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/fastAlloc32Accounting") {
|
||||
#ifndef __linux__
|
||||
return Void(); // see /coverage for rationale
|
||||
#endif
|
||||
KnobOverride ko;
|
||||
constexpr int N = 30;
|
||||
|
||||
std::vector<void*> ptrs;
|
||||
ptrs.reserve(N);
|
||||
|
||||
memTrackerResetForTest();
|
||||
void* sentinel = allocateFastAlloc32Sentinel(N, ptrs);
|
||||
|
||||
auto pre = collectAccounting(sentinel);
|
||||
if (pre.sitesWithSentinelFrames != 1) {
|
||||
dumpSitesForFailure("fastAlloc32Accounting/post-alloc");
|
||||
}
|
||||
ASSERT_EQ(pre.sitesWithSentinelFrames, 1);
|
||||
ASSERT_EQ(pre.cumAllocsSentinel, N);
|
||||
ASSERT_EQ(pre.liveCountSentinel, N);
|
||||
ASSERT_EQ(pre.cumBytesSentinel, int64_t(N) * 32);
|
||||
ASSERT_EQ(pre.liveBytesSentinel, pre.cumBytesSentinel);
|
||||
// Global totals are intentionally not asserted: at inverse=1 a foreign-thread
|
||||
// allocation in the window would break a strict global equality (flaky at
|
||||
// Joshua scale); the sentinel-scoped checks above pin the regression.
|
||||
|
||||
releaseFastAlloc32(ptrs);
|
||||
|
||||
auto post = collectAccounting(sentinel);
|
||||
ASSERT_EQ(post.liveBytesSentinel, 0);
|
||||
ASSERT_EQ(post.liveCountSentinel, 0);
|
||||
// Global live totals intentionally not asserted (flaky at inverse=1; see above).
|
||||
ASSERT_EQ(post.cumAllocsSentinel, N); // cumulative never decrements
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/arenaSmallAccounting") {
|
||||
#ifndef __linux__
|
||||
return Void(); // see /coverage for rationale
|
||||
#endif
|
||||
KnobOverride ko;
|
||||
constexpr int N = 30;
|
||||
|
||||
std::vector<Arena> arenas;
|
||||
arenas.reserve(N);
|
||||
|
||||
memTrackerResetForTest();
|
||||
void* sentinel = allocateArenaSmallSentinel(N, arenas);
|
||||
|
||||
auto pre = collectAccounting(sentinel);
|
||||
if (pre.sitesWithSentinelFrames != 1) {
|
||||
dumpSitesForFailure("arenaSmallAccounting/post-alloc");
|
||||
}
|
||||
ASSERT_EQ(pre.sitesWithSentinelFrames, 1);
|
||||
ASSERT_EQ(pre.cumAllocsSentinel, N);
|
||||
ASSERT_EQ(pre.liveCountSentinel, N);
|
||||
ASSERT_EQ(pre.liveBytesSentinel, pre.cumBytesSentinel);
|
||||
// Global totals are intentionally not asserted: at inverse=1 a foreign-thread
|
||||
// allocation in the window would break a strict global equality (flaky at
|
||||
// Joshua scale); the sentinel-scoped checks above pin the regression.
|
||||
|
||||
arenas.clear();
|
||||
|
||||
auto post = collectAccounting(sentinel);
|
||||
ASSERT_EQ(post.liveBytesSentinel, 0);
|
||||
ASSERT_EQ(post.liveCountSentinel, 0);
|
||||
// Global live totals intentionally not asserted (flaky at inverse=1; see above).
|
||||
ASSERT_EQ(post.cumAllocsSentinel, N);
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/arenaMediumAccounting") {
|
||||
#ifndef __linux__
|
||||
return Void(); // see /coverage for rationale
|
||||
#endif
|
||||
// Make sure arenas aren't counted twice, once due to their direct
|
||||
// instrumentation and a second time due to their use of operator new.
|
||||
KnobOverride ko;
|
||||
constexpr int N = 30;
|
||||
|
||||
std::vector<Arena> arenas;
|
||||
arenas.reserve(N);
|
||||
|
||||
memTrackerResetForTest();
|
||||
void* sentinel = allocateArenaMediumSentinel(N, arenas);
|
||||
|
||||
auto pre = collectAccounting(sentinel);
|
||||
if (pre.sitesWithSentinelFrames != 1) {
|
||||
dumpSitesForFailure("arenaMediumAccounting/post-alloc");
|
||||
}
|
||||
ASSERT_EQ(pre.sitesWithSentinelFrames, 1);
|
||||
ASSERT_EQ(pre.cumAllocsSentinel, N);
|
||||
ASSERT_EQ(pre.liveCountSentinel, N);
|
||||
ASSERT_EQ(pre.liveBytesSentinel, pre.cumBytesSentinel);
|
||||
// Global totals intentionally not asserted (flaky at inverse=1; see above).
|
||||
|
||||
arenas.clear();
|
||||
|
||||
auto post = collectAccounting(sentinel);
|
||||
ASSERT_EQ(post.liveBytesSentinel, 0);
|
||||
ASSERT_EQ(post.liveCountSentinel, 0);
|
||||
// Global live totals intentionally not asserted (flaky at inverse=1; see above).
|
||||
ASSERT_EQ(post.cumAllocsSentinel, N);
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/arenaHugeAccounting") {
|
||||
#ifndef __linux__
|
||||
return Void(); // see /coverage for rationale
|
||||
#endif
|
||||
// Verify the huge Arena-block path (reqSize >= LARGE), including that a block is
|
||||
// tracked once (not double-counted by both the explicit Arena hook and the inner
|
||||
// operator new[]). This is the deepest tracked call chain, and under a real
|
||||
// (non-simulation) network its frame-pointer backtrace is unreliable — the
|
||||
// best-effort walker may, run to run and even alloc to alloc, fail to climb to
|
||||
// the test's frame or attribute the blocks to varying fingerprints. So instead of
|
||||
// the sentinel-frame approach the other *Accounting tests use, we identify the
|
||||
// huge blocks by their unmistakable ~100 KB size signature: the recorded size is
|
||||
// correct regardless of which frames were captured, and no incidental or
|
||||
// foreign-thread allocation comes anywhere near this large.
|
||||
KnobOverride ko;
|
||||
constexpr int N = 10;
|
||||
constexpr int64_t BLOCK = 100000;
|
||||
constexpr int64_t HUGE_MIN = 90000; // mean bytes/alloc of a huge site; nothing else is this big
|
||||
|
||||
std::vector<Arena> arenas;
|
||||
arenas.reserve(N);
|
||||
|
||||
memTrackerResetForTest();
|
||||
allocateArenaHugeSentinel(N, arenas);
|
||||
|
||||
int64_t cumAllocs = 0, cumBytes = 0, liveBytes = 0, liveCount = 0;
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite& s) {
|
||||
if (s.cumulativeAllocs > 0 && s.cumulativeBytes / s.cumulativeAllocs >= HUGE_MIN) {
|
||||
cumAllocs += s.cumulativeAllocs;
|
||||
cumBytes += s.cumulativeBytes;
|
||||
liveBytes += s.liveBytes;
|
||||
liveCount += s.liveCount;
|
||||
}
|
||||
});
|
||||
// Exactly N huge blocks, each tracked once (double-tracking would show as 2N),
|
||||
// all currently live, with live bytes == cumulative bytes (nothing freed yet).
|
||||
ASSERT_EQ(cumAllocs, N);
|
||||
ASSERT_EQ(liveCount, N);
|
||||
ASSERT_EQ(liveBytes, cumBytes);
|
||||
ASSERT(cumBytes >= int64_t(N) * BLOCK);
|
||||
|
||||
arenas.clear();
|
||||
|
||||
int64_t cumAllocsPost = 0, liveBytesPost = 0, liveCountPost = 0;
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite& s) {
|
||||
if (s.cumulativeAllocs > 0 && s.cumulativeBytes / s.cumulativeAllocs >= HUGE_MIN) {
|
||||
cumAllocsPost += s.cumulativeAllocs;
|
||||
liveBytesPost += s.liveBytes;
|
||||
liveCountPost += s.liveCount;
|
||||
}
|
||||
});
|
||||
// After freeing, the huge blocks are debited: live returns to 0, cumulative persists.
|
||||
ASSERT_EQ(liveBytesPost, 0);
|
||||
ASSERT_EQ(liveCountPost, 0);
|
||||
ASSERT_EQ(cumAllocsPost, N);
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/operatorNewAccounting") {
|
||||
#ifndef __linux__
|
||||
return Void(); // see /coverage for rationale
|
||||
#endif
|
||||
KnobOverride ko;
|
||||
constexpr int N = 30;
|
||||
constexpr int K = 8; // new int[8] -> 32 bytes; int is trivial so no array cookie
|
||||
|
||||
std::vector<int*> ptrs;
|
||||
ptrs.reserve(N);
|
||||
|
||||
memTrackerResetForTest();
|
||||
void* sentinel = allocateOperatorNewSentinel(N, K, ptrs);
|
||||
|
||||
auto pre = collectAccounting(sentinel);
|
||||
if (pre.sitesWithSentinelFrames != 1) {
|
||||
dumpSitesForFailure("operatorNewAccounting/post-alloc");
|
||||
}
|
||||
ASSERT_EQ(pre.sitesWithSentinelFrames, 1);
|
||||
ASSERT_EQ(pre.cumAllocsSentinel, N);
|
||||
ASSERT_EQ(pre.liveCountSentinel, N);
|
||||
ASSERT_EQ(pre.cumBytesSentinel, static_cast<int64_t>(N) * K * static_cast<int64_t>(sizeof(int)));
|
||||
ASSERT_EQ(pre.liveBytesSentinel, pre.cumBytesSentinel);
|
||||
// Global totals intentionally not asserted (flaky at inverse=1; see above).
|
||||
|
||||
for (auto* p : ptrs) {
|
||||
delete[] p;
|
||||
}
|
||||
ptrs.clear();
|
||||
|
||||
auto post = collectAccounting(sentinel);
|
||||
ASSERT_EQ(post.liveBytesSentinel, 0);
|
||||
ASSERT_EQ(post.liveCountSentinel, 0);
|
||||
// Global live totals intentionally not asserted (flaky at inverse=1; see above).
|
||||
ASSERT_EQ(post.cumAllocsSentinel, N);
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/failOpenOnMetadataAllocFailure") {
|
||||
// The tracker must fail open: if its own metadata allocation throws, the
|
||||
// underlying user allocation still succeeds and tracking recovers on this
|
||||
// thread afterward (the reentrancy guard is restored, not leaked). Runs on all
|
||||
// platforms — no frame inspection.
|
||||
KnobOverride ko; // inverse = 1: sample every allocation
|
||||
memTrackerResetForTest();
|
||||
|
||||
// Arm the one-shot; it is consumed by the next sampled allocation, which throws
|
||||
// inside the tracker. new/delete must not observe that exception.
|
||||
memTrackerFailNextSampleForTest();
|
||||
int* p = new int[4];
|
||||
ASSERT(p != nullptr);
|
||||
p[0] = 42;
|
||||
int observed = p[0];
|
||||
delete[] p;
|
||||
ASSERT_EQ(observed, 42);
|
||||
|
||||
// Tracking must still work after the injected failure.
|
||||
memTrackerResetForTest();
|
||||
std::vector<int*> ptrs;
|
||||
ptrs.reserve(8);
|
||||
for (int i = 0; i < 8; i++) {
|
||||
auto* q = new int[4];
|
||||
escape(q);
|
||||
ptrs.push_back(q);
|
||||
}
|
||||
int siteCount = 0;
|
||||
memTrackerForEachSite([&](const MemoryTrackerCallSite&) { siteCount++; });
|
||||
for (auto* q : ptrs) {
|
||||
delete[] q;
|
||||
}
|
||||
ASSERT(siteCount > 0);
|
||||
return Void();
|
||||
}
|
||||
|
||||
TEST_CASE("/flow/MemoryTracker/initEnablesFromKnob") {
|
||||
// Exercises the *production* enablement path (memTrackerInit), not the test-only
|
||||
// reset — the reviewer noted that memTrackerResetForTest masks the startup race.
|
||||
// memTrackerInit must set the global enabled flag and this thread's fast-path
|
||||
// off-latch straight from MEMORY_TRACKING_SAMPLE_INVERSE, and (the regression)
|
||||
// must re-arm a thread that a prior off-configuration had already latched off.
|
||||
auto* k = const_cast<FlowKnobs*>(FLOW_KNOBS);
|
||||
int prev = k->MEMORY_TRACKING_SAMPLE_INVERSE;
|
||||
|
||||
// Sampling off: init publishes disabled and latches this thread off. This is the
|
||||
// state an early startup allocation used to get stuck in before init existed.
|
||||
k->MEMORY_TRACKING_SAMPLE_INVERSE = 0;
|
||||
memTrackerInit();
|
||||
ASSERT(!g_memTrackerEnabled.value.load(std::memory_order_relaxed));
|
||||
ASSERT(gMemTrackerOff); // alloc hot path short-circuits
|
||||
|
||||
// Knobs now configured with sampling on: init must re-arm THIS thread. The bug
|
||||
// was that nothing re-armed the network thread once it latched off before the
|
||||
// knobs were ready, so sampling stayed dead for the life of the process.
|
||||
k->MEMORY_TRACKING_SAMPLE_INVERSE = 8;
|
||||
memTrackerInit();
|
||||
ASSERT(g_memTrackerEnabled.value.load(std::memory_order_relaxed));
|
||||
ASSERT(!gMemTrackerOff); // re-armed: alloc hot path now reaches the sampler
|
||||
|
||||
// And the reverse transition: a subsequent off-configuration re-latches it.
|
||||
k->MEMORY_TRACKING_SAMPLE_INVERSE = 0;
|
||||
memTrackerInit();
|
||||
ASSERT(!g_memTrackerEnabled.value.load(std::memory_order_relaxed));
|
||||
ASSERT(gMemTrackerOff);
|
||||
|
||||
k->MEMORY_TRACKING_SAMPLE_INVERSE = prev;
|
||||
memTrackerInit(); // restore tracker state from the baseline knob for later tests
|
||||
return Void();
|
||||
}
|
||||
|
||||
#endif // FDB_MEMORY_TRACKER
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
/*
|
||||
* BenchMain.cpp
|
||||
*
|
||||
* This source file is part of the FoundationDB open source project
|
||||
*
|
||||
* Copyright 2013-2026 Apple Inc. and the FoundationDB project authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
// Most microbenchmarks belong in flow/bench; this fdbserver-local benchmark
|
||||
// binary exists only for benchmarks that must link against fdbserver-only code.
|
||||
// See BenchMemoryTracker.cpp for why that one lives here.
|
||||
|
||||
#include "flow/BenchMain.h"
|
||||
|
||||
int main(int argc, char** argv) {
|
||||
return runBenchmarks(argc, argv);
|
||||
}
|
||||
|
|
@ -0,0 +1,167 @@
|
|||
/*
|
||||
* BenchMemoryTracker.cpp
|
||||
*
|
||||
* This source file is part of the FoundationDB open source project
|
||||
*
|
||||
* Copyright 2013-2026 Apple Inc. and the FoundationDB project authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
// Microbenchmarks for the per-call-site memory tracker (see
|
||||
// design/memory-tracker.md, Testing Considerations -> Microbenchmarks).
|
||||
//
|
||||
// Two questions, both about "is it cheap enough to leave on?" (R0):
|
||||
// * off-state cost — the always-compiled hooks with sampling disabled;
|
||||
// * enabled-state cost — hooks at the envisioned production 1% rate
|
||||
// and the pessimal every-allocation rate, which includes both the
|
||||
// sampled-alloc slow path and the per-free lock+probe (the
|
||||
// dominant enabled-state cost).
|
||||
//
|
||||
// Run: bin/fdbserver_bench --benchmark_filter=memtracker
|
||||
//
|
||||
// This bench lives under fdbserver/ (not flow/) and its CMake compiles
|
||||
// fdbserver/GlobalNewDelete.cpp into the executable, so the real global
|
||||
// operator new/delete override is active here and bench_memtracker_operator_new
|
||||
// actually exercises the tracker. flow_bench links only flow (no override), so
|
||||
// the operator-new path could not be measured there.
|
||||
//
|
||||
// FLOW_KNOBS points at the process-default (non-simulated) bootstrap knobs in
|
||||
// fdbserver_bench, so MEMORY_TRACKING_SAMPLE_INVERSE starts at 0 (off); we drive
|
||||
// it per benchmark via const_cast, exactly like the unit tests do.
|
||||
|
||||
#include "benchmark/benchmark.h"
|
||||
|
||||
#include "flow/Arena.h"
|
||||
#include "flow/FastAlloc.h"
|
||||
#include "flow/Knobs.h"
|
||||
#include "flow/MemoryTracker.h"
|
||||
|
||||
#include <cstdlib>
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int kSize = 64;
|
||||
|
||||
// Set the sample-inverse knob (0=off, N=1-in-N) and clear tracker state so the
|
||||
// run starts clean. Returns the previous inverse for restoration.
|
||||
int setInverseAndReset(int inverse) {
|
||||
auto* k = const_cast<FlowKnobs*>(FLOW_KNOBS);
|
||||
int prev = k->MEMORY_TRACKING_SAMPLE_INVERSE;
|
||||
k->MEMORY_TRACKING_SAMPLE_INVERSE = inverse;
|
||||
memTrackerResetForTest();
|
||||
return prev;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// Baseline: raw libc malloc/free. std::malloc is NOT hooked (we override
|
||||
// operator new, not libc malloc), so this is the tracker-free reference the
|
||||
// operator-new benchmark is compared against.
|
||||
static void bench_memtracker_malloc_free(benchmark::State& state) {
|
||||
for (auto _ : state) {
|
||||
void* p = std::malloc(kSize);
|
||||
benchmark::DoNotOptimize(p);
|
||||
std::free(p);
|
||||
}
|
||||
state.SetItemsProcessed(state.iterations());
|
||||
}
|
||||
|
||||
// End-to-end cost of a hooked allocation: global operator new[]/delete[] (which
|
||||
// fire memTrackerOnAlloc/OnFree) at sample inverse Arg(0). 0 = off, 100 = prod
|
||||
// 1%, 1 = every allocation. Compare Arg(0) against bench_memtracker_malloc_free
|
||||
// for the disabled-hook cost, and Arg(100)/Arg(1) against Arg(0) for sampling.
|
||||
static void bench_memtracker_operator_new(benchmark::State& state) {
|
||||
int prev = setInverseAndReset(state.range(0));
|
||||
for (auto _ : state) {
|
||||
char* p = new char[kSize];
|
||||
benchmark::DoNotOptimize(p);
|
||||
delete[] p;
|
||||
}
|
||||
state.SetItemsProcessed(state.iterations());
|
||||
setInverseAndReset(prev); // restore so later benchmarks aren't sampled
|
||||
}
|
||||
|
||||
// Isolated tracker-hook cost: call memTrackerOnAlloc/OnFree directly on one
|
||||
// preallocated buffer, with no real allocation in the loop, so only the
|
||||
// tracker's own work is measured. At inverse>0 with live-tracking on, every
|
||||
// OnFree still takes the global lock and probes the live table (the dominant
|
||||
// enabled-state cost), while ~1/inverse of the OnAlloc calls take the sampling
|
||||
// slow path (frame walk + table insert).
|
||||
//
|
||||
// This is doing a stack unwind against the same stack, and is going
|
||||
// to hit the same hash table entries each iteration, so this is definitely
|
||||
// a best-case estimate.
|
||||
static void bench_memtracker_hooks(benchmark::State& state) {
|
||||
int prev = setInverseAndReset(state.range(0));
|
||||
void* p = std::malloc(kSize);
|
||||
for (auto _ : state) {
|
||||
memTrackerOnAlloc(p, kSize);
|
||||
memTrackerOnFree(p);
|
||||
}
|
||||
state.SetItemsProcessed(state.iterations());
|
||||
std::free(p);
|
||||
setInverseAndReset(prev);
|
||||
}
|
||||
|
||||
BENCHMARK(bench_memtracker_malloc_free);
|
||||
BENCHMARK(bench_memtracker_operator_new)->Arg(0)->Arg(100)->Arg(1);
|
||||
BENCHMARK(bench_memtracker_hooks)->Arg(0)->Arg(100)->Arg(1);
|
||||
|
||||
// --- Per-path unweighted overhead (for the FDB_MEMORY_TRACKER compile gate) ---
|
||||
// Plain allocation loops with no knob manipulation: in a default build
|
||||
// (FDB_MEMORY_TRACKER=1) they run with the tracker present but sampling off; in a
|
||||
// FDB_MEMORY_TRACKER=0 build the tracker code is absent. Diffing the two builds
|
||||
// gives each path's always-compiled off-state cost per op, unweighted by how
|
||||
// often the path is actually taken at runtime.
|
||||
|
||||
// operator new[]/delete[] at size Arg(0): exercises GlobalNewDelete.cpp (our
|
||||
// override when FDB_MEMORY_TRACKER, else libc++'s operator new).
|
||||
static void bench_path_operator_new(benchmark::State& state) {
|
||||
size_t n = state.range(0);
|
||||
for (auto _ : state) {
|
||||
char* p = new char[n];
|
||||
benchmark::DoNotOptimize(p);
|
||||
delete[] p;
|
||||
}
|
||||
state.SetItemsProcessed(state.iterations());
|
||||
}
|
||||
|
||||
// FastAllocator<Size> allocate/release: exercises the flow/FastAlloc.cpp hook.
|
||||
template <int Size>
|
||||
static void bench_path_fastalloc(benchmark::State& state) {
|
||||
for (auto _ : state) {
|
||||
void* p = FastAllocator<Size>::allocate();
|
||||
benchmark::DoNotOptimize(p);
|
||||
FastAllocator<Size>::release(p);
|
||||
}
|
||||
state.SetItemsProcessed(state.iterations());
|
||||
}
|
||||
|
||||
// Arena block create/destroy: exercises the flow/Arena.cpp hook. Arg(0) is the
|
||||
// user allocation size, which selects the block class (medium vs huge).
|
||||
static void bench_path_arena(benchmark::State& state) {
|
||||
int n = state.range(0);
|
||||
for (auto _ : state) {
|
||||
Arena a;
|
||||
uint8_t* p = new (a) uint8_t[n];
|
||||
benchmark::DoNotOptimize(p);
|
||||
}
|
||||
state.SetItemsProcessed(state.iterations());
|
||||
}
|
||||
|
||||
BENCHMARK(bench_path_operator_new)->Arg(64)->Arg(96)->Arg(256)->Arg(100000);
|
||||
BENCHMARK_TEMPLATE(bench_path_fastalloc, 64);
|
||||
BENCHMARK_TEMPLATE(bench_path_fastalloc, 96);
|
||||
BENCHMARK_TEMPLATE(bench_path_fastalloc, 256);
|
||||
BENCHMARK(bench_path_arena)->Arg(600)->Arg(100000);
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
# Most bench stuff is in flow/bench. This is in fdbserver because it links with
|
||||
# code we specifically do not want in flow/ (to avoid pulling into clients).
|
||||
|
||||
include(FDBBenchmark)
|
||||
|
||||
fdb_find_sources(FDBSERVER_BENCH_SRCS)
|
||||
|
||||
# Compile fdbserver/GlobalNewDelete.cpp directly into this benchmark executable.
|
||||
# It defines the global operator new/delete overrides that route through the
|
||||
# memory tracker; as a strong definition in the executable link it wins over
|
||||
# libc++'s weak one and is active process-wide here, exactly as in fdbserver.
|
||||
# Without it the bench would link libc++'s operator new and the operator-new
|
||||
# microbench would measure the unhooked allocator (its Arg(100)/Arg(1) rows
|
||||
# would equal Arg(0)). It only needs flow (the tracker hooks it calls), not the
|
||||
# rest of the fdbserver dependency graph.
|
||||
add_flow_target(EXECUTABLE NAME fdbserver_bench
|
||||
SRCS ${FDBSERVER_BENCH_SRCS}
|
||||
ADDL_SRCS "${CMAKE_CURRENT_SOURCE_DIR}/../GlobalNewDelete.cpp")
|
||||
|
||||
fdb_setup_googlebenchmark()
|
||||
|
||||
target_include_directories(
|
||||
fdbserver_bench
|
||||
PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}"
|
||||
"${CMAKE_CURRENT_SOURCE_DIR}/.."
|
||||
"${CMAKE_CURRENT_SOURCE_DIR}/include")
|
||||
|
||||
target_link_libraries(fdbserver_bench PRIVATE Threads::Threads fdb_google_benchmark flow)
|
||||
|
|
@ -77,6 +77,7 @@
|
|||
#include "flow/ProtocolVersion.h"
|
||||
#include "SimpleOpt/SimpleOpt.h"
|
||||
#include "flow/SystemMonitor.h"
|
||||
#include "flow/MemoryTracker.h"
|
||||
#include "flow/TLSConfig.h"
|
||||
#include "fdbclient/Tracing.h"
|
||||
#include "flow/WriteOnlySet.h"
|
||||
|
|
@ -705,54 +706,9 @@ static void printUsage(const char* name, bool devhelp) {
|
|||
|
||||
extern bool g_crashOnError;
|
||||
|
||||
#if defined(ALLOC_INSTRUMENTATION) || defined(ALLOC_INSTRUMENTATION_STDOUT)
|
||||
void* operator new(std::size_t size) {
|
||||
void* p = malloc(size);
|
||||
if (!p)
|
||||
throw std::bad_alloc();
|
||||
recordAllocation(p, size);
|
||||
return p;
|
||||
}
|
||||
void operator delete(void* ptr) throw() {
|
||||
recordDeallocation(ptr);
|
||||
free(ptr);
|
||||
}
|
||||
|
||||
// scalar, nothrow new and it matching delete
|
||||
void* operator new(std::size_t size, const std::nothrow_t&) throw() {
|
||||
void* p = malloc(size);
|
||||
recordAllocation(p, size);
|
||||
return p;
|
||||
}
|
||||
void operator delete(void* ptr, const std::nothrow_t&) throw() {
|
||||
recordDeallocation(ptr);
|
||||
free(ptr);
|
||||
}
|
||||
|
||||
// array throwing new and matching delete[]
|
||||
void* operator new[](std::size_t size) {
|
||||
void* p = malloc(size);
|
||||
if (!p)
|
||||
throw std::bad_alloc();
|
||||
recordAllocation(p, size);
|
||||
return p;
|
||||
}
|
||||
void operator delete[](void* ptr) throw() {
|
||||
recordDeallocation(ptr);
|
||||
free(ptr);
|
||||
}
|
||||
|
||||
// array, nothrow new and matching delete[]
|
||||
void* operator new[](std::size_t size, const std::nothrow_t&) throw() {
|
||||
void* p = malloc(size);
|
||||
recordAllocation(p, size);
|
||||
return p;
|
||||
}
|
||||
void operator delete[](void* ptr, const std::nothrow_t&) throw() {
|
||||
recordDeallocation(ptr);
|
||||
free(ptr);
|
||||
}
|
||||
#endif
|
||||
// The global operator new / operator delete replacements (both the legacy
|
||||
// ALLOC_INSTRUMENTATION accounting hooks and the sampled memory tracker) live in
|
||||
// fdbserver/GlobalNewDelete.cpp.
|
||||
|
||||
Optional<bool> checkBuggifyOverride(const char* testFile) {
|
||||
std::ifstream ifs;
|
||||
|
|
@ -1955,6 +1911,13 @@ int main(int argc, char* argv[]) {
|
|||
// Reinitialize knobs in order to update knobs that are dependent on explicitly set knobs
|
||||
initializeServerKnobs(Randomize::True, role == ServerRole::Simulation ? IsSimulated::True : IsSimulated::False);
|
||||
|
||||
// Knobs are now final; initialize the sampled memory tracker from them on
|
||||
// this (soon-to-be network) thread, before any serving role starts. Reading
|
||||
// the sample-inverse knob explicitly here — rather than inferring it from
|
||||
// the first allocation — keeps early startup allocations from latching the
|
||||
// tracker off before the knobs were configured. See design/memory-tracker.md.
|
||||
memTrackerInit();
|
||||
|
||||
// evictionPolicyStringToEnum will throw an exception if the string is not recognized as a valid
|
||||
EvictablePageCache::evictionPolicyStringToEnum(FLOW_KNOBS->CACHE_EVICTION_POLICY);
|
||||
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@ void forceLinkJsonWebKeySetTests();
|
|||
void forceLinkVersionVectorTests();
|
||||
void forceLinkRESTClientTests();
|
||||
void forceLinkRESTUtilsTests();
|
||||
void forceLinkMemoryTrackerTests();
|
||||
void forceLinkCompressedIntTests();
|
||||
void forceLinkAtomicTests();
|
||||
void forceLinkIdempotencyIdTests();
|
||||
|
|
@ -115,6 +116,7 @@ struct UnitTestWorkload : TestWorkload {
|
|||
forceLinkVersionVectorTests();
|
||||
forceLinkRESTClientTests();
|
||||
forceLinkRESTUtilsTests();
|
||||
forceLinkMemoryTrackerTests();
|
||||
forceLinkCompressedIntTests();
|
||||
forceLinkAtomicTests();
|
||||
forceLinkIdempotencyIdTests();
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@
|
|||
*/
|
||||
|
||||
#include "flow/Arena.h"
|
||||
#include "flow/MemoryTracker.h"
|
||||
#include "flow/ScopeExit.h"
|
||||
#include "flow/SimpleCounter.h"
|
||||
#include "flow/UnitTest.h"
|
||||
|
|
@ -460,36 +461,55 @@ ArenaBlock* ArenaBlock::create(int dataSize, Reference<ArenaBlock>& next) {
|
|||
b = (ArenaBlock*)FastAllocator<256>::allocate();
|
||||
b->bigSize = 256;
|
||||
INSTRUMENT_ALLOCATE("Arena256");
|
||||
} else if (reqSize <= 512) {
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(512);
|
||||
b->bigSize = 512;
|
||||
INSTRUMENT_ALLOCATE("Arena512");
|
||||
} else if (reqSize <= 1024) {
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(1024);
|
||||
b->bigSize = 1024;
|
||||
INSTRUMENT_ALLOCATE("Arena1024");
|
||||
} else if (reqSize <= 2048) {
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(2048);
|
||||
b->bigSize = 2048;
|
||||
INSTRUMENT_ALLOCATE("Arena2048");
|
||||
} else if (reqSize <= 4096) {
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(4096);
|
||||
b->bigSize = 4096;
|
||||
INSTRUMENT_ALLOCATE("Arena4096");
|
||||
} else {
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(8192);
|
||||
b->bigSize = 8192;
|
||||
INSTRUMENT_ALLOCATE("Arena8192");
|
||||
// Suppress the operator-new[] memory-tracker hook around the
|
||||
// underlying `new uint8_t[]`; the explicit memTrackerOnAlloc
|
||||
// below is the sole hook for these blocks. Without this
|
||||
// guard the same pointer would be tracked twice under two
|
||||
// different fingerprints.
|
||||
MemTrackerSuppress _suppress;
|
||||
if (reqSize <= 512) {
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(512);
|
||||
b->bigSize = 512;
|
||||
INSTRUMENT_ALLOCATE("Arena512");
|
||||
} else if (reqSize <= 1024) {
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(1024);
|
||||
b->bigSize = 1024;
|
||||
INSTRUMENT_ALLOCATE("Arena1024");
|
||||
} else if (reqSize <= 2048) {
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(2048);
|
||||
b->bigSize = 2048;
|
||||
INSTRUMENT_ALLOCATE("Arena2048");
|
||||
} else if (reqSize <= 4096) {
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(4096);
|
||||
b->bigSize = 4096;
|
||||
INSTRUMENT_ALLOCATE("Arena4096");
|
||||
} else {
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(8192);
|
||||
b->bigSize = 8192;
|
||||
INSTRUMENT_ALLOCATE("Arena8192");
|
||||
}
|
||||
}
|
||||
b->totalSizeEstimate = b->bigSize;
|
||||
b->tinySize = b->tinyUsed = NOT_TINY;
|
||||
b->bigUsed = sizeof(ArenaBlock);
|
||||
b->secure = 0;
|
||||
// Block-level attribution for >256 sizes (sizes <=256 use FastAllocator,
|
||||
// which fires its own memTrackerOnAlloc hook).
|
||||
if (b->bigSize > 256) {
|
||||
memTrackerOnAlloc(b, b->bigSize);
|
||||
}
|
||||
} else {
|
||||
#ifdef ALLOC_INSTRUMENTATION
|
||||
allocInstr["ArenaHugeKB"].alloc((reqSize + 1023) >> 10);
|
||||
#endif
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(reqSize);
|
||||
{
|
||||
// Suppress the operator-new[] hook so the explicit
|
||||
// memTrackerOnAlloc below is the sole tracker for huge
|
||||
// arena blocks (see comment in the small-block branch).
|
||||
MemTrackerSuppress _suppress;
|
||||
b = (ArenaBlock*)allocateAndMaybeKeepalive(reqSize);
|
||||
}
|
||||
b->tinySize = b->tinyUsed = NOT_TINY;
|
||||
b->bigSize = reqSize;
|
||||
b->totalSizeEstimate = b->bigSize;
|
||||
|
|
@ -505,6 +525,9 @@ ArenaBlock* ArenaBlock::create(int dataSize, Reference<ArenaBlock>& next) {
|
|||
}
|
||||
#endif
|
||||
g_hugeArenaMemory.fetch_add(reqSize);
|
||||
// Block-level attribution for huge arena blocks. allocateAndMaybeKeepalive
|
||||
// bypasses FastAllocator, so this is the only hook for these blocks.
|
||||
memTrackerOnAlloc(b, reqSize);
|
||||
|
||||
// If the new block has less free space than the old block, make the old block depend on it
|
||||
if (next && !next->isTiny() && next->unused() >= reqSize - dataSize) {
|
||||
|
|
@ -585,26 +608,50 @@ void ArenaBlock::destroyLeaf() {
|
|||
FastAllocator<256>::release(this);
|
||||
INSTRUMENT_RELEASE("Arena256");
|
||||
} else if (bigSize <= 512) {
|
||||
freeOrMaybeKeepalive(this);
|
||||
memTrackerOnFree(this);
|
||||
{
|
||||
MemTrackerSuppress _suppress;
|
||||
freeOrMaybeKeepalive(this);
|
||||
}
|
||||
INSTRUMENT_RELEASE("Arena512");
|
||||
} else if (bigSize <= 1024) {
|
||||
freeOrMaybeKeepalive(this);
|
||||
memTrackerOnFree(this);
|
||||
{
|
||||
MemTrackerSuppress _suppress;
|
||||
freeOrMaybeKeepalive(this);
|
||||
}
|
||||
INSTRUMENT_RELEASE("Arena1024");
|
||||
} else if (bigSize <= 2048) {
|
||||
freeOrMaybeKeepalive(this);
|
||||
memTrackerOnFree(this);
|
||||
{
|
||||
MemTrackerSuppress _suppress;
|
||||
freeOrMaybeKeepalive(this);
|
||||
}
|
||||
INSTRUMENT_RELEASE("Arena2048");
|
||||
} else if (bigSize <= 4096) {
|
||||
freeOrMaybeKeepalive(this);
|
||||
memTrackerOnFree(this);
|
||||
{
|
||||
MemTrackerSuppress _suppress;
|
||||
freeOrMaybeKeepalive(this);
|
||||
}
|
||||
INSTRUMENT_RELEASE("Arena4096");
|
||||
} else if (bigSize <= 8192) {
|
||||
freeOrMaybeKeepalive(this);
|
||||
memTrackerOnFree(this);
|
||||
{
|
||||
MemTrackerSuppress _suppress;
|
||||
freeOrMaybeKeepalive(this);
|
||||
}
|
||||
INSTRUMENT_RELEASE("Arena8192");
|
||||
} else {
|
||||
#ifdef ALLOC_INSTRUMENTATION
|
||||
allocInstr["ArenaHugeKB"].dealloc((bigSize + 1023) >> 10);
|
||||
#endif
|
||||
g_hugeArenaMemory.fetch_sub(bigSize);
|
||||
freeOrMaybeKeepalive(this);
|
||||
memTrackerOnFree(this);
|
||||
{
|
||||
MemTrackerSuppress _suppress;
|
||||
freeOrMaybeKeepalive(this);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@
|
|||
|
||||
#include "flow/FastAlloc.h"
|
||||
|
||||
#include "flow/MemoryTracker.h"
|
||||
#include "flow/ThreadPrimitives.h"
|
||||
#include "flow/Trace.h"
|
||||
#include "flow/Error.h"
|
||||
|
|
@ -432,6 +433,7 @@ void* FastAllocator<Size>::allocate() {
|
|||
#if defined(ALLOC_INSTRUMENTATION) || defined(ALLOC_INSTRUMENTATION_STDOUT)
|
||||
recordAllocation(p, Size);
|
||||
#endif
|
||||
memTrackerOnAlloc(p, Size);
|
||||
return p;
|
||||
}
|
||||
|
||||
|
|
@ -509,6 +511,7 @@ void FastAllocator<Size>::release(void* ptr) {
|
|||
#if defined(ALLOC_INSTRUMENTATION) || defined(ALLOC_INSTRUMENTATION_STDOUT)
|
||||
recordDeallocation(ptr);
|
||||
#endif
|
||||
memTrackerOnFree(ptr);
|
||||
}
|
||||
|
||||
template <int Size>
|
||||
|
|
|
|||
|
|
@ -86,6 +86,15 @@ void FlowKnobs::initialize(Randomize randomize, IsSimulated isSimulated) {
|
|||
|
||||
init( MEMORY_USAGE_CHECK_INTERVAL, 1.0 );
|
||||
|
||||
// Per-call-site sampled memory tracker. See design/memory-tracker.md.
|
||||
// Initial rollout: prod default off (=0). Simulation defaults to 1-in-10 sampling so the path is exercised.
|
||||
init( MEMORY_TRACKING_SAMPLE_INVERSE, 0 ); if( isSimulated ) MEMORY_TRACKING_SAMPLE_INVERSE = 10;
|
||||
init( MEMORY_TRACKING_FORCE_SAMPLE_BYTES, 100000 );
|
||||
init( MEMORY_TRACKING_LIVE_TRACKING, true );
|
||||
init( MEMORY_TRACKING_REPORT_INTERVAL, 600.0 ); if( isSimulated ) MEMORY_TRACKING_REPORT_INTERVAL = 30.0;
|
||||
init( MEMORY_TRACKING_REPORT_BYTES_THRESHOLD, 80000000 ); if( isSimulated ) MEMORY_TRACKING_REPORT_BYTES_THRESHOLD = 1000000;
|
||||
init( MEMORY_TRACKING_FRAMES, 6 );
|
||||
|
||||
// Chaos testing - enabled for simulation by default
|
||||
init( ENABLE_CHAOS_FEATURES, isSimulated );
|
||||
init( CHAOS_LOGGING_INTERVAL, 5.0 );
|
||||
|
|
|
|||
|
|
@ -0,0 +1,695 @@
|
|||
/*
|
||||
* MemoryTracker.cpp
|
||||
*
|
||||
* This source file is part of the FoundationDB open source project
|
||||
*
|
||||
* Copyright 2013-2026 Apple Inc. and the FoundationDB project authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
// Implementation of the sampled per-call-site memory tracker.
|
||||
// See design/memory-tracker.md and flow/include/flow/MemoryTracker.h.
|
||||
//
|
||||
// What this is for: finding memory LEAKS and untuned / oversized allocations
|
||||
// that drive RSS growth. FDB effectively never sees a real malloc / operator new
|
||||
// failure — a process is killed by fdbmonitor when its RSS crosses a configured
|
||||
// ceiling (typically ~12-16 GB against an ~8 GB target), i.e. "OOM" here is a
|
||||
// self-imposed RSS threshold, not an allocator failure. So the interesting range
|
||||
// is memory growth well SHORT of any allocation failure, and the tracker's
|
||||
// behaviour under an actual malloc/new failure is not a scenario we optimize for:
|
||||
// the hooks fail open (drop the sample; see memTrackerSampleAlloc). The sampled
|
||||
// path is nonetheless ordered so its table growth happens before any counter
|
||||
// update, so even that never-in-practice case leaves the accounting consistent.
|
||||
|
||||
#include "flow/MemoryTracker.h"
|
||||
|
||||
#include "flow/Knobs.h"
|
||||
#include "flow/Platform.h"
|
||||
#include "flow/ThreadPrimitives.h"
|
||||
#include "flow/Trace.h"
|
||||
#include "flow/flow.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cstdint>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <new>
|
||||
#include <unordered_map>
|
||||
#include <vector>
|
||||
|
||||
#ifdef __linux__
|
||||
#include <pthread.h>
|
||||
#endif
|
||||
|
||||
#if FDB_MEMORY_TRACKER
|
||||
|
||||
// Thread-local sampling state.
|
||||
// gMemTrackerCounter starts at 1 so the first allocation per thread is
|
||||
// sampled (and the slow path then reseeds from the knob).
|
||||
// gForceSampleBytes initialized to ~0 so force-sample never fires before we've
|
||||
// loaded the knob value at least once.
|
||||
thread_local bool gInMemTracker = false;
|
||||
thread_local int gMemTrackerCounter = 1;
|
||||
thread_local std::size_t gForceSampleBytes = static_cast<std::size_t>(-1);
|
||||
// Starts false so the first allocation on each thread reaches the slow path to
|
||||
// read the knob; set true there if sampling is off (see memTrackerSampleAlloc).
|
||||
thread_local bool gMemTrackerOff = false;
|
||||
|
||||
// Test-only one-shot: when set, the next sampled allocation throws to simulate a
|
||||
// tracker metadata-allocation failure (see memTrackerFailNextSampleForTest).
|
||||
static thread_local bool gFailNextSampleForTest = false;
|
||||
|
||||
// Definition of the cache-line-isolated enabled flag declared in the header.
|
||||
MemTrackerEnabledFlag g_memTrackerEnabled;
|
||||
// Same initial seed for every thread; cheap and adequate. Threads in
|
||||
// production start at different times and call into the slow path at
|
||||
// uncorrelated rates, so any phase correlation washes out within the
|
||||
// first handful of samples. If profiling ever shows correlated bursts
|
||||
// at startup, mix in a thread-id-derived value here.
|
||||
thread_local uint32_t gMemTrackerSeed = 0x9E3779B9u;
|
||||
|
||||
namespace {
|
||||
|
||||
// FNV-1a 64-bit over the captured frame array.
|
||||
uint64_t fnv64(const void* data, std::size_t len) {
|
||||
uint64_t h = 0xcbf29ce484222325ULL;
|
||||
const auto* p = static_cast<const std::uint8_t*>(data);
|
||||
for (std::size_t i = 0; i < len; i++) {
|
||||
h ^= p[i];
|
||||
h *= 0x100000001b3ULL;
|
||||
}
|
||||
return h;
|
||||
}
|
||||
|
||||
inline std::uint32_t xorshift32(std::uint32_t& s) {
|
||||
std::uint32_t x = s ? s : 0x9E3779B9u;
|
||||
x ^= x << 13;
|
||||
x ^= x >> 17;
|
||||
x ^= x << 5;
|
||||
s = x;
|
||||
return x;
|
||||
}
|
||||
|
||||
// Per-thread stack bounds, populated lazily on first use of captureFramesFP.
|
||||
// Used by captureFramesFP to terminate the FP walk when it crosses into
|
||||
// FP-elided code (notably glibc's pthread shutdown / TLS-destructor
|
||||
// machinery). Without this guard, the FP-elided frame leaves an
|
||||
// uninitialized saved-FP slot and the walk dereferences garbage. See
|
||||
// design/memory-tracker.md, "Side-thread safety".
|
||||
//
|
||||
// You've heard of optimistic concurrency control in database systems? This is
|
||||
// basically *optimistic segfault avoidance* to enable fast stack unwinding.
|
||||
// Caveat: the heuristics here aren't perfect. They seem pretty effective
|
||||
// so far.
|
||||
thread_local uintptr_t gStackLow = 0;
|
||||
thread_local uintptr_t gStackHigh = 0;
|
||||
|
||||
#ifdef __linux__
|
||||
|
||||
void initStackBoundsForThread() {
|
||||
pthread_attr_t attr;
|
||||
if (pthread_getattr_np(pthread_self(), &attr) == 0) {
|
||||
void* base = nullptr;
|
||||
size_t size = 0;
|
||||
if (pthread_attr_getstack(&attr, &base, &size) == 0) {
|
||||
gStackLow = reinterpret_cast<uintptr_t>(base);
|
||||
gStackHigh = gStackLow + size;
|
||||
}
|
||||
pthread_attr_destroy(&attr);
|
||||
}
|
||||
}
|
||||
|
||||
// Manual frame-pointer walk. Captures the return-address chain starting at
|
||||
// the caller of this function (and up). Relies on -fno-omit-frame-pointer.
|
||||
// Annotated noinline + no_instrument_function so the compiler can't fold the
|
||||
// frame chain in unexpected ways.
|
||||
//
|
||||
// Bounds the walk by the current thread's stack range so that crossing into
|
||||
// FP-elided code (which leaves the saved-FP slot uninitialized rather than
|
||||
// NULL) terminates cleanly instead of dereferencing garbage (see above).
|
||||
__attribute__((no_instrument_function, noinline)) int captureFramesFP(void** out, int max) {
|
||||
if (!gStackLow) {
|
||||
initStackBoundsForThread();
|
||||
}
|
||||
void** fp = static_cast<void**>(__builtin_frame_address(0));
|
||||
// Fallback for threads where pthread_getattr_np failed: ±8 MB around
|
||||
// the initial frame.
|
||||
// Caveat: this may need to be constrained more tightly to deal with
|
||||
// smaller stacks.
|
||||
uintptr_t lo = gStackLow ? gStackLow : reinterpret_cast<uintptr_t>(fp);
|
||||
uintptr_t hi = gStackHigh ? gStackHigh : reinterpret_cast<uintptr_t>(fp) + (8u << 20);
|
||||
int n = 0;
|
||||
while (fp && n < max) {
|
||||
uintptr_t a = reinterpret_cast<uintptr_t>(fp);
|
||||
// Reject out-of-stack or misaligned fp before dereferencing.
|
||||
if (a < lo || a + 16 > hi) {
|
||||
break;
|
||||
}
|
||||
if (a & (sizeof(void*) - 1)) {
|
||||
break;
|
||||
}
|
||||
void* ra = fp[1];
|
||||
if (!ra) {
|
||||
break;
|
||||
}
|
||||
out[n++] = ra;
|
||||
void** next = static_cast<void**>(fp[0]);
|
||||
// Sanity: stack grows down, so each next frame address must be larger.
|
||||
if (next <= fp) {
|
||||
break;
|
||||
}
|
||||
fp = next;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
#else // !__linux__
|
||||
|
||||
// NOTE: We (Apple) do not maintain a local facility to build FDB with
|
||||
// MSVC on Windows. The code in this file **may** have issues. We are
|
||||
// doing a best-effort attempt not to break the build. Support for
|
||||
// this memory tracking feature by community users of Windows would be
|
||||
// welcome.
|
||||
|
||||
// macOS / non-Linux: stack walking is unreliable here (system runtime
|
||||
// has -fomit-frame-pointer in places we can't avoid, and pthread_getattr_np
|
||||
// is Linux-specific). FDB is required to compile on macOS/Windows but is not run
|
||||
// in production there, so we just no-op the walker. The rest of the
|
||||
// tracker still compiles and runs; per-call-site reports will simply
|
||||
// lack stack attribution.
|
||||
force_noinline int captureFramesFP(void**, int) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
#endif // __linux__
|
||||
|
||||
// TODO: when memory tracking is enabled, regardless of the sampling
|
||||
// rate, every deallocation has to acquire this mutex. At O(1M)
|
||||
// frees/second this is noticeable overhead. This could be sped up by
|
||||
// sharding the mutex and the global state protected by the mutex
|
||||
// (the global state is the 2 maps and ~10 scalars defined below).
|
||||
// Presumably hash of the malloc buffer address itself could direct
|
||||
// the sharding. Consumers of the global state would have to merge
|
||||
// across shards but that is intended only to be the periodic log
|
||||
// reporter so making it do costly work is fine since it only runs
|
||||
// O(1/minute).
|
||||
|
||||
ThreadSpinLock g_mtLock;
|
||||
|
||||
struct LiveEntry {
|
||||
std::uint64_t fingerprint;
|
||||
std::uint64_t size;
|
||||
std::int64_t weight; // inverse inclusion probability at sample time (≈ SampleInverse, or 1 if
|
||||
// force-sampled); estimated contribution of this block is size * weight.
|
||||
// Stored so free debits the estimate by exactly what alloc credited, even
|
||||
// if the sampling knob changed in between.
|
||||
};
|
||||
|
||||
// Lazily-constructed maps. Allocated under the spinlock the first time we
|
||||
// reach the sampled path. Heap allocations from the maps' internals go
|
||||
// through our overridden operator new, which short-circuits (gInMemTracker
|
||||
// is true on the sampled path) and falls through to std::malloc — so map
|
||||
// growth never recurses into tracking.
|
||||
std::unordered_map<std::uint64_t, MemoryTrackerCallSite>* g_aggMap = nullptr;
|
||||
std::unordered_map<std::uintptr_t, LiveEntry>* g_liveMap = nullptr;
|
||||
|
||||
// Sampled-totals (i.e. across what we actually saw, not population estimates).
|
||||
std::int64_t g_liveBytesTotal = 0;
|
||||
std::int64_t g_liveBlocksTotal = 0;
|
||||
std::int64_t g_cumulativeBytesTotal = 0;
|
||||
std::int64_t g_cumulativeAllocsTotal = 0;
|
||||
std::int64_t g_samplesEmitted = 0;
|
||||
|
||||
// Estimated population totals (sampling correction applied; see LiveEntry::weight).
|
||||
std::int64_t g_estLiveBytesTotal = 0;
|
||||
std::int64_t g_estLiveBlocksTotal = 0;
|
||||
std::int64_t g_estCumulativeBytesTotal = 0;
|
||||
std::int64_t g_estCumulativeAllocsTotal = 0;
|
||||
|
||||
void ensureMaps() {
|
||||
if (!g_aggMap) {
|
||||
g_aggMap = new std::unordered_map<std::uint64_t, MemoryTrackerCallSite>();
|
||||
}
|
||||
if (!g_liveMap) {
|
||||
g_liveMap = new std::unordered_map<std::uintptr_t, LiveEntry>();
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// Publish the enabled flag and arm the calling thread from the current
|
||||
// MEMORY_TRACKING_SAMPLE_INVERSE. Shared by memTrackerInit (once, at startup)
|
||||
// and memTrackerResetForTest. Reading the knob explicitly here — rather than
|
||||
// inferring the enabled state from the first sampled allocation — is what keeps
|
||||
// an early main-thread allocation from latching the tracker off before the
|
||||
// knobs are configured.
|
||||
static void memTrackerArmFromKnobs() {
|
||||
int inverse = FLOW_KNOBS ? FLOW_KNOBS->MEMORY_TRACKING_SAMPLE_INVERSE : 0;
|
||||
bool enabled = (inverse > 0);
|
||||
g_memTrackerEnabled.value.store(enabled, std::memory_order_relaxed);
|
||||
// If enabled, clear this thread's off-latch and force its next allocation onto
|
||||
// the slow path (counter==1) to seed the reseed; if disabled, latch off so the
|
||||
// alloc hot path short-circuits on a single TLS load.
|
||||
gMemTrackerOff = !enabled;
|
||||
gMemTrackerCounter = 1;
|
||||
gForceSampleBytes = FLOW_KNOBS ? static_cast<std::size_t>(FLOW_KNOBS->MEMORY_TRACKING_FORCE_SAMPLE_BYTES)
|
||||
: static_cast<std::size_t>(-1);
|
||||
}
|
||||
|
||||
void memTrackerInit() {
|
||||
memTrackerArmFromKnobs();
|
||||
}
|
||||
|
||||
static void memTrackerSampleAllocImpl(void* p, std::size_t n) {
|
||||
if (gFailNextSampleForTest) {
|
||||
gFailNextSampleForTest = false;
|
||||
throw std::bad_alloc(); // simulate a tracker metadata-allocation failure (test only)
|
||||
}
|
||||
int inverse = 0;
|
||||
int frames = 6;
|
||||
bool liveTracking = true;
|
||||
if (FLOW_KNOBS) {
|
||||
inverse = FLOW_KNOBS->MEMORY_TRACKING_SAMPLE_INVERSE;
|
||||
frames = FLOW_KNOBS->MEMORY_TRACKING_FRAMES;
|
||||
liveTracking = FLOW_KNOBS->MEMORY_TRACKING_LIVE_TRACKING;
|
||||
gForceSampleBytes = static_cast<std::size_t>(FLOW_KNOBS->MEMORY_TRACKING_FORCE_SAMPLE_BYTES);
|
||||
}
|
||||
if (frames < 1) {
|
||||
frames = 1;
|
||||
}
|
||||
if (frames > MEMORY_TRACKER_MAX_FRAMES) {
|
||||
frames = MEMORY_TRACKER_MAX_FRAMES;
|
||||
}
|
||||
// Bound the reseed's `2 * inverse` arithmetic to int range for absurd knob
|
||||
// values; 1-in-256M sampling is already effectively off.
|
||||
if (inverse > (1 << 28)) {
|
||||
inverse = 1 << 28;
|
||||
}
|
||||
|
||||
if (inverse <= 0) {
|
||||
// Off (knob is startup-only). Flag this thread so the alloc hot path
|
||||
// short-circuits on one TLS load, without touching the counter again.
|
||||
gMemTrackerOff = true;
|
||||
return;
|
||||
}
|
||||
if (inverse == 1) {
|
||||
// Sample every allocation — keep counter at 1 so the next decrement
|
||||
// drops it to 0 and re-enters the slow path. Bypass the random
|
||||
// reseed below, which would otherwise leave counter==2 half the
|
||||
// time and cause us to miss every other allocation.
|
||||
gMemTrackerCounter = 1;
|
||||
} else if (gMemTrackerCounter <= 0) {
|
||||
// The random countdown actually expired: draw a fresh gap uniformly from
|
||||
// [1, 2*inverse-1], whose mean is exactly `inverse` — so the 1-in-inverse
|
||||
// sampling rate is unbiased and the integer `weight` below is exact.
|
||||
std::uint32_t r = xorshift32(gMemTrackerSeed);
|
||||
gMemTrackerCounter = 1 + static_cast<int>(r % static_cast<std::uint32_t>(2 * inverse - 1));
|
||||
}
|
||||
|
||||
bool isForceSampled = (n >= gForceSampleBytes);
|
||||
|
||||
// Weight = inverse inclusion probability of this sample, i.e. how many
|
||||
// allocations in the population it stands in for. A randomly-sampled block
|
||||
// (1-in-inverse) represents ~inverse allocations; a force-sampled block was
|
||||
// captured with certainty and represents only itself. The reseed draws
|
||||
// uniformly from [1, 2*inverse-1] (mean exactly `inverse`), so this integer
|
||||
// weight matches the true mean sampling gap with no bias.
|
||||
std::int64_t weight = (isForceSampled || inverse <= 1) ? 1 : inverse;
|
||||
|
||||
// Capture frames; skip the topmost two (this function and captureFramesFP
|
||||
// itself) so the recorded stack starts at the caller of memTrackerOnAlloc.
|
||||
// The strip count of 2 assumes memTrackerOnAlloc is inlined into its
|
||||
// caller (it's declared `inline` and the body is trivial). Production
|
||||
// builds run at -O3 and the inliner cooperates; at -O0 the inline hint
|
||||
// can be ignored and the recorded stack starts one frame too deep
|
||||
// (frame 0 = memTrackerOnAlloc body rather than the user's
|
||||
// allocation site). Acceptable: -O0 builds are not load-bearing for
|
||||
// memory attribution; the off-by-one is harmless for that workflow.
|
||||
void* tmp[MEMORY_TRACKER_MAX_FRAMES + 4];
|
||||
int captured = captureFramesFP(tmp, frames + 2);
|
||||
int kept = 0;
|
||||
void* keep[MEMORY_TRACKER_MAX_FRAMES];
|
||||
for (int i = 2; i < captured && kept < frames; i++) {
|
||||
keep[kept++] = tmp[i];
|
||||
}
|
||||
|
||||
std::uint64_t fp = (kept == 0) ? 0 : fnv64(keep, static_cast<std::size_t>(kept) * sizeof(void*));
|
||||
|
||||
ThreadSpinLockHolder lk(g_mtLock);
|
||||
ensureMaps();
|
||||
|
||||
auto nBytes = static_cast<std::int64_t>(n);
|
||||
auto estBytes = nBytes * weight;
|
||||
|
||||
// Do both node-allocating map operations up front, before mutating any
|
||||
// counter, so that if one throws (a bad_alloc while a table grows) the
|
||||
// exception unwinds with all per-site and global totals still consistent and
|
||||
// the fail-open catch in memTrackerSampleAlloc simply drops the sample. FDB
|
||||
// never sees a real malloc/new failure in practice (see the file header), so
|
||||
// this is cheap hygiene rather than a hot path.
|
||||
auto& site = (*g_aggMap)[fp]; // inserts a node for a new fingerprint; may throw
|
||||
if (site.fingerprint == 0 && site.cumulativeAllocs == 0) {
|
||||
site.fingerprint = fp;
|
||||
site.exemplarFrameCount = static_cast<std::uint8_t>(kept);
|
||||
for (int i = 0; i < kept; i++) {
|
||||
site.exemplarFrames[i] = keep[i];
|
||||
}
|
||||
}
|
||||
|
||||
// Reserve the live-block slot before crediting anything. A brand-new key
|
||||
// allocates a node here (may throw); an existing key — a stale entry whose
|
||||
// free was suppressed (e.g. during memTrackerDump or a memTrackerForEachSite
|
||||
// callback) so it was never debited — reuses its slot and cannot throw. We
|
||||
// capture the stale value so it can be debited below; otherwise its live
|
||||
// credit would leak once the address is reused and live totals would creep up.
|
||||
bool hadStale = false;
|
||||
LiveEntry stalePrev{};
|
||||
if (liveTracking) {
|
||||
auto key = reinterpret_cast<std::uintptr_t>(p);
|
||||
auto res = g_liveMap->try_emplace(key, LiveEntry{ fp, static_cast<std::uint64_t>(n), weight });
|
||||
if (!res.second) {
|
||||
hadStale = true;
|
||||
stalePrev = res.first->second;
|
||||
res.first->second = LiveEntry{ fp, static_cast<std::uint64_t>(n), weight };
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Nothing below allocates or throws; per-site and global totals move in lockstep. ----
|
||||
|
||||
if (hadStale) {
|
||||
auto oldBytes = static_cast<std::int64_t>(stalePrev.size);
|
||||
auto oldEst = oldBytes * stalePrev.weight;
|
||||
auto oldSite = g_aggMap->find(stalePrev.fingerprint);
|
||||
if (oldSite != g_aggMap->end()) {
|
||||
oldSite->second.liveBytes -= oldBytes;
|
||||
oldSite->second.liveCount -= 1;
|
||||
oldSite->second.estLiveBytes -= oldEst;
|
||||
oldSite->second.estLiveCount -= stalePrev.weight;
|
||||
}
|
||||
g_liveBytesTotal -= oldBytes;
|
||||
g_liveBlocksTotal -= 1;
|
||||
g_estLiveBytesTotal -= oldEst;
|
||||
g_estLiveBlocksTotal -= stalePrev.weight;
|
||||
}
|
||||
|
||||
site.cumulativeAllocs += 1;
|
||||
site.cumulativeBytes += nBytes;
|
||||
site.estCumulativeAllocs += weight;
|
||||
site.estCumulativeBytes += estBytes;
|
||||
if (isForceSampled) {
|
||||
site.forceSampledCount += 1;
|
||||
}
|
||||
|
||||
if (liveTracking) {
|
||||
site.liveBytes += nBytes;
|
||||
site.liveCount += 1;
|
||||
if (site.liveBytes > site.peakBytes) {
|
||||
site.peakBytes = site.liveBytes;
|
||||
}
|
||||
site.estLiveBytes += estBytes;
|
||||
site.estLiveCount += weight;
|
||||
if (site.estLiveBytes > site.estPeakBytes) {
|
||||
site.estPeakBytes = site.estLiveBytes;
|
||||
}
|
||||
g_liveBytesTotal += nBytes;
|
||||
g_liveBlocksTotal += 1;
|
||||
g_estLiveBytesTotal += estBytes;
|
||||
g_estLiveBlocksTotal += weight;
|
||||
}
|
||||
g_cumulativeBytesTotal += nBytes;
|
||||
g_cumulativeAllocsTotal += 1;
|
||||
g_estCumulativeBytesTotal += estBytes;
|
||||
g_estCumulativeAllocsTotal += weight;
|
||||
g_samplesEmitted += 1;
|
||||
}
|
||||
|
||||
void memTrackerSampleAlloc(void* p, std::size_t n) {
|
||||
// Fail open: the tracker is a diagnostic; a std::bad_alloc from its own map
|
||||
// growth (or the test injection) must never propagate into the caller's
|
||||
// allocation path, which has already handed out the underlying block.
|
||||
try {
|
||||
memTrackerSampleAllocImpl(p, n);
|
||||
} catch (...) {
|
||||
}
|
||||
}
|
||||
|
||||
static void memTrackerSampleFreeImpl(void* p) {
|
||||
bool liveTracking = FLOW_KNOBS ? FLOW_KNOBS->MEMORY_TRACKING_LIVE_TRACKING : true;
|
||||
if (!liveTracking) {
|
||||
return;
|
||||
}
|
||||
|
||||
ThreadSpinLockHolder lk(g_mtLock);
|
||||
if (!g_liveMap) {
|
||||
return;
|
||||
}
|
||||
auto it = g_liveMap->find(reinterpret_cast<std::uintptr_t>(p));
|
||||
if (it == g_liveMap->end()) {
|
||||
return;
|
||||
}
|
||||
|
||||
LiveEntry e = it->second;
|
||||
g_liveMap->erase(it);
|
||||
|
||||
auto eBytes = static_cast<std::int64_t>(e.size);
|
||||
auto eEstBytes = eBytes * e.weight;
|
||||
if (g_aggMap) {
|
||||
auto sit = g_aggMap->find(e.fingerprint);
|
||||
if (sit != g_aggMap->end()) {
|
||||
sit->second.liveBytes -= eBytes;
|
||||
sit->second.liveCount -= 1;
|
||||
sit->second.estLiveBytes -= eEstBytes;
|
||||
sit->second.estLiveCount -= e.weight;
|
||||
}
|
||||
}
|
||||
g_liveBytesTotal -= eBytes;
|
||||
g_liveBlocksTotal -= 1;
|
||||
g_estLiveBytesTotal -= eEstBytes;
|
||||
g_estLiveBlocksTotal -= e.weight;
|
||||
}
|
||||
|
||||
void memTrackerSampleFree(void* p) {
|
||||
// Fail open (see memTrackerSampleAlloc): operator delete is noexcept, so the
|
||||
// tracker must never let an exception escape the free path.
|
||||
try {
|
||||
memTrackerSampleFreeImpl(p);
|
||||
} catch (...) {
|
||||
}
|
||||
}
|
||||
|
||||
void memTrackerForEachSite(std::function<void(const MemoryTrackerCallSite&)> cb) {
|
||||
// Suppress for the entire call so callbacks that allocate (e.g.
|
||||
// fprintf or std::vector growth in test failure paths) don't
|
||||
// re-enter the tracker and pollute the agg map mid-iteration.
|
||||
MemTrackerSuppress _suppress;
|
||||
std::vector<MemoryTrackerCallSite> snapshot;
|
||||
{
|
||||
ThreadSpinLockHolder lk(g_mtLock);
|
||||
if (g_aggMap) {
|
||||
snapshot.reserve(g_aggMap->size());
|
||||
for (auto& kv : *g_aggMap) {
|
||||
snapshot.push_back(kv.second);
|
||||
}
|
||||
}
|
||||
}
|
||||
for (auto& s : snapshot) {
|
||||
cb(s);
|
||||
}
|
||||
}
|
||||
|
||||
void memTrackerResetForTest() {
|
||||
MemTrackerSuppress _suppress;
|
||||
{
|
||||
ThreadSpinLockHolder lk(g_mtLock);
|
||||
if (g_aggMap) {
|
||||
g_aggMap->clear();
|
||||
}
|
||||
if (g_liveMap) {
|
||||
g_liveMap->clear();
|
||||
}
|
||||
g_liveBytesTotal = 0;
|
||||
g_liveBlocksTotal = 0;
|
||||
g_cumulativeBytesTotal = 0;
|
||||
g_cumulativeAllocsTotal = 0;
|
||||
g_samplesEmitted = 0;
|
||||
g_estLiveBytesTotal = 0;
|
||||
g_estLiveBlocksTotal = 0;
|
||||
g_estCumulativeBytesTotal = 0;
|
||||
g_estCumulativeAllocsTotal = 0;
|
||||
}
|
||||
// Publish the enabled flag and arm this thread from the current knob value
|
||||
// (a test typically sets MEMORY_TRACKING_SAMPLE_INVERSE via KnobOverride just
|
||||
// before calling this), mirroring memTrackerInit at process startup.
|
||||
gFailNextSampleForTest = false;
|
||||
memTrackerArmFromKnobs();
|
||||
}
|
||||
|
||||
void memTrackerFailNextSampleForTest() {
|
||||
gFailNextSampleForTest = true;
|
||||
}
|
||||
|
||||
static void memTrackerDumpImpl(int64_t bytesThreshold) {
|
||||
MemTrackerSuppress _suppress;
|
||||
|
||||
std::vector<MemoryTrackerCallSite> sites;
|
||||
int aggSize = 0;
|
||||
int liveSize = 0;
|
||||
std::int64_t liveBytesTotalSnap = 0;
|
||||
std::int64_t liveBlocksTotalSnap = 0;
|
||||
std::int64_t cumBytesSnap = 0;
|
||||
std::int64_t cumAllocsSnap = 0;
|
||||
std::int64_t samplesEmittedSnap = 0;
|
||||
std::int64_t estLiveBytesTotalSnap = 0;
|
||||
std::int64_t estLiveBlocksTotalSnap = 0;
|
||||
std::int64_t estCumBytesSnap = 0;
|
||||
std::int64_t estCumAllocsSnap = 0;
|
||||
{
|
||||
ThreadSpinLockHolder lk(g_mtLock);
|
||||
if (g_aggMap) {
|
||||
sites.reserve(g_aggMap->size());
|
||||
for (auto& kv : *g_aggMap) {
|
||||
sites.push_back(kv.second);
|
||||
}
|
||||
aggSize = static_cast<int>(g_aggMap->size());
|
||||
}
|
||||
liveSize = g_liveMap ? static_cast<int>(g_liveMap->size()) : 0;
|
||||
liveBytesTotalSnap = g_liveBytesTotal;
|
||||
liveBlocksTotalSnap = g_liveBlocksTotal;
|
||||
cumBytesSnap = g_cumulativeBytesTotal;
|
||||
cumAllocsSnap = g_cumulativeAllocsTotal;
|
||||
samplesEmittedSnap = g_samplesEmitted;
|
||||
estLiveBytesTotalSnap = g_estLiveBytesTotal;
|
||||
estLiveBlocksTotalSnap = g_estLiveBlocksTotal;
|
||||
estCumBytesSnap = g_estCumulativeBytesTotal;
|
||||
estCumAllocsSnap = g_estCumulativeAllocsTotal;
|
||||
}
|
||||
|
||||
bool liveTracking = FLOW_KNOBS ? FLOW_KNOBS->MEMORY_TRACKING_LIVE_TRACKING : true;
|
||||
// Rank and threshold on the *estimated* usage, since that is the real
|
||||
// per-site cost the report is about; the threshold knob is expressed in
|
||||
// real bytes (~1% of target RSS), not sampled bytes.
|
||||
// std::sort is unstable and unordered_map iteration is bucket-order, so
|
||||
// MemoryTrackerSite events for sites with tied byte values may appear in
|
||||
// different orders across same-seed sim2 runs. The R5 determinism
|
||||
// requirement is on aggregate counts, not event ordering — those are
|
||||
// unaffected — so we don't pay for stable_sort here.
|
||||
auto byLive = [](const MemoryTrackerCallSite& a, const MemoryTrackerCallSite& b) {
|
||||
return a.estLiveBytes > b.estLiveBytes;
|
||||
};
|
||||
auto byCum = [](const MemoryTrackerCallSite& a, const MemoryTrackerCallSite& b) {
|
||||
return a.estCumulativeBytes > b.estCumulativeBytes;
|
||||
};
|
||||
if (liveTracking) {
|
||||
std::sort(sites.begin(), sites.end(), byLive);
|
||||
} else {
|
||||
std::sort(sites.begin(), sites.end(), byCum);
|
||||
}
|
||||
|
||||
// Filter: a site qualifies when its estimated currently-live bytes (or
|
||||
// estimated cumulative bytes in degraded mode) exceed the threshold. Sites
|
||||
// are already sorted descending, so we can stop at the first non-qualifier.
|
||||
std::vector<MemoryTrackerCallSite> qualifying;
|
||||
qualifying.reserve(sites.size());
|
||||
for (const auto& s : sites) {
|
||||
int64_t v = liveTracking ? s.estLiveBytes : s.estCumulativeBytes;
|
||||
if (v < bytesThreshold) {
|
||||
break;
|
||||
}
|
||||
qualifying.push_back(s);
|
||||
}
|
||||
|
||||
// Build addr2line prefix once per dump. Built directly here rather
|
||||
// than via platform::format_backtrace, which deliberately drops index
|
||||
// 0 of its input (its single-site use case treats that as the helper's
|
||||
// caller); we want every captured frame including the leaf.
|
||||
std::string addrCmdPrefix;
|
||||
uintptr_t pieOffset = 0;
|
||||
if (!qualifying.empty()) {
|
||||
platform::ImageInfo img = platform::getImageInfo();
|
||||
#ifdef __clang__
|
||||
const char* addr2lineTool = "/usr/local/bin/llvm-addr2line";
|
||||
#else
|
||||
const char* addr2lineTool = "/usr/bin/addr2line";
|
||||
#endif
|
||||
addrCmdPrefix = format("%s -e %s -p -C -f -i", addr2lineTool, img.symbolFileName.c_str());
|
||||
pieOffset = reinterpret_cast<uintptr_t>(img.offset);
|
||||
}
|
||||
|
||||
for (const auto& s : qualifying) {
|
||||
std::string addrCmd = addrCmdPrefix;
|
||||
for (int i = 0; i < s.exemplarFrameCount; i++) {
|
||||
uintptr_t pieRelative = reinterpret_cast<uintptr_t>(s.exemplarFrames[i]) - pieOffset;
|
||||
addrCmd += format(" 0x%lx", pieRelative);
|
||||
}
|
||||
TraceEvent("MemoryTrackerSite")
|
||||
.detail("Fingerprint", format("%016llx", static_cast<unsigned long long>(s.fingerprint)))
|
||||
.detail("EstLiveBytes", s.estLiveBytes)
|
||||
.detail("EstLiveCount", s.estLiveCount)
|
||||
.detail("EstPeakBytes", s.estPeakBytes)
|
||||
.detail("EstCumulativeBytes", s.estCumulativeBytes)
|
||||
.detail("EstCumulativeAllocs", s.estCumulativeAllocs)
|
||||
.detail("LiveBytes", s.liveBytes)
|
||||
.detail("LiveCount", s.liveCount)
|
||||
.detail("PeakBytes", s.peakBytes)
|
||||
.detail("CumulativeBytes", s.cumulativeBytes)
|
||||
.detail("CumulativeAllocs", s.cumulativeAllocs)
|
||||
.detail("ForceSampledCount", s.forceSampledCount)
|
||||
.detail("AddrCmd", addrCmd);
|
||||
}
|
||||
|
||||
TraceEvent("MemoryTrackerSummary")
|
||||
.detail("SitesTracked", aggSize)
|
||||
.detail("SitesReported", static_cast<int>(qualifying.size()))
|
||||
.detail("EstLiveBytesTotal", estLiveBytesTotalSnap)
|
||||
.detail("EstLiveBlocksTotal", estLiveBlocksTotalSnap)
|
||||
.detail("EstCumulativeBytes", estCumBytesSnap)
|
||||
.detail("EstCumulativeAllocs", estCumAllocsSnap)
|
||||
.detail("LiveBlocks", liveSize)
|
||||
.detail("LiveBytesTotal", liveBytesTotalSnap)
|
||||
.detail("LiveBlocksTotal", liveBlocksTotalSnap)
|
||||
.detail("CumulativeAllocs", cumAllocsSnap)
|
||||
.detail("CumulativeBytes", cumBytesSnap)
|
||||
.detail("SamplesEmitted", samplesEmittedSnap)
|
||||
.detail("SampleInverse", FLOW_KNOBS ? FLOW_KNOBS->MEMORY_TRACKING_SAMPLE_INVERSE : 0)
|
||||
.detail("ForceSampleBytes",
|
||||
FLOW_KNOBS ? FLOW_KNOBS->MEMORY_TRACKING_FORCE_SAMPLE_BYTES : static_cast<std::int64_t>(-1))
|
||||
.detail("ReportBytesThreshold", bytesThreshold)
|
||||
// Caveat: Est* values are statistical estimates. Each randomly-sampled block is
|
||||
// scaled by SampleInverse; force-sampled blocks (>= ForceSampleBytes) count once.
|
||||
// Accuracy improves with SamplesEmitted; a site with few samples is noisy.
|
||||
.detail("EstimateBasis", "Est*=sampled*SampleInverse; force-sampled weight 1; statistical estimate");
|
||||
}
|
||||
|
||||
void memTrackerDump(int64_t bytesThreshold) {
|
||||
// Disabled: nothing is sampled, so skip the dump entirely rather than emit an
|
||||
// empty MemoryTrackerSummary every report interval (the production default is
|
||||
// off, and SystemMonitor calls this on a fixed cadence regardless).
|
||||
if (!g_memTrackerEnabled.value.load(std::memory_order_relaxed)) {
|
||||
return;
|
||||
}
|
||||
// Fail open (see memTrackerSampleAlloc): a diagnostic dump must never crash the
|
||||
// server, e.g. on an allocation failure while building the report.
|
||||
try {
|
||||
memTrackerDumpImpl(bytesThreshold);
|
||||
} catch (...) {
|
||||
}
|
||||
}
|
||||
|
||||
// The global operator new / operator delete replacements that route through
|
||||
// memTrackerOnAlloc/OnFree live in fdbserver/GlobalNewDelete.cpp, not here, so
|
||||
// the interposition is confined to the fdbserver executable and never ships in
|
||||
// libfdb_c / client bindings. This TU provides only the tracker machinery those
|
||||
// overrides (and the FastAllocator / ArenaBlock hooks) call into.
|
||||
|
||||
#endif // FDB_MEMORY_TRACKER
|
||||
|
|
@ -22,6 +22,8 @@
|
|||
#include "flow/Platform.h"
|
||||
#include "flow/TDMetric.h"
|
||||
#include "flow/SystemMonitor.h"
|
||||
#include "flow/Knobs.h"
|
||||
#include "flow/MemoryTracker.h"
|
||||
|
||||
#if defined(ALLOC_INSTRUMENTATION) && defined(__linux__)
|
||||
#include <cxxabi.h>
|
||||
|
|
@ -489,6 +491,20 @@ SystemStatistics customSystemMonitor(std::string const& eventName, StatisticsSta
|
|||
#endif
|
||||
statState->networkMetricsState = g_network->networkInfo.metrics;
|
||||
statState->networkState = netData;
|
||||
|
||||
// Periodic dump of the per-call-site memory tracker; cadence from the
|
||||
// MEMORY_TRACKING_REPORT_INTERVAL knob (<=0 disables). In simulation the
|
||||
// tracker's tables and this static are shared across all simulated
|
||||
// processes, so one dump fires per interval cluster-wide and its site
|
||||
// numbers blend every process — correct only in a real single-process server.
|
||||
if (FLOW_KNOBS && FLOW_KNOBS->MEMORY_TRACKING_REPORT_INTERVAL > 0) {
|
||||
static double lastMemTrackerDump = 0;
|
||||
if (now() - lastMemTrackerDump >= FLOW_KNOBS->MEMORY_TRACKING_REPORT_INTERVAL) {
|
||||
memTrackerDump(FLOW_KNOBS->MEMORY_TRACKING_REPORT_BYTES_THRESHOLD);
|
||||
lastMemTrackerDump = now();
|
||||
}
|
||||
}
|
||||
|
||||
return currentStats;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -127,6 +127,17 @@ public:
|
|||
|
||||
double MEMORY_USAGE_CHECK_INTERVAL;
|
||||
|
||||
// Per-call-site sampled memory tracker. See design/memory-tracker.md and flow/MemoryTracker.h.
|
||||
// All of these are startup-only: they are read once and not meant to change at runtime
|
||||
// (dynamic enable/disable is a Non-requirement -- edit the config and restart).
|
||||
int MEMORY_TRACKING_SAMPLE_INVERSE; // 0=off, N=1-in-N
|
||||
int64_t MEMORY_TRACKING_FORCE_SAMPLE_BYTES; // always sample allocations >= this many bytes; -1 disables
|
||||
bool MEMORY_TRACKING_LIVE_TRACKING; // when false, skip the pointer-keyed live-block table
|
||||
double MEMORY_TRACKING_REPORT_INTERVAL; // seconds between dumps; 0 disables reporting
|
||||
int64_t MEMORY_TRACKING_REPORT_BYTES_THRESHOLD; // sites with live bytes >= this are reported each dump (~1% of an 8
|
||||
// GB target RSS)
|
||||
int MEMORY_TRACKING_FRAMES; // captured stack depth (1..MEMORY_TRACKER_MAX_FRAMES)
|
||||
|
||||
// Chaos testing
|
||||
bool ENABLE_CHAOS_FEATURES;
|
||||
double CHAOS_LOGGING_INTERVAL;
|
||||
|
|
|
|||
|
|
@ -0,0 +1,223 @@
|
|||
/*
|
||||
* MemoryTracker.h
|
||||
*
|
||||
* This source file is part of the FoundationDB open source project
|
||||
*
|
||||
* Copyright 2013-2026 Apple Inc. and the FoundationDB project authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
// Sampled per-call-site memory attribution.
|
||||
//
|
||||
// See design/memory-tracker.md for the full design.
|
||||
//
|
||||
// Hot path: memTrackerOnAlloc is header-inlined; when the feature is disabled
|
||||
// (the common production default) it short-circuits on a single per-thread TLS
|
||||
// load + branch. memTrackerOnFree is header-inlined, one relaxed read of a
|
||||
// cache-line-isolated enabled flag + one branch when disabled -- no lock, no
|
||||
// table probe.
|
||||
//
|
||||
// Sampled path delegates to memTrackerSampleAlloc / memTrackerSampleFree,
|
||||
// which take a private spinlock, capture a frame-pointer-walk backtrace, and
|
||||
// update two tables (aggregation by fingerprint, and an optional pointer-
|
||||
// keyed live-block table).
|
||||
//
|
||||
// Reentrancy: the gInMemTracker thread-local guard is set to true while the
|
||||
// tracker is doing its own work. Any allocator hook called recursively
|
||||
// during that window observes the guard and bails out, leaving the
|
||||
// underlying allocation un-tracked. Higher-level hooks (e.g. ArenaBlock::create)
|
||||
// may also set this guard to suppress an inner allocator hook so the same
|
||||
// block is attributed at exactly one level.
|
||||
|
||||
#ifndef FLOW_MEMORY_TRACKER_H
|
||||
#define FLOW_MEMORY_TRACKER_H
|
||||
#pragma once
|
||||
|
||||
#include <atomic>
|
||||
#include <cstddef>
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
|
||||
// FDB_MEMORY_TRACKER gates the whole feature at compile time; on (1) by default.
|
||||
// Build with -DFDB_MEMORY_TRACKER=0 (cmake -DFDB_MEMORY_TRACKER=OFF) to compile it
|
||||
// out entirely: the hooks become no-ops and the global operator new/delete override
|
||||
// (fdbserver/GlobalNewDelete.cpp) is not defined, so libc++'s allocator is used.
|
||||
#ifndef FDB_MEMORY_TRACKER
|
||||
#define FDB_MEMORY_TRACKER 1
|
||||
#endif
|
||||
|
||||
#if FDB_MEMORY_TRACKER
|
||||
|
||||
// Maximum number of stack frames the tracker can capture per sample.
|
||||
// MEMORY_TRACKING_FRAMES knob controls the runtime depth (1..MEMORY_TRACKER_MAX_FRAMES).
|
||||
constexpr int MEMORY_TRACKER_MAX_FRAMES = 10;
|
||||
|
||||
// Per-site aggregate, exposed for tests via memTrackerForEachSite.
|
||||
//
|
||||
// Two families of numbers are kept per site:
|
||||
// * Est* — the estimated *population* usage, i.e. what the site is really
|
||||
// costing. Each sampled block is weighted by its inverse inclusion
|
||||
// probability (≈ SampleInverse for randomly-sampled blocks, 1 for
|
||||
// force-sampled blocks) at sample time, so these already have the sampling
|
||||
// math applied — a consumer (logging, etc) reads them directly, no scaling required.
|
||||
// * the raw sampled counters (liveBytes, cumulativeAllocs, …) — the
|
||||
// uninterpreted "what we actually observed" numbers, kept for auditing the
|
||||
// estimate and gauging its confidence (few samples ⇒ noisy estimate).
|
||||
struct MemoryTrackerCallSite {
|
||||
uint64_t fingerprint;
|
||||
|
||||
int64_t estLiveBytes;
|
||||
int64_t estLiveCount;
|
||||
int64_t estPeakBytes;
|
||||
int64_t estCumulativeBytes;
|
||||
int64_t estCumulativeAllocs;
|
||||
|
||||
int64_t liveBytes;
|
||||
int64_t liveCount;
|
||||
int64_t peakBytes;
|
||||
int64_t cumulativeAllocs;
|
||||
int64_t cumulativeBytes;
|
||||
int64_t forceSampledCount;
|
||||
|
||||
void* exemplarFrames[MEMORY_TRACKER_MAX_FRAMES];
|
||||
uint8_t exemplarFrameCount;
|
||||
};
|
||||
|
||||
extern thread_local bool gInMemTracker;
|
||||
extern thread_local int gMemTrackerCounter;
|
||||
extern thread_local std::size_t gForceSampleBytes;
|
||||
// Set true (per thread) once this thread's slow path observes sampling is off,
|
||||
// so the alloc hot path then short-circuits on a single TLS load instead of
|
||||
// decrementing the counter and reading gForceSampleBytes every call. Per-thread
|
||||
// (not global) so an early main-thread allocation before FLOW_KNOBS is ready
|
||||
// can't disable sampling on worker threads that bootstrap later. Cleared by
|
||||
// memTrackerResetForTest.
|
||||
extern thread_local bool gMemTrackerOff;
|
||||
|
||||
// Global "is the tracker enabled" flag, kept in its own cache line. Published
|
||||
// once, from the first slow-path visit, and thereafter constant: the sample-
|
||||
// inverse knob is read at startup only (dynamic enable/disable is a
|
||||
// Non-requirement -- see design/memory-tracker.md). The flag stays in MESI
|
||||
// shared state across cores, so the free hot path's relaxed read is cached:
|
||||
// when disabled, a free is one read + one branch, no lock. This is the
|
||||
// free-path off switch (a free has no per-thread sampling counter to gate on,
|
||||
// unlike an alloc).
|
||||
//
|
||||
// Relaxed ordering is sufficient: a free of a sampled pointer is always
|
||||
// preceded (via the pointer handoff that let the freeing thread learn the
|
||||
// pointer at all) by the sampling alloc that inserted it, and that alloc set
|
||||
// this flag true before inserting -- so the happens-before edge guarantees the
|
||||
// freeing thread observes the flag as true.
|
||||
struct alignas(64) MemTrackerEnabledFlag {
|
||||
std::atomic<bool> value{ false };
|
||||
char pad[64 - sizeof(std::atomic<bool>)];
|
||||
};
|
||||
extern MemTrackerEnabledFlag g_memTrackerEnabled;
|
||||
|
||||
// RAII suppressor: while alive, allocator hooks short-circuit. Used by code
|
||||
// paths that call into a lower-level allocator (e.g. ArenaBlock wrapping
|
||||
// `new uint8_t[]`) and want their explicit memTrackerOnAlloc/OnFree call to
|
||||
// be the sole tracker for the block — without this guard the inner
|
||||
// allocator's hook fires too and the same pointer is double-tracked under
|
||||
// two different fingerprints. Nest-safe: saves and restores prev.
|
||||
class MemTrackerSuppress {
|
||||
bool prev;
|
||||
|
||||
public:
|
||||
MemTrackerSuppress() : prev(gInMemTracker) { gInMemTracker = true; }
|
||||
~MemTrackerSuppress() { gInMemTracker = prev; }
|
||||
MemTrackerSuppress(const MemTrackerSuppress&) = delete;
|
||||
MemTrackerSuppress& operator=(const MemTrackerSuppress&) = delete;
|
||||
};
|
||||
|
||||
// Initialize the tracker from the current knob values. Call once, from process
|
||||
// startup, AFTER all knobs are finalized and BEFORE any serving role starts (see
|
||||
// fdbserver.cpp). Publishes the enabled state and arms the calling (network)
|
||||
// thread from MEMORY_TRACKING_SAMPLE_INVERSE, so early startup allocations on the
|
||||
// main thread cannot latch the tracker off before the knob is configured. The
|
||||
// sample-inverse knob is read here (and in memTrackerResetForTest) rather than
|
||||
// inferred from the first allocation; dynamic runtime enable/disable is a
|
||||
// Non-requirement (see design/memory-tracker.md).
|
||||
void memTrackerInit();
|
||||
|
||||
void memTrackerSampleAlloc(void* p, std::size_t n);
|
||||
void memTrackerSampleFree(void* p);
|
||||
|
||||
inline void memTrackerOnAlloc(void* p, std::size_t n) {
|
||||
if (gMemTrackerOff) [[likely]] {
|
||||
return;
|
||||
}
|
||||
if (gInMemTracker || !p) {
|
||||
return;
|
||||
}
|
||||
if (--gMemTrackerCounter > 0 && n < gForceSampleBytes) {
|
||||
return;
|
||||
}
|
||||
MemTrackerSuppress _suppress;
|
||||
memTrackerSampleAlloc(p, n);
|
||||
}
|
||||
|
||||
inline void memTrackerOnFree(void* p) {
|
||||
if (gInMemTracker || !p) {
|
||||
return;
|
||||
}
|
||||
// Cheap cache-line-shared read: when the tracker is disabled there is no
|
||||
// live-block table to debit, so skip all lock/table work.
|
||||
if (!g_memTrackerEnabled.value.load(std::memory_order_relaxed)) {
|
||||
return;
|
||||
}
|
||||
MemTrackerSuppress _suppress;
|
||||
memTrackerSampleFree(p);
|
||||
}
|
||||
|
||||
// Periodic dump — emits one TraceEvent("MemoryTrackerSite") per site whose
|
||||
// estLiveBytes (or estCumulativeBytes when MEMORY_TRACKING_LIVE_TRACKING is off)
|
||||
// exceeds bytesThreshold. The threshold is compared against the sampling-corrected
|
||||
// estimate, not the raw sampled bytes. Each site event carries an "AddrCmd"
|
||||
// detail: a ready-to-paste addr2line invocation covering just that site's frames.
|
||||
// A final TraceEvent("MemoryTrackerSummary") reports aggregate totals.
|
||||
void memTrackerDump(int64_t bytesThreshold);
|
||||
|
||||
// Snapshot iteration for tests. The callback runs while a copy of the
|
||||
// aggregation table is held; the spinlock is not held during the callback.
|
||||
void memTrackerForEachSite(std::function<void(const MemoryTrackerCallSite&)> cb);
|
||||
|
||||
// Reset all state. Tests only — not safe for production use.
|
||||
void memTrackerResetForTest();
|
||||
|
||||
// Test only: arm a one-shot so the next sampled allocation simulates a tracker
|
||||
// metadata-allocation failure (the sampled path throws std::bad_alloc, which the
|
||||
// tracker swallows). Lets tests verify the tracker fails open — the underlying
|
||||
// allocation still succeeds and tracking recovers afterward. Never used in
|
||||
// production.
|
||||
void memTrackerFailNextSampleForTest();
|
||||
|
||||
#else // !FDB_MEMORY_TRACKER — compiled out: hooks are no-ops, no operator new override.
|
||||
|
||||
inline void memTrackerInit() {}
|
||||
inline void memTrackerOnAlloc(void*, std::size_t) {}
|
||||
inline void memTrackerOnFree(void*) {}
|
||||
inline void memTrackerDump(int64_t) {}
|
||||
inline void memTrackerResetForTest() {}
|
||||
class MemTrackerSuppress {
|
||||
public:
|
||||
MemTrackerSuppress() {}
|
||||
~MemTrackerSuppress() {}
|
||||
MemTrackerSuppress(const MemTrackerSuppress&) = delete;
|
||||
MemTrackerSuppress& operator=(const MemTrackerSuppress&) = delete;
|
||||
};
|
||||
|
||||
#endif // FDB_MEMORY_TRACKER
|
||||
|
||||
#endif // FLOW_MEMORY_TRACKER_H
|
||||
|
|
@ -95,6 +95,20 @@
|
|||
#error Missing force inline
|
||||
#endif
|
||||
|
||||
// Keep a function un-inlined and (on GCC) un-cloned so its address stays stable
|
||||
// for return-address matching in tests. GCC -O3 IPA cloning (.constprop/.isra)
|
||||
// otherwise runs the code under a synthetic clone symbol at a different address
|
||||
// than &fn, so a captured frame won't fall in [&fn, &fn+size); noclone disables
|
||||
// it. clang lacks noclone and doesn't clone this way, so it gets noinline only;
|
||||
// empty elsewhere (e.g. MSVC, which we cannot compile-test).
|
||||
#if defined(__clang__)
|
||||
#define force_noinline __attribute__((noinline))
|
||||
#elif defined(__GNUC__)
|
||||
#define force_noinline __attribute__((noinline, noclone))
|
||||
#else
|
||||
#define force_noinline
|
||||
#endif
|
||||
|
||||
/*
|
||||
* Visual Studio (.NET 2003 and beyond) has an __assume compiler
|
||||
* intrinsic to hint to the compiler that a given condition is true
|
||||
|
|
|
|||
Loading…
Reference in New Issue