* Arena/FastAlloc: add comments where potential metrics can be added (#12306)
* Arena/FastAlloc: add comments where potential metrics can be incremented.
The intent is to count allocations and bytes.
Remove a commented-out ifdef block that has been disabled for many years and
which does not work (per the explanation in the comment). We don't need to
keep reading about the results of a small failed experiment from many years ago.
* add more one FIXME comment
* add one more METRICS-FIXME
* Add a SimpleCounter template for counter metrics (#12326)
* ignore TAGS (from etags/ctags)
* Add initial SimpleCounter interface/implementation/unit tests
* Add initial SimpleCounter interface/implementation/unit tests
* Fix unit test
* Improve clarity on unit test
* Update FIXME comments
* Address review comments. Must use function local static mutex
* update comment
* Go back to template specializations to handle older C++ versions
* SimpleCounter: periodically log the counters to TraceEvent (#12329)
* SimpleCounter: periodically log the counters to TraceEvent. Muck with hierarchical names to comply with random rules.
* Update doc about Prometheus metric names
* relax assertion about counter count, because unit test is actually running in fdbserver and that causes a unrelated counter to be created
* Update SimpleCounter unit tests not to use metric names that break Trace.cpp simulation-only checks (#12333)
* Add a pointed comment in UnitTest.h about some weaknesses
* Use counter names that will get converted to field names that TraceEvent does not complain about
* unit test: do not use a counter name that will cause Trace.cpp to emit errors in simulation
* update comment about caveats with unittests breaking simulation
* yet another field name fix
* just call validateField() directly from simple counters
* run report loop in unit tests
* fix build, fix comment
* blah blah blah
* always be munging metric names
* Instrument Arena, FastAlloc, Platform.cpp with SimpleCounter metrics to count allocations and bytes (#12339)
* emit a simpleCounterReport when we declare out of memory
* FastAlloc.h: initial pass of adding byte/object allocation/deallocation metrics
* Avoid conflict over the name SimpleCounter by eliminating this private definition of a name which is too valuable for this one random file to claim for its own use
* FastAlloc.cpp, Platform.actor.cpp: initial pass at adding SimpleCounter metrics to count allocations and bytes
* rename wrapper calls and update comments
* Count bytes copied in StringRef
* Arena.cpp: instrument allocations and some other stuff
* Arena.cpp: simplify use of SimpleCounter
* simpleCounterReport: generate TraceEvent in batches of MAX_TRACE_EVENT_LENGTH / 100 counters to avoid trace buffer overflow
* Eliminate poorly motivated trace field name validation, and change SimpleCounter to emit Prometheus-compatible metric names (#12356)
Trace.cpp does not provide a rationale for validateField() and validateFormat(). It appears to be some kind of
XML related validation. Why we should care about this is not clear. The output is going to Splunk. As far as I know, Splunk is supposed to be pretty liberal in what it accepts as input.
Add logic in SimpleCounter.cpp to convert hierarchical metric names to Prometheus compatible metric names
by the simple rule of converting intermediate '/' chars into '_', i.e. something like /flow/arena/bytesAllocated becomes
flow_arena_bytesAllocated. I feel hierarchical names are still slightly better, and very easy to reason about when
creating new metric names on the fly, but ensuring that they are at least Prometheus compatible should allow targeting
to future metrics platforms down the road.
Testing:
20250905-010257-gglass-25f3ef43ccc1c130 compressed=True data_size=41538755 duration=6389116 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=1:00:34 sanity=False started=100000 stopped=20250905-020331 submitted=20250905-010257 timeout=5400 username=gglass
* replace undocumented trace event field name rules with a rule that enforces that field names must be valid Prometheus metric names. No idea why the old code declines to even state what it is trying to be compatible with
* move Prometheus metric name validation to SimpleCounter.cpp. Remove validation from Trace.cpp. This stuff is going to Splunk. Splunk takes what we give it.
* Use simple counter to replace recently added net2 counters (#12358)
* use simple counter to replace recent added net2 counters
* allow unit test to use SimpleCounter Trace event
---------
Co-authored-by: Zhe Wang <zhe.wang@wustl.edu>
* Define API for unsuppressable TraceEvent types
Add trace checking tests for authz trace events
* Revert temporary configurations used for debugging
* Simplify/Modernize flow audit logging API
- Do event type whitelist checks at compile time
- Use ""_audit literal API instead of a tag struct
- Replace int with a lightweight struct for tracking/modifying TraceEvent enablement
* Revert installing signal handler for SIGTERM and refactor test script
Move trace checker to local_cluster.py
* Lengthen public key refresh interval and add more audited events
* Try and make MSVC and Mac build happy
* consteval > constexpr
'inline consteval' still causes link errors in Mac builds
- Revert JWT timestamp from double back to uint64_t
- Describe in comment the need for wait loop at the beginning of watchPublicKeyJwksFile()
- Revert CLI args definition in fdbserver.actor.cpp that was
incorrectly tabbed with a bad clang-format run
* Update network address in trace logs; Add system monitor for flowprocess
* Create a new trace file with the correct process address for flowprocess
* Remove unused debugging traces
* Add a new error lock_file_failure; Change please_reboot_remote_kv_store to please_reboot_kv_store; Add the code to only reboot the kv store but not the worker; Remove some unnecessay traces
* Add error handling for file_not_found in handleIOErrors
* Format worker.actor.cpp file
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli
* Auto format python files used in upgrade tests
* Add a test for upgrading to a future FDB version
* Emphasize that the options for using future protocol version are intended for test purposes only
* Make the global variable for current protocol version visible only locally
* Refactirng to avoid using currentProtocolVersion() in static intialization
* Update go bindings
* proof of concept
* use code-probe instead of test
* code probe working on gcc
* code probe implemented
* renamed TestProbe to CodeProbe
* fixed refactoring typo
* support filtered output
* print probes at end of simulation
* fix missed probes print
* fix deduplication
* Fix refactoring issues
* revert bad refactor
* make sure file paths are relative
* fix more wrong refactor changes
gxglass