Add START_MOVE_KEYS_MAX_RETRIES knob (default 50, BUGGIFY=10) and throw
start_move_keys_too_many_retries when exceeded. Wire the new error into
DDRelocationQueue (same light-weight re-queue handler as
finish_move_keys_too_many_retries and move_to_removed_server) and
normalDDQueueErrors so DD does not restart. Also propagate actor_cancelled
before the retry limit check in startMoveKeys.
Ports apple/foundationdb#13381 to release-7.4:
- DDRelocationQueue: the relocation pipeline gate now holds moves of all
priorities when the pipeline is full; only cancellations bypass (they
reduce tracked metadata rather than add to it). Previously moves with
priority >= PRIORITY_TEAM_UNHEALTHY passed through unconditionally. Add a
"DD Pipeline Full" CODE_PROBE where pipelineFull is set.
- ServerKnobs: raise the buggified DD_MAX_PIPELINE_MOVES from 5 to 20 so
simulation does not test under artificial scarcity (degenerate cases).
- Add fast/DDPipelineSaturation.toml, which saturates the back-pressure
gate via many small shards plus failure-recovery relocations and trips
the new code probe.
Paths adapted from main's layout (fdbserver/core, fdbserver/datadistributor)
to release-7.4 (fdbclient/ServerKnobs.cpp, fdbserver/DDRelocationQueue.actor.cpp);
the BUGGIFY macro is retained.
Testing: Release build; ran fdbserver -r simulation on DDPipelineSaturation
across several seeds, buggify on and off. The gate change is behaviorally
inert in these runs -- a baseline binary with the priority condition restored
produces bit-identical transaction counts and there are no SevError events --
and the "DD Pipeline Full" code probe is hit every run. The test currently
trips the Cycle workload's "Rate below desired rate" check because this branch
still carries the Cycle minimum-rate check that main removed in #13297; it
passes once that check is gone. Joshua validation deferred until the #13297
backport is sequenced onto the branch.
Originally this was PR #13112. That was ported to release-7.3 as PR #13243. Since release-7.4 is closer to release-7.3 than to main, port 13243 to release-7.4.
* DD 7.3 finish movekeys backoff (#12991)
* Add jitter to finishMoveKeys backoff
Jitter the exponential backoff delay to [0.75x, 1.25x] of the base
value. This prevents all 15 FlowLock slots from retrying in lockstep
when they all hit transaction_too_old at the same time.
* Cap finishMoveKeys backoff at 5s after jitter
Apply the 5.0s cap after jitter so the final delay never exceeds
the documented maximum.
It might have been easier to backport PR #12940 from main directly to release-7.4.
* Backport PR #12940 to release-7.3 (async status related) (#13068)
* Make status async and return before we timeout (#12940)
* Fix clusterGetStatus declaration for both actor compiler and clang-ide builds
The actor compiler adds const& to ACTOR params, so the source must use
value params. Clang-ide compiles .actor.cpp directly without the actor
compiler, so it needs a matching value-param declaration. Use
NO_INTELLISENSE guard to provide both.
* formatting
* Fix some misc backporting related issues
* Fix build for async status backport: remove tenant stuff, grey failure stuff, and update includes
* formatting
* remove dead code found in subsequent review
---------
Co-authored-by: Han Xu <hgcrpd@gmail.com>
* fix merge damage
* fix some more merge damage
* formatting
---------
Co-authored-by: Han Xu <hgcrpd@gmail.com>
Cherrypick PR#13072 into release-7.4.
This function does a lot of stuff and has been observed to be expensive from time to time. This PR tracks its latency explicitly. This is done in 5 minute windows which seems about the right granularity for the sort of incidents which we might anticipate. Note too short, not too long.
Testing (7.4 specific):
20260428-142636-gglass-3b52791173c99235 compressed=True data_size=41487471 duration=4813496 ended=100000 fail_fast=1000 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=1:00:16 sanity=False started=100000 stopped=20260428-152652 submitted=20260428-142636 timeout=5400 username=gglass
* latency tracking for dataDistributionRelocator (#13072)
* add latency tracking for dataDistributionRelocator
* simplify approach, and try to avoid negative latency samples
* formatting
* Arena/FastAlloc: add comments where potential metrics can be added (#12306)
* Arena/FastAlloc: add comments where potential metrics can be incremented.
The intent is to count allocations and bytes.
Remove a commented-out ifdef block that has been disabled for many years and
which does not work (per the explanation in the comment). We don't need to
keep reading about the results of a small failed experiment from many years ago.
* add more one FIXME comment
* add one more METRICS-FIXME
* Add a SimpleCounter template for counter metrics (#12326)
* ignore TAGS (from etags/ctags)
* Add initial SimpleCounter interface/implementation/unit tests
* Add initial SimpleCounter interface/implementation/unit tests
* Fix unit test
* Improve clarity on unit test
* Update FIXME comments
* Address review comments. Must use function local static mutex
* update comment
* Go back to template specializations to handle older C++ versions
* SimpleCounter: periodically log the counters to TraceEvent (#12329)
* SimpleCounter: periodically log the counters to TraceEvent. Muck with hierarchical names to comply with random rules.
* Update doc about Prometheus metric names
* relax assertion about counter count, because unit test is actually running in fdbserver and that causes a unrelated counter to be created
* Update SimpleCounter unit tests not to use metric names that break Trace.cpp simulation-only checks (#12333)
* Add a pointed comment in UnitTest.h about some weaknesses
* Use counter names that will get converted to field names that TraceEvent does not complain about
* unit test: do not use a counter name that will cause Trace.cpp to emit errors in simulation
* update comment about caveats with unittests breaking simulation
* yet another field name fix
* just call validateField() directly from simple counters
* run report loop in unit tests
* fix build, fix comment
* blah blah blah
* always be munging metric names
* Instrument Arena, FastAlloc, Platform.cpp with SimpleCounter metrics to count allocations and bytes (#12339)
* emit a simpleCounterReport when we declare out of memory
* FastAlloc.h: initial pass of adding byte/object allocation/deallocation metrics
* Avoid conflict over the name SimpleCounter by eliminating this private definition of a name which is too valuable for this one random file to claim for its own use
* FastAlloc.cpp, Platform.actor.cpp: initial pass at adding SimpleCounter metrics to count allocations and bytes
* rename wrapper calls and update comments
* Count bytes copied in StringRef
* Arena.cpp: instrument allocations and some other stuff
* Arena.cpp: simplify use of SimpleCounter
* simpleCounterReport: generate TraceEvent in batches of MAX_TRACE_EVENT_LENGTH / 100 counters to avoid trace buffer overflow
* Eliminate poorly motivated trace field name validation, and change SimpleCounter to emit Prometheus-compatible metric names (#12356)
Trace.cpp does not provide a rationale for validateField() and validateFormat(). It appears to be some kind of
XML related validation. Why we should care about this is not clear. The output is going to Splunk. As far as I know, Splunk is supposed to be pretty liberal in what it accepts as input.
Add logic in SimpleCounter.cpp to convert hierarchical metric names to Prometheus compatible metric names
by the simple rule of converting intermediate '/' chars into '_', i.e. something like /flow/arena/bytesAllocated becomes
flow_arena_bytesAllocated. I feel hierarchical names are still slightly better, and very easy to reason about when
creating new metric names on the fly, but ensuring that they are at least Prometheus compatible should allow targeting
to future metrics platforms down the road.
Testing:
20250905-010257-gglass-25f3ef43ccc1c130 compressed=True data_size=41538755 duration=6389116 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=1:00:34 sanity=False started=100000 stopped=20250905-020331 submitted=20250905-010257 timeout=5400 username=gglass
* replace undocumented trace event field name rules with a rule that enforces that field names must be valid Prometheus metric names. No idea why the old code declines to even state what it is trying to be compatible with
* move Prometheus metric name validation to SimpleCounter.cpp. Remove validation from Trace.cpp. This stuff is going to Splunk. Splunk takes what we give it.
* Use simple counter to replace recently added net2 counters (#12358)
* use simple counter to replace recent added net2 counters
* allow unit test to use SimpleCounter Trace event
---------
Co-authored-by: Zhe Wang <zhe.wang@wustl.edu>
* Add multipart retry delay configuration. (#12076)
Add documentation to .h file.
Co-authored-by: stack <stack@duboce.com>
* Add a quickstart section on the bulkdump/bulkload feature to the user doc. (#12090)
* Fix assertion failure in fdbcli (#12095)
* Add Option to Turn Off Failure Injection to BulkLoad Process (#12096)
* add option to turn off failure injection to bulkload process in simulation
* nit
* DD BulkLoadTask Load Balance (#12103)
* bulkload load balance
* improve bulkload task submission efficiency
* address comments
* address comments
* cleanup
* Take a --proxy argument as backup_agent does. (#12110)
* Add "--proxy" to fdbserver (#12109)
* add --proxy to fdbserver
* fmt
* address comments
* use g_network to store proxy for s3client_ci
* Add 'ls' to s3client for the case where 'aws s3' is not installable (#12118)
* Add 'ls' to s3client for the case where 'aws s3' is not installable
-- i.e. production -- and we want a tool to test s3 connectivity.
Use it like this:
~/build_output/bin/s3client \
--tls-ca-file /etc/ssl/cert.pem \
--blob-credentials /path/to/credentials.json \
ls "blobstore://backup-us-west-2.s3.amazonaws.com/x?bucket=backup-us-west-2®ion=us-west-2"
* fdbclient/S3BlobStore.actor.cpp
* fdbclient/include/fdbclient/S3BlobStore.h
Make constructResourcePath method on endpoint so accessible making ls
URLs. Handling for encoded ampersands too. Allow 404 when requesting
resource that doesn't exist.
* fdbclient/S3Client.actor.cpp
* fdbclient/S3Client_cli.actor.cpp
Add in ls implementation.
* fdbclient/tests/s3client_test.sh
Add test for new ls facility.
* See if this fixes compile issue
---------
Co-authored-by: michael stack <stack@duboce.com>
* Ingest sst files rather than their keyvalue content (2nd attempt) (#12108)
* Ingest sst files rather than their keyvalue content.
* fdbclient/ServerKnobs.cpp
* fdbclient/include/fdbclient/ServerKnobs.h
Add BULK_LOAD_USE_SST_INGEST knob.
* fdbclient/include/fdbclient/IKeyValueStore.actor.h
Add ingestSSTFiles and supportSStIngestion.
* fdbserver/KeyValueStoreRocksDB.actor.cpp
* fdbserver/KeyValueStoreShardedRocksDB.actor.cpp
Implement ingestSSTFiles and supportSStIngestion
* fdbserver/storageserver.actor.cpp
If BULK_LOAD_USE_SST_INGEST and BulkLoadType::SST, ingest sst file
rather than read keyvalues.
* fdbclient/tests/fdb_cluster_fixture.sh
Use rocksdb instead of sqllite in tests.
* Revert "Ingest sst files rather than their keyvalue content."
This reverts commit 6d50bf15d3566de17fa6febf2afba0f220846ca6.
* ctests pass
* Restore ingest minus metric
* Add back ingestDuration metric
* Formatting
---------
Co-authored-by: michael stack <stack@duboce.com>
* Allow resource to be empty (#12129)
* RocksDB Do Compact After BulkLoad (#12130)
* rocksdb bulkload compact after load
* address comments
* address comments
* job overwrite task root path (#12138)
* BulkLoad Load Balance Based on DataMove Busy Map (#12137)
* bulkload load balance in ddqueue
* cleanup
* address comments
* nit
* bug fix
* nit
* address comments
* avoid scheduler busy loop
* address comments
* address comments
* avoid busy loop (#12157)
* First set of fixes... sizes and implement deleteResource (#12156)
Co-authored-by: michael stack <stack@duboce.com>
---------
Co-authored-by: Michael Stack <saintstack@users.noreply.github.com>
Co-authored-by: stack <stack@duboce.com>
Increase the buffer size for backup workers and rename the knob.
Seed: -f ./tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml -s 3120109021 -b on
* improve bulkload code
* address CI
* disable audit storage replica check and distributed consistency check in bulkload and bulkdump simulation test
* fix ci
* disable waitForQuiescence in bulkload and bulkdump tests
* bulkload support general engine and fix bugs
* add comments
* improve test coverage and fix bug
* nits and address comments
* nit
* nits
* fix data inconsistency bug due to bulkload metadata
* fix ss bulkload task metadata bugs
* nit and fix CI issue
* fix bugs of restore ss bulkload metadata
* use ssBulkLoadMetadata for fetchKey and general kv engine
* cleanup bulkload file for fetchkey
* fix CI issue
* fix simulation stuck due to repeated re-recruitment of unfit dd
* randomly do available space check when finding the dest team for bulkload in simulation
* address conflict
* code clean up
* update BulkDumping.toml same to BulkLoading.toml
* consolidate ss fetchkey and fetchshard failed to read bulkload task metadata
* fix DD bulkload job busy loop bug which causes segfault and test terminate unexpectedly in joshua test
* nit
* fix ss busy loop for bulkload in fetchkey
* use sqlite for bulkload ctest
* fix bulkload ctest stuck issue due to merge and change storage engine to ssd
* fix comments for CC recruit DD
* address comments
* address comments
* add comments
* fix ci format issue
* address comments
* add comments
* add bulkload job framework and fix bugs
* add BulkLoadChecksum, fix CI issue
* nits
* nits
* address comments
* mitigate perpetual wiggle to make sure DD can select a valid team to inject data
* fix submitBulkDumpJob and submitBulkLoadJob
* change remoteRoot to jobRoot
* add comments
* Improve BulkLoad/Dump implementation
* make bulkload test data folder inside simfdb folder
* simplify code
* use manifest in bulkdump metadata
* use manifest in bulkload
* apply bulkload fileset to bulkload and fix bugs of bytesampling value generation
* remove BulkDumpFileFullPathSet
* address comments
* address comments
* address comments
michael stack