Ports apple/foundationdb#13381 to release-7.4:
- DDRelocationQueue: the relocation pipeline gate now holds moves of all
priorities when the pipeline is full; only cancellations bypass (they
reduce tracked metadata rather than add to it). Previously moves with
priority >= PRIORITY_TEAM_UNHEALTHY passed through unconditionally. Add a
"DD Pipeline Full" CODE_PROBE where pipelineFull is set.
- ServerKnobs: raise the buggified DD_MAX_PIPELINE_MOVES from 5 to 20 so
simulation does not test under artificial scarcity (degenerate cases).
- Add fast/DDPipelineSaturation.toml, which saturates the back-pressure
gate via many small shards plus failure-recovery relocations and trips
the new code probe.
Paths adapted from main's layout (fdbserver/core, fdbserver/datadistributor)
to release-7.4 (fdbclient/ServerKnobs.cpp, fdbserver/DDRelocationQueue.actor.cpp);
the BUGGIFY macro is retained.
Testing: Release build; ran fdbserver -r simulation on DDPipelineSaturation
across several seeds, buggify on and off. The gate change is behaviorally
inert in these runs -- a baseline binary with the priority condition restored
produces bit-identical transaction counts and there are no SevError events --
and the "DD Pipeline Full" code probe is hit every run. The test currently
trips the Cycle workload's "Rate below desired rate" check because this branch
still carries the Cycle minimum-rate check that main removed in #13297; it
passes once that check is gone. Joshua validation deferred until the #13297
backport is sequenced onto the branch.
Originally this was PR #13112. That was ported to release-7.3 as PR #13243. Since release-7.4 is closer to release-7.3 than to main, port 13243 to release-7.4.
* DD 7.3 finish movekeys backoff (#12991)
* Add jitter to finishMoveKeys backoff
Jitter the exponential backoff delay to [0.75x, 1.25x] of the base
value. This prevents all 15 FlowLock slots from retrying in lockstep
when they all hit transaction_too_old at the same time.
* Cap finishMoveKeys backoff at 5s after jitter
Apply the 5.0s cap after jitter so the final delay never exceeds
the documented maximum.
Cherrypick PR#13072 into release-7.4.
This function does a lot of stuff and has been observed to be expensive from time to time. This PR tracks its latency explicitly. This is done in 5 minute windows which seems about the right granularity for the sort of incidents which we might anticipate. Note too short, not too long.
Testing (7.4 specific):
20260428-142636-gglass-3b52791173c99235 compressed=True data_size=41487471 duration=4813496 ended=100000 fail_fast=1000 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=1:00:16 sanity=False started=100000 stopped=20260428-152652 submitted=20260428-142636 timeout=5400 username=gglass
* latency tracking for dataDistributionRelocator (#13072)
* add latency tracking for dataDistributionRelocator
* simplify approach, and try to avoid negative latency samples
* formatting
* Add multipart retry delay configuration. (#12076)
Add documentation to .h file.
Co-authored-by: stack <stack@duboce.com>
* Add a quickstart section on the bulkdump/bulkload feature to the user doc. (#12090)
* Fix assertion failure in fdbcli (#12095)
* Add Option to Turn Off Failure Injection to BulkLoad Process (#12096)
* add option to turn off failure injection to bulkload process in simulation
* nit
* DD BulkLoadTask Load Balance (#12103)
* bulkload load balance
* improve bulkload task submission efficiency
* address comments
* address comments
* cleanup
* Take a --proxy argument as backup_agent does. (#12110)
* Add "--proxy" to fdbserver (#12109)
* add --proxy to fdbserver
* fmt
* address comments
* use g_network to store proxy for s3client_ci
* Add 'ls' to s3client for the case where 'aws s3' is not installable (#12118)
* Add 'ls' to s3client for the case where 'aws s3' is not installable
-- i.e. production -- and we want a tool to test s3 connectivity.
Use it like this:
~/build_output/bin/s3client \
--tls-ca-file /etc/ssl/cert.pem \
--blob-credentials /path/to/credentials.json \
ls "blobstore://backup-us-west-2.s3.amazonaws.com/x?bucket=backup-us-west-2®ion=us-west-2"
* fdbclient/S3BlobStore.actor.cpp
* fdbclient/include/fdbclient/S3BlobStore.h
Make constructResourcePath method on endpoint so accessible making ls
URLs. Handling for encoded ampersands too. Allow 404 when requesting
resource that doesn't exist.
* fdbclient/S3Client.actor.cpp
* fdbclient/S3Client_cli.actor.cpp
Add in ls implementation.
* fdbclient/tests/s3client_test.sh
Add test for new ls facility.
* See if this fixes compile issue
---------
Co-authored-by: michael stack <stack@duboce.com>
* Ingest sst files rather than their keyvalue content (2nd attempt) (#12108)
* Ingest sst files rather than their keyvalue content.
* fdbclient/ServerKnobs.cpp
* fdbclient/include/fdbclient/ServerKnobs.h
Add BULK_LOAD_USE_SST_INGEST knob.
* fdbclient/include/fdbclient/IKeyValueStore.actor.h
Add ingestSSTFiles and supportSStIngestion.
* fdbserver/KeyValueStoreRocksDB.actor.cpp
* fdbserver/KeyValueStoreShardedRocksDB.actor.cpp
Implement ingestSSTFiles and supportSStIngestion
* fdbserver/storageserver.actor.cpp
If BULK_LOAD_USE_SST_INGEST and BulkLoadType::SST, ingest sst file
rather than read keyvalues.
* fdbclient/tests/fdb_cluster_fixture.sh
Use rocksdb instead of sqllite in tests.
* Revert "Ingest sst files rather than their keyvalue content."
This reverts commit 6d50bf15d3566de17fa6febf2afba0f220846ca6.
* ctests pass
* Restore ingest minus metric
* Add back ingestDuration metric
* Formatting
---------
Co-authored-by: michael stack <stack@duboce.com>
* Allow resource to be empty (#12129)
* RocksDB Do Compact After BulkLoad (#12130)
* rocksdb bulkload compact after load
* address comments
* address comments
* job overwrite task root path (#12138)
* BulkLoad Load Balance Based on DataMove Busy Map (#12137)
* bulkload load balance in ddqueue
* cleanup
* address comments
* nit
* bug fix
* nit
* address comments
* avoid scheduler busy loop
* address comments
* address comments
* avoid busy loop (#12157)
* First set of fixes... sizes and implement deleteResource (#12156)
Co-authored-by: michael stack <stack@duboce.com>
---------
Co-authored-by: Michael Stack <saintstack@users.noreply.github.com>
Co-authored-by: stack <stack@duboce.com>
* bulkload support general engine and fix bugs
* add comments
* improve test coverage and fix bug
* nits and address comments
* nit
* nits
* fix data inconsistency bug due to bulkload metadata
* fix ss bulkload task metadata bugs
* nit and fix CI issue
* fix bugs of restore ss bulkload metadata
* use ssBulkLoadMetadata for fetchKey and general kv engine
* cleanup bulkload file for fetchkey
* fix CI issue
* fix simulation stuck due to repeated re-recruitment of unfit dd
* randomly do available space check when finding the dest team for bulkload in simulation
* address conflict
* code clean up
* update BulkDumping.toml same to BulkLoading.toml
* consolidate ss fetchkey and fetchshard failed to read bulkload task metadata
* fix DD bulkload job busy loop bug which causes segfault and test terminate unexpectedly in joshua test
* nit
* fix ss busy loop for bulkload in fetchkey
* use sqlite for bulkload ctest
* fix bulkload ctest stuck issue due to merge and change storage engine to ssd
* fix comments for CC recruit DD
* address comments
* address comments
* add comments
* fix ci format issue
* address comments
* add comments
* Improve BulkLoad/Dump implementation
* make bulkload test data folder inside simfdb folder
* simplify code
* use manifest in bulkdump metadata
* use manifest in bulkload
* apply bulkload fileset to bulkload and fix bugs of bytesampling value generation
* remove BulkDumpFileFullPathSet
* address comments
* address comments
* address comments
* [Release-7.3] TeamRedundant and TeamUnhealthy data moves choose best destination with probability (#11668)
* team redundant and unhealthy data moves can choose best dest with probability
* nits
* nits
* enable wantTrueBestIfMoveout
* fix getteam stuck
* [Release-7.3] Delay team remover when space pivot is low (#11665)
* [Release-7.3] Validate ServerTeam count per server in simulation (#11678)
* validate server team count in simulation
* change naming (not relevant to the PR title)
* address comments and add a new trace event BuildTeamsLastBuildTeamsFailed triggered when buildTeam failed
* Add usable region check per shard for encode shard location metadata
* nits
* nit
* address comments
* fix SS assertion failed for a wrong data move type generated by an old binary which does not encode the data move type in the data move id
* fix ClientTransactionProfilingCorrectness 7.3 upgrade test considering physical shard move compatibility
* code clean
* split CycleTestRestart in upgrading test from release-7.3
* address comments
* nits
* Terminate DD if SHARD_ENCODE_LOCATION_METADATA is not enabled and storage_engine_type is ShardedRocksDB.
* Fixed Error in non-main thread.
* Minor improvements.
* Log data move reasons in DataMoveStats.
* Remove a move-in-shard only when it is no longer used.
* Dont remove ranges if restore failed due to actor_cancelled.
gxglass