Problem: some of these transactions have been observed to involve cascades of many shards trying to execute them simultaneously owing to somewhat unpredictable DD dynamics. This then results in DD being pegged on CPU with many failing transactions.
Solution: the counters here will make it clear if this is happening and if so, where, so that further mitigations/solutions can be devised. Some of these code paths have TraceEvents, but some don't and of the existing TraceEvents, many are sampled. This PR avoids all of that and counts every time through, making us less reliant on guesswork.
An alternative approach (not taken here for many reasons) would be to instrument transaction code to let callers pass a tag. Then have the transaction client and/or server side emit a few metrics (started, committed, aborted) parameterized by tag.
One may reasonably object that the boilerplate here is kind of fugly. I guess my view is that at this late date the time for more subtle approaches has come and gone. We need this code to tell us what it is doing and if this looks a little intrusive, so be it.
Testing:
20260422-234326-gglass-2f514436068d99e6 compressed=True data_size=35519571 duration=5298275 ended=100000 fail=1 fail_fast=10 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:55:44 sanity=False started=100000 stopped=20260423-003910 submitted=20260422-234326 timeout=5400 username=gglass
* Add DD init visibility, metrics retries, shard tracking, scan progress, and team collection logging (#12913)
Add logging throughout DD startup and runtime to diagnose slow startups,
stuck data moves, and undesired server classification.
DDTxnProcessor: Log elapsed time for server list + data move read
transaction and keyServer scan. Warn when getRange(dataMoveKeys) takes
over 5 seconds.
DataDistribution: Add NumShards and NumServers to DDInitGotInitialDD.
Add DDInitResumedDataMoves summary with ValidMoves, CancelledMoves,
EmptyMoves counts and elapsed time. Log DD exit reason as DDExiting
at SevWarn. Add DDInit-prefixed trace events throughout startup sequence.
DDTeamCollection: Add Reason and Address details to UndesiredStorageServer
trace events to distinguish version lag, same-address, wrong-class, and
exclusion causes.
DDShardTracker: Log TrackInitialShardsComplete with shard count and
TrackInitialShardsMetricsComplete with elapsed time.
NativeAPI: Add retry counting and logging for getStorageMetrics timeouts.
* Remove double trace
* Fix DD trace event issues: rate-limit warns, deduplicate events, fix DDExiting
- Wire up unused lastLogTime in waitStorageMetrics to rate-limit SevWarn
events to once per 10s after the 60s threshold. Previously every retry
(up to 100/s with 10ms WRONG_SHARD_SERVER_DELAY) emitted SevWarn.
- Remove redundant DataDistributorRunning and DDInitRunning events inside
the try block. Rename the original DataDistributorRunning to DDInitRunning
so a single event serves both purposes and fits the DDInit* query pattern.
- Move DDExiting to the three actual throw sites so it only fires on
terminal exits, not on retries of movekeys_conflict/dd_config_changed
when DD is disabled.
* Trim over-apologetic DDInitDone comment
* Add multipart retry delay configuration. (#12076)
Add documentation to .h file.
Co-authored-by: stack <stack@duboce.com>
* Add a quickstart section on the bulkdump/bulkload feature to the user doc. (#12090)
* Fix assertion failure in fdbcli (#12095)
* Add Option to Turn Off Failure Injection to BulkLoad Process (#12096)
* add option to turn off failure injection to bulkload process in simulation
* nit
* DD BulkLoadTask Load Balance (#12103)
* bulkload load balance
* improve bulkload task submission efficiency
* address comments
* address comments
* cleanup
* Take a --proxy argument as backup_agent does. (#12110)
* Add "--proxy" to fdbserver (#12109)
* add --proxy to fdbserver
* fmt
* address comments
* use g_network to store proxy for s3client_ci
* Add 'ls' to s3client for the case where 'aws s3' is not installable (#12118)
* Add 'ls' to s3client for the case where 'aws s3' is not installable
-- i.e. production -- and we want a tool to test s3 connectivity.
Use it like this:
~/build_output/bin/s3client \
--tls-ca-file /etc/ssl/cert.pem \
--blob-credentials /path/to/credentials.json \
ls "blobstore://backup-us-west-2.s3.amazonaws.com/x?bucket=backup-us-west-2®ion=us-west-2"
* fdbclient/S3BlobStore.actor.cpp
* fdbclient/include/fdbclient/S3BlobStore.h
Make constructResourcePath method on endpoint so accessible making ls
URLs. Handling for encoded ampersands too. Allow 404 when requesting
resource that doesn't exist.
* fdbclient/S3Client.actor.cpp
* fdbclient/S3Client_cli.actor.cpp
Add in ls implementation.
* fdbclient/tests/s3client_test.sh
Add test for new ls facility.
* See if this fixes compile issue
---------
Co-authored-by: michael stack <stack@duboce.com>
* Ingest sst files rather than their keyvalue content (2nd attempt) (#12108)
* Ingest sst files rather than their keyvalue content.
* fdbclient/ServerKnobs.cpp
* fdbclient/include/fdbclient/ServerKnobs.h
Add BULK_LOAD_USE_SST_INGEST knob.
* fdbclient/include/fdbclient/IKeyValueStore.actor.h
Add ingestSSTFiles and supportSStIngestion.
* fdbserver/KeyValueStoreRocksDB.actor.cpp
* fdbserver/KeyValueStoreShardedRocksDB.actor.cpp
Implement ingestSSTFiles and supportSStIngestion
* fdbserver/storageserver.actor.cpp
If BULK_LOAD_USE_SST_INGEST and BulkLoadType::SST, ingest sst file
rather than read keyvalues.
* fdbclient/tests/fdb_cluster_fixture.sh
Use rocksdb instead of sqllite in tests.
* Revert "Ingest sst files rather than their keyvalue content."
This reverts commit 6d50bf15d3566de17fa6febf2afba0f220846ca6.
* ctests pass
* Restore ingest minus metric
* Add back ingestDuration metric
* Formatting
---------
Co-authored-by: michael stack <stack@duboce.com>
* Allow resource to be empty (#12129)
* RocksDB Do Compact After BulkLoad (#12130)
* rocksdb bulkload compact after load
* address comments
* address comments
* job overwrite task root path (#12138)
* BulkLoad Load Balance Based on DataMove Busy Map (#12137)
* bulkload load balance in ddqueue
* cleanup
* address comments
* nit
* bug fix
* nit
* address comments
* avoid scheduler busy loop
* address comments
* address comments
* avoid busy loop (#12157)
* First set of fixes... sizes and implement deleteResource (#12156)
Co-authored-by: michael stack <stack@duboce.com>
---------
Co-authored-by: Michael Stack <saintstack@users.noreply.github.com>
Co-authored-by: stack <stack@duboce.com>
* bulkload support general engine and fix bugs
* add comments
* improve test coverage and fix bug
* nits and address comments
* nit
* nits
* fix data inconsistency bug due to bulkload metadata
* fix ss bulkload task metadata bugs
* nit and fix CI issue
* fix bugs of restore ss bulkload metadata
* use ssBulkLoadMetadata for fetchKey and general kv engine
* cleanup bulkload file for fetchkey
* fix CI issue
* fix simulation stuck due to repeated re-recruitment of unfit dd
* randomly do available space check when finding the dest team for bulkload in simulation
* address conflict
* code clean up
* update BulkDumping.toml same to BulkLoading.toml
* consolidate ss fetchkey and fetchshard failed to read bulkload task metadata
* fix DD bulkload job busy loop bug which causes segfault and test terminate unexpectedly in joshua test
* nit
* fix ss busy loop for bulkload in fetchkey
* use sqlite for bulkload ctest
* fix bulkload ctest stuck issue due to merge and change storage engine to ssd
* fix comments for CC recruit DD
* address comments
* address comments
* add comments
* fix ci format issue
* address comments
* add comments
* add bulkload job framework and fix bugs
* add BulkLoadChecksum, fix CI issue
* nits
* nits
* address comments
* mitigate perpetual wiggle to make sure DD can select a valid team to inject data
* fix submitBulkDumpJob and submitBulkLoadJob
* change remoteRoot to jobRoot
* add comments
* Improve BulkLoad/Dump implementation
* make bulkload test data folder inside simfdb folder
* simplify code
* use manifest in bulkdump metadata
* use manifest in bulkload
* apply bulkload fileset to bulkload and fix bugs of bytesampling value generation
* remove BulkDumpFileFullPathSet
* address comments
* address comments
* address comments
* [Release-7.3] TeamRedundant and TeamUnhealthy data moves choose best destination with probability (#11668)
* team redundant and unhealthy data moves can choose best dest with probability
* nits
* nits
* enable wantTrueBestIfMoveout
* fix getteam stuck
* [Release-7.3] Delay team remover when space pivot is low (#11665)
* [Release-7.3] Validate ServerTeam count per server in simulation (#11678)
* validate server team count in simulation
* change naming (not relevant to the PR title)
* address comments and add a new trace event BuildTeamsLastBuildTeamsFailed triggered when buildTeam failed
To retrieve storage metadata for every status json request is very expensive
for clusters with a large number of storage servers. So I change the logic so
that ClusterController actively monitors changes to storage metadata, and only
retrieves them when there is a change.
gxglass