PR #12913 upgraded WaitStorageMetricsHandleError from SevDebug to SevWarn
after 60s of retrying, intending to give operators visibility into stuck
shard metric loops. In production this fires ~10/sec/cluster, putting it
in the top 35 most frequent TraceEvents.
The 60s threshold doesn't filter for stuck shards. The SS-side
waitMetrics handler is a long-poll with STORAGE_METRIC_TIMEOUT = 600s
(fdbserver/storageserver/storageserver.actor.cpp:11476). On timeout, the
SS deliberately returns wrong_shard_server with probability
WAIT_METRICS_WRONG_SHARD_CHANCE = 0.1 to force clients to refresh their
location cache (fdbserver/core/StorageMetrics.cpp:742). So most calls
that ever reach this catch block already have Elapsed >= 600s by design,
and the SevWarn fires on normal quiet-cluster operation, not stuck
shards. DD-init stall visibility (the actual goal of PR #12913) is
covered by the DDInitServerListAndDataMoveReadComplete /
DDInitKeyServerScanComplete / DDInitSlowDataMoveRead events that PR also
added — those are at the right layer.
Revert this event to plain SevDebug and add a comment explaining why,
so a future reader doesn't reintroduce the upgrade.
* Remove dead client-side storage cache code left behind by #12486
LocationInfo::hasCaches, addCaches(), updateLocationCacheWithCaches(),
DatabaseContext::{cacheListMonitor, updateCache}, and the dead hasCaches branch in loadBalance() - all unreachable since #12486.
Follow-up sweep in the same spirit as #13119.
* Inline trivial loadBalance wrapper into its callers
Addresses review feedback on #13326. With the hasCaches branch gone, the anonymous-namespace loadBalance(DatabaseContext*, Reference<LocationInfo>, ...) wrapper is a pure forwarder. Delete it and update its five callers to call loadBalance(alternatives->locations(), ...) directly. No behavior change.
* Detect missing commit version after automatic idempotency replay
Adds a post-commit assertion in AutomaticIdempotencyWorkload that every
successful transaction reports a non-invalid getCommittedVersion(). The
replay path in NativeAPI exercised by CLIENT_BUGGIFY currently leaves
trState->committedVersion unset, which this regression guard catches.
Related to #12582.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Populate committedVersion on automatic idempotency replay path
When a commit request returns commit_unknown_result for a transaction
carrying an idempotency id, determineCommitStatus reads the system
keyspace and may discover the original commit succeeded. In that case
we reconstruct the versionstamp from CommitResult but previously forgot
to record the commit version back on the TransactionState, so callers
observing getCommittedVersion() after success would see invalidVersion.
Mirror the normal-path assignment from line ~4456 on the replay branch.
Fixes#12582.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: use thread-local mt19937_64 for automatic idempotency ID
* feat: add platform::getRandomBytes and use it for idempotency IDs
* fix: pr fixes + getentropy support
* fix: pr review changes
* Add DD init and team collection logging for diagnosing slow startups
When SHARD_ENCODE_LOCATION_METADATA=true we take new codepaths often
opaque. Add logging.
For example, DD init hung for 14-16 minutes with zero visibility into
what was stuck. The only clue was a gap between DDInitUpdatedReplicaKeys
and DDInitGotInitialDD trace events. Diagnosing the root cause required
extensive log splunking of SS metrics to determine that a single
getRange(dataMoveKeys) read was queued on an overloaded storage server.
DDTxnProcessor.actor.cpp:
- Log elapsed time for the server list + data move read transaction
(DDInitServerListAndDataMoveReadComplete) with NumDataMoves, NumServers
- Log elapsed time for the keyServer scan (DDInitKeyServerScanComplete)
with NumShards
- Warn when getRange(dataMoveKeys) takes >5 seconds
(DDInitSlowDataMoveRead)
DataDistribution.actor.cpp:
- Add NumShards and NumServers to DDInitGotInitialDD
- Promote DDInitFoundDataMove from SevDebug to SevInfo so individual
data moves are visible in production logs
- Add DDInitResumedDataMoves summary event with ValidMoves,
CancelledMoves, EmptyMoves counts and elapsed time
DDTeamCollection.actor.cpp:
- Add Reason and Address details to UndesiredStorageServer trace events
to distinguish version lag, same-address, wrong-class, and exclusion
causes without needing to correlate with other log lines
* Revert DDInitFoundDataMove to SevDebug to avoid log spam with many data moves
* Add DD startup visibility: metrics retries, shard tracking, scan progress
Additional logging to address DD operational opacity during startup,
based on past incidents where DD hung with no visibility into the cause.
NativeAPI.actor.cpp:
- Log WaitStorageMetricsRetrying every 60s when waitStorageMetrics is
stuck retrying wrong_shard_server or all_alternatives_failed, with
the key range, retry count, and elapsed time. Previously these retries
were silent (SevDebug only), making it impossible to identify which
shard was stuck or that retries were even happening.
DDShardTracker.actor.cpp:
- Log TrackInitialShardsComplete after shard tracker setup with count
- Log TrackInitialShardsMetricsComplete after changeSizes() finishes
with elapsed time. changeSizes() waits for ALL shards to report
metrics via getFirstSize/waitStorageMetrics -- if any shard metrics
never arrive, this hangs silently.
DDTxnProcessor.actor.cpp:
- Log DDInitKeyServerScanProgress every 30s during the multi-transaction
keyServer scan with current beginKey, batch count, shards scanned,
and elapsed time. With 255K shards this scan requires many transactions
and a stuck one was previously invisible.
DataDistribution.actor.cpp:
- Log DDInitComplete with elapsed time after DataDistributor::init()
finishes, providing a single event showing total init duration.
- NativeAPI.actor.cpp: Move retry logging outside the error-type if block
so all errors get keys/elapsed/retries details. Use severity upgrade
(SevDebug -> SevWarn after 60s) on the existing WaitStorageMetricsHandleError
event instead of a separate event name.
- DataDistribution.actor.cpp: Add periodic progress logging (every 30s) in
resumeFromDataMoves loop so operators can watch counts go up during long
data move recovery.
- CompileBoost.cmake: Remove BOOST_NO_CXX98_FUNCTION_BASE since 7.3 CI is
broken independently of this change.
DD exits (e.g. movekeys_conflict) were invisible because
reportErrorsExcept suppresses logging for "normal" DD errors.
Add DDExiting trace event at SevWarn with error and code so
every DD death is visible in trace logs.
Trevor Clinkenbeard