wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Revert WaitStorageMetricsHandleError SevWarn upgrade (#13336) (#13338) 

PR #12913 upgraded WaitStorageMetricsHandleError from SevDebug to SevWarn
after 60s of retrying, intending to give operators visibility into stuck
shard metric loops. In production this fires ~10/sec/cluster, putting it
in the top 35 most frequent TraceEvents.

The 60s threshold doesn't filter for stuck shards. The SS-side
waitMetrics handler is a long-poll with STORAGE_METRIC_TIMEOUT = 600s
(fdbserver/storageserver/storageserver.actor.cpp:11476). On timeout, the
SS deliberately returns wrong_shard_server with probability
WAIT_METRICS_WRONG_SHARD_CHANCE = 0.1 to force clients to refresh their
location cache (fdbserver/core/StorageMetrics.cpp:742). So most calls
that ever reach this catch block already have Elapsed >= 600s by design,
and the SevWarn fires on normal quiet-cluster operation, not stuck
shards. DD-init stall visibility (the actual goal of PR #12913) is
covered by the DDInitServerListAndDataMoveReadComplete /
DDInitKeyServerScanComplete / DDInitSlowDataMoveRead events that PR also
added — those are at the right layer.

Revert this event to plain SevDebug.
This commit is contained in:
Michael Stack 2026-06-15 18:57:31 -07:00 committed by GitHub
parent 164d2e7aed
commit d5223da940
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
fdbclient/NativeAPI.actor.cpp
View File

@ -8165,7 +8165,6 @@ ACTOR Future<std::pair<Optional<StorageMetrics>, int>> waitStorageMetrics(
Optional<Reference<TransactionState>> trState) {
state Span span("NAPI:WaitStorageMetrics"_loc, generateSpanID(cx->transactionTracingSample));
state double startTime = now();
state double lastLogTime = 0;
state int retryCount = 0;
loop {
if (trState.present()) {
@ -8215,16 +8214,7 @@ ACTOR Future<std::pair<Optional<StorageMetrics>, int>> waitStorageMetrics(
}
} catch (Error& e) {
retryCount++;
// Upgrade from SevDebug to SevWarn after 60 seconds of retrying,
// but rate-limit warns to avoid flooding the trace log on fast retries.
Severity sev = SevDebug;
if (now() - startTime > 60.0) {
if (now() - lastLogTime >= 10.0) {
sev = SevWarn;
lastLogTime = now();
}
}
TraceEvent(sev, "WaitStorageMetricsHandleError")
TraceEvent(SevDebug, "WaitStorageMetricsHandleError")
.error(e)
.detail("Keys", keys)
.detail("Elapsed", now() - startTime)