PR #12913 upgraded WaitStorageMetricsHandleError from SevDebug to SevWarn
after 60s of retrying, intending to give operators visibility into stuck
shard metric loops. In production this fires ~10/sec/cluster, putting it
in the top 35 most frequent TraceEvents.
The 60s threshold doesn't filter for stuck shards. The SS-side
waitMetrics handler is a long-poll with STORAGE_METRIC_TIMEOUT = 600s
(fdbserver/storageserver/storageserver.actor.cpp:11476). On timeout, the
SS deliberately returns wrong_shard_server with probability
WAIT_METRICS_WRONG_SHARD_CHANCE = 0.1 to force clients to refresh their
location cache (fdbserver/core/StorageMetrics.cpp:742). So most calls
that ever reach this catch block already have Elapsed >= 600s by design,
and the SevWarn fires on normal quiet-cluster operation, not stuck
shards. DD-init stall visibility (the actual goal of PR #12913) is
covered by the DDInitServerListAndDataMoveReadComplete /
DDInitKeyServerScanComplete / DDInitSlowDataMoveRead events that PR also
added — those are at the right layer.
Revert this event to plain SevDebug.
Backport of #13038 to release-7.4.
When a commit request returns commit_unknown_result for a transaction
carrying an idempotency id, determineCommitStatus reads the system
keyspace and may discover the original commit succeeded. In that case
we reconstruct the versionstamp from CommitResult but previously forgot
to record the commit version back on the TransactionState, so callers
observing getCommittedVersion() after success would see invalidVersion.
Mirror the normal-path assignment on the replay branch, and add a
regression guard in AutomaticIdempotencyWorkload asserting every
successful commit reports a non-invalid version. Adapted to the
actor-based workload still in use on 7.4.
Fixes#12582.
* Add DD init visibility, metrics retries, shard tracking, scan progress, and team collection logging (#12913)
Add logging throughout DD startup and runtime to diagnose slow startups,
stuck data moves, and undesired server classification.
DDTxnProcessor: Log elapsed time for server list + data move read
transaction and keyServer scan. Warn when getRange(dataMoveKeys) takes
over 5 seconds.
DataDistribution: Add NumShards and NumServers to DDInitGotInitialDD.
Add DDInitResumedDataMoves summary with ValidMoves, CancelledMoves,
EmptyMoves counts and elapsed time. Log DD exit reason as DDExiting
at SevWarn. Add DDInit-prefixed trace events throughout startup sequence.
DDTeamCollection: Add Reason and Address details to UndesiredStorageServer
trace events to distinguish version lag, same-address, wrong-class, and
exclusion causes.
DDShardTracker: Log TrackInitialShardsComplete with shard count and
TrackInitialShardsMetricsComplete with elapsed time.
NativeAPI: Add retry counting and logging for getStorageMetrics timeouts.
* Remove double trace
* Fix DD trace event issues: rate-limit warns, deduplicate events, fix DDExiting
- Wire up unused lastLogTime in waitStorageMetrics to rate-limit SevWarn
events to once per 10s after the 60s threshold. Previously every retry
(up to 100/s with 10ms WRONG_SHARD_SERVER_DELAY) emitted SevWarn.
- Remove redundant DataDistributorRunning and DDInitRunning events inside
the try block. Rename the original DataDistributorRunning to DDInitRunning
so a single event serves both purposes and fits the DDInit* query pattern.
- Move DDExiting to the three actual throw sites so it only fires on
terminal exits, not on retries of movekeys_conflict/dd_config_changed
when DD is disabled.
* Trim over-apologetic DDInitDone comment
handleTssMismatches(DatabaseContext* cx) uses a pointer to DatabaseContext
object, which can be destroyed when "tr" is reset within this actor. However,
the actor can't be destroyed because it's on the stack. Introducing this delay
gives a chance to cancel the actor.
1. Only start `clientStatusUpdateActor` when `DatabaseContext` successfully established connection
to the cluster.
2. `DatabaseContext` starts few actors for monitoring as well as update client status to server.
These sometimes pass pointers to `DatabaseContext` and the `Transaction` object created within these
actors will increment the refcount. This can lead to cyclic references or holding `DatabaseContext`
objects for long period of times if `Transaction` object is not cleaned up or we keep retrying
forever without any limit.
3. Some actors don't handle `actor_cancelled` errors which can lead then to stay alive forever. This
patch fixes some of those.
rdar://155780163 found that in multi-version client if primary is incompatible with the cluster, we
keep trying to reconnect with cluster, and spamming with IncompatibleConnectionClosed messages. (1)
should be enough to fix that, but other issues were found during investigation which can potentially
lead to similar issues in future.
Testing:
Manually started a 7.1 cluster, with 7.4 primary client and 7.1. secondary client. Started a client.
Without this patch we'll see bunch of IncompatibleConnectionClosed messages, and with this patch
they will be gone.
* range lock framework
* improve the framework
* persist to txnStateStore
* fix bugs
* code clean
* code clean
* bug fix
* address comments
* add complex test workload and fix bugs found by the workload
* add workload correctness check and fix bugs
* code clean up
* add random range lock injection
* fix bugs in RandomRangeLock.actor.cpp
* enable random range lock injection in general workloads
* add rangelockcycle test
* disable random range lock in backup workloads
* nits
* add range lock ownership concept
* enable lock ownership to rangeLock
* api deal with tenant
* fix CI
* add test for multiple rangeLock owners
* nits
* address comments and renaming
* address comments
This is a rewrite of BUGGIFY function/macros. Seems the performance
improved a lot during the simulation, e.g.
fdbserver -r simulation -b on -f ../CycleTest.toml -s 99438
Without this patch:
Unseed: 54646
Elapsed: 494.091327 simsec, 14.586831 real seconds
With this patch:
Unseed: 54646
Elapsed: 494.091327 simsec, 12.580612 real seconds
I expected the improvement but did not expect a ~13% improvement.
CC sets a version to int_max in ClientDBInfo indicating a refresh, however,
proxy server would reject this version for the error of future_version.
This change fixes this issue by not sending int_max, instead maintaining a
lastKnown in memory and send it to grvproxy to get latest globalconfig.
this change also fixes some java tests that were used to test the fix
In the current code, errors are retried in getConsistentReadVersion, so it's
possible that the client has cancelled the GRV request, but readVersionBatcher
continue retrying, which can lead to many clients DDoS GRV proxies, especially
when the database has become unavailable for a while and clients are issuing
many GRV requests.
* - Compare storage replicas on reads (in "loadBalance()")
* - Do consistency check on reads in loadbalance
* - Do replica consistency check in the case where loadBalance issues
requests to multiple storage servers
* - Address a state variable related bug
* - Code formatting
* - API simplification
* - Simplify code
* - Code formatting
* - Address a review comment
* throttle hot shards
* expire throttled shards over time
* add backoff
* Parallelize messaging from RK to CP
* Obtain shards from a single SS
* handle expired transactions
* bump transaction_throttled_hot_shard
* Change SevError to SevWarn for CannotMonitorHotShardForSS
* Add log per request
* Make CodeProbeImpl::_hitCount atomic
* Structure access to TraceLog::logTraceEventMetrics so that it is written before a trace log is opened and only read from one thread after it is opened.
* Fix condition in assert
* Rename TraceLog::log to logMetrics and move initialization of trace log metrics into TraceLog::open
---------
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
* Add networkoption to disable non-TLS connections
* add disable plaintext connection to fdbserver
* python doc
* Formatting
* Add tls disable plaintext connection to client api test
* review
* fix negative test
* formatting
* add TLS support to c client config tests
Adds support for TLS in the client and server separately
* add tests for disable_plaintext_connections
Test TLS and Plaintext Clusters and Clients
* Fix documentation
* Rename option to indicate it is client-only
* clearer formatting
* default to allowing plaintext connections
* add SetTLSDisablePlaintextConnection to go bindings
* fixing bugs with tenant_mode required on external clients and changing test to find them
* Update fdbcli/BlobKeyCommand.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
---------
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
Michael Stack