The logic to determine the validity of a process joining a cluster now
belongs on the worker and the cluster controller. It is no longer
restricted to tlogs and storages, but instead applies to all processes
(even stateless ones).
The cluster ID is now stored in the database instead of in the
txnStateStore. The cluster controller will read it on boot and send it
to all processes to persist.
And have these processes enter a "zombie" state where they cancel all
their actors and then wait forever, refusing to do any additional work
until they are manually handled by the operator.
* Recruit new singleton for consistency checker.
* Recruit the consistency checker only if enabled.
* Add a yield in monitorConsistencyChecker().
* Minor fixes.
* Consistency check workload enhancements.
* Minor fixes and clarifications.
* clang format
* Clang format.
* Minor fixes, cleanup, debug tracing.
* Misc.
* Move the consistency scan information from dbconfig to a key backed object.
* Move consistency scan config out of db cofig to a state object and feature rename.
* ConsistencyCheck workload refactor.
* devFormat
* Update fdbcli/ConsistencyScanCommand.actor.cpp
* Review Comments.
Co-authored-by: negoyal <neelam.goyal@gmail.com>
Co-authored-by: Ata E Husain Bohra <ata.husain@snowflake.com>
The `--no-config-db` flag, passed to `fdbserver`, will disable the
configuration database. When this flag is specified, no `ConfigNode`s
will be started, the `ConfigBroadcaster` will not be started, and on a
coordinator change no attempt will be made to lock `ConfigNode`s.
Configuration database data lives on the coordinators. When a change
coordinators command is issued, the data must be sent to the new
coordinators to keep the database consistent.
Description
In current code the leader replacement handling is done by
ClusterControllerWatchDog, it is possible that if the cluster
recruitment is in-progress, the leader replacement handling gets
delayed causing the multiple leaders to present simulateneously
impacting cluster recovery.
Move the cluster leader replacement logic to ClusterController
core to avoid above described scenario.
Testing
* do not count recently created change feeds for throttling
* fix: blocked assignments were not decremented when force purging
* fix: created needs to be updated when the changefeed is reset
* added asserts to detect if ratekeeper is throttled on blob workers
* fix: the cluster controller did not halt singletons if their role wait failure client triggered but their worker wait failure client did not trigger
Description
Register worker maintain 'id_worker' map to track the worker interface
registeration, it also sets 'WorkerAvailabilityWatcher' actor to track
for failures. However, if worker is already registered but interface gets
updated, the existing code doesn't actively cancel the watcher. One possible
race condition is, old watcher detects worker failure and removes the worker
from id_worker map even before the new watcher started monitoring the new
interface. If such a scenario is hit, it trips an assert in rebootAndCheck
routine which expects the worker should be present in the id_worker map.
Patch addresses the race condition by actively cancelling existing watcher
actor before registering the new watcher actor.
Testing
devRunCorrectness - 100K
* assorted bug fixes for blob granules
* Fixing transaction used after commit in blob manager recovery
* fixing race with granule merging across hard boundaries because it hadn't loaded them yet
Trevor Clinkenbeard