In rare circumstances where the cluster controller dies / moves to a new
machine, sometimes only a minority of `ConfigNode`s received messages
telling them they were registered. When the `ConfigNode`s attempt to
register with the new broadcaster (on the new cluster controller), the
knob system would get stuck because only a minority would be registered.
Part of this change allows registration of unregistered `ConfigNode`s if
there is no path to a majority of registered nodes.
This commit also removes an attempt to read the latest configuration
snapshot when a rollforward timeout occurs. The normal retry loop will
eventually fetch an up to date snapshot and the rollforward will be
retried.
The calculation to determine how many non-timeout replies had been
received was incorrect, causing rollback/rollforward requests to not be
sent, causing the dynamic knob subsystem to get stuck.
Fixes an issue where commit versions from previous requests sent to
ConfigNodes were being reused when a new quorum of commit versions was
requested. This was occurring due to a failure to reset the state of
GetCommittedVersionQuorum after a full snapshot request.
Fixes an issue where commit versions from previous requests sent to
ConfigNodes were being reused when a new quorum of commit versions was
requested. This was occurring due to a failure to reset the state of
GetCommittedVersionQuorum after a full snapshot request.
Chaoguang Lin