Description
Patch addresses occurrences where Status.actor ends up bubbling timeout error
up to the ClusterController causing recovery to be triggered when
ClusterGetStatus timeout for some reason.
Testing
* fix: Non-storage processes were not being checked for locality exclusions
fix: Data distribution when not detect a newly added process was locality excluded
fix: RemoveServerSafely did not wait for processes to be excluded before killing them when excluding localities
* fix: do not allow locality based excludes if they cannot exclude the required addresses
* Remove duplicate getRange() for DB handles and update existing GetRange to accept DB handles.
* Initial progress checkpoint on new ConsistencyScan role.
* Updated TODOs, finished most if not all state updates.
* placeholder
* Add more TODOs, documentation and comment improvements.
* Checkpoint round state to avoid advancing progress if commit fails.
* Bug fix, check is supposed to be for overlap, not lack of overlap.
* Added more TODO's and added faked read results / exceptions and faked DB size retrieval to prove the consistencyScanCore logic works.
* Update JSON schemas and command help.
* Add comment about lifetime stats reset.
* More TODO comments and some renames for clarity, some bug fixes.
* properly stopping consistency scan in simulation so that it doesn't run forever and cause quiet database to fail
* removing trailing comma from consistency_scan json schema
* Making CC inconsistency not an error if it's intentional tss corruption
* consistency scan actually reads storage locations
* added check that consistency scan actually completes a round in simulation, fixed bug and added debugging around consistency scan getting stuck
* made consistency scan properly fetch database size
* refactoring data check to be used in both consistency scan and consistency check
* checking that consistency scan always completes at least one round and doesn't get stuck
* cleanup
* fixing ide build
* consistencyscan fdbcli command wasn't actually changing db state
* consistencyscan fdbcli command always said enabled even when it wasn't
---------
Co-authored-by: Steve Atherton <steve.atherton@snowflake.com>
* Add processID, networkAddress, and locality to layer status JSON for Backup Agents.
* Backup/dr agent determines network address to report in Layer Status only once, when the status updater loop begins, since it is a blocking call which connects to the cluster. And lots of code cleanup.
Adds a new `version_epoch` object to `status json`, which includes the
status of the feature, and the current epoch if it is enabled. If the
version epoch is disabled, the `epoch` field will not be present.
```
{
"client" : {
...
},
"cluster" : {
...
"version_epoch" : {
"enabled" : "true",
"epoch" : "100000"
},
...
}
}
```
There are a few issues here:
1. The cgroup reporting added looks for top level metrics of cgroup v1.
2. Looking at the whole machine tells us nothing about fdbserver.
3. The reporting itself expects the values to be there rather than
failing silently if we're not on v1. As cgroup v2 is significantly
more prominent.
This fixes 3, but 1 and 2 are open and quite questionable. These really
ought to be reported by a 3rd party observer and not fdb itself.
Previously, `status json` would report a machine as excluded when any of
its processes were excluded. But a machine should only be reported as
excluded when all of its processes are excluded.
I tested this change by running a small, three process test cluster. I
excluded one machine, and verified `status json` reported the overall
machine as not excluded.
Ata E Husain Bohra