3a3ab5664b
fix: team trackers for bad teams that contain a removed servers must be cancelled or the cluster will falsely report those teams as failed
2019-11-22 10:20:13 -08:00
f8e44d2f71
fix: If a storage server was offline, it would not be checked for being in an undesired dc
2019-10-23 23:04:39 -07:00
86bcb84b45
Raised the data distribution priority of splitting shards above restoring fault tolerance to avoid hot write shards
2019-10-11 17:50:43 -07:00
4b5080fbea
added a few more missing data distribution priorities
2019-09-27 19:39:53 -07:00
324d0bd3b0
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-cleanup-mutations
2019-09-27 19:15:14 -07:00
3bb62e008c
lowered the priority of some delays in data distribution so that the process will prefer other work
2019-09-27 18:33:13 -07:00
32ebd08f9f
DD:Trigger storage recruitment when an invalid address locality is corrected
2019-09-24 13:35:38 -07:00
515689d07b
Update fdbserver/DataDistribution.actor.cpp
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-09-18 14:45:18 -07:00
d175b61a62
DD:Trace when invalid locality is corrected
...
Change getWorkers param from txn handler to db
2019-09-18 13:52:54 -07:00
93bbc26a35
set:erase:Use return value of erase iterator as next iterator
2019-09-17 15:28:30 -07:00
d2fd1f4931
DD:MisconfiguredLocality:Fix review comments
2019-09-17 13:04:21 -07:00
37d2318eed
DD:Handle worker with incorrect locality
...
When a worker has incorrect locality, the worker will be excluded from
storage recruitment.
When the worker has its locality corrected by system operators,
the worker will be reincluded for storage recruitment.
2019-09-14 12:12:56 -07:00
c3960aba17
DD:initializeStorage:Exclude worker with invalid locality
2019-09-13 22:05:41 -07:00
75460089e1
DD_VALIDATE_LOCALITY:Add comment for our future selves
...
When we add simulation test that misconfigure a cluster by not setting some
locality entries, we should set DD_VALIDATE_LOCALITY always true.
Otherwise, simulation tests may fail.
2019-09-13 16:26:54 -07:00
78b8e48cef
DD:ValidLocality:Resolve review comment
2019-09-13 15:35:16 -07:00
e1dcdbf3d2
LocalityData:Remove verbose check for valid locality
2019-09-13 15:11:13 -07:00
8970d9858b
DD:isValidLocality:A generic way to check any replicationPolicy
2019-09-13 14:55:51 -07:00
1196841b3d
DD:IsValidLocality:Clang format
2019-09-13 13:56:43 -07:00
e8878b16d4
DD:Valid locality includes an empty but set locality entry
2019-09-13 13:55:46 -07:00
1596e2e4a5
DD:TCMachine:Use processID as machineID if zoneID is unset
2019-09-13 13:43:41 -07:00
3ad7e3adb3
DD:DD_VALIDATE_LOCALITY:Guard the checking of locality validity
2019-09-13 13:19:35 -07:00
90d6a27a0d
DD:IsValidLocality:Consider configured replica policy
2019-09-13 12:04:49 -07:00
52f6297b52
DD:Introduce isValidLocality
...
A server or machine has a valid locality only if it sets correct
locality entries.
Build teams should only use the valid locality servers or machines
2019-09-13 11:30:26 -07:00
cc41f3e2fc
fix: an unhealthy server with a low number of teams could cause data distribution to build every possible teams
2019-09-12 14:18:10 -07:00
5d87443323
improved error msgs for snapshot cmd
2019-08-27 16:43:52 -07:00
297b65236f
added additional trace events to warn when different parts of shard relocations take more than 10 minutes
2019-08-16 14:56:58 -07:00
ba54508c47
code cleanup
2019-08-06 16:30:30 -07:00
5dc4c80d44
fix: the machineAttrition workload did not ensure that healthyZone was always cleared
...
fix: an assert could trigger spuriously
2019-08-05 15:00:17 -07:00
7d7aa27c2d
Merge pull request #1814 from dongxinEric/feature/1508/finer-grained-dd-controls
...
Added finer grained controls to DataDistribution in fdbcli.
2019-07-31 17:36:20 -07:00
bba01c6531
fix: add subsetOfEmergencyTeam could add an unsorted team
2019-07-31 16:02:08 -07:00
b653ddb30d
Final clean ups after rebasing master
2019-07-30 22:35:34 -07:00
5d20364423
Address review comments
2019-07-30 22:24:30 -07:00
1922c39377
Resolve review comments. 100K run shows one suspecious ASSERT_WE_THINK failure which I think could be a race.
2019-07-30 22:24:30 -07:00
c6e5472d8d
Apply suggestions from code review
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-07-30 22:20:45 -07:00
f5d6e3a5b3
- Addressed review commends
...
- Added test for the storage server failure disable switch
2019-07-30 22:20:45 -07:00
ae11efcb0a
Made following changes:
...
- Make sure the disabled data distribution won't be accidentally enabled by the 'maintenance' command
- Make sure the status json reflects the status of DD accordingly
- Make sure the CLI can play with the new DD states correctly, i.e. print out warns when necessary
2019-07-30 22:20:45 -07:00
4ecfc9830f
Added finer grained controls to DataDistribution in fdbcli. What's happening under the hood is:
...
- Use pre-existing 'healthZone' key and write a special value to it in order to disable DD for all storage server failures
- Use a new system key 'rebalanceDDIgnored' key to disable/enable DD for all rebalance reasons(MountainChopper and ValleyFiller)
Kicked off two 200K correctness and showed no related errors.
2019-07-30 22:17:21 -07:00
dd4ab63d90
fixed another bad trace event name
2019-07-30 19:36:26 -07:00
b8cd51c4d3
fixed invalid trace event name
2019-07-30 19:23:54 -07:00
a78a97f186
Merge pull request #1908 from etschannen/feature-better-dd
...
A few data distribution improvements
2019-07-30 17:34:50 -07:00
a88aaa0f04
review comment
2019-07-30 17:04:51 -07:00
63941e0d96
disable DD with a in-memory flag and use in snapv2
2019-07-30 17:04:51 -07:00
5dd9043fd3
addressed review comments
2019-07-30 17:04:41 -07:00
481642fbd4
Merge branch 'master' into feature-better-dd
2019-07-30 16:56:27 -07:00
a3fe3d4324
Merge pull request #1923 from xumengpanda/mengxu/evan-dd-improvement-minor-improvement
...
DD:Change condition for lastBuildTeamsFailed
2019-07-30 16:54:42 -07:00
14648e20f9
Merge pull request #1901 from ajbeamon/data-distribution-receives-bytes-input-rate
...
Send bytes input rate to data distribution
2019-07-30 15:01:36 -07:00
0e50656c7f
DD:Change condition for lastBuildTeamsFailed
...
Change the threshold team number per server that should set lastBuildTeamsFailed
from DESIRED_TEAMS_PER_SERVER to
(SERVER_KNOBS->DESIRED_TEAMS_PER_SERVER * (configuration.storageTeamSize + 1)) / 2;
2019-07-30 11:07:02 -07:00
a0f26b604c
Merge pull request #1907 from etschannen/master
...
A number of bug fixes for rare problems found by correctness testing
2019-07-29 21:04:38 -07:00
5a56f6b456
minor snap create client improvement and bug fixes
2019-07-29 20:28:22 -07:00
cc4481b71a
team builders prefer to make teams which overlap less with existing teams
2019-07-28 23:44:23 -07:00
Evan Tschannen