Add a flag in HealthMetrics to indicate that batch priority is rate limited.
Data distributor pulls this flag from proxy to know roughly when rate limiting
happens.
DD uses this information to determine when to do the rebalance in the background,
i.e., moving data from heavily loaded servers to lighter ones. If the cluster is
currently rate limited for batch commits, then the rebalance will use longer
time intervals, otherwise use shorter intervals. See BgDDMountainChopper() and
BgDDValleyFiller() in DataDistributionQueue.actor.cpp.
Add a new role for ratekeeper.
Remove StorageServerChanges from data distribution.
Ratekeeper monitors storage servers, which borrows the idea from
DataDistribution.
After we add a new data distributor role, we publish the data
related to data distributor and rate keeper through the new
role (and new worker).
So the status needs to contact the data distributor, instead of master,
to get the status information.
In addTeam(), to determine the team is badTeam or not, we should check
redundantTeam before check satisfiesPolicy. Because if a team is
redundantTeam, it has been removed from the system before we call addTeam().
The only reason we call addTeam() for a removed redundantTeam is to
kick off the badTeam cleanup logic.
When we remove a machine team in teamRemover function,
we should always find the machine team in the global machineTeams.
Change the ASSERT to the above invariant.
The previous commit merge with the master, which just merges
the pull request #1062 from jzhou77/PR that adds a new DataDistribution role.
The merge causes conflicts and errors in simulation tests.
This commit resolves the code conflicts and
tries to fix the new errors after incorporating the new DataDistribution role
We do NOT enforce the removing order of removing a machine team
and the server teams on the machine team.
This is for the benefit of clear code logic.
When a storage server locality changes, we first remove the server
and its machine if needed, before we handle the server team removal
and addition.
We do not actively remove a machine team when it has no server team on it.
But since adding a server team may add a machine team, we need to be
careful that the machine team number is not larger than the desired number
due to server team creation.
So whenever a server team is removed, we should check if the teamRemover
should be kicked in.
When the machine number changes due to machine remove event,
the desired machine team number changes. Then we need to
make sure the teamRemover actor is running to clean up the
redundant teams.
getTeam is called very frequently and does not create a new team.
So no need to call teamRemover in getTeam.
teamRemover should be called only when a new team may be added.
The buildTeam() can create teams with undesired storage servers, which are
considered unhealthy. As a result, the data movement can become stuck.
Fix this by adding an ACTOR monitorHealthyTeams that builds team every one
second whenever there is no healthy teams.
Clean up storageServerTracker() interface.
When zeroHealthyTeams signals and the storage server becomes healthy, we could
attempt buildTeam before the ServerStatusMap is updated. As a result, the
healthy server is not available for use. Fix by delaying the buildTeam after the
status map is updated.
This bug was introduced in cee23ee3. During a configuration change, the data
distributor was restarted, which destroys previous DDTeamCollection and cancels
all previous teamTracker(). In this case, even though the healthy team count
reaches 0, there is no need to try to rebuild teams. The bug is triggered when
trying rebuilding teams, DDTeamCollection is already destroyed.
When moving keys to a team, if one of the server in the target team died, then
the move can become stuck. This is because the DDTeamCollection waits for all
the data movement of the failed server to be completed. However, in this case,
because the movement has not finished yet, checking the database tells us there
is no key assocated with this server and it is safe to go ahead. In reality,
only the in-memory structure knows there is pending movement, i.e., unfinished
move causes some keys to be attributed to the failed server. Thus, the server
can't be removed yet. Fix by adding a check with in-memory structure in
waitForAllDataRemoved().
Use const& to optimize a few function parameters.
Evan Tschannen