Commit Graph

290 Commits

Author SHA1 Message Date
Xiaoxi Wang 4b92f8f546 add relocate reason and set teamSorter in relocator 2022-03-18 16:39:31 -07:00
sfc-gh-tclinkenbeard 4a0720e8ad Provide destTeam parameter to rebalanceTeams again.
There is a wait before the load bytes of the dest team is used, so the
last commit inadvertently caused a behaviour change. Instead, update the
comment for the function, and pass const IDataDistributionTeam
references.
2022-03-18 10:25:13 -07:00
Josh Slocum 5fbdbb5c4a Better cancelling logic that reflects whether move has actually started 2022-03-18 10:43:24 -05:00
sfc-gh-tclinkenbeard a04ca65ad5 Change signature of rebalanceTeams 2022-03-17 23:27:33 -07:00
Josh Slocum 67eba5ec7c Limiting DD Moves by destination SS. 2022-03-15 13:52:19 -05:00
Xiaoxi Wang 10eb082888 format code; fix DD ignore option bug" 2022-03-15 10:31:43 -07:00
Xiaoxi Wang d38b827906 fix DD ignore bug 2022-03-13 23:47:07 -07:00
Xiaoxi Wang 8d20ee8432 change storage metrics of read sample calculation 2022-03-07 17:36:43 -08:00
Xiaoxi Wang 970a84bf63 fix teamSorter setting 2022-03-03 12:58:45 -08:00
Xiaoxi Wang a28b0f4361 Merge branch 'main' of https://github.com/apple/foundationdb into readaware 2022-03-03 11:15:18 -08:00
Xiaoxi Wang a05f6f851a fix reference assign bug 2022-03-02 21:56:03 -08:00
Xiaoxi Wang f48ba989c3 add status 2022-03-01 15:27:27 -08:00
Xiaoxi Wang 5e74b5006e fix uninitialized member 2022-02-28 10:22:32 -08:00
Xiaoxi Wang 40a1f562a7 temporary change special key for data distributor 2022-02-25 11:01:23 -08:00
Xiaoxi Wang a07e413eae metrics comparator; rebalanceReadLoad() 2022-02-24 16:41:01 -08:00
A.J. Beamon 250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Evan Tschannen 1afef44f19 fix: self->shardsAffectedByTeamFailure->moveShard must be called without any waits after getting the destination team or we could miss failure notifications for the storage servers in the destination team 2021-11-23 15:26:38 -08:00
Xiaoxi Wang d1959122af consider wiggling when waitUntilHealthy 2021-10-27 09:08:37 -07:00
Chang Liu 8761960cdc fix roll trace event issue for data distribution(master)
Description

Testing
2021-09-24 15:40:17 -07:00
Chang Liu 814e3c729f fix roll trace event issue for data distribution(master)
Description

Testing
2021-09-24 15:40:17 -07:00
Chang Liu c10dd0df4b fix roll trace event issue for data distribution
Description

Testing
2021-09-24 15:40:17 -07:00
Xiaoge Su abf73047ca Enforce std:: specifier rather than using namespace 2021-09-16 19:40:28 -07:00
sfc-gh-tclinkenbeard 3442ebd3b7 Fix more -Wreorder-ctor warnings across many files 2021-07-24 11:20:51 -07:00
Daniel Smith 8efe3b296a Delete remaining extern declarations for noUnseed 2021-07-08 19:19:22 -04:00
Xiaoxi Wang 0493d149e6 wait remove 2021-06-21 05:18:42 +00:00
Xiaoxi Wang d46fccc30f Revert "Revert "Properly set simulation test for perpetual storage wiggle and bug fixing""
This reverts commit ad576e8c20.
2021-06-11 22:58:05 +00:00
Xiaoxi Wang ad576e8c20
Revert "Properly set simulation test for perpetual storage wiggle and bug fixing" 2021-06-11 09:07:45 -07:00
Xiaoxi Wang 17ac91bac4
Merge pull request #4929 from sfc-gh-xwang/ppwtest
Properly set simulation test for perpetual storage wiggle and bug fixing
2021-06-10 14:09:50 -07:00
Xiaoxi Wang 4220a548ce use the same health check as exclude to avoid 'best team get stuck' 2021-06-09 22:51:46 +00:00
sfc-gh-tclinkenbeard 371a38e6e5 Merge remote-tracking branch 'origin/master' into remove-extra-copies 2021-06-07 10:26:06 -07:00
Xiaoxi Wang 85cd2b9945 add perpetualStorageWiggler 2021-05-20 23:31:08 +00:00
sfc-gh-tclinkenbeard f28ac955c3 Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>> 2021-05-10 16:32:50 -07:00
sfc-gh-tclinkenbeard 5c2d7b6080 Create RangeResult type alias 2021-05-03 13:14:16 -07:00
FDB Formatster df90cc89de apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-10 10:18:07 -08:00
Andrew Noyes 4ee97c0784 Use clang-tidy to automatically fix missing overrides
Use `clang-tidy -p . $file -checks='-*,modernize-use-override' -header-filter='.*' -fix`
to fix missing overrides, and then use git clang-format to reformat just
those changes. This went pretty well for most files.

Formatting the following files went off the rails, so I'm going to
follow up with a commit that's just clang-tidy and no clang-format.

- fdbclient/DatabaseBackupAgent.actor.cpp
- fdbclient/FileBackupAgent.actor.cpp
- fdbserver/OldTLogServer_4_6.actor.cpp
- fdbmonitor/SimpleIni.h
- fdbserver/workloads/ClientTransactionProfileCorrectness.actor.cpp
2021-01-26 02:04:12 +00:00
Andrew Noyes 7fbc4d7391 Resolve conflicts 2020-12-04 23:58:42 +00:00
Andrew Noyes 877997632d Merge branch 'release-6.3' into anoyes/merge-release-6.3-master
Include conflict markers for review purposes
2020-12-04 01:38:07 +00:00
Andrew Noyes dc2bac5670 Resolve conflicts 2020-11-24 19:09:42 +00:00
Andrew Noyes 1f541f02be Merge branch 'anoyes/merge-6.2-to-6.3' into anoyes/release-6.3-merge
Merge, leaving conflict markers for now
2020-11-24 16:55:34 +00:00
David Youngworth d64cf8b9e3 Merge branch 6.3 into master 2020-11-17 11:22:45 -08:00
David Youngworth 489ba20641 Fix several merge issues 2020-11-16 14:46:36 -08:00
David Youngworth d0391db862 Merge branch 'release-6.2' into release-6.3 2020-11-16 10:15:23 -08:00
Xiaoge Su 15d10d68cd Cleanup DataDistributionQueue.actor.cpp and storageserver.actor.cpp 2020-11-09 15:23:51 -08:00
Xin Dong 2909e35415 When moving dispaching shard relocations, log the source and destination storage servers. 2020-11-05 16:13:18 -08:00
Xin Dong cedcfc7d87 Removed unused uid field inside ParallelTCInfo 2020-10-26 16:48:09 -07:00
Xin Dong 9ef29d0cea Changed getTeamID() to return a string instead of UID as suggested by reviews. 2020-10-26 16:44:52 -07:00
Xin Dong 7ebb2e5c09 Piggy back this PR to polish more TraceEvent by:
- Making it clear that it's tracking machine team info or server team info
- Added ID to both machine team and server team for better trackability
- Attach distributor id to some trace events.
2020-10-26 16:44:09 -07:00
sfc-gh-tclinkenbeard e0b1f95740 Merge remote-tracking branch 'origin/master' into remove-global-ddenabled-flag 2020-10-21 18:22:08 -07:00
Young Liu 8cc3e4d3c6 Merge release-6.3 into master 2020-10-19 22:51:56 -07:00
sfc-gh-tclinkenbeard 652d753daf Remove global ddEnabled flag 2020-10-17 11:23:52 -07:00
A.J. Beamon 3b66a1f2d4 Fix a couple places where we were creating vectors with default elements rather than reserving space. 2020-10-09 10:51:06 -07:00
sfc-gh-tclinkenbeard a9607bdcec Explicitly seal classes that inherit but aren't inherited from 2020-10-07 21:58:24 -07:00
sfc-gh-tclinkenbeard 8571dcfe28 Use override where applicable in fdbserver 2020-10-07 18:41:19 -07:00
Evan Tschannen a49cb41de7 Merge branch 'release-6.3'
# Conflicts:
#	CMakeLists.txt
#	cmake/ConfigureCompiler.cmake
#	fdbserver/Knobs.cpp
#	fdbserver/StorageCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	flow/ThreadHelper.actor.h
#	flow/serialize.h
#	tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
Russell Sears ab0d8b0626
Merge pull request #3509 from sfc-gh-anoyes/anoyes/remove-using-relops
Remove using namespace std::rel_ops
2020-07-22 11:58:25 -07:00
sfc-gh-tclinkenbeard 83c5a30f62 Add encapsulation to TCTeamInfo and ParallelTCInfo 2020-07-21 11:05:41 -07:00
sfc-gh-tclinkenbeard 9a2ce4c981 Make IDataDistributionTeam const-correct 2020-07-21 11:05:34 -07:00
Evan Tschannen 7affda3c8b Merge branch 'release-6.3'
# Conflicts:
#	CMakeLists.txt
2020-07-14 14:57:24 -07:00
Meng Xu 815e403009 Add comment to DDQueue GetTeam
Comments to help understand Evan's PR 3487
that fix the problem: replication factor could drop unexpected in fearless config.
2020-07-13 17:05:50 -07:00
Meng Xu 47ae66bd61 Merge branch 'master' into mengxu/tmp-minor-comment-PR
Resolve conflict at waitFailureClient
2020-07-13 16:17:50 -07:00
Meng Xu efc446f529 Add a comment to DD 2020-07-13 10:12:39 -07:00
Meng Xu 6f2e12be42 Minor improvement on comments 2020-07-12 18:32:47 -07:00
Andrew Noyes f470ba8316 Remove using namespace std::rel_ops
This causes the following to not compile anymore

\#include <utility>
\#include <vector>

using namespace std::rel_ops;

int main() {
    std::vector<int> xs;
    return xs.rbegin() != xs.rend();
}

See https://godbolt.org/z/s1977n
2020-07-10 22:58:15 +00:00
A.J. Beamon b09dddc07e Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
# Conflicts:
#	cmake/ConfigureCompiler.cmake
#	documentation/sphinx/source/downloads.rst
#	fdbrpc/FlowTransport.actor.cpp
#	fdbrpc/fdbrpc.vcxproj
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/LogSystemPeekCursor.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/Status.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	flow/flow.vcxproj
2020-07-10 15:06:34 -07:00
Evan Tschannen 0e2f5e8bb5 Added a flow lock to prevent too many source server fetches from happening at the same time and running the data distributor out of memory 2020-07-09 10:38:19 -07:00
Evan Tschannen 5e02fd490e fix: the check for if a teamCollection was tracking a source server was unreliable, leading to scenarios where we would temporarily replicate a shard less than teamSIze 2020-06-29 10:02:27 -07:00
Evan Tschannen 48b1b20f67 Fixed a crash related to destruction order in data distribution 2020-05-10 23:14:19 -07:00
A.J. Beamon b1172417f5 Merge branch 'master' into per-priority-busy-logging
# Conflicts:
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/Net2.actor.cpp
2020-04-14 14:22:12 -07:00
Alex Miller 1439de37b5 Convert GetRangeLimits() -> TOO_MANY + ASSERT(). 2020-04-12 18:23:14 -07:00
Alex Miller 6078fd1b18 Convert UID to Tag in keyServers to reduce txnStateStore size 2020-04-05 14:30:09 -07:00
Evan Tschannen e08f0201f1 merge release 6.2 into master 2020-03-17 12:51:47 -07:00
Evan Tschannen 12f2b32770 added additional logging in data distribution 2020-03-13 15:19:33 -07:00
Evan Tschannen 303df197cf Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	bindings/c/test/mako/mako.c
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/NativeAPI.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/LogRouter.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/fdbserver.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/flow.vcxproj
#	flow/flow.vcxproj.filters
#	versions.target
2020-03-06 18:22:46 -08:00
Evan Tschannen e219c1671f Merge branch 'release-6.2' into feature-dd-region-queue
# Conflicts:
#	fdbserver/Knobs.h
2020-03-04 16:25:38 -08:00
Evan Tschannen 6d6f184e2f added a knob which reverts the new queue behavior 2020-03-04 16:23:49 -08:00
Evan Tschannen b7834b2995
Merge pull request #2774 from etschannen/feature-dd-repopulate-priority
Make the DD priority of populating a region lower than machine failures
2020-03-04 16:15:18 -08:00
Evan Tschannen 125bd13198 fix: in multi-region configurations, the data distribution queue could start too much work, expecting that the remote region would contribute to the read workload 2020-03-04 14:17:17 -08:00
Evan Tschannen 6296465e07 Make the DD priority associated with populating a remote region lower than machine failures 2020-03-04 14:07:32 -08:00
Meng Xu 1ef4cb432b Merge branch 'master' into mengxu/fast-restore-robust-and-visibility-PR-v2 2020-03-01 20:08:07 -08:00
Meng Xu ad9b3fb4a8 DD:Add trace for detailed relocate shard info 2020-02-29 13:45:10 -08:00
Meng Xu 2657d41bb2 FastRestore:Add debug msg when memory is over threshold 2020-02-27 18:32:11 -08:00
Alvin Moore 0f64505d0b Merge branch 'release-6.2' of github.com:apple/foundationdb
Needed to pull in changes to build docker
2020-02-23 23:27:53 -08:00
Evan Tschannen 96258b9809 Merge branch 'release-6.2'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/FlowTransport.actor.cpp
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/StorageMetrics.actor.h
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/KVStoreTest.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/genericactors.actor.cpp
#	flow/serialize.h
2020-02-21 19:09:16 -08:00
A.J. Beamon 4c696d5bf2 Merge branch 'release-6.2' into dd-better-rebalance-logging
# Conflicts:
#	fdbserver/DataDistributionQueue.actor.cpp
2020-02-21 17:41:00 -08:00
A.J. Beamon dfa5f76c01 Remove unused parameter. Don't put check for g_network presence in ASSERT_WE_THINK. 2020-02-21 16:28:03 -08:00
Evan Tschannen 08914a2acd Once available space ratio falls below 0.3 avoid moving data to teams with less free space than the median team 2020-02-21 15:14:32 -08:00
A.J. Beamon 2e699fef55 Don't suppress actor cancellation because we've already initialized the trace event by adding details. 2020-02-21 11:28:59 -08:00
A.J. Beamon 6810a03283 Add more logging to valley filler and mountain chopper 2020-02-21 10:55:14 -08:00
Evan Tschannen 819c55556c More aggressively attempt to find teams that do not have low disk space 2020-02-20 16:47:50 -08:00
A.J. Beamon e1fb568fd1 Merge branch 'release-6.2' into dd-use-available-space
# Conflicts:
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
2020-02-20 16:12:42 -08:00
A.J. Beamon e4b483796d Combine some logic that was doing similar computations for free space ratio. 2020-02-20 14:52:08 -08:00
A.J. Beamon 4c9c736253 Data distribution uses available space instead of free space when evaluating whether processes are low on space and penalizing them. 2020-02-20 11:21:03 -08:00
A.J. Beamon c164acb88d Add new criteria to DD's GetTeamRequest that allow you to require shards be present on the team and that the team have a minimum free ratio. This avoids scenarios where the team chosen when processing the request is later rejected by the requestor, causing rebalancing movements to get stuck. 2020-02-20 09:32:00 -08:00
A.J. Beamon abb75f7eb7 Add logging to indicate the time spent at each priority that exceeds some minimum busyness threshold 2020-02-07 14:34:24 -08:00
Evan Tschannen 3f9d9d8b84 Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	cmake/FlowCommands.cmake
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/StorageServerInterface.h
#	fdbserver/DataDistributionTracker.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	flow/Knobs.h
#	flow/Platform.cpp
#	versions.target
2020-01-16 18:37:47 -08:00
Evan Tschannen e65760eb46
Merge pull request #2536 from etschannen/feature-commit-latency
Improved commit latency in large clusters
2020-01-13 19:12:02 -08:00
Evan Tschannen ab7071932f Data distribution no longer attempts to pick teams which share members of the source unless the team matches exactly 2020-01-09 16:59:37 -08:00
Evan Tschannen 3157d8a375 fixed typo 2019-12-18 16:57:39 -08:00
Evan Tschannen 688940b685 merge 6.2 into master 2019-10-21 11:43:46 -07:00
Evan Tschannen 5667331729 added a buggify + minor code cleanup 2019-10-11 18:31:43 -07:00
Evan Tschannen 86bcb84b45 Raised the data distribution priority of splitting shards above restoring fault tolerance to avoid hot write shards 2019-10-11 17:50:43 -07:00
A.J. Beamon 22c3fa867c
Merge pull request #2074 from xumengpanda/mengxu/fix-correctness-bug
DD:fix:getTeam may fail to get a team when it can
2019-10-07 09:33:57 -07:00
Meng Xu 3c4dc1003d DD:clang-format the PR 2019-09-11 11:16:29 -07:00
Meng Xu a83afa5f64 DD:BgDDValleyFiller:Do not move shard to a team with no healthy space 2019-09-11 11:15:00 -07:00
Meng Xu bd80a67d46 Merge branch 'master' into mengxu/storage-engine-switch-PR-v2 2019-09-03 14:11:33 -07:00
Meng Xu a377261740 StorageEngineSwitch:Remove questions in comments 2019-08-22 11:49:39 -07:00
Evan Tschannen ac68c8e4fd added sources servers to the warning message 2019-08-21 11:48:29 -07:00
Meng Xu 39680fa515 StorageEngineSwitch:Clean up unnecessary trace
And do not trigger storage recruitment unnecessarily.
2019-08-19 14:11:57 -07:00
Evan Tschannen d30d4cb955 Added a duration to regular relocateShard trace events 2019-08-16 15:15:36 -07:00
Evan Tschannen 297b65236f added additional trace events to warn when different parts of shard relocations take more than 10 minutes 2019-08-16 14:56:58 -07:00
Meng Xu a588710376 StorageEngineSwitch:Graceful switch
When fdbcli change storeType for storage engines,
we switch the store type of storage servers one by one gracefully.
This avoids recruiting multiple storage servers on the same process,
which can cause OOM error.
2019-08-12 17:37:52 -07:00
Xin Dong b653ddb30d Final clean ups after rebasing master 2019-07-30 22:35:34 -07:00
Xin Dong cda70700cc Address review comments. 50K correctness with no failures. 2019-07-30 22:24:30 -07:00
Xin Dong 5d20364423 Address review comments 2019-07-30 22:24:30 -07:00
Xin Dong 1922c39377 Resolve review comments. 100K run shows one suspecious ASSERT_WE_THINK failure which I think could be a race. 2019-07-30 22:24:30 -07:00
Xin Dong f5d6e3a5b3 - Addressed review commends
- Added test for the storage server failure disable switch
2019-07-30 22:20:45 -07:00
Xin Dong 4ecfc9830f Added finer grained controls to DataDistribution in fdbcli. What's happening under the hood is:
- Use pre-existing 'healthZone' key and write a special value to it in order to disable DD for all storage server failures
- Use a new system key 'rebalanceDDIgnored' key to disable/enable DD for all rebalance reasons(MountainChopper and ValleyFiller)

Kicked off two 200K correctness and showed no related errors.
2019-07-30 22:17:21 -07:00
Evan Tschannen a78a97f186
Merge pull request #1908 from etschannen/feature-better-dd
A few data distribution improvements
2019-07-30 17:34:50 -07:00
sramamoorthy 63941e0d96 disable DD with a in-memory flag and use in snapv2 2019-07-30 17:04:51 -07:00
Evan Tschannen 5dd9043fd3 addressed review comments 2019-07-30 17:04:41 -07:00
Evan Tschannen 481642fbd4 Merge branch 'master' into feature-better-dd 2019-07-30 16:56:27 -07:00
A.J. Beamon 41605735f5
Merge pull request #1916 from ajbeamon/merge-onto-new-servers
Add knob to control whether merges request new servers or not.
2019-07-30 15:04:37 -07:00
A.J. Beamon 14648e20f9
Merge pull request #1901 from ajbeamon/data-distribution-receives-bytes-input-rate
Send bytes input rate to data distribution
2019-07-30 15:01:36 -07:00
A.J. Beamon bc536757df Add knob to control whether merges request new servers or not. Set the default to request new servers in \xff but not in main key space. 2019-07-29 15:47:34 -07:00
Evan Tschannen 6b5e683de5 The mountainChopper and valleyFiller only move larger than average shards, to avoid moving high bandwidth shards which are generally smaller. 2019-07-28 23:50:42 -07:00
Evan Tschannen 04dd293af0
Merge pull request #1874 from xumengpanda/mengxu/DD-code-read
DataDistribution:Add comments to help understand the code
2019-07-26 13:30:44 -07:00
A.J. Beamon b91795d288 Send bytes input rate to DD. 2019-07-25 16:27:32 -07:00
Meng Xu e582219ec5 Remove unnecessary condition in DDQueue
Resolve the review comment.
2019-07-22 17:00:37 -07:00
Meng Xu b7478f5dd3 DD:Add comments to help understand code
Add comments to explain the functionalities of some code.
2019-07-22 11:23:16 -07:00
Meng Xu 612a51fe00 Apply Clang format to PRIORITY_TEAM_REDUNDANT 2019-07-19 18:32:22 -07:00
Meng Xu ea76451f15 Count PRIORITY_TEAM_REDUNDANT as count PRIORITY_TEAM_UNHEALTHY 2019-07-19 18:30:01 -07:00
Alex Miller 7a500cd37f A giant translation of TaskFooPriority -> TaskPriority::Foo
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
A.J. Beamon 5f55f3f613 Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used. 2019-05-10 14:01:52 -07:00
Evan Tschannen 2d5043c665 Merge branch 'release-6.1'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	versions.target
2019-04-30 18:27:04 -07:00
Evan Tschannen e0f7ec96aa Data distribution needs to build new teams as old teams are removed to ensure data remains balanced across servers 2019-04-22 17:29:46 -07:00
mpilman d01cbf3455 Addressed code review comments 2019-04-05 13:12:20 -07:00
mpilman 1c16f87a4e Remove trace-calls to printable (in non-workloads) 2019-04-05 13:12:19 -07:00
anoyes 981426bac9 More ide fixes 2019-03-05 18:03:57 -08:00
Evan Tschannen d008de576e
Merge pull request #1139 from xumengpanda/mengxu/machine-team-upgrade-PR
Add background actor to remove redundant teams
2019-02-22 14:22:07 -08:00
Meng Xu 9445ac0b0c Status: Use new data distributor worker to publish status
After we add a new data distributor role, we publish the data
related to data distributor and rate keeper through the new
role (and new worker).

So the status needs to contact the data distributor, instead of master,
to get the status information.
2019-02-21 18:05:50 -08:00
Meng Xu 7cca439e00 TeamRemover: Add status to show redundant team removing
Distinguish the removal of unhealthy team and redundant team.
Change status report to include redundant team removal report.
2019-02-21 14:16:46 -08:00
mpilman 27a3153719 Use ACTOR forward declarations in MoveKeys
Also MoveKeys.h -> MoveKeys.actor.h
2019-02-19 15:16:59 -08:00
mpilman 3a0f9839b9 Fix minor IDE build errors 2019-02-19 15:16:59 -08:00
Meng Xu 6d09ac483c Merge with master 2019-02-15 17:03:40 -08:00
Jingyu Zhou bf6da81bf9 Remove recovery version from data distribution queue
This parameter is no longer used/needed.
2019-02-14 16:37:16 -08:00
Jingyu Zhou 07dab56133 Fix a data movement stuck bug
When moving keys to a team, if one of the server in the target team died, then
the move can become stuck. This is because the DDTeamCollection waits for all
the data movement of the failed server to be completed. However, in this case,
because the movement has not finished yet, checking the database tells us there
is no key assocated with this server and it is safe to go ahead. In reality,
only the in-memory structure knows there is pending movement, i.e., unfinished
move causes some keys to be attributed to the failed server. Thus, the server
can't be removed yet. Fix by adding a check with in-memory structure in
waitForAllDataRemoved().

Use const& to optimize a few function parameters.
2019-02-14 16:37:16 -08:00
Jingyu Zhou 886e7ab2ba Add a new DataDistributor role.
Let cluster controller to start a new data distributor role by sending a
message to a chosen worker.
Change MasterInterface usage in DataDistribution to masterId

Add DataDistributor rejoin handling.

This allows the data distributor to tell the new cluster controller of its
existence so that the controller doesn't spawn a new one. I.e., there should
be only ONE data distributor in the cluster.

If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries
to recruit one as DD. CC also monitors DD and restarts one if it failed.

The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for
the new DD.

Add GetRecoveryInfo RPC to master server, which is called by data distributor
to obtain the recovery Transaction version from the master server.
2019-02-14 16:30:13 -08:00
Meng Xu fe4f43203d TeamCollection: getTeam may add a new team
getTeam function may add a new team for the GetTeamRequest.
We need to check if the number of teams is larger than the desired team number.
2019-02-12 14:57:35 -08:00
Meng Xu 8de031f9a6 TeamCollection: clang-format
Format the changes with git clang-format.
No functional changes.

Signed-off-by: Meng Xu <meng_xu@apple.com>
2018-11-21 11:18:26 -08:00
Meng Xu f7a7e069f0 TeamCollection: Remove unnecessary comments
Pass 41806 tests with no failure

Signed-off-by: Meng Xu <meng_xu@apple.com>
2018-11-16 15:56:35 -08:00