Xiaoxi Wang
4b92f8f546
add relocate reason and set teamSorter in relocator
2022-03-18 16:39:31 -07:00
sfc-gh-tclinkenbeard
4a0720e8ad
Provide destTeam parameter to rebalanceTeams again.
...
There is a wait before the load bytes of the dest team is used, so the
last commit inadvertently caused a behaviour change. Instead, update the
comment for the function, and pass const IDataDistributionTeam
references.
2022-03-18 10:25:13 -07:00
Josh Slocum
5fbdbb5c4a
Better cancelling logic that reflects whether move has actually started
2022-03-18 10:43:24 -05:00
sfc-gh-tclinkenbeard
a04ca65ad5
Change signature of rebalanceTeams
2022-03-17 23:27:33 -07:00
Josh Slocum
67eba5ec7c
Limiting DD Moves by destination SS.
2022-03-15 13:52:19 -05:00
Xiaoxi Wang
10eb082888
format code; fix DD ignore option bug"
2022-03-15 10:31:43 -07:00
Xiaoxi Wang
d38b827906
fix DD ignore bug
2022-03-13 23:47:07 -07:00
Xiaoxi Wang
8d20ee8432
change storage metrics of read sample calculation
2022-03-07 17:36:43 -08:00
Xiaoxi Wang
970a84bf63
fix teamSorter setting
2022-03-03 12:58:45 -08:00
Xiaoxi Wang
a28b0f4361
Merge branch 'main' of https://github.com/apple/foundationdb into readaware
2022-03-03 11:15:18 -08:00
Xiaoxi Wang
a05f6f851a
fix reference assign bug
2022-03-02 21:56:03 -08:00
Xiaoxi Wang
f48ba989c3
add status
2022-03-01 15:27:27 -08:00
Xiaoxi Wang
5e74b5006e
fix uninitialized member
2022-02-28 10:22:32 -08:00
Xiaoxi Wang
40a1f562a7
temporary change special key for data distributor
2022-02-25 11:01:23 -08:00
Xiaoxi Wang
a07e413eae
metrics comparator; rebalanceReadLoad()
2022-02-24 16:41:01 -08:00
A.J. Beamon
250a88e682
Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement.
2022-02-24 12:25:52 -08:00
Evan Tschannen
1afef44f19
fix: self->shardsAffectedByTeamFailure->moveShard must be called without any waits after getting the destination team or we could miss failure notifications for the storage servers in the destination team
2021-11-23 15:26:38 -08:00
Xiaoxi Wang
d1959122af
consider wiggling when waitUntilHealthy
2021-10-27 09:08:37 -07:00
Chang Liu
8761960cdc
fix roll trace event issue for data distribution(master)
...
Description
Testing
2021-09-24 15:40:17 -07:00
Chang Liu
814e3c729f
fix roll trace event issue for data distribution(master)
...
Description
Testing
2021-09-24 15:40:17 -07:00
Chang Liu
c10dd0df4b
fix roll trace event issue for data distribution
...
Description
Testing
2021-09-24 15:40:17 -07:00
Xiaoge Su
abf73047ca
Enforce std:: specifier rather than using namespace
2021-09-16 19:40:28 -07:00
sfc-gh-tclinkenbeard
3442ebd3b7
Fix more -Wreorder-ctor warnings across many files
2021-07-24 11:20:51 -07:00
Daniel Smith
8efe3b296a
Delete remaining extern declarations for noUnseed
2021-07-08 19:19:22 -04:00
Xiaoxi Wang
0493d149e6
wait remove
2021-06-21 05:18:42 +00:00
Xiaoxi Wang
d46fccc30f
Revert "Revert "Properly set simulation test for perpetual storage wiggle and bug fixing""
...
This reverts commit ad576e8c20 .
2021-06-11 22:58:05 +00:00
Xiaoxi Wang
ad576e8c20
Revert "Properly set simulation test for perpetual storage wiggle and bug fixing"
2021-06-11 09:07:45 -07:00
Xiaoxi Wang
17ac91bac4
Merge pull request #4929 from sfc-gh-xwang/ppwtest
...
Properly set simulation test for perpetual storage wiggle and bug fixing
2021-06-10 14:09:50 -07:00
Xiaoxi Wang
4220a548ce
use the same health check as exclude to avoid 'best team get stuck'
2021-06-09 22:51:46 +00:00
sfc-gh-tclinkenbeard
371a38e6e5
Merge remote-tracking branch 'origin/master' into remove-extra-copies
2021-06-07 10:26:06 -07:00
Xiaoxi Wang
85cd2b9945
add perpetualStorageWiggler
2021-05-20 23:31:08 +00:00
sfc-gh-tclinkenbeard
f28ac955c3
Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>>
2021-05-10 16:32:50 -07:00
sfc-gh-tclinkenbeard
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
FDB Formatster
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
Andrew Noyes
4ee97c0784
Use clang-tidy to automatically fix missing overrides
...
Use `clang-tidy -p . $file -checks='-*,modernize-use-override' -header-filter='.*' -fix`
to fix missing overrides, and then use git clang-format to reformat just
those changes. This went pretty well for most files.
Formatting the following files went off the rails, so I'm going to
follow up with a commit that's just clang-tidy and no clang-format.
- fdbclient/DatabaseBackupAgent.actor.cpp
- fdbclient/FileBackupAgent.actor.cpp
- fdbserver/OldTLogServer_4_6.actor.cpp
- fdbmonitor/SimpleIni.h
- fdbserver/workloads/ClientTransactionProfileCorrectness.actor.cpp
2021-01-26 02:04:12 +00:00
Andrew Noyes
7fbc4d7391
Resolve conflicts
2020-12-04 23:58:42 +00:00
Andrew Noyes
877997632d
Merge branch 'release-6.3' into anoyes/merge-release-6.3-master
...
Include conflict markers for review purposes
2020-12-04 01:38:07 +00:00
Andrew Noyes
dc2bac5670
Resolve conflicts
2020-11-24 19:09:42 +00:00
Andrew Noyes
1f541f02be
Merge branch 'anoyes/merge-6.2-to-6.3' into anoyes/release-6.3-merge
...
Merge, leaving conflict markers for now
2020-11-24 16:55:34 +00:00
David Youngworth
d64cf8b9e3
Merge branch 6.3 into master
2020-11-17 11:22:45 -08:00
David Youngworth
489ba20641
Fix several merge issues
2020-11-16 14:46:36 -08:00
David Youngworth
d0391db862
Merge branch 'release-6.2' into release-6.3
2020-11-16 10:15:23 -08:00
Xiaoge Su
15d10d68cd
Cleanup DataDistributionQueue.actor.cpp and storageserver.actor.cpp
2020-11-09 15:23:51 -08:00
Xin Dong
2909e35415
When moving dispaching shard relocations, log the source and destination storage servers.
2020-11-05 16:13:18 -08:00
Xin Dong
cedcfc7d87
Removed unused uid field inside ParallelTCInfo
2020-10-26 16:48:09 -07:00
Xin Dong
9ef29d0cea
Changed getTeamID() to return a string instead of UID as suggested by reviews.
2020-10-26 16:44:52 -07:00
Xin Dong
7ebb2e5c09
Piggy back this PR to polish more TraceEvent by:
...
- Making it clear that it's tracking machine team info or server team info
- Added ID to both machine team and server team for better trackability
- Attach distributor id to some trace events.
2020-10-26 16:44:09 -07:00
sfc-gh-tclinkenbeard
e0b1f95740
Merge remote-tracking branch 'origin/master' into remove-global-ddenabled-flag
2020-10-21 18:22:08 -07:00
Young Liu
8cc3e4d3c6
Merge release-6.3 into master
2020-10-19 22:51:56 -07:00
sfc-gh-tclinkenbeard
652d753daf
Remove global ddEnabled flag
2020-10-17 11:23:52 -07:00
A.J. Beamon
3b66a1f2d4
Fix a couple places where we were creating vectors with default elements rather than reserving space.
2020-10-09 10:51:06 -07:00
sfc-gh-tclinkenbeard
a9607bdcec
Explicitly seal classes that inherit but aren't inherited from
2020-10-07 21:58:24 -07:00
sfc-gh-tclinkenbeard
8571dcfe28
Use override where applicable in fdbserver
2020-10-07 18:41:19 -07:00
Evan Tschannen
a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
Russell Sears
ab0d8b0626
Merge pull request #3509 from sfc-gh-anoyes/anoyes/remove-using-relops
...
Remove using namespace std::rel_ops
2020-07-22 11:58:25 -07:00
sfc-gh-tclinkenbeard
83c5a30f62
Add encapsulation to TCTeamInfo and ParallelTCInfo
2020-07-21 11:05:41 -07:00
sfc-gh-tclinkenbeard
9a2ce4c981
Make IDataDistributionTeam const-correct
2020-07-21 11:05:34 -07:00
Evan Tschannen
7affda3c8b
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
2020-07-14 14:57:24 -07:00
Meng Xu
815e403009
Add comment to DDQueue GetTeam
...
Comments to help understand Evan's PR 3487
that fix the problem: replication factor could drop unexpected in fearless config.
2020-07-13 17:05:50 -07:00
Meng Xu
47ae66bd61
Merge branch 'master' into mengxu/tmp-minor-comment-PR
...
Resolve conflict at waitFailureClient
2020-07-13 16:17:50 -07:00
Meng Xu
efc446f529
Add a comment to DD
2020-07-13 10:12:39 -07:00
Meng Xu
6f2e12be42
Minor improvement on comments
2020-07-12 18:32:47 -07:00
Andrew Noyes
f470ba8316
Remove using namespace std::rel_ops
...
This causes the following to not compile anymore
\#include <utility>
\#include <vector>
using namespace std::rel_ops;
int main() {
std::vector<int> xs;
return xs.rbegin() != xs.rend();
}
See https://godbolt.org/z/s1977n
2020-07-10 22:58:15 +00:00
A.J. Beamon
b09dddc07e
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# cmake/ConfigureCompiler.cmake
# documentation/sphinx/source/downloads.rst
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/fdbrpc.vcxproj
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/flow.vcxproj
2020-07-10 15:06:34 -07:00
Evan Tschannen
0e2f5e8bb5
Added a flow lock to prevent too many source server fetches from happening at the same time and running the data distributor out of memory
2020-07-09 10:38:19 -07:00
Evan Tschannen
5e02fd490e
fix: the check for if a teamCollection was tracking a source server was unreliable, leading to scenarios where we would temporarily replicate a shard less than teamSIze
2020-06-29 10:02:27 -07:00
Evan Tschannen
48b1b20f67
Fixed a crash related to destruction order in data distribution
2020-05-10 23:14:19 -07:00
A.J. Beamon
b1172417f5
Merge branch 'master' into per-priority-busy-logging
...
# Conflicts:
# flow/Knobs.cpp
# flow/Knobs.h
# flow/Net2.actor.cpp
2020-04-14 14:22:12 -07:00
Alex Miller
1439de37b5
Convert GetRangeLimits() -> TOO_MANY + ASSERT().
2020-04-12 18:23:14 -07:00
Alex Miller
6078fd1b18
Convert UID to Tag in keyServers to reduce txnStateStore size
2020-04-05 14:30:09 -07:00
Evan Tschannen
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
Evan Tschannen
12f2b32770
added additional logging in data distribution
2020-03-13 15:19:33 -07:00
Evan Tschannen
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
Evan Tschannen
e219c1671f
Merge branch 'release-6.2' into feature-dd-region-queue
...
# Conflicts:
# fdbserver/Knobs.h
2020-03-04 16:25:38 -08:00
Evan Tschannen
6d6f184e2f
added a knob which reverts the new queue behavior
2020-03-04 16:23:49 -08:00
Evan Tschannen
b7834b2995
Merge pull request #2774 from etschannen/feature-dd-repopulate-priority
...
Make the DD priority of populating a region lower than machine failures
2020-03-04 16:15:18 -08:00
Evan Tschannen
125bd13198
fix: in multi-region configurations, the data distribution queue could start too much work, expecting that the remote region would contribute to the read workload
2020-03-04 14:17:17 -08:00
Evan Tschannen
6296465e07
Make the DD priority associated with populating a remote region lower than machine failures
2020-03-04 14:07:32 -08:00
Meng Xu
1ef4cb432b
Merge branch 'master' into mengxu/fast-restore-robust-and-visibility-PR-v2
2020-03-01 20:08:07 -08:00
Meng Xu
ad9b3fb4a8
DD:Add trace for detailed relocate shard info
2020-02-29 13:45:10 -08:00
Meng Xu
2657d41bb2
FastRestore:Add debug msg when memory is over threshold
2020-02-27 18:32:11 -08:00
Alvin Moore
0f64505d0b
Merge branch 'release-6.2' of github.com:apple/foundationdb
...
Needed to pull in changes to build docker
2020-02-23 23:27:53 -08:00
Evan Tschannen
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
A.J. Beamon
4c696d5bf2
Merge branch 'release-6.2' into dd-better-rebalance-logging
...
# Conflicts:
# fdbserver/DataDistributionQueue.actor.cpp
2020-02-21 17:41:00 -08:00
A.J. Beamon
dfa5f76c01
Remove unused parameter. Don't put check for g_network presence in ASSERT_WE_THINK.
2020-02-21 16:28:03 -08:00
Evan Tschannen
08914a2acd
Once available space ratio falls below 0.3 avoid moving data to teams with less free space than the median team
2020-02-21 15:14:32 -08:00
A.J. Beamon
2e699fef55
Don't suppress actor cancellation because we've already initialized the trace event by adding details.
2020-02-21 11:28:59 -08:00
A.J. Beamon
6810a03283
Add more logging to valley filler and mountain chopper
2020-02-21 10:55:14 -08:00
Evan Tschannen
819c55556c
More aggressively attempt to find teams that do not have low disk space
2020-02-20 16:47:50 -08:00
A.J. Beamon
e1fb568fd1
Merge branch 'release-6.2' into dd-use-available-space
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
2020-02-20 16:12:42 -08:00
A.J. Beamon
e4b483796d
Combine some logic that was doing similar computations for free space ratio.
2020-02-20 14:52:08 -08:00
A.J. Beamon
4c9c736253
Data distribution uses available space instead of free space when evaluating whether processes are low on space and penalizing them.
2020-02-20 11:21:03 -08:00
A.J. Beamon
c164acb88d
Add new criteria to DD's GetTeamRequest that allow you to require shards be present on the team and that the team have a minimum free ratio. This avoids scenarios where the team chosen when processing the request is later rejected by the requestor, causing rebalancing movements to get stuck.
2020-02-20 09:32:00 -08:00
A.J. Beamon
abb75f7eb7
Add logging to indicate the time spent at each priority that exceeds some minimum busyness threshold
2020-02-07 14:34:24 -08:00
Evan Tschannen
3f9d9d8b84
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# cmake/FlowCommands.cmake
# documentation/sphinx/source/release-notes.rst
# fdbclient/StorageServerInterface.h
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# flow/Knobs.h
# flow/Platform.cpp
# versions.target
2020-01-16 18:37:47 -08:00
Evan Tschannen
e65760eb46
Merge pull request #2536 from etschannen/feature-commit-latency
...
Improved commit latency in large clusters
2020-01-13 19:12:02 -08:00
Evan Tschannen
ab7071932f
Data distribution no longer attempts to pick teams which share members of the source unless the team matches exactly
2020-01-09 16:59:37 -08:00
Evan Tschannen
3157d8a375
fixed typo
2019-12-18 16:57:39 -08:00
Evan Tschannen
688940b685
merge 6.2 into master
2019-10-21 11:43:46 -07:00
Evan Tschannen
5667331729
added a buggify + minor code cleanup
2019-10-11 18:31:43 -07:00
Evan Tschannen
86bcb84b45
Raised the data distribution priority of splitting shards above restoring fault tolerance to avoid hot write shards
2019-10-11 17:50:43 -07:00
A.J. Beamon
22c3fa867c
Merge pull request #2074 from xumengpanda/mengxu/fix-correctness-bug
...
DD:fix:getTeam may fail to get a team when it can
2019-10-07 09:33:57 -07:00
Meng Xu
3c4dc1003d
DD:clang-format the PR
2019-09-11 11:16:29 -07:00
Meng Xu
a83afa5f64
DD:BgDDValleyFiller:Do not move shard to a team with no healthy space
2019-09-11 11:15:00 -07:00
Meng Xu
bd80a67d46
Merge branch 'master' into mengxu/storage-engine-switch-PR-v2
2019-09-03 14:11:33 -07:00
Meng Xu
a377261740
StorageEngineSwitch:Remove questions in comments
2019-08-22 11:49:39 -07:00
Evan Tschannen
ac68c8e4fd
added sources servers to the warning message
2019-08-21 11:48:29 -07:00
Meng Xu
39680fa515
StorageEngineSwitch:Clean up unnecessary trace
...
And do not trigger storage recruitment unnecessarily.
2019-08-19 14:11:57 -07:00
Evan Tschannen
d30d4cb955
Added a duration to regular relocateShard trace events
2019-08-16 15:15:36 -07:00
Evan Tschannen
297b65236f
added additional trace events to warn when different parts of shard relocations take more than 10 minutes
2019-08-16 14:56:58 -07:00
Meng Xu
a588710376
StorageEngineSwitch:Graceful switch
...
When fdbcli change storeType for storage engines,
we switch the store type of storage servers one by one gracefully.
This avoids recruiting multiple storage servers on the same process,
which can cause OOM error.
2019-08-12 17:37:52 -07:00
Xin Dong
b653ddb30d
Final clean ups after rebasing master
2019-07-30 22:35:34 -07:00
Xin Dong
cda70700cc
Address review comments. 50K correctness with no failures.
2019-07-30 22:24:30 -07:00
Xin Dong
5d20364423
Address review comments
2019-07-30 22:24:30 -07:00
Xin Dong
1922c39377
Resolve review comments. 100K run shows one suspecious ASSERT_WE_THINK failure which I think could be a race.
2019-07-30 22:24:30 -07:00
Xin Dong
f5d6e3a5b3
- Addressed review commends
...
- Added test for the storage server failure disable switch
2019-07-30 22:20:45 -07:00
Xin Dong
4ecfc9830f
Added finer grained controls to DataDistribution in fdbcli. What's happening under the hood is:
...
- Use pre-existing 'healthZone' key and write a special value to it in order to disable DD for all storage server failures
- Use a new system key 'rebalanceDDIgnored' key to disable/enable DD for all rebalance reasons(MountainChopper and ValleyFiller)
Kicked off two 200K correctness and showed no related errors.
2019-07-30 22:17:21 -07:00
Evan Tschannen
a78a97f186
Merge pull request #1908 from etschannen/feature-better-dd
...
A few data distribution improvements
2019-07-30 17:34:50 -07:00
sramamoorthy
63941e0d96
disable DD with a in-memory flag and use in snapv2
2019-07-30 17:04:51 -07:00
Evan Tschannen
5dd9043fd3
addressed review comments
2019-07-30 17:04:41 -07:00
Evan Tschannen
481642fbd4
Merge branch 'master' into feature-better-dd
2019-07-30 16:56:27 -07:00
A.J. Beamon
41605735f5
Merge pull request #1916 from ajbeamon/merge-onto-new-servers
...
Add knob to control whether merges request new servers or not.
2019-07-30 15:04:37 -07:00
A.J. Beamon
14648e20f9
Merge pull request #1901 from ajbeamon/data-distribution-receives-bytes-input-rate
...
Send bytes input rate to data distribution
2019-07-30 15:01:36 -07:00
A.J. Beamon
bc536757df
Add knob to control whether merges request new servers or not. Set the default to request new servers in \xff but not in main key space.
2019-07-29 15:47:34 -07:00
Evan Tschannen
6b5e683de5
The mountainChopper and valleyFiller only move larger than average shards, to avoid moving high bandwidth shards which are generally smaller.
2019-07-28 23:50:42 -07:00
Evan Tschannen
04dd293af0
Merge pull request #1874 from xumengpanda/mengxu/DD-code-read
...
DataDistribution:Add comments to help understand the code
2019-07-26 13:30:44 -07:00
A.J. Beamon
b91795d288
Send bytes input rate to DD.
2019-07-25 16:27:32 -07:00
Meng Xu
e582219ec5
Remove unnecessary condition in DDQueue
...
Resolve the review comment.
2019-07-22 17:00:37 -07:00
Meng Xu
b7478f5dd3
DD:Add comments to help understand code
...
Add comments to explain the functionalities of some code.
2019-07-22 11:23:16 -07:00
Meng Xu
612a51fe00
Apply Clang format to PRIORITY_TEAM_REDUNDANT
2019-07-19 18:32:22 -07:00
Meng Xu
ea76451f15
Count PRIORITY_TEAM_REDUNDANT as count PRIORITY_TEAM_UNHEALTHY
2019-07-19 18:30:01 -07:00
Alex Miller
7a500cd37f
A giant translation of TaskFooPriority -> TaskPriority::Foo
...
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
A.J. Beamon
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
Evan Tschannen
2d5043c665
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-30 18:27:04 -07:00
Evan Tschannen
e0f7ec96aa
Data distribution needs to build new teams as old teams are removed to ensure data remains balanced across servers
2019-04-22 17:29:46 -07:00
mpilman
d01cbf3455
Addressed code review comments
2019-04-05 13:12:20 -07:00
mpilman
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
anoyes
981426bac9
More ide fixes
2019-03-05 18:03:57 -08:00
Evan Tschannen
d008de576e
Merge pull request #1139 from xumengpanda/mengxu/machine-team-upgrade-PR
...
Add background actor to remove redundant teams
2019-02-22 14:22:07 -08:00
Meng Xu
9445ac0b0c
Status: Use new data distributor worker to publish status
...
After we add a new data distributor role, we publish the data
related to data distributor and rate keeper through the new
role (and new worker).
So the status needs to contact the data distributor, instead of master,
to get the status information.
2019-02-21 18:05:50 -08:00
Meng Xu
7cca439e00
TeamRemover: Add status to show redundant team removing
...
Distinguish the removal of unhealthy team and redundant team.
Change status report to include redundant team removal report.
2019-02-21 14:16:46 -08:00
mpilman
27a3153719
Use ACTOR forward declarations in MoveKeys
...
Also MoveKeys.h -> MoveKeys.actor.h
2019-02-19 15:16:59 -08:00
mpilman
3a0f9839b9
Fix minor IDE build errors
2019-02-19 15:16:59 -08:00
Meng Xu
6d09ac483c
Merge with master
2019-02-15 17:03:40 -08:00
Jingyu Zhou
bf6da81bf9
Remove recovery version from data distribution queue
...
This parameter is no longer used/needed.
2019-02-14 16:37:16 -08:00
Jingyu Zhou
07dab56133
Fix a data movement stuck bug
...
When moving keys to a team, if one of the server in the target team died, then
the move can become stuck. This is because the DDTeamCollection waits for all
the data movement of the failed server to be completed. However, in this case,
because the movement has not finished yet, checking the database tells us there
is no key assocated with this server and it is safe to go ahead. In reality,
only the in-memory structure knows there is pending movement, i.e., unfinished
move causes some keys to be attributed to the failed server. Thus, the server
can't be removed yet. Fix by adding a check with in-memory structure in
waitForAllDataRemoved().
Use const& to optimize a few function parameters.
2019-02-14 16:37:16 -08:00
Jingyu Zhou
886e7ab2ba
Add a new DataDistributor role.
...
Let cluster controller to start a new data distributor role by sending a
message to a chosen worker.
Change MasterInterface usage in DataDistribution to masterId
Add DataDistributor rejoin handling.
This allows the data distributor to tell the new cluster controller of its
existence so that the controller doesn't spawn a new one. I.e., there should
be only ONE data distributor in the cluster.
If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries
to recruit one as DD. CC also monitors DD and restarts one if it failed.
The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for
the new DD.
Add GetRecoveryInfo RPC to master server, which is called by data distributor
to obtain the recovery Transaction version from the master server.
2019-02-14 16:30:13 -08:00
Meng Xu
fe4f43203d
TeamCollection: getTeam may add a new team
...
getTeam function may add a new team for the GetTeamRequest.
We need to check if the number of teams is larger than the desired team number.
2019-02-12 14:57:35 -08:00
Meng Xu
8de031f9a6
TeamCollection: clang-format
...
Format the changes with git clang-format.
No functional changes.
Signed-off-by: Meng Xu <meng_xu@apple.com>
2018-11-21 11:18:26 -08:00
Meng Xu
f7a7e069f0
TeamCollection: Remove unnecessary comments
...
Pass 41806 tests with no failure
Signed-off-by: Meng Xu <meng_xu@apple.com>
2018-11-16 15:56:35 -08:00