a04ca65ad5
Change signature of rebalanceTeams
2022-03-17 23:27:33 -07:00
67eba5ec7c
Limiting DD Moves by destination SS.
2022-03-15 13:52:19 -05:00
10eb082888
format code; fix DD ignore option bug"
2022-03-15 10:31:43 -07:00
d38b827906
fix DD ignore bug
2022-03-13 23:47:07 -07:00
8d20ee8432
change storage metrics of read sample calculation
2022-03-07 17:36:43 -08:00
970a84bf63
fix teamSorter setting
2022-03-03 12:58:45 -08:00
a28b0f4361
Merge branch 'main' of https://github.com/apple/foundationdb into readaware
2022-03-03 11:15:18 -08:00
a05f6f851a
fix reference assign bug
2022-03-02 21:56:03 -08:00
f48ba989c3
add status
2022-03-01 15:27:27 -08:00
5e74b5006e
fix uninitialized member
2022-02-28 10:22:32 -08:00
40a1f562a7
temporary change special key for data distributor
2022-02-25 11:01:23 -08:00
a07e413eae
metrics comparator; rebalanceReadLoad()
2022-02-24 16:41:01 -08:00
250a88e682
Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement.
2022-02-24 12:25:52 -08:00
1afef44f19
fix: self->shardsAffectedByTeamFailure->moveShard must be called without any waits after getting the destination team or we could miss failure notifications for the storage servers in the destination team
2021-11-23 15:26:38 -08:00
d1959122af
consider wiggling when waitUntilHealthy
2021-10-27 09:08:37 -07:00
8761960cdc
fix roll trace event issue for data distribution(master)
...
Description
Testing
2021-09-24 15:40:17 -07:00
814e3c729f
fix roll trace event issue for data distribution(master)
...
Description
Testing
2021-09-24 15:40:17 -07:00
c10dd0df4b
fix roll trace event issue for data distribution
...
Description
Testing
2021-09-24 15:40:17 -07:00
abf73047ca
Enforce std:: specifier rather than using namespace
2021-09-16 19:40:28 -07:00
3442ebd3b7
Fix more -Wreorder-ctor warnings across many files
2021-07-24 11:20:51 -07:00
8efe3b296a
Delete remaining extern declarations for noUnseed
2021-07-08 19:19:22 -04:00
0493d149e6
wait remove
2021-06-21 05:18:42 +00:00
d46fccc30f
Revert "Revert "Properly set simulation test for perpetual storage wiggle and bug fixing""
...
This reverts commit ad576e8c20
2021-06-11 22:58:05 +00:00
ad576e8c20
Revert "Properly set simulation test for perpetual storage wiggle and bug fixing"
2021-06-11 09:07:45 -07:00
17ac91bac4
Merge pull request #4929 from sfc-gh-xwang/ppwtest
...
Properly set simulation test for perpetual storage wiggle and bug fixing
2021-06-10 14:09:50 -07:00
4220a548ce
use the same health check as exclude to avoid 'best team get stuck'
2021-06-09 22:51:46 +00:00
371a38e6e5
Merge remote-tracking branch 'origin/master' into remove-extra-copies
2021-06-07 10:26:06 -07:00
85cd2b9945
add perpetualStorageWiggler
2021-05-20 23:31:08 +00:00
f28ac955c3
Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>>
2021-05-10 16:32:50 -07:00
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
4ee97c0784
Use clang-tidy to automatically fix missing overrides
...
Use `clang-tidy -p . $file -checks='-*,modernize-use-override' -header-filter='.*' -fix`
to fix missing overrides, and then use git clang-format to reformat just
those changes. This went pretty well for most files.
Formatting the following files went off the rails, so I'm going to
follow up with a commit that's just clang-tidy and no clang-format.
- fdbclient/DatabaseBackupAgent.actor.cpp
- fdbclient/FileBackupAgent.actor.cpp
- fdbserver/OldTLogServer_4_6.actor.cpp
- fdbmonitor/SimpleIni.h
- fdbserver/workloads/ClientTransactionProfileCorrectness.actor.cpp
2021-01-26 02:04:12 +00:00
7fbc4d7391
Resolve conflicts
2020-12-04 23:58:42 +00:00
877997632d
Merge branch 'release-6.3' into anoyes/merge-release-6.3-master
...
Include conflict markers for review purposes
2020-12-04 01:38:07 +00:00
dc2bac5670
Resolve conflicts
2020-11-24 19:09:42 +00:00
1f541f02be
Merge branch 'anoyes/merge-6.2-to-6.3' into anoyes/release-6.3-merge
...
Merge, leaving conflict markers for now
2020-11-24 16:55:34 +00:00
d64cf8b9e3
Merge branch 6.3 into master
2020-11-17 11:22:45 -08:00
489ba20641
Fix several merge issues
2020-11-16 14:46:36 -08:00
d0391db862
Merge branch 'release-6.2' into release-6.3
2020-11-16 10:15:23 -08:00
15d10d68cd
Cleanup DataDistributionQueue.actor.cpp and storageserver.actor.cpp
2020-11-09 15:23:51 -08:00
2909e35415
When moving dispaching shard relocations, log the source and destination storage servers.
2020-11-05 16:13:18 -08:00
cedcfc7d87
Removed unused uid field inside ParallelTCInfo
2020-10-26 16:48:09 -07:00
9ef29d0cea
Changed getTeamID() to return a string instead of UID as suggested by reviews.
2020-10-26 16:44:52 -07:00
7ebb2e5c09
Piggy back this PR to polish more TraceEvent by:
...
- Making it clear that it's tracking machine team info or server team info
- Added ID to both machine team and server team for better trackability
- Attach distributor id to some trace events.
2020-10-26 16:44:09 -07:00
e0b1f95740
Merge remote-tracking branch 'origin/master' into remove-global-ddenabled-flag
2020-10-21 18:22:08 -07:00
8cc3e4d3c6
Merge release-6.3 into master
2020-10-19 22:51:56 -07:00
652d753daf
Remove global ddEnabled flag
2020-10-17 11:23:52 -07:00
3b66a1f2d4
Fix a couple places where we were creating vectors with default elements rather than reserving space.
2020-10-09 10:51:06 -07:00
a9607bdcec
Explicitly seal classes that inherit but aren't inherited from
2020-10-07 21:58:24 -07:00
8571dcfe28
Use override where applicable in fdbserver
2020-10-07 18:41:19 -07:00
a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
ab0d8b0626
Merge pull request #3509 from sfc-gh-anoyes/anoyes/remove-using-relops
...
Remove using namespace std::rel_ops
2020-07-22 11:58:25 -07:00
83c5a30f62
Add encapsulation to TCTeamInfo and ParallelTCInfo
2020-07-21 11:05:41 -07:00
9a2ce4c981
Make IDataDistributionTeam const-correct
2020-07-21 11:05:34 -07:00
7affda3c8b
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
2020-07-14 14:57:24 -07:00
815e403009
Add comment to DDQueue GetTeam
...
Comments to help understand Evan's PR 3487
that fix the problem: replication factor could drop unexpected in fearless config.
2020-07-13 17:05:50 -07:00
47ae66bd61
Merge branch 'master' into mengxu/tmp-minor-comment-PR
...
Resolve conflict at waitFailureClient
2020-07-13 16:17:50 -07:00
efc446f529
Add a comment to DD
2020-07-13 10:12:39 -07:00
6f2e12be42
Minor improvement on comments
2020-07-12 18:32:47 -07:00
f470ba8316
Remove using namespace std::rel_ops
...
This causes the following to not compile anymore
\#include <utility>
\#include <vector>
using namespace std::rel_ops;
int main() {
std::vector<int> xs;
return xs.rbegin() != xs.rend();
}
See https://godbolt.org/z/s1977n
2020-07-10 22:58:15 +00:00
b09dddc07e
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# cmake/ConfigureCompiler.cmake
# documentation/sphinx/source/downloads.rst
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/fdbrpc.vcxproj
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/flow.vcxproj
2020-07-10 15:06:34 -07:00
0e2f5e8bb5
Added a flow lock to prevent too many source server fetches from happening at the same time and running the data distributor out of memory
2020-07-09 10:38:19 -07:00
5e02fd490e
fix: the check for if a teamCollection was tracking a source server was unreliable, leading to scenarios where we would temporarily replicate a shard less than teamSIze
2020-06-29 10:02:27 -07:00
48b1b20f67
Fixed a crash related to destruction order in data distribution
2020-05-10 23:14:19 -07:00
b1172417f5
Merge branch 'master' into per-priority-busy-logging
...
# Conflicts:
# flow/Knobs.cpp
# flow/Knobs.h
# flow/Net2.actor.cpp
2020-04-14 14:22:12 -07:00
1439de37b5
Convert GetRangeLimits() -> TOO_MANY + ASSERT().
2020-04-12 18:23:14 -07:00
6078fd1b18
Convert UID to Tag in keyServers to reduce txnStateStore size
2020-04-05 14:30:09 -07:00
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
12f2b32770
added additional logging in data distribution
2020-03-13 15:19:33 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
e219c1671f
Merge branch 'release-6.2' into feature-dd-region-queue
...
# Conflicts:
# fdbserver/Knobs.h
2020-03-04 16:25:38 -08:00
6d6f184e2f
added a knob which reverts the new queue behavior
2020-03-04 16:23:49 -08:00
b7834b2995
Merge pull request #2774 from etschannen/feature-dd-repopulate-priority
...
Make the DD priority of populating a region lower than machine failures
2020-03-04 16:15:18 -08:00
125bd13198
fix: in multi-region configurations, the data distribution queue could start too much work, expecting that the remote region would contribute to the read workload
2020-03-04 14:17:17 -08:00
6296465e07
Make the DD priority associated with populating a remote region lower than machine failures
2020-03-04 14:07:32 -08:00
1ef4cb432b
Merge branch 'master' into mengxu/fast-restore-robust-and-visibility-PR-v2
2020-03-01 20:08:07 -08:00
ad9b3fb4a8
DD:Add trace for detailed relocate shard info
2020-02-29 13:45:10 -08:00
2657d41bb2
FastRestore:Add debug msg when memory is over threshold
2020-02-27 18:32:11 -08:00
0f64505d0b
Merge branch 'release-6.2' of github.com:apple/foundationdb
...
Needed to pull in changes to build docker
2020-02-23 23:27:53 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
4c696d5bf2
Merge branch 'release-6.2' into dd-better-rebalance-logging
...
# Conflicts:
# fdbserver/DataDistributionQueue.actor.cpp
2020-02-21 17:41:00 -08:00
dfa5f76c01
Remove unused parameter. Don't put check for g_network presence in ASSERT_WE_THINK.
2020-02-21 16:28:03 -08:00
08914a2acd
Once available space ratio falls below 0.3 avoid moving data to teams with less free space than the median team
2020-02-21 15:14:32 -08:00
2e699fef55
Don't suppress actor cancellation because we've already initialized the trace event by adding details.
2020-02-21 11:28:59 -08:00
6810a03283
Add more logging to valley filler and mountain chopper
2020-02-21 10:55:14 -08:00
819c55556c
More aggressively attempt to find teams that do not have low disk space
2020-02-20 16:47:50 -08:00
e1fb568fd1
Merge branch 'release-6.2' into dd-use-available-space
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
2020-02-20 16:12:42 -08:00
e4b483796d
Combine some logic that was doing similar computations for free space ratio.
2020-02-20 14:52:08 -08:00
4c9c736253
Data distribution uses available space instead of free space when evaluating whether processes are low on space and penalizing them.
2020-02-20 11:21:03 -08:00
c164acb88d
Add new criteria to DD's GetTeamRequest that allow you to require shards be present on the team and that the team have a minimum free ratio. This avoids scenarios where the team chosen when processing the request is later rejected by the requestor, causing rebalancing movements to get stuck.
2020-02-20 09:32:00 -08:00
abb75f7eb7
Add logging to indicate the time spent at each priority that exceeds some minimum busyness threshold
2020-02-07 14:34:24 -08:00
3f9d9d8b84
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# cmake/FlowCommands.cmake
# documentation/sphinx/source/release-notes.rst
# fdbclient/StorageServerInterface.h
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# flow/Knobs.h
# flow/Platform.cpp
# versions.target
2020-01-16 18:37:47 -08:00
e65760eb46
Merge pull request #2536 from etschannen/feature-commit-latency
...
Improved commit latency in large clusters
2020-01-13 19:12:02 -08:00
ab7071932f
Data distribution no longer attempts to pick teams which share members of the source unless the team matches exactly
2020-01-09 16:59:37 -08:00
3157d8a375
fixed typo
2019-12-18 16:57:39 -08:00
688940b685
merge 6.2 into master
2019-10-21 11:43:46 -07:00
5667331729
added a buggify + minor code cleanup
2019-10-11 18:31:43 -07:00
86bcb84b45
Raised the data distribution priority of splitting shards above restoring fault tolerance to avoid hot write shards
2019-10-11 17:50:43 -07:00
22c3fa867c
Merge pull request #2074 from xumengpanda/mengxu/fix-correctness-bug
...
DD:fix:getTeam may fail to get a team when it can
2019-10-07 09:33:57 -07:00
3c4dc1003d
DD:clang-format the PR
2019-09-11 11:16:29 -07:00
sfc-gh-tclinkenbeard