db7b48b71c
wiggling teams calculation replace
2021-10-27 09:08:37 -07:00
3a6359e202
minus wiggling teams when build team
2021-10-27 09:08:37 -07:00
16ae2b76e5
Merge branch 'master' of https://github.com/apple/foundationdb into clean-sim-test-data-loss
2021-10-21 09:16:53 -07:00
504d0b71b2
Fix invalid memory access when dataDistribution actor is cancelled ( #5791 )
...
* Fix valgrind error when dataDistribution actor is cancelled
* Trace Sev30 when dataDistribution actor is cancelled outside of simulation
* Rethrow actor_cancelled error in dataDistribution catch block
2021-10-18 14:21:29 -07:00
dbfeb06c97
Reproduced user data loss incident, and tested the improved exclude tool
...
can fix the system metadata.
2021-10-14 14:08:39 -07:00
f339b603a5
Bug fix: printSnapshotTeamsInfo() could crash when looking up status for a storage server that was very recently added because its entry in server_status was not yet created.
...
Bug fix: printSnapshotTeamsInfo()'s local server_status map would not see status updates for server UIDs that already existed in the map.
2021-10-10 01:48:31 -07:00
3ea7209013
Simulation changes to randomly wiggle with locality filter and review comments.
2021-09-30 10:00:33 -07:00
3e79299898
Locality filter support to perpetual storage wiggler feature.
2021-09-30 10:00:33 -07:00
f50a3f08de
Fix format problem for file fdbserver/DataDistribution.actor.cpp.
...
Description
Testing
2021-09-24 15:40:17 -07:00
8761960cdc
fix roll trace event issue for data distribution(master)
...
Description
Testing
2021-09-24 15:40:17 -07:00
731c1fffac
fix roll trace event issue for data distribution(master)
...
Description
Testing
2021-09-24 15:40:17 -07:00
814e3c729f
fix roll trace event issue for data distribution(master)
...
Description
Testing
2021-09-24 15:40:17 -07:00
c10dd0df4b
fix roll trace event issue for data distribution
...
Description
Testing
2021-09-24 15:40:17 -07:00
a1f6dcc2d5
merge from master
2021-09-22 13:07:38 -07:00
1730d75f73
change configure test
...
add store type check
add test file
2021-09-21 18:11:04 -07:00
2246a0bee7
switch to plain random selection
2021-09-20 11:23:54 -07:00
4d5bf08da8
address comments
2021-09-19 17:03:06 -07:00
abf73047ca
Enforce std:: specifier rather than using namespace
2021-09-16 19:40:28 -07:00
ef7fdc0781
fmt
2021-09-15 10:32:09 -07:00
c8a3413820
exclude to-be-dropped server from the random team
2021-09-15 09:07:50 -07:00
fd6d088945
choose team before removing server
2021-09-14 19:24:59 -07:00
7e53f8662d
added comments
2021-09-13 13:28:55 -07:00
8e0b572a18
Added comments.
2021-09-02 22:29:07 -07:00
6612cc00b6
Check if the src server list will be empty before removing a failed server."
2021-09-01 14:52:07 -07:00
66df75c570
Merge pull request #5385 from sfc-gh-tclinkenbeard/debug-dd
...
Capture deep copy of `machine_info` in `printSnapshotTeamsInfo`
2021-08-20 13:25:50 -07:00
9458a6975d
Remove std::map::at usage from DataDistribution.actor.cpp
2021-08-20 12:35:26 -07:00
556e4bc283
Add assertion to overlappingMachineMembers
2021-08-13 14:56:22 -07:00
a0a4207ce2
Capture deep copy of machine_info in printSnapshotTeamsInfo
2021-08-13 12:13:54 -07:00
3f0e07d79c
Remove dead code
2021-08-13 09:54:02 -07:00
ea4f6850da
Mark ServerStatus::excludeOnRecruit const
2021-08-13 09:52:07 -07:00
0aafd9d5f0
Mark TCServerInfo::isCorrectStoreType const
2021-08-13 09:49:22 -07:00
e444d3781c
Various TSS improvements from snowblower testing
2021-08-13 10:24:15 -05:00
904deb9516
Improve DDTeamCollection const-correctness
2021-08-12 18:52:57 -07:00
cfe677c100
storageRecruiter only responds to changes in recruitStorage endpoint
2021-08-12 16:24:03 -07:00
45ac667271
Add IsPrimary boolean parameter
2021-08-12 14:05:04 -07:00
ca9f60baef
Fix heap use after free
2021-08-11 15:42:01 -07:00
2337ec7d4e
remove unused variables
2021-08-03 10:15:34 -07:00
c74047c665
Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings
2021-07-28 11:51:02 -07:00
507c1f11e3
Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use.
2021-07-26 19:55:10 -07:00
a27d7c86f4
Fix more -Wreorder-ctor warnings in DataDistribution.actor.cpp
2021-07-24 22:14:43 -07:00
da50e13f3e
Fix more -Wreorder-ctor warnings in DataDistribution.actor.cpp, OldTLogServer_4_6.actor.cpp, and Net2.actor.cpp
2021-07-24 17:33:11 -07:00
6f81155784
Merge remote-tracking branch 'origin/master' into const-serverdbinfo
2021-07-20 10:18:40 -07:00
f596a81073
Rename ::TRUE and ::FALSE in BooleanParams to ::True and ::False so as to not conflict with the TRUE and FALSE macros provided by the Windows and MacOS SDKs.
2021-07-17 00:11:40 -07:00
501dc339a9
relax perpetual wiggle pause condition; add trace log; correct perpetual wiggle priority setting
2021-07-12 05:46:55 +00:00
8a212862f0
Prevent dataDistributor from modifying ServerDBInfo object
2021-07-11 22:04:54 -07:00
79ff07a071
Added *BOOLEAN_PARAM macros to enforce documentation of boolean parameters
2021-07-02 15:04:42 -07:00
73752f441b
exclude locality:clang-format, ranged loops, documentation, tracking addStoragesever for exclusion.
2021-06-23 18:03:27 -07:00
62355571d0
exclude servers based on locality match
2021-06-23 18:03:27 -07:00
7b713f7fd2
add knob
2021-06-23 05:49:55 +00:00
f2daf20927
TEST condition
2021-06-21 06:56:03 +00:00
0493d149e6
wait remove
2021-06-21 05:18:42 +00:00
783520ce85
add and remove some healthy check to solve cluster status oscillation when #ss is little; simplify some code
2021-06-19 16:57:04 +00:00
647138145d
adjust default value of stopWiggleSignal; better trace logic
2021-06-17 20:59:47 +00:00
fdd9c30794
code refactor;change stopSignal;
2021-06-16 05:30:58 +00:00
d33e43fd2b
code format
2021-06-14 23:00:02 +00:00
2cd4e6d62f
check healthy team count, dd queue and disk space;
...
code refactor
2021-06-14 22:09:45 +00:00
d46fccc30f
Revert "Revert "Properly set simulation test for perpetual storage wiggle and bug fixing""
...
This reverts commit ad576e8c20
2021-06-11 22:58:05 +00:00
ad576e8c20
Revert "Properly set simulation test for perpetual storage wiggle and bug fixing"
2021-06-11 09:07:45 -07:00
17ac91bac4
Merge pull request #4929 from sfc-gh-xwang/ppwtest
...
Properly set simulation test for perpetual storage wiggle and bug fixing
2021-06-10 14:09:50 -07:00
cd58c0c149
add useful trace; add invalid wiggling server check
2021-06-10 06:50:44 +00:00
4220a548ce
use the same health check as exclude to avoid 'best team get stuck'
2021-06-09 22:51:46 +00:00
51b4cb89c2
fix server_status bug
2021-06-08 23:47:59 +00:00
45ebdb1a9d
fix perpetual wiggle bug caused by multiple DCs and removeStorageServer
2021-06-08 23:33:25 +00:00
6ab0ea3d0f
properly set perpetual_storage_wiggle value during tests
2021-06-07 17:55:20 +00:00
371a38e6e5
Merge remote-tracking branch 'origin/master' into remove-extra-copies
2021-06-07 10:26:06 -07:00
838d847d4e
Merge pull request #4860 from sfc-gh-xwang/ppwtest
...
implement perpetual storage wiggling feature
2021-06-04 16:18:39 -07:00
5be65fab5e
add comment
2021-06-04 18:40:18 +00:00
e0981d6732
add code coverage mark
2021-06-03 19:58:28 +00:00
351325b3af
comment modification; wait perpetual wiggling close
2021-06-03 05:13:20 +00:00
21e175b16c
add comments for new actors
2021-06-02 18:49:01 +00:00
944c9ad8d9
fix memory bug
2021-06-02 17:53:44 +00:00
b3e4f182ef
TSS Mapping Change
2021-06-02 17:30:09 +00:00
9684d78a6e
solve recruiting conflict with TSS
2021-06-02 06:12:45 +00:00
8b9c8b33fc
manually merge with master
2021-06-01 17:51:42 +00:00
ce308edc5e
fix wiggler logic bug
2021-05-26 21:57:58 +00:00
4257ac2b4d
More TSS Changes/Fixes
2021-05-25 20:37:48 +00:00
ce82c9653e
Testing Storage Server implementation
2021-05-25 20:28:50 +00:00
e9a23840ea
fix promise bug
2021-05-25 20:25:21 +00:00
f11b7ffa5f
merge master, fix promise callback bug
2021-05-25 18:43:08 +00:00
7bc55448aa
fix iterator bug
2021-05-24 19:11:28 +00:00
85cd2b9945
add perpetualStorageWiggler
2021-05-20 23:31:08 +00:00
3f3a81b3d9
add pid2server_info to maintain Process id set
2021-05-20 03:32:15 +00:00
f28ac955c3
Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>>
2021-05-10 16:32:50 -07:00
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
0db28f6ea0
Merge pull request #4535 from jzhou77/fix-dd
...
Fix DD Assertion failed in canBeSet
2021-03-24 10:50:04 -07:00
0c3bc09524
Remove the shuttingDown flag
2021-03-21 20:12:37 -07:00
cb26576b95
Fix DD assertion failure
...
This fixes #4493 , where DDTeamCollection::~DDTeamCollection creates new teams
that hold pointer to the DDTeamCollection, thus later causes assertion failure
because the memory is invalid.
The fix is to cancel teamBuilder at the begining of the ~DDTeamCollection.
2021-03-21 19:54:44 -07:00
d2f9bf7eb6
added comments and fixed style
2021-03-16 15:44:49 -07:00
edefcff3ac
do not kill the data distributor after removing a failed server, completely remove the failed server
2021-03-15 16:48:08 -07:00
c0a1362478
fixed a bug where DD was shutdown while still in a callback from trackExcludedServers
2021-03-15 16:26:57 -07:00
c570a7b718
added trace events
2021-03-15 15:55:02 -07:00
403d933329
fixed trace event
2021-03-15 10:51:53 -07:00
831224df99
fixed complier error
2021-03-15 10:48:48 -07:00
4e4149b070
exclude failed shuts down data distribution while the server is being removed to avoid two processes making changes to the key servers at the same time
2021-03-15 10:43:06 -07:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
2bb4f2e59f
Merge branch 'release-6.3-pre-format' into master-format
...
This merges release-6.3 branch right before it was fully formatted.
There were quite a few conflicts that are resolved here. CoroFlow had
a check for OOM errors introduced in 6.3, but didn't seem applicable in
the new implmentation which seems to use boost.
2021-03-10 09:37:41 -08:00
9645f489e6
Fix base trace event name inconsistency
2021-03-08 15:20:50 -08:00
59181245c1
Change SSVersionDiffLarge event log level to warning
2021-03-03 23:33:48 -08:00
79cec09255
Apply clang-tidy's performance-inefficient-vector-operation fix
...
I ran this command in my build directory after compiling with
OPEN_FOR_IDE. It took a few small tweaks to get it to compile, which is
outside the scope of this commit.
$ python run-clang-tidy.py -j $(nproc) -checks='-*,performance-inefficient-vector-operation' -fix
2021-03-04 03:58:25 +00:00
346a4e3ecd
Merge branch 'release-6.3'
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/MultiInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2021-03-01 18:52:06 -08:00
32486a2785
Reenable tlog pops in ddSnapCreateCore even if some disable requests fail
...
If some tlogs successfully disable pops but others fail to, we do not
want to wait TLOG_IGNORE_POP_AUTO_ENABLE_DELAY seconds before reenabling
pops
2021-02-20 18:24:21 -08:00
8025cc4571
Merge remote-tracking branch 'upstream/release-6.3' into merge-6.3-to-master-1-26-21
2021-01-26 19:46:47 +00:00
0ef44739ea
Fix OPEN_FOR_IDE build in preparation for using clang-tidy
2021-01-26 02:04:11 +00:00
0cde3cc48f
Make trace event happy
2021-01-25 14:04:16 -08:00
dce8af5b63
Change back to SevWarnAlways for now.
2021-01-25 13:54:24 -08:00
bdf58d0b2f
Add assertion to DDTeamCollection::overlappingMembers
2021-01-21 14:39:39 -08:00
ad99bf0471
Merge remote-tracking branch 'origin' into misc-changes
2021-01-21 10:03:07 -08:00
83506cda87
Log SevError instead of SevWarnAlways when all replicas of some data are lost.
2021-01-15 15:00:48 -08:00
ff7d306b09
Merge branch 'release-6.3' into anoyes/merge-6.3-to-master
...
Include conflict markers for now. Will resolve.
2021-01-15 18:04:09 +00:00
5b2e88b187
Use structured bindings in for loops
2020-12-27 01:46:20 -04:00
0d4e81e6b4
Use unique_ptr in DataDistribution.actor.cpp
2020-12-26 23:40:54 -04:00
19816ccdbf
Improve DataDistribution const-correctness
2020-12-26 20:22:27 -04:00
26a4884eef
Mark TCMachineTeamInfo::size const
2020-12-26 19:23:01 -04:00
bbb56e4089
Merge branch 'release-6.2' of https://github.com/apple/foundationdb into release-6.3
2020-12-23 14:26:59 -08:00
877997632d
Merge branch 'release-6.3' into anoyes/merge-release-6.3-master
...
Include conflict markers for review purposes
2020-12-04 01:38:07 +00:00
78503db523
Reset and retry transaction errors
2020-12-03 14:42:30 -08:00
ac02329d7d
Added a command in fdbcli to allow user to manually trigger the detailed teams info loggings in data distributor
2020-12-03 14:42:30 -08:00
b8a9807336
Move trackerCancelled higher in catch block
2020-11-24 20:34:06 +00:00
dc2bac5670
Resolve conflicts
2020-11-24 19:09:42 +00:00
1f541f02be
Merge branch 'anoyes/merge-6.2-to-6.3' into anoyes/release-6.3-merge
...
Merge, leaving conflict markers for now
2020-11-24 16:55:34 +00:00
fc9b78737f
Fix some merge bugs
2020-11-17 14:53:02 -08:00
d64cf8b9e3
Merge branch 6.3 into master
2020-11-17 11:22:45 -08:00
489ba20641
Fix several merge issues
2020-11-16 14:46:36 -08:00
d0391db862
Merge branch 'release-6.2' into release-6.3
2020-11-16 10:15:23 -08:00
ca8ea3b6ff
Fix memory issues caused by cancelling data distribution tracker
2020-11-15 23:52:36 -08:00
1343f40117
don't allow empty coments
2020-11-11 14:07:54 -07:00
bdd3dbfa7d
remove duplicates
2020-11-10 14:01:07 -07:00
4669f837fa
Add uses of makeReference
2020-11-07 22:10:18 -08:00
3ae611d668
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-pause-backup-snapshot
2020-11-04 14:26:49 -05:00
bda72d9a3d
first draft at changing snapshot backup behaviour
2020-11-02 17:12:30 -05:00
cf4c8e375f
Merge remote-tracking branch 'origin/release-6.3' into merge
2020-10-29 22:15:41 -07:00
99c1880a83
Merge commit 'f9581de2005e6b085776e81b9fcaa16442b32589' into merge-6.2-to-6.3
...
# Conflicts:
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
2020-10-27 12:21:26 -07:00
be7944773f
Fix a typo
2020-10-26 16:44:52 -07:00
9ef29d0cea
Changed getTeamID() to return a string instead of UID as suggested by reviews.
2020-10-26 16:44:52 -07:00
bec2cfb167
Fix typos.
2020-10-26 16:44:52 -07:00
0bc51bb780
Resolve review comments
2020-10-26 16:44:50 -07:00
7ebb2e5c09
Piggy back this PR to polish more TraceEvent by:
...
- Making it clear that it's tracking machine team info or server team info
- Added ID to both machine team and server team for better trackability
- Attach distributor id to some trace events.
2020-10-26 16:44:09 -07:00
c037bfd001
Added detailed logging when there is no servers left in a server team, because that may indicate a data loss incident.
2020-10-26 16:44:07 -07:00
6395b76d8c
Address more review comments
2020-10-23 15:29:08 -07:00
f757cae786
Address review comments
2020-10-23 14:01:53 -07:00
e0b1f95740
Merge remote-tracking branch 'origin/master' into remove-global-ddenabled-flag
2020-10-21 18:22:08 -07:00
8cc3e4d3c6
Merge release-6.3 into master
2020-10-19 22:51:56 -07:00
44c62b2d51
Merge pull request #3922 from jzhou77/release-6.3
...
Merge Release 6.2 to Release 6.3
2020-10-19 14:38:36 -07:00
652d753daf
Remove global ddEnabled flag
2020-10-17 11:23:52 -07:00
70c1ac2131
Use TraceEvent::error
2020-10-16 16:55:09 -07:00
8d0aa02a63
Do not periodically print detailed DD teams info
2020-10-16 16:11:14 -07:00
9dec4bc46a
Add ErrorCode to StorageServerTrackerCancelled trace event
2020-10-16 15:40:14 -07:00
8f17a1a5d6
Merge branch 'release-6.2' into release-6.3
2020-10-16 15:25:39 -07:00
30488df5ea
Fix build
2020-10-16 14:26:40 -07:00
81193a9226
Move TraceEvent upward
2020-10-16 14:23:27 -07:00
Xiaoxi Wang