1311fec45a
Add an option to get minKnownCommittedVersion from Proxies
...
The backup worker needs to use this version for popping when running in a NOOP
mode. This option is added to GetReadVersionRequest and proxies will send back
minKnownCommittedVersion if the option is set.
Also add a couple of knobs for backup workers.
2020-01-22 19:42:13 -08:00
0e5f5b50f0
Remove unused backup worker knobs
2020-01-22 19:38:46 -08:00
a4d6ebe79e
Recruit backup worker in newEpoch
2020-01-22 19:37:48 -08:00
de8d953865
Add backup role, class, and worker skeleton
2020-01-22 19:35:30 -08:00
54d77d20b2
Merge branch 'release-6.2'
2020-01-19 15:22:49 -08:00
8197f0562f
merge priority did not need to be raised, because we no longer merge shards until they are untrackable
...
max_commit_updates was too large, and could cause proxies to run out of memory
2020-01-17 14:24:58 -08:00
3f9d9d8b84
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# cmake/FlowCommands.cmake
# documentation/sphinx/source/release-notes.rst
# fdbclient/StorageServerInterface.h
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# flow/Knobs.h
# flow/Platform.cpp
# versions.target
2020-01-16 18:37:47 -08:00
4b90487b90
occasionally throw wrong_shard_server when waitMetrics times out so that the waitMetrics request can get the correct number of shards if two shards have been merged or split and the same storage server owns all the chunks
2020-01-15 13:22:18 -08:00
9668c4471f
Clamp infinite limit in ratekeeper
2020-01-14 15:45:24 -08:00
17e97f24e4
Merge pull request #2526 from etschannen/feature-dd-improvements
...
Data distribution improvements
2020-01-10 17:53:22 -08:00
fde53cbeef
HasBeenTrueFor was ready immediately after a previous shard merge
2020-01-10 16:28:56 -08:00
9b80498180
Added a trace event to warn if a shard is merged before enough time has elapses from becoming low bandwidth
2020-01-10 14:58:38 -08:00
4aab9b7bc8
fix: clients would waste time attempting to read from a remote region when it was in the process of catching up
2020-01-10 12:23:59 -08:00
02a8e8d1e9
batch priority must be heavily throttled before stopping data distribution rebalancing
2020-01-09 17:05:22 -08:00
9842272ced
raised the priority of shard merges, because the tracker cannot track an unmerged shard
2020-01-09 17:04:17 -08:00
e4fa4ad0c9
Data distribution will not merge a shard unless it has been low bandwidth for 5 minutes
2020-01-09 17:02:49 -08:00
ab7071932f
Data distribution no longer attempts to pick teams which share members of the source unless the team matches exactly
2020-01-09 16:59:37 -08:00
39a4f2372f
Change FASTRESTORE_SAMPLING_PERCENT to 0 to 100
2019-12-04 21:26:27 -08:00
c6b36dbffb
FastRestore:Sampling:Resolve review comments
2019-12-04 17:35:11 -08:00
dd91d26dfa
FastRestore:Sampling:Add FASTRESTORE_SAMPLING_RATE knob
2019-12-04 11:46:29 -08:00
07331ab5fd
Merge pull request #2362 from etschannen/master
...
Merge 6.2 into master
2019-12-02 15:04:27 -08:00
ebcb2f79ed
Merge branch 'master' of github.com:apple/foundationdb
2019-11-22 15:34:49 -08:00
14dd5626d7
Resolve review comments
2019-11-22 10:11:45 -08:00
b6e1839d84
Code clean up
2019-11-21 13:39:19 -08:00
b282e180d5
Added a knob to disable read sampling
2019-11-20 14:03:20 -08:00
8d3ef89540
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbclient/MutationList.h
# fdbserver/MasterProxyServer.actor.cpp
# versions.target
2019-11-14 15:49:56 -08:00
a4a0bf18f9
Merging with Master.
2019-11-12 13:01:29 -08:00
396dccbc98
when peeking from satellites we do not need to limit the amount of peeking on log router tags, because that is the only thing that can be peeked from a satellite log
2019-11-08 18:34:05 -08:00
afc9713005
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbclient/FDBTypes.h
# fdbserver/LogSystem.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/OldTLogServer_6_0.actor.cpp
# fdbserver/TLogServer.actor.cpp
# versions.target
2019-11-06 13:45:37 -08:00
daac8a2c22
Knobified a few variables
2019-11-04 20:21:38 -08:00
71dfaa3f95
Merge pull request #2275 from dongxinEric/bugfix/2273/fix-read-key-sampling
...
Resolves #2273 : Use a large value for read sampling size threshold. Also at sampling …
2019-10-31 10:21:58 -07:00
199a34b827
Defined a minimum read cost (a penalty) for empty read or read size smaller than it. Fixed several review comments.
2019-10-30 10:04:19 -07:00
3325980c03
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbserver/DataDistribution.actor.cpp
# fdbserver/OldTLogServer_6_0.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/WorkerInterface.actor.h
# fdbserver/worker.actor.cpp
# versions.target
2019-10-24 17:38:15 -07:00
a290e2cb2b
Use 8 MiB for real
2019-10-24 11:02:17 -07:00
fe54a4bde1
- Changed SHARD_MAX_BYTES_READ_PRE_KEYSEC to be equivalent to 8MiB/s, which when times the sample expire interval(120 seconds) yields 960MiB/s. A shard having a read rate larger than that will be marked as read-hot. The number 960MiB was chosen to be roughtly twice the size of the max allowed shard size to avoid wrongly marking a shard as read-hot when doing a table scan on it.
...
- Also tuned down the empty key sampling percentage to be 5%.
2019-10-23 12:00:19 -07:00
e676348710
Merge pull request #1955 from fzhjon/mark-ss-failed
...
Add fdbcli and API command to mark storage servers as permanently failed
2019-10-22 23:36:30 -07:00
af72d15566
Update fdbserver/Knobs.cpp
...
From AJ: to match typical aligned format used on other variables.
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-10-22 13:53:28 -07:00
e6f5748791
Use a large value for read sampling size threshold. Also at sampling site, don't round up small values to avoid sampling every key.
2019-10-22 13:47:58 -07:00
29a0014b41
Fix "bandwith" typo
2019-10-22 09:51:59 -07:00
12c517ab16
limit the number of committed version updates in progress simultaneously to prevent running out of memory
2019-10-21 16:01:45 -07:00
fca9aab17a
Merge pull request #2046 from dongxinEric/feature/hot-read-key-detection
...
Added metrics for read hot key detection
2019-10-21 14:31:48 -07:00
d2b6626d5c
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-21 13:47:06 -07:00
688940b685
merge 6.2 into master
2019-10-21 11:43:46 -07:00
8b09cd16b2
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-share-mutations
2019-10-16 14:50:37 -07:00
5667331729
added a buggify + minor code cleanup
2019-10-11 18:31:43 -07:00
86bcb84b45
Raised the data distribution priority of splitting shards above restoring fault tolerance to avoid hot write shards
2019-10-11 17:50:43 -07:00
62ffdd54a3
Updated some comments to reflect the correct knob value and also used a more appropiate value for read bandwidth. Set the default value for read bandwidth in some cases.
2019-10-09 16:42:42 -07:00
cd4757b06c
Address review comments
2019-10-09 16:42:42 -07:00
6b0f771cc0
Fixex a typo in knobs. Addressed some review comments. Added code for actual metric collecting.
2019-10-09 16:42:42 -07:00
12293d5497
Added metrics for read hot key detection
2019-10-09 16:42:42 -07:00
Jingyu Zhou