247ab84323
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-03-23 17:01:17 -07:00
26b7e02d4c
Some initial work to support tagging transactions and passing them around.
2020-03-20 11:23:11 -07:00
804fe1b22e
Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key"
...
This reverts commit 648dc4a933487d131b38
2020-03-19 21:34:28 -07:00
648dc4a933
Merge pull request #2257 from zjuLcg/report-conflicting-key
...
Report conflicting keys
2020-03-18 13:39:42 -07:00
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
04b752b40a
Added additional logging related to memory errors (including in status)
2020-03-13 18:31:22 -07:00
a71e61f57b
fixed compiler issue
2020-03-13 18:22:38 -07:00
ebbf4490b3
use a Deque for each priority instead of a priority queue to improve CPU with large numbers of outstanding requests
2020-03-13 18:07:48 -07:00
243c268d9d
Limit the amount of requests the proxy can queue up in memory
2020-03-13 10:17:49 -07:00
02ee4f4c46
Update comments
2020-03-11 22:22:51 -07:00
7a76e9556d
Merge remote-tracking branch 'upstream/master' into report-conflicting-key
2020-03-04 11:24:39 -08:00
993c6e478e
Merge branch 'master' into grv-proxy-perf-improvements
2020-02-28 14:25:42 -08:00
1ec55f21f7
Add some comments for documentation; strengthen an assert.
2020-02-28 14:25:19 -08:00
924d335aa7
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# flow/Knobs.cpp
# flow/Knobs.h
2020-02-25 18:25:19 -08:00
6e7d2ff7dd
prevent the proxy from delaying too long based on an incorrect estimate of the compute time
2020-02-25 15:46:13 -08:00
80c2848af6
Change the algorithm for the proxy handing out read versions to improve performance and increase responsiveness to changes in workload.
2020-02-24 09:52:31 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
d7c841a28a
Merge pull request #2589 from etschannen/feature-proxy-delay
...
Improve version pipelining on the proxy
2020-02-20 15:23:30 -08:00
7d54acf4ca
removed an unnecessary yield
2020-02-20 14:41:49 -08:00
df2b0452b4
Step 3 of fixing storage server range reads: change return type of readRange from VectorRef<KeyValueRef> to RangeResultRef.
2020-02-06 13:19:24 -08:00
058de86b88
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-02-05 10:07:36 -08:00
8d28c2a7f0
Added two new counters for transaction throttled error and remove the verbose trace event logging. Also changed a chain of 'if' statements into 'if-else' statements since they are mutal exclusive
2020-01-31 14:16:39 -08:00
800d2e560c
Apply review comments.
2020-01-31 09:32:22 -08:00
f8513327dc
Update fdbserver/MasterProxyServer.actor.cpp
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-01-31 09:30:37 -08:00
ca8a4c0096
Fix CTest failure.
2020-01-30 16:32:32 -08:00
babd3fa919
Rebased with master
2020-01-30 14:13:56 -08:00
1d0d4adb81
Address review comments from Evan
2020-01-30 14:13:56 -08:00
c901fc269b
Changed to use 'rate' instead of 'limit' after some discussion with Evan and AJ
2020-01-30 14:13:56 -08:00
65c607bc13
Fix the error after the rebase
2020-01-30 14:13:56 -08:00
1b313a4f7e
Address review comments. Rebased with latest master
2020-01-30 14:13:56 -08:00
9aaf4bc107
Add code coverage mark when sending out the throttled error.
2020-01-30 14:13:56 -08:00
e21426d12a
Send error back to the GRV requests with batch priority when the cluster is saturated, instead of blindly enqueue the requests and let the client timeout.
2020-01-30 14:13:56 -08:00
231d7830a0
more accurate calculation on the amount of time that proxy should wait before getting a version from the master
2020-01-26 19:47:12 -08:00
e167e63eaf
Add delays between proxy batches which roughly corresponding to the amount of work the proxy needs to do. This will help avoid getting a version from the master and then waiting a long time before committing it.
2020-01-23 18:31:51 -08:00
c9363e7e28
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-01-22 21:02:21 -08:00
8b67a89eed
More review comments fixed.
2020-01-22 19:42:13 -08:00
1311fec45a
Add an option to get minKnownCommittedVersion from Proxies
...
The backup worker needs to use this version for popping when running in a NOOP
mode. This option is added to GetReadVersionRequest and proxies will send back
minKnownCommittedVersion if the option is set.
Also add a couple of knobs for backup workers.
2020-01-22 19:42:13 -08:00
12e91240cc
Fix a typo
2020-01-22 19:38:45 -08:00
670c3e629f
Fix arena issue and change std::vector back to VectorRef
2020-01-17 14:24:13 -08:00
3f9d9d8b84
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# cmake/FlowCommands.cmake
# documentation/sphinx/source/release-notes.rst
# fdbclient/StorageServerInterface.h
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# flow/Knobs.h
# flow/Platform.cpp
# versions.target
2020-01-16 18:37:47 -08:00
db901d626a
Update some var names and make returned list in ResolveTransactionBatchReply as Optional
2020-01-16 14:41:06 -08:00
1fa934837d
Update some var names
2020-01-16 13:14:13 -08:00
30b9858cf5
using std::vector instead of VectorRef to avoid unknown arena change behavior
2020-01-16 13:06:36 -08:00
1d6ac716a1
Merge remote-tracking branch 'origin' into add-data-distribution-metrics
2020-01-15 13:20:04 -08:00
0749a5a1ab
update comment
2020-01-15 11:06:40 -08:00
72d39a31f1
Change to send back index of read_conflict_range not keys, which speeds up but fail to give narrowed keyrange
2020-01-15 10:21:51 -08:00
1c2b116688
Push sorting and union of overlapped keyranges to clients
2020-01-13 18:08:54 -08:00
83ad9caf54
implemented a load balancing algorithm which evens out the number of requests processes by each proxy
2020-01-08 01:59:01 -08:00
10719200c3
A hack way to call API through getRange("\xff\xff/conflicting_keys\<start_key>", "\xff\xff/conflicting_keys\<end_key>").
2020-01-06 11:22:11 -08:00
d8c3c2fda4
Improved prioritization of commit path on the proxies
2019-12-18 16:56:35 -08:00
eacfdf6da3
Add a simple workload, ReportConflictingKeysWorkload, to test correctness of the API and performance overhead added to the resovler.
2019-12-06 16:21:03 -08:00
2b987d1945
FastRestore:typedef Standalone<VectorRef<MutationRef>> MutationsVec
2019-12-04 11:39:55 -08:00
edf52e8c97
First version for reporting conflicting keys
2019-12-02 13:45:39 -08:00
3c769fcf60
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# versions.target
2019-11-22 15:39:19 -08:00
ebcb2f79ed
Merge branch 'master' of github.com:apple/foundationdb
2019-11-22 15:34:49 -08:00
7c801513e2
Fix cases where latency band config could be discarded during recovery or process start.
2019-11-20 11:44:18 -08:00
8d3ef89540
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbclient/MutationList.h
# fdbserver/MasterProxyServer.actor.cpp
# versions.target
2019-11-14 15:49:56 -08:00
11525f6922
added comments
2019-11-13 12:53:23 -08:00
8f725db92e
serialization of logRangeMutation->second caused long slow tasks
2019-11-12 23:06:58 -08:00
a4a0bf18f9
Merging with Master.
2019-11-12 13:01:29 -08:00
3325980c03
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbserver/DataDistribution.actor.cpp
# fdbserver/OldTLogServer_6_0.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/WorkerInterface.actor.h
# fdbserver/worker.actor.cpp
# versions.target
2019-10-24 17:38:15 -07:00
a1bed51d34
Ignore batch priority GRVs for latency band tracking
2019-10-23 10:29:58 -07:00
12c517ab16
limit the number of committed version updates in progress simultaneously to prevent running out of memory
2019-10-21 16:01:45 -07:00
d2b6626d5c
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-21 13:47:06 -07:00
688940b685
merge 6.2 into master
2019-10-21 11:43:46 -07:00
ac28e96bbf
added a yield on the proxy to remove a slow task when processing large transactions
2019-10-16 14:31:59 -07:00
450a09e117
Code Review Changes
2019-09-24 15:48:50 -07:00
40ad6f0931
created new reply types to work with flatbuffer serialization requirements
2019-09-18 13:40:18 -07:00
471e283128
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-09-18 11:49:07 -07:00
02230e4ef6
Refactor: moving addition of backup mutations out of commitBatch
...
commitBatch() is too complicated and should be simplified.
2019-09-12 18:37:08 -07:00
dcbb19816b
Merge pull request #1999 from ajbeamon/fix-proxy-grv-budgeting
...
The master proxy was too slow to erase a GRV budget deficit if no GRV requests were coming in.
2019-08-30 13:32:54 -07:00
5d87443323
improved error msgs for snapshot cmd
2019-08-27 16:43:52 -07:00
3666c0c776
added more trace lines and added timeout to safety check in test workload
2019-08-27 14:39:44 -07:00
04d514c483
added a wait to check for master proxies changed and put in a few more trace events
2019-08-27 14:39:44 -07:00
e515691d7d
do not continuously loop if maybe_request_delivered
2019-08-27 14:39:44 -07:00
00c2025d4b
fixed removeKeys impl, adjusted test workload, and introduced extra safety checks to NativeAPI and proxy
2019-08-27 14:39:44 -07:00
5a877d6b14
added safety check on client to prevent removing all servers from a team
2019-08-27 14:39:43 -07:00
0b1fc91a9c
Revert "Don't grow the budget deficit once it's exceeded some number of seconds of transactions. Decay the deficit if the rate changes and it exceeds the new limit."
...
This reverts commit 90cb73d472
2019-08-22 10:05:29 -07:00
90cb73d472
Don't grow the budget deficit once it's exceeded some number of seconds of transactions. Decay the deficit if the rate changes and it exceeds the new limit.
2019-08-19 14:56:59 -07:00
b2af17fb08
Simplify logic by removing an unneeded condition.
2019-08-15 08:23:13 -07:00
717ede25b3
Fix: the master proxy was too slow to erase a GRV budget deficit if no GRV requests were coming in.
2019-08-14 15:01:09 -07:00
ba54508c47
code cleanup
2019-08-06 16:30:30 -07:00
4c9a392f05
the master checks the popped version of the txsTag before recovering the txnStateStore, to avoid restoring data that is later found to be popped
2019-08-05 17:01:48 -07:00
1bad0fd44e
Make requestTime private
2019-07-31 17:59:35 -07:00
6dbaddd0a7
Added a knob to always use CAUSAL_READ_RISKY for GRV
2019-07-30 18:21:46 -07:00
63941e0d96
disable DD with a in-memory flag and use in snapv2
2019-07-30 17:04:51 -07:00
5c98dcce6d
revert the proxy forwarding path, because it is no longer necessary as clients keep a persistent connection open with coordinators
2019-07-27 16:46:22 -07:00
b509a441e7
Merge branch 'master' into feature-skip-confirm
...
# Conflicts:
# bindings/flow/tester/Tester.actor.cpp
# bindings/go/src/_stacktester/stacktester.go
# bindings/java/src/test/com/apple/foundationdb/test/AsyncStackTester.java
# bindings/java/src/test/com/apple/foundationdb/test/StackTester.java
# bindings/python/tests/tester.py
# bindings/ruby/tests/tester.rb
# documentation/sphinx/source/api-c.rst
# documentation/sphinx/source/api-python.rst
# documentation/sphinx/source/api-ruby.rst
# documentation/sphinx/source/data-modeling.rst
# documentation/sphinx/source/developer-guide.rst
# fdbclient/vexillographer/fdb.options
# fdbserver/MasterProxyServer.actor.cpp
2019-07-27 15:08:13 -07:00
9afd162e2f
remove snap v1 related code
2019-07-25 17:29:31 -07:00
a65c9f92ed
get rid of all timeouts and other changes
2019-07-24 15:36:28 -07:00
a2f2ad96ff
code review comments and merge to master changes
2019-07-24 15:36:28 -07:00
869f77aef1
Few cosmetic edits and fixes
2019-07-24 15:36:28 -07:00
62c14dae72
disable dd during snap and enable in restore
2019-07-24 15:36:28 -07:00
95d6807740
tryGetReply instead of getReply for ddSnapReq
2019-07-24 15:36:28 -07:00
d0793f5ca2
snap v2: master proxy related changes
2019-07-24 15:36:28 -07:00
63e37aebaf
Reorder include files.
2019-07-19 11:24:26 -07:00
d8fb1ea2d3
Log large transactions at proxy
...
This can help debugging where large transactions are coming from.
2019-07-19 11:10:48 -07:00
d4c332e2cf
update lastCommitTime when doing GRV’s without causal read risky
2019-07-14 14:46:00 -07:00
02de53160d
only skip confirm epoch live if CAUSAL_READ_RISKY is enabled
...
time checked on the proxy should be less than the time waited by the master to account for clock speed differences
setting REQUIRED_MIN_RECOVERY_DURATION and ENFORCED_MIN_RECOVERY_DURATION to 0 will go back to the old behavior
2019-07-12 17:58:16 -07:00
a63969afb3
enforce a minimum recovery duration, which allows proxies to avoid checking if the epoch is alive as long as its last commit has been less than MINIMUM_RECOVERY_DURATION ago
2019-07-12 13:10:21 -07:00
tclinken