Commit Graph

871 Commits

Author SHA1 Message Date
Daniel Smith d0210fd54f Rename option 2020-04-02 21:41:16 +00:00
negoyal a0c8946f31 Merge branch 'master' into fdb_cache_subfeature2 2020-04-02 12:27:04 -07:00
tclinken 884e92bb49 Atomically update dependent knobs 2020-04-01 15:18:49 -07:00
chaoguang c8abd9c493 Reformat some code 2020-03-31 18:53:36 -07:00
Xin Dong 6820167d77
Merge branch 'master' into feature/1689/allow-custome-trace-log-file-identifier 2020-03-31 16:50:46 -07:00
Xin Dong 459c2e34cd Now limit the file identifier to a). has a length no longer than the value defined in knobs; b). can only contain [A-Za-z0-9_] chars. 2020-03-31 16:42:00 -07:00
chaoguang a63f8b839c Merge remote-tracking branch 'upstream/master' into private-keyspace-framework 2020-03-31 12:05:42 -07:00
chaoguang 82a2196d7d update space range 2020-03-31 09:34:53 -07:00
Meng Xu 60f6edc3b5
Merge pull request #2860 from zjuLcg/report-conflicting-key-roll-forward
Report conflicting key roll forward
2020-03-30 17:33:56 -07:00
negoyal e358510e27 Minor compilation fix. 2020-03-26 16:50:58 -07:00
chaoguang 64148469e8 clang-format the pr 2020-03-26 15:52:30 -07:00
negoyal acaf91ac47 Merge branch 'master' into fdb_cache_subfeature2 2020-03-26 13:33:08 -07:00
Meng Xu 1052b23ee1
Merge pull request #2370 from atn34/test-watch-outliving-transaction
Test watch outliving transaction
2020-03-26 12:40:38 -07:00
Xin Dong a0177a9335 Allow the user to provide a custome trace log file identifier that will be used as the prefix of all trace log files created at the client side. 2020-03-26 11:25:05 -07:00
Andrew Noyes f6af6ec777 Avoid passing self to watch actor 2020-03-26 10:29:23 -07:00
Andrew Noyes cdb6bbfc85 Test watch outliving transaction 2020-03-26 10:09:03 -07:00
tclinken 7d5ed53215 Allow trace log group to be set after database is created 2020-03-25 13:40:43 -07:00
Andrew Noyes 289487559d Revert "Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key""
This reverts commit 804fe1b22e.
2020-03-24 18:11:15 -07:00
chaoguang 6df92a8b8d A work version of strict workload test 2020-03-23 23:55:56 -07:00
tclinken 247ab84323 Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics 2020-03-23 17:01:17 -07:00
A.J. Beamon 197f866824 Don't include internal tools such as fdbcli, backup, and DR in the status client list. 2020-03-23 14:44:04 -07:00
A.J. Beamon 26b7e02d4c Some initial work to support tagging transactions and passing them around. 2020-03-20 11:23:11 -07:00
Balachandar Namasivayam 804fe1b22e Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key"
This reverts commit 648dc4a933, reversing
changes made to 487d131b38.
2020-03-19 21:34:28 -07:00
Evan Tschannen 648dc4a933
Merge pull request #2257 from zjuLcg/report-conflicting-key
Report conflicting keys
2020-03-18 13:39:42 -07:00
Daniel Smith 572f08e5fc Add option to set transaction as debug 2020-03-17 22:21:31 +00:00
Evan Tschannen e08f0201f1 merge release 6.2 into master 2020-03-17 12:51:47 -07:00
Evan Tschannen ed4d02a3e4
Merge pull request #2812 from etschannen/feature-proxy-mem-limit
Limit the amount of requests the proxy can queue up in memory
2020-03-16 14:56:56 -07:00
Evan Tschannen a068d4063f renamed ProxyGetConsistentReadVersion 2020-03-16 12:11:32 -07:00
negoyal 99a5cb0572 Fix a corruption bug. 2020-03-13 18:52:34 -07:00
A.J. Beamon 031b579ede Increase priority of the logging of various metrics trace events. 2020-03-13 16:20:23 -07:00
chaoguang 39a37531db Fix issues according to Andrew's comments 2020-03-13 15:42:15 -07:00
chaoguang c4c38c5eca Delete commented code 2020-03-13 12:58:12 -07:00
Evan Tschannen 243c268d9d Limit the amount of requests the proxy can queue up in memory 2020-03-13 10:17:49 -07:00
chaoguang 6f90228a0b change to krmSetRangeCoalescing 2020-03-12 11:31:36 -07:00
chaoguang c2f0c41c52 use krmSetRange 2020-03-11 23:12:38 -07:00
chaoguang 6ae60870fc use krmSetRange 2020-03-11 13:20:40 -07:00
chaoguang 157d026ba2 debugging 2020-03-10 18:42:49 -07:00
chaoguang 698198a09e Merge remote-tracking branch 'upstream/master' into report-conflicting-key 2020-03-09 10:50:33 -07:00
Evan Tschannen 303df197cf Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	bindings/c/test/mako/mako.c
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/NativeAPI.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/LogRouter.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/fdbserver.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/flow.vcxproj
#	flow/flow.vcxproj.filters
#	versions.target
2020-03-06 18:22:46 -08:00
negoyal 927e59b1ce Fix a memory bug. 2020-03-06 12:32:56 -08:00
Evan Tschannen dbfc0cbcc0
Merge pull request #2781 from alexmiller-apple/certificate-refresh
Refresh certificates used for handshaking when they change on disk
2020-03-06 11:12:04 -08:00
A.J. Beamon fd8d569b91 Fix a few typos. 2020-03-05 14:42:07 -08:00
A.J. Beamon 6479034645 Add more metrics to the TransactionMetrics event 2020-03-05 14:00:44 -08:00
Alex Miller 595dd77ed1 Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh 2020-03-04 20:25:42 -08:00
Alex Miller 9b5ef3416e Refactor TLSParams into TLSConfig + LoadedTLSConfig
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.

initTLS now just takes the in-memory bytes and applies them to the ssl
context.

This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
2020-03-04 20:14:47 -08:00
chaoguang 7a76e9556d Merge remote-tracking branch 'upstream/master' into report-conflicting-key 2020-03-04 11:24:39 -08:00
chaoguang 9c50b8369d Change name from private-key-space to special-key-space 2020-03-03 18:35:04 -08:00
chaoguang 94102969fd Merge code from upstream and change ACTOR to member class 2020-03-03 14:55:03 -08:00
chaoguang 409718c964 Merge remote-tracking branch 'upstream/master' into private-keyspace-framework 2020-03-03 13:37:51 -08:00
chaoguang 4ee9ce15fc Change from NULL to nullptr 2020-03-03 00:29:40 -08:00
Evan Tschannen 6054c05963 Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	documentation/sphinx/source/release-notes.rst
#	fdbserver/fdbserver.actor.cpp
#	versions.target
2020-02-28 12:11:05 -08:00
A.J. Beamon d1e1fea42d Our binaries that act like clients (fdbcli, backup and DR binaries) were reporting an unknown client version. Clients did not react if the list of supported versions changed. 2020-02-28 09:35:21 -08:00
Evan Tschannen c3299b8ebe if tls cannot be initialized, throw an error from createDatabase 2020-02-26 18:53:06 -08:00
Evan Tschannen d1598e7c99 set_verify_peers throws an error instead of returning a value 2020-02-26 16:06:16 -08:00
Evan Tschannen 2586bade68 re-added support for configuration TLS options with environment variables 2020-02-26 15:33:48 -08:00
negoyal 7fa1f105c0 Fix compilation error after merging master. 2020-02-26 11:34:41 -08:00
negoyal cd949eca71 Merge branch 'master' into fdb_cache_subfeature2 2020-02-26 11:22:08 -08:00
Evan Tschannen 924d335aa7 Merge branch 'release-6.2'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	flow/Knobs.cpp
#	flow/Knobs.h
2020-02-25 18:25:19 -08:00
Evan Tschannen 13a523a355 fix: commit on first proxy did not always commit to the first proxy 2020-02-25 12:34:31 -08:00
negoyal 8c4cdb1f80 More fixes. 2020-02-25 10:48:57 -08:00
negoyal 7193fa87b8 And some more compilation fixes. 2020-02-25 10:45:02 -08:00
negoyal 746523b090 Yet more compilation fixes. 2020-02-24 17:47:33 -08:00
Alvin Moore 0f64505d0b Merge branch 'release-6.2' of github.com:apple/foundationdb
Needed to pull in changes to build docker
2020-02-23 23:27:53 -08:00
Evan Tschannen 96258b9809 Merge branch 'release-6.2'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/FlowTransport.actor.cpp
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/StorageMetrics.actor.h
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/KVStoreTest.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/genericactors.actor.cpp
#	flow/serialize.h
2020-02-21 19:09:16 -08:00
A.J. Beamon 4c696d5bf2 Merge branch 'release-6.2' into dd-better-rebalance-logging
# Conflicts:
#	fdbserver/DataDistributionQueue.actor.cpp
2020-02-21 17:41:00 -08:00
A.J. Beamon 6810a03283 Add more logging to valley filler and mountain chopper 2020-02-21 10:55:14 -08:00
Evan Tschannen 08c318d28a re-added the connect lock in the fdbcli so that the timeout is not spent before a connection has been initiated (because of the handshake lock) 2020-02-20 10:43:34 -08:00
Evan Tschannen fd8a58b035 re-added support for the TLS_DISABLED flag 2020-02-19 18:37:47 -08:00
Evan Tschannen 761da5a059 code cleanup 2020-02-19 17:59:45 -08:00
negoyal a21f29b586 Bugfixes. Identify the cached range correctly. 2020-02-19 16:27:06 -08:00
Alex Miller 88d36af9c7 Fix --tls_password and add better error logging
This refactors all tls settings into a TLSParams object so that we can
set the password before loading any certificates.

It turns out that the FDBLibTLS code did really nice things with error
logging, but I just didn't understand openssl enough before to realize
what pieces I should be copying.
2020-02-19 00:57:05 -08:00
A.J. Beamon 649fc6ba94
Merge pull request #2329 from davisp/trace-clock-source-network-option
Add network option for the trace clock source
2020-02-15 10:43:00 -08:00
Paul J. Davis 32e285a761 Add network option for the trace clock source
This option allows clients to select the clock source for trace events
similar to the `--traceclock` command line parameter for `fdbserver`.
Using the `realtime` clock sources makes loading event data into
OpenTracing systems like Jaeger more useful.
2020-02-15 11:30:43 -06:00
Xin Dong 1849939bc3 Added a delay to avoid get stuck in a loop because the request is not versioned and thus if a storage server is behind it might not know it has been assigned a shard range that a proxy thinks it has. 2020-02-12 15:01:26 -08:00
Xin Dong 2e1d03cbe7 Addressed AJ's review comments 2020-02-12 14:57:40 -08:00
Xin Dong 03287a0214 Fix build error. 2020-02-12 14:57:40 -08:00
Xin Dong 57f0c11712 Address Evan's review comments 2020-02-12 14:57:40 -08:00
Xin Dong d934aed1d7 Because when the user issue 'getStorageByteSample' on a large key range, which can be as large as the whole DB, we need to change the behavior of 'waitStorageMetricsMultipleLocation' to avoid the case where a target key range got moved/splited by DD and thus the call to 'waitMetircs' on the corresponding storage server will return 'wrong_shard_server' error and thus the whole 'waitStorageMetricsMultipleLocation' will be retried on the large key range. What we want is to do the retry only for the key range that caused the error. 2020-02-12 14:57:40 -08:00
A.J. Beamon ff44bd2b33
Merge pull request #2639 from atn34/atn34/include-port-in-address-default
Enable include_port_in_address by default for api version 700
2020-02-10 09:50:59 -08:00
Markus Pilman e71fe44ee3
Merge branch 'master' into features/icc 2020-02-08 21:33:02 -08:00
Alex Miller e390dbd36c Add a non-FDBLibTLS verify peers framework to new TLS impl 2020-02-06 21:06:52 -08:00
Evan Tschannen 38d8d0d675 fixed simulation 2020-02-06 19:29:31 -08:00
negoyal 85cc35e81e Merge branch 'master' into HEAD 2020-02-05 14:59:55 -08:00
tclinken 058de86b88 Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics 2020-02-05 10:07:36 -08:00
Evan Tschannen 84853dd1fd switched SSL implementation to use boost ssl 2020-02-04 14:56:40 -08:00
Evan Tschannen 8449badb3e
Merge pull request #1868 from dongxinEric/fix/1827/error_instead_of_timeout
Send error back before put the GRV request with PRIORITY_BATCH into t…
2020-02-04 14:32:47 -08:00
mpilman d09e07f1f5 Merge remote-tracking branch 'upstream/master' into features/icc 2020-02-04 10:26:18 -08:00
Evan Tschannen 4524831456
Merge pull request #2518 from vishesh/task/failmon-remove-server
FailureMonitoring: Server processes no longer need to talk to ClusterController
2020-02-03 17:22:50 -08:00
Andrew Noyes 2ce887012c Respect api version for include_port_in_address 2020-02-03 15:25:30 -08:00
Andrew Noyes 07a3051f0e Enable include_port_in_address by default for api version 700
Resolves #2607
2020-02-03 15:10:00 -08:00
Xin Dong 7016f7903b Fixed another build error. Do not use timeReplyIgnoreError since we don not want the logging inside that function and thus that's unnecessary anymore. Change to use ready() which basically ignores the error. 2020-01-31 15:48:29 -08:00
Xin Dong c1f992667b Fix build failure 2020-01-31 14:27:47 -08:00
Xin Dong 8d28c2a7f0 Added two new counters for transaction throttled error and remove the verbose trace event logging. Also changed a chain of 'if' statements into 'if-else' statements since they are mutal exclusive 2020-01-31 14:16:39 -08:00
Xin Dong 1b313a4f7e Address review comments. Rebased with latest master 2020-01-30 14:13:56 -08:00
Xin Dong e21426d12a Send error back to the GRV requests with batch priority when the cluster is saturated, instead of blindly enqueue the requests and let the client timeout. 2020-01-30 14:13:56 -08:00
A.J. Beamon 182dac7cd5 Convert the slow task profiler into a run loop profiler that also logs when the run loop is 100% busy for a knob-configurable duration. 2020-01-28 12:09:37 -08:00
mpilman ec418a4dcb LocationCache should now be updated 2020-01-27 17:03:36 -08:00
mpilman 291d524ffe Some bugfixes, started to add logic for locationCache update 2020-01-27 14:13:54 -08:00
mpilman 8d774d1b81 Refactoring for readability 2020-01-26 21:25:15 -08:00
mpilman 1cca70d4ea Added logic to keep locationCache up to date 2020-01-26 20:53:50 -08:00
mpilman 51717c970d Fixed management api 2020-01-24 11:00:50 -08:00
tclinken c9363e7e28 Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics 2020-01-22 21:02:21 -08:00
Vishesh Yadav daef5f011a Merge remote-tracking branch 'apple/master' into task/failmon-remove-server 2020-01-21 13:20:15 -08:00
Xin Dong b0a1af1288 Added the actual read hot detection algorithm and logging machanism.
- When a shard has a read bandwidth larger than a threshold value(configurable via knob), and it's read-bandwidth/byte-size ratio is also larger than a threshold(configurable via knob), the corresponding shard tracker will run the algorithm
- The algorithm will divide the shard into 10MB(configurable via knob) chunks and try to find the chunk(s) that has large aforementioned ratio
- Then those ranges will be logged into TraceEvents. This will later do more like actually cache them.
2020-01-21 11:19:52 -08:00
Alex Miller eb64eede8d Make a smaller range inaccessable after writing a versionstamped key
A transaction's read version is the lower bound of what a transaction's
commit version could be.  Thus, we can narrow the conflict range of a
versionstamped key, and thus reduce the amount of the keyspace that is
rendered inaccessable, by filling in the read version on the
versionstamped key and using that as the lower bound of the conflict
range.

This allows reads to still be done to versionstampled keys lower than
the read version of the current transaction.
2020-01-16 21:41:59 -08:00
Evan Tschannen 3f9d9d8b84 Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	cmake/FlowCommands.cmake
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/StorageServerInterface.h
#	fdbserver/DataDistributionTracker.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	flow/Knobs.h
#	flow/Platform.cpp
#	versions.target
2020-01-16 18:37:47 -08:00
chaoguang db901d626a Update some var names and make returned list in ResolveTransactionBatchReply as Optional 2020-01-16 14:41:06 -08:00
chaoguang 30b9858cf5 using std::vector instead of VectorRef to avoid unknown arena change behavior 2020-01-16 13:06:36 -08:00
tclinken 1d6ac716a1 Merge remote-tracking branch 'origin' into add-data-distribution-metrics 2020-01-15 13:20:04 -08:00
chaoguang 72d39a31f1 Change to send back index of read_conflict_range not keys, which speeds up but fail to give narrowed keyrange 2020-01-15 10:21:51 -08:00
Evan Tschannen c93ca04ea6 Do not merge more than 100 shards together to avoid creating untrackable shards 2020-01-15 09:33:27 -08:00
chaoguang 1c2b116688 Push sorting and union of overlapped keyranges to clients 2020-01-13 18:08:54 -08:00
Evan Tschannen 4aab9b7bc8 fix: clients would waste time attempting to read from a remote region when it was in the process of catching up 2020-01-10 12:23:59 -08:00
chaoguang 1cdb22c4a4 refine code according to comments 2020-01-09 15:42:21 -08:00
Evan Tschannen 83ad9caf54 implemented a load balancing algorithm which evens out the number of requests processes by each proxy 2020-01-08 01:59:01 -08:00
mpilman c7de801935 Merge remote-tracking branch 'upstream/master' into features/cache-initialization 2020-01-07 16:41:51 -08:00
Vishesh Yadav 6e6cfaff16 Cleanup old Failure Monitoring code 2020-01-07 15:53:32 -08:00
chaoguang b8ffc72cca Use a RYW in transactionInfo to track conflicting keys 2020-01-07 13:55:05 -08:00
mpilman 9c641c7a8e First part of client handling 2020-01-07 11:42:57 -08:00
chaoguang 10719200c3 A hack way to call API through getRange("\xff\xff/conflicting_keys\<start_key>", "\xff\xff/conflicting_keys\<end_key>"). 2020-01-06 11:22:11 -08:00
Alvin Moore 3bf971ba8b Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbserver/storageserver.actor.cpp
2019-12-12 07:13:12 -08:00
A.J. Beamon 290cbb7628 Fix an internal error that occurred when using client buggify and a successful transaction was failed with commit_unknown_result.
Move some buggify errors into a try catch block so that the normal error handling path can run.
Initialize startTime so that it's valid in the case of errors thrown early.
2019-12-09 09:44:48 -08:00
chaoguang b376b947d5 Encode conflicting keys using base64. 2019-12-06 16:21:19 -08:00
chaoguang eacfdf6da3 Add a simple workload, ReportConflictingKeysWorkload, to test correctness of the API and performance overhead added to the resovler. 2019-12-06 16:21:03 -08:00
chaoguang edf52e8c97 First version for reporting conflicting keys 2019-12-02 13:45:39 -08:00
A.J. Beamon ed8d3f163c Rename hgVersion to sourceVersion. 2019-11-15 12:26:51 -08:00
Jon Fu 4d8cfc00f6 added comment explaining quorum 2019-10-18 15:00:00 -07:00
Jon Fu f38d52b82d re-introduced coordinator quorum and added some comments 2019-10-18 14:52:07 -07:00
Jon Fu f67eb2f371 removed smartquorum for coordinators check 2019-10-18 09:44:23 -07:00
Jon Fu efa2f5df6b avoid double overwrite on failed servers and reordered removal of keys 2019-09-26 13:52:53 -07:00
Jon Fu 450a09e117 Code Review Changes 2019-09-24 15:48:50 -07:00
Jon Fu 40ad6f0931 created new reply types to work with flatbuffer serialization requirements 2019-09-18 13:40:18 -07:00
Jon Fu 471e283128 Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed 2019-09-18 11:49:07 -07:00
Evan Tschannen b495cc697b Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	documentation/sphinx/source/release-notes.rst
#	versions.target
2019-09-13 09:25:08 -07:00
A.J. Beamon 41fd3d9467 Merge branch 'master' into remove-unused-ssi-get-version
# Conflicts:
#	fdbclient/StorageServerInterface.h
#	fdbserver/storageserver.actor.cpp
2019-09-10 08:11:43 -07:00
Andrew Noyes 34dedc9a62 Fix whitespace 2019-09-05 16:44:58 -07:00
Andrew Noyes c18c4c1b83 Use a transaction option to control includePort behavior 2019-09-05 14:58:39 -07:00
Andrew Noyes 11f6adf645 Treat \xff\xff prefix as 'includePort' for get_addresses_for_key 2019-09-04 17:47:40 -07:00
Evan Tschannen dc1d055b27
Merge pull request #2042 from senthil-ram/snap_cli_fix
fix fdbcli --exec 'snapshot create.sh' failure
2019-08-30 13:40:38 -07:00
Evan Tschannen a7237c4302
Merge pull request #2045 from atn34/disallow-scalar-network-messages
Disallow scalar network messages
2019-08-30 13:38:54 -07:00
sramamoorthy b3277f2982 Fix #2009 posix compliant args for snapshot binary 2019-08-30 12:54:09 -07:00
Andrew Noyes b5f9e9f307 Move comment above if 2019-08-28 15:21:58 -07:00
Andrew Noyes 6aa0ada7b1 Replace scalar root types with proper messages 2019-08-28 14:40:50 -07:00
sramamoorthy 7a9097ea01 make fdbcli --exec 'snapshot create.sh' to succeed 2019-08-27 16:44:19 -07:00
sramamoorthy 5d87443323 improved error msgs for snapshot cmd 2019-08-27 16:43:52 -07:00
Jon Fu 7414ddd891 cleanup merge/rebase error 2019-08-27 14:39:44 -07:00
Jon Fu 3666c0c776 added more trace lines and added timeout to safety check in test workload 2019-08-27 14:39:44 -07:00
Jon Fu 04d514c483 added a wait to check for master proxies changed and put in a few more trace events 2019-08-27 14:39:44 -07:00
Jon Fu 00c2025d4b fixed removeKeys impl, adjusted test workload, and introduced extra safety checks to NativeAPI and proxy 2019-08-27 14:39:44 -07:00
Jon Fu a07cf5d038 addressed code review comments 2019-08-27 14:39:44 -07:00
Jon Fu 5a877d6b14 added safety check on client to prevent removing all servers from a team 2019-08-27 14:39:43 -07:00
A.J. Beamon 64ce0c3285 Remove the unused getVersion from StorageServerInterface. 2019-08-26 13:53:54 -07:00
A.J. Beamon c4004a4eea Don't count read version requests if we've already started one. Also avoid some other work that only needs to be done if we haven't started a read version request. 2019-08-12 15:55:48 -07:00
Evan Tschannen c9fa7237f1 Merge branch 'master' of github.com:apple/foundationdb 2019-08-06 16:40:14 -07:00
Evan Tschannen ba54508c47 code cleanup 2019-08-06 16:30:30 -07:00
mpilman 370ba8b841 Remove --object-serializer flag from executables 2019-08-06 09:25:40 -07:00
Evan Tschannen 69e7ed3e53
Merge pull request #1932 from etschannen/master
Bug fixes for rare bugs found by simulation
2019-07-30 17:18:30 -07:00
sramamoorthy 63941e0d96 disable DD with a in-memory flag and use in snapv2 2019-07-30 17:04:51 -07:00
Evan Tschannen 5c978f6129 fix: switchConnectionFile could get the proxies out of the clientInfo and continue connecting to the wrong cluster 2019-07-30 16:32:26 -07:00
Evan Tschannen 5c98dcce6d revert the proxy forwarding path, because it is no longer necessary as clients keep a persistent connection open with coordinators 2019-07-27 16:46:22 -07:00
Evan Tschannen b509a441e7 Merge branch 'master' into feature-skip-confirm
# Conflicts:
#	bindings/flow/tester/Tester.actor.cpp
#	bindings/go/src/_stacktester/stacktester.go
#	bindings/java/src/test/com/apple/foundationdb/test/AsyncStackTester.java
#	bindings/java/src/test/com/apple/foundationdb/test/StackTester.java
#	bindings/python/tests/tester.py
#	bindings/ruby/tests/tester.rb
#	documentation/sphinx/source/api-c.rst
#	documentation/sphinx/source/api-python.rst
#	documentation/sphinx/source/api-ruby.rst
#	documentation/sphinx/source/data-modeling.rst
#	documentation/sphinx/source/developer-guide.rst
#	fdbclient/vexillographer/fdb.options
#	fdbserver/MasterProxyServer.actor.cpp
2019-07-27 15:08:13 -07:00
Evan Tschannen d1c7ab325b fix: getConnectionFile could crash when connectionFile is null 2019-07-27 13:02:06 -07:00
Evan Tschannen 1c4028d71e fixed merge conflict error 2019-07-26 16:09:46 -07:00
Evan Tschannen 90e3b50213 Merge branch 'master' into feature-coordinator-connection
# Conflicts:
#	fdbclient/DatabaseContext.h
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/NativeAPI.actor.h
#	fdbserver/workloads/KillRegion.actor.cpp
2019-07-26 15:05:02 -07:00
Evan Tschannen 8149b5b352
Merge pull request #1413 from atn34/change-connection-file
Switch cluster file feature
2019-07-26 13:27:37 -07:00
sramamoorthy 9afd162e2f remove snap v1 related code 2019-07-25 17:29:31 -07:00
Evan Tschannen be5d144b8b added status information on connected clients 2019-07-25 17:15:31 -07:00
sramamoorthy a65c9f92ed get rid of all timeouts and other changes 2019-07-24 15:36:28 -07:00
sramamoorthy a2f2ad96ff code review comments and merge to master changes 2019-07-24 15:36:28 -07:00
sramamoorthy 869f77aef1 Few cosmetic edits and fixes 2019-07-24 15:36:28 -07:00
sramamoorthy ddd4523816 bug fix in timeout & header file re-arrange in DD 2019-07-24 15:36:28 -07:00
sramamoorthy 0598779a40 snap create timeout for client call 2019-07-24 15:36:28 -07:00
sramamoorthy ec9af66fc5 timeout snapshotdatabase call 2019-07-24 15:36:28 -07:00
sramamoorthy 62c14dae72 disable dd during snap and enable in restore 2019-07-24 15:36:28 -07:00
sramamoorthy 209448807d snap v2: fdbclient related changes 2019-07-24 15:36:28 -07:00
Trevor Clinkenbeard 9ad9bd4c1f Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file 2019-07-24 15:22:26 -07:00
Evan Tschannen 4a866290b7 Clients keep a persistent connection open with coordinators to get updates to the list of proxies
Status still needs to be updated with client information with information from the coordinators
2019-07-23 19:22:44 -07:00
A.J. Beamon e29a6ea280
Merge pull request #1871 from bnamasivayam/tr-priority-add-client-log
Track the priority of sampled Transaction as part of GetReadVersion e…
2019-07-22 13:04:52 -07:00
Trevor Clinkenbeard 3507bfd52f Trigger masterProxiesChangeTrigger in switchConnectionFileImpl 2019-07-22 11:13:07 -07:00
Balachandar Namasivayam af267ba053 Track the priority of sampled Transaction as part of GetReadVersion event. 2019-07-19 17:31:49 -07:00
Alex Miller c3a8ae4752
Merge pull request #1791 from fzhjon/fetch-keys-requests-priority
Introduce priority to fetchKeys requests from data distribution
2019-07-19 14:54:51 -07:00
A.J. Beamon bc5c65e5ab
Merge pull request #1756 from jzhou77/db-option
Add transaction getApproximateSize() API
2019-07-19 08:33:24 -07:00
Evan Tschannen 02de53160d only skip confirm epoch live if CAUSAL_READ_RISKY is enabled
time checked on the proxy should be less than the time waited by the master to account for clock speed differences
setting REQUIRED_MIN_RECOVERY_DURATION and ENFORCED_MIN_RECOVERY_DURATION to 0 will go back to the old behavior
2019-07-12 17:58:16 -07:00
A.J. Beamon d5051b08dd Make trace event field lengths (and total event sizes) default knobified and configurable. Add a transaction option to control the field length of transaction debug logging. Make the program start command line field less likely to be truncated. 2019-07-12 16:12:35 -07:00
Andrew Noyes 9001f5d23f Fix bug introduced in merge commit 2019-07-12 14:02:38 -07:00
Andrew Noyes 969957e619 Merge branch 'master' into change-connection-file 2019-07-12 11:39:19 -07:00
A.J. Beamon f4366e69ca Unknown options should not be used internally (i.e. underneath thread-safe API). This commit removes various checks that options exist and replaces them with an ASSERT. 2019-07-11 11:25:39 -07:00
Jon Fu 652cd77689 fixed merge conflicts and use new TaskPriority enum class 2019-07-11 09:56:59 -07:00
Jon Fu f12a3909f3 renamed workloads and made code style adjustments 2019-07-11 09:56:58 -07:00
Jon Fu 1e9d31597c removed extra parameter from getRange, added knob to guard new changes, and adjusted style/formatting in several places 2019-07-11 09:56:58 -07:00
Jon Fu f707d186fe added new priority for fetchkeys requests and adjusted ddmetrics workload to run parallel with mako 2019-07-11 09:56:58 -07:00
Evan Tschannen bbef631872 fix: do not access optionInfo unless the option already exists in the map 2019-07-10 18:48:54 -07:00
Jingyu Zhou 0ad2d2d16e Add binding test for getApproximateSize API 2019-07-10 14:58:07 -07:00
Jingyu Zhou 8ef8b59fcc Use ThreadFuture for getApproximateSize
Change return type to int64_t and fix C and Python binding to use the correct
type.
2019-07-10 14:58:07 -07:00
Jingyu Zhou c50a675bf0 Add transaction getApproximateSize() API
The size is the summation of expected size of mutations, read conflict ranges,
and write conflict ranges.
2019-07-10 14:51:52 -07:00
Evan Tschannen 7e919e361c
Merge pull request #1817 from etschannen/feature-proxy-forward
Proxies will forward clients to the next generation
2019-07-10 13:53:12 -07:00
A.J. Beamon a174178be1 Merge branch 'master' into fix-mvc-default-options
# Conflicts:
#	fdbclient/NativeAPI.actor.cpp
2019-07-09 18:33:14 -07:00
Evan Tschannen 7ad0d1a12b Merge branch 'master' into feature-proxy-forward
# Conflicts:
#	fdbclient/NativeAPI.actor.cpp
2019-07-09 17:26:15 -07:00
Meng Xu cce00bb413
Merge pull request #1808 from ajbeamon/improved-transaction-metrics
Improve TransactionMetrics
2019-07-09 16:46:17 -07:00
A.J. Beamon 15ecba59db Merge branch 'master' into fix-mvc-default-options
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
2019-07-09 15:28:12 -07:00
A.J. Beamon fdd580c878 Restore some variable initializations that were unintentionally removed. 2019-07-09 15:00:11 -07:00
Vishesh Yadav ae6c3e013a monitorClientInfo: Wait for master proxy endpoint failures than triggers
This will not initiate request to get get new set of proxy unless we
know for a fact that endpoint has indeed failed, not just because the
connection to Peer was closed as it was sitting idle.
2019-07-09 14:24:16 -07:00
Trevor Clinkenbeard 1582a2a24d Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file 2019-07-09 13:41:54 -07:00
A.J. Beamon a5a6f8431c Add a random UID to TransactionMetrics in case a client opens multiple connections and also a field to indicate whether the connection is internal. Convert some of the metrics to our Counter object instead of running totals. 2019-07-08 14:01:04 -07:00
Evan Tschannen c348b3da51 After a proxy dies, it will remain alive for an additional 10 seconds to forward clients to the new proxies 2019-07-08 12:53:40 -07:00
A.J. Beamon 6b6012ee7b Add a break to setOption() switch statement. Better detection of missing options (and logging for present options). 2019-07-02 15:42:53 -07:00
A.J. Beamon 3f6ba3d737 Remove space 2019-07-02 11:18:45 -07:00
A.J. Beamon 1cf449db10 Undo formatting changes to otherwise unchanged code 2019-07-02 11:17:47 -07:00
A.J. Beamon e7218bbb28 Restore retry limiting on the client sampling transaction 2019-07-02 11:16:00 -07:00
A.J. Beamon 2035b36257 Make default and persistent options specifyable via annotations to fdb.options. Fix some issues with persisting these options in the multi-version client. Make size limit option not persistent. 2019-06-28 13:24:32 -07:00
Alex Miller bf883d7055 Merge remote-tracking branch 'upstream/master' into flowlock-api 2019-06-25 14:26:50 -07:00
Evan Tschannen 24937d8125
Merge pull request #1744 from vishesh/task/monitor-leader-on-demand
Fix setting enClientFailureMonitor global for client
2019-06-25 13:38:59 -07:00
Alex Miller 7a500cd37f A giant translation of TaskFooPriority -> TaskPriority::Foo
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
Trevor Clinkenbeard f38414e26d Changes error type on aborted read
Throw transaction_too_old instead of all_alternatives_failed when the
cluster file changes while a read request is outstanding
2019-06-24 10:01:41 -07:00
sramamoorthy 5abc891b12 undo the partial retry logic in NativeAPI 2019-06-24 09:36:07 -07:00
Andrew Noyes f7cd9438d3 Use camelCase instead of snake_case 2019-06-24 09:32:14 -07:00
Andrew Noyes 70ebcb3baf Fix quietDatabase timeouts
Update the implementation to interact with the new "don't maintain a
connection to the cluster controller unless necessary" change, and
unlock the originalDB at the end of the workload.
2019-06-22 14:27:30 -07:00
Alec Grieser e8c75505d3
Merge pull request #1725 from jzhou77/db-option
Add transaction size option
2019-06-21 08:25:34 -07:00
Trevor Clinkenbeard afb0dbcd1c Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file 2019-06-20 19:11:29 -07:00
Jingyu Zhou 3a63d053e9 Address review comments for PR#1725 2019-06-20 14:06:32 -07:00
mpilman 844dd60202 FDB compiling with intel compiler 2019-06-20 09:29:01 -07:00
Jingyu Zhou ebc7e7cb4a Use old transaction size limt during reset 2019-06-19 23:16:27 -07:00
Jingyu Zhou 357c9ba0fb Refactor code 2019-06-19 20:41:53 -07:00
Trevor Clinkenbeard 8af92a99ed Abort reads when connection file changes 2019-06-19 16:53:14 -07:00
Jingyu Zhou 9c2257a0e5 Add transaction size option 2019-06-19 07:45:23 -07:00
Vishesh Yadav 6e753dc614 Fix setting enClientFailureMonitor global for client 2019-06-19 02:44:30 -07:00
mpilman 978a5b2d14 Buggified `get`, added comment 2019-06-17 10:27:52 -07:00
mpilman 5b7dc4eeb8 Added proxy_memory_limit_exceeded to possible commit errors 2019-06-17 09:53:43 -07:00
mpilman 24ca369425 Fixed probability calculation 2019-06-17 09:51:44 -07:00
mpilman 7483cba26e Address review comments and added documentation 2019-06-16 09:52:40 -07:00
mpilman 6ea75713cb Overall framework and first buggify entries 2019-06-16 09:09:09 -07:00
Vishesh Yadav f9bfd74dd0 Monitor leader only when proxies are unknown or any dies
FDB clients talk to coordinators to monitor the leader, and then ask
them for proxies. Once proxies are known, the clients still keep
talking to coordinators. This patch, stops this monitoring and
reconnects to coordinators only if one of the proxy is no longer
available.
2019-06-14 13:27:57 -07:00
Jon Fu e2b6acffce Merge branch 'master' into add-data-distribution-metrics 2019-06-12 16:12:16 -07:00
Jon Fu b473a8a830 changed on-the-wire format to use serialized flatbuffers, added cycletest to workload, and fixed small bug in trace 2019-06-11 15:45:06 -07:00
Andrew Noyes bc03421d05 Open Database as switchable only for client 2019-06-11 13:58:22 -07:00
Andrew Noyes d4578a49d8 Use 'SwitchConnectionFile' language uniformly 2019-06-11 13:58:22 -07:00
Andrew Noyes b17956fef5 Remove unnecessary TraceEvent 2019-06-11 13:58:22 -07:00
Andrew Noyes 02e173b601 Add changeConnectionFile method to Transaction
Also add tests
2019-06-11 13:58:22 -07:00
Evan Tschannen 5bdf5aaeb6
Merge pull request #1662 from etschannen/master
Merge 6.1 into master
2019-06-06 13:57:34 -07:00
A.J. Beamon dbfa746494 Don't set the deterministic random seed with platform::getRandomSeed since that is what it does by default. 2019-06-05 11:48:29 -07:00
A.J. Beamon 31e82a6d6d Clients should seed the deterministic random number generator on the network thread. 2019-06-05 11:48:29 -07:00
Evan Tschannen 29b96414e2 Merge branch 'release-6.1'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/NativeAPI.actor.cpp
#	fdbserver/Coordination.actor.cpp
#	flow/Arena.h
#	versions.target
2019-06-03 18:49:35 -07:00
Jon Fu a6bee65f11
Merge branch 'master' into add-data-distribution-metrics 2019-05-30 11:11:49 -07:00
Evan Tschannen 362c2bf1e6 improved the cpu efficiency of printable 2019-05-29 14:55:45 -07:00
sramamoorthy 4bcb590f12 g_random -> deterministicRandom() 2019-05-28 22:07:46 -07:00
sramamoorthy b17ad85497 exec op not supported when log_anti_quorum > 0 2019-05-28 22:07:46 -07:00
sramamoorthy dcb99c5138 txn to disable tlogPop to be timedout
If the disable tlog pop txn takes more than 30 seconds then
tlog will automatically start enabling pop, fix is to timeout
the txn if it takes more than 10 seconds and retry a new
txn to disable tlog pop.
2019-05-28 22:07:46 -07:00
sramamoorthy d3a179b6f9 Multiple bug fixes
- wait for snapTLogFailKeys in a loop, otherwise in some race
  condition it can cause a false assert
- in single region, there does not seem to be a guarantee of
  tagLocalityListKey for a given DC ID, avoiding that assert for now
- to find the workers that are coordinators, looking up by primary
  address is not sufficient in some cases, hence looking by both
  primary and secondary address
- test make files to reflect the location of the new test cases
2019-05-28 22:07:46 -07:00
sramamoorthy bb474dc323 if recovery < fully_recovered then fail the exec
Will do more cleanup, pushing it for a test run in CI
2019-05-28 22:07:46 -07:00
sramamoorthy 925499954b New status cluster_not_fully_recovered 2019-05-28 22:07:46 -07:00
sramamoorthy 591ff96b93 increase retry and use eat instead of parsing 2019-05-28 22:07:46 -07:00
sramamoorthy 936ffc2dde rebase related changes 2019-05-28 22:07:46 -07:00
sramamoorthy ec7834e2f7 code re-orgnaization and address comments 2019-05-28 22:07:46 -07:00
sramamoorthy 61e93a9304 Address review comments and minor fixes 2019-05-28 22:07:46 -07:00
sramamoorthy 89b7a052f5 Bug fixes for snapping coordinators 2019-05-28 22:07:46 -07:00
sramamoorthy 17ecba8313 trace cleanup and other indentation changes 2019-05-28 22:07:46 -07:00
sramamoorthy 898bed66c1 Allow only whitelisted binary path for exec op 2019-05-28 22:07:46 -07:00
sramamoorthy aa79480d69 changes to make fdbfork asynchronous 2019-05-28 22:07:46 -07:00
sramamoorthy f7ba0635ef Make Exec op the first op in the batch 2019-05-28 22:07:46 -07:00
sramamoorthy 382b246930 trace change and retain fitness file after restore 2019-05-28 22:07:46 -07:00
sramamoorthy 3d5998e9dd tlog: when pops are disabled, store them & replay
In Tlogs, disable pop is done whlie taking snapshots. Earlier, tlogs
were ignoring the pops if it got pop requests when pops were
disabled. In this change, instead of ignoring the pop - it remembers
the list of pops in-memory and plays them once the popping is
enabled.
2019-05-28 22:07:46 -07:00
sramamoorthy 72dd067173 Trace message changes and fix few FIXMEs 2019-05-28 22:07:46 -07:00
sramamoorthy 69edefe68b Snapshot based backup and resotre implementation 2019-05-28 22:07:46 -07:00
Jon Fu 03cebd6032 renamed function from storage metrics to data distribution metrics 2019-05-28 09:52:27 -07:00
A.J. Beamon 603721e125 Merge branch 'master' into thread-safe-random-number-generation
# Conflicts:
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/AsyncFileCached.actor.h
#	fdbrpc/genericactors.actor.cpp
#	fdbrpc/sim2.actor.cpp
#	fdbserver/DiskQueue.actor.cpp
#	fdbserver/workloads/BulkSetup.actor.h
#	flow/ActorCollection.actor.cpp
#	flow/Net2.actor.cpp
#	flow/Trace.cpp
#	flow/flow.cpp
2019-05-23 08:35:47 -07:00
Jon Fu 7b4f2907da addressed code review changes 2019-05-22 16:58:28 -07:00
Jon Fu e339ab890b added timeout and shard limiting behaviour 2019-05-21 14:13:09 -07:00
Jon Fu 0984d272e1 initial implementation of get-dd-metrics 2019-05-21 14:13:09 -07:00
Jingyu Zhou b8e7fc1b84 Refactor: add std:: qualifier and use emplace_back 2019-05-17 09:38:50 -10:00
mpilman 9eeb48c43d Allow to turn on object serializer
This commit includes functionality to turn on
the object serializer for network communication.
This is done the following way:

- On incoming connections, a process will detect
  whether the client supports the object serializer
  and will only serialize responses with it, if it does
- On outgoing connections, the command line flag is used
  to determine whether the object serializer should be used
  to send data.

This way, a cluster can run in mixed mode. To upgrade one
can upgrade one process at a time and set the flag one process
at a time.

This is how this is tested on the simulator:
- The command line flag can take three options: on, off,
  and random.
- For off, the object serializer will never we used.
- For on, the object serializer will be always used.
- For random, the simulator will flip a coin for each
  process it starts up.
2019-05-13 14:15:22 -07:00
A.J. Beamon 5f55f3f613 Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used. 2019-05-10 14:01:52 -07:00
Andrew Noyes 781b6ece77 Fix OPEN_FOR_IDE -Wunused-variable warnings
CC #1255, #1173
2019-04-16 15:28:01 -07:00
Andrew Noyes 6207d724f8 Fix all -Wunused-variable warnings 2019-04-15 18:13:00 -07:00
Evan Tschannen cd5c9d91fa
Merge pull request #1443 from etschannen/master
Merge 6.1 into master
2019-04-10 17:43:07 -07:00
Evan Tschannen 21c0ba555c Merge branch 'release-6.1'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	versions.target
2019-04-08 18:38:42 -07:00
A.J. Beamon 538b431656 Apply suggestions from code review 2019-04-08 14:55:58 -07:00
A.J. Beamon a7288e1325 Throw process_behind instead of future_version when all storage nodes on a team are behind. process_behind gets the same backoff behavior as not_committed. Add proxy_memory_limit_exceeded to the retryable predicate. 2019-04-08 14:21:24 -07:00
mpilman d01cbf3455 Addressed code review comments 2019-04-05 13:12:20 -07:00
mpilman 39ecbedd74 Fixed compilation errors on OS X & gcc8 2019-04-05 13:12:19 -07:00
mpilman 1c16f87a4e Remove trace-calls to printable (in non-workloads) 2019-04-05 13:12:19 -07:00
Evan Tschannen 72203ba47a Merge commit '56f3f0b1bc60604f965152d856ae29a591227703' 2019-04-01 18:45:38 -07:00
Evan Tschannen 8714394d42 increase the priority of the client’s version batch timeout, so that we prefer issuing the batch over other possible work 2019-04-01 18:37:40 -07:00
Evan Tschannen d882c060bf Merge commit '5dd6396eed0de0dfea6cf9eecc307995eff5cedc' 2019-03-28 18:00:55 -07:00
Evan Tschannen b6008558d3 renamed BinaryWriter.toStringRef() to .toValue(), because the function now returns a Standalone<StringRef>()
eliminated an unnecessary copy from the proxy commit path
eliminated an unnecessary copy from buffered peek cursor
2019-03-28 11:52:50 -07:00
Jingyu Zhou 7c02ee6fdd Fix compiler warning about unreferenced exception variable 2019-03-26 13:43:47 -07:00
Evan Tschannen 5e03e178de
Merge pull request #1345 from ajbeamon/support-multiple-client-or-worker-issues
Add support for a client or worker having multiple issues.
2019-03-24 17:27:50 -07:00
Evan Tschannen e3400c13ae fixed a performance regression related to broadcasting a read version to too many transactions simultaneously 2019-03-22 18:37:39 -07:00
Alec Grieser 55a9db1994
spaces to tabs 🤮 2019-03-22 12:58:32 -04:00
A.J. Beamon 4eb5715689 Add support for a client or worker having multiple issues. 2019-03-22 08:29:41 -07:00
Alec Grieser 64e45e6826
retry limit and max delay transaction options are no longer reset after onError 2019-03-21 18:50:02 -04:00
Alec Grieser 22f592ce6e
reset the timeout only if the API version is less than 610 to allow transactions with longer timeouts than the database default 2019-03-21 16:47:12 -04:00
Alec Grieser 148a21a31c
set the timeout actor upon transaction creation if necessary
initialize lockaware and maxbackoff in options constructor rather than NativeAPI transaction constructor
2019-03-20 11:13:38 -06:00
Alec Grieser 5f73b7d6ae
Resolves #775: Support setting Transaction options at the Database level
This selects a few useful transaction options that users may want to set for every transaction and adds them to the database options.
2019-03-20 11:13:38 -06:00
Evan Tschannen f9aad46573 made use_provisional_proxies a transaction option 2019-03-19 18:44:37 -07:00
Evan Tschannen 5b9c45ea0b clients do not attempt to connect to provisional proxies 2019-03-19 13:37:50 -07:00
A.J. Beamon 19d27b6271 Merge branch 'master' into fix-compiler-warnings 2019-03-15 10:35:36 -07:00
A.J. Beamon 85b3f11e71 Fix various compiler warnings 2019-03-15 10:34:57 -07:00
A.J. Beamon 59b41b03ed Merge branch 'master' into fix-threadsafedatabase-safety
# Conflicts:
#	fdbclient/NativeAPI.actor.cpp
2019-03-14 14:50:14 -07:00
A.J. Beamon e12d242d18 Address review comments. 2019-03-14 13:42:03 -07:00
Meng Xu 5a10bf5dfc Merge branch 'master' into mengxu/tls-switch-status-PR 2019-03-14 10:35:12 -07:00
A.J. Beamon cab284027d Be more careful about accessing ThreadSafeDatabase's db member. Preallocate the db on the calling thread so that other callers can access the pointer immediately. 2019-03-14 10:26:22 -07:00
Meng Xu e30e2af1f3 ClientKnobs: Add CHECK_CONNECTED_COORDINATOR_NUM_DELAY 2019-03-13 16:54:56 -07:00
Meng Xu 85c24b0067 Merge branch 'master' into mengxu/tls-switch-status-PR 2019-03-12 15:20:54 -07:00
Meng Xu 22f5624494 TLS Status: Reduce cluster controller load
When the coordinator changes, we use delayedAsyncVar() to reduce
the frequency for cluster controller to send the updated connectedCoordinatorsNumDelayed
to clients.
This help reduce the cluster controllers workload
2019-03-12 15:08:10 -07:00
Meng Xu 46f4b02807 TLS Status: Resolve review comments
Use connectedCoordinatorsNumDelayed to reduce the load on cluster controller;
Set connectedCoordinatorsNum to null by default for monitorLeader()
2019-03-11 17:10:08 -07:00
Evan Tschannen 2627bcd35e Merge branch 'master' into feature-metadata-version 2019-03-10 21:13:28 -07:00
Meng Xu 04880e3d4d Merge branch 'master' into mengxu/tls-switch-status-PR 2019-03-06 13:41:16 -08:00
Alex Miller c6a65389ae Remove noexcept macro and replace with BOOST_NOEXCEPT.
BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a
way that is correctly maintained over time.
2019-03-05 22:06:12 -08:00
Meng Xu 820548223a Status: connected_coordinators misc minor changes
Change the rst document file;
Change the coding style to be consistent with the nearby code;
Ensure we always initilize the connectedCoordinatesNum to 0
even when the variable is not used.
2019-03-05 21:45:18 -08:00
Meng Xu b7a52e81e2 Status: Count connected coordinators per client
A client will always try to connect all coordinators.
This commit let Status track the number of connected coordinators
for each client.

This allows us to do canary in coordinators. For example,
when we switch from non-TLS to TLS, we can switch 1 coordinator
from non-TLS to TLS. This can help check if a client has the ability
to connect through TLS.
We can make the non-TLS to TLS switch for each coordinators
one by one. This avoid the risk of losing connection in the switch.
2019-03-05 21:21:23 -08:00
Evan Tschannen 82d957e0bb
Merge pull request #1178 from vishesh/task/issue-963-IPv6
IPv6 Support
2019-03-05 17:14:16 -08:00
Evan Tschannen f1897f3eb6 Merge branch 'master' into feature-metadata-version
# Conflicts:
#	fdbclient/NativeAPI.actor.cpp
2019-03-04 21:06:16 -08:00
Evan Tschannen 988add9fb5 cache the metadataVersion for commits, so that doing setVersion with the commit version of a different transaction will 2019-03-04 16:48:34 -08:00
Trevor Clinkenbeard cbc5df5e93 Removed unnecessary Trace line 2019-03-04 16:31:40 -08:00
Trevor Clinkenbeard 56ae46f89e Client lazily fetches health metrics from proxies 2019-03-04 14:16:39 -08:00
Vishesh Yadav e93cd0ff21 Add some checks and comments to IPv6 changes #963 2019-03-04 14:12:45 -08:00
Vishesh Yadav 41d18db7b9 fix: update the encoding of AddressExclusion in SystemData #963 2019-03-04 14:12:45 -08:00
Vishesh Yadav 57832e625d net: Support IPv6 #963
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.

- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.

- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.

- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
Evan Tschannen 075fdef31a Merge branch 'master' into feature-metadata-version
# Conflicts:
#	fdbclient/DatabaseContext.h
2019-03-03 22:58:45 -08:00
Trevor Clinkenbeard 1576d50ee9 Removed unnecessary Trace line 2019-03-03 17:09:03 -08:00
Trevor Clinkenbeard 015997f2f9 Simplified updateHealthMetricsActor 2019-03-02 18:01:23 -08:00
Trevor Clinkenbeard 39f612d132 Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics 2019-03-02 17:07:00 -08:00
Trevor Clinkenbeard 6c45ffc56f Added exception handling to setNetworkOption
Handle non-integer option values for sendDetailedHealthMetrics option by
throwing invalid_option_value
2019-03-02 15:51:51 -08:00
Evan Tschannen c1de93f467 fix: binary search could get stuck in an infinite loop
fix: avoid picking a read version which could be before the last real commit
fix: we must wait on metadataVersionKey in case it is not already cached
fixed review comments
2019-03-02 13:55:41 -08:00
Meng Xu 94385447bc Status: Get if client configured TLS
To understand if all clients have configured TLS,
we check the tlsoption when a client tries to open database.
This is similar to how we track the versions of multi-version clients.
2019-03-01 15:17:01 -08:00
Evan Tschannen 2168b14834 Merge branch 'master' into feature-metadata-version 2019-02-28 17:45:55 -08:00
Evan Tschannen 3da85f3acd implemented the \xff/metadataVersion key, which can be used by layers to help them cheaply cache metadata and know when their cache is invalid 2019-02-28 17:45:00 -08:00
A.J. Beamon af69ba035a
Merge pull request #1200 from bnamasivayam/master
User provided Client Transaction id merging
2019-02-28 11:16:44 -05:00
Trevor Clinkenbeard d2bde4e55b Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics 2019-02-27 16:30:25 -08:00
Balachandar Namasivayam cc0aac588a Merge branch 'client-transaction-id'
* client-transaction-id:
  Add user provided transaction id as part of the format in ClientTransactionProfileCorrectness workload.
  Addressed review comments.
  Apply suggestions from code review
  Add support for client provided transaction identifier to be logged as part of trace logging or transaction profiling.
2019-02-27 13:11:15 -08:00
Balachandar Namasivayam 7d7db54515 Addressed review comments. 2019-02-27 12:42:45 -08:00
A.J. Beamon 999ee68609 Improve avoidance of transient issues when logging IncorrectClusterFileContents SevWarnAlways events by making it time based. 2019-02-27 10:08:24 -08:00
Trevor Clinkenbeard ff9a7cb2f1 Combined proxy health metrics replies into single message type 2019-02-23 10:13:43 -08:00
Trevor Clinkenbeard edc0c5bf2b Client resends health metrics request when proxies change 2019-02-22 12:30:36 -08:00
Trevor Clinkenbeard fa96b8dd33 Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics 2019-02-20 16:56:16 -08:00
mpilman 0bb60e5a3b Use proper fwd decl in NativeAPI
Also NativeAPI.h -> NativeAPI.actor.h
2019-02-19 15:16:59 -08:00
Vishesh Yadav 907446d0ce Merge remote-tracking branch 'apple/master' into task/tls-upgrade 2019-02-14 11:37:38 -08:00
Andrew Noyes 65136a2ecd Forward declare actors with ACTOR keyword. #1148
There are several more occurrences of this, but they're in .h files that
now need to be .actor.h files. This gets the easy ones out of the way.
2019-02-12 17:56:20 -08:00
Andrew Noyes 067a445e06 Replace unused _ variables with wait(success(...)) 2019-02-12 17:30:30 -08:00
Balachandar Namasivayam 91d859d791 Add support for client provided transaction identifier to be logged as part of trace logging or transaction profiling. 2019-02-12 13:54:24 -08:00
mpilman 6da5971e79 Guard all versions.h to not break old WIN32 build 2019-02-08 16:06:00 -08:00
A.J. Beamon 685242fbfc
Merge pull request #1092 from atn34/trace-format-network-option
Add trace_format network option
2019-02-08 10:52:50 -08:00
mpilman 7e26b4ef0d Address comments from PR 2019-02-07 15:37:04 -08:00
mpilman 8a94d80deb fdbservice and fdbrpc now compiling 2019-02-07 15:37:04 -08:00
Trevor Clinkenbeard 5d15aa2794 Client must actually update health metrics after receiving GetHealthMetricsReply 2019-02-02 17:04:53 -08:00
Trevor Clinkenbeard c22b3c35ad Added updateHealthMetricsActor actor to each DatabaseContext object, to poll proxies for health metrics 2019-02-01 13:23:10 -08:00
Trevor Clinkenbeard e549def5a9 Added send_detailed_health_metrics network option, to be enabled when the client wants to receive per-process health metrics. 2019-01-31 18:46:11 -08:00
Evan Tschannen e9ddd94e27 The failure monitor is given a list of all IP addresses associated with a process
The connect packet includes the correct remote address
Did a lot of code cleanup
Simulation test mixed TLS and non-TLS listeners on the same process
2019-01-31 18:20:14 -08:00
Andrew Noyes e055fdab03 Better diagnostics for unrecognized trace format 2019-01-29 13:08:47 -08:00
Andrew Noyes 768d7678be Add trace_format network option 2019-01-28 15:14:17 -08:00
Evan Tschannen 1d7fec3074 Merge commit '048bfc5c368063d9e009513078dab88be0cbd5b0' into task/tls-upgrade-2
# Conflicts:
#	.gitignore
2019-01-24 17:43:06 -08:00
Alec Grieser 04b94e74c7
Merge pull request #942 from ajbeamon/remove-cluster-from-bindings
Remove cluster from bindings
2019-01-24 14:39:09 -08:00
A.J. Beamon d4d5740282 * Add Optional.map and ErrorOr.map.
* Rename Optional/ErrorOr cast_to to castTo.
* Make printable(Optional<T>) templated rather than restricted to StringRef types.
* Fixes bug in (unused) ErrorOr.castTo where an ErrorOr that was not set would lose its error.
2019-01-11 09:03:38 -08:00
Vishesh Yadav 51b89ae083 WIP 2019-01-09 07:41:02 -08:00
Vishesh Yadav e8e01b2406 Remove unused localAddress parameter from newNet2 and Net2 classes 2018-12-13 13:36:52 -08:00
A.J. Beamon 890a0f3e82 Merge branch 'master' into remove-cluster-from-bindings
# Conflicts:
#	bindings/flow/fdb_flow.actor.cpp
2018-11-26 12:01:03 -08:00
Evan Tschannen 4b5d0b4e2c Merge branch 'release-6.0'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/AsyncFileBlobStore.actor.cpp
#	fdbclient/AsyncFileBlobStore.actor.h
#	fdbclient/BlobStore.actor.cpp
#	fdbclient/BlobStore.h
#	fdbclient/HTTP.actor.cpp
#	fdbclient/ManagementAPI.actor.cpp
#	fdbclient/NativeAPI.actor.cpp
#	fdbrpc/LoadBalance.actor.h
#	fdbrpc/batcher.actor.h
#	fdbrpc/fdbrpc.vcxproj
#	fdbrpc/sim2.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistributionTracker.actor.cpp
#	fdbserver/SimulatedCluster.actor.cpp
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/masterserver.actor.cpp
2018-11-10 13:04:24 -08:00
Evan Tschannen a654183f63
Merge pull request #791 from ajbeamon/remove-cluster-from-iclientapi
Remove cluster from IClientApi (phase 2 of removing DB names)
2018-11-10 10:16:18 -08:00
Evan Tschannen 6a406bae72
Merge pull request #896 from ajbeamon/downgrade-incorrect-cluster-file-event
Downgrade the severity of IncorrectClusterFileContents the first time…
2018-11-10 10:06:36 -08:00
A.J. Beamon 187e507e53 Downgrade the severity of IncorrectClusterFileContents the first time it is logged to avoid transient issues that appear like the cluster file hasn't been updated (e.g. the cluster file is shared between multiple processes). 2018-11-05 09:28:08 -08:00
Evan Tschannen 593fa092f3 added a fixme 2018-11-04 19:54:20 -08:00
Evan Tschannen bf6545a9cf clients cache storage server interfaces individually, instead of as a team. This is needed because in fearless every shard has storage servers from two separate teams, leading to a lot of possible combinations
allAlternatives failed logic was simplified, because we are already doing a global rate limiting, so a per shard limit is unnecessary
reduced unnecessary state variables in waitMetrics requests
2018-11-02 13:15:09 -07:00
Robert Escriva 268093a96d Adjust all includes to be relative to the root.
Remove the use of relative paths.  A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h".  Adjust so that every include references such a header with the
latter form.

Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
A.J. Beamon d5357d2439 Update all of the bindings to use the new C API and deprecate or remove unneeded parts of their APIs. 2018-10-02 15:28:46 -07:00
A.J. Beamon 3eb4355a48 Some various cleanup and fixes. Added "Cluster" to TransactionMetrics trace event. 2018-09-25 15:06:19 -07:00
A.J. Beamon c831051474 This removes the idea of clusters from IClientApi. 2018-09-21 15:58:14 -07:00
Evan Tschannen 90301f497f Merge branch 'release-6.0'
# Conflicts:
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/FlowTransport.actor.cpp
#	fdbrpc/TLSConnection.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/Status.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/StatusWorkload.actor.cpp
#	versions.target
2018-09-05 16:06:33 -07:00
Evan Tschannen d42f0e9c44
Merge pull request #618 from ajbeamon/log-options
Log network, cluster, and database options when they are set.
2018-09-05 10:36:40 -07:00
A.J. Beamon 2a97139d5d This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated. 2018-08-16 10:24:12 -07:00
A.J. Beamon bc0832bb55 Deprecate TLS_plugin option. Don't remove deprecated methods from Go bindings, since that breaks compatibility with old API versions. 2018-08-15 10:11:08 -07:00
A.J. Beamon 3c03ddf91e Remove value logging for options. Add some additional logging for setting an external client library and setting knobs. 2018-08-15 09:42:49 -07:00
Alex Miller fb31a6999f Rewrite all files to have #include actorcompiler.h as the last include. 2018-08-14 15:50:26 -07:00
Alex Miller 535b5701e5 Rewrite all `Void _ = wait(...)` -> `wait(...)`.
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
Alex Miller c38d5247fd Initialize client TLSOptions before first use.
Because initializing them after first use is called SEGFAULT.

This affects both fdbcli and fdbbackup when passed TLS options.  I didn't track
down what introduced this, I just fixed it.
2018-08-06 18:08:45 -07:00
Evan Tschannen 538e684f1c Merge branch 'release-6.0'
# Conflicts:
#	versions.target
2018-08-03 11:41:46 -07:00
Evan Tschannen 1c29275672 call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details. 2018-08-01 14:30:57 -07:00
Alex Miller d7f26cb744 Initialize client TLSOptions before first use.
Because initializing them after first use is called SEGFAULT.

This affects both fdbcli and fdbbackup when passed TLS options.  I didn't track
down what introduced this, I just fixed it.
2018-07-31 16:09:11 -07:00
Alvin Moore 9ea0f0a5ae Fixed problem with stack initialization of TLS Options class 2018-07-03 15:02:53 -07:00
Alvin Moore c3f88dbfe1 Merge branch 'master' of github.com:apple/foundationdb into tls-static 2018-07-01 23:13:57 -07:00
Alvin Moore ef8de426d3 Changed the TLS_DISABLED macro
Disable TLS within Windows until working
2018-06-26 12:08:32 -07:00
Evan Tschannen c6a2207577 fix: stop client sampling during the consistency check 2018-06-20 16:24:37 -07:00
Alvin Moore f8ce1de601 Added support for compiling TLS into binaries 2018-06-20 09:21:23 -07:00
A.J. Beamon 99c9958db7 Some more trace event normalization 2018-06-08 13:57:00 -07:00
A.J. Beamon e5488419cc Attempt to normalize trace events:
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.

Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.

This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
A.J. Beamon 216404de45 Merge branch 'release-5.2' of github.com:apple/foundationdb
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	versions.target
2018-06-06 15:25:37 -07:00
A.J. Beamon e4e06321c7 fix: Read-only transactions that get committed would fail if the readOnly option is set. They would also be counted in the transactionsCommitStarted metric. 2018-06-05 12:10:28 -07:00
Balachandar Namasivayam 529d0497f1 Proxy going OOM when applying high volumes of writes to a proxy, particular in a sudden fashion before ratekeeper can control the workload.
Address this issue by proactively monitoring the memory used by commit batches and dropping requests if a certain memory limit is exceeded.
2018-06-01 15:21:40 -07:00
A.J. Beamon d9c702a9e3 Merge release-5.1 into release-5.2 2018-05-30 09:09:55 -07:00
Evan Tschannen 520aaf731d merge release 5.2 into master 2018-05-10 14:33:08 -07:00
Evan Tschannen 714da951a6 refactored getKeyLocation code to remove duplication 2018-05-10 10:05:41 -07:00
Evan Tschannen 8f984cb2c9 Merge branch 'release-5.2'
# Conflicts:
#	fdbrpc/TLSConnection.h
2018-05-10 09:13:22 -07:00
Evan Tschannen e860cc5994 Merge branch 'release-5.2' of github.com:apple/foundationdb into release-5.2 2018-05-09 17:22:19 -07:00
Evan Tschannen 98e07fd92e fix: clear the client's storage server interface cache on endpoint failure for all request types 2018-05-09 17:11:54 -07:00
Evan Tschannen d3450ce5b0
Merge pull request #343 from bnamasivayam/tls-plugin
Tls plugin
2018-05-09 16:35:53 -07:00
Balachandar Namasivayam 7591931a09 Revert "Make tls_verify_peers as a comma separated string of constraints."
This reverts commit 2033847e4b.
2018-05-09 14:40:36 -07:00
Balachandar Namasivayam 2033847e4b Make tls_verify_peers as a comma separated string of constraints. 2018-05-09 14:37:39 -07:00
Alec Grieser 8de914a81f
use contents() instead of address of in withPrefix and withSuffix ; whitespace fixes 2018-05-09 09:01:22 -07:00
Balachandar Namasivayam e8b7f4b190 Add password support for tls. 2018-05-08 20:46:31 -07:00
Balachandar Namasivayam 49af5d685b Restore previous behavior of not specifying peer_verify option means disable checking. 2018-05-08 18:54:44 -07:00
Balachandar Namasivayam d3b5cfb93c Support latest TLS plugin.
Add support for https in backup.
2018-05-08 16:28:13 -07:00
A.J. Beamon ca720e1540
Merge pull request #297 from apple/release-5.2
Merge 5.2 to Master
2018-05-08 12:04:20 -07:00
Alec Grieser 8987d21df3
use new semantics when writing position of vs key in native api 2018-05-08 08:57:09 -07:00
Alec Grieser 464e2cdbf0
change SetVersionstampedKey and SetVersionstampedValue behavior based on API version to make them consistent 2018-05-08 08:57:09 -07:00
A.J. Beamon f848c68b73 TraceEvents in FileBackupAgent that include BeginFile should be encoding them with printable because of the possible inclusion of the null byte. This causes the detail to terminate prematurely. 2018-05-01 15:35:48 -07:00
Dennis Schafroth 290122637b Using ASSERT_ABORT in destructors 2018-04-23 14:05:10 +02:00
Evan Tschannen 19762b847d Merge branch 'release-5.2'
# Conflicts:
#	fdbserver/DatabaseConfiguration.cpp
#	fdbserver/SimulatedCluster.actor.cpp
2018-04-10 17:02:43 -07:00
Evan Tschannen d3fb17d30a
Merge pull request #74 from bnamasivayam/client-profiling-tests
Client profiling tests - Part 1
2018-03-23 16:52:49 -07:00
Balachandar Namasivayam 89d7cc1093 Minor Bug fixes... 2018-03-15 11:00:47 -07:00
Balachandar Namasivayam 856d2a0a9d Add correctness tests for Client transaction profiling data format. It also includes format check across upgrades. 2018-03-14 12:39:50 -07:00
A.J. Beamon f2c804e14f Reverting changes from merge of master into release-5.2 (b25810711c). Note that we never intend to release master into release-5.2, but if we did we would need to revert this commit. 2018-03-06 10:15:04 -08:00
A.J. Beamon fd4c9d5b54 Fix line endings 2018-03-05 10:37:52 -08:00
Evan Tschannen 37a6a81634 Merge commit '7f6fc3e039c911cd84b8540f7f799fc38a1c1822' into feature-remote-logs
# Conflicts:
#	fdbserver/workloads/RestartRecovery.actor.cpp
2018-02-23 12:33:28 -08:00
Evan Tschannen 719bb5bd0c
Merge pull request #4 from bnamasivayam/getKeyServers-refactor
Having 1000 as the limit for Limit for GetKeyServerLocationsRequest s…
2018-02-22 12:39:48 -08:00
Balachandar Namasivayam 2fe2b522d5 Simple refactor of code... 2018-02-22 12:38:14 -08:00
Alec Grieser e1162e9238 Merge remote-tracking branch 'upstream/release-5.1' 2018-02-22 11:16:12 -08:00
Balachandar Namasivayam e2030db5a8 Refactor getKeyServers to be more readable.
Fix possible memory corruption by returning KeyRange instead of KeyRangeRef in getKeyServers.
Simplify getMasterProxies on DatabaseContext class.
2018-02-21 17:11:50 -08:00
Alec Grieser 0bae9880f1 remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py 2018-02-21 10:25:11 -08:00
Evan Tschannen 42405c78a5 Merge commit '4038bd2fd968d88861f2cebd442ce511724816cb' into feature-remote-logs
# Conflicts:
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/Knobs.cpp
2018-02-10 12:08:52 -08:00
Evan Tschannen fbadcc6eea changing a storage server’s tag must be the first mutations applied in a version, because privatized mutations applied earlier in the same version will use the old tag 2018-02-09 18:21:29 -08:00
A.J. Beamon e06f80bb6e Add client logging for number of logical and physical reads at the NativeAPI layer as well as committed mutation counts and bytes. 2018-02-07 11:56:47 -08:00
A.J. Beamon 0c601d6f85 Purge past version references 2018-01-31 12:05:41 -08:00
A.J. Beamon 41ebc34097 Fix some weird spacing 2018-01-31 11:13:57 -08:00
Stephen Atherton 4dec5423f7 Optimization in backup snapshot dispatching. If the next dispatch version is not ahead of the recently known current version then do not set a scheduled time for new range tasks in order to avoid the overhead and delay of task timeout handling. Adjusted task timeout knobs to avoid large transaction warnings. Removed backtrace() from LargeTransaction trace event. Tweaked suppression in backup trace events. 2018-01-24 17:40:02 -08:00
Stephen Atherton aebbe1dcfd Changed core_versionspersecond knob to int64_t type to avoid integer overflow. Cleaned up backup TraceEvent suppression. Added backtrace to LargeTransaction TraceEvent to make it easier to find the source of large commits in applications using NativeAPI directly. 2018-01-24 11:59:37 -08:00
Evan Tschannen 698ef4117e Merge branch 'master' into feature-remote-logs 2018-01-20 10:34:30 -08:00
A.J. Beamon 16cd0c8f75 Change WaitStorageMetricsPenalty to not be SevWarnAlways unless it happens multiple times in a row (about 2.5 minutes with current knobs). 2018-01-19 16:41:15 -08:00
Evan Tschannen 21482a45e1 Merge branch 'master' into feature-remote-logs
# Conflicts:
#	fdbserver/DBCoreState.h
#	fdbserver/LogSystem.h
#	fdbserver/LogSystemPeekCursor.actor.cpp
#	fdbserver/TLogServer.actor.cpp
2018-01-14 13:40:24 -08:00
Evan Tschannen 645dc5ead6 warmRange needs to get a read version occasionally to prevent it from overwhelming the proxy
quietDatabase waits for all data distribution to be completely finished so that databases are cached in a cleaner state
2018-01-14 12:50:52 -08:00
Evan Tschannen 721a891d1f fix: never request more than 100 shards from the proxy at a time to avoid large packets 2018-01-12 10:51:53 -08:00
Evan Tschannen 4e8bc273b3 added a version of getKeyRangeLocations that checks for endpoint failures
fix: did not add the cluster controller to id_used in all cases
removed obsolete fixmes
2018-01-07 15:32:43 -08:00
Evan Tschannen 3ec45d38a0 Merge branch 'master' into feature-remote-logs
# Conflicts:
#	tests/fast/SidebandWithStatus.txt
#	tests/rare/LargeApiCorrectnessStatus.txt
#	tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-06 13:54:45 -08:00
Evan Tschannen 14e956c4fc fix: getRange did not check for endpoint failures
refactored checking endpoint failures
2018-01-06 13:49:47 -08:00
Evan Tschannen 63751fb0e2 fix: remote logs are not in the log system until the recovery is complete so they cannot be used to determine if this is the correct log system to recover from 2018-01-05 14:15:25 -08:00
Evan Tschannen 2a869a4178 fixed merge errors 2018-01-05 12:17:17 -08:00
Evan Tschannen 5ac4f73978 Merge branch 'release-5.1' into feature-remote-logs
# Conflicts:
#	fdbclient/NativeAPI.actor.cpp
#	fdbrpc/Locality.h
#	fdbrpc/simulator.h
#	fdbserver/ApplyMetadataMutation.h
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/LogSystemPeekCursor.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/SimulatedCluster.actor.cpp
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/TagPartitionedLogSystem.actor.cpp
#	fdbserver/WorkerInterface.h
#	fdbserver/masterserver.actor.cpp
#	flow/Net2.actor.cpp
#	tests/fast/SidebandWithStatus.txt
#	tests/rare/LargeApiCorrectnessStatus.txt
#	tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-05 11:33:42 -08:00
A.J. Beamon 5015119115 Generalize the message that gets displayed in status if a cluster file's contents are incorrect. 2018-01-05 10:29:47 -08:00
Evan Tschannen 982f0dcb1e Merge pull request #222 from cie/alexmiller/drtimefix2
Fix yet another VersionStamp DR issue.
2017-12-20 15:09:23 -08:00
Alex Miller b5a6bc0ab7 Fix VersionStamp problems by instead adding a COMMIT_ON_FIRST_PROXY transaction option.
Simulation identified the fact that we can violate the
VersionStamps-are-always-increasing promise via the following series of events:

1. On proxy 0, dumpData adds commit requests to proxy 0's commit promise stream
2. To any proxy, a client submits the first transaction of abortBackup, which stops further dumpData calls on proxy 0.
3. To any proxy that is not proxy 0, submit a transaction that checks if it needs to upgrade the destination version.
4. The transaction from (3) is committed
5. Transactions from (1) are committed

This is possible because the dumpData transactions have no read conflict
ranges, and thus it's impossible to make them abort due to "conflicting"
transactions.  There's also no promise that if client C sends a commit to proxy
A, and later a client D sends a commit to proxy B, that B must log its commit
after A.  (We only promise that if C is told it was committed before D is told
it was committed, then A committed before B.)

There was a failed attempt to fix this problem.  We tried to add read conflict
ranges to dumpData transactions so that they could be aborted by "conflicting"
transactions.  However, this failed because this now means that dumpData
transactions require conflict resolution, and the stale read version that they
use can cause them to be aborted with a transaction_too_old error.
(Transactions that don't have read conflict ranges will never return
transaction_too_old, because with no reads, the read snapshot version is
effectively meaningless.)  This was never previously possible, so the existing
code doesn't retry commits, and to make things more complicated, the dumpData
commits must be applied in order.  This would require either adding
dependencies to transactions (if A is going to commit then B must also be/have
committed), which would be complicated, or submitting transactions with a fixed
read version, and replaying the failed commits with a higher read version once
we get a transaction_too_old error, which would unacceptably slow down the
maximum throughput of dumpData.

Thus, we've instead elected to add a special transaction option that bypasses
proxy load balancing for commits, and always commits against proxy 0.  We can
know for certain that after the transaction from (2) is committed, all of the
dumpData transactions that will be committed have been added to the commit
promise stream on proxy 0.  Thus, if we enqueue another transaction against
proxy 0, we can know that it will be placed into the promise stream after all
of the dumpData transactions, thus providing the semantics that we require:  no
dumpData transaction can commit after the destination version upgrade
transaction.
2017-12-20 15:04:04 -08:00
Evan Tschannen 1dc9eceb6d optimize GetKeyLocationRequests on the proxy so they only require a single map lookup, instead of doing 3 + (3* [number of ranges]) lookups 2017-12-15 20:13:44 -08:00
Evan Tschannen 73a0a07eac clients ask for key location information directly from the proxy, instead of reading it from the database 2017-12-09 16:10:22 -08:00
Alex Miller e900333dbf Fix a subtle valgrind error.
If there was an error in waiting for the read version, we would attempt to
serialize and eventually commit a CommitTransactionRef that had an
uninitialized read_snapshot.
2017-11-15 19:21:20 -08:00
A.J. Beamon cd085764f1 Do not automatically change a cluster file that does not match what you expect. 2017-11-10 14:12:45 -08:00
Evan Tschannen 33eef4c76b Merge branch 'master' into getrange-perf-improvements 2017-11-04 14:25:45 -07:00
Balachandar Namasivayam 33cc644ba8 Copy CommitTransactionRequest during tryCommit function call as it is anyway copied inside the function. 2017-11-02 17:00:44 -07:00
A.J. Beamon 3cad2676cc Collapse some of the getRange actors into a single actor. Avoid unnecessary comparisons. 2017-11-02 13:39:06 -07:00
A.J. Beamon 07c80484eb Merge branch 'master' into getrange-perf-improvements 2017-11-02 08:42:26 -07:00
Balachandar Namasivayam 3efaaec479 onMasterProxiesChanged was being triggered when any member of ClientDBInfo changed. Change the behavior to be triggered only when proxies field in ClientDBInfo is changed. 2017-11-01 18:29:56 -07:00
Balachandar Namasivayam 1d95e1bfd6 Log actor cancellation error to Client Transaction sampling records. 2017-11-01 11:51:31 -07:00
A.J. Beamon 7cf17df821 Merge branch 'master' into log-group-for-unsupported-clients
# Conflicts:
#	flow/Net2.actor.cpp
#	tests/fast/SidebandWithStatus.txt
#	tests/rare/LargeApiCorrectnessStatus.txt
#	tests/slow/DDBalanceAndRemoveStatus.txt
2017-11-01 11:31:02 -07:00
A.J. Beamon e52f46064a Don't use the results arena for conflict ranges. Rearrange the code to avoid the copy unless necessary. 2017-10-26 09:35:34 -07:00