Commit Graph

599 Commits

Author SHA1 Message Date
Andrew Noyes e1c3c73d4c Rudimentary read your own read conflict range 2020-04-30 00:30:37 +00:00
A.J. Beamon 573b10f9f5 Fix a merge bug and cleanup some argument passing. 2020-04-27 15:03:56 -07:00
A.J. Beamon 41c517a5dd Merge branch 'master' into transaction-tagging
# Conflicts:
#	fdbclient/NativeAPI.actor.cpp
2020-04-27 13:05:24 -07:00
A.J. Beamon a65e97209a Fix bug in flag management. Pass priority into ratekeeper's updateRate and use it when setting autothrottles. 2020-04-27 11:34:12 -07:00
A.J. Beamon 239876351b Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites). 2020-04-24 11:31:16 -07:00
A.J. Beamon 35c18ac60a Improvements to expiration. Encode throttles with auto/manual and priority in the key to support throttling the same tag with different values in these parameters. 2020-04-23 20:50:40 -07:00
Evan Tschannen c87aa33941 Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	bindings/go/src/fdb/generated.go
#	documentation/sphinx/source/api-common.rst.inc
#	documentation/sphinx/source/api-ruby.rst
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/FailureMonitorClient.actor.cpp
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/vexillographer/fdb.options
#	fdbrpc/FlowTransport.actor.cpp
#	fdbserver/OldTLogServer_6_0.actor.cpp
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	versions.target
2020-04-23 13:47:53 -07:00
A.J. Beamon 434704fbd9 Various bug fixes 2020-04-22 12:28:51 -07:00
A.J. Beamon 444b338f63 Client clears throttles when the proxy responds to a request with a throttle being absent. 2020-04-21 16:24:02 -07:00
A.J. Beamon 2f69eb42a5 Update client last check time for a throttle when the request is sent, not when the response is received. This prevents sending lots of requests when the check time has expired before we get a response. 2020-04-21 16:23:24 -07:00
A.J. Beamon ee8447765f Throttled tags on the client periodically check in with the cluster to update their rates if they get heavily throttled. 2020-04-21 11:56:58 -07:00
Alex Miller e51d0365cf Cleanup: Use the shutdown callback for destroying TLS state. 2020-04-20 13:16:16 -07:00
Alex Miller da8e47ea25 Merge remote-tracking branch 'upstream/release-6.2' into tls-background-eio-thread 2020-04-20 13:15:05 -07:00
Alex Miller 2ce539ef6d Respect flow<->fdbrpc module boundaries.
Which fixes a compilation error due to a circular dependency between
flow.a and fdbrpc.a.  However, this is now done at the cost of newNet2
users have to remember to add Net2FileSystem::stop() as a callback.
2020-04-20 02:53:07 -07:00
Alex Miller 94b4f78ea9 Fix clients crashing in TLS code on exit.
If client code initiates an FDB operation to a TLS cluster, and then
immediately exits the main thread, then OpenSSL's atexit handler would
potentially run while the network thread is attempting to do TLS
operations, and thus crash.

This commit removes the OpenSSL atexit hander, and instead relies on a
client intentionally ending the network thread to do TLS cleanup.  If
the client code exits without stopping the network thread, then we'll
never free OpenSSL data structures, which is the safer thing to do.
2020-04-18 15:48:02 -07:00
A.J. Beamon 78d48a0dad Merge branch 'master' into transaction-tagging
# Conflicts:
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/Knobs.h
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/fdbclient.vcxproj
#	fdbserver/MasterProxyServer.actor.cpp
2020-04-17 09:23:18 -07:00
A.J. Beamon 2b66dcd24a Some more refactoring. Reduce what is sent from RK->MP->clients 2020-04-17 08:07:01 -07:00
Xin Dong 7dd7406c59
Merge branch 'master' into feature/hot-read-key-detection-part-2 2020-04-16 14:54:05 -07:00
A.J. Beamon 0fba8c47be Checkpoint: Ratekeeper sets absolute limits for tag throttles and enforces them by distributing requests to proxies, who distribute them to clients.
A few refactorings.
2020-04-16 14:43:22 -07:00
chaoguang b011efa8df Merge remote-tracking branch 'upstream/master' into private-keyspace-framework 2020-04-16 10:58:04 -07:00
A.J. Beamon e104a2e3a6 Merge commit 'cf01233f28a2c42908656a39f458a4475c1d44a3' into run-loop-busy-profiler
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/NativeAPI.actor.h
#	fdbserver/fdbserver.actor.cpp
#	flow/Net2.actor.cpp
2020-04-14 14:02:24 -07:00
chaoguang ae1de060c2 Reformat SpecialKeyRangeBaseImpl constructor 2020-04-14 09:10:40 -07:00
chaoguang 65620b00f3 Merge remote-tracking branch 'upstream/master' into private-keyspace-framework 2020-04-13 14:21:07 -07:00
Alex Miller 1439de37b5 Convert GetRangeLimits() -> TOO_MANY + ASSERT(). 2020-04-12 18:23:14 -07:00
Alex Miller db071256db Merge remote-tracking branch 'upstream/master' into reduce-txn-state-store-size 2020-04-12 18:06:10 -07:00
A.J. Beamon 6508c891fc Make the TagSet sent to the storage servers optional so we can distinguish no tags from unsampled. 2020-04-10 13:29:28 -07:00
A.J. Beamon 55a0d00ad4 Encoding of tags in the database now supports multiple tags per throttle. Remove throttle prefix search. 2020-04-10 10:12:26 -07:00
A.J. Beamon ebeca10bce Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients. 2020-04-09 16:55:56 -07:00
chaoguang 34cddd675c clang-format 2020-04-08 14:33:41 -07:00
chaoguang 20b2984841 fix issues according to andrew's comments 2020-04-08 12:43:25 -07:00
tclinken 52860043c9 Merge remote-tracking branch 'origin' into atomically-update-dependent-knobs 2020-04-08 12:26:21 -07:00
chaoguang 5aee6880a8 fix issues according to andrew's comments 2020-04-08 11:33:07 -07:00
A.J. Beamon 36da61dd9c Merge branch 'master' into transaction-tagging
# Conflicts:
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/vexillographer/fdb.options
2020-04-07 21:12:14 -07:00
A.J. Beamon a1d8623e5f Various changes to the throttling scheme, including most notably that clients now enforce the throttles with info they receive from the proxy. 2020-04-07 16:28:09 -07:00
Markus Pilman e4611e8ae4 fix versions.h stupidity 2020-04-06 10:28:55 -07:00
Markus Pilman 8b5780c36c don't include source and binary dir
This forces users to use include paths from the sources root.

So `#include "Arena.h"` won't work anymore, only
`#include "flow/Arena.h"` will.
2020-04-06 10:13:49 -07:00
Alex Miller 6078fd1b18 Convert UID to Tag in keyServers to reduce txnStateStore size 2020-04-05 14:30:09 -07:00
chaoguang aad8f850f3 refactor conflicting keys feature using special key space 2020-04-04 13:26:11 -07:00
Balachandar Namasivayam 6d570cd865 Change version from 7.0 to 6.3 2020-04-03 21:38:58 -07:00
chaoguang 5b6ae4da7f A simple intergration to put conflictingKeys into special-key-space 2020-04-03 16:11:20 -07:00
A.J. Beamon 2336f073ad Checkpointing a bunch of work on throttles. Rudimentary implementation of auto-throttling. Support for manual throttling via fdbcli. Throttles are stored in the system keyspace. 2020-04-03 15:24:14 -07:00
Xin Dong 760dc68b7f
Merge pull request #2869 from dongxinEric/feature/1689/allow-custome-trace-log-file-identifier
Allow the user to provide a custome trace log file identifier that wi…
2020-04-03 14:08:17 -07:00
Xin Dong 2ef8f1eb96 Fixed the indentation. Addressed review comments 2020-04-03 13:21:57 -07:00
A.J. Beamon 0816423d78
Merge pull request #2853 from Daniel-B-Smith/debug-transactions
Add ability to mark transactions to add debug logging
2020-04-03 11:43:45 -07:00
Daniel Smith f6a2a7cee9 Move log statement to correct section 2020-04-03 17:31:38 +00:00
Daniel Smith 5092f52042
Fix location of logging code. 2020-04-03 13:19:48 -04:00
Daniel Smith 1ec7b1ebe3 Add logging for server tracing 2020-04-02 22:16:58 +00:00
Daniel Smith d0210fd54f Rename option 2020-04-02 21:41:16 +00:00
tclinken 884e92bb49 Atomically update dependent knobs 2020-04-01 15:18:49 -07:00
chaoguang c8abd9c493 Reformat some code 2020-03-31 18:53:36 -07:00
Xin Dong 6820167d77
Merge branch 'master' into feature/1689/allow-custome-trace-log-file-identifier 2020-03-31 16:50:46 -07:00
Xin Dong 459c2e34cd Now limit the file identifier to a). has a length no longer than the value defined in knobs; b). can only contain [A-Za-z0-9_] chars. 2020-03-31 16:42:00 -07:00
chaoguang a63f8b839c Merge remote-tracking branch 'upstream/master' into private-keyspace-framework 2020-03-31 12:05:42 -07:00
chaoguang 82a2196d7d update space range 2020-03-31 09:34:53 -07:00
Meng Xu 60f6edc3b5
Merge pull request #2860 from zjuLcg/report-conflicting-key-roll-forward
Report conflicting key roll forward
2020-03-30 17:33:56 -07:00
chaoguang 64148469e8 clang-format the pr 2020-03-26 15:52:30 -07:00
Meng Xu 1052b23ee1
Merge pull request #2370 from atn34/test-watch-outliving-transaction
Test watch outliving transaction
2020-03-26 12:40:38 -07:00
Xin Dong a0177a9335 Allow the user to provide a custome trace log file identifier that will be used as the prefix of all trace log files created at the client side. 2020-03-26 11:25:05 -07:00
Andrew Noyes f6af6ec777 Avoid passing self to watch actor 2020-03-26 10:29:23 -07:00
Andrew Noyes cdb6bbfc85 Test watch outliving transaction 2020-03-26 10:09:03 -07:00
tclinken 7d5ed53215 Allow trace log group to be set after database is created 2020-03-25 13:40:43 -07:00
Andrew Noyes 289487559d Revert "Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key""
This reverts commit 804fe1b22e.
2020-03-24 18:11:15 -07:00
chaoguang 6df92a8b8d A work version of strict workload test 2020-03-23 23:55:56 -07:00
tclinken 247ab84323 Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics 2020-03-23 17:01:17 -07:00
A.J. Beamon 197f866824 Don't include internal tools such as fdbcli, backup, and DR in the status client list. 2020-03-23 14:44:04 -07:00
A.J. Beamon 26b7e02d4c Some initial work to support tagging transactions and passing them around. 2020-03-20 11:23:11 -07:00
Balachandar Namasivayam 804fe1b22e Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key"
This reverts commit 648dc4a933, reversing
changes made to 487d131b38.
2020-03-19 21:34:28 -07:00
Evan Tschannen 648dc4a933
Merge pull request #2257 from zjuLcg/report-conflicting-key
Report conflicting keys
2020-03-18 13:39:42 -07:00
Daniel Smith 572f08e5fc Add option to set transaction as debug 2020-03-17 22:21:31 +00:00
Evan Tschannen e08f0201f1 merge release 6.2 into master 2020-03-17 12:51:47 -07:00
Evan Tschannen ed4d02a3e4
Merge pull request #2812 from etschannen/feature-proxy-mem-limit
Limit the amount of requests the proxy can queue up in memory
2020-03-16 14:56:56 -07:00
Evan Tschannen a068d4063f renamed ProxyGetConsistentReadVersion 2020-03-16 12:11:32 -07:00
A.J. Beamon 031b579ede Increase priority of the logging of various metrics trace events. 2020-03-13 16:20:23 -07:00
chaoguang 39a37531db Fix issues according to Andrew's comments 2020-03-13 15:42:15 -07:00
chaoguang c4c38c5eca Delete commented code 2020-03-13 12:58:12 -07:00
Evan Tschannen 243c268d9d Limit the amount of requests the proxy can queue up in memory 2020-03-13 10:17:49 -07:00
chaoguang 6f90228a0b change to krmSetRangeCoalescing 2020-03-12 11:31:36 -07:00
chaoguang c2f0c41c52 use krmSetRange 2020-03-11 23:12:38 -07:00
chaoguang 6ae60870fc use krmSetRange 2020-03-11 13:20:40 -07:00
chaoguang 157d026ba2 debugging 2020-03-10 18:42:49 -07:00
chaoguang 698198a09e Merge remote-tracking branch 'upstream/master' into report-conflicting-key 2020-03-09 10:50:33 -07:00
Evan Tschannen 303df197cf Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	bindings/c/test/mako/mako.c
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/NativeAPI.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/LogRouter.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/fdbserver.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/flow.vcxproj
#	flow/flow.vcxproj.filters
#	versions.target
2020-03-06 18:22:46 -08:00
Evan Tschannen dbfc0cbcc0
Merge pull request #2781 from alexmiller-apple/certificate-refresh
Refresh certificates used for handshaking when they change on disk
2020-03-06 11:12:04 -08:00
A.J. Beamon fd8d569b91 Fix a few typos. 2020-03-05 14:42:07 -08:00
A.J. Beamon 6479034645 Add more metrics to the TransactionMetrics event 2020-03-05 14:00:44 -08:00
Alex Miller 595dd77ed1 Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh 2020-03-04 20:25:42 -08:00
Alex Miller 9b5ef3416e Refactor TLSParams into TLSConfig + LoadedTLSConfig
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.

initTLS now just takes the in-memory bytes and applies them to the ssl
context.

This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
2020-03-04 20:14:47 -08:00
chaoguang 7a76e9556d Merge remote-tracking branch 'upstream/master' into report-conflicting-key 2020-03-04 11:24:39 -08:00
chaoguang 9c50b8369d Change name from private-key-space to special-key-space 2020-03-03 18:35:04 -08:00
chaoguang 94102969fd Merge code from upstream and change ACTOR to member class 2020-03-03 14:55:03 -08:00
chaoguang 409718c964 Merge remote-tracking branch 'upstream/master' into private-keyspace-framework 2020-03-03 13:37:51 -08:00
chaoguang 4ee9ce15fc Change from NULL to nullptr 2020-03-03 00:29:40 -08:00
Evan Tschannen 6054c05963 Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	documentation/sphinx/source/release-notes.rst
#	fdbserver/fdbserver.actor.cpp
#	versions.target
2020-02-28 12:11:05 -08:00
A.J. Beamon d1e1fea42d Our binaries that act like clients (fdbcli, backup and DR binaries) were reporting an unknown client version. Clients did not react if the list of supported versions changed. 2020-02-28 09:35:21 -08:00
Evan Tschannen c3299b8ebe if tls cannot be initialized, throw an error from createDatabase 2020-02-26 18:53:06 -08:00
Evan Tschannen d1598e7c99 set_verify_peers throws an error instead of returning a value 2020-02-26 16:06:16 -08:00
Evan Tschannen 2586bade68 re-added support for configuration TLS options with environment variables 2020-02-26 15:33:48 -08:00
Evan Tschannen 924d335aa7 Merge branch 'release-6.2'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	flow/Knobs.cpp
#	flow/Knobs.h
2020-02-25 18:25:19 -08:00
Evan Tschannen 13a523a355 fix: commit on first proxy did not always commit to the first proxy 2020-02-25 12:34:31 -08:00
Alvin Moore 0f64505d0b Merge branch 'release-6.2' of github.com:apple/foundationdb
Needed to pull in changes to build docker
2020-02-23 23:27:53 -08:00
Evan Tschannen 96258b9809 Merge branch 'release-6.2'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/FlowTransport.actor.cpp
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/StorageMetrics.actor.h
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/KVStoreTest.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/genericactors.actor.cpp
#	flow/serialize.h
2020-02-21 19:09:16 -08:00
A.J. Beamon 4c696d5bf2 Merge branch 'release-6.2' into dd-better-rebalance-logging
# Conflicts:
#	fdbserver/DataDistributionQueue.actor.cpp
2020-02-21 17:41:00 -08:00
A.J. Beamon 6810a03283 Add more logging to valley filler and mountain chopper 2020-02-21 10:55:14 -08:00
Evan Tschannen 08c318d28a re-added the connect lock in the fdbcli so that the timeout is not spent before a connection has been initiated (because of the handshake lock) 2020-02-20 10:43:34 -08:00
Evan Tschannen fd8a58b035 re-added support for the TLS_DISABLED flag 2020-02-19 18:37:47 -08:00
Evan Tschannen 761da5a059 code cleanup 2020-02-19 17:59:45 -08:00
Alex Miller 88d36af9c7 Fix --tls_password and add better error logging
This refactors all tls settings into a TLSParams object so that we can
set the password before loading any certificates.

It turns out that the FDBLibTLS code did really nice things with error
logging, but I just didn't understand openssl enough before to realize
what pieces I should be copying.
2020-02-19 00:57:05 -08:00
A.J. Beamon 649fc6ba94
Merge pull request #2329 from davisp/trace-clock-source-network-option
Add network option for the trace clock source
2020-02-15 10:43:00 -08:00
Paul J. Davis 32e285a761 Add network option for the trace clock source
This option allows clients to select the clock source for trace events
similar to the `--traceclock` command line parameter for `fdbserver`.
Using the `realtime` clock sources makes loading event data into
OpenTracing systems like Jaeger more useful.
2020-02-15 11:30:43 -06:00
Xin Dong 1849939bc3 Added a delay to avoid get stuck in a loop because the request is not versioned and thus if a storage server is behind it might not know it has been assigned a shard range that a proxy thinks it has. 2020-02-12 15:01:26 -08:00
Xin Dong 2e1d03cbe7 Addressed AJ's review comments 2020-02-12 14:57:40 -08:00
Xin Dong 03287a0214 Fix build error. 2020-02-12 14:57:40 -08:00
Xin Dong 57f0c11712 Address Evan's review comments 2020-02-12 14:57:40 -08:00
Xin Dong d934aed1d7 Because when the user issue 'getStorageByteSample' on a large key range, which can be as large as the whole DB, we need to change the behavior of 'waitStorageMetricsMultipleLocation' to avoid the case where a target key range got moved/splited by DD and thus the call to 'waitMetircs' on the corresponding storage server will return 'wrong_shard_server' error and thus the whole 'waitStorageMetricsMultipleLocation' will be retried on the large key range. What we want is to do the retry only for the key range that caused the error. 2020-02-12 14:57:40 -08:00
A.J. Beamon ff44bd2b33
Merge pull request #2639 from atn34/atn34/include-port-in-address-default
Enable include_port_in_address by default for api version 700
2020-02-10 09:50:59 -08:00
Markus Pilman e71fe44ee3
Merge branch 'master' into features/icc 2020-02-08 21:33:02 -08:00
Alex Miller e390dbd36c Add a non-FDBLibTLS verify peers framework to new TLS impl 2020-02-06 21:06:52 -08:00
Evan Tschannen 38d8d0d675 fixed simulation 2020-02-06 19:29:31 -08:00
tclinken 058de86b88 Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics 2020-02-05 10:07:36 -08:00
Evan Tschannen 84853dd1fd switched SSL implementation to use boost ssl 2020-02-04 14:56:40 -08:00
Evan Tschannen 8449badb3e
Merge pull request #1868 from dongxinEric/fix/1827/error_instead_of_timeout
Send error back before put the GRV request with PRIORITY_BATCH into t…
2020-02-04 14:32:47 -08:00
mpilman d09e07f1f5 Merge remote-tracking branch 'upstream/master' into features/icc 2020-02-04 10:26:18 -08:00
Evan Tschannen 4524831456
Merge pull request #2518 from vishesh/task/failmon-remove-server
FailureMonitoring: Server processes no longer need to talk to ClusterController
2020-02-03 17:22:50 -08:00
Andrew Noyes 2ce887012c Respect api version for include_port_in_address 2020-02-03 15:25:30 -08:00
Andrew Noyes 07a3051f0e Enable include_port_in_address by default for api version 700
Resolves #2607
2020-02-03 15:10:00 -08:00
Xin Dong 7016f7903b Fixed another build error. Do not use timeReplyIgnoreError since we don not want the logging inside that function and thus that's unnecessary anymore. Change to use ready() which basically ignores the error. 2020-01-31 15:48:29 -08:00
Xin Dong c1f992667b Fix build failure 2020-01-31 14:27:47 -08:00
Xin Dong 8d28c2a7f0 Added two new counters for transaction throttled error and remove the verbose trace event logging. Also changed a chain of 'if' statements into 'if-else' statements since they are mutal exclusive 2020-01-31 14:16:39 -08:00
Xin Dong 1b313a4f7e Address review comments. Rebased with latest master 2020-01-30 14:13:56 -08:00
Xin Dong e21426d12a Send error back to the GRV requests with batch priority when the cluster is saturated, instead of blindly enqueue the requests and let the client timeout. 2020-01-30 14:13:56 -08:00
A.J. Beamon 182dac7cd5 Convert the slow task profiler into a run loop profiler that also logs when the run loop is 100% busy for a knob-configurable duration. 2020-01-28 12:09:37 -08:00
tclinken c9363e7e28 Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics 2020-01-22 21:02:21 -08:00
Vishesh Yadav daef5f011a Merge remote-tracking branch 'apple/master' into task/failmon-remove-server 2020-01-21 13:20:15 -08:00
Xin Dong b0a1af1288 Added the actual read hot detection algorithm and logging machanism.
- When a shard has a read bandwidth larger than a threshold value(configurable via knob), and it's read-bandwidth/byte-size ratio is also larger than a threshold(configurable via knob), the corresponding shard tracker will run the algorithm
- The algorithm will divide the shard into 10MB(configurable via knob) chunks and try to find the chunk(s) that has large aforementioned ratio
- Then those ranges will be logged into TraceEvents. This will later do more like actually cache them.
2020-01-21 11:19:52 -08:00
Alex Miller eb64eede8d Make a smaller range inaccessable after writing a versionstamped key
A transaction's read version is the lower bound of what a transaction's
commit version could be.  Thus, we can narrow the conflict range of a
versionstamped key, and thus reduce the amount of the keyspace that is
rendered inaccessable, by filling in the read version on the
versionstamped key and using that as the lower bound of the conflict
range.

This allows reads to still be done to versionstampled keys lower than
the read version of the current transaction.
2020-01-16 21:41:59 -08:00
Evan Tschannen 3f9d9d8b84 Merge branch 'release-6.2'
# Conflicts:
#	CMakeLists.txt
#	cmake/FlowCommands.cmake
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/StorageServerInterface.h
#	fdbserver/DataDistributionTracker.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	flow/Knobs.h
#	flow/Platform.cpp
#	versions.target
2020-01-16 18:37:47 -08:00
chaoguang db901d626a Update some var names and make returned list in ResolveTransactionBatchReply as Optional 2020-01-16 14:41:06 -08:00
chaoguang 30b9858cf5 using std::vector instead of VectorRef to avoid unknown arena change behavior 2020-01-16 13:06:36 -08:00
tclinken 1d6ac716a1 Merge remote-tracking branch 'origin' into add-data-distribution-metrics 2020-01-15 13:20:04 -08:00
chaoguang 72d39a31f1 Change to send back index of read_conflict_range not keys, which speeds up but fail to give narrowed keyrange 2020-01-15 10:21:51 -08:00
Evan Tschannen c93ca04ea6 Do not merge more than 100 shards together to avoid creating untrackable shards 2020-01-15 09:33:27 -08:00
chaoguang 1c2b116688 Push sorting and union of overlapped keyranges to clients 2020-01-13 18:08:54 -08:00
Evan Tschannen 4aab9b7bc8 fix: clients would waste time attempting to read from a remote region when it was in the process of catching up 2020-01-10 12:23:59 -08:00
chaoguang 1cdb22c4a4 refine code according to comments 2020-01-09 15:42:21 -08:00
Evan Tschannen 83ad9caf54 implemented a load balancing algorithm which evens out the number of requests processes by each proxy 2020-01-08 01:59:01 -08:00
Vishesh Yadav 6e6cfaff16 Cleanup old Failure Monitoring code 2020-01-07 15:53:32 -08:00
chaoguang b8ffc72cca Use a RYW in transactionInfo to track conflicting keys 2020-01-07 13:55:05 -08:00
chaoguang 10719200c3 A hack way to call API through getRange("\xff\xff/conflicting_keys\<start_key>", "\xff\xff/conflicting_keys\<end_key>"). 2020-01-06 11:22:11 -08:00
Alvin Moore 3bf971ba8b Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbserver/storageserver.actor.cpp
2019-12-12 07:13:12 -08:00
A.J. Beamon 290cbb7628 Fix an internal error that occurred when using client buggify and a successful transaction was failed with commit_unknown_result.
Move some buggify errors into a try catch block so that the normal error handling path can run.
Initialize startTime so that it's valid in the case of errors thrown early.
2019-12-09 09:44:48 -08:00