Andrew Noyes
e1c3c73d4c
Rudimentary read your own read conflict range
2020-04-30 00:30:37 +00:00
A.J. Beamon
573b10f9f5
Fix a merge bug and cleanup some argument passing.
2020-04-27 15:03:56 -07:00
A.J. Beamon
41c517a5dd
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2020-04-27 13:05:24 -07:00
A.J. Beamon
a65e97209a
Fix bug in flag management. Pass priority into ratekeeper's updateRate and use it when setting autothrottles.
2020-04-27 11:34:12 -07:00
A.J. Beamon
239876351b
Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites).
2020-04-24 11:31:16 -07:00
A.J. Beamon
35c18ac60a
Improvements to expiration. Encode throttles with auto/manual and priority in the key to support throttling the same tag with different values in these parameters.
2020-04-23 20:50:40 -07:00
Evan Tschannen
c87aa33941
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/go/src/fdb/generated.go
# documentation/sphinx/source/api-common.rst.inc
# documentation/sphinx/source/api-ruby.rst
# documentation/sphinx/source/release-notes.rst
# fdbclient/FailureMonitorClient.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/OldTLogServer_6_0.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# versions.target
2020-04-23 13:47:53 -07:00
A.J. Beamon
434704fbd9
Various bug fixes
2020-04-22 12:28:51 -07:00
A.J. Beamon
444b338f63
Client clears throttles when the proxy responds to a request with a throttle being absent.
2020-04-21 16:24:02 -07:00
A.J. Beamon
2f69eb42a5
Update client last check time for a throttle when the request is sent, not when the response is received. This prevents sending lots of requests when the check time has expired before we get a response.
2020-04-21 16:23:24 -07:00
A.J. Beamon
ee8447765f
Throttled tags on the client periodically check in with the cluster to update their rates if they get heavily throttled.
2020-04-21 11:56:58 -07:00
Alex Miller
e51d0365cf
Cleanup: Use the shutdown callback for destroying TLS state.
2020-04-20 13:16:16 -07:00
Alex Miller
da8e47ea25
Merge remote-tracking branch 'upstream/release-6.2' into tls-background-eio-thread
2020-04-20 13:15:05 -07:00
Alex Miller
2ce539ef6d
Respect flow<->fdbrpc module boundaries.
...
Which fixes a compilation error due to a circular dependency between
flow.a and fdbrpc.a. However, this is now done at the cost of newNet2
users have to remember to add Net2FileSystem::stop() as a callback.
2020-04-20 02:53:07 -07:00
Alex Miller
94b4f78ea9
Fix clients crashing in TLS code on exit.
...
If client code initiates an FDB operation to a TLS cluster, and then
immediately exits the main thread, then OpenSSL's atexit handler would
potentially run while the network thread is attempting to do TLS
operations, and thus crash.
This commit removes the OpenSSL atexit hander, and instead relies on a
client intentionally ending the network thread to do TLS cleanup. If
the client code exits without stopping the network thread, then we'll
never free OpenSSL data structures, which is the safer thing to do.
2020-04-18 15:48:02 -07:00
A.J. Beamon
78d48a0dad
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbclient/Knobs.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/fdbclient.vcxproj
# fdbserver/MasterProxyServer.actor.cpp
2020-04-17 09:23:18 -07:00
A.J. Beamon
2b66dcd24a
Some more refactoring. Reduce what is sent from RK->MP->clients
2020-04-17 08:07:01 -07:00
Xin Dong
7dd7406c59
Merge branch 'master' into feature/hot-read-key-detection-part-2
2020-04-16 14:54:05 -07:00
A.J. Beamon
0fba8c47be
Checkpoint: Ratekeeper sets absolute limits for tag throttles and enforces them by distributing requests to proxies, who distribute them to clients.
...
A few refactorings.
2020-04-16 14:43:22 -07:00
chaoguang
b011efa8df
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-04-16 10:58:04 -07:00
A.J. Beamon
e104a2e3a6
Merge commit 'cf01233f28a2c42908656a39f458a4475c1d44a3' into run-loop-busy-profiler
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/NativeAPI.actor.h
# fdbserver/fdbserver.actor.cpp
# flow/Net2.actor.cpp
2020-04-14 14:02:24 -07:00
chaoguang
ae1de060c2
Reformat SpecialKeyRangeBaseImpl constructor
2020-04-14 09:10:40 -07:00
chaoguang
65620b00f3
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-04-13 14:21:07 -07:00
Alex Miller
1439de37b5
Convert GetRangeLimits() -> TOO_MANY + ASSERT().
2020-04-12 18:23:14 -07:00
Alex Miller
db071256db
Merge remote-tracking branch 'upstream/master' into reduce-txn-state-store-size
2020-04-12 18:06:10 -07:00
A.J. Beamon
6508c891fc
Make the TagSet sent to the storage servers optional so we can distinguish no tags from unsampled.
2020-04-10 13:29:28 -07:00
A.J. Beamon
55a0d00ad4
Encoding of tags in the database now supports multiple tags per throttle. Remove throttle prefix search.
2020-04-10 10:12:26 -07:00
A.J. Beamon
ebeca10bce
Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients.
2020-04-09 16:55:56 -07:00
chaoguang
34cddd675c
clang-format
2020-04-08 14:33:41 -07:00
chaoguang
20b2984841
fix issues according to andrew's comments
2020-04-08 12:43:25 -07:00
tclinken
52860043c9
Merge remote-tracking branch 'origin' into atomically-update-dependent-knobs
2020-04-08 12:26:21 -07:00
chaoguang
5aee6880a8
fix issues according to andrew's comments
2020-04-08 11:33:07 -07:00
A.J. Beamon
36da61dd9c
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
2020-04-07 21:12:14 -07:00
A.J. Beamon
a1d8623e5f
Various changes to the throttling scheme, including most notably that clients now enforce the throttles with info they receive from the proxy.
2020-04-07 16:28:09 -07:00
Markus Pilman
e4611e8ae4
fix versions.h stupidity
2020-04-06 10:28:55 -07:00
Markus Pilman
8b5780c36c
don't include source and binary dir
...
This forces users to use include paths from the sources root.
So `#include "Arena.h"` won't work anymore, only
`#include "flow/Arena.h"` will.
2020-04-06 10:13:49 -07:00
Alex Miller
6078fd1b18
Convert UID to Tag in keyServers to reduce txnStateStore size
2020-04-05 14:30:09 -07:00
chaoguang
aad8f850f3
refactor conflicting keys feature using special key space
2020-04-04 13:26:11 -07:00
Balachandar Namasivayam
6d570cd865
Change version from 7.0 to 6.3
2020-04-03 21:38:58 -07:00
chaoguang
5b6ae4da7f
A simple intergration to put conflictingKeys into special-key-space
2020-04-03 16:11:20 -07:00
A.J. Beamon
2336f073ad
Checkpointing a bunch of work on throttles. Rudimentary implementation of auto-throttling. Support for manual throttling via fdbcli. Throttles are stored in the system keyspace.
2020-04-03 15:24:14 -07:00
Xin Dong
760dc68b7f
Merge pull request #2869 from dongxinEric/feature/1689/allow-custome-trace-log-file-identifier
...
Allow the user to provide a custome trace log file identifier that wi…
2020-04-03 14:08:17 -07:00
Xin Dong
2ef8f1eb96
Fixed the indentation. Addressed review comments
2020-04-03 13:21:57 -07:00
A.J. Beamon
0816423d78
Merge pull request #2853 from Daniel-B-Smith/debug-transactions
...
Add ability to mark transactions to add debug logging
2020-04-03 11:43:45 -07:00
Daniel Smith
f6a2a7cee9
Move log statement to correct section
2020-04-03 17:31:38 +00:00
Daniel Smith
5092f52042
Fix location of logging code.
2020-04-03 13:19:48 -04:00
Daniel Smith
1ec7b1ebe3
Add logging for server tracing
2020-04-02 22:16:58 +00:00
Daniel Smith
d0210fd54f
Rename option
2020-04-02 21:41:16 +00:00
tclinken
884e92bb49
Atomically update dependent knobs
2020-04-01 15:18:49 -07:00
chaoguang
c8abd9c493
Reformat some code
2020-03-31 18:53:36 -07:00
Xin Dong
6820167d77
Merge branch 'master' into feature/1689/allow-custome-trace-log-file-identifier
2020-03-31 16:50:46 -07:00
Xin Dong
459c2e34cd
Now limit the file identifier to a). has a length no longer than the value defined in knobs; b). can only contain [A-Za-z0-9_] chars.
2020-03-31 16:42:00 -07:00
chaoguang
a63f8b839c
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-03-31 12:05:42 -07:00
chaoguang
82a2196d7d
update space range
2020-03-31 09:34:53 -07:00
Meng Xu
60f6edc3b5
Merge pull request #2860 from zjuLcg/report-conflicting-key-roll-forward
...
Report conflicting key roll forward
2020-03-30 17:33:56 -07:00
chaoguang
64148469e8
clang-format the pr
2020-03-26 15:52:30 -07:00
Meng Xu
1052b23ee1
Merge pull request #2370 from atn34/test-watch-outliving-transaction
...
Test watch outliving transaction
2020-03-26 12:40:38 -07:00
Xin Dong
a0177a9335
Allow the user to provide a custome trace log file identifier that will be used as the prefix of all trace log files created at the client side.
2020-03-26 11:25:05 -07:00
Andrew Noyes
f6af6ec777
Avoid passing self to watch actor
2020-03-26 10:29:23 -07:00
Andrew Noyes
cdb6bbfc85
Test watch outliving transaction
2020-03-26 10:09:03 -07:00
tclinken
7d5ed53215
Allow trace log group to be set after database is created
2020-03-25 13:40:43 -07:00
Andrew Noyes
289487559d
Revert "Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key""
...
This reverts commit 804fe1b22e .
2020-03-24 18:11:15 -07:00
chaoguang
6df92a8b8d
A work version of strict workload test
2020-03-23 23:55:56 -07:00
tclinken
247ab84323
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-03-23 17:01:17 -07:00
A.J. Beamon
197f866824
Don't include internal tools such as fdbcli, backup, and DR in the status client list.
2020-03-23 14:44:04 -07:00
A.J. Beamon
26b7e02d4c
Some initial work to support tagging transactions and passing them around.
2020-03-20 11:23:11 -07:00
Balachandar Namasivayam
804fe1b22e
Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key"
...
This reverts commit 648dc4a933 , reversing
changes made to 487d131b38 .
2020-03-19 21:34:28 -07:00
Evan Tschannen
648dc4a933
Merge pull request #2257 from zjuLcg/report-conflicting-key
...
Report conflicting keys
2020-03-18 13:39:42 -07:00
Daniel Smith
572f08e5fc
Add option to set transaction as debug
2020-03-17 22:21:31 +00:00
Evan Tschannen
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
Evan Tschannen
ed4d02a3e4
Merge pull request #2812 from etschannen/feature-proxy-mem-limit
...
Limit the amount of requests the proxy can queue up in memory
2020-03-16 14:56:56 -07:00
Evan Tschannen
a068d4063f
renamed ProxyGetConsistentReadVersion
2020-03-16 12:11:32 -07:00
A.J. Beamon
031b579ede
Increase priority of the logging of various metrics trace events.
2020-03-13 16:20:23 -07:00
chaoguang
39a37531db
Fix issues according to Andrew's comments
2020-03-13 15:42:15 -07:00
chaoguang
c4c38c5eca
Delete commented code
2020-03-13 12:58:12 -07:00
Evan Tschannen
243c268d9d
Limit the amount of requests the proxy can queue up in memory
2020-03-13 10:17:49 -07:00
chaoguang
6f90228a0b
change to krmSetRangeCoalescing
2020-03-12 11:31:36 -07:00
chaoguang
c2f0c41c52
use krmSetRange
2020-03-11 23:12:38 -07:00
chaoguang
6ae60870fc
use krmSetRange
2020-03-11 13:20:40 -07:00
chaoguang
157d026ba2
debugging
2020-03-10 18:42:49 -07:00
chaoguang
698198a09e
Merge remote-tracking branch 'upstream/master' into report-conflicting-key
2020-03-09 10:50:33 -07:00
Evan Tschannen
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
Evan Tschannen
dbfc0cbcc0
Merge pull request #2781 from alexmiller-apple/certificate-refresh
...
Refresh certificates used for handshaking when they change on disk
2020-03-06 11:12:04 -08:00
A.J. Beamon
fd8d569b91
Fix a few typos.
2020-03-05 14:42:07 -08:00
A.J. Beamon
6479034645
Add more metrics to the TransactionMetrics event
2020-03-05 14:00:44 -08:00
Alex Miller
595dd77ed1
Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh
2020-03-04 20:25:42 -08:00
Alex Miller
9b5ef3416e
Refactor TLSParams into TLSConfig + LoadedTLSConfig
...
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.
initTLS now just takes the in-memory bytes and applies them to the ssl
context.
This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
2020-03-04 20:14:47 -08:00
chaoguang
7a76e9556d
Merge remote-tracking branch 'upstream/master' into report-conflicting-key
2020-03-04 11:24:39 -08:00
chaoguang
9c50b8369d
Change name from private-key-space to special-key-space
2020-03-03 18:35:04 -08:00
chaoguang
94102969fd
Merge code from upstream and change ACTOR to member class
2020-03-03 14:55:03 -08:00
chaoguang
409718c964
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-03-03 13:37:51 -08:00
chaoguang
4ee9ce15fc
Change from NULL to nullptr
2020-03-03 00:29:40 -08:00
Evan Tschannen
6054c05963
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbserver/fdbserver.actor.cpp
# versions.target
2020-02-28 12:11:05 -08:00
A.J. Beamon
d1e1fea42d
Our binaries that act like clients (fdbcli, backup and DR binaries) were reporting an unknown client version. Clients did not react if the list of supported versions changed.
2020-02-28 09:35:21 -08:00
Evan Tschannen
c3299b8ebe
if tls cannot be initialized, throw an error from createDatabase
2020-02-26 18:53:06 -08:00
Evan Tschannen
d1598e7c99
set_verify_peers throws an error instead of returning a value
2020-02-26 16:06:16 -08:00
Evan Tschannen
2586bade68
re-added support for configuration TLS options with environment variables
2020-02-26 15:33:48 -08:00
Evan Tschannen
924d335aa7
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# flow/Knobs.cpp
# flow/Knobs.h
2020-02-25 18:25:19 -08:00
Evan Tschannen
13a523a355
fix: commit on first proxy did not always commit to the first proxy
2020-02-25 12:34:31 -08:00
Alvin Moore
0f64505d0b
Merge branch 'release-6.2' of github.com:apple/foundationdb
...
Needed to pull in changes to build docker
2020-02-23 23:27:53 -08:00
Evan Tschannen
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
A.J. Beamon
4c696d5bf2
Merge branch 'release-6.2' into dd-better-rebalance-logging
...
# Conflicts:
# fdbserver/DataDistributionQueue.actor.cpp
2020-02-21 17:41:00 -08:00
A.J. Beamon
6810a03283
Add more logging to valley filler and mountain chopper
2020-02-21 10:55:14 -08:00
Evan Tschannen
08c318d28a
re-added the connect lock in the fdbcli so that the timeout is not spent before a connection has been initiated (because of the handshake lock)
2020-02-20 10:43:34 -08:00
Evan Tschannen
fd8a58b035
re-added support for the TLS_DISABLED flag
2020-02-19 18:37:47 -08:00
Evan Tschannen
761da5a059
code cleanup
2020-02-19 17:59:45 -08:00
Alex Miller
88d36af9c7
Fix --tls_password and add better error logging
...
This refactors all tls settings into a TLSParams object so that we can
set the password before loading any certificates.
It turns out that the FDBLibTLS code did really nice things with error
logging, but I just didn't understand openssl enough before to realize
what pieces I should be copying.
2020-02-19 00:57:05 -08:00
A.J. Beamon
649fc6ba94
Merge pull request #2329 from davisp/trace-clock-source-network-option
...
Add network option for the trace clock source
2020-02-15 10:43:00 -08:00
Paul J. Davis
32e285a761
Add network option for the trace clock source
...
This option allows clients to select the clock source for trace events
similar to the `--traceclock` command line parameter for `fdbserver`.
Using the `realtime` clock sources makes loading event data into
OpenTracing systems like Jaeger more useful.
2020-02-15 11:30:43 -06:00
Xin Dong
1849939bc3
Added a delay to avoid get stuck in a loop because the request is not versioned and thus if a storage server is behind it might not know it has been assigned a shard range that a proxy thinks it has.
2020-02-12 15:01:26 -08:00
Xin Dong
2e1d03cbe7
Addressed AJ's review comments
2020-02-12 14:57:40 -08:00
Xin Dong
03287a0214
Fix build error.
2020-02-12 14:57:40 -08:00
Xin Dong
57f0c11712
Address Evan's review comments
2020-02-12 14:57:40 -08:00
Xin Dong
d934aed1d7
Because when the user issue 'getStorageByteSample' on a large key range, which can be as large as the whole DB, we need to change the behavior of 'waitStorageMetricsMultipleLocation' to avoid the case where a target key range got moved/splited by DD and thus the call to 'waitMetircs' on the corresponding storage server will return 'wrong_shard_server' error and thus the whole 'waitStorageMetricsMultipleLocation' will be retried on the large key range. What we want is to do the retry only for the key range that caused the error.
2020-02-12 14:57:40 -08:00
A.J. Beamon
ff44bd2b33
Merge pull request #2639 from atn34/atn34/include-port-in-address-default
...
Enable include_port_in_address by default for api version 700
2020-02-10 09:50:59 -08:00
Markus Pilman
e71fe44ee3
Merge branch 'master' into features/icc
2020-02-08 21:33:02 -08:00
Alex Miller
e390dbd36c
Add a non-FDBLibTLS verify peers framework to new TLS impl
2020-02-06 21:06:52 -08:00
Evan Tschannen
38d8d0d675
fixed simulation
2020-02-06 19:29:31 -08:00
tclinken
058de86b88
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-02-05 10:07:36 -08:00
Evan Tschannen
84853dd1fd
switched SSL implementation to use boost ssl
2020-02-04 14:56:40 -08:00
Evan Tschannen
8449badb3e
Merge pull request #1868 from dongxinEric/fix/1827/error_instead_of_timeout
...
Send error back before put the GRV request with PRIORITY_BATCH into t…
2020-02-04 14:32:47 -08:00
mpilman
d09e07f1f5
Merge remote-tracking branch 'upstream/master' into features/icc
2020-02-04 10:26:18 -08:00
Evan Tschannen
4524831456
Merge pull request #2518 from vishesh/task/failmon-remove-server
...
FailureMonitoring: Server processes no longer need to talk to ClusterController
2020-02-03 17:22:50 -08:00
Andrew Noyes
2ce887012c
Respect api version for include_port_in_address
2020-02-03 15:25:30 -08:00
Andrew Noyes
07a3051f0e
Enable include_port_in_address by default for api version 700
...
Resolves #2607
2020-02-03 15:10:00 -08:00
Xin Dong
7016f7903b
Fixed another build error. Do not use timeReplyIgnoreError since we don not want the logging inside that function and thus that's unnecessary anymore. Change to use ready() which basically ignores the error.
2020-01-31 15:48:29 -08:00
Xin Dong
c1f992667b
Fix build failure
2020-01-31 14:27:47 -08:00
Xin Dong
8d28c2a7f0
Added two new counters for transaction throttled error and remove the verbose trace event logging. Also changed a chain of 'if' statements into 'if-else' statements since they are mutal exclusive
2020-01-31 14:16:39 -08:00
Xin Dong
1b313a4f7e
Address review comments. Rebased with latest master
2020-01-30 14:13:56 -08:00
Xin Dong
e21426d12a
Send error back to the GRV requests with batch priority when the cluster is saturated, instead of blindly enqueue the requests and let the client timeout.
2020-01-30 14:13:56 -08:00
A.J. Beamon
182dac7cd5
Convert the slow task profiler into a run loop profiler that also logs when the run loop is 100% busy for a knob-configurable duration.
2020-01-28 12:09:37 -08:00
tclinken
c9363e7e28
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-01-22 21:02:21 -08:00
Vishesh Yadav
daef5f011a
Merge remote-tracking branch 'apple/master' into task/failmon-remove-server
2020-01-21 13:20:15 -08:00
Xin Dong
b0a1af1288
Added the actual read hot detection algorithm and logging machanism.
...
- When a shard has a read bandwidth larger than a threshold value(configurable via knob), and it's read-bandwidth/byte-size ratio is also larger than a threshold(configurable via knob), the corresponding shard tracker will run the algorithm
- The algorithm will divide the shard into 10MB(configurable via knob) chunks and try to find the chunk(s) that has large aforementioned ratio
- Then those ranges will be logged into TraceEvents. This will later do more like actually cache them.
2020-01-21 11:19:52 -08:00
Alex Miller
eb64eede8d
Make a smaller range inaccessable after writing a versionstamped key
...
A transaction's read version is the lower bound of what a transaction's
commit version could be. Thus, we can narrow the conflict range of a
versionstamped key, and thus reduce the amount of the keyspace that is
rendered inaccessable, by filling in the read version on the
versionstamped key and using that as the lower bound of the conflict
range.
This allows reads to still be done to versionstampled keys lower than
the read version of the current transaction.
2020-01-16 21:41:59 -08:00
Evan Tschannen
3f9d9d8b84
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# cmake/FlowCommands.cmake
# documentation/sphinx/source/release-notes.rst
# fdbclient/StorageServerInterface.h
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# flow/Knobs.h
# flow/Platform.cpp
# versions.target
2020-01-16 18:37:47 -08:00
chaoguang
db901d626a
Update some var names and make returned list in ResolveTransactionBatchReply as Optional
2020-01-16 14:41:06 -08:00
chaoguang
30b9858cf5
using std::vector instead of VectorRef to avoid unknown arena change behavior
2020-01-16 13:06:36 -08:00
tclinken
1d6ac716a1
Merge remote-tracking branch 'origin' into add-data-distribution-metrics
2020-01-15 13:20:04 -08:00
chaoguang
72d39a31f1
Change to send back index of read_conflict_range not keys, which speeds up but fail to give narrowed keyrange
2020-01-15 10:21:51 -08:00
Evan Tschannen
c93ca04ea6
Do not merge more than 100 shards together to avoid creating untrackable shards
2020-01-15 09:33:27 -08:00
chaoguang
1c2b116688
Push sorting and union of overlapped keyranges to clients
2020-01-13 18:08:54 -08:00
Evan Tschannen
4aab9b7bc8
fix: clients would waste time attempting to read from a remote region when it was in the process of catching up
2020-01-10 12:23:59 -08:00
chaoguang
1cdb22c4a4
refine code according to comments
2020-01-09 15:42:21 -08:00
Evan Tschannen
83ad9caf54
implemented a load balancing algorithm which evens out the number of requests processes by each proxy
2020-01-08 01:59:01 -08:00
Vishesh Yadav
6e6cfaff16
Cleanup old Failure Monitoring code
2020-01-07 15:53:32 -08:00
chaoguang
b8ffc72cca
Use a RYW in transactionInfo to track conflicting keys
2020-01-07 13:55:05 -08:00
chaoguang
10719200c3
A hack way to call API through getRange("\xff\xff/conflicting_keys\<start_key>", "\xff\xff/conflicting_keys\<end_key>").
2020-01-06 11:22:11 -08:00
Alvin Moore
3bf971ba8b
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/storageserver.actor.cpp
2019-12-12 07:13:12 -08:00
A.J. Beamon
290cbb7628
Fix an internal error that occurred when using client buggify and a successful transaction was failed with commit_unknown_result.
...
Move some buggify errors into a try catch block so that the normal error handling path can run.
Initialize startTime so that it's valid in the case of errors thrown early.
2019-12-09 09:44:48 -08:00