Daniel Smith
d0210fd54f
Rename option
2020-04-02 21:41:16 +00:00
negoyal
a0c8946f31
Merge branch 'master' into fdb_cache_subfeature2
2020-04-02 12:27:04 -07:00
tclinken
884e92bb49
Atomically update dependent knobs
2020-04-01 15:18:49 -07:00
chaoguang
c8abd9c493
Reformat some code
2020-03-31 18:53:36 -07:00
Xin Dong
6820167d77
Merge branch 'master' into feature/1689/allow-custome-trace-log-file-identifier
2020-03-31 16:50:46 -07:00
Xin Dong
459c2e34cd
Now limit the file identifier to a). has a length no longer than the value defined in knobs; b). can only contain [A-Za-z0-9_] chars.
2020-03-31 16:42:00 -07:00
chaoguang
a63f8b839c
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-03-31 12:05:42 -07:00
chaoguang
82a2196d7d
update space range
2020-03-31 09:34:53 -07:00
Meng Xu
60f6edc3b5
Merge pull request #2860 from zjuLcg/report-conflicting-key-roll-forward
...
Report conflicting key roll forward
2020-03-30 17:33:56 -07:00
negoyal
e358510e27
Minor compilation fix.
2020-03-26 16:50:58 -07:00
chaoguang
64148469e8
clang-format the pr
2020-03-26 15:52:30 -07:00
negoyal
acaf91ac47
Merge branch 'master' into fdb_cache_subfeature2
2020-03-26 13:33:08 -07:00
Meng Xu
1052b23ee1
Merge pull request #2370 from atn34/test-watch-outliving-transaction
...
Test watch outliving transaction
2020-03-26 12:40:38 -07:00
Xin Dong
a0177a9335
Allow the user to provide a custome trace log file identifier that will be used as the prefix of all trace log files created at the client side.
2020-03-26 11:25:05 -07:00
Andrew Noyes
f6af6ec777
Avoid passing self to watch actor
2020-03-26 10:29:23 -07:00
Andrew Noyes
cdb6bbfc85
Test watch outliving transaction
2020-03-26 10:09:03 -07:00
tclinken
7d5ed53215
Allow trace log group to be set after database is created
2020-03-25 13:40:43 -07:00
Andrew Noyes
289487559d
Revert "Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key""
...
This reverts commit 804fe1b22e .
2020-03-24 18:11:15 -07:00
chaoguang
6df92a8b8d
A work version of strict workload test
2020-03-23 23:55:56 -07:00
tclinken
247ab84323
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-03-23 17:01:17 -07:00
A.J. Beamon
197f866824
Don't include internal tools such as fdbcli, backup, and DR in the status client list.
2020-03-23 14:44:04 -07:00
A.J. Beamon
26b7e02d4c
Some initial work to support tagging transactions and passing them around.
2020-03-20 11:23:11 -07:00
Balachandar Namasivayam
804fe1b22e
Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key"
...
This reverts commit 648dc4a933 , reversing
changes made to 487d131b38 .
2020-03-19 21:34:28 -07:00
Evan Tschannen
648dc4a933
Merge pull request #2257 from zjuLcg/report-conflicting-key
...
Report conflicting keys
2020-03-18 13:39:42 -07:00
Daniel Smith
572f08e5fc
Add option to set transaction as debug
2020-03-17 22:21:31 +00:00
Evan Tschannen
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
Evan Tschannen
ed4d02a3e4
Merge pull request #2812 from etschannen/feature-proxy-mem-limit
...
Limit the amount of requests the proxy can queue up in memory
2020-03-16 14:56:56 -07:00
Evan Tschannen
a068d4063f
renamed ProxyGetConsistentReadVersion
2020-03-16 12:11:32 -07:00
negoyal
99a5cb0572
Fix a corruption bug.
2020-03-13 18:52:34 -07:00
A.J. Beamon
031b579ede
Increase priority of the logging of various metrics trace events.
2020-03-13 16:20:23 -07:00
chaoguang
39a37531db
Fix issues according to Andrew's comments
2020-03-13 15:42:15 -07:00
chaoguang
c4c38c5eca
Delete commented code
2020-03-13 12:58:12 -07:00
Evan Tschannen
243c268d9d
Limit the amount of requests the proxy can queue up in memory
2020-03-13 10:17:49 -07:00
chaoguang
6f90228a0b
change to krmSetRangeCoalescing
2020-03-12 11:31:36 -07:00
chaoguang
c2f0c41c52
use krmSetRange
2020-03-11 23:12:38 -07:00
chaoguang
6ae60870fc
use krmSetRange
2020-03-11 13:20:40 -07:00
chaoguang
157d026ba2
debugging
2020-03-10 18:42:49 -07:00
chaoguang
698198a09e
Merge remote-tracking branch 'upstream/master' into report-conflicting-key
2020-03-09 10:50:33 -07:00
Evan Tschannen
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
negoyal
927e59b1ce
Fix a memory bug.
2020-03-06 12:32:56 -08:00
Evan Tschannen
dbfc0cbcc0
Merge pull request #2781 from alexmiller-apple/certificate-refresh
...
Refresh certificates used for handshaking when they change on disk
2020-03-06 11:12:04 -08:00
A.J. Beamon
fd8d569b91
Fix a few typos.
2020-03-05 14:42:07 -08:00
A.J. Beamon
6479034645
Add more metrics to the TransactionMetrics event
2020-03-05 14:00:44 -08:00
Alex Miller
595dd77ed1
Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh
2020-03-04 20:25:42 -08:00
Alex Miller
9b5ef3416e
Refactor TLSParams into TLSConfig + LoadedTLSConfig
...
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.
initTLS now just takes the in-memory bytes and applies them to the ssl
context.
This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
2020-03-04 20:14:47 -08:00
chaoguang
7a76e9556d
Merge remote-tracking branch 'upstream/master' into report-conflicting-key
2020-03-04 11:24:39 -08:00
chaoguang
9c50b8369d
Change name from private-key-space to special-key-space
2020-03-03 18:35:04 -08:00
chaoguang
94102969fd
Merge code from upstream and change ACTOR to member class
2020-03-03 14:55:03 -08:00
chaoguang
409718c964
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-03-03 13:37:51 -08:00
chaoguang
4ee9ce15fc
Change from NULL to nullptr
2020-03-03 00:29:40 -08:00
Evan Tschannen
6054c05963
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbserver/fdbserver.actor.cpp
# versions.target
2020-02-28 12:11:05 -08:00
A.J. Beamon
d1e1fea42d
Our binaries that act like clients (fdbcli, backup and DR binaries) were reporting an unknown client version. Clients did not react if the list of supported versions changed.
2020-02-28 09:35:21 -08:00
Evan Tschannen
c3299b8ebe
if tls cannot be initialized, throw an error from createDatabase
2020-02-26 18:53:06 -08:00
Evan Tschannen
d1598e7c99
set_verify_peers throws an error instead of returning a value
2020-02-26 16:06:16 -08:00
Evan Tschannen
2586bade68
re-added support for configuration TLS options with environment variables
2020-02-26 15:33:48 -08:00
negoyal
7fa1f105c0
Fix compilation error after merging master.
2020-02-26 11:34:41 -08:00
negoyal
cd949eca71
Merge branch 'master' into fdb_cache_subfeature2
2020-02-26 11:22:08 -08:00
Evan Tschannen
924d335aa7
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# flow/Knobs.cpp
# flow/Knobs.h
2020-02-25 18:25:19 -08:00
Evan Tschannen
13a523a355
fix: commit on first proxy did not always commit to the first proxy
2020-02-25 12:34:31 -08:00
negoyal
8c4cdb1f80
More fixes.
2020-02-25 10:48:57 -08:00
negoyal
7193fa87b8
And some more compilation fixes.
2020-02-25 10:45:02 -08:00
negoyal
746523b090
Yet more compilation fixes.
2020-02-24 17:47:33 -08:00
Alvin Moore
0f64505d0b
Merge branch 'release-6.2' of github.com:apple/foundationdb
...
Needed to pull in changes to build docker
2020-02-23 23:27:53 -08:00
Evan Tschannen
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
A.J. Beamon
4c696d5bf2
Merge branch 'release-6.2' into dd-better-rebalance-logging
...
# Conflicts:
# fdbserver/DataDistributionQueue.actor.cpp
2020-02-21 17:41:00 -08:00
A.J. Beamon
6810a03283
Add more logging to valley filler and mountain chopper
2020-02-21 10:55:14 -08:00
Evan Tschannen
08c318d28a
re-added the connect lock in the fdbcli so that the timeout is not spent before a connection has been initiated (because of the handshake lock)
2020-02-20 10:43:34 -08:00
Evan Tschannen
fd8a58b035
re-added support for the TLS_DISABLED flag
2020-02-19 18:37:47 -08:00
Evan Tschannen
761da5a059
code cleanup
2020-02-19 17:59:45 -08:00
negoyal
a21f29b586
Bugfixes. Identify the cached range correctly.
2020-02-19 16:27:06 -08:00
Alex Miller
88d36af9c7
Fix --tls_password and add better error logging
...
This refactors all tls settings into a TLSParams object so that we can
set the password before loading any certificates.
It turns out that the FDBLibTLS code did really nice things with error
logging, but I just didn't understand openssl enough before to realize
what pieces I should be copying.
2020-02-19 00:57:05 -08:00
A.J. Beamon
649fc6ba94
Merge pull request #2329 from davisp/trace-clock-source-network-option
...
Add network option for the trace clock source
2020-02-15 10:43:00 -08:00
Paul J. Davis
32e285a761
Add network option for the trace clock source
...
This option allows clients to select the clock source for trace events
similar to the `--traceclock` command line parameter for `fdbserver`.
Using the `realtime` clock sources makes loading event data into
OpenTracing systems like Jaeger more useful.
2020-02-15 11:30:43 -06:00
Xin Dong
1849939bc3
Added a delay to avoid get stuck in a loop because the request is not versioned and thus if a storage server is behind it might not know it has been assigned a shard range that a proxy thinks it has.
2020-02-12 15:01:26 -08:00
Xin Dong
2e1d03cbe7
Addressed AJ's review comments
2020-02-12 14:57:40 -08:00
Xin Dong
03287a0214
Fix build error.
2020-02-12 14:57:40 -08:00
Xin Dong
57f0c11712
Address Evan's review comments
2020-02-12 14:57:40 -08:00
Xin Dong
d934aed1d7
Because when the user issue 'getStorageByteSample' on a large key range, which can be as large as the whole DB, we need to change the behavior of 'waitStorageMetricsMultipleLocation' to avoid the case where a target key range got moved/splited by DD and thus the call to 'waitMetircs' on the corresponding storage server will return 'wrong_shard_server' error and thus the whole 'waitStorageMetricsMultipleLocation' will be retried on the large key range. What we want is to do the retry only for the key range that caused the error.
2020-02-12 14:57:40 -08:00
A.J. Beamon
ff44bd2b33
Merge pull request #2639 from atn34/atn34/include-port-in-address-default
...
Enable include_port_in_address by default for api version 700
2020-02-10 09:50:59 -08:00
Markus Pilman
e71fe44ee3
Merge branch 'master' into features/icc
2020-02-08 21:33:02 -08:00
Alex Miller
e390dbd36c
Add a non-FDBLibTLS verify peers framework to new TLS impl
2020-02-06 21:06:52 -08:00
Evan Tschannen
38d8d0d675
fixed simulation
2020-02-06 19:29:31 -08:00
negoyal
85cc35e81e
Merge branch 'master' into HEAD
2020-02-05 14:59:55 -08:00
tclinken
058de86b88
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-02-05 10:07:36 -08:00
Evan Tschannen
84853dd1fd
switched SSL implementation to use boost ssl
2020-02-04 14:56:40 -08:00
Evan Tschannen
8449badb3e
Merge pull request #1868 from dongxinEric/fix/1827/error_instead_of_timeout
...
Send error back before put the GRV request with PRIORITY_BATCH into t…
2020-02-04 14:32:47 -08:00
mpilman
d09e07f1f5
Merge remote-tracking branch 'upstream/master' into features/icc
2020-02-04 10:26:18 -08:00
Evan Tschannen
4524831456
Merge pull request #2518 from vishesh/task/failmon-remove-server
...
FailureMonitoring: Server processes no longer need to talk to ClusterController
2020-02-03 17:22:50 -08:00
Andrew Noyes
2ce887012c
Respect api version for include_port_in_address
2020-02-03 15:25:30 -08:00
Andrew Noyes
07a3051f0e
Enable include_port_in_address by default for api version 700
...
Resolves #2607
2020-02-03 15:10:00 -08:00
Xin Dong
7016f7903b
Fixed another build error. Do not use timeReplyIgnoreError since we don not want the logging inside that function and thus that's unnecessary anymore. Change to use ready() which basically ignores the error.
2020-01-31 15:48:29 -08:00
Xin Dong
c1f992667b
Fix build failure
2020-01-31 14:27:47 -08:00
Xin Dong
8d28c2a7f0
Added two new counters for transaction throttled error and remove the verbose trace event logging. Also changed a chain of 'if' statements into 'if-else' statements since they are mutal exclusive
2020-01-31 14:16:39 -08:00
Xin Dong
1b313a4f7e
Address review comments. Rebased with latest master
2020-01-30 14:13:56 -08:00
Xin Dong
e21426d12a
Send error back to the GRV requests with batch priority when the cluster is saturated, instead of blindly enqueue the requests and let the client timeout.
2020-01-30 14:13:56 -08:00
A.J. Beamon
182dac7cd5
Convert the slow task profiler into a run loop profiler that also logs when the run loop is 100% busy for a knob-configurable duration.
2020-01-28 12:09:37 -08:00
mpilman
ec418a4dcb
LocationCache should now be updated
2020-01-27 17:03:36 -08:00
mpilman
291d524ffe
Some bugfixes, started to add logic for locationCache update
2020-01-27 14:13:54 -08:00
mpilman
8d774d1b81
Refactoring for readability
2020-01-26 21:25:15 -08:00
mpilman
1cca70d4ea
Added logic to keep locationCache up to date
2020-01-26 20:53:50 -08:00
mpilman
51717c970d
Fixed management api
2020-01-24 11:00:50 -08:00
tclinken
c9363e7e28
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-01-22 21:02:21 -08:00
Vishesh Yadav
daef5f011a
Merge remote-tracking branch 'apple/master' into task/failmon-remove-server
2020-01-21 13:20:15 -08:00
Xin Dong
b0a1af1288
Added the actual read hot detection algorithm and logging machanism.
...
- When a shard has a read bandwidth larger than a threshold value(configurable via knob), and it's read-bandwidth/byte-size ratio is also larger than a threshold(configurable via knob), the corresponding shard tracker will run the algorithm
- The algorithm will divide the shard into 10MB(configurable via knob) chunks and try to find the chunk(s) that has large aforementioned ratio
- Then those ranges will be logged into TraceEvents. This will later do more like actually cache them.
2020-01-21 11:19:52 -08:00
Alex Miller
eb64eede8d
Make a smaller range inaccessable after writing a versionstamped key
...
A transaction's read version is the lower bound of what a transaction's
commit version could be. Thus, we can narrow the conflict range of a
versionstamped key, and thus reduce the amount of the keyspace that is
rendered inaccessable, by filling in the read version on the
versionstamped key and using that as the lower bound of the conflict
range.
This allows reads to still be done to versionstampled keys lower than
the read version of the current transaction.
2020-01-16 21:41:59 -08:00
Evan Tschannen
3f9d9d8b84
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# cmake/FlowCommands.cmake
# documentation/sphinx/source/release-notes.rst
# fdbclient/StorageServerInterface.h
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# flow/Knobs.h
# flow/Platform.cpp
# versions.target
2020-01-16 18:37:47 -08:00
chaoguang
db901d626a
Update some var names and make returned list in ResolveTransactionBatchReply as Optional
2020-01-16 14:41:06 -08:00
chaoguang
30b9858cf5
using std::vector instead of VectorRef to avoid unknown arena change behavior
2020-01-16 13:06:36 -08:00
tclinken
1d6ac716a1
Merge remote-tracking branch 'origin' into add-data-distribution-metrics
2020-01-15 13:20:04 -08:00
chaoguang
72d39a31f1
Change to send back index of read_conflict_range not keys, which speeds up but fail to give narrowed keyrange
2020-01-15 10:21:51 -08:00
Evan Tschannen
c93ca04ea6
Do not merge more than 100 shards together to avoid creating untrackable shards
2020-01-15 09:33:27 -08:00
chaoguang
1c2b116688
Push sorting and union of overlapped keyranges to clients
2020-01-13 18:08:54 -08:00
Evan Tschannen
4aab9b7bc8
fix: clients would waste time attempting to read from a remote region when it was in the process of catching up
2020-01-10 12:23:59 -08:00
chaoguang
1cdb22c4a4
refine code according to comments
2020-01-09 15:42:21 -08:00
Evan Tschannen
83ad9caf54
implemented a load balancing algorithm which evens out the number of requests processes by each proxy
2020-01-08 01:59:01 -08:00
mpilman
c7de801935
Merge remote-tracking branch 'upstream/master' into features/cache-initialization
2020-01-07 16:41:51 -08:00
Vishesh Yadav
6e6cfaff16
Cleanup old Failure Monitoring code
2020-01-07 15:53:32 -08:00
chaoguang
b8ffc72cca
Use a RYW in transactionInfo to track conflicting keys
2020-01-07 13:55:05 -08:00
mpilman
9c641c7a8e
First part of client handling
2020-01-07 11:42:57 -08:00
chaoguang
10719200c3
A hack way to call API through getRange("\xff\xff/conflicting_keys\<start_key>", "\xff\xff/conflicting_keys\<end_key>").
2020-01-06 11:22:11 -08:00
Alvin Moore
3bf971ba8b
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/storageserver.actor.cpp
2019-12-12 07:13:12 -08:00
A.J. Beamon
290cbb7628
Fix an internal error that occurred when using client buggify and a successful transaction was failed with commit_unknown_result.
...
Move some buggify errors into a try catch block so that the normal error handling path can run.
Initialize startTime so that it's valid in the case of errors thrown early.
2019-12-09 09:44:48 -08:00
chaoguang
b376b947d5
Encode conflicting keys using base64.
2019-12-06 16:21:19 -08:00
chaoguang
eacfdf6da3
Add a simple workload, ReportConflictingKeysWorkload, to test correctness of the API and performance overhead added to the resovler.
2019-12-06 16:21:03 -08:00
chaoguang
edf52e8c97
First version for reporting conflicting keys
2019-12-02 13:45:39 -08:00
A.J. Beamon
ed8d3f163c
Rename hgVersion to sourceVersion.
2019-11-15 12:26:51 -08:00
Jon Fu
4d8cfc00f6
added comment explaining quorum
2019-10-18 15:00:00 -07:00
Jon Fu
f38d52b82d
re-introduced coordinator quorum and added some comments
2019-10-18 14:52:07 -07:00
Jon Fu
f67eb2f371
removed smartquorum for coordinators check
2019-10-18 09:44:23 -07:00
Jon Fu
efa2f5df6b
avoid double overwrite on failed servers and reordered removal of keys
2019-09-26 13:52:53 -07:00
Jon Fu
450a09e117
Code Review Changes
2019-09-24 15:48:50 -07:00
Jon Fu
40ad6f0931
created new reply types to work with flatbuffer serialization requirements
2019-09-18 13:40:18 -07:00
Jon Fu
471e283128
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-09-18 11:49:07 -07:00
Evan Tschannen
b495cc697b
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-09-13 09:25:08 -07:00
A.J. Beamon
41fd3d9467
Merge branch 'master' into remove-unused-ssi-get-version
...
# Conflicts:
# fdbclient/StorageServerInterface.h
# fdbserver/storageserver.actor.cpp
2019-09-10 08:11:43 -07:00
Andrew Noyes
34dedc9a62
Fix whitespace
2019-09-05 16:44:58 -07:00
Andrew Noyes
c18c4c1b83
Use a transaction option to control includePort behavior
2019-09-05 14:58:39 -07:00
Andrew Noyes
11f6adf645
Treat \xff\xff prefix as 'includePort' for get_addresses_for_key
2019-09-04 17:47:40 -07:00
Evan Tschannen
dc1d055b27
Merge pull request #2042 from senthil-ram/snap_cli_fix
...
fix fdbcli --exec 'snapshot create.sh' failure
2019-08-30 13:40:38 -07:00
Evan Tschannen
a7237c4302
Merge pull request #2045 from atn34/disallow-scalar-network-messages
...
Disallow scalar network messages
2019-08-30 13:38:54 -07:00
sramamoorthy
b3277f2982
Fix #2009 posix compliant args for snapshot binary
2019-08-30 12:54:09 -07:00
Andrew Noyes
b5f9e9f307
Move comment above if
2019-08-28 15:21:58 -07:00
Andrew Noyes
6aa0ada7b1
Replace scalar root types with proper messages
2019-08-28 14:40:50 -07:00
sramamoorthy
7a9097ea01
make fdbcli --exec 'snapshot create.sh' to succeed
2019-08-27 16:44:19 -07:00
sramamoorthy
5d87443323
improved error msgs for snapshot cmd
2019-08-27 16:43:52 -07:00
Jon Fu
7414ddd891
cleanup merge/rebase error
2019-08-27 14:39:44 -07:00
Jon Fu
3666c0c776
added more trace lines and added timeout to safety check in test workload
2019-08-27 14:39:44 -07:00
Jon Fu
04d514c483
added a wait to check for master proxies changed and put in a few more trace events
2019-08-27 14:39:44 -07:00
Jon Fu
00c2025d4b
fixed removeKeys impl, adjusted test workload, and introduced extra safety checks to NativeAPI and proxy
2019-08-27 14:39:44 -07:00
Jon Fu
a07cf5d038
addressed code review comments
2019-08-27 14:39:44 -07:00
Jon Fu
5a877d6b14
added safety check on client to prevent removing all servers from a team
2019-08-27 14:39:43 -07:00
A.J. Beamon
64ce0c3285
Remove the unused getVersion from StorageServerInterface.
2019-08-26 13:53:54 -07:00
A.J. Beamon
c4004a4eea
Don't count read version requests if we've already started one. Also avoid some other work that only needs to be done if we haven't started a read version request.
2019-08-12 15:55:48 -07:00
Evan Tschannen
c9fa7237f1
Merge branch 'master' of github.com:apple/foundationdb
2019-08-06 16:40:14 -07:00
Evan Tschannen
ba54508c47
code cleanup
2019-08-06 16:30:30 -07:00
mpilman
370ba8b841
Remove --object-serializer flag from executables
2019-08-06 09:25:40 -07:00
Evan Tschannen
69e7ed3e53
Merge pull request #1932 from etschannen/master
...
Bug fixes for rare bugs found by simulation
2019-07-30 17:18:30 -07:00
sramamoorthy
63941e0d96
disable DD with a in-memory flag and use in snapv2
2019-07-30 17:04:51 -07:00
Evan Tschannen
5c978f6129
fix: switchConnectionFile could get the proxies out of the clientInfo and continue connecting to the wrong cluster
2019-07-30 16:32:26 -07:00
Evan Tschannen
5c98dcce6d
revert the proxy forwarding path, because it is no longer necessary as clients keep a persistent connection open with coordinators
2019-07-27 16:46:22 -07:00
Evan Tschannen
b509a441e7
Merge branch 'master' into feature-skip-confirm
...
# Conflicts:
# bindings/flow/tester/Tester.actor.cpp
# bindings/go/src/_stacktester/stacktester.go
# bindings/java/src/test/com/apple/foundationdb/test/AsyncStackTester.java
# bindings/java/src/test/com/apple/foundationdb/test/StackTester.java
# bindings/python/tests/tester.py
# bindings/ruby/tests/tester.rb
# documentation/sphinx/source/api-c.rst
# documentation/sphinx/source/api-python.rst
# documentation/sphinx/source/api-ruby.rst
# documentation/sphinx/source/data-modeling.rst
# documentation/sphinx/source/developer-guide.rst
# fdbclient/vexillographer/fdb.options
# fdbserver/MasterProxyServer.actor.cpp
2019-07-27 15:08:13 -07:00
Evan Tschannen
d1c7ab325b
fix: getConnectionFile could crash when connectionFile is null
2019-07-27 13:02:06 -07:00
Evan Tschannen
1c4028d71e
fixed merge conflict error
2019-07-26 16:09:46 -07:00
Evan Tschannen
90e3b50213
Merge branch 'master' into feature-coordinator-connection
...
# Conflicts:
# fdbclient/DatabaseContext.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/workloads/KillRegion.actor.cpp
2019-07-26 15:05:02 -07:00
Evan Tschannen
8149b5b352
Merge pull request #1413 from atn34/change-connection-file
...
Switch cluster file feature
2019-07-26 13:27:37 -07:00
sramamoorthy
9afd162e2f
remove snap v1 related code
2019-07-25 17:29:31 -07:00
Evan Tschannen
be5d144b8b
added status information on connected clients
2019-07-25 17:15:31 -07:00
sramamoorthy
a65c9f92ed
get rid of all timeouts and other changes
2019-07-24 15:36:28 -07:00
sramamoorthy
a2f2ad96ff
code review comments and merge to master changes
2019-07-24 15:36:28 -07:00
sramamoorthy
869f77aef1
Few cosmetic edits and fixes
2019-07-24 15:36:28 -07:00
sramamoorthy
ddd4523816
bug fix in timeout & header file re-arrange in DD
2019-07-24 15:36:28 -07:00
sramamoorthy
0598779a40
snap create timeout for client call
2019-07-24 15:36:28 -07:00
sramamoorthy
ec9af66fc5
timeout snapshotdatabase call
2019-07-24 15:36:28 -07:00
sramamoorthy
62c14dae72
disable dd during snap and enable in restore
2019-07-24 15:36:28 -07:00
sramamoorthy
209448807d
snap v2: fdbclient related changes
2019-07-24 15:36:28 -07:00
Trevor Clinkenbeard
9ad9bd4c1f
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-07-24 15:22:26 -07:00
Evan Tschannen
4a866290b7
Clients keep a persistent connection open with coordinators to get updates to the list of proxies
...
Status still needs to be updated with client information with information from the coordinators
2019-07-23 19:22:44 -07:00
A.J. Beamon
e29a6ea280
Merge pull request #1871 from bnamasivayam/tr-priority-add-client-log
...
Track the priority of sampled Transaction as part of GetReadVersion e…
2019-07-22 13:04:52 -07:00
Trevor Clinkenbeard
3507bfd52f
Trigger masterProxiesChangeTrigger in switchConnectionFileImpl
2019-07-22 11:13:07 -07:00
Balachandar Namasivayam
af267ba053
Track the priority of sampled Transaction as part of GetReadVersion event.
2019-07-19 17:31:49 -07:00
Alex Miller
c3a8ae4752
Merge pull request #1791 from fzhjon/fetch-keys-requests-priority
...
Introduce priority to fetchKeys requests from data distribution
2019-07-19 14:54:51 -07:00
A.J. Beamon
bc5c65e5ab
Merge pull request #1756 from jzhou77/db-option
...
Add transaction getApproximateSize() API
2019-07-19 08:33:24 -07:00
Evan Tschannen
02de53160d
only skip confirm epoch live if CAUSAL_READ_RISKY is enabled
...
time checked on the proxy should be less than the time waited by the master to account for clock speed differences
setting REQUIRED_MIN_RECOVERY_DURATION and ENFORCED_MIN_RECOVERY_DURATION to 0 will go back to the old behavior
2019-07-12 17:58:16 -07:00
A.J. Beamon
d5051b08dd
Make trace event field lengths (and total event sizes) default knobified and configurable. Add a transaction option to control the field length of transaction debug logging. Make the program start command line field less likely to be truncated.
2019-07-12 16:12:35 -07:00
Andrew Noyes
9001f5d23f
Fix bug introduced in merge commit
2019-07-12 14:02:38 -07:00
Andrew Noyes
969957e619
Merge branch 'master' into change-connection-file
2019-07-12 11:39:19 -07:00
A.J. Beamon
f4366e69ca
Unknown options should not be used internally (i.e. underneath thread-safe API). This commit removes various checks that options exist and replaces them with an ASSERT.
2019-07-11 11:25:39 -07:00
Jon Fu
652cd77689
fixed merge conflicts and use new TaskPriority enum class
2019-07-11 09:56:59 -07:00
Jon Fu
f12a3909f3
renamed workloads and made code style adjustments
2019-07-11 09:56:58 -07:00
Jon Fu
1e9d31597c
removed extra parameter from getRange, added knob to guard new changes, and adjusted style/formatting in several places
2019-07-11 09:56:58 -07:00
Jon Fu
f707d186fe
added new priority for fetchkeys requests and adjusted ddmetrics workload to run parallel with mako
2019-07-11 09:56:58 -07:00
Evan Tschannen
bbef631872
fix: do not access optionInfo unless the option already exists in the map
2019-07-10 18:48:54 -07:00
Jingyu Zhou
0ad2d2d16e
Add binding test for getApproximateSize API
2019-07-10 14:58:07 -07:00
Jingyu Zhou
8ef8b59fcc
Use ThreadFuture for getApproximateSize
...
Change return type to int64_t and fix C and Python binding to use the correct
type.
2019-07-10 14:58:07 -07:00
Jingyu Zhou
c50a675bf0
Add transaction getApproximateSize() API
...
The size is the summation of expected size of mutations, read conflict ranges,
and write conflict ranges.
2019-07-10 14:51:52 -07:00
Evan Tschannen
7e919e361c
Merge pull request #1817 from etschannen/feature-proxy-forward
...
Proxies will forward clients to the next generation
2019-07-10 13:53:12 -07:00
A.J. Beamon
a174178be1
Merge branch 'master' into fix-mvc-default-options
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2019-07-09 18:33:14 -07:00
Evan Tschannen
7ad0d1a12b
Merge branch 'master' into feature-proxy-forward
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2019-07-09 17:26:15 -07:00
Meng Xu
cce00bb413
Merge pull request #1808 from ajbeamon/improved-transaction-metrics
...
Improve TransactionMetrics
2019-07-09 16:46:17 -07:00
A.J. Beamon
15ecba59db
Merge branch 'master' into fix-mvc-default-options
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-07-09 15:28:12 -07:00
A.J. Beamon
fdd580c878
Restore some variable initializations that were unintentionally removed.
2019-07-09 15:00:11 -07:00
Vishesh Yadav
ae6c3e013a
monitorClientInfo: Wait for master proxy endpoint failures than triggers
...
This will not initiate request to get get new set of proxy unless we
know for a fact that endpoint has indeed failed, not just because the
connection to Peer was closed as it was sitting idle.
2019-07-09 14:24:16 -07:00
Trevor Clinkenbeard
1582a2a24d
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-07-09 13:41:54 -07:00
A.J. Beamon
a5a6f8431c
Add a random UID to TransactionMetrics in case a client opens multiple connections and also a field to indicate whether the connection is internal. Convert some of the metrics to our Counter object instead of running totals.
2019-07-08 14:01:04 -07:00
Evan Tschannen
c348b3da51
After a proxy dies, it will remain alive for an additional 10 seconds to forward clients to the new proxies
2019-07-08 12:53:40 -07:00
A.J. Beamon
6b6012ee7b
Add a break to setOption() switch statement. Better detection of missing options (and logging for present options).
2019-07-02 15:42:53 -07:00
A.J. Beamon
3f6ba3d737
Remove space
2019-07-02 11:18:45 -07:00
A.J. Beamon
1cf449db10
Undo formatting changes to otherwise unchanged code
2019-07-02 11:17:47 -07:00
A.J. Beamon
e7218bbb28
Restore retry limiting on the client sampling transaction
2019-07-02 11:16:00 -07:00
A.J. Beamon
2035b36257
Make default and persistent options specifyable via annotations to fdb.options. Fix some issues with persisting these options in the multi-version client. Make size limit option not persistent.
2019-06-28 13:24:32 -07:00
Alex Miller
bf883d7055
Merge remote-tracking branch 'upstream/master' into flowlock-api
2019-06-25 14:26:50 -07:00
Evan Tschannen
24937d8125
Merge pull request #1744 from vishesh/task/monitor-leader-on-demand
...
Fix setting enClientFailureMonitor global for client
2019-06-25 13:38:59 -07:00
Alex Miller
7a500cd37f
A giant translation of TaskFooPriority -> TaskPriority::Foo
...
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
Trevor Clinkenbeard
f38414e26d
Changes error type on aborted read
...
Throw transaction_too_old instead of all_alternatives_failed when the
cluster file changes while a read request is outstanding
2019-06-24 10:01:41 -07:00
sramamoorthy
5abc891b12
undo the partial retry logic in NativeAPI
2019-06-24 09:36:07 -07:00
Andrew Noyes
f7cd9438d3
Use camelCase instead of snake_case
2019-06-24 09:32:14 -07:00
Andrew Noyes
70ebcb3baf
Fix quietDatabase timeouts
...
Update the implementation to interact with the new "don't maintain a
connection to the cluster controller unless necessary" change, and
unlock the originalDB at the end of the workload.
2019-06-22 14:27:30 -07:00
Alec Grieser
e8c75505d3
Merge pull request #1725 from jzhou77/db-option
...
Add transaction size option
2019-06-21 08:25:34 -07:00
Trevor Clinkenbeard
afb0dbcd1c
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-06-20 19:11:29 -07:00
Jingyu Zhou
3a63d053e9
Address review comments for PR#1725
2019-06-20 14:06:32 -07:00
mpilman
844dd60202
FDB compiling with intel compiler
2019-06-20 09:29:01 -07:00
Jingyu Zhou
ebc7e7cb4a
Use old transaction size limt during reset
2019-06-19 23:16:27 -07:00
Jingyu Zhou
357c9ba0fb
Refactor code
2019-06-19 20:41:53 -07:00
Trevor Clinkenbeard
8af92a99ed
Abort reads when connection file changes
2019-06-19 16:53:14 -07:00
Jingyu Zhou
9c2257a0e5
Add transaction size option
2019-06-19 07:45:23 -07:00
Vishesh Yadav
6e753dc614
Fix setting enClientFailureMonitor global for client
2019-06-19 02:44:30 -07:00
mpilman
978a5b2d14
Buggified `get`, added comment
2019-06-17 10:27:52 -07:00
mpilman
5b7dc4eeb8
Added proxy_memory_limit_exceeded to possible commit errors
2019-06-17 09:53:43 -07:00
mpilman
24ca369425
Fixed probability calculation
2019-06-17 09:51:44 -07:00
mpilman
7483cba26e
Address review comments and added documentation
2019-06-16 09:52:40 -07:00
mpilman
6ea75713cb
Overall framework and first buggify entries
2019-06-16 09:09:09 -07:00
Vishesh Yadav
f9bfd74dd0
Monitor leader only when proxies are unknown or any dies
...
FDB clients talk to coordinators to monitor the leader, and then ask
them for proxies. Once proxies are known, the clients still keep
talking to coordinators. This patch, stops this monitoring and
reconnects to coordinators only if one of the proxy is no longer
available.
2019-06-14 13:27:57 -07:00
Jon Fu
e2b6acffce
Merge branch 'master' into add-data-distribution-metrics
2019-06-12 16:12:16 -07:00
Jon Fu
b473a8a830
changed on-the-wire format to use serialized flatbuffers, added cycletest to workload, and fixed small bug in trace
2019-06-11 15:45:06 -07:00
Andrew Noyes
bc03421d05
Open Database as switchable only for client
2019-06-11 13:58:22 -07:00
Andrew Noyes
d4578a49d8
Use 'SwitchConnectionFile' language uniformly
2019-06-11 13:58:22 -07:00
Andrew Noyes
b17956fef5
Remove unnecessary TraceEvent
2019-06-11 13:58:22 -07:00
Andrew Noyes
02e173b601
Add changeConnectionFile method to Transaction
...
Also add tests
2019-06-11 13:58:22 -07:00
Evan Tschannen
5bdf5aaeb6
Merge pull request #1662 from etschannen/master
...
Merge 6.1 into master
2019-06-06 13:57:34 -07:00
A.J. Beamon
dbfa746494
Don't set the deterministic random seed with platform::getRandomSeed since that is what it does by default.
2019-06-05 11:48:29 -07:00
A.J. Beamon
31e82a6d6d
Clients should seed the deterministic random number generator on the network thread.
2019-06-05 11:48:29 -07:00
Evan Tschannen
29b96414e2
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/NativeAPI.actor.cpp
# fdbserver/Coordination.actor.cpp
# flow/Arena.h
# versions.target
2019-06-03 18:49:35 -07:00
Jon Fu
a6bee65f11
Merge branch 'master' into add-data-distribution-metrics
2019-05-30 11:11:49 -07:00
Evan Tschannen
362c2bf1e6
improved the cpu efficiency of printable
2019-05-29 14:55:45 -07:00
sramamoorthy
4bcb590f12
g_random -> deterministicRandom()
2019-05-28 22:07:46 -07:00
sramamoorthy
b17ad85497
exec op not supported when log_anti_quorum > 0
2019-05-28 22:07:46 -07:00
sramamoorthy
dcb99c5138
txn to disable tlogPop to be timedout
...
If the disable tlog pop txn takes more than 30 seconds then
tlog will automatically start enabling pop, fix is to timeout
the txn if it takes more than 10 seconds and retry a new
txn to disable tlog pop.
2019-05-28 22:07:46 -07:00
sramamoorthy
d3a179b6f9
Multiple bug fixes
...
- wait for snapTLogFailKeys in a loop, otherwise in some race
condition it can cause a false assert
- in single region, there does not seem to be a guarantee of
tagLocalityListKey for a given DC ID, avoiding that assert for now
- to find the workers that are coordinators, looking up by primary
address is not sufficient in some cases, hence looking by both
primary and secondary address
- test make files to reflect the location of the new test cases
2019-05-28 22:07:46 -07:00
sramamoorthy
bb474dc323
if recovery < fully_recovered then fail the exec
...
Will do more cleanup, pushing it for a test run in CI
2019-05-28 22:07:46 -07:00
sramamoorthy
925499954b
New status cluster_not_fully_recovered
2019-05-28 22:07:46 -07:00
sramamoorthy
591ff96b93
increase retry and use eat instead of parsing
2019-05-28 22:07:46 -07:00
sramamoorthy
936ffc2dde
rebase related changes
2019-05-28 22:07:46 -07:00
sramamoorthy
ec7834e2f7
code re-orgnaization and address comments
2019-05-28 22:07:46 -07:00
sramamoorthy
61e93a9304
Address review comments and minor fixes
2019-05-28 22:07:46 -07:00
sramamoorthy
89b7a052f5
Bug fixes for snapping coordinators
2019-05-28 22:07:46 -07:00
sramamoorthy
17ecba8313
trace cleanup and other indentation changes
2019-05-28 22:07:46 -07:00
sramamoorthy
898bed66c1
Allow only whitelisted binary path for exec op
2019-05-28 22:07:46 -07:00
sramamoorthy
aa79480d69
changes to make fdbfork asynchronous
2019-05-28 22:07:46 -07:00
sramamoorthy
f7ba0635ef
Make Exec op the first op in the batch
2019-05-28 22:07:46 -07:00
sramamoorthy
382b246930
trace change and retain fitness file after restore
2019-05-28 22:07:46 -07:00
sramamoorthy
3d5998e9dd
tlog: when pops are disabled, store them & replay
...
In Tlogs, disable pop is done whlie taking snapshots. Earlier, tlogs
were ignoring the pops if it got pop requests when pops were
disabled. In this change, instead of ignoring the pop - it remembers
the list of pops in-memory and plays them once the popping is
enabled.
2019-05-28 22:07:46 -07:00
sramamoorthy
72dd067173
Trace message changes and fix few FIXMEs
2019-05-28 22:07:46 -07:00
sramamoorthy
69edefe68b
Snapshot based backup and resotre implementation
2019-05-28 22:07:46 -07:00
Jon Fu
03cebd6032
renamed function from storage metrics to data distribution metrics
2019-05-28 09:52:27 -07:00
A.J. Beamon
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
Jon Fu
7b4f2907da
addressed code review changes
2019-05-22 16:58:28 -07:00
Jon Fu
e339ab890b
added timeout and shard limiting behaviour
2019-05-21 14:13:09 -07:00
Jon Fu
0984d272e1
initial implementation of get-dd-metrics
2019-05-21 14:13:09 -07:00
Jingyu Zhou
b8e7fc1b84
Refactor: add std:: qualifier and use emplace_back
2019-05-17 09:38:50 -10:00
mpilman
9eeb48c43d
Allow to turn on object serializer
...
This commit includes functionality to turn on
the object serializer for network communication.
This is done the following way:
- On incoming connections, a process will detect
whether the client supports the object serializer
and will only serialize responses with it, if it does
- On outgoing connections, the command line flag is used
to determine whether the object serializer should be used
to send data.
This way, a cluster can run in mixed mode. To upgrade one
can upgrade one process at a time and set the flag one process
at a time.
This is how this is tested on the simulator:
- The command line flag can take three options: on, off,
and random.
- For off, the object serializer will never we used.
- For on, the object serializer will be always used.
- For random, the simulator will flip a coin for each
process it starts up.
2019-05-13 14:15:22 -07:00
A.J. Beamon
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
Andrew Noyes
781b6ece77
Fix OPEN_FOR_IDE -Wunused-variable warnings
...
CC #1255 , #1173
2019-04-16 15:28:01 -07:00
Andrew Noyes
6207d724f8
Fix all -Wunused-variable warnings
2019-04-15 18:13:00 -07:00
Evan Tschannen
cd5c9d91fa
Merge pull request #1443 from etschannen/master
...
Merge 6.1 into master
2019-04-10 17:43:07 -07:00
Evan Tschannen
21c0ba555c
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-08 18:38:42 -07:00
A.J. Beamon
538b431656
Apply suggestions from code review
2019-04-08 14:55:58 -07:00
A.J. Beamon
a7288e1325
Throw process_behind instead of future_version when all storage nodes on a team are behind. process_behind gets the same backoff behavior as not_committed. Add proxy_memory_limit_exceeded to the retryable predicate.
2019-04-08 14:21:24 -07:00
mpilman
d01cbf3455
Addressed code review comments
2019-04-05 13:12:20 -07:00
mpilman
39ecbedd74
Fixed compilation errors on OS X & gcc8
2019-04-05 13:12:19 -07:00
mpilman
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
Evan Tschannen
72203ba47a
Merge commit '56f3f0b1bc60604f965152d856ae29a591227703'
2019-04-01 18:45:38 -07:00
Evan Tschannen
8714394d42
increase the priority of the client’s version batch timeout, so that we prefer issuing the batch over other possible work
2019-04-01 18:37:40 -07:00
Evan Tschannen
d882c060bf
Merge commit '5dd6396eed0de0dfea6cf9eecc307995eff5cedc'
2019-03-28 18:00:55 -07:00
Evan Tschannen
b6008558d3
renamed BinaryWriter.toStringRef() to .toValue(), because the function now returns a Standalone<StringRef>()
...
eliminated an unnecessary copy from the proxy commit path
eliminated an unnecessary copy from buffered peek cursor
2019-03-28 11:52:50 -07:00
Jingyu Zhou
7c02ee6fdd
Fix compiler warning about unreferenced exception variable
2019-03-26 13:43:47 -07:00
Evan Tschannen
5e03e178de
Merge pull request #1345 from ajbeamon/support-multiple-client-or-worker-issues
...
Add support for a client or worker having multiple issues.
2019-03-24 17:27:50 -07:00
Evan Tschannen
e3400c13ae
fixed a performance regression related to broadcasting a read version to too many transactions simultaneously
2019-03-22 18:37:39 -07:00
Alec Grieser
55a9db1994
spaces to tabs 🤮
2019-03-22 12:58:32 -04:00
A.J. Beamon
4eb5715689
Add support for a client or worker having multiple issues.
2019-03-22 08:29:41 -07:00
Alec Grieser
64e45e6826
retry limit and max delay transaction options are no longer reset after onError
2019-03-21 18:50:02 -04:00
Alec Grieser
22f592ce6e
reset the timeout only if the API version is less than 610 to allow transactions with longer timeouts than the database default
2019-03-21 16:47:12 -04:00
Alec Grieser
148a21a31c
set the timeout actor upon transaction creation if necessary
...
initialize lockaware and maxbackoff in options constructor rather than NativeAPI transaction constructor
2019-03-20 11:13:38 -06:00
Alec Grieser
5f73b7d6ae
Resolves #775 : Support setting Transaction options at the Database level
...
This selects a few useful transaction options that users may want to set for every transaction and adds them to the database options.
2019-03-20 11:13:38 -06:00
Evan Tschannen
f9aad46573
made use_provisional_proxies a transaction option
2019-03-19 18:44:37 -07:00
Evan Tschannen
5b9c45ea0b
clients do not attempt to connect to provisional proxies
2019-03-19 13:37:50 -07:00
A.J. Beamon
19d27b6271
Merge branch 'master' into fix-compiler-warnings
2019-03-15 10:35:36 -07:00
A.J. Beamon
85b3f11e71
Fix various compiler warnings
2019-03-15 10:34:57 -07:00
A.J. Beamon
59b41b03ed
Merge branch 'master' into fix-threadsafedatabase-safety
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2019-03-14 14:50:14 -07:00
A.J. Beamon
e12d242d18
Address review comments.
2019-03-14 13:42:03 -07:00
Meng Xu
5a10bf5dfc
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-14 10:35:12 -07:00
A.J. Beamon
cab284027d
Be more careful about accessing ThreadSafeDatabase's db member. Preallocate the db on the calling thread so that other callers can access the pointer immediately.
2019-03-14 10:26:22 -07:00
Meng Xu
e30e2af1f3
ClientKnobs: Add CHECK_CONNECTED_COORDINATOR_NUM_DELAY
2019-03-13 16:54:56 -07:00
Meng Xu
85c24b0067
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-12 15:20:54 -07:00
Meng Xu
22f5624494
TLS Status: Reduce cluster controller load
...
When the coordinator changes, we use delayedAsyncVar() to reduce
the frequency for cluster controller to send the updated connectedCoordinatorsNumDelayed
to clients.
This help reduce the cluster controllers workload
2019-03-12 15:08:10 -07:00
Meng Xu
46f4b02807
TLS Status: Resolve review comments
...
Use connectedCoordinatorsNumDelayed to reduce the load on cluster controller;
Set connectedCoordinatorsNum to null by default for monitorLeader()
2019-03-11 17:10:08 -07:00
Evan Tschannen
2627bcd35e
Merge branch 'master' into feature-metadata-version
2019-03-10 21:13:28 -07:00
Meng Xu
04880e3d4d
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-06 13:41:16 -08:00
Alex Miller
c6a65389ae
Remove noexcept macro and replace with BOOST_NOEXCEPT.
...
BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a
way that is correctly maintained over time.
2019-03-05 22:06:12 -08:00
Meng Xu
820548223a
Status: connected_coordinators misc minor changes
...
Change the rst document file;
Change the coding style to be consistent with the nearby code;
Ensure we always initilize the connectedCoordinatesNum to 0
even when the variable is not used.
2019-03-05 21:45:18 -08:00
Meng Xu
b7a52e81e2
Status: Count connected coordinators per client
...
A client will always try to connect all coordinators.
This commit let Status track the number of connected coordinators
for each client.
This allows us to do canary in coordinators. For example,
when we switch from non-TLS to TLS, we can switch 1 coordinator
from non-TLS to TLS. This can help check if a client has the ability
to connect through TLS.
We can make the non-TLS to TLS switch for each coordinators
one by one. This avoid the risk of losing connection in the switch.
2019-03-05 21:21:23 -08:00
Evan Tschannen
82d957e0bb
Merge pull request #1178 from vishesh/task/issue-963-IPv6
...
IPv6 Support
2019-03-05 17:14:16 -08:00
Evan Tschannen
f1897f3eb6
Merge branch 'master' into feature-metadata-version
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2019-03-04 21:06:16 -08:00
Evan Tschannen
988add9fb5
cache the metadataVersion for commits, so that doing setVersion with the commit version of a different transaction will
2019-03-04 16:48:34 -08:00
Trevor Clinkenbeard
cbc5df5e93
Removed unnecessary Trace line
2019-03-04 16:31:40 -08:00
Trevor Clinkenbeard
56ae46f89e
Client lazily fetches health metrics from proxies
2019-03-04 14:16:39 -08:00
Vishesh Yadav
e93cd0ff21
Add some checks and comments to IPv6 changes #963
2019-03-04 14:12:45 -08:00
Vishesh Yadav
41d18db7b9
fix: update the encoding of AddressExclusion in SystemData #963
2019-03-04 14:12:45 -08:00
Vishesh Yadav
57832e625d
net: Support IPv6 #963
...
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.
- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.
- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.
- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
Evan Tschannen
075fdef31a
Merge branch 'master' into feature-metadata-version
...
# Conflicts:
# fdbclient/DatabaseContext.h
2019-03-03 22:58:45 -08:00
Trevor Clinkenbeard
1576d50ee9
Removed unnecessary Trace line
2019-03-03 17:09:03 -08:00
Trevor Clinkenbeard
015997f2f9
Simplified updateHealthMetricsActor
2019-03-02 18:01:23 -08:00
Trevor Clinkenbeard
39f612d132
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-03-02 17:07:00 -08:00
Trevor Clinkenbeard
6c45ffc56f
Added exception handling to setNetworkOption
...
Handle non-integer option values for sendDetailedHealthMetrics option by
throwing invalid_option_value
2019-03-02 15:51:51 -08:00
Evan Tschannen
c1de93f467
fix: binary search could get stuck in an infinite loop
...
fix: avoid picking a read version which could be before the last real commit
fix: we must wait on metadataVersionKey in case it is not already cached
fixed review comments
2019-03-02 13:55:41 -08:00
Meng Xu
94385447bc
Status: Get if client configured TLS
...
To understand if all clients have configured TLS,
we check the tlsoption when a client tries to open database.
This is similar to how we track the versions of multi-version clients.
2019-03-01 15:17:01 -08:00
Evan Tschannen
2168b14834
Merge branch 'master' into feature-metadata-version
2019-02-28 17:45:55 -08:00
Evan Tschannen
3da85f3acd
implemented the \xff/metadataVersion key, which can be used by layers to help them cheaply cache metadata and know when their cache is invalid
2019-02-28 17:45:00 -08:00
A.J. Beamon
af69ba035a
Merge pull request #1200 from bnamasivayam/master
...
User provided Client Transaction id merging
2019-02-28 11:16:44 -05:00
Trevor Clinkenbeard
d2bde4e55b
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-27 16:30:25 -08:00
Balachandar Namasivayam
cc0aac588a
Merge branch 'client-transaction-id'
...
* client-transaction-id:
Add user provided transaction id as part of the format in ClientTransactionProfileCorrectness workload.
Addressed review comments.
Apply suggestions from code review
Add support for client provided transaction identifier to be logged as part of trace logging or transaction profiling.
2019-02-27 13:11:15 -08:00
Balachandar Namasivayam
7d7db54515
Addressed review comments.
2019-02-27 12:42:45 -08:00
A.J. Beamon
999ee68609
Improve avoidance of transient issues when logging IncorrectClusterFileContents SevWarnAlways events by making it time based.
2019-02-27 10:08:24 -08:00
Trevor Clinkenbeard
ff9a7cb2f1
Combined proxy health metrics replies into single message type
2019-02-23 10:13:43 -08:00
Trevor Clinkenbeard
edc0c5bf2b
Client resends health metrics request when proxies change
2019-02-22 12:30:36 -08:00
Trevor Clinkenbeard
fa96b8dd33
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-20 16:56:16 -08:00
mpilman
0bb60e5a3b
Use proper fwd decl in NativeAPI
...
Also NativeAPI.h -> NativeAPI.actor.h
2019-02-19 15:16:59 -08:00
Vishesh Yadav
907446d0ce
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-14 11:37:38 -08:00
Andrew Noyes
65136a2ecd
Forward declare actors with ACTOR keyword. #1148
...
There are several more occurrences of this, but they're in .h files that
now need to be .actor.h files. This gets the easy ones out of the way.
2019-02-12 17:56:20 -08:00
Andrew Noyes
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
Balachandar Namasivayam
91d859d791
Add support for client provided transaction identifier to be logged as part of trace logging or transaction profiling.
2019-02-12 13:54:24 -08:00
mpilman
6da5971e79
Guard all versions.h to not break old WIN32 build
2019-02-08 16:06:00 -08:00
A.J. Beamon
685242fbfc
Merge pull request #1092 from atn34/trace-format-network-option
...
Add trace_format network option
2019-02-08 10:52:50 -08:00
mpilman
7e26b4ef0d
Address comments from PR
2019-02-07 15:37:04 -08:00
mpilman
8a94d80deb
fdbservice and fdbrpc now compiling
2019-02-07 15:37:04 -08:00
Trevor Clinkenbeard
5d15aa2794
Client must actually update health metrics after receiving GetHealthMetricsReply
2019-02-02 17:04:53 -08:00
Trevor Clinkenbeard
c22b3c35ad
Added updateHealthMetricsActor actor to each DatabaseContext object, to poll proxies for health metrics
2019-02-01 13:23:10 -08:00
Trevor Clinkenbeard
e549def5a9
Added send_detailed_health_metrics network option, to be enabled when the client wants to receive per-process health metrics.
2019-01-31 18:46:11 -08:00
Evan Tschannen
e9ddd94e27
The failure monitor is given a list of all IP addresses associated with a process
...
The connect packet includes the correct remote address
Did a lot of code cleanup
Simulation test mixed TLS and non-TLS listeners on the same process
2019-01-31 18:20:14 -08:00
Andrew Noyes
e055fdab03
Better diagnostics for unrecognized trace format
2019-01-29 13:08:47 -08:00
Andrew Noyes
768d7678be
Add trace_format network option
2019-01-28 15:14:17 -08:00
Evan Tschannen
1d7fec3074
Merge commit '048bfc5c368063d9e009513078dab88be0cbd5b0' into task/tls-upgrade-2
...
# Conflicts:
# .gitignore
2019-01-24 17:43:06 -08:00
Alec Grieser
04b94e74c7
Merge pull request #942 from ajbeamon/remove-cluster-from-bindings
...
Remove cluster from bindings
2019-01-24 14:39:09 -08:00
A.J. Beamon
d4d5740282
* Add Optional.map and ErrorOr.map.
...
* Rename Optional/ErrorOr cast_to to castTo.
* Make printable(Optional<T>) templated rather than restricted to StringRef types.
* Fixes bug in (unused) ErrorOr.castTo where an ErrorOr that was not set would lose its error.
2019-01-11 09:03:38 -08:00
Vishesh Yadav
51b89ae083
WIP
2019-01-09 07:41:02 -08:00
Vishesh Yadav
e8e01b2406
Remove unused localAddress parameter from newNet2 and Net2 classes
2018-12-13 13:36:52 -08:00
A.J. Beamon
890a0f3e82
Merge branch 'master' into remove-cluster-from-bindings
...
# Conflicts:
# bindings/flow/fdb_flow.actor.cpp
2018-11-26 12:01:03 -08:00
Evan Tschannen
4b5d0b4e2c
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/AsyncFileBlobStore.actor.cpp
# fdbclient/AsyncFileBlobStore.actor.h
# fdbclient/BlobStore.actor.cpp
# fdbclient/BlobStore.h
# fdbclient/HTTP.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/batcher.actor.h
# fdbrpc/fdbrpc.vcxproj
# fdbrpc/sim2.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2018-11-10 13:04:24 -08:00
Evan Tschannen
a654183f63
Merge pull request #791 from ajbeamon/remove-cluster-from-iclientapi
...
Remove cluster from IClientApi (phase 2 of removing DB names)
2018-11-10 10:16:18 -08:00
Evan Tschannen
6a406bae72
Merge pull request #896 from ajbeamon/downgrade-incorrect-cluster-file-event
...
Downgrade the severity of IncorrectClusterFileContents the first time…
2018-11-10 10:06:36 -08:00
A.J. Beamon
187e507e53
Downgrade the severity of IncorrectClusterFileContents the first time it is logged to avoid transient issues that appear like the cluster file hasn't been updated (e.g. the cluster file is shared between multiple processes).
2018-11-05 09:28:08 -08:00
Evan Tschannen
593fa092f3
added a fixme
2018-11-04 19:54:20 -08:00
Evan Tschannen
bf6545a9cf
clients cache storage server interfaces individually, instead of as a team. This is needed because in fearless every shard has storage servers from two separate teams, leading to a lot of possible combinations
...
allAlternatives failed logic was simplified, because we are already doing a global rate limiting, so a per shard limit is unnecessary
reduced unnecessary state variables in waitMetrics requests
2018-11-02 13:15:09 -07:00
Robert Escriva
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
A.J. Beamon
d5357d2439
Update all of the bindings to use the new C API and deprecate or remove unneeded parts of their APIs.
2018-10-02 15:28:46 -07:00
A.J. Beamon
3eb4355a48
Some various cleanup and fixes. Added "Cluster" to TransactionMetrics trace event.
2018-09-25 15:06:19 -07:00
A.J. Beamon
c831051474
This removes the idea of clusters from IClientApi.
2018-09-21 15:58:14 -07:00
Evan Tschannen
90301f497f
Merge branch 'release-6.0'
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/TLSConnection.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# versions.target
2018-09-05 16:06:33 -07:00
Evan Tschannen
d42f0e9c44
Merge pull request #618 from ajbeamon/log-options
...
Log network, cluster, and database options when they are set.
2018-09-05 10:36:40 -07:00
A.J. Beamon
2a97139d5d
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-16 10:24:12 -07:00
A.J. Beamon
bc0832bb55
Deprecate TLS_plugin option. Don't remove deprecated methods from Go bindings, since that breaks compatibility with old API versions.
2018-08-15 10:11:08 -07:00
A.J. Beamon
3c03ddf91e
Remove value logging for options. Add some additional logging for setting an external client library and setting knobs.
2018-08-15 09:42:49 -07:00
Alex Miller
fb31a6999f
Rewrite all files to have #include actorcompiler.h as the last include.
2018-08-14 15:50:26 -07:00
Alex Miller
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
Alex Miller
c38d5247fd
Initialize client TLSOptions before first use.
...
Because initializing them after first use is called SEGFAULT.
This affects both fdbcli and fdbbackup when passed TLS options. I didn't track
down what introduced this, I just fixed it.
2018-08-06 18:08:45 -07:00
Evan Tschannen
538e684f1c
Merge branch 'release-6.0'
...
# Conflicts:
# versions.target
2018-08-03 11:41:46 -07:00
Evan Tschannen
1c29275672
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-01 14:30:57 -07:00
Alex Miller
d7f26cb744
Initialize client TLSOptions before first use.
...
Because initializing them after first use is called SEGFAULT.
This affects both fdbcli and fdbbackup when passed TLS options. I didn't track
down what introduced this, I just fixed it.
2018-07-31 16:09:11 -07:00
Alvin Moore
9ea0f0a5ae
Fixed problem with stack initialization of TLS Options class
2018-07-03 15:02:53 -07:00
Alvin Moore
c3f88dbfe1
Merge branch 'master' of github.com:apple/foundationdb into tls-static
2018-07-01 23:13:57 -07:00
Alvin Moore
ef8de426d3
Changed the TLS_DISABLED macro
...
Disable TLS within Windows until working
2018-06-26 12:08:32 -07:00
Evan Tschannen
c6a2207577
fix: stop client sampling during the consistency check
2018-06-20 16:24:37 -07:00
Alvin Moore
f8ce1de601
Added support for compiling TLS into binaries
2018-06-20 09:21:23 -07:00
A.J. Beamon
99c9958db7
Some more trace event normalization
2018-06-08 13:57:00 -07:00
A.J. Beamon
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
A.J. Beamon
216404de45
Merge branch 'release-5.2' of github.com:apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2018-06-06 15:25:37 -07:00
A.J. Beamon
e4e06321c7
fix: Read-only transactions that get committed would fail if the readOnly option is set. They would also be counted in the transactionsCommitStarted metric.
2018-06-05 12:10:28 -07:00
Balachandar Namasivayam
529d0497f1
Proxy going OOM when applying high volumes of writes to a proxy, particular in a sudden fashion before ratekeeper can control the workload.
...
Address this issue by proactively monitoring the memory used by commit batches and dropping requests if a certain memory limit is exceeded.
2018-06-01 15:21:40 -07:00
A.J. Beamon
d9c702a9e3
Merge release-5.1 into release-5.2
2018-05-30 09:09:55 -07:00
Evan Tschannen
520aaf731d
merge release 5.2 into master
2018-05-10 14:33:08 -07:00
Evan Tschannen
714da951a6
refactored getKeyLocation code to remove duplication
2018-05-10 10:05:41 -07:00
Evan Tschannen
8f984cb2c9
Merge branch 'release-5.2'
...
# Conflicts:
# fdbrpc/TLSConnection.h
2018-05-10 09:13:22 -07:00
Evan Tschannen
e860cc5994
Merge branch 'release-5.2' of github.com:apple/foundationdb into release-5.2
2018-05-09 17:22:19 -07:00
Evan Tschannen
98e07fd92e
fix: clear the client's storage server interface cache on endpoint failure for all request types
2018-05-09 17:11:54 -07:00
Evan Tschannen
d3450ce5b0
Merge pull request #343 from bnamasivayam/tls-plugin
...
Tls plugin
2018-05-09 16:35:53 -07:00
Balachandar Namasivayam
7591931a09
Revert "Make tls_verify_peers as a comma separated string of constraints."
...
This reverts commit 2033847e4b .
2018-05-09 14:40:36 -07:00
Balachandar Namasivayam
2033847e4b
Make tls_verify_peers as a comma separated string of constraints.
2018-05-09 14:37:39 -07:00
Alec Grieser
8de914a81f
use contents() instead of address of in withPrefix and withSuffix ; whitespace fixes
2018-05-09 09:01:22 -07:00
Balachandar Namasivayam
e8b7f4b190
Add password support for tls.
2018-05-08 20:46:31 -07:00
Balachandar Namasivayam
49af5d685b
Restore previous behavior of not specifying peer_verify option means disable checking.
2018-05-08 18:54:44 -07:00
Balachandar Namasivayam
d3b5cfb93c
Support latest TLS plugin.
...
Add support for https in backup.
2018-05-08 16:28:13 -07:00
A.J. Beamon
ca720e1540
Merge pull request #297 from apple/release-5.2
...
Merge 5.2 to Master
2018-05-08 12:04:20 -07:00
Alec Grieser
8987d21df3
use new semantics when writing position of vs key in native api
2018-05-08 08:57:09 -07:00
Alec Grieser
464e2cdbf0
change SetVersionstampedKey and SetVersionstampedValue behavior based on API version to make them consistent
2018-05-08 08:57:09 -07:00
A.J. Beamon
f848c68b73
TraceEvents in FileBackupAgent that include BeginFile should be encoding them with printable because of the possible inclusion of the null byte. This causes the detail to terminate prematurely.
2018-05-01 15:35:48 -07:00
Dennis Schafroth
290122637b
Using ASSERT_ABORT in destructors
2018-04-23 14:05:10 +02:00
Evan Tschannen
19762b847d
Merge branch 'release-5.2'
...
# Conflicts:
# fdbserver/DatabaseConfiguration.cpp
# fdbserver/SimulatedCluster.actor.cpp
2018-04-10 17:02:43 -07:00
Evan Tschannen
d3fb17d30a
Merge pull request #74 from bnamasivayam/client-profiling-tests
...
Client profiling tests - Part 1
2018-03-23 16:52:49 -07:00
Balachandar Namasivayam
89d7cc1093
Minor Bug fixes...
2018-03-15 11:00:47 -07:00
Balachandar Namasivayam
856d2a0a9d
Add correctness tests for Client transaction profiling data format. It also includes format check across upgrades.
2018-03-14 12:39:50 -07:00
A.J. Beamon
f2c804e14f
Reverting changes from merge of master into release-5.2 ( b25810711c). Note that we never intend to release master into release-5.2, but if we did we would need to revert this commit.
2018-03-06 10:15:04 -08:00
A.J. Beamon
fd4c9d5b54
Fix line endings
2018-03-05 10:37:52 -08:00
Evan Tschannen
37a6a81634
Merge commit '7f6fc3e039c911cd84b8540f7f799fc38a1c1822' into feature-remote-logs
...
# Conflicts:
# fdbserver/workloads/RestartRecovery.actor.cpp
2018-02-23 12:33:28 -08:00
Evan Tschannen
719bb5bd0c
Merge pull request #4 from bnamasivayam/getKeyServers-refactor
...
Having 1000 as the limit for Limit for GetKeyServerLocationsRequest s…
2018-02-22 12:39:48 -08:00
Balachandar Namasivayam
2fe2b522d5
Simple refactor of code...
2018-02-22 12:38:14 -08:00
Alec Grieser
e1162e9238
Merge remote-tracking branch 'upstream/release-5.1'
2018-02-22 11:16:12 -08:00
Balachandar Namasivayam
e2030db5a8
Refactor getKeyServers to be more readable.
...
Fix possible memory corruption by returning KeyRange instead of KeyRangeRef in getKeyServers.
Simplify getMasterProxies on DatabaseContext class.
2018-02-21 17:11:50 -08:00
Alec Grieser
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
Evan Tschannen
42405c78a5
Merge commit '4038bd2fd968d88861f2cebd442ce511724816cb' into feature-remote-logs
...
# Conflicts:
# fdbserver/ClusterController.actor.cpp
# fdbserver/Knobs.cpp
2018-02-10 12:08:52 -08:00
Evan Tschannen
fbadcc6eea
changing a storage server’s tag must be the first mutations applied in a version, because privatized mutations applied earlier in the same version will use the old tag
2018-02-09 18:21:29 -08:00
A.J. Beamon
e06f80bb6e
Add client logging for number of logical and physical reads at the NativeAPI layer as well as committed mutation counts and bytes.
2018-02-07 11:56:47 -08:00
A.J. Beamon
0c601d6f85
Purge past version references
2018-01-31 12:05:41 -08:00
A.J. Beamon
41ebc34097
Fix some weird spacing
2018-01-31 11:13:57 -08:00
Stephen Atherton
4dec5423f7
Optimization in backup snapshot dispatching. If the next dispatch version is not ahead of the recently known current version then do not set a scheduled time for new range tasks in order to avoid the overhead and delay of task timeout handling. Adjusted task timeout knobs to avoid large transaction warnings. Removed backtrace() from LargeTransaction trace event. Tweaked suppression in backup trace events.
2018-01-24 17:40:02 -08:00
Stephen Atherton
aebbe1dcfd
Changed core_versionspersecond knob to int64_t type to avoid integer overflow. Cleaned up backup TraceEvent suppression. Added backtrace to LargeTransaction TraceEvent to make it easier to find the source of large commits in applications using NativeAPI directly.
2018-01-24 11:59:37 -08:00
Evan Tschannen
698ef4117e
Merge branch 'master' into feature-remote-logs
2018-01-20 10:34:30 -08:00
A.J. Beamon
16cd0c8f75
Change WaitStorageMetricsPenalty to not be SevWarnAlways unless it happens multiple times in a row (about 2.5 minutes with current knobs).
2018-01-19 16:41:15 -08:00
Evan Tschannen
21482a45e1
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/DBCoreState.h
# fdbserver/LogSystem.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-01-14 13:40:24 -08:00
Evan Tschannen
645dc5ead6
warmRange needs to get a read version occasionally to prevent it from overwhelming the proxy
...
quietDatabase waits for all data distribution to be completely finished so that databases are cached in a cleaner state
2018-01-14 12:50:52 -08:00
Evan Tschannen
721a891d1f
fix: never request more than 100 shards from the proxy at a time to avoid large packets
2018-01-12 10:51:53 -08:00
Evan Tschannen
4e8bc273b3
added a version of getKeyRangeLocations that checks for endpoint failures
...
fix: did not add the cluster controller to id_used in all cases
removed obsolete fixmes
2018-01-07 15:32:43 -08:00
Evan Tschannen
3ec45d38a0
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-06 13:54:45 -08:00
Evan Tschannen
14e956c4fc
fix: getRange did not check for endpoint failures
...
refactored checking endpoint failures
2018-01-06 13:49:47 -08:00
Evan Tschannen
63751fb0e2
fix: remote logs are not in the log system until the recovery is complete so they cannot be used to determine if this is the correct log system to recover from
2018-01-05 14:15:25 -08:00
Evan Tschannen
2a869a4178
fixed merge errors
2018-01-05 12:17:17 -08:00
Evan Tschannen
5ac4f73978
Merge branch 'release-5.1' into feature-remote-logs
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/Locality.h
# fdbrpc/simulator.h
# fdbserver/ApplyMetadataMutation.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
# fdbserver/WorkerInterface.h
# fdbserver/masterserver.actor.cpp
# flow/Net2.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-05 11:33:42 -08:00
A.J. Beamon
5015119115
Generalize the message that gets displayed in status if a cluster file's contents are incorrect.
2018-01-05 10:29:47 -08:00
Evan Tschannen
982f0dcb1e
Merge pull request #222 from cie/alexmiller/drtimefix2
...
Fix yet another VersionStamp DR issue.
2017-12-20 15:09:23 -08:00
Alex Miller
b5a6bc0ab7
Fix VersionStamp problems by instead adding a COMMIT_ON_FIRST_PROXY transaction option.
...
Simulation identified the fact that we can violate the
VersionStamps-are-always-increasing promise via the following series of events:
1. On proxy 0, dumpData adds commit requests to proxy 0's commit promise stream
2. To any proxy, a client submits the first transaction of abortBackup, which stops further dumpData calls on proxy 0.
3. To any proxy that is not proxy 0, submit a transaction that checks if it needs to upgrade the destination version.
4. The transaction from (3) is committed
5. Transactions from (1) are committed
This is possible because the dumpData transactions have no read conflict
ranges, and thus it's impossible to make them abort due to "conflicting"
transactions. There's also no promise that if client C sends a commit to proxy
A, and later a client D sends a commit to proxy B, that B must log its commit
after A. (We only promise that if C is told it was committed before D is told
it was committed, then A committed before B.)
There was a failed attempt to fix this problem. We tried to add read conflict
ranges to dumpData transactions so that they could be aborted by "conflicting"
transactions. However, this failed because this now means that dumpData
transactions require conflict resolution, and the stale read version that they
use can cause them to be aborted with a transaction_too_old error.
(Transactions that don't have read conflict ranges will never return
transaction_too_old, because with no reads, the read snapshot version is
effectively meaningless.) This was never previously possible, so the existing
code doesn't retry commits, and to make things more complicated, the dumpData
commits must be applied in order. This would require either adding
dependencies to transactions (if A is going to commit then B must also be/have
committed), which would be complicated, or submitting transactions with a fixed
read version, and replaying the failed commits with a higher read version once
we get a transaction_too_old error, which would unacceptably slow down the
maximum throughput of dumpData.
Thus, we've instead elected to add a special transaction option that bypasses
proxy load balancing for commits, and always commits against proxy 0. We can
know for certain that after the transaction from (2) is committed, all of the
dumpData transactions that will be committed have been added to the commit
promise stream on proxy 0. Thus, if we enqueue another transaction against
proxy 0, we can know that it will be placed into the promise stream after all
of the dumpData transactions, thus providing the semantics that we require: no
dumpData transaction can commit after the destination version upgrade
transaction.
2017-12-20 15:04:04 -08:00
Evan Tschannen
1dc9eceb6d
optimize GetKeyLocationRequests on the proxy so they only require a single map lookup, instead of doing 3 + (3* [number of ranges]) lookups
2017-12-15 20:13:44 -08:00
Evan Tschannen
73a0a07eac
clients ask for key location information directly from the proxy, instead of reading it from the database
2017-12-09 16:10:22 -08:00
Alex Miller
e900333dbf
Fix a subtle valgrind error.
...
If there was an error in waiting for the read version, we would attempt to
serialize and eventually commit a CommitTransactionRef that had an
uninitialized read_snapshot.
2017-11-15 19:21:20 -08:00
A.J. Beamon
cd085764f1
Do not automatically change a cluster file that does not match what you expect.
2017-11-10 14:12:45 -08:00
Evan Tschannen
33eef4c76b
Merge branch 'master' into getrange-perf-improvements
2017-11-04 14:25:45 -07:00
Balachandar Namasivayam
33cc644ba8
Copy CommitTransactionRequest during tryCommit function call as it is anyway copied inside the function.
2017-11-02 17:00:44 -07:00
A.J. Beamon
3cad2676cc
Collapse some of the getRange actors into a single actor. Avoid unnecessary comparisons.
2017-11-02 13:39:06 -07:00
A.J. Beamon
07c80484eb
Merge branch 'master' into getrange-perf-improvements
2017-11-02 08:42:26 -07:00
Balachandar Namasivayam
3efaaec479
onMasterProxiesChanged was being triggered when any member of ClientDBInfo changed. Change the behavior to be triggered only when proxies field in ClientDBInfo is changed.
2017-11-01 18:29:56 -07:00
Balachandar Namasivayam
1d95e1bfd6
Log actor cancellation error to Client Transaction sampling records.
2017-11-01 11:51:31 -07:00
A.J. Beamon
7cf17df821
Merge branch 'master' into log-group-for-unsupported-clients
...
# Conflicts:
# flow/Net2.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2017-11-01 11:31:02 -07:00
A.J. Beamon
e52f46064a
Don't use the results arena for conflict ranges. Rearrange the code to avoid the copy unless necessary.
2017-10-26 09:35:34 -07:00