ec7834e2f7
code re-orgnaization and address comments
2019-05-28 22:07:46 -07:00
61e93a9304
Address review comments and minor fixes
2019-05-28 22:07:46 -07:00
89b7a052f5
Bug fixes for snapping coordinators
2019-05-28 22:07:46 -07:00
17ecba8313
trace cleanup and other indentation changes
2019-05-28 22:07:46 -07:00
898bed66c1
Allow only whitelisted binary path for exec op
2019-05-28 22:07:46 -07:00
aa79480d69
changes to make fdbfork asynchronous
2019-05-28 22:07:46 -07:00
f7ba0635ef
Make Exec op the first op in the batch
2019-05-28 22:07:46 -07:00
382b246930
trace change and retain fitness file after restore
2019-05-28 22:07:46 -07:00
3d5998e9dd
tlog: when pops are disabled, store them & replay
...
In Tlogs, disable pop is done whlie taking snapshots. Earlier, tlogs
were ignoring the pops if it got pop requests when pops were
disabled. In this change, instead of ignoring the pop - it remembers
the list of pops in-memory and plays them once the popping is
enabled.
2019-05-28 22:07:46 -07:00
72dd067173
Trace message changes and fix few FIXMEs
2019-05-28 22:07:46 -07:00
69edefe68b
Snapshot based backup and resotre implementation
2019-05-28 22:07:46 -07:00
03cebd6032
renamed function from storage metrics to data distribution metrics
2019-05-28 09:52:27 -07:00
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
7b4f2907da
addressed code review changes
2019-05-22 16:58:28 -07:00
e339ab890b
added timeout and shard limiting behaviour
2019-05-21 14:13:09 -07:00
0984d272e1
initial implementation of get-dd-metrics
2019-05-21 14:13:09 -07:00
b8e7fc1b84
Refactor: add std:: qualifier and use emplace_back
2019-05-17 09:38:50 -10:00
9eeb48c43d
Allow to turn on object serializer
...
This commit includes functionality to turn on
the object serializer for network communication.
This is done the following way:
- On incoming connections, a process will detect
whether the client supports the object serializer
and will only serialize responses with it, if it does
- On outgoing connections, the command line flag is used
to determine whether the object serializer should be used
to send data.
This way, a cluster can run in mixed mode. To upgrade one
can upgrade one process at a time and set the flag one process
at a time.
This is how this is tested on the simulator:
- The command line flag can take three options: on, off,
and random.
- For off, the object serializer will never we used.
- For on, the object serializer will be always used.
- For random, the simulator will flip a coin for each
process it starts up.
2019-05-13 14:15:22 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
781b6ece77
Fix OPEN_FOR_IDE -Wunused-variable warnings
...
CC #1255 , #1173
2019-04-16 15:28:01 -07:00
6207d724f8
Fix all -Wunused-variable warnings
2019-04-15 18:13:00 -07:00
cd5c9d91fa
Merge pull request #1443 from etschannen/master
...
Merge 6.1 into master
2019-04-10 17:43:07 -07:00
21c0ba555c
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-08 18:38:42 -07:00
538b431656
Apply suggestions from code review
2019-04-08 14:55:58 -07:00
a7288e1325
Throw process_behind instead of future_version when all storage nodes on a team are behind. process_behind gets the same backoff behavior as not_committed. Add proxy_memory_limit_exceeded to the retryable predicate.
2019-04-08 14:21:24 -07:00
d01cbf3455
Addressed code review comments
2019-04-05 13:12:20 -07:00
39ecbedd74
Fixed compilation errors on OS X & gcc8
2019-04-05 13:12:19 -07:00
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
72203ba47a
Merge commit '56f3f0b1bc60604f965152d856ae29a591227703'
2019-04-01 18:45:38 -07:00
8714394d42
increase the priority of the client’s version batch timeout, so that we prefer issuing the batch over other possible work
2019-04-01 18:37:40 -07:00
d882c060bf
Merge commit '5dd6396eed0de0dfea6cf9eecc307995eff5cedc'
2019-03-28 18:00:55 -07:00
b6008558d3
renamed BinaryWriter.toStringRef() to .toValue(), because the function now returns a Standalone<StringRef>()
...
eliminated an unnecessary copy from the proxy commit path
eliminated an unnecessary copy from buffered peek cursor
2019-03-28 11:52:50 -07:00
7c02ee6fdd
Fix compiler warning about unreferenced exception variable
2019-03-26 13:43:47 -07:00
5e03e178de
Merge pull request #1345 from ajbeamon/support-multiple-client-or-worker-issues
...
Add support for a client or worker having multiple issues.
2019-03-24 17:27:50 -07:00
e3400c13ae
fixed a performance regression related to broadcasting a read version to too many transactions simultaneously
2019-03-22 18:37:39 -07:00
55a9db1994
spaces to tabs 🤮
2019-03-22 12:58:32 -04:00
4eb5715689
Add support for a client or worker having multiple issues.
2019-03-22 08:29:41 -07:00
64e45e6826
retry limit and max delay transaction options are no longer reset after onError
2019-03-21 18:50:02 -04:00
22f592ce6e
reset the timeout only if the API version is less than 610 to allow transactions with longer timeouts than the database default
2019-03-21 16:47:12 -04:00
148a21a31c
set the timeout actor upon transaction creation if necessary
...
initialize lockaware and maxbackoff in options constructor rather than NativeAPI transaction constructor
2019-03-20 11:13:38 -06:00
5f73b7d6ae
Resolves #775 : Support setting Transaction options at the Database level
...
This selects a few useful transaction options that users may want to set for every transaction and adds them to the database options.
2019-03-20 11:13:38 -06:00
f9aad46573
made use_provisional_proxies a transaction option
2019-03-19 18:44:37 -07:00
5b9c45ea0b
clients do not attempt to connect to provisional proxies
2019-03-19 13:37:50 -07:00
19d27b6271
Merge branch 'master' into fix-compiler-warnings
2019-03-15 10:35:36 -07:00
85b3f11e71
Fix various compiler warnings
2019-03-15 10:34:57 -07:00
59b41b03ed
Merge branch 'master' into fix-threadsafedatabase-safety
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2019-03-14 14:50:14 -07:00
e12d242d18
Address review comments.
2019-03-14 13:42:03 -07:00
5a10bf5dfc
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-14 10:35:12 -07:00
cab284027d
Be more careful about accessing ThreadSafeDatabase's db member. Preallocate the db on the calling thread so that other callers can access the pointer immediately.
2019-03-14 10:26:22 -07:00
e30e2af1f3
ClientKnobs: Add CHECK_CONNECTED_COORDINATOR_NUM_DELAY
2019-03-13 16:54:56 -07:00
85c24b0067
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-12 15:20:54 -07:00
22f5624494
TLS Status: Reduce cluster controller load
...
When the coordinator changes, we use delayedAsyncVar() to reduce
the frequency for cluster controller to send the updated connectedCoordinatorsNumDelayed
to clients.
This help reduce the cluster controllers workload
2019-03-12 15:08:10 -07:00
46f4b02807
TLS Status: Resolve review comments
...
Use connectedCoordinatorsNumDelayed to reduce the load on cluster controller;
Set connectedCoordinatorsNum to null by default for monitorLeader()
2019-03-11 17:10:08 -07:00
2627bcd35e
Merge branch 'master' into feature-metadata-version
2019-03-10 21:13:28 -07:00
04880e3d4d
Merge branch 'master' into mengxu/tls-switch-status-PR
2019-03-06 13:41:16 -08:00
c6a65389ae
Remove noexcept macro and replace with BOOST_NOEXCEPT.
...
BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a
way that is correctly maintained over time.
2019-03-05 22:06:12 -08:00
820548223a
Status: connected_coordinators misc minor changes
...
Change the rst document file;
Change the coding style to be consistent with the nearby code;
Ensure we always initilize the connectedCoordinatesNum to 0
even when the variable is not used.
2019-03-05 21:45:18 -08:00
b7a52e81e2
Status: Count connected coordinators per client
...
A client will always try to connect all coordinators.
This commit let Status track the number of connected coordinators
for each client.
This allows us to do canary in coordinators. For example,
when we switch from non-TLS to TLS, we can switch 1 coordinator
from non-TLS to TLS. This can help check if a client has the ability
to connect through TLS.
We can make the non-TLS to TLS switch for each coordinators
one by one. This avoid the risk of losing connection in the switch.
2019-03-05 21:21:23 -08:00
82d957e0bb
Merge pull request #1178 from vishesh/task/issue-963-IPv6
...
IPv6 Support
2019-03-05 17:14:16 -08:00
f1897f3eb6
Merge branch 'master' into feature-metadata-version
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2019-03-04 21:06:16 -08:00
988add9fb5
cache the metadataVersion for commits, so that doing setVersion with the commit version of a different transaction will
2019-03-04 16:48:34 -08:00
cbc5df5e93
Removed unnecessary Trace line
2019-03-04 16:31:40 -08:00
56ae46f89e
Client lazily fetches health metrics from proxies
2019-03-04 14:16:39 -08:00
e93cd0ff21
Add some checks and comments to IPv6 changes #963
2019-03-04 14:12:45 -08:00
41d18db7b9
fix: update the encoding of AddressExclusion in SystemData #963
2019-03-04 14:12:45 -08:00
57832e625d
net: Support IPv6 #963
...
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.
- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.
- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.
- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
075fdef31a
Merge branch 'master' into feature-metadata-version
...
# Conflicts:
# fdbclient/DatabaseContext.h
2019-03-03 22:58:45 -08:00
1576d50ee9
Removed unnecessary Trace line
2019-03-03 17:09:03 -08:00
015997f2f9
Simplified updateHealthMetricsActor
2019-03-02 18:01:23 -08:00
39f612d132
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-03-02 17:07:00 -08:00
6c45ffc56f
Added exception handling to setNetworkOption
...
Handle non-integer option values for sendDetailedHealthMetrics option by
throwing invalid_option_value
2019-03-02 15:51:51 -08:00
c1de93f467
fix: binary search could get stuck in an infinite loop
...
fix: avoid picking a read version which could be before the last real commit
fix: we must wait on metadataVersionKey in case it is not already cached
fixed review comments
2019-03-02 13:55:41 -08:00
94385447bc
Status: Get if client configured TLS
...
To understand if all clients have configured TLS,
we check the tlsoption when a client tries to open database.
This is similar to how we track the versions of multi-version clients.
2019-03-01 15:17:01 -08:00
2168b14834
Merge branch 'master' into feature-metadata-version
2019-02-28 17:45:55 -08:00
3da85f3acd
implemented the \xff/metadataVersion key, which can be used by layers to help them cheaply cache metadata and know when their cache is invalid
2019-02-28 17:45:00 -08:00
af69ba035a
Merge pull request #1200 from bnamasivayam/master
...
User provided Client Transaction id merging
2019-02-28 11:16:44 -05:00
d2bde4e55b
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-27 16:30:25 -08:00
cc0aac588a
Merge branch 'client-transaction-id'
...
* client-transaction-id:
Add user provided transaction id as part of the format in ClientTransactionProfileCorrectness workload.
Addressed review comments.
Apply suggestions from code review
Add support for client provided transaction identifier to be logged as part of trace logging or transaction profiling.
2019-02-27 13:11:15 -08:00
7d7db54515
Addressed review comments.
2019-02-27 12:42:45 -08:00
999ee68609
Improve avoidance of transient issues when logging IncorrectClusterFileContents SevWarnAlways events by making it time based.
2019-02-27 10:08:24 -08:00
ff9a7cb2f1
Combined proxy health metrics replies into single message type
2019-02-23 10:13:43 -08:00
edc0c5bf2b
Client resends health metrics request when proxies change
2019-02-22 12:30:36 -08:00
fa96b8dd33
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-20 16:56:16 -08:00
0bb60e5a3b
Use proper fwd decl in NativeAPI
...
Also NativeAPI.h -> NativeAPI.actor.h
2019-02-19 15:16:59 -08:00
907446d0ce
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-14 11:37:38 -08:00
65136a2ecd
Forward declare actors with ACTOR keyword. #1148
...
There are several more occurrences of this, but they're in .h files that
now need to be .actor.h files. This gets the easy ones out of the way.
2019-02-12 17:56:20 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
91d859d791
Add support for client provided transaction identifier to be logged as part of trace logging or transaction profiling.
2019-02-12 13:54:24 -08:00
6da5971e79
Guard all versions.h to not break old WIN32 build
2019-02-08 16:06:00 -08:00
685242fbfc
Merge pull request #1092 from atn34/trace-format-network-option
...
Add trace_format network option
2019-02-08 10:52:50 -08:00
7e26b4ef0d
Address comments from PR
2019-02-07 15:37:04 -08:00
8a94d80deb
fdbservice and fdbrpc now compiling
2019-02-07 15:37:04 -08:00
5d15aa2794
Client must actually update health metrics after receiving GetHealthMetricsReply
2019-02-02 17:04:53 -08:00
c22b3c35ad
Added updateHealthMetricsActor actor to each DatabaseContext object, to poll proxies for health metrics
2019-02-01 13:23:10 -08:00
e549def5a9
Added send_detailed_health_metrics network option, to be enabled when the client wants to receive per-process health metrics.
2019-01-31 18:46:11 -08:00
e9ddd94e27
The failure monitor is given a list of all IP addresses associated with a process
...
The connect packet includes the correct remote address
Did a lot of code cleanup
Simulation test mixed TLS and non-TLS listeners on the same process
2019-01-31 18:20:14 -08:00
e055fdab03
Better diagnostics for unrecognized trace format
2019-01-29 13:08:47 -08:00
768d7678be
Add trace_format network option
2019-01-28 15:14:17 -08:00
1d7fec3074
Merge commit '048bfc5c368063d9e009513078dab88be0cbd5b0' into task/tls-upgrade-2
...
# Conflicts:
# .gitignore
2019-01-24 17:43:06 -08:00
04b94e74c7
Merge pull request #942 from ajbeamon/remove-cluster-from-bindings
...
Remove cluster from bindings
2019-01-24 14:39:09 -08:00
d4d5740282
* Add Optional.map and ErrorOr.map.
...
* Rename Optional/ErrorOr cast_to to castTo.
* Make printable(Optional<T>) templated rather than restricted to StringRef types.
* Fixes bug in (unused) ErrorOr.castTo where an ErrorOr that was not set would lose its error.
2019-01-11 09:03:38 -08:00
51b89ae083
WIP
2019-01-09 07:41:02 -08:00
e8e01b2406
Remove unused localAddress parameter from newNet2 and Net2 classes
2018-12-13 13:36:52 -08:00
890a0f3e82
Merge branch 'master' into remove-cluster-from-bindings
...
# Conflicts:
# bindings/flow/fdb_flow.actor.cpp
2018-11-26 12:01:03 -08:00
4b5d0b4e2c
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/AsyncFileBlobStore.actor.cpp
# fdbclient/AsyncFileBlobStore.actor.h
# fdbclient/BlobStore.actor.cpp
# fdbclient/BlobStore.h
# fdbclient/HTTP.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/batcher.actor.h
# fdbrpc/fdbrpc.vcxproj
# fdbrpc/sim2.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2018-11-10 13:04:24 -08:00
a654183f63
Merge pull request #791 from ajbeamon/remove-cluster-from-iclientapi
...
Remove cluster from IClientApi (phase 2 of removing DB names)
2018-11-10 10:16:18 -08:00
6a406bae72
Merge pull request #896 from ajbeamon/downgrade-incorrect-cluster-file-event
...
Downgrade the severity of IncorrectClusterFileContents the first time…
2018-11-10 10:06:36 -08:00
187e507e53
Downgrade the severity of IncorrectClusterFileContents the first time it is logged to avoid transient issues that appear like the cluster file hasn't been updated (e.g. the cluster file is shared between multiple processes).
2018-11-05 09:28:08 -08:00
593fa092f3
added a fixme
2018-11-04 19:54:20 -08:00
bf6545a9cf
clients cache storage server interfaces individually, instead of as a team. This is needed because in fearless every shard has storage servers from two separate teams, leading to a lot of possible combinations
...
allAlternatives failed logic was simplified, because we are already doing a global rate limiting, so a per shard limit is unnecessary
reduced unnecessary state variables in waitMetrics requests
2018-11-02 13:15:09 -07:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
d5357d2439
Update all of the bindings to use the new C API and deprecate or remove unneeded parts of their APIs.
2018-10-02 15:28:46 -07:00
3eb4355a48
Some various cleanup and fixes. Added "Cluster" to TransactionMetrics trace event.
2018-09-25 15:06:19 -07:00
c831051474
This removes the idea of clusters from IClientApi.
2018-09-21 15:58:14 -07:00
90301f497f
Merge branch 'release-6.0'
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/TLSConnection.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# versions.target
2018-09-05 16:06:33 -07:00
d42f0e9c44
Merge pull request #618 from ajbeamon/log-options
...
Log network, cluster, and database options when they are set.
2018-09-05 10:36:40 -07:00
2a97139d5d
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-16 10:24:12 -07:00
bc0832bb55
Deprecate TLS_plugin option. Don't remove deprecated methods from Go bindings, since that breaks compatibility with old API versions.
2018-08-15 10:11:08 -07:00
3c03ddf91e
Remove value logging for options. Add some additional logging for setting an external client library and setting knobs.
2018-08-15 09:42:49 -07:00
fb31a6999f
Rewrite all files to have #include actorcompiler.h as the last include.
2018-08-14 15:50:26 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
c38d5247fd
Initialize client TLSOptions before first use.
...
Because initializing them after first use is called SEGFAULT.
This affects both fdbcli and fdbbackup when passed TLS options. I didn't track
down what introduced this, I just fixed it.
2018-08-06 18:08:45 -07:00
538e684f1c
Merge branch 'release-6.0'
...
# Conflicts:
# versions.target
2018-08-03 11:41:46 -07:00
1c29275672
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-01 14:30:57 -07:00
d7f26cb744
Initialize client TLSOptions before first use.
...
Because initializing them after first use is called SEGFAULT.
This affects both fdbcli and fdbbackup when passed TLS options. I didn't track
down what introduced this, I just fixed it.
2018-07-31 16:09:11 -07:00
9ea0f0a5ae
Fixed problem with stack initialization of TLS Options class
2018-07-03 15:02:53 -07:00
c3f88dbfe1
Merge branch 'master' of github.com:apple/foundationdb into tls-static
2018-07-01 23:13:57 -07:00
ef8de426d3
Changed the TLS_DISABLED macro
...
Disable TLS within Windows until working
2018-06-26 12:08:32 -07:00
c6a2207577
fix: stop client sampling during the consistency check
2018-06-20 16:24:37 -07:00
f8ce1de601
Added support for compiling TLS into binaries
2018-06-20 09:21:23 -07:00
99c9958db7
Some more trace event normalization
2018-06-08 13:57:00 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
216404de45
Merge branch 'release-5.2' of github.com:apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2018-06-06 15:25:37 -07:00
e4e06321c7
fix: Read-only transactions that get committed would fail if the readOnly option is set. They would also be counted in the transactionsCommitStarted metric.
2018-06-05 12:10:28 -07:00
529d0497f1
Proxy going OOM when applying high volumes of writes to a proxy, particular in a sudden fashion before ratekeeper can control the workload.
...
Address this issue by proactively monitoring the memory used by commit batches and dropping requests if a certain memory limit is exceeded.
2018-06-01 15:21:40 -07:00
d9c702a9e3
Merge release-5.1 into release-5.2
2018-05-30 09:09:55 -07:00
520aaf731d
merge release 5.2 into master
2018-05-10 14:33:08 -07:00
714da951a6
refactored getKeyLocation code to remove duplication
2018-05-10 10:05:41 -07:00
8f984cb2c9
Merge branch 'release-5.2'
...
# Conflicts:
# fdbrpc/TLSConnection.h
2018-05-10 09:13:22 -07:00
e860cc5994
Merge branch 'release-5.2' of github.com:apple/foundationdb into release-5.2
2018-05-09 17:22:19 -07:00
98e07fd92e
fix: clear the client's storage server interface cache on endpoint failure for all request types
2018-05-09 17:11:54 -07:00
d3450ce5b0
Merge pull request #343 from bnamasivayam/tls-plugin
...
Tls plugin
2018-05-09 16:35:53 -07:00
7591931a09
Revert "Make tls_verify_peers as a comma separated string of constraints."
...
This reverts commit 2033847e4b
2018-05-09 14:40:36 -07:00
2033847e4b
Make tls_verify_peers as a comma separated string of constraints.
2018-05-09 14:37:39 -07:00
8de914a81f
use contents() instead of address of in withPrefix and withSuffix ; whitespace fixes
2018-05-09 09:01:22 -07:00
e8b7f4b190
Add password support for tls.
2018-05-08 20:46:31 -07:00
49af5d685b
Restore previous behavior of not specifying peer_verify option means disable checking.
2018-05-08 18:54:44 -07:00
d3b5cfb93c
Support latest TLS plugin.
...
Add support for https in backup.
2018-05-08 16:28:13 -07:00
ca720e1540
Merge pull request #297 from apple/release-5.2
...
Merge 5.2 to Master
2018-05-08 12:04:20 -07:00
8987d21df3
use new semantics when writing position of vs key in native api
2018-05-08 08:57:09 -07:00
464e2cdbf0
change SetVersionstampedKey and SetVersionstampedValue behavior based on API version to make them consistent
2018-05-08 08:57:09 -07:00
f848c68b73
TraceEvents in FileBackupAgent that include BeginFile should be encoding them with printable because of the possible inclusion of the null byte. This causes the detail to terminate prematurely.
2018-05-01 15:35:48 -07:00
290122637b
Using ASSERT_ABORT in destructors
2018-04-23 14:05:10 +02:00
19762b847d
Merge branch 'release-5.2'
...
# Conflicts:
# fdbserver/DatabaseConfiguration.cpp
# fdbserver/SimulatedCluster.actor.cpp
2018-04-10 17:02:43 -07:00
d3fb17d30a
Merge pull request #74 from bnamasivayam/client-profiling-tests
...
Client profiling tests - Part 1
2018-03-23 16:52:49 -07:00
89d7cc1093
Minor Bug fixes...
2018-03-15 11:00:47 -07:00
856d2a0a9d
Add correctness tests for Client transaction profiling data format. It also includes format check across upgrades.
2018-03-14 12:39:50 -07:00
f2c804e14f
Reverting changes from merge of master into release-5.2 ( b25810711c). Note that we never intend to release master into release-5.2, but if we did we would need to revert this commit.
2018-03-06 10:15:04 -08:00
fd4c9d5b54
Fix line endings
2018-03-05 10:37:52 -08:00
37a6a81634
Merge commit '7f6fc3e039c911cd84b8540f7f799fc38a1c1822' into feature-remote-logs
...
# Conflicts:
# fdbserver/workloads/RestartRecovery.actor.cpp
2018-02-23 12:33:28 -08:00
719bb5bd0c
Merge pull request #4 from bnamasivayam/getKeyServers-refactor
...
Having 1000 as the limit for Limit for GetKeyServerLocationsRequest s…
2018-02-22 12:39:48 -08:00
2fe2b522d5
Simple refactor of code...
2018-02-22 12:38:14 -08:00
e1162e9238
Merge remote-tracking branch 'upstream/release-5.1'
2018-02-22 11:16:12 -08:00
e2030db5a8
Refactor getKeyServers to be more readable.
...
Fix possible memory corruption by returning KeyRange instead of KeyRangeRef in getKeyServers.
Simplify getMasterProxies on DatabaseContext class.
2018-02-21 17:11:50 -08:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
42405c78a5
Merge commit '4038bd2fd968d88861f2cebd442ce511724816cb' into feature-remote-logs
...
# Conflicts:
# fdbserver/ClusterController.actor.cpp
# fdbserver/Knobs.cpp
2018-02-10 12:08:52 -08:00
fbadcc6eea
changing a storage server’s tag must be the first mutations applied in a version, because privatized mutations applied earlier in the same version will use the old tag
2018-02-09 18:21:29 -08:00
e06f80bb6e
Add client logging for number of logical and physical reads at the NativeAPI layer as well as committed mutation counts and bytes.
2018-02-07 11:56:47 -08:00
0c601d6f85
Purge past version references
2018-01-31 12:05:41 -08:00
41ebc34097
Fix some weird spacing
2018-01-31 11:13:57 -08:00
4dec5423f7
Optimization in backup snapshot dispatching. If the next dispatch version is not ahead of the recently known current version then do not set a scheduled time for new range tasks in order to avoid the overhead and delay of task timeout handling. Adjusted task timeout knobs to avoid large transaction warnings. Removed backtrace() from LargeTransaction trace event. Tweaked suppression in backup trace events.
2018-01-24 17:40:02 -08:00
aebbe1dcfd
Changed core_versionspersecond knob to int64_t type to avoid integer overflow. Cleaned up backup TraceEvent suppression. Added backtrace to LargeTransaction TraceEvent to make it easier to find the source of large commits in applications using NativeAPI directly.
2018-01-24 11:59:37 -08:00
698ef4117e
Merge branch 'master' into feature-remote-logs
2018-01-20 10:34:30 -08:00
16cd0c8f75
Change WaitStorageMetricsPenalty to not be SevWarnAlways unless it happens multiple times in a row (about 2.5 minutes with current knobs).
2018-01-19 16:41:15 -08:00
21482a45e1
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/DBCoreState.h
# fdbserver/LogSystem.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-01-14 13:40:24 -08:00
645dc5ead6
warmRange needs to get a read version occasionally to prevent it from overwhelming the proxy
...
quietDatabase waits for all data distribution to be completely finished so that databases are cached in a cleaner state
2018-01-14 12:50:52 -08:00
721a891d1f
fix: never request more than 100 shards from the proxy at a time to avoid large packets
2018-01-12 10:51:53 -08:00
4e8bc273b3
added a version of getKeyRangeLocations that checks for endpoint failures
...
fix: did not add the cluster controller to id_used in all cases
removed obsolete fixmes
2018-01-07 15:32:43 -08:00
3ec45d38a0
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-06 13:54:45 -08:00
14e956c4fc
fix: getRange did not check for endpoint failures
...
refactored checking endpoint failures
2018-01-06 13:49:47 -08:00
63751fb0e2
fix: remote logs are not in the log system until the recovery is complete so they cannot be used to determine if this is the correct log system to recover from
2018-01-05 14:15:25 -08:00
2a869a4178
fixed merge errors
2018-01-05 12:17:17 -08:00
5ac4f73978
Merge branch 'release-5.1' into feature-remote-logs
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/Locality.h
# fdbrpc/simulator.h
# fdbserver/ApplyMetadataMutation.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
# fdbserver/WorkerInterface.h
# fdbserver/masterserver.actor.cpp
# flow/Net2.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-05 11:33:42 -08:00
5015119115
Generalize the message that gets displayed in status if a cluster file's contents are incorrect.
2018-01-05 10:29:47 -08:00
982f0dcb1e
Merge pull request #222 from cie/alexmiller/drtimefix2
...
Fix yet another VersionStamp DR issue.
2017-12-20 15:09:23 -08:00
b5a6bc0ab7
Fix VersionStamp problems by instead adding a COMMIT_ON_FIRST_PROXY transaction option.
...
Simulation identified the fact that we can violate the
VersionStamps-are-always-increasing promise via the following series of events:
1. On proxy 0, dumpData adds commit requests to proxy 0's commit promise stream
2. To any proxy, a client submits the first transaction of abortBackup, which stops further dumpData calls on proxy 0.
3. To any proxy that is not proxy 0, submit a transaction that checks if it needs to upgrade the destination version.
4. The transaction from (3) is committed
5. Transactions from (1) are committed
This is possible because the dumpData transactions have no read conflict
ranges, and thus it's impossible to make them abort due to "conflicting"
transactions. There's also no promise that if client C sends a commit to proxy
A, and later a client D sends a commit to proxy B, that B must log its commit
after A. (We only promise that if C is told it was committed before D is told
it was committed, then A committed before B.)
There was a failed attempt to fix this problem. We tried to add read conflict
ranges to dumpData transactions so that they could be aborted by "conflicting"
transactions. However, this failed because this now means that dumpData
transactions require conflict resolution, and the stale read version that they
use can cause them to be aborted with a transaction_too_old error.
(Transactions that don't have read conflict ranges will never return
transaction_too_old, because with no reads, the read snapshot version is
effectively meaningless.) This was never previously possible, so the existing
code doesn't retry commits, and to make things more complicated, the dumpData
commits must be applied in order. This would require either adding
dependencies to transactions (if A is going to commit then B must also be/have
committed), which would be complicated, or submitting transactions with a fixed
read version, and replaying the failed commits with a higher read version once
we get a transaction_too_old error, which would unacceptably slow down the
maximum throughput of dumpData.
Thus, we've instead elected to add a special transaction option that bypasses
proxy load balancing for commits, and always commits against proxy 0. We can
know for certain that after the transaction from (2) is committed, all of the
dumpData transactions that will be committed have been added to the commit
promise stream on proxy 0. Thus, if we enqueue another transaction against
proxy 0, we can know that it will be placed into the promise stream after all
of the dumpData transactions, thus providing the semantics that we require: no
dumpData transaction can commit after the destination version upgrade
transaction.
2017-12-20 15:04:04 -08:00
1dc9eceb6d
optimize GetKeyLocationRequests on the proxy so they only require a single map lookup, instead of doing 3 + (3* [number of ranges]) lookups
2017-12-15 20:13:44 -08:00
73a0a07eac
clients ask for key location information directly from the proxy, instead of reading it from the database
2017-12-09 16:10:22 -08:00
e900333dbf
Fix a subtle valgrind error.
...
If there was an error in waiting for the read version, we would attempt to
serialize and eventually commit a CommitTransactionRef that had an
uninitialized read_snapshot.
2017-11-15 19:21:20 -08:00
cd085764f1
Do not automatically change a cluster file that does not match what you expect.
2017-11-10 14:12:45 -08:00
33eef4c76b
Merge branch 'master' into getrange-perf-improvements
2017-11-04 14:25:45 -07:00
33cc644ba8
Copy CommitTransactionRequest during tryCommit function call as it is anyway copied inside the function.
2017-11-02 17:00:44 -07:00
3cad2676cc
Collapse some of the getRange actors into a single actor. Avoid unnecessary comparisons.
2017-11-02 13:39:06 -07:00
07c80484eb
Merge branch 'master' into getrange-perf-improvements
2017-11-02 08:42:26 -07:00
3efaaec479
onMasterProxiesChanged was being triggered when any member of ClientDBInfo changed. Change the behavior to be triggered only when proxies field in ClientDBInfo is changed.
2017-11-01 18:29:56 -07:00
1d95e1bfd6
Log actor cancellation error to Client Transaction sampling records.
2017-11-01 11:51:31 -07:00
7cf17df821
Merge branch 'master' into log-group-for-unsupported-clients
...
# Conflicts:
# flow/Net2.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2017-11-01 11:31:02 -07:00
e52f46064a
Don't use the results arena for conflict ranges. Rearrange the code to avoid the copy unless necessary.
2017-10-26 09:35:34 -07:00
0d68db0dac
Merge branch 'master' into getrange-perf-improvements
2017-10-26 09:25:04 -07:00
9dd588dcce
Addressed review comments.
...
Changed naming for NewMin and NewAnd to MinV2 and AndV2
2017-10-25 14:48:05 -07:00
a2e059be11
Eliminate some more copies
2017-10-20 09:27:17 -07:00
5fcd58b637
Whitespace fix
2017-10-20 09:24:49 -07:00
55bea14b9e
Convert extraConflictRanges from KeyRange to std::pair<Key, Key> to avoid copies.
2017-10-20 09:17:47 -07:00
7bab9a0276
Eliminate some copies
2017-10-20 09:11:26 -07:00
986a99a39a
Change the reply priority for NativeAPI requests to the cluster to TaskDefaultPromiseEndpoint.
2017-10-20 09:03:48 -07:00
e2c1e87df6
made a large number of fixes to make fearless DR correctness clean.
2017-10-19 15:36:32 -07:00
eeebf10030
Modified existing behavior of MIN and AND atomic ops. The new behavior results in a 'SET' if the atomic op is performed on a non -existing key.
...
Added new atomic ops ByteMin and ByteMax that does lexicographic comparison of byte strings.
2017-10-10 13:02:22 -07:00
ef41b07bb3
renamed past_version to transaction_too_old
...
implemented read_lock_aware option
2017-09-28 16:35:08 -07:00
d4f75630de
Support log group field in status json
2017-09-28 16:31:29 -07:00
53a4a3280a
fix: we cannot add to the trLog when cancelled
2017-09-20 14:47:57 -07:00
24aa616a7a
Merge pull request #154 from cie/additional-client-profiling
...
Additional client profiling
2017-09-19 18:15:02 -07:00
dc1f7ca6b7
testers now use client locality load balancing
2017-09-01 12:53:01 -07:00
aac82074af
Avoid calling setCachedLocation twice
2017-08-08 10:03:04 -07:00
e767860010
Addressed review comments.
...
Changed current protocol version to match master
Added operation details for operations that failed.
2017-08-07 18:45:42 -07:00
3e90fdfae7
Added extra client transactional profiling info
...
1) Key has been added to GET
2) KeyRange has been added to GETRANGE
3) ReadConflict, WriteConflict, Mutation info has been added to COMMIT
2017-08-01 18:33:39 -07:00
53e1ae9f60
shard system keyspace
2017-07-26 13:47:31 -07:00
035efd79cf
fix: if a database gets locked after an unknown result, the dummyTransaction will be stuck until the database is unlocked
2017-06-26 12:12:47 -07:00
03d2b1787a
Merge branch 'release-4.6' into release-5.0
...
# Conflicts:
# fdbrpc/sim2.actor.cpp
2017-06-22 16:56:25 -07:00
a674cb4ef4
Initial repository commit
2017-05-25 13:48:44 -07:00
sramamoorthy