29e328cf32
Use addressExcluded to compare exclusions and addresses
...
Don't convert the exclusion to a NetworkAddress use operator==, since that's problematic for two reasons:
1. The network address of the not-yet-excluded server might compare differently due to flags like private or public
2. The network address of the not-yet-excluded server will never compare equal to a whole-machine exclusion
2020-07-27 18:52:07 +00:00
09cf3e2883
Revert "Use IPAddress instead of NetworkAddress for exclude"
...
This reverts commit 331ee4dab4
2020-07-27 18:38:57 +00:00
331ee4dab4
Use IPAddress instead of NetworkAddress for exclude
...
NetworkAddress has other fields like flags that aren't relevant for
exclude. Network addresses with equal ip addresses not comparing equal
appears to be the cause of #3565
2020-07-27 05:40:14 +00:00
e1dedff7b3
Merge branch 'release-6.2' into release-6.3
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# cmake/ConfigureCompiler.cmake
# documentation/sphinx/source/downloads.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/FileBackupAgent.actor.cpp
# packaging/msi/FDBInstaller.wxs
# versions.target
2020-07-24 12:10:44 -07:00
5a8a8153ce
updated documentation
2020-07-23 11:22:16 -07:00
f5e7ee23a8
updated text of fdbcli output
2020-07-23 11:09:59 -07:00
229ab0d5f1
Fix some conflicts and remote debugging trace events
2020-07-22 23:35:46 -07:00
49bf42e66a
added the ability to suspend processes from fdbcli
2020-07-22 16:37:00 -07:00
302cf5c45f
Remove debug trace events
2020-07-22 12:20:22 -07:00
5b06d69d25
Pass watches test
2020-07-15 00:37:41 -07:00
e10704fd76
Cherry-pick region related status changes from 6.3
2020-06-09 14:56:21 -07:00
12225c0f46
Adopt consistent use of ... for repeating arguments
2020-05-27 15:27:47 -07:00
a632b26beb
Add documentation for several fdbcli commands that were present in fdbcli help but were not in our online documentation.
2020-05-27 13:29:01 -07:00
1f1a8df7ef
Auto completion for "throttle disable auto" were wrong because of logical operator grouping problems.
2020-05-22 09:48:53 -07:00
3a8efb2d0b
Added advanceversion command to fdbcli
2020-05-20 15:47:31 -07:00
40aec61d30
Added fdbcli getversion command
2020-05-20 15:47:30 -07:00
4f5bf41ed8
Print explanation when fdbcli unlock fails
2020-05-20 15:47:30 -07:00
9c6bec1af6
Add unlock command to fdbcli
2020-05-20 15:47:30 -07:00
b3aa54e7eb
Add lock command to fdbcli
2020-05-20 15:47:30 -07:00
cc4874918a
Merge branch 'release-6.3' into tag-throttling-by-priority
...
# Conflicts:
# fdbserver/Ratekeeper.actor.cpp
2020-05-20 14:26:35 -07:00
a3d4f3f4de
Merge pull request #3175 from ajbeamon/fdbcli-throttle-improve-hints-and-completion
...
Better hint and completion help with throttle command in fdbcli
2020-05-20 14:25:03 -07:00
b589c3c940
Remove IncludeVersions.h, which is no longer needed for cmake-only builds
2020-05-20 13:23:02 -07:00
14b23c146f
Support throttling and unthrottling tags by priority and their auto/manual state in fdbcli.
2020-05-15 12:47:55 -07:00
f1477b09e9
Better hint and completion help with throttle command in fdbcli.
2020-05-14 16:04:03 -07:00
8bd5dcaff8
Merge branch 'master' into atn34/special-key-versioning
2020-05-09 15:34:20 -07:00
a7c21cdadd
Use new worker_interfaces key range in fdbcli
2020-05-08 05:37:37 +00:00
b1055a8501
Merge branch 'master' into transaction-tagging
2020-05-05 16:03:39 -07:00
78d644f673
Merge branch 'master' into bugfixes/cmake-includes
2020-05-04 02:11:46 -07:00
decf3e82b0
Fix various bugs and make sure to cleanup throttles from the database when they expire
2020-05-01 21:36:28 -07:00
41c517a5dd
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2020-04-27 13:05:24 -07:00
239876351b
Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites).
2020-04-24 11:31:16 -07:00
35c18ac60a
Improvements to expiration. Encode throttles with auto/manual and priority in the key to support throttling the same tag with different values in these parameters.
2020-04-23 20:50:40 -07:00
f1dd0ee298
Protect against a ratekeeper starting up with a clock set in the past (compared to old ratekeeper) extending the duration of throttles excessively.
2020-04-21 16:35:25 -07:00
78d48a0dad
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbclient/Knobs.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/fdbclient.vcxproj
# fdbserver/MasterProxyServer.actor.cpp
2020-04-17 09:23:18 -07:00
2b66dcd24a
Some more refactoring. Reduce what is sent from RK->MP->clients
2020-04-17 08:07:01 -07:00
0fba8c47be
Checkpoint: Ratekeeper sets absolute limits for tag throttles and enforces them by distributing requests to proxies, who distribute them to clients.
...
A few refactorings.
2020-04-16 14:43:22 -07:00
93400d25c8
Added advanceversion command to fdbcli
2020-04-15 20:01:01 -07:00
bcb0a4cc78
Merge pull request #2890 from tclinken/expose-lock-fdbcli
...
Add lock and unlock commands to fdbcli
2020-04-10 13:54:47 -07:00
7f3fa00897
Reorganize a bit of code.
2020-04-10 13:53:23 -07:00
55a0d00ad4
Encoding of tags in the database now supports multiple tags per throttle. Remove throttle prefix search.
2020-04-10 10:12:26 -07:00
b44105b54c
Print explanation when fdbcli unlock fails
2020-04-08 16:38:30 -07:00
52860043c9
Merge remote-tracking branch 'origin' into atomically-update-dependent-knobs
2020-04-08 12:26:21 -07:00
36da61dd9c
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
2020-04-07 21:12:14 -07:00
a1d8623e5f
Various changes to the throttling scheme, including most notably that clients now enforce the throttles with info they receive from the proxy.
2020-04-07 16:28:09 -07:00
e4611e8ae4
fix versions.h stupidity
2020-04-06 10:28:55 -07:00
8b5780c36c
don't include source and binary dir
...
This forces users to use include paths from the sources root.
So `#include "Arena.h"` won't work anymore, only
`#include "flow/Arena.h"` will.
2020-04-06 10:13:49 -07:00
2336f073ad
Checkpointing a bunch of work on throttles. Rudimentary implementation of auto-throttling. Support for manual throttling via fdbcli. Throttles are stored in the system keyspace.
2020-04-03 15:24:14 -07:00
638369492f
Merge pull request #2863 from dongxinEric/feature/2290/expose-region-config-via-status-default-or-details
...
Expose region related configurations in status
2020-04-03 11:42:43 -07:00
eaae9397e5
Address review comments
2020-04-02 11:04:58 -07:00
aa7d5d8816
Add unlock command to fdbcli
2020-04-01 17:39:16 -07:00
884e92bb49
Atomically update dependent knobs
2020-04-01 15:18:49 -07:00
e755583c07
Address review comments.
2020-04-01 15:13:04 -07:00
8e489e7031
Merge pull request #2870 from dongxinEric/feature/2399/provide-comprehensive-info-about-exclude-in-fdbcli
...
Instead of providing partial information about exclude results in dif…
2020-03-31 11:58:41 -07:00
18356ec4e6
Apply suggestions from code review
...
Co-Authored-By: John Brownlee <mail@johnbrownlee.com>
2020-03-31 11:11:24 -07:00
bee6acb717
Added fdbcli getversion command
2020-03-30 17:12:52 -07:00
bb6a8bf7ab
Add lock command to fdbcli
2020-03-30 16:19:18 -07:00
012d41548e
Address review comments
2020-03-30 13:55:59 -07:00
52b6754598
Align up the outputs
2020-03-27 15:45:51 -07:00
248a097c58
Instead of providing partial information about exclude results in different scenarios, provide comprehensive result for all cases.
2020-03-26 13:15:47 -07:00
d7c7bbbea5
Fix macOS build failures
2020-03-25 16:39:41 -07:00
040c953f74
Expose region related configurations in status
2020-03-25 16:34:24 -07:00
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
b44747fe12
Merge pull request #2726 from alexmiller-apple/fdbcli-hints
...
Update LineNoise, and use hints to provide inline argument descriptions.
2020-03-17 11:24:44 -07:00
8e96e5a525
Expand comment about ignoring malformed escape sequences
2020-03-16 18:25:56 -07:00
6ee992eb35
Add that debug tls exits after running.
2020-03-13 15:47:16 -07:00
a5568b2fc6
Rewrite tlsinfo into --debug-tls, and print out configuration.
2020-03-13 15:46:03 -07:00
0c558efcfe
Add a `tlsinfo` command to fdbcli that prints the certificate chain.
...
This requires the certificate chain to load successfully, otherwise
fdbcli will error out at an earlier point due to Net2 not being able to
configure TLS.
2020-03-13 00:11:53 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
595dd77ed1
Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh
2020-03-04 20:25:42 -08:00
9b5ef3416e
Refactor TLSParams into TLSConfig + LoadedTLSConfig
...
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.
initTLS now just takes the in-memory bytes and applies them to the ssl
context.
This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
2020-03-04 20:14:47 -08:00
58e621eca1
Invalid knobs or knob values are treated as warnings rather than errors. Apply this change to backup as well.
2020-03-04 15:50:04 -08:00
9862aa8bed
Add support for setting knobs in fdbcli
2020-03-04 11:15:32 -08:00
243a4c4480
Reduce code indentation
2020-03-03 18:14:57 -08:00
6d14aacb40
I forgot to add the last change before committing and pushing the last commit.
2020-03-01 02:00:45 -08:00
e997be3e2d
Make fdbcli act like redis-cli, and a couple fixes.
...
Hints now show up as a darker gray text.
Hint components now only show for pieces of the command string that
hasn't yet been typed.
The code now correctly handles offering hints if multiple commands are
issued at once seperated by ';'.
A hint of "{malformed escape sequence}" is displayed if a bad escape
sequence was typed. e.g. \e
2020-02-29 03:07:39 -08:00
c11c24b79d
removed the fdbrpc version of platform.h
2020-02-28 14:56:10 -08:00
6054c05963
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbserver/fdbserver.actor.cpp
# versions.target
2020-02-28 12:11:05 -08:00
0f5c999d4b
Better containment of boost errors related to TLS.
2020-02-26 12:26:43 -08:00
b76f68d0e2
Add a --no-hints flag to fdbcli to disable linenoise hints
2020-02-24 16:08:04 -08:00
9545091899
Merge remote-tracking branch 'upstream/master' into fdbcli-hints
2020-02-24 16:02:30 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
390330176f
Update LineNoise, and use hints to provide inline argument descriptions.
...
This makes it so that if you type:
fdbcli> setclass
Then fdbcli shows
fdbcli> setclass <ADDRESS> <unset|storage|transaction|default>
^
With the cursor located where the ^ is pointing.
2020-02-21 17:15:40 -08:00
3a3f6afc4c
Update fdbcli/fdbcli.actor.cpp
2020-02-20 18:04:44 -08:00
f04e311a1e
Merge commit 'b46d6e25e24993ab5a5f04091fd3235050b7cd09' into feature-boost-ssl
...
# Conflicts:
# fdbserver/SimulatedCluster.actor.cpp
# flow/Net2.actor.cpp
2020-02-20 17:36:38 -08:00
927cff3317
Report errors on TLS misconfigurations ... or at least try to.
2020-02-20 16:57:29 -08:00
4f1301b2dd
Merge pull request #2583 from etschannen/feature-keep-status-connected
...
Clients should not disconnect from the CC after fetching status
2020-02-20 13:12:30 -08:00
08c318d28a
re-added the connect lock in the fdbcli so that the timeout is not spent before a connection has been initiated (because of the handshake lock)
2020-02-20 10:43:34 -08:00
761da5a059
code cleanup
2020-02-19 17:59:45 -08:00
53d0867a17
limit the number of connections a process can attempt to establish in parallel
2020-02-04 18:15:10 -08:00
d09e07f1f5
Merge remote-tracking branch 'upstream/master' into features/icc
2020-02-04 10:26:18 -08:00
61a727d701
Allow new databases to be configured as locked
2020-02-03 15:32:01 -08:00
ee6490c9d1
Merge pull request #2314 from mengranwo/memory-engine
...
New Radix-Tree based Memory Storage Engine
2020-01-30 16:20:13 -08:00
73ad702d14
Clients which fetch status should not disconnect from the coordinators and cluster controller between each retrieval
2020-01-22 15:41:22 -08:00
3f9d9d8b84
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# cmake/FlowCommands.cmake
# documentation/sphinx/source/release-notes.rst
# fdbclient/StorageServerInterface.h
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# flow/Knobs.h
# flow/Platform.cpp
# versions.target
2020-01-16 18:37:47 -08:00
227edd4248
change memory storage engine name from memory-radixtree to memory-radixtree-beta
2020-01-15 13:49:45 -08:00
f597aa7e18
WIP : deployable/stable version since Nov 3. Start rebase to master branch
2020-01-15 13:49:45 -08:00
9a3dfec7c5
open a connection with processes before attempting to kill them to improve the reliability of the kill process
...
secondaryAddresses are included in the list of processes which can be killed
2020-01-03 16:10:44 -08:00
92892d4168
Merge branch 'master' of https://github.com/apple/foundationdb into include-failed-ss
2019-11-25 11:05:35 -08:00
ed8d3f163c
Rename hgVersion to sourceVersion.
2019-11-15 12:26:51 -08:00
d4de608bb6
Fix OPEN_FOR_IDE build
2019-10-25 10:42:22 -07:00
bedb38d3e8
fixed compilation error
2019-10-24 13:05:28 -07:00
8930172e4a
fixed includeall case
2019-10-23 11:12:10 -07:00
9db95bd976
initial commit to allow re-inclusion of servers marked as failed
2019-10-23 11:05:48 -07:00
f38d52b82d
re-introduced coordinator quorum and added some comments
2019-10-18 14:52:07 -07:00
896701006f
addressed code review changes
2019-10-16 11:30:20 -07:00
d96a7b2c69
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-03 09:47:45 -07:00
d0147e5e5d
Merge branch 'release-6.2' into mengxu/merge-release620-to-master-v3
...
Resolved Conflicts:
documentation/sphinx/source/release-notes.rst
fdbserver/DataDistribution.actor.cpp
versions.target
2019-10-02 13:22:56 -07:00
450a09e117
Code Review Changes
2019-09-24 15:48:50 -07:00
471e283128
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-09-18 11:49:07 -07:00
9a64dc7f24
Merge pull request #1792 from ajbeamon/remove-dl-iterate-phdr-hack
...
Remove dl_iterate_phdr caching used by slow task profiler
2019-09-17 16:51:30 -07:00
a4d38f1158
Fix #2057 snapshot cli to print UID in failure too
2019-09-17 05:18:28 -07:00
bd7678e71b
Remove --object-serializer help text
2019-09-05 11:57:59 -07:00
dc1d055b27
Merge pull request #2042 from senthil-ram/snap_cli_fix
...
fix fdbcli --exec 'snapshot create.sh' failure
2019-08-30 13:40:38 -07:00
b3277f2982
Fix #2009 posix compliant args for snapshot binary
2019-08-30 12:54:09 -07:00
3f9e392668
Merge pull request #2014 from etschannen/feature-fdbcli-sleep
...
Added a sleep command to fdbcli
2019-08-30 11:22:13 -07:00
f3bc7e0abd
do not duplicate data distribution disabled fields in status
...
fixed a few bugs related to the existing data distribution disabled fields in status
2019-08-29 18:41:34 -07:00
00c2025d4b
fixed removeKeys impl, adjusted test workload, and introduced extra safety checks to NativeAPI and proxy
2019-08-27 14:39:44 -07:00
a07cf5d038
addressed code review comments
2019-08-27 14:39:44 -07:00
080fbc63dc
updated keyServers removal to be multi-transactional in order to avoid hitting transaction timeout
2019-08-27 14:39:43 -07:00
5a877d6b14
added safety check on client to prevent removing all servers from a team
2019-08-27 14:39:43 -07:00
807b02551e
updated help message and changed existing workload to use mark as failed feature
2019-08-27 14:39:43 -07:00
c908c6c1db
added command to fdbcli and changes to SystemData and ManagementAPI
2019-08-27 14:39:43 -07:00
e465cbef07
Merge branch 'master' into remove-dl-iterate-phdr-hack
...
# Conflicts:
# fdbserver/fdbserver.actor.cpp
2019-08-22 10:13:23 -07:00
0b0c9fe0ff
data distribution status was combined into regular status
2019-08-21 14:44:15 -07:00
7953545331
Fix an unknown_error when the file passed to fileconfigure doesn't contain a valid object (e.g. if you omit the enclosing {} of your object).
...
Fix an internal error when configuring regions with some storage servers that don't have a datacenter set.
2019-08-19 11:28:15 -07:00
d8ab48ce7f
added a sleep command to fdbcli
2019-08-16 18:13:35 -07:00
f08704557a
Merge pull request #1959 from alexmiller-apple/fix-fdbcli-help
...
Reword consistencycheck and remove snapshot.
2019-08-06 16:38:52 -07:00
370ba8b841
Remove --object-serializer flag from executables
2019-08-06 09:25:40 -07:00
aaf72c903d
Reword consistencycheck and remove snapshot.
...
`consistencycheck` help text made it sound like it would cause
consistency checking to be done, which was not the case. Consistency
checking still requires dedicated `-r consistencycheck` processes to be
running.
Snapshotting requires documentation and a bit more work, cleanup, and
polish before it could be used by users without great confusion. Thus,
it's being shuffled over to a hidden command, until that work is done.
2019-08-05 19:10:00 -07:00
476641a087
Merge pull request #1929 from jzhou77/fix-warning
...
Fix compiler warnings
2019-08-01 11:15:41 -07:00
37450be706
Fix format usage for currentProtocolVersion
...
ProtocolVersion now is a class.
2019-08-01 10:19:46 -07:00
b653ddb30d
Final clean ups after rebasing master
2019-07-30 22:35:34 -07:00
5d20364423
Address review comments
2019-07-30 22:24:30 -07:00
1922c39377
Resolve review comments. 100K run shows one suspecious ASSERT_WE_THINK failure which I think could be a race.
2019-07-30 22:24:30 -07:00
a1afafc17c
Accept suggested change.
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-07-30 22:20:45 -07:00
0f87ae91ac
Remove the unused variable
2019-07-30 22:20:45 -07:00
ae11efcb0a
Made following changes:
...
- Make sure the disabled data distribution won't be accidentally enabled by the 'maintenance' command
- Make sure the status json reflects the status of DD accordingly
- Make sure the CLI can play with the new DD states correctly, i.e. print out warns when necessary
2019-07-30 22:20:45 -07:00
4ecfc9830f
Added finer grained controls to DataDistribution in fdbcli. What's happening under the hood is:
...
- Use pre-existing 'healthZone' key and write a special value to it in order to disable DD for all storage server failures
- Use a new system key 'rebalanceDDIgnored' key to disable/enable DD for all rebalance reasons(MountainChopper and ValleyFiller)
Kicked off two 200K correctness and showed no related errors.
2019-07-30 22:17:21 -07:00
59dd5916c5
Merge branch 'master' into rename-fault-tolerance-status-fields
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-07-30 15:11:25 -07:00
438bc636d5
Rename max_machine_failures_without_losing_X to max_zone_failures_without_losing_X in status.
2019-07-30 14:02:31 -07:00
ec7e71ed93
Add reporting for zone count; fault tolerance reports in terms of zones unless machine ID matches zone ID.
2019-07-30 12:24:13 -07:00
5a56f6b456
minor snap create client improvement and bug fixes
2019-07-29 20:28:22 -07:00
8149b5b352
Merge pull request #1413 from atn34/change-connection-file
...
Switch cluster file feature
2019-07-26 13:27:37 -07:00
9afd162e2f
remove snap v1 related code
2019-07-25 17:29:31 -07:00
a2f2ad96ff
code review comments and merge to master changes
2019-07-24 15:36:28 -07:00
30a90e546d
make snap v2 the default version
2019-07-24 15:36:28 -07:00
209448807d
snap v2: fdbclient related changes
2019-07-24 15:36:28 -07:00
9ad9bd4c1f
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-07-24 15:22:26 -07:00
846038b0e6
Merge pull request #1858 from bnamasivayam/rk-ssfetch-throttle
...
Ratekeeper throttling aggressively when unable to fetch storage server list
2019-07-19 16:41:58 -07:00
3045826e3c
Merge pull request #1819 from mpilman/flatbuffers-fixes2
...
Flatbuffers fixes2
2019-07-19 16:33:50 -07:00
f6183df8b9
Merge pull request #1852 from vishesh/task/issue-1840-non-blocking-exclusion
...
fdbcli: Add `no_wait` option in `exclude` command to avoid blocking
2019-07-19 12:49:29 -07:00
d9a8657096
fdbcli: Add `no_wait` option in `exclude` command to avoid blocking
...
RESOLVES #1840
2019-07-18 13:07:31 -07:00
94c66f8d58
Merge pull request #1738 from bnamasivayam/consistency-check-disable
...
Disable/Re-enable consistency check through a database key.
2019-07-18 10:56:02 -07:00
1ac2d01b03
Merge remote-tracking branch 'upstream/master' into flatbuffers-fixes2
2019-07-18 09:50:08 -07:00
406bcebdc4
Ratekeeper to throttle tpsLimit to 1 if it is not able to fetch storage server list for some configurable amount of time.
2019-07-17 18:08:17 -07:00
d5caf0c1b4
Merge branch 'flatbuffers-fixes2' of github.com:mpilman/foundationdb into flatbuffers-fixes2
2019-07-16 14:47:40 -07:00
d5051b08dd
Make trace event field lengths (and total event sizes) default knobified and configurable. Add a transaction option to control the field length of transaction debug logging. Make the program start command line field less likely to be truncated.
2019-07-12 16:12:35 -07:00
969957e619
Merge branch 'master' into change-connection-file
2019-07-12 11:39:19 -07:00
f4366e69ca
Unknown options should not be used internally (i.e. underneath thread-safe API). This commit removes various checks that options exist and replaces them with an ASSERT.
2019-07-11 11:25:39 -07:00
1582a2a24d
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-07-09 13:41:54 -07:00
a5a6f8431c
Add a random UID to TransactionMetrics in case a client opens multiple connections and also a field to indicate whether the connection is internal. Convert some of the metrics to our Counter object instead of running totals.
2019-07-08 14:01:04 -07:00
9ed8eb2cdb
Explain strange use of literal byte strings
2019-07-05 14:07:02 -07:00
889e153b81
Add object serializer flag to fdbcli
2019-07-05 14:07:02 -07:00
6a899ddff3
Remove dl_iterate_phdr results caching that is used by slow task profiler, instead favoring disabling and reenabling profiling around the call. Add a mechanism to handle deferred profile requests.
2019-07-03 12:48:36 -07:00
7489f83a7f
Disable/Re-enable consistency check through a database key.
...
fdbcli has a new command 'consistencycheck' to disable/re-enable consistency check.
cluster_healthy metric in status becomes false if consistencycheck is disabled.
2019-06-20 21:38:45 -07:00
844dd60202
FDB compiling with intel compiler
2019-06-20 09:29:01 -07:00
02e173b601
Add changeConnectionFile method to Transaction
...
Also add tests
2019-06-11 13:58:22 -07:00
61e93a9304
Address review comments and minor fixes
2019-05-28 22:07:46 -07:00
69edefe68b
Snapshot based backup and resotre implementation
2019-05-28 22:07:46 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
1ce585b5ce
fdbcli: fix some print/scan format warnings
...
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-06 13:35:29 -07:00
2d5043c665
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-04-30 18:27:04 -07:00
b2d19eebc4
fdbcli would return success even when configure failed for a variety of error types
...
the existing configure safety check would fail when attempting to change between three_datacenter and region configuration
2019-04-30 16:30:19 -07:00
6220a5ce0f
Merge pull request #1370 from jzhou77/fix-unreferenced
...
Remove unused functions
2019-04-09 11:49:45 -07:00
dfad79d577
Update fdbcli/fdbcli.actor.cpp
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-04-05 13:12:19 -07:00
9595b43a2d
Update fdbcli/fdbcli.actor.cpp
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-04-05 13:12:19 -07:00
a05350bfa7
Update fdbcli/fdbcli.actor.cpp
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-04-05 13:12:19 -07:00
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
3371cf22d4
Add manually triggered heap profiling
...
At client side:
fdb> profile
ERROR: Usage: profile <client|list|flow|heap>
fdb> profile heap 127.0.0.1:4500
On the server side:
$ HEAPPROFILE=/tmp/fdbserver bin/fdbserver -C ../test.cluster -p 127.0.0.1:4500
Starting tracking the heap
FDBD joined cluster.
Dumping heap profile to /tmp/fdbserver.0001.heap (1024 MB allocated cumulatively, 13 MB currently in use)
Dumping heap profile to /tmp/fdbserver.0002.heap (User triggered heap dump)
2019-04-03 16:00:54 -07:00
39c595223b
Merge branch 'release-6.1'
2019-04-02 22:30:02 -07:00
628fec8c8b
updated status with information about ongoing maintenance
...
clear the maintenance zone if a different storage server is detected failed
2019-04-02 14:15:51 -07:00
72203ba47a
Merge commit '56f3f0b1bc60604f965152d856ae29a591227703'
2019-04-01 18:45:38 -07:00
781cf9b5a0
added the ability to make a zoneId for maintenance in fdbcli
2019-04-01 17:55:13 -07:00
f7f8ddd894
Fix warnings on unused variables
...
Found by -Wunused-variable flag.
2019-04-01 14:00:20 -07:00
1ba0b4e682
fix: Parse IPv6 addresses correctly in status details
2019-04-01 12:58:26 -07:00
7c02ee6fdd
Fix compiler warning about unreferenced exception variable
2019-03-26 13:43:47 -07:00
555405d1ab
Set TRACE_FORMAT network option in fdbcli
...
Previously --trace_format did not work since the network option has the
last say in what trace format to use.
2019-03-22 10:35:05 -07:00
eb54a700ba
changed the old memory configuration to memory-1
2019-03-18 15:10:04 -07:00
a372c7cf18
configure memory now selects the ssd engine for transaction log spilling. Transaction log spilling is only used when the transaction logs cannot keep all of the unpopped mutations it has in memory. If we are only using this data structure because we do not have enough memory, it is much less safe to use the memory storage engine for this purpose.
2019-03-16 22:48:24 -07:00
ed49d603a0
Allows cluster string to contain coordinators with different TLS states
...
During live TLS upgrades, we can hence switch one coordinator at a time
to TLS than all of them together.
2019-03-06 16:05:10 -08:00
57832e625d
net: Support IPv6 #963
...
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.
- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.
- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.
- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
b8910ba7cd
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.h
# fdbserver/DataDistribution.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-22 14:38:13 -08:00
0bb60e5a3b
Use proper fwd decl in NativeAPI
...
Also NativeAPI.h -> NativeAPI.actor.h
2019-02-19 15:16:59 -08:00
3cb2391b58
use proper fwd declarations in ManagementAPI
...
Also ManagementAPI.h -> ManagementAPI.actor.h
2019-02-19 15:16:59 -08:00
065a45e05f
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-18 17:09:06 -08:00
8f2af8bed1
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller.
...
fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date.
fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-18 14:54:28 -08:00
e05b53d755
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-15 20:37:07 -08:00
c03de6c7b6
Update CLI to take addresses with mixed TLS states
2019-02-15 20:23:07 -08:00
b435d51061
Merge branch 'master' into track-server-request-latencies
2019-02-14 08:07:32 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
6da5971e79
Guard all versions.h to not break old WIN32 build
2019-02-08 16:06:00 -08:00
7e26b4ef0d
Address comments from PR
2019-02-07 15:37:04 -08:00
8a94d80deb
fdbservice and fdbrpc now compiling
2019-02-07 15:37:04 -08:00
750124f083
Accounting only non-excluded processes in the user output if recruiting is going on.
2019-02-06 17:04:38 -08:00
9c19d9e9eb
Improve status output to reflect that tlogs are recruited across availability zones according to the configured replication factor.
2019-02-04 18:14:00 -08:00
2198d24ce1
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies
...
# Conflicts:
# fdbclient/MasterProxyInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/ServerDBInfo.h
# fdbserver/Status.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/storageserver.actor.cpp
2019-01-24 11:43:26 -08:00
8e05e95045
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-18 16:18:34 -08:00
d4d5740282
* Add Optional.map and ErrorOr.map.
...
* Rename Optional/ErrorOr cast_to to castTo.
* Make printable(Optional<T>) templated rather than restricted to StringRef types.
* Fixes bug in (unused) ErrorOr.castTo where an ErrorOr that was not set would lose its error.
2019-01-11 09:03:38 -08:00
7eb6765698
Mention that xml is the default
2019-01-03 08:48:31 -08:00
bce5b03340
Fix whitespace
2019-01-02 15:24:11 -08:00
1bca665b29
Document --trace_format flag
2018-12-20 16:22:41 -08:00
03b48fb452
s/--trace-format/--trace_format/
2018-12-20 15:58:26 -08:00
b8df5acc15
Add --trace_format flag to fdbserver
2018-12-20 15:02:01 -08:00
d2d68aa171
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# versions.target
2018-12-03 18:26:52 -08:00
6e94d7d71a
changing the database configuration now checks that enough processes exist to support the new configuration
2018-11-30 18:52:24 -08:00
4e54690005
Merge branch 'release-6.0'
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
# fdbserver/MoveKeys.actor.cpp
2018-11-12 20:26:58 -08:00
ccdc83036d
simplified region configuration protections
2018-11-12 17:45:20 -08:00
6353a6724b
strengthened the protections related to changing regions
2018-11-12 17:40:40 -08:00
4b5d0b4e2c
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/AsyncFileBlobStore.actor.cpp
# fdbclient/AsyncFileBlobStore.actor.h
# fdbclient/BlobStore.actor.cpp
# fdbclient/BlobStore.h
# fdbclient/HTTP.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/batcher.actor.h
# fdbrpc/fdbrpc.vcxproj
# fdbrpc/sim2.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2018-11-10 13:04:24 -08:00
a654183f63
Merge pull request #791 from ajbeamon/remove-cluster-from-iclientapi
...
Remove cluster from IClientApi (phase 2 of removing DB names)
2018-11-10 10:16:18 -08:00
c02690471d
added protection against configuration changes which cannot be immediately reverted
...
the configure database workload tests region configurations
2018-11-04 19:53:55 -08:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
3eb4355a48
Some various cleanup and fixes. Added "Cluster" to TransactionMetrics trace event.
2018-09-25 15:06:19 -07:00
c831051474
This removes the idea of clusters from IClientApi.
2018-09-21 15:58:14 -07:00
90301f497f
Merge branch 'release-6.0'
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/TLSConnection.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# versions.target
2018-09-05 16:06:33 -07:00
4eaff42e4f
Merge pull request #712 from ajbeamon/remove-database-name-internal
...
Eliminate use of database names (phase 1)
2018-09-05 10:35:00 -07:00
40f5dbe423
fixed issues from review, added a safeguard to prevent configuring a cluster to an invalid configuration
2018-09-04 22:16:35 -07:00
e1fafbf259
Add some missing actorcompiler.h includes
2018-08-22 09:40:45 -07:00
d8ea3dbf9a
Added the ability to configure a cluster from a JSON file
2018-08-16 17:34:59 -07:00
2a97139d5d
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-16 10:24:12 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
1c29275672
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-01 14:30:57 -07:00
507b3bacb0
fix: kill all tlogs in one region prevents the remote logs from recovering in that region, do not allow that to prevent us from configuring usable_regions=1.
...
added more recovery states.
2018-07-05 00:08:51 -07:00
cbdf598fa2
Add force_recovery_with_data_loss to hidden command list.
2018-07-03 15:04:11 -07:00
e67f951c06
Merge branch 'master' into feature-remote-logs
2018-07-02 02:18:20 -04:00
c3f88dbfe1
Merge branch 'master' of github.com:apple/foundationdb into tls-static
2018-07-01 23:13:57 -07:00
7a12d3e130
added the (untested) ability to force a recovery to the remote datacenter, even if that results in data loss. If the DR lag is more than 1 week there could be potential data corruption if any primary storage servers are still alive.
2018-07-01 09:39:04 -04:00
45849d1f95
Added support for no-op legacy TLS options
2018-06-27 09:25:05 -07:00
dd967bd9e2
Removed developmental debug messages
2018-06-26 14:44:21 -07:00
ef8de426d3
Changed the TLS_DISABLED macro
...
Disable TLS within Windows until working
2018-06-26 12:08:32 -07:00
8a8914f046
re-added the ability to configure the number of log routers. Many log routers are needed to get a sufficient number of sockets involved in copying data across the WAN
2018-06-22 00:04:00 -07:00
f8ce1de601
Added support for compiling TLS into binaries
2018-06-20 09:21:23 -07:00
39894ea798
Merge remote-tracking branch 'apple/release-5.2'
2018-06-12 18:31:20 -07:00
cfa7fe8866
Identify processes with host:port regardless of if TLS is enabled or not.
...
This makes `kill` and `profile` behave like how `exclude` functions, and means
commands don't have to change depending on TLS status.
Verified via starting a TLS cluster and killing a process before and after this change.
2018-06-11 16:49:20 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
026458baf3
Merge release-5.2 into master
2018-05-23 15:32:56 -07:00
086700aeb1
Plumb through TLS key password to CLI and from environment
2018-05-21 10:56:10 -07:00
8f984cb2c9
Merge branch 'release-5.2'
...
# Conflicts:
# fdbrpc/TLSConnection.h
2018-05-10 09:13:22 -07:00
e8b7f4b190
Add password support for tls.
2018-05-08 20:46:31 -07:00
d3b5cfb93c
Support latest TLS plugin.
...
Add support for https in backup.
2018-05-08 16:28:13 -07:00
Andrew Noyes