5f079904d4
add unit test
2021-03-16 18:30:39 +00:00
ffb8e27f43
Merge pull request #4461 from sfc-gh-anoyes/anoyes/fix-thread-leak
...
Fix several leaks
2021-03-15 22:03:45 -07:00
f0d79b3d86
Inital implementation of network busyness
2021-03-15 23:23:56 +00:00
4bee976b37
completely ignore messages to unknown endpoint
2021-03-12 09:35:53 -07:00
a76c7b9754
Fix several memory leaks and a thread leak
2021-03-11 23:58:32 +00:00
2cd3f45fd6
Merge remote-tracking branch 'apple/release-6.3' into master-format-final
...
release-6.3 was recently merged, and there were two PRs which were
merged in between and got those changes in here. Hence, since all the
changes were in, discarded the incoming changes and accepted all
current.
2021-03-10 16:50:44 -08:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
36bf68ed2e
Merge branch 'release-6.3' into clang_format_the_world/6.3
2021-03-08 14:40:52 -08:00
a7ccb6669d
Change Net2::connect to only call initTLS for tls
2021-03-08 16:27:04 +00:00
8a8c488ede
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-05 18:13:38 -06:00
23b85256ae
Change default to 64, and only start threads if using TLS
2021-03-04 20:52:46 +00:00
d718e7baf3
Merge pull request #4179 from sfc-gh-tclinkenbeard/run-minio-joshua
...
Support Testing S3BlobStore backup in simulation
2021-01-26 14:39:57 -08:00
564c70acb4
Merge branch 'master' into anoyes/modernize-use-override
2021-01-25 18:26:18 -08:00
4ee97c0784
Use clang-tidy to automatically fix missing overrides
...
Use `clang-tidy -p . $file -checks='-*,modernize-use-override' -header-filter='.*' -fix`
to fix missing overrides, and then use git clang-format to reformat just
those changes. This went pretty well for most files.
Formatting the following files went off the rails, so I'm going to
follow up with a commit that's just clang-tidy and no clang-format.
- fdbclient/DatabaseBackupAgent.actor.cpp
- fdbclient/FileBackupAgent.actor.cpp
- fdbserver/OldTLogServer_4_6.actor.cpp
- fdbmonitor/SimpleIni.h
- fdbserver/workloads/ClientTransactionProfileCorrectness.actor.cpp
2021-01-26 02:04:12 +00:00
adde95bea5
Merge branch 'release-6.3' into anoyes/merge-6.3-to-master
...
Commit merge conflicts for now
2021-01-25 18:55:24 +00:00
51cbc494f1
Merge remote-tracking branch 'origin/master' into run-minio-joshua
2021-01-23 11:31:19 -08:00
065c7e570c
Revert "Make g_net2 a unique_ptr" and add comment
2021-01-21 14:31:03 -08:00
fb9b4b7626
Merge remote-tracking branch 'origin/master' into misc-changes
2021-01-21 10:45:50 -08:00
281151ff20
Merge remote-tracking branch 'upstream/release-6.2' into anoyes/merge-6.2-to-6.3
...
Commit conflict markers for now
2021-01-21 17:54:26 +00:00
3f7e19a182
Add tags to spans to provide additional metadata
2021-01-20 13:57:36 -08:00
98e6ba6962
Add POSIX send functionality
2021-01-20 13:57:36 -08:00
d2143f184a
Add fast UDP tracer
2021-01-20 13:57:36 -08:00
4ddf3385a8
Reserve the priority queue container
2021-01-20 10:34:48 -08:00
61a29ecfc8
Merge remote-tracking branch 'origin/master' into run-minio-joshua
2021-01-01 09:38:36 -04:00
5bfa6cea98
Merge remote-tracking branch 'origin/master' into misc-changes
2020-12-26 20:47:00 -04:00
913401aaee
Move SendBufferIterator out of network.h
...
This class is only needed for implementation in Net2.actor.cpp and
SimExternalConnection.actor.cpp, so this class should not be included
everywhere network.h is included
2020-12-26 16:30:00 -04:00
201857541d
Remove unnecessary copies from INetworkConnections interface
2020-12-24 21:20:03 -04:00
1e7ea8ece1
Implemented SimExternalConnection::write
2020-12-24 21:20:02 -04:00
fed1a03950
Added SimExternalConnection (most methods not implemented yet)
2020-12-24 21:20:02 -04:00
c8786a65c3
Remove unnecessary uses of virtual
2020-12-08 09:09:33 -08:00
fad43687d1
Make g_net2 a unique_ptr
2020-12-08 09:09:31 -08:00
2b92cfbde8
Fix Net2 method signatures
2020-12-08 09:09:30 -08:00
c77d9e4abe
merge conflicts
2020-12-02 21:53:19 +00:00
d64cf8b9e3
Merge branch 6.3 into master
2020-11-17 11:22:45 -08:00
d3598bfaa4
address review comments
2020-11-13 15:37:03 -07:00
54df295186
remove dead code
2020-11-12 14:00:34 -07:00
0f205620bf
Merge branch 'master' into features/udp
2020-11-12 13:51:17 -07:00
1c05b70942
Add INetwork::timer_monotonic
2020-11-05 17:07:49 +00:00
d1f38077f5
remove virtual
2020-11-02 19:41:20 +00:00
055add9682
conflicts
2020-10-23 06:33:00 +00:00
8cc3e4d3c6
Merge release-6.3 into master
2020-10-19 22:51:56 -07:00
545ee4269d
master conflicts
2020-10-19 01:03:54 +00:00
bbf5bdf6da
fix stable interfaces test and corresponding changes in simulator
2020-10-12 18:25:12 +00:00
41843f07e6
add simulator support for different process versions and ProtocolVersion test
2020-10-12 18:19:31 +00:00
a9607bdcec
Explicitly seal classes that inherit but aren't inherited from
2020-10-07 21:58:24 -07:00
8571dcfe28
Use override where applicable in fdbserver
2020-10-07 18:41:19 -07:00
2c5a2909cd
fixed compiler error
2020-10-06 09:44:16 -07:00
59d4e8b629
added a timeout to TLS handshakes
2020-10-05 15:24:03 -07:00
0ac08f6a9b
Replace NULL with nullptr in flow
2020-09-20 11:31:49 -07:00
6d84bcb568
Merge remote-tracking branch 'origin/master' into features/udp
2020-08-06 14:08:34 -06:00
8976694ba1
UDP implementation (untested)
2020-08-06 14:06:50 -06:00
a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
c4f3553826
TLS handshake limit changed to regular flow lock, added new knob for simultaneous handshakes limit and removed old knobs. Added knob to control how much data the FlowTransport connectionWriter() can write using multiple write() calls before yielding to other tasks, and updated p2p network test to use the same logic. Changed knob defaults for send limit on single writes and TLS handshake thread count.
2020-07-16 02:31:17 -07:00
5151f1167c
Added TLS option to limit the number of glibc malloc arenas if TLS background threads are created. TLS initialization now has three phases in order to avoid creating TLS threads or limit the malloc arena count unnecessarily.
2020-06-28 17:29:13 -07:00
813dbfb297
TLS handshake threads are now created with TLS initialization instead of when the network is created since they may not be needed and it avoids calling startThread() with a custom stack size too early in process execution. StartThread() now logs a warning if the given stack size can't be used. Reorganized TLS handshake thread state near the other TLS related stuff.
2020-06-23 03:09:56 -07:00
c3ce0034bf
Background threads for TLS handshakes use a stack size based on a knob. Platform::startThread() now accepts a stack size. The generic thread pool implementation takes an optional stack size override which is used for each added thread.
2020-06-23 02:08:01 -07:00
5b19233989
Implemented count of used background TLS handshake threads differently using a generic utility class that updates the count on destruction.
2020-06-23 00:45:07 -07:00
a48d005f61
Bug fixes, stats improvements.
2020-06-22 13:37:13 -07:00
d75b4c9a69
Bug fix, accidental local variable redeclaration.
2020-06-22 05:53:37 -07:00
d681e9d350
Follow TLS_DISABLED macro setting.
2020-06-21 11:21:24 -07:00
6330f8b458
TLS handshakes can now be done using up to a configured number of background threads if any of them are not busy.
2020-06-21 06:23:32 -07:00
d7cb54d2b2
Fixed Windows CMake build
2020-06-20 23:03:18 -07:00
a59925dd73
Added more const-correctness improvements
2020-06-20 22:15:19 -07:00
99bf993815
Replace BOOST_NOEXCEPT with noexcept
2020-06-09 22:39:19 -07:00
30bfd606c0
Merge branch 'release-6.2' into release-6.3
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/downloads.rst
# documentation/sphinx/source/release-notes.rst
# fdbserver/worker.actor.cpp
# packaging/msi/FDBInstaller.wxs
# versions.target
2020-06-04 19:21:32 -07:00
bf072d68ec
Workarounds for strange behaviors in Boost ssl sockets on MacOS and Linux. When writing to the ssl socket, write_some() would sometimes return BrokenPipe instead of WouldBlock unless onWriteable on the raw socket was checked first. On MacOS, even with the onWriteable check using a send size greater than 2016 (determined experimentally) would still result in the error. Also consolidated two identical copies of SendBufferIterator.
2020-06-04 01:44:05 -07:00
86f712657e
Eliminate the undefined behavior of calling run_network twice, instead returning an error.
2020-05-22 13:31:06 -07:00
c2e80fe47b
Linux aarch64 port
2020-05-09 22:14:03 +00:00
95bc24de11
flow: update headers and includes
2020-04-30 18:11:23 +00:00
c87aa33941
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/go/src/fdb/generated.go
# documentation/sphinx/source/api-common.rst.inc
# documentation/sphinx/source/api-ruby.rst
# documentation/sphinx/source/release-notes.rst
# fdbclient/FailureMonitorClient.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/OldTLogServer_6_0.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# versions.target
2020-04-23 13:47:53 -07:00
c6df20a179
Use nullptr instead of NULL
2020-04-21 20:39:45 -07:00
a51746b307
Match 6.2.15's behavior in how invalid/unreadable/non-existent certs are handled.
...
Which is to proceed past Net2 creation, and allow certificate refresh to
try and eventually load valid certs. Additionally, fix certificate
refeshing dieing if the certificate is not readable when first called.
In testing, I also found and fixed an issue where if a cert went from
unreadable to readable, we wouldn't reload the TLS context, due to not
considering it as a file change.
2020-04-20 21:38:04 -07:00
20fe068863
Merge branch 'tls-background-eio-thread' into tls-permission-errors
2020-04-20 20:51:05 -07:00
49c6bb90ef
Merge pull request #2982 from alexmiller-apple/tls-log-settings
...
Log Net2TLSConfig with paths and settings when using TLS.
2020-04-20 15:46:26 -07:00
75a4f3b7c9
Remove comment about ignoring runOnMainThread errors.
...
If we got an exception, it wouldn't be of type `Error` anyway, so
it seems like things would crash regardless.
2020-04-20 13:19:42 -07:00
da8e47ea25
Merge remote-tracking branch 'upstream/release-6.2' into tls-background-eio-thread
2020-04-20 13:15:05 -07:00
5c399bf725
Move the callbacks into ::run() right before it exits.
...
stopped=true doesn't cause the run loop to immediately exit.
2020-04-20 13:14:19 -07:00
2ce539ef6d
Respect flow<->fdbrpc module boundaries.
...
Which fixes a compilation error due to a circular dependency between
flow.a and fdbrpc.a. However, this is now done at the cost of newNet2
users have to remember to add Net2FileSystem::stop() as a callback.
2020-04-20 02:53:07 -07:00
cbb6ffb431
Only log OpenSSL error strings for OpenSSL errors.
...
Normal "connection refused" messages would show up with a long verbose
string that doesn't really provide any useful information otherwise.
2020-04-18 20:39:02 -07:00
11eebc4a48
Log Net2TLSConfig with paths and settings when using TLS.
...
There were similar TraceEvents in the FDBLibTLS/LibreSSL TLS
implementaiton that were accidentally dropped in the TLS rewrite.
This makes it so that one does not have to use magic to figure out if a
process was configued with TLS correctly when some of the settings come
from environment variables.
2020-04-18 20:21:10 -07:00
1398e9a82e
Stop background eio threads on Net2::stop().
...
This will stop eio threads for both the client (`fdb_stop_network()`)
and the server. This change is being done more for the former, but I
don't see any harm in doing the latter as well.
2020-04-18 19:40:55 -07:00
d8690d31cd
Merge branch 'master' into per-priority-busy-logging
...
# Conflicts:
# flow/Net2.actor.cpp
2020-04-15 08:31:30 -07:00
b1172417f5
Merge branch 'master' into per-priority-busy-logging
...
# Conflicts:
# flow/Knobs.cpp
# flow/Knobs.h
# flow/Net2.actor.cpp
2020-04-14 14:22:12 -07:00
e104a2e3a6
Merge commit 'cf01233f28a2c42908656a39f458a4475c1d44a3' into run-loop-busy-profiler
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/NativeAPI.actor.h
# fdbserver/fdbserver.actor.cpp
# flow/Net2.actor.cpp
2020-04-14 14:02:24 -07:00
cf01233f28
Merge pull request #2549 from ajbeamon/slow-task-and-priority-tracking-improvements
...
Slow task and priority tracker improvements
2020-04-14 13:26:32 -07:00
903128a36f
Fix boost error trace event fields to match other trace events by the same name.
2020-04-01 08:18:50 -07:00
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
db5863145a
Merge remote-tracking branch 'upstream/release-6.2' into fdbcli-tlsinfo
2020-03-16 12:33:50 -07:00
75e2fffe5a
Add a ProcessMetrics.TLSPolicyFailures metric
...
This reports the number of policy failures over the past 5s interval.
It also is step 1 towards getting this information into status json.
2020-03-13 02:24:37 -07:00
0c558efcfe
Add a `tlsinfo` command to fdbcli that prints the certificate chain.
...
This requires the certificate chain to load successfully, otherwise
fdbcli will error out at an earlier point due to Net2 not being able to
configure TLS.
2020-03-13 00:11:53 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
188d9b8239
Don't swallow actor cancellation in certificate refreshing.
2020-03-06 11:09:17 -08:00
9b760fae2d
Rewrite all Errors into tls_errors if they happen as part of initializing TLS.
2020-03-06 11:06:19 -08:00
ac52b6b474
Rework a bit of error and exception handling.
...
I went back and dug through all of the "what functions can throw what
types", and made sane decisions about them. boost errors are
aggressively translated into FDB ones, whcih might result in multiple
lines of logging about errors, but this is in infrequently run code, so
it should be fine.
2020-03-06 02:33:16 -08:00
ccef3f7d05
Attempt to fix TLS_DISABLED compiles.
2020-03-05 17:32:10 -08:00
2d95a1e64d
Implement certificate refreshing
2020-03-05 17:25:33 -08:00
595dd77ed1
Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh
2020-03-04 20:25:42 -08:00
9b5ef3416e
Refactor TLSParams into TLSConfig + LoadedTLSConfig
...
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.
initTLS now just takes the in-memory bytes and applies them to the ssl
context.
This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
2020-03-04 20:14:47 -08:00
7cbabca124
remove printing to stderr from initTLS because that could cause problems on clients
2020-03-04 15:06:22 -08:00
35a1ac6482
prepare net2 for new versions of boost
2020-03-04 14:26:01 -08:00
Nim Wijetunga