Commit Graph

297 Commits

Author SHA1 Message Date
Chaoguang Lin 7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
sfc-gh-tclinkenbeard a32547cbbe Merge remote-tracking branch 'origin/main' into flow-enhancements 2022-02-15 15:56:42 -08:00
Ray Jenkins dd45805312
Merge branch 'apple:main' into threadname-issue-6064 2022-02-01 17:40:07 -06:00
Renxuan Wang 2ea4146e1f Add resolveTCPEndpointBlocking() to resolve hostnames where async resolving is impossible. 2022-01-28 12:20:41 -08:00
Ray Jenkins ff49bfaeb5 Add thread name for SSLHandshakerThread. 2022-01-24 20:00:00 -06:00
Renxuan Wang 28832a99d6 Address comment. 2022-01-18 14:34:18 -08:00
Renxuan Wang b8bab06e16 Add the functions to set and get mock DNS.
These functions will be used in restarting tests, where mock DNS needs to be saved to and read from files.
2022-01-18 14:34:18 -08:00
sfc-gh-tclinkenbeard 8076806e76 Add ITERATIONS_PER_REACTOR_CHECK 2022-01-01 20:14:57 -08:00
sfc-gh-tclinkenbeard 2ad8955dd3 Cleanup for Flow Enhancements PR 2022-01-01 20:04:00 -08:00
Jon Fu 476aaa73c6 modify benchmark to avoid threadsleep and change runloop to process IO properly 2021-12-16 16:20:22 -05:00
Jon Fu 169838cec3 stub changes for IO queue and attempt at new benchmark 2021-12-16 13:28:38 -05:00
Andrew Noyes def41697bf
Merge pull request #6083 from sfc-gh-tclinkenbeard/remove-temporaries
Avoid creating unnecessary temporary objects
2021-12-06 13:24:56 -08:00
sfc-gh-tclinkenbeard 3d36dfe5e9 Fix compilation error in resolveTCPEndpoint_impl 2021-12-02 12:51:00 -08:00
sfc-gh-tclinkenbeard 70c8f98eb9 Apply clang-format to Net2.actor.cpp 2021-12-02 10:22:22 -08:00
sfc-gh-tclinkenbeard d01a363e29 Avoid creating unnecessary temporary objects 2021-12-01 23:48:34 -08:00
sfc-gh-tclinkenbeard 90ced244eb Fix -Wunused-but-set-variable warnings 2021-12-01 18:15:53 -08:00
Renxuan Wang 22e34bd6b9 Replace <boost/bind.hpp> with <boost/bind/bind.hpp>.
This eliminates many useless warnings when compiling.
`#pragma message: The practice of declaring the Bind placeholders (_1, _2, ...) in the global namespace is deprecated. Please use <boost/bind/bind.hpp> + using namespace boost::placeholders, or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior.`
2021-11-18 14:00:13 -08:00
Lukas Joswiak 18243351e7 Fix possible data race
Transactions (created on a separate thread) can read the `globals` field
at the same time as `setGlobal` is called on the main thread, causing a
potential race. TSAN surfaced this issue.
2021-11-18 10:16:20 -08:00
Steve Atherton 035e0d6e52
Merge branch 'master' into bit-flipping-workload 2021-11-16 14:42:22 -08:00
Renxuan Wang 4630b0ccea Move DNS mock from SimExternalConnection to Sim2.
This is a revise PR of #5934. In simulation, we don't have direct access to SimExternalConnection.
2021-11-15 17:02:51 -08:00
negoyal 1e7338b6c3 Merge branch 'master' into bit-flipping-workload 2021-10-28 14:24:49 -07:00
sfc-gh-tclinkenbeard 49a667c29b Improve const-correctness of INetwork 2021-10-25 14:42:31 -07:00
negoyal f913dfed97 Merge branch 'master' into bit-flipping-workload 2021-10-11 16:34:57 -07:00
Xiaoge Su abf73047ca Enforce std:: specifier rather than using namespace 2021-09-16 19:40:28 -07:00
negoyal 3b34423248 Merge branch 'master' into bit-flipping-workload 2021-08-31 12:14:51 -07:00
Lukas Joswiak 5dc9a97230 Merge branch 'master' into fixes/alp6 2021-08-01 20:42:52 -07:00
Andrew Noyes 353efe7db2
Merge pull request #5264 from sfc-gh-tclinkenbeard/fix-more-clang-warnings
Enable more warnings for `clang`
2021-07-29 15:43:54 -07:00
negoyal 40b4f3b2f1 Merge branch 'master' into bit-flipping-workload 2021-07-28 18:06:07 -07:00
negoyal 050c218502 New Disk Delay Logic and ChaosMetrics. 2021-07-28 16:03:37 -07:00
sfc-gh-tclinkenbeard 94a65865d9 Merge remote-tracking branch 'origin/master' into fix-clang-warnings 2021-07-28 12:29:27 -07:00
sfc-gh-tclinkenbeard c74047c665 Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings 2021-07-28 11:51:02 -07:00
Evan Tschannen 81f93d794f Merge branch 'master' of https://github.com/apple/foundationdb into fix-ordered-delay 2021-07-27 13:56:35 -07:00
Evan Tschannen 256a18e43b Flow transport uses an ordered delay to avoid out of order reply promise stream messages 2021-07-27 12:01:32 -07:00
Lukas Joswiak 3eed4084e2 Merge branch 'master' into fixes/alp6 2021-07-27 11:26:53 -07:00
Lukas Joswiak 59d535149e Merge branch 'master' into fixes/alp6 2021-07-27 10:07:18 -07:00
Lukas Joswiak e9a1679467 Disable sampling everywhere except fdbserver 2021-07-27 09:53:23 -07:00
Steve Atherton 507c1f11e3 Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use. 2021-07-26 19:55:10 -07:00
sfc-gh-tclinkenbeard da50e13f3e Fix more -Wreorder-ctor warnings in DataDistribution.actor.cpp, OldTLogServer_4_6.actor.cpp, and Net2.actor.cpp 2021-07-24 17:33:11 -07:00
sfc-gh-tclinkenbeard 3442ebd3b7 Fix more -Wreorder-ctor warnings across many files 2021-07-24 11:20:51 -07:00
sfc-gh-tclinkenbeard e62e6503ac Fix most delete-non-virtual-dtor clang warnings 2021-07-21 23:32:44 -07:00
Daniel Smith 3fbd6b6143 Enable IThreadPool in simulation 2021-07-08 18:51:01 -04:00
Lukas Joswiak 250d20e7bf Even more cleanup 2021-06-15 15:17:54 -07:00
Lukas Joswiak 0301072690 Refactor 2021-06-11 13:01:32 -07:00
Lukas Joswiak e9fdbb9c86 Enable sampling 2021-06-07 18:01:14 -07:00
Lukas Joswiak 486a04659f Lazy inititialization 2021-06-04 15:01:18 -07:00
Lukas Joswiak ca79b8eaab Various ALP fixes 2021-06-04 15:01:18 -07:00
Lukas Joswiak 153de33f57 Revert "Merge pull request #4802 from sfc-gh-ljoswiak/revert/actor-lineage"
This reverts commit 6499fa178e, reversing
changes made to 1512631957.
2021-06-04 13:31:55 -07:00
A.J. Beamon a162682d6d Fix accounting for time spent in run loop after breaking due to yield 2021-05-14 11:12:47 -07:00
Lukas Joswiak 4ea760b2a9 Revert "Merge pull request #4136 from sfc-gh-mpilman/features/actor-lineage"
This reverts commit da41534618, reversing
changes made to e6300905d6.
2021-05-10 20:26:12 -07:00