Commit Graph

393 Commits

Author SHA1 Message Date
A.J. Beamon 4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
sfc-gh-tclinkenbeard 82adc1e856 Make g_simulator a pointer 2022-09-15 09:00:33 -07:00
Junhyun Shim b4e428fa33 Address review comments
- Revert JWT timestamp from double back to uint64_t
- Describe in comment the need for wait loop at the beginning of watchPublicKeyJwksFile()
- Revert CLI args definition in fdbserver.actor.cpp that was
  incorrectly tabbed with a bad clang-format run
2022-08-29 12:49:03 +02:00
Junhyun Shim 6bcfbf421e Merge remote-tracking branch 'origin/main' into authz-general-tls-and-integration-test 2022-08-26 09:20:31 +02:00
Chaoguang Lin 06aa6ee5ff
Add system monitor for flowprocess (#6925)
* Update network address in trace logs; Add system monitor for flowprocess

* Create a new trace file with the correct process address for flowprocess

* Remove unused debugging traces

* Add a new error lock_file_failure; Change please_reboot_remote_kv_store to please_reboot_kv_store; Add the code to only reboot the kv store but not the worker; Remove some unnecessay traces

* Add error handling for file_not_found in handleIOErrors

* Format worker.actor.cpp file
2022-08-24 00:40:38 -07:00
Junhyun Shim fc461fb701 Apply Clang format 2022-08-23 17:13:55 +02:00
Junhyun Shim a312002333 Merge remote-tracking branch 'origin/main' into authz-general-tls-and-integration-test 2022-08-17 17:32:00 +02:00
Junhyun Shim bcb9bc7e62 Set up authz integration test environment using Python venv
Add Authz-related support to {tmp|local}_cluster.py
Also fix public key set load/refresh mechanism
2022-08-17 14:57:05 +02:00
Vaidas Gasiunas 79571dd2b4
Testing upgrades to a future version of FDB (#7780)
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli

* Auto format python files used in upgrade tests

* Add a test for upgrading to a future FDB version

* Emphasize that the options for using future protocol version are intended for test purposes only

* Make the global variable for current protocol version visible only locally

* Refactirng to avoid using currentProtocolVersion() in static intialization

* Update go bindings
2022-08-08 17:29:49 +02:00
Junhyun Shim 65b441bd1b Watch public key file at CLI-arg path
Argument name: authorization-public-key-file
2022-07-28 20:03:19 +02:00
Junhyun Shim 1dbb13a899 Accept non-mTLS clients as untrusted 2022-07-28 20:03:07 +02:00
Junhyun Shim 78ea2b40b6 Add assertion to check g_currentDelivery* is unset before delivery 2022-07-28 01:16:42 +02:00
Junhyun Shim e2a3fedfc7
Merge branch 'main' into features/authz 2022-07-27 00:08:57 +02:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
Junhyun Shim ac5436a090 Catch up on changes to main 2022-07-12 16:48:09 +02:00
Junhyun Shim 545a9a8043 Fix bugs and add token timeout-in-cache test 2022-07-11 16:58:04 +02:00
Markus Pilman 56541167c2 send token with tenant info object 2022-07-06 20:58:17 -06:00
Markus Pilman 624430b6ae addess review comments 2022-06-27 14:22:23 -06:00
Markus Pilman 13d8b13722 Migrated Authz code to use JWT 2022-06-13 18:20:27 -06:00
Markus Pilman 799fe32346 Merge remote-tracking branch 'origin/main' into features/authz 2022-06-13 18:02:11 -06:00
Josh Slocum 5621fcff40 formatting fix 2022-05-25 17:16:56 -05:00
Josh Slocum fd7b662767 cleaning up 2022-05-25 17:16:56 -05:00
Josh Slocum e3a570d50d Blob worker tenant integration mostly works 2022-05-25 17:16:56 -05:00
A.J. Beamon 993a79b42c
Restart protocol monitoring logic when we cycle the coordinator, even if its the same coordinator. Add some defensive code in case we fetch the protocol version for an absent peer. (#6397) 2022-05-19 10:00:23 +02:00
Vaidas Gasiunas b55863d958
Merge pull request #7132 from sfc-gh-vgasiunas/vgasiunas-fix-version-check-for-downgrades
Fixing the problem with client getting stuck on server downgrades
2022-05-13 10:53:18 +02:00
Andrew Noyes a92ef37d44 Log a backtrace before throwing serialization_failed 2022-05-12 20:08:36 -07:00
Vaidas Gasiunas 2f67637701 Fixing the problem with client getting stuck on server downgrades:
- Restoring the original check for strict protocol compatibility before sending packets
 - Resetting the compatibility flag when connection is closed, so that the protocol compatibility is checked again for a new connection
2022-05-11 19:12:38 +02:00
A.J. Beamon 0597ae7513 When an incompatible connection is closed, clear the state that prevents us from sending messages to it 2022-05-10 15:29:36 -07:00
Ray Jenkins dc9e782ccc
OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
Markus Pilman 145f3d15bc implemented token caching 2022-04-26 10:21:22 -06:00
Markus Pilman dc65b7e1c0 fix gcc build error, rename tenantcycle workload 2022-04-25 14:49:48 -06:00
Markus Pilman bf2a723889 fix formatting 2022-04-25 09:03:12 -06:00
Markus Pilman ccf97eb187 Make transport work 2022-04-22 17:06:05 -06:00
Markus Pilman 1da1f8cc0f Fix memory issue 2022-04-21 14:12:16 -06:00
Markus Pilman b61c9ab329 Merge remote-tracking branch 'origin/main' into features/authz 2022-04-20 14:40:14 -06:00
Binglin Chang 408c0cf1c9
Fix compile errors on ubuntu 20.04 (#4931) 2022-04-20 10:00:46 -07:00
Markus Pilman 436d15ea98 Implemented token management and testing
Code is still untested at this point, but testing code was written

Implemented token management and testing

Code is still untested at this point, but testing code was written
2022-04-16 11:29:13 -06:00
Markus Pilman 577f0dca99 Make context variables type safe 2022-04-14 11:24:19 -06:00
Markus Pilman 420a901938 Added verification logic 2022-04-13 12:30:18 -06:00
Markus Pilman f9eca3884b implemented base logic 2022-04-12 12:02:43 -06:00
Markus Pilman bf956f5630 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-07 13:29:27 -06:00
Markus Pilman e2d7d4075d multiple bug fixes 2022-04-07 11:08:07 -06:00
Chaoguang Lin 7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
Markus Pilman b595d4462f Throw error on unauthorized access 2022-03-29 14:58:43 -06:00
Josh Slocum f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Markus Pilman 1fbeca8038 fix memory issue 2022-03-19 11:03:32 +01:00
Josh Slocum 37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
Evan Tschannen 7908ea54f3 fix: NetNotifiedQueueWithAcknowledgements could miss disconnects that happen during the delay(0) in deliver() 2022-03-17 09:57:46 -07:00
Markus Pilman 158da462f7 fix memory bug 2022-03-17 14:37:05 +01:00