Commit Graph

247 Commits

Author SHA1 Message Date
Josh Slocum c11b8a3625 Improved comments 2021-12-01 17:04:55 -06:00
Josh Slocum a82845af43 Merge branch 'master' into bg_bindings 2021-12-01 16:55:28 -06:00
Josh Slocum 7f4fcc8c2c Added FDBResult and made readBlobGranules use it 2021-12-01 16:22:05 -06:00
Andrew Noyes bccda9f6fe Don't provide default for requireFunction in loadClientFunction
Recently there was an issue where we added a new symbol to the c api and
we did not include the right header version guard to see if it was
required. Let's make this not a default so that we force callers to
think about what the right header versions to require it are.
2021-11-29 14:08:28 -08:00
A.J. Beamon b8bd89f88d Shorten the name of external client threads. Add a thread name for trace logging threads. 2021-11-29 09:57:10 -08:00
A.J. Beamon 264c75b9a6 Add some extra client logging details:
1. Add a trace event when a database is created and move the cluster file / connection string from ClientStart to the new trace event
2. Add a detail for the path to the image being loaded
3. Add a detail for whether a client library is primary or not
4. Set a thread name for each external client thread that includes the release version
2021-11-29 09:57:10 -08:00
Andrew Noyes c6472790e3 Don't require fdb_transaction_get_range_and_flat_map in older clients 2021-11-22 10:49:26 -08:00
A.J. Beamon d129b726e1 Don't use assertion for bad option usage; don't disable local client until setup network is called. 2021-11-12 10:12:49 -08:00
Josh Slocum 329091e14f Merge branch 'master' into bg_bindings 2021-11-11 10:13:37 -06:00
Josh Slocum 77fa1e701a cleanup 2021-11-11 09:43:23 -06:00
Josh Slocum b8ac4213a1 Switched BG APIs to transaction instead of database 2021-11-11 08:59:06 -06:00
Vaidas Gasiunas 51b8ccf7d3 Merge remote-tracking branch 'apple/master' into notify-client-lib-changes 2021-11-10 18:40:34 +01:00
Vaidas Gasiunas d6a5cc8460 MVC2.0: Rollback static initialization of the local client API instance 2021-11-10 15:55:33 +01:00
Tao Lin fdb3b72e35 Introduce GetRangeAndFlatMap to push computations down to FDB
Re-introduce #5609
2021-11-09 13:52:28 -08:00
Vaidas Gasiunas 31386c0f03 MVC2.0: Fix refactoring to remove dependency of MVC on Native API 2021-11-08 17:25:43 +01:00
Tao Lin 586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB" 2021-11-04 08:46:56 -07:00
Tao Lin 6c98e35893 Rename Hop to FlatMap 2021-11-03 13:32:01 -07:00
Tao Lin 0853661d13 Introduce getRangeAndHop to push computations down to FDB 2021-11-03 13:21:16 -07:00
Josh Slocum e3839e198a Cleanup 2021-11-03 13:41:41 -05:00
Vaidas Gasiunas 276dbd4d37 MVC2.0: Refactoring - remove dependency of MVC on Native API 2021-11-03 16:30:18 +01:00
Josh Slocum 382882f1c1 mako successfully calls read_blob_granules and gets stuff back 2021-11-02 13:43:42 -05:00
Josh Slocum d6a31078fe C API for blob granules 2021-11-02 10:01:23 -05:00
Dan Lambright befe1993c4 fix conflict on rebase 2021-10-29 12:25:26 -04:00
A.J. Beamon abab45760d Add some additional logging if the network thread finishes, fails with an error, gets stopped, or is blocked. 2021-10-19 10:05:29 -07:00
Dan Lambright f68d018a06 conflicts 2021-10-08 12:39:39 -04:00
A.J. Beamon a3d53c57b1 Fix a thread-safety issue with timeoutTsav 2021-09-29 11:07:47 -07:00
Sreenath Bodagala 950de29d5a Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype 2021-09-20 21:29:57 +00:00
Chang Liu 110d6f2d24 Enable support for integers in environment variable network options
Description
Description
Previously, the environment setting via shell cannot be extracted correctly, due to the code is't ablle to
determine input parameter type.
Now the logic to determien parameter and give it different processing is added.

Resolves #4127

Testing
1. Pass an invalid env var:
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=xyz - error
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=123xyz) - error
2. Pass an out-of-ranged env var:
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=9999999999999999999999999999 -error
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=-9999999999999999999999999999 -error
3. Pass a valid env var
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=1000 -ok
4. Pass a float env var(not support)
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=3.1415926535 - error
2021-09-20 11:48:07 -07:00
Chang Liu 3ece92b93d Enable support for integers in environment variable network options 2021-09-20 11:48:07 -07:00
Chang Liu 1e16ab446f Enable support for integers in environment variable network options 2021-09-20 11:48:07 -07:00
Chang Liu c36758c08a Enable support for integers in environment variable network options 2021-09-20 11:48:07 -07:00
Chang Liu 72e97775be Enable support for integers in environment variable network options 2021-09-20 11:48:07 -07:00
Chang Liu d78e53893d Update for the env variable type check function
Description

Testing
2021-09-20 11:48:07 -07:00
Chang Liu 19cfc12378 Enable multiple environment type input
Description
Previously, the environment setting via shell cannot be extracted correctly, due to the code is't ablle to
determine input parameter type.
Now the logic to determien parameter and give it different processing is added.

Testing
2021-09-20 11:48:07 -07:00
A.J. Beamon d51b93390d Proper handling of reset and transaction destruction. Add some comments. 2021-09-17 18:10:56 -06:00
A.J. Beamon b5fbd07155 Transactions would not honor the transaction timeout option if the MVC did not have an active database. 2021-09-17 18:10:56 -06:00
Sreenath Bodagala 2aa3b44d4e Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
- Conflicts:
	fdbserver/LogSystem.h
	fdbserver/LogSystemConfig.h
	fdbserver/TagPartitionedLogSystem.actor.cpp

- Files modified during merge:

modified:   fdbserver/LogSystem.cpp
modified:   fdbserver/LogSystemConfig.cpp
2021-09-17 19:36:18 +00:00
Sreenath Bodagala ffbdc21ca8 - Address review comments 2021-09-16 12:00:26 -04:00
Sreenath Bodagala 852fc96200 Address simulation test failures caused by:
- Assertion failures in MoveKeys.actor.cpp
- Wrong results returned by getRange()

Changes:

DatabaseContext.h, NativeAPI.actor.[h,cpp]:
- Introduce a new flag, TransactionInfo::readVersionObtainedFromGrvProxy.
- Set this flag to true by default, and clear it when the read version of a
transaction is explicitly set (by using setVersion()).
- Modify getLatestCommitVersions() to not populate "latestCommitVersions" if
this flag is not set. (This will cause storage server to read at the specified
read version.)

- Modify getRange() actor to always use the specified version as the read
version (except when the specified version is latestVersion).

- Modify waitForCommittedVersion(), getRawVersion(), and getConsistentReadVersion()
to update local version vector cache after receiving GetReadVersionReply.

IClientApi.h, IConfigTransaction.h, ISingleThreadTransaction.h,
MultiVersionTransaction[.actor].[h,cpp], ThreadSafeTransaction.[h,cpp],
ApiWorkload.h:
- Add methods to get the spanID of a transaction and also the version vector
cached in a transaction. (Likely to be useful for debugging simulation test
failures.)

VersionVector.h:
- Update "maxVersion" when populating/applying a delta. (Note that empty
mutation messages only update VersionVector::maxVersion.)

BackupWorker.actor.cpp:
- Update local version vector cache after receiving GetReadVersionReply message.

Status.actor.cpp:
- Update local version vector cache and
TransactionInfo::info.readVersionObtainedFromGrvProxy after setting the
read version.
2021-09-16 12:00:26 -04:00
Lukas Joswiak 30e7469b19
Update fdbclient/MultiVersionTransaction.actor.cpp 2021-09-08 17:37:33 -07:00
Chaoguang Lin af5643c998 Fix the issue in CheckStatus where it stuck as we cannot talk to any of the coordinators 2021-09-08 10:05:13 -07:00
A.J. Beamon def5822d42 When using an external client, use the larger of the primary and external busyness. 2021-08-27 10:46:34 -07:00
sfc-gh-tclinkenbeard c74047c665 Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings 2021-07-28 11:51:02 -07:00
Steve Atherton 507c1f11e3 Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use. 2021-07-26 19:55:10 -07:00
sfc-gh-tclinkenbeard 7f727fd616 Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings 2021-07-24 23:18:21 -07:00
sfc-gh-tclinkenbeard 3442ebd3b7 Fix more -Wreorder-ctor warnings across many files 2021-07-24 11:20:51 -07:00
A.J. Beamon 0e9dabcabb Remove mutex that was only needed for a minor optimization. 2021-07-23 10:20:50 -07:00
A.J. Beamon 052e32ae18 Update comment about failure to update version monitor DB 2021-07-23 15:58:29 +00:00
A.J. Beamon 11b803fe0b Handle database creation errors 2021-07-22 13:22:04 -07:00
A.J. Beamon bdb740b7b8 Propogate errors if creating a database on an external client fails 2021-07-21 15:59:23 -07:00
A.J. Beamon 39af41ebb7 Closing a multi-version database causes us to cancel the protocol version monitor, invalidating its future. If a version change is triggered after that happens, then an assertion will be triggered that expects the future to be valid.
This changes the behavior so that closing the database prevents us from doing any work to update the version.
2021-07-19 11:01:09 -07:00
sfc-gh-tclinkenbeard 475abe301c Merge remote-tracking branch 'origin/master' into fix-ub 2021-07-14 10:47:02 -07:00
Daniel Smith 8efe3b296a Delete remaining extern declarations for noUnseed 2021-07-08 19:19:22 -04:00
sfc-gh-tclinkenbeard fe5e7f8c2a Remove ISingleThreadTransaction::create function 2021-06-29 10:29:41 -07:00
Andrew Noyes 7f35a66383 Fix the types of dl functions in the multi-version client
This fixes undefined behavior of invoking a function pointer with the
wrong type.
2021-06-23 10:38:04 -07:00
sfc-gh-tclinkenbeard f28ac955c3 Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>> 2021-05-10 16:32:50 -07:00
sfc-gh-tclinkenbeard 5c2d7b6080 Create RangeResult type alias 2021-05-03 13:14:16 -07:00
A.J. Beamon ab3f96f16d Fix: simulation doesn't have a protocol version monitor and can't cancel it. 2021-04-30 11:56:35 -07:00
A.J. Beamon eaf1e0f64e Be more defensive with cancellation by not capturing the this pointer in lambdas, instead capturing a full Reference. 2021-04-30 11:11:02 -07:00
A.J. Beamon cb3d2bfec7 Add cancellation of the protocol monitor when the database is destroyed. Avoid using any state when cancelled. Fix race between setting up the protocol version monitor and destroying the database. 2021-04-30 10:35:44 -07:00
A.J. Beamon f61f13a0ba Explicitly cancel thread futures for the protocol version monitors in MVC 2021-04-29 22:02:38 -07:00
Andrew Noyes 904a39e473
Merge pull request #4667 from sfc-gh-ajbeamon/feature-mvc-monitor-protocol-version
Use fewer connections in the multi-version client
2021-04-28 14:13:17 -07:00
A.J. Beamon 16dfb2b2f2 Keep connections older than 6.2 open indefinitely to avoid weird bugs around quickly closing the database. 2021-04-27 15:00:56 -07:00
A.J. Beamon 823873a9aa Address review comments:
Use nullptr instead of NULL
Use const& for a parameter
Add some comments
2021-04-26 14:39:27 -07:00
Andrew Noyes 656c9a6c47 Add benchmark and document entities touched 2021-04-26 17:46:35 +00:00
A.J. Beamon a794fca932 Support 5.0 (and earlier) client versions by adding GRV probing for old versions. Update the C bindings implementation of get_server_protocol to convert the ProtocolVersion object into a uint64_t. Rename a misleading protocol version alias. 2021-04-23 15:00:21 -07:00
Andrew Noyes 6fc59379d8 Add /fdbclient/multiversionclient/ to ctest, and fix thread safety 2021-04-23 21:17:41 +00:00
A.J. Beamon f485d7fa5e Fix comment typo 2021-04-21 12:25:03 -07:00
A.J. Beamon 9e89159efb Don't use DLDatabase objects before they are ready (applicable for API versions < 610). Fix reference counting of DLDatabase objects to avoid leaking the underlying database handle. Update release notes to note that clients older than 6.2 still create extra connections. 2021-04-20 16:21:01 -07:00
A.J. Beamon 486260e944 Fix infinite loop with stable interface protocol monitoring. Fix case where getting an error with a network option didn't properly terminate the database connection. Reduce option lock critical section. 2021-04-15 13:36:31 -07:00
A.J. Beamon b2d6930103 The multi-version client monitors the cluster's protocol version and only activates the client library that can connect. 2021-04-15 11:45:14 -07:00
A.J. Beamon 3ed0d614d2 Move fdb_get_server_protocol to be a function on the database object. Add an argument for expected_version that can be used to signal that the function shouldn't return unless the protocol version is different. 2021-04-14 12:50:30 -07:00
Nim Wijetunga 73571ad52e Merge branch 'master' of github.com:sfc-gh-nwijetunga/foundationdb into network_busyness
* 'master' of github.com:sfc-gh-nwijetunga/foundationdb: (55 commits)
  Port lost changes from #4004
  Fix markdown
  Factor our refreshTransaction gadget
  Align FastAllocator memory to 4096 for size >= 4096
  Fix heap-use-after-free's
  Disable lsan in joshua
  Manually apply changes made to BackupContainer.actor.cpp on release-6.3 to the new files where that code is located.
  add release branch instructions
  Fix PR number in release note link
  fixed a simulation bug where a process on an unreliable machine would be considered reliable by the simulator
  Add tini as PID 1 to docker image (#4363)
  Fix typo
  Fix broken_promise bug
  Explain FastAllocator<4096> special case
  Inform LSAN that we have unaligned pointers
  apply clang-format to flow/Platform.h
  Fix typos
  Address review comments
  Change the macro that forbids exit() calls to be a static assertion
  Add versionstamp serialization information
  ...
2021-03-18 18:58:43 +00:00
Nim Wijetunga bf5f83d2bf address pr comments 2021-03-16 21:49:36 +00:00
A.J. Beamon 25c4880ebe Merge branch 'release-6.3' into merge-release-6.3-into-master (temporarily discard all changes to BackupContainer.actor.cpp)
# Conflicts:
#	fdbclient/BackupContainer.actor.cpp
#	fdbserver/Knobs.h
2021-03-15 16:41:04 -07:00
Nim Wijetunga f0d79b3d86 Inital implementation of network busyness 2021-03-15 23:23:56 +00:00
A.J. Beamon 0dd9fdf9de Throw an error if the local client is disabled and there are no external clients 2021-03-11 14:09:52 -08:00
A.J. Beamon fe4ff420a3 Don't copy the external libraries if you aren't using more than one thread 2021-03-10 15:09:30 -08:00
FDB Formatster df90cc89de apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-10 10:18:07 -08:00
Vishesh Yadav 2bb4f2e59f Merge branch 'release-6.3-pre-format' into master-format
This merges release-6.3 branch right before it was fully formatted.
There were quite a few conflicts that are resolved here. CoroFlow had
a check for OOM errors introduced in 6.3, but didn't seem applicable in
the new implmentation which seems to use boost.
2021-03-10 09:37:41 -08:00
FDB Formatster 8a8c488ede apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-05 18:13:38 -06:00
Markus Pilman 37d9e975e9 Fix multiple compiler warnings 2021-03-03 10:18:03 -07:00
Evan Tschannen 346a4e3ecd Merge branch 'release-6.3'
# Conflicts:
#	fdbcli/fdbcli.actor.cpp
#	fdbrpc/LoadBalance.actor.h
#	fdbrpc/MultiInterface.h
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/masterserver.actor.cpp
2021-03-01 18:52:06 -08:00
Vishesh Yadav 1ee9c3e45d
Apply suggestions from code review
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2021-02-22 20:18:01 -07:00
negoyal e9b7e22e02 Merge branch 'rel-6.3' 2021-02-22 08:47:22 -08:00
Russell Sears 5324f127c9 Address skipped PR comments from multithreaded client PR 2021-02-17 21:10:42 +00:00
Russell Sears bf3e6db9e9 Fix build issues; move to new assert style 2021-02-12 23:52:55 +00:00
Russell Sears 36915e848a clang format 2021-02-12 22:42:57 +00:00
Russell Sears 555f3ebfaa Implement + test corner case configurations for mutlithreaded client. 2021-02-12 22:37:24 +00:00
Vishesh Yadav 6e483b6da8 Remove sendfile.h to fix non-Linux builds 2021-02-12 09:31:22 -08:00
Russell Sears 4938aee113 Address PR comments, remove some old cruft + comments 2021-02-11 01:27:57 +00:00
Russell Sears e4f6a39e48 change client thread count from knob to fdb.option (to be more consistent with other multiversion client options) 2021-02-10 16:29:27 +00:00
Chaoguang Lin 90686a95a0 format code 2021-02-08 22:17:16 -08:00
Chaoguang Lin 61082e70e1 format code, add comments 2021-02-08 19:11:30 -08:00
Russell Sears 9e9bc4abe5 Create one client thread for each one that was requested (assumes local client is disabled) 2021-02-08 21:20:37 +00:00
Chaoguang Lin 97925de9d4 Update fdb_database_create_snapshot to take a UID as input 2021-02-05 14:14:38 -08:00
Russell Sears ad3de5c9e0 WIP: seems to work? 2021-02-03 23:19:26 +00:00
Russell Sears 1032a5d005 WIP 2021-02-03 21:45:35 +00:00
Vishesh Yadav f50206aabb fdbclient: Copy external libraries MULTI_VERSION_CLIENT_THREADS_PER_VERSION times
To support multiple threads in for each multi-version clients, we make a copy of
external library for each thread in `/tmp/` directory and load that.

(cherry-picked by Rusty; there were some conflicts, so this has commented out cruft in it)
2021-02-03 20:47:15 +00:00
Russell Sears 87aeafb294 partial refactor to support multiple threads per client library 2021-02-03 19:58:37 +00:00