e2c7c30faf
GetMappedRange support serializable & check RYW & continuation ( #6181 )
2022-03-10 10:05:44 -08:00
250a88e682
Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement.
2022-02-24 12:25:52 -08:00
72e06369a4
Merge remote-tracking branch 'apple-upstream/main' into version-vector-prototype
2022-02-08 17:47:57 +00:00
57a073b3b9
Add thread names for release, cancel, destroy and abort.
2022-01-24 19:42:21 -06:00
9544379cdf
rebase
2022-01-20 11:12:33 -05:00
3afe9fb6e0
MVC bug fixes
2021-12-10 12:47:53 -06:00
b7b2ad0a6f
Handling timeout and transaction changes for ThreadResults properly
2021-12-10 12:47:53 -06:00
c11b8a3625
Improved comments
2021-12-01 17:04:55 -06:00
a82845af43
Merge branch 'master' into bg_bindings
2021-12-01 16:55:28 -06:00
7f4fcc8c2c
Added FDBResult and made readBlobGranules use it
2021-12-01 16:22:05 -06:00
bccda9f6fe
Don't provide default for requireFunction in loadClientFunction
...
Recently there was an issue where we added a new symbol to the c api and
we did not include the right header version guard to see if it was
required. Let's make this not a default so that we force callers to
think about what the right header versions to require it are.
2021-11-29 14:08:28 -08:00
b8bd89f88d
Shorten the name of external client threads. Add a thread name for trace logging threads.
2021-11-29 09:57:10 -08:00
264c75b9a6
Add some extra client logging details:
...
1. Add a trace event when a database is created and move the cluster file / connection string from ClientStart to the new trace event
2. Add a detail for the path to the image being loaded
3. Add a detail for whether a client library is primary or not
4. Set a thread name for each external client thread that includes the release version
2021-11-29 09:57:10 -08:00
c6472790e3
Don't require fdb_transaction_get_range_and_flat_map in older clients
2021-11-22 10:49:26 -08:00
d129b726e1
Don't use assertion for bad option usage; don't disable local client until setup network is called.
2021-11-12 10:12:49 -08:00
329091e14f
Merge branch 'master' into bg_bindings
2021-11-11 10:13:37 -06:00
77fa1e701a
cleanup
2021-11-11 09:43:23 -06:00
b8ac4213a1
Switched BG APIs to transaction instead of database
2021-11-11 08:59:06 -06:00
51b8ccf7d3
Merge remote-tracking branch 'apple/master' into notify-client-lib-changes
2021-11-10 18:40:34 +01:00
d6a5cc8460
MVC2.0: Rollback static initialization of the local client API instance
2021-11-10 15:55:33 +01:00
fdb3b72e35
Introduce GetRangeAndFlatMap to push computations down to FDB
...
Re-introduce #5609
2021-11-09 13:52:28 -08:00
31386c0f03
MVC2.0: Fix refactoring to remove dependency of MVC on Native API
2021-11-08 17:25:43 +01:00
586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB"
2021-11-04 08:46:56 -07:00
6c98e35893
Rename Hop to FlatMap
2021-11-03 13:32:01 -07:00
0853661d13
Introduce getRangeAndHop to push computations down to FDB
2021-11-03 13:21:16 -07:00
e3839e198a
Cleanup
2021-11-03 13:41:41 -05:00
276dbd4d37
MVC2.0: Refactoring - remove dependency of MVC on Native API
2021-11-03 16:30:18 +01:00
382882f1c1
mako successfully calls read_blob_granules and gets stuff back
2021-11-02 13:43:42 -05:00
d6a31078fe
C API for blob granules
2021-11-02 10:01:23 -05:00
befe1993c4
fix conflict on rebase
2021-10-29 12:25:26 -04:00
abab45760d
Add some additional logging if the network thread finishes, fails with an error, gets stopped, or is blocked.
2021-10-19 10:05:29 -07:00
f68d018a06
conflicts
2021-10-08 12:39:39 -04:00
a3d53c57b1
Fix a thread-safety issue with timeoutTsav
2021-09-29 11:07:47 -07:00
950de29d5a
Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
2021-09-20 21:29:57 +00:00
110d6f2d24
Enable support for integers in environment variable network options
...
Description
Description
Previously, the environment setting via shell cannot be extracted correctly, due to the code is't ablle to
determine input parameter type.
Now the logic to determien parameter and give it different processing is added.
Resolves #4127
Testing
1. Pass an invalid env var:
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=xyz - error
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=123xyz) - error
2. Pass an out-of-ranged env var:
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=9999999999999999999999999999 -error
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=-9999999999999999999999999999 -error
3. Pass a valid env var
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=1000 -ok
4. Pass a float env var(not support)
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=3.1415926535 - error
2021-09-20 11:48:07 -07:00
3ece92b93d
Enable support for integers in environment variable network options
2021-09-20 11:48:07 -07:00
1e16ab446f
Enable support for integers in environment variable network options
2021-09-20 11:48:07 -07:00
c36758c08a
Enable support for integers in environment variable network options
2021-09-20 11:48:07 -07:00
72e97775be
Enable support for integers in environment variable network options
2021-09-20 11:48:07 -07:00
d78e53893d
Update for the env variable type check function
...
Description
Testing
2021-09-20 11:48:07 -07:00
19cfc12378
Enable multiple environment type input
...
Description
Previously, the environment setting via shell cannot be extracted correctly, due to the code is't ablle to
determine input parameter type.
Now the logic to determien parameter and give it different processing is added.
Testing
2021-09-20 11:48:07 -07:00
d51b93390d
Proper handling of reset and transaction destruction. Add some comments.
2021-09-17 18:10:56 -06:00
b5fbd07155
Transactions would not honor the transaction timeout option if the MVC did not have an active database.
2021-09-17 18:10:56 -06:00
2aa3b44d4e
Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
...
- Conflicts:
fdbserver/LogSystem.h
fdbserver/LogSystemConfig.h
fdbserver/TagPartitionedLogSystem.actor.cpp
- Files modified during merge:
modified: fdbserver/LogSystem.cpp
modified: fdbserver/LogSystemConfig.cpp
2021-09-17 19:36:18 +00:00
ffbdc21ca8
- Address review comments
2021-09-16 12:00:26 -04:00
852fc96200
Address simulation test failures caused by:
...
- Assertion failures in MoveKeys.actor.cpp
- Wrong results returned by getRange()
Changes:
DatabaseContext.h, NativeAPI.actor.[h,cpp]:
- Introduce a new flag, TransactionInfo::readVersionObtainedFromGrvProxy.
- Set this flag to true by default, and clear it when the read version of a
transaction is explicitly set (by using setVersion()).
- Modify getLatestCommitVersions() to not populate "latestCommitVersions" if
this flag is not set. (This will cause storage server to read at the specified
read version.)
- Modify getRange() actor to always use the specified version as the read
version (except when the specified version is latestVersion).
- Modify waitForCommittedVersion(), getRawVersion(), and getConsistentReadVersion()
to update local version vector cache after receiving GetReadVersionReply.
IClientApi.h, IConfigTransaction.h, ISingleThreadTransaction.h,
MultiVersionTransaction[.actor].[h,cpp], ThreadSafeTransaction.[h,cpp],
ApiWorkload.h:
- Add methods to get the spanID of a transaction and also the version vector
cached in a transaction. (Likely to be useful for debugging simulation test
failures.)
VersionVector.h:
- Update "maxVersion" when populating/applying a delta. (Note that empty
mutation messages only update VersionVector::maxVersion.)
BackupWorker.actor.cpp:
- Update local version vector cache after receiving GetReadVersionReply message.
Status.actor.cpp:
- Update local version vector cache and
TransactionInfo::info.readVersionObtainedFromGrvProxy after setting the
read version.
2021-09-16 12:00:26 -04:00
30e7469b19
Update fdbclient/MultiVersionTransaction.actor.cpp
2021-09-08 17:37:33 -07:00
af5643c998
Fix the issue in CheckStatus where it stuck as we cannot talk to any of the coordinators
2021-09-08 10:05:13 -07:00
def5822d42
When using an external client, use the larger of the primary and external busyness.
2021-08-27 10:46:34 -07:00
c74047c665
Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings
2021-07-28 11:51:02 -07:00
507c1f11e3
Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use.
2021-07-26 19:55:10 -07:00
7f727fd616
Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings
2021-07-24 23:18:21 -07:00
3442ebd3b7
Fix more -Wreorder-ctor warnings across many files
2021-07-24 11:20:51 -07:00
0e9dabcabb
Remove mutex that was only needed for a minor optimization.
2021-07-23 10:20:50 -07:00
052e32ae18
Update comment about failure to update version monitor DB
2021-07-23 15:58:29 +00:00
11b803fe0b
Handle database creation errors
2021-07-22 13:22:04 -07:00
bdb740b7b8
Propogate errors if creating a database on an external client fails
2021-07-21 15:59:23 -07:00
39af41ebb7
Closing a multi-version database causes us to cancel the protocol version monitor, invalidating its future. If a version change is triggered after that happens, then an assertion will be triggered that expects the future to be valid.
...
This changes the behavior so that closing the database prevents us from doing any work to update the version.
2021-07-19 11:01:09 -07:00
475abe301c
Merge remote-tracking branch 'origin/master' into fix-ub
2021-07-14 10:47:02 -07:00
8efe3b296a
Delete remaining extern declarations for noUnseed
2021-07-08 19:19:22 -04:00
fe5e7f8c2a
Remove ISingleThreadTransaction::create function
2021-06-29 10:29:41 -07:00
7f35a66383
Fix the types of dl functions in the multi-version client
...
This fixes undefined behavior of invoking a function pointer with the
wrong type.
2021-06-23 10:38:04 -07:00
f28ac955c3
Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>>
2021-05-10 16:32:50 -07:00
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
ab3f96f16d
Fix: simulation doesn't have a protocol version monitor and can't cancel it.
2021-04-30 11:56:35 -07:00
eaf1e0f64e
Be more defensive with cancellation by not capturing the this pointer in lambdas, instead capturing a full Reference.
2021-04-30 11:11:02 -07:00
cb3d2bfec7
Add cancellation of the protocol monitor when the database is destroyed. Avoid using any state when cancelled. Fix race between setting up the protocol version monitor and destroying the database.
2021-04-30 10:35:44 -07:00
f61f13a0ba
Explicitly cancel thread futures for the protocol version monitors in MVC
2021-04-29 22:02:38 -07:00
904a39e473
Merge pull request #4667 from sfc-gh-ajbeamon/feature-mvc-monitor-protocol-version
...
Use fewer connections in the multi-version client
2021-04-28 14:13:17 -07:00
16dfb2b2f2
Keep connections older than 6.2 open indefinitely to avoid weird bugs around quickly closing the database.
2021-04-27 15:00:56 -07:00
823873a9aa
Address review comments:
...
Use nullptr instead of NULL
Use const& for a parameter
Add some comments
2021-04-26 14:39:27 -07:00
656c9a6c47
Add benchmark and document entities touched
2021-04-26 17:46:35 +00:00
a794fca932
Support 5.0 (and earlier) client versions by adding GRV probing for old versions. Update the C bindings implementation of get_server_protocol to convert the ProtocolVersion object into a uint64_t. Rename a misleading protocol version alias.
2021-04-23 15:00:21 -07:00
6fc59379d8
Add /fdbclient/multiversionclient/ to ctest, and fix thread safety
2021-04-23 21:17:41 +00:00
f485d7fa5e
Fix comment typo
2021-04-21 12:25:03 -07:00
9e89159efb
Don't use DLDatabase objects before they are ready (applicable for API versions < 610). Fix reference counting of DLDatabase objects to avoid leaking the underlying database handle. Update release notes to note that clients older than 6.2 still create extra connections.
2021-04-20 16:21:01 -07:00
486260e944
Fix infinite loop with stable interface protocol monitoring. Fix case where getting an error with a network option didn't properly terminate the database connection. Reduce option lock critical section.
2021-04-15 13:36:31 -07:00
b2d6930103
The multi-version client monitors the cluster's protocol version and only activates the client library that can connect.
2021-04-15 11:45:14 -07:00
3ed0d614d2
Move fdb_get_server_protocol to be a function on the database object. Add an argument for expected_version that can be used to signal that the function shouldn't return unless the protocol version is different.
2021-04-14 12:50:30 -07:00
73571ad52e
Merge branch 'master' of github.com:sfc-gh-nwijetunga/foundationdb into network_busyness
...
* 'master' of github.com:sfc-gh-nwijetunga/foundationdb: (55 commits)
Port lost changes from #4004
Fix markdown
Factor our refreshTransaction gadget
Align FastAllocator memory to 4096 for size >= 4096
Fix heap-use-after-free's
Disable lsan in joshua
Manually apply changes made to BackupContainer.actor.cpp on release-6.3 to the new files where that code is located.
add release branch instructions
Fix PR number in release note link
fixed a simulation bug where a process on an unreliable machine would be considered reliable by the simulator
Add tini as PID 1 to docker image (#4363 )
Fix typo
Fix broken_promise bug
Explain FastAllocator<4096> special case
Inform LSAN that we have unaligned pointers
apply clang-format to flow/Platform.h
Fix typos
Address review comments
Change the macro that forbids exit() calls to be a static assertion
Add versionstamp serialization information
...
2021-03-18 18:58:43 +00:00
bf5f83d2bf
address pr comments
2021-03-16 21:49:36 +00:00
25c4880ebe
Merge branch 'release-6.3' into merge-release-6.3-into-master (temporarily discard all changes to BackupContainer.actor.cpp)
...
# Conflicts:
# fdbclient/BackupContainer.actor.cpp
# fdbserver/Knobs.h
2021-03-15 16:41:04 -07:00
f0d79b3d86
Inital implementation of network busyness
2021-03-15 23:23:56 +00:00
0dd9fdf9de
Throw an error if the local client is disabled and there are no external clients
2021-03-11 14:09:52 -08:00
fe4ff420a3
Don't copy the external libraries if you aren't using more than one thread
2021-03-10 15:09:30 -08:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
2bb4f2e59f
Merge branch 'release-6.3-pre-format' into master-format
...
This merges release-6.3 branch right before it was fully formatted.
There were quite a few conflicts that are resolved here. CoroFlow had
a check for OOM errors introduced in 6.3, but didn't seem applicable in
the new implmentation which seems to use boost.
2021-03-10 09:37:41 -08:00
8a8c488ede
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-05 18:13:38 -06:00
37d9e975e9
Fix multiple compiler warnings
2021-03-03 10:18:03 -07:00
346a4e3ecd
Merge branch 'release-6.3'
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/MultiInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2021-03-01 18:52:06 -08:00
1ee9c3e45d
Apply suggestions from code review
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2021-02-22 20:18:01 -07:00
e9b7e22e02
Merge branch 'rel-6.3'
2021-02-22 08:47:22 -08:00
5324f127c9
Address skipped PR comments from multithreaded client PR
2021-02-17 21:10:42 +00:00
bf3e6db9e9
Fix build issues; move to new assert style
2021-02-12 23:52:55 +00:00
36915e848a
clang format
2021-02-12 22:42:57 +00:00
555f3ebfaa
Implement + test corner case configurations for mutlithreaded client.
2021-02-12 22:37:24 +00:00
6e483b6da8
Remove sendfile.h to fix non-Linux builds
2021-02-12 09:31:22 -08:00
4938aee113
Address PR comments, remove some old cruft + comments
2021-02-11 01:27:57 +00:00
e4f6a39e48
change client thread count from knob to fdb.option (to be more consistent with other multiversion client options)
2021-02-10 16:29:27 +00:00
90686a95a0
format code
2021-02-08 22:17:16 -08:00
Tao Lin