d129b726e1
Don't use assertion for bad option usage; don't disable local client until setup network is called.
2021-11-12 10:12:49 -08:00
329091e14f
Merge branch 'master' into bg_bindings
2021-11-11 10:13:37 -06:00
77fa1e701a
cleanup
2021-11-11 09:43:23 -06:00
b8ac4213a1
Switched BG APIs to transaction instead of database
2021-11-11 08:59:06 -06:00
51b8ccf7d3
Merge remote-tracking branch 'apple/master' into notify-client-lib-changes
2021-11-10 18:40:34 +01:00
d6a5cc8460
MVC2.0: Rollback static initialization of the local client API instance
2021-11-10 15:55:33 +01:00
fdb3b72e35
Introduce GetRangeAndFlatMap to push computations down to FDB
...
Re-introduce #5609
2021-11-09 13:52:28 -08:00
31386c0f03
MVC2.0: Fix refactoring to remove dependency of MVC on Native API
2021-11-08 17:25:43 +01:00
586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB"
2021-11-04 08:46:56 -07:00
6c98e35893
Rename Hop to FlatMap
2021-11-03 13:32:01 -07:00
0853661d13
Introduce getRangeAndHop to push computations down to FDB
2021-11-03 13:21:16 -07:00
e3839e198a
Cleanup
2021-11-03 13:41:41 -05:00
276dbd4d37
MVC2.0: Refactoring - remove dependency of MVC on Native API
2021-11-03 16:30:18 +01:00
382882f1c1
mako successfully calls read_blob_granules and gets stuff back
2021-11-02 13:43:42 -05:00
d6a31078fe
C API for blob granules
2021-11-02 10:01:23 -05:00
befe1993c4
fix conflict on rebase
2021-10-29 12:25:26 -04:00
abab45760d
Add some additional logging if the network thread finishes, fails with an error, gets stopped, or is blocked.
2021-10-19 10:05:29 -07:00
f68d018a06
conflicts
2021-10-08 12:39:39 -04:00
a3d53c57b1
Fix a thread-safety issue with timeoutTsav
2021-09-29 11:07:47 -07:00
950de29d5a
Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
2021-09-20 21:29:57 +00:00
110d6f2d24
Enable support for integers in environment variable network options
...
Description
Description
Previously, the environment setting via shell cannot be extracted correctly, due to the code is't ablle to
determine input parameter type.
Now the logic to determien parameter and give it different processing is added.
Resolves #4127
Testing
1. Pass an invalid env var:
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=xyz - error
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=123xyz) - error
2. Pass an out-of-ranged env var:
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=9999999999999999999999999999 -error
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=-9999999999999999999999999999 -error
3. Pass a valid env var
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=1000 -ok
4. Pass a float env var(not support)
FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=3.1415926535 - error
2021-09-20 11:48:07 -07:00
3ece92b93d
Enable support for integers in environment variable network options
2021-09-20 11:48:07 -07:00
1e16ab446f
Enable support for integers in environment variable network options
2021-09-20 11:48:07 -07:00
c36758c08a
Enable support for integers in environment variable network options
2021-09-20 11:48:07 -07:00
72e97775be
Enable support for integers in environment variable network options
2021-09-20 11:48:07 -07:00
d78e53893d
Update for the env variable type check function
...
Description
Testing
2021-09-20 11:48:07 -07:00
19cfc12378
Enable multiple environment type input
...
Description
Previously, the environment setting via shell cannot be extracted correctly, due to the code is't ablle to
determine input parameter type.
Now the logic to determien parameter and give it different processing is added.
Testing
2021-09-20 11:48:07 -07:00
d51b93390d
Proper handling of reset and transaction destruction. Add some comments.
2021-09-17 18:10:56 -06:00
b5fbd07155
Transactions would not honor the transaction timeout option if the MVC did not have an active database.
2021-09-17 18:10:56 -06:00
2aa3b44d4e
Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
...
- Conflicts:
fdbserver/LogSystem.h
fdbserver/LogSystemConfig.h
fdbserver/TagPartitionedLogSystem.actor.cpp
- Files modified during merge:
modified: fdbserver/LogSystem.cpp
modified: fdbserver/LogSystemConfig.cpp
2021-09-17 19:36:18 +00:00
ffbdc21ca8
- Address review comments
2021-09-16 12:00:26 -04:00
852fc96200
Address simulation test failures caused by:
...
- Assertion failures in MoveKeys.actor.cpp
- Wrong results returned by getRange()
Changes:
DatabaseContext.h, NativeAPI.actor.[h,cpp]:
- Introduce a new flag, TransactionInfo::readVersionObtainedFromGrvProxy.
- Set this flag to true by default, and clear it when the read version of a
transaction is explicitly set (by using setVersion()).
- Modify getLatestCommitVersions() to not populate "latestCommitVersions" if
this flag is not set. (This will cause storage server to read at the specified
read version.)
- Modify getRange() actor to always use the specified version as the read
version (except when the specified version is latestVersion).
- Modify waitForCommittedVersion(), getRawVersion(), and getConsistentReadVersion()
to update local version vector cache after receiving GetReadVersionReply.
IClientApi.h, IConfigTransaction.h, ISingleThreadTransaction.h,
MultiVersionTransaction[.actor].[h,cpp], ThreadSafeTransaction.[h,cpp],
ApiWorkload.h:
- Add methods to get the spanID of a transaction and also the version vector
cached in a transaction. (Likely to be useful for debugging simulation test
failures.)
VersionVector.h:
- Update "maxVersion" when populating/applying a delta. (Note that empty
mutation messages only update VersionVector::maxVersion.)
BackupWorker.actor.cpp:
- Update local version vector cache after receiving GetReadVersionReply message.
Status.actor.cpp:
- Update local version vector cache and
TransactionInfo::info.readVersionObtainedFromGrvProxy after setting the
read version.
2021-09-16 12:00:26 -04:00
30e7469b19
Update fdbclient/MultiVersionTransaction.actor.cpp
2021-09-08 17:37:33 -07:00
af5643c998
Fix the issue in CheckStatus where it stuck as we cannot talk to any of the coordinators
2021-09-08 10:05:13 -07:00
def5822d42
When using an external client, use the larger of the primary and external busyness.
2021-08-27 10:46:34 -07:00
c74047c665
Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings
2021-07-28 11:51:02 -07:00
507c1f11e3
Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use.
2021-07-26 19:55:10 -07:00
7f727fd616
Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings
2021-07-24 23:18:21 -07:00
3442ebd3b7
Fix more -Wreorder-ctor warnings across many files
2021-07-24 11:20:51 -07:00
0e9dabcabb
Remove mutex that was only needed for a minor optimization.
2021-07-23 10:20:50 -07:00
052e32ae18
Update comment about failure to update version monitor DB
2021-07-23 15:58:29 +00:00
11b803fe0b
Handle database creation errors
2021-07-22 13:22:04 -07:00
bdb740b7b8
Propogate errors if creating a database on an external client fails
2021-07-21 15:59:23 -07:00
39af41ebb7
Closing a multi-version database causes us to cancel the protocol version monitor, invalidating its future. If a version change is triggered after that happens, then an assertion will be triggered that expects the future to be valid.
...
This changes the behavior so that closing the database prevents us from doing any work to update the version.
2021-07-19 11:01:09 -07:00
475abe301c
Merge remote-tracking branch 'origin/master' into fix-ub
2021-07-14 10:47:02 -07:00
8efe3b296a
Delete remaining extern declarations for noUnseed
2021-07-08 19:19:22 -04:00
fe5e7f8c2a
Remove ISingleThreadTransaction::create function
2021-06-29 10:29:41 -07:00
7f35a66383
Fix the types of dl functions in the multi-version client
...
This fixes undefined behavior of invoking a function pointer with the
wrong type.
2021-06-23 10:38:04 -07:00
f28ac955c3
Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>>
2021-05-10 16:32:50 -07:00
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
ab3f96f16d
Fix: simulation doesn't have a protocol version monitor and can't cancel it.
2021-04-30 11:56:35 -07:00
eaf1e0f64e
Be more defensive with cancellation by not capturing the this pointer in lambdas, instead capturing a full Reference.
2021-04-30 11:11:02 -07:00
cb3d2bfec7
Add cancellation of the protocol monitor when the database is destroyed. Avoid using any state when cancelled. Fix race between setting up the protocol version monitor and destroying the database.
2021-04-30 10:35:44 -07:00
f61f13a0ba
Explicitly cancel thread futures for the protocol version monitors in MVC
2021-04-29 22:02:38 -07:00
904a39e473
Merge pull request #4667 from sfc-gh-ajbeamon/feature-mvc-monitor-protocol-version
...
Use fewer connections in the multi-version client
2021-04-28 14:13:17 -07:00
16dfb2b2f2
Keep connections older than 6.2 open indefinitely to avoid weird bugs around quickly closing the database.
2021-04-27 15:00:56 -07:00
823873a9aa
Address review comments:
...
Use nullptr instead of NULL
Use const& for a parameter
Add some comments
2021-04-26 14:39:27 -07:00
656c9a6c47
Add benchmark and document entities touched
2021-04-26 17:46:35 +00:00
a794fca932
Support 5.0 (and earlier) client versions by adding GRV probing for old versions. Update the C bindings implementation of get_server_protocol to convert the ProtocolVersion object into a uint64_t. Rename a misleading protocol version alias.
2021-04-23 15:00:21 -07:00
6fc59379d8
Add /fdbclient/multiversionclient/ to ctest, and fix thread safety
2021-04-23 21:17:41 +00:00
f485d7fa5e
Fix comment typo
2021-04-21 12:25:03 -07:00
9e89159efb
Don't use DLDatabase objects before they are ready (applicable for API versions < 610). Fix reference counting of DLDatabase objects to avoid leaking the underlying database handle. Update release notes to note that clients older than 6.2 still create extra connections.
2021-04-20 16:21:01 -07:00
486260e944
Fix infinite loop with stable interface protocol monitoring. Fix case where getting an error with a network option didn't properly terminate the database connection. Reduce option lock critical section.
2021-04-15 13:36:31 -07:00
b2d6930103
The multi-version client monitors the cluster's protocol version and only activates the client library that can connect.
2021-04-15 11:45:14 -07:00
3ed0d614d2
Move fdb_get_server_protocol to be a function on the database object. Add an argument for expected_version that can be used to signal that the function shouldn't return unless the protocol version is different.
2021-04-14 12:50:30 -07:00
73571ad52e
Merge branch 'master' of github.com:sfc-gh-nwijetunga/foundationdb into network_busyness
...
* 'master' of github.com:sfc-gh-nwijetunga/foundationdb: (55 commits)
Port lost changes from #4004
Fix markdown
Factor our refreshTransaction gadget
Align FastAllocator memory to 4096 for size >= 4096
Fix heap-use-after-free's
Disable lsan in joshua
Manually apply changes made to BackupContainer.actor.cpp on release-6.3 to the new files where that code is located.
add release branch instructions
Fix PR number in release note link
fixed a simulation bug where a process on an unreliable machine would be considered reliable by the simulator
Add tini as PID 1 to docker image (#4363 )
Fix typo
Fix broken_promise bug
Explain FastAllocator<4096> special case
Inform LSAN that we have unaligned pointers
apply clang-format to flow/Platform.h
Fix typos
Address review comments
Change the macro that forbids exit() calls to be a static assertion
Add versionstamp serialization information
...
2021-03-18 18:58:43 +00:00
bf5f83d2bf
address pr comments
2021-03-16 21:49:36 +00:00
25c4880ebe
Merge branch 'release-6.3' into merge-release-6.3-into-master (temporarily discard all changes to BackupContainer.actor.cpp)
...
# Conflicts:
# fdbclient/BackupContainer.actor.cpp
# fdbserver/Knobs.h
2021-03-15 16:41:04 -07:00
f0d79b3d86
Inital implementation of network busyness
2021-03-15 23:23:56 +00:00
0dd9fdf9de
Throw an error if the local client is disabled and there are no external clients
2021-03-11 14:09:52 -08:00
fe4ff420a3
Don't copy the external libraries if you aren't using more than one thread
2021-03-10 15:09:30 -08:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
2bb4f2e59f
Merge branch 'release-6.3-pre-format' into master-format
...
This merges release-6.3 branch right before it was fully formatted.
There were quite a few conflicts that are resolved here. CoroFlow had
a check for OOM errors introduced in 6.3, but didn't seem applicable in
the new implmentation which seems to use boost.
2021-03-10 09:37:41 -08:00
8a8c488ede
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-05 18:13:38 -06:00
37d9e975e9
Fix multiple compiler warnings
2021-03-03 10:18:03 -07:00
346a4e3ecd
Merge branch 'release-6.3'
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/MultiInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2021-03-01 18:52:06 -08:00
1ee9c3e45d
Apply suggestions from code review
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2021-02-22 20:18:01 -07:00
e9b7e22e02
Merge branch 'rel-6.3'
2021-02-22 08:47:22 -08:00
5324f127c9
Address skipped PR comments from multithreaded client PR
2021-02-17 21:10:42 +00:00
bf3e6db9e9
Fix build issues; move to new assert style
2021-02-12 23:52:55 +00:00
36915e848a
clang format
2021-02-12 22:42:57 +00:00
555f3ebfaa
Implement + test corner case configurations for mutlithreaded client.
2021-02-12 22:37:24 +00:00
6e483b6da8
Remove sendfile.h to fix non-Linux builds
2021-02-12 09:31:22 -08:00
4938aee113
Address PR comments, remove some old cruft + comments
2021-02-11 01:27:57 +00:00
e4f6a39e48
change client thread count from knob to fdb.option (to be more consistent with other multiversion client options)
2021-02-10 16:29:27 +00:00
90686a95a0
format code
2021-02-08 22:17:16 -08:00
61082e70e1
format code, add comments
2021-02-08 19:11:30 -08:00
9e9bc4abe5
Create one client thread for each one that was requested (assumes local client is disabled)
2021-02-08 21:20:37 +00:00
97925de9d4
Update fdb_database_create_snapshot to take a UID as input
2021-02-05 14:14:38 -08:00
ad3de5c9e0
WIP: seems to work?
2021-02-03 23:19:26 +00:00
1032a5d005
WIP
2021-02-03 21:45:35 +00:00
f50206aabb
fdbclient: Copy external libraries MULTI_VERSION_CLIENT_THREADS_PER_VERSION times
...
To support multiple threads in for each multi-version clients, we make a copy of
external library for each thread in `/tmp/` directory and load that.
(cherry-picked by Rusty; there were some conflicts, so this has commented out cruft in it)
2021-02-03 20:47:15 +00:00
87aeafb294
partial refactor to support multiple threads per client library
2021-02-03 19:58:37 +00:00
cef53b2b40
clang-format
2021-01-26 12:06:40 -08:00
5f912f0deb
Add snapshot c function
2021-01-26 11:53:35 -08:00
6e619e9157
Update the returned future when local client is disabled
2021-01-25 00:41:50 -08:00
116f034933
update var names
2021-01-21 01:11:40 -08:00
9cd7c43a2c
Add c function fdb_database_force_recovery_with_data_loss
2021-01-21 00:42:23 -08:00
e097c98956
Remove headed added by IDE, add comments
2021-01-04 00:18:01 -08:00
da7b65cd71
Change the interface back to return int64_t
2021-01-03 22:45:09 -08:00
b68d84aaea
Merge remote-tracking branch 'upstream/master' into add-c-function-for-management-commands
2020-12-21 12:32:31 -08:00
8d59dae26d
Add check for existence of databaseRebootWorker in c api
2020-12-03 13:47:59 -08:00
88e66545bb
Change the api to return bool instead of int
2020-12-03 12:01:32 -08:00
c77d9e4abe
merge conflicts
2020-12-02 21:53:19 +00:00
2320f970c1
Add reboot_worker and pythong binding
2020-11-19 16:43:39 -08:00
aec0398fd0
Merge remote-tracking branch 'upstream/master' into add-c-function-for-management-commands
2020-11-17 11:39:12 -08:00
4669f837fa
Add uses of makeReference
2020-11-07 22:10:18 -08:00
a72bb52eae
Merge branch 'master' of github.com:apple/foundationdb into add-c-function-for-management-commands
2020-10-28 22:32:44 -07:00
309e38dc06
add unit test and some cleanup
2020-10-23 19:34:18 +00:00
90caa24af1
add function to get correct checksum class. Remove IChecksum::checkSum function
2020-10-23 02:24:29 +00:00
c84a5f4014
change IChecksum interface and remove coordinator protocol version optional
2020-10-19 21:39:07 +00:00
a4ccf8c5b6
add checksum class to change checksum in the future. expose get server protocol api
2020-10-19 16:57:11 +00:00
dc4612712d
Merge remote-tracking branch 'origin/master' into use-override-more
2020-10-09 07:12:41 -07:00
5d902f9177
Merge pull request #3394 from dongxinEric/feature/range-split-points-based-on-size
...
Feature/range split points based on size
2020-10-08 14:25:11 -07:00
a9607bdcec
Explicitly seal classes that inherit but aren't inherited from
2020-10-07 21:58:24 -07:00
a3de8c333b
Use override where applicable in fdbclient
2020-10-07 16:57:38 -07:00
f03a355c4e
Merge branch 'release-6.2' into release-6.3
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/downloads.rst
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/FlowTransport.h
# flow/Arena.h
# flow/Knobs.cpp
# flow/Knobs.h
# packaging/msi/FDBInstaller.wxs
2020-10-07 12:28:22 -07:00
76da0a7cbc
Backport #3426 to release-6.2
2020-10-05 13:53:51 -07:00
b4e209bd4e
Replace NULL with nullptr in fdbclient
2020-09-20 11:33:09 -07:00
0282a1745a
Address more review comments
2020-09-11 16:44:09 -07:00
1b923477f7
Apply suggestions from code review
...
Co-authored-by: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-09-11 14:28:56 -07:00
23ef23357c
Test code to add c function for kill/suspend
2020-08-21 14:30:52 -07:00
a46e620baa
Merge branch 'master' into feature/range-split-points-based-on-size
2020-08-06 15:23:11 -07:00
a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
e1dedff7b3
Merge branch 'release-6.2' into release-6.3
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# cmake/ConfigureCompiler.cmake
# documentation/sphinx/source/downloads.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/FileBackupAgent.actor.cpp
# packaging/msi/FDBInstaller.wxs
# versions.target
2020-07-24 12:10:44 -07:00
6446b4c082
WIP
2020-07-09 22:02:43 +00:00
2126f46195
Added client side support for range split
2020-07-02 14:05:11 -07:00
1082ab2caa
fix ClientVersionRef versionString should be passed by ref
2020-06-29 21:23:56 +02:00
6d570cd865
Change version from 7.0 to 6.3
2020-04-03 21:38:58 -07:00
d20ce99774
Resolved the review comment and renamed the functions
2020-02-12 14:57:40 -08:00
807204e676
Update fdbclient/MultiVersionTransaction.actor.cpp
...
Apply A.J's suggestion.
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-02-12 14:57:40 -08:00
d5c3f821e2
Added missing pieces.
2020-02-12 14:57:40 -08:00
70f89042fd
Remove comment that does not apply anymore
2020-02-12 14:57:40 -08:00
0c16d43c2f
Added necessary plumbings to expose byte sample collected by storage servers to fdb_c library
2020-02-12 14:57:40 -08:00
a51ab0003d
Don't require fdb_future_get_database from new client binaries. The previous code, which technically incorrect, is believed to be asymptomatic because new versions of the client do have this symbol in order to support old API header versions.
2019-09-24 13:11:30 -07:00
bc5c65e5ab
Merge pull request #1756 from jzhou77/db-option
...
Add transaction getApproximateSize() API
2019-07-19 08:33:24 -07:00
14cb21285f
Remove futureGetVersion in C binding and FutureVersion in Python binding
2019-07-16 10:46:07 -07:00
2dcc3cfd0a
Deprecate fdb_future_get_version for version 620
...
Use fdb_future_get_int64 in all bindings.
2019-07-11 21:17:31 -07:00
bbef631872
fix: do not access optionInfo unless the option already exists in the map
2019-07-10 18:48:54 -07:00
2c2836c6c7
Require API version to 620 for approximate size
2019-07-10 15:00:50 -07:00
5d1437c8e0
Push int directly to stack for getApproximateSize
2019-07-10 15:00:50 -07:00
0802df2c8f
Convert size from int to string before pushing onto stack
...
Using int is troublesome because the size of int can be different from the
desired 64 bits. So, using a string representation seems to be more consistent.
2019-07-10 14:58:35 -07:00
8ef8b59fcc
Use ThreadFuture for getApproximateSize
...
Change return type to int64_t and fix C and Python binding to use the correct
type.
2019-07-10 14:58:07 -07:00
c50a675bf0
Add transaction getApproximateSize() API
...
The size is the summation of expected size of mutations, read conflict ranges,
and write conflict ranges.
2019-07-10 14:51:52 -07:00
15ecba59db
Merge branch 'master' into fix-mvc-default-options
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-07-09 15:28:12 -07:00
6b6012ee7b
Add a break to setOption() switch statement. Better detection of missing options (and logging for present options).
2019-07-02 15:42:53 -07:00
7e5b5a0536
Apply suggestions from code review
...
Use emplace_back instead of push_back
Co-Authored-By: Jingyu Zhou <jingyuzhou@gmail.com>
2019-07-02 11:09:46 -07:00
aa1bc0087e
Address some review comments
2019-06-28 14:17:25 -07:00
2035b36257
Make default and persistent options specifyable via annotations to fdb.options. Fix some issues with persisting these options in the multi-version client. Make size limit option not persistent.
2019-06-28 13:24:32 -07:00
cf30c47a29
If onError fails with cluster_version_changed, retry the error on the new transaction.
2019-06-20 11:21:05 -07:00
A.J. Beamon