ab3f96f16d
Fix: simulation doesn't have a protocol version monitor and can't cancel it.
2021-04-30 11:56:35 -07:00
eaf1e0f64e
Be more defensive with cancellation by not capturing the this pointer in lambdas, instead capturing a full Reference.
2021-04-30 11:11:02 -07:00
cb3d2bfec7
Add cancellation of the protocol monitor when the database is destroyed. Avoid using any state when cancelled. Fix race between setting up the protocol version monitor and destroying the database.
2021-04-30 10:35:44 -07:00
f61f13a0ba
Explicitly cancel thread futures for the protocol version monitors in MVC
2021-04-29 22:02:38 -07:00
904a39e473
Merge pull request #4667 from sfc-gh-ajbeamon/feature-mvc-monitor-protocol-version
...
Use fewer connections in the multi-version client
2021-04-28 14:13:17 -07:00
16dfb2b2f2
Keep connections older than 6.2 open indefinitely to avoid weird bugs around quickly closing the database.
2021-04-27 15:00:56 -07:00
823873a9aa
Address review comments:
...
Use nullptr instead of NULL
Use const& for a parameter
Add some comments
2021-04-26 14:39:27 -07:00
656c9a6c47
Add benchmark and document entities touched
2021-04-26 17:46:35 +00:00
a794fca932
Support 5.0 (and earlier) client versions by adding GRV probing for old versions. Update the C bindings implementation of get_server_protocol to convert the ProtocolVersion object into a uint64_t. Rename a misleading protocol version alias.
2021-04-23 15:00:21 -07:00
6fc59379d8
Add /fdbclient/multiversionclient/ to ctest, and fix thread safety
2021-04-23 21:17:41 +00:00
f485d7fa5e
Fix comment typo
2021-04-21 12:25:03 -07:00
9e89159efb
Don't use DLDatabase objects before they are ready (applicable for API versions < 610). Fix reference counting of DLDatabase objects to avoid leaking the underlying database handle. Update release notes to note that clients older than 6.2 still create extra connections.
2021-04-20 16:21:01 -07:00
486260e944
Fix infinite loop with stable interface protocol monitoring. Fix case where getting an error with a network option didn't properly terminate the database connection. Reduce option lock critical section.
2021-04-15 13:36:31 -07:00
b2d6930103
The multi-version client monitors the cluster's protocol version and only activates the client library that can connect.
2021-04-15 11:45:14 -07:00
3ed0d614d2
Move fdb_get_server_protocol to be a function on the database object. Add an argument for expected_version that can be used to signal that the function shouldn't return unless the protocol version is different.
2021-04-14 12:50:30 -07:00
73571ad52e
Merge branch 'master' of github.com:sfc-gh-nwijetunga/foundationdb into network_busyness
...
* 'master' of github.com:sfc-gh-nwijetunga/foundationdb: (55 commits)
Port lost changes from #4004
Fix markdown
Factor our refreshTransaction gadget
Align FastAllocator memory to 4096 for size >= 4096
Fix heap-use-after-free's
Disable lsan in joshua
Manually apply changes made to BackupContainer.actor.cpp on release-6.3 to the new files where that code is located.
add release branch instructions
Fix PR number in release note link
fixed a simulation bug where a process on an unreliable machine would be considered reliable by the simulator
Add tini as PID 1 to docker image (#4363 )
Fix typo
Fix broken_promise bug
Explain FastAllocator<4096> special case
Inform LSAN that we have unaligned pointers
apply clang-format to flow/Platform.h
Fix typos
Address review comments
Change the macro that forbids exit() calls to be a static assertion
Add versionstamp serialization information
...
2021-03-18 18:58:43 +00:00
bf5f83d2bf
address pr comments
2021-03-16 21:49:36 +00:00
25c4880ebe
Merge branch 'release-6.3' into merge-release-6.3-into-master (temporarily discard all changes to BackupContainer.actor.cpp)
...
# Conflicts:
# fdbclient/BackupContainer.actor.cpp
# fdbserver/Knobs.h
2021-03-15 16:41:04 -07:00
f0d79b3d86
Inital implementation of network busyness
2021-03-15 23:23:56 +00:00
0dd9fdf9de
Throw an error if the local client is disabled and there are no external clients
2021-03-11 14:09:52 -08:00
fe4ff420a3
Don't copy the external libraries if you aren't using more than one thread
2021-03-10 15:09:30 -08:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
2bb4f2e59f
Merge branch 'release-6.3-pre-format' into master-format
...
This merges release-6.3 branch right before it was fully formatted.
There were quite a few conflicts that are resolved here. CoroFlow had
a check for OOM errors introduced in 6.3, but didn't seem applicable in
the new implmentation which seems to use boost.
2021-03-10 09:37:41 -08:00
8a8c488ede
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-05 18:13:38 -06:00
37d9e975e9
Fix multiple compiler warnings
2021-03-03 10:18:03 -07:00
346a4e3ecd
Merge branch 'release-6.3'
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/MultiInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2021-03-01 18:52:06 -08:00
1ee9c3e45d
Apply suggestions from code review
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2021-02-22 20:18:01 -07:00
e9b7e22e02
Merge branch 'rel-6.3'
2021-02-22 08:47:22 -08:00
5324f127c9
Address skipped PR comments from multithreaded client PR
2021-02-17 21:10:42 +00:00
bf3e6db9e9
Fix build issues; move to new assert style
2021-02-12 23:52:55 +00:00
36915e848a
clang format
2021-02-12 22:42:57 +00:00
555f3ebfaa
Implement + test corner case configurations for mutlithreaded client.
2021-02-12 22:37:24 +00:00
6e483b6da8
Remove sendfile.h to fix non-Linux builds
2021-02-12 09:31:22 -08:00
4938aee113
Address PR comments, remove some old cruft + comments
2021-02-11 01:27:57 +00:00
e4f6a39e48
change client thread count from knob to fdb.option (to be more consistent with other multiversion client options)
2021-02-10 16:29:27 +00:00
90686a95a0
format code
2021-02-08 22:17:16 -08:00
61082e70e1
format code, add comments
2021-02-08 19:11:30 -08:00
9e9bc4abe5
Create one client thread for each one that was requested (assumes local client is disabled)
2021-02-08 21:20:37 +00:00
97925de9d4
Update fdb_database_create_snapshot to take a UID as input
2021-02-05 14:14:38 -08:00
ad3de5c9e0
WIP: seems to work?
2021-02-03 23:19:26 +00:00
1032a5d005
WIP
2021-02-03 21:45:35 +00:00
f50206aabb
fdbclient: Copy external libraries MULTI_VERSION_CLIENT_THREADS_PER_VERSION times
...
To support multiple threads in for each multi-version clients, we make a copy of
external library for each thread in `/tmp/` directory and load that.
(cherry-picked by Rusty; there were some conflicts, so this has commented out cruft in it)
2021-02-03 20:47:15 +00:00
87aeafb294
partial refactor to support multiple threads per client library
2021-02-03 19:58:37 +00:00
cef53b2b40
clang-format
2021-01-26 12:06:40 -08:00
5f912f0deb
Add snapshot c function
2021-01-26 11:53:35 -08:00
6e619e9157
Update the returned future when local client is disabled
2021-01-25 00:41:50 -08:00
116f034933
update var names
2021-01-21 01:11:40 -08:00
9cd7c43a2c
Add c function fdb_database_force_recovery_with_data_loss
2021-01-21 00:42:23 -08:00
e097c98956
Remove headed added by IDE, add comments
2021-01-04 00:18:01 -08:00
da7b65cd71
Change the interface back to return int64_t
2021-01-03 22:45:09 -08:00
b68d84aaea
Merge remote-tracking branch 'upstream/master' into add-c-function-for-management-commands
2020-12-21 12:32:31 -08:00
8d59dae26d
Add check for existence of databaseRebootWorker in c api
2020-12-03 13:47:59 -08:00
88e66545bb
Change the api to return bool instead of int
2020-12-03 12:01:32 -08:00
c77d9e4abe
merge conflicts
2020-12-02 21:53:19 +00:00
2320f970c1
Add reboot_worker and pythong binding
2020-11-19 16:43:39 -08:00
aec0398fd0
Merge remote-tracking branch 'upstream/master' into add-c-function-for-management-commands
2020-11-17 11:39:12 -08:00
4669f837fa
Add uses of makeReference
2020-11-07 22:10:18 -08:00
a72bb52eae
Merge branch 'master' of github.com:apple/foundationdb into add-c-function-for-management-commands
2020-10-28 22:32:44 -07:00
309e38dc06
add unit test and some cleanup
2020-10-23 19:34:18 +00:00
90caa24af1
add function to get correct checksum class. Remove IChecksum::checkSum function
2020-10-23 02:24:29 +00:00
c84a5f4014
change IChecksum interface and remove coordinator protocol version optional
2020-10-19 21:39:07 +00:00
a4ccf8c5b6
add checksum class to change checksum in the future. expose get server protocol api
2020-10-19 16:57:11 +00:00
dc4612712d
Merge remote-tracking branch 'origin/master' into use-override-more
2020-10-09 07:12:41 -07:00
5d902f9177
Merge pull request #3394 from dongxinEric/feature/range-split-points-based-on-size
...
Feature/range split points based on size
2020-10-08 14:25:11 -07:00
a9607bdcec
Explicitly seal classes that inherit but aren't inherited from
2020-10-07 21:58:24 -07:00
a3de8c333b
Use override where applicable in fdbclient
2020-10-07 16:57:38 -07:00
f03a355c4e
Merge branch 'release-6.2' into release-6.3
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/downloads.rst
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/FlowTransport.h
# flow/Arena.h
# flow/Knobs.cpp
# flow/Knobs.h
# packaging/msi/FDBInstaller.wxs
2020-10-07 12:28:22 -07:00
76da0a7cbc
Backport #3426 to release-6.2
2020-10-05 13:53:51 -07:00
b4e209bd4e
Replace NULL with nullptr in fdbclient
2020-09-20 11:33:09 -07:00
0282a1745a
Address more review comments
2020-09-11 16:44:09 -07:00
1b923477f7
Apply suggestions from code review
...
Co-authored-by: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-09-11 14:28:56 -07:00
23ef23357c
Test code to add c function for kill/suspend
2020-08-21 14:30:52 -07:00
a46e620baa
Merge branch 'master' into feature/range-split-points-based-on-size
2020-08-06 15:23:11 -07:00
a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
e1dedff7b3
Merge branch 'release-6.2' into release-6.3
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# cmake/ConfigureCompiler.cmake
# documentation/sphinx/source/downloads.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/FileBackupAgent.actor.cpp
# packaging/msi/FDBInstaller.wxs
# versions.target
2020-07-24 12:10:44 -07:00
6446b4c082
WIP
2020-07-09 22:02:43 +00:00
2126f46195
Added client side support for range split
2020-07-02 14:05:11 -07:00
1082ab2caa
fix ClientVersionRef versionString should be passed by ref
2020-06-29 21:23:56 +02:00
6d570cd865
Change version from 7.0 to 6.3
2020-04-03 21:38:58 -07:00
d20ce99774
Resolved the review comment and renamed the functions
2020-02-12 14:57:40 -08:00
807204e676
Update fdbclient/MultiVersionTransaction.actor.cpp
...
Apply A.J's suggestion.
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-02-12 14:57:40 -08:00
d5c3f821e2
Added missing pieces.
2020-02-12 14:57:40 -08:00
70f89042fd
Remove comment that does not apply anymore
2020-02-12 14:57:40 -08:00
0c16d43c2f
Added necessary plumbings to expose byte sample collected by storage servers to fdb_c library
2020-02-12 14:57:40 -08:00
a51ab0003d
Don't require fdb_future_get_database from new client binaries. The previous code, which technically incorrect, is believed to be asymptomatic because new versions of the client do have this symbol in order to support old API header versions.
2019-09-24 13:11:30 -07:00
bc5c65e5ab
Merge pull request #1756 from jzhou77/db-option
...
Add transaction getApproximateSize() API
2019-07-19 08:33:24 -07:00
14cb21285f
Remove futureGetVersion in C binding and FutureVersion in Python binding
2019-07-16 10:46:07 -07:00
2dcc3cfd0a
Deprecate fdb_future_get_version for version 620
...
Use fdb_future_get_int64 in all bindings.
2019-07-11 21:17:31 -07:00
bbef631872
fix: do not access optionInfo unless the option already exists in the map
2019-07-10 18:48:54 -07:00
2c2836c6c7
Require API version to 620 for approximate size
2019-07-10 15:00:50 -07:00
5d1437c8e0
Push int directly to stack for getApproximateSize
2019-07-10 15:00:50 -07:00
0802df2c8f
Convert size from int to string before pushing onto stack
...
Using int is troublesome because the size of int can be different from the
desired 64 bits. So, using a string representation seems to be more consistent.
2019-07-10 14:58:35 -07:00
8ef8b59fcc
Use ThreadFuture for getApproximateSize
...
Change return type to int64_t and fix C and Python binding to use the correct
type.
2019-07-10 14:58:07 -07:00
c50a675bf0
Add transaction getApproximateSize() API
...
The size is the summation of expected size of mutations, read conflict ranges,
and write conflict ranges.
2019-07-10 14:51:52 -07:00
15ecba59db
Merge branch 'master' into fix-mvc-default-options
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2019-07-09 15:28:12 -07:00
6b6012ee7b
Add a break to setOption() switch statement. Better detection of missing options (and logging for present options).
2019-07-02 15:42:53 -07:00
7e5b5a0536
Apply suggestions from code review
...
Use emplace_back instead of push_back
Co-Authored-By: Jingyu Zhou <jingyuzhou@gmail.com>
2019-07-02 11:09:46 -07:00
aa1bc0087e
Address some review comments
2019-06-28 14:17:25 -07:00
2035b36257
Make default and persistent options specifyable via annotations to fdb.options. Fix some issues with persisting these options in the multi-version client. Make size limit option not persistent.
2019-06-28 13:24:32 -07:00
cf30c47a29
If onError fails with cluster_version_changed, retry the error on the new transaction.
2019-06-20 11:21:05 -07:00
8576665a90
Revert "Revert "Make protocol version a type""
...
This reverts commit 455bf3b3ec
2019-06-18 14:49:04 -07:00
455bf3b3ec
Revert "Make protocol version a type"
2019-06-18 10:59:17 -07:00
da53a92bec
Make protocol version a type
...
This fixes #1214
The basic idea is that ProtocolVersion is now its own type. This
alone is an improvement as it makes many things more typesafe. For
each version, we can now add breaking features (for example Fearless).
After that, there's no need to test against actual (confusing) version
numbers. Instead a developer can simply test
`protocolVersion->hasFearless()` and this will return true iff the
protocolVersion is newer than the newest version that didn't support
fearless.
2019-06-16 09:59:15 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
6207d724f8
Fix all -Wunused-variable warnings
2019-04-15 18:13:00 -07:00
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
85b3f11e71
Fix various compiler warnings
2019-03-15 10:34:57 -07:00
04b94e74c7
Merge pull request #942 from ajbeamon/remove-cluster-from-bindings
...
Remove cluster from bindings
2019-01-24 14:39:09 -08:00
65aa5105c7
Fix: The multi-version client would block when trying to connect to an external cluster.
2019-01-14 10:53:03 -08:00
d4d5740282
* Add Optional.map and ErrorOr.map.
...
* Rename Optional/ErrorOr cast_to to castTo.
* Make printable(Optional<T>) templated rather than restricted to StringRef types.
* Fixes bug in (unused) ErrorOr.castTo where an ErrorOr that was not set would lose its error.
2019-01-11 09:03:38 -08:00
890a0f3e82
Merge branch 'master' into remove-cluster-from-bindings
...
# Conflicts:
# bindings/flow/fdb_flow.actor.cpp
2018-11-26 12:01:03 -08:00
a654183f63
Merge pull request #791 from ajbeamon/remove-cluster-from-iclientapi
...
Remove cluster from IClientApi (phase 2 of removing DB names)
2018-11-10 10:16:18 -08:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
22f8a4efa9
Normalized all unit test names to begin with "/" if they should be included in random unit testing.
2018-10-05 22:09:58 -07:00
1b259dbc9d
Initial work to remove clusters from our C API
2018-09-26 10:27:55 -07:00
3eb4355a48
Some various cleanup and fixes. Added "Cluster" to TransactionMetrics trace event.
2018-09-25 15:06:19 -07:00
6ae50f7016
Change cluster objects to be a char* for the cluster file path. Remove IDatabase::createDatabase. DLApi::createDatabase deletes the temporary cluster it creates.
2018-09-25 11:01:09 -07:00
c831051474
This removes the idea of clusters from IClientApi.
2018-09-21 15:58:14 -07:00
d42f0e9c44
Merge pull request #618 from ajbeamon/log-options
...
Log network, cluster, and database options when they are set.
2018-09-05 10:36:40 -07:00
4eaff42e4f
Merge pull request #712 from ajbeamon/remove-database-name-internal
...
Eliminate use of database names (phase 1)
2018-09-05 10:35:00 -07:00
e1fafbf259
Add some missing actorcompiler.h includes
2018-08-22 09:40:45 -07:00
2a97139d5d
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-16 10:24:12 -07:00
3c03ddf91e
Remove value logging for options. Add some additional logging for setting an external client library and setting knobs.
2018-08-15 09:42:49 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
a2a2064a50
Don't log a SevWarnAlways event for external_client_already_loaded errors.
2018-08-03 14:26:04 -07:00
1c29275672
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-01 14:30:57 -07:00
227d36cdc4
Log network, cluster, and database options when they are set. Throw an exception if the option being set doesn't exist.
2018-07-17 16:23:16 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
c5d0a04b9e
Move external completion hook handling to DLApi. Use camel case for hookParameter.
2018-05-09 12:28:51 -07:00
01a3360982
Move addNetworkThreadCompletionHook to IClientApi.
2018-05-08 16:33:43 -07:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
84f7565d04
MultiVersionApi::createCluster was throwing network_not_setup rather than returning it.
2018-01-31 10:03:09 -08:00
d886b95628
Merge pull request #131 from cie/33300740-with-shutdown-hooks
...
<rdar://problem/33300740> Java: support callbacks from external multi-version client threads
2017-10-04 09:17:25 -07:00
d30c730f75
Add the ability to access name and description in Error. Update error descriptions.
2017-09-28 12:35:03 -07:00
80f559d148
changed name from thread_completion_hook to network_thread_completion_hook
2017-09-27 11:30:39 -07:00
d7e1b267be
changed name from shutdown hook to thread completion hook ; added hook parameter
2017-09-26 17:00:04 -07:00
59aae5e994
added catch all for client shutdown hooks
2017-07-26 15:39:03 -07:00
83bf2ee312
added add_shutdown_hook to fdb_c api and used it to detach java threads where appropriate
2017-07-25 15:57:26 -07:00
93509133ad
Attempt to eliminate a race in DLSingleAssignmentVar between cancelling a future and checking its ref count. Reduce the amount of iterations in the test because it’s timing out on the pie workers.
2017-05-26 10:47:37 -07:00
a674cb4ef4
Initial repository commit
2017-05-25 13:48:44 -07:00
A.J. Beamon