1519f24f77
Merge branch 'main' into feature-metacluster
2022-07-07 09:35:40 -07:00
c4b0f6eaae
Add an internal C API to support connection to a cluster using a connection string ( #7438 )
...
* Add an internal C API to support memory connection records
* Track shared state in the client using a unique and immutable cluster ID from the cluster
* Add missing code to store the clusterId in the database state object
* Update some arguments to pass by const&
2022-07-07 10:12:49 +02:00
56541167c2
send token with tenant info object
2022-07-06 20:58:17 -06:00
c9b553663e
Merge branch 'main' into feature-metacluster
...
# Conflicts:
# bindings/java/src/main/com/apple/foundationdb/TenantManagement.java
# fdbcli/TenantCommands.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/TenantSpecialKeys.actor.cpp
# fdbclient/include/fdbclient/KeyBackedTypes.h
# fdbclient/include/fdbclient/RunTransaction.actor.h
# fdbclient/include/fdbclient/SpecialKeySpace.actor.h
# fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-06 15:44:21 -07:00
1e8225a19e
Merge pull request #7511 from sfc-gh-ajbeamon/add-subrange-support-to-tenant-special-keys
...
Update the tenant special keys submodule to support multiple sub-ranges
2022-07-06 10:33:14 -07:00
2eae95aabe
Merge branch 'main' into feature-metacluster
2022-07-01 14:07:34 -07:00
037c06a688
Merge remote-tracking branch 'apple-upstream/main'
2022-07-01 04:51:17 +00:00
aea4d802c6
Merge branch 'main' into feature-metacluster
2022-06-30 15:08:09 -07:00
2f67328a0c
Update the tenant special keys submodule to support multiple sub-ranges. This will enable future work that allows configuring tenants at the same time as creating them.
2022-06-30 15:03:37 -07:00
48bcf6eaba
parameterizing min split size instead of hardcoding it to min_shard_bytes ( #7404 )
2022-06-30 12:28:22 -07:00
c74710f6d5
- Address a review comment
...
- Optiize out the code in DatabaseContext::getLatestCommitVersions()
in the case where the feature is turned off.
2022-06-29 10:13:08 +00:00
e1a93988ef
Merge branch 'main' into feature-metacluster
2022-06-28 14:58:07 -07:00
9d16260822
Merge remote-tracking branch 'origin/main' into features/authz
2022-06-28 13:13:51 -06:00
086e4bff06
Merge remote-tracking branch 'origin/main' into global-tag-throttling3
2022-06-28 10:18:13 -07:00
0da13a7797
- Optimize out the version vector specific code (on the client)
...
when the version vector feature is disabled.
2022-06-28 09:17:53 +00:00
4452e53342
Fix a problem caused by missing state.
2022-06-27 23:52:54 -04:00
62d4ac550a
Merge branch 'main' of https://github.com/apple/foundationdb into feature/ddtxn
2022-06-27 11:35:22 -07:00
3aaae9c521
Merge remote-tracking branch 'origin/main' into features/authz
2022-06-27 11:07:14 -06:00
8af056e7b0
fdbclient now compiling
2022-06-23 18:05:36 -06:00
ffaf15c12a
moved wellknownendpoints and fixed some includes
2022-06-23 17:03:53 -06:00
a5054b2beb
move getServerListAndProcessClasses to NativeAPI
2022-06-23 15:28:45 -07:00
d35445a868
enforce include modularization in cmake
2022-06-23 14:37:35 -06:00
9f3819752f
Change the command to create a metacluster from using 'configure tenant_mode=management' to 'metacluster create <NAME>'. Distribute this name to all processes in a metacluster. Eliminate the tenant mode entirely from metacluster clusters, instead relying on a metacluster registration key.
2022-06-22 12:15:43 -07:00
2391e58fb2
Merge remote-tracking branch 'origin/main' into global-tag-throttling3
2022-06-21 10:09:15 -07:00
bcd8785767
Remove old debug tracing
2022-06-20 00:59:22 -07:00
34e6a8f942
Merge pull request #7399 from sfc-gh-jslocum/bg_tenant_improvements
...
Bg tenant improvements
2022-06-17 11:19:41 -05:00
1cc466e068
fixes and review comments
2022-06-17 08:17:44 -05:00
b3597ef3a8
Added plumbing for tenant-aware purge granules
2022-06-16 13:04:34 -05:00
13d8b13722
Migrated Authz code to use JWT
2022-06-13 18:20:27 -06:00
799fe32346
Merge remote-tracking branch 'origin/main' into features/authz
2022-06-13 18:02:11 -06:00
207e0bc105
Fix a few places we weren't doing exponential backoff ( #7349 )
...
* Fix a few places we weren't doing exponential backoff
We re-create the transaction every iteration of each of these retry
loops, so we need to manage exponential backoff here ourselves.
Closes #7301
* Remove former Backoff definition
2022-06-13 13:18:58 -07:00
df71a49bf6
Merge remote-tracking branch 'origin/main' into global-tag-throttling3
2022-06-13 10:03:10 -07:00
75c858eb2c
Differentiate between different quotas in GlobalTagThrottling workload
2022-06-13 10:01:55 -07:00
710612442c
small fixes
2022-06-09 17:21:21 -05:00
d6920cde28
Implemented blob granule merging
2022-06-09 10:50:53 -05:00
a3289f9cab
adding tenant prefix to bg ranges call
2022-06-06 14:09:10 -05:00
2907aafa60
Merge branch 'main' into feature-metacluster
2022-06-02 12:18:42 -07:00
8cc51064a5
Add NAPI: prefix to VerifyCausalReadRisky span. ( #7285 )
2022-06-01 11:47:28 -05:00
29ebcdbfa9
Merge remote-tracking branch 'origin/main' into global-tag-throttling3
2022-05-31 14:53:51 -07:00
4d6ec7663f
Close file at the end of runNetwork rather than in stopNetwork in order to avoid a race condition,
...
also in order to continue tracing the pending network thread activity.
Poll event throttler only in the network thread in order to avoid a race condition.
2022-05-31 13:59:09 +02:00
4c07064632
Fix reference cycle in GlobalConfig
2022-05-27 10:25:29 -07:00
cd2a575e02
Move the resolve of coordinator hostname from getCoordinatorProtocol() to getClusterProtocolImpl(). ( #7245 )
...
* Move the resolve of coordinator hostname from getCoordinatorProtocol() to getClusterProtocolImpl().
* Guard DNS Cache behind a knob.
2022-05-26 09:45:54 -07:00
458e8018e1
review comments
2022-05-25 17:16:56 -05:00
fd7b662767
cleaning up
2022-05-25 17:16:56 -05:00
dd637dd6a7
passes tenant simulation test
2022-05-25 17:16:56 -05:00
e3a570d50d
Blob worker tenant integration mostly works
2022-05-25 17:16:56 -05:00
4015b5b2a0
Several Blob Granule bug fixes found in nightlies ( #7170 )
...
* Fixing simulation validation assert that was tripping incorrectly
* Commenting out debugging prints
* Fixed multiple error propagation issues in blob worker
2022-05-23 10:07:16 -07:00
16cf087667
Allow fetching invalid tenant id for getKeyLocation
2022-05-23 10:00:27 -06:00
d784173f7f
Add first implementation of tenant creation and deletion in a metacluster
2022-05-20 15:21:21 -07:00
5e8c3130e6
Extend the reboot interface to accept comma-delimited string;
...
Only fetch requested worker interfaces in rebootWorkerActor
Add killall test to multiprocess_fdbcli_tests
Make the rebootWorker only sends requests if all the requested workers are valid
Add comments; Organize print texts; Fix the bug clear the map every time run kill/suspend/expensive_data_check
2022-05-20 12:06:33 -07:00
61490022b4
Enabled checkpoint, restore test for RocksDB format.
2022-05-19 13:27:21 -07:00
69261f9f10
Merge branch 'main' into feature-metacluster
...
# Conflicts:
# fdbclient/MultiVersionTransaction.actor.cpp
# fdbclient/MultiVersionTransaction.h
# fdbserver/tester.actor.cpp
2022-05-19 09:11:30 -07:00
993a79b42c
Restart protocol monitoring logic when we cycle the coordinator, even if its the same coordinator. Add some defensive code in case we fetch the protocol version for an absent peer. ( #6397 )
2022-05-19 10:00:23 +02:00
f00606fd71
Enable MATCHED and UNMATCHED mode for index prefetch ( #7162 )
...
* Enable MATCHED and UNMATCHED mode for index prefetch
MATCHED mode returns index entries whose secondary KVs are present,
UNMATCHED mode returns index entries whose secondary KVs are absent.
Note that the conflict read range of this txn is set in 2 steps:
* Set the conflict range for primary query according to request
* Set the conflict ranges for secondary queries according to responses.
As a result, conflicts of different match_index mode are taken care of.
* Fix c binding
2022-05-18 09:16:28 -07:00
e8e26c9f7c
Remove the internal special key-space mechanism (at least for now), instead relying on the local logic to implement the metacluster functions. Add ability to force remove a cluster. Improve idempotency of register operation.
2022-05-18 08:18:25 -07:00
853e6a346b
Optimization: support removing index conditionally ( #7116 )
2022-05-13 10:10:43 -07:00
58c818985d
Apply clang-format to several files
2022-05-10 11:21:45 -07:00
051bd3102d
Revert bad merge change
2022-05-09 14:54:51 -07:00
f88cbf9309
Address review comments
2022-05-09 14:54:51 -07:00
f14baf2af8
clang-format changes
2022-05-09 14:54:51 -07:00
9173e2e19b
Move GlobalConfig to DatabaseContext
2022-05-09 14:54:51 -07:00
7578d5ebc7
Create GlobalConfig object for each database instance
...
Currently, GlobalConfig is a singleton that means for each process there is only
one GlobalConfig object. This is bug from clients perspective as a client can
keep connections to several databases. This patch tracks GlobalConfig for each
database using an unordered_map in flowGlobals.
We discovered this bug while testing multi-version client, where the client got
stuck. This was lucky, as normally it'd just write down config to the wrong
database.
2022-05-09 14:54:51 -07:00
d6ac3e57a4
Add internal metacluster special key-space modules
2022-05-07 20:14:03 -07:00
c9a5b6d789
Avoid re-adding transaction tags in ReadWriteWorkload
2022-05-07 15:58:05 -07:00
154ae7559e
- If getRange() is called with "latestVersion" as the read version
...
then both GetKeyValuesFamilyRequest and getRangeFallback should do
reads at the same version.
2022-05-06 18:10:37 +00:00
7c3b99f5bd
Add support for connection to clusters via a connection string. Start work on multi-cluster operations with data cluster registration.
2022-05-06 09:53:53 -07:00
0c41225d76
Add support for setting and storing the tenant group for a tenant
2022-05-05 08:57:01 -07:00
2102ed1eaa
- Remove "stale_version_vector" error code.
2022-05-03 21:56:11 +00:00
7e860dc92c
Some initial metacluster implementation
2022-05-03 12:53:35 -07:00
484558250b
- Do not consult version vector if the client has obtained the read
...
version from its GRV cache.
2022-05-03 19:23:37 +00:00
dc9e782ccc
OpenTelemetry Tracing Perf Fixes ( #6990 )
2022-05-02 14:56:51 -05:00
c69a07a858
Check in the new Hostname logic. ( #6926 )
...
* Revert #6655 .
20220407-031010-renxuan-c101052c21da8346 compressed=True data_size=31004844 duration=4310801 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=1:04:15 sanity=False started=100047 stopped=20220407-041425 submitted=20220407-031010 timeout=5400 username=renxuan
* Revert #6271 .
20220407-051532-renxuan-470f0fe6aac1c217 compressed=True data_size=30982370 duration=3491067 ended=100002 fail_fast=10 max_runs=100000 pass=100002 priority=100 remaining=0 runtime=0:59:57 sanity=False started=100141 stopped=20220407-061529 submitted=20220407-051532 timeout=5400 username=renxuan
* Revert #6266 .
Remove resolving-related functionalities in connection string. Connection string will be used for storing purpose only, and non-mutable.
20220407-175119-renxuan-55d30ee1a4b42c2f compressed=True data_size=30970443 duration=5437659 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:59:31 sanity=False started=100154 stopped=20220407-185050 submitted=20220407-175119 timeout=5400 username=renxuan
* Add hostname to coordinator interfaces.
* Turn on the new hostname logic.
* Add the corresponding change in config txns.
The most notable change is before calling basicLoadBalance(), we need to call tryInitializeRequestStream() to initialize request streams first.
Passed correctness tests.
* Return error when hostnames cannot be resolved in coordinators command.
* Minor fixes.
2022-04-27 21:54:13 -07:00
79063d5fc4
Better accounting for tenant prefixes when working with key size limits ( #6973 )
...
* Consider tenant prefix sizes when doing key size limit checks
* Fix formatting
2022-04-27 14:38:59 -07:00
cbe4a873d2
Merge remote-tracking branch 'origin/main' into features/validate-trace-events-in-simulation
2022-04-25 17:39:29 -06:00
21336fbf12
Merge remote-tracking branch 'origin/main' into features/authz
2022-04-25 08:45:23 -06:00
1c5bf135d5
Revert "Migrate to OpenTelemetry tracing. ( #6855 )" ( #6941 )
...
This reverts commit 5df3bac110
2022-04-25 09:29:56 -05:00
9e65e15b45
Merge remote-tracking branch 'origin/main' into features/validate-trace-events-in-simulation
2022-04-22 15:39:55 -06:00
1352083d4c
Merge pull request #6884 from sfc-gh-clin/deprecate-speical-keys
...
Remove the client profiling special keys and update related documentations
2022-04-21 21:51:16 -07:00
879eb98dec
Rename registerSpecialKeySpaceModule to registerSpecialKeysImpl
2022-04-21 12:54:34 -07:00
b61c9ab329
Merge remote-tracking branch 'origin/main' into features/authz
2022-04-20 14:40:14 -06:00
3335b2686e
Merge branch 'main' into features/validate-trace-events-in-simulation
2022-04-20 12:03:33 -06:00
5df3bac110
Migrate to OpenTelemetry tracing. ( #6855 )
2022-04-20 09:26:37 -05:00
d4ee7be1d7
Reduce excessive tracing and fail after 1M traces
2022-04-19 21:11:51 -06:00
6bfa4f6ac4
Fix typos; Polish documentations according to comments
2022-04-19 13:57:11 -07:00
36c876b769
Fix a typo; Fix a format error in special-keys.rst
2022-04-18 18:10:17 -07:00
c0264a8522
Remove the client profiling special keys and update related documentations
2022-04-18 17:54:50 -07:00
58c24fdaf2
Fix compilation problems
2022-04-13 13:15:41 -06:00
f9eca3884b
implemented base logic
2022-04-12 12:02:43 -06:00
16467262f0
Merge remote-tracking branch 'origin/main' into features/private-request-streams
2022-04-10 14:12:37 -06:00
e43fde16ec
formatting
2022-04-08 17:28:16 -04:00
1b3b4166c6
Merge branch 'main' into vv
2022-04-08 17:18:13 -04:00
6276cebad9
Blob integration ( #6808 )
...
* Fixing leaked stream with explicit notify failed before destructor
* better logic to prevent races in change feed fetching
* Found new race that makes assert incorrect
* handle server overloaded in initial read from fdb
* Handling more blob error types in granule retry
* Fixing rollback metadata problem, added better debugging
* Fixing version race when fetching change feed metadata
* Better racing split request handling
* fixing assert
* Handle change feed popped check in the blob worker
* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768 )
* more merge conflict issues
* Change feed destroy fixes
* Fixing change feed destroy and move race
* Check error condition in BG file req
* Using relative endpoints for blob worker interface
* Fixing bug in previous fix
* More destroy and move race fixes
* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read
* Bug fix (#6796 )
* fix: do not use a RYW transaction for a versionstamp because of randomize API version
* fix: if the initialSnapshotVersion was pruned, granule history was incorrect
* added a way to compress null bytes in printable()
* Fixing durability issue with moving and destroying change feeds
* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot
* More destroy and move races
* Fixing change feed destroy and pop races
* Renaming bg prune to purge, and adding a C api and unit test for it
* more cleanup
* review comments
* Observability for granule purging
* better handling for change feed not registered
* Fixed purging bugs (#6815 )
* fix: do not use a RYW transaction for a versionstamp because of randomize API version
* fix: if the initialSnapshotVersion was pruned, granule history was incorrect
* added a way to compress null bytes in printable()
* fixed a few purging bugs
Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
73a7c32982
Add fdbcli command to read/write version epoch ( #6480 )
...
* Initialize cluster version at wall-clock time
Previously, new clusters would begin at version 0. After this change,
clusters will initialize at a version matching wall-clock time. Instead
of using the Unix epoch (or Windows epoch), FDB clusters will use a new
epoch, defaulting to January 1, 2010, 01:00:00+00:00. In the future,
this base epoch will be modifiable through fdbcli, allowing
administrators to advance the cluster version.
Basing the version off of time allows different FDB clusters to share
data without running into version issues.
* Send version epoch to master
* Cleanup
* Update fdbserver/storageserver.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
* Jump directly to expected version if possible
* Fix initial version issue on storage servers
* Add random recovery offset to start version in simulation
* Type fixes
* Disable reference time by default
Enable on a cluster using the fdbcli command `versionepoch add 0`.
* Use correct recoveryTransactionVersion when recovering
* Allow version epoch to be adjusted forwards (to decrease the version)
* Set version epoch in simulation
* Add quiet database check to ensure small version offset
* Fix initial version issue on storage servers
* Disable reference time by default
Enable on a cluster using the fdbcli command `versionepoch add 0`.
* Add fdbcli command to read/write version epoch
* Cause recovery when version epoch is set
* Handle optional version epoch key
* Add ability to clear the version epoch
This causes version advancement to revert to the old methodology whereas
versions attempt to advance by about a million versions per second,
instead of trying to match the clock.
* Update transaction access
* Modify version epoch to use microseconds instead of seconds
* Modify fdbcli version target API
Move commands from `versionepoch` to `targetversion` top level command.
* Add fdbcli tests for
* Temporarily disable targetversion cli tests
* Fix version epoch fetch issue
* Fix Arena issue
* Reduce max version jump in simulation to 1,000,000
* Rework fdbcli API
It now requires two commands to fully switch a cluster to using the
version epoch. First, enable the version epoch with `versionepoch
enable` or `versionepoch set <versionepoch>`. At this point, versions
will be given out at a faster or slower rate in an attempt to reach the
expected version. Then, run `versionepoch commit` to perform a one time
jump to the expected version. This is essentially irreversible.
* Temporarily disable old targetversion tests
* Cleanup
* Move version epoch buggify to sequencer
This will cause some issues with the QuietDatabase check for the version
offset - namely, it won't do anything, since the version epoch is not
being written to the txnStateStore in simulation. This will get fixed in
the future.
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-04-08 12:33:19 -07:00
e2c144ba8e
fix conflict NativeAPI.actor.cpp
2022-04-08 14:35:20 -04:00
5bdc525353
Merge branch 'main' into vv
2022-04-08 13:16:04 -04:00
bf956f5630
Merge remote-tracking branch 'origin/main' into features/private-request-streams
2022-04-07 13:29:27 -06:00
124aa6e5fd
clean up debug traces and fix details with delRef placement
2022-04-06 12:27:39 -04:00
1b9bd597c5
address code review comments and add some traces for manual testing
2022-04-05 15:29:02 -04:00
a1be7abdad
Clear VV cache instead of throw
2022-04-01 15:19:24 -07:00
4fd414a8ed
Merge branch 'main' into vv
...
Fix Conflicts:
fdbclient/NativeAPI.actor.cpp
2022-03-31 09:38:36 -07:00
cfcf0f152c
Merge branch 'main-4a085fc84' into vv
...
Fix Conflicts:
fdbclient/NativeAPI.actor.cpp
fdbserver/ClusterRecovery.actor.cpp
fdbserver/MasterInterface.h
fdbserver/masterserver.actor.cpp
flow/error_definitions.h
2022-03-30 22:28:06 -07:00
b34f4052cd
Merge branch 'main-f28dfc12b' into vv
...
Fix Conflicts:
fdbclient/MultiVersionTransaction.actor.cpp
fdbclient/NativeAPI.actor.cpp
fdbclient/NativeAPI.actor.h
fdbserver/storageserver.actor.cpp
2022-03-30 21:01:25 -07:00
00b57d4cce
Merge branch 'main-67eba5ec7' into vv
...
Fix Conflicts:
fdbclient/CommitProxyInterface.h
fdbclient/NativeAPI.actor.cpp
fdbclient/StorageServerInterface.h
fdbserver/CommitProxyServer.actor.cpp
fdbserver/storageserver.actor.cpp
2022-03-30 20:05:55 -07:00
0cc1b67b7a
Temporary fix for stale version vector assertion
...
Need to investigate why and fix later.
2022-03-30 19:34:58 -07:00
e9659b5dd4
Merge branch 'master-PR-6500' into vv
...
Fix Conflicts:
fdbclient/CommitProxyInterface.h
fdbclient/NativeAPI.actor.cpp
fdbserver/masterserver.actor.cpp
2022-03-30 14:53:49 -07:00
f266695a76
clear shared state map entries on version upgrade and use currentProtocolVersion instead of piping version through function parameters
2022-03-30 15:28:55 -04:00
9d0a520fe0
add ProtocolVersion to shared state and state cluster map
2022-03-30 13:28:06 -04:00
53d252fda0
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded
2022-03-29 17:23:08 -04:00
bdbe78e48f
address code review comments
2022-03-29 17:20:22 -04:00
7fc6dfa6c5
Adding useful debugging trace events
2022-03-29 14:48:28 -05:00
fd5734c39e
Fix the case where we access the memory after cleaning
2022-03-28 13:35:42 -07:00
0ff1cd102e
Merge branch 'main' into bg_client_bindings_improvements
2022-03-24 17:19:01 -05:00
989dd8d7eb
Finished implementing beginVersion
2022-03-24 17:15:11 -05:00
b92c6ea92c
granule parallelism, metrics, and other improvements to bg client bindings
2022-03-24 13:44:09 -05:00
7af4b24966
move trace event to within if-block
2022-03-23 17:09:28 -04:00
9f72d68f1e
Make trace event SevDebug to avoid spamming logs by default
2022-03-23 17:08:17 -04:00
7c3151142e
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded
2022-03-23 16:41:33 -04:00
4466e00716
Fix multithreaded GRV cache code and enable grv caching in multithreaded ctest suite
2022-03-23 16:09:12 -04:00
f27475e2f4
Merge branch 'main' into blob_integration
2022-03-22 11:41:58 -05:00
a71099471b
Update copyright header dates
2022-03-21 13:36:23 -07:00
35f7843d84
Merge remote-tracking branch 'origin/main' into features/private-request-streams
2022-03-21 12:33:53 +01:00
8205771e8f
ReplyPromiseStreams need synchronous disconnect to avoid change feed races
2022-03-18 17:05:12 -05:00
37e7c80f26
Merge branch 'main' into blob_integration
2022-03-17 18:45:42 -05:00
513e8887dd
Add a new C unit test that uses tenants. Fix simulation issue where resetting a transaction could cause in-flight operations to pick up the simulated default tenant for the next retry. Use the new list tenant JSON format in the tenant management test. Fix missing raw_access option in special key space correctness test.
2022-03-17 12:10:19 -07:00
4079b9f23d
Lazily assign the default tenant in simulated tests so that we can enforce that raw access isn't used on tenant transactions. Update various tests to avoid using raw access or system key access after they have performed other operations on the transaction.
2022-03-17 12:10:19 -07:00
d0dc756c6d
Allow disabling tenant mode in simulation. Fix a few bugs.
2022-03-17 12:10:18 -07:00
05495908b8
Implement some tenant tests
2022-03-17 12:10:18 -07:00
118b53b7cf
Merge remote-tracking branch 'origin/main' into features/private-request-streams
2022-03-17 12:06:44 +01:00
58de6e22cc
Add BalanceOnRequests boolean parameter for ModelInterface
2022-03-16 14:25:32 -07:00
9731d96797
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded
2022-03-16 14:32:33 -04:00
7ab205391b
Revert "remove all multi-version client code related to grv cache"
...
This reverts commit 6d05ecffc2
2022-03-16 14:26:36 -04:00
2acc222d9a
Detect stale GRV proxy replies
...
Each GetReadVersionReply now has the GRV proxy ID that sent it so that the
client can detect old GRV proxy's replies and avoid apply stale version vector
changes.
2022-03-15 14:23:07 -07:00
a1dfd92a43
Fix merge conflict between tenants and checkpointing API. Avoid using .toString() in trace event details and rename (and move) a couple variables.
2022-03-15 14:18:06 -07:00
7464c7747c
Merge pull request #6600 from sfc-gh-ajbeamon/add-tenant-support-back-to-mapped-range
...
Add tenant support back to mapped range requests
2022-03-15 13:46:32 -07:00
e8e69e7c56
Fix formatting
2022-03-15 15:15:16 -05:00
c3a68d661e
Physical Shard Move ( #6264 )
...
Physical Shard Move part I: Checkpoint creation, transfer and restore.
2022-03-15 13:03:23 -07:00
ce5b567971
Cleanup the cleanup
2022-03-15 15:02:28 -05:00
a04934465c
Add tenant support back to mapped range requests. Fix ACTOR warning.
2022-03-15 12:41:08 -07:00
e8077b65e1
Merge pull request #6559 from sfc-gh-ajbeamon/fdb-tenant-client
...
Add client support for tenants
2022-03-15 12:40:17 -07:00
22faf8e5b3
Ignore tenants when setting watches on the metadataVersionKey or sending mutations or conflict ranges that use the metadataVersionKey.
2022-03-15 10:46:31 -07:00
a70adf9fcd
Convert cached ranges to relative ranges when using tenants.
2022-03-15 09:23:30 -07:00
149ca44b9b
Disallow setting raw access on a transaction that has specified a tenant
2022-03-15 09:23:30 -07:00
0536e56725
The tenant cache uses a simpler eviction policy. Some various improvement around the use of futures. Add documentation for C API and fix inaccurate statement in special keys documentation.
2022-03-15 09:23:30 -07:00
06f088e088
Pass along versions with GetKeyServerLocationsRequests, if we have them.
2022-03-15 09:23:30 -07:00
68069c9784
Undo some unused changes. Fix tenant cache eviction.
2022-03-15 09:23:30 -07:00
8bc2b283e1
Create a special keys API to create, delete, and read tenants. Remove the C API to create/delete tenants.
2022-03-15 09:23:30 -07:00
1e1098ca9a
Don't check whether tenants are enabled when watch is called. Support unknown_tenant error.
2022-03-15 09:23:30 -07:00
ecccfd0868
Add cache invalidation to tenant cache. Send tenant ID along with tenant name in requests to validate that the tenant hasn't changed. Fix a few bugs.
2022-03-15 09:23:30 -07:00
fb73e1857a
Some get key requests should not happen within a tenant. Don't set read_system_keys in watch logic unless necessary. Set access_system_keys in a couple movekeys functions.
2022-03-15 09:23:30 -07:00
2a21126028
Don't apply read prefixes on the client. Cache tenant data locally.
2022-03-15 09:23:30 -07:00
a168faf219
Fix bugs in serialization and tenant prefix management on the client.
2022-03-15 09:21:27 -07:00
c635dcd3ad
Add tenant support in the FDB native client
2022-03-15 09:21:27 -07:00
117ee637db
Merge remote-tracking branch 'origin/main' into features/private-request-streams
2022-03-15 17:17:47 +01:00
bed799220a
Addressed review comments, added test
2022-03-15 16:57:26 +01:00
800478ad5e
Add a new client error code stale_version_vector
2022-03-14 20:50:12 -07:00
8d88b7ca41
making range assigner boundary changes fully synchronous to ensure no races with things reading assignments
2022-03-14 14:39:00 -05:00
98eeaac1b7
More debugging and fixing a stuck merge cursor
2022-03-14 11:57:56 -05:00
8dcac2f76d
Fix typos
2022-03-13 10:02:11 -03:00
9b8c101158
Fix a rare race with version vector cache
...
The events that led to the assertion failure are:
Many txn started, some get ReadVersion1, some get ReadVersion2.
Recovery happened.
GRV proxy change detected and the client version cache is cleared.
Old GRV responded with ReadVersion1, and client version cache is updated to ReadVersion1.
A txn starts reading with ReadVersion2, triggering the assertion.
The fix is to throw an error to trigger transaction restart so that a new GRV
will update the version cache.
2022-03-12 12:03:01 -08:00
82eef0c7f7
Better CF tracing for client and server
2022-03-11 16:50:38 -06:00
2a506f2dff
Updating SS desired every mutation to avoid CF merge deadlock
2022-03-10 20:11:49 -06:00
211e5b579a
Adding back debug stream uid to help track down bugs
2022-03-10 13:55:42 -06:00
903f7adbc4
more cleanup
2022-03-10 13:53:06 -06:00
e2c7c30faf
GetMappedRange support serializable & check RYW & continuation ( #6181 )
2022-03-10 10:05:44 -08:00
1f964ac085
BM focused cleanup
2022-03-09 15:04:17 -06:00
b21d0943b9
client-focused cleanup
2022-03-09 10:01:25 -06:00
e71b3533f9
Merge branch 'main' into blob_integration
2022-03-09 08:59:56 -06:00
8fac0081a8
Merge remote-tracking branch 'origin/main' into features/private-request-streams
2022-03-09 11:00:00 +01:00
402fa4dd9e
Merge pull request #6428 from sfc-gh-ajbeamon/fdb-tenant-add-interfaces
...
Add server-side and protocol support for tenants
2022-03-08 15:33:18 -08:00
63b8b838a2
Merge pull request #6558 from sfc-gh-ajbeamon/special-keys-limits
...
Add support for a limits hint in special keys range reads
2022-03-08 15:32:59 -08:00
6f7adb04a5
Send a version along with GetKeyServerLocationsRequests to allow the proxy to reject missing tenants more quickly. Change the commit proxy tenant map to be unversioned.
2022-03-08 13:33:56 -08:00
58f5cea39b
fix typo
2022-03-07 11:20:24 -06:00
04f678d515
Fixing sample logic
2022-03-07 11:20:24 -06:00
665e182bcc
TSS Metrics improvements
2022-03-07 11:20:24 -06:00
c99c7cf55f
Apply tenant prefixes on the storage server. The commit proxy can serve tenant location requests. Fix some bugs.
2022-03-06 21:54:21 -08:00
5fa9d3e1b7
Add a tenant parameter to read and commit requests. Store a map of all tenants on commit proxy and storage servers. Add an option to require tenant mode.
2022-03-06 21:54:21 -08:00
36435af4f5
Add support for a limits hint in special keys. Fix a typo and resolve some comment formatting weirdness.
2022-03-06 21:52:39 -08:00
1a5bf25b5c
Update code base to use fmt 8.1.1
2022-03-04 15:52:06 -08:00
1555c0e2aa
Fix conflict between two recent merges by using the new errorUnsuppressed function.
2022-03-04 12:41:21 -08:00
cdebda35ab
Merge pull request #5725 from sfc-gh-jfu/jfu-grv-cache
...
Add transaction option for clients to use cached read versions
2022-03-04 09:17:27 -08:00
2b8882ca0a
Adding exponential backoff for consecutive change feed stream failures
2022-03-02 18:46:40 -06:00
e73c0a31e6
add wiggle_server_ids and wiggle_server_addresses in status json
2022-03-02 10:03:23 -08:00
5e39da7741
fixup! Reformat the source
2022-02-28 18:03:52 -08:00
02e396274f
Fix the compile error on NativeAPI.actor.cpp
2022-02-28 18:03:52 -08:00
aadd706a0a
Port back addressed comments for fixing get key location issue
2022-02-28 16:08:57 -08:00
0bdae35a2c
Address comments and move updateFailedEndpointRefreshTime
2022-02-24 16:35:30 -08:00
9c99c59b0c
Fix get key location overloading proxies
2022-02-24 16:35:30 -08:00
e5b4fb3d80
Adding explicit popVersion to change feed reply, like tlogs have
2022-02-24 15:20:18 -06:00
250a88e682
Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement.
2022-02-24 12:25:52 -08:00
8cc9a5af1a
Rebase 02/23
2022-02-23 14:23:28 -05:00
ce1d71472d
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache
2022-02-23 12:35:25 -05:00
cf31e14904
Merge remote-tracking branch 'origin/main' into features/private-request-streams
2022-02-23 10:29:32 +01:00
622d89b552
Rebase on main.
...
Since we changed ClusterConnectionString's status flag from boolean to enum in #6422 , we need to update this PR correspondingly.
2022-02-22 16:29:59 -08:00
481587a8c6
Turn on hostname logic.
2022-02-22 16:29:59 -08:00
ae071a7211
clean up debug trace lines
2022-02-22 17:16:11 -05:00
b60cc02aa2
fix compile error
2022-02-22 13:10:30 -05:00
31d56b2f50
adjust knob setting, revert trace changes, and disable cache if onError retries transactions
2022-02-22 12:42:53 -05:00
dc973fb67e
Allow List and first test
2022-02-22 11:15:16 +01:00
a7a3dccba5
TSS Mismatch Quarantine fix
2022-02-17 19:16:07 -06:00
ef25e4fdfb
Sending empty version back immediately to establish stream connection and acknowledgement queue for change feed stream requests
2022-02-17 18:10:17 -06:00
38a75a8b89
Merge branch 'main' into blob_integration
2022-02-17 17:47:38 -06:00
26480c4705
change feed popping fixes
2022-02-15 15:12:23 -06:00
d399daebed
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache
2022-02-15 15:09:40 -05:00
6f1c3d50bb
add debug traces for testing
2022-02-15 15:08:53 -05:00
092b5cee4b
MVC2.0: Rollback added code
2022-02-14 13:50:42 -08:00
a63d218e9d
simplify test workload and adjust ratekeeper throttling strategy
2022-02-11 16:41:14 -05:00
2e63ac6963
code review changes: additional test checks, cleanup extra code, mark internal option hidden
2022-02-10 17:01:09 -05:00
458e708272
addressed code review comments: renamed variables, small functional changes, style changes
2022-02-10 16:17:54 -05:00
f4c235f134
propagating change_feed_popped correctly
2022-02-09 09:34:13 -06:00
72e06369a4
Merge remote-tracking branch 'apple-upstream/main' into version-vector-prototype
2022-02-08 17:47:57 +00:00
7e5ad88780
formatting for Github CI
2022-02-07 15:31:59 -05:00
9c0a512cf5
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache
2022-02-07 14:51:12 -05:00
ec2bbf0343
clean up some more trace lines and leftover code snippets
2022-02-07 14:50:04 -05:00
9b3f6fe452
More bugs in streaming split
2022-02-07 13:25:14 -06:00
88b1593681
Removing incorrect assert
2022-02-07 11:56:14 -06:00
88cab7fb67
More change feed fetching improvements and optimizations
2022-02-07 11:19:51 -06:00
c48ca9430d
Fixes for streaming split keys
2022-02-07 10:36:22 -06:00
809ec24184
More change feed buffer tuning
2022-02-07 08:19:19 -06:00
1736eabd33
fixes for split metrics stream
2022-02-04 18:24:55 -06:00
41a274ed9c
Fixing bug and refactoring iterative range splitting
2022-02-04 17:10:46 -06:00
ddfc301d74
Improving memory footprint of change feeds and making it configurable
2022-02-04 16:41:25 -06:00
a974773676
Handled change feed getting moved away or cancelled during move
2022-02-04 09:33:04 -06:00
2592c3f0ae
Implemented less efficient version of reply stream closing
2022-02-03 20:07:49 -06:00
9c8afdf35c
Debugged and fixed stuck change feed issue
2022-02-02 15:15:54 -06:00
6d05ecffc2
remove all multi-version client code related to grv cache
2022-02-02 14:39:17 -05:00
d8e7fea421
clean up some comments and debug changes
2022-02-02 14:03:32 -05:00
d7819f8387
Merge pull request #6304 from sfc-gh-ajbeamon/reset-copy-trstate
...
Copy transaction state during reset
2022-01-31 09:11:04 -08:00
89f4ed7481
Merge pull request #6296 from sfc-gh-ajbeamon/mark-external-system-transactions
...
Mark various externally exposed system transactions as read/access system keys
2022-01-28 14:16:40 -08:00
f9f3735f73
Add resolveHostnamesBlocking() in ConnectionString and IClusterConnectionRecord.
...
Also, combine IClusterConnectionRecord::getConnectionString() and IClusterConnectionRecord::getMutableConnectionString() to IClusterConnectionRecord::getConnectionString(), and rename setConnectionString() to setAndPersistConnectionString().
2022-01-28 12:20:41 -08:00
ac1fd056dd
Added change feed popped read guard for blob workers
2022-01-28 10:45:33 -06:00
c26e11c2c3
Adding extra debugid to change feed stream requests to track things better
2022-01-27 18:53:27 -06:00
8418cb2839
Mark various externally exposed system transactions as read/access system keys. Also adjust a few lock aware declarations to read lock aware.
2022-01-27 13:58:33 -08:00
29f0bcda21
Copy transaction state during reset to avoid potential issues with operations that were in-flight during the reset
2022-01-27 13:13:48 -08:00
bc7b76b407
Fix for gotAll in getChangeFeedMutations
2022-01-27 14:25:08 -06:00
1800259d7c
Merge pull request #6136 from sfc-gh-ajbeamon/client-description-trace-field
...
Add ClientDescription field for client processes to identify the client logging an event.
2022-01-26 11:37:31 -08:00
9db49484d4
add reference counting to shared state and create new fdb_c_types header
2022-01-25 16:10:03 -05:00
5d9843ec32
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache
2022-01-24 13:20:51 -05:00
133ee2c113
added fdb_c_internal.h and set up shared space initialization in multiversion api db creation
2022-01-24 12:55:24 -05:00
dd1d48754d
revert create_database signatures and added new shared state struct
2022-01-21 14:32:04 -05:00
3a21438bb8
remove getclusterid api
2022-01-21 13:12:30 -05:00
a8f1d6fa00
StorageMetrics and and halting fixes
2022-01-20 19:27:25 -06:00
915e2f6c1c
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache
2022-01-20 16:17:20 -05:00
ca1839c405
add sim_validation back to check staleness
2022-01-20 15:45:50 -05:00
9544379cdf
rebase
2022-01-20 11:12:33 -05:00
79704aa05f
Fixing change feed private mutation handling
2022-01-18 20:17:39 -06:00
58bc3a78ea
Fixed range assignment bug and fixed printf compiler warnings
2022-01-14 17:46:32 -06:00
e44dae0e5b
Removing incorrect assert
2022-01-14 15:47:51 -06:00
6b1e387df7
removed debugging
2022-01-14 15:44:31 -06:00
9174bfbb4e
bug fix and extra check in merge cursor
2022-01-14 15:26:37 -06:00
e708072fd8
Cleaned up new merge cursor and fixed memory lifetime issue with cancelling a change feed
2022-01-14 14:47:23 -06:00
be80aa35a4
Fix memory issue with client watch metadata
2022-01-14 09:42:28 -08:00
0e58478c4d
Initial rework of merge cursor
2022-01-14 10:28:54 -06:00
07e5319477
Merge pull request #6165 from sfc-gh-ajbeamon/native-api-refactor
...
Refactor Native API Transactions
2022-01-13 14:53:19 -08:00
661c50d29f
Handled server_overloaded in storageFeedVersionUpdater
2022-01-13 16:29:25 -06:00
cef5d400f9
Fix issue where enabling debug transaction after the transaction has been used could cause an assertion failure.
2022-01-13 13:43:42 -08:00
17415168b6
Make the use provisional proxies parameter be of type UseProvisionalProxies everywhere.
2022-01-13 12:41:20 -08:00
e7d15b2947
more explicit empty version problems
2022-01-13 12:10:44 -06:00
6e361a3283
Need delay(0) for empty check at start of change feed
2022-01-13 09:48:31 -06:00
515dc40153
Fixed issue with out of order empty versions
2022-01-13 08:43:29 -06:00
5dccca4081
Implement initial concept of shared cache space in NativeAPI DatabaseContext
2022-01-12 16:49:00 -05:00
fe9a3d27d8
Not allowing whenAtLeast after change feed end version
2022-01-12 13:16:49 -06:00
da76abbdf6
expose cluster ID through NativeAPI and C API
2022-01-11 17:06:25 -05:00
6b202fc9a8
Fixed stuck change feed storage updater and improved debugging
2022-01-11 15:35:54 -06:00
3b2e58ada8
Fixed rollback of snapshot files bug
2022-01-11 15:35:54 -06:00
b7ca2a656c
remove epoch/generations for grv cache
2022-01-11 13:35:43 -05:00
8b77c8d79a
Remove unused OpenSSL code
2022-01-11 10:34:37 -08:00
64090dd2c0
Suppress a small leak that LSAN is reporting
...
I cannot seem to figure out why LSAN is reporting this, but if it is a
real leak than it's only a few bytes. Better to have the ASAN tests
actually passing IMO.
2022-01-10 13:44:09 -08:00
5bd8350eac
also handling server overloaded in change feed pop
2022-01-10 13:13:00 -06:00
91574fa0b9
fix: poppingChangeFeeds did not properly handle broken promise errors
2022-01-10 10:14:54 -08:00
21309fb55b
Change feed merge cursor whenAtLeast fix
2022-01-07 16:16:29 -06:00
807ec37e84
BG request retry on failed blob worker properly
2022-01-05 17:38:49 -06:00
0f66cca8e0
Fixing change feed race with empty mutation and error
2022-01-05 16:40:07 -06:00
138932c12f
Fixed a memory bug in the watch map. A few other formatting tweaks, etc.
2022-01-05 11:26:43 -08:00
bc69521a91
Several fixes with restarting BW/BM
2022-01-05 12:48:53 -06:00
7fab9ea5c0
adjust sideband workload to expose another consistency error
2022-01-05 12:51:39 -05:00
431d3de43b
Fix some small bugs
2022-01-04 14:23:00 -08:00
329a4b1c85
Handling end of stream correctly in merge cursor optimization
2021-12-22 12:09:20 -06:00
738b72918a
send merge cursor's buffered data if all sub-streams have something at this version
2021-12-21 16:32:56 -06:00
b0aea91895
Broadening explicit disconnect handling to explicit error handling of all types
2021-12-21 14:12:09 -06:00
d337e8fbe8
Handled disconnects explicitly in CF streams
2021-12-21 10:38:04 -06:00
abdaf5c9cd
extra debugging for CF empty versions
2021-12-21 08:58:20 -06:00
d8e161f89e
Refactor NativeAPI transactions to create and pass around a reference counted state object. Watches no longer use the tranasction info object but instead use their own state.
2021-12-17 11:57:39 -08:00
6ffe2067fe
Set the CommitTransactionRequest span context upon construction/reset rather than in tryCommit.
2021-12-16 14:08:08 -08:00
af931ad989
Add support for universal trace fields that apply to all events in a process. Add a universal field on clients called ClientDescription that identifies the client logging the event.
2021-12-10 15:03:14 -08:00
ea7a5114f5
Merge branch 'bw_rollback_grv' into blob_integration
2021-12-10 16:59:59 -06:00
90a652ccfb
fix merge conflict
2021-12-10 16:43:58 -06:00
e9d1f36cae
More cleanup.
2021-12-10 16:43:58 -06:00
310d990b12
Add debugging.
2021-12-10 16:43:58 -06:00
307d049c9d
Cleaning up some memory lifetime issues
2021-12-10 16:12:06 -06:00
ff2cd691cd
Switching back to GRV for committed version checking, with proper rollback checking
2021-12-10 15:27:25 -06:00
9fbccb21d2
Remove checks from native API.
2021-12-10 14:00:34 -06:00
fdc94d3356
Fix blobGranuleVerifier setup.
2021-12-10 14:00:34 -06:00
0f3318a8ba
add TODO
2021-12-10 14:00:34 -06:00
8dc5f79dc7
Made blob worker assume no empty mutations, and made merge cursor not send them
2021-12-09 15:00:46 -06:00
1ee0b16bfa
Fixed bug in merge cursor whenAtLeast
2021-12-09 14:19:00 -06:00
86f6f73518
Merge cursor debugging and fix in BW::waitForVersion
2021-12-09 10:51:31 -06:00
c24826430f
Removing this check as it still won't work if the knob is set
2021-12-08 18:13:41 -06:00
ef74d309d8
Adding delay after onEmpty for merge cursor as well
2021-12-08 12:48:30 -06:00
6c79750412
Added a bunch of debugging for tracking whenAtLatest problems, and fixed a bug there
2021-12-08 12:41:45 -06:00
4a9c2f4172
fixing merge conflicts
2021-12-08 08:47:58 -06:00
c5b2b384da
Fixing ASAN issues
2021-12-08 08:42:35 -06:00
3ee6d69f3d
Wait for empty even for whenAtLatest>0
2021-12-08 08:42:35 -06:00
75267aba80
fix: do not update lastReturnedVersion until the client has consumed the mutations
2021-12-07 13:03:50 -08:00
fdd35070a1
fix: merge cursors can have more mutations sent to self->mutations after it goes empty
2021-12-07 09:57:39 -08:00
def41697bf
Merge pull request #6083 from sfc-gh-tclinkenbeard/remove-temporaries
...
Avoid creating unnecessary temporary objects
2021-12-06 13:24:56 -08:00
528df86a28
fix: refactored and fixed changeFeedWhenAtLatest returning early
2021-12-06 14:06:17 -06:00
02c650c6e7
fix: sending to mutations can cause whenAtLeast to update lastReturnedVersion before we have a chance to
2021-12-05 16:16:20 -08:00
da9c42fd9e
do not log transaction metrics in simulation
2021-12-05 14:59:24 -08:00
f711ef56a3
fix: whenAtLeast updates lastReturnedVersion
2021-12-03 14:27:11 -08:00
2d7c44af31
Fixing bug with readVersionOut being null
2021-12-02 16:42:14 -06:00
bc16e23d2c
Merge branch 'master' into blob_integration
2021-12-02 15:51:18 -06:00
d85eb330e0
retooling some waitForVersion stuff and adding asserts
2021-12-02 14:52:16 -06:00
f43169cb7b
bug fixes
2021-12-02 12:40:37 -06:00
632f701c8a
First pass at using ChangeFeedData in blob worker
2021-12-02 11:42:16 -06:00
85c6e30aec
Fix: several empty version issues with large single-version change feed entries
2021-12-02 09:56:15 -06:00
3b5d23ef88
Remove unnecessary temporary objects when appending to vector of pairs
2021-12-01 18:15:16 -08:00
2c63d0ae03
cleaning up prints
2021-12-01 17:17:19 -06:00
a82845af43
Merge branch 'master' into bg_bindings
2021-12-01 16:55:28 -06:00
13a4743e76
Check whether connRecord is null before logging it.
2021-11-29 14:22:44 -08:00
dcb0fc82e9
remove future wait optimization with NotifiedDouble and use regular GRV path if cache is too stale
2021-11-29 15:13:48 -05:00
b5b0afb96d
narrow down places where to advance cache generation
2021-11-29 14:38:52 -05:00
1fb537cde5
Add cache generations to create time boundaries when commits time out or fail
2021-11-29 14:16:49 -05:00
264c75b9a6
Add some extra client logging details:
...
1. Add a trace event when a database is created and move the cluster file / connection string from ClientStart to the new trace event
2. Add a detail for the path to the image being loaded
3. Add a detail for whether a client library is primary or not
4. Set a thread name for each external client thread that includes the release version
2021-11-29 09:57:10 -08:00
2b0ade5250
Change throttling threshold to count loop iterations instead of time
2021-11-25 13:31:55 -05:00
bba7ed8ac0
Add panic knob to shut down cache from outside txn options
2021-11-25 13:22:28 -05:00
efb21ca6a1
Merge branch 'master' into blob_integration
2021-11-24 13:17:54 -06:00
e3422b2443
fix: change feed streams were not being closed properly
2021-11-23 15:01:44 -08:00
3360000a91
fix: an empty mutation vector did not mean that all of the data was returned
2021-11-23 15:01:44 -08:00
9df287ea06
fixed empty version merge logic
2021-11-23 15:01:44 -08:00
6d9f134bf0
fix: merge needs to update on empty versions to release other mutations blocked by the empty version
2021-11-23 15:01:44 -08:00
014fb2f31c
fix: onEmpty must check all merged streams
2021-11-23 15:01:44 -08:00
a0afe530f8
protect against the changeFeedStorageData getting destroyed
2021-11-23 15:01:44 -08:00
547c5899f8
whenAtLeast could return an incorrect result when the connection closed
2021-11-23 15:01:44 -08:00
31b2414706
fixed a crash while popping change feeds
2021-11-23 15:01:44 -08:00
3f24128da4
Merge branch 'master' of github.com:apple/foundationdb into jfu-grv-cache
2021-11-19 14:46:55 -05:00
e9c58d9f86
Check for sustained throttling in the proxy to lower threshold time and avoid false positives
2021-11-19 14:33:06 -05:00
3d0ed2083c
Merge pull request #6000 from sfc-gh-tclinkenbeard/setup-flow-knobs
...
Setup global flow knobs in `setupNetwork`
2021-11-18 19:30:26 -08:00
2613ec7561
Expand use of fmt to get rid of %ld usage
2021-11-17 17:03:32 -08:00
1e2060a52b
add extra tracking in dbcontext to periodically contact GRV proxies
2021-11-17 15:24:37 -05:00
766a05d33c
Merge remote-tracking branch 'origin/master' into add-format-warning
2021-11-17 12:14:01 -08:00
e04526b345
Allow knobs to be set via network options before network is setup
2021-11-16 13:47:37 -08:00
8714480370
fix some compile issues
2021-11-16 13:59:35 -05:00
16e551e834
change knob to be an adjustable probability instead of binary on/off
2021-11-16 13:41:19 -05:00
deb0eb85e8
Initialize global flow knobs in setupNetwork
2021-11-16 00:41:34 -08:00
f140028086
Merge branch 'master' into blob_integration
2021-11-15 15:09:40 -06:00
416d94d5b0
add some more trace details and remove spammy trace throttling temporarily
2021-11-15 13:45:23 -05:00
94a51e57a5
Merge branch 'master' into feature-changefeed-empty-versions
...
# Conflicts:
# fdbclient/StorageServerInterface.h
2021-11-14 19:13:05 -08:00
6909754b21
changefeeds now have a whenAtLeast function for efficiently learning when the version has updated but no mutations have been committed
2021-11-14 19:08:46 -08:00
62efeb6812
Merge remote-tracking branch 'origin/master' into add-format-warning
2021-11-12 11:50:36 -08:00
bab8756e17
fixed bugs in small splits
2021-11-11 13:50:19 -06:00
329091e14f
Merge branch 'master' into bg_bindings
2021-11-11 10:13:37 -06:00
b8ac4213a1
Switched BG APIs to transaction instead of database
2021-11-11 08:59:06 -06:00
0dfb72176e
Merge pull request #5857 from sfc-gh-vgasiunas/notify-client-lib-changes
...
A mechanism to notify MVC about relevant client library status changes on the cluster
2021-11-11 07:43:20 -07:00
1da288822f
Remove distributed trace database option
2021-11-10 13:33:49 -08:00
8bc0c3e8a2
Convert distributed trace sampling rate to a database option
2021-11-10 13:33:49 -08:00
c93052121f
Fix issue where transaction spans would not be recorded
2021-11-10 13:33:49 -08:00
51b8ccf7d3
Merge remote-tracking branch 'apple/master' into notify-client-lib-changes
2021-11-10 18:40:34 +01:00
fdb3b72e35
Introduce GetRangeAndFlatMap to push computations down to FDB
...
Re-introduce #5609
2021-11-09 13:52:28 -08:00
e984b2f5d0
remove sevdebug temporarily and add extra trace to verify knob values
2021-11-09 15:13:50 -05:00
6540bef9ee
Clear client VV cache after recovery
...
This is to prevent the client from sending older VV to the new GRV proxies,
because the client side cache can be stale, i.e., not knowing commits for a tag.
This was found in simulation:
A client has a VV cache of [{0:1,841885664} maxversion: 841885664]; client
issues a transaction commits at 841905696. There is a recovery. The client got
Delta=[ maxversion: -1] back. Client issues a read at 946662688, which uses the
cached version 841885664 < 841905696 , thus reading corrupted data.
2021-11-09 08:37:22 -08:00
586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB"
2021-11-04 08:46:56 -07:00
6c98e35893
Rename Hop to FlatMap
2021-11-03 13:32:01 -07:00
0853661d13
Introduce getRangeAndHop to push computations down to FDB
2021-11-03 13:21:16 -07:00
e3839e198a
Cleanup
2021-11-03 13:41:41 -05:00
382882f1c1
mako successfully calls read_blob_granules and gets stuff back
2021-11-02 13:43:42 -05:00
d6a31078fe
C API for blob granules
2021-11-02 10:01:23 -05:00
13bb7838aa
Enable clang -Wformat warning
2021-10-30 21:07:38 -07:00
befe1993c4
fix conflict on rebase
2021-10-29 12:25:26 -04:00
875824b186
MVC2.0: Notify clients about relevant changes of client libraries
2021-10-27 23:43:40 +02:00
19076ad4d2
Changed blob manager checkin to by async and to have blob worker speculatively consume change feed ahead of re-snapshot
2021-10-27 16:23:51 -05:00
fd50a011b1
Address PR comments.
2021-10-26 21:39:41 -04:00
5a9d9921d0
Fixes and final cleanup for BM failure handling
2021-10-26 16:16:00 -04:00
17b30f188a
Working impl
2021-10-26 16:16:00 -04:00
0ff8ddc2b6
Merge branch 'master' into blob_full_clean
2021-10-25 13:38:48 -05:00
7273d352a2
Add extra debug traces and ignore "IMMEDIATE" priority txns for rk throttling
2021-10-25 13:44:45 -04:00
6f7558b8ea
Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
...
# Conflicts:
# tests/CMakeLists.txt
2021-10-24 21:06:33 -07:00
0e327d3d0a
fix: do not duplicate lastEpochEnd from different servers
2021-10-24 19:17:11 -07:00
4039dbd8da
fix: combine mutations from the same version into a single MutationsAndVesionRef
2021-10-24 13:46:45 -07:00
e882eb33fc
Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem.
2021-10-22 11:05:18 -07:00
6d74239760
Track throttling by measuring time spent left in queue on the proxy
2021-10-22 13:55:01 -04:00
773886515e
Merge branch 'feature-range-feed' into blob_full_clean
2021-10-22 11:07:51 -05:00
d778cea9b5
fix: prevent infinite loop
2021-10-21 15:16:22 -07:00
f1158371a7
Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
...
# Conflicts:
# flow/error_definitions.h
2021-10-21 00:55:12 -07:00
4e79296a9f
fixed a few bugs with fetching change feeds
2021-10-21 00:44:51 -07:00
3ebabb6edc
fixed incorrect use of change feed errors
2021-10-20 22:37:31 -07:00
f1c8d3fbc8
Add code to disable cache when ratekeeper begins throttling
2021-10-20 15:52:43 -04:00
3f7df58a77
fixed a number of issues
2021-10-19 13:56:52 -07:00
abab45760d
Add some additional logging if the network thread finishes, fails with an error, gets stopped, or is blocked.
2021-10-19 10:05:29 -07:00
912ef76f1c
cleanup before merge
2021-10-18 17:11:14 -05:00
246f035afe
Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
2021-10-18 19:07:33 +00:00
f75c8b25ad
add small random delay before returning a cached read version
2021-10-18 13:59:04 -04:00
e2e852e515
Mitigate transitive includes.
2021-10-18 10:49:25 -04:00
b5074fd597
Reworked all of the system data to encode granule data more efficiently for persistence
2021-10-13 16:28:04 -05:00
5f0ec0612a
Merge branch 'feature-range-feed' into blob_full
2021-10-13 15:44:35 -05:00
bde90f058b
Gate the client API.
2021-10-13 13:26:27 -04:00
e543570343
Add locality cache size to TransactionMetrics. ( #5758 )
...
Co-authored-by: Yao Xiao <yaoxiao@Yaos-MacBook-Pro.local>
2021-10-12 13:10:15 -07:00
282f84c807
added handling for broken promise
2021-10-12 10:36:10 -07:00
5066d67dec
Merge pull request #5715 from sfc-gh-ljoswiak/fixes/tracing-various
...
Add knob to control distributed trace recording percentage
2021-10-11 14:03:53 -06:00
d51edf18dc
fixed merge conflicts
2021-10-09 19:47:24 -07:00
5c642f706e
Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
2021-10-09 19:34:16 -07:00
efc4cec53f
fixed a variety of bugs with change feeds
2021-10-09 19:24:01 -07:00
4fbc193e19
fix priority
2021-10-09 16:40:24 -06:00
9077e15b11
Address review comments
2021-10-09 16:40:24 -06:00
4e723ba3d9
Apply suggestions from code review
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2021-10-09 16:40:24 -06:00
ffdba4a133
assert we never get a stale read version
2021-10-09 16:40:24 -06:00
2b2d059039
Specify option to specifically skip grv cache for the background updater when using debug knob
2021-10-08 17:32:44 -04:00
b770936cde
move background updater actor initialization
2021-10-08 16:38:21 -04:00
ecdafba336
add debug traces to background grv updater
2021-10-08 15:17:28 -04:00
f68d018a06
conflicts
2021-10-08 12:39:39 -04:00
efd75ccc74
Add histograms to client GRV batcher. ( #5727 )
2021-10-07 13:30:25 -07:00
31664ddb78
Fixing change feed error retry in pop
2021-10-07 10:07:18 -05:00
54d5e8366d
fixing another change feed hang error
2021-10-06 16:04:01 -05:00
cc38c48123
Add debug knob to use grv cache for performance testing
2021-10-06 14:35:42 -04:00
bbe7c7ca9f
found change feed liveness bug
2021-10-06 12:32:16 -05:00
9a18cc8f41
Change debug time bound update logic to be located in masterserver
2021-10-05 17:31:31 -04:00
44a854772f
Merge branch 'master' of github.com:apple/foundationdb into jfu-grv-cache
2021-10-05 12:55:02 -04:00
ae31f01dc4
remove cache update from extractreadversion and set lower time bound to 0
2021-10-05 12:54:22 -04:00
d560eb1fea
debug time bounds using sim_validation
2021-10-04 14:12:31 -04:00
e034f66fc2
Add knob to control trace recording percentage
2021-10-01 16:30:46 -07:00
fa1fe5f08b
added blob worker rollbacks that handle (most) cases
2021-09-24 17:52:36 -05:00
a986679b7f
More bug fixes
2021-09-24 10:02:02 -05:00
32a834b0c2
The computation of tag throttling backoff was taking a minimum with a knob for how long a throttle lasts. When this knob was buggified to 0 in simulation, the backoff would then be 0, and this could cause some transactions to get stuck in a retry loop that failed to advance time. Eventually the test would timeout or potentially run out of memory.
2021-09-23 10:03:57 -07:00
4cd5d138ff
Fixed segfault involving missing futures and infinite looping in background actor
2021-09-22 13:13:35 -04:00
16171d8252
Refactoring well-known endpoint registration
...
- List all well-known endpoints of FDB in a single enum
- Identify well-known endpoints by plain IDs
2021-09-21 11:05:31 -06:00
d4f9a09a4f
- Address test failures
2021-09-17 21:21:44 +00:00
2aa3b44d4e
Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
...
- Conflicts:
fdbserver/LogSystem.h
fdbserver/LogSystemConfig.h
fdbserver/TagPartitionedLogSystem.actor.cpp
- Files modified during merge:
modified: fdbserver/LogSystem.cpp
modified: fdbserver/LogSystemConfig.cpp
2021-09-17 19:36:18 +00:00
abf73047ca
Enforce std:: specifier rather than using namespace
2021-09-16 19:40:28 -07:00
35621a255e
- Address review comments
2021-09-16 12:00:26 -04:00
ffbdc21ca8
- Address review comments
2021-09-16 12:00:26 -04:00
852fc96200
Address simulation test failures caused by:
...
- Assertion failures in MoveKeys.actor.cpp
- Wrong results returned by getRange()
Changes:
DatabaseContext.h, NativeAPI.actor.[h,cpp]:
- Introduce a new flag, TransactionInfo::readVersionObtainedFromGrvProxy.
- Set this flag to true by default, and clear it when the read version of a
transaction is explicitly set (by using setVersion()).
- Modify getLatestCommitVersions() to not populate "latestCommitVersions" if
this flag is not set. (This will cause storage server to read at the specified
read version.)
- Modify getRange() actor to always use the specified version as the read
version (except when the specified version is latestVersion).
- Modify waitForCommittedVersion(), getRawVersion(), and getConsistentReadVersion()
to update local version vector cache after receiving GetReadVersionReply.
IClientApi.h, IConfigTransaction.h, ISingleThreadTransaction.h,
MultiVersionTransaction[.actor].[h,cpp], ThreadSafeTransaction.[h,cpp],
ApiWorkload.h:
- Add methods to get the spanID of a transaction and also the version vector
cached in a transaction. (Likely to be useful for debugging simulation test
failures.)
VersionVector.h:
- Update "maxVersion" when populating/applying a delta. (Note that empty
mutation messages only update VersionVector::maxVersion.)
BackupWorker.actor.cpp:
- Update local version vector cache after receiving GetReadVersionReply message.
Status.actor.cpp:
- Update local version vector cache and
TransactionInfo::info.readVersionObtainedFromGrvProxy after setting the
read version.
2021-09-16 12:00:26 -04:00
6673540553
TSS Metrics fix
2021-09-14 13:15:05 -05:00
87fac6e301
adjust to changed function signature after merge
2021-09-13 13:04:55 -04:00
d971b63bbf
Merge branch 'master' of github.com:apple/foundationdb into jfu-grv-cache
2021-09-13 12:47:42 -04:00
cfbe5eca9c
Use notifieddouble to track GRV request time and include grvdelay in timing calculations
2021-09-13 11:57:19 -04:00
cefb66d64c
fixing a couple bugs
2021-09-10 12:52:33 -05:00
c2d1d1704f
Merge branch 'feature-range-feed' into blob_full
2021-09-10 11:21:52 -05:00
ac5b580e2d
Merge branch 'master' into feature-range-feed
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbclient/StorageServerInterface.cpp
# fdbclient/StorageServerInterface.h
# fdbserver/ApplyMetadataMutation.cpp
# fdbserver/TLogServer.actor.cpp
# flow/error_definitions.h
2021-09-09 23:13:22 -07:00
1ce2457f7e
initial commit
2021-09-09 14:01:42 -04:00
eb76343dfb
Added blob granule reassignment and splitting
2021-09-08 14:09:14 -05:00
d9782bc29d
fixed a few bugs with merging change feeds on the client
2021-09-07 21:35:00 -07:00
f53ddf6720
fix: handle the case where I change feed is added while a fetch is in progress
2021-09-06 13:51:53 -07:00
af0792b1f5
fixes that allow change feeds to pass correctness in tests with no failures
2021-09-03 17:24:33 -07:00
b02e8d99c6
added a workload to test change feeds
2021-09-03 14:19:27 -07:00
8309c2ce26
filter mutations in change feeds to the requested range
2021-09-03 12:06:13 -07:00
afd8df4bb9
Merge commit '7f2b6c2c7ec2efcd7339e3c6531e4906322477c3' into network-thread-busyness-burstiness-score
2021-09-03 09:48:11 -07:00
4bbae59bb0
Fixed a few bugs and added a backup popping mechanism
2021-09-02 21:11:44 -07:00
27151b0831
support reads from range feeds that span multiple storage servers
2021-09-01 14:35:51 -07:00
46adada5ff
Cleaned up debugging and fixed a couple bugs
2021-08-31 12:30:43 -05:00
2fa97f60fe
Update fdbclient/NativeAPI.actor.cpp
...
Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
2021-08-31 08:53:37 -07:00
8689e1f106
merge with master
2021-08-30 15:29:08 -04:00
A.J. Beamon