Commit Graph

161 Commits

Author SHA1 Message Date
Josh Slocum 34e6a8f942
Merge pull request #7399 from sfc-gh-jslocum/bg_tenant_improvements
Bg tenant improvements
2022-06-17 11:19:41 -05:00
Josh Slocum b3597ef3a8 Added plumbing for tenant-aware purge granules 2022-06-16 13:04:34 -05:00
Vishesh Yadav fd6f6eb06a
Merge pull request #7364 from sfc-gh-ljoswiak/fixes/unnecessary-transaction-initialization
Remove unnecessary ReadYourWritesTransaction initialization
2022-06-14 11:02:31 -07:00
Renxuan Wang 839af5701e
Fix bug in resolveTCPEndpoint() when hostname resolving fails. (#7375)
* Close trace file when error happens in runNetwork().

* Improve the bestCount algorithm in getLeader().

In the current implementation, if the nominees are [0,1], the chosen leader will be 1, which is an exception to other cases and our expectation that if 2 nominees have the same frequency, the one with lower id will be the leader.

* Remove unnecessary new statement.

stream will never be a nullptr.

* Move self->dnsCache out of lambda capture.

Member variables are not capture by default, thus, `host` and `service` are not captured. This somehow successfully compile, but throws std::bad_alloc or basic_string::_S_create exceptions when we call `host+":"+service` in dnsCache.remove().

* Revert unintended change.

* Address comments.
2022-06-13 20:24:30 -07:00
Lukas Joswiak 3b3ef49d40 Remove unnecessary transaction initialization
`ReadYourWritesTransaction` has memory allocated before being passed to
the main thread. This allows both threads to continue to access the
transaction object. Currently, the transaction gets allocated and
initialized on the foreign thread, and then re-initialized on the main
thread. This causes a bunch of extra, unnecessary work for each
`ReadYourWritesTransaction` where the temporary object gets destructed.

The fix is to only allocate memory for the `ReadYourWritesTransaction`
on the foreign thread, and then initialize it once on the main thread.
2022-06-10 16:53:19 -07:00
Vaidas Gasiunas 4d6ec7663f Close file at the end of runNetwork rather than in stopNetwork in order to avoid a race condition,
also in order to continue tracing the pending network thread activity.
Poll event throttler only in the network thread in order to avoid a race condition.
2022-05-31 13:59:09 +02:00
Hao Fu 853e6a346b
Optimization: support removing index conditionally (#7116) 2022-05-13 10:10:43 -07:00
Ray Jenkins dc9e782ccc
OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
Ray Jenkins 1c5bf135d5
Revert "Migrate to OpenTelemetry tracing. (#6855)" (#6941)
This reverts commit 5df3bac110.
2022-04-25 09:29:56 -05:00
Ray Jenkins 5df3bac110
Migrate to OpenTelemetry tracing. (#6855) 2022-04-20 09:26:37 -05:00
Dan Lambright 1b3b4166c6
Merge branch 'main' into vv 2022-04-08 17:18:13 -04:00
Josh Slocum 6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Dan Lambright 5bdc525353
Merge branch 'main' into vv 2022-04-08 13:16:04 -04:00
Jingyu Zhou 4fd414a8ed Merge branch 'main' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
2022-03-31 09:38:36 -07:00
Jingyu Zhou cfcf0f152c Merge branch 'main-4a085fc84' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
	fdbserver/ClusterRecovery.actor.cpp
	fdbserver/MasterInterface.h
	fdbserver/masterserver.actor.cpp
	flow/error_definitions.h
2022-03-30 22:28:06 -07:00
Jingyu Zhou b34f4052cd Merge branch 'main-f28dfc12b' into vv
Fix Conflicts:
	fdbclient/MultiVersionTransaction.actor.cpp
	fdbclient/NativeAPI.actor.cpp
	fdbclient/NativeAPI.actor.h
	fdbserver/storageserver.actor.cpp
2022-03-30 21:01:25 -07:00
Jingyu Zhou 00b57d4cce Merge branch 'main-67eba5ec7' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbclient/StorageServerInterface.h
	fdbserver/CommitProxyServer.actor.cpp
	fdbserver/storageserver.actor.cpp
2022-03-30 20:05:55 -07:00
Jon Fu f266695a76 clear shared state map entries on version upgrade and use currentProtocolVersion instead of piping version through function parameters 2022-03-30 15:28:55 -04:00
Jon Fu 9d0a520fe0 add ProtocolVersion to shared state and state cluster map 2022-03-30 13:28:06 -04:00
Jon Fu 53d252fda0 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-29 17:23:08 -04:00
Jon Fu bdbe78e48f address code review comments 2022-03-29 17:20:22 -04:00
Josh Slocum 721c64b698 Adding Blob Granule Client C Unit Tests 2022-03-28 15:42:33 -05:00
Jon Fu 7c3151142e Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-23 16:41:33 -04:00
Jon Fu 4466e00716 Fix multithreaded GRV cache code and enable grv caching in multithreaded ctest suite 2022-03-23 16:09:12 -04:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Jon Fu 9731d96797 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-16 14:32:33 -04:00
Jon Fu 7ab205391b Revert "remove all multi-version client code related to grv cache"
This reverts commit 6d05ecffc2.
2022-03-16 14:26:36 -04:00
A.J. Beamon 8bc2b283e1 Create a special keys API to create, delete, and read tenants. Remove the C API to create/delete tenants. 2022-03-15 09:23:30 -07:00
A.J. Beamon c89b1e3537 Use TenantName type instead of StringRef in various places. Add a function on transactions to get the tenant name being used for the transaction. 2022-03-15 09:21:27 -07:00
A.J. Beamon c635dcd3ad Add tenant support in the FDB native client 2022-03-15 09:21:27 -07:00
Tao Lin e2c7c30faf
GetMappedRange support serializable & check RYW & continuation (#6181) 2022-03-10 10:05:44 -08:00
Dan Lambright 9544379cdf rebase 2022-01-20 11:12:33 -05:00
Josh Slocum 3afe9fb6e0 MVC bug fixes 2021-12-10 12:47:53 -06:00
Josh Slocum 7f4fcc8c2c Added FDBResult and made readBlobGranules use it 2021-12-01 16:22:05 -06:00
Josh Slocum 329091e14f Merge branch 'master' into bg_bindings 2021-11-11 10:13:37 -06:00
Josh Slocum b8ac4213a1 Switched BG APIs to transaction instead of database 2021-11-11 08:59:06 -06:00
Vaidas Gasiunas 51b8ccf7d3 Merge remote-tracking branch 'apple/master' into notify-client-lib-changes 2021-11-10 18:40:34 +01:00
Vaidas Gasiunas d6a5cc8460 MVC2.0: Rollback static initialization of the local client API instance 2021-11-10 15:55:33 +01:00
Tao Lin fdb3b72e35 Introduce GetRangeAndFlatMap to push computations down to FDB
Re-introduce #5609
2021-11-09 13:52:28 -08:00
Tao Lin 586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB" 2021-11-04 08:46:56 -07:00
Tao Lin 6c98e35893 Rename Hop to FlatMap 2021-11-03 13:32:01 -07:00
Tao Lin 0853661d13 Introduce getRangeAndHop to push computations down to FDB 2021-11-03 13:21:16 -07:00
Josh Slocum e3839e198a Cleanup 2021-11-03 13:41:41 -05:00
Josh Slocum a5b2f464e3 Refactored granule materialization code to be called by c client as well as internal granule management 2021-11-03 13:19:07 -05:00
Vaidas Gasiunas 276dbd4d37 MVC2.0: Refactoring - remove dependency of MVC on Native API 2021-11-03 16:30:18 +01:00
Josh Slocum 5b2617a524 Added local granule file reading to mako 2021-11-03 09:33:30 -05:00
Josh Slocum 382882f1c1 mako successfully calls read_blob_granules and gets stuff back 2021-11-02 13:43:42 -05:00
Josh Slocum d6a31078fe C API for blob granules 2021-11-02 10:01:23 -05:00
Dan Lambright befe1993c4 fix conflict on rebase 2021-10-29 12:25:26 -04:00
A.J. Beamon e882eb33fc Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem. 2021-10-22 11:05:18 -07:00
A.J. Beamon abab45760d Add some additional logging if the network thread finishes, fails with an error, gets stopped, or is blocked. 2021-10-19 10:05:29 -07:00
Sreenath Bodagala ffbdc21ca8 - Address review comments 2021-09-16 12:00:26 -04:00
Sreenath Bodagala 852fc96200 Address simulation test failures caused by:
- Assertion failures in MoveKeys.actor.cpp
- Wrong results returned by getRange()

Changes:

DatabaseContext.h, NativeAPI.actor.[h,cpp]:
- Introduce a new flag, TransactionInfo::readVersionObtainedFromGrvProxy.
- Set this flag to true by default, and clear it when the read version of a
transaction is explicitly set (by using setVersion()).
- Modify getLatestCommitVersions() to not populate "latestCommitVersions" if
this flag is not set. (This will cause storage server to read at the specified
read version.)

- Modify getRange() actor to always use the specified version as the read
version (except when the specified version is latestVersion).

- Modify waitForCommittedVersion(), getRawVersion(), and getConsistentReadVersion()
to update local version vector cache after receiving GetReadVersionReply.

IClientApi.h, IConfigTransaction.h, ISingleThreadTransaction.h,
MultiVersionTransaction[.actor].[h,cpp], ThreadSafeTransaction.[h,cpp],
ApiWorkload.h:
- Add methods to get the spanID of a transaction and also the version vector
cached in a transaction. (Likely to be useful for debugging simulation test
failures.)

VersionVector.h:
- Update "maxVersion" when populating/applying a delta. (Note that empty
mutation messages only update VersionVector::maxVersion.)

BackupWorker.actor.cpp:
- Update local version vector cache after receiving GetReadVersionReply message.

Status.actor.cpp:
- Update local version vector cache and
TransactionInfo::info.readVersionObtainedFromGrvProxy after setting the
read version.
2021-09-16 12:00:26 -04:00
FDB Formatster 2c788c233d apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-08-27 17:07:47 -07:00
Steve Atherton f596a81073 Rename ::TRUE and ::FALSE in BooleanParams to ::True and ::False so as to not conflict with the TRUE and FALSE macros provided by the Windows and MacOS SDKs. 2021-07-17 00:11:40 -07:00
sfc-gh-tclinkenbeard 475abe301c Merge remote-tracking branch 'origin/master' into fix-ub 2021-07-14 10:47:02 -07:00
sfc-gh-tclinkenbeard 8cc40e3a2b Expand use of BOOLEAN_PARAM 2021-07-02 21:41:50 -07:00
sfc-gh-tclinkenbeard 79ff07a071 Added *BOOLEAN_PARAM macros to enforce documentation of boolean parameters 2021-07-02 15:04:42 -07:00
sfc-gh-tclinkenbeard fe5e7f8c2a Remove ISingleThreadTransaction::create function 2021-06-29 10:29:41 -07:00
sfc-gh-tclinkenbeard 41c790b299 Merge remote-tracking branch 'origin/master' into config-db 2021-06-10 22:31:23 -07:00
sfc-gh-tclinkenbeard 371a38e6e5 Merge remote-tracking branch 'origin/master' into remove-extra-copies 2021-06-07 10:26:06 -07:00
sfc-gh-tclinkenbeard 6fc2f1ddcb Fix/remove some TODO comments in configuration database code 2021-06-06 09:27:51 -07:00
sfc-gh-tclinkenbeard 905f0065e5 Store DatabaseContext reference in configuration transaction objects 2021-06-02 22:14:58 -07:00
sfc-gh-tclinkenbeard a775f92fca Merge remote-tracking branch 'origin/master' into config-db 2021-06-01 15:39:34 -07:00
sfc-gh-tclinkenbeard dce15d90d5 Add ThreadSafeDatabase::isConfigDB field 2021-05-10 18:15:56 -07:00
sfc-gh-tclinkenbeard f28ac955c3 Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>> 2021-05-10 16:32:50 -07:00
sfc-gh-tclinkenbeard 121c48359d Revert "Make ThreadSafeTransaction implementation less verbose"
This reverts commit c7d5bbd2e0.
2021-05-10 15:00:01 -07:00
sfc-gh-tclinkenbeard 28f923c087 Remove ISingleThreadTransaction::getMutableDeferredError 2021-05-10 14:09:08 -07:00
sfc-gh-tclinkenbeard 331dd2e377 Replace config_transaction option with use_config_database 2021-05-10 13:18:31 -07:00
Chaoguang Lin 7da5004690 Merge branch 'master' of github.com:apple/foundationdb into refactor-fdbcli 2021-05-06 15:44:26 -07:00
sfc-gh-tclinkenbeard 5c2d7b6080 Create RangeResult type alias 2021-05-03 13:14:16 -07:00
sfc-gh-tclinkenbeard c7d5bbd2e0 Make ThreadSafeTransaction implementation less verbose 2021-04-30 12:05:58 -07:00
sfc-gh-tclinkenbeard 0d9eaa5d6d Add ISingleThreadTransaction.h 2021-04-29 22:31:16 -07:00
Chaoguang Lin 283b774c2f Merge branch 'master' of github.com:apple/foundationdb into refactor-fdbcli 2021-04-29 12:04:54 -07:00
A.J. Beamon 823873a9aa Address review comments:
Use nullptr instead of NULL
Use const& for a parameter
Add some comments
2021-04-26 14:39:27 -07:00
Chaoguang Lin de4753a5db Add a workaround to temporily use the ryw to create a ThreadTransaction; Make sure we are using the same underlying ryw object 2021-04-23 01:32:30 -07:00
A.J. Beamon b2d6930103 The multi-version client monitors the cluster's protocol version and only activates the client library that can connect. 2021-04-15 11:45:14 -07:00
A.J. Beamon 3ed0d614d2 Move fdb_get_server_protocol to be a function on the database object. Add an argument for expected_version that can be used to signal that the function shouldn't return unless the protocol version is different. 2021-04-14 12:50:30 -07:00
A.J. Beamon ed811008ca
Merge pull request #4504 from sfc-gh-nwijetunga/network_busyness
Monitor Network Thread Busyness
2021-03-24 09:44:09 -07:00
Nim Wijetunga de1c354b12 address pr comments 2021-03-23 20:56:37 +00:00
Andrew Noyes 7f4bde74ef Add some documentation 2021-03-18 23:34:44 +00:00
Andrew Noyes 0919eff187 Fix data race in fdb_get_server_protocol 2021-03-18 18:24:12 +00:00
Nim Wijetunga 822cbf895d address pr comments 2021-03-16 23:29:02 +00:00
Nim Wijetunga 0c6b9bc541 remove guard around g_network 2021-03-16 21:53:24 +00:00
Nim Wijetunga 17f9da6962 remove traces 2021-03-16 16:47:50 +00:00
Nim Wijetunga f0d79b3d86 Inital implementation of network busyness 2021-03-15 23:23:56 +00:00
FDB Formatster df90cc89de apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-10 10:18:07 -08:00
Chaoguang Lin 90686a95a0 format code 2021-02-08 22:17:16 -08:00
Chaoguang Lin 7c1ef37bcb format code 2021-02-08 21:42:40 -08:00
Chaoguang Lin 97925de9d4 Update fdb_database_create_snapshot to take a UID as input 2021-02-05 14:14:38 -08:00
Chaoguang Lin cef53b2b40 clang-format 2021-01-26 12:06:40 -08:00
Chaoguang Lin 5f912f0deb Add snapshot c function 2021-01-26 11:53:35 -08:00
Chaoguang Lin 9b5fc3b638 Update code 2021-01-21 01:22:53 -08:00
Chaoguang Lin 116f034933 update var names 2021-01-21 01:11:40 -08:00
Chaoguang Lin 9cd7c43a2c Add c function fdb_database_force_recovery_with_data_loss 2021-01-21 00:42:23 -08:00
Chaoguang Lin e097c98956 Remove headed added by IDE, add comments 2021-01-04 00:18:01 -08:00
Chaoguang Lin da7b65cd71 Change the interface back to return int64_t 2021-01-03 22:45:09 -08:00
Chaoguang Lin b68d84aaea Merge remote-tracking branch 'upstream/master' into add-c-function-for-management-commands 2020-12-21 12:32:31 -08:00
Chaoguang Lin adc916121e Copy the network address stringref 2020-12-20 22:44:04 -08:00
Chaoguang Lin 88e66545bb Change the api to return bool instead of int 2020-12-03 12:01:32 -08:00