Commit Graph

290 Commits

Author SHA1 Message Date
Josh Slocum 40052c1394 Added Blob Granule API Tester 2022-04-20 10:15:43 -05:00
Ray Jenkins 5df3bac110
Migrate to OpenTelemetry tracing. (#6855) 2022-04-20 09:26:37 -05:00
Dan Lambright 1b3b4166c6
Merge branch 'main' into vv 2022-04-08 17:18:13 -04:00
Josh Slocum 6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Dan Lambright c106847e3e
Merge branch 'main' into vv 2022-04-08 15:05:51 -04:00
Dan Lambright 5bdc525353
Merge branch 'main' into vv 2022-04-08 13:16:04 -04:00
Tao Lin 73ed34d2fb Fixes #6793: Upgrade from 7.0.0 to 7.1.0 fails because of missing function fdb_transaction_get_mapped_range 2022-04-08 09:32:18 -07:00
Jon Fu 1a3b5f78c4 trace sevError if no map entry is found 2022-04-08 10:32:24 -04:00
Jon Fu 68a75dd9cc additional safety checks in the upgrade scenario for GRV cache 2022-04-08 10:17:18 -04:00
Jon Fu b0ae22f16d Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-04-06 12:28:28 -04:00
Jon Fu 124aa6e5fd clean up debug traces and fix details with delRef placement 2022-04-06 12:27:39 -04:00
Jon Fu 1b9bd597c5 address code review comments and add some traces for manual testing 2022-04-05 15:29:02 -04:00
Jingyu Zhou 64d4658034 Merge branch 'main' into vv
Fix Conflicts:
	flow/error_definitions.h
2022-04-01 21:49:24 -07:00
Jon Fu c46ad3ce75 Only clear state map if it isn't the initial protocol version change from a null value 2022-04-01 13:51:57 -04:00
A.J. Beamon 68f15650a1 Make sure closed and tenantUpdater are read/written in the same critical section. 2022-03-31 11:12:01 -07:00
Jingyu Zhou 4fd414a8ed Merge branch 'main' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
2022-03-31 09:38:36 -07:00
Jingyu Zhou cfcf0f152c Merge branch 'main-4a085fc84' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
	fdbserver/ClusterRecovery.actor.cpp
	fdbserver/MasterInterface.h
	fdbserver/masterserver.actor.cpp
	flow/error_definitions.h
2022-03-30 22:28:06 -07:00
Jingyu Zhou b34f4052cd Merge branch 'main-f28dfc12b' into vv
Fix Conflicts:
	fdbclient/MultiVersionTransaction.actor.cpp
	fdbclient/NativeAPI.actor.cpp
	fdbclient/NativeAPI.actor.h
	fdbserver/storageserver.actor.cpp
2022-03-30 21:01:25 -07:00
Jingyu Zhou 00b57d4cce Merge branch 'main-67eba5ec7' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbclient/StorageServerInterface.h
	fdbserver/CommitProxyServer.actor.cpp
	fdbserver/storageserver.actor.cpp
2022-03-30 20:05:55 -07:00
Jingyu Zhou e9659b5dd4 Merge branch 'master-PR-6500' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbserver/masterserver.actor.cpp
2022-03-30 14:53:49 -07:00
A.J. Beamon c7d53b31ee Use a TenantState object in the MVC implementation to help manage tenant lifetime. 2022-03-30 12:52:27 -07:00
Jon Fu f266695a76 clear shared state map entries on version upgrade and use currentProtocolVersion instead of piping version through function parameters 2022-03-30 15:28:55 -04:00
Jon Fu 9d0a520fe0 add ProtocolVersion to shared state and state cluster map 2022-03-30 13:28:06 -04:00
Jon Fu 53d252fda0 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-29 17:23:08 -04:00
Jon Fu bdbe78e48f address code review comments 2022-03-29 17:20:22 -04:00
Josh Slocum b92c6ea92c granule parallelism, metrics, and other improvements to bg client bindings 2022-03-24 13:44:09 -05:00
Jon Fu 7c3151142e Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-23 16:41:33 -04:00
Jon Fu 4466e00716 Fix multithreaded GRV cache code and enable grv caching in multithreaded ctest suite 2022-03-23 16:09:12 -04:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Jon Fu 9731d96797 Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache-multi-threaded 2022-03-16 14:32:33 -04:00
Jon Fu 7ab205391b Revert "remove all multi-version client code related to grv cache"
This reverts commit 6d05ecffc2.
2022-03-16 14:26:36 -04:00
A.J. Beamon 8a5107af03 Fix a couple variable initialization issues in the tenant MVC implementation. Encode tenant prefixes with unicode escape sequences for the list command. 2022-03-15 09:23:30 -07:00
A.J. Beamon 68069c9784 Undo some unused changes. Fix tenant cache eviction. 2022-03-15 09:23:30 -07:00
A.J. Beamon 8bc2b283e1 Create a special keys API to create, delete, and read tenants. Remove the C API to create/delete tenants. 2022-03-15 09:23:30 -07:00
A.J. Beamon c89b1e3537 Use TenantName type instead of StringRef in various places. Add a function on transactions to get the tenant name being used for the transaction. 2022-03-15 09:21:27 -07:00
A.J. Beamon c635dcd3ad Add tenant support in the FDB native client 2022-03-15 09:21:27 -07:00
Tao Lin e2c7c30faf
GetMappedRange support serializable & check RYW & continuation (#6181) 2022-03-10 10:05:44 -08:00
A.J. Beamon 250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Sreenath Bodagala 72e06369a4 Merge remote-tracking branch 'apple-upstream/main' into version-vector-prototype 2022-02-08 17:47:57 +00:00
Ray Jenkins 57a073b3b9 Add thread names for release, cancel, destroy and abort. 2022-01-24 19:42:21 -06:00
Dan Lambright 9544379cdf rebase 2022-01-20 11:12:33 -05:00
Josh Slocum 3afe9fb6e0 MVC bug fixes 2021-12-10 12:47:53 -06:00
Josh Slocum b7b2ad0a6f Handling timeout and transaction changes for ThreadResults properly 2021-12-10 12:47:53 -06:00
Josh Slocum c11b8a3625 Improved comments 2021-12-01 17:04:55 -06:00
Josh Slocum a82845af43 Merge branch 'master' into bg_bindings 2021-12-01 16:55:28 -06:00
Josh Slocum 7f4fcc8c2c Added FDBResult and made readBlobGranules use it 2021-12-01 16:22:05 -06:00
Andrew Noyes bccda9f6fe Don't provide default for requireFunction in loadClientFunction
Recently there was an issue where we added a new symbol to the c api and
we did not include the right header version guard to see if it was
required. Let's make this not a default so that we force callers to
think about what the right header versions to require it are.
2021-11-29 14:08:28 -08:00
A.J. Beamon b8bd89f88d Shorten the name of external client threads. Add a thread name for trace logging threads. 2021-11-29 09:57:10 -08:00
A.J. Beamon 264c75b9a6 Add some extra client logging details:
1. Add a trace event when a database is created and move the cluster file / connection string from ClientStart to the new trace event
2. Add a detail for the path to the image being loaded
3. Add a detail for whether a client library is primary or not
4. Set a thread name for each external client thread that includes the release version
2021-11-29 09:57:10 -08:00
Andrew Noyes c6472790e3 Don't require fdb_transaction_get_range_and_flat_map in older clients 2021-11-22 10:49:26 -08:00
A.J. Beamon d129b726e1 Don't use assertion for bad option usage; don't disable local client until setup network is called. 2021-11-12 10:12:49 -08:00
Josh Slocum 329091e14f Merge branch 'master' into bg_bindings 2021-11-11 10:13:37 -06:00
Josh Slocum 77fa1e701a cleanup 2021-11-11 09:43:23 -06:00
Josh Slocum b8ac4213a1 Switched BG APIs to transaction instead of database 2021-11-11 08:59:06 -06:00
Vaidas Gasiunas 51b8ccf7d3 Merge remote-tracking branch 'apple/master' into notify-client-lib-changes 2021-11-10 18:40:34 +01:00
Vaidas Gasiunas d6a5cc8460 MVC2.0: Rollback static initialization of the local client API instance 2021-11-10 15:55:33 +01:00
Tao Lin fdb3b72e35 Introduce GetRangeAndFlatMap to push computations down to FDB
Re-introduce #5609
2021-11-09 13:52:28 -08:00
Vaidas Gasiunas 31386c0f03 MVC2.0: Fix refactoring to remove dependency of MVC on Native API 2021-11-08 17:25:43 +01:00
Tao Lin 586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB" 2021-11-04 08:46:56 -07:00
Tao Lin 6c98e35893 Rename Hop to FlatMap 2021-11-03 13:32:01 -07:00
Tao Lin 0853661d13 Introduce getRangeAndHop to push computations down to FDB 2021-11-03 13:21:16 -07:00
Josh Slocum e3839e198a Cleanup 2021-11-03 13:41:41 -05:00
Vaidas Gasiunas 276dbd4d37 MVC2.0: Refactoring - remove dependency of MVC on Native API 2021-11-03 16:30:18 +01:00
Josh Slocum 382882f1c1 mako successfully calls read_blob_granules and gets stuff back 2021-11-02 13:43:42 -05:00
Josh Slocum d6a31078fe C API for blob granules 2021-11-02 10:01:23 -05:00
Dan Lambright befe1993c4 fix conflict on rebase 2021-10-29 12:25:26 -04:00
A.J. Beamon abab45760d Add some additional logging if the network thread finishes, fails with an error, gets stopped, or is blocked. 2021-10-19 10:05:29 -07:00
Dan Lambright f68d018a06 conflicts 2021-10-08 12:39:39 -04:00
A.J. Beamon a3d53c57b1 Fix a thread-safety issue with timeoutTsav 2021-09-29 11:07:47 -07:00
Sreenath Bodagala 950de29d5a Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype 2021-09-20 21:29:57 +00:00
Chang Liu 110d6f2d24 Enable support for integers in environment variable network options
Description
Description
Previously, the environment setting via shell cannot be extracted correctly, due to the code is't ablle to
determine input parameter type.
Now the logic to determien parameter and give it different processing is added.

Resolves #4127

Testing
1. Pass an invalid env var:
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=xyz - error
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=123xyz) - error
2. Pass an out-of-ranged env var:
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=9999999999999999999999999999 -error
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=-9999999999999999999999999999 -error
3. Pass a valid env var
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=1000 -ok
4. Pass a float env var(not support)
	FDB_NETWORK_OPTION_TRACE_ROLL_SIZE=3.1415926535 - error
2021-09-20 11:48:07 -07:00
Chang Liu 3ece92b93d Enable support for integers in environment variable network options 2021-09-20 11:48:07 -07:00
Chang Liu 1e16ab446f Enable support for integers in environment variable network options 2021-09-20 11:48:07 -07:00
Chang Liu c36758c08a Enable support for integers in environment variable network options 2021-09-20 11:48:07 -07:00
Chang Liu 72e97775be Enable support for integers in environment variable network options 2021-09-20 11:48:07 -07:00
Chang Liu d78e53893d Update for the env variable type check function
Description

Testing
2021-09-20 11:48:07 -07:00
Chang Liu 19cfc12378 Enable multiple environment type input
Description
Previously, the environment setting via shell cannot be extracted correctly, due to the code is't ablle to
determine input parameter type.
Now the logic to determien parameter and give it different processing is added.

Testing
2021-09-20 11:48:07 -07:00
A.J. Beamon d51b93390d Proper handling of reset and transaction destruction. Add some comments. 2021-09-17 18:10:56 -06:00
A.J. Beamon b5fbd07155 Transactions would not honor the transaction timeout option if the MVC did not have an active database. 2021-09-17 18:10:56 -06:00
Sreenath Bodagala 2aa3b44d4e Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
- Conflicts:
	fdbserver/LogSystem.h
	fdbserver/LogSystemConfig.h
	fdbserver/TagPartitionedLogSystem.actor.cpp

- Files modified during merge:

modified:   fdbserver/LogSystem.cpp
modified:   fdbserver/LogSystemConfig.cpp
2021-09-17 19:36:18 +00:00
Sreenath Bodagala ffbdc21ca8 - Address review comments 2021-09-16 12:00:26 -04:00
Sreenath Bodagala 852fc96200 Address simulation test failures caused by:
- Assertion failures in MoveKeys.actor.cpp
- Wrong results returned by getRange()

Changes:

DatabaseContext.h, NativeAPI.actor.[h,cpp]:
- Introduce a new flag, TransactionInfo::readVersionObtainedFromGrvProxy.
- Set this flag to true by default, and clear it when the read version of a
transaction is explicitly set (by using setVersion()).
- Modify getLatestCommitVersions() to not populate "latestCommitVersions" if
this flag is not set. (This will cause storage server to read at the specified
read version.)

- Modify getRange() actor to always use the specified version as the read
version (except when the specified version is latestVersion).

- Modify waitForCommittedVersion(), getRawVersion(), and getConsistentReadVersion()
to update local version vector cache after receiving GetReadVersionReply.

IClientApi.h, IConfigTransaction.h, ISingleThreadTransaction.h,
MultiVersionTransaction[.actor].[h,cpp], ThreadSafeTransaction.[h,cpp],
ApiWorkload.h:
- Add methods to get the spanID of a transaction and also the version vector
cached in a transaction. (Likely to be useful for debugging simulation test
failures.)

VersionVector.h:
- Update "maxVersion" when populating/applying a delta. (Note that empty
mutation messages only update VersionVector::maxVersion.)

BackupWorker.actor.cpp:
- Update local version vector cache after receiving GetReadVersionReply message.

Status.actor.cpp:
- Update local version vector cache and
TransactionInfo::info.readVersionObtainedFromGrvProxy after setting the
read version.
2021-09-16 12:00:26 -04:00
Lukas Joswiak 30e7469b19
Update fdbclient/MultiVersionTransaction.actor.cpp 2021-09-08 17:37:33 -07:00
Chaoguang Lin af5643c998 Fix the issue in CheckStatus where it stuck as we cannot talk to any of the coordinators 2021-09-08 10:05:13 -07:00
A.J. Beamon def5822d42 When using an external client, use the larger of the primary and external busyness. 2021-08-27 10:46:34 -07:00
sfc-gh-tclinkenbeard c74047c665 Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings 2021-07-28 11:51:02 -07:00
Steve Atherton 507c1f11e3 Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use. 2021-07-26 19:55:10 -07:00
sfc-gh-tclinkenbeard 7f727fd616 Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings 2021-07-24 23:18:21 -07:00
sfc-gh-tclinkenbeard 3442ebd3b7 Fix more -Wreorder-ctor warnings across many files 2021-07-24 11:20:51 -07:00
A.J. Beamon 0e9dabcabb Remove mutex that was only needed for a minor optimization. 2021-07-23 10:20:50 -07:00
A.J. Beamon 052e32ae18 Update comment about failure to update version monitor DB 2021-07-23 15:58:29 +00:00
A.J. Beamon 11b803fe0b Handle database creation errors 2021-07-22 13:22:04 -07:00
A.J. Beamon bdb740b7b8 Propogate errors if creating a database on an external client fails 2021-07-21 15:59:23 -07:00
A.J. Beamon 39af41ebb7 Closing a multi-version database causes us to cancel the protocol version monitor, invalidating its future. If a version change is triggered after that happens, then an assertion will be triggered that expects the future to be valid.
This changes the behavior so that closing the database prevents us from doing any work to update the version.
2021-07-19 11:01:09 -07:00
sfc-gh-tclinkenbeard 475abe301c Merge remote-tracking branch 'origin/master' into fix-ub 2021-07-14 10:47:02 -07:00
Daniel Smith 8efe3b296a Delete remaining extern declarations for noUnseed 2021-07-08 19:19:22 -04:00
sfc-gh-tclinkenbeard fe5e7f8c2a Remove ISingleThreadTransaction::create function 2021-06-29 10:29:41 -07:00
Andrew Noyes 7f35a66383 Fix the types of dl functions in the multi-version client
This fixes undefined behavior of invoking a function pointer with the
wrong type.
2021-06-23 10:38:04 -07:00
sfc-gh-tclinkenbeard f28ac955c3 Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>> 2021-05-10 16:32:50 -07:00
sfc-gh-tclinkenbeard 5c2d7b6080 Create RangeResult type alias 2021-05-03 13:14:16 -07:00