Commit Graph

1507 Commits

Author SHA1 Message Date
Josh Slocum c5b2b384da Fixing ASAN issues 2021-12-08 08:42:35 -06:00
Josh Slocum 3ee6d69f3d Wait for empty even for whenAtLatest>0 2021-12-08 08:42:35 -06:00
Evan Tschannen 75267aba80 fix: do not update lastReturnedVersion until the client has consumed the mutations 2021-12-07 13:03:50 -08:00
Evan Tschannen fdd35070a1 fix: merge cursors can have more mutations sent to self->mutations after it goes empty 2021-12-07 09:57:39 -08:00
Andrew Noyes def41697bf
Merge pull request #6083 from sfc-gh-tclinkenbeard/remove-temporaries
Avoid creating unnecessary temporary objects
2021-12-06 13:24:56 -08:00
Josh Slocum 528df86a28 fix: refactored and fixed changeFeedWhenAtLatest returning early 2021-12-06 14:06:17 -06:00
Evan Tschannen 02c650c6e7 fix: sending to mutations can cause whenAtLeast to update lastReturnedVersion before we have a chance to 2021-12-05 16:16:20 -08:00
Evan Tschannen da9c42fd9e do not log transaction metrics in simulation 2021-12-05 14:59:24 -08:00
Evan Tschannen f711ef56a3 fix: whenAtLeast updates lastReturnedVersion 2021-12-03 14:27:11 -08:00
Josh Slocum 2d7c44af31 Fixing bug with readVersionOut being null 2021-12-02 16:42:14 -06:00
Josh Slocum bc16e23d2c Merge branch 'master' into blob_integration 2021-12-02 15:51:18 -06:00
Josh Slocum d85eb330e0 retooling some waitForVersion stuff and adding asserts 2021-12-02 14:52:16 -06:00
Josh Slocum f43169cb7b bug fixes 2021-12-02 12:40:37 -06:00
Josh Slocum 632f701c8a First pass at using ChangeFeedData in blob worker 2021-12-02 11:42:16 -06:00
Josh Slocum 85c6e30aec Fix: several empty version issues with large single-version change feed entries 2021-12-02 09:56:15 -06:00
sfc-gh-tclinkenbeard 3b5d23ef88 Remove unnecessary temporary objects when appending to vector of pairs 2021-12-01 18:15:16 -08:00
Josh Slocum 2c63d0ae03 cleaning up prints 2021-12-01 17:17:19 -06:00
Josh Slocum a82845af43 Merge branch 'master' into bg_bindings 2021-12-01 16:55:28 -06:00
A.J. Beamon 13a4743e76 Check whether connRecord is null before logging it. 2021-11-29 14:22:44 -08:00
Jon Fu dcb0fc82e9 remove future wait optimization with NotifiedDouble and use regular GRV path if cache is too stale 2021-11-29 15:13:48 -05:00
Jon Fu b5b0afb96d narrow down places where to advance cache generation 2021-11-29 14:38:52 -05:00
Jon Fu 1fb537cde5 Add cache generations to create time boundaries when commits time out or fail 2021-11-29 14:16:49 -05:00
A.J. Beamon 264c75b9a6 Add some extra client logging details:
1. Add a trace event when a database is created and move the cluster file / connection string from ClientStart to the new trace event
2. Add a detail for the path to the image being loaded
3. Add a detail for whether a client library is primary or not
4. Set a thread name for each external client thread that includes the release version
2021-11-29 09:57:10 -08:00
Jon Fu 2b0ade5250 Change throttling threshold to count loop iterations instead of time 2021-11-25 13:31:55 -05:00
Jon Fu bba7ed8ac0 Add panic knob to shut down cache from outside txn options 2021-11-25 13:22:28 -05:00
Josh Slocum efb21ca6a1 Merge branch 'master' into blob_integration 2021-11-24 13:17:54 -06:00
Evan Tschannen e3422b2443 fix: change feed streams were not being closed properly 2021-11-23 15:01:44 -08:00
Evan Tschannen 3360000a91 fix: an empty mutation vector did not mean that all of the data was returned 2021-11-23 15:01:44 -08:00
Evan Tschannen 9df287ea06 fixed empty version merge logic 2021-11-23 15:01:44 -08:00
Evan Tschannen 6d9f134bf0 fix: merge needs to update on empty versions to release other mutations blocked by the empty version 2021-11-23 15:01:44 -08:00
Evan Tschannen 014fb2f31c fix: onEmpty must check all merged streams 2021-11-23 15:01:44 -08:00
Evan Tschannen a0afe530f8 protect against the changeFeedStorageData getting destroyed 2021-11-23 15:01:44 -08:00
Evan Tschannen 547c5899f8 whenAtLeast could return an incorrect result when the connection closed 2021-11-23 15:01:44 -08:00
Evan Tschannen 31b2414706 fixed a crash while popping change feeds 2021-11-23 15:01:44 -08:00
Jon Fu 3f24128da4 Merge branch 'master' of github.com:apple/foundationdb into jfu-grv-cache 2021-11-19 14:46:55 -05:00
Jon Fu e9c58d9f86 Check for sustained throttling in the proxy to lower threshold time and avoid false positives 2021-11-19 14:33:06 -05:00
Trevor Clinkenbeard 3d0ed2083c
Merge pull request #6000 from sfc-gh-tclinkenbeard/setup-flow-knobs
Setup global flow knobs in `setupNetwork`
2021-11-18 19:30:26 -08:00
sfc-gh-tclinkenbeard 2613ec7561 Expand use of fmt to get rid of %ld usage 2021-11-17 17:03:32 -08:00
Jon Fu 1e2060a52b add extra tracking in dbcontext to periodically contact GRV proxies 2021-11-17 15:24:37 -05:00
sfc-gh-tclinkenbeard 766a05d33c Merge remote-tracking branch 'origin/master' into add-format-warning 2021-11-17 12:14:01 -08:00
sfc-gh-tclinkenbeard e04526b345 Allow knobs to be set via network options before network is setup 2021-11-16 13:47:37 -08:00
Jon Fu 8714480370 fix some compile issues 2021-11-16 13:59:35 -05:00
Jon Fu 16e551e834 change knob to be an adjustable probability instead of binary on/off 2021-11-16 13:41:19 -05:00
sfc-gh-tclinkenbeard deb0eb85e8 Initialize global flow knobs in setupNetwork 2021-11-16 00:41:34 -08:00
Josh Slocum f140028086 Merge branch 'master' into blob_integration 2021-11-15 15:09:40 -06:00
Jon Fu 416d94d5b0 add some more trace details and remove spammy trace throttling temporarily 2021-11-15 13:45:23 -05:00
Evan Tschannen 94a51e57a5 Merge branch 'master' into feature-changefeed-empty-versions
# Conflicts:
#	fdbclient/StorageServerInterface.h
2021-11-14 19:13:05 -08:00
Evan Tschannen 6909754b21 changefeeds now have a whenAtLeast function for efficiently learning when the version has updated but no mutations have been committed 2021-11-14 19:08:46 -08:00
sfc-gh-tclinkenbeard 62efeb6812 Merge remote-tracking branch 'origin/master' into add-format-warning 2021-11-12 11:50:36 -08:00
Josh Slocum bab8756e17 fixed bugs in small splits 2021-11-11 13:50:19 -06:00
Josh Slocum 329091e14f Merge branch 'master' into bg_bindings 2021-11-11 10:13:37 -06:00
Josh Slocum b8ac4213a1 Switched BG APIs to transaction instead of database 2021-11-11 08:59:06 -06:00
Markus Pilman 0dfb72176e
Merge pull request #5857 from sfc-gh-vgasiunas/notify-client-lib-changes
A mechanism to notify MVC about relevant client library status changes on the cluster
2021-11-11 07:43:20 -07:00
Lukas Joswiak 1da288822f Remove distributed trace database option 2021-11-10 13:33:49 -08:00
Lukas Joswiak 8bc0c3e8a2 Convert distributed trace sampling rate to a database option 2021-11-10 13:33:49 -08:00
Lukas Joswiak c93052121f Fix issue where transaction spans would not be recorded 2021-11-10 13:33:49 -08:00
Vaidas Gasiunas 51b8ccf7d3 Merge remote-tracking branch 'apple/master' into notify-client-lib-changes 2021-11-10 18:40:34 +01:00
Tao Lin fdb3b72e35 Introduce GetRangeAndFlatMap to push computations down to FDB
Re-introduce #5609
2021-11-09 13:52:28 -08:00
Jon Fu e984b2f5d0 remove sevdebug temporarily and add extra trace to verify knob values 2021-11-09 15:13:50 -05:00
Jingyu Zhou 6540bef9ee Clear client VV cache after recovery
This is to prevent the client from sending older VV to the new GRV proxies,
because the client side cache can be stale, i.e., not knowing commits for a tag.

This was found in simulation:
A client has a VV cache of [{0:1,841885664} maxversion: 841885664]; client
issues a transaction commits at 841905696. There is a recovery. The client got
Delta=[ maxversion: -1] back. Client issues a read at 946662688, which uses the
cached version 841885664 < 841905696 , thus reading corrupted data.
2021-11-09 08:37:22 -08:00
Tao Lin 586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB" 2021-11-04 08:46:56 -07:00
Tao Lin 6c98e35893 Rename Hop to FlatMap 2021-11-03 13:32:01 -07:00
Tao Lin 0853661d13 Introduce getRangeAndHop to push computations down to FDB 2021-11-03 13:21:16 -07:00
Josh Slocum e3839e198a Cleanup 2021-11-03 13:41:41 -05:00
Josh Slocum 382882f1c1 mako successfully calls read_blob_granules and gets stuff back 2021-11-02 13:43:42 -05:00
Josh Slocum d6a31078fe C API for blob granules 2021-11-02 10:01:23 -05:00
sfc-gh-tclinkenbeard 13bb7838aa Enable clang -Wformat warning 2021-10-30 21:07:38 -07:00
Dan Lambright befe1993c4 fix conflict on rebase 2021-10-29 12:25:26 -04:00
Vaidas Gasiunas 875824b186 MVC2.0: Notify clients about relevant changes of client libraries 2021-10-27 23:43:40 +02:00
Josh Slocum 19076ad4d2 Changed blob manager checkin to by async and to have blob worker speculatively consume change feed ahead of re-snapshot 2021-10-27 16:23:51 -05:00
Suraj Gupta fd50a011b1 Address PR comments. 2021-10-26 21:39:41 -04:00
Suraj Gupta 5a9d9921d0 Fixes and final cleanup for BM failure handling 2021-10-26 16:16:00 -04:00
Suraj Gupta 17b30f188a Working impl 2021-10-26 16:16:00 -04:00
Josh Slocum 0ff8ddc2b6 Merge branch 'master' into blob_full_clean 2021-10-25 13:38:48 -05:00
Jon Fu 7273d352a2 Add extra debug traces and ignore "IMMEDIATE" priority txns for rk throttling 2021-10-25 13:44:45 -04:00
Evan Tschannen 6f7558b8ea Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
# Conflicts:
#	tests/CMakeLists.txt
2021-10-24 21:06:33 -07:00
Evan Tschannen 0e327d3d0a fix: do not duplicate lastEpochEnd from different servers 2021-10-24 19:17:11 -07:00
Evan Tschannen 4039dbd8da fix: combine mutations from the same version into a single MutationsAndVesionRef 2021-10-24 13:46:45 -07:00
A.J. Beamon e882eb33fc Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem. 2021-10-22 11:05:18 -07:00
Jon Fu 6d74239760 Track throttling by measuring time spent left in queue on the proxy 2021-10-22 13:55:01 -04:00
Josh Slocum 773886515e Merge branch 'feature-range-feed' into blob_full_clean 2021-10-22 11:07:51 -05:00
Evan Tschannen d778cea9b5 fix: prevent infinite loop 2021-10-21 15:16:22 -07:00
Evan Tschannen f1158371a7 Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
# Conflicts:
#	flow/error_definitions.h
2021-10-21 00:55:12 -07:00
Evan Tschannen 4e79296a9f fixed a few bugs with fetching change feeds 2021-10-21 00:44:51 -07:00
Evan Tschannen 3ebabb6edc fixed incorrect use of change feed errors 2021-10-20 22:37:31 -07:00
Jon Fu f1c8d3fbc8 Add code to disable cache when ratekeeper begins throttling 2021-10-20 15:52:43 -04:00
Evan Tschannen 3f7df58a77 fixed a number of issues 2021-10-19 13:56:52 -07:00
A.J. Beamon abab45760d Add some additional logging if the network thread finishes, fails with an error, gets stopped, or is blocked. 2021-10-19 10:05:29 -07:00
Josh Slocum 912ef76f1c cleanup before merge 2021-10-18 17:11:14 -05:00
Sreenath Bodagala 246f035afe Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype 2021-10-18 19:07:33 +00:00
Jon Fu f75c8b25ad add small random delay before returning a cached read version 2021-10-18 13:59:04 -04:00
Suraj Gupta e2e852e515 Mitigate transitive includes. 2021-10-18 10:49:25 -04:00
Josh Slocum b5074fd597 Reworked all of the system data to encode granule data more efficiently for persistence 2021-10-13 16:28:04 -05:00
Josh Slocum 5f0ec0612a Merge branch 'feature-range-feed' into blob_full 2021-10-13 15:44:35 -05:00
Suraj Gupta bde90f058b Gate the client API. 2021-10-13 13:26:27 -04:00
Yao Xiao e543570343
Add locality cache size to TransactionMetrics. (#5758)
Co-authored-by: Yao Xiao <yaoxiao@Yaos-MacBook-Pro.local>
2021-10-12 13:10:15 -07:00
Evan Tschannen 282f84c807 added handling for broken promise 2021-10-12 10:36:10 -07:00
Markus Pilman 5066d67dec
Merge pull request #5715 from sfc-gh-ljoswiak/fixes/tracing-various
Add knob to control distributed trace recording percentage
2021-10-11 14:03:53 -06:00
Evan Tschannen d51edf18dc fixed merge conflicts 2021-10-09 19:47:24 -07:00
Evan Tschannen 5c642f706e Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
# Conflicts:
#	fdbcli/fdbcli.actor.cpp
2021-10-09 19:34:16 -07:00
Evan Tschannen efc4cec53f fixed a variety of bugs with change feeds 2021-10-09 19:24:01 -07:00
Markus Pilman 4fbc193e19 fix priority 2021-10-09 16:40:24 -06:00
Markus Pilman 9077e15b11 Address review comments 2021-10-09 16:40:24 -06:00
Markus Pilman 4e723ba3d9 Apply suggestions from code review
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2021-10-09 16:40:24 -06:00
Markus Pilman ffdba4a133 assert we never get a stale read version 2021-10-09 16:40:24 -06:00
Jon Fu 2b2d059039 Specify option to specifically skip grv cache for the background updater when using debug knob 2021-10-08 17:32:44 -04:00
Jon Fu b770936cde move background updater actor initialization 2021-10-08 16:38:21 -04:00
Jon Fu ecdafba336 add debug traces to background grv updater 2021-10-08 15:17:28 -04:00
Dan Lambright f68d018a06 conflicts 2021-10-08 12:39:39 -04:00
Yao Xiao efd75ccc74
Add histograms to client GRV batcher. (#5727) 2021-10-07 13:30:25 -07:00
Josh Slocum 31664ddb78 Fixing change feed error retry in pop 2021-10-07 10:07:18 -05:00
Josh Slocum 54d5e8366d fixing another change feed hang error 2021-10-06 16:04:01 -05:00
Jon Fu cc38c48123 Add debug knob to use grv cache for performance testing 2021-10-06 14:35:42 -04:00
Josh Slocum bbe7c7ca9f found change feed liveness bug 2021-10-06 12:32:16 -05:00
Jon Fu 9a18cc8f41 Change debug time bound update logic to be located in masterserver 2021-10-05 17:31:31 -04:00
Jon Fu 44a854772f Merge branch 'master' of github.com:apple/foundationdb into jfu-grv-cache 2021-10-05 12:55:02 -04:00
Jon Fu ae31f01dc4 remove cache update from extractreadversion and set lower time bound to 0 2021-10-05 12:54:22 -04:00
Jon Fu d560eb1fea debug time bounds using sim_validation 2021-10-04 14:12:31 -04:00
Lukas Joswiak e034f66fc2 Add knob to control trace recording percentage 2021-10-01 16:30:46 -07:00
Josh Slocum fa1fe5f08b added blob worker rollbacks that handle (most) cases 2021-09-24 17:52:36 -05:00
Josh Slocum a986679b7f More bug fixes 2021-09-24 10:02:02 -05:00
A.J. Beamon 32a834b0c2 The computation of tag throttling backoff was taking a minimum with a knob for how long a throttle lasts. When this knob was buggified to 0 in simulation, the backoff would then be 0, and this could cause some transactions to get stuck in a retry loop that failed to advance time. Eventually the test would timeout or potentially run out of memory. 2021-09-23 10:03:57 -07:00
Jon Fu 4cd5d138ff Fixed segfault involving missing futures and infinite looping in background actor 2021-09-22 13:13:35 -04:00
Vaidas Gasiunas 16171d8252 Refactoring well-known endpoint registration
- List all well-known endpoints of FDB in a single enum
- Identify well-known endpoints by plain IDs
2021-09-21 11:05:31 -06:00
Sreenath Bodagala d4f9a09a4f - Address test failures 2021-09-17 21:21:44 +00:00
Sreenath Bodagala 2aa3b44d4e Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
- Conflicts:
	fdbserver/LogSystem.h
	fdbserver/LogSystemConfig.h
	fdbserver/TagPartitionedLogSystem.actor.cpp

- Files modified during merge:

modified:   fdbserver/LogSystem.cpp
modified:   fdbserver/LogSystemConfig.cpp
2021-09-17 19:36:18 +00:00
Xiaoge Su abf73047ca Enforce std:: specifier rather than using namespace 2021-09-16 19:40:28 -07:00
Sreenath Bodagala 35621a255e - Address review comments 2021-09-16 12:00:26 -04:00
Sreenath Bodagala ffbdc21ca8 - Address review comments 2021-09-16 12:00:26 -04:00
Sreenath Bodagala 852fc96200 Address simulation test failures caused by:
- Assertion failures in MoveKeys.actor.cpp
- Wrong results returned by getRange()

Changes:

DatabaseContext.h, NativeAPI.actor.[h,cpp]:
- Introduce a new flag, TransactionInfo::readVersionObtainedFromGrvProxy.
- Set this flag to true by default, and clear it when the read version of a
transaction is explicitly set (by using setVersion()).
- Modify getLatestCommitVersions() to not populate "latestCommitVersions" if
this flag is not set. (This will cause storage server to read at the specified
read version.)

- Modify getRange() actor to always use the specified version as the read
version (except when the specified version is latestVersion).

- Modify waitForCommittedVersion(), getRawVersion(), and getConsistentReadVersion()
to update local version vector cache after receiving GetReadVersionReply.

IClientApi.h, IConfigTransaction.h, ISingleThreadTransaction.h,
MultiVersionTransaction[.actor].[h,cpp], ThreadSafeTransaction.[h,cpp],
ApiWorkload.h:
- Add methods to get the spanID of a transaction and also the version vector
cached in a transaction. (Likely to be useful for debugging simulation test
failures.)

VersionVector.h:
- Update "maxVersion" when populating/applying a delta. (Note that empty
mutation messages only update VersionVector::maxVersion.)

BackupWorker.actor.cpp:
- Update local version vector cache after receiving GetReadVersionReply message.

Status.actor.cpp:
- Update local version vector cache and
TransactionInfo::info.readVersionObtainedFromGrvProxy after setting the
read version.
2021-09-16 12:00:26 -04:00
Josh Slocum 6673540553 TSS Metrics fix 2021-09-14 13:15:05 -05:00
Jon Fu 87fac6e301 adjust to changed function signature after merge 2021-09-13 13:04:55 -04:00
Jon Fu d971b63bbf Merge branch 'master' of github.com:apple/foundationdb into jfu-grv-cache 2021-09-13 12:47:42 -04:00
Jon Fu cfbe5eca9c Use notifieddouble to track GRV request time and include grvdelay in timing calculations 2021-09-13 11:57:19 -04:00
Josh Slocum cefb66d64c fixing a couple bugs 2021-09-10 12:52:33 -05:00
Josh Slocum c2d1d1704f Merge branch 'feature-range-feed' into blob_full 2021-09-10 11:21:52 -05:00
Evan Tschannen ac5b580e2d Merge branch 'master' into feature-range-feed
# Conflicts:
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/StorageServerInterface.cpp
#	fdbclient/StorageServerInterface.h
#	fdbserver/ApplyMetadataMutation.cpp
#	fdbserver/TLogServer.actor.cpp
#	flow/error_definitions.h
2021-09-09 23:13:22 -07:00
Jon Fu 1ce2457f7e initial commit 2021-09-09 14:01:42 -04:00
Josh Slocum eb76343dfb Added blob granule reassignment and splitting 2021-09-08 14:09:14 -05:00
Evan Tschannen d9782bc29d fixed a few bugs with merging change feeds on the client 2021-09-07 21:35:00 -07:00
Evan Tschannen f53ddf6720 fix: handle the case where I change feed is added while a fetch is in progress 2021-09-06 13:51:53 -07:00
Evan Tschannen af0792b1f5 fixes that allow change feeds to pass correctness in tests with no failures 2021-09-03 17:24:33 -07:00
Evan Tschannen b02e8d99c6 added a workload to test change feeds 2021-09-03 14:19:27 -07:00
Evan Tschannen 8309c2ce26 filter mutations in change feeds to the requested range 2021-09-03 12:06:13 -07:00
A.J. Beamon afd8df4bb9 Merge commit '7f2b6c2c7ec2efcd7339e3c6531e4906322477c3' into network-thread-busyness-burstiness-score 2021-09-03 09:48:11 -07:00
Evan Tschannen 4bbae59bb0 Fixed a few bugs and added a backup popping mechanism 2021-09-02 21:11:44 -07:00
Evan Tschannen 27151b0831 support reads from range feeds that span multiple storage servers 2021-09-01 14:35:51 -07:00
Josh Slocum 46adada5ff Cleaned up debugging and fixed a couple bugs 2021-08-31 12:30:43 -05:00
A.J. Beamon 2fa97f60fe
Update fdbclient/NativeAPI.actor.cpp
Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
2021-08-31 08:53:37 -07:00
Dan Lambright 8689e1f106 merge with master 2021-08-30 15:29:08 -04:00
FDB Formatster 2c788c233d apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-08-27 17:07:47 -07:00
A.J. Beamon 2d6fb93316 Compute a burstiness score based on the max busyness interval in a measurement period. 2021-08-27 16:29:15 -07:00
Josh Slocum 8d49c98a41 Added simulation workload for blob granules and fixed some bugs 2021-08-26 13:48:05 -05:00
Josh Slocum 714aca4f3c Refactored blob granule reads to actually be in NativeAPI 2021-08-25 15:01:38 -05:00
Chaoguang Lin b9628bdb50 Merge branch 'master' of github.com:apple/foundationdb into add-flow-db-interface 2021-08-17 20:24:41 +00:00
Chaoguang Lin 53e83cf653 Remove change to Database 2021-08-17 20:10:43 +00:00
Chaoguang Lin 179c313e08 Use DatabaseContext instead of Database in template functions 2021-08-17 20:02:25 +00:00
hao fu 925c4a935d Log size of locationCache in client
We observed that multi-threaded client has more memory usage,
and we would like to understand it better.

One suspect is that this shard-to-SS cache is playing a role,
logging the size of it to get better vision.
2021-08-16 10:31:14 -07:00
Evan Tschannen a278d2977a renamed range feeds to change feeds, data distribution support almost complete 2021-08-13 14:27:15 -07:00
Josh Slocum e444d3781c Various TSS improvements from snowblower testing 2021-08-13 10:24:15 -05:00
Andrew Noyes 28f3ffd2ec Fix a crash in debug mode
For unknown reasons, The previous implementation crashed while
(according to valgrind and ubsan) trying to interpret uninitialized memory as a
shared_ptr. This only happened when CMAKE_BUILD_TYPE=Debug. This change
works for debug mode and also still addresses this TSAN issue the
original change addressed.
2021-08-12 16:19:20 -07:00
Lukas Joswiak a605fb3852
Merge pull request #5026 from sfc-gh-ljoswiak/fixes/alp6
Actor sampling
2021-08-11 13:44:17 -07:00
Sreenath Bodagala 2de3a1947e - Address an assertion failure in waitForVersion()
Specify the correct "readVersion" when finding the latest commit versions
of storage server replicas (which will serve the read request).
2021-08-11 15:36:35 +00:00
Evan Tschannen 94239098a5 added additional rangefeed commands to fdbcli 2021-08-10 23:59:05 -07:00
Evan Tschannen 7891477a7c added support for streaming range feeds 2021-08-10 21:28:25 -07:00
Chaoguang Lin 618c72f7e2 Draft to use template to rewrite ThrottleApi 2021-08-11 01:07:36 +00:00
Steve Atherton e5e8a56b66 Merge branch 'master' of https://github.com/apple/foundationdb into debugging-cleanup
# Conflicts:
#	fdbbackup/FileDecoder.actor.cpp
2021-08-09 23:36:57 -07:00
Evan Tschannen ed28aecde0 Merge branch 'master' into feature-range-feed 2021-08-09 20:40:55 -07:00
Evan Tschannen 42ae870c84 added support for querying specific range feed versions 2021-08-09 20:39:28 -07:00
Andrew Noyes ecc078a42f Fix TSAN warnings
Avoid doing thread-unsafe things in the ReadYourWritesTransaction
default constructor since we now will be calling it in a foreign thread.

Here's the warning we're fixing:

Write of size 8 at 0x7b6800000398 by thread T1:
    #0 N2::Net2::updateNow() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/flow/Net2.actor.cpp:192:33 (libfdb_c.so+0xe74568)
    #1 N2::Net2::run() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/flow/Net2.actor.cpp:1459:3 (libfdb_c.so+0xe74568)
    #2 runNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/NativeAPI.actor.cpp:2036:13 (libfdb_c.so+0x575004)
    #3 ThreadSafeApi::runNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ThreadSafeTransaction.cpp:445:3 (libfdb_c.so+0xcda9cc)
    #4 MultiVersionApi::runNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/MultiVersionTransaction.actor.cpp:1741:20 (libfdb_c.so+0x4eab8d)
    #5 fdb_run_network /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/fdb_c.cpp:130:2 (libfdb_c.so+0x4bb425)
    #6 decltype(std::__1::forward<int (*)()>(fp)()) std::__1::__invoke<int (*)()>(int (*&&)()) /usr/local/bin/../include/c++/v1/type_traits:3899:1 (fdb_c_unit_tests+0x389816)
    #7 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (fdb_c_unit_tests+0x389816)
    #8 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (fdb_c_unit_tests+0x389816)

  Previous read of size 8 at 0x7b6800000398 by main thread:
    #0 N2::Net2::now() const /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/flow/Net2.actor.cpp:160:39 (libfdb_c.so+0xe7ea5d)
    #1 Span::Span(UID, Location, std::initializer_list<UID> const&) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../flow/Tracing.h:38:41 (libfdb_c.so+0x5837b4)
    #2 Transaction::Transaction() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/NativeAPI.actor.cpp:4092:64 (libfdb_c.so+0x5837b4)
    #3 ReadYourWritesTransaction::ReadYourWritesTransaction() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ReadYourWrites.h:134:2 (libfdb_c.so+0x7cb4a8)
    #4 ISingleThreadTransaction::allocateOnForeignThread(ISingleThreadTransaction::Type) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ISingleThreadTransaction.cpp:29:17 (libfdb_c.so+0xcf785f)
    #5 ThreadSafeTransaction::ThreadSafeTransaction(DatabaseContext*, ISingleThreadTransaction::Type) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ThreadSafeTransaction.cpp:149:23 (libfdb_c.so+0xcc9540)
    #6 ThreadSafeDatabase::createTransaction() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ThreadSafeTransaction.cpp:50:37 (libfdb_c.so+0xcc4b1c)
    #7 fdb_database_create_transaction /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/fdb_c.cpp:332:2 (libfdb_c.so+0x4bce1d)
    #8 fdb::Transaction::Transaction(FDB_database*) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/test/unit/fdb_api.cpp:111:24 (fdb_c_unit_tests+0x3925d0)
    #9 _DOCTEST_ANON_FUNC_10() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/test/unit/unit_tests.cpp:250:19 (fdb_c_unit_tests+0x32a7a8)
    #10 doctest::Context::run() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/doctest/src/doctest/doctest/doctest.h:6112:21 (fdb_c_unit_tests+0x3262e4)
    #11 main /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/test/unit/unit_tests.cpp:2286:20 (fdb_c_unit_tests+0x36303d)
2021-08-09 11:16:52 -07:00
Evan Tschannen bc9a0e1315 first attempt to add data distribution support for range feeds 2021-08-09 10:05:56 -07:00
Steve Atherton 2bf7b9b7a9 Remove many calls to .toString(), .printable(), and printable(...) from TraceEvent .detail() values to avoid double-escaping characters such that the same values look different in different events and can't be easily searched for. Also fixed a few warnings found by clang. 2021-08-08 00:04:54 -07:00
Sreenath Bodagala a081c0baa5 Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype 2021-08-05 22:40:32 +00:00
Andrew Noyes 39eff8c569 Add trace_partial_file_suffix network option 2021-08-03 09:43:42 -07:00
Lukas Joswiak 5dc9a97230 Merge branch 'master' into fixes/alp6 2021-08-01 20:42:52 -07:00
Evan Tschannen 0989c28a6b made range feeds durable on the storage server 2021-07-30 15:23:42 -07:00
Evan Tschannen e9409b02fe fixed the build 2021-07-28 14:02:12 -07:00
sfc-gh-tclinkenbeard c74047c665 Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings 2021-07-28 11:51:02 -07:00
A.J. Beamon 9f571255bd
Merge pull request #5271 from sfc-gh-ajbeamon/do-not-reset-committed-transaction
Do not partially reset a transaction when it is committed or fails to commit with an error
2021-07-28 09:02:54 -07:00
Evan Tschannen 406562b282 Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed 2021-07-27 17:04:58 -07:00
Lukas Joswiak 3eed4084e2 Merge branch 'master' into fixes/alp6 2021-07-27 11:26:53 -07:00
Lukas Joswiak 59d535149e Merge branch 'master' into fixes/alp6 2021-07-27 10:07:18 -07:00
Lukas Joswiak e9a1679467 Disable sampling everywhere except fdbserver 2021-07-27 09:53:23 -07:00
Steve Atherton 507c1f11e3 Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use. 2021-07-26 19:55:10 -07:00
A.J. Beamon 34f82e7a15 Do not partially reset a transaction when it is committed or fails to commit with an error. 2021-07-26 09:51:44 -07:00
sfc-gh-tclinkenbeard 64dc1dc185 Fix -Wreorder-ctor warnings in NativeAPI.actor.cpp and several other files 2021-07-24 00:23:06 -07:00
Dan Lambright 1bb01ef591 use version vector in GetRange 2021-07-22 10:49:08 -04:00
sfc-gh-tclinkenbeard 6f81155784 Merge remote-tracking branch 'origin/master' into const-serverdbinfo 2021-07-20 10:18:40 -07:00
Steve Atherton f596a81073 Rename ::TRUE and ::FALSE in BooleanParams to ::True and ::False so as to not conflict with the TRUE and FALSE macros provided by the Windows and MacOS SDKs. 2021-07-17 00:11:40 -07:00
Sreenath Bodagala 81001edb2e - Propagate version vector deltas between processes 2021-07-15 19:49:20 +00:00
Sreenath Bodagala b6f89df060 - Propagate the latest commit version of a storage server as part of read request.
Make storage server read at the specified version.
2021-07-15 19:48:42 +00:00
sfc-gh-tclinkenbeard 4f853b19a6 More const-correctness improvements for Reference<AsyncVar<T>> objects 2021-07-12 21:28:38 -07:00
Evan Tschannen 15fe00927a Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
# Conflicts:
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/StorageServerInterface.h
#	fdbclient/SystemData.cpp
#	fdbclient/SystemData.h
#	fdbserver/storageserver.actor.cpp
#	flow/ProtocolVersion.h
2021-07-12 19:10:14 -07:00
Andrew Noyes 4059215180 Value should not be present for ENABLE_RUN_LOOP_PROFILING
This fixes an unintentional change from
79ff07a071. I reviewed other options
changed in that commit and this was the only one of its kind.
2021-07-12 16:07:56 -07:00
sfc-gh-tclinkenbeard 77cd95473d Merge remote-tracking branch 'origin/master' into const-serverdbinfo 2021-07-12 11:00:22 -07:00
Josh Slocum 02f243c821
Merge pull request #5110 from sfc-gh-jslocum/tss_streaming_master
Added Streaming Read comparison for TSS
2021-07-12 10:44:15 -07:00
sfc-gh-tclinkenbeard 6c1d913ab8 Prevent masterServer from modifying db 2021-07-11 21:11:21 -07:00
Sreenath Bodagala baceacacf4 - Maintain a cache of the version vector on both the Grv proxy and
the client. Update these caches on the (transaction) read path.
2021-07-07 19:35:26 +00:00
sfc-gh-tclinkenbeard ccffd6f47f Fix option validation for DISABLE_CLIENT_STATISTICS_LOGGING 2021-07-04 19:06:39 -07:00
sfc-gh-tclinkenbeard 8cc40e3a2b Expand use of BOOLEAN_PARAM 2021-07-02 21:41:50 -07:00
sfc-gh-tclinkenbeard 79ff07a071 Added *BOOLEAN_PARAM macros to enforce documentation of boolean parameters 2021-07-02 15:04:42 -07:00
Josh Slocum 9f1afbb709 Added Streaming Read comparison for TSS 2021-07-01 19:00:05 -05:00
Josh Slocum ab0ab765c8
Merge pull request #4965 from sfc-gh-jslocum/tss_mapping_changes
TSS Mismatch Changes
2021-06-29 08:24:25 -07:00
Neethu Haneesha Bingi 73752f441b exclude locality:clang-format, ranged loops, documentation, tracking addStoragesever for exclusion. 2021-06-23 18:03:27 -07:00
Neethu Haneesha Bingi 62355571d0 exclude servers based on locality match 2021-06-23 18:03:27 -07:00
Evan Tschannen 244e8f3527 Merge branch 'master' of https://github.com/apple/foundationdb into feature-streaming-reply-master
# Conflicts:
#	fdbclient/CMakeLists.txt
#	fdbclient/Knobs.h
#	fdbserver/Knobs.h
2021-06-21 16:35:21 -07:00
Evan Tschannen 1cec223434 cherry pick streaming replies to master 2021-06-19 09:47:13 -07:00
Lukas Joswiak 2d248a5926 Collect actors waiting on disk and network 2021-06-17 16:50:34 -07:00
sfc-gh-tclinkenbeard 826e70916c Merge remote-tracking branch 'origin/master' into config-db 2021-06-17 09:47:41 -07:00
Lukas Joswiak 805c4200ce Add collector to get running actor name 2021-06-16 18:08:51 -07:00
RenxuanW fe936207a9 Replace lower priority txn request when limit is hit. 2021-06-15 14:00:06 -07:00
Lukas Joswiak 0ff041222a Add removals back 2021-06-13 18:23:59 -07:00
Lukas Joswiak 112be7a763 Remove most work 2021-06-13 16:19:02 -07:00
Josh Slocum 56dadaa428 TSS Mismatch Changes 2021-06-11 23:13:16 +00:00
sfc-gh-tclinkenbeard 41c790b299 Merge remote-tracking branch 'origin/master' into config-db 2021-06-10 22:31:23 -07:00
sfc-gh-tclinkenbeard c272304e60 Manage global flow knobs with global knob collection 2021-06-09 22:33:00 -07:00
Josh Slocum c6d96c4124 Added TSS Quarantine and fixed TSS mapping bug in CommitProxy 2021-06-09 14:42:50 +00:00
Xiaoxi Wang 838d847d4e
Merge pull request #4860 from sfc-gh-xwang/ppwtest
implement perpetual storage wiggling feature
2021-06-04 16:18:39 -07:00
Lukas Joswiak 486a04659f Lazy inititialization 2021-06-04 15:01:18 -07:00
Josh Slocum 9b36f69b8d
Merge pull request #4892 from sfc-gh-jslocum/tss_mappingv2
TSS Mapping Change
2021-06-04 14:57:51 -07:00
Lukas Joswiak 153de33f57 Revert "Merge pull request #4802 from sfc-gh-ljoswiak/revert/actor-lineage"
This reverts commit 6499fa178e, reversing
changes made to 1512631957.
2021-06-04 13:31:55 -07:00
Lukas Joswiak ba25b95c6a Fix global config not updating on server processes 2021-06-03 14:27:24 -07:00
Xiaoxi Wang 365b0bc4d8 better api 2021-06-03 19:57:44 +00:00
sfc-gh-tclinkenbeard 89dadf1475 Create IKnobCollection interface 2021-06-02 23:40:52 -07:00
Xiaoxi Wang 351325b3af comment modification; wait perpetual wiggling close 2021-06-03 05:13:20 +00:00
Josh Slocum b3e4f182ef TSS Mapping Change 2021-06-02 17:30:09 +00:00
sfc-gh-tclinkenbeard 4c059670a0 Parse configuration knob values on the client 2021-06-02 10:04:46 -07:00
sfc-gh-tclinkenbeard a775f92fca Merge remote-tracking branch 'origin/master' into config-db 2021-06-01 15:39:34 -07:00
Xiaoxi Wang 8b9c8b33fc manually merge with master 2021-06-01 17:51:42 +00:00
Josh Slocum d67184163b
Merge pull request #4556 from sfc-gh-jslocum/tss
Testing Storage Server
2021-06-01 09:11:10 -07:00
A.J. Beamon 750901dd1d Reduce the frequency that buggified reads are failed so that transactions with a lot of reads aren't doomed to almost always fail. 2021-05-27 21:54:59 -07:00
Xiaoxi Wang 24c0c3361a change quietdatabase 2021-05-25 21:20:24 +00:00
Josh Slocum 4257ac2b4d More TSS Changes/Fixes 2021-05-25 20:37:48 +00:00
Josh Slocum ce82c9653e Testing Storage Server implementation 2021-05-25 20:28:50 +00:00
Evan Tschannen 6165af81fa Merge branch 'apple:master' into feature-range-feed 2021-05-21 15:21:34 -07:00
sfc-gh-tclinkenbeard bb61a20851 Improve const-correctness of ISingleThreadTransaction 2021-05-15 00:23:21 -07:00
Lukas Joswiak 4ea760b2a9 Revert "Merge pull request #4136 from sfc-gh-mpilman/features/actor-lineage"
This reverts commit da41534618, reversing
changes made to e6300905d6.
2021-05-10 20:26:12 -07:00
sfc-gh-tclinkenbeard 331dd2e377 Replace config_transaction option with use_config_database 2021-05-10 13:18:31 -07:00
Lukas Joswiak cdf98f987d
Revert "Fix global config not triggering changes on server processes" 2021-05-10 12:18:28 -07:00
Lukas Joswiak 71ba6f4501 Switch std::function to AsyncTrigger 2021-05-07 16:05:03 -07:00
Lukas Joswiak 0fc1552683 Fix global config updates not triggering on fdbserver processes 2021-05-06 15:46:34 -07:00
sfc-gh-tclinkenbeard 5c2d7b6080 Create RangeResult type alias 2021-05-03 13:14:16 -07:00
Lukas Joswiak 8dcd779fc4
Merge branch 'master' into features/actor-lineage 2021-05-02 14:11:42 -07:00
Evan Tschannen 2621c7153a clang format PR 2021-04-30 10:41:35 -07:00
Evan Tschannen d652287bae Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
# Conflicts:
#	fdbclient/DatabaseContext.h
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/StorageServerInterface.h
#	fdbserver/storageserver.actor.cpp
2021-04-30 10:29:42 -07:00
Markus Pilman f513543305 fix special keyspace register 2021-04-28 16:35:09 -06:00
Andrew Noyes 904a39e473
Merge pull request #4667 from sfc-gh-ajbeamon/feature-mvc-monitor-protocol-version
Use fewer connections in the multi-version client
2021-04-28 14:13:17 -07:00
Markus Pilman f5b0008346 Merge branch 'features/actor-lineage' into features/actor-lineage-fluentd 2021-04-28 09:31:20 -06:00
Markus Pilman 54919d4f3b Merge remote-tracking branch 'sfc/features/actor-lineage' into features/actor-lineage 2021-04-28 09:22:14 -06:00
Markus Pilman 4fab2ecd30 Merge remote-tracking branch 'origin/master' into features/actor-lineage 2021-04-28 09:20:54 -06:00
Markus Pilman 2d6fafde64 Implemented configuration 2021-04-27 10:26:42 -06:00
A.J. Beamon 823873a9aa Address review comments:
Use nullptr instead of NULL
Use const& for a parameter
Add some comments
2021-04-26 14:39:27 -07:00
A.J. Beamon a794fca932 Support 5.0 (and earlier) client versions by adding GRV probing for old versions. Update the C bindings implementation of get_server_protocol to convert the ProtocolVersion object into a uint64_t. Rename a misleading protocol version alias. 2021-04-23 15:00:21 -07:00
Lukas Joswiak 52bba82e8e Add window size configuration key 2021-04-23 14:05:05 -07:00
Markus Pilman a6a8a97e1f Merge remote-tracking branch 'sfc/features/actor-lineage' into features/actor-lineage-instrument 2021-04-22 17:50:36 -06:00
Lukas Joswiak 15336ca274 Add callback for specific global configuration key changes 2021-04-20 17:51:38 -07:00
A.J. Beamon eaaae2e16d Merge branch master into 'feature-mvc-monitor-protocol-version' 2021-04-20 15:07:02 -07:00
Markus Pilman d76b32da18 Annotate read paths on the server side 2021-04-20 15:10:01 -06:00
Chaoguang Lin b34825a0e6 Merge branch 'master' of github.com:apple/foundationdb into add-dd-and-maintenance 2021-04-19 14:52:10 -07:00
Markus Pilman 51ce278d2d
Merge branch 'features/actor-lineage' into features/actor-lineage-collector 2021-04-19 12:55:29 -06:00
Markus Pilman 09ddcb3bae remove old sample thread 2021-04-19 11:55:35 -06:00
Markus Pilman 7307750e5e Merge remote-tracking branch 'origin/master' into features/actor-lineage 2021-04-19 11:29:52 -06:00
Markus Pilman c8e9e2fab4 Merge remote-tracking branch 'sfc/features/actor-lineage' into features/actor-lineage-collector 2021-04-16 17:33:40 -06:00
Lukas Joswiak 551268b0f2 Add well known endpoint for worker communication 2021-04-15 13:50:50 -07:00
A.J. Beamon 486260e944 Fix infinite loop with stable interface protocol monitoring. Fix case where getting an error with a network option didn't properly terminate the database connection. Reduce option lock critical section. 2021-04-15 13:36:31 -07:00
A.J. Beamon b2d6930103 The multi-version client monitors the cluster's protocol version and only activates the client library that can connect. 2021-04-15 11:45:14 -07:00
Markus Pilman d51f94f16e
Merge pull request #4330 from sfc-gh-ljoswiak/features/global-configuration
Add global configuration framework implementation
2021-04-14 16:10:08 -06:00
A.J. Beamon 3ed0d614d2 Move fdb_get_server_protocol to be a function on the database object. Add an argument for expected_version that can be used to signal that the function shouldn't return unless the protocol version is different. 2021-04-14 12:50:30 -07:00
Lukas Joswiak 7de23918c0 Add comments, fix erase bug, make optimizations 2021-04-14 10:56:33 -07:00
Lukas Joswiak 6de28dd916 clang-format 2021-04-14 10:56:33 -07:00
Lukas Joswiak 4a799baa1d Add clear range for global configuration 2021-04-14 10:56:32 -07:00
Lukas Joswiak c3f68831af Move existing ClientDBInfo variables to global configuration 2021-04-14 10:56:32 -07:00
Lukas Joswiak f1415412f1 Add global configuration framework implementation 2021-04-14 10:56:32 -07:00
A.J. Beamon eab468fecc Remove extra line caused by commit issue 2021-04-14 09:32:48 -07:00
Markus Pilman 1d362d8869 Merge remote-tracking branch 'origin/master' into features/actor-lineage 2021-04-14 09:55:04 -06:00
A.J. Beamon feede1d2f6 Fix line length of test macro + comments to be within the 120 character limit 2021-04-13 10:48:52 -07:00
Markus Pilman ed8d43cb87
Merge pull request #5 from sfc-gh-ljoswiak/features/network-actors
Sample actors waiting on network
2021-04-09 15:07:53 -06:00
Lukas Joswiak 5c79d29140 Use object lifetimes instead of function calls 2021-04-07 10:59:45 -07:00
Lukas Joswiak c481ba2cfa Update annotation class name 2021-04-06 17:32:02 -07:00
Lukas Joswiak 433872e17d Sample actors waiting on network 2021-04-06 17:28:28 -07:00
Chaoguang Lin 747fa2a353 Merge branch 'master' of github.com:apple/foundationdb into add-dd-and-maintenance 2021-03-26 12:22:37 -07:00
Chaoguang Lin 0eff74f205 Add special keys for maintenance and datadistribution 2021-03-26 12:19:33 -07:00
Nim Wijetunga de1c354b12 address pr comments 2021-03-23 20:56:37 +00:00
Nim Wijetunga 73571ad52e Merge branch 'master' of github.com:sfc-gh-nwijetunga/foundationdb into network_busyness
* 'master' of github.com:sfc-gh-nwijetunga/foundationdb: (55 commits)
  Port lost changes from #4004
  Fix markdown
  Factor our refreshTransaction gadget
  Align FastAllocator memory to 4096 for size >= 4096
  Fix heap-use-after-free's
  Disable lsan in joshua
  Manually apply changes made to BackupContainer.actor.cpp on release-6.3 to the new files where that code is located.
  add release branch instructions
  Fix PR number in release note link
  fixed a simulation bug where a process on an unreliable machine would be considered reliable by the simulator
  Add tini as PID 1 to docker image (#4363)
  Fix typo
  Fix broken_promise bug
  Explain FastAllocator<4096> special case
  Inform LSAN that we have unaligned pointers
  apply clang-format to flow/Platform.h
  Fix typos
  Address review comments
  Change the macro that forbids exit() calls to be a static assertion
  Add versionstamp serialization information
  ...
2021-03-18 18:58:43 +00:00
Nim Wijetunga 822cbf895d address pr comments 2021-03-16 23:29:02 +00:00
Nim Wijetunga bf5f83d2bf address pr comments 2021-03-16 21:49:36 +00:00
Nim Wijetunga 5f079904d4 add unit test 2021-03-16 18:30:39 +00:00
Nim Wijetunga 17f9da6962 remove traces 2021-03-16 16:47:50 +00:00
Andrew Noyes 191230a7b3 Factor our refreshTransaction gadget 2021-03-16 00:51:20 +00:00
Nim Wijetunga f0d79b3d86 Inital implementation of network busyness 2021-03-15 23:23:56 +00:00
Andrew Noyes d45481decc Fix broken_promise bug 2021-03-13 01:16:27 +00:00
Andrew Noyes 6cacf3e002 Fix potential heap-use-after-free
If we destroy tr here then we might not be able to dereference |self|.
Keep the window where |self| might have been deleted small.
2021-03-12 00:26:26 +00:00
Andrew Noyes a76c7b9754 Fix several memory leaks and a thread leak 2021-03-11 23:58:32 +00:00
Vishesh Yadav 2c316db94d Merge remote-tracking branch 'apple/master' into master-format-final
Also format the changes
2021-03-11 12:54:26 -08:00
Andrew Noyes be980ebb55
Merge pull request #4354 from sfc-gh-clin/add-advanceversion
Add advanceversion into special keys
2021-03-11 11:43:53 -08:00
Vishesh Yadav d7252da951 clang-format: Fix the TEST() macros which require comments in line 2021-03-10 16:50:53 -08:00
Chaoguang Lin 7404a29d58 Merge branch 'master' of github.com:apple/foundationdb into add-dd-and-maintenance 2021-03-10 11:47:39 -08:00
FDB Formatster df90cc89de apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-10 10:18:07 -08:00
Evan Tschannen 6cfba6e54b added support for popping rangefeeds 2021-03-05 11:46:33 -08:00
Evan Tschannen 21822e4949 more fixes 2021-03-04 19:46:32 -08:00
Evan Tschannen adaa67df63 fix registerRangeFeedActor 2021-03-04 19:42:36 -08:00
Evan Tschannen bc757f3e4b basic framework for range feed support 2021-03-04 19:14:54 -08:00
Andrew Noyes 79cec09255 Apply clang-tidy's performance-inefficient-vector-operation fix
I ran this command in my build directory after compiling with
OPEN_FOR_IDE. It took a few small tweaks to get it to compile, which is
outside the scope of this commit.

    $ python run-clang-tidy.py -j $(nproc) -checks='-*,performance-inefficient-vector-operation' -fix
2021-03-04 03:58:25 +00:00
Chaoguang Lin 9ce9cba0e5 Merge branch 'master' of github.com:apple/foundationdb into add-advanceversion 2021-02-19 14:29:08 -08:00
Chaoguang Lin 703107332a Add special keys for profile client get/set 2021-02-19 12:22:00 -08:00
Andrew Noyes 9fb396e372
Merge pull request #4255 from sfc-gh-clin/add-coordinators-into-special-keys
Add coordinators into special keys
2021-02-18 18:26:07 -08:00
Nim Wijetunga fa3834f115 fix bug 2021-02-18 18:23:28 -05:00
Chaoguang Lin ac860b3c22 Add advanceversion into special keys 2021-02-18 01:27:14 -08:00
Andrew Noyes c075415a51
Merge pull request #4344 from sfc-gh-ljoswiak/fixes/remove-async-tracer
Remove async tracer
2021-02-17 09:43:24 -08:00
Chaoguang Lin f3a849de10 Update ChangeConfig test for auto coordinators change 2021-02-16 23:55:58 -08:00
Lukas Joswiak 30d2d9ff82 Remove async tracer 2021-02-16 16:36:33 -08:00
Markus Pilman 82cda5448d
Merge pull request #4266 from sfc-gh-nwijetunga/optimize-watches-client
Optimize Watches Client
2021-02-16 13:28:51 -07:00
Nim Wijetunga a59f07e6fe address pr comment 2021-02-16 13:32:03 -05:00
Nim Wijetunga 13e4f6d7e1 small formatting fix 2021-02-12 12:57:33 -05:00
Nim Wijetunga 058557a3b1 fix pr issues 2021-02-11 15:51:37 -05:00
Nim Wijetunga 270a18f089 formatting fixes 2021-02-10 13:19:59 -05:00
Nim Wijetunga 7acfeda955 cleanup 2021-02-09 20:57:16 -05:00
Nim Wijetunga 386e3b0838 minor change 2021-02-09 18:36:45 -05:00
Nim Wijetunga 4982679ef2 change function name 2021-02-09 18:06:36 -05:00
Nim Wijetunga 3b48942f9a resolve pr comments 2021-02-09 17:56:30 -05:00
Nim Wijetunga d2233ffa5a whitespace change 2021-02-09 13:38:24 -05:00
Nim Wijetunga 82e9f54938 small change 2021-02-09 13:36:44 -05:00
Nim Wijetunga 2d46cfcb80 remove traces 2021-02-09 13:33:19 -05:00
Nim Wijetunga 2560dc0227 add test workload 2021-02-09 12:32:37 -05:00
Chaoguang Lin 90686a95a0 format code 2021-02-08 22:17:16 -08:00
Chaoguang Lin 61082e70e1 format code, add comments 2021-02-08 19:11:30 -08:00
Nim Wijetunga 83af53b1bb correctness fixes 2021-02-05 19:23:37 -05:00
Chaoguang Lin a85458dd1e Merge branch 'master' of github.com:apple/foundationdb into add-c-function-snapshot 2021-02-05 14:14:52 -08:00
Chaoguang Lin 97925de9d4 Update fdb_database_create_snapshot to take a UID as input 2021-02-05 14:14:38 -08:00
Nim Wijetunga 6c687067a6 refactor 2021-02-04 12:51:53 -05:00
Nim Wijetunga ec6226bde9 change logic 2021-02-03 20:01:26 -05:00
Nim Wijetunga b6e7961b78 change to references 2021-02-02 20:29:03 -05:00
Nim Wijetunga c67aac8212 remove the use of pointers 2021-02-02 13:26:12 -05:00
Nim Wijetunga 6f38105e48 fix issue with case 4 2021-02-01 19:07:14 -05:00
Nim Wijetunga 6bee8ac0aa Inital implementation 2021-02-01 18:42:13 -05:00
Chaoguang Lin c2aedb0b94 clang-format, remove comments 2021-01-29 18:20:09 -08:00
Chaoguang Lin 7d32045506 Merge branch 'master' of github.com:apple/foundationdb into add-coordinators-into-special-keys 2021-01-29 11:46:01 -08:00
Chaoguang Lin e1182f2f86 Add coordinators to special keys 2021-01-29 11:45:52 -08:00
Vishesh Yadav 5b5dd0a0d4
Merge pull request #4204 from sfc-gh-ljoswiak/features/network-tracer
Add network option to set tracer in fdbclient
2021-01-27 20:00:49 -08:00
Chaoguang Lin 5f912f0deb Add snapshot c function 2021-01-26 11:53:35 -08:00
sfc-gh-tclinkenbeard 7558cd8e06 Merge remote-tracking branch 'origin/master' into simplify-global-knobs 2021-01-26 10:00:25 -08:00
Andrew Noyes a51de2f73d
Merge pull request #4220 from sfc-gh-clin/add-c-function-force-recovery
Add c function fdb_database_force_recovery_with_data_loss
2021-01-25 18:22:41 -08:00
Lukas Joswiak 29c0e05ea6 Clarify naming of distributed tracing options 2021-01-22 17:30:33 -08:00
Lukas Joswiak 500eabf6dd Add network option to set tracer in fdbclient 2021-01-21 12:04:28 -08:00
sfc-gh-tclinkenbeard fb9b4b7626 Merge remote-tracking branch 'origin/master' into misc-changes 2021-01-21 10:45:50 -08:00
sfc-gh-tclinkenbeard ad99bf0471 Merge remote-tracking branch 'origin' into misc-changes 2021-01-21 10:03:07 -08:00
Chaoguang Lin 9cd7c43a2c Add c function fdb_database_force_recovery_with_data_loss 2021-01-21 00:42:23 -08:00
Lukas Joswiak fd20011276 Remove tags 2021-01-20 17:05:58 -08:00
Lukas Joswiak 4d6fbf099c Use StringRef for span tags 2021-01-20 13:57:36 -08:00
Lukas Joswiak 7ae5e98c5a Add metadata to a few spans 2021-01-20 13:57:36 -08:00
Andrew Noyes b0f61fb74f Resolve the simple-looking conflicts 2021-01-15 19:35:10 +00:00
Andrew Noyes ff7d306b09 Merge branch 'release-6.3' into anoyes/merge-6.3-to-master
Include conflict markers for now. Will resolve.
2021-01-15 18:04:09 +00:00
Chaoguang Lin 8ce9bd4e8e Merge branch 'master' of github.com:apple/foundationdb into add-c-function-for-management-commands 2021-01-14 17:31:26 -08:00
Andrew Noyes e32a0fb742 Resolve conflicts 2021-01-14 20:13:29 +00:00
Andrew Noyes 6d6b2843da Merge remote-tracking branch 'upstream/release-6.2' into anoyes/merge-release-6.2
I'm including the conflict markers for now and I'll resolve them in a
subsequent commit
2021-01-14 17:20:11 +00:00
Chaoguang Lin 87b4463305 Solve comments: update c api description; refine code 2021-01-13 14:56:41 -08:00
sfc-gh-tclinkenbeard 8ff14878fe Merge remote-tracking branch 'origin/master' into simplify-global-knobs 2021-01-13 14:39:35 -08:00
A.J. Beamon 58360a192f Add a counter for the number of status requests made by a client 2021-01-05 10:18:43 -08:00
Chaoguang Lin e097c98956 Remove headed added by IDE, add comments 2021-01-04 00:18:01 -08:00
Chaoguang Lin da7b65cd71 Change the interface back to return int64_t 2021-01-03 22:45:09 -08:00
sfc-gh-tclinkenbeard 5b2e88b187 Use structured bindings in for loops 2020-12-27 01:46:20 -04:00
Jingyu Zhou 0ecfaf138e
Merge pull request #4150 from sfc-gh-ljoswiak/fixes/sps-tracing
Remove "a/" from tracing keys in special-key-space
2020-12-21 13:48:01 -08:00
Chaoguang Lin 9db9a84798 Ignore negative duration value 2020-12-21 13:29:16 -08:00
Chaoguang Lin b68d84aaea Merge remote-tracking branch 'upstream/master' into add-c-function-for-management-commands 2020-12-21 12:32:31 -08:00
Chaoguang Lin 7b4f7ce4e0 update comments 2020-12-20 22:47:55 -08:00
Andrew Noyes 9601769b01
Merge pull request #3858 from sfc-gh-rchen/stable_interfaces
Stable interfaces
2020-12-11 09:34:27 -08:00
Richard Chen 5f57d72a59 remove print statements and format protocol version workload 2020-12-11 04:46:20 +00:00
Russell Sears 4cb821cd63 Merge remote-tracking branch 'upstream/release-6.2' into merge-6.2-to-6.3 2020-12-09 15:47:44 -08:00
Lukas Joswiak 43fd545f06 Remove a/ from tracing path in special-key-space 2020-12-09 13:40:16 -08:00
Trevor Clinkenbeard 9581bc0279
Merge pull request #4129 from sfc-gh-anoyes/anoyes/merge-release-6.3-master
Merge release 6.3 master
2020-12-08 08:57:54 -08:00
Lukas Joswiak 54237f28ea Throw error on commit for late trace options 2020-12-07 14:43:44 -08:00
Lukas Joswiak 18201d6f4f Add option to enable transaction tracing 2020-12-07 11:24:14 -08:00
A.J. Beamon 75836d6114 Fix double counting of range reads in TransactionMetrics 2020-12-07 09:01:19 -08:00
Andrew Noyes 7fbc4d7391 Resolve conflicts 2020-12-04 23:58:42 +00:00
Lukas Joswiak 76142549e0 Return keys from range read in lexicographic order 2020-12-04 11:21:39 -08:00
Andrew Noyes 877997632d Merge branch 'release-6.3' into anoyes/merge-release-6.3-master
Include conflict markers for review purposes
2020-12-04 01:38:07 +00:00
Lukas Joswiak 3669615e4b Refactor and clean up 2020-12-03 14:06:11 -08:00
Chaoguang Lin 88e66545bb Change the api to return bool instead of int 2020-12-03 12:01:32 -08:00
Richard Chen c77d9e4abe merge conflicts 2020-12-02 21:53:19 +00:00
Lukas Joswiak 1f166bc183 Add options to set custom transaction ID and disable tracing 2020-11-30 14:57:17 -08:00
Andrew Noyes 1f541f02be Merge branch 'anoyes/merge-6.2-to-6.3' into anoyes/release-6.3-merge
Merge, leaving conflict markers for now
2020-11-24 16:55:34 +00:00
Chaoguang Lin 2fae7333c0 Add duration parameter in the api 2020-11-23 14:55:29 -08:00
sfc-gh-tclinkenbeard b1e3478267 Simplified global knobs 2020-11-21 13:27:48 -08:00
Chaoguang Lin 2320f970c1 Add reboot_worker and pythong binding 2020-11-19 16:43:39 -08:00
Chaoguang Lin aec0398fd0 Merge remote-tracking branch 'upstream/master' into add-c-function-for-management-commands 2020-11-17 11:39:12 -08:00
sfc-gh-tclinkenbeard 45c9a0abc7 Revert "Revert "Add limiting health metrics""
This reverts commit 209ebcc595.
2020-11-13 17:24:57 -08:00
Trevor Clinkenbeard 209ebcc595
Revert "Add limiting health metrics" 2020-11-13 17:08:46 -08:00
Trevor Clinkenbeard 8c0b4dbe4c
Merge pull request #4067 from sfc-gh-tclinkenbeard/add-limiting-health-metrics
Add limiting health metrics
2020-11-13 16:04:44 -08:00
sfc-gh-tclinkenbeard 6c4493166f Add limiting storage queue and durability lag to health metrics 2020-11-12 20:14:41 -08:00
Markus Pilman 0f205620bf
Merge branch 'master' into features/udp 2020-11-12 13:51:17 -07:00
Markus Pilman 1343f40117 don't allow empty coments 2020-11-11 14:07:54 -07:00
sfc-gh-tclinkenbeard 4669f837fa Add uses of makeReference 2020-11-07 22:10:18 -08:00
Meng Xu 4788544a6f Revise comments based on review suggestions
Ack. Jingyu and Xin for their suggestions.
2020-11-06 08:51:13 -08:00
Meng Xu 063700e4d6 Add comments and questions to HA and tLog code reading
The comments' correctness need to be confirmed by reviewers.
2020-10-30 12:14:57 -07:00
sfc-gh-tclinkenbeard cf4c8e375f Merge remote-tracking branch 'origin/release-6.3' into merge 2020-10-29 22:15:41 -07:00
Chaoguang Lin a72bb52eae Merge branch 'master' of github.com:apple/foundationdb into add-c-function-for-management-commands 2020-10-28 22:32:44 -07:00
Richard Chen bf7bf470d9 Merge branch 'master' of https://github.com/apple/foundationdb into stable_interfaces 2020-10-26 19:36:19 +00:00
Andrew Noyes a15bc32b14
Merge pull request #3945 from sfc-gh-clin/add-consistencycheck-to-special-keys
Add consistencycheck command to special keys
2020-10-26 10:59:17 -07:00
Richard Chen 309e38dc06 add unit test and some cleanup 2020-10-23 19:34:18 +00:00
Richard Chen 055add9682 conflicts 2020-10-23 06:33:00 +00:00