Commit Graph

217 Commits

Author SHA1 Message Date
A.J. Beamon d39c0b773a Add a limit to the number of tenants that can be created in a cluster 2022-07-27 08:21:03 -07:00
Josh Slocum 77956dc7ae
Merge pull request #7639 from sfc-gh-jslocum/cf_metadata_rewrite
Change Feed Metadata Rewrite and adding targeted fault injection
2022-07-26 18:10:37 -05:00
Josh Slocum 6115a406d4 formatting 2022-07-25 13:52:47 -05:00
Josh Slocum af60e2ea32 Fixed granule purging bug and improved debugging for purging 2022-07-25 12:19:41 -05:00
Josh Slocum ee1b0cdd43 making purge failures fail test 2022-07-23 13:10:03 -05:00
A.J. Beamon 17146c484b Use key-backed types for tenants. Add a tenant state field that will be used in upcoming work. Some other tenant related refactoring. 2022-07-21 20:33:28 -07:00
Josh Slocum 316b7a5344 Merge branch 'main' into granule_merging_converge 2022-07-20 12:13:48 -05:00
Josh Slocum fd9201f60b Merge branch 'main' into cf_metadata_rewrite 2022-07-20 07:55:00 -05:00
Josh Slocum 44885aa719 cleanup 2022-07-20 07:46:57 -05:00
Josh Slocum 78b6a96006 Merge branch 'main' into granule_merging_batch 2022-07-20 07:42:26 -05:00
Josh Slocum 4000682578 moving comment 2022-07-20 07:35:30 -05:00
Josh Slocum 12b6f386cb Refactoring granule flush to retry properly on granule rollback 2022-07-19 19:49:20 -05:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
Josh Slocum 2573c0018c cleanup after passing tests 2022-07-19 08:00:25 -05:00
Josh Slocum 0a4aef16d0 Several bugs with new change feed metadata 2022-07-18 19:32:48 -05:00
Josh Slocum bfab550435 cleanup 2022-07-18 16:26:03 -05:00
Josh Slocum 150346af64 Explicitly avoiding large keys problem in single value 2022-07-15 21:08:46 -05:00
Josh Slocum 08186f9245 More efficient merge intent and granule history serialization 2022-07-15 20:38:06 -05:00
Josh Slocum 306610bfcb batch periodic merging in blob manager 2022-07-15 15:52:10 -05:00
Josh Slocum 866dda5763 Adding check for merge convergence 2022-07-14 20:49:43 -05:00
Josh Slocum 0720b358ff comments and bug fix 2022-07-13 08:01:26 -05:00
Josh Slocum b2a96b64e6 refactoring one function into two 2022-07-13 07:49:19 -05:00
Josh Slocum f707626213 Cleanup 2022-07-12 17:43:44 -05:00
Josh Slocum 0b0ac16a4c Merge branch 'main' into granule_merging 2022-07-12 09:09:30 -05:00
Josh Slocum 48bcf6eaba
parameterizing min split size instead of hardcoding it to min_shard_bytes (#7404) 2022-06-30 12:28:22 -07:00
A.J. Beamon 26b35c07cd Refactor how tenant map entries are encoded and decoded. Add a specific version to the encoding that matches the version used when this feature was introduced (and the only version in which it was used). 2022-06-29 10:58:58 -07:00
A.J. Beamon 4bafe77889 Some refactoring of tenant code:
* extract tenant management into its own file and namespace
* rename the tenant management workload source file
* extract tenant special keys functions to a separate file
* extract some helper functions to GenericTransactionHelper.h
* convert StringRef -> TenantNameRef
* move some TenantMapEntry implementation into the cpp file
* add some helper functions to decode/encode a tenant mode
2022-06-27 12:32:49 -07:00
Markus Pilman d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Josh Slocum 710612442c small fixes 2022-06-09 17:21:21 -05:00
Josh Slocum d6920cde28 Implemented blob granule merging 2022-06-09 10:50:53 -05:00
Josh Slocum ae865027d6 fixes for blob metadata memory from valgrind 2022-06-07 13:50:11 -05:00
Josh Slocum fcd20c479d addressing review comments 2022-06-03 08:36:07 -05:00
Josh Slocum 567b1d35f7 Full integration with BlobConnectionProvider for blob granules 2022-06-01 12:39:51 -05:00
Josh Slocum 85af0a25b2 Enabling BM to understand tenant boundaries, and changing BlobGranuleCorrectness to use tenants 2022-05-25 17:16:56 -05:00
Josh Slocum 6276cebad9
Blob integration (#6808)
* Fixing leaked stream with explicit notify failed before destructor

* better logic to prevent races in change feed fetching

* Found new race that makes assert incorrect

* handle server overloaded in initial read from fdb

* Handling more blob error types in granule retry

* Fixing rollback metadata problem, added better debugging

* Fixing version race when fetching change feed metadata

* Better racing split request handling

* fixing assert

* Handle change feed popped check in the blob worker

* fix: do not use a RYW transaction for a versionstamp because of randomize API version (#6768)

* more merge conflict issues

* Change feed destroy fixes

* Fixing change feed destroy and move race

* Check error condition in BG file req

* Using relative endpoints for blob worker interface

* Fixing bug in previous fix

* More destroy and move race fixes

* Don't update empty version on destroy in case it gets rolled back. moved() and removing will take care of ensuring it is not read

* Bug fix (#6796)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* Fixing durability issue with moving and destroying change feeds

* Adding fix for not fully deleting files for a granule that child granules need to re-snapshot

* More destroy and move races

* Fixing change feed destroy and pop races

* Renaming bg prune to purge, and adding a C api and unit test for it

* more cleanup

* review comments

* Observability for granule purging

* better handling for change feed not registered

* Fixed purging bugs (#6815)

* fix: do not use a RYW transaction for a versionstamp because of randomize API version

* fix: if the initialSnapshotVersion was pruned, granule history was incorrect

* added a way to compress null bytes in printable()

* fixed a few purging bugs

Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com>
2022-04-08 14:15:25 -07:00
Josh Slocum aaaf42525a misc bg operational fixes and improvements 2022-04-05 12:26:00 -05:00
Josh Slocum cb918b9cef Added basic blob granule consistency check 2022-04-04 11:38:42 -05:00
Josh Slocum 377e252fcf
Better split sizing in blob manager (#6725) 2022-04-01 16:09:46 -07:00
Josh Slocum 7fc6dfa6c5 Adding useful debugging trace events 2022-03-29 14:48:28 -05:00
Jingyu Zhou da0673ccce
Merge pull request #6705 from RenxuanW/another
Add proxy option to backup and restore params.
2022-03-29 11:36:13 -07:00
Josh Slocum 2f8e9d9de0 misc bg fixes 2022-03-29 08:05:52 -05:00
Renxuan Wang 0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
Josh Slocum 1b1182f414 Blob Worker side of beginVersion done, added unit test 2022-03-24 17:15:11 -05:00
Josh Slocum 0f9e88572a Cleaning up debugging and fixing race in blob manager recruitment 2022-03-17 14:57:43 -05:00
Josh Slocum a85b578d2b Added simulation validation that multiple blob maangers don't have same epoch 2022-03-17 14:57:43 -05:00
Evan Tschannen 2c88a189a9 more pruning bug fixes 2022-03-15 13:34:59 -07:00
Evan Tschannen d46e551f11 merge 2022-03-15 13:09:10 -07:00
Josh Slocum ce5b567971 Cleanup the cleanup 2022-03-15 15:02:28 -05:00
Josh Slocum 9d9b70abd4 Fixing another BM recovery and granule splitting race 2022-03-14 18:56:44 -05:00
Josh Slocum 8c35ceb684 switching bg file keys to sort on version then type, and using that to fix race where 2 blob workers try to split the same granule at different versions 2022-03-14 17:38:31 -05:00