Commit Graph

257 Commits

Author SHA1 Message Date
Nim Wijetunga c63b30f698 address pr comments 2022-08-08 11:32:49 -07:00
Nim Wijetunga 369494ab19 merge 2022-08-05 18:52:39 -07:00
Josh Slocum f866ffc36b
Better granule conversion (#7787)
* better check for granule-ification

* Handling blob granule initial split too large

* Re-evaluating split size if too large, even if read doesn't get transaction_too_old

* reworked to have blob worker propose split key

* New GranuleStatusReply to avoid seqno issue stream side effects

* Handling retries on reevaluateInitialSplit properly

* Waiting for stream to be initialized

* Checking reevaluate split for additional split points beyond proposed

* Fixing more races in reevaluate initial split

* properly handling cleaning up old change feed after split re-evaluate

* fixing granule conversion bug with hard boundaries

* fixing clear and merge check race with cycle test

* refactor missed knob check for clearAndMerge

* Fixing formatting

* review comments and improving large range conversion

* fixing typo

* more formatting
2022-08-05 18:12:17 -05:00
Hui Liu cd0020ccaf
add counters for blob compression (#7799) 2022-08-05 18:07:13 -05:00
Josh Slocum b2835921ba
Using knownBlobRanges for blob granule ranges whether tenants are enabled or not (#7788)
* Using knownBlobRanges for blob granule ranges whether tenants are enabled or not

* Effectively disabled blob granule tests when tenants enabled to fix ctest
2022-08-05 11:46:09 -05:00
Josh Slocum d721d1b850
More cf bug fixes (#7789)
* Fixing change feed fetch and rollback race

* Fixing validation issue for change feed validation

* Fixing shutdown segfault in blob worker
2022-08-04 15:57:42 -05:00
Josh Slocum 7f45cccb56
More granule purging fixes (#7756)
* Granule purge cannot delete history entry for fully deleting granule until all children are completely done splitting

* Several purging fixes related to granule history

* Fixed typo in refactor

* fixing memory model for purgeRange

* formatting

* weakening granule purge test for now

* cleanup

* review comments
2022-08-03 16:43:27 -05:00
Nim Wijetunga b922470ceb fix pr issues 2022-08-03 13:26:14 -07:00
Nim Wijetunga 8d591fc5e7 address pr comments 2022-08-02 10:51:41 -07:00
Nim Wijetunga 9a4fb8ad4e merge 2022-08-02 09:49:16 -07:00
Ata E Husain Bohra ef6012c1d1
Encrypt BlobGranule delta files (#7735)
* Encrypt  BlobGranule delta files

Description

 diff-1: Address review comments

Major changes proposed by the patch are:
1. Refactor code to allow caching of 'encryption key ctx' as part of
BlobFilePointerRef. The refactoring allows snapshot and/or delta files
to store their own file encryption context.
2. Enable BlobGranule delta file encryption/decryption semantics.

Testing

BlobGranuleCorrrectness  
BlobGranuleCorrectnessClean
BlobGranuleFileUnitTestToml

Description

Testing
2022-08-01 16:34:44 -07:00
Nim Wijetunga af6db42b1b temp 2022-08-01 15:19:21 -07:00
Junhyun Shim c6342a6e5b
Merge branch 'main' into features/authz 2022-07-27 20:51:32 +02:00
A.J. Beamon d39c0b773a Add a limit to the number of tenants that can be created in a cluster 2022-07-27 08:21:03 -07:00
Junhyun Shim 5169616b16 Fix unresolved merge conflicts 2022-07-27 00:38:16 +02:00
Josh Slocum c32e1da908
Merge pull request #7673 from sfc-gh-jslocum/delta_files_v2
Sorted Delta Files
2022-07-26 16:04:55 -05:00
Josh Slocum 15e7a4b186 addressing review comments 2022-07-26 14:20:35 -05:00
Josh Slocum ea9018460a cleanup and polish 2022-07-22 15:13:32 -05:00
Josh Slocum 095a5a4868 First version of key-sorted delta files 2022-07-22 11:43:49 -05:00
A.J. Beamon 17146c484b Use key-backed types for tenants. Add a tenant state field that will be used in upcoming work. Some other tenant related refactoring. 2022-07-21 20:33:28 -07:00
Josh Slocum 6fc0d61146 delta file test and delta generation 2022-07-21 11:49:13 -05:00
Josh Slocum 316b7a5344 Merge branch 'main' into granule_merging_converge 2022-07-20 12:13:48 -05:00
Josh Slocum 78b6a96006 Merge branch 'main' into granule_merging_batch 2022-07-20 07:42:26 -05:00
Josh Slocum 12b6f386cb Refactoring granule flush to retry properly on granule rollback 2022-07-19 19:49:20 -05:00
Josh Slocum 166de4b704 Fixing merge convergence bugs 2022-07-19 18:44:40 -05:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
Josh Slocum 08186f9245 More efficient merge intent and granule history serialization 2022-07-15 20:38:06 -05:00
Josh Slocum 306610bfcb batch periodic merging in blob manager 2022-07-15 15:52:10 -05:00
Ata E Husain Bohra 1da88d5d8c BlobFile Encryption and compression support
Fix formatting issues

Description

Testing
2022-07-14 22:46:14 -07:00
Ata E Husain Bohra f288abebc2 BlobFile Encryption and compression support
Fix formatting issues and rename KNOB

Description

Testing
2022-07-14 17:22:00 -07:00
Ata E Husain Bohra f6f117592d BlobFile Encryption and compression support
- Limit verbose logging under DEBUG_MACRO
 - Update/Add code documentation

Description

Testing
2022-07-14 17:04:14 -07:00
Ata E Husain Bohra 3a3811d6cc BlobFile Encryption and compression support
Few fixes
 - Update BlobGranuleCipherKeysMeta to use 'std::string' to
   persist 'Initialization Vector'
 - Update WriteSnapshot to persist 'BlobGranuleCipherKeysMeta' as part
   of BlobFileIndex

Description

Testing
2022-07-14 17:04:14 -07:00
Ata E Husain Bohra 24b2de8de8 BlobFile Encryption and compression support
Description

Testing
2022-07-14 17:04:14 -07:00
Josh Slocum b85fbaef52
Merge pull request #7395 from sfc-gh-jslocum/bg_file_chunking
Chunked Snapshot Files
2022-07-13 17:22:34 -05:00
Josh Slocum b9437e9a8d BW re-sending merge candidate if manager changes 2022-07-13 10:39:55 -05:00
Josh Slocum f707626213 Cleanup 2022-07-12 17:43:44 -05:00
Josh Slocum 0b0ac16a4c Merge branch 'main' into granule_merging 2022-07-12 09:09:30 -05:00
Josh Slocum 9e64037b25 Merge branch 'main' into bg_file_chunking 2022-06-30 17:13:02 -05:00
A.J. Beamon 26b35c07cd Refactor how tenant map entries are encoded and decoded. Add a specific version to the encoding that matches the version used when this feature was introduced (and the only version in which it was used). 2022-06-29 10:58:58 -07:00
A.J. Beamon 4bafe77889 Some refactoring of tenant code:
* extract tenant management into its own file and namespace
* rename the tenant management workload source file
* extract tenant special keys functions to a separate file
* extract some helper functions to GenericTransactionHelper.h
* convert StringRef -> TenantNameRef
* move some TenantMapEntry implementation into the cpp file
* add some helper functions to decode/encode a tenant mode
2022-06-27 12:32:49 -07:00
Markus Pilman d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Josh Slocum 3f871f4123 Chunked Snapshot Files passes unit test 2022-06-15 14:52:28 -05:00
Josh Slocum 058f44a4ae Fixed req begin version when time travelling back from merge 2022-06-10 15:11:30 -05:00
Josh Slocum 710612442c small fixes 2022-06-09 17:21:21 -05:00
Josh Slocum d6920cde28 Implemented blob granule merging 2022-06-09 10:50:53 -05:00
Josh Slocum ae865027d6 fixes for blob metadata memory from valgrind 2022-06-07 13:50:11 -05:00
Josh Slocum ff643cce76 compile fix 2022-06-03 11:50:33 -05:00
Josh Slocum b650410a48 Merge branch 'main' into blob_granule_kms 2022-06-03 09:13:49 -05:00
Josh Slocum fcd20c479d addressing review comments 2022-06-03 08:36:07 -05:00
Josh Slocum 567b1d35f7 Full integration with BlobConnectionProvider for blob granules 2022-06-01 12:39:51 -05:00