wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
23,445 Commits 36 Branches 335 Tags 299 MiB
Commit Graph

435 Commits

Author SHA1 Message Date
Josh Slocum ab6953be7d
Blob Granule read-driven compaction (#8572) 2022-10-26 09:02:50 -07:00
sfc-gh-tclinkenbeard 31a48f4046 Disable GLOBAL_TAG_THROTTLING by default 2022-10-25 13:10:15 -07:00
sfc-gh-tclinkenbeard f339819758 Merge remote-tracking branch 'origin/main' into reject-tag-throttled-txns 2022-10-25 11:59:35 -07:00
Ankita Kejriwal ce733cd1a1
Merge pull request #8538 from sfc-gh-akejriwal/monitorusage 
Add functionality to get tenants over storage quota and improve the relevant monitors
 2022-10-24 16:28:07 -07:00
neethuhaneesha a1eb1d4a48
Rocksdb storage using single_key_deletes instead of deleterange on clearrange operation. (#8452) 2022-10-21 15:47:19 -07:00
Ankita Kejriwal 7647cea4e5 Improve the storage quota monitor code + add a knob for refresh interval 2022-10-21 15:44:13 -07:00
Jingyu Zhou 0872cbfb2f
Fix a test timeout (#8488) 
* Fix a test timeout due to buggified knob MAX_WRITE_TRANSACTION_LIFE_VERSIONS

The buggified knob MAX_WRITE_TRANSACTION_LIFE_VERSIONS can be only 1M. In some
tests, this transaction always end up commitVersion - readVersion is a little
above 1M, thus always getting transaction_too_old error.

* Change MAX_COMMIT_BATCH_INTERVAL instead

So that the master may give out versions fast enough.

* Fix an assertion failure in a unit test

48125>>8 = 187, 48125 = 0xbbfd
48128>>8 = 188, 48128 = 0xbc00
So if 48125 is chosen as the index, 48128 changes the higher order byte.

48125 & 0xff7f = 47997 = 0xbb7d. Thus +5 won't change the higher order byte.
 2022-10-19 09:45:56 -07:00
Ankita Kejriwal 3d890ced9a Code cleanup 2022-10-18 15:53:22 -07:00
sfc-gh-tclinkenbeard 003986fdb0 Randomize GLOBAL_TAG_THROTTLING knob 2022-10-18 15:16:24 -07:00
sfc-gh-tclinkenbeard 300840ea2e Enable GLOBAL_TAG_THROTTLING by default 2022-10-18 15:16:24 -07:00
Ankita Kejriwal fb014a4834 Merge branch 'main' of github.com:apple/foundationdb into monitorusage 2022-10-18 13:14:47 -07:00
sfc-gh-tclinkenbeard a556f21ed2 Merge remote-tracking branch 'origin/main' into limit-gtt-size 2022-10-15 10:26:16 -07:00
sfc-gh-tclinkenbeard 4ab947a9b3 Merge remote-tracking branch 'origin/main' into limit-gtt-size 2022-10-14 15:59:57 -07:00
Josh Slocum 914dfd7438
REST Blob metadata kms connector (#8474) 
* Changing RESTKMSConnector request handling to not be synchronous, as that would be a huge perf bottleneck

* Implementation of blob kms fetch

* cleanup

* review comments
 2022-10-14 17:49:00 -05:00
Josh Slocum 7cec0a5249
Blob metadata refresh (#8456) 
* Adding EKP refresh of blob metadata

* Adding re-fetching blob metadata from BGTenantMap

* adding buggifies from code review comments
 2022-10-14 08:17:50 -05:00
Hui Liu 169c341f79
Merge pull request #8386 from sfc-gh-huliu/blobmigrator 
Add blob migrator to assist data copy from blob to storage server
 2022-10-13 14:46:04 -07:00
Hui Liu 049df622f1 add a blob migrator 2022-10-13 13:21:45 -07:00
He Liu a43e424d8a Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency 2022-10-12 13:01:07 -07:00
Steve Atherton d74c072382
Merge pull request #8448 from sfc-gh-yiwu/encrypt_disable 
Add a knob to disable Redwood tenant page split
 2022-10-10 21:17:43 -07:00
Yi Wu 5dc14accfd Add a knob to disable Redwood tenant page split 2022-10-10 17:02:37 -07:00
He Liu a730e32164 Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency 2022-10-10 13:17:17 -07:00
neethuhaneesha 4b238e3985
RocksDB disable WAL for experimentation. (#8443) 2022-10-10 12:32:05 -07:00
sfc-gh-tclinkenbeard b887b1e85c Limit number of tags tracked by GlobalTagThrottler 2022-10-10 12:28:26 -07:00
He Liu b52edd8658 Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency 2022-10-10 11:00:05 -07:00
He Liu 88c37c81d8 Merge branch 'main' of https://github.com/apple/foundationdb into thread-priority 2022-10-05 14:20:53 -07:00
Hui Liu 9799329b99
Merge pull request #8390 from sfc-gh-huliu/actor 
Misc changes for blob manifest dumper
 2022-10-05 10:30:56 -07:00
neethuhaneesha a565863189
RocksDB compaction knobs and stats. (#8392) 2022-10-04 15:13:50 -07:00
Hui Liu 50b3a8d3ba misc changes for manifest dumper 
1) move dumper to a dedicated actor
2) include blob manager epoch into the manifest name
3) keep multiple manifest files(up to 5)
 2022-10-04 10:33:43 -07:00
He Liu 5f975623fb Merge branch 'main' of https://github.com/apple/foundationdb into thread-priority 2022-09-30 08:57:53 -07:00
Jingyu Zhou c321d633c7
Merge pull request #8361 from halfprice/zhewu/update-default-gray-latency-percentile 
Update default peer latency degradation percentile to 0.5
 2022-09-30 08:34:31 -07:00
He Liu 63b8d775a3
Knob for RocksDb behaviors (#8360) 
* Added knob for rocksdb suggest compact.

* Added ROCKSDB_LEVEL_COMPACTION_DYNAMIC_LEVEL_BYTES knob.

Co-authored-by: He Liu <heliu@apple.com>
 2022-09-29 15:40:29 -07:00
Zhe Wu 78f29ecd88 Update default PEER_LATENCY_DEGRADATION_PERCENTILE to 0.5 2022-09-29 14:18:26 -07:00
Trevor Clinkenbeard ef13985feb
Merge pull request #8320 from sfc-gh-tclinkenbeard/read-write-fungible 
Make read and write quotas fungible
 2022-09-28 10:59:37 -07:00
sfc-gh-tclinkenbeard ba8fbc9573 Merge remote-tracking branch 'origin/main' into read-write-fungible 2022-09-26 23:26:14 -07:00
Zhe Wu 0188f2712b Update default PEER_DEGRADATION_CONNECTION_FAILURE_COUNT value 2022-09-26 14:45:47 -07:00
Evan Tschannen a900d8dfa9
increase the target blob worker lag to account for the slow startup time when a blob worker reboots (#8294) 2022-09-26 09:19:38 -07:00
sfc-gh-tclinkenbeard 7fc5c196c4 Make read and write quotas fungible 2022-09-25 21:00:11 -07:00
Markus Pilman 5774249e5b
Revert "[DRAFT] Redwood PriorityMultiLock enable different launch limits to be specified based on different priority level." 2022-09-23 12:22:47 -06:00
Ata E Husain Bohra 52169d2b8e
Enable ZSTD compression support (#8014) 
* Enable ZSTD compression filter

Description

  diff-4: Randomize Knob Compression filter selection
  diff-3: Minor refactoring
  diff-2: Limit ZSTD availability to CLANG compiler
  diff-1: Add ZSTD compression option to BlobGranule tests

Major changes includes:
1. Update FDB CMake to download, install and build Boost with
ZSTD compatibility
2. Update CompressionUtils to enable boost::iostreams::zstd
compression filter

Testing

CompressionUtilsUnit.toml
BlobGranuleCorrectness/BlobGranuleCorrectnessClean
devRunCorrectness - 100K (in-progress)
 2022-09-22 14:31:49 -07:00
Steve Atherton 04b4960786 Merge branch 'main' into fzhao/RedwoodIOLaunchLimit 
# Conflicts:
#	fdbserver/VersionedBTree.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/ReadWrite.actor.cpp
 2022-09-22 00:39:51 -07:00
neethuhaneesha c4f59ba654
Merge pull request #8261 from neethuhaneesha/rocksdb-cachesize 
RocksDB block cache size increase.
 2022-09-21 14:50:31 -07:00
Josh Slocum 6270016bed
Seq insert perf fixes main (#8264) 
* Force flushing granules post-split to guarantee parent feeds get cleaned up

* fixing bug and cleaning up split finalize code
 2022-09-21 12:36:02 -07:00
neethuhaneesha bc9e806f1c RocksDB block cache size increase. 2022-09-21 11:41:27 -07:00
Fuheng Zhao 92aee50dbf update format 2022-09-20 22:51:45 -07:00
Fuheng Zhao 7002150188 add storage server read concurrency knob 2022-09-20 21:23:54 -07:00
Steve Atherton 3c306cc558 Merge branch 'fzhao/feature-testing' into fzhao/RedwoodIOLaunchLimit 2022-09-20 01:14:04 -07:00
He Liu 2e2c198df7 Added knobs for RocksDB reader/writer thread priorities. 2022-09-19 11:20:17 -07:00
Steve Atherton 74b152e550 Removed two obsolete things: explicit maxPriority argument from PriorityMultiLock as it is redundant after the launch limit refactor, and the redwood read concurrency lock which is no longer needed after the StorageServer priority refactor as it will control the concurrency of requests sent to the KVS. 2022-09-18 18:23:15 -07:00
Steve Atherton 31a66f9f12
Add comment explaining STORAGESERVER_READ_RANKS 2022-09-17 22:44:45 -07:00
sfc-gh-ngoyal 1bd97fe628
Recruit new singleton for consistency checker. (#5804) 
* Recruit new singleton for consistency checker.

* Recruit the consistency checker only if enabled.

* Add a yield in monitorConsistencyChecker().

* Minor fixes.

* Consistency check workload enhancements.

* Minor fixes and clarifications.

* clang format

* Clang format.

* Minor fixes, cleanup, debug tracing.

* Misc.

* Move the consistency scan information from dbconfig to a key backed object.

* Move consistency scan config out of db cofig to a state object and feature rename.

* ConsistencyCheck workload refactor.

* devFormat

* Update fdbcli/ConsistencyScanCommand.actor.cpp

* Review Comments.

Co-authored-by: negoyal <neelam.goyal@gmail.com>
Co-authored-by: Ata E Husain Bohra <ata.husain@snowflake.com>
 2022-09-16 09:03:06 -07:00