b20dcf23a9
Support periodic compaction for sharded rocksdb. ( #10815 )
2023-08-25 15:38:01 -07:00
c63ee571e5
Export file metrics and add knob for file size multiplier. ( #10785 )
2023-08-16 11:27:33 -07:00
5868173a3e
nits
2023-08-10 16:51:28 -05:00
81db1da5a9
address comments
2023-08-10 16:51:28 -05:00
5598f4f28f
trace shardedrocks manifest size
2023-08-10 16:51:28 -05:00
ae7fe844d2
Merge pull request #10689 from xis19/rocksdbflag
...
Add a flag for enabling/disabling RocksDB storage engine in simulations
2023-08-01 10:50:09 -07:00
d0742c79ac
Improving visibility to debug sharded rocksdb ( #10694 )
...
* logging storage commit stats
* add rocks flush and compaction listener
* remove used field in FlushStats and fix CI error
* reduce LOGGING_ROCKSDB_BG_WORK_PROBABILITY
* merge rocks event listeners
* avoid using mutex/spinloop in rocksdb event listener
* code clean
* fix OnCompactionBegin and OnFlushBegin
* add logReason to RecentRocksDBBackgroundWorkStats
* add error listener back
2023-07-31 14:45:26 -07:00
0036bab8a4
Rename SSD_ROCKSDB_EXPERIMENTAL to WITH_ROCKSDB
2023-07-27 14:04:10 -07:00
24968674ce
Merge pull request #10643 from yao-xiao-github/write-buffer
...
Add knob.
2023-07-21 08:31:14 -07:00
e1aa6d3dd1
Log counters. ( #10626 )
2023-07-20 13:01:40 -07:00
391c2041d1
Add logs for write stalls. ( #10640 )
2023-07-20 13:00:05 -07:00
ff2f752bf9
Add knob.
2023-07-18 16:54:06 -07:00
ab72951034
Add knob for manifest file size and log rocksdb status. ( #10567 )
2023-07-05 10:40:43 -07:00
6337125712
Several minor improvements for ShardedRocksDB ( #10520 )
...
* Terminate DD if SHARD_ENCODE_LOCATION_METADATA is not enabled and storage_engine_type is ShardedRocksDB.
* Fixed Error in non-main thread.
* Minor improvements.
2023-06-24 16:07:14 -07:00
ef682d304e
fix IKeyValueStore include
2023-06-16 13:28:40 -07:00
7d1c6e7f17
Improve sharded rocksdb init time. ( #10475 )
2023-06-15 09:37:32 +08:00
7ccc734f87
Avoiding read timeout checks for eager reads(write path) and system keys. ( #10486 )
2023-06-14 09:56:23 -07:00
74614161b2
Merge pull request #10460 from sfc-gh-etschannen/feature-durable-change-feed
...
Cache change feeds durably on blob workers
2023-06-14 08:35:52 -07:00
c286c9ca3e
Fixed Iterator Pool `reset` issue & Improve psm test ( #10462 )
...
* Cancel a data move if the source or dest server lists are modified.
* error_.
* Enabled DD ealier.
* fixed iterator pool `reset` error.
* fmt.
2023-06-13 12:06:20 -07:00
335ccf1d87
correctly include server knobs
2023-06-13 07:33:01 -07:00
a2c5554de7
include server knobs because of CI failure
2023-06-11 15:27:57 -07:00
359e178dcd
Merge branch 'main' into feature-durable-change-feed
...
# Conflicts:
# fdbclient/ClientKnobs.cpp
# fdbserver/BlobManager.actor.cpp
# fdbserver/worker.actor.cpp
2023-06-11 13:58:35 -07:00
415bf4faea
Psm minor fixes ( #10407 )
...
* Log data move reasons in DataMoveStats.
* Remove a move-in-shard only when it is no longer used.
* Dont remove ranges if restore failed due to actor_cancelled.
2023-06-08 12:10:37 -07:00
cefb6c7ec3
Clear metadata before destroyDB. ( #10393 )
2023-06-02 11:20:14 -07:00
5d3f923177
Improve logging ( #10387 )
2023-05-31 18:36:53 -07:00
8ad7ec6fdf
Psm ss ( #9817 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* Refactored ShardedRocks checkpoint/restore for psm.
* Complete ShardedRocks::restore.
* dismiss operation_obsolete, and throw actor_cancelled.
* Validate checkpoint when !asKeyValues.
* fmt.
* Don't read from uninitialized physical shard.
* Resolved commments.
* cleanup.
* Added verify_checksum_before_restore for ShardedRocks.
* Added ShardedRocksDB checkpoint/restore unit test.
* Populate CheckpointMetaData::dir in RocksDB.
* Rename MovingIn as Adding.
* Added StorageServerUtils.
* Added physical shard move in SS.
* Fix on ApplyMetaData, doFetchFile error handling etc.
* Debugging incorrect shard size.
* Create/delete checkpoints only when Physical shard move is enabled.
* Added back SHARD_ENCODE_LOCATION_METADATA.
* Fixed bytesSample incorrect issue.
Essentially dedicated CheckpointRocksDBCF as key-value based checkpoint, will need to add a new format for the file-based checkpoint.
* Cleanup.
* Cleanup & compile rocksdb with 8.1 branch.
* clean up.
* clean up.
* Allowed request_maybe_delivered error type in FetchShard.
* Added FDBRocksDBVersion.h.
* Fixed stuck fetchShard.
* Don't create checkpoint on TSS.
* Upgrade to RocksDB 8.1.1
* Cleanup.
* Fixed accidently deleted db_path and name fields.
* Improved trace event.
* Removed redundants from previuos ShardedrocksDB.
* Cleanup.
* cleanup.
* cleanup.
* reanme `state`.
* Cleanup.
* Removed excessive TraceEvent.
* * Fixed shardMap race condition on different threads
* Added *Stats, logging data move rates.
* Added `DD_PHYSICAL_SHARD_MOVE_PROBABILITY` to support hybrid data move.
* Resolved comments.
* fmt.
* Use physical shard move in PhysicalShardMoveTest.
* Enforce physical-shard-move for PhysicalShardMoveTest.
* fmt
2023-05-23 11:18:35 -07:00
bbf15be05f
Knobs to speed up DB open. ( #10301 )
2023-05-22 16:21:05 -07:00
cef93f7d22
knobs ( #10253 )
2023-05-18 14:58:09 -07:00
854464a6af
Hex values in TSS logs and rocksb debuglogs mode knob ( #10231 )
2023-05-16 10:34:58 -07:00
92d1da79a9
RocksDB WAL archive options. ( #10211 )
2023-05-10 21:36:18 -07:00
3dd86d6c22
move IKeyValueStore.h to the client
2023-05-10 15:41:47 -07:00
182d2cafbf
Log physical shard size in KVS
2023-05-10 12:54:59 -07:00
bdb54c5375
initialize counter
2023-05-10 10:23:01 -07:00
1c018f066d
Fix build error.
2023-05-10 10:23:01 -07:00
2d1b5d02e2
Range deletion memory usage improvements ( #10048 )
2023-05-10 10:23:01 -07:00
fa101e1e11
Log background error and add knobs for memory tuning. ( #9841 )
...
* error logger
* recovery mode
2023-05-10 10:23:01 -07:00
fa821c0ed6
Cherrypick #9746
2023-05-10 10:23:01 -07:00
abd45c4486
Cherrypick #9665
2023-05-10 10:23:01 -07:00
8b2f3bcfdc
Rocksdb paranoid file checks knob.
2023-05-04 11:49:38 -07:00
4c75cee2e3
Upgrade to RocksDB 8.1.1 ( #10039 )
...
* Upgrade to RocksDB 8.1.1
* Revert an ASSERT on the previous rocksdb bug. The test was a guard to show the failure.
2023-04-27 12:32:04 -07:00
53fe07a709
Enabling auto_prefix_mode to true in rocksdb. ( #10050 )
2023-04-27 12:11:48 -07:00
0ab691b707
Merge pull request #10002 from sfc-gh-jazhuang/readThrough
...
Fix RangeResult.readThrough misuse
2023-04-27 09:59:11 -07:00
e11f804f96
ShardedRocks checkpoint/restore for physical shard move ( #9752 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* Refactored ShardedRocks checkpoint/restore for psm.
* Complete ShardedRocks::restore.
* dismiss operation_obsolete, and throw actor_cancelled.
* Validate checkpoint when !asKeyValues.
* fmt.
* Don't read from uninitialized physical shard.
* Resolved commments.
* cleanup.
* Added verify_checksum_before_restore for ShardedRocks.
* Added ShardedRocksDB checkpoint/restore unit test.
* Populate CheckpointMetaData::dir in RocksDB.
* Addressed comments.
2023-04-26 09:17:18 -07:00
b7da2ed16c
Fix RangeResult.readThrough misuse
...
Fix `RangeResult.readThrough` misuses:
1. KeyValueStores do not need to set readThrough, as it will not be
serialized and return. Also setting it to the last key of the result
is not right, it should at least be the keyAfter of the last key;
2. Fix NativeAPI doesn't set `RangeResult.more` in a few places;
3. Avoid `tryGetRange()` setting `readThrough` when `more` is false,
which was a workaround for the above item 2;
4. `tryGetRangeFromBlob()` doesn't set `more` but set `readThrough` to
indicate it is end, which was following the same above workaround I
think. Fixed that.
5. `getRangeStream()` is going to set `more` to true and then let the
`readThrough` be it's boundary.
Also added readThrough getter/setter function to validate it's usage.
2023-04-17 21:37:51 -07:00
1d6908d3b4
Changing single key deletions to delete based on number of deletes instead of bytelimit. ( #9867 )
2023-04-03 13:55:58 -07:00
44fba4e84c
RocksDB 7.10.2 version upgrade ( #9828 )
2023-04-03 10:01:22 -07:00
81c3cb8c50
Psm checkpoint ( #9636 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* dismiss operation_obsolete, and throw actor_cancelled.
* fmt.
* Resolved commments.
2023-03-21 09:14:10 -07:00
c677f4c705
tiny fix
2023-03-03 09:04:25 -08:00
e283e067d3
clean and address comments
2023-03-03 09:04:25 -08:00
b5639339ad
address comments
2023-03-03 09:04:25 -08:00
Yao Xiao