d3b7addbc5
Don't crash DD if dest team cannot be found for a restored data move. ( #10651 )
2023-07-21 10:29:49 -07:00
6337125712
Several minor improvements for ShardedRocksDB ( #10520 )
...
* Terminate DD if SHARD_ENCODE_LOCATION_METADATA is not enabled and storage_engine_type is ShardedRocksDB.
* Fixed Error in non-main thread.
* Minor improvements.
2023-06-24 16:07:14 -07:00
8eb526684a
Merge pull request #10437 from halfprice/zhewu/one-shard-per-physical-shard
...
Add an option to limit the number of shard per team
2023-06-20 19:43:31 -07:00
8fec871904
fix an issue where rrs.keys = rd.keys
2023-06-12 13:29:28 -07:00
9a4a4afb43
add coments on the comp function of std::set<RelocateData, std::greater<RelocateData>>
2023-06-12 13:29:28 -07:00
e92bb6ecc2
fix the issue that queuedRelocations not matching with fetchingSourcesQueue.size()
2023-06-12 13:29:28 -07:00
415bf4faea
Psm minor fixes ( #10407 )
...
* Log data move reasons in DataMoveStats.
* Remove a move-in-shard only when it is no longer used.
* Dont remove ranges if restore failed due to actor_cancelled.
2023-06-08 12:10:37 -07:00
e139f5bf90
Correctly handle buggify errors in MGSWaitStorageMetrics
2023-06-07 22:01:33 -07:00
ac16dbd0d8
Fix mock DD incompatible places
2023-06-07 22:01:33 -07:00
0efc2ec2bd
Use PreferWithinShardLimit
2023-06-07 14:45:10 -07:00
6b17f9fcf3
Adding PreferWithinShardLimit option
2023-06-07 14:38:58 -07:00
241907d8ad
Clear move-in-shards before terminating SS.
2023-05-24 19:02:30 -07:00
8ad7ec6fdf
Psm ss ( #9817 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* Refactored ShardedRocks checkpoint/restore for psm.
* Complete ShardedRocks::restore.
* dismiss operation_obsolete, and throw actor_cancelled.
* Validate checkpoint when !asKeyValues.
* fmt.
* Don't read from uninitialized physical shard.
* Resolved commments.
* cleanup.
* Added verify_checksum_before_restore for ShardedRocks.
* Added ShardedRocksDB checkpoint/restore unit test.
* Populate CheckpointMetaData::dir in RocksDB.
* Rename MovingIn as Adding.
* Added StorageServerUtils.
* Added physical shard move in SS.
* Fix on ApplyMetaData, doFetchFile error handling etc.
* Debugging incorrect shard size.
* Create/delete checkpoints only when Physical shard move is enabled.
* Added back SHARD_ENCODE_LOCATION_METADATA.
* Fixed bytesSample incorrect issue.
Essentially dedicated CheckpointRocksDBCF as key-value based checkpoint, will need to add a new format for the file-based checkpoint.
* Cleanup.
* Cleanup & compile rocksdb with 8.1 branch.
* clean up.
* clean up.
* Allowed request_maybe_delivered error type in FetchShard.
* Added FDBRocksDBVersion.h.
* Fixed stuck fetchShard.
* Don't create checkpoint on TSS.
* Upgrade to RocksDB 8.1.1
* Cleanup.
* Fixed accidently deleted db_path and name fields.
* Improved trace event.
* Removed redundants from previuos ShardedrocksDB.
* Cleanup.
* cleanup.
* cleanup.
* reanme `state`.
* Cleanup.
* Removed excessive TraceEvent.
* * Fixed shardMap race condition on different threads
* Added *Stats, logging data move rates.
* Added `DD_PHYSICAL_SHARD_MOVE_PROBABILITY` to support hybrid data move.
* Resolved comments.
* fmt.
* Use physical shard move in PhysicalShardMoveTest.
* Enforce physical-shard-move for PhysicalShardMoveTest.
* fmt
2023-05-23 11:18:35 -07:00
eaa934dac6
Added more logs about shard management. ( #10303 )
2023-05-22 18:00:00 -07:00
34569b0058
Merge pull request #10076 from apple/feature/overallBytesWritten
...
Enable MovingData to show overall moved bytes rather than just one copy
2023-05-04 10:20:03 -07:00
b31dce9f33
add comments to explain difference between completeSources and src
2023-05-03 14:48:22 -07:00
f6da0480fc
add comments to DDEnabledState and rename some methods
2023-05-02 09:40:11 -07:00
880274ab48
add PrepareBlobRestoreRequest, and make DD handle the BlobRestore
...
prepare transactions
add DD Restore Preparing state; actor accept blob migrator requests
Refactor DDEnabledState and PrepareBlobRestoreReply
Improve the initialization, remove unused DDSharedContext method
move prepareBlobRestore to moveKeys
Make context->lock and DataDistributor.lock share the reference; change checkMoveKeysLock to checkPersistenMoveKeysLock; Add more debug trace
fix requestId assignment bug
Add DDEnabledState::sameId method
Throw movekeys_conflict after hybrid restore preparation to force reload
server list and shard mapping; format code; remove unused
methods,definition and comments
fix rebase conflicts
Make DD only load initial Data Distribution after enabled
use dd_config_changed error and throw it in serveBlobMigratorRequests
move empty range check before blob restore to the transaction lock database
Rename BlobRestore*
2023-05-02 09:40:11 -07:00
d81d2019da
Enable MovingData to show overall moved bytes rather than just one copy
2023-04-28 13:32:22 -07:00
d618c07c71
Fix clean up of data moves when encode location metatdata ( #9986 )
...
* fix
* add comments
* fmt
* serialize cancelDataMove
* cancelDataMove should not overwrite a new relocator
* clean
* disable SHARD_ENCODE_LOCATION_METADATA
* add assertions
2023-04-25 15:05:05 -07:00
6b8e35fd19
Merge pull request #9964 from sfc-gh-xwang/fix/main/comments
...
fix HealthMetrics update bug and correct comments about readLoadKSecond()
2023-04-18 16:42:42 -07:00
bb225cb208
add assertion for setParentRange and add more trace
2023-04-18 16:16:46 -07:00
31f2de8cf0
create struct RelocateDecision
2023-04-18 16:16:46 -07:00
5816142e11
add parent_range
2023-04-18 16:16:46 -07:00
50e1f629fe
EligibilityCounter use int type; resolve review comments;
2023-04-17 12:24:17 -07:00
16a8ea0867
move EligibilityCounter to DataDistributionTeam.h; fix ParallelTCInfo compilation error; add MockDD definition of getStorageStats
2023-04-12 09:33:05 -07:00
2419edd874
rename getLoadReadBandwidth to getReadLoad; change the implementation of getReadLoad
2023-04-12 09:33:05 -07:00
27c45f5f45
solve merge conflicts
2023-04-12 09:33:05 -07:00
a78d800e8a
Clean up data move tombstone when DD init ( #9901 )
...
* init
* move cleanup out of getInitDD
* fmt
* address comments
* fixes
2023-04-06 13:48:18 -07:00
8102ac9a41
Serialize concurrent datamoves and their cleanups ( #9421 )
...
* retry when concurrent dm cleanups happens
* cleanup
* fix conflict data move and cleanup
* clean
* make code looks better
* use background cleanup
* fmt
* unset StartMoveShardsFoundConflictingDataMove error
* address comments
2023-04-04 11:38:24 -07:00
9968925c02
Merge pull request #9713 from sfc-gh-zhzhang/refactor-GetTeamRequest
...
refactor GetTeamRequest.
2023-03-30 11:00:16 -07:00
d5dc848626
Use special BlobMigrator UID and remove the data movement restrictions for blob migrator ( #9839 )
2023-03-30 08:25:35 -07:00
72ff00a5ce
change TeamSelect variables names, and update comment
2023-03-29 13:46:04 -07:00
4f4f201138
delete TeamSelect::AvoidMovement, this PR is merely for refactor
2023-03-21 20:46:56 -07:00
93ef3e990e
add the underscore to TeamSelect::value, and move out 'TeamSelect::INVALID'
2023-03-21 15:48:33 -07:00
bad4ee745b
Merge remote-tracking branch 'upstream/main' into refactor-GetTeamRequest
2023-03-21 14:57:28 -07:00
6f687ece4d
change struct to class, add comment for DD Components class
2023-03-21 10:10:54 -07:00
303248b0b7
solve conflicts with MovingWindow PR #9591
2023-03-20 16:06:38 -07:00
dea76f62cf
merge upstream/main without completely clean up conflicts
2023-03-20 16:01:00 -07:00
a258d775c3
Merge pull request #9710 from sfc-gh-etschannen/feature-custom-dd
...
Added the ability to manually create a shard and also increase its replication factor
2023-03-20 15:35:10 -07:00
d2c1b3124e
add a field to show average data movement bytes in MovingData trace ( #9591 )
...
* add a field to show average data movement bytes in MovingData trace
* change variable type
* Make changes to variable/function naming and add more comments
* move rolling window struct to a new file; deal with corner case: dd startup, elements are full
* format
* simplify codes
* modify file/struct name, universal to moving window
* fix typo
* add a new Knob to control MovingWindow::Deque size
* make the general use of dequeSize limit
* format
* format, use space rather than tab
2023-03-20 14:33:32 -07:00
4615926d3f
merge upstream/main and solve conflicts
2023-03-17 21:27:19 -07:00
5f4e3a95c5
change run function use reference parameter rather than pointer
2023-03-17 20:59:18 -07:00
73767501d4
Merge branch 'main' into feature-custom-dd
...
# Conflicts:
# fdbserver/tester.actor.cpp
2023-03-17 10:33:38 -07:00
0f5e75b34b
Added newDataMoveId(). ( #9647 )
...
* Added newDataMoveId().
* Added `ENABLE_DD_PHYSICAL_SHARD_MOVE`
* fmt.
* Replace `teamId` with `shardId`.
2023-03-16 18:06:06 -07:00
ac54962533
code cleanup
2023-03-16 09:47:21 -07:00
8d9e905258
refactor GetTeamRequest
2023-03-15 14:47:34 -07:00
aaf7b9b32b
Added the ability to manually create a shard and also increase its replication factor
2023-03-15 11:26:15 -07:00
0c718f7a04
Fixing double-completing of destination servers in busy map ( #9629 )
2023-03-09 09:43:45 -06:00
26ffcf6b4a
add knob for min read rebalance shard bandwidth
2023-03-02 11:26:27 -08:00
He Liu