Commit Graph

113 Commits

Author SHA1 Message Date
Zhe Wang a345d66ec3
A Couple of Fixes and Improvements for BulkLoad/Dump (#12040) 2025-03-19 09:00:36 -07:00
Zhe Wang 0e736c68e7
Allow One BulkloadTask Do Multiple Manifests (#12036) 2025-03-17 11:45:15 -07:00
Zhe Wang eb0d9f2028
Add Verbose Level for BulkLoad Trace Events (#12034)
* add level for DDBulkLoad except for datadistribution

* nits
2025-03-14 19:15:41 -07:00
Zhe Wang 6156975979
Improve BulkLoad Test Coverage And Fix Bugs (#12009) 2025-03-08 20:26:51 -08:00
Zhe Wang 5f9f5358a8
Improve BulkLoad TraceEvent (#11971)
* improve bulkload event

* fmt
2025-02-25 14:37:21 -08:00
Zhe Wang 94faec13d5
Enable BulkLoad Job to Give Up Unretrievable Task and Fix DDStuck Bug (#11952)
* enable bulkload job to give up unretriable task

* fix ddstuck bug
2025-02-17 17:27:32 -08:00
Zhe Wang d141eea3e1
Allow BulkLoadEngine to Handle Non-Retriable Task (#11950)
* enable-bulkload-engine-accept-unretriable-task

* nit and fmt

* fix bug
2025-02-14 10:52:29 -08:00
Zhe Wang e070698ed0
DataMove Should Decide BulkLoading After Old DataMove Actor Has Been Cleared (#11947)
* fix bulkload bug

* fix CI
2025-02-13 15:35:55 -08:00
Zhe Wang 0f6fa090ce
Bulkload Engine Support General Storage Engine and Fix BulkLoad Bugs (#11898)
* bulkload support general engine and fix bugs

* add comments

* improve test coverage and fix bug

* nits and address comments

* nit

* nits

* fix data inconsistency bug due to bulkload metadata

* fix ss bulkload task metadata bugs

* nit and fix CI issue

* fix bugs of restore ss bulkload metadata

* use ssBulkLoadMetadata for fetchKey and general kv engine

* cleanup bulkload file for fetchkey

* fix CI issue

* fix simulation stuck due to repeated re-recruitment of unfit dd

* randomly do available space check when finding the dest team for bulkload in simulation

* address conflict

* code clean up

* update BulkDumping.toml same to BulkLoading.toml

* consolidate ss fetchkey and fetchshard failed to read bulkload task metadata

* fix DD bulkload job busy loop bug which causes segfault and test terminate unexpectedly in joshua test

* nit

* fix ss busy loop for bulkload in fetchkey

* use sqlite for bulkload ctest

* fix bulkload ctest stuck issue due to merge and change storage engine to ssd

* fix comments for CC recruit DD

* address comments

* address comments

* add comments

* fix ci format issue

* address comments

* add comments
2025-02-06 12:04:13 -08:00
Zhe Wang d3532e4478
Improve BulkLoad/Dump implementation (#11842)
* Improve BulkLoad/Dump implementation

* make bulkload test data folder inside simfdb folder

* simplify code

* use manifest in bulkdump metadata

* use manifest in bulkload

* apply bulkload fileset to bulkload and fix bugs of bytesampling value generation

* remove BulkDumpFileFullPathSet

* address comments

* address comments

* address comments
2025-01-06 13:02:23 -08:00
Zhe Wang ca4ab1eca9
Fix traceTooManyEvents and externalTimeouts in BulkLoad test (#11769) 2024-11-11 11:05:43 -08:00
Zhe Wang ab9ce0df15
Cherrypick recent DD changes from release-7.3 (#11754)
* [Release-7.3] TeamRedundant and TeamUnhealthy data moves choose best destination with probability (#11668)

* team redundant and unhealthy data moves can choose best dest with probability

* nits

* nits

* enable wantTrueBestIfMoveout

* fix getteam stuck

* [Release-7.3] Delay team remover when space pivot is low (#11665)

* [Release-7.3] Validate ServerTeam count per server in simulation (#11678)

* validate server team count in simulation

* change naming (not relevant to the PR title)

* address comments and add a new trace event BuildTeamsLastBuildTeamsFailed triggered when buildTeam failed
2024-11-05 18:52:40 -08:00
Syed Paymaan Raza c146ee0869
[fdbserver] Use STL contains method and std::find for containment checks (#11702) 2024-10-15 11:40:02 -07:00
Syed Paymaan Raza c3e7542cda Update end year in copyright header 2024-08-02 09:40:11 -07:00
Zhe Wang a245b9622c
Fix a couple of simulation failures (#11543)
* Add usable region check per shard for encode shard location metadata

* nits

* nit

* address comments

* fix SS assertion failed for a wrong data move type generated by an old binary which does not encode the data move type in the data move id

* fix ClientTransactionProfilingCorrectness 7.3 upgrade test considering physical shard move compatibility

* code clean

* split CycleTestRestart in upgrading test from release-7.3

* address comments

* nits
2024-08-01 22:32:32 -07:00
Zhe Wang 1a91498161
Fix Compatibility Issue of DataMoveId with BulkLoad type (#11533)
* fix compatibility issue of dataMoveId with BulkLoad type

* nit
2024-07-27 00:19:53 -07:00
Zhe Wang 74990e44bd
Bulk Loading Framework (#11369) 2024-07-23 14:57:28 -07:00
Xiaoge Su c21c6c6ac3 fixup! Another ASSERT with side effect 2024-06-17 11:41:06 -07:00
Yao Xiao 67a588380e
shard size log (#11342) 2024-04-29 13:42:19 -07:00
Zhe Wang 09444c7657
Rebalance Storage Queue (Cherrypick from release-7.1) (#11172)
* cherry-pick-storage-queue-rebalance

* address comments

* address comments
2024-02-07 13:14:08 -08:00
Yao Xiao 9004e2f5ce
Adjust rebalance polling interval based on load. (#11163)
* Adjust rebalance polling interval based on load.

* clang-fmt
2024-02-07 10:50:38 -08:00
Dimitris Apostolou a88114c222
Fix typos 2024-02-07 01:16:00 +02:00
Zhe Wang 970175a8a2
cherrypick storage queue aware getteam (#11154) 2024-01-30 15:15:18 -08:00
Yao Xiao 3af49dc07e
read knob (#11111)
Co-authored-by: yaoxiao-github <yaoxiao@Yaos-MacBook-Pro-14.local>
2024-01-03 14:34:59 -08:00
He Liu 29eab90528
Clean up dd traces (#11090)
* Clean up DD traces.

* clean up dd traces.
2023-12-06 15:53:04 -08:00
Zhe Wang 1e9c5bb390
Propagate data move reason from DD to SS (#11063)
* encode reason to data move id

* address comments

* fix data move id decode bug and add assert for data move decode invariant

* address comments
2023-11-15 13:07:11 -08:00
He Liu b8f1670a0e
Physical shard move tss (#11057)
* Refactored newDataMoveId() and decodeServerKeysValue().

* Enabled physical shard move for tss.

* Added unit test & cleanup.

* clean up test configs.
2023-11-13 11:34:07 -08:00
Zhe Wang be90185dd6
Throttle wiggling data moves (#10953)
* throttle wiggling data moves

* address comments
2023-10-03 09:29:24 -07:00
He Liu d3b7addbc5
Don't crash DD if dest team cannot be found for a restored data move. (#10651) 2023-07-21 10:29:49 -07:00
He Liu 6337125712
Several minor improvements for ShardedRocksDB (#10520)
* Terminate DD if SHARD_ENCODE_LOCATION_METADATA is not enabled and storage_engine_type is ShardedRocksDB.

* Fixed Error in non-main thread.

* Minor improvements.
2023-06-24 16:07:14 -07:00
Zhe Wu 8eb526684a
Merge pull request #10437 from halfprice/zhewu/one-shard-per-physical-shard
Add an option to limit the number of shard per team
2023-06-20 19:43:31 -07:00
zhongxing.zhang 8fec871904 fix an issue where rrs.keys = rd.keys 2023-06-12 13:29:28 -07:00
zhongxing.zhang 9a4a4afb43 add coments on the comp function of std::set<RelocateData, std::greater<RelocateData>> 2023-06-12 13:29:28 -07:00
zhongxing.zhang e92bb6ecc2 fix the issue that queuedRelocations not matching with fetchingSourcesQueue.size() 2023-06-12 13:29:28 -07:00
He Liu 415bf4faea
Psm minor fixes (#10407)
* Log data move reasons in DataMoveStats.

* Remove a move-in-shard only when it is no longer used.

* Dont remove ranges if restore failed due to actor_cancelled.
2023-06-08 12:10:37 -07:00
Xiaoxi Wang e139f5bf90 Correctly handle buggify errors in MGSWaitStorageMetrics 2023-06-07 22:01:33 -07:00
Xiaoxi Wang ac16dbd0d8 Fix mock DD incompatible places 2023-06-07 22:01:33 -07:00
Zhe Wu 0efc2ec2bd Use PreferWithinShardLimit 2023-06-07 14:45:10 -07:00
Zhe Wu 6b17f9fcf3 Adding PreferWithinShardLimit option 2023-06-07 14:38:58 -07:00
He Liu 241907d8ad Clear move-in-shards before terminating SS. 2023-05-24 19:02:30 -07:00
He Liu 8ad7ec6fdf
Psm ss (#9817)
* Update NativeAPI getCheckpointForRange().

* Implemented checkpoint in SS.

* clean up.

* Disabled StorageServerCheckpointTest.

* Serialized checkpoint creation and deletion.

Simplified checkpoint GC, via deleting CheckpointMetaData::dir.

* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.

* Minor improvements on CheckpointMetaData and DataMoveMetaData.

* fmt.

* Optimized PhysicalShardMove test

cleanup.

* Refactored ShardedRocks checkpoint/restore for psm.

* Complete ShardedRocks::restore.

* dismiss operation_obsolete, and throw actor_cancelled.

* Validate checkpoint when !asKeyValues.

* fmt.

* Don't read from uninitialized physical shard.

* Resolved commments.

* cleanup.

* Added verify_checksum_before_restore for ShardedRocks.

* Added ShardedRocksDB checkpoint/restore unit test.

* Populate CheckpointMetaData::dir in RocksDB.

* Rename MovingIn as Adding.

* Added StorageServerUtils.

* Added physical shard move in SS.

* Fix on ApplyMetaData, doFetchFile error handling etc.

* Debugging incorrect shard size.

* Create/delete checkpoints only when Physical shard move is enabled.

* Added back SHARD_ENCODE_LOCATION_METADATA.

* Fixed bytesSample incorrect issue.

Essentially dedicated CheckpointRocksDBCF as key-value based checkpoint, will need to add a new format for the file-based checkpoint.

* Cleanup.

* Cleanup & compile rocksdb with 8.1 branch.

* clean up.

* clean up.

* Allowed request_maybe_delivered error type in FetchShard.

* Added FDBRocksDBVersion.h.

* Fixed stuck fetchShard.

* Don't create checkpoint on TSS.

* Upgrade to RocksDB 8.1.1

* Cleanup.

* Fixed accidently deleted db_path and name fields.

* Improved trace event.

* Removed redundants from previuos ShardedrocksDB.

* Cleanup.

* cleanup.

* cleanup.

* reanme `state`.

* Cleanup.

* Removed excessive TraceEvent.

* * Fixed shardMap race condition on different threads
* Added *Stats, logging data move rates.
* Added `DD_PHYSICAL_SHARD_MOVE_PROBABILITY` to support hybrid data move.

* Resolved comments.

* fmt.

* Use physical shard move in PhysicalShardMoveTest.

* Enforce physical-shard-move for PhysicalShardMoveTest.

* fmt
2023-05-23 11:18:35 -07:00
He Liu eaa934dac6
Added more logs about shard management. (#10303) 2023-05-22 18:00:00 -07:00
Zhongxing Zhang 34569b0058
Merge pull request #10076 from apple/feature/overallBytesWritten
Enable MovingData to show overall moved bytes rather than just one copy
2023-05-04 10:20:03 -07:00
zhongxing.zhang b31dce9f33 add comments to explain difference between completeSources and src 2023-05-03 14:48:22 -07:00
Xiaoxi Wang f6da0480fc add comments to DDEnabledState and rename some methods 2023-05-02 09:40:11 -07:00
Xiaoxi Wang 880274ab48 add PrepareBlobRestoreRequest, and make DD handle the BlobRestore
prepare transactions

add DD Restore Preparing state; actor accept blob migrator requests

Refactor DDEnabledState and PrepareBlobRestoreReply

Improve the initialization, remove unused DDSharedContext method

move prepareBlobRestore to moveKeys

Make context->lock and DataDistributor.lock share the reference; change checkMoveKeysLock to checkPersistenMoveKeysLock; Add more debug trace

fix requestId assignment bug

Add DDEnabledState::sameId method

Throw movekeys_conflict after hybrid restore preparation to force reload
server list and shard mapping; format code; remove unused
methods,definition and comments

fix rebase conflicts

Make DD only load initial Data Distribution after enabled

use dd_config_changed error and throw it in serveBlobMigratorRequests

move empty range check before blob restore to the transaction lock database

Rename BlobRestore*
2023-05-02 09:40:11 -07:00
zhongxing.zhang d81d2019da Enable MovingData to show overall moved bytes rather than just one copy 2023-04-28 13:32:22 -07:00
Zhe Wang d618c07c71
Fix clean up of data moves when encode location metatdata (#9986)
* fix

* add comments

* fmt

* serialize cancelDataMove

* cancelDataMove should not overwrite a new relocator

* clean

* disable SHARD_ENCODE_LOCATION_METADATA

* add assertions
2023-04-25 15:05:05 -07:00
Ankita Kejriwal 6b8e35fd19
Merge pull request #9964 from sfc-gh-xwang/fix/main/comments
fix HealthMetrics update bug and correct comments about readLoadKSecond()
2023-04-18 16:42:42 -07:00
Xiaoxi Wang bb225cb208 add assertion for setParentRange and add more trace 2023-04-18 16:16:46 -07:00