wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
26,043 Commits 36 Branches 335 Tags 299 MiB
Commit Graph

68 Commits

Author SHA1 Message Date
Xiaoxi Wang f6da0480fc add comments to DDEnabledState and rename some methods 2023-05-02 09:40:11 -07:00
Xiaoxi Wang 880274ab48 add PrepareBlobRestoreRequest, and make DD handle the BlobRestore 
prepare transactions

add DD Restore Preparing state; actor accept blob migrator requests

Refactor DDEnabledState and PrepareBlobRestoreReply

Improve the initialization, remove unused DDSharedContext method

move prepareBlobRestore to moveKeys

Make context->lock and DataDistributor.lock share the reference; change checkMoveKeysLock to checkPersistenMoveKeysLock; Add more debug trace

fix requestId assignment bug

Add DDEnabledState::sameId method

Throw movekeys_conflict after hybrid restore preparation to force reload
server list and shard mapping; format code; remove unused
methods,definition and comments

fix rebase conflicts

Make DD only load initial Data Distribution after enabled

use dd_config_changed error and throw it in serveBlobMigratorRequests

move empty range check before blob restore to the transaction lock database

Rename BlobRestore*
 2023-05-02 09:40:11 -07:00
Zhe Wang d618c07c71
Fix clean up of data moves when encode location metatdata (#9986) 
* fix

* add comments

* fmt

* serialize cancelDataMove

* cancelDataMove should not overwrite a new relocator

* clean

* disable SHARD_ENCODE_LOCATION_METADATA

* add assertions
 2023-04-25 15:05:05 -07:00
Ankita Kejriwal 6b8e35fd19
Merge pull request #9964 from sfc-gh-xwang/fix/main/comments 
fix HealthMetrics update bug and correct comments about readLoadKSecond()
 2023-04-18 16:42:42 -07:00
Xiaoxi Wang bb225cb208 add assertion for setParentRange and add more trace 2023-04-18 16:16:46 -07:00
Xiaoxi Wang 31f2de8cf0 create struct RelocateDecision 2023-04-18 16:16:46 -07:00
Xiaoxi Wang 5816142e11 add parent_range 2023-04-18 16:16:46 -07:00
Xiaoxi Wang 50e1f629fe EligibilityCounter use int type; resolve review comments; 2023-04-17 12:24:17 -07:00
Xiaoxi Wang 16a8ea0867 move EligibilityCounter to DataDistributionTeam.h; fix ParallelTCInfo compilation error; add MockDD definition of getStorageStats 2023-04-12 09:33:05 -07:00
Xiaoxi Wang 2419edd874 rename getLoadReadBandwidth to getReadLoad; change the implementation of getReadLoad 2023-04-12 09:33:05 -07:00
Xiaoxi Wang 27c45f5f45 solve merge conflicts 2023-04-12 09:33:05 -07:00
Zhe Wang a78d800e8a
Clean up data move tombstone when DD init (#9901) 
* init

* move cleanup out of getInitDD

* fmt

* address comments

* fixes
 2023-04-06 13:48:18 -07:00
Zhe Wang 8102ac9a41
Serialize concurrent datamoves and their cleanups (#9421) 
* retry when concurrent dm cleanups happens

* cleanup

* fix conflict data move and cleanup

* clean

* make code looks better

* use background cleanup

* fmt

* unset StartMoveShardsFoundConflictingDataMove error

* address comments
 2023-04-04 11:38:24 -07:00
Zhongxing Zhang 9968925c02
Merge pull request #9713 from sfc-gh-zhzhang/refactor-GetTeamRequest 
refactor GetTeamRequest.
 2023-03-30 11:00:16 -07:00
Hui Liu d5dc848626
Use special BlobMigrator UID and remove the data movement restrictions for blob migrator (#9839) 2023-03-30 08:25:35 -07:00
zhongxing.zhang 72ff00a5ce change TeamSelect variables names, and update comment 2023-03-29 13:46:04 -07:00
zhongxing.zhang 4f4f201138 delete TeamSelect::AvoidMovement, this PR is merely for refactor 2023-03-21 20:46:56 -07:00
zhongxing.zhang 93ef3e990e add the underscore to TeamSelect::value, and move out 'TeamSelect::INVALID' 2023-03-21 15:48:33 -07:00
zhongxing.zhang bad4ee745b Merge remote-tracking branch 'upstream/main' into refactor-GetTeamRequest 2023-03-21 14:57:28 -07:00
Xiaoxi Wang 6f687ece4d change struct to class, add comment for DD Components class 2023-03-21 10:10:54 -07:00
Xiaoxi Wang 303248b0b7 solve conflicts with MovingWindow PR #9591 2023-03-20 16:06:38 -07:00
Xiaoxi Wang dea76f62cf merge upstream/main without completely clean up conflicts 2023-03-20 16:01:00 -07:00
Evan Tschannen a258d775c3
Merge pull request #9710 from sfc-gh-etschannen/feature-custom-dd 
Added the ability to manually create a shard and also increase its replication factor
 2023-03-20 15:35:10 -07:00
Zhongxing Zhang d2c1b3124e
add a field to show average data movement bytes in MovingData trace (#9591) 
* add a field to show average data movement bytes in MovingData trace

* change variable type

* Make changes to variable/function naming and add more comments

* move rolling window struct to a new file; deal with corner case: dd startup, elements are full

* format

* simplify codes

* modify file/struct name, universal to moving window

* fix typo

* add a new Knob to control MovingWindow::Deque size

* make the general use of dequeSize limit

* format

* format, use space rather than tab
 2023-03-20 14:33:32 -07:00
Xiaoxi Wang 4615926d3f merge upstream/main and solve conflicts 2023-03-17 21:27:19 -07:00
Xiaoxi Wang 5f4e3a95c5 change run function use reference parameter rather than pointer 2023-03-17 20:59:18 -07:00
Evan Tschannen 73767501d4 Merge branch 'main' into feature-custom-dd 
# Conflicts:
#	fdbserver/tester.actor.cpp
 2023-03-17 10:33:38 -07:00
He Liu 0f5e75b34b
Added newDataMoveId(). (#9647) 
* Added newDataMoveId().

* Added `ENABLE_DD_PHYSICAL_SHARD_MOVE`

* fmt.

* Replace `teamId` with `shardId`.
 2023-03-16 18:06:06 -07:00
Evan Tschannen ac54962533 code cleanup 2023-03-16 09:47:21 -07:00
zhongxing.zhang 8d9e905258 refactor GetTeamRequest 2023-03-15 14:47:34 -07:00
Evan Tschannen aaf7b9b32b Added the ability to manually create a shard and also increase its replication factor 2023-03-15 11:26:15 -07:00
Josh Slocum 0c718f7a04
Fixing double-completing of destination servers in busy map (#9629) 2023-03-09 09:43:45 -06:00
Xiaoxi Wang 26ffcf6b4a add knob for min read rebalance shard bandwidth 2023-03-02 11:26:27 -08:00
Xiaoxi Wang 613391f40f Make DDQueue and DDShardTracker reference counted 2023-02-27 22:36:31 -08:00
Xiaoxi Wang 7e14324cc1 remove dataDistributionQueue actor and replace it with class DDQueue 2023-02-23 17:15:33 -08:00
Xiaoxi Wang 9cda7ebe5a encapsulate DDQueue constructor params into DDQueueInitParams 2023-02-23 15:38:48 -08:00
Xiaoxi Wang cdfaf7d4e0 move DDRelocationQueue to header file 2023-02-23 13:12:38 -08:00
A.J. Beamon 985bbe66d4 Add support for flatMap in optional and erroror. Allow calling map with a lambda without specifying the return type template parameter. 2022-12-15 15:32:02 -08:00
sfc-gh-tclinkenbeard 68f14f017c Fix clang 15 compiler warnings 2022-12-08 13:59:37 -08:00
FoundationDB CI 86d6106dc1
format source code after switch to clang 15 2022-12-08 17:26:45 +00:00
Zhe Wu 7e55522cb0 Fix simulation failure with the new MoveKeysParams 2022-11-29 13:29:02 -08:00
Zhe Wu 3c9130e6d1 Make startMoveShards take a list of key ranges 2022-11-29 13:29:02 -08:00
Zhe Wu f6612ebd00 When selecting physical shard and it is not available for the remote team, retry selecting a new dst physical shard 2022-11-17 15:44:17 -08:00
Zhe Wu 3a5cf32f0b Trying to re-use existing physical shard during data movement 2022-11-16 16:36:00 -08:00
Zhe Wu c09092921b Revert debugging info and apply clang-format 2022-11-15 09:41:21 -08:00
Zhe Wu 18b07368ed Add more detailed reasons for physical shard creation 2022-11-15 09:41:21 -08:00
Zhe Wu f25577f4c2 Apply clang-format 2022-11-11 17:21:32 -08:00
Zhe Wu b90aecad2f Update shard move flag name 2022-11-11 17:21:32 -08:00
Zhe Wu 30af886389 Clean up DD remote team selection in physical shard move 2022-11-11 17:21:32 -08:00
Zhe Wu 0140991d15 Rename NewPhysicalShardReason to RetryFindDstReason 2022-10-24 15:18:20 -07:00