Commit Graph

814 Commits

Author SHA1 Message Date
Xiaoxi Wang 47af598749
Merge pull request #7701 from liquid-helium/resume-data-move-timing
Make sure resumeFromDataMoves() starts after resumeFromShards().
2022-07-26 09:45:14 -07:00
He Liu e0b8530ad1 Make sure resumeFromDataMoves() starts after resumeFromShards(). 2022-07-25 22:33:24 -07:00
Bharadwaj V.R dcf09fd691
Move DD queue code over to using movement-reasons rather than priority (#7614)
* Use enum variables to invoke Priority checking

* add an explicit isDataMovementForReadBalancing function

* Set up RelocateShard in terms of data movement reason instead of priority

* Remove isMountainChopperPriority

* Remove isDiskRebalancePriority

* Fix formatting

* Fix misnamed DataMovementReason::TEAM_HEALTHY

Co-authored-by: Zhongxing Zhang <zhongxing.zhang@snowflake.com>
2022-07-25 00:50:37 -07:00
Xiaoxi Wang 01569963da update comments 2022-07-21 09:26:20 -07:00
Xiaoxi Wang 7b8ea1d14f function naming using intention rather than implementation details 2022-07-21 09:19:05 -07:00
Xiaoxi Wang 41032500d0 only do check after bootstraping test 2022-07-20 14:47:54 -07:00
Xiaoxi Wang a5fba6031f resolve conflict upstream/main 2022-07-19 13:55:51 -07:00
Xiaoxi Wang 43099bda0d resolve conflict upstream/main 2022-07-19 13:48:39 -07:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
Xiaoxi Wang 1c543e3b4a add unittest 2022-07-19 11:58:24 -07:00
Xiaoxi Wang 2909614b0c smaller function 2022-07-18 14:21:50 -07:00
Xiaoxi Wang f53c108e4f
Update fdbserver/DataDistribution.actor.cpp 2022-07-14 12:17:34 -07:00
Xiaoxi Wang 42bef3b265
Update fdbserver/DataDistribution.actor.cpp 2022-07-14 12:17:29 -07:00
Xiaoxi Wang 7ee091d1da add test namespace 2022-07-14 12:13:23 -07:00
Xiaoxi Wang 015026f146 add comment; add resumeRelocation 2022-07-14 11:32:27 -07:00
Xiaoxi Wang 1592f6ad34 create DataDistributor::init() method 2022-07-13 12:53:04 -07:00
Xiaoxi Wang a9e9772b98 fix argument bug 2022-07-12 16:49:09 -07:00
Xiaoxi Wang d3686143a1 merge upstream/main 2022-07-12 14:28:49 -07:00
Xiaoxi Wang bf98dc2a2b move waitDataDistributorEnabled to txnProcessor 2022-07-12 11:25:59 -07:00
Xiaoxi Wang 7e067ee4de move getInitialDataDistribution to DDTxnProcessor 2022-07-11 21:54:47 -07:00
Xiaoxi Wang cc185c51c6
Merge pull request #7563 from sfc-gh-xwang/feature/dd-refactor-incremental
[DD Testability] move updateReplicaKey to txnProcessor
2022-07-11 17:11:55 -07:00
Xiaoxi Wang 447eaa1d81 move updateReplicaKey to txnProcessor 2022-07-11 11:16:17 -07:00
Chaoguang Lin 901d988de9
Add a knob SNAPSHOT_ALL_STATEFUL_PROCESSES to snapshot all processes with stateful class type(storage, log, transaction) even if they are not recruited (#7554) 2022-07-08 20:53:49 -07:00
Xiaoxi Wang 9fe76ecf21 format 2022-07-08 16:34:55 -07:00
Xiaoxi Wang b48b1e93f9 remove header dependency; move getDatabaseConfiguration to txnProcessor 2022-07-08 16:01:23 -07:00
Xiaoxi Wang f5a81154ef move takeMoveKeysLock to DDTxnProcessor 2022-07-08 14:11:31 -07:00
He Liu bc5bfaffda
Shard based move (#6981)
* Shard based move.

* Clean up.

* Clear results on retry in getInitialDataDistribution.

* Remove assertion on SHARD_ENCODE_LOCATION_METADATA for compatibility.

* Resolved comments.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-07 20:49:16 -07:00
Chaoguang Lin 29f98f3654
Avoid duplicate snapshot on one process if it serves as multiple roles (#7294)
* Fix comments

* Add simulation value for SERVER_KNOBS->SNAP_CREATE_MAX_TIMEOUT

* A work version with correctness clean

* Remove unnecessay comments; debugging symbols

* Only check secondary address for coordinators, same as before

* Change the trace to SevError and remove the ASSERT(false)

* Remove TLogSnapRequest handling on TlogServer, which is changed to use WorkerSnapRequest

* Add retry for network failures

* Add retry limit for network failures; still allow duplicate snapshots on processes are both tlog and storage to avoid race

* Add retry limit as a knob and make backoff exponentail

* Add getDatabaseConfiguration(Transaction* tr)

* revert back to send request for each role once

* update some comments
2022-06-29 11:23:07 -07:00
Xiaoxi Wang a5054b2beb move getServerListAndProcessClasses to NativeAPI 2022-06-23 15:28:45 -07:00
Bharadwaj V.R 8cf2be030f
Build a TenantCache for use by DD (#7207)
* Add an DD tenant-cache-assembly actor
* Add basic tenant list monitoring for tenant cache. 
* Update DD tenant cache refresh to be more efficient and unit-testable
* Remove the DD prefix in the tenant cache class name (and associated impl and UT class names); there is nothing specific to DD in it; DD uses it; other modules may use it in the future
* Disable DD tenant awareness by default
2022-06-21 16:29:30 -07:00
Xiaoxi Wang 6bb4e341f9
Merge pull request #7110 from sfc-gh-xwang/features/ppw-pause-state
Adding paused/running wiggling status to status json and also the last running/paused timestamp
2022-06-16 14:27:18 -07:00
Bharadwaj V.R 24b2be1c4f Remove last-limited check from DDMountainChopper and DDValleyFiller 2022-05-30 21:57:34 -07:00
Xiaoxi Wang 13a77dd5a2 change priority knob; change PromiseStream to FutureStream; remove comments; add on_sr check 2022-05-25 17:09:34 -07:00
Xiaoxi Wang 2717cee1f9 Merge branch 'features/read-skew' into readaware 2022-05-09 16:12:09 -07:00
Xiaoxi Wang 487a81b62c add lastStateChange 2022-05-08 23:05:43 -07:00
Xiaoxi Wang 13bbd062c4 add storage wiggler state 2022-05-08 22:06:11 -07:00
Xiaoxi Wang a3d0b005dc reset several method use getShardMetrics 2022-05-04 00:00:03 -07:00
Xiaoxi Wang 1723bee639 add fetchTopKShardMetrics to dd tracker 2022-05-03 23:42:09 -07:00
Xiaoxi Wang 940512f208 simplify the storageWiggler ordering; add unittests 2022-05-02 12:25:02 -07:00
Xiaoxi Wang 99d2335220 add storeType to metadata; updateStorageMetadata; combine storeTypeTracker 2022-04-23 00:03:57 -07:00
Xiaoxi Wang ed97a35dc0 Merge branch 'main' into readaware 2022-04-12 16:47:15 -07:00
sfc-gh-tclinkenbeard 41c3bb03c3 Fix storageFaultTolerance calculation 2022-04-08 11:35:24 -07:00
sfc-gh-tclinkenbeard 3fcaf4dda3 Account for storage team size when computing storageFaultTolerance 2022-04-08 10:39:44 -07:00
sfc-gh-tclinkenbeard e27b0d9ab5 Merge remote-tracking branch 'origin/main' into improve-snapshot-fault-tolerance 2022-04-07 23:30:16 -07:00
sfc-gh-tclinkenbeard f4a988fe36 Remove unnecessary call to getDatabaseConfiguration in ddSnapCreateCore 2022-04-07 23:27:59 -07:00
sfc-gh-tclinkenbeard 91930b8040 Remove getMinReplicasRemaining PromiseStream.
Instead, in order to enforce the maximum fault tolerance for snapshots,
update getStorageWorkers to return the number of unavailable storage
servers (instead of throwing an error when unavailable storage servers
exist).
2022-04-07 23:23:23 -07:00
Xiaoxi Wang aba9d85560 merge main 2022-04-06 09:57:52 -07:00
sfc-gh-tclinkenbeard 70f378bacc Restrict write access to getUnhealthyRelocationCount 2022-04-03 23:47:54 -07:00
sfc-gh-tclinkenbeard 33fb6ab983 Prevent coordFaultTolerance from dropping below 0 2022-04-03 23:37:42 -07:00
sfc-gh-tclinkenbeard 4f61c86b69 Add MAX_COORDINATOR_SNAPSHOT_FAULT_TOLERANCE knob 2022-04-03 23:28:57 -07:00