12fe500633
ClusterController watches changes to storage metadata
...
To retrieve storage metadata for every status json request is very expensive
for clusters with a large number of storage servers. So I change the logic so
that ClusterController actively monitors changes to storage metadata, and only
retrieves them when there is a change.
2023-09-15 14:19:04 -07:00
3426fc3c1a
Add DD Security Mode ( #10646 )
...
* dd-security-mode
* address comments
* cleanup
* revise tr option set in loadAndUpdateAuditMetadataWithNewDDId
* address comments
* reset auditStorageInitStarted before DD init
* decouple audit resume and audit launch
* audit launch new request should wait for resuming existing requests
* address comment/clean up/fix
* fix
* fix initAuditMetadata retry
* fix initAuditMetadata retry should reset tr
2023-07-21 17:06:25 -07:00
a6bd2699d2
cleanups, no logic changes. ( #10566 )
2023-06-29 13:20:06 -07:00
37689af3f2
Detect inconsistency of KeyServers and ServerKeys in real time ( #10484 )
...
* add framework
* add audit logic
* refactor audit loc metadata
* address comments
* add realtime audit timeout, add post validation logic
* fix input empty range to compareKeyServersAndServerKeys
* add context for auditKeyServersAndServerKeysInRealTime
* focus on moveShard
* remove space
* address comments
* cleanup
* add audit cleanup
* make validateRangeAssignment simple
* change trace name
* add shardAssigned
* stop DD when inconsistency detected
* fix ci
* small fix
* revert ss and auditUtl and simplify rt audit
* cleanup ss
* tiny change
* address comments and refactor code
* make auditLocationMetadataPreCheck retriable
* handle actor cancel in auditLocationMetadataPreCheck
* rm timeout and add new protection for failure of audit
* fix bugs
* import dataMoveId to validation
* improve trace event
* carefully propagate error and stop DD
* tiny fix
* small change
* remove a state var
* nit
* clean comments
* fmt
2023-06-23 17:40:21 -07:00
7d3cc86860
check serverList before update storage metadata ( #10540 )
2023-06-22 12:02:09 -07:00
883761e8e4
Delete a data move if key range is not consistent. ( #10334 )
2023-06-01 19:14:36 -07:00
5160f91e78
Removed SHARD_ENCODE_LOCATION_METADATA.
2023-05-23 13:39:25 -07:00
8ad7ec6fdf
Psm ss ( #9817 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* Refactored ShardedRocks checkpoint/restore for psm.
* Complete ShardedRocks::restore.
* dismiss operation_obsolete, and throw actor_cancelled.
* Validate checkpoint when !asKeyValues.
* fmt.
* Don't read from uninitialized physical shard.
* Resolved commments.
* cleanup.
* Added verify_checksum_before_restore for ShardedRocks.
* Added ShardedRocksDB checkpoint/restore unit test.
* Populate CheckpointMetaData::dir in RocksDB.
* Rename MovingIn as Adding.
* Added StorageServerUtils.
* Added physical shard move in SS.
* Fix on ApplyMetaData, doFetchFile error handling etc.
* Debugging incorrect shard size.
* Create/delete checkpoints only when Physical shard move is enabled.
* Added back SHARD_ENCODE_LOCATION_METADATA.
* Fixed bytesSample incorrect issue.
Essentially dedicated CheckpointRocksDBCF as key-value based checkpoint, will need to add a new format for the file-based checkpoint.
* Cleanup.
* Cleanup & compile rocksdb with 8.1 branch.
* clean up.
* clean up.
* Allowed request_maybe_delivered error type in FetchShard.
* Added FDBRocksDBVersion.h.
* Fixed stuck fetchShard.
* Don't create checkpoint on TSS.
* Upgrade to RocksDB 8.1.1
* Cleanup.
* Fixed accidently deleted db_path and name fields.
* Improved trace event.
* Removed redundants from previuos ShardedrocksDB.
* Cleanup.
* cleanup.
* cleanup.
* reanme `state`.
* Cleanup.
* Removed excessive TraceEvent.
* * Fixed shardMap race condition on different threads
* Added *Stats, logging data move rates.
* Added `DD_PHYSICAL_SHARD_MOVE_PROBABILITY` to support hybrid data move.
* Resolved comments.
* fmt.
* Use physical shard move in PhysicalShardMoveTest.
* Enforce physical-shard-move for PhysicalShardMoveTest.
* fmt
2023-05-23 11:18:35 -07:00
7ca13d8f9c
support blob restore in fdbrestore ( #10248 )
2023-05-19 14:45:14 -07:00
a5f639f859
Fix psm test ( #10273 )
2023-05-18 14:54:26 -07:00
96322b703c
Remove an assertion for a condition that can happen
...
Revert part of logic introduced in #10110
Reproduction:
./fdbserver-7.1.27 -r simulation -f ./tests/restarting/from_7.1.0_until_7.2.0/ConfigureTestRestart-1.toml -s 1969136453 -b on
./fdbserver -r simulation -f ./tests/restarting/from_7.1.0_until_7.2.0/ConfigureTestRestart-2.toml --restarting -s 1969136454 -b on
2023-05-11 12:39:42 -07:00
96bb675c87
Fix bug: isEnabled() to !isEnabled() ( #10163 )
2023-05-05 15:31:34 -04:00
cdef4523aa
Update a few Data Distribution code probes ( #10110 )
2023-05-03 18:26:51 -07:00
ea9a31b4d7
sameId(requesterId)->!sameId(requesterId)
2023-05-02 09:40:11 -07:00
f6da0480fc
add comments to DDEnabledState and rename some methods
2023-05-02 09:40:11 -07:00
a958436bf5
force blob migrator's ssi to be undesired; remove TCServerInfo from prepareBlobMigrator parameter list
2023-05-02 09:40:11 -07:00
f33a5a5f91
Remove unused function and rename some method
2023-05-02 09:40:11 -07:00
880274ab48
add PrepareBlobRestoreRequest, and make DD handle the BlobRestore
...
prepare transactions
add DD Restore Preparing state; actor accept blob migrator requests
Refactor DDEnabledState and PrepareBlobRestoreReply
Improve the initialization, remove unused DDSharedContext method
move prepareBlobRestore to moveKeys
Make context->lock and DataDistributor.lock share the reference; change checkMoveKeysLock to checkPersistenMoveKeysLock; Add more debug trace
fix requestId assignment bug
Add DDEnabledState::sameId method
Throw movekeys_conflict after hybrid restore preparation to force reload
server list and shard mapping; format code; remove unused
methods,definition and comments
fix rebase conflicts
Make DD only load initial Data Distribution after enabled
use dd_config_changed error and throw it in serveBlobMigratorRequests
move empty range check before blob restore to the transaction lock database
Rename BlobRestore*
2023-05-02 09:40:11 -07:00
3605d8c74c
populate storage metadata for tss
2023-05-01 18:08:08 -07:00
53ee26d758
Changed KeyBackedTypes to an actor file. Added TypedKeySelectors for Map and Set classes and getRange() keySelector methods. Added debug macro for KeyBackedTypes. Rewrote KeyBackedRangeMap using keyselectors on KeyBackedMap.
2023-04-18 22:21:19 -07:00
a4438d4542
Refactored and simplified KeyBackedTypes to base types KeyBackedProperty, KeyBackedMap, and KeyBackedSet. Object and BinaryValue variations are now redefined as customizations of the base types. Added WatchableTrigger for using a key to track a last updated version, supported by all KeyBackedTypes. KeyBackedStruct is renamed to KeyBackedClass contains a WatchableTrigger to pass to contained KeyBackTypes.
2023-04-18 21:48:55 -07:00
a78d800e8a
Clean up data move tombstone when DD init ( #9901 )
...
* init
* move cleanup out of getInitDD
* fmt
* address comments
* fixes
2023-04-06 13:48:18 -07:00
8102ac9a41
Serialize concurrent datamoves and their cleanups ( #9421 )
...
* retry when concurrent dm cleanups happens
* cleanup
* fix conflict data move and cleanup
* clean
* make code looks better
* use background cleanup
* fmt
* unset StartMoveShardsFoundConflictingDataMove error
* address comments
2023-04-04 11:38:24 -07:00
2cb76d06d1
The sharded rocks implementation skipped on TSS handling. Backfill. ( #9759 )
2023-04-04 08:35:30 -07:00
64b6a5d257
Allow boolean parameters to be nested inside of namespaces or classes
2023-03-30 15:09:59 -07:00
81c3cb8c50
Psm checkpoint ( #9636 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* dismiss operation_obsolete, and throw actor_cancelled.
* fmt.
* Resolved commments.
2023-03-21 09:14:10 -07:00
73767501d4
Merge branch 'main' into feature-custom-dd
...
# Conflicts:
# fdbserver/tester.actor.cpp
2023-03-17 10:33:38 -07:00
0f5e75b34b
Added newDataMoveId(). ( #9647 )
...
* Added newDataMoveId().
* Added `ENABLE_DD_PHYSICAL_SHARD_MOVE`
* fmt.
* Replace `teamId` with `shardId`.
2023-03-16 18:06:06 -07:00
aaf7b9b32b
Added the ability to manually create a shard and also increase its replication factor
2023-03-15 11:26:15 -07:00
6c4a9b5f23
Fix DD stuck when remote DC is dead
...
When remote DC is down, the remote team collection of DD can initializing
waiting for the remote to recover (all_tlog_recruited state). However, the
getTeam request can already be served by the remote team collection. So, for
a RelocateShard (data movement such as split, move), it will get a team for
the remote DC. But the data movement can't make progress on the remote team
because the remote DC hasn't recovered yet. Because of the stuck of data
movement, the primary cannot reach the "storage_recovered" state and stay in
accepting_commit state.
The specifc test failure: slow/ApiCorrectness.toml -s 339026305 -b on
at commit: 0edd899d65
2023-02-10 11:11:07 -08:00
33a43cde79
Fixed error type. ( #9101 )
2023-01-06 10:45:58 -08:00
34b3172e34
Improved SHARD_ENCODE_LOCATION_METADATA migration. ( #9077 )
...
* Improved SHARD_ENCODE_LOCATION_METADATA migration.
* Cleanup.
* Cancel itself if a data move finds a conflicting data move. Fixed
transaction reset issue.
* Cancel data move in a retry loop to avoid corrupted mutations.
Co-authored-by: He Liu <heliu@apple.com>
2023-01-04 12:57:33 -08:00
5f51129834
Fix other call site using new MoveKeysParams
2022-11-29 13:29:02 -08:00
e3410268b5
Make finishMoveShards and DDMockTxnProcessor work with list of key ranges for data movement
2022-11-29 13:29:02 -08:00
3c9130e6d1
Make startMoveShards take a list of key ranges
2022-11-29 13:29:02 -08:00
b2451e600a
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/dataApi
2022-11-15 17:04:49 -08:00
4691a35215
format code
2022-11-10 13:11:24 -08:00
7c9334121a
mark MoveKeysParams const&
2022-11-10 12:51:22 -08:00
ec78424543
Do not require missing remote destinations to complete a fetch
2022-11-08 12:14:18 -07:00
96cf3f855b
add rawMoveShard function
2022-11-07 16:47:14 -08:00
8d76636a7f
Don't unassign ranges from dest servers if they are skipped. ( #8611 )
...
Co-authored-by: He Liu <heliu@apple.com>
2022-10-31 12:07:42 -07:00
7bb823edbe
Replace KeyRange with std::vector<KeyRange> in DataMoveMetaData and ( #8591 )
...
* Replace KeyRange with std::vector<KeyRange> in DataMoveMetaData and
CheckpointMetaData.
* Checked if ranges.empty().
* fmt.
* Resolved some comments.
Co-authored-by: He Liu <heliu@apple.com>
2022-10-28 15:22:55 -07:00
d7a408928a
Revert "extract startMoveKeysTransaction method"
...
This reverts commit 014398a5a0d9b3442a5c626197f1d65ff6c177e3.
2022-10-16 21:38:11 -07:00
885f8242d9
Revert "extract finishMoveKeysTransaction method"
...
This reverts commit 66e91a0faa8badff00ce8b7a296c5ff726be432b.
2022-10-15 23:02:06 -07:00
73efbd7e41
Revert "remove strange naming"
...
This reverts commit ba8e8c6719e1edc2e69739c4cabf00863a7d3cfa.
2022-10-15 22:55:28 -07:00
e8e6f47e09
format code
2022-10-15 22:52:43 -07:00
00c19ec9ed
expose rawStartMovement and rawFinishMovement
2022-10-15 22:52:42 -07:00
cb86b38496
remove strange naming
2022-10-15 22:52:42 -07:00
16ca02f048
extract finishMoveKeysTransaction method
2022-10-15 22:52:42 -07:00
e959bdef8d
extract startMoveKeysTransaction method
2022-10-15 22:52:42 -07:00
Jingyu Zhou