wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
26,674 Commits 36 Branches 335 Tags 297 MiB
Commit Graph

192 Commits

Author SHA1 Message Date
Zhe Wang 3426fc3c1a
Add DD Security Mode (#10646) 
* dd-security-mode

* address comments

* cleanup

* revise tr option set in loadAndUpdateAuditMetadataWithNewDDId

* address comments

* reset auditStorageInitStarted before DD init

* decouple audit resume and audit launch

* audit launch new request should wait for resuming existing requests

* address comment/clean up/fix

* fix

* fix initAuditMetadata retry

* fix initAuditMetadata retry should reset tr
 2023-07-21 17:06:25 -07:00
He Liu a6bd2699d2
cleanups, no logic changes. (#10566) 2023-06-29 13:20:06 -07:00
Zhe Wang 37689af3f2
Detect inconsistency of KeyServers and ServerKeys in real time (#10484) 
* add framework

* add audit logic

* refactor audit loc metadata

* address comments

* add realtime audit timeout, add post validation logic

* fix input empty range to compareKeyServersAndServerKeys

* add context for auditKeyServersAndServerKeysInRealTime

* focus on moveShard

* remove space

* address comments

* cleanup

* add audit cleanup

* make validateRangeAssignment simple

* change trace name

* add shardAssigned

* stop DD when inconsistency detected

* fix ci

* small fix

* revert ss and auditUtl and simplify rt audit

* cleanup ss

* tiny change

* address comments and refactor code

* make auditLocationMetadataPreCheck retriable

* handle actor cancel in auditLocationMetadataPreCheck

* rm timeout and add new protection for failure of audit

* fix bugs

* import dataMoveId to validation

* improve trace event

* carefully propagate error and stop DD

* tiny fix

* small change

* remove a state var

* nit

* clean comments

* fmt
 2023-06-23 17:40:21 -07:00
Xiaoxi Wang 7d3cc86860
check serverList before update storage metadata (#10540) 2023-06-22 12:02:09 -07:00
He Liu 883761e8e4
Delete a data move if key range is not consistent. (#10334) 2023-06-01 19:14:36 -07:00
He Liu 5160f91e78 Removed SHARD_ENCODE_LOCATION_METADATA. 2023-05-23 13:39:25 -07:00
He Liu 8ad7ec6fdf
Psm ss (#9817) 
* Update NativeAPI getCheckpointForRange().

* Implemented checkpoint in SS.

* clean up.

* Disabled StorageServerCheckpointTest.

* Serialized checkpoint creation and deletion.

Simplified checkpoint GC, via deleting CheckpointMetaData::dir.

* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.

* Minor improvements on CheckpointMetaData and DataMoveMetaData.

* fmt.

* Optimized PhysicalShardMove test

cleanup.

* Refactored ShardedRocks checkpoint/restore for psm.

* Complete ShardedRocks::restore.

* dismiss operation_obsolete, and throw actor_cancelled.

* Validate checkpoint when !asKeyValues.

* fmt.

* Don't read from uninitialized physical shard.

* Resolved commments.

* cleanup.

* Added verify_checksum_before_restore for ShardedRocks.

* Added ShardedRocksDB checkpoint/restore unit test.

* Populate CheckpointMetaData::dir in RocksDB.

* Rename MovingIn as Adding.

* Added StorageServerUtils.

* Added physical shard move in SS.

* Fix on ApplyMetaData, doFetchFile error handling etc.

* Debugging incorrect shard size.

* Create/delete checkpoints only when Physical shard move is enabled.

* Added back SHARD_ENCODE_LOCATION_METADATA.

* Fixed bytesSample incorrect issue.

Essentially dedicated CheckpointRocksDBCF as key-value based checkpoint, will need to add a new format for the file-based checkpoint.

* Cleanup.

* Cleanup & compile rocksdb with 8.1 branch.

* clean up.

* clean up.

* Allowed request_maybe_delivered error type in FetchShard.

* Added FDBRocksDBVersion.h.

* Fixed stuck fetchShard.

* Don't create checkpoint on TSS.

* Upgrade to RocksDB 8.1.1

* Cleanup.

* Fixed accidently deleted db_path and name fields.

* Improved trace event.

* Removed redundants from previuos ShardedrocksDB.

* Cleanup.

* cleanup.

* cleanup.

* reanme `state`.

* Cleanup.

* Removed excessive TraceEvent.

* * Fixed shardMap race condition on different threads
* Added *Stats, logging data move rates.
* Added `DD_PHYSICAL_SHARD_MOVE_PROBABILITY` to support hybrid data move.

* Resolved comments.

* fmt.

* Use physical shard move in PhysicalShardMoveTest.

* Enforce physical-shard-move for PhysicalShardMoveTest.

* fmt
 2023-05-23 11:18:35 -07:00
Hui Liu 7ca13d8f9c
support blob restore in fdbrestore (#10248) 2023-05-19 14:45:14 -07:00
He Liu a5f639f859
Fix psm test (#10273) 2023-05-18 14:54:26 -07:00
Jingyu Zhou 96322b703c Remove an assertion for a condition that can happen 
Revert part of logic introduced in #10110

Reproduction:
./fdbserver-7.1.27 -r simulation -f ./tests/restarting/from_7.1.0_until_7.2.0/ConfigureTestRestart-1.toml -s 1969136453 -b on
./fdbserver -r simulation -f ./tests/restarting/from_7.1.0_until_7.2.0/ConfigureTestRestart-2.toml --restarting -s 1969136454 -b on
 2023-05-11 12:39:42 -07:00
Xiaoxi Wang 96bb675c87
Fix bug: isEnabled() to !isEnabled() (#10163) 2023-05-05 15:31:34 -04:00
Ankita Kejriwal cdef4523aa
Update a few Data Distribution code probes (#10110) 2023-05-03 18:26:51 -07:00
Xiaoxi Wang ea9a31b4d7 sameId(requesterId)->!sameId(requesterId) 2023-05-02 09:40:11 -07:00
Xiaoxi Wang f6da0480fc add comments to DDEnabledState and rename some methods 2023-05-02 09:40:11 -07:00
Xiaoxi Wang a958436bf5 force blob migrator's ssi to be undesired; remove TCServerInfo from prepareBlobMigrator parameter list 2023-05-02 09:40:11 -07:00
Xiaoxi Wang f33a5a5f91 Remove unused function and rename some method 2023-05-02 09:40:11 -07:00
Xiaoxi Wang 880274ab48 add PrepareBlobRestoreRequest, and make DD handle the BlobRestore 
prepare transactions

add DD Restore Preparing state; actor accept blob migrator requests

Refactor DDEnabledState and PrepareBlobRestoreReply

Improve the initialization, remove unused DDSharedContext method

move prepareBlobRestore to moveKeys

Make context->lock and DataDistributor.lock share the reference; change checkMoveKeysLock to checkPersistenMoveKeysLock; Add more debug trace

fix requestId assignment bug

Add DDEnabledState::sameId method

Throw movekeys_conflict after hybrid restore preparation to force reload
server list and shard mapping; format code; remove unused
methods,definition and comments

fix rebase conflicts

Make DD only load initial Data Distribution after enabled

use dd_config_changed error and throw it in serveBlobMigratorRequests

move empty range check before blob restore to the transaction lock database

Rename BlobRestore*
 2023-05-02 09:40:11 -07:00
Xiaoxi Wang 3605d8c74c populate storage metadata for tss 2023-05-01 18:08:08 -07:00
Steve Atherton 53ee26d758 Changed KeyBackedTypes to an actor file. Added TypedKeySelectors for Map and Set classes and getRange() keySelector methods. Added debug macro for KeyBackedTypes. Rewrote KeyBackedRangeMap using keyselectors on KeyBackedMap. 2023-04-18 22:21:19 -07:00
Steve Atherton a4438d4542 Refactored and simplified KeyBackedTypes to base types KeyBackedProperty, KeyBackedMap, and KeyBackedSet. Object and BinaryValue variations are now redefined as customizations of the base types. Added WatchableTrigger for using a key to track a last updated version, supported by all KeyBackedTypes. KeyBackedStruct is renamed to KeyBackedClass contains a WatchableTrigger to pass to contained KeyBackTypes. 2023-04-18 21:48:55 -07:00
Zhe Wang a78d800e8a
Clean up data move tombstone when DD init (#9901) 
* init

* move cleanup out of getInitDD

* fmt

* address comments

* fixes
 2023-04-06 13:48:18 -07:00
Zhe Wang 8102ac9a41
Serialize concurrent datamoves and their cleanups (#9421) 
* retry when concurrent dm cleanups happens

* cleanup

* fix conflict data move and cleanup

* clean

* make code looks better

* use background cleanup

* fmt

* unset StartMoveShardsFoundConflictingDataMove error

* address comments
 2023-04-04 11:38:24 -07:00
Michael Stack 2cb76d06d1
The sharded rocks implementation skipped on TSS handling. Backfill. (#9759) 2023-04-04 08:35:30 -07:00
A.J. Beamon 64b6a5d257 Allow boolean parameters to be nested inside of namespaces or classes 2023-03-30 15:09:59 -07:00
He Liu 81c3cb8c50
Psm checkpoint (#9636) 
* Update NativeAPI getCheckpointForRange().

* Implemented checkpoint in SS.

* clean up.

* Disabled StorageServerCheckpointTest.

* Serialized checkpoint creation and deletion.

Simplified checkpoint GC, via deleting CheckpointMetaData::dir.

* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.

* Minor improvements on CheckpointMetaData and DataMoveMetaData.

* fmt.

* Optimized PhysicalShardMove test

cleanup.

* dismiss operation_obsolete, and throw actor_cancelled.

* fmt.

* Resolved commments.
 2023-03-21 09:14:10 -07:00
Evan Tschannen 73767501d4 Merge branch 'main' into feature-custom-dd 
# Conflicts:
#	fdbserver/tester.actor.cpp
 2023-03-17 10:33:38 -07:00
He Liu 0f5e75b34b
Added newDataMoveId(). (#9647) 
* Added newDataMoveId().

* Added `ENABLE_DD_PHYSICAL_SHARD_MOVE`

* fmt.

* Replace `teamId` with `shardId`.
 2023-03-16 18:06:06 -07:00
Evan Tschannen aaf7b9b32b Added the ability to manually create a shard and also increase its replication factor 2023-03-15 11:26:15 -07:00
Jingyu Zhou 6c4a9b5f23 Fix DD stuck when remote DC is dead 
When remote DC is down, the remote team collection of DD can initializing
waiting for the remote to recover (all_tlog_recruited state). However, the
getTeam request can already be served by the remote team collection. So, for
a RelocateShard (data movement such as split, move), it will get a team for
the remote DC. But the data movement can't make progress on the remote team
because the remote DC hasn't recovered yet. Because of the stuck of data
movement, the primary cannot reach the "storage_recovered" state and stay in
accepting_commit state.

The specifc test failure: slow/ApiCorrectness.toml -s 339026305 -b on
at commit:  0edd899d65

In this test, primary DC has 1 SS killed, remote DC has 2 TLog and 2 SS killed.
So the remote is dead, the remaining 2 SSes can't make progress because of the
loss of 2 TLogs. The repairDeadDatacenter() can't reach the "storage_recovered"
state due to DD's failure of moving shards away from the killed SS in the
primary.

The fix is to exclude all remote in repairDeadDatacenter() so that tells DD to
mark all SSes in the remote as unhealthy. Another fix is to return empty
results for getTeam request if the remote team collection is not ready. This
will allow the data movement to continue, essentially remote team is not changed
for the data movement.
 2023-02-10 11:11:07 -08:00
He Liu 33a43cde79
Fixed error type. (#9101) 2023-01-06 10:45:58 -08:00
He Liu 34b3172e34
Improved SHARD_ENCODE_LOCATION_METADATA migration. (#9077) 
* Improved SHARD_ENCODE_LOCATION_METADATA migration.

* Cleanup.

* Cancel itself if a data move finds a conflicting data move. Fixed
transaction reset issue.

* Cancel data move in a retry loop to avoid corrupted mutations.

Co-authored-by: He Liu <heliu@apple.com>
 2023-01-04 12:57:33 -08:00
Zhe Wu 5f51129834 Fix other call site using new MoveKeysParams 2022-11-29 13:29:02 -08:00
Zhe Wu e3410268b5 Make finishMoveShards and DDMockTxnProcessor work with list of key ranges for data movement 2022-11-29 13:29:02 -08:00
Zhe Wu 3c9130e6d1 Make startMoveShards take a list of key ranges 2022-11-29 13:29:02 -08:00
Xiaoxi Wang b2451e600a Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/dataApi 2022-11-15 17:04:49 -08:00
Xiaoxi Wang 4691a35215 format code 2022-11-10 13:11:24 -08:00
Xiaoxi Wang 7c9334121a mark MoveKeysParams const& 2022-11-10 12:51:22 -08:00
Markus Pilman ec78424543 Do not require missing remote destinations to complete a fetch 2022-11-08 12:14:18 -07:00
Xiaoxi Wang 96cf3f855b add rawMoveShard function 2022-11-07 16:47:14 -08:00
He Liu 8d76636a7f
Don't unassign ranges from dest servers if they are skipped. (#8611) 
Co-authored-by: He Liu <heliu@apple.com>
 2022-10-31 12:07:42 -07:00
He Liu 7bb823edbe
Replace KeyRange with std::vector<KeyRange> in DataMoveMetaData and (#8591) 
* Replace KeyRange with std::vector<KeyRange> in DataMoveMetaData and
CheckpointMetaData.

* Checked if ranges.empty().

* fmt.

* Resolved some comments.

Co-authored-by: He Liu <heliu@apple.com>
 2022-10-28 15:22:55 -07:00
Xiaoxi Wang d7a408928a Revert "extract startMoveKeysTransaction method" 
This reverts commit 014398a5a0d9b3442a5c626197f1d65ff6c177e3.
 2022-10-16 21:38:11 -07:00
Xiaoxi Wang 885f8242d9 Revert "extract finishMoveKeysTransaction method" 
This reverts commit 66e91a0faa8badff00ce8b7a296c5ff726be432b.
 2022-10-15 23:02:06 -07:00
Xiaoxi Wang 73efbd7e41 Revert "remove strange naming" 
This reverts commit ba8e8c6719e1edc2e69739c4cabf00863a7d3cfa.
 2022-10-15 22:55:28 -07:00
Xiaoxi Wang e8e6f47e09 format code 2022-10-15 22:52:43 -07:00
Xiaoxi Wang 00c19ec9ed expose rawStartMovement and rawFinishMovement 2022-10-15 22:52:42 -07:00
Xiaoxi Wang cb86b38496 remove strange naming 2022-10-15 22:52:42 -07:00
Xiaoxi Wang 16ca02f048 extract finishMoveKeysTransaction method 2022-10-15 22:52:42 -07:00
Xiaoxi Wang e959bdef8d extract startMoveKeysTransaction method 2022-10-15 22:52:42 -07:00
Markus Pilman ea1325a552
Merge pull request #8319 from sfc-gh-tclinkenbeard/add-rare-code-probe-annotation 
Add `rare` code probe decoration
 2022-10-07 09:39:00 -06:00