a345d66ec3
A Couple of Fixes and Improvements for BulkLoad/Dump ( #12040 )
2025-03-19 09:00:36 -07:00
0e736c68e7
Allow One BulkloadTask Do Multiple Manifests ( #12036 )
2025-03-17 11:45:15 -07:00
d5946157f0
avoid shard merge when bulkload ( #12035 )
2025-03-15 13:20:51 -07:00
eb0d9f2028
Add Verbose Level for BulkLoad Trace Events ( #12034 )
...
* add level for DDBulkLoad except for datadistribution
* nits
2025-03-14 19:15:41 -07:00
5f9f5358a8
Improve BulkLoad TraceEvent ( #11971 )
...
* improve bulkload event
* fmt
2025-02-25 14:37:21 -08:00
09b9fac5d8
Fix startMoveShards() caused corruption ( #11933 )
...
At commit: fff5439e
2025-02-10 13:00:41 -08:00
0f6fa090ce
Bulkload Engine Support General Storage Engine and Fix BulkLoad Bugs ( #11898 )
...
* bulkload support general engine and fix bugs
* add comments
* improve test coverage and fix bug
* nits and address comments
* nit
* nits
* fix data inconsistency bug due to bulkload metadata
* fix ss bulkload task metadata bugs
* nit and fix CI issue
* fix bugs of restore ss bulkload metadata
* use ssBulkLoadMetadata for fetchKey and general kv engine
* cleanup bulkload file for fetchkey
* fix CI issue
* fix simulation stuck due to repeated re-recruitment of unfit dd
* randomly do available space check when finding the dest team for bulkload in simulation
* address conflict
* code clean up
* update BulkDumping.toml same to BulkLoading.toml
* consolidate ss fetchkey and fetchshard failed to read bulkload task metadata
* fix DD bulkload job busy loop bug which causes segfault and test terminate unexpectedly in joshua test
* nit
* fix ss busy loop for bulkload in fetchkey
* use sqlite for bulkload ctest
* fix bulkload ctest stuck issue due to merge and change storage engine to ssd
* fix comments for CC recruit DD
* address comments
* address comments
* add comments
* fix ci format issue
* address comments
* add comments
2025-02-06 12:04:13 -08:00
ed7a0d44e6
Improve AuditLocationMetadataPostCheck coverage ( #11888 )
...
* improve-auditLocationMetadataPostCheck-coverage
* address comments
* nit
2025-01-17 16:21:29 -08:00
d735250798
Fix the scope of sharded rocks checkpoint determinism flag ( #11893 )
2025-01-16 22:44:48 -08:00
d3532e4478
Improve BulkLoad/Dump implementation ( #11842 )
...
* Improve BulkLoad/Dump implementation
* make bulkload test data folder inside simfdb folder
* simplify code
* use manifest in bulkdump metadata
* use manifest in bulkload
* apply bulkload fileset to bulkload and fix bugs of bytesampling value generation
* remove BulkDumpFileFullPathSet
* address comments
* address comments
* address comments
2025-01-06 13:02:23 -08:00
122cb96b82
Make sharded rocks deterministic in simulation (phase 1) ( #11841 )
2024-12-18 16:05:04 -08:00
c146ee0869
[fdbserver] Use STL contains method and std::find for containment checks ( #11702 )
2024-10-15 11:40:02 -07:00
37a62349f9
remove complete flag in moveShard related functions ( #11642 )
2024-09-09 21:17:45 -07:00
c3e7542cda
Update end year in copyright header
2024-08-02 09:40:11 -07:00
a245b9622c
Fix a couple of simulation failures ( #11543 )
...
* Add usable region check per shard for encode shard location metadata
* nits
* nit
* address comments
* fix SS assertion failed for a wrong data move type generated by an old binary which does not encode the data move type in the data move id
* fix ClientTransactionProfilingCorrectness 7.3 upgrade test considering physical shard move compatibility
* code clean
* split CycleTestRestart in upgrading test from release-7.3
* address comments
* nits
2024-08-01 22:32:32 -07:00
74990e44bd
Bulk Loading Framework ( #11369 )
2024-07-23 14:57:28 -07:00
a733e48048
Merge pull request #11517 from jzhou77/release-notes
...
Fix Wunused-but-set-variable warnings
2024-07-18 13:51:21 -07:00
c630fa2296
Fix wait ( #11474 )
2024-07-18 11:28:34 -07:00
3a3ee247ab
Fix Wunused-but-set-variable warnings
2024-07-17 13:09:32 -07:00
29eab90528
Clean up dd traces ( #11090 )
...
* Clean up DD traces.
* clean up dd traces.
2023-12-06 15:53:04 -08:00
4f167f50be
Adding field length to audit storage trace events. ( #11079 )
2023-12-01 15:45:17 -08:00
1e9c5bb390
Propagate data move reason from DD to SS ( #11063 )
...
* encode reason to data move id
* address comments
* fix data move id decode bug and add assert for data move decode invariant
* address comments
2023-11-15 13:07:11 -08:00
b8f1670a0e
Physical shard move tss ( #11057 )
...
* Refactored newDataMoveId() and decodeServerKeysValue().
* Enabled physical shard move for tss.
* Added unit test & cleanup.
* clean up test configs.
2023-11-13 11:34:07 -08:00
12fe500633
ClusterController watches changes to storage metadata
...
To retrieve storage metadata for every status json request is very expensive
for clusters with a large number of storage servers. So I change the logic so
that ClusterController actively monitors changes to storage metadata, and only
retrieves them when there is a change.
2023-09-15 14:19:04 -07:00
3426fc3c1a
Add DD Security Mode ( #10646 )
...
* dd-security-mode
* address comments
* cleanup
* revise tr option set in loadAndUpdateAuditMetadataWithNewDDId
* address comments
* reset auditStorageInitStarted before DD init
* decouple audit resume and audit launch
* audit launch new request should wait for resuming existing requests
* address comment/clean up/fix
* fix
* fix initAuditMetadata retry
* fix initAuditMetadata retry should reset tr
2023-07-21 17:06:25 -07:00
a6bd2699d2
cleanups, no logic changes. ( #10566 )
2023-06-29 13:20:06 -07:00
37689af3f2
Detect inconsistency of KeyServers and ServerKeys in real time ( #10484 )
...
* add framework
* add audit logic
* refactor audit loc metadata
* address comments
* add realtime audit timeout, add post validation logic
* fix input empty range to compareKeyServersAndServerKeys
* add context for auditKeyServersAndServerKeysInRealTime
* focus on moveShard
* remove space
* address comments
* cleanup
* add audit cleanup
* make validateRangeAssignment simple
* change trace name
* add shardAssigned
* stop DD when inconsistency detected
* fix ci
* small fix
* revert ss and auditUtl and simplify rt audit
* cleanup ss
* tiny change
* address comments and refactor code
* make auditLocationMetadataPreCheck retriable
* handle actor cancel in auditLocationMetadataPreCheck
* rm timeout and add new protection for failure of audit
* fix bugs
* import dataMoveId to validation
* improve trace event
* carefully propagate error and stop DD
* tiny fix
* small change
* remove a state var
* nit
* clean comments
* fmt
2023-06-23 17:40:21 -07:00
7d3cc86860
check serverList before update storage metadata ( #10540 )
2023-06-22 12:02:09 -07:00
883761e8e4
Delete a data move if key range is not consistent. ( #10334 )
2023-06-01 19:14:36 -07:00
5160f91e78
Removed SHARD_ENCODE_LOCATION_METADATA.
2023-05-23 13:39:25 -07:00
8ad7ec6fdf
Psm ss ( #9817 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* Refactored ShardedRocks checkpoint/restore for psm.
* Complete ShardedRocks::restore.
* dismiss operation_obsolete, and throw actor_cancelled.
* Validate checkpoint when !asKeyValues.
* fmt.
* Don't read from uninitialized physical shard.
* Resolved commments.
* cleanup.
* Added verify_checksum_before_restore for ShardedRocks.
* Added ShardedRocksDB checkpoint/restore unit test.
* Populate CheckpointMetaData::dir in RocksDB.
* Rename MovingIn as Adding.
* Added StorageServerUtils.
* Added physical shard move in SS.
* Fix on ApplyMetaData, doFetchFile error handling etc.
* Debugging incorrect shard size.
* Create/delete checkpoints only when Physical shard move is enabled.
* Added back SHARD_ENCODE_LOCATION_METADATA.
* Fixed bytesSample incorrect issue.
Essentially dedicated CheckpointRocksDBCF as key-value based checkpoint, will need to add a new format for the file-based checkpoint.
* Cleanup.
* Cleanup & compile rocksdb with 8.1 branch.
* clean up.
* clean up.
* Allowed request_maybe_delivered error type in FetchShard.
* Added FDBRocksDBVersion.h.
* Fixed stuck fetchShard.
* Don't create checkpoint on TSS.
* Upgrade to RocksDB 8.1.1
* Cleanup.
* Fixed accidently deleted db_path and name fields.
* Improved trace event.
* Removed redundants from previuos ShardedrocksDB.
* Cleanup.
* cleanup.
* cleanup.
* reanme `state`.
* Cleanup.
* Removed excessive TraceEvent.
* * Fixed shardMap race condition on different threads
* Added *Stats, logging data move rates.
* Added `DD_PHYSICAL_SHARD_MOVE_PROBABILITY` to support hybrid data move.
* Resolved comments.
* fmt.
* Use physical shard move in PhysicalShardMoveTest.
* Enforce physical-shard-move for PhysicalShardMoveTest.
* fmt
2023-05-23 11:18:35 -07:00
7ca13d8f9c
support blob restore in fdbrestore ( #10248 )
2023-05-19 14:45:14 -07:00
a5f639f859
Fix psm test ( #10273 )
2023-05-18 14:54:26 -07:00
96322b703c
Remove an assertion for a condition that can happen
...
Revert part of logic introduced in #10110
Reproduction:
./fdbserver-7.1.27 -r simulation -f ./tests/restarting/from_7.1.0_until_7.2.0/ConfigureTestRestart-1.toml -s 1969136453 -b on
./fdbserver -r simulation -f ./tests/restarting/from_7.1.0_until_7.2.0/ConfigureTestRestart-2.toml --restarting -s 1969136454 -b on
2023-05-11 12:39:42 -07:00
96bb675c87
Fix bug: isEnabled() to !isEnabled() ( #10163 )
2023-05-05 15:31:34 -04:00
cdef4523aa
Update a few Data Distribution code probes ( #10110 )
2023-05-03 18:26:51 -07:00
ea9a31b4d7
sameId(requesterId)->!sameId(requesterId)
2023-05-02 09:40:11 -07:00
f6da0480fc
add comments to DDEnabledState and rename some methods
2023-05-02 09:40:11 -07:00
a958436bf5
force blob migrator's ssi to be undesired; remove TCServerInfo from prepareBlobMigrator parameter list
2023-05-02 09:40:11 -07:00
f33a5a5f91
Remove unused function and rename some method
2023-05-02 09:40:11 -07:00
880274ab48
add PrepareBlobRestoreRequest, and make DD handle the BlobRestore
...
prepare transactions
add DD Restore Preparing state; actor accept blob migrator requests
Refactor DDEnabledState and PrepareBlobRestoreReply
Improve the initialization, remove unused DDSharedContext method
move prepareBlobRestore to moveKeys
Make context->lock and DataDistributor.lock share the reference; change checkMoveKeysLock to checkPersistenMoveKeysLock; Add more debug trace
fix requestId assignment bug
Add DDEnabledState::sameId method
Throw movekeys_conflict after hybrid restore preparation to force reload
server list and shard mapping; format code; remove unused
methods,definition and comments
fix rebase conflicts
Make DD only load initial Data Distribution after enabled
use dd_config_changed error and throw it in serveBlobMigratorRequests
move empty range check before blob restore to the transaction lock database
Rename BlobRestore*
2023-05-02 09:40:11 -07:00
3605d8c74c
populate storage metadata for tss
2023-05-01 18:08:08 -07:00
53ee26d758
Changed KeyBackedTypes to an actor file. Added TypedKeySelectors for Map and Set classes and getRange() keySelector methods. Added debug macro for KeyBackedTypes. Rewrote KeyBackedRangeMap using keyselectors on KeyBackedMap.
2023-04-18 22:21:19 -07:00
a4438d4542
Refactored and simplified KeyBackedTypes to base types KeyBackedProperty, KeyBackedMap, and KeyBackedSet. Object and BinaryValue variations are now redefined as customizations of the base types. Added WatchableTrigger for using a key to track a last updated version, supported by all KeyBackedTypes. KeyBackedStruct is renamed to KeyBackedClass contains a WatchableTrigger to pass to contained KeyBackTypes.
2023-04-18 21:48:55 -07:00
a78d800e8a
Clean up data move tombstone when DD init ( #9901 )
...
* init
* move cleanup out of getInitDD
* fmt
* address comments
* fixes
2023-04-06 13:48:18 -07:00
8102ac9a41
Serialize concurrent datamoves and their cleanups ( #9421 )
...
* retry when concurrent dm cleanups happens
* cleanup
* fix conflict data move and cleanup
* clean
* make code looks better
* use background cleanup
* fmt
* unset StartMoveShardsFoundConflictingDataMove error
* address comments
2023-04-04 11:38:24 -07:00
2cb76d06d1
The sharded rocks implementation skipped on TSS handling. Backfill. ( #9759 )
2023-04-04 08:35:30 -07:00
64b6a5d257
Allow boolean parameters to be nested inside of namespaces or classes
2023-03-30 15:09:59 -07:00
81c3cb8c50
Psm checkpoint ( #9636 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* dismiss operation_obsolete, and throw actor_cancelled.
* fmt.
* Resolved commments.
2023-03-21 09:14:10 -07:00
73767501d4
Merge branch 'main' into feature-custom-dd
...
# Conflicts:
# fdbserver/tester.actor.cpp
2023-03-17 10:33:38 -07:00
Zhe Wang