fee87e03b2
Add compile time switch NO_MULTIREGION_TEST. ( #11931 )
...
* Add compile time switch NO_MULTIREGION_TEST. When set, simulation tests
will not create configurations with more than one region. Tests requiring
multiple regions are ignored.
* While the RUN_IGNORED_TESTS setting allows running tests that have been
marked as ignored, this should not apply to multiregion tests. Multiregion
tests must be completely disabled if the NO_MULTIREGION setting is enabled.
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
2025-02-13 14:38:41 -05:00
b0ff9187ad
fix shardedrocksdb knob and add ENFORCE_SHARDED_ROCKSDB_SIM_IF_AVALIABLE ( #11916 )
2025-01-29 23:40:37 -08:00
cf7c8f41b2
BulkLoad Job Framework and Co-Testing BulkLoad and BulkDump ( #11865 )
...
* add bulkload job framework and fix bugs
* add BulkLoadChecksum, fix CI issue
* nits
* nits
* address comments
* mitigate perpetual wiggle to make sure DD can select a valid team to inject data
* fix submitBulkDumpJob and submitBulkLoadJob
* change remoteRoot to jobRoot
* add comments
2025-01-14 11:28:42 -08:00
122cb96b82
Make sharded rocks deterministic in simulation (phase 1) ( #11841 )
2024-12-18 16:05:04 -08:00
5a615df85f
Make rocksdb deterministic in simulation ( #11790 )
2024-11-19 15:42:15 -08:00
31d67404a3
Do not choose two_satellite_fast in simulation tests when version vector unicast enabled. ( #11765 )
...
Co-authored-by: Dan Lambright <hlambright@apple.com>
2024-11-08 16:44:09 -05:00
5f480947ad
[fdbserver] Gray failure and simulator improvements related to remote processes ( #11717 )
...
* [fdbserver][simulator] Add remoteDesiredTLogCount option
* [fdbserver][simulator] Allow explicitly specifying number of stateless classes in each DC
* [fdbserver][gray_failure] RemoteTLog lagging SS simulation test
* [fdbserver][gray_failure] Consider remote processes + CC inter/intra latency awareness
* [fdbserver][cc] Make processInSameDC O(1)
2024-10-23 13:15:29 -07:00
ab93517da4
address comments
2024-10-10 13:11:16 -07:00
fcb0030883
add probability that Memory gets selected
2024-10-10 11:24:40 -07:00
7d95b87483
improve the probability that sharded rocksdb is selected in simulation tests
2024-10-10 09:48:23 -07:00
70f9440e7d
Merge pull request #11648 from sbodagala/main
...
Refactor version vector related recovery version computation logic
2024-09-23 14:06:21 -07:00
2313fdaa0e
Add rocksdb, sharded rocksdb to configure workload ( #11654 )
...
* Add rocksdb, sharded rocksdb to configure workload
Also remove mentioning of ssd-redwood-1-experimental.
* Fix test failure when SHARD_ENCODE_LOCATION_METADATA is off
2024-09-12 21:03:06 -07:00
9ebb1b7391
- Disable logWriteAntiQuorum when version vector is enabled
...
- Refactor the version vector related recovery version computation
logic in order to make it work better with the recovery restart logic
2024-09-11 19:51:41 +00:00
48064f6cf1
Make some codeprobes rare ( #11607 )
...
* Make BlobGranule code probes rare
* Make encryption related code probes rare
* fixup! Fix formatting
2024-08-26 22:33:38 -07:00
3305d2e3ee
fix storage engine selection ( #11586 )
2024-08-20 09:24:30 -07:00
c3e7542cda
Update end year in copyright header
2024-08-02 09:40:11 -07:00
dcebf1a9bc
Add extraStorageMachineCountPerDC config to simulation ( #11529 )
2024-07-26 09:05:41 -07:00
74990e44bd
Bulk Loading Framework ( #11369 )
2024-07-23 14:57:28 -07:00
e9505506df
add consistency-check-urgent-mode to tester process class ( #11484 )
2024-07-09 13:21:37 -07:00
3e3eee98fc
fixup! Reformat source
2024-06-17 11:41:06 -07:00
afc04366fb
Rewrite BUGGIFY related code
...
This is a rewrite of BUGGIFY function/macros. Seems the performance
improved a lot during the simulation, e.g.
fdbserver -r simulation -b on -f ../CycleTest.toml -s 99438
Without this patch:
Unseed: 54646
Elapsed: 494.091327 simsec, 14.586831 real seconds
With this patch:
Unseed: 54646
Elapsed: 494.091327 simsec, 12.580612 real seconds
I expected the improvement but did not expect a ~13% improvement.
2024-06-17 11:41:06 -07:00
0b96d52eec
fixup! Remove a stateful ASSERT statement
2024-06-17 11:41:06 -07:00
a88114c222
Fix typos
2024-02-07 01:16:00 +02:00
b8f1670a0e
Physical shard move tss ( #11057 )
...
* Refactored newDataMoveId() and decodeServerKeysValue().
* Enabled physical shard move for tss.
* Added unit test & cleanup.
* clean up test configs.
2023-11-13 11:34:07 -08:00
0f752473be
Merge branch 'main' into radixtree-production
2023-09-25 09:52:20 +02:00
cb99a8f56f
fixup! Reformat source
2023-08-28 14:56:39 -07:00
5a12d12774
fixup! Enable sharded RocksDB storage engine for PhysicalShardMove
2023-08-28 14:56:39 -07:00
47efd890ad
Better error report when specified storage engine is not supported
2023-08-22 15:52:08 -07:00
a4e4ab6411
Fix a crash error
2023-08-09 13:44:52 -07:00
fff6b745d2
EaR: turn down the probability of enabling encryption in simulation
2023-08-02 10:11:18 -07:00
d75b167bf1
fixup! Reformat source
2023-07-27 14:14:31 -07:00
0036bab8a4
Rename SSD_ROCKSDB_EXPERIMENTAL to WITH_ROCKSDB
2023-07-27 14:04:10 -07:00
c5a8642706
fixup! fix the compile error
2023-07-27 10:56:33 -07:00
e655feb07a
Remove WITH_ROCKSDB_EXPERIMENTAL, enforce using enum for StorageEngine type in simulation
2023-07-27 10:31:48 -07:00
fac2f09641
fixup! Reformat source
2023-07-26 16:00:25 -07:00
874642a15b
fixup! Add message when a storage engine is randomly chosen
2023-07-26 15:46:53 -07:00
b40ba09303
Add a flag for enabling/disabling RocksDB storage engine in simulations
...
Previously we have to modify SimulatedCluster source code to
enable/disable RocksDB storage engine in simulations. Now it is doable
by using cmake flags/ccmake gui by setting the OPTION
ENABLE_ROCKSDB_IN_SIMULATION
ON/OFF.
2023-07-26 15:42:59 -07:00
8ce1796a7a
Stable http ports ( #10604 )
...
* Ensuring HTTP ports are stable for the same servers, and adding a test http server that ensures no other users accidentally talk to it
* fixing warning with werror
* more werror fixes
2023-07-13 14:06:52 -04:00
b16c7518b4
fix worker server reporting IoTimeoutError as injected fault
2023-07-05 13:30:56 -07:00
3ea1ba1648
Remove beta status from RadixTree storage engine
2023-07-05 17:54:54 +02:00
6afd585a3c
Address review: mock assertion and shard number condition
2023-06-07 22:01:33 -07:00
e3c9a1a7f5
Disable same address detection in mock DD and implement MockStorageServer::getStorageMetrics
2023-06-07 22:01:33 -07:00
e139f5bf90
Correctly handle buggify errors in MGSWaitStorageMetrics
2023-06-07 22:01:33 -07:00
b444eb1f22
Make DataDistributor use the configuration object in DDSharedContext; Change Mock test config
2023-06-07 22:01:33 -07:00
8ad7ec6fdf
Psm ss ( #9817 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* Refactored ShardedRocks checkpoint/restore for psm.
* Complete ShardedRocks::restore.
* dismiss operation_obsolete, and throw actor_cancelled.
* Validate checkpoint when !asKeyValues.
* fmt.
* Don't read from uninitialized physical shard.
* Resolved commments.
* cleanup.
* Added verify_checksum_before_restore for ShardedRocks.
* Added ShardedRocksDB checkpoint/restore unit test.
* Populate CheckpointMetaData::dir in RocksDB.
* Rename MovingIn as Adding.
* Added StorageServerUtils.
* Added physical shard move in SS.
* Fix on ApplyMetaData, doFetchFile error handling etc.
* Debugging incorrect shard size.
* Create/delete checkpoints only when Physical shard move is enabled.
* Added back SHARD_ENCODE_LOCATION_METADATA.
* Fixed bytesSample incorrect issue.
Essentially dedicated CheckpointRocksDBCF as key-value based checkpoint, will need to add a new format for the file-based checkpoint.
* Cleanup.
* Cleanup & compile rocksdb with 8.1 branch.
* clean up.
* clean up.
* Allowed request_maybe_delivered error type in FetchShard.
* Added FDBRocksDBVersion.h.
* Fixed stuck fetchShard.
* Don't create checkpoint on TSS.
* Upgrade to RocksDB 8.1.1
* Cleanup.
* Fixed accidently deleted db_path and name fields.
* Improved trace event.
* Removed redundants from previuos ShardedrocksDB.
* Cleanup.
* cleanup.
* cleanup.
* reanme `state`.
* Cleanup.
* Removed excessive TraceEvent.
* * Fixed shardMap race condition on different threads
* Added *Stats, logging data move rates.
* Added `DD_PHYSICAL_SHARD_MOVE_PROBABILITY` to support hybrid data move.
* Resolved comments.
* fmt.
* Use physical shard move in PhysicalShardMoveTest.
* Enforce physical-shard-move for PhysicalShardMoveTest.
* fmt
2023-05-23 11:18:35 -07:00
712fefd59f
Merge pull request #10213 from sfc-gh-ajbeamon/tenant-code-probes
...
Add code probes for tenant and metacluster code
2023-05-15 12:13:00 -07:00
9c081f8a08
Sim http server improvements ( #10217 )
...
* Passes existing tests
* adding http unit test for wrong md5 sum
* Added new HTTPKeyValueStore workload to test long-running http clients
* fixing warnings
2023-05-12 16:33:32 -05:00
d8141c049d
Add code probes for tenant code
2023-05-10 20:44:39 -07:00
77cd187c21
Fix a corner case failure where rebooting flag is forgot to set to true when the process is executing RebootProcessAndSwitch ( #10195 )
2023-05-09 16:18:48 -07:00
50df914c80
Add snapshot restarting tests back to correctness on main ( #10164 )
...
* Enable fault injection;
Ignore MAX_COORDINATOR_SNAPSHOT_FAULT_TOLERANCE in simulation
Fix the issue when there's not fdb.cluster file and disable randomMoveKeys in SnapTest workload
* Move snap tests to from_7.3.0
* Add comments for the change
* Change SevError to SevWarn when Tlog process is not found in the worker map
2023-05-09 11:02:27 -07:00
Dan Lambright