* Separate backup and restore into two workloads
This allows more flexible testing as well as cleaner code.
* Keep sim backup agents after the Backup workload
Otherwise, the Restore workload doesn't have agents, thus can't make progress.
100k partitioned restore tests, i.e., BackupAndRestore.toml and
BackupCorrectnessPartitioned.toml:
20250311-200411-jzhou-9d34d22d5225d6fe
* Remove TransformPartitionedLog parameter from restore()
Instead, infer the flag from backup description.
* Add usePartitionedLog option for Backup workload
This allows us to specify old or new style of backup to be used. Added two
tests that switch between them and randomly choose one backup to restore.
20250314-034057-jzhou-c27ca23b6c69cecf
* Enable buggify in these backup tests
20250314-034541-jzhou-13ed090d0b111474
* Disable backup encryptions for two tests
Because we separate backup and restore into two workloads, they may not choose
the same encryption option, i.e., one encrypted and the other unencrypted.
20250320-013757-jzhou-12b4c8e4504ffd96
* A small refactor
20250321-222324-jzhou-fdcd6f145f3ac0f8
* Fix backup agent being randomly paused
This can cause subsequent backup and restore workload to fail.
20250322-040333-jzhou-15c32299d18f4456
100k backup tests:
20250322-040453-jzhou-2bdb4e0ddc265632
* Refactor RestoreLogDataPartitionedTaskFunc::_execute
* Batch multiple versions together when applying backup mutations
If done version by version, it is inefficient and causes the task to be
interrupted in simulation, thus never finishing the RestoreLogDataPartitionedTaskFunc.
20250324-040712-jzhou-cd8501d3890a6b56
100k backup tests:
20250324-040751-jzhou-8cec93182e6d3acb
* Correct how many log bytes are written during restore
20250326-182125-jzhou-375d243c097c3b5a
20250325-221139-jzhou-5dac71c4525d414c
100k backup tests:
20250326-162525-jzhou-f08e3fc12887a3e9
* Skip consistency check and Quiescence for backup tests
Do these checks in the last [[test]] specified in TOML file.
* Disable backup workers when no active partitioned backups
Currently, when submitting backup, backup workers will be enabled for partitioned backups.
However, we didn't clear the backup worker setting if no partitioned backup is active,
which will cause backup workers to be recruited, but doing nothing.
This PR changes the behavior so that when submitting, aborting, or discontinuing
backups, we'll disable backup workers if there is no active partitioned backup jobs.
20250517-162642-jzhou-4966348e89f1794d
20250517-044345-jzhou-f02f7defca3ea010
* Fix a test assertion failure
The continuous log end version could be less than min restorable version, when
the snapshot is a single version. I.e., min and max restorable versions are the
same.
20250519-042009-jzhou-88a2e0c67e8bed92
100k backup tests 20250519-155545-jzhou-c0aeaaf4a933cff9
* Fix tests where backup agents are not resumed
Also consolidate key updates for pausing backups.
20250520-175022-jzhou-10106ade7e0ad74f
100k backup 20250520-175208-jzhou-15255334e0c57eec
* Add compile time switch NO_MULTIREGION_TEST. When set, simulation tests
will not create configurations with more than one region. Tests requiring
multiple regions are ignored.
* While the RUN_IGNORED_TESTS setting allows running tests that have been
marked as ignored, this should not apply to multiregion tests. Multiregion
tests must be completely disabled if the NO_MULTIREGION setting is enabled.
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* add bulkload job framework and fix bugs
* add BulkLoadChecksum, fix CI issue
* nits
* nits
* address comments
* mitigate perpetual wiggle to make sure DD can select a valid team to inject data
* fix submitBulkDumpJob and submitBulkLoadJob
* change remoteRoot to jobRoot
* add comments
* [fdbserver][simulator] Add remoteDesiredTLogCount option
* [fdbserver][simulator] Allow explicitly specifying number of stateless classes in each DC
* [fdbserver][gray_failure] RemoteTLog lagging SS simulation test
* [fdbserver][gray_failure] Consider remote processes + CC inter/intra latency awareness
* [fdbserver][cc] Make processInSameDC O(1)
* Add rocksdb, sharded rocksdb to configure workload
Also remove mentioning of ssd-redwood-1-experimental.
* Fix test failure when SHARD_ENCODE_LOCATION_METADATA is off
This is a rewrite of BUGGIFY function/macros. Seems the performance
improved a lot during the simulation, e.g.
fdbserver -r simulation -b on -f ../CycleTest.toml -s 99438
Without this patch:
Unseed: 54646
Elapsed: 494.091327 simsec, 14.586831 real seconds
With this patch:
Unseed: 54646
Elapsed: 494.091327 simsec, 12.580612 real seconds
I expected the improvement but did not expect a ~13% improvement.
Previously we have to modify SimulatedCluster source code to
enable/disable RocksDB storage engine in simulations. Now it is doable
by using cmake flags/ccmake gui by setting the OPTION
ENABLE_ROCKSDB_IN_SIMULATION
ON/OFF.
* Ensuring HTTP ports are stable for the same servers, and adding a test http server that ensures no other users accidentally talk to it
* fixing warning with werror
* more werror fixes
* Passes existing tests
* adding http unit test for wrong md5 sum
* Added new HTTPKeyValueStore workload to test long-running http clients
* fixing warnings
Jingyu Zhou