* Separate backup and restore into two workloads
This allows more flexible testing as well as cleaner code.
* Keep sim backup agents after the Backup workload
Otherwise, the Restore workload doesn't have agents, thus can't make progress.
100k partitioned restore tests, i.e., BackupAndRestore.toml and
BackupCorrectnessPartitioned.toml:
20250311-200411-jzhou-9d34d22d5225d6fe
* Remove TransformPartitionedLog parameter from restore()
Instead, infer the flag from backup description.
* Add usePartitionedLog option for Backup workload
This allows us to specify old or new style of backup to be used. Added two
tests that switch between them and randomly choose one backup to restore.
20250314-034057-jzhou-c27ca23b6c69cecf
* Enable buggify in these backup tests
20250314-034541-jzhou-13ed090d0b111474
* Disable backup encryptions for two tests
Because we separate backup and restore into two workloads, they may not choose
the same encryption option, i.e., one encrypted and the other unencrypted.
20250320-013757-jzhou-12b4c8e4504ffd96
* A small refactor
20250321-222324-jzhou-fdcd6f145f3ac0f8
* Fix backup agent being randomly paused
This can cause subsequent backup and restore workload to fail.
20250322-040333-jzhou-15c32299d18f4456
100k backup tests:
20250322-040453-jzhou-2bdb4e0ddc265632
* Refactor RestoreLogDataPartitionedTaskFunc::_execute
* Batch multiple versions together when applying backup mutations
If done version by version, it is inefficient and causes the task to be
interrupted in simulation, thus never finishing the RestoreLogDataPartitionedTaskFunc.
20250324-040712-jzhou-cd8501d3890a6b56
100k backup tests:
20250324-040751-jzhou-8cec93182e6d3acb
* Correct how many log bytes are written during restore
20250326-182125-jzhou-375d243c097c3b5a
20250325-221139-jzhou-5dac71c4525d414c
100k backup tests:
20250326-162525-jzhou-f08e3fc12887a3e9
* Skip consistency check and Quiescence for backup tests
Do these checks in the last [[test]] specified in TOML file.
* Disable backup workers when no active partitioned backups
Currently, when submitting backup, backup workers will be enabled for partitioned backups.
However, we didn't clear the backup worker setting if no partitioned backup is active,
which will cause backup workers to be recruited, but doing nothing.
This PR changes the behavior so that when submitting, aborting, or discontinuing
backups, we'll disable backup workers if there is no active partitioned backup jobs.
20250517-162642-jzhou-4966348e89f1794d
20250517-044345-jzhou-f02f7defca3ea010
* Fix a test assertion failure
The continuous log end version could be less than min restorable version, when
the snapshot is a single version. I.e., min and max restorable versions are the
same.
20250519-042009-jzhou-88a2e0c67e8bed92
100k backup tests 20250519-155545-jzhou-c0aeaaf4a933cff9
* Fix tests where backup agents are not resumed
Also consolidate key updates for pausing backups.
20250520-175022-jzhou-10106ade7e0ad74f
100k backup 20250520-175208-jzhou-15255334e0c57eec
* fdbcli/BulkDumpCommand.actor.cpp
* fdbcli/BulkLoadCommand.actor.cpp
Print out the bulkdump description rather than usage so user
has a chance of figuring out what it is they entered incorrectly.
Make bulkdump and bulkload align by using 'cancel' instead of
'clear' in both and ordering the sub-commands the same for
bulkload and bulkdump. Add more help to the description.
Bulkload was missing mention of the jobid needed
specifying a bulkload.
* documentation/sphinx/source/bulkdump.rst
s/clearBulkDumpJob/cancelBulkDumpJob/
Co-authored-by: stack <stack@duboce.com>
* add bulkload job framework and fix bugs
* add BulkLoadChecksum, fix CI issue
* nits
* nits
* address comments
* mitigate perpetual wiggle to make sure DD can select a valid team to inject data
* fix submitBulkDumpJob and submitBulkLoadJob
* change remoteRoot to jobRoot
* add comments
* Improve BulkLoad/Dump implementation
* make bulkload test data folder inside simfdb folder
* simplify code
* use manifest in bulkdump metadata
* use manifest in bulkload
* apply bulkload fileset to bulkload and fix bugs of bytesampling value generation
* remove BulkDumpFileFullPathSet
* address comments
* address comments
* address comments
* range lock framework
* improve the framework
* persist to txnStateStore
* fix bugs
* code clean
* code clean
* bug fix
* address comments
* add complex test workload and fix bugs found by the workload
* add workload correctness check and fix bugs
* code clean up
* add random range lock injection
* fix bugs in RandomRangeLock.actor.cpp
* enable random range lock injection in general workloads
* add rangelockcycle test
* disable random range lock in backup workloads
* nits
* add range lock ownership concept
* enable lock ownership to rangeLock
* api deal with tenant
* fix CI
* add test for multiple rangeLock owners
* nits
* address comments and renaming
* address comments
* Add rocksdb, sharded rocksdb to configure workload
Also remove mentioning of ssd-redwood-1-experimental.
* Fix test failure when SHARD_ENCODE_LOCATION_METADATA is off
This workload can have timeout error when using locality-based exclusion. The
sequence is:
1. RemoveServerSafely workload exclude locality by processid
2. Attrition reboots the target process, thus changing the processid, because
processid is generated for each worker process at fdbd()
3. RemoveServerSafely waits for the process exclusion, which never succeed
4. Timeout
The fix monitors processid locality changes and reissue the exclusion with the
correct locality.
To reproduce:
seed: -f ./tests/fast/SwizzledRollbackSideband.toml -s 879108103 -b on
commit: a3dbd4baf release-7.1
* list audits
* cancel audits and corresponding tests
* make audit storage dblock aware
* increase audit retry since we are able to cancel
* fix updateAuditState and fdb github ci
* fmt
* fix fdbcli audit_storage and fix CI issue
* fix fdb cli
* address comments
* fmt
Jingyu Zhou