Commit Graph

843 Commits

Author SHA1 Message Date
Syed Paymaan Raza a7c536e2ab
Disable ENABLE_REPLICA_CONSISTENCY_CHECK_ON_DATA_MOVEMENT by default, randomly enable in simulation (#12136) 2025-05-09 15:39:34 -07:00
Jingyu Zhou 1693a718ae
Fix assertion failure due to small knob value (#12114)
Increase the buffer size for backup workers and rename the knob.

Seed: -f ./tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml -s 3120109021 -b on
2025-04-29 14:22:47 -07:00
neethuhaneesha 3b98e53c70
Limit the number of clearRanges per commit to a knob value only for rocksdb storage engine and added few metrics. (#12054) (#12062) 2025-04-01 21:38:09 -07:00
Zhe Wang a345d66ec3
A Couple of Fixes and Improvements for BulkLoad/Dump (#12040) 2025-03-19 09:00:36 -07:00
Zhe Wang 0e736c68e7
Allow One BulkloadTask Do Multiple Manifests (#12036) 2025-03-17 11:45:15 -07:00
Zhe Wang d5946157f0
avoid shard merge when bulkload (#12035) 2025-03-15 13:20:51 -07:00
neethuhaneesha 1d9f16bf07
Added compaction knobs. (#12018) 2025-03-12 12:38:23 -07:00
Zhe Wang 51ad8428e0
A Couple for Fixes for BulkDump and RangeLock (#12013)
* fix lockrange test and improve bulk dump

* fix bulkdump stuck error

* remove unnecessary yield when read/write bulk files

* remove unnecessary string creation in read/write bulk files
2025-03-11 15:58:01 -07:00
Syed Paymaan Raza 610ab21936 Increase TLOG_MAX_CREATE_DURATION in simulation 2025-03-10 19:47:23 -07:00
Zhe Wang 79a38c1dc0
Fix RangeLock in BulkDump Test and Avoid Memory Copy For Async Read/Write Bulk Files (#12007) 2025-03-10 15:13:29 -07:00
Jingyu Zhou add710d7f6
Enable TRACK_TLOG_RECOVERY as default (#11987)
Test RECORD_RECOVER_AT_IN_CSTATE and TRACK_TLOG_RECOVERY in buggify with random
on or off.
2025-03-02 19:15:12 -08:00
Zhe Wang 2116547ad3
Improve BulkDump Implementation (#11974)
* bulkdump code refactor

* fix bugs

* improve
2025-02-26 13:58:45 -08:00
Yao Xiao 67b9b5c9f3
Remove per thread histogram in storage engine and fix bugs in range scan. (#11967) 2025-02-25 10:52:46 -08:00
Yao Xiao 76d514bf56
Update shared rocksdb knobs. #11936 (#11938) 2025-02-11 15:41:02 -08:00
Zhe Wang d1efff1511
Improve BulkLoad Implementation (#11929)
* improve bulkload code

* address CI

* disable audit storage replica check and distributed consistency check in bulkload and bulkdump simulation test

* fix ci

* disable waitForQuiescence in bulkload and bulkdump tests
2025-02-06 21:25:49 -08:00
Zhe Wang 0f6fa090ce
Bulkload Engine Support General Storage Engine and Fix BulkLoad Bugs (#11898)
* bulkload support general engine and fix bugs

* add comments

* improve test coverage and fix bug

* nits and address comments

* nit

* nits

* fix data inconsistency bug due to bulkload metadata

* fix ss bulkload task metadata bugs

* nit and fix CI issue

* fix bugs of restore ss bulkload metadata

* use ssBulkLoadMetadata for fetchKey and general kv engine

* cleanup bulkload file for fetchkey

* fix CI issue

* fix simulation stuck due to repeated re-recruitment of unfit dd

* randomly do available space check when finding the dest team for bulkload in simulation

* address conflict

* code clean up

* update BulkDumping.toml same to BulkLoading.toml

* consolidate ss fetchkey and fetchshard failed to read bulkload task metadata

* fix DD bulkload job busy loop bug which causes segfault and test terminate unexpectedly in joshua test

* nit

* fix ss busy loop for bulkload in fetchkey

* use sqlite for bulkload ctest

* fix bulkload ctest stuck issue due to merge and change storage engine to ssd

* fix comments for CC recruit DD

* address comments

* address comments

* add comments

* fix ci format issue

* address comments

* add comments
2025-02-06 12:04:13 -08:00
Zhe Wang b0ff9187ad
fix shardedrocksdb knob and add ENFORCE_SHARDED_ROCKSDB_SIM_IF_AVALIABLE (#11916) 2025-01-29 23:40:37 -08:00
neethuhaneesha 06cdf2e030
Pause store wiggle if all SS does not have minimum available space. (#11905) 2025-01-24 17:29:23 -08:00
Yao Xiao 786e2a6093 Add custom compaction policy based on number of range deletions in file
* compaction policy

* fix build error
2025-01-17 14:14:14 -08:00
Yao Xiao 9ca82b2fda Add knob for direct IO 2025-01-17 14:11:35 -08:00
neethuhaneesha f4c3565aff
Rocksdb manual flush code changes (#11849) 2025-01-17 12:44:44 -08:00
Dan Lambright 78d4490acf
Add ENABLE_VERSION_VECTOR_REPLY_RECOVERY switch (#11889)
Co-authored-by: Dan Lambright <hlambright@apple.com>
2025-01-16 15:10:06 -05:00
Zhe Wang cf7c8f41b2
BulkLoad Job Framework and Co-Testing BulkLoad and BulkDump (#11865)
* add bulkload job framework and fix bugs

* add BulkLoadChecksum, fix CI issue

* nits

* nits

* address comments

* mitigate perpetual wiggle to make sure DD can select a valid team to inject data

* fix submitBulkDumpJob and submitBulkLoadJob

* change remoteRoot to jobRoot

* add comments
2025-01-14 11:28:42 -08:00
Zhe Wang d3532e4478
Improve BulkLoad/Dump implementation (#11842)
* Improve BulkLoad/Dump implementation

* make bulkload test data folder inside simfdb folder

* simplify code

* use manifest in bulkdump metadata

* use manifest in bulkload

* apply bulkload fileset to bulkload and fix bugs of bytesampling value generation

* remove BulkDumpFileFullPathSet

* address comments

* address comments

* address comments
2025-01-06 13:02:23 -08:00
Syed Paymaan Raza c2862b8728
Lower bound version of CC_DEGRADED_PEER_DEGREE_TO_EXCLUDE (#11840) 2024-12-18 23:33:01 -08:00
Yao Xiao 6ccfbf2f42
Pause perpetual storage wiggle when TSS count target is met. (#11823)
* TSS pause

* Add condition
2024-12-16 11:36:37 -08:00
Zhe Wang 31cf800ef9 batch dumping at SS 2024-12-04 11:14:51 -08:00
Zhe Wang 86979abdb0 add parallelism check 2024-12-03 22:01:06 -08:00
Zhe Wang 3bc612e286 add bulkdump metadata and manifest 2024-12-03 22:01:06 -08:00
Zhe Wang 75f7693874 add SS bulkdump 2024-12-03 22:01:06 -08:00
Zhe Wang db8c7d96b8 add bulkdump mode and task scheduling in DD 2024-12-03 22:01:06 -08:00
Syed Paymaan Raza 5a615df85f
Make rocksdb deterministic in simulation (#11790) 2024-11-19 15:42:15 -08:00
neethuhaneesha 1f7a9f4768
Manual flush if the rocksdb flush does not happen within a time interval. (#11791) 2024-11-19 11:57:32 -08:00
neethuhaneesha 5be60d50d3
Histogram sample rate updated to 1. (#11793) 2024-11-19 11:53:16 -08:00
Syed Paymaan Raza 1f8b72c066
Gray failure machine readable status (#11758) 2024-11-12 15:50:08 -08:00
Zhe Wang ca4ab1eca9
Fix traceTooManyEvents and externalTimeouts in BulkLoad test (#11769) 2024-11-11 11:05:43 -08:00
Syed Paymaan Raza 7a5f61cc65
Address feedback in PR #11753 (#11757) 2024-11-05 20:30:40 -08:00
Zhe Wang ab9ce0df15
Cherrypick recent DD changes from release-7.3 (#11754)
* [Release-7.3] TeamRedundant and TeamUnhealthy data moves choose best destination with probability (#11668)

* team redundant and unhealthy data moves can choose best dest with probability

* nits

* nits

* enable wantTrueBestIfMoveout

* fix getteam stuck

* [Release-7.3] Delay team remover when space pivot is low (#11665)

* [Release-7.3] Validate ServerTeam count per server in simulation (#11678)

* validate server team count in simulation

* change naming (not relevant to the PR title)

* address comments and add a new trace event BuildTeamsLastBuildTeamsFailed triggered when buildTeam failed
2024-11-05 18:52:40 -08:00
Syed Paymaan Raza 84fb8f843c
Gray failure allows storage servers to complain (#11753) 2024-11-05 16:53:02 -08:00
Syed Paymaan Raza 7d529ec724
Invalidate gray failure complaints from excluded processes (#11749) 2024-11-04 13:49:25 -08:00
neethuhaneesha 64030de741
Max range deletions knob update to prevent OOMs. (#11738) 2024-10-28 11:45:44 -07:00
Yao Xiao afbcf5ef5f
Enable backward read in consistency checker. (#11721)
* Do backward reads in consistency checker.

* Add knob for read options in consistency checker.
2024-10-26 09:57:36 -07:00
Zhe Wang 43446204ed
Database Per-Range Lock (#11693)
* range lock framework

* improve the framework

* persist to txnStateStore

* fix bugs

* code clean

* code clean

* bug fix

* address comments

* add complex test workload and fix bugs found by the workload

* add workload correctness check and fix bugs

* code clean up

* add random range lock injection

* fix bugs in RandomRangeLock.actor.cpp

* enable random range lock injection in general workloads

* add rangelockcycle test

* disable random range lock in backup workloads

* nits

* add range lock ownership concept

* enable lock ownership to rangeLock

* api deal with tenant

* fix CI

* add test for multiple rangeLock owners

* nits

* address comments and renaming

* address comments
2024-10-23 16:25:56 -07:00
Syed Paymaan Raza 5f480947ad
[fdbserver] Gray failure and simulator improvements related to remote processes (#11717)
* [fdbserver][simulator] Add remoteDesiredTLogCount option

* [fdbserver][simulator] Allow explicitly specifying number of stateless classes in each DC

* [fdbserver][gray_failure] RemoteTLog lagging SS simulation test

* [fdbserver][gray_failure] Consider remote processes + CC inter/intra latency awareness

* [fdbserver][cc] Make processInSameDC O(1)
2024-10-23 13:15:29 -07:00
Yao Xiao 7290369aac
Use a single iterator pool for all physical shards. (#11699)
* Rewrite iterator pool.

* simulation fix
2024-10-15 17:28:54 -07:00
Jingyu Zhou 7c6c8ae095
Merge pull request #11709 from yao-xiao-github/knob-main
Update sharded rocksdb knobs.
2024-10-10 15:11:43 -07:00
Yao Xiao 6a87e6042f update knobs 2024-10-10 13:51:50 -07:00
Zhe Wang fcb0030883 add probability that Memory gets selected 2024-10-10 11:24:40 -07:00
Zhe Wang 7d95b87483 improve the probability that sharded rocksdb is selected in simulation tests 2024-10-10 09:48:23 -07:00
Syed Paymaan Raza 0946f49579
[gray-failure] Remove CC_PAUSE_HEALTH_MONITOR (#11675) 2024-10-07 11:59:19 -07:00