Commit Graph

509 Commits

Author SHA1 Message Date
Yi Wu 845cc62a39
Redwood: fix tree height overgrowth with per-tenant encryption (#9020)
* Fix Redwood tree height overgrowth when EaR and tenant page split are enabled, by removing the buildNewSubtree() logic.
* Fixing incorrect page upper bound for the last page created by writePages() without the buildNewSubtree() logic.
* Enable tenant page split if encryption mode is domain-aware encryption.
* Related test fixes:
  - In simulation, pass encryption mode to storage/Redwood via knobs. This is a workaround to enable testing with Redwood encryption before we correctly pass the encryption mode via db config. Also temporarily disable tenant page split for restart tests.
  - Disable raw access in FuzzApiCorrectness test if domain-aware encryption is enabled, to avoid test timeout
  - Disable encryption for DrUpgradeRestart test, which is likely to fail due to a rare EKP deadlock issue blocking recovery. Will re-enable after the deadlock issue is fixed.
2023-01-06 15:56:37 -08:00
Nim Wijetunga 3a238d638a
Commit Proxy uses Encryption DB Config (#8911)
* add encryption db config

* address pr comments

* address pr comments

* add comments

* cp uses db config

* remove includes

* fix tests

* fix tests

* modify comment

* add encryption enabled method

* modify simulation

* Trigger Build

* change commit proxy encrypt mode fetching

* address pr comments

* address pr comments

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build
2022-12-12 21:23:55 -08:00
sfc-gh-tclinkenbeard 68f14f017c Fix clang 15 compiler warnings 2022-12-08 13:59:37 -08:00
neethuhaneesha f3b0cfb2cb Enabling backup tests with rocksdb storage engine. 2022-11-29 16:05:28 -08:00
Nim Wijetunga 97713cadff
Update Encryption Mode DB Config Values (#8839)
* add encryption db config

* address pr comments

* address pr comments

* add comments

* add comment

* modify check

* change condition

* address pr comments

* simplify check

* address pr comments
2022-11-22 16:33:59 -08:00
Markus Pilman f105cb1809 Merge remote-tracking branch 'origin/main' into bugfixes/machines-attrition-debugging 2022-11-14 10:11:52 -07:00
Chaoguang Lin 9a643286cb
Make the hardcoded timeouts as parameters in test files; Also longRunningTest para to disable Simulation timeout restriction (#8765) 2022-11-09 16:57:51 -08:00
He Liu d28187ca90
Do not set timeout for RocksDB reads in simulation. (#8716)
* Do not set timeout for RocksDB reads in simulation.

* Cleanup.

* Ignore validateStorage test.
2022-11-07 15:59:52 -08:00
Markus Pilman f1fea14255 Merge remote-tracking branch 'origin/main' into bugfixes/machines-attrition-debugging 2022-11-01 13:51:35 -06:00
Lukas Joswiak 5ca2b89bdf Fix simulation issue where process switch was ignored
The simulator tracks only active processes. Rebooted or killed processes
are removed from the list of processes, and only get added back when the
process is rebooted and starts up again. This causes a problem for the
`RebootProcessAndSwitch` kill type, which wants to simultaneously reboot
all machines in a cluster and change their cluster file. If a machine is
currently being rebooted, it will miss the reboot process and switch
command.

The fix is to add a check when a process is being started in simulation.
If the process has had its cluster file changed and the cluster is in a
state where all processes should have had their cluster files reverted
to the original value, the simulator will now send a
`RebootProcessAndSwitch` signal right when the process is started. This
will cause an extra reboot, but should correctly switch the process back
to its original, correct cluster file, allowing the cluster to fully
recover all clusters.

Note that the above issue should only affect simulation, due to how the
simulator tracks processes and handles kill signals.

This commit also adds a field to each process struct to determine
whether the process is being run in a DR cluster in the simulation run.
This is needed because simulation does not differentiate between
processes in different clusters (other than by the IP), and some
processes needed to switch clusters and some simply needed to be
rebooted.
2022-10-27 13:56:13 -07:00
Lukas Joswiak a72066be33 Add simulation support for changing the cluster file 2022-10-27 13:56:13 -07:00
Markus Pilman e7b5b870a3 Merge remote-tracking branch 'origin/main' into bugfixes/machines-attrition-debugging 2022-10-24 15:24:36 -06:00
Jon Fu 60e76ef4a7 Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests 2022-10-20 17:57:22 -07:00
Jingyu Zhou a8391caf23 Revert "Data loss protection v2" 2022-10-20 18:09:58 -05:00
Jon Fu 3533cff94d process '_experimental' suffix in fromString helper 2022-10-20 13:38:13 -07:00
Jon Fu c6998a7185 change tenantModes option to accept array of string 2022-10-20 11:44:03 -07:00
Lukas Joswiak c1e5dd662c Protect availability of DR coordinators in simulation
Otherwise, simulation may reboot and delete a majority of coordinator
processes in the DR cluster, causing `configuration_never_created`
errors.
2022-10-19 16:56:08 -07:00
Jon Fu 04f709c7cb Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests 2022-10-19 14:30:55 -07:00
Jon Fu 702fcd1274 remove allowDisablingTenants flag and add new downgrade test 2022-10-19 14:15:03 -07:00
Lukas Joswiak 7f889c87e3 Fix simulation issue where process switch was ignored
The simulator tracks only active processes. Rebooted or killed processes
are removed from the list of processes, and only get added back when the
process is rebooted and starts up again. This causes a problem for the
`RebootProcessAndSwitch` kill type, which wants to simultaneously reboot
all machines in a cluster and change their cluster file. If a machine is
currently being rebooted, it will miss the reboot process and switch
command.

The fix is to add a check when a process is being started in simulation.
If the process has had its cluster file changed and the cluster is in a
state where all processes should have had their cluster files reverted
to the original value, the simulator will now send a
`RebootProcessAndSwitch` signal right when the process is started. This
will cause an extra reboot, but should correctly switch the process back
to its original, correct cluster file, allowing the cluster to fully
recover all clusters.

Note that the above issue should only affect simulation, due to how the
simulator tracks processes and handles kill signals.

This commit also adds a field to each process struct to determine
whether the process is being run in a DR cluster in the simulation run.
This is needed because simulation does not differentiate between
processes in different clusters (other than by the IP), and some
processes needed to switch clusters and some simply needed to be
rebooted.
2022-10-18 21:37:42 -07:00
Lukas Joswiak 43854c5ac8 Add simulation support for changing the cluster file 2022-10-18 21:37:16 -07:00
Jon Fu 930801b483 allow tenantmodes option to specify an array of possible choices 2022-10-18 10:48:27 -07:00
Jon Fu 7969b08f52 change tenantmode type to int and rewrite some reboot logic 2022-10-17 09:38:11 -07:00
Jon Fu ee0027ba76 remove tenantModeRequired flag and disable defaulttenant for cyclerestart test 2022-10-12 12:32:43 -07:00
Jon Fu a3d42c6073 Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests 2022-10-12 11:59:27 -07:00
Jon Fu 056f938991 add tenantmode to testconfig and change precedence rules for some tenant options 2022-10-12 11:51:50 -07:00
Nim Wijetunga 8ca6397e97
Enable Backup Tests for Encrypted Snapshots (#8419)
* enable encrypted snapshot backup tests

* address pr comments

* remove tenant cache when decoding

* remove client knob
2022-10-12 10:45:30 -07:00
Jon Fu 2c0f936740 Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests 2022-10-11 12:41:03 -07:00
Jon Fu 3c7d17e2bb work in progress commit for defaulttenant in restarting tests 2022-10-11 12:40:36 -07:00
Markus Pilman ea1325a552
Merge pull request #8319 from sfc-gh-tclinkenbeard/add-rare-code-probe-annotation
Add `rare` code probe decoration
2022-10-07 09:39:00 -06:00
Josh Slocum dc917453c1
Targeted blob granules fault injection (#8231) 2022-10-05 13:44:38 -05:00
Hui Liu 1f87286424
Merge pull request #8366 from sfc-gh-huliu/fix7
fix killMachine - make sure we have at least 1 blob worker in a dc
2022-09-30 13:56:32 -07:00
Hui Liu 9db48eb10c fix killMachine - make sure we have at least 1 blob worker in a dc 2022-09-30 11:18:27 -07:00
Lukas Joswiak 020e28a63f Disable RocksDB storage engine in unit test TestConfig 2022-09-30 08:59:47 -07:00
Xiaoxi Wang 3578832c69 merge upstream/main 2022-09-28 14:01:50 -07:00
A.J. Beamon c8bb15e8ee
Merge pull request #8216 from sfc-gh-ajbeamon/backup-support-for-tenants
Include tenant and metacluster metadata in backups
2022-09-28 12:00:24 -07:00
Nim Wijetunga 251afa3610
Set g_simulator fields for restart tests (#8318)
* fix tests

* fix tenant concurrency

* Set g_simulator variables for restart tests

* fix machine attrition for restarting
2022-09-28 10:09:26 -07:00
Xiaoxi Wang 3428fe467a Merge branch 'main' of https://github.com/apple/foundationdb into feature/dd-refactor-simple 2022-09-27 11:55:35 -07:00
Xiaoxi Wang e476e814b2 merge upstream/main 2022-09-27 10:15:26 -07:00
Markus Pilman 437efc60f0
Merge pull request #8199 from sfc-gh-tclinkenbeard/improve-code-coverage
Add `probe::assert::RocksDB` code probe annotation
2022-09-26 16:09:08 -06:00
Markus Pilman c503b2dc53 fix probe::assert::RocksDB 2022-09-26 15:03:35 -06:00
sfc-gh-tclinkenbeard 985958c260 Add rare code probe decoration 2022-09-25 15:28:32 -07:00
Dan Adkins 48e1b06d29 Add comments about the use of protectedAddresses. 2022-09-22 17:01:01 -07:00
A.J. Beamon fda0d7223d Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants. 2022-09-22 10:00:13 -07:00
Xiaoxi Wang 7169a8b132 merge upstream/main 2022-09-15 21:27:26 -07:00
sfc-gh-tclinkenbeard c0fd93869c Add probe::assert::rocksDB code probe annotation 2022-09-15 11:54:02 -07:00
sfc-gh-tclinkenbeard 82adc1e856 Make g_simulator a pointer 2022-09-15 09:00:33 -07:00
Xiaoxi Wang 1f1a66be39 merge upstream/main 2022-09-14 12:32:28 -07:00
Lukas Joswiak cd2bbffa4c Add flag to disable the configuration database
The `--no-config-db` flag, passed to `fdbserver`, will disable the
configuration database. When this flag is specified, no `ConfigNode`s
will be started, the `ConfigBroadcaster` will not be started, and on a
coordinator change no attempt will be made to lock `ConfigNode`s.
2022-09-13 16:53:54 -07:00
Xiaoxi Wang 806d0a8599 merge upstream/main; solve conflicts 2022-09-12 12:34:35 -07:00