Commit Graph

37 Commits

Author SHA1 Message Date
A.J. Beamon cc68320333 Add testing that a metacluster can be used after a restore. Fix some bugs found by this related to the restore ID and tenant tombstones. 2023-06-21 08:59:22 -07:00
Yanqin Jin 626a8a1a5f SNOW-804199 Support restoring a cluster with a tenant in the error state (#357)
If we restore a cluster and a previously created tenant was not included in the backup, then the tenant will be marked in an error state on the management cluster. It is then up to the operator to resolve the error, generally by deleting the tenant and recreating it if needed.

There is, however, the possibility that we restored a backup that was older than we wanted, and a newer backup would have the tenant. If we tried to restore the newer backup, it would not leave the previously missing tenant in a fully usable state.

We need to have a way to deal with this case. One option is to allow us to clear the error state of a tenant, and that can be performed before (or maybe even after) the second restore.

Test plan:
Joshua test
100K ensemble: 20230613-225414-yajin-439d13ef3c6b3afd fail=0
2023-06-15 22:23:46 -07:00
A.J. Beamon eacf817b2f Add metacluster code probes 2023-05-12 12:32:24 -07:00
Yanqin Jin 01fddb7799
Add `ignore_capacity_limit` to `tenant create` (#10173)
Similar to `tenant configure`, this PR adds `ignore_capacity_limit` as an optional argument to `tenant create`.
This allows the user of fdbcli to create a new tenant on an **assigned** cluster, ignoring the tenant group capacity
on that specific cluster.
When creating a tenant with `ignore_capacity_limit`.
- If the user does not specify `assigned_cluster`, this is an error.
- If the user specifies `assigned_cluster`,
  - user does not specify `tenant_group`, then the new tenant will be an ungrouped tenant on the `assigned_cluster` ignoring the capacity limit
  - user specifies `tenant_group`,
    - if `tenant_group` does not exist, then the new tenant will be created on the assigned cluster and the tenant group will be implicitly created.
    - if `tenant_group` already exists, then additional check will make sure the tenant_group's cluster matches what the user specifies.

Test plan:
Simulation and metacluster_fdbcli_tests.py
---------

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-05-10 13:19:32 -07:00
A.J. Beamon cd54ee9556 Rewrite asserts that check metacluster size 2023-04-26 12:47:39 -07:00
A.J. Beamon eb3f495ac9 Add a utility function for creating a simulated metacluster and use it in most workloads that create metaclusters 2023-04-26 11:01:47 -07:00
A.J. Beamon e27908556a Update the tenant group index to be a tuple of (tenant group name, tenant name, tenant ID) 2023-04-04 14:46:15 -07:00
A.J. Beamon 807646675c Refactor the metacluster project into smaller files, and reorganize the namespaces. Move some metacluster and tenant testing helpers into the metacluster project. 2023-03-30 16:20:09 -07:00
A.J. Beamon e61748c7d5 Move metacluster into its own directory and static library 2023-03-30 16:07:49 -07:00
A.J. Beamon b22ee61497 Speed up the runtime of the metacluster recovery workload by using a simplified configuration for the metacluster 2023-03-24 08:23:59 -07:00
A.J. Beamon cbc330697c Disallow repopulating a management cluster from a data cluster with matching tenant ID prefix unless forced. Remember the largest used tenant ID on the data cluster and use it to update the management cluster tenant ID when force repopulating the same ID. 2023-03-10 15:36:37 -08:00
A.J. Beamon 8c3ee768a2 Add an option to allow exceeding the tenant group capacity limit when changing tenant configuration 2023-02-24 21:01:36 -08:00
Nim Wijetunga c8b7cff10c fix api test 2023-02-24 15:26:59 -08:00
Nim Wijetunga eca98afcb0 metacluster check tenant mode 2023-02-24 13:59:54 -08:00
A.J. Beamon 2b25cfef8b Merge branch 'main' into metacluster-concurrent-restore-testing 2023-02-23 16:06:47 -08:00
Jon Fu 33f8e90f9f
Split tenant group metadata (#9446)
* initial commit to split tenant group metadata

* attempt to fix merge errors

* fix compile errors and adjust existing tests

* fix infinite loop and extra ACTOR tag

* direct assignment instead of store

* direct assign instead of store (missed a few)
2023-02-23 18:11:49 -05:00
A.J. Beamon dcae48cbbd Add concurrent restore testing to the metacluster restore workload 2023-02-23 13:27:20 -08:00
A.J. Beamon ec79ecce73 Add a boolean parameter for ForceRemove; rename ForceJoinNewMetacluster to ForceJoin 2023-02-22 12:26:24 -08:00
A.J. Beamon cb57541c98 Add testing that the metacluster restore dry-run mode doesn't change anything 2023-02-22 08:41:16 -08:00
Jon Fu 2d74d10a91 use MetaclusterAPI namespace over TenantAPI for TenantState 2023-02-21 11:33:36 -08:00
Jon Fu edb7a51b7e Revert "let client supply restore id"
This reverts commit 5fe32b8503.
2023-02-17 14:37:22 -08:00
Jon Fu 5fe32b8503 let client supply restore id 2023-02-17 14:01:58 -08:00
Jon Fu 036078234e fix some leftover issues from branch merge 2023-02-16 12:54:45 -08:00
A.J. Beamon 7284e691fb Fix a few minor restore bugs and add a dry-run mode. Some improvements to the fdbcli output. 2023-02-14 12:28:55 -08:00
A.J. Beamon f3b58a063f Fix some merge issues and review comments 2023-02-13 15:32:44 -08:00
A.J. Beamon 958ff862e0 Fix some merge issues 2023-02-13 12:59:48 -08:00
A.J. Beamon 4b13c9c211 Make a few minor fixes, refactor some code for clarity, and improve throughput of repopulating a management cluster 2023-02-10 10:41:55 -08:00
A.J. Beamon f9a68056ac Add support for modifying a data cluster that is being restored so that we can manage conflicts 2023-02-09 15:33:40 -08:00
A.J. Beamon 2d59c5681d Bug fixes and test improvements for management cluster restoration 2023-02-09 08:42:23 -08:00
A.J. Beamon 0410f7d9a1 Add intial support for restoring a management cluster in a metacluster 2023-02-07 15:08:46 -08:00
A.J. Beamon 6665eb6654 Update cluster recovery for tenant ID changes 2023-01-31 16:06:45 -08:00
A.J. Beamon 795e8aa63f Update the metacluster restore test to run operations while the backup is in flight 2023-01-30 13:34:45 -08:00
A.J. Beamon 56b716040f Various fixes and cleanup 2023-01-30 13:34:45 -08:00
A.J. Beamon f84dd5ac2b Formatting fix 2023-01-30 13:34:45 -08:00
A.J. Beamon 54eb847dcb Fix a bug in the tenant configuration function where the tenant configuration wouldn't get applied if it retried after a successful commit. Fix some formatting and and merge issues. 2023-01-30 13:34:45 -08:00
A.J. Beamon be5b0f451d Expand the restore test; fix some bugs mainly around renaming and changing tenant configuration during a restore 2023-01-30 13:34:45 -08:00
A.J. Beamon 91f5347551 Add some simulated testing to the metacluster restore process; fix various issues found by the testing. 2023-01-30 13:34:18 -08:00