Commit Graph

108 Commits

Author SHA1 Message Date
A.J. Beamon cc68320333 Add testing that a metacluster can be used after a restore. Fix some bugs found by this related to the restore ID and tenant tombstones. 2023-06-21 08:59:22 -07:00
Yanqin Jin 686bd52953
Update a test to check data cluster's version only when registered (#10275)
This PR fixes an unitialized-value error reported by valgrind.

Steps to reproduce:
Ensemble: 20230516-050214-nightly_valgrind_main_x86_64_apple-b77b4ac44a15911e
Profile: team valgrind
Commit hash: 0ce1ab3162 on apple/main
Command: devRetryCorrectnessTest valgrind bin/fdbserver -r simulation -f tests/slow/MetaclusterManagement.toml -s 1695330198 -b off --crash --trace_format json
2023-05-18 15:28:31 -07:00
A.J. Beamon eacf817b2f Add metacluster code probes 2023-05-12 12:32:24 -07:00
Yanqin Jin 01fddb7799
Add `ignore_capacity_limit` to `tenant create` (#10173)
Similar to `tenant configure`, this PR adds `ignore_capacity_limit` as an optional argument to `tenant create`.
This allows the user of fdbcli to create a new tenant on an **assigned** cluster, ignoring the tenant group capacity
on that specific cluster.
When creating a tenant with `ignore_capacity_limit`.
- If the user does not specify `assigned_cluster`, this is an error.
- If the user specifies `assigned_cluster`,
  - user does not specify `tenant_group`, then the new tenant will be an ungrouped tenant on the `assigned_cluster` ignoring the capacity limit
  - user specifies `tenant_group`,
    - if `tenant_group` does not exist, then the new tenant will be created on the assigned cluster and the tenant group will be implicitly created.
    - if `tenant_group` already exists, then additional check will make sure the tenant_group's cluster matches what the user specifies.

Test plan:
Simulation and metacluster_fdbcli_tests.py
---------

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-05-10 13:19:32 -07:00
Yanqin Jin f1e22f9a1e
Account for 'tenant_locked' error in MetaclusterManagementWorkload (#10175)
In MetaclusterManagementWorkload, if we lock a tenant that has already been locked with a different lockId, it's possible that a tenant_locked error is thrown. We need to account for this instead of terminating the test.

Test plan:
Regular simulation and a deterministic reproduction

Branch: https://github.com/sfc-gh-yajin/foundationdb/tree/snow-791039
Commit hash: 7356504723
Profile: team
Command: devRetryCorrectnessTest bin/fdbserver -r simulation -f tests/slow/MetaclusterManagement.toml -s 3475230174 -b off --crash --trace_format json
2023-05-08 13:38:05 -07:00
Yanqin Jin 8b1fe728be
Add configuration option `auto_tenant_assignment` to data clusters (#10058)
This PR adds auto_tenant_assignment option to register/configure data clusters.
Setting auto_tenant_assignment to disabled means the data cluster is a dedicated one and won't be
used for auto tenant assignment. This option is enabled by default (allowing auto tenant assignment).

Test plan:
simulation tests and metacluster_fdbcli_tests.py
---------

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-05-01 21:58:49 -07:00
A.J. Beamon b258159d3a Change enum capitalization. Improve error reporting if we cannot read metacluster registration when fetching metacluster metrics. Improve timeliness of metacluster metrics updates. 2023-05-01 11:21:42 -07:00
A.J. Beamon 882ec9998b Add testing for the metacluster version feature to metacluster management workload. This also adds additional operations to the workload (create/decommission metacluster, lock tenant) to ensure they are covered by this testing. 2023-04-27 10:18:34 -07:00
A.J. Beamon 807646675c Refactor the metacluster project into smaller files, and reorganize the namespaces. Move some metacluster and tenant testing helpers into the metacluster project. 2023-03-30 16:20:09 -07:00
A.J. Beamon e61748c7d5 Move metacluster into its own directory and static library 2023-03-30 16:07:49 -07:00
A.J. Beamon 64b6a5d257 Allow boolean parameters to be nested inside of namespaces or classes 2023-03-30 15:09:59 -07:00
A.J. Beamon d39cda610a Merge branch 'main' into metacluster-improvements
# Conflicts:
#	fdbcli/TenantCommands.actor.cpp
2023-03-13 15:58:39 -07:00
A.J. Beamon cbc330697c Disallow repopulating a management cluster from a data cluster with matching tenant ID prefix unless forced. Remember the largest used tenant ID on the data cluster and use it to update the management cluster tenant ID when force repopulating the same ID. 2023-03-10 15:36:37 -08:00
A.J. Beamon de5f2c0fee Disallow cluster names that start with the `\xff` byte 2023-03-07 11:46:34 -08:00
A.J. Beamon 2898a95c81 Fix two metacluster issues:
1. When retrying the transaction to register a restoring cluster, don't choose a new ID if the current ID matches the one recorded for the restoring cluster
2. A metacluster test was incorrectly handling the case where a transaction was retried with unknown result and had committed successfully
2023-02-28 15:40:04 -08:00
A.J. Beamon 8c3ee768a2 Add an option to allow exceeding the tenant group capacity limit when changing tenant configuration 2023-02-24 21:01:36 -08:00
A.J. Beamon 1c71056e26
Merge pull request #9479 from sfc-gh-nwijetunga/nim/enforce-metacluster-tenant-mode
Enforce Disabled Tenant Mode in Metacluster
2023-02-24 19:27:57 -08:00
Nim Wijetunga c8b7cff10c fix api test 2023-02-24 15:26:59 -08:00
Nim Wijetunga eca98afcb0 metacluster check tenant mode 2023-02-24 13:59:54 -08:00
A.J. Beamon 2d936acbdd Fix memory error from using invalidated iterator 2023-02-24 10:25:38 -08:00
A.J. Beamon 344f6977c9 Restores can now throw a cluster_already_exists error in the metacluster management workload if we timeout a restore and have to retry 2023-02-23 20:32:18 -08:00
Jon Fu 33f8e90f9f
Split tenant group metadata (#9446)
* initial commit to split tenant group metadata

* attempt to fix merge errors

* fix compile errors and adjust existing tests

* fix infinite loop and extra ACTOR tag

* direct assignment instead of store

* direct assign instead of store (missed a few)
2023-02-23 18:11:49 -05:00
A.J. Beamon 9e9a31c0f1 Use error variable consistently 2023-02-23 11:27:53 -08:00
A.J. Beamon a76af5c696 Improve restore support in the metacluster management workload 2023-02-22 17:42:36 -08:00
A.J. Beamon a5a8c57a38 Fix some merge issues and missing updates to use new boolean parameters 2023-02-22 12:52:58 -08:00
Jon Fu 2d74d10a91 use MetaclusterAPI namespace over TenantAPI for TenantState 2023-02-21 11:33:36 -08:00
Jon Fu f27c3e24a8 fix code formatting 2023-02-21 11:09:14 -08:00
Jon Fu 428eb07766 Merge branch 'main' of github.com:apple/foundationdb into split-tenant-metadata 2023-02-21 10:11:35 -08:00
A.J. Beamon 3163201201 Restore ID fixes: we weren't generating a restore ID; we weren't setting the restore ID on the management cluster in some restore modes; it is possible in some test scenarios to encounter a restore conflict, in which case we need to retry. 2023-02-17 21:20:06 -08:00
Jon Fu edb7a51b7e Revert "let client supply restore id"
This reverts commit 5fe32b8503.
2023-02-17 14:37:22 -08:00
Jon Fu 5fe32b8503 let client supply restore id 2023-02-17 14:01:58 -08:00
Jon Fu 0d7b6d626b update restoreCluster test to account for conflicting_restore 2023-02-17 10:36:28 -08:00
Jon Fu 53fc43a3a6 Merge branch 'main' of github.com:apple/foundationdb into split-tenant-metadata 2023-02-16 14:46:03 -08:00
Yanqin Jin c3d6ae0213
Throw invalid_tenant_configuration when changing assigned cluster (#9350)
Since we currently do not support tenant movement, we should as well explicitly disallow changing the assigned
cluster of a tenant during configuration by throwing `invalid_tenant_configuration` for now.

Test plan:
- add coverage for changing assigned cluster during tenant configuration
- fdbcli
- simulation tests
2023-02-16 14:20:59 -08:00
Jon Fu 036078234e fix some leftover issues from branch merge 2023-02-16 12:54:45 -08:00
Jon Fu ab15478ef9 Merge branch 'main' of github.com:apple/foundationdb into split-tenant-metadata 2023-02-16 10:04:07 -08:00
Jon Fu 8f4aec7d7f address review comments and fix some test errors 2023-02-15 15:58:19 -08:00
A.J. Beamon 545e08e1af Try again to fix the logic in choosing the assigned cluster in the metacluster management test 2023-02-14 21:39:42 -08:00
A.J. Beamon 7284e691fb Fix a few minor restore bugs and add a dry-run mode. Some improvements to the fdbcli output. 2023-02-14 12:28:55 -08:00
A.J. Beamon 98407809d9 Merge branch 'main' into metacluster-mgmt-restore
# Conflicts:
#	fdbcli/MetaclusterCommands.actor.cpp
#	fdbclient/Metacluster.cpp
#	fdbclient/include/fdbclient/MetaclusterManagement.actor.h
#	fdbserver/workloads/MetaclusterManagementWorkload.actor.cpp
#	tests/CMakeLists.txt
2023-02-13 12:30:33 -08:00
A.J. Beamon 0127dd4b5a
Merge pull request #9356 from sfc-gh-ajbeamon/metacluster-concurrency-testing
Add metacluster concurrency test and fix various bugs that it found
2023-02-13 11:57:47 -08:00
A.J. Beamon 93b1e04aa9
Merge pull request #9355 from sfc-gh-ajbeamon/fix-assigned-cluster-test-retry-logic
Fix logic in MetaclusterManagementWorkload when retrying a tenant creation with an invalid assigned cluster
2023-02-13 10:43:09 -08:00
A.J. Beamon e6021f8326 Add Jon's metacluster concurrency test and fix various bugs that it found 2023-02-11 15:15:32 -08:00
Xiaoxi Wang a9c7632c83 Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/tenantList 2023-02-11 13:54:27 -08:00
A.J. Beamon b4f45a0a87 Fix logic in MetaclusterManagementWorkload when retrying a tenant creation with an invalid assigned cluster 2023-02-11 12:09:17 -08:00
A.J. Beamon ee1b48323d
Merge pull request #9346 from sfc-gh-nwijetunga/nim/global-tenant-ids
Support for Two Byte Prefix for Tenant IDs
2023-02-11 11:31:24 -08:00
A.J. Beamon a6b47c1da4 Fix merge issue 2023-02-10 11:12:36 -08:00
Jon Fu 1c0e784a7a attempt to fix broken test workloads 2023-02-10 11:01:59 -08:00
A.J. Beamon 13eee09ce8 Merge branch 'main' into metacluster-mgmt-restore 2023-02-10 10:58:01 -08:00
A.J. Beamon 4b13c9c211 Make a few minor fixes, refactor some code for clarity, and improve throughput of repopulating a management cluster 2023-02-10 10:41:55 -08:00