diff --git a/doc/modules/cassandra/pages/operating/denylisting_partitions.adoc b/doc/modules/cassandra/pages/operating/denylisting_partitions.adoc new file mode 100644 index 0000000000..a81dc8009c --- /dev/null +++ b/doc/modules/cassandra/pages/operating/denylisting_partitions.adoc @@ -0,0 +1,143 @@ += Denylisting Partitions + +Due to access patterns and data modeling, sometimes there are specific partitions +that are "hot" and can cause instability in a Cassandra cluster. This often occurs +when your data model includes many update or insert operations on a single partition, +causing the partition to grow very large over time and in turn making it very expensive +to read and maintain. + +Cassandra supports "denylisting" these problematic partitions so that when clients +issue point reads (`SELECT` statements with the partition key specified) or range +reads (`SELECT *`, etc that pull a range of data) that intersect with a blocked +partition key, the query will be immediately rejected with an `InvalidQueryException`. + +== How to denylist a partition key + +The ``system_distributed.denylisted_partitions`` table can be used to denylist partitions. +There are a couple of ways to interact with and mutate this data. First: directly +via CQL by inserting a record with the following details: + +- Keyspace name (ks_name) +- Table name (table_name) +- Partition Key (partition_key) + +The partition key format needs to be in the same form required by ``nodetool getendpoints``. + +Following are several examples for denylisting partition keys in keyspace `ks` and +table `table1` for different data types on the primary key `Id`: + +- Id is a simple type - `INSERT INTO system_distributed.denylisted_partitions (ks_name, table_name, partition_key) VALUES ('ks','table1','1');` +- Id is a blob - `INSERT INTO system_distributed.denylisted_partitions (ks_name, table_name, partition_key) VALUES ('ks','table1','12345f');` +- Id has a colon - `INSERT INTO system_distributed.denylisted_partitions (ks_name, table_name, partition_key) VALUES ('ks','table1','1\:2');` + +In the case of composite column partition keys (Key1, Key2): + +- `INSERT INTO system_distributed.denylisted_partitions (ks_name, table_name, partition_key) VALUES ('ks', 'table1', 'k11:k21')` + + +=== Special considerations + +The denylist has the property in that you want to keep your cache (see below) and +CQL data on a replica set as close together as possible, so you don't have different +nodes in your cluster denying or allowing different keys. To best achieve this, +the workflow for a denylist change (addition or deletion) should always be as follows: + +JMX PATH (preferred for single changes): + +1. Call the JMX hook for ``denylistKey()`` with the desired key +2. Double-check the cache reloaded with ``isKeyDenylisted()`` +3. Check for warnings about unrecognized keyspace/table combinations, limits, or +consistency level. If you get a message about nodes being down and not hitting CL +for denylist, recover the downed nodes and then trigger a re-load of the cache on each +node with ``loadPartitionDenylist()`` + +CQL PATH (preferred for bulk changes): + +1. Mutate the denylisted partition lists via CQL +2. Trigger a re-load of the denylist cache on each node via JMX ``loadPartitionDenylist()`` (see below) +3. Check for warnings about lack of availability for a denylist refresh. In the event nodes are down, recover them, then go to 2. + +Due to conditions on known unavailable range slices leading to alert storming on +startup, the denylist cache will not load on node start unless it can achieve the +configured consistency level in `cassandra.yaml` - `denylist_consistency_level`. +The JMX call to `loadPartitionDenylist` will, however, load the cache regardless +of the number of nodes available. This leaves the control for denylisting or not +denylisting during degraded cluster states in the hands of the operator. + +== Denylisted Partitions Cache + +Cassandra internally maintains an on-heap cache of denylisted partitions loaded +from ``system_distributed.denylisted_partitions``. The values for a table will be +automatically repopulated every ``denylist_refresh`` as specified in the +`conf/cassandra.yaml` file, defaulting to `600s`, or 10 minutes. Invalid records +(unknown keyspaces, tables, or keys) will be ignored and not cached on load. + +The cache can be refreshed in the following ways: + +- During Cassandra node startup +- Via the automatic on-heap cache refresh mechanisms. Note: this will occur asynchronously +on query after the ``denylist_refresh`` time is hit. +- Via the JMX command: ``loadPartitionDenylist`` in ``the org.apache.cassandra.service. +StorageProxyMBean`` invocation point. + +The Cache size is bounded by the following two config properties + +- denylist_max_keys_per_table +- denylist_max_keys_total + +On cache load, if a table exceeds the value allowed in `denylist_max_keys_per_table` (defaults to 1000), +a warning will be printed to the logs and the remainder of the keys will not be cached. +Similarly, if the total allowed size is exceeded, subsequent ks_name + table_name +combinations (in clustering / lexicographical order) will be skipped as well, and a +warning logged to the server logs. + +[NOTE] +==== +Given the required workflow of 1) Mutate, 2) Reload cache, the auto-reload +property seems superfluous. It exists to ensure that, should an operator make a +mistake and denylist (or undenylist) a key but forget to reload the cache, that +intent will be captured on the next cache reload. +==== + +== JMX Interface + +[cols="1,1"] +|=== +| Command | Effect + +| loadPartitionDenylist() +| Reloads cached denylist from CQL table + +| getPartitionDenylistLoadAttempts() +| Gets the count of cache reload attempts + +| getPartitionDenylistLoadSuccesses() +| Gets the count of cache reload successes + +| setEnablePartitionDenylist(boolean enabled) +| Enables or disables the partition denylisting functionality + +| setEnableDenylistWrites(boolean enabled) +| Enables or disables write denylisting functionality + +| setEnableDenylistReads(boolean enabled) +| Enables or disables read denylisting functionality + +| setEnableDenylistRangeReads(boolean enabled) +| Enables or disables range read denylisting functionality + +| denylistKey(String keyspace, String table, String partitionKeyAsString) +| Adds a specific keyspace, table, and partition key combo to the denylist + +| removeDenylistKey(String keyspace, String cf, String partitionKeyAsString) +| Removes a specific keyspace, table, and partition key combo from the denylist + +| setDenylistMaxKeysPerTable(int value) +| Limits count of allowed keys per table in the denylist + +| setDenylistMaxKeysTotal(int value) +| Limits the total count of allowable denylisted keys in the system + +| isKeyDenylisted(String keyspace, String table, String partitionKeyAsString) +| Indicates whether the keyspace.table has the input partition key denied +|=== diff --git a/doc/modules/cassandra/pages/operating/index.adoc b/doc/modules/cassandra/pages/operating/index.adoc index 367430a2ef..41ee1b60ca 100644 --- a/doc/modules/cassandra/pages/operating/index.adoc +++ b/doc/modules/cassandra/pages/operating/index.adoc @@ -13,3 +13,4 @@ * xref:operating/bulk_loading.adoc[Bulk loading] * xref:operating/cdc.adoc[CDC] * xref:operating/bloom_filters.adoc[Bloom filters] +* xref:operating/denylisting_partitions.adoc[Denylisting partitions]