mirror of https://github.com/apache/cassandra
Added production recommendations and improved compaction doc organization.
Each compaction strategy gets its own page. No longer recommending m1 instances in AWS. Patch by Jon Haddad; Reviewed by Jordan West for CASSANDRA-15618.
This commit is contained in:
Jon Haddad jon@jonhaddad.com
parent
03424b4fd7
commit
4dcee71b2d
|
|
@ -1,4 +1,5 @@
|
|||
4.0-alpha4
|
||||
* Added production recommendations and improved compaction doc organization
|
||||
* Document usage of EC2Snitch with intra-region VPC peering (CASSANDRA-15337)
|
||||
* Fixed flakey test in SASIIndexTest by shutting down its ExecutorService (CASSANDRA-15528)
|
||||
* Fixed empty check in TrieMemIndex due to potential state inconsistency in ConcurrentSkipListMap (CASSANDRA-15526)
|
||||
|
|
|
|||
|
|
@ -29,5 +29,6 @@ Cassandra.
|
|||
configuring
|
||||
querying
|
||||
drivers
|
||||
production
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,156 @@
|
|||
.. Licensed to the Apache Software Foundation (ASF) under one
|
||||
.. or more contributor license agreements. See the NOTICE file
|
||||
.. distributed with this work for additional information
|
||||
.. regarding copyright ownership. The ASF licenses this file
|
||||
.. to you under the Apache License, Version 2.0 (the
|
||||
.. "License"); you may not use this file except in compliance
|
||||
.. with the License. You may obtain a copy of the License at
|
||||
..
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0
|
||||
..
|
||||
.. Unless required by applicable law or agreed to in writing, software
|
||||
.. distributed under the License is distributed on an "AS IS" BASIS,
|
||||
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
.. See the License for the specific language governing permissions and
|
||||
.. limitations under the License.
|
||||
|
||||
Production Recommendations
|
||||
----------------------------
|
||||
|
||||
The ``cassandra.yaml`` and ``jvm.options`` files have a number of notes and recommendations for production usage. This page
|
||||
expands on some of the notes in these files with additional information.
|
||||
|
||||
Tokens
|
||||
^^^^^^^
|
||||
|
||||
Using more than 1 token (referred to as vnodes) allows for more flexible expansion and more streaming peers when
|
||||
bootstrapping new nodes into the cluster. This can limit the negative impact of streaming (I/O and CPU overhead)
|
||||
as well as allow for incremental cluster expansion.
|
||||
|
||||
As a tradeoff, more tokens will lead to sharing data with more peers, which can result in decreased availability. To learn more about this we
|
||||
recommend reading `this paper <https://github.com/jolynch/python_performance_toolkit/raw/master/notebooks/cassandra_availability/whitepaper/cassandra-availability-virtual.pdf>`_.
|
||||
|
||||
The number of tokens can be changed using the following setting:
|
||||
|
||||
``num_tokens: 16``
|
||||
|
||||
|
||||
Here are the most common token counts with a brief explanation of when and why you would use each one.
|
||||
|
||||
+-------------+---------------------------------------------------------------------------------------------------+
|
||||
| Token Count | Description |
|
||||
+=============+===================================================================================================+
|
||||
| 1 | Maximum availablility, maximum cluster size, fewest peers, |
|
||||
| | but inflexible expansion. Must always |
|
||||
| | double size of cluster to expand and remain balanced. |
|
||||
+-------------+---------------------------------------------------------------------------------------------------+
|
||||
| 4 | A healthy mix of elasticity and availability. Recommended for clusters which will eventually |
|
||||
| | reach over 30 nodes. Requires adding approximately 20% more nodes to remain balanced. |
|
||||
| | Shrinking a cluster may result in cluster imbalance. |
|
||||
+-------------+---------------------------------------------------------------------------------------------------+
|
||||
| 16 | Best for heavily elastic clusters which expand and shrink regularly, but may have issues |
|
||||
| | availability with larger clusters. Not recommended for clusters over 50 nodes. |
|
||||
+-------------+---------------------------------------------------------------------------------------------------+
|
||||
|
||||
|
||||
In addition to setting the token count, it's extremely important that ``allocate_tokens_for_local_replication_factor`` be
|
||||
set as well, to ensure even token allocation.
|
||||
|
||||
.. _read-ahead:
|
||||
|
||||
Read Ahead
|
||||
^^^^^^^^^^^
|
||||
|
||||
Read ahead is an operating system feature that attempts to keep as much data loaded in the page cache as possible. The
|
||||
goal is to decrease latency by using additional throughput on reads where the latency penalty is high due to seek times
|
||||
on spinning disks. By leveraging read ahead, the OS can pull additional data into memory without the cost of additional
|
||||
seeks. This works well when available RAM is greater than the size of the hot dataset, but can be problematic when the
|
||||
hot dataset is much larger than available RAM. The benefit of read ahead decreases as the size of your hot dataset gets
|
||||
bigger in proportion to available memory.
|
||||
|
||||
With small partitions (usually tables with no partition key, but not limited to this case) and solid state drives, read
|
||||
ahead can increase disk usage without any of the latency benefits, and in some cases can result in up to
|
||||
a 5x latency and throughput performance penalty. Read heavy, key/value tables with small (under 1KB) rows are especially
|
||||
prone to this problem.
|
||||
|
||||
We recommend the following read ahead settings:
|
||||
|
||||
+----------------+-------------------------+
|
||||
| Hardware | Initial Recommendation |
|
||||
+================+=========================+
|
||||
|Spinning Disks | 64KB |
|
||||
+----------------+-------------------------+
|
||||
|SSD | 4KB |
|
||||
+----------------+-------------------------+
|
||||
|
||||
Read ahead can be adjusted on Linux systems by using the `blockdev` tool.
|
||||
|
||||
For example, we can set read ahead of ``/dev/sda1` to 4KB by doing the following::
|
||||
|
||||
blockdev --setra 8 /dev/sda1
|
||||
|
||||
**Note**: blockdev accepts the number of 512 byte sectors to read ahead. The argument of 8 above is equivilent to 4KB.
|
||||
|
||||
Since each system is different, use the above recommendations as a starting point and tuning based on your SLA and
|
||||
throughput requirements. To understand how read ahead impacts disk resource usage we recommend carefully reading through the
|
||||
:ref:`troubleshooting <use-os-tools>` portion of the documentation.
|
||||
|
||||
|
||||
Compression
|
||||
^^^^^^^^^^^^
|
||||
|
||||
Compressed data is stored by compressing fixed size byte buffers and writing the data to disk. The buffer size is
|
||||
determined by the ``chunk_length_in_kb`` element in the compression map of the schema settings.
|
||||
|
||||
The default setting is 16KB starting with Cassandra 4.0.
|
||||
|
||||
Since the entire compressed buffer must be read off disk, using too high of a compression chunk length can lead to
|
||||
significant overhead when reading small records. Combined with the default read ahead setting this can result in massive
|
||||
read amplification for certain workloads.
|
||||
|
||||
LZ4Compressor is the default and recommended compression algorithm.
|
||||
|
||||
There is additional information on this topic on `The Last Pickle Blog <https://thelastpickle.com/blog/2018/08/08/compression_performance.html>`_.
|
||||
|
||||
Compaction
|
||||
^^^^^^^^^^^^
|
||||
|
||||
There are different :ref:`compaction <compaction>` strategies available for different workloads.
|
||||
We recommend reading up on the different strategies to understand which is the best for your environment. Different tables
|
||||
may (and frequently do) use different compaction strategies on the same cluster.
|
||||
|
||||
Encryption
|
||||
^^^^^^^^^^^
|
||||
|
||||
It is significantly easier to set up peer to peer encryption and client server encryption when setting up your production
|
||||
cluster as opposed to setting it up once the cluster is already serving production traffic. If you are planning on using network encryption
|
||||
eventually (in any form), we recommend setting it up now. Changing these configurations down the line is not impossible,
|
||||
but mistakes can result in downtime or data loss.
|
||||
|
||||
Ensure Keyspaces are Created with NetworkTopologyStrategy
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Production clusters should never use SimpleStrategy. Production keyspaces should use the NetworkTopologyStrategy (NTS).
|
||||
|
||||
For example::
|
||||
|
||||
create KEYSPACE mykeyspace WITH replication =
|
||||
{'class': 'NetworkTopologyStrategy', 'datacenter1': 3};
|
||||
|
||||
NetworkTopologyStrategy allows Cassandra to take advantage of multiple racks and data centers.
|
||||
|
||||
Configure Racks and Snitch
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
**Correctly configuring or changing racks after a cluster has been provisioned is an unsupported process**. Migrating from
|
||||
a single rack to multiple racks is also unsupported and can result in data loss.
|
||||
|
||||
Using ``GossipingPropertyFileSnitch`` is the most flexible solution for on premise or mixed cloud environments. ``Ec2Snitch``
|
||||
is reliable for AWS EC2 only environments.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -21,6 +21,25 @@
|
|||
Compaction
|
||||
----------
|
||||
|
||||
Strategies
|
||||
^^^^^^^^^^
|
||||
|
||||
Picking the right compaction strategy for your workload will ensure the best performance for both querying and for compaction itself.
|
||||
|
||||
:ref:`Size Tiered Compaction Strategy <stcs>`
|
||||
The default compaction strategy. Useful as a fallback when other strategies don't fit the workload. Most useful for
|
||||
non pure time series workloads with spinning disks, or when the I/O from :ref:`LCS <lcs>` is too high.
|
||||
|
||||
|
||||
:ref:`Leveled Compaction Strategy <lcs>`
|
||||
Leveled Compaction Strategy (LCS) is optimized for read heavy workloads, or workloads with lots of updates and deletes. It is not a good choice for immutable time series data.
|
||||
|
||||
|
||||
:ref:`Time Window Compaction Strategy <twcs>`
|
||||
Time Window Compaction Strategy is designed for TTL'ed, mostly immutable time series data.
|
||||
|
||||
|
||||
|
||||
Types of compaction
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
|
@ -276,172 +295,7 @@ More detailed compaction logging
|
|||
Enable with the compaction option ``log_all`` and a more detailed compaction log file will be produced in your log
|
||||
directory.
|
||||
|
||||
.. _STCS:
|
||||
|
||||
Size Tiered Compaction Strategy
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The basic idea of ``SizeTieredCompactionStrategy`` (STCS) is to merge sstables of approximately the same size. All
|
||||
sstables are put in different buckets depending on their size. An sstable is added to the bucket if size of the sstable
|
||||
is within ``bucket_low`` and ``bucket_high`` of the current average size of the sstables already in the bucket. This
|
||||
will create several buckets and the most interesting of those buckets will be compacted. The most interesting one is
|
||||
decided by figuring out which bucket's sstables takes the most reads.
|
||||
|
||||
Major compaction
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
When running a major compaction with STCS you will end up with two sstables per data directory (one for repaired data
|
||||
and one for unrepaired data). There is also an option (-s) to do a major compaction that splits the output into several
|
||||
sstables. The sizes of the sstables are approximately 50%, 25%, 12.5%... of the total size.
|
||||
|
||||
.. _stcs-options:
|
||||
|
||||
STCS options
|
||||
~~~~~~~~~~~~
|
||||
|
||||
``min_sstable_size`` (default: 50MB)
|
||||
Sstables smaller than this are put in the same bucket.
|
||||
``bucket_low`` (default: 0.5)
|
||||
How much smaller than the average size of a bucket a sstable should be before not being included in the bucket. That
|
||||
is, if ``bucket_low * avg_bucket_size < sstable_size`` (and the ``bucket_high`` condition holds, see below), then
|
||||
the sstable is added to the bucket.
|
||||
``bucket_high`` (default: 1.5)
|
||||
How much bigger than the average size of a bucket a sstable should be before not being included in the bucket. That
|
||||
is, if ``sstable_size < bucket_high * avg_bucket_size`` (and the ``bucket_low`` condition holds, see above), then
|
||||
the sstable is added to the bucket.
|
||||
|
||||
Defragmentation
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
Defragmentation is done when many sstables are touched during a read. The result of the read is put in to the memtable
|
||||
so that the next read will not have to touch as many sstables. This can cause writes on a read-only-cluster.
|
||||
|
||||
.. _LCS:
|
||||
|
||||
Leveled Compaction Strategy
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The idea of ``LeveledCompactionStrategy`` (LCS) is that all sstables are put into different levels where we guarantee
|
||||
that no overlapping sstables are in the same level. By overlapping we mean that the first/last token of a single sstable
|
||||
are never overlapping with other sstables. This means that for a SELECT we will only have to look for the partition key
|
||||
in a single sstable per level. Each level is 10x the size of the previous one and each sstable is 160MB by default. L0
|
||||
is where sstables are streamed/flushed - no overlap guarantees are given here.
|
||||
|
||||
When picking compaction candidates we have to make sure that the compaction does not create overlap in the target level.
|
||||
This is done by always including all overlapping sstables in the next level. For example if we select an sstable in L3,
|
||||
we need to guarantee that we pick all overlapping sstables in L4 and make sure that no currently ongoing compactions
|
||||
will create overlap if we start that compaction. We can start many parallel compactions in a level if we guarantee that
|
||||
we wont create overlap. For L0 -> L1 compactions we almost always need to include all L1 sstables since most L0 sstables
|
||||
cover the full range. We also can't compact all L0 sstables with all L1 sstables in a single compaction since that can
|
||||
use too much memory.
|
||||
|
||||
When deciding which level to compact LCS checks the higher levels first (with LCS, a "higher" level is one with a higher
|
||||
number, L0 being the lowest one) and if the level is behind a compaction will be started in that level.
|
||||
|
||||
Major compaction
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
It is possible to do a major compaction with LCS - it will currently start by filling out L1 and then once L1 is full,
|
||||
it continues with L2 etc. This is sub optimal and will change to create all the sstables in a high level instead,
|
||||
CASSANDRA-11817.
|
||||
|
||||
Bootstrapping
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
During bootstrap sstables are streamed from other nodes. The level of the remote sstable is kept to avoid many
|
||||
compactions after the bootstrap is done. During bootstrap the new node also takes writes while it is streaming the data
|
||||
from a remote node - these writes are flushed to L0 like all other writes and to avoid those sstables blocking the
|
||||
remote sstables from going to the correct level, we only do STCS in L0 until the bootstrap is done.
|
||||
|
||||
STCS in L0
|
||||
~~~~~~~~~~
|
||||
|
||||
If LCS gets very many L0 sstables reads are going to hit all (or most) of the L0 sstables since they are likely to be
|
||||
overlapping. To more quickly remedy this LCS does STCS compactions in L0 if there are more than 32 sstables there. This
|
||||
should improve read performance more quickly compared to letting LCS do its L0 -> L1 compactions. If you keep getting
|
||||
too many sstables in L0 it is likely that LCS is not the best fit for your workload and STCS could work out better.
|
||||
|
||||
Starved sstables
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
If a node ends up with a leveling where there are a few very high level sstables that are not getting compacted they
|
||||
might make it impossible for lower levels to drop tombstones etc. For example, if there are sstables in L6 but there is
|
||||
only enough data to actually get a L4 on the node the left over sstables in L6 will get starved and not compacted. This
|
||||
can happen if a user changes sstable\_size\_in\_mb from 5MB to 160MB for example. To avoid this LCS tries to include
|
||||
those starved high level sstables in other compactions if there has been 25 compaction rounds where the highest level
|
||||
has not been involved.
|
||||
|
||||
.. _lcs-options:
|
||||
|
||||
LCS options
|
||||
~~~~~~~~~~~
|
||||
|
||||
``sstable_size_in_mb`` (default: 160MB)
|
||||
The target compressed (if using compression) sstable size - the sstables can end up being larger if there are very
|
||||
large partitions on the node.
|
||||
|
||||
``fanout_size`` (default: 10)
|
||||
The target size of levels increases by this fanout_size multiplier. You can reduce the space amplification by tuning
|
||||
this option.
|
||||
|
||||
LCS also support the ``cassandra.disable_stcs_in_l0`` startup option (``-Dcassandra.disable_stcs_in_l0=true``) to avoid
|
||||
doing STCS in L0.
|
||||
|
||||
.. _TWCS:
|
||||
|
||||
Time Window CompactionStrategy
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
``TimeWindowCompactionStrategy`` (TWCS) is designed specifically for workloads where it's beneficial to have data on
|
||||
disk grouped by the timestamp of the data, a common goal when the workload is time-series in nature or when all data is
|
||||
written with a TTL. In an expiring/TTL workload, the contents of an entire SSTable likely expire at approximately the
|
||||
same time, allowing them to be dropped completely, and space reclaimed much more reliably than when using
|
||||
``SizeTieredCompactionStrategy`` or ``LeveledCompactionStrategy``. The basic concept is that
|
||||
``TimeWindowCompactionStrategy`` will create 1 sstable per file for a given window, where a window is simply calculated
|
||||
as the combination of two primary options:
|
||||
|
||||
``compaction_window_unit`` (default: DAYS)
|
||||
A Java TimeUnit (MINUTES, HOURS, or DAYS).
|
||||
``compaction_window_size`` (default: 1)
|
||||
The number of units that make up a window.
|
||||
``unsafe_aggressive_sstable_expiration`` (default: false)
|
||||
Expired sstables will be dropped without checking its data is shadowing other sstables. This is a potentially
|
||||
risky option that can lead to data loss or deleted data re-appearing, going beyond what
|
||||
`unchecked_tombstone_compaction` does for single sstable compaction. Due to the risk the jvm must also be
|
||||
started with `-Dcassandra.unsafe_aggressive_sstable_expiration=true`.
|
||||
|
||||
Taken together, the operator can specify windows of virtually any size, and `TimeWindowCompactionStrategy` will work to
|
||||
create a single sstable for writes within that window. For efficiency during writing, the newest window will be
|
||||
compacted using `SizeTieredCompactionStrategy`.
|
||||
|
||||
Ideally, operators should select a ``compaction_window_unit`` and ``compaction_window_size`` pair that produces
|
||||
approximately 20-30 windows - if writing with a 90 day TTL, for example, a 3 Day window would be a reasonable choice
|
||||
(``'compaction_window_unit':'DAYS','compaction_window_size':3``).
|
||||
|
||||
TimeWindowCompactionStrategy Operational Concerns
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The primary motivation for TWCS is to separate data on disk by timestamp and to allow fully expired SSTables to drop
|
||||
more efficiently. One potential way this optimal behavior can be subverted is if data is written to SSTables out of
|
||||
order, with new data and old data in the same SSTable. Out of order data can appear in two ways:
|
||||
|
||||
- If the user mixes old data and new data in the traditional write path, the data will be comingled in the memtables
|
||||
and flushed into the same SSTable, where it will remain comingled.
|
||||
- If the user's read requests for old data cause read repairs that pull old data into the current memtable, that data
|
||||
will be comingled and flushed into the same SSTable.
|
||||
|
||||
While TWCS tries to minimize the impact of comingled data, users should attempt to avoid this behavior. Specifically,
|
||||
users should avoid queries that explicitly set the timestamp via CQL ``USING TIMESTAMP``. Additionally, users should run
|
||||
frequent repairs (which streams data in such a way that it does not become comingled).
|
||||
|
||||
Changing TimeWindowCompactionStrategy Options
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Operators wishing to enable ``TimeWindowCompactionStrategy`` on existing data should consider running a major compaction
|
||||
first, placing all existing data into a single (old) window. Subsequent newer writes will then create typical SSTables
|
||||
as expected.
|
||||
|
||||
Operators wishing to change ``compaction_window_unit`` or ``compaction_window_size`` can do so, but may trigger
|
||||
additional compactions as adjacent windows are joined together. If the window size is decrease d (for example, from 24
|
||||
hours to 12 hours), then the existing SSTables will not be modified - TWCS can not split existing SSTables into multiple
|
||||
windows.
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
.. Licensed to the Apache Software Foundation (ASF) under one
|
||||
.. or more contributor license agreements. See the NOTICE file
|
||||
.. distributed with this work for additional information
|
||||
.. regarding copyright ownership. The ASF licenses this file
|
||||
.. to you under the Apache License, Version 2.0 (the
|
||||
.. "License"); you may not use this file except in compliance
|
||||
.. with the License. You may obtain a copy of the License at
|
||||
..
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0
|
||||
..
|
||||
.. Unless required by applicable law or agreed to in writing, software
|
||||
.. distributed under the License is distributed on an "AS IS" BASIS,
|
||||
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
.. See the License for the specific language governing permissions and
|
||||
.. limitations under the License.
|
||||
|
||||
|
||||
|
||||
.. _LCS:
|
||||
|
||||
Leveled Compaction Strategy
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The idea of ``LeveledCompactionStrategy`` (LCS) is that all sstables are put into different levels where we guarantee
|
||||
that no overlapping sstables are in the same level. By overlapping we mean that the first/last token of a single sstable
|
||||
are never overlapping with other sstables. This means that for a SELECT we will only have to look for the partition key
|
||||
in a single sstable per level. Each level is 10x the size of the previous one and each sstable is 160MB by default. L0
|
||||
is where sstables are streamed/flushed - no overlap guarantees are given here.
|
||||
|
||||
When picking compaction candidates we have to make sure that the compaction does not create overlap in the target level.
|
||||
This is done by always including all overlapping sstables in the next level. For example if we select an sstable in L3,
|
||||
we need to guarantee that we pick all overlapping sstables in L4 and make sure that no currently ongoing compactions
|
||||
will create overlap if we start that compaction. We can start many parallel compactions in a level if we guarantee that
|
||||
we wont create overlap. For L0 -> L1 compactions we almost always need to include all L1 sstables since most L0 sstables
|
||||
cover the full range. We also can't compact all L0 sstables with all L1 sstables in a single compaction since that can
|
||||
use too much memory.
|
||||
|
||||
When deciding which level to compact LCS checks the higher levels first (with LCS, a "higher" level is one with a higher
|
||||
number, L0 being the lowest one) and if the level is behind a compaction will be started in that level.
|
||||
|
||||
Major compaction
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
It is possible to do a major compaction with LCS - it will currently start by filling out L1 and then once L1 is full,
|
||||
it continues with L2 etc. This is sub optimal and will change to create all the sstables in a high level instead,
|
||||
CASSANDRA-11817.
|
||||
|
||||
Bootstrapping
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
During bootstrap sstables are streamed from other nodes. The level of the remote sstable is kept to avoid many
|
||||
compactions after the bootstrap is done. During bootstrap the new node also takes writes while it is streaming the data
|
||||
from a remote node - these writes are flushed to L0 like all other writes and to avoid those sstables blocking the
|
||||
remote sstables from going to the correct level, we only do STCS in L0 until the bootstrap is done.
|
||||
|
||||
STCS in L0
|
||||
~~~~~~~~~~
|
||||
|
||||
If LCS gets very many L0 sstables reads are going to hit all (or most) of the L0 sstables since they are likely to be
|
||||
overlapping. To more quickly remedy this LCS does STCS compactions in L0 if there are more than 32 sstables there. This
|
||||
should improve read performance more quickly compared to letting LCS do its L0 -> L1 compactions. If you keep getting
|
||||
too many sstables in L0 it is likely that LCS is not the best fit for your workload and STCS could work out better.
|
||||
|
||||
Starved sstables
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
If a node ends up with a leveling where there are a few very high level sstables that are not getting compacted they
|
||||
might make it impossible for lower levels to drop tombstones etc. For example, if there are sstables in L6 but there is
|
||||
only enough data to actually get a L4 on the node the left over sstables in L6 will get starved and not compacted. This
|
||||
can happen if a user changes sstable\_size\_in\_mb from 5MB to 160MB for example. To avoid this LCS tries to include
|
||||
those starved high level sstables in other compactions if there has been 25 compaction rounds where the highest level
|
||||
has not been involved.
|
||||
|
||||
.. _lcs-options:
|
||||
|
||||
LCS options
|
||||
~~~~~~~~~~~
|
||||
|
||||
``sstable_size_in_mb`` (default: 160MB)
|
||||
The target compressed (if using compression) sstable size - the sstables can end up being larger if there are very
|
||||
large partitions on the node.
|
||||
|
||||
``fanout_size`` (default: 10)
|
||||
The target size of levels increases by this fanout_size multiplier. You can reduce the space amplification by tuning
|
||||
this option.
|
||||
|
||||
LCS also support the ``cassandra.disable_stcs_in_l0`` startup option (``-Dcassandra.disable_stcs_in_l0=true``) to avoid
|
||||
doing STCS in L0.
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
.. Licensed to the Apache Software Foundation (ASF) under one
|
||||
.. or more contributor license agreements. See the NOTICE file
|
||||
.. distributed with this work for additional information
|
||||
.. regarding copyright ownership. The ASF licenses this file
|
||||
.. to you under the Apache License, Version 2.0 (the
|
||||
.. "License"); you may not use this file except in compliance
|
||||
.. with the License. You may obtain a copy of the License at
|
||||
..
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0
|
||||
..
|
||||
.. Unless required by applicable law or agreed to in writing, software
|
||||
.. distributed under the License is distributed on an "AS IS" BASIS,
|
||||
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
.. See the License for the specific language governing permissions and
|
||||
.. limitations under the License.
|
||||
|
||||
|
||||
.. _STCS:
|
||||
|
||||
Leveled Compaction Strategy
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The basic idea of ``SizeTieredCompactionStrategy`` (STCS) is to merge sstables of approximately the same size. All
|
||||
sstables are put in different buckets depending on their size. An sstable is added to the bucket if size of the sstable
|
||||
is within ``bucket_low`` and ``bucket_high`` of the current average size of the sstables already in the bucket. This
|
||||
will create several buckets and the most interesting of those buckets will be compacted. The most interesting one is
|
||||
decided by figuring out which bucket's sstables takes the most reads.
|
||||
|
||||
Major compaction
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
When running a major compaction with STCS you will end up with two sstables per data directory (one for repaired data
|
||||
and one for unrepaired data). There is also an option (-s) to do a major compaction that splits the output into several
|
||||
sstables. The sizes of the sstables are approximately 50%, 25%, 12.5%... of the total size.
|
||||
|
||||
.. _stcs-options:
|
||||
|
||||
STCS options
|
||||
~~~~~~~~~~~~
|
||||
|
||||
``min_sstable_size`` (default: 50MB)
|
||||
Sstables smaller than this are put in the same bucket.
|
||||
``bucket_low`` (default: 0.5)
|
||||
How much smaller than the average size of a bucket a sstable should be before not being included in the bucket. That
|
||||
is, if ``bucket_low * avg_bucket_size < sstable_size`` (and the ``bucket_high`` condition holds, see below), then
|
||||
the sstable is added to the bucket.
|
||||
``bucket_high`` (default: 1.5)
|
||||
How much bigger than the average size of a bucket a sstable should be before not being included in the bucket. That
|
||||
is, if ``sstable_size < bucket_high * avg_bucket_size`` (and the ``bucket_low`` condition holds, see above), then
|
||||
the sstable is added to the bucket.
|
||||
|
||||
Defragmentation
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
Defragmentation is done when many sstables are touched during a read. The result of the read is put in to the memtable
|
||||
so that the next read will not have to touch as many sstables. This can cause writes on a read-only-cluster.
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,76 @@
|
|||
.. Licensed to the Apache Software Foundation (ASF) under one
|
||||
.. or more contributor license agreements. See the NOTICE file
|
||||
.. distributed with this work for additional information
|
||||
.. regarding copyright ownership. The ASF licenses this file
|
||||
.. to you under the Apache License, Version 2.0 (the
|
||||
.. "License"); you may not use this file except in compliance
|
||||
.. with the License. You may obtain a copy of the License at
|
||||
..
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0
|
||||
..
|
||||
.. Unless required by applicable law or agreed to in writing, software
|
||||
.. distributed under the License is distributed on an "AS IS" BASIS,
|
||||
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
.. See the License for the specific language governing permissions and
|
||||
.. limitations under the License.
|
||||
|
||||
|
||||
.. _TWCS:
|
||||
|
||||
Time Window CompactionStrategy
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
``TimeWindowCompactionStrategy`` (TWCS) is designed specifically for workloads where it's beneficial to have data on
|
||||
disk grouped by the timestamp of the data, a common goal when the workload is time-series in nature or when all data is
|
||||
written with a TTL. In an expiring/TTL workload, the contents of an entire SSTable likely expire at approximately the
|
||||
same time, allowing them to be dropped completely, and space reclaimed much more reliably than when using
|
||||
``SizeTieredCompactionStrategy`` or ``LeveledCompactionStrategy``. The basic concept is that
|
||||
``TimeWindowCompactionStrategy`` will create 1 sstable per file for a given window, where a window is simply calculated
|
||||
as the combination of two primary options:
|
||||
|
||||
``compaction_window_unit`` (default: DAYS)
|
||||
A Java TimeUnit (MINUTES, HOURS, or DAYS).
|
||||
``compaction_window_size`` (default: 1)
|
||||
The number of units that make up a window.
|
||||
``unsafe_aggressive_sstable_expiration`` (default: false)
|
||||
Expired sstables will be dropped without checking its data is shadowing other sstables. This is a potentially
|
||||
risky option that can lead to data loss or deleted data re-appearing, going beyond what
|
||||
`unchecked_tombstone_compaction` does for single sstable compaction. Due to the risk the jvm must also be
|
||||
started with `-Dcassandra.unsafe_aggressive_sstable_expiration=true`.
|
||||
|
||||
Taken together, the operator can specify windows of virtually any size, and `TimeWindowCompactionStrategy` will work to
|
||||
create a single sstable for writes within that window. For efficiency during writing, the newest window will be
|
||||
compacted using `SizeTieredCompactionStrategy`.
|
||||
|
||||
Ideally, operators should select a ``compaction_window_unit`` and ``compaction_window_size`` pair that produces
|
||||
approximately 20-30 windows - if writing with a 90 day TTL, for example, a 3 Day window would be a reasonable choice
|
||||
(``'compaction_window_unit':'DAYS','compaction_window_size':3``).
|
||||
|
||||
TimeWindowCompactionStrategy Operational Concerns
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The primary motivation for TWCS is to separate data on disk by timestamp and to allow fully expired SSTables to drop
|
||||
more efficiently. One potential way this optimal behavior can be subverted is if data is written to SSTables out of
|
||||
order, with new data and old data in the same SSTable. Out of order data can appear in two ways:
|
||||
|
||||
- If the user mixes old data and new data in the traditional write path, the data will be comingled in the memtables
|
||||
and flushed into the same SSTable, where it will remain comingled.
|
||||
- If the user's read requests for old data cause read repairs that pull old data into the current memtable, that data
|
||||
will be comingled and flushed into the same SSTable.
|
||||
|
||||
While TWCS tries to minimize the impact of comingled data, users should attempt to avoid this behavior. Specifically,
|
||||
users should avoid queries that explicitly set the timestamp via CQL ``USING TIMESTAMP``. Additionally, users should run
|
||||
frequent repairs (which streams data in such a way that it does not become comingled).
|
||||
|
||||
Changing TimeWindowCompactionStrategy Options
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Operators wishing to enable ``TimeWindowCompactionStrategy`` on existing data should consider running a major compaction
|
||||
first, placing all existing data into a single (old) window. Subsequent newer writes will then create typical SSTables
|
||||
as expected.
|
||||
|
||||
Operators wishing to change ``compaction_window_unit`` or ``compaction_window_size`` can do so, but may trigger
|
||||
additional compactions as adjacent windows are joined together. If the window size is decrease d (for example, from 24
|
||||
hours to 12 hours), then the existing SSTables will not be modified - TWCS can not split existing SSTables into multiple
|
||||
windows.
|
||||
|
||||
|
|
@ -77,8 +77,6 @@ Many large users of Cassandra run in various clouds, including AWS, Azure, and G
|
|||
of these environments. Users should choose similar hardware to what would be needed in physical space. In EC2, popular
|
||||
options include:
|
||||
|
||||
- m1.xlarge instances, which provide 1.6TB of local ephemeral spinning storage and sufficient RAM to run moderate
|
||||
workloads
|
||||
- i2 instances, which provide both a high RAM:CPU ratio and local ephemeral SSDs
|
||||
- m4.2xlarge / c4.4xlarge instances, which provide modern CPUs, enhanced networking and work well with EBS GP2 (SSD)
|
||||
storage
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ Operating Cassandra
|
|||
repair
|
||||
read_repair
|
||||
hints
|
||||
compaction
|
||||
compaction/index
|
||||
bloom_filters
|
||||
compression
|
||||
cdc
|
||||
|
|
|
|||
Loading…
Reference in New Issue