mirror of https://github.com/apache/cassandra
CASSANDRA-21291: Fix duplicate section and minor formatting typos in compaction overview
This commit is contained in:
arvindksi274-ksolves
Patrick McFadin
parent
aec3d90ce4
commit
0742f15f10
|
|
@ -2,18 +2,18 @@
|
|||
|
||||
== What is compaction?
|
||||
|
||||
Data in {cassandra} is created in xref:cassandra:architecture/storage-engine.adoc#memtables[memtables].
|
||||
Data in {cassandra} is created in xref:cassandra:architecture/storage-engine.adoc#memtables[memtables].
|
||||
Once a memory threshold is reached, to free up memory again, the data is written to an xref:cassandra:architecture/storage-engine.adoc#SSTables[SSTable], an https://cassandra.apache.org/_/glossary.html#immutable[immutable] file residing on disk.
|
||||
|
||||
Because SSTables are immutable, when data is updated or deleted, the old data is not overwritten with inserts or updates, or removed from the SSTable.
|
||||
Instead, a new SSTable is created with the updated data with a new timestamp, and the old SSTable is marked for deletion.
|
||||
Because SSTables are immutable, when data is updated or deleted, the old data is not overwritten with inserts or updates, or removed from the SSTable.
|
||||
Instead, a new SSTable is created with the updated data with a new timestamp, and the old SSTable is marked for deletion.
|
||||
The piece of deleted data is known as a https://cassandra.apache.org/_/glossary.html#tombstone[tombstone].
|
||||
|
||||
Over time, Cassandra may write many versions of a row in different SSTables.
|
||||
Each version may have a unique set of columns stored with a different timestamp.
|
||||
Over time, Cassandra may write many versions of a row in different SSTables.
|
||||
Each version may have a unique set of columns stored with a different timestamp.
|
||||
As SSTables accumulate, the distribution of data can require accessing more and more SSTables to retrieve a complete row.
|
||||
|
||||
To keep the database healthy, Cassandra periodically merges SSTables and discards old data.
|
||||
To keep the database healthy, Cassandra periodically merges SSTables and discards old data.
|
||||
This process is called https://cassandra.apache.org/_/glossary.html#compaction[compaction].
|
||||
|
||||
== Why must compaction be run?
|
||||
|
|
@ -26,22 +26,22 @@ Deleting, updating, or expiring data are all valid triggers for compaction.
|
|||
== What does compaction accomplish?
|
||||
|
||||
Two important factors accomplished by compaction are performance improvement and disk space reclamation.
|
||||
If SSTables have duplicate data that must be read, read operations are slower.
|
||||
If SSTables have duplicate data that must be read, read operations are slower.
|
||||
Once tombstones and duplicates are removed, read operations are faster.
|
||||
SSTables use disk space, and reducing the size of SSTables through compaction frees up disk space.
|
||||
|
||||
== How does compaction work?
|
||||
|
||||
Compaction works on a collection of SSTables.
|
||||
From these SSTables, compaction collects all versions of each unique row and assembles one complete row, using the most up-to-date version (by timestamp) of each of the row's columns.
|
||||
The merge process is performant, because rows are sorted by partition key within each SSTable, and the merge process does not use random I/O.
|
||||
The new versions of each row is written to a new SSTable.
|
||||
Compaction works on a collection of SSTables.
|
||||
From these SSTables, compaction collects all versions of each unique row and assembles one complete row, using the most up-to-date version (by timestamp) of each of the row's columns.
|
||||
The merge process is performant, because rows are sorted by partition key within each SSTable, and the merge process does not use random I/O.
|
||||
The new versions of each row is written to a new SSTable.
|
||||
The old versions, along with any rows that are ready for deletion, are left in the old SSTables, and are deleted as soon as pending reads are completed.
|
||||
|
||||
== Types of compaction
|
||||
|
||||
The concept of compaction is used for different kinds of operations in
|
||||
{cassandra}, the common thing about these operations is that it takes one
|
||||
{cassandra}, the common thing about these operations is that they take one
|
||||
or more SSTables, merges, and outputs new SSTables. The types of compactions are:
|
||||
|
||||
Minor compaction::
|
||||
|
|
@ -56,11 +56,11 @@ A major compaction is triggered when a user executes a compaction over all SSTab
|
|||
User defined compaction::
|
||||
Similar to a major compaction, a user-defined compaction executes when a user triggers a compaction on a given set of SSTables.
|
||||
Scrub::
|
||||
A scrub triggers a compaction to try to fix any broken SSTables.
|
||||
A scrub triggers a compaction to try to fix any broken SSTables.
|
||||
This can actually remove valid data if that data is corrupted.
|
||||
If that happens you will need to run a full repair on the node.
|
||||
UpgradeSSTables::
|
||||
A compaction occurs when you upgrade SSTables to the latest version.
|
||||
A compaction occurs when you upgrade SSTables to the latest version.
|
||||
Run this after upgrading to a new major version.
|
||||
Cleanup::
|
||||
Compaction executes to remove any ranges that a node no longer owns.
|
||||
|
|
@ -71,8 +71,8 @@ Anticompaction::
|
|||
After repair, the ranges that were actually repaired are split out of the SSTables that existed when repair started. This type of compaction rewrites SSTables to accomplish this task.
|
||||
Sub range compaction::
|
||||
It is possible to only compact a given sub range - this action is useful if you know a token that has been misbehaving - either gathering many updates or many deletes.
|
||||
The command `nodetool compact -st x -et y` will pick all SSTables containing the range between x and y and issue a compaction for those SSTables.
|
||||
For Size Tiered Compaction Strategy, this will most likely include all SSTables, but with Leveled Compaction Strategy, it can issue the compaction for a subset of the SSTables.
|
||||
The command `nodetool compact -st x -et y` will pick all SSTables containing the range between x and y and issue a compaction for those SSTables.
|
||||
For Size Tiered Compaction Strategy, this will most likely include all SSTables, but with Leveled Compaction Strategy, it can issue the compaction for a subset of the SSTables.
|
||||
With LCS the resulting SSTable will end up in L0.
|
||||
|
||||
== Strategies
|
||||
|
|
@ -82,14 +82,14 @@ Picking the right compaction strategy for your workload will ensure the best per
|
|||
|
||||
xref:cassandra:managing/operating/compaction/ucs.adoc[`Unified Compaction Strategy (UCS)`]::
|
||||
UCS is a good choice for most workloads and is recommended for new workloads.
|
||||
This compaction strategy is designed to handle a wide variety of workloads.
|
||||
It is designed to be able to handle both immutable time-series data and workloads with lots of updates and deletes.
|
||||
It is also designed to be able to handle both spinning disks and SSDs.
|
||||
xref:cassandra:managing/operating/compaction/stcs.adoc[`Size Tiered Compaction Strategy (STCS)`]::
|
||||
STCS is the default compaction strategy, because it is useful as a fallback when other strategies don't fit the workload.
|
||||
This compaction strategy is designed to handle a wide variety of workloads.
|
||||
It is designed to be able to handle both immutable time-series data and workloads with lots of updates and deletes.
|
||||
It is also designed to be able to handle both spinning disks and SSDs.
|
||||
xref:cassandra:managing/operating/compaction/stcs.adoc[`Size Tiered Compaction Strategy (STCS)`]::
|
||||
STCS is the default compaction strategy, because it is useful as a fallback when other strategies don't fit the workload.
|
||||
Most useful for not strictly time-series workloads with spinning disks, or when the I/O from `LCS` is too high.
|
||||
xref:cassandra:managing/operating/compaction/lcs.adoc[`Leveled Compaction Strategy (LCS)`]::
|
||||
Leveled Compaction Strategy (LCS) is optimized for read heavy workloads, or workloads with lots of updates and deletes.
|
||||
Leveled Compaction Strategy (LCS) is optimized for read heavy workloads, or workloads with lots of updates and deletes.
|
||||
It is not a good choice for immutable time-series data.
|
||||
xref:cassandra:managing/operating/compaction/twcs.adoc[`Time Window Compaction Strategy (TWCS)`]::
|
||||
Time Window Compaction Strategy is designed for TTL'ed, mostly immutable time-series data.
|
||||
|
|
@ -107,19 +107,6 @@ of the TTL) Cassandra will have a hard time dropping the tombstones
|
|||
created since the partition might span many SSTables and not all are
|
||||
compacted at once.
|
||||
|
||||
== Fully expired SSTables
|
||||
|
||||
If an SSTable contains only tombstones and it is guaranteed that
|
||||
SSTable is not shadowing data in any other SSTable, then the compaction can drop
|
||||
that SSTable. If you see SSTables with only tombstones (note that TTL-ed
|
||||
data is considered tombstones once the time-to-live has expired), but it
|
||||
is not being dropped by compaction, it is likely that other SSTables
|
||||
contain older data. There is a tool called `sstableexpiredblockers` that
|
||||
will list which SSTables are droppable and which are blocking them from
|
||||
being dropped. With `TimeWindowCompactionStrategy` it
|
||||
is possible to remove the guarantee (not check for shadowing data) by
|
||||
enabling `unsafe_aggressive_sstable_expiration`.
|
||||
|
||||
== Repaired/unrepaired data
|
||||
|
||||
With incremental repairs Cassandra must keep track of what data is
|
||||
|
|
@ -161,8 +148,8 @@ When an SSTable is written a histogram with the tombstone expiry times
|
|||
is created and this is used to try to find SSTables with very many
|
||||
tombstones and run single SSTable compaction on that SSTable in hope of
|
||||
being able to drop tombstones in that SSTable. Before starting this it
|
||||
is also checked how likely it is that any tombstones will actually will
|
||||
be able to be dropped how much this SSTable overlaps with other
|
||||
is also checked how likely it is that any tombstones will actually
|
||||
be able to be dropped and how much this SSTable overlaps with other
|
||||
SSTables. To avoid most of these checks the compaction option
|
||||
`unchecked_tombstone_compaction` can be enabled.
|
||||
|
||||
|
|
@ -178,11 +165,11 @@ How much of the SSTable should be tombstones for us to consider doing a single S
|
|||
`tombstone_compaction_interval` (default: 86400s (1 day))::
|
||||
Since it might not be possible to drop any tombstones when doing a single SSTable compaction we need to make sure that one SSTable is not constantly getting recompacted - this option states how often we should try for a given SSTable.
|
||||
`log_all` (default: false)::
|
||||
New detailed compaction logging, see `below <detailed-compaction-logging>`.
|
||||
New detailed compaction logging, see <<detailed-compaction-logging, below>>.
|
||||
`unchecked_tombstone_compaction` (default: false)::
|
||||
The single SSTable compaction has quite strict checks for whether it should be started, this option disables those checks and for some use cases this might be needed.
|
||||
The single SSTable compaction has quite strict checks for whether it should be started, this option disables those checks and for some use cases this might be needed.
|
||||
Note that this does not change anything for the actual compaction, tombstones are only dropped if it is safe to do so - it might just rewrite an SSTable without being able to drop any tombstones.
|
||||
`only_purge_repaired_tombstone` (default: false)::
|
||||
`only_purge_repaired_tombstones` (default: false)::
|
||||
Option to enable the extra safety of making sure that tombstones are only dropped if the data has been repaired.
|
||||
`min_threshold` (default: 4)::
|
||||
Lower limit of number of SSTables before a compaction is triggered.
|
||||
|
|
@ -195,7 +182,7 @@ Further, see the section on each strategy for specific additional options.
|
|||
|
||||
== Compaction nodetool commands
|
||||
|
||||
The `nodetool <nodetool>` utility provides a number of commands related to compaction:
|
||||
The `nodetool` utility provides a number of commands related to compaction:
|
||||
|
||||
`enableautocompaction`::
|
||||
Enable compaction.
|
||||
|
|
@ -212,7 +199,7 @@ Set the min/max SSTable count for when to trigger compaction, defaults to 4/32.
|
|||
|
||||
== Switching the compaction strategy and options using JMX
|
||||
|
||||
It is possible to switch compaction strategies and its options on just a single node using JMX, this is a great way to experiment with settings without affecting the whole cluster.
|
||||
It is possible to switch compaction strategies and its options on just a single node using JMX, this is a great way to experiment with settings without affecting the whole cluster.
|
||||
The mbean is:
|
||||
|
||||
[source,console]
|
||||
|
|
|
|||
|
|
@ -69,22 +69,21 @@ This is basically the same as in the "Deletes without Tombstones" section.
|
|||
|
||||
=== Deletes without tombstones
|
||||
|
||||
Imagine a three node cluster which has the value [A] replicated to every
|
||||
node.:
|
||||
Imagine a three node cluster which has the value [A] replicated to every node:
|
||||
|
||||
[source,none]
|
||||
----
|
||||
[A], [A], [A]
|
||||
----
|
||||
|
||||
If one of the nodes fails and and our delete operation only removes existing values, we can end up with a cluster that looks like:
|
||||
If one of the nodes fails and our delete operation only removes existing values, we can end up with a cluster that looks like:
|
||||
|
||||
[source,none]
|
||||
----
|
||||
[], [], [A]
|
||||
----
|
||||
|
||||
Then a repair operation would replace the value of [A] back onto the two nodes which are missing the value.:
|
||||
Then a repair operation would replace the value of [A] back onto the two nodes which are missing the value:
|
||||
|
||||
[source,none]
|
||||
----
|
||||
|
|
@ -95,7 +94,7 @@ This would cause our data to be resurrected as a zombie even though it had been
|
|||
|
||||
=== Deletes with tombstones
|
||||
|
||||
Starting again with a three node cluster which has the value [A] replicated to every node.:
|
||||
Starting again with a three node cluster which has the value [A] replicated to every node:
|
||||
|
||||
[source,none]
|
||||
----
|
||||
|
|
|
|||
Loading…
Reference in New Issue