mirror of https://github.com/apache/cassandra
Updated documentation related to commitlog, storage engine and other improvements
patch by Arra Praveen; reviewed by Stefan Miklosovic for CASSANDRA-20454
This commit is contained in:
Arra, Praveen R
Stefan Miklosovic
parent
4226a7770e
commit
f7d39dfff4
|
|
@ -110,7 +110,7 @@ internode_application_receive_queue_reserve_global_capacity: 512MiB
|
|||
== Virtual Tables for Messaging Metrics
|
||||
|
||||
Metrics is improved by keeping metrics using virtual tables for
|
||||
inter-node inbound and outbound messaging
|
||||
internode inbound and outbound messaging
|
||||
(https://issues.apache.org/jira/browse/CASSANDRA-15066[CASSANDRA-15066]).
|
||||
For inbound messaging a virtual table (`internode_inbound`) has been
|
||||
added to keep metrics for:
|
||||
|
|
@ -122,10 +122,10 @@ due to an error
|
|||
* Bytes and count of messages successfully received
|
||||
* Nanos and count of messages throttled
|
||||
* Bytes and count of messages expired
|
||||
* Corrupt frames recovered and unrecovered
|
||||
* Corrupt frames recovered and un-recovered
|
||||
|
||||
A separate virtual table (`internode_outbound`) has been added for
|
||||
outbound inter-node messaging. The outbound virtual table keeps metrics
|
||||
outbound internode messaging. The outbound virtual table keeps metrics
|
||||
for:
|
||||
|
||||
* Bytes and count of messages pending
|
||||
|
|
@ -216,7 +216,7 @@ The Unprotected option is still available.
|
|||
For resilience, all frames are written with a separate CRC protected
|
||||
header, of 8 and 6 bytes respectively. If corruption occurs in this
|
||||
header, the connection must be reset, as before. If corruption occurs
|
||||
anywhere outside of the header, the corrupt frame will be skipped,
|
||||
outside the header, the corrupt frame will be skipped,
|
||||
leaving the connection intact and avoiding the loss of any messages
|
||||
unnecessarily.
|
||||
|
||||
|
|
@ -255,7 +255,7 @@ expired at the time when a message is encountered the message is just
|
|||
skipped in the byte stream altogether. And if a message fails to be
|
||||
deserialized while still on the receiving side - say, because of table
|
||||
id or column being unknown - bytes are skipped, without dropping the
|
||||
entire connection and losing all the buffered messages. An immediately
|
||||
entire connection and losing all the buffered messages. An immediate
|
||||
reply back is sent to the coordinator node with the failure reason,
|
||||
rather than waiting for the coordinator callback to expire. This logic
|
||||
is extended to a corrupted frame; a corrupted frame is safely skipped
|
||||
|
|
@ -342,7 +342,7 @@ sent.
|
|||
== Added a Message size limit
|
||||
|
||||
Cassandra pre-4.0 doesn't protect the server from allocating huge
|
||||
buffers for the inter-node Message objects. Adding a message size limit
|
||||
buffers for the internode Message objects. Adding a message size limit
|
||||
would be good to deal with issues such as a malfunctioning cluster
|
||||
participant. Version 4.0 introduced max message size config param, akin
|
||||
to max mutation size - set to endpoint reserve capacity by default.
|
||||
|
|
@ -356,5 +356,5 @@ message from another node. Pre-4.0, we close the connection and
|
|||
reconnect, which can cause other concurrent queries to fail. Version 4.0
|
||||
fixes the issue by wrapping message in-stream with
|
||||
`TrackedDataInputPlus`, catching `UnknownCFException`, and skipping the
|
||||
remaining bytes in this message. TCP won't be closed and it will remain
|
||||
remaining bytes in this message. TCP won't be closed, and it will remain
|
||||
connected for other messages.
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ It implements a partitioned wide-column storage model with eventually consistent
|
|||
This initial design implemented a combination of Amazon's https://www.cs.cornell.edu/courses/cs5414/2017fa/papers/dynamo.pdf[Dynamo] distributed storage and replication techniques and Google's https://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf[Bigtable] data and storage engine model.
|
||||
Dynamo and Bigtable were both developed to meet emerging requirements for scalable, reliable and highly available storage systems, but each had areas that could be improved.
|
||||
|
||||
{product} was designed as a best-in-class combination of both systems to meet emerging largescale, both in data footprint and query volume, storage requirements.
|
||||
{product} was designed as a best-in-class combination of both systems to meet emerging large scale, both in data footprint and query volume, storage requirements.
|
||||
As applications began to require full global replication and always available low-latency reads and writes, a new kind of database model was required to meet these new requirements.
|
||||
Relational database systems at that time struggled to meet the requirements.
|
||||
|
||||
|
|
@ -28,9 +28,9 @@ Relational database systems at that time struggled to meet the requirements.
|
|||
language, to create, modify, and delete database schema, as well as access data.
|
||||
CQL allows users to organize data within a cluster of {cassandra} nodes using:
|
||||
|
||||
* Keyspace: Defines how a dataset is replicated, per datacenter.
|
||||
Replication is the number of copies saved per cluster.
|
||||
Keyspaces contain tables.
|
||||
* Keyspace: Specifies the replication strategy for a dataset across different datacenters.
|
||||
Replication refers to the number of copies stored within a cluster.
|
||||
Keyspaces serve as containers for tables.
|
||||
* Table: Tables are composed of rows and columns.
|
||||
Columns define the typed schema for a single datum in a table.
|
||||
Tables are partitioned based on the columns provided in the partition key.
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
= Storage Engine
|
||||
|
||||
The {Cassandra} storage engine is optimized for high performance, write-oriented workloads. The architecture is based on Log Structured Merge (LSM) trees, which utilize an append-only approach instead of the traditional relational database design with B-trees. This creates a write path free of read lookups and bottlenecks.
|
||||
The Cassandra storage engine is optimized for high performance, write-oriented workloads. The architecture is based on Log Structured Merge (LSM) trees, which utilize an append-only approach instead of the traditional relational database design with B-trees. This creates a write path free of read lookups and bottlenecks.
|
||||
|
||||
While the write path is highly optimized, it comes with tradeoffs in terms of read performance and write amplification. To enhance read operations, Cassandra uses Bloom filters when accessing data from stables. Bloom filters are remarkably efficient, leading to generally well-balanced performance for both reads and writes.
|
||||
|
||||
Compaction is a necessary background activity required by the ‘merge’ phase of Log Structured Merge trees. Compaction creates write amplification when several small SSTables on disk are read, merged, updates and deletes processed, and a new ssstable is re-written. Every write of data in Cassandra is re-written multiple times, known as write amplification, and this adds background I/O to the database workload.
|
||||
Compaction is a necessary background activity required by the ‘merge’ phase of Log Structured Merge trees. Compaction creates write amplification when several small SSTables on disk are read, merged, updates and deletes processed, and a new SSTable is re-written. Every write of data in Cassandra is re-written multiple times, known as write amplification, and this adds background I/O to the database workload.
|
||||
|
||||
The core storage engine consists of memtables for in-memory data and immutable SSTables (Sorted String Tables) on disk. Data in SSTables is stored sorted to enable efficent merge sort during compaction. Additionally, a write-ahead log (WAL), referred to as the commit log, ensures resiliency for crash and transaction recovery.
|
||||
The core storage engine consists of memtables for in-memory data and immutable SSTables (Sorted String Tables) on disk. Data in SSTables is stored sorted to enable efficient merge sort during compaction. Additionally, a write-ahead log (WAL), referred to as the commit log, ensures resiliency for crash and transaction recovery.
|
||||
|
||||
The sequence of the steps in the write path:
|
||||
|
||||
|
|
@ -18,19 +18,19 @@ The sequence of the steps in the write path:
|
|||
[[commit-log]]
|
||||
== Logging writes to commit logs
|
||||
|
||||
When a write occurs, {cassandra} writes the data to a local append-only (https://cassandra.apache.org/_/glossary.html#commit-log)[commit log] on disk.
|
||||
This action provides xref:cassandra:managing/configuration/cass_yaml_file.adoc[configurable durability] by logging every write made to a {cassandra} node.
|
||||
When a write operation takes place, Cassandra records the data in a local append-only https://cassandra.apache.org/_/glossary.html#commit-log[commit log] on disk.
|
||||
This action provides xref:cassandra:managing/configuration/cass_yaml_file.adoc[configurable durability] by logging every write made to a Cassandra node.
|
||||
If an unexpected shutdown occurs, the commit log provides permanent durable writes of the data.
|
||||
On startup, any mutations in the commit log will be applied to (https://cassandra.apache.org/_/glossary.html#memtable)[memtables].
|
||||
On startup, any mutations in the commit log will be applied to https://cassandra.apache.org/_/glossary.html#memtable[memtables].
|
||||
The commit log is shared among tables.
|
||||
|
||||
All mutations are write-optimized on storage in commit log segments, reducing the number of seeks needed to write to disk.
|
||||
Commit log segments are limited by the xref:cassandra:managing/configuration/cass_yaml_file.adoc#commitlog_segment_size[`commitlog_segment_size`] option.
|
||||
Once the defined size is reached, a new commit log segment is created.
|
||||
Commit log segments can be archived, deleted, or recycled once all the data is flushed to
|
||||
(https://cassandra.apache.org/_/glossary.html#sstable)[SSTables].
|
||||
Commit log segments are truncated when {cassandra} has written data older than a certain point to the SSTables.
|
||||
Running xref:managing:tools/nodetool/drain.adoc[`nodetool drain`] before stopping {cassandra} will write everything in the memtables
|
||||
https://cassandra.apache.org/_/glossary.html#sstable[SSTables].
|
||||
Commit log segments are truncated when Cassandra has written data older than a certain point to the SSTables.
|
||||
Running xref:managing:tools/nodetool/drain.adoc[`nodetool drain`] before stopping Cassandra will write everything in the memtables
|
||||
to SSTables and remove the need to sync with the commit logs on startup.
|
||||
|
||||
* xref:cassandra:managing/configuration/cass_yaml_file.adoc#commitlog_segment_size [`commitlog_segment_size`]: The default size is 32MiB, which is almost always fine, but if you are archiving commitlog segments (see commitlog_archiving.properties), then you probably want a finer granularity of archiving; 8 or 16 MiB is reasonable.
|
||||
|
|
@ -42,18 +42,18 @@ If `max_mutation_size` is set explicitly then `commitlog_segment_size` must be s
|
|||
====
|
||||
|
||||
* xref:cassandra:managing/configuration/cass_yaml_file.adoc#commitlog_sync[`commitlog_sync`]: may be either _periodic_ or _batch_.
|
||||
** `batch`: In batch mode, {cassandra} won't acknowledge writes until the commit log has been fsynced to disk.
|
||||
** `batch`: In batch mode, Cassandra won't acknowledge writes until the commit log has been fsynced to disk.
|
||||
+
|
||||
** `periodic`: In periodic mode, writes are immediately acknowledged, and the commit log is simply synced every "commitlog_sync_period" milliseconds.
|
||||
+
|
||||
- `commitlog_sync_period`: Time to wait between "periodic" fsyncs
|
||||
_Default Value:_ 10000ms
|
||||
|
||||
_Default Value:_ batch
|
||||
_Default Value:_ periodic
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
In the event of an unexpected shutdown, {cassandra} can lose up to the sync period or more if the sync is delayed.
|
||||
In the event of an unexpected shutdown, Cassandra can lose up to the sync period or more if the sync is delayed.
|
||||
If using `batch` mode, it is recommended to store commit logs in a separate, dedicated device.
|
||||
====
|
||||
|
||||
|
|
@ -67,7 +67,7 @@ _Default Value:_ `/var/lib/cassandra/commitlog`
|
|||
If omitted, the commit log will be written uncompressed.
|
||||
LZ4, Snappy,Deflate and Zstd compressors are supported.
|
||||
|
||||
_Default Value:_ (complex option):
|
||||
_Default Value:_
|
||||
|
||||
[source, yaml]
|
||||
----
|
||||
|
|
@ -77,7 +77,7 @@ _Default Value:_ (complex option):
|
|||
|
||||
* xref:cassandra:managing/configuration/cass_yaml_file.adoc#commitlog_total_space[`commitlog_total_space`]: Total space to use for commit logs on disk.
|
||||
This option is commented out by default.
|
||||
If space gets above this value, {cassandra} will flush every dirty table in the oldest segment and remove it.
|
||||
If space gets above this value, Cassandra will flush every dirty table in the oldest segment and remove it.
|
||||
So a small total commit log space will tend to cause more flush activity on less-active tables.
|
||||
The default value is the smallest between 8192 and 1/4 of the total space of the commitlog volume.
|
||||
|
||||
|
|
@ -85,11 +85,12 @@ _Default Value:_ 8192MiB
|
|||
|
||||
== Memtables
|
||||
|
||||
When a write occurs, {cassandra} also writes the data to a memtable.
|
||||
Memtables are in-memory structures where {cassandra} buffers writes.
|
||||
In general, there is one active memtable per table.
|
||||
The memtable is a write-back cache of data partitions that {cassandra} looks up by key.
|
||||
Memtables may be stored entirely on-heap or partially off-heap, depending on xref:cassandra:managing/configuration/cass_yaml_file.adoc#memtable_allocation_type[`memtable_allocation_type`].
|
||||
When Cassandra accepts new write requests, it saves the data in two places: an in-memory write-back cache called a memtable and the CommitLog.
|
||||
The memtable buffers writes and allows serving reads without accessing the disk, while the CommitLog ensures durability by appending new mutations.
|
||||
Typically, there is one active memtable per table, acting as a cache for data partitions that Cassandra accesses by key.
|
||||
|
||||
Depending on the xref:cassandra:managing/configuration/cass_yaml_file.adoc#memtable_allocation_type[`memtable_allocation_type`], memtables can be stored entirely on-heap or partially off-heap.
|
||||
If Cassandra crashes before flushing the memtable, it can restore acknowledged writes by replaying the commitLog.
|
||||
|
||||
The memtable stores writes in sorted order until reaching a configurable limit.
|
||||
When the limit is reached, memtables are flushed onto disk and become immutable xref:#sstables[SSTables].
|
||||
|
|
@ -102,8 +103,8 @@ When a triggering event occurs, the memtable is put in a queue that is flushed t
|
|||
Flushing writes the data to disk, in the memtable-sorted order.
|
||||
A partition index is also created on the disk that maps the tokens to a location on disk.
|
||||
|
||||
The queue can be configured with either the xref:cassandra:managing/configuration/cass_yaml_file.adoc#memtable_heap_space[`memtable_heap_space`] or xref:cassandra:managing/configuration/cass_yaml_file.adoc#memtable_offheap_space[`memtable_offheap_space`] setting in the `cassandra.yaml` file.
|
||||
If the data to be flushed exceeds the `memtable_cleanup_threshold`, {cassandra} blocks writes until the next flush succeeds.
|
||||
The queue can be configured with either the xref:cassandra:managing/configuration/cass_yaml_file.adoc#memtable_heap_space[`memtable_heap_space`] or xref:cassandra:managing/configuration/cass_yaml_file.adoc#memtable_offheap_space[`memtable_offheap_space`] setting in the `cassandra.yaml` file.
|
||||
If the data to be flushed exceeds the `memtable_cleanup_threshold`, Cassandra blocks writes until the next flush succeeds.
|
||||
You can manually flush a table using xref:managing:tools/nodetool/flush.adoc[`nodetool flush`] or `nodetool drain` (flushes memtables without listening for connections to other nodes).
|
||||
To reduce the commit log replay time, the recommended best practice is to flush the memtable before you restart the nodes.
|
||||
If a node stops working, replaying the commit log restores writes to the memtable that were there before it stopped.
|
||||
|
|
@ -112,12 +113,12 @@ Data in the commit log is purged after its corresponding data in the memtable is
|
|||
|
||||
== SSTables
|
||||
|
||||
https://cassandra.apache.org/_/glossary.html#sstable[SSTables] are the immutable data files that {cassandra} uses for persisting data on disk.
|
||||
https://cassandra.apache.org/_/glossary.html#sstable[SSTables] are the immutable data files that Cassandra uses for persisting data on disk.
|
||||
SSTables are maintained per table.
|
||||
SSTables are immutable, and never written to again after the memtable is flushed.
|
||||
Thus, a partition is typically stored across multiple SSTable files, as data is added or modified.
|
||||
|
||||
Each SSTable is comprised of multiple components stored in separate files:
|
||||
Each SSTable consists of multiple components stored in separate files:
|
||||
|
||||
`Data.db`::
|
||||
The actual data, i.e. the contents of rows.
|
||||
|
|
@ -148,7 +149,7 @@ Only present if SAI is enabled for the table.
|
|||
[NOTE]
|
||||
====
|
||||
Note that the `Index.db` file type is replaced by `Partitions.db` and `Rows.db`.
|
||||
This change is a consequence of the inclusion of Big Trie indexes in Cassandra (https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-25%3A+Trie-indexed+SSTable+format[CEP-25]).
|
||||
This change is a consequence of the inclusion of Big Trie indexes in Cassandra https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-25%3A+Trie-indexed+SSTable+format[CEP-25].
|
||||
====
|
||||
|
||||
Within the `Data.db` file, rows are organized by partition.
|
||||
|
|
@ -157,18 +158,18 @@ Within a partition, rows are stored in the order of their clustering keys.
|
|||
|
||||
SSTables can be optionally compressed using block-based compression.
|
||||
|
||||
As SSTables are flushed to disk from `memtables` or are streamed from other nodes, {cassandra} triggers compactions which combine multiple SSTables into one.
|
||||
As SSTables are flushed to disk from `memtables` or are streamed from other nodes, Cassandra triggers compactions which combine multiple SSTables into one.
|
||||
Once the new SSTable has been written, the old SSTables can be removed.
|
||||
|
||||
== SSTable Versions
|
||||
|
||||
From (https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L338)[BigFormat#BigVersion].
|
||||
From https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L338[BigFormat#BigVersion].
|
||||
|
||||
The version numbers, to date are:
|
||||
|
||||
=== Version 0
|
||||
|
||||
* b (0.7.0): added version to sstable filenames
|
||||
* b (0.7.0): added version to SSTable filenames
|
||||
* c (0.7.0): bloom filter component computes hashes over raw key bytes
|
||||
instead of strings
|
||||
* d (0.7.0): row size in data component becomes a long instead of int
|
||||
|
|
@ -199,7 +200,7 @@ millis-based id (see CASSANDRA-4782)
|
|||
real format change, this is mostly a marker to know if we should expect
|
||||
super columns or not. We do need a major version bump however, because
|
||||
we should not allow streaming of super columns into this new format)
|
||||
** tracks max local deletiontime in sstable metadata
|
||||
** tracks max local deletiontime in SSTable metadata
|
||||
** records bloom_filter_fp_chance in metadata component
|
||||
** remove data size and column count from data file (CASSANDRA-4180)
|
||||
** tracks max/min column values (according to comparator)
|
||||
|
|
@ -208,7 +209,7 @@ we should not allow streaming of super columns into this new format)
|
|||
** checksum the compressed data
|
||||
* ka (2.1.0):
|
||||
** new Statistics.db file format
|
||||
** index summaries can be downsampled and the sampling level is
|
||||
** index summaries can be down sampled and the sampling level is
|
||||
persisted
|
||||
** switch uncompressed checksums to adler32
|
||||
** tracks presence of legacy (local and remote) counter shards
|
||||
|
|
@ -222,12 +223,12 @@ persisted
|
|||
** store rows natively
|
||||
* mb (3.0.7, 3.7): commit log lower bound included
|
||||
* mc (3.0.8, 3.9): commit log intervals included
|
||||
* md (3.0.18, 3.11.4): corrected sstable min/max clustering
|
||||
* me (3.0.25, 3.11.11): added hostId of the node from which the sstable originated
|
||||
* md (3.0.18, 3.11.4): corrected SSTable min/max clustering
|
||||
* me (3.0.25, 3.11.11): added hostId of the node from which the SSTable originated
|
||||
|
||||
=== Version 4
|
||||
|
||||
* na (4.0-rc1): uncompressed chunks, pending repair session, isTransient, checksummed sstable metadata file, new Bloomfilter format
|
||||
* na (4.0-rc1): uncompressed chunks, pending repair session, isTransient, checksummed SSTable metadata file, new Bloom filter format
|
||||
* nb (4.0.0): originating host id
|
||||
|
||||
=== Version 5
|
||||
|
|
@ -241,14 +242,15 @@ persisted
|
|||
Cassandra 5.0 introduced new SSTable formats BTI for Trie-indexed SSTables.
|
||||
To use the BTI formats configure it `cassandra.yaml` like
|
||||
|
||||
```
|
||||
[source]
|
||||
----
|
||||
sstable:
|
||||
selected_format: bti
|
||||
```
|
||||
----
|
||||
|
||||
Versions come from (https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/io/sstable/format/bti/BtiFormat.java#L287)[BtiFormat#BtiVersion].
|
||||
Versions come from https://github.com/apache/cassandra/blob/f16fb6765b8a3ff8f49accf61c908791520c0d6e/src/java/org/apache/cassandra/io/sstable/format/bti/BtiFormat.java#L302[BtiFormat#BtiVersion].
|
||||
|
||||
For implementation docs see (https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/io/sstable/format/bti/BtiFormat.md)[BtiFormat.md].
|
||||
For implementation docs see https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/io/sstable/format/bti/BtiFormat.md[BtiFormat.md].
|
||||
|
||||
=== Version 5
|
||||
|
||||
|
|
@ -256,7 +258,7 @@ For implementation docs see (https://github.com/apache/cassandra/blob/cassandra-
|
|||
|
||||
=== Example Code
|
||||
|
||||
The following example is useful for finding all sstables that do not
|
||||
The following example is useful for finding all SSTables that do not
|
||||
match the "ib" SSTable version
|
||||
|
||||
[source,bash]
|
||||
|
|
|
|||
|
|
@ -68,8 +68,7 @@ Zero copy streaming is enabled by setting the following setting in
|
|||
stream_entire_sstables: true
|
||||
....
|
||||
|
||||
By default zero copy streaming is enabled.
|
||||
|
||||
Zero copy streaming is enabled by default.
|
||||
=== SSTables Eligible for Zero Copy Streaming
|
||||
|
||||
Zero copy streaming is used if all partitions within the SSTable need to
|
||||
|
|
@ -88,7 +87,7 @@ network transfer significantly subject to throttling specified by
|
|||
Enabling this will reduce the GC pressure on sending and receiving node.
|
||||
While this feature tries to keep the disks balanced, it cannot guarantee
|
||||
it. This feature will be automatically disabled if internode encryption
|
||||
is enabled. Currently this can be used with Leveled Compaction.
|
||||
is enabled. Currently, this can be used with Leveled Compaction.
|
||||
|
||||
=== Configuring for Zero Copy Streaming
|
||||
|
||||
|
|
@ -161,7 +160,7 @@ Repair with `nodetool repair` involves streaming of repaired SSTables
|
|||
and a repair preview has been added to provide an estimate of the amount
|
||||
of repair streaming that would need to be performed. Repair preview
|
||||
(https://issues.apache.org/jira/browse/CASSANDRA-13257[CASSANDRA-13257])
|
||||
is invoke with `nodetool repair --preview` using option:
|
||||
is invoked with `nodetool repair --preview` using option:
|
||||
|
||||
....
|
||||
-prv, --preview
|
||||
|
|
|
|||
Loading…
Reference in New Issue