This patch adds a new configuration file, cassandra_latest.yaml, which changes
some of the settings to use new features we usually recommend to users,
including:
- Big Trie-Indexed SSTables
- Trie memtables
- Unified Compaction Strategy
- Storage Attached Index
- Off-heap objects memtables
- Direct-write commit log
- No storage compatibility
These changes are applied in new test targets for unit, in-jvm and python dtests that are
run pre-commit. When changes are made to the new config file, python dtest will use it
directly, but unit and in-jvm dtests require updates, respectively, to latest_diff.yaml
and InstanceConfig.java.
patch by Branimir Lambov; reviewed by Berenguer Blasi for CASSANDRA-18753
/bin/sh (our shebang in bin/cassandra) does not recognize the
`if` construct which is available in /bin/bash only hence if
MX4J_ADDRESS or MX4J_PORT was not commented out, the script would
fail to execute it.
patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-19416
patch by Andrés de la Peña; reviewed by David Capwell for CASSANDRA-19336
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
Co-authored-by: David Capwell <dcapwell@apache.org>
patch by Claude Warren; reviewed by Stefan Miklosovic, Jacek Lewandowski, Michael Semb Wever for CASSANDRA-16565
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
Patch by Amit Pawar and Jacek Lewandowski; reviewed by Branimir Lambov and Maxwell Guo for CASSANDRA-18464
Co-authored-by: Amit Pawar <Amit.Pawar@amd.com>
Co-authored-by: Jacek Lewandowski <lewandowski.jacek@gmail.com>
An overview of the core components can be found in the included
TransactionalClusterMetadata.md
patch by Alex Petrov, Marcus Eriksson and Sam Tunnicliffe; reviewed by
Alex Petrov, Marcus Eriksson and Sam Tunnicliffe for CASSANDRA-18330
Co-authored-by: Marcus Eriksson <marcuse@apache.org>
Co-authored-by: Alex Petrov <oleksandr.petrov@gmail.com>
Co-authored-by: Sam Tunnicliffe <samt@apache.org>
- Add Bouncycastle dependency
- Upgrade tcnative boringssl
- Add TLSv1.3 to encryption options tests
- Revert defaults after changes in Netty 4.1.75
- Remove Guava 18 from deps - we accidentally ended with Guava 30+ and 18 on the classpath because JimFS includes it as a transient dependency.
Patch by Jacek Lewandowski and Brandon Williams; reviewed by Ekaterina Dimitrova and Berenguer Blasi for CASSANDRA-17992
Co-authored-by: Jacek Lewandowski <lewandowski.jacek@gmail.com>
Co-authored-by: Brandon Williams <driftx@gmail.com>
patch by Ayushi Singh; reviewed by Stefan Miklosovic, Michael Semb Wever and Maxim Muzafarov for CASSANDRA-18624
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
This commit contains the following additions
to SAI:
- The index write path and index building
based around StorageAttachedIndexBuilder
and StorageAttachedIndexWriter
- The on-disk index versioning using the
SSTable Descriptor analog IndexDescriptor
with Version and OnDiskFormat
- The literal on-disk index using the
LiteralIndexWriter
patch by Mike Adamson; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-18062
Co-authored-by: Mike Adamson <mikeatdot@gmail.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Andres de la Peña <a.penya.garcia@gmail.com>
Co-authored-by: Piotr Kołaczkowski <pkolaczk@gmail.com>
Co-authored-by: Jason Rutherglen <jason.rutherglen@gmail.com>
Remove --add-opens java.base/java.nio=ALL-UNNAMED which is not needed anymore
patch by Ekaterina Dimitrova; reviewed by Jacek Lewandowki for CASSANDRA-17909
This upgrade also fixes issues with PhantomReferences and the test problems from CASSANDRA-17884 anad CASSANDRA-16304
patch by Benjamin Lerer; reviewed by Ekaterina Dimitrova for CASSANDRA-18239
As we were implementing the snitch itself, we noticed that the constructors
for cloud-based snitches are unnecessarilly complicated and
we took the opportunity to make them simpler.
patch by Stefan Miklosovic; reviewed by German Eichberger and Jacek Lewandowski for CASSANDRA-18646
The approach is documented in the included UnifiedCompactionStrategy.md.
Also included:
- Generalize prettyPrint to prettyPrintBinary (with 2^10 multiplier,
e.g. MiB) and prettyPrintDecimal (with 1000 multiplier, e.g. ms)
that cover the full range of double, and add a parseHumanReadable
method that can read both.
- CASSANDRA-18123: Fix invalid reuse of metadata collector during flushing
- Fix invalid token range splitting with non-unit weights
- Add flushSizeOnDisk metric
- Add ability to change compaction default in YAML
patch by Alex Sorokoumov, Branimir Lambov, Dimitar Dimitrov and Stefania Alborghetti
reviewed by Alex Sorokoumov, Jaroslaw Grabowski and Maxwell Guo for CASSANDRA-18397
The patch also refactors existing cloud snitches to get rid of the duplicate code,
this is the logical follow-up of CASSANDRA-16555 where AbstractCloudMetadataServiceConnector was introduced.
patch by Stefan Miklosovic; reviewed by Jacek Lewandowski, Jackson Fleming and Maxwell Guo for CASSANDRA-18438
patch by Stefan Miklosovic; reviewed by Jacek Lewandowski and Brandon Williams for CASSANDRA-16555
Co-authored-by: Jacek Lewandowski <lewandowski.jacek@gmail.com>
Co-authored-by: Paul Rütter <paul@blueconic.com>
patch by Stefan Podkowinski; reviewed by Robert Stupp for CASSANDRA-10508
backported in CASSANDRA-18575 by German Eichberger; reviewed by
brandonwilliams
Documentation in the included BTIFormat.md.
patch by Branimir Lambov and Jacek Lewandowski; reviewed by Caleb Rackliffe and Maxwell Guo for CASSANDRA-18398
- refactored sstable format configuration
- sstable formats are discovered via ServiceLoader
- options configuration for sstable formats can be included in yaml
- yaml may include selected sstable format and version (version is not yet supported)
- auto saved caches refactored - they include additional metadata component which contains necessary mappings
patch by Jacek Lewandowski; reviewed by David Capwell for CASSANDRA-18441
Allows to attach the native masking functions and UDFs to the definitions of
table columns in the schema, as defined by CEP-20.
The functions masking a column can be specified on CREATE TABLE queries, right
at the end of the column definition. The mask of a column can also be changed
or dropped with an ALTER TABLE query. Once a column is masked, SELECT queries
will always return the masked value of the column. That masking is done on the
coordinator, at the end of the query execution. Thus, masking won't affect any
filters or ordering, which would be based on the clear values of the masked
columns. Column masks are stored on the table system_schema.column_masks.
A new UNMASK permission allows to see the clear data of columns with an
attached mask. Also, a new SELECT_MASKED permission allows to run SELECT
queries restricting the clear values of masked columns. Superusers have both
permissions by default, whereas regular users don't have them.
Dynamic data masking can be enabled/disabled with the config property
dynamic_data_masking_enabled in cassandra.yaml. It is disabled by default.
This is the combination of multiple tickets:
* Add masking functions to column metadata (CASSANDRA-18068)
* Add UNMASK permission (CASSANDRA-18069)
* Add SELECT_MASKED permission (CASSANDRA-18070)
* Add support for using UDFs as masking functions (CASSANDRA-18071)
* Add feature flag for dynamic data masking (CASSANDRA-18316)
patch by Andrés de la Peña; reviewed by Benjamin Lerer and Berenguer Blasi for CASSANDRA-17940
Some transitive dependencies to these have also been added to raise their versions. asm-* transitive dependencies have been excluded.
patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova for CASSANDRA-18049
Summary of the changes:
Format, reader and writer
---------------------------
There are a lot of refactorings around sstable related classes aiming to extract the most generic functionality to the top-level entities and push down implementation-specific stuff to the actual implementation. In Particular, the top-level, implementation agnostic classes/interfaces are SSTableFormat interface, SSTable, SSTableReader, SSTableWriter, IVerifier, and IScrubber. The rest of the codebase has been reviewed for explicit usages of big table format-specific usages of sstable classes and refactored. SSTable, SSTableReader, and SSTableWriter have their builders. Builders make a hierarchy that follows the same inheritance structure as readers and writers.
There are also partial implementations that add support for some features and may or may not be used by the custom implementations. They include:
- AbstractSSTableFormat - adds an implementation of some initialization methods - in practice, all of the format implementations should extend this class
- SSTableReaderWithFilter - add support for Bloom filter to the reader
- SortedTableWriter - generic implementation for a writer which writes partitions in the default order to the data file, supports Bloom filter and some index of partitions
- IndexSummarySupport - interface implemented by the readers using index summaries
- KeyCacheSupport - interface implemented by the readers using row key cache
Descriptor
---------------------------
Refactored the Descriptor class so that:
- All paths are created from the base directory File rather than from a String
- All the methods named *filename* producing full paths were made private; their current implementations are returning file names rather than paths (the naming was inconsistent)
- The usages of the `filenameFor` method were refactored to use the `fileFor` method
- The usages of the `fromFilename` method were refactored to use a `fromFileWithComponent(..., false).left` expression
In essence, the Descriptor class is no longer working on String-based paths.
Index summaries
---------------------------
Removed the index summary from the generic SSTableReader class and created an interface IndexSummarySupport to be implemented by the readers that need it. Methods in related classes that refer back to the reader were refactored to support just readers of the SSTableReader & IndexSummarySupport type. Therefore, we will no longer need to assume that the generic SSTableReader has anything to do with an index summary.
A new IndexSummaryComponent class encloses data fields from the index summary file (note that aside from the index summary itself, the file includes the first and last partition of the sstable). The class has been extracted to deal with those fields and have that logic in a single place.
Filter
---------------------------
Refactored IFilter and its serialization - in particular, added the `serialize` method to the IFilter interface and moved loading/saving logic to a separate utility class FilterComponent.
Extracted the SSTableReaderWithFilter abstract reader extending the generic SSTableReader with filter support.
Extracted bloom filter metrics into separate entities allowing to plug them in if the implementation uses a filter.
Cache
---------------------------
Refactored CacheService to support different key-cache values. CacheService now supports arbitrary IRowIndexEntry implementation as a key-cache value. A new version of the auto-saving cache was created ("g") because some information about the type of serialized row index entry needs to be known before it is deserialized (or skipped). Therefore, the SSTableFormat type ordinal number is stored, which is sufficient because the IRowIndexEntry serializer is specific to the sstable format type.
Similarly to the IndexSummarySupport, a new KeyCacheSupport interface has to be implemented to mark the reader as supporting key-cache. It contains the default implementation of several methods the rest of the system relies on when the key-cache is supported.
Other changes
---------------------------
- Fixed disabling chunk cache - enable(boolean) method in ChunkCache does not make any sense - it makes a false impression it can disable chunk cache once enabled, while in fact, it only clears it. Added setFileCacheEnabled to DatabaseDescriptor
- Made WrappingUnfilteredRowIterator an interface
- DataInputStreamPlus extends InputStream - this makes it possible for input stream-based inheritors of DataInputPlus to extend DataInputStreamPlus. It simplifies coding because sometimes we want to get DataInputPlus implementation extending InputStream as an argument.
- Table and keyspace metrics were made pluggable - in particular, added the ability for a certain format to register gauges that are specific only to that format and make no sense for others
- Implemented mmapped region extension for compressed data
- Refactored FileHandle so that it is no longer closable
- Implemented WrappingRebufferer
- Introduced the SSTable.Owner interface to make SSTable implementation not reference higher-level entities directly. SSTable accepts passing null as the owner when there is no owner (like sometimes in offline tools) or passing a mock when needed in tests.
Individual commits
---------------------------
[4a87cd36fe] Fix disabling chunk cache
[c84c75ccf3] Made WrappingUnfilteredRowIterator an interface
[253d2b828e] Add getType to SSTableFormat
[3f169dcc20] Remove getIndexSerializer from SSTableFormat
[05bae1833b] Pull down rowIndexEntrySerializer field
[da675f2809] Moved RowIndexEntry
[673f0c5c39] Reduce usages of RowIndexEntry
[c72538be91] Refactor CacheService to support for different key cache values
[54d33ee656] Minor refactoring of ColumnIndex
[93862df967] Just moved AbstractSSTableIterator to o.a.c.io.sstable.format
[9e4566a1de] Refactored AbstractSSTableIterator
[a4e61e80bb] Extracted IScrubber and IVerifier interfaces
[20f78c7419] Push down implementation of SSTableReader.firstKeyBeyond
[f2c24e5774] Moved SSTableReader.getSampleIndexesForRanges to IndexSummary
[b6c3a6c1ea] Moved SSTableReader.getKeySamples implementation to IndexSummary
[c4b90ebb33] Refactor InstanceTidier so that it is more generic
[918d5a9e74] Refactor dropping page cache
[a52fb4d558] Refactor sstable metrics
[f6d10f930f] NEW (fix up) - DataInputStreamPlus extends InputStream
[8f6a56d972] Getting rid of index summary in SSTableReader
[4a918bf725] Removed direct usages of primary index from SSTableReader
[358fa32602] Refactor KeyIterator so that it is sstable format agnostic
[14c09d89c2] Remove explicit usage of Components outside of format specific classes
[feff14e137] Move clone methods implementation from SSTableReader to BigTableReader
[64e9787b10] Move saveIndexSummary and saveBloomFilter to SSTableReaderBuilder
[ae71fe6ed8] Moved indexSummary field to BigTableReader and made it private
[df9fd8c4b9] Moved ifile field to BigTableReader and made it private
[2be6ea9ecf] Moved static open methods for BigTableReader to the reader factory
[bc0e55ac48] Minor refactoring around IFilter and its serialization
[5b95704beb] Minor refactorings around IndexSummary
[87812335e8] Extracted TOCComponent class to deal with TOC file
[fdad092a6a] Extracted CompressionInfoComponent class
[39b47e388d] Extracted StatsComponent as a helper for elements of SSTable metadata
[cdb55bff47] Fix SSTable.getMinimalKey
[b99c6d5805] Refactor FileHandle so that it is no longer closable
[77b7f7ace5] Implement WrappingRebufferer
[b6868914dd] Add progressPercentage to ProgressInfo
[7fd4956e5b] Moved copy/rename/hardLink methods from SSTableWriter to SSTable
[1ccc6bf148] Create generic SSTableBuilder and IOOptions
[da58a81102] Refactor SSTableReaderBuilder
[4501ddba1c] Refactor ColumnIndex
[d4f9e1a64b] Extracted non-big-table-specific functionality from BigTableWriter to SortedTableWriter
[379525d01e] Refactor BigTableZeroCopyWriter to SSTableZeroCopyWriter as it is not specific to big format
[8ac37f83bc] Extract EmptySSTableScanner out from BigTableScanner
[ee6673f1cf] Implement SSTableWriterBuilder
[bb26629235] Refactor opening early / final
[a327595015] Refactored SSTableWriter factory
[16ffd7334b] Extract non-big-format-specific logic from scrubber and verifier
[75e02db6af] Allow to specify the default SSTableFormat via system property
[a7b9d0d628] Small fixes around streaming
[407f977c36] Move guard collection size
[0529e57d2f] Remove explicit references to big format
[61509963ec] Unclassified minor changes
[da28d1af3a] Replaced getCreationTimeFor(Component) with getDataCreationTime()
[e99c834de6] !!! Reformatting
[882b7baa5a] Rename SSTableReader.maybePresent and fix its redundant usages
[b70c983bea] Implement mmapped region extension for compressed data
[d7ff3970de] Introduce SSTable.Owner interface
[e9feb9c462] Replaced getCreationTimeFor(Component) with getDataCreationTime()
[ee8082fb07] Created SSTableFormat.deleteOrphanedComponents
[e62950fd3d] Refactor metrics further
[cefa5b3814] Extract key cache support into separate entity
[dd55101ca1] Extracted SSTableReaderWithFilter
[510b651824] Implement customizable component types
[2be512d9fa] Pluggable SSTableFormat by making SSTableFormat.Type not an enum
[670836b55d] Refactor CRC and digest validators
[00c91103bc] Extract delete method to delete SSTables and purge row cache entries
[0819dc9fc2] Extracted trySkipFileCacheBefore(key) to SSTableReader
[732f841750] Added missing overrides in ForwardingSSTableReader
[db623218fd] Update DatabaseDescriptorRefTest
[c018c468e5] Cleanup
[eafc836242] Add @SuppressWarnings("resource") where needed
[3b7c911dd6] Documentation
patch by Jacek Lewandowski, reviewed by Branimir Lambov for CASSANDRA-17056
Co-authored-by: @jacek-lewandowski
Co-authored-by: @blambov
patch by Ben Bromhead; reviewed by Stefan Miklosovic and Andrés de la Peña for CASSANDRA-14361
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
patch by Stefan Miklosovic; reviewed by Andrés de la Peña, Josh McKenzie, Brandon Williams and Brad Schoening for CASSANDRA-18042
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
* Add optional reason for guarding an operation to every guardrail
* Add reason for disabling ALLOW FILTERING into cassandra.yaml
* Consider allow_filtering_enabled guardrail in the error message for needed ALLOW FILTERING
patch by Andrés de la Peña; reviewed by Berenguer Blasi and Josh McKenzie for CASSANDRA-17967
Patch by Caleb Rackliffe; reviewed by Josh McKenzie, David Capwell, and Jon Meredith for CASSANDRA-17795
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Josh McKenzie <jmckenzie@apache.org>
commented out by mistake in a previous patch;
Fix breaking change with cache_load_timeout; cache_load_timeout_seconds <=0 and cache_load_timeout=0 are equivalent
and they both mean disabled;
Deprecate public method setRate(final double throughputMbPerSec) in Compaction Manager in favor of
setRateInBytes(final double throughputBytesPerSec);
Revert breaking change removal of StressCQLSSTableWriter.Builder.withBufferSizeInMB(int size). Deprecate it in favor
of StressCQLSSTableWriter.Builder.withBufferSizeInMiB(int size);
Fix precision issues, add new -m flag (for nodetool/setstreamthroughput, nodetool/setinterdcstreamthroughput,
nodetool/getstreamthroughput and nodetoo/getinterdcstreamthroughput), add new -d flags (nodetool/getstreamthroughput,
nodetool/getinterdcstreamthroughput, nodetool/getcompactionthroughput);
Fix a bug with precision in nodetool/compactionstats;
Deprecate StorageService methods and add new ones for stream_throughput_outbound, inter_dc_stream_throughput_outbound,
compaction_throughput_outbound in the JMX MBean `org.apache.cassandra.db:type=StorageService`;
Removed getEntireSSTableStreamThroughputMebibytesPerSec in favor of new getEntireSSTableStreamThroughputMebibytesPerSecAsDouble
in the JMX MBean `org.apache.cassandra.db:type=StorageService`;
Removed getEntireSSTableInterDCStreamThroughputMebibytesPerSec in favor of getEntireSSTableInterDCStreamThroughputMebibytesPerSecAsDouble
in the JMX MBean `org.apache.cassandra.db:type=StorageService`
Patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, Francisco Guerrero for CASSANDRA-17225
patch by Ekaterina Dimitrova and Andres de la Pena; reviewed by Andres de la Pena and Caleb Rackliffe for CASSANDRA-17571
Co-authored-by: Ekaterina Dimitrova <ekaterina.dimitrova@datastax.com>
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova and Stefan Miklosovic for CASSANDRA-17150
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
Co-authored-by: Zhao Yang <jasonstack.zhao@gmail.com>
Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>
patch by Andrés de la Peña; reviewed by Berenguer Blasi and Stefan Miklosovic for CASSANDRA-17197
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
Co-authored-by: Aleksandr Sorokoumov <aleksandr.sorokoumov@gmail.com>
patch by Andrés de la Peña; reviewed by Caleb Rackliffe for CASSANDRA-17188
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
Co-authored-by: Aleksandr Sorokoumov <aleksandr.sorokoumov@gmail.com>
patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova for CASSANDRA-17187
Co-authored-by: Aleksandr Sorokoumov <aleksandr.sorokoumov@gmail.com>
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
Branimir Lambov