This patch removes deprecated chunk_length_kb and sstable_compression in favor of
chunk_length_in_kb and class parameters.
The only deprecated compression parameter left, crc_check_chance (which is configurable outside
of compression parameters as a standalone table parameter) is meant to be treated separately.
patch by Stefan Miklosovic; reviewed by Maxim Muzafarov and Brandon Williams for CASSANDRA-18742
Refactoring prevents the situation where some sstable components, like
data or index, are created before the new sstable is registered with
lifecycle transaction, which leads to a problem such that there is
a short time when incomplete sstable components are present. At the same
time, no transaction file is created, which leads to the possibility
that the sstable can be recognized as completed by various
transaction-aware listers.
Patch by Jacek Lewandowski; reviewed by Branimir Lambov, Mike Adamson for CASSANDRA-18737
This patch fixes a bug in the SegmentMetadata that
was only storing the partition key for min and max
primary keys for a segment. It also contains some
refactoring of the PrimaryKey to remove the deferred
loading of PrimaryKeys by the PrimaryKeyMaps.
Patch by Mike Adamson; reviewed by Caleb Rackliffe and Andrés de la Peña for CASSANDRA-18734
This patch removes the PRIMARY_KEY_TRIE component and adds KeyLookup.Cursor#clusteredSeekToKey() to
search for clustering keys within a partition. To do this a new on-disk component
PARTITION_SIZES has been added that holds the size of each partition in the SSTable.
patch by Mike Adamson; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-18673
patch by Aleksey Yeschenko; reviewed by Blake Eggleston, Sam Tunnicliffe, and Caleb Rackliffe for CASSANDRA-16905
Co-authored by Aleksey Yeschenko (aleksey@apache.org)
Co-authored by Josh McKenzie (jmckenzie@apache.org)
This patch also resolves CASSANDRA-18723 by bumping bcpkix-jdk15on of 1.70
to bcpkix-jdk18on of 1.76.
patch by Stefan Miklosovic; reviewed by Jacek Lewandowski CASSANDRA-18729
Contrary to expectations, we do permit negative timestamps.
This reverts checks that treat such timestamps as corruption
introduced in CASSANDRA-18676.
patch by Branimir Lambov; reviewed by Ekaterina Dimitrova for CASSANDRA-18735
This commit skips the initialization of `topPartitions` in `org.apache.cassandra.db.ColumnFamilyStore`
when running in client or tool mode. The `TopPartitionTracker` class will attempt to query the system
keyspace, which when running in client or tool mode will not be part of the KeyspaceMetadata. This
causes a warning to be printed out with a stacktrace that can be misleading. The warning is similar to
this:
```
WARN org.apache.cassandra.db.SystemKeyspace: Could not load stored top SIZES partitions for ...
org.apache.cassandra.db.KeyspaceNotDefinedException: keyspace system does not exist
at org.apache.cassandra.schema.Schema.validateTable(Schema.java:xxx) ~[?:?]
at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:xxx) ~[?:?]
at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:xxx) ~[?:?]
at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:xxx) ~[?:?]
at org.apache.cassandra.cql3.QueryProcessor.parseAndPrepare(QueryProcessor.java:xxx) ~[?:?]
...
```
In this commit, we check whether we run in client or tool mode, and skip initialization
of `topPartitions` in those cases.
Patch by Francisco Guerrero; Reviewed by Dinesh Joshi, Yifan Cai for CASSANDRA-18697
patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova and Maxwell Guo for CASSANDRA-18640
Co-authored-by: Jonathan Ellis <jbellis@datastax.com>
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
patch by Ayushi Singh; reviewed by Stefan Miklosovic, Michael Semb Wever and Maxim Muzafarov for CASSANDRA-18624
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
- Avoid validation in response to SSTableAddedNotification, as it should already have been done somewhere else
- Change SSTableWriter to prevent commit when a failure is thrown out of an index build
patch by Caleb Rackliffe; reviewed by Mike Adamson and Andres de la Peña for CASSANDRA-18670
- Fix checksum calculation in IncrementalChecksumSequentialWriter
- Checksum per-SSTable and per-column components after streaming
- Avoid validating indexes when full rebuild is requested
patch by Piotr Kołaczkowski; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-18490
Co-authored-by: Piotr Kołaczkowski <pkolaczk@gmail.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Includes:
- The disk/v1/kdtree package containing the
kdtree writer and reader
- The implementation code to tie these into
the existing read and write paths. The main parts
of this are the NumericIndexWriter and the
NumericIndexSegmentSearcher
- Additional testing for the new code
patch by Mike Adamson; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-18067
Co-authored-by: Mike Adamson <madamson@datastax.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Piotr Kołaczkowski <pkolaczk@gmail.com>
Co-authored-by: Jason Rutherglen <jason.rutherglen@gmail.com>
Co-authored-by: Zhao Yang <zhaoyangsingapore@gmail.com>
Notes on the upgrade path:
- RamIndexOutput is replaced with ResettableByteBuffersIndexOutput, an extension of ByteBuffersIndexOutput, which was the closest thing to a replacement of RamIndexOutput.
- Lucene exposes the code we needed from DirectReaders more or less directly in DirectReader now, so the old copied code has been deleted.
- Lucene changed its data files to be little endian, but to keep its compatibility story simple it retained BE for the header and footer ints. That's the cause of the changes in SAICodecUtils.
- We could gain a bit of performance making our own code natively little endian but that is too big of a change for this patch.
patch by Jonathan Ellis; reviewed by Andrés de la Peña, Caleb Rackliffe, and Mike Adamson for CASSANDRA-18494
This commit contains the following additions
to SAI:
- The index write path and index building
based around StorageAttachedIndexBuilder
and StorageAttachedIndexWriter
- The on-disk index versioning using the
SSTable Descriptor analog IndexDescriptor
with Version and OnDiskFormat
- The literal on-disk index using the
LiteralIndexWriter
patch by Mike Adamson; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-18062
Co-authored-by: Mike Adamson <mikeatdot@gmail.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Andres de la Peña <a.penya.garcia@gmail.com>
Co-authored-by: Piotr Kołaczkowski <pkolaczk@gmail.com>
Co-authored-by: Jason Rutherglen <jason.rutherglen@gmail.com>
This includes the following elements of the Storage Attached Index:
- Memtable-attached indexes backed by an in-memory trie structure for byte-comparable values
- Query path for the in-memory index
- Index status propagation
- Randomized testing for Memtable-attached indexes
patch my Mike Adamson; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-18058
Co-authored-by: Mike Adamson <madamson@datastax.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
patch by Zhao Yang; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-16092
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Andres de la Peña <a.penya.garcia@gmail.com>
Treat AssertionError as corruption, assert positive deletion timestamps and
TTLs and treat localDeletionTime < TTL as invalid.
patch by Branimir Lambov; reviewed by Berenguer Blasi for CASSANDRA-18676
Remove --add-opens java.base/java.nio=ALL-UNNAMED which is not needed anymore
patch by Ekaterina Dimitrova; reviewed by Jacek Lewandowki for CASSANDRA-17909
Bump source and target version to 11 for Java UDFs
ICompilationUnit has new default methods
patch by Ekaterina Dimitrova; reviewed by Andres de la Pena and Berenguer Blasi for CASSANRA-18190
Fix ClassCastException from jdk GaloisCounterMode when using JDK17 provider
Add javac exports to build.xml for java.base/jdk.internal.ref and java.base/sun.nio.ch
Replace checktestnameshelper ant task in build.xml with _check-test-names java target to be able to pass jvm args to TestNameCheckTask
patch by Dan Janieks; revewed by Ekaterina Dimitrova and Andres de la Pena for CASSANDRA-18180
This upgrade also fixes issues with PhantomReferences and the test problems from CASSANDRA-17884 anad CASSANDRA-16304
patch by Benjamin Lerer; reviewed by Ekaterina Dimitrova for CASSANDRA-18239
As we were implementing the snitch itself, we noticed that the constructors
for cloud-based snitches are unnecessarilly complicated and
we took the opportunity to make them simpler.
patch by Stefan Miklosovic; reviewed by German Eichberger and Jacek Lewandowski for CASSANDRA-18646
The approach is documented in the included UnifiedCompactionStrategy.md.
Also included:
- Generalize prettyPrint to prettyPrintBinary (with 2^10 multiplier,
e.g. MiB) and prettyPrintDecimal (with 1000 multiplier, e.g. ms)
that cover the full range of double, and add a parseHumanReadable
method that can read both.
- CASSANDRA-18123: Fix invalid reuse of metadata collector during flushing
- Fix invalid token range splitting with non-unit weights
- Add flushSizeOnDisk metric
- Add ability to change compaction default in YAML
patch by Alex Sorokoumov, Branimir Lambov, Dimitar Dimitrov and Stefania Alborghetti
reviewed by Alex Sorokoumov, Jaroslaw Grabowski and Maxwell Guo for CASSANDRA-18397
The patch also refactors existing cloud snitches to get rid of the duplicate code,
this is the logical follow-up of CASSANDRA-16555 where AbstractCloudMetadataServiceConnector was introduced.
patch by Stefan Miklosovic; reviewed by Jacek Lewandowski, Jackson Fleming and Maxwell Guo for CASSANDRA-18438
this patch also fixes incorrect visualisation of SSTable min/max local deletion time when no
tombstones are present in a SSTable
patch by Stefan Miklosovic; reviewed by Marcus Eriksson and Brandon Williams for CASSANDRA-18639
patch by Vincent White; reviewed by Stefan Miklosovic and Branimir Lambov for CASSANDRA-14204
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
patch by Chris Lohfink; reviewed by Caleb Rackliffe and Benedict Elliott Smith for CASSANDRA-15241
Co-authored-by: Chris Lohfink <clohfink@apple.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Benedict Elliott Smith <benedict@apache.org>
patch by Stefan Miklosovic; reviewed by Jacek Lewandowski and Brandon Williams for CASSANDRA-16555
Co-authored-by: Jacek Lewandowski <lewandowski.jacek@gmail.com>
Co-authored-by: Paul Rütter <paul@blueconic.com>
patch by Manish Ghildiyal; reviewed by Stefan Miklosovic, Brandon Williams for CASSANDRA-18305
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
patch by Stefan Podkowinski; reviewed by Robert Stupp for CASSANDRA-10508
backported in CASSANDRA-18575 by German Eichberger; reviewed by
brandonwilliams
Patch by Ningzi Zhan and Tomasz Lasica; reviewed by brandonwilliams,
edimitrova and Maxwell Guo for CASSANDRA-17818
Co-Authored-By: Tomek Lasica <tomasz.lasica@datastax.com>
patch by Benjamin Lerer and Andrés de la Peña; reviewed by Benjamin Lerer for CASSANDRA-18566
Co-authored-by: Benjamin Lerer <b.lerer@gmail.com>
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
If an sstable is corrupted in a nasty way, we may read invalid cell sizes and try to read much more data for a row than we should. In rare scenarios this can lead even to OOMs.
This simple fix adds tracking and limiting the amount of data that is read per row. Row has its size stored in preamble which can be used as a limit. If the deserialization code tries to read more than that, it will simply fail with EOF which will prevent more serious problems later.
Patch by Jacek Lewandowski; reviewed by Berenguer Blasi and Maxwell Guo for CASSANDRA-18513
# Conflicts:
# src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java
Stefan Miklosovic