This is a partial solution to IllegalStateException thrown by VectorPostings. It works by using a single shard at L0 when a vector index is present. As noted in the jira ticket, there are edge cases that may still produce errors, notably the case where there are multiple data directories.
The key trade offs here are related to the time complexity for search. Since graph search is log(n), and searching m graphs is m * log(n), we see better search performance by building bigger graphs which is essentially log(m * n). We could pre-shard, which comes at a cost of increased search time complexity.
patch by Michael Marshall,Dmitry Konstantinov; reviewed by Caleb Rackliffe,Dmitry Konstantinov,Michael Semb Wever for CASSANDRA-19661
Co-authored-by: Michael Marshall <mmarshall@apache.org>
Co-authored-by: Dmitry Konstantinov <netudima@gmail.com>
Introduce the ability to override compaction strategy for specific keyspaces
and tables at startup via two new system properties:
- cassandra.override_compaction.entities: comma-separated list of keyspaces
and keyspace.table pairs (e.g. "ks1,ks2.tbl1,ks3.tbl2")
- cassandra.override_compaction.params: JSON compaction parameters to apply
Patch by Paulo Motta; Reviewed by Jaydeepkumar Chovatia for CASSANDRA-21169
Add test container memory events printing to check if Linux OOM killer was active
Fix thread dump printing on junit test timeout
Process.pid() API is available since JDK9
patch by Dmitry Konstantinov; reviewed by Michael Semb Wever for CASSANDRA-21172
Too many threads can lead to OOM, so new Thread is replaced with an executor
spin wait is added to HintsServiceTest.testPageSeek to avoid flaky NPE (HintsStore is populated in an async way)
patch by Dmitry Konstantinov; reviewed by Brandon Williams for CASSANDRA-21166
The runtime overflow guard test (testLargeIndexSummary) required ~2 GiB of
off-heap memory to trigger the Integer.MAX_VALUE boundary in
IndexSummaryBuilder.maybeAddEntry. SafeMemoryWriter's 50% buffer growth
strategy caused ~5 GiB peak memory during reallocation (old + new buffer),
exceeding CI medium node limits (3.5-5 GB).
Extract the hardcoded Integer.MAX_VALUE threshold in IndexSummaryBuilder
into a @VisibleForTesting field (maxEntriesSize), allowing the test to
exercise the same guard logic with a 2 MB threshold instead. Rename both
large summary tests to reflect their distinct code paths: runtime truncation
vs constructor-initiated downsampling.
patch by Sam Lightfoot; reviewed by Brandon Williams,Dmitry Konstantinov,Michael Semb Wever for CASSANDRA-20599
The two deletes use the same CQL timestamp, so compaction should merge
their overlapping range tombstones into a single open/close pair (no
boundary markers). However, DeletionTime also includes local_delete_time
(wall-clock seconds), which may differ between deletes if a second
boundary is crossed during the flush. Differing local_delete_time causes
the merger to treat them as distinct deletions, producing boundary
markers instead. Using executeInternalWithNowInSec with a fixed nowInSec
ensures both deletes get identical DeletionTime values.
patch by Sam Lightfoot; reviewed by Dmitry Konstantinov,Brandon Williams for CASSANDRA-21163
It restores the original approach when data files are read only in an extended mode, as it takes more time
It fixes sstable.VerifyTest.testVerifyCorruptRowCorrectDigest test failure
patch by Arvind Kandpal; reviewed by Dmitry Konstantinov,Brandon Williams for CASSANDRA-21150
avoid ClusteringIndexSliceFilter allocation if a write does not required a read (plain usual write), avoid iterator allocation, use array instead of ArrayList for perStatementOptions which does not grow dynamically
BatchStatement: check if many similar rows for the same table are written unconditionally, in this case we can avoid columns info merging and builders allocation
add fast path in nonTokenRestrictionValues logic, when we specify a single partition key (a single row) to modify, optimize also the case if a partition or clustering key is a single column
add fast path in valuesAsClustering logic, when we specify a single clustering key (a single row) to modify
UpdateParameters: allocate DeletionTime on demand (it is not needed if we do insert/updates)
serializedRowBodySize: avoid capturing lamda allocation per cell by moving capturing arguments to SerializationHelper (same optimization as it was done in serializeRowBody)
force hash3_x64_128 inlining to help JIT with escape analysis and long[] heap allocation elimination
patch by Dmitry Konstantinov; reviewed by Chris Lohfink for CASSANDRA-21141
This is unnecessary code as the current implementation of CEP-54 is not implemeting / exercising these features.
patch by Stefan Miklosovic; reviewed by Yifan Cai for CASSANDRA-21154
This change adds support for using O_DIRECT (direct I/O) when reading SSTables
during compaction. Direct I/O bypasses the OS page cache, which can be beneficial
for compaction workloads where data is typically read once and not accessed again
soon after.
Key changes:
- Add DiskAccessMode.direct option for scan operations
- Introduce DirectThreadLocalByteBufferHolder and DirectThreadLocalReadAheadBuffer
for aligned buffer management required by O_DIRECT
- Add ByteBufferHolder interface to abstract buffer management
- Modify FileHandle to support building direct I/O readers via toBuilder()
- Add startup check to verify Direct IO support on configured data directories
- Enable read-ahead for uncompressed data in Direct CompressedChunkReader
- Move direct IO support logic into FileHandle (from SSTableReader)
- Add comprehensive tests for direct I/O chunk readers and buffer holders
patch by Sam Lightfoot; reviewed by Ariel Weisberg and Maxwell Guo for CASSANDRA-19987
Add microbench-test target for quick test run
Unify declaration and add assertions on target types
Fix dirname usages for macos
Parameter for docker_timeout
Add more jmh class ignores (that are broken)
Add disableResume() and better host debug to Jenkinsfile
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov for CASSANDRA-18873
Legacy Cassandra 3.x sstables use an old bloom filter format that is
incompatible with zero-copy streaming in Cassandra 5.0+. This patch
automatically detects sstables with the old bloom filter format
(pre-4.0) and disables zero-copy streaming for them, allowing legacy
sstables to be loaded via sstableloader without requiring manual flags.
The fix adds a version check in CassandraOutgoingFile.computeShouldStreamEntireSSTables()
that calls descriptor.version.hasOldBfFormat() to detect legacy sstables.
patch by Paulo Motta; reviewed by Stefan Miklosovic for CASSANDRA-21092
This commit adds a new configuration, data_disk_usage_keyspace_wide_protection_enabled, which ensures that if any node which replicates a given keyspace is full, all writes to that keyspace are blocked.
patch by Isaac Reath; reviewed by Stefan Miklosovic, Paulo Motta for CASSANDRA-20124
Closes#4547
Michael Marshall