This is a partial solution to IllegalStateException thrown by VectorPostings. It works by using a single shard at L0 when a vector index is present. As noted in the jira ticket, there are edge cases that may still produce errors, notably the case where there are multiple data directories.
The key trade offs here are related to the time complexity for search. Since graph search is log(n), and searching m graphs is m * log(n), we see better search performance by building bigger graphs which is essentially log(m * n). We could pre-shard, which comes at a cost of increased search time complexity.
patch by Michael Marshall,Dmitry Konstantinov; reviewed by Caleb Rackliffe,Dmitry Konstantinov,Michael Semb Wever for CASSANDRA-19661
Co-authored-by: Michael Marshall <mmarshall@apache.org>
Co-authored-by: Dmitry Konstantinov <netudima@gmail.com>
Introduce the ability to override compaction strategy for specific keyspaces
and tables at startup via two new system properties:
- cassandra.override_compaction.entities: comma-separated list of keyspaces
and keyspace.table pairs (e.g. "ks1,ks2.tbl1,ks3.tbl2")
- cassandra.override_compaction.params: JSON compaction parameters to apply
Patch by Paulo Motta; Reviewed by Jaydeepkumar Chovatia for CASSANDRA-21169
Add test container memory events printing to check if Linux OOM killer was active
Fix thread dump printing on junit test timeout
Process.pid() API is available since JDK9
patch by Dmitry Konstantinov; reviewed by Michael Semb Wever for CASSANDRA-21172
Too many threads can lead to OOM, so new Thread is replaced with an executor
spin wait is added to HintsServiceTest.testPageSeek to avoid flaky NPE (HintsStore is populated in an async way)
patch by Dmitry Konstantinov; reviewed by Brandon Williams for CASSANDRA-21166
The runtime overflow guard test (testLargeIndexSummary) required ~2 GiB of
off-heap memory to trigger the Integer.MAX_VALUE boundary in
IndexSummaryBuilder.maybeAddEntry. SafeMemoryWriter's 50% buffer growth
strategy caused ~5 GiB peak memory during reallocation (old + new buffer),
exceeding CI medium node limits (3.5-5 GB).
Extract the hardcoded Integer.MAX_VALUE threshold in IndexSummaryBuilder
into a @VisibleForTesting field (maxEntriesSize), allowing the test to
exercise the same guard logic with a 2 MB threshold instead. Rename both
large summary tests to reflect their distinct code paths: runtime truncation
vs constructor-initiated downsampling.
patch by Sam Lightfoot; reviewed by Brandon Williams,Dmitry Konstantinov,Michael Semb Wever for CASSANDRA-20599
The two deletes use the same CQL timestamp, so compaction should merge
their overlapping range tombstones into a single open/close pair (no
boundary markers). However, DeletionTime also includes local_delete_time
(wall-clock seconds), which may differ between deletes if a second
boundary is crossed during the flush. Differing local_delete_time causes
the merger to treat them as distinct deletions, producing boundary
markers instead. Using executeInternalWithNowInSec with a fixed nowInSec
ensures both deletes get identical DeletionTime values.
patch by Sam Lightfoot; reviewed by Dmitry Konstantinov,Brandon Williams for CASSANDRA-21163
This is unnecessary code as the current implementation of CEP-54 is not implemeting / exercising these features.
patch by Stefan Miklosovic; reviewed by Yifan Cai for CASSANDRA-21154
This change adds support for using O_DIRECT (direct I/O) when reading SSTables
during compaction. Direct I/O bypasses the OS page cache, which can be beneficial
for compaction workloads where data is typically read once and not accessed again
soon after.
Key changes:
- Add DiskAccessMode.direct option for scan operations
- Introduce DirectThreadLocalByteBufferHolder and DirectThreadLocalReadAheadBuffer
for aligned buffer management required by O_DIRECT
- Add ByteBufferHolder interface to abstract buffer management
- Modify FileHandle to support building direct I/O readers via toBuilder()
- Add startup check to verify Direct IO support on configured data directories
- Enable read-ahead for uncompressed data in Direct CompressedChunkReader
- Move direct IO support logic into FileHandle (from SSTableReader)
- Add comprehensive tests for direct I/O chunk readers and buffer holders
patch by Sam Lightfoot; reviewed by Ariel Weisberg and Maxwell Guo for CASSANDRA-19987
Add microbench-test target for quick test run
Unify declaration and add assertions on target types
Fix dirname usages for macos
Parameter for docker_timeout
Add more jmh class ignores (that are broken)
Add disableResume() and better host debug to Jenkinsfile
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov for CASSANDRA-18873
Legacy Cassandra 3.x sstables use an old bloom filter format that is
incompatible with zero-copy streaming in Cassandra 5.0+. This patch
automatically detects sstables with the old bloom filter format
(pre-4.0) and disables zero-copy streaming for them, allowing legacy
sstables to be loaded via sstableloader without requiring manual flags.
The fix adds a version check in CassandraOutgoingFile.computeShouldStreamEntireSSTables()
that calls descriptor.version.hasOldBfFormat() to detect legacy sstables.
patch by Paulo Motta; reviewed by Stefan Miklosovic for CASSANDRA-21092
This commit adds a new configuration, data_disk_usage_keyspace_wide_protection_enabled, which ensures that if any node which replicates a given keyspace is full, all writes to that keyspace are blocked.
patch by Isaac Reath; reviewed by Stefan Miklosovic, Paulo Motta for CASSANDRA-20124
Closes#4547
Based on the official documentation here (1), AsyncProfiler works best with these kernel parameters.
Not all CI environments have them set. That would fail the tests which run the startup check
if kernel parameters are not like that.
It might be empirically observed that some AsyncProfiler functionality works even without these
kernel parameters, it might be investigated in follow-up work if we should not relax these conditions a bit.
(1) https://github.com/async-profiler/async-profiler/blob/master/docs/GettingStarted.md#before-profiling
It is also possible to override them via nodetool if necessary.
This patch also fixes the computation of sampling ratio to not lose the precision.
patch by Stefan Miklosovic; reviewed by Jyothsna Konisha, Yifan Cai for CASSANDRA-21078
Adds a compaction implementation utilizing new fixed allocation SSTable reader/writer implementations, and other purpose built code, leading to improved efficiencies.
patch by Nitsan Wakart; reviewed by Branimir Lambov, Dmitry Konstantinov for CASSANDRA-20918
By default the expiry time is calculated on each peer independently. It can be
made to converge by disabling gossip quarantine using the configuration setting
gossip_quarantine_disabled or via a hotprop on GossiperMBean.
Patch by Sam Tunnicliffe; reviewed by Marcus Eriksson for CASSANDRA-21035
* cassandra-5.0:
Fix CQLSSTableWriter serialization of vector of date and time patch by Lukasz Antoniak; reviewed by Andres de la Pena, Yifan Cai for CASSANDRA-20979
Also Fix Cassandra:
- In memory size calculation for CommandsForKey include Unmanaged
- Accord load out-of-band cleanup should use SafeRedundantBefore
ALso Improve Cassandra:
- Report replay information on begin replay
- Improve AccordService shutdown
- Log command store RedundantBefore on shutdown
- Segment compaction should wait for readOrder barrier to replace segments, for additional safety
- Journal segments should share readOrder with sstables
Also Improve Accord:
- Iterate LocalListeners in order, so can query more effectively on node
- Refine AbstractReplay.minReplay/shouldReplay
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21804
Use a plain loop to check if it is ASCII symbol before going into more complicated UTF8 parsing.
Avoid ValueAccessor to get extra boost for the ASCII check, especially in non-monomorphic cases.
Patch by Dmitry Konstantinov; reviewed by Jyothsna Konisa, Stefan Miklosovic for CASSANDRA-21075
- DefaultLocalListeners.ComplexListeners iterator IndexOutOfBoundsException
- Race condition initialising empty ActiveEpochs, when minimum pending epoch can move backwards
- SyncPoints must be declared in an epoch containing the ranges, and PENDING_REMOVAL ranges will reject non-syncpoint transactions
- AccordExecutorMetrics is now registered on startup
- getRecentValues for non-cumulative histogram should not subtract prior values
Improve:
- Report ephemeral read, epoch waits and timeout metrics
- Remove Topologies.SelectNodeOwnership, as no need to SLICE anymore
- Introduce SystemEventListener for epoch waiting and timeout metrics
- No-op but log if gcBefore provided to CFK is in the past
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21076
Additionally replace List with array for bind values (we know the size in advance during a decoding), so in total: List<List> is replaced with byte[][] QueryOptions classes support both ways to get values now: using an old API with ByteBuffer and a new API with byte[].
Patch by Dmitry Konstantinov; reviewed by Michael Semb Wever for CASSANDRA-20166
Converting collections or UDTs to raw bytes is nonsensical - it
allows reading raw serialized bytes which have no meaningful
interpretation.
patch by Mikołaj Diakowski; reviewed by Stefan Miklosovic, Brandon Williams for CASSANDRA-20982
- DurabilityQueue/ShardScheduler deadlock
- MemtableCleanerThread.Cleanup assumes Boolean parameter is non-null, which is invalid if an exception has been thrown
- AccordDurableOnFlush may be invoked while Accord is starting up, so should use AccordService.unsafeInstance
- AccordCache shrink without lock regression
- Cleanup system_accord compaction leftovers before starting up
- system_accord_debug.txn order
- system_accord_debug.txn_blocked_by order
- system_accord_debug.shard_epochs order
Improve:
- Set DefaultProgressLog.setMode(Catchup) during Catchup
- IdentityAccumulators only need to readLast, not readAll
- Limit number of static segments we compact at once to sstable
- If too many static segments on startup, wait for them to be compacted
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21053
Also Fix:
- Restore MaxDecidedRX on replay
- When catchup_on_start_exit_on_failure == false, should startup on any kind of failure, not only timeout
- lazy vtable LIMIT clause regression
- DurabilityService.onEpochRetired
- Command.validate when uniqueHlc differs
- Avoid unsafe publication of AccordExecutor to scheduledFastTasks
- AccordCache hitRate metric names
- use long for return type of DurationSpec.toNanoseconds
- Repair without all replicas should not request all Accord replicas participate
- ExecuteAtSerializer
- SyncPoints should be coordinated in an epoch that contains the ranges
Also Improve:
- Split Accord startup into local+distributed, ensure we
- Add logging to FetchDurableBefore on startup
- Add randomised testing of AbstractLazyVirtualTable
- Add validation of lazy virtual table key ordering
- Don't send requests to faulty replicas
- shrinkOrEvict large objects without holding lock
- Accord dtest shutdown
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21042
* Better handling of DOWN unupgraded nodes
* Test that aborted CMS initialization cleans up state properly
* Clean up orphaned PreInitialize entries in the log on bounce
* Unconditionally reset initiator during abort
* Ensure that metadata log entries aren't exchanged before CMS
initialization is complete
* Precalculate common serialization version, excluding non-upgraded and
LEFT nodes
* Decide if metadata-impacting upgrade is in progress using min common
version
* Add metadata identifier to nodetool cms output
Patch by Sam Tunnicliffe and Marcus Eriksson; reviewed by Sam
Tunnicliffe and Marcus Eriksson for CASSANDRA-21036
Co-authored-by: Marcus Eriksson <marcuse@apache.org>
Co-authored-by: Sam Tunnicliffe <samt@apache.org>
For GC like ShenandoahGC/ZGC etc., there are GC events that do not have STW pauses (Concurrent phases).
Operator might find it reasonable to use lower thresholds for events require STW pauses and higher thresholds for concurrent phases.
gc_concurrent_phase_log_threshold and gc_concurrent_phase_warn_threshold configuration options are introduced in cassandra.yaml
Patch by Yuqi Yan; reviewed by Dmitry Konstantinov, Stefan Miklosovic for CASSANDRA-20980
Co-authored-by: Stefan Miklosovic
- Estimate memory required to allocate for applying a partition update into a memtable and allocate this memory in one shot, then use it as a request-local SLAB
- Reduce contention by switching MemtableAllocator.SubAllocator#owns from updates via AtomicLongFieldUpdater to LongAdder usage. MemtableAllocator.SubAllocator#acquired(..) method updates "owns" value but does not use the updated result.
- Reduce contention by replacing of CAS loop in MemtablePool.SubPool#tryAllocate with allocatedUpdater.addAndGet(this, size)
Patch by Dmitry Konstantinov; reviewed by Michael Semb Wever for CASSANDRA-20226
Before the fix during a capacity extension BufferPoolAllocator returned to BufferPool a sliced ByteBuffer wrapper object instead of the originally allocated one, so the ByteBuffer was not recycled by BufferPool
Adjust BufferPoolAllocatorTest to test the ByteBuf capacity extension with a real BufferPool behavior
Patch by Dmitry Konstantinov; reviewed by Michael Semb Wever for CASSANDRA-20753
Also Introduce:
- Sharded/LogLinearDecayingHistogram
Also Improve:
- Do not take a reference to CFK unless relevant
Also Fix:
- Sharded/LogLinearHistogram
- ExecuteFlags serialization bug in ReadData
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21017
- All retries are appended to a delay queue so overlaps can be pruned
- Quorum successes are not retried if there are superseding sync points covering the ranges
- User-initiated requests are not purged unless the request has timed out or otherwise completed
- Overlapping requests are queued up against the next to run
Alsp (C*):
- Catch-up with quorums on restart
- Manage an ordered set of keys in cache for faster range searches
Also (Accord):
- Update copy of BTree and import IntervalBTree
- Fix RedundantStatus WAS_OWNED_OVERRIDE_MASK
- Add Catchup mechanism to reach parity with a quorum on restart
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21013
CASSANDRA-20389 implemented limit on table name length of 222
characters. This commit updates the documentation to reflect this.
It also adds an assert in CreateTableValidationTest to ensure that the
documented limit matches the actual constant.
patch by Ruslan Fomkin; reviewed by Brad Schoening, Dmitry Konstantinov for CASSANDRA-20914
Merge ConfigurationService with TopologyManager to remove cyclic dependency and duplicated work.
Also:
- Improve visibility of work blocking topology processing
- Ensure we cannot double-count peers when deciding an epoch's distributed readiness
- Remove the possibility of distributed stalls, by processing new topologies as soon as a
contiguous sequence is known locally, regardless of whether anyprior local epoch is ready to
coordinate or has recorded this fact to peers. The notification of local readiness continues
to be processed only once all prior epochs are ready, but we can begin using and readying
later epochs immediately.
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20998
Also fix:
- Paxos should not update minHlc if AccordService is not setup
- Remove EndpointMapper methods that require non-null, to ensure call-sites handle null case explicitly
- AccordRepair should specify the Node Ids that must participate, but these should not affect the durability requirements for GC
- Avoid erroneously marking UniversalOrInvalidated when only known to quorum
- AccordSyncPropagator should not consult FailureDetector as it may throw UnknownEndpointException
- Refuse ReplaceSameAddress if accord enabled
- Reset Accord topologies on restart if no local command stores to ensure we can catch up (as intervening epochs may have otherwise been GC'd)
- MaybeRecover should not abort if found durable if either: 1) prevProgressToken already witnessed durable; or 2) Home state has requested we recover anyway
- Invariant in MaybeRecover is too strong now we permit recovering when known to be stable
- ReadData must respond with InsufficientAndWaiting if not Stable to ensure Durability is inferred correctly by ExecuteTxn
- Avoid StackOverflowException in NotifyWaitingOn
- PreAccept should honour the REJECTED flag of any member of the latest topology if the vote would be relied upon for any prior epoch's quorum
- Resume Accord bootstrap on restart
- Fix burn test non-determinism
- Reset Accord topologies on restart if no local command stores to ensure we can catch up (as intervening epochs may have otherwise been GC'd)
- Exclude stale or removed ids from sync points
- Terminate a failed DurabilityRequest if a required participant is now removed/hardRemoved
Also improve:
- Introduce accord_debug_keyspace.{listeners_txn, listeners_local}
- Filter and record faulty in AbstractCoordination
- If we are the home shard, and our home progress status is done, we should not wait for the home shard to advance the waiting progress state
- Introduce CommandStore.tryExecuteListening to try to execute all waiting transactions and their dependencies
- ExecuteTxn on recovery should not wait, to avoid consuming progress log concurrency slots for transactions that cannot execute
- HomeState should update its phase based on the command status to handle operator request to retry all states
- Separate home/wait queue in DefaultProgressLog
- Additional tracing
- Do not call trySendMore from prerecordFailure
- Force full recovery for sync points if we have lost quorum
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20921
- Introduce pattern tracing, that can intercept failed or new coordinations matching various filters
- Support additional tracing event collection modes (SAMPLE and RING)
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20911
Also Improve:
- Searchable system_accord_debug.txn
- Integrate txn_blocked_by deadline and depth filter with execution logic, to ensure we terminate promptly and get a best effort reply
patch by Benedict; reviewed by Aleksey Yeschenko for CASSANDRA-20899
To support recovering a node that has lost some of its local transaction log, introduce rebootstrap and unsafe bootstrap modes, where Accord ensures no responses are produced for transactions the node cannot be certain it had not previously answered.
patch by Benedict and Alex Petrov for CASSANDRA-20908
* cassandra-5.0:
ninja-fix – Fix eclipse-warnings error for CASSANDRA-19564
ReadCommandController should close fast to avoid deadlock when building secondary index
* cassandra-4.1:
ninja-fix – Fix eclipse-warnings error for CASSANDRA-19564
ReadCommandController should close fast to avoid deadlock when building secondary index
Codahale metrics do not provide the ability to create custom metric implementations, so we have to inherit from Codahale classes.
For better cache locality rate and counter values are extracted to a common thread-local arrays.
Threads death is tracked using 2 approaches: FastThreadLocal.onRemoval callback and phantom references to Thread objects.
Phantom references are used to track aliveness of metric users and reusing of metric IDs.
Patch by Dmitry Konstantinov; reviewed by Benedict Elliott Smith for CASSANDRA-20250
Dmitry Konstantinov