org.apache.cassandra.utils.MergeIterator.Candidate - has Comparator field which is the same for all Candidates under an MergeIterator instance, we can move the field to MergeIterator level. Candidate is 4.8% of all allocations and we can save 20% (~1% of all allocations) of it for JDK 21 with compressed references on.
switch from ArrayList to array in org.apache.cassandra.db.rows.Row.Merger
The ArrayList iterator costs 0.72% and BulkIterator.Adapter costs 0.35%. So, in total we can save here ~1% of total allocations.
patch by Dmitry Konstantinov; reviewed by Francisco Guerrero for CASSANDRA-21359
The following subcommands have been added:
* abortbootstrap Cancel and undo an inflight bootstrap
* abortmove Cancel and undo an inflight token move
* abortdecommission Cancel and undo an inflight decommission
* assassinate Remove a node from cluster metadata
* resetcms Redefine the CMS membership
* move Modify a node's (single) owned token
* describe Output a summary of the cluster metadata
* forcejoin Force a registered or joining node into a JOINED state
* print Output a full rendering of cluster metadata
* printdirectory Output a full rendering of the directory
* printdataplacements Output the full data placements
Patch by Venkata Harikrishna Nukala; reviewed by Alex Petrov and Sam Tunnicliffe
for CASSANDRA-19151
- TransactionStatement token aware routing
- Mitigate convoy effect:
- ExecuteTxnBacklog v1 can execute single key transaction backlogs and disseminate the result
- Direct local execution possible for single key transactions
- Use shardAppliedBefore instead of gcBefore to cleanup CFK faster
Also Improve:
- Introduce distributed tracing
- Combine RejectBefore and MaxConflicts
- Self-addressed messages are delivered directly
- Disable AccordSyncPropagator by default, as incompatible with large clusters
- FastPathStrategy.SIMPLE should not modify fast path contents; FastPathStrategy.UP introduced to adopt this behaviour
- Configurable queue prioritisation
- ProtocolModifiers mostly final
- Remove unnecessary condition check when serializing TxnUpdate
Also Fix:
- SequentialExecutor ownership bug
- BTree.XUpdater.reset()
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for CASSANDRA-21361
use compression ratio to scale the amount of remaining disk usage
CompactionInfo::estimatedRemainingWriteBytes is uncompressed, causing compaction to be rejected due to size when it has space to complete
patch by Matt Byrd; reviewed by Dmitry Konstantinov, Marcus Eriksson for CASSANDRA-21245
Co-authored-by: Jon Meredith <jmeredithco@apple.com>
SinglePartitionReadCommand.Group#create - typical single partition select, use singletonList instead of ArrayList + Object[]
SelectStatement#getSliceCommands - remove unnessesary cloning of partition key, it is a part of a very old patch, now we copy all values from Netty buffers
SelectStatement#getSliceCommands - typical single partition select, use singletonList instead of ArrayList + Object[]
SinglePartitionReadQuery.Group#maybeValidateIndex, avoid iterators for restrictionSet if a full partition is selected
patch by Dmitry Konstantinov; reviewed by Francisco Guerrero, Stefan Miklosovic for CASSANDRA-21351
Best effort attempt to get tableId from CFS, fallback to null which
still allows snapshot to be loaded
End result is that nodetool clearsnapshot and listsnapshot function for such
snapshots on restart
In addition to 5.0 patch we move starting SnapshotManager to after TCM is initiliazed.
The consequence of such a move is that cleanup of ephemeral snapshots happens after startup checks pass.
patch by Matt Byrd; reviewed by Stefan Miklosovic, Brandon Williams and Francisco Guerrero for CASSANDRA-21246
* cassandra-6.0:
Differentiate between legitimate cases where the first entry is the same as the last entry and empty bounds in SSTableCursorWriter#addIndexBlock()
Add an extra optimization if it is enabled: invoke Thread.setName only if a phase changed
patch by Dmitry Konstantinov; reviewed by Francisco Guerrero, Jyothsna Konisa for CASSANDRA-21089
patch by C. Scott Andreas; reviewed by Caleb Rackliffe and Dmitry Konstantinov for CASSANDRA-21285
Co-authored-by: C. Scott Andreas <cscotta@users.noreply.github.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
With negotiated authentication (CEP-50), nodes may be configured with multiple authenticators. Prior to this change,
a number of areas in the code assumed that there was a single configured authenticator and contained logic that
switched depending on the authenticator type. This logic won't work when multiple authenticators can be configured.
This change eliminates most calls to DataDescriptor.getAuthenticator(), by enabling individual authenticators to
declare the role attributes they support, requiring callers to specify the type of authenticator they're looking
for, and directly returning whether the node can enforce authn or not rather than inferring it by the presence of
an authenticator.
Testing done: Unit tests for auth and config packages; d-tests for auth-related functionality (e.g. ColumnMasks).
patch by Joel Shepherd; reviewed by Stefan Miklosovic, Andy Tolbert for CASSANDRA-20834
The same idea was applied previously in UnfilteredSerializer#serializeRowBody. A state is passed into a consumer function using the existing helper object.
patch by Dmitry Konstantinov; reviewed by Caleb Rackliffe, Francisco Guerrero for CASSANDRA-21289
Includes bug fixes and features:
- Improved observability in AutoRepair (CASSANDRA-20581)
- Stop repair scheduler if two major versions detected (CASSANDRA-20048)
- Safeguard Full repair against disk protection (CASSANDRA-20045)
- Stop AutoRepair monitoring thread upon shutdown (CASSANDRA-20623)
- Fix race condition in auto-repair scheduler (CASSANDRA-20265)
- Minimum repair task duration setting (CASSANDRA-20160)
- Preview_repaired auto-repair type (CASSANDRA-20046)
- Gate auto-repair behind cassandra.autorepair.enable JVM property
- Add cassandra.autorepair.check_min_version to gate minimum version enforcement
- Prevent auto-repair from running if any node is below 5.0.7
- Make system_distributed auto-repair schema conditional on feature being enabled
- Add user-friendly errors for disabled auto-repair and schema incompatibility
patch by Paulo Motta; reviewed by Andy Tolbert, Jaydeepkumar Chovatia for CASSANDRA-21138
Co-Authored-By: Andy Tolbert <andy_tolbert@apple.com>
Co-Authored-By: Chris Lohfink <clohfink@netflix.com>
Co-Authored-By: Francisco Guerrero <frankgh@apache.org>
Co-Authored-By: Himanshu Jindal <himanshj@amazon.com>
Co-Authored-By: Jaydeepkumar Chovatia <jchovati@uber.com>
Co-Authored-By: Kristijonas Zalys <kzalys@uber.com>
Co-Authored-By: jaydeepkumar1984 <chovatia.jaydeep@gmail.com>
this diff was supposed to be included in 5f9de8ca13
Rate limit password changes
Introduces property cassandra.role_password_update_interval_in_ms, with default value of 5000.
Logging added when passwords on roles are changed or are rate-limited.
patch by Mick Semb Wever; reviewed by Doug Rohrer, Jeremiah Jordan for CASSANDRA-21202
Add an implementation of isExpired to ReusableLivenessInfo, before the change the method implementation was interited from an interface and always returned false, so livenessInfo().supersedes(...) did not work properly.
patch by Dmitry Konstantinov; reviewed by Branimir Lambov, Nitsan Wakart for CASSANDRA-21152
Best effort attempt to get tableId from CFS, fallback to null which, still allows snapshot to be loaded
End result is that nodetool clearsnapshot and listsnapshot function for such snapshots on restart.
patch by Matt Byrd; reviewed by Stefan Miklosovic, Brandon Williams and Francisco Guerrero for CASSANDRA-21173
We can disable saving of the history either via command-line parameter --disable-history, or by setting disabled = True in the history section of the cqlshrc. Both options will read existing history, and just won't save new commands.
Update help and docs for cqlsh history.
Add startup info logline whenr history logging is enabled.
Add a fix for cqlshrc file path not correctly expanding.
Includes the Backport of
Allows users to change cqlsh history location using env variable
patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-17448
patch by Ekaterina Dimitrova; reviewed by Mick Semb Wever for CASSANDRA-XXX
Co-authored-by: Alex Ott alex.ott@datastax.com
Co-authored-by: Jaroslaw Grabowski jaroslaw.grabowski@datastax.com
Rewrites ANN search query execution logic to more efficiently merge graph search results using
similarity score-ordered (descending) iterators to merge segments efficiently. Allows for reduced memory
consumption during queries, reduced impact of overwrites and tombstones, selective re-querying of
minimally necessary graphs, and reduced shuffling of PrimaryKey objects.
patch by Michael Marshall; reviewed by Caleb Rackliffe and Michael Semb Wever for CASSANDRA-20086
Only inflight MSOs which affect the local node need to be
applied to determine the settled local ranges
Patch by Sam Tunnicliffe; reviewed by Marcus Eriksson for
CASSANDRA-21144
Introduces property cassandra.role_password_update_interval_in_ms, with default value of 5000.
Logging added when passwords on roles are changed or are rate-limited.
patch by Mick Semb Wever; reviewed by Doug Rohrer, Jeremiah Jordan for CASSANDRA-21202
* Expand affected ranges to include removals
* Don't create waiters for replica groups of future ranges not yet in placements
Patch by Sam Tunnicliffe; reviewed by Marcus Eriksson for
CASSANDRA-21142
Make MemoryUtil.clean() throw for slices/duplicates that cannot be cleaned and fix mmap cleanup
Parameterise main compaction tests for cursor/iterator and buffered/direct combinations
patch by Sam Lightfoot; reviewed by Ariel Weisberg, Dmitry Konstantinov for CASSANDRA-21147
This is a partial solution to IllegalStateException thrown by VectorPostings. It works by using a single shard at L0 when a vector index is present. As noted in the jira ticket, there are edge cases that may still produce errors, notably the case where there are multiple data directories.
The key trade offs here are related to the time complexity for search. Since graph search is log(n), and searching m graphs is m * log(n), we see better search performance by building bigger graphs which is essentially log(m * n). We could pre-shard, which comes at a cost of increased search time complexity.
patch by Michael Marshall,Dmitry Konstantinov; reviewed by Caleb Rackliffe,Dmitry Konstantinov,Michael Semb Wever for CASSANDRA-19661
Co-authored-by: Michael Marshall <mmarshall@apache.org>
Co-authored-by: Dmitry Konstantinov <netudima@gmail.com>
Introduce the ability to override compaction strategy for specific keyspaces
and tables at startup via two new system properties:
- cassandra.override_compaction.entities: comma-separated list of keyspaces
and keyspace.table pairs (e.g. "ks1,ks2.tbl1,ks3.tbl2")
- cassandra.override_compaction.params: JSON compaction parameters to apply
Patch by Paulo Motta; Reviewed by Jaydeepkumar Chovatia for CASSANDRA-21169
It restores the original approach when data files are read only in an extended mode, as it takes more time
It fixes sstable.VerifyTest.testVerifyCorruptRowCorrectDigest test failure
patch by Arvind Kandpal; reviewed by Dmitry Konstantinov,Brandon Williams for CASSANDRA-21150
avoid ClusteringIndexSliceFilter allocation if a write does not required a read (plain usual write), avoid iterator allocation, use array instead of ArrayList for perStatementOptions which does not grow dynamically
BatchStatement: check if many similar rows for the same table are written unconditionally, in this case we can avoid columns info merging and builders allocation
add fast path in nonTokenRestrictionValues logic, when we specify a single partition key (a single row) to modify, optimize also the case if a partition or clustering key is a single column
add fast path in valuesAsClustering logic, when we specify a single clustering key (a single row) to modify
UpdateParameters: allocate DeletionTime on demand (it is not needed if we do insert/updates)
serializedRowBodySize: avoid capturing lamda allocation per cell by moving capturing arguments to SerializationHelper (same optimization as it was done in serializeRowBody)
force hash3_x64_128 inlining to help JIT with escape analysis and long[] heap allocation elimination
patch by Dmitry Konstantinov; reviewed by Chris Lohfink for CASSANDRA-21141
This change adds support for using O_DIRECT (direct I/O) when reading SSTables
during compaction. Direct I/O bypasses the OS page cache, which can be beneficial
for compaction workloads where data is typically read once and not accessed again
soon after.
Key changes:
- Add DiskAccessMode.direct option for scan operations
- Introduce DirectThreadLocalByteBufferHolder and DirectThreadLocalReadAheadBuffer
for aligned buffer management required by O_DIRECT
- Add ByteBufferHolder interface to abstract buffer management
- Modify FileHandle to support building direct I/O readers via toBuilder()
- Add startup check to verify Direct IO support on configured data directories
- Enable read-ahead for uncompressed data in Direct CompressedChunkReader
- Move direct IO support logic into FileHandle (from SSTableReader)
- Add comprehensive tests for direct I/O chunk readers and buffer holders
patch by Sam Lightfoot; reviewed by Ariel Weisberg and Maxwell Guo for CASSANDRA-19987
Legacy Cassandra 3.x sstables use an old bloom filter format that is
incompatible with zero-copy streaming in Cassandra 5.0+. This patch
automatically detects sstables with the old bloom filter format
(pre-4.0) and disables zero-copy streaming for them, allowing legacy
sstables to be loaded via sstableloader without requiring manual flags.
The fix adds a version check in CassandraOutgoingFile.computeShouldStreamEntireSSTables()
that calls descriptor.version.hasOldBfFormat() to detect legacy sstables.
patch by Paulo Motta; reviewed by Stefan Miklosovic for CASSANDRA-21092
This commit adds a new configuration, data_disk_usage_keyspace_wide_protection_enabled, which ensures that if any node which replicates a given keyspace is full, all writes to that keyspace are blocked.
patch by Isaac Reath; reviewed by Stefan Miklosovic, Paulo Motta for CASSANDRA-20124
Closes#4547
add more flushing stats: partitions/rows, bytes rate, CPU and heap allocation for the flushing thread
avoid columns filtering overheads for unfilteredIterator
do not re-map colums in serializeRowBody if they haven't changed
reduce allocations during serialization of NativeClustering
add fast return for BTreeRow.hasComplexDeletion, avoid column.name.bytes.hashCode if not needed, avoid capturing lambda allocation in UnfilteredSerializer.serializeRowBody
check if Guardrails enabled at the beginning of writing, avoid hidden auto-boxing for logging of primitive parameters
split call sites for in Cell serialize logic, make isCounterCell cheaper (avoid megamorphic call + cache isCounterColumn)
invoke metadataCollector.updateClusteringValues only for first and last clustering key in a partition
enforce inlining for MinMaxIntTracker/MinMaxLongTracker
Patch by Dmitry Konstantinov; reviewed by Branimir Lambov for CASSANDRA-21083
Avoid BTree iterators allocation in ColumnsCollector.update
reduce amout of virtual calls for Cell.dataSize
avoid capturing lambda allocation in BTreeRow.clone
do not recalculate minLocalDeletionTime when BTreeRow is cloned
avoid CAS write if EncodingStats is not changed after merge
Patch by Dmitry Konstantinov; reviewed by Jyothsna Konisa, Michael Semb Wever for CASSANDRA-21088
It is also possible to override them via nodetool if necessary.
This patch also fixes the computation of sampling ratio to not lose the precision.
patch by Stefan Miklosovic; reviewed by Jyothsna Konisha, Yifan Cai for CASSANDRA-21078
Adds a compaction implementation utilizing new fixed allocation SSTable reader/writer implementations, and other purpose built code, leading to improved efficiencies.
patch by Nitsan Wakart; reviewed by Branimir Lambov, Dmitry Konstantinov for CASSANDRA-20918
By default the expiry time is calculated on each peer independently. It can be
made to converge by disabling gossip quarantine using the configuration setting
gossip_quarantine_disabled or via a hotprop on GossiperMBean.
Patch by Sam Tunnicliffe; reviewed by Marcus Eriksson for CASSANDRA-21035
* cassandra-5.0:
Fix CQLSSTableWriter serialization of vector of date and time patch by Lukasz Antoniak; reviewed by Andres de la Pena, Yifan Cai for CASSANDRA-20979
Use a plain loop to check if it is ASCII symbol before going into more complicated UTF8 parsing.
Avoid ValueAccessor to get extra boost for the ASCII check, especially in non-monomorphic cases.
Patch by Dmitry Konstantinov; reviewed by Jyothsna Konisa, Stefan Miklosovic for CASSANDRA-21075
Additionally replace List with array for bind values (we know the size in advance during a decoding), so in total: List<List> is replaced with byte[][] QueryOptions classes support both ways to get values now: using an old API with ByteBuffer and a new API with byte[].
Patch by Dmitry Konstantinov; reviewed by Michael Semb Wever for CASSANDRA-20166
Converting collections or UDTs to raw bytes is nonsensical - it
allows reading raw serialized bytes which have no meaningful
interpretation.
patch by Mikołaj Diakowski; reviewed by Stefan Miklosovic, Brandon Williams for CASSANDRA-20982
Dmitry Konstantinov