Set 1 thread and 1 second for the measurement iteration (note: it is a smoke test only, we do not analyze results from such short measurements)
Exclude some options for heavy benchmarks by default
Shutdown all threads once JMH test is completed to avoid awaiting of a forked JVM till timeout at the end
patch by Dmitry Konstantinov; reviewed by Michael Semb Wever for CASSANDRA-21388
- now() functions must be deterministic (and derived from Accord timestamp)
- tombstone GC and TTL evaluation must be deterministic and based on both Accord timestamp and Accord GC mechanisms (that guarantee completeness of execution)
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for CASSANDRA-21376
- consumer and producer threads wait signal without acquiring the lock first
- lock owners may prepare more work than they need, moving some dynamically-adjusted portion of the work from the prioritised lock-managed structures onto a non-blocking queue, so that other threads may consume work from there; the portion is continually micro-adjusted to target some available work whenever the lock is acquired.
- adopts/supports some features of the SEPExecutor:
- threads may auto-adjust the number of running threads based on how much time is collectively spent waiting, to minimise time spent signalling
- consumers may (timed-sleep) poll rather than await a signal, reducing the number of kernel interactions needed; relying on the auto-adjustment to bound the time the scheduler spends waking threads with no work to do
Also Fix:
- Client Result should not be persisted or included in Command object at any time
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for CASSANDRA-21375
The following subcommands have been added:
* abortbootstrap Cancel and undo an inflight bootstrap
* abortmove Cancel and undo an inflight token move
* abortdecommission Cancel and undo an inflight decommission
* assassinate Remove a node from cluster metadata
* resetcms Redefine the CMS membership
* move Modify a node's (single) owned token
* describe Output a summary of the cluster metadata
* forcejoin Force a registered or joining node into a JOINED state
* print Output a full rendering of cluster metadata
* printdirectory Output a full rendering of the directory
* printdataplacements Output the full data placements
Patch by Venkata Harikrishna Nukala; reviewed by Alex Petrov and Sam Tunnicliffe
for CASSANDRA-19151
- TransactionStatement token aware routing
- Mitigate convoy effect:
- ExecuteTxnBacklog v1 can execute single key transaction backlogs and disseminate the result
- Direct local execution possible for single key transactions
- Use shardAppliedBefore instead of gcBefore to cleanup CFK faster
Also Improve:
- Introduce distributed tracing
- Combine RejectBefore and MaxConflicts
- Self-addressed messages are delivered directly
- Disable AccordSyncPropagator by default, as incompatible with large clusters
- FastPathStrategy.SIMPLE should not modify fast path contents; FastPathStrategy.UP introduced to adopt this behaviour
- Configurable queue prioritisation
- ProtocolModifiers mostly final
- Remove unnecessary condition check when serializing TxnUpdate
Also Fix:
- SequentialExecutor ownership bug
- BTree.XUpdater.reset()
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for CASSANDRA-21361
use compression ratio to scale the amount of remaining disk usage
CompactionInfo::estimatedRemainingWriteBytes is uncompressed, causing compaction to be rejected due to size when it has space to complete
patch by Matt Byrd; reviewed by Dmitry Konstantinov, Marcus Eriksson for CASSANDRA-21245
Co-authored-by: Jon Meredith <jmeredithco@apple.com>
- Clean Shutdown/Restart
- Rebootstrap to allow nodes to rejoin consensus if they are out of sync, or did not shutdown cleanly
Improve:
- Improve efficiency of BTreeReducingRangeMap
- DurableBefore backed by BTreeReducingRangeMap
- Soft reject new transactions when a replica has a backlog of work
- system_accord_debug.command_store_ops supports starting journal replay
Fix:
- Ensure SequentialExecutor.owner is unset only by the owner
- Detect and avoid taking two AccordExecutor locks simultaneously
- MaxConflicts serializer
- tryToExecuteListening after replay, to handle invalidated dependencies
- TxnNamedRead does not close a RowIterator, leading to native memory leaks
- GetLatestDepsNack serialization
- journal.readLast() did not read last
- DefaultRemoteListeners not correctly synchronised
- RangeTxnScanner cancellation assumes was already started
- Start cfk compaction before replay
- txn_graph should avoid visiting parents twice (possible if two different dependency chains connecting them in the graph)
- Topology.cloneEquivalentWithEpoch should also share nodeLookup and ranges, especially to accelerate computeWaitForEpoch/computeScope
- Do not invoke slowReplicaDelay or slowCoordinatorDelay during restore
- Do not fail if slowReplicaDelay or slowCoordinatorDelay are invoked without knowing the transaction
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for CASSANDRA-21355
Best effort attempt to get tableId from CFS, fallback to null which
still allows snapshot to be loaded
End result is that nodetool clearsnapshot and listsnapshot function for such
snapshots on restart
In addition to 5.0 patch we move starting SnapshotManager to after TCM is initiliazed.
The consequence of such a move is that cleanup of ephemeral snapshots happens after startup checks pass.
patch by Matt Byrd; reviewed by Stefan Miklosovic, Brandon Williams and Francisco Guerrero for CASSANDRA-21246
* cassandra-6.0:
Differentiate between legitimate cases where the first entry is the same as the last entry and empty bounds in SSTableCursorWriter#addIndexBlock()
In 6.0, only rename test/data/serialization/5.1 to 6.0
In 7.0, same plus update edges in UpgradeTestBase
patch by Stefan Miklosovic; reviewed by Dmitry Konstantinov for CASSANDRA-21319
Also Add:
- txn_graph and txn_graph_all virtual tables for debugging transaction dependency graphs
- txn_cache virtual table to debug accord cache contents
Also Fix:
- NPE in AccordDebugKeyspace.TxnOpsTable
- AccordCoordinatorMetrics latency metrics should use Timer not Histogram
- OnDiskIndex.readLast
Also Improve:
- Catchup should run after progress log is started
- Cancelling AccordTask should early-terminate range scanning
- Improve RangeTxnScanner description
- Add ballot to txn tables
- Add NodeId to logging output
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21306
With negotiated authentication (CEP-50), nodes may be configured with multiple authenticators. Prior to this change,
a number of areas in the code assumed that there was a single configured authenticator and contained logic that
switched depending on the authenticator type. This logic won't work when multiple authenticators can be configured.
This change eliminates most calls to DataDescriptor.getAuthenticator(), by enabling individual authenticators to
declare the role attributes they support, requiring callers to specify the type of authenticator they're looking
for, and directly returning whether the node can enforce authn or not rather than inferring it by the presence of
an authenticator.
Testing done: Unit tests for auth and config packages; d-tests for auth-related functionality (e.g. ColumnMasks).
patch by Joel Shepherd; reviewed by Stefan Miklosovic, Andy Tolbert for CASSANDRA-20834
Includes bug fixes and features:
- Improved observability in AutoRepair (CASSANDRA-20581)
- Stop repair scheduler if two major versions detected (CASSANDRA-20048)
- Safeguard Full repair against disk protection (CASSANDRA-20045)
- Stop AutoRepair monitoring thread upon shutdown (CASSANDRA-20623)
- Fix race condition in auto-repair scheduler (CASSANDRA-20265)
- Minimum repair task duration setting (CASSANDRA-20160)
- Preview_repaired auto-repair type (CASSANDRA-20046)
- Gate auto-repair behind cassandra.autorepair.enable JVM property
- Add cassandra.autorepair.check_min_version to gate minimum version enforcement
- Prevent auto-repair from running if any node is below 5.0.7
- Make system_distributed auto-repair schema conditional on feature being enabled
- Add user-friendly errors for disabled auto-repair and schema incompatibility
patch by Paulo Motta; reviewed by Andy Tolbert, Jaydeepkumar Chovatia for CASSANDRA-21138
Co-Authored-By: Andy Tolbert <andy_tolbert@apple.com>
Co-Authored-By: Chris Lohfink <clohfink@netflix.com>
Co-Authored-By: Francisco Guerrero <frankgh@apache.org>
Co-Authored-By: Himanshu Jindal <himanshj@amazon.com>
Co-Authored-By: Jaydeepkumar Chovatia <jchovati@uber.com>
Co-Authored-By: Kristijonas Zalys <kzalys@uber.com>
Co-Authored-By: jaydeepkumar1984 <chovatia.jaydeep@gmail.com>
this diff was supposed to be included in 5f9de8ca13
Rate limit password changes
Introduces property cassandra.role_password_update_interval_in_ms, with default value of 5000.
Logging added when passwords on roles are changed or are rate-limited.
patch by Mick Semb Wever; reviewed by Doug Rohrer, Jeremiah Jordan for CASSANDRA-21202
Add an implementation of isExpired to ReusableLivenessInfo, before the change the method implementation was interited from an interface and always returned false, so livenessInfo().supersedes(...) did not work properly.
patch by Dmitry Konstantinov; reviewed by Branimir Lambov, Nitsan Wakart for CASSANDRA-21152
Best effort attempt to get tableId from CFS, fallback to null which, still allows snapshot to be loaded
End result is that nodetool clearsnapshot and listsnapshot function for such snapshots on restart.
patch by Matt Byrd; reviewed by Stefan Miklosovic, Brandon Williams and Francisco Guerrero for CASSANDRA-21173
Rewrites ANN search query execution logic to more efficiently merge graph search results using
similarity score-ordered (descending) iterators to merge segments efficiently. Allows for reduced memory
consumption during queries, reduced impact of overwrites and tombstones, selective re-querying of
minimally necessary graphs, and reduced shuffling of PrimaryKey objects.
patch by Michael Marshall; reviewed by Caleb Rackliffe and Michael Semb Wever for CASSANDRA-20086
Only inflight MSOs which affect the local node need to be
applied to determine the settled local ranges
Patch by Sam Tunnicliffe; reviewed by Marcus Eriksson for
CASSANDRA-21144
Introduces property cassandra.role_password_update_interval_in_ms, with default value of 5000.
Logging added when passwords on roles are changed or are rate-limited.
patch by Mick Semb Wever; reviewed by Doug Rohrer, Jeremiah Jordan for CASSANDRA-21202
* Expand affected ranges to include removals
* Don't create waiters for replica groups of future ranges not yet in placements
Patch by Sam Tunnicliffe; reviewed by Marcus Eriksson for
CASSANDRA-21142
Make MemoryUtil.clean() throw for slices/duplicates that cannot be cleaned and fix mmap cleanup
Parameterise main compaction tests for cursor/iterator and buffered/direct combinations
patch by Sam Lightfoot; reviewed by Ariel Weisberg, Dmitry Konstantinov for CASSANDRA-21147
This is a partial solution to IllegalStateException thrown by VectorPostings. It works by using a single shard at L0 when a vector index is present. As noted in the jira ticket, there are edge cases that may still produce errors, notably the case where there are multiple data directories.
The key trade offs here are related to the time complexity for search. Since graph search is log(n), and searching m graphs is m * log(n), we see better search performance by building bigger graphs which is essentially log(m * n). We could pre-shard, which comes at a cost of increased search time complexity.
patch by Michael Marshall,Dmitry Konstantinov; reviewed by Caleb Rackliffe,Dmitry Konstantinov,Michael Semb Wever for CASSANDRA-19661
Co-authored-by: Michael Marshall <mmarshall@apache.org>
Co-authored-by: Dmitry Konstantinov <netudima@gmail.com>
Introduce the ability to override compaction strategy for specific keyspaces
and tables at startup via two new system properties:
- cassandra.override_compaction.entities: comma-separated list of keyspaces
and keyspace.table pairs (e.g. "ks1,ks2.tbl1,ks3.tbl2")
- cassandra.override_compaction.params: JSON compaction parameters to apply
Patch by Paulo Motta; Reviewed by Jaydeepkumar Chovatia for CASSANDRA-21169
Add test container memory events printing to check if Linux OOM killer was active
Fix thread dump printing on junit test timeout
Process.pid() API is available since JDK9
patch by Dmitry Konstantinov; reviewed by Michael Semb Wever for CASSANDRA-21172
Too many threads can lead to OOM, so new Thread is replaced with an executor
spin wait is added to HintsServiceTest.testPageSeek to avoid flaky NPE (HintsStore is populated in an async way)
patch by Dmitry Konstantinov; reviewed by Brandon Williams for CASSANDRA-21166
The runtime overflow guard test (testLargeIndexSummary) required ~2 GiB of
off-heap memory to trigger the Integer.MAX_VALUE boundary in
IndexSummaryBuilder.maybeAddEntry. SafeMemoryWriter's 50% buffer growth
strategy caused ~5 GiB peak memory during reallocation (old + new buffer),
exceeding CI medium node limits (3.5-5 GB).
Extract the hardcoded Integer.MAX_VALUE threshold in IndexSummaryBuilder
into a @VisibleForTesting field (maxEntriesSize), allowing the test to
exercise the same guard logic with a 2 MB threshold instead. Rename both
large summary tests to reflect their distinct code paths: runtime truncation
vs constructor-initiated downsampling.
patch by Sam Lightfoot; reviewed by Brandon Williams,Dmitry Konstantinov,Michael Semb Wever for CASSANDRA-20599
The two deletes use the same CQL timestamp, so compaction should merge
their overlapping range tombstones into a single open/close pair (no
boundary markers). However, DeletionTime also includes local_delete_time
(wall-clock seconds), which may differ between deletes if a second
boundary is crossed during the flush. Differing local_delete_time causes
the merger to treat them as distinct deletions, producing boundary
markers instead. Using executeInternalWithNowInSec with a fixed nowInSec
ensures both deletes get identical DeletionTime values.
patch by Sam Lightfoot; reviewed by Dmitry Konstantinov,Brandon Williams for CASSANDRA-21163
This is unnecessary code as the current implementation of CEP-54 is not implemeting / exercising these features.
patch by Stefan Miklosovic; reviewed by Yifan Cai for CASSANDRA-21154
This change adds support for using O_DIRECT (direct I/O) when reading SSTables
during compaction. Direct I/O bypasses the OS page cache, which can be beneficial
for compaction workloads where data is typically read once and not accessed again
soon after.
Key changes:
- Add DiskAccessMode.direct option for scan operations
- Introduce DirectThreadLocalByteBufferHolder and DirectThreadLocalReadAheadBuffer
for aligned buffer management required by O_DIRECT
- Add ByteBufferHolder interface to abstract buffer management
- Modify FileHandle to support building direct I/O readers via toBuilder()
- Add startup check to verify Direct IO support on configured data directories
- Enable read-ahead for uncompressed data in Direct CompressedChunkReader
- Move direct IO support logic into FileHandle (from SSTableReader)
- Add comprehensive tests for direct I/O chunk readers and buffer holders
patch by Sam Lightfoot; reviewed by Ariel Weisberg and Maxwell Guo for CASSANDRA-19987
Add microbench-test target for quick test run
Unify declaration and add assertions on target types
Fix dirname usages for macos
Parameter for docker_timeout
Add more jmh class ignores (that are broken)
Add disableResume() and better host debug to Jenkinsfile
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov for CASSANDRA-18873
Legacy Cassandra 3.x sstables use an old bloom filter format that is
incompatible with zero-copy streaming in Cassandra 5.0+. This patch
automatically detects sstables with the old bloom filter format
(pre-4.0) and disables zero-copy streaming for them, allowing legacy
sstables to be loaded via sstableloader without requiring manual flags.
The fix adds a version check in CassandraOutgoingFile.computeShouldStreamEntireSSTables()
that calls descriptor.version.hasOldBfFormat() to detect legacy sstables.
patch by Paulo Motta; reviewed by Stefan Miklosovic for CASSANDRA-21092
This commit adds a new configuration, data_disk_usage_keyspace_wide_protection_enabled, which ensures that if any node which replicates a given keyspace is full, all writes to that keyspace are blocked.
patch by Isaac Reath; reviewed by Stefan Miklosovic, Paulo Motta for CASSANDRA-20124
Closes#4547
Based on the official documentation here (1), AsyncProfiler works best with these kernel parameters.
Not all CI environments have them set. That would fail the tests which run the startup check
if kernel parameters are not like that.
It might be empirically observed that some AsyncProfiler functionality works even without these
kernel parameters, it might be investigated in follow-up work if we should not relax these conditions a bit.
(1) https://github.com/async-profiler/async-profiler/blob/master/docs/GettingStarted.md#before-profiling
It is also possible to override them via nodetool if necessary.
This patch also fixes the computation of sampling ratio to not lose the precision.
patch by Stefan Miklosovic; reviewed by Jyothsna Konisha, Yifan Cai for CASSANDRA-21078
Adds a compaction implementation utilizing new fixed allocation SSTable reader/writer implementations, and other purpose built code, leading to improved efficiencies.
patch by Nitsan Wakart; reviewed by Branimir Lambov, Dmitry Konstantinov for CASSANDRA-20918
By default the expiry time is calculated on each peer independently. It can be
made to converge by disabling gossip quarantine using the configuration setting
gossip_quarantine_disabled or via a hotprop on GossiperMBean.
Patch by Sam Tunnicliffe; reviewed by Marcus Eriksson for CASSANDRA-21035
* cassandra-5.0:
Fix CQLSSTableWriter serialization of vector of date and time patch by Lukasz Antoniak; reviewed by Andres de la Pena, Yifan Cai for CASSANDRA-20979
Also Fix Cassandra:
- In memory size calculation for CommandsForKey include Unmanaged
- Accord load out-of-band cleanup should use SafeRedundantBefore
ALso Improve Cassandra:
- Report replay information on begin replay
- Improve AccordService shutdown
- Log command store RedundantBefore on shutdown
- Segment compaction should wait for readOrder barrier to replace segments, for additional safety
- Journal segments should share readOrder with sstables
Also Improve Accord:
- Iterate LocalListeners in order, so can query more effectively on node
- Refine AbstractReplay.minReplay/shouldReplay
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21804
Use a plain loop to check if it is ASCII symbol before going into more complicated UTF8 parsing.
Avoid ValueAccessor to get extra boost for the ASCII check, especially in non-monomorphic cases.
Patch by Dmitry Konstantinov; reviewed by Jyothsna Konisa, Stefan Miklosovic for CASSANDRA-21075
- DefaultLocalListeners.ComplexListeners iterator IndexOutOfBoundsException
- Race condition initialising empty ActiveEpochs, when minimum pending epoch can move backwards
- SyncPoints must be declared in an epoch containing the ranges, and PENDING_REMOVAL ranges will reject non-syncpoint transactions
- AccordExecutorMetrics is now registered on startup
- getRecentValues for non-cumulative histogram should not subtract prior values
Improve:
- Report ephemeral read, epoch waits and timeout metrics
- Remove Topologies.SelectNodeOwnership, as no need to SLICE anymore
- Introduce SystemEventListener for epoch waiting and timeout metrics
- No-op but log if gcBefore provided to CFK is in the past
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21076
Additionally replace List with array for bind values (we know the size in advance during a decoding), so in total: List<List> is replaced with byte[][] QueryOptions classes support both ways to get values now: using an old API with ByteBuffer and a new API with byte[].
Patch by Dmitry Konstantinov; reviewed by Michael Semb Wever for CASSANDRA-20166
Converting collections or UDTs to raw bytes is nonsensical - it
allows reading raw serialized bytes which have no meaningful
interpretation.
patch by Mikołaj Diakowski; reviewed by Stefan Miklosovic, Brandon Williams for CASSANDRA-20982
- DurabilityQueue/ShardScheduler deadlock
- MemtableCleanerThread.Cleanup assumes Boolean parameter is non-null, which is invalid if an exception has been thrown
- AccordDurableOnFlush may be invoked while Accord is starting up, so should use AccordService.unsafeInstance
- AccordCache shrink without lock regression
- Cleanup system_accord compaction leftovers before starting up
- system_accord_debug.txn order
- system_accord_debug.txn_blocked_by order
- system_accord_debug.shard_epochs order
Improve:
- Set DefaultProgressLog.setMode(Catchup) during Catchup
- IdentityAccumulators only need to readLast, not readAll
- Limit number of static segments we compact at once to sstable
- If too many static segments on startup, wait for them to be compacted
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21053
Also Fix:
- Restore MaxDecidedRX on replay
- When catchup_on_start_exit_on_failure == false, should startup on any kind of failure, not only timeout
- lazy vtable LIMIT clause regression
- DurabilityService.onEpochRetired
- Command.validate when uniqueHlc differs
- Avoid unsafe publication of AccordExecutor to scheduledFastTasks
- AccordCache hitRate metric names
- use long for return type of DurationSpec.toNanoseconds
- Repair without all replicas should not request all Accord replicas participate
- ExecuteAtSerializer
- SyncPoints should be coordinated in an epoch that contains the ranges
Also Improve:
- Split Accord startup into local+distributed, ensure we
- Add logging to FetchDurableBefore on startup
- Add randomised testing of AbstractLazyVirtualTable
- Add validation of lazy virtual table key ordering
- Don't send requests to faulty replicas
- shrinkOrEvict large objects without holding lock
- Accord dtest shutdown
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21042
David Capwell