* cassandra-5.0:
Add LittleEndianMemoryUtil and NativeEndianMemoryUtil, switch memtable-related off-heap objects and Memory to use them and have Little Endian now. Add BE offsets detection on Summary loading. Add test SSTables in an old format with BE offsets in Summary component to LegacySSTableTest.
Add BE offsets detection on Summary loading.
Add test SSTables in an old format with BE offsets in Summary component to LegacySSTableTest.
Patch by Dmitry Konstantinov; reviewed by Branimir Lambov, Michael Semb Wever for CASSANDRA-20190
- Share TableMetadatas and PartitionKey for PartialTxn serialization
Also:
- TxnReference et al should reference uniqueId, and avoid serializing ksName/cfName
- Don't double count shared keys when estimating size on heap
patch by Benedict; reviewed by David Capwell for CASSANDRA-20578
This removes the usage of index files during compaction and simplifies
and improves the performance of compaction.
patch by Branimir Lambov; reviewed by Sylvain Lebresne for CASSANDRA-20092
Also improve:
- TxnId serialization
- StoreParticipants serialization
- compareUnsigned Node.Id for consistency with serialized TxnId
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20546
* avoid running python3-dependent tasks when running simulator tasks
* fix a problem with simulated snitch rebase
* add a distinction between short-lived daemon threads and infinite loop ones for cases when we need to simulate user-implemented infinite loops
Patch by Alex Petrov; reviewed by Benedict Elliott Smith for CASSANDRA-20542
Improve:
- InMemoryJournal compaction should simulate files to ensure we compact the same cohorts of records (so shadowing applied correctly)
- Don't update CFK deps if already known
- avoid unnecessary heapification of LogGroupTimers
- begin removal of Guava (mostly unused)
- consider medium path on fast path delayed
- add Seekables.indexOf to support faster serialization
- unboxed Invariant.requires variant
- AsyncChains.addCallback -> invoke
Fix:
- Recovery must wait for earlier transactions on shards that have not yet been accepted, even if we recover a fast Stable record on another shard
- only skip Dep calculation on PreAccept if vote is required by coordinator
- ReadTracker must slice Minimal the first unavailable collection
- If PreLoadContext cannot acquire shared caches, still consider existing contents
- CFK: make sure to insert UNSTABLE into missing array
- Fix failed task stops DelayedCommandStore task queue processing further tasks
- short circuit AbstractRanges.equals()
- Handle edge case where we can take fast/medium path but one of the Deps replies contains a future TxnId
- update executeAtEpoch when retryInFutureEpoch
- Fix incorrect validation in validateMissing (should only validate newInfo is not in missing when in witnessedBy; all other additions should be included)
patch by Benedict; reviewed by David Capwell for CASSANDRA-20522
* Fix short accord simulation test (seed 0x6bea128ae851724b), ConcurrentModificationException
* Increase wait time during closing to avoid Unterminated threads
* Increase timeouts, improve test stability
* More descriptive output from CQL test
* Shorten max CMS delay
* Improve future handling in config service
Patch by Alex Petrov; reviewed by Benedict Elliott Smith for CASSANDRA-20440
- Accord Journal purging was disabled
- remove unique_id from schema keyspace
- avoid String.format in Compactor hot path
- avoid string concatenation on hot path; improve segment compactor partition build efficiency
- Partial compaction should update records in place to ensure truncation of discontiguous compactions do not lead to an incorrect field version being used
- StoreParticipants.touches behaviour for RX was erroneously modified; should touch all non-redundant ranges including those no longer owned
- SetShardDurable should correctly set DurableBefore Majority/Universal based on the Durability parameter
- fix erroneous prunedBefore invariant
- Journal compaction should not rewrite fields shadowed by a newer record
- Don't save updates to ERASED commands
- Simplify CommandChange.getFlags
- fix handling of Durability for Invalidated
- Don't use ApplyAt for GC_BEFORE with partial input, as might be a saveStatus >= ApplyAtKnown but with executeAt < ApplyAtKnown
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20441
- Decouple command serialization from TableMetadata version; introduce ColumnMetadata ids; gracefully handle missing TableId
- DataInputPlus.readLeastSignificantBytes must truncate high bits
- Fix RandomPartitioner accord serialization
- Fast path stable commits must not override recovery propose/commit decisions regarding visibility of a transaction
- RejectBefore must mergeMax, not merge, to ensure we maintain epoch and hlc increasing independently
- Bad commitInvalidate decision
- consistent filtering for touches and stillTouches
- ensure TRUNCATE_BEFORE implies SHARD_APPLIED
- TopologyManager.unsyncedOnly off-by-one error
- DurabilityQueue should not retry SyncPointErased
- handle rare case of no deps but none needed
- not updating CFK synchronously on recovery, which can lead to erroneous recovery decisions for other transactions
- Don't return partial read response when one commandStore rejects the commit
- Filter touches/stillTouches consistently
- WaitingState computeLowEpoch must use hasTouched to handle historic key with no route
Improve:
- Use format parameters to defer building Invariants.requireArgument string
- streamline RedundantStatus/RedundantBefore
Also improve:
- Introduce DurabilityService
- Retire SyncPoint, replace Barrier with Write and RX
- MessageType -> enum, restore GetMaxConflict
- Standardise backoff logic with WaitStrategy
- improve TimeoutStrategy/RetryStrategy specification strings
- Forbid KX, remove directKeyDeps
- Introduce UniqueTimeService, permitting hlc reservations for sync points avoid delay when min TxnId is sufficiently in the past
- Remove ListStore custom purge logic
Also fix:
- RejectBefore should reject on both epoch and hlc
- Do not record sync success for removed nodes
- Support GlobalDurability detecting no command store to run on
- Incorrect ballot constructor
- Serializing 15-bit ballot flags incorrectly
- TopologyManager.hasEpoch deadlock
- Computing withOpenEpochs incorrectly, sometimes stopping one epoch short
- PartitionKey serializer should not depend on schema information that can be erased
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20395
- Bad ArrayBuffers recycling logic
- RX must ensure dependencies TRANSITIVE_VISIBLE
- Permit constructing "antiRange" that spans multiple prefixes
- Not computing range CommandSummary IsDep correctly
- Truncated commands that aren't shard durable could not repopulate CFK on replay, permitting recovery of another command to make an incorrect decision
- NPE on async persist of RX (i.e. supplying no callback)
- NPE in Builder.shouldCleanup when durability is null
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20370
- Only use persisted RedundantBefore for compaction
- RouteIndex should index only touches, not Route
- Flush RangesForEpoch updates to journal immediately, so we do not rely on the command we are processing succeeding
- DurableBefore updates must wait for the epochs to be known locally
- Shard.mustWitnessEpoch to support guaranteeing to witness relevant non-topology schema changes
- We must propagate RedundantBefore RX shard bounds along with epoch syncs
- Prevent a truncated transaction FetchData infinite loop
- GC_BEFORE status being overwritten by bootstrappedAt, permitting old transaction state to be resurrected
- Avoid CFK.maxUniqueHlc read race on bootstrap
- TopologyManager.awaitEpoch could wait for wrong epoch
- Journal fsync thread could miss notifications
Also improve:
- CommandStores uses SearchableRangeList for finding matching stores
- Refactor RedundantBefore to use a sorted array of TxnId/RedundantStatus pairs (to better fix GC_BEFORE issue)
- Accord debug keyspace operates on keyspace/table, and sorts correctly by token
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20361
Improve:
- Introduce pre/accept fast execution flags
- Introduce searchable Deps serialization
- Flatten AccordRoutingKey(s) into single type, using sentinel bits
- Introduce new fast byte-comparable serialization methods for Token and TableId to support above
Fix:
- Fix journal re-serialization logic
- Enable RandomPartitioner for Accord by supporting fixed-width serialization for RouteIndex
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20349
Also fix:
- Topology slicing must declare whether we share/slice node ownership (to assist above)
- CFK.visit removes transitive dependencies too eagerly across epoch change
- apply cleanup to builder consistently, and construct the same value we would produce by purge (so that replay is idempotent)
- Invoke ExecuteTxn.LocalExecute callbacks on originating CommandStore
- misc other minor issues
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20325
Doug Rohrer