CommandChanges.shouldCleanup short-circuits to NO if there is no data, but this is incorrect as Cleanup.EXPUNGE may have dropped the data and the record must receive cleanup EXPUNGE and be reported as ERASED.
Also Fix:
- Populate CFK system table row markers so we can stop reading sstables as soon as we have data
- system_accord_debug.executors has wrong clustering type
- RemoteToLocalVirtualTable should lazily allocate the partition collector to avoid LIMIT clause filtering removing it before it's populated
- ActiveEpochs.withNewEpochs should handle transition from 0 -> more than 1
- RedundantBefore.minGcBefore should be NONE if empty
- Update RangesForEpoch directly, so that we cannot have race conditions where the ownership is unknown
- Avoid reentrancy on local callbacks
- Ensure ReadCoordinator callbacks are invoked on owning thread
- Avoid deadlock when notifying ComplexListener(s)
- Release IntrusivePriorityHeap memory from large capacity heaps when empty
- Prevent SynchronousRecoverAwait reentrancy when invoking onDone (by exposing and invoking invokeOnDone that first sets isDone)
- maybeExecute must invoke either notWaiting or notifyWaiting to ensure tryExecuteListening terminates
Also Improve:
- Configurable execute_waiting_on_start
patch by Benedict; reviewed by Alan Wang and Alex Petrov for CASSANDRA-21440
- consumer and producer threads wait signal without acquiring the lock first
- lock owners may prepare more work than they need, moving some dynamically-adjusted portion of the work from the prioritised lock-managed structures onto a non-blocking queue, so that other threads may consume work from there; the portion is continually micro-adjusted to target some available work whenever the lock is acquired.
- adopts/supports some features of the SEPExecutor:
- threads may auto-adjust the number of running threads based on how much time is collectively spent waiting, to minimise time spent signalling
- consumers may (timed-sleep) poll rather than await a signal, reducing the number of kernel interactions needed; relying on the auto-adjustment to bound the time the scheduler spends waking threads with no work to do
Also Fix:
- Client Result should not be persisted or included in Command object at any time
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for CASSANDRA-21375
- TransactionStatement token aware routing
- Mitigate convoy effect:
- ExecuteTxnBacklog v1 can execute single key transaction backlogs and disseminate the result
- Direct local execution possible for single key transactions
- Use shardAppliedBefore instead of gcBefore to cleanup CFK faster
Also Improve:
- Introduce distributed tracing
- Combine RejectBefore and MaxConflicts
- Self-addressed messages are delivered directly
- Disable AccordSyncPropagator by default, as incompatible with large clusters
- FastPathStrategy.SIMPLE should not modify fast path contents; FastPathStrategy.UP introduced to adopt this behaviour
- Configurable queue prioritisation
- ProtocolModifiers mostly final
- Remove unnecessary condition check when serializing TxnUpdate
Also Fix:
- SequentialExecutor ownership bug
- BTree.XUpdater.reset()
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for CASSANDRA-21361
- Clean Shutdown/Restart
- Rebootstrap to allow nodes to rejoin consensus if they are out of sync, or did not shutdown cleanly
Improve:
- Improve efficiency of BTreeReducingRangeMap
- DurableBefore backed by BTreeReducingRangeMap
- Soft reject new transactions when a replica has a backlog of work
- system_accord_debug.command_store_ops supports starting journal replay
Fix:
- Ensure SequentialExecutor.owner is unset only by the owner
- Detect and avoid taking two AccordExecutor locks simultaneously
- MaxConflicts serializer
- tryToExecuteListening after replay, to handle invalidated dependencies
- TxnNamedRead does not close a RowIterator, leading to native memory leaks
- GetLatestDepsNack serialization
- journal.readLast() did not read last
- DefaultRemoteListeners not correctly synchronised
- RangeTxnScanner cancellation assumes was already started
- Start cfk compaction before replay
- txn_graph should avoid visiting parents twice (possible if two different dependency chains connecting them in the graph)
- Topology.cloneEquivalentWithEpoch should also share nodeLookup and ranges, especially to accelerate computeWaitForEpoch/computeScope
- Do not invoke slowReplicaDelay or slowCoordinatorDelay during restore
- Do not fail if slowReplicaDelay or slowCoordinatorDelay are invoked without knowing the transaction
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for CASSANDRA-21355
Also Add:
- txn_graph and txn_graph_all virtual tables for debugging transaction dependency graphs
- txn_cache virtual table to debug accord cache contents
Also Fix:
- NPE in AccordDebugKeyspace.TxnOpsTable
- AccordCoordinatorMetrics latency metrics should use Timer not Histogram
- OnDiskIndex.readLast
Also Improve:
- Catchup should run after progress log is started
- Cancelling AccordTask should early-terminate range scanning
- Improve RangeTxnScanner description
- Add ballot to txn tables
- Add NodeId to logging output
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21306
Also Fix Cassandra:
- In memory size calculation for CommandsForKey include Unmanaged
- Accord load out-of-band cleanup should use SafeRedundantBefore
ALso Improve Cassandra:
- Report replay information on begin replay
- Improve AccordService shutdown
- Log command store RedundantBefore on shutdown
- Segment compaction should wait for readOrder barrier to replace segments, for additional safety
- Journal segments should share readOrder with sstables
Also Improve Accord:
- Iterate LocalListeners in order, so can query more effectively on node
- Refine AbstractReplay.minReplay/shouldReplay
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21804
- DefaultLocalListeners.ComplexListeners iterator IndexOutOfBoundsException
- Race condition initialising empty ActiveEpochs, when minimum pending epoch can move backwards
- SyncPoints must be declared in an epoch containing the ranges, and PENDING_REMOVAL ranges will reject non-syncpoint transactions
- AccordExecutorMetrics is now registered on startup
- getRecentValues for non-cumulative histogram should not subtract prior values
Improve:
- Report ephemeral read, epoch waits and timeout metrics
- Remove Topologies.SelectNodeOwnership, as no need to SLICE anymore
- Introduce SystemEventListener for epoch waiting and timeout metrics
- No-op but log if gcBefore provided to CFK is in the past
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21076
- DurabilityQueue/ShardScheduler deadlock
- MemtableCleanerThread.Cleanup assumes Boolean parameter is non-null, which is invalid if an exception has been thrown
- AccordDurableOnFlush may be invoked while Accord is starting up, so should use AccordService.unsafeInstance
- AccordCache shrink without lock regression
- Cleanup system_accord compaction leftovers before starting up
- system_accord_debug.txn order
- system_accord_debug.txn_blocked_by order
- system_accord_debug.shard_epochs order
Improve:
- Set DefaultProgressLog.setMode(Catchup) during Catchup
- IdentityAccumulators only need to readLast, not readAll
- Limit number of static segments we compact at once to sstable
- If too many static segments on startup, wait for them to be compacted
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21053
Also Fix:
- Restore MaxDecidedRX on replay
- When catchup_on_start_exit_on_failure == false, should startup on any kind of failure, not only timeout
- lazy vtable LIMIT clause regression
- DurabilityService.onEpochRetired
- Command.validate when uniqueHlc differs
- Avoid unsafe publication of AccordExecutor to scheduledFastTasks
- AccordCache hitRate metric names
- use long for return type of DurationSpec.toNanoseconds
- Repair without all replicas should not request all Accord replicas participate
- ExecuteAtSerializer
- SyncPoints should be coordinated in an epoch that contains the ranges
Also Improve:
- Split Accord startup into local+distributed, ensure we
- Add logging to FetchDurableBefore on startup
- Add randomised testing of AbstractLazyVirtualTable
- Add validation of lazy virtual table key ordering
- Don't send requests to faulty replicas
- shrinkOrEvict large objects without holding lock
- Accord dtest shutdown
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21042
Also Introduce:
- Sharded/LogLinearDecayingHistogram
Also Improve:
- Do not take a reference to CFK unless relevant
Also Fix:
- Sharded/LogLinearHistogram
- ExecuteFlags serialization bug in ReadData
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21017
- All retries are appended to a delay queue so overlaps can be pruned
- Quorum successes are not retried if there are superseding sync points covering the ranges
- User-initiated requests are not purged unless the request has timed out or otherwise completed
- Overlapping requests are queued up against the next to run
Alsp (C*):
- Catch-up with quorums on restart
- Manage an ordered set of keys in cache for faster range searches
Also (Accord):
- Update copy of BTree and import IntervalBTree
- Fix RedundantStatus WAS_OWNED_OVERRIDE_MASK
- Add Catchup mechanism to reach parity with a quorum on restart
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21013
Merge ConfigurationService with TopologyManager to remove cyclic dependency and duplicated work.
Also:
- Improve visibility of work blocking topology processing
- Ensure we cannot double-count peers when deciding an epoch's distributed readiness
- Remove the possibility of distributed stalls, by processing new topologies as soon as a
contiguous sequence is known locally, regardless of whether anyprior local epoch is ready to
coordinate or has recorded this fact to peers. The notification of local readiness continues
to be processed only once all prior epochs are ready, but we can begin using and readying
later epochs immediately.
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20998
Also fix:
- Paxos should not update minHlc if AccordService is not setup
- Remove EndpointMapper methods that require non-null, to ensure call-sites handle null case explicitly
- AccordRepair should specify the Node Ids that must participate, but these should not affect the durability requirements for GC
- Avoid erroneously marking UniversalOrInvalidated when only known to quorum
- AccordSyncPropagator should not consult FailureDetector as it may throw UnknownEndpointException
- Refuse ReplaceSameAddress if accord enabled
- Reset Accord topologies on restart if no local command stores to ensure we can catch up (as intervening epochs may have otherwise been GC'd)
- MaybeRecover should not abort if found durable if either: 1) prevProgressToken already witnessed durable; or 2) Home state has requested we recover anyway
- Invariant in MaybeRecover is too strong now we permit recovering when known to be stable
- ReadData must respond with InsufficientAndWaiting if not Stable to ensure Durability is inferred correctly by ExecuteTxn
- Avoid StackOverflowException in NotifyWaitingOn
- PreAccept should honour the REJECTED flag of any member of the latest topology if the vote would be relied upon for any prior epoch's quorum
- Resume Accord bootstrap on restart
- Fix burn test non-determinism
- Reset Accord topologies on restart if no local command stores to ensure we can catch up (as intervening epochs may have otherwise been GC'd)
- Exclude stale or removed ids from sync points
- Terminate a failed DurabilityRequest if a required participant is now removed/hardRemoved
Also improve:
- Introduce accord_debug_keyspace.{listeners_txn, listeners_local}
- Filter and record faulty in AbstractCoordination
- If we are the home shard, and our home progress status is done, we should not wait for the home shard to advance the waiting progress state
- Introduce CommandStore.tryExecuteListening to try to execute all waiting transactions and their dependencies
- ExecuteTxn on recovery should not wait, to avoid consuming progress log concurrency slots for transactions that cannot execute
- HomeState should update its phase based on the command status to handle operator request to retry all states
- Separate home/wait queue in DefaultProgressLog
- Additional tracing
- Do not call trySendMore from prerecordFailure
- Force full recovery for sync points if we have lost quorum
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20921
- Introduce pattern tracing, that can intercept failed or new coordinations matching various filters
- Support additional tracing event collection modes (SAMPLE and RING)
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20911
To support recovering a node that has lost some of its local transaction log, introduce rebootstrap and unsafe bootstrap modes, where Accord ensures no responses are produced for transactions the node cannot be certain it had not previously answered.
patch by Benedict and Alex Petrov for CASSANDRA-20908
Also Fix:
- RegisteredCallback should remove itself from callback map when cancelled
- Do not throw CancellationException when processing requests that have been aborted, as may be caused by a successful meaningful reply that can be overridden
- system_accord_debug.{executors,coordinations} fail with ClassCastException
- CommandStore.updateMinHlc eats up CPU as called much too often
- AccordCache not notifying flushed on shutdown
Also Improve:
- Support skipping Deps
- Violation information reported
- Sort CommandStore shards by id
patch by Benedict; reviewed by Aleksey Yeschenko for CASSANDRA-20896
Also Fix:
- JournalAndTableKeyIterator not merging in consistent order, which can lead to replaying topologies in non-ascending order
- Invariant failure when reporting topologies if fetchTopologies on startup yields a gap
- removeRedundantMissing could erroneously remove missing dependencies occurring after the new appliedBeforeIndex
- Invariant failure for some propagate calls supplying 'wrong' addRoute
- Disambiguate nextTxnId and nextTxnIdWithDefaultFlags
Also Improve:
- LocalLog should not use NoSuchElementException for control flow (instead use ConcurrentSkipListMap)
patch by Benedict; reviewed by Aleksey Yeschenko for CASSANDRA-20886
- JournalAndTableKeyIterator not merging in consistent order
- Track all active Coordinations
- Refactor Replica/Coordinator metrics and report Coordinator exhausted/preempted/timeout
- DurabilityQueue metrics and visibility
Also Fix:
- WaitingState can get cause distributed stall when asked to wait for CanApply if not yet PreCommitted; track separate querying state and advance this to the next achievable state rather than the desired final state
- Stalled coordinators should not prevent recovery
- Edge case with fetch unable to make progress when pre-bootstrap and all peers have GC'd
- Dependency initialisation for sync points across certain ownership changes
- SyncPoint propagation may not include all of the epochs required on the receiving node for ranges they have lost but not closed, and receiving node does not validate them
- Stable tracker accounting with LocalExecute
- Do not prune non-durable APPLIED as must be reported in dependencies until durably applied (so as not to break recovery)
- Ensure we cannot race with replies when initiating Coordination
- ProgressLog does not guarantee to clear home or waiting states when erased or invalidated by compaction
- WaitingState on non-home shard cannot guarantee progress once home shard is Erased
- WaitingOnSync handles retired ranges incorrectly
Also Improve:
- Standardise failure accounting, use null to represent single reply timeouts
- BurnTest record/replay to/from file
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20878
- Fix erroneous call to registerWithDelay with absolute deadline
- ExecuteSyncPoint should discount votes from unbootstrapped replicas
- Bootstrap no longer needs to first ensure durability (now ExecuteSyncPoint discounts votes from unbootstrapped replicas)
- Progress stall with home shard stopping before Stable is propagated to all shards
- Only upgrade Durability.HasPhase if we have queried all shards
- Incorrectly inferring Durability in CheckStatus.finish
- Incorrectly inferring/propagating stable from fast unstable reply; clarify to prevent further mistakes
- MaxDecidedRX should track separate hlc bounds for filtering non-RX dependencies
- minGcBefore.hlc() can move backwards across epochs, even if each shard moves forwards; must track minHlc directly
- slowTimeout init in ExecuteTxn.LocalExecute
- TxnId.parse for RV
Also improve:
- Don't query recovery state if fast path durably decided
- Support deferred partial dep deserialisation
- Support both orientations of KeyDeps/RangeDeps
- Support partialDeps as ByteBuffer
Also improve in Cassandra:
- Deps serialization and Journal skipping
- Don't inflate when reading from cache for CommandsForRanges
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20847
and more reliably avoid initialising recovery progress log state of transactions that cannot yet make progress
Also fix:
- Harden AccordExecutor state cleanup to failures
- Handle SAVING state in AccordCache.tryEvict, as now possible to save for reasons besides eviction so normal to both be in evict queue and saving
- Infer invalid in MaybeRecover and FetchData
- MaybeRecover sometimes aborts before home shard knows outcome
- Epoch sync with VisibilitySyncPoint
- Retired implies synced
- Don't interpret force repair as excluding nodes from Accord sync conditions
- TxnData.without
Also improve:
- Add Topology.removedNodes
- If Durability implies we can fetch a status, update the waiting state to fetch it
- DurableBefore debug table should have searchable txnId
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20832
Also fix:
- Catch exceptions thrown by Task.cleanupExclusive to ensure we do not corrupt SequentialExecutor state
- ConcurrentModificationException with CommandStore.waitingOnSync
- Exclude promisedHlc that are in the future, even if not equal to Long.MAX_VALUE
- Filter CFK on load as not expected to be up to date (for later on demand filtering validation)
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20821
since later quorums may include the bootstrapping node which does not participate in the durability of preceding transactions
Also Improve:
- Apply MaxDecidedRX filtering to CommandsForKey RX dependencies
- Optimise AbstractRanges.sliceMinimal (burn test hotspot)
- Don't start shard schedulers on topology changes if not already started
- Only registerTransitive if waitingOnSync
- Reserve DurabilityRequest until ShardDurability is started
- Don't notify progress log if stopped
- Remove duplicated CFK unmanaged filtering
Also fix:
- Edge case in notification across bootstrap boundary by resubmitting Unmanaged to be recomputed
- Serialization of CommandsForKey.TxnInfo.missing() collection when non-identity flag bits are present
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20811
- Schedule a fallback timeout for active requests to ensure progress with lost callbacks
- Clear active task for a txnId when new RunInvoker is registered
- Consult DurableBefore prior to invoking recovery
- Support user invoked reset of a command, clearing any active work and requeueing it
- (Testing): Treat progress log for RX as recurring tasks for burn test termination
Also:
- Add TxnOps RECOVER, FETCH, RESET_PROGRESS_LOG
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20815
- Remove from CFK any unapplied transactions we know cannot apply
- Force notification of waiting commands in CFK on replay
Also fix:
- Don't truncateWithOutcome if pre bootstrap or stale
- Fix RangeDeps.without(RangeDeps)
- Fix InMemoryCommandStore replay bug with clearing DefaultLocalListeners
- Ensure SaveStatus and executeAt are updated together to prevent corruption via expunge
Improve:
- Inform home shard that command is decided if we cannot execute, to avoid recovery contention
- Don't recover sync points on the fast path
- Don't calculate recovery info for RX (since we don't use it anymore, so no need to do the work)
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20797
- DurableBefore.fullyContainedIn should compare txnId
Also fix:
- Messages with nullable state should check if they are cancelled after reading the state but before using it
- Don't snapshot during replay
- CFK.updateRedundantBefore incorrectly updating mayExecute and sharing byId in some cases
- Refine GC with global durability bound
- Fix JournalGCTest
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20786
Also fix:
- Limit truncation to TruncatedApplyWithOutcome until data is persisted durably to local store
- IntervalUpdater not invoking super.close()
- Do not invoke preRunExclusive without holding lock
- IntervalUpdater not correctly initialise BranchBuilder.inUse
- AccordExecutor should notify if more work on unlock of caches
- Relax paranoid CFK validation during restart
Also improve:
- Flush logs before System.exit
- Start/stop progress log explicitly
- Limit progress log concurrency
- Clear heavy fields in some messages once processed
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20780
- PreLoadContext descriptions
- Introduce LoadKeysFor so we can avoid loading range transactions except where necessary
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20758
Also fix:
- slowCoordinatorDelay calculation for locally truncated command throws ISE
- Bad cast of Truncated during replay
- Don't throw IllegalStateException in Invariants.expect if accord.testing == false
- Replay of empty segment throws NPE
Also improve:
- Support tracing of recovery and home progress
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20771
- Don't try to update metrics when no TxnId (e.g. GetMaxConflict)
- maybeExecuteImmediately did not guarantee mutual exclusivity
- Cancellation of a running 'plain' task could corrupt AccordExecutor state
- GetLatestDeps message serializer
- txn_blocked_by StackOverflowError
- Not updating CommandsForKey in all cases on restart
Also Improve:
- TableId.from/toString
- Route toString methods
- Tracing coverage of FetchRoute
- Replay command store parallelism
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20763
- WatermarkCollector should not report same closed/retired epoch N times
- AccordSyncPropagator can merge pending requests and back-off retries
- AccordCommandLoader should notify listeners
- AccordSegmentCompactor should estimate number of keys to ensure bloom filters work
- txn_blocked_by table should report what it can, not throw IllegalStateException
- Permit uncompressed system tables
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20754
Also Fix:
- Initialise home state when calling waiting() if not already initialised
- Don't reportClosed/reportRetired for epochs that are already closed/retired
Also Improve:
- Implement waitForQuiescence in AccordExecutor
- Permit replay parallelism
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20750
- Journal debugging vtable support
- Background task tracing support
Fix:
- HLC_BOUND only valid for strictly lower HLC
- HAS_UNIQUE_HLC can only be safely computed if READY_TO_EXECUTE
- Break recursion in CommandStore.ensureReadyToCoordinate
- Fix find intersecting shard scheduler
- Separate adhoc ShardScheduler from normal to avoid overwriting
- Should still use execution listeners to detect invalid reads for certain transactions even with dataStoreDetectsFutureReads
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20746
- Lock inversion when restarting durability for erased sync point
- Initialise durability cycle start time
Improve:
- Improve durability reporting
- Timeout durability requests
- DurabilityQueue concurrency should limit only until quorum is achieved
- Some exceptions that may be thrown when starting coordination may not be propagated
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20328
- Cannot shrink CommandsForKey if loading pruned
- NoSpamLogger to suppress AccordSyncPropagator repeats
- Minor performance improvement to BTree.Subtraction
- EphemeralReads should retry in a later epoch if replication factor changes
- Fix no local epoch for NotAccept
- Invalidate a command with no route but for which we know we own some key that has applied locally
- Don't pre-merge existing DurableBefore, to reduce duplicate/redundant persistence
- Misc purging bugs and improve testing of purging
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20739
- Fix shouldCleanup handling of erase/expunge
- Fix CommandChange handling of minUniqueHlc being cleared
- Don't clear minUniqueHlc when fast applying; instead simply validate !isWaiting
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20726
Benedict Elliott Smith