Also fix:
- Catch exceptions thrown by Task.cleanupExclusive to ensure we do not corrupt SequentialExecutor state
- ConcurrentModificationException with CommandStore.waitingOnSync
- Exclude promisedHlc that are in the future, even if not equal to Long.MAX_VALUE
- Filter CFK on load as not expected to be up to date (for later on demand filtering validation)
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20821
since later quorums may include the bootstrapping node which does not participate in the durability of preceding transactions
Also Improve:
- Apply MaxDecidedRX filtering to CommandsForKey RX dependencies
- Optimise AbstractRanges.sliceMinimal (burn test hotspot)
- Don't start shard schedulers on topology changes if not already started
- Only registerTransitive if waitingOnSync
- Reserve DurabilityRequest until ShardDurability is started
- Don't notify progress log if stopped
- Remove duplicated CFK unmanaged filtering
Also fix:
- Edge case in notification across bootstrap boundary by resubmitting Unmanaged to be recomputed
- Serialization of CommandsForKey.TxnInfo.missing() collection when non-identity flag bits are present
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20811
- Schedule a fallback timeout for active requests to ensure progress with lost callbacks
- Clear active task for a txnId when new RunInvoker is registered
- Consult DurableBefore prior to invoking recovery
- Support user invoked reset of a command, clearing any active work and requeueing it
- (Testing): Treat progress log for RX as recurring tasks for burn test termination
Also:
- Add TxnOps RECOVER, FETCH, RESET_PROGRESS_LOG
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20815
- Remove from CFK any unapplied transactions we know cannot apply
- Force notification of waiting commands in CFK on replay
Also fix:
- Don't truncateWithOutcome if pre bootstrap or stale
- Fix RangeDeps.without(RangeDeps)
- Fix InMemoryCommandStore replay bug with clearing DefaultLocalListeners
- Ensure SaveStatus and executeAt are updated together to prevent corruption via expunge
Improve:
- Inform home shard that command is decided if we cannot execute, to avoid recovery contention
- Don't recover sync points on the fast path
- Don't calculate recovery info for RX (since we don't use it anymore, so no need to do the work)
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20797
- DurableBefore.fullyContainedIn should compare txnId
Also fix:
- Messages with nullable state should check if they are cancelled after reading the state but before using it
- Don't snapshot during replay
- CFK.updateRedundantBefore incorrectly updating mayExecute and sharing byId in some cases
- Refine GC with global durability bound
- Fix JournalGCTest
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20786
Also fix:
- Limit truncation to TruncatedApplyWithOutcome until data is persisted durably to local store
- IntervalUpdater not invoking super.close()
- Do not invoke preRunExclusive without holding lock
- IntervalUpdater not correctly initialise BranchBuilder.inUse
- AccordExecutor should notify if more work on unlock of caches
- Relax paranoid CFK validation during restart
Also improve:
- Flush logs before System.exit
- Start/stop progress log explicitly
- Limit progress log concurrency
- Clear heavy fields in some messages once processed
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20780
- PreLoadContext descriptions
- Introduce LoadKeysFor so we can avoid loading range transactions except where necessary
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20758
Also fix:
- slowCoordinatorDelay calculation for locally truncated command throws ISE
- Bad cast of Truncated during replay
- Don't throw IllegalStateException in Invariants.expect if accord.testing == false
- Replay of empty segment throws NPE
Also improve:
- Support tracing of recovery and home progress
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20771
- Don't try to update metrics when no TxnId (e.g. GetMaxConflict)
- maybeExecuteImmediately did not guarantee mutual exclusivity
- Cancellation of a running 'plain' task could corrupt AccordExecutor state
- GetLatestDeps message serializer
- txn_blocked_by StackOverflowError
- Not updating CommandsForKey in all cases on restart
Also Improve:
- TableId.from/toString
- Route toString methods
- Tracing coverage of FetchRoute
- Replay command store parallelism
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20763
- WatermarkCollector should not report same closed/retired epoch N times
- AccordSyncPropagator can merge pending requests and back-off retries
- AccordCommandLoader should notify listeners
- AccordSegmentCompactor should estimate number of keys to ensure bloom filters work
- txn_blocked_by table should report what it can, not throw IllegalStateException
- Permit uncompressed system tables
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20754
Also Fix:
- Initialise home state when calling waiting() if not already initialised
- Don't reportClosed/reportRetired for epochs that are already closed/retired
Also Improve:
- Implement waitForQuiescence in AccordExecutor
- Permit replay parallelism
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20750
- Journal debugging vtable support
- Background task tracing support
Fix:
- HLC_BOUND only valid for strictly lower HLC
- HAS_UNIQUE_HLC can only be safely computed if READY_TO_EXECUTE
- Break recursion in CommandStore.ensureReadyToCoordinate
- Fix find intersecting shard scheduler
- Separate adhoc ShardScheduler from normal to avoid overwriting
- Should still use execution listeners to detect invalid reads for certain transactions even with dataStoreDetectsFutureReads
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20746
- Lock inversion when restarting durability for erased sync point
- Initialise durability cycle start time
Improve:
- Improve durability reporting
- Timeout durability requests
- DurabilityQueue concurrency should limit only until quorum is achieved
- Some exceptions that may be thrown when starting coordination may not be propagated
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20328
- Cannot shrink CommandsForKey if loading pruned
- NoSpamLogger to suppress AccordSyncPropagator repeats
- Minor performance improvement to BTree.Subtraction
- EphemeralReads should retry in a later epoch if replication factor changes
- Fix no local epoch for NotAccept
- Invalidate a command with no route but for which we know we own some key that has applied locally
- Don't pre-merge existing DurableBefore, to reduce duplicate/redundant persistence
- Misc purging bugs and improve testing of purging
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20739
- Fix shouldCleanup handling of erase/expunge
- Fix CommandChange handling of minUniqueHlc being cleared
- Don't clear minUniqueHlc when fast applying; instead simply validate !isWaiting
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20726
- Attempt to fix CommandsForKey StackOverflowError (presumed to be reachable via SaveStatus->InternalStatus map returning null)
- Bound recursion with SafeCommandStore.tryRecurse()
- IndexOutOfBoundsException in CINTIA checkpoint list encoding bounds logic
- Maintain MaxDecidedRX to save majority of work when deciding RX that are newer than those we have previously agreed
- Introduce IntervalBTree and use in CommandsForRanges to limit time spent in critical section when there are millions of RX
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20727
- Introduce SequentialAgentExecutor
- ExecuteEphemeralRead.LocalExecute (matching ExecuteTxn.LocalExecute)
- remove AgentExecutor (agent() only used as implementation detail)
- Skip CommandStore/CommandsForKey on read or apply, when safe to do so
- Implement maybeExecuteImmediately
- Faster maxTimestamp
Fix:
- Marking vestigial without knowing all covering keys
- Incorrect size in WaitingOn.none
- Cleanup RX that are not known but are no longer needed, due to being defunct
- Home should not abort because of local truncation; may still be incomplete on another shard
- Invalidate infinite loop due to interpreting Vestigial as a possible decision (as though it were another Truncated state)
- Known.isTruncated reporting incorrect answer in some cases, leading to infinite RecoverWithRoute loops as not slicing to correct subset and some shards are truncated and reject the deps collection
- BurnTest slowCoordinatorDelay should grow with retryCount
- NotAccept should check both ballot and saveStatus; PreCommitted or later can be propagated by Apply that has an earlier ballot
- Handle awaiting locally retired epoch
- Fix propagate low epoch of sync points
- Fix GetEphemeralReadDeps.reduce NPE
- Don't use tombstones to ERASE journal entries, as want to produce an Erased SaveStatus until EXPUNGE
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20726
Also Fix:
- slowCoordinatorDelay should bound its start point, and log more detail if its expectations are breached
- Erased SaveStatus can be reported for all queried owned participants on a replica (the saved participants have been erased)
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20722
- AccordJournal should not attempt to construct expunged commands
- ready/notReady logic can break if we receive a partial Route that does not cover the local store
- Invariant.require -> TopologyRetiredException
- validate min/max in TopologyManager.preciseEpochs
Improve:
- ExecuteSyncPoint should send Apply
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20713
- Do not query local topology when deciding what keys to fetch to avoid TopologyRetiredException
- Ensure we propagate information back to the requesting CommandStore by using the store id to ensure it is included
- BurnTest topology fetching was broken by earlier patch
- Topology callbacks were not being invoked as we were not calling .begin()
- Topology mismatch failure during notAccept phase was not being reported due to CoordinatePreAccept already having isDone==true
patch by Benedict; reviewed by David Capwell for CASSANDRA-20711
- Don't assume stillExecutes applies to remote request - must mark unavailable any keys we don't have the txn definition for
- Don't exit notifyManagedPreBootstrap notify loop early, as could have later transactions with lower txnId
- CoordinateEphemeralRead must retry if insufficient responses from replicas that still own the range
- Burn test terminates while non-recurring tasks pending on command store queues
- Infinite recovery due to not sending InformDurable when partially truncated
- Incorrect participants when invoking removeRedundantDependencies
- Infinite bootstrap loop on retired topology
Improve
- Do not perform linear filters of CommandStores or Topologies
- Some default implementations of forEach/iterator
- TopologyRetiredException messages
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20707
- cfk pruning+prebootstrap=invalid future dependency
- exclude retired ranges when filtering RX stillTouches
- propagate uses incorrect lowEpoch when fetch finds additional owned/touched ranges
- node.withEpoch should callback with TopologyRetiredException, not throw
- Recovery can race with durable-applied pruning; must not send durable unless latest ballot on apply
- removeRedundantDependencies was not slicing pre-bootstrap range calculation to participating ranges
- NPE in TopologyManager.atLeast caused by referencing an epoch that has been GC'd
- use journal durableBeforePersister in burn test, not NOOP_PERSISTER
- ServerUtils.cleanupDirectory use tryDeleteRecursive
- FsyncRunnable shutdown
- fix NPE in AccordJournalBurnTest
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20688
- Share TableMetadatas and PartitionKey for PartialTxn serialization
Also:
- TxnReference et al should reference uniqueId, and avoid serializing ksName/cfName
- Don't double count shared keys when estimating size on heap
patch by Benedict; reviewed by David Capwell for CASSANDRA-20578
Also improve:
- TxnId serialization
- StoreParticipants serialization
- compareUnsigned Node.Id for consistency with serialized TxnId
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20546
Improve:
- InMemoryJournal compaction should simulate files to ensure we compact the same cohorts of records (so shadowing applied correctly)
- Don't update CFK deps if already known
- avoid unnecessary heapification of LogGroupTimers
- begin removal of Guava (mostly unused)
- consider medium path on fast path delayed
- add Seekables.indexOf to support faster serialization
- unboxed Invariant.requires variant
- AsyncChains.addCallback -> invoke
Fix:
- Recovery must wait for earlier transactions on shards that have not yet been accepted, even if we recover a fast Stable record on another shard
- only skip Dep calculation on PreAccept if vote is required by coordinator
- ReadTracker must slice Minimal the first unavailable collection
- If PreLoadContext cannot acquire shared caches, still consider existing contents
- CFK: make sure to insert UNSTABLE into missing array
- Fix failed task stops DelayedCommandStore task queue processing further tasks
- short circuit AbstractRanges.equals()
- Handle edge case where we can take fast/medium path but one of the Deps replies contains a future TxnId
- update executeAtEpoch when retryInFutureEpoch
- Fix incorrect validation in validateMissing (should only validate newInfo is not in missing when in witnessedBy; all other additions should be included)
patch by Benedict; reviewed by David Capwell for CASSANDRA-20522
- Accord Journal purging was disabled
- remove unique_id from schema keyspace
- avoid String.format in Compactor hot path
- avoid string concatenation on hot path; improve segment compactor partition build efficiency
- Partial compaction should update records in place to ensure truncation of discontiguous compactions do not lead to an incorrect field version being used
- StoreParticipants.touches behaviour for RX was erroneously modified; should touch all non-redundant ranges including those no longer owned
- SetShardDurable should correctly set DurableBefore Majority/Universal based on the Durability parameter
- fix erroneous prunedBefore invariant
- Journal compaction should not rewrite fields shadowed by a newer record
- Don't save updates to ERASED commands
- Simplify CommandChange.getFlags
- fix handling of Durability for Invalidated
- Don't use ApplyAt for GC_BEFORE with partial input, as might be a saveStatus >= ApplyAtKnown but with executeAt < ApplyAtKnown
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20441
- Decouple command serialization from TableMetadata version; introduce ColumnMetadata ids; gracefully handle missing TableId
- DataInputPlus.readLeastSignificantBytes must truncate high bits
- Fix RandomPartitioner accord serialization
- Fast path stable commits must not override recovery propose/commit decisions regarding visibility of a transaction
- RejectBefore must mergeMax, not merge, to ensure we maintain epoch and hlc increasing independently
- Bad commitInvalidate decision
- consistent filtering for touches and stillTouches
- ensure TRUNCATE_BEFORE implies SHARD_APPLIED
- TopologyManager.unsyncedOnly off-by-one error
- DurabilityQueue should not retry SyncPointErased
- handle rare case of no deps but none needed
- not updating CFK synchronously on recovery, which can lead to erroneous recovery decisions for other transactions
- Don't return partial read response when one commandStore rejects the commit
- Filter touches/stillTouches consistently
- WaitingState computeLowEpoch must use hasTouched to handle historic key with no route
Improve:
- Use format parameters to defer building Invariants.requireArgument string
- streamline RedundantStatus/RedundantBefore
Also improve:
- Introduce DurabilityService
- Retire SyncPoint, replace Barrier with Write and RX
- MessageType -> enum, restore GetMaxConflict
- Standardise backoff logic with WaitStrategy
- improve TimeoutStrategy/RetryStrategy specification strings
- Forbid KX, remove directKeyDeps
- Introduce UniqueTimeService, permitting hlc reservations for sync points avoid delay when min TxnId is sufficiently in the past
- Remove ListStore custom purge logic
Also fix:
- RejectBefore should reject on both epoch and hlc
- Do not record sync success for removed nodes
- Support GlobalDurability detecting no command store to run on
- Incorrect ballot constructor
- Serializing 15-bit ballot flags incorrectly
- TopologyManager.hasEpoch deadlock
- Computing withOpenEpochs incorrectly, sometimes stopping one epoch short
- PartitionKey serializer should not depend on schema information that can be erased
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20395
- Bad ArrayBuffers recycling logic
- RX must ensure dependencies TRANSITIVE_VISIBLE
- Permit constructing "antiRange" that spans multiple prefixes
- Not computing range CommandSummary IsDep correctly
- Truncated commands that aren't shard durable could not repopulate CFK on replay, permitting recovery of another command to make an incorrect decision
- NPE on async persist of RX (i.e. supplying no callback)
- NPE in Builder.shouldCleanup when durability is null
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20370
- Only use persisted RedundantBefore for compaction
- RouteIndex should index only touches, not Route
- Flush RangesForEpoch updates to journal immediately, so we do not rely on the command we are processing succeeding
- DurableBefore updates must wait for the epochs to be known locally
- Shard.mustWitnessEpoch to support guaranteeing to witness relevant non-topology schema changes
- We must propagate RedundantBefore RX shard bounds along with epoch syncs
- Prevent a truncated transaction FetchData infinite loop
- GC_BEFORE status being overwritten by bootstrappedAt, permitting old transaction state to be resurrected
- Avoid CFK.maxUniqueHlc read race on bootstrap
- TopologyManager.awaitEpoch could wait for wrong epoch
- Journal fsync thread could miss notifications
Also improve:
- CommandStores uses SearchableRangeList for finding matching stores
- Refactor RedundantBefore to use a sorted array of TxnId/RedundantStatus pairs (to better fix GC_BEFORE issue)
- Accord debug keyspace operates on keyspace/table, and sorts correctly by token
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20361
Improve:
- Introduce pre/accept fast execution flags
- Introduce searchable Deps serialization
- Flatten AccordRoutingKey(s) into single type, using sentinel bits
- Introduce new fast byte-comparable serialization methods for Token and TableId to support above
Fix:
- Fix journal re-serialization logic
- Enable RandomPartitioner for Accord by supporting fixed-width serialization for RouteIndex
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20349
Also fix:
- Topology slicing must declare whether we share/slice node ownership (to assist above)
- CFK.visit removes transitive dependencies too eagerly across epoch change
- apply cleanup to builder consistently, and construct the same value we would produce by purge (so that replay is idempotent)
- Invoke ExecuteTxn.LocalExecute callbacks on originating CommandStore
- misc other minor issues
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20325
Use initializeTopologyUnsafe to for-load the last topology and create shards rather than replaying all topologies.
Patch by Alex Petrov; reviewed by Benedict Elliott Smith for CASSANDRA-20294
- Even if we can decide autonomously that we took the fast path, we must still wait for earlier transactions to decide themselves
- We must update a command that is in CFK.loadingPruned whether or not it is outOfRange
- We must visit pruned commands that are transitive dependencies of RX to ensure dependencies are propagated
- flagsWithoutDomainAndKind() -> flagsWithoutDomainOrKindOrCardinality() [to fix integration regression]
- Don't invoke uniqueNow() twice when allocate nextTxnId
- BTreeReducingRangeMap edge case that corrupts map when merge function does not return one of its inputs
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20292
Benedict Elliott Smith